Magellan and the feed pond — bound IDs turn sites into living entities

systems/magellan-and-feeds.md · System · magellan, feeds, identity, calendar, accuracy, strategy

unverified: no trust signals recorded yet

The business_id_map spine (built v2.2.549) plus the feed adapters that hang off it. A bound external ID is what lets a generated page carry current, authoritative, self-maintained content instead of static prose.

Magellan and the feed pond

The operator direction (Aug 21 2026): "Cross link as much as we can and turn feeds, sites, and other directories into data feeds. Doozer sites need to be living entities."

The spine (BUILT — includes/magellan.php, v2.2.549)

business_id_map (business_id, service, external_id, source, confidence, verified_at), unique per (business_id, service), reverse-indexed on (service, external_id). This is the table specified in magellan-id-mapping in June 2026 and finally built. magellanBind/Id/Find/All/BackfillHot/Stats; dz magellan.

Live coverage: google 3,632 · gcal (growing) · ticketmaster 101.

The hot columns (businesses.google_place_id, businesses.tm_venue_id) are NOT touched; magellanBackfillHot() copies them IN so both views agree while callers migrate.

The rule stays: join by ID, never by name+city+state. Name matching guesses, duplicates, and fails silently.

Why a bound ID matters (the actual point)

A generated page describing a business is one of thousands and gets skipped. A page carrying what is on tonight is the only page answering that question. A bound ID is what makes the second thing possible, and it is also an accuracy engine: every feed is a second opinion on the facts.

Feed adapters

gcal — public Google Calendar (includes/feed-gcal.php, GENERIC)

A public Google Calendar exposes iCal at a predictable URL with no key and no quota. Any page embedding one names a live, self-maintained event feed.

<iframe src="...calendar/embed?src=<id>"> -> https://calendar.google.com/calendar/ical/<urlencoded id>/public/basic.ics

gcalIdsFromHtml (handles the addr@gmail.com and hex @group.calendar.google.com forms, and src not being the first query param), gcalFetchIcs, gcalParseUpcoming, gcalSyncBusiness. RFC 5545: line unfolding, VALUE=DATE, TZID, EXDATE, cancelled events, recurrence for FREQ=DAILY/WEEKLY/MONTHLY/YEARLY with INTERVAL/BYDAY/COUNT/ UNTIL, hard-bounded. BYSETPOS/BYWEEKNO are NOT expanded (stated, not hidden).

GOTCHA — timezone is a correctness bug, not cosmetics. Store the instant AND the calendar's tz, and format with gcalLocalString(). First pass showed San Diego's "Friday Dinner" as Saturday 00:00 because correct UTC instants were formatted in the server's zone. Events belong to the place that holds them.

GOTCHA — the reader strips iframes. Calendar discovery must fetch RAW HTML (a raw-HTML fetcher), not sourceFetchMarkdown(). This is why a calendar page looked empty and was nearly written off as unusable.

Member registries

Some organizations publish a registry with a page per member chapter: address, meeting times, officers, official site, social links, facilities hours, news, and often an embedded calendar. When the chapter's identifier is already part of the site's address, the path is deterministic and needs no search API key. One member-registry adapter is built on this pattern; its specifics are kept in a private node.

• Read the registry's robots.txt first, and keep to what it allows.
• Parse the body slice only. Everything outside the chapter's own section is the

registry's chrome, which made a national vendor link look like a chapter website on the first pass (a directions link did the same).

• Officers may be prose, not a labelled list. Parse sentences, not fields.
• A contact form is surfaced for a HUMAN, never auto-submitted.

Rules every adapter honours

• GEO-ANCHOR. Feed addresses are REFERENCE only, never applied to the geo-verified

location. Enrich the content, never the coordinates.

• Facts, never prose. Copying source prose is what caused a root's scaled-content

cliff. Extract data; let the site render it.

• No automated outbound. Reading public pages is fine. Submitting a third party's

contact form at scale is not, whatever the volume argument.

• Dry-run by default, live=1 writes (the dz harvest convention).
• Be a polite guest: delay between fetches, cache hard.

Next

Render the bound data on the site (facilities hours, upcoming events). The facts are captured but the page does not use them yet — the same gap [[project_foundation_sprint]] notes for source_facts. Then more services: a chapter's own site, regional associations, and other directories that publish structured member data.

Related: [[the-flywheel]], [[event-feeds]], [[money-clouds]], [[source-of-truth-cache]], [[project_where_the_traffic_is]].