Magellan and the feed pond — bound IDs turn sites into living entities

systems/magellan-and-feeds.md · System · magellan, feeds, identity, calendar, elks, accuracy, strategy

unverified: no trust signals recorded yet

The business_id_map spine (built v2.2.549) plus the feed adapters that hang off it. A bound external ID is what lets a generated page carry current, authoritative, self-maintained content instead of static prose.

Magellan and the feed pond

The operator direction (Aug 21 2026): "Cross link as much as we can and turn feeds, sites, and other directories into data feeds. Doozer sites need to be living entities."

The spine (BUILT — includes/magellan.php, v2.2.549)

business_id_map (business_id, service, external_id, source, confidence, verified_at), unique per (business_id, service), reverse-indexed on (service, external_id). This is the table specified in magellan-id-mapping in June 2026 and finally built. magellanBind/Id/Find/All/BackfillHot/Stats; dz magellan.

Live coverage: google 3,632 · elks_org 1,704 · gcal (growing) · ticketmaster 101.

The hot columns (businesses.google_place_id, businesses.tm_venue_id) are NOT touched; magellanBackfillHot() copies them IN so both views agree while callers migrate.

The rule stays: join by ID, never by name+city+state. Name matching guesses, duplicates, and fails silently.

Why a bound ID matters (the actual point)

A generated page describing a business is one of thousands and gets skipped. A page carrying what is on tonight is the only page answering that question. A bound ID is what makes the second thing possible, and it is also an accuracy engine: every feed is a second opinion on the facts.

Feed adapters

gcal — public Google Calendar (includes/feed-gcal.php, GENERIC)

A public Google Calendar exposes iCal at a predictable URL with no key and no quota. Any page embedding one names a live, self-maintained event feed.

<iframe src="...calendar/embed?src=<id>"> -> https://calendar.google.com/calendar/ical/<urlencoded id>/public/basic.ics

gcalIdsFromHtml (handles the addr@gmail.com and hex @group.calendar.google.com forms, and src not being the first query param), gcalFetchIcs, gcalParseUpcoming, gcalSyncBusiness. RFC 5545: line unfolding, VALUE=DATE, TZID, EXDATE, cancelled events, recurrence for FREQ=DAILY/WEEKLY/MONTHLY/YEARLY with INTERVAL/BYDAY/COUNT/ UNTIL, hard-bounded. BYSETPOS/BYWEEKNO are NOT expanded (stated, not hidden).

GOTCHA — timezone is a correctness bug, not cosmetics. Store the instant AND the calendar's tz, and format with gcalLocalString(). First pass showed San Diego's "Friday Dinner" as Saturday 00:00 because correct UTC instants were formatted in the server's zone. Events belong to the place that holds them.

GOTCHA — the reader strips iframes. Calendar discovery must fetch RAW HTML (elksOrgFetchHtml), not sourceFetchMarkdown(). This is why the calendar page looked empty and was nearly written off as unusable.

elks_org — the BPOE registry (includes/elks-lodge-contacts.php)

Lodge number is in every elks.club subdomain (1,710/1,710), so the path is deterministic and needs no search API key. elks.org/robots.txt is permissive (only three history PDF paths disallowed). Per-lodge pages: home (address, meeting times, officers, official site, Facebook), lodgeFacilities (social-quarter and hall hours — RICH, and exactly what the real leads ask for), lodgeNews, lodgeCalendar (the Google embed), contactUs (form URL — surfaced for a HUMAN, never auto-submitted).

Measured on a 24-lodge random sample: 25% list their own website, 20% Facebook only, 100% a verified address, 17% embed a calendar.

GOTCHA — parse the BODY SLICE only, between ## <City>, ST Lodge No. <N> and ##### Navigation. Outside it is elks.org chrome, which made the national vendor link elksbenefits.com look like a lodge website on the first pass (a MapQuest directions link did the same). Officers are PROSE ("The Exalted Ruler of the Lodge is X, and the Secretary is Y"), not a labelled list.

Rules every adapter honours

• GEO-ANCHOR. Feed addresses are REFERENCE only, never applied to the geo-verified

location. Enrich the content, never the coordinates.

• Facts, never prose. Copying source prose is what caused the elks.club scaled-content

cliff. Extract data; let the site render it.

• No automated outbound. Reading public pages is fine. Submitting a third party's

contact form at scale is not, whatever the volume argument.

• Dry-run by default, live=1 writes (the dz harvest convention).
• Be a polite guest: delay between fetches, cache hard.

Next

Render the bound data on the site (facilities hours, upcoming events). The facts are captured but the page does not use them yet — the same gap [[project_foundation_sprint]] notes for source_facts. Then more services: the lodge's own site, state associations, and other directories that publish structured member data.

Related: [[the-flywheel]], [[event-feeds]], [[money-clouds]], [[source-of-truth-cache]], [[project_where_the_traffic_is]].