The Photo Pipeline (harvest, store, serve)

systems/photo-pipeline.md · System · photos, images, cache, spaces, cdn, cost, places

unverified: no trust signals recorded yet

How business photographs are referenced from Google, bought once on first view, stored as objects on DO Spaces, and served back through a predictable cache key that lets any page ask "do we already own this?" before spending.

The Photo Pipeline

One sentence: **we store photo references for free, buy the bytes once when a real person first looks, keep them forever as public objects, and every URL is derived from a key we can predict — so any page can ask "do we already own this?" before spending.**

Related: Architecture · Source of Truth Cache · The Observed World

---

1. Harvest — we take the reference, not the picture

When a business is discovered, Google Places returns photo handles. Those land in businesses.photos (JSON) in one of two shapes:

API	field	looks like
Places API (New)	name	places/ChIJ…/photos/AaVGc3n…
Places API (Old)	photo_reference	AaVGc3n…

Nothing is downloaded at this point. A reference costs nothing to hold, and most businesses in the network are never opened by anybody. This is what makes par-created skeletons and the directory's blueprint plates cheap: we know a place has photographs without having paid for them.

2. Choose — which photo to show

resolveBusinessImageUrl($row, $width) in business-image-resolver.php is the canonical picker, used by every surface so quality matches everywhere. It returns ['url' => …, 'is_logo' => bool], preferring, in order: an already-resolved photo_url, then Google photos, then scraped / deep-research images, then the logo. When it returns nothing the caller draws a category tile instead.

business_photo_meta (58,545 rows) holds per-photo vision analysis — subject class, aesthetic, quality, caption — for ranking and hero selection.

3. Derive the URL — never link Google directly

googlePlacePhotoProxyUrl($photo, $maxWidth) builds a URL on our own host:

https://<any-doozer-host>/?e=photo&name=<urlencoded reference>&w=<W>&h=<H>

Linking googleapis.com directly would leak the API key and bill us for every view by every visitor and every bot. The proxy is what makes the rest of this possible.

4. The cache key — the load-bearing detail

$cacheFilename = generateCacheFilename($cacheKey, 'google', ['w' => $w, 'h' => $h]);
// google_<md5( photoName . json_encode(['w'=>W,'h'=>H]) )>.jpg

The width and height are part of the key. A size nobody has requested before is a guaranteed cache miss and a fresh purchase — and under warm-only rendering it means the photo silently never appears, no matter how many copies of that same image we already own at other sizes.

Reuse the sizes the network already accumulates:

use	width
directory + nearby thumbnails	400
business cards	600
hero	1200

5. Serve — the order ?e=photo tries

1. Registry first. imageRegistryLookup($cacheFilename); if status = 'cdn',

302 straight to cdn_url. No local file needed. Gated by the platform setting image_serve_from_cdn (currently 1).

2. Local disk cache — cache/images/<cache_key> (path in image_cache_path).
3. Negative cache — neg_<cache_key>. Stale Google references return HTTP 400

forever; the marker stops us re-buying them on every pageview. Ordering matters: the positive cache is checked first, or one transient 4xx hides a perfectly good image behind a negative marker.

4. Fetch from Google — New API places.googleapis.com/v1/<name>/media, or Old API

maps.googleapis.com/maps/api/place/photo.

5. Determine the content type by SNIFFING MAGIC BYTES, not by trusting the upstream

header (see gotchas).

6. Serve the visitor, then persist. After fastcgi_finish_request() — so the

visitor waits for none of it — the bytes are written to Spaces and recorded in the registry. Gated by image_spaces_dualwrite.

6. Store — objects on DO Spaces

• Bucket, region and CDN base live in platform_settings: spaces_bucket,

spaces_cdn_base. Credentials are spaces_key / spaces_secret in api_keys.

• Object key: images/<cache_key>
• Written public-read with Cache-Control: public, max-age=31536000, immutable
• Public URL: <spaces_cdn_base>/images/<cache_key>

7. The registry — image_assets

128,168 rows: 113,506 on the CDN, 14,662 negative.

column	meaning
cache_key	google_<md5>.jpg — the join key for everything
spaces_key	images/<cache_key>
cdn_url	the public URL
content_type	what the object is served as
bytes, source_type	size and origin
status	cdn (we own it) / negative (known-bad reference)
cdn_uploaded_at, ct_fixed_at	when stored, when its type was repaired

8. Knowing what we already own — the warm-only rule

Because the key is predictable from the URL alone, a page can ask about a whole screen of photos in one query before rendering any of them:

dzcPhotoCacheKey($url)          // reverse the URL back to its cache key
dzcPrimePhotoWarmth([$urls])    // ONE batched lookup for the whole page
dzcPhotoIsWarm($url)            // free thereafter

Photos we own are shown; photos we do not are replaced by the category plate, and the first real visit to that business buys them for everyone after. Used by the NBMN directory, the NBMN home list, the neighborhood block on every business site, and the JSON-LD emitter.

It fails open: if the registry is unreachable everything is treated as warm. A directory that silently loses its imagery to a database hiccup is a worse failure than one that spends a little.

9. Gotchas, all of them paid for

1. Width is in the key. Asking for 160px to fill a 44px tile meant every card fell

back to a plate while its photo sat on the CDN at 400.

2. Never trust the upstream Content-Type. Read the last header across redirects

and sniff the magic bytes. 32,223 objects were stored as application/json — valid JPEGs that browsers refused to render, showing blank boxes across the network for months.

3. Correcting an object does not correct the cached response. The CDN edge had

already cached the bad header under immutable, max-age=31536000. Healing the object changes nothing a browser sees. The registry-first 302 now appends ?v=<hash of ct_fixed_at> for healed objects only — a new edge cache key exactly once, leaving every never-healed image on its warm URL.

4. Structured data must publish warm images only. ?e=photo buys on first request

and 83% of network traffic is bots; a cold photo URL in JSON-LD is an invitation for crawlers to spend money on pictures no person asked to see.

5. A stamped row is not a fixed row. An early version of the healer stamped

ct_fixed_at without correcting content_type; those rows looked done and nothing retried them. dz imgct reset=1 clears the stamp.

10. Operator commands

dz imgct [limit=] [rounds=] [all=1] [probe=1] [reset=1]   # repair content types
dz photo_audit                                            # find broken/missing images
dz photo_heal                                             # re-resolve and re-upload