The Doozer Brainlive from the knowledge bundle · v2.2.618 · 2026-09-13 17:17 ETall pages · one page

Price the pull before you pull

craft/price-the-pull-before-you-pull.md · Process · craft, cost, api, cache, discipline, refresh

unverified: no trust signals recorded yet

A data refresh is a bill. Before re-pulling from any paid source, check what each pull costs, reuse the cache, dry-run the total, and pull only the unavoidable remainder.

Price the pull before you pull

At Doozer's scale, "just refresh the data" is not a neutral operation. Every pull from a paid source (Google Places, photos, deep research, AI) is money, multiplied by thousands of rows. The instinct to freshen everything is exactly the instinct that burns a budget.

The story that makes the rule concrete

A photo refresh was needed. The naive implementation — re-pull every business's photos from the source — would have cost roughly $300. The actual requirement, once we asked which photos genuinely must be re-fetched, was a small subset already missing or broken. Reusing everything still valid and pulling only the real remainder cost about $1.40. Same visible outcome. Two orders of magnitude apart.

The discipline

Before any bulk refresh:

  1. Ask if each pull costs. Not every source is metered; know which are.
  2. Reuse the cache first. We never expire cached images
  3. precisely so a "refresh" is not a re-buy. Cached-and-still-valid means do nothing.

  4. Dry-run the cost. Compute how many pulls the operation would make and what
  5. that bills, and look at that number, before running it for real.

  6. Pull only the unavoidable remainder. The delta, not the world.

Why it is craft

The clever part is treating a refresh as a bill to be estimated, not a button to be pressed. It reframes an ops task as a budgeting decision, and it turns "regenerate everything" — which feels responsible — into the reckless option. This is the cost half of the skeleton-first instinct (do the expensive thing only when it is truly needed) and it is why the regeneration engine reports cost per mode before it runs. A silent money-burn from not doing this is what motivated the admin alerting roadmap.

Inline the bytes, never the proxy URLSelf-healing email backfill