Disaster Recovery & Resilience
The DR program protects distribution: if Cloudflare fails for any reason (outage, account lockout, restriction), podcast players keep receiving RSS and audio from an independent standby path (Backblaze B2 origin, Bunny CDN edge, Porkbun DNS custody). Since Epic 16 GATE-2 (2026-08-24) the protected surfaces answer on two zones — the legacy feed. / media.podcasterplus.com and the structural-domain successors rss. / m.cdn.media — all four hostnames serving the same workers, buckets, and keys, so one replica covers every URL a player might hold. Both pairs are protected indefinitely (the legacy URLs stay load-bearing forever). Explicitly out of scope by decision: a full infrastructure mirror and a warm second Cloudflare account; during a failover the app, marketing site, customer show pages, and embed player are down and that is accepted.
Authoritative operational documents (cited here, not copied):
docs/planning/infrastructure/resilience-disaster-recovery/00-runbook-feed-media-failover.md(the cutover procedure, recorded DNS/DNSSEC facts, standby hygiene checklists)docs/planning/infrastructure/resilience-disaster-recovery/00-implementation-plan.md(per-workstream design and as-built status)- The numbered issue files in the same directory (PITR drill
04-#35, logical backups08-#36, stale fallback03-#33, snapshots07-#34, replication09-#32, and so on)
Architecture
Verification runs outside Cloudflare (GitHub Actions) so the watcher does not share fate with the watched: .github/workflows/dr-reconcile.yml (nightly rclone reconcile R2 to B2, plus a feed-snapshot freshness check against Postgres) and .github/workflows/dr-db-backup.yml (nightly database dump). dr-zone-export.yml commits nightly read-only zone exports plus derived Porkbun cutover zone files to the repo — one pair per zone: the two DR zones (podcasterplus.com, cdn.media) and show.fm (no feed/media overrides; staged for its post-transfer registrar posture) — and dr-standby-cert.yml covers the standby TLS certificate: since 2026-08-24 one SAN cert spanning *.podcasterplus.com + *.cdn.media, uploaded per Bunny custom hostname (four) and weekly-verified on all four SNIs, with an every-run probe that the token still sees both zones.
Off-site database backup
Concept level (implementation and as-built detail live in the plan's Phase D and issue 08-#36; the restore procedure is issue 04-#35):
- Nightly encrypted
pg_dumpto the B2pp-backupsbucket via GitHub Actions (.github/workflows/dr-db-backup.yml), streamed throughageencryption so plaintext never lands on disk, uploaded under B2 Object Lock. - Restore-tested: the program treats a backup as real only once it has been restored; the PITR/restore drill is documented in
04-#35-issue-resilience-supabase-pitr-restore-drill.md. - The nightly reconcile workflow includes a stale-backup guard that checks the dump's age, so a silently broken backup job fails the nightly run instead of rotting unnoticed.
Media replication (dr-replicator, DR-A2)
workers/dr-replicator/ is a queue-only worker (no HTTP surface, no cron) consuming R2 event notifications for podcasterplus-media and podcasterplus-feeds from the r2-object-events queue, and mirroring every object mutation key-for-key into the matching B2 bucket (pp-media-replica / pp-feeds) over B2's S3-compatible API.
Key design points (all in workers/dr-replicator/src/index.ts and its wrangler.toml):
- Whole-bucket, key-for-key. No curated prefix lists:
media-deliverymaps URL path directly to R2 key, so key parity in B2 means byte-identical URLs after cutover with zero feed rewriting. Bucket-level events cover every write path (Worker binding, presigned S3 PUT, multipart complete, deletes, lifecycle) with zero per-feature wiring. - Routing by source bucket name: the event's
bucketfield selects the R2 binding, the B2 target bucket, and the credential pair. B2 application keys are single-bucket, so each target has its own SigV4 client (aws4fetch). - Creates at or below 32MiB replicate as one buffered
PutObject(ArrayBuffer body so the runtime computes a real Content-Length). - Creates above 32MiB use an S3 multipart upload built from ranged R2 reads: each part is an independent
get(key, { range })buffered to exactlyPART_SIZE(32MiB) and PUT with its own Content-Length. This path exists because a 2026-07-08 smoke test proved B2 returns 413 for large streamed PUTs (streamed bodies go out chunked), while buffering a 500MB audio file whole is impossible in a 128MiB isolate. Parts are pinned to the initial etag viaonlyIf: { etagMatches }, so an object replaced mid-replication aborts (and retries) rather than interleaving bytes of two versions; any failure aborts the MPU best-effort and a retry starts a fresh one. - Deletes are S3
DeleteObjectby name with no versionId, producing a delete marker; B2 versioning retains prior versions for 30 days (the program's accidental-delete protection, since R2 has no object versioning). Thedr-*-rwkeys deliberately lack the B2deleteFilescapability, so hard-purging a version is impossible even on key compromise. - Content-Type is preserved so feed snapshots survive the round trip as
application/rss+xml(Bunny passes origin Content-Type through). - Idempotent by construction (PUT/DELETE by key); out-of-order create/delete converges. Queue consumer config:
max_batch_size 10,max_retries 5, DLQr2-object-events-dlq. Malformed messages, unknown buckets, and unknown actions are acked loudly rather than poisoning the queue; the nightly reconcile is the correctness backstop, this path optimizes freshness.
Feed snapshots (DR-B1)
workers/rss-feed/src/cache/snapshot.ts persists generated feed XML to the podcasterplus-feeds R2 bucket at key {slug}. The identity {slug} = feed URL path = R2 key = B2 key = Bunny request path means nothing needs rewriting at cutover.
writeFeedSnapshot()runs fire-and-forget after every cache-miss feed generation (workers/rss-feed/src/index.ts), and is content-conditional: the feed's strong ETag is stored as R2 custom metadata and an unchanged ETag skips the PUT, so hourly lazy regeneration of unchanged content does not churn R2 writes, replication, or B2 versions.- The rss-invalidation queue consumer also proactively regenerates the feed and refreshes the snapshot on every content change (publish, edit, unpublish, artwork, settings), so a feed nobody polls still has a fresh snapshot. It deliberately refreshes only the snapshot, not the KV live cache (a Hyperdrive read can be ~60s stale right after a write).
- Podcasts that stop being servable get their snapshot deleted (
deleteFeedSnapshot()), and the replication pipeline propagates the deletion to B2. - Everything is best-effort and never throws: a snapshot failure must not affect feed serving or block a queue ack. The nightly reconcile's snapshot freshness check (feeds' last content change vs the R2 snapshot's write time, enforced via
ENFORCE_SNAPSHOT_FRESHNESSindr-reconcile.yml) is the backstop that catches a persistently failing snapshot.
Stale-fallback serving tiers (03-#33)
When live feed generation fails (almost always a DB/Hyperdrive outage), the rss-feed worker serves the most recent good XML instead of a 500 that would pull the feed out from under every podcast app. Tier order in workers/rss-feed/src/index.ts:
| Tier | Source | Properties |
|---|---|---|
| 1 | Live KV cache (feed:{slug}, 1h TTL) | The normal path; not a fallback |
| 2 | KV last-known-good (stale:{slug}, 48h TTL) | Written on every successful generation (cacheFeed() in src/cache/kv.ts); deliberately never cleared by invalidateCache(), so it survives content updates and short outages |
| 3 | R2 snapshot (podcasterplus-feeds/{slug}) | Survives KV eviction; refreshed proactively on every invalidation |
| 4 | 500 | Only when both fallback tiers are empty |
Fallback responses are marked X-Cache: STALE plus X-Fallback: kv or X-Fallback: r2 (createStaleResponse()), log a structured feed_served_stale event, and still count as a feed poll in analytics. Their Cache-Control keeps a short browser TTL and omits s-maxage so no shared cache pins the stale copy once the origin recovers.
Note the division of labor: the stale tiers handle DB-side blips without any failover; the Bunny/B2 path in the runbook handles Cloudflare itself failing.
The feed-referenced-assets invariant (standing rule)
From .claude/rules/backend/workers.md and runbook section B3:
Anything an RSS feed references MUST be a replicated key in the
podcasterplus-mediabucket, served via the media hostnames (media.podcasterplus.com/m.cdn.media— same worker, bucket, and keys).
During a failover the feed is served from a B2 snapshot and its URLs resolve via a media hostname to Bunny to B2, so any asset URL a feed emits that is not a replicated media-bucket key becomes a dead link exactly when it matters. The feed and media hostname pairs also move zones together, never separately — a feed on one zone must never reference enclosures in a different failure domain.
- Enclosures comply by construction:
episodes.audio_urlholds raw media-host URLs, and download analytics happen transparently inside themedia-deliveryworker at the same URL (there is no tracking-wrapper URL to break). - The item-level asset columns in
workers/rss-feed/src/db/client.ts(cover_image_url,transcript_url,chapters_url) carry the same obligation: any feature that populates them (for example AI transcription) must store the asset in thepodcasterplus-mediabucket under a canonical key, never a bucket or host that R2 events do not replicate. - Externally hosted episodes (
audio_urlon a third-party host) bypass the media hostnames entirely and are simply unavailable during a failover; accepted, no action needed.
Losing download analytics during a failover is acceptable; broken enclosures are not. Check this rule on any feed-generation change.
What stays manual
The cutover itself is a human procedure by design; consult the runbook's checklists as the operational source of truth. Deliberately manual pieces recorded there:
- Scenario 1 cutover (Cloudflare DNS still editable): edit the four records —
feed./media.on thepodcasterplus.comzone andrss./m.on thecdn.mediazone — to the Bunny pull zones; no DNSSEC change needed. - Scenario 2 cutover (total Cloudflare loss): per zone at Porkbun (
podcasterplus.comandcdn.mediaare both registered there): delete the DS record, flip nameservers to Porkbun, import the committed staged zone by hand. There is intentionally no standing Porkbun API credential: the same API that writes DNS also changes nameservers, so a standing key in CI would be a domain-takeover vector worse than the outage it defends against (runbook A4, redesigned 2026-07-09). The brand zoneshow.fmis registered at Cloudflare until its Porkbun transfer (~late October 2026) and has no Scenario-2 lever until then — accepted, because no DR-critical distribution surface lives on it. - The runbook's Part C (external uptime detection on both zones' feed/media hostnames plus standby-side cert/drift alerts) and Part B4 (failover rehearsals against throwaway hostnames) are tracked as checklist items in the runbook itself; treat those checklists, not this page, as current status.
- DR-E2 drill preparation (the new-estate rehearsal, runbook B4) is a three-step hand-off: (1) Bunny dashboard — attach
rss-dr.cdn.mediato thepp-feedspull zone andm-dr.cdn.mediatopp-media(skip Bunny's own SSL); (2) dispatch theDR — Standby TLS Certworkflow withinclude_drill_hosts=true— it binds the existing SAN cert to the two drill hostnames and REFUSES, via a Bunny API precondition probe, if step 1 was skipped (weekly cron runs never touch the drill hosts); (3) grey-CNAME the two-drrecords to the pull zones in the Cloudflare DNS panel and run the runbook's Part G checks against them, recording the result asdrill-DR-E2-<date>.mdbeside the runbook.