show.fm Docs (internal)

Appearance

Lifecycle Manager Worker ​

The Lifecycle Manager is the project's dual-cadence housekeeping worker. It covers two unrelated problems behind one deployment:

  1. Daily lane (04:00 UTC) — pause-expiry reactivation, hard-deletion of pending_deletion podcasts, R2 cleanup, Stripe billing resumption, the Epic 13 storage reconciler (safe subset: server-measured audio sizes, aged orphan sweep, invalidation-outbox drain), and the hosted-content overage escalation sweep (downgrade-overages §7 — arm/nag/suspend/restore, never delete). A secret-guarded POST /reconcile endpoint exposes the reconciler's destructive one-time backfills on demand.
  2. Fast lane (every minute) — three independent self-healing sweeps that run in parallel:
    • sweepStalledImports — re-dispatches lost podcast-imports queue messages.
    • sweepStalledExternalLinks — re-enqueues lost external-episode-link queue messages (Epic 11).
    • sweepStuckHostingMigrationPreviews — cancels hosting-migration rows wedged in the preview staging window (Epic 11).

The two cadences share one worker because each fast-lane sweep is cheap (single indexed query + bounded queue.send() calls) and keeping them in the same deploy avoids introducing more workers just for self-healing.

Overview ​

PropertyValue
Worker Namepodcasterplus-lifecycle-manager
Source Location/workers/lifecycle-manager/
TriggersCron: 0 4 * * * (daily lane) and * * * * * (fast lane)
DatabaseSupabase PostgreSQL via Hyperdrive
StorageR2 (MEDIA_BUCKET) for cleanup + storage reconciliation
Queue producersrss-invalidation, podcast-imports, external-episode-link, public-api-invalidation
HTTP endpointsGET /health, GET /_health, POST /reconcile (secret-guarded — A.5 storage reconciler)

The scheduled() handler dispatches by event.cron: * * * * * runs the three fast-lane sweeps in parallel (via Promise.all — a failing sweep does not block the others) and returns; anything else runs the daily lane.

Responsibilities ​

Daily lane ​

1. Auto-Reactivate Expired Pauses ​

Finds podcasts where status = 'paused' and pause_expires_at <= NOW(), then:

  1. Sets status = 'active', clears paused_at and pause_expires_at
  2. Records last_pause_ended_at = NOW() (starts the 12-month cooldown)
  3. Resumes Stripe billing for the podcast owner
sql
SELECT id, slug, title, pause_expires_at
FROM podcasts
WHERE status = 'paused'
  AND pause_expires_at <= NOW()

2. Hard-Delete Expired Podcasts ​

Finds podcasts where status = 'pending_deletion' and deletion_scheduled_at <= NOW(), then:

  1. DELETE FROM podcasts WHERE id = $id (CASCADE handles all related tables)
  2. R2 cleanup: Deletes all objects under podcasts/{podcastId}/
  3. RSS cache invalidation: Sends message to rss-invalidation queue (or HTTP fallback)
sql
SELECT id, slug, title, deletion_scheduled_at, deletion_redirect_url
FROM podcasts
WHERE status = 'pending_deletion'
  AND deletion_scheduled_at <= NOW()

3. Reap Unconfirmed Signups ​

Email confirmation is on, so an auth.users row without email_confirmed_at can never sign in, and since migration 20260906200000 it owns nothing (no profile, billing account, guest links or CRM contact). The lane calls public.reap_unconfirmed_users(interval) (src/reap-unconfirmed-users.ts, retention UNCONFIRMED_USER_RETENTION_DAYS = 7), which de-provisions and deletes every unconfirmed, non-invited user older than the retention. A row still held by a NO ACTION foreign key is skipped with a Postgres WARNING and counted in skipped; the lane logs unconfirmed_users_reaped with deleted, skipped and retention_days, and a failure logs unconfirmed_users_reap_failed without blocking the rest of the lane. Seven days is generous on purpose: the confirmation code lives 15 minutes, and a person who comes back later signs up again. Runbook: Bot protection.

Billing Resumption (Self-Healing) ​

When a paused podcast is auto-reactivated, the worker resumes Stripe billing for the owner. The resumeOwnerBillingAfterReactivation() function:

  1. Looks up the podcast owner via podcast_members + user_profiles
  2. Skips non-subscription billing models (enterprise_manual, ltd)
  3. Self-healing: If stripe_subscription_id is not cached, resolves it from Stripe via stripe_customer_id and caches it
  4. Clears pause_collection on the Stripe subscription

Worker uses raw Stripe REST API

Unlike the main app (which uses the Stripe SDK), this worker calls the Stripe REST API directly via fetch(). This avoids the SDK dependency in the Worker bundle.

R2 Cleanup ​

Deleted podcast storage is cleaned up by listing and deleting all objects under the podcast's R2 prefix:

podcasts/{podcastId}/images/   → cover, header, banner images
podcasts/{podcastId}/e/        → episode audio files

Cleanup is best-effort: R2 failures are logged but do not prevent the database deletion.

RSS Cache Invalidation ​

After a hard delete, the RSS cache must be invalidated so the feed returns 404 immediately. The worker uses a two-tier strategy:

  1. Queue (preferred): Send { type: 'podcast.deleted', podcast_id, podcast_slug } to rss-invalidation queue
  2. HTTP fallback: POST to the feed worker's /_internal/invalidate with the RSS invalidation secret; canonical addressing is { id: podcastId } (Epic 17 — slug stays accepted as a transition alias, and an unresolvable slug triggers legacy-key cleanup)

Storage Reconciliation (Epic 13 A.5) ​

The daily lane also runs the storage reconciler — folded in from the former standalone podcasterplus-storage-reconciler worker (it shared this worker's daily 0 4 * * * cron and exact bindings — HYPERDRIVE, MEDIA_BUCKET, RSS_INVALIDATION_SECRET, PUBLIC_FEED_URL — so a second deployment, secret set, and cron added cost with no isolation benefit). The engine lives in workers/lifecycle-manager/src/reconcile.ts (+ pure helpers in reconcile-core.ts); the daily lane calls runReconcile(env, SCHEDULED_OPTIONS).

It reconciles R2 audio objects against the episodes audio metadata. It exists because production was exercised against the pre-hardening upload path (Workstream A): audio byte sizes were client-declared (so audio_file_size_bytes can be wrong/NULL), and the old removeAudio cleared the DB row but left the R2 object behind (leaked orphans). It is also the runtime garbage collector for superseded audio: the upload finalize and the removeAudio path deliberately retain a replaced/removed object for a cache-exposed (published/archived) episode instead of deleting it inline — so a stale edge/cache copy never 404s — and the orphan sweep (age-guarded ≥24h + a final DB-reference check) reclaims it later.

Four jobs ​

  1. Size backfill — for every episode with audio_url, derive the R2 key and HEAD it. If the object exists and audio_file_size_bytes is NULL/0 or disagrees with the real size, record the measured size (apply: write it). A write on a published episode changes the feed's <enclosure length>, so it also records a feed_invalidation_outbox intent (reason size_backfill, drained by job 3) — the snapshot refreshes the same run instead of the nightly DR check having to heal it.

    The comparison MUST coerce (sizeNeedsWrite in reconcile-core.ts)

    audio_file_size_bytes is a Postgres BIGINT, and the postgres client returns int8 as a string (OID 20 has no number parser). The original inline recorded !== actual compared string to number, was true for every row, and rewrote every audio-bearing episode every day with its existing value — 149 rows across 10 podcasts, bumping episodes.updated_at, the parent podcast (touch trigger) and <lastBuildDate> — which kept the nightly dr-reconcile feed-freshness check red for every podcast without organic feed traffic (11 of 12 nights to 2026-08-20). The comparison now lives in sizeNeedsWrite() (unit-tested, coerces with Number(...) like overage-escalation.ts always did). When reading BIGINT/NUMERIC columns through the postgres client, coerce at the boundary and type the row field as string | null, never number.

  2. Missing objects — episodes whose audio_url object no longer exists. Always reported; only quarantined with quarantineMissing. Quarantine is a COMPLETE transition in a SINGLE TRANSACTION: it locks the row and reads its current status (never the scan-time status — a draft can go live between scan and apply), then clears the audio columns AND — if the episode is published/scheduled — demotes it to draft, clears published_at/scheduled_for, and cancels its pending time-based automation jobs. A failure rolls everything back so the next run rediscovers the row. For a published demotion, the same transaction records a durable intent in feed_invalidation_outbox.

  3. Invalidation-outbox drain — runs on every apply run (independent of quarantineMissing). Delivers undelivered feed_invalidation_outbox rows via the synchronous, confirmable HTTP path (POST {PUBLIC_FEED_URL}/_internal/invalidate — returns 2xx only after the cache is busted), marking delivered_at only on a confirmed 2xx / else bumping attempts + last_error and leaving the row pending. The rss-invalidation queue is deliberately not used here: queue acceptance ≠ invalidation, and that consumer has no DLQ, so a dropped message would be lost while the outbox had already marked it delivered (round-6 finding 2).

  4. Orphan sweep — R2 audio objects no episode references (old removeAudio leaks + retained superseded objects, plus post-normalise superseded sources). The referenced set covers both audio columns — audio_url (enclosure) and original_audio_url (the media plane's retained master) — and the final pre-delete re-check matches either; without the union every FLAC master would be swept 24h after normalisation (media plane §6.3). Reported always; deleted in apply mode unless skipOrphanDelete, only when older than minOrphanAgeHours, and only after a final per-candidate DB re-check immediately before the delete. A companion episode-image sweep (job 3b, same age guard + final re-check against episodes.cover_image_url) reclaims per-episode cover-art objects (podcasts/{p}/e/{e}/images/…) as the backstop for the inline cleanups in the episode page's removeCoverImage/delete actions; it skips the pending-invalidation protection since a stale cached feed pointing at a deleted image only degrades to the channel artwork. Podcast-level images (podcasts/{p}/images/…) are singletons overwritten in place and are never swept.

Plus an over-cap report — per billing account, SUM(audio_file_size_bytes + original_audio_file_size_bytes) (count-both: enclosure + master, mirroring the storage cap trigger — media plane §6.4) vs effective_cap(account, 'storage_gb_per_account') (NULL = unlimited). Report only — it will never delete audio to force an account under cap. The report survives unchanged, but the same position now also drives the overage escalation sweep below ("Job 4's graduation").

Two media-plane legs (plan 2026-07-18-media-processing-plane.md):

  • Prep-artifact TTL sweep (job 3d, daily run) — transcription-prep objects (podcasts/{p}/e/{e}/transcribe-prep/…) older than 7 days are deleted with no reference check: they are re-derivable on demand (the ai-processor enqueues a fresh prep job when the manifest is gone).
  • media_jobs watchdog (sweepWedgedMediaJobs, MINUTE lane) — pending/running rows older than 3 hours (a dead workflow, or a lost spawn whose failure-flip was itself lost) are flagged failed + notified via notify_media_job_result; a wedged row would otherwise hold the one-active unique index and block every future job for that episode+type. The sweep runs with the other minute watchdogs — inside the daily reconcile alone, a row wedged just after 04:00 would sit ~24h against a 3h promise; the daily run (job 5) still calls it so the reconcile report carries the flagged ids.

Daily cron (safe subset) ​

The daily lane runs SCHEDULED_OPTIONS: mode: apply, size backfill on, orphan sweep on (minOrphanAgeHours: 24), quarantineMissing and bypassCapForBackfill OFF. So the cron only ever writes measured sizes, deletes aged reference-checked orphans, and drains any pending invalidation outbox — it never quarantines (nulls) a row or disables the cap trigger. It is isolated in its own catch, so a reconcile failure cannot block the rest of daily maintenance. The run summary is logged as scheduled_reconcile.

Manual endpoint — POST /reconcile (destructive one-time backfills) ​

The destructive backfills stay manual, behind the worker's secret-guarded HTTP endpoint (handleReconcileRequest). Always dryrun first, review the JSON report, then apply:

bash
# 1) Dry run — reports only, mutates nothing
curl -X POST "https://<lifecycle-manager-host>/reconcile" \
  -H "Authorization: Bearer $RECONCILER_SECRET"

# 2) Apply — size backfill + aged orphan deletion (missing-quarantine still opt-in)
curl -X POST "https://<lifecycle-manager-host>/reconcile?mode=apply" \
  -H "Authorization: Bearer $RECONCILER_SECRET"

A request without a valid Authorization: Bearer <RECONCILER_SECRET> returns 401 (fails closed if the secret is unset, before the options are even parsed). No native R2 S3 credentials are required — it uses the native MEDIA_BUCKET binding, so it is not blocked by the presigned-upload (A-6) deploy blocker.

FlagDefaultEffect
modedryrunapply to perform mutations. dryrun reports only.
quarantineMissingfalse(apply) Null audio_url + audio_file_size_bytes for rows whose R2 object is gone (full demotion transition).
skipOrphanDeletefalseForce the orphan sweep to report-only even in apply mode.
bypassCapForBackfillfalse(apply) Disable only the storage-cap trigger during the size backfill (caveat below).
minOrphanAgeHours24Minimum age before an orphan object may be deleted (clamped 24–720). The 24h floor is a hard guard — it cannot be lowered — so an in-flight upload's object is never swept early.

The cap-bypass caveat: by default the size backfill runs with the storage-cap trigger active — a row whose true size would push its account over cap raises 23514, which the worker catches and lists under sizeBackfill.blockedOverCap (the run does not fail). With bypassCapForBackfill=true (apply only) the worker wraps the backfill in a transaction that DISABLEs only trigger_enforce_storage_account_cap (re-enabled in a finally; rolled back on error; falls back to the active path with capBypassFallback: true if the Hyperdrive role lacks ALTER privilege). Use only when backfilling already-stored historical audio whose true sizes legitimately exceed a now-stricter cap. The ALTER TABLE takes a brief ACCESS EXCLUSIVE lock on episodes — keep the run small / off-peak.

Guarantees ​

  • Dryrun mutates nothing. Mutations only occur with mode=apply, and each is independently flag-gated.
  • Orphan deletion respects the 24h age guard so it cannot race a finalize, plus a final per-candidate DB re-check before each delete.
  • Over-cap is report-only. The worker never deletes a user's audio to bring an account under cap.
  • It runs as the Hyperdrive/owner role (NULL auth.uid()), so its size/quarantine writes are exempt from the browser-only audio-column lock (42501), and quarantine (size → NULL) is a shrink that never trips the storage cap.

Hosted-content overage escalation sweep (downgrade-overages) ​

The daily lane's last job is runOverageEscalation() (workers/lifecycle-manager/src/overage-escalation.ts) — Job 4's graduation from the report-only over-cap report into the 30-day hosted-content state machine (design: docs/planning/plans/2026-07-12-downgrade-overages.md §7). The Stripe webhook's flip-time reconciler (src/api/utils/overage-reconciler.ts) is the other entry point into the same machine; the sweep is its daily correctness backstop (missed webhooks, resource reductions, admin overrides). It is isolated in its own catch, so a failure cannot block the rest of daily maintenance.

What it scans ​

One query (Job 4's byte sums with podcast-slot counts added alongside, same shape) selects only accounts that can possibly need work:

  • over the podcasts_per_account or storage_gb_per_account effective cap (escalation occupancy = active/paused podcasts only, with no hosting-type filter — externally hosted podcasts count; pending_deletion is deliberately excluded even though it holds a create-time slot, because scheduling the excess feed for deletion is the resolution and counting it would drive an empty-sanction suspension; storage = SUM(episodes.audio_file_size_bytes) over non-pending_deletion podcasts), or
  • armed (billing_accounts.content_overage_since IS NOT NULL), or
  • stamped (any podcast with overage_suspended_at IS NOT NULL).

effective_cap() resolves live — overrides and credit pools included — so an admin override is a support rescue lever the sweep honours by construction. Everything else never enters the machine.

Four legs ​

LegConditionAction
1. ArmOver either cap, content_overage_since IS NULLConditional UPDATE ... WHERE content_overage_since IS NULL sets since = NOW(), deadline = NOW() + 30d, nag_stage = -1. First-detection-wins: an armed clock is never re-armed or extended. Losing the arm race to the webhook reconciler re-reads the clock and continues.
2. NagArmed, email stage dueCompute the highest email stage due by elapsed time (stages 0/1/2/3 at days 0/7/21/27, clamped to 3), send stage D's email first, then claim it atomically once the send outcome is final: UPDATE billing_accounts SET content_overage_nag_stage = D, content_overage_last_nag_at = NOW() WHERE ... AND content_overage_nag_stage < D RETURNING. A transient send failure (Resend 429/5xx/network) leaves the stage unclaimed so the next run retries the notice; a permanent fault (unconfigured Resend, no resolvable owner, definite 4xx) advances best-effort so a misconfiguration never stalls the escalation. A cron gap collapses skipped reminders into one email (the latest due) and a gap past day 30 still stops at the stage-3 warning, never at suspend.
3. Suspendnag_stage = 3 AND content_overage_last_nag_at + 3d <= NOW()Gated on the final warning having aged its full 3-day notice, so suspension can never fire in the same run as the warning. The sweep stamps the sanction set first (suspension takes effect immediately), invalidates both caches, sends the suspension notice, then flips nag_stage to 4 via a separate atomic claim only when the notice's send outcome is final — a transient notice failure leaves nag_stage = 3 (feeds already stamped) so the next run re-derives, re-stamps, and retries the notice; a user is never suspended with no warning. Accounts already at stage 4 get their sanction set re-derived from live state every run (stamp missing rows, clear rows no longer sanctioned), so partial resolutions converge instead of replaying history.
4. RestoreArmed or stamped, now under both capsDisarm (clear all four clock columns) + clear every overage_suspended_at stamp, invalidate both caches, send the restored email. Restore clears the stamp and nothing else — never is_active or status, so a podcast the owner switched off themselves stays off.

Send first, claim after — two guards, never a prior read (§7.2). Both entry points (this sweep and the webhook reconciler) send the due stage's notice, then advance through the same conditional-UPDATE claim. The stable Resend Idempotency-Key is the send-once guard — overlapping entry points may both send, and Resend collapses them to one delivery — while the row-locked conditional UPDATE is the advance-once guard (exactly one caller's claim matches). Claim-then-send would not be loss-free: a send failing after a committed claim would never be retried, suspending a user with no warning. The canonical key is overage:{account}:{since}:{stage}, scoped by content_overage_since so a resolved-then-reopened overage re-sends its fresh stage 0 instead of deduping against the prior episode (src/api/utils/overage-reconciler.ts:267). ⚠ The sweep currently sends overage:{account}:{stage} without the {since} scoping (overage-escalation.ts:363, :409) — until it is aligned, overlapping entry points do not dedupe and a same-day re-armed overage's stage-0 can be dropped. Keep the two entry points' keys in lockstep.

Sanction set ​

  • Storage-byte overage (with or without count overage): all occupying feeds — bytes have no per-feed excess.
  • Podcast-count overage only: the excess podcasts, newest-first by created_at (tiebreak id), keeping the oldest cap occupying podcasts serving.
  • pending_deletion podcasts are never selected — the deletion flow owns their feed behaviour.
  • Suspension is one platform-owned stamp, podcasts.overage_suspended_at (guard-triggered — a user-JWT write raises 42501). The escalation never reads or writes is_active or status; this worker's own pause-expiry auto-reactivation therefore cannot un-suspend a feed, and suspension survives every lifecycle transition.
  • Suspension frees no slots (the stamp sits outside slot occupancy), so it cannot be used as slot arbitrage.

Cache invalidation (both caches, always) ​

Every suspend/restore stamp change fans out to both serving caches per podcast:

  1. RSS feed — the worker's established synchronous HTTP lane (deliverInvalidation() → POST {PUBLIC_FEED_URL}/_internal/invalidate?wait=1), not the rss-invalidation queue: queue acceptance ≠ invalidation and that consumer has no DLQ. A 2xx confirms the KV entry is gone and the DR feed snapshot refreshed/purged.
  2. Embed player / v1 API — the PUBLIC_API_INVALIDATION_QUEUE producer (net-new for this worker), same message shape as RSS invalidation, dropping the public-api worker's KV payload cache.

Failures are collected into the run report's errors — the next run re-derives and retries.

Email delivery (Resend, quota-exempt) ​

Stage emails (0/7/21/27-day nags, the suspension notice, the restore confirmation) are platform billing mail sent directly via the Resend API (the crm-sync digest pattern) to the billing-account owner — they never consume the account's email/notification quotas and never touch the automation system. Requires the RESEND_API_KEY + RESEND_FROM_EMAIL secrets: unset = emails are skipped with a logged warning (overage_email_skipped) while the state machine still advances — the atomic claims, not the sends, are the correctness guard.

Never deletes ​

Job 4's invariant survives its graduation: the sweep suspends serving and restores it, but never deletes a podcast, an episode, or audio to bring an account under cap. Resolution is upgrade, admin override, or the user deleting content themselves (detected by the next sweep, which restores).

Fast lane — three sweeps ​

The minute-cadence cron fires all three sweeps in parallel. They share the worker and the structured import_watchdog_completed summary log, but operate on independent state.

Stalled-Import Watchdog ​

sweepStalledImports() re-dispatches import_items rows that are stuck at status='pending' (or rescues status='processing' rows whose worker died) under a parent podcast_imports row that is still running. This compensates for a rare Cloudflare Queues failure mode where queue.send() succeeds from the producer's perspective but the message never reaches the consumer — observed on 2026-04-22 on a 69-episode import (67 clean, 2 hung with attempts=0 until manually resumed).

The stale-pending predicate is per-attempt, sized to the executor's backoff() formula plus a 60 s grace. attempts=0 rows use pi.dispatch_end_at (the actual last delaySeconds moment of the dispatch loop) plus a short propagation grace; attempts>0 rows use the per-attempt table below.

Constants (workers/lifecycle-manager/src/index.ts):

ConstantValuePurpose
ATTEMPTS_THRESHOLDS_SECONDS[1..4]67 / 85 / 160 / 410 secondsPer-attempt rescue threshold = backoff(attempts) max + 60 s grace.
STALLED_IMPORT_THRESHOLD_MINUTES10Safety cap for attempts ≥ 5 (the executor terminalises before reaching this in practice).
STALLED_IMPORT_LEGACY_FALLBACK_MINUTES25Used only for rows predating podcast_imports.dispatch_end_at (rolling-upgrade transition).
STALLED_IMPORT_RESEND_COOLDOWN_SECONDS60Excludes rows whose updated_at was bumped in the previous minute, preventing duplicate resends.
STALLED_PROCESSING_THRESHOLD_MINUTES10How long a processing claim may sit before the worker is assumed dead and the row is reset to pending.
STALLED_IMPORT_MAX_RESEND500Upper bound on re-dispatches per tick — cheap runaway protection.

For each stalled item the worker sends a fresh podcast-imports queue message with the original import_id / import_item_id / podcast_id and bumps updated_at = NOW() so the cooldown clause excludes the row on the next tick. It makes no other state mutation — the executor's claimPendingItem does the atomic status flip when it picks up the message.

Safety properties:

  • Idempotent. claimPendingItem in the executor is an atomic UPDATE filtered on status='pending'. A late-arriving original message ack-skips when it loses the race.
  • Bounded. Capped at 500 re-dispatches per tick to avoid a degenerate loop if the executor itself is broken.
  • No status rollback. The worker only sends queue messages and resets processing claims back to pending — it never advances import_items.status toward a terminal state, and never touches podcast_imports.status.

sweepStalledExternalLinks() re-enqueues external-episode-link messages for externally-hosted episodes whose external_link_status='pending' and whose external_link_next_attempt_at is past due. It catches two failure modes:

  1. Lost initial messages — the publish-handoff producer set next_attempt_at but the consumer never ran (queue drop). last_attempt_at is still NULL, which is why next_attempt_at (not last_attempt_at) is the primary predicate.
  2. Lost retry messages — the consumer's self-retry queue.send was dropped. attempt and next_attempt_at reflect the consumer's last write, so re-enqueuing with the same attempt preserves the backoff progression.

The sweep is gated on podcasts.hosting_type = 'external': after a hosting migration cutover flips the row to podcasterplus, any lingering pending external-link rows are stale audit metadata and are ignored.

Constants:

ConstantValuePurpose
STALLED_LINK_GRACE_SECONDS90Grace past next_attempt_at before re-enqueueing (absorbs queue propagation lag).
STALLED_LINK_RESEND_COOLDOWN_SECONDS90Pushes next_attempt_at forward by this much on resend so the next sweep skips the row.
STALLED_LINK_MAX_RESEND500Upper bound on stalled-link re-dispatches per tick.

The consumer's UPDATE filter on external_link_status='pending' makes redelivery safe — a still-alive original message and a watchdog resend cannot both complete (the row's status check dedupes the last writer).

If external_link_next_attempt_at IS NULL (schema-drift defence), the sweep still rescues the row and logs external_link_watchdog_null_next_attempt.

Stuck Hosting-Migration Preview Watchdog (Epic 11) ​

sweepStuckHostingMigrationPreviews() cancels podcast_imports rows that wedged in the preview staging window of the hosting-migration commit endpoint. The commit flow inserts the row as status='preview' with started_at=NULL, dispatches every queue message, then flips to status='running' (started_at=NOW()). If the API request dies between insert and activation (container kill, network drop, worker eviction, timeout), the row stays preview forever and the unique partial index idx_podcast_imports_hosting_migration_active blocks every future commit attempt for that podcast with a 409.

The sweep detects exactly that state — is_hosting_migration = TRUE, status = 'preview', started_at IS NULL, older than the threshold — and flips the row to canceled so the user can retry. Any queue messages dispatched before the API died will see the now-canceled parent and ack-skip (processItem treats both preview and canceled parents as ack-skip signals).

Constants:

ConstantValuePurpose
STUCK_HOSTING_MIGRATION_PREVIEW_MINUTES15Generous enough not to cancel a slow-but-healthy commit; tight enough that same-session retry works.
STUCK_HOSTING_MIGRATION_PREVIEW_MAX50Upper bound on cancellations per run.

The SQL uses FOR UPDATE SKIP LOCKED so concurrent worker invocations cannot double-cancel the same row.

Worker activation (status='preview' → 'running') is filtered on eq('status', 'preview') and inspects the returned row count; if the watchdog has already canceled the row, the commit endpoint returns a clean retry-friendly 409 instead of misleading the user with a 201.

Environment Bindings ​

wrangler.toml ​

toml
name = "podcasterplus-lifecycle-manager"
main = "src/index.ts"
compatibility_date = "2024-12-30"
compatibility_flags = ["nodejs_compat"]

# Cron triggers:
#   "0 4 * * *" — daily at 04:00 UTC: pause expiry, pending_deletion hard-delete, R2 cleanup
#   "* * * * *" — every minute: three parallel watchdog sweeps
[triggers]
crons = ["0 4 * * *", "* * * * *"]

[[hyperdrive]]
binding = "HYPERDRIVE"
id = "a81d477ff9264805989f5a72f0354ee8"

[[r2_buckets]]
binding = "MEDIA_BUCKET"
bucket_name = "podcasterplus-media"

[[queues.producers]]
queue = "rss-invalidation"
binding = "RSS_INVALIDATION_QUEUE"

# Used by sweepStalledImports to re-dispatch lost messages to the
# podcast-import-executor consumer.
[[queues.producers]]
queue = "podcast-imports"
binding = "PODCAST_IMPORTS_QUEUE"

# Used by sweepStalledExternalLinks (Epic 11) to re-dispatch lost
# external-episode link messages.
[[queues.producers]]
queue = "external-episode-link"
binding = "EXTERNAL_EPISODE_LINK_QUEUE"

# The public-api worker's payload cache (embed player / v1 API). The overage
# escalation fans its suspend/restore invalidations out to BOTH caches: the
# RSS leg via the synchronous HTTP lane, this queue for public-api.
[[queues.producers]]
queue = "public-api-invalidation"
binding = "PUBLIC_API_INVALIDATION_QUEUE"

[vars]
PUBLIC_FEED_URL = "https://feed.podcasterplus.com"
# Public base for R2 objects — the storage reconciler (A.5) derives object keys
# from audio_url against this base.
R2_PUBLIC_URL = "https://media.podcasterplus.com"
# Deep-link base for the overage escalation emails.
PUBLIC_APP_URL = "https://app.podcasterplus.com"

# The daily lane now also lists every R2 audio object + HEADs each audio-bearing
# episode (the reconciler). Match the former standalone worker's CPU budget.
[limits]
cpu_ms = 30000

Secrets ​

Set via wrangler secret put:

SecretPurpose
STRIPE_SECRET_KEYResume billing after auto-reactivation
RSS_INVALIDATION_SECRETHTTP fallback for RSS cache invalidation; also drains the storage reconciler's feed-invalidation outbox (A.5) and confirms the overage escalation's suspend/restore feed invalidations (wait=1 lane)
RECONCILER_SECRETBearer token required to invoke the storage reconciler's destructive one-time backfills via POST /reconcile (A.5). The safe daily-cron subset does not need it.
RESEND_API_KEYPlatform billing mail for the overage escalation (nags/suspension/restore). Unset = emails are skipped with a logged warning; the state machine still advances (the atomic claims are the guard).
RESEND_FROM_EMAILFrom-address for the overage escalation mail (e.g. [email protected])

Health Check ​

The worker also exposes HTTP endpoints:

  • GET /health or GET /_health → { "status": "ok", "service": "lifecycle-manager" }
  • POST /reconcile — secret-guarded storage reconciler (A.5; see Storage Reconciliation). Returns 401 without a valid RECONCILER_SECRET Bearer token.

Logging ​

All events are logged as structured JSON. Every scheduled() invocation emits lifecycle_cron_started with the cron field set, so you can filter by lane in wrangler tail.

Daily lane ​

EventDescription
lifecycle_cron_startedCron trigger fired (any lane)
lifecycle_cron_completedDaily run finished with counts
lifecycle_cron_failedTop-level failure
pause_auto_reactivatedPodcast pause expired, reactivated
pause_reactivation_failedFailed to reactivate a specific podcast
billing_resumed_after_reactivationStripe billing resumed
billing_resume_failedStripe billing resume failed
podcast_hard_deletedPodcast row deleted
podcast_deletion_failedFailed to delete a specific podcast
r2_cleanup_completedR2 objects deleted
r2_cleanup_failedR2 cleanup error
cache_invalidation_failedRSS cache invalidation error
scheduled_reconcileStorage reconciler (A.5) finished — sizeWritten, orphansDeleted, overCapAccounts, outboxDrained, errors, durationMs
scheduled_reconcile_failedStorage reconciler threw; rest of the daily lane unaffected
overage_escalation_completedOverage escalation sweep finished — scanned, armed, emails_sent, suspended_accounts, suspended_podcasts, restored_accounts, restored_podcasts, errors
overage_escalation_failedOverage escalation sweep threw; rest of the daily lane unaffected
overage_escalation_account_failedOne account's escalation processing threw; the sweep continues
overage_email_skippedResend not configured — stage email skipped, state machine advanced anyway
overage_email_no_ownerNo owner email resolvable for a billing account's stage email
overage_email_failedResend API rejected or threw for a stage email

Fast lane ​

EventDescription
import_watchdog_completedSweep finished — combined summary: items_requeued, links_requeued, previews_canceled (each 0 on idle ticks)
import_watchdog_found_stalledPer-import summary: import_id, count, oldest_stale_minutes
import_watchdog_rescued_processingReset processing rows back to pending after the worker was assumed dead
import_watchdog_resend_failedqueue.send() threw for a specific item; sweep continues
import_watchdog_rescue_resend_failedResend of a rescued (processing→pending) item failed
import_watchdog_misconfiguredPODCAST_IMPORTS_QUEUE binding missing at runtime
import_watchdog_failedsweepStalledImports threw; other sweeps unaffected
external_link_watchdog_resentPer-tick count of resent external-link messages
external_link_watchdog_resend_failedqueue.send() threw for a specific episode
external_link_watchdog_null_next_attemptDefensive log: pending row had next_attempt_at = NULL (rescued anyway)
external_link_watchdog_skipped_no_targetPending row had no published_at or scheduled_for to use as target_published_at
external_link_watchdog_misconfiguredEXTERNAL_EPISODE_LINK_QUEUE binding missing
external_link_watchdog_failedsweepStalledExternalLinks threw; other sweeps unaffected
hosting_migration_preview_watchdog_canceledPer-tick count of stuck preview rows flipped to canceled
hosting_migration_preview_watchdog_failedsweepStuckHostingMigrationPreviews threw; other sweeps unaffected

Deployment ​

bash
cd workers/lifecycle-manager
npx wrangler deploy

# Set secrets
npx wrangler secret put STRIPE_SECRET_KEY
npx wrangler secret put RSS_INVALIDATION_SECRET
npx wrangler secret put RECONCILER_SECRET   # A.5 storage reconciler (POST /reconcile)
npx wrangler secret put RESEND_API_KEY      # overage escalation billing mail
npx wrangler secret put RESEND_FROM_EMAIL   # overage escalation from-address

# View logs
npx wrangler tail podcasterplus-lifecycle-manager

Failure Isolation ​

Each podcast is processed independently. If one podcast fails to reactivate or delete, the error is logged and processing continues for the remaining podcasts. Fast-lane sweeps run inside Promise.all with per-sweep catch blocks, so a thrown sweep cannot block the other two.

Internal documentation - Not for public distribution

show.fm Team