Media Processor

The media processing plane: runs media_jobs as Cloudflare Workflows driving ffmpeg Containers. First consumers: audio normalisation (WAV → MP3 enclosure + lossless FLAC master; OGG/WebM → MP3 enclosure with the ORIGINAL lossy object retained as the master, never FLAC-encoded; raw AAC → lossless M4A remux, no master) and transcription-prep (silence-aligned chunk manifests for the AI Processor). Designed so Studio transcode and export-archive jobs slot in later as new job types.

Plan of record: docs/planning/plans/2026-07-18-media-processing-plane.md.

Architecture

App / import-executor / ai-processor / sweep
  → INSERT media_jobs (service-role or worker; clients hold NO write privilege)
  → media-jobs queue { jobId, jobType }
      → queue consumer (thin spawn; workflow instance id = job row id)
          → NormaliseWorkflow | TranscriptionPrepWorkflow
              → MediaContainer instance per ATTEMPT ({jobId}:{attempt}, fresh disk)
                  → ffmpeg (Debian-stable, pinned image on the Cloudflare registry)

Piece	Detail
Queue	media-jobs (max_retries=3, DLQ media-jobs-dlq); malformed messages ack'd, duplicate deliveries dedupe via workflow.get()
Job table	media_jobs (status enum pending/running/complete/failed/skipped, one ACTIVE job per episode+type via partial unique index; Realtime-published for the Media-tab card)
Container	standard-1 (½ vCPU / 4 GiB / 8 GB ephemeral disk); one-shot entry script; sleepAfter 75m; SIGTERM-tolerant (idempotent re-runs)
Retry model	Per-step Workflow retries; container attempts ≤2 on fresh instances (waitForEvent 60m timeout, zombie destroy() between attempts)
Watchdog	lifecycle-manager minute lane (sweepWedgedMediaJobs) flags pending/running rows older than 3h → failed + notify; the daily reconcile also runs it so its report carries the flagged ids. The 3h promise only holds on the minute lane — and the consumer's indeterminate-spawn ack deliberately leans on this backstop

Normalise flow (audio_normalise)

validate (re-derives the source key from the episode's CURRENT audio_url — replaced audio ⇒ skipped) → plan-outputs (durable versioned keys) → container run (download via presigned GET → ffprobe → FLAC master -compression_level 8 (wav only) → MP3 CBR 128k/44.1kHz + ID3v2 → optional prep chunks → upload via presigned PUT) → verify (R2 HEAD vs the container report) → guarded enclosure/master swap (0 rows = stale → skip + delete outputs; SQLSTATE 23514 = storage cap → terminal with the upsell message) → mark-complete → best-effort side effects: dual cache invalidation (rss-invalidation and public-api-invalidation) for published/scheduled episodes, and auto-transcribe chaining (inserts an ai_jobs row + sends on ai-jobs when the podcast opted in and the feature is granted).

The superseded source object is never deleted inline — once unreferenced it ages into the lifecycle-manager orphan sweep (whose referenced set covers BOTH audio_url and original_audio_url).

Transcription-prep flow (transcription_prep)

Decodes the FLAC master when present (else the enclosure) to 16 kHz mono PCM, cuts silence-aligned chunks (45–75s window, target 60s, ffmpeg silencedetect), encodes each chunk independently as 64 kbps mono MP3, and writes podcasts/{p}/e/{e}/transcribe-prep/{jobId}/chunk-NNN.mp3 + manifest.json (versioned contract — PrepManifest mirrored in workers/ai-processor/src/lib/prep-manifest.ts). Prep artifacts are ephemeral: a 7-day TTL sweep reclaims them; the ai-processor re-derives on demand. Prep failures never notify the user directly — they surface through the transcription job that polls for the manifest.

Security model

• Zero credentials in the container. Capabilities are short-TTL exact-key presigned URLs minted inside the start step per attempt, never persisted in step results.
• Per-host interception; the data plane goes DIRECT (revised after live E2E): only the virtual host media-plane.internal is intercepted (outboundByHost in the Worker runtime — progress writes, prep-chunk PUTs via the R2 binding, completion sendEvent); presigned R2 transfers flow straight to R2. Never add a catch-all outbound/allowedHosts/interceptHttps — any of those flips intercept-ALL and routes the media data plane through the ContainerProxy hop (it collapsed a 415MB stream in E2E). Accepted consequence: egress beyond the callback host is not platform-restricted; isolation = VM-per-attempt + zero credentials + short-TTL exact-key URLs.
• DO-pinned job identity: the Workflow writes the job context into the container's Durable Object before start(); callback handlers resolve it from ctx.containerId — container-supplied ids are never trusted, and a forged prepManifestKey outside the job prefix is stripped.
• Verified before committed: the Workflow HEADs every claimed output and cross-checks byte sizes before any DB write.
• ffmpeg CVE hygiene: Debian-stable ffmpeg in the image; rebuild + redeploy on security updates (standing ops task).

Endpoints

POST /sweep-normalise (Bearer SWEEP_SECRET, constant-time compare): enumerates episodes whose audio_content_type is not in PUBLISHABLE_AUDIO_TYPES with a managed audio_url and no active job, and enqueues normalise jobs in bounded batches ({batchSize?, dryRun?} → {scanned, enqueued, skippedActive, enqueueFailures, exhausted}). Idempotent; re-run until exhausted. Also the re-run lever for future target-spec changes.

Bindings

Hyperdrive (shared config) · R2 MEDIA_BUCKET (podcasterplus-media) · queue consumer media-jobs · producers media-jobs (sweep), rss-invalidation, public-api-invalidation, ai-jobs · Workflows NORMALISE_WORKFLOW, PREP_WORKFLOW · container DO MEDIA_CONTAINER · secrets R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, SWEEP_SECRET.

Deploy

cd workers/media-processor
npx wrangler queues create media-jobs        # once
npx wrangler queues create media-jobs-dlq    # once
CLOUDFLARE_ACCOUNT_ID=b8eff1b484adf398bda38644efc85bce npx wrangler deploy

The first deploy builds and pushes the container image — Docker must be running locally. Tests run from the repo root: npx vitest run workers/media-processor/.