Video Processing Pipeline

01 / Context

The videos involved are typically 30–60 minutes long, and each one can take around 30 minutes to process. That rules out handling the work inline within a single request — this is an engineering problem in the long-running, compute-intensive category, not a request/response one.

02 / Constraints

• Videos in the roughly 30–60 minute range.
• Processing time of around 30 minutes per video.
• The workload is compute-intensive and long-running — not a fit for an inline request/response step.
• Processing runs in containers.

03 / Architecture

A processing flow, not a request/response API: S3 → SQS → Lambda → AWS Batch → S3.

04 / Processing lifecycle

1. 01A video is submitted and stored in S3.
2. 02A message is placed on SQS, decoupling submission from processing.
3. 03Lambda picks up the message and orchestrates the processing step.
4. 04Lambda hands the heavy lifting to AWS Batch, which runs the containerized, long-running processing job.
5. 05The processed result is written back to S3.

05 / Key decisions

• Queue work with SQS instead of triggering processing directly.

  Why / Decouples work submission from processing.

• Use Lambda as the controller step, not the processing step.

  Why / Lambda orchestrates the flow; it does not run the video processing itself.

• Run the actual processing in AWS Batch.

  Why / AWS Batch is built to run the intensive, containerized, long-running video-processing workload.

06 / Trade-offs

Pending — not yet documented for this case study.

07 / What I learned

Pending — not yet written.