Resumable ML analysis of 4K/60 video

The problem
We run a multi-stage machine-learning pipeline entirely on device, over full-length 4K/60 fps clips. A single clip is thousands of frames, and every frame passes through a chain of Core ML models. Start to finish, that is minutes of sustained GPU work on a phone.
Nobody watches a progress bar for minutes. Users background the app, take a call, or the device gets hot — and iOS treats a long GPU job on a backgrounded app as fair game to tear down. If the pipeline treats any of that as a failure, the user loses all progress and restarts a multi-minute job from zero.
The requirement was blunt: analysis has to survive interruption. Pause when the environment turns against us, resume exactly where we left off, and never throw away more than a sliver of work.
Our approach
The pipeline is a sequence of stages — detection → stabilization → keypoint estimation — each implemented as its own Swift actor, coordinated by a single coordinator actor that owns an explicit state machine:
idle → starting → running ⇄ paused → stopping → (completed | cancelled | failed)
Two things make it resumable.
A durable checkpoint in SwiftData. One @Model row per recording holds the current stage, per-stage metadata (output file locations, telemetry, completion flags) and storage URLs. The coordinator never passes the live model object around — it works off an immutable, Sendable snapshot() struct, so persistence state never leaks across actor boundaries. On launch, checkUnfinishedAnalysis() reads the row and knows precisely which stage to pick up from. That is coarse-grained resume: finished stages are skipped outright, and a stage that already produced and uploaded its result short-circuits to completion.
A backpressured frame extractor. Frame reading is its own actor wrapping AVAssetReader, driven by a structured-concurrency task group with a bounded number of frames in flight — 10 for the keypoint stage. The reader blocks when the queue is full and drains as tasks complete, so we never buffer 4K frames without limit. It records lastProcessedFrame, which gives us fine-grained resume within a stage.
// Backpressure: block the reader until a slot frees up.
if activeTasks >= maxConcurrentFrames {
try Task.checkCancellation()
if let result = try await group.next() {
self.lastProcessedFrame = result
activeTasks -= 1
await report(result, progressHandler: progressHandler)
}
}
The hard decisions and trade-offs
Where to checkpoint: per frame or per stage. We keep a stage's per-frame results in the actor's memory ([Int: KeypointDetections]) and persist the finished result only once the stage completes. Writing every frame to disk during a 60 fps sweep is its own bottleneck, and it would bloat the checkpoint with half-finished state.
A pause releases what is expensive to hold, such as the AVAssetReader, the decoded 4K frame buffers and the in-flight model requests. It keeps the per-frame results in the actor. If iOS kills the app outright, those results go with it and that stage starts over; the SwiftData checkpoint guarantees finished stages are never redone. That is the price of checkpointing per stage: saving to SwiftData after every frame would cost more than it protects. Saving results every N frames would shrink the loss, and it is the obvious next step if we ever need it. So far we have not. Resuming means opening the reader again and seeking back, and that is where the cost is: frames that were in flight when we stopped have no result yet. So a pause is not free. But it is bounded, which leads to the next decision.
Rewind overlap: recompute a little to guarantee no gaps. When we resume a stage, we do not restart the reader exactly at lastProcessedFrame. Up to maxConcurrentFrames frames were in flight when we stopped — some finished, some did not — so restarting at the last reported frame could leave a hole in the sequence. We deliberately rewind by two in-flight batches and re-decode a small overlap:
let minimumFrameCountToStartSkipping = maxConcurrentFrames * frameInterval * 2
var targetStartFrameIndex = 0
if lastProcessedFrame > minimumFrameCountToStartSkipping {
targetStartFrameIndex = lastProcessedFrame - minimumFrameCountToStartSkipping
}
// ...then fast-forward the reader, skipping frames below targetStartFrameIndex.
In the keypoint stage — 10 concurrent frames, every frame processed — that is at most 20 frames re-run through the model, about a third of a second of 60 fps video, no matter how long the clip is. We trade a sliver of recompute for a hard guarantee that the frame sequence has no gaps.
Progress has to stay monotonic even when the reader does not. The rewind means the reader genuinely goes backwards on every resume. Persisting reportedProgress — throttled to roughly 0.5% steps — lets the UI ignore that entirely. The user never sees the bar move back, because the bar is not reading the frame index.
Every pause carries a reason. Lifecycle and thermal observers do not call a generic pause(). They call pause(reason:) with .userPaused, .backgrounded, .thermal and so on. Thermal and background pauses auto-resume when their condition clears; a user pause waits for the user. Without explicit reasons the resume paths cross wires — a clip paused because the phone was hot gets woken up merely because the app returned to the foreground, and starts cooking the device again.
The outcome
- A pause and resume costs at most the rewind overlap — 20 re-processed frames per stage, roughly 0.3 s of 60 fps video — instead of the entire clip, regardless of clip length. If iOS kills the app outright, only the current stage is redone.
- Backgrounding or overheating mid-analysis went from a hard failure to a clean, invisible pause and resume.
- For scale: the full multi-stage analysis of a ~1-minute 4K/60 clip runs 90 s to 2 minutes on an iPhone 17 Pro, of which the keypoint stage is about 30 s. That is what an interruption at the wrong moment used to cost the user, in full.
| Metric | Before | After |
|---|---|---|
| Work lost on pause and resume | Whole clip (90 s–2 min of compute) | ≤ 20 frames per stage |
| Work lost if iOS kills the app | Whole clip | Current stage only |
| Backgrounding mid-run | Job fails every time | Clean pause → resume |
Stack & takeaways
Stack: Swift 6 concurrency (actors, structured task groups, async let), Core ML and Vision, SwiftData for durable checkpoints, AVFoundation (AVAssetReader), Accelerate (vDSP/vImage) on the decode path.
- Treat interruption as a first-class state, not an error path. A pipeline that can pause and resume is a fundamentally different design from one that only knows "running" and "crashed", and it is far easier to get right if you commit to it up front rather than bolting it on.
- Checkpoint at the coarsest level that stays cheap. Per-stage persistence plus a small bounded overlap on resume beat per-frame persistence, in both code complexity and runtime cost.
- Make resume overlap a constant, not a variable. Rewinding by a fixed two in-flight batches means worst-case wasted work is identical for a 10-second clip and a 10-minute one.
- Keep progress monotonic even when the work is not. Persisting the reported fraction lets the reader rewind invisibly.
- Name your pause reasons. A boolean
isPausedcannot tell you what is allowed to resume it, and you will find that out the hard way.
Related: When the OS revokes your GPU covers how iOS tears down background GPU work, and why we stopped fighting it.

