← All postsOffline-First on Android · Part 2 of 4

Offline-First on Android, Part 2 — The Outbox Engine: Architecture & Draining

An engineer working at a laptop at his desk

How we made an offline edit survive the process being killed, and drain to the server in order.

Part 1 built the local half: a single source of truth read via delta pull, so the network never touches the screen. This part is the other half: what happens when users change things offline and expect those changes to survive, sync once, and reconcile.

The problem

A user renames a door while offline. What should happen?

The tempting answer was "disable the button when offline." We rejected it. It is a cop-out, users hate it, and it breaks the one promise a local-first app makes: your own actions always work, immediately, regardless of the network. The network's job is to catch up later.

So every write in the app had to:

  1. Apply to the local database now, so the UI reflects it instantly.
  2. Get recorded as a durable intent to sync: an outbox entry.
  3. Reach the server at-least-once and be applied idempotently, so that even if the app is killed and relaunched three times mid-sync, the net effect is exactly one application. (Exactly-once delivery is impossible over an unreliable network; exactly-once effect is what we engineered.)
  4. Reconcile cleanly when the server's version of events differs from ours.

Point 4 is the hard one and gets Part 3 to itself. This part is the machine that makes 1–3 reliable.

Our approach

The outbox: a durable to-do list for the network

Every mutation writes two things in one local transaction: the new state, and a durable record of the operation that produced it. We made that record a real table, and every field in it is there because something later needed it:

@Entity(
    tableName = "outbox_operations",
    indices = [                                          // drain, pending-skip, and cascade all hit these
        Index(value = ["status", "nextAttemptAt"]),      // the drain selector
        Index(value = ["status", "targetEntityId"]),     // the pending-skip + cascade walk
    ],
)

data class OutboxOperationEntity(
    @PrimaryKey(autoGenerate = true)                     // Room emits INTEGER PRIMARY KEY AUTOINCREMENT
    val sequenceNumber: Long = 0,    // monotonic AND never-reused — matters, because we DELETE rows
    val operationId: String = UUID.randomUUID().toString(),  // retry-dedup key on the wire
    val targetEntityId: String,      // entity identity (NON-null — the skip SQL depends on it)
    val type: String,                // "CREATE" | "RENAME" | "DELETE" | ...
    val payloadJson: String,         // the operation's arguments (JSON-sized args only, never bytes)
    val payloadVersion: Int = 1,     // schema version of payloadJson — survives app upgrades
    val preImageJson: String?,       // entity state BEFORE this op — the rollback snapshot
    val status: OutboxStatus,        // PENDING | IN_FLIGHT | FAILED_RETRYABLE | FAILED_FATAL
    val leaseUntil: Long = 0,        // while IN_FLIGHT: deadline after which a *crashed* op is reclaimable
    val retryCount: Int = 0,         // feeds backoff + poison-message cutoff
    val nextAttemptAt: Long = 0,     // earliest wall-clock at which to retry — the backoff gate
    val lastError: String? = null,   // what went wrong, for surfacing + debugging
    val createdAt: Long = System.currentTimeMillis(),  // display/debug only — NOT a decision input
)

sequenceNumber is the auto-incrementing primary key. Room maps @PrimaryKey(autoGenerate = true) to INTEGER PRIMARY KEY AUTOINCREMENT, and AUTOINCREMENT matters here: it never reuses a value even after rows are deleted (which squashing and completion both do), so a recycled rowid can never scramble the drain order. We never hand-rolled SELECT MAX(seq)+1: two coroutines racing it get the same number.

The two IDs do different jobs, and mixing them up is a classic bug. operationId (a client-minted UUID) is the retry-dedup key the server keys idempotency on: minted once at enqueue and re-sent unchanged on every retry. Regenerate it and dedup silently breaks. targetEntityId is the entity identity used for 409-adopt and the pending-skip. preImageJson is the snapshot of the entity before this op, the exact value restored if the op dies. payloadVersion stamps the schema the payload was written under, so an op queued on v1.2 still parses after the update to v1.3.

With the table in place, the mutation itself is small: apply the state change and enqueue the op, together.

suspend fun renameDoor(id: String, newName: String) = db.withTransaction {
    val before = doorDao.find(id)                      // 0. snapshot for rollback
        ?: error("renameDoor on a row that isn't local — a pre-image must exist")
    doorDao.updateName(id, newName)                    // 1. local truth updates now
    outboxDao.enqueue(
        OutboxOperationEntity(                         // sequenceNumber auto-assigned by SQLite
            targetEntityId = id,
            type = "RENAME",
            payloadJson = json.encodeToString(RenamePayload(newName)),
            preImageJson = json.encodeToString(before), // 2. how to undo, if this op dies
            status = OutboxStatus.PENDING,
        )
    )
}

Both writes live in one transaction, so the app can never land in the impossible state where the UI shows the rename but nothing remembers to sync it. Crash between them and neither happened.

Gotcha. @Transaction on a repository method does nothing — use db.withTransaction { }. Inside it, Room's transaction context survives withContext (suspend DAO calls are routed back to the transaction dispatcher via its context element); what actually breaks the transaction is escaping coroutines entirely: runBlocking, GlobalScope, executors/CompletableFuture, or blocking DAO calls on a foreign thread.

One database setting mattered under all this write traffic: WAL (write-ahead logging) stays on. It is Room's default on everything but a low-RAM device (journal mode AUTOMATIC), so the rule was simply never to turn it off. WAL lets the UI's Flow reads keep hitting the last committed snapshot while an outbox write is in flight, so a busy queue never shows up as UI jank from lock contention.

Draining the queue

suspend fun drainOutbox(): DrainResult {
    outboxDao.reclaimExpiredLeases(now())          // A) only IN_FLIGHT whose leaseUntil < now → PENDING
    val batch = outboxDao.drainable(now(), limit = 50)  // B) PENDING|FAILED_RETRYABLE, nextAttemptAt<=now, ORDER BY sequenceNumber — stops at the first op still IN_FLIGHT or backing off, so nothing overtakes it
    for (op in batch) {
        // atomic claim: UPDATE ... SET status=IN_FLIGHT WHERE status IN (drainable). Returns false if
        // the row is no longer drainable -- e.g. cascade-rejected by a Fatal EARLIER IN THIS BATCH.
        // Without this guard the stale in-memory list would fire network calls for rejected dependents.
        if (!outboxDao.claim(op.sequenceNumber, leaseUntil = now() + LEASE_MS)) continue
        val result = try {
            op.execute(api)                             // network — OUTSIDE any DB transaction
        } catch (e: CancellationException) {
            throw e                                     // never swallow cancellation
        } catch (e: IOException) {
            Retryable(e)                                // dropped connection: reschedule, never counted toward the cutoff
        }
        when (result) {
            is Success   -> db.withTransaction {                  // apply server's CANONICAL entity, then finish
                doorDao.upsertCanonical(result.entity)            //   ← restore skipped fields (field-scoped if other ops pending)
                outboxDao.delete(op.sequenceNumber)
            }
            is Conflict  -> reconcile(op, result)                 // adopt / rebase per policy
            is Retryable -> { outboxDao.reschedule(op, backoffWithJitter(op.retryCount)); break }  // stop — keep order
            is Fatal     -> cascadeReject(op)                     // reject op + dependents; the claim guard above skips them for the rest of this batch
        }
    }
    return if (outboxDao.hasMore(now())) DrainResult.Retry else DrainResult.Ok
}

A few details in that loop were easy to get wrong. OkHttp throws IOException on transient failures (it does not return them), so the try/catch that maps them to Retryable is load-bearing, and CancellationException is always re-thrown, never folded into a broad catch. Marking IN_FLIGHT and calling the network are unavoidably separate transactions (you cannot hold a DB transaction across a network call); the lease plus the idempotent operationId are what make that gap safe. We drain in bounded batches of 50: a week's backlog in one List is a memory problem, and enough serial round-trips blow past WorkManager's ~10-minute ceiling.

The hard decisions and trade-offs

Four outcomes, not two. The naive version handles success and failure, and that is where fragile offline support usually starts. Retryable versus fatal has to be decided explicitly: a dropped connection is retryable; a 400 "illegal characters" is fatal, and retrying it forever jams the queue behind one poison message. A Fatal rolls the user's edit back for good, so it has to be certain. Anything unclassified is retried, and a cutoff on retryCount stops an unknown error from looping forever. We decided it per HTTP status:

Status Bucket Why
400, 422, 403 (this op only) Fatal client sent data, or asked for an action, the server will never accept — a retry can't fix it
404 (update) Fatal the entity no longer exists upstream — roll back, then the targeted re-fetch removes it locally
404 (delete) Success the goal state is already reached (Part 3)
408, 500, 502, 503, 504 Retryable timeout or transient server hiccup — back off and try again
429 Retryable throttled — honor Retry-After, then back off and try again
409 Conflict adopt the server's state, then reconcile per the field's policy
401 (expired) Refresh + retry still authenticated — refresh once (single-flight) and retry
401 (refresh fails) Park + clearCredentials() signed out elsewhere — keep the user's edits, park the queue, prompt re-login
unclassified Retryable (poison cutoff) an unknown error must not destroy the user's edit; retryCount demotes it to Fatal if it never clears

Within a batch, outcomes are per-op: successes complete, the first Retryable breaks to preserve order, and fatals cascade-reject. Because the batch is an in-memory snapshot, the atomic claim guard (claim returns false, continue) is what stops the loop from sending dependents it has just cascade-rejected.

Leases instead of resetting everything. A process death mid-request can strand a row in IN_FLIGHT, so each claim carries a lease deadline (leaseUntil = now + LEASE_MS, about 30 s) and the drainer reclaims only expired leases back to PENDING. Reclaiming every IN_FLIGHT row unconditionally would reset a request that is merely slow (a 15-second call on 3G is not a crash) and fire a needless duplicate.

Order is part of the contract. A user, offline, creates a door and then renames it. The rename is meaningless until the create has landed and been assigned a real identity. Drain them out of order, or drain the rename while the create is still in flight, and the server 404s the rename against an ID it has never heard of. So the outbox is an ordered log, drained sequentially, and a transient failure stops the drain rather than skipping ahead (the break above). A single ordered queue was the cheap, correct default for us. Its one weakness is an op stuck Retryable forever (a 503 that never clears for one entity), which would block everything behind it, so retryCount doubles as a poison cutoff: past a threshold, the op is demoted to FAILED_FATAL and cascade-rejected. Only server answers count toward it (5xx, 429), never a dropped connection: a dropped connection reschedules without touching retryCount, because a week of flaky Wi-Fi must not be able to roll back a user's edit.

Cascade rollback on fatal. If "create door" is rejected, the "rename" behind it refers to an entity that will never exist, and draining it can only 404 forever. A fatal failure has to cascade: reject every dependent and roll the local optimistic state back. Dependents here are the later ops on the same entity, the walk the second index serves; chains that span entities are Part 4's groupId. The rollback is the preImageJson, applied newest-first. Walking the chain in reverse, each pre-image is restored: an edit's pre-image is the entity before that edit; a CREATE's pre-image is null, so "restoring" it means deleting the row; a soft-delete resurrects. Local state lands exactly where it stood before the doomed chain began. This is why the schema carries a pre-image snapshot and not just an isDirty flag: undo has to know the actual prior value.

Compacting the queue, but only where it is provably safe. A user offline for a week who renames the same door thirty times leaves thirty RENAME ops: thirty round-trips to reach a state the twenty-ninth already superseded. Before draining, we squash the one case that is provably safe: an unbroken run of the same scalar update to the same field of the same entity collapses to the last one. We deliberately did not collapse across op types (folding a CREATE with the edits after it, or dropping an entity a later op deletes): if any op in between established a cross-entity dependency, that corrupts referential integrity. Squashing only touches PENDING ops that have never been sent, and the merged op inherits the earliest collapsed op's operationId, preImageJson and sequenceNumber, not the latest. Keep the last op's pre-image instead and a later fatal cascade rolls back to an intermediate value the user never saw.

No bytes in the queue. payloadJson is for arguments, never for files. A photo must not be base64'd into a row: Android reads rows through a fixed-size CursorWindow (2 MB by default), and a row carrying a multi-megabyte blob throws SQLiteBlobTooBigException and drags every query on the table down with it. Binaries live on disk and upload on a separate, slower channel; the CRUD op (a profile update with a new avatar) carries a stable assetId reference, never the bytes, so one slow 2 MB avatar upload never blocks a 2 KB rename.

The outcome

The life of one outbox operation
The life of one outbox operation
  • Every edit works offline and shows up instantly; nothing in the UI needs a "disabled while offline" state.
  • A write survives the process being killed at any point: the state change and its outbox entry commit together or not at all.
  • The queue drains in order, backs off politely, and a single bad request can no longer jam it.
  • After squashing, the queue holds one op per edit the user actually meant, not one per keystroke.

The machine was durable now. It still could not handle the messy cases: a token that expires mid-drain, a retry that would double-apply, a 409 from the server, a delete that fails after the row is already gone from the UI. That is Part 3.

Stack & takeaways

Stack: Kotlin, coroutines, Room (withTransaction, WAL), OkHttp.

  • Never disable a button because the network is gone. Apply locally, record the intent, sync later.
  • Write the state change and the outbox entry in one transaction. That single rule removes a whole class of "the UI shows it but it never synced" bugs.
  • A drain has four outcomes, not two. Decide retryable versus fatal per status, and retry anything you have not classified, with a poison cutoff.
  • Keep a pre-image, not a dirty flag. Cascade rollback needs the real prior value.
  • Squash only what is provably safe, and keep the earliest op's identity when you do.