← All postsOffline-First on Android · Part 3 of 4

Offline-First on Android, Part 3 — Idempotency, Conflicts & Edge Cases

Two engineers with laptops in a meeting room

Expired tokens mid-drain, retries that must not duplicate, conflicts settled per field, and deletes you can take back.

Part 2 built the machine: a durable outbox that survives process death and drains in order. This part is everything that pushes back on it, ending with the fourth requirement from Part 2: reconciling cleanly when the server disagrees.

The problem

A durable queue is only half of the job. Once the outbox existed, it had to survive the situations real users would put it in:

  • A login that expires while the phone is offline, so the whole queue comes back to a wall of 401s the moment the signal returns.
  • A "create door" that the server applies, but whose response dies on the way home. The client never sees the ack, retries, and now there are two doors.
  • A delta from the server that still carries a door's old name, because the server has not heard about the rename yet, and overwrites the rename the user is looking at right now.
  • Two writers, a phone and a tablet, or two people sharing an account, editing the same door.
  • A delete that the server rejects after the row has already vanished from the screen.

None of these is exotic. Each of them, left alone, turns into a support ticket that says "my rename didn't take" or "the door I deleted came back."

Our approach

Single-flight token refresh on 401

A token can expire while the user is offline, so the outbox can come back to a 401 wall on reconnect. We handle it as "refresh once, then retry," and the refresh is single-flight, so twenty queued ops do not fire twenty concurrent refreshes the instant the signal returns. The hook is okhttp3.Authenticator, not an interceptor: an interceptor wrapping the refresh in runBlocking blocks one of OkHttp's finite dispatcher threads and can deadlock the client against itself. Authenticator is invoked only on a 401, on a thread where a synchronous refresh is expected.

class TokenAuthenticator(
    private val tokens: TokenProvider,
) : Authenticator {

    @Synchronized                                       // serialize concurrent 401s — the single-flight guard
    override fun authenticate(route: Route?, response: Response): Request? {
        if (responseCount(response) >= 2) return null   // already retried once — stop, don't loop forever

        val failed = response.request.header("Authorization")?.removePrefix("Bearer ")
        val current = tokens.accessToken()
        val fresh = if (current == failed) {
            // We're first. null = terminal (revoked / account gone) → WIPE and give up.
            tokens.refreshBlocking() ?: run { tokens.clearCredentials(); return null }
        } else {
            current                                     // someone already refreshed — reuse it
        }
        return response.request.newBuilder()
            .header("Authorization", "Bearer $fresh")
            .build()
    }

    private fun responseCount(r: Response): Int =            // not built-in: OkHttp chains retries
        generateSequence(r) { it.priorResponse }.count()      // via priorResponse; count the links
}

@Synchronized buys single-flight with no Mutex and no runBlocking: the first 401 refreshes, the ones queued behind it see the changed token and reuse it, and responseCount caps retries. It has preconditions we had to respect: refreshBlocking() runs through a separate client, so a refresh cannot recurse into the authenticator; the authenticator is a true singleton (@Synchronized locks on this, so a per-request instance serializes nothing); and if an identity provider rotates the token on every fetch, you gate on a monotonic refresh-generation counter instead of the current == failed equality. The terminal path must call clearCredentials(): skip that wipe and every queued 401 keeps seeing the dead token and thrashes.

Client-minted identity, so retries cannot duplicate

The fix for the duplicate-create trap was to stop letting the server assign identity. The client generates a UUID at creation time and sends it as the operation key; a create for an ID the server has already seen resolves to "you already did this, here is the result."

db.withTransaction {                               // same rule as Part 2: state + outbox entry commit together
    val localId = UUID.randomUUID().toString()     // stable PRIMARY KEY, minted once, never rewritten
    doorDao.insert(Door(localId = localId, serverId = null, name = name, synced = false))
    outboxDao.enqueue(createDoorOp(target = localId, name = name))
}

When the server assigns its own canonical ID, we do not rewrite the local primary key to it: a PK rewrite cascades to every foreign key pointing at it and can violate uniqueness mid-update. Each row has a stable localId (the PK, minted once) and a nullable serverId; adopting the server's ID simply fills in serverId, inside one withTransaction.

The same idea covers the "unknown outcome" problem for every other op. A RENAME that times out after the server applied it is safe to retry only if the client re-sends the stored operationId and the server dedups on it for every op type, for at least as long as it honors delta cursors (Part 1): if a months-old cursor still syncs, a months-old operationId must still be recognized. The window only matters for an op that was sent once and whose ack was lost; an op that never left the phone cannot duplicate however old it is. For a sent-but-unconfirmed op older than the window, the app re-fetches the entity before retrying instead of trusting the dedup. We kept the two answers distinct on the wire: a replayed operationId the server has already applied comes back as the original success response (the canonical entity), never as a 409. 409 stays reserved for a genuine version conflict, which the drain hands to the per-field policy below. Blur the two and the client cannot tell "just adopt this" from "reconcile."

Merge rules: absence is not deletion, and unsent edits win

One merge rule prevents the worst class of bug here. When you merge server state into the local DB, the naive approach is: anything on the server I do not have, I add; anything I have that the server does not, I delete. That last clause is a landmine. The thing you have that the server does not is not garbage; it is the write the user made 200 ms ago that has not synced yet. So our rule is: an entity present locally but absent on the server is not stale, it is not-yet-sent. Leave it alone. Deletions flow only as explicit tombstones, never inferred from absence. That is the rule for deltas; the full bootstrap snapshot of Part 1 is the one exception, and even it spares rows with pending ops.

The mirror case is just as sharp: the entity exists on both sides, but there is an unsent local edit. A blanket upsertAll walks straight into it: a delta arrives carrying the old name and overwrites the rename the user is looking at, flickering it back. The rule that closes it: a delta may not overwrite an entity that has a PENDING or IN_FLIGHT op in the outbox. Your own unsent writes always win. We do that check as a set operation, not a loop (an N+1 turns one sync into 501 queries):

-- inside upsertChanged / deleteConfirmed
WHERE NOT EXISTS (
    SELECT 1 FROM outbox_operations o
    WHERE o.targetEntityId = :deltaRowTargetId
      AND o.status IN ('PENDING','IN_FLIGHT')
)

Gotcha. x NOT IN (SELECT …) returns nothing the instant its subquery yields a single NULL, silently dropping the whole delta. NOT EXISTS is null-immune, and it drives the index.

The hard decisions and trade-offs

A skip is a deferral, and it has to land somewhere. Say a delta carries an unrelated change to a row the user is mid-editing: some other field of the door changed on the server while a rename is pending. Skip the whole row and the cursor moves on, so that server-side change is lost forever. The skip is only safe if the entity is reconciled the moment its op resolves. On Success, the app applies the server's canonical entity from the write response (the upsertCanonical in Part 2's drain loop), which echoes back the new name and the other field's new value. On a fatal rollback, the entity is flagged for a targeted re-fetch. For a server-wins field the skip is wrong entirely: a pending local edit is exactly what must not win, so the delta lands.

Field-scoped tracking only where it pays. For entities that merge per field, all three paths (skip, rollback, canonical echo) have to be field-scoped: two pending ops on different fields (rename + move-room) mean the rename's canonical echo still carries the old room, so a blind whole-row upsert clobbers the move. Field-merge needs per-field dirty tracking. Everywhere else, a whole-entity pre-image is the right, simple tool, and we did not build the finer grain where we had no genuine field-merge entities.

A conflict policy per field, not one global rule. When two writers disagree there is no universal answer, so we decided it per field, on purpose:

  • Last-write-wins (LWW). The later write wins, but "later" is decided by a server-assigned version, never the client wall clock. In practice that makes "later" mean "last to reach the server," not "last edited": an edit typed a week ago on a device that was offline beats an edit someone made yesterday, simply because it reached the server last. That is the trade we chose; the alternative was trusting client clocks. It is right for independent scalars (a name, a color), and because it silently discards the loser, we used it only where discarding is acceptable.
  • Field-level merge. Two edits touching different fields (rename + move-room) are not really a conflict: merge per field and both survive. It costs per-field dirty tracking, which was worth it for our richer entities.
  • Server-wins (rebase). For invariants the client cannot enforce (permissions, billing, membership), the server is the authority: discard the local change, re-derive, and tell the user.

The policy is baked into the schema per field. On one Door, name is "LWW by server version" while isLocked is strictly "server-wins": the client must never decide a lock state on its own, even offline. Because the policy is a property of the field rather than an ad-hoc if in the sync code, nobody can accidentally pick the wrong one for a security-sensitive field.

No CRDTs. LWW and field-merge both resolve by picking a winner. For genuinely collaborative data (two people typing into one note) that silently loses work, and you reach for CRDTs. Our entities had effectively one writer at a time, so LWW and field-merge were the right tools and a CRDT would have been over-engineering.

Deletes are writes that run backward, so we made them reversible. Two rules. First, 404 on a delete is success: if the server says the thing does not exist, the goal state is achieved. Retrying until you get a 204, and treating 404 as an error, is how you build another stuck queue. Second, local deletes are optimistic but reversible. Instead of hard-deleting the instant the user taps, the app soft-deletes (a deletedAt tombstone, hidden from the UI) and enqueues. The row vanishes immediately, so it feels instant; but if the server rejects the delete fatally, the row is resurrected instead of leaving the user staring at a ghost or watching it silently reappear later. The tombstone survives until the server confirms 204 or 404, and then it is purged. Users forgive a slow delete; they do not forgive one that comes back.

The outcome

  • An expired login on reconnect costs one refresh, not twenty, and a revoked one parks the queue and asks the user to sign in again instead of thrashing.
  • A lost ack can no longer create a duplicate door: every op carries an ID the server dedups on.
  • A sync can no longer flicker an unsent rename back, or treat "not on the server yet" as "deleted."
  • Every conflict resolves by a policy chosen per field, and lock state is never decided by the client.
  • A rejected delete brings the door back with an explanation instead of a ghost.

That covered a single device. Part 4 takes it beyond the phone in your hand: how an edit reaches your other devices, what the backend had to guarantee, when the app actually syncs, and what changed when we went past one entity.

Stack & takeaways

Stack: Kotlin, OkHttp (Authenticator), Room, SQLite (NOT EXISTS set queries).

  • Refresh tokens in an Authenticator, single-flight. An interceptor with runBlocking can deadlock OkHttp against itself.
  • Let the client mint identity. Retries and lost acks become non-events, and adopting the server's ID fills in serverId instead of rewriting the primary key.
  • Absence is never deletion, and a delta never overwrites an unsent edit. Deletes travel only as tombstones.
  • Make conflict policy a property of each field, decided by server versions, never client clocks.
  • Treat 404 on delete as success, and make deletes soft until the server confirms.