← All postsOffline-First on Android · Part 4 of 4

Offline-First on Android, Part 4 — Multi-Device, the Backend Contract & Operating at Scale

Two engineers looking at a phone together

How one phone's change reaches another, the contract the backend had to sign, when the app actually syncs, and what changes when you go from one entity to ten.

Parts 1–3 built one device: local-truth reads with delta pull, a durable outbox engine, and reconciliation when the server disagrees. This part is where it became a system: across devices, against a real backend, across the whole user base.

The problem

Our users rarely had just one device. A household had a phone and a tablet, or two people sharing one account, and the moment that happens you have a tiny distributed system. Three questions followed:

  • How does a change made on the phone show up on the tablet, without anyone pulling to refresh?
  • What does the backend have to guarantee for any of Parts 1–3 to hold? Nearly every "impossible" sync bug turns out to be a contract violation the client cannot compensate for.
  • When should the app actually sync? A timer is the obvious answer, and the wrong one.

And all of it had to work on every phone running the app, none of which we could SSH into.

Our approach

Multi-device: we had already built both halves

Push (device → server) is the outbox. Pull (server → device) is the delta-with-a-cursor. The tablet pulls on the same event triggers as everything else, plus one addition: while a screen is in the foreground, the app runs a light periodic delta pull, so an open, idle device converges within seconds without pull-to-refresh. That is the only foreground timer in the design. It runs only while the app is visible, when the radio is awake anyway, and it only pulls; writes still go out on the event triggers below.

The failure mode we designed against: two devices, both offline, both edit, both reconnect. If the merge treated "absent from my last delta" as "deleted," device B would propagate a phantom delete of the work device A just did. Absence-is-not-deletion plus server-versioned conflicts (Part 3) is what stops the two from ping-ponging each other's state.

Multi-device is exactly where people reach for timestamps, so we wrote the rule down: use timestamps to show humans things; never use them to decide machine things.

  • Display: "last updated 3 minutes ago," sorting a feed for the eye. Timestamps are perfect here.
  • Decisions: "which of these two edits wins," or "what changed since I last synced." Phones have wrong clocks, users change them by hand, and NTP corrects them backward. These are decided by a server-assigned version (for conflicts) and an opaque server cursor (for deltas): the server has one clock.

The createdAt in the outbox row is display-only. This device's own pending ops are ordered by sequenceNumber, not by any clock; across devices, only a server version decides.

The backend contract, written down

Everything in this series assumes a backend with specific properties, so we turned them into a written contract with the backend team:

The backend MUST… …or this breaks
Assign a monotonic, per-entity version on every write LWW / field-merge have no "later" → conflict resolution corrupts
Return the canonical entity (with its new version) in every write response the delta-skip can't recover concurrent server-side changes → permanent staleness
Emit an opaque, resumable cursor with at-least-once delta semantics a skipped change → silent divergence; a non-resumable page → cold sync can't finish
Include tombstones in the delta (not merely omit deleted rows) deletes never propagate — the flip side of absence ≠ deletion
Honor the idempotency key for every op type, dedup retained at least as long as delta cursors, and answer a replay with the original response, not a 409 the retry story collapses into duplicates or spurious 409s
Treat 404-on-delete as success; distinguish 401 expired vs revoked the delete-is-success and single-flight-401 branches misfire
(For cross-entity mutations) expose an atomic batch endpoint multi-entity operations half-apply

The protocol itself is versioned too: an old client (offline for weeks) will talk to a newer server, so the app sends a protocol-version header, delta changes stay additive-only, and "you must update before syncing" has a path that does not eat the unsent outbox. Writing it down early also surfaces the rows a backend cannot satisfy (commonly no per-entity version, only an updatedAt) before they turn into a corruption report, and that row decides whether LWW is even possible.

Syncing on events, not on a timer

A polling timer is almost never the answer: a 30-second poll is both too aggressive (waking the radio for nothing) and too slow (half a minute of stale data after a reconnect). The app syncs on events, with a durable backstop underneath:

  • Connectivity regained. A NetworkCallback drains when the network returns, but only after NET_CAPABILITY_VALIDATED, not merely NET_CAPABILITY_INTERNET. The gap between those two is the captive-portal hotel WiFi that fails every request.
  • App start / cold launch. The app may have died with ops pending.
  • Return to foreground. Sync before the user touches anything; this is also the main multi-device pull trigger.
  • Right after a local mutation. Try immediately; if it fails, the op is already safe in the outbox.

Those triggers love to fire in the same second, so overlapping syncs are coalesced with the cheapest single-flight guard there is, a one-slot flow:

private val kicks = MutableSharedFlow<Unit>(extraBufferCapacity = 1, onBufferOverflow = BufferOverflow.DROP_OLDEST)

fun requestSync() { kicks.tryEmit(Unit) }          // never suspends — safe from a NetworkCallback thread

scope.launch { kicks.collect { drainOutbox() } }   // one drain at a time; a burst of kicks collapses to "one more"

While a drain runs, a burst of kicks keeps exactly one pending, so you get one more drain after the last kick: never five, never zero. Dropping is lossless because every kick says the same thing; the intent itself stays in the durable table, which is why this trick is never applied to the operations themselves. The collector is subscribed at startup, because with no subscriber the emissions are dropped. (One-time WorkManager drains are different: enqueueUniqueWork(..., ExistingWorkPolicy.KEEP) drops a kick that arrives while a drain is running, so there the drain's own DrainResult.Retry from Part 2 is what catches the work that came in meanwhile.)

WorkManager as the durable, battery-aware backstop

Every trigger above needs the process alive. But the outbox promises a write survives the app being killed, so something has to drain it after a force-stop, a memory-reap or a reboot. That is the one place a schedule earns its keep:

val work = PeriodicWorkRequestBuilder<SyncWorker>(15, TimeUnit.MINUTES)
    .setConstraints(
        Constraints.Builder()
            .setRequiredNetworkType(NetworkType.CONNECTED)   // don't even wake without a network
            .setRequiresBatteryNotLow(true)                  // respect the user's battery
            .build()
    )
    .setBackoffCriteria(BackoffPolicy.EXPONENTIAL, 10, TimeUnit.SECONDS)
    .build()

WorkManager.getInstance(ctx).enqueueUniquePeriodicWork(
    "outbox-sync", ExistingPeriodicWorkPolicy.UPDATE, work,  // UPDATE: unique AND self-healing on upgrade
)

Telling the user the truth about sync state

The reactive DB→UI stream shows what the data is, never whether it is confirmed: an optimistic write and a server-acked one look identical, right up until the write fails and silently reverts. So each row carries a status (pending, syncing, synced, failed), and a fatal drain shows an explicit, retryable error instead of a quiet rollback: "couldn't save the new name, tap to retry."

when (row.syncStatus) {
    PENDING -> pendingIndicator()                      // subtle "waiting to sync"
    SYNCING -> syncingIndicator()                      // the op is in flight
    FAILED  -> retryAffordance { repo.retry(row.id) }  // actionable, never a dead end
    SYNCED  -> Unit                                    // the default — invisible
}

The hard decisions and trade-offs

UPDATE, not KEEP, for the periodic work. KEEP looks right and does keep the schedule unique, but it also tells WorkManager to ignore the new spec entirely: ship v1.1 with a new constraint and every existing user keeps running the v1.0 config forever. ExistingPeriodicWorkPolicy.UPDATE (WorkManager 2.8+) is unique and applies the new configuration.

Two constraints, and no more. CONNECTED means the worker does not wake without a network; batteryNotLow means the app does not spin up the radio while the user nurses 8% on the last train home. We stopped there on purpose: add charging and idle, and a user who lives at 20% never syncs.

Our own jitter. Nothing in that setup adds jitter for you. A mass trigger, such as an outage ending, enqueues a drain on every device at the same instant: a self-inflicted DDoS on a backend that is just recovering. With enough phones out there that is a real risk, so we compute our own jittered delay and spend it as the WorkRequest's setInitialDelay. (Result.retry() rides the jitter-free grid, which is fine for one device's own cadence and wrong for a stampede.)

fun backoffWithJitter(attempt: Int): Long {
    val base = min(1_000L * (1L shl attempt.coerceAtMost(30)), MAX_BACKOFF_MS)  // 1s,2s,4s… then cap
    val jitter = Random.nextLong(0, 1_000L)                                     // up to 1 s of spread, also at the cap
    return base + jitter                                                        // cap first, THEN jitter: capped devices still spread out
}

We also designed for "sync may be delayed." Periodic work is best-effort: App Standby buckets throttle rarely-opened apps hard, so the 15-minute floor can stretch to hours. The periodic tick is the backstop, not the main path; most drains happen because the user reconnected or brought the app to the foreground.

No "which version do you want to keep?" dialog. Conflicts resolve by policy (Part 3), so the UI reports outcomes and never makes the user arbitrate a merge. For a value quietly updated from another device, a small "updated elsewhere" hint prevents surprises.

The local database became real user data. Local-first promotes the device from throwaway cache to primary store, and that brought four responsibilities tutorials tend to skip:

  • Encryption at rest. Sensitive data uses SQLCipher with the passphrase wrapped by the Android Keystore. Since that key is unavailable until the first unlock after a reboot, background sync waits for it (WorkManager already does).
  • Migrations that never eat the outbox. Real Room Migrations only; fallbackToDestructiveMigration() never ships, because it silently wipes unsynced operations.
  • Keeping the database out of Auto Backup. With the default allowBackup, Android restores app databases on reinstall, outbox and cursor included, so a fresh install could replay weeks-old edits under whatever account signs in next. The database and the sync metadata are excluded with dataExtractionRules (fullBackupContent before API 31); and since the Keystore key is not restored either, a restored SQLCipher file cannot be opened, so "cannot open" is treated as "empty store."
  • Surviving a cold, empty store. "Clear storage" and reinstalls both happen, and on Android they wipe DataStore and SharedPreferences along with the database, so no local flag survives to say which one it was. An empty store is treated as the bootstrap from Part 1: the server knows whether this account already has data, and because deletes travel only as explicit outbox operations, an empty device is never a mass delete to propagate back.

Testing the parts only offline-first needs. Two tests this domain cannot do without: a fake network that can succeed but drop the ACK (the server writes, the client never sees the 200), which is how a duplicate-create gets caught in CI instead of by a user; and a harness that kills the process mid-withTransaction to prove the outbox actually survives process death. For the emergent multi-device failures, a seeded N-device simulation asserts one invariant after everything goes quiet: every device converges, and no write is lost.

The outcome

  • An edit on one device shows up on another open device within seconds, with no pull-to-refresh, and two devices editing offline no longer undo each other's work.
  • The backend contract is a versioned, written document both teams agreed on, so sync bugs are traced to a broken guarantee instead of argued about.
  • The app syncs when something actually happened, coalesces bursts into one drain, and a WorkManager backstop drains whatever a killed process left behind, without stampeding the backend after an outage.
  • Users can see which of their changes are pending, synced or failed, and a failed one is one tap from a retry.

The test that matters most is the support inbox: you know it worked when it goes quiet, with no more "my rename didn't take" and no more "the door I deleted came back."

What comes next: from a recipe to an engine

Everything in this series was shown for one entity, the door. Hand-copy it for a tenth entity and you have ten copies of the subtlest code in the app, so the next step is to lift it into one generic engine driven by a per-entity Syncable<E> registration: its DAO, its delta endpoint, its per-field conflict policy, its dependencies. The engine owns the drain loop, coalescing, backoff and merge, and the eleventh entity becomes registering two objects. Four concerns become the engine's job the moment it exists:

  • Cross-entity atomicity. Related ops are tagged with one groupId so they drain, and cascade-reject, as a unit. That needs the atomic-batch endpoint from the contract.
  • Multi-account isolation. The store is partitioned by accountId (a separate DB file and key per account); on switch, the drainer pauses and waits out in-flight ops. Draining account A's queue under account B's token is a data leak.
  • Retention and recovery. Success already deletes its row; FAILED_FATAL rows are purged after a short debug window, and a supervised "drop the poison op, re-fetch its entity" path keeps one dead op from freezing an account's sync.
  • Observability and anti-entropy. You are running a database on a fleet of phones you cannot SSH into, so the engine emits queue depth, oldest-pending age and fatal-rate, and runs a cheap periodic digest check per scope (count, xor-of-versions); a mismatch resets that slice's cursor and re-pulls.

Stack & takeaways

Stack: Kotlin, coroutines and Flow, Room, OkHttp, WorkManager 2.8+, ConnectivityManager NetworkCallback, SQLCipher with the Android Keystore.

The whole series in eight rules:

  1. The device owns the truth for the user's own actions. Every other decision falls out of this.
  2. The UI reads the database, never the network, and pulls deltas with an opaque server cursor stored in Room, so data and watermark commit together.
  3. Writes apply locally and enqueue in one transaction. The outbox drains in order, with four outcomes and cascade rollback on fatal.
  4. Identity is client-minted. Retries and lost acks become non-events; adopting the server's ID fills in serverId and never rewrites the PK.
  5. Absence is never deletion, and a delta never clobbers an unsent edit. Deletes flow as tombstones, soft and reversible on the client; 404 is success.
  6. Conflict policy is a property of the field, decided by server versions. Timestamps show humans; versions decide machines.
  7. Sync on events, backstopped by WorkManager: validated connectivity, coalesced triggers, jittered backoff, and honest per-row sync state in the UI.
  8. The backend is an agreed, versioned contract, and past one entity, the recipe gets lifted into an engine.

None of this is exotic. For an app like ours (one writer per entity at a time, a moderate number of mostly hierarchical entities) there was no clever algorithm to adopt, just a handful of stubborn principles applied consistently, native-first, and owned by our team. Genuinely collaborative data wants a CRDT library, not LWW; an entity count that outgrows what one team can keep correct wants a turnkey sync engine such as PowerSync. Hand-roll when your sync semantics are simple and the backend contract is yours to shape; buy when they are not.