The Architecture Of A Local-First Sync System

In the first post, I explained why I am not treating this project as a CRUD todo app.

The short version:

User intent becomes an operation. The operation commits once. Clients converge through the committed log.

This post is about the target system shape around that idea.

A local-first task app has more moving parts than a normal online-only app:

• local client databases
• pending operation queues
• sync cursors
• an API boundary
• authentication and sessions
• a command/write path
• a sync/read path
• PostgreSQL shards
• an outbox
• Kafka
• derived projections
• WebSocket hints
• object storage
• edge protection

The hard part is not drawing boxes. The hard part is deciding which boxes are allowed to be true.

For this app, the rule is:

PostgreSQL workspace state is canonical.
Everything else is either a client replica, a cache, a hint, or a projection.

That rule keeps the architecture from turning into a pile of services that all think they own the truth.

This post describes the intended production architecture. The current core demo is a smaller slice: one Go API process, one local PostgreSQL database, operation-first writes, /sync, IndexedDB client state, and the Go auth/session foundation. Kafka publishing, WebSocket hints, runtime shard catalog routing, attachments, snapshots, object storage, Valkey, and Keycloak login are staged future slices.

The System In One Diagram

Here is the high-level architecture:

This looks like a lot, but the core loop is small:

Client operation
  -> command transaction
  -> PostgreSQL workspace log
  -> /sync
  -> client convergence

Everything else supports that loop.

Start With The Source Of Truth

The most important architecture decision is where truth lives.

Not where data is copied. Not where data is cached. Not where data is indexed. Where truth lives.

For this project, canonical workspace state lives in PostgreSQL. More specifically, in a sharded deployment it lives in the PostgreSQL shard that owns a given workspace_id.

That shard owns:

• workspace_events
• current-state tables
• workspace_sequences
• processed_operations
• outbox_events
• workspace membership and permission state
• attachment metadata
• snapshot metadata

The command transaction writes the operation log, current state, idempotency record, and outbox row together.

If this transaction commits, the operation happened. If it rolls back, the operation did not happen. Ultimately, PostgreSQL decides.

Canonical, Derived, And Ephemeral

A lot of architecture mistakes come from mixing up these categories.

Category	Examples	Can it lag?	Can it be rebuilt?	Can it authorize workspace writes?
Canonical	PostgreSQL workspace log, current-state tables, permissions, idempotency records	No	From backups, retained events, repair tools, and controlled workflows	Yes
Derived	Search indexes, reminder schedules, notification records, analytics, activity feed	Yes	Yes	No
Ephemeral	Valkey cache, WebSocket connections, in-memory worker state	Yes	Yes	No
Client local	IndexedDB/SQLite materialized state, local search, pending operations	Can be ahead or behind	From snapshot plus sync, except unsent local work	No

The architecture is easier to reason about once this is clear.

A projection can request repair. A client can submit an operation. An admin workflow can submit a controlled operation. But nothing gets to directly mutate canonical workspace state outside the command path.

Clients Are Local Replicas

The clients are not thin views over server state.

The web client uses IndexedDB. Mobile and desktop clients will use SQLite.

Each client stores:

• local materialized workspace state
• pending operations
• device_id
• local_counter
• last_seen_sequence per workspace
• retry metadata
• local search data
• auth/session state

That gives the app local-first behavior. The client can render tasks without fetching every screen from the server. It can accept edits while offline. It can restart and still remember unsent work.

But the client is not canonical.

A client can be ahead of the server because it has optimistic pending operations. It can be behind because it has not synced yet. It can miss WebSocket hints. It can have old local data.

The client is allowed to hold local state, but it must converge through the server's committed sequence order.

The Client Contract

The client contract is small but strict.

The client must:

1. create a stable device_id;
2. keep a monotonic local_counter;
3. allocate operation_id from device_id + local_counter; (kind of like snowflake Id's by twitter, not exactly)
4. persist the pending operation before treating it as durable;
5. retry with the same operation_id;
6. call /sync after commits, reconnects, restarts, and WebSocket hints;
7. apply events in server sequence order;
8. advance last_seen_sequence only after local apply succeeds.

That gives us this loop:

This is the client's side of the architecture. The server's side starts at the edge.

The Public Edge

Production traffic should not hit random origin servers directly. I have built the web app to be a Single Page application which can be put on cloudflare network for delivery; it is a static web app which get populated by data from database after login.

The public path should look like this:

client
  -> Cloudflare or equivalent edge
  -> regional load balancer / ingress
  -> Go services
  -> private infrastructure

The edge owns DNS, public TLS, WAF, DDoS protection, static asset caching, coarse rate limits, bot/abuse filtering, and origin IP hiding. I choose this to get better security for a lower price, since cloudflare provides these services, and I can focus more on making the app for now.

The edge does not own task state, workspace authorization, operation ordering, durable sync, refresh-token storage, database access, Kafka access, or object storage internals. (I might consider using D1 for caching but nothing for now)

The edge is protection and routing. It is not product truth.

Public Hostnames

The public hostnames should make the system easier to reason about.

A production layout could be:

app.example.com    -> static web app
api.example.com    -> HTTP API
auth.example.com   -> auth/session flows
sync.example.com   -> WebSocket hints
files.example.com  -> attachment gateway
admin.example.com  -> restricted admin

For a smaller deployment, some of these can collapse. For example, wss://api.example.com/v1/realtime is fine early on.

The important boundary is:

public clients -> edge -> Go services -> private data systems

Never this:

public clients -> Postgres
public clients -> Kafka
public clients -> Valkey
public clients -> Ceph internal endpoint

If the architecture allows a browser or mobile client to know about internal database or broker addresses, the boundary is already broken. I am trying to reduce the attack surfaces to make the app more secure.

Go Backend Services

The backend is written as a small number of coarse services, not one microservice per entity. That is intentional.

The consistency boundary is the workspace, not the table.

A task.create operation touches permissions, task state, workspace sequence, operation log, idempotency, and outbox. A task.move may validate project, section, parent task, ordering, permissions, and task state. A comment.create may affect comment state, activity history, notifications, and search indexing.

Splitting this into task-service, comment-service, label-service, section-service, and notification-service too early would create coordination problems that do not need to exist.

So the service design is coarse:

Service	Job	Canonical writer?
api-gateway	Public HTTP API entry point, routing, coarse auth, error envelope	No
auth-session-service	App sessions, device sessions, refresh-token rotation	Yes, for session data
workspace-command-service	Commits workspace operations	Yes, for workspace task state
sync-query-service	Serves /sync and snapshots	No
realtime-gateway	Sends WebSocket hints	No
outbox-publisher	Publishes committed outbox rows to Kafka	No
attachment-service	Attachment metadata and object access flow	Metadata through command path
search-indexer	Maintains search projection if needed	No
reminder-service	Maintains reminder schedules and triggers	No
notification-service	Creates and delivers notifications	No, for task state
analytics-ingestor	Derived analytics	No
admin-service	Operational repair and admin flows	Depends on the workflow

The initial deployment can run several logical services inside one binary. That is fine. Logical boundaries matter before physical deployment boundaries.

The API Gateway

The API gateway is the public HTTPS entry point.

It should authenticate access tokens at the application boundary, apply coarse rate limits, enforce request size limits, attach request IDs and trace context, route operation submissions, route sync reads, route auth/session requests, route attachment requests, normalize error responses, and enforce API versioning.

It should not assign workspace sequence numbers, directly mutate workspace tables, make final workspace write authorization decisions, publish durable sync events, or talk directly to Kafka for command commits.

The gateway is a boundary and router. The command service owns workspace writes.

The Command Service

The command service is the core backend service. It accepts operations and commits them as canonical workspace state.

The flow is:

This service must be robust in the best way:

No partial writes.
No direct Kafka publish.
No stale JWT role as final authorization.
No task mutation outside an operation.
No sequence assignment outside the transaction.

When this service is correct, the rest of the system has a stable foundation. When this service is wrong, the rest of the system inherits the damage, which cannot be fixed.

The Sync Query Service

The sync service is how clients recover.

The main endpoint is:

GET /v1/workspaces/{workspace_id}/sync?after_sequence=N

The client asks:

Give me committed events after the last sequence I have durably applied.

The sync service authenticates the user, checks read access, routes by workspace_id, reads ordered workspace_events, returns a page of events, and tells the client whether more events remain.

This is why WebSocket is not a correctness mechanism. The sync service is the durable recovery path.

If the client misses a hint, it can still call /sync. If the app restarts, it can still call /sync. If a request timed out, it can retry and then call /sync. If the client was offline for a week, it can call /sync, or install a snapshot and then call /sync.

The Realtime Gateway

Realtime is still useful. Users expect updates to appear quickly.

But realtime delivery should not be durable sync. The WebSocket gateway sends hints.

A hint means:

Something may have changed. Go ask /sync.

Not:

Here is the canonical state. Trust this forever.

That distinction removes a lot of complexity.

The WebSocket message can be dropped, duplicated, late, or delivered while the client is backgrounded. The client still converges through /sync.

Identity: Keycloak Plus A Go Session Service

Identity and app sessions are separate concerns. Since, I have limited myself to open-source alternatives rather than relying on any services, I picked Keycloak.

Keycloak owns identity provider behavior: login, OIDC, passkeys/WebAuthn, MFA, optional social login, email verification, password reset if passwords are enabled, and identity-provider sessions.

The Go auth-session service owns product session behavior:

• local app users linked to Keycloak subjects
• app sessions
• device sessions
• short-lived app access tokens
• opaque rotating refresh tokens
• refresh-token family tracking
• refresh-token reuse detection
• device logout
• account-wide session revocation
• app-level auth audit events

The split is:

Keycloak proves who the user is.
The Go session service manages app sessions.
The command service decides whether a workspace write is allowed.

A valid login does not mean a valid workspace write. Final write authorization happens inside the command transaction.

The current repository has the Go auth/session foundation and token verification in place. Keycloak/OIDC login and WebAuthn/passkey UX are still staged work.

PostgreSQL Shards

The target canonical data plane is PostgreSQL, sharded by workspace_id.

The workspace is the consistency boundary. Most operations affect one workspace: create task, complete task, move task, add comment, assign task, set due date, create reminder, add label, create project, and reorder section.

Those operations need one canonical sequence number per workspace, current membership checks, idempotency by (workspace_id, operation_id), current-state mutation, event append, outbox row, and efficient /sync by (workspace_id, sequence).

So the normal production write path is:

workspace_id
  -> workspace catalog
  -> shard_id
  -> shard writer endpoint
  -> PostgreSQL transaction

This avoids distributed transactions for normal task operations. Do not split one workspace's normal writes across multiple primaries. Do not make every task update a global consensus problem. Keep the product boundary and the data boundary aligned.

The current core demo keeps this as a single-cell PostgreSQL deployment with runtime catalog routing deferred.

The Data Model Shape

At a high level, the data model has four categories.

Category	Examples	Storage
Identity/session	users, devices, app sessions, refresh-token families	auth/session database in production; same PostgreSQL in the single-cell demo
Workspace canonical data	workspaces, members, projects, sections, tasks, comments, labels, reminders, events	PostgreSQL shard
Control-plane data	workspace catalog, resharding state, workflow state	catalog/control-plane database
Derived data	search indexes, reminders, notifications, analytics, cache	projection stores

Workspace-owned tables include workspace_id. That keeps constraints and queries shard-local.

Important primary keys look like:

tasks:                (workspace_id, task_id)
workspace_events:     (workspace_id, sequence)
processed_operations: (workspace_id, operation_id)
projects:             (workspace_id, project_id)
sections:             (workspace_id, section_id)
comments:             (workspace_id, comment_id)
attachments:          (workspace_id, attachment_id)

A simplified entity view:

The details will evolve. The important boundary is:

Workspace-owned state has workspace_id.
Normal operations stay inside one workspace.
One workspace lives on one canonical shard at a time.

#todo revise

The Outbox

The outbox is the bridge between PostgreSQL and Kafka.

The command transaction writes an outbox_events row together with the workspace event and current-state mutation. Then a separate outbox publisher reads committed outbox rows and publishes them to Kafka.

Why not publish to Kafka directly inside the request handler?

Because dual writes create bad failure cases:

Failure	Result
PostgreSQL commits, process crashes before Kafka publish	Canonical state changed, projections miss the event
Kafka publishes, PostgreSQL rolls back	Projections see an event for a write that did not happen
Kafka acknowledgement is lost	Handler may publish duplicates
Kafka is down	Command path fails even though canonical DB is healthy

The rule is:

Commit canonical state first.
Publish after commit.
Make consumers idempotent.

If Kafka is down, command writes can still commit. Outbox lag grows. When Kafka recovers, the publisher drains the backlog.

In the current core demo, command transactions write outbox rows, but the Kafka publisher is not implemented yet.

Kafka And Projections

Kafka distributes committed events. It does not decide whether an operation happened.

The first important topic is:

workspace-events.v1

Later topics can split work by projection:

search-events.v1
reminder-events.v1
notification-events.v1
activity-events.v1
analytics-events.v1
dead-letter-events.v1

A projection is a derived view of committed state: search documents, reminder schedules, notification records, activity feed rows, analytics facts, cache entries, and realtime hint fanout.

Projection lag is normal.

After a command commits:

1. /sync can read the committed event from PostgreSQL.
2. The outbox publisher later publishes to Kafka.
3. Projection workers consume the event.
4. Derived stores update.

During that window, search may be stale, reminders may be delayed, notifications may be delayed, analytics may be behind, caches may contain old data, and WebSocket hints may not have fired yet.

That is acceptable. Correctness comes from PostgreSQL and /sync.

Search

Search starts simple.

The initial search stack should be:

client local search index
+
PostgreSQL full-text search

OpenSearch can come later if needed. It should not be required for v1. If it is added later, it is still a projection.

Search is not allowed to become canonical state. If search is unavailable or stale, task writes still work, /sync still works, clients still converge, and search can be rebuilt.

Reminders And Notifications

Reminders and notifications are also projections.

The canonical facts live in workspace state: task due date, recurrence rule, reminder config, completion state, deletion state, assignee, comments, membership, and notification preferences.

The reminder service consumes committed events and maintains a schedule optimized for due reminders. The notification service turns reminder and activity events into user-visible records and channel sends.

Important rule:

Do not call notification providers inside the command transaction.

External providers fail, time out, retry, and have rate limits. None of that belongs in the canonical write path.

Object Storage

PostgreSQL should not store attachment file blobs.

Attachment metadata belongs in PostgreSQL because it is part of workspace state: attachment ID, workspace ID, task/comment relationship, uploader, object key, content type, size, created time, deleted/tombstoned state, and permission-relevant metadata.

The bytes belong in object storage:

Ceph / Rook / S3-compatible storage

Object storage is canonical for the file bytes. It is not canonical for task state. The relationship between a task and an attachment is stored in PostgreSQL metadata.

Attachments are not part of the current core demo slice.

Admin And Repair

Admin tools are dangerous if they bypass the model.

A support tool that directly edits tasks can break the operation log. A repair script that writes current-state rows without events can break sync. A worker that fixes projection data by writing canonical tables can create hidden drift.

So the rule is:

Admin tools should use normal operations or documented repair workflows.

Some repair workflows need special access. That is fine. They should be explicit, audited, and built around the canonical model. The operation log is not optional just because the caller is internal.

The Read Paths

The system has different read paths depending on what the caller needs.

Normal screens read current-state tables. Sync reads workspace_events. Search reads search indexes or PostgreSQL FTS. WebSocket provides hints. The attachment gateway authorizes and serves object access.

Different read models. One canonical write model.

The Write Paths

There should be one normal write path for workspace task state:

POST /v1/workspaces/{workspace_id}/operations

Everything user-visible goes through it: creating tasks, editing tasks, moving tasks, completing tasks, adding comments, assigning users, setting reminders, adding labels, changing due dates, and attaching file metadata.

Cross-workspace workflows can exist later, but they should submit ordinary operations to each workspace involved. They should not create a hidden multi-shard write path.

What This Architecture Does Not Do

Just as important as the design is what it refuses to do.

For v1, this system does not do:

• peer-to-peer sync
• LAN sync
• browser-to-browser sync
• mDNS
• STUN/TURN
• Cassandra as canonical task storage
• Kafka as source of truth
• OpenSearch as source of truth
• Valkey as durable state
• distributed transactions for normal workspace operations
• active-active multi-region writes
• claims of 10-million-user readiness without evidence

That is scope control. The architecture should be able to grow, but the first version should prove the core correctness loop before adding global complexity.

The Small Deployment

The local or early deployment does not need every production component.

The current core demo is smaller than the full architecture:

That is enough to prove local-first client behavior, operation submission, idempotency, the command transaction, /sync, outbox rows, and basic auth/session primitives.

As projection, attachment, and OIDC slices land, the small deployment can grow without changing the core model:

Do not prematurely split everything. Do not run ten services before the first correctness loop works.

The Production Shape

A production deployment grows the same model.

The production version adds HA PostgreSQL, multiple shards, Kafka replication, projection workers, network policies, backups and WAL archiving, observability, runbooks, controlled deployment, security hardening, and capacity planning.

But the central idea does not change:

operation -> command transaction -> workspace log -> sync -> convergence

How The Pieces Fit Together

The architecture can be summarized by path.

Write Path

client pending operation
  -> POST /operations
  -> API gateway
  -> workspace command service
  -> workspace catalog
  -> PostgreSQL shard transaction
  -> workspace_events + current state + processed_operations + outbox_events

Sync Path

client last_seen_sequence
  -> GET /sync?after_sequence=N
  -> sync query service
  -> workspace catalog
  -> PostgreSQL shard
  -> ordered events
  -> local apply

Projection Path

outbox_events
  -> outbox publisher
  -> Kafka
  -> projection workers
  -> search / reminders / notifications / analytics / cache / hints

Realtime Path

committed event
  -> Kafka
  -> realtime gateway
  -> WebSocket hint
  -> client calls /sync

Attachment Path

client upload/download
  -> attachment gateway
  -> metadata authorization
  -> PostgreSQL metadata
  -> object storage bytes

Auth Path

user login
  -> Keycloak
  -> Go session service
  -> app access token + rotating refresh token
  -> API gateway
  -> command-time workspace authorization

Different paths. One source-of-truth boundary.

The Architecture In One Table

Layer	Main responsibility	Not responsible for
Client	Local state, pending operations, optimistic UI, sync cursor	Canonical ordering
Edge	TLS, WAF, DDoS, public routing, caching	Task state or authorization
API gateway	Public API boundary, coarse auth, routing	Final write authorization
Auth session service	Product sessions and refresh tokens	Workspace write decisions
Command service	Canonical workspace writes	Kafka publishing or notifications
Sync service	Durable catch-up from operation log	Realtime delivery
PostgreSQL shards	Canonical workspace state	Derived search/analytics fanout
Outbox publisher	Publish committed events	Deciding whether writes happened
Kafka	Event distribution	Source of truth
Projections	Search, reminders, notifications, analytics	Canonical task state
Realtime gateway	Wake-up hints	Durable sync
Object storage	File bytes	Task relationships or permissions

The Design Bias

My bias in this project is data first.

Not services first. Not framework first. Not infrastructure first.

The important questions are:

What is canonical?
What is derived?
What can lag?
What must be atomic?
What can be rebuilt?
What needs ordering?
What needs idempotency?
What happens after timeout?
What happens after reconnect?
What happens after a missed hint?

Once those are answered, the architecture follows.

The product may look like a task app. The system is a local-first sync system with a canonical operation log. That is the thing to protect.

What Comes Next

The next post goes deeper into the operation log itself.

I will cover operation envelopes, operation IDs, request hashes, event envelopes, sequence numbers, idempotency records, rejected operations, conflict responses, and operation schema versioning.

The operation log is not just an implementation detail. It is the contract between offline clients, the server, sync, and every derived system.