When We Choose Consistency Over Availability

WatchDock acts on things it doesn't own: your servers, your GitHub repository, your payment provider. Each of those actions has its state in two places, and this is how we decide when to refuse an operation rather than tell you something that isn't true.

Watchdock Engineering·9 min read

Most of what WatchDock does is read. It collects metrics, tails logs, checks URLs and draws the results on a dashboard. A growing part of it acts, though. From the browser you can restart a container or prune Docker on a server, approve an AI-written fix and get a pull request in your repository, or upgrade your plan. Every one of those actions has its state in two places: our database, and the thing being acted on, which is a machine on someone else's network, a repository on GitHub or a charge at a payment provider. A network we don't control sits between the two copies, and they disagree more often than you'd expect. This post is about how we decide which copy a user gets to act on, and it starts with a Cancel button I added on a Thursday.

What CAP actually says

People reach for the CAP theorem loosely, so it's worth being precise. Brewer's conjecture, and the proof Gilbert and Lynch published in 2002, is about a replicated data store during a network partition. If the replicas can't talk to each other, each operation either waits or fails until they can, which is consistency in the strict linearizable sense, or it answers from whatever the replica it reached happens to know, which is availability. Partition tolerance isn't a third option you get to decline, because networks partition whether or not your architecture diagram admits it. Brewer said as much in his 2012 follow-up, where he called "pick two of three" misleading and described the real decision as what you do during a partition and how you recover afterwards.

Our backend is a single Postgres primary, which on its own is as boring as consistency stories get. The interesting replicas are the ones outside it. A server's agent polls us over the public internet and is routinely unreachable for minutes at a time, because it's a laptop VM, or behind a flaky NAT, or rebooting. GitHub's copy of a file changes whenever somebody pushes. A card can be declined after the checkout page has closed. So the question CAP asks is one we answer every day: when our database and the other copy disagree, what do we let the user do?

A Cancel button with a race behind it

WatchDock lets you queue actions against a server from the dashboard. The agent on the box polls for them every five seconds, runs them and posts the result back. If the agent is offline the command waits, sometimes for hours, which is why it needed a Cancel button. A restart you queued on Monday while a box was down shouldn't fire on Wednesday when somebody finally brings it back.

The poll endpoint was written months ago and did what you'd sketch on a whiteboard. It selected this server's pending command ids, marked them running and returned them. Names are changed for illustration, but the shape was this:

python
1ids = list(
2 Command.objects
3 .filter(server=server, status="PENDING")
4 .values_list("id", flat=True)
5)
6Command.objects.filter(id__in=ids).update(
7 status="RUNNING", started_at=timezone.now()
8)
9return Command.objects.filter(id__in=ids)

Before cancel existed, nothing else moved a command out of PENDING, so the gap between the select and the update didn't matter. A second writer changes that. If a user cancels in the milliseconds between them, the cancel commits, the poll's update overwrites CANCELLED with RUNNING because it only filters on id, and the agent receives a command the dashboard has just told the user is cancelled. It's a textbook lost update. The window is small, but small isn't a property I want to rely on when the command is docker system prune on someone's production box.

There was a second way in. If a customer bakes a VM image with the agent installed and forgets to clear its identity, every clone reports as the same server. Two agents polling one queue would both select the same ids and both be handed the same command, so one click in the dashboard would prune two machines.

Every transition is conditional

The fix is to make every state change a compare-and-set, so a row only moves if it's still in the state the writer expects:

python
1claimed_at = timezone.now()
2Command.objects.filter(id__in=ids, status="PENDING").update(
3 status="RUNNING", started_at=claimed_at
4)
5return Command.objects.filter(
6 id__in=ids, status="RUNNING", started_at=claimed_at
7)

This is safe under Postgres's default READ COMMITTED isolation without a SELECT ... FOR UPDATE. When an UPDATE finds a row that a concurrent transaction has modified, it waits for that transaction to finish and then re-evaluates its WHERE clause against the newest version of the row. If the cancel commits first, the poll's update sees CANCELLED and skips the row. The started_at marker exists because Django's .update() returns a row count rather than the rows it touched, where raw SQL would just use UPDATE ... RETURNING id. Several consumers draining one queue would call for FOR UPDATE SKIP LOCKED, the standard Postgres job-queue pattern, but one agent per server doesn't need it, and the conditional update already handles the cloned VM: the second agent's update matches zero rows and it gets an empty list.

Cancel is the same move pointed the other way:

python
1cancelled = Command.objects.filter(id=pk, status="PENDING").update(
2 status="CANCELLED"
3)
4if not cancelled:
5 return Response(
6 {"error": "The agent already picked this command up."},
7 status=409,
8 )

That 409 is the CAP decision. The friendly alternative is to accept every cancel, show a grey "Cancelled" pill and sort it out when the agent reports back. That system is available, because it always answers, and it's lying whenever the agent has already started, because a prune halfway through deleting images doesn't stop when a row in our database changes. So once the agent has claimed a command we refuse the cancel, say why, and refresh the list so the user sees it running.

Restarts are a partition you cause yourself

Restarting or upgrading the agent from the dashboard adds a twist, because the command's last act is to cut the link it would report back on. If the agent restarts and then tries to post "done", there's no process left to post it, and the row sits in RUNNING until someone gets suspicious. So the agent asks systemd to run the restart as a separate transient unit five seconds later, reports success for the launch, and only then goes down.

That fixes the stuck row but changes what "Done" means, and I'd rather say so than pretend. Done on an agent update means the upgrade was launched. It can't mean the new version is running, because nobody on our side can know that until the partition heals. The real confirmation is the next heartbeat carrying the new version number, which is what the agents card shows. We also deduplicate while a command is in flight, so a second click on Restart, or a browser retrying a request it thinks failed, returns the command already queued instead of stacking another behind it. The claim covers the agent's side: once the restart has moved to RUNNING, the agent coming back up can't pick it up and restart itself again.

The repository is a replica too

Our AI fix agent proposes a code change for an error and waits for someone to approve it before opening a pull request. Approve and Reject use the same compare-and-set as cancel. A proposal only leaves review if it's still in review, so two teammates clicking different buttons at the same moment get one result and one conflict, and never a pull request for a fix somebody rejected.

The more interesting part is the commit. When the agent writes a fix it records the version hash of the file it read, and the commit goes to GitHub with that hash attached. If anyone has pushed to that file since, GitHub refuses the write, and we mark the proposal failed and ask the user to regenerate it. Retrying on top of the newer file would succeed far more often, and that's exactly what worries me about it. A patch written against code that no longer exists is a guess, and committing a guess to somebody's repository is worse than asking them to click once more.

Duplicates are a consistency bug

Creating a GitHub issue from an error has a smaller version of the cancel race. Two workers can pick up the same error at nearly the same moment, for instance when an ingest is redelivered, and each sees that no issue exists yet, calls GitHub and opens its own. The task now locks the error's row and holds the lock through the GitHub call, which takes about two seconds. Holding a database lock across a network call is something I'd normally flag in review, because every other writer to that row ends up waiting on someone else's API. Here the only other writer is the duplicate we're trying to stop, so the waiting is the point. The second task gets the row after the first has saved the issue link, sees it, and does nothing.

Incident numbers work the same way. Each workspace numbers its incidents from 1, and when two monitors fail in the same second we lock the workspace's counter while handing out the next number. Incident numbers get read out on calls and pasted into postmortems, and two outages both called #41 would cause confusion at the worst possible time. A few milliseconds of waiting is cheap by comparison.

Billing waits for the money

Starting a checkout records how you intend to pay, and that's all it does. The new plan only takes effect when the payment provider confirms the charge. Switching plans the moment you click would feel faster, and it would be wrong every time a card is declined after the checkout page has closed, which leaves us either taking features away from someone who thought they'd paid or giving them to someone who hadn't.

Where we chose availability instead

None of this means WatchDock is consistent everywhere, and a monitoring product shouldn't be. When the dashboard can't reach our API, the servers page keeps showing the last data it had, with a banner that says "You're looking at data from 35s ago" and a retry every minute. An SRE in the middle of an incident is better served by a slightly stale fleet view than by a spinner, as long as the staleness is spelled out on screen. Our plan quota check counts a month of metric reports and is cached for sixty seconds, because a quota a minute out of date costs nothing and the uncached count cost 76 ms on every report. The agent config shown in the server drawer is built by the same function that serves the agent, so the two can't drift, but the agent only picks up a change on its next fetch, and the panel says it shows what the agent pulls rather than what it's running.

The rule we ended up with

It's less elegant than a theorem but easier to apply. Reads that only inform a person can be available and stale, provided they say how stale. Writes that change something on a customer's machine, in their repository or on their bill have to be consistent. In practice that means every state transition is conditional on the state it expects, duplicates are prevented with a lock rather than cleaned up afterwards, the API returns a conflict when it can't make the write true, and the UI says "launched" when it can't yet say "finished". PACELC, Daniel Abadi's extension of CAP, describes the everyday half of this better than CAP does: even without a partition you're trading latency against consistency, and the sixty-second quota cache and the two-second lock on an issue row are that trade made in opposite directions. The cancel race never reached production, because the button and the fix shipped in the same pull request. A cancel button that's right almost every time doesn't work, because the one time it's wrong is the one somebody remembers.

Keep reading
One email when we publish. Nothing else.Postmortems and engineering notes, roughly twice a month.