Two hands on one wheel

We ran two of something that was supposed to be one, because two felt safer.

It was an important record, the kind you do not want to lose, so we kept a second copy live — not a cold backup, but a second working copy, able to take writes, ready to step in. Insurance, we called it. For a while the two agreed, and we felt clever.

Two clocks on one wall agree for a while too. Then one drifts a minute, then five, and the room with two clocks no longer knows the time. It has two answers and no way to choose. We had built exactly that and called it redundancy.

The drift, when it came, was silent. Nothing crashed. Both copies kept running, kept taking writes, kept moving apart, each certain it held the real version. By the time we went looking there was no fact of the matter about which was right — two confident answers and a quiet corruption underneath that had been there a while, wearing the face of a healthy system.

The lesson was not keep fewer copies. Copies are fine; read from as many as you like. The lesson was about writing. One thing turns the wheel. Everyone else keeps their hands in their lap and reads. A single writer can fail, and when it does you see it — the clock stopped, go fix the clock. Two writers do not fail. They disagree, which is worse, because disagreement does not look like failure until the day you audit it and find it has been lying to you in proportion to how much you trusted it.

We took the second hand off the wheel. We should not have put it there. A man with one watch always knows the time.