Tailscale Tracks Down 16-Year-Old SQLite Corruption Bug

A mysterious database flaw triggered 19 outages at Tailscale before engineers caught a 16-year-old bug in SQLite.

Tailscale runs its control plane on SQLite, using a single Go process per internal shard. Starting last August, backup checks flagged corrupt database files, leading to 19 separate downtime incidents over six months. With no easy way to reproduce the glitch synthetically, Tailscale worked with Antithesis and deployed live telemetry to find the culprit: a 16-year-old flaw in SQLite's write-ahead log reset logic.

Why it matters: SQLite is the gold standard for simple, reliable software. This bug shows that even widely trusted code can hide rare concurrency bugs for decades until high-scale workloads push it to the edge.

Here's the gist: Tailscale lost no private encryption keys or network traffic, but repairing the corrupt shards caused recurring downtime. The team has since helped fix the bug upstream.

Even the most dependable tech has a few ancient ghosts in the codebase.