05 Aug 2026
Database Crash Simulator
Databases advertise durability, but that guarantee is rarely tested against the narrow set of on-disk states a real crash can actually produce. I wanted something that enumerated those states instead of guessing at them, so this records a real syscall trace of a database under load, models what each filesystem journaling mode actually guarantees, and replays every legal crash state through the database's own recovery path.
I pointed it at redb, a production Rust embedded database, across 20 combinations of filesystem mode and workload shape. It ran 10,942 crash states in 21 minutes and found a genuine data-loss bug, where recovery permanently lost an intact file after one specific interrupted write. Checking redb's git history and all 75 released tags showed the maintainer had already found and fixed the same bug before I picked the target, so I found it independently rather than first. I packaged the disclosure the way it would have been written and left the credit where it belonged. The fix ships in redb 4.2.0. Getting to zero false positives meant triaging every one of the 9 flags: 7 were bugs in my own tool, 1 was the bug planted in the positive control, and 1 was the real redb bug.
- Role
- Solo
- Stack
- C, TypeScript on Bun, Rust, Linux
- Scale
- 10,942 crash states in 21 minutes, 77 tests
- Status
- Public on GitHub
Screens
Output from the committed campaign run, not a mock-up.
01 / 02 · The campaign
01
The campaign
Every filesystem and workload shape, 10,942 states in total. Three rows produced a finding, and all three are the same bug on a different filesystem.
02
The bug
The header persisted but the ftruncate that grew the file did not, so the file ends up smaller than the layout its own header describes. redb panics instead of recovering.
What it does
- Traces the real thing
- An LD_PRELOAD shim in C intercepts the database's actual writes. No source changes to the target and no instrumentation it could behave differently under.
- Four journaling modes
- ext4 ordered and journal, xfs and btrfs each guarantee different things. The models deliberately diverge instead of collapsing into one conservative approximation.
- Real recovery, not a simulation
- Each crash state is materialized as an actual filesystem image on a loopback device, and the database's own recovery code runs against it.
- One-command reproducer
- Findings are deduplicated by root-cause signature and packaged so anyone can reproduce one in about six seconds from a clean machine.
How it's built
- Trace
- C shim loaded with LD_PRELOAD. Every intercepted syscall gets submission and completion timestamps from one counter in a shared page, which gives a total order without a lock in the hot path.
- Enumeration
- TypeScript on Bun builds a persistence graph per journaling mode and enumerates the legal on-disk states at each crash point.
- Oracle
- Derived mechanically from the database's own documented durability contract rather than hand-written per workload, so it cannot be quietly tuned to pass.
- Targets
- redb as the real target, SQLite as a working control, and a deliberately broken key-value store as a positive control to prove the oracle catches what it should.