Pull the plug

Goal

Recover from a torn write, tell a torn write apart from corruption, and see why Stash keeps your data at every crash point.

Tear a write

Make a fresh log in the shell, as in chapter 02 (set acorn:1 under the oak, set acorn:2 behind the shed, set acorn:1 in the flower pot, del acorn:2). Then cut 7 bytes off the end, as if the plug came out during the delete:

uv run python -m stash tear data/nuts.stash 7
uv run python -m stash inspect

  offset  size  key         value
       0    32  acorn:1     under the oak
      32    34  acorn:2     behind the shed
      66    36  acorn:1     in the flower pot
torn tail: 12 bytes of a write that never finished
3 records, 114 bytes

The reader found three whole records, then 12 bytes that claim to be a record but end too soon. Open the store:

uv run python -m stash shell

recovered: cut 12 bytes of a write that never finished
stash on .../lab/data/nuts.stash: 2 keys. Try: set acorn:1 under the oak

Two keys: acorn:2 is back. The delete never finished, so it never happened. That is the promise: every write is all or nothing.

Stash also truncates the torn bytes before writing anything new. Otherwise the next record would land after the garbage, and a later open would stop reading at the garbage and never see it.

How the reader knows

From lab/stash/record.py, the heart of recovery:

        if end > len(data):
            break  # cut short: the write in flight when the plug came out
        if zlib.crc32(data[offset + 4 : end]) != crc:
            # A bad last record is a torn write. A bad record with more after it is rot.
            return Log(records, offset, damaged=end < len(data))

Two checks. If the lengths point past the end of the file, the record was cut short. If the lengths fit but the checksum does not match, the bytes are wrong. Disks do not always write in order, so after a power cut the last record can have the right length and the wrong bytes. The checksum catches that.

This is why stash plug reported 100% in chapter 01. It cuts the log at every byte, runs this same reader, and compares the result with the writes that had finished.

A cosmic ray

A bad record in the middle of the log is different. That is not a crash; that is rot: a failing disk, a bad cable, a real cosmic ray. Simulate one:

uv run python -m stash cosmic-ray data/nuts.stash
uv run python -m stash stats data/nuts.stash

a cosmic ray flipped bit 4 of byte 13, inside acorn:1
refusing to open data/nuts.stash: the record at byte 0 fails its checksum

Your bit and byte will differ: the ray is random. Stash refuses to open, on purpose. Cutting the log at the bad record would silently throw away every good record after it. Real databases do the same, and then you restore from a backup or a replica.

Check it works

• After tear, the shell reports the cut, and acorn:2 is back.
• After cosmic-ray, stats refuses to open the file. Delete data/nuts.stash to start again.

Exercise

The cosmic ray never hits the last record, because cmd_cosmic_ray skips it. Change that and flip a bit in the last record. What does Stash do? Is it right to treat that as a torn write? Write down the argument for and against.

More to try

• Run uv run python -m stash plug --writes 300. Do both numbers hold?
• CRC32 is fast but not cryptographic. Who could fool it, and does that matter for this job?