What "saved" means

Goal

Know the three places a written byte can be, measure what each guarantee costs, and learn the trick real databases use to make the strongest one fast.

Three places a byte can be

When Python's file.write() returns, your bytes may not have gone anywhere yet:

From lab/stash/store.py:

    def commit(self) -> None:
        """Push written bytes as far towards the disk as the sync mode asks."""
        if self.sync in ("flush", "fsync"):
            self.file.flush()  # from this process to the operating system
        if self.sync == "fsync":
            os.fsync(self.file.fileno())  # from the operating system to the disk

Kill a real writer

stash plug --kill starts a real writer process. It writes nut:0, nut:1, nut:2 and so on as fast as it can, and after each set returns it prints ok N, a promise that write N is done. Then it gets killed with no warning, and a fresh process checks every promise:

uv run python -m stash plug --kill --rounds 5 --sync none

pulling the plug on a real writer, 5 times, sync=none
  round  1    98,569 writes acked    87 lost
  round  2   179,529 writes acked    73 lost
  round  3   171,642 writes acked   106 lost
  round  4   112,188 writes acked   172 lost
  round  5   140,969 writes acked   153 lost
total: 702,897 acked, 591 lost

With none, every crash loses the writes still sitting in Python's 8 KB buffer, even though the writer had promised they were done. Now with flush:

uv run python -m stash plug --kill --rounds 5 --sync flush

pulling the plug on a real writer, 5 times, sync=flush
  round  1    51,706 writes acked     0 lost
  round  2    95,550 writes acked     0 lost
  round  3    90,488 writes acked     0 lost
  round  4    62,085 writes acked     0 lost
  round  5    76,188 writes acked     0 lost
total: 376,017 acked, 0 lost

Zero lost out of 376,017. Once the operating system has the bytes, a dead process cannot take them back. A power cut still could, and you cannot pull your laptop's plug from Python. That is what fsync is for.

What it costs

uv run python -m stash bench

  sync=none                     381,428 writes/s
  sync=flush                    155,247 writes/s
  sync=fsync                      1,855 writes/s
  sync=fsync, batches of 100    146,567 writes/s

The benchmark lives in lab/stash/bench.py: it writes for two seconds per line and counts. These numbers are from one Windows laptop with an SSD; yours will differ, sometimes a lot. The shape will not: fsync on every write is about a hundred times slower, because each one waits for the disk.

The last line is the trick: group commit. Write a batch of records, then fsync once for all of them. Every write in the batch is safe when the batch returns, and the disk wait is shared. Postgres, MySQL and SQLite all do a version of this.

with db.batch():
    for n in range(100):
        db.set(f"nut:{n}", f"acorn {n}")

Check it works

• With --sync none, writes that were acked get lost. With --sync flush, none do.
• bench shows fsync far slower than flush, and batching wins most of it back.

Exercise

Run plug --kill --sync fsync. How many writes does it ack per round, compared with flush? Then decide: for a shopping cart, a bank transfer and a game's high score, which sync mode would you choose, and why?

More to try

• A batch makes callers wait for each other. Sketch a writer that collects writes for up to 2 ms, fsyncs once, then answers everyone. What is the worst-case wait?
• Look up synchronous in SQLite's documentation and match its settings to Stash's three modes.