Can you prove software has no bugs, and who checks the proof?

An AI wrote proofs so long nobody will ever read them. Weirdly, that might be fine.

Think about how we normally hunt bugs. We test. A lot. One standard technique, called fuzzing, throws millions of weird, half-random inputs at a program and waits for it to crash. It is brilliant and it is brute force. And no matter how long it runs, it can only say "I haven't found one yet."

A proof says something different. It says "for every input in this range, this cannot happen." Not "I didn't see it." Cannot.

What just happened

Anthropic took existing fuzz tests and turned them into statements to prove: this program never crashes on any input up to 64 megabytes. Then it set Claude loose on the compiled machine code of 123 open-source libraries. Over a few weeks, Claude patched the bugs it hit and proved the no-crash property for 239 entry points. We are talking about roughly 180 million lines of proof, with a typical proof running to around 600,000 lines.

And along the way it found bugs that years of testing had missed. One hid in a decompressor and only showed up on inputs of about 20 megabytes with one rare layout. Another needed a 64 megabyte file and over 2 gigabytes of memory. A fuzzer capped at small inputs would basically never get there. On a widely used PDF reader, it found as many security bugs in one week as are usually reported in a year.

Wait. A proof of 180 million lines that no human will read. Why on earth should we trust it?

Here is the trick. You do not trust the proof. You trust a tiny checker that confirms it. A small, well-understood proof-checking kernel verifies every step mechanically, and the only things a human has to trust are the formal models of the processor and the operating system, the exact statements being proved, and the checker itself. You swap a mountain of code you cannot read for a small pile you can.

Where it still wobbles

  • "Formally verified" does not mean "bug-free." If the statement you proved does not match reality, you have a gorgeous proof of the wrong thing.
  • These proofs cover single-threaded programs on one processor design with one operating system model. Most people run something different.
  • There is a cheating risk: an AI could rewrite a program so it trivially never crashes. The team penalised such patches and audited them.

So who verifies what the AI shows us? Increasingly, a small machine we can inspect, plus humans reading the question being asked rather than the answer. The risk moves from "is the proof right?" to "did we ask the right thing?" Also worth saying: this is a company reporting on its own work, with outside checking only just starting. But if it holds up, it is the first real answer to a scary trend, AI finding flaws faster than humans can patch them. Maybe the same AI can also prove where they are not.