OpenAI's bug that hid inside a 100-picosecond window

How OpenAI stopped debugging like a doctor and started thinking like an epidemiologist and caught a race condition in the process.

A crash that shouldn’t be possible.

A normal C++ function inside OpenAI’s Rockset service runs, finishes, and returns to garbage. Sometimes the saved return address on the stack is NULL. Sometimes the stack pointer is off by 8 bytes, as if %rsp quietly decremented mid-execution. No inline assembly. No longjmp. No setcontext. None of the things allowed to move the stack pointer like that.

Every hypothesis had strong evidence against it. Which is another way of saying the bug looked impossible. It turned out to be two bugs wearing a trench coat.

First, they debugged like doctors

Rockset, the data system OpenAI acquired in 2024 and now runs behind ChatGPT search and connectors, keeps its execution layer in C++ for the performance. That also means it inherits C++’s favorite party trick: memory corruption that surfaces far from its cause.

The team did what most of us do. Grabbed a few core dumps, stared hard, formed hypotheses, knocked them down one by one. Compiler bug? Linker issue? Kernel bug in signal delivery? They went deep, spending days reconstructing the pre-crash history of a single misaligned-%rsp crash from register and stack contents.

They also made one early call that poisoned everything after it. They ruled out hardware, because the crashes showed up across multiple regions and hardware types. Reasonable. Also wrong. And because they assumed all the crashes shared one root cause, every counterexample from one bug muddied the investigation of the other.

This is the same trap Pinterest fell into when a 50% ML training slowdown sent them chasing PyTorch before the real cause turned out to be page tables. One patient, lots of tests, wrong frame.

Then they debugged like epidemiologists

The pivot is the whole story. Instead of diagnosing one patient, look at the whole population and ask what a single case can never show you. Did the crashes start at a specific release? Correlate with one CPU SKU, one region, one kernel? Are there two diseases hiding inside one set of symptoms?

Log searches had failed them. Stack-corruption crashes produce corrupted stack traces, so the logs were full of false positives and false negatives. So they built a real pipeline. A script pulled a prefix of every core file, extracted registers, filtered known false positives, and auto-labeled each crash as return-to-null, misaligned-stack, or other. Then they ran it across every Rockset core dump from the past year.

That was the turning point. With clean population data, the correlations jumped out, and the “one weird bug” split cleanly into two.

Return-to-null crashes: spread across many clusters and regions, rising in frequency, no crisp start date.

Misaligned-stack crashes: all from one region, a clear start date, never on long-lived nodes. The fingerprint of a single bad physical machine poisoning whichever VM landed on it.

Bug #1: the host that couldn’t do math

With a clean list of nodes and timestamps, the misaligned-stack crashes traced back to one Azure host whose CPU was silently corrupting register state. They never reproduced it in a controlled test. But denylisting the host made those crashes vanish. They also hardened the fatal signal handler to log register state, so next time they catch it from logs alone, no core dump required.

Pulling the hardware crashes out of the data set is what finally made bug #2 legible.

Bug #2: an 18-year-old race in GNU libunwind

With the hardware noise gone, the remaining return-to-null crashes revealed a pattern the team had explicitly ruled out earlier. They were all happening during C++ exception unwinding.

Exception unwinding isn’t a normal return. It’s a dynamic control transfer, closer to a fiber switch. GNU libunwind synthesizes a ucontext_t on the stack, then hands it to an assembly routine, _Ux86_64_setcontext, to restore registers and jump.

Look at the last handful of instructions and the bug is right there:

mov  UC_MCONTEXT_GREGS_RSP(%rdi),%rsp    # race window opens here
mov  UC_MCONTEXT_GREGS_RIP(%rdi),%rcx
push %rcx
...
retq

The instant %rsp is updated, the ucontext_t that %rdi still points to is no longer part of the active stack or red zone. So the kernel considers that memory fair game. If a signal arrives right then, the kernel builds its signal frame over that memory and clobbers the saved instruction pointer before the next instruction reads it. Restored %rip becomes NULL. Crash on transfer.

The race window is one instruction wide. Roughly 100 picoseconds on a modern out-of-order CPU.

So how does a 100-picosecond window crash a fleet a dozen times a day? Fermi estimation. Rockset throws exceptions as backpressure at ~10⁴/sec on an overloaded host, and its coarse_thread_cputime_clock fires SIGUSR2 every few milliseconds of CPU time. Multiply it out and you get a mean time between failures of hours per host. At fleet scale, that’s your dozen daily crashes.

The bug has existed since the first x86_64 version of libunwind to support exception unwinding. Over 18 years. It only surfaced now because Rockset is extreme on all three variables that matter (exception rate × signal rate × signal-handler stack usage), and a recent change that made the SIGUSR2 handler use more stack pushed it over the threshold.

This is exactly the shape of the Cloudflare hyper race condition class of bug. A window measured in instructions, dormant for years, until one workload makes the improbable routine.

The fix: switch to libgcc’s unwinder, a win anyway because it means less lock contention at scale, plus an upstreamed reproducer and patch to GNU libunwind.

What to Take Back to Your System

The clever part of this story, reading assembly, reasoning about the System V ABI red zone, DWARF unwind metadata, is not the lesson. None of it helped until they built a high-quality data set over the whole population of crashes.

When a bug seems impossible, suspect your data before you suspect physics. Contaminated evidence makes every smart deduction fight itself. The logs lied here because the corruption corrupted the logs.

Check whether you’re staring at two problems, not one. A single assumed root cause is what let two unrelated bugs blend into one unsolvable narrative. The Heroku foreign key incident is a reminder that the immediate fix can even make things worse when you’ve misread the shape of the problem.

Debug like an epidemiologist when one patient won’t talk. Population-level correlation across release, region, SKU, and kernel finds structure a single core dump never will. Build the pipeline before you build the deduction.

---

Read the original: Core dump epidemiology: fixing an 18-year-old bug, OpenAI Engineering, by Nathan Bronson (Jun 30, 2026)