Core dump epidemiology: fixing an 18-year-old bug

By Nathan Bronson, Member of Technical Staff

OpenAI’s models and agents increasingly rely on scalable data infrastructure in order to search for relevant data at inference time: when the models are thinking about your question. Some of these services are written in C++, whose low-level control of the system lets us maximize performance and minimize memory usage. Those efficiency benefits are important as we scale, but C++’s lack of memory safety means that bugs can cause crashes by writing to incorrect or non-existent memory addresses.

A few months ago we observed some crashes from inside the Rockset service, a bespoke part of our ChatGPT data infrastructure which is key to many data plugins and to searching over conversations. In each of these crashes, a normal C++ function seemed to finish and then return to a bogus address, causing the kernel to stop the program because the instruction pointer no longer pointed at code. Sometimes the return address slot in the stack frame was NULL. Sometimes the stack pointer CPU register itself seemed to be off by 8 bytes, as if %rsp had somehow been decremented in the middle of normal execution. In both cases the crash happened on return.

These are not normal failure modes for application code. A stray write that lands only on a saved return address is possible, but extremely unlikely. A bug that misaligns %rsp by 8 without involving inline assembly, setcontext, or longjmp (none of which we use) is even stranger, because compiled code only adjusts that register directly in the function prologue and epilogue. Every hypothesis we (or ChatGPT) could think of had strong evidence against it, so the bug seemed impossible.

What we assumed was one problem eventually turned out to be two unrelated bugs, coincidentally discovered at the same time. First, silent hardware corruption on one Azure host, where the CPU just didn’t do math correctly. Second, an 18-year-old race condition in GNU libunwind, an unnoticed bug in a widely used open source library.

This post is the story of how we identified and fixed seemingly inexplicable crashes by thinking like an epidemiologist and building a high-quality data set about the entire population of crashes.

First, let’s go deeper on Rockset. It’s a cloud-native data system for search and real-time analytics that we use for many internal use cases at OpenAI, such as sync connectors (Rockset was acquired by OpenAI in 2024). Streaming updates are used to maintain an up-to-date index of a workspace’s knowledge base so that ChatGPT can search for relevant information when answering questions or performing actions.

Rockset’s execution layer is written in C++. The C++ language provides low-level access to the CPU, which is good for performance and efficiency, but it means that application bugs can lead to invalid memory accesses and segfaults. To help track these down we use folly’s fatal signal handler to log a stack trace when a crash happens, and we upload the corresponding core dumps (a snapshot of the state of the program when it crashed) to Azure blob storage for later analysis. All of Rockset’s query processing leaves are replicated, which minimizes the client impact of a crash. However, each segfault corresponds to a bug that needs to be fixed to meet our reliability and quality goals.

Our initial approach was to treat these cores like a conventional debugging problem: inspect a few core dumps very closely, form hypotheses, and rule them out one by one.

Most of the crashes occurred in a method called DocumentTree::updateDocument. In these crashes it appeared that updateDocument had called some unknown function X, the stack had become corrupted while X was active, then X had returned to an address that wasn’t executable code. In some cases X’s just-popped frame looked valid except that its saved return address was NULL. In other cases the stack pointer itself looked wrong, but the next valid frame still seemed to be updateDocument.

We didn’t know when the stack was getting corrupted, which left a huge search space. updateDocument is a large method that undergoes a lot of inlining, so the number of candidates for X was overwhelming.

Was this a bug in our C++ code? A compiler or linkage issue? A problem in one of our runtime libraries? A Linux kernel bug around signal delivery or context switching? Something even rarer? If this was a stray write, why wasn’t it caught by our ASAN staging environment?

We tried to use our application-level logs to identify all occurrences of the problem, but stack-corruption bugs are hard to classify from logs alone because the logged stack traces are themselves corrupted or missing. We weren’t able to construct a log query that didn’t have both false positives and false negatives. We manually inspected more cores and found some additional examples, but that process was too labor-intensive to give us a trustworthy data set.

At this stage of the investigation, we (incorrectly) ruled out a hardware bug, because we saw crashes across multiple regions and multiple hardware types, so we were still looking for software-only causes. For a few days, we went super-deep on a single misaligned-%rsp crash, reconstructing the pre-crash history using stack and register contents. This produced some possible clues, but because we didn’t let go of our initial conclusions that all of the bugs had the same cause, this didn’t get us unstuck.

Before getting to the turning point of our investigation, it’s important to explain what kind of information we were extracting from the core files.

Rockset is compiled with -fno-omit-frame-pointer, so the active stack frame is always reachable through %rbp, and callers form a linked list of frame pointers.

On Linux x86_64, the AMD64 System V ABI also reserves 128 bytes below %rsp as the red zone. That region is available to userspace code and, importantly, the kernel promises not to clobber it when it delivers a signal, as part of the ABI contract.

The red zone was central to our debugging of a post-return crash, because it preserves some information from before the return. When a SIGSEGV is triggered, folly’s fatal signal handler runs on the crashing thread’s stack. Stack frames that are no longer active (because their function has returned) will get clobbered by the signal handler, except for the last 128 bytes. That’s why we can say things like “X’s just-popped stack frame looked valid, except for a NULL return address.” The red zone preserves some of the inactive frames, or sometimes just the tail of one inactive frame.

We found one misaligned-stack crash in which all of the functions involved were very small. That let us see that %rsp had become misaligned during execution of a relatively simple function, and that more calls had succeeded afterward. The program only crashed when the active function finally tried to return. None of those code paths used exceptions, inline assembly, setcontext, or longjmp, so if the stack pointer truly changed in the way the core suggested, no plausible bug in userspace code explained the issue.

That pushed us toward the kernel.

Rockset uses signals more aggressively than most programs. Query execution is broken into many lightweight tasks that exchange data. This is important for handling high-QPS workloads efficiently, but it makes per-query CPU accounting awkward as work for many queries is multiplexed onto the same thread pool.

Our solution is something we call coarse_thread_cputime_clock, which approximates clock_gettime(CLOCK_THREAD_CPUTIME_ID, ...) cheaply enough to sample at every task boundary. The timer_create API can be used to schedule a periodic signal delivery based on several notions of the passage of time, including the accumulation of CPU time. We schedule a signal (SIGUSR2) to be delivered every few milliseconds of CPU time, at which point the signal handler updates a thread-local value. Even though many tasks don’t see the coarse clock advance while they are executing, summing all of the deltas produces an unbiased estimate of the actual CPU time for a query.

Because we deliver signals so often, a rare kernel bug around context switching or signal delivery seemed plausible. We spent time reading bug reports, kernel source code, and the Azure-specific kernel patches. We tried stress tests. We weren’t able to find anything that seemed related.

At that point we decided to step back and try a different approach.

There are two broad ways to debug a problem like this.

One is to act like a doctor of sorts: focus on one patient, run lots of tests, and try to diagnose a single case from detailed evidence.

The other is to act more like an epidemiologist: look at the entire population and ask whether there are patterns that a single case cannot reveal. Did the bug start at a specific release? Does it correlate with one hardware SKU (the specific CPU and server model), one region, or one kernel version? Are there multiple distinct clusters hiding inside what looks like one syndrome?

We had mostly been in doctor mode. The key shift was deciding that we needed to gather high-quality population data.

Our previous attempts to automatically find all of the instances of the problem failed because we were trying to use text searches over the logs. The core dumps themselves have a lot more information, but looking at them manually didn’t scale. We decided to invest the effort to build a pipeline that could automatically analyze the core dumps.

We had ChatGPT write a script that downloaded a prefix of each core file, extracted the registers, filtered known false positives using the logs, and automatically labeled the crash as return-to-null, misaligned-stack, or other. Then we ran that script in parallel over every production Rockset core dump from the previous year.

This was the turning point.

Once we had a clean data set, correlations appeared immediately. What we had been treating as one weird bug was actually two separate crash populations.

The return-to-null cores were spread across many clusters and geographic regions. Their frequency had increased recently, but there was no crisp start date and no clean infrastructure boundary.

The misaligned-stack crashes looked completely different. They all came from one region, had a clear start date, and never happened on nodes that had been running for a long time. Even though they involved multiple Azure VMs (virtual machines hosted in the cloud), the pattern looked like one physical machine with bad hardware causing problems for whichever VM happened to land on it.

That was the moment we realized we had been mentally conflating two bugs. Because we had been mixing counterexamples from both bugs, we couldn’t find a single coherent explanation.

Annagoo haysanna liis nadiif ah oo ah mashiinnada adeegga Kubernetes iyo calaamadaha waqtiga, waxaan awoodnay inaan burburradii rasada qalloocan dib ugu raadraacno hal martigeliye jireed, kaas oo ay fududayd in lagu daro liiska la diido.

Ma awoodin inaan si xakameysan ugu celinno xumaanta diiwaanka ee martigeliyahaas, xitaa kadib dhowr toddobaad oo tijaabin walbahaar ah. Markii host-kii dhibaatada lahaa laga saaray adeegga, si kastaba, burburadii rasada aan is-waafaqsaneyn way baaba’een.

Ka saarista martigeliyaha xun ma aha xal joogto ah, macnaha ah inaysan ka hortagayn in isla dhibaatadu mar kale dhacdo. Si kastaba ha ahaatee, waxaan beddeli karnaa barnaamijka si haddii arrin la mid ahi mar kale soo noqoto, si fudud loo ogaado loona maareeyo. Waxaan hagaajinnay maareeyahayaga calaamadaha halista ah si uu ugu daro xaaladda diiwaannada, taasoo noo suurtagelinaysa inaan soo noqoshada dhibaatada ka ogaanno diiwaannada oo keliya (iyadoo aan loo baahnayn burbur barnaamijyo). Waxaan beddelnay lakabka xakamaynta si mashiinnada dalwadda ah inta badan dib loogu isticmaalo halkii dib loo warshadayn lahaa, taasoo ka dhigaysa ogaanshaha mashiinka adeegga ee xun mid aad uga fudud heerkeenna kaabayaasha. Waxaan sidoo kale cusboonaysiinnay hagayaashayada hawlgalka (iyo qaababka fikirka ee kooxdeennu leedahay) si ay ugu daraan suurtagalnimadan.

Markii burburradii martigeliyaha xun gaar loo saaray, burburada barnaamijyada ee haray ee return-to-null waxay noqdeen kuwo aad uga fudud in laga fekero. Hore waxaan meesha uga saarnay furfuridda reebitaannada, sababtoo ah waxaan moodaynay inaan hayno tusaalooyin ka hor imaanaya: burburro ka dhacay waddooyin koodh oo reebitaanno si dhab ah aan loo isticmaalin. Laakiin tusaalooyinkaas ka hor imaanayay dhammaantood waxay ka yimaadeen kooxda musuqmaasuqa qalabka.

Markii aan dib ugu noqonnay burburkii xusuusta ee haray annagoo taas maskaxda ku hayna, waxaan ogaanay in gunaanadkaasi uu gebi ahaanba ka soo horjeeday xaqiiqda: burburradu dhammaantood waxay dhacayeen intii lagu jiray furfuridda reebitaannada.

Marka C++ ay tuurto reebitaan, waqtiga socodsiintu waa inuu ogaadaa xannibka qabashada ee ay tahay inuu reebitaankaas helo iyo baabi’iyeyaasha ama maareeyayaasha nadiifinta ee ay tahay inay socodsamaan inta jidka lagu jiro. Isku-dubbariduhu wuxuu soo saaraa xog-sifeedkan, balse is-waafajinta dhabta ahi waxay si firfircoon u dhacdaa waqtiga socodsiinta.

Furfuridda reebitaannada dhab ahaantii ma fuliso shaqada yeerisa throw, balse waxaa fuliya shaqooyin caawiye ah oo uu waco koodhka la isku-dubbariday ee ka dhasha. Hab-raacyadaas waqtiga socodsiintu waxay baaraan rasada, waxay soo qaataan xog-sifeed ku saabsan shaqooyinka laga helo rasada, waxay si firfircoon u raadiyaan maareeyayaasha nadiifinta iyo xannibyada qabashada, kadibna waxay xakamaynta u wareejiyaan mid ka mid ah goobahaas. Wareejinta xakamayntu waxay ka mid tahay furfuridda dhammaan qaab-dhismeedyada rasada ee u dhexeeya (oo ay ku jiraan kuwa shaqooyinka caawiyeyaasha ah).

Hawlgal ahaan, tani aad bay uga dhowdahay longjmp ama fiber switch marka loo eego wicitaan iyo soo noqosho caadi ah. Callee save registers waa in la soo celiyaa, sidoo kale diiwaannada qaabka rasada ee %rbp iyo %rsp.

Binary-gayagu wuxuu ku xirmaa laba maktabadood oo ay ku jiraan hirgelinnada hawlaha sameeya exception unwinding-ka C++: libgcc iyo GNU libunwind. Qeexitaannada GNU libunwind ayaa ahaa kuwa uu doortay dynamic linker-ku. Taasi way naga yaabisay; waxaan filaynay in hirgelinta libgcc ay guulaysan doonto sababta oo ah xeerarka noocaynta summadaha; hase yeeshee, baaritaanka faylasha fulinta ee socda wuxuu muujiyay in arrintu sidaas ahayn.

Halkaas, mala-awaalkayagii shaqadu wuu is beddelay, maadaama aan dabcinay mala-awaal kale oo aan samaynay markii aan moodnay in hal cillad oo keliya jirto.

Waxaa laga yaabaa inaanan aragayn hawl caadi ah oo NULL ku noqonaysa. Waxaa laga yaabaa inaan aragayn wareejin unwind ah—si wax ku ool ah soo celin diiwaanno oo u eg setcontext—halkaas oo tilmaamaha amarka bartilmaameedku NULL noqday ka hor intaan xakamaynta la wareejin. Si kale haddii loo dhigo, xog khaldan oo ka timid maktabadda unwind, halkii ay ka ahaan lahayd boos cinwaan soo-celin oo khaldan oo rasada ku jira.

Taasi si weyn bay dhibaatada u soo koobtay. Ama GNU libunwind ayaa xisaabinayay xaalad bartilmaameed oo khaldan, ama xaalad sax ah ayuu xisaabinayay balse wax baa xumaynayay ka hor intaan la dabaqin.

Waxaan akhrinnay koodhka isha ee GNU libunwind, waxaana ogaanay inuu rasada ku sameeyo ucontext_t, buuxiyo xaaladda diiwaannada ee la doonayo ee qaab-dhismeedka maareeyaha nadiifinta, kadibna tilmaame u gudbiyo qaab-dhismeedka xogeedkaas una dhiibo hab-raac hoose oo isku-ururin gudaha ah: _Ux86_64_setcontext.

Halkaas waxaan haysannay dhammaan qaybihii.

ucontext_t-ga la sameeyey wuxuu ku nool yahay mid ka mid ah stack frames-ka uu _Ux86_64_setcontext la furfuro inta hawshaas socoto. _Ux86_64_setcontext ma ka akhrinayey qaab-dhismeedka xogeedka kadib markuu beddelay %rsp, xilligaas oo qaab-dhismeedka xogeedku uusan mar dambe qayb ka ahayn rasada firfircoon? Taasi waxay ka dhigi lahayd mid u nugul in gaarsiinta calaamada, sida SIGUSR2-geena joogtada ah.

Jawaabtu waxay ahayd haa.

Kuwani waa lixdii amar ee ugu dambeeyay ee _Ux86_64_setcontext ee noocii GNU libunwind ee aan isticmaalaynay, kuwaas oo inta badan ka kooban amarro mov ah oo ka rarta xusuusta una rarta diiwaanka bartilmaameed:

(%rdi wuxuu tilmaamayaa ucontext_t rasada lagu qoondeeyey, iyo macros-ka UC_MCONTEXT_* waxay si fudud ugu ballaartaan masaafada go’an ee diiwaan gaar ah lagu kaydiyo.)

Amarka koowaad waa bilowga daaqadda xaalad-tartanka. Wuxuu cusboonaysiiyaa %rsp si uu ugu jeedo hoosta cusub ee rasada firfircoon. Isla marka ay tani dhacdo, qaab-dhismeedka xogeed ee uu tilmaamayo %rdi mar dambe kama tirsana rasada firfircoon (ama aagga cas), mana aha wax xudun-dhexaadka ka reebban tahay inay taabato.

Caadiyan tani dhibaato ma keento, laakiin haddii signal yimaado daqiiqadda saxda ah (ama khaldan?), xarun-dhexaadku wuxuu qaabka calaamadda ka dhisi doonaa %rsp-128. Taasi waxay ku dul-qori kartaa xusuusta uu %rdi tilmaamayo.

Haddii taasi dhacdo ka hor instruction-ka xiga ee akhriya UC_MCONTEXT_GREGS_RIP(%rdi), markaas tusiyaha amarka ee la soo celiyay waa la musuqmaasuqi karaa. Burburradeennii, wuxuu noqday NULL.

Taasi waa cillada.

This assembly also explains one of the observations that confused us: why function X had a NULL in the return address slot of the preceding stack frame.

setcontext was written to restore all registers, including %rdi, so it can’t use that register to read UC_MCONTEXT_GREGS_RIP(%rdi) at the final moment of the control transfer. Instead, it reads the value earlier, saves it to the stack, restores a few more registers, then uses retq to read the saved value and transfer control.

What looked in the cores like “a function returned to NULL” was actually “the unwinder synthesized a target return address on the stack, but that target had been corrupted before the transfer completed.” We had assumed that corruption of the return address slot must happen in-place, because we didn’t know of any places where (corruptible) data was written to the return address slot on purpose.

What makes this bug seem absurd is how narrow this race window is. In this kind of race condition, the external event (the signal) needs to happen in between two steps taken by another thread. The closer those steps are to each other, the less likely the race condition is to happen.

In this case the vulnerable window is literally one instruction wide! A signal must be delivered after %rsp has been changed, but before the next instruction loads %rip. Several simple instructions like this can be run per cycle on a modern super-scalar out-of-order CPU, so the race window is roughly a hundred picoseconds.

When we found this race, our first reaction was that it must be too rare to explain the observed crash rate. We were seeing more than a dozen return-to-null crashes per day across the fleet. Could a one-instruction race during exception cleanup really account for that?

We turned to Fermi estimation. If the vulnerable window is on the order of $10^{-10}$ seconds and SIGUSR2 arrives every $10^{-2}$ seconds of CPU time, then each exception cleanup handler or catch block has a roughly $10^{-8}$ probability of losing the race.

Rockset uses exceptions as part of its internal ingest backpressure mechanism. A single overloaded host can throw on the order of $10^{4}$ exceptions per second. That implies the mean time between failures of a host using backpressure is $10^{4}$ seconds, or one crash every few hours. At fleet scale, that is more than enough to explain the observed crash frequency.

The GNU libunwind bug is old—more than 18 years old, present in the first x86_64 version that supported C++ exception unwinding.

So why did it show up now?

The crash rate is roughly proportional to how many exceptions are thrown and how many signals are delivered. It’s also dependent on how much stack the signal handler consumes.

Rockset is unusual on all three axes. We throw exceptions at high rates as part of normal overload control; we deliver SIGUSR2 unusually often because of coarse_thread_cputime_clock; and earlier this year we made the SIGUSR2 handler use more stack by adding a call to timer_getoverrun, so we could account for merged signals.

That last change seems to have been important. If the handler uses little enough stack, it may not reach and overwrite the stale ucontext_t memory. Before that change, we do not observe these crashes at all. After the change the rate remained low until we ramped up load for some use cases that stressed the backpressure mechanism.

In other words, the libunwind bug has always been there, but the product of our exception rate, signal rate, and handler stack usage had only recently crossed the threshold where it became operationally visible.

This mechanism also explains the coincidence that both the hardware bug and the libunwind bug crashed mostly inside DocumentTree::updateDocument. Crashes from libunwind were heavily biased toward this method, because it’s always active at the point we throw an exception to apply ingest backpressure. It was also heavily selected for the %rsp-misalignment crashes because the bad hardware node was of a SKU that we use for bulk ingest, which spends the majority of its CPU time in that method.

Our immediate mitigation was to switch from GNU libunwind to libgcc’s unwinder. That was a good trade on its own: libgcc’s implementation has benefited from a lot of work to reduce lock contention, which matters when scaling to large VMs.

We also upstreamed a self-contained reproducer and a fix⁠(ku furmaa daaqad cusub) to GNU libunwind, and verified that the other unwinders don’t have a similar issue.

This debugging journey taught us a lot about the specific details of dynamic linking, DWARF unwind metadata, Linux signal delivery, the System V ABI, and C++ exception machinery. But the main lesson was simpler than any of that.

The most important step was not the clever assembly reading or deep knowledge of the details. It was building a high-quality data set. In the absence of this data set, we were mixing two distinct phenomena into one story and trying to reason our way out of the confusion. Once we had accurate and complete population data, the structure of the problem became obvious: one crash population belonged to a bad host, and the other belonged to a race in libunwind. Once the data got better, the debugging got easier.

For infrastructure systems like Rockset, that matters a lot. This investigation reinforced our commitment to deep instrumentation, automated investigations, and continual improvements in our operational tooling. Reliability is not just about fixing bugs after they happen—it’s about building the data, workflows, and skills that turn impossible problems into diagnosable and solvable ones.