Core dump epidemiology: fixing an 18-year-old bug

By Nathan Bronson, Member of Technical Staff

OpenAI’s models and agents increasingly rely on scalable data infrastructure in order to search for relevant data at inference time: when the models are thinking about your question. Some of these services are written in C++, whose low-level control of the system lets us maximize performance and minimize memory usage. Those efficiency benefits are important as we scale, but C++’s lack of memory safety means that bugs can cause crashes by writing to incorrect or non-existent memory addresses.

A few months ago we observed some crashes from inside the Rockset service, a bespoke part of our ChatGPT data infrastructure which is key to many data plugins and to searching over conversations. In each of these crashes, a normal C++ function seemed to finish and then return to a bogus address, causing the kernel to stop the program because the instruction pointer no longer pointed at code. Sometimes the return address slot in the stack frame was NULL. Sometimes the stack pointer CPU register itself seemed to be off by 8 bytes, as if %rsp had somehow been decremented in the middle of normal execution. In both cases the crash happened on return.

These are not normal failure modes for application code. A stray write that lands only on a saved return address is possible, but extremely unlikely. A bug that misaligns %rsp by 8 without involving inline assembly, setcontext, or longjmp (none of which we use) is even stranger, because compiled code only adjusts that register directly in the function prologue and epilogue. Every hypothesis we (or ChatGPT) could think of had strong evidence against it, so the bug seemed impossible.

What we assumed was one problem eventually turned out to be two unrelated bugs, coincidentally discovered at the same time. First, silent hardware corruption on one Azure host, where the CPU just didn’t do math correctly. Second, an 18-year-old race condition in GNU libunwind, an unnoticed bug in a widely used open source library.

This post is the story of how we identified and fixed seemingly inexplicable crashes by thinking like an epidemiologist and building a high-quality data set about the entire population of crashes.

First, let’s go deeper on Rockset. It’s a cloud-native data system for search and real-time analytics that we use for many internal use cases at OpenAI, such as sync connectors (Rockset was acquired by OpenAI in 2024). Streaming updates are used to maintain an up-to-date index of a workspace’s knowledge base so that ChatGPT can search for relevant information when answering questions or performing actions.

Rockset’s execution layer is written in C++. The C++ language provides low-level access to the CPU, which is good for performance and efficiency, but it means that application bugs can lead to invalid memory accesses and segfaults. To help track these down we use folly’s fatal signal handler to log a stack trace when a crash happens, and we upload the corresponding core dumps (a snapshot of the state of the program when it crashed) to Azure blob storage for later analysis. All of Rockset’s query processing leaves are replicated, which minimizes the client impact of a crash. However, each segfault corresponds to a bug that needs to be fixed to meet our reliability and quality goals.

Our initial approach was to treat these cores like a conventional debugging problem: inspect a few core dumps very closely, form hypotheses, and rule them out one by one.

Most of the crashes occurred in a method called DocumentTree::updateDocument. In these crashes it appeared that updateDocument had called some unknown function X, the stack had become corrupted while X was active, then X had returned to an address that wasn’t executable code. In some cases X’s just-popped frame looked valid except that its saved return address was NULL. In other cases the stack pointer itself looked wrong, but the next valid frame still seemed to be updateDocument.

We didn’t know when the stack was getting corrupted, which left a huge search space. updateDocument is a large method that undergoes a lot of inlining, so the number of candidates for X was overwhelming.

Was this a bug in our C++ code? A compiler or linkage issue? A problem in one of our runtime libraries? A Linux kernel bug around signal delivery or context switching? Something even rarer? If this was a stray write, why wasn’t it caught by our ASAN staging environment?

We tried to use our application-level logs to identify all occurrences of the problem, but stack-corruption bugs are hard to classify from logs alone because the logged stack traces are themselves corrupted or missing. We weren’t able to construct a log query that didn’t have both false positives and false negatives. We manually inspected more cores and found some additional examples, but that process was too labor-intensive to give us a trustworthy data set.

At this stage of the investigation, we (incorrectly) ruled out a hardware bug, because we saw crashes across multiple regions and multiple hardware types, so we were still looking for software-only causes. For a few days, we went super-deep on a single misaligned-%rsp crash, reconstructing the pre-crash history using stack and register contents. This produced some possible clues, but because we didn’t let go of our initial conclusions that all of the bugs had the same cause, this didn’t get us unstuck.

Before getting to the turning point of our investigation, it’s important to explain what kind of information we were extracting from the core files.

Rockset is compiled with -fno-omit-frame-pointer, so the active stack frame is always reachable through %rbp, and callers form a linked list of frame pointers.

On Linux x86_64, the AMD64 System V ABI also reserves 128 bytes below %rsp as the red zone. That region is available to userspace code and, importantly, the kernel promises not to clobber it when it delivers a signal, as part of the ABI contract.

The red zone was central to our debugging of a post-return crash, because it preserves some information from before the return. When a SIGSEGV is triggered, folly’s fatal signal handler runs on the crashing thread’s stack. Stack frames that are no longer active (because their function has returned) will get clobbered by the signal handler, except for the last 128 bytes. That’s why we can say things like “X’s just-popped stack frame looked valid, except for a NULL return address.” The red zone preserves some of the inactive frames, or sometimes just the tail of one inactive frame.

We found one misaligned-stack crash in which all of the functions involved were very small. That let us see that %rsp had become misaligned during execution of a relatively simple function, and that more calls had succeeded afterward. The program only crashed when the active function finally tried to return. None of those code paths used exceptions, inline assembly, setcontext, or longjmp, so if the stack pointer truly changed in the way the core suggested, no plausible bug in userspace code explained the issue.

That pushed us toward the kernel.

Rockset uses signals more aggressively than most programs. Query execution is broken into many lightweight tasks that exchange data. This is important for handling high-QPS workloads efficiently, but it makes per-query CPU accounting awkward as work for many queries is multiplexed onto the same thread pool.

Our solution is something we call coarse_thread_cputime_clock, which approximates clock_gettime(CLOCK_THREAD_CPUTIME_ID, ...) cheaply enough to sample at every task boundary. The timer_create API can be used to schedule a periodic signal delivery based on several notions of the passage of time, including the accumulation of CPU time. We schedule a signal (SIGUSR2) to be delivered every few milliseconds of CPU time, at which point the signal handler updates a thread-local value. Even though many tasks don’t see the coarse clock advance while they are executing, summing all of the deltas produces an unbiased estimate of the actual CPU time for a query.

Because we deliver signals so often, a rare kernel bug around context switching or signal delivery seemed plausible. We spent time reading bug reports, kernel source code, and the Azure-specific kernel patches. We tried stress tests. We weren’t able to find anything that seemed related.

At that point we decided to step back and try a different approach.

There are two broad ways to debug a problem like this.

One is to act like a doctor of sorts: focus on one patient, run lots of tests, and try to diagnose a single case from detailed evidence.

The other is to act more like an epidemiologist: look at the entire population and ask whether there are patterns that a single case cannot reveal. Did the bug start at a specific release? Does it correlate with one hardware SKU (the specific CPU and server model), one region, or one kernel version? Are there multiple distinct clusters hiding inside what looks like one syndrome?

We had mostly been in doctor mode. The key shift was deciding that we needed to gather high-quality population data.

Our previous attempts to automatically find all of the instances of the problem failed because we were trying to use text searches over the logs. The core dumps themselves have a lot more information, but looking at them manually didn’t scale. We decided to invest the effort to build a pipeline that could automatically analyze the core dumps.

We had ChatGPT write a script that downloaded a prefix of each core file, extracted the registers, filtered known false positives using the logs, and automatically labeled the crash as return-to-null, misaligned-stack, or other. Then we ran that script in parallel over every production Rockset core dump from the previous year.

This was the turning point.

Once we had a clean data set, correlations appeared immediately. What we had been treating as one weird bug was actually two separate crash populations.

The return-to-null cores were spread across many clusters and geographic regions. Their frequency had increased recently, but there was no crisp start date and no clean infrastructure boundary.

The misaligned-stack crashes looked completely different. They all came from one region, had a clear start date, and never happened on nodes that had been running for a long time. Even though they involved multiple Azure VMs (virtual machines hosted in the cloud), the pattern looked like one physical machine with bad hardware causing problems for whichever VM happened to land on it.

That was the moment we realized we had been mentally conflating two bugs. Because we had been mixing counterexamples from both bugs, we couldn’t find a single coherent explanation.

Agus liosta glan de nóid Kubernetes agus stampaí ama againn, bhíomar in ann na tuairteanna cruach mhí-ailínithe a rianú siar chuig óstach fisiciúil amháin, a bhí furasta a chur ar liosta diúltaithe.

Níor éirigh linn truailliú na gclár ar an óstach sin a atáirgeadh i dtimpeallacht rialaithe, fiú tar éis roinnt seachtainí de thástáil struis. Ach a luaithe a baineadh an t-óstach fadhbach as seirbhís, d’imigh na tuairteanna cruach mhí-ailínithe.

Ní réiteach buan é an drochóstach a bhaint, sa chiall nach gcuireann sé cosc ar tharlú nua den fhadhb chéanna. Is féidir linn, áfach, na bogearraí a athrú ionas go mbeidh ceist chomhchosúil furasta a bhrath agus a láimhseáil má tharlaíonn sí arís. D’fheabhsaíomar ár láimhseálaí comhartha marfach chun staid na gclár a chur san áireamh, ionas gur féidir linn atarlú a bhrath ó na logaí amháin (ní gá core dump). D’athraíomar an plána rialaithe ionas go n-athúsáidtear VManna de ghnáth in ionad iad a athchúrsáil, rud a fhágann go bhfuil sé i bhfad níos éasca drochnóid a bhrath ag ár leibhéal den chruach bhonneagair. Nuashonraíomar ár leabhair reatha freisin (agus samhlacha meabhracha ár bhfoirne) chun an fhéidearthacht seo a chur san áireamh.

Agus na tuairteanna drochóstaigh scartha amach, bhí sé i bhfad níos éasca réasúnú a dhéanamh faoi na core dumps filleadh-go-null a bhí fágtha. Níos luaithe, bhí dífhilleadh eisceachtaí curtha as an áireamh againn mar cheapamar go raibh frithshamplaí againn: tuairteanna i gconairí cód nár úsáideadh eisceachtaí iontu go cinnte. Ach tháinig na frithshamplaí sin go léir ón gcnuasach truaillithe crua-earraí.

Nuair a d’fhilleamar ar na cores a bhí fágtha leis sin san áireamh, fuair muid amach go raibh an chonclúid seo díreach bun os cionn: bhí na tuairteanna go léir ag tarlú le linn dífhilleadh eisceachtaí.

Nuair a chaitheann C++ eisceacht, caithfidh an t-am rite a fháil amach cén bloc gabhála ba cheart í a fháil agus cé na dítheachtóirí nó láimhseálaithe glanta ba cheart a rith ar an mbealach. Astaíonn an tiomsaitheoir na meiteashonraí seo, ach tarlaíonn an meaitseáil iarbhír go dinimiciúil ag am rite.

Ní dhéanann an fheidhm a ghlaonn throw dífhilleadh eisceachtaí i ndáiríre, ach déanann feidhmeanna cúnta a ghlaonn an cód tiomsaithe a eascraíonn as. Scrúdaíonn na gnáthaimh am rite sin an chruach, faigheann siad meiteashonraí faoi na feidhmeanna atá ar an gcruach, lorgaíonn siad láimhseálaithe glanta agus bloic ghabhála go dinimiciúil, agus ansin aistríonn siad rialú chuig ceann de na háiteanna sin. Áirítear le haistriú rialaithe na frámaí cruaiche idir eatarthu go léir a dhífhilleadh (lena n-áirítear frámaí na bhfeidhmeanna cúnta).

Ó thaobh oibríochta de, tá sé seo i bhfad níos gaire do longjmp nó d’athrú snáithín ná do ghnáthghlao agus filleadh. Ní mór cláir sábhála an ghlaoite a athbhunú, chomh maith le cláir fhráma na cruaiche %rbp agus %rsp.

Nascann ár ndénártha le dhá leabharlann ina bhfuil cur chun feidhme na bhfeidhmeanna a dhéanann dífhilleadh eisceachtaí C++: libgcc agus GNU libunwind. Ba iad sainmhínithe GNU libunwind a roghnaigh an nascóir dinimiciúil. Chuir sé sin iontas orainn; bhíomar ag súil go mbuafadh cur chun feidhme libgcc mar gheall ar rialacha leaganaithe siombailí; áfach, léirigh iniúchadh ar dhénárthaí reatha nach mar sin a bhí.

Ag an bpointe seo, d’athraigh ár hipitéis oibre, agus muid ag maolú toimhde eile a rinneamar nuair a cheapamar nach raibh ann ach fabht amháin.

B’fhéidir nach raibh muid ag feiceáil gnáthfheidhm ag filleadh ar NULL. B’fhéidir go raibh muid ag feiceáil aistriú dífhillte—athbhunú clár de stíl setcontext go bunúsach—ina raibh pointeoir treorach an chinn scríbe éirithe NULL sular aistríodh rialú. I bhfocail eile, sonraí míchearta ón leabharlann dífhillte seachas sliotán seolta fillte mícheart ar an gcruach.

Chúngraigh sé sin an fhadhb go mór. Bhí GNU libunwind ag ríomh staid chinn scríbe mhícheart, nó bhí sé ag ríomh na staide cearta agus bhí rud éigin á truailliú sula bhféadfaí í a chur i bhfeidhm.

Léamar foinse GNU libunwind agus fuair muid amach go sintéisíonn sé ucontext_t ar an gcruach, go líonann sé isteach an staid cláir inmhianaithe d’fhráma an láimhseálaí glanta, agus ansin go dtugann sé pointeoir chuig an struchtúr sin do ghnáthamh cóimeála inmheánach: _Ux86_64_setcontext.

Ag an bpointe seo bhí na píosaí go léir againn.

Tá an ucontext_t sintéisithe ina chónaí i gceann de na frámaí cruaiche a dhífhillteann _Ux86_64_setcontext le linn forghníomhú na feidhme sin. An raibh _Ux86_64_setcontext ag léamh ón struchtúr tar éis dó %rsp a athrú, agus ag an bpointe sin nach raibh an struchtúr mar chuid den chruach ghníomhach a thuilleadh? D’fhágfadh sé sin é leochaileach do scríobh anuas ó sheachadadh comhartha, amhail ár SIGUSR2 minic.

Ba é an freagra: bhí.

Seo iad na sé threoir dheireanacha de _Ux86_64_setcontext sa leagan de GNU libunwind a bhí á úsáid againn, ar treoracha mov iad den chuid is mó a lódálann ó chuimhne chuig clár cinn scríbe:

(díríonn %rdi ar an ucontext_t a leithdháiltear ar an gcruach, agus leathnaíonn na macraí UC_MCONTEXT_* go dtí an fritháireamh seasta ina stóráiltear clár ar leith.)

Is í an chéad treoir tús na fuinneoige ráis. Nuashonraíonn sí %rsp chun díriú ar bhun nua na cruaiche gníomhaí. A luaithe a tharlaíonn sé seo, níl an struchtúr a léiríonn %rdi mar chuid den chruach ghníomhach (ná den limistéar dearg) a thuilleadh, agus níl sé cosanta ón eithne a thuilleadh.

De ghnáth ní chruthaíonn sé seo fadhbanna, ach má thagann comhartha ag an nóiméad ceart (mícheart?) go díreach, tógfaidh an eithne an fráma comhartha ag %rsp-128. Féadfaidh sé sin an chuimhne a léiríonn %rdi a scríobh anuas.

Má tharlaíonn sé sin sula léann an chéad treoir eile UC_MCONTEXT_GREGS_RIP(%rdi), féadfar an pointeoir treorach athbhunaithe a thruailliú. Inár dtuairteanna, d’éirigh sé NULL.

Sin é an fabht.

Míníonn an tionól seo ceann de na breathnuithe a chuir mearbhall orainn freisin: cén fáth a raibh NULL ag feidhm X sa sliotán seolta fillte den fhráma cruacha roimhe sin.

Scríobhadh setcontext chun gach clár a athbhunú, lena n-áirítear %rdi, agus mar sin ní féidir leis an gclár sin a úsáid chun UC_MCONTEXT_GREGS_RIP(%rdi) a léamh ag an nóiméad deiridh den aistriú rialaithe. Ina ionad sin, léann sé an luach níos luaithe, sábhálann sé ar an gcruach é, athbhunaíonn sé cúpla clár eile, agus ansin úsáideann sé retq chun an luach sábháilte a léamh agus rialú a aistriú.

An rud a bhí le feiceáil sna croíthe mar “d’fhill feidhm ar NULL” ba ea i ndáiríre “shintéisigh an dífhillteoir spriocsheoladh fillte ar an gcruach, ach bhí an sprioc sin truaillithe sular críochnaíodh an t-aistriú.” Bhíomar den tuairim go gcaithfeadh truailliú an tsliotáin seolta fillte tarlú ina áit féin, toisc nach raibh a fhios againn faoi aon áiteanna ina scríobhtaí sonraí (a d’fhéadfaí a thruailliú) isteach sa sliotán seolta fillte d’aon ghnó.

Is é an rud a fhágann go bhfuil cuma áiféiseach ar an bhfabht seo ná cé chomh caol is atá an fhuinneog rása seo. Sa chineál seo de choinníoll rása, caithfidh an teagmhas seachtrach (an comhartha) tarlú idir dhá chéim a ghlacann snáithe eile. Dá ghaire na céimeanna sin dá chéile, is ea is lú an seans go dtarlóidh an coinníoll rása.

Sa chás seo, níl san fhuinneog leochaileach ach treoir amháin ar leithead, go litriúil! Caithfear comhartha a sheachadadh tar éis %rsp a bheith athraithe, ach sula lódálann an chéad treoir eile %rip. Is féidir roinnt treoracha simplí mar seo a rith in aghaidh an timthrialla ar LAP nua-aimseartha sárscálach as ord, mar sin tá an fhuinneog rása thart ar chéad picishoicind.

Nuair a fuaireamar an race seo, ba é ár gcéad imoibriú gur dócha go raibh sé ró-annamh chun an ráta tuairte a bhí á bhreathnú againn a mhíniú. Bhíomar ag feiceáil níos mó ná dosaen tuairt return-to-null in aghaidh an lae ar fud an fhleasc. An bhféadfadh race aon treorach amháin le linn glantacháin eisceachta cuntas a thabhairt air sin i ndáiríre?

D’iompaíomar ar mheastachán Fermat. Má tá an fhuinneog leochaileach ar ord $10^{-10}$ soicind agus má thagann SIGUSR2 gach $10^{-2}$ soicind d’am LAP, ansin tá dóchúlacht thart ar $10^{-8}$ ag gach láimhseálaí glantacháin eisceachta nó bloc catch an rás a chailleadh.

Úsáideann Rockset eisceachtaí mar chuid dá mheicníocht inmheánach backpressure ionghabhála. Is féidir le hóstach amháin atá ró-ualaithe eisceachtaí ar ord $10^{4}$ a chaitheamh in aghaidh an tsoicind. Tugann sé sin le fios gurb é $10^{4}$ soicind an meán-am idir teipeanna d’óstach a úsáideann backpressure, nó tuairt amháin gach cúpla uair an chloig. Ar scála an fhleáit, is mó ná a dhóthain é sin chun minicíocht na dtuairteanna a breathnaíodh a mhíniú.

Tá an fabht GNU libunwind sean, níos mó ná 18 mbliana d’aois, agus bhí sé i láthair sa chéad leagan x86_64 a thacaigh le dífhoirceannadh eisceachtaí C++.

Mar sin, cén fáth ar tháinig sé chun cinn anois?

Tá an ráta tuairte comhréireach go garbh leis an méid eisceachtaí a chaitear agus leis an méid comharthaí a sheachadtar. Braitheann sé freisin ar an méid cruacha a ídíonn an láimhseálaí comharthaí.

Tá Rockset neamhghnách ar na trí ais ar fad. Caithimid eisceachtaí ag rátaí arda mar chuid de ghnáthrialú ró-ualaigh; seachadann muid SIGUSR2 go neamhghnách minic mar gheall ar coarse_thread_cputime_clock; agus níos luaithe i mbliana chuir muid ar an láimhseálaí SIGUSR2 níos mó cruacha a úsáid trí ghlao ar timer_getoverrun a chur leis, ionas go bhféadfaimis comharthaí cumaiscthe a chur san áireamh.

Is cosúil go raibh an t-athrú deireanach sin tábhachtach. Má úsáideann an láimhseálaí go leor beag cruach, b’fhéidir nach sroichfidh sé agus nach bhforscríobhfaidh sé an chuimhne ucontext_t atá as dáta. Roimh an athrú sin, ní fheicimid na tuairteanna seo ar chor ar bith. Tar éis an athraithe, d’fhan an ráta íseal go dtí gur mhéadaíomar an t-ualach do roinnt cásanna úsáide a chuir strus ar an meicníocht backpressure.

I bhfocail eile, bhí an fabht libunwind ann i gcónaí, ach ní raibh táirge ár ráta eisceachtaí, ár ráta comharthaí, agus úsáid chruaiche an láimhseálaí tar éis an tairseach a shárú go dtí le déanaí, nuair a d’éirigh sé infheicthe ó thaobh oibríochtaí de.

Míníonn an mheicníocht seo freisin an comhtharlú gur tharla tuairteanna ón bhfabht crua-earraí agus ón bhfabht libunwind den chuid is mó laistigh de DocumentTree::updateDocument. Bhí tuairteanna ó libunwind claonta go mór i dtreo an mhodha seo, mar go mbíonn sé gníomhach i gcónaí ag an bpointe a gcaithimid eisceacht chun cúl-bhrú ionghabhála a chur i bhfeidhm. Roghnaíodh go mór é freisin do na tuairteanna mí-ailínithe %rsp, toisc gur de SKU a úsáidimid le haghaidh ionghabháil ar an mórchóir a bhí an droch-nód crua-earraí, agus caitheann sé formhór a ama LAP sa mhodh sin.

Ba é ár maolú láithreach ná aistriú ó GNU libunwind go dífhillteoir libgcc. Ba mhalartú maith é sin ann féin: bhain cur i bhfeidhm libgcc tairbhe as go leor oibre chun iomaíocht ar ghlasanna a laghdú, rud atá tábhachtach agus VManna móra á scálú.

Chuireamar suas an sruth freisin atáirgeoir féinchuimsitheach agus ceartúchán⁠(osclaíonn i bhfuinneog nua) chuig GNU libunwind, agus dheimhnigh muid nach bhfuil fadhb chomhchosúil ag na dífhoirceannairí eile.

Mhúin an turas dífhabhtaithe seo go leor dúinn faoi shonraí sonracha nascadh dinimiciúil, meiteashonraí dífhoirceannadh DWARF, seachadadh comharthaí Linux, an ABI System V, agus innealra eisceachtaí C++. Ach bhí an príomhcheacht níos simplí ná aon cheann de sin.

Ba é an chéim ba thábhachtaí ná ní an léamh cliste ar an tionól ná eolas domhain ar na mionsonraí. Ba é tacar sonraí ardchaighdeáin a thógáil. In éagmais an tacair sonraí seo, bhíomar ag meascadh dhá fheiniméan ar leith in aon scéal amháin agus ag iarraidh réasúnú a dhéanamh chun muid féin a bhaint as an mearbhall. Nuair a bhí sonraí cruinne agus iomlána againn faoin bpobal, d’éirigh struchtúr na faidhbe soiléir: bhain pobal tuairteanna amháin le hóstach lochtach, agus bhain an ceann eile le rás in libunwind. De réir mar a d’éirigh na sonraí níos fearr, d’éirigh an dífhabhtú níos éasca.

I gcás córas bonneagair cosúil le Rockset, tá tábhacht mhór leis sin. Neartaigh an t-imscrúdú seo ár dtiomantas d’ionstraimíocht dhomhain, d’imscrúduithe uathoibrithe, agus d’fheabhsuithe leanúnacha ar ár n-uirlisí oibríochtúla. Ní bhaineann iontaofacht le fabhtanna a cheartú tar éis dóibh tarlú amháin—baineann sí le sonraí, sreafaí oibre, agus scileanna a thógáil a iompraíonn fadhbanna dodhéanta ina bhfadhbanna is féidir a dhiagnóisiú agus a réiteach.