The Dark Art of Postmortem Debugging

Reading a crash out of frozen memory, from an Android ramdump to a Linux kernel vmcore.

There’s a kind of debugging most engineers never do, and half believe they can’t.

A device crashes hard. No panic on the console, no stack trace, nothing written to disk. Just a reboot and a few gigabytes of frozen RAM the device dumped on its way down. Turning that memory back into an account of what killed the device has a reputation. Arcane. Reserved for people with a JTAG rig, an open vendor support case, and a decade of kernel internals in their heads. Everyone else reflashes and moves on.

That reputation is the most useful lie in the field. Reading a dump is not sorcery. It’s engineering, and the only reason it feels like sorcery is that the workflow was gated at every step behind money, hardware, or a vendor you have to wait on. Take the gates away and what’s left is difficult, learnable, and mostly mechanical. This is about what’s actually behind the curtain.

Start with who gets to look, because that’s the first surprise.

If you maintain a ROM for a phone you own, you probably can’t even collect a full dump. Full-memory capture on a Qualcomm device is an engineering-mode feature. On a retail build it’s off by default, debug is fused down, and the recovery paths that used to let you in, EDL and friends, are increasingly blown shut at the factory. You can sometimes get a minidump, a handful of pre-selected regions, and even that ships with the vendor’s own warning that the data may be incomplete or not present at all. So the person with the most motivation to read a dump, the one debugging their own device on their own time, is locked out at collection.

The OEM engineer has the opposite problem. They can collect a dump. They just can’t read it, so they don’t. A dump is never the first debugging step, because the first step has to be something you can act on, and raw memory isn’t that for most teams. So the dump gets collected at the end, when the silicon vendor asks for one to attach to a support case. The richest artifact the device produces becomes a formality you generate on your way to waiting for someone else.

A few of the largest OEMs do read dumps in-house. They do it the old way, with a Lauterbach and TRACE32, tens of thousands of dollars a seat and a specialist who lives in it. Even there it’s a heavyweight, late-stage move. So across the industry the same thing is true from opposite directions. The dump sits at the end of the funnel, if it gets collected at all.

And it feeds on itself. You don’t collect dumps because you can’t read them, and you never learn to read them because you don’t collect them. The skill atrophies on both ends, and the reputation for sorcery hardens, because the only people still doing it are the few who paid the toll.

Here’s the tell that it was never really about difficulty. Take that same engineer and hand them a Linux server that just kernel-panicked. Completely different experience. kdump already caught a vmcore. The distro ships the matching debug symbols as a package you install in one command. You open it in crash or drgn, both free, both open, and start reading. Same underlying problem, frozen memory you have to reason about from the outside, and every door is standing open. The wall around Android dumps was never the memory. It’s the provisioning.

What actually makes it hard#

So what does make a dump hard to read, once you’re holding one?

Start with the shift that breaks people first. Every other artifact you’ve debugged is a recording. A log, a trace, a profile: a sequence, cause turning into effect, read front to back. A dump is a photograph. One instant, the machine frozen at the moment it died, and you reason backward from what’s left. That inversion is why the first attempt feels like staring at a wall. You’re not following a story. You’re doing pathology.

Then the mechanical walls, in the order they hit you.

There’s no addressing. A running kernel has hardware translating virtual addresses to physical ones on every access. In a dump that machinery is gone, and every pointer in memory is a number that means nothing until you rebuild the translation the CPU was using and walk it yourself. The dump isn’t one clean image either. It’s a set of physical regions with holes: carve-outs, device space, pages the collector never captured. A structure walk can run off the end of a captured region, and “this reads as zero” and “this was never captured” look identical unless you deliberately built the difference in.

The kernel’s base address is randomized every boot. Even with the exact matching debug image in hand, nothing resolves until you recover that per-boot offset from the dump itself. There’s a small bootstrap trap in that: you need the address translation to read the kernel’s symbols, and you want the symbols to fix up the translation. Solving that chicken and egg is most of what a good extractor does.

Now the one people leave out. Symbols tell you where a structure lives, not what shape it has. The offset of a field inside a kernel struct depends on the build config, which debug options were compiled in, which compiler ran. Two devices on the same kernel version can lay the same struct out differently. Hardcode an offset and it won’t fail loudly on the next build. It returns a number that looks fine. Confident garbage is worse than an error. An error stops you. A plausible wrong value sends you a day down the wrong path and lets you write a wrong conclusion with confidence.

One more, because it burns people who think they’ve found the answer. Present is not enabled. A kernel debug feature’s data structures are in the image because someone compiled them in. That says nothing about whether the feature was running. Some kernel instrumentation shuts itself off on the first error and freezes, so its records are all there, decode perfectly, and describe a moment long before the crash. Ask the image what was built. Ask the dump what was running. Never let one answer for the other.

That’s the wall. It’s real, and it’s entirely mechanical. None of it needs a hardware debugger. It needs the translation done right and a refusal to trust a number you can’t source.

You might expect the vendor’s own tooling to carry you over it. Qualcomm publishes an open-source ramdump parser, and for anyone without a JTAG rig it’s the whole toolbox. Point it at a current kernel and it often won’t start, or worse, it starts and quietly lies. One I worked assumed the kernel image sits at the base of DRAM. On that platform the image loaded higher, so the constant it derived for the entire address translation was wrong, and every structure it walked afterward came out plausible and false. Nothing errored. A parser like that is fitted to the kernels the vendor’s own baseline shipped, and the public drop lags the internal one, which lags whatever your device actually runs. So it has to be re-fitted per board and per kernel generation, and the person doing the re-fitting is the same person who was trying to debug a crash. You become an expert in the tool before the tool tells you anything about your bug.

What a dump gives up#

Here’s the part that would change how you debug, if you believed it. Once you can read one, a dump answers questions a log physically cannot.

Take a crash with no backtrace. Most engineers hit that and stop, because the whole mental model is: get symbols, symbolize the stack, read the answer. On one dump I worked, Android 16, kernel 6.12 on a mid-tier Qualcomm part, the kernel log came out clean for about 4,000 records and then stopped. No panic, no oops, no trace. That absence is the evidence. The kernel never faulted. The secure world caught an asynchronous interconnect error and brought the system down from underneath it, which means there’s no faulting stack to find, ever, for that class of crash. The thing people read as a dead end is a signature.

Take “you can’t prove a race from a snapshot.” The root cause on that same dump was a race between a storage controller gating its own clock and a key-derivation step reaching into that controller’s crypto registers. You can’t watch a race in a photograph. But the controller’s own driver state was in memory, and its clock-gating field read as gating in flight at the instant of capture. No symbol pointed at that object. It was found by locating the device’s name string in memory, finding what pointed at it, and following that to the driver’s private data. The inference became a measurement. That is the thing a dump does that nothing else can.

Take symbols as a precondition. Most tooling treats the matching kernel image as a gate: no image, no output. A lot is readable from raw memory with none of it. The reset reason. The interconnect fault syndrome. The secure world’s own event record. Each co-processor’s account of why it stopped. Symbols are a tier that adds resolution, not a gate that decides whether you get anything at all. Reframing that is probably the most useful single thing here, because the common real-world case is exactly the one where nobody can find the image.

And the one that kills the “memory is unreadable” instinct for good. The user-visible Android log doesn’t live in a file. It lives in a userspace daemon’s heap, in compressed chunks scattered across physical pages. Carve those pages physically and every frame is corrupt. To read them you reconstruct the process’s virtual address space and read the chunks in virtual order. Do it right and nearly 28,000 ordered log entries come out of a device that wrote nothing to disk. Symbols get you to the process. Geometry is what turns its scattered pages back into a log.

For scale, once extraction runs clean on a dump like that: 760 live processes, each with a symbolized kernel stack. 5,375 memory regions for one process, read out of a tree whose shape the kernel changed a couple of versions back. 22,484 hardware register values from the SoC’s own black-box recorder, stored in a compressed encoding you have to decode before a single value belongs to a register. The build fingerprint. Two dozen frames of what was on the screen. None of it was in a log. All of it was in the RAM the device already dumped.

One discipline holds the whole thing up, and it’s the line between reading memory and hallucinating over it. Before you trust anything, prove you’re reading the right kernel at all. Find something in the dump whose value you already know independently, and confirm it reads back correctly through the translation you built. If it doesn’t, everything downstream is confident fiction. Verify the lens before you report what you saw through it.

The role of LLMs, and where they are allowed to think#

A language model has a job in this work. It begins after the extraction is done.

Memory forensics is deterministic. The bytes are the bytes, the offset is the offset, and a probabilistic model has no business guessing at any of it. That is exactly why the model should never touch the extraction. It should read nothing out of memory itself. What it does instead is reason over evidence that was already pulled out deterministically, and there’s a failure lurking in that handoff that matters more the moment a model is involved.

“We looked and there’s nothing there” and “we couldn’t look” produce the same empty result. One of them is a lie. There aren’t two states of a missing thing, there are four. We didn’t try. It genuinely isn’t there. We read it, here it is. We tried and failed partway through. Collapse them into “absent” and everyone downstream reads absence as health.

This stops being philosophy the moment a model narrates the output. Hand it a table that failed to read and it will fluently tell you the thermal subsystem was nominal. It isn’t lying on purpose. It was handed an empty cell with no way to know why the cell was empty. So the extraction has to carry its own honesty: this was read, this was truly absent, this failed here and stopped. Same with limits. Dropping a whole item and shortening every item are different failures. One says a driver might be missing entirely. The other says everyone’s present but partial. A single truncation flag that forces the worse reading onto the ordinary case quietly turns a report into a story.

There’s a division of labor inside that, and it’s the discipline in one line. Deterministic extraction produces the evidence. The model reasons over evidence it isn’t allowed to invent. Never let the model produce the evidence, and never let the extraction produce the verdict.

What it can’t do#

The reputation for sorcery partly comes from people overselling what a dump proves, so here are the honest limits.

A snapshot can’t prove a missing lock directly. What it can show you is the wreck. Two threads sitting in a path that should only ever hold one. A structure left in a state you can only reach if two writers overlapped. The dump hands you the collision frozen in place. It doesn’t hand you the instant of overlap, because a photograph has no motion in it. That a lock was missing is an inference from the wreck plus the code, the same reasoning a vendor’s own engineer uses to write the fix. Reading the dump doesn’t skip that step. It gets you to it in an afternoon instead of a support cycle.

And a full memory dump is the most sensitive thing a device can produce. Every process’s memory, the whole user-visible log with whatever rode along in it, key material, a literal picture of what was on the screen. Read one somewhere the device’s owner doesn’t control and you’ve built a data-exfiltration problem out of a debugging tool. That is why this belongs inside a boundary they own, and nowhere else.

The point#

None of what I described needed new evidence. The crash with no log, the race you supposedly can’t prove from a snapshot, the 28,000-line log nobody could read: all of it was already sitting in a dump the device handed over on its way down. Nobody was missing information. They were missing a way to look.

That’s the whole of the dark art. Not talent, not sorcery. A pile of gates, some fused into silicon, some priced in five figures, some just the habit of treating your richest evidence as a last resort. Take the gates away and the dump on your bench turns back into what it always was. The machine, telling you exactly what happened, in its own memory, for anyone with a way to read it.