Debug Runtime Errors in Production

Last updated: September 2026

Quick answer. To debug a runtime error in production, first check whether it left a stack trace. If it did, the standard method works: read the trace, reproduce it, set a breakpoint. The hard runtime errors leave no trace, because the kernel ended the process before it could write one. An out-of-memory kill sends SIGKILL, which the manual says "cannot be caught, blocked, or ignored," so no handler runs and nothing is logged by your application; the record is in the kernel log, not your app's. The same is true of a SIGSEGV segfault, a disk that returns ENOSPC, or a syscall that returns EACCES. For these, you stop looking in the application and read what the system did to the process, from the kernel: journalctl -k for the kill record, and a live view of the running system for the ones still happening, with something like yeet graph query.

A runtime error is any error that happens while your program runs rather than when it compiles, and on your laptop the drill is well worn: the program throws, you get a stack trace, you read it, you set a breakpoint near the offending line, and you step through until you see the bad value. That method is correct, and for a runtime error you can reproduce you should use it and stop reading here.

The reason this needs its own post is that the runtime errors that cost the most time in production are the ones where that method has nothing to work with. The process is gone, there is no stack trace, the log ends mid-sentence, and the thing that killed it left its record somewhere your application never looked. Debugging those is a different skill, and it starts with knowing why the trace is missing.

Why does my process die in production with no stack trace? (SIGKILL leaves none)

Because something outside your program ended it, and that something does not write to your log. A stack trace is produced by your language runtime when your code hits an exception it can see. When the kernel kills the process, or the process receives a fatal signal, your runtime never runs the code that would have written the trace, so there is nothing in the log by design rather than by accident.

The clearest case is the out-of-memory killer. When the machine runs out of memory the kernel selects a process and kills it, and it does so by sending SIGKILL. The signal manual is explicit about what that means for your chances of logging it: SIGKILL and SIGSTOP "cannot be caught, blocked, or ignored." A signal you cannot catch is a signal you cannot handle, which means no finally block runs, no shutdown hook fires, no last log line is written. From your application's point of view the process simply ceases to exist between one line and the next. The record of what happened is real, but it is in the kernel's log, reachable with journalctl -k or dmesg, not in the application log you were reading.

So the first move when a production process dies with no trace is to stop treating the missing trace as a mystery and start treating it as a signal in itself: a clean disappearance with no application error usually means the process did not fail, it was ended.

How do I debug an OOM kill on Linux?

Read the kernel log for the kill record, then find why the memory pressure built, because the kill is the symptom and the growth is the bug. The kill itself is logged by the kernel, and journalctl -k | grep -i "killed process" or dmesg | grep -i oom shows you the victim, its process id, and how much memory it was using when the kernel chose it. That confirms the diagnosis in seconds and tells you which process to investigate, which is often not the one you assumed.

The harder half is that the kill tells you the end state, not the trajectory, and the trajectory is where the fix lives. A process killed at 4GB might have grown there slowly over a day or spiked there in a second, and those are different bugs with different fixes. This is where a live view of the running system earns its place, because you want to watch memory per process over time before the next kill rather than reconstruct it after. Reading resident memory and growth for every process from the kernel, on the live host, turns "it got OOM-killed again overnight" into "this process climbs 200MB an hour and never releases it." The container memory question is the same measurement problem one layer in, where the per-process and per-cgroup numbers disagree by design and the disagreement is the clue.

How do I debug a segfault (SIGSEGV) in production?

A segfault is the kernel telling you the process tried to touch memory it was not allowed to, and the manual names it exactly that: SIGSEGV is an "invalid memory reference." Unlike SIGKILL you can catch it, and a core dump is the classic artifact, but in production two things usually go wrong with the classic approach. Core dumps are often disabled or size-limited on production hosts, so the dump you were counting on is not there, and even when it is, a segfault in a stripped release binary or across a foreign-language boundary gives you a backtrace that stops at an address rather than a line.

So the production-honest version of segfault debugging is to capture the context around the crash rather than only the crash itself. Confirm the signal in the kernel log, check whether a core pattern is configured with cat /proc/sys/kernel/core_pattern, and if the crash is recurring, watch the process live for what it was doing in the moments before, the files it opened, the syscalls it made, the memory it mapped. A segfault that only happens under real load is a runtime error in the truest sense: it exists only while the process runs against production input, and it is invisible to a debugger you cannot attach to a live service.

Why does my app get "No space left on device" (ENOSPC) when there is free space?

Because ENOSPC, which the errno list defines as "No space left on device," is reported against a specific filesystem or a specific limit, not against the disk in general, and the free space you are looking at is usually on a different one. The three that catch people are inodes, a full /tmp or a full /var/log on a separate mount, and a container or cgroup writable-layer limit that is smaller than the host disk. df -h shows bytes and df -i shows inodes, and a filesystem can be at 5% bytes and 100% inodes at the same time, which produces ENOSPC on every new file while the disk looks nearly empty.

The runtime-debugging version of this is to catch the write that fails at the moment it fails, because the error message names the errno but not the path or the process, and in production both are the question. Watching failed write and openat syscalls on the live host tells you which process hit ENOSPC and which path it was writing to, which turns a generic error into a specific full mount. This is the same shape as every other error here: the errno is a fact the kernel returned, and the context that makes it fixable is only available from the kernel at the moment the syscall failed.

How do I debug an EACCES "Permission denied" error at runtime?

Find which file the process was actually denied, because EACCES, defined in the errno list as "Permission denied," is thrown for a path the error rarely names, and the path is the whole answer. A process can hit EACCES on the file you expect, on a parent directory whose execute bit is missing, on a file blocked by a mandatory access control policy like SELinux or AppArmor even though the ordinary permissions look correct, or on a socket or device it does not have the capability to open. The permission bits you check by hand are only the first of those four.

The fastest way to resolve it in production is to watch the failing openat call and read the exact path and the exact error the kernel returned, rather than guessing which of the four causes applies. That is a runtime observation, not a static one, because the path a process opens is often computed at runtime from configuration or input you do not have in front of you. For an AI agent hitting EACCES on a hardened host specifically, the privilege side of this is worked through in AI agent privilege on Linux, where a dedicated user and capability set decide which of those four denials you will see.

How do I read what a live process was doing with eBPF instead of a debugger?

You read it from the kernel, because the kernel is the one vantage point that sees a process's errors, signals and failing syscalls without being attached to the process and without stopping it. This is the case every error above collapses into: the trace is missing or unhelpful, you cannot reproduce it locally, and you cannot pause a live production service to inspect it. The kernel already saw what happened; the task is to read its record.

This is the half of the problem I work on. I build kernel side tooling for Linux, and the reason I reach for the kernel on a production error is that it is where the truthful record already exists. yeet is a JavaScript runtime for Linux that reads the running system with eBPF and exposes it as a queryable graph, so you can ask a live host what a process is doing rather than attaching to it. A single daemon holds the privilege, and the graph is the read surface: processes with command lines, resident memory, page-fault counts, open file descriptors, plus CPU, memory, network and containers. For the memory-growth question behind most OOM kills, you read resident memory per process straight from the kernel:

curl -fsSL https://yeet.cx | sh                          # install the yeet daemon, once
yeet graph query '{ procs { pid comm status { vmrss } stat { majflt } } }'   # resident memory and major faults, per process

vmrss is the resident set size the OOM killer weighs when it chooses a victim, and majflt is the major-fault count that climbs when a box starts thrashing before it kills anything. The query reads current state from the kernel while everything keeps running, and a subscription streams the same fields live, so you can watch memory climb toward the next kill rather than reconstruct it from a dmesg line afterward. It is read-only by construction: the schema is built as Schema<Query, EmptyMutation, Subscription>, so there is no query that changes anything on the host, which is what makes it safe to run on production and safe to give to an AI agent investigating an incident.

For the errors that are about what a process did rather than how much memory it held, the failing openat behind an ENOSPC or EACCES, or the command a process ran right before it crashed, you attach a scoped monitor to the process tree instead of querying state. exectop follows one process tree through fork in the kernel and folds every command it ran into one row per kind:

yeet run gh:yeet-src/exectop -- --pid $(pgrep -n node)   # what this process tree actually ran, live

Where all of this sits among the other runtime debugging tools, and why a typed read-only graph beats a stream of terminal events for a production incident, is covered in the top runtime debugging tools comparison. What none of it does is show you a variable's value inside your application; for a reproducible application error that is still a debugger's job, and the standard stack-trace method is the right one.

Where does the record of an OOM kill, ENOSPC or EACCES actually live?

Every error above collapses into one table, because the reason the stack trace failed you is always the same: the record is not in the application, it is one layer down. Read across the row to see where to look instead.

Runtime error	What the app log shows	Where the real record is	How to read it live
OOM kill (SIGKILL, exit 137)	Nothing; the process vanished	Kernel OOM record, plus rising vmrss before it	journalctl -k after; yeet graph query on vmrss before
Segfault (SIGSEGV)	Nothing, or a truncated native frame	Kernel signal + a core dump if enabled	dmesg; watch the process's syscalls and maps live
Disk full (ENOSPC)	"No space left on device", no path	The failing write/openat and its target mount	Watch failed openat syscalls with the path attached
Permission denied (EACCES)	"Permission denied", no path	The failing openat and the exact denied path	Watch failed openat; check the four denial causes
Silent exit (no error)	The log simply ends	The kernel's view of the exit and last syscalls	exectop on the process tree; the exit is in the kernel

The pattern the table makes visible is the whole argument: an application-level tool reports what the application chose to log, and a process that was killed, faulted, or denied never chose to log the thing that ended it. The kernel is the one observer present for all five, which is why the production-honest move is to read from there rather than to add more application logging for an event the application will never reach.

How do I capture the record before the next crash (exit 137 and beyond)?

Put the kernel-side view in place before the incident, because the defining feature of these errors is that the evidence exists only at the moment they happen, and after the process is gone it is gone. The three steps, cheapest first: keep journalctl -k and dmesg within reach and know the exit-code-to-signal mapping so a 137 or a 139 reads instantly as a kill or a segfault rather than a mystery. For a recurring error you cannot yet catch, run a live read of the running system so the next occurrence is observed rather than reconstructed, watching vmrss for the OOM case or failing syscalls for the ENOSPC and EACCES cases. And for a process whose crash you need to explain after the fact, attach a scoped monitor like exectop to its tree so the commands and children it produced are on record independent of whether the process logged them.

None of this replaces a debugger for a reproducible application bug, and it is not meant to. It is the layer that answers the production runtime errors a debugger cannot reach, because you cannot attach one to a live service and the process you would attach to is already gone. Reading the kernel is not a workaround for the missing stack trace. On these errors it is the primary record, and the stack trace was the workaround.

Frequently asked questions

How do I debug a runtime error that has no stack trace?

Treat the missing trace as evidence rather than an obstacle. A runtime error with no application trace usually means the process was ended by the kernel or a fatal signal rather than by an exception your code could catch, since a signal like SIGKILL "cannot be caught, blocked, or ignored" and so runs no logging code. Look in the kernel log with journalctl -k or dmesg for a kill or OOM record, identify the process and the reason, then read the live system to see the conditions that produced it.

Why does my container get killed with exit code 137?

Exit code 137 is 128 plus 9, and signal 9 is SIGKILL, so a 137 exit means the process was killed by SIGKILL, most often by the out-of-memory killer when the container exceeded its memory limit. Check journalctl -k or dmesg for an OOM message naming the process, and compare the container's memory limit against its actual usage over time. The kill is the symptom; the memory growth before it is the bug to find.

Can I debug a production runtime error without stopping the service?

Yes, and for a live service you have to, because attaching a traditional debugger stops the process. Read the error from the kernel instead: the signal, the OOM record, the failing syscall and its errno are all observable from the kernel without touching the process, using journalctl -k for what already happened and an eBPF tool like yeet for what is happening now. Stopping the service to debug it is the outage you were trying to prevent.

What is the difference between a runtime error and a crash?

A runtime error is any error during execution, from a caught exception your code handles to a fatal condition that ends the process. A crash is the subset that terminates the process, whether from an uncaught exception, a fatal signal like SIGSEGV, or an external kill like the OOM killer's SIGKILL. The distinction matters for debugging because a caught runtime error leaves a stack trace and a crash from a signal usually does not, which changes where you look for the record.

How do I find which process is causing OOM kills?

Read the kernel log, which names the victim of every OOM kill: journalctl -k | grep -i "killed process" or dmesg | grep -i oom shows the process id and its memory usage at the moment it was chosen. To prevent the next one rather than explain the last one, watch resident memory per process on the live host over time, because the process that gets killed is not always the one that caused the pressure, and the growth pattern tells you which is which.

Sources

1. Linux signal(7) manual page, on SIGKILL and SIGSTOP being unable to be caught, blocked, or ignored, and on SIGSEGV as an invalid memory reference. https://man7.org/linux/man-pages/man7/signal.7.html
2. Linux errno(3) manual page, defining ENOSPC as "No space left on device" and EACCES as "Permission denied." https://man7.org/linux/man-pages/man3/errno.3.html
3. Linux kernel documentation on the out-of-memory killer selecting and killing a process under memory pressure. https://www.kernel.org/doc/gorman/html/understand/understand016.html

Related resources

1. Top 7 runtime debugging tools for Linux, where reading the kernel fits among the tools for debugging a running process. https://yeet.cx/topical-takes/runtime-debugging-tools-linux
2. Find which process is slowing your Linux machine, reading per-process kernel accounting without installing anything. https://yeet.cx/topical-takes/find-which-process-is-slowing-your-machine-linux
3. The kernel metrics your APM agent misses, on the errors and delays that never reach an application-level agent. https://yeet.cx/topical-takes/kernel-metrics-your-apm-agent-misses