How to Find Which Process Is Slowing Down Your Linux Machine

Necco Ceresani·August 25, 2026·34 min read

Fifteen years building engineering platforms, currently focused on advanced AI infrastructure at yeet. I love turning the deeply complex topics into something everyone can understand. I relate deeply with the core yeet philosophy that you can just build things.

Book a demo with Necco More from Necco →

schedstat context-switches proc-filesystem cpu-accounting profiling eBPF linux yeet

Quick answer. A process that feels like it is destroying your machine and a process that is actually consuming it look identical in top, and three files tell them apart. Read /proc/PID/schedstat for nanoseconds actually spent on CPU and nanoseconds spent waiting for a core, /proc/PID/status for the voluntary and nonvoluntary context switch counters, and /proc/PID/io for read syscalls against block-device bytes. A process with high voluntary switches, near-zero nonvoluntary switches and a run queue wait of 0.00 ms was blocked on the network, not competing for your CPU, and a faster machine would not have made it quicker. Start with cat /proc/$(pgrep -n node)/schedstat and read the second number.

I build kernel-side tools and watch engineers point them at things, which means I mostly see the moment after somebody has already decided what is wrong. The decision is usually made from a top reading and a fan. I have not measured your workstation and I am not going to tell you which process to kill, because the interesting part of this problem is that the instrument everyone reaches for first cannot answer the question being asked of it. A CPU percentage in top is a number about a sampling interval. "This process is why my machine feels slow" is a claim about causation. Those are different statements, and the counters that connect them have been in /proc since 2.6.23 and cost nothing to read.

What follows is one measurement pair I ran on 2026-08-25, on purpose, to see what the shape looks like in kernel accounting. Two node processes, same runtime, same 12-second window, on one Debian 13 arm64 Lima VM (kernel 6.12.101+deb13-cloud-arm64, 14 CPUs visible, 7938 MB RAM). One did request-then-read-then-parse work. One sat in an arithmetic loop. They land at opposite extremes on every counter below, and the split is what makes the counters diagnostic rather than trivia.

The routes for finding which process is slowing a Linux machine

Six places to stand, and they disagree about what "slow" means. The column that matters is whether the route can tell you a process was denied a resource rather than merely using one, because that is the difference between a machine that is too small and a machine that is fine.

Route	Install cost	Tells you time on CPU	Tells you time denied CPU	Per-process I/O attribution	Scope
`cat /proc/PID/{schedstat,status,io}`	none, already there	yes, nanoseconds	yes, `run_delay`	yes, syscall and block layer	one process, cumulative since start
`top` / `htop`	usually preinstalled	percentage over an interval	no	no	every process, live
`pidstat -w -d`	`sysstat` package	yes, plus switch rates	no	disk-layer only	per process, sampled per second
`schedlat`	one Python script	yes, as a percentage	yes, as a percentage	no	one process, live
`hotspot` (yeet + eBPF)	`curl -fsSL https://yeet.cx \| sh`	yes, per function	no, on-CPU only	no	one process, live, user and kernel frames
`perf sched`	`linux-tools` matched to kernel	yes	yes, per wakeup	no	every scheduler event, high overhead

Two things fall out of that table before any of the routes are worth explaining. The first is that the free route is the only one in the top half that answers the denied column, which is the column that kills the "I need a faster machine" theory. The second is that a CPU profiler appears in the table and does not answer it either, which is a structural limitation of sampling profilers rather than a knock on any particular one.

`cat /proc/PID/schedstat`, `status` and `io`: the counters that are already running

Nothing to install, and the kernel maintains these whether or not anyone reads them. /proc/PID/schedstat is three numbers, documented in the kernel's scheduler statistics page as "time spent on the cpu (in nanoseconds)", "time spent waiting on a runqueue (in nanoseconds)", and "# of timeslices run on this cpu". The second number is the one nobody reads, and it is the one that decides whether contention is real.

PID=$(pgrep -n node)
cat /proc/$PID/schedstat            # on-CPU ns, runqueue-wait ns, timeslices
grep ctxt_switches /proc/$PID/status
grep -E 'syscr|rchar|read_bytes' /proc/$PID/io

These are cumulative counters since the process started, so a single read tells you about the whole lifetime of the process and not about the last ten seconds. That matters more than it sounds: a process that spent its first minute compiling and has been idle since will look busy forever. Read them twice with a delay and subtract, which is what every measurement in this post does over a fixed 12-second window.

`top` and `htop`: the percentage that started the argument

Right about what it reports and wrong for the question, which is a worse failure than being inaccurate. top gives you CPU percentage per process over its refresh interval, sorted, with no notion of why a process was or was not running. It is the correct first look and a bad last one. Every publicly reported case of an AI coding agent eating a machine that I can find is argued at this layer: an open Claude Code issue reports 105 to 130 percent CPU from an idle instance on Ubuntu under WSL2 with terminal lag and constant fan activity, measured with top and ps aux. A dev.to writeup reports a load average of 122.91 on a ten-core machine and ten reparented zsh processes at roughly 60 percent CPU each, measured with ps. Both are real findings. Neither includes a single per-process kernel counter, and the second turns out to be orphaned shell busy-loops rather than the agent itself, which is exactly the attribution error the ps layer invites.

`pidstat -w -d`: switch rates per second, from the sysstat package

The closest thing to a shipped tool for the argument this post is making, and it is worth installing. pidstat -w reports the two context switch counters as rates, and the man page's own definitions are the clearest statement of the distinction anywhere in the tooling: cswch/s is "the total number of voluntary context switches the task made per second. A voluntary context switch occurs when a task blocks because it requires a resource that is unavailable", while nvcswch/s counts involuntary switches, which take "place when a task executes for the duration of its time slice and then is forced to relinquish the processor". pidstat -d adds disk I/O with an iodelay column in clock ticks.

What it does not give you is the run queue wait, and it reads at the disk layer rather than showing you the syscall-versus-block-device split. So pidstat will tell you a process is blocking, in rates, over an interval you choose. It will not tell you the process was never denied a core.

`schedlat`: the prior art for splitting a process's time three ways

Tanel Poder's schedlat is the closest thing to this post's method as a running tool, and it predates it by years. It reads /proc/PID/schedstat once per second and reports %CPU as "percent of time the task spent running on CPU", %LAT as "percent of time the task spent trying to get onto CPU (in runqueue)", and %SLP as the remainder, which is time the task was "not on CPU, not in runqueue, thus sleeping/waiting". That third bucket is the whole answer for a blocked process, arrived at by subtraction, from the same two counters this post reads by hand. If you want this as a live percentage rather than a delta you compute yourself, use it.

`hotspot`: an eBPF sampling profiler, for after the counters say the process is on CPU

hotspot is an eBPF sampling CPU profiler for Linux: pick a process out of a live table, click it, and watch a flat self-time profile build with user and kernel frames side by side. It arms a perf_event BPF program at 499 Hz per CPU scoped to the process's cgroup, so worker threads are covered, and it symbolizes through the live process maps rather than a capture file. No linux-tools package matched to the running kernel, no perf record, no post-processing step.

It is also the wrong first instrument for the question in this post's title, and its README says so plainly in what it can't see: "On-CPU time only. A blocked process produces no samples. Time waiting on I/O, a lock, a futex or the scheduler is invisible, so a process that's slow because it's waiting looks idle." That sentence is the entire reason the counters come first. Point a sampling profiler at a process that is parked on a socket and you get an empty profile, which reads as "nothing here" when the truth is "nothing here on CPU". Read schedstat first, and reach for hotspot once the first number is large enough to explain what you are feeling.

`perf sched`: every scheduler event, when you need the wakeups themselves

The heavyweight option, and the one to reach for when you need to know which task preempted which and when, rather than how much time a process lost in total. perf sched timehist separates wait time before a wakeup from the scheduler delay between wakeup and running, which is finer-grained than run_delay can be because run_delay is a single accumulated total. The cost is real and Brendan Gregg states it directly: these events "can be very frequent, millions per second, costing CPU, memory, and disk overhead to record", and one second of recording produced 1.9 MB and 13,502 samples in his example. It also needs a linux-tools build matched to your running kernel, which a minimal or slightly-behind image does not have.

My node process shows 5 percent CPU but the machine feels sluggish. What is it doing?

Waiting, almost certainly, and the counters say so in a way a percentage cannot. Here is the request-wait-parse workload over a 12-second window on the Debian 13 arm64 VM described above, read from the kernel's per-process accounting rather than from top:

pid 628750  comm node  threads 11
vmrss 62068 kB   rssanon 18348 kB
cpu                    616.89 ms   =  5.14% of one core
runqueue wait            0.00 ms
voluntary ctxt switches    320
nonvoluntary ctxt switches   0
syscr 1646   syscw 442   rchar 753666 bytes
read_bytes 0  write_bytes 0

616.89 ms of CPU in 12 seconds is 5.14 percent of a single core, on a box with 14 of them. The process was on CPU for about one twentieth of one fourteenth of the machine's capacity. Whatever is making the machine feel slow, this accounting says this process was not doing it by consuming CPU, and the run_delay of 0.00 ms says it was not being starved either. It asked for a core 320 times and got one every time, immediately.

The shape of the work is legible in the same numbers, which is the part that makes this worth reading rather than just reassuring. 320 voluntary context switches with zero involuntary ones is a process that blocked, yielded, was woken, ran briefly, and blocked again, 320 times. 1,646 read syscalls moving 753,666 bytes is a lot of small reads. Neither of those is CPU work. The vmrss of 62 MB, of which 18 MB is anonymous, is a resident set nobody would notice on a 7938 MB host.

One honest boundary before anyone quotes this: no AI coding agent produced these numbers. This is a synthetic node workload written to have an agent turn's shape, which is an HTTPS request to a remote API, then three small file reads, then a roughly 3-million-iteration arithmetic burst, looped. It is one run of one workload on one host, with no repetitions and no variance, so treat it as a single honest observation of what request-wait-parse looks like in kernel accounting and not as a measurement of any product. What transfers is the counter signature, not the values.

What does the difference between voluntary and nonvoluntary context switches actually tell me?

It tells you whether a process is waiting or competing, and it is the only pair of counters that separates those two mechanically. A voluntary switch means the task gave up the CPU itself because it had nothing to do until something else happened. A nonvoluntary switch means the scheduler took the CPU away because the task's timeslice expired and someone else wanted it. kernel-internals.org puts the mechanism in one line each: a voluntary switch is when "the task called schedule() directly because it has nothing to do, blocked on I/O, sleeping, waiting for a lock", and an involuntary one is when "the scheduler preempted the task, its timeslice expired or a higher-priority task woke up".

Every source that defines these counters is correct and almost none of them use the pair to decide anything. Here is the same runtime, the same 12-second window and the same host, running a pure arithmetic loop with no network and no file I/O:

pid 628968  comm node  threads 7
vmrss 54200 kB   rssanon 13948 kB
cpu                  12013.64 ms   = 100.11% of one core
runqueue wait            2.35 ms
voluntary ctxt switches      0
nonvoluntary ctxt switches  65
syscr 0   syscw 0   rchar 0

Put the two side by side and the counters stop being definitions.

	request-wait-parse workload	CPU-bound control
CPU in a 12 s window	616.89 ms (5.14% of one core)	12013.64 ms (100.11% of one core)
Run queue wait	0.00 ms	2.35 ms
Voluntary context switches	320	0
Nonvoluntary context switches	0	65
Read syscalls	1646	0
Bytes through the syscall layer	753666	0

Same language, same runtime, same window, same machine, and roughly a 19x difference in CPU consumed. The two switch counters are at perfect opposite extremes: 320 and 0 for the process that spends its life blocking, 0 and 65 for the process that always has work and keeps losing the CPU. Nothing about the workload had to be known in advance to read that. The ratio is the diagnostic, and the reading generalizes: a high nonvoluntary count relative to voluntary means the task always has work to do and keeps getting preempted, which is a machine with too much runnable work on it. A high voluntary count with almost no nonvoluntary ones means the task keeps stopping to wait, which is a machine that has capacity to spare and a process that is not using it.

The reason a percentage cannot make this call is that both of these processes can produce a modest number in top depending on when you look and how many cores you divide by. 100.11 percent of one core is 7 percent of this 14-CPU box. A process pinning a core and a process blocking on sockets can render the same tidy figure, and the counters are what disambiguate them.

Will a faster CPU make my coding agent faster?

Not if run_delay reads zero, and that is the most useful thing on this page. The second field of /proc/PID/schedstat is nanoseconds this task spent on a run queue, runnable, wanting a CPU and not getting one. It is the direct measure of "the machine was too busy for me". On the request-wait-parse workload it was 0.00 ms across the entire 12-second window, meaning the process was never once made to wait for a core. Adding cores, removing neighbors, or buying a faster machine gives a process with zero run queue wait nothing it did not already have.

Note the direction of that inference carefully, because it is easy to over-claim. Zero run queue wait refutes "this process was starved of CPU". It does not prove the process was fast, or that nothing else on the box was starved, or that the user-visible slowness was imaginary. What it does is remove one specific and very commonly assumed cause, cheaply, before anyone spends money. The control process is the useful contrast again: even a workload pinning a core produced only 2.35 ms of run queue wait on an otherwise idle 14-CPU host, because there was nothing to compete with. Run queue wait is a contention measurement, so it reads near zero on a quiet machine regardless of what any single process is doing, and it climbs when runnable work exceeds cores.

So the sequence that answers the money question is short. Read run_delay; if it is near zero, contention is not your problem and the next question is what the process is waiting for. If it is large relative to the window, you have real contention, and at that point run queue latency profiling on a production host is a different investigation than this one. The kernel metrics an APM agent misses covers that route for a service with an SLO.

How do I tell if a process is blocked on I/O or fighting for CPU?

Read /proc/PID/io and compare the syscall-layer fields against the block-device fields, per process, with no dashboard involved. The kernel documents the distinction in the field definitions themselves: rchar is "the number of bytes which this task has caused to be read from storage. This is simply the sum of bytes which this process passed to read() and pread()", syscr is an "attempt to count the number of read I/O operations, i.e. syscalls like read() and pread()", while read_bytes is an "attempt to count the number of bytes which this process really did cause to be fetched from the storage layer. Done at the submit_bio() level, so it is accurate for block-backed filesystems".

Those two layers are supposed to disagree, and the mechanism for why is already written up field by field in the kernel metrics your APM agent misses, so it does not need re-arguing here. What matters for one process on your own machine is what the gap lets you rule out. On the request-wait-parse workload, read_bytes was 0 while rchar was 753666 across 1646 read syscalls. Three quarters of a megabyte requested, 1,646 times asked for, and not one byte fetched from a block device. Those reads were served without touching the disk, so a disk-latency theory of the slowness is dead from the same sample that produced the CPU numbers, and it took no additional measurement to kill it.

That is the move the host-level route cannot make. %iowait in top or vmstat is a whole-host figure with no owner attached, and the standard iowait recipe walks you from top to vmstat to iotop to iostat, which ends at per-process disk activity and per-device latency. Useful, and it never distinguishes read syscalls from block-device reads, so it cannot tell you the difference between a process hammering the page cache and a process waiting on a spindle. Those look nothing alike to the machine and identical in an I/O percentage.

A three-line check, per process, that separates the three cases:

PID=$(pgrep -n node)
awk '{printf "on-cpu %.2f ms  runqueue %.2f ms\n", $1/1e6, $2/1e6}' /proc/$PID/schedstat
grep -E 'voluntary_ctxt|nonvoluntary_ctxt' /proc/$PID/status
grep -E '^(syscr|rchar|read_bytes)' /proc/$PID/io

Read it as a decision table rather than three facts.

What the counters show	The process is	What to do next
High on-CPU, nonvoluntary switches climbing, `run_delay` climbing	competing for CPU on a busy box	reduce runnable work, or profile it with `hotspot` or `perf`
High on-CPU, nonvoluntary switches climbing, `run_delay` near zero	doing real CPU work, uncontended	profile it, the time is real and in your code
Low on-CPU, voluntary switches high, `read_bytes` zero	blocked on network or on cached reads	look at what it is waiting for, not at the CPU
Low on-CPU, voluntary switches high, `read_bytes` large	blocked on actual disk	`iostat`, `biolatency`, the storage path
Low on-CPU, `run_delay` large	starved, not slow	something else on the box is the problem

Why doesn't a CPU profiler show me anything for a process that feels slow?

Because sampling profilers interrupt the CPU and record what is executing, and a blocked thread is not executing. This is not a sampling-rate problem that a higher frequency fixes; it is structural. At 499 Hz per CPU, a process sitting on a socket read produces zero samples per second, the same as a process doing nothing at all, because from the sampler's point of view those are the same state. hotspot's README states the consequence as a limit rather than burying it: "a process that's slow because it's waiting looks idle."

That is why the ordering in this post is counters first, profiler second, and why an empty profile is a result rather than a failure. An empty profile plus 320 voluntary context switches is a positive finding: the process is off CPU and the time is going somewhere the profiler cannot see. An empty profile with no counters read alongside it is just confusing, and it is the point where people usually conclude the tool is broken.

Once the counters say a process really is burning CPU, a profiler is exactly right and the counters are exhausted. run_delay near zero with hundreds of milliseconds of sum_exec_runtime per second of wall clock means the time is real and in the code, and the next question is which function, which no /proc file can answer. That is the handoff: /proc decides whether there is CPU time worth attributing, and the profiler attributes it.

Can I read schedstat and the context switch counters without installing anything?

Yes, and this is the property that makes the route usable on a box you do not control. Every number in this post came out of /proc, which the kernel maintains as ordinary accounting whether or not anyone reads it. There is no probe to attach, no module to load, no package matched to the running kernel, and nothing to remove afterwards. cat and awk are enough, and the two schedstat fields have been there since well before any kernel you are likely to be running.

Worth being precise about the cost of the measurement rather than making a claim I cannot support: reading these files is two reads of counters the kernel already maintains, which is a different kind of operation from tracing scheduler events. That contrast is the honest one, and Gregg's figure above is the reference point: recording scheduler events at millions per second costs CPU, memory and disk, which is why in-kernel aggregation exists. Reading an accumulated total does not, because the accumulation already happened. I have not published a measured overhead number for the reading itself, and a properly measured one would need per-thread accounting across a daemon's tasks rather than a single main-pid sample.

The trade is that cumulative counters describe a whole process lifetime, not the ten seconds you care about. Read them twice around a fixed window and subtract, and record the window with the numbers, which is what makes a reading comparable to anyone else's. Any number in this post without its 12-second window attached is meaningless.

Which process is actually slowing my machine, if not the one at the top of top?

The honest answer is that these counters identify the wrong suspects far better than they nominate the right one, and that is still most of the work. Ruling a process out costs three cat commands and takes a theory off the table permanently, which for a performance investigation is the cheap half. What they will not do is scan the box for you: /proc/PID/* is per process and you have to name a PID, so a machine-wide answer means iterating, and at that point top sorted by CPU is the right way to get the candidate list.

Three suspects worth reading before concluding the process you launched is guilty, based on what the public reports of this problem actually turned out to be. An editor or IDE renderer process, which in one closed Claude Code issue the reported process was a VS Code "Code Helper (Renderer)" at 99.8 percent CPU for 211 minutes on macOS, measured in Activity Monitor, rather than the agent binary itself. That report is macOS, so the counters in this post do not apply to it directly; the transferable part is which process the number belonged to. Orphaned children reparented to PID 1, which is what the dev.to load-average-of-122 case turned out to be, and which ps -Ao pcpu,pid,ppid,user,comm -r surfaces by showing you a PPID of 1 next to a busy loop. And a file watcher or language server, which is a separate process from the thing that started it and gets blamed by proximity.

The pattern in all three is that the process you are angry at is not the process on the CPU, and per-process counters are how you check rather than assume. Read them for each candidate rather than for the one you already suspect.

What can per-process kernel accounting not tell me?

Their limits are sharp and worth stating at the point you would rely on them, because each one has a different next step.

No function-level attribution. sum_exec_runtime says 616.89 ms of CPU happened. It cannot say in what. That needs a sampling profiler such as hotspot, perf, or a language-level tool like py-spy or async-profiler for interpreter frames.
No reason for a block. 320 voluntary context switches say the process yielded 320 times. They do not say whether it was a socket, a lock, a futex or a page fault. Scheduler-tracepoint tooling gets the blocking stack; the counters get only the count.
Cumulative since process start, so a single read describes a lifetime and not a moment. Two reads and a subtraction, with the window recorded.
run_delay is a whole-host contention signal in disguise. It reads near zero on an idle machine no matter what one process does, so a zero is only informative about starvation, never about speed.
Write-side I/O attribution is weak. write_bytes is counted at page-dirtying time and writeback happens later in a kernel thread, so the process credited with a block write is often not the one that dirtied the page. Read-side attribution is the reliable half.
VmRSS and RssAnon are per process, not per cgroup. The man page flags both as inaccurate and points at /proc/pid/statm, and container-level memory is a different accounting question worked through in why a container uses more memory than the process inside it.
Nothing about what the process did. These counters are quantity, not identity. Which files an agent opened and which commands it ran is a separate route entirely, covered in auditing what an AI coding agent actually ran.

The bottom line: read run queue wait before you buy a faster machine

top tells you a process is using CPU. /proc/PID/schedstat tells you whether it was ever denied CPU, and those are different claims with different fixes. A process at 5.14 percent of one core with 320 voluntary context switches, zero nonvoluntary ones and 0.00 ms of run queue wait was blocked and yielding, not competing, and no amount of hardware helps it. The CPU-bound control in the same runtime and window sat at the opposite extreme on every one of those counters, at 100.11 percent of a core with 0 voluntary and 65 nonvoluntary switches, which is what makes the split a signature rather than a definition.

Reach for cat on the three /proc files first, on any box, including one you do not own. Add pidstat -w -d from sysstat when you want the switch counters as live rates, or Tanel Poder's schedlat when you want on-CPU, runqueue and sleeping as three percentages without doing the arithmetic. Move to hotspot once the counters confirm there is CPU time worth attributing to a function, and to perf sched when you need individual wakeups rather than accumulated totals and can afford the overhead and the matching linux-tools. Keep iotop and iostat for the case where read_bytes comes back large, which is the case where the disk actually is involved.

And keep the discipline that makes any of it quotable: state the window, state the host, and state which counter you read. A percentage without an interval and a machine behind it is how this argument gets started in the first place.

Frequently asked questions

Do I need root to read the schedstat file for a process?

No, for your own processes. The schedstat, status and io files are readable by the owning user, so reading a process you launched needs no privileges at all. Reading another user's process is where restrictions apply, and on some distributions /proc/PID/io is more tightly restricted than the others because byte counts can leak information about what a process is doing.

Are schedstat and the context switch counters available in a container?

Yes, and they describe the process rather than the container. A process inside a container reads its own /proc/PID/schedstat normally, and the numbers are that task's own accounting. What changes is the surrounding context: run queue wait inside a CPU-quota-limited cgroup can climb because of throttling rather than because other processes are competing, so a large run_delay in a container has a second possible cause that the same number on a bare workstation does not.

Why does my process show more than 100 percent of one core?

Because the counter sums across threads. sum_exec_runtime accumulates CPU time for the task, and a multi-threaded process running on several cores at once accumulates faster than wall clock. A value of 100.11 percent of one core over a window means roughly one core's worth of CPU was consumed, whether by one thread pinned to a core or several threads sharing the work.

What is the difference between run queue wait and iowait?

They measure opposite states. Run queue wait is time a task was runnable and wanted a CPU but did not get one, which means the machine had more runnable work than cores. Iowait is time a CPU sat idle with at least one task blocked on I/O, which means the machine had capacity and something was waiting on a device. High run queue wait means too little CPU; high iowait means too little I/O throughput, and a task blocked on the network shows up in neither.

Does a high context switch count mean I have a performance problem?

Not on its own, and treating switches as a number to minimize is the common misreading. A process making 320 voluntary switches in 12 seconds is doing exactly what a network-bound process should do, which is get off the CPU while waiting. The count is a description of the workload's shape. What is worth investigating is a change in the ratio between voluntary and nonvoluntary, or nonvoluntary switches climbing on a process you expected to be idle.

Can I read schedstat for a process that already exited?

No. Everything under /proc/PID/ disappears when the process does, so a post-mortem needs something that was recording at the time. This is the argument for reading counters on a long-running process you are suspicious of rather than waiting for it to finish, and for process accounting or an audit trail if you need the history.

Why is my run queue wait large when the CPUs look idle?

Usually a boundary rather than load. A cgroup CPU quota, CPU affinity pinning that confines a task to cores that are busy while others sit free, or a NUMA layout keeping work away from available cores all produce runnable-but-not-running time on a machine with visible idle capacity. Check the cgroup's CPU limits before concluding the scheduler is at fault.

Does the arithmetic burst in the test workload matter to the result?

It is why the request-wait-parse process registered any CPU at all. The workload loops an HTTPS request, three small file reads, then roughly 3 million arithmetic iterations, so the 616.89 ms is mostly that burst plus syscall overhead. Removing it would push CPU lower and the voluntary switch count would stay high, which sharpens the signature rather than changing it.

What kernel version do I need for schedstat and run_delay?

Anything current, and much older. The voluntary and nonvoluntary context switch counters in /proc/PID/status have been present since Linux 2.6.23, RssAnon since 4.5, and the schedstat and io files predate both. The measurements here ran on 6.12.101+deb13-cloud-arm64. The kernel's own schedstat documentation covers version 17 of the format, which changed load-balancer fields in the domain statistics rather than the three per-task fields this post reads.

Is this a replacement for a continuous profiler or an APM?

No, and it does not overlap with one. There is no retention, no query language, no aggregation across hosts, no alerting and no history: every number here is one process on one box, read now, gone when the process exits. What it replaces is the guess you would otherwise make before deciding which of those tools to open, and on a workstation with neither installed, it is the whole investigation.

Sources

Linux kernel scheduler statistics documentation (kernel.org, 2026): the authoritative definition of the three /proc/<pid>/schedstat fields used throughout, namely "time spent on the cpu (in nanoseconds)", "time spent waiting on a runqueue (in nanoseconds)" and "# of timeslices run on this cpu". Also documents the domain-level statistics in /proc/schedstat and the current schedstat version 17, which removed lb_imbalance in favor of lb_imbalance_load, lb_imbalance_util, lb_imbalance_task and lb_imbalance_misfit, and added domain name printing; those changes affect the load-balancer fields rather than the three per-task fields read here.
Linux kernel proc filesystem documentation (kernel.org, 2026): the field definitions behind the I/O half of the argument. rchar is "simply the sum of bytes which this process passed to read() and pread()"; syscr is an "attempt to count the number of read I/O operations"; read_bytes counts bytes "really did cause to be fetched from the storage layer. Done at the submit_bio() level, so it is accurate for block-backed filesystems"; write_bytes is counted "at page-dirtying time", which is why write attribution is weaker than read attribution; cancelled_write_bytes covers bytes a process "caused to not happen, by truncating pagecache". Also documents VmRSS as the sum of RssAnon, RssFile and RssShmem.
proc_pid_status(5) man page (man7.org, 2026): the availability floor for the two counters this post's argument rests on, stated as "number of voluntary and involuntary context switches (since Linux 2.6.23)". Also dates RssAnon to Linux 4.5 and flags both VmRSS and RssAnon as inaccurate, pointing at /proc/pid/statm, which is why the memory numbers here are reported as context rather than as the finding.
pidstat(1) man page (man7.org, 2026): the clearest tooling-side statement of the voluntary/nonvoluntary distinction, and the shipped alternative to reading /proc by hand. cswch/s counts voluntary switches, occurring "when a task blocks because it requires a resource that is unavailable"; nvcswch/s counts involuntary switches, taking place "when a task executes for the duration of its time slice and then is forced to relinquish the processor". -d adds per-process disk I/O with kB_rd/s, kB_wr/s, kB_ccwr/s and an iodelay column in clock ticks. No run queue wait column.
schedlat: measuring CPU scheduling latency on Linux (Tanel Poder, 2026): the nearest prior art and the reason this route is not novel: a Python script reading /proc/PID/schedstat once per second to report %CPU as "percent of time the task spent running on CPU", %LAT as "percent of time the task spent trying to get onto CPU (in runqueue)", and %SLP as the remainder, time when the task is "not on CPU, not in runqueue, thus sleeping/waiting". The three-way split by subtraction is the live version of the deltas computed by hand in this post.
Context switches in the Linux scheduler (kernel-internals.org, 2026): the mechanism behind the two counters. A voluntary switch is when "the task called schedule() directly because it has nothing to do, blocked on I/O, sleeping, waiting for a lock"; an involuntary one is when "the scheduler preempted the task, its timeslice expired or a higher-priority task woke up". States the diagnostic reading directly: "a high involuntary count relative to voluntary means the task is being preempted frequently, it always has work to do but keeps losing the CPU to other tasks", and shows the dequeue path taken when a blocking task yields.
perf sched for Linux CPU scheduler analysis (Brendan Gregg, 2017): the overhead reference point for tracing scheduler events rather than reading accumulated counters: these events "can be very frequent, millions per second, costing CPU, memory, and disk overhead to record", with one second of recording producing 1.9 MB and 13,502 samples. Also the finer-grained alternative to run_delay, since perf sched timehist separates wait time before a wakeup from the scheduler delay between wakeup and running.
Claude Code issue 19393: high CPU while idle (GitHub, open): evidence that the felt-slowness complaint is real and that it is argued entirely at the top layer. Reports two processes at 105 to 130 percent and 59 to 70 percent CPU sustained while idle and waiting for input, on Ubuntu under WSL2 with Node v22.20.0 and 24GB RAM, with terminal unresponsiveness, typing lag and constant fan activity. Measured with top and ps aux; no per-process kernel accounting anywhere in the thread.
Claude Code issue 11615: renderer process at 99.8 percent CPU (GitHub, closed as not planned): the misattribution case: the process at fault was VS Code's "Code Helper (Renderer)" at 99.8 percent CPU for 211 minutes with about 1.03 GB resident, on an M3 Pro MacBook Pro running macOS 15.6.1, affecting both VS Code and Cursor 2.0.75. Measured with Activity Monitor and ps aux. Cited for the pattern that the busy process is often adjacent to the one being blamed, not for any Linux kernel counter.
Claude might be saturating your machine (dev.to, 2026): the closest incumbent framing of this question, and a good illustration of what the ps layer can and cannot settle. Reports a load average of "122.91 167.84 162.08" on a ten-core machine, ten zsh processes at roughly 60 percent CPU each with PPID 1, an elapsed time of nearly two days, and 850.3 percent total CPU across 12 processes. The cause turned out to be orphaned shell busy-loops rather than the agent, and the recommended detection is ps -Ao pcpu,pid,ppid,user,comm -r. No /proc per-process accounting.
How to troubleshoot high I/O wait on Linux (Site24x7, 2026): the incumbent iowait recipe this post's /proc/PID/io route sits beside: top to spot the iowait percentage, vmstat for memory context, iotop -o for per-process disk I/O including the percentage of time a process spent doing I/O, then iostat per device. Ends at the disk layer; does not mention the page cache and never distinguishes read syscalls from block-device reads, so it cannot separate a process hammering cache from one waiting on a device.
hotspot (yeet-src, 2026): the eBPF sampling profiler for the case the counters hand off to, and the source of this post's clearest limit statement. Arms a perf_event BPF program at 499 Hz per CPU scoped to the process's cgroup, symbolizing through live process maps with no capture file; 499 is prime so sampling never phase-locks with kernel ticks. Its what it can't see section states the constraint that orders this whole post: "On-CPU time only. A blocked process produces no samples... so a process that's slow because it's waiting looks idle."

Related resources

Top 5 kernel metrics your APM agent misses: the production version of this question, for a service with an SLO and a real confirmed latency problem, including the page-cache layer disagreement worked through in full.
Why is my container using more memory than the process inside it?: the same method applied to memory, where the per-process and per-cgroup numbers disagree by design.
How to audit what an AI coding agent actually ran on Linux: the identity question rather than the quantity question: which commands ran and which files were touched.
How to sandbox an AI coding agent on Linux: what to do once you have decided the problem is not resource consumption but permission.
Why is my Redis slow?: another case where the obvious instrument answers a different question than the one being asked.
yeet documentation: reading the same /proc counters as a typed, queryable graph, and writing a probe when the counters run out.

Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.