Top 5 Kernel Metrics Your APM Agent Misses
Fifteen years building engineering platforms, currently focused on advanced AI infrastructure at yeet. I love turning the deeply complex topics into something everyone can understand. I relate deeply with the core yeet philosophy that you can just build things.
Last updated: August 2026
Quick answer. Your agent collects what your application reports and what the network stack counts in aggregate. It does not collect the time your threads spent blocked rather than running, which process owned a retransmitted segment, the shape of your disk latency distribution rather than its mean, how long a runnable thread waited for a core, or the difference between the bytes your application requested and the bytes that reached the device. Five kernel-side answers, ranked below by how often each one ends an investigation rather than extending it. The fastest way to get the first one is
offcputime-bpfcc -p $(pgrep -n yourapp) 30.
I build kernel-side tools for a living and watch engineers use them, which mostly means watching what they reach for after the dashboard has already failed them. I do not run your fleet, I have never had to justify your agent's per-host price to a finance team, and I would not tell you to remove it. What I see instead is a specific pattern in the questions that arrive: they are almost never about a metric that is missing from a dashboard. They are about a number that is present, correct, and unable to explain the thing it is measuring. A p99 of 400ms with a flat CPU profile is not a gap in coverage. It is a measurement taken at a layer where the answer does not live.
The ranking below is by how often each question ends an investigation rather than extending it. That is a judgment from watching people use these tools, not a benchmark, and I have said what each one is based on so you can disagree with the order and still use the list. What the five have in common is that no amount of budget on the tools you already run produces them, because each one is a fact the kernel holds and never reports upward on its own.
What types of metrics does an APM agent collect, and what does it miss?
An APM agent collects four families well, and none of them describe time your process spent not running. It collects request-level metrics from your traces, runtime metrics from the language VM, host metrics scraped off the system, and whatever custom metrics you submit. Those cover the request path and the machine's aggregate state, which is most of what you need most of the time. What falls outside them is not a coverage gap a vendor could close in a release; it is a consequence of where the agent stands.
| Metric family | Examples an agent collects | What that family cannot tell you |
|---|---|---|
| Request and trace metrics | trace.<SPAN_NAME>.hits, .errors, .duration and .apdex, tagged by env, service, resource and http.status_code | Where the time inside a slow span went, when the span is slow because the thread was blocked rather than working |
| Runtime metrics | jvm.heap_memory, jvm.gc.major_collection_time, runtime.go.num_goroutine, runtime.node.event_loop.delay.avg | Anything below the language VM: the scheduler, the block layer, the socket. The runtime reports its own bookkeeping, not the kernel's |
| Host metrics | system.cpu.iowait, system.load.1, system.mem.used, system.io.await, system.io.r_s | Which process or request caused any of it. These are host-wide aggregates with no owner attached |
| Custom metrics | Counts, rates and gauges you submit, plus histograms aggregated agent-side and distributions aggregated server-side | Anything you did not already know to instrument, which is the usual situation when the cause is new |
The line falls where it does for a structural reason. An agent reports what the application tells it, what the runtime exposes, and what the kernel already counts in /proc. Everything below that line requires attaching to a kernel event and doing arithmetic at the point the event fires, which is a different mechanism and not something a vendor withholds to sell you an upgrade.
Three categories sit below it. The first is time your process was not running. Sampling profilers interrupt the CPU and record what is executing. A thread blocked on a lock, a page fault, or a socket read is not executing, so it appears in no sample, at any sampling frequency. This is not a resolution problem that a higher rate fixes.
The second is per-event attribution for things the kernel counts in aggregate. Your host has counted 4,412 retransmitted segments since boot. That counter is a single integer with no owner attached, and by the time you read it the socket that caused the increment may not exist. Getting an owner means being present when the counter increments.
The third is distribution shape where a mean is reported. system.io.await and node_exporter's node_disk_io_time_seconds_total both give you an average, and an average over a bimodal distribution describes neither mode. Bimodal is the normal state of a disk serving both cache-warm and cache-cold traffic. Datadog's own metric types make the distinction available, since a distribution sends raw values and aggregates server-side, but that only helps for values you submit yourself, not for kernel events nobody is measuring.
Worth being fair about what the agent does better, because the reason to keep it is real: retention, fleet aggregation, a query language, and traces that follow a request across service boundaries. Nothing below replaces any of those. The five items are each about one host, right now, and about a fact that has to be captured at the moment it happens or not at all.
1. Off-CPU time: p99 latency high but traces look fine
The time went off-CPU: your threads spent it blocked and not running, and both instruments you checked are structurally incapable of showing that. The bcc tool that measures it directly is offcputime, and its own man page states the relationship plainly: it records "stack traces and task names that were blocked and 'off-CPU', and the total duration they were not running: their 'off-CPU time'", and describes itself as "complementary to CPU profiling (e.g., CPU flame graphs) which shows the time spent on-CPU". Complementary is doing real work in that sentence. It means the two views do not overlap, so a flat on-CPU profile is not evidence that nothing is slow. It is evidence that whatever is slow is not burning CPU.
This ranks first because it is the single most common shape of an investigation that has stalled. The reader has traces on, the spans do not sum to the latency users report, the profile shows nothing dominant, and they are out of instrumented places to look. Off-CPU time is not a place they have looked and rejected. It is an axis that most ranked answers to this question never mention. The standard prescription, which you will find at the top of the results for any tail-latency query, is distributed tracing plus continuous profiling plus high-cardinality analysis. That advice is what already failed, because continuous profiling in practice means on-CPU sampling.
The blocking reasons offcputime covers are broad: the man page names "disk I/O, network I/O, locks, page faults, involuntary context switches, etc." That breadth is why the tool is a good first move rather than a hypothesis-confirming one. You do not need to guess whether it is lock contention or a page fault before you measure; the stack trace tells you which.
What the kernel sees that your agent doesn't. The scheduler knows the exact nanosecond a thread stopped running and the exact nanosecond it resumed, along with the kernel stack that led to the block. Nothing in userspace has both halves. The runtime knows it called read() and that read() eventually returned, and the difference between those is where your latency is hiding.
Best for. A p99 or p99.9 regression with a flat profile, a bimodal latency distribution, or any case where the spans in a trace do not add up to the wall-clock time the user experienced.
What it costs you. Real overhead, and the man page is direct about it: scheduler events "can exceed 1 million events per second, and so caution should still be used. Test before production use." The mitigation is built in. Raise the MINBLOCK_US tunable so short blocks are filtered in the kernel and never reach userspace, which is the right trade when you are hunting a tail event anyway. You also get kernel stacks that need symbols, so a stripped binary gives you the same [unknown] frames it gives perf.
How to start.
# Off-CPU time for one process, 30 seconds, blocks longer than 1ms only.
# -p limits to one PID; -m 1000 sets the minimum block in microseconds,
# which drops the high-frequency short blocks that cause the overhead.
offcputime-bpfcc -p $(pgrep -n yourapp) -m 1000 30
2. TCP retransmit attribution: which process is causing retransmissions on Linux
Neither ss nor netstat -s will tell you, and the tool everyone recommends for this has a caveat in its own documentation that most recipe posts omit. netstat -s gives you a host-wide count of segments retransmitted with no owner. ss -ti gives you per-socket retransmit counters, which is genuinely closer, but only for sockets that still exist when you run it, and a connection that already failed and closed is gone.
The standard next recommendation is bcc's tcpretrans, which prints a PID column and looks like exactly the answer. Read the man page before you trust it. The column is documented as the "Process ID that was on-CPU", and the page adds the caveat directly underneath: "This is less useful than it might sound, as it may usually be 0, for the kernel, for timer-based retransmits."
That single sentence turns a widely repeated recommendation into a known-wrong one, and it ranks second here because the failure is silent. You run the tool, you get a table with a PID column populated with zeros, and if you have not read the man page the natural conclusion is that the kernel is doing something odd rather than that the column cannot answer your question. The mechanism is not mysterious. A timer-based retransmit fires from a kernel timer context, so the process on-CPU at that instant is whatever happened to be scheduled, which is usually nothing.
The attribution you actually want comes from the socket, not from the scheduler, which is the same vantage-point argument that separates a kernel view from a proxy's. The 4-tuple on the retransmitted segment is reliable, and mapping a local port back to an owning process is a lookup you can do at capture time. That is the shape of a probe worth writing rather than a tool worth installing: hook tcp_retransmit_skb, read the socket's addresses and ports out of the sock struct, and resolve the owner from the port rather than from the PID the kernel hands you.
What the kernel sees that your agent doesn't. The socket at the moment of retransmission, including the 4-tuple, the connection state, and the accumulated retransmit count for that specific connection. Your agent sees a host-level counter. Datadog's Cloud Network Monitoring does collect network data via eBPF and identifies "TCP failures including resets, refusals, and timeouts", which is real coverage, and it also documents a floor: Linux kernel 4.4.0+ or backported eBPF features. Below that line you are not getting eBPF collection at all.
Best for. A host where retransmit counters are climbing, multiple services share the NIC, and you need to know which team to page before you can do anything about it.
What it costs you. Retransmits are rare events, so the capture cost is genuinely low. The honest limits are elsewhere. The PID column is unreliable for the reason above, so you write the port-to-process lookup yourself. And a retransmit is a symptom that gets misread constantly: a moderate retransmit rate on a healthy connection is normal TCP behavior doing its job, not a defect to chase.
How to start.
# Per-socket retransmit counts for sockets that currently exist.
# -t restricts to TCP, -i shows the internal TCP info including retrans.
ss -ti state established | grep -B1 retrans
# Live retransmit events with the 4-tuple. Read the PID column with the
# man page's caveat in hand: it is usually 0 for timer-based retransmits.
tcpretrans-bpfcc
3. Per-disk io latency histograms: node_exporter shows an average, writes still stall
Bimodal, almost certainly, and the average is the problem rather than the measurement. A disk serving a mix of cache-warm reads and cache-cold writes produces two clusters, one in the tens of microseconds and one in the tens of milliseconds. The mean of those two lands in a gap where no actual request lives, and it moves when the ratio between the modes changes even though neither mode moved.
The metric shape that answers this is a per-disk latency histogram with buckets, and Cloudflare's ebpf_exporter is the canonical implementation. It ships a block IO example that "attaches to block io subsystem and reports disk latency as a prometheus histogram, allowing you to compute percentiles", emitting series like ebpf_exporter_bio_latency_seconds_bucket{device="nvme0n1",operation="write",le="0.000128"}. Note what is in those labels: the device and the operation, separately. A single averaged number collapses both dimensions, so a write regression on one device disappears into read traffic on another.
This ranks third rather than higher because the fix is often a change to what you already scrape rather than a new investigation. If you are running ebpf_exporter alongside node_exporter, you get percentiles from the histogram and the question never comes up. It earns its place because the failure mode is so quiet: the average looks fine, it has always looked fine, and it will keep looking fine while p99 writes stall.
What the kernel sees that your agent doesn't. Every individual block I/O request with its queue time and its service time, separately, per device and per operation type. biosnoop prints both columns: QUE(ms) for time in the OS queue and LAT(ms) for issue to completion. Time spent queued in the kernel and time spent waiting on the device are different problems with different fixes, and any single latency number has already added them together.
Best for. Storage that looks healthy in aggregate while a specific operation type stalls, and any case where you need a p99 rather than a mean from a disk.
What it costs you. Block I/O is lower frequency than scheduler events, so histogram collection is cheap enough to run continuously, which is why Cloudflare runs it as an exporter rather than an ad-hoc tool. The caveat is attribution rather than cost, and it appears in the next item.
How to start.
# Latency histogram per disk, powers of 2, 10 second interval.
# -D breaks the histogram out per disk instead of aggregating all devices.
biolatency-bpfcc -D 10 1
4. Runqueue latency: idle cores and threads still waiting
Run queue latency is the answer, and runqlat reports it as a histogram: "the time a task spends waiting on a run queue (or equivalent scheduler data structure) for a turn on-CPU". This is the interval between a thread becoming runnable and a core actually picking it up, and it is invisible to every utilization metric you have, because a thread waiting in the run queue is neither on-CPU nor blocked on anything you can point at.
The reason this deserves its own item rather than folding into off-CPU time is that the two describe different failures with opposite fixes. Off-CPU time is a thread that cannot proceed because it is waiting for something: a lock, a disk, a socket. Run queue latency is a thread that can proceed and is not being given a core. The first is a dependency problem and the second is a capacity or affinity problem, and treating one as the other sends you optimizing the wrong layer.
The man page states the load relationship: the wait time "should be small, but a task may need to wait its turn due to CPU load. The higher the CPU load, the longer a task will generally need to wait its turn." The case worth watching is where that relationship appears to break. Rising run queue latency with cores showing idle usually means the runnable threads and the free cores are not on the same side of a boundary, which in practice is cgroup CPU quota, CPU affinity pinning, or a NUMA layout that keeps work away from the cores that are free.
This ranks fourth because it is narrower than the three above it. When it is the answer it is decisive and nothing else finds it, but the number of investigations where scheduling delay is the cause is smaller than the number where blocked time or a misread average is.
What the kernel sees that your agent doesn't. The gap between sched_wakeup and sched_switch for every thread, as a distribution. Load average is a decayed count of runnable-plus-uninterruptible tasks over one, five and fifteen minutes, which tells you roughly how many things wanted to run and nothing about how long any of them waited. A 200ms spike in scheduling delay is invisible in a one-minute decayed average and is entirely visible in a histogram.
Best for. Throughput plateauing well below CPU saturation, latency that correlates with load but not with any single service's work, and containers where the CPU quota is the suspected constraint.
What it costs you. The same scheduler-frequency problem as off-CPU time, and the man page says so: despite efficient in-kernel storage, "the overhead of this tool may become significant for some workloads", with the advice to "measure in a lab environment to quantify the overhead before use." A histogram is cheaper than per-event output because the aggregation happens in the kernel and only the buckets cross into userspace, but the probe still fires on every scheduler event.
How to start.
# Run queue latency histogram in microseconds, per second, 10 times.
# -m would give milliseconds; microseconds is the right resolution here
# because a healthy queue wait is usually double digits of microseconds.
runqlat-bpfcc 1 10
5. Requested vs issued bytes: the app wrote 40MB, iostat shows 3MB
Neither. You are looking at two different layers and the page cache sits between them, which means the two views are supposed to disagree and the size of the disagreement is itself the diagnostic. On the read side the mechanism is that the kernel "checks whether the pages are present in Page Cache and immediately returns them to the caller if so", with zero disk operations performed, and any later read of those pages is served the same way "without any disk IOP until these pages have not been evicted." On the write side, buffered writes land in cache and are flushed later by writeback, so the timing and the size of what reaches the device are both decided by the kernel rather than your application.
This matters because the gap is where a whole class of confusing behavior lives. A read-heavy service that appears to do no disk I/O until a cache eviction turns it into a service that suddenly does a lot. A write path that reports fast completions until the writeback threshold is crossed and the process is throttled. In both cases the application-side number and the device-side number are each correct, and the story is entirely in the difference.
Getting both halves means measuring at both layers and comparing, which is the part no single existing tool does for you. VFS-level counts tell you what was requested. Block-level counts tell you what was issued. The ratio over an interval is the cache hit rate for that specific workload, which is a more useful number than the system-wide one because it is scoped to the process you care about.
This ranks fifth because it is the most situational of the five. It is decisive when it applies and it does not apply to most latency investigations. It is on the list because when a team is arguing about which of two dashboards is lying, this is the answer, and the argument is common.
What the kernel sees that your agent doesn't. Both sides of the boundary, with a process attached to each. The caveat belongs right here, because it is the one that bites: block-layer tools attribute I/O to a cached process ID, and biosnoop's man page is explicit that this "usually (but isn't guaranteed) to identify the responsible process for the I/O". Writeback happens asynchronously in a kernel thread, so the process credited with a block write is frequently not the process that dirtied the page. Attribution is reliable on the read side and approximate on the write side, and any tool claiming otherwise is not telling you about the flush.
Best for. A disagreement between what an application reports and what iostat shows, cache-eviction cliffs, and capacity work where you need to know how much of your read traffic is real disk traffic.
What it costs you. You are running two probes and correlating them, which is more assembly than the other four items. The write-side attribution limit above is structural rather than a tool defect, so the honest scope is that you can measure the volume gap precisely and attribute the write half approximately.
How to start.
# What the block layer actually issued, with queue and service time split out.
# Compare the summed BYTES column against what your application logged.
biosnoop-bpfcc
# Cache hit and miss counts system-wide, as the coarse first check
# before you go per-process.
cachestat-bpfcc 1 10
Which eBPF tool gets me off-CPU time, retransmit attribution or disk latency histograms?
The bcc tools cover all five out of the box, bpftrace and yeet cover them if you write the probe, and ebpf_exporter covers the two that work as standing metrics. What separates them is not capability but what you install, who writes the program, and whether it survives the terminal closing.
| Tool | What it gets you of the five | What you install | Who writes the program | Runs across a fleet | What it leaves behind |
|---|---|---|---|---|---|
| yeet scripts | Any of the five, plus the correlation in item 5 that no single tool does | Runtime and daemon, one curl | You, in JavaScript, or run an existing script | Yes, same script per host | Nothing after the script exits |
| bcc tools | Items 1 through 5 individually, as separate ready-made tools | Package install, plus LLVM and kernel headers | Nobody, for the shipped tools | One host at a time, per terminal | Nothing after exit |
| bpftrace | Any of them, if you write the probe | Package install | You, in the bpftrace language | One host at a time, per terminal | Nothing after Ctrl-C |
| ebpf_exporter | Items 3 and 4 continuously, as Prometheus histograms | Exporter plus your Prometheus | Nobody, for the shipped examples | Yes, scraped like any exporter | A running exporter and stored series |
| Datadog | Network-level TCP failures, not per-process retransmit attribution | Agent per host, kernel 4.4.0+ | Nobody | Yes, with retention and a query language | An agent and your data in their storage |
| Pixie | Protocol messages, CPU stack samples, TLS-library capture | Kubernetes, PEMs as a DaemonSet | Nobody, or PxL scripts | Yes, within one cluster | A DaemonSet and its footprint |
| Coroot | Continuous on-CPU profiling with container metadata | Node agent and collector | Nobody | Yes, across nodes | An agent and stored profiles |
Two rows deserve a note, because the interesting fact about each is what its own documentation does not say. Pixie's eBPF page describes probes that snoop send() and recv() data, user-space probes on encryption libraries that "capture the data before it is encrypted", periodic CPU stack sampling, and generated code for tracing function arguments. That is a substantial capability list, and the page states no limitation, no coverage boundary, and no kernel minimum anywhere on it. Coroot's profiling page says its agent "continuously profiles all processes running on a node, associates them with container metadata, and sends the results to the collector", and likewise names no kernel floor and no boundary. Neither page mentions off-CPU profiling.
That absence is not an accusation of dishonesty. Vendor capability pages document capability, which is their job. It does matter for this specific decision, because "continuously profiles all processes" reads like complete coverage to someone deciding whether item 1 is already handled, and on-CPU profiling of all processes is still on-CPU profiling. The check worth running is to search a vendor's own docs for the thing you need before assuming a broad claim includes it.
When should I just use Datadog, Pixie or Coroot instead of writing a probe?
Whenever the question is continuous, fleet-wide, or historical, which is most questions most of the time. Every item on this list is a one-host, right-now measurement, and turning any of them into a time series across a hundred machines with alerting and ninety days of history is a storage and aggregation problem that no kernel probe solves. That is what you are paying an agent for, and it is a real thing to be paying for.
Reach for a platform when you need the answer to survive the host rebooting, when you need to compare this week to last quarter, or when the question spans service boundaries. A trace that follows one request across four services depends on context your application propagates, and no amount of kernel-side capture reconstructs it, because the relationship exists in your code rather than on the wire.
Reach for ebpf_exporter specifically when the metric you want is continuous and per-host but your existing exporter aggregates it away. Items 3 and 4 are both in that shape, both ship as examples, and both land in the Prometheus you already run. This is the cheapest fix on the whole list and it is the one people skip, because it looks like adopting a new tool when it is really adding two scrape targets.
Reach for a ready-made bcc tool when the question is one-off and one of the hundred shipped tools already asks it. Items 1 through 5 all have a bcc tool that covers the common case, and installing bcc is faster than writing anything. Reach for bpftrace or yeet when the shipped tool's shape is wrong, and pick between bpftrace and bcc on what you need to install rather than on syntax: when you need the port-to-process resolution from item 2 that tcpretrans cannot give you, or the VFS-to-block correlation from item 5 that no single tool performs.
Why isn't container memory or HTTP latency on this list?
Because the list is ranked by how often the answer ends the investigation, and both are better served by a dedicated post than by a summary line here. Ranking requires a criterion, and that is the one used throughout. Adding items with a lower hit rate makes the list longer and the ranking less useful, which is why this is five items rather than fifteen.
Container memory has its own failure mode worth understanding properly, where the number a container reports and the number the process inside it reports diverge for reasons that are entirely about cgroup accounting rather than a leak. That is covered in Why is my container using more memory than the process inside it?, which goes through every relevant memory.stat field. HTTP-level visibility, datastore traffic attribution and syscall-level tracing are each their own investigation with their own tooling, and the kernel data conversation about them differs enough that folding them in would produce paragraphs of unrelated advice rather than a ranking.
My p99 is bad and I do not know which of these five to measure first. How do I pick?
Start with what the existing instruments already ruled out, because each of the five is the answer to a different dead end. If the CPU profile is flat, the time is not being burned and item 1 is the first move. If the profile is flat and off-CPU time points at network waits, item 2 tells you whether retransmits are behind them. If the waits are disk, item 3 tells you whether the device is slow and item 5 tells you whether the request reached the device at all. If nothing is blocked and throughput still plateaus below saturation, item 4 is the one left.
| What you already know | Start with | Why that one |
|---|---|---|
| Traces are clean, CPU profile is flat | Item 1, offcputime | The time is being spent blocked, which no on-CPU sample can show |
| Off-CPU stacks point at socket reads or writes | Item 2, retransmit attribution | Names which connection and which owner, where host counters cannot |
| Off-CPU stacks point at block I/O | Item 3, biolatency | An average hides a bimodal disk; the histogram shows which mode moved |
| Disk looks busy but the app disagrees about volume | Item 5, VFS versus block | The page cache sits between them, so both numbers can be right |
| Nothing is blocked, throughput plateaus under saturation | Item 4, runqlat | Runnable threads waiting for a core is invisible to utilization metrics |
The order matters more than it looks. Off-CPU time comes first not because it is the most common cause but because it is the cheapest way to eliminate four of the five: the stack traces it returns tell you whether you are looking at a disk problem, a network problem, or a scheduling problem, before you have committed to measuring any of them. Running item 2 or item 3 first works when you already have a hypothesis, and wastes a probe when you do not.
The bottom line: keep the agent, and know which five questions it was never going to answer
If you need retention, fleet aggregation, or a query language over history, keep paying for Datadog or whatever you run, because none of this replaces storage. If you need traces across service boundaries, that requires context propagation from your application and no kernel probe substitutes for it. If you run Kubernetes and want cluster-wide protocol visibility as a standing capability, Pixie is the right shape. If you want continuous on-CPU profiles with container metadata attached, Coroot does that well.
If your p99 is bad and the profile is flat, run offcputime before anything else, because on-CPU sampling structurally cannot show you blocked time and that is where the latency usually is. If you need per-disk latency percentiles or run queue delay as standing metrics, add ebpf_exporter to the Prometheus you already run. If you need which process owns a retransmit, do not trust tcpretrans' PID column, because its own man page says that column is usually zero; resolve the owner from the socket's port instead, with bpftrace or yeet.
The failure worth avoiding is concluding that a question has no answer because the tools you have return nothing. A flat profile, a zeroed PID column, and a reasonable-looking average are three different ways of being told that you measured at the wrong layer, and all three read as "nothing to see here" if you do not know what the instrument structurally cannot report.
Frequently asked questions
Can a sampling profiler detect a 1-in-10,000 slow request?
Not reliably, and not by raising the sample rate. A sampling profiler records what is on-CPU at each interrupt, so a rare event has to coincide with a sample to appear at all, and a request that is slow because it is blocked is not on-CPU during the interesting part. Event-triggered tracing catches the rare case because it fires on the event rather than on a clock.
What is off-CPU time and how is it different from CPU time?
Off-CPU time is the duration a thread spent blocked and not running, including waits on disk, network, locks, page faults and involuntary context switches. CPU time is the duration it spent executing. They are complementary and non-overlapping views of the same thread, which is why a flat CPU profile is not evidence that a service is fast.
Why does tcpretrans show PID 0?
Because the retransmit fired from a kernel timer rather than from a process context, so the process on-CPU at that instant was the kernel itself. The bcc man page documents this directly, saying the PID column may usually be 0 for timer-based retransmits. Attribute the connection from the socket 4-tuple instead of from that column.
Does eBPF tracing slow down production?
It depends entirely on event frequency. Block I/O and TCP retransmit probes are cheap because the events are rare. Scheduler probes are expensive because scheduler events can exceed a million per second, and the bcc man pages for both offcputime and runqlat say to measure overhead in a lab before production use. Filtering in the kernel with a minimum threshold is the standard mitigation.
What is run queue latency and what causes it?
It is the time a runnable thread waits for a core after becoming runnable and before actually being scheduled. CPU load is the usual cause, and the relationship is direct. Rising queue latency with idle cores available usually points at a boundary instead: a cgroup CPU quota, CPU affinity pinning, or a NUMA layout keeping the work away from the free cores.
Do I need root to run these tools?
Yes for the bcc tools, which require root because loading a BPF program does. The bcc man pages state the root requirement alongside CONFIG_BPF being enabled in the kernel. Some runtimes handle the privileged load through a daemon so the invoking command does not need sudo, which changes who needs the privilege rather than whether it is needed.
Why do node_exporter and my application disagree about disk I/O?
Because the page cache sits between them and they are measuring different layers. Application-level counts are what was requested through VFS; block-level counts are what was actually issued to the device. Reads served from cache perform zero disk operations, and buffered writes reach the device later during writeback, so the two numbers are both correct.
Can eBPF attribute disk writes to the process that caused them?
Only approximately, on the write side. Writeback is asynchronous and happens in a kernel thread, so the process credited with a block write is frequently not the one that dirtied the page. The biosnoop man page says the cached process ID usually but not guaranteed identifies the responsible process. Read-side attribution is reliable.
What is the lowest-overhead way to get disk latency percentiles on Linux?
An in-kernel histogram, where the bucketing happens at the probe and only bucket counts cross into userspace. Cloudflare's ebpf_exporter ships a block I/O example that emits per-device, per-operation Prometheus histograms, which is cheap enough to run continuously and gives you real percentiles instead of an average derived from node_exporter's totals.
Does Pixie document what its eBPF probes cannot see?
Not on its eBPF overview page. That page describes snooping send and recv data, user-space probes on encryption libraries, periodic CPU stack sampling and dynamic function argument tracing, and states no limitation, coverage boundary or kernel minimum. Check the specific capability you need in a vendor's docs rather than inferring it from a broad coverage claim.
What kernel version do I need for these measurements?
The tracepoints and kprobes behind all five have been stable for many kernel generations, so the measurements work well below current kernels, including 4.19. The constraint is building the probe rather than the kernel refusing to answer: kernels without BTF need matching kernel headers at compile time, which is what makes a bcc install on a minimal host harder than the package name suggests. Check bpftool feature probe on the oldest kernel in your fleet before committing. Datadog documents its own eBPF floor as 4.4.0 or backported features, with RHEL and CentOS 7.6+ working below it, while Pixie and Coroot state no kernel minimum at all.
Is high TCP retransmission always a problem?
No. TCP retransmission is the protocol recovering from loss, which is normal behavior, and a low steady rate on a busy host is not a defect. Retransmit counters are widely misread for this reason. What is worth investigating is a change in rate, retransmits concentrated on one peer or one socket, or retransmits paired with a latency regression on the same path.
Can I get these metrics without installing an agent on every host?
For one-off investigation, yes: the bcc tools, bpftrace and yeet scripts all run per host, answer, and exit, leaving nothing behind. For continuous collection across a fleet you need something resident that scrapes or ships, whether that is ebpf_exporter into Prometheus or a vendor agent. The distinction is whether the question is asked once or asked forever.
What does an APM agent do better than kernel-side probes?
Retention, fleet aggregation, a query language, alerting, and distributed traces that follow a request across service boundaries. All five are storage and correlation capabilities rather than capture capabilities, and no kernel probe produces any of them. The two are different altitudes and most teams end up running both.
Sources
- bcc tools and man pages (IO Visor, 2026) — the toolkit behind all five items, ~100 tools, needing LLVM and kernel headers. Its man pages carry the sharpest caveats: tcpretrans says its PID column "may usually be 0, for the kernel, for timer-based retransmits"; biosnoop says the cached PID "usually (but isn't guaranteed) to identify the responsible process", splitting QUE(ms) from LAT(ms); offcputime records time threads were "blocked and 'off-CPU'" and is "complementary to CPU profiling"; runqlat measures "the time a task spends waiting on a run queue... for a turn on-CPU". Both scheduler tools warn overhead "may become significant".
- Cloudflare ebpf_exporter (Cloudflare, 2026) — the canonical per-disk histogram shape in item 3: a Prometheus exporter for custom eBPF metrics whose block IO example "attaches to block io subsystem and reports disk latency as a prometheus histogram, allowing you to compute percentiles", emitting series such as
ebpf_exporter_bio_latency_seconds_bucket{device="nvme0n1",operation="write",le="0.000128"}. Ships runqueue latency, kernel timer and per-CPU softirq examples, and supports exp2, exp2zero, linear and fixed bucket types. - Pixie eBPF documentation (Pixie docs, 2026) — cited for both what it claims and what it omits: eBPF probes "snoop the data" on network syscalls such as
send()andrecv(), user-space probes on encryption libraries "capture the data before it is encrypted", the profiler periodically interrupts the CPU to collect stack traces, and dynamic logging generates eBPF code for tracing function arguments. The page states no limitation, coverage gap, retention boundary or kernel minimum, and does not mention off-CPU profiling. - Coroot eBPF profiling (Coroot docs, 2026) — the agent "continuously profiles all processes running on a node, associates them with container metadata, and sends the results to the collector" and "in most cases works out of the box with no configuration", with extra symbolization guidance for Java and Node.js. The page names no minimum kernel version, no coverage boundary and no limitation, and does not mention off-CPU profiling, which is the gap item 1 turns on.
- Datadog APM and infrastructure metrics (Datadog docs, 2026) — the four metric families an APM agent collects. Trace metrics:
trace.<SPAN_NAME>.hits,.errorsand.apdex, tagged byenv,service,resource_nameandhttp.status_code, on "100% of the application's traffic". Runtime metrics:jvm.heap_memory,jvm.gc.major_collection_time,runtime.go.num_goroutine,runtime.node.event_loop.delay.avg, for Java, Python, Ruby, Go, Node.js and .NET but not PHP or C++. System core:system.cpu.iowait,system.load.1,system.mem.used,system.io.await,system.io.r_s, all host-wide with no attribution. Metric types: a HISTOGRAM is "calculated Agent-side" while a DISTRIBUTION "sends all the raw data" and "aggregations occur on the server-side". - Datadog Cloud Network Monitoring setup (Datadog docs, 2026) — the vendor kernel floor cited in item 2 and the version FAQ: requires "Linux kernel versions of 4.4.0+ or have eBPF features backported", with CentOS and RHEL 7.6+ supported below that line via backports. Collects traffic between services, containers and availability zones, identifies "TCP failures including resets, refusals, and timeouts", and uses eBPF on Linux with a kernel device driver on Windows.
- Essential page cache theory (Viacheslav Biriukov, 2026) — the mechanism behind item 5: on a read the kernel "checks whether the pages are present in Page Cache and immediately returns them to the caller if so", performing zero disk operations, and any later read of that range "will be handled by Page Cache without any disk IOP until these pages have not been evicted", regardless of which process or cgroup issues it. Disk I/O occurs only on a miss, which is why VFS and block-level counts disagree by design.
- bpftrace (bpftrace, 2026) — the high-level tracing language for the cases where a shipped tool's shape is wrong, covering kprobes, uprobes, tracepoints, USDT and perf events. Cited here as the route for the two custom probes this post recommends: socket-based retransmit attribution, and correlating VFS-level request volume against block-level issued volume.
Related resources
- Why is my container using more memory than the process inside it? — the memory question this post deliberately does not rank, worked through field by field.
- bpftrace vs BCC — which of the two to write a probe in once you have decided a shipped tool is the wrong shape.
- Why is my Redis slow in 2026? — the datastore version of item 5's layer-disagreement problem, including the client-versus-server latency gap.
- What the kernel sees that your proxy doesn't — the L7 traffic vantage point, which is a different layer from everything ranked here.
- yeet documentation — writing your own probe when the ready-made tool cannot answer the question.
Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.