Writing an eBPF Protocol Tracer

Quick answer. Use bpftrace if the question fits in one line, BCC or libbpf if you want a Python or C userspace, and yeet if you need an iterative parser or a UI, because that is where the decoder can be JavaScript above a C object. Whichever you pick, do not write the parser in the kernel: a length-prefixed format needs loops whose trip count comes from a length field, and the verifier accepts a loop only when it can prove the exit condition becomes true, which it cannot do for a value read out of user memory. Validate the header in the kernel, copy a fixed window with bpf_probe_read_user, and walk the real structure in userspace. mongosnoop copies 192 bytes and does every BSON element walk in JavaScript.

I write eBPF programs that read protocol traffic, and the mistake I made early and watch other people make is treating the kernel as a place to be clever. The instinct is reasonable: you are already in the send path with the buffer in front of you, so parsing it there feels like the efficient choice. Then the verifier rejects it, and the next three hours go into restructuring a loop that was never going to be accepted. What changed my results was giving up on kernel-side parsing entirely and treating the BPF program as a tap with a filter on it, which turns out to be a better design and not just a concession to a tool.

bpftrace vs BCC vs libbpf vs yeet: which should I use for a protocol tracer?

Pick by how much structure you need out of the bytes, because that decides how much userspace you are about to write. All four load the same kind of BPF program; they differ in what happens to the bytes after the kernel hands them over.

Tool	Where the parser lives	Iterative parser	UI or custom output	Best for
yeet	JavaScript, above a C object	Yes	Yes, terminal or JSON	a protocol decoder, or a tool with a UI
bpftrace	its own language, in-kernel	No	printf and maps only	a one-line question, answered now
BCC	Python	Yes	whatever you write	a maintained tool with a Python half
libbpf	C or Rust	Yes	whatever you write	a shipped binary with no runtime

1. A one-off question about kernel behaviour. bpftrace, in one line. Counting calls, histogramming latencies, printing arguments. Nothing is close on time-to-answer when your question fits in its language, and bpftrace vs BCC covers that choice in more depth. It is the wrong tool the moment you need an iterative parser, because there is nowhere to put one.
2. A tool you will maintain, with a Python or C++ userspace. BCC or libbpf directly. The mainstream path, the most examples, and the right answer when the userspace half wants to be a real program with its own build and dependencies. BPF from scratch in Rust walks that road at full depth if you want to see what building on the raw interface involves.
3. A protocol decoder, or any of these with a UI. yeet, where the BPF object is still C but everything above it is JavaScript. This is the split the rest of this post argues for, with the language boundary landing exactly where the kernel/userspace boundary already is.
4. Anything already solved. Fork it. mongosnoop and redissnoop are both this pattern, and the interesting work is rarely the probe.

Why the userspace language decides the project

Testing, mostly. mongosnoop's BSON decoder has round-trip tests built on an independent encoder, so a decoder bug and an encoder bug cannot cancel each other, and they run with node test/bson.test.mjs against no kernel, no database and no privileges. That is only possible because the parser never went into the kernel, and it is the difference between a decoder you can refactor and one you change by guessing.

The second thing it decides is what the tool becomes after it works. Once decoded records are ordinary JavaScript objects, a terminal UI, a JSON sink, an HTTP endpoint or an aggregation are all just code you already know how to write, and none of them require touching the BPF object again. mongosnoop's split shows the proportions: 545 lines of BPF C against about 1,900 lines of JavaScript, and none of that JavaScript is kernel-specific. It is a parser, a classifier, a formatter and a UI.

Customizing an existing probe instead of starting over

The reason to fork rather than build is that the kernel half is the part you would spend a week on and the part that generalizes. A working socket tracer already contains the header validation, the request/reply correlation, the per-CPU scratch maps and the iov_iter branch that the rest of this post works through. Changing which protocol it decodes usually does not touch any of that.

Run one to see whether it is close, since a gh: target clones, builds and runs in one step:

yeet run gh:yeet-src/mongosnoop

Three seams are where forks actually happen, and each is a small, local edit:

• The decoder. src/lib/bson.js is a cursor over a byte window with no kernel dependency. Swapping BSON for another length-prefixed format is a rewrite of that one module, and the probe, the correlation and the UI keep working. This is the whole job when the protocol changes.

• The output. src/probes/mongo.js holds every decoded record in a RingBuf.subscribe callback:

  import { BpfObject, RingBuf } from "yeet:bpf";
  new RingBuf(ctl, "mongo_events").subscribe((w) => { /* every command lands here */ });
  

  A JSON, HTTP or Kafka sink is a branch in that callback rather than a fork of the project, which is why the README says there is no --json mode and does not need one.

• The kernel-side filter. The BPF program exposes min_latency_us in its .data section, bound with .bind("probe.data", { kind: "data" }) and patched live from JavaScript. Raising the floor drops fast commands before they reach the ring buffer, so filtering costs nothing in userspace. Any threshold your tool needs can work the same way.

What you do not get by forking is a different attach story, and the attach point is what bounds a tool more than its code does; what the TC layer sees makes that case for the network side. A BPF program attaches once, so a probe built around SSL_write on one target stays that shape, and the plaintext and TLS coverage limits in what it can't see are inherited along with the working code. Those are properties of the seam rather than of the implementation, which is the honest reason to read the limits before assuming a fork will clear them.

BPF verifier error: "back-edge from insn X to Y" in a parsing loop

A back-edge error means the verifier found a jump backwards to an earlier instruction and could not prove the loop it forms will terminate. Its first pass is a depth-first search that checks the program is a directed acyclic graph, and a backward jump breaks that, so the message names the instruction pair forming the cycle. The verifier evaluates all possible execution paths before it will load anything, so a loop is acceptable only when it can establish an upper bound on the number of iterations. The eBPF documentation puts the requirement directly: programs may contain bounded loops, but a program is accepted only if the verifier can ensure the loop contains an exit condition guaranteed to become true.

A binary format parser violates this by construction, and it is worth being clear that this is a property of eBPF's safety model rather than a missing feature; what is eBPF covers why the verifier exists at all. Walking BSON means reading an element's type byte, reading its length, advancing by that length, and repeating until you reach the document's end. The trip count is data. It comes from bytes in a buffer that the verifier has no knowledge of and no reason to trust, so there is no constant it can bound the loop by, and no amount of restructuring changes that. The same applies to any TLV format, to protobuf wire format, and to anything else where element sizes are declared rather than fixed.

Two more limits shape what is possible even when you do bound your loops. The verifier explores up to 1 million instructions during analysis, which the kernel documentation describes as meaning the largest program can consist of a million NOP instructions; a nested bounded loop over a few hundred iterations can blow through that in path exploration long before it exhausts anything else. And unprivileged programs are capped at BPF_MAXINSNS, 4096 instructions. These are why a program that verifies today can stop verifying after you add one condition inside a loop: complexity is multiplicative across paths, and you are spending a budget you cannot see.

The practical read is that the verifier is not an obstacle between you and a kernel-side parser. It is telling you the kernel is the wrong place for that code, and it happens to be right, because a parser in BPF C would also be the hardest part of your program to test.

"Stack size exceeded": how do I use a struct larger than 512 bytes in eBPF?

Put it in a per-CPU array map and take a pointer. A combined stack size ... is too large rejection, or clang's own stack size exceeded warning, means the structure cannot live as a local at all: the kernel's BPF design FAQ states that all program types are limited to 512 bytes of stack space, and that the verifier computes the actual amount used. An event struct carrying a process name, a few integers and any payload window at all exceeds that immediately, so it cannot be a local variable:

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, struct mongo_event);
} event_scratch SEC(".maps");

__u32 zero = 0;
struct mongo_event *e = bpf_map_lookup_elem(&event_scratch, &zero);
if (!e) return 0;

A per-CPU array with a single entry gives every CPU its own copy of the structure, so there is no locking, no contention and no chance of two CPUs writing the same scratch space. The if (!e) check is not defensive habit; the verifier requires it, because bpf_map_lookup_elem returns a pointer that may be null and it will not let you dereference one you have not tested.

mongosnoop uses two of these, one for the event under construction and one for the in-flight record, precisely because its event carries a 192-byte payload window and is well past the stack limit. The cost is one map lookup per event, which is a fair trade for being able to build a structure of any size.

Should I parse a protocol in the eBPF program or in userspace?

In userspace, with the kernel doing only the cheap filtering that shrinks the volume. That splits into three jobs on the kernel side, none of which requires knowing the payload's structure:

1. Validate that this is your protocol at all. A kprobe on tcp_sendmsg fires for every TCP write on the box, most of which is not what you are looking for. This check is the highest-value code in the program, because everything it rejects costs nothing further.
2. Lift the fields that sit at fixed offsets. A protocol header is fixed-width by design, so reading it needs no loops. MongoDB's is 16 bytes: messageLength, requestID, responseTo, opCode, all little-endian int32s.
3. Copy a fixed window of the payload with bpf_probe_read_user and send it up. Not the whole payload, which is unbounded, but a constant-size window the verifier can check statically.

The validation step deserves emphasis because it is where correctness is won or lost. mongosnoop requires that messageLength match the size of the write and that responseTo be zero on a request. Without both checks, a TCP segment that split mid-message, or an HTTPS call from the same process, parses into a garbage verb and pollutes the output. The header check is what makes the tap trustworthy, and it is also the cheapest possible filter, since it rejects non-matching traffic before any copy happens.

The window size is a real trade with no clean answer. 192 bytes covers a typical command's leading fields, and it truncates a bulk insert that carries documents. Widening it cannot fix the general case, because a payload is unbounded, so the honest move is to pick a size, mark truncated events, and be explicit about the limit rather than implying full coverage. That is what the tool's limits section does.

How do I correlate a request with its reply in eBPF to measure latency?

Key on a correlation id the protocol already carries. The tempting key is the socket pointer, on the reasoning that the next reply on this socket belongs to the last request on it. That works for a strictly serial protocol and breaks on anything pipelined, which includes every database driver with a connection pool. With four commands outstanding on one socket, socket-keyed pairing matches replies to the wrong requests and produces latencies that are wrong while looking entirely plausible, under exactly the concurrency you were trying to measure.

MongoDB hands you the right key. A request carries a fresh requestID and responseTo == 0; the reply echoes that value back in responseTo. So the reply names its own request, and pairing is exact regardless of how many are in flight.

The subtlety that costs a debugging session is scope. Drivers restart requestID at 1 per connection, so the id is unique within a connection and not on the host. Two short-lived clients each send request 1, and one client's reply looks up the other's pending entry, which produced a 2349 second latency in one real demo run before the key was fixed. Combining the pid with the request id gives each process its own numbering space:

__u64 key = ((__u64)pid << 32) | req_id;

Store the pending request in a BPF_MAP_TYPE_LRU_HASH rather than a plain hash. In-flight entries get orphaned constantly: a connection drops, a process is killed, a reply never arrives. A plain hash fills with those and then starts rejecting inserts, at which point you silently stop tracing. An LRU map evicts the least recently used entry instead, so the stale ones are reclaimed without any cleanup logic.

Why does my kprobe on tcp_recvmsg read an empty buffer?

Because the buffer is empty when the entry probe runs. A receive function is handed a pointer to memory the kernel is about to fill, so reading it on entry gives you whatever was there before, which is either stale data or zeroes. The bytes exist only after the copy, which means after the function returns.

The pattern is a two-probe handoff through a scratch map keyed by pid_tgid:

SEC("kprobe/tcp_recvmsg")
int on_recvmsg(struct pt_regs *ctx) {
    /* record where the bytes will land */
}

SEC("kretprobe/tcp_recvmsg")
int on_recvmsg_ret(struct pt_regs *ctx) {
    /* now read them, and trust the return value for how many */
}

pid_tgid is the correct key because it identifies the specific thread that is inside the call, so two threads receiving concurrently do not collide. The return probe also gets the return value, which tells you how many bytes actually arrived; the length passed on entry is a buffer capacity and is usually larger. Reading the entry length as though it were a byte count is a bug that shows up as garbage at the end of every capture.

Send paths do not need this, which is why a send-side probe is where you start when building one of these. At tcp_sendmsg the data is already in the buffer the caller passed, so one probe is enough.

How do I read msghdr iov_iter in eBPF across kernel versions?

Check iter_type and handle both cases, because a modern kernel can pass the buffer two different ways:

__u8 itype = BPF_CORE_READ(msg, msg_iter.iter_type);
if (itype == ITER_UBUF)
    return BPF_CORE_READ(msg, msg_iter.ubuf);
if (itype == ITER_IOVEC) {
    const struct iovec *iov = BPF_CORE_READ(msg, msg_iter.__iov);
    if (iov)
        return BPF_CORE_READ(iov, iov_base);
}
return NULL;

ITER_UBUF stores a single user buffer inline in ubuf, while ITER_IOVEC points at a classic iovec array. Which one you get depends on the kernel version and on how the caller issued the write, so a program handling only one of them works on some machines and returns null on others. This is the most fragile read in a socket-tracing program, and it is worth marking as such in a comment, because it is the line that breaks when someone tries the tool on an unfamiliar kernel.

BPF_CORE_READ is doing real work here rather than being a stylistic wrapper. It emits a CO-RE relocation, so libbpf resolves and matches the types and fields at load time and updates the offsets to match the running kernel, reading BTF from /sys/kernel/btf/vmlinux. If a field moves because new members were added to a struct, libbpf adjusts the offset automatically and your program keeps working.

"no member named __iov in struct iov_iter": CO-RE and renamed fields

Because CO-RE relocates offsets, and a rename is not an offset change. A compile error naming a missing struct member, rather than a load-time verifier rejection, is the signature of this problem: the field exists on your kernel under one name and on the target kernel under another, and no relocation runs because the build never finished. This is the distinction that catches people who have read the CO-RE documentation and reasonably concluded that portability is handled.

Kernel change	CO-RE handles it	Why
A field moves because members were added	Yes	libbpf updates the offset at load time from BTF
A struct grows	Yes	Same mechanism
A field is renamed	No	The name is resolved at compile time, before BTF matching
A field is removed	No	Nothing to relocate to
A type changes shape	Partially	Depends on whether the access still makes sense

iov_iter.iov was renamed to __iov in kernel 6.4. mongosnoop compiles against __iov, which means the object builds and loads on 6.12 and is a compile error on 6.1, because vmlinux.h is generated from the running kernel's BTF and carries only the name that kernel has. There is no relocation that fixes this, because the failure happens in the compiler, before any BTF matching could occur. Handling it means either a version-conditional access or the bpf_core_field_exists family of checks, and either way it is a source-level problem rather than a load-time one.

The general lesson is that a compile against your development kernel's vmlinux.h is not evidence of portability, and neither is a successful load on your development kernel. Those are two different failures with two different remedies, and one build on one machine tests for neither.

How do I test an eBPF program against multiple kernel versions in CI?

Boot them and run the verifier, in CI, because nothing short of that is evidence. A program that verifies on 6.12 can be rejected on 6.1 by an older verifier with less capable range tracking, and a program that compiles on 6.12 may not compile on 6.1 at all, as the __iov rename shows. Neither failure is visible from your laptop.

The setup that works is a matrix job doing three things per kernel:

1. Build the object once, so every kernel is tested against identical bytecode.
2. Boot that kernel in a VM, using prebuilt images from a project like cilium's little-vm-helper so you are not maintaining kernel builds.
3. Run veristat inside the VM against the object, which loads each program and reports a per-program verdict, and fail the job if anything is rejected.

mongosnoop runs this against 6.1, 6.6, 6.12 and bpf-next, and pivots the per-kernel results into one grid. This is where the __iov rename was caught, which is the kind of bug that otherwise ships and becomes a stranger's issue report about a tool that does not build. A per-program verdict matters more than a pass or fail for the object, because a rejection usually implicates one program and knowing which one is most of the fix.

The bottom line: the kernel filters, userspace parses

Put the header validation in the kernel, because it is the filter that makes everything downstream cheap, and it needs no loops. Keep every event struct in a BPF_MAP_TYPE_PERCPU_ARRAY once it passes 512 bytes, and expect the verifier to require a null check on the lookup. Key in-flight state on the protocol's own correlation id combined with the pid, in an LRU_HASH, and use a kretprobe for anything on a receive path. Copy a fixed window rather than a declared length, mark what you truncated, and put the parser wherever parsers are easy to write and easy to test. Run veristat against every kernel you claim to support, in CI, because a successful build proves nothing about an older verifier.

The failure worth avoiding is the one that looks like diligence: spending days making a parser pass the verifier. If you succeed you have the least testable code in your project sitting in the hottest path in the kernel, with error handling that BPF makes awkward and a bug surface that reaches every process on the box. The verifier objecting to your loop is the cheapest design review you will get.

Frequently asked questions

Can eBPF programs use loops?

Only bounded ones. The kernel documentation notes that BPF developers spent years working out how to support loops at all, and current kernels accept a loop only if the verifier can ensure it contains an exit condition guaranteed to become true. A loop whose trip count depends on a length field read from user memory does not satisfy that, because the verifier cannot bound a value it has to trust. That single constraint is what makes parsing variable-length binary formats in the kernel impractical.

What is the eBPF stack size limit?

512 bytes. The kernel BPF design FAQ states that all program types are limited to 512 bytes of stack space, and that the verifier computes the actual amount used. Any structure larger than that cannot be a local variable, which is why protocol-tracing programs keep their event structs in a per-CPU array map and get a pointer to one rather than declaring it on the stack.

How do I handle a struct larger than 512 bytes in eBPF?

Put it in a BPF_MAP_TYPE_PERCPU_ARRAY with one entry and look up index zero to get a pointer. Each CPU gets its own copy, so there is no locking and no cross-CPU interference, and the map lives in map memory rather than on the 512-byte stack. This is the standard workaround for event structs that carry a payload buffer, and it costs one map lookup per event.

Why does my eBPF program fail the verifier with a variable-length read?

Because the verifier must prove every memory access is in bounds, and a length it cannot bound makes that impossible. Reading a length prefix from user memory and using it as a copy size gives the verifier an unknown value, so it rejects the access. The fix is to bound the value explicitly with a comparison against a compile-time constant before using it, or to copy a fixed-size window and let userspace deal with the real length.

Should I parse protocol data in the kernel or in userspace?

Parse in userspace unless the parse decides whether to keep the event. The kernel is the right place for the cheap filtering that reduces volume: validating a header, matching a port, checking a magic number. It is the wrong place for walking typed, variable-length elements, because that needs unbounded loops the verifier rejects and error handling that BPF makes painful. Copy a fixed window and parse it where a parser is easy to write and easy to test.

How do I correlate a request with its reply in eBPF?

Use a correlation id the protocol already carries, not the socket pointer. Many protocols pipeline several in-flight requests over one connection, so pairing the next reply on a socket mismatches under concurrency and produces confidently wrong latencies. MongoDB's wire protocol carries a requestID that the reply echoes back in responseTo, which pairs correctly with four commands outstanding. Store the pending request in an LRU hash keyed by that id.

Why key an eBPF map on pid plus a request id?

Because protocol-level ids are usually unique per connection, not per host. Many drivers restart their request numbering at 1 for each new connection, so two short-lived clients both produce request 1 and one client's reply can look up the other's pending entry. The subtraction then yields nonsense, in one real case a latency of 2349 seconds. Combining the pid with the request id gives each process its own numbering space.

What is a BPF_MAP_TYPE_LRU_HASH and when should I use it?

A hash map that evicts the least recently used entry when it is full, instead of failing the insert. It is the right choice for tracking in-flight state, because entries can be orphaned by a dropped connection, a killed process or a reply that never arrives, and a plain hash map would fill up with them and then start rejecting new work. With an LRU map the stale entries are reclaimed automatically.

Why do I need a kretprobe to read data from a receive path?

Because at function entry the buffer is not filled yet. A receive call is passed a pointer to where the kernel will copy data, so reading it on entry gives you whatever was there before. The pattern is to record the destination pointer in a scratch map on entry, keyed by pid_tgid, then read the bytes in the return probe once the copy has happened and the return value tells you how many bytes actually arrived.

What is CO-RE in eBPF and what does it actually fix?

Compile Once Run Everywhere, which lets one compiled BPF object run on kernels whose struct layouts differ. libbpf resolves and matches types and fields at load time and updates offsets so the program's logic works on that specific kernel, using BTF from /sys/kernel/btf/vmlinux. It fixes fields moving position when new members are added. It does not fix a field being renamed, because a rename is a source-level change and vmlinux.h carries only the name the generating kernel had.

How do I test that an eBPF program loads on older kernels?

Boot those kernels and run the verifier against the object, because building successfully proves nothing about whether an older verifier accepts the result. A practical setup builds the object once, boots each target kernel in a VM using images from a project like cilium's little-vm-helper, and runs veristat inside each one to get a per-program verdict. This catches both verifier rejections and struct changes that a single development kernel hides.

Is a BPF ring buffer better than a perf buffer?

For most tracing work, yes, and the reason is ordering rather than throughput. A perf buffer is per-CPU, so events from different CPUs arrive interleaved and a consumer has to reorder them. A BPF ring buffer is shared across CPUs and preserves ordering, and it supports a reserve and commit pattern that lets a program claim space before deciding whether to keep the event. It drops rather than blocks when the consumer falls behind.

Sources

• BPF design Q&A (kernel.org BPF documentation) — the authoritative statement of the limits this post is built around: "Currently all program types are limited to 512 bytes of stack space, but the verifier computes the actual amount of stack used and both interpreter and most JITed code consume necessary amount"; the unprivileged instruction cap of BPF_MAXINSNS at 4096; and the verifier's exploration ceiling, "Currently, that limit is set to 1 million. Which essentially means that the largest program can consist of 1 million NOP instructions." Also the historical note that BPF developers were still working out how to support bounded loops.
• What is eBPF? (ebpf.io) — the loop rule as currently implemented: "eBPF programs may contain so called bounded loops but the program is only accepted if the verifier can ensure that the loop contains an exit condition which is guaranteed to become true", plus the finite-complexity requirement that the verifier "will evaluate all possible execution paths and must be capable of completing the analysis within the limits of the configured upper complexity limit", which together are why a data-driven parse loop cannot be accepted.
• libbpf overview (kernel.org BPF documentation) — what CO-RE does and does not cover: it brings together BTF type information, libbpf and the compiler to produce a single executable binary, relying on the running kernel's BTF via /sys/kernel/btf/vmlinux, and at load time "libbpf then resolves and matches all the types and fields, and updates necessary offsets and other relocatable data to ensure that BPF program's logic functions correctly for a specific kernel", with the worked example of a task_struct field shifting position and libbpf automatically adjusting the offset.
• BPF verifier (kernel.org BPF documentation) — the two-step safety process, beginning with a DAG check that disallows loops and performs CFG validation, followed by register state tracking across all reachable paths, plus the pruning mechanism that keeps path exploration tractable. This is the pass that rejects an unbounded parse loop before any of the program's logic is considered.
• cilium/little-vm-helper (Cilium, 2026) — the VM tooling used to boot a range of kernels in CI from prebuilt images, which is what makes a per-kernel verifier check practical without maintaining kernel builds; paired with veristat, which loads each program in the object and reports a per-program verdict rather than a single pass or fail for the whole file.
• mongosnoop BPF source (yeet-src, 2026) — the worked example in full: six programs across a kprobe path and a uprobe path feeding one 512 KB ring buffer, a 192-byte BODY_LEN window copied with bpf_probe_read_user, two PERCPU_ARRAY scratch maps for structs past the stack limit, an LRU_HASH of 16384 in-flight commands keyed on (pid << 32 | requestID), the iter_type branch handling both ITER_UBUF and ITER_IOVEC, and the header validation requiring messageLength to match the write size and responseTo to be zero on a request.
• mongosnoop (yeet-src, 2026) — the userspace half of the split: BSON element walks, shape extraction, value decoding and flag checks all in JavaScript, a decoder with round-trip tests built on an independent encoder so a decoder bug and an encoder bug cannot cancel, and a kernel matrix over 6.1, 6.6, 6.12 and bpf-next that caught iov_iter.iov being renamed to __iov in 6.4.

---

Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.