
Founding engineer at yeet, working on kernel-side observability and the tooling around it. I write about eBPF, Linux internals, and why your telemetry bill looks the way it does.
Last updated: November 2026
Quick answer. To see what customer code does inside your sandboxes, record it from a kernel the platform controls and the tenant's code does not: the guest kernel of a Firecracker microVM, or the shared host kernel under plain containers, with every event keyed to a tenant by its cgroup ID. Hooks in that kernel fire on the workload's exec, file opens and connections without the workload running any of your code, so you get untrusted code execution observability per tenant with no instrumentation and no reliance on the workload's own logs. The runtime decides how well this works. A microVM's guest kernel and image are yours, so an eBPF daemon in the guest image sits below the workload. Containers share the host kernel, so one host daemon covers every tenant on the machine. gVisor is the weak fit: its Sentry services the workload's syscalls in userspace, so host eBPF sees the Sentry's syscalls and not the workload's.
I work on yeetd, the daemon in yeet that loads eBPF programs into a Linux kernel and hands what they record to JavaScript, so most of my week goes into one narrow question: what a program sitting in the kernel can attribute, and where it loses the thread. I have never run a sandbox platform, a CI cloud or a GPU fleet, and I have never held the pager when a tenant's session went sideways at 3am. What I can describe is the boundary those platforms sit on. Each of them runs code its own engineers did not write, and the records they keep about that code come from two places: the platform's API, which knows a session started and stopped, and the workload, which says whatever it was written to say. The record you need comes from a third place, below both of them.
In the kernel underneath the session, and in no record the session wrote. A platform holding only its own execution log knows the tenant, the session ID, the start and stop times, the exit code and maybe a resource total. It does not know which processes the session started, which files they opened or which hosts they connected to, and those three facts are what an abuse report, an egress spike or a "why did my session fail" ticket is about. The workload's own stdout and telemetry can fill some of that in, if the code chose to emit it and chose to emit it truthfully. For code you didn't write, neither condition is yours to set.
The three records a sandbox platform can hold about one session, and what each one can answer:
| Record | Written by | What it knows about a session | What it cannot tell you |
|---|---|---|---|
| Kernel-side record from eBPF, in a kernel you control | Hooks on exec, file open and connect | Every process, file open and connection, attributed to the session's cgroup or VM | What happens inside one process's memory; plaintext of encrypted traffic |
| Platform API or execution log | Your control plane | Tenant, session ID, start, stop, exit code, resource totals | Anything that happened between start and stop |
| In-sandbox logs, stdout, OpenTelemetry from the workload | The tenant's code | Whatever the code chose to emit | Anything it did not emit, or chose to hide |
The sandbox vendors' own guidance describes the target and stops short of the mechanism. Modal's roundup of code-agent sandbox products says "a good provider exposes granular process logs & metrics and throttles egress", and Blaxel's guide to remote code execution sandboxes asks for one view of "what the agent decided, what code it generated, and what the sandbox did with that code", delivered through an OpenTelemetry pipeline. The first two clauses of that sentence come from the agent framework and are application-level data. The third, what the sandbox did with the code, is the part only the kernel underneath the sandbox can report, because the code that did it is the tenant's and nothing in it was written to tell you.
When any one of three things is true, and on a platform that runs other people's code all three usually are. Kernel-level vs application-level telemetry is normally framed as a choice between layers you can both have, and for your own services that framing is right: you add OpenTelemetry spans to code you own, and the kernel fills in the scheduler and network detail around them. The choice stops being a choice when the application layer is not available to you. This is the test I use, and each case points at a different platform:
A code-execution vendor running model-written code for thousands of tenants meets all three on every session. The code isn't theirs, the incidents they get asked about are network and resource incidents, and the code is the subject of the question. For a platform in that position there is no instrumentation for kernel data to supplement, so the kernel record is the only data about the workload that exists. That is a narrower claim than "the kernel sees everything", and it moves the hard question from whether to use the kernel to which kernel the hooks belong in.
No. Observation has to live below the tenant's code, in a kernel the platform controls, and not inside the tenant's process. An agent inside the tenant's process shares its address space and its fate: the code under observation can read the agent's memory, feed it false data, or exit before it flushes. A sidecar process in the same container is one step better and has the same flaw once the workload holds the same user ID. An eBPF program attached in the kernel is a different arrangement. It runs in kernel context on events the workload triggers, the workload executes none of its code, and removing it takes kernel privileges a well-run sandbox does not give the tenant.
"Below" does not always mean "on the host". In a microVM, the kernel below the workload is the guest kernel, and the guest kernel and guest image belong to the platform, so the daemon ships in the guest image and runs beneath the workload inside the VM. Under plain containers, the kernel below every tenant is the host kernel, so the daemon runs on the host. The rule is about which kernel serves the workload's syscalls, which is why the runtime choice decides how much of this is available.
One boundary comes first. AI Agents on Linux: Two Threat Models describes confidential computing, where a platform running other people's agents deliberately makes itself unable to read them, with the kernel and hypervisor on the adversary list. If you sell that guarantee, kernel-side visibility into tenant workloads is the thing you promised not to have, and nothing below applies. Everything here assumes the standard model, where whoever controls the infrastructure is expected to know what runs on it.
Firecracker and plain containers both work well, for different reasons, and gVisor is the weak fit. The question that sorts them is whose kernel services the workload's syscalls, because an eBPF program sees the syscalls of the kernel it is loaded into and nothing else. Fly.io's comparison of Firecracker and gVisor states the split in one pair of sentences: "Firecracker isolates untrusted code by giving it a separate guest kernel inside a hardware-virtualized microVM", while "gVisor isolates it by intercepting its system calls in a userspace kernel written in Go". It compares the two on isolation and says nothing about observability, and neither does any sandbox comparison I could find.
| Runtime | Kernel that serves the workload's syscalls | Where the eBPF daemon goes | What it sees | Fit |
|---|---|---|---|---|
| Firecracker microVM | A guest kernel per microVM, booted from an image you supply | In the guest image, below the workload | Every exec, file open and connection in that VM | Strong |
| Plain containers (runc) | The shared host kernel | On the host, one daemon per machine | Every tenant on the machine, keyed by cgroup ID | Strong |
| gVisor (runsc) | The Sentry, a userspace application kernel | On the host | The Sentry's own host syscalls, not the workload's; packets on the sandbox's network device | Weak |
From the host you get the network; for processes and files the daemon has to run in the guest. The Firecracker README describes a virtual machine monitor that "uses the Linux Kernel Virtual Machine (KVM) to create and run microVMs", started with "a given kernel image, root file system, and boot arguments". The workload's syscalls land in that guest kernel. On the host, Firecracker's design doc describes one process with API, VMM and vCPU threads, where the vCPU threads "run the KVM_RUN main loop" and network devices are "backed by TAP devices on the host". That is everything a host-side probe can reach: the VMM's own syscalls, the KVM_RUN ioctls and the TAP traffic, and none of the guest's execs or opens. The guest image and the guest kernel are both artifacts the platform builds, which is what makes the guest the right place for the daemon: it boots with the VM, before any tenant code runs, and the guest kernel has to be built with BPF support, a config choice the platform owns.
Network is the exception, and there is a public precedent for it. Blacksmith's July 2026 post What is your CI job talking to? describes a CI platform where "all of that VM's traffic crosses a single veth pair on the host", with eBPF programs attached through TCX, "the traffic-control hook that landed in Linux 6.6", and "nothing inside the guest at all". That design attributes egress to a domain name per job and per step without touching the guest, and it is candid about its limits: it is IPv4-only, and "domain attribution is admittedly best-effort". For a microVM platform, host-side network observation like that is the half of the record no guest process can reach.
The guest half carries one condition, and it is the detail most likely to change the strong rating in the microVM row of the table. If tenant code runs as root inside the guest, it can kill any process there, the daemon included, and a guest daemon in that setup is a convenience and no longer a boundary. Run the workload as an unprivileged user inside the guest and keep the daemon as the only privileged resident, or treat the host-side veth record as the part you rely on and the guest record as best effort. Which of the two you have is a property of your guest image, so decide it on purpose before an incident decides it for you. The same goes for platforms that boot tenant-supplied root filesystems: the daemon is below the workload only if it ships in a layer the platform adds and the tenant cannot replace.
Key every event by the cgroup of the task that caused it. Container runtimes put each container in its own cgroup, and the kernel exposes that cgroup to eBPF programs directly. The bpf-helpers(7) man page documents bpf_get_current_cgroup_id() as returning "the current cgroup id based on the cgroup within which the current task is running", which covers exec, file open and connect hooks, and bpf_skb_cgroup_id() as returning "the cgroup v2 id of the socket associated with the skb", usable "on TC egress path, but not on ingress". That is eBPF per cgroup tenant attribution without instrumentation: one host daemon, every event stamped with a tenant key at the moment it happens, and no change to any tenant's image, code or configuration.
The cgroup ID is a number, and the work left over is mapping it back to a tenant and a session. On the host, the container runtime already knows which container owns which cgroup, and yeet's system graph exposes docker and proc nodes through a GraphQL interface in the same runtime that loaded the probes, so the container and process metadata a script needs for that mapping is a query away. Whether the step from cgroup ID to container is one query or a lookup you write depends on the schema, and yeet graph dump prints the schema the graph offers on that machine, which beats trusting a list from a blog post.
What ships as of November 2026 is the runtime and the graph. A scoped per-sandbox lifecycle API, where a sandbox's start opens a recording scope and its exit closes it and hands you the session's record, is the shape platforms ask for first, and it is not something you can install today. You can build that join yourself in a script, and the eBPF and GraphQL modules docs cover the two surfaces you would build it from.
Because under gVisor the workload's syscalls never reach the host kernel. gVisor's own documentation calls it "an application kernel that implements a Linux-like interface", and says that when an application makes a call the platform "redirects the call to the Sentry", and that "the Sentry does not pass system calls through to the host kernel". It adds that "the Sentry will make some host system calls to support its operation". A host eBPF program hooked on exec, open or connect therefore sees a smaller, different set of syscalls made by the Sentry process, and cannot map them back to the workload's own calls. That is the isolation working as designed, and it is the reason gVisor is a weak fit for host-side kernel observation.
gVisor offers its own route instead. Its Runtime Monitoring feature streams "trace points" that "are available for all syscalls and other important events in the system" to "a process running alongside the sandbox, which is isolated from the sandbox for security reasons". Falco is the best-known consumer: the CNCF post Getting started with gVisor support in Falco explains that "it is not possible to install a kernel module or eBPF probe in such an environment", that Falco 0.32.1 first added gVisor support by receiving Sentry events over a Unix domain socket, and that the first release covered "the most important events used in the default rulesets", not all of them. yeet does not consume gVisor's runtime monitor, so on a gVisor fleet that route is Falco or your own monitor process.
Network is partly recoverable. gVisor's networking guide says the Sentry runs its own stack, netstack, and that "data link layer packets are written directly to the virtual device" in the network namespace Docker or Kubernetes set up. The same page says the default mode writes them "with an AF_PACKET socket". Docker joins that namespace to the host with a veth pair, and per veth(4), "packets transmitted on one device in the pair are immediately received on the other device", so a tc program on the host end of the pair sees the frames the sandbox sends. That gives you destinations and byte counts per sandbox, and none of the process or file detail. The --network=host passthrough mode puts the workload on the host stack, and the same page warns that it "decreases the isolation to the host". If the workloads that need kernel visibility are also the ones on gVisor, the practical options are runtime monitoring through Falco, or moving that class of workload to microVMs. Best AI agent tools for DevOps covers gVisor's isolation side for agents.
Three event types, each keyed to the session that caused it: every exec with its arguments and parent, every file open with the path and whether it succeeded, and every connection with the destination and bytes moved. Those three answer most of what an abuse report asks, which is what ran, what it touched and what it talked to. A sandbox audit trail of syscalls, file access and network per session is only as useful as its key, so the cgroup or VM attribution from the previous sections is what turns a host-wide stream into a per-tenant record you can hand to a support engineer.
The same hook that records an event can refuse it, which is where an audit trail becomes enforcement. yeet's Agent Security Kit is the packaged version of this for agents on a host: Flight Recorder keeps a kernel-level audit of every agent action that the agent cannot see or tamper with, File Wall does program-level file access control with the agent attributed, and PII Filter catches PII in encrypted egress before it leaves the NIC. Each ships with default rules, and rules are JavaScript. The Flight Recorder write-up walks through the recording side. These were built for one host running agents, and scoping them per tenant session is the same unshipped lifecycle API described above, so treat them as the building blocks and the per-session scope as your part.
Recording is not a substitute for confinement. A sandbox decides what a workload can do before it acts; a record says what it did after. How to Sandbox an AI Coding Agent on Linux compares the confinement options, and a platform running untrusted code needs both: the runtime's isolation for what you could enumerate in advance, and a kernel-side record for everything you could not.
For CPU, memory and disk, mostly yes, and that is why usage metering is not the headline case for kernel data. The cgroup v2 documentation describes cpu.stat as always reporting usage_usec, user_usec and system_usec, which "account for all the processes in the cgroup"; memory.current as "the total amount of memory currently being used by the cgroup and its descendants"; and io.stat with per-device rbytes, wbytes, rios and wios. If you bill containers by CPU-seconds, memory and IO, read those files and skip eBPF entirely.
The gaps are narrower and specific. The cgroup v2 controller list (CPU, memory, IO, PID, cpuset, device, RDMA, DMEM, HugeTLB, misc) has no network controller, so egress by destination needs a hook such as bpf_skb_cgroup_id() on TC egress. GPU use per process is outside cgroup accounting altogether; yeet's system graph has an nvidia node, and I cannot point you at a shipped tool that attributes GPU time per tenant process. And tenants that share one process, such as a multi-tenant worker serving many customers from one PID, are below cgroup granularity entirely, because the kernel sees one process and the tenant boundary exists only in the application's memory. That last case is where application-level data wins outright.
Split the slow step's time into network vs CPU per step, with process startup broken out, and the split usually answers the ticket. A CI vendor sells speed and cannot see inside a customer's build, so the weekly "your runners are slow" ticket tends to get answered with a guess or a generic caching tip. The question underneath is whether the time went to something the vendor controls (the runner's CPU, its network path) or to something the build chose (fetching 340 MB of dependencies, launching thousands of processes). Build cloud visibility into customer builds means answering that with the build's own numbers, collected from the kernel, without reading the build. Blacksmith's host-side TCX design is one vendor's eBPF CI network observability, and it covers the network share; eBPF CI job observability per step needs the process side too.
ciprof is an eBPF build profiler for GitHub Actions: it reports where each step's time went, network vs CPU vs process startup overhead, broken down by step. Per its how-it-works table, it hooks fentry/do_execveat_common and tp_btf/sched_process_exit for every exec and exit, fentry/tcp_connect, fexit/inet_csk_accept and fentry/tcp_close for every TCP connection with bytes sent and received, and walks parent PIDs back to the Actions runner process to decide which step each process and connection belongs to. As shipped, it runs as two steps in the workflow, one first and one last around the customer's own steps, which makes it a support move: ask the customer filing the ticket to add these two and rerun. The README's quick start, trimmed to the steps:
steps:
- name: Start ciprof
run: yeet run https://github.com/yeet-src/ciprof -- --start
- uses: actions/checkout@v4
- run: npm ci
- run: npm test
- name: ciprof report
if: always()
run: yeet run https://github.com/yeet-src/ciprof -- --report
The --start step forks a profiler into the background, and --report stops it and prints the report; --json emits JSON only, for a support tool to parse. The README's sample report shows the shape of an answer. In it, npm ci takes 4m 12s of a 4m 52s job, spawns 3,241 processes and pulls 340.8 MB, with 847 connections and 338.1 MB coming from registry.npmjs.org:443, and the observation line reads "npm ci spent an estimated 66% of step time on network". Those numbers are the README's illustration, not a benchmark, and they read the way a ticket answer should: the time went to downloading dependencies on every run, which a dependency cache fixes and a faster runner does not.
Read the split in three directions. Network-heavy steps send you to the destinations: a package registry points at the build's caching, while a slow path to a destination every customer uses points at your egress. Process-heavy steps, thousands of launches with short lifetimes, are the build's shape, and Why Is My Build Slow? is the end-user side of that diagnosis. A step that is neither network- nor launch-heavy and still slower on your runner than the customer's baseline is the case that points back at your compute. How to See Every Command Your Build Runs in CI on Linux covers capturing the commands themselves.
ciprof's caveats bound what it can claim, and they belong in any ticket reply built on it. Process timing is wall-clock, not CPU time, so a sleeping process counts the same as a computing one, and a step waiting on a lock reads the same as a step compiling. Network time is inferred from bytes transferred, and the "estimated X% network" line is a heuristic. Step boundaries come from direct children of the runner PID, so a runner using a wrapper can misattribute. Destinations print as IP:port with no reverse DNS, argv is capped at 128 bytes, it needs kernel 5.10 or later and CAP_BPF or root, and it is alpha. Blacksmith's host-side TCX design covers network per job and step with no workflow change; ciprof adds process and per-step time at the cost of two workflow steps.
Whenever the case fails the three-part test above, or the data you need sits somewhere a kernel hook cannot reach. Kernel data about tenant workloads is the right tool for a specific job, and the places it is the wrong tool are as specific:
cpu.stat, memory.current and io.stat from cgroup v2. They already count per container, with no BPF program to load or maintain.SSL_write or SSL_read, which is a different tool with different trust implications for tenant data.Each item marks an edge of the claim. For code you didn't write, the kernel you control is the only vantage point that does not depend on the workload's cooperation, and every case in the list is one where cooperation is available or the vantage point is blocked. Outside those cases, the kernel record is the one to build on.
In four steps, each optional, and the first takes a minute on any Linux host you own. Run a ready-made tool with yeet run github:yeet-src/httpinspect, the first command the yeet docs suggest, to see what the runtime hands back from one host. Then change a tool: rules are JavaScript, so the predicate that decides what to keep or refuse is a file you edit, not a vendor's setting. Then leave it running as a service, "a set of scripts the daemon keeps running for you", enabled to survive a reboot with a restart policy for when a unit goes down. The last step is where a platform differs from a single operator: the kernel data becomes a feature of your platform, shown in your dashboard, collected on the tenant's host and sent wherever your script sends it.
curl -fsSL https://yeet.cx | sh # install the daemon and CLI
yeet run github:yeet-src/httpinspect # a ready-made tool, no sudo
yeet graph dump # the system graph's schema on this host
yeet run never takes sudo; the daemon handles the privileged load. The step platforms want to skip to is the per-sandbox scope: a session-shaped API where your orchestrator opens a recording when a sandbox starts and collects it when the sandbox exits, and that does not ship today. The join lives in a script you write against the cgroup ID and the graph, and the cheapest way to find out whether it is worth building is to run the host-level version on one machine through one incident and check whether the record would have answered the ticket.
For code you didn't write, observability comes from a kernel the platform controls and the tenant's code does not, never from inside the tenant's process. On Firecracker microVMs that is the guest kernel, with the daemon in a guest image you build and the workload kept off root, plus host-side TCX on the VM's veth for traffic, the pattern Blacksmith published. On plain containers it is one host daemon with every event keyed by bpf_get_current_cgroup_id(). On gVisor, host eBPF sees the Sentry and not the workload, so syscall-level visibility comes from gVisor's Runtime Monitoring and Falco. cgroup v2 already covers CPU, memory and IO billing, OpenTelemetry stays the right tool for your own services, and for CI tickets ciprof turns "your runners are slow" into a per-step split of network, process startup and the rest that the customer can see too.
Not without privileges a sandbox should not grant. Privileged BPF operations go through the bpf() syscall and need CAP_SYS_ADMIN, or CAP_BPF on Linux 5.8 and later, so a container running without those capabilities cannot unload a program the host attached. The weak point is configuration: a container started privileged, or a microVM guest where the workload runs as root, puts the monitor back within the workload's reach.
Yes, provided the guest kernel is built with BPF support, because eBPF programs load into whichever kernel the loader runs on, and in a microVM that is the guest kernel. The platform builds both the guest kernel and the guest image, so it controls whether BPF is enabled and whether a daemon is installed. A host-side program cannot see the guest's processes or files, only the VM's traffic on the host network device.
Yes, through gVisor's runtime monitoring and not through Falco's usual kernel driver. Falco 0.32.1 added gVisor support by receiving events from each sandbox's Sentry over a Unix domain socket, with gVisor configured through runsc to send them. The first release covered the events used by Falco's default rules and not every syscall, so check current coverage against the event types your rules depend on.
An execution log is written by the platform's control plane and records that a session ran: tenant, start, stop, exit code and resource totals. A sandbox audit trail records what happened during the session: each process started, each file opened, each connection made. The log answers billing and uptime questions, and the audit trail answers abuse reports and failure tickets, which are about the middle of the session.
No, for plain containers. A host daemon attaches its programs to the shared host kernel and identifies each container by the cgroup ID of the task that triggered an event, so nothing is added to the image, the entrypoint or the environment. MicroVMs differ because the daemon lives in the guest, but the guest kernel and the base layer it boots are the platform's artifacts, so tenants still change nothing as long as the platform owns the layer the daemon ships in.
It scales with the number of matched events, not with total activity, because the in-kernel program filters before anything reaches userspace, so the cost depends on which hooks you attach and how busy they are. As one reference point, the ciprof README puts its own cost at about 2 microseconds per exec or exit event and about 1 microsecond per TCP event. Measure on your own workload mix before committing to a fleet rollout.
Yes. Connection hooks and traffic-control programs see the destination IP address, port and byte counts for every connection regardless of encryption, which is enough to answer most egress questions. What they do not see is the plaintext inside the TLS session. Resolving IP addresses back to names is a separate step, and tools differ: ciprof prints raw IP and port, while Blacksmith's design attributes domains on a best-effort basis.
No. Isolation decides what a workload is able to reach before it acts, and monitoring records what it did. A platform running untrusted code needs both, because isolation alone permits everything inside its boundary without reporting any of it, and monitoring alone reports an action only after it completes. The same eBPF hook can refuse an event as well as record it, which narrows the gap without closing it.
--network=host uses the host stack instead and "decreases the isolation to the host".release-20220704.0 or later; the first release covered "the most important events used in the default rulesets", not every syscall.cpu.stat always reports usage_usec, user_usec and system_usec, which "account for all the processes in the cgroup"; memory.current is "the total amount of memory currently being used by the cgroup and its descendants"; io.stat carries per-device rbytes, wbytes, rios, wios, dbytes and dios; the controller list covers CPU, memory, IO, PID, cpuset, device, RDMA, DMEM, HugeTLB and misc, with no network controller.bpf_get_current_cgroup_id() returns "the current cgroup id based on the cgroup within which the current task is running"; bpf_skb_cgroup_id() returns "the cgroup v2 id of the socket associated with the skb", "can be used on TC egress path, but not on ingress", and needs CONFIG_SOCK_CGROUP_DATA; the companion capabilities(7) page documents CAP_BPF as added in Linux 5.8 "to separate out BPF functionality from the overloaded CAP_SYS_ADMIN capability".fentry/do_execveat_common, tp_btf/sched_process_exit, fentry/tcp_connect, fexit/inet_csk_accept and fentry/tcp_close; attributes to steps by parent-PID walk from the runner; process timing is wall-clock, network time inferred from bytes; argv capped at 128 bytes; no reverse DNS; kernel 5.10+ and CAP_BPF or root; overhead "~2µs per exec/exit event, ~1µs per TCP event"; status alpha.yeetd.See your machine the way the kernel sees it. Start on one host you own:
curl -fsSL https://yeet.cx | sh
Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.