L7 Enforcement Without a CNI Migration

Jacob Pradels
Jacob Pradels··7 min read

Founding engineer at yeet, working on kernel-side observability and the tooling around it. I write about eBPF, Linux internals, and why your telemetry bill looks the way it does.

The question that brings most people here is narrower than "which CNI should I run." It is usually some version of: I want to make decisions about L7 traffic at wire speed, and I have been told that means eBPF. Does that mean I have to replace my cluster networking to get it?

That question deserves a straight answer, and the straight answer has two halves. If what you want is a faster, iptables-free datapath for pod and service traffic across your whole cluster, then yes — that is a CNI-level concern, and Cilium's high-performance CNI positioning is pointing at something real. If what you want is to inspect and act on application-layer traffic at specific points in your infrastructure, that is a different layer, and the answer is no.

We build yeet, which lives on the second half of that split. So before anything else, the part where we are not the alternative.

Cilium's datapath, stated the way its authors would state it

Cilium replaces kube-proxy with eBPF programs that handle ClusterIP, NodePort, LoadBalancer, externalIP, and hostPort service load-balancing — in its own words, "all without relying on iptables or netfilter" (docs). This is not a thin wrapper over the same primitives. It is a different datapath, and the mechanisms are individually strong:

  • Socket-LB translates service IPs at the socket layer during connect(), sendmsg(), and recvmsg(), so pod-to-service traffic never pays for lower-layer NAT at all. The translation happens once, at connection setup, instead of per packet.
  • NodePort XDP acceleration (loadBalancer.acceleration=native) handles NodePort requests inside the network driver when the backend is remote. Cilium's framing: "Cilium is able to process those requests right out of the network driver layer. This helps to reduce latency and scale-out of services given a single node's forwarding capacity is dramatically increased."
  • DSR lets backends reply directly to clients rather than hairpinning through the ingress node, with three dispatch modes — IPv4 Option / IPv6 Extension Header, Geneve encapsulation, and IPIP.
  • Maglev consistent hashing gives deterministic backend selection across nodes, so every node independently agrees on where a flow goes.
  • netkit device mode replaces veth pairs and, per Cilium's tuning guide, "reduces the datapath overhead for network namespaces down to zero."
  • eBPF host-routing exists to "fully bypass iptables and the upper host stack."
  • BIG TCP raises GSO/GRO packet sizes to "reduce the number of times the stack is traversed," aimed explicitly at "100Gbit/s and beyond."

Stack those together and you get the fastest generally available path for pod and service traffic in Kubernetes today. We do not think that is arguable, and we are not going to hedge it. Cilium also gets something structurally right: because it is the CNI, its policy engine knows pod identity for every flow in the cluster, without anyone having to place a probe anywhere. Identity-aware policy that covers everything by default is a real property, and it is not one you get by instrumenting chokepoints.

Where the top-end wins get expensive

The gap between "Cilium is fast" and "Cilium is fast in my cluster" is a list of preconditions, and Cilium documents them honestly. That is what makes them worth discussing — none of this is inference on our part.

The fastest modes are not the defaults, and turning them on is a cluster-level event. From the tuning guide:

  • netkit is beta, requires kernel 6.8 or newer, and "cannot upgrade in-place" on an existing cluster — enabling it requires new Pod creation, and veth and netkit cannot run in parallel during a transition.
  • eBPF host-routing is "Incompatible with Istio", and incompatible with anything relying on netfilter hooks, GKE Workload Identities included. There are workarounds (bpf.hostLegacyRouting=true, Local Redirect Policy), but they walk back part of the speed you enabled it for.
  • BIG TCP requires host-routing, kube-proxy replacement, eBPF masquerading, tunneling and encryption disabled, and a NIC from the mlx4, mlx5, or ice families. Also no in-place upgrade — Pods restart.
  • Distributed LRU and the eBPF clock probe are both documented as disruptive to enable on-the-fly.
  • Cilium's own recommended combined configuration is thirteen Helm flags, and the minimum kernel for that full stack is 6.8.

The limitations page for the kube-proxy replacement is similarly candid, and the entries there are the kind that surface in production rather than in evaluation. NFS and SMB mounts targeting service cluster IPs may break without four specific kernel commits backported. TCP Fast Open is incompatible with DSR NodePort mode. SCTP is unsupported outside "a few basic cases." Unconnected UDP sockets and UDP aborts fail to clean up reverse SK maps, which the docs describe as causing gradual exhaustion — mitigated with socketLB.hostNamespaceOnly=true. And there is a flat recommendation worth reading twice: "It is highly recommended not to expose a backend endpoint via multiple VIPs."

None of these are indictments. Every one is the ordinary cost of owning a datapath, and a project that publishes them in this detail is behaving well. But they add up to a specific claim: the headline performance profile is available to clusters that can commit to a kernel floor, a NIC family, a service-mesh decision, and a Pod-recreating rollout. Many clusters can. If yours can, the ceiling is genuinely high.

The part we want to isolate is what happens when your reason for wanting eBPF was never the pod datapath in the first place.

What we do instead, and what it costs

yeet is a Linux CLI platform for building eBPF tools — monitoring, debugging, and enforcement — as programs you write and ship, rather than features you configure. The piece relevant here is that it does HTTP/2 header inspection at XDP, before a socket buffer exists. Our published figures for that path: median decision latency under 200 ns, roughly 5M header decisions per second per core, about 4,000 user-agent patterns held in rules, at under 1% CPU. Those are our numbers, from our own measurements; we are not putting them next to a Cilium figure because Cilium's docs do not publish comparable throughput or latency benchmarks, and we are not going to manufacture the other side of a table.

The structural difference is scope. Enforcement at XDP on the hosts you choose does not require the cluster to agree on anything. No kernel floor for the whole fleet, no NIC family requirement, no Pod recreation, no decision about whether your service mesh survives the change. You put a program where the traffic is and it makes decisions there.

Now the honest side of the ledger:

  • We are not a CNI, and we are not going to become your CNI. No pod networking, no service load-balancing, no IPAM. If those are the problem, this post is not the answer to it — Cilium or another CNI is.
  • You write a tool, not a flag. Cilium's whole configuration surface is Helm values. Ours is a program you author against our API. That is more expressive and more work, and for teams that want a supported knob rather than a codebase, more work is the wrong trade.
  • Linux only. No portability story beyond that.
  • Enforcement at chokepoints is not enforcement everywhere. Cilium's cluster-wide identity-aware policy has coverage properties that a set of deliberately placed programs does not. If your threat model needs default-deny across every pod-to-pod flow, that is a CNI-shaped problem.

Choose Cilium when

We would tell you to pick Cilium, without qualification, in these cases:

  • Your problem is the pod and service datapath. Getting iptables out of the path for east-west traffic is what Cilium is for. Nothing at our layer addresses it.
  • You want one system for networking, policy, and observability. A single project covering CNI, NetworkPolicy, and Hubble is an operational simplification that is worth real money, and assembling equivalent coverage from separate tools is worse unless you have a reason.
  • You control your kernel and your hardware. If you can standardize on 6.8+ and mlx5 or ice NICs, the tuning matrix stops being a list of obstacles and becomes a checklist you complete once. That is the configuration Cilium's best numbers describe, and you can actually get there.
  • You need identity-aware policy across the entire cluster. Cluster-wide default coverage is a property of being the CNI. We do not have it and are not claiming it.
  • You want configuration, not code. Thirteen Helm flags is a real answer when the alternative is owning a program.

Choose yeet when the L7 decision is the thing you care about, the traffic you want to act on passes through hosts you can reach, and re-platforming cluster networking to get there is out of proportion to the problem — especially if you are on a mixed or older kernel fleet, on NICs outside the BIG TCP list, running Istio, or simply unwilling to recreate every Pod for a datapath change you did not need.

These are different layers. The mistake worth avoiding is concluding that because the fast path is written in eBPF, the entry price is a CNI migration. Sometimes it is. Often the thing you actually wanted was a decision made a little earlier in the packet's life, and that you can have without moving the floor underneath your cluster.