How to Find Which Tenant Is the Noisy Neighbor

Jacob Pradels
Jacob Pradels
·27 min read

Founding engineer at yeet, working on kernel-side observability and the tooling around it. I write about eBPF, Linux internals, and why your telemetry bill looks the way it does.

Last updated: November 2026

I work on yeetd, the daemon in yeet that loads eBPF programs into a Linux kernel and hands their records to JavaScript, and the scheduler is the part of the kernel I explain most often. I have never run a container platform or answered a performance ticket for one, so I cannot tell you how to word the reply. I can tell you what the kernel records while one tenant waits for a CPU another tenant holds, and which of those records you have to collect before the reply is evidence instead of a guess.

Short answer. Answer the ticket in three reads, in this order. Check steal time in /proc/stat on the host: if it rose, the slowdown came from under the host and no tenant caused it. Then read the customer's cgroup. cpu.pressure (Pressure Stall Information, or PSI) shows whether its tasks stalled waiting for CPU in the complaint window, and nr_throttled in cpu.stat shows whether the stall was its own cpu.max quota running out, the only thing that counter counts. If it stalled and was not throttled, the cause was contention, and only the scheduler names the noisy neighbor: eBPF hooks on sched_wakeup and sched_switch that record, for each long wait, the cgroup of the task holding the CPU when the waiting task finally got it. PSI names the victim, cpu.stat and io.stat rank suspects, and only the scheduler names a culprit. All of it comes from the host kernel, with nothing added to the tenant's container.

A customer says their app got slow on my platform and the host dashboards look fine. What do I check first?

Check, in order, whether the host lost CPU to its own hypervisor, whether the tenant hit its own limit, and only then whether a neighbor took the CPU it needed. Host dashboards look fine in these tickets because they show utilization, and utilization does not show waiting. The Kubernetes announcement of PSI metrics going GA in v1.36 puts it directly: "A node may report XX% (below 100%) CPU utilization while certain tasks are experiencing severe latency due to scheduling delays." A host at 60% can have one tenant's threads queued behind another tenant's on the same four cores, and the host graph averages that away.

In a May 2026 thread on Railway's community forum, a customer reported that every backend container across two environments began running pure-CPU work "3–4× slower" at the same moment, and asked whether their compute had been "migrated, re‑quota'd, or rescheduled onto contended hosts". Railway staff found "a noisy-neighbor issue" where the host "became partially I/O-saturated", and said it "was not a quota or migration change on our side". The customer replied that they were still at four times their baseline, staff could not see anything further, and the thread closed itself for inactivity. Neither side had a per-tenant record of who waited, on what, and behind whom.

That record has four layers, and only the last names who caused the slowdown. PSI and cpu.stat are per-cgroup counters: they describe what happened to one cgroup, not who did it. Only the last row describes two tenants at once, and nobody ships it:

SignalWhere it livesQuestion it answersNames the noisy neighbor?Needs eBPF?
cpu.pressure, io.pressure, memory.pressure (PSI)Every cgroup directory; kubelet /metrics/cadvisor and Summary API, GA in Kubernetes v1.36Did this tenant's tasks stall, on which resource, for how longNo. The noisy neighbor stalls tooNo
cpu.stat nr_throttled, throttled_usecEvery cgroup with the cpu controller enabledDid this tenant run out of its own cpu.max quotaNo, it rules the tenant's own limit in or outNo
cpu.stat usage_usec, io.stat rbytes / wbytesEvery cgroup directoryWhich tenants consumed the most CPU or disk bytes in the windowSuspects only: correlation, with no CPU or timing detailNo
steal in /proc/statThe hostDid the hypervisor take CPU away from this hostNo, it names your providerNo
sched_wakeup + sched_switch, keyed by cgroup IDAn eBPF program on the hostWhose task held the CPU while this tenant's task waitedYes, per wait, per CPUYes

Steal rules every tenant out at once. If your container hosts are cloud VMs and steal rose during the window, no tenant took the lost time, the conversation is with your cloud provider, and moving the customer to another host is the fix that matches the evidence. Steal is counted per CPU, not per cgroup, so PSI, not steal, says which tenant was hurt most.

The first three layers are files. The cgroup v2 documentation says /proc/$PID/cgroup "lists a process's cgroup membership", in the form 0::$PATH, so any PID from the customer's container gets you its cgroup directory on the host. Read these at the start of the window and again at the end, and diff the counters:

PID=<any pid in the customer's container>          # from your runtime, e.g. a process list on the host
CG=/sys/fs/cgroup$(cut -d: -f3 /proc/$PID/cgroup)  # the 0::$PATH line, on a cgroup v2 host

awk '/^cpu /{print "host steal ticks:", $9}' /proc/stat           # 8th value after "cpu", in USER_HZ
grep -E 'nr_periods|nr_throttled|throttled_usec' $CG/cpu.stat     # the tenant's own quota
cat $CG/cpu.pressure $CG/io.pressure $CG/memory.pressure          # PSI: did it stall, and on what

$9 is the eighth number after the cpu label, which proc_stat(5) defines as steal, "the time spent in other operating systems when running in a virtualized environment". Diff each PSI file's total, not avg10; the PSI section below says why.

Two more sources of lost CPU belong to the platform. Kernel threads such as kworker and ksoftirqd sit in the root cgroup (0::/, ID 1 on my test VM), so a scheduler record whose holder is the root cgroup is the platform's own work. Softirqs are harder: the kernel's hacking guide says pending softirqs run "whenever a system call is about to return to userspace, or a hardware interrupt handler exits", on whatever task is current, so they stretch that task's turn without a context switch and the delay is charged to its cgroup. The cgroup v2 doc offers irq.pressure for "IRQ/SOFTIRQ" stall, but /proc/pressure/irq was absent on my Debian 13 cloud arm64 kernel, built without CONFIG_IRQ_TIME_ACCOUNTING.

Kubernetes v1.36 made PSI metrics GA. Is that enough to find noisy neighbors?

It is enough to find the victims, rank stalled tenants and alert on them, and not enough to name a culprit, because PSI measures a cgroup's stall and says nothing about its cause. The kernel PSI documentation defines the some line as "the share of time in which at least some tasks are stalled on a given resource" and full as the share in which "all non-idle tasks are stalled on a given resource simultaneously", and each cgroup directory "contains cpu.pressure, memory.pressure, and io.pressure files" in the same format. A tenant whose threads waited 40% of the last ten seconds has a 40% stall whether the CPU went to a neighbor, to a kernel thread or to its own quota running out.

As of v1.36 the Understanding PSI Metrics page says "the KubeletPSI feature gate is locked to true and cannot be disabled", and the GA announcement lists the requirements: Linux 4.20 or later, cgroup v2, and a kernel "compiled with CONFIG_PSI=y" that was not booted with psi=0. The data reaches you through the kubelet's Summary API and in Prometheus format at /metrics/cadvisor, where the docs use container_pressure_cpu_waiting_seconds_total; KEP-4205 lists it alongside container_pressure_cpu_stalled_seconds_total and the matching memory and IO series. If you already scrape cAdvisor, per-container PSI is a query away with no agent and no BPF.

Kubernetes' own CPU pressure demo shows the gap: a pod running stress --cpus 1 under a 500m limit, with no neighbor, so the pressure it shows is its own quota throttling it. PSI reports that the same way it would report a neighbor taking the CPU, which is why a PSI alert cannot be your reply to a customer.

The noisy neighbor is stalled too. On a 12-vCPU arm64 VM running Debian 13's 6.12 kernel, I pinned two unlimited busy loops in two cgroups to the same CPU for eleven seconds. Each cgroup ran for 5.49 seconds and each recorded 5.48 to 5.49 seconds of cpu.pressure some stall, within a few milliseconds of each other. Two CPU-bound tenants sharing a core each see the other as the noisy neighbor, PSI reports them identically, and only the latency-sensitive one's customer files a ticket.

The averages cover "ten, sixty, and three hundred second windows", so a five-second spike hours ago is gone from avg10; the kernel doc exports total stall time in microseconds "to allow detection of latency spikes which wouldn't necessarily make a dent in the time averages". Diff total over the complaint window. And PSI supports triggers: write some 150000 1000000 to a pressure file and poll() wakes you when 150 ms of stall accumulates in any one-second window, with unprivileged monitors limited to windows that are "a multiple of 2s". A trigger on a tenant's cpu.pressure is the moment to start the scheduler probe below.

nr_throttled is flat but the customer's latency spiked. Is it their CPU limit or a noisy neighbor?

Most likely a neighbor or the host, because nr_throttled counts only one cause of waiting: the tenant's own CFS bandwidth quota running out. The kernel's CFS bandwidth control doc says "Within each given 'period' (microseconds), a task group is allocated up to 'quota' microseconds of CPU time", and once the quota is assigned "any additional requests for quota will result in those threads being throttled", unable to run "until the next period when the quota is replenished". The cgroup v2 doc says the throttling fields in cpu.stat "only account for throttling caused by this cgroup's own bandwidth limit". Preemption by another tenant's task is not a quota event, so it never moves the victim's nr_throttled.

The two cases look identical in PSI and opposite in cpu.stat, and you can watch both on any cgroup v2 host in about twenty seconds. As root:

cd /sys/fs/cgroup && mkdir -p nntest/victim nntest/noisy nntest/capped
echo +cpu > nntest/cgroup.subtree_control
spin() { sh -c "echo \$\$ > nntest/$1/cgroup.procs; exec taskset -c $2 timeout 10 sh -c 'while :; do :; done'" & }

spin victim 3; spin noisy 3; sleep 10                  # A: two tenants share CPU 3, no limits
cat nntest/victim/cpu.pressure; grep -E 'usage|throttled' nntest/victim/cpu.stat

echo "20000 100000" > nntest/capped/cpu.max            # B: one tenant alone, capped at 20% of a CPU
spin capped 4; sleep 10
cat nntest/capped/cpu.pressure; grep -E 'usage|throttled' nntest/capped/cpu.stat
wait; rmdir nntest/victim nntest/noisy nntest/capped nntest

taskset -c pins each loop to one CPU so the contention is guaranteed, and cpu.max takes $MAX $PERIOD in microseconds. On the same 6.12 VM, the victim in case A ran 4.99 seconds, recorded 4.99 seconds of some stall, and finished with nr_throttled 0. The capped tenant in case B ran 2.02 seconds, recorded 7.96 seconds of stall, and finished with nr_throttled 100 and throttled_usec 7928420. PSI said "stalled" both times. Only cpu.stat told the two apart, and case B matches the CFS bandwidth doc's expectation that for a CPU-bound group "nr_periods roughly equal nr_throttled".

So a rising nr_throttled is the "raise your limit" reply. A flat one is weaker than it looks: four cases keep it flat while the tenant's own configuration is still to blame:

  • A limit on a parent cgroup: the counters are "non-hierarchical", so a cap on a pod or slice above the container throttles it without moving the container's nr_throttled. Read the parent's cpu.stat, or throttled_usec in the child's cpu.stat.local, which "may include throttling inherited from ancestor cgroup bandwidth limits".
  • The tenant's own threads queueing behind each other: a service running more busy threads than it has CPUs waits on itself, which the scheduler record shows as a holder in the same cgroup.
  • A lower cpu.weight under contention: weights split CPU "by adding up the weights of all active children and giving each the fraction matching the ratio of its weight against the sum", so a tenant can get less than it wants with no quota involved.
  • Tasks outside the fair class: the throttling stats "account for only the processes under the fair-class scheduler", so SCHED_FIFO and SCHED_RR work never appears in them.

The single-host version, reading one process's run_delay, is covered in How to Find Which Process Is Slowing Down Your Linux Machine.

Discover yeet apps

browse all
pktscope71
by yeet-src·updated Oct 2026#245

Wireshark-style terminal packet analyzer on eBPF/TCX — live packet list, protocol detail tree, section-colorized hex; works on raw-IP tunnel devices

How do I identify which container is the noisy neighbor from the scheduler?

Record a timestamp when a task becomes runnable, and when the scheduler finally switches it in, record the wait along with the cgroup of the task switched out. Three tracepoints in the kernel's sched.h carry everything needed: sched_wakeup and sched_wakeup_new fire with the task becoming runnable, and sched_switch fires with prev and next task pointers plus prev_state. A prev still in the running state at the switch was preempted, so its wait starts there too. When the waiting task is next, the wait is now minus its timestamp, and prev is the task that held that CPU immediately before it got it. Read the cgroup ID of both and you have a victim, a holder and a duration.

The cgroup ID is the tenant key, by the same mechanism covered in How to See What Untrusted Customer Code Does in Your Sandboxes: each container gets its own cgroup, and the kernel hands that ID to eBPF without the container running anything of yours. A scheduler event involves two tasks, so you read the ID from each task_struct. On the 6.12 kernel I tested on, that ID matched the inode number stat -c %i printed for each cgroup's directory, which lets you map IDs back to tenants from a shell.

Netflix published this design for Titus in 2024. The Noisy Neighbor Detection with eBPF post (a syndicated copy of the Netflix TechBlog original) uses exactly the three hooks, reads the cgroup ID as task->cgroups->dfl_cgrp->kn->id under the bpf_rcu_read_lock() kfunc, tags each preemption by whether the cause was "a process within the same container (or cgroup), a process in another container, or a system service". It also names the throttling trap: "if a container is at or over its cgroup CPU limit, the scheduler will throttle it, resulting in an apparent spike in run queue latency", which is why they read the preemption cause alongside the latency. The post shows the probe code only in fragments and offers no installable detector.

The off-the-shelf tools stop one step short. The bcc runqslower man page offers -P to "show previous task comm and TID" next to each long wait, the right mechanism with a process name instead of a tenant, and its filters are -p PID and -t TID. The libbpf-tools runqlat takes -c CG to "Trace process in cgroup path", which gives you the victim tenant's wait histogram and no holder at all. If run queue latency itself is new ground, Top 5 Kernel Metrics Your APM Agent Misses covers runqlat for code you own.

yeet ships no noisy-neighbor tool, so what follows is a sketch of the script you would write, in two halves: ordinary BPF C compiled to a .bpf.o, and an ES module that imports it. The full BPF program the C fragment below is cut from compiles with clang, passed the verifier and attached on that 6.12 kernel through a plain libbpf loader. The JavaScript half follows the documented API and has not been run, so treat both as a starting point, not a product.

// Sketch: runq_culprit.bpf.c, the sched_switch half.
// on_wakeup and on_wakeup_new (tp_btf/sched_wakeup, sched_wakeup_new) store
// bpf_ktime_get_ns() in `enqueued`, keyed by pid, the same way mark_runnable does.
const volatile __u64 min_wait_ns = 1000000;     /* only waits of 1 ms or more leave the kernel */

static __always_inline __u64 task_cgid(struct task_struct *t)
{
	__u64 id;
	bpf_rcu_read_lock();
	id = t->cgroups->dfl_cgrp->kn->id;
	bpf_rcu_read_unlock();
	return id;
}

SEC("tp_btf/sched_switch")
int BPF_PROG(on_switch, bool preempt, struct task_struct *prev, struct task_struct *next)
{
	if (prev->__state == 0)                         /* preempted, still runnable: its wait starts now */
		mark_runnable(prev);

	__u32 pid = next->pid;
	__u64 *ts = pid ? bpf_map_lookup_elem(&enqueued, &pid) : NULL;
	if (!ts)
		return 0;
	__u64 wait = bpf_ktime_get_ns() - *ts;
	bpf_map_delete_elem(&enqueued, &pid);
	if (wait < min_wait_ns)
		return 0;

	struct wait_event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
	if (!e)
		return 0;
	e->victim_cgid = task_cgid(next);               /* the tenant that waited */
	e->holder_cgid = task_cgid(prev);               /* whoever had this CPU just before it */
	e->wait_ns = wait;
	e->victim_pid = pid;
	e->holder_pid = prev->pid;
	e->cpu = bpf_get_smp_processor_id();
	bpf_ringbuf_submit(e, 0);
	return 0;
}
// Sketch, not a shipped tool: total wait per (victim, holder) cgroup pair on one host.
import probe from "./runq_culprit.bpf.o";
import { RingBuf } from "yeet:bpf";

// tp_btf programs auto-attach by section name when start() loads the object
const control = await probe
  .bind("events", { kind: "ringbuf", btf_struct: "wait_event" })
  .start();

const paths = new Map();   // cgroup ID -> cgroup path
const waits = new Map();   // "victim waited behind holder" -> total wait, ns (BigInt)

async function cgroupPath(cgid, pid) {
  if (!paths.has(cgid)) {
    const { data } = await yeet.graph.query(`{ proc(pid: ${pid}) { cgroups { pathname } } }`);
    paths.set(cgid, data?.proc?.cgroups?.[0]?.pathname ?? `cgroup ${cgid}`);
  }
  return paths.get(cgid);
}

const events = new RingBuf(control, "events");
await events.subscribe(async ({ wait_event: e }) => {
  const victim = await cgroupPath(e.victim_cgid, e.victim_pid);
  const holder = e.holder_cgid === e.victim_cgid ? "itself" : await cgroupPath(e.holder_cgid, e.holder_pid);
  const key = `${victim} waited behind ${holder}`;
  waits.set(key, (waits.get(key) ?? 0n) + e.wait_ns);
});

setInterval(() => {
  for (const [pair, ns] of waits) console.log(`${pair}: ${ns / 1_000_000n} ms`);
  waits.clear();
}, 10_000);

The cgroup ID and the wait arrive as 64-bit BigInt values, and the system graph resolves a PID to the cgroup path the kernel lists for it, which is the join from ID to container. The graph also carries docker nodes if you want the container's name. A "waited behind itself" line is the tenant's own threads; a line naming another tenant's path, growing during the complaint window while the victim's nr_throttled stays flat, is the attribution the customer was asking for.

The neighbor is hammering the disk, not the CPU. What do io.pressure and io.stat show?

io.pressure shows the victim stalled on IO, and io.stat shows who moved the bytes, but no scheduler event names the IO culprit, because a task waiting on a disk is blocked rather than runnable and never sits in a run queue behind anyone. That is the Railway case: a host "partially I/O-saturated", with the customer's CPU-bound code slowing alongside it. The cgroup v2 doc's io.stat gives per-device lines "keyed by $MAJ:$MIN device numbers" with rbytes, wbytes, rios, wios, dbytes and dios, so the suspects are the tenants whose counters grew fastest on the device the victim uses. That ranking is correlation, the same layer as usage_usec for CPU, and it is the strongest attribution the cgroup files offer for disk.

Writeback makes the ranking less certain than it looks. The cgroup v2 doc says "an inode is assigned to a cgroup and all IO requests to write dirty pages from the inode are attributed to that cgroup". cgroup writeback "requires explicit support from the underlying filesystem" and is implemented on ext2, ext4, btrfs, f2fs and xfs; "on other filesystems, all writeback IOs are attributed to the root cgroup". Where several cgroups write one inode at once, "a significant portion of IOs are likely to be attributed incorrectly". A tenant flushing a large buffered write can show up under its own cgroup, another tenant's or the root, depending on the filesystem and on who dirtied the inode first.

Memory follows the same victim-only pattern. A tenant's memory.pressure rises when its tasks stall on reclaim, and nothing in that file says whose allocation forced the reclaim. Why Is My Container Using More Memory Than the Process Inside It? covers memory.pressure and memory.stat for the memory side.

What can't cross-cgroup scheduler attribution see?

It sees who held a CPU, and several real causes of a slow tenant never hold one:

  • More than the last holder: prev is the task switched out immediately before the victim ran. If three tenants' tasks took turns during a 20 ms wait, the record names the last one. Netflix's tag has the same shape. It becomes attribution only in aggregate, over many waits.
  • Waits that moved CPUs: a task can wake on one CPU and run on another after load balancing, so the holder is the last occupant of the CPU it ran on, not of the one it queued on.
  • SMT siblings: two hyperthreads share one core, and the kernel's core scheduling doc describes workloads that do or do not "need the same hardware resources of the shared core". A neighbor on the sibling thread slows the victim with no scheduler event on the victim's CPU.
  • Cache and memory bandwidth: a neighbor thrashing the last-level cache or saturating memory bandwidth slows everyone on the socket without preempting anyone. On x86 parts with Intel RDT or AMD QoS, the resctrl filesystem exposes llc_occupancy, mbm_total_bytes and mbm_local_bytes per monitoring group.
  • Disk, memory and lock waits: a blocked task is not runnable, so IO, reclaim and futex waits never produce a run queue record.
  • Tenants that share one process: a multi-tenant worker serving many customers from one PID is one cgroup to the kernel, and only the application knows which request belonged to whom.
  • gVisor and microVM tenants: under gVisor the host sees the Sentry's threads, and in a microVM the host's run queues hold vCPU threads rather than the guest's tasks; the sandbox post's runtime table covers where a probe sees what.

Keep the reply's wording to what the record supports: "your tasks waited 1.8 seconds behind another tenant's tasks on these CPUs in this window" is a statement the data backs, and "tenant X slowed you down" is one it does not when an SMT sibling or a bandwidth-hungry neighbor is the cause.

Can I leave noisy neighbor detection running on every host without slowing it down?

Yes, if you filter in the kernel and measure the per-hook cost on your own workload first, because sched_switch fires on every context switch on every CPU. The ctxt line in /proc/stat, "the number of context switches that the system underwent", read twice and diffed, is your event rate per host, and the rate times your per-hook cost is the overhead. Netflix measured its version with bpftop at "less than 600 nanoseconds to each sched_* hook" and found the difference with profiling on and off "not measurable in milliseconds" for a Java service, after early exits for PID 0 and per-cgroup, per-CPU rate limiting. The runqslower man page warns that scheduler event frequency "may be high enough that the overhead of this tool becomes significant" and says to measure first.

The sketch's min_wait_ns is the main lever. Every wakeup still costs a map update, but only waits over the threshold reserve ring buffer space and wake the JavaScript side, so on a healthy host almost nothing crosses into userspace.

Running it on every host needs no change to any tenant image. Run the script by hand with yeet run on one host and check that its output would have answered last week's ticket, then make it a service, "a set of scripts the daemon keeps running for you", and roll it across hosts with yeet service apply, which makes "a service match a file, rolling the change out". yeet's FAQ says "Nothing leaves the host unless a script sends it, and nothing is retained by yeet itself", so where the per-tenant numbers go is a line in your script, and retention is yours to build or buy.

When should I use Coroot, Kubernetes PSI metrics or tenant telemetry instead?

Whenever the question in front of you is not "whose task held the CPU while this tenant waited", the only question the scheduler hooks answer better than what you already run:

  • Alerting on stalled tenants: Kubernetes v1.36 kubelet PSI metrics from /metrics/cadvisor, or cpu.pressure directly. GA, no BPF, and they rank victims across the fleet.
  • "Did they hit their limit": nr_throttled and throttled_usec, plus cpu.stat.local or the parent cgroup's cpu.stat. No probe needed.
  • A shipping product: Coroot collects per-TGID CPU delay over Netlink, aggregates it per container, adds container_resources_cpu_throttled_seconds_total so throttling is not mistaken for contention, and in its noisy-neighbor example concludes that "the stats-aggregator has caused the issue" by "comparing these metrics". It names a suspect by consumption and timing, not by which task held the CPU during a particular wait.
  • Services you wrote: the Azure noisy neighbor antipattern says to "include the tenant's identifier in the telemetry so that you can aggregate request unit usage for each tenant". That works when requests pass through your code; a container host runs the tenant's own process, with no request path of yours to stamp.

The bottom line: read PSI for the victim, cpu.stat for the alibi, sched_switch for the culprit

A "my app got slow on your platform" ticket has three possible answers, and the host kernel holds evidence for each. Steal in /proc/stat covers your host. nr_throttled and throttled_usec in cpu.stat cover the customer's own cpu.max quota, and a neighbor's preemption never moves them. cpu.pressure and io.pressure (PSI), now GA as per-container metrics in Kubernetes v1.36, say which tenants stalled. usage_usec and io.stat, and Coroot's side-by-side comparison, rank suspects. Naming the noisy neighbor takes a hook on sched_wakeup and sched_switch that records the cgroup ID of the task holding the CPU during each long wait: the design Netflix described for Titus without an installable detector, and one a platform can write as a yeet script and leave running on every host without touching a tenant's container.

Frequently asked questions

What is a noisy neighbor on a container platform?

A noisy neighbor is a tenant whose workload uses enough of a shared host's CPU, disk, memory bandwidth or cache to slow other tenants on the same host. Netflix's definition for its Titus platform is a container or system service that heavily uses the server's resources, causing performance degradation in adjacent containers. The victim usually sees higher latency with no change in its own code or configuration.

What is the difference between PSI some and full?

The some line is the share of time at least one task in the cgroup was stalled on the resource; full is the share of time all non-idle tasks in it were stalled at once. For a cgroup with one busy thread the two are equal. CPU full is undefined at the system level and reported as zero there, but it is meaningful per cgroup.

Does cpu.weight stop a noisy neighbor?

Only partly. cpu.weight divides CPU among cgroups that want it at the same time, in proportion to their weights, so a low-weight batch tenant gives way to a high-weight latency-sensitive one under contention. It does not cap anyone when the CPU is free, it does not help against cache or memory bandwidth contention, and two tenants with equal weights still share a busy core equally.

Sources

  • PSI: Pressure Stall Information (Linux kernel documentation): "some" is "the share of time in which at least some tasks are stalled on a given resource" and "full" the share in which "all non-idle tasks are stalled on a given resource simultaneously"; averages over ten, sixty and three hundred seconds plus a total in microseconds "to allow detection of latency spikes which wouldn't necessarily make a dent in the time averages"; triggers via poll() with windows from 500ms to 10s, unprivileged windows "a multiple of 2s"; each cgroup directory "contains cpu.pressure, memory.pressure, and io.pressure files".
  • Control Group v2 (Linux kernel documentation): cpu.stat always reports usage_usec, user_usec and system_usec, plus nr_periods, nr_throttled, throttled_usec, nr_bursts and burst_usec when the controller is enabled, which "only account for throttling caused by this cgroup's own bandwidth limit"; cpu.stat.local throttled_usec "may include throttling inherited from ancestor cgroup bandwidth limits"; io.stat per $MAJ:$MIN device; cgroup writeback on ext2, ext4, btrfs, f2fs and xfs, root cgroup elsewhere; irq.pressure for IRQ/SOFTIRQ.
  • CFS Bandwidth Control (Linux kernel documentation): a group gets up to "quota" microseconds per "period", after which "threads being throttled" cannot run "until the next period"; default period 100ms, minimum quota 1ms, maximum period 1s; quota moves to CPUs in slices, sched_cfs_bandwidth_slice_us default 5ms; for CPU-bound constrained groups "nr_periods roughly equal nr_throttled".
  • Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA (Kubernetes blog, May 12, 2026): PSI at "the node, pod, and container levels"; requires Linux 4.20+, cgroup v2 and CONFIG_PSI=y without psi=0; no feature gate opt-in as of v1.36; available at /metrics/cadvisor and the Summary API; alpha in v1.33, beta in v1.34; kernel PSI overhead measured at 0.037 to 0.125 cores on a 4-core node at 80-pod density.
  • Understanding PSI Metrics (Kubernetes documentation): "the KubeletPSI feature gate is locked to true and cannot be disabled" from v1.36; CPU pressure demo runs stress --cpus 1 with a 500m limit and observes container_pressure_cpu_waiting_seconds_total at /metrics/cadvisor; avg10, avg60 and avg300 are percentages of wall-clock time and total is a cumulative stall counter; requires Linux 4.20+, CONFIG_PSI=y and cgroup v2.
  • Noisy Neighbor Detection with eBPF (Netflix Technology Blog, September 2024, syndicated copy): hooks on sched_wakeup, sched_wakeup_new and sched_switch; cgroup ID from task->cgroups->dfl_cgrp->kn->id under bpf_rcu_read_lock; preemption cause tagged as same container, another container or a system service; per-cgroup per-CPU rate limiting; a 131 ms P99 spike when a neighbor saturated all CPUs; "less than 600 nanoseconds to each sched_* hook" measured with bpftop.
  • Delay accounting: an underrated feature of the Linux kernel (Coroot blog): per-TGID delay statistics collected over Netlink and aggregated per container; a noisy neighbor identified by "comparing these metrics"; container_resources_cpu_throttled_seconds_total used alongside CPU delay to recognize a container that reached its own limit; a best-effort neighbor at 100% node CPU did not hurt the latency-sensitive app.
  • bcc-runqslower(8) (bcc-tools man page): measures time "waiting on a run queue ... for a turn on-CPU" for waits over a threshold; -P shows "previous task comm and TID"; filters are -p PID and -t TID; covers both wakeup-to-run and preempted-to-run waits; warns scheduler event frequency can make overhead "significant" and to measure first.
  • Performance degradation for pure-CPU work (Railway Central Station, May 2026): a customer reports containers "3–4× slower" for pure-CPU work at one instant and asks about being "rescheduled onto contended hosts"; staff find a noisy-neighbor issue on a host that "became partially I/O-saturated", "not a quota or migration change on our side"; the customer still reports four times baseline; the thread auto-closes.
  • proc_stat(5) (Linux man-pages): the cpu lines report user, nice, system, idle, iowait, irq, softirq and steal in USER_HZ units; steal is "the time spent in other operating systems when running in a virtualized environment"; ctxt is "the number of context switches that the system underwent".
  • User Interface for Resource Control feature (resctrl) (Linux kernel documentation): Intel RDT and AMD Platform Quality of Service; cache allocation and monitoring, Memory Bandwidth Monitoring and Allocation; monitoring events llc_occupancy, mbm_total_bytes and mbm_local_bytes.

Related resources

See what other people have built on the kernel in Discover, or sign up and start on one host you own.

Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.