What AI Agents Get Wrong on Linux

Last updated: September 2026

TL;DR. The best agents now resolve over 90% of Terminal-Bench 2.1 tasks, so "agents are bad at Linux" is no longer true and was never the useful claim. The ICLR 2026 error analysis is more interesting than the headline score: of the execution errors it catalogues, 35.1% are command not found, and a further cluster it labels verification failures covers agents that declared success without checking, or checked the wrong thing. Both are failures of knowing this machine, not failures of knowing Linux, and they are the failures that get worse on your host rather than better, because your host has no test suite to catch them.

I write eBPF tooling and watch people point agents at real boxes, which means the failures I see are skewed toward the ones that happen outside a container. I have not run Terminal-Bench myself and I am reading its published error analysis rather than reproducing it. I am also not going to tell you agents are unreliable, because the numbers do not support that and the people saying it are mostly arguing with a 2024 memory. The pattern that actually matters is narrower and less comfortable: the model gets better at Linux every quarter and no better at your machine, because nothing on your machine is telling it anything.

Are AI agents actually bad at Linux, or is that a 2024 memory?

They are good and getting better fast, and any post built on the opposite claim will be wrong within two quarters. Terminal-Bench 2.0 is 89 hand-built terminal tasks, each with a containerized environment, a human-written reference solution and tests, described in the paper as requiring "extensive domain knowledge, long chains of interdependent actions, and autonomous problem-solving". When the ICLR paper was written, frontier models resolved "less than 65% of tasks, with smaller models scoring around 15%".

That number is already stale, which is the point. The current leaderboard has Claude Fable 5.1 at 91.4% on v2.1. In under a year the frontier went from failing a third of these tasks to failing under a tenth of them.

So the honest framing is not that agents cannot do this work. It is that the residual failures are concentrated somewhere specific, and the paper is unusually useful because it went and catalogued them instead of just publishing a score.

What do AI agents fail at on Linux, according to the Terminal-Bench error analysis?

Executing commands that do not exist, and declaring victory without checking. The paper ran an LLM-as-judge over the failing transcripts, validated at 92.4% agreement with three human annotators across 66 pairs, and sorted the failures into a taxonomy. The execution errors break down like this:

Execution error class	Share
Invocation (command not found, shell syntax)	35.1%
REPL (module not found, script syntax)	19.1%
Runtime (application failure)	15.5%
Filesystem (file not found)	14.1%
Other	16.3%

And the behavioural failure modes, which are the more interesting half:

Failure mode	Prevalence
Disobey specification	24.1%
No or incorrect verification	11.1%
Reasoning-action mismatch	9.6%
Unaware of termination conditions	8.3%
Step repetition	6.4%
Context loss	6.1%
Premature termination	5.8%
Task derailment	4.6%
Weak verification	3.0%

Read the two tables together and a shape appears. command not found and file not found are an agent guessing what exists on a machine. "No or incorrect verification" and "weak verification" are an agent guessing whether what it did worked. Neither is a gap in Linux knowledge. Both are gaps in knowing the state of the specific system in front of it, which is a different quantity entirely: one improves with every model release and the other is not in any training set, because it is a fact about your box that came into existence after the weights were frozen. Note also that these two tables are counting different things. The first sorts errors by what the shell reported; the second sorts transcripts by what the agent did wrong. A single failed run contributes to both, so the percentages describe two views of the same failures rather than two disjoint populations.

Why does my AI agent run a correct top command and still reach a wrong conclusion?

Because top is a correct command that returns output an agent cannot calibrate without knowing the system. This is the failure that survives every model upgrade, and the paper's verification categories are the closest published measurement of it.

The paper defines weak verification precisely: verification is weak "when checks do not cover properties that must hold for true correctness". Its companion category is sharper still. Data fabrication occurs "when the agent creates or alters data that should have been measured, recovered, or derived from specified sources, and treats it as authentic".

Sit with that second definition for a moment in the context of a debugging session on your host. Data that should have been measured, and was not, and got treated as real anyway. That is not a hypothetical benchmark artifact. That is an agent telling you the service is fine because it ran a command that returned zero, on a box where nothing was measuring the thing you actually asked about. The reason this shape is worth naming rather than filing under "models make mistakes" is that it is invisible at the point of failure. A wrong command errors out and the transcript shows it. A weak verification returns a clean result and reads exactly like a strong one, so the run that fabricated a number and the run that measured it produce the same confident paragraph, and nothing in the output distinguishes them.

Why is my Linux host harder for an AI agent than a benchmark container?

Because every Terminal-Bench task ships with tests and your machine does not. Each task in the dataset has, in the paper's own description, "(1) a containerized environment initialized with relevant packages and files, (2) an instruction that describes the task, (3) a set of tests to verify completion, and (4) a reference solution manually written to solve this task".

Three of those four do not exist when an agent works on your host. There is no reference solution, so nothing defines what "done" looks like other than the agent's own reading of your request. There is no test that returns pass or fail, so the check that would have caught a premature completion is absent exactly when it is needed. And the environment was not initialized with the relevant packages; it accumulated over four years, carries a broken symlink somebody made in 2023, and has three Python versions on $PATH for reasons nobody documented.

This matters more than it sounds, because the benchmark's verification failures happen even with tests available. An agent that verifies weakly when a test suite is sitting right there will verify weakly on a box that offers it nothing. The 91.4% is measured under conditions strictly friendlier than production, and the residual failures are exactly the ones those friendly conditions were catching.

What does an AI agent do when it cannot measure something, and what is data fabrication?

It reaches for a snapshot tool, runs it twice, and reasons about a moving system from two still frames. This is the concrete shape of the verification gap on a real host, and it is worth walking because it looks like competence.

Ask an agent why a process is slow. It runs top, which reports a CPU percentage over a sampling interval. It runs it again a few seconds later and gets a different number. It now has two data points and no motion between them, and the competent next move, attaching to the process and watching what it does, is the one move it cannot make because nothing on the host offers it. So it reasons from the two frames it has, and produces an answer with the confident register of an answer derived from measurement.

What makes this hard to catch in review is that the reasoning is sound given the inputs. Two samples showing high CPU do support "this process is busy" as a hypothesis. The defect is not in the inference; it is that a different measurement, available on the same machine for nothing, would have discriminated between two explanations that the sampled one cannot separate. The check that does it is cheap and it is not a CPU percentage:

cat /proc/$(pgrep -n node)/schedstat

The second field is nanoseconds spent waiting on a run queue. A process with a high top percentage and a run-queue wait near zero was not competing for CPU, and a faster machine would not have helped. That distinction is a whole post on its own, and it belongs here because an agent will not make it unless something puts the number in front of it.

Does command not found mean my AI agent does not know Linux?

No, and this is the most misread number in the analysis. At 35.1% of execution errors, invocation failures are the single largest execution category, and the intuitive reading is that the model does not know which command to use. The likelier reading is that it knows exactly which command it wants and that command is not installed on this box.

ss versus netstat, ip versus ifconfig, bpftrace on a host that has never had it, jq on a minimal container image. A model trained on the world's Linux writing knows the canonical tool for a job. What it cannot know is which subset of those tools exists on the machine it just landed on, and the shell tells it only by failing.

The fix on your own host is unglamorous and it works: tell the agent what is there. A CLAUDE.md naming the distro, the kernel, the package manager and the handful of tools that are and are not installed removes a whole class of failure at the cost of six lines. This is the cheapest reliability win available and almost nobody does it.

Should I trust an AI agent to debug my production Linux server?

Trust it to execute and to hypothesize, and do not trust its verification, which is precisely the split the error analysis measures. An agent is genuinely good at knowing which command answers a question, sequencing a long chain of them, and reading output that is put in front of it. It is measurably weaker at deciding whether the thing it just did worked, and that weakness is concentrated where you can least afford it.

Practically, on a box that matters:

• Give it the observation, do not make it infer the observation. The failure modes cluster on verification, so the highest-value thing you can add is a measurement it did not have to construct.
• Make the environment legible. Distro, kernel version, what is installed. This is aimed directly at the 35.1%.
• Keep an independent record of what it ran. The agent's own transcript stops at the shell boundary, so the record has to come from below it.
• Treat a confident "fixed" as a hypothesis. Premature termination is 5.8% of failures in an environment with tests. On your host it has nothing to trip over.

How do I give an AI agent a measurement instead of a guess on Linux?

Put the number where the agent will read it, which on Linux means one of three places depending on how hard the question is. The failure modes above are not a reason to stop using agents; they are a specification for what to hand one.

/proc files, for anything a counter already answers. This costs nothing and covers more than people expect. /proc/PID/schedstat separates on-CPU nanoseconds from run-queue wait, which is the top ambiguity resolved. /proc/PID/status carries voluntary against nonvoluntary context switches. /proc/PID/io has read syscalls against block-device bytes. Name these in a CLAUDE.md and an agent reads them correctly without further help, because the hard part was never the parsing.

auditd or strace, when you need a record of what ran. auditd ships with the distro, survives reboots, and compliance teams already read it, which makes it the right answer when the record has to be durable and auditable. strace -f is the fastest way to watch one process tree interactively. Both have known edges: auditd gets noisy at agent volumes and has no per-agent scoping, and strace misses a server the agent launched through an intermediary.

A kernel probe, when the question needs a stream rather than a sample. Some of what an agent gets wrong is invisible to any polling tool, because the event starts and finishes between two samples: which process opened which file, which connection belongs to which pid, what a build actually executed across ten thousand forks. That is where yeet fits. It is a JavaScript runtime for writing eBPF programs, so a probe attaches to the event itself and the aggregation runs in JS you can edit while it is running:

curl -fsSL https://yeet.cx | sh
yeet login
yeet run gh:yeet-src/exectop -- --pid $(pgrep -n claude)

exectop folds every command the agent's process tree executed into one row per kind. Run it during a session and the command not found failures stop being a statistic: you see which binary the agent reached for and whether it existed. That is the 35.1% made legible on your own box, and it is the input to the CLAUDE.md fix above rather than a replacement for it.

A kernel probe tells you what happened; it does not tell the agent anything unless you put the output in front of the model, and today that is a copy-paste or a file the agent reads. It needs a BTF-capable kernel and a privileged daemon install, so it is a fleet decision rather than a laptop one. And none of the three tiers here touches the behavioural failures: an agent that skips verification skips it whether or not good data was available. What instrumentation fixes is the case where the agent would have checked and had nothing to check against.

The bottom line: the gap is measurement, and it is the part that does not improve on its own

Read the benchmark honestly and the conclusion is not that agents are unreliable. Frontier agents resolve over 90% of Terminal-Bench 2.1, the trajectory is steep, and betting against it has been a losing position for two years running. What the error analysis shows is where the residual failures live: 35.1% of execution errors are the agent guessing what exists on the machine, and the verification categories are the agent guessing whether its own work landed. Both are questions about your specific host, and a better model does not answer either one, because the information was never in the training data and is not in the room.

Two of those are fixable this afternoon, and the routing is by how hard the question is rather than by vendor. Write down what is installed, which is aimed at the invocation failures and costs six lines. Point the agent at /proc/PID/schedstat, /proc/PID/status and /proc/PID/io instead of top, which is aimed at the verification failures and costs nothing. Use auditd when the record has to survive a reboot and compliance already reads it. Reach for exectop on yeet when the question needs a stream rather than a sample and you can run privileged on the box, and know it tells you what happened rather than telling the agent anything. The third problem, knowing what the agent actually did rather than what it reported, needs a record the agent does not author, and that is the kernel's job.

The mistake this sets up is reading a 91.4% and concluding the remaining 8.6% is more of the same, only harder. It is not. It is a different kind of failure, it is concentrated in the part of the loop where nobody is checking, and on your machine it has none of the guardrails the benchmark provided.

Frequently asked questions

What is Terminal-Bench?

A benchmark of 89 hand-built terminal tasks published at ICLR 2026, each with a containerized environment, a human-written reference solution and tests for verification. Tasks range from configuring legacy systems to reimplementing research papers. It is designed to measure agents on realistic command-line work rather than on isolated question answering.

How good are AI agents at Linux command line tasks in 2026?

Better than most people assume. The Terminal-Bench 2.1 leaderboard has frontier models above 90% resolution, up from under 65% when the paper was written. Smaller models sit far lower, around 15% in the paper's measurements, so the gap between frontier and local models is much wider on agentic tasks than on single-turn ones.

What is the most common failure mode for AI coding agents?

By the ICLR error analysis, disobeying the specification at 24.1% of behavioural failures, and command not found at 35.1% of execution errors. The verification categories together, covering agents that did not check their work or checked the wrong property, account for another 14.1%.

Why does my AI agent say it fixed something when it did not?

The error analysis has categories for exactly this. Premature termination is declaring completion before meeting the task objectives, and weak verification is running checks that do not cover the properties that must hold for correctness. Both occur even in an environment with a test suite available, so they occur more readily on a host with no tests at all.

What should a human check when an AI agent says it fixed a Linux problem?

Check the thing the fix was supposed to change, measured independently of the agent. The error analysis names premature termination, declaring completion before meeting the objectives, and weak verification, running checks that miss the properties that must hold. Both produce a confident summary, so the summary is not the artifact to read. Re-run the failing case yourself, or read a counter the agent did not produce.

Do AI agents hallucinate Linux commands?

Sometimes, but the benchmark data suggests a more common cause for invocation failures: the command is real and correct and simply is not installed on that machine. A model knows the canonical tool for a job from its training data and cannot know which subset exists on the host it just landed on.

How do I make an AI agent more reliable on my own server?

Tell it what is on the machine, which addresses the largest execution error category, and give it real measurements rather than making it infer state from repeated snapshots, which addresses the verification categories. A short file naming the distro, kernel, package manager and installed tooling is the cheapest reliability improvement available.

Is a local model as good as a frontier model for agentic work?

Not currently, and the gap is much larger for agentic tasks than for single-turn ones. The paper reports smaller models scoring around 15% against frontier models above 90% on the current leaderboard. Long chains of interdependent actions are where small models fall off, and agentic work is made of those chains.

What is data fabrication in the context of an AI agent?

A failure mode the ICLR analysis defines as the agent creating or altering data that should have been measured, recovered or derived from a specified source, and then treating that result as authentic. It is counted only when the fabricated artifact is actually used to satisfy a requirement or pass a check.

Does giving an agent more context fix these failures?

It fixes the environment-knowledge failures and not the verification ones. Telling an agent what is installed directly addresses invocation errors. Whether it checks its own work is a behavioural property of the harness and model, and more context does not make an agent that skipped verification start verifying.

Sources

• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. ICLR 2026, Merrill, Shaw, Carlini et al. The source for every number in this post: 89 tasks, the "less than 65%" and "around 15%" figures, the four-part task structure, the LLM-as-judge methodology at 92.4% human agreement over 66 pairs, and both failure taxonomies. Also the source for the weak-verification and data-fabrication definitions quoted above.
• Terminal-Bench v2.1 leaderboard. Current standings, showing frontier resolution above 90% and therefore that the paper's headline figure is already historical. Read this before citing any score from the paper as current.
• Terminal-Bench project site. The benchmark's own home, with the task set and the evaluation harness.
• terminal-bench on GitHub. The dataset and harness as published, for anyone wanting to reproduce the numbers rather than take them from a blog post.
• Claude Code sandboxing documentation. Referenced for what an agent running on your own host can reach by default, which is the condition under which these failure modes play out in production rather than in a container.
• yeet documentation. The runtime behind the kernel-side records referenced here, for putting a measurement in front of an agent that it did not have to infer.

Related resources

• How to Run AI Agents Locally on Linux. The setup these failure modes play out on
• How to Find Which Process Is Slowing Down Your Linux Machine. The schedstat check, in full
• How to Audit What an AI Coding Agent Actually Ran on Linux. Keeping a record the agent did not write
• AI Agents on Linux: Two Threat Models. Why the record has to come from below

Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.