Why Is My Build Slow?

Necco Ceresani
Necco Ceresani··21 min read

Fifteen years building engineering platforms, currently focused on advanced AI infrastructure at yeet. I love turning the deeply complex topics into something everyone can understand. I relate deeply with the core yeet philosophy that you can just build things.

Quick answer. A slow build is usually not one slow step, it is a cheap command running far more often than anyone intended. Count process launches grouped by kind of command and the answer is normally one row: ./bin/exectop -- make -j8 folds every exec into one row per command and sorts by count, so a utility running 2,420 times is immediately visible where a build log hides it in 40,000 lines. Read the fork-to-exec median in the same table to separate a slow program from a slow process launch.

The builds I have spent the longest failing to speed up were the ones where every individual step was fast. Nothing in the log stood out, no single command looked expensive, and the total was somehow four minutes. I build eBPF tooling rather than maintain build systems for a living, so this is not advice about your particular toolchain. What I keep seeing, in my own repos and in the ones people bring to Discord, is that the interesting number is almost never a duration. It is a count, and build logs are structured to hide counts.

Why is my build slow when no single step looks slow?

Because cost is duration multiplied by count, and a log shows you only the duration. Three milliseconds is invisible on any single log line, and 4,000 invocations of that same command is 12 seconds of wall clock. Nothing was slow. The build was still 12 seconds longer than it needed to be, and no profiler aimed at a single process would have found it, because no single process misbehaved.

This is the arithmetic that makes process launches worth counting. One fork and exec for a small program is roughly a hundred microseconds of kernel work, which is cheap on its own, and then the new program pays dynamic linking and its own startup on top. At that scale, ten thousand launches is a second of pure process creation before any of those programs do a unit of useful work. Builds do not lose minutes to one bad step nearly as often as they lose them to a cheap step repeated past the point of sense. So the fix is to stop reading the log and start counting the launches, grouped so that repetition becomes a number rather than scrollback. A log is ordered by time, which is the wrong axis for this question: it interleaves 4,000 invocations of one command with everything else that happened, and reading it back tells you the order things occurred in rather than how many times each one did.

How do I count how many times make ran gcc, echo and sh?

Run the build under a process-launch tracer that groups identical commands, and read the count column. Launching it under the probe rather than attaching partway through is what makes the counts complete, since nothing escapes in the first second.

curl -fsSL https://yeet.cx | sh        # install yeet, once
./bin/exectop -- make -j8              # run the build, count everything it starts

The result is one row per kind of command, sorted by how many times that kind ran. The top of a real capture from a small C build looks like this:

  command                              count   share          fork→exec  parent
▸ echo ⟨1 arg⟩                          ×124  ██······  29.2%     149µs  bash
▸ date ⟨1 arg⟩                           ×90  █·······  21.2%     137µs  bash
▸ sleep ⟨1 arg⟩                          ×46  █·······  10.8%     145µs  bash
▸ as -EL -mabi=lp64 -o ⟨2 args⟩          ×27  ········   6.4%     128µs  gcc
▸ gcc -Wall -O2 -c -o ⟨2 args⟩           ×26  ········   6.1%     243µs  bash
▸ cc1 -quiet -imultiarch -quiet …        ×26  ········   6.1%     103µs  gcc,?

Read that top down and the shape of the build is obvious in a way it never is in a log: half of everything this build did was echo and date. The compiler invocations, which are the point of the build, are 6% of the launches each. A build whose top rows are shell utilities rather than compilers is telling you where its time went.

The parent column is what turns a count into an action. Knowing echo ran 124 times is interesting; knowing all 124 came from bash tells you a recipe is doing it, and knowing which recipe follows from expanding the row.

How do I group thousands of similar commands without losing the differences?

Group on the command plus its flag names, and drop the positional paths. Two cc1 invocations compiling different files are the same kind of work and belong in one row, while the same compiler with a different optimization level is a real difference and must not merge. Getting that key wrong makes the whole view useless. The same compiler with a different optimization level is not, and merging them would hide a real difference.

So the key is the command plus its flag names, with positional paths dropped and repeated flags deduplicated. cc1 -quiet a.c and cc1 -quiet b.c fold together; cc1 -O2 stays separate. Two carve-outs came out of real data rather than from design, and both matter for builds specifically:

  • Subcommands count as part of the identity. git add and git gc do very different work and must not merge into a git row that tells you nothing. The subcommand is part of what the command is.
  • sh -c folds on the first command inside the script, not the script text. A generated Makefile emits a different script per target, so folding on the text shatters one recipe into dozens of single-use rows and the count you needed disappears. This is the case that matters most for autotools and CMake builds, where nearly everything arrives as sh -c.

Without that second carve-out, a generated build looks like a few hundred unique commands that each ran once, which is precisely the view you were trying to escape. Autotools and CMake are the common cases: both emit per-target shell fragments, so the text of the script differs on every invocation while the command inside it is the same one repeated, and folding on the outer text turns one recipe into as many rows as there are targets.

Is my build slow to start processes, or slow to run them?

Read the fork→exec column to tell them apart: it is the median time between a process being created and its program beginning to run, so a large value means the delay is in the launch rather than in the program. That gap is kernel work, setting up the new address space and, on a loaded machine, waiting for a CPU to be free. It is not time the program spends doing anything, because the program has not started yet.

Ordinary values sit in the low hundreds of microseconds on an unloaded machine. What makes the column useful is not the absolute number but watching it move:

What you seeWhat it points at
Medians in the low hundreds of microseconds, stableNormal process creation; look at counts instead
Medians climbing as the build progressesContention, memory pressure, or a -j value the machine cannot sustain
One command with a much higher median than its siblingsA large binary, or a program whose dynamic linking is expensive
Low medians, high total time, low launch countThe time is inside a program; use a CPU profiler, not this

That last row is the honest boundary of the whole approach and it belongs here rather than in a footnote. A build spending four minutes in one enormous link step has a launch count in the dozens and nothing in this view will help. For time inside a single process, hotspot is the sibling tool, and it deliberately excludes forked children, which is the exact inverse of what this post is about.

Why does make compile the same source file twice?

Usually because the target sits in two dependency chains, and a doubled exec count is how you spot it. A project with 40 source files showing 80 compiler invocations is compiling everything twice, and a count that is a clean multiple of what you expect is the tell. In a log, a second compilation pass looks like more of the same output scrolling past, which is why doubling survives for months in build systems nobody has counted.

Three causes account for most of it, in rough order of how often I have seen them:

  1. A target appearing in two dependency chains, so it is built once for each, and neither path knows the other did it. Common when a shared library is a dependency of both a binary and a test binary.
  2. A recursive make invoked from more than one place, where each invocation re-derives the same subtree. The count multiplies by the number of entry points.
  3. A configure or codegen step with no output-based dependency, so make cannot tell it already ran and runs it on every invocation.

Cross-check with make's own tooling, which answers a different question well. make -n prints the recipes without running them, so it shows what make intended before any shell expansion. make -d explains why each target was considered out of date, which is the direct answer to "why did this rebuild". Use those to explain a doubling that launch counts revealed; use launch counts to notice it in the first place, since neither -n nor -d counts anything for you.

Why does the same build take twice as long in CI?

Usually because a different program ran, not because the machine was slower. That distinction is worth establishing before anyone buys faster runners, and comparing launches settles it faster than comparing timings does. Two wall-clock numbers tell you the pipeline is slower and nothing about why; two fold lists tell you which commands the pipeline ran that your laptop did not. The pipeline side of this, including running a tracer inside a CI job with no terminal, is covered properly in capturing every command a build runs.

Run the build in both places and diff the fold lists. A row present in one run and absent in the other is the signal, and it is a much sharper one than a timing difference because it names the command rather than the symptom. Four causes account for nearly all of it:

  • A different compiler or interpreter version, visible as a different flag set on the compiler row even when the binary name matches.
  • A tool that was not installed, so the build took a fallback path. This is the one that never appears in a log, because a well-written build falls back quietly.
  • A cold cache, visible as compiler invocations in CI where the local run had almost none.
  • An optional dependency resolving differently, producing extra configure or download commands that the local run skipped entirely.

For the CI side you need output without a terminal, since the interactive view requires a real TTY and a CI job has none. The data layer runs standalone and prints a plain-text report:

yeet run src/probes/capture.js -- <root-pid> 30

That seeds the same traced set, aggregates for the given number of seconds, then prints the totals, the bucket breakdown and the top folds as text before exiting. Save that from both environments and the diff is the answer.

Is my build spending its time compiling or shelling out?

Compare the compiler rows against the shell rows: gcc at the top is healthy, echo, date and sh at the top means the build is spending its launches on plumbing rather than on the work it exists to do. That is the whole heuristic, and it is more useful than it sounds because the inverse is so common. Grouping every exec into behavior buckets makes the ratio immediate, since a bucket total answers the question a per-row count cannot, which is what kind of work this build mostly consists of:

── doing ────────────────────────────────────────────────────────────────────
  text plumbing    ███████████████████████·······    323  76.0%
  compiling        ██████························     84  19.8%
  moving files     ······························      8   1.9%
  other            ······························      7   1.6%
  network          ······························      2   0.5%
  shelling out     ······························      1   0.2%

Three quarters of that build's process launches were text plumbing and a fifth was compilation. For a build whose job is to compile nine files, that ratio is the finding. It does not tell you which recipe to fix, and it does tell you that the answer is in the shell layer rather than in the compiler flags, which is usually where people start.

A caveat on reading proportions: these are shares of launches, not shares of time. A build could launch 500 cheap shell commands and one compiler invocation that runs for three minutes, and the buckets would show 99% text plumbing while the compiler owned the wall clock. Use the buckets to find where the launches are and confirm against total build time before acting.

Will make -j16 fix a build that launches 10,000 processes?

No. A build launching 10,000 unnecessary processes at -j1 launches the same 10,000 at -j16, and simply pays for them concurrently. Parallelism changes how many processes run at once and never how many run, which is why it is the first thing people reach for and cannot touch this failure mode.

Where raising -j helps is a build that is compiler-bound with idle cores, which the launch profile above identifies directly: compilers at the top, counts matching your file count, cores not saturated. Where it hurts is subtler and shows up in the fork→exec column. More concurrent processes means more memory in use and more I/O in flight, and past the width your machine can sustain, process creation itself starts to queue. A fork-to-exec median that climbs when you raise -j is the machine telling you the value is too high, and it is a better signal than total build time because it moves earlier.

When should I use a CPU profiler or strace instead?

When the time is inside a program rather than in the number of programs. These tools answer different questions and picking the wrong one costs an afternoon, because a launch tracer aimed at a build with 30 launches reports 30 rows and no insight, while a CPU profiler aimed at a build with 40,000 short-lived processes profiles each one separately and finds nothing significant in any of them:

ToolAnswersUse it when
exectopWhat did the build launch, and how many timesThe launch count is high or you do not know what is running
hotspotWhere is one process spending CPUOne long step dominates and you know which
bcc execsnoopA greppable stream of every execYou want to pipe, script or log launches, or watch a whole host
strace -fEvery syscall for a process and its childrenYou need file, network or syscall detail for a small number of processes
make -dWhy did make decide to rebuild thisA target rebuilds when you believe it should not

strace -f deserves a specific warning for build work. It attaches to each child through ptrace and stops the process on every syscall it reports, so across the hundreds of short-lived processes a build creates, the observation cost dominates and the build you measure is meaningfully slower than the build you run. It is the right tool for a handful of processes and the wrong one for a tree of them. If what you want is a flat greppable stream of launches rather than a folded table, bcc's execsnoop is built for exactly that, and the comparison between the two shapes covers when each wins. For the same five tools ranked by what each can scope to one process tree and what each does with the resulting volume, capturing every command a build runs does that comparison directly.

The bottom line: count the launches before you change anything

Start by counting what the build launched, grouped by kind of command, because the answer is a row with a surprising count more often than it is a slow step. Use exectop when you do not yet know what is running and want repetition folded; use make -n and make -d to explain a doubling once you have found it; use hotspot when the launch count is low and one program owns the clock, since a launch tracer has nothing to say about time spent inside a process. Reach for ccache or sccache only after you have confirmed compilation is actually where the time goes, because a compiler cache does nothing for a build losing its minutes to shell plumbing. Raise -j when compilers are at the top of the profile and cores are idle, and treat a climbing fork-to-exec median as the signal that you have raised it too far. And if the whole build is one enormous link step, none of this applies and a CPU profiler is your tool.

Frequently asked questions

Why is my build slow when every individual step is fast?

Because the cost is in the count, not the duration. A command that takes 3 milliseconds and runs 4,000 times costs 12 seconds, and no profiler pointed at a single process will show it, because no single process was slow. Counting process launches by kind of command is what makes this visible: the row with a count in the thousands is the answer, and it is usually a cheap utility invoked per file by a recipe that could have invoked it once.

How do I find out what a Makefile is actually running?

Run make under a process-launch tracer, or use make's own dry-run and debug output. make -n prints the recipes without executing them and make -d explains why each target was considered out of date. The difference is that a tracer shows what actually executed, including commands a recipe generated at runtime, whereas make -n shows what make intended to run before any shell expansion happened.

What does a high fork-to-exec time mean?

It means the delay is in creating the process rather than in the program itself. The gap between a process being created by fork and the new program starting at exec is dominated by kernel work: setting up the address space, and on a loaded machine, waiting for CPU. A median in the hundreds of microseconds is normal. A median that climbs while the machine is under load points at contention or memory pressure rather than at any program you could optimize.

Does make -j actually make my build faster?

Up to the point where something else becomes the constraint, usually disk or memory rather than CPU. Raising parallelism multiplies the number of processes alive at once, which multiplies memory use and I/O contention, and past a certain width the total time stops improving or gets worse. The count of processes launched does not change with -j, only how many run concurrently, so parallelism cannot fix a build that is running a command ten thousand unnecessary times.

Why does my build behave differently in CI than on my laptop?

Usually because a different program ran. A different compiler version, a tool that was not installed so a fallback path was taken, a cache that was cold, or an optional dependency that resolved differently. Comparing what each environment launched, rather than comparing timings, finds this quickly: the difference shows up as a command that appears in one run and not the other, which no amount of reading the build log makes obvious.

How do I tell if my build is CPU-bound or launching too many processes?

Compare the count of process launches against the wall-clock time. A build spending its time in a handful of long-running compiler invocations is CPU-bound, and the fix is compiler flags, caching or parallelism. A build launching thousands of short-lived processes is spending real time on process creation itself, and the fix is to launch fewer of them. The two look identical in a wall-clock number and completely different in a launch count.

Is process creation on Linux actually expensive?

One fork and exec is cheap, on the order of a hundred microseconds or so of kernel work for a small program. The problem is arithmetic rather than any single call being slow. At a hundred microseconds, ten thousand launches is a second of pure process creation before any of those programs do work, and each also pays dynamic linking and startup on the way in. Cheap multiplied by a large count is how builds lose minutes invisibly.

What is the difference between fork and exec on Linux?

fork creates a new process as a copy of the caller, and exec replaces the program running inside a process with a different one. Running a command from a shell is normally both: fork makes the process, then exec loads the program into it. Measuring the gap between them isolates the cost of process creation from the runtime of the program, which is why the two events are traced separately.

Will ccache or sccache fix a slow build?

They fix repeated compilation of unchanged sources, which is a large share of incremental build time and a real win. They do nothing for a build whose time goes into shell plumbing, redundant configure steps, or a utility invoked per file, because none of that is compilation. Checking what the build actually launches before adding a compiler cache tells you whether compilation is where your time is going at all.

How do I see which step of my build is running twice?

Look for a fold count that is an exact multiple of what you expect. A build with 40 source files showing 80 compiler invocations is compiling everything twice, and the usual causes are a target that appears in two dependency chains or a recursive make invoked from more than one place. The doubling is obvious in a count and nearly invisible in a log, where the second pass looks like more of the same output.

Can I profile a build without changing the build system?

Yes, by observing from outside it. A process-launch tracer attaches to kernel tracepoints rather than to your build, so nothing in the Makefile, the package scripts or the CI configuration changes, and no wrapper script is inserted around the compiler. This matters most for build systems that generate their recipes at runtime, where adding instrumentation means modifying a generator rather than a file you can read.

What can process-launch tracing not tell me about a build?

Anything happening inside a process. Time spent in a compiler optimizing a large function, a linker doing symbol resolution, a bundler walking a dependency graph in memory, all of it is one process launch and reveals nothing about the work inside. Process tracing finds builds losing time to too many launches, and it will not find a build losing time to one slow program, which needs a CPU profiler.

Sources

  • GNU make options summary (GNU make manual, 2026) — documents -n as printing the recipes that would be executed without running them, -d as printing all debugging information including why each target was considered out of date, and -j as specifying the number of recipes to run simultaneously, with the note that recipes run in parallel unless a target is marked as needing serial execution.
  • exectop: what you're looking at (yeet-src, 2026) — documents the folded table's columns including fork→exec as the median time between the process being created and the program starting, the parent column listing which programs started this kind of command, and the behavior buckets that group every exec into categories such as text plumbing, compiling and network; includes the sample of 425 execs in 42 seconds with echo at ×124 and date at ×90.
  • exectop: how it works (yeet-src, 2026) — records the fold key as the command plus its flag names with positional paths dropped and repeated flags deduplicated, and the two carve-outs that came from real data: subcommands counting as part of the identity so git add and git gc stay separate, and sh -c folding on the first command inside the script because a generated Makefile emits a different script per target.
  • exectop: reading it without a TTY (yeet-src, 2026) — documents the headless path yeet run src/probes/capture.js -- <root-pid> <seconds>, which seeds the same traced set, aggregates for the given number of seconds and prints totals, bucket breakdown, findings and top folds as plain text before exiting, and notes there is no JSON mode.
  • fork(2) man page (man7.org, 2026) — specifies that fork creates a new process by duplicating the calling process, with the child receiving a copy of the parent's address space using copy-on-write, which is the kernel work that the fork-to-exec interval measures and the reason that interval grows under memory pressure rather than staying constant.
  • execve(2) man page (man7.org, 2026) — specifies that execve replaces the current process image with a new program, so the calling process's text, data and stack are discarded; this is the point at which the new program begins running and therefore the end of the interval that separates process-creation cost from program runtime.
  • sched tracepoints in the kernel tree (kernel.org documentation, 2026) — the tracepoint infrastructure behind sched_process_fork and sched_process_exec, the two hooks needed to time the interval between them; tracepoints are a stable instrumentation interface rather than a syscall ABI, so one BPF object works across architectures without per-arch entry points.
  • strace man page (man7.org, 2026) — documents -f as tracing child processes as they are created by currently traced processes, and the ptrace attach mechanism that stops the tracee on each traced event; that per-syscall stop is the observation cost that makes strace -f impractical across the hundreds of short-lived processes a build creates.
  • ccache documentation (ccache, 2026) — describes ccache as a compiler cache that speeds up recompilation by caching previous compilations and detecting when the same compilation is being repeated, which bounds what it can help with: repeated compilation of unchanged sources, and not shell plumbing, configure steps or per-file utility invocations.

Related resources