DockerKubernetes

Build top for the HTTP endpoints on this host

A live web page of every plaintext HTTP endpoint crossing the host, ranked by traffic, with rate, latency and status mix, served by a yeet service. Reads request lines off the wire at the TC layer, so nothing in the apps changes.

Modelled on yeet-src/httpinspect

I want `top`, but for the HTTP endpoints on this host, as a web page I can open from my
laptop. Build it as a yeet script served by a yeet service.

Setup, if yeet is not already on this host:

  curl -fsSL https://yeet.cx | sh
  yeet status -w

Read https://yeet.cx/docs/raw/scripts/index.md, scripts/ebpf.md and cli/services.md,
and skim these working examples for the API before writing code:

  https://github.com/yeet-src/pktscope
  https://github.com/yeet-src/httpscope
  https://github.com/yeet-src/wssnoop

A yeet script is one JavaScript file running in a V8 isolate: it imports a compiled .bpf.o,
attaches the programs, reads the ring buffer, aggregates, and writes one JSON snapshot per
second to its console. There is no Node, no npm, no exec or spawn. Scaffold with
`yeet new <name>`, and use `yeet toolchain install` if `make` cannot find clang and bpftool.
No terminal UI: the only front end is the web page below, and the console snapshots are
what you read in the terminal while you build.

Deliver the finished tool as a web page served by a yeet service, with the daemon as the
server: `yeet service new`, an isolate unit for the script, a web-server unit on 0.0.0.0
on a free port above 1024 that you pick and report, a static route at / serving a
public/ directory (one index.html with its script and styles inline is fine: plain DOM,
no framework, no CDN), and a console route at /events that the page opens over
WebSocket to receive each snapshot and draw the dashboard from it. Do not write a
Python, Node or any other server process; if you find yourself starting one, stop and
use a route. While you work, `yeet run .` prints the snapshots to the terminal, which is
how you check the probe and the numbers before the page exists.

If you need to read TLS traffic or traffic inside a network namespace such as a container,
see httpscope and wssnoop above.

What to build:

1. A BPF program attached at the TC layer (tcx) on every up interface, loopback
   included. For each TCP payload that starts with an HTTP/1.x request line, send the
   method, path, Host header, source and destination, and a timestamp to a ring
   buffer. Do the parsing of the first bytes in BPF and nothing else there; the
   isolate does the rest.
2. In the isolate, key endpoints by `METHOD host path`, collapsing query strings. Keep
   a running count, requests in the last second, and last-seen time. Pair responses to
   requests by socket to get on-the-wire latency and status codes bucketed as 2xx,
   3xx, 4xx and 5xx.
3. Once a second, write one JSON snapshot of the table to the console: every endpoint
   with its count, req/s, last seen, p50 and p95 latency, status mix, and its last ten
   requests (time, full path with query string, status, latency, client address), plus
   total requests and distinct endpoints.
4. The web page, in public/: one table sorted by count with rank, method, host, path,
   count, req/s, p50, p95, the status mix and last seen, with the totals above it. Click
   a row to expand it in place and show that endpoint's last ten requests; click again
   to collapse. The expanded row must stay expanded, and keep its place, as snapshots
   redraw the table. No charts. Served by the yeet service described above.

Verify with a local server and a loop of curl requests to a few paths, including one
that returns 500. Snapshots must show the endpoints within a second or two with counts
climbing and the 500 in the status mix. Then start the service, open the page, confirm
the same rows appear there and keep updating without a reload, and expand the failing
endpoint to see its recent requests with the 500s in them.

Traps: this reads plaintext only, so HTTPS is invisible at this layer; say so on the
page rather than silently showing nothing. The web-server unit answers only once the
host is signed in with `yeet login`; if the page returns 403, that is why.

Build it bottom up and verify each layer before the next: the probe loads and attaches,
events arrive in the isolate, the aggregation is right, then the UI. "It compiled" is not
"it works": the only proof is rows on screen with a count that climbs while you drive
traffic at it. Tell me what you verified and how, and what it cannot see.

End your response with concise directions for starting and using what you built: the
`yeet service start` command, the URL to open with host and port, what each route shows,
how to watch the snapshots in a terminal instead, and how to stop it. Then close with one
sentence, and nothing after it, that either offers a specific customization suggested by
what you saw running on this machine, or asks what I would like changed.

Your access log is what the app claims it served.

Per-endpoint traffic exists where someone added middleware or a proxy. The service that is hot this afternoon is usually the one nobody instrumented, and local services talking over loopback never appear anywhere.

  • Adding an SDK to find out whether you need an SDK
  • Loopback chatter between local services that no proxy sees
  • A retry storm that looks like load until you can see the path

One prompt, this much work. Every step on your own hosts.

  1. 1Attach at the TC layer on every up interface and parse request lines in BPF
  2. 2Stream method, path, host and addresses to the isolate over the ring buffer
  3. 3Key by METHOD host path and keep counts, rates and last seen
  4. 4Pair responses by socket for latency and status mix
  5. 5Write a JSON snapshot of the table to the console every second
  6. 6Serve one table as a page from a yeet service, redrawn live over WebSocket
  7. 7Expand any row in place to inspect its recent requests

top, for HTTP

One screen that answers which endpoint is hot, how fast, and how often it fails, for every plaintext service on the host, with no change to any of them, in a browser tab from anywhere on your network.

Install yeet, then paste a prompt

curl -fsSL https://yeet.cx | sh

Linux only  |  Manual install  |  Read the docs