Debugging database queries

With yeet, your agent can see every query and command your app sends, from the process that sent it. 

curl -fsSL https://yeet.cx | sh

Linux only  |  Manual install  |  Read the docs

When the slow query log isn’t enough

yeet comes in when the database is busy and nobody can say why. We sit alongside Datadog and Grafana, so you don’t have to turn on ORM debug logging or run MONITOR in production.

Your agent without yeet

p99 on GET /orders went from 80 ms to 2.4 s

  • what is it sending to Postgres?pg_stat_statements shows one SELECT called far more than the rest. Not who calls it, or why.
  • how many queries per request?ORM query logging is off in production.
  • is the cache helping?redis-cli MONITOR would slow production down, so nobody runs it.
  • is the pool exhausted?ssh: Permission denied (publickey).

A guess: “the database is overloaded, scale it up.” The rest waits for someone with access to the host.

Your agent with yeet

p99 on GET /orders went from 80 ms to 2.4 s

  • what is it sending to Postgres?orders-api sends SELECT … FROM order_items WHERE order_id = $1: one prepared statement, executed 36,000 times in two minutes.
  • how many queries per request?About 180 per GET /orders: 36,000 executions against 200 requests in the same window.
  • is the cache helping?No. Every GET for orders:* comes back empty. The service now writes order:*, singular.
  • is the pool exhausted?20 of 20 connections to Postgres open and busy. Requests queue for one.

A verdict: it’s the code, not the database. The deploy renamed the cache key on the write path only, from orders: to order:, so every read misses and falls through to a query that runs once per item. Fix the key, then batch the per-item query.

Query budgets

The probe your agent writes for one slow endpoint doesn’t have to go away. yeet can run it as a service on the host, all day.

Your agent counts queries per service around the clock and serves the numbers to your CI. A deploy that adds an N+1 fails the build.

  • An N+1 alert when a service’s query count jumps
  • A slow query feed with the process that sent each one
  • A Prometheus metric for a cache hit rate your app never exported
  • A Slack alert when anything runs KEYS in production
Query budgets, as a loopOn your host, a yeet service counts every query and command your app sends and serves the counts as budgets over an HTTP route. Your CI test suite pulls them and checks each test’s queries against the budget. Deploys that pass ship. An N+1 fails the build.your appqueries and commandsyeet serviceruns all daybudget routeserved over HTTPCI test suitequeries against budgetdeployan N+1 fails the buildcountspulls budgetsshipsevery deploy is held to the queries production actually runs

How your agent works with yeet

Three layers your agent can read, from the statement down to the socket.

  • Queries

    What the app asked the database, read off the wire.

    Postgres, MySQL, MongoDB

    • Which query is slow, and which one runs 200 times?

      Postgres and MySQL statements by shape, latency and count, tied to the process that sent them.

    • How many queries does each request make?

      Executions per statement against requests to the same service, so an N+1 shows up as a ratio.

    • Which query returned 50,000 rows?

      Row counts per statement, read from the database’s own replies, tied to the process that asked.

    • What changed after the deploy?

      Statement shapes and counts on the new version against the hosts still on the old one.

    • What is the app asking MongoDB?

      Finds, aggregates and updates per client process, with latency.

  • Caches

    What the app asked its cache, and what came back.

    Redis, Memcached, in-process caches

    • What is the app doing to its cache?

      Commands and latency per client, read off the wire, whether it speaks to Redis, Valkey, Memcached or anything else over a socket.

    • Which reads miss the cache?

      Reads that come back empty, by key pattern, so a renamed key shows up as a wall of misses.

    • Which keys are hot or oversized?

      The keys read most and the values that are largest, per key pattern.

    • Who ran a blocking command in production?

      KEYS, FLUSHALL and long scripts, with the client process that sent them and how long they took.

    • Is the in-process cache doing anything?

      Hit and miss counts read from the app’s own cache functions, with no code change and nothing on the wire.

  • Between the app and the database

    The connections and lookups on the way to your data stores.

    TCP, TLS, DNS, the kernel

    • Is the connection pool exhausted?

      Open connections per process and database, against requests waiting for one.

    • Who reset the connection to the database?

      Which side sent the reset, the process that owned the socket, and the database it was talking to.

    • What is still connecting to the old primary?

      What each process resolved for the database host, and where it is still connecting.

    • Why are TLS handshakes to the database failing?

      The handshake error, the server name and the process that made the call.

    • What is the latency to the database from each zone?

      Round-trip times on real connections, grouped by host and zone.

Make your machines readable

See every query your app sends, in under a minute.

curl -fsSL https://yeet.cx | sh

Linux only  |  Manual install  |  Read the docs

Sign up