Detect API Drift After a Deploy

Last updated: October 2026

Quick answer. API drift is a change in the shape of what an endpoint sends or returns, rather than an error, which is why nothing alerts on it. A provider stops sending a field, an id changes from a number to a string, a new nullable key appears, and every participant reports success while the strictest consumer breaks. You cannot catch it from status codes because the statuses do not change, and you cannot catch it from a schema file because the schema describes what was agreed rather than what is being sent. What does catch it is learning each endpoint's shape from live traffic and reporting the differences: a field missing, a type changed, a new status, a new endpoint, latency stepping up. On Linux that comes from reading the actual request and response bodies off the kernel with eBPF, so the comparison is against what really crossed the wire.

The failure has a signature. Something was deployed, somewhere, possibly not by you. Error rates look normal on both sides. One client starts failing, usually the one with the strictest parsing, and its error says something unhelpful about an unexpected null or a type mismatch. Everyone checks their dashboards, everyone is green, and the investigation becomes archaeology on two sets of logs that each record a success.

The reason this class of bug survives modern tooling is that every layer is working correctly. The provider returned 200 with valid JSON. Your service received 200 with valid JSON. The disagreement is about a key that used to be there, and no status code can express that.

What is API drift, and why doesn't it show up as an error?

API drift is a change in an endpoint's observed shape: the keys a body contains, the types of their values, the statuses it answers with, the parameters it accepts. It is distinct from an outage because nothing fails at the protocol level. The request is well formed, the response is valid, and the status is 200.

It does not show up as an error because errors are a contract between the two endpoints of one exchange, and drift is a contract between the exchange and the code that will later read it. The provider's handler succeeded by its own definition. Your service's HTTP client succeeded by its own definition. The failure happens afterwards, in a parser, a validator or a line of business logic that assumed a key would be present, and by then the HTTP transaction is over and was recorded as a success on both sides.

The common shapes are worth naming, because recognising them is most of the diagnosis:

• A field stops being sent. Often a cleanup, often undocumented, and it only breaks consumers that read it without a default.
• A type changes. The classic is an id moving from number to string when the provider outgrows the 53-bit integers JSON numbers safely carry. Strict parsers reject it; lenient ones silently coerce and corrupt data later.
• A field becomes nullable. The key is present, the value is now sometimes null, and the consumer that never checked begins throwing on a subset of records.
• A new status appears. An endpoint that only ever answered 200 and 404 starts returning 429 under load, and nobody wrote a handler for it.
• A new endpoint shows up. Usually benign, occasionally the sign that a deploy rolled out something nobody told you about.

Why don't OpenAPI specs or contract tests catch API drift?

Because they check what was agreed, and drift is the gap between the agreement and the traffic. An OpenAPI spec is a document that describes an intention; it is accurate exactly as long as somebody updates it when the implementation changes, which is the step that gets skipped. A spec that says a field is required stays saying that after the provider stops sending it.

Contract tests are better because they execute, but they test the requests the test suite makes against the environment the suite points at. They catch a provider breaking a contract it knew about, in a scenario somebody wrote down, at the moment CI runs. They do not catch the field that only appears on records created before a migration, the type that changes only for large ids, or the third-party API that changed on their schedule without telling anyone, because none of those are in your test matrix.

The gap both share is that they are comparisons against a description. What makes drift detectable is comparing against the traffic itself, which has no opinion about what should be there and simply records what was.

How do I detect that an API response changed shape?

Learn the shape from the traffic, keep it, and compare new traffic against it. That is the whole mechanism, and the useful part is that it requires no schema, no instrumentation and no cooperation from the provider. For each endpoint you accumulate, from the bodies that actually crossed the wire, which keys appear, how often each appears relative to the number of samples, and which JSON types each key has taken.

Those two numbers are what turn a difference into a signal. A key's presence rate separates a field that is legitimately optional, and has always been around 60% present, from a field that was at 100% for a week and dropped to zero after Tuesday's deploy. A key's set of observed types separates a field that is honestly polymorphic from one that just became a string. Without the history, a single response tells you nothing; with it, the change is the event.

The practical form is a set of drift kinds, each a named difference you can ask for. This is what httpscope models: it learns each endpoint's shape from captured traffic and emits drift as it happens: response.field.missing, response.type.changed, request.field.added, status.new, endpoint.new, latency.up. That vocabulary matters more than it sounds, because it is what makes drift queryable after the fact. "What changed about this endpoint in the last ten minutes" becomes a question with an answer rather than a comparison of two screenshots taken a week apart.

What changed	Drift kind	Who it breaks
A key stopped appearing	response.field.missing	Consumers reading it without a default
A key's type changed	response.type.changed	Strict parsers; lenient ones corrupt quietly
A new key appeared in requests	request.field.added	Providers validating strictly on unknown fields
An endpoint answered a status it never had	status.new	Clients with no branch for it, commonly 429
An endpoint appeared	endpoint.new	Nobody, usually; sometimes an unannounced deploy
Latency stepped up and stayed	latency.up	Callers whose timeouts were tuned to the old shape

How do I catch a field that disappeared from a JSON response?

Watch the key's presence rate rather than inspecting a response. A single sample cannot tell you a field is missing, because an absent optional key looks identical to a removed required one. What distinguishes them is the trend: a field that was present in every one of the last ten thousand responses and is now present in none of them changed; a field that has always been at 70% is doing what it always did.

This is why shape is learned from samples rather than declared. The model is built by accumulating observed bodies per endpoint, tracking each key's presence and the types it has carried, which is an inference over real traffic rather than a schema anybody wrote. That has one honest consequence worth stating: the model is only as good as the traffic it has seen. An endpoint called twice has a weak shape, a field that is legitimately rare looks like an anomaly, and a watcher that just restarted knows nothing about last week. Treat a drift event on a thinly sampled endpoint as a lead rather than a finding.

The inverse case is just as useful and gets less attention. A field that appears where it never did before is often the earliest visible sign of a deploy, and it is how you discover that a provider shipped something on their schedule rather than yours.

Can an AI agent monitor my APIs for drift?

This is the part of API debugging that suits an agent best, because it is pure comparison over structured history and nobody enjoys doing it by hand. The agent's advantage is not cleverness; it is that it will actually look at every endpoint rather than the three you suspect.

What makes it work is giving the agent the shape data as something it can query with its own criteria rather than a feed of pre-judged alerts. With yeet the agent reads one page describing the available fields, served as markdown to anything that is not a browser, and then composes the question it actually has: endpoints that drifted in the last ten minutes with kinds limited to a missing field or a changed type, with the evidence attached. Because nothing in the surface is a verdict, the threshold for "interesting" is the agent's to set, which matters when the honest answer varies by endpoint.

The investigations that pay off immediately are narrow and specific. Which endpoints have a key whose presence has fallen below 90%, meaning something that used to be reliable now is not. Which have a key that has carried more than one type, meaning a parser somewhere is coercing or failing. What changed on this endpoint in the window around the deploy, which is the question you actually have at 2am. Each is a single query, and each would take a person an afternoon of log reading.

What should I run to start detecting API drift on a box?

Begin with apiwatch, because the first problem is usually that nobody has the list of APIs at all, and drift detection on an inventory you do not have is premature.

curl -fsSL https://yeet.cx | sh    # install yeet, once
yeet run gh:yeet-src/apiwatch      # list this box's APIs from 30s of live traffic

That gives you every HTTP API the machine serves and calls, from traffic rather than from configuration, with the process behind each one and the status codes it returns. --watch then keeps it running and alerts when an API returns 5xx, when its 4xx share jumps well above the normal it learned, or when its port goes quiet, with the failing request and response in the message.

Be clear about where that stops, because it is the boundary between alerting and drift: apiwatch watches statuses and availability, not body shapes. An endpoint that quietly drops a field keeps returning 200 and will not alert. Catching that needs the shape model described above, which is the layer beyond the status watcher, and it is worth knowing which problem you are solving before you choose a tool.

The bottom line: drift is a diff against traffic, not against a schema

Every tool that compares your API to a document will keep missing this, because the document is the thing that fell out of date. Specs describe intentions, contract tests cover the cases somebody wrote down, and status codes cannot express a key that stopped appearing.

The comparison that works is against what actually crossed the wire: this endpoint's keys, their presence rates, and the types they have carried, accumulated from real traffic and diffed as new traffic arrives. Then a field going missing is an event with a timestamp rather than a mystery that surfaces three days later as a null pointer in the strictest consumer you have.

Frequently asked questions

What is API drift?

API drift is a change in an endpoint's observed shape rather than its availability: a response field that stops being sent, a value whose type changes, a key that becomes nullable, a new status code, or a new endpoint appearing. Nothing errors at the protocol level, which is why monitoring based on status codes and uptime does not detect it.

How do I detect a breaking API change after a deploy?

Compare each endpoint's shape before and after. If you have accumulated which keys appear in its bodies, how often, and with what types, a deploy that removes a field or changes a type shows up as a difference in that model within minutes of the first affected response. Status-based alerting will not show it because the statuses are unchanged.

Why didn't my OpenAPI spec catch the change?

Because the spec describes the agreement, not the traffic, and it stays accurate only while somebody updates it alongside the implementation. A provider that stops sending a field does not edit your copy of their spec. Validating responses against a spec at runtime helps, but only for providers whose spec you have and trust, which excludes most third-party drift.

Can contract tests find API drift?

They find the subset that occurs in scenarios you wrote down, against the environment your suite points at, at the moment CI runs. They do not find a field that only vanishes for records created before a migration, a type that changes only for large identifiers, or a third-party API that changed on their schedule between your test runs.

How do I know if a missing field is a bug or just optional?

By its history. A key that has been present in essentially every response and is suddenly absent has changed; a key that has always appeared in some fraction of responses is optional and behaving normally. That is why presence rate matters more than any single sample, and why a shape model needs enough traffic before its conclusions are worth acting on.

What happens when an API changes a field from a number to a string?

Strict parsers reject the response outright, which at least fails loudly. Lenient ones coerce it, which is worse, because the wrong type propagates into storage and comparisons silently: an integer id that becomes a string still looks fine until something sorts on it or joins against it. Detecting the type change at the traffic layer catches both cases before the consequences spread.

Can I detect drift on a third-party API I don't control?

Yes, and it is the case with the most value, because you have no visibility into their deploys and no say in their timing. Reading the responses your own machine receives gives you their shape history whether or not they publish one, so a field they removed on a Tuesday is an event you can see rather than an outage you infer.

Does drift detection need me to instrument my services?

No, if the shapes are learned from traffic read at the kernel with eBPF rather than from code you added. The bodies cross the socket regardless of instrumentation, so the model is built from what the machine actually sent and received. That also means it covers services you did not write and runtimes you have no agent for.

How much traffic does a shape model need to be useful?

Enough that presence rates mean something, which is more than a handful of requests and less than you would guess. The practical rule is to treat endpoints with few samples as weakly known: a drift event on an endpoint called twice is a lead, and the same event on one called ten thousand times is a finding. Any tool worth using should let you see the sample count beside the claim.

Can an AI agent watch for API drift automatically?

That is a good fit, because the work is comparison across many endpoints and agents do not get bored. The requirement is that the agent can query the shape data with its own criteria rather than consume somebody's pre-set alerts, so it can ask for the endpoints whose key presence has fallen, or whose keys have carried mixed types, and bring the evidence back with the finding.

Sources

1. yeet API debugging use case, which names API drift after a deploy as one of the questions an agent answers from traffic. https://yeet.cx/use-cases/api-debugging
2. apiwatch, the published eBPF API watchdog that lists a machine's APIs from live traffic and alerts on 5xx, 4xx jumps and quiet ports. https://github.com/yeet-src/apiwatch
3. RFC 9110, HTTP Semantics, for the status codes a drift event appears alongside without changing. https://www.rfc-editor.org/rfc/rfc9110.html
4. RFC 8259, the JSON data interchange format, for the value types a body's shape is described in. https://www.rfc-editor.org/rfc/rfc8259.html
5. yeet documentation, the runtime the capture runs on. https://yeet.cx/docs/

Related resources

1. How do I get an AI agent to debug my API, on giving an agent the traffic instead of the code. https://yeet.cx/topical-takes/ai-agent-debug-api
2. How to monitor HTTP traffic on Linux, on capturing requests without a sidecar or an app change. https://yeet.cx/topical-takes/monitor-http-traffic-linux
3. Why is my TLS handshake failing on Linux, for the layer below a response that never arrives. https://yeet.cx/topical-takes/why-is-my-tls-handshake-failing