
Fifteen years building engineering platforms, currently focused on advanced AI infrastructure at yeet. I love turning the deeply complex topics into something everyone can understand. I relate deeply with the core yeet philosophy that you can just build things.
TL;DR. Check the client metadata first, because MongoDB already recorded some of what you want: every driver sends a handshake document at connection time, and
mongodlogs it with the remote address, the driver name and an optionalappName. If your connection strings setappName, it also appears incurrentOp.appNameand insystem.profile.appName, and you may be done. If they do not, or if your clients sit behind NAT and all arrive as one address, the server cannot help, because everything it knows about the sender is self-declared by the sender. Move to the client host and read the socket instead:ss -tnpmaps connections to pids, andyeet run gh:yeet-src/mongosnoopreports the process behind every individual command ascomm/pid.
Most of the MongoDB questions I get shown are not really about MongoDB. They are attribution questions: something is hammering a cluster, the cluster is doing its job and reporting healthy, and nobody can say which of eleven services is responsible. I write kernel probes for a living, so the version of this I see is always the one where the easy answers are already exhausted. There is one connection pool, one set of credentials, one source address, and a graph of connection count that has been climbing for two weeks. The reason it stalls is a property of the protocol rather than a gap in anyone's tooling: identity is not something a MongoDB command carries.
Ask the client host, not the database. The server's knowledge of who you are comes entirely from two sources, and both are weaker than they look. The first is the TCP source address, which tells you a machine and gets rewritten by any NAT, proxy or Kubernetes node between the client and the server. The second is the handshake document the driver sends at connection time, which is self-declared: the client says what it is, the server writes it down, and nothing verifies it.
That handshake is worth knowing in detail, because it is the best server-side answer available and most people have never read one. Drivers and client applications send identifying information at connection time, and mongod records it as a client metadata log entry with a remote field carrying the address and port, plus a doc field naming the driver and version, the OS type, name, architecture and kernel version, and optionally an application name. The documentation is explicit about the lifecycle: after the connection is established the client does not send that information again unless the connection is dropped and reestablished. So it describes a connection, once, at the beginning.
What it does not describe is a command. A pooled connection carries commands from every code path in your process, and the handshake happened before any of them ran. So even in the good case, where metadata was sent and logged, the resolution you get is per-connection and per-process-at-startup, never per-query and never per-request. That is the ceiling on the server-side answer, and no amount of log parsing raises it.
appName attaches a custom application name to a connection, and MongoDB surfaces it in three places: the mongod and mongos logs, the currentOp.appName field, and the system.profile.appName field. Setting it is the cheapest improvement available to anyone running several services against one cluster. It is supported by the drivers, by mongosh from 1.1.9 and by Compass from 1.28.4.
mongodb://db.internal:27017/orders?appName=orders-api-worker
Set it in every connection string you own and a large class of attribution question becomes answerable from the database, which is the right place to answer it if you can. A profiler document that carries appName: orders-api-worker tells you which service issued the operation without anyone touching a client host.
Three limits decide whether this is enough, and they are the reason the rest of this post exists:
appName to a running process, and you cannot add it retroactively to the traffic you are currently investigating. It is preparation, not diagnosis.orders-api-worker are one appName. If the question is which replica, or which of four code paths inside it, the field does not have the resolution.Something between the client and the server rewrote the source address, and the original is not recoverable from the server side. This is worth stating plainly because it is where most server-side investigations end without the investigator realizing the answer was destroyed rather than hidden.
Three rewrites account for nearly all of it. Containers on a bridge network leave the host through NAT, so every container on that host arrives at MongoDB as the host's address. A Kubernetes pod without host networking is translated at the node, so a whole node's worth of pods collapses into one address. A connection proxy or load balancer terminates the client's connection and opens its own, so what MongoDB sees is the proxy, with the client's address absent from the packet entirely.
In all three cases mongod logs exactly what it received, and it received a rewritten address. The information did not get lost in transit; it was replaced by design, by an intermediary doing the job it was deployed to do. The same asymmetry shows up whenever an intermediary is the thing reporting: what the TC layer sees works through the proxy version of it. The recovery path is always the same: go to the host where the address had not yet been rewritten. That is the client host, and on the client host the sending socket still belongs to a specific process.
ss -tnp, which prints TCP sockets with the owning process and pid:
ss -tnp 'dport = :27017'
-t limits it to TCP, -n prints numeric ports rather than resolving service names, -p shows the owning process, and the filter narrows it to connections whose destination port is 27017. lsof -i :27017 answers the same question if ss is unavailable, and both need privileges to report processes owned by other users. This is the fastest useful command in the whole post and it is often where the investigation should have started.
What it gives you is the set of processes holding connections right now. What it cannot give you is which of them sent a particular command, and the gap between those two things is larger than it sounds. Connection pooling is the reason: a pool is opened once, usually by a framework during startup, and every subsequent command from every code path in that process travels over connections the pool already holds. So ss tells you a pid has eight connections to your cluster, and says nothing about whether that pid sent one command in the last minute or forty thousand.
The second limit is timing. ss reports what exists at the moment you run it. A short-lived process, a cron job, a migration script or a serverless invocation that has already exited leaves no socket to find, and those are exactly the clients that produce mysterious traffic spikes. Sampling ss in a loop narrows that window without closing it.
Read the command in the kernel at the moment the process writes it, because that is the only place where the command content and the sender's identity exist at the same time. Once bytes leave the socket they carry no process identity; once they arrive at the server they carry a rewritten address. The one instant where both facts are available is the write itself.
yeet run gh:yeet-src/mongosnoop
mongosnoop attaches a kprobe to tcp_sendmsg and a kprobe plus kretprobe to tcp_recvmsg, so it sees each command as the client hands it to the kernel and the reply when it lands. Because a kprobe runs in the context of the calling process, the probe already knows which task triggered it, and each captured command carries comm/pid with no correlation step and nothing to join against. One run covers every MongoDB client on the box at once, which is the property that matters when you do not yet know which process to suspect.
Two structural facts make this the right tool for the attribution question specifically:
The honest boundary is the host. This reads one machine's clients, so a fleet question needs one run per host and something to aggregate them, which mongosnoop does not provide. It also retains only the most recent 2000 commands in memory and forgets them on exit, and it reports what was asked without ever reporting how the server answered it. For queries about the server's execution plan, explain is the tool and the enumerated limits are worth reading before you plan around it.
Because the kernel stores a task's command name in a fixed 16-byte field called TASK_COMM_LEN, including the terminating null, so any tool that reads comm from the kernel gets at most 15 characters of name. A process called orders-api-worker-prod arrives as orders-api-work, and two services whose names agree for the first fifteen characters are indistinguishable by comm alone.
The pid resolves the ambiguity, and for the full invocation /proc/<pid>/cmdline has it while the process is alive:
tr '\0' ' ' < /proc/1234/cmdline; echo
The null-to-space translation is needed because the kernel stores the arguments null-separated, so cat on that file runs them together. This is the correct move whenever a truncated comm is ambiguous, and it works only while the process exists, which is the same liveness constraint that applies to ss.
Six routes, and they differ mainly in where they stand relative to the address rewriting and the connection pool. The two questions worth keeping separate as you read this: does it identify a process, and does it attribute an individual command.
| Tool | Where it reads | Identifies the process | Attributes one command | Survives NAT | Needs preparation |
|---|---|---|---|---|---|
| mongosnoop | client socket and TLS boundary | Yes, comm/pid | Yes | Yes | No |
ss -tnp | client host socket table | Yes, pid | No | Yes | No |
| appName | server, self-declared | Service name only | Yes, in profiler | Yes | Yes, before connect |
| Client metadata log | mongod log | Driver and OS, once | No | No | No |
| currentOp | server, right now | appName if set | In-flight only | Partially | No |
| Application tracing | inside your process | Yes | Yes | Yes | Yes, instrumented |
The column that matters most is the last one. Two routes need nothing set up in advance and work on traffic that is already flowing, which is the situation you are in when the question becomes urgent. Everything else is either preparation you wish you had done or a server-side record that the network already degraded.
Best when you do not know which process to suspect and cannot modify anything. Reads at tcp_sendmsg and at the SSL_write boundary for TLS, tags every command with comm/pid, and covers every client on the host in one run. Gives up retention, fleet scope, and everything on the server side of the socket.
Best as the first command you run, because it is one line and often sufficient. ss -tnp resolves sockets to pids immediately and is usually already installed. Gives up per-command attribution and misses anything that has already exited.
Best as preparation, and the only route here that makes server-side records useful without touching a client host. Surfaces in the logs, currentOp and system.profile. Gives up per-process resolution and depends on the client telling the truth.
Best for the question "what kinds of clients are connecting to this cluster", answered without any client-side access. Records driver name and version, OS details and the remote address at connection time. Gives up per-command attribution entirely, and its address field is the one the network rewrote.
Best for catching a long-running operation in the act and seeing its appName if one was set. Reports in-progress operations only, with $ownOps letting an unprivileged user see their own. Gives up everything that completed, which on sub-millisecond commands is nearly all of it.
Best when it exists, because a span per query carries request context that no host-level tool can reconstruct: the HTTP route, the trace id, the user. Gives up any coverage of code nobody instrumented, which is where attribution problems tend to live, and gives up the kernel-side signals catalogued in kernel metrics your APM agent misses.
Yes, and the process identity is actually easier to get than the command content. TLS encrypts the payload before it reaches the socket, so a kprobe on the send path sees ciphertext and cannot decode a command. The pid is unaffected, because the probe fires in the calling process regardless of what the bytes contain. To recover the command as well, move the capture point inside the process:
yeet run gh:yeet-src/mongosnoop -- --tls-binary "$(command -v node)"
A uprobe on SSL_write reads the plaintext out of the application's own buffer before encryption, and a uretprobe on SSL_read reads the reply after decryption, so identity and content arrive together. The -- routes the flag to the script rather than to the runtime. Which target to pass depends on how the client links its TLS: libssl.so covers Python's pymongo, distro-packaged Node and .NET, while Node's official builds and mongosh need their own binary by path, because they statically link BoringSSL but keep the OpenSSL symbol names and ship unstripped.
Two clients are out of reach and no flag changes that. Go's crypto/tls is implemented in Go, so there is no C symbol to hook, and Java's TLS lives inside the JVM's JSSE. For a Go service talking to Atlas the routes that still work are ss -tnp for connection ownership, appName for server-side attribution, and application-level instrumentation for per-command detail. The same constraint applies to redissnoop, because it is a property of those runtimes.
The ordering constraint matters more here than anywhere else in this post: a uprobe fires only for processes that start after it attaches. Start the capture first, then start or restart the client. An already-running TLS client is invisible until it restarts, while the plaintext kprobe path has no such limit and sees processes that were running all along.
Whenever the question is not actually "which process", which is more often than the framing of this post suggests. Match the tool to the unit you need identified:
appName plus your log pipeline, or an APM. Retention and fleet aggregation are exactly what a host-local probe does not have, and retrofitting them onto one is a worse plan than using a tool built for it.ss -tnp first, then socket-level capture if you need per-command detail. This is the gap the kernel probe exists for.currentOp. It is a snapshot, so it finds stuck operations and index builds and misses everything fast.For the same attribution question asked of CPU and disk instead of a database, find which process is slowing your machine covers the host-level version. If the query you actually need answered is why the traffic is slow rather than who is sending it, why is my Redis slow works through the equivalent reasoning for a different datastore. And when none of the ready-made tools is the right shape, the probes are ordinary JavaScript on yeet, so a JSON feed into whatever you already run is a branch in a callback.
Set appName in every connection string you control, today, because it costs one URI parameter and it is the only thing on this list that makes the database's own records attributable. Run ss -tnp 'dport = :27017' first when a question is already urgent, since one line resolves connections to pids and frequently ends the investigation. Use mongosnoop when you need to know which process sent which command, on a host where nothing can be modified or restarted, accepting one host, 2000 commands of memory and no server-side execution detail. Reach for currentOp only for operations slow enough to still be running, and for an APM when the real question is which request or which user, because request context exists only inside your process.
The trap is treating the server's client metadata as an answer. It is a self-declared document sent once per connection, carrying an address that any NAT or proxy has already rewritten, and reading it as authoritative is how a service spends a week convinced the traffic comes from the wrong place. The database records what it was told. Attribution lives on the client host, where the write happened.
Start with the client metadata the driver sent at connection time, which mongod records in a client metadata log entry containing a remote address and port plus a document naming the driver, the OS and an optional application name. If the connection string set appName, that name also appears in currentOp and in system.profile documents. All of it is self-declared by the client, so it identifies whoever configured the connection string and not the process that is actually running.
Not from the database if the pods use the default networking, because a pod without host networking is source-translated at the node, so every pod on that node reaches MongoDB as one address. From the node itself you can, since the sending socket belongs to a pid in the pod's namespace and a host-wide kernel probe reports it regardless of namespace. The alternative that works cluster-wide is setting appName per deployment, which makes the server's own records attributable at the service level.
Not from the database, if the containers share a host and go out through NAT, because the server sees one source address for all of them. From the host running the containers you can, because the sending socket belongs to a pid in a specific namespace. A kernel probe on the socket path reports the process that wrote each command regardless of namespace, so a containerized client appears alongside a host process in the same output.
Run ss -tnp on the client host and match the remote address to your MongoDB port. ss prints the owning process and pid for each socket, so a connection to port 27017 resolves to a pid immediately. This tells you which processes hold connections, not which one issued a specific query, because a pooled connection is shared and a pool is often opened by a framework rather than by the code path you are chasing.
Because something between the client and the server rewrote the source address. Containers on a bridge network leave through NAT with the host's address, a Kubernetes pod without host networking is translated at the node, and a connection proxy or load balancer terminates and reopens the connection. The server logs what it received, so once an address is rewritten the original sender is not recoverable from the server side at all.
It records an appName field when the client set one, and nothing else that identifies the sender. A system.profile document describes the operation as the server executed it: the namespace, the command, the duration and the plan summary. There is no pid, no container id and no request context, because none of that ever crossed the wire. If appName was not set, profiler documents from different services on one cluster are indistinguishable.
Yes, if you capture at the socket rather than inside the application. A kprobe on tcp_sendmsg reads the bytes the process handed the kernel, which requires no cooperation from the program, no source code, no instrumentation library and no restart. This is how you see queries from a vendor binary, a background worker nobody owns, or a framework issuing its own commands underneath your code.
Not for plaintext MongoDB connections. A kprobe on the kernel socket path is host-wide and sees processes that were already running when it attached, so nothing restarts. TLS capture is the exception: a uprobe fires only for processes that start after it attaches, so an already-running encrypted client stays invisible until it restarts. Plan the order of operations around that difference.
Run ss -tnp for TCP sockets with process information, or lsof -i :27017 for a specific port. ss is the modern replacement for netstat and is usually already installed. Both need privileges to show processes owned by other users, and both report socket ownership at the moment you run them, so a short-lived client that has already exited leaves nothing to find.
Because the kernel stores a task's command name in a fixed 16-byte field, TASK_COMM_LEN, including the terminating null. Any tool reading comm from the kernel inherits that limit, so a long process name arrives cut short and two services whose names agree for the first fifteen characters are indistinguishable by comm alone. The pid disambiguates them, and reading the full command line means going to /proc/<pid>/cmdline instead.
Yes, because the capture happens inside the process before encryption. A uprobe on SSL_write reads the plaintext command out of the application's own buffer and the probe already knows which pid it fired in, so process identity and command content arrive together. This works for OpenSSL and for statically linked BoringSSL builds that keep the OpenSSL symbol names, and not for Go's crypto/tls or Java's JSSE, which expose no C symbol to hook.
mongosh can send identifying information at the time of connection and that "after the connection is established, the client does not send the identifying information again unless the connection is dropped and reestablished"; the worked client metadata entry showing a remote field with address and port, a client context, and a doc document carrying application name, driver name and version, and OS type, name, architecture and kernel version.appName specifies a custom app name for the connection and is surfaced in three places: the mongod and mongos logs, the currentOp.appName field of the currentOp command and db.currentOp() method, and the system.profile.appName field in database profiler output. Supported by the drivers, by mongosh starting in 1.1.9 and by Compass starting in 1.28.4, with no documented length limit.mongod instance in an inprog array, reflecting only what is executing at the moment the command runs, so a completed operation is absent from subsequent calls. Requires the inprog privilege action on systems running with authorization, with the documented exception that $ownOps lets a user view their own operations without it.ss dumps socket statistics and is documented as intended to replace netstat; -t restricts output to TCP sockets, -n prevents service-name resolution, -p shows the process using each socket, and dport/sport filter expressions narrow output by port, which is the whole basis of resolving a MongoDB connection to an owning pid.usedDisk indicator when an aggregation stage wrote temporary files because it exceeded the documented 100 megabyte per-stage in-memory limit, which is an example of the kind of execution detail that exists only server-side and never appears in a client-side capture.tcp_sendmsg and tcp_recvmsg plus uprobes on SSL_write and SSL_read, six BPF programs feeding one 512 KB ring buffer, each captured command tagged with the sending process as comm/pid, host-wide kprobe coverage that includes containerized clients regardless of namespace, and a 2000-command in-memory log with no retention past exit.crypto/tls and Java's JSSE expose no C symbol so those clients are uncapturable over TLS, processes already running when the TLS uprobes attach stay invisible until they restart while the plaintext kprobe path has no such limit, comm is truncated to 16 bytes by the kernel, and nothing beyond the host or older than 2000 commands is retained.Built with yeet, a JS runtime for writing eBPF programs on Linux machines. Join us on discord.