It started with a seemingly straightforward problem: a container failed to start correctly because an expected D-Bus socket was missing. Inside the affected container, the expected socket was:
/run/dbus/system_bus_socket
At first glance, this looked like a D-Bus problem. The socket was missing, and the D-Bus daemon was reporting that it had no socket to receive connections on.
But there was something strange about the behavior.
First, other containers on the same host were working normally. If D-Bus itself were fundamentally broken, why was the problem isolated to one container?
Second, migrating the affected Pod to another host resolved the problem. This suggested that something specific to the original host might be involved, rather than a problem inherent to the service or its configuration.
I now had a more interesting question: what was different about the original host?
I tried restarting the Pod and restarting the D-Bus daemon, but neither helped.
I also consulted an AI assistant, which suggested that D-Bus should start working automatically after restarting the daemon.
That sounded reasonable, but it didn’t solve the problem.
At this point, I decided to stop focusing on the missing socket and investigate what was happening underneath.
I’d heard people talk about how interesting bpftrace was, so I decided this would be a good opportunity to try it in a real production debugging scenario.
After several attempts, I came up with the following script (with AI support):
bpftrace -v -e '
tracepoint:raw_syscalls:sys_exit
/comm == "systemd"/
{
$cg = curtask->cgroups->dfl_cgrp;
$parent = str($cg->kn->parent->name);
if ($parent == "pod56875133-fd05-42c9-9d8a-f563be4f1365") {
$ret = (int64)args->ret;
$nr = args->id;
if ($ret < 0 ||
$nr == 41 || $nr == 42 ||
$nr == 49 || $nr == 50) {
printf("pid=%d syscall=%ld ret=%ld\n",
pid, $nr, $ret);
}
}
}
interval:s:120
{
exit();
}
The script traces system-call exit events, filters for processes whose command name is systemd, and narrows the results to the target Pod’s cgroup. It prints failed system calls, along with selected networking-related system calls, for a two-minute observation window.
The output was long and, without decoding the system-call numbers, difficult to interpret.
Here are some representative results:
pid=1919284 syscall=34 ret=-514
pid=1015626 syscall=21 ret=-2
pid=1015626 syscall=21 ret=-2
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=21 ret=-2
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=21 ret=-2
pid=1015626 syscall=112 ret=-1
pid=1015626 syscall=2 ret=-6
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=41 ret=3
pid=1015626 syscall=54 ret=-1
pid=1015626 syscall=42 ret=-2
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=41 ret=3
pid=1015626 syscall=49 ret=0
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=169 ret=-1
pid=1015626 syscall=2 ret=-2
pid=1015626 syscall=294 ret=-24
pid=1015626 syscall=2 ret=-2
There were many other failures, including ENOENT (-2), which can be perfectly normal when a process probes for files or configuration paths that don’t exist.
The challenge was finding the failure that actually mattered.
I used an AI assistant to help interpret the system-call numbers and error codes.
One line immediately stood out:
pid=1015626 syscall=294 ret=-24
On the x86-64 Linux architecture, system-call number 294 corresponds to inotify_init1(), and -24 represents EMFILE.
In other words, the traced process had attempted to create an inotify instance, but the operation had failed.
Inotify is a Linux mechanism that allows applications to monitor filesystem events. Creating an instance is the first step; applications can then attach watches to files and directories.
The important clue was the error.
EMFILE can indicate that the per-real-UID inotify instance limit has been reached. It can also indicate that the process has exhausted its own file-descriptor limit. So I needed to check which limit was responsible.
I inspected the host’s inotify configuration:
sysctl fs.inotify.max_user_instances
The result was:
fs.inotify.max_user_instances = 128
The configured limit was only 128 instances per real user ID.
This was a strong lead. Multiple processes running as the same user could contribute to the same inotify instance limit, even if those processes belonged to different containers.
In this environment, processes were running as root. That meant the inotify instances created by other processes sharing the same real UID could potentially consume the available capacity.
The evidence now pointed toward a host-level resource exhaustion issue rather than a simple D-Bus configuration problem.
I increased the inotify instance limit to approximately 20,000:
sysctl -w fs.inotify.max_user_instances=20000
After the change, the previously failing Pod could start normally again.
This result supported the diagnosis: the original host had an insufficient inotify instance limit for the workload, and increasing it resolved the startup failure.
The fix itself was simple. The interesting part was finding it.
A missing D-Bus socket naturally suggests a D-Bus problem. But restarting the daemon didn’t help, and moving the Pod to another host changed the outcome.
Those observations were reasons to investigate beyond the application layer.
Instead of guessing which component was failing, I traced system-call results and found a specific failure: inotify_init1() returning EMFILE.
That gave me a concrete direction for further investigation.
The inotify instance limit is associated with the real user ID, rather than being inherently independent for each container.
When several processes share the same UID, their resource consumption can interact in unexpected ways. A container can therefore encounter a resource constraint that is difficult to explain by looking at that container alone.
AI helped me decode an otherwise cryptic line of output. But the real investigation involved connecting that clue to the kernel configuration, understanding the resource limit, and validating the fix.
The useful result wasn’t simply an AI-generated explanation. It was a hypothesis that could be tested against the running system.
I learned that even when a production problem doesn’t initially make sense, there is usually a deeper explanation waiting to be discovered.
A missing socket led me to system-call tracing. System-call tracing led me to inotify_init1(). And that led me to a host-level resource limit.
It was also a good reminder of why I enjoy systems engineering: the most satisfying part isn’t applying the fix. It’s figuring out why the system behaved the way it did in the first place.
And yes, bpftrace really is cool.