These days I'm spending a lot of time dealing with containers. All kinds of containers. Bloated and slim, containing web-servers and CLIs, written in compiled and interpreted languages, running atop Docker and Kubernetes... And these containers often misbehave.

My typical container debugging routine involves:

• Running an interactive shell in a container that may lack its own shell.
• Using advanced debugging tools no one should really have in their containers (like tshark).
• Editing code and configs right in the container (luckily, vanilla vim is everything I need for that).
• Executing commands in a crashed container.
• Extracting the container's filesystem to a local folder for further analysis (easy).
• Extracting the image's filesystem to a local folder (not much harder but more tedious).
• Forwarding container ports to the host system (because I forgot to publish them in advance).
• Accessing the container's localhost from the host system (e.g., a container with a reverse proxy sitting in front of a private app).
• Kubernetes-specific: Exposing cluster resources to the host system and vice versa.

I can solve most (if not all) of the above tasks by juggling standard tools and sub-commands (1, 2, 3, 4), but how nice it would be if there existed one ring tool to rule them all.

I started researching the available tools, and here is what I found.

Container debugging tools

​debug-ctr - "a command-line tool for interactive troubleshooting when a container has crashed or a container image doesn't include debugging utilities, such as distroless images." A very nice CLI that either 1) mounts a volume with debugging tools into a running container (using some black magic of modern Linux kernels) or 2) "clones" the target container with a new mount containing the debugging tools (handy when the target has crashed). But it's Docker- and probably Linux-only, and no image- and network-related functionality (yet?).

​docker-opener - "shell-in to any Docker container easily." This tool runs a shell in the target container. If there is no shell, it brings its own (busybox) to the target container and runs it. Despite the name, there is also a whole bunch of other helper commands, including (rather primitive) port forwarding. Almost meets my goals, but again seems to be limited to Docker (and Docker Compose).

​cntr - "a container debugging tool based on FUSE." This is probably the most impressive one, at least judging by the amount of effort needed to implement something like that. It allows you to mount the host filesystem into a running container (or rather to mount the container's filesystem to the host and then launch a host's shell re-using the container's namespaces, IIUC). Since the implementation is based on FUSE and namespace manipulation, it supports an extremely wide range of runtimes (Docker, Podman, LXC/LXD, rkt, systemd-nspawn, containerd, etc). But it has only two (and very similar) commands - "exec" and "attach" and assumes access to the runtime's machine (i.e., no Kubernetes ephemeral containers "by design").

Two other honorable mentions are the docker-slim debug command and the docker-debug tool, that are essentially the subsets of the debug-ctr and docker-opener offerings.

cdebug - a Swiss Army Knife Of Container Debugging

As you can see, none of the above projects was even close to fully satisfying my needs. So, as usually happens with programmers, I decided to write my own tool 🙈

Behold github.com/iximiuz/cdebug!

The goals that I'm chasing with cdebug are:

• Portability - the tool should leverage the underlying runtime's public API to work the same way on Linux and macOS and be able to deal with "remote" workloads (e.g., Docker Desktop's VMs or remote Kubernetes clusters).
• Completeness - not just a fancy way to run a shell in the target container but also a helper tool for the network-, image- and potentially other use cases (the "swiss army knife" part).
• Unified UX - subcommands like "cdebug exec" or "cdebug port-forward" should behave the same way for Docker, containerd, and Kubernetes.
• Automation - the tool should be packing multiple tedious and (human-)error-prone operations into higher-level commands reducing the time for debugging.
• Last by not least, not being limited to busybox-provided tools 🧙

You can think of "cdebug" as a marketplace for various container debugging commands. And the way it's designed should make it work for the majority of the most popular contemporary runtimes, including but not limited to Docker, containerd, and Kubernetes.

The tool is WiP, and the following commands are currently implemented.

cdebug exec: an interactive shell in a scratch, slim, or distroless container

The "cdebug exec" command is a crossbreeding of the "docker exec" and "kubectl debug" commands. You point the tool at a running container, say what toolkit image to use, and it starts a debugging "sidecar" container that feels like a "docker exec" session to the target container:

• The root filesystem of the debugger is the root filesystem of the target container.
• The target container isn't recreated and/or restarted.
• No extra volumes or copying of debugging tools is needed.
• The debugging tools are available in the target container.

Here is how it works under the hood:

By default, the "busybox:latest" image is used for the debugger sidecar, but you can override it with the "--image" flag. Combining this with the superpower of Nix and Nixery, you can get all your favorite debugging tools by simply listing them in the image name:

cdebug exec -it --image nixery.dev/shell/ps/vim/tshark <target-container>

cdebug port-forward: forward local ports to containers and vice versa

This command is another crossbreeding. This time it's "kubectl port-forward" and "ssh -L".

Currently, only local port forwarding is supported, but remote port forwarding ("ssh -R") is under active development.

Local port forwarding use cases:

• Publish "unpublished" port 80 to a random port on the host.
• Expose the container's localhost to the host system.
• Proxy local traffic to a remote host via the target.
• 🛠️ Expose a Kubernetes service to the host system.

Here is how the port forwarding command is implemented in the simple ("direct") case:

So, all the heavy lifting is done by socat (forwarding) and the container runtime (port publishing). Accessing the container's localhost is also possible, but with a trick:

Instead of conclusion

The tool is very early, but I already rely on it in my daily work. Looking forward to the calm(er) Christmas season to add support for more commands and runtimes. And, of course, I'm really curious to hear back from you!

Have a productive week ahead!

Ivan