Ivan Velichko

DevOps Challenge of the Week - Issue #6

Published about 2 months ago • 1 min read

Hello there! 👋

Debugging containerized applications is... challenging. Debugging apps that use slim variants of container images is double challenging. And debugging slim containers in hardened production environments is often close to impossible.

Before jumping to the DevOps problems that I prepared for you this week, let's review a few tricks that can be used to troubleshoot containers.

If the container has a shell inside, running commands in it with docker exec (or kubectl exec) is probably the most obvious choice when things go sideways:

However, good security practices demand that our containers don't include debugging tools (or even shells) by default. Here is what you can do if the misbehaving container is based on a slim or a distroless image:

  • Install debugging tools on the fly (slim).
  • Temporarily switch to a "fat" base image (slim & distroless).
  • Run an improvised debugger "sidecar" container using docker run and sharing target's namespaces.
  • Run a proper debugger "sidecar" container with kubectl debug.

Quite a few options, huh? Well, from my experience, none of them is super user-friendly. Installing debugging tools on demand is tedious, getting the docker run flags by heart from the first attempt is hard, and kubectl debug is not as flexible as I'd like it to be (you cannot run the debugger as the given user, make the sidecar privileged, etc.).

Third-party tools to the rescue!

cdebug is a my favorite tool that I wrote specifically to streamline the container debugging UX. It allows executing commands in scratch, distroless, and slim containers, using the debugger image of choice and making the debugger sidecar see the target's filesystem as-is. And it works the same for Docker, container, and, since recently, Kubernetes 🚀

Do recommend giving cdebug a try while solving today's challenges:

Good luck!

P.S. Traditional reminder - with iximiuz Labs Premium, you can get 2-4x faster VMs, unlimited daily playtime, and full content access. Monthly, yearly, and lifetime plans are available, with proper invoices, so that you can include this expense in your employer's dev education budget 😎

Ivan Velichko

Software Engineer at day. Tech Storyteller at night. Helping people master Containers.

Read more from Ivan Velichko

Hello friends! Ivan's here - with a well overdue February roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙 What I was working on A lot of stuff on the dev side - not so much on the content side. But things are soon to reverse 🤞 Announcing labCTL - the long-awaited iximiuz Labs CLI A dozen people have asked me over the past year-ish if there'll be access to the playgrounds from the local terminal and not only from the browser. And while I myself wanted this feature...

about 1 month ago • 7 min read

Hey hey! Are you ready for your next DevOps challenge? Last week, we all witnessed yet another terrifying cyber-security event, and this time, it was a direct hit - researchers from Snyk discovered a way to break out of containers! 🤯 The vulnerability was found in the fundamental component of the containerization ecosystem - the most popular implementation of the (low-level) OCI container runtime - runc. Notice how, on the diagram above, most high-level container runtimes actually rely on the...

2 months ago • 1 min read

Hello friends! Ivan's here - with my traditional monthly roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙 What I was working on After my announcement of the iximiuz Labs GA earlier this month, the platform's usage has more than doubled, so I had to focus on the system's stability and UX. As a result, I increased observability and test coverage, added one more bare-metal server, streamlined a bunch of use cases, and fixed a few bugs. The most notable user-facing...

2 months ago • 3 min read
Share this post