Ivan Velichko

Ivan on Containers, Kubernetes, and Backend Development

Published over 1 year agoΒ β€’Β 6 min read

Hi friends!

It's Ivan again, with my traditional roundup of all things around

This month was mostly about container images and container runtimes. Here is what I've got for you:

  • Some random container thoughts and explanatory drawings
  • A bunch of handy tricks I wish I had learned sooner on my Cloud Native journey
  • A couple of streams on containerization tech (DockerSlim and containerd)
  • A new GitHub repository with yet another awesome list of resources
  • One big and one small(er) blog posts based on my recent research
  • And, of course, a whole lot of great reads, podcasts, and videos πŸ”₯

Let's get started!

SPONSORED Packages and software supply chains get compromised more and more often. So, everyone these days should be a security expert. I already confined every my project to a separate VM, but how can I know if there is some malicious activity happening inside? Learn how to use the Linux audit system in this practical post by Teleport (no marketing fluff).


Container Runtime Visualised

While preparing for one of the workshops, I came up with the following visual explanation of the low-level container runtime:

If this topic sounds interesting, you can find a few useful links in this Twitter thread.


Random Evening Thoughts

Might be not fully accurate, but this explanation could help get through a popular misconception:

twitter profile avatar
Ivan Velichko
Twitter Logo
September 24th 2022


Docker Tips & Tricks

To improve your container game 😎

Dealing With Containers Faster

Since (a very recent) discovery, this trick has saved me hours! Turns out, you can exec, stop, kill, remove, etc docker containers using just a short prefix of their IDs! Instead of typing (or rather copy-pasting):

docker rm b5758e341c2a can just go with:

docker rm b5

Credit goes to Michael Irwin!


Store Build Results In Local Folders

Another Docker trick that I learned on Twitter (thanks to Chris Guest). Turns out, the buildx plugin supports this for quite a while:

docker buildx build -o <some-dir> .

This is pretty powerful since it allows you to use Docker as a build system. You can find more details here (bake is the keyword to look for).

Non-Docker Tips & Tricks

Not so much about containers, but I must share them:


What I Was... Talking About

This is a new (and quite challenging) category for me, but I'm glad I gave it a try:

  • ​Hands-on Introduction to DockerSlim | Rawkode Live - Learn how to create extremely small container images using DockerSlim (and what magic is hidden behind the tool). As always on Rawkode Academy, it was a highly practical session.
  • ​containerd - a secret hero of the Cloud Native world - My first ever workshop where you can learn about the most widespread container runtime and how you can use it to dig deeper into containers, troubleshoot Kubernetes issues, or even as a building block for your own software.


What I Was Writing

On the blog, Twitter, ...and GitHub!

Awesome Container Tinkering

A yet another awesome list of resources? Nope, it's a list of awesome resources!

I started this GitHub project to collect links to (and thoughts about):

  • Lower-level building blocks (runtimes, libraries, CLIs)
  • Solutions for running containers locally and remotely
  • Helper tools and libraries to inspect, edit, or move images
  • In-depth write-ups and video resources on the above.

And I'm hoping for your contribution!

Distroless Container Images

Over the past few months, I spent quite some time researching the topic of how to create decent container images. Distroless (1.0) is one of the ways. You can learn more about the limitations of the scratch base images and the pros and cons of the distroless (aka "scratch for everyone else") base images in the below Twitter thread or in a long-form write-up on my blog - What's Inside Of a Distroless Container Image: Taking a Deeper Look:

twitter profile avatar
Ivan Velichko
Twitter Logo
September 5th 2022


How To Extract Image Filesystem Without Running Any Containers

Collected a bunch of receipts in this blog article.

Containers Are Just Linux Processes another popular misconception. Intrigued? Then read on!

Docker History And Evolution

For the digital archeologists (like myself), I called out on Twitter asking folks if the Docker's history is documented somewhere. The thread received lots of great replies (with some of them coming from people who are/were directly involved), shedding light on Docker's original architecture (monolith written in Go, essentially just a UX layer on top of LXC), its evolution (LXC's functionality got re-implemented in Go giving birth to, first, libcontainer, and then runc) and reshaping (containerd being factored out). Go check it out πŸ˜‰


What I Was Reading

  • ​LXC and LXD: a different container story - when a superior UX beats the superior tech. This article will make you feel that LXC and LXD are a viable alternative to Docker (in reality, Docker-like containers and the corresponding UX & ecosystem became the de facto standard). As an engineer, I'm on the LXC side, but as a pragmatic developer, I'm siding with Docker.
  • ​Yet Another Brief History of container(d) - a related one. From the old-day VMs, through Linux namespaces & cgroups, to LXC, then Docker, libcontainer, runc, and containerd. Well worth a read if you feel like a history lesson.
  • ​Extending Docker’s Integration with containerd - containerd was factored out back in 2015-2016, but the work continues even in 2022. The article is not about history, though. It's about the practical implications of this move. Docker starts using even more features of containerd. And this is good (at least) for two reasons: 1) more code reuse means better compatibility; 2) Docker's image dealing capabilities will start being enriched faster.
  • ​Where are containerd’s graph drivers? - if you want to learn more about the above move - the story behind storage drivers.
  • ​Vulnerability Management for Go - this is just great! A much-needed addition to any modern programming language.
  • ​In Defense of Package Managers - (in the context of deplorable software security situation) Package Managers isn't the problem. The problem is how we write (with lots of bugs & vulnerabilities) and release (developers are quicker than distro maintainers) our software. LTS Linux distros can't keep up with the industry's pace. Is stream distros with frequent patches the future?
  • ​What your scanner doesn't know **can** hurt you - continuing the above theme...
  • ​HackTricks - a worthwhile resource on container security with some practical examples, including Docker containers breakout.
  • ​Securing Containers with Seccomp: Part 1 & Part 2 - a highly-practical mini-series on how to use seccomp profiles to fine-tune your containers.
  • ​Istio: Introducing Ambient Mesh - this does sound like a reasonable redesign to me! Migrating legacy workloads to Istio is not easy (I know it first-hand), but with the new basic L4 layer implemented on the ambient infra side, this initial step might become much easier. And since there is no need for sidecar injection anymore, the final steps will also likely be simplified. The single-tenant, scalable, and optional L7 proxy layer implemented as a bunch of Envoy Pods does sound like a good idea too. If, from the app's code standpoint, mesh remains fully transparent, these changes are a very strong reason to start liking Istio again!
    • 🎬 Istio Ambient Mesh Launch Demo - nice demo (only 10m), but you can skip the hands-on part and instead look at three diagrams that Christian Posta shows there. They might shed more light than the above blog post.
    • ​[Kubernetes Podcast] Ambient Mesh, with Justin Pettit and Ethan Jackson - a lot of technical details, historical reasons for the redesign, insights on how meshes are actually used (mostly for mTLS), and how this new design was actually done by [virtual] networking people (vs. the sidecar-based design that was influenced by a background in application development).
  • ​Why fix Kubernetes and Systemd? - Kubernetes vs systemd. Wait, what? But on second thought, it makes total sense. Kubelet's and systemd's responsibilities do intersect substantially. Kris NΓ³va continues the "war on sidecars" started by service meshes some time ago.

And that's all the links for today!


Stay Tuned

Well, I thought the previous one was big... I should probably start doing this newsletter twice a month, what do you think? Hit reply and let me know :)



Ivan Velichko


Ivan Velichko

Software Engineer at day. Tech Storyteller at night. Helping people master Containers.

Read more from Ivan Velichko

Hello friends! Ivan's here - with a well overdue February roundup of all things Linux, Containers, Kubernetes, and Server-Side craft πŸ§™ What I was working on A lot of stuff on the dev side - not so much on the content side. But things are soon to reverse 🀞 Announcing labCTL - the long-awaited iximiuz Labs CLI A dozen people have asked me over the past year-ish if there'll be access to the playgrounds from the local terminal and not only from the browser. And while I myself wanted this feature...

about 1 month agoΒ β€’Β 7 min read

Hello there! πŸ‘‹ Debugging containerized applications is... challenging. Debugging apps that use slim variants of container images is double challenging. And debugging slim containers in hardened production environments is often close to impossible. Before jumping to the DevOps problems that I prepared for you this week, let's review a few tricks that can be used to troubleshoot containers. If the container has a shell inside, running commands in it with docker exec (or kubectl exec) is...

about 2 months agoΒ β€’Β 1 min read

Hey hey! Are you ready for your next DevOps challenge? Last week, we all witnessed yet another terrifying cyber-security event, and this time, it was a direct hit - researchers from Snyk discovered a way to break out of containers! 🀯 The vulnerability was found in the fundamental component of the containerization ecosystem - the most popular implementation of the (low-level) OCI container runtime - runc. Notice how, on the diagram above, most high-level container runtimes actually rely on the...

2 months agoΒ β€’Β 1 min read
Share this post