Hello friends!

Ivan's here - with a well overdue February roundup of all things Linux, Containers, Kubernetes, and Server-Side craft 🧙

What I was working on

A lot of stuff on the dev side - not so much on the content side. But things are soon to reverse 🤞

Announcing labCTL - the long-awaited iximiuz Labs CLI

A dozen people have asked me over the past year-ish if there'll be access to the playgrounds from the local terminal and not only from the browser. And while I myself wanted this feature pretty badly, too, I couldn't justify spending time on developing it. Until February...

When David offered to prepare a piece of content or two for iximiuz Labs, I simply couldn't miss this opportunity! But the "author machinery" wasn't a thing yet - I was keeping my markdown files next to the platform's code, so I had to build something to let "external" authors create challenges, tutorials, and maybe even courses. And I couldn't come up with a better UX than CLI 🙈

However, building a CLI just for authors didn't make much sense - there is a lot of "common" work, like designing the project structure, creating an installer, adding the CLI session authentication support to the platform, etc. So, why not build the general-purpose CLI then?

Here is what you can already do with labctl today:

• Start a playground - e.g. labctl playground start k3s.
• SSH into a playground with labctl ssh <playground-id>.
• Forward local ports (including TCP and UDP) from the playground to your laptop with labctl port-forward <playground-id> -L <local>:<remote>.
• Forward remote ports from your laptop to the playground with good old ssh -R.
• Stop running playgrounds with labctl playground stop.
• Access playgrounds from VS Code, JetBrains, and other IDEs via their Remote SSH extensions.

I tried summarizing the main use case in this short video:

​

What do the above commands actually mean?

When you need a Docker host to try out a fishy container or a Kubernetes cluster to play with yet another Helm chart, you can get one available right in your terminal with a simple "labctl p start docker --ssh" or "labctl p k3s --ssh". The addition of the --ssh flag makes the command SSH into the new playground as soon as it's ready, and it usually takes just a few seconds 🚀

Port publishing in playgrounds is no longer limited to HTTP ports, so the choice of sandboxed services you can access has significantly widened. Notice how I'm using my local GUI client to access a MongoDB container running in the Docker playground, and it took me just one labctl port-forward command.

Last but not least, you can access the playgrounds from your local IDE now, which means you can bring all your extensions, including Copilot, with you! Check out another short video for the example:

​

The new features came out so cool and handy that I'm even considering moving the CLI into the premium tier 🙈 But it'll definitely remain free for a month or two while I polish it.

cdebug gets Kubernetes support

A while ago, I created a tool that allows executing commands in slim and distroless containers lacking their own shells. The tool uses an interesting trick that combines the portability of statically linked binaries (or Nix-based debug images) and chroot-ing to make the cdebug exec sessions feel very close to the native docker exec experience (i.e., as if your shell would be running in the target container itself, seeing the same filesystem).

In Kubernetes, kubectl debug was supposed to (partially) solve this problem, but still - you cannot use kubectl debug --target <cont> if the target container lacks the shell inside. And in addition to it, I keep running into other limitations of kubectl debug like the inability to set the user for the ephemeral container or override the ephemeral container's spec.

So, this month I finally found some time to implement my own version of Kubernetes debugger.

Here is what cdebug exec pod/mypod[/mycont] can do when running against a Kubernetes Pod:

• Execute commands and start sessions in shell-less containers.
• Make the shell see the target's filesystem, including mounted volumes, as it's seen by the target itself.
• Run ephemeral containers as the given user (kubectl debug lacks this ability completely).
• Auto-detect runAsNonRoot targets and adjust the debug container's security context accordingly (kubectl debug would simply hang if the chosen debug image would, by default, run as root).
• Override the ephemeral container spec arbitrarily (via cdebug exec --override '{ json patch }').

k'exp can now visualize Ephemeral Containers

A by-product of my cdebug work - k'exp, the Visual Kubernetes Explorer, can now visualize Ephemeral Containers, so you can quickly see how many of them have been started for the given Pod and what their specs and statuses are. Pretty handy for learning about the behavior of this special type of Kubernetes containers.

​

What I was writing

Unfortunately, with all the above development work (that coincided with a tough period at my primary job - yes, iximiuz Labs is still just a side-project), I haven't had much time for long-form writing. However, I managed to prepare a few challenges:

• ​Send a Signal to a Kubernetes App: a Simple Case​
• ​Send a Signal to a Kubernetes App: a Slim Container Case​
• ​Send a Signal to a Kubernetes App: a Non-Root Container Case​
• ​What Port Does This Kubernetes Pod Listen On?​
• ​When a FROM scratch Container Image Is Not Good Enough​
• ​Pick the Right Distroless Base Image For Your Application​
• ​Edit a File in a Running Container: a Simple Case​
• ​Edit a File in a Running Container: a Slim Base Image Case​
• ​Edit a File in a Running Kubernetes Pod: a Tricky Case​

Don't forget about cdebug and k'exp while solving them 😉

Also, it's not exactly writing, but I had my very first podcast recorded! Many thanks to wonderful Alexa Griffith for giving me this opportunity! 🙏

• ​Tech Learning Made Simple: Ivan Velichko's Educational Journey​

What I was reading

​All you need is Wide Events, not “Metrics, Logs and Traces” - I can definitely relate to the debugging experience in the author’s made-up example - I’ve done quite a few similar investigations during my SRE days at Booking. With a system in maintenance mode and under considerable load, you can do wonders by simply analyzing trends in your metrics. And funnily, most of the time, I used ages-old Graphite metrics for that, even though Booking also has (had?) the internal “wide event” machinery (similar to what the authors described). However, things work differently in a green field project where every other error is something you see for the first time and with no well-established historical usage patterns. That’s how you quickly start relying more on logs (including DEBUG and TRACE) than on metrics. Add microservices to the picture, and and that’s how you find yourself in the need to correlate these log lines, so you have to come up with something like (or exactly is) trace_id (to correlate the logs between services) and span_id (to correlate the log lines within a service). My problem with “wide events” is that they miss the temporality aspect - unlike OTel’s span that lasts in time, a “wide event” is instant, and I often want to look at a piece of telemetry that corresponds to a whole time frame spanning the entire (but only one) action, not to a single point in time. From my standpoint, every (structured) log line is a wide event. A bunch of such log lines produced by one service while performing one action (e.g., handling an HTTP request) logically represents a span, and a bunch of bunches produced by a tree of actions of different services represents a trace. IMO, OpenTelemetry got the model right, however it doesn’t quite get the DevEx yet. If I were to try to solve this problem, I’d bet on structured logging (a.k.a “wide events”) on the DevEx side but try to transparently transform the produced log lines into spans, traces, and metrics during the log collection phase.

​The end of 0% interest rates: what it means for software engineering practices - yet another good read by Gergely Orosz, which echoes with my favorite saying that microservices solve org problems but create technical problems. If you think a microservice should be used to solve problem X solely because of the encapsulation and/or architectural purity reasons, a separate module in a monolith is likely a better choice. Without the expected hypergrowth in the engineering headcount and the need to deal with the real load in production, post-ZIRP era engineers should be optimizing for technical simplicity and ease of development and operation. Read Gergerly’s post to learn how to become more valuable in the current employer-driven market.

​The Lazy Tyranny of the Wait Calculation - we’re all tired of AI buzz, but this blog post is a good read. It explained my subconscious decision to let the dust settle before getting my hands dirty with this hot new tech (if even). Sometimes, waiting in inaction is the winning strategy.

​Images as Code: The pursuit of declarative image builds - an interesting read from many perspectives (especially the historical one). However, from my standpoint, the “Images as Code” problem was solved long ago. Docker did it for all of us, and Dockerfiles gained extremely wide adoption. What’s not solved yet is declarative and binary reproducible builds. Of course, there is a niche for them, but do you really need them? “We don’t even have a practical use case for a service mesh.” (binary reproducible builds, WASM, edge deployment, add your favorite tech to the list).

​npm packages caught exfiltrating Kubernetes config, SSH keys - one of the reasons I use ephemeral and project-scoped dev environments and try not to install anything dev-related to my primary system.

​A different method to debug Kubernetes Pods - I enjoyed the plot, and the author’s skillset deserves a kudo, but the article describes exactly what you shouldn’t be doing while debugging containers or Pods. Use kubectl debug cdebug, folks.

​Fargate Is Not Firecracker - I have no reason not to trust Justin Garrison when he says there is no Firecracker in AWS Fargate. But I also agree with his “Should I care? Not really” sentiment. There is another hidden gem in this blog post, though. “Does Fargate lower operational burden by never having to worry about an operating system ever again? Nope. The operational burden shifts to other areas.” IMO, the same is true about almost every managed service out there. Simple, especially demo-like, use cases get easier and easier to solve with these numerous SaaS’s, while the complex services remain hard to build and operate. So when you start asking, “Why is this costing us so much more money than EC2? […] make sure you’re not lying to yourself about how much “heavy lifting” is being removed and how much is being shifted to something else.” Maybe DHH is not that unreasonable after all, with all his "moving off of the Cloud" posts. Have you tried Kamal yet?

Wrapping up

I haven't been luring new users to the platform actively enough in February. However, the platform's usage remained unbelievably high, and the sales of iximiuz Labs Premium kept happening - both facts are reassuring. Maybe I built something useful after all?

Thanks for reading, and have a creative week ahead!

Cheers
Ivan