The future is minimal and immutable - a new generation of operating systems

What are minimal, monolithic, and immutable Linux distributions?

To start with, we need to cover some fundamental definitions. What is a Linux distribution, I don’t think, needs explaining. What needs explaining and is often thrown into one bag are the types of distributions listed above. So what is this all about?

• Minimal means that the system contains an absolute bare minimum set of tools needed to perform its job, in most cases, such systems are container-oriented and are not meant to be interacted with by humans, often not even having tools such as “sudo”.
• Monolithic very often comes in pair with minimal, and means that the system takes care of itself. Automatic updates, automatic health checks, automatic backups, automatic rollbacks. A “set it and forget it” type of system, so to speak.
• Immutable means that the core base of the system - everything except logs, personal configuration, and user files - are read-only, and cannot be tampered with by the user. The whole system is a solid image that is exactly the same across all existing installations.

As you can imagine, this can be mixed and matched however you want, minimal and monolithic, monolithic and immutable. This just boils down to the use case, what specific user needs and wants in their system. Each property gives different possibilities and has its advantages and flaws that need to be considered before deciding to switch to it.

But why? Isn’t this just complicating everything?

No, in fact - quite the opposite. Of course, a typical server or desktop OS doesn’t have these properties, and can and will work perfectly fine. But in some applications, it’s just convenient and easier to use for example minimal, monolithic or immutable distribution, both in server and in desktop space.

Let’s say we want to run a HA redirection proxy for our critical service. Yes, of course, we can use a typical, full-featured Linux server distribution, such as RHEL, SUSE, or UBUNTU, and it will work perfectly fine. But this needs resources - even as a virtual machine a typical server OS needs at least 10 GB of storage, 2 vCPUs and at least 1 GB of RAM, it’s just the bare minimum. On top of that, we need to consider regular updates. All of that is for one package, that we will set up once and not change for the next 2, 5 or 10 years. How convenient would it be to shrink this VM to just 2 GB of storage, 1 vCPU and 256 MB of RAM, set it and leave, knowing it will automatically update, backup and rollback if something goes wrong? Paired with a reproducible installation feature, such as Ignition in Fedora CoreOS, where you can deploy as many preconfigured servers as you like with one network-hosted file this sounds like a perfect solution.

And what if a company wants to have a simple, free, easy to use OS for their office workers, where they absolutely can not break anything? In this case, immutable desktop Linux distributions come into play, such as Fedora Silverblue/Kinoite, NixOS or VanillaOS. Their end users don’t have to worry about installations, configuration, or updates - the filesystem is read-only, updates are atomic (meaning they are committed fully or not committed at all), and users only have access to their home directory. Of course, they still can install applications they need from distro-agnostic packaging formats, such as Flatpak, Snap or AppImage, however, these applications are sandboxed, isolated from the main filesystem and even if they’re hacked or break, absolutely nothing happens to the OS itself. As a fun fact - MacOS is mainly an immutable OS and works on the same premise as the aforementioned NixOS or Fedora Silverblue.

Most recognisable examples of such systems

Here we have to make an important distinction between server and desktop OS. Both can be vastly different, and immutability is more prevalent in desktop space, whereas minimalism and monolithicity are rather associated with servers.

Here are some server examples:

• Fedora CoreOS is probably the oldest immutable, monolithic and minimalistic distribution, oriented towards this “fire and forget” approach. Originally named CoreOS Linux, it was founded in 2013, in 2018 bought by RedHat and then merged with Fedora project. It can be installed with predefined configuration, thanks to ignition, is fully packed with Docker and Podman, as well as all plugins needed to run containers. This OS also updates automatically in an atomic fashion, same as RancherOS, and has automatic rollbacks in case something goes wrong. It’s worth noting, that it houses Fedora’s own great feature - rpm-ostree, which is a package manager working much like git, creating a new “branch” of OS after installing or updating a package, which can be either merged into main or discarded and rolled back.
• SUSE MicroOS is SUSE’s answer to Fedora CoreOS, which works pretty much the same. Minimalistic, monolithic, immutable distribution, oriented towards hosting Docker containers on a smaller scale. It also updates automatically, and rolls back, but contrary to Fedora, uses SUSE’s solution, which is automated BTRFS snapshots (BTRFS being a tree-like filesystem with branches and live snapshots) with the help of snapper - SUSE’s in-house solution.
• Microsoft Azure Linux - or rather Common Base Linux Mariner. Yes, Microsoft created their own immutable, container, and Kubernetes-oriented Linux distribution! As of right now, it’s not very well explored, as it’s not officially and fully released yet, but the goal is to use it to host Microsoft Azure services, such as databases, web applications, and even Active Directory Administrator using Kubernetes.
• Upcoming Ubuntu 24.04 LTS will also reportedly offer a Snap-only, immutable variant, although we don’t know much about it yet. It’s worth noting that Ubuntu Server currently  offers a minimalistic installation option and in the upcoming 24.04 release this will be the default behaviour, with full installation as an optional feature.

And here’s a list of desktop examples:

• Fedora Silverblue/Kinoite was probably the first immutable Linux distribution that gained widespread popularity, thanks to being featured on the main page of the Fedora Foundation, and extensive advertising. As described above this OS offers a read-only filesystem, with the exception of the home directory, allows installing applications from Flatpak or AppImage, and uses the same rpm-ostree package manager as CoreOS, allowing it to create branches, backups and perform easy rollbacks. Sadly, this means you cannot use applications simply packaged as .rpm, such as TeamViewer, but for non-technical users this creates a rock-solid, unbreakable environment. To avoid confusion there are two names for the same OS to differentiate desktop environments they use - Silverblue uses GNOME, whereas Kinoite uses KDE Plasma.
• NixOS is way less recognizable than Fedora, but it is very cool nonetheless. Its main benefit is that its installations are completely reproducible. All configurations, starting from users present on the system, locales, file privileges, and even installed packages are listed in one, simple configuration file. This allows administrators to prepare one config and ensure that every single computer in the company is running exactly the same setup. You can even host this configuration file on version control platform, such as GitHub and have your workstations periodically fetch it and rebuild, so you can for example install an application on all computers in the company by just adding it to the file and waiting for all endpoints to fetch it and rebuild.
• VanillaOS is the “new kid on the block”. Based on Ubuntu 22.10 in its current release but in the process of rebasing to Debian Sid, it aims to completely erase the lines between different distributions. It does that by offering an immutable base system using a custom-developed ABRoot system to maintain stability and rollback capabilities. However, the biggest feature of VanillaOS is its custom apx package manager, based on Distrobox, that can install any package from any repository of any Linux distribution in their own isolated and sandboxed environment. I’ve been following this project closely and am very impressed with the state of the development, and what it aims to offer. I’m certain when it reaches a stable release it will stir up on the Linux desktop market. Because if you can have any package from any distribution, why wouldn’t you?

So why aren’t these widely used?

Well first of all they’re quite new. Immutability started gaining popularity only a few years ago and only in the Linux space, and as we all know - people don’t like changes. Especially in enterprise space, where if something works it mostly is being left untouched, in fear of breaking something. Why should a company switch their production website from an “old and trusty” Ubuntu (pun intended) server to this new and unexplored CoreOS, if it still works? What, they’re expected to abandon placing config files all around the filesystem and instead use containers? This mindset is starting to change but it’s a long way off being commonplace.

The second reason partially connects with the first - this requires a complete change of approach. Realising that you’re not supposed to tamper with anything outside your home directory and that everything should be concise and compressed in one place can be hard. A great example of this is Jenkins on Kubernetes. The old, traditional way is to have a server or virtual machine with a master node and several separate servers/virtual machines with huge amounts of disk space and resources as build agents. It was done this way for years. But the current consensus is to have agents as stateless containers, run on Docker or better yet, on Kubernetes, with no disk space, spun up when they are needed to do their job, push artifacts outside and be deleted. 

The third reason, which is strictly connected to using desktop immutable distros, is just that… Linux desktop is not very popular in the first place. Recently it reached 3% of the market share on desktops, but it’s still a far cry from Windows, MacOS, or even ChromeOS for that matter. And among this small percentage of users, an even smaller percentage will use immutable distros, because of reasons one and two. Plus, most Linux users are tinkerers who like to mess around and change their system, and immutability directly forbids them from doing that.

The fourth and final cause is - they are only fit for some use cases. While it’s easy to imagine a web server, Jenkins agent, or even a proxy server as a container, certain things just cannot be run this way. Things like VPN servers, DNS servers, and databases are just not meant to be stateless and discardable. While there are solutions to this, such as MariaDB or PostgreSQL containers, they are rather not recommended and not widely used in production environments, simply because of the need for data persistence in case of databases and extensive, frequent configuration changes in case of VPN or DNS. Some obstacles just cannot be overcome, at least not right now.

As in most things in life, it boils down to this - a habit is the enemy of change. We might see a surge in the popularity of such server solutions as time passes, when younger, technology-enthusiastic people will start their careers in tech with fresh minds and no habits. As much as many hate to admit it - this is the future, and even Microsoft recently switched Azure services to custom-made immutable Linux distribution instead of keeping Microsoft Server and RedHat Enterprise Linux as they used to.

And when it comes to desktop distributions - personally, I don’t think they will be much more than a fun novelty, at least not until big companies decide to switch from Windows and MacOS to Linux for their desktop needs. I really doubt that’ll ever happen though.