{"id":5327,"date":"2026-03-29T23:54:18","date_gmt":"2026-03-30T06:54:18","guid":{"rendered":"https:\/\/catbradley.io\/?p=5327"},"modified":"2026-03-29T23:54:18","modified_gmt":"2026-03-30T06:54:18","slug":"how-i-add-tools-to-my-immutable-linux-without-rebooting","status":"publish","type":"post","link":"https:\/\/catbradley.io\/?p=5327","title":{"rendered":"How I Add Tools to My Immutable Linux Without Rebooting"},"content":{"rendered":"

If you\u2019ve recently dipped your toes into the world of immutable Linux distributions<\/a> like Fedora Silverblue<\/a>, openSUSE MicroOS<\/a>, or even the Steam Deck, you’ll encounter this issue eventually. <\/p>\n

You try to perform a basic task, like adding a custom script to \/usr\/bin<\/code> or creating a global configuration directory, and the terminal throws an error: Read-only file system.<\/strong><\/p>\n

It\u2019s a frustrating moment. You chose an immutable OS <\/a>for the stability, the atomic updates, and the “unbreakable” nature of the system. But now you feel like a guest in your own house.<\/p>\n

The traditional fixes, manually mounting an overlay filesystem or using rpm-ostree to layer packages, either require a reboot or complex manual management.<\/p>\n

systemd-sysext<\/a> was built specifically to solve this problem. This often-overlooked utility uses OverlayFS under the hood but adds compatibility checking, systemd integration, and a standardized format, allowing you to dynamically merge binaries and libraries into \/usr<\/code> at runtime, without touching the underlying read-only image and without a reboot<\/strong>.<\/p>\n

Quick Look at Immutability<\/h2>\n

To understand why we need sysext<\/code>, you first have to understand why the Linux world is moving toward immutability. In a traditional “mutable” distribution like Ubuntu <\/a>or Arch<\/a>, the root filesystem is a giant, writable scratchpad. Any process with root privileges can modify any file in \/usr<\/code> or \/bin<\/code>. <\/p>\n

While this gives us total freedom, it\u2019s also a major source of system drift. Over time, manual changes, conflicting libraries, and failed package installations make the system unpredictable.<\/p>\n

Immutable distributions solve this by treating the operating system as a read-only image. When you update the system, you aren’t just changing individual files; you are switching to a completely new, pre-verified version of the OS. This makes the system “atomic”, it either works perfectly, or it rolls back to the previous version.<\/p>\n

The Problem: Seeing the “Read-Only” Barrier<\/h2>\n

While immutability is great for stability, it\u2019s a nightmare for “on-the-fly” troubleshooting. On a standard system, if I need to see why a network port is blocked, I might quickly install nmap<\/code> or tcpdump<\/code>. On an immutable system, I\u2019m stuck.<\/p>\n

You can see this in action by trying to manually add a file to your system binaries:<\/p>\n

sudo touch \/usr\/bin\/test_file<\/code><\/pre>\n
\"read<\/figure>\n
Instead of creating a file, you\u2019ll get a rejection message: touch: cannot touch '\/usr\/bin\/test_file': Read-only file system<\/code>. This proves that even with sudo<\/code>, the core of your OS is locked. <\/p>\n
To add a tool “the official way” (layering), you\u2019d have to run a command like rpm-ostree install<\/code> and then restart your computer. For a quick task, that’s a massive interruption. And this “rpm-ostree” is more of a Fedora Silverblue thing, it won’t work on non-Fedora atomic distros.<\/p>\n
How System Extensions Actually Work<\/h2>\n
I’d like to think of systemd-sysext<\/code> as a digital “overlay.” Instead of fighting the read-only filesystem, we are going to build a small directory structure that contains our tools and tell the system to virtually “merge” it on top of the existing \/usr<\/code>.<\/p>\n
This uses a kernel feature called OverlayFS<\/strong>. It takes your base (read-only) system as the “Lower” layer and your extension as the “Upper” layer. The result is a “Merged” view that the user interacts with. To your applications, it looks like the files were there all along.<\/p>\n
Step 1: Building Your First System Extension<\/h3>\n
You don’t need complex build systems to create a system extension. At its simplest, a sysext<\/code> is just a directory structure that mirrors the Linux root. Let’s build a workspace for a custom tool.<\/p>\n
First, mirror the Linux filesystem hierarchy:<\/p>\n
mkdir -p my-tool-ext\/usr\/bin\nmkdir -p my-tool-ext\/usr\/lib\/extension-release.d\/\n<\/code><\/pre>\n
\"Creating<\/figure>\n
Next, let’s create a simple test tool. In a real-world scenario, you could drop a compiled binary like ncdu<\/code> or htop<\/code> here, but for this guide, a script works perfectly:<\/p>\n
echo -e '#!\/bin\/shnecho \"Sysext is active on Fedora!\"' > my-tool-ext\/usr\/bin\/foss-tool\nchmod +x my-tool-ext\/usr\/bin\/foss-tool\n<\/code><\/pre>\n
\"Creating<\/figure>\n
Step 2: The Metadata “Passport”<\/h3>\n
This is the most critical step and where you can get stuck. systemd-sysext<\/code> acts as a gatekeeper. It will not merge an extension unless it knows exactly which OS version the extension is built for. To find out what your system expects, run:<\/p>\n
cat \/etc\/os-release | grep -E '^ID=|^VERSION_ID='<\/code><\/p>\n
\"Checking<\/figure>\n
On my setup, this returns ID=fedora<\/code> and VERSION_ID=43<\/code>.  <\/strong>If you are following along, make sure to replace these values with whatever your specific system reports. If you are on Silverblue 39, use those numbers. <\/p>\n
Now, create the mandatory release file:<\/p>\n
echo \"ID=fedora\" > my-tool-ext\/usr\/lib\/extension-release.d\/extension-release.my-tool-ext\necho \"VERSION_ID=43\" >> my-tool-ext\/usr\/lib\/extension-release.d\/extension-release.my-tool-ext\n<\/code><\/pre>\n
\"Creating<\/figure>\n
Before we merge anything into the live system, it\u2019s worth double-checking our work. A systemd-sysext<\/code> image is essentially a mirror of your root directory, so the file hierarchy must be exact. You can verify your layout by running: <\/p>\n
ls -R my-tool-ext\n<\/code><\/pre>\n
\"Verifying<\/figure>\n
You should see your binary sitting in usr\/bin<\/code> and your metadata ‘passport’ tucked away in usr\/lib\/extension-release.d\/<\/code>. If these aren’t in the right place, the system simply won’t know how to ‘map’ them during the merge.<\/p>\n
Step 3: The “Merge” Moment<\/h3>\n
Now that our extension has its “passport” ready, we move it to the system’s extension path and trigger the merge. This is the moment where the read-only barrier is bypassed:<\/p>\n
sudo cp -r my-tool-ext \/var\/lib\/extensions\/\nsudo systemd-sysext merge\n<\/code><\/pre>\n
\"Executing<\/figure>\n
Next, confirm the binary location. The system should see it as a standard system tool:<\/p>\n
ls -l \/usr\/bin\/foss-tool\nfoss-tool\n<\/code><\/pre>\n
\"Verifying<\/figure>\n
You can verify the status and run your new tool:<\/p>\n
systemd-sysext status\n<\/code><\/pre>\n
\"Checking<\/figure>\n
Your system is still technically read-only, but you\u2019ve successfully injected new functionality into it without a single reboot.<\/p>\n
Troubleshooting: When the Merge Fails<\/h2>\n
One of the most common frustrations is seeing the error: No suitable extensions found (1 ignored due to incompatible image)<\/code>. This isn’t a bug; it’s a safety feature.<\/p>\n
If your extension-release<\/code> file says you are on Fedora 42 but you actually just upgraded to Fedora 43, systemd<\/code> will block the merge. <\/p>\n
It does this because libraries often change between versions, and merging an incompatible binary could cause system instability. If you hit this error, simply update your metadata to match your current os-release<\/code> and re-run the merge.<\/p>\n
Reverting Without a Trace<\/h2>\n
The most powerful feature of systemd-sysext<\/code> isn’t just how easily it adds tools, it\u2019s how cleanly it removes them. Traditional package management often leaves behind config files or libraries that clutter your system over time. <\/p>\n
With sysext<\/code>, unmerging is a clean break:<\/p>\n
sudo systemd-sysext unmerge\n<\/code><\/pre>\n
\"Running<\/figure>\n
If you try to run your tool now, the shell will return a No such file or directory<\/code> error. The overlay has been lifted, and your \/usr<\/code> directory is exactly as it was when you first installed the OS.<\/p>\n
Why This Beats the Container Approach<\/h2>\n
A common question is: “Why not just use Distrobox<\/a>?” Containers are amazing for general applications, but they run in an isolated namespace. If you are trying to debug a kernel issue or analyze hardware peripherals, that isolation can get in the way.<\/p>\n
systemd-sysext<\/code> puts the tool directly on the host<\/strong>. It has the same permissions and visibility as a tool shipped with the OS itself. If you need a tool to “be” the system rather than just “run on” the system, sysext<\/code> is the surgical choice.<\/p>\n
Conclusion<\/h2>\n
The move toward immutable Linux shouldn’t feel like a move toward a “locked-down” experience. Tools like systemd-sysext<\/code> prove that we can have our cake and eat it too. We get the security of a read-only core and the flexibility to inject any tool we need instantly.<\/p>\n
<\/p>","protected":false},"excerpt":{"rendered":"
If you\u2019ve recently dipped your toes into the world of immutable Linux distributions like Fedora Silverblue, openSUSE MicroOS, or even the Steam Deck, you’ll encounter this issue eventually. You try…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5327","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-rss"],"_links":{"self":[{"href":"https:\/\/catbradley.io\/index.php?rest_route=\/wp\/v2\/posts\/5327","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/catbradley.io\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/catbradley.io\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/catbradley.io\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/catbradley.io\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5327"}],"version-history":[{"count":0,"href":"https:\/\/catbradley.io\/index.php?rest_route=\/wp\/v2\/posts\/5327\/revisions"}],"wp:attachment":[{"href":"https:\/\/catbradley.io\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5327"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/catbradley.io\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5327"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/catbradley.io\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5327"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}