Using Nix as a Yocto Alternative Building a bootable Linux image from the ground up in a declarative way
11.07.2024
Building system images for embedded devices from the ground up is a very complex process, that involves many different kinds of requirements for the build tooling around it. Traditionally, the most popular build systems used in this context are the Yocto project and buildroot.
These build systems make it easy to set up toolchains for cross-compilation to the embedded target architecture, so that the lengthy compilation process can be offloaded to a more beefy machine, but they also help with all the little details that come up when building an image for what effectively amounts to building an entire custom Linux distribution.
More often than not, an embedded Linux kernel image requires changing the kernel config, as well as setting compiler flags and patching files in the source tree directly. A good build system can take a lot of pain away for these steps, by simplifying this with a declarative interface.
Finally, there is a lot of value in deterministic and reproducible builds, as this allows one to get the very same output image regardless of the context and circumstances where and when the compilation is performed.
Introducing Nix
Today we take a look at Nix as an alternative to Yocto and buildroot. Nix is a purely functional language that fits all of the above criteria perfectly. The project started as a PhD thesis for purely functional software deployment and has been around for over 20 years already. In the last few years, it has gained a lot of popularity in the Server and Desktop Linux scene, due to its ability to configure an entire system and solve complex packaging-related use cases in a declarative fashion.
In the embedded scene, Nix is not yet as popular, but there have already been success stories of Nix being used as an alternative to Yocto. And with the vast collection of over 80,000 officially maintained packages in the nixpkgs repo (this is more than all official packages of Debian and Arch Linux combined), Nix certainly has an edge over the competition, as most software stacks are already packaged. For most common hardware you will also find an overlay of hardware-specific quirks in the nixos-hardware repository. However, since the user demographic of Nix is slightly different at the moment, for more obscure embedded platforms you are still much more likely to find an OpenEmbedded layer.
For the Nix syntax it is best to refer to the official documentation, but being a declarative language, the following parts should be easy to comprehend even without further introduction. The only uncommon caveat is the syntax for lambdas, which essentially boils down to { x, y }: x + y being a lambda, that takes a set with two attributes x and y and returns their sum.
Cross-Compilation
On Nix there are two possible approaches to do cross-compilation. The first one would be to just pull in the packages for the target architecture (in this case aarch64) and compile it on a x86_64 system by configuring qemu-user as a binfmt_misc handler for user space emulation. While this effectively cheats around the actual cross-compilation by emulating the target instruction set, it has some advantages such as simplifying the build process for all packages, but most importantly it allows to reuse the official package cache, which has already built binaries for most aarch64 packages. While most of the build process can be shortcut with this, packages that need to be actually built will build extremely slow due to the emulation.
For that reason we use the second approach instead, which is actual cross-compilation: Instead of pulling in packages normally, we can use the special pkgsCross.${targetArch} attribute to cross-compile our packages to whatever we pass as ${targetArch}. The majority of the packages will just work, but rarely some packages need extra configuration to cross-compile. For example, for Qt we need to set QT_HOST_PATH to point to a Qt installation on the build host, as it needs to run some tools such as moc during the actual build on the build host. The disadvantage of this approach is that for most packages the official Nix cache does not provide binaries, so we have to build everything ourselves.
Of course builds are already cached locally by default (because they end up as a derivation in /nix/store), but it is also possible to set up a custom self-hosted Nix cache server, so that binaries have to be built only once even across multiple machines.
Building an image with Nix
As an example we will be looking into building an entire system image for the Raspberry Pi 3 Model B from the ground up. This includes compiling a custom Linux kernel from source, building the entire graphics stack including Mesa and the whole rest of the software stack. This also means we will build Qt 6 from source.
As example application we will deploy GammaRay, which is already packaged in the official Nix repository. This is to illustrate the advantage of having the large collection of nixpkgs at our disposal. Building a custom Qt application would not be much more involved, for reference take a look at how GammaRay itself is packaged.
Then at the end of the build pipeline, we will have an actual image that can be flashed onto the Raspberry Pi to boot a custom NixOS image with all the settings we have configured.
To build a system image, nixpkgs provides a lot of utility helper functions. For example, to build a normal bootable ISO that can install NixOS like the official installer, the isoImage module can be used. Even more helper functions are available in the nixos-generators repository. However, we do not want to create an “installer” image, instead we are interested in creating an image that can be booted straight away and already has all of the correct software installed. And because the Raspberry Pi uses an SD card, we can make use of the sd-card/sd-image.nix module for that. This module already does a lot of the extra work for us, i.e. it creates a MBR partitioned SD card image, that contains a FAT boot partition and an ext4 partitioned root partition. Of course it is possible to customize all these settings, for example, to add other partitions, we could simply append elements to the config.fileSystems attribute set.
Leaving out some slight Nix flakes boilerplate (we will get to this at the end), with the following two snippets we would already create a bootable image:
nixosConfigurations.pi = system.nixos {
imports = [
"${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
nixosModules.pi
];
images.pi = nixosConfigurations.pi.config.system.build.sdImage;
};
This first snippet imports the sd-image module mentioned above and links to a further nixosModules.pi configuration, that we define in the following snippet and that we can use to configure the entire system to our liking. This includes installed packages, setup of users, boot flags and more.
nixosModules.pi = ({ lib, config, pkgs, ... }: {
environment.systemPackages = with pkgs; [
gammaray
];
users.groups = {
pi = {
gid = 1000;
name = "pi";
};
};
users.users = {
pi = {
uid = 1000;
password = "pi";
isSystemUser = true;
group = "pi";
extraGroups = ["wheel" "video"];
shell = pkgs.bash;
};
};
services = {
getty.autologinUser = "pi";
};
time.timeZone = "Europe/Berlin";
boot = {
kernelPackages = lib.mkForce pkgs.linuxKernel.packages.linux_rpi3;
kernelParams = ["earlyprintk" "loglevel=8" "console=ttyAMA0,115200" "cma=256M"];
};
networking.hostName = "pi";
});
Thus, with this configuration we install GammaRay, set up a user that is logged in by default and uses a custom Raspberry Linux kernel as the default kernel to boot. Most of these configuration options should be self-explanatory and they only show a glimpse of what is possible to configure. In total, there are over 10,000 official NixOS configuration options, that can be searched with the official web interface.
The line where we add GammaRay to the system packages, also automatically adds Qt 6 as a dependency. However, as mentioned previously, this does not work quite well with cross-compilation. In order for it to build we need to patch a dependency of GammaRay to add the QT_HOST_PATH variable to the cmake flags. What would involve bizarre gymnastics with most other build systems, becomes incredibly simple with Nix: we just add an overlay that overrides the Qt6 package, there is no need to touch the definition of GammaRay at all:
nixpkgs.overlays = [
(final: super: {
qt6 = super.qt6.overrideScope' (qf: qp: {
qtbase = qp.qtbase.overrideAttrs (p: {
cmakeFlags = ["-DQT_HOST_PATH=${nixpkgs.legacyPackages.${hostSystem}.qt6.qtbase}"];
});
});
})
];
And note, how we pass the path to Qt built on the host system to QT_HOST_PATH. Due to lazy evaluation, this will build Qt (or rather download it from the Nix binary cache) for the host architecture and pass the resulting derivation as string at evaluation time.
In order to quickly test an image, we can write a support script to test the output directly in qemu instead of having to flash it on real hardware:
qemuTest = pkgs.writeScript "qemuTest" ''
zstd --decompress ${images.pi.outPath}/sd-image/*.img.zst -o qemu.img
chmod +w qemu.img
qemu-img resize -f raw qemu.img 4G
qemu-system-aarch64 -machine raspi3b -kernel "${uboot}/u-boot.bin" -cpu cortex-a53 -m 1G -smp 4 -drive file=qemu.img,format=raw -device usb-net,netdev=net0 -netdev type=user,id=net0 -usb -device usb-mouse -device usb-kbd -serial stdio
'';
Here we decompress the output image, resize it so that qemu can start it and then use the uboot boot loader to finally boot it.
Taking a final look at our config, we now have the following flake.nix file:
{
description = "Cross compile for Raspberry";
inputs = {
nixpkgs.url = "nixpkgs/nixos-23.11";
};
outputs = { self, nixpkgs, nixos-hardware }: let
hostSystem = "x86_64-linux";
system = nixpkgs.legacyPackages.${hostSystem}.pkgsCross.aarch64-multiplatform;
pkgs = system.pkgs;
uboot = pkgs.ubootRaspberryPi3_64bit;
in rec {
nixosConfigurations.pi = system.nixos {
imports = [
"${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
nixosModules.pi
];
};
images.pi = nixosConfigurations.pi.config.system.build.sdImage;
qemuTest = pkgs.writeScript "qemuTest" ''
zstd --decompress ${images.pi.outPath}/sd-image/*.img.zst -o qemu.img
chmod +w qemu.img
qemu-img resize -f raw qemu.img 4G
qemu-system-aarch64 -machine raspi3b -kernel "${uboot}/u-boot.bin" -cpu cortex-a53 -m 1G -smp 4 -drive file=qemu.img,format=raw -device usb-net,netdev=net0 -netdev type=user,id=net0 -usb -device usb-mouse -device usb-kbd -serial stdio
'';
nixosModules.pi = ({ lib, config, pkgs, ... }: {
environment.systemPackages = with pkgs; [
uboot
gammaray
];
services = {
getty.autologinUser = "pi";
};
users.groups = {
pi = {
gid = 1000;
name = "pi";
};
};
users.users = {
pi = {
uid = 1000;
password = "pi";
isSystemUser = true;
group = "pi";
extraGroups = ["wheel" "video"];
shell = pkgs.bash;
};
};
time.timeZone = "Europe/Berlin";
boot = {
kernelParams = ["earlyprintk" "loglevel=8" "console=ttyAMA0,115200" "cma=256M"];
kernelPackages = lib.mkForce pkgs.linuxKernel.packages.linux_rpi3;
};
networking.hostName = "pi";
nixpkgs.overlays = [
(final: super: {
makeModulesClosure = x: super.makeModulesClosure (x // { allowMissing = true; }); # workaround for https://github.com/NixOS/nixpkgs/issues/154163
qt6 = super.qt6.overrideScope' (qf: qp: {
qtbase = qp.qtbase.overrideAttrs (p: {
cmakeFlags = ["-DQT_HOST_PATH=${nixpkgs.legacyPackages.${hostSystem}.qt6.qtbase}"];
});
});
unixODBCDrivers = super.unixODBCDrivers // {
psql = super.unixODBCDrivers.psql.overrideAttrs (p: {
nativeBuildInputs = with nixpkgs.legacyPackages.${hostSystem}.pkgsCross.aarch64-multiplatform.pkgs; [unixODBC postgresql];
});
}; # workaround for odbc not building
})
];
system.stateVersion = "23.11";
});
};
}
And that’s it, now we can build and test the entire image with:
nix build .#images.pi --print-build-logs
nix build .#qemuTest
./result
Note that this will take quite a while to build, as everything is compiled from source. This will also create a flake.lock file pinning all the inputs to a specific version, so that subsequent runs will be reproducible.
Conclusion
Nix has been growing a lot in recent years, and not without reason. The Nix language allows to solve some otherwise very complicated packaging tasks in a very concise way. The fully hermetic and reproducible builds are a perfect fit for building embedded Linux images, and the vast collection of packages and configuration options allow to perform most tasks without ever having to leave the declarative world.
However, there are also some downsides when compared to the Yocto project. Due to the less frequent use of Nix on embedded, it is harder to find answers and support for embedded-related questions and you are quickly on your own, especially when using more obscure embedded platforms.
And while the Nix syntax in and of itself is very simple, it should not go unmentioned that there is a lot of complexity around the language constructs such as derivations and how everything interacts with each other. Thus, there is definitely a steep learning curve involved, though most of this comes with the territory and is also true for the Yocto project.
Hence overall, Nix is a suitable alternative for building embedded system images (keeping in mind that some extra work is involved for more obscure embedded platforms), and its purely functional language makes it possible to solve most tasks in a very elegant way.
If you like this article and want to read similar material, consider subscribing via our RSS feed.
Subscribe to KDAB TV for similar informative short video content.
KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.
Amazing stuff! I have been also building RPi4 images with Nix for the OpenMower project. I love the fact, that the same project codebase can provide the means to build an image with the current configuration, and then later one can incrementally deploy changes as the project Nix configuration evolves, with tools like deploy-rs.
Also, for testing locally, I have been using https://github.com/Mic92/nixos-shell to start a vm with the build sd card image. Very similar to ‘qemuTest’ described here.