Yocto

Adding Key-Based SSH Authentication to Yocto

I recently needed to add SSH keys to a Yocto image, but realised that I have no idea how that is done. When I tried to figure out how to do it, I realised that I didn’t even fully know what I actually wanted to achieve. This text is supposed to be a quick crash course on the different keys used in SSH servers, and how to generate and use them in Yocto.

Step zero of this SSH process is adding either ssh-server-dropbear or ssh-server-openssh to your IMAGE_FEATURES to select the SSH server you want to use. After the choice is made, we are good to go and ready to start logging in.

I feel like 90% of my blog posts either talk about how to get into a system, or how to prevent people from getting into systems.

Host Key

The host key is the key that is used to identify the SSH server. When you’re connecting for the first time to an SSH server, you’ll get a warning that the identity of the server isn’t known. Once you accept the server identity, it’ll get added to known_hosts file. If the server generates a new host key every time it boots up, you’ll get a new warning that the identity doesn’t match the expected one. If a server’s identity suddenly changes, it can also mean that something malicious is going on, and the connection is usually aborted.

The host key consists of both public and private portions, and both are stored on the server. The public key is given to the client so that it can check it during subsequent connection attempts, and the private key is used to sign messages that the client can then verify with the public key. Note that the host key should be unique for each SSH server, meaning that building the key into the firmware image should be reserved for development and testing environments. For real-world deployments, you want to generate a random host key on the device to persistent storage before shipping the device into the wild. Storing the generated public key is a good idea so that you can confirm the device’s identity later on.

Generating Keys

OpenSSH

First, we need to generate the keys. This step varies a bit, depending on the SSH server you’re using. To generate keys for OpenSSH, you can run the following:

ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key
ssh-keygen -t ed25519 -f ssh_host_ed25519_key

Having multiple key types offers security, flexibility, and both backwards compatibility and future-proofing. RSA is quite old, but using it with longer keys should still be an acceptable fallback for older clients without ed25519 support. ed25519 is a more modern algorithm that’s faster and hopefully harder to crack, but not fully supported in all SSH clients. ecdsa is also a modern algorithm that can be used with OpenSSH. If you know that you’re not going to require the fallback for the older clients you can skip the RSA keys.

Dropbear

Creating keys for Dropbear is quite similar, but we need to use dropbearkey command instead:

dropbearkey -t rsa -s 4096 -f dropbear_rsa_host_key
dropbearkey -t ed25519 -f dropbear_ed25519_host_key

It’s worth noting that the OpenSSH command creates two separate files, one for the public and another one for the private key, while Dropbear only creates one file that contains both. That’s because they have their own key format.

Yocto Recipe

In Poky there already is a recipe that installs pre-generated host keys to the system: ssh-pregen-hostkeys. We can use that recipe as a basis, copy the keys to a location where the recipe can find them and write a recipe like this:

SUMMARY = "Pre-generated host keys"

SRC_URI = "file://dropbear_rsa_host_key \
           file://dropbear_ed25519_host_key \
           file://ssh_host_rsa_key \
           file://ssh_host_rsa_key.pub \
           file://ssh_host_ed25519_key \
           file://ssh_host_ed25519_key.pub"

LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

INHIBIT_DEFAULT_DEPS = "1"

do_install () {
    install -d ${D}${sysconfdir}/dropbear
    install ${WORKDIR}/dropbear_*_host_key -m 0600 ${D}${sysconfdir}/dropbear/

    install -d ${D}${sysconfdir}/ssh
    install ${WORKDIR}/ssh_host_*_key* ${D}${sysconfdir}/ssh/
    chmod 0600 ${D}${sysconfdir}/ssh/*
    chmod 0644 ${D}${sysconfdir}/ssh/*.pub
}
It’s open source, I don’t have to explain anything.

Note that you should remove the parts that do not apply to your use-case because you most likely do not want to have both Dropbear and OpenSSH keys in the system.

What if I want to generate the keys during the build?

I’d recommend creating a pre-build script that runs the key-generation commands before starting the build. The SSH packages in Yocto don’t provide native versions, meaning that the ssh-keygen and dropbearkey won’t get compiled for the build architecture. This in turn means that you cannot call these commands in recipes without host contamination. Instead of trying to call the commands in a recipe, a shell script like this could work:

ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N ""
ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N ""
dropbearkey -t rsa -s 4096 -f dropbear_rsa_host_key
dropbearkey -t ed25519 -f dropbear_ed25519_host_key
cp ssh_host_*_key* <RECIPE_SRC_URI_DIR>
cp dropbear_*_host_key <RECIPE_SRC_URI_DIR>

Authentication Key

The second type of key we’re usually interested in is the authentication key. This can be used for SSH login without having to enter a password. It’s also usually recommended over the password login because passwords are a quite poor method of authentication. In key-based authentication, the public key gets stored on the server (the Yocto image in our case), and the client uses the private key it has to prove it should be allowed in.

Generating Keys

OpenSSH

I recommend using ssh-keygen, because the public keys generated with it are compatible with both Dropbear and OpenSSH. The command to create the keys is the same as before. I’ll stick to ed25519 for security:

ssh-keygen -t ed25519 -f ssh_auth_ed25519_key

Dropbear

Dropbear is a bit trickier, in the sense that it requires TWO commands instead of one. First, we create the key pair:

dropbearkey -t ed25519 -f dropbear_ed25519_auth_key

Then we’ll extract the public key from the key file:

dropbearkey -y -f dropbear_ed25519_auth_key > ./dropbear_ed25519_auth_key.pub

This may create a file with multiple lines that contain some unnecessary information. You should remove all the other lines except the one that begins with ssh-ed25519.

Yocto Recipe

On the Yocto side, we want to add a generated public key to the ~/.ssh/authorized_keys file for each of the users that should be allowed to log in. Adding the following snippet to an image recipe can be used to do that:

# Add example user
inherit extrausers

# Hashed password, unhashed value is "password"
PASSWD = "\$6\$vRcGS0O8nEeug1zJ\$YnRLFm/w1y/JtgGOQRTfm57c1.QVSZfbJEHzzLUAFmwcf6N72tDQ7xlsmhEF.3JdVL9iz75DVnmmtxVnNIFvp0"

EXTRA_USERS_PARAMS:append = "\
    useradd -u 1200 -d /home/serviceuser -p '${PASSWD}' -s /bin/sh serviceuser; \
"

# Location of the authentication keys
AUTH_KEYS_DIR ??= "${TOPDIR}/../auth-keys"

# Function to install the auth keys
configure_ssh_auth_key() {
    AUTH_KEYS=$(ls ${AUTH_KEYS_DIR}/*-auth-key.pub)
    for auth_key in ${AUTH_KEYS}; do
        user=$(basename "$auth_key" | cut -d'-' -f1)
        mkdir ${IMAGE_ROOTFS}/home/${user}/.ssh
        cat ${auth_key} >> ${IMAGE_ROOTFS}/home/${user}/.ssh/authorized_keys
    done
}

# Leading ; shouldn't be required, but seems to fail without it
ROOTFS_POSTPROCESS_COMMAND:append = ";configure_ssh_auth_key;"

This approach assumes that the keys have the following naming scheme: <USER_NAME>-<ALGORITHM>-auth-key.pub. It also assumes that the keys are located in a directory named auth-keys one level above the build directory. Also, it is assumed that the user has their home directory already created when the key is copied. That’s a lot of assumptions, but assuming you follow the key file naming scheme things should work just fine. You can configure the key folder to a custom location using AUTH_KEYS_DIR variable.

The example above adds a new user named serviceuser to the system for demonstration purposes. So, if the AUTH_KEYS_DIR points to a directory that contains serviceuser-ed25519-auth-key.pub, the key should get copied to the user’s authorized_keys and it should be possible to log in as them using SSH. If you want to read more about adding users to your Yocto images, you can check out my other blog text.

The first thought I had was using SRC_URI and installing the files from WORKDIR, but that doesn’t work because image.bbclass defines do_fetch as noexec. According to this StackOverflow answer, you can define those as executable with Python function, make bitbake fetch the certificates from SRC_URI, and install them in a custom task, but it’s a bit risky to edit the core classes like that, so I’d advise against it.

Also, once again, if you want to generate the keys during the build, I recommend using a pre-build script to create and store the keys.

Connecting

The next logical step is of course connecting. I’m going to assume you have a service that starts the SSH server on the image, such service should exist by default if you install an SSH server using IMAGE_FEATURES. Note that OpenSSH and Dropbear clients expect the private key in their specific format, so you can’t use OpenSSH private key on Dropbear client or vice versa. Unless you convert them, more on that a bit later.

One thing to note when attempting an SSH connection. If you get asked for a password even when you shouldn’t, there’s a high chance that the user you’re trying to log in as has been locked. This happens for example if you don’t define a password when creating a user. Check the system log for relevant messages to debug the issue if you encounter it.

OpenSSH

OpenSSH client is the classic, fan-favourite ssh. To connect to a remote server with your private key, simply run:

ssh -i <PRIVATE_KEY_FILE> <USERNAME>@<IP>

The magic is the -i flag. You may want to add the key to the ssh-agent so that you won’t have to use the flag every time:

# Start ssh-agent
eval "$(ssh-agent -s)"
# Add key
ssh-add <PRIVATE_KEY_FILE>

After that the ssh command “should just work”. I personally don’t like things that “should just work”, but it saves some typing if you’re constantly connecting to a server.

Dropbear

Honestly, I didn’t even know that Dropbear has its own SSH client before writing this text. It turns out it has, which makes sense considering that it has its own key format. The client is called dbclient, and as one can guess, it focuses on being lightweight. One thing that may not be obvious from its name is that it has nothing to do with databases though. To connect to a remote server with the Dropbear format key, use the following command:

dbclient -i <PRIVATE_KEY_FILE> <USERNAME>@<IP>

Converting Keys

If you end up in a situation where you have the wrong type of private keys for your SSH client, you can use dropbearconvert to convert a key from Dropbear format into OpenSSH format:

dropbearconvert dropbear openssh <SOURCE_KEY> <DESTINATION_KEY>

The conversion from OpenSSH to Dropbear isn’t much harder either:

dropbearconvert openssh dropbear <SOURCE_KEY> <DESTINATION_KEY>

It’s worth noting that dropbearconvert converts only private keys. That’s because both OpenSSH and Dropbear use a similar type of public keys.

That’s all for this topic. These instructions should help you build an image that has a constant host certificate, and that you can connect to easily and password-free. Both are quite basic yet important features for a firmware image. If you have questions or comments let me know in the comments.

Yocto Hardening: Measured Boot

You can find the other Yocto hardening posts from here!

Oh yes, it’s time for more of the security stuff. We are getting into the difficult things now. So far we have mostly been focusing on hardening the kernel and userspace separately, but this time we will zoom out a bit and take a look at securing the entire system. First, we are going to start hardening the boot process to prevent unwanted bootflows and loading undesired binaries.

I know that there is a philosophical and moral question of whether doing this is “right”, potentially locking the devices from the people using them. I’m not going to argue too much in either direction. I’d like everything to be open and easily hackable (in the good sense of the word), but because the real world is the way it is, keeping the embedded devices open doesn’t always make sense. Mostly because of the hacking (in the bad sense of the word). Anyway, I hope you use the power you will learn for good.

What Is Measured Boot

Simply put, the measured boot is a boot feature that hashes different boot components and then stores the hashes in immutable hash chains. The measured boot can perform hashing during different stages of the boot. The hashed items can for example be the kernel binary, devicetree, boot arguments, disk partitions, etc. These calculated hashes usually then get written to the platform configuration registers (PCR) in TPM.

These registers can only be extended, meaning that the existing value in the register and the new value get hashed together, and this combined hash then gets stored in the register. This creates a chain of hashes. This can then be called a “blockchain”, and it can be used to raise unlimited venture capital funding (or at least it was possible before AI became the hype train locomotive). The hash chain can also be used to detect unwanted changes in the chain because if one of the hashes in the chain changes, all the subsequent hashes after that will be changed as well.

To make actual use of these registers and their contents, attestation should be performed. In attestation, it is decided whether the system is in an acceptable state or not for performing some actions. For example, there could be a check that the PCRs contain certain expected values. Then, if the system is considered to be cool, for example filesystems may be decrypted, services could be started, or remote connections may be made.

I asked ChatGPT to create a meme about measured boot, and I’m not sure if this is genius or not.

It’s worth noting that measured boot doesn’t prevent loading or running unwanted binaries or configurations, it just makes a note if such a thing happened. Attestation on the other hand may prevent some unexpected things from happening if it is configured to do so. To prevent loading naughty stuff into your system, you may want to read about the secure boot. There’s even a summary of the differences coming sooner than you think!

Measured Boot vs. Secure Boot

Secure boot is a term that’s often confused with the measured boot, or the trusted boot, or the verified boot, so it is worth clarifying how these differ. This document sums it up nicely, but I’ll briefly summarize the differences in the next paragraphs. The original link contains some more pros and cons explained in an actually professional manner, so I recommend checking it out if you have the time.

In secure boot (also known as verified boot) each boot component checks the signatures of the next boot item (e.g U-boot checks Linux kernel, etc.), and if these don’t match with the keys stored in the device, the boot fails. If they match, the component doing the measurement transfers the control to the next component in boot chain. This fairly rigid system gives more control over the boot process, but signature verification and key storage aren’t trivial problems to solve. Also, updates to this kind of system are difficult.

Measured boot (also known as trusted boot) only measures the boot items and stores their hashes to TPM’s PCRs. It is then the responsibility of the attestation process to decide if the event log is acceptable or not for proceeding. This is more flexible and allows more options than a simple “boot or no boot”, but it is quite complex, and in theory, may allow booting some bad configurations if attestation isn’t sufficient. Performing the attestation itself isn’t that easy either. While local attestation is simpler to set up, it’s susceptible to local attacks, and with remote attestation, you have a server to set up and need a secure way of transferring the hashes to the attestation server.

So, despite having quite similar names, secure boot and measured boot are quite different things. Therefore a single system can have both systems in place. It’s actually a good idea, assuming the performance and complexity hits are acceptable. The performance hit is usually tolerable, as the actions need to be performed once per boot (as opposed to some encrypted filesystems where every filesystem operation takes a hit). Complexity on the other hand, well… In my experience, things won’t surely become easier after implementing these systems. All in all, everything requires more work and makes life miserable (but hopefully for the bad actors as well).

Considering the nature of the human meme culture, I’m not sure if ChatGPT was actually that bad.

Adding Measured Boot to Yocto

Now that we know what we’re trying to achieve, we can start working towards that goal. As you can guess, the exact steps vary a lot depending on your hardware and software. Therefore, it’s difficult to give the exact instructions on how to enable measured boot on your device. But, to give some useful advice, I’m going to utilize the virtual QEMU machine I’ve been working on a few earlier blog texts.

Yocto Emulation: Setting Up QEMU with U-Boot
Yocto Emulation: Setting Up QEMU with TPM

I’ve enabled measured boot also on Raspberry Pi 4 & LetsTrust TPM module combination using almost the same steps as outlined here, so the instructions should work on actual hardware as well. I’ll write a text about this a bit later…

Edit: The text for enabling the measured boot on Raspberry Pi 4 is now available, check it out here.

Configuring U-Boot

You want to start measuring the boot as early as possible to have a long hash chain. In an actual board, this could be something like the boot ROM (if boot ROM supports that) or SPL/FSBL. In our emulated example, the first piece doing the measurement is the U-boot bootloader. This is fairly late because we can only measure the kernel boot parameters, but we can’t change the boot ROM and don’t have SPL so it’s the best we can do.

Since we’re using U-Boot, according to the documentation enabling the boot measurement requires CONFIG_MEASURED_BOOT to be added into the U-Boot build configuration. This requires hashing and TPM2 support as well. You’ll most likely also want CONFIG_MEASURE_DEVICETREE to hash the device tree. It should be enabled automatically by default, at least in U-boot 2024.01 which I’m using it is, but you can add it just in case. The configuration fragment looks like this:

# Dependencies
CONFIG_HASH=y
CONFIG_TPM_V2=y
# The actual stuff
CONFIG_MEASURED_BOOT=y
CONFIG_MEASURE_DEVICETREE=y

Measured boot should be enabled by default in qemu_arm_defconfig used by our virtual machine, so no action is required to enable the measured boot for that device. If you’re using some other device you may need to add the configs. On the other hand, if you’re using something else than U-Boot as the bootloader, you have to consult the documentation of that bootloader. Or, in the worst case, write the boot measurement code yourself. U-Boot measures OS image, initial ramdisk image (if present), and bootargs variable. And the device tree, if the configuration option is enabled.

Editing the devicetree

Next, if you checked out the link to U-Boot documentation, it mentions that we also have to make some changes to our device tree. We need to define where the measurement event log is located in the memory. There are two ways of doing this: either by defining a memory-region of tcg_event_log type for the TPM node, or by adding linux,sml-base and linux,sml-size parameters to the TPM node. We’re going to go with the first option because the second option didn’t work with the QEMU for some reason (with the Raspberry Pi 4 it was the other way around, only linux,sml-base method worked. Go figure.)

For this, we first need to decompile our QEMU devicetree binary that has been dumped in the Yocto emulation blog texts (check those out if you haven’t already). The decompilation can be done with the following command:

dtc -I dtb -O dts -o qemu.dts qemu.dtb

Then, you can add memory-region = <&event_log>; to the TPM node in the source so that it looks like the following:

tpm_tis@0 {
    reg = <0x00 0x5000>;
    compatible = "tcg,tpm-tis-mmio";
    memory-region = <&event_log>;
};

After that, add the event log memory region to the root of the device tree. My node looks like this:

reserved-memory {
	#address-cells = <0x01>;
	#size-cells = <0x01>;
	ranges;
	event_log: tcg_event_log {
		#address-cells = <0x01>;
		#size-cells = <0x01>;
		no-map;
		reg = <0x45000000 0x6000>;
	};
};

Commit showing an example of this can be found from here. I had some trouble finding the correct location and addresses for the reserved-memory. In the end, I added reserved-memory node to the root of the device tree. The address is defined to be inside the device memory range, and that range is (usually) defined in the memory node at the root of the devicetree. The size of the event log comes from one of the U-Boot devicetree examples if I remember right.

Note that my reserved memory region is a bit poorly aligned to be in the middle of the memory, causing some segmentation. You can move it to some other address, just make sure that the address is not inside kernel code or kernel data sections. You can check these address ranges from a live system by reading /proc/iomem. For example, in my emulator device they look like this;

root@qemuarm-uboot:~# cat /proc/iomem
09000000-09000fff : pl011@9000000
09000000-09000fff : 9000000.pl011 pl011@9000000
09010000-09010fff : pl031@9010000
09010000-09010fff : rtc-pl031
09030000-09030fff : pl061@9030000
0a003c00-0a003dff : a003c00.virtio_mmio virtio_mmio@a003c00
0a003e00-0a003fff : a003e00.virtio_mmio virtio_mmio@a003e00
0c000000-0c004fff : c000000.tpm_tis tpm_tis@0
10000000-3efeffff : pcie@10000000
10000000-10003fff : 0000:00:01.0
10000000-10003fff : virtio-pci-modern
10004000-10007fff : 0000:00:02.0
10004000-10007fff : xhci-hcd
10008000-1000bfff : 0000:00:03.0
10008000-1000bfff : virtio-pci-modern
1000c000-1000cfff : 0000:00:01.0
1000d000-1000dfff : 0000:00:03.0
3f000000-3fffffff : PCI ECAM
40000000-4fffffff : System RAM
40008000-40ffffff : Kernel code
41200000-413c108f : Kernel data

After adding the reserved block of memory, you can check the reserved memory blocks in U-boot with bdinfo command:

=> bdinfo
boot_params = 0x00000000
DRAM bank   = 0x00000000
-> start    = 0x40000000
-> size     = 0x10000000
flashstart  = 0x00000000
flashsize   = 0x04000000
flashoffset = 0x000d7074
baudrate    = 115200 bps
relocaddr   = 0x4f722000
reloc off   = 0x4f722000
Build       = 32-bit
current eth = virtio-net#31
ethaddr     = 52:54:00:12:34:02
IP addr     = <NULL>
fdt_blob    = 0x4e6d9160
new_fdt     = 0x4e6d9160
fdt_size    = 0x00008d40
lmb_dump_all:
 memory.cnt = 0x1 / max = 0x10
 memory[0]      [0x40000000-0x4fffffff], 0x10000000 bytes flags: 0
 reserved.cnt = 0x2 / max = 0x10
 reserved[0]    [0x45000000-0x45005fff], 0x00006000 bytes flags: 4
 reserved[1]    [0x4d6d4000-0x4fffffff], 0x0292c000 bytes flags: 0
devicetree  = board
arch_number = 0x00000000
TLB addr    = 0x4fff0000
irq_sp      = 0x4e6d9150
sp start    = 0x4e6d9140
Early malloc usage: 2c0 / 2000

Once you’re done with the device tree, you can compile the source back into binary with the following command (this will print warnings, I guess the QEMU-generated device tree isn’t 100% perfect and my additions didn’t most likely help):

dtc -I dts -O dtb -o qemu.dtb qemu.dts

Booting the Device

That should be the hard part done. Since we have edited the devicetree and the modifications need to be present already in the U-Boot, QEMU can’t use the on-the-fly generated devicetree. Instead, we need to pass the self-compiled devicetree with the dtb option. The whole runqemu command looks like this:

BIOS=/<path>/<to>/u-boot.bin \
runqemu \
core-image-base nographic wic.qcow2 \
qemuparams="-chardev \
socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
-tpmdev emulator,id=tpm0,chardev=chrtpm \
-device tpm-tis-device,tpmdev=tpm0 \
-dtb /<path>/<to>/qemu.dtb"

Note that you need to source the Yocto build environment to have access to runqemu command. Also, remember to set up the swtpm TPM as instructed in the Yocto Emulation texts before booting up the system. You can use the same boot script that was used in the QEMU emulation texts.

Now, when the QEMU device boots, U-Boot will perform the measurements, store them into TPM PCRs, and the kernel is aware of this fabled measurement log. To read the event log in the Linux-land, you want to make sure that the securityfs is mounted. If not, you can mount it manually with:

mount -t securityfs securityfs /sys/kernel/security

If you face issues, make sure CONFIG_SECURITYFS is present in the kernel configuration. Once that is done, you should be able to read the event log with the following command:

tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements

This outputs the event log and the contents of the PCRs. You can also use tpm2_pcrread command to directly read the current values in the PCR registers. If you turn off the emulator and re-launch it, the hashes should stay the same. And if you make a small change to for example the U-Boot bootargs variable and boot the device, register 1 should have a different value.

The Limitations

Then, the bad news. Rebooting does not quite work as expected. If you reboot the device (as opposed to shutting QEMU down and re-starting it), the PCR values output by tpm2_pcrread change on subsequent boots even though they should always be the same. The binary_bios_measurements on the other hand stays the same after reboot even if the bootargs changes, indicating that it doesn’t get properly updated either.

From what I’ve understood, this happens because PCRs are supposed to be volatile, but the emulated TPM doesn’t really “reset” the “volatile” memory during reboot because the emulator doesn’t get powered off. With the actual hardware Raspberry Pi 4 TPM module this isn’t an issue, and tpm2_pcrread results are consistent between reboots and binary_bios_measurements gets updated on every boot as expected. It took me almost 6 months of banging my head on this virtual wall to figure out that this was most likely an emulation issue. Oh well.

Closing Words

Now we have (mostly) enabled measured boot to our example machine. Magnificient! There isn’t any attestation, though, so the measurement isn’t all that useful yet. The measurements could also be extended to the Linux side with IMA. These things will be addressed in future editions of Yocto hardening, so stay tuned!

While waiting for that, you can read the other Yocto hardening posts here!

Yocto Emulation: Setting Up QEMU with TPM

As promised, it’s time for the QEMU follow-up. Last time we got Yocto’s runqemu command to launch u-boot, boot up a kernel, and mount a virtual drive with multiple partitions. Quite a lot of stuff. This time we are “just” going to add a TPM device to the virtual machine. As before, you can find the example meta-layer from Github. It contains the example snippets presented in this blog text, and should be ready to use.

Why is this virtualized TPM worth the effort? Well, if you have ever been in a painful situation where you’re working with TPMs and you’re writing some scripts or programs using them, you know that the development is not as straightforward as one would hope. The flows tend to be confusing, frustrating, and difficult. Using a virtual environment that’s easy to reset and that’s quite close to the actual hardware is a nice aid for developing and testing these types of applications.

In a nutshell, the idea is to run swtpm TPM emulator on the host machine, and then launch QEMU Arm device emulator that talks with the swtpm process. QEMU has an option for a TPM device that can be passed through to the guest device, so the process is fairly easy. With these systems in place, we can have a virtual TPM chip inside the virtual machine. *insert yo dawg meme here*

TPM Emulation With swtpm

Because I’m terrible at explaining things understandably, I’m going to ask my co-author ChatGPT to summarise in one paragraph what a TPM is:

Trusted Platform Module (TPM) is a hardware-based security feature integrated into computer systems to provide a secure foundation for various cryptographic functions and protect sensitive data. TPM securely stores cryptographic keys, certificates, and passwords, ensuring they remain inaccessible to unauthorized entities. It enables secure boot processes, integrity measurement, and secure storage of credentials, enhancing the overall security of computing devices by thwarting attacks such as tampering, unauthorized access, and data breaches.

I’m not sure if this is easier to understand than my ramblings, but I guess it makes the point clear. It’s a hardware chip that can be used to store and generate secrets. One extra thing worth knowing is that there are two notable versions of the TPM specification: 1.2 and 2.0. When I’m talking about TPM in this blog text, I mean TPM 2.0.

Since we’re using emulated hardware, we don’t have the “hardware” part in the system. Well, QEMU has a passthrough option for hardware TPMs, but for development purposes it’s easier to have an emulated TPM, something that swtpm can be used for. Installing swtpm is straightforward, as it can be found in most of the package repositories. For example, on Ubuntu, you can just run:

sudo apt install swtpm

Building swtpm is also an option. It has quite a few dependencies though, so you may want to consider just fetching the packages. Sometimes taking the easy route is allowed.

Whichever option you choose, once you’re ready you can run the following commands to set up the swtpm and launch the swtpm process:

mkdir /tmp/mytpm1
swtpm_setup --tpmstate /tmp/mytpm1 \
  --create-ek-cert \
  --create-platform-cert \
  --create-spk \
  --tpm2 \
  --overwrite
swtpm socket --tpmstate dir=/tmp/mytpm1 \
  --ctrl type=unixio,path=/tmp/mytpm1/swtpm-sock \
  --tpm2 \
  --log level=20

Once the process launches, it opens a Unix domain socket that listens to the incoming connections. It’s worth knowing that the process gets launched as a foreground job, and once a connected process exits swtpm exits as well. Next, we’re going to make QEMU talk with the swtpm daemon.

QEMU TPM

Fortunately, making QEMU communicate with TPM isn’t anything groundbreaking. There’s a whole page of documentation dedicated to this topic, so we’re just going to follow it. For Arm devices, we want to pass the following additional parameters to QEMU:

-chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
-tpmdev emulator,id=tpm0,chardev=chrtpm \
-device tpm-spapr,tpmdev=tpm0 \

These parameters should result in the QEMU connecting to the swtpm, and using the emulated software TPM as a TPM in the emulated machine. Simple as.

One thing worth noting though. Since we’re adding a new device to the virtual machine, the device tree changes as well. Therefore, we need to dump the device tree again. This was discussed more in-depth in the first part of this emulation exercise, so I recommend reading that. In summary, you can dump the device tree with the following runqemu command:

BIOS=tmp/deploy/images/qemuarm-uboot/u-boot.bin \
runqemu \
core-image-base nographic wic.qcow2 \
qemuparams="-chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
-tpmdev emulator,id=tpm0,chardev=chrtpm \
-device tpm-tis-device,tpmdev=tpm0 \
-machine dumpdtb=qemu.dtb"

Then, you need to move the dumped binary to a location where it can get installed to the boot partition as a part of the Yocto build. This was also discussed in the first blog text.

TPM2.0 Software Stack

Configuring Yocto

Now that we have the virtualized hardware in order, it’s time to get the software part sorted out. Yocto has a meta-layer that contains security features and programs. That layer is aptly named meta-security. To add the TPM-related stuff into the firmware image, add sub-layer meta-tpm to bblayers.conf. meta-tpm has dependencies to meta-openembedded sub-layers meta-oe and meta-python, so add those as well.

Once the layers are added, we still need to configure the build a bit. The following should be added to your distro.conf, or if you don’t have one, local.conf should suffice:

DISTRO_FEATURES:append = " tpm"

Configuring Linux Kernel

Next, to get the TPM device working together with Linux, we need to configure the kernel. First of all, the TPM feature needs to be enabled, and then the driver for our emulated chip needs to be added. If you were curious enough to decompile the QEMU device tree binary, you maybe noticed that the emulated TPM device is compatible with tcg,tpm-tis-mmio. Therefore, we don’t need a specific driver, the generic tpm-tis driver should do. The following two config lines both enable TPM and add the driver:

CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y

If you’re wondering what TCG means, it stands for Trusted Computing Group, the organization that has developed the TPM standard. TIS on the other hand stands for TPM Interface Specification. There are a lot of TLAs here that begin with the letter T, and we haven’t even seen all of them yet.

Well, here’s the yo dawg meme.

Configuring U-Boot

Configuring TPM support for U-Boot is quite simple. Actually, the U-Boot I built worked straight away with the defconfig. However, if you have issues with TPM in U-Boot, you should ensure that you have the following configuration items enabled:

# Enable TPM 2.0
CONFIG_TPM=y
CONFIG_TPM_V2=y
# Add MMIO interface for device
CONFIG_TPM2_MMIO=y
# Add TPM command
CONFIG_CMD_TPM=y
# This should be enabled automatically if
# CMD_TPM and TPM_V2 are enabled
CONFIG_CMD_TPM_V2=y

Installing tpm2-tools

In theory, we now should have completed the original goal of booting a Yocto image on an emulator that has a virtual TPM. However, there’s still nothing that uses the TPM. To add plenty of packages, tpm2-tools among them, we can add the following to the image configuration:

IMAGE_INSTALL:append = " \
    packagegroup-security-tpm2 \
    libtss2-tcti-device \
"

packagegroup-security-tpm2 contains the following packages:

tpm2-tools
trousers
tpm2-tss
libtss2
tpm2-abrmd
tpm2-pkcs11

For our testing purposes, we are mostly interested in tpm2-tools and tpm2-tss, and libtss2 that tpm2-tools requires. TSS here stands for TPM2 Software Stack. trousers is an older implementation of the stack, tpm2-abrmd (=access broker & resource manager daemon) didn’t work for me (and AFAIK using a kernel-managed device is preferred anyway), and PKCS#11 isn’t required for our simple example. libtss2-tcti-device is required to enable a TCTI (TPM Command Transmission Interface) for communication with Linux kernel TPM device files. These are the last acronyms, so now you can let out a sigh of relief.

Running QEMU

Now you can rebuild the image to compile a suitable kernel and user-space tools. Once the build finishes, you can use the following command to launch QEMU (ensure that swtpm is running):

BIOS=tmp/deploy/images/qemuarm-uboot/u-boot.bin \
runqemu \
core-image-base nographic wic.qcow2 \
qemuparams="-chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
-tpmdev emulator,id=tpm0,chardev=chrtpm \
-device tpm-tis-device,tpmdev=tpm0"

Then, stop the booting process to drop into the U-Boot terminal. We wrote the boot script in the previous blog, but now we can add tpm2 commands to initialize and self-test the TPM. The first three commands of this complete boot script set-up and self-test the TPM:

# Initalize TPM
tpm2 init
tpm2 startup TPM2_SU_CLEAR
tpm2 self_test full
# Set boot arguments for the kernel
setenv bootargs root=/dev/vda2 console=ttyAMA0
# Load kernel image
setenv loadaddr 0x40200000
fatload virtio 0:1 ${loadaddr} zImage
# Load device tree binary
setenv loadaddr_dtb 0x49000000
fatload virtio 0:1 ${loadaddr_dtb} qemu.dtb
# Boot the kernel
bootz ${loadaddr} - ${loadaddr_dtb}

Now, once the machine boots up, you should see /dev/tpm0 and /dev/tpmrm0 devices present in the system. tpm0 is a direct access device, and tpmrm0 is a device using the kernel’s resource manager. The latter of these is the alternative to tpm2-abrmd, and we’re going to be using it for a demo.

TPM Demo

Before we proceed, I warn you that my knowledge of actual TPM usage is a bit shallow. So, the example presented here may not necessarily follow the best practices, but it should perform a simple task that should prove that the QEMU TPM works. We are going to create a key, store it in the TPM, sign a file and verify the signature. When you’ve got the device booted with the swtpm running in the background, you can start trying out these commands:

# Set environment variable for selecting TPM device
# instead of the abrmd.
export TPM2TOOLS_TCTI="device:/dev/tpmrm0"
# Create contexts
tpm2_createprimary -C e -c primary.ctx
tpm2_create -G rsa -u rsa.pub -r rsa.priv -C primary.ctx
# Load and store contexts
tpm2_load -C primary.ctx -u rsa.pub -r rsa.priv -c rsa.ctx
tpm2_evictcontrol -C o -c primary.ctx 0x81010002
tpm2_evictcontrol -C o -c rsa.ctx 0x81010003
# Remove generated files and create message
rm rsa.pub rsa.priv rsa.ctx primary.ctx
echo "my message" > message.dat
# Sign and verify signature with TPM handles
tpm2_sign -c 0x81010003 -g sha256 -o sig.rssa message.dat
tpm2_verifysignature -c 0x81010003 -g sha256 -s sig.rssa -m message.dat

If life goes your way, all the commands should succeed without issues and you can create and verify the signature using the handles in the TPM. Usually, things aren’t that simple. If you see errors related to abrmd, you may need to define the TCTI as the tpmrm0 device. The TPM2TOOLS_TCTI environment variable should do that. However, if that doesn’t work you can try adding -T "device:/dev/tpmrm0" to the tpm2_* commands, so for example the first command looks like this:

tpm2_createprimary -C e -c primary.ctx -T "device:/dev/tpmrm0"

When running the tpm2_* commands, you should see swtpm printing out plenty of information. This information includes requests and responses received and sent by the daemon. To make some sense of these hexadecimal dumps, you can use tpmstream tool.

That should wrap up my texts about QEMU, Yocto and TPM. Hopefully, these will help you set up a QEMU device that has a TPM in it. I also hope that in the long run this setup helps you to develop and debug secure Linux systems that utilize TPM properly. Perhaps I’ll write more about TPMs in the future, it was quite difficult to find understandable sources and examples utilizing its features. But maybe first I’d need to understand the TPMs a bit better myself.

Yocto Emulation: Setting Up QEMU with U-Boot

I’ve been thinking about the next topic for the Yocto Hardening blog series, and it’s starting to feel like the easy topics are running out. Adding and using non-root users, basic stuff. Running a tool to check kernel configuration, should be simple enough. Firewalls, even your grandma knows what a firewall is.

So, I started to look into things like encryption and secure boot, but turns out they are quite complicated topics. Also, they more or less require a TPM (Trusted Platform Module), and I don’t have a board with such a chip. And even if I did, it’d be more useful to have flexible hardware for future experiments. And for writing blog texts that can be easily followed along it’d be beneficial if that hardware would be easily available for everyone.

Hardware emulation sounds like a solution to all of these problems. Yocto provides a script for using QEMU (Quick EMUlator) in the form of runqemu wrapper. However, by default that script seems to just boot up the kernel and root file system using whatever method QEMU considers the best (depending on the architecture). Also, runqemu passes just the root file system partition as a single drive to the emulator. Emulating a device with a bootloader and a partitioned disk image is a bit tricky thing to do, but that’s exactly what we’re going to do in this text. In the next part we’re going to throw a TPM into the mix, but for now, let’s focus on the basics.

Configuring the Yocto Build

Before we start, I’ll say that you can find a meta-layer containing the code presented here from GitHub. So if you don’t want to copy-paste everything, you can clone the repo. It’ll contain some more features in the future but the basic functionality created in this blog text should be present in the commit cf4372a.

Machine Configuration

To start, we’re going to define some variables related to the image being built. To do that, we will define our machine configuration that is an extension of a qemuarm configuration:

require conf/machine/qemuarm.conf

# Use the same overrides as qemuarm machine
MACHINEOVERRIDES:append = ":qemuarm"

# Set the required entrypoint and loadaddress
# These are usually 00008000 for Arm machines
UBOOT_ENTRYPOINT =       "0x00008000"
UBOOT_LOADADDRESS =      "0x00008000"

# Set the imagetype
KERNEL_IMAGETYPE = "zImage"
# Set kernel loaddaddr, should match the one u-boot uses
KERNEL_EXTRA_ARGS += "LOADADDR=${UBOOT_ENTRYPOINT}"

# Add wic.qcow2 image that can be used by QEMU for drive image
IMAGE_FSTYPES:append = " wic.qcow2"

# Add wks file for image partition definition
WKS_FILE = "qemu-test.wks"

# List artifacts in deploy dir that we want to be in boot partition
IMAGE_BOOT_FILES = "zImage qemu.dtb"

# Ensure things get deployed before wic builder tries to access them
do_image_wic[depends] += " \
    u-boot:do_deploy \
    qemu-devicetree:do_deploy \
"

# Configure the rootfs drive options. Biggest difference to original is
# format=qcow2, in original the default format is raw
QB_ROOTFS_OPT = "-drive id=disk0,file=@ROOTFS@,if=none,format=qcow2 -device virtio-blk-device,drive=disk0"

Drive Image Configuration with WIC

Once that is done, we can write the wks file that’ll guide the process that creates the wic image. wic image can be considered as a drive image with partitions and such. Writing wks files is worth a blog text of its own, but here’s the wks file I’ve been using that creates a drive containing two partitions:

part /boot --source bootimg-partition --ondisk vda --fstype=vfat --label boot --active --align 1024
part / --source rootfs --use-uuid --ondisk vda --fstype=ext4 --label platform --align 1024

The first partition is a FAT boot partition where we will store the kernel and device tree so that the bootloader can load them. Second is the ext4 root file system, containing all the lovely binaries Yocto spends a long time building.

Device Tree

We have defined the machine and the image. The only thing that is still missing is the device tree. The device tree defines the hardware of the machine in a tree-like format and should be passed to the kernel by the bootloader. QEMU generates a device tree on-the-fly, based on the parameters passed to it. The generated device tree binary can be dumped by adding -machine dumpdtb=qemu.dtb to the QEMU command. With runqemu, you can use the following command to pass the parameter:

runqemu core-image-base nographic wic.qcow2 qemuparams="-machine dumpdtb=qemu.dtb"

However, here we have a circular dependency. The image depends on the qemu-devicetree recipe to deploy the qemu.dtb, but runqemu cannot be run without an image, so the image needs to built to dump the device tree. To sort this out, remove the qemu-devicetree dependency from the machine configuration, build once, and dump the device tree. Then re-enable the dependency.

After this, you can give the device tree binary to a recipe and deploy it from there. Or you could maybe decompile it to a source file, and then re-compile the source as a part of kernel build to do things “correctly”. I was lazy and just wrote a recipe that deploys the binary:

SUMMARY = "QEMU device tree binary"
DESCRIPTION = "Recipe deploying the generated QEMU device tree binary blob"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

SRC_URI = "file://qemu.dtb"

inherit deploy

do_deploy() {
    install -d ${DEPLOYDIR}
    install -m 0664 ${WORKDIR}/*.dtb ${DEPLOYDIR}
}

addtask do_deploy after do_compile before do_build

Once that is done, you should be able to build the image. I recommend checking out the meta-layer repo if you found this explanation confusing. I’m using core-image-base as the image recipe, but you should be able to use pretty much any image, assuming it doesn’t overwrite variables in machine configuration.

Setting up QEMU

Running runqemu

We should now have an image that contains everything needed to emulate a boot process: it has a bootloader, a kernel and a file system. We just need to get the runqemu to play along nicely. To start booting from the bootloader, we want to pass the bootloader as a BIOS for QEMU. Also, we need to load the wic.qcow2 file instead of the rootfs.ext4 as the drive source so that we have the boot partition present for the bootloader. All this can be achieved with the following command:

BIOS=tmp/deploy/images/qemuarm-uboot/u-boot.bin runqemu core-image-base nographic wic.qcow2

nographic isn’t mandatory if you’re running in an environment that has visual display capabilities. To this day I still don’t quite understand how the runqemu argument parsing works, even though I tried going through the script source. It simultaneously feels like it’s very picky about the order of the parameters, and that it doesn’t matter at all what you pass and at what position. But at least the command above works.

Booting the Kernel

If things go well, you should be greeted with the u-boot log. If you’re quick, spam any key to stop the boot, and if you’re not, spam Ctrl-C to stop bootloader’s desperate efforts of TFTP booting. I’m not 100% sure why the default boot script fails to load the kernel, I think the boot script doesn’t like the boot partition being a FAT partition on a virtio interface. To be honest, I would have been more surprised if the stock script would have worked out of the box. However, what works is the script below:

# Set boot arguments for the kernel
setenv bootargs root=/dev/vda2 console=ttyAMA0
# Load kernel image
setenv loadaddr 0x40200000
fatload virtio 0:1 ${loadaddr} zImage
# Load device tree binary
setenv loadaddr_dtb 0x49000000
fatload virtio 0:1 ${loadaddr_dtb} qemu.dtb
# Boot the kernel
bootz ${loadaddr} - ${loadaddr_dtb}

This script does exactly what the comments say: it loads the two artefacts from the boot partition and boots the board. We don’t have an init RAM disk, so we skip the second parameter of bootz. I also tried to create a FIT (firmware image tree) image with uImage to avoid having multiple boot files in the boot partition. Unfortunately, that didn’t quite work out. Loading the uImage got the device stuck with a nefarious "Starting kernel ..." message for some reason.

Back to the task at hand: if things went as they should have, the kernel should boot with the bootz, and eventually you should be dropped to the kernel login prompt. You can run mount command to see that the boot partition gets mounted, and cat /proc/cmdline to check that vda2 indeed was the root device that was used.

Closing Words And What’s Next

Congratulations! You got the first part of the QEMU set-up done. The second half with the TPM setup will follow soon. The example presented here could be improved in a few ways, like by adding a custom boot script for u-boot so that the user doesn’t have to input the script manually to boot the device, and by getting that darn FIT image working. But those will be classified as “future work” for now. Until next time!

The second part where the TPM gets enabled is out now!

Yocto Hardening: Kernel and GCC Configuration

Find all of the Yocto hardening texts from here!

Would you like to make your Yocto image a tiny bit harder to hack ‘n’ crack? Of course you would. This time we’re going to be doing two things to improve its security: hardening the Linux kernel, and setting the hardening flags for GCC. The motivation for these is quite obvious. Kernel is the privileged core of the system, so it better be as hardened as possible. GCC compilation flags on the other hand affect almost every C and C++ binary and library that gets compiled into the system. As you may know, over the years we’ve gotten quite a few of them, so it’s a good idea to use any help the compiler can provide with hardening them.

On the other hand, who wouldn’t like to live a bit dangerously?

Kernel Configuration Hardening

Linux kernel is the heart of the operating system and environment. As one can guess, configuring the kernel incorrectly or enabling everything in the kernel “just in case” will in the best situation lead to suboptimal performance and/or size, and in the worst case, it’ll provide unnecessary attack surfaces. However, optimizing the configuration manually for size, speed, or safety is a massive undertaking. According to Linux from Scratch, there are almost 12,000 configuration switches in the kernel, so going through all of them isn’t really an option.

Fortunately, there are automatic kernel configuration checkers that can help guide the way. Alexander Popov’s Kernel Hardening Checker is one such tool, focusing on the safety of the kernel. It combines a few different security recommendations into one checker. The project’s README contains the list of recommendations it uses as the guideline for a safe configuration. The README also contains plenty of other useful information, like how to run the checker. Who would have guessed! For the sake of example, let’s go through the usage here as well.

Obtaining and Analyzing Kernel Hardening Information

The kernel-hardening-checker doesn’t actually only check the kernel configuration that defines the build time hardening, but it also checks the command line and sysctl parameters for boot-time and runtime hardening as well. Here’s how you can obtain the info for each of the checks:

  • Kernel configuration: in Yocto, you can usually find this from ${STAGING_KERNEL_BUILDDIR}/.config, e.g. <build>/tmp/work-shared/<machine>/kernel-build-artifacts/.config
  • Command line parameters: run cat /proc/cmdline on the system to print the command line parameters
  • Sysctl parameters: run sysctl -a on the system to print the sysctl information

Once you’ve collected all the information you want to check, you can install and run the tool in a Python virtual environment like this:

python3 -m venv venv
source venv/bin/activate
pip install git+https://github.com/a13xp0p0v/kernel-hardening-checker
kernel-hardening-checker -c <path-to-config> -l <path-to-cmdline> -s <path-to-sysctl>

Note that you don’t have to perform all the checks if you don’t want to. The command will print out the report, most likely recommending plenty of fixes. Green text is better than red text. Note that not all of the recommendations necessarily apply to your system. However, at least disabling the unused features is usually a good idea because it reduces the attack surface and (possibly) optimizes the kernel.

To generate the config fragment that contains the recommended configuration, you can use the -g flag without the input files. As the README states, the configuration flags may have performance and/or size impacts on the kernel. This is listed as recommended reading about the performance impact.

GCC Hardening

Whether you like it or not, GCC is the default compiler in Yocto builds. Well, there exists meta-clang for building with clang, and as far as I know, the support is already in quite good shape, but that’s beside the point. Yocto has had hardening flags for GCC compilation for quite some time. To check these flags, you can run the following command:

bitbake <image-name> -e | grep ^SECURITY_CFLAGS=

How the security flags get applied to the actual build flags may vary between Yocto versions. In Kirkstone, SECURITY_CFLAGS gets added to TARGET_CC_ARCH variable, which gets set to HOST_CC_ARCH, which finally gets added to CC command. HOST_CC_ARCH gets also added to CXX and CPP commands, so SECURITY_CFLAGS apply also to C++ programs. bitbake -e is your friend when trying to figure out what gets set and where.

I don’t think any other meme can capture this feeling of madness

So, in addition to checking the SECURITY_CFLAGS, you most likely want to check the CC variable as well to see that the flags actually get added to the command that gets run:

# Note that the CC variable gets exported so grep is slightly different
bitbake <image-name> -e |grep "^export CC="

The flags are defined in security_flags.inc file in Poky (link goes to Kirkstone version of the file). It also shows how to make package-specific exceptions with pn-<package-name> override. The PIE (position-independent executables) flags are perhaps worth mentioning as they’re a bit special. The compiler is built to create position-independent executables by default (seen in GCCPIE variable), so PIE flags are empty and not part of the SECURITY_CFLAGS. Only if PIE flags are not wanted, they are explicitly disabled.

Extra Flags for GCC

Are the flags defined in security_flags.inc any good? Yes, they are, but they can also be expanded a bit. GCC will most likely get in early 2024 new -fhardened flag that sets some options not present in Yocto’s security flags:

-D_FORTIFY_SOURCE=3 (or =2 for older glibcs) 
-D_GLIBCXX_ASSERTIONS 
-ftrivial-auto-var-init=pattern 
-fPIE -pie -Wl,-z,relro,-z,now
-fstack-protector-strong
-fstack-clash-protection
-fcf-protection=full (x86 GNU/Linux only)

Lines 2, 3, and 6 are not present in the Yocto flags. Those could be added using a SECURITY_CFLAGS:append in a suitable place if so desired. I had some trouble with the trivial-auto-var-init flag though, seems like it is introduced in GCC version 12.1 while Yocto Kirkstone is still using 11 series. Most of the aforementioned flags are explained quite well in this Red Hat article. Considering there’s plenty of overlap with the SECURITY_CFLAGS and -fhardened, it may be that in future versions of Poky the security flags will contain just -fhardened (assuming that the flag actually gets implemented).

All in all, assuming you have a fairly modern version of Yocto, this GCC hardening chapter consists mostly of just checking that you have the SECURITY_CFLAGS present in CC variable and adding a few flags. Note once again that using the hardening flags has its own performance hit, so if you are writing something really time- or resource-critical you need to find a suitable balance with the hardening and optimization levels.

In Closing

While this was quite a short text, and the GCC hardening chapter mostly consisted of two grep commands and silly memes, the Linux kernel hardening work is something that actually takes a long time to complete and verify. At least my simple check for core-image-minimal with systemd enabled resulted in 136 recommendation fails and 110 passes. Fixing it would most likely take quite a bit longer than writing this text. Perhaps not all of the issues need to be fixed but deciding what’s an actual issue and what isn’t takes its own time as well. So good luck with that, and until next time!

As a reminder, if you liked this text and/or found it useful, you can find the whole Yocto hardening series from here.


The Movember blog series continues with this text! As usual, after reading this text I ask you to do something good. Good is a bit subjective, so you most likely know what’s something good that you can do. I’m going to eat a hamburger, change ventilation filters, and donate a bit to charity.

Yocto Hardening: Firewalls, Part 2: firewalld

Find all of the Yocto hardening texts from here!

People often ask me two things. The first question is “Why did you choose to write this firewall text in two parts?”. The answer to that is I actually started writing this over a year ago, the scope of the text swelled like crazy, and in the end, to get something published I chose to write the thing in two parts to have a better focus. The second question that I get asked is “Do you often have imaginary discussions with imaginary people in your head?”. To that, the answer is yes.

Comic about imaginary conversations
I prepare for daily stand-up by mentally going through conversations where I get deservedly fired because I didn’t close three tickets the previous day.

But without further ado, let’s continue where we left off in part 1. As promised, let’s take a look at another way of setting up and configuring a firewall: firewalld.

One Step Forward, Two Steps Back: Configuring Kernel (Again)

To get the firewalld running we need a few more kernel configuration items enabled. How many? Well, I spent a few days trying to figure out the minimal possible configuration to run some simple firewalld commands, all without success. firewalld is not too helpful with its error messages, as it basically says “something is missing” without really specifying what that special something is.

Meme about Linux kernel modules
I’ve always liked Lionel’s ‘stash, but I feel like he’s looking extra fine in this pic.

After banging my head on a wall for a few days I attempted to enable every single Netfilter and nftables configuration item (because firewalld uses nftables), but no success. After some further searching, I found this blog post that contained a kernel configuration that had worked for the author of the blog. I combined that with my desperate efforts, and the resulting config actually worked! In the end, my full configuration fragment looked like this:

As you can see, my shotgun approach to configuration wasn’t 100% accurate because there were few NF and NFT configuration items I missed. Is everything here absolutely necessary? Most likely not. There actually are some warnings about unused config options for the Linux kernel version 5.15, but I’m afraid to touch the configuration at this point.

I spent some more time optimizing this and trying to make the “minimal config”, just to realize that I’m actually minimizing something that can’t be really minimized. The “minimal config” really depends on what your actual firewall configuration contains. After this realization, I decided that it’s better to have too much than too little, and found an inner peace. You can think that the fragment provides a starting point from which you can start to remove useless-sounding options if you feel like it.

While we’re still talking about the build side of things, remember to add the kernel-modules to your IMAGE_INSTALL as instructed in part 1. Or if you chose to go through the RRECOMMENDS route, remember to add all 36 modules to the dependencies.

Meme about saving disk space by removing kernel modules
To be fair, I’ve been in a situation where a few KBs of root file system content have made the difference between a success and a catastrophic failure.

The Hard Firewall – firewalld

Now that I’ve spent three chapters basically apologizing for my Linux config, it’s time to get to the actual firewall stuff. The “hard” way of setting up a firewall consists of setting up firewalld. This is in my opinion suitable if you have multiple interfaces with changing configurations, possibly edited by a human user, meaning that things change (and usually in a more complex direction).

firewalld is a firewall management tool designed for Linux distributions, providing a dynamic interface for network security (another sentence from ChatGPT, thank you AI overlords). Or to put it into more understandable words, it’s a front-end for nftables providing a D-Bus interface for configuring the firewall. The nice thing about firewalld is that it operates in zones & services, allowing different network interfaces to operate in different network zones with different available services, all of which can be edited run-time using D-Bus, creating mind-boggling, evolving, and Lovecraftian configurations. This is the nice thing, so you can imagine what the not-so-nice things are.

Now that we are somewhat aware of what we’re getting into, we can add firewalld to IMAGE_INSTALL:

IMAGE_INSTALL:append = " firewalld"

One thing worth noting is that firewalld is not part of the Poky repository, but it comes as a part of meta-openembedded. Most likely this is not a problem, because every project I’ve worked on has meta-openembedded, but it’s worth knowing nevertheless.

Now that everything is in place we can start hammering the firewalld to the desired shape. To edit the configuration we can use the firewall-cmd shipped with firewalld package. This firewall-cmd (among some other firewall-* tools) uses the D-Bus interface to command the firewalld, which in turn commands nftables, which finally sets the Netfilter in the kernel. This picture illustrates it all nicely:

Graph showing firewalld internal structure
The picture originates from here:
https://firewalld.org/2018/07/nftables-backend

So, about the actual configuration process: each network interface can be assigned a zone, and the zones can enable a certain set of services. Here’s a short command reference for how to add a custom zone, add a custom service, add the new service to the new zone, and set the new zone to a network interface:

# List available zones
firewall-cmd --get-zones
# These zones are also listed in /usr/lib/firewalld/zones/
# and /etc/firewalld/zones/

# Get the default zone with the following command
# (This is usually public zone)
firewall-cmd --get-default-zone

# This'll return "no zone" if nothing is set,
# meaning that the default zone will be used
firewall-cmd --get-zone-of-interface=eth0

# Create a custom zone blocking everything
firewall-cmd --permanent --new-zone=test_zone
firewall-cmd --permanent --zone=test_zone --set-target=DROP
# If permanent option is not used, changes will be lost on reload.
# This applies to pretty much all of the commands

# Reload the firewalld to make the new zone visible
firewall-cmd --reload

#List services, and add a custom service
firewall-cmd --list-services
firewall-cmd --permanent --new-service=my-custom-service
firewall-cmd --permanent --service=my-custom-service --add-port=22/tcp
firewall-cmd --reload

# Add the service to the zone
firewall-cmd --permanent --zone=test_zone --add-service=my-custom-service

# Set zone
firewall-cmd --zone=home --change-interface=eth0
# Add --permanent flag to make the change, well, permanent

You can (and should) check between the commands how the ports appear to the outside world using nmap to get a bit better grasp on how the different commands actually work. The changes are not always as immediate and intuitive as one would hope. Good rule of thumb is that if the settings survive --reload, they’re set up properly.

How does this all work with the file-based configuration and Yocto? You could create the zone and service files for your system using the firewall-cmd, install them during build time, and change the default zone in the configuration file /etc/firewalld/firewalld.conf to have the rules active by default. This is similar to nftables approach, but is a bit easier to customize for example from a graphical user interface or scripts, and doesn’t require custom start-up scripts. An example of adding some zones and services, and a patch setting the default zone, can be found in the meta-firewall-examples repo created for this blog text.

Comparison and Closing Words

So, now we have two options for firewalling. Which one should be used? Well, I’d pick nftables, use it as long as possible, and move to firewalld if it would be required later on. This is suitable when the target is quite simple and lightweight, at least at the beginning of the lifespan of a device. On the other hand, if it’s already known from the get-go that the device is going to be a full-fledged configurable router, it’d be better to pick the firewalld right off the bat.

Meme about differences of nftables and firewalld

What about iptables and ufw then? Both have bitbake recipes and can definitely be installed, and I suppose they even work. iptables is a bit of a legacy software these days, so maybe avoid that one though. It should be possible to use iptables syntax with nftables backend if that’s what you want. ufw in Yocto uses iptables by default, but it should be possible to use nftables as the backend for ufw. So as usual with the open-source projects, everything is possible with a bit of effort. And as you can guess from the amount of weasel words in this chapter, I don’t really know these two that well, I just took a quick glance at their recipes.

Note that the strength of the firewall depends also on how protected the rulesets are. If the rules are in a plaintext file that everyone can write to, it’s safe to assume that everyone will do so at some point. So give a thought to who has the access and ability to edit the rules. Also, it may be worthwhile to consider some tampering detection for the rule files, but something like that is worth a text of its own.

In closing, I hope this helped you to set up your firewall. There are plenty of ways to set it up, this two-parter provides two options. It’s important to check with external tooling that the firewall actually works as it should and there are no unwanted surprises. Depending on the type of device you want to configure you can choose the simple way or the configurable way. I’d recommend sticking to the easy solutions if they’re possible. And finally, these are just my suggestions. I don’t claim to know anything, except that I know nothing, so you should check your system’s overall security with someone who’s an actual pro.

Yocto Hardening: Firewalls, Part 1: nftables

Find all of the Yocto hardening texts from here!

The eternal task of making the Yocto Linux build an impenetrable fortress continues. Next, we’ll look into setting up a firewall two different ways: the easy way with nftables, and in the part two the easily configurable way with firewalld. I must warn you beforehand that this text contains a bit of resentment towards kernel configuration. I also used ChatGPT as a tool for writing a part of this text, but I’ll inform you of the section I asked it to write.

So yeah, firewall. The system keeping the barbarians out of our Linux empire. Also, one of the most bad-ass names for a program, at least if taken literally. Reality is a bit less exciting, turns out “firewall” is some technical construction term. In the magical world of computers, a firewall is a piece of software used to allow or deny network traffic coming in and out of a machine. It’s mighty useful for improving the overall security of the system to be able to prevent unwanted remote connections and such.

Step 0 – Configuring Kernel

The first step of installing and configuring a firewall on a Yocto image is ensuring that the kernel is configured correctly. What, do you think you can just start configuring the firewall and blocking your nefarious enemies just like that? Pfft, get out with that casual attitude (please don’t).

To get some basic traffic filtering with nftables done, we need to ensure that the Netfilter is enabled in the kernel, as well as the Netfilter support for nftables, and that some nftables modules are installed as well. It’s all a bit confusing, but to summarize: Netfilter is the kernel framework for packet filtering and nftables is the front-end for Netfilter that can be used from user space. nftables can then be built with or without modules to extend its functionality. The following kernel configuration fragment should do the trick:

Getting this config fragment working was where things got problematic for me. I thought that I’d just slap =y to everything and be done with it, no need to install and load modules. Well, the configuration process of the kernel consists of merging the defconfig, kernel features, and configuration fragments, and as it turns out, the kernel feature enabling the Netfilter overrode some parts of my config fragment, changing =y to =m. For some reason bitbake didn’t give a warning about this. I’m quite sure it used to do.

But where is this mythical Netfilter kernel feature located? The fragment above enables only nftables related things, how to get the Netfilter working? Well, Kirkstone release (3.1.27) of Poky has this line that enables the feature by default in the kernel, and the related configuration fragment looks like this. The multiple configs lead to funny situations where you try to remember the difference between CONFIG_NF_CONNTRACK and CONFIG_NFT_CT and where they are defined and to what value.

Ensure that you’re also installing the kernel modules to the image by adding the following line somewhere in the image configuration:

IMAGE_INSTALL:append = " kernel-modules"

No use enabling the modules if they’re not installed, and by default nothing installs them, so they won’t get added to the final image. At least to the qemu64 core-image-minimal image they won’t be installed by default, guess if I found that out the hard way as well. If you need more fine-grained control over the modules that get installed, you can add them as a runtime recommendation to some package:

RRECOMMENDS:${PN} += "kernel-module-nf-tables kernel-module-nf-conntrack kernel-module-nft-ct ...

Most likely nftables package is the best place for this. However, I’d advise against this approach if possible, because there will be a lot of modules later on. If you choose to go down this route, the naming format of the modules is fortunately quite self-explanatory. CONFIG_NF_TABLES being built as a module generates kernel-module-nf-tables package, CONFIG_NFT_CT results in kernel-module-nft-ct being built etc, so this should be easily scriptable (although painful to maintain).

The Easy Firewall – nftables

The easy way of getting some firewalling done involves installing nftables, adding some rules to it, and loading them as a part of the start-up. This is suitable if you have a few interfaces with a simple configuration that doesn’t change often, preferably ever.

The actual first step of getting the firewall up and running is adding the nftables package to the image if you don’t have it already installed. This can be easily checked by running nft command. If it fails, add the package to the image configuration:

IMAGE_INSTALL:append = " nftables"

If you see the following types of errors when running the nft command it means that your kernel configuration is missing either nftables support (the first error) or some nftables module (the second error):

root@qemux86-64:~# nft
../../nftables-1.0.2/src/mnl.c:60: Unable to initialize Netlink socket: Protocol not supported
root@qemux86-64:~# 
root@qemux86-64:~# nft -f /tmp/test.conf
/tmp/test2:32:9-16: Error: Could not process rule: No such file or directory
        ct state vmap { established : accept, related : accept, invalid : drop } 
        ^^^^^^^^
root@qemux86-64:~# 

Quite often the firewall in Yocto is empty by default. The active rules of the firewall can be checked by running nft list ruleset command to print the current ruleset. If nothing is output, no rules are present. If something gets output, there are some rules present. Here’s a short reference of commands on how to set some basic rules and list them on the command line.

# List ruleset
nft list ruleset

# Add a table named filter for IP traffic 
nft add table inet filter

# Add an input chain that drops traffic by default
nft add chain inet filter input \{ type filter hook input priority 0 \; policy drop \}

# Add rule to allow traffic to port 22
nft add rule inet filter input tcp dport \{ 22 \} accept

# List tables and rules in a table
nft list tables
nft list table inet firewall

In addition to listing the tables and rulesets from inside the device, you should also try a black-box approach. Running nmap command against your device is a good way to check how the outside world sees your device. In this example, I have added the Dropbear SSH server to the image. Because the server is listening on port 22 and there are no rules defined in the firewall, the port is seen as open:

esa@ubuntu:~$ nmap 192.168.7.2
Starting Nmap 7.80 ( https://nmap.org ) at 2023-07-22 16:14 UTC
Nmap scan report for 192.168.7.2
Host is up (0.0024s latency).
Not shown: 999 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds

The command line commands are useful for editing the configuration run-time, but for the rest of the text we’re going to focus on a file-based configuration that is set during the Yocto build. The configurations presented here are mostly based on “a simple ruleset for a server” in nftables wiki. It contains some useful extra things not presented here, like a rule for the loopback interface, differentiating between ipv4 and ipv6 traffic and logging, and I recommend checking it out.

Usually, a good way to start creating a firewall ruleset is to block everything by default, and then start poking holes as needed. To block everything, you can write a configuration file with the following content:

flush ruleset

table inet firewall {
    chain inbound {
        type filter hook input priority 0; policy drop;
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy drop;
    }
}

This config creates one rule table named firewall for inet traffic. The table contains three chains: inbound, forward and output. You can load the ruleset with nft -f <config_file> command. Once you’ve blocked all traffic, you can give the nmap command another go. As you can see, blocking is really effective. nmap actually considers the device to be non-existent at this point:

esa@ubuntu:~$ nmap 192.168.7.2
Starting Nmap 7.80 ( https://nmap.org ) at 2023-07-22 16:18 UTC
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.03 seconds

Trying to ping from the device itself to the outside world also fails because the output is blocked. In theory, allowing outgoing traffic is not the worst idea, usually many sample configurations actually allow it. But if you know the traffic that’s going to go out and the ports that will be used there’s in my opinion no sense allowing all of the unnecessary traffic. That’s just an attitude that will lead to botnets. But if you want/need to allow all outgoing traffic, you can just drop the output chain from the config. By default everything will be accepted.

After nothing goes in or out, it’s time to start allowing some traffic. For example, if you want to allow SSH traffic from the barbarians to port 22 (generally a Bad Idea, but for the sake of an example), you can use the following ruleset:

flush ruleset

table inet firewall {
    chain inbound {
        type filter hook input priority 0; policy drop;

        # Allow SSH on port TCP/22 for IPv4 and IPv6.
        tcp dport { 22 } accept
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy drop;

        # Allow established and related traffic
        ct state vmap { established : accept, related : accept, invalid : drop } 
    }
}

If you want to define multiple allowed destination ports, separate them using commas, like { 22, 80, 443 }. If you choose not to block outgoing connections, you obviously don’t need the output chain presented here.

On the other hand, if there’s a process in the Yocto image that wants to contact the outside world, you could consider adding a ruleset like below:

flush ruleset

table inet firewall {
    chain inbound {
        type filter hook input priority 0; policy drop;

        # Allow established and related traffic
        ct state vmap { established : accept, related : accept, invalid : drop } 
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy drop;

        # You can check the ephemeral port range of kernel 
        # from /proc/sys/net/ipv4/ip_local_port_range
        tcp sport 32768-60999 accept
    }
}

Note how we need to use a port range for the output. Outgoing connections usually use a port from an ephemeral range for their connection, meaning that you may not know beforehand the exact port that needs to be allowed. Having fixed ports for outgoing traffic makes the device more susceptible to port scanning and prevents multiple connections from the same service because the port is already used, but if you choose to create a service that uses a static source port, you can use a single port number in the rule.

You can combine the rules in the chains as required by your system and services. If you want to enable the ping command (for ipv4), you can add the following piece of configuration to the input chain (note that this is just a single line of configuration, not a full config):

icmp type echo-request limit rate 5/second accept

In the end, creating the firewall ruleset is fairly simple: block everything and allow only necessary things. The difficulty really comes from keeping the rules up-to-date when something changes. However, when developing an embedded device with Yocto you generally should have a well-defined list of allowed ports & traffic (as opposed to a general-purpose computer used by a living human where all sorts of traffic and configs may come and go at the user’s whim). If that’s not the case, the part 2 where we work with firewalld may be useful to you.

So, now we know how to write the perfect configuration file. But how to add this to a Yocto build? You can create a bbappend file for nftables into your own meta-layer. This append then installs the configuration file, because nftables does not install any configuration by default. An example of how to add a configuration to nftables can be found in meta-firewall-examples repository I made for this blog post.

You should also note that nftables package doesn’t contain any mechanism to activate the rules during the start-up. Therefore you should append the nftables recipe so that it adds an init.d script or a service file that activates the rules from the configuration file before the networking is started. You can find an example of this as well from the same meta-firewall-examples repository.

Note that if you do run-time edits to the configuration with nft command, the changes will be lost when the device is reset. To ensure that the changes are kept, you can run the following command to overwrite the default config with the new, enhanced configuration:

nft list ruleset > /path/to/your/configuration-file

And one more thing: if you face some No such file or directory issues when trying more exotic configurations you are most likely missing some kernel configurations. In part 2, we’ll be enabling pretty much every feature of the Netfilter, so that’ll (most likely) be helpful. On that cliffhanger, we eagerly await the next chapter in this ever-evolving journey of discovery and innovation (I asked Chat-GPT to write that sentence).

Yocto Hardening: Finding & Fixing CVEs

Find all of the Yocto hardening texts from here!

Last time when we were talking about Yocto hardening we focused on setting up extra users to prevent unauthorized root access and to avoid situations where the users would have too open permissions. Getting these right is a good low-hanging fruit, but there are plenty of other things to consider for hardening the system. One of the most important things to do is to eliminate (or at least minimize) the security vulnerabilities weakening the overall system defences.

What are CVEs?

Despite what one may be inclined to think, open-source code is not always perfect. It may contain bugs, which is a bit annoying, but it may also contain vulnerabilities, which is potentially a bit more dangerous. These can lead to denial of service attacks, data leaks, or even unauthorized accesses & system takeovers. Not good. Very bad. The usual stuff.

This is what I assume an average security operations centre looks like.

CVE (Common Vulnerabilities and Exposures) is a system maintained by The United States National Cybersecurity FFRDC and operated by Mitre Corporation. The system contains information about, well, vulnerabilities and exposures. “CVE” as a noun can also mean vulnerability or exposure. Or at least I often use the acronym with that meaning. Checking your Yocto system against this database of vulnerabilities can help detect security issues lurking in the open-source code components.

Checking vulnerabilities in Yocto

Every CVE doesn’t necessarily lead to unauthorised root access, and there may be vulnerabilities that don’t have a CVE identifier. However, scanning through the CVEs in your system should provide a fairly good overview of how vulnerable your system is against possible CVE exploits. At least at the time of the scan, there are constantly new issues popping up which means you have to keep doing it often. Also, it’s good to keep in mind that this kind of scan does not check for other kind of weaknesses in the system. But at least performing the CVE check in the Yocto system is fairly simple. Something is simple in Yocto, what?

My whole worldview is being shattered.

In all simplicity, just add this line to local.conf to check your packages & images for common vulnerabilities.

INHERIT += "cve-check"
# I had some issues with slow download speeds. If you get weird timeouts
# when fetching issue database, try adding this line as well:
CVE_SOCKET_TIMEOUT = "180"

Yocto has an include file containing some exclusions for CVEs that are deemed impractical to fix from the Yocto project’s point of view. To add this exclusion list to your build, add this to the local.conf as well:

include conf/distro/include/cve-extra-exclusions.inc

Now when you run the build you’ll most likely end up with more warnings than before. Yocto’s CVE checker prints a warning to the build output for every vulnerability it detects, and depending on the complexity and age of your system your terminal may end up quite yellow. It’s more useful to inspect the CVE reports than to try and remember the build output as it scrolls by. These reports can be found in <build-folder>/tmp/deploy/cve directory, and we’ll inspect them a bit later. It’s worth knowing that this CVE checker may not be perfect, meaning that there may be some vulnerabilities lurking around that it does not detect. But it’s at least better than going through all of the source code eyeballing it.

Analysing the results

Ok, now you’ve proved to yourself that your system has holes that need to be plugged. How to get over this uneasy feeling of constant doom looming over your system? The first thing to do is listing out all the CVE IDs and checking their status & severity to become even more stressed. Or relaxed, depends a bit on how well your system is maintained.

This is the most likely scenario though.

Further information for the CVEs can be found on MITRE’s CVE site, and some more complementary information can be found on NIST’s (National Institute of Standards and Technology) NVD (National Vulnerability Database). Plenty of acronyms. NVD is a system containing the same CVE IDs as MITRE’s site but with severity scores and usually some extra information. Both are useful, but for practical purposes, NVD tends to be more useful. MITRE’s site can contain some new info not yet present in NVD, but NVD contains the information in a more easily digestible format.

Let’s take CVE-2023-22451 as an example. CVE listing for the vulnerability shows a summary, affected versions, and a bunch of links. NVD entry has the same information, severity score, modification date, and a bit more information about the links so you won’t have to guess which link is the patch and which is advisory (this can be useful on bigger issues where there are a dozen links). For what it’s worth, Yocto’s CVE checker uses NVD’s database.

Now that we’ve gone through some background information, we can investigate the results of the CVE check. The aforementioned <build-folder>/tmp/deploy/cve contains two summary files: cve-summary and cve-summary.json. These contain all the detected issues, both patched and unpatched. The format of the files isn’t all that intuitive for getting an actual summary, so I wrote a Python tool for summarising the results. The design is very human:

# Replace the file path with your path
python3 ./cve-parser.py -f build/tmp/log/cve/cve-summary.json

Very easy to use. By default, the parser will output the unpatched CVEs, and from each CVE it will print the package name, CVE ID, CVSS scores, and the link to the NVD site for more info. There is more information per issue available, the results can be sorted, and the ignored & patched issues can also be printed if needed. To get the complete and current list of options, run:

python3 ./cve-parser.py -h

I’m not going to document the whole program here in case I decide to change it at some point. But for the sake of example, if your boss asks you to get the total amount of unpatched CVEs in the system, you can run the command below to get “extra info” from the parser. Currently, “extra info” just contains the number of displayed issues, but if you’ve got ideas for it feel free to let me know:

# -u display unpatched issues (default behavior, but added just in case)
# -e displays extra info
python3 ./cve-parser.py -u -e -f build/tmp/log/cve/cve-summary.json

Then, if your boss asks how many of them actually need to be fixed, use this to sort the CVEs by severity score and count how many have a score higher than 8.3

# -u display unpatched issues
# -ss2 sort by CVSSv2 score
python3 ./cve-parser.py -u -ss2 -f build/tmp/log/cve/cve-summary.json

Just kidding, please fix them all (if applicable).

Fixing the CVEs

After you’ve justified your fears of getting hacked by realizing that your system has seven CVEs with a CVSS score higher than 9.5 (three of them in the kernel) and a few dozen more with just a slightly lower priority, it’s time to ask the big question: what can I do? What can I possibly do? This didn’t help with the feeling of constant doom at all.

The first step is going through the list of unpatched issues. There tend to be quite a few false positives that do not apply to the current system. For example, a brand new Poky build (at the time of writing) reports an unpatched CVE-2017-6264 that’s been around six years already. This is a vulnerability in the NVIDIA GPU driver, and it applies to Android products. Most certainly it’s a false positive in your Yocto system, but since the bad code is present in the source code, it’s reported as unpatched. You can ignore these false positives by adding the line below somewhere to your build configuration (local.conf works, but it’s maybe not the best place for it):

# The format is CVE_CHECK_IGNORE="<cve-id>", e.g.
CVE_CHECK_IGNORE="CVE-2017-6264"
# In the older Yocto versions this was called CVE_CHECK_WHITELIST
# so if ignore doesn't work, try the old whitelist variable

After ignoring the false positives fixing the rest of the CVE issues is easy, in theory. Just keep the Yocto at its newest (or relatively new LTS) version. This should keep the packages fresh. Sounds simple, but unfortunately, this process tends to cause a big headache with incompatibilities and such. However, that’s the best advice I can give.

If there is a need to update a recipe to a newer version than officially supported by the version of Yocto you’re using, copy the recipe in question to your meta-layer, update the recipe version in the filename and fix SRCREV to match. When hacking around like this remember to hope that nothing falls apart in the system even more than during a regular Yocto version update.

Sometimes, updating the packages is not an option. Perhaps updating the package breaks the build, a newer version of a library isn’t compatible with your application, or there is some other Perfectly Good Reason you’re not allowed to do that. That’s when it’s time to do some patching to make the confusing build system even more spaghettified.

Basically, this approach consists of heading to the NVD entry for the CVE, checking if there’s a patch for the vulnerability and if there is, porting the patch to the Yocto build. For example, the aforementioned CVE-2023-22451 mentions this commit as its patch in the NVD entry. Copy the contents of the commit to a patch file, create a bbappend file for the recipe and add the patch to the recipe with that bbappend.

If there’s no patch you can wait. Or if you’re like a kid on Christmas and can’t wait, you can try digging the package’s git repositories for the corrective commit before it’s been released. It isn’t fun, and rarely it’s a productive use of time, but occasionally, some fixes can be found like this before they are made official. In the worst case you can always fix it yourself, most likely causing another vulnerability in the process.

That’s the short explanation of how you can check and fix the CVE vulnerabilities in your system. The theory of the process is fairly simple. However, it tends to get a bit more complicated in the actual world, especially when trying to update older legacy systems where the stale, non-updated packages contain more patches than original code. But I guess that’s the problem with real life, it tends to mess up good theories.

You can find the next part about Yocto hardening here. It’s about firewalls.

How to build Yocto with Apple Silicon

If you’re like me, you have more money than brains, and not too much of either really. This can lead to situations where you end up buying a MacBook without actually checking if it supports Yocto builds, the one thing you inexplicably like to waste your little free time on. As it turns out after getting the nearly 2K€ laptop, builds on Mac are not supported. But don’t worry, where there is a problem, there usually is a convoluted solution.

Rube Goldberg Win GIF by Millions - Find & Share on GIPHY

As one might expect, the answer is virtualization. Specifically Docker virtualization this time. Writing this kind of instructional text is becoming my second nature, earlier I wrote how to build Yocto on Windows/WSL, and now how to build the damn thing on Mac. One day I’ll get an actual Linux computer suitable for this kind of development work, but not today. I’m out of money at the moment. Anyways, to the actual guide. I’ll warn you beforehand that my knowledge of Docker is quite superficial, but I’ll try to explain things as well as I can.

Installing Docker

When doing Docker builds the first step obviously is to install Docker. Get it from here and come back when you’re done.

Gonna Go Homer Simpson GIF - Find & Share on GIPHY
Don’t worry, they definitely are.

Getting correct images

Now, the actual tricky part: getting the Docker images suitable for building Yocto. Docker images are used to create the containers that are the virtualized environment we can do the builds in. Yocto Development Tasks Manual mentions CROPS (CROss PlatformS) Docker images that can be used for building in non-Linux hosts. CROPS itself also has this nice Mac guide that almost works.

Well, I guess the manual works for the older amd64 MacBooks because the pre-built Docker images are only available in amd64 format. Since Apple Silicon is arm64, Docker gives a nice warning that the emulated image may or may not work, and spoiler alert, it does not. Attempting to run bitbake after following the guide gives the following error:

OSError: Cannot initialize new instance of inotify, Errno=Function not implemented (ENOSYS)

However, I recommend keeping that CROPS guide nearby because we are going to follow it once we get the suitable images built. The first step of the journey is pulling the Docker image source repositories for the two images mentioned in the CROPS Mac guide:

% git clone https://github.com/crops/samba.git
% git clone https://github.com/crops/poky-container.git

The first one is responsible for creating a samba file server to provide the contents of the build container to the Mac file system via a Docker volume, and the second one is the actual builder image. Building the samba image is simple enough:

% cd samba
% docker build . -t crops/samba:selfbuilt
% cd ..

The poky-container image is a bit trickier. It attempts to fetch an amd64 operating system image provided by CROPS as its base layer. Instead of using that image, we need to find something else that’s more suitable for us and arm64.

Season 3 Episode 20 GIF by SpongeBob SquarePants - Find & Share on GIPHY
Not much.

Docker Hub is a site containing Docker images for a variety of uses. Among many others, it also contains the CROPS amd64 images that we are not able to use. By searching for “Yocto”, and limiting the results to arm64 architecture we can find quite a few results. However, none of these (at least at the time of writing) fit into the CROPS workflow or are up to date, meaning that they won’t work with the newer versions of Yocto. Therefore we need to build the operating system base image by ourselves.

The Dockerfiles used to build the images are open source, but unfortunately, the scripts in them use GNU versions of some commands, meaning that the scripts won’t work with the BSD commands Mac has. But fear not, I made a fork for building the base images on Mac! It works at least for the Ubuntu 22.04 image, I was a bit lazy and didn’t check the dozen other options. To build the Ubuntu base image, run the following commands:

git clone git@github.com:ejaaskel/yocto-dockerfiles.git
cd yocto-dockerfiles
export REPO=ejaaskel/yocto
export DISTRO_TO_BUILD=ubuntu-22.04
./build_container
cd ..

This will create two images: ejaaskel/yocto:ubuntu-22.04-base and ejaaskel/yocto:ubuntu-22.04-builder. Now we just need to go to the CROPS poky-container Dockerfile and replace their base image with the one we just built. After that, we should be able to build the poky-container as an arm64 image and get to actually building Yocto.

% cd poky-container
## Open your favourite text editor (it better be nvim),
## and replace these lines in Dockerfile:
ARG BASE_DISTRO=SPECIFY_ME
FROM crops/yocto:$BASE_DISTRO-base
## With these lines to specify the distro version and 
## use the Docker image we built:
ARG BASE_DISTRO=ubuntu-22.04
FROM ejaaskel/yocto:$BASE_DISTRO-base
% docker build . -t crops/poky:selfbuilt
% cd ..

Now we’re again on track to follow that CROPS guide. We just have to suffix every mention of the CROPS images with :selfbuilt so that we’ll use the built images and not pull images from Docker Hub. So whenever there’s a mention of crops/samba in the guide, write crops/samba:selfbuilt, and when there’s a mention of crops/poky, use crops/poky:selfbuilt instead.

Building Yocto

If you followed the instructions in the CROPS guide you should end up with an Ubuntu terminal in /workdir and a Finder window that’s connected to the same location in the Docker container. After that, it’s just a matter of following Yocto’s Quick Build manual or building something else that’s actually interesting. Once the build finishes, you can access the files in Finder as you normally do, or if you prefer to use the command line, docker cp should do the trick as well:

# Use docker ps to get crops/poky:selfbuilt container id
% docker ps
% docker cp <container-id>:workdir/<whatever-you-want-to-copy> .

By the way, if you stick to docker cp for moving stuff around, you don’t actually need to start the samba container. Saves a bit of effort.

Fight Club Work GIF - Find & Share on GIPHY
of a copy

If you happen to need a root shell in the container for some reason, like installing packages, you can open it like this:

# Use docker ps to get crops/poky:selfbuilt container id
% docker ps
% docker exec -it --user=root <container-id> bash

If (and when) you run out of disk space, you can increase the virtual disk size from the Docker Desktop program. Select the cog from the upper right corner to enter settings, navigate to Resources->Advanced and from there you can increase the virtual disk limit. I’m quite sure there is a way to do this in the command line as well, but hopefully this isn’t a task that I’ll have to do so often that I’d need to figure it out.

Can you see how artistic those red circles are? All thanks to the new MacBook.

That’s all for this time! Hopefully, this helped you to get started with Yocto builds on Apple Silicon a bit quicker. Fortunately, this wasn’t really anything groundbreaking, mostly it was just a matter of combining a few different information sources and a bit of script editing. The biggest sidestep from the CROPS instructions was building the Docker images instead of using the ready-built ones. But for someone who’s new to Mac & Docker, it may take a surprisingly long time to figure this out. At least it did for me. Writing this also helped a bit with my buyer’s remorse. Until next time!

Yocto hardening: Non-root users, sudo configuration & disabling root

Find all of the Yocto hardening texts from here!

Cybersecurity. The never-ending race between you trying to secure your precious IoT device and some propeller head who’s finding the wildest exploits and destroying your system just in the time between breakfast and lunch (although true hackers work during night, not in the mornings). Or perhaps you forgot to update your WordPress plugins, and now your fantastic development blog is hacked. Or perhaps you’ve just been postponing that security update on your Android phone for six months. We all have some experience with cybersecurity.

Usually, there are some simple things to do to prevent the worst catastrophes. Update software, make sure the run-time environment is sane & secure, and encrypt the secrets to mention a few. In this new blog series, I try to explain how to make your Yocto systems more secure by adding some common-sense security to the system. All of these ideas are familiar from Linux desktop & server environments, but I’m going to show how to apply them to Yocto builds. So, unfortunately, I’m not actually going to cover how to keep a fantastic WordPress site secure.

Well okay, no need to be upset. Just press the red “update plugins” button every now and then. And if you’re self hosting, I wish luck to your distro upgrades.

As usual with any security advice, it’s good to be slightly skeptical when reading it. This means that some rando writing blog posts on their blog may not be a 100% trustworthy or up-to-date source on everything. Also, those randos will take no responsibility for the results of their advice. However, their advice may provide useful guidance and general tips that you can apply to your system’s security hardening plan. You do have a security plan, right?

So, the first topic of this Yocto hardening exercise is users. As you may have heard, running and doing everything as the root user is generally considered a Bad Idea. Having services running under the root user unnecessarily can result in root user executing arbitrary code using the most imaginative methods. Or, if the only user available for logging in is the root user you’re not only giving away root permissions to malicious users, but to incompetent users as well.

This text will assume intermediate knowledge of Yocto, meaning that I won’t explain every step in depth along the way. Hopefully, you know what an append-file and local.conf are. The code and config snippets were originally written for Yocto Kirkstone checked out “at some point after 4.0.3 release”, and later re-tested with version 4.0.15.

Before we get started with the actual hardening, there’s one preliminary task to do. Ensure that you don’t have DEBUG_TWEAKS in either IMAGE_FEATURES or EXTRA_IMAGE_FEATURES. Not only is it unsafe as it allows root login without a password, and a bunch of other debug stuff, but it also makes some examples shown here behave in unexpected ways.

Creating non-root users

Here is a code snippet that creates a service user for the system. This user can be logged in with, and they have sudo capabilities (if sudo-package is installed). Insert this code either into a image recipe or a configuration file:

# If adding to configuration file, use INHERIT += "extrausers" instead.
inherit extrausers

IMAGE_INSTALL:append = " sudo"

# This password is generated with `openssl passwd -6 password`, 
# where -6 stands for SHA-512 hashing alorithgm
# The resulting string is in format $<ALGORITHM_ID>$<SALT>$<PASSWORD_HASH>,
# the dollar signs have been escaped
# This'll allow user to login with the least secure password there is, "password" (without quotes)
PASSWD = "\$6\$vRcGS0O8nEeug1zJ\$YnRLFm/w1y/JtgGOQRTfm57c1.QVSZfbJEHzzLUAFmwcf6N72tDQ7xlsmhEF.3JdVL9iz75DVnmmtxVnNIFvp0"

# This creates a user with name serviceuser and UID 1200. 
# The password is stored in the aforementioned PASSWD variable
# and home-folder is /home/serviceuser, and the login-shell is set as sh.
# Finally, this user is added to the sudo-group.
EXTRA_USERS_PARAMS:append = "\
    useradd -u 1200 -d /home/serviceuser -s /bin/sh -p '${PASSWD}' serviceuser; \
    usermod -a -G sudo serviceuser; \
    "

The extrausers class allows creating additional users to the image. This is done simply by defining the desired commands to create & configure users in the EXTRA_USERS_PARAMS variable. As a side note, it’s good to have static UIDs for the users as this will make the builds more reproducible.

In a perfect world, every custom service you create for your system would run under a non-root user. When writing a recipe that creates a daemon or other kind of service, you can use the snippet below to add a new user in a non-image recipe:

inherit useradd
USERADD_PACKAGES = "${PN}"
GROUPADD_PARAM:${PN} = "--system systemuser"
# This creates a non-root user that cannot be logged in as
USERADD_PARAM:${PN} = "--system -s /sbin/nologin -g systemuser systemuser"

As you can see, adding a new user in a non-image recipe is slightly different than in the image recipe. This time we’re only giving parameters to useradd and groupadd commands. After adding this, you should be able to start your daemon as the systemuser in a startup script. If you need root permissions for some functionality in your service but don’t want the service to be run as root, I’d recommend reading into capabilities. It’s worth noting that when adding users in a service recipe like this, the additions are done per-package basis, not per-recipe basis. This means that you can create and add new users in a quite flexible manner.

The flexibility tends to come with some complications though…

Editing sudoers configuration

By default, when adding the sudo-package to the build it doesn’t do much. It doesn’t actually allow anything special to be done by the users, even if they are in the sudoers group. That’s why we need to edit the sudo-configuration. There are two ways of doing this, either by editing the default sudoers file or by adding drop-in sudoers.d configurations. First, editing the sudoers file (which is the worse method in my opinion).

There are two ways of doing this, and I’m not 100% sure which one is less bad, so I’ll leave it to you to decide. The first option is adding a ROOTFS_POSTPROCESS_COMMAND to the image recipe:

enable_sudo_group() {
    # This magic looking sed will uncomment the following line from sudoers:
    #   %sudo   ALL=(ALL:ALL) ALL
    sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/'  ${IMAGE_ROOTFS}/etc/sudoers
}

ROOTFS_POSTPROCESS_COMMAND += "enable_sudo_group;"

The second option is creating a bbappend file for the sudo recipe, and adding something like this there:

do_install:append() {
    # Effectively the same magic sed command
    sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/'  ${D}/${sysconfdir}/sudoers
}

Both do the same thing, and both are hacky. The correct choice really depends on if you want to edit the sudoers file in a different way for each image, or if you want the sudo configuration to depend on something else. You can also of course supply your own sudoers file instead of sed-editing it, and that can be a lot more maintainable way of doing this if you have more than just one change.

However, a more flexible way of configuring sudo is with the sudoers.d drop-in files. It also involves a lot fewer cryptic sed commands. Let’s assume that you have a recipe that creates a new user lsuser (not to be confused with a loser), and you want that user to have sudo rights just for ls-command (considering rights given to this user, they may actually be a loser). For this purpose you need to create a bbappend for sudo, and add something like this snippet to create a drop-in configuration:

do_install:append() {
    echo "lsuser ALL= /bin/ls " > ${D}${sysconfdir}/sudoers.d/lsuser
}

FILES_${PN} += " ${sysconfdir}/sudoers.d/lsuser"

Again, if your drop-in configuration is more complex than one line, you can provide it as a configuration file through SRC_URI and install that in your image instead of echoing configuration lines. I recommend reading sudo-configuration documentation for more info on how to precisely set the permissions because let’s be honest, these two examples aren’t the shining examples of sudo configuration.

The downside of this approach is that you need to add the drop-in configuration in a sudo bbappend because /etc/sudoers.d will conflict when creating the root filesystem if you add the drop-in in another recipe. This means that when you’re creating a user in a non-sudo recipe and add the drop-in conf in the sudo recipe you’ll have the user creation & configuration handled in two different places, which is perfect for forgetting to maintain things.

Disabling root-login

Disabling root login is useful for a multitude of reasons.

  1. You usually want to make sure that no user has the full command set available
  2. You also want that the users’ privileged actions to end up logged in auth.log when they use sudo
  3. In general, it’s “a bit of a risk” to allow users to log in as root

Disabling the root user can be achieved in multiple ways, and I’m going to cover three of them here: fully locking the root account, removing the login shell, and disabling the SSH login. In theory, locking the root account should be enough, assuming non-root users cannot unlock it, but I’ll present all the methods as they may be useful for different situations.

First, disabling the root account. This is a method I found from this StackOverflow answer, kudos there. The simple snippet below pasted in the image recipe should lock and expire the root account:

inherit extrausers
EXTRA_USERS_PARAMS:append = " usermod -L -e 1 root; "

Then, removing the login shell. Or more like setting the login shell to nologin. This isn’t strictly required after locking the root account but can be achieved by creating a bbappend shown below to base-passwd recipe:

do_install:append() {
    # What this magic sed does:
    # "In the line beginning with root
    # replace /bin/sh with /sbin/nologin
    # in file passwd.master"
    # (passwd file gets generated from this template during install)
    sed -i '/^root/ s/\/bin\/sh/\/sbin\/nologin/' ${D}${datadir}/base-passwd/passwd.master
    # sry for the picket fence

    # I think it's peak sed that I need a four line
    # comment to explain a one-liner
}

EDIT: It was suggested in the comments that usermod can be used for disabling the root login shell as well. That solution is a lot cleaner than this sed & bbappend. Check the comments below for an example.

And finally, disabling the SSH login. You should always disable the root login over remote networks. If you need, you can just disable the SSH login for root and still allow other service users to log in as root. This is still a better option than doing nothing, as this will prevent logging in as root remotely and attackers would need to know two passwords to gain root access (one for the service user, and the other one for the root user).

Disabling the root login varies a bit between the SSH servers. Dropbear disables root login by default with -w parameter in DROPBEAR_EXTRA_ARGS variable located in the /etc/defaults/dropbear file (note that if you have debug-tweaks enabled, the file actually contains -B, allowing root-login). You can overwrite the file with your own dropbear.default file during the build if you want to add more options.

Similarly, OpenSSH-server disables root logins with passwords if debug-tweaks is removed from the IMAGE_FEATURES (and allows them if it’s present). This is achieved by sed-editing SSH configuration files. If you want to see how this is exactly done, check ssh_allow_root_login function in meta/classes/rootfs-postcommands.bbclass (part of poky).

However, it’s worth noting that this default behaviour doesn’t prevent public key authentication. If you want to disable that as well, you can add this function as a rootfs post-process task to the image recipe. And if needed, it could of course be modified to work in a bbappend as well.

# The function here searches sshd_config and sshd_config_readonly files for a
# commented line containing PermitRootLogin, and replaces it with "PermitRootLogin
# no" to prevent any sort of root login.

disable_rootlogin() {
    for config in sshd_config sshd_config_readonly; do
        if [ -e ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config ]; then
            sed -i 's/^[#[:space:]]*PermitRootLogin.*/PermitRootLogin no/' ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config
        fi
    done
}

ROOTFS_POSTPROCESS_COMMAND += "disable_rootlogin;"

The thing to note about disabling root login is that too lenient sudo- or filesystem-permissions for non-root users can make the whole thing useless. For example, if a service user has sudo access to passwd command they can easily unlock the root account. If a service user has write permissions to /etc they can set the root’s login shell and edit SSH configuration. And finally, disabling root login like shown in this text does nothing to prevent shell escapes made possible by incorrectly set sudo-permissions.

Might as well roll out the red carpet

That’s it for this time! Hopefully, these tips helped you to achieve the minimum security in your Yocto images. Do these code snippets make your system unhackable? Not even close. Do they make it slightly more inconvenient? I hope so (but no guarantees for that either). But at least now you can show your Yocto image to an independent professional security consultant and say that you tried before they tear the whole system to pieces.

You can find the second part of the Yocto hardening series here. It’s about fixing CVEs.