You can find the other Yocto hardening posts from here!

Oh yes, it’s time for more of the security stuff. We are getting into the difficult things now. So far we have mostly been focusing on hardening the kernel and userspace separately, but this time we will zoom out a bit and take a look at securing the entire system. First, we are going to start hardening the boot process to prevent unwanted bootflows and loading undesired binaries.

I know that there is a philosophical and moral question of whether doing this is “right”, potentially locking the devices from the people using them. I’m not going to argue too much in either direction. I’d like everything to be open and easily hackable (in the good sense of the word), but because the real world is the way it is, keeping the embedded devices open doesn’t always make sense. Mostly because of the hacking (in the bad sense of the word). Anyway, I hope you use the power you will learn for good.

What Is Measured Boot

Simply put, the measured boot is a boot feature that hashes different boot components and then stores the hashes in immutable hash chains. The measured boot can perform hashing during different stages of the boot. The hashed items can for example be the kernel binary, devicetree, boot arguments, disk partitions, etc. These calculated hashes usually then get written to the platform configuration registers (PCR) in TPM.

These registers can only be extended, meaning that the existing value in the register and the new value get hashed together, and this combined hash then gets stored in the register. This creates a chain of hashes. This can then be called a “blockchain”, and it can be used to raise unlimited venture capital funding (or at least it was possible before AI became the hype train locomotive). The hash chain can also be used to detect unwanted changes in the chain because if one of the hashes in the chain changes, all the subsequent hashes after that will be changed as well.

To make actual use of these registers and their contents, attestation should be performed. In attestation, it is decided whether the system is in an acceptable state or not for performing some actions. For example, there could be a check that the PCRs contain certain expected values. Then, if the system is considered to be cool, for example filesystems may be decrypted, services could be started, or remote connections may be made.

It’s worth noting that measured boot doesn’t prevent loading or running unwanted binaries or configurations, it just makes a note if such a thing happened. Attestation on the other hand may prevent some unexpected things from happening if it is configured to do so. To prevent loading naughty stuff into your system, you may want to read about the secure boot. There’s even a summary of the differences coming sooner than you think!

Measured Boot vs. Secure Boot

Secure boot is a term that’s often confused with the measured boot, or the trusted boot, or the verified boot, so it is worth clarifying how these differ. This document sums it up nicely, but I’ll briefly summarize the differences in the next paragraphs. The original link contains some more pros and cons explained in an actually professional manner, so I recommend checking it out if you have the time.

In secure boot (also known as verified boot) each boot component checks the signatures of the next boot item (e.g U-boot checks Linux kernel, etc.), and if these don’t match with the keys stored in the device, the boot fails. If they match, the component doing the measurement transfers the control to the next component in boot chain. This fairly rigid system gives more control over the boot process, but signature verification and key storage aren’t trivial problems to solve. Also, updates to this kind of system are difficult.

Measured boot (also known as trusted boot) only measures the boot items and stores their hashes to TPM’s PCRs. It is then the responsibility of the attestation process to decide if the event log is acceptable or not for proceeding. This is more flexible and allows more options than a simple “boot or no boot”, but it is quite complex, and in theory, may allow booting some bad configurations if attestation isn’t sufficient. Performing the attestation itself isn’t that easy either. While local attestation is simpler to set up, it’s susceptible to local attacks, and with remote attestation, you have a server to set up and need a secure way of transferring the hashes to the attestation server.

So, despite having quite similar names, secure boot and measured boot are quite different things. Therefore a single system can have both systems in place. It’s actually a good idea, assuming the performance and complexity hits are acceptable. The performance hit is usually tolerable, as the actions need to be performed once per boot (as opposed to some encrypted filesystems where every filesystem operation takes a hit). Complexity on the other hand, well… In my experience, things won’t surely become easier after implementing these systems. All in all, everything requires more work and makes life miserable (but hopefully for the bad actors as well).

Adding Measured Boot to Yocto

Now that we know what we’re trying to achieve, we can start working towards that goal. As you can guess, the exact steps vary a lot depending on your hardware and software. Therefore, it’s difficult to give the exact instructions on how to enable measured boot on your device. But, to give some useful advice, I’m going to utilize the virtual QEMU machine I’ve been working on a few earlier blog texts.

Yocto Emulation: Setting Up QEMU with U-Boot
Yocto Emulation: Setting Up QEMU with TPM

I’ve enabled measured boot also on Raspberry Pi 4 & LetsTrust TPM module combination using almost the same steps as outlined here, so the instructions should work on actual hardware as well. I’ll write a text about this a bit later…

Edit: The text for enabling the measured boot on Raspberry Pi 4 is now available, check it out here.

Configuring U-Boot

You want to start measuring the boot as early as possible to have a long hash chain. In an actual board, this could be something like the boot ROM (if boot ROM supports that) or SPL/FSBL. In our emulated example, the first piece doing the measurement is the U-boot bootloader. This is fairly late because we can only measure the kernel boot parameters, but we can’t change the boot ROM and don’t have SPL so it’s the best we can do.

Since we’re using U-Boot, according to the documentation enabling the boot measurement requires CONFIG_MEASURED_BOOT to be added into the U-Boot build configuration. This requires hashing and TPM2 support as well. You’ll most likely also want CONFIG_MEASURE_DEVICETREE to hash the device tree. It should be enabled automatically by default, at least in U-boot 2024.01 which I’m using it is, but you can add it just in case. The configuration fragment looks like this:

# Dependencies
CONFIG_HASH=y
CONFIG_TPM_V2=y
# The actual stuff
CONFIG_MEASURED_BOOT=y
CONFIG_MEASURE_DEVICETREE=y

Measured boot should be enabled by default in qemu_arm_defconfig used by our virtual machine, so no action is required to enable the measured boot for that device. If you’re using some other device you may need to add the configs. On the other hand, if you’re using something else than U-Boot as the bootloader, you have to consult the documentation of that bootloader. Or, in the worst case, write the boot measurement code yourself. U-Boot measures OS image, initial ramdisk image (if present), and bootargs variable. And the device tree, if the configuration option is enabled.

Editing the devicetree

Next, if you checked out the link to U-Boot documentation, it mentions that we also have to make some changes to our device tree. We need to define where the measurement event log is located in the memory. There are two ways of doing this: either by defining a memory-region of tcg_event_log type for the TPM node, or by adding linux,sml-base and linux,sml-size parameters to the TPM node. We’re going to go with the first option because the second option didn’t work with the QEMU for some reason (with the Raspberry Pi 4 it was the other way around, only linux,sml-base method worked. Go figure.)

For this, we first need to decompile our QEMU devicetree binary that has been dumped in the Yocto emulation blog texts (check those out if you haven’t already). The decompilation can be done with the following command:

dtc -I dtb -O dts -o qemu.dts qemu.dtb

Then, you can add memory-region = <&event_log>; to the TPM node in the source so that it looks like the following:

tpm_tis@0 {
    reg = <0x00 0x5000>;
    compatible = "tcg,tpm-tis-mmio";
    memory-region = <&event_log>;
};

After that, add the event log memory region to the root of the device tree. My node looks like this:

reserved-memory {
	#address-cells = <0x01>;
	#size-cells = <0x01>;
	ranges;
	event_log: tcg_event_log {
		#address-cells = <0x01>;
		#size-cells = <0x01>;
		no-map;
		reg = <0x45000000 0x6000>;
	};
};

Commit showing an example of this can be found from here. I had some trouble finding the correct location and addresses for the reserved-memory. In the end, I added reserved-memory node to the root of the device tree. The address is defined to be inside the device memory range, and that range is (usually) defined in the memory node at the root of the devicetree. The size of the event log comes from one of the U-Boot devicetree examples if I remember right.

Note that my reserved memory region is a bit poorly aligned to be in the middle of the memory, causing some segmentation. You can move it to some other address, just make sure that the address is not inside kernel code or kernel data sections. You can check these address ranges from a live system by reading /proc/iomem. For example, in my emulator device they look like this;

root@qemuarm-uboot:~# cat /proc/iomem
09000000-09000fff : pl011@9000000
09000000-09000fff : 9000000.pl011 pl011@9000000
09010000-09010fff : pl031@9010000
09010000-09010fff : rtc-pl031
09030000-09030fff : pl061@9030000
0a003c00-0a003dff : a003c00.virtio_mmio virtio_mmio@a003c00
0a003e00-0a003fff : a003e00.virtio_mmio virtio_mmio@a003e00
0c000000-0c004fff : c000000.tpm_tis tpm_tis@0
10000000-3efeffff : pcie@10000000
10000000-10003fff : 0000:00:01.0
10000000-10003fff : virtio-pci-modern
10004000-10007fff : 0000:00:02.0
10004000-10007fff : xhci-hcd
10008000-1000bfff : 0000:00:03.0
10008000-1000bfff : virtio-pci-modern
1000c000-1000cfff : 0000:00:01.0
1000d000-1000dfff : 0000:00:03.0
3f000000-3fffffff : PCI ECAM
40000000-4fffffff : System RAM
40008000-40ffffff : Kernel code
41200000-413c108f : Kernel data

After adding the reserved block of memory, you can check the reserved memory blocks in U-boot with bdinfo command:

=> bdinfo
boot_params = 0x00000000
DRAM bank   = 0x00000000
-> start    = 0x40000000
-> size     = 0x10000000
flashstart  = 0x00000000
flashsize   = 0x04000000
flashoffset = 0x000d7074
baudrate    = 115200 bps
relocaddr   = 0x4f722000
reloc off   = 0x4f722000
Build       = 32-bit
current eth = virtio-net#31
ethaddr     = 52:54:00:12:34:02
IP addr     = <NULL>
fdt_blob    = 0x4e6d9160
new_fdt     = 0x4e6d9160
fdt_size    = 0x00008d40
lmb_dump_all:
 memory.cnt = 0x1 / max = 0x10
 memory[0]      [0x40000000-0x4fffffff], 0x10000000 bytes flags: 0
 reserved.cnt = 0x2 / max = 0x10
 reserved[0]    [0x45000000-0x45005fff], 0x00006000 bytes flags: 4
 reserved[1]    [0x4d6d4000-0x4fffffff], 0x0292c000 bytes flags: 0
devicetree  = board
arch_number = 0x00000000
TLB addr    = 0x4fff0000
irq_sp      = 0x4e6d9150
sp start    = 0x4e6d9140
Early malloc usage: 2c0 / 2000

Once you’re done with the device tree, you can compile the source back into binary with the following command (this will print warnings, I guess the QEMU-generated device tree isn’t 100% perfect and my additions didn’t most likely help):

dtc -I dts -O dtb -o qemu.dtb qemu.dts

Booting the Device

That should be the hard part done. Since we have edited the devicetree and the modifications need to be present already in the U-Boot, QEMU can’t use the on-the-fly generated devicetree. Instead, we need to pass the self-compiled devicetree with the dtb option. The whole runqemu command looks like this:

BIOS=/<path>/<to>/u-boot.bin \
runqemu \
core-image-base nographic wic.qcow2 \
qemuparams="-chardev \
socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock \
-tpmdev emulator,id=tpm0,chardev=chrtpm \
-device tpm-tis-device,tpmdev=tpm0 \
-dtb /<path>/<to>/qemu.dtb"

Note that you need to source the Yocto build environment to have access to runqemu command. Also, remember to set up the swtpm TPM as instructed in the Yocto Emulation texts before booting up the system. You can use the same boot script that was used in the QEMU emulation texts.

Now, when the QEMU device boots, U-Boot will perform the measurements, store them into TPM PCRs, and the kernel is aware of this fabled measurement log. To read the event log in the Linux-land, you want to make sure that the securityfs is mounted. If not, you can mount it manually with:

mount -t securityfs securityfs /sys/kernel/security

If you face issues, make sure CONFIG_SECURITYFS is present in the kernel configuration. Once that is done, you should be able to read the event log with the following command:

tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements

This outputs the event log and the contents of the PCRs. You can also use tpm2_pcrread command to directly read the current values in the PCR registers. If you turn off the emulator and re-launch it, the hashes should stay the same. And if you make a small change to for example the U-Boot bootargs variable and boot the device, register 1 should have a different value.

The Limitations

Then, the bad news. Rebooting does not quite work as expected. If you reboot the device (as opposed to shutting QEMU down and re-starting it), the PCR values output by tpm2_pcrread change on subsequent boots even though they should always be the same. The binary_bios_measurements on the other hand stays the same after reboot even if the bootargs changes, indicating that it doesn’t get properly updated either.

From what I’ve understood, this happens because PCRs are supposed to be volatile, but the emulated TPM doesn’t really “reset” the “volatile” memory during reboot because the emulator doesn’t get powered off. With the actual hardware Raspberry Pi 4 TPM module this isn’t an issue, and tpm2_pcrread results are consistent between reboots and binary_bios_measurements gets updated on every boot as expected. It took me almost 6 months of banging my head on this virtual wall to figure out that this was most likely an emulation issue. Oh well.

Closing Words

Now we have (mostly) enabled measured boot to our example machine. Magnificient! There isn’t any attestation, though, so the measurement isn’t all that useful yet. The measurements could also be extended to the Linux side with IMA. These things will be addressed in future editions of Yocto hardening, so stay tuned!

While waiting for that, you can read the other Yocto hardening posts here!