Let’s continue encryption with the second part and move on to file system encryption. You can find the first part covering block device encryption from here. I recommend reading at least the first few sections about encryption and block device encryption because I won’t write again about the differences between block device encryption and file system encryption. Reading about the tools, and the actual block device encryption is not that important if you’re just interested in file system encryption. Let’s jump right into it, and talk about file system encryption.
About File System Encryption in Linux
The file system encryption in Linux can be handled in two ways: as a stacked file system, or a native file system. A stacked encryption file system operates on top of an existing file system. This upper layer encrypts and decrypts data on the fly as it is written and read from the lower layer file system. The lower layer file system sees “normal” files, but their content is scrambled. Native file system encryption on the other hand happens in the file system itself, without an upper layer. This means that the file system itself has to support encryption. At least Ext4, F2FS, UBIFS, and CephFS file systems support native encryption. I’m listing these because they are managed using fscrypt.
fscrypt is a Linux kernel library that enables encryption support for file systems. There is also a userspace tool called fscrypt that is used to manage the encrypted file systems. fscrypt does not mark files or their content as encrypted. Instead, a folder is marked as encrypted, and its contents will be encrypted. It’s worth noting that the only metadata that gets encrypted is the file name. Other file systems may support native encryption without fscrypt, like ZFS, so it’s good to be aware that there are other alternatives as well.
The native file system encryption is currently more popular than the stacked encryption. It’s more memory-efficient as there is no file system stack, and it does not (necessarily) require root permissions for setup. eCryptFs was a commonly used stacked encryption file system, but it’s currently not actively developed, so I’d advise against selecting it. Some other stacked encryption file system may make sense if your file system does not support native encryption. If you have chosen the file system first but cannot find any suitable file system encryption solution for it, you should consider block device encryption instead.
File System Encryption in Embedded Devices
File system-level encryption usually makes the most sense in situations where a device has multiple users, and it is not sensible to split the disk for each user and encrypt the split volumes. This kind of situation needs fine-grained control over encryption, where multiple keys may be used to lock/unlock multiple directories on a single file system. Something like this is not that common in embedded systems. Of course, this varies from system to system, but usually, there are not that many users in the system, and a situation where someone logs into the system is fairly uncommon (by system, I mean the Linux system, not for example a web control panel that has its own user management).
However, there may be situations where this kind of flexible encryption makes sense. For example, you could use block device encryption to ensure that the data in the storage media is encrypted when the device is turned off. You could then add additional file system encryption that would keep the secret directories encrypted as long as required when the device is running. This, combined with proper access controls, could help with keeping the data secret if someone manages to break into a live system. Also, if you’re using the unattended TPM2 encrypted block device unlocking, it may be a good idea to complement that with a more fine-grained file system encryption if the performance impact from double encryption is acceptable.
And finally, if for some reason block device encryption is not an option for you, then it is better to use file system encryption than no encryption at all. However, if you have to choose only one, usually it’s better to go with block device encryption. It encrypts the whole file system in one go, and with detached headers, it is difficult to guess how the file system is encrypted. If you need flexible key management, then file system encryption may make more sense as the only encryption method.
Hardware
The hardware used in the forthcoming file system encryption example is the same as in the block device encryption example: Raspberry Pi 4 with a LetsTrust TPM module. However, I’m not going to be utilizing TPM features this time. Ta-dah, I managed to write an even shorter hardware description this time.
Yocto
First, I added the meta-raspberrypi for the board support. I also added an extra partition to the default SD card image .wks file. Note the mkfs extra option that prepares the file system for encryption:
part /crypted --ondisk mmcblk0 --fstype=ext4 --mkfs-extraopts="-O encrypt" --label crypted --align 4096 --size 100To get the encryption working, we need the fscrypt userspace tool. The recipe for it is available in meta-security, but unfortunately, that recipe does not work. The commit hash in it does not exist for some reason. In addition to that, the compilation task attempts to fetch Go dependencies (which is a fatal error in Yocto), and the resulting binaries are always stripped, causing a QA failure. I had to do some patching to get the recipe working, but applying this should work:
The patch may not necessarily apply cleanly, but you can see the required changes from it. Note that this patch allows the fscrypt recipe to fetch the dependencies during the compilation step, which may be problematic for reproducible builds. However, the dependency versions in go.mod seem to be fixed, and other Go recipes seem to allow network access during compilation if they need to fetch something, so I guess it is acceptable.
Note that the recipe has a dependency on a DISTRO_FEATURE named pam. PAM stands for “pluggable authentication module”, and it’d be worth a blog post of its own. To summarize, it allows a plugin-based authentication process on Linux. PAM separates the authentication process from the applications, meaning that the applications call PAM API instead of doing the authentication themselves. This allows more flexible and configurable authentication. And as the name suggests, it’s plugin-based, meaning that custom plugins can be added to control the authentication.
fscrypt has this dependency because it provides a PAM module that allows automatic encryption unlocking when a user logs in. The recipe in meta-security does not install the module, but the aforementioned patch adds fscrypt-pam package that does. Note that the DISTRO_FEATURE dependency exists whether or not you install the extra package.
This time I won’t be adding initramfs, so the build is a lot simpler. Also, we won’t have to deal with TPM either, so all that’s required for file system encryption is a suitable partition, and the fscrypt package in IMAGE_INSTALL. Well, those and the CONFIG_FS_ENCRYPTION kernel configuration flag, but that should be enabled by default.
fscrypt Encryption
Now we can boot the image, and get encrypting. As with the block device encryption, the file system encryption needs to be initialized in a live system. The reason is the same as before: if we set up the encryption during an image build, all the devices flashed with the built image have identical encryption keys, which is not a good thing. It’s actually very bad. I’ve been thinking if it would be possible to automate these steps with package post-install scripts that run during the first boot, but I haven’t tried that yet. However, for demonstration purposes, it’s better to go through these commands manually.
First, we’ll need to run the setup to generate the global configuration file /etc/fscrypt.conf and the root fscrypt metadata directory /.fscrypt. Having this folder in the root does not necessarily mean that we will be encrypting anything in the root file system. The directory just contains global metadata for the encryption. The setup process will prompt if we want to allow other users than the current user to encrypt directories. For example, if you want to allow each user to set up their encrypted folders, you should allow this. If you prepare everything as root, this is not mandatory. The setup command is as follows:
fscrypt setupThen, we can create the mount point, mount our test partition, and set up the fscrypt metadata for that partition. This creates a .fscrypt directory to the root of the file system. The directory contains metadata to perform the encryption and decryption, and removing it will prevent decryption. So don’t remove it. The setup process will again prompt if we want to allow other users to encrypt directories, this time for this specific mount point.
mkdir /crypted
mount /dev/mmcblk0p3 /crypted
fscrypt setup /cryptedIf you didn’t use the encrypted option in the .wic file to prepare the partition for encryption, you can run tune2fs -O encrypt /dev/mmcblk0p3 to do it later.
Now that we have set up the partition for encryption, we can create the folder we want to encrypt and then encrypt it. Also, we should create a key that can be used as a protector. I’ll create a key file using dd, but using any file that is 32 bytes is possible. You could also use a passphrase instead of a key file, but then you’ll need someone to type in the passphrase.
mkdir /crypted/test_dir
# Note that the key needs to be stored in a secure place
# with strict access control
dd if=/dev/urandom of=/secret.key bs=1 count=32
fscrypt encrypt /crypted/test_dirThe fscrypt command asks for the protector type. I’ll use number three, as it is the key file protector:
The following protector sources are available:
1 - Your login passphrase (pam_passphrase)
2 - A custom passphrase (custom_passphrase)
3 - A raw 256-bit key (raw_key)
Enter the source number for the new protector [2 - custom_passphrase]:After defining the protector name and the key file location, the directory gets encrypted and automatically unlocked. To lock and unlock the directory, you can use these commands:
fscrypt lock /crypted/test_dir
fscrypt unlock /crypted/test_dir
# To unlock without having to manually input the key file:
fscrypt unlock /crypted/test_dir --key=/secret.keyYou can use fscrypt status to see information about the encrypted directory:
root@raspberrypi4-64:/# fscrypt status /crypted/test_dir
"/crypted/test_dir" is encrypted with fscrypt.
Policy: f22117a6747964bfa8d97b5bc7104124
Options: padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
Unlocked: Yes
Protected with 1 protector:
PROTECTOR LINKED DESCRIPTION
7bd1c4e5d45bd9c0 No raw key protector "keyfile"Of course, when using a key file protector, you have to take care of the confidentiality of the protector file. If you’re using a random suitable file on the file system as a key, you’ll have to ensure that the secondary use of the key “hidden in plain sight” won’t be revealed. If you’re using a purpose-made key file (like in this example), you should ensure that its content does not leak. It could be stored in a separate device, sent over a network (and shredded after use), or have its content sealed in a TPM (although, a passphrase may be more suitable in this case). Encrypting the key is also an option, but then you’ll have another key that needs to be secured.
Another thing to consider is the automatic unlocking at boot. You’ll need to write an init service that performs the unlocking if you want to achieve that. This should be a fairly trivial thing to do but may make it difficult to keep the location of the key file a secret. TPM could provide a good option for keeping the key content private. Or you could use the PAM module if there is a log-in performed during boot.
fscrypt PAM Module
Sometimes it may make sense to skip the passphrases and key files altogether and bind the encryption to a user login instead. As mentioned in the background chapters in the beginning, it isn’t that common to have user logins in the embedded systems. However, if that’s not the case for your system, it could make sense to utilize the fscrypt PAM module, and unlock a partition automatically when a user signs in.
Integrating the PAM module is quite easy. Instructions for it can be found on the fscrypt documentation page. It mostly consists of editing the common PAM configuration files and creating a configuration file for the fscrypt tool. The documentation lists the following modifications:
- Add
password optional pam_fscrypt.soafterpam_unix.soin/etc/pam.d/common-password - Add
auth optional pam_fscrypt.soafterpam_unix.soin/etc/pam.d/common-auth - Add
session optional pam_fscrypt.soafterpam_unix.soin/etc/pam.d/common-session, but before any module that may access the encrypted content (e.g.pam_systemd.so) - Add
auth required pam_unix.soto/etc/pam.d/fscrypt
I did these, but I made the mistake of adding pam_fscrypt.so right on the line after the pam_unix.so. This led to the authorization failing and me getting locked out of my system. Oops. The pam_fscrypt.so was supposed to be added after pam_unix.so indeed, but a bit later on in the PAM configuration files. The properly functioning spot was in the “Additional” block. In practice, at least in my sysvinit system, this was the very end of each PAM configuration file.
After adding those lines to the correct place, the automatic PAM unlocking worked nicely. To test this out, I added another user to the image. You can read more about adding extra users to Yocto images from my earlier blog post. The steps to perform the encryption are mostly the same:
# Run these as root
# This time allow other users to encrypt directories
fscrypt setup
mkdir /crypted
mount /dev/mmcblk0p3 /crypted
# Again, allow other users to encrypt directories
fscrypt setup /crypted
mkdir /crypted/test_dir
chown <OTHER_USER>:<OTHER_USER> /crypted/test_dirThen, switch to the other user, and run:
fscrypt encrypt /crypted/test_dirThis time, you’ll want to pick the PAM protector, which is the number one. fscrypt will then prompt for the login password of the user, and if you type it in correctly, the protector will get set up. fscrypt also creates a recovery passphrase that is stored in a generated file in the encrypted directory. You should read the generated file for information about the passphrase, and possibly remove the file and store the passphrase in a secure location. Or even disable the passphrase, but that carries a risk if something goes sideways in the PAM authentication.
After that, it should be possible to log out, log in as a different user (e.g. root), and see that the files are now encrypted. It’s worth noting that the file name is encrypted as well (but other metadata is not):
root@raspberrypi4-64:~# ls /crypted/test_dir/
rC4Rwunn9SEbx41-iuW_6r2IbWMVMbWsi5Urm2uypnNplWWp3LP6ww Then, you can log in again as the user that performed the encryption, and see that the files are now automatically decrypted:
raspberrypi4-64:~$ ls /crypted/test_dir/
fscrypt_recovery_readme.txtAfter you have finished setting up the encryption and aren’t expecting any more changes to it, you may want to lock the .fscrypt directories from non-root users:
chmod 0755 /.fscrypt/*
chmod 0755 /crypted/.fscrypt/*To reverse the operation, use 1777 mode.
Security Implications of fscrypt PAM Protector
This kind of PAM protector results in two security requirements:
- The user login password should be strong because it is an encryption passphrase as well
- Hashing of the passwords in the
/etc/shadowshould be strong
fscrypt documentation has instructions on how to configure the hashing with PAM. You’ll want to add rounds=1000000 to pam_unix.so module in /etc/pam.d/passwd (or some other amount of rounds that result in about one second of hashing). Then the password can be set to a secure one with passwd, and the value in /etc/shadow should get properly hashed.
Adding Multiple Protectors
There may be a situation where you’ll need multiple protectors for a single directory. Understanding how this happens requires a bit more knowledge of how the fscrypt works. Technically, a protector does not decrypt a directory, but a policy. The “policy key” is immutable, and it’s the key that is passed to the kernel for actual decrypting. Protectors on the other hand are mutable, and they are used to decrypt the policies. This allows flexible protector management without a need for re-encryption if a protector has to be changed. Both protectors and policies are stored in the .fscrypt metadata directory.
With this new knowledge, we can edit the fscrypt metadata so that we add a new protector to an existing policy. First, we should get the policy ID by reading the status of a directory. Then, we can use metadata subcommands to create a protector, and add it to a policy. Note how the mount point is required as an argument. This is because we need to specify what metadata is being edited in case there are multiple mount points:
# Get POLICY_ID from this command
fscrypt status /crypted/test_dir
# Take note of the generated PROTECTOR_ID
fscrypt metadata create protector /crypted
fscrypt metadata add-protector-to-policy --protector /crypted:<PROTECTOR_ID> --policy /crypted:<POLICY_ID>That’s all. The metadata subcommand can also be used to remove protectors from a policy, and even destroy metadata if so required.
Conclusion
This was a quite comprehensive walkthrough of the fscrypt. Its usefulness may be a bit more limited than dm-crypt, but it’s good to be aware of this alternative in case you ever end up needing extra encryption or some flexibility. This should cover the encryption topic of the Yocto hardening series, I hope you found it helpful, and the other Yocto hardening posts as well.
The other Yocto Hardening posts can be found here!
Recommended Reading
- Data-at-Rest encryption – Arch Wiki
- This recommended reading was in the block device encryption text as well, but it fits here too. And it’s a good read, so why not?
- fscrypt – Arch Wiki
- The mandatory Arch Wiki article on the topic at hand. Plenty of useful information, just don’t follow the instructions on how to enable the PAM module because they don’t apply.
- fscrypt tool README
- More useful information is here. And the PAM module instructions that should be used instead.
- Filesystem-level encryption (fscrypt) – The Linux Kernel documentation
- Kernel-side documentation of the
fscrypt. This may help to understand the big picture, but reading it in its entirety isn’t most likely useful (unless you’re implementingfscryptsupport for a filesystem)
- Kernel-side documentation of the
