Esa Jaaskela

Find all of the Yocto hardening texts from here!

Last time when we were talking about Yocto hardening we focused on setting up extra users to prevent unauthorized root access and to avoid situations where the users would have too open permission sets. Getting these right is a good low-hanging fruit, but there are plenty of other things to consider for hardening the system. One of the most important things to do is to eliminate (or at least minimize) the security vulnerabilities weakening the overall system defenses.

What are CVEs?

Despite what one may be inclined to think, open-source code is not always perfect. It may contain bugs, which is a bit annoying, but it may also contain vulnerabilities, which is potentially a bit dangerous. These can lead to denial of service attacks, data leaks, or even unauthorized accesses & system takeovers. Not good. Very bad. The usual stuff.

CVE (Common Vulnerabilities and Exposures) is a system maintained by The United States National Cybersecurity FFRDC and operated by Mitre Corporation. The system contains information about, well, vulnerabilities and exposures. “CVE” as a noun can also mean vulnerability or exposure. Or at least I often use the acronym in with that meaning. Checking your Yocto system against this database of vulnerabilities can help to detect security issues lurking in the open-source code components.

Checking vulnerabilities in Yocto

Every CVE doesn’t necessarily lead to unauthorised root access, and there may be vulnerabilities that don’t have a CVE identifier. However, scanning through the CVEs in your system should provide a fairly good image of how vulnerable your system is against possible CVE exploits. At least at the time of the scan, there are constantly new issues popping up which means you have to keep doing it often. Also it’s good to keep in mind that this kind of a scan does not check for other kind of weaknesses in the system. But at least performing the CVE check in the Yocto system is fairly simple. Something is simple in Yocto, what?

In all simplicity, just add this line to local.conf to check your packages & images for common vulnerabilities.

INHERIT += "cve-check"
# I had some issues with slow download speeds. If you get weird timeouts
# when fetching issue database, try adding this line as well:
CVE_SOCKET_TIMEOUT = "180"

Now when you run the build you’ll most likely end up with more warnings than before. Yocto’s CVE checker prints a warning to the build output for every vulnerability it detects, and depending on the complexity and age of your system your terminal may end up quite yellow. It’s more useful to inspect the CVE reports than to try and remember the build output as it scrolls by. These reports can be found in <build-folder>/tmp/deploy/cve folder, and we’ll inspect them a bit later. It’s worth knowing that this CVE checker may not be perfect, meaning that there may be some vulnerabilities lurking around that it does not detect. But it’s at least better than going through all the source code eyeballing it.

Analysing the results

Ok, now you’ve proved to yourself that your system has holes that need to be plugged. How to get over this uneasy feeling of constant doom looming over your system? The first thing is listing out all the CVE IDs and checking their status & severity to get a bit more stressed. Or relaxed, depends a bit on how well your system is maintained.

Further information for the CVEs can be found on MITRE’s CVE site, and some more complementary information can be found on NIST’s (National Institute of Standards and Technology) NVD (National Vulnerability Database). Plenty of acronyms. NVD is a system containing the same CVE IDs as MITRE’s site, but with severity scores and usually some extra information. Both are definitely useful, but for practical purposes NVD tends to be more useful. MITRE’s site can contain some new info not yet present in NVD, but NVD contains the information in a more easily digestible format.

Let’s take CVE-2023-22451 as an example. CVE listing for the vulnerability shows a short summary, affected versions, and a bunch of links. NVD entry has the same information, severity score, modification date, and a bit more information about the links so you won’t have to guess which link is the patch and which is some advisory (this can be useful on bigger issues where there are a dozen of links). For what it’s worth, Yocto’s CVE checker uses NVD’s database.

Now that we’ve gone through some background information, we can investigate the results of the CVE check. The aforementioned <build-folder>/tmp/deploy/cve contains two summary files: cve-summary and cve-summary.json. These contain all the detected issues, both patched and unpatched, and the format of the files isn’t all that intuitive for getting an actual summary, so I wrote a simple Python-tool for summarising the results. The design is very human:

# Replace the file path with your path
python3 ./cve-parser.py -f build/tmp/log/cve/cve-summary.json

Very easy to use. By default the parser will output the unpatched CVEs, and from each CVE it will print the package name, CVE ID, CVSS scores, and the link to the NVD site for more info. There is more information per issue available, the results can be sorted, and the ignored & patched issues can also be printed if needed. To get the complete list of options, run:

python3 ./cve-parser.py -h

I’m not going to document the whole program here in case I decide to change it at some point. If your boss asks you to get the total amount of unpatched CVEs in the system, you can run the command below to get “extra info” from the parser. Currently, “extra info” just contains the number of displayed issues, but if you’ve got ideas for it feel free to let me know:

# -u display unpatched issues (default behavior, but added just in case)
# -e displays extra info
python3 ./cve-parser.py -u -e -f build/tmp/log/cve/cve-summary.json

If your boss asks how many of them actually need to be fixed, use this to sort the CVEs by severity score and count how many have a score higher than 8.3

# -u display unpatched issues
# -ss2 sort by CVSSv2 score
python3 ./cve-parser.py -u -ss2 -f build/tmp/log/cve/cve-summary.json

Just kidding, please fix them all.

Fixing the CVEs

After you’ve justified your fears of getting hacked by realizing that your system has seven CVEs with a CVSS score higher than 9.5 (three of them in the kernel) and a few dozen more with just a slightly lower priority, it’s time to ask the big question: what can I do? What can I possibly do? This didn’t help with the feeling of constant doom at all.

The first step is going through the list of unpatched issues. There tend to be quite a few false positives that are not applicable to the current system. For example, a brand new Poky build reports an unpatched CVE-2017-6264 that’s been around six years already. This is a vulnerability in the NVIDIA GPU driver, and it’s applicable to Android products. Most certainly it’s a false positive in your Yocto system, but since the bad code is present in the source code, it’s reported as unpatched. You can ignore these false positives by adding the line below somewhere to your build configuration (local.conf works, but it’s maybe not the best place for it):

# The format is CVE_CHECK_IGNORE="<cve-id>", e.g.
CVE_CHECK_IGNORE="CVE-2017-6264"
# In the older Yocto versions this was called CVE_CHECK_WHITELIST
# so if ignore doesn't work, try the old whitelist variable

After ignoring the false positives fixing the rest of the CVE issues is easy, in theory. Just keep the Yocto at its newest (or relatively new LTS) version. This should keep the packages fresh. Sounds simple, but unfortunately this process tends to cause a big headache with incompatibilities and such. However, that’s the best advice I can give.

If there is a need to update the recipes even further than officially supported, or if you want to update a single recipe, copy the recipe in question to your own meta-layer, update the recipe version in the filename and fix SRCREV to match. When hacking around like this remember to hope that nothing falls apart in the system even more than during a regular Yocto version update.

Sometimes, updating the packages is not an option. Perhaps updating the package breaks the build, a newer version of the library isn’t compatible with your application, or there is some other Perfectly Good Reason you’re not allowed to do that. That’s when it’s time to do some patching to make the confusing build system even more spaghettified.

Basically, this approach consists of heading to the NVD entry for the CVE, checking if there’s a patch for the vulnerability, and if there is, porting the patch to the Yocto build. For example, the aforementioned CVE-2023-22451 mentions this commit as its patch in the NVD entry. Copy the contents of the commit to a patch file, create bbappend file for the recipe and add the patch to the recipe with that bbappend.

If there’s no patch you can wait. Or if you’re like a kid on Christmas and can’t wait, you can try digging the package’s git repositories for the corrective commit before it’s been released. It ain’t fun, and rarely it’s a productive use of time, but every now and then some fixes can be found like this before they are made official.

That’s a short explanation of how you can check and fix the CVE vulnerabilities in your system. The theory of the process is fairly simple. However, it tends to get a bit more complicated in the actual world, especially when trying to update older legacy systems where the stale, non-updated packages contain more patches than original code. But I guess that’s the problem with real life, it tends to mess up good theories.

You can find the next part about Yocto hardening here. It’s about firewalls.

Find all of the Yocto hardening texts from here!

Cybersecurity. The never-ending race between you trying to secure your precious IoT device and some propeller head who’s finding the wildest exploits and destroying your system just in the time between breakfast and lunch (although true hackers work during night, not in the mornings). Or perhaps you forgot to update your WordPress plugins, and now your fantastic development blog is hacked. Or perhaps you’ve just been postponing that security update on your Android phone for six months. We all have some experience with cybersecurity.

Usually, there are some simple things to do to prevent the worst catastrophes. Update software, make sure the run-time environment is sane & secure, and encrypt the secrets to mention a few. In this new blog series, I try to explain how to make your Yocto systems more secure by adding some common-sense security to the system. All of these ideas are familiar from Linux desktop & server environments, but I’m going to show how to apply them to Yocto builds. So, unfortunately, I’m not actually going to cover how to keep a fantastic WordPress site secure.

As usual with any security advice, it’s good to be slightly skeptical when reading it. This means that some rando writing blog posts on their blog may not be a 100% trustworthy or up-to-date source on everything. Also, those randos will take no responsibility for the results of their advice. However, their advice may provide useful guidance and general tips that you can apply to your system’s security hardening plan. You do have a security plan, right?

So, the first topic of this Yocto hardening exercise is users. As you may have heard, running and doing everything as the root user is generally considered a Bad Idea. Having services running under the root user unnecessarily can result in root user executing arbitrary code using the most imaginative methods. Or, if the only user available for logging in is the root user you’re not only giving away root permissions to malicious users, but to incompetent users as well.

This text will assume intermediate knowledge of Yocto, meaning that I won’t explain every step in depth along the way. Hopefully, you know what an append-file and local.conf are. The code and config snippets were originally written for Yocto Kirkstone checked out “at some point after 4.0.3 release”, and later re-tested with version 4.0.15.

Before we get started with the actual hardening, there’s one preliminary task to do. Ensure that you don’t have DEBUG_TWEAKS in either IMAGE_FEATURES or EXTRA_IMAGE_FEATURES. Not only is it unsafe as it allows root login without a password, and a bunch of other debug stuff, but it also makes some examples shown here behave in unexpected ways.

Creating non-root users

Here is a code snippet that creates a service user for the system. This user can be logged in with, and they have sudo capabilities (if sudo-package is installed). Insert this code either into a image recipe or a configuration file:

# If adding to configuration file, use INHERIT += "extrausers" instead.
inherit extrausers

IMAGE_INSTALL:append = " sudo"

# This password is generated with `openssl passwd -6 password`, 
# where -6 stands for SHA-512 hashing alorithgm
# The resulting string is in format $<ALGORITHM_ID>$<SALT>$<PASSWORD_HASH>,
# the dollar signs have been escaped
# This'll allow user to login with the least secure password there is, "password" (without quotes)
PASSWD = "\$6\$vRcGS0O8nEeug1zJ\$YnRLFm/w1y/JtgGOQRTfm57c1.QVSZfbJEHzzLUAFmwcf6N72tDQ7xlsmhEF.3JdVL9iz75DVnmmtxVnNIFvp0"

# This creates a user with name serviceuser and UID 1200. 
# The password is stored in the aforementioned PASSWD variable
# and home-folder is /home/serviceuser, and the login-shell is set as sh.
# Finally, this user is added to the sudo-group.
EXTRA_USERS_PARAMS:append = "\
    useradd -u 1200 -d /home/serviceuser -s /bin/sh -p '${PASSWD}' serviceuser; \
    usermod -a -G sudo serviceuser; \
    "

The extrausers class allows creating additional users to the image. This is done simply by defining the desired commands to create & configure users in the EXTRA_USERS_PARAMS variable. As a side note, it’s good to have static UIDs for the users as this will make the builds more reproducible.

In a perfect world, every custom service you create for your system would run under a non-root user. When writing a recipe that creates a daemon or other kind of service, you can use the snippet below to add a new user in a non-image recipe:

inherit useradd
USERADD_PACKAGES = "${PN}"
GROUPADD_PARAM:${PN} = "--system systemuser"
# This creates a non-root user that cannot be logged in as
USERADD_PARAM:${PN} = "--system -s /sbin/nologin -g systemuser systemuser"

As you can see, adding a new user in a non-image recipe is slightly different than in the image recipe. This time we’re only giving parameters to useradd and groupadd commands. After adding this, you should be able to start your daemon as the systemuser in a startup script. If you need root permissions for some functionality in your service but don’t want the service to be run as root, I’d recommend reading into capabilities. It’s worth noting that when adding users in a service recipe like this, the additions are done per-package basis, not per-recipe basis. This means that you can create and add new users in a quite flexible manner.

Editing sudoers configuration

By default, when adding the sudo-package to the build it doesn’t do much. It doesn’t actually allow anything special to be done by the users, even if they are in the sudoers group. That’s why we need to edit the sudo-configuration. There are two ways of doing this, either by editing the default sudoers file or by adding drop-in sudoers.d configurations. First, editing the sudoers file (which is the worse method in my opinion).

There are two ways of doing this, and I’m not 100% sure which one is less bad, so I’ll leave it to you to decide. The first option is adding a ROOTFS_POSTPROCESS_COMMAND to the image recipe:

enable_sudo_group() {
    # This magic looking sed will uncomment the following line from sudoers:
    #   %sudo   ALL=(ALL:ALL) ALL
    sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/'  ${IMAGE_ROOTFS}/etc/sudoers
}

ROOTFS_POSTPROCESS_COMMAND += "enable_sudo_group;"

The second option is creating a bbappend file for the sudo recipe, and adding something like this there:

do_install:append() {
    # Effectively the same magic sed command
    sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/'  ${D}/${sysconfdir}/sudoers
}

Both do the same thing, and both are hacky. The correct choice really depends on if you want to edit the sudoers file in a different way for each image, or if you want the sudo configuration to depend on something else. You can also of course supply your own sudoers file instead of sed-editing it, and that can be a lot more maintainable way of doing this if you have more than just one change.

However, a more flexible way of configuring sudo is with the sudoers.d drop-in files. It also involves a lot fewer cryptic sed commands. Let’s assume that you have a recipe that creates a new user lsuser (not to be confused with a loser), and you want that user to have sudo rights just for ls-command (considering rights given to this user, they may actually be a loser). For this purpose you need to create a bbappend for sudo, and add something like this snippet to create a drop-in configuration:

do_install:append() {
    echo "lsuser ALL= /bin/ls " > ${D}${sysconfdir}/sudoers.d/lsuser
}

FILES_${PN} += " ${sysconfdir}/sudoers.d/lsuser"

Again, if your drop-in configuration is more complex than one line, you can provide it as a configuration file through SRC_URI and install that in your image instead of echoing configuration lines. I recommend reading sudo-configuration documentation for more info on how to precisely set the permissions because let’s be honest, these two examples aren’t the shining examples of sudo configuration.

The downside of this approach is that you need to add the drop-in configuration in a sudo bbappend because /etc/sudoers.d will conflict when creating the root filesystem if you add the drop-in in another recipe. This means that when you’re creating a user in a non-sudo recipe and add the drop-in conf in the sudo recipe you’ll have the user creation & configuration handled in two different places, which is perfect for forgetting to maintain things.

Disabling root-login

Disabling root login is useful for a multitude of reasons.

1. You usually want to make sure that no user has the full command set available
2. You also want that the users’ privileged actions to end up logged in auth.log when they use sudo
3. In general, it’s “a bit of a risk” to allow users to log in as root

Disabling the root user can be achieved in multiple ways, and I’m going to cover three of them here: fully locking the root account, removing the login shell, and disabling the SSH login. In theory, locking the root account should be enough, assuming non-root users cannot unlock it, but I’ll present all the methods as they may be useful for different situations.

First, disabling the root account. This is a method I found from this StackOverflow answer, kudos there. The simple snippet below pasted in the image recipe should lock and expire the root account:

inherit extrausers
EXTRA_USERS_PARAMS:append = " usermod -L -e 1 root; "

Then, removing the login shell. Or more like setting the login shell to nologin. This isn’t strictly required after locking the root account but can be achieved by creating a bbappend shown below to base-passwd recipe:

do_install:append() {
    # What this magic sed does:
    # "In the line beginning with root
    # replace /bin/sh with /sbin/nologin
    # in file passwd.master"
    # (passwd file gets generated from this template during install)
    sed -i '/^root/ s/\/bin\/sh/\/sbin\/nologin/' ${D}${datadir}/base-passwd/passwd.master
    # sry for the picket fence

    # I think it's peak sed that I need a four line
    # comment to explain a one-liner
}

And finally, disabling the SSH login. You should always disable the root login over remote networks. If you need, you can just disable the SSH login for root and still allow other service users to log in as root. This is still a better option than doing nothing, as this will prevent logging in as root remotely and attackers would need to know two passwords to gain root access (one for the service user, and the other one for the root user).

Disabling the root login varies a bit between the SSH servers. Dropbear disables root login by default with -w parameter in DROPBEAR_EXTRA_ARGS variable located in the /etc/defaults/dropbear file (note that if you have debug-tweaks enabled, the file actually contains -B, allowing root-login). You can overwrite the file with your own dropbear.default file during the build if you want to add more options.

Similarly, OpenSSH-server disables root logins with passwords if debug-tweaks is removed from the IMAGE_FEATURES (and allows them if it’s present). This is achieved by sed-editing SSH configuration files. If you want to see how this is exactly done, check ssh_allow_root_login function in meta/classes/rootfs-postcommands.bbclass (part of poky).

However, it’s worth noting that this default behaviour doesn’t prevent public key authentication. If you want to disable that as well, you can add this function as a rootfs post-process task to the image recipe. And if needed, it could of course be modified to work in a bbappend as well.

# The function here searches sshd_config and sshd_config_readonly files for a
# commented line containing PermitRootLogin, and replaces it with "PermitRootLogin
# no" to prevent any sort of root login.

disable_rootlogin() {
    for config in sshd_config sshd_config_readonly; do
        if [ -e ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config ]; then
            sed -i 's/^[#[:space:]]*PermitRootLogin.*/PermitRootLogin no/' ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config
        fi
    done
}

ROOTFS_POSTPROCESS_COMMAND += "disable_rootlogin;"

The thing to note about disabling root login is that too lenient sudo- or filesystem-permissions for non-root users can make the whole thing useless. For example, if a service user has sudo access to passwd command they can easily unlock the root account. If a service user has write permissions to /etc they can set the root’s login shell and edit SSH configuration. And finally, disabling root login like shown in this text does nothing to prevent shell escapes made possible by incorrectly set sudo-permissions.

That’s it for this time! Hopefully, these tips helped you to achieve the minimum security in your Yocto images. Do these code snippets make your system unhackable? Not even close. Do they make it slightly more inconvenient? I hope so (but no guarantees for that either). But at least now you can show your Yocto image to an independent professional security consultant and say that you tried before they tear the whole system to pieces.

You can find the second part of the Yocto hardening series here. It’s about fixing CVEs.