About half a year ago, I started a fun little hobby project of building a hardened Yocto distro, Sulka. Somewhat surprisingly, it’s been going great so far! I thought that I’d share a bit of what’s happened over the past few months, and what is coming up next. If you’d like to give Sulka a try, you can start from the Kas repository and the quick start documentation.
What’s Been Happening
It’s been a busy six months. According to Codeberg, I have over 700 contributions from the last year, but that is a bit inflated number. I mean, one pull request can count up to four contributions: creating a branch, opening a pull request, merging a pull request, and deleting a branch all count as one contribution, while the result may be a single merged commit.
However, there are over 200 commits across all repositories, so there’s still some work that’s been done. I think it’s impossible to list everything, but I’ll try to list most of the features that are in the distro, not in the order of importance:
- Improved user management. In practice, this means locking root-user and allowing the distro maintainer to easily create an admin user for their image. One thing that is missing is easily enabling sudo for that service user; there’s a template configuration for that, but no easy way to enable it.
- Dangerous variable checking. Yocto has some debugging variables that are problematic from the security point of view, so I added checks for them. As a side note, I think handling of these debug settings and the defaults related to them has gotten better in Yocto as well lately.
- Firewall. The core network security feature, now added to the distro by default! Firewall is implemented with
nftables. And with a ruleset that drops everything. There are a few other templates as well, but by default, no traffic is allowed. It’s for the safety, you see. - Secure SSH configuration. This includes a hardened default configuration for OpenSSH, and disabled password logins because passwords are always used insecurely.
- Somewhat sane SELinux policy. As you can read from my previous blog post, sanity and SELinux do not always go hand in hand. However, by default, SELinux is enabled in Sulka, the targeted refpolicy is used, and it does not raise denial errors to the audit logs, so I consider it sane.
- Audit daemon with STIG ruleset. The daemon can be quite easily added to Yocto builds, but finding a good ruleset is not trivial. Audit package has some default rulesets, but none of those are installed by default. So, I chose to use the STIG ruleset and patch it for Sulka’s use.
- Some other miscellaneous hardening for the root file system. I’ve been constantly running Lynis and
oscapon the distro, fixing most of the issues they report. There are still some warnings, but I think the most glaring issues should be fixed now. - Hardened kernel. Basically, I ran
kernel-hardening-checkeron the defaultlinux-yoctokernel and fixed the issues using a kmeta. I also added a bbclass that can be used to enable the hardening configuration on other Linux recipes, and it can even be used independently of Sulka if so desired. - Kernel module signing. Useful for preventing unsigned code from being run in the kernel space, which is quite dangerous. Disabled by default at the moment, but this will most likely change in the future.
- Hardened U-Boot. I wrote a blog post about protecting the U-Boot command line, but in addition to that, there is command allowlisting, safe failure, disabled external environments, and a limited set of commands. I think this could be hardened further, though.
- Raspberry Pi reference project. The default target of Sulka is (at least at the moment)
qemux86-64virtual target, which is fine for development and testing, but it is not real hardware. As a physical reference target, I used a Raspberry Pi 4 and ported the distro to it. I’m aware of the RasPi’s security shortcomings, so I may change this into something else in the future, but it is the board I had available. - System tests. There are some tests for the distro. They are a bit lacking, but I’m hoping this will improve in the future, and I can get some proper confidence in them. This should allow keeping the development pace tolerable and quality reliable when I don’t have to manually check things and then fail to notice some weird issue until it pops up two weeks later in a completely unrelated situation.
- Jenkins server that runs the system tests. I spun up a small cloud server that runs the test set once a day, simple as that. I noticed at some point that I’m not running the tests myself, so the automation became mandatory.
- Documentation. This should be revisited soon. The content is still correct, but I’m not sure if the structure etc., makes sense anymore. However, it provides a quick start and a deeper developer guide that hopefully helps with understanding the distro.
- Website. I made a simple website and a server that hosts the documentation so that the interested people don’t have to go through
.rstfiles, hoping to find the information they’re looking for.
What’s Currently Happening
The 0.3.0 version was published a few weeks ago, so I’ve been working on 0.4.0. The most notable change that has already been completed is the removal of GPLv3-licensed code from the core distro configuration. The GPLv3 license’s installation clause can be problematic from a security point of view, as the manufacturer has to provide the means for the users to install their custom versions of GPLv3 licensed code on their devices, effectively preventing things like secure boot. Personally, I’m not against this clause, but I can see the problems that may arise from it, and I don’t think an embedded distro should force GPLv3-licensed code to its users.
Another thing that I’ve been doing is testing improvements. As mentioned, the tests are a bit lacking, so I’m trying to improve and automate them as much as possible. This should free up some time for the development when I have to stress a bit less during each pull request.
The biggest thing that I’m still considering is moving from sysvinit to systemd. I’m sure this will happen at some point. Originally, I was thinking the change would happen during the 1.0.0 release when updating the Yocto version to the next LTS version, but I’m starting to feel that the sooner this init manager change is done, the better. I’m currently maintaining two init managers, and it’s a bit of a wasted effort; and since I’ve decided that I’ll be moving to systemd, there really isn’t any reason not to do it.
What’s Happening Next
The interesting question is what will happen next. There are a few features I’d like to see in the near future.
1. Firmware Update Example
Now that the distro is ported to Raspberry Pi, it’d be a good idea to add a firmware update to it. After all, it is going to be a requirement for pretty much everyone due EU CRA requirements. However, the security features may cause some unexpected conflicts with the firmware updates, so it’d be a good idea to provide a reference implementation of a working update. This will most likely be quite a big task and cause some headaches.
2. Remote Logging Example
Another thing which is quite important from the compliance point-of-view is remote logging. This allows for detecting the exploited vulnerabilities. I have added syslog-ng to the distro for this purpose already, so creating a properly functioning example would be a good addition. This shouldn’t require much effort.
3. Update Yocto Meta-Layers to Wrynose
The distro is currently based on the Scarthgap version of Yocto, which will become EOL once the next LTS release, Wrynose, drops. So, needless to say, the distro should be updated to the new LTS. Shouldn’t be too difficult, but there may be some surprises that I’m not yet aware of.
4. Change Init Manager to Systemd
This is the thing that I mentioned earlier. Despite all my complaining about systemd, from the security point of view, it is a better choice for the init manager than sysvinit. Also, sysvinit has required a surprising amount of patching so far. I’ll most likely still keep the sysvinit scripts around, but supporting both init managers is really not feasible with current “resourcing” (=one guy working on the distro when he has a spare moment).
5. Read-Only Rootfs & Integrity
I have done quite a lot of work ensuring that the root file system and its contents are secure, but at the moment, the system is a read-write system. This in practice makes it quite easy to dismantle all the security features, even if they require superuser permissions. To counter this, the root file system should be read-only, preferably with proper integrity checking, and perhaps even with encryption.
Conclusion
It’s been quite an intensive half a year with this project. I’m hoping to finish the features listed above soon, after which I think most of the big pieces of the system are in place. Then there’s of course the continuous work of updating packages and applying hardening configurations, etc., but those are a lot simpler things to do than, for example, adding a firmware upgrade functionality.
All things considered, this whole thing has been extremely educational and useful (at least to me). I rarely work with greenfield projects, so it’s been interesting to kick off and manage everything from the beginning. There hasn’t been that much new technical information, though. I’ve mostly made choices I’m familiar with, and that I’ve utilised before in my work, so there haven’t been big revelations. Here’s hoping the next six months will go smoothly as well. I’ll write another status update then.
As a reminder, if you’d like to give Sulka a try, you can start from the Kas repository and the quick start documentation.
