Check out the previous part of the Aioli Audiostreamer saga here. In case you don’t want to check it out, here’s a quick recap: I started a new project in which the goal is to stream audio from one Raspberry Pi to another over an IP network. The last time we got the streaming to work in theory, but the practical part of it was (and is still) missing. This time we’re not going to to address that.
Instead of focusing on getting the streaming working robustly and automatically, I chose to add a Bluetooth connection between the Raspberry Pi controller device and an external audio source. This way the system can stream something else than just the audio files present in the controller device. Here’s the graph with a chunky red line showing what’s the focus for today:
Unfortunately, this means confronting my old nemesis: BlueZ stack. Or Bluetooth in general. Something about it rubs me the wrong way. I’m not sure if it’s actually that bad. However, every time my headphones fail to connect to my phone I curse the whole protocol to the ninth circle of hell. Which happens every single morning. And it’s still the best choice for this kind of project. But yeah, plenty of that coming up.
Bluetooth Connectivity
The first step is making our Raspberry Pi audio server advertise itself as a Bluetooth device wanting to receive audio: headphones, speakers, or anything along those lines. In theory, this sounds like a lot of work, but once again, the open-source community comes to the rescue. This bt-speaker project makes a Raspberry Pi act as a Bluetooth speaker, which is exactly what we want. The phone (or some other audio source) can connect to the bt-speaker daemon running on Raspberry Pi and stream audio to it. bt-speaker then outputs the received audio to the desired audio device.
The program required some tweaking for cross-compilation, and some things weren’t quite as generic as they could have been, causing some QA errors in Yocto. However, it mostly worked quite nicely out of the box. I guess because the bulk of the program is written in Python there are not that many compilation issues to wrestle with. There was also one codec that needed to be compiled, and then there was the issue of figuring out the correct dependencies, but all in all fairly simple stuff. Yocto recipe can be found here.
The Actual Troubles
What actually took a long time was getting the start-up script working. I’m still sticking to SysVinit for simplicity, which means that I ended up using start-stop-daemon to launch the program. However, it turned out Busybox’s implementation of start-stop-daemon was missing the -d/--chdir option. bt-speaker loads a codec from a relative path, meaning that the program fails at start-up because it’s launched from the root directory. Because I’m not much of a Python programmer, I chose to patch the feature into Busybox instead of doing the sensible thing and installing the codec into the correct location and fixing bt-speaker. An open-source contribution to Busybox coming soon I hope.
Well, after that came the second problem: the Bluetooth chip in Raspberry Pi wasn’t stable during the startup. The script starting BlueZ worked well, and the bt-speaker launched successfully as well. However, after a few seconds, the Bluetooth device became undiscoverable. I tried to check all the Bluetooth-related changes that happened during boot in the system: changes in the Bluetooth device information and BlueZ status, reading syslog & dmesg, but no. The Bluetooth chip just reset a few seconds after the BlueZ launched. So I did what any sane person would do: power cycled BT chip as a part of the start-up, added a “reasonable amount” of sleep, and moved on with my life.
The Less Actual Troubles
After getting the thing starting automatically during the boot there’s still a small problem. The problems never end with Bluetooth, don’t they? Well, even better, there are two problems. First, for some reason, my phone says that it has trouble connecting to this Frankensteinian BlueZ device. Streaming music from Spotify works nicely though, and even the volume control behaves as expected. So all in all, this sounds like a very typical Bluetooth device already: nothing works, except that it works, except when it doesn’t. I’m not yet sure if this is actually a problem or not to anyone else except my phone.
I just wanted to flex my phone and watch with this screenshot. The ironic thing here is that if I “turn device off & back on” as suggested, it’ll be forever unable to connect again unless the BlueZ cache is cleared on Raspberry Pi. That may be the fault of dodgy Bluetooth code and not the protocol itself though.
The bigger issue is that we don’t want the audio to be output from the Raspberry Pi we’re connected to. Instead, we want to stream the audio to the other Raspberry Pis in LAN and have those output the audio. GStreamer has an alsasrc source that can take input from an ALSA device and work with that. However, we have a bit of a mismatch here: GStreamer wants an input device to receive the audio from (e.g. microphone), but bt-speaker generates audio that goes to an output device (e.g. speaker).
Loop Devices to the Rescue!
Loop device is a virtual audio device that redirects audio from a virtual output device to a corresponding virtual input device (or vice versa). This means that we can have a virtual “microphone” that outputs the audio that bt-speaker has been received through Bluetooth. Maybe my explanation just made it worse, the idea is quite simple. This blog post explains the functionality quite well.
To explain more: probing snd-aloop module creates two loopback sound cards, both for input and output (four cards in total). The virtual cards have two devices, and each device has eight subdevices. This results in a lot of devices being created. These devices are special because the output of an output device gets directed to the input of a corresponding input device. For example, if I play music with aplay to card 1, device 0, subdevice 0, the same music can be captured from the input card 1, device 1, subdevice 0. Notice how the device number is flipped. Output to input, and vice versa, as I’ve been repeating myself for two chapters now.
Google Translate don’t fail me now.
With this method, when bt-speaker uses aplay to output the audio it receives, we can define the output device to be a loopback device. Then, GStreamer can use the corresponding loopback input device to receive the Bluetooth audio and pass it to the LAN. A bit of patching to the bt-speaker, and something like this seems to do the trick:
# Play command that bt-speaker uses when it receives audio
aplay -D hw:2,0,1 -f cd -
# Streaming command to send audio to 192.168.1.182
gst-launch-1.0 alsasrc device=hw:2,1,1 ! audioconvert ! audioresample ! rtpL24pay ! udpsink host=192.168.1.182 port=5001
Is This a Good Idea?
I’m not sure. I think it would be possible for the bt-speaker daemon to launch the GStreamer directly once the Bluetooth connection has been initialized. This approach would skip aplay altogether and wouldn’t require the loop devices. However, keeping the Bluetooth and networking separate should keep the system simpler. Both processes do their own thing without knowledge of each other. bt-speaker can output audio without caring if anyone listens to it. On the other end, GStreamer can stream whatever it happens to receive through the loopback device.
This also allows kicking the bt-speaker and GStreamer individually when they eventually and inevitably start misbehaving. The drawback is that GStreamer streams silence if nothing is received from Bluetooth, but I think that’s an acceptable weakness for now. After all, I’ve paid for a WLAN router to route some bits, so I’m going to route them bits, even if they’re all zeros.
I noticed that there actually is a BlueZ plugin for GStreamer. However, it’s labelled as one of the “bad” plugins, and dabbling with such dark magic sounds like a bad idea. If something is labelled “bad” even in the official documentation it’s usually better to avoid it. It doesn’t necessarily mean that the quality itself is bad as the label may also mean a lack of testing or maintenance, but still.
TBH I don’t exactly know what WASAPI is, but after a quick Google search, I’m not sure I even want to know.
Closing Words
This text was a bit shorter than anticipated, but some things in life are unexpectedly short. Next time we’ll get rid of the static WLAN configuration. Instead, we’ll create a mechanism for passing the SSID and password during run-time. We’ll be doing that mostly because I already started working on that feature. Now I have DHCP servers running amok in my home LAN causing trouble and slowing down development.
One question still remains: does this system contain any code that I have written? Not really at the moment. But in my experience, that’s the story for most of the embedded Linux projects: find half a dozen somewhat working pieces of software and glue them together with some scripts. Until next time!
This blog text is a part of my Movember 2023 series. If you found this text useful, I’d ask you to do “something good”. That doesn’t necessarily mean shoving your money to charities or volunteering weekends away (although those are good ideas), it can be something as simple as asking a family member or colleague how they’re doing. Or it can mean selling your earthly possessions away and becoming a monk. It’s really up to you.
People often ask me two things. The first question is “Why did you choose to write this firewall text in two parts?”. The answer to that is I actually started writing this over a year ago, the scope of the text swelled like crazy, and in the end, to get something published I chose to write the thing in two parts to have a better focus. The second question that I get asked is “Do you often have imaginary discussions with imaginary people in your head?”. To that, the answer is yes.
I prepare for daily stand-up by mentally going through conversations where I get deservedly fired because I didn’t close three tickets the previous day.
But without further ado, let’s continue where we left off in part 1. As promised, let’s take a look at another way of setting up and configuring a firewall: firewalld.
One Step Forward, Two Steps Back: Configuring Kernel (Again)
To get the firewalld running we need a few more kernel configuration items enabled. How many? Well, I spent a few days trying to figure out the minimal possible configuration to run some simple firewalld commands, all without success. firewalld is not too helpful with its error messages, as it basically says “something is missing” without really specifying what that special something is.
I’ve always liked Lionel’s ‘stash, but I feel like he’s looking extra fine in this pic.
After banging my head on a wall for a few days I attempted to enable every single Netfilter and nftables configuration item (because firewalld uses nftables), but no success. After some further searching, I found this blog post that contained a kernel configuration that had worked for the author of the blog. I combined that with my desperate efforts, and the resulting config actually worked! In the end, my full configuration fragment looked like this:
As you can see, my shotgun approach to configuration wasn’t 100% accurate because there were few NF and NFT configuration items I missed. Is everything here absolutely necessary? Most likely not. There actually are some warnings about unused config options for the Linux kernel version 5.15, but I’m afraid to touch the configuration at this point.
I spent some more time optimizing this and trying to make the “minimal config”, just to realize that I’m actually minimizing something that can’t be really minimized. The “minimal config” really depends on what your actual firewall configuration contains. After this realization, I decided that it’s better to have too much than too little, and found an inner peace. You can think that the fragment provides a starting point from which you can start to remove useless-sounding options if you feel like it.
While we’re still talking about the build side of things, remember to add the kernel-modules to your IMAGE_INSTALL as instructed in part 1. Or if you chose to go through the RRECOMMENDS route, remember to add all 36 modules to the dependencies.
To be fair, I’ve been in a situation where a few KBs of root file system content have made the difference between a success and a catastrophic failure.
The Hard Firewall – firewalld
Now that I’ve spent three chapters basically apologizing for my Linux config, it’s time to get to the actual firewall stuff. The “hard” way of setting up a firewall consists of setting up firewalld. This is in my opinion suitable if you have multiple interfaces with changing configurations, possibly edited by a human user, meaning that things change (and usually in a more complex direction).
firewalld is a firewall management tool designed for Linux distributions, providing a dynamic interface for network security (another sentence from ChatGPT, thank you AI overlords). Or to put it into more understandable words, it’s a front-end for nftables providing a D-Bus interface for configuring the firewall. The nice thing about firewalld is that it operates in zones & services, allowing different network interfaces to operate in different network zones with different available services, all of which can be edited run-time using D-Bus, creating mind-boggling, evolving, and Lovecraftian configurations. This is the nice thing, so you can imagine what the not-so-nice things are.
Now that we are somewhat aware of what we’re getting into, we can add firewalld to IMAGE_INSTALL:
IMAGE_INSTALL:append = " firewalld"
One thing worth noting is that firewalld is not part of the Poky repository, but it comes as a part of meta-openembedded. Most likely this is not a problem, because every project I’ve worked on has meta-openembedded, but it’s worth knowing nevertheless.
Now that everything is in place we can start hammering the firewalld to the desired shape. To edit the configuration we can use the firewall-cmd shipped with firewalld package. This firewall-cmd (among some other firewall-* tools) uses the D-Bus interface to command the firewalld, which in turn commands nftables, which finally sets the Netfilter in the kernel. This picture illustrates it all nicely:
So, about the actual configuration process: each network interface can be assigned a zone, and the zones can enable a certain set of services. Here’s a short command reference for how to add a custom zone, add a custom service, add the new service to the new zone, and set the new zone to a network interface:
# List available zones
firewall-cmd --get-zones
# These zones are also listed in /usr/lib/firewalld/zones/
# and /etc/firewalld/zones/
# Get the default zone with the following command
# (This is usually public zone)
firewall-cmd --get-default-zone
# This'll return "no zone" if nothing is set,
# meaning that the default zone will be used
firewall-cmd --get-zone-of-interface=eth0
# Create a custom zone blocking everything
firewall-cmd --permanent --new-zone=test_zone
firewall-cmd --permanent --zone=test_zone --set-target=DROP
# If permanent option is not used, changes will be lost on reload.
# This applies to pretty much all of the commands
# Reload the firewalld to make the new zone visible
firewall-cmd --reload
#List services, and add a custom service
firewall-cmd --list-services
firewall-cmd --permanent --new-service=my-custom-service
firewall-cmd --permanent --service=my-custom-service --add-port=22/tcp
firewall-cmd --reload
# Add the service to the zone
firewall-cmd --permanent --zone=test_zone --add-service=my-custom-service
# Set zone
firewall-cmd --zone=home --change-interface=eth0
# Add --permanent flag to make the change, well, permanent
You can (and should) check between the commands how the ports appear to the outside world using nmap to get a bit better grasp on how the different commands actually work. The changes are not always as immediate and intuitive as one would hope. Good rule of thumb is that if the settings survive --reload, they’re set up properly.
How does this all work with the file-based configuration and Yocto? You could create the zone and service files for your system using the firewall-cmd, install them during build time, and change the default zone in the configuration file /etc/firewalld/firewalld.conf to have the rules active by default. This is similar to nftables approach, but is a bit easier to customize for example from a graphical user interface or scripts, and doesn’t require custom start-up scripts. An example of adding some zones and services, and a patch setting the default zone, can be found in the meta-firewall-examples repo created for this blog text.
Comparison and Closing Words
So, now we have two options for firewalling. Which one should be used? Well, I’d pick nftables, use it as long as possible, and move to firewalld if it would be required later on. This is suitable when the target is quite simple and lightweight, at least at the beginning of the lifespan of a device. On the other hand, if it’s already known from the get-go that the device is going to be a full-fledged configurable router, it’d be better to pick the firewalld right off the bat.
What about iptables and ufw then? Both have bitbake recipes and can definitely be installed, and I suppose they even work. iptables is a bit of a legacy software these days, so maybe avoid that one though. It should be possible to use iptables syntax with nftables backend if that’s what you want. ufw in Yocto uses iptables by default, but it should be possible to use nftables as the backend for ufw. So as usual with the open-source projects, everything is possible with a bit of effort. And as you can guess from the amount of weasel words in this chapter, I don’t really know these two that well, I just took a quick glance at their recipes.
Note that the strength of the firewall depends also on how protected the rulesets are. If the rules are in a plaintext file that everyone can write to, it’s safe to assume that everyone will do so at some point. So give a thought to who has the access and ability to edit the rules. Also, it may be worthwhile to consider some tampering detection for the rule files, but something like that is worth a text of its own.
In closing, I hope this helped you to set up your firewall. There are plenty of ways to set it up, this two-parter provides two options. It’s important to check with external tooling that the firewall actually works as it should and there are no unwanted surprises. Depending on the type of device you want to configure you can choose the simple way or the configurable way. I’d recommend sticking to the easy solutions if they’re possible. And finally, these are just my suggestions. I don’t claim to know anything, except that I know nothing, so you should check your system’s overall security with someone who’s an actual pro.
The eternal task of making the Yocto Linux build an impenetrable fortress continues. Next, we’ll look into setting up a firewall two different ways: the easy way with nftables, and in the part two the easily configurable way with firewalld. I must warn you beforehand that this text contains a bit of resentment towards kernel configuration. I also used ChatGPT as a tool for writing a part of this text, but I’ll inform you of the section I asked it to write.
So yeah, firewall. The system keeping the barbarians out of our Linux empire. Also, one of the most bad-ass names for a program, at least if taken literally. Reality is a bit less exciting, turns out “firewall” is some technical construction term. In the magical world of computers, a firewall is a piece of software used to allow or deny network traffic coming in and out of a machine. It’s mighty useful for improving the overall security of the system to be able to prevent unwanted remote connections and such.
Step 0 – Configuring Kernel
The first step of installing and configuring a firewall on a Yocto image is ensuring that the kernel is configured correctly. What, do you think you can just start configuring the firewall and blocking your nefarious enemies just like that? Pfft, get out with that casual attitude (please don’t).
To get some basic traffic filtering with nftables done, we need to ensure that the Netfilter is enabled in the kernel, as well as the Netfilter support for nftables, and that some nftables modules are installed as well. It’s all a bit confusing, but to summarize: Netfilter is the kernel framework for packet filtering and nftables is the front-end for Netfilter that can be used from user space. nftables can then be built with or without modules to extend its functionality. The following kernel configuration fragment should do the trick:
Getting this config fragment working was where things got problematic for me. I thought that I’d just slap =y to everything and be done with it, no need to install and load modules. Well, the configuration process of the kernel consists of merging the defconfig, kernel features, and configuration fragments, and as it turns out, the kernel feature enabling the Netfilter overrode some parts of my config fragment, changing =y to =m. For some reason bitbake didn’t give a warning about this. I’m quite sure it used to do.
But where is this mythical Netfilter kernel feature located? The fragment above enables only nftables related things, how to get the Netfilter working? Well, Kirkstone release (3.1.27) of Poky has this line that enables the feature by default in the kernel, and the related configuration fragment looks like this. The multiple configs lead to funny situations where you try to remember the difference between CONFIG_NF_CONNTRACK and CONFIG_NFT_CT and where they are defined and to what value.
Ensure that you’re also installing the kernel modules to the image by adding the following line somewhere in the image configuration:
IMAGE_INSTALL:append = " kernel-modules"
No use enabling the modules if they’re not installed, and by default nothing installs them, so they won’t get added to the final image. At least to the qemu64 core-image-minimal image they won’t be installed by default, guess if I found that out the hard way as well. If you need more fine-grained control over the modules that get installed, you can add them as a runtime recommendation to some package:
Most likely nftables package is the best place for this. However, I’d advise against this approach if possible, because there will be a lot of modules later on. If you choose to go down this route, the naming format of the modules is fortunately quite self-explanatory. CONFIG_NF_TABLES being built as a module generates kernel-module-nf-tables package, CONFIG_NFT_CT results in kernel-module-nft-ct being built etc, so this should be easily scriptable (although painful to maintain).
The Easy Firewall – nftables
The easy way of getting some firewalling done involves installing nftables, adding some rules to it, and loading them as a part of the start-up. This is suitable if you have a few interfaces with a simple configuration that doesn’t change often, preferably ever.
The actual first step of getting the firewall up and running is adding the nftables package to the image if you don’t have it already installed. This can be easily checked by running nft command. If it fails, add the package to the image configuration:
IMAGE_INSTALL:append = " nftables"
If you see the following types of errors when running the nft command it means that your kernel configuration is missing either nftables support (the first error) or some nftables module (the second error):
root@qemux86-64:~# nft
../../nftables-1.0.2/src/mnl.c:60: Unable to initialize Netlink socket: Protocol not supported
root@qemux86-64:~#
root@qemux86-64:~# nft -f /tmp/test.conf
/tmp/test2:32:9-16: Error: Could not process rule: No such file or directory
ct state vmap { established : accept, related : accept, invalid : drop }
^^^^^^^^
root@qemux86-64:~#
Quite often the firewall in Yocto is empty by default. The active rules of the firewall can be checked by running nft list ruleset command to print the current ruleset. If nothing is output, no rules are present. If something gets output, there are some rules present. Here’s a short reference of commands on how to set some basic rules and list them on the command line.
# List ruleset
nft list ruleset
# Add a table named filter for IP traffic
nft add table inet filter
# Add an input chain that drops traffic by default
nft add chain inet filter input \{ type filter hook input priority 0 \; policy drop \}
# Add rule to allow traffic to port 22
nft add rule inet filter input tcp dport \{ 22 \} accept
# List tables and rules in a table
nft list tables
nft list table inet firewall
In addition to listing the tables and rulesets from inside the device, you should also try a black-box approach. Running nmap command against your device is a good way to check how the outside world sees your device. In this example, I have added the Dropbear SSH server to the image. Because the server is listening on port 22 and there are no rules defined in the firewall, the port is seen as open:
esa@ubuntu:~$ nmap 192.168.7.2
Starting Nmap 7.80 ( https://nmap.org ) at 2023-07-22 16:14 UTC
Nmap scan report for 192.168.7.2
Host is up (0.0024s latency).
Not shown: 999 closed ports
PORT STATE SERVICE
22/tcp open ssh
Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds
The command line commands are useful for editing the configuration run-time, but for the rest of the text we’re going to focus on a file-based configuration that is set during the Yocto build. The configurations presented here are mostly based on “a simple ruleset for a server” in nftables wiki. It contains some useful extra things not presented here, like a rule for the loopback interface, differentiating between ipv4 and ipv6 traffic and logging, and I recommend checking it out.
Usually, a good way to start creating a firewall ruleset is to block everything by default, and then start poking holes as needed. To block everything, you can write a configuration file with the following content:
This config creates one rule table named firewall for inet traffic. The table contains three chains: inbound, forward and output. You can load the ruleset with nft -f <config_file> command. Once you’ve blocked all traffic, you can give the nmap command another go. As you can see, blocking is really effective. nmap actually considers the device to be non-existent at this point:
esa@ubuntu:~$ nmap 192.168.7.2 Starting Nmap 7.80 ( https://nmap.org ) at 2023-07-22 16:18 UTC Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn Nmap done: 1 IP address (0 hosts up) scanned in 3.03 seconds
Trying to ping from the device itself to the outside world also fails because the output is blocked. In theory, allowing outgoing traffic is not the worst idea, usually many sample configurations actually allow it. But if you know the traffic that’s going to go out and the ports that will be used there’s in my opinion no sense allowing all of the unnecessary traffic. That’s just an attitude that will lead to botnets. But if you want/need to allow all outgoing traffic, you can just drop the output chain from the config. By default everything will be accepted.
After nothing goes in or out, it’s time to start allowing some traffic. For example, if you want to allow SSH traffic from the barbarians to port 22 (generally a Bad Idea, but for the sake of an example), you can use the following ruleset:
flush ruleset
table inet firewall {
chain inbound {
type filter hook input priority 0; policy drop;
# Allow SSH on port TCP/22 for IPv4 and IPv6.
tcp dport { 22 } accept
}
chain forward {
type filter hook forward priority 0; policy drop;
}
chain output {
type filter hook output priority 0; policy drop;
# Allow established and related traffic
ct state vmap { established : accept, related : accept, invalid : drop }
}
}
If you want to define multiple allowed destination ports, separate them using commas, like { 22, 80, 443 }. If you choose not to block outgoing connections, you obviously don’t need the output chain presented here.
On the other hand, if there’s a process in the Yocto image that wants to contact the outside world, you could consider adding a ruleset like below:
flush ruleset
table inet firewall {
chain inbound {
type filter hook input priority 0; policy drop;
# Allow established and related traffic
ct state vmap { established : accept, related : accept, invalid : drop }
}
chain forward {
type filter hook forward priority 0; policy drop;
}
chain output {
type filter hook output priority 0; policy drop;
# You can check the ephemeral port range of kernel
# from /proc/sys/net/ipv4/ip_local_port_range
tcp sport 32768-60999 accept
}
}
Note how we need to use a port range for the output. Outgoing connections usually use a port from an ephemeral range for their connection, meaning that you may not know beforehand the exact port that needs to be allowed. Having fixed ports for outgoing traffic makes the device more susceptible to port scanning and prevents multiple connections from the same service because the port is already used, but if you choose to create a service that uses a static source port, you can use a single port number in the rule.
You can combine the rules in the chains as required by your system and services. If you want to enable the ping command (for ipv4), you can add the following piece of configuration to the input chain (note that this is just a single line of configuration, not a full config):
icmp type echo-request limit rate 5/second accept
In the end, creating the firewall ruleset is fairly simple: block everything and allow only necessary things. The difficulty really comes from keeping the rules up-to-date when something changes. However, when developing an embedded device with Yocto you generally should have a well-defined list of allowed ports & traffic (as opposed to a general-purpose computer used by a living human where all sorts of traffic and configs may come and go at the user’s whim). If that’s not the case, the part 2 where we work with firewalld may be useful to you.
So, now we know how to write the perfect configuration file. But how to add this to a Yocto build? You can create a bbappend file for nftables into your own meta-layer. This append then installs the configuration file, because nftables does not install any configuration by default. An example of how to add a configuration to nftables can be found in meta-firewall-examples repository I made for this blog post.
You should also note that nftables package doesn’t contain any mechanism to activate the rules during the start-up. Therefore you should append the nftables recipe so that it adds an init.d script or a service file that activates the rules from the configuration file before the networking is started. You can find an example of this as well from the same meta-firewall-examples repository.
Note that if you do run-time edits to the configuration with nft command, the changes will be lost when the device is reset. To ensure that the changes are kept, you can run the following command to overwrite the default config with the new, enhanced configuration:
nft list ruleset > /path/to/your/configuration-file
And one more thing: if you face some No such file or directory issues when trying more exotic configurations you are most likely missing some kernel configurations. In part 2, we’ll be enabling pretty much every feature of the Netfilter, so that’ll (most likely) be helpful. On that cliffhanger, we eagerly await the next chapter in this ever-evolving journey of discovery and innovation (I asked Chat-GPT to write that sentence).
In my last blog post, I said that I wanted to try finishing a project instead of starting a new one. Let’s forget that and kick off a new project. In my defense, I started working on this idea a few months ago already, and now I am (mostly) finishing it.
Back then I thought that it would be fun to have a retro-style camera. An analog camera would be neat, but developing films in this day and age is a bit of a pain. Early 2000s digital camera could be an option, but paying for one of those would be a bit of a waste. I mean, they cost maybe 10€, but it’s more a matter of principle. A Polaroid camera would tick all the boxes, but that idea didn’t come to my mind.
Then I thought why not build a digital camera? Well, for starters, it takes some effort that’s definitely worth more than 10€. Also, it takes some materials that are worth more than 10€. So yeah, all around a silly idea. Let’s do it.
Camera Lens
When I say that I’m going to build a camera it doesn’t mean building a lens. Those are quite precise devices, and it would be quite a bold claim to say that I possess the skills or facilities to make them. Instead, I had to salvage one from somewhere. Fortunately, every laptop contains a camera lens in the form of a webcam, and I have some spare laptops lying around. Also, the benefit of using a laptop webcam as a camera lens is that it contains a controller that can usually be interfaced with USB.
So, it’s time to get the fine-tuning hammer and give my ancient laptop a light tap with it:
*bonk*
This process of course varies from laptop to laptop, but usually “<laptop name> disassembly” search from Google is a good starting point. There is always a small random repair shop that has made a disassembly video for your laptop. After the webcam is within reach, it’s just a matter of cutting the wires and terminating them. Not a good idea to have unterminated wires inside a laptop, at least if it’s still going to be used.
So now I have the camera module for my camera. I just need to figure out how to connect it to anything. The wires didn’t seem to follow any standard coloring scheme, and after some googling, I couldn’t find any standard order of the wires either. However, there is this useful information printed on the silk screen underneath a sticker in the back:
I wonder what these ancient hieroglyphs mean
The next thing to do is solder the power of the webcam to the power of a USB cable, ground to ground, D+ to D+, and D- to D-, right? Wrong (maybe). I’m not sure if someone has mislabeled the wires or if I just messed them up, but soldering D+ of the webcam to D+ of the USB cable (and the same for D-) resulted in following errors in Linux when plugging in the device.
kernel: usb 4-1: new low-speed USB device number 14 using ohci-pci
kernel: usb 4-1: device descriptor read/64, error -62
kernel: usb 4-1: device descriptor read/64, error -62
kernel: usb 4-1: new low-speed USB device number 15 using ohci-pci
kernel: usb 4-1: device descriptor read/64, error -62
kernel: usb 4-1: device descriptor read/64, error -62
kernel: usb usb4-port1: attempt power cycle
kernel: usb 4-1: new low-speed USB device number 16 using ohci-pci
kernel: usb 4-1: device not accepting address 16, error -62
kernel: usb 4-1: new low-speed USB device number 17 using ohci-pci
kernel: usb 4-1: device not accepting address 17, error -62
kernel: usb usb4-port1: unable to enumerate USB device
This is where I gave up a few months ago on my first try. I was too sad to go on.
The Project: Rebirth
Fast forward two months. I saw an ad for a contest. The competition was looking for something that could be described as “overengineered DiWhy” projects, but maybe in a bit more positive sense. After seeing the ad, and realizing the fact that a 3D printer was available as a reward, I knew what I had to do: finish the digital camera.
So I dug up the abandoned project and soldered D+ to D- and D- to D+, plugged the thing in, and was pleasantly surprised that it actually now worked and I hadn’t caused a heat death of the device with careless soldering. The final schematic ended up looking like this:
Note that D+ and D- are subjective truth in this reality. Mixing them up shouldn’t break the device but it won’t work either.
Two 1N4001 diodes are used to drop the voltage from the USB’s 5V closer to the 3.3V expected by the camera module. After reading some blog posts about similar projects it seemed like some other people had had the same problem with D+ and D-, so it may be a common point of confusion.
The quality of the connections has nothing to do with the possible communication issues.
The next step was coming up with a plan for what to actually do with this thing. I ended up using a Raspberry Pi powered by a power bank and creating a small control/status board using GPIO pins of Raspberry Pi.
Control/Status Board
I put together all my entry-level electronics knowledge and tried to remember which way the LED should be connected. I failed. After googling some basic Arduino tutorials for children, I came up with this kind of schematic.
LEDs are connected to GPIO pins 23 and 24, button is connected to 22.
To be honest, I don’t quite understand electronics as well as I would want to, and I’m not 100% sure if the resistors are the correct size. At least the board works and doesn’t produce audio-visual bang-smoke output, so I guess it’s good? Or maybe I’m slowly but steadily causing some irreparable damage that will manifest itself in surprising and slightly disappointing ways? Only time will tell.
Can a button be anything besides big and red? I think not.
The purpose of the button is quite obvious: press it to take a picture. The LEDs give an indication of the system’s status. One LED turns on when the device is listening to button presses and ready to take pictures. The second one turns on when a picture is being taken (it’s a surprisingly long process).
I’ve Got the Power
To power up the Raspberry Pi I used a power bank that has enough juice to power up the device. It’s Anker Powercore II 10000 that outputs 5V and 3A, which is within the recommended limits. In addition, I “soldered” a switch to the power wire of the USB cable to have a rudimentary power button. “Soldered” is in quotes, because I tried new lead-free solder for the first time, and nothing really stuck on anything despite the maximum heat and effort, so it was closer to suffering than soldering.
I haven’t tried if the data lines work, but if I had to guess, I’d say no. I’m surprised that even the power line works.
Software
This type of device could use two software components:
A program that handles GPIO input and output
A program that captures the camera frame and outputs the image
I wanted to write a program that would actually communicate with the camera module, but because I had to finish the camera in time for the contest, I opted for an existing solution. fswebcam is a command line software that can be used to capture frames from a webcam, which is exactly what we’re going to do.
The first piece of code I wrote is camera-gpio. It turns on the standby LED, polls the button state, and turns on another LED if an image is being taken. If the button is pressed camera-gpio launches the second program that actually takes the picture. Quite self-explanatory. It’s a C program that’s built with CMake, because I wanted to refresh my memory on how those work.
The second code repo is camera-handler, and it’s for the program that takes the photo. Currently, it’s pretty much just a wrapper for fswebcam. camera-handler also generates filenames for the images from the system time, which is a bit useless because the board doesn’t have RTC or NTP to store or sync the time. But you have to consider the opportunities, all the things it could be! It’s a C++ program that’s built with autotools, because I wanted to refresh my memory on how those work.
If you’ve been reading this blog before, you may know what comes next: Yocto build. In addition to these code repos, I made a meta-layer named meta-camera with the required files to build these into a Linux firmware image ready to be flashed into an SD card.
Final Schematic and Photos
Once all the pieces were ready, all that was left to do was plug in the thingys, boot up the thingy, and hope for the best.
Hope is a powerful thing. It may lead to a semi-functional thingy that takes pictures. Here’s a picture of the final build:
Doesn’t look quite as good as it did in the schematic. Just don’t take it with you on an airplane.
Beautiful. I posted this picture earlier on Linkedin and considered sorting out the wires. Decided to keep them as they are, because I can’t be bothered to be honest.
How do the actual pictures look? Well, I’m going to be polite and say “quite retro”, which was the original goal of the project. See for yourself:
“Coffee break of the piggies threatened by a long-legged spider”
“Sunrise in Suburbia (picture taken at 13:02)”
“Oh wow didn’t expect this close-up to actually look tolerable”
At the least the file sizes are small.
Future Work
Honestly, I think this project could still use plenty of work. Here’s the list of things that could be fixed or improved:
Make the boot-up time shorter: While it’s delightfully old-school to wait 20 seconds for the camera to start up, it’s also a quite nerve-racking experience to wait every time for almost half a minute to see if the device still works.
Make the picture capture time shorter: It takes quite a few seconds to take a photo. Well, taking the picture itself is almost immediate, but using fswebcam is quite slow.
Store the pictures on their own partition: Currently, images are stored in the boot partition because that’s where they can be easily accessed. This is a hilariously bad idea for multiple reasons.
Audio feedback: It feels weird to use a camera that doesn’t make an artificial shutter sound.
PTP device: Currently, the SD card needs to be removed from the camera and inserted into a computer to access the images. It’d be nifty if the device could just be plugged into a computer to browse the photos.
Fix kernel issue at boot: Oh, did I forget to mention that there’s a kernel error message printed during boot? Well, there is. And it should most likely be looked at.
Create a nice case for the camera: I wish I had a 3D printer to print a case for the camera. I wonder where I could get one. *wink wink nudge nudge*
All in all, I think there are more things that should be done than actually have been done. However, I’m also quite happy with the things that are already done. Good starting point for whatever comes next.
People need projects to consume their free time. I’ve lately felt that I want to actually try finishing a project (instead of just starting them), and that the project should be somehow related to audio, and it would be nice if it would have a real-world use, and it would use the old Raspberry Pis that have lying around. Plenty of requirements then. I think this is still better a better-formulated train wreck than an average customer project.
After considering few different options, I ended up attempting to create a multi-speaker streaming system named Aioli (so yeah, I started another project). This text is closer to a devlog than tutorial, but there will be open source code repositories in case you want to see how it’s done. Enough with the blabber, let’s move on.
Overview Of The Project
Basically, in this project I want to have one audio source, and the audio from that single source would get wirelessly transmitted to multiple speakers. To be more specific, in my case there’s one Raspberry Pi 4 connected to an external audio source, and then there would be other Raspberry Pi 2’s connected to the speakers. The Raspberry Pis in this scenario would handle at least streaming, networking, receiving the audio, and playing the audio. This graph attempts to explain the situation:
To start out the project I decided to focus on the streaming between Raspberry Pis because I didn’t feel masochistic enough to start working with Bluetooth yet. Everything is all fun and games until Bluetooth is added into the mix, and I want to have a bit of fun and games.
Today’s focus is this part to be exact
Obviously, the first thing to do is to create a custom Yocto distro, because every self-respecting hobby project needs its own Linux distribution. Perhaps further down the line this distro can contain some useful configs and other things that actually justify its existence, but for now, it’s just a renamed example Poky distro.
Creating The Network Of Raspberries
To get the Raspberry Pis talking to each other the first step is getting the devices connected to the same LAN. I wanted to use WLAN to not have cables around the house. Using ethernet cables would defeat the whole point of the system anyway as I could just use audio cables then. I considered also an ad hoc network but decided to use WLAN to keep things familiar for now. The Raspberry Pi 4 I own does have an internal Wifi chip, so that was easy to sort out, but the two Raspberry Pi 2’s did not. I had one Wifi dongle that worked out of the box, but another dongle required some extra work. You can read about it from my previous blog post if you’re interested.
After getting the hardware sorted out it was time to get the devices actually connected to WLAN. For that purpose, I added wpa_supplicant to the distro. wpa_supplicant is a program that in layman’s terms “connects the device to wifi” (or so I’ve understood as a layman). A properly configured supplicant that launches during boot should in theory automatically connect the device to WLAN. Surprisingly enough, it usually does. Following simple configuration in /etc/wpa_supplicant.conf added during a build to a Raspberry Pi does the trick:
network={
ssid="WLANname"
psk="SecretPassword"
}
This of course means that you have a statically defined network you want to connect to, and the password is stored in plaintext in the device. Both are bad things for different reasons, but they’ll do for now because it’s the simplest solution. This simplicity will be fixed later on in another text. If you have a WLAN network without a password or want to use a calculated key instead of a plaintext password, you can read more about wpa_supplicant from Arch wiki. It’s a good read. Pay attention to the quotation marks in psk-variable, they caused a lot of headache to me.
With quotation marks the value is a plaintext password, without them it’s a calculated key value. Makes “sense”.
After the devices wirelessly connected to the router, I gave them static IP address leases to make the development somewhat easier. I also ran a quick ping test to check that the Raspberry Pis can reach Google and each other before proceeding.
Moving The Audio Bits
Making the audio streaming work was actually fairly simple because there already is an open-source solution, as there usually is. GStreamer is an “open source multimedia framework”, which can mean many things. This is quite fitting because GStreamer does many things, and with the help of its plugins, it can do pretty much anything you can dream of. Assuming your dreams revolve around handling and processing multimedia.
My dream was to find a way to stream audio over IP network. And dreams, they sometimes do come true. Actually, a bit too much so, it was slightly difficult to find the best options for streaming the audio with all encoding options, protocols, and what-not. I’m still not sure I picked the right things. To keep the prototyping fast I worked with command line tools provided by GStreamer (as opposed to using the API, which may be worth looking into in the future).
GStreamer works with pipelines. Pipelines have sources where the media originates from, sinks where the media ends up, and parsers, encoders, and other types of things in between that manipulate input and pass it forward. For example, here’s a simple pipeline that reads an audio file, and then parses, converts, resamples, and outputs it to the appropriate default sink:
This command may result in a sound being output from your speakers. Quite often not. Depends on what your default ALSA output device is, if you’re using PulseAudio, and if it’s the third Tuesday of the month. In the case of Raspberry Pi, the default output device is the HDMI audio, and I’m not using PulseAudio, and it’s not the third Tuesday today, meaning that I actually got some sound out from a television connected to the HDMI port. If you want to get the audio output from the Raspberry Pi’s headphone jack, you can be a bit more specific about the sink:
gst-launch-1.0 filesrc location=/opt/sample-files/sample1.wav ! wavparse ! audioconvert ! audioresample ! alsasink device=hw:1
# use "aplay -l" command to list the available ALSA devices
To get the audio sent over the network we can use the RTP protocol that’s meant for delivering audio and video. Basic GStreamer functionality can be easily extended with plugins, and as it turns out, there exists a plugin for RTP. It’s weird how these things work out nicely. Almost like someone has had the same ideas before me. Now we can package the audio to 16-bit RTP payloads, and instead of using an alsasink we can use a udpsink (from another plugin) to output the stream to a target in a network instead of an audio device.
Then, the intended receiver of the stream can use udpsrc instead of filersrc to read the stream, decode, and deliver the contents to its own audio sink. Simple as.
In theory, we could use multicast streaming instead of multiple streams but for some reason I couldn’t get it working. Most likely it had something to with the third Tuesday of the month. I couldn’t even complete a simple multicast test on my network of Raspberry Pis, so I guess something is wrong with my setup. For the sake of completeness, AFAIK these commands (should (in theory)) work, but don’t. I’ll look into this later on because multicasting seems like a more sensible approach to this problem:
Considering the amount of multicast memes floating around the internet, I’m not the only one having issues with it.
By using these commands we can send the audio over network from the controller device to the speaker devices. However, this is still a bit cumbersome, because we need to manually run the gst-launch-1.0 commands, figure out the intended receivers & their IP addresses, and so on. Later on I plan to introduce a manager process that’s dynamically able to find the clients in LAN and control the streaming, but that’s a topic for another text.
There’s a recipe for GStreamer and its plugins in Yocto, so to get these things installed into the new custom distro is just a matter of adding a few packages. It’s almost simpler than using a package manager. At least if you’ve spent the last five years learning the ins and outs of Yocto, and don’t need to install them during runtime. Something like this should do the trick:
Plugins are sorted to good, bad, and ugly (I guess it’s no big surprise that bluez plugin is “bad”). To figure out which group plugin belongs to you can check the documentation. The documentation is quite good by the way, I recommend reading it. For example, udp plugin page contains information about the pipeline elements it provides, and also mention which group it belongs to.
That mostly covers all for this text. We’re now able to send sound over the network from one device to another. Next time we’ll stop this goofing off, and get painfully serious by adding Bluetooth to the system, and instead of using sample audio files we’ll actually stream something from a phone.
You can find the top-level repo-tool manifest repository from here. Please note that the progress of the project is a bit further than what’s presented in this blog text, and the progress is also “a bit” all over the place, so the manifest repository and the subrepositories contain plenty of spoilers and confusion.
One more question remains: why the name Aioli? Well, it kinda sounds like audio combined with I/O, and I like garlic flavoured condiments. That’s as good reason as any.
This is a story of how I became a useful member of society by doing my first open-source contribution.
It all began one fateful afternoon, when I purchased a TP-Link Wifi dongle, thinking that it would allow me to connect my old Raspberry Pi 2 wirelessly to the internet. It was running my own Poky-based distro, but what could really go wrong with random USB devices and Linux?
Well quite a plenty really. I plugged the device in, but I couldn’t connect to the highway of data. No delicious internet cookies for me. Not even a blinking led.
To begin troubleshooting the issue, I tried checking if the network interface was seen by kernel by running both ifconfig -a and ip link show. No wlan devices were found. Some googling suggested running lsusb. That showed the device, which at least proved that it wasn’t broken and was recognized by kernel. Some sort of network driver was clearly needed.
Bus 001 Device 004: ID 2357:011f TP-Link 802.11ac WLAN Adapter
Bus 001 Device 003: ID 0424:ec00 Microchip Technology, Inc. (formerly SMSC) SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Microchip Technology, Inc. (formerly SMSC) SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Finding the correct driver turned out to be a bit tougher than expected. First I tried googling the name of the wifi dongle suffixed with “driver”. Bad Idea. This led to a lot of ancient forum posts that suggested all kinds of Realtek drivers for (almost) similarly named devices that were installed by enabling a variety of kernel configuration options. None of the drivers worked.
After some more of the furious googling I found out that the wifi dongle I bought required an out-of-tree kernel module instead. That meant I couldn’t just enable a kernel configuration to build the driver in my distro. Finding the correct driver was another trial and error type of affair. Someone suggested a driver for the 8812au chip. It did not work but helped me to find a correct trail.
Fortunately there’s a lot less diseases on this trail.
RTL8812AU driver repo contained a file supported-device-IDs that expectedly did not contain the device id output by lsusb. However, that gave me an idea (that I really should have gotten from the beginning): googling “driver 2357:011f”. Who would have guessed that searching for a driver with an exact device id instead of vague product names would yield the correct driver(s)? This search also helped me to find the name of the Realtek chip, 8821au, which I confused plenty of times with 8812au. I’m not sure if this info would have been available on the manual of the dongle because I did not read it.
After finding the driver & chip I connected some dots and realized that there actually is a kernel configuration driver named CONFIG_RTL8XXXU that I tried. Despite what the name suggests, it does not work with rtl8821au.
Once the correct driver was figured out it was time to add it to the Yocto build. Some more googling revealed that there is a meta layer called meta-rtlwifi for these Realtek out-of-tree modules. Unfortunately, it didn’t contain the RTL8821AU driver. Fortunately, I’ve been using git at work so I could fix that myself. You can see where this is heading.
So I took the RTL8812AU driver recipe as I suspected that it should mostly work, and updated the relevant parts, i.e. the repo to fetch the driver from. I was pleasantly surprised that the build worked just like that. Even more shocking was that the module worked as well. After that, it was just a matter of a pull request to get the driver added to the meta-layer alongside the other friendly drivers.
There were actually multiple drivers available for 8821au. At least morrownr, ulli-kroll and ivanborislav provide RTL8821AU drivers. In the end, I chose morrownr driver because their driver worked satisfactorily out of the box and their driver is also used for 8812au. I first gave a shot at ivanborislav driver but it filled my TTY with logs about power save mode. Most likely a configuration mistake from my side, but usually a thing that works without extra tinkering is the better choice.
It’s almost weird that there’s a meme for literally everything.
That’s how I got quite familiar with my wifi dongle, and made my first open-source contribution in the process. I also learned something. I’m not yet 100% sure what that is. Perhaps it’s that the device id is quite important when trying to find a suitable driver. And googling can give all kinds of interesting useful information. Until next time!
Last time when we were talking about Yocto hardening we focused on setting up extra users to prevent unauthorized root access and to avoid situations where the users would have too open permission sets. Getting these right is a good low-hanging fruit, but there are plenty of other things to consider for hardening the system. One of the most important things to do is to eliminate (or at least minimize) the security vulnerabilities weakening the overall system defenses.
What are CVEs?
Despite what one may be inclined to think, open-source code is not always perfect. It may contain bugs, which is a bit annoying, but it may also contain vulnerabilities, which is potentially a bit dangerous. These can lead to denial of service attacks, data leaks, or even unauthorized accesses & system takeovers. Not good. Very bad. The usual stuff.
CVE (Common Vulnerabilities and Exposures) is a system maintained by The United States National Cybersecurity FFRDC and operated by Mitre Corporation. The system contains information about, well, vulnerabilities and exposures. “CVE” as a noun can also mean vulnerability or exposure. Or at least I often use the acronym in with that meaning. Checking your Yocto system against this database of vulnerabilities can help to detect security issues lurking in the open-source code components.
Checking vulnerabilities in Yocto
Every CVE doesn’t necessarily lead to unauthorised root access, and there may be vulnerabilities that don’t have a CVE identifier. However, scanning through the CVEs in your system should provide a fairly good image of how vulnerable your system is against possible CVE exploits. At least at the time of the scan, there are constantly new issues popping up which means you have to keep doing it often. Also it’s good to keep in mind that this kind of a scan does not check for other kind of weaknesses in the system. But at least performing the CVE check in the Yocto system is fairly simple. Something is simple in Yocto, what?
My whole worldview is being shattered.
In all simplicity, just add this line to local.conf to check your packages & images for common vulnerabilities.
INHERIT += "cve-check"
# I had some issues with slow download speeds. If you get weird timeouts
# when fetching issue database, try adding this line as well:
CVE_SOCKET_TIMEOUT = "180"
Now when you run the build you’ll most likely end up with more warnings than before. Yocto’s CVE checker prints a warning to the build output for every vulnerability it detects, and depending on the complexity and age of your system your terminal may end up quite yellow. It’s more useful to inspect the CVE reports than to try and remember the build output as it scrolls by. These reports can be found in <build-folder>/tmp/deploy/cve folder, and we’ll inspect them a bit later. It’s worth knowing that this CVE checker may not be perfect, meaning that there may be some vulnerabilities lurking around that it does not detect. But it’s at least better than going through all the source code eyeballing it.
Analysing the results
Ok, now you’ve proved to yourself that your system has holes that need to be plugged. How to get over this uneasy feeling of constant doom looming over your system? The first thing is listing out all the CVE IDs and checking their status & severity to get a bit more stressed. Or relaxed, depends a bit on how well your system is maintained.
Let’s be real, this is most likely going to be the most accurate reaction.
Further information for the CVEs can be found on MITRE’s CVE site, and some more complementary information can be found on NIST’s (National Institute of Standards and Technology) NVD (National Vulnerability Database). Plenty of acronyms. NVD is a system containing the same CVE IDs as MITRE’s site, but with severity scores and usually some extra information. Both are definitely useful, but for practical purposes NVD tends to be more useful. MITRE’s site can contain some new info not yet present in NVD, but NVD contains the information in a more easily digestible format.
Let’s take CVE-2023-22451 as an example. CVE listing for the vulnerability shows a short summary, affected versions, and a bunch of links. NVD entry has the same information, severity score, modification date, and a bit more information about the links so you won’t have to guess which link is the patch and which is some advisory (this can be useful on bigger issues where there are a dozen of links). For what it’s worth, Yocto’s CVE checker uses NVD’s database.
Now that we’ve gone through some background information, we can investigate the results of the CVE check. The aforementioned <build-folder>/tmp/deploy/cve contains two summary files: cve-summary and cve-summary.json. These contain all the detected issues, both patched and unpatched, and the format of the files isn’t all that intuitive for getting an actual summary, so I wrote a simple Python-tool for summarising the results. The design is very human:
# Replace the file path with your path
python3 ./cve-parser.py -f build/tmp/log/cve/cve-summary.json
Very easy to use. By default the parser will output the unpatched CVEs, and from each CVE it will print the package name, CVE ID, CVSS scores, and the link to the NVD site for more info. There is more information per issue available, the results can be sorted, and the ignored & patched issues can also be printed if needed. To get the complete list of options, run:
python3 ./cve-parser.py -h
I’m not going to document the whole program here in case I decide to change it at some point. If your boss asks you to get the total amount of unpatched CVEs in the system, you can run the command below to get “extra info” from the parser. Currently, “extra info” just contains the number of displayed issues, but if you’ve got ideas for it feel free to let me know:
# -u display unpatched issues (default behavior, but added just in case)
# -e displays extra info
python3 ./cve-parser.py -u -e -f build/tmp/log/cve/cve-summary.json
If your boss asks how many of them actually need to be fixed, use this to sort the CVEs by severity score and count how many have a score higher than 8.3
After you’ve justified your fears of getting hacked by realizing that your system has seven CVEs with a CVSS score higher than 9.5 (three of them in the kernel) and a few dozen more with just a slightly lower priority, it’s time to ask the big question: what can I do? What can I possibly do? This didn’t help with the feeling of constant doom at all.
The first step is going through the list of unpatched issues. There tend to be quite a few false positives that are not applicable to the current system. For example, a brand new Poky build reports an unpatched CVE-2017-6264 that’s been around six years already. This is a vulnerability in the NVIDIA GPU driver, and it’s applicable to Android products. Most certainly it’s a false positive in your Yocto system, but since the bad code is present in the source code, it’s reported as unpatched. You can ignore these false positives by adding the line below somewhere to your build configuration (local.conf works, but it’s maybe not the best place for it):
# The format is CVE_CHECK_IGNORE="<cve-id>", e.g.
CVE_CHECK_IGNORE="CVE-2017-6264"
# In the older Yocto versions this was called CVE_CHECK_WHITELIST
# so if ignore doesn't work, try the old whitelist variable
After ignoring the false positives fixing the rest of the CVE issues is easy, in theory. Just keep the Yocto at its newest (or relatively new LTS) version. This should keep the packages fresh. Sounds simple, but unfortunately this process tends to cause a big headache with incompatibilities and such. However, that’s the best advice I can give.
If there is a need to update the recipes even further than officially supported, or if you want to update a single recipe, copy the recipe in question to your own meta-layer, update the recipe version in the filename and fix SRCREV to match. When hacking around like this remember to hope that nothing falls apart in the system even more than during a regular Yocto version update.
Sometimes, updating the packages is not an option. Perhaps updating the package breaks the build, a newer version of the library isn’t compatible with your application, or there is some other Perfectly Good Reason you’re not allowed to do that. That’s when it’s time to do some patching to make the confusing build system even more spaghettified.
Basically, this approach consists of heading to the NVD entry for the CVE, checking if there’s a patch for the vulnerability, and if there is, porting the patch to the Yocto build. For example, the aforementioned CVE-2023-22451 mentions this commit as its patch in the NVD entry. Copy the contents of the commit to a patch file, create bbappend file for the recipe and add the patch to the recipe with that bbappend.
If there’s no patch you can wait. Or if you’re like a kid on Christmas and can’t wait, you can try digging the package’s git repositories for the corrective commit before it’s been released. It ain’t fun, and rarely it’s a productive use of time, but every now and then some fixes can be found like this before they are made official.
That’s a short explanation of how you can check and fix the CVE vulnerabilities in your system. The theory of the process is fairly simple. However, it tends to get a bit more complicated in the actual world, especially when trying to update older legacy systems where the stale, non-updated packages contain more patches than original code. But I guess that’s the problem with real life, it tends to mess up good theories.
If you’re like me, you have more money than brains, and not too much of either really. This can lead to situations where you end up buying a MacBook without actually checking if it supports Yocto builds, the one thing you inexplicably like to waste your little free time on. As it turns out after getting the nearly 2K€ laptop, builds on Mac are not supported. But don’t worry, where there is a problem, there usually is a convoluted solution.
As one might expect, the answer is virtualization. Specifically Docker virtualization this time. Writing this kind of instructional text is becoming my second nature, earlier I wrote how to build Yocto on Windows/WSL, and now how to build the damn thing on Mac. One day I’ll get an actual Linux computer suitable for this kind of development work, but not today. I’m out of money at the moment. Anyways, to the actual guide. I’ll warn you beforehand that my knowledge of Docker is quite superficial, but I’ll try to explain things as well as I can.
Installing Docker
When doing Docker builds the first step obviously is to install Docker. Get it from here and come back when you’re done.
Don’t worry, they definitely are.
Getting correct images
Now, the actual tricky part: getting the Docker images suitable for building Yocto. Docker images are used to create the containers that are the virtualized environment we can do the builds in. Yocto Development Tasks Manual mentions CROPS (CROss PlatformS) Docker images that can be used for building in non-Linux hosts. CROPS itself also has this nice Mac guide that almost works.
Well, I guess the manual works for the older amd64 MacBooks because the pre-built Docker images are only available in amd64 format. Since Apple Silicon is arm64, Docker gives a nice warning that the emulated image may or may not work, and spoiler alert, it does not. Attempting to run bitbake after following the guide gives the following error:
OSError: Cannot initialize new instance of inotify, Errno=Function not implemented (ENOSYS)
However, I recommend keeping that CROPS guide nearby because we are going to follow it once we get the suitable images built. The first step of the journey is pulling the Docker image source repositories for the two images mentioned in the CROPS Mac guide:
The first one is responsible for creating a samba file server to provide the contents of the build container to the Mac file system via a Docker volume, and the second one is the actual builder image. Building the samba image is simple enough:
% cd samba
% docker build . -t crops/samba:selfbuilt
% cd ..
The poky-container image is a bit trickier. It attempts to fetch an amd64 operating system image provided by CROPS as its base layer. Instead of using that image, we need to find something else that’s more suitable for us and arm64.
Not much.
Docker Hub is a site containing Docker images for a variety of uses. Among many others, it also contains the CROPS amd64 images that we are not able to use. By searching for “Yocto”, and limiting the results to arm64 architecture we can find quite a few results. However, none of these (at least at the time of writing) fit into the CROPS workflow or are up to date, meaning that they won’t work with the newer versions of Yocto. Therefore we need to build the operating system base image by ourselves.
The Dockerfiles used to build the images are open source, but unfortunately, the scripts in them use GNU versions of some commands, meaning that the scripts won’t work with the BSD commands Mac has. But fear not, I made a fork for building the base images on Mac! It works at least for the Ubuntu 22.04 image, I was a bit lazy and didn’t check the dozen other options. To build the Ubuntu base image, run the following commands:
git clone git@github.com:ejaaskel/yocto-dockerfiles.git
cd yocto-dockerfiles
export REPO=ejaaskel/yocto
export DISTRO_TO_BUILD=ubuntu-22.04
./build_container
cd ..
This will create two images: ejaaskel/yocto:ubuntu-22.04-base and ejaaskel/yocto:ubuntu-22.04-builder. Now we just need to go to the CROPS poky-container Dockerfile and replace their base image with the one we just built. After that, we should be able to build the poky-container as an arm64 image and get to actually building Yocto.
% cd poky-container
## Open your favourite text editor (it better be nvim),
## and replace these lines in Dockerfile:
ARG BASE_DISTRO=SPECIFY_ME
FROM crops/yocto:$BASE_DISTRO-base
## With these lines to specify the distro version and
## use the Docker image we built:
ARG BASE_DISTRO=ubuntu-22.04
FROM ejaaskel/yocto:$BASE_DISTRO-base
% docker build . -t crops/poky:selfbuilt
% cd ..
Now we’re again on track to follow that CROPS guide. We just have to suffix every mention of the CROPS images with :selfbuilt so that we’ll use the built images and not pull images from Docker Hub. So whenever there’s a mention of crops/samba in the guide, write crops/samba:selfbuilt, and when there’s a mention of crops/poky, use crops/poky:selfbuilt instead.
Building Yocto
If you followed the instructions in the CROPS guide you should end up with an Ubuntu terminal in /workdir and a Finder window that’s connected to the same location in the Docker container. After that, it’s just a matter of following Yocto’s Quick Build manual or building something else that’s actually interesting. Once the build finishes, you can access the files in Finder as you normally do, or if you prefer to use the command line, docker cp should do the trick as well:
# Use docker ps to get crops/poky:selfbuilt container id
% docker ps
% docker cp <container-id>:workdir/<whatever-you-want-to-copy> .
By the way, if you stick to docker cp for moving stuff around, you don’t actually need to start the samba container. Saves a bit of effort.
of a copy
If you happen to need a root shell in the container for some reason, like installing packages, you can open it like this:
# Use docker ps to get crops/poky:selfbuilt container id
% docker ps
% docker exec -it --user=root <container-id> bash
If (and when) you run out of disk space, you can increase the virtual disk size from the Docker Desktop program. Select the cog from the upper right corner to enter settings, navigate to Resources->Advanced and from there you can increase the virtual disk limit. I’m quite sure there is a way to do this in the command line as well, but hopefully this isn’t a task that I’ll have to do so often that I’d need to figure it out.
Can you see how artistic those red circles are? All thanks to the new MacBook.
That’s all for this time! Hopefully, this helped you to get started with Yocto builds on Apple Silicon a bit quicker. Fortunately, this wasn’t really anything groundbreaking, mostly it was just a matter of combining a few different information sources and a bit of script editing. The biggest sidestep from the CROPS instructions was building the Docker images instead of using the ready-built ones. But for someone who’s new to Mac & Docker, it may take a surprisingly long time to figure this out. At least it did for me. Writing this also helped a bit with my buyer’s remorse. Until next time!
Testing. The final frontier. Or so it often feels. It’s the place where no man boldly goes, it’s the place where they tend to crawl to when being forced to do so after the test coverage checker lets them know that there’s less than 60% of the line coverage. Hopefully, this is just a tired stereotype, because testing is mighty useful in all kinds of applications for quite obvious reasons: it saves time, helps catch bugs, and a good & passing test set gives a better peace of mind than a ten-minute mindfulness session. In this text, I’m writing a bit about unit testing in audio processing applications and presenting my small example plugin that contains some unit tests. It’s basically my upcoming ADC 2022 talk in a text format, unless something surprising happens in the upcoming week. Let’s hope not, I react poorly to surprises.
Testing
Unit testing is a type of testing where small sections of the program code are individually and independently scrutinized. Usually, this is done by the means of feeding known input to the functions being tested, and comparing the outputs, function calls, etc. against an expected set of values. Basic stuff, really.
However, audio applications are slightly different, or at least have some features that don’t fully follow this idea of known inputs and outputs. For example, think about a high-pass filter that adds some “character” to the signal it filters. The character here means something extra being added to the audio signal during processing. Depending on the type of “character” we may know the output only partially. Sure, the audio gets filtered according to the filter parameters, but the processing may also add some random elements, like a tiny smidgen of noise to keep things exciting. In such a situation we can’t directly compare the output of the processing function to some expected value, because the expected value is only partially known.
Of course, the test in such a case could be reduced to smaller components that have fully predictable outputs. And it could even be argued that testing these small components is the whole idea of unit testing. But even if we break down the filter into the smallest possible pieces, there still would be that one noise generator element that can’t be fully predicted (unless using the same random seed every time). How can we know what to expect when we only have a general idea of what something is supposed to sound like, but it can’t be exactly defined? As a sidenote, this kind of vague hand-waving describes my music-making fairly well.
Basically me whenever I’m asked to explain anything (please don’t ask difficult questions)
Math to the rescue
I dislike math as much as anyone else, but no one can really disagree with the fact that it’s definitely useful. Without it, we wouldn’t have trebuchets. And from what I’ve heard, mathematics plays some role in most of the computer systems as well.
Fast fourier transform is a magica… mathematical algorithm that can perform discrete fourier transformations. Fourier transformation on the other hand can convert time or space functions to their frequency components. Or, if these words mean nothing to you, it can convert this kind of waveform image (i.e. time domain representation):
Difficult to say what it sounds like, but at least it’s loud.
Into this kind of frequency spectrum image (frequency domain representation):
Ah, it’s a sine sweep.
How does this happen? I wish I understood. Wikipedia article for Fourier transform is filled with sentences like “for each frequency, the magnitude of the complex value represents the amplitude of a constituent complex sinusoid with that frequency, and the argument of the complex value represents that complex sinusoid’s phase offset” and to be honest, sentences like this make me scared. For all I know, it could say that there’s a tiny wizard somewhere in the computer that has nothing better to do than some signal processing. Although, all the signal-processing people I know tend to be wizards, so maybe there’s some truth in that.
However, with the FFT we can get the characteristics of the audio, like what frequency range has a lot of energy, or where there is no energy at all. Audio signals are signals, and as such, they have some amount of energy. These kinds of frequency characteristics can be compared in non-exact, fuzzy ways by setting thresholds for the allowed energy amounts for different frequency ranges.
For example, if we have a simple gain function, we can use FFT to check that the audio signal energy is indeed increasing or decreasing. Or if there’s a filter that needs to be tested we can check that some frequency bands have no energy to ensure that the filter doesn’t allow frequencies to pass from where they shouldn’t. Or if there’s a white noise generator, it can be checked that there is energy across the whole frequency range. Or with a synthesizer, we can check that there are no unexpected artefacts in the signal. Without fast fourier transform finding out these kinds of things automatically can be tricky.
Pamplejuce
Integrating a unit testing framework with an audio development framework can be challenging. It took me surprisingly much googling to find anything useful on the topic. There were plenty of tutorials/advertisements about building different unit testing framework demos. I’m using JUCE, and it has some level of unit testing support, but that seemed insufficient. There also were conference talks confirming that yes, you indeed should test your software even if it handles audio signals, that’s not an acceptable reason to skip testing even though that’s a tempting thought.
Finally, after failing to do what I wanted with JUCE’s unit tests, and failing to integrate two different unit testing frameworks into my JUCE plugin, I came across Pamplejuce. It’s a template project using CMake workflow, and more importantly (at least for me), it integrates Catch2 unit testing framework to JUCE. It’s good that someone around here is a competent developer.
Catch2 is a unit testing framework. I don’t have much else to say about it. It’s being actively developed and it has all the features I could ask for. This isn’t all that much, I have about as many requirements for unit testing frameworks as I do for hotel rooms: hotel rooms should have a bed, a shower and a lockable door. Unit testing frameworks should have the possibility to compare two things and an option to write custom matchers. And I should be able to integrate them into my projects. The last one was surprisingly rare.
I couldn’t yet find “Unit testing framework integration for dummies”, but once I find one I’ll be sure to get one.
FilterUnitTest
To test out Pamplejuce I created a new imaginatively named project FilterUnitTest from the template. To have some actual audio processing to test I made a simple highpass filter plugin using JUCE’s LadderFilter. No GUI, just simple filtering pleasure. Well, it also has a parameter for the wet amount. I’ll give a warning that there’s still plenty of refactoring & warning fixing to be done in the FilterUnitTest, but I’m currently working on it whenever I have the time. I’m quite certain that the tests at least leak memory, but I haven’t yet gotten around to fixing that (I know, I know, it isn’t very C++17 to have memory leaks).
So, after the base of the project was done, it was time to finally implement some tests. First is the dummy test already part of the Pamplejuce template, which simply checks that the name of the plugin can be fetched and that it is actually the expected name. The best test is the one someone writes for you. The test cases presented in this blog text can be found from Tests/PluginBasics.cpp file in the FilterUnitTest repository.
TEST_CASE("Plugin instance name", "[name]")
{
testPluginProcessor = new AudioPluginAudioProcessor();
CHECK_THAT(testPluginProcessor->getName().toStdString(),
Catch::Matchers::Equals("Filter Unit Test"));
delete testPluginProcessor;
}
To verify the actual functionality of the program, I identified two things that need to be tested:
Testing that the filter performs filtering to the audio signal
Testing that the wet parameter controls the amount of applied filtering
The first test is quite simple: create an audio signal buffer, or read it from an audio file. Run it through the audio processing function, and check that the output contains less energy than before filtering. This of course requires writing a custom matcher, more on that a bit later. If you’ve set your filtering function in stone, you could consider storing the output from one of the test runs and comparing the test results to that instead. During the development phase, it may be easier to do some non-exact comparisons of the FFT values.
If you choose to go the exact value comparison route, you can check out the writeBufferToFile and readBufferFromFile helper functions in Tests/Helpers.h. They serialize and deserialize an audio buffer to/from a file. These helpers can be used to create the exact expected values, and they can also be used to fetch the expected value and compare the output to it. This dummy test basically writes a random buffer to a file, reads the file and ensures that the two buffers have identical contents.
As you can see, this type of test requires a custom matcher, AudioBuffersMatch. As does the FFT comparison, and any other custom comparison. For FilterUnitTest, I wrote four different types of comparators, these can be found from Tests/Matchers.h:
Audiobuffers are equal
Audiobuffer has higher energy than another audio buffer
Audiobuffer has a maximum energy of N in its frequency bands (N can vary between different bands, and the check for a band can also be skipped)
Audiobuffer has minimum energy of N in its frequency bands (Same here)
The second approach of using FFT to ensure that the audio buffer has lower energy after filtering can use a combination of the second and third matcher. By combining these two, we can ensure that the total energy of the signal is indeed lower and that the amount of lower frequencies is within a certain limit:
TEST_CASE("Filter", "[functionality]")
{
int samplesPerBlock = 4096;
int sampleRate = 44100;
testPluginProcessor = new AudioPluginAudioProcessor();
//Helper to read a sine sweep wav
juce::MemoryMappedAudioFormatReader *reader = Helpers::readSineSweep();
juce::AudioBuffer<float> *buffer = new juce::AudioBuffer<float>(reader->numChannels, reader->lengthInSamples);
reader->read(buffer->getArrayOfWritePointers(), 1, 0, reader->lengthInSamples);
juce::AudioBuffer<float> originalBuffer(*buffer);
//Dismiss the partial chunk for now
int chunkAmount = buffer->getNumSamples() / samplesPerBlock;
juce::MidiBuffer midiBuffer;
testPluginProcessor->prepareToPlay(sampleRate, samplesPerBlock);
//Process the sine sweep, one chunk at a time
for (int i = 0; i < chunkAmount; i++) {
juce::AudioBuffer<float> processBuffer(buffer->getNumChannels(), samplesPerBlock);
for (int ch = 0; ch < buffer->getNumChannels(); ++ch) {
processBuffer.copyFrom(0, 0, *buffer, ch, i * samplesPerBlock, samplesPerBlock);
}
testPluginProcessor->processBlock(processBuffer, midiBuffer);
for (int ch = 0; ch < buffer->getNumChannels(); ++ch) {
buffer->copyFrom(0, i * samplesPerBlock, processBuffer, ch, 0, samplesPerBlock);
}
}
//Check that originalBuffer has higher total energy
CHECK_THAT(originalBuffer,
!AudioBufferHigherEnergy(*buffer));
juce::Array<float> maxEnergies;
for (int i = 0; i < fft_size / 2; i++) {
//Set the threshold to some value for the lowest 32 frequency bands
if (i < 32) {
maxEnergies.set(i, 100);
}
//Skip the rest
else {
maxEnergies.set(i, -1);
}
}
//Check that lower end frequencies are within limits
CHECK_THAT(*buffer,
AudioBufferCheckMaxEnergy(maxEnergies));
//I guess programming C++ like this in the year 2022 isn't a good idea to do publicly
delete buffer;
delete reader;
delete testPluginProcessor;
}
The second test for testing the wet parameter is basically a continuation of this. Get your audio buffer and run it through the audio processing function with varying levels of wet-parameter. Ensure that the higher the wet parameter is, the higher the filtering effect. This means there’s again less low-end energy, and less energy in general. Or if you want to do a super simple test as I did, just check that with a wet value of 0 signal doesn’t change, and with the max wet parameter value of 1 it does.
TEST_CASE("Wet Parameter", "[parameters]")
{
testPluginProcessor = new AudioPluginAudioProcessor();
//Helper to generate a buffer filled with noise
juce::AudioBuffer<float> *buffer = Helpers::generateAudioSampleBuffer();
juce::AudioBuffer<float> originalBuffer(*buffer);
juce::MidiBuffer midiBuffer;
testPluginProcessor->prepareToPlay(44100, 4096);
testPluginProcessor->processBlock(*buffer, midiBuffer);
//Check that initial value of wet is not zero, i.e. filtering happens
CHECK_THAT(*buffer,
!AudioBuffersMatch(originalBuffer));
delete buffer;
buffer = Helpers::generateAudioSampleBuffer();
//Get and set parameter
auto *parameters = testPluginProcessor->getParameters();
juce::RangedAudioParameter* pParam = parameters->getParameter ( "WET" );
pParam->setValueNotifyingHost( 0.0f );
for (int ch = 0; ch < buffer->getNumChannels(); ++ch)
originalBuffer.copyFrom (ch, 0, *buffer, ch, 0, buffer->getNumSamples());
testPluginProcessor->processBlock(*buffer, midiBuffer);
//Check that filter now doesnt affect the audio signal
CHECK_THAT(*buffer,
AudioBuffersMatch(originalBuffer));
delete buffer;
buffer = Helpers::generateAudioSampleBuffer();
pParam->setValueNotifyingHost( 1.0f );
for (int ch = 0; ch < buffer->getNumChannels(); ++ch)
originalBuffer.copyFrom (ch, 0, *buffer, ch, 0, buffer->getNumSamples());
testPluginProcessor->processBlock(*buffer, midiBuffer);
//Finally, check that with max wet the signal is again affected
CHECK_THAT(*buffer,
!AudioBuffersMatch(originalBuffer));
delete buffer;
delete testPluginProcessor;
}
I wish I could argue which approach is better here, but I think it’s quite apparent whether or not it’s better to do proper or half-assed testing. I’ll leave it as an exercise for the reader to figure out how to do the proper way of testing.
Images
If you’re given the following sample values, can you figure out what the audio signal sounds like?
I don’t know either. In general, humans tend to absorb information via visual means. A bunch of decimal numbers isn’t the most intuitive way of understanding something unless you’re one of the wizards I talked about earlier. But what if I showed you this:
Seems like it’s a signal of sorts.
As you can see (pun intended) visual information is a lot easier to digest. It’s not the most impressive of graphs really, nor the easiest to read, but there’s a good explanation for that: I wrote the code for it. At least the code allows easy drawing of images as a part of unit tests to see what’s going on with the inputs and outputs, as opposed to printing out audio buffer contents and FFT results and hoping that staring at the numbers absorbs them into the brain. You can find the image drawing function from Tests/ImageProcessing.h, here it is in action:
Just give it a buffer and a filename without the .png extension and it’ll handle the rest. So, for example to make sure you’ve hooked things up correctly in your testing set-up, you can call the drawing function before and after doing some processing to the audio signal to see if the changes are at least somewhat sensible.
As you can guess, this was one of the earlier attempts of getting things working.
Benchmarking
Imagine you’re in an all-you-can-eat buffet. You can eat whatever as much as you want and the staff won’t kick you out for a few hours. Usually eating a lot feels like a good idea. However, after you’ve stopped eating, the reality of the situation settles in and you realize that it wasn’t a good idea. You feel sick.
The same applies to coding. It’s fun to code without limits. More features, more, MORE, you’ll think to yourself. However, after coding for a while you’ll realize that this wasn’t a good idea either. The program starts to become slow and sluggish. You’ve introduced latency to your code, and that is the second worst thing that can be done. The only thing worse is a 127 dBFS pop that was caused by careless buffer handling when you were starting out with audio signal processing.
To keep things in check, Catch2 has some simple benchmarking macros. There are a few example usages of those in FilterUnitTest-repo. It’s quite basic C++, meaning that it took me about three compilation attempts and one illegal memory access to get the syntax right. After some trials and a lot of errors, I ended up with something like this:
Attempting to read C++ documentation on lambdas and pretending to understand what’s going on.
Closing words
I hope you found this blog post useful. As mentioned, I was struggling to get started with JUCE and unit testing, so hopefully this writing helps you to think about how to test your application, assists in integrating a unit testing framework, and contains some useful and practical resources to get you started with testing. Also, I want to say that this type of FFT matching isn’t the only solution for unit-testing audio applications. You can for example remove the random elements from your tests, use pre-determined random seeds, or mock some parts of your code if needed. I’ve just found the FFT approach really intuitive and flexible after I got my head wrapped around it. Thanks for reading!
Comments are welcome but due to a quite hefty amount of bot spam, the comments will go through moderation so it may take some time to see your prose in the comments section. As long as you’re not trying to sell Viagra or women’s haircuts to me it’ll eventually appear there. If you happen to be going to ADC 2022 feel free to let me know!
I don’t know the context of this gif, but knowing it’s South Park it must be something very nice and wholesome.
Cybersecurity. The never-ending race between you trying to secure your precious IoT device and some propeller head who’s finding the wildest exploits and destroying your system just in the time between breakfast and lunch (although true hackers work during night, not in the mornings). Or perhaps you forgot to update your WordPress plugins, and now your fantastic development blog is hacked. Or perhaps you’ve just been postponing that security update on your Android phone for six months. We all have some experience with cybersecurity.
Usually, there are some simple things to do to prevent the worst catastrophes. Update software, make sure the run-time environment is sane & secure, and encrypt the secrets to mention a few. In this new blog series, I try to explain how to make your Yocto systems more secure by adding some common-sense security to the system. All of these ideas are familiar from Linux desktop & server environments, but I’m going to show how to apply them to Yocto builds. So, unfortunately, I’m not actually going to cover how to keep a fantastic WordPress site secure.
Well okay, no need to be upset. Just press the red “update plugins” button every now and then. And if you’re self hosting, I wish luck to your distro upgrades.
As usual with any security advice, it’s good to be slightly skeptical when reading it. This means that some rando writing blog posts on their blog may not be a 100% trustworthy or up-to-date source on everything. Also, those randos will take no responsibility for the results of their advice. However, their advice may provide useful guidance and general tips that you can apply to your system’s security hardening plan. You do have a security plan, right?
So, the first topic of this Yocto hardening exercise is users. As you may have heard, running and doing everything as the root user is generally considered a Bad Idea. Having services running under the root user unnecessarily can result in root user executing arbitrary code using the most imaginative methods. Or, if the only user available for logging in is the root user you’re not only giving away root permissions to malicious users, but to incompetent users as well.
This text will assume intermediate knowledge of Yocto, meaning that I won’t explain every step in depth along the way. Hopefully, you know what an append-file and local.conf are. The code and config snippets were originally written for Yocto Kirkstone checked out “at some point after 4.0.3 release”, and later re-tested with version 4.0.15.
Before we get started with the actual hardening, there’s one preliminary task to do. Ensure that you don’t have DEBUG_TWEAKS in either IMAGE_FEATURES or EXTRA_IMAGE_FEATURES. Not only is it unsafe as it allows root login without a password, and a bunch of other debug stuff, but it also makes some examples shown here behave in unexpected ways.
Creating non-root users
Here is a code snippet that creates a service user for the system. This user can be logged in with, and they have sudo capabilities (if sudo-package is installed). Insert this code either into a image recipe or a configuration file:
# If adding to configuration file, use INHERIT += "extrausers" instead.
inherit extrausers
IMAGE_INSTALL:append = " sudo"
# This password is generated with `openssl passwd -6 password`,
# where -6 stands for SHA-512 hashing alorithgm
# The resulting string is in format $<ALGORITHM_ID>$<SALT>$<PASSWORD_HASH>,
# the dollar signs have been escaped
# This'll allow user to login with the least secure password there is, "password" (without quotes)
PASSWD = "\$6\$vRcGS0O8nEeug1zJ\$YnRLFm/w1y/JtgGOQRTfm57c1.QVSZfbJEHzzLUAFmwcf6N72tDQ7xlsmhEF.3JdVL9iz75DVnmmtxVnNIFvp0"
# This creates a user with name serviceuser and UID 1200.
# The password is stored in the aforementioned PASSWD variable
# and home-folder is /home/serviceuser, and the login-shell is set as sh.
# Finally, this user is added to the sudo-group.
EXTRA_USERS_PARAMS:append = "\
useradd -u 1200 -d /home/serviceuser -s /bin/sh -p '${PASSWD}' serviceuser; \
usermod -a -G sudo serviceuser; \
"
The extrausers class allows creating additional users to the image. This is done simply by defining the desired commands to create & configure users in the EXTRA_USERS_PARAMS variable. As a side note, it’s good to have static UIDs for the users as this will make the builds more reproducible.
In a perfect world, every custom service you create for your system would run under a non-root user. When writing a recipe that creates a daemon or other kind of service, you can use the snippet below to add a new user in a non-image recipe:
inherit useradd
USERADD_PACKAGES = "${PN}"
GROUPADD_PARAM:${PN} = "--system systemuser"
# This creates a non-root user that cannot be logged in as
USERADD_PARAM:${PN} = "--system -s /sbin/nologin -g systemuser systemuser"
As you can see, adding a new user in a non-image recipe is slightly different than in the image recipe. This time we’re only giving parameters to useradd and groupadd commands. After adding this, you should be able to start your daemon as the systemuser in a startup script. If you need root permissions for some functionality in your service but don’t want the service to be run as root, I’d recommend reading into capabilities. It’s worth noting that when adding users in a service recipe like this, the additions are done per-package basis, not per-recipe basis. This means that you can create and add new users in a quite flexible manner.
The flexibility tends to come with some complications though…
Editing sudoers configuration
By default, when adding the sudo-package to the build it doesn’t do much. It doesn’t actually allow anything special to be done by the users, even if they are in the sudoers group. That’s why we need to edit the sudo-configuration. There are two ways of doing this, either by editing the default sudoers file or by adding drop-in sudoers.d configurations. First, editing the sudoers file (which is the worse method in my opinion).
There are two ways of doing this, and I’m not 100% sure which one is less bad, so I’ll leave it to you to decide. The first option is adding a ROOTFS_POSTPROCESS_COMMAND to the image recipe:
enable_sudo_group() {
# This magic looking sed will uncomment the following line from sudoers:
# %sudo ALL=(ALL:ALL) ALL
sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/' ${IMAGE_ROOTFS}/etc/sudoers
}
ROOTFS_POSTPROCESS_COMMAND += "enable_sudo_group;"
The second option is creating a bbappend file for the sudo recipe, and adding something like this there:
do_install:append() {
# Effectively the same magic sed command
sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/' ${D}/${sysconfdir}/sudoers
}
Both do the same thing, and both are hacky. The correct choice really depends on if you want to edit the sudoers file in a different way for each image, or if you want the sudo configuration to depend on something else. You can also of course supply your own sudoers file instead of sed-editing it, and that can be a lot more maintainable way of doing this if you have more than just one change.
However, a more flexible way of configuring sudo is with the sudoers.d drop-in files. It also involves a lot fewer cryptic sed commands. Let’s assume that you have a recipe that creates a new user lsuser (not to be confused with a loser), and you want that user to have sudo rights just for ls-command (considering rights given to this user, they may actually be a loser). For this purpose you need to create a bbappend for sudo, and add something like this snippet to create a drop-in configuration:
Again, if your drop-in configuration is more complex than one line, you can provide it as a configuration file through SRC_URI and install that in your image instead of echoing configuration lines. I recommend reading sudo-configuration documentation for more info on how to precisely set the permissions because let’s be honest, these two examples aren’t the shining examples of sudo configuration.
The downside of this approach is that you need to add the drop-in configuration in a sudo bbappend because /etc/sudoers.d will conflict when creating the root filesystem if you add the drop-in in another recipe. This means that when you’re creating a user in a non-sudo recipe and add the drop-in conf in the sudo recipe you’ll have the user creation & configuration handled in two different places, which is perfect for forgetting to maintain things.
Disabling root-login
Disabling root login is useful for a multitude of reasons.
You usually want to make sure that no user has the full command set available
You also want that the users’ privileged actions to end up logged in auth.log when they use sudo
In general, it’s “a bit of a risk” to allow users to log in as root
Disabling the root user can be achieved in multiple ways, and I’m going to cover three of them here: fully locking the root account, removing the login shell, and disabling the SSH login. In theory, locking the root account should be enough, assuming non-root users cannot unlock it, but I’ll present all the methods as they may be useful for different situations.
First, disabling the root account. This is a method I found from this StackOverflow answer, kudos there. The simple snippet below pasted in the image recipe should lock and expire the root account:
Then, removing the login shell. Or more like setting the login shell to nologin. This isn’t strictly required after locking the root account but can be achieved by creating a bbappend shown below to base-passwd recipe:
do_install:append() {
# What this magic sed does:
# "In the line beginning with root
# replace /bin/sh with /sbin/nologin
# in file passwd.master"
# (passwd file gets generated from this template during install)
sed -i '/^root/ s/\/bin\/sh/\/sbin\/nologin/' ${D}${datadir}/base-passwd/passwd.master
# sry for the picket fence
# I think it's peak sed that I need a four line
# comment to explain a one-liner
}
EDIT: It was suggested in the comments that usermod can be used for disabling the root login shell as well. That solution is a lot cleaner than this sed & bbappend. Check the comments below for an example.
And finally, disabling the SSH login. You should always disable the root login over remote networks. If you need, you can just disable the SSH login for root and still allow other service users to log in as root. This is still a better option than doing nothing, as this will prevent logging in as root remotely and attackers would need to know two passwords to gain root access (one for the service user, and the other one for the root user).
Disabling the root login varies a bit between the SSH servers. Dropbear disables root login by default with -w parameter in DROPBEAR_EXTRA_ARGS variable located in the /etc/defaults/dropbear file (note that if you have debug-tweaks enabled, the file actually contains -B, allowing root-login). You can overwrite the file with your own dropbear.default file during the build if you want to add more options.
Similarly, OpenSSH-server disables root logins with passwords if debug-tweaks is removed from the IMAGE_FEATURES (and allows them if it’s present). This is achieved by sed-editing SSH configuration files. If you want to see how this is exactly done, check ssh_allow_root_login function in meta/classes/rootfs-postcommands.bbclass (part of poky).
However, it’s worth noting that this default behaviour doesn’t prevent public key authentication. If you want to disable that as well, you can add this function as a rootfs post-process task to the image recipe. And if needed, it could of course be modified to work in a bbappend as well.
# The function here searches sshd_config and sshd_config_readonly files for a
# commented line containing PermitRootLogin, and replaces it with "PermitRootLogin
# no" to prevent any sort of root login.
disable_rootlogin() {
for config in sshd_config sshd_config_readonly; do
if [ -e ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config ]; then
sed -i 's/^[#[:space:]]*PermitRootLogin.*/PermitRootLogin no/' ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config
fi
done
}
ROOTFS_POSTPROCESS_COMMAND += "disable_rootlogin;"
The thing to note about disabling root login is that too lenient sudo- or filesystem-permissions for non-root users can make the whole thing useless. For example, if a service user has sudo access to passwd command they can easily unlock the root account. If a service user has write permissions to /etc they can set the root’s login shell and edit SSH configuration. And finally, disabling root login like shown in this text does nothing to prevent shell escapes made possible by incorrectly set sudo-permissions.
Might as well roll out the red carpet
That’s it for this time! Hopefully, these tips helped you to achieve the minimum security in your Yocto images. Do these code snippets make your system unhackable? Not even close. Do they make it slightly more inconvenient? I hope so (but no guarantees for that either). But at least now you can show your Yocto image to an independent professional security consultant and say that you tried before they tear the whole system to pieces.
