yocto

How to build Yocto with Apple Silicon

If you’re like me, you have more money than brains, and not too much of either really. This can lead to situations where you end up buying a MacBook without actually checking if it supports Yocto builds, the one thing you inexplicably like to waste your little free time on. As it turns out after getting the nearly 2K€ laptop, builds on Mac are not supported. But don’t worry, where there is a problem, there usually is a convoluted solution.

Rube Goldberg Win GIF by Millions - Find & Share on GIPHY

As one might expect, the answer is virtualization. Specifically Docker virtualization this time. Writing this kind of instructional text is becoming my second nature, earlier I wrote how to build Yocto on Windows/WSL, and now how to build the damn thing on Mac. One day I’ll get an actual Linux computer suitable for this kind of development work, but not today. I’m out of money at the moment. Anyways, to the actual guide. I’ll warn you beforehand that my knowledge of Docker is quite superficial, but I’ll try to explain things as well as I can.

Installing Docker

When doing Docker builds the first step obviously is to install Docker. Get it from here and come back when you’re done.

Gonna Go Homer Simpson GIF - Find & Share on GIPHY
Don’t worry, they definitely are.

Getting correct images

Now, the actual tricky part: getting the Docker images suitable for building Yocto. Docker images are used to create the containers that are the virtualized environment we can do the builds in. Yocto Development Tasks Manual mentions CROPS (CROss PlatformS) Docker images that can be used for building in non-Linux hosts. CROPS itself also has this nice Mac guide that almost works.

Well, I guess the manual works for the older amd64 MacBooks because the pre-built Docker images are only available in amd64 format. Since Apple Silicon is arm64, Docker gives a nice warning that the emulated image may or may not work, and spoiler alert, it does not. Attempting to run bitbake after following the guide gives the following error:

OSError: Cannot initialize new instance of inotify, Errno=Function not implemented (ENOSYS)

However, I recommend keeping that CROPS guide nearby because we are going to follow it once we get the suitable images built. The first step of the journey is pulling the Docker image source repositories for the two images mentioned in the CROPS Mac guide:

% git clone https://github.com/crops/samba.git
% git clone https://github.com/crops/poky-container.git

The first one is responsible for creating a samba file server to provide the contents of the build container to the Mac file system via a Docker volume, and the second one is the actual builder image. Building the samba image is simple enough:

% cd samba
% docker build . -t crops/samba:selfbuilt
% cd ..

The poky-container image is a bit trickier. It attempts to fetch an amd64 operating system image provided by CROPS as its base layer. Instead of using that image, we need to find something else that’s more suitable for us and arm64.

Season 3 Episode 20 GIF by SpongeBob SquarePants - Find & Share on GIPHY
Not much.

Docker Hub is a site containing Docker images for a variety of uses. Among many others, it also contains the CROPS amd64 images that we are not able to use. By searching for “Yocto”, and limiting the results to arm64 architecture we can find quite a few results. However, none of these (at least at the time of writing) fit into the CROPS workflow or are up to date, meaning that they won’t work with the newer versions of Yocto. Therefore we need to build the operating system base image by ourselves.

The Dockerfiles used to build the images are open source, but unfortunately, the scripts in them use GNU versions of some commands, meaning that the scripts won’t work with the BSD commands Mac has. But fear not, I made a fork for building the base images on Mac! It works at least for the Ubuntu 22.04 image, I was a bit lazy and didn’t check the dozen other options. To build the Ubuntu base image, run the following commands:

git clone git@github.com:ejaaskel/yocto-dockerfiles.git
cd yocto-dockerfiles
export REPO=ejaaskel/yocto
export DISTRO_TO_BUILD=ubuntu-22.04
./build_container
cd ..

This will create two images: ejaaskel/yocto:ubuntu-22.04-base and ejaaskel/yocto:ubuntu-22.04-builder. Now we just need to go to the CROPS poky-container Dockerfile and replace their base image with the one we just built. After that, we should be able to build the poky-container as an arm64 image and get to actually building Yocto.

% cd poky-container
## Open your favourite text editor (it better be nvim),
## and replace these lines in Dockerfile:
ARG BASE_DISTRO=SPECIFY_ME
FROM crops/yocto:$BASE_DISTRO-base
## With these lines to specify the distro version and 
## use the Docker image we built:
ARG BASE_DISTRO=ubuntu-22.04
FROM ejaaskel/yocto:$BASE_DISTRO-base
% docker build . -t crops/poky:selfbuilt
% cd ..

Now we’re again on track to follow that CROPS guide. We just have to suffix every mention of the CROPS images with :selfbuilt so that we’ll use the built images and not pull images from Docker Hub. So whenever there’s a mention of crops/samba in the guide, write crops/samba:selfbuilt, and when there’s a mention of crops/poky, use crops/poky:selfbuilt instead.

Building Yocto

If you followed the instructions in the CROPS guide you should end up with an Ubuntu terminal in /workdir and a Finder window that’s connected to the same location in the Docker container. After that, it’s just a matter of following Yocto’s Quick Build manual or building something else that’s actually interesting. Once the build finishes, you can access the files in Finder as you normally do, or if you prefer to use the command line, docker cp should do the trick as well:

# Use docker ps to get crops/poky:selfbuilt container id
% docker ps
% docker cp <container-id>:workdir/<whatever-you-want-to-copy> .

By the way, if you stick to docker cp for moving stuff around, you don’t actually need to start the samba container. Saves a bit of effort.

Fight Club Work GIF - Find & Share on GIPHY
of a copy

If you happen to need a root shell in the container for some reason, like installing packages, you can open it like this:

# Use docker ps to get crops/poky:selfbuilt container id
% docker ps
% docker exec -it --user=root <container-id> bash

If (and when) you run out of disk space, you can increase the virtual disk size from the Docker Desktop program. Select the cog from the upper right corner to enter settings, navigate to Resources->Advanced and from there you can increase the virtual disk limit. I’m quite sure there is a way to do this in the command line as well, but hopefully this isn’t a task that I’ll have to do so often that I’d need to figure it out.

Can you see how artistic those red circles are? All thanks to the new MacBook.

That’s all for this time! Hopefully, this helped you to get started with Yocto builds on Apple Silicon a bit quicker. Fortunately, this wasn’t really anything groundbreaking, mostly it was just a matter of combining a few different information sources and a bit of script editing. The biggest sidestep from the CROPS instructions was building the Docker images instead of using the ready-built ones. But for someone who’s new to Mac & Docker, it may take a surprisingly long time to figure this out. At least it did for me. Writing this also helped a bit with my buyer’s remorse. Until next time!

Yocto hardening: Non-root users, sudo configuration & disabling root

Find all of the Yocto hardening texts from here!

Cybersecurity. The never-ending race between you trying to secure your precious IoT device and some propeller head who’s finding the wildest exploits and destroying your system just in the time between breakfast and lunch (although true hackers work during night, not in the mornings). Or perhaps you forgot to update your WordPress plugins, and now your fantastic development blog is hacked. Or perhaps you’ve just been postponing that security update on your Android phone for six months. We all have some experience with cybersecurity.

Usually, there are some simple things to do to prevent the worst catastrophes. Update software, make sure the run-time environment is sane & secure, and encrypt the secrets to mention a few. In this new blog series, I try to explain how to make your Yocto systems more secure by adding some common-sense security to the system. All of these ideas are familiar from Linux desktop & server environments, but I’m going to show how to apply them to Yocto builds. So, unfortunately, I’m not actually going to cover how to keep a fantastic WordPress site secure.

Well okay, no need to be upset. Just press the red “update plugins” button every now and then. And if you’re self hosting, I wish luck to your distro upgrades.

As usual with any security advice, it’s good to be slightly skeptical when reading it. This means that some rando writing blog posts on their blog may not be a 100% trustworthy or up-to-date source on everything. Also, those randos will take no responsibility for the results of their advice. However, their advice may provide useful guidance and general tips that you can apply to your system’s security hardening plan. You do have a security plan, right?

So, the first topic of this Yocto hardening exercise is users. As you may have heard, running and doing everything as the root user is generally considered a Bad Idea. Having services running under the root user unnecessarily can result in root user executing arbitrary code using the most imaginative methods. Or, if the only user available for logging in is the root user you’re not only giving away root permissions to malicious users, but to incompetent users as well.

This text will assume intermediate knowledge of Yocto, meaning that I won’t explain every step in depth along the way. Hopefully, you know what an append-file and local.conf are. The code and config snippets were originally written for Yocto Kirkstone checked out “at some point after 4.0.3 release”, and later re-tested with version 4.0.15.

Before we get started with the actual hardening, there’s one preliminary task to do. Ensure that you don’t have DEBUG_TWEAKS in either IMAGE_FEATURES or EXTRA_IMAGE_FEATURES. Not only is it unsafe as it allows root login without a password, and a bunch of other debug stuff, but it also makes some examples shown here behave in unexpected ways.

Creating non-root users

Here is a code snippet that creates a service user for the system. This user can be logged in with, and they have sudo capabilities (if sudo-package is installed). Insert this code either into a image recipe or a configuration file:

# If adding to configuration file, use INHERIT += "extrausers" instead.
inherit extrausers

IMAGE_INSTALL:append = " sudo"

# This password is generated with `openssl passwd -6 password`, 
# where -6 stands for SHA-512 hashing alorithgm
# The resulting string is in format $<ALGORITHM_ID>$<SALT>$<PASSWORD_HASH>,
# the dollar signs have been escaped
# This'll allow user to login with the least secure password there is, "password" (without quotes)
PASSWD = "\$6\$vRcGS0O8nEeug1zJ\$YnRLFm/w1y/JtgGOQRTfm57c1.QVSZfbJEHzzLUAFmwcf6N72tDQ7xlsmhEF.3JdVL9iz75DVnmmtxVnNIFvp0"

# This creates a user with name serviceuser and UID 1200. 
# The password is stored in the aforementioned PASSWD variable
# and home-folder is /home/serviceuser, and the login-shell is set as sh.
# Finally, this user is added to the sudo-group.
EXTRA_USERS_PARAMS:append = "\
    useradd -u 1200 -d /home/serviceuser -s /bin/sh -p '${PASSWD}' serviceuser; \
    usermod -a -G sudo serviceuser; \
    "

The extrausers class allows creating additional users to the image. This is done simply by defining the desired commands to create & configure users in the EXTRA_USERS_PARAMS variable. As a side note, it’s good to have static UIDs for the users as this will make the builds more reproducible.

In a perfect world, every custom service you create for your system would run under a non-root user. When writing a recipe that creates a daemon or other kind of service, you can use the snippet below to add a new user in a non-image recipe:

inherit useradd
USERADD_PACKAGES = "${PN}"
GROUPADD_PARAM:${PN} = "--system systemuser"
# This creates a non-root user that cannot be logged in as
USERADD_PARAM:${PN} = "--system -s /sbin/nologin -g systemuser systemuser"

As you can see, adding a new user in a non-image recipe is slightly different than in the image recipe. This time we’re only giving parameters to useradd and groupadd commands. After adding this, you should be able to start your daemon as the systemuser in a startup script. If you need root permissions for some functionality in your service but don’t want the service to be run as root, I’d recommend reading into capabilities. It’s worth noting that when adding users in a service recipe like this, the additions are done per-package basis, not per-recipe basis. This means that you can create and add new users in a quite flexible manner.

The flexibility tends to come with some complications though…

Editing sudoers configuration

By default, when adding the sudo-package to the build it doesn’t do much. It doesn’t actually allow anything special to be done by the users, even if they are in the sudoers group. That’s why we need to edit the sudo-configuration. There are two ways of doing this, either by editing the default sudoers file or by adding drop-in sudoers.d configurations. First, editing the sudoers file (which is the worse method in my opinion).

There are two ways of doing this, and I’m not 100% sure which one is less bad, so I’ll leave it to you to decide. The first option is adding a ROOTFS_POSTPROCESS_COMMAND to the image recipe:

enable_sudo_group() {
    # This magic looking sed will uncomment the following line from sudoers:
    #   %sudo   ALL=(ALL:ALL) ALL
    sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/'  ${IMAGE_ROOTFS}/etc/sudoers
}

ROOTFS_POSTPROCESS_COMMAND += "enable_sudo_group;"

The second option is creating a bbappend file for the sudo recipe, and adding something like this there:

do_install:append() {
    # Effectively the same magic sed command
    sed -i 's/^#\s*\(%sudo\s*ALL=(ALL:ALL)\s*ALL\)/\1/'  ${D}/${sysconfdir}/sudoers
}

Both do the same thing, and both are hacky. The correct choice really depends on if you want to edit the sudoers file in a different way for each image, or if you want the sudo configuration to depend on something else. You can also of course supply your own sudoers file instead of sed-editing it, and that can be a lot more maintainable way of doing this if you have more than just one change.

However, a more flexible way of configuring sudo is with the sudoers.d drop-in files. It also involves a lot fewer cryptic sed commands. Let’s assume that you have a recipe that creates a new user lsuser (not to be confused with a loser), and you want that user to have sudo rights just for ls-command (considering rights given to this user, they may actually be a loser). For this purpose you need to create a bbappend for sudo, and add something like this snippet to create a drop-in configuration:

do_install:append() {
    echo "lsuser ALL= /bin/ls " > ${D}${sysconfdir}/sudoers.d/lsuser
}

FILES_${PN} += " ${sysconfdir}/sudoers.d/lsuser"

Again, if your drop-in configuration is more complex than one line, you can provide it as a configuration file through SRC_URI and install that in your image instead of echoing configuration lines. I recommend reading sudo-configuration documentation for more info on how to precisely set the permissions because let’s be honest, these two examples aren’t the shining examples of sudo configuration.

The downside of this approach is that you need to add the drop-in configuration in a sudo bbappend because /etc/sudoers.d will conflict when creating the root filesystem if you add the drop-in in another recipe. This means that when you’re creating a user in a non-sudo recipe and add the drop-in conf in the sudo recipe you’ll have the user creation & configuration handled in two different places, which is perfect for forgetting to maintain things.

Disabling root-login

Disabling root login is useful for a multitude of reasons.

  1. You usually want to make sure that no user has the full command set available
  2. You also want that the users’ privileged actions to end up logged in auth.log when they use sudo
  3. In general, it’s “a bit of a risk” to allow users to log in as root

Disabling the root user can be achieved in multiple ways, and I’m going to cover three of them here: fully locking the root account, removing the login shell, and disabling the SSH login. In theory, locking the root account should be enough, assuming non-root users cannot unlock it, but I’ll present all the methods as they may be useful for different situations.

First, disabling the root account. This is a method I found from this StackOverflow answer, kudos there. The simple snippet below pasted in the image recipe should lock and expire the root account:

inherit extrausers
EXTRA_USERS_PARAMS:append = " usermod -L -e 1 root; "

Then, removing the login shell. Or more like setting the login shell to nologin. This isn’t strictly required after locking the root account but can be achieved by creating a bbappend shown below to base-passwd recipe:

do_install:append() {
    # What this magic sed does:
    # "In the line beginning with root
    # replace /bin/sh with /sbin/nologin
    # in file passwd.master"
    # (passwd file gets generated from this template during install)
    sed -i '/^root/ s/\/bin\/sh/\/sbin\/nologin/' ${D}${datadir}/base-passwd/passwd.master
    # sry for the picket fence

    # I think it's peak sed that I need a four line
    # comment to explain a one-liner
}

And finally, disabling the SSH login. You should always disable the root login over remote networks. If you need, you can just disable the SSH login for root and still allow other service users to log in as root. This is still a better option than doing nothing, as this will prevent logging in as root remotely and attackers would need to know two passwords to gain root access (one for the service user, and the other one for the root user).

Disabling the root login varies a bit between the SSH servers. Dropbear disables root login by default with -w parameter in DROPBEAR_EXTRA_ARGS variable located in the /etc/defaults/dropbear file (note that if you have debug-tweaks enabled, the file actually contains -B, allowing root-login). You can overwrite the file with your own dropbear.default file during the build if you want to add more options.

Similarly, OpenSSH-server disables root logins with passwords if debug-tweaks is removed from the IMAGE_FEATURES (and allows them if it’s present). This is achieved by sed-editing SSH configuration files. If you want to see how this is exactly done, check ssh_allow_root_login function in meta/classes/rootfs-postcommands.bbclass (part of poky).

However, it’s worth noting that this default behaviour doesn’t prevent public key authentication. If you want to disable that as well, you can add this function as a rootfs post-process task to the image recipe. And if needed, it could of course be modified to work in a bbappend as well.

# The function here searches sshd_config and sshd_config_readonly files for a
# commented line containing PermitRootLogin, and replaces it with "PermitRootLogin
# no" to prevent any sort of root login.

disable_rootlogin() {
    for config in sshd_config sshd_config_readonly; do
        if [ -e ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config ]; then
            sed -i 's/^[#[:space:]]*PermitRootLogin.*/PermitRootLogin no/' ${IMAGE_ROOTFS}${sysconfdir}/ssh/$config
        fi
    done
}

ROOTFS_POSTPROCESS_COMMAND += "disable_rootlogin;"

The thing to note about disabling root login is that too lenient sudo- or filesystem-permissions for non-root users can make the whole thing useless. For example, if a service user has sudo access to passwd command they can easily unlock the root account. If a service user has write permissions to /etc they can set the root’s login shell and edit SSH configuration. And finally, disabling root login like shown in this text does nothing to prevent shell escapes made possible by incorrectly set sudo-permissions.

Might as well roll out the red carpet

That’s it for this time! Hopefully, these tips helped you to achieve the minimum security in your Yocto images. Do these code snippets make your system unhackable? Not even close. Do they make it slightly more inconvenient? I hope so (but no guarantees for that either). But at least now you can show your Yocto image to an independent professional security consultant and say that you tried before they tear the whole system to pieces.

You can find the second part of the Yocto hardening series here. It’s about fixing CVEs.

Yocto and WSL, part 3: WSL vs. VMWare

Read the previous parts of “Yocto and WSL” series:
Yocto? On WSL2? Easier than you think!
Yocto and WSL, part 2: The Graphic Boogaloo

The ultimate showdown of the ultimate destiny!

All the good things are trilogies. Star Wars original trilogy, Nolan’s Batman trilogy, and The Hobbit trilogy. And now, Yocto & WSL trilogy. As per usual with the trilogies, the last part may be a bit of a letdown for the hardcore fans. This time we’re not doing really anything technical or exciting. Instead, we’re comparing some numbers while trying to decide if this whole exercise was worth it or not.

To judge the usefulness of WSL in the Yocto build context were doing a comparison between WSL machine and a “regular” virtual machine to see which one is actually better for building Yocto (if you’re forced to use Windows). Better means faster in this case. For this purpose, I created a VMWare virtual machine that matched my WSL machine’s specs: 4 cores, 8GB, and enough disk space to accommodate the build area. For the operating system I chose Ubuntu Server 20.04.4, and as a reference build target I used the simplest core-image-minimal recipe with both machines.

But wait, there’s more!

As a bit of an extra, I’ll be adding a comparison to a cloud build server as well for the same price! For this purpose, I prepared a similar 4-core 8GB build server on Hetzner. This is a virtual machine as well, but it’s interesting to have more things to compare to. Especially because the cloud instances provide an option to scale up in case they aren’t powerful enough. Also, like WSL and VMWare, they can be used as an alternative for a Linux build machine. Without further ado, let’s get into crunching numbers.

First, to actually make a worthwhile comparison between different build times, I’m going to separate the process of downloading the sources from the actual build itself. I did three runs of downloads because “third time’s the charm”. VMWare was using a NAT network connection here. The separation was done because the time it takes to download the sources depends on the load on the source code servers, how much my neighbor happens to be using the internet at the given moment, and the alignment of the stars. Or what do you think of these results:

MeanMedianRun 1Run 2Run 3
WSL44m 4s52m 1s52m 1s59m 29s20m 42s
VMWare46m 20s45m 5s37m 28s56m 29s45m 5s
Cloud build35m 18s33m 40s33m 40s41m 49s30m 26s

There were two packages that became bottlenecks in these tests: cross-localedef-native and linux-yocto. Especially the former. Other packages were downloaded in a few minutes but cross-localedef-native took almost always at least half an hour to download. In the WSL run #3 cross-localedef-native took only about 15 minutes to download, and instantly the whole download process was a lot quicker. In general, I would still boldly claim that Hetzner build is potentially the fastest on average. Or at least it would make sense, as the Internet pipes in their data centers are most likely wider than mine at my home.

That good ol’ YouTube player brings a nostalgic tear to my eye

Then, the actual build times. For these, I actually did four runs, because “third time’s the charm and one more for a good measure” (this blog text isn’t quite as scientific as I make it out to be). As mentioned earlier, I built the core-image-minimal for all these runs. These builds were done without downloading the sources in an attempt to get the build times to be a bit more stable. Here’s the table:

MeanMedianRun 1Run 2Run 3Run 4
WSL109m 15s106m 48s106m 11s121m 26s107m 26s103m 37s
VMWare125m 51s125m 51s126m 33s126m 32s125m 10s125m 9s
Cloud build106m 30s105m 35s111m 10s105m 29s105m 3s105m 40s

As perhaps expected, the virtual machine was constantly the slowest option. What however is slightly surprising is that the WSL was almost as fast as the cloud build server with the same resources. Or actually, with better resources as the build server had an SSD disk while WSL was running on an old HDD I bought from a friend of mine for 30 euros some years ago.

But yeah, in the light of this evidence I’d say that if you’ve got some dollars to spare, getting a cloud build server is the best option for building Yocto (if you can’t get a native Linux build machine that is). It’s not only the fastest option, but it’s also scalable, so when you’re building something else than the “trivial” core-image-minimal you can easily increase your build performance. I’ll admit that the time difference isn’t big in this comparison, but once you build more complex images the differences in the build times will become more apparent.

The cloud server used in this comparison costs about 15€ a month. If you want to give the cloud build a go, you can use this super-neat affiliate link to sign up and get 20€ worth of free credits. And maybe give me 10€ in the process if you happen to spend some actual money on the platform.

Shooting Easy Money GIF by iHeartRadio San Francisco - Find & Share on GIPHY
How well I handle my money every time the paycheck comes in

However, if you don’t have enough dollars for about two avocado toasts or three lattes every month, WSL is a better alternative than VMWare for building Yocto. Virtualbox seemed to be even worse than VMWare as it didn’t even want to build anything for me. It just spat out constant disk I/O errors that I simply couldn’t get fixed. Different caching options, physical disk drives, or even guest operating systems didn’t fix that.

But this is not all! You thought that you don’t have to read any more of these tables, didn’t you? I did some generic performance tests to get a bit better understanding of the different build machines, so I’ll add their results here as a bit of a bonus. The first one is the network speed test. I used the Ookla’s Speedtest command line tool for these. The values are averages of “about a dozen” runs.

LatencyDownloadUpload
WSL3.76ms104.22 Mb/s59.85 Mb/s
VMWare3.94ms104.13 Mb/s59.88 Mb/s
Cloud build0ms9266 Mb/s9220 Mb/s

There’s a bit of a difference in the latency between WSL and virtual machine, but the differences in the download and upload speeds are marginal. Cloud build is something entirely different than the other two, as expected.

Next is the hard drive speed, both reading and writing. For this, I used dd command-line tool. For the write test I used this command:

dd if=/dev/zero of=test bs=1G count=4 oflag=dsync

For the read test I used this command:

dd if=./test of=/dev/null

For the un-cached timings I cleared the caches with this command between the runs:

echo 3 | sudo tee /proc/sys/vm/drop_caches

For the cached timings I didn’t. Here are the results:

WriteWrite (cached)ReadRead (cached)
WSL91.6 MB/s93.7 MB/s91.9 MB/s628 MB/s
VMWare50.6 MB/s107.5 MB/s338 MB/s559 MB/s
Cloud build1.2 GB/s1.3 GB/s603 MB/s1.2 GB/s

This is where the cloud machine’s SSD seems to shine, even if it didn’t do so well when doing the actual build work. The differences between WSL and VMWare on the other hand are a bit varied. What these results seem to suggest is that the uncached write performance is quite important for Yocto build performance. However, results also seem to be showing that the mass storage performance isn’t quite as important for the Yocto builds.

Finally, the CPU speed test. This was done with sysbench, using the command sysbench cpu --threads=4 run. Here I compared the CPU events per second:

CPU events per second
WSL5359.31
VMWare5284.58
Cloud build14582.81

All things considered, it’s slightly surprising how small the cloud build’s margin of victory when comparing the build times actually was. Especially considering how much more performant the CPU is according to the raw numbers. The most important thing actually seemed to be the number of cores, because a three-core cloud machine would already be slower for Yocto builds than a four-core VMWare virtual machine. So please remember this when you’re setting up your Hetzner cloud instance after you’ve registered through this link (I promise this affiliate pushing won’t become a thing).

That’s all for now. This is also most likely the last text about Yocto builds with WSL for time being. Next time it’s either something different that’s not related to Yocto at all, or then it’s something similar that is related to Yocto but not WSL.

tl;dr: WSL is faster than a virtual machine, but I recommend using cloud servers if possible

Writer of this blog post is wondering how he ever managed to finish his thesis

Yocto and WSL, part 2: The Graphic Boogaloo

Read the other parts of “Yocto and WSL” series:
Yocto? On WSL2? Easier than you think!
Yocto and WSL, part 3: WSL vs. VMWare

So, in the previous part, we got the Yocto built with WSL and tested it out with a text-based terminal interface. This time we’re going to improve things a bit by getting the graphics working! Like every engineer always says, “it’s really nice to have this fancy graphical interface with buttons that have rounded corners instead of an efficient, text-based interface”.

(After publishing this text it was brought to my attention that WSL on Windows 11 actually supports GUI applications natively. I’m not sure how this works with QEMU but I’d recommend trying that out before going through all the hassle mentioned in this text. If you’re still stuck on Windows 10 like me these steps are at least currently applicable)

The process of getting graphics up and running in the WSL consists of three steps:

  1. Make a hole into the Windows firewall (starts promisingly) so that WSL can communicate with an X server
  2. Launch an X server with the access control disabled (this sounds great as well)
  3. Set the environment variables in WSL

The instructions I used to achieve this can be found in this great blog post. However, for the sake of documentation and out of the fear of broken hyperlinks, I’ll go through the steps here as well.

The hole in the firewall needs to be made because the networking in WSL2 is implemented in a way where the traffic is considered to be coming from an external source (as opposed to WSL1). The firewall rule can be set using the following steps:

  1. Search “Windows Defender Firewall with Advanced Security” from the Start Menu
  2. Click “Inbound Rules” on the left-hand side menu, and then click “New Rule…” on the right-hand side menu
  3. Select rule type to be “Port”, set the port to 6000 and give it a good name. The rest of the settings should be good by default (TCP protocol & allowing all kinds of access)
  4. Right click on the newly created rule in the rule list, select Properties and from there select Scope-tab. The limited scope prevents unwanted entities from exploiting the poor firewall’s new rule. Check the image below for Scope-settings:
Basically, we’re limiting the connections just to private IP-addresses

And just like that, the firewall should allow the traffic from WSL2 to the X server. The X server that we haven’t even installed yet. I have been using VcXsrv X server, and the next step is downloading and installing it. After VcXsrv has been installed we can continue.

As briefly mentioned earlier, the access control needs to be disabled when launching the server to make things go smoothly. Just use -ac flag when launching the X server from the command line to disable access control. The full command to launch VcXsrv from a Windows terminal is:

vcxsrv.exe :0 -multiwindow -wgl -ac -silent-dup-error 

The first parameter sets the display number to zero, the second one allows multiple windows, the third one doesn’t seem to be necessary after a quick test but I’ll still leave it here because it was in the original post, the fourth parameter is the important one that disables the access control and the last one prevents errors if launching multiple VcXsrv-servers. After getting the server running we’re almost ready to get into the exciting world of graphic content. Open up a WSL session, and export the following two variables (in WSL terminal):

export DISPLAY=$(awk '/nameserver / {print $2; exit}' /etc/resolv.conf 2>/dev/null):0
export LIBGL_ALWAYS_INDIRECT=1

The first variable sets the display we want to use (basically WSL nameserver address and display 0) and the second variable should prevent skipping X server when calling draw commands. I’m not 100% what that means, but it sounds important so it’s better to set it.

And now, finally, we’re ready to check out the graphical interface! If you’ve closed your WSL session in the past two months it took me to write this second blog post, first head to your poky-folder and run source oe-init-build-env again. Then you should be able to run:

runqemu
#look, no no-graphic option this time, wowzers!
Wait, I was promised graphics, what is this? I want to click icons and drag-n-drop stuff around.

Well, the thing is that the core-image-minimal image we built in the previous blog post doesn’t have a graphical interface built. While technically this new window is a graphical, emulated window, it’s not a graphical user interface. To get an image with these capabilities, we need to build core-image-sato instead (sorry).

So yeah, navigate to your build-folder and run bitbake core-image-sato. Even though the build reuses some of the artifacts from the previous build this will take some time, so be prepared to find something else to do for a while. After the build has been completed, you should be able to run runqemu again, and see something like this:

Awww yisssss

The apparent problem is the quite significant latency introduced by QEMU, and running QEMU on top of the WSL doesn’t do the situation any favors. Technically you can run the image also on a Windows native QEMU, but I didn’t see any noticeable difference in the GUI performance. My method of analysis was not the most scientific one though: I just pressed a button and tried to internally analyze how frustrated I got before something happened in the GUI. With both WSL QEMU and Windows native QEMU I got “quite frustrated”, so I’d call it a tie.

I didn’t do any actual benchmarking on if there’s a difference in the computing performance, perhaps in some upcoming blog post I could write a bit about that. It seems like the Windows native QEMU lacks some options (I couldn’t get the networking working (more like notworking amirite)), so it may be better to stick to WSL QEMU.

That’s all for this time. Hopefully, you learned something from this post, I at least did. Once again big shout-out to Skeptric-blog, plenty of other interesting texts in there as well.

Part 3 is now available here!

PS: I disabled the comments on this blog. It seems like the only ones writing those were freaky spambots. If you’ve got something to ask or say you can reach me on LinkedIn.

Yocto? On WSL2? Easier than you think!

Read the other parts of “Yocto and WSL” series:
Yocto and WSL, part 2: The Graphic Boogaloo
Yocto and WSL, part 3: WSL vs. VMWare

(Sorry for the clickbaity title, but the WordPress Headline Analyzer gave it the best score I’ve ever seen so I can’t change it)

Let’s begin this post with the basics. What are these words and acronyms in the title? Some nerd stuff? Well, yes. Yocto is a project/tool that can be used to generate a cross-compilation toolchain and a firmware image for an embedded system. This sort of work includes never-ending woes with bootloaders, Linux, and much, MUCH more. Windows Subsystem for Linux (= WSL) is “a compatibility layer for running Linux inside Windows”. It’s not really a virtual machine (at least according to Microsoft) or a container, despite running a virtual machine and virtualization being its core (or so I understood). What is it then exactly?

Well, let’s call it a virtualization layer and settle for a fact that it’s a way of running a Linux-like environment inside Windows with relative ease. And as you can guess from the Yocto project’s description, we’re going to be needing plenty of that Linux.

First things first: setting up WSL2. There are plenty of tutorials and official documentation on this topic, so I’m not going too much in-depth with this one (although I’ll go through the issues I faced along the way). I recommend checking at least this official documentation out (There actually seems to be an easier way of installing WSL these days, but I haven’t tried it). In a nutshell, the steps that should be done are:

  • Enable WSL feature in Windows (and reboot)
  • Install WSL2 kernel update (most likely reboot after this one won’t hurt either)
  • Set WSL2 as the default WSL version (reboot isn’t really needed here, but it’s always fun to stare at that picture of a cave that’s the log-in screen background and think how you could be somewhere else. Somewhere warmer where the sun shines)
  • Install the desired distribution.

It’s Ubuntu that you desire. Or at least I recommend it because it’s the most widely supported Linux distro [citation needed] and the Yocto guide explains the steps a bit more specifically for Ubuntu users. Other distros still work, or at least should work. Yocto claims to support “recent releases of Fedora, openSUSE, CentOS, Debian, or Ubuntu”, whatever that means.

However, this installation phase was where I started to face problems. When installing Ubuntu I got the following error:

Installing, this may take a few minutes…
WslRegisterDistribution failed with error: 0x80004005
Error: 0x80004005 Unspecified error

Despite the ominous-sounding “Unspecified error” this was caused by the simple fact that I didn’t have enough disk space on the disk that contained AppData-folder (I guess this defaults to C:). There seem to be some other explanations for this error on the Internet, but I recommend freeing space if you’ve filled your C: with junk (as I assume every Windows user has done). 2GB of free space wasn’t enough, 7GB seemed to do the trick.

After doing this and retrying, I got the following error instead:

WslRegisterDistribution failed with error: 0xc03a001a
Error: 0xc03a001a The requested operation could not be completed due to a virtual disk system limitation. Virtual hard disk files must be uncompressed and unencrypted and must not be sparse.

This one was a bit trickier, but I found the fix here. Basically, you need to navigate to %LOCALAPPDATA%/packages/, check the (advanced) properties of CanonicalGroupLimited.UbuntuonWindows-folder and ensure that it isn’t compressed or encrypted.

What, an actually useful image in this blog? Didn’t see this one coming. Fortunately, my bad paintwriting makes it feel a bit less useful.

And after these trials, the Ubuntu image should finally install without further problems. Or at least it did for me, YMMV. Once you get the WSL up and running, I recommend playing around with it a bit if it’s not familiar to you yet. If you don’t know where to start, this video can be useful (About 3:40 onwards). To be honest, so far the biggest issue I’ve had with WSL is how simple it is, I was expecting it to be a lot more complex and cumbersome.

After getting the basics of WSL down it’s time to get started with Yocto. Well, almost. As you may guess, the Yocto project takes up disk space. A lot it. The official requirement at the time of writing is 50GB. However, as you may remember from the earlier chapters, I have barely 7GB free on the disk where the WSL image gets installed by default.

A bit of googling revealed two useful options to the Windows’ WSL command-line tool: export and import. These can be used to pack the image and then load it in a non-default location. unlink option could be used to get rid of the obsolete machine. So in short, the Powershell commands to be run on Windows are:

wsl --export Ubuntu E:\yoctotest\testDistro.tar
wsl --unlink Ubuntu
wsl --import UbuntuCustom E:\yoctotest\UbuntuCustom E:\yoctotest\testDistro.tar
#Or something similar, depends a bit on what you actually want and what drives you have

As you may have noticed at this point, all of my problems were caused by the lack of disk space on C: drive, so if you don’t have that problem I guess your experience will be a lot smoother.

Just to prove to you that I have learned absolutely nothing from this “out of disk space”-experience, here is a 1.5MB gif.

Finally, after getting the WSL image ready and in the correct location, it is time to properly start working on the Linux side of things. There’s one more thing to do before moving to Yocto: updating apt and installing zstd. I’m not sure why zstd isn’t listed in Yocto requirements, but my build failed without it. These commands can be run in the Ubuntu shell to get the things ready for Yocto:

sudo apt update
sudo apt install zstd

After that, it’s mostly just a matter of following the Yocto quick start, with the difference of building the core-image-minimal instead of core-image-sato. Sato-image is the GUI-image, but we don’t have graphics yet so it’s better to stick to core-image-minimal. In a nutshell, the steps for Yocto-build are:

  • Install dependencies (no need to reboot)
  • Clone poky-repository and check out the latest LTS (=long-term support) branch dunfell (interestingly, no reboot)
  • Source the build environment (don’t reboot, you’ll lose the sourced environment)
  • Build the core-image-minimal. Did I mention core-image-minimal yet?

Yocto-build with WSL will print a warning that you better be sure that you have enough disk space, but besides that it should work normally. With the default settings, the build will eat pretty much all of the resources your machine has (it’s been optimized that way), so don’t be afraid if things start to freeze a bit. Especially memory tends to be a bottleneck, partially because of WSL. However, I was able to watch a Youtube video most of the time while waiting for the thing to finish, so I guess it doesn’t render the computer completely useless. During the processing of the heavier components it may be better to do something with a video game console of your choice though.

Bleep bloop

And if the stars are aligned, the build will complete in an hour or two (or six)! The dunfell-branch is usually quite stable (as LTS branches should be), so the build shouldn’t be in a broken state. After the compilation is done, you can try out the generated image with the following command:

runqemu -nographic

runqemu command is Yocto’s wrapper for QEMU, which is an emulator that can be used to run the built images (and plenty of other things outside Yocto as well). runqemu helps to automatically find and set up the image for the emulator. -nographic option prevents loading a graphical shell in a separate window, as the graphics are not set up for WSL yet.

And that’s it! It is easy once you know how to do it. Like pretty much everything else I guess. But I was pleasantly surprised that Yocto officially supports building in WSL, so there was no need for (dirty) hacks. The memory seems to be a bit of a problem, but that’s always the case. In the next post there’ll hopefully be some graphics!

Part 2 is now available here!