This is a sequel to a similarly named blog post Black-Box Fuzzing Kernel Modules in Yocto. In that blog post, I briefly went through what fuzzing is and presented an easy but fairly naive approach to fuzzing. This time I will present a more refined approach to fuzzing using Syzkaller.

Syzkaller

Syzkaller is an unsupervised coverage-guided kernel fuzzer from Google. Simply put, it creates programs that perform different kernel syscalls in varying order with evolving parameters and then analyzes the coverage to see what test programs reach new code coverage. These programs are added to the corpus, which can be used to create even more programs with even larger coverage. Sounds simple? Well, the architecture looks like this:

Could be worse. There are two core components in the system, syz-manager and syz-executor. syz-manager runs on the “main fuzzing server”, and syz-executor runs on the fuzzing target. This can either be an actual hardware device or a QEMU emulator system that syz-manager controls. syz-manager is responsible for the fuzzing work, generating the test programs, and storing the corpus & crashes. The executor receives the test programs, runs them and reports back the results. This communication happens over a network interface.

Smoke Test

A good way to smoke test the setup is to run Syzkaller with the default built-in syscall definitions. This ensures that your image is suitable for fuzzing without having to worry that your additions are breaking things. I’m using Yocto to build an image for x86_64 QEMU. You can use pretty much anything to build the image as long as you have the kernel source and object files, and a disk image for rootfs and of course a kernel. In addition to x86_64 there are plenty of other supported architectures. The setup guide lists arm, arm64, and riscv64 for example.

Building the Image

First, the disk image requirements. The image should have an SSH root login that either has key authentication or a passwordless login. This naturally means that there needs to be networking support. Optional, but strongly recommended, is a DHCP client for getting an IP address. This is required for QEMU port forwarding that syz-manager utilizes. Yocto’s default QEMU core-image-base image does not require any special changes to the disk image as it has network capabilities, passwordless root, and a DHCP client.

The kernel should be built with instrumentation and a few other debugging options enabled. The exact options depend on the target architecture, the x86_64 setup guide lists “at the very least” the following as a requirement:

CONFIG_KCOV=y
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y

The full documentation of the kernel configuration can be found here, and it lists plenty of additional configuration flags. I added both the minimum and the maximum configuration to my meta-fuzzing layer. I’ll be using the minimum configuration.

Writing Syzkaller configuration

Next, we need to write the configuration for Syzkaller. Let’s pick the example QEMU-configuration, and modify it a bit. This configuration defines the web interface address, location of required files, the syscalls to call, and the fuzzing target. We are going to use emulated QEMU targets in this test:

{
	"target": "linux/amd64",
	"http": "<YOUR-IP-HERE>:56741",
	"workdir": "./workdir",
	"kernel_obj": "/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build/",
	"kernel_src": "/poky/build/tmp/work-shared/qemux86-64/kernel-source",
	"image": "/poky/build/tmp/deploy/images/qemux86-64/core-image-base-qemux86-64.rootfs.ext4",
	"syzkaller": ".",
	"disable_syscalls": ["keyctl", "add_key", "request_key"],
	"procs": 4,
	"type": "qemu",
	"vm": {
		"count": 2,
		"cpu": 2,
		"mem": 2048,
		"kernel": "/poky/build/tmp/deploy/images/qemux86-64/bzImage",
		"cmdline": "ip=dhcp"
	}
}

Different configuration items are quite well documented in the manager configuration and QEMU VM configuration files. The amount of virtual machines and processes here is quite low because I have a poor PC for running these tests, so you may want to increase them.

This configuration assumes you have the Poky folder directly under the root folder, adjust this if necessary. Also, the configuration assumes that the process is launched in the root of the Syzkaller repo. The ip=dhcp cmdline option is also worth noting. This will be passed to the kernel, and it should ensure that the Ethernet interface uses DHCP to get IP address from QEMU. I think you can also hardcode the IP address if you know what it should be. tcpdump can be used to check the incoming ARP requests to see what the IP address is expected to be. There may be an easier way, but that’s what I did when poking around.

There’s one weird hack I had to do though. In my experience (and I may be wrong here), it seems that Syzkaller expects a certain format from the kernel source tree, and that expected format is not the actual structure of the kernel source. There seems to be an expectation that under the path defined in kernel_src there is a usr/src/kernel folder that points to the source, otherwise the coverage information generation will fail. However, if I move the kernel source to a usr/src/kernel folder, the report generation will fail because the scripts folder cannot be found anymore. To create a suitable directory structure, I used the following script:

cd <KERNEL_SOURCE>
mkdir -p usr/src/
cd usr/src/
ln -s ../../. kernel

Again, I’m not sure if this is necessary, it may be that I misconfigured something, but I had to do this to get the syz-manager running, reporting coverage, and formatting error reports.

Running Syzkaller

Once the configuration is ready, we can start to wonder what to do with it. The first thing is fetching the Syzkaller source from GitHub. Then we can compile Syzkaller using syz-env that is a Docker script that can be used to ensure that the build environment is the expected one:

./tools/syz-env make

This of course requires Docker. You can also use make directly, but then you need to take care of the build dependencies yourself. If you’re doing cross-arch testing, you need to define the target variables, so check out the setup guide for those. Once the build completes, copy the configuration created in the previous chapter to the root of the Syzkaller repo, and run:

./bin/syz-manager -config <CONFIG_FILE>

syz-manager should start up, and after a moment the QEMU machines should boot up as well. If you navigate your web browser to http://<YOUR-IP-HERE>:56741, you should see the web interface that shows the status of fuzzing, collected coverage and crashes:

If not, you can try adding -debug option to the syz-manager command to see what’s going wrong. Also, for some reason my web interface doesn’t load if there are no crashes, there’s just an error message about missing crashes folder. So, I had to create manually ./workdir/crashes folder, and then it seemed to work just fine.

Once you get to the web interface, you can leave the fuzzer running for a while to see how it works and explores new coverage paths while increasing corpus. It’s quite unlikely it’ll find a crash, as it’s fuzzing against a mainline kernel, but if it does, you have potentially found a kernel bug!

Adding Custom Syscalls

However, it’s not very interesting to fuzz the mainline kernel. It’s been so done (and is constantly being done). What’s more interesting is fuzzing our custom kernel module, and seeing if the Syzkaller can find a poorly hidden error from it.

I asked ChatGPT to write a small driver that has an IOCTL interface, and then I added a bug to it. The single command in the interface takes a string as an input. It tokenizes the string, and if the string contains five commas, an invalid free will be performed. There are some extra checks to guide the fuzzer towards the crash.

In-tree vs. Out-of-tree Module Build

While writing this text, I had to consider the in-tree vs. out-of-tree kernel module building. The difference is that an in-tree module is built as a part of the kernel build, and an out-of-tree module is built against the kernel headers after the kernel has been built. The out-of-tree method allows compiling modules for pre-built kernels, assuming the headers are available. For example, if you’ve ever built a Hello World -module for Ubuntu, you’ve most likely run apt-get install linux-headers-`uname -r`. This means you’ve pulled the development headers for building the out-of-tree module, and you’re not compiling an entire kernel.

Since Yocto builds the kernel it is quite easy to build the modules in-tree. This has a few advantages. For example, the module can be easily built as a built-in feature. From the fuzzing point of view, obtaining the coverage for the in-tree modules is a lot easier. Also getting the correct line numbers for crash reports is a lot simpler because there’s no need to decrypt offsets with objdump. So I’d recommend building the modules in-tree for fuzzing.

The example driver has both a module recipe (because that’s what I tried first) for out-of-tree build and linux-yocto append for in-tree build. It’s a bit silly way of supporting both methods, but at least it works (until it doesn’t). To add the IOCTL example module to the in-tree kernel build, add this line somewhere in your configuration:

IOCTL_STRING_PARSE_INTREE = "1"

Describing New Syscalls in Syzkaller

This is where the magic happens. Defining our own syscalls to Syzkaller so that it knows about the non-default syscalls and can create fuzzing sequences utilizing them. To achieve that, we need to write the syscall definitions in Syzkaller’s syntax, which is kind of simple, but still a bit frustrating to get right. Fortunately, there are plenty of examples. After staring at those, and reading this blog post, here’s what I came up with:

include <linux/fcntl.h>
include <linux/ioctl_string_parse.h>

resource fd_vuln_ioctl[fd]
openat$ioctl_string_parse(fd const[AT_FDCWD], file ptr[in, string["/dev/ioctl_example"]], flags const[0x2], mode const[0x0]) fd_vuln_ioctl
ioctl$IOCTL_STRING_PARSE_CMD(fd fd_vuln_ioctl, cmd const[IOCTL_CMD_PARSE_STRING], arg ptr[in, string])

A file with this content should be added to <PATH_TO_SYZKALLER>/sys/linux. The file name can be arbitrarily chosen, but it should have the .txt suffix.

The includes here are from the kernel source tree. fcntl.h is for the AT_FDCWD macro, and our own ioctl_string_parse.h is for the IOCTL_CMD_PARSE_STRING that contains the IOCTL command to use when making IOCTL calls to our driver. resource defines the file descriptor resource that is shared between the two other calls. openat opens the ioctl_example device in read/write mode with no special mode flags. This call should be quite static as the goal isn’t to fuzz the openat command, so most values are constants.

The final definition is the IOCTL command on the opened device, using the IOCTL_CMD_PARSE_STRING defined in the header as the command, and a random string as an argument for the IOCTL call.

Once the definitions are done, it’s time to compile them into Syzkaller. We first need to compile a tool called syz-extract, then extract syscalls from the our .txt file to a .const file, update the generated code, and re-compile the binary. The four commands below do exactly that (for 64-bit Linux systems)

./tools/syz-env make bin/syz-extract
./bin/syz-extract -os linux -arch amd64 -sourcedir /poky/linked-linux-src/usr/src/kernel -builddir /poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build <CUSTOM_DEFINITIONS>.txt
./tools/syz-env make generate
./tools/syz-env make

Note how you need to pass the build and source directories for extracting syscalls into .const file that will be used in the code generation. Again, update paths and file names if necessary

After that, we should be almost good to go. The configuration file still needs some tweaking. We do want to focus only on our custom syscalls, and we do not want to mutate the openat command. The following configuration should work:

{
	"target": "linux/amd64",
	"http": "<YOUR-IP-HERE>:56741",
	"workdir": "./workdir",
	"kernel_obj": "/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build/",
	"kernel_src": "/poky/build/tmp/work-shared/qemux86-64/kernel-source",
	"image": "/poky/build/tmp/deploy/images/qemux86-64/core-image-base-qemux86-64.rootfs.ext4",
	"enable_syscalls": ["openat$ioctl_string_parse", "ioctl$IOCTL_STRING_PARSE_CMD"],
	"no_mutate_syscalls": ["openat$ioctl_string_parse"],
	"syzkaller": ".",
	"procs": 8,
	"type": "qemu",
	"vm": {
		"count": 2,
		"cpu": 2,
		"mem": 2048,
		"kernel": "/poky/build/tmp/deploy/images/qemux86-64/bzImage",
		"cmdline": "ip=dhcp"
	}
}

After that, the fuzzer can be started with the same command as before:

./bin/syz-manager -config <CONFIG_FILE>

Now, after waiting a few minutes, there should be some crashes visible:

Interestingly enough, if we are using the maximum debug kernel configuration these get reported as “potential deadlocks”. I guess enabling 40+ debug flags has some side effects. (As a side note, after enabling all the configuration items the basic runqemu machine boot time slows down from 10 seconds to 5 minutes). Regardless of which configuration we’re using, if we check out the report we can see the expected root cause for the crash:

It’s quite fascinating to watch the coverage information and see how the fuzzer approaches the problematic line when it attempts to increase the coverage. Fascinating in the same sense it’s exciting to watch paint dry:

Extra Bonus Issue

Can you see what’s the issue with this code:

input_buffer = kmalloc(input_len, GFP_KERNEL);
switch (cmd) {
    case IOCTL_CMD_PARSE_STRING:
        ret = copy_from_user(input_buffer, (char *)arg, input_len);
        if (ret != 0) {
            printk(KERN_ALERT "Failed to copy string from user space\n");
            kfree(input_buffer);
            return -EFAULT;
        }

        // Some processing happens here

        kfree(input_buffer);
        break;

    default:
        printk(KERN_ALERT "Invalid IOCTL command\n");
        kfree(input_buffer);
        return -EINVAL;
}

I didn’t, and neither did ChatGPT when it suggested this for the first version of the driver. However, when Syzkaller calls this kind of IOCTL function multiple times in a rapid fashion it results in some unexpected invalid-frees. I guess constantly performing allocations and frees in an IOCTL command isn’t the best idea. The second implementation of the driver uses a memory pool to avoid having to allocate memory after initialization, and that seems to work. But yeah, another point for fuzzing for finding a bug from a seemingly functional code.

But Wait, There’s More!

In addition to this, I also created a test program with ChatGPT for debugging purposes. Later I realized that this could also be used for black-box fuzzing the example kernel module. So, if you want to, you can run the following script to fuzz the module with Radamsa (assuming you have installed Radamsa, check the black-box fuzzing blog text for more info):

while true; do 
    test-program-ioctl "$(echo ,,, | radamsa)";
done

Mandatory Final Chapter

This should cover all for now. With these instructions, you can hopefully fuzz your kernel module with Syzkaller. However, this only performs fuzzing on virtual QEMU targets. Sometimes it’d be better to fuzz on the actual hardware, especially if using specialized hardware. I’ll cover that in a follow-up text. If you want to get notified when that text goes out, consider joining my mailing list. Thanks for reading, and happy bug-hunting.

It’s been almost ten years since I wrote my thesis. It was about guided fuzz testing, and as usual, I have done zero days of actual work related to the topic of my thesis. However, I was feeling nostalgic one day and thought that I’d fire up a good ol’ fuzzer and see what I could do with it. In the end, not much. But it was fun to try to break something and relive the golden days of my youth.

To shake things up a bit, this time I tried fuzzing a Linux kernel module in a Yocto image, because it seems that I just can’t help but cram Yocto into every blog post I write. But let’s start from the beginning.

What Is Fuzzing?

Fuzzing is a type of testing where more or less broken input is used to check how a program behaves in unexpected situations. Usually, the process consists of collecting input samples, good or bad, running them through a fuzzer that does “something” to the sample, and then feeding this mystery sample to the program being tested. Well-behaving programs handle the erroneus input gracefully, but the badly behaving programs may hang, crash, or even worse, use the bad input like nothing is wrong.

Fuzz testing can be subcategorized into a few different groups: black-box, grey-box and white-box fuzzing. In black-box fuzzing there is no knowledge of the internals of the program, and no test feedback is used to guide the fuzzer. On the other hand, when using the white-box fuzzing the full knowledge of the program flow and protocols is available. In grey-box, there is no “deep” knowledge of the program, but for example code coverage may be used to guide the fuzzer.

As one can guess, a black-box fuzzer is the simplest to set up, but generally it is inefficient. White-box fuzzing is the opposite, where the initial effort may not even be worth it in the end. Grey-box, once again, lands somewhere in the middle. The instrumentation and feedback may require some effort, but it is (usually) worth it in the form of improved results.

Fuzzing on Embedded Target

Even though fuzzing can reveal some fascinating bugs, it’s worth noting that performing fuzzing on an embedded device may not always be a good idea. Usually, the efficiency of the fuzzing is directly proportional to the amount of tests being run per second. “Real” computers tend to be more powerful, resulting in more tests getting churned out compared to the embedded systems. The requirement for speed is especially true for black-box fuzzing which is basically brute forcing bugs out of the system. Therefore, you may want to consider fuzzing high-level application code on a more powerful computer, or in a virtualized environment to reveal more complex issues.

Fuzzing on the actual hardware makes the most sense in the following scenarios:

• The code you’re testing relies on some architecture-specific functionality
• The code relies on some hardware functionality that cannot be easily simulated
• The hardware can generate samples and run target programs with “tolerable” efficiency
• You want to do a quick smoke test type of fuzzing run

However, despite trying to talk you out of fuzzing on the target HW, I personally think it’s a good idea to give a quick black-box fuzzing session at least a try. It can reveal some low-hanging bugs, and setting up a black-box fuzzer takes little to no effort. Just be aware of the limitations, and the fact that it’s not going to be as efficient as it could.

Finally, it’s worth knowing that things can go really wrong with fuzzing, so consider the potential risks, and if there’s a possibility of some hardware breaking. It’s usually unlikely, but aggressively fuzzing for example a poorly written device driver can result in bricking.

Radamsa

There are plenty of black-box fuzzers available for various purposes. Protocol fuzzers, web-app fuzzers, cloud fuzzers, etc. In this example, I’m using Radamsa. It’s a generic command line fuzzer that is simple to use yet it is fairly powerful. Not coincidentally, I also used it 10 years ago when writing my thesis.

Radamsa takes input either from stdin or from a file, and outputs fuzz either to stdout or to a file. This can then either be piped to the tested program, or the tested program can be instructed to open the file. Radamsa can also act as a TCP client or server, but I haven’t tried either of those so I can’t comment much on that. You can read more about Radamsa from it’s git repo.

The program is written in Owl Lisp, which gets translated into C, so the cross-compilation is quite straightforward once the Owl Lisp is set up. Because we don’t have to do any compilation time instrumentation for grey-box fuzzing guidance, the steps to build the fuzzer and the testable software are quite simple. The testable software in our case is going to be a kernel module. We still want to do some error instrumentation that will be covered in the next chapter, but since we’re fuzzing in kernel, it’s easier than one would guess (for once).

The Yocto recipe for building Radamsa can be found from meta-fuzzing repo I made to accompany this blog text.

Instrumentation

Breaking stuff with no consideration is rude. Breaking stuff and analyzing the results can be considered science. Therefore, to get something useful out of the fuzzing efforts we should figure out how to get as much information as possible from the system when it’s being bombarded. While black-box fuzzing doesn’t really need instrumentation, it makes fuzzing a lot more useful when we can detect more errors.

So, usually with all types of fuzzing some amount of compile-time instrumentation is used. This allows injecting extra code into the compiled binaries that may prove useful information when things start going wrong. A commonly used tool for this is AddressSanitizer (ASAN) and its fellow sanitizers. AddressSanitizer is a memory error detector that can detect things like use-after-frees, buffer overflows, and double-frees. As the nature of these bugs implies, it’s meant for C and C++ programs.

Of course, this comes with a price. On average, AddressSanitizer tends to slow down the programs 2x. Who would have guessed that injecting code into binaries has some side effects? For debugging purposes, this is still usually acceptable.

The best part of the AddressSanitizer is that it’s readily available in the Linux kernel! To enable KernelAddressSanitizer KASAN, all that needs to be done is to set two configuration flags:

CONFIG_KASAN=y
CONFIG_KASAN_GENERIC=y

You can read more about the different KASAN modes from the KASAN documentation, but in summary, generic is the heaviest, but also the most compatible mode. There are faster modes, but they may be architecture and compiler specific. After enabling these flags, we can detect memory errors not only in the kernel but also in the modules we are building for that kernel.

Linux also has undefined behaviour sanitizer (UBSAN), Kernel concurrency sanitizer (KCSAN), and Kernel memory leak detector (no fun acronym), but let’s leave them out for now. They can be enabled similarly by toggling configuration flags, so no special work is needed from the driver side.

Example Module

To have something to fuzz, I wrote a simple Linux kernel module (with help from ChatGPT). The module creates two sysfs files, one that takes input and one that gives output. Anything written to the first file can be read from the second file. This allows passing data from user space to kernel space, and is a suitable input surface for fuzzing. sysfs interface isn’t maybe the most interesting one, because there is some processing that happens before the input written by user ends up in the kernel module, but it’s a simple test for verifying that the set-up works.

The code for this module can be found in meta-fuzzing repo as well.

Putting It All Together

Rest of the stuff is quite simple. If you’re using Yocto, add the meta-fuzzing layer to your Yocto build, add the kernel configuration into your kernel config, and install Radamsa (and the test module) to the image. If you’re using something else, then you do the same things but with a different system. Then, run the image, log into it, and run the following:

echo test | radamsa

Most likely something other than test gets printed. If not, give it a few more tries. If the output doesn’t look like t ejSt after a few tries something may be wrong.

To fuzz the actual test kernel module, you can run the following:

modprobe sysfs_attribute_echo
while true
  do cat /sys/kernel/sysfs_attribute_echo/output | radamsa > /sys/kernel/sysfs_attribute_echo/input
done

This probes the module, and then in a neverending loop reads the output from the kernel module, fuzzes it and passes it back to the input file. As an example of the sample file-based fuzzing, check this out:

mkdir /tmp/samples
echo aaa > /tmp/samples/sample-1
echo bbb > /tmp/samples/sample-2
echo ccc > /tmp/samples/sample-3
while true
  do radamsa -n 1 /tmp/samples/* > /sys/kernel/sysfs_attribute_echo/input
done

We create three sample files, and fuzz randomly one of them. Radamsa can output the fuzzed data into a file, but we still use stdout to send it to the kernel module. The samples in this case are quite trivial, but with more interesting sample files it would be possible to generate quite exotic fuzzed data.

Does this find bugs from our module or kernel? No. Or at least it is highly unlikely. The kernel module itself is simple, and shouldn’t contain bugs (famous last words). Or, if there’s a bug, it’s either in the Linux kernel sysfs or kstrdup functions and those are already quite extensively tested (more famous last words). Unless there’s a regression of course.

However, this script demonstrates one admittedly simple approach of passing fuzzed data into the kernel space. The parsing of the data could be more exciting in a more complex module, which could in turn lead to actual bugs.

Closing Words

That’s all for this time. As shown here, the whole black-box fuzzing of the kernel can be straightforward. As mentioned about a dozen times in this text, the example was quite simple but demonstrates the point. The same ideas apply to more complex setups as well. The advantage of the black-box fuzzing is that it is easy to set up, so I recommend giving it a go and seeing what happens. Hopefully something exciting!