This is the promised follow-up to my earlier Syzkaller text. This time I’ll show how to make Syzkaller work on an actual hardware that is not emulated. At least in theory, for demonstration I’m still using an emulated target, but it could be easily replaced with real hardware. If you haven’t read the first part, I recommend doing it now, because I’ll recommend doing it a few more times later on, so it’s better to get it out of the way.

Requirements

The image requirements are mostly the same as for the emulated fuzzing: network communication, some debug flags in the kernel, etc. The biggest difference networking-wise is that the IP address isn’t assigned by QEMU. Therefore, we need to make sure that there is a network connection between the syz-manager fuzzer machine and the target hardware. We also need to know the IP address of the target hardware before launching Syzkaller. The exact details of this network management depend on your setup. In my example, I’m going to use a hardcoded IP address.

In addition to these, there is a requirement that the dmesg command needs to support the -w/--follow option. This means that the dmesg from Busybox does not work, but you should use the util-linux one instead. To install this into your Yocto image, you can use:

IMAGE_INSTALL:append = " util-linux-dmesg"

The util-linux version should have a higher priority than Busybox and get automatically selected as the default dmesg command.

Syzkaller Configuration

I didn’t talk too much about Syzkaller configuration last time, but maybe I’ll mention a few more things about it. The configuration JSON consists of two things: Main configuration and target configuration, also called virtual machine configuration in the configuration file.

Main configuration primarily consists of setting different paths, like kernel source location, working directory location, and Syzkaller location. The configuration also contains the target architecture, the type of the target, syscalls to fuzz, and the address for the management HTTP server. There are few options that on a high-level define the target, but the target configuration happens mostly in the vm section of the configuration.

Counterintuitively the “virtual machine” target configuration is also for the non-virtual target configuration. Last time we configured a target of type qemu, this time we’re using isolated type. An isolated target is a type of target over which syz-manager has only limited control, and it is typically physically separate from the main fuzzing computer. In my demonstration I’m going to be using a QEMU machine as an isolated target, meaning that the isolated machine is a virtual machine, and is physically located in the same computer. However, it could just as well be elsewhere as long as it can be pinged.

For an isolated target, we need to define the IP address where the target can be found, and the working directory in the target machine. You may also want to control the reboot behaviour if the remote syz-executor hangs. For the qemu target, we configured the emulated machine properties like memory size and CPU count, but these are not required anymore since we cannot control those.

Strongly recommended for the isolated configuration is also a key-based SSH authentication. If you want to know how to add key-based authentication to your Yocto image, check out my earlier blog text.

We can use the example configuration from the isolated target documentation, and modify it a bit by adding the custom syscalls presented in the previous Syzkaller blog post:

{
        "target": "linux/amd64",
        "http": "<YOUR-IP-HERE>:56741",
        "workdir": "./workdir",
        "sshkey" : "/poky/auth-keys/root-ed25519-auth-key",
        "kernel_obj": "/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build/",
        "kernel_src": "/poky/build/tmp/work-shared/qemux86-64/kernel-source",
        "enable_syscalls": ["openat$ioctl_string_parse", "ioctl$IOCTL_STRING_PARSE_CMD"],
        "no_mutate_syscalls": ["openat$ioctl_string_parse"],
        "syzkaller": ".",
        "procs": 1,
        "type": "isolated",
        "vm": {
                "targets" : [ "192.168.7.2" ],
                "pstore": false,
                "target_dir" : "/home/root/tmp/syzkaller",
                "target_reboot" : false
        }
}

I also dropped the sandbox option that was in the example configuration, because we need the root permissions for writing to the example device. The isolated target configuration documentation contains plenty of other useful information as well, so I recommend reading it.

Fuzzing

To begin fuzzing, we first need to launch the target. If you have an actual HW target, you can press the power button and wait until the SSH server is started. In my case, I launch the QEMU machine using Yocto’s runqemu command:

runqemu nographic qemuparams="-m 2048"

Note that I had to add some extra memory. By default, runqemu creates a VM with 256 megabytes of memory which is not enough to run the syz-executor. Without extra memory, the out-of-memory killer starts doing its work quite fast after starting the fuzzing. This memory requirement limits usage on the memory-constrained embedded targets, so I hope you have enough RAM. I guess the amount of required RAM depends from many things (configuration, architecture, kernel configuration, etc.), so you may want to try different values to see how low can you go.

Once the machine is ready and the SSH server is accepting connections, you can launch the syz-manager with the same command as before. I’m not going to go through again how to build the binaries, so if this line looks cryptic I again recommend checking out the earlier Syzkaller blog post:

./bin/syz-manager -config ./isolated.cfg

After that, the fuzzer should work pretty much as usual. Which makes sense, all we’ve changed is some fields in the configuration file. If you leave the fuzzer running for a while, it should find the hidden crash from our demo driver again quite fast.

One thing I noticed is that the coverage information is not as accurate with the isolated target type as it was with the qemu type. Sometimes the crash is triggered before it is visible in the coverage, and sometimes the problematic line is marked covered but no crash occurs.

Tale of Woe: Trying Cross-Arch Fuzzing

In addition to this, I wanted to try cross-arch fuzzing with arm64 target because that is a quite common scenario, at least in the embedded world. The result was a mixed success. With a little modification in the kernel, I could get the fuzzer running for about 4 minutes. After that “something” happens, the Syzkaller gets a “no output received” error and starts reproducing it. Once the reproducer finishes, the Syzkaller manages to run four syscalls before it hangs again. From what I understood after three evenings spent with log files is that the syz-executor running on the target machine stops doing the one thing it should, executing.

In the logs it can be seen that syz-manager normally logs that it receives output, then suddenly there are a few minutes of just SSH keepalive messages, and then there’s the error that no output is received. The system is responsive the whole time, so it seems to be either a connection issue or then the syz-executor really hangs for some reason. Or maybe it’s syz-manager, I didn’t dig too deep into this, but if there’s interest I could take another look at it. I also wouldn’t be surprised if this works on someone else’s machine with a slightly different configuration. Anyway, here’s what I did to get the fuzzer running.

Hacking Kernel for arm64 Fuzzing

If you remember from the previous blog text, we need to enable a feature called Kernel Address Sanitizer (KASAN) for Syzkaller. For the arm64 KASAN can be configured in a few different modes. Usually, KASAN is used in generic mode, but for better performance, in arm64 it can also be configured in software tag-based or hardware tag-based modes. Understanding the exact operation of each mode is a bit above my pay grade (I get nothing for writing this), but basically software tag-based mode should have a low enough performance overhead to be used on real hardware, and hardware tag-based mode should be usable even in production, assuming the CPU supports it.

Sounds good, but in our situation this has one unfortunate side effect. For arm64 architecture KASAN_SHADOW_SCALE_SHIFT isn’t defined in kasan.h like it is done for the other architectures. Instead, it is defined in the architecture Makefile and is given as a command line define for the compiler. This kind of approach doesn’t work with Syzkaller, because when it works with a kernel source tree it doesn’t seem to be using the information from Makefile. Therefore we want to add the information from the Makefile to memory.h header for the arm64 that fails when compiling syscalls for Syzkaller. A patch like the below works:

diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index fde4186cc387..8d05375d6b2b 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -70,6 +70,15 @@
  * significantly, so double the (minimum) stack size when they are in use.
  */
 #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
+#ifndef KASAN_SHADOW_SCALE_SHIFT
+#if defined(CONFIG_KASAN_SW_TAGS)
+#define KASAN_SHADOW_SCALE_SHIFT       4
+#elif defined(CONFIG_KASAN_GENERIC)
+#define KASAN_SHADOW_SCALE_SHIFT       3
+#endif // CONFIG_KASAN_SW_TAGS
+#endif // KASAN_SHADOW_SCALE_SHIFT
+
 #define KASAN_SHADOW_OFFSET    _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
 #define KASAN_SHADOW_END       ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) \
                                        + KASAN_SHADOW_OFFSET)

Basically, depending on the KASAN mode we want to have a different value for the KASAN_SHADOW_SCALE_SHIFT. I added a patch achieving the same effect to my meta-fuzzing layer. It’s a bit more complicated because it tries to define the KASAN variables in the same fashion as they are defined in arm architecture (and other architectures as well) in the kasan.h header.

Building, Configuring and Running Syzkaller for arm64

When building Syzkaller, this time we need to define the cross-compilation variables. The whole process of generating the custom syscall definitions and building the Syzkaller for the arm64 target looks like this:

# Compile syz-extract with proper target
./tools/syz-env make bin/syz-extract TARGETOS=linux TARGETVMARCH=arm64 TARGETARCH=arm64
# Extract the syscall definitions
./bin/syz-extract -os linux -arch arm64 -sourcedir /poky/build/tmp/work-shared/qemuarm64/kernel-source -builddir /poky/build/tmp/work/qemuarm64-poky-linux/linux-yocto/6.6.50+git/linux-qemuarm64-standard-build <SYSCALL_DEFINITION_FILE>.txt
# Generate the code
./tools/syz-env make generate TARGETOS=linux TARGETVMARCH=arm64 TARGETARCH=arm64
# Compile syzkaller
./tools/syz-env make TARGETOS=linux TARGETVMARCH=arm64 TARGETARCH=arm64

The configuration file for the syz-manager is almost the same, we just need to change the first line target from "target": "linux/amd64" to "target": "linux/arm64". Also, you of course want to update the kernel_obj and kernel_src paths. After that is done, we can run the Syzkaller as usual:

./bin/syz-manager -config ./arm64-isolated.cfg

Then it should work. And it does, for a while at least. If I launch this process maybe like 50 times I can get the hidden crash from the driver before the process hangs, so Syzkaller does indeed find bugs from different architectures as well.

The End?

That’s all I wanted to write about Syzkaller for now. This second part maybe proved that the fuzzing should be primarily done in a virtualized environment if possible. Then, the parts that cannot be fuzzed with the virtualized hardware should be done with the isolated targets. This sounds quite sensible, although sometimes setting up the virtualized environment can be a hassle. Fortunately, Yocto has quite a good support for QEMU. There are still some open questions with Syzkaller, like why the arm64 fuzzing didn’t quite work as expected, but all in all, it seems like quite a good tool for performing some robustness testing on your kernel and its modules.

This is a sequel to a similarly named blog post Black-Box Fuzzing Kernel Modules in Yocto. In that blog post, I briefly went through what fuzzing is and presented an easy but fairly naive approach to fuzzing. This time I will present a more refined approach to fuzzing using Syzkaller.

Syzkaller

Syzkaller is an unsupervised coverage-guided kernel fuzzer from Google. Simply put, it creates programs that perform different kernel syscalls in varying order with evolving parameters and then analyzes the coverage to see what test programs reach new code coverage. These programs are added to the corpus, which can be used to create even more programs with even larger coverage. Sounds simple? Well, the architecture looks like this:

Could be worse. There are two core components in the system, syz-manager and syz-executor. syz-manager runs on the “main fuzzing server”, and syz-executor runs on the fuzzing target. This can either be an actual hardware device or a QEMU emulator system that syz-manager controls. syz-manager is responsible for the fuzzing work, generating the test programs, and storing the corpus & crashes. The executor receives the test programs, runs them and reports back the results. This communication happens over a network interface.

Smoke Test

A good way to smoke test the setup is to run Syzkaller with the default built-in syscall definitions. This ensures that your image is suitable for fuzzing without having to worry that your additions are breaking things. I’m using Yocto to build an image for x86_64 QEMU. You can use pretty much anything to build the image as long as you have the kernel source and object files, and a disk image for rootfs and of course a kernel. In addition to x86_64 there are plenty of other supported architectures. The setup guide lists arm, arm64, and riscv64 for example.

Building the Image

First, the disk image requirements. The image should have an SSH root login that either has key authentication or a passwordless login. This naturally means that there needs to be networking support. Optional, but strongly recommended, is a DHCP client for getting an IP address. This is required for QEMU port forwarding that syz-manager utilizes. Yocto’s default QEMU core-image-base image does not require any special changes to the disk image as it has network capabilities, passwordless root, and a DHCP client.

The kernel should be built with instrumentation and a few other debugging options enabled. The exact options depend on the target architecture, the x86_64 setup guide lists “at the very least” the following as a requirement:

CONFIG_KCOV=y
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y

The full documentation of the kernel configuration can be found here, and it lists plenty of additional configuration flags. I added both the minimum and the maximum configuration to my meta-fuzzing layer. I’ll be using the minimum configuration.

Writing Syzkaller configuration

Next, we need to write the configuration for Syzkaller. Let’s pick the example QEMU-configuration, and modify it a bit. This configuration defines the web interface address, location of required files, the syscalls to call, and the fuzzing target. We are going to use emulated QEMU targets in this test:

{
	"target": "linux/amd64",
	"http": "<YOUR-IP-HERE>:56741",
	"workdir": "./workdir",
	"kernel_obj": "/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build/",
	"kernel_src": "/poky/build/tmp/work-shared/qemux86-64/kernel-source",
	"image": "/poky/build/tmp/deploy/images/qemux86-64/core-image-base-qemux86-64.rootfs.ext4",
	"syzkaller": ".",
	"disable_syscalls": ["keyctl", "add_key", "request_key"],
	"procs": 4,
	"type": "qemu",
	"vm": {
		"count": 2,
		"cpu": 2,
		"mem": 2048,
		"kernel": "/poky/build/tmp/deploy/images/qemux86-64/bzImage",
		"cmdline": "ip=dhcp"
	}
}

Different configuration items are quite well documented in the manager configuration and QEMU VM configuration files. The amount of virtual machines and processes here is quite low because I have a poor PC for running these tests, so you may want to increase them.

This configuration assumes you have the Poky folder directly under the root folder, adjust this if necessary. Also, the configuration assumes that the process is launched in the root of the Syzkaller repo. The ip=dhcp cmdline option is also worth noting. This will be passed to the kernel, and it should ensure that the Ethernet interface uses DHCP to get IP address from QEMU. I think you can also hardcode the IP address if you know what it should be. tcpdump can be used to check the incoming ARP requests to see what the IP address is expected to be. There may be an easier way, but that’s what I did when poking around.

There’s one weird hack I had to do though. In my experience (and I may be wrong here), it seems that Syzkaller expects a certain format from the kernel source tree, and that expected format is not the actual structure of the kernel source. There seems to be an expectation that under the path defined in kernel_src there is a usr/src/kernel folder that points to the source, otherwise the coverage information generation will fail. However, if I move the kernel source to a usr/src/kernel folder, the report generation will fail because the scripts folder cannot be found anymore. To create a suitable directory structure, I used the following script:

cd <KERNEL_SOURCE>
mkdir -p usr/src/
cd usr/src/
ln -s ../../. kernel

Again, I’m not sure if this is necessary, it may be that I misconfigured something, but I had to do this to get the syz-manager running, reporting coverage, and formatting error reports.

Running Syzkaller

Once the configuration is ready, we can start to wonder what to do with it. The first thing is fetching the Syzkaller source from GitHub. Then we can compile Syzkaller using syz-env that is a Docker script that can be used to ensure that the build environment is the expected one:

./tools/syz-env make

This of course requires Docker. You can also use make directly, but then you need to take care of the build dependencies yourself. If you’re doing cross-arch testing, you need to define the target variables, so check out the setup guide for those. Once the build completes, copy the configuration created in the previous chapter to the root of the Syzkaller repo, and run:

./bin/syz-manager -config <CONFIG_FILE>

syz-manager should start up, and after a moment the QEMU machines should boot up as well. If you navigate your web browser to http://<YOUR-IP-HERE>:56741, you should see the web interface that shows the status of fuzzing, collected coverage and crashes:

If not, you can try adding -debug option to the syz-manager command to see what’s going wrong. Also, for some reason my web interface doesn’t load if there are no crashes, there’s just an error message about missing crashes folder. So, I had to create manually ./workdir/crashes folder, and then it seemed to work just fine.

Once you get to the web interface, you can leave the fuzzer running for a while to see how it works and explores new coverage paths while increasing corpus. It’s quite unlikely it’ll find a crash, as it’s fuzzing against a mainline kernel, but if it does, you have potentially found a kernel bug!

Adding Custom Syscalls

However, it’s not very interesting to fuzz the mainline kernel. It’s been so done (and is constantly being done). What’s more interesting is fuzzing our custom kernel module, and seeing if the Syzkaller can find a poorly hidden error from it.

I asked ChatGPT to write a small driver that has an IOCTL interface, and then I added a bug to it. The single command in the interface takes a string as an input. It tokenizes the string, and if the string contains five commas, an invalid free will be performed. There are some extra checks to guide the fuzzer towards the crash.

In-tree vs. Out-of-tree Module Build

While writing this text, I had to consider the in-tree vs. out-of-tree kernel module building. The difference is that an in-tree module is built as a part of the kernel build, and an out-of-tree module is built against the kernel headers after the kernel has been built. The out-of-tree method allows compiling modules for pre-built kernels, assuming the headers are available. For example, if you’ve ever built a Hello World -module for Ubuntu, you’ve most likely run apt-get install linux-headers-`uname -r`. This means you’ve pulled the development headers for building the out-of-tree module, and you’re not compiling an entire kernel.

Since Yocto builds the kernel it is quite easy to build the modules in-tree. This has a few advantages. For example, the module can be easily built as a built-in feature. From the fuzzing point of view, obtaining the coverage for the in-tree modules is a lot easier. Also getting the correct line numbers for crash reports is a lot simpler because there’s no need to decrypt offsets with objdump. So I’d recommend building the modules in-tree for fuzzing.

The example driver has both a module recipe (because that’s what I tried first) for out-of-tree build and linux-yocto append for in-tree build. It’s a bit silly way of supporting both methods, but at least it works (until it doesn’t). To add the IOCTL example module to the in-tree kernel build, add this line somewhere in your configuration:

IOCTL_STRING_PARSE_INTREE = "1"

Describing New Syscalls in Syzkaller

This is where the magic happens. Defining our own syscalls to Syzkaller so that it knows about the non-default syscalls and can create fuzzing sequences utilizing them. To achieve that, we need to write the syscall definitions in Syzkaller’s syntax, which is kind of simple, but still a bit frustrating to get right. Fortunately, there are plenty of examples. After staring at those, and reading this blog post, here’s what I came up with:

include <linux/fcntl.h>
include <linux/ioctl_string_parse.h>

resource fd_vuln_ioctl[fd]
openat$ioctl_string_parse(fd const[AT_FDCWD], file ptr[in, string["/dev/ioctl_example"]], flags const[0x2], mode const[0x0]) fd_vuln_ioctl
ioctl$IOCTL_STRING_PARSE_CMD(fd fd_vuln_ioctl, cmd const[IOCTL_CMD_PARSE_STRING], arg ptr[in, string])

A file with this content should be added to <PATH_TO_SYZKALLER>/sys/linux. The file name can be arbitrarily chosen, but it should have the .txt suffix.

The includes here are from the kernel source tree. fcntl.h is for the AT_FDCWD macro, and our own ioctl_string_parse.h is for the IOCTL_CMD_PARSE_STRING that contains the IOCTL command to use when making IOCTL calls to our driver. resource defines the file descriptor resource that is shared between the two other calls. openat opens the ioctl_example device in read/write mode with no special mode flags. This call should be quite static as the goal isn’t to fuzz the openat command, so most values are constants.

The final definition is the IOCTL command on the opened device, using the IOCTL_CMD_PARSE_STRING defined in the header as the command, and a random string as an argument for the IOCTL call.

Once the definitions are done, it’s time to compile them into Syzkaller. We first need to compile a tool called syz-extract, then extract syscalls from the our .txt file to a .const file, update the generated code, and re-compile the binary. The four commands below do exactly that (for 64-bit Linux systems)

./tools/syz-env make bin/syz-extract
./bin/syz-extract -os linux -arch amd64 -sourcedir /poky/linked-linux-src/usr/src/kernel -builddir /poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build <CUSTOM_DEFINITIONS>.txt
./tools/syz-env make generate
./tools/syz-env make

Note how you need to pass the build and source directories for extracting syscalls into .const file that will be used in the code generation. Again, update paths and file names if necessary

After that, we should be almost good to go. The configuration file still needs some tweaking. We do want to focus only on our custom syscalls, and we do not want to mutate the openat command. The following configuration should work:

{
	"target": "linux/amd64",
	"http": "<YOUR-IP-HERE>:56741",
	"workdir": "./workdir",
	"kernel_obj": "/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build/",
	"kernel_src": "/poky/build/tmp/work-shared/qemux86-64/kernel-source",
	"image": "/poky/build/tmp/deploy/images/qemux86-64/core-image-base-qemux86-64.rootfs.ext4",
	"enable_syscalls": ["openat$ioctl_string_parse", "ioctl$IOCTL_STRING_PARSE_CMD"],
	"no_mutate_syscalls": ["openat$ioctl_string_parse"],
	"syzkaller": ".",
	"procs": 8,
	"type": "qemu",
	"vm": {
		"count": 2,
		"cpu": 2,
		"mem": 2048,
		"kernel": "/poky/build/tmp/deploy/images/qemux86-64/bzImage",
		"cmdline": "ip=dhcp"
	}
}

After that, the fuzzer can be started with the same command as before:

./bin/syz-manager -config <CONFIG_FILE>

Now, after waiting a few minutes, there should be some crashes visible:

Interestingly enough, if we are using the maximum debug kernel configuration these get reported as “potential deadlocks”. I guess enabling 40+ debug flags has some side effects. (As a side note, after enabling all the configuration items the basic runqemu machine boot time slows down from 10 seconds to 5 minutes). Regardless of which configuration we’re using, if we check out the report we can see the expected root cause for the crash:

It’s quite fascinating to watch the coverage information and see how the fuzzer approaches the problematic line when it attempts to increase the coverage. Fascinating in the same sense it’s exciting to watch paint dry:

Extra Bonus Issue

Can you see what’s the issue with this code:

input_buffer = kmalloc(input_len, GFP_KERNEL);
switch (cmd) {
    case IOCTL_CMD_PARSE_STRING:
        ret = copy_from_user(input_buffer, (char *)arg, input_len);
        if (ret != 0) {
            printk(KERN_ALERT "Failed to copy string from user space\n");
            kfree(input_buffer);
            return -EFAULT;
        }

        // Some processing happens here

        kfree(input_buffer);
        break;

    default:
        printk(KERN_ALERT "Invalid IOCTL command\n");
        kfree(input_buffer);
        return -EINVAL;
}

I didn’t, and neither did ChatGPT when it suggested this for the first version of the driver. However, when Syzkaller calls this kind of IOCTL function multiple times in a rapid fashion it results in some unexpected invalid-frees. I guess constantly performing allocations and frees in an IOCTL command isn’t the best idea. The second implementation of the driver uses a memory pool to avoid having to allocate memory after initialization, and that seems to work. But yeah, another point for fuzzing for finding a bug from a seemingly functional code.

But Wait, There’s More!

In addition to this, I also created a test program with ChatGPT for debugging purposes. Later I realized that this could also be used for black-box fuzzing the example kernel module. So, if you want to, you can run the following script to fuzz the module with Radamsa (assuming you have installed Radamsa, check the black-box fuzzing blog text for more info):

while true; do 
    test-program-ioctl "$(echo ,,, | radamsa)";
done

Mandatory Final Chapter

This should cover all for now. With these instructions, you can hopefully fuzz your kernel module with Syzkaller. However, this only performs fuzzing on virtual QEMU targets. Sometimes it’d be better to fuzz on the actual hardware, especially if using specialized hardware. I’ll cover that in a follow-up text. If you want to get notified when that text goes out, consider joining my mailing list. Thanks for reading, and happy bug-hunting.