This is the promised follow-up to my earlier Syzkaller text. This time I’ll show how to make Syzkaller work on an actual hardware that is not emulated. At least in theory, for demonstration I’m still using an emulated target, but it could be easily replaced with real hardware. If you haven’t read the first part, I recommend doing it now, because I’ll recommend doing it a few more times later on, so it’s better to get it out of the way.
Requirements
The image requirements are mostly the same as for the emulated fuzzing: network communication, some debug flags in the kernel, etc. The biggest difference networking-wise is that the IP address isn’t assigned by QEMU. Therefore, we need to make sure that there is a network connection between the syz-manager fuzzer machine and the target hardware. We also need to know the IP address of the target hardware before launching Syzkaller. The exact details of this network management depend on your setup. In my example, I’m going to use a hardcoded IP address.
In addition to these, there is a requirement that the dmesg command needs to support the -w/--follow option. This means that the dmesg from Busybox does not work, but you should use the util-linux one instead. To install this into your Yocto image, you can use:
IMAGE_INSTALL:append = " util-linux-dmesg"The util-linux version should have a higher priority than Busybox and get automatically selected as the default dmesg command.
Syzkaller Configuration
I didn’t talk too much about Syzkaller configuration last time, but maybe I’ll mention a few more things about it. The configuration JSON consists of two things: Main configuration and target configuration, also called virtual machine configuration in the configuration file.
Main configuration primarily consists of setting different paths, like kernel source location, working directory location, and Syzkaller location. The configuration also contains the target architecture, the type of the target, syscalls to fuzz, and the address for the management HTTP server. There are few options that on a high-level define the target, but the target configuration happens mostly in the vm section of the configuration.
Counterintuitively the “virtual machine” target configuration is also for the non-virtual target configuration. Last time we configured a target of type qemu, this time we’re using isolated type. An isolated target is a type of target over which syz-manager has only limited control, and it is typically physically separate from the main fuzzing computer. In my demonstration I’m going to be using a QEMU machine as an isolated target, meaning that the isolated machine is a virtual machine, and is physically located in the same computer. However, it could just as well be elsewhere as long as it can be pinged.
For an isolated target, we need to define the IP address where the target can be found, and the working directory in the target machine. You may also want to control the reboot behaviour if the remote syz-executor hangs. For the qemu target, we configured the emulated machine properties like memory size and CPU count, but these are not required anymore since we cannot control those.
Strongly recommended for the isolated configuration is also a key-based SSH authentication. If you want to know how to add key-based authentication to your Yocto image, check out my earlier blog text.
We can use the example configuration from the isolated target documentation, and modify it a bit by adding the custom syscalls presented in the previous Syzkaller blog post:
{
"target": "linux/amd64",
"http": "<YOUR-IP-HERE>:56741",
"workdir": "./workdir",
"sshkey" : "/poky/auth-keys/root-ed25519-auth-key",
"kernel_obj": "/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.50+git/linux-qemux86_64-standard-build/",
"kernel_src": "/poky/build/tmp/work-shared/qemux86-64/kernel-source",
"enable_syscalls": ["openat$ioctl_string_parse", "ioctl$IOCTL_STRING_PARSE_CMD"],
"no_mutate_syscalls": ["openat$ioctl_string_parse"],
"syzkaller": ".",
"procs": 1,
"type": "isolated",
"vm": {
"targets" : [ "192.168.7.2" ],
"pstore": false,
"target_dir" : "/home/root/tmp/syzkaller",
"target_reboot" : false
}
}I also dropped the sandbox option that was in the example configuration, because we need the root permissions for writing to the example device. The isolated target configuration documentation contains plenty of other useful information as well, so I recommend reading it.
Fuzzing
To begin fuzzing, we first need to launch the target. If you have an actual HW target, you can press the power button and wait until the SSH server is started. In my case, I launch the QEMU machine using Yocto’s runqemu command:
runqemu nographic qemuparams="-m 2048"Note that I had to add some extra memory. By default, runqemu creates a VM with 256 megabytes of memory which is not enough to run the syz-executor. Without extra memory, the out-of-memory killer starts doing its work quite fast after starting the fuzzing. This memory requirement limits usage on the memory-constrained embedded targets, so I hope you have enough RAM. I guess the amount of required RAM depends from many things (configuration, architecture, kernel configuration, etc.), so you may want to try different values to see how low can you go.
Once the machine is ready and the SSH server is accepting connections, you can launch the syz-manager with the same command as before. I’m not going to go through again how to build the binaries, so if this line looks cryptic I again recommend checking out the earlier Syzkaller blog post:
./bin/syz-manager -config ./isolated.cfgAfter that, the fuzzer should work pretty much as usual. Which makes sense, all we’ve changed is some fields in the configuration file. If you leave the fuzzer running for a while, it should find the hidden crash from our demo driver again quite fast.
One thing I noticed is that the coverage information is not as accurate with the isolated target type as it was with the qemu type. Sometimes the crash is triggered before it is visible in the coverage, and sometimes the problematic line is marked covered but no crash occurs.
Tale of Woe: Trying Cross-Arch Fuzzing
In addition to this, I wanted to try cross-arch fuzzing with arm64 target because that is a quite common scenario, at least in the embedded world. The result was a mixed success. With a little modification in the kernel, I could get the fuzzer running for about 4 minutes. After that “something” happens, the Syzkaller gets a “no output received” error and starts reproducing it. Once the reproducer finishes, the Syzkaller manages to run four syscalls before it hangs again. From what I understood after three evenings spent with log files is that the syz-executor running on the target machine stops doing the one thing it should, executing.
In the logs it can be seen that syz-manager normally logs that it receives output, then suddenly there are a few minutes of just SSH keepalive messages, and then there’s the error that no output is received. The system is responsive the whole time, so it seems to be either a connection issue or then the syz-executor really hangs for some reason. Or maybe it’s syz-manager, I didn’t dig too deep into this, but if there’s interest I could take another look at it. I also wouldn’t be surprised if this works on someone else’s machine with a slightly different configuration. Anyway, here’s what I did to get the fuzzer running.
Hacking Kernel for arm64 Fuzzing
If you remember from the previous blog text, we need to enable a feature called Kernel Address Sanitizer (KASAN) for Syzkaller. For the arm64 KASAN can be configured in a few different modes. Usually, KASAN is used in generic mode, but for better performance, in arm64 it can also be configured in software tag-based or hardware tag-based modes. Understanding the exact operation of each mode is a bit above my pay grade (I get nothing for writing this), but basically software tag-based mode should have a low enough performance overhead to be used on real hardware, and hardware tag-based mode should be usable even in production, assuming the CPU supports it.
Sounds good, but in our situation this has one unfortunate side effect. For arm64 architecture KASAN_SHADOW_SCALE_SHIFT isn’t defined in kasan.h like it is done for the other architectures. Instead, it is defined in the architecture Makefile and is given as a command line define for the compiler. This kind of approach doesn’t work with Syzkaller, because when it works with a kernel source tree it doesn’t seem to be using the information from Makefile. Therefore we want to add the information from the Makefile to memory.h header for the arm64 that fails when compiling syscalls for Syzkaller. A patch like the below works:
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index fde4186cc387..8d05375d6b2b 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -70,6 +70,15 @@
* significantly, so double the (minimum) stack size when they are in use.
*/
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
+
+#ifndef KASAN_SHADOW_SCALE_SHIFT
+#if defined(CONFIG_KASAN_SW_TAGS)
+#define KASAN_SHADOW_SCALE_SHIFT 4
+#elif defined(CONFIG_KASAN_GENERIC)
+#define KASAN_SHADOW_SCALE_SHIFT 3
+#endif // CONFIG_KASAN_SW_TAGS
+#endif // KASAN_SHADOW_SCALE_SHIFT
+
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
#define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) \
+ KASAN_SHADOW_OFFSET)Basically, depending on the KASAN mode we want to have a different value for the KASAN_SHADOW_SCALE_SHIFT. I added a patch achieving the same effect to my meta-fuzzing layer. It’s a bit more complicated because it tries to define the KASAN variables in the same fashion as they are defined in arm architecture (and other architectures as well) in the kasan.h header.
Building, Configuring and Running Syzkaller for arm64
When building Syzkaller, this time we need to define the cross-compilation variables. The whole process of generating the custom syscall definitions and building the Syzkaller for the arm64 target looks like this:
# Compile syz-extract with proper target
./tools/syz-env make bin/syz-extract TARGETOS=linux TARGETVMARCH=arm64 TARGETARCH=arm64
# Extract the syscall definitions
./bin/syz-extract -os linux -arch arm64 -sourcedir /poky/build/tmp/work-shared/qemuarm64/kernel-source -builddir /poky/build/tmp/work/qemuarm64-poky-linux/linux-yocto/6.6.50+git/linux-qemuarm64-standard-build <SYSCALL_DEFINITION_FILE>.txt
# Generate the code
./tools/syz-env make generate TARGETOS=linux TARGETVMARCH=arm64 TARGETARCH=arm64
# Compile syzkaller
./tools/syz-env make TARGETOS=linux TARGETVMARCH=arm64 TARGETARCH=arm64The configuration file for the syz-manager is almost the same, we just need to change the first line target from "target": "linux/amd64" to "target": "linux/arm64". Also, you of course want to update the kernel_obj and kernel_src paths. After that is done, we can run the Syzkaller as usual:
./bin/syz-manager -config ./arm64-isolated.cfgThen it should work. And it does, for a while at least. If I launch this process maybe like 50 times I can get the hidden crash from the driver before the process hangs, so Syzkaller does indeed find bugs from different architectures as well.
The End?
That’s all I wanted to write about Syzkaller for now. This second part maybe proved that the fuzzing should be primarily done in a virtualized environment if possible. Then, the parts that cannot be fuzzed with the virtualized hardware should be done with the isolated targets. This sounds quite sensible, although sometimes setting up the virtualized environment can be a hassle. Fortunately, Yocto has quite a good support for QEMU. There are still some open questions with Syzkaller, like why the arm64 fuzzing didn’t quite work as expected, but all in all, it seems like quite a good tool for performing some robustness testing on your kernel and its modules.
