Hi,
We recently purchased 4x Milk-V Jupiter 2 (K3) boards to use in our continuous integration fleet for Zig. We’d previously been using the same number of Milk-V Jupiter (M1) boards.
Unfortunately, while doing build benchmarking in preparation for adding the K3 boards to the CI fleet, we ran into what looks to either be some kind of kernel bug or, more worryingly, a hardware bug.
In short, we sometimes see SIGSEGVs that make no sense, because the pc is pointing to e.g. an add or slli instruction, which obviously do not touch memory. When looking at the supposedly-problematic instruction both on disk, in GDB, and in dmesg output, everything agrees that it’s (e.g.) an add. But the dmesg output claims that it’s a store/AMO page fault (note the cause):
[166986.196678] zig1[229411]: unhandled signal 11 code 0x1 at 0x0000000000000009 in zig1[21aab6,109d000+44f000]
[166986.196692] CPU: 4 UID: 1004 PID: 229411 Comm: zig1 Not tainted 6.18.3-generic #1.0.2.4 PREEMPT(voluntary)
[166986.196698] Hardware name: SpacemiT K3 Pico ITX (DT)
[166986.196701] epc : 000000000121bab6 ra : 0000000003c10978 sp : 0000003fffaf31d0
[166986.196706] gp : 0000000001000800 tp : 00000000014ed6c8 t0 : 00000000014ed000
[166986.196709] t1 : 0000000003b3fe54 t2 : 0000000000023f95 s0 : 00000000000092d1
[166986.196712] s1 : 0000003f8fe7544c a0 : 0000000000000009 a1 : 0000000002df52b8
[166986.196716] a2 : 0000003f8fe7545c a3 : 0000003f8fe75440 a4 : 0000003f8d07e010
[166986.196719] a5 : 0000000003cd1c5c a6 : 000000000001825d a7 : 0000000002df40f0
[166986.196723] s2 : 000000000002031a s3 : 0000000002df40ec s4 : 0000000003950008
[166986.196726] s5 : 0000000003950004 s6 : 000000000394fffc s7 : 00000000000000a1
[166986.196730] s8 : 0000000002df7434 s9 : 0000000000000001 s10: 0000000001003b60
[166986.196733] s11: 0000000000000000 t3 : 0000000003c10974 t4 : 0000000003bb0000
[166986.196736] t5 : 0000000002df40e8 t6 : 0000000003950000
[166986.196739] status: 8000000200006020 badaddr: 0000000000000009 cause: 000000000000000f
[166986.196746] Code: eb63 68bb 058a 95ea 418c 8582 040e 9406 1402 9001 (943a) 400c
Below are the steps to reproduce. Please note: We have only been able to reproduce the problem while running these steps on an M.2 NVMe drive, not the built-in UFS. It also did not reproduce on the A100 cores, only on X100.
$ git clone https://codeberg.org/alexrp/zig.git --branch k3-bug-repro
$ wget https://ziglang.org/deps/zig+llvm+lld+clang-riscv64-linux-musl-0.17.0-dev.203+073889523.tar.xz
$ tar xvf zig+llvm+lld+clang-riscv64-linux-musl-0.17.0-dev.203+073889523.tar.xz
$ cd zig
$ git clean -xdff && ./repro.sh
It may take a few repeats of that last command to trigger the bug. (You can speed up the process by making multiple copies of the zig directory to run repro.sh in parallel.) When it happens, the failure will look like this:
+ ninja zig2
[13/17] Running zig1.wasm to produce /home/ci/zig1/build-release/compiler_rt.c
FAILED: [code=139] compiler_rt.c /home/ci/zig1/build-release/compiler_rt.c
cd /home/ci/zig1 && /home/ci/zig1/build-release/zig1 /home/ci/zig1/lib build-obj -ofmt=c -OReleaseSmall --name compiler_rt -femit-bin="/home/ci/zig1/build-release/compiler_rt.c" -target riscv64-linux -Mroot=lib/compiler_rt.zig
Segmentation fault
[14/17] Running zig1.wasm to produce /home/ci/zig1/build-release/zig2.c
FAILED: [code=139] zig2.c /home/ci/zig1/build-release/zig2.c
cd /home/ci/zig1 && /home/ci/zig1/build-release/zig1 /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
Segmentation fault
(Note that it can happen when generating either compiler_rt.c or zig2.c, or both.)
From here, you can now consistently trigger the SIGSEGV multiple times:
$ /home/ci/zig1/build-release/zig1 /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
Segmentation fault
$ /home/ci/zig1/build-release/zig1 /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
Segmentation fault
Running it in GDB, we see:
Program received signal SIGSEGV, Segmentation fault.
0x000000000121bab6 in f2688 ()
(gdb) disas $pc-20,+40
Dump of assembler code from 0x121baa2 to 0x121baca:
0x000000000121baa2 <f2688+250>: bltu s7,a1,0x121c138 <f2688+1936>
0x000000000121baa6 <f2688+254>: slli a1,a1,0x2
0x000000000121baa8 <f2688+256>: add a1,a1,s10
0x000000000121baaa <f2688+258>: lw a1,0(a1)
0x000000000121baac <f2688+260>: jr a1
0x000000000121baae <f2688+262>: slli s0,s0,0x3
0x000000000121bab0 <f2688+264>: add s0,s0,ra
0x000000000121bab2 <f2688+266>: slli s0,s0,0x20
0x000000000121bab4 <f2688+268>: srli s0,s0,0x20
=> 0x000000000121bab6 <f2688+270>: add s0,s0,a4
0x000000000121bab8 <f2688+272>: lw a1,0(s0)
0x000000000121baba <f2688+274>: j 0x121ba8e <f2688+230>
0x000000000121babc <f2688+276>: slli s0,s0,0x3
0x000000000121babe <f2688+278>: add s0,s0,t3
0x000000000121bac0 <f2688+280>: j 0x121bab2 <f2688+266>
0x000000000121bac2 <f2688+282>: slli s0,s0,0x3
0x000000000121bac4 <f2688+284>: add s0,s0,ra
0x000000000121bac6 <f2688+286>: slli s0,s0,0x20
0x000000000121bac8 <f2688+288>: srli s0,s0,0x20
So that clearly indicates that the CPU is running different code than is on disk or even in memory.
Now for the fun part:
$ cp build-release/zig1 build-release/zig1.cp
$ /home/ci/zig1/build-release/zig1.cp /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
$ echo $?
0
$ /home/ci/zig1/build-release/zig1 /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
Segmentation fault
$ sha256sum build-release/{zig1,zig1.cp}
5bf0342502b90f61510239534eaa13bf08d6261394ce17db8a404c5c9c82472c build-release/zig1
5bf0342502b90f61510239534eaa13bf08d6261394ce17db8a404c5c9c82472c build-release/zig1.cp
$ mv build-release/zig1 build-release/zig1.mv
$ /home/ci/zig1/build-release/zig1.mv /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
Segmentation fault
$ sha256sum build-release/zig1.mv
5bf0342502b90f61510239534eaa13bf08d6261394ce17db8a404c5c9c82472c build-release/zig1.mv
It turns out that if we force the kernel to flush its page cache, the bug vanishes:
$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
$ /home/ci/zig1/build-release/zig1.mv /home/ci/zig1/lib build-exe -ofmt=c -lc -OReleaseSmall --name zig2 -femit-bin="/home/ci/zig1/build-release/zig2.c" -target riscv64-linux --dep build_options -Mroot=src/main.zig -Mbuild_options=/home/ci/zig1/build-release/config.zig
$ echo $?
0
So that’s where we’re at; at this point, it must either be a kernel bug or a hardware bug.
For the record, we already tried writing a small program that mmaps the program headers of zig1 at the exact same addresses the kernel does when loading it, and then doing a global SYS_riscv_flush_icache on those ranges. This did not make any difference.
Let me know if we can provide any more information to assist in debugging this. This is unfortunately completely blocking us from using our 4 boards as it reproduces on all of them.