Re: Problem testing with S390x under QEMU on x86_64

[Ilya Leoshkevich <iii@linux.ibm.com>]

On Sat, 2024-08-17 at 14:57 -0700, Tony Ambardar wrote:
> Hello all,
> 
> I'd appreciate some help from the BPF and s390x communities...
> 
> Some background: I'm finalizing a patch series enabling full cross-
> endian
> support for libbpf and selftests/bpf, and testing with mips64 so far.
> This
> arch makes switching the build byte-order trivial and has been very
> handy
> for A/B testing, but lacks some basic support (bpf2bpf call, kfuncs,
> etc.)
> making for poor selftests coverage.
> 
> To finish testing I thought (optimistically?) to cross-build kernel
> and
> bpf selftests from x86_64 to s390x. That configuration might also be
> used
> later on bpf-ci for regression testing endian support and sharing the
> load
> of s390x builds.
> 
> Locally, the build succeeds but when running it under QEMU I see
> kernel
> crashes trying to load any modules (e.g. prng or bpf_testmod).
> 
> 
> The build/test setup uses Ubuntu and gcc: 
> 
> kodidev:~/linux$ lsb_release -d
> Description:    Ubuntu 22.04.4 LTS
> 
> kodidev:~/linux$ s390x-linux-gnu-gcc --version
> s390x-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
> Copyright (C) 2021 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There
> is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> PURPOSE.
> 
> 
> The code targets bpf/master, and I've researched QEMU/s390x usage
> details
> online but mainly follow https://docs.kernel.org/bpf/s390.html. For
> rootfs
> I take the same s390x image used by the kernel-patches bpf-ci.
> 
> 
> The kernel .config used is attached, and the QEMU command-line is
> below:
> 
> qemu-system-s390x -cpu max,zpci=on -smp 2 -nographic -m 1G \
> -object rng-random,filename=/dev/urandom,id=rng0 \
> -device virtio-rng-ccw,rng=rng0 \
> -device virtio-net-ccw,netdev=eth0 \
> -netdev user,id=eth0,hostfwd=tcp::2224-:22 \
> -serial mon:stdio \
> -nodefaults \
> -kernel bzImage-s390x \
> -drive file=root-s390x-glibc.qcow2,if=virtio,format=qcow2   \
> -append "rootwait root=/dev/vda rw net.ifnames=0 biosdevname=0
> console=ttyS1"
> 
> 
> After successfully booting, the crashes are easily reproduced:
> 
> root@(none):/# uname -a
> Linux (none) 6.10.0-12706-g853081e84612-dirty #111 SMP Sat Aug 17
> 00:49:23
> PDT 2024 s390x GNU/Linux
> 
> # modprobe prng
> Unable to handle kernel pointer dereference in virtual kernel address
> space
> Failing address: 000003fee0011000 TEID: 000003fee0011803
> Fault in home space mode while using kernel ASCE.
> AS:0000000001dac007 R3:0000000000000024
> Oops: 003b ilc:1 [#1] SMP
> Modules linked in: prng(+)
> CPU: 1 UID: 0 PID: 81 Comm: modprobe Not tainted 6.10.0-12691-
> g52e8c345c9f0 #106
> Hardware name: QEMU 3906 QEMU (KVM/Linux)
> Krnl PSW : 0704d00180000000 000003fee0011ea0 (0x3fee0011ea0)
>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0
> EA:3
> Krnl GPRS: 000002f2820db180 000003ffe1ca49a8 0000000000000000
> 000003ff6000f498
>            0000000000000000 0000000000000a22 0000000000000000
> 0000000000000000
>            000003ffe18111e0 000002f2806a48e8 000003ff6000f498
> 0000000000000000
>            000003ff8eaacfa8 000002aa0bbafa00 000003ff6000f4be
> 0000037fe0c33b58
> Krnl Code: Bad PSW.
> Call Trace:
>  [<000003fee0011ea0>] 0x3fee0011ea0
>  [<000003ffe0000b24>] do_one_initcall+0x64/0x258
>  [<000003ffe01566e8>] do_init_module+0x78/0x258
>  [<000003ffe0158160>] init_module_from_file+0x88/0xa8
>  [<000003ffe01582e8>] idempotent_init_module+0x168/0x230
>  [<000003ffe0158430>] __s390x_sys_finit_module+0x80/0xb8
>  [<000003ffe0b6f58a>] __do_syscall+0x232/0x2b0
>  [<000003ffe0b81b90>] system_call+0x70/0x98
> INFO: lockdep is turned off.
> Last Breaking-Event-Address:
>  [<000003ff6000f4b8>]
> cpu_feature_match_S390_CPU_FEATURE_MSA_init+0x20/0xb68 [prng]
> Kernel panic - not syncing: Fatal exception: panic_on_oops
> 
> 
> and also:
> 
> # ./test_progs -a xdpwall
> bpf_testmod: loading out-of-tree module taints kernel.
> Unable to handle kernel pointer dereference in virtual kernel address
> space
> Failing address: 000003fee0293000 TEID: 000003fee0293803
> Fault in home space mode while using kernel ASCE.
> AS:0000000001dac007 R3:0000000000000024
> Oops: 003b ilc:1 [#1] SMP
> Modules linked in: bpf_testmod(O+)
> CPU: 2 UID: 0 PID: 91 Comm: test_progs Tainted: G           O      
> 6.10.0-12691-g52e8c345c9f0 #106
> Tainted: [O]=OOT_MODULE
> Hardware name: QEMU 3906 QEMU (KVM/Linux)
> Krnl PSW : 0704d00180000000 000003fee0293998 (0x3fee0293998)
>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0
> EA:3
> Krnl GPRS: 000002f20374e760 0001677600016805 0000000000000000
> 000003ff6000e288
>            0000000000000000 000000000000007a 0000000000000000
> 0000000000000000
>            000003ffe18111e0 000002f204c6d2e8 000003ff6000ab90
> 0000000000000000
>            000003ffb73acfa8 0000000000000000 000003ff6000abce
> 0000037fe0803b50
> Krnl Code: Bad PSW.
> Call Trace:
>  [<000003fee0293998>] 0x3fee0293998
>  [<000003ffe0000b24>] do_one_initcall+0x64/0x258
>  [<000003ffe01566e8>] do_init_module+0x78/0x258
>  [<000003ffe0158160>] init_module_from_file+0x88/0xa8
>  [<000003ffe01582e8>] idempotent_init_module+0x168/0x230
>  [<000003ffe0158430>] __s390x_sys_finit_module+0x80/0xb8
>  [<000003ffe0b6f58a>] __do_syscall+0x232/0x2b0
>  [<000003ffe0b81b90>] system_call+0x70/0x98
> INFO: lockdep is turned off.
> Last Breaking-Event-Address:
>  [<000003ff6000abc8>] bpf_testmod_init+0x38/0x160 [bpf_testmod]
> Kernel panic - not syncing: Fatal exception: panic_on_oops
> 
> 
> Has anyone encountered this before, or is able to reproduce?
> Could someone share a "known good" kernel .config working in the
> past?
> 
> I'd be grateful for any advice or helpful suggestions.
> 
> Thanks,
> Tony

Hi,

I assume you are using the distro qemu v6.2?
Could you please try v9.1.0-rc2?
It contains quite a few emulation bugfixes.

Best regards,
Ilya