Re: Problem testing with S390x under QEMU on x86_64

[Tony Ambardar <tony.ambardar@gmail.com>]

On Tue, Aug 20, 2024 at 05:38:49PM -0700, Tony Ambardar wrote:
> 
> Hi Ilya,
> 
> Thanks for following up. As it happens, I did this the day before out of
> desperation after trying various kernel config and rootfs changes
> with no luck, and can confirm the system runs faster and without the
> kernel crashes noted above. Certainly the latest QEMU seems mandatory.
> 
> The good news is that 99% of tests with my cross-compiled test_progs
> work as expected out of the box, and some of the failing ones helped
> troubleshoot a few hidden libbpf issues. I'll outline the remaining
> failures for your feedback and comparison with native-built tests.
> 
> I used the command line:
>     ./test_progs -d get_stack_raw_tp,stacktrace_build_id,verifier_iterating_callbacks,tailcalls
> 

[snip]

> Aside from the tests above, I see only 3 failing tests:
> 
> All error logs:
> test_map_ptr:PASS:skel_open 0 nsec
> test_map_ptr:FAIL:skel_load unexpected error: -22 (errno 22)
> #165     map_ptr:FAIL
> subtest_userns:PASS:socketpair 0 nsec
> subtest_userns:PASS:fork 0 nsec
> recvfd:PASS:recvmsg 0 nsec
> recvfd:PASS:cmsg_null 0 nsec
> recvfd:PASS:cmsg_len 0 nsec
> recvfd:PASS:cmsg_level 0 nsec
> recvfd:PASS:cmsg_type 0 nsec
> parent:PASS:recv_bpffs_fd 0 nsec
> materialize_bpffs_fd:PASS:fs_cfg_cmds 0 nsec
> materialize_bpffs_fd:PASS:fs_cfg_maps 0 nsec
> materialize_bpffs_fd:PASS:fs_cfg_progs 0 nsec
> materialize_bpffs_fd:PASS:fs_cfg_attachs 0 nsec
> parent:PASS:materialize_bpffs_fd 0 nsec
> sendfd:PASS:sendmsg 0 nsec
> parent:PASS:send_mnt_fd 0 nsec
> recvfd:PASS:recvmsg 0 nsec
> recvfd:PASS:cmsg_null 0 nsec
> recvfd:PASS:cmsg_len 0 nsec
> recvfd:PASS:cmsg_level 0 nsec
> recvfd:PASS:cmsg_type 0 nsec
> parent:PASS:recv_token_fd 0 nsec
> parent:FAIL:waitpid_child unexpected error: 22 (errno 3)
> #402/9   token/obj_priv_implicit_token_envvar:FAIL
> #402     token:FAIL
> libbpf: prog 'on_event': BPF program load failed: Bad address
> libbpf: prog 'on_event': -- BEGIN PROG LOAD LOG --
> The sequence of 8193 jumps is too complex.
> verification time 2816240 usec
> stack depth 360
> processed 116096 insns (limit 1000000) max_states_per_insn 1 total_states 5061 peak_states 5061 mark_read 2540
> -- END PROG LOAD LOG --
> libbpf: prog 'on_event': failed to load: -14
> libbpf: failed to load object 'pyperf600.bpf.o'
> scale_test:FAIL:expect_success unexpected error: -14 (errno 14)
> #525     verif_scale_pyperf600:FAIL
> Summary: 559/4166 PASSED, 98 SKIPPED, 3 FAILED
> 

Hi Ilya,

A brief update with some good news: the 3 test failures above have been
resolved and all expected tests now pass on QEMU/s390x under x86_64.

Test '#165 map_ptr:FAIL' was a bug in my light-skeleton code, and fixed in
my patch series v2:
https://lore.kernel.org/bpf/cover.1724313164.git.tony.ambardar@gmail.com/

Test '#402/9 token/obj_priv_implicit_token_envvar:FAIL' was a problem in my
rootfs configuration and now passes after resolving.

Test '#525 verif_scale_pyperf600:FAIL' was caused by clang miscompilation
exposed by my use of clang-19 and clang-20. The test passes when built
with clang-17 (used by BPF CI) or clang-18 which I switched to use.

One symptom of the problem is easily seen by manually compiling:

$ clang-18  -g -Wall -Werror -D__TARGET_ARCH_s390 -mbig-endian -Itools/testing/selftests/bpf/tools/include -Itools/testing/selftests/bpf -Itools/include/uapi -Itools/testing/selftests/usr/include -Wno-compare-distinct-pointer-types -idirafter /usr/lib/llvm-18/lib/clang/18/include -idirafter /usr/local/include -idirafter /usr/lib/gcc-cross/s390x-linux-gnu/11/../../../../s390x-linux-gnu/include -idirafter /usr/include/s390x-linux-gnu -idirafter /usr/include -DENABLE_ATOMICS_TESTS -O2 --target=bpfeb -c tools/testing/selftests/bpf/progs/pyperf600.c -mcpu=v3 -o pyperf600.clang18.bpf.o

$ clang-19  -g -Wall -Werror -D__TARGET_ARCH_s390 -mbig-endian -Itools/testing/selftests/bpf/tools/include -Itools/testing/selftests/bpf -Itools/include/uapi -Itools/testing/selftests/usr/include -Wno-compare-distinct-pointer-types -idirafter /usr/lib/llvm-19/lib/clang/19/include -idirafter /usr/local/include -idirafter /usr/lib/gcc-cross/s390x-linux-gnu/11/../../../../s390x-linux-gnu/include -idirafter /usr/include/s390x-linux-gnu -idirafter /usr/include -DENABLE_ATOMICS_TESTS -O2 --target=bpfeb -c tools/testing/selftests/bpf/progs/pyperf600.c -mcpu=v3 -o pyperf600.clang19.bpf.o

$ llvm-readelf-18 -S pyperf600.clang{18,19}.bpf.o |grep .symtab
  [27] .symtab           SYMTAB          0000000000000000 1739d0 01ad60 18      1 4572  8
  [27] .symtab           SYMTAB          0000000000000000 14f048 0001e0 18      1  12  8

Notice that the .symtab has shrunk by ~200X for example going to clang-19!
(CCing llvm maintainers)


Kind regards,
Tony