Re: Problem testing with S390x under QEMU on x86_64

[Leon Hwang <hffilwlqm@gmail.com>]

On 2024/8/22 01:28, Ilya Leoshkevich wrote:
> On Tue, 2024-08-20 at 17:38 -0700, Tony Ambardar wrote:
> 
> 
> [...]
> 
>> I used the command line:
>>     ./test_progs -d
>> get_stack_raw_tp,stacktrace_build_id,verifier_iterating_callbacks,tai
>> lcalls
>>
>> which includes the current DENYLIST.s390x as well as 'tailcalls',
>> which
>> is also excluded by the kernel-patches/bpf s390x CI. I note the CI
>> excludes several more tests that seem to work. Any idea why that is?
>>
>> For reference, the issue with 'tailcalls/tailcall_hierarchy_count' is
>> an
>> RCU stall and kernel hang:
>>
>> root@(none):/usr/libexec/kselftests-bpf# ./test_progs -v --debug -n
>> 332/19
>> bpf_testmod.ko is already unloaded.
>> Loading bpf_testmod.ko...
>> Successfully loaded bpf_testmod.ko.
>> test_tailcall_hierarchy_count:PASS:load obj 0 nsec
>> test_tailcall_hierarchy_count:PASS:find entry prog 0 nsec
>> test_tailcall_hierarchy_count:PASS:prog_fd 0 nsec
>> test_tailcall_hierarchy_count:PASS:find jmp_table 0 nsec
>> test_tailcall_hierarchy_count:PASS:map_fd 0 nsec
>> test_tailcall_hierarchy_count:PASS:update jmp_table 0 nsec
>> test_tailcall_hierarchy_count:PASS:find data_map 0 nsec
>> test_tailcall_hierarchy_count:PASS:open fentry_obj file 0 nsec
>> test_tailcall_hierarchy_count:PASS:find fentry prog 0 nsec
>> test_tailcall_hierarchy_count:PASS:set_attach_target subprog_tail 0
>> nsec
>> test_tailcall_hierarchy_count:PASS:load fentry_obj 0 nsec
>> test_tailcall_hierarchy_count:PASS:attach_trace 0 nsec
>> rcu: INFO: rcu_sched self-detected stall on CPU
>> rcu:    0-....: (1 GPs behind) idle=4eb4/1/0x4000000000000000
>> softirq=527/528 fqs=1050
>> rcu:    (t=2100 jiffies g=-379 q=20 ncpus=2)
>> CPU: 0 UID: 0 PID: 84 Comm: test_progs Tainted: G           O      
>> 6.10.0-12706-g853081e84612-dirty #111
>> Tainted: [O]=OOT_MODULE
>> Hardware name: QEMU 8561 QEMU (KVM/Linux)
>> Krnl PSW : 0704f00180000000 000003ffe00f8fca
>> (lock_release+0xf2/0x190)
>>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0
>> EA:3
>> Krnl GPRS: 00000000b298dd12 0000000000000000 000002f23fd767c8
>> 000003ffe1848800
>>            0000000000000001 0000037fe034edbc 0000037fe034fd74
>> 0000000000000001
>>            0700037fe034edc8 000003ffe0249e48 000003ffe1848800
>> 000003ffe19ba7c8
>>            000003ff9f7a7f90 0000037fe034ef00 000003ffe00f8f96
>> 0000037fe034ed78
>> Krnl Code: 000003ffe00f8fbe: a7820300           tmhh    %r8,768
>>            000003ffe00f8fc2: a7840004           brc    
>> 8,000003ffe00f8fca
>>           #000003ffe00f8fc6: ad03f0a0           stosm   160(%r15),3
>>           >000003ffe00f8fca: eb8ff0a80004       lmg    
>> %r8,%r15,168(%r15)
>>            000003ffe00f8fd0: 07fe               bcr     15,%r14
>>            000003ffe00f8fd2: c0e500011057       brasl  
>> %r14,000003ffe011b080
>>            000003ffe00f8fd8: ec26ffa6007e       cij    
>> %r2,0,6,000003ffe00f8f24
>>            000003ffe00f8fde: c01000b78b96       larl   
>> %r1,000003ffe17ea70a
>> Call Trace:
>>  [<000003ffe00f8fca>] lock_release+0xf2/0x190
>> ([<000003ffe00f8f96>] lock_release+0xbe/0x190)
>>  [<000003ffe0249ea4>] __bpf_prog_exit_recur+0x5c/0x68
>>  [<000003ff6001e0b0>] bpf_trampoline_73014444060+0xb0/0xd2
>>  [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8
>>  [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8
>>  [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8
>>  [<000003ff60024d2a>] bpf_prog_eb7edc599e93dcc8_entry+0x72/0xc8
>>  [<000003ff60024d2a>] bpf_prog_eb7edc599e93dcc8_entry+0x72/0xc8
>>  [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8
>>  [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8
>>  [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8
>>  [<000003ffe084ecee>] bpf_test_run+0x216/0x3a8
>>  [<000003ffe084f9cc>] bpf_prog_test_run_skb+0x21c/0x630
>>  [<000003ffe0202ad2>] __sys_bpf+0x7ea/0xbb0
>>  [<000003ffe0203114>] __s390x_sys_bpf+0x44/0
> 
> Thanks for the detailed analysis! I will need to port
> 
> commit 116e04ba1459fc08f80cf27b8c9f9f188be0fcb2
> Author: Leon Hwang <hffilwlqm@gmail.com>
> Date:   Sun Jul 14 20:39:00 2024 +0800
> 
>     bpf, x64: Fix tailcall hierarchy
> 
> to s390x to fix this.
> 

Hi Ilya,

I think you should wait for a while, as I'll fix another tailcall issue
in near future. After fixing the issue, you can port them both in one shot.

Thanks,
Leon