Re: [PATCH bpf] selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep

[Puranjay Mohan <puranjay@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 9791 bytes --]


[SNIP]

>
> Hm, the latest run actually hangs in fexit_sleep (which is the test right after
> fexit_bpf2bpf). So looks like this was too early. It seems some CI runs pass on
> arm64 but others fail:
>
>    https://github.com/kernel-patches/bpf/actions/runs/9859826851/job/27224868398 (fail)
>    https://github.com/kernel-patches/bpf/actions/runs/9859837213/job/27224955045 (pass)
>
> Puranjay, do you have a chance to look into this again?
>
> Probably unrelated... but when I tried to reproduce this using qemu in full emulation mode [0], I am getting a kernel crash for fexit_sleep, but also for fexit_bpf2bpf, fentry_fexit
>
> stacktraces look like (for fentry_fexit)
>
>
> root@(none):/mnt/vmtest/selftests/bpf# ./test_progs -v -t fentry_fexit
> bpf_testmod.ko is already unloaded.
> Loading bpf_testmod.ko...
> Successfully loaded bpf_testmod.ko.
> test_fentry_fexit:PASS:fentry_skel_load 0 nsec
> test_fentry_fexit:PASS:fexit_skel_load 0 nsec
>
> test_fentry_fexit:PASS:fentry_attach 0 nsec
> test_fentry_fexit:PASS:fexit_attach 0 nsec
> Unable to handle kernel paging request at virtual address ffff0000c2a80e68
> Mem abort info:
>   ESR = 0x0000000096000004
>   EC = 0x25: DABT (current EL), IL = 32 bits
>   SET = 0, FnV = 0
>   EA = 0, S1PTW = 0
>   FSC = 0x04: level 0 translation fault
> Data abort info:
>   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
>   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
>   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000
> [ffff0000c2a80e68] pgd=1000000042f28003, p4d=0000000000000000
> Internal error: Oops: 0000000096000004 [#1] SMP
> Modules linked in: bpf_testmod(OE) [last unloaded: bpf_testmod(OE)]
> CPU: 0 PID: 97 Comm: test_progs Tainted: G           OE      6.10.0-rc6-gb0eedd920017-dirty #67
> Hardware name: linux,dummy-virt (DT)
> pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> pc : __bpf_tramp_enter+0x58/0x190
> lr : __bpf_tramp_enter+0xd8/0x190
> sp : ffff800084afbc10
> x29: ffff800084afbc10 x28: fff00000c28c2e80 x27: 0000000000000000
> x26: 0000000000000000 x25: 0000000000000050 x24: 0000000000000000
> x23: 000000000000000a x22: fff00000c28c2e80 x21: 0000ffffed100070
> x20: ffff800082032938 x19: ffff0000c2a80c00 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffffed100070
> x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000
> x11: 0000000000020007 x10: 0000000000000007 x9 : 00000000ffffffff
> x8 : 0000000000004008 x7 : ffff80008218fa78 x6 : 0000000000000000
> x5 : 0000000000000001 x4 : 0000000086db7919 x3 : 0000000095481a34
> x2 : 0000000000000001 x1 : fff00000c28c2e80 x0 : 0000000000000001
> Call trace:
>  __bpf_tramp_enter+0x58/0x190
>  bpf_trampoline_6442499844+0x44/0x158
>  bpf_fentry_test1+0x8/0x10
>  bpf_prog_test_run_tracing+0x190/0x328
>  __sys_bpf+0x844/0x2148
>  __arm64_sys_bpf+0x2c/0x48
>  invoke_syscall+0x4c/0x118
>  el0_svc_common.constprop.0+0x48/0xf0
>  do_el0_svc+0x24/0x38
>  el0_svc+0x4c/0x120
>  el0t_64_sync_handler+0xc0/0xc8
>  el0t_64_sync+0x190/0x198
> Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660)
> ---[ end trace 0000000000000000 ]---
> Kernel panic - not syncing: Oops: Fatal exception
> SMP: stopping secondary CPUs
> Kernel Offset: disabled
> CPU features: 0x00,00000006,8c13bd78,576676af
> Memory Limit: none
>
> For "fexit_sleep" and "fexit_bpf2bpf" respectively:
>
>
>  $ ( cd  9859826851 && vmtest -k kbuild-output/arch/arm64/boot/Image.gz -r ../aarch64-rootfs -a aarch64 '/bin/mount bpffs /sys/fs/bpf -t bpf && ip link set lo up && cd /mnt/vmtest/selftests/bpf/ && ./test_progs -v -t fexit_sleep' )
> => Image.gz
> ===> Booting
> ===> Setting up VM
> ===> Running command
> root@(none):/# bpf_testmod: loading out-of-tree module taints kernel.
> bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel
> Unable to handle kernel paging request at virtual address ffff0000c19c2668
> Mem abort info:
>   ESR = 0x0000000096000004
>   EC = 0x25: DABT (current EL), IL = 32 bits
>   SET = 0, FnV = 0
>   EA = 0, S1PTW = 0
>   FSC = 0x04: level 0 translation fault
> Data abort info:
>   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
>   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
>   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000
> [ffff0000c19c2668] pgd=1000000042f28003, p4d=0000000000000000
> Internal error: Oops: 0000000096000004 [#1] SMP
> Modules linked in: bpf_testmod(OE)
> CPU: 1 PID: 91 Comm: test_progs Tainted: G           OE      6.10.0-rc6-gb0eedd920017-dirty #67
> Hardware name: linux,dummy-virt (DT)
> pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> pc : __bpf_tramp_enter+0x58/0x190
> lr : __bpf_tramp_enter+0xd8/0x190
> sp : ffff800084c4bda0
> x29: ffff800084c4bda0 x28: fff00000c274ae80 x27: 0000000000000000
> x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
> x23: 0000000060001000 x22: 0000ffffa36b7a54 x21: 00000000ffffffff
> x20: ffff800082032938 x19: ffff0000c19c2400 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000
> x11: 0000000000020007 x10: 0000000000000007 x9 : 00000000ffffffff
> x8 : 0000000000004008 x7 : ffff80008218fa78 x6 : 0000000000000000
> x5 : 0000000000000001 x4 : 0000000086db7919 x3 : 0000000095481a34
> x2 : 0000000000000001 x1 : fff00000c274ae80 x0 : 0000000000000001
> Call trace:
>  __bpf_tramp_enter+0x58/0x190
>  bpf_trampoline_6442487232+0x44/0x158
>  __arm64_sys_nanosleep+0x8/0xf0
>  invoke_syscall+0x4c/0x118
>  el0_svc_common.constprop.0+0x48/0xf0
>  do_el0_svc+0x24/0x38
>  el0_svc+0x4c/0x120
>  el0t_64_sync_handler+0xc0/0xc8
>  el0t_64_sync+0x190/0x198
> Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660)
> ---[ end trace 0000000000000000 ]---
> Kernel panic - not syncing: Oops: Fatal exception
> SMP: stopping secondary CPUs
> Kernel Offset: disabled
> CPU features: 0x00,00000006,8c13bd78,576676af
> Memory Limit: none
> Failed to run command
>
> Caused by:
>     0: Failed to QGA guest-exec-status
>     1: error running guest_exec_status
>     2: Broken pipe (os error 32)
>     3: Broken pipe (os error 32)
> [11:46:14] chantra@devvm17937:scratchpad $
> [11:47:56] chantra@devvm17937:scratchpad $
> [11:47:57] chantra@devvm17937:scratchpad $ ( cd  9859826851 && vmtest -k kbuild-output/arch/arm64/boot/Image.gz -r ../aarch64-rootfs -a aarch64 '/bin/mount bpffs /sys/fs/bpf -t bpf && ip link set lo up && cd /mnt/vmtest/selftests/bpf/ && ./test_progs -v -t fexit_bpf2bpf' )
> => Image.gz
> ===> Booting
> ===> Setting up VM
> ===> Running command
> root@(none):/# bpf_testmod: loading out-of-tree module taints kernel.
> bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel
> Unable to handle kernel paging request at virtual address ffff0000c278de68
> Mem abort info:
>   ESR = 0x0000000096000004
>   EC = 0x25: DABT (current EL), IL = 32 bits
>   SET = 0, FnV = 0
>   EA = 0, S1PTW = 0
>   FSC = 0x04: level 0 translation fault
> Data abort info:
>   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
>   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
>   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000
> [ffff0000c278de68] pgd=1000000042f28003, p4d=0000000000000000
> Internal error: Oops: 0000000096000004 [#1] SMP
> Modules linked in: bpf_testmod(OE)
> CPU: 1 PID: 87 Comm: test_progs Tainted: G           OE      6.10.0-rc6-gb0eedd920017-dirty #67
> Hardware name: linux,dummy-virt (DT)
> pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> pc : __bpf_tramp_enter+0x58/0x190
> lr : __bpf_tramp_enter+0xd8/0x190
> sp : ffff800084c4ba90
> x29: ffff800084c4ba90 x28: ffff800080a32d10 x27: ffff800080a32d80
> x26: ffff8000813e0ad8 x25: ffff800084c4bce4 x24: ffff800082fbd048
> x23: 0000000000000001 x22: fff00000c2732e80 x21: fff00000c18a3200
> x20: ffff800082032938 x19: ffff0000c278dc00 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000aaaabcc22aa0
> x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000
> x11: 0000000000000000 x10: 000000000ac0d5af x9 : 000000000ac0d5af
> x8 : 00000000a4d7a457 x7 : ffff80008218fa78 x6 : 0000000000000000
> x5 : 0000000000000002 x4 : 0000000006fa0785 x3 : 0000000081d7cd4c
> x2 : 0000000000000202 x1 : fff00000c2732e80 x0 : 0000000000000001
> Call trace:
>  __bpf_tramp_enter+0x58/0x190
>  bpf_trampoline_34359738386+0x44/0xf8
>  bpf_prog_3b052b77318ab7c4_test_pkt_md_access+0x8/0x118
>  bpf_test_run+0x200/0x3a0
>  bpf_prog_test_run_skb+0x328/0x6d8
>  __sys_bpf+0x844/0x2148
>  __arm64_sys_bpf+0x2c/0x48
>  invoke_syscall+0x4c/0x118
>  el0_svc_common.constprop.0+0x48/0xf0
>  do_el0_svc+0x24/0x38
>  el0_svc+0x4c/0x120
>  el0t_64_sync_handler+0xc0/0xc8
>  el0t_64_sync+0x190/0x198
> Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660)
> ---[ end trace 0000000000000000 ]---
> Kernel panic - not syncing: Oops: Fatal exception in interrupt
> SMP: stopping secondary CPUs
> Kernel Offset: disabled
> CPU features: 0x00,00000006,8c13bd78,576676af
> Memory Limit: none
> Failed to run command
>
> Caused by:
>     0: Failed to QGA guest-exec-status
>     1: error running guest_exec_status
>     2: Broken pipe (os error 32)
>     3: Broken pipe (os error 32)
>
>
> [0] https://chantra.github.io/bpfcitools/bpfci-troubleshooting.html


Thanks for sharing the logs,
I will try to reproduce this and find the root cause.

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]