* Re: [Syzkaller & bisect] There is perf_event_open triggered soft lockup BUG in v6.5-rc5 kernel
[not found] <ZNhVUnPCs/SLlm12@xpf.sh.intel.com>
@ 2023-08-13 4:06 ` Pengfei Xu
0 siblings, 0 replies; only message in thread
From: Pengfei Xu @ 2023-08-13 4:06 UTC (permalink / raw)
To: paul.kocialkowski
Cc: linux-media-owner, heng.su, isaku.yamahata, lkp, hverkuil-cisco,
mchehab, linux-media
++ linux-media emial list.
On 2023-08-13 at 12:00:18 +0800, Pengfei Xu wrote:
> Hi Paul,
>
> Greeting!
>
> There is perf_event_open triggered soft lockup BUG in v6.5-rc5 kernel.
>
> All detailed info link: https://github.com/xupengfe/syzkaller_logs/tree/main/230812_084142_BUG_soft_lockup
> Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/repro.c
> Syzkaller reproduced step: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/repro.prog
> Syzkaller report analysis: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/repro.report
> Kconfig(Need "make olddefconfig" to generate target kconfig): https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/kconfig_origin
> Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/bisect_info.log
> Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230812_084142_BUG_soft_lockup/52a93d39b17dc7eb98b6aa3edb93943248e03b2f_dmesg.log
> v6.5-rc5 bzImage: https://github.com/xupengfe/syzkaller_logs/raw/main/230812_084142_BUG_soft_lockup/bzImage_52a93d39b17dc7eb98b6aa3edb93943248e03b2f.tar.gz
>
> This issue could be reproduced in 253s.
>
> Bisected and found first bad commit is:
> f042b08b833d media: sun6i-mipi-csi2: Require both pads to be connected for streaming
>
> "
> [ 28.083677] cgroup: Unknown subsys name 'rlimit'
> [ 73.568711] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [repro:1253]
> [ 73.573792] Kernel panic - not syncing: softlockup: hung tasks
> [ 73.574833] CPU: 0 PID: 1253 Comm: repro Tainted: G L 6.5.0-rc5-52a93d39b17d #1
> [ 73.576363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
> [ 73.578364] Call Trace:
> [ 73.578824] <IRQ>
> [ 73.579210] dump_stack_lvl+0xa9/0x110
> [ 73.579938] dump_stack+0x19/0x20
> [ 73.580562] panic+0x660/0x6d0
> [ 73.581165] ? __pfx_panic+0x10/0x10
> [ 73.581866] ? watchdog_timer_fn+0x4ab/0x5c0
> [ 73.582674] watchdog_timer_fn+0x4c9/0x5c0
> [ 73.583421] ? __pfx_watchdog_timer_fn+0x10/0x10
> [ 73.584258] __hrtimer_run_queues+0x61a/0xba0
> [ 73.585068] ? __pfx___hrtimer_run_queues+0x10/0x10
> [ 73.585970] ? ktime_get_update_offsets_now+0x25b/0x340
> [ 73.586932] ? write_comp_data+0x2f/0x90
> [ 73.587678] hrtimer_interrupt+0x326/0x7b0
> [ 73.588453] __sysvec_apic_timer_interrupt+0x14f/0x3f0
> [ 73.589411] sysvec_apic_timer_interrupt+0x48/0xd0
> [ 73.590308] asm_sysvec_apic_timer_interrupt+0x1f/0x30
> [ 73.591252] RIP: 0010:is_bpf_text_address+0x139/0x1c0
> [ 73.592184] Code: ff e8 7b 3b e7 ff 48 c7 c2 80 05 93 85 be ef 02 00 00 48 c7 c7 e0 05 93 85 c6 05 ca 5f fa 05 01 e8 dc 6e c5 ff e9 22 ff ff ff <e8> 52 3b e7 ff e8 5d b2 ce ff 31 ff 89 c3 89 c6 e8 a2 3c e7 ff 84
> [ 73.595436] RSP: 0018:ffff88806ca096e0 EFLAGS: 00000202
> [ 73.596374] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81772d6f
> [ 73.597646] RDX: 0000000000000000 RSI: ffff88800ff7a500 RDI: 0000000000000003
> [ 73.598921] RBP: ffff88806ca096f0 R08: fffffbfff133afd9 R09: fffffbfff133afd9
> [ 73.600185] R10: fffffbfff133afd8 R11: ffffffff899d7ec7 R12: 0000000000000000
> [ 73.601454] R13: 0000000000000000 R14: ffff88800ff7a500 R15: 0000000000000000
> [ 73.602740] ? is_bpf_text_address+0x7f/0x1c0
> [ 73.603557] kernel_text_address+0xd5/0xe0
> [ 73.604313] __kernel_text_address+0x16/0x50
> [ 73.605109] unwind_get_return_address+0x6e/0xc0
> [ 73.605979] ? __pfx_stack_trace_consume_entry+0x10/0x10
> [ 73.606951] arch_stack_walk+0xa5/0x100
> [ 73.607671] stack_trace_save+0x8b/0xc0
> [ 73.608385] ? __pfx_stack_trace_save+0x10/0x10
> [ 73.609237] kasan_save_stack+0x2a/0x50
> [ 73.609964] ? kasan_save_stack+0x2a/0x50
> [ 73.610707] ? kasan_set_track+0x29/0x40
> [ 73.611458] ? kasan_save_alloc_info+0x22/0x30
> [ 73.612274] ? __kasan_slab_alloc+0x64/0x80
> [ 73.613046] ? slab_post_alloc_hook+0x84/0x3b0
> [ 73.613872] ? kmem_cache_alloc_node+0x18a/0x2e0
> [ 73.614725] ? __alloc_skb+0x2b6/0x360
> [ 73.615435] ? __neigh_notify+0x91/0x170
> [ 73.616171] ? neigh_timer_handler+0x2b8/0xfa0
> [ 73.616981] ? call_timer_fn+0x1a5/0x570
> [ 73.617712] ? run_timer_softirq+0x714/0x14d0
> [ 73.618511] ? __do_softirq+0x1b7/0x77c
> [ 73.619222] ? irq_exit_rcu+0xc1/0x110
> [ 73.619928] ? sysvec_apic_timer_interrupt+0xa1/0xd0
> [ 73.620828] ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
> [ 73.621797] ? generic_exec_single+0x278/0x540
> [ 73.622619] ? smp_call_function_single+0x193/0x460
> [ 73.623506] ? task_function_call+0x10e/0x1a0
> [ 73.624318] ? perf_install_in_context+0x2f9/0x5b0
> [ 73.625191] ? __do_sys_perf_event_open+0x1a9a/0x2a50
> [ 73.626130] ? __x64_sys_perf_event_open+0xc7/0x160
> [ 73.627024] ? do_syscall_64+0x3b/0x90
> [ 73.627721] ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [ 73.628676] ? write_comp_data+0x2f/0x90
> [ 73.629422] ? debug_smp_processor_id+0x20/0x30
> [ 73.630266] ? __sanitizer_cov_trace_pc+0x25/0x60
> [ 73.631131] ? write_comp_data+0x2f/0x90
> [ 73.631871] ? mark_held_locks+0xb7/0x140
> [ 73.632608] ? irqentry_exit+0x3e/0xa0
> [ 73.633311] ? sysvec_irq_work+0x58/0xd0
> [ 73.634065] ? __this_cpu_preempt_check+0x20/0x30
> [ 73.634928] ? lockdep_hardirqs_on+0x8a/0x110
> [ 73.635731] ? irqentry_exit+0x3e/0xa0
> [ 73.636433] ? trace_hardirqs_on+0x26/0x120
> [ 73.637208] kasan_set_track+0x29/0x40
> [ 73.637939] kasan_save_alloc_info+0x22/0x30
> [ 73.638726] __kasan_slab_alloc+0x64/0x80
> [ 73.639487] slab_post_alloc_hook+0x84/0x3b0
> [ 73.640283] kmem_cache_alloc_node+0x18a/0x2e0
> [ 73.641101] ? __alloc_skb+0x2b6/0x360
> [ 73.641817] __alloc_skb+0x2b6/0x360
> [ 73.642494] ? __pfx___alloc_skb+0x10/0x10
> [ 73.643255] ? asm_sysvec_irq_work+0x1f/0x30
> [ 73.644063] __neigh_notify+0x91/0x170
> [ 73.644768] neigh_timer_handler+0x2b8/0xfa0
> [ 73.645572] ? __pfx_neigh_timer_handler+0x10/0x10
> [ 73.646448] call_timer_fn+0x1a5/0x570
> [ 73.647137] ? asm_sysvec_irq_work+0x1f/0x30
> [ 73.647929] ? __pfx_call_timer_fn+0x10/0x10
> [ 73.648725] ? __pfx_neigh_timer_handler+0x10/0x10
> [ 73.649611] run_timer_softirq+0x714/0x14d0
> [ 73.650401] ? __pfx_run_timer_softirq+0x10/0x10
> [ 73.651240] ? asm_sysvec_irq_work+0x1f/0x30
> [ 73.652040] ? __pfx_run_timer_softirq+0x10/0x10
> [ 73.652890] __do_softirq+0x1b7/0x77c
> [ 73.653593] irq_exit_rcu+0xc1/0x110
> [ 73.654261] sysvec_apic_timer_interrupt+0xa1/0xd0
> [ 73.655132] </IRQ>
> [ 73.655532] <TASK>
> [ 73.655943] asm_sysvec_apic_timer_interrupt+0x1f/0x30
> [ 73.656889] RIP: 0010:generic_exec_single+0x278/0x540
> [ 73.657832] Code: e8 0d 85 0b 00 4c 89 e6 31 ff 31 db e8 11 87 0b 00 4d 85 e4 0f 84 3e fe ff ff e8 f3 84 0b 00 e8 8e f0 17 00 fb 0f 1f 44 00 00 <e9> 29 fe ff ff e8 de 84 0b 00 65 8b 1d a3 6a b1 7e 89 db be 08 00
> [ 73.661070] RSP: 0018:ffff88800fa97ac8 EFLAGS: 00000202
> [ 73.662013] RAX: 0000000000000485 RBX: 0000000000000000 RCX: 1ffffffff1343c28
> [ 73.663275] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
> [ 73.664538] RBP: ffff88800fa97af0 R08: 0000000000000001 R09: fffffbfff133afee
> [ 73.665808] R10: 0000000000000001 R11: ffffffff899d7f6f R12: 0000000000000200
> [ 73.667071] R13: ffffffff818e9ad0 R14: ffff88800fa97c20 R15: ffff88800fa97b48
> [ 73.668333] ? __pfx_remote_function+0x10/0x10
> [ 73.669163] ? generic_exec_single+0x272/0x540
> [ 73.670005] smp_call_function_single+0x193/0x460
> [ 73.670866] ? write_comp_data+0x2f/0x90
> [ 73.671590] ? __pfx_remote_function+0x10/0x10
> [ 73.672406] ? __pfx_smp_call_function_single+0x10/0x10
> [ 73.673371] ? __pfx_remote_function+0x10/0x10
> [ 73.674209] ? __sanitizer_cov_trace_pc+0x25/0x60
> [ 73.675081] task_function_call+0x10e/0x1a0
> [ 73.675855] ? __pfx_task_function_call+0x10/0x10
> [ 73.676720] ? __pfx___perf_install_in_context+0x10/0x10
> [ 73.677693] ? exclusive_event_installable+0x25b/0x320
> [ 73.678626] ? write_comp_data+0x2f/0x90
> [ 73.679358] perf_install_in_context+0x2f9/0x5b0
> [ 73.680206] ? __pfx_perf_install_in_context+0x10/0x10
> [ 73.681145] ? write_comp_data+0x2f/0x90
> [ 73.681897] ? __sanitizer_cov_trace_pc+0x25/0x60
> [ 73.682767] __do_sys_perf_event_open+0x1a9a/0x2a50
> [ 73.683672] ? __pfx___do_sys_perf_event_open+0x10/0x10
> [ 73.684649] __x64_sys_perf_event_open+0xc7/0x160
> [ 73.685526] ? syscall_enter_from_user_mode+0x51/0x60
> [ 73.686449] do_syscall_64+0x3b/0x90
> [ 73.687114] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [ 73.688035] RIP: 0033:0x7f904803ee5d
> [ 73.688694] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 93 af 1b 00 f7 d8 64 89 01 48
> [ 73.691934] RSP: 002b:00007ffcb1c151a8 EFLAGS: 00000206 ORIG_RAX: 000000000000012a
> [ 73.693269] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f904803ee5d
> [ 73.694545] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000280
> [ 73.695804] RBP: 00007ffcb1c151b0 R08: 0000000000000000 R09: 00007ffcb1c15200
> [ 73.697064] R10: 00000000ffffffff R11: 0000000000000206 R12: 00007ffcb1c15348
> [ 73.698337] R13: 0000000000407f0f R14: 000000000040adf8 R15: 00007f9048381000
> [ 73.699619] </TASK>
> [ 73.700900] Kernel Offset: disabled
> [ 73.701602] ---[ end Kernel panic - not syncing: softlockup: hung tasks ]---
> "
>
> ---
>
> If you don't need the following environment to reproduce the problem or if you
> already have one, please ignore the following information.
>
> How to reproduce:
> git clone https://gitlab.com/xupengfe/repro_vm_env.git
> cd repro_vm_env
> tar -xvf repro_vm_env.tar.gz
> cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
> // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
> // You could change the bzImage_xxx as you want
> // Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
> You could use below command to log in, there is no password for root.
> ssh -p 10023 root@localhost
>
> After login vm(virtual machine) successfully, you could transfer reproduced
> binary to the vm by below way, and reproduce the problem in vm:
> gcc -pthread -o repro repro.c
> scp -P 10023 repro root@localhost:/root/
>
> Get the bzImage for target kernel:
> Please use target kconfig and copy it to kernel_src/.config
> make olddefconfig
> make -jx bzImage //x should equal or less than cpu num your pc has
>
> Fill the bzImage file name into above start3.sh to load the target kernel in vm.
>
>
> Tips:
> If you already have qemu-system-x86_64, please ignore below info.
> If you want to install qemu v7.1.0 version:
> git clone https://github.com/qemu/qemu.git
> cd qemu
> git checkout -f v7.1.0
> mkdir build
> cd build
> yum install -y ninja-build.x86_64
> yum -y install libslirp-devel.x86_64
> ../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp
> make
> make install
>
> Best Regards,
> Thanks!
^ permalink raw reply [flat|nested] only message in thread