* Re: [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
@ 2024-11-23 13:17 ` syzbot
2024-11-23 23:32 ` Hillf Danton
2024-11-26 14:24 ` syzbot
` (5 subsequent siblings)
6 siblings, 1 reply; 12+ messages in thread
From: syzbot @ 2024-11-23 13:17 UTC (permalink / raw)
To: bp, dave.hansen, dwmw2, hpa, kvm, linux-kernel, mingo, paul,
pbonzini, seanjc, syzkaller-bugs, tglx, x86
syzbot has found a reproducer for the following issue on:
HEAD commit: 06afb0f36106 Merge tag 'trace-v6.13' of git://git.kernel.o..
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=17ff7930580000
kernel config: https://syzkaller.appspot.com/x/.config?x=95b76860fd16c857
dashboard link: https://syzkaller.appspot.com/bug?extid=919877893c9d28162dc2
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=142981c0580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1371975f980000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/49111529582a/disk-06afb0f3.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/f04577ad9add/vmlinux-06afb0f3.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b352b4fae995/bzImage-06afb0f3.xz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com
=============================
[ BUG: Invalid wait context ]
6.12.0-syzkaller-07834-g06afb0f36106 #0 Not tainted
-----------------------------
kworker/0:1/9 is trying to lock:
ffffc90003bca460 (&gpc->lock){....}-{3:3}, at: kvm_xen_set_evtchn_fast+0x1ee/0xa00 arch/x86/kvm/xen.c:1755
other info that might help us debug this:
context-{2:2}
6 locks held by kworker/0:1/9:
#0: ffff888144a92148 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3204 [inline]
#0: ffff888144a92148 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: process_scheduled_works+0x93b/0x1850 kernel/workqueue.c:3310
#1: ffffc900000e7d00 ((work_completion)(&hub->events)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3205 [inline]
#1: ffffc900000e7d00 ((work_completion)(&hub->events)){+.+.}-{0:0}, at: process_scheduled_works+0x976/0x1850 kernel/workqueue.c:3310
#2: ffff888145711190 (&dev->mutex){....}-{4:4}, at: device_lock include/linux/device.h:1014 [inline]
#2: ffff888145711190 (&dev->mutex){....}-{4:4}, at: hub_event+0x1fe/0x5150 drivers/usb/core/hub.c:5849
#3: ffffffff8e817de0 (console_lock){+.+.}-{0:0}, at: dev_vprintk_emit+0x2ae/0x330 drivers/base/core.c:4942
#4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: rcu_try_lock_acquire include/linux/rcupdate.h:342 [inline]
#4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: srcu_read_lock_nmisafe include/linux/srcu.h:297 [inline]
#4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: console_srcu_read_lock kernel/printk/printk.c:288 [inline]
#4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: console_flush_all+0x1a3/0xeb0 kernel/printk/printk.c:3187
#5: ffffc90003bca8c8 (&kvm->srcu){.?.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
#5: ffffc90003bca8c8 (&kvm->srcu){.?.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
#5: ffffc90003bca8c8 (&kvm->srcu){.?.+}-{0:0}, at: kvm_xen_set_evtchn_fast+0x1bb/0xa00 arch/x86/kvm/xen.c:1753
stack backtrace:
CPU: 0 UID: 0 PID: 9 Comm: kworker/0:1 Not tainted 6.12.0-syzkaller-07834-g06afb0f36106 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: usb_hub_wq hub_event
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:94 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
print_lock_invalid_wait_context kernel/locking/lockdep.c:4826 [inline]
check_wait_context kernel/locking/lockdep.c:4898 [inline]
__lock_acquire+0x15a8/0x2100 kernel/locking/lockdep.c:5176
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
__raw_read_lock_irqsave include/linux/rwlock_api_smp.h:160 [inline]
_raw_read_lock_irqsave+0xdd/0x130 kernel/locking/spinlock.c:236
kvm_xen_set_evtchn_fast+0x1ee/0xa00 arch/x86/kvm/xen.c:1755
xen_timer_callback+0x1a0/0x380 arch/x86/kvm/xen.c:140
__run_hrtimer kernel/time/hrtimer.c:1739 [inline]
__hrtimer_run_queues+0x551/0xd50 kernel/time/hrtimer.c:1803
hrtimer_interrupt+0x403/0xa40 kernel/time/hrtimer.c:1865
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1038 [inline]
__sysvec_apic_timer_interrupt+0x110/0x420 arch/x86/kernel/apic/apic.c:1055
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1049
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:console_flush_all+0x996/0xeb0
Code: 48 21 c3 0f 85 16 02 00 00 e8 66 aa 20 00 4c 8b 7c 24 10 4d 85 f6 75 07 e8 57 aa 20 00 eb 06 e8 50 aa 20 00 fb 48 8b 5c 24 18 <48> 8b 44 24 30 42 80 3c 28 00 74 08 48 89 df e8 76 61 8b 00 4c 8b
RSP: 0018:ffffc900000e7000 EFLAGS: 00000293
RAX: ffffffff8174a2e0 RBX: ffffffff8f17fa58 RCX: ffff88801bef8000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc900000e71b0 R08: ffffffff8174a2b7 R09: 1ffffffff285cb10
R10: dffffc0000000000 R11: fffffbfff285cb11 R12: ffffffff8f17fa00
R13: dffffc0000000000 R14: 0000000000000200 R15: ffffc900000e7200
__console_flush_and_unlock kernel/printk/printk.c:3269 [inline]
console_unlock+0x14f/0x3b0 kernel/printk/printk.c:3309
vprintk_emit+0x730/0xa10 kernel/printk/printk.c:2432
dev_vprintk_emit+0x2ae/0x330 drivers/base/core.c:4942
dev_printk_emit+0xdd/0x120 drivers/base/core.c:4953
_dev_info+0x122/0x170 drivers/base/core.c:5011
show_string drivers/usb/core/hub.c:2357 [inline]
announce_device drivers/usb/core/hub.c:2375 [inline]
usb_new_device+0xd02/0x19a0 drivers/usb/core/hub.c:2632
hub_port_connect drivers/usb/core/hub.c:5521 [inline]
hub_port_connect_change drivers/usb/core/hub.c:5661 [inline]
port_event drivers/usb/core/hub.c:5821 [inline]
hub_event+0x2d6d/0x5150 drivers/usb/core/hub.c:5903
process_one_work kernel/workqueue.c:3229 [inline]
process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310
worker_thread+0x870/0xd30 kernel/workqueue.c:3391
kthread+0x2f0/0x390 kernel/kthread.c:389
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
</TASK>
----------------
Code disassembly (best guess):
0: 48 21 c3 and %rax,%rbx
3: 0f 85 16 02 00 00 jne 0x21f
9: e8 66 aa 20 00 call 0x20aa74
e: 4c 8b 7c 24 10 mov 0x10(%rsp),%r15
13: 4d 85 f6 test %r14,%r14
16: 75 07 jne 0x1f
18: e8 57 aa 20 00 call 0x20aa74
1d: eb 06 jmp 0x25
1f: e8 50 aa 20 00 call 0x20aa74
24: fb sti
25: 48 8b 5c 24 18 mov 0x18(%rsp),%rbx
* 2a: 48 8b 44 24 30 mov 0x30(%rsp),%rax <-- trapping instruction
2f: 42 80 3c 28 00 cmpb $0x0,(%rax,%r13,1)
34: 74 08 je 0x3e
36: 48 89 df mov %rbx,%rdi
39: e8 76 61 8b 00 call 0x8b61b4
3e: 4c rex.WR
3f: 8b .byte 0x8b
---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-23 13:17 ` syzbot
@ 2024-11-23 23:32 ` Hillf Danton
0 siblings, 0 replies; 12+ messages in thread
From: Hillf Danton @ 2024-11-23 23:32 UTC (permalink / raw)
To: Sebastian Andrzej Siewior, Boqun Feng
Cc: syzbot, kvm, linux-kernel, pbonzini, seanjc, syzkaller-bugs
Loop in lock people.
On Sat, 23 Nov 2024 05:17:19 -0800
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 06afb0f36106 Merge tag 'trace-v6.13' of git://git.kernel.o..
> git tree: upstream
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=17ff7930580000
> kernel config: https://syzkaller.appspot.com/x/.config?x=95b76860fd16c857
> dashboard link: https://syzkaller.appspot.com/bug?extid=919877893c9d28162dc2
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=142981c0580000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1371975f980000
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/49111529582a/disk-06afb0f3.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/f04577ad9add/vmlinux-06afb0f3.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/b352b4fae995/bzImage-06afb0f3.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com
>
> =============================
> [ BUG: Invalid wait context ]
> 6.12.0-syzkaller-07834-g06afb0f36106 #0 Not tainted
> -----------------------------
> kworker/0:1/9 is trying to lock:
> ffffc90003bca460 (&gpc->lock){....}-{3:3}, at: kvm_xen_set_evtchn_fast+0x1ee/0xa00 arch/x86/kvm/xen.c:1755
> other info that might help us debug this:
> context-{2:2}
> 6 locks held by kworker/0:1/9:
> #0: ffff888144a92148 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3204 [inline]
> #0: ffff888144a92148 ((wq_completion)usb_hub_wq){+.+.}-{0:0}, at: process_scheduled_works+0x93b/0x1850 kernel/workqueue.c:3310
> #1: ffffc900000e7d00 ((work_completion)(&hub->events)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3205 [inline]
> #1: ffffc900000e7d00 ((work_completion)(&hub->events)){+.+.}-{0:0}, at: process_scheduled_works+0x976/0x1850 kernel/workqueue.c:3310
> #2: ffff888145711190 (&dev->mutex){....}-{4:4}, at: device_lock include/linux/device.h:1014 [inline]
> #2: ffff888145711190 (&dev->mutex){....}-{4:4}, at: hub_event+0x1fe/0x5150 drivers/usb/core/hub.c:5849
> #3: ffffffff8e817de0 (console_lock){+.+.}-{0:0}, at: dev_vprintk_emit+0x2ae/0x330 drivers/base/core.c:4942
> #4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: rcu_try_lock_acquire include/linux/rcupdate.h:342 [inline]
> #4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: srcu_read_lock_nmisafe include/linux/srcu.h:297 [inline]
> #4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: console_srcu_read_lock kernel/printk/printk.c:288 [inline]
> #4: ffffffff8e8179f0 (console_srcu){....}-{0:0}, at: console_flush_all+0x1a3/0xeb0 kernel/printk/printk.c:3187
> #5: ffffc90003bca8c8 (&kvm->srcu){.?.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> #5: ffffc90003bca8c8 (&kvm->srcu){.?.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> #5: ffffc90003bca8c8 (&kvm->srcu){.?.+}-{0:0}, at: kvm_xen_set_evtchn_fast+0x1bb/0xa00 arch/x86/kvm/xen.c:1753
> stack backtrace:
> CPU: 0 UID: 0 PID: 9 Comm: kworker/0:1 Not tainted 6.12.0-syzkaller-07834-g06afb0f36106 #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
> Workqueue: usb_hub_wq hub_event
> Call Trace:
> <IRQ>
> __dump_stack lib/dump_stack.c:94 [inline]
> dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
> print_lock_invalid_wait_context kernel/locking/lockdep.c:4826 [inline]
> check_wait_context kernel/locking/lockdep.c:4898 [inline]
> __lock_acquire+0x15a8/0x2100 kernel/locking/lockdep.c:5176
> lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
> __raw_read_lock_irqsave include/linux/rwlock_api_smp.h:160 [inline]
> _raw_read_lock_irqsave+0xdd/0x130 kernel/locking/spinlock.c:236
> kvm_xen_set_evtchn_fast+0x1ee/0xa00 arch/x86/kvm/xen.c:1755
> xen_timer_callback+0x1a0/0x380 arch/x86/kvm/xen.c:140
> __run_hrtimer kernel/time/hrtimer.c:1739 [inline]
> __hrtimer_run_queues+0x551/0xd50 kernel/time/hrtimer.c:1803
> hrtimer_interrupt+0x403/0xa40 kernel/time/hrtimer.c:1865
> local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1038 [inline]
> __sysvec_apic_timer_interrupt+0x110/0x420 arch/x86/kernel/apic/apic.c:1055
> instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
> sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1049
> </IRQ>
Another locking issue in irq context [1]
[1] https://lore.kernel.org/lkml/20241116232957.1223-1-hdanton@sina.com/
> <TASK>
> asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
> RIP: 0010:console_flush_all+0x996/0xeb0
> Code: 48 21 c3 0f 85 16 02 00 00 e8 66 aa 20 00 4c 8b 7c 24 10 4d 85 f6 75 07 e8 57 aa 20 00 eb 06 e8 50 aa 20 00 fb 48 8b 5c 24 18 <48> 8b 44 24 30 42 80 3c 28 00 74 08 48 89 df e8 76 61 8b 00 4c 8b
> RSP: 0018:ffffc900000e7000 EFLAGS: 00000293
> RAX: ffffffff8174a2e0 RBX: ffffffff8f17fa58 RCX: ffff88801bef8000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffffc900000e71b0 R08: ffffffff8174a2b7 R09: 1ffffffff285cb10
> R10: dffffc0000000000 R11: fffffbfff285cb11 R12: ffffffff8f17fa00
> R13: dffffc0000000000 R14: 0000000000000200 R15: ffffc900000e7200
> __console_flush_and_unlock kernel/printk/printk.c:3269 [inline]
> console_unlock+0x14f/0x3b0 kernel/printk/printk.c:3309
> vprintk_emit+0x730/0xa10 kernel/printk/printk.c:2432
> dev_vprintk_emit+0x2ae/0x330 drivers/base/core.c:4942
> dev_printk_emit+0xdd/0x120 drivers/base/core.c:4953
> _dev_info+0x122/0x170 drivers/base/core.c:5011
> show_string drivers/usb/core/hub.c:2357 [inline]
> announce_device drivers/usb/core/hub.c:2375 [inline]
> usb_new_device+0xd02/0x19a0 drivers/usb/core/hub.c:2632
> hub_port_connect drivers/usb/core/hub.c:5521 [inline]
> hub_port_connect_change drivers/usb/core/hub.c:5661 [inline]
> port_event drivers/usb/core/hub.c:5821 [inline]
> hub_event+0x2d6d/0x5150 drivers/usb/core/hub.c:5903
> process_one_work kernel/workqueue.c:3229 [inline]
> process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310
> worker_thread+0x870/0xd30 kernel/workqueue.c:3391
> kthread+0x2f0/0x390 kernel/kthread.c:389
> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> </TASK>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
2024-11-23 13:17 ` syzbot
@ 2024-11-26 14:24 ` syzbot
2024-11-26 14:49 ` David Woodhouse
2026-03-15 13:58 ` Forwarded: syzbot
` (4 subsequent siblings)
6 siblings, 1 reply; 12+ messages in thread
From: syzbot @ 2024-11-26 14:24 UTC (permalink / raw)
To: bigeasy, boqun.feng, bp, dave.hansen, dwmw2, hdanton, hpa, kvm,
linux-kernel, longman, mingo, paul, pbonzini, seanjc,
syzkaller-bugs, tglx, x86
syzbot has bisected this issue to:
commit 560af5dc839eef08a273908f390cfefefb82aa04
Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Date: Wed Oct 9 15:45:03 2024 +0000
lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING.
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=162ef5c0580000
start commit: 06afb0f36106 Merge tag 'trace-v6.13' of git://git.kernel.o..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=152ef5c0580000
console output: https://syzkaller.appspot.com/x/log.txt?x=112ef5c0580000
kernel config: https://syzkaller.appspot.com/x/.config?x=95b76860fd16c857
dashboard link: https://syzkaller.appspot.com/bug?extid=919877893c9d28162dc2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=142981c0580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1371975f980000
Reported-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com
Fixes: 560af5dc839e ("lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING.")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-26 14:24 ` syzbot
@ 2024-11-26 14:49 ` David Woodhouse
2024-11-26 15:03 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 12+ messages in thread
From: David Woodhouse @ 2024-11-26 14:49 UTC (permalink / raw)
To: syzbot, bigeasy, boqun.feng, bp, dave.hansen, hdanton, hpa, kvm,
linux-kernel, longman, mingo, paul, pbonzini, seanjc,
syzkaller-bugs, tglx, x86
[-- Attachment #1: Type: text/plain, Size: 535 bytes --]
On Tue, 2024-11-26 at 06:24 -0800, syzbot wrote:
> syzbot has bisected this issue to:
>
> commit 560af5dc839eef08a273908f390cfefefb82aa04
> Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Date: Wed Oct 9 15:45:03 2024 +0000
>
> lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING.
That's not it; this has always been broken with PREEMPT_RT I think.
There was an attempt to fix it in
https://lore.kernel.org/all/20240227115648.3104-8-dwmw2@infradead.org/
I'll dust that off and try again.
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-26 14:49 ` David Woodhouse
@ 2024-11-26 15:03 ` Sebastian Andrzej Siewior
2024-11-26 16:26 ` David Woodhouse
0 siblings, 1 reply; 12+ messages in thread
From: Sebastian Andrzej Siewior @ 2024-11-26 15:03 UTC (permalink / raw)
To: David Woodhouse
Cc: syzbot, boqun.feng, bp, dave.hansen, hdanton, hpa, kvm,
linux-kernel, longman, mingo, paul, pbonzini, seanjc,
syzkaller-bugs, tglx, x86
On 2024-11-26 14:49:40 [+0000], David Woodhouse wrote:
> On Tue, 2024-11-26 at 06:24 -0800, syzbot wrote:
> > syzbot has bisected this issue to:
> >
> > commit 560af5dc839eef08a273908f390cfefefb82aa04
> > Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> > Date: Wed Oct 9 15:45:03 2024 +0000
> >
> > lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING.
>
> That's not it; this has always been broken with PREEMPT_RT I think.
> There was an attempt to fix it in
> https://lore.kernel.org/all/20240227115648.3104-8-dwmw2@infradead.org/
>
> I'll dust that off and try again.
Oh thank you. The timer has been made to always expire in hardirq due to
HRTIMER_MODE_ABS_HARD, this is why you see the splat. If the hardirq
invocation is needed/ possible then the callback needs to be updated.
The linked patch has this hunk:
|- read_lock_irqsave(&gpc->lock, flags);
|+ local_irq_save(flags);
|+ if (!read_trylock(&gpc->lock)) {
…
|+ if (in_interrupt())
|+ goto out;
|+
|+ read_lock(&gpc->lock);
This does not work. If interrupts are disabled (due to local_irq_save())
then read_lock() must not be used. in_interrupt() does not matter.
Side note: Using HRTIMER_MODE_ABS would avoid the splat at the cost that
on PREEMPT_RT the timer will be invoked in softirq context (as with
HRTIMER_MODE_ABS_SOFT on !PREEMPT_RT). There is no changed behaviour on
!PREEMPT_RT.
Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-26 15:03 ` Sebastian Andrzej Siewior
@ 2024-11-26 16:26 ` David Woodhouse
0 siblings, 0 replies; 12+ messages in thread
From: David Woodhouse @ 2024-11-26 16:26 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: syzbot, boqun.feng, bp, dave.hansen, hdanton, hpa, kvm,
linux-kernel, longman, mingo, paul, pbonzini, seanjc,
syzkaller-bugs, tglx, x86
[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]
On Tue, 2024-11-26 at 16:03 +0100, Sebastian Andrzej Siewior wrote:
> On 2024-11-26 14:49:40 [+0000], David Woodhouse wrote:
> > On Tue, 2024-11-26 at 06:24 -0800, syzbot wrote:
> > > syzbot has bisected this issue to:
> > >
> > > commit 560af5dc839eef08a273908f390cfefefb82aa04
> > > Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> > > Date: Wed Oct 9 15:45:03 2024 +0000
> > >
> > > lockdep: Enable PROVE_RAW_LOCK_NESTING with PROVE_LOCKING.
> >
> > That's not it; this has always been broken with PREEMPT_RT I think.
> > There was an attempt to fix it in
> > https://lore.kernel.org/all/20240227115648.3104-8-dwmw2@infradead.org/
> >
> > I'll dust that off and try again.
>
> Oh thank you. The timer has been made to always expire in hardirq due to
> HRTIMER_MODE_ABS_HARD, this is why you see the splat. If the hardirq
> invocation is needed/ possible then the callback needs to be updated.
>
> The linked patch has this hunk:
> > - read_lock_irqsave(&gpc->lock, flags);
> > + local_irq_save(flags);
> > + if (!read_trylock(&gpc->lock)) {
> …
> > + if (in_interrupt())
> > + goto out;
> > +
> > + read_lock(&gpc->lock);
>
> This does not work. If interrupts are disabled (due to local_irq_save())
> then read_lock() must not be used. in_interrupt() does not matter.
Right. At the end of that discussion, I think I concluded that if we
make it use read_trylock() and fall back to the slow path, then it
doesn't actually need to disable interrupts at all anyway.
> Side note: Using HRTIMER_MODE_ABS would avoid the splat at the cost that
> on PREEMPT_RT the timer will be invoked in softirq context (as with
> HRTIMER_MODE_ABS_SOFT on !PREEMPT_RT). There is no changed behaviour on
> !PREEMPT_RT.
Ah, shiny. If that *only* pushes it to softirq context for PREEMPT_RT
and leaves it in hardirq context for everything else, I think that's a
good choice.
I'll have a quick look at eliminating the _irqsave completely though,
as it may be beenficial.
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Forwarded:
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
2024-11-23 13:17 ` syzbot
2024-11-26 14:24 ` syzbot
@ 2026-03-15 13:58 ` syzbot
2026-03-15 15:00 ` Forwarded: (No Subject) syzbot
` (3 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: syzbot @ 2026-03-15 13:58 UTC (permalink / raw)
To: linux-kernel, syzkaller-bugs
For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com.
***
Subject:
Author: klnm1908v@gmail.com
#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
master
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 91fd3673c09a..e588a188f50a 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -126,23 +126,10 @@ static enum hrtimer_restart
xen_timer_callback(struct hrtimer *timer)
{
struct kvm_vcpu *vcpu = container_of(timer, struct kvm_vcpu,
arch.xen.timer);
- struct kvm_xen_evtchn e;
- int rc;
if (atomic_read(&vcpu->arch.xen.timer_pending))
return HRTIMER_NORESTART;
- e.vcpu_id = vcpu->vcpu_id;
- e.vcpu_idx = vcpu->vcpu_idx;
- e.port = vcpu->arch.xen.timer_virq;
- e.priority = KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL;
-
- rc = kvm_xen_set_evtchn_fast(&e, vcpu->kvm);
- if (rc != -EWOULDBLOCK) {
- vcpu->arch.xen.timer_expires = 0;
- return HRTIMER_NORESTART;
- }
-
atomic_inc(&vcpu->arch.xen.timer_pending);
kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
kvm_vcpu_kick(vcpu);
^ permalink raw reply related [flat|nested] 12+ messages in thread* Forwarded: (No Subject)
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
` (2 preceding siblings ...)
2026-03-15 13:58 ` Forwarded: syzbot
@ 2026-03-15 15:00 ` syzbot
2026-03-15 15:07 ` syzbot
` (2 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: syzbot @ 2026-03-15 15:00 UTC (permalink / raw)
To: linux-kernel, syzkaller-bugs
For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com.
***
Subject: (No Subject)
Author: zxcyui967@proton.me
#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 91fd3673c09a..e588a188f50a 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -126,23 +126,10 @@ static enum hrtimer_restart xen_timer_callback(struct hrtimer *timer)
{
struct kvm_vcpu *vcpu = container_of(timer, struct kvm_vcpu,
arch.xen.timer);
- struct kvm_xen_evtchn e;
- int rc;
if (atomic_read(&vcpu->arch.xen.timer_pending))
return HRTIMER_NORESTART;
- e.vcpu_id = vcpu->vcpu_id;
- e.vcpu_idx = vcpu->vcpu_idx;
- e.port = vcpu->arch.xen.timer_virq;
- e.priority = KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL;
-
- rc = kvm_xen_set_evtchn_fast(&e, vcpu->kvm);
- if (rc != -EWOULDBLOCK) {
- vcpu->arch.xen.timer_expires = 0;
- return HRTIMER_NORESTART;
- }
-
atomic_inc(&vcpu->arch.xen.timer_pending);
kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
kvm_vcpu_kick(vcpu);
-- 2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* Forwarded: (No Subject)
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
` (3 preceding siblings ...)
2026-03-15 15:00 ` Forwarded: (No Subject) syzbot
@ 2026-03-15 15:07 ` syzbot
2026-08-09 9:42 ` Forwarded: Re: [PATCH] KVM: x86/xen: Use read_trylock() for GPC locks in hardirq paths syzbot
2026-08-10 11:28 ` Forwarded: Re: [syzbot] [kvm-x86?] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
6 siblings, 0 replies; 12+ messages in thread
From: syzbot @ 2026-03-15 15:07 UTC (permalink / raw)
To: linux-kernel, syzkaller-bugs
For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com.
***
Subject: (No Subject)
Author: zxcyui967@proton.me
#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
From kettlebellok Mon Sep 17 00:00:00 2001
From: kettlebellok <zxcyui967@proton.me>
Date: Wed, 12 Mar 2026 00:00:00 +0800
Subject: [PATCH] KVM: xen: fix sleeping lock in hardirq context in
xen_timer_callback()
xen_timer_callback() calls kvm_xen_set_evtchn_fast(), which acquires
gpc->lock via read_lock_irqsave(). This is a regular rwlock, which
becomes a sleeping lock on PREEMPT_RT kernels. Since xen_timer_callback()
runs in hardirq context (hrtimer), this is invalid and triggers:
BUG: Invalid wait context
kvm_xen_set_evtchn_fast
xen_timer_callback
__hrtimer_run_queues
hrtimer_interrupt
Fix this by removing the kvm_xen_set_evtchn_fast() call from
xen_timer_callback() and always deferring event delivery via the
existing timer_pending mechanism. The vCPU will then deliver the
event through kvm_xen_inject_timer_irqs() in a safe process context.
This was already the fallback path when kvm_xen_set_evtchn_fast()
returned -EWOULDBLOCK.
Reported-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=919877893c9d28162dc2
Signed-off-by: kettlebellok <zxcyui967@proton.me>
---
arch/x86/kvm/xen.c | 13 -------------
1 file changed, 13 deletions(-)
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 91fd3673c09a..e588a188f50a 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -126,23 +126,10 @@ static enum hrtimer_restart xen_timer_callback(struct hrtimer *timer)
{
struct kvm_vcpu *vcpu = container_of(timer, struct kvm_vcpu,
arch.xen.timer);
- struct kvm_xen_evtchn e;
- int rc;
if (atomic_read(&vcpu->arch.xen.timer_pending))
return HRTIMER_NORESTART;
- e.vcpu_id = vcpu->vcpu_id;
- e.vcpu_idx = vcpu->vcpu_idx;
- e.port = vcpu->arch.xen.timer_virq;
- e.priority = KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL;
-
- rc = kvm_xen_set_evtchn_fast(&e, vcpu->kvm);
- if (rc != -EWOULDBLOCK) {
- vcpu->arch.xen.timer_expires = 0;
- return HRTIMER_NORESTART;
- }
-
atomic_inc(&vcpu->arch.xen.timer_pending);
kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
kvm_vcpu_kick(vcpu);
-- 2.43.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* Forwarded: Re: [PATCH] KVM: x86/xen: Use read_trylock() for GPC locks in hardirq paths
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
` (4 preceding siblings ...)
2026-03-15 15:07 ` syzbot
@ 2026-08-09 9:42 ` syzbot
2026-08-10 11:28 ` Forwarded: Re: [syzbot] [kvm-x86?] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
6 siblings, 0 replies; 12+ messages in thread
From: syzbot @ 2026-08-09 9:42 UTC (permalink / raw)
To: linux-kernel, syzkaller-bugs
For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com.
***
Subject: Re: [PATCH] KVM: x86/xen: Use read_trylock() for GPC locks in hardirq paths
Author: dwmw2@infradead.org
On Fri, 2026-08-07 at 17:49 +0530, sai madhu wrote:
> kvm_xen_set_evtchn_fast() is called from hardirq context (xen timer
> callback, kvm_arch_set_irq_inatomic()). On PREEMPT_RT (and under
> lockdep), read_lock_irqsave(&gpc->lock) is invalid there because
> rwlock_t may sleep.
>
> Switch to read_trylock() and return -EWOULDBLOCK when the lock is
> contended or the cache is invalid. xen_timer_callback() already
> defers to the slow path (timer_pending + KVM_REQ_UNBLOCK +
> kvm_xen_inject_timer_irqs()).
>
> Also fix __kvm_xen_has_interrupt() the same way for atomic callers.
>
> Based on David Woodhouse's upstream series (May 2026, patch 3/7).
> Fixes syzbot: https://syzkaller.appspot.com/bug?extid=919877893c9d28162dc2
Thanks... but why? We're literally iterating on this right now.
I'm fairly convinced I want to ditch rwlocks completely and move to RCU:
https://lore.kernel.org/all/20260805195528.3853473-1-dwmw@amazon.co.uk/
Testing found some issues in my conversion which I've since fixed, and
the conversion to SRCU at Sean's request is currently an *incremental*
patch, which I'll eventually squash back into the first.
I'm in the middle of running a weekend-long soak test on this tree:
https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/xen-rcu-fixed
Let's see what syzbot thinks of it...
#syz test: git://git.infradead.org/users/dwmw2/linux.git xen-rcu-fixed
^ permalink raw reply [flat|nested] 12+ messages in thread* Forwarded: Re: [syzbot] [kvm-x86?] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
2024-11-21 15:03 [syzbot] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast syzbot
` (5 preceding siblings ...)
2026-08-09 9:42 ` Forwarded: Re: [PATCH] KVM: x86/xen: Use read_trylock() for GPC locks in hardirq paths syzbot
@ 2026-08-10 11:28 ` syzbot
6 siblings, 0 replies; 12+ messages in thread
From: syzbot @ 2026-08-10 11:28 UTC (permalink / raw)
To: linux-kernel, syzkaller-bugs
For archival purposes, forwarding an incoming command email to
linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com.
***
Subject: Re: [syzbot] [kvm-x86?] [kvm?] WARNING: locking bug in kvm_xen_set_evtchn_fast
Author: dwmw2@infradead.org
On Sun, 2026-08-09 at 04:34 +0000, syzbot wrote:
> syzbot has tested the proposed patch and the reproducer did not trigger any issue
I found some though. Reworked:
#syz test: git://git.infradead.org/users/dwmw2/linux.git xen-srcu-fixed
^ permalink raw reply [flat|nested] 12+ messages in thread