* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw [not found] ` <5237824E.5090802@windriver.com> @ 2013-09-20 17:51 ` Paul Gortmaker 2013-09-20 18:04 ` Jan Kiszka 0 siblings, 1 reply; 5+ messages in thread From: Paul Gortmaker @ 2013-09-20 17:51 UTC (permalink / raw) To: Paolo Bonzini; +Cc: linux-kernel, kvm, gleb, jan.kiszka, linux-rt-users [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote: > On 13-09-16 10:06 AM, Paolo Bonzini wrote: > > Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the > > mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a > > patch that shrunk the kvm_lock critical section so that the mmu_lock > > critical section does not nest with it, but in the end there is no reason > > for the vm_list to be protected by a raw spinlock. Only manipulations > > of kvm_usage_count and the consequent hardware_enable/disable operations > > are not preemptable. > > > > This small series thus splits the kvm_lock in the "raw" part and the > > "non-raw" part. > > > > Paul, could you please provide your Tested-by? > > Sure, I'll go back and see if I can find what triggered it in the > original report, and give the patches a spin on 3.4.x-rt (and probably > 3.10.x-rt, since that is where rt-current is presently). Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other issues, probably not explicitly related to this patchset (see below). Paul. -- e1000e 0000:00:19.0 eth1: removed PHC assign device 0:0:19.0 pci 0000:00:19.0: irq 43 for MSI/MSI-X pci 0000:00:19.0: irq 43 for MSI/MSI-X pci 0000:00:19.0: irq 43 for MSI/MSI-X pci 0000:00:19.0: irq 43 for MSI/MSI-X BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659 in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0 2 locks held by swapper/0/0: #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 irq event stamp: 6121390 hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f softirqs last enabled at (0): [< (null)>] (null) softirqs last disabled at (0): [< (null)>] (null) Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430 CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18 ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001 ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110 Call Trace: <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b [<ffffffff810c90d3>] __might_sleep+0x153/0x250 [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60 [<ffffffff810ccdd6>] __wake_up+0x36/0x70 [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 [<ffffffff819db767>] rest_init+0x137/0x140 [<ffffffff819db635>] ? rest_init+0x5/0x140 [<ffffffff822fde18>] start_kernel+0x3af/0x3bc [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf ================================= [ INFO: inconsistent lock state ] 3.10.10-rt7 #2 Not tainted --------------------------------- inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes: (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 {HARDIRQ-ON-W} state was registered at: [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0 [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 [<ffffffff810ccdd6>] __wake_up+0x36/0x70 [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390 [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0 [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60 [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340 [<ffffffff810bbf4d>] kthread+0xcd/0xe0 [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0 irq event stamp: 6121390 hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f softirqs last enabled at (0): [< (null)>] (null) softirqs last disabled at (0): [< (null)>] (null) other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&(&(&q->lock)->lock)->wait_lock); <Interrupt> lock(&(&(&q->lock)->lock)->wait_lock); *** DEADLOCK *** 2 locks held by swapper/0/0: #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90 ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000 Call Trace: <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150 [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0 [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0 [<ffffffff811091f2>] ? __module_text_address+0x12/0x60 [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60 [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80 [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0 [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 [<ffffffff810ccdd6>] __wake_up+0x36/0x70 [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 [<ffffffff819db767>] rest_init+0x137/0x140 [<ffffffff819db635>] ? rest_init+0x5/0x140 [<ffffffff822fde18>] start_kernel+0x3af/0x3bc [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf > > Paul. > -- > > > > > Thanks, > > > > Paolo > > > > Paolo Bonzini (3): > > KVM: cleanup (physical) CPU hotplug > > KVM: protect kvm_usage_count with its own spinlock > > KVM: Convert kvm_lock back to non-raw spinlock > > > > Documentation/virtual/kvm/locking.txt | 8 ++++-- > > arch/x86/kvm/mmu.c | 4 +-- > > arch/x86/kvm/x86.c | 8 +++--- > > include/linux/kvm_host.h | 2 +- > > virt/kvm/kvm_main.c | 51 ++++++++++++++++++----------------- > > 5 files changed, 40 insertions(+), 33 deletions(-) > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw 2013-09-20 17:51 ` [PATCH 0/3] KVM: Make kvm_lock non-raw Paul Gortmaker @ 2013-09-20 18:04 ` Jan Kiszka 2013-09-20 18:18 ` Paul Gortmaker 2013-09-21 20:26 ` Michael S. Tsirkin 0 siblings, 2 replies; 5+ messages in thread From: Jan Kiszka @ 2013-09-20 18:04 UTC (permalink / raw) To: Paul Gortmaker Cc: Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users, Alex Williamson, Michael S. Tsirkin On 2013-09-20 19:51, Paul Gortmaker wrote: > [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote: > >> On 13-09-16 10:06 AM, Paolo Bonzini wrote: >>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the >>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a >>> patch that shrunk the kvm_lock critical section so that the mmu_lock >>> critical section does not nest with it, but in the end there is no reason >>> for the vm_list to be protected by a raw spinlock. Only manipulations >>> of kvm_usage_count and the consequent hardware_enable/disable operations >>> are not preemptable. >>> >>> This small series thus splits the kvm_lock in the "raw" part and the >>> "non-raw" part. >>> >>> Paul, could you please provide your Tested-by? >> >> Sure, I'll go back and see if I can find what triggered it in the >> original report, and give the patches a spin on 3.4.x-rt (and probably >> 3.10.x-rt, since that is where rt-current is presently). > > Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other > issues, probably not explicitly related to this patchset (see below). > > Paul. > -- > > e1000e 0000:00:19.0 eth1: removed PHC > assign device 0:0:19.0 > pci 0000:00:19.0: irq 43 for MSI/MSI-X > pci 0000:00:19.0: irq 43 for MSI/MSI-X > pci 0000:00:19.0: irq 43 for MSI/MSI-X > pci 0000:00:19.0: irq 43 for MSI/MSI-X > BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659 > in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0 > 2 locks held by swapper/0/0: > #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 > #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > irq event stamp: 6121390 > hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 > hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f > softirqs last enabled at (0): [< (null)>] (null) > softirqs last disabled at (0): [< (null)>] (null) > Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430 > > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 > Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 > ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18 > ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001 > ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110 > Call Trace: > <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b > [<ffffffff810c90d3>] __might_sleep+0x153/0x250 > [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60 > [<ffffffff810ccdd6>] __wake_up+0x36/0x70 > [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from assigned devices directly from the host IRQ handler. We need to disable this fast-path in -rt or introduce such an abstraction (I did this once over 2.6.33-rt). IIRC, VFIO goes the slower patch via a kernel thread unconditionally, thus cannot trigger this. Only legacy device assignment is affected. Jan > [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 > [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 > [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 > [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 > [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 > [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 > [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 > [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 > [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 > [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 > [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f > <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 > [<ffffffff819db767>] rest_init+0x137/0x140 > [<ffffffff819db635>] ? rest_init+0x5/0x140 > [<ffffffff822fde18>] start_kernel+0x3af/0x3bc > [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e > [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c > [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf > > ================================= > [ INFO: inconsistent lock state ] > 3.10.10-rt7 #2 Not tainted > --------------------------------- > inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. > swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes: > (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 > {HARDIRQ-ON-W} state was registered at: > [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0 > [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 > [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 > [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 > [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 > [<ffffffff810ccdd6>] __wake_up+0x36/0x70 > [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390 > [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0 > [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60 > [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340 > [<ffffffff810bbf4d>] kthread+0xcd/0xe0 > [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0 > irq event stamp: 6121390 > hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 > hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f > softirqs last enabled at (0): [< (null)>] (null) > softirqs last disabled at (0): [< (null)>] (null) > > other info that might help us debug this: > Possible unsafe locking scenario: > > CPU0 > ---- > lock(&(&(&q->lock)->lock)->wait_lock); > <Interrupt> > lock(&(&(&q->lock)->lock)->wait_lock); > > *** DEADLOCK *** > > 2 locks held by swapper/0/0: > #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 > #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > > stack backtrace: > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 > Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 > ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90 > ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001 > 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000 > Call Trace: > <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b > [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a > [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150 > [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0 > [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0 > [<ffffffff811091f2>] ? __module_text_address+0x12/0x60 > [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60 > [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80 > [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0 > [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 > [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 > [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 > [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 > [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 > [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 > [<ffffffff810ccdd6>] __wake_up+0x36/0x70 > [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 > [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 > [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 > [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 > [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 > [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 > [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 > [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 > [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 > [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 > [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 > [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f > <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 > [<ffffffff819db767>] rest_init+0x137/0x140 > [<ffffffff819db635>] ? rest_init+0x5/0x140 > [<ffffffff822fde18>] start_kernel+0x3af/0x3bc > [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e > [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c > [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw 2013-09-20 18:04 ` Jan Kiszka @ 2013-09-20 18:18 ` Paul Gortmaker 2013-09-20 18:27 ` Jan Kiszka 2013-09-21 20:26 ` Michael S. Tsirkin 1 sibling, 1 reply; 5+ messages in thread From: Paul Gortmaker @ 2013-09-20 18:18 UTC (permalink / raw) To: Jan Kiszka Cc: Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users, Alex Williamson, Michael S. Tsirkin On 13-09-20 02:04 PM, Jan Kiszka wrote: > On 2013-09-20 19:51, Paul Gortmaker wrote: >> [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote: >> >>> On 13-09-16 10:06 AM, Paolo Bonzini wrote: >>>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the >>>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a >>>> patch that shrunk the kvm_lock critical section so that the mmu_lock >>>> critical section does not nest with it, but in the end there is no reason >>>> for the vm_list to be protected by a raw spinlock. Only manipulations >>>> of kvm_usage_count and the consequent hardware_enable/disable operations >>>> are not preemptable. >>>> >>>> This small series thus splits the kvm_lock in the "raw" part and the >>>> "non-raw" part. >>>> >>>> Paul, could you please provide your Tested-by? >>> >>> Sure, I'll go back and see if I can find what triggered it in the >>> original report, and give the patches a spin on 3.4.x-rt (and probably >>> 3.10.x-rt, since that is where rt-current is presently). >> >> Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other >> issues, probably not explicitly related to this patchset (see below). >> >> Paul. >> -- >> >> e1000e 0000:00:19.0 eth1: removed PHC >> assign device 0:0:19.0 >> pci 0000:00:19.0: irq 43 for MSI/MSI-X >> pci 0000:00:19.0: irq 43 for MSI/MSI-X >> pci 0000:00:19.0: irq 43 for MSI/MSI-X >> pci 0000:00:19.0: irq 43 for MSI/MSI-X >> BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659 >> in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0 >> 2 locks held by swapper/0/0: >> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 >> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 >> irq event stamp: 6121390 >> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 >> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f >> softirqs last enabled at (0): [< (null)>] (null) >> softirqs last disabled at (0): [< (null)>] (null) >> Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430 >> >> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 >> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 >> ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18 >> ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001 >> ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110 >> Call Trace: >> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b >> [<ffffffff810c90d3>] __might_sleep+0x153/0x250 >> [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60 >> [<ffffffff810ccdd6>] __wake_up+0x36/0x70 >> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 > > -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from > assigned devices directly from the host IRQ handler. We need to disable > this fast-path in -rt or introduce such an abstraction (I did this once > over 2.6.33-rt). Ah, right -- the simple wait queue support (currently -rt specific) would have to be used here. It is on the todo list to get that moved from -rt into mainline. Paul. -- > > IIRC, VFIO goes the slower patch via a kernel thread unconditionally, > thus cannot trigger this. Only legacy device assignment is affected. > > Jan > >> [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 >> [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 >> [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 >> [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 >> [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 >> [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 >> [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 >> [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 >> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 >> [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 >> [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 >> [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 >> [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 >> [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f >> <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe >> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 >> [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 >> [<ffffffff819db767>] rest_init+0x137/0x140 >> [<ffffffff819db635>] ? rest_init+0x5/0x140 >> [<ffffffff822fde18>] start_kernel+0x3af/0x3bc >> [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e >> [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c >> [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf >> >> ================================= >> [ INFO: inconsistent lock state ] >> 3.10.10-rt7 #2 Not tainted >> --------------------------------- >> inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. >> swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes: >> (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 >> {HARDIRQ-ON-W} state was registered at: >> [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0 >> [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 >> [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 >> [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 >> [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 >> [<ffffffff810ccdd6>] __wake_up+0x36/0x70 >> [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390 >> [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0 >> [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60 >> [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340 >> [<ffffffff810bbf4d>] kthread+0xcd/0xe0 >> [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0 >> irq event stamp: 6121390 >> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 >> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f >> softirqs last enabled at (0): [< (null)>] (null) >> softirqs last disabled at (0): [< (null)>] (null) >> >> other info that might help us debug this: >> Possible unsafe locking scenario: >> >> CPU0 >> ---- >> lock(&(&(&q->lock)->lock)->wait_lock); >> <Interrupt> >> lock(&(&(&q->lock)->lock)->wait_lock); >> >> *** DEADLOCK *** >> >> 2 locks held by swapper/0/0: >> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 >> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 >> >> stack backtrace: >> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 >> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 >> ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90 >> ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001 >> 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000 >> Call Trace: >> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b >> [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a >> [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150 >> [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0 >> [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0 >> [<ffffffff811091f2>] ? __module_text_address+0x12/0x60 >> [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60 >> [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80 >> [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0 >> [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 >> [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 >> [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 >> [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 >> [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 >> [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 >> [<ffffffff810ccdd6>] __wake_up+0x36/0x70 >> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 >> [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 >> [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 >> [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 >> [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 >> [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 >> [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 >> [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 >> [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 >> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 >> [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 >> [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 >> [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 >> [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 >> [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f >> <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe >> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 >> [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 >> [<ffffffff819db767>] rest_init+0x137/0x140 >> [<ffffffff819db635>] ? rest_init+0x5/0x140 >> [<ffffffff822fde18>] start_kernel+0x3af/0x3bc >> [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e >> [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c >> [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw 2013-09-20 18:18 ` Paul Gortmaker @ 2013-09-20 18:27 ` Jan Kiszka 0 siblings, 0 replies; 5+ messages in thread From: Jan Kiszka @ 2013-09-20 18:27 UTC (permalink / raw) To: Paul Gortmaker Cc: Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users, Alex Williamson, Michael S. Tsirkin On 2013-09-20 20:18, Paul Gortmaker wrote: > On 13-09-20 02:04 PM, Jan Kiszka wrote: >> On 2013-09-20 19:51, Paul Gortmaker wrote: >>> [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote: >>> >>>> On 13-09-16 10:06 AM, Paolo Bonzini wrote: >>>>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the >>>>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a >>>>> patch that shrunk the kvm_lock critical section so that the mmu_lock >>>>> critical section does not nest with it, but in the end there is no reason >>>>> for the vm_list to be protected by a raw spinlock. Only manipulations >>>>> of kvm_usage_count and the consequent hardware_enable/disable operations >>>>> are not preemptable. >>>>> >>>>> This small series thus splits the kvm_lock in the "raw" part and the >>>>> "non-raw" part. >>>>> >>>>> Paul, could you please provide your Tested-by? >>>> >>>> Sure, I'll go back and see if I can find what triggered it in the >>>> original report, and give the patches a spin on 3.4.x-rt (and probably >>>> 3.10.x-rt, since that is where rt-current is presently). >>> >>> Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other >>> issues, probably not explicitly related to this patchset (see below). >>> >>> Paul. >>> -- >>> >>> e1000e 0000:00:19.0 eth1: removed PHC >>> assign device 0:0:19.0 >>> pci 0000:00:19.0: irq 43 for MSI/MSI-X >>> pci 0000:00:19.0: irq 43 for MSI/MSI-X >>> pci 0000:00:19.0: irq 43 for MSI/MSI-X >>> pci 0000:00:19.0: irq 43 for MSI/MSI-X >>> BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659 >>> in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0 >>> 2 locks held by swapper/0/0: >>> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 >>> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 >>> irq event stamp: 6121390 >>> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 >>> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f >>> softirqs last enabled at (0): [< (null)>] (null) >>> softirqs last disabled at (0): [< (null)>] (null) >>> Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430 >>> >>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 >>> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 >>> ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18 >>> ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001 >>> ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110 >>> Call Trace: >>> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b >>> [<ffffffff810c90d3>] __might_sleep+0x153/0x250 >>> [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60 >>> [<ffffffff810ccdd6>] __wake_up+0x36/0x70 >>> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 >> >> -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from >> assigned devices directly from the host IRQ handler. We need to disable >> this fast-path in -rt or introduce such an abstraction (I did this once >> over 2.6.33-rt). > > Ah, right -- the simple wait queue support (currently -rt specific) > would have to be used here. It is on the todo list to get that moved > from -rt into mainline. Oh, it's there in -rt already - perfect! If there is a good reason for upstream, kvm can switch of course. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw 2013-09-20 18:04 ` Jan Kiszka 2013-09-20 18:18 ` Paul Gortmaker @ 2013-09-21 20:26 ` Michael S. Tsirkin 1 sibling, 0 replies; 5+ messages in thread From: Michael S. Tsirkin @ 2013-09-21 20:26 UTC (permalink / raw) To: Jan Kiszka Cc: Paul Gortmaker, Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users, Alex Williamson On Fri, Sep 20, 2013 at 08:04:19PM +0200, Jan Kiszka wrote: > On 2013-09-20 19:51, Paul Gortmaker wrote: > > [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote: > > > >> On 13-09-16 10:06 AM, Paolo Bonzini wrote: > >>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the > >>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a > >>> patch that shrunk the kvm_lock critical section so that the mmu_lock > >>> critical section does not nest with it, but in the end there is no reason > >>> for the vm_list to be protected by a raw spinlock. Only manipulations > >>> of kvm_usage_count and the consequent hardware_enable/disable operations > >>> are not preemptable. > >>> > >>> This small series thus splits the kvm_lock in the "raw" part and the > >>> "non-raw" part. > >>> > >>> Paul, could you please provide your Tested-by? > >> > >> Sure, I'll go back and see if I can find what triggered it in the > >> original report, and give the patches a spin on 3.4.x-rt (and probably > >> 3.10.x-rt, since that is where rt-current is presently). > > > > Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other > > issues, probably not explicitly related to this patchset (see below). > > > > Paul. > > -- > > > > e1000e 0000:00:19.0 eth1: removed PHC > > assign device 0:0:19.0 > > pci 0000:00:19.0: irq 43 for MSI/MSI-X > > pci 0000:00:19.0: irq 43 for MSI/MSI-X > > pci 0000:00:19.0: irq 43 for MSI/MSI-X > > pci 0000:00:19.0: irq 43 for MSI/MSI-X > > BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659 > > in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0 > > 2 locks held by swapper/0/0: > > #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 > > #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > > irq event stamp: 6121390 > > hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 > > hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f > > softirqs last enabled at (0): [< (null)>] (null) > > softirqs last disabled at (0): [< (null)>] (null) > > Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430 > > > > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 > > Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 > > ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18 > > ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001 > > ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110 > > Call Trace: > > <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b > > [<ffffffff810c90d3>] __might_sleep+0x153/0x250 > > [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60 > > [<ffffffff810ccdd6>] __wake_up+0x36/0x70 > > [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 > > -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from > assigned devices directly from the host IRQ handler. We need to disable > this fast-path in -rt or introduce such an abstraction (I did this once > over 2.6.33-rt). > > IIRC, VFIO goes the slower patch via a kernel thread unconditionally, > thus cannot trigger this. AFAIK VFIO just uses eventfds and these can inject MSI interrupts directly from IRQ without going through a thread. > Only legacy device assignment is affected. > > Jan > > > [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 > > [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 > > [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 > > [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > > [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 > > [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 > > [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 > > [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 > > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > > [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 > > [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 > > [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 > > [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 > > [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f > > <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe > > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > > [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 > > [<ffffffff819db767>] rest_init+0x137/0x140 > > [<ffffffff819db635>] ? rest_init+0x5/0x140 > > [<ffffffff822fde18>] start_kernel+0x3af/0x3bc > > [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e > > [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c > > [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf > > > > ================================= > > [ INFO: inconsistent lock state ] > > 3.10.10-rt7 #2 Not tainted > > --------------------------------- > > inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. > > swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes: > > (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 > > {HARDIRQ-ON-W} state was registered at: > > [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0 > > [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 > > [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 > > [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 > > [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 > > [<ffffffff810ccdd6>] __wake_up+0x36/0x70 > > [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390 > > [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0 > > [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60 > > [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340 > > [<ffffffff810bbf4d>] kthread+0xcd/0xe0 > > [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0 > > irq event stamp: 6121390 > > hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30 > > hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f > > softirqs last enabled at (0): [< (null)>] (null) > > softirqs last disabled at (0): [< (null)>] (null) > > > > other info that might help us debug this: > > Possible unsafe locking scenario: > > > > CPU0 > > ---- > > lock(&(&(&q->lock)->lock)->wait_lock); > > <Interrupt> > > lock(&(&(&q->lock)->lock)->wait_lock); > > > > *** DEADLOCK *** > > > > 2 locks held by swapper/0/0: > > #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0 > > #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > > > > stack backtrace: > > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 > > Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 > > ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90 > > ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001 > > 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000 > > Call Trace: > > <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b > > [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a > > [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150 > > [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0 > > [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0 > > [<ffffffff811091f2>] ? __module_text_address+0x12/0x60 > > [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60 > > [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80 > > [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0 > > [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0 > > [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 > > [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80 > > [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370 > > [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370 > > [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60 > > [<ffffffff810ccdd6>] __wake_up+0x36/0x70 > > [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0 > > [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0 > > [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30 > > [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 > > [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 > > [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0 > > [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0 > > [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40 > > [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0 > > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > > [<ffffffff8113cec8>] handle_irq_event+0x48/0x70 > > [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120 > > [<ffffffff8104c6ae>] handle_irq+0x1e/0x30 > > [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0 > > [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f > > <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe > > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430 > > [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430 > > [<ffffffff819db767>] rest_init+0x137/0x140 > > [<ffffffff819db635>] ? rest_init+0x5/0x140 > > [<ffffffff822fde18>] start_kernel+0x3af/0x3bc > > [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e > > [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c > > [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf > > -- > Siemens AG, Corporate Technology, CT RTC ITP SES-DE > Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-09-21 20:26 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1379340373-5135-1-git-send-email-pbonzini@redhat.com>
[not found] ` <5237824E.5090802@windriver.com>
2013-09-20 17:51 ` [PATCH 0/3] KVM: Make kvm_lock non-raw Paul Gortmaker
2013-09-20 18:04 ` Jan Kiszka
2013-09-20 18:18 ` Paul Gortmaker
2013-09-20 18:27 ` Jan Kiszka
2013-09-21 20:26 ` Michael S. Tsirkin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).