* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw
[not found] ` <5237824E.5090802@windriver.com>
@ 2013-09-20 17:51 ` Paul Gortmaker
2013-09-20 18:04 ` Jan Kiszka
0 siblings, 1 reply; 5+ messages in thread
From: Paul Gortmaker @ 2013-09-20 17:51 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: linux-kernel, kvm, gleb, jan.kiszka, linux-rt-users
[Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote:
> On 13-09-16 10:06 AM, Paolo Bonzini wrote:
> > Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the
> > mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a
> > patch that shrunk the kvm_lock critical section so that the mmu_lock
> > critical section does not nest with it, but in the end there is no reason
> > for the vm_list to be protected by a raw spinlock. Only manipulations
> > of kvm_usage_count and the consequent hardware_enable/disable operations
> > are not preemptable.
> >
> > This small series thus splits the kvm_lock in the "raw" part and the
> > "non-raw" part.
> >
> > Paul, could you please provide your Tested-by?
>
> Sure, I'll go back and see if I can find what triggered it in the
> original report, and give the patches a spin on 3.4.x-rt (and probably
> 3.10.x-rt, since that is where rt-current is presently).
Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other
issues, probably not explicitly related to this patchset (see below).
Paul.
--
e1000e 0000:00:19.0 eth1: removed PHC
assign device 0:0:19.0
pci 0000:00:19.0: irq 43 for MSI/MSI-X
pci 0000:00:19.0: irq 43 for MSI/MSI-X
pci 0000:00:19.0: irq 43 for MSI/MSI-X
pci 0000:00:19.0: irq 43 for MSI/MSI-X
BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659
in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0
2 locks held by swapper/0/0:
#0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
#1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
irq event stamp: 6121390
hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
softirqs last enabled at (0): [< (null)>] (null)
softirqs last disabled at (0): [< (null)>] (null)
Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18
ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001
ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110
Call Trace:
<IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
[<ffffffff810c90d3>] __might_sleep+0x153/0x250
[<ffffffff819f89a4>] rt_spin_lock+0x24/0x60
[<ffffffff810ccdd6>] __wake_up+0x36/0x70
[<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
[<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
[<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
[<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
[<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
[<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
[<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
[<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
[<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff8113cec8>] handle_irq_event+0x48/0x70
[<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
[<ffffffff8104c6ae>] handle_irq+0x1e/0x30
[<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
[<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
<EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
[<ffffffff819db767>] rest_init+0x137/0x140
[<ffffffff819db635>] ? rest_init+0x5/0x140
[<ffffffff822fde18>] start_kernel+0x3af/0x3bc
[<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
[<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
[<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
=================================
[ INFO: inconsistent lock state ]
3.10.10-rt7 #2 Not tainted
---------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
(&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
{HARDIRQ-ON-W} state was registered at:
[<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0
[<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
[<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
[<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
[<ffffffff810ccdd6>] __wake_up+0x36/0x70
[<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390
[<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0
[<ffffffff81092db8>] run_ksoftirqd+0x38/0x60
[<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340
[<ffffffff810bbf4d>] kthread+0xcd/0xe0
[<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0
irq event stamp: 6121390
hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
softirqs last enabled at (0): [< (null)>] (null)
softirqs last disabled at (0): [< (null)>] (null)
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&(&q->lock)->lock)->wait_lock);
<Interrupt>
lock(&(&(&q->lock)->lock)->wait_lock);
*** DEADLOCK ***
2 locks held by swapper/0/0:
#0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
#1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
stack backtrace:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90
ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001
0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000
Call Trace:
<IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
[<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a
[<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150
[<ffffffff810f9dab>] mark_lock+0x28b/0x6a0
[<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0
[<ffffffff811091f2>] ? __module_text_address+0x12/0x60
[<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60
[<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80
[<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0
[<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
[<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
[<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
[<ffffffff810ccdd6>] __wake_up+0x36/0x70
[<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
[<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
[<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
[<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
[<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
[<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
[<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
[<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
[<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff8113cec8>] handle_irq_event+0x48/0x70
[<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
[<ffffffff8104c6ae>] handle_irq+0x1e/0x30
[<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
[<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
<EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
[<ffffffff819db767>] rest_init+0x137/0x140
[<ffffffff819db635>] ? rest_init+0x5/0x140
[<ffffffff822fde18>] start_kernel+0x3af/0x3bc
[<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
[<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
[<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
>
> Paul.
> --
>
> >
> > Thanks,
> >
> > Paolo
> >
> > Paolo Bonzini (3):
> > KVM: cleanup (physical) CPU hotplug
> > KVM: protect kvm_usage_count with its own spinlock
> > KVM: Convert kvm_lock back to non-raw spinlock
> >
> > Documentation/virtual/kvm/locking.txt | 8 ++++--
> > arch/x86/kvm/mmu.c | 4 +--
> > arch/x86/kvm/x86.c | 8 +++---
> > include/linux/kvm_host.h | 2 +-
> > virt/kvm/kvm_main.c | 51 ++++++++++++++++++-----------------
> > 5 files changed, 40 insertions(+), 33 deletions(-)
> >
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw
2013-09-20 17:51 ` [PATCH 0/3] KVM: Make kvm_lock non-raw Paul Gortmaker
@ 2013-09-20 18:04 ` Jan Kiszka
2013-09-20 18:18 ` Paul Gortmaker
2013-09-21 20:26 ` Michael S. Tsirkin
0 siblings, 2 replies; 5+ messages in thread
From: Jan Kiszka @ 2013-09-20 18:04 UTC (permalink / raw)
To: Paul Gortmaker
Cc: Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users,
Alex Williamson, Michael S. Tsirkin
On 2013-09-20 19:51, Paul Gortmaker wrote:
> [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote:
>
>> On 13-09-16 10:06 AM, Paolo Bonzini wrote:
>>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the
>>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a
>>> patch that shrunk the kvm_lock critical section so that the mmu_lock
>>> critical section does not nest with it, but in the end there is no reason
>>> for the vm_list to be protected by a raw spinlock. Only manipulations
>>> of kvm_usage_count and the consequent hardware_enable/disable operations
>>> are not preemptable.
>>>
>>> This small series thus splits the kvm_lock in the "raw" part and the
>>> "non-raw" part.
>>>
>>> Paul, could you please provide your Tested-by?
>>
>> Sure, I'll go back and see if I can find what triggered it in the
>> original report, and give the patches a spin on 3.4.x-rt (and probably
>> 3.10.x-rt, since that is where rt-current is presently).
>
> Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other
> issues, probably not explicitly related to this patchset (see below).
>
> Paul.
> --
>
> e1000e 0000:00:19.0 eth1: removed PHC
> assign device 0:0:19.0
> pci 0000:00:19.0: irq 43 for MSI/MSI-X
> pci 0000:00:19.0: irq 43 for MSI/MSI-X
> pci 0000:00:19.0: irq 43 for MSI/MSI-X
> pci 0000:00:19.0: irq 43 for MSI/MSI-X
> BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659
> in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0
> 2 locks held by swapper/0/0:
> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> irq event stamp: 6121390
> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
> softirqs last enabled at (0): [< (null)>] (null)
> softirqs last disabled at (0): [< (null)>] (null)
> Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430
>
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
> ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18
> ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001
> ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110
> Call Trace:
> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
> [<ffffffff810c90d3>] __might_sleep+0x153/0x250
> [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60
> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
-rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from
assigned devices directly from the host IRQ handler. We need to disable
this fast-path in -rt or introduce such an abstraction (I did this once
over 2.6.33-rt).
IIRC, VFIO goes the slower patch via a kernel thread unconditionally,
thus cannot trigger this. Only legacy device assignment is affected.
Jan
> [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
> [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
> [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
> [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
> [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
> [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
> [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> [<ffffffff8113cec8>] handle_irq_event+0x48/0x70
> [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
> [<ffffffff8104c6ae>] handle_irq+0x1e/0x30
> [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
> [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
> <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
> [<ffffffff819db767>] rest_init+0x137/0x140
> [<ffffffff819db635>] ? rest_init+0x5/0x140
> [<ffffffff822fde18>] start_kernel+0x3af/0x3bc
> [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
> [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
> [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
>
> =================================
> [ INFO: inconsistent lock state ]
> 3.10.10-rt7 #2 Not tainted
> ---------------------------------
> inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
> swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
> (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
> {HARDIRQ-ON-W} state was registered at:
> [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0
> [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
> [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
> [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
> [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
> [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390
> [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0
> [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60
> [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340
> [<ffffffff810bbf4d>] kthread+0xcd/0xe0
> [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0
> irq event stamp: 6121390
> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
> softirqs last enabled at (0): [< (null)>] (null)
> softirqs last disabled at (0): [< (null)>] (null)
>
> other info that might help us debug this:
> Possible unsafe locking scenario:
>
> CPU0
> ----
> lock(&(&(&q->lock)->lock)->wait_lock);
> <Interrupt>
> lock(&(&(&q->lock)->lock)->wait_lock);
>
> *** DEADLOCK ***
>
> 2 locks held by swapper/0/0:
> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
>
> stack backtrace:
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
> ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90
> ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001
> 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000
> Call Trace:
> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
> [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a
> [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150
> [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0
> [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0
> [<ffffffff811091f2>] ? __module_text_address+0x12/0x60
> [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60
> [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80
> [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0
> [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
> [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
> [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
> [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
> [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
> [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
> [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
> [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
> [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
> [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
> [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
> [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
> [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> [<ffffffff8113cec8>] handle_irq_event+0x48/0x70
> [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
> [<ffffffff8104c6ae>] handle_irq+0x1e/0x30
> [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
> [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
> <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
> [<ffffffff819db767>] rest_init+0x137/0x140
> [<ffffffff819db635>] ? rest_init+0x5/0x140
> [<ffffffff822fde18>] start_kernel+0x3af/0x3bc
> [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
> [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
> [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw
2013-09-20 18:04 ` Jan Kiszka
@ 2013-09-20 18:18 ` Paul Gortmaker
2013-09-20 18:27 ` Jan Kiszka
2013-09-21 20:26 ` Michael S. Tsirkin
1 sibling, 1 reply; 5+ messages in thread
From: Paul Gortmaker @ 2013-09-20 18:18 UTC (permalink / raw)
To: Jan Kiszka
Cc: Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users,
Alex Williamson, Michael S. Tsirkin
On 13-09-20 02:04 PM, Jan Kiszka wrote:
> On 2013-09-20 19:51, Paul Gortmaker wrote:
>> [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote:
>>
>>> On 13-09-16 10:06 AM, Paolo Bonzini wrote:
>>>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the
>>>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a
>>>> patch that shrunk the kvm_lock critical section so that the mmu_lock
>>>> critical section does not nest with it, but in the end there is no reason
>>>> for the vm_list to be protected by a raw spinlock. Only manipulations
>>>> of kvm_usage_count and the consequent hardware_enable/disable operations
>>>> are not preemptable.
>>>>
>>>> This small series thus splits the kvm_lock in the "raw" part and the
>>>> "non-raw" part.
>>>>
>>>> Paul, could you please provide your Tested-by?
>>>
>>> Sure, I'll go back and see if I can find what triggered it in the
>>> original report, and give the patches a spin on 3.4.x-rt (and probably
>>> 3.10.x-rt, since that is where rt-current is presently).
>>
>> Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other
>> issues, probably not explicitly related to this patchset (see below).
>>
>> Paul.
>> --
>>
>> e1000e 0000:00:19.0 eth1: removed PHC
>> assign device 0:0:19.0
>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>> BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659
>> in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0
>> 2 locks held by swapper/0/0:
>> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
>> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
>> irq event stamp: 6121390
>> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
>> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
>> softirqs last enabled at (0): [< (null)>] (null)
>> softirqs last disabled at (0): [< (null)>] (null)
>> Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430
>>
>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
>> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
>> ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18
>> ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001
>> ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110
>> Call Trace:
>> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
>> [<ffffffff810c90d3>] __might_sleep+0x153/0x250
>> [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60
>> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
>> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
>
> -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from
> assigned devices directly from the host IRQ handler. We need to disable
> this fast-path in -rt or introduce such an abstraction (I did this once
> over 2.6.33-rt).
Ah, right -- the simple wait queue support (currently -rt specific)
would have to be used here. It is on the todo list to get that moved
from -rt into mainline.
Paul.
--
>
> IIRC, VFIO goes the slower patch via a kernel thread unconditionally,
> thus cannot trigger this. Only legacy device assignment is affected.
>
> Jan
>
>> [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
>> [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
>> [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
>> [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
>> [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
>> [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
>> [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
>> [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
>> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
>> [<ffffffff8113cec8>] handle_irq_event+0x48/0x70
>> [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
>> [<ffffffff8104c6ae>] handle_irq+0x1e/0x30
>> [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
>> [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
>> <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
>> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
>> [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
>> [<ffffffff819db767>] rest_init+0x137/0x140
>> [<ffffffff819db635>] ? rest_init+0x5/0x140
>> [<ffffffff822fde18>] start_kernel+0x3af/0x3bc
>> [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
>> [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
>> [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
>>
>> =================================
>> [ INFO: inconsistent lock state ]
>> 3.10.10-rt7 #2 Not tainted
>> ---------------------------------
>> inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
>> swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
>> (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
>> {HARDIRQ-ON-W} state was registered at:
>> [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0
>> [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
>> [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
>> [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
>> [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
>> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
>> [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390
>> [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0
>> [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60
>> [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340
>> [<ffffffff810bbf4d>] kthread+0xcd/0xe0
>> [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0
>> irq event stamp: 6121390
>> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
>> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
>> softirqs last enabled at (0): [< (null)>] (null)
>> softirqs last disabled at (0): [< (null)>] (null)
>>
>> other info that might help us debug this:
>> Possible unsafe locking scenario:
>>
>> CPU0
>> ----
>> lock(&(&(&q->lock)->lock)->wait_lock);
>> <Interrupt>
>> lock(&(&(&q->lock)->lock)->wait_lock);
>>
>> *** DEADLOCK ***
>>
>> 2 locks held by swapper/0/0:
>> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
>> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
>>
>> stack backtrace:
>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
>> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
>> ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90
>> ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001
>> 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000
>> Call Trace:
>> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
>> [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a
>> [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150
>> [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0
>> [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0
>> [<ffffffff811091f2>] ? __module_text_address+0x12/0x60
>> [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60
>> [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80
>> [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0
>> [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
>> [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
>> [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
>> [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
>> [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
>> [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
>> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
>> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
>> [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
>> [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
>> [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
>> [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
>> [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
>> [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
>> [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
>> [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
>> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
>> [<ffffffff8113cec8>] handle_irq_event+0x48/0x70
>> [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
>> [<ffffffff8104c6ae>] handle_irq+0x1e/0x30
>> [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
>> [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
>> <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
>> [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
>> [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
>> [<ffffffff819db767>] rest_init+0x137/0x140
>> [<ffffffff819db635>] ? rest_init+0x5/0x140
>> [<ffffffff822fde18>] start_kernel+0x3af/0x3bc
>> [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
>> [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
>> [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw
2013-09-20 18:18 ` Paul Gortmaker
@ 2013-09-20 18:27 ` Jan Kiszka
0 siblings, 0 replies; 5+ messages in thread
From: Jan Kiszka @ 2013-09-20 18:27 UTC (permalink / raw)
To: Paul Gortmaker
Cc: Paolo Bonzini, linux-kernel, kvm, gleb, linux-rt-users,
Alex Williamson, Michael S. Tsirkin
On 2013-09-20 20:18, Paul Gortmaker wrote:
> On 13-09-20 02:04 PM, Jan Kiszka wrote:
>> On 2013-09-20 19:51, Paul Gortmaker wrote:
>>> [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote:
>>>
>>>> On 13-09-16 10:06 AM, Paolo Bonzini wrote:
>>>>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the
>>>>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a
>>>>> patch that shrunk the kvm_lock critical section so that the mmu_lock
>>>>> critical section does not nest with it, but in the end there is no reason
>>>>> for the vm_list to be protected by a raw spinlock. Only manipulations
>>>>> of kvm_usage_count and the consequent hardware_enable/disable operations
>>>>> are not preemptable.
>>>>>
>>>>> This small series thus splits the kvm_lock in the "raw" part and the
>>>>> "non-raw" part.
>>>>>
>>>>> Paul, could you please provide your Tested-by?
>>>>
>>>> Sure, I'll go back and see if I can find what triggered it in the
>>>> original report, and give the patches a spin on 3.4.x-rt (and probably
>>>> 3.10.x-rt, since that is where rt-current is presently).
>>>
>>> Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other
>>> issues, probably not explicitly related to this patchset (see below).
>>>
>>> Paul.
>>> --
>>>
>>> e1000e 0000:00:19.0 eth1: removed PHC
>>> assign device 0:0:19.0
>>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>>> pci 0000:00:19.0: irq 43 for MSI/MSI-X
>>> BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659
>>> in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0
>>> 2 locks held by swapper/0/0:
>>> #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
>>> #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
>>> irq event stamp: 6121390
>>> hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
>>> hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
>>> softirqs last enabled at (0): [< (null)>] (null)
>>> softirqs last disabled at (0): [< (null)>] (null)
>>> Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430
>>>
>>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
>>> Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
>>> ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18
>>> ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001
>>> ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110
>>> Call Trace:
>>> <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
>>> [<ffffffff810c90d3>] __might_sleep+0x153/0x250
>>> [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60
>>> [<ffffffff810ccdd6>] __wake_up+0x36/0x70
>>> [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
>>
>> -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from
>> assigned devices directly from the host IRQ handler. We need to disable
>> this fast-path in -rt or introduce such an abstraction (I did this once
>> over 2.6.33-rt).
>
> Ah, right -- the simple wait queue support (currently -rt specific)
> would have to be used here. It is on the todo list to get that moved
> from -rt into mainline.
Oh, it's there in -rt already - perfect! If there is a good reason for
upstream, kvm can switch of course.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] KVM: Make kvm_lock non-raw
2013-09-20 18:04 ` Jan Kiszka
2013-09-20 18:18 ` Paul Gortmaker
@ 2013-09-21 20:26 ` Michael S. Tsirkin
1 sibling, 0 replies; 5+ messages in thread
From: Michael S. Tsirkin @ 2013-09-21 20:26 UTC (permalink / raw)
To: Jan Kiszka
Cc: Paul Gortmaker, Paolo Bonzini, linux-kernel, kvm, gleb,
linux-rt-users, Alex Williamson
On Fri, Sep 20, 2013 at 08:04:19PM +0200, Jan Kiszka wrote:
> On 2013-09-20 19:51, Paul Gortmaker wrote:
> > [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote:
> >
> >> On 13-09-16 10:06 AM, Paolo Bonzini wrote:
> >>> Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the
> >>> mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a
> >>> patch that shrunk the kvm_lock critical section so that the mmu_lock
> >>> critical section does not nest with it, but in the end there is no reason
> >>> for the vm_list to be protected by a raw spinlock. Only manipulations
> >>> of kvm_usage_count and the consequent hardware_enable/disable operations
> >>> are not preemptable.
> >>>
> >>> This small series thus splits the kvm_lock in the "raw" part and the
> >>> "non-raw" part.
> >>>
> >>> Paul, could you please provide your Tested-by?
> >>
> >> Sure, I'll go back and see if I can find what triggered it in the
> >> original report, and give the patches a spin on 3.4.x-rt (and probably
> >> 3.10.x-rt, since that is where rt-current is presently).
> >
> > Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other
> > issues, probably not explicitly related to this patchset (see below).
> >
> > Paul.
> > --
> >
> > e1000e 0000:00:19.0 eth1: removed PHC
> > assign device 0:0:19.0
> > pci 0000:00:19.0: irq 43 for MSI/MSI-X
> > pci 0000:00:19.0: irq 43 for MSI/MSI-X
> > pci 0000:00:19.0: irq 43 for MSI/MSI-X
> > pci 0000:00:19.0: irq 43 for MSI/MSI-X
> > BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659
> > in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0
> > 2 locks held by swapper/0/0:
> > #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
> > #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> > irq event stamp: 6121390
> > hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
> > hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
> > softirqs last enabled at (0): [< (null)>] (null)
> > softirqs last disabled at (0): [< (null)>] (null)
> > Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430
> >
> > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
> > Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
> > ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18
> > ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001
> > ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110
> > Call Trace:
> > <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
> > [<ffffffff810c90d3>] __might_sleep+0x153/0x250
> > [<ffffffff819f89a4>] rt_spin_lock+0x24/0x60
> > [<ffffffff810ccdd6>] __wake_up+0x36/0x70
> > [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
>
> -rt lacks an atomic waitqueue for triggering VCPU wakeups on MSIs from
> assigned devices directly from the host IRQ handler. We need to disable
> this fast-path in -rt or introduce such an abstraction (I did this once
> over 2.6.33-rt).
>
> IIRC, VFIO goes the slower patch via a kernel thread unconditionally,
> thus cannot trigger this.
AFAIK VFIO just uses eventfds and these can
inject MSI interrupts directly from IRQ without going through a thread.
> Only legacy device assignment is affected.
>
> Jan
>
> > [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
> > [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
> > [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
> > [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> > [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
> > [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
> > [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
> > [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
> > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> > [<ffffffff8113cec8>] handle_irq_event+0x48/0x70
> > [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
> > [<ffffffff8104c6ae>] handle_irq+0x1e/0x30
> > [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
> > [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
> > <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
> > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> > [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
> > [<ffffffff819db767>] rest_init+0x137/0x140
> > [<ffffffff819db635>] ? rest_init+0x5/0x140
> > [<ffffffff822fde18>] start_kernel+0x3af/0x3bc
> > [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
> > [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
> > [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
> >
> > =================================
> > [ INFO: inconsistent lock state ]
> > 3.10.10-rt7 #2 Not tainted
> > ---------------------------------
> > inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
> > swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
> > (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
> > {HARDIRQ-ON-W} state was registered at:
> > [<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0
> > [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
> > [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
> > [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
> > [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
> > [<ffffffff810ccdd6>] __wake_up+0x36/0x70
> > [<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390
> > [<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0
> > [<ffffffff81092db8>] run_ksoftirqd+0x38/0x60
> > [<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340
> > [<ffffffff810bbf4d>] kthread+0xcd/0xe0
> > [<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0
> > irq event stamp: 6121390
> > hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
> > hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
> > softirqs last enabled at (0): [< (null)>] (null)
> > softirqs last disabled at (0): [< (null)>] (null)
> >
> > other info that might help us debug this:
> > Possible unsafe locking scenario:
> >
> > CPU0
> > ----
> > lock(&(&(&q->lock)->lock)->wait_lock);
> > <Interrupt>
> > lock(&(&(&q->lock)->lock)->wait_lock);
> >
> > *** DEADLOCK ***
> >
> > 2 locks held by swapper/0/0:
> > #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
> > #1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> >
> > stack backtrace:
> > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
> > Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
> > ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90
> > ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001
> > 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000
> > Call Trace:
> > <IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
> > [<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a
> > [<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150
> > [<ffffffff810f9dab>] mark_lock+0x28b/0x6a0
> > [<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0
> > [<ffffffff811091f2>] ? __module_text_address+0x12/0x60
> > [<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60
> > [<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80
> > [<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0
> > [<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
> > [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
> > [<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
> > [<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
> > [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
> > [<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
> > [<ffffffff810ccdd6>] __wake_up+0x36/0x70
> > [<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
> > [<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
> > [<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
> > [<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
> > [<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
> > [<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
> > [<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
> > [<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
> > [<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
> > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> > [<ffffffff8113cec8>] handle_irq_event+0x48/0x70
> > [<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
> > [<ffffffff8104c6ae>] handle_irq+0x1e/0x30
> > [<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
> > [<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
> > <EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
> > [<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
> > [<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
> > [<ffffffff819db767>] rest_init+0x137/0x140
> > [<ffffffff819db635>] ? rest_init+0x5/0x140
> > [<ffffffff822fde18>] start_kernel+0x3af/0x3bc
> > [<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
> > [<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
> > [<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-09-21 20:26 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1379340373-5135-1-git-send-email-pbonzini@redhat.com>
[not found] ` <5237824E.5090802@windriver.com>
2013-09-20 17:51 ` [PATCH 0/3] KVM: Make kvm_lock non-raw Paul Gortmaker
2013-09-20 18:04 ` Jan Kiszka
2013-09-20 18:18 ` Paul Gortmaker
2013-09-20 18:27 ` Jan Kiszka
2013-09-21 20:26 ` Michael S. Tsirkin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).