From: Paul Gortmaker <paul.gortmaker@windriver.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: <linux-kernel@vger.kernel.org>, <kvm@vger.kernel.org>,
<gleb@redhat.com>, <jan.kiszka@siemens.com>,
<linux-rt-users@vger.kernel.org>
Subject: Re: [PATCH 0/3] KVM: Make kvm_lock non-raw
Date: Fri, 20 Sep 2013 13:51:50 -0400 [thread overview]
Message-ID: <20130920175150.GA24770@windriver.com> (raw)
In-Reply-To: <5237824E.5090802@windriver.com>
[Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote:
> On 13-09-16 10:06 AM, Paolo Bonzini wrote:
> > Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the
> > mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a
> > patch that shrunk the kvm_lock critical section so that the mmu_lock
> > critical section does not nest with it, but in the end there is no reason
> > for the vm_list to be protected by a raw spinlock. Only manipulations
> > of kvm_usage_count and the consequent hardware_enable/disable operations
> > are not preemptable.
> >
> > This small series thus splits the kvm_lock in the "raw" part and the
> > "non-raw" part.
> >
> > Paul, could you please provide your Tested-by?
>
> Sure, I'll go back and see if I can find what triggered it in the
> original report, and give the patches a spin on 3.4.x-rt (and probably
> 3.10.x-rt, since that is where rt-current is presently).
Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other
issues, probably not explicitly related to this patchset (see below).
Paul.
--
e1000e 0000:00:19.0 eth1: removed PHC
assign device 0:0:19.0
pci 0000:00:19.0: irq 43 for MSI/MSI-X
pci 0000:00:19.0: irq 43 for MSI/MSI-X
pci 0000:00:19.0: irq 43 for MSI/MSI-X
pci 0000:00:19.0: irq 43 for MSI/MSI-X
BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659
in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0
2 locks held by swapper/0/0:
#0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
#1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
irq event stamp: 6121390
hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
softirqs last enabled at (0): [< (null)>] (null)
softirqs last disabled at (0): [< (null)>] (null)
Preemption disabled at:[<ffffffff810ebb9a>] cpu_startup_entry+0x1ba/0x430
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18
ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001
ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110
Call Trace:
<IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
[<ffffffff810c90d3>] __might_sleep+0x153/0x250
[<ffffffff819f89a4>] rt_spin_lock+0x24/0x60
[<ffffffff810ccdd6>] __wake_up+0x36/0x70
[<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
[<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
[<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
[<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
[<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
[<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
[<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
[<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
[<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff8113cec8>] handle_irq_event+0x48/0x70
[<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
[<ffffffff8104c6ae>] handle_irq+0x1e/0x30
[<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
[<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
<EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
[<ffffffff819db767>] rest_init+0x137/0x140
[<ffffffff819db635>] ? rest_init+0x5/0x140
[<ffffffff822fde18>] start_kernel+0x3af/0x3bc
[<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
[<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
[<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
=================================
[ INFO: inconsistent lock state ]
3.10.10-rt7 #2 Not tainted
---------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
(&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
{HARDIRQ-ON-W} state was registered at:
[<ffffffff810fc94d>] __lock_acquire+0x69d/0x20e0
[<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
[<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
[<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
[<ffffffff810ccdd6>] __wake_up+0x36/0x70
[<ffffffff8109c5ce>] run_timer_softirq+0x1be/0x390
[<ffffffff81092a09>] do_current_softirqs+0x239/0x5b0
[<ffffffff81092db8>] run_ksoftirqd+0x38/0x60
[<ffffffff810c5d7c>] smpboot_thread_fn+0x22c/0x340
[<ffffffff810bbf4d>] kthread+0xcd/0xe0
[<ffffffff81a019dc>] ret_from_fork+0x7c/0xb0
irq event stamp: 6121390
hardirqs last enabled at (6121389): [<ffffffff819f9ae0>] restore_args+0x0/0x30
hardirqs last disabled at (6121390): [<ffffffff819f9a2a>] common_interrupt+0x6a/0x6f
softirqs last enabled at (0): [< (null)>] (null)
softirqs last disabled at (0): [< (null)>] (null)
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&(&q->lock)->lock)->wait_lock);
<Interrupt>
lock(&(&(&q->lock)->lock)->wait_lock);
*** DEADLOCK ***
2 locks held by swapper/0/0:
#0: (rcu_read_lock){.+.+.+}, at: [<ffffffff8100998a>] kvm_set_irq_inatomic+0x2a/0x4a0
#1: (rcu_read_lock){.+.+.+}, at: [<ffffffff81038800>] kvm_irq_delivery_to_apic_fast+0x60/0x3d0
stack backtrace:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2
Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013
ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90
ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001
0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000
Call Trace:
<IRQ> [<ffffffff819f177d>] dump_stack+0x19/0x1b
[<ffffffff819ec532>] print_usage_bug.part.36+0x28b/0x29a
[<ffffffff810f9040>] ? check_usage_backwards+0x150/0x150
[<ffffffff810f9dab>] mark_lock+0x28b/0x6a0
[<ffffffff810fcbf9>] __lock_acquire+0x949/0x20e0
[<ffffffff811091f2>] ? __module_text_address+0x12/0x60
[<ffffffff8110ea8f>] ? is_module_text_address+0x2f/0x60
[<ffffffff810b8408>] ? __kernel_text_address+0x58/0x80
[<ffffffff8104dbb2>] ? print_context_stack+0x62/0xf0
[<ffffffff810feaee>] lock_acquire+0x9e/0x1f0
[<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f9090>] _raw_spin_lock+0x40/0x80
[<ffffffff819f7e98>] ? rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f7e98>] rt_spin_lock_slowlock+0x48/0x370
[<ffffffff819f89ac>] rt_spin_lock+0x2c/0x60
[<ffffffff810ccdd6>] __wake_up+0x36/0x70
[<ffffffff81003bbb>] kvm_vcpu_kick+0x3b/0xd0
[<ffffffff810371a2>] __apic_accept_irq+0x2b2/0x3a0
[<ffffffff810385f7>] kvm_apic_set_irq+0x27/0x30
[<ffffffff8103894e>] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0
[<ffffffff81038800>] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0
[<ffffffff81009a8b>] kvm_set_irq_inatomic+0x12b/0x4a0
[<ffffffff8100998a>] ? kvm_set_irq_inatomic+0x2a/0x4a0
[<ffffffff8100c5b3>] kvm_assigned_dev_msi+0x23/0x40
[<ffffffff8113cb38>] handle_irq_event_percpu+0x88/0x3d0
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff8113cec8>] handle_irq_event+0x48/0x70
[<ffffffff8113f9b7>] handle_edge_irq+0x77/0x120
[<ffffffff8104c6ae>] handle_irq+0x1e/0x30
[<ffffffff81a035ca>] do_IRQ+0x5a/0xd0
[<ffffffff819f9a2f>] common_interrupt+0x6f/0x6f
<EOI> [<ffffffff819f9ae0>] ? retint_restore_args+0xe/0xe
[<ffffffff810ebb7c>] ? cpu_startup_entry+0x19c/0x430
[<ffffffff810ebb38>] ? cpu_startup_entry+0x158/0x430
[<ffffffff819db767>] rest_init+0x137/0x140
[<ffffffff819db635>] ? rest_init+0x5/0x140
[<ffffffff822fde18>] start_kernel+0x3af/0x3bc
[<ffffffff822fd870>] ? repair_env_string+0x5e/0x5e
[<ffffffff822fd5a5>] x86_64_start_reservations+0x2a/0x2c
[<ffffffff822fd673>] x86_64_start_kernel+0xcc/0xcf
>
> Paul.
> --
>
> >
> > Thanks,
> >
> > Paolo
> >
> > Paolo Bonzini (3):
> > KVM: cleanup (physical) CPU hotplug
> > KVM: protect kvm_usage_count with its own spinlock
> > KVM: Convert kvm_lock back to non-raw spinlock
> >
> > Documentation/virtual/kvm/locking.txt | 8 ++++--
> > arch/x86/kvm/mmu.c | 4 +--
> > arch/x86/kvm/x86.c | 8 +++---
> > include/linux/kvm_host.h | 2 +-
> > virt/kvm/kvm_main.c | 51 ++++++++++++++++++-----------------
> > 5 files changed, 40 insertions(+), 33 deletions(-)
> >
next prev parent reply other threads:[~2013-09-20 17:51 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-16 14:06 [PATCH 0/3] KVM: Make kvm_lock non-raw Paolo Bonzini
2013-09-16 14:06 ` [PATCH 1/3] KVM: cleanup (physical) CPU hotplug Paolo Bonzini
2013-09-17 7:57 ` Jan Kiszka
2013-09-17 23:19 ` Paolo Bonzini
2013-09-16 14:06 ` [PATCH 2/3] KVM: protect kvm_usage_count with its own spinlock Paolo Bonzini
2013-09-16 14:06 ` [PATCH 3/3] KVM: Convert kvm_lock back to non-raw spinlock Paolo Bonzini
2013-09-16 22:12 ` [PATCH 0/3] KVM: Make kvm_lock non-raw Paul Gortmaker
2013-09-20 17:51 ` Paul Gortmaker [this message]
2013-09-20 18:04 ` Jan Kiszka
2013-09-20 18:18 ` Paul Gortmaker
2013-09-20 18:27 ` Jan Kiszka
2013-09-21 20:26 ` Michael S. Tsirkin
2013-09-22 7:42 ` Gleb Natapov
2013-09-22 8:53 ` Paolo Bonzini
2013-09-22 9:53 ` Gleb Natapov
2013-09-23 6:30 ` Jan Kiszka
2013-09-23 13:36 ` Paul Gortmaker
2013-09-23 13:44 ` Paolo Bonzini
2013-09-23 14:59 ` Gleb Natapov
2013-09-23 15:05 ` Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130920175150.GA24770@windriver.com \
--to=paul.gortmaker@windriver.com \
--cc=gleb@redhat.com \
--cc=jan.kiszka@siemens.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=pbonzini@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.