[kvm.git & 2.6.37-rc1] KVM deadlock with CONFIG_PREEMPT host

[kvm.git & 2.6.37-rc1] KVM deadlock with CONFIG_PREEMPT host

[Jan Kiszka]

Hi,

I'm seeing lock-ups of the QEMU process on kvm.git as well as current
upstream kernels. This is a backtrace of the hanging VCPU thread:

[<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
[<ffffffff810a286f>] try_stop_cpus+0x40/0x59
[<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
[<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
[<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
[<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
[<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
[<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
[<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
[<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
[<ffffffff81143cbd>] sys_ioctl+0x56/0x79
[<ffffffff81002df2>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

This issue disappears when disabling CONFIG_PREEMPT on the host.
According to some rough bisecting, it was imported into kvm.git with
merge 146d3bb06b. Given that RCU is involved, I also tried
force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no
difference as long as PREEMPT is on.

Can anyone confirm this or does someone have an idea what goes wrong? Of
course, .config will be provided if required.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [kvm.git & 2.6.37-rc1] KVM deadlock with CONFIG_PREEMPT host

[Markus Trippelsdorf]

On Mon, Nov 08, 2010 at 10:04:00AM +0100, Jan Kiszka wrote:
> Hi,
> 
> I'm seeing lock-ups of the QEMU process on kvm.git as well as current
> upstream kernels. This is a backtrace of the hanging VCPU thread:
> 
> [<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
> [<ffffffff810a286f>] try_stop_cpus+0x40/0x59
> [<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
> [<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
> [<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
> [<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
> [<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
> [<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
> [<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
> [<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
> [<ffffffff81143cbd>] sys_ioctl+0x56/0x79
> [<ffffffff81002df2>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> 
> This issue disappears when disabling CONFIG_PREEMPT on the host.
> According to some rough bisecting, it was imported into kvm.git with
> merge 146d3bb06b. Given that RCU is involved, I also tried
> force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no
> difference as long as PREEMPT is on.
> 
> Can anyone confirm this or does someone have an idea what goes wrong? Of
> course, .config will be provided if required.

This patch should help (,it fixes the problem in my case):
http://article.gmane.org/gmane.linux.kernel/1058018
-- 
Markus

Re: [kvm.git & 2.6.37-rc1] KVM deadlock with CONFIG_PREEMPT host

[Jan Kiszka]

Am 08.11.2010 10:18, Markus Trippelsdorf wrote:
> On Mon, Nov 08, 2010 at 10:04:00AM +0100, Jan Kiszka wrote:
>> Hi,
>>
>> I'm seeing lock-ups of the QEMU process on kvm.git as well as current
>> upstream kernels. This is a backtrace of the hanging VCPU thread:
>>
>> [<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
>> [<ffffffff810a286f>] try_stop_cpus+0x40/0x59
>> [<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
>> [<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
>> [<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
>> [<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
>> [<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
>> [<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
>> [<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
>> [<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
>> [<ffffffff81143cbd>] sys_ioctl+0x56/0x79
>> [<ffffffff81002df2>] system_call_fastpath+0x16/0x1b
>> [<ffffffffffffffff>] 0xffffffffffffffff
>>
>> This issue disappears when disabling CONFIG_PREEMPT on the host.
>> According to some rough bisecting, it was imported into kvm.git with
>> merge 146d3bb06b. Given that RCU is involved, I also tried
>> force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no
>> difference as long as PREEMPT is on.
>>
>> Can anyone confirm this or does someone have an idea what goes wrong? Of
>> course, .config will be provided if required.
> 
> This patch should help (,it fixes the problem in my case):
> http://article.gmane.org/gmane.linux.kernel/1058018

Yeah, that works here as well.

Would be a nice-to-have in kvm.git until it's pull via the next upstream
merge. At least for me this bug triggers at every VM boot.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [kvm.git & 2.6.37-rc1] KVM deadlock with CONFIG_PREEMPT host

[Avi Kivity]

On 11/08/2010 12:28 PM, Jan Kiszka wrote:
> Am 08.11.2010 10:18, Markus Trippelsdorf wrote:
> >  On Mon, Nov 08, 2010 at 10:04:00AM +0100, Jan Kiszka wrote:
> >>  Hi,
> >>
> >>  I'm seeing lock-ups of the QEMU process on kvm.git as well as current
> >>  upstream kernels. This is a backtrace of the hanging VCPU thread:
> >>
> >>  [<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
> >>  [<ffffffff810a286f>] try_stop_cpus+0x40/0x59
> >>  [<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
> >>  [<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
> >>  [<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
> >>  [<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
> >>  [<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
> >>  [<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
> >>  [<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
> >>  [<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
> >>  [<ffffffff81143cbd>] sys_ioctl+0x56/0x79
> >>  [<ffffffff81002df2>] system_call_fastpath+0x16/0x1b
> >>  [<ffffffffffffffff>] 0xffffffffffffffff
> >>
> >>  This issue disappears when disabling CONFIG_PREEMPT on the host.
> >>  According to some rough bisecting, it was imported into kvm.git with
> >>  merge 146d3bb06b. Given that RCU is involved, I also tried
> >>  force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no
> >>  difference as long as PREEMPT is on.
> >>
> >>  Can anyone confirm this or does someone have an idea what goes wrong? Of
> >>  course, .config will be provided if required.
> >
> >  This patch should help (,it fixes the problem in my case):
> >  http://article.gmane.org/gmane.linux.kernel/1058018
>
> Yeah, that works here as well.
>
> Would be a nice-to-have in kvm.git until it's pull via the next upstream
> merge. At least for me this bug triggers at every VM boot.
>

I just did so.  Testing with CONFIG_PREEMPT is important.

-- 
error compiling committee.c: too many arguments to function