BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Boris Dolgov]

Hello!

I try to use KVM on slackware host and centos5 guest with SMP.

When the server is idle, I get following errors in /var/log/messages:
http://pastebin.com/4RrMVuXq

Guest:
[root@centos-5-x64 ~]# uname -a
Linux centos-5-x64.slave 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10
19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Start command: /usr/local/qemu/bin/qemu-system-x86_64 -name
centos-5-x64 -m 384 -smp 2 -cdrom
/root/CentOS-5.5-x86_64-netinstall.iso -drive
index=0,media=disk,if=virtio,file=/dev/mapper/vgvm-centos5x64,boot=on
-vnc 0.0.0.0:1 -net nic,macaddr=fe:e1:de:ad:57:22 -net
tap,script=/usr/local/qemu/bin/qemu-ifup -monitor stdio -usb
-usbdevice tablet -enable-kvm

Host:
root@superserver:~# uname -a
Linux superserver 2.6.35.3 #4 SMP Mon Aug 23 12:17:01 MSD 2010 x86_64
Intel(R) Xeon(R) CPU           X3460  @ 2.80GHz GenuineIntel GNU/Linux
root@superserver:~# /usr/local/qemu/bin/qemu-system-x86_64 -version
QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c)
2003-2008 Fabrice Bellard

What does it mean?

-- 
Boris Dolgov.

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Avi Kivity]

  On 08/25/2010 04:15 PM, Boris Dolgov wrote:
> Hello!
>
> I try to use KVM on slackware host and centos5 guest with SMP.
>
> When the server is idle, I get following errors in /var/log/messages:
> http://pastebin.com/4RrMVuXq
>
> Guest:
> [root@centos-5-x64 ~]# uname -a
> Linux centos-5-x64.slave 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10
> 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
>
> Start command: /usr/local/qemu/bin/qemu-system-x86_64 -name
> centos-5-x64 -m 384 -smp 2 -cdrom
> /root/CentOS-5.5-x86_64-netinstall.iso -drive
> index=0,media=disk,if=virtio,file=/dev/mapper/vgvm-centos5x64,boot=on
> -vnc 0.0.0.0:1 -net nic,macaddr=fe:e1:de:ad:57:22 -net
> tap,script=/usr/local/qemu/bin/qemu-ifup -monitor stdio -usb
> -usbdevice tablet -enable-kvm
>
> Host:
> root@superserver:~# uname -a
> Linux superserver 2.6.35.3 #4 SMP Mon Aug 23 12:17:01 MSD 2010 x86_64
> Intel(R) Xeon(R) CPU           X3460  @ 2.80GHz GenuineIntel GNU/Linux
> root@superserver:~# /usr/local/qemu/bin/qemu-system-x86_64 -version
> QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c)
> 2003-2008 Fabrice Bellard
>
> What does it mean?

Looks like a bug in our USB emulation.  Copying qemu-devel to see if 
anyone has a clue.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Anthony Liguori]

On 08/25/2010 08:19 AM, Avi Kivity wrote:
>  On 08/25/2010 04:15 PM, Boris Dolgov wrote:
>> Hello!
>>
>> I try to use KVM on slackware host and centos5 guest with SMP.
>>
>> When the server is idle, I get following errors in /var/log/messages:
>> http://pastebin.com/4RrMVuXq
>>
>> Guest:
>> [root@centos-5-x64 ~]# uname -a
>> Linux centos-5-x64.slave 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10
>> 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
>>
>> Start command: /usr/local/qemu/bin/qemu-system-x86_64 -name
>> centos-5-x64 -m 384 -smp 2 -cdrom
>> /root/CentOS-5.5-x86_64-netinstall.iso -drive
>> index=0,media=disk,if=virtio,file=/dev/mapper/vgvm-centos5x64,boot=on
>> -vnc 0.0.0.0:1 -net nic,macaddr=fe:e1:de:ad:57:22 -net
>> tap,script=/usr/local/qemu/bin/qemu-ifup -monitor stdio -usb
>> -usbdevice tablet -enable-kvm
>>
>> Host:
>> root@superserver:~# uname -a
>> Linux superserver 2.6.35.3 #4 SMP Mon Aug 23 12:17:01 MSD 2010 x86_64
>> Intel(R) Xeon(R) CPU           X3460  @ 2.80GHz GenuineIntel GNU/Linux
>> root@superserver:~# /usr/local/qemu/bin/qemu-system-x86_64 -version
>> QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c)
>> 2003-2008 Fabrice Bellard
>>
>> What does it mean?
>
> Looks like a bug in our USB emulation.  Copying qemu-devel to see if 
> anyone has a clue.

USB tablet emulation would not do anything that causes QEMU to block so 
it seems unlikely to me.

What's more likely is that since USB is polling based, statistically 
speaking it's more likely that a VCPU is executing that call when the 
soft lockup is detected.

Regards,

Anthony Liguori

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Boris Dolgov]

On Wed, Aug 25, 2010 at 5:19 PM, Avi Kivity <avi@redhat.com> wrote:
> Looks like a bug in our USB emulation.  Copying qemu-devel to see if anyone
> has a clue.
I have disabled USB support, but the bug have not disappeared.
Please, pay attention that CPU#1 was stuck too.


-- 
Boris Dolgov.

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Boris Dolgov]

On Wed, Aug 25, 2010 at 5:15 PM, Boris Dolgov <boris@dolgov.name> wrote:
> Hello!
>
> I try to use KVM on slackware host and centos5 guest with SMP.
>
> When the server is idle, I get following errors in /var/log/messages:
> http://pastebin.com/4RrMVuXq
>
> Guest:
> [root@centos-5-x64 ~]# uname -a
> Linux centos-5-x64.slave 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10
> 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
>
> Start command: /usr/local/qemu/bin/qemu-system-x86_64 -name
> centos-5-x64 -m 384 -smp 2 -cdrom
> /root/CentOS-5.5-x86_64-netinstall.iso -drive
> index=0,media=disk,if=virtio,file=/dev/mapper/vgvm-centos5x64,boot=on
> -vnc 0.0.0.0:1 -net nic,macaddr=fe:e1:de:ad:57:22 -net
> tap,script=/usr/local/qemu/bin/qemu-ifup -monitor stdio -usb
> -usbdevice tablet -enable-kvm
>
> Host:
> root@superserver:~# uname -a
> Linux superserver 2.6.35.3 #4 SMP Mon Aug 23 12:17:01 MSD 2010 x86_64
> Intel(R) Xeon(R) CPU           X3460  @ 2.80GHz GenuineIntel GNU/Linux
> root@superserver:~# /usr/local/qemu/bin/qemu-system-x86_64 -version
> QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c)
> 2003-2008 Fabrice Bellard
>
> What does it mean?
The problem also reproduces without SMP or USB for a guest.

I have installed Debian 5.0.5 to another VM, and the problem is
unreproducible under it:
debian-5-x64:~# uname -a
Linux debian-5-x64.slave 2.6.26-2-amd64 #1 SMP Thu Aug 19 00:37:36 UTC
2010 x86_64 GNU/Linux

Should I post it to redhat bugzilla as redhat bug, or it is KVM bug?

Also, I have noticed a strange thing. In top on host, Debian qemu
process consumes nearly 1% of cpu, but CentOS qemu process consumes
nearly 20-30%, even when VM is idle.

The problem also reproduces on centos-i386 guest and doesn't reproduce
on debian-5-i386.

-- 
Boris Dolgov.

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Anthony Liguori]

On 08/25/2010 11:14 AM, Boris Dolgov wrote:
> On Wed, Aug 25, 2010 at 5:15 PM, Boris Dolgov<boris@dolgov.name>  wrote:
>    
>> Hello!
>>
>> I try to use KVM on slackware host and centos5 guest with SMP.
>>
>> When the server is idle, I get following errors in /var/log/messages:
>> http://pastebin.com/4RrMVuXq
>>
>> Guest:
>> [root@centos-5-x64 ~]# uname -a
>> Linux centos-5-x64.slave 2.6.18-194.11.1.el5 #1 SMP Tue Aug 10
>> 19:05:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
>>
>> Start command: /usr/local/qemu/bin/qemu-system-x86_64 -name
>> centos-5-x64 -m 384 -smp 2 -cdrom
>> /root/CentOS-5.5-x86_64-netinstall.iso -drive
>> index=0,media=disk,if=virtio,file=/dev/mapper/vgvm-centos5x64,boot=on
>> -vnc 0.0.0.0:1 -net nic,macaddr=fe:e1:de:ad:57:22 -net
>> tap,script=/usr/local/qemu/bin/qemu-ifup -monitor stdio -usb
>> -usbdevice tablet -enable-kvm
>>
>> Host:
>> root@superserver:~# uname -a
>> Linux superserver 2.6.35.3 #4 SMP Mon Aug 23 12:17:01 MSD 2010 x86_64
>> Intel(R) Xeon(R) CPU           X3460  @ 2.80GHz GenuineIntel GNU/Linux
>> root@superserver:~# /usr/local/qemu/bin/qemu-system-x86_64 -version
>> QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c)
>> 2003-2008 Fabrice Bellard
>>
>> What does it mean?
>>      
> The problem also reproduces without SMP or USB for a guest.
>
> I have installed Debian 5.0.5 to another VM, and the problem is
> unreproducible under it:
> debian-5-x64:~# uname -a
> Linux debian-5-x64.slave 2.6.26-2-amd64 #1 SMP Thu Aug 19 00:37:36 UTC
> 2010 x86_64 GNU/Linux
>
> Should I post it to redhat bugzilla as redhat bug, or it is KVM bug?
>    

Or post it to the CentOS bugzilla since you're not using Red Hat.

There's been various time related issues in RHEL5.x so my inclination 
would be that it's not an upstream issue until you verify it with the 
latest upstream.

Regards,

Anthony Liguori

> Also, I have noticed a strange thing. In top on host, Debian qemu
> process consumes nearly 1% of cpu, but CentOS qemu process consumes
> nearly 20-30%, even when VM is idle.
>
> The problem also reproduces on centos-i386 guest and doesn't reproduce
> on debian-5-i386.
>
>    

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Boris Dolgov]

On Wed, Aug 25, 2010 at 8:19 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Or post it to the CentOS bugzilla since you're not using Red Hat.
I'am using redhat, but not on this VM.

> There's been various time related issues in RHEL5.x so my inclination would
> be that it's not an upstream issue until you verify it with the latest
> upstream.
Do you mean latest kvm or latest RHEL?
I tried to use qemu-kvm from GIT today, but it doesn't compile.

-- 
Boris Dolgov.

Re: BUG: soft lockup - CPU#0 stuck for 13s! [swapper:0]

[Boris Dolgov]

On Wed, Aug 25, 2010 at 8:19 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Or post it to the CentOS bugzilla since you're not using Red Hat.
>
> There's been various time related issues in RHEL5.x so my inclination would
> be that it's not an upstream issue until you verify it with the latest
> upstream.
After setting clocksource to rtc (instead of dynticks), the problem disappeared.


-- 
Boris Dolgov.