Soft lockups and cpu frequency scaling

Soft lockups and cpu frequency scaling

[Martin Schmitt]

All,

I'm running a manually compiled KVM on CentOS 5.4. The KVM installation
has been carried over from CentOS 5.3, when KVM wasn't distributed with
the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't
able to get along with RedHat's interpretation of KVM.)

The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't
seem to compile on CentOS' current 2.6.18-164.9.1.el5.

Only on CentOS guests, I see very frequent "soft lockup" messages and
excessively hanging KVM instances. Long-term stability is fine (several
months uptime), but disturbed by the hangs. The problem already was
there on CentOS 5.3 as well. With the Debian guests on the same host, I
have never had any apparent problems.

A number of google results suggest that I should work with CPU scaling
on the CentOS guest systems, but unfortunately, CPU scaling is not
available in my guests. So, here's my question: How do I enable CPU
scaling in KVM guests? Or is there any other measure against these soft
lockups that you can recommend?

Thanks everyone,

-martin

-- 
Martin Schmitt / Schmitt Systemberatung / www.scsy.de
--> http://www.pug.org/index.php/Benutzer:Martin <--

Re: Soft lockups and cpu frequency scaling

[Marcelo Tosatti]

On Tue, Dec 22, 2009 at 02:16:49PM +0100, Martin Schmitt wrote:
> All,
> 
> I'm running a manually compiled KVM on CentOS 5.4. The KVM installation
> has been carried over from CentOS 5.3, when KVM wasn't distributed with
> the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't
> able to get along with RedHat's interpretation of KVM.)
> 
> The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't
> seem to compile on CentOS' current 2.6.18-164.9.1.el5.
> 
> Only on CentOS guests, I see very frequent "soft lockup" messages and
> excessively hanging KVM instances.

Can you please share some of the soft lockup messages.

And how exactly are the VMs hanging?

>  Long-term stability is fine (several months uptime), but disturbed
> by the hangs. The problem already was there on CentOS 5.3 as well.
> With the Debian guests on the same host, I have never had any apparent
> problems.

Questions:

- Is there significant swapping on the host?
- Are you migrating vm's? 

> A number of google results suggest that I should work with CPU scaling
> on the CentOS guest systems, but unfortunately, CPU scaling is not
> available in my guests. So, here's my question: How do I enable CPU
> scaling in KVM guests? Or is there any other measure against these soft
> lockups that you can recommend?

What probably was suggested is to disable cpu frequency scaling on the
host. Please provide more details on the host system.

Re: Soft lockups and cpu frequency scaling

[Martin Schmitt]

[-- Attachment #1: Type: text/plain, Size: 4350 bytes --]

Ciao Marcelo,

sorry for getting back so late. Thanks for your patience. :-)

Marcelo Tosatti schrieb:

>> I'm running a manually compiled KVM on CentOS 5.4. The KVM installation
>> has been carried over from CentOS 5.3, when KVM wasn't distributed with
>> the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't
>> able to get along with RedHat's interpretation of KVM.)
>>
>> The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't
>> seem to compile on CentOS' current 2.6.18-164.9.1.el5.
>>
>> Only on CentOS guests, I see very frequent "soft lockup" messages and
>> excessively hanging KVM instances.
> 
> Can you please share some of the soft lockup messages.
> 
> And how exactly are the VMs hanging?

They are unresponsive for a few seconds. More "hiccuping" than hanging.
It appears to be I/O-related in some way, because it happens most
frequently when I do things on the file system.

Dmesg is full of these:

BUG: soft lockup - CPU#0 stuck for 10s! [kblockd/0:10]

Pid: 10, comm:            kblockd/0
EIP: 0060:[<c056f931>] CPU: 0
EIP is at ide_outb+0x4/0x5
 EFLAGS: 00000202    Not tainted  (2.6.18-164.6.1.el5 #1)
EAX: 00000001 EBX: c07e2f80 ECX: 00000286 EDX: 0000c000
ESI: 00000011 EDI: 00000000 EBP: c07e3014 DS: 007b ES: 007b
CR0: 8005003b CR2: b7f3c000 CR3: 12122000 CR4: 000006d0
 [<c0573cab>] ide_dma_start+0x22/0x2e
 [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
 [<c056de34>] ide_do_request+0x533/0x6bf
 [<c04de1b9>] freed_request+0x1d/0x37
 [<c056d8d0>] ide_end_request+0xcc/0xd4
 [<c056e221>] ide_intr+0x167/0x190
 [<c044da39>] handle_IRQ_event+0x45/0x8c
 [<c044db04>] __do_IRQ+0x84/0xd6
 [<c044da80>] __do_IRQ+0x0/0xd6
 [<c04074b2>] do_IRQ+0x99/0xc3
 [<c0405946>] common_interrupt+0x1a/0x20
 [<c04291ab>] __do_softirq+0x57/0x114
 [<c04073cf>] do_softirq+0x52/0x9c
 [<c04059d7>] apic_timer_interrupt+0x1f/0x24
 [<c056f931>] ide_outb+0x4/0x5
 [<c0573cab>] ide_dma_start+0x22/0x2e
 [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
 [<c056de34>] ide_do_request+0x533/0x6bf
 [<c04e710f>] cfq_kick_queue+0x70/0x80
 [<c0431e8a>] run_workqueue+0x78/0xb5
 [<c04e709f>] cfq_kick_queue+0x0/0x80
 [<c043273e>] worker_thread+0xd9/0x10b
 [<c041e727>] default_wake_function+0x0/0xc
 [<c0432665>] worker_thread+0x0/0x10b
 [<c0434b55>] kthread+0xc0/0xeb
 [<c0434a95>] kthread+0x0/0xeb
 [<c0405c53>] kernel_thread_helper+0x7/0x10
 =======================

>>  Long-term stability is fine (several months uptime), but disturbed
>> by the hangs. The problem already was there on CentOS 5.3 as well.
>> With the Debian guests on the same host, I have never had any apparent
>> problems.
> 
> Questions:
> 
> - Is there significant swapping on the host?
> - Are you migrating vm's? 

No migration and no swap activity. The host has plenty of idle RAM:

[root@zulu ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7987       7904         82          0        667       5101
-/+ buffers/cache:       2135       5851
Swap:         1983          0       1983

>> A number of google results suggest that I should work with CPU scaling
>> on the CentOS guest systems, but unfortunately, CPU scaling is not
>> available in my guests. So, here's my question: How do I enable CPU
>> scaling in KVM guests? Or is there any other measure against these soft
>> lockups that you can recommend?
> 
> What probably was suggested is to disable cpu frequency scaling on the
> host. Please provide more details on the host system.

Host is a Quadcore Xeon HP DL320 G5 with CentOS 5.4, old Kernel
2.6.18-128.7.1.el5.

There are no hints toward CPU scaling in /sys/devices/system/ on the host:

[root@zulu ~]# ls -l /sys/devices/system/cpu/cpu0
total 0
drwxr-xr-x 5 root root    0 Nov  7 13:47 cache
-r-------- 1 root root 4096 Jan  4 14:55 crash_notes
drwxr-xr-x 2 root root    0 Nov  7 13:48 topology

The file "Crash Notes" contains the following number: 22792b400

Thanks for your help,

-martin

-- 
Martin Schmitt - Schmitt Systemberatung - http://www.scsy.de
DE 35415 Pohlheim, Gießener Str. 18
DE 65307 Bad Schwalbach, Am Bräunchesberg 9
Linux/UNIX - Internet - E-Mail Infrastructure - Antispam/Antivirus
- "What goes up, must come down. Ask any system administrator." -


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]

Re: Soft lockups and cpu frequency scaling

[Martin Schmitt]

Marcelo Tosatti schrieb:

> Can you please share a few more "soft lockup" messages? (with
> backtrace included).

Full dmesg from guest: http://pastebin.com/f51a966df

> Also qemu command line.

>From ps:

/usr/local/kvm/bin/qemu-system-x86_64 -hda /drbd/vweb/vweb.vmdk -vnc
127.0.0.1:4 -m 512 -net nic,macaddr=00:16:3e:6a:a5:10 -net tap
-enable-kvm -k de -pidfile /drbd/vweb/pidfile.pid -monitor
tcp:127.0.0.1:1004,server,nowait -name /drbd/vweb/vweb.vmdk -cpu host
-daemonize

> And boot-up messages of host and guest.

Does the dmesg above cover that for the guest or are you asking for
anything further than that?

Host side dmesg: http://pastebin.com/f47a30d22

Thanks for your time,

-martin

-- 
Martin Schmitt / Schmitt Systemberatung / www.scsy.de
--> http://www.pug.org/index.php/Benutzer:Martin <--

Re: Soft lockups and cpu frequency scaling

[Marcelo Tosatti]

On Mon, Jan 04, 2010 at 03:01:29PM +0100, Martin Schmitt wrote:
> Ciao Marcelo,
> 
> sorry for getting back so late. Thanks for your patience. :-)
> 
> Marcelo Tosatti schrieb:
> 
> >> I'm running a manually compiled KVM on CentOS 5.4. The KVM installation
> >> has been carried over from CentOS 5.3, when KVM wasn't distributed with
> >> the OS. (I tried to migrate to CentOS 5.4 native KVM support, but wasn't
> >> able to get along with RedHat's interpretation of KVM.)
> >>
> >> The KVM version used is 88, on Kernel 2.6.18-128.7.1.el5, as KVM doesn't
> >> seem to compile on CentOS' current 2.6.18-164.9.1.el5.
> >>
> >> Only on CentOS guests, I see very frequent "soft lockup" messages and
> >> excessively hanging KVM instances.
> > 
> > Can you please share some of the soft lockup messages.
> > 
> > And how exactly are the VMs hanging?
> 
> They are unresponsive for a few seconds. More "hiccuping" than hanging.
> It appears to be I/O-related in some way, because it happens most
> frequently when I do things on the file system.
> 
> Dmesg is full of these:
> 
> BUG: soft lockup - CPU#0 stuck for 10s! [kblockd/0:10]
> 
> Pid: 10, comm:            kblockd/0
> EIP: 0060:[<c056f931>] CPU: 0
> EIP is at ide_outb+0x4/0x5
>  EFLAGS: 00000202    Not tainted  (2.6.18-164.6.1.el5 #1)
> EAX: 00000001 EBX: c07e2f80 ECX: 00000286 EDX: 0000c000
> ESI: 00000011 EDI: 00000000 EBP: c07e3014 DS: 007b ES: 007b
> CR0: 8005003b CR2: b7f3c000 CR3: 12122000 CR4: 000006d0
>  [<c0573cab>] ide_dma_start+0x22/0x2e
>  [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
>  [<c056de34>] ide_do_request+0x533/0x6bf
>  [<c04de1b9>] freed_request+0x1d/0x37
>  [<c056d8d0>] ide_end_request+0xcc/0xd4
>  [<c056e221>] ide_intr+0x167/0x190
>  [<c044da39>] handle_IRQ_event+0x45/0x8c
>  [<c044db04>] __do_IRQ+0x84/0xd6
>  [<c044da80>] __do_IRQ+0x0/0xd6
>  [<c04074b2>] do_IRQ+0x99/0xc3
>  [<c0405946>] common_interrupt+0x1a/0x20
>  [<c04291ab>] __do_softirq+0x57/0x114
>  [<c04073cf>] do_softirq+0x52/0x9c
>  [<c04059d7>] apic_timer_interrupt+0x1f/0x24
>  [<c056f931>] ide_outb+0x4/0x5
>  [<c0573cab>] ide_dma_start+0x22/0x2e
>  [<c0576474>] ide_do_rw_disk+0x3b2/0x4a6
>  [<c056de34>] ide_do_request+0x533/0x6bf
>  [<c04e710f>] cfq_kick_queue+0x70/0x80
>  [<c0431e8a>] run_workqueue+0x78/0xb5
>  [<c04e709f>] cfq_kick_queue+0x0/0x80
>  [<c043273e>] worker_thread+0xd9/0x10b
>  [<c041e727>] default_wake_function+0x0/0xc
>  [<c0432665>] worker_thread+0x0/0x10b
>  [<c0434b55>] kthread+0xc0/0xeb
>  [<c0434a95>] kthread+0x0/0xeb
>  [<c0405c53>] kernel_thread_helper+0x7/0x10
>  =======================
> >> by the hangs. The problem already was there on CentOS 5.3 as well.
> >> With the Debian guests on the same host, I have never had any apparent
> >> problems.
> > 
> > Questions:
> > 
> > - Is there significant swapping on the host?
> > - Are you migrating vm's? 
> 
> No migration and no swap activity. The host has plenty of idle RAM:
> 
> [root@zulu ~]# free -m
>              total       used       free     shared    buffers     cached
> Mem:          7987       7904         82          0        667       5101
> -/+ buffers/cache:       2135       5851
> Swap:         1983          0       1983
> 
> >> A number of google results suggest that I should work with CPU scaling
> >> on the CentOS guest systems, but unfortunately, CPU scaling is not
> >> available in my guests. So, here's my question: How do I enable CPU
> >> scaling in KVM guests? Or is there any other measure against these soft
> >> lockups that you can recommend?
> > 
> > What probably was suggested is to disable cpu frequency scaling on the
> > host. Please provide more details on the host system.
> 
> Host is a Quadcore Xeon HP DL320 G5 with CentOS 5.4, old Kernel
> 2.6.18-128.7.1.el5.
> 
> There are no hints toward CPU scaling in /sys/devices/system/ on the host:
> 
> [root@zulu ~]# ls -l /sys/devices/system/cpu/cpu0
> total 0
> drwxr-xr-x 5 root root    0 Nov  7 13:47 cache
> -r-------- 1 root root 4096 Jan  4 14:55 crash_notes
> drwxr-xr-x 2 root root    0 Nov  7 13:48 topology
> 
> The file "Crash Notes" contains the following number: 22792b400
> 
> Thanks for your help,


Martin,

Can you please share a few more "soft lockup" messages? (with
backtrace included).

Also qemu command line.

And boot-up messages of host and guest.

Thanks

Re: Soft lockups and cpu frequency scaling

[Marcelo Tosatti]

On Wed, Jan 06, 2010 at 09:15:02AM +0100, Martin Schmitt wrote:
> Marcelo Tosatti schrieb:
> 
> > Can you please share a few more "soft lockup" messages? (with
> > backtrace included).
> 
> Full dmesg from guest: http://pastebin.com/f51a966df
> 
> > Also qemu command line.
> 
> >From ps:
> 
> /usr/local/kvm/bin/qemu-system-x86_64 -hda /drbd/vweb/vweb.vmdk -vnc
> 127.0.0.1:4 -m 512 -net nic,macaddr=00:16:3e:6a:a5:10 -net tap
> -enable-kvm -k de -pidfile /drbd/vweb/pidfile.pid -monitor
> tcp:127.0.0.1:1004,server,nowait -name /drbd/vweb/vweb.vmdk -cpu host
> -daemonize

Hum, can you try converting that vmdk image to qcow2 or raw? (with
qemu-img convert).

AFAICS the QEMU vmdk implementation is synchronous, so the guest 
waits on IO operations to complete on the host side.

> > And boot-up messages of host and guest.
> 
> Does the dmesg above cover that for the guest or are you asking for
> anything further than that?
> 
> Host side dmesg: http://pastebin.com/f47a30d22
> 
> Thanks for your time,
> 
> -martin

Re: Soft lockups and cpu frequency scaling

[Martin Schmitt]

Marcelo Tosatti schrieb:

> Hum, can you try converting that vmdk image to qcow2 or raw? (with
> qemu-img convert).
> 
> AFAICS the QEMU vmdk implementation is synchronous, so the guest 
> waits on IO operations to complete on the host side.

I'll do so, but I can only make the conversion early morning or late
evening, so it won't happen before tomorrow.

The reason why I'm on vmdk is that these were migrated from VMware
server. The free VMware server isn't much fun anymore, and I just need
"simple" virtualization, no big-iron enterprise BS compliance. ;-)

Thanks,

-martin

-- 
Martin Schmitt / Schmitt Systemberatung / www.scsy.de
--> http://www.pug.org/index.php/Benutzer:Martin <--