KVM got slow after adding more physical memory to host

KVM got slow after adding more physical memory to host

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1452 bytes --]

Hello,

I'm trying to solve quite a weird problem on one of our customers' box.
It's quad core Xeon X3430  @ 2.40GHz, running 64bit centos with 2.6.38 (I also tried upgrading to
2.6.39).
After increasing physical memory from 16 to 20GB, all guest got incredibly slow, starting just
one windows or linux guest causes host to be quite slow, and quest seems to be even slower.
hardware virtualisation is on, KVM is used, no relevant update since the update.
here's about 3 seconds of kvm_stat:

 kvm_exit                                    659513  103922
 kvm_entry                                   659505  103920
 kvm_emulate_insn                            631118   99344
 kvm_exit(IO_INSTRUCTION)                    625223   98398
 kvm_userspace_exit                          624979   98359
 kvm_pio                                     624970   98357

but according to atop, there is almost no I/O activity from guests, and running top in them
shows them to really be idle...
but they eat like 60-70% of host CPU core time each..

Do You have any idea on what should I check?
thanks a lot in advance

nik
-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM got slow after adding more physical memory to host

[Avi Kivity]

On 09/06/2011 05:02 PM, Nikola Ciprich wrote:
> Hello,
>
> I'm trying to solve quite a weird problem on one of our customers' box.
> It's quad core Xeon X3430  @ 2.40GHz, running 64bit centos with 2.6.38 (I also tried upgrading to
> 2.6.39).
> After increasing physical memory from 16 to 20GB, all guest got incredibly slow, starting just
> one windows or linux guest causes host to be quite slow, and quest seems to be even slower.
> hardware virtualisation is on, KVM is used, no relevant update since the update.
> here's about 3 seconds of kvm_stat:
>
>   kvm_exit                                    659513  103922
>   kvm_entry                                   659505  103920
>   kvm_emulate_insn                            631118   99344
>   kvm_exit(IO_INSTRUCTION)                    625223   98398
>   kvm_userspace_exit                          624979   98359
>   kvm_pio                                     624970   98357

How many guests in there?

Please post a trace as per http://www.linux-kvm.org/page/Tracing.

> but according to atop, there is almost no I/O activity from guests, and running top in them
> shows them to really be idle...
> but they eat like 60-70% of host CPU core time each..
>
> Do You have any idea on what should I check?
> thanks a lot in advance

Please post /proc/mtrr and /proc/iomem.

-- 
error compiling committee.c: too many arguments to function

Re: KVM got slow after adding more physical memory to host

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 3312 bytes --]

Hello Avi,
thanks for quick reply!

> How many guests in there?
two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.

>
> Please post a trace as per http://www.linux-kvm.org/page/Tracing.
here it goes:
http://nelide.cz/nik/trace.dat.bz2

>
> Please post /proc/mtrr and /proc/iomem.
[root@virtualbox ~]# cat /proc/mtrr 
reg00: base=0x000000000 (    0MB), size=16384MB, count=1: write-back
reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable


[root@virtualbox ~]# cat /proc/iomem 
00000000-0000ffff : reserved
00010000-0009d7ff : System RAM
0009d800-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000cffff : pnp 00:0f
000d0000-000dffff : PCI Bus 0000:00
000e0000-000fffff : reserved
00100000-bf78ffff : System RAM
  01000000-013917ae : Kernel code
  013917af-016c88ff : Kernel data
  0176a000-0182ffff : Kernel bss
bf790000-bf79dfff : RAM buffer
bf79e000-bf79ffff : reserved
bf7a0000-bf7adfff : ACPI Tables
bf7ae000-bf7cffff : ACPI Non-volatile Storage
bf7d0000-bf7dffff : reserved
bf7e0000-bf7ecfff : RAM buffer
bf7ed000-bfffffff : reserved
c0000000-dfffffff : PCI Bus 0000:00
  c0000000-c01fffff : PCI Bus 0000:08
  c0200000-c03fffff : PCI Bus 0000:07
  c0400000-c05fffff : PCI Bus 0000:06
  c0600000-c07fffff : PCI Bus 0000:06
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
  e0000000-efffffff : reserved
    e0000000-efffffff : pnp 00:0e
f0000000-fed8ffff : PCI Bus 0000:00
  fa000000-faffffff : PCI Bus 0000:09
    fa000000-faffffff : 0000:09:03.0
  fb4f6000-fb4f60ff : 0000:00:1f.3
  fb4f8000-fb4f87ff : 0000:00:1f.2
    fb4f8000-fb4f87ff : ahci
  fb4fa000-fb4fa3ff : 0000:00:1d.0
    fb4fa000-fb4fa3ff : ehci_hcd
  fb4fc000-fb4fc3ff : 0000:00:1a.0
    fb4fc000-fb4fc3ff : ehci_hcd
  fb500000-fb5fffff : PCI Bus 0000:07
    fb5dc000-fb5dffff : 0000:07:00.0
      fb5dc000-fb5dffff : e1000e
    fb5e0000-fb5fffff : 0000:07:00.0
      fb5e0000-fb5fffff : e1000e
  fb600000-fb6fffff : PCI Bus 0000:08
    fb6dc000-fb6dffff : 0000:08:00.0
      fb6dc000-fb6dffff : e1000e
    fb6e0000-fb6fffff : 0000:08:00.0
      fb6e0000-fb6fffff : e1000e
  fb700000-fbffffff : PCI Bus 0000:09
    fb7fc000-fb7fffff : 0000:09:03.0
    fb800000-fbffffff : 0000:09:03.0
  fc000000-fcffffff : pnp 00:01
  fd000000-fdffffff : pnp 00:01
  fe000000-febfffff : pnp 00:01
  fec00000-fec003ff : IOAPIC 0
  fed00000-fed003ff : HPET 0
  fed14000-fed19fff : pnp 00:01
  fed1c000-fed1ffff : pnp 00:0b
  fed20000-fed3ffff : reserved
    fed20000-fed3ffff : pnp 00:0b
  fed40000-fed8ffff : pnp 00:0b
fee00000-fee00fff : Local APIC
  fee00000-fee00fff : reserved
    fee00000-fee00fff : pnp 00:0d
100000000-53fffffff : System RAM

n.


>
> -- 
> error compiling committee.c: too many arguments to function
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM got slow after adding more physical memory to host

[Avi Kivity]

On 09/06/2011 06:38 PM, Nikola Ciprich wrote:
> Hello Avi,
> thanks for quick reply!
>
> >  How many guests in there?
> two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.
>
> >
> >  Please post a trace as per http://www.linux-kvm.org/page/Tracing.
> here it goes:
> http://nelide.cz/nik/trace.dat.bz2

404

>
> >
> >  Please post /proc/mtrr and /proc/iomem.
> [root@virtualbox ~]# cat /proc/mtrr
> reg00: base=0x000000000 (    0MB), size=16384MB, count=1: write-back
> reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
> reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
> reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable
>
>

These seem fine - all your memory is write-back.

-- 
error compiling committee.c: too many arguments to function

Re: KVM got slow after adding more physical memory to host

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 941 bytes --]

> 404
ouch, dumb me :-/
http://nelide.cz/downloads/nik/trace.dat.bz2
sorry
n.

>
>>
>> >
>> >  Please post /proc/mtrr and /proc/iomem.
>> [root@virtualbox ~]# cat /proc/mtrr
>> reg00: base=0x000000000 (    0MB), size=16384MB, count=1: write-back
>> reg01: base=0x400000000 (16384MB), size= 4096MB, count=1: write-back
>> reg02: base=0x500000000 (20480MB), size= 1024MB, count=1: write-back
>> reg03: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable
>>
>>
>
> These seem fine - all your memory is write-back.
>
> -- 
> error compiling committee.c: too many arguments to function
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM got slow after adding more physical memory to host

[Avi Kivity]

On 09/06/2011 07:01 PM, Nikola Ciprich wrote:
> >  404
> ouch, dumb me :-/
> http://nelide.cz/downloads/nik/trace.dat.bz2
> sorry
> n.
>

         qemu-kvm-4618  [000] 13698.803169: kvm_emulate_insn:     
0:806edb12: rep outsw
         qemu-kvm-4618  [000] 13698.803170: kvm_pio:              
pio_write at 0x1f0 size 2 count 1
         qemu-kvm-4618  [000] 13698.803171: kvm_userspace_exit:   reason 
KVM_EXIT_IO (2)
         qemu-kvm-4618  [000] 13698.803186: kvm_entry:            vcpu 0
         qemu-kvm-4618  [000] 13698.803189: kvm_exit:             reason 
IO_INSTRUCTION rip 0x806edb12 info 1f00031 0

         qemu-kvm-4618  [000] 13698.803191: kvm_emulate_insn:     
0:806edb12: rep outsw
         qemu-kvm-4618  [000] 13698.803192: kvm_pio:              
pio_write at 0x1f0 size 2 count 1
         qemu-kvm-4618  [000] 13698.803193: kvm_userspace_exit:   reason 
KVM_EXIT_IO (2)
         qemu-kvm-4618  [000] 13698.803195: kvm_entry:            vcpu 0
         qemu-kvm-4618  [000] 13698.803198: kvm_exit:             reason 
IO_INSTRUCTION rip 0x806edb12 info 1f00031 0


One of your guests is using IDE PIO instead of DMA.  Switch it back to DMA.

Gleb, is rep/outsw our pio regression or was it another instruction?

-- 
error compiling committee.c: too many arguments to function

Re: KVM got slow after adding more physical memory to host

[Avi Kivity]

On 09/06/2011 07:06 PM, Avi Kivity wrote:
> On 09/06/2011 07:01 PM, Nikola Ciprich wrote:
>> >  404
>> ouch, dumb me :-/
>> http://nelide.cz/downloads/nik/trace.dat.bz2
>> sorry
>> n.
>>
>
>         qemu-kvm-4618  [000] 13698.803169: kvm_emulate_insn:     
> 0:806edb12: rep outsw
>         qemu-kvm-4618  [000] 13698.803170: kvm_pio:              
> pio_write at 0x1f0 size 2 count 1
>         qemu-kvm-4618  [000] 13698.803171: kvm_userspace_exit:   
> reason KVM_EXIT_IO (2)
>         qemu-kvm-4618  [000] 13698.803186: kvm_entry:            vcpu 0
>         qemu-kvm-4618  [000] 13698.803189: kvm_exit:             
> reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0
>
>         qemu-kvm-4618  [000] 13698.803191: kvm_emulate_insn:     
> 0:806edb12: rep outsw
>         qemu-kvm-4618  [000] 13698.803192: kvm_pio:              
> pio_write at 0x1f0 size 2 count 1
>         qemu-kvm-4618  [000] 13698.803193: kvm_userspace_exit:   
> reason KVM_EXIT_IO (2)
>         qemu-kvm-4618  [000] 13698.803195: kvm_entry:            vcpu 0
>         qemu-kvm-4618  [000] 13698.803198: kvm_exit:             
> reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0
>
>
> One of your guests is using IDE PIO instead of DMA.  Switch it back to 
> DMA.

Even better, switch it to virtio.

-- 
error compiling committee.c: too many arguments to function

Re: KVM got slow after adding more physical memory to host

[Avi Kivity]

On 09/06/2011 06:38 PM, Nikola Ciprich wrote:
> Hello Avi,
> thanks for quick reply!
>
> >  How many guests in there?
> two currently (one x86_64 centos, one winXP), with more, the box get even more unresponsive.
>

I don't really understand why the box is unresponsive.  What do top and 
'vmstat 1' say?

-- 
error compiling committee.c: too many arguments to function

Re: KVM got slow after adding more physical memory to host

[Gleb Natapov]

On Tue, Sep 06, 2011 at 07:06:34PM +0300, Avi Kivity wrote:
> On 09/06/2011 07:01 PM, Nikola Ciprich wrote:
> >>  404
> >ouch, dumb me :-/
> >http://nelide.cz/downloads/nik/trace.dat.bz2
> >sorry
> >n.
> >
> 
>         qemu-kvm-4618  [000] 13698.803169: kvm_emulate_insn:
> 0:806edb12: rep outsw
>         qemu-kvm-4618  [000] 13698.803170: kvm_pio:
> pio_write at 0x1f0 size 2 count 1
>         qemu-kvm-4618  [000] 13698.803171: kvm_userspace_exit:
> reason KVM_EXIT_IO (2)
>         qemu-kvm-4618  [000] 13698.803186: kvm_entry:            vcpu 0
>         qemu-kvm-4618  [000] 13698.803189: kvm_exit:
> reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0
> 
>         qemu-kvm-4618  [000] 13698.803191: kvm_emulate_insn:
> 0:806edb12: rep outsw
>         qemu-kvm-4618  [000] 13698.803192: kvm_pio:
> pio_write at 0x1f0 size 2 count 1
>         qemu-kvm-4618  [000] 13698.803193: kvm_userspace_exit:
> reason KVM_EXIT_IO (2)
>         qemu-kvm-4618  [000] 13698.803195: kvm_entry:            vcpu 0
>         qemu-kvm-4618  [000] 13698.803198: kvm_exit:
> reason IO_INSTRUCTION rip 0x806edb12 info 1f00031 0
> 
> 
> One of your guests is using IDE PIO instead of DMA.  Switch it back to DMA.
> 
> Gleb, is rep/outsw our pio regression or was it another instruction?
> 
It is rep/in-out. But why it became noticeable only after adding more
physical memory to the host is a mystery.

--
			Gleb.

Re: KVM got slow after adding more physical memory to host

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

OK, seems like Avi's right, at least the windows guest is not using DMA
though the question is, why did it got disabled and how can I reenable it,
I'm not experienced that much with XP :(


> It is rep/in-out. But why it became noticeable only after adding more
> physical memory to the host is a mystery.

now that I'm thinking about it, we had similar problem in the past, the windows guest
was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly
solved this issue, but maybe the DMA just got randomly enabled, and now after adding
the memory (and restarting of course), it just got disabled again?
but why could this be??


> 
> --
> 			Gleb.
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM got slow after adding more physical memory to host

[Avi Kivity]

On 09/06/2011 07:30 PM, Nikola Ciprich wrote:
> OK, seems like Avi's right, at least the windows guest is not using DMA
> though the question is, why did it got disabled

It can happen due to a timeout, perhaps a temporary load spike.

> and how can I reenable it,

Google knows.

> I'm not experienced that much with XP :(
>
>
> >  It is rep/in-out. But why it became noticeable only after adding more
> >  physical memory to the host is a mystery.
>
> now that I'm thinking about it, we had similar problem in the past, the windows guest
> was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly
> solved this issue, but maybe the DMA just got randomly enabled, and now after adding
> the memory (and restarting of course), it just got disabled again?
> but why could this be??
>

I don't think it's related to the memory.

-- 
error compiling committee.c: too many arguments to function

Re: KVM got slow after adding more physical memory to host - SOLVED

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 2522 bytes --]

Hello guys,

thanks to both of You for Your replies. The problem is solved,
exactly as Avi said, the DMA in windows got somehow disabled.
So this certainly was not related to adding the memory...

anyways, note for further generations:
in windows XP, the DMA usage can be checked in 
Device Manager->IDE ATA/ATAPI Controllers -> Properties->Advanced Settings
Current Transfer Mode must be Multi-Word DMA2 or something similar,
NOT PIO!

The way I enabled this, was to uninstall both primary and secondary controller
in device controller, then also uninstall Intel controller, and THEN rebooting (NOT sooner!)
after reboot, controllers got detected and installed again, with DMA properly enabled.
Note that when controller is in PIO mode, this is really a patience test, switching to DMA took
me like half an hour to complete, so slow the system was :-/

So thanks a lot for Your help once more.

have a nice day.

nik




On Tue, Sep 06, 2011 at 08:01:47PM +0300, Avi Kivity wrote:
> On 09/06/2011 07:30 PM, Nikola Ciprich wrote:
>> OK, seems like Avi's right, at least the windows guest is not using DMA
>> though the question is, why did it got disabled
>
> It can happen due to a timeout, perhaps a temporary load spike.
>
>> and how can I reenable it,
>
> Google knows.
>
>> I'm not experienced that much with XP :(
>>
>>
>> >  It is rep/in-out. But why it became noticeable only after adding more
>> >  physical memory to the host is a mystery.
>>
>> now that I'm thinking about it, we had similar problem in the past, the windows guest
>> was unusably slow, and the whole host was slow as well... upgrading to 0.14.1 seemingly
>> solved this issue, but maybe the DMA just got randomly enabled, and now after adding
>> the memory (and restarting of course), it just got disabled again?
>> but why could this be??
>>
>
> I don't think it's related to the memory.
>
> -- 
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM got slow after adding more physical memory to host - SOLVED

[Michael Tokarev]

On 06.09.2011 23:22, Nikola Ciprich wrote:
> Hello guys,
> 
> thanks to both of You for Your replies. The problem is solved,
> exactly as Avi said, the DMA in windows got somehow disabled.
> So this certainly was not related to adding the memory...
> 
> anyways, note for further generations:
> in windows XP, the DMA usage can be checked in 
> Device Manager->IDE ATA/ATAPI Controllers -> Properties->Advanced Settings
> Current Transfer Mode must be Multi-Word DMA2 or something similar,
> NOT PIO!
> 
> The way I enabled this, was to uninstall both primary and secondary controller
> in device controller, then also uninstall Intel controller, and THEN rebooting (NOT sooner!)
> after reboot, controllers got detected and installed again, with DMA properly enabled.
> Note that when controller is in PIO mode, this is really a patience test, switching to DMA took
> me like half an hour to complete, so slow the system was :-/

You can open regedit and search for "NoIDE" - it should find one key
with that name, with value "1" or "yes" - just delete it, there's no
need to go that route with removing/reinstalling device drivers.

The reason why it has been disabled is - most likely - due to some
timeout while handling i/o -- eg, when your host was loaded too much
or were swapping - you should be able  to find something about that
in windows event log.

/mjt