KVM performance Java server/MySQL...

KVM performance Java server/MySQL...

[Erik Brakkee]

Hi,


We have been benchmarking a java server application (java 6 update 29)
that requires a mysql database. The scenario is quite simple. We open a
web page which displays a lot of search results. To get the content of the
page one big query is done with many smaller queries to retrieve the data.
The test from the java side is single threaded.

We have used the following deployment scenarios:
1. JBoss in VM, MySql in separate VM
2. JBoss in VM, MySQL native
3. JBoss native, MySQL in vm.
4. JBoss native and MySQL native on the same physical machine
5. JBoss and MySQL virtualized on the same VM.

What we see is that the performance (time to execute) is practically the
same for all scenarios (approx. 30 seconds), except for scenario 4 that
takes approx. 21 seconds. This difference is quite large and contrasts
many other test on the internet and other benchmarks we did previously.

We have tried pinning the VMs, turning hyperthreading off, varying the CPU
model (including host-passthrough), but this did not have any significant
impact.

The hardware on which we are running is a dual socket E5-2650 machine with
64 GB memory. The server is a Dell poweredge R720 server with SAS disks,
RAID controller with battery backup (writeback cache). Transparent huge
pages is turned on.

We are at a loss to explain the differences in the test. In particular, we
would have expected the least performance when both were running
virtualized and we would have expected a better performance when JBoss and
MySQL were running virtualized in the same VM as compared to JBoss and
MySQL both running in different virtual machines. It looks like we are
dealing with multiple issues here and not just one.

Right now we have a 30% penalty for running virtualized which is too much
for us; 10% would be allright. What would you suggest to do to
troubleshoot this further?

Cheers
  Erik

Re: KVM performance Java server/MySQL...

[Gleb Natapov]

On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote:
> Hi,
> 
> 
> We have been benchmarking a java server application (java 6 update 29)
> that requires a mysql database. The scenario is quite simple. We open a
> web page which displays a lot of search results. To get the content of the
> page one big query is done with many smaller queries to retrieve the data.
> The test from the java side is single threaded.
> 
> We have used the following deployment scenarios:
> 1. JBoss in VM, MySql in separate VM
> 2. JBoss in VM, MySQL native
> 3. JBoss native, MySQL in vm.
> 4. JBoss native and MySQL native on the same physical machine
> 5. JBoss and MySQL virtualized on the same VM.
> 
> What we see is that the performance (time to execute) is practically the
> same for all scenarios (approx. 30 seconds), except for scenario 4 that
> takes approx. 21 seconds. This difference is quite large and contrasts
> many other test on the internet and other benchmarks we did previously.
> 
> We have tried pinning the VMs, turning hyperthreading off, varying the CPU
> model (including host-passthrough), but this did not have any significant
> impact.
> 
> The hardware on which we are running is a dual socket E5-2650 machine with
> 64 GB memory. The server is a Dell poweredge R720 server with SAS disks,
> RAID controller with battery backup (writeback cache). Transparent huge
> pages is turned on.
> 
> We are at a loss to explain the differences in the test. In particular, we
> would have expected the least performance when both were running
> virtualized and we would have expected a better performance when JBoss and
> MySQL were running virtualized in the same VM as compared to JBoss and
> MySQL both running in different virtual machines. It looks like we are
> dealing with multiple issues here and not just one.
> 
> Right now we have a 30% penalty for running virtualized which is too much
> for us; 10% would be allright. What would you suggest to do to
> troubleshoot this further?
>

What is you kernel/qemu versions and command line you are using to start
a VM?
 
--
			Gleb.

Re: KVM performance Java server/MySQL...

[Erik Brakkee]

[-- Attachment #1: Type: text/plain, Size: 6287 bytes --]

<quote who="Gleb Natapov">
> On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote:
>> Hi,
>>
>>
>> We have been benchmarking a java server application (java 6 update 29)
>> that requires a mysql database. The scenario is quite simple. We open a
>> web page which displays a lot of search results. To get the content of
>> the
>> page one big query is done with many smaller queries to retrieve the
>> data.
>> The test from the java side is single threaded.
>>
>> We have used the following deployment scenarios:
>> 1. JBoss in VM, MySql in separate VM
>> 2. JBoss in VM, MySQL native
>> 3. JBoss native, MySQL in vm.
>> 4. JBoss native and MySQL native on the same physical machine
>> 5. JBoss and MySQL virtualized on the same VM.
>>
>> What we see is that the performance (time to execute) is practically the
>> same for all scenarios (approx. 30 seconds), except for scenario 4 that
>> takes approx. 21 seconds. This difference is quite large and contrasts
>> many other test on the internet and other benchmarks we did previously.
>>
>> We have tried pinning the VMs, turning hyperthreading off, varying the
>> CPU
>> model (including host-passthrough), but this did not have any
>> significant
>> impact.
>>
>> The hardware on which we are running is a dual socket E5-2650 machine
>> with
>> 64 GB memory. The server is a Dell poweredge R720 server with SAS disks,
>> RAID controller with battery backup (writeback cache). Transparent huge
>> pages is turned on.
>>
>> We are at a loss to explain the differences in the test. In particular,
>> we
>> would have expected the least performance when both were running
>> virtualized and we would have expected a better performance when JBoss
>> and
>> MySQL were running virtualized in the same VM as compared to JBoss and
>> MySQL both running in different virtual machines. It looks like we are
>> dealing with multiple issues here and not just one.
>>
>> Right now we have a 30% penalty for running virtualized which is too
>> much
>> for us; 10% would be allright. What would you suggest to do to
>> troubleshoot this further?
>>
>
> What is you kernel/qemu versions and command line you are using to start
> a VM?

Centos 6.3, 2.6.32-279.22.1.el6.x86_64
> rpm -qf /usr/libexec/qemu-kvm
qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64

The guest is also running centos 6.3 with the same settings. Settings that
can influence Java performance (such as transparent huge pages) are turned
on on both the host and guest (see the remark on hugepages below).

The command-line is as follows:

/usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 8192 -mem-prealloc
-mem-path /hugepages/libvirt/qemu -smp 4,sockets=4,cores=1,threads=1 -name
master-data05-v50 -uuid 79ddd84d-937e-357b-8e57-c7f487dc3464 -nodefconfig
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/master-data05-v50.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-drive
file=/dev/raid5/v50disk1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/dev/vg_raid1/v50disk2,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/dev/raid5/v50disk3,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2
-drive
file=/var/data/images/configdisks/v50/configdisk.img,if=none,id=drive-virtio-disk25,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk25,id=virtio-disk25
-netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:01:50,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6



This virtual machine has three virtio disks, and one file based disk. The
last disk is about 100MB in size and is used only during startup (contains
configurationd data for initializing the vm) and is only read and never
written. It has one CDrom which is not used. It also uses old-style
hugepages. Apparently this did not have any significant effect on
performance over transparent hugepages (as would be expected). We
configured these old style hugepages just to rule out any issue with
transparent hugepages.

Initially we got 30% performance penalty with 16 processors, but in the
current setting of using 4 processors we see a reduced performance penalty
of 15-20%. Also on the physical host, we are not running the numad daemon
at the moment. Also, we tried disabling hyperthreading in the host's BIOS
but the measurements do not change significantly.

The IO scheduler on the host and on the guest is CFS. We also tried with
deadline scheduler on the host but this did not make any measurable
difference. We did not try no-op on the host. Additionally, the test is a
read-only test. It queries the database for data. We are excluding the
first measurement, so disk IO cannot be a problem as far as the database
is concerned. For the application (running on JBoss), the IO is basically
limited to network IO of a single web page and it writes a few lines in a
server log file.

This means that we can effectively rule out disk IO as a problem I think,
which limits it to CPU and memory issues. I also epxiermented with the cpu
mode 'host-passthrough' but that also did not result in an improvement.

Other things I could try is to remove the 'memballoon' configuration from
the VM.

Do you have any idea what could be the cause of this? Do you think it is a
numa problem? Memory related perhaps? Do you have any suggestions on
things to configure and/or measure?

Cheers
  Erik


>
> --
> 			Gleb.
>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v50.xml --]
[-- Type: text/xml; name="v50.xml", Size: 6387 bytes --]

<domain type='kvm' id='1'>                                                           
  <name>master-data05-v50</name>                                                     
  <uuid>79ddd84d-937e-357b-8e57-c7f487dc3464</uuid>                                  
  <memory unit='KiB'>8388608</memory>                                                
  <currentMemory unit='KiB'>8388608</currentMemory>                                  
  <memoryBacking>                                                                    
    <hugepages/>                                                                     
  </memoryBacking>                                                                   
  <vcpu placement='static'>4</vcpu>                                                  
  <os>                                                                               
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>                               
    <boot dev='cdrom'/>                                                              
    <boot dev='hd'/>                                                                 
  </os>                                                                              
  <features>                                                                         
    <acpi/>                                                                          
    <apic/>                                                                          
  </features>                                                                        
  <clock offset='utc'/>                                                              
  <on_poweroff>destroy</on_poweroff>                                                 
  <on_reboot>restart</on_reboot>                                                     
  <on_crash>restart</on_crash>                                                       
  <devices>                                                                          
    <emulator>/usr/libexec/qemu-kvm</emulator>                                       
    <disk type='file' device='cdrom'>                                                
      <driver name='qemu' type='raw'/>                                               
      <target dev='hdc' bus='ide'/>                                                  
      <readonly/>                                                                    
      <alias name='ide0-1-0'/>                                                       
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>             
    </disk>                                                                          
    <disk type='block' device='disk'>                                                
      <driver name='qemu' type='raw' cache='none' io='native'/>                      
      <source dev='/dev/raid5/v50disk1'/>                                            
      <target dev='vda' bus='virtio'/>                                               
      <alias name='virtio-disk0'/>                                                   
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>    
    </disk>                                                                          
    <disk type='block' device='disk'>                                                
      <driver name='qemu' type='raw' cache='none' io='native'/>                      
      <source dev='/dev/vg_raid1/v50disk2'/>                                         
      <target dev='vdb' bus='virtio'/>                                               
      <alias name='virtio-disk1'/>                                                   
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>    
    </disk>                                                                          
    <disk type='block' device='disk'>                                                
      <driver name='qemu' type='raw' cache='none' io='native'/>                      
      <source dev='/dev/raid5/v50disk3'/>                                            
      <target dev='vdc' bus='virtio'/>                                               
      <alias name='virtio-disk2'/>                                                   
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>    
    </disk>                                                                          
    <disk type='file' device='disk'>                                                 
      <driver name='qemu' type='raw' cache='none'/>                                  
      <source file='/var/data/images/configdisks/v50/configdisk.img'/>               
      <target dev='vdz' bus='virtio'/>
      <alias name='virtio-disk25'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:00:01:50'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
</domain>

Re: KVM performance Java server/MySQL...

[Erik Brakkee]

<quote who="Erik Brakkee">
> The IO scheduler on the host and on the guest is CFS. We also tried with
> deadline scheduler on the host but this did not make any measurable
> difference. We did not try no-op on the host.

I mean of course that we did not try no-op on the guest (not on the host).

Re: KVM performance Java server/MySQL...

[Erik Brakkee]

<quote who="Erik Brakkee">
> <quote who="Gleb Natapov">
>> On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote:
>>> Hi,
>>>
>>>
>>> We have been benchmarking a java server application (java 6 update 29)
>>> that requires a mysql database. The scenario is quite simple. We open a
>>> web page which displays a lot of search results. To get the content of
>>> the
>>> page one big query is done with many smaller queries to retrieve the
>>> data.
>>> The test from the java side is single threaded.
>>>
>>> We have used the following deployment scenarios:
>>> 1. JBoss in VM, MySql in separate VM
>>> 2. JBoss in VM, MySQL native
>>> 3. JBoss native, MySQL in vm.
>>> 4. JBoss native and MySQL native on the same physical machine
>>> 5. JBoss and MySQL virtualized on the same VM.
>>>
>>> What we see is that the performance (time to execute) is practically
>>> the
>>> same for all scenarios (approx. 30 seconds), except for scenario 4 that
>>> takes approx. 21 seconds. This difference is quite large and contrasts
>>> many other test on the internet and other benchmarks we did previously.
>>>
>>> We have tried pinning the VMs, turning hyperthreading off, varying the
>>> CPU
>>> model (including host-passthrough), but this did not have any
>>> significant
>>> impact.
>>>
>>> The hardware on which we are running is a dual socket E5-2650 machine
>>> with
>>> 64 GB memory. The server is a Dell poweredge R720 server with SAS
>>> disks,
>>> RAID controller with battery backup (writeback cache). Transparent huge
>>> pages is turned on.
>>>
>>> We are at a loss to explain the differences in the test. In particular,
>>> we
>>> would have expected the least performance when both were running
>>> virtualized and we would have expected a better performance when JBoss
>>> and
>>> MySQL were running virtualized in the same VM as compared to JBoss and
>>> MySQL both running in different virtual machines. It looks like we are
>>> dealing with multiple issues here and not just one.
>>>
>>> Right now we have a 30% penalty for running virtualized which is too
>>> much
>>> for us; 10% would be allright. What would you suggest to do to
>>> troubleshoot this further?
>>>
>>
>> What is you kernel/qemu versions and command line you are using to start
>> a VM?
>
> Centos 6.3, 2.6.32-279.22.1.el6.x86_64
>> rpm -qf /usr/libexec/qemu-kvm
> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
>
> The guest is also running centos 6.3 with the same settings. Settings that
> can influence Java performance (such as transparent huge pages) are turned
> on on both the host and guest (see the remark on hugepages below).
>
> The command-line is as follows:
>
> /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 8192 -mem-prealloc
> -mem-path /hugepages/libvirt/qemu -smp 4,sockets=4,cores=1,threads=1 -name
> master-data05-v50 -uuid 79ddd84d-937e-357b-8e57-c7f487dc3464 -nodefconfig
> -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/master-data05-v50.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> -drive
> file=/dev/raid5/v50disk1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
> -drive
> file=/dev/vg_raid1/v50disk2,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1
> -drive
> file=/dev/raid5/v50disk3,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2
> -drive
> file=/var/data/images/configdisks/v50/configdisk.img,if=none,id=drive-virtio-disk25,format=raw,cache=none
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk25,id=virtio-disk25
> -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:01:50,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
>
>
>
> This virtual machine has three virtio disks, and one file based disk. The
> last disk is about 100MB in size and is used only during startup (contains
> configurationd data for initializing the vm) and is only read and never
> written. It has one CDrom which is not used. It also uses old-style
> hugepages. Apparently this did not have any significant effect on
> performance over transparent hugepages (as would be expected). We
> configured these old style hugepages just to rule out any issue with
> transparent hugepages.
>
> Initially we got 30% performance penalty with 16 processors, but in the
> current setting of using 4 processors we see a reduced performance penalty
> of 15-20%. Also on the physical host, we are not running the numad daemon
> at the moment. Also, we tried disabling hyperthreading in the host's BIOS
> but the measurements do not change significantly.
>
> The IO scheduler on the host and on the guest is CFS. We also tried with
> deadline scheduler on the host but this did not make any measurable
> difference. We did not try no-op on the host. Additionally, the test is a
> read-only test. It queries the database for data. We are excluding the
> first measurement, so disk IO cannot be a problem as far as the database
> is concerned. For the application (running on JBoss), the IO is basically
> limited to network IO of a single web page and it writes a few lines in a
> server log file.
>
> This means that we can effectively rule out disk IO as a problem I think,
> which limits it to CPU and memory issues. I also epxiermented with the cpu
> mode 'host-passthrough' but that also did not result in an improvement.
>
> Other things I could try is to remove the 'memballoon' configuration from
> the VM.
>
> Do you have any idea what could be the cause of this? Do you think it is a
> numa problem? Memory related perhaps? Do you have any suggestions on
> things to configure and/or measure?
>

One of the changes I made was disabling the memballoon configuration for
the virtual machine. This also helped performance a bit.

In addition, I have been comparing the server configurations now in detail
of the physical and virtual server and I found one difference in the mysql
configuration which was the use of the tcmalloc memory allocator from the
gperftools-libs RPM in mysql on the physical host.

After updating the virtual machine to also use this memory allocator, the
difference between virtual and physical is now only a 10% penalty for
virtual, which is an acceptable overhead. This is comparing the scenario
with mysql and jboss colocated on the same virtual/physical machine.

Nevertheless, we still welcome any suggestions for improving this further.

Is 10% an acceptable penalty for virtualization or do you think we should
be able to get more out of it?

Re: KVM performance Java server/MySQL...

[David Cruz]

Other optimizations people are testing out there.

- use "nohz=off" in the kernel loading line y menu.lst
- Disable Cgroups completely. Using cgclear, and turning off cgred
cg-config daemons.

And from a Personal point of view, we've always tried to use MySQL in
a different server from JBoss.
99% of the times is far better for performance and tuning.

David

2013/2/8 Erik Brakkee <erik@brakkee.org>:
> <quote who="Erik Brakkee">
>> <quote who="Gleb Natapov">
>>> On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote:
>>>> Hi,
>>>>
>>>>
>>>> We have been benchmarking a java server application (java 6 update 29)
>>>> that requires a mysql database. The scenario is quite simple. We open a
>>>> web page which displays a lot of search results. To get the content of
>>>> the
>>>> page one big query is done with many smaller queries to retrieve the
>>>> data.
>>>> The test from the java side is single threaded.
>>>>
>>>> We have used the following deployment scenarios:
>>>> 1. JBoss in VM, MySql in separate VM
>>>> 2. JBoss in VM, MySQL native
>>>> 3. JBoss native, MySQL in vm.
>>>> 4. JBoss native and MySQL native on the same physical machine
>>>> 5. JBoss and MySQL virtualized on the same VM.
>>>>
>>>> What we see is that the performance (time to execute) is practically
>>>> the
>>>> same for all scenarios (approx. 30 seconds), except for scenario 4 that
>>>> takes approx. 21 seconds. This difference is quite large and contrasts
>>>> many other test on the internet and other benchmarks we did previously.
>>>>
>>>> We have tried pinning the VMs, turning hyperthreading off, varying the
>>>> CPU
>>>> model (including host-passthrough), but this did not have any
>>>> significant
>>>> impact.
>>>>
>>>> The hardware on which we are running is a dual socket E5-2650 machine
>>>> with
>>>> 64 GB memory. The server is a Dell poweredge R720 server with SAS
>>>> disks,
>>>> RAID controller with battery backup (writeback cache). Transparent huge
>>>> pages is turned on.
>>>>
>>>> We are at a loss to explain the differences in the test. In particular,
>>>> we
>>>> would have expected the least performance when both were running
>>>> virtualized and we would have expected a better performance when JBoss
>>>> and
>>>> MySQL were running virtualized in the same VM as compared to JBoss and
>>>> MySQL both running in different virtual machines. It looks like we are
>>>> dealing with multiple issues here and not just one.
>>>>
>>>> Right now we have a 30% penalty for running virtualized which is too
>>>> much
>>>> for us; 10% would be allright. What would you suggest to do to
>>>> troubleshoot this further?
>>>>
>>>
>>> What is you kernel/qemu versions and command line you are using to start
>>> a VM?
>>
>> Centos 6.3, 2.6.32-279.22.1.el6.x86_64
>>> rpm -qf /usr/libexec/qemu-kvm
>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
>>
>> The guest is also running centos 6.3 with the same settings. Settings that
>> can influence Java performance (such as transparent huge pages) are turned
>> on on both the host and guest (see the remark on hugepages below).
>>
>> The command-line is as follows:
>>
>> /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 8192 -mem-prealloc
>> -mem-path /hugepages/libvirt/qemu -smp 4,sockets=4,cores=1,threads=1 -name
>> master-data05-v50 -uuid 79ddd84d-937e-357b-8e57-c7f487dc3464 -nodefconfig
>> -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/master-data05-v50.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>> if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
>> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
>> -drive
>> file=/dev/raid5/v50disk1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
>> -device
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
>> -drive
>> file=/dev/vg_raid1/v50disk2,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
>> -device
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1
>> -drive
>> file=/dev/raid5/v50disk3,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
>> -device
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2
>> -drive
>> file=/var/data/images/configdisks/v50/configdisk.img,if=none,id=drive-virtio-disk25,format=raw,cache=none
>> -device
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk25,id=virtio-disk25
>> -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:01:50,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
>>
>>
>>
>> This virtual machine has three virtio disks, and one file based disk. The
>> last disk is about 100MB in size and is used only during startup (contains
>> configurationd data for initializing the vm) and is only read and never
>> written. It has one CDrom which is not used. It also uses old-style
>> hugepages. Apparently this did not have any significant effect on
>> performance over transparent hugepages (as would be expected). We
>> configured these old style hugepages just to rule out any issue with
>> transparent hugepages.
>>
>> Initially we got 30% performance penalty with 16 processors, but in the
>> current setting of using 4 processors we see a reduced performance penalty
>> of 15-20%. Also on the physical host, we are not running the numad daemon
>> at the moment. Also, we tried disabling hyperthreading in the host's BIOS
>> but the measurements do not change significantly.
>>
>> The IO scheduler on the host and on the guest is CFS. We also tried with
>> deadline scheduler on the host but this did not make any measurable
>> difference. We did not try no-op on the host. Additionally, the test is a
>> read-only test. It queries the database for data. We are excluding the
>> first measurement, so disk IO cannot be a problem as far as the database
>> is concerned. For the application (running on JBoss), the IO is basically
>> limited to network IO of a single web page and it writes a few lines in a
>> server log file.
>>
>> This means that we can effectively rule out disk IO as a problem I think,
>> which limits it to CPU and memory issues. I also epxiermented with the cpu
>> mode 'host-passthrough' but that also did not result in an improvement.
>>
>> Other things I could try is to remove the 'memballoon' configuration from
>> the VM.
>>
>> Do you have any idea what could be the cause of this? Do you think it is a
>> numa problem? Memory related perhaps? Do you have any suggestions on
>> things to configure and/or measure?
>>
>
> One of the changes I made was disabling the memballoon configuration for
> the virtual machine. This also helped performance a bit.
>
> In addition, I have been comparing the server configurations now in detail
> of the physical and virtual server and I found one difference in the mysql
> configuration which was the use of the tcmalloc memory allocator from the
> gperftools-libs RPM in mysql on the physical host.
>
> After updating the virtual machine to also use this memory allocator, the
> difference between virtual and physical is now only a 10% penalty for
> virtual, which is an acceptable overhead. This is comparing the scenario
> with mysql and jboss colocated on the same virtual/physical machine.
>
> Nevertheless, we still welcome any suggestions for improving this further.
>
> Is 10% an acceptable penalty for virtualization or do you think we should
> be able to get more out of it?
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM performance Java server/MySQL...

[Erik Brakkee]

<quote who="David Cruz">
> Other optimizations people are testing out there.
>
> - use "nohz=off" in the kernel loading line y menu.lst
> - Disable Cgroups completely. Using cgclear, and turning off cgred
> cg-config daemons.
>
> And from a Personal point of view, we've always tried to use MySQL in
> a different server from JBoss.
> 99% of the times is far better for performance and tuning.

I am still having problems. Running mysql and JBoss on different VMs is
significantly slower, but that appears to be a pure application issue
relating to the network overhead in a huge amount of queries. This is of
course not an issue for this mailing list.

However, I got a performance degradation of running the same test with
MySQL and JBoss colocated and increasing the CPU count from 4 to 8. Then I
did various tricks to improve performance. Pinning the processes did not
have a significant effect. What appeared to have an effect was to remove
memory ballooning.

But now I get inconsistent results, varying from 21 seconds to 27 seconds.
The 21 figure is acceptable but the 27 figure is not. The tests are being
done on new hardware (see previous posts), with basically the only thing
running the virtual machine.

Summary of configuration:
- host and guest: centos 6.3, transparent hugepages
- vm: cpu mode is host-passthrough, pinning to the cores of one processor,
  removed tablet, sound, and USB devices
- host: hyperthreading switched off in BIOS

Do you have any idea what this could be? I expect it is somehow NUMA
related, but how would I troubleshoot this? Ideally I would like to make
sure the entire VM runs on one CPU and allocates memory from that CPU and
never moves (or both CPU and memory move together). I saw some
presentations on the internet about NUMA work being done for linux. Do you
have any suggestions?

My domain.xml is given below:

<domain type='kvm' id='37'>
  <name>master-data05-v50</name>
  <uuid>79ddd84d-937e-357b-8e57-c7f487dc3464</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='2'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='6'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='12'/>
    <vcpupin vcpu='7' cpuset='14'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/raid5/v50disk1'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/vg_system/v50disk2'/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/raid5/v50disk3'/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/mydata/images/configdisks/v50/configdisk.img'/>
      <target dev='vdz' bus='virtio'/>
      <alias name='virtio-disk25'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:00:01:50'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
  </devices>
  <seclabel type='none'/>
</domain>

The output of virsh capabilities:

<capabilities>

  <host>
    <uuid>44454c4c-4600-1031-8043-b9c04f47354a</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>SandyBridge</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='8' threads='1'/>
      <feature name='pdpe1gb'/>
      <feature name='osxsave'/>
      <feature name='tsc-deadline'/>
      <feature name='dca'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_disk/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='2'>
        <cell id='0'>
          <cpus num='8'>
            <cpu id='0'/>
            <cpu id='2'/>
            <cpu id='4'/>
            <cpu id='6'/>
            <cpu id='8'/>
            <cpu id='10'/>
            <cpu id='12'/>
            <cpu id='14'/>
          </cpus>
        </cell>
        <cell id='1'>
          <cpus num='8'>
            <cpu id='1'/>
            <cpu id='3'/>
            <cpu id='5'/>
            <cpu id='7'/>
            <cpu id='9'/>
            <cpu id='11'/>
            <cpu id='13'/>
            <cpu id='15'/>
          </cpus>
        </cell>
      </cells>
    </topology>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.3.0</machine>
      <machine canonical='rhel6.3.0'>pc</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <pae/>
      <nonpae/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.3.0</machine>
      <machine canonical='rhel6.3.0'>pc</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

The output of virsh vcpuinfo master-data05-v50:

VCPU:           0
CPU:            0
State:          running
CPU time:       119.4s
CPU Affinity:   y---------------

VCPU:           1
CPU:            2
State:          running
CPU time:       78.9s
CPU Affinity:   --y-------------

VCPU:           2
CPU:            4
State:          running
CPU time:       74.4s
CPU Affinity:   ----y-----------

VCPU:           3
CPU:            6
State:          running
CPU time:       61.6s
CPU Affinity:   ------y---------

VCPU:           4
CPU:            8
State:          running
CPU time:       91.4s
CPU Affinity:   --------y-------

VCPU:           5
CPU:            10
State:          running
CPU time:       67.4s
CPU Affinity:   ----------y-----

VCPU:           6
CPU:            12
State:          running
CPU time:       52.8s
CPU Affinity:   ------------y---

VCPU:           7
CPU:            14
State:          running
CPU time:       66.8s
CPU Affinity:   --------------y-

I also checked x2apic using 'grep -i x2apic /var/log/messages*'. this
shows x2apic is not working on the host but it is working on the guest.
Should I enable it on the host as well as suggested by the output?

On the host I get:

/var/log/messages-20130210:Feb  8 08:51:00 localhost kernel: Your BIOS has
requested that x2apic be disabled.
/var/log/messages-20130210:Feb  8 08:51:00 localhost kernel: Use
'intremap=no_x2apic_optout' to override BIOS request.
/var/log/messages-20130210:Feb  8 08:51:00 localhost kernel: x2apic not
enabled, IRQ remapping is in xapic mode

On the guest I get:

/var/log/messages-20130213:Feb 13 02:00:19 localhost kernel: Setting APIC
routing to physical x2apic
/var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Enabling x2apic
/var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Enabled x2apic
/var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Setting APIC
routing to physical x2apic

>
> David
>

Re: KVM performance Java server/MySQL...

[Erik Brakkee]

<quote who="Erik Brakkee">
> <quote who="David Cruz">
>> Other optimizations people are testing out there.
>>
>> - use "nohz=off" in the kernel loading line y menu.lst
>> - Disable Cgroups completely. Using cgclear, and turning off cgred
>> cg-config daemons.

I also tried this option but it did not have a significant effect and
degraded performance a bit, although it is difficult to tell because of
the variations in the measured times.

On a positive note, I have managed to get a significant performance
improvement by checking the BIOS. When we got the server, it was
configured to use the Performance Per Watt profile. All current tests were
run with CPU scaling enabled (and also recognized by the linux kernel).
What I did to get increased performance was to enable the 'Performance'
profile instead and let the hardware do the CPU scaling (this is what I
understand that the option means).

Now the measured times have dropped from a consistent 19 seconds on the
physical host to a consistent 10 seconds on the physical host. On the
virtual host, the variable times from 21-27 seconds have now turned into
consistent times of approximately 13.5 seconds. This is with the nohz=off
kernel option. With the nohz=off option removed, I get consistent times
that are mostly a bit under 13 seconds (12.8 upwards). These results are
for a VM with 4 virtual CPUs that are also pinned to cores of the first
CPU.

>From the looks of it, the kernel we are using does not fully understand
the CPU it is dealing with. Also, I am doubtful that we ever saw the
actual CPU frequences when doing 'cat /proc/cpuinfo'.

Also, positively, increasing the VCPU count from 4 to 8 and also not
pinning them does not deteriorate performance. In fact, it looks like it
increases performance even a bit, leading to an overhead of 20%
approximately.

All in all, the performance results are acceptable now looking at the
absolute figures. In fact, I am quite happy with this because it means we
can continue to take this setup in production.  However, there still
appears to be a 20% penalty for virtualization in our specific case.

Do you have any ideas on how to get to the bottom of this? Being able to
squeeze the last 20% out of it would really make us even more happy (my
day is already made with this BIOS setting).

Any ideas?



> However, I got a performance degradation of running the same test with
> MySQL and JBoss colocated and increasing the CPU count from 4 to 8. Then I
> did various tricks to improve performance. Pinning the processes did not
> have a significant effect. What appeared to have an effect was to remove
> memory ballooning.
>
> But now I get inconsistent results, varying from 21 seconds to 27 seconds.
> The 21 figure is acceptable but the 27 figure is not. The tests are being
> done on new hardware (see previous posts), with basically the only thing
> running the virtual machine.
>
> Summary of configuration:
> - host and guest: centos 6.3, transparent hugepages
> - vm: cpu mode is host-passthrough, pinning to the cores of one processor,
>   removed tablet, sound, and USB devices
> - host: hyperthreading switched off in BIOS
>
> Do you have any idea what this could be? I expect it is somehow NUMA
> related, but how would I troubleshoot this? Ideally I would like to make
> sure the entire VM runs on one CPU and allocates memory from that CPU and
> never moves (or both CPU and memory move together). I saw some
> presentations on the internet about NUMA work being done for linux. Do you
> have any suggestions?
>
> My domain.xml is given below:
>
> <domain type='kvm' id='37'>
>   <name>master-data05-v50</name>
>   <uuid>79ddd84d-937e-357b-8e57-c7f487dc3464</uuid>
>   <memory unit='KiB'>8388608</memory>
>   <currentMemory unit='KiB'>8388608</currentMemory>
>   <vcpu placement='static'>8</vcpu>
>   <cputune>
>     <vcpupin vcpu='0' cpuset='0'/>
>     <vcpupin vcpu='1' cpuset='2'/>
>     <vcpupin vcpu='2' cpuset='4'/>
>     <vcpupin vcpu='3' cpuset='6'/>
>     <vcpupin vcpu='4' cpuset='8'/>
>     <vcpupin vcpu='5' cpuset='10'/>
>     <vcpupin vcpu='6' cpuset='12'/>
>     <vcpupin vcpu='7' cpuset='14'/>
>   </cputune>
>   <os>
>     <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
>     <boot dev='cdrom'/>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>   </features>
>   <cpu mode='host-passthrough'>
>   </cpu>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/libexec/qemu-kvm</emulator>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <target dev='hdc' bus='ide'/>
>       <readonly/>
>       <alias name='ide0-1-0'/>
>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>     </disk>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/raid5/v50disk1'/>
>       <target dev='vda' bus='virtio'/>
>       <alias name='virtio-disk0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
>     </disk>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/vg_system/v50disk2'/>
>       <target dev='vdb' bus='virtio'/>
>       <alias name='virtio-disk1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
>     </disk>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source dev='/dev/raid5/v50disk3'/>
>       <target dev='vdc' bus='virtio'/>
>       <alias name='virtio-disk2'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>     </disk>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source file='/var/mydata/images/configdisks/v50/configdisk.img'/>
>       <target dev='vdz' bus='virtio'/>
>       <alias name='virtio-disk25'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </disk>
>     <controller type='usb' index='0'>
>       <alias name='usb0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
>     </controller>
>     <controller type='ide' index='0'>
>       <alias name='ide0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:00:01:50'/>
>       <source bridge='br0'/>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <source path='/dev/pts/1'/>
>       <target port='0'/>
>       <alias name='serial0'/>
>     </serial>
>     <console type='pty' tty='/dev/pts/1'>
>       <source path='/dev/pts/1'/>
>       <target type='serial' port='0'/>
>       <alias name='serial0'/>
>     </console>
>     <input type='mouse' bus='ps2'/>
>     <graphics type='vnc' port='5900' autoport='yes'/>
>     <video>
>       <model type='cirrus' vram='9216' heads='1'/>
>       <alias name='video0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>   </devices>
>   <seclabel type='none'/>
> </domain>
>
> The output of virsh capabilities:
>
> <capabilities>
>
>   <host>
>     <uuid>44454c4c-4600-1031-8043-b9c04f47354a</uuid>
>     <cpu>
>       <arch>x86_64</arch>
>       <model>SandyBridge</model>
>       <vendor>Intel</vendor>
>       <topology sockets='1' cores='8' threads='1'/>
>       <feature name='pdpe1gb'/>
>       <feature name='osxsave'/>
>       <feature name='tsc-deadline'/>
>       <feature name='dca'/>
>       <feature name='pdcm'/>
>       <feature name='xtpr'/>
>       <feature name='tm2'/>
>       <feature name='est'/>
>       <feature name='smx'/>
>       <feature name='vmx'/>
>       <feature name='ds_cpl'/>
>       <feature name='monitor'/>
>       <feature name='dtes64'/>
>       <feature name='pbe'/>
>       <feature name='tm'/>
>       <feature name='ht'/>
>       <feature name='ss'/>
>       <feature name='acpi'/>
>       <feature name='ds'/>
>       <feature name='vme'/>
>     </cpu>
>     <power_management>
>       <suspend_disk/>
>     </power_management>
>     <migration_features>
>       <live/>
>       <uri_transports>
>         <uri_transport>tcp</uri_transport>
>       </uri_transports>
>     </migration_features>
>     <topology>
>       <cells num='2'>
>         <cell id='0'>
>           <cpus num='8'>
>             <cpu id='0'/>
>             <cpu id='2'/>
>             <cpu id='4'/>
>             <cpu id='6'/>
>             <cpu id='8'/>
>             <cpu id='10'/>
>             <cpu id='12'/>
>             <cpu id='14'/>
>           </cpus>
>         </cell>
>         <cell id='1'>
>           <cpus num='8'>
>             <cpu id='1'/>
>             <cpu id='3'/>
>             <cpu id='5'/>
>             <cpu id='7'/>
>             <cpu id='9'/>
>             <cpu id='11'/>
>             <cpu id='13'/>
>             <cpu id='15'/>
>           </cpus>
>         </cell>
>       </cells>
>     </topology>
>   </host>
>
>   <guest>
>     <os_type>hvm</os_type>
>     <arch name='i686'>
>       <wordsize>32</wordsize>
>       <emulator>/usr/libexec/qemu-kvm</emulator>
>       <machine>rhel6.3.0</machine>
>       <machine canonical='rhel6.3.0'>pc</machine>
>       <machine>rhel6.2.0</machine>
>       <machine>rhel6.1.0</machine>
>       <machine>rhel6.0.0</machine>
>       <machine>rhel5.5.0</machine>
>       <machine>rhel5.4.4</machine>
>       <machine>rhel5.4.0</machine>
>       <domain type='qemu'>
>       </domain>
>       <domain type='kvm'>
>         <emulator>/usr/libexec/qemu-kvm</emulator>
>       </domain>
>     </arch>
>     <features>
>       <cpuselection/>
>       <deviceboot/>
>       <pae/>
>       <nonpae/>
>       <acpi default='on' toggle='yes'/>
>       <apic default='on' toggle='no'/>
>     </features>
>   </guest>
>
>   <guest>
>     <os_type>hvm</os_type>
>     <arch name='x86_64'>
>       <wordsize>64</wordsize>
>       <emulator>/usr/libexec/qemu-kvm</emulator>
>       <machine>rhel6.3.0</machine>
>       <machine canonical='rhel6.3.0'>pc</machine>
>       <machine>rhel6.2.0</machine>
>       <machine>rhel6.1.0</machine>
>       <machine>rhel6.0.0</machine>
>       <machine>rhel5.5.0</machine>
>       <machine>rhel5.4.4</machine>
>       <machine>rhel5.4.0</machine>
>       <domain type='qemu'>
>       </domain>
>       <domain type='kvm'>
>         <emulator>/usr/libexec/qemu-kvm</emulator>
>       </domain>
>     </arch>
>     <features>
>       <cpuselection/>
>       <deviceboot/>
>       <acpi default='on' toggle='yes'/>
>       <apic default='on' toggle='no'/>
>     </features>
>   </guest>
>
> </capabilities>
>
> The output of virsh vcpuinfo master-data05-v50:
>
> VCPU:           0
> CPU:            0
> State:          running
> CPU time:       119.4s
> CPU Affinity:   y---------------
>
> VCPU:           1
> CPU:            2
> State:          running
> CPU time:       78.9s
> CPU Affinity:   --y-------------
>
> VCPU:           2
> CPU:            4
> State:          running
> CPU time:       74.4s
> CPU Affinity:   ----y-----------
>
> VCPU:           3
> CPU:            6
> State:          running
> CPU time:       61.6s
> CPU Affinity:   ------y---------
>
> VCPU:           4
> CPU:            8
> State:          running
> CPU time:       91.4s
> CPU Affinity:   --------y-------
>
> VCPU:           5
> CPU:            10
> State:          running
> CPU time:       67.4s
> CPU Affinity:   ----------y-----
>
> VCPU:           6
> CPU:            12
> State:          running
> CPU time:       52.8s
> CPU Affinity:   ------------y---
>
> VCPU:           7
> CPU:            14
> State:          running
> CPU time:       66.8s
> CPU Affinity:   --------------y-
>
> I also checked x2apic using 'grep -i x2apic /var/log/messages*'. this
> shows x2apic is not working on the host but it is working on the guest.
> Should I enable it on the host as well as suggested by the output?
>
> On the host I get:
>
> /var/log/messages-20130210:Feb  8 08:51:00 localhost kernel: Your BIOS has
> requested that x2apic be disabled.
> /var/log/messages-20130210:Feb  8 08:51:00 localhost kernel: Use
> 'intremap=no_x2apic_optout' to override BIOS request.
> /var/log/messages-20130210:Feb  8 08:51:00 localhost kernel: x2apic not
> enabled, IRQ remapping is in xapic mode
>
> On the guest I get:
>
> /var/log/messages-20130213:Feb 13 02:00:19 localhost kernel: Setting APIC
> routing to physical x2apic
> /var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Enabling
> x2apic
> /var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Enabled
> x2apic
> /var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Setting APIC
> routing to physical x2apic
>
>>
>> David
>>
>
>