Guest performance is reduced after live migration

Guest performance is reduced after live migration

[Mark Petersen]

Hello KVM,

I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.

Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.

If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:

linux-image-2.6.32-32-server - 2.6.32-32.62
linux-image-2.6.32-38-server - 2.6.32-38.83
linux-image-2.6.32-43-server - 2.6.32-43.97
linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1
linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1
linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1 
linux-image-3.2-5 - mainline 3.2.5 kernel

I'm guess it's a libc issue (or at least a libc change causing the issue) as it doesn't seem to a be kernel related.

I'll try other distributions as a guest (probably Debian/Ubuntu) with newer libc's and see if I can pinpoint the issue to a libc version.  Any other ideas?

Shared disk backend is clvm/LV via FC to EMC SAN, not sure what else might be relevant.

Thanks,
Mark


______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

Re: Guest performance is reduced after live migration

[Marcelo Tosatti]

Can you describe more details of the test you are performing? 

If transparent hugepages are being used then there is the possibility
that there has been no time for khugepaged to back guest memory
with huge pages, in the destination (don't recall the interface for
retrieving number of hugepages for a given process, probably somewhere
in /proc/pid/).

On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote:
> Hello KVM,
> 
> I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.
> 
> Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.
> 
> If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:
> 
> linux-image-2.6.32-32-server - 2.6.32-32.62
> linux-image-2.6.32-38-server - 2.6.32-38.83
> linux-image-2.6.32-43-server - 2.6.32-43.97
> linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1
> linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1
> linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1 
> linux-image-3.2-5 - mainline 3.2.5 kernel
> 
> I'm guess it's a libc issue (or at least a libc change causing the issue) as it doesn't seem to a be kernel related.
> 
> I'll try other distributions as a guest (probably Debian/Ubuntu) with newer libc's and see if I can pinpoint the issue to a libc version.  Any other ideas?
> 
> Shared disk backend is clvm/LV via FC to EMC SAN, not sure what else might be relevant.
> 
> Thanks,
> Mark
> 
> 
> ______________________________________________
> 
> See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Guest performance is reduced after live migration

[Mark Petersen]

I don't think it's related to huge pages...

I was using phoronix-test-suite to run benchmarks.  The 'batch/compilation' group shows the slowdown for all tests, the 'batch/computation' show some performance degradation, but not nearly as significant.

You could probably easily test this way without phoronix -  Start a VM with almost nothing running.  Download mainline Linux kernel, compile.  This takes about 45 seconds in my case (72GB memory, 16 virtual CPUs, idle physical host running this VM.)  Run as many times as you want, still takes ~45 seconds.

Migrate to a new idle host, kernel compile now takes ~90 seconds, wait 3 hours (should give khugepaged a change to do its thing I imagine), kernel compiles still take 90 seconds.  Reboot virtual machine (run 'shutdown -r now', reboot, whatever.)  First compile will take ~45 seconds after reboot.  You don't even need to reset/destroy/shutdown the VM, just a reboot in the guest fixes the issue.

I'm going to test more with qemu-kvm 1.3 tomorrow as I have a new/dedicated lab setup and recently built the 1.3 code base.  I'd be happy to run any test that would help in diagnosing the real issue here, I'm just not sure how to best diagnose this issue.

Thanks,
Mark
 
-----Original Message-----

Can you describe more details of the test you are performing? 

If transparent hugepages are being used then there is the possibility that there has been no time for khugepaged to back guest memory with huge pages, in the destination (don't recall the interface for retrieving number of hugepages for a given process, probably somewhere in /proc/pid/).

On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote:
> Hello KVM,
> 
> I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.
> 
> Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.
> 
> If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:
> 
> linux-image-2.6.32-32-server - 2.6.32-32.62 
> linux-image-2.6.32-38-server - 2.6.32-38.83 
> linux-image-2.6.32-43-server - 2.6.32-43.97 
> linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1 
> linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1 
> linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1
> linux-image-3.2-5 - mainline 3.2.5 kernel
> 
> I'm guess it's a libc issue (or at least a libc change causing the issue) as it doesn't seem to a be kernel related.
> 
> I'll try other distributions as a guest (probably Debian/Ubuntu) with newer libc's and see if I can pinpoint the issue to a libc version.  Any other ideas?
> 
> Shared disk backend is clvm/LV via FC to EMC SAN, not sure what else might be relevant.
> 
> Thanks,
> Mark
> 
> 
> ______________________________________________
> 
> See http://www.peak6.com/email_disclaimer/ for terms and conditions 
> related to this email
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in the 
> body of a message to majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html

Re: Guest performance is reduced after live migration

[Marcelo Tosatti]

On Wed, Jan 02, 2013 at 11:56:11PM +0000, Mark Petersen wrote:
> I don't think it's related to huge pages...
> 
> I was using phoronix-test-suite to run benchmarks.  The 'batch/compilation' group shows the slowdown for all tests, the 'batch/computation' show some performance degradation, but not nearly as significant.

Huge pages in the host, for the qemu-kvm process, i mean.
Without huge pages backing guest memory in the host, 4k EPT TLB entries
will be used instead of 2MB EPT TLB entries.

> You could probably easily test this way without phoronix -  Start a VM with almost nothing running.  Download mainline Linux kernel, compile.  This takes about 45 seconds in my case (72GB memory, 16 virtual CPUs, idle physical host running this VM.)  Run as many times as you want, still takes ~45 seconds.
> 
> Migrate to a new idle host, kernel compile now takes ~90 seconds, wait 3 hours (should give khugepaged a change to do its thing I imagine),

Please verify its the case (by checking how much memory is backed by
hugepages).

http://www.mjmwired.net/kernel/Documentation/vm/transhuge.txt
"Monitoring Usage".


> kernel compiles still take 90 seconds.  Reboot virtual machine (run 'shutdown -r now', reboot, whatever.)  First compile will take ~45 seconds after reboot.  You don't even need to reset/destroy/shutdown the VM, just a reboot in the guest fixes the issue.

What is the qemu command line?

> I'm going to test more with qemu-kvm 1.3 tomorrow as I have a new/dedicated lab setup and recently built the 1.3 code base.  I'd be happy to run any test that would help in diagnosing the real issue here, I'm just not sure how to best diagnose this issue.
> 
> Thanks,
> Mark
>  
> -----Original Message-----
> 
> Can you describe more details of the test you are performing? 
> 
> If transparent hugepages are being used then there is the possibility that there has been no time for khugepaged to back guest memory with huge pages, in the destination (don't recall the interface for retrieving number of hugepages for a given process, probably somewhere in /proc/pid/).
> 
> On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote:
> > Hello KVM,
> > 
> > I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.
> > 
> > Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.
> > 
> > If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:
> > 
> > linux-image-2.6.32-32-server - 2.6.32-32.62 
> > linux-image-2.6.32-38-server - 2.6.32-38.83 
> > linux-image-2.6.32-43-server - 2.6.32-43.97 
> > linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1 
> > linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1 
> > linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1
> > linux-image-3.2-5 - mainline 3.2.5 kernel
> > 
> > I'm guess it's a libc issue (or at least a libc change causing the issue) as it doesn't seem to a be kernel related.
> > 
> > I'll try other distributions as a guest (probably Debian/Ubuntu) with newer libc's and see if I can pinpoint the issue to a libc version.  Any other ideas?
> > 
> > Shared disk backend is clvm/LV via FC to EMC SAN, not sure what else might be relevant.
> > 
> > Thanks,
> > Mark
> > 
> > 
> > ______________________________________________
> > 
> > See http://www.peak6.com/email_disclaimer/ for terms and conditions 
> > related to this email
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in the 
> > body of a message to majordomo@vger.kernel.org More majordomo info at  
> > http://vger.kernel.org/majordomo-info.html

RE: Guest performance is reduced after live migration

[Mark Petersen]

I believe I disabled huge pages on the guest and host previously, but I'll test a few scenarios and look at transparent hugepage usage specifically again over the next couple days and report back.


Below is a command line used for testing.

/usr/bin/kvm -> qemu-x86_64

/usr/bin/kvm -name one-483 -S -M pc-1.2 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid a844146a-0d72-a661-fe6c-cb6b7a4ba240 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-483.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/483/disk.0,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/483/disk.1,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:0a:02:02:4b,bus=pci.0,addr=0x3 -vnc 0.0.0.0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


-----Original Message-----
From: Marcelo Tosatti [mailto:mtosatti@redhat.com] 
Sent: Wednesday, January 02, 2013 6:49 PM
To: Mark Petersen
Cc: kvm@vger.kernel.org; Shouta.Uehara@jp.yokogawa.com
Subject: Re: Guest performance is reduced after live migration

On Wed, Jan 02, 2013 at 11:56:11PM +0000, Mark Petersen wrote:
> I don't think it's related to huge pages...
> 
> I was using phoronix-test-suite to run benchmarks.  The 'batch/compilation' group shows the slowdown for all tests, the 'batch/computation' show some performance degradation, but not nearly as significant.

Huge pages in the host, for the qemu-kvm process, i mean.
Without huge pages backing guest memory in the host, 4k EPT TLB entries will be used instead of 2MB EPT TLB entries.

> You could probably easily test this way without phoronix -  Start a VM with almost nothing running.  Download mainline Linux kernel, compile.  This takes about 45 seconds in my case (72GB memory, 16 virtual CPUs, idle physical host running this VM.)  Run as many times as you want, still takes ~45 seconds.
> 
> Migrate to a new idle host, kernel compile now takes ~90 seconds, wait 
> 3 hours (should give khugepaged a change to do its thing I imagine),

Please verify its the case (by checking how much memory is backed by hugepages).

http://www.mjmwired.net/kernel/Documentation/vm/transhuge.txt
"Monitoring Usage".


> kernel compiles still take 90 seconds.  Reboot virtual machine (run 'shutdown -r now', reboot, whatever.)  First compile will take ~45 seconds after reboot.  You don't even need to reset/destroy/shutdown the VM, just a reboot in the guest fixes the issue.

What is the qemu command line?

> I'm going to test more with qemu-kvm 1.3 tomorrow as I have a new/dedicated lab setup and recently built the 1.3 code base.  I'd be happy to run any test that would help in diagnosing the real issue here, I'm just not sure how to best diagnose this issue.
> 
> Thanks,
> Mark
>  
> -----Original Message-----
> 
> Can you describe more details of the test you are performing? 
> 
> If transparent hugepages are being used then there is the possibility that there has been no time for khugepaged to back guest memory with huge pages, in the destination (don't recall the interface for retrieving number of hugepages for a given process, probably somewhere in /proc/pid/).
> 
> On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote:
> > Hello KVM,
> > 
> > I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.
> > 
> > Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.
> > 
> > If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:
> > 
> > linux-image-2.6.32-32-server - 2.6.32-32.62 
> > linux-image-2.6.32-38-server - 2.6.32-38.83 
> > linux-image-2.6.32-43-server - 2.6.32-43.97 
> > linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1 
> > linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1 
> > linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1
> > linux-image-3.2-5 - mainline 3.2.5 kernel
> > 
> > I'm guess it's a libc issue (or at least a libc change causing the issue) as it doesn't seem to a be kernel related.
> > 
> > I'll try other distributions as a guest (probably Debian/Ubuntu) with newer libc's and see if I can pinpoint the issue to a libc version.  Any other ideas?
> > 
> > Shared disk backend is clvm/LV via FC to EMC SAN, not sure what else might be relevant.
> > 
> > Thanks,
> > Mark
> > 
> > 
> > ______________________________________________
> > 
> > See http://www.peak6.com/email_disclaimer/ for terms and conditions 
> > related to this email
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in 
> > the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at http://vger.kernel.org/majordomo-info.html

RE: Guest performance is reduced after live migration

[Mark Petersen]

Here's what I have found so far...

Ubuntu 10.04 performed within +/- 2% so I'm not including those results.    It seems that it's more of an issue of disk access, so I'm going to run some more disk specific benchmarks and I'll those results later.   I'd be happy to run any other perf tests that might help track down the problem as well.

Qemu command line:
/usr/bin/kvm -name one-3 -S -M pc-1.2 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid 3ebea329-cfbb-3447-0b49-b41e078a3ede -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-3.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/3/disk.0,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/3/disk.1,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:0a:64:02:fe,bus=pci.0,addr=0x3 -vnc 0.0.0.0:3,password -vga cirrus -incoming tcp:0.0.0.0:49155 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Initial Boot Benchmarks
===========================
Huge Page Usage
Physical Host: 2627584kB
QEMU Process: 2478080kB

Using phoronix pts/compuational
ImageMagick - 31.54s
Linux Kernel 3.1 - 43.91s
Mplayer - 30.49s
PHP - 22.25s


After Live Migration Benchmarks
===========================
Huge Page Usage
Physical Host: 3174400kB
QEMU Process: 3151872kB

Using phoronix pts/compilation
ImageMagick - 43.29s
Linux Kernel 3.1 - 76.67s
Mplayer - 45.41s
PHP - 29.1s


	
-----Original Message-----
From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Mark Petersen
Sent: Wednesday, January 02, 2013 7:32 PM
To: Marcelo Tosatti
Cc: kvm@vger.kernel.org; Shouta.Uehara@jp.yokogawa.com
Subject: RE: Guest performance is reduced after live migration

I believe I disabled huge pages on the guest and host previously, but I'll test a few scenarios and look at transparent hugepage usage specifically again over the next couple days and report back.


Below is a command line used for testing.

/usr/bin/kvm -> qemu-x86_64

/usr/bin/kvm -name one-483 -S -M pc-1.2 -cpu Westmere -enable-kvm -m 73728 -smp 16,sockets=2,cores=8,threads=1 -uuid a844146a-0d72-a661-fe6c-cb6b7a4ba240 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-483.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/one//datastores/0/483/disk.0,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/one//datastores/0/483/disk.1,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:0a:02:02:4b,bus=pci.0,addr=0x3 -vnc 0.0.0.0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


-----Original Message-----
From: Marcelo Tosatti [mailto:mtosatti@redhat.com]
Sent: Wednesday, January 02, 2013 6:49 PM
To: Mark Petersen
Cc: kvm@vger.kernel.org; Shouta.Uehara@jp.yokogawa.com
Subject: Re: Guest performance is reduced after live migration

On Wed, Jan 02, 2013 at 11:56:11PM +0000, Mark Petersen wrote:
> I don't think it's related to huge pages...
> 
> I was using phoronix-test-suite to run benchmarks.  The 'batch/compilation' group shows the slowdown for all tests, the 'batch/computation' show some performance degradation, but not nearly as significant.

Huge pages in the host, for the qemu-kvm process, i mean.
Without huge pages backing guest memory in the host, 4k EPT TLB entries will be used instead of 2MB EPT TLB entries.

> You could probably easily test this way without phoronix -  Start a VM with almost nothing running.  Download mainline Linux kernel, compile.  This takes about 45 seconds in my case (72GB memory, 16 virtual CPUs, idle physical host running this VM.)  Run as many times as you want, still takes ~45 seconds.
> 
> Migrate to a new idle host, kernel compile now takes ~90 seconds, wait
> 3 hours (should give khugepaged a change to do its thing I imagine),

Please verify its the case (by checking how much memory is backed by hugepages).

http://www.mjmwired.net/kernel/Documentation/vm/transhuge.txt
"Monitoring Usage".


> kernel compiles still take 90 seconds.  Reboot virtual machine (run 'shutdown -r now', reboot, whatever.)  First compile will take ~45 seconds after reboot.  You don't even need to reset/destroy/shutdown the VM, just a reboot in the guest fixes the issue.

What is the qemu command line?

> I'm going to test more with qemu-kvm 1.3 tomorrow as I have a new/dedicated lab setup and recently built the 1.3 code base.  I'd be happy to run any test that would help in diagnosing the real issue here, I'm just not sure how to best diagnose this issue.
> 
> Thanks,
> Mark
>  
> -----Original Message-----
> 
> Can you describe more details of the test you are performing? 
> 
> If transparent hugepages are being used then there is the possibility that there has been no time for khugepaged to back guest memory with huge pages, in the destination (don't recall the interface for retrieving number of hugepages for a given process, probably somewhere in /proc/pid/).
> 
> On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote:
> > Hello KVM,
> > 
> > I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.
> > 
> > Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.
> > 
> > If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:
> > 
> > linux-image-2.6.32-32-server - 2.6.32-32.62 
> > linux-image-2.6.32-38-server - 2.6.32-38.83 
> > linux-image-2.6.32-43-server - 2.6.32-43.97 
> > linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1 
> > linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1 
> > linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1
> > linux-image-3.2-5 - mainline 3.2.5 kernel
> > 
> > I'm guess it's a libc issue (or at least a libc change causing the issue) as it doesn't seem to a be kernel related.
> > 
> > I'll try other distributions as a guest (probably Debian/Ubuntu) with newer libc's and see if I can pinpoint the issue to a libc version.  Any other ideas?
> > 
> > Shared disk backend is clvm/LV via FC to EMC SAN, not sure what else might be relevant.
> > 
> > Thanks,
> > Mark
> > 
> > 
> > ______________________________________________
> > 
> > See http://www.peak6.com/email_disclaimer/ for terms and conditions 
> > related to this email
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in 
> > the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Guest performance is reduced after live migration

[Mark Petersen]

I used lmbench, and even simple test shows the difference here.  Simple read/writes (but not open/close?) seem to take much longer.  Perhaps this over simplifies things, but it's repeatable.   Any thoughts?  Other tests that might be interesting? 

ubuntu-12.04 - first boot
==========================================
Simple syscall: 0.0527 microseconds
Simple read: 0.1143 microseconds
Simple write: 0.0953 microseconds
Simple open/close: 1.0432 microseconds

ubuntu-12.04 - post live migration
==========================================
Simple syscall: 0.0621 microseconds
Simple read: 0.2485 microseconds
Simple write: 0.2252 microseconds
Simple open/close: 1.4626 microseconds

ubuntu-10.04 - first boot
==========================================
Simple syscall: 0.0524 microseconds
Simple read: 0.1135 microseconds
Simple write: 0.0972 microseconds
Simple open/close: 1.1261 microseconds


ubuntu-10.04 - post live migration
==========================================
Simple syscall: 0.0526 microseconds
Simple read: 0.1075 microseconds
Simple write: 0.0951 microseconds
Simple open/close: 1.0413 microseconds



Regards,
mark

> On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote:
> > Hello KVM,
> > 
> > I'm seeing something similar to this (http://thread.gmane.org/gmane.comp.emulators.kvm.devel/100592) as well when doing live migrations on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu-kvm 1.2 (improved performance for live migrations on guests with large memory guests is great!)  The default libvirt  0.9.8 and qemu-kvm 1.0 have the same issue.
> > 
> > Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.  I'm seeing similar issues with both virtio and ide bus.  Hugetblfs is not used, but transparent hugepages are.  Host machines are dual core Xeon E5-2660 processors.  I tried disabling EPT but that doesn't seem to make a difference so I don't think it's a requirement to reproduce.
> > 
> > If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kernels I don't seem to have the issue:
> > 
> > linux-image-2.6.32-32-server - 2.6.32-32.62 
> > linux-image-2.6.32-38-server - 2.6.32-38.83 
> > linux-image-2.6.32-43-server - 2.6.32-43.97 
> > linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1 
> > linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1 
> > linux-image-3.0.0-26-server  - 3.0.0-26.43~lucid1
> > linux-image-3.2-5 - mainline 3.2.5 kernel

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

Guest performance is reduced after live migration

[Shouta.Uehara]

Hello.

I have a problem with the performance of the guest Linux after live migration.
When I analyze the file I/O latency of the guest using LMbench3, the latency of
the guest on the destination host is about 2 times bigger than the guest on the
source host. As a result that I investigated it, this problem occurs if three
following conditions are right.

  1. Use the kernel version 2.6.38.6-26.fc15.x86_64 or later.
  2. Execute system calls for low level file I/O (read, write, open etc.) on the
     guest.
  3. Enable EPT.

Performance cannot decrease on other tests of LMbench3; numerical operation,
context switching, memory latencies etc.

Is this an existing problem ? What should I do to resolve it ?

---
The situation the problem occurs

CPU models     : Intel Xeon E3-1245 (Both hosts have same cpu.)
Host kernel    : 3.6.4
Host OS        : Fedora 16 64bit ver. (Both hosts have same OS.)
Guest OS       : Fedora 16 64bit ver., kernel 3.1.0-7.fc16.x86_64
QEMU version   : qemu-kvm 1.2.0
QEMU parameter : qemu-kvm -m 1024 -drive file=guest.qcow2,if=virtio \
                 -boot order=c,menu=on -k ja
                 (-incoming tcp:192.168.0.51:4444   set only destination)
QEMU command   : migrate -d tcp:192.168.0.51:4444
Analyses       : LMbench version 3

        [Guest on the source host]
        $ ./lat_syscall open
          Simple open/close: 1.4281 microseconds
        $ ./lat_syscall read
          Simple read: 0.1416 microseconds
        $ ./lat_syscall write
          Simple write: 0.1194 microseconds
        $ ./lat_fs -s 0
          0k      550     98726   247712
          1k      383     69751   193985
          4k      381     69348   193383
          10k     313     54369   155697

        [Guest on the source host]
        $ ./lat_syscall open
          Simple open/close: 1.6126 microseconds
        $ ./lat_syscall read
          Simple read: 0.2206 microseconds
        $ ./lat_syscall write
          Simple write: 0.1980 microseconds
        $ ./lat_fs -s 0
          0k      427     83252   243222
          1k      315     60530   191650
          4k      184     34535   190947
          10k     65      11922   152608
---

Thanks.

        Shota Uehara <Shouta.Uehara()jp.yokogawa.com>


-----
CONFIDENTIAL: This e-mail may contain information that is confidential or otherwise protected from disclosure and intended only for the party to whom it is addressed. If you are not the intended recipient, please notify the sender by return and delete this e-mail. You are hereby formally advised that any unauthorized use, disclosure or copying of this email is strictly prohibited and may be unlawful.

Re: Guest performance is reduced after live migration

[Xiao Guangrong]

Shouta,

Can it be reproduced if thp/hugetlbfs is disabled on both source and destination?

On 11/01/2012 08:12 AM, Shouta.Uehara@jp.yokogawa.com wrote:
> Hello.
> 
> I have a problem with the performance of the guest Linux after live migration.
> When I analyze the file I/O latency of the guest using LMbench3, the latency of
> the guest on the destination host is about 2 times bigger than the guest on the
> source host. As a result that I investigated it, this problem occurs if three
> following conditions are right.
> 
>   1. Use the kernel version 2.6.38.6-26.fc15.x86_64 or later.
>   2. Execute system calls for low level file I/O (read, write, open etc.) on the
>      guest.
>   3. Enable EPT.
> 
> Performance cannot decrease on other tests of LMbench3; numerical operation,
> context switching, memory latencies etc.
> 
> Is this an existing problem ? What should I do to resolve it ?
> 
> ---
> The situation the problem occurs
> 
> CPU models     : Intel Xeon E3-1245 (Both hosts have same cpu.)
> Host kernel    : 3.6.4
> Host OS        : Fedora 16 64bit ver. (Both hosts have same OS.)
> Guest OS       : Fedora 16 64bit ver., kernel 3.1.0-7.fc16.x86_64
> QEMU version   : qemu-kvm 1.2.0
> QEMU parameter : qemu-kvm -m 1024 -drive file=guest.qcow2,if=virtio \
>                  -boot order=c,menu=on -k ja
>                  (-incoming tcp:192.168.0.51:4444   set only destination)
> QEMU command   : migrate -d tcp:192.168.0.51:4444
> Analyses       : LMbench version 3
> 
>         [Guest on the source host]
>         $ ./lat_syscall open
>           Simple open/close: 1.4281 microseconds
>         $ ./lat_syscall read
>           Simple read: 0.1416 microseconds
>         $ ./lat_syscall write
>           Simple write: 0.1194 microseconds
>         $ ./lat_fs -s 0
>           0k      550     98726   247712
>           1k      383     69751   193985
>           4k      381     69348   193383
>           10k     313     54369   155697
> 
>         [Guest on the source host]
>         $ ./lat_syscall open
>           Simple open/close: 1.6126 microseconds
>         $ ./lat_syscall read
>           Simple read: 0.2206 microseconds
>         $ ./lat_syscall write
>           Simple write: 0.1980 microseconds
>         $ ./lat_fs -s 0
>           0k      427     83252   243222
>           1k      315     60530   191650
>           4k      184     34535   190947
>           10k     65      11922   152608
> ---
> 
> Thanks.
> 
>         Shota Uehara <Shouta.Uehara()jp.yokogawa.com>
> 
> 
> -----
> CONFIDENTIAL: This e-mail may contain information that is confidential or otherwise protected from disclosure and intended only for the party to whom it is addressed. If you are not the intended recipient, please notify the sender by return and delete this e-mail. You are hereby formally advised that any unauthorized use, disclosure or copying of this email is strictly prohibited and may be unlawful.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: Guest performance is reduced after live migration

[Shouta.Uehara]

> Can it be reproduced if thp/hugetlbfs is disabled on both source and
> destination?

Yes, the problem occurs even if thp/hugetlbfs is disabled.

Shota

> -----Original Message-----
> From: Xiao Guangrong [mailto:xiaoguangrong@linux.vnet.ibm.com]
> Sent: Thursday, November 01, 2012 1:45 PM
> To: Uehara, Shouta (Shouta.Uehara@jp.yokogawa.com)
> Cc: kvm@vger.kernel.org
> Subject: Re: Guest performance is reduced after live migration
>
> Shouta,
>
> Can it be reproduced if thp/hugetlbfs is disabled on both source and
> destination?
>
> On 11/01/2012 08:12 AM, Shouta.Uehara@jp.yokogawa.com wrote:
> > Hello.
> >
> > I have a problem with the performance of the guest Linux after live
> migration.
> > When I analyze the file I/O latency of the guest using LMbench3, the
> > latency of the guest on the destination host is about 2 times bigger
> > than the guest on the source host. As a result that I investigated it,
> > this problem occurs if three following conditions are right.
> >
> >   1. Use the kernel version 2.6.38.6-26.fc15.x86_64 or later.
> >   2. Execute system calls for low level file I/O (read, write, open etc.)
> on the
> >      guest.
> >   3. Enable EPT.
> >
> > Performance cannot decrease on other tests of LMbench3; numerical
> > operation, context switching, memory latencies etc.
> >
> > Is this an existing problem ? What should I do to resolve it ?
> >
> > ---
> > The situation the problem occurs
> >
> > CPU models     : Intel Xeon E3-1245 (Both hosts have same cpu.)
> > Host kernel    : 3.6.4
> > Host OS        : Fedora 16 64bit ver. (Both hosts have same OS.)
> > Guest OS       : Fedora 16 64bit ver., kernel 3.1.0-7.fc16.x86_64
> > QEMU version   : qemu-kvm 1.2.0
> > QEMU parameter : qemu-kvm -m 1024 -drive file=guest.qcow2,if=virtio \
> >                  -boot order=c,menu=on -k ja
> >                  (-incoming tcp:192.168.0.51:4444   set only
> destination)
> > QEMU command   : migrate -d tcp:192.168.0.51:4444
> > Analyses       : LMbench version 3
> >
> >         [Guest on the source host]
> >         $ ./lat_syscall open
> >           Simple open/close: 1.4281 microseconds
> >         $ ./lat_syscall read
> >           Simple read: 0.1416 microseconds
> >         $ ./lat_syscall write
> >           Simple write: 0.1194 microseconds
> >         $ ./lat_fs -s 0
> >           0k      550     98726   247712
> >           1k      383     69751   193985
> >           4k      381     69348   193383
> >           10k     313     54369   155697
> >
> >         [Guest on the source host]
> >         $ ./lat_syscall open
> >           Simple open/close: 1.6126 microseconds
> >         $ ./lat_syscall read
> >           Simple read: 0.2206 microseconds
> >         $ ./lat_syscall write
> >           Simple write: 0.1980 microseconds
> >         $ ./lat_fs -s 0
> >           0k      427     83252   243222
> >           1k      315     60530   191650
> >           4k      184     34535   190947
> >           10k     65      11922   152608
> > ---
> >
> > Thanks.
> >
> >         Shota Uehara <Shouta.Uehara()jp.yokogawa.com>

-----
CONFIDENTIAL: This e-mail may contain information that is confidential or otherwise protected from disclosure and intended only for the party to whom it is addressed. If you are not the intended recipient, please notify the sender by return and delete this e-mail. You are hereby formally advised that any unauthorized use, disclosure or copying of this email is strictly prohibited and may be unlawful.

RE: Guest performance is reduced after live migration

[Shouta.Uehara]

I've analysed the problem with migration using perf-events, and it is confirmed
cost of glibc is significantly increased.
I made a simple test code to execute read system call on the guest as follows,
and examined the performance from host and guest.

    fd = open("/dev/zero", O_RDONLY);
    while (1) { read(fd &ch, 1); }

---
[Source host]

$ sudo perf kvm --host --guest record -a -o src_host.perf sleep 10

Events: 10K cycles
 89.36%  qemu-system-x86  [unknown]                      [g] 0xffffffff810a36ef
  9.04%  qemu-system-x86  [unknown]                      [u] 0x3fd20e2020
  0.16%          swapper  [kernel.kallsyms]              [k] intel_idle
  0.11%            sleep  [kernel.kallsyms]              [k] page_fault

[Guest on source host]

$ perf record -o src_guest.perf ./loop_read

Events: 29K cpu-clock
    11.71%  loop_read  [kernel.kallsyms]  [k] system_call_after_swapgs
     9.58%  loop_read  libc-2.14.90.so    [.] __GI___libc_read
     6.92%  loop_read  [kernel.kallsyms]  [k] vfs_read
     5.53%  loop_read  [kernel.kallsyms]  [k] fsnotify

__GI___libc_read
         :        0000003fd20e2010 <__read>:
    2.31 :          3fd20e2010:   83 3d 1d 22 2d 00 00    cmpl   $0x0,0x2d221d(
    5.67 :          3fd20e2017:   75 10                   jne    3fd20e2029 <__
         :
         :        0000003fd20e2019 <__read_nocancel>:
    1.82 :          3fd20e2019:   b8 00 00 00 00          mov    $0x0,%eax
    0.00 :          3fd20e201e:   0f 05                   syscall
   87.78 :          3fd20e2020:   48 3d 01 f0 ff ff       cmp    $0xfffffffffff
    0.00 :          3fd20e2026:   73 31                   jae    3fd20e2059 <__
    2.42 :          3fd20e2028:   c3                      retq

[Destination host]

$ sudo perf kvm --host --guest record -a -o dst_host.perf sleep 10

Events: 10K cycles
 58.39%  qemu-system-x86  [unknown]                      [g] 0xffffffff810a3a6e
 40.14%  qemu-system-x86  [unknown]                      [u] 0x3fd20e2017
  0.13%      gnome-shell  nouveau_dri.so                 [.] 0xbd7c9
  0.11%          swapper  [kernel.kallsyms]              [k] intel_idle

[Guest on destination host]

$ perf record -o dst_guest.perf ./loop_read

Events: 29K cpu-clock
    41.95%  loop_read  libc-2.14.90.so    [.] __GI___libc_read
     7.90%  loop_read  [kernel.kallsyms]  [k] system_call_after_swapgs
     4.61%  loop_read  [kernel.kallsyms]  [k] vfs_read
     3.72%  loop_read  [kernel.kallsyms]  [k] fsnotify

__GI___libc_read
         :        0000003fd20e2010 <__read>:
    0.41 :          3fd20e2010:   83 3d 1d 22 2d 00 00    cmpl   $0x0,0x2d221d(
   86.10 :          3fd20e2017:   75 10                   jne    3fd20e2029 <__
         :
         :        0000003fd20e2019 <__read_nocancel>:
    0.33 :          3fd20e2019:   b8 00 00 00 00          mov    $0x0,%eax
    0.00 :          3fd20e201e:   0f 05                   syscall
   12.84 :          3fd20e2020:   48 3d 01 f0 ff ff       cmp    $0xfffffffffff
    0.00 :          3fd20e2026:   73 31                   jae    3fd20e2059 <__
    0.33 :          3fd20e2028:   c3                      retq

---

After the migration, "jne" instruction in __read became to take a long time.
This is due to increase the overhead of read system call. The performance was
not improved even if thp/hugetlb was disabled.
Does this problem happen only to me? Should I also ask other community about
this problem?

   Shota

> -----Original Message-----
> From: Xiao Guangrong [mailto:xiaoguangrong@linux.vnet.ibm.com]
> Sent: Thursday, November 01, 2012 1:45 PM
> To: Uehara, Shouta (Shouta.Uehara@jp.yokogawa.com)
> Cc: kvm@vger.kernel.org
> Subject: Re: Guest performance is reduced after live migration
>
> Shouta,
>
> Can it be reproduced if thp/hugetlbfs is disabled on both source and
> destination?
>
> On 11/01/2012 08:12 AM, Shouta.Uehara@jp.yokogawa.com wrote:
> > Hello.
> >
> > I have a problem with the performance of the guest Linux after live
> migration.
> > When I analyze the file I/O latency of the guest using LMbench3, the
> > latency of the guest on the destination host is about 2 times bigger
> > than the guest on the source host. As a result that I investigated it,
> > this problem occurs if three following conditions are right.
> >
> >   1. Use the kernel version 2.6.38.6-26.fc15.x86_64 or later.
> >   2. Execute system calls for low level file I/O (read, write, open etc.)
> on the
> >      guest.
> >   3. Enable EPT.
> >
> > Performance cannot decrease on other tests of LMbench3; numerical
> > operation, context switching, memory latencies etc.
> >
> > Is this an existing problem ? What should I do to resolve it ?
> >
> > ---
> > The situation the problem occurs
> >
> > CPU models     : Intel Xeon E3-1245 (Both hosts have same cpu.)
> > Host kernel    : 3.6.4
> > Host OS        : Fedora 16 64bit ver. (Both hosts have same OS.)
> > Guest OS       : Fedora 16 64bit ver., kernel 3.1.0-7.fc16.x86_64
> > QEMU version   : qemu-kvm 1.2.0
> > QEMU parameter : qemu-kvm -m 1024 -drive file=guest.qcow2,if=virtio \
> >                  -boot order=c,menu=on -k ja
> >                  (-incoming tcp:192.168.0.51:4444   set only
> destination)
> > QEMU command   : migrate -d tcp:192.168.0.51:4444
> > Analyses       : LMbench version 3
> >
> >         [Guest on the source host]
> >         $ ./lat_syscall open
> >           Simple open/close: 1.4281 microseconds
> >         $ ./lat_syscall read
> >           Simple read: 0.1416 microseconds
> >         $ ./lat_syscall write
> >           Simple write: 0.1194 microseconds
> >         $ ./lat_fs -s 0
> >           0k      550     98726   247712
> >           1k      383     69751   193985
> >           4k      381     69348   193383
> >           10k     313     54369   155697
> >
> >         [Guest on the source host]
> >         $ ./lat_syscall open
> >           Simple open/close: 1.6126 microseconds
> >         $ ./lat_syscall read
> >           Simple read: 0.2206 microseconds
> >         $ ./lat_syscall write
> >           Simple write: 0.1980 microseconds
> >         $ ./lat_fs -s 0
> >           0k      427     83252   243222
> >           1k      315     60530   191650
> >           4k      184     34535   190947
> >           10k     65      11922   152608
> > ---
> >
> > Thanks.
> >
> >         Shota Uehara <Shouta.Uehara()jp.yokogawa.com>
> >
> >
> > -----
> > CONFIDENTIAL: This e-mail may contain information that is confidential
> or otherwise protected from disclosure and intended only for the party to
> whom it is addressed. If you are not the intended recipient, please notify
> the sender by return and delete this e-mail. You are hereby formally advised
> that any unauthorized use, disclosure or copying of this email is strictly
> prohibited and may be unlawful.
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in the
> > body of a message to majordomo@vger.kernel.org More majordomo info at
> > http://vger.kernel.org/majordomo-info.html
> >
> >


-----
CONFIDENTIAL: This e-mail may contain information that is confidential or otherwise protected from disclosure and intended only for the party to whom it is addressed. If you are not the intended recipient, please notify the sender by return and delete this e-mail. You are hereby formally advised that any unauthorized use, disclosure or copying of this email is strictly prohibited and may be unlawful.

RE: Guest performance is reduced after live migration

[Shouta.Uehara]

Dear all

I continue watching a mailing list whether a similar problem is reported because a problem does not seem to happen to others.
Any information, however small, would be appreciated.

regards

> -----Original Message-----
> From: Uehara, Shouta (Shouta.Uehara@jp.yokogawa.com)
> Sent: Friday, November 09, 2012 6:52 PM
> To: 'kvm@vger.kernel.org'; 'Xiao Guangrong
> (xiaoguangrong@linux.vnet.ibm.com)'
> Subject: RE: Guest performance is reduced after live migration
>
> I've analysed the problem with migration using perf-events, and it is
> confirmed cost of glibc is significantly increased.
> I made a simple test code to execute read system call on the guest as follows,
> and examined the performance from host and guest.
>
>     fd = open("/dev/zero", O_RDONLY);
>     while (1) { read(fd &ch, 1); }
>
> ---
> [Source host]
>
> $ sudo perf kvm --host --guest record -a -o src_host.perf sleep 10
>
> Events: 10K cycles
>  89.36%  qemu-system-x86  [unknown]                      [g]
> 0xffffffff810a36ef
>   9.04%  qemu-system-x86  [unknown]                      [u]
> 0x3fd20e2020
>   0.16%          swapper  [kernel.kallsyms]              [k] intel_idle
>   0.11%            sleep  [kernel.kallsyms]              [k] page_fault
>
> [Guest on source host]
>
> $ perf record -o src_guest.perf ./loop_read
>
> Events: 29K cpu-clock
>     11.71%  loop_read  [kernel.kallsyms]  [k] system_call_after_swapgs
>      9.58%  loop_read  libc-2.14.90.so    [.] __GI___libc_read
>      6.92%  loop_read  [kernel.kallsyms]  [k] vfs_read
>      5.53%  loop_read  [kernel.kallsyms]  [k] fsnotify
>
> __GI___libc_read
>          :        0000003fd20e2010 <__read>:
>     2.31 :          3fd20e2010:   83 3d 1d 22 2d 00 00    cmpl
> $0x0,0x2d221d(
>     5.67 :          3fd20e2017:   75 10                   jne
> 3fd20e2029 <__
>          :
>          :        0000003fd20e2019 <__read_nocancel>:
>     1.82 :          3fd20e2019:   b8 00 00 00 00          mov
> $0x0,%eax
>     0.00 :          3fd20e201e:   0f 05                   syscall
>    87.78 :          3fd20e2020:   48 3d 01 f0 ff ff       cmp
> $0xfffffffffff
>     0.00 :          3fd20e2026:   73 31                   jae
> 3fd20e2059 <__
>     2.42 :          3fd20e2028:   c3                      retq
>
> [Destination host]
>
> $ sudo perf kvm --host --guest record -a -o dst_host.perf sleep 10
>
> Events: 10K cycles
>  58.39%  qemu-system-x86  [unknown]                      [g]
> 0xffffffff810a3a6e
>  40.14%  qemu-system-x86  [unknown]                      [u]
> 0x3fd20e2017
>   0.13%      gnome-shell  nouveau_dri.so                 [.] 0xbd7c9
>   0.11%          swapper  [kernel.kallsyms]              [k] intel_idle
>
> [Guest on destination host]
>
> $ perf record -o dst_guest.perf ./loop_read
>
> Events: 29K cpu-clock
>     41.95%  loop_read  libc-2.14.90.so    [.] __GI___libc_read
>      7.90%  loop_read  [kernel.kallsyms]  [k] system_call_after_swapgs
>      4.61%  loop_read  [kernel.kallsyms]  [k] vfs_read
>      3.72%  loop_read  [kernel.kallsyms]  [k] fsnotify
>
> __GI___libc_read
>          :        0000003fd20e2010 <__read>:
>     0.41 :          3fd20e2010:   83 3d 1d 22 2d 00 00    cmpl
> $0x0,0x2d221d(
>    86.10 :          3fd20e2017:   75 10                   jne
> 3fd20e2029 <__
>          :
>          :        0000003fd20e2019 <__read_nocancel>:
>     0.33 :          3fd20e2019:   b8 00 00 00 00          mov
> $0x0,%eax
>     0.00 :          3fd20e201e:   0f 05                   syscall
>    12.84 :          3fd20e2020:   48 3d 01 f0 ff ff       cmp
> $0xfffffffffff
>     0.00 :          3fd20e2026:   73 31                   jae
> 3fd20e2059 <__
>     0.33 :          3fd20e2028:   c3                      retq
>
> ---
>
> After the migration, "jne" instruction in __read became to take a long time.
> This is due to increase the overhead of read system call. The performance
> was not improved even if thp/hugetlb was disabled.
> Does this problem happen only to me? Should I also ask other community about
> this problem?
>
>    Shota
>
> > -----Original Message-----
> > From: Xiao Guangrong [mailto:xiaoguangrong@linux.vnet.ibm.com]
> > Sent: Thursday, November 01, 2012 1:45 PM
> > To: Uehara, Shouta (Shouta.Uehara@jp.yokogawa.com)
> > Cc: kvm@vger.kernel.org
> > Subject: Re: Guest performance is reduced after live migration
> >
> > Shouta,
> >
> > Can it be reproduced if thp/hugetlbfs is disabled on both source and
> > destination?
> >
> > On 11/01/2012 08:12 AM, Shouta.Uehara@jp.yokogawa.com wrote:
> > > Hello.
> > >
> > > I have a problem with the performance of the guest Linux after live
> > migration.
> > > When I analyze the file I/O latency of the guest using LMbench3, the
> > > latency of the guest on the destination host is about 2 times bigger
> > > than the guest on the source host. As a result that I investigated
> > > it, this problem occurs if three following conditions are right.
> > >
> > >   1. Use the kernel version 2.6.38.6-26.fc15.x86_64 or later.
> > >   2. Execute system calls for low level file I/O (read, write, open
> > > etc.)
> > on the
> > >      guest.
> > >   3. Enable EPT.
> > >
> > > Performance cannot decrease on other tests of LMbench3; numerical
> > > operation, context switching, memory latencies etc.
> > >
> > > Is this an existing problem ? What should I do to resolve it ?
> > >
> > > ---
> > > The situation the problem occurs
> > >
> > > CPU models     : Intel Xeon E3-1245 (Both hosts have same cpu.)
> > > Host kernel    : 3.6.4
> > > Host OS        : Fedora 16 64bit ver. (Both hosts have same OS.)
> > > Guest OS       : Fedora 16 64bit ver., kernel 3.1.0-7.fc16.x86_64
> > > QEMU version   : qemu-kvm 1.2.0
> > > QEMU parameter : qemu-kvm -m 1024 -drive file=guest.qcow2,if=virtio
> \
> > >                  -boot order=c,menu=on -k ja
> > >                  (-incoming tcp:192.168.0.51:4444   set only
> > destination)
> > > QEMU command   : migrate -d tcp:192.168.0.51:4444
> > > Analyses       : LMbench version 3
> > >
> > >         [Guest on the source host]
> > >         $ ./lat_syscall open
> > >           Simple open/close: 1.4281 microseconds
> > >         $ ./lat_syscall read
> > >           Simple read: 0.1416 microseconds
> > >         $ ./lat_syscall write
> > >           Simple write: 0.1194 microseconds
> > >         $ ./lat_fs -s 0
> > >           0k      550     98726   247712
> > >           1k      383     69751   193985
> > >           4k      381     69348   193383
> > >           10k     313     54369   155697
> > >
> > >         [Guest on the source host]
> > >         $ ./lat_syscall open
> > >           Simple open/close: 1.6126 microseconds
> > >         $ ./lat_syscall read
> > >           Simple read: 0.2206 microseconds
> > >         $ ./lat_syscall write
> > >           Simple write: 0.1980 microseconds
> > >         $ ./lat_fs -s 0
> > >           0k      427     83252   243222
> > >           1k      315     60530   191650
> > >           4k      184     34535   190947
> > >           10k     65      11922   152608
> > > ---
> > >
> > > Thanks.
> > >
> > >         Shota Uehara <Shouta.Uehara()jp.yokogawa.com>


-----
CONFIDENTIAL: This e-mail may contain information that is confidential or otherwise protected from disclosure and intended only for the party to whom it is addressed. If you are not the intended recipient, please notify the sender by return and delete this e-mail. You are hereby formally advised that any unauthorized use, disclosure or copying of this email is strictly prohibited and may be unlawful.

Re: Guest performance is reduced after live migration

[Xiao Guangrong]

On 11/21/2012 09:25 AM, Shouta.Uehara@jp.yokogawa.com wrote:
> Dear allI
> 
> I continue watching a mailing list whether a similar problem is reported because a problem does not seem to happen to others.
> Any information, however small, would be appreciated.
> 

I am digging into it, but did not get useful hint so far...