* [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists @ 2012-02-20 18:17 Peter Lieven 2012-02-20 18:40 ` Gleb Natapov 0 siblings, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-20 18:17 UTC (permalink / raw) To: qemu-devel, kvm Hi, I came a across an issue with a Windows 7 (32-bit) as well as with a Windows 2008 R2 (64-bit) guest. If I transfer a file from the VM via CIFS or FTP to a remote machine, i get very poor read performance (around 13MB/s). The VM peaks at 100% cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat efer_reload 0 0 exits 2260976 79620 fpu_reload 6197 11 halt_exits 114734 5011 halt_wakeup 111195 4876 host_state_reload 1499659 60962 hypercalls 0 0 insn_emulation 1577325 58488 insn_emulation_fail 0 0 invlpg 0 0 io_exits 943949 40249 irq_exits 108679 5434 irq_injections 236545 10788 irq_window 7606 246 largepages 672 5 mmio_exits 460020 16082 mmu_cache_miss 119 0 mmu_flooded 0 0 mmu_pde_zapped 0 0 mmu_pte_updated 0 0 mmu_pte_write 13474 9 mmu_recycled 0 0 mmu_shadow_zapped 141 0 mmu_unsync 0 0 nmi_injections 0 0 nmi_window 0 0 pf_fixed 22803 35 pf_guest 0 0 remote_tlb_flush 239 2 request_irq 0 0 signal_exits 0 0 tlb_flush 20933 0 If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s throughput. The kvm_stats look a lot more sane. efer_reload 0 0 exits 6132004 17931 fpu_reload 19863 3 halt_exits 264961 3083 halt_wakeup 236468 2959 host_state_reload 1104468 3104 hypercalls 0 0 insn_emulation 1417443 7518 insn_emulation_fail 0 0 invlpg 0 0 io_exits 869380 2795 irq_exits 253501 2362 irq_injections 616967 6804 irq_window 201186 2161 largepages 1019 0 mmio_exits 205268 0 mmu_cache_miss 192 0 mmu_flooded 0 0 mmu_pde_zapped 0 0 mmu_pte_updated 0 0 mmu_pte_write 7440546 0 mmu_recycled 0 0 mmu_shadow_zapped 259 0 mmu_unsync 0 0 nmi_injections 0 0 nmi_window 0 0 pf_fixed 38529 30 pf_guest 0 0 remote_tlb_flush 761 1 request_irq 0 0 signal_exits 0 0 tlb_flush 0 0 I use virtio-net (with vhost-net) and virtio-blk. I tried disabling hpet (which basically illiminated the mmio_exits, but does not increase performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 ) - no improvement. My commandline: /usr/bin/qemu-kvm-1.0 -netdev type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name lieven-win7-vc -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet What further information is needed to debug this further? Thanks, Peter ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 18:17 [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists Peter Lieven @ 2012-02-20 18:40 ` Gleb Natapov 2012-02-20 19:04 ` Gleb Natapov 2012-02-20 19:15 ` Peter Lieven 0 siblings, 2 replies; 19+ messages in thread From: Gleb Natapov @ 2012-02-20 18:40 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: > Hi, > > I came a across an issue with a Windows 7 (32-bit) as well as with a > Windows 2008 R2 (64-bit) guest. > > If I transfer a file from the VM via CIFS or FTP to a remote machine, > i get very poor read performance (around 13MB/s). The VM peaks at 100% > cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat > > efer_reload 0 0 > exits 2260976 79620 > fpu_reload 6197 11 > halt_exits 114734 5011 > halt_wakeup 111195 4876 > host_state_reload 1499659 60962 > hypercalls 0 0 > insn_emulation 1577325 58488 > insn_emulation_fail 0 0 > invlpg 0 0 > io_exits 943949 40249 Hmm, too many of those. > irq_exits 108679 5434 > irq_injections 236545 10788 > irq_window 7606 246 > largepages 672 5 > mmio_exits 460020 16082 > mmu_cache_miss 119 0 > mmu_flooded 0 0 > mmu_pde_zapped 0 0 > mmu_pte_updated 0 0 > mmu_pte_write 13474 9 > mmu_recycled 0 0 > mmu_shadow_zapped 141 0 > mmu_unsync 0 0 > nmi_injections 0 0 > nmi_window 0 0 > pf_fixed 22803 35 > pf_guest 0 0 > remote_tlb_flush 239 2 > request_irq 0 0 > signal_exits 0 0 > tlb_flush 20933 0 > > If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s > throughput. The kvm_stats look a lot more sane. > > efer_reload 0 0 > exits 6132004 17931 > fpu_reload 19863 3 > halt_exits 264961 3083 > halt_wakeup 236468 2959 > host_state_reload 1104468 3104 > hypercalls 0 0 > insn_emulation 1417443 7518 > insn_emulation_fail 0 0 > invlpg 0 0 > io_exits 869380 2795 > irq_exits 253501 2362 > irq_injections 616967 6804 > irq_window 201186 2161 > largepages 1019 0 > mmio_exits 205268 0 > mmu_cache_miss 192 0 > mmu_flooded 0 0 > mmu_pde_zapped 0 0 > mmu_pte_updated 0 0 > mmu_pte_write 7440546 0 > mmu_recycled 0 0 > mmu_shadow_zapped 259 0 > mmu_unsync 0 0 > nmi_injections 0 0 > nmi_window 0 0 > pf_fixed 38529 30 > pf_guest 0 0 > remote_tlb_flush 761 1 > request_irq 0 0 > signal_exits 0 0 > tlb_flush 0 0 > > I use virtio-net (with vhost-net) and virtio-blk. I tried disabling > hpet (which basically illiminated the mmio_exits, but does not > increase > performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 > ) - no improvement. > > My commandline: > /usr/bin/qemu-kvm-1.0 -netdev > type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on > -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native > -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name > lieven-win7-vc -boot order=dc,menu=off -k de -pidfile > /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu > host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet > > What further information is needed to debug this further? > Which kernel version (looks like something recent)? Which host CPU (looks like something old)? Which Windows' virtio drivers are you using? Take a trace like described here http://www.linux-kvm.org/page/Tracing (with -no-hpet please). Try to use -cpu host,+x2apic. It may help Linux guest performance. -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 18:40 ` Gleb Natapov @ 2012-02-20 19:04 ` Gleb Natapov 2012-02-20 19:24 ` Peter Lieven 2012-02-20 19:59 ` Peter Lieven 2012-02-20 19:15 ` Peter Lieven 1 sibling, 2 replies; 19+ messages in thread From: Gleb Natapov @ 2012-02-20 19:04 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm On Mon, Feb 20, 2012 at 08:40:08PM +0200, Gleb Natapov wrote: > On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: > > Hi, > > > > I came a across an issue with a Windows 7 (32-bit) as well as with a > > Windows 2008 R2 (64-bit) guest. > > > > If I transfer a file from the VM via CIFS or FTP to a remote machine, > > i get very poor read performance (around 13MB/s). The VM peaks at 100% > > cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat > > > > efer_reload 0 0 > > exits 2260976 79620 > > fpu_reload 6197 11 > > halt_exits 114734 5011 > > halt_wakeup 111195 4876 > > host_state_reload 1499659 60962 > > hypercalls 0 0 > > insn_emulation 1577325 58488 > > insn_emulation_fail 0 0 > > invlpg 0 0 > > io_exits 943949 40249 > Hmm, too many of those. > > > irq_exits 108679 5434 > > irq_injections 236545 10788 > > irq_window 7606 246 > > largepages 672 5 > > mmio_exits 460020 16082 > > mmu_cache_miss 119 0 > > mmu_flooded 0 0 > > mmu_pde_zapped 0 0 > > mmu_pte_updated 0 0 > > mmu_pte_write 13474 9 > > mmu_recycled 0 0 > > mmu_shadow_zapped 141 0 > > mmu_unsync 0 0 > > nmi_injections 0 0 > > nmi_window 0 0 > > pf_fixed 22803 35 > > pf_guest 0 0 > > remote_tlb_flush 239 2 > > request_irq 0 0 > > signal_exits 0 0 > > tlb_flush 20933 0 > > > > If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s > > throughput. The kvm_stats look a lot more sane. > > > > efer_reload 0 0 > > exits 6132004 17931 > > fpu_reload 19863 3 > > halt_exits 264961 3083 > > halt_wakeup 236468 2959 > > host_state_reload 1104468 3104 > > hypercalls 0 0 > > insn_emulation 1417443 7518 > > insn_emulation_fail 0 0 > > invlpg 0 0 > > io_exits 869380 2795 > > irq_exits 253501 2362 > > irq_injections 616967 6804 > > irq_window 201186 2161 > > largepages 1019 0 > > mmio_exits 205268 0 > > mmu_cache_miss 192 0 > > mmu_flooded 0 0 > > mmu_pde_zapped 0 0 > > mmu_pte_updated 0 0 > > mmu_pte_write 7440546 0 > > mmu_recycled 0 0 > > mmu_shadow_zapped 259 0 > > mmu_unsync 0 0 > > nmi_injections 0 0 > > nmi_window 0 0 > > pf_fixed 38529 30 > > pf_guest 0 0 > > remote_tlb_flush 761 1 > > request_irq 0 0 > > signal_exits 0 0 > > tlb_flush 0 0 > > > > I use virtio-net (with vhost-net) and virtio-blk. I tried disabling > > hpet (which basically illiminated the mmio_exits, but does not > > increase > > performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 > > ) - no improvement. > > > > My commandline: > > /usr/bin/qemu-kvm-1.0 -netdev > > type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on > > -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native > > -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name > > lieven-win7-vc -boot order=dc,menu=off -k de -pidfile > > /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu > > host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet > > > > What further information is needed to debug this further? > > > Which kernel version (looks like something recent)? > Which host CPU (looks like something old)? Output of cat /proc/cpuinfo > Which Windows' virtio drivers are you using? > > Take a trace like described here http://www.linux-kvm.org/page/Tracing > (with -no-hpet please). > And also "info pci" output from qemu monitor while we are at it. > Try to use -cpu host,+x2apic. It may help Linux guest performance. > > -- > Gleb. > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 19:04 ` Gleb Natapov @ 2012-02-20 19:24 ` Peter Lieven 2012-02-20 19:59 ` Peter Lieven 1 sibling, 0 replies; 19+ messages in thread From: Peter Lieven @ 2012-02-20 19:24 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm On 20.02.2012 20:04, Gleb Natapov wrote: > On Mon, Feb 20, 2012 at 08:40:08PM +0200, Gleb Natapov wrote: >> On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: >>> Hi, >>> >>> I came a across an issue with a Windows 7 (32-bit) as well as with a >>> Windows 2008 R2 (64-bit) guest. >>> >>> If I transfer a file from the VM via CIFS or FTP to a remote machine, >>> i get very poor read performance (around 13MB/s). The VM peaks at 100% >>> cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat >>> >>> efer_reload 0 0 >>> exits 2260976 79620 >>> fpu_reload 6197 11 >>> halt_exits 114734 5011 >>> halt_wakeup 111195 4876 >>> host_state_reload 1499659 60962 >>> hypercalls 0 0 >>> insn_emulation 1577325 58488 >>> insn_emulation_fail 0 0 >>> invlpg 0 0 >>> io_exits 943949 40249 >> Hmm, too many of those. >> >>> irq_exits 108679 5434 >>> irq_injections 236545 10788 >>> irq_window 7606 246 >>> largepages 672 5 >>> mmio_exits 460020 16082 >>> mmu_cache_miss 119 0 >>> mmu_flooded 0 0 >>> mmu_pde_zapped 0 0 >>> mmu_pte_updated 0 0 >>> mmu_pte_write 13474 9 >>> mmu_recycled 0 0 >>> mmu_shadow_zapped 141 0 >>> mmu_unsync 0 0 >>> nmi_injections 0 0 >>> nmi_window 0 0 >>> pf_fixed 22803 35 >>> pf_guest 0 0 >>> remote_tlb_flush 239 2 >>> request_irq 0 0 >>> signal_exits 0 0 >>> tlb_flush 20933 0 >>> >>> If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s >>> throughput. The kvm_stats look a lot more sane. >>> >>> efer_reload 0 0 >>> exits 6132004 17931 >>> fpu_reload 19863 3 >>> halt_exits 264961 3083 >>> halt_wakeup 236468 2959 >>> host_state_reload 1104468 3104 >>> hypercalls 0 0 >>> insn_emulation 1417443 7518 >>> insn_emulation_fail 0 0 >>> invlpg 0 0 >>> io_exits 869380 2795 >>> irq_exits 253501 2362 >>> irq_injections 616967 6804 >>> irq_window 201186 2161 >>> largepages 1019 0 >>> mmio_exits 205268 0 >>> mmu_cache_miss 192 0 >>> mmu_flooded 0 0 >>> mmu_pde_zapped 0 0 >>> mmu_pte_updated 0 0 >>> mmu_pte_write 7440546 0 >>> mmu_recycled 0 0 >>> mmu_shadow_zapped 259 0 >>> mmu_unsync 0 0 >>> nmi_injections 0 0 >>> nmi_window 0 0 >>> pf_fixed 38529 30 >>> pf_guest 0 0 >>> remote_tlb_flush 761 1 >>> request_irq 0 0 >>> signal_exits 0 0 >>> tlb_flush 0 0 >>> >>> I use virtio-net (with vhost-net) and virtio-blk. I tried disabling >>> hpet (which basically illiminated the mmio_exits, but does not >>> increase >>> performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 >>> ) - no improvement. >>> >>> My commandline: >>> /usr/bin/qemu-kvm-1.0 -netdev >>> type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on >>> -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native >>> -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name >>> lieven-win7-vc -boot order=dc,menu=off -k de -pidfile >>> /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu >>> host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet >>> >>> What further information is needed to debug this further? >>> >> Which kernel version (looks like something recent)? >> Which host CPU (looks like something old)? > Output of cat /proc/cpuinfo > >> Which Windows' virtio drivers are you using? >> >> Take a trace like described here http://www.linux-kvm.org/page/Tracing >> (with -no-hpet please). >> > And also "info pci" output from qemu monitor while we are at it. here we go. i stumbled across IRQ 0 ? QEMU 1.0 monitor - type 'help' for more information (qemu) info pci info pci Bus 0, device 0, function 0: Host bridge: PCI device 8086:1237 id "" Bus 0, device 1, function 0: ISA bridge: PCI device 8086:7000 id "" Bus 0, device 1, function 1: IDE controller: PCI device 8086:7010 BAR4: I/O at 0xc080 [0xc08f]. id "" Bus 0, device 1, function 2: USB controller: PCI device 8086:7020 IRQ 5. BAR4: I/O at 0xc040 [0xc05f]. id "" Bus 0, device 1, function 3: Bridge: PCI device 8086:7113 IRQ 9. id "" Bus 0, device 2, function 0: VGA controller: PCI device 1234:1111 BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 3, function 0: Ethernet controller: PCI device 1af4:1000 IRQ 0. BAR0: I/O at 0xc060 [0xc07f]. BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 4, function 0: SCSI controller: PCI device 1af4:1001 IRQ 0. BAR0: I/O at 0xc000 [0xc03f]. BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff]. id "" >> Try to use -cpu host,+x2apic. It may help Linux guest performance. >> >> -- >> Gleb. >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 19:04 ` Gleb Natapov 2012-02-20 19:24 ` Peter Lieven @ 2012-02-20 19:59 ` Peter Lieven 2012-02-20 20:45 ` Gleb Natapov 1 sibling, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-20 19:59 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm On 20.02.2012 20:04, Gleb Natapov wrote: > On Mon, Feb 20, 2012 at 08:40:08PM +0200, Gleb Natapov wrote: >> On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: >>> Hi, >>> >>> I came a across an issue with a Windows 7 (32-bit) as well as with a >>> Windows 2008 R2 (64-bit) guest. >>> >>> If I transfer a file from the VM via CIFS or FTP to a remote machine, >>> i get very poor read performance (around 13MB/s). The VM peaks at 100% >>> cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat >>> >>> efer_reload 0 0 >>> exits 2260976 79620 >>> fpu_reload 6197 11 >>> halt_exits 114734 5011 >>> halt_wakeup 111195 4876 >>> host_state_reload 1499659 60962 >>> hypercalls 0 0 >>> insn_emulation 1577325 58488 >>> insn_emulation_fail 0 0 >>> invlpg 0 0 >>> io_exits 943949 40249 >> Hmm, too many of those. >> >>> irq_exits 108679 5434 >>> irq_injections 236545 10788 >>> irq_window 7606 246 >>> largepages 672 5 >>> mmio_exits 460020 16082 >>> mmu_cache_miss 119 0 >>> mmu_flooded 0 0 >>> mmu_pde_zapped 0 0 >>> mmu_pte_updated 0 0 >>> mmu_pte_write 13474 9 >>> mmu_recycled 0 0 >>> mmu_shadow_zapped 141 0 >>> mmu_unsync 0 0 >>> nmi_injections 0 0 >>> nmi_window 0 0 >>> pf_fixed 22803 35 >>> pf_guest 0 0 >>> remote_tlb_flush 239 2 >>> request_irq 0 0 >>> signal_exits 0 0 >>> tlb_flush 20933 0 >>> >>> If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s >>> throughput. The kvm_stats look a lot more sane. >>> >>> efer_reload 0 0 >>> exits 6132004 17931 >>> fpu_reload 19863 3 >>> halt_exits 264961 3083 >>> halt_wakeup 236468 2959 >>> host_state_reload 1104468 3104 >>> hypercalls 0 0 >>> insn_emulation 1417443 7518 >>> insn_emulation_fail 0 0 >>> invlpg 0 0 >>> io_exits 869380 2795 >>> irq_exits 253501 2362 >>> irq_injections 616967 6804 >>> irq_window 201186 2161 >>> largepages 1019 0 >>> mmio_exits 205268 0 >>> mmu_cache_miss 192 0 >>> mmu_flooded 0 0 >>> mmu_pde_zapped 0 0 >>> mmu_pte_updated 0 0 >>> mmu_pte_write 7440546 0 >>> mmu_recycled 0 0 >>> mmu_shadow_zapped 259 0 >>> mmu_unsync 0 0 >>> nmi_injections 0 0 >>> nmi_window 0 0 >>> pf_fixed 38529 30 >>> pf_guest 0 0 >>> remote_tlb_flush 761 1 >>> request_irq 0 0 >>> signal_exits 0 0 >>> tlb_flush 0 0 >>> >>> I use virtio-net (with vhost-net) and virtio-blk. I tried disabling >>> hpet (which basically illiminated the mmio_exits, but does not >>> increase >>> performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 >>> ) - no improvement. >>> >>> My commandline: >>> /usr/bin/qemu-kvm-1.0 -netdev >>> type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on >>> -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native >>> -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name >>> lieven-win7-vc -boot order=dc,menu=off -k de -pidfile >>> /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu >>> host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet >>> >>> What further information is needed to debug this further? >>> >> Which kernel version (looks like something recent)? >> Which host CPU (looks like something old)? > Output of cat /proc/cpuinfo > >> Which Windows' virtio drivers are you using? >> >> Take a trace like described here http://www.linux-kvm.org/page/Tracing >> (with -no-hpet please). >> > And also "info pci" output from qemu monitor while we are at it. here is the output while i was tracing. you can download the trace i took while i did a ftp transfer from the vm: -> http://82.141.21.156/report.txt.gz QEMU 1.0 monitor - type 'help' for more information (qemu) info pci info pci Bus 0, device 0, function 0: Host bridge: PCI device 8086:1237 id "" Bus 0, device 1, function 0: ISA bridge: PCI device 8086:7000 id "" Bus 0, device 1, function 1: IDE controller: PCI device 8086:7010 BAR4: I/O at 0xc080 [0xc08f]. id "" Bus 0, device 1, function 2: USB controller: PCI device 8086:7020 IRQ 5. BAR4: I/O at 0xc040 [0xc05f]. id "" Bus 0, device 1, function 3: Bridge: PCI device 8086:7113 IRQ 9. id "" Bus 0, device 2, function 0: VGA controller: PCI device 1234:1111 BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 3, function 0: Ethernet controller: PCI device 1af4:1000 IRQ 0. BAR0: I/O at 0xc060 [0xc07f]. BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 4, function 0: SCSI controller: PCI device 1af4:1001 IRQ 0. BAR0: I/O at 0xc000 [0xc03f]. BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff]. id "" thanks for your help, peter ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 19:59 ` Peter Lieven @ 2012-02-20 20:45 ` Gleb Natapov 2012-02-21 10:50 ` Peter Lieven 0 siblings, 1 reply; 19+ messages in thread From: Gleb Natapov @ 2012-02-20 20:45 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm On Mon, Feb 20, 2012 at 08:59:38PM +0100, Peter Lieven wrote: > On 20.02.2012 20:04, Gleb Natapov wrote: > >On Mon, Feb 20, 2012 at 08:40:08PM +0200, Gleb Natapov wrote: > >>On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: > >>>Hi, > >>> > >>>I came a across an issue with a Windows 7 (32-bit) as well as with a > >>>Windows 2008 R2 (64-bit) guest. > >>> > >>>If I transfer a file from the VM via CIFS or FTP to a remote machine, > >>>i get very poor read performance (around 13MB/s). The VM peaks at 100% > >>>cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat > >>> > >>>efer_reload 0 0 > >>>exits 2260976 79620 > >>>fpu_reload 6197 11 > >>>halt_exits 114734 5011 > >>>halt_wakeup 111195 4876 > >>>host_state_reload 1499659 60962 > >>>hypercalls 0 0 > >>>insn_emulation 1577325 58488 > >>>insn_emulation_fail 0 0 > >>>invlpg 0 0 > >>>io_exits 943949 40249 > >>Hmm, too many of those. > >> > >>>irq_exits 108679 5434 > >>>irq_injections 236545 10788 > >>>irq_window 7606 246 > >>>largepages 672 5 > >>>mmio_exits 460020 16082 > >>>mmu_cache_miss 119 0 > >>>mmu_flooded 0 0 > >>>mmu_pde_zapped 0 0 > >>>mmu_pte_updated 0 0 > >>>mmu_pte_write 13474 9 > >>>mmu_recycled 0 0 > >>>mmu_shadow_zapped 141 0 > >>>mmu_unsync 0 0 > >>>nmi_injections 0 0 > >>>nmi_window 0 0 > >>>pf_fixed 22803 35 > >>>pf_guest 0 0 > >>>remote_tlb_flush 239 2 > >>>request_irq 0 0 > >>>signal_exits 0 0 > >>>tlb_flush 20933 0 > >>> > >>>If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s > >>>throughput. The kvm_stats look a lot more sane. > >>> > >>>efer_reload 0 0 > >>>exits 6132004 17931 > >>>fpu_reload 19863 3 > >>>halt_exits 264961 3083 > >>>halt_wakeup 236468 2959 > >>>host_state_reload 1104468 3104 > >>>hypercalls 0 0 > >>>insn_emulation 1417443 7518 > >>>insn_emulation_fail 0 0 > >>>invlpg 0 0 > >>>io_exits 869380 2795 > >>>irq_exits 253501 2362 > >>>irq_injections 616967 6804 > >>>irq_window 201186 2161 > >>>largepages 1019 0 > >>>mmio_exits 205268 0 > >>>mmu_cache_miss 192 0 > >>>mmu_flooded 0 0 > >>>mmu_pde_zapped 0 0 > >>>mmu_pte_updated 0 0 > >>>mmu_pte_write 7440546 0 > >>>mmu_recycled 0 0 > >>>mmu_shadow_zapped 259 0 > >>>mmu_unsync 0 0 > >>>nmi_injections 0 0 > >>>nmi_window 0 0 > >>>pf_fixed 38529 30 > >>>pf_guest 0 0 > >>>remote_tlb_flush 761 1 > >>>request_irq 0 0 > >>>signal_exits 0 0 > >>>tlb_flush 0 0 > >>> > >>>I use virtio-net (with vhost-net) and virtio-blk. I tried disabling > >>>hpet (which basically illiminated the mmio_exits, but does not > >>>increase > >>>performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 > >>>) - no improvement. > >>> > >>>My commandline: > >>>/usr/bin/qemu-kvm-1.0 -netdev > >>>type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on > >>>-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native > >>>-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name > >>>lieven-win7-vc -boot order=dc,menu=off -k de -pidfile > >>>/var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu > >>>host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet > >>> > >>>What further information is needed to debug this further? > >>> > >>Which kernel version (looks like something recent)? > >>Which host CPU (looks like something old)? > >Output of cat /proc/cpuinfo > > > >>Which Windows' virtio drivers are you using? > >> > >>Take a trace like described here http://www.linux-kvm.org/page/Tracing > >>(with -no-hpet please). > >> > >And also "info pci" output from qemu monitor while we are at it. > here is the output while i was tracing. you can download the trace > i took while i did a ftp transfer from the vm: > > -> http://82.141.21.156/report.txt.gz > Windows reads PM timer. A lot. 15152 times per second. Can you try to run this command in Windows guest: bcdedit /set {default} useplatformclock false I hope it will make Windows use TSC instead, but you can't be sure about anything with Windows :( > QEMU 1.0 monitor - type 'help' for more information > (qemu) info pci > info pci > Bus 0, device 0, function 0: > Host bridge: PCI device 8086:1237 > id "" > Bus 0, device 1, function 0: > ISA bridge: PCI device 8086:7000 > id "" > Bus 0, device 1, function 1: > IDE controller: PCI device 8086:7010 > BAR4: I/O at 0xc080 [0xc08f]. > id "" > Bus 0, device 1, function 2: > USB controller: PCI device 8086:7020 > IRQ 5. > BAR4: I/O at 0xc040 [0xc05f]. > id "" > Bus 0, device 1, function 3: > Bridge: PCI device 8086:7113 > IRQ 9. > id "" > Bus 0, device 2, function 0: > VGA controller: PCI device 1234:1111 > BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff]. > BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. > id "" > Bus 0, device 3, function 0: > Ethernet controller: PCI device 1af4:1000 > IRQ 0. > BAR0: I/O at 0xc060 [0xc07f]. > BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff]. > BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. > id "" > Bus 0, device 4, function 0: > SCSI controller: PCI device 1af4:1001 > IRQ 0. > BAR0: I/O at 0xc000 [0xc03f]. > BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff]. > id "" > > thanks for your help, > peter -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 20:45 ` Gleb Natapov @ 2012-02-21 10:50 ` Peter Lieven 2012-02-21 10:56 ` Gleb Natapov 0 siblings, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-21 10:50 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm On 20.02.2012 21:45, Gleb Natapov wrote: > On Mon, Feb 20, 2012 at 08:59:38PM +0100, Peter Lieven wrote: >> On 20.02.2012 20:04, Gleb Natapov wrote: >>> On Mon, Feb 20, 2012 at 08:40:08PM +0200, Gleb Natapov wrote: >>>> On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: >>>>> Hi, >>>>> >>>>> I came a across an issue with a Windows 7 (32-bit) as well as with a >>>>> Windows 2008 R2 (64-bit) guest. >>>>> >>>>> If I transfer a file from the VM via CIFS or FTP to a remote machine, >>>>> i get very poor read performance (around 13MB/s). The VM peaks at 100% >>>>> cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat >>>>> >>>>> efer_reload 0 0 >>>>> exits 2260976 79620 >>>>> fpu_reload 6197 11 >>>>> halt_exits 114734 5011 >>>>> halt_wakeup 111195 4876 >>>>> host_state_reload 1499659 60962 >>>>> hypercalls 0 0 >>>>> insn_emulation 1577325 58488 >>>>> insn_emulation_fail 0 0 >>>>> invlpg 0 0 >>>>> io_exits 943949 40249 >>>> Hmm, too many of those. >>>> >>>>> irq_exits 108679 5434 >>>>> irq_injections 236545 10788 >>>>> irq_window 7606 246 >>>>> largepages 672 5 >>>>> mmio_exits 460020 16082 >>>>> mmu_cache_miss 119 0 >>>>> mmu_flooded 0 0 >>>>> mmu_pde_zapped 0 0 >>>>> mmu_pte_updated 0 0 >>>>> mmu_pte_write 13474 9 >>>>> mmu_recycled 0 0 >>>>> mmu_shadow_zapped 141 0 >>>>> mmu_unsync 0 0 >>>>> nmi_injections 0 0 >>>>> nmi_window 0 0 >>>>> pf_fixed 22803 35 >>>>> pf_guest 0 0 >>>>> remote_tlb_flush 239 2 >>>>> request_irq 0 0 >>>>> signal_exits 0 0 >>>>> tlb_flush 20933 0 >>>>> >>>>> If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s >>>>> throughput. The kvm_stats look a lot more sane. >>>>> >>>>> efer_reload 0 0 >>>>> exits 6132004 17931 >>>>> fpu_reload 19863 3 >>>>> halt_exits 264961 3083 >>>>> halt_wakeup 236468 2959 >>>>> host_state_reload 1104468 3104 >>>>> hypercalls 0 0 >>>>> insn_emulation 1417443 7518 >>>>> insn_emulation_fail 0 0 >>>>> invlpg 0 0 >>>>> io_exits 869380 2795 >>>>> irq_exits 253501 2362 >>>>> irq_injections 616967 6804 >>>>> irq_window 201186 2161 >>>>> largepages 1019 0 >>>>> mmio_exits 205268 0 >>>>> mmu_cache_miss 192 0 >>>>> mmu_flooded 0 0 >>>>> mmu_pde_zapped 0 0 >>>>> mmu_pte_updated 0 0 >>>>> mmu_pte_write 7440546 0 >>>>> mmu_recycled 0 0 >>>>> mmu_shadow_zapped 259 0 >>>>> mmu_unsync 0 0 >>>>> nmi_injections 0 0 >>>>> nmi_window 0 0 >>>>> pf_fixed 38529 30 >>>>> pf_guest 0 0 >>>>> remote_tlb_flush 761 1 >>>>> request_irq 0 0 >>>>> signal_exits 0 0 >>>>> tlb_flush 0 0 >>>>> >>>>> I use virtio-net (with vhost-net) and virtio-blk. I tried disabling >>>>> hpet (which basically illiminated the mmio_exits, but does not >>>>> increase >>>>> performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 >>>>> ) - no improvement. >>>>> >>>>> My commandline: >>>>> /usr/bin/qemu-kvm-1.0 -netdev >>>>> type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on >>>>> -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native >>>>> -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name >>>>> lieven-win7-vc -boot order=dc,menu=off -k de -pidfile >>>>> /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu >>>>> host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet >>>>> >>>>> What further information is needed to debug this further? >>>>> >>>> Which kernel version (looks like something recent)? >>>> Which host CPU (looks like something old)? >>> Output of cat /proc/cpuinfo >>> >>>> Which Windows' virtio drivers are you using? >>>> >>>> Take a trace like described here http://www.linux-kvm.org/page/Tracing >>>> (with -no-hpet please). >>>> >>> And also "info pci" output from qemu monitor while we are at it. >> here is the output while i was tracing. you can download the trace >> i took while i did a ftp transfer from the vm: >> >> -> http://82.141.21.156/report.txt.gz >> > Windows reads PM timer. A lot. 15152 times per second. > > Can you try to run this command in Windows guest: > > bcdedit /set {default} useplatformclock false > > I hope it will make Windows use TSC instead, but you can't be sure > about anything with Windows :( Whatever it does now it eates more CPU has almost equal number of exits and throughput is about the same (15MB/s). If pmtimer is at 0xb008 it still reads it like hell. I checked with bcedit /v that useplatformclock is set to "No". I still wonder why both virtio devices are on IRQ0 ? New Trace: http://82.141.21.156/report2.txt.gz efer_reload 0 0 exits 1510993 59343 fpu_reload 6729 10 halt_exits 93603 5913 halt_wakeup 95698 5849 host_state_reload 738523 24727 hypercalls 0 0 insn_emulation 678416 20107 insn_emulation_fail 0 0 invlpg 0 0 io_exits 703291 28436 irq_exits 102117 7527 irq_injections 217335 14344 irq_window 9926 650 largepages 573 8 mmio_exits 27 0 mmu_cache_miss 148 0 mmu_flooded 0 0 mmu_pde_zapped 0 0 mmu_pte_updated 0 0 mmu_pte_write 0 0 mmu_recycled 0 0 mmu_shadow_zapped 190 0 mmu_unsync 0 0 nmi_injections 0 0 nmi_window 0 0 pf_fixed 21938 38 pf_guest 0 0 remote_tlb_flush 20 0 request_irq 0 0 signal_exits 0 0 tlb_flush 11711 0 QEMU 1.0 monitor - type 'help' for more information (qemu) info pci info pci Bus 0, device 0, function 0: Host bridge: PCI device 8086:1237 id "" Bus 0, device 1, function 0: ISA bridge: PCI device 8086:7000 id "" Bus 0, device 1, function 1: IDE controller: PCI device 8086:7010 BAR4: I/O at 0xc080 [0xc08f]. id "" Bus 0, device 1, function 2: USB controller: PCI device 8086:7020 IRQ 5. BAR4: I/O at 0xc040 [0xc05f]. id "" Bus 0, device 1, function 3: Bridge: PCI device 8086:7113 IRQ 9. id "" Bus 0, device 2, function 0: VGA controller: PCI device 1234:1111 BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 3, function 0: Ethernet controller: PCI device 1af4:1000 IRQ 0. BAR0: I/O at 0xc060 [0xc07f]. BAR1: 32 bit memory at 0xfebf0000 [0xfebf0fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" Bus 0, device 4, function 0: SCSI controller: PCI device 1af4:1001 IRQ 0. BAR0: I/O at 0xc000 [0xc03f]. BAR1: 32 bit memory at 0xfebf1000 [0xfebf1fff]. id "" ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 10:50 ` Peter Lieven @ 2012-02-21 10:56 ` Gleb Natapov 2012-02-21 10:59 ` Peter Lieven 0 siblings, 1 reply; 19+ messages in thread From: Gleb Natapov @ 2012-02-21 10:56 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm, vrozenfe On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: > >I hope it will make Windows use TSC instead, but you can't be sure > >about anything with Windows :( > Whatever it does now it eates more CPU has almost equal > number of exits and throughput is about the same (15MB/s). > If pmtimer is at 0xb008 it still reads it like hell. > > I checked with bcedit /v that useplatformclock is set to "No". Yeah, today I noticed that it is likely virtio drivers that hammer on PM timer (at least rip of the instruction that access it is very close to rip of the instruction that access virtio pio). Vadim, Windows driver developer, is CCed. > > I still wonder why both virtio devices are on IRQ0 ? > They use MSI like they should. -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 10:56 ` Gleb Natapov @ 2012-02-21 10:59 ` Peter Lieven 2012-02-21 11:00 ` Gleb Natapov 0 siblings, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-21 10:59 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm, vrozenfe On 21.02.2012 11:56, Gleb Natapov wrote: > On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>> I hope it will make Windows use TSC instead, but you can't be sure >>> about anything with Windows :( >> Whatever it does now it eates more CPU has almost equal >> number of exits and throughput is about the same (15MB/s). >> If pmtimer is at 0xb008 it still reads it like hell. >> >> I checked with bcedit /v that useplatformclock is set to "No". > Yeah, today I noticed that it is likely virtio drivers that hammer > on PM timer (at least rip of the instruction that access it is > very close to rip of the instruction that access virtio pio). > Vadim, Windows driver developer, is CCed. Ok, I will switch to IDE and e1000 to confirm this? Or does it not make sense? Peter ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 10:59 ` Peter Lieven @ 2012-02-21 11:00 ` Gleb Natapov 2012-02-21 11:16 ` Peter Lieven 0 siblings, 1 reply; 19+ messages in thread From: Gleb Natapov @ 2012-02-21 11:00 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm, vrozenfe On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: > On 21.02.2012 11:56, Gleb Natapov wrote: > >On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: > >>>I hope it will make Windows use TSC instead, but you can't be sure > >>>about anything with Windows :( > >>Whatever it does now it eates more CPU has almost equal > >>number of exits and throughput is about the same (15MB/s). > >>If pmtimer is at 0xb008 it still reads it like hell. > >> > >>I checked with bcedit /v that useplatformclock is set to "No". > >Yeah, today I noticed that it is likely virtio drivers that hammer > >on PM timer (at least rip of the instruction that access it is > >very close to rip of the instruction that access virtio pio). > >Vadim, Windows driver developer, is CCed. > Ok, I will switch to IDE and e1000 to confirm this? Or does it not > make sense? > It make perfect sense! Please try it. -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 11:00 ` Gleb Natapov @ 2012-02-21 11:16 ` Peter Lieven 2012-02-21 11:46 ` Gleb Natapov 0 siblings, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-21 11:16 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm, vrozenfe On 21.02.2012 12:00, Gleb Natapov wrote: > On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: >> On 21.02.2012 11:56, Gleb Natapov wrote: >>> On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>>>> I hope it will make Windows use TSC instead, but you can't be sure >>>>> about anything with Windows :( >>>> Whatever it does now it eates more CPU has almost equal >>>> number of exits and throughput is about the same (15MB/s). >>>> If pmtimer is at 0xb008 it still reads it like hell. >>>> >>>> I checked with bcedit /v that useplatformclock is set to "No". >>> Yeah, today I noticed that it is likely virtio drivers that hammer >>> on PM timer (at least rip of the instruction that access it is >>> very close to rip of the instruction that access virtio pio). >>> Vadim, Windows driver developer, is CCed. >> Ok, I will switch to IDE and e1000 to confirm this? Or does it not >> make sense? >> > It make perfect sense! Please try it. ~10MB/s. still a lot of 0xb008 reads. efer_reload 0 0 exits 4389875 72341 fpu_reload 36729 342 halt_exits 206204 3451 halt_wakeup 212953 3474 host_state_reload 2976799 59043 hypercalls 0 0 insn_emulation 2936091 54921 insn_emulation_fail 0 0 invlpg 0 0 io_exits 1821386 14108 irq_exits 81999 2798 irq_injections 343720 8560 irq_window 12712 153 largepages 754 2 mmio_exits 37 0 mmu_cache_miss 148 0 mmu_flooded 0 0 mmu_pde_zapped 0 0 mmu_pte_updated 0 0 mmu_pte_write 0 0 mmu_recycled 0 0 mmu_shadow_zapped 189 0 mmu_unsync 0 0 nmi_injections 0 0 nmi_window 0 0 pf_fixed 139461 21 pf_guest 0 0 remote_tlb_flush 248 0 request_irq 0 0 signal_exits 0 0 tlb_flush 15366 0 trace at http://82.141.21.156/report3.txt.gz Peter > -- > Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 11:16 ` Peter Lieven @ 2012-02-21 11:46 ` Gleb Natapov 2012-02-21 12:05 ` Peter Lieven 0 siblings, 1 reply; 19+ messages in thread From: Gleb Natapov @ 2012-02-21 11:46 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm, vrozenfe On Tue, Feb 21, 2012 at 12:16:16PM +0100, Peter Lieven wrote: > On 21.02.2012 12:00, Gleb Natapov wrote: > >On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: > >>On 21.02.2012 11:56, Gleb Natapov wrote: > >>>On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: > >>>>>I hope it will make Windows use TSC instead, but you can't be sure > >>>>>about anything with Windows :( > >>>>Whatever it does now it eates more CPU has almost equal > >>>>number of exits and throughput is about the same (15MB/s). > >>>>If pmtimer is at 0xb008 it still reads it like hell. > >>>> > >>>>I checked with bcedit /v that useplatformclock is set to "No". > >>>Yeah, today I noticed that it is likely virtio drivers that hammer > >>>on PM timer (at least rip of the instruction that access it is > >>>very close to rip of the instruction that access virtio pio). > >>>Vadim, Windows driver developer, is CCed. > >>Ok, I will switch to IDE and e1000 to confirm this? Or does it not > >>make sense? > >> > >It make perfect sense! Please try it. > ~10MB/s. still a lot of 0xb008 reads. > The same amount of reads essentially. So my theory is incorrect. Virtio driver probably calls Windows function to do IO and the function happens to be near the function that access PM timer. I wonder why time stamps in your traces are so coarse-grained. What do you see in /sys/bus/clocksource/devices/clocksource0/current_clocksource ? -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 11:46 ` Gleb Natapov @ 2012-02-21 12:05 ` Peter Lieven 2012-02-21 13:56 ` Vadim Rozenfeld 0 siblings, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-21 12:05 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm, vrozenfe On 21.02.2012 12:46, Gleb Natapov wrote: > On Tue, Feb 21, 2012 at 12:16:16PM +0100, Peter Lieven wrote: >> On 21.02.2012 12:00, Gleb Natapov wrote: >>> On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: >>>> On 21.02.2012 11:56, Gleb Natapov wrote: >>>>> On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>>>>>> I hope it will make Windows use TSC instead, but you can't be sure >>>>>>> about anything with Windows :( >>>>>> Whatever it does now it eates more CPU has almost equal >>>>>> number of exits and throughput is about the same (15MB/s). >>>>>> If pmtimer is at 0xb008 it still reads it like hell. >>>>>> >>>>>> I checked with bcedit /v that useplatformclock is set to "No". >>>>> Yeah, today I noticed that it is likely virtio drivers that hammer >>>>> on PM timer (at least rip of the instruction that access it is >>>>> very close to rip of the instruction that access virtio pio). >>>>> Vadim, Windows driver developer, is CCed. >>>> Ok, I will switch to IDE and e1000 to confirm this? Or does it not >>>> make sense? >>>> >>> It make perfect sense! Please try it. >> ~10MB/s. still a lot of 0xb008 reads. >> > The same amount of reads essentially. So my theory is incorrect. Virtio > driver probably calls Windows function to do IO and the function > happens to be near the function that access PM timer. > > I wonder why time stamps in your traces are so coarse-grained. What do > you see in /sys/bus/clocksource/devices/clocksource0/current_clocksource ? its set to acpi_pm on the host. we changed that from tsc (choosen by kernel) after we encountered a kernel bug which ooops all hosts after approx. 270 days uptime. (https://lkml.org/lkml/2011/7/21/343). i am not sure if this is fixed in 2.6.38 or later kernels and we could go back to tsc. for testing i already checked this, but it doesn't give better performance. peter > -- > Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 12:05 ` Peter Lieven @ 2012-02-21 13:56 ` Vadim Rozenfeld 2012-02-21 14:10 ` Peter Lieven 0 siblings, 1 reply; 19+ messages in thread From: Vadim Rozenfeld @ 2012-02-21 13:56 UTC (permalink / raw) To: Peter Lieven; +Cc: Gleb Natapov, qemu-devel, kvm ----- Original Message ----- From: "Peter Lieven" <pl@dlh.net> To: "Gleb Natapov" <gleb@redhat.com> Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, vrozenfe@redhat.com Sent: Tuesday, February 21, 2012 2:05:25 PM Subject: Re: win7 bad i/o performance, high insn_emulation and exists On 21.02.2012 12:46, Gleb Natapov wrote: > On Tue, Feb 21, 2012 at 12:16:16PM +0100, Peter Lieven wrote: >> On 21.02.2012 12:00, Gleb Natapov wrote: >>> On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: >>>> On 21.02.2012 11:56, Gleb Natapov wrote: >>>>> On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>>>>>> I hope it will make Windows use TSC instead, but you can't be sure >>>>>>> about anything with Windows :( >>>>>> Whatever it does now it eates more CPU has almost equal >>>>>> number of exits and throughput is about the same (15MB/s). >>>>>> If pmtimer is at 0xb008 it still reads it like hell. >>>>>> >>>>>> I checked with bcedit /v that useplatformclock is set to "No". >>>>> Yeah, today I noticed that it is likely virtio drivers that hammer >>>>> on PM timer (at least rip of the instruction that access it is >>>>> very close to rip of the instruction that access virtio pio). >>>>> Vadim, Windows driver developer, is CCed. >>>> Ok, I will switch to IDE and e1000 to confirm this? Or does it not >>>> make sense? >>>> >>> It make perfect sense! Please try it. >> ~10MB/s. still a lot of 0xb008 reads. >> [VR] Could it be that you have Driver Verifier running in you system? > The same amount of reads essentially. So my theory is incorrect. Virtio > driver probably calls Windows function to do IO and the function > happens to be near the function that access PM timer. > > I wonder why time stamps in your traces are so coarse-grained. What do > you see in /sys/bus/clocksource/devices/clocksource0/current_clocksource ? its set to acpi_pm on the host. we changed that from tsc (choosen by kernel) after we encountered a kernel bug which ooops all hosts after approx. 270 days uptime. (https://lkml.org/lkml/2011/7/21/343). i am not sure if this is fixed in 2.6.38 or later kernels and we could go back to tsc. for testing i already checked this, but it doesn't give better performance. peter > -- > Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 13:56 ` Vadim Rozenfeld @ 2012-02-21 14:10 ` Peter Lieven 2012-02-21 16:48 ` Vadim Rozenfeld 0 siblings, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-21 14:10 UTC (permalink / raw) To: Vadim Rozenfeld; +Cc: Gleb Natapov, qemu-devel, kvm On 21.02.2012 14:56, Vadim Rozenfeld wrote: > > ----- Original Message ----- > From: "Peter Lieven"<pl@dlh.net> > To: "Gleb Natapov"<gleb@redhat.com> > Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, vrozenfe@redhat.com > Sent: Tuesday, February 21, 2012 2:05:25 PM > Subject: Re: win7 bad i/o performance, high insn_emulation and exists > > On 21.02.2012 12:46, Gleb Natapov wrote: >> On Tue, Feb 21, 2012 at 12:16:16PM +0100, Peter Lieven wrote: >>> On 21.02.2012 12:00, Gleb Natapov wrote: >>>> On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: >>>>> On 21.02.2012 11:56, Gleb Natapov wrote: >>>>>> On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>>>>>>> I hope it will make Windows use TSC instead, but you can't be sure >>>>>>>> about anything with Windows :( >>>>>>> Whatever it does now it eates more CPU has almost equal >>>>>>> number of exits and throughput is about the same (15MB/s). >>>>>>> If pmtimer is at 0xb008 it still reads it like hell. >>>>>>> >>>>>>> I checked with bcedit /v that useplatformclock is set to "No". >>>>>> Yeah, today I noticed that it is likely virtio drivers that hammer >>>>>> on PM timer (at least rip of the instruction that access it is >>>>>> very close to rip of the instruction that access virtio pio). >>>>>> Vadim, Windows driver developer, is CCed. >>>>> Ok, I will switch to IDE and e1000 to confirm this? Or does it not >>>>> make sense? >>>>> >>>> It make perfect sense! Please try it. >>> ~10MB/s. still a lot of 0xb008 reads. >>> > [VR] > Could it be that you have Driver Verifier running in you system? > unfortunately not. i found the following in an old knowledge base article (http://support.microsoft.com/kb/938448): "Only Windows Server 2003 with Service Pack 2 uniprocessor ACPI HALs use *PMTIMER* for QPC by default. Multiprocessor ACPI HALs will use *PMTIMER* only if *USE_PLATFORM_CLOCK *flag is set by the BIOS or if the */usepmtimer *boot.ini option is used. Other HAL types don’t support *PMTIMER* and will use *TSC* by default for QPC By default, Windows Server 2003 Service Pack 2 (SP2) uses the PM timer for all Advanced Configuration and Power Interface (ACPI) HALs unless one of the following conditions aretrue: * The check process to determine whether the BIOS supports the APIC or ACPI HALs fails. * * Note:* If the BIOS does not support the ACPI HAL, contact the original equipment manufacturer to determine whether a BIOS update is available that will resolve the problem. If a BIOS update is not available, you must use the PM timer by using the */usepmtimer* switch. If you are not running Windows Server 2003 SP2, you must force the AMD computer to use the PM timer by using the */usepmtimer* switch. *Note* The decision to use the PM timer or the TSC timer is made during a check that is performed at startup to query the BIOS and to determine whether the BIOS will support the PM timer functions. This check is not completely accurate on AMD chipsets. Therefore, you must use the */usepmtimer* switch. In Windows Server 2003 SP2, this section of code was rewritten. Therefore, the correct performance monitor data appears on AMD chipsets that have Windows Server 2003 SP2 installed, and you do not have to use the */usepmtimer* switch. For more information about ACPI and APCI hardware support, click the following article number to view the article in the Microsoft Knowledge Base: 309283 <http://support.microsoft.com/kb/309283> HAL options after Windows XP or Windows Server 2003 Setup The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products." - so it seems windows prefers pmtimer over tsc. has anyone an idea/hack to make the acpi_pm timer fail without disabling acpi completely? thanks, peter ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 14:10 ` Peter Lieven @ 2012-02-21 16:48 ` Vadim Rozenfeld 2012-02-21 18:21 ` Peter Lieven 0 siblings, 1 reply; 19+ messages in thread From: Vadim Rozenfeld @ 2012-02-21 16:48 UTC (permalink / raw) To: Peter Lieven; +Cc: Gleb Natapov, qemu-devel, kvm ----- Original Message ----- From: "Peter Lieven" <pl@dlh.net> To: "Vadim Rozenfeld" <vrozenfe@redhat.com> Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, "Gleb Natapov" <gleb@redhat.com> Sent: Tuesday, February 21, 2012 4:10:22 PM Subject: Re: win7 bad i/o performance, high insn_emulation and exists On 21.02.2012 14:56, Vadim Rozenfeld wrote: > > ----- Original Message ----- > From: "Peter Lieven"<pl@dlh.net> > To: "Gleb Natapov"<gleb@redhat.com> > Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, vrozenfe@redhat.com > Sent: Tuesday, February 21, 2012 2:05:25 PM > Subject: Re: win7 bad i/o performance, high insn_emulation and exists > > On 21.02.2012 12:46, Gleb Natapov wrote: >> On Tue, Feb 21, 2012 at 12:16:16PM +0100, Peter Lieven wrote: >>> On 21.02.2012 12:00, Gleb Natapov wrote: >>>> On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: >>>>> On 21.02.2012 11:56, Gleb Natapov wrote: >>>>>> On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>>>>>>> I hope it will make Windows use TSC instead, but you can't be sure >>>>>>>> about anything with Windows :( >>>>>>> Whatever it does now it eates more CPU has almost equal >>>>>>> number of exits and throughput is about the same (15MB/s). >>>>>>> If pmtimer is at 0xb008 it still reads it like hell. >>>>>>> >>>>>>> I checked with bcedit /v that useplatformclock is set to "No". >>>>>> Yeah, today I noticed that it is likely virtio drivers that hammer >>>>>> on PM timer (at least rip of the instruction that access it is >>>>>> very close to rip of the instruction that access virtio pio). >>>>>> Vadim, Windows driver developer, is CCed. >>>>> Ok, I will switch to IDE and e1000 to confirm this? Or does it not >>>>> make sense? >>>>> >>>> It make perfect sense! Please try it. >>> ~10MB/s. still a lot of 0xb008 reads. >>> > [VR] > Could it be that you have Driver Verifier running in you system? > unfortunately not. [VR] Then could you try booting into "Safe Mode"? i found the following in an old knowledge base article (http://support.microsoft.com/kb/938448): "Only Windows Server 2003 with Service Pack 2 uniprocessor ACPI HALs use *PMTIMER* for QPC by default. Multiprocessor ACPI HALs will use *PMTIMER* only if *USE_PLATFORM_CLOCK *flag is set by the BIOS or if the */usepmtimer *boot.ini option is used. Other HAL types don’t support *PMTIMER* and will use *TSC* by default for QPC By default, Windows Server 2003 Service Pack 2 (SP2) uses the PM timer for all Advanced Configuration and Power Interface (ACPI) HALs unless one of the following conditions aretrue: * The check process to determine whether the BIOS supports the APIC or ACPI HALs fails. * * Note:* If the BIOS does not support the ACPI HAL, contact the original equipment manufacturer to determine whether a BIOS update is available that will resolve the problem. If a BIOS update is not available, you must use the PM timer by using the */usepmtimer* switch. If you are not running Windows Server 2003 SP2, you must force the AMD computer to use the PM timer by using the */usepmtimer* switch. *Note* The decision to use the PM timer or the TSC timer is made during a check that is performed at startup to query the BIOS and to determine whether the BIOS will support the PM timer functions. This check is not completely accurate on AMD chipsets. Therefore, you must use the */usepmtimer* switch. In Windows Server 2003 SP2, this section of code was rewritten. Therefore, the correct performance monitor data appears on AMD chipsets that have Windows Server 2003 SP2 installed, and you do not have to use the */usepmtimer* switch. For more information about ACPI and APCI hardware support, click the following article number to view the article in the Microsoft Knowledge Base: 309283 <http://support.microsoft.com/kb/309283> HAL options after Windows XP or Windows Server 2003 Setup The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products." - so it seems windows prefers pmtimer over tsc. has anyone an idea/hack to make the acpi_pm timer fail without disabling acpi completely? thanks, peter ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-21 16:48 ` Vadim Rozenfeld @ 2012-02-21 18:21 ` Peter Lieven 0 siblings, 0 replies; 19+ messages in thread From: Peter Lieven @ 2012-02-21 18:21 UTC (permalink / raw) To: Vadim Rozenfeld; +Cc: Gleb Natapov, qemu-devel, kvm On 21.02.2012 17:48, Vadim Rozenfeld wrote: > > ----- Original Message ----- > From: "Peter Lieven"<pl@dlh.net> > To: "Vadim Rozenfeld"<vrozenfe@redhat.com> > Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, "Gleb Natapov"<gleb@redhat.com> > Sent: Tuesday, February 21, 2012 4:10:22 PM > Subject: Re: win7 bad i/o performance, high insn_emulation and exists > > On 21.02.2012 14:56, Vadim Rozenfeld wrote: >> ----- Original Message ----- >> From: "Peter Lieven"<pl@dlh.net> >> To: "Gleb Natapov"<gleb@redhat.com> >> Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, vrozenfe@redhat.com >> Sent: Tuesday, February 21, 2012 2:05:25 PM >> Subject: Re: win7 bad i/o performance, high insn_emulation and exists >> >> On 21.02.2012 12:46, Gleb Natapov wrote: >>> On Tue, Feb 21, 2012 at 12:16:16PM +0100, Peter Lieven wrote: >>>> On 21.02.2012 12:00, Gleb Natapov wrote: >>>>> On Tue, Feb 21, 2012 at 11:59:23AM +0100, Peter Lieven wrote: >>>>>> On 21.02.2012 11:56, Gleb Natapov wrote: >>>>>>> On Tue, Feb 21, 2012 at 11:50:47AM +0100, Peter Lieven wrote: >>>>>>>>> I hope it will make Windows use TSC instead, but you can't be sure >>>>>>>>> about anything with Windows :( >>>>>>>> Whatever it does now it eates more CPU has almost equal >>>>>>>> number of exits and throughput is about the same (15MB/s). >>>>>>>> If pmtimer is at 0xb008 it still reads it like hell. >>>>>>>> >>>>>>>> I checked with bcedit /v that useplatformclock is set to "No". >>>>>>> Yeah, today I noticed that it is likely virtio drivers that hammer >>>>>>> on PM timer (at least rip of the instruction that access it is >>>>>>> very close to rip of the instruction that access virtio pio). >>>>>>> Vadim, Windows driver developer, is CCed. >>>>>> Ok, I will switch to IDE and e1000 to confirm this? Or does it not >>>>>> make sense? >>>>>> >>>>> It make perfect sense! Please try it. >>>> ~10MB/s. still a lot of 0xb008 reads. >>>> >> [VR] >> Could it be that you have Driver Verifier running in you system? >> > unfortunately not. > > [VR] > Then could you try booting into "Safe Mode"? same picture. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 18:40 ` Gleb Natapov 2012-02-20 19:04 ` Gleb Natapov @ 2012-02-20 19:15 ` Peter Lieven 2012-02-20 20:42 ` Gleb Natapov 1 sibling, 1 reply; 19+ messages in thread From: Peter Lieven @ 2012-02-20 19:15 UTC (permalink / raw) To: Gleb Natapov; +Cc: qemu-devel, kvm On 20.02.2012 19:40, Gleb Natapov wrote: > On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: >> Hi, >> >> I came a across an issue with a Windows 7 (32-bit) as well as with a >> Windows 2008 R2 (64-bit) guest. >> >> If I transfer a file from the VM via CIFS or FTP to a remote machine, >> i get very poor read performance (around 13MB/s). The VM peaks at 100% >> cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat >> >> efer_reload 0 0 >> exits 2260976 79620 >> fpu_reload 6197 11 >> halt_exits 114734 5011 >> halt_wakeup 111195 4876 >> host_state_reload 1499659 60962 >> hypercalls 0 0 >> insn_emulation 1577325 58488 >> insn_emulation_fail 0 0 >> invlpg 0 0 >> io_exits 943949 40249 > Hmm, too many of those. > >> irq_exits 108679 5434 >> irq_injections 236545 10788 >> irq_window 7606 246 >> largepages 672 5 >> mmio_exits 460020 16082 >> mmu_cache_miss 119 0 >> mmu_flooded 0 0 >> mmu_pde_zapped 0 0 >> mmu_pte_updated 0 0 >> mmu_pte_write 13474 9 >> mmu_recycled 0 0 >> mmu_shadow_zapped 141 0 >> mmu_unsync 0 0 >> nmi_injections 0 0 >> nmi_window 0 0 >> pf_fixed 22803 35 >> pf_guest 0 0 >> remote_tlb_flush 239 2 >> request_irq 0 0 >> signal_exits 0 0 >> tlb_flush 20933 0 >> >> If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s >> throughput. The kvm_stats look a lot more sane. >> >> efer_reload 0 0 >> exits 6132004 17931 >> fpu_reload 19863 3 >> halt_exits 264961 3083 >> halt_wakeup 236468 2959 >> host_state_reload 1104468 3104 >> hypercalls 0 0 >> insn_emulation 1417443 7518 >> insn_emulation_fail 0 0 >> invlpg 0 0 >> io_exits 869380 2795 >> irq_exits 253501 2362 >> irq_injections 616967 6804 >> irq_window 201186 2161 >> largepages 1019 0 >> mmio_exits 205268 0 >> mmu_cache_miss 192 0 >> mmu_flooded 0 0 >> mmu_pde_zapped 0 0 >> mmu_pte_updated 0 0 >> mmu_pte_write 7440546 0 >> mmu_recycled 0 0 >> mmu_shadow_zapped 259 0 >> mmu_unsync 0 0 >> nmi_injections 0 0 >> nmi_window 0 0 >> pf_fixed 38529 30 >> pf_guest 0 0 >> remote_tlb_flush 761 1 >> request_irq 0 0 >> signal_exits 0 0 >> tlb_flush 0 0 >> >> I use virtio-net (with vhost-net) and virtio-blk. I tried disabling >> hpet (which basically illiminated the mmio_exits, but does not >> increase >> performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 >> ) - no improvement. >> >> My commandline: >> /usr/bin/qemu-kvm-1.0 -netdev >> type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on >> -device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native >> -m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name >> lieven-win7-vc -boot order=dc,menu=off -k de -pidfile >> /var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu >> host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet >> >> What further information is needed to debug this further? >> > Which kernel version (looks like something recent)? 2.6.38 with kvm-kmod 3.2 > Which host CPU (looks like something old)? why? i guess its (quite) new. vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz stepping : 2 cpu MHz : 1596.000 cache size : 12288 KB physical id : 1 siblings : 6 core id : 10 cpu cores : 6 apicid : 52 initial apicid : 52 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm arat dts tpr_shadow vnmi flexpriority ept vpid bogomips : 2254.43 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual > Which Windows' virtio drivers are you using? i used to use 0.1-16 and today also tried 0.1-22 from http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/ > Take a trace like described here http://www.linux-kvm.org/page/Tracing > (with -no-hpet please). will prepare this. > Try to use -cpu host,+x2apic. It may help Linux guest performance. Thanks, it improved throughput a little while lowering the cpu usage. Windows does not support this? Thanks Peter > -- > Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists 2012-02-20 19:15 ` Peter Lieven @ 2012-02-20 20:42 ` Gleb Natapov 0 siblings, 0 replies; 19+ messages in thread From: Gleb Natapov @ 2012-02-20 20:42 UTC (permalink / raw) To: Peter Lieven; +Cc: qemu-devel, kvm On Mon, Feb 20, 2012 at 08:15:15PM +0100, Peter Lieven wrote: > On 20.02.2012 19:40, Gleb Natapov wrote: > >On Mon, Feb 20, 2012 at 07:17:55PM +0100, Peter Lieven wrote: > >>Hi, > >> > >>I came a across an issue with a Windows 7 (32-bit) as well as with a > >>Windows 2008 R2 (64-bit) guest. > >> > >>If I transfer a file from the VM via CIFS or FTP to a remote machine, > >>i get very poor read performance (around 13MB/s). The VM peaks at 100% > >>cpu and I see a lot of insn_emulations and all kinds of exists in kvm_stat > >> > >>efer_reload 0 0 > >>exits 2260976 79620 > >>fpu_reload 6197 11 > >>halt_exits 114734 5011 > >>halt_wakeup 111195 4876 > >>host_state_reload 1499659 60962 > >>hypercalls 0 0 > >>insn_emulation 1577325 58488 > >>insn_emulation_fail 0 0 > >>invlpg 0 0 > >>io_exits 943949 40249 > >Hmm, too many of those. > > > >>irq_exits 108679 5434 > >>irq_injections 236545 10788 > >>irq_window 7606 246 > >>largepages 672 5 > >>mmio_exits 460020 16082 > >>mmu_cache_miss 119 0 > >>mmu_flooded 0 0 > >>mmu_pde_zapped 0 0 > >>mmu_pte_updated 0 0 > >>mmu_pte_write 13474 9 > >>mmu_recycled 0 0 > >>mmu_shadow_zapped 141 0 > >>mmu_unsync 0 0 > >>nmi_injections 0 0 > >>nmi_window 0 0 > >>pf_fixed 22803 35 > >>pf_guest 0 0 > >>remote_tlb_flush 239 2 > >>request_irq 0 0 > >>signal_exits 0 0 > >>tlb_flush 20933 0 > >> > >>If I run the same VM with a Ubuntu 10.04.4 guest I get around 60MB/s > >>throughput. The kvm_stats look a lot more sane. > >> > >>efer_reload 0 0 > >>exits 6132004 17931 > >>fpu_reload 19863 3 > >>halt_exits 264961 3083 > >>halt_wakeup 236468 2959 > >>host_state_reload 1104468 3104 > >>hypercalls 0 0 > >>insn_emulation 1417443 7518 > >>insn_emulation_fail 0 0 > >>invlpg 0 0 > >>io_exits 869380 2795 > >>irq_exits 253501 2362 > >>irq_injections 616967 6804 > >>irq_window 201186 2161 > >>largepages 1019 0 > >>mmio_exits 205268 0 > >>mmu_cache_miss 192 0 > >>mmu_flooded 0 0 > >>mmu_pde_zapped 0 0 > >>mmu_pte_updated 0 0 > >>mmu_pte_write 7440546 0 > >>mmu_recycled 0 0 > >>mmu_shadow_zapped 259 0 > >>mmu_unsync 0 0 > >>nmi_injections 0 0 > >>nmi_window 0 0 > >>pf_fixed 38529 30 > >>pf_guest 0 0 > >>remote_tlb_flush 761 1 > >>request_irq 0 0 > >>signal_exits 0 0 > >>tlb_flush 0 0 > >> > >>I use virtio-net (with vhost-net) and virtio-blk. I tried disabling > >>hpet (which basically illiminated the mmio_exits, but does not > >>increase > >>performance) and also commit (39a7a362e16bb27e98738d63f24d1ab5811e26a8 > >>) - no improvement. > >> > >>My commandline: > >>/usr/bin/qemu-kvm-1.0 -netdev > >>type=tap,id=guest8,script=no,downscript=no,ifname=tap0,vhost=on > >>-device virtio-net-pci,netdev=guest8,mac=52:54:00:ff:00:d3 -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-eeef4e007-a8a9f3818674f2fc-lieven-windows7-vc-r80788,if=virtio,cache=none,aio=native > >>-m 2048 -smp 2 -monitor tcp:0:4001,server,nowait -vnc :1 -name > >>lieven-win7-vc -boot order=dc,menu=off -k de -pidfile > >>/var/run/qemu/vm-187.pid -mem-path /hugepages -mem-prealloc -cpu > >>host -rtc base=localtime -vga std -usb -usbdevice tablet -no-hpet > >> > >>What further information is needed to debug this further? > >> > >Which kernel version (looks like something recent)? > 2.6.38 with kvm-kmod 3.2 > >Which host CPU (looks like something old)? > why? i guess its (quite) new. > Yes, it is. I misread the kvm_stat data and thought that CPU does not have ept, but it does. > vendor_id : GenuineIntel > cpu family : 6 > model : 44 > model name : Intel(R) Xeon(R) CPU L5640 @ 2.27GHz > stepping : 2 > cpu MHz : 1596.000 > cache size : 12288 KB > physical id : 1 > siblings : 6 > core id : 10 > cpu cores : 6 > apicid : 52 > initial apicid : 52 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts > rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 > monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 > sse4_2 popcnt aes lahf_lm arat dts tpr_shadow vnmi flexpriority ept > vpid > bogomips : 2254.43 > clflush size : 64 > cache_alignment : 64 > address sizes : 40 bits physical, 48 bits virtual > > >Which Windows' virtio drivers are you using? > i used to use 0.1-16 and today also tried 0.1-22 from > http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/ > >Take a trace like described here http://www.linux-kvm.org/page/Tracing > >(with -no-hpet please). > will prepare this. > >Try to use -cpu host,+x2apic. It may help Linux guest performance. > Thanks, it improved throughput a little while lowering the > cpu usage. Windows does not support this? > Not yet. It will support something similar when qemu will have hyper-v support. -- Gleb. ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2012-02-21 18:21 UTC | newest] Thread overview: 19+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-02-20 18:17 [Qemu-devel] win7 bad i/o performance, high insn_emulation and exists Peter Lieven 2012-02-20 18:40 ` Gleb Natapov 2012-02-20 19:04 ` Gleb Natapov 2012-02-20 19:24 ` Peter Lieven 2012-02-20 19:59 ` Peter Lieven 2012-02-20 20:45 ` Gleb Natapov 2012-02-21 10:50 ` Peter Lieven 2012-02-21 10:56 ` Gleb Natapov 2012-02-21 10:59 ` Peter Lieven 2012-02-21 11:00 ` Gleb Natapov 2012-02-21 11:16 ` Peter Lieven 2012-02-21 11:46 ` Gleb Natapov 2012-02-21 12:05 ` Peter Lieven 2012-02-21 13:56 ` Vadim Rozenfeld 2012-02-21 14:10 ` Peter Lieven 2012-02-21 16:48 ` Vadim Rozenfeld 2012-02-21 18:21 ` Peter Lieven 2012-02-20 19:15 ` Peter Lieven 2012-02-20 20:42 ` Gleb Natapov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).