From: zhanghailiang <zhang.zhanghailiang@huawei.com>
To: Paolo Bonzini <pbonzini@redhat.com>, <kvm@vger.kernel.org>
Cc: <gleb@kernel.org>, <peter.huangpeng@huawei.com>
Subject: Re: [PATCH] KVM: add KVM_CAP_VMX_APICV to advertise hardware apic-v support
Date: Thu, 11 Dec 2014 20:53:44 +0800 [thread overview]
Message-ID: <548993D8.6010004@huawei.com> (raw)
In-Reply-To: <5489849C.4010308@redhat.com>
On 2014/12/11 19:48, Paolo Bonzini wrote:
>
>
> On 11/12/2014 09:07, zhanghailiang wrote:
>> User space (i.e. QEMU) should be able to check whether KVM
>> supports apic-v. User space will use this to decide whether enable
>> emulated MSR-based APIC (i.e. hyperv-vapic).
>
> Did you see an improvement in some workload?
>
Yes, compared with using hyperv-vapic, using apic-v reduces lots of VM exit.
The bellow is the qemu command and perf data:
qemu/x86_64-softmmu/qemu-system-x86_64 -name win2008_st_r2_64_2U -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -m 4096 -realtime mlock=off -smp 6,maxcpus=12,sockets=1,cores=12,threads=1 -uuid e7fca4b5-10a2-4fa5-a71b-b5e9eaa41d4d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_st_r2_64_2U.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/data/win2008_st_r2_64_2U.DataCenter.40G.base.virtio.02,if=none,id=drive-ide0-0-0,format=raw,cache=none,
aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=!
52:54:00
:48:70:80,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,vgamem_mb=8,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
(note: -cpu is Haswell, so x2apic is enable in default)
(1) Using Hyperv-vapic
UVP:/data # perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
1215390 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
0 kvm:kvm_hv_hypercall [100.00%]
8397 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
693288 kvm:kvm_apic [100.00%]
1228026 kvm:kvm_exit [100.00%]
0 kvm:kvm_inj_virq [100.00%]
3 kvm:kvm_inj_exception [100.00%]
39 kvm:kvm_page_fault [100.00%]
536450 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
1340 kvm:kvm_pic_set_irq [100.00%]
164441 kvm:kvm_apic_ipi [100.00%]
376239 kvm:kvm_apic_accept_irq [100.00%]
377100 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
528 kvm:kvm_emulate_insn [100.00%]
444 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
630 kvm:kvm_userspace_exit [100.00%]
1394 kvm:kvm_set_irq [100.00%]
1396 kvm:kvm_ioapic_set_irq [100.00%]
212003 kvm:kvm_msi_set_irq [100.00%]
4092 kvm:kvm_ack_irq [100.00%]
901 kvm:kvm_mmio [100.00%]
710971 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.000959015 seconds time elapsed
(2) Using apic-v
perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
888948 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
6 kvm:kvm_hv_hypercall [100.00%]
12510 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
324926 kvm:kvm_apic [100.00%]
898841 kvm:kvm_exit [100.00%]
0 kvm:kvm_inj_virq [100.00%]
0 kvm:kvm_inj_exception [100.00%]
259 kvm:kvm_page_fault [100.00%]
60 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
2046 kvm:kvm_pic_set_irq [100.00%]
164641 kvm:kvm_apic_ipi [100.00%]
371867 kvm:kvm_apic_accept_irq [100.00%]
665 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
1418 kvm:kvm_emulate_insn [100.00%]
465 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
368303 kvm:kvm_ple_window [100.00%]
3948 kvm:kvm_userspace_exit [100.00%]
2088 kvm:kvm_set_irq [100.00%]
2090 kvm:kvm_ioapic_set_irq [100.00%]
0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%]
208975 kvm:kvm_msi_set_irq [100.00%]
688 kvm:kvm_ack_irq [100.00%]
949 kvm:kvm_mmio [100.00%]
743624 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.000829758 seconds time elapsed
> I think it's a Windows bug---it should prefer x2apic to hv-vapic if both
> are available.
>
No, i don't think it is a windows bug, it has nothing to do with x2apic, but apic-v
(need hardware support, i.e. Haswell cpu). When we don't passthough host cpu model to Guest os,
it has no idea about whether it supports apic-v in host, so the choose should be done by qemu.
Actually, qemu has a option 'hv_vapic' for -cpu, we can choose not to configure it if we know
there is apic-v support in host. But IMHO, we'd better to do it automatically.
PS:
This is the perf data when we configure x2apic but without configure hv-vapic and disable apic-v in kvm.
It is the worst result, compared with the above two cases.
UVP:/data # perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
1616665 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
0 kvm:kvm_hv_hypercall [100.00%]
8766 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
896485 kvm:kvm_apic [100.00%]
1634073 kvm:kvm_exit [100.00%]
363649 kvm:kvm_inj_virq [100.00%]
0 kvm:kvm_inj_exception [100.00%]
38 kvm:kvm_page_fault [100.00%]
0 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
1444 kvm:kvm_pic_set_irq [100.00%]
181544 kvm:kvm_apic_ipi [100.00%]
368012 kvm:kvm_apic_accept_irq [100.00%]
368745 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
556213 kvm:kvm_emulate_insn [100.00%]
463 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
378556 kvm:kvm_ple_window [100.00%]
1125 kvm:kvm_userspace_exit [100.00%]
1490 kvm:kvm_set_irq [100.00%]
1492 kvm:kvm_ioapic_set_irq [100.00%]
0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%]
187316 kvm:kvm_msi_set_irq [100.00%]
4137 kvm:kvm_ack_irq [100.00%]
568249 kvm:kvm_mmio [100.00%]
763264 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.001000787 seconds time elapsed
Thanks,
zhanghailiang
next prev parent reply other threads:[~2014-12-11 12:54 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-11 8:07 [PATCH] KVM: add KVM_CAP_VMX_APICV to advertise hardware apic-v support zhanghailiang
2014-12-11 11:48 ` Paolo Bonzini
2014-12-11 12:53 ` zhanghailiang [this message]
2014-12-11 13:28 ` Radim Krčmář
2014-12-11 17:15 ` Paolo Bonzini
2014-12-17 22:17 ` Stefan Fritsch
2014-12-18 3:37 ` zhanghailiang
2014-12-18 8:30 ` Paolo Bonzini
2014-12-18 8:57 ` Stefan Fritsch
2014-12-18 9:23 ` Paolo Bonzini
2014-12-11 13:08 ` Radim Krčmář
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=548993D8.6010004@huawei.com \
--to=zhang.zhanghailiang@huawei.com \
--cc=gleb@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=peter.huangpeng@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox