From: zhanghailiang <zhang.zhanghailiang@huawei.com>
To: Paolo Bonzini <pbonzini@redhat.com>, <kvm@vger.kernel.org>
Cc: <gleb@kernel.org>, <peter.huangpeng@huawei.com>
Subject: Re: [PATCH] KVM: add KVM_CAP_VMX_APICV to advertise hardware apic-v support
Date: Thu, 11 Dec 2014 20:53:44 +0800 [thread overview]
Message-ID: <548993D8.6010004@huawei.com> (raw)
In-Reply-To: <5489849C.4010308@redhat.com>
On 2014/12/11 19:48, Paolo Bonzini wrote:
>
>
> On 11/12/2014 09:07, zhanghailiang wrote:
>> User space (i.e. QEMU) should be able to check whether KVM
>> supports apic-v. User space will use this to decide whether enable
>> emulated MSR-based APIC (i.e. hyperv-vapic).
>
> Did you see an improvement in some workload?
>
Yes, compared with using hyperv-vapic, using apic-v reduces lots of VM exit.
The bellow is the qemu command and perf data:
qemu/x86_64-softmmu/qemu-system-x86_64 -name win2008_st_r2_64_2U -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -m 4096 -realtime mlock=off -smp 6,maxcpus=12,sockets=1,cores=12,threads=1 -uuid e7fca4b5-10a2-4fa5-a71b-b5e9eaa41d4d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_st_r2_64_2U.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/data/win2008_st_r2_64_2U.DataCenter.40G.base.virtio.02,if=none,id=drive-ide0-0-0,format=raw,cache=none,
aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=!
52:54:00
:48:70:80,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,vgamem_mb=8,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
(note: -cpu is Haswell, so x2apic is enable in default)
(1) Using Hyperv-vapic
UVP:/data # perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
1215390 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
0 kvm:kvm_hv_hypercall [100.00%]
8397 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
693288 kvm:kvm_apic [100.00%]
1228026 kvm:kvm_exit [100.00%]
0 kvm:kvm_inj_virq [100.00%]
3 kvm:kvm_inj_exception [100.00%]
39 kvm:kvm_page_fault [100.00%]
536450 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
1340 kvm:kvm_pic_set_irq [100.00%]
164441 kvm:kvm_apic_ipi [100.00%]
376239 kvm:kvm_apic_accept_irq [100.00%]
377100 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
528 kvm:kvm_emulate_insn [100.00%]
444 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
630 kvm:kvm_userspace_exit [100.00%]
1394 kvm:kvm_set_irq [100.00%]
1396 kvm:kvm_ioapic_set_irq [100.00%]
212003 kvm:kvm_msi_set_irq [100.00%]
4092 kvm:kvm_ack_irq [100.00%]
901 kvm:kvm_mmio [100.00%]
710971 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.000959015 seconds time elapsed
(2) Using apic-v
perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
888948 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
6 kvm:kvm_hv_hypercall [100.00%]
12510 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
324926 kvm:kvm_apic [100.00%]
898841 kvm:kvm_exit [100.00%]
0 kvm:kvm_inj_virq [100.00%]
0 kvm:kvm_inj_exception [100.00%]
259 kvm:kvm_page_fault [100.00%]
60 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
2046 kvm:kvm_pic_set_irq [100.00%]
164641 kvm:kvm_apic_ipi [100.00%]
371867 kvm:kvm_apic_accept_irq [100.00%]
665 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
1418 kvm:kvm_emulate_insn [100.00%]
465 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
368303 kvm:kvm_ple_window [100.00%]
3948 kvm:kvm_userspace_exit [100.00%]
2088 kvm:kvm_set_irq [100.00%]
2090 kvm:kvm_ioapic_set_irq [100.00%]
0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%]
208975 kvm:kvm_msi_set_irq [100.00%]
688 kvm:kvm_ack_irq [100.00%]
949 kvm:kvm_mmio [100.00%]
743624 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.000829758 seconds time elapsed
> I think it's a Windows bug---it should prefer x2apic to hv-vapic if both
> are available.
>
No, i don't think it is a windows bug, it has nothing to do with x2apic, but apic-v
(need hardware support, i.e. Haswell cpu). When we don't passthough host cpu model to Guest os,
it has no idea about whether it supports apic-v in host, so the choose should be done by qemu.
Actually, qemu has a option 'hv_vapic' for -cpu, we can choose not to configure it if we know
there is apic-v support in host. But IMHO, we'd better to do it automatically.
PS:
This is the perf data when we configure x2apic but without configure hv-vapic and disable apic-v in kvm.
It is the worst result, compared with the above two cases.
UVP:/data # perf stat -e "kvm:*" -a sleep 10s
Performance counter stats for 'sleep 10s':
1616665 kvm:kvm_entry [100.00%]
0 kvm:kvm_hypercall [100.00%]
0 kvm:kvm_hv_hypercall [100.00%]
8766 kvm:kvm_pio [100.00%]
0 kvm:kvm_cpuid [100.00%]
896485 kvm:kvm_apic [100.00%]
1634073 kvm:kvm_exit [100.00%]
363649 kvm:kvm_inj_virq [100.00%]
0 kvm:kvm_inj_exception [100.00%]
38 kvm:kvm_page_fault [100.00%]
0 kvm:kvm_msr [100.00%]
0 kvm:kvm_cr [100.00%]
1444 kvm:kvm_pic_set_irq [100.00%]
181544 kvm:kvm_apic_ipi [100.00%]
368012 kvm:kvm_apic_accept_irq [100.00%]
368745 kvm:kvm_eoi [100.00%]
0 kvm:kvm_pv_eoi [100.00%]
0 kvm:kvm_nested_vmrun [100.00%]
0 kvm:kvm_nested_intercepts [100.00%]
0 kvm:kvm_nested_vmexit [100.00%]
0 kvm:kvm_nested_vmexit_inject [100.00%]
0 kvm:kvm_nested_intr_vmexit [100.00%]
0 kvm:kvm_invlpga [100.00%]
0 kvm:kvm_skinit [100.00%]
556213 kvm:kvm_emulate_insn [100.00%]
463 kvm:vcpu_match_mmio [100.00%]
0 kvm:kvm_write_tsc_offset [100.00%]
0 kvm:kvm_update_master_clock [100.00%]
0 kvm:kvm_track_tsc [100.00%]
378556 kvm:kvm_ple_window [100.00%]
1125 kvm:kvm_userspace_exit [100.00%]
1490 kvm:kvm_set_irq [100.00%]
1492 kvm:kvm_ioapic_set_irq [100.00%]
0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%]
187316 kvm:kvm_msi_set_irq [100.00%]
4137 kvm:kvm_ack_irq [100.00%]
568249 kvm:kvm_mmio [100.00%]
763264 kvm:kvm_fpu [100.00%]
0 kvm:kvm_age_page [100.00%]
0 kvm:kvm_try_async_get_page [100.00%]
0 kvm:kvm_async_pf_doublefault [100.00%]
0 kvm:kvm_async_pf_not_present [100.00%]
0 kvm:kvm_async_pf_ready [100.00%]
0 kvm:kvm_async_pf_completed
10.001000787 seconds time elapsed
Thanks,
zhanghailiang
next prev parent reply other threads:[~2014-12-11 12:54 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-11 8:07 [PATCH] KVM: add KVM_CAP_VMX_APICV to advertise hardware apic-v support zhanghailiang
2014-12-11 11:48 ` Paolo Bonzini
2014-12-11 12:53 ` zhanghailiang [this message]
2014-12-11 13:28 ` Radim Krčmář
2014-12-11 17:15 ` Paolo Bonzini
2014-12-17 22:17 ` Stefan Fritsch
2014-12-18 3:37 ` zhanghailiang
2014-12-18 8:30 ` Paolo Bonzini
2014-12-18 8:57 ` Stefan Fritsch
2014-12-18 9:23 ` Paolo Bonzini
2014-12-11 13:08 ` Radim Krčmář
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=548993D8.6010004@huawei.com \
--to=zhang.zhanghailiang@huawei.com \
--cc=gleb@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=peter.huangpeng@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.