From mboxrd@z Thu Jan 1 00:00:00 1970 From: zhanghailiang Subject: Re: [PATCH] KVM: add KVM_CAP_VMX_APICV to advertise hardware apic-v support Date: Thu, 11 Dec 2014 20:53:44 +0800 Message-ID: <548993D8.6010004@huawei.com> References: <1418285221-14256-1-git-send-email-zhang.zhanghailiang@huawei.com> <5489849C.4010308@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Cc: , To: Paolo Bonzini , Return-path: Received: from szxga03-in.huawei.com ([119.145.14.66]:58172 "EHLO szxga03-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932622AbaLKMyQ (ORCPT ); Thu, 11 Dec 2014 07:54:16 -0500 In-Reply-To: <5489849C.4010308@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On 2014/12/11 19:48, Paolo Bonzini wrote: > > > On 11/12/2014 09:07, zhanghailiang wrote: >> User space (i.e. QEMU) should be able to check whether KVM >> supports apic-v. User space will use this to decide whether enable >> emulated MSR-based APIC (i.e. hyperv-vapic). > > Did you see an improvement in some workload? > Yes, compared with using hyperv-vapic, using apic-v reduces lots of VM exit. The bellow is the qemu command and perf data: qemu/x86_64-softmmu/qemu-system-x86_64 -name win2008_st_r2_64_2U -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -m 4096 -realtime mlock=off -smp 6,maxcpus=12,sockets=1,cores=12,threads=1 -uuid e7fca4b5-10a2-4fa5-a71b-b5e9eaa41d4d -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008_st_r2_64_2U.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/data/win2008_st_r2_64_2U.DataCenter.40G.base.virtio.02,if=none,id=drive-ide0-0-0,format=raw,cache=none, aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=! 52:54:00 :48:70:80,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,vgamem_mb=8,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on (note: -cpu is Haswell, so x2apic is enable in default) (1) Using Hyperv-vapic UVP:/data # perf stat -e "kvm:*" -a sleep 10s Performance counter stats for 'sleep 10s': 1215390 kvm:kvm_entry [100.00%] 0 kvm:kvm_hypercall [100.00%] 0 kvm:kvm_hv_hypercall [100.00%] 8397 kvm:kvm_pio [100.00%] 0 kvm:kvm_cpuid [100.00%] 693288 kvm:kvm_apic [100.00%] 1228026 kvm:kvm_exit [100.00%] 0 kvm:kvm_inj_virq [100.00%] 3 kvm:kvm_inj_exception [100.00%] 39 kvm:kvm_page_fault [100.00%] 536450 kvm:kvm_msr [100.00%] 0 kvm:kvm_cr [100.00%] 1340 kvm:kvm_pic_set_irq [100.00%] 164441 kvm:kvm_apic_ipi [100.00%] 376239 kvm:kvm_apic_accept_irq [100.00%] 377100 kvm:kvm_eoi [100.00%] 0 kvm:kvm_pv_eoi [100.00%] 0 kvm:kvm_nested_vmrun [100.00%] 0 kvm:kvm_nested_intercepts [100.00%] 0 kvm:kvm_nested_vmexit [100.00%] 0 kvm:kvm_nested_vmexit_inject [100.00%] 0 kvm:kvm_nested_intr_vmexit [100.00%] 0 kvm:kvm_invlpga [100.00%] 0 kvm:kvm_skinit [100.00%] 528 kvm:kvm_emulate_insn [100.00%] 444 kvm:vcpu_match_mmio [100.00%] 0 kvm:kvm_write_tsc_offset [100.00%] 0 kvm:kvm_update_master_clock [100.00%] 0 kvm:kvm_track_tsc [100.00%] 630 kvm:kvm_userspace_exit [100.00%] 1394 kvm:kvm_set_irq [100.00%] 1396 kvm:kvm_ioapic_set_irq [100.00%] 212003 kvm:kvm_msi_set_irq [100.00%] 4092 kvm:kvm_ack_irq [100.00%] 901 kvm:kvm_mmio [100.00%] 710971 kvm:kvm_fpu [100.00%] 0 kvm:kvm_age_page [100.00%] 0 kvm:kvm_try_async_get_page [100.00%] 0 kvm:kvm_async_pf_doublefault [100.00%] 0 kvm:kvm_async_pf_not_present [100.00%] 0 kvm:kvm_async_pf_ready [100.00%] 0 kvm:kvm_async_pf_completed 10.000959015 seconds time elapsed (2) Using apic-v perf stat -e "kvm:*" -a sleep 10s Performance counter stats for 'sleep 10s': 888948 kvm:kvm_entry [100.00%] 0 kvm:kvm_hypercall [100.00%] 6 kvm:kvm_hv_hypercall [100.00%] 12510 kvm:kvm_pio [100.00%] 0 kvm:kvm_cpuid [100.00%] 324926 kvm:kvm_apic [100.00%] 898841 kvm:kvm_exit [100.00%] 0 kvm:kvm_inj_virq [100.00%] 0 kvm:kvm_inj_exception [100.00%] 259 kvm:kvm_page_fault [100.00%] 60 kvm:kvm_msr [100.00%] 0 kvm:kvm_cr [100.00%] 2046 kvm:kvm_pic_set_irq [100.00%] 164641 kvm:kvm_apic_ipi [100.00%] 371867 kvm:kvm_apic_accept_irq [100.00%] 665 kvm:kvm_eoi [100.00%] 0 kvm:kvm_pv_eoi [100.00%] 0 kvm:kvm_nested_vmrun [100.00%] 0 kvm:kvm_nested_intercepts [100.00%] 0 kvm:kvm_nested_vmexit [100.00%] 0 kvm:kvm_nested_vmexit_inject [100.00%] 0 kvm:kvm_nested_intr_vmexit [100.00%] 0 kvm:kvm_invlpga [100.00%] 0 kvm:kvm_skinit [100.00%] 1418 kvm:kvm_emulate_insn [100.00%] 465 kvm:vcpu_match_mmio [100.00%] 0 kvm:kvm_write_tsc_offset [100.00%] 0 kvm:kvm_update_master_clock [100.00%] 0 kvm:kvm_track_tsc [100.00%] 368303 kvm:kvm_ple_window [100.00%] 3948 kvm:kvm_userspace_exit [100.00%] 2088 kvm:kvm_set_irq [100.00%] 2090 kvm:kvm_ioapic_set_irq [100.00%] 0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%] 208975 kvm:kvm_msi_set_irq [100.00%] 688 kvm:kvm_ack_irq [100.00%] 949 kvm:kvm_mmio [100.00%] 743624 kvm:kvm_fpu [100.00%] 0 kvm:kvm_age_page [100.00%] 0 kvm:kvm_try_async_get_page [100.00%] 0 kvm:kvm_async_pf_doublefault [100.00%] 0 kvm:kvm_async_pf_not_present [100.00%] 0 kvm:kvm_async_pf_ready [100.00%] 0 kvm:kvm_async_pf_completed 10.000829758 seconds time elapsed > I think it's a Windows bug---it should prefer x2apic to hv-vapic if both > are available. > No, i don't think it is a windows bug, it has nothing to do with x2apic, but apic-v (need hardware support, i.e. Haswell cpu). When we don't passthough host cpu model to Guest os, it has no idea about whether it supports apic-v in host, so the choose should be done by qemu. Actually, qemu has a option 'hv_vapic' for -cpu, we can choose not to configure it if we know there is apic-v support in host. But IMHO, we'd better to do it automatically. PS: This is the perf data when we configure x2apic but without configure hv-vapic and disable apic-v in kvm. It is the worst result, compared with the above two cases. UVP:/data # perf stat -e "kvm:*" -a sleep 10s Performance counter stats for 'sleep 10s': 1616665 kvm:kvm_entry [100.00%] 0 kvm:kvm_hypercall [100.00%] 0 kvm:kvm_hv_hypercall [100.00%] 8766 kvm:kvm_pio [100.00%] 0 kvm:kvm_cpuid [100.00%] 896485 kvm:kvm_apic [100.00%] 1634073 kvm:kvm_exit [100.00%] 363649 kvm:kvm_inj_virq [100.00%] 0 kvm:kvm_inj_exception [100.00%] 38 kvm:kvm_page_fault [100.00%] 0 kvm:kvm_msr [100.00%] 0 kvm:kvm_cr [100.00%] 1444 kvm:kvm_pic_set_irq [100.00%] 181544 kvm:kvm_apic_ipi [100.00%] 368012 kvm:kvm_apic_accept_irq [100.00%] 368745 kvm:kvm_eoi [100.00%] 0 kvm:kvm_pv_eoi [100.00%] 0 kvm:kvm_nested_vmrun [100.00%] 0 kvm:kvm_nested_intercepts [100.00%] 0 kvm:kvm_nested_vmexit [100.00%] 0 kvm:kvm_nested_vmexit_inject [100.00%] 0 kvm:kvm_nested_intr_vmexit [100.00%] 0 kvm:kvm_invlpga [100.00%] 0 kvm:kvm_skinit [100.00%] 556213 kvm:kvm_emulate_insn [100.00%] 463 kvm:vcpu_match_mmio [100.00%] 0 kvm:kvm_write_tsc_offset [100.00%] 0 kvm:kvm_update_master_clock [100.00%] 0 kvm:kvm_track_tsc [100.00%] 378556 kvm:kvm_ple_window [100.00%] 1125 kvm:kvm_userspace_exit [100.00%] 1490 kvm:kvm_set_irq [100.00%] 1492 kvm:kvm_ioapic_set_irq [100.00%] 0 kvm:kvm_ioapic_delayed_eoi_inj [100.00%] 187316 kvm:kvm_msi_set_irq [100.00%] 4137 kvm:kvm_ack_irq [100.00%] 568249 kvm:kvm_mmio [100.00%] 763264 kvm:kvm_fpu [100.00%] 0 kvm:kvm_age_page [100.00%] 0 kvm:kvm_try_async_get_page [100.00%] 0 kvm:kvm_async_pf_doublefault [100.00%] 0 kvm:kvm_async_pf_not_present [100.00%] 0 kvm:kvm_async_pf_ready [100.00%] 0 kvm:kvm_async_pf_completed 10.001000787 seconds time elapsed Thanks, zhanghailiang