apic-v reduce network performance in my test case

["Liuqiming (John)" <john.liuqiming@huawei.com>]

Hi all,

     Recently I met an odd performance problem: when I turn on APIC 
Virtualization feature (apicv=1), the network performance of a windows 
guest become worse.

     My test case like this: host only have one windows 2008 R2 HVM 
guest running,and this guest has a SR-IOV VF network passthrough to it.
Guest using this network access a NAS device. No fontend or backend of 
network and storage, all data transfered through network.

     The xentrace data shows: the mainly difference between apicv and 
non-apicv, is the way guest write apic registers, and 
EXIT_REASON_MSR_WRITE vmexit cost much more time than 
EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much 
less than using APIC-v.
This is the odd part,any ideas?

APIC-v OFF:
      4099582 VMEXIT   3467051359128 TSC HLT
     10135140 VMEXIT     42484175528 TSC WRMSR
      1651714 VMEXIT      9785961276 TSC I/O instruction
       532702 VMEXIT      3887971388 TSC External interrupt
       290546 VMEXIT      2262312440 TSC PAUSE
       588077 VMEXIT       914905312 TSC Control-register accesses
       383617 VMEXIT       453329940 TSC Exception or non-maskable 
interrupt (NMI)
       132717 VMEXIT       232289792 TSC Interrupt window
        25534 VMEXIT       198718764 TSC EPT violation
        53969 VMEXIT        62886752 TSC TPR below threshold
         7996 VMEXIT        34735376 TSC RDMSR
         1615 VMEXIT        16042768 TSC VMCALL
          147 VMEXIT          272320 TSC CPUID
            7 VMEXIT            6484 TSC WBINVD
            2 VMEXIT            6308 TSC MOV DR	

APIC-v ON:
      3717629 VMEXIT   3459905385332 TSC HLT
      2282403 VMEXIT     23099880196 TSC APIC write
      3900448 VMEXIT     13073253548 TSC PAUSE
      1643729 VMEXIT     11719626776 TSC I/O instruction
      2194667 VMEXIT      5321640708 TSC WRMSR
       214425 VMEXIT      2198994944 TSC External interrupt
       566795 VMEXIT      1940710108 TSC Control-register accesses
       342688 VMEXIT       659665532 TSC Exception or non-maskable 
interrupt (NMI)
       190623 VMEXIT       644411612 TSC VMCALL
       188657 VMEXIT       295956932 TSC Virtualized EOI
        24350 VMEXIT       194817152 TSC EPT violation
         4393 VMEXIT        23282044 TSC RDMSR
	 179 VMEXIT         1688676 TSC CPUID
            7 VMEXIT            6884 TSC WBINVD
            1 VMEXIT            4200 TSC MOV DR

In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V", 
msr based apic is disabled when apic-v is on, I wonder can they co-exist 
in some way? seems for windows guest msr-based apic has better performance.