apic-v reduce network performance in my test case

apic-v reduce network performance in my test case

[Liuqiming (John)]

Hi all,

     Recently I met an odd performance problem: when I turn on APIC 
Virtualization feature (apicv=1), the network performance of a windows 
guest become worse.

     My test case like this: host only have one windows 2008 R2 HVM 
guest running,and this guest has a SR-IOV VF network passthrough to it.
Guest using this network access a NAS device. No fontend or backend of 
network and storage, all data transfered through network.

     The xentrace data shows: the mainly difference between apicv and 
non-apicv, is the way guest write apic registers, and 
EXIT_REASON_MSR_WRITE vmexit cost much more time than 
EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much 
less than using APIC-v.
This is the odd part,any ideas?

APIC-v OFF:
      4099582 VMEXIT   3467051359128 TSC HLT
     10135140 VMEXIT     42484175528 TSC WRMSR
      1651714 VMEXIT      9785961276 TSC I/O instruction
       532702 VMEXIT      3887971388 TSC External interrupt
       290546 VMEXIT      2262312440 TSC PAUSE
       588077 VMEXIT       914905312 TSC Control-register accesses
       383617 VMEXIT       453329940 TSC Exception or non-maskable 
interrupt (NMI)
       132717 VMEXIT       232289792 TSC Interrupt window
        25534 VMEXIT       198718764 TSC EPT violation
        53969 VMEXIT        62886752 TSC TPR below threshold
         7996 VMEXIT        34735376 TSC RDMSR
         1615 VMEXIT        16042768 TSC VMCALL
          147 VMEXIT          272320 TSC CPUID
            7 VMEXIT            6484 TSC WBINVD
            2 VMEXIT            6308 TSC MOV DR	

APIC-v ON:
      3717629 VMEXIT   3459905385332 TSC HLT
      2282403 VMEXIT     23099880196 TSC APIC write
      3900448 VMEXIT     13073253548 TSC PAUSE
      1643729 VMEXIT     11719626776 TSC I/O instruction
      2194667 VMEXIT      5321640708 TSC WRMSR
       214425 VMEXIT      2198994944 TSC External interrupt
       566795 VMEXIT      1940710108 TSC Control-register accesses
       342688 VMEXIT       659665532 TSC Exception or non-maskable 
interrupt (NMI)
       190623 VMEXIT       644411612 TSC VMCALL
       188657 VMEXIT       295956932 TSC Virtualized EOI
        24350 VMEXIT       194817152 TSC EPT violation
         4393 VMEXIT        23282044 TSC RDMSR
	 179 VMEXIT         1688676 TSC CPUID
            7 VMEXIT            6884 TSC WBINVD
            1 VMEXIT            4200 TSC MOV DR

In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V", 
msr based apic is disabled when apic-v is on, I wonder can they co-exist 
in some way? seems for windows guest msr-based apic has better performance.

Re: apic-v reduce network performance in my test case

[Jan Beulich]

>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
>      Recently I met an odd performance problem: when I turn on APIC 
> Virtualization feature (apicv=1), the network performance of a windows 
> guest become worse.
> 
>      My test case like this: host only have one windows 2008 R2 HVM 
> guest running,and this guest has a SR-IOV VF network passthrough to it.
> Guest using this network access a NAS device. No fontend or backend of 
> network and storage, all data transfered through network.
> 
>      The xentrace data shows: the mainly difference between apicv and 
> non-apicv, is the way guest write apic registers, and 
> EXIT_REASON_MSR_WRITE vmexit cost much more time than 
> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much 
> less than using APIC-v.

There being heavier use of the pause VMEXIT doesn't by itself say
anything, I'm afraid. It may suggest that you have a C-state exit
latency problem - try lowering the maximum C-state allowed, or
disabling use of C-states altogether.

> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V", 
> msr based apic is disabled when apic-v is on, I wonder can they co-exist 
> in some way? seems for windows guest msr-based apic has better performance.

The whole purpose is to avoid the costly MSR access exits. Why
would you want to reintroduce that overhead?

Jan

Re: apic-v reduce network performance in my test case

[Liuqiming (John)]

Hi Jan,
Thanks for the reply.

On 2015/2/2 18:12, Jan Beulich wrote:
> >>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >       Recently I met an odd performance problem: when I turn on APIC
> > Virtualization feature (apicv=1), the network performance of a windows
> > guest become worse.
> >
> >       My test case like this: host only have one windows 2008 R2 HVM
> > guest running,and this guest has a SR-IOV VF network passthrough to it.
> > Guest using this network access a NAS device. No fontend or backend of
> > network and storage, all data transfered through network.
> >
> >       The xentrace data shows: the mainly difference between apicv and
> > non-apicv, is the way guest write apic registers, and
> > EXIT_REASON_MSR_WRITE vmexit cost much more time than
> > EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much
> > less than using APIC-v.
>
> There being heavier use of the pause VMEXIT doesn't by itself say
> anything, I'm afraid. It may suggest that you have a C-state exit
> latency problem - try lowering the maximum C-state allowed, or
> disabling use of C-states altogether.
Sorry, I forgot to mention  my test scenario:
Its a video test suite,I am not sure what the logic inside the tools exactly (not opensource tool).
The basic flow is:
      1) test suite start several thread to read video file from disk (from NAS through network in my case)
      2) decode these video data as a frame one by one
      3) if  any frame delay more than 40ms, then mark as lost

test result:
        apicv=1,  there can be 15 thread running at the same time without lost frame
        apicv=0,  there can be 22 thread running at the same time without lost frame

so when I'm saying apicv reduce the performance, I got the conclusion from the test result not from what xentrace shows.
>
> > In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> > "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V",
> > msr based apic is disabled when apic-v is on, I wonder can they co-exist
> > in some way? seems for windows guest msr-based apic has better performance.
>
> The whole purpose is to avoid the costly MSR access exits. Why
> would you want to reintroduce that overhead?
>
> Jan
>
I agree to avoid the MSR access vmexit by using apicv, I just do not know what's the side effect.
Because from the test result,  apicv replacing  msr-based access brings performance reduction.
>
> .
>

Re: apic-v reduce network performance in my test case

[Konrad Rzeszutek Wilk]

On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> Hi Jan,
> Thanks for the reply.
> 
> On 2015/2/2 18:12, Jan Beulich wrote:
> >>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >>       Recently I met an odd performance problem: when I turn on APIC
> >> Virtualization feature (apicv=1), the network performance of a windows
> >> guest become worse.
> >>
> >>       My test case like this: host only have one windows 2008 R2 HVM
> >> guest running,and this guest has a SR-IOV VF network passthrough to it.
> >> Guest using this network access a NAS device. No fontend or backend of
> >> network and storage, all data transfered through network.
> >>
> >>       The xentrace data shows: the mainly difference between apicv and
> >> non-apicv, is the way guest write apic registers, and
> >> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much
> >> less than using APIC-v.
> >
> >There being heavier use of the pause VMEXIT doesn't by itself say
> >anything, I'm afraid. It may suggest that you have a C-state exit
> >latency problem - try lowering the maximum C-state allowed, or
> >disabling use of C-states altogether.
> Sorry, I forgot to mention  my test scenario:
> Its a video test suite,I am not sure what the logic inside the tools exactly (not opensource tool).
> The basic flow is:
>      1) test suite start several thread to read video file from disk (from NAS through network in my case)
>      2) decode these video data as a frame one by one
>      3) if  any frame delay more than 40ms, then mark as lost
> 
> test result:
>        apicv=1,  there can be 15 thread running at the same time without lost frame
>        apicv=0,  there can be 22 thread running at the same time without lost frame
> 
> so when I'm saying apicv reduce the performance, I got the conclusion from the test result not from what xentrace shows.
> >
> >> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >> "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V",
> >> msr based apic is disabled when apic-v is on, I wonder can they co-exist
> >> in some way? seems for windows guest msr-based apic has better performance.
> >
> >The whole purpose is to avoid the costly MSR access exits. Why
> >would you want to reintroduce that overhead?
> >
> >Jan
> >
> I agree to avoid the MSR access vmexit by using apicv, I just do not know what's the side effect.
> Because from the test result,  apicv replacing  msr-based access brings performance reduction.

It sounds like it brings latency disruption, not neccessarily performance reduction.

What happens if you run with the cpufreq turned to performance, or as Jan suggested - 
with disabling C-states?

> >
> >.
> >
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: apic-v reduce network performance in my test case

[Liuqiming (John)]

On 2015/2/5 10:57, Liuqiming (John) wrote:
> sorry for late replay
>
> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> > On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> > > On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> > >> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> > >>> Hi Jan,
> > >>> Thanks for the reply.
> > >>>
> > >>> On 2015/2/2 18:12, Jan Beulich wrote:
> > >>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> > >>>>>         Recently I met an odd performance problem: when I turn
> > on APIC
> > >>>>> Virtualization feature (apicv=1), the network performance of a
> > windows
> > >>>>> guest become worse.
> > >>>>>
> > >>>>>         My test case like this: host only have one windows 2008
> > R2 HVM
> > >>>>> guest running,and this guest has a SR-IOV VF network passthrough
> > to it.
> > >>>>> Guest using this network access a NAS device. No fontend or
> > backend of
> > >>>>> network and storage, all data transfered through network.
> > >>>>>
> > >>>>>         The xentrace data shows: the mainly difference between
> > apicv and
> > >>>>> non-apicv, is the way guest write apic registers, and
> > >>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> > >>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> > is much
> > >>>>> less than using APIC-v.
> > >>>>
> > >>>> There being heavier use of the pause VMEXIT doesn't by itself say
> > >>>> anything, I'm afraid. It may suggest that you have a C-state exit
> > >>>> latency problem - try lowering the maximum C-state allowed, or
> > >>>> disabling use of C-states altogether.
> > >>> Sorry, I forgot to mention  my test scenario:
> > >>> Its a video test suite,I am not sure what the logic inside the
> > tools exactly (not opensource tool).
> > >>> The basic flow is:
> > >>>        1) test suite start several thread to read video file from
> > disk (from NAS through network in my case)
> > >>>        2) decode these video data as a frame one by one
> > >>>        3) if  any frame delay more than 40ms, then mark as lost
> > >>>
> > >>> test result:
> > >>>          apicv=1,  there can be 15 thread running at the same time
> > without lost frame
> > >>>          apicv=0,  there can be 22 thread running at the same time
> > without lost frame
> > >>>
> > >>> so when I'm saying apicv reduce the performance, I got the
> > conclusion from the test result not from what xentrace shows.
> > >>>>
> > >>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> > >>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> > APIC-V",
> > >>>>> msr based apic is disabled when apic-v is on, I wonder can they
> > co-exist
> > >>>>> in some way? seems for windows guest msr-based apic has better
> > performance.
> > >>>>
> > >>>> The whole purpose is to avoid the costly MSR access exits. Why
> > >>>> would you want to reintroduce that overhead?
> > >>>>
> > >>>> Jan
> > >>>>
> > >>> I agree to avoid the MSR access vmexit by using apicv, I just do
> > not know what's the side effect.
> > >>> Because from the test result,  apicv replacing  msr-based access
> > brings performance reduction.
> > >>
> > >> It sounds like it brings latency disruption, not neccessarily
> > performance reduction.
> > >>
> > >> What happens if you run with the cpufreq turned to performance, or
> > as Jan suggested -
> > >> with disabling C-states?
> > >>
> > > I have tried disable C-states in BIOS, the test result is still the
> > same.
> > > There is no frontend-backend device in my test, dom0 is not
> > involved, so you mean cpufreq in DomU, right?
> >
> > I meant on Xen (cpufreq=performance). But if you disable C-states then
> > that should
> > not matter.
> >
> > > I have no idea how to change cpufreq of a windows os but I will
> > figure out and try.
> >
> > That should not be needed.
> >
> > But back to your problem. When you say you have 'frame delay' - is it
> > because
> > of the NAS storage not getting the I/Os fast enough?
> >
> The  NAS storage should not be the bottleneck, it's a ramdisk connected
> with 10G network card.
> Besides, I'am using the same test environment, the only difference is
> whether apicv=1 or not.
>
> > What are you using as network device? Is it an SR-IOV device or an
> > emulated
> > device (e1000, ne2k, rtl81..?)?
> The vm using Intel Corporation 82599 Ethernet Controller Virtual
> Function passthrough network and has the corresponding driver installed.
> > >>
> > >>> _______________________________________________
> > >>> Xen-devel mailing list
> > >>> Xen-devel@lists.xen.org
> > >>> http://lists.xen.org/xen-devel
> > >>
> > >> .
> > >>
> > >
> When  tracing the performance problem, I found whether apicv is on
> change the result.
> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> presents or not is the key factor.
>
>
>
>
sorry, forget cc the list

Re: apic-v reduce network performance in my test case

[Konrad Rzeszutek Wilk]

On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote:
> On 2015/2/5 10:57, Liuqiming (John) wrote:
> >sorry for late replay
> >
> >On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> >> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> >> > On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> >> >> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> >> >>> Hi Jan,
> >> >>> Thanks for the reply.
> >> >>>
> >> >>> On 2015/2/2 18:12, Jan Beulich wrote:
> >> >>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >> >>>>>         Recently I met an odd performance problem: when I turn
> >> on APIC
> >> >>>>> Virtualization feature (apicv=1), the network performance of a
> >> windows
> >> >>>>> guest become worse.
> >> >>>>>
> >> >>>>>         My test case like this: host only have one windows 2008
> >> R2 HVM
> >> >>>>> guest running,and this guest has a SR-IOV VF network passthrough
> >> to it.
> >> >>>>> Guest using this network access a NAS device. No fontend or
> >> backend of
> >> >>>>> network and storage, all data transfered through network.
> >> >>>>>
> >> >>>>>         The xentrace data shows: the mainly difference between
> >> apicv and
> >> >>>>> non-apicv, is the way guest write apic registers, and
> >> >>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >> >>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> >> is much
> >> >>>>> less than using APIC-v.
> >> >>>>
> >> >>>> There being heavier use of the pause VMEXIT doesn't by itself say
> >> >>>> anything, I'm afraid. It may suggest that you have a C-state exit
> >> >>>> latency problem - try lowering the maximum C-state allowed, or
> >> >>>> disabling use of C-states altogether.
> >> >>> Sorry, I forgot to mention  my test scenario:
> >> >>> Its a video test suite,I am not sure what the logic inside the
> >> tools exactly (not opensource tool).
> >> >>> The basic flow is:
> >> >>>        1) test suite start several thread to read video file from
> >> disk (from NAS through network in my case)
> >> >>>        2) decode these video data as a frame one by one
> >> >>>        3) if  any frame delay more than 40ms, then mark as lost
> >> >>>
> >> >>> test result:
> >> >>>          apicv=1,  there can be 15 thread running at the same time
> >> without lost frame
> >> >>>          apicv=0,  there can be 22 thread running at the same time
> >> without lost frame
> >> >>>
> >> >>> so when I'm saying apicv reduce the performance, I got the
> >> conclusion from the test result not from what xentrace shows.
> >> >>>>
> >> >>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >> >>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> >> APIC-V",
> >> >>>>> msr based apic is disabled when apic-v is on, I wonder can they
> >> co-exist
> >> >>>>> in some way? seems for windows guest msr-based apic has better
> >> performance.
> >> >>>>
> >> >>>> The whole purpose is to avoid the costly MSR access exits. Why
> >> >>>> would you want to reintroduce that overhead?
> >> >>>>
> >> >>>> Jan
> >> >>>>
> >> >>> I agree to avoid the MSR access vmexit by using apicv, I just do
> >> not know what's the side effect.
> >> >>> Because from the test result,  apicv replacing  msr-based access
> >> brings performance reduction.
> >> >>
> >> >> It sounds like it brings latency disruption, not neccessarily
> >> performance reduction.
> >> >>
> >> >> What happens if you run with the cpufreq turned to performance, or
> >> as Jan suggested -
> >> >> with disabling C-states?
> >> >>
> >> > I have tried disable C-states in BIOS, the test result is still the
> >> same.
> >> > There is no frontend-backend device in my test, dom0 is not
> >> involved, so you mean cpufreq in DomU, right?
> >>
> >> I meant on Xen (cpufreq=performance). But if you disable C-states then
> >> that should
> >> not matter.
> >>
> >> > I have no idea how to change cpufreq of a windows os but I will
> >> figure out and try.
> >>
> >> That should not be needed.
> >>
> >> But back to your problem. When you say you have 'frame delay' - is it
> >> because
> >> of the NAS storage not getting the I/Os fast enough?
> >>
> >The  NAS storage should not be the bottleneck, it's a ramdisk connected
> >with 10G network card.
> >Besides, I'am using the same test environment, the only difference is
> >whether apicv=1 or not.
> >
> >> What are you using as network device? Is it an SR-IOV device or an
> >> emulated
> >> device (e1000, ne2k, rtl81..?)?
> >The vm using Intel Corporation 82599 Ethernet Controller Virtual
> >Function passthrough network and has the corresponding driver installed.
> >> >>
> >> >>> _______________________________________________
> >> >>> Xen-devel mailing list
> >> >>> Xen-devel@lists.xen.org
> >> >>> http://lists.xen.org/xen-devel
> >> >>
> >> >> .
> >> >>
> >> >
> >When  tracing the performance problem, I found whether apicv is on
> >change the result.
> >And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> >presents or not is the key factor.

Ah, willing to send out a patch for that?
> >
> >
> >
> >
> sorry, forget cc the list
> 
> 

Re: apic-v reduce network performance in my test case

[Liuqiming (John)]

On 2015/2/6 3:26, Konrad Rzeszutek Wilk wrote:
> On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote:
> > On 2015/2/5 10:57, Liuqiming (John) wrote:
> >> sorry for late replay
> >>
> >> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> >>> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> >>>> On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> >>>>> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> >>>>>> Hi Jan,
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> On 2015/2/2 18:12, Jan Beulich wrote:
> >>>>>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >>>>>>>>          Recently I met an odd performance problem: when I turn
> >>> on APIC
> >>>>>>>> Virtualization feature (apicv=1), the network performance of a
> >>> windows
> >>>>>>>> guest become worse.
> >>>>>>>>
> >>>>>>>>          My test case like this: host only have one windows 2008
> >>> R2 HVM
> >>>>>>>> guest running,and this guest has a SR-IOV VF network passthrough
> >>> to it.
> >>>>>>>> Guest using this network access a NAS device. No fontend or
> >>> backend of
> >>>>>>>> network and storage, all data transfered through network.
> >>>>>>>>
> >>>>>>>>          The xentrace data shows: the mainly difference between
> >>> apicv and
> >>>>>>>> non-apicv, is the way guest write apic registers, and
> >>>>>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >>>>>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> >>> is much
> >>>>>>>> less than using APIC-v.
> >>>>>>>
> >>>>>>> There being heavier use of the pause VMEXIT doesn't by itself say
> >>>>>>> anything, I'm afraid. It may suggest that you have a C-state exit
> >>>>>>> latency problem - try lowering the maximum C-state allowed, or
> >>>>>>> disabling use of C-states altogether.
> >>>>>> Sorry, I forgot to mention  my test scenario:
> >>>>>> Its a video test suite,I am not sure what the logic inside the
> >>> tools exactly (not opensource tool).
> >>>>>> The basic flow is:
> >>>>>>         1) test suite start several thread to read video file from
> >>> disk (from NAS through network in my case)
> >>>>>>         2) decode these video data as a frame one by one
> >>>>>>         3) if  any frame delay more than 40ms, then mark as lost
> >>>>>>
> >>>>>> test result:
> >>>>>>           apicv=1,  there can be 15 thread running at the same time
> >>> without lost frame
> >>>>>>           apicv=0,  there can be 22 thread running at the same time
> >>> without lost frame
> >>>>>>
> >>>>>> so when I'm saying apicv reduce the performance, I got the
> >>> conclusion from the test result not from what xentrace shows.
> >>>>>>>
> >>>>>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >>>>>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> >>> APIC-V",
> >>>>>>>> msr based apic is disabled when apic-v is on, I wonder can they
> >>> co-exist
> >>>>>>>> in some way? seems for windows guest msr-based apic has better
> >>> performance.
> >>>>>>>
> >>>>>>> The whole purpose is to avoid the costly MSR access exits. Why
> >>>>>>> would you want to reintroduce that overhead?
> >>>>>>>
> >>>>>>> Jan
> >>>>>>>
> >>>>>> I agree to avoid the MSR access vmexit by using apicv, I just do
> >>> not know what's the side effect.
> >>>>>> Because from the test result,  apicv replacing  msr-based access
> >>> brings performance reduction.
> >>>>>
> >>>>> It sounds like it brings latency disruption, not neccessarily
> >>> performance reduction.
> >>>>>
> >>>>> What happens if you run with the cpufreq turned to performance, or
> >>> as Jan suggested -
> >>>>> with disabling C-states?
> >>>>>
> >>>> I have tried disable C-states in BIOS, the test result is still the
> >>> same.
> >>>> There is no frontend-backend device in my test, dom0 is not
> >>> involved, so you mean cpufreq in DomU, right?
> >>>
> >>> I meant on Xen (cpufreq=performance). But if you disable C-states then
> >>> that should
> >>> not matter.
> >>>
> >>>> I have no idea how to change cpufreq of a windows os but I will
> >>> figure out and try.
> >>>
> >>> That should not be needed.
> >>>
> >>> But back to your problem. When you say you have 'frame delay' - is it
> >>> because
> >>> of the NAS storage not getting the I/Os fast enough?
> >>>
> >> The  NAS storage should not be the bottleneck, it's a ramdisk connected
> >> with 10G network card.
> >> Besides, I'am using the same test environment, the only difference is
> >> whether apicv=1 or not.
> >>
> >>> What are you using as network device? Is it an SR-IOV device or an
> >>> emulated
> >>> device (e1000, ne2k, rtl81..?)?
> >> The vm using Intel Corporation 82599 Ethernet Controller Virtual
> >> Function passthrough network and has the corresponding driver installed.
> >>>>>
> >>>>>> _______________________________________________
> >>>>>> Xen-devel mailing list
> >>>>>> Xen-devel@lists.xen.org
> >>>>>> http://lists.xen.org/xen-devel
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >> When  tracing the performance problem, I found whether apicv is on
> >> change the result.
> >> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> >> presents or not is the key factor.
>
> Ah, willing to send out a patch for that?
I have not get the conclusion: why CPUID4A_MSR_BASED_APIC affect performance.
But I will continue to dig in, and happy to send out what I found.
> >>
> >>
> >>
> >>
> > sorry, forget cc the list
> >
> >
>
for testing purpose, reverting this commit will do:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=7f2e992b824ec62a2818e64390ac2ccfbd74e6b7

"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V"

> .
>