Re: apic-v reduce network performance in my test case

["Liuqiming (John)" <john.liuqiming@huawei.com>]

On 2015/2/5 10:57, Liuqiming (John) wrote:
> sorry for late replay
>
> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> > On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> > > On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> > >> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> > >>> Hi Jan,
> > >>> Thanks for the reply.
> > >>>
> > >>> On 2015/2/2 18:12, Jan Beulich wrote:
> > >>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> > >>>>>         Recently I met an odd performance problem: when I turn
> > on APIC
> > >>>>> Virtualization feature (apicv=1), the network performance of a
> > windows
> > >>>>> guest become worse.
> > >>>>>
> > >>>>>         My test case like this: host only have one windows 2008
> > R2 HVM
> > >>>>> guest running,and this guest has a SR-IOV VF network passthrough
> > to it.
> > >>>>> Guest using this network access a NAS device. No fontend or
> > backend of
> > >>>>> network and storage, all data transfered through network.
> > >>>>>
> > >>>>>         The xentrace data shows: the mainly difference between
> > apicv and
> > >>>>> non-apicv, is the way guest write apic registers, and
> > >>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> > >>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> > is much
> > >>>>> less than using APIC-v.
> > >>>>
> > >>>> There being heavier use of the pause VMEXIT doesn't by itself say
> > >>>> anything, I'm afraid. It may suggest that you have a C-state exit
> > >>>> latency problem - try lowering the maximum C-state allowed, or
> > >>>> disabling use of C-states altogether.
> > >>> Sorry, I forgot to mention  my test scenario:
> > >>> Its a video test suite,I am not sure what the logic inside the
> > tools exactly (not opensource tool).
> > >>> The basic flow is:
> > >>>        1) test suite start several thread to read video file from
> > disk (from NAS through network in my case)
> > >>>        2) decode these video data as a frame one by one
> > >>>        3) if  any frame delay more than 40ms, then mark as lost
> > >>>
> > >>> test result:
> > >>>          apicv=1,  there can be 15 thread running at the same time
> > without lost frame
> > >>>          apicv=0,  there can be 22 thread running at the same time
> > without lost frame
> > >>>
> > >>> so when I'm saying apicv reduce the performance, I got the
> > conclusion from the test result not from what xentrace shows.
> > >>>>
> > >>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> > >>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> > APIC-V",
> > >>>>> msr based apic is disabled when apic-v is on, I wonder can they
> > co-exist
> > >>>>> in some way? seems for windows guest msr-based apic has better
> > performance.
> > >>>>
> > >>>> The whole purpose is to avoid the costly MSR access exits. Why
> > >>>> would you want to reintroduce that overhead?
> > >>>>
> > >>>> Jan
> > >>>>
> > >>> I agree to avoid the MSR access vmexit by using apicv, I just do
> > not know what's the side effect.
> > >>> Because from the test result,  apicv replacing  msr-based access
> > brings performance reduction.
> > >>
> > >> It sounds like it brings latency disruption, not neccessarily
> > performance reduction.
> > >>
> > >> What happens if you run with the cpufreq turned to performance, or
> > as Jan suggested -
> > >> with disabling C-states?
> > >>
> > > I have tried disable C-states in BIOS, the test result is still the
> > same.
> > > There is no frontend-backend device in my test, dom0 is not
> > involved, so you mean cpufreq in DomU, right?
> >
> > I meant on Xen (cpufreq=performance). But if you disable C-states then
> > that should
> > not matter.
> >
> > > I have no idea how to change cpufreq of a windows os but I will
> > figure out and try.
> >
> > That should not be needed.
> >
> > But back to your problem. When you say you have 'frame delay' - is it
> > because
> > of the NAS storage not getting the I/Os fast enough?
> >
> The  NAS storage should not be the bottleneck, it's a ramdisk connected
> with 10G network card.
> Besides, I'am using the same test environment, the only difference is
> whether apicv=1 or not.
>
> > What are you using as network device? Is it an SR-IOV device or an
> > emulated
> > device (e1000, ne2k, rtl81..?)?
> The vm using Intel Corporation 82599 Ethernet Controller Virtual
> Function passthrough network and has the corresponding driver installed.
> > >>
> > >>> _______________________________________________
> > >>> Xen-devel mailing list
> > >>> Xen-devel@lists.xen.org
> > >>> http://lists.xen.org/xen-devel
> > >>
> > >> .
> > >>
> > >
> When  tracing the performance problem, I found whether apicv is on
> change the result.
> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> presents or not is the key factor.
>
>
>
>
sorry, forget cc the list