From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Liuqiming (John)" Subject: Re: apic-v reduce network performance in my test case Date: Fri, 6 Feb 2015 10:48:09 +0800 Message-ID: <54D42B69.7020800@huawei.com> References: <54CCAE71.5070006@huawei.com> <54CF5BA7020000780005BCF4@mail.emea.novell.com> <54CF82A1.3060505@huawei.com> <20150202145923.GA2186@l.oracle.com> <54D03148.50604@huawei.com> <20150203153228.GD9371@l.oracle.com> <54D2DC35.6070205@huawei.com> <54D2DDC5.3050206@huawei.com> <20150205192622.GA11646@x230.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20150205192622.GA11646@x230.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 2015/2/6 3:26, Konrad Rzeszutek Wilk wrote: > On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote: > > On 2015/2/5 10:57, Liuqiming (John) wrote: > >> sorry for late replay > >> > >> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote: > >>> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote: > >>>> On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote: > >>>>> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote: > >>>>>> Hi Jan, > >>>>>> Thanks for the reply. > >>>>>> > >>>>>> On 2015/2/2 18:12, Jan Beulich wrote: > >>>>>>>>>> On 31.01.15 at 11:29, wrote: > >>>>>>>> Recently I met an odd performance problem: when I turn > >>> on APIC > >>>>>>>> Virtualization feature (apicv=1), the network performance of a > >>> windows > >>>>>>>> guest become worse. > >>>>>>>> > >>>>>>>> My test case like this: host only have one windows 2008 > >>> R2 HVM > >>>>>>>> guest running,and this guest has a SR-IOV VF network passthrough > >>> to it. > >>>>>>>> Guest using this network access a NAS device. No fontend or > >>> backend of > >>>>>>>> network and storage, all data transfered through network. > >>>>>>>> > >>>>>>>> The xentrace data shows: the mainly difference between > >>> apicv and > >>>>>>>> non-apicv, is the way guest write apic registers, and > >>>>>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than > >>>>>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit > >>> is much > >>>>>>>> less than using APIC-v. > >>>>>>> > >>>>>>> There being heavier use of the pause VMEXIT doesn't by itself say > >>>>>>> anything, I'm afraid. It may suggest that you have a C-state exit > >>>>>>> latency problem - try lowering the maximum C-state allowed, or > >>>>>>> disabling use of C-states altogether. > >>>>>> Sorry, I forgot to mention my test scenario: > >>>>>> Its a video test suite,I am not sure what the logic inside the > >>> tools exactly (not opensource tool). > >>>>>> The basic flow is: > >>>>>> 1) test suite start several thread to read video file from > >>> disk (from NAS through network in my case) > >>>>>> 2) decode these video data as a frame one by one > >>>>>> 3) if any frame delay more than 40ms, then mark as lost > >>>>>> > >>>>>> test result: > >>>>>> apicv=1, there can be 15 thread running at the same time > >>> without lost frame > >>>>>> apicv=0, there can be 22 thread running at the same time > >>> without lost frame > >>>>>> > >>>>>> so when I'm saying apicv reduce the performance, I got the > >>> conclusion from the test result not from what xentrace shows. > >>>>>>> > >>>>>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7 > >>>>>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having > >>> APIC-V", > >>>>>>>> msr based apic is disabled when apic-v is on, I wonder can they > >>> co-exist > >>>>>>>> in some way? seems for windows guest msr-based apic has better > >>> performance. > >>>>>>> > >>>>>>> The whole purpose is to avoid the costly MSR access exits. Why > >>>>>>> would you want to reintroduce that overhead? > >>>>>>> > >>>>>>> Jan > >>>>>>> > >>>>>> I agree to avoid the MSR access vmexit by using apicv, I just do > >>> not know what's the side effect. > >>>>>> Because from the test result, apicv replacing msr-based access > >>> brings performance reduction. > >>>>> > >>>>> It sounds like it brings latency disruption, not neccessarily > >>> performance reduction. > >>>>> > >>>>> What happens if you run with the cpufreq turned to performance, or > >>> as Jan suggested - > >>>>> with disabling C-states? > >>>>> > >>>> I have tried disable C-states in BIOS, the test result is still the > >>> same. > >>>> There is no frontend-backend device in my test, dom0 is not > >>> involved, so you mean cpufreq in DomU, right? > >>> > >>> I meant on Xen (cpufreq=performance). But if you disable C-states then > >>> that should > >>> not matter. > >>> > >>>> I have no idea how to change cpufreq of a windows os but I will > >>> figure out and try. > >>> > >>> That should not be needed. > >>> > >>> But back to your problem. When you say you have 'frame delay' - is it > >>> because > >>> of the NAS storage not getting the I/Os fast enough? > >>> > >> The NAS storage should not be the bottleneck, it's a ramdisk connected > >> with 10G network card. > >> Besides, I'am using the same test environment, the only difference is > >> whether apicv=1 or not. > >> > >>> What are you using as network device? Is it an SR-IOV device or an > >>> emulated > >>> device (e1000, ne2k, rtl81..?)? > >> The vm using Intel Corporation 82599 Ethernet Controller Virtual > >> Function passthrough network and has the corresponding driver installed. > >>>>> > >>>>>> _______________________________________________ > >>>>>> Xen-devel mailing list > >>>>>> Xen-devel@lists.xen.org > >>>>>> http://lists.xen.org/xen-devel > >>>>> > >>>>> . > >>>>> > >>>> > >> When tracing the performance problem, I found whether apicv is on > >> change the result. > >> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC > >> presents or not is the key factor. > > Ah, willing to send out a patch for that? I have not get the conclusion: why CPUID4A_MSR_BASED_APIC affect performance. But I will continue to dig in, and happy to send out what I found. > >> > >> > >> > >> > > sorry, forget cc the list > > > > > for testing purpose, reverting this commit will do: http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=7f2e992b824ec62a2818e64390ac2ccfbd74e6b7 "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V" > . >