From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Liuqiming (John)" Subject: Re: apic-v reduce network performance in my test case Date: Thu, 5 Feb 2015 11:04:37 +0800 Message-ID: <54D2DDC5.3050206@huawei.com> References: <54CCAE71.5070006@huawei.com> <54CF5BA7020000780005BCF4@mail.emea.novell.com> <54CF82A1.3060505@huawei.com> <20150202145923.GA2186@l.oracle.com> <54D03148.50604@huawei.com> <20150203153228.GD9371@l.oracle.com> <54D2DC35.6070205@huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <54D2DC35.6070205@huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk , xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 2015/2/5 10:57, Liuqiming (John) wrote: > sorry for late replay > > On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote: > > On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote: > > > On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote: > > >> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote: > > >>> Hi Jan, > > >>> Thanks for the reply. > > >>> > > >>> On 2015/2/2 18:12, Jan Beulich wrote: > > >>>>>>> On 31.01.15 at 11:29, wrote: > > >>>>> Recently I met an odd performance problem: when I turn > > on APIC > > >>>>> Virtualization feature (apicv=1), the network performance of a > > windows > > >>>>> guest become worse. > > >>>>> > > >>>>> My test case like this: host only have one windows 2008 > > R2 HVM > > >>>>> guest running,and this guest has a SR-IOV VF network passthrough > > to it. > > >>>>> Guest using this network access a NAS device. No fontend or > > backend of > > >>>>> network and storage, all data transfered through network. > > >>>>> > > >>>>> The xentrace data shows: the mainly difference between > > apicv and > > >>>>> non-apicv, is the way guest write apic registers, and > > >>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than > > >>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit > > is much > > >>>>> less than using APIC-v. > > >>>> > > >>>> There being heavier use of the pause VMEXIT doesn't by itself say > > >>>> anything, I'm afraid. It may suggest that you have a C-state exit > > >>>> latency problem - try lowering the maximum C-state allowed, or > > >>>> disabling use of C-states altogether. > > >>> Sorry, I forgot to mention my test scenario: > > >>> Its a video test suite,I am not sure what the logic inside the > > tools exactly (not opensource tool). > > >>> The basic flow is: > > >>> 1) test suite start several thread to read video file from > > disk (from NAS through network in my case) > > >>> 2) decode these video data as a frame one by one > > >>> 3) if any frame delay more than 40ms, then mark as lost > > >>> > > >>> test result: > > >>> apicv=1, there can be 15 thread running at the same time > > without lost frame > > >>> apicv=0, there can be 22 thread running at the same time > > without lost frame > > >>> > > >>> so when I'm saying apicv reduce the performance, I got the > > conclusion from the test result not from what xentrace shows. > > >>>> > > >>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7 > > >>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having > > APIC-V", > > >>>>> msr based apic is disabled when apic-v is on, I wonder can they > > co-exist > > >>>>> in some way? seems for windows guest msr-based apic has better > > performance. > > >>>> > > >>>> The whole purpose is to avoid the costly MSR access exits. Why > > >>>> would you want to reintroduce that overhead? > > >>>> > > >>>> Jan > > >>>> > > >>> I agree to avoid the MSR access vmexit by using apicv, I just do > > not know what's the side effect. > > >>> Because from the test result, apicv replacing msr-based access > > brings performance reduction. > > >> > > >> It sounds like it brings latency disruption, not neccessarily > > performance reduction. > > >> > > >> What happens if you run with the cpufreq turned to performance, or > > as Jan suggested - > > >> with disabling C-states? > > >> > > > I have tried disable C-states in BIOS, the test result is still the > > same. > > > There is no frontend-backend device in my test, dom0 is not > > involved, so you mean cpufreq in DomU, right? > > > > I meant on Xen (cpufreq=performance). But if you disable C-states then > > that should > > not matter. > > > > > I have no idea how to change cpufreq of a windows os but I will > > figure out and try. > > > > That should not be needed. > > > > But back to your problem. When you say you have 'frame delay' - is it > > because > > of the NAS storage not getting the I/Os fast enough? > > > The NAS storage should not be the bottleneck, it's a ramdisk connected > with 10G network card. > Besides, I'am using the same test environment, the only difference is > whether apicv=1 or not. > > > What are you using as network device? Is it an SR-IOV device or an > > emulated > > device (e1000, ne2k, rtl81..?)? > The vm using Intel Corporation 82599 Ethernet Controller Virtual > Function passthrough network and has the corresponding driver installed. > > >> > > >>> _______________________________________________ > > >>> Xen-devel mailing list > > >>> Xen-devel@lists.xen.org > > >>> http://lists.xen.org/xen-devel > > >> > > >> . > > >> > > > > When tracing the performance problem, I found whether apicv is on > change the result. > And after investigating the code, turns out CPUID4A_MSR_BASED_APIC > presents or not is the key factor. > > > > sorry, forget cc the list