From mboxrd@z Thu Jan 1 00:00:00 1970 From: Abhinav Srivastava Subject: Re: device-mmio emulation in Xen Date: Mon, 18 Jan 2010 20:45:29 +0530 (IST) Message-ID: <73770.94342.qm@web7908.mail.in.yahoo.com> References: <20100105163831.GC30054@whitby.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20100105163831.GC30054@whitby.uk.xensource.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Tim Deegan Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Hi Tim,=0A=0AThanks for your reply. On the same topic, I see a lot of MMIO = operations and corresponding faults on network copy operations. However, I = do not see faults at the same rate on disk operations or graphics operation= s. Is it my perceptions or there are different tricks/methods to emulate di= fferent devices? Why networking load is showing so many MMIO faults, but no= t other workloads?=0A=0AThanks,=0AAbhinav=0A=0A--- On Tue, 5/1/10, Tim Deeg= an wrote:=0A=0A> From: Tim Deegan =0A> Subject: Re: [Xen-devel] device-mmio emulation in Xen=0A> To: "A= bhinav Srivastava" =0A> Cc: "xen-devel@lists.x= ensource.com" =0A> Date: Tuesday, 5 January,= 2010, 10:08 PM=0A> At 16:32 +0000 on 05 Jan=0A> (1262709121), Abhinav Sriv= astava wrote:=0A> > 1. How does Xen know which code path to execute? What= =0A> I am noticing is=0A> > that sometimes it goes to all the way till the = end of=0A> the page fault=0A> > handler code to call handle_mmio function, = and=0A> sometimes it goes into=0A> > the fast path to call this function, b= ypassing all the=0A> page fault=0A> > handlers code? Why it does not go int= o the fast path=0A> all the time? OR=0A> > Xen does something at the first = time so that second=0A> time, the execution=0A> > can directly go into the = fast path? =0A> =0A> Yes.=A0 In the slow path, it creates a deliberately=0A= > invalid shadow PTE.=0A> When the MMU signals an invalid-PTE fault, Xen kn= ows tro=0A> take the fast=0A> path.=0A> =0A> > And, does this needs to be d= one for all physical pages=0A> given to MMIO?=0A> =0A> Yes.=A0 Some kind of= pagefault needs to happen for=0A> emulated MMIO.=A0 The=0A> fast-path tric= k just makes the fault handler faster. =0A> =0A> Tim.=0A> =0A> > =0A> > Tha= nks,=0A> > Abhinav=0A> > =0A> > =0A> > =0A> > --- On Tue, 5/1/10, Tim Deega= n =0A> wrote:=0A> > =0A> > > From: Tim Deegan =0A> > > Subject: Re: [Xen-devel] device-mmio emulation in= =0A> Xen=0A> > > To: "Abhinav Srivastava" =0A>= > > Cc: "xen-devel@lists.xensource.com"=0A> =0A> > > Date: Tuesday, 5 January, 2010, 9:30 PM=0A> > > Hi, =0A> > > =0A>= > > At 15:32 +0000 on 05 Jan (1262705557), Abhinav=0A> Srivastava=0A> > > = wrote:=0A> > > > I noticed that during a network copy=0A> operation Xen=0A>= > > page faults a lot and control goes to=0A> sh_page_fault=0A> > > functi= on. When I printed some debugging info, it=0A> showed me=0A> > > gmfn =3D -= 1. Then the execution goes through the=0A> regular path=0A> > > of the page= fault handler code, which means it=0A> creates an=0A> > > entry using shad= ow_get_and_create_l1e, propagates=0A> it using=0A> > > l1e_propagate_from_g= uest, and finally updates the=0A> entry=0A> > > using shadow_set_l1e. It fi= nally goes into the=0A> device-model=0A> > > mmio condition. In this condit= ion, it extracts a=0A> guest=0A> > > physical address and calls "goto mmio"= , which in=0A> turn calls=0A> > > handle_mmio function that emulates the=0A= > instruction.=0A> > > > =0A> > > > However with the gmfn =3D -1 condition,= the=0A> execution=0A> > > sometimes directly goto=0A> > > > to handle_mmio= function using the=0A> fast_fault_path with=0A> > > going through the regu= lar path. It seems like=0A> there are two=0A> > > possible execution paths,= and I did not=0A> understand which one=0A> > > is chosen when?=0A> > > > = =0A> > > =0A> > >=0A> /****************************************************= **************************=0A> > >=A0 * We implement a "fast path" for two= =0A> special cases: faults=0A> > > that require=0A> > >=A0 * MMIO emulation= , and faults where the=0A> guest PTE is not=0A> > > present.=A0 We=0A> > >= =A0 * record these as shadow l1 entries that=0A> have reserved=0A> > > bits= set in=0A> > >=A0 * them, so we can spot them immediately in=0A> the fault= =0A> > > handler and handle=0A> > >=A0 * them without needing to hold the s= hadow=0A> lock or walk the=0A> > > guest=0A> > >=A0 * pagetables.=0A> > >= =A0 *=0A> > >=A0 * This is only feasible for PAE and 64bit=0A> Xen: 32-bit= =0A> > > non-PAE PTEs don't=0A> > >=A0 * have reserved bits that we can use= for=0A> this.=0A> > >=A0 */=0A> > > =0A> > > > I have some questions relat= ed to this=0A> behavior:=0A> > > > =0A> > > > 1. Why are there so many faul= ts duing=0A> network copy=0A> > > operation?=0A> > > =0A> > > The faults ar= e a HVM guest doing MMIO operations=0A> to control=0A> > > its emulated=0A>= > > network card.=0A> > > =0A> > > > 2. What does gmfn =3D -1 signify? Is = it=0A> reserved for=0A> > > mmio addresses?=0A> > > =0A> > > It's INVALID_M= FN.=A0 It means there is no memory=0A> mapped=0A> > > at the address the=0A= > > > guest accessed.=A0 Xen treats that as MMIO,=0A> though=0A> > > there'= s a plan to have=0A> > > MMIO regions explicitly registered instead.=0A> > = > =0A> > > > 3. How does Xen handle this gmfn =3D -1? It=0A> seems like=0A>= > > on the regular path=0A> > > > it still creates, propagates, and update= s=0A> entries for=0A> > > gmfn =3D -1. How does Xen handles this at the=0A>= shadow page=0A> > > table level?=0A> > > =0A> > > Yes, it creates a delibe= rately invalid shadow=0A> PTE; then in=0A> > > the pagefault=0A> > > handle= r it can use the pagefault error code and=0A> the=0A> > > contents of the P= TE =0A> > > to skip the bulk of the shadow fault handling and=0A> go=0A> > = > straight to handle_mmio.=0A> > > =0A> > > > 4. What are these two code ex= ecution paths,=0A> and when=0A> > > does Xen decide which=0A> > > > path to= choose?=0A> > > > =0A> > > > 5. Finally, is there anyway these faults can= =0A> be=0A> > > reduced?=0A> > > =0A> > > Yes, use PV drivers in the guest.= =A0 That=0A> eliminates the=0A> > > whole=0A> > > emulated-MMIO system, and= all the pagefaults=0A> associated=0A> > > with it. =0A> > > =0A> > > Cheer= s,=0A> > > =0A> > > Tim.=0A> > > =0A> > > > I would very appreciate any hel= p in this=0A> regard.=0A> > > > =0A> > > > Thanks,=0A> > > > Abhinav=0A> > = > > =0A> > > > =0A> > > > =0A> > > >=A0 =A0 =A0=A0=A0The INTERNET now has a= =0A> > > personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/= =0A> > > > =0A> > > >=0A> _______________________________________________= =0A> > > > Xen-devel mailing list=0A> > > > Xen-devel@lists.xensource.com= =0A> > > > http://lists.xensource.com/xen-devel=0A> > > =0A> > > -- =0A> > = > Tim Deegan =0A> > > Principal Software Engineer, C= itrix Systems=0A> (R&D) Ltd.=0A> > > [Company #02300071, SL9 0DZ, UK.]=0A> = > > =0A> > =0A> > =0A> >=A0 =A0 =A0=A0=A0The INTERNET now has a=0A> persona= lity. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/=0A> =0A> -- =0A= > Tim Deegan =0A> Principal Software Engineer, Citri= x Systems (R&D) Ltd.=0A> [Company #02300071, SL9 0DZ, UK.]=0A> =0A> _______= ________________________________________=0A> Xen-devel mailing list=0A> Xen= -devel@lists.xensource.com=0A> http://lists.xensource.com/xen-devel=0A> =0A= =0A=0A The INTERNET now has a personality. YOURS! See your Yahoo! Home= page. http://in.yahoo.com/