Re: device-mmio emulation in Xen

[Abhinav Srivastava <abhinavs_iitkgp@yahoo.co.in>]

Hi Tim,

Thanks for your reply. On the same topic, I see a lot of MMIO operations and corresponding faults on network copy operations. However, I do not see faults at the same rate on disk operations or graphics operations. Is it my perceptions or there are different tricks/methods to emulate different devices? Why networking load is showing so many MMIO faults, but not other workloads?

Thanks,
Abhinav

--- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com> wrote:

> From: Tim Deegan <Tim.Deegan@citrix.com>
> Subject: Re: [Xen-devel] device-mmio emulation in Xen
> To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
> Date: Tuesday, 5 January, 2010, 10:08 PM
> At 16:32 +0000 on 05 Jan
> (1262709121), Abhinav Srivastava wrote:
> > 1. How does Xen know which code path to execute? What
> I am noticing is
> > that sometimes it goes to all the way till the end of
> the page fault
> > handler code to call handle_mmio function, and
> sometimes it goes into
> > the fast path to call this function, bypassing all the
> page fault
> > handlers code? Why it does not go into the fast path
> all the time? OR
> > Xen does something at the first time so that second
> time, the execution
> > can directly go into the fast path? 
> 
> Yes.  In the slow path, it creates a deliberately
> invalid shadow PTE.
> When the MMU signals an invalid-PTE fault, Xen knows tro
> take the fast
> path.
> 
> > And, does this needs to be done for all physical pages
> given to MMIO?
> 
> Yes.  Some kind of pagefault needs to happen for
> emulated MMIO.  The
> fast-path trick just makes the fault handler faster. 
> 
> Tim.
> 
> > 
> > Thanks,
> > Abhinav
> > 
> > 
> > 
> > --- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com>
> wrote:
> > 
> > > From: Tim Deegan <Tim.Deegan@citrix.com>
> > > Subject: Re: [Xen-devel] device-mmio emulation in
> Xen
> > > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > > Cc: "xen-devel@lists.xensource.com"
> <xen-devel@lists.xensource.com>
> > > Date: Tuesday, 5 January, 2010, 9:30 PM
> > > Hi, 
> > > 
> > > At 15:32 +0000 on 05 Jan (1262705557), Abhinav
> Srivastava
> > > wrote:
> > > > I noticed that during a network copy
> operation Xen
> > > page faults a lot and control goes to
> sh_page_fault
> > > function. When I printed some debugging info, it
> showed me
> > > gmfn = -1. Then the execution goes through the
> regular path
> > > of the page fault handler code, which means it
> creates an
> > > entry using shadow_get_and_create_l1e, propagates
> it using
> > > l1e_propagate_from_guest, and finally updates the
> entry
> > > using shadow_set_l1e. It finally goes into the
> device-model
> > > mmio condition. In this condition, it extracts a
> guest
> > > physical address and calls "goto mmio", which in
> turn calls
> > > handle_mmio function that emulates the
> instruction.
> > > > 
> > > > However with the gmfn = -1 condition, the
> execution
> > > sometimes directly goto
> > > > to handle_mmio function using the
> fast_fault_path with
> > > going through the regular path. It seems like
> there are two
> > > possible execution paths, and I did not
> understand which one
> > > is chosen when?
> > > > 
> > > 
> > >
> /******************************************************************************
> > >  * We implement a "fast path" for two
> special cases: faults
> > > that require
> > >  * MMIO emulation, and faults where the
> guest PTE is not
> > > present.  We
> > >  * record these as shadow l1 entries that
> have reserved
> > > bits set in
> > >  * them, so we can spot them immediately in
> the fault
> > > handler and handle
> > >  * them without needing to hold the shadow
> lock or walk the
> > > guest
> > >  * pagetables.
> > >  *
> > >  * This is only feasible for PAE and 64bit
> Xen: 32-bit
> > > non-PAE PTEs don't
> > >  * have reserved bits that we can use for
> this.
> > >  */
> > > 
> > > > I have some questions related to this
> behavior:
> > > > 
> > > > 1. Why are there so many faults duing
> network copy
> > > operation?
> > > 
> > > The faults are a HVM guest doing MMIO operations
> to control
> > > its emulated
> > > network card.
> > > 
> > > > 2. What does gmfn = -1 signify? Is it
> reserved for
> > > mmio addresses?
> > > 
> > > It's INVALID_MFN.  It means there is no memory
> mapped
> > > at the address the
> > > guest accessed.  Xen treats that as MMIO,
> though
> > > there's a plan to have
> > > MMIO regions explicitly registered instead.
> > > 
> > > > 3. How does Xen handle this gmfn = -1? It
> seems like
> > > on the regular path
> > > > it still creates, propagates, and updates
> entries for
> > > gmfn = -1. How does Xen handles this at the
> shadow page
> > > table level?
> > > 
> > > Yes, it creates a deliberately invalid shadow
> PTE; then in
> > > the pagefault
> > > handler it can use the pagefault error code and
> the
> > > contents of the PTE 
> > > to skip the bulk of the shadow fault handling and
> go
> > > straight to handle_mmio.
> > > 
> > > > 4. What are these two code execution paths,
> and when
> > > does Xen decide which
> > > > path to choose?
> > > > 
> > > > 5. Finally, is there anyway these faults can
> be
> > > reduced?
> > > 
> > > Yes, use PV drivers in the guest.  That
> eliminates the
> > > whole
> > > emulated-MMIO system, and all the pagefaults
> associated
> > > with it. 
> > > 
> > > Cheers,
> > > 
> > > Tim.
> > > 
> > > > I would very appreciate any help in this
> regard.
> > > > 
> > > > Thanks,
> > > > Abhinav
> > > > 
> > > > 
> > > > 
> > > >       The INTERNET now has a
> > > personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> > > > 
> > > >
> _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > 
> > > -- 
> > > Tim Deegan <Tim.Deegan@citrix.com>
> > > Principal Software Engineer, Citrix Systems
> (R&D) Ltd.
> > > [Company #02300071, SL9 0DZ, UK.]
> > > 
> > 
> > 
> >       The INTERNET now has a
> personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


      The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/