Re: device-mmio emulation in Xen

[Tim Deegan <Tim.Deegan@citrix.com>]

At 16:32 +0000 on 05 Jan (1262709121), Abhinav Srivastava wrote:
> 1. How does Xen know which code path to execute? What I am noticing is
> that sometimes it goes to all the way till the end of the page fault
> handler code to call handle_mmio function, and sometimes it goes into
> the fast path to call this function, bypassing all the page fault
> handlers code? Why it does not go into the fast path all the time? OR
> Xen does something at the first time so that second time, the execution
> can directly go into the fast path? 

Yes.  In the slow path, it creates a deliberately invalid shadow PTE.
When the MMU signals an invalid-PTE fault, Xen knows tro take the fast
path.

> And, does this needs to be done for all physical pages given to MMIO?

Yes.  Some kind of pagefault needs to happen for emulated MMIO.  The
fast-path trick just makes the fault handler faster. 

Tim.

> 
> Thanks,
> Abhinav
> 
> 
> 
> --- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com> wrote:
> 
> > From: Tim Deegan <Tim.Deegan@citrix.com>
> > Subject: Re: [Xen-devel] device-mmio emulation in Xen
> > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
> > Date: Tuesday, 5 January, 2010, 9:30 PM
> > Hi, 
> > 
> > At 15:32 +0000 on 05 Jan (1262705557), Abhinav Srivastava
> > wrote:
> > > I noticed that during a network copy operation Xen
> > page faults a lot and control goes to sh_page_fault
> > function. When I printed some debugging info, it showed me
> > gmfn = -1. Then the execution goes through the regular path
> > of the page fault handler code, which means it creates an
> > entry using shadow_get_and_create_l1e, propagates it using
> > l1e_propagate_from_guest, and finally updates the entry
> > using shadow_set_l1e. It finally goes into the device-model
> > mmio condition. In this condition, it extracts a guest
> > physical address and calls "goto mmio", which in turn calls
> > handle_mmio function that emulates the instruction.
> > > 
> > > However with the gmfn = -1 condition, the execution
> > sometimes directly goto
> > > to handle_mmio function using the fast_fault_path with
> > going through the regular path. It seems like there are two
> > possible execution paths, and I did not understand which one
> > is chosen when?
> > > 
> > 
> > /******************************************************************************
> >  * We implement a "fast path" for two special cases: faults
> > that require
> >  * MMIO emulation, and faults where the guest PTE is not
> > present.  We
> >  * record these as shadow l1 entries that have reserved
> > bits set in
> >  * them, so we can spot them immediately in the fault
> > handler and handle
> >  * them without needing to hold the shadow lock or walk the
> > guest
> >  * pagetables.
> >  *
> >  * This is only feasible for PAE and 64bit Xen: 32-bit
> > non-PAE PTEs don't
> >  * have reserved bits that we can use for this.
> >  */
> > 
> > > I have some questions related to this behavior:
> > > 
> > > 1. Why are there so many faults duing network copy
> > operation?
> > 
> > The faults are a HVM guest doing MMIO operations to control
> > its emulated
> > network card.
> > 
> > > 2. What does gmfn = -1 signify? Is it reserved for
> > mmio addresses?
> > 
> > It's INVALID_MFN.  It means there is no memory mapped
> > at the address the
> > guest accessed.  Xen treats that as MMIO, though
> > there's a plan to have
> > MMIO regions explicitly registered instead.
> > 
> > > 3. How does Xen handle this gmfn = -1? It seems like
> > on the regular path
> > > it still creates, propagates, and updates entries for
> > gmfn = -1. How does Xen handles this at the shadow page
> > table level?
> > 
> > Yes, it creates a deliberately invalid shadow PTE; then in
> > the pagefault
> > handler it can use the pagefault error code and the
> > contents of the PTE 
> > to skip the bulk of the shadow fault handling and go
> > straight to handle_mmio.
> > 
> > > 4. What are these two code execution paths, and when
> > does Xen decide which
> > > path to choose?
> > > 
> > > 5. Finally, is there anyway these faults can be
> > reduced?
> > 
> > Yes, use PV drivers in the guest.  That eliminates the
> > whole
> > emulated-MMIO system, and all the pagefaults associated
> > with it. 
> > 
> > Cheers,
> > 
> > Tim.
> > 
> > > I would very appreciate any help in this regard.
> > > 
> > > Thanks,
> > > Abhinav
> > > 
> > > 
> > > 
> > >       The INTERNET now has a
> > personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 
> > -- 
> > Tim Deegan <Tim.Deegan@citrix.com>
> > Principal Software Engineer, Citrix Systems (R&D) Ltd.
> > [Company #02300071, SL9 0DZ, UK.]
> > 
> 
> 
>       The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]