device-mmio emulation in Xen

device-mmio emulation in Xen

[Abhinav Srivastava]

Hi there,

I am trying to understand how does Xen handle device MMIO. More specifically, I am looking at the network behavior, however my understanding is that other devices such as disk will use the same logic.

I noticed that during a network copy operation Xen page faults a lot and control goes to sh_page_fault function. When I printed some debugging info, it showed me gmfn = -1. Then the execution goes through the regular path of the page fault handler code, which means it creates an entry using shadow_get_and_create_l1e, propagates it using l1e_propagate_from_guest, and finally updates the entry using shadow_set_l1e. It finally goes into the device-model mmio condition. In this condition, it extracts a guest physical address and calls "goto mmio", which in turn calls handle_mmio function that emulates the instruction.

However with the gmfn = -1 condition, the execution sometimes directly goto
to handle_mmio function using the fast_fault_path with going through the regular path. It seems like there are two possible execution paths, and I did not understand which one is chosen when?

I have some questions related to this behavior:

1. Why are there so many faults duing network copy operation?

2. What does gmfn = -1 signify? Is it reserved for mmio addresses?

3. How does Xen handle this gmfn = -1? It seems like on the regular path
it still creates, propagates, and updates entries for gmfn = -1. How does Xen handles this at the shadow page table level?

4. What are these two code execution paths, and when does Xen decide which
path to choose?

5. Finally, is there anyway these faults can be reduced?

I would very appreciate any help in this regard.

Thanks,
Abhinav



      The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

Re: device-mmio emulation in Xen

[Tim Deegan]

Hi, 

At 15:32 +0000 on 05 Jan (1262705557), Abhinav Srivastava wrote:
> I noticed that during a network copy operation Xen page faults a lot and control goes to sh_page_fault function. When I printed some debugging info, it showed me gmfn = -1. Then the execution goes through the regular path of the page fault handler code, which means it creates an entry using shadow_get_and_create_l1e, propagates it using l1e_propagate_from_guest, and finally updates the entry using shadow_set_l1e. It finally goes into the device-model mmio condition. In this condition, it extracts a guest physical address and calls "goto mmio", which in turn calls handle_mmio function that emulates the instruction.
> 
> However with the gmfn = -1 condition, the execution sometimes directly goto
> to handle_mmio function using the fast_fault_path with going through the regular path. It seems like there are two possible execution paths, and I did not understand which one is chosen when?
> 

/******************************************************************************
 * We implement a "fast path" for two special cases: faults that require
 * MMIO emulation, and faults where the guest PTE is not present.  We
 * record these as shadow l1 entries that have reserved bits set in
 * them, so we can spot them immediately in the fault handler and handle
 * them without needing to hold the shadow lock or walk the guest
 * pagetables.
 *
 * This is only feasible for PAE and 64bit Xen: 32-bit non-PAE PTEs don't
 * have reserved bits that we can use for this.
 */

> I have some questions related to this behavior:
> 
> 1. Why are there so many faults duing network copy operation?

The faults are a HVM guest doing MMIO operations to control its emulated
network card.

> 2. What does gmfn = -1 signify? Is it reserved for mmio addresses?

It's INVALID_MFN.  It means there is no memory mapped at the address the
guest accessed.  Xen treats that as MMIO, though there's a plan to have
MMIO regions explicitly registered instead.

> 3. How does Xen handle this gmfn = -1? It seems like on the regular path
> it still creates, propagates, and updates entries for gmfn = -1. How does Xen handles this at the shadow page table level?

Yes, it creates a deliberately invalid shadow PTE; then in the pagefault
handler it can use the pagefault error code and the contents of the PTE 
to skip the bulk of the shadow fault handling and go straight to handle_mmio.

> 4. What are these two code execution paths, and when does Xen decide which
> path to choose?
> 
> 5. Finally, is there anyway these faults can be reduced?

Yes, use PV drivers in the guest.  That eliminates the whole
emulated-MMIO system, and all the pagefaults associated with it. 

Cheers,

Tim.

> I would very appreciate any help in this regard.
> 
> Thanks,
> Abhinav
> 
> 
> 
>       The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: device-mmio emulation in Xen

[Abhinav Srivastava]

Hi Tim,

Thanks for the reply. I understood what you had described. However, I have one question:

1. How does Xen know which code path to execute? What I am noticing is that sometimes it goes to all the way till the end of the page fault handler code to call handle_mmio function, and sometimes it goes into the fast path to call this function, bypassing all the page fault handlers code? Why it does not go into the fast path all the time? OR Xen does something at the first time so that second time, the execution can directly go into the fast path? And, does this needs to be done for all physical pages given to MMIO?

Thanks,
Abhinav



--- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com> wrote:

> From: Tim Deegan <Tim.Deegan@citrix.com>
> Subject: Re: [Xen-devel] device-mmio emulation in Xen
> To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
> Date: Tuesday, 5 January, 2010, 9:30 PM
> Hi, 
> 
> At 15:32 +0000 on 05 Jan (1262705557), Abhinav Srivastava
> wrote:
> > I noticed that during a network copy operation Xen
> page faults a lot and control goes to sh_page_fault
> function. When I printed some debugging info, it showed me
> gmfn = -1. Then the execution goes through the regular path
> of the page fault handler code, which means it creates an
> entry using shadow_get_and_create_l1e, propagates it using
> l1e_propagate_from_guest, and finally updates the entry
> using shadow_set_l1e. It finally goes into the device-model
> mmio condition. In this condition, it extracts a guest
> physical address and calls "goto mmio", which in turn calls
> handle_mmio function that emulates the instruction.
> > 
> > However with the gmfn = -1 condition, the execution
> sometimes directly goto
> > to handle_mmio function using the fast_fault_path with
> going through the regular path. It seems like there are two
> possible execution paths, and I did not understand which one
> is chosen when?
> > 
> 
> /******************************************************************************
>  * We implement a "fast path" for two special cases: faults
> that require
>  * MMIO emulation, and faults where the guest PTE is not
> present.  We
>  * record these as shadow l1 entries that have reserved
> bits set in
>  * them, so we can spot them immediately in the fault
> handler and handle
>  * them without needing to hold the shadow lock or walk the
> guest
>  * pagetables.
>  *
>  * This is only feasible for PAE and 64bit Xen: 32-bit
> non-PAE PTEs don't
>  * have reserved bits that we can use for this.
>  */
> 
> > I have some questions related to this behavior:
> > 
> > 1. Why are there so many faults duing network copy
> operation?
> 
> The faults are a HVM guest doing MMIO operations to control
> its emulated
> network card.
> 
> > 2. What does gmfn = -1 signify? Is it reserved for
> mmio addresses?
> 
> It's INVALID_MFN.  It means there is no memory mapped
> at the address the
> guest accessed.  Xen treats that as MMIO, though
> there's a plan to have
> MMIO regions explicitly registered instead.
> 
> > 3. How does Xen handle this gmfn = -1? It seems like
> on the regular path
> > it still creates, propagates, and updates entries for
> gmfn = -1. How does Xen handles this at the shadow page
> table level?
> 
> Yes, it creates a deliberately invalid shadow PTE; then in
> the pagefault
> handler it can use the pagefault error code and the
> contents of the PTE 
> to skip the bulk of the shadow fault handling and go
> straight to handle_mmio.
> 
> > 4. What are these two code execution paths, and when
> does Xen decide which
> > path to choose?
> > 
> > 5. Finally, is there anyway these faults can be
> reduced?
> 
> Yes, use PV drivers in the guest.  That eliminates the
> whole
> emulated-MMIO system, and all the pagefaults associated
> with it. 
> 
> Cheers,
> 
> Tim.
> 
> > I would very appreciate any help in this regard.
> > 
> > Thanks,
> > Abhinav
> > 
> > 
> > 
> >       The INTERNET now has a
> personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
> 


      The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

Re: device-mmio emulation in Xen

[Tim Deegan]

At 16:32 +0000 on 05 Jan (1262709121), Abhinav Srivastava wrote:
> 1. How does Xen know which code path to execute? What I am noticing is
> that sometimes it goes to all the way till the end of the page fault
> handler code to call handle_mmio function, and sometimes it goes into
> the fast path to call this function, bypassing all the page fault
> handlers code? Why it does not go into the fast path all the time? OR
> Xen does something at the first time so that second time, the execution
> can directly go into the fast path? 

Yes.  In the slow path, it creates a deliberately invalid shadow PTE.
When the MMU signals an invalid-PTE fault, Xen knows tro take the fast
path.

> And, does this needs to be done for all physical pages given to MMIO?

Yes.  Some kind of pagefault needs to happen for emulated MMIO.  The
fast-path trick just makes the fault handler faster. 

Tim.

> 
> Thanks,
> Abhinav
> 
> 
> 
> --- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com> wrote:
> 
> > From: Tim Deegan <Tim.Deegan@citrix.com>
> > Subject: Re: [Xen-devel] device-mmio emulation in Xen
> > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
> > Date: Tuesday, 5 January, 2010, 9:30 PM
> > Hi, 
> > 
> > At 15:32 +0000 on 05 Jan (1262705557), Abhinav Srivastava
> > wrote:
> > > I noticed that during a network copy operation Xen
> > page faults a lot and control goes to sh_page_fault
> > function. When I printed some debugging info, it showed me
> > gmfn = -1. Then the execution goes through the regular path
> > of the page fault handler code, which means it creates an
> > entry using shadow_get_and_create_l1e, propagates it using
> > l1e_propagate_from_guest, and finally updates the entry
> > using shadow_set_l1e. It finally goes into the device-model
> > mmio condition. In this condition, it extracts a guest
> > physical address and calls "goto mmio", which in turn calls
> > handle_mmio function that emulates the instruction.
> > > 
> > > However with the gmfn = -1 condition, the execution
> > sometimes directly goto
> > > to handle_mmio function using the fast_fault_path with
> > going through the regular path. It seems like there are two
> > possible execution paths, and I did not understand which one
> > is chosen when?
> > > 
> > 
> > /******************************************************************************
> >  * We implement a "fast path" for two special cases: faults
> > that require
> >  * MMIO emulation, and faults where the guest PTE is not
> > present.  We
> >  * record these as shadow l1 entries that have reserved
> > bits set in
> >  * them, so we can spot them immediately in the fault
> > handler and handle
> >  * them without needing to hold the shadow lock or walk the
> > guest
> >  * pagetables.
> >  *
> >  * This is only feasible for PAE and 64bit Xen: 32-bit
> > non-PAE PTEs don't
> >  * have reserved bits that we can use for this.
> >  */
> > 
> > > I have some questions related to this behavior:
> > > 
> > > 1. Why are there so many faults duing network copy
> > operation?
> > 
> > The faults are a HVM guest doing MMIO operations to control
> > its emulated
> > network card.
> > 
> > > 2. What does gmfn = -1 signify? Is it reserved for
> > mmio addresses?
> > 
> > It's INVALID_MFN.  It means there is no memory mapped
> > at the address the
> > guest accessed.  Xen treats that as MMIO, though
> > there's a plan to have
> > MMIO regions explicitly registered instead.
> > 
> > > 3. How does Xen handle this gmfn = -1? It seems like
> > on the regular path
> > > it still creates, propagates, and updates entries for
> > gmfn = -1. How does Xen handles this at the shadow page
> > table level?
> > 
> > Yes, it creates a deliberately invalid shadow PTE; then in
> > the pagefault
> > handler it can use the pagefault error code and the
> > contents of the PTE 
> > to skip the bulk of the shadow fault handling and go
> > straight to handle_mmio.
> > 
> > > 4. What are these two code execution paths, and when
> > does Xen decide which
> > > path to choose?
> > > 
> > > 5. Finally, is there anyway these faults can be
> > reduced?
> > 
> > Yes, use PV drivers in the guest.  That eliminates the
> > whole
> > emulated-MMIO system, and all the pagefaults associated
> > with it. 
> > 
> > Cheers,
> > 
> > Tim.
> > 
> > > I would very appreciate any help in this regard.
> > > 
> > > Thanks,
> > > Abhinav
> > > 
> > > 
> > > 
> > >       The INTERNET now has a
> > personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 
> > -- 
> > Tim Deegan <Tim.Deegan@citrix.com>
> > Principal Software Engineer, Citrix Systems (R&D) Ltd.
> > [Company #02300071, SL9 0DZ, UK.]
> > 
> 
> 
>       The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: device-mmio emulation in Xen

[Abhinav Srivastava]

Hi Tim,

Thanks for your reply. On the same topic, I see a lot of MMIO operations and corresponding faults on network copy operations. However, I do not see faults at the same rate on disk operations or graphics operations. Is it my perceptions or there are different tricks/methods to emulate different devices? Why networking load is showing so many MMIO faults, but not other workloads?

Thanks,
Abhinav

--- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com> wrote:

> From: Tim Deegan <Tim.Deegan@citrix.com>
> Subject: Re: [Xen-devel] device-mmio emulation in Xen
> To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> Cc: "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>
> Date: Tuesday, 5 January, 2010, 10:08 PM
> At 16:32 +0000 on 05 Jan
> (1262709121), Abhinav Srivastava wrote:
> > 1. How does Xen know which code path to execute? What
> I am noticing is
> > that sometimes it goes to all the way till the end of
> the page fault
> > handler code to call handle_mmio function, and
> sometimes it goes into
> > the fast path to call this function, bypassing all the
> page fault
> > handlers code? Why it does not go into the fast path
> all the time? OR
> > Xen does something at the first time so that second
> time, the execution
> > can directly go into the fast path? 
> 
> Yes.  In the slow path, it creates a deliberately
> invalid shadow PTE.
> When the MMU signals an invalid-PTE fault, Xen knows tro
> take the fast
> path.
> 
> > And, does this needs to be done for all physical pages
> given to MMIO?
> 
> Yes.  Some kind of pagefault needs to happen for
> emulated MMIO.  The
> fast-path trick just makes the fault handler faster. 
> 
> Tim.
> 
> > 
> > Thanks,
> > Abhinav
> > 
> > 
> > 
> > --- On Tue, 5/1/10, Tim Deegan <Tim.Deegan@citrix.com>
> wrote:
> > 
> > > From: Tim Deegan <Tim.Deegan@citrix.com>
> > > Subject: Re: [Xen-devel] device-mmio emulation in
> Xen
> > > To: "Abhinav Srivastava" <abhinavs_iitkgp@yahoo.co.in>
> > > Cc: "xen-devel@lists.xensource.com"
> <xen-devel@lists.xensource.com>
> > > Date: Tuesday, 5 January, 2010, 9:30 PM
> > > Hi, 
> > > 
> > > At 15:32 +0000 on 05 Jan (1262705557), Abhinav
> Srivastava
> > > wrote:
> > > > I noticed that during a network copy
> operation Xen
> > > page faults a lot and control goes to
> sh_page_fault
> > > function. When I printed some debugging info, it
> showed me
> > > gmfn = -1. Then the execution goes through the
> regular path
> > > of the page fault handler code, which means it
> creates an
> > > entry using shadow_get_and_create_l1e, propagates
> it using
> > > l1e_propagate_from_guest, and finally updates the
> entry
> > > using shadow_set_l1e. It finally goes into the
> device-model
> > > mmio condition. In this condition, it extracts a
> guest
> > > physical address and calls "goto mmio", which in
> turn calls
> > > handle_mmio function that emulates the
> instruction.
> > > > 
> > > > However with the gmfn = -1 condition, the
> execution
> > > sometimes directly goto
> > > > to handle_mmio function using the
> fast_fault_path with
> > > going through the regular path. It seems like
> there are two
> > > possible execution paths, and I did not
> understand which one
> > > is chosen when?
> > > > 
> > > 
> > >
> /******************************************************************************
> > >  * We implement a "fast path" for two
> special cases: faults
> > > that require
> > >  * MMIO emulation, and faults where the
> guest PTE is not
> > > present.  We
> > >  * record these as shadow l1 entries that
> have reserved
> > > bits set in
> > >  * them, so we can spot them immediately in
> the fault
> > > handler and handle
> > >  * them without needing to hold the shadow
> lock or walk the
> > > guest
> > >  * pagetables.
> > >  *
> > >  * This is only feasible for PAE and 64bit
> Xen: 32-bit
> > > non-PAE PTEs don't
> > >  * have reserved bits that we can use for
> this.
> > >  */
> > > 
> > > > I have some questions related to this
> behavior:
> > > > 
> > > > 1. Why are there so many faults duing
> network copy
> > > operation?
> > > 
> > > The faults are a HVM guest doing MMIO operations
> to control
> > > its emulated
> > > network card.
> > > 
> > > > 2. What does gmfn = -1 signify? Is it
> reserved for
> > > mmio addresses?
> > > 
> > > It's INVALID_MFN.  It means there is no memory
> mapped
> > > at the address the
> > > guest accessed.  Xen treats that as MMIO,
> though
> > > there's a plan to have
> > > MMIO regions explicitly registered instead.
> > > 
> > > > 3. How does Xen handle this gmfn = -1? It
> seems like
> > > on the regular path
> > > > it still creates, propagates, and updates
> entries for
> > > gmfn = -1. How does Xen handles this at the
> shadow page
> > > table level?
> > > 
> > > Yes, it creates a deliberately invalid shadow
> PTE; then in
> > > the pagefault
> > > handler it can use the pagefault error code and
> the
> > > contents of the PTE 
> > > to skip the bulk of the shadow fault handling and
> go
> > > straight to handle_mmio.
> > > 
> > > > 4. What are these two code execution paths,
> and when
> > > does Xen decide which
> > > > path to choose?
> > > > 
> > > > 5. Finally, is there anyway these faults can
> be
> > > reduced?
> > > 
> > > Yes, use PV drivers in the guest.  That
> eliminates the
> > > whole
> > > emulated-MMIO system, and all the pagefaults
> associated
> > > with it. 
> > > 
> > > Cheers,
> > > 
> > > Tim.
> > > 
> > > > I would very appreciate any help in this
> regard.
> > > > 
> > > > Thanks,
> > > > Abhinav
> > > > 
> > > > 
> > > > 
> > > >       The INTERNET now has a
> > > personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> > > > 
> > > >
> _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > 
> > > -- 
> > > Tim Deegan <Tim.Deegan@citrix.com>
> > > Principal Software Engineer, Citrix Systems
> (R&D) Ltd.
> > > [Company #02300071, SL9 0DZ, UK.]
> > > 
> > 
> > 
> >       The INTERNET now has a
> personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/
> 
> -- 
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


      The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/