Re: support for sharing huge pages with grant table?

[Ian Campbell <Ian.Campbell@citrix.com>]

On Tue, 2014-11-18 at 14:50 -0500, Timothy Wood wrote:
> On Mon, Nov 17, 2014 at 5:34 AM, Ian Campbell
> <Ian.Campbell@citrix.com> wrote:
> 
>         On Sun, 2014-11-16 at 23:39 -0500, Tim Wood wrote:
>         > Hi,
>         >
>         > I am curious if Xen currently supports sharing hugepages
>         between
>         > domains (specifically ones originally allocated in Dom-0 and
>         shared
>         > with a guest r/w).  I've seen some references to huge pages
>         in the
>         > archives of this list, but not in relation to the grant
>         mechanism.
>         
>         I don't think the grant table has any specific superpage
>         support. It
>         might be an interesting extension to consider though (for the
>         sorts of
>         reasons you would like it).
>         
>         You could grant a superpage using multiple 4K grants to cover
>         whichever
>         subset of the superpage you need to expose to the other end.
>         Now granted
>         (no pun intended ;-)) that might suck up 512 grefs in the
>         worst case,
>         which is a bit mad...
> 
> 
> If the grants just need to be setup once at system startup and then
> are never changed, is this likely to pose any particular problem
> (i.e., does the size of the grant table only affect things when new
> grants are being setup, or is it checked regularly at runtime)?

I think the main concern would be the amount of space used by the grant
table, rather than the frequency of access to it.

At 512 refs per superpage, each ref being 64-bits (for a v1 grant table,
lets ignore v2 for now) then we are talking about a page worth of grant
table for each such superpage mapping. I think the maximum grant table
size is 32 pages by default, although this can be increased on the Xen
command line, I'm not sure how far though.

>  
>         
>         > Also, can someone confirm that "superpages" are another term
>         for "huge
>         > pages" in Xen?
>         
>         Yes. Or at least I think so.
>         
>         > This would be helpful for some work we are doing on high
>         speed
>         > networking to VMs---DPDK stores packets into huge pages and
>         we'd like
>         > to get those to VMs as quickly as possible.
>         
>         This seems like a reasonable usecase to me. Having added this
>         to the
>         grant table interface I suppose you would also need to
>         consider
>         extensions to the individual PV I/O protocols (netif.h) to
>         allow them to
>         signal when a grant was huge. You might have issues with e.g.
>         finding
>         enough bits to represent the larger sizes...
> 
> 
> I hope you guys will think it is useful enough to someday implement
> it.. I've got a handful of undergrad and grad students who work with
> me, but this might be beyond our capabilities at this point ;)
> 
> 
> In our prior system, we did this on KVM.  In that case we used QEMU to
> define a virtual PCI device that mapped memory from the host OS and
> let it be accessed by a VM.  Any thoughts on whether the same approach
> would work with Xen?  I'd originally thought just using the grant
> table to enable sharing would make this a lot easier in Xen, but maybe
> not since it doesn't support huge pages.

That approach might work for an HVM guest, I'm not sure. It probably
won't work for a PV guest since they don't have emulated PCI devices
(only actual ones passed through).

Ian.