support for sharing huge pages with grant table?

support for sharing huge pages with grant table?

[Tim Wood]

[-- Attachment #1.1: Type: text/plain, Size: 549 bytes --]

Hi,
I am curious if Xen currently supports sharing hugepages between domains
(specifically ones originally allocated in Dom-0 and shared with a guest
r/w).  I've seen some references to huge pages in the archives of this
list, but not in relation to the grant mechanism.

Also, can someone confirm that "superpages" are another term for "huge
pages" in Xen?

This would be helpful for some work we are doing on high speed networking
to VMs---DPDK stores packets into huge pages and we'd like to get those to
VMs as quickly as possible.

thanks!
Tim

[-- Attachment #1.2: Type: text/html, Size: 683 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: support for sharing huge pages with grant table?

[Ian Campbell]

On Sun, 2014-11-16 at 23:39 -0500, Tim Wood wrote:
> Hi,
> 
> I am curious if Xen currently supports sharing hugepages between
> domains (specifically ones originally allocated in Dom-0 and shared
> with a guest r/w).  I've seen some references to huge pages in the
> archives of this list, but not in relation to the grant mechanism.

I don't think the grant table has any specific superpage support. It
might be an interesting extension to consider though (for the sorts of
reasons you would like it).

You could grant a superpage using multiple 4K grants to cover whichever
subset of the superpage you need to expose to the other end. Now granted
(no pun intended ;-)) that might suck up 512 grefs in the worst case,
which is a bit mad...

> Also, can someone confirm that "superpages" are another term for "huge
> pages" in Xen?

Yes. Or at least I think so.

> This would be helpful for some work we are doing on high speed
> networking to VMs---DPDK stores packets into huge pages and we'd like
> to get those to VMs as quickly as possible.

This seems like a reasonable usecase to me. Having added this to the
grant table interface I suppose you would also need to consider
extensions to the individual PV I/O protocols (netif.h) to allow them to
signal when a grant was huge. You might have issues with e.g. finding
enough bits to represent the larger sizes...

Ian.

Re: support for sharing huge pages with grant table?

[Timothy Wood]

[-- Attachment #1.1: Type: text/plain, Size: 2375 bytes --]

On Mon, Nov 17, 2014 at 5:34 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:

> On Sun, 2014-11-16 at 23:39 -0500, Tim Wood wrote:
> > Hi,
> >
> > I am curious if Xen currently supports sharing hugepages between
> > domains (specifically ones originally allocated in Dom-0 and shared
> > with a guest r/w).  I've seen some references to huge pages in the
> > archives of this list, but not in relation to the grant mechanism.
>
> I don't think the grant table has any specific superpage support. It
> might be an interesting extension to consider though (for the sorts of
> reasons you would like it).
>
> You could grant a superpage using multiple 4K grants to cover whichever
> subset of the superpage you need to expose to the other end. Now granted
> (no pun intended ;-)) that might suck up 512 grefs in the worst case,
> which is a bit mad...
>

If the grants just need to be setup once at system startup and then are
never changed, is this likely to pose any particular problem (i.e., does
the size of the grant table only affect things when new grants are being
setup, or is it checked regularly at runtime)?


>
> > Also, can someone confirm that "superpages" are another term for "huge
> > pages" in Xen?
>
> Yes. Or at least I think so.
>
> > This would be helpful for some work we are doing on high speed
> > networking to VMs---DPDK stores packets into huge pages and we'd like
> > to get those to VMs as quickly as possible.
>
> This seems like a reasonable usecase to me. Having added this to the
> grant table interface I suppose you would also need to consider
> extensions to the individual PV I/O protocols (netif.h) to allow them to
> signal when a grant was huge. You might have issues with e.g. finding
> enough bits to represent the larger sizes...
>

I hope you guys will think it is useful enough to someday implement it..
I've got a handful of undergrad and grad students who work with me, but
this might be beyond our capabilities at this point ;)

In our prior system, we did this on KVM.  In that case we used QEMU to
define a virtual PCI device that mapped memory from the host OS and let it
be accessed by a VM.  Any thoughts on whether the same approach would work
with Xen?  I'd originally thought just using the grant table to enable
sharing would make this a lot easier in Xen, but maybe not since it doesn't
support huge pages.

thanks!

[-- Attachment #1.2: Type: text/html, Size: 3124 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: support for sharing huge pages with grant table?

[Ian Campbell]

On Tue, 2014-11-18 at 14:50 -0500, Timothy Wood wrote:
> On Mon, Nov 17, 2014 at 5:34 AM, Ian Campbell
> <Ian.Campbell@citrix.com> wrote:
> 
>         On Sun, 2014-11-16 at 23:39 -0500, Tim Wood wrote:
>         > Hi,
>         >
>         > I am curious if Xen currently supports sharing hugepages
>         between
>         > domains (specifically ones originally allocated in Dom-0 and
>         shared
>         > with a guest r/w).  I've seen some references to huge pages
>         in the
>         > archives of this list, but not in relation to the grant
>         mechanism.
>         
>         I don't think the grant table has any specific superpage
>         support. It
>         might be an interesting extension to consider though (for the
>         sorts of
>         reasons you would like it).
>         
>         You could grant a superpage using multiple 4K grants to cover
>         whichever
>         subset of the superpage you need to expose to the other end.
>         Now granted
>         (no pun intended ;-)) that might suck up 512 grefs in the
>         worst case,
>         which is a bit mad...
> 
> 
> If the grants just need to be setup once at system startup and then
> are never changed, is this likely to pose any particular problem
> (i.e., does the size of the grant table only affect things when new
> grants are being setup, or is it checked regularly at runtime)?

I think the main concern would be the amount of space used by the grant
table, rather than the frequency of access to it.

At 512 refs per superpage, each ref being 64-bits (for a v1 grant table,
lets ignore v2 for now) then we are talking about a page worth of grant
table for each such superpage mapping. I think the maximum grant table
size is 32 pages by default, although this can be increased on the Xen
command line, I'm not sure how far though.

>  
>         
>         > Also, can someone confirm that "superpages" are another term
>         for "huge
>         > pages" in Xen?
>         
>         Yes. Or at least I think so.
>         
>         > This would be helpful for some work we are doing on high
>         speed
>         > networking to VMs---DPDK stores packets into huge pages and
>         we'd like
>         > to get those to VMs as quickly as possible.
>         
>         This seems like a reasonable usecase to me. Having added this
>         to the
>         grant table interface I suppose you would also need to
>         consider
>         extensions to the individual PV I/O protocols (netif.h) to
>         allow them to
>         signal when a grant was huge. You might have issues with e.g.
>         finding
>         enough bits to represent the larger sizes...
> 
> 
> I hope you guys will think it is useful enough to someday implement
> it.. I've got a handful of undergrad and grad students who work with
> me, but this might be beyond our capabilities at this point ;)
> 
> 
> In our prior system, we did this on KVM.  In that case we used QEMU to
> define a virtual PCI device that mapped memory from the host OS and
> let it be accessed by a VM.  Any thoughts on whether the same approach
> would work with Xen?  I'd originally thought just using the grant
> table to enable sharing would make this a lot easier in Xen, but maybe
> not since it doesn't support huge pages.

That approach might work for an HVM guest, I'm not sure. It probably
won't work for a PV guest since they don't have emulated PCI devices
(only actual ones passed through).

Ian.