Inter-domain shared memory facility in Xen?

Inter-domain shared memory facility in Xen?

[David Stone]

Hopefully this question belongs in xen-devel and not xen-users:

Is there any kind of inter-domain shared memory facility in Xen?  E.g.
an API callable from DomU user-mode that would cause the system sets
up two DomU's page tables so they a range of virtual addresses in both
resolve to the same set of machine pages?

Thanks,
Dave

Re: Inter-domain shared memory facility in Xen?

[Akio Takebe]

Hi,

>Hopefully this question belongs in xen-devel and not xen-users:
>
>Is there any kind of inter-domain shared memory facility in Xen?  E.g.
>an API callable from DomU user-mode that would cause the system sets
>up two DomU's page tables so they a range of virtual addresses in both
>resolve to the same set of machine pages?
You may be able to use grant table.
Grant table can make shared memroy.

If you want to share memory between dom0 and domU,
you can use the following fanction.

1. HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref) and
   gnttab_grant_foreign_access()
       It is kernel API used in blk split driver.

2. xc_map_foreign_range()
       It is dom0 user API used in xc_core.c.
       It can share memory between dom0 user vaddr and domU guest maddr.

If you want to share memory between domU user vaddr and the other domU's,
you need to make the API.

Best Regards,

Akio Takebe

Re: Inter-domain shared memory facility in Xen?

[Haifeng He]

Hi,

I had a similar question on another email thread and I have a
following question.
If I understand correctly, I need to change the kernels of domUs to make the
sharing happen (explicitly call those APIs). Is it possible to do the
sharing without
changing domUs? In another word, is it possible to let Xen, the hypervisor,
to perform the sharing transparently without modifying domUs.
For example, Xen can share all the identical pages among domUs and
apply Copy-On-Write on them. I guess, it requires changes inside the Xen's
memory management. Any suggestion on how it may be possible?

Thanks

Haifeng

On 10/30/07, Akio Takebe <takebe_akio@jp.fujitsu.com> wrote:
> Hi,
>
> >Hopefully this question belongs in xen-devel and not xen-users:
> >
> >Is there any kind of inter-domain shared memory facility in Xen?  E.g.
> >an API callable from DomU user-mode that would cause the system sets
> >up two DomU's page tables so they a range of virtual addresses in both
> >resolve to the same set of machine pages?
> You may be able to use grant table.
> Grant table can make shared memroy.
>
> If you want to share memory between dom0 and domU,
> you can use the following fanction.
>
> 1. HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref) and
>    gnttab_grant_foreign_access()
>        It is kernel API used in blk split driver.
>
> 2. xc_map_foreign_range()
>        It is dom0 user API used in xc_core.c.
>        It can share memory between dom0 user vaddr and domU guest maddr.
>
> If you want to share memory between domU user vaddr and the other domU's,
> you need to make the API.
>
> Best Regards,
>
> Akio Takebe
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Inter-domain shared memory facility in Xen?

[Akio Takebe]

Hi,

http://sourceforge.net/projects/xvmsocket/
http://xen.xensource.com/files/xensummit_4/SuzanneMcIntosh_XenSummit_2007.pdf
See XenSocket. I think it is your help.

http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00122.html
Xway is also the similar facility.

Best Regards,

Akio Takebe

>Hi,
>
>I had a similar question on another email thread and I have a
>following question.
>If I understand correctly, I need to change the kernels of domUs to make the
>sharing happen (explicitly call those APIs). Is it possible to do the
>sharing without
>changing domUs? In another word, is it possible to let Xen, the hypervisor,
>to perform the sharing transparently without modifying domUs.
>For example, Xen can share all the identical pages among domUs and
>apply Copy-On-Write on them. I guess, it requires changes inside the Xen's
>memory management. Any suggestion on how it may be possible?
>
>Thanks
>
>Haifeng
>
>On 10/30/07, Akio Takebe <takebe_akio@jp.fujitsu.com> wrote:
>> Hi,
>>
>> >Hopefully this question belongs in xen-devel and not xen-users:
>> >
>> >Is there any kind of inter-domain shared memory facility in Xen?  E.g.
>> >an API callable from DomU user-mode that would cause the system sets
>> >up two DomU's page tables so they a range of virtual addresses in both
>> >resolve to the same set of machine pages?
>> You may be able to use grant table.
>> Grant table can make shared memroy.
>>
>> If you want to share memory between dom0 and domU,
>> you can use the following fanction.
>>
>> 1. HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref) and
>>    gnttab_grant_foreign_access()
>>        It is kernel API used in blk split driver.
>>
>> 2. xc_map_foreign_range()
>>        It is dom0 user API used in xc_core.c.
>>        It can share memory between dom0 user vaddr and domU guest maddr.
>>
>> If you want to share memory between domU user vaddr and the other domU's,
>> you need to make the API.
>>
>> Best Regards,
>>
>> Akio Takebe
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>

Fwd: Inter-domain shared memory facility in Xen?

[David Stone]

(Forgot to include the list)


---------- Forwarded message ----------
From: David Stone <unclestoner@gmail.com>
Date: Nov 1, 2007 9:39 AM
Subject: Re: [Xen-devel] Inter-domain shared memory facility in Xen?
To: Akio Takebe <takebe_akio@jp.fujitsu.com>


Thanks for your help on this.  So based on your responses I just want
to confirm my understanding is correct...is it true that:

1. It is currently possible to establish shared memory between Dom0
and DomU, but
2. It is currently not possible to establish shared memory between two
DomUs in an released Xen versions, nor the current xen-unstable branch
nor any other branch on http://xenbits.xensource.org.  There are
projects that address this that you mentioned though.

Thanks again,
Dave


> http://sourceforge.net/projects/xvmsocket/
> http://xen.xensource.com/files/xensummit_4/SuzanneMcIntosh_XenSummit_2007.pdf
> See XenSocket. I think it is your help.
>
> http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00122.html
> Xway is also the similar facility.

Re: Fwd: Inter-domain shared memory facility in Xen?

[Daniel Stodden]

On Thu, 2007-11-01 at 09:40 -0400, David Stone wrote:

> Thanks for your help on this.  So based on your responses I just want
> to confirm my understanding is correct...is it true that:
> 
> 1. It is currently possible to establish shared memory between Dom0
> and DomU, but
> 2. It is currently not possible to establish shared memory between two
> DomUs in an released Xen versions, nor the current xen-unstable branch
> nor any other branch on http://xenbits.xensource.org.  There are
> projects that address this that you mentioned though.

i did not follow this discussion until now, but are you aware of the
grant table facility part of the PV interface?

xensocket and xway are projects targeting sockets-based APIs. grant
table ops are, like all hypercalls, a kernel-level interface, but the
xen-specific modules in dom0 as well as the control libraries already
contain some code to let userland use it.

regarding item 2.: no, shared memory can be established between
arbitrary domains. otherwise, the sockets interfaces could not work,
right? grants work about as follows: dom1 'grants' a page to dom2, which
creates a 'grant reference'. that grant reference can be used by dom2 to
(map|transfer) that page into its respective own address space(s). 

this is what e.g. the pv driver codebase is using for bulk data
transfers. similarly, a userland interface may be built to e.g.
implement something similar to posix shm on top of it. if that is what
you need.

not sure what exactly you're looking for, though.

grant refs are concisely described in the interface manual.

regards,
dns

-- 
dns@somacoma.net

Wire up your home and stay there.

Re: Fwd: Inter-domain shared memory facility in Xen?

[Derek Murray]

David Stone wrote:
> Thanks for your help on this.  So based on your responses I just want
> to confirm my understanding is correct...is it true that:
> 
> 1. It is currently possible to establish shared memory between Dom0
> and DomU, but
> 2. It is currently not possible to establish shared memory between two
> DomUs in an released Xen versions, nor the current xen-unstable branch
> nor any other branch on http://xenbits.xensource.org.  There are
> projects that address this that you mentioned though.

It *is* possible to establish shared memory between two DomUs, using the 
grant table mechanism. At present, it is only possible to grant access 
to memory from a VM kernel, but it is possible to map that granted 
memory into either the kernel (using the kernel wrappers to the 
necessary hypercalls) or user-space (using gntdev, the user-space 
granted memory driver).

Up to and including the current stable release (3.1.1), it is only 
possible for domains that have I/O memory permissions (typically only 
Dom0) to map granted pages. Therefore, it is necessary to give some 
dummy permissions to any domain that you want to allow to map grants. 
However, the issue that led to this restriction has been fixed in the 
current version of xen-unstable.

As I understand it, the other projects that were mentioned provide a 
socket-like communication mechanism for Xen, with the advantage that 
they do not (necessarily) use the networking stack. I presume that you 
are looking for something simpler, such as creating a shared data 
structure between two or more domains. In that case, the mechanisms that 
  exist in Xen today and that I have described above are sufficient for 
your needs.

Regards,

Derek Murray.

Re: Fwd: Inter-domain shared memory facility in Xen?

[David Stone]

Thanks guys, I'm clear now.

On Nov 1, 2007 10:16 AM, Derek Murray <Derek.Murray@cl.cam.ac.uk> wrote:
> David Stone wrote:
> > Thanks for your help on this.  So based on your responses I just want
> > to confirm my understanding is correct...is it true that:
> >
> > 1. It is currently possible to establish shared memory between Dom0
> > and DomU, but
> > 2. It is currently not possible to establish shared memory between two
> > DomUs in an released Xen versions, nor the current xen-unstable branch
> > nor any other branch on http://xenbits.xensource.org.  There are
> > projects that address this that you mentioned though.
>
> It *is* possible to establish shared memory between two DomUs, using the
> grant table mechanism. At present, it is only possible to grant access
> to memory from a VM kernel, but it is possible to map that granted
> memory into either the kernel (using the kernel wrappers to the
> necessary hypercalls) or user-space (using gntdev, the user-space
> granted memory driver).
>
> Up to and including the current stable release (3.1.1), it is only
> possible for domains that have I/O memory permissions (typically only
> Dom0) to map granted pages. Therefore, it is necessary to give some
> dummy permissions to any domain that you want to allow to map grants.
> However, the issue that led to this restriction has been fixed in the
> current version of xen-unstable.
>
> As I understand it, the other projects that were mentioned provide a
> socket-like communication mechanism for Xen, with the advantage that
> they do not (necessarily) use the networking stack. I presume that you
> are looking for something simpler, such as creating a shared data
> structure between two or more domains. In that case, the mechanisms that
>  exist in Xen today and that I have described above are sufficient for
> your needs.
>
> Regards,
>
> Derek Murray.
>

Re: Fwd: Inter-domain shared memory facility in Xen?

[Mark Williamson]

A couple of my longterm personal projects involve interdomain shared memory 
too.

One of these is a Xen version of the IBM zSeries DCSS (which is basically a 
big lump of shared memory, which can optionally be used as a disk).  The 
other is XenFS which presents a filesystem interface.

The end goal for either of those would include the ability to share userspace 
application memory between domains using just an mmap() call, but I've not 
got that working yet ;-)

Cheers,
Mark

On Thursday 01 November 2007, David Stone wrote:
> Thanks guys, I'm clear now.
>
> On Nov 1, 2007 10:16 AM, Derek Murray <Derek.Murray@cl.cam.ac.uk> wrote:
> > David Stone wrote:
> > > Thanks for your help on this.  So based on your responses I just want
> > > to confirm my understanding is correct...is it true that:
> > >
> > > 1. It is currently possible to establish shared memory between Dom0
> > > and DomU, but
> > > 2. It is currently not possible to establish shared memory between two
> > > DomUs in an released Xen versions, nor the current xen-unstable branch
> > > nor any other branch on http://xenbits.xensource.org.  There are
> > > projects that address this that you mentioned though.
> >
> > It *is* possible to establish shared memory between two DomUs, using the
> > grant table mechanism. At present, it is only possible to grant access
> > to memory from a VM kernel, but it is possible to map that granted
> > memory into either the kernel (using the kernel wrappers to the
> > necessary hypercalls) or user-space (using gntdev, the user-space
> > granted memory driver).
> >
> > Up to and including the current stable release (3.1.1), it is only
> > possible for domains that have I/O memory permissions (typically only
> > Dom0) to map granted pages. Therefore, it is necessary to give some
> > dummy permissions to any domain that you want to allow to map grants.
> > However, the issue that led to this restriction has been fixed in the
> > current version of xen-unstable.
> >
> > As I understand it, the other projects that were mentioned provide a
> > socket-like communication mechanism for Xen, with the advantage that
> > they do not (necessarily) use the networking stack. I presume that you
> > are looking for something simpler, such as creating a shared data
> > structure between two or more domains. In that case, the mechanisms that
> >  exist in Xen today and that I have described above are sufficient for
> > your needs.
> >
> > Regards,
> >
> > Derek Murray.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!