[RFC] [0/4] User-space grants for Console and XenStore

[RFC] [0/4] User-space grants for Console and XenStore

[Derek Murray]

Following on from the introduction of a User-space grant table device  
("gntdev" - see http://lists.xensource.com/archives/html/xen-devel/ 
2007-03/msg01337.html for more details), I have modified the domain  
building process so that the grant table mechanism is used for  
communication with the console and XenStore daemons. This replaces  
the use of xc_map_foreign_range().

The development was carried out as follows:

* Added a new gnttabop, which installs grant table entries into the  
reserved section of the grant table.
* Modified the domain builder to install entries for the console and  
XenStore pages for each new domain (using fixed entry numbers).
* Modified the console and XenStore daemons to use  
xc_gnttab_map_grant_ref() in place of xc_map_foreign_range().

I've tested this locally (on x86_32 and x86_64) and it seems to work.  
There are a few remaining concerns that I have:

* Will this work on ia64 and PowerPC?
* How will this integrate with XSM? The install-grant operation is  
clearly as privileged as the ability to map arbitrary memory from a  
foreign domain, so could these share a hook?
* Solaris support? I don't know enough about Solaris to know what  
changes must be made to gntdev for it to run there. Since these  
changes hit presumably portable code, would it be acceptable to have  
an #ifdef in the console and XenStore daemons?

The patch is structured as follows:

1. gnttabop.patch - Changes to the hypervisor to add the install- 
entry gnttabop.
2. libxc.patch - Changes to libxc to install grants for the console  
and XenStore pages.
3. xenconsoled.patch - Changes to the console daemon to use grant  
references.
4. xenstored.patch - Changes to the XenStore daemon to use grant  
references.

I would welcome your input on this.

Regards,

Derek Murray.

Re: [RFC] [0/4] User-space grants for Console and XenStore

[John Levon]

On Wed, May 02, 2007 at 12:15:18PM +0100, Derek Murray wrote:

> Following on from the introduction of a User-space grant table device  
> ("gntdev" - see http://lists.xensource.com/archives/html/xen-devel/ 
> 2007-03/msg01337.html for more details), I have modified the domain  
>
> * Solaris support? I don't know enough about Solaris to know what  
> changes must be made to gntdev for it to run there. Since these  

It would be nice to have the old method still work, but if you feel the
need to there's no problem in dropping the old code. We're always
playing catch-up anyway...

regards
john

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Derek Murray]

Hi John,

On 2 May 2007, at 13:15, John Levon wrote:

> It would be nice to have the old method still work, but if you feel  
> the
> need to there's no problem in dropping the old code. We're always
> playing catch-up anyway...

One possibility would be to have a Makefile option that controls  
whether or not xc_map_foreign_range() is available, and swaps out the  
calls for xc_gnttab_map_grant_ref() where necessary. That way, if/ 
when Solaris gets gntdev, it will be easy to swap in the new code,  
and nothing will be broken in the mean time. I'd also be willing to  
provide any support needed for porting gntdev to Solaris.

Regards,

Derek.

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Keir Fraser]

On 2/5/07 13:29, "Derek Murray" <Derek.Murray@cl.cam.ac.uk> wrote:

>> It would be nice to have the old method still work, but if you feel
>> the
>> need to there's no problem in dropping the old code. We're always
>> playing catch-up anyway...
> 
> One possibility would be to have a Makefile option that controls
> whether or not xc_map_foreign_range() is available, and swaps out the
> calls for xc_gnttab_map_grant_ref() where necessary. That way, if/
> when Solaris gets gntdev, it will be easy to swap in the new code,
> and nothing will be broken in the mean time. I'd also be willing to
> provide any support needed for porting gntdev to Solaris.

It should be easy to support the old way and choose dynamically at run time.
We should somehow tag the values passed to or stored in xenstore explicitly
as being grant-refs, at the very least.

I would like to continue to avoid dependence on particular dom0 kernel
features, so new tool stack on old kernel (with no gnttab driver) should
really continue to work.

 -- Keir

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Derek Murray]

On 2 May 2007, at 13:46, Keir Fraser wrote:

> It should be easy to support the old way and choose dynamically at  
> run time.
> We should somehow tag the values passed to or stored in xenstore  
> explicitly
> as being grant-refs, at the very least.

At present, I don't store the grant-refs in xenstore, because they  
are constant (always 0 for xenstore, 1 for the console). I suppose I  
can see an argument where two different tool stacks are used with  
different conventions, but the same console and xenstore daemons are  
used. The tricky part is getting the grant references out of the  
domain builder without changing the interface to xc_linux_build().  
Would it be reasonable to assume that the current versions of xend  
and libxc are consistent, and therefore the fixed grant references  
could be written into xenstore (from constants in the Python code) at  
the same time as the console and xenstore mfns are?

> I would like to continue to avoid dependence on particular dom0 kernel
> features, so new tool stack on old kernel (with no gnttab driver)  
> should
> really continue to work.

This should be quite simple: if the call to xc_gnttab_open() fails,  
it could set a flag which determines the method to be used. All that  
needs to be done in the mean time is to add stub functions to  
xc_solaris.c - I'll do that.

Regards,

Derek.

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Keir Fraser]

On 2/5/07 15:38, "Derek Murray" <Derek.Murray@cl.cam.ac.uk> wrote:

> At present, I don't store the grant-refs in xenstore, because they
> are constant (always 0 for xenstore, 1 for the console). I suppose I
> can see an argument where two different tool stacks are used with
> different conventions, but the same console and xenstore daemons are
> used. The tricky part is getting the grant references out of the
> domain builder without changing the interface to xc_linux_build().
> Would it be reasonable to assume that the current versions of xend
> and libxc are consistent, and therefore the fixed grant references
> could be written into xenstore (from constants in the Python code) at
> the same time as the console and xenstore mfns are?

I would accept that degree of binding between libxenctrl.so and the Python
binding Xc.so, yes. The existing xenstore arguments/fields might already be
unhelpfully called grant-ref, by the way. :-) Very stupid of me if so.
You'll just have to be a bit imaginative about naming of new args/fields in
that case.

> This should be quite simple: if the call to xc_gnttab_open() fails,
> it could set a flag which determines the method to be used. All that
> needs to be done in the mean time is to add stub functions to
> xc_solaris.c - I'll do that.

It's as simple as try grant-table method, fall back to direct-map method,
then fail outright, isn't it? Maybe that's what you meant.

 -- Keir

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Derek Murray]

On 2 May 2007, at 15:48, Keir Fraser wrote:

> I would accept that degree of binding between libxenctrl.so and the  
> Python
> binding Xc.so, yes. The existing xenstore arguments/fields might  
> already be
> unhelpfully called grant-ref, by the way. :-) Very stupid of me if so.
> You'll just have to be a bit imaginative about naming of new args/ 
> fields in
> that case.

Not quite so bad - the current keys are called "ring-ref", so I'll  
probably add "ring-grant-ref" for the new entries.

> It's as simple as try grant-table method, fall back to direct-map  
> method,
> then fail outright, isn't it? Maybe that's what you meant.

It's even simpler in practice. If the gntdev isn't available, then  
xc_gnttab_open() will fail (as long as there's a stub in the Solaris  
version of libxc), so there will be no "xcg" handle with which to try  
to map the grant reference. Therefore we can go straight to the  
direct map method. I suppose, in addition, if we do have an xcg  
handle and the grant mapping fails for some reason, we could still  
fall back to the direct map method, though this failure should  
definitely be noted.

Regards,

Derek.

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Isaku Yamahata]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

On Wed, May 02, 2007 at 12:15:18PM +0100, Derek Murray wrote:
> * Will this work on ia64 and PowerPC?

With the attached patches, I was able to boot dom0 and create/destroy domU.
I used gcc (GCC) 3.4.4 20050721 (Red Hat 3.4.4-2).

BTW save/restore doesn't seem to work because when domain is
restored grant table is reset so that grant table entries should be
reinstalled again before introducing domain.

-- 
yamahata

[-- Attachment #2: 15129_13267399e63b_libxc_compilation_fix.patch --]
[-- Type: text/x-diff, Size: 1808 bytes --]

# HG changeset patch
# User yamahata@valinux.co.jp
# Date 1178524804 -32400
# Node ID 13267399e63b83125a12791c21b46811fc1878d5
# Parent  1ed965d4314ea1f3c186460eecda1a044a6db38e
libxc compilation fix
PATCHNAME: libxc_compilation_fix

Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>

diff -r 1ed965d4314e -r 13267399e63b tools/libxc/xc_dom_core.c
--- a/tools/libxc/xc_dom_core.c	Mon May 07 16:21:21 2007 +0900
+++ b/tools/libxc/xc_dom_core.c	Mon May 07 17:00:04 2007 +0900
@@ -774,7 +774,7 @@ int xc_dom_install_gnttab_entries(struct
 int xc_dom_install_gnttab_entries(struct xc_dom_image *dom,
                                   grant_handle_t start_handle,
                                   uint32_t count,
-                                  grant_entry_t entries[count])
+                                  grant_entry_t *entries)
 {
     int i, rc = -1;
     gnttab_install_entry_t *ops;    
diff -r 1ed965d4314e -r 13267399e63b tools/libxc/xc_dom_ia64.c
--- a/tools/libxc/xc_dom_ia64.c	Mon May 07 16:21:21 2007 +0900
+++ b/tools/libxc/xc_dom_ia64.c	Mon May 07 17:00:04 2007 +0900
@@ -28,6 +28,8 @@
 
 static int alloc_magic_pages(struct xc_dom_image *dom)
 {
+    grant_entry_t gnt_entries[2];
+
     /* allocate special pages */
     dom->console_pfn = dom->total_pages -1;
     dom->xenstore_pfn = dom->total_pages -2;
diff -r 1ed965d4314e -r 13267399e63b tools/libxc/xc_dom_powerpc.c
--- a/tools/libxc/xc_dom_powerpc.c	Mon May 07 16:21:21 2007 +0900
+++ b/tools/libxc/xc_dom_powerpc.c	Mon May 07 17:00:04 2007 +0900
@@ -37,6 +37,7 @@ static int alloc_magic_pages(struct xc_d
     unsigned long shadow_mb;
     int rma_pages;
     int rc;
+    grant_entry_t gnt_entries[2];
 
     /* Allocate special pages from the end of the RMA. */
     rma_pages = 1 << (dom->realmodearea_log - PAGE_SHIFT);

[-- Attachment #3: 15130_0417bce1329f_xencomm_gnttabop_isntall_entry.patch --]
[-- Type: text/x-diff, Size: 1991 bytes --]

# HG changeset patch
# User yamahata@valinux.co.jp
# Date 1178524861 -32400
# Node ID 0417bce1329f9b58e678140a6d14192bd98cde31
# Parent  13267399e63b83125a12791c21b46811fc1878d5
xencomm stub for GNTTABOP_install_entry
PATCHNAME: xencomm_gnttabop_isntall_entry

Signed-off-by: Isaku Yamahata <yamahata@valinux.co.jp>

diff -r 13267399e63b -r 0417bce1329f linux-2.6-xen-sparse/arch/ia64/xen/xcom_privcmd.c
--- a/linux-2.6-xen-sparse/arch/ia64/xen/xcom_privcmd.c	Mon May 07 17:00:04 2007 +0900
+++ b/linux-2.6-xen-sparse/arch/ia64/xen/xcom_privcmd.c	Mon May 07 17:01:01 2007 +0900
@@ -30,6 +30,7 @@
 #include <xen/interface/version.h>
 #include <xen/interface/event_channel.h>
 #include <xen/interface/acm_ops.h>
+#include <xen/interface/grant_table.h>
 #include <xen/interface/hvm/params.h>
 #include <xen/public/privcmd.h>
 #include <asm/hypercall.h>
@@ -710,6 +711,35 @@ xencomm_privcmd_ia64_dom0vp_op(privcmd_h
 		ret = -EINVAL;
 		break;
 	}
+	return ret;
+}
+
+static int
+xencomm_privcmd_grant_table_op(privcmd_hypercall_t *hypercall)
+{
+	int cmd = hypercall->arg[0];
+	unsigned int count = hypercall->arg[2];
+	unsigned int argsize;
+	struct xencomm_handle *desc;
+	int ret;
+	
+	switch (cmd) {
+	case GNTTABOP_install_entry:
+		argsize = sizeof(gnttab_install_entry_t) * count;
+		break;
+	default:
+		printk("%s: unknown GNTTABOP %d\n", __func__, cmd);
+		return -EINVAL;
+	}
+
+	ret = xencomm_create((void *)hypercall->arg[1], argsize,
+			     &desc, GFP_KERNEL);
+	if (ret)
+		return ret;
+
+	ret = xencomm_arch_hypercall_grant_table_op(cmd, desc, count);
+
+	xencomm_free(desc);
 	return ret;
 }
 
@@ -737,6 +767,8 @@ privcmd_hypercall(privcmd_hypercall_t *h
 		return xencomm_privcmd_sched_op(hypercall);
 	case __HYPERVISOR_ia64_dom0vp_op:
 		return xencomm_privcmd_ia64_dom0vp_op(hypercall);
+	case __HYPERVISOR_grant_table_op:
+		return xencomm_privcmd_grant_table_op(hypercall);
 	default:
 		printk("%s: unknown hcall (%ld)\n", __func__, hypercall->op);
 		return -ENOSYS;

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Derek Murray]

On 7 May 2007, at 09:46, Isaku Yamahata wrote:

> On Wed, May 02, 2007 at 12:15:18PM +0100, Derek Murray wrote:
>> * Will this work on ia64 and PowerPC?
>
> With the attached patches, I was able to boot dom0 and create/ 
> destroy domU.
> I used gcc (GCC) 3.4.4 20050721 (Red Hat 3.4.4-2).

Thanks for these patches! I'll incorporate them into the revised  
version of my patchset. It's good to know that gntdev is working on  
these architectures.

> BTW save/restore doesn't seem to work because when domain is
> restored grant table is reset so that grant table entries should be
> reinstalled again before introducing domain.

I'll take care of this as well.

Regards,

Derek Murray.

Re: [RFC] [0/4] User-space grants for Console and XenStore

[Isaku Yamahata]

On Tue, May 08, 2007 at 03:40:41PM +0100, Derek Murray wrote:
> 
> On 7 May 2007, at 09:46, Isaku Yamahata wrote:
> 
> >On Wed, May 02, 2007 at 12:15:18PM +0100, Derek Murray wrote:
> >>* Will this work on ia64 and PowerPC?
> >
> >With the attached patches, I was able to boot dom0 and create/ 
> >destroy domU.
> >I used gcc (GCC) 3.4.4 20050721 (Red Hat 3.4.4-2).
> 
> Thanks for these patches! I'll incorporate them into the revised  
> version of my patchset. It's good to know that gntdev is working on  
> these architectures.

I forgot to say that I only tested on IA64.

-- 
yamahata