[PATCH] Xen backend support for paged out grant targets V4.

[PATCH] Xen backend support for paged out grant targets V4.

[Andres Lagar-Cavilla]

Since Xen-4.2, hvm domains may have portions of their memory paged out. When a
foreign domain (such as dom0) attempts to map these frames, the map will
initially fail. The hypervisor returns a suitable errno, and kicks an
asynchronous page-in operation carried out by a helper. The foreign domain is
expected to retry the mapping operation until it eventually succeeds. The
foreign domain is not put to sleep because itself could be the one running the
pager assist (typical scenario for dom0).

This patch adds support for this mechanism for backend drivers using grant
mapping and copying operations. Specifically, this covers the blkback and
gntdev drivers (which map foregin grants), and the netback driver (which copies
foreign grants).

* Add a retry method for grants that fail with GNTST_eagain (i.e. because the
  target foregin frame is paged out).
* Insert hooks with appropriate wrappers in the aforementioned drivers.

The retry loop is only invoked if the grant operation status is GNTST_eagain.
It guarantees to leave a new status code different from GNTST_eagain. Any other
status code results in identical code execution as before.

The retry loop performs 256 attempts with increasing time intervals through a
32 second period. It uses msleep to yield while waiting for the next retry.

V2 after feedback from David Vrabel:
* Explicit MAX_DELAY instead of wrap-around delay into zero
* Abstract GNTST_eagain check into core grant table code for netback module.

V3 after feedback from Ian Campbell:
* Add placeholder in array of grant table error descriptions for unrelated
  error code we jump over.
* Eliminate single map and retry macro in favor of a generic batch flavor.
* Some renaming.
* Bury most implementation in grant_table.c, cleaner interface.

V4 rebased on top of sync of Xen grant table interface headers.

Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
---
 drivers/net/xen-netback/netback.c  |   11 ++------
 drivers/xen/grant-table.c          |   53 ++++++++++++++++++++++++++++++++++++
 drivers/xen/xenbus/xenbus_client.c |    6 ++--
 include/xen/grant_table.h          |   12 ++++++++
 4 files changed, 70 insertions(+), 12 deletions(-)

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 682633b..05593d8 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -635,9 +635,7 @@ static void xen_netbk_rx_action(struct xen_netbk *netbk)
 		return;
 
 	BUG_ON(npo.copy_prod > ARRAY_SIZE(netbk->grant_copy_op));
-	ret = HYPERVISOR_grant_table_op(GNTTABOP_copy, &netbk->grant_copy_op,
-					npo.copy_prod);
-	BUG_ON(ret != 0);
+	gnttab_batch_copy(netbk->grant_copy_op, npo.copy_prod);
 
 	while ((skb = __skb_dequeue(&rxq)) != NULL) {
 		sco = (struct skb_cb_overlay *)skb->cb;
@@ -1460,18 +1458,15 @@ static void xen_netbk_tx_submit(struct xen_netbk *netbk)
 static void xen_netbk_tx_action(struct xen_netbk *netbk)
 {
 	unsigned nr_gops;
-	int ret;
 
 	nr_gops = xen_netbk_tx_build_gops(netbk);
 
 	if (nr_gops == 0)
 		return;
-	ret = HYPERVISOR_grant_table_op(GNTTABOP_copy,
-					netbk->tx_copy_ops, nr_gops);
-	BUG_ON(ret);
 
-	xen_netbk_tx_submit(netbk);
+	gnttab_batch_copy(netbk->tx_copy_ops, nr_gops);
 
+	xen_netbk_tx_submit(netbk);
 }
 
 static void xen_netbk_idx_release(struct xen_netbk *netbk, u16 pending_idx)
diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
index eea81cf..f5681c8 100644
--- a/drivers/xen/grant-table.c
+++ b/drivers/xen/grant-table.c
@@ -38,6 +38,7 @@
 #include <linux/vmalloc.h>
 #include <linux/uaccess.h>
 #include <linux/io.h>
+#include <linux/delay.h>
 #include <linux/hardirq.h>
 
 #include <xen/xen.h>
@@ -823,6 +824,52 @@ unsigned int gnttab_max_grant_frames(void)
 }
 EXPORT_SYMBOL_GPL(gnttab_max_grant_frames);
 
+/* Handling of paged out grant targets (GNTST_eagain) */
+#define MAX_DELAY 256
+static inline void
+gnttab_retry_eagain_gop(unsigned int cmd, void *gop, int16_t *status,
+						const char *func)
+{
+	unsigned delay = 1;
+
+	do {
+		BUG_ON(HYPERVISOR_grant_table_op(cmd, gop, 1));
+		if (*status == GNTST_eagain)
+			msleep(delay++);
+	} while ((*status == GNTST_eagain) && (delay < MAX_DELAY));
+
+	if (delay >= MAX_DELAY) {
+		printk(KERN_ERR "%s: %s eagain grant\n", func, current->comm);
+		*status = GNTST_bad_page;
+	}
+}
+
+void gnttab_batch_map(struct gnttab_map_grant_ref *batch, unsigned count)
+{
+	struct gnttab_map_grant_ref *op;
+
+	if (HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref, batch, count))
+		BUG();
+	for (op = batch; op < batch + count; op++)
+		if (op->status == GNTST_eagain)
+			gnttab_retry_eagain_gop(GNTTABOP_map_grant_ref, op,
+									&op->status, __func__);
+}
+EXPORT_SYMBOL_GPL(gnttab_batch_map);
+
+void gnttab_batch_copy(struct gnttab_copy *batch, unsigned count)
+{
+	struct gnttab_copy *op;
+
+	if (HYPERVISOR_grant_table_op(GNTTABOP_copy, batch, count))
+		BUG();
+	for (op = batch; op < batch + count; op++)
+		if (op->status == GNTST_eagain)
+			gnttab_retry_eagain_gop(GNTTABOP_copy, op, &op->status,
+									__func__);
+}
+EXPORT_SYMBOL_GPL(gnttab_batch_copy);
+
 int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 		    struct gnttab_map_grant_ref *kmap_ops,
 		    struct page **pages, unsigned int count)
@@ -836,6 +883,12 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 	if (ret)
 		return ret;
 
+	/* Retry eagain maps */
+	for (i = 0; i < count; i++)
+		if (map_ops[i].status == GNTST_eagain)
+			gnttab_retry_eagain_gop(GNTTABOP_map_grant_ref, map_ops + i,
+                                    &map_ops[i].status, __func__);
+
 	if (xen_feature(XENFEAT_auto_translated_physmap))
 		return ret;
 
diff --git a/drivers/xen/xenbus/xenbus_client.c b/drivers/xen/xenbus/xenbus_client.c
index b3e146e..bcf3ba4 100644
--- a/drivers/xen/xenbus/xenbus_client.c
+++ b/drivers/xen/xenbus/xenbus_client.c
@@ -490,8 +490,7 @@ static int xenbus_map_ring_valloc_pv(struct xenbus_device *dev,
 
 	op.host_addr = arbitrary_virt_to_machine(pte).maddr;
 
-	if (HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref, &op, 1))
-		BUG();
+	gnttab_batch_map(&op, 1);
 
 	if (op.status != GNTST_okay) {
 		free_vm_area(area);
@@ -572,8 +571,7 @@ int xenbus_map_ring(struct xenbus_device *dev, int gnt_ref,
 	gnttab_set_map_op(&op, (unsigned long)vaddr, GNTMAP_host_map, gnt_ref,
 			  dev->otherend_id);
 
-	if (HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref, &op, 1))
-		BUG();
+	gnttab_batch_map(&op, 1);
 
 	if (op.status != GNTST_okay) {
 		xenbus_dev_fatal(dev, op.status,
diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
index 11e27c3..da9386e 100644
--- a/include/xen/grant_table.h
+++ b/include/xen/grant_table.h
@@ -189,4 +189,16 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
 int gnttab_unmap_refs(struct gnttab_unmap_grant_ref *unmap_ops,
 		      struct page **pages, unsigned int count, bool clear_pte);
 
+/* Perform a batch of grant map/copy operations. Retry every batch slot
+ * for which the hypervisor returns GNTST_eagain. This is typically due 
+ * to paged out target frames.
+ *
+ * Will retry for 1, 2, ... 255 ms, i.e. 256 times during 32 seconds.
+ *
+ * Return value in each iand every status field of the batch guaranteed
+ * to not be GNTST_eagain. 
+ */
+void gnttab_batch_map(struct gnttab_map_grant_ref *batch, unsigned count);
+void gnttab_batch_copy(struct gnttab_copy *batch, unsigned count);
+
 #endif /* __ASM_GNTTAB_H__ */
-- 
1.7.9.5

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Ian Campbell]

(I think I forgot to hit send on this on Friday, sorry. Also
s/xen.lists.org/lists.xen.org in the CC line...)

On Fri, 2012-09-14 at 15:26 +0100, Andres Lagar-Cavilla wrote:
> Since Xen-4.2, hvm domains may have portions of their memory paged out. When a
> foreign domain (such as dom0) attempts to map these frames, the map will
> initially fail. The hypervisor returns a suitable errno, and kicks an
> asynchronous page-in operation carried out by a helper. The foreign domain is
> expected to retry the mapping operation until it eventually succeeds. The
> foreign domain is not put to sleep because itself could be the one running the
> pager assist (typical scenario for dom0).
> 
> This patch adds support for this mechanism for backend drivers using grant
> mapping and copying operations. Specifically, this covers the blkback and
> gntdev drivers (which map foregin grants), and the netback driver (which copies

                           foreign

> foreign grants).
> 
> * Add a retry method for grants that fail with GNTST_eagain (i.e. because the
>   target foregin frame is paged out).

          foreign

> * Insert hooks with appropriate wrappers in the aforementioned drivers.
> 
> The retry loop is only invoked if the grant operation status is GNTST_eagain.
> It guarantees to leave a new status code different from GNTST_eagain. Any other
> status code results in identical code execution as before.
> 
> The retry loop performs 256 attempts with increasing time intervals through a
> 32 second period. It uses msleep to yield while waiting for the next retry.
[...]
> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Since this is more about grant tables than netback this should probably
go via Konrad rather than Dave, is that OK with you Dave?

(there's one more typo below)

> ---
>  drivers/net/xen-netback/netback.c  |   11 ++------
>  drivers/xen/grant-table.c          |   53 ++++++++++++++++++++++++++++++++++++
>  drivers/xen/xenbus/xenbus_client.c |    6 ++--
>  include/xen/grant_table.h          |   12 ++++++++
>  4 files changed, 70 insertions(+), 12 deletions(-)
[...]
> diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
> index 11e27c3..da9386e 100644
> --- a/include/xen/grant_table.h
> +++ b/include/xen/grant_table.h
> @@ -189,4 +189,16 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
>  int gnttab_unmap_refs(struct gnttab_unmap_grant_ref *unmap_ops,
>  		      struct page **pages, unsigned int count, bool clear_pte);
>  
> +/* Perform a batch of grant map/copy operations. Retry every batch slot
> + * for which the hypervisor returns GNTST_eagain. This is typically due 
> + * to paged out target frames.
> + *
> + * Will retry for 1, 2, ... 255 ms, i.e. 256 times during 32 seconds.
> + *
> + * Return value in each iand every status field of the batch guaranteed

                          and

> + * to not be GNTST_eagain. 
> + */
> +void gnttab_batch_map(struct gnttab_map_grant_ref *batch, unsigned count);
> +void gnttab_batch_copy(struct gnttab_copy *batch, unsigned count);
> +
>  #endif /* __ASM_GNTTAB_H__ */

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Andres Lagar-Cavilla]

On Sep 17, 2012, at 4:17 AM, Ian Campbell wrote:

> (I think I forgot to hit send on this on Friday, sorry. Also
> s/xen.lists.org/lists.xen.org in the CC line…)
I'm on a roll here…

> 
> On Fri, 2012-09-14 at 15:26 +0100, Andres Lagar-Cavilla wrote:
>> Since Xen-4.2, hvm domains may have portions of their memory paged out. When a
>> foreign domain (such as dom0) attempts to map these frames, the map will
>> initially fail. The hypervisor returns a suitable errno, and kicks an
>> asynchronous page-in operation carried out by a helper. The foreign domain is
>> expected to retry the mapping operation until it eventually succeeds. The
>> foreign domain is not put to sleep because itself could be the one running the
>> pager assist (typical scenario for dom0).
>> 
>> This patch adds support for this mechanism for backend drivers using grant
>> mapping and copying operations. Specifically, this covers the blkback and
>> gntdev drivers (which map foregin grants), and the netback driver (which copies
> 
>                           foreign
> 
>> foreign grants).
>> 
>> * Add a retry method for grants that fail with GNTST_eagain (i.e. because the
>>  target foregin frame is paged out).
> 
>          foreign
> 
>> * Insert hooks with appropriate wrappers in the aforementioned drivers.
>> 
>> The retry loop is only invoked if the grant operation status is GNTST_eagain.
>> It guarantees to leave a new status code different from GNTST_eagain. Any other
>> status code results in identical code execution as before.
>> 
>> The retry loop performs 256 attempts with increasing time intervals through a
>> 32 second period. It uses msleep to yield while waiting for the next retry.
> [...]
>> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
> 
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Since this is more about grant tables than netback this should probably
> go via Konrad rather than Dave, is that OK with you Dave?

If that is the case hopefully Konrad can deal with the two typos? Otherwise happy to re-spin the patch.
Thanks!
Andres

> 
> (there's one more typo below)
> 
>> ---
>> drivers/net/xen-netback/netback.c  |   11 ++------
>> drivers/xen/grant-table.c          |   53 ++++++++++++++++++++++++++++++++++++
>> drivers/xen/xenbus/xenbus_client.c |    6 ++--
>> include/xen/grant_table.h          |   12 ++++++++
>> 4 files changed, 70 insertions(+), 12 deletions(-)
> [...]
>> diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
>> index 11e27c3..da9386e 100644
>> --- a/include/xen/grant_table.h
>> +++ b/include/xen/grant_table.h
>> @@ -189,4 +189,16 @@ int gnttab_map_refs(struct gnttab_map_grant_ref *map_ops,
>> int gnttab_unmap_refs(struct gnttab_unmap_grant_ref *unmap_ops,
>> 		      struct page **pages, unsigned int count, bool clear_pte);
>> 
>> +/* Perform a batch of grant map/copy operations. Retry every batch slot
>> + * for which the hypervisor returns GNTST_eagain. This is typically due 
>> + * to paged out target frames.
>> + *
>> + * Will retry for 1, 2, ... 255 ms, i.e. 256 times during 32 seconds.
>> + *
>> + * Return value in each iand every status field of the batch guaranteed
> 
>                          and
> 
>> + * to not be GNTST_eagain. 
>> + */
>> +void gnttab_batch_map(struct gnttab_map_grant_ref *batch, unsigned count);
>> +void gnttab_batch_copy(struct gnttab_copy *batch, unsigned count);
>> +
>> #endif /* __ASM_GNTTAB_H__ */
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Konrad Rzeszutek Wilk]

> >> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
> > 
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> > 
> > Since this is more about grant tables than netback this should probably
> > go via Konrad rather than Dave, is that OK with you Dave?
> 
> If that is the case hopefully Konrad can deal with the two typos? Otherwise happy to re-spin the patch.

Sure. Just waiting for an Ack from David.

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Konrad Rzeszutek Wilk]

> >> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
> > 
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> > 
> > Since this is more about grant tables than netback this should probably
> > go via Konrad rather than Dave, is that OK with you Dave?
> 
> If that is the case hopefully Konrad can deal with the two typos? Otherwise happy to re-spin the patch.
> Thanks!

David, I pulled it in my tree since the only changes it does to drivers/net/xen-* is
change the name of the function to call in the bowels of grant API.

HYPERVISOR_grant_table_op(GNTTABOP_copy, netbk->tx_copy_ops, nr_gops);
to
gnttab_batch_copy(netbk->tx_copy_ops, nr_gops);

Hope that is OK - if not I can prep a branch that has patches that this depends
on that you can pull.

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Konrad Rzeszutek Wilk]

On Mon, Sep 17, 2012 at 05:29:24AM -0400, Andres Lagar-Cavilla wrote:
> On Sep 17, 2012, at 4:17 AM, Ian Campbell wrote:
> 
> > (I think I forgot to hit send on this on Friday, sorry. Also
> > s/xen.lists.org/lists.xen.org in the CC line…)
> I'm on a roll here…
> 
> > 
> > On Fri, 2012-09-14 at 15:26 +0100, Andres Lagar-Cavilla wrote:
> >> Since Xen-4.2, hvm domains may have portions of their memory paged out. When a
> >> foreign domain (such as dom0) attempts to map these frames, the map will
> >> initially fail. The hypervisor returns a suitable errno, and kicks an
> >> asynchronous page-in operation carried out by a helper. The foreign domain is
> >> expected to retry the mapping operation until it eventually succeeds. The
> >> foreign domain is not put to sleep because itself could be the one running the
> >> pager assist (typical scenario for dom0).
> >> 
> >> This patch adds support for this mechanism for backend drivers using grant
> >> mapping and copying operations. Specifically, this covers the blkback and
> >> gntdev drivers (which map foregin grants), and the netback driver (which copies
> > 
> >                           foreign
> > 
> >> foreign grants).
> >> 
> >> * Add a retry method for grants that fail with GNTST_eagain (i.e. because the
> >>  target foregin frame is paged out).
> > 
> >          foreign
> > 
> >> * Insert hooks with appropriate wrappers in the aforementioned drivers.
> >> 
> >> The retry loop is only invoked if the grant operation status is GNTST_eagain.
> >> It guarantees to leave a new status code different from GNTST_eagain. Any other
> >> status code results in identical code execution as before.
> >> 
> >> The retry loop performs 256 attempts with increasing time intervals through a
> >> 32 second period. It uses msleep to yield while waiting for the next retry.
> > [...]
> >> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
> > 
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> > 
> > Since this is more about grant tables than netback this should probably
> > go via Konrad rather than Dave, is that OK with you Dave?
> 
> If that is the case hopefully Konrad can deal with the two typos? Otherwise happy to re-spin the patch.

So with this patch when I launch an PVHVM guest on Xen 4.1 I get this
in the initial domain and the guest is crashed:

[  261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000

guest config:
> more /mnt/lab/latest/hvm.xm 
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory=1024
#maxmem=1024
maxvcpus = 2
serial='pty'
vcpus = 2
disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
boot="dn"
#vif = [ 'type=ioemu,model=e1000,mac=00:0F:4B:00:00:71, bridge=switch' ]
vif = [ 'type=netfront, bridge=switch' ]
#vfb = [ 'vnc=1, vnclisten=0.0.0.0 ,vncunused=1']
vnc=1
vnclisten="0.0.0.0"
usb=1
xen_platform_pci=1

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Andres Lagar-Cavilla]

On Sep 21, 2012, at 2:52 PM, Konrad Rzeszutek Wilk wrote:

> On Mon, Sep 17, 2012 at 05:29:24AM -0400, Andres Lagar-Cavilla wrote:
>> On Sep 17, 2012, at 4:17 AM, Ian Campbell wrote:
>> 
>>> (I think I forgot to hit send on this on Friday, sorry. Also
>>> s/xen.lists.org/lists.xen.org in the CC line…)
>> I'm on a roll here…
>> 
>>> 
>>> On Fri, 2012-09-14 at 15:26 +0100, Andres Lagar-Cavilla wrote:
>>>> Since Xen-4.2, hvm domains may have portions of their memory paged out. When a
>>>> foreign domain (such as dom0) attempts to map these frames, the map will
>>>> initially fail. The hypervisor returns a suitable errno, and kicks an
>>>> asynchronous page-in operation carried out by a helper. The foreign domain is
>>>> expected to retry the mapping operation until it eventually succeeds. The
>>>> foreign domain is not put to sleep because itself could be the one running the
>>>> pager assist (typical scenario for dom0).
>>>> 
>>>> This patch adds support for this mechanism for backend drivers using grant
>>>> mapping and copying operations. Specifically, this covers the blkback and
>>>> gntdev drivers (which map foregin grants), and the netback driver (which copies
>>> 
>>>                          foreign
>>> 
>>>> foreign grants).
>>>> 
>>>> * Add a retry method for grants that fail with GNTST_eagain (i.e. because the
>>>> target foregin frame is paged out).
>>> 
>>>         foreign
>>> 
>>>> * Insert hooks with appropriate wrappers in the aforementioned drivers.
>>>> 
>>>> The retry loop is only invoked if the grant operation status is GNTST_eagain.
>>>> It guarantees to leave a new status code different from GNTST_eagain. Any other
>>>> status code results in identical code execution as before.
>>>> 
>>>> The retry loop performs 256 attempts with increasing time intervals through a
>>>> 32 second period. It uses msleep to yield while waiting for the next retry.
>>> [...]
>>>> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
>>> 
>>> Acked-by: Ian Campbell <ian.campbell@citrix.com>
>>> 
>>> Since this is more about grant tables than netback this should probably
>>> go via Konrad rather than Dave, is that OK with you Dave?
>> 
>> If that is the case hopefully Konrad can deal with the two typos? Otherwise happy to re-spin the patch.
> 
> So with this patch when I launch an PVHVM guest on Xen 4.1 I get this
> in the initial domain and the guest is crashed:
> 
> [  261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000

With this patch? Or with the mmapbatch v2? This is a page fault in a foreign-mapped VMA. Not touched by this grant backend patch we are talking about.

Does the hypervisor dump anything to its console?

At which point during xc_hvm_build do you see this? (or elsewhere in the toolstack?)

Thanks
Andres

> 
> guest config:
>> more /mnt/lab/latest/hvm.xm 
> kernel = "/usr/lib/xen/boot/hvmloader"
> builder='hvm'
> memory=1024
> #maxmem=1024
> maxvcpus = 2
> serial='pty'
> vcpus = 2
> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> boot="dn"
> #vif = [ 'type=ioemu,model=e1000,mac=00:0F:4B:00:00:71, bridge=switch' ]
> vif = [ 'type=netfront, bridge=switch' ]
> #vfb = [ 'vnc=1, vnclisten=0.0.0.0 ,vncunused=1']
> vnc=1
> vnclisten="0.0.0.0"
> usb=1
> xen_platform_pci=1
> 
> 

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Konrad Rzeszutek Wilk]

On Fri, Sep 21, 2012 at 03:30:01PM -0400, Andres Lagar-Cavilla wrote:
> On Sep 21, 2012, at 2:52 PM, Konrad Rzeszutek Wilk wrote:
> 
> > On Mon, Sep 17, 2012 at 05:29:24AM -0400, Andres Lagar-Cavilla wrote:
> >> On Sep 17, 2012, at 4:17 AM, Ian Campbell wrote:
> >> 
> >>> (I think I forgot to hit send on this on Friday, sorry. Also
> >>> s/xen.lists.org/lists.xen.org in the CC line…)
> >> I'm on a roll here…
> >> 
> >>> 
> >>> On Fri, 2012-09-14 at 15:26 +0100, Andres Lagar-Cavilla wrote:
> >>>> Since Xen-4.2, hvm domains may have portions of their memory paged out. When a
> >>>> foreign domain (such as dom0) attempts to map these frames, the map will
> >>>> initially fail. The hypervisor returns a suitable errno, and kicks an
> >>>> asynchronous page-in operation carried out by a helper. The foreign domain is
> >>>> expected to retry the mapping operation until it eventually succeeds. The
> >>>> foreign domain is not put to sleep because itself could be the one running the
> >>>> pager assist (typical scenario for dom0).
> >>>> 
> >>>> This patch adds support for this mechanism for backend drivers using grant
> >>>> mapping and copying operations. Specifically, this covers the blkback and
> >>>> gntdev drivers (which map foregin grants), and the netback driver (which copies
> >>> 
> >>>                          foreign
> >>> 
> >>>> foreign grants).
> >>>> 
> >>>> * Add a retry method for grants that fail with GNTST_eagain (i.e. because the
> >>>> target foregin frame is paged out).
> >>> 
> >>>         foreign
> >>> 
> >>>> * Insert hooks with appropriate wrappers in the aforementioned drivers.
> >>>> 
> >>>> The retry loop is only invoked if the grant operation status is GNTST_eagain.
> >>>> It guarantees to leave a new status code different from GNTST_eagain. Any other
> >>>> status code results in identical code execution as before.
> >>>> 
> >>>> The retry loop performs 256 attempts with increasing time intervals through a
> >>>> 32 second period. It uses msleep to yield while waiting for the next retry.
> >>> [...]
> >>>> Signed-off-by: Andres Lagar-Cavilla <andres@lagarcavilla.org>
> >>> 
> >>> Acked-by: Ian Campbell <ian.campbell@citrix.com>
> >>> 
> >>> Since this is more about grant tables than netback this should probably
> >>> go via Konrad rather than Dave, is that OK with you Dave?
> >> 
> >> If that is the case hopefully Konrad can deal with the two typos? Otherwise happy to re-spin the patch.
> > 
> > So with this patch when I launch an PVHVM guest on Xen 4.1 I get this
> > in the initial domain and the guest is crashed:
> > 
> > [  261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
> 
> With this patch? Or with the mmapbatch v2? This is a page fault in a foreign-mapped VMA. Not touched by this grant backend patch we are talking about.

This patch. But I also had a modified blkback. So let me double check
that it is not the persistent grants and this patch doing something naughty.

> 
> Does the hypervisor dump anything to its console?
> 
> At which point during xc_hvm_build do you see this? (or elsewhere in the toolstack?)

No idea. Didn't examine that much.

Re: [PATCH] Xen backend support for paged out grant targets V4.

[Konrad Rzeszutek Wilk]

> > So with this patch when I launch an PVHVM guest on Xen 4.1 I get this
> > in the initial domain and the guest is crashed:
> > 
> > [  261.927218] privcmd_fault: vma=ffff88002a31dce8 7f4edc095000-7f4edc195000, pgoff=c8, uv=00007f4edc15d000
> 
> With this patch? Or with the mmapbatch v2? This is a page fault in a foreign-mapped VMA. Not touched by this grant backend patch we are talking about.
> 
> Does the hypervisor dump anything to its console?
> 
> At which point during xc_hvm_build do you see this? (or elsewhere in the toolstack?)

My apologies. It was Oliver's persistent grant patch that does not like your patch.