[PATCH 00/21] xenpaging changes for xen-unstable

[PATCH 00/21] xenpaging changes for xen-unstable

[Olaf Hering]

Here are some changes for xenpaging in xen-unstable.

Patches 1 to 14 are likely non-controversial and could be applied.
All later patches need more review (15/16), and more work from my side (17+).

With a similar series of changes for 4.0.1 my testcase, which is a continuous
reboot of a Linux guest, passes now. It ran for more than 1000 iterations and
exercised the early bootcode.

There are still some issues like guests not starting up when running the
reboot loop with 4x1G guests parallel on a 4GB box with all guest memory paged.
This seems to happen when the guest initially needs the full 1G until
xenpaging frees that memory again.

Other than that, xenpaging is now more mature.
I havent seen data corruption anymore.

Olaf

[PATCH 01/21] xenpaging: break endless loop during inital page-out with large pagefiles

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.policy_linear.patch --]
[-- Type: text/plain, Size: 3879 bytes --]

To allow the starting for xenpaging right after 'xm start XYZ', I
specified a pagefile size equal to the guest memory size in the hope to
catch more errors where the paged-out state of a p2mt is not checked.

While doing that, xenpaging got into an endless loop because some pages
cant be paged out right away. Now the policy reports an error if the gfn
number wraps.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
Already-Acked-by: Patrick Colp <pjcolp@cs.ubc.ca>
Already-Acked-by: Keir Fraser <keir.fraser@citrix.com>

---
 tools/xenpaging/policy_default.c |   35 ++++++++++++++++++++++++++++-------
 tools/xenpaging/xenpaging.c      |    7 +++++--
 2 files changed, 33 insertions(+), 9 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/policy_default.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/policy_default.c
@@ -30,8 +30,12 @@
 
 
 static unsigned long mru[MRU_SIZE];
-static unsigned int i_mru = 0;
+static unsigned int i_mru;
 static unsigned long *bitmap;
+static unsigned long *unconsumed;
+static unsigned long current_gfn;
+static unsigned long bitmap_size;
+static unsigned long max_pages;
 
 
 int policy_init(xenpaging_t *paging)
@@ -43,6 +47,14 @@ int policy_init(xenpaging_t *paging)
     rc = alloc_bitmap(&bitmap, paging->bitmap_size);
     if ( rc != 0 )
         goto out;
+    /* Allocate bitmap to track unusable pages */
+    rc = alloc_bitmap(&unconsumed, paging->bitmap_size);
+    if ( rc != 0 )
+        goto out;
+
+    /* record bitmap_size */
+    bitmap_size = paging->bitmap_size;
+    max_pages = paging->domain_info->max_pages;
 
     /* Initialise MRU list of paged in pages */
     for ( i = 0; i < MRU_SIZE; i++ )
@@ -51,8 +63,6 @@ int policy_init(xenpaging_t *paging)
     /* Don't page out page 0 */
     set_bit(0, bitmap);
 
-    rc = 0;
-
  out:
     return rc;
 }
@@ -61,17 +71,27 @@ int policy_choose_victim(xc_interface *x
                          xenpaging_t *paging, domid_t domain_id,
                          xenpaging_victim_t *victim)
 {
+    unsigned long wrap = current_gfn;
     ASSERT(victim != NULL);
 
     /* Domain to pick on */
     victim->domain_id = domain_id;
-    
+
     do
     {
-        /* Randomly choose a gfn to evict */
-        victim->gfn = rand() % paging->domain_info->max_pages;
+        current_gfn++;
+        if ( current_gfn >= max_pages )
+            current_gfn = 0;
+        if ( wrap == current_gfn )
+        {
+            victim->gfn = INVALID_MFN;
+            return -ENOSPC;
+        }
     }
-    while ( test_bit(victim->gfn, bitmap) );
+    while ( test_bit(current_gfn, bitmap) || test_bit(current_gfn, unconsumed) );
+
+    set_bit(current_gfn, unconsumed);
+    victim->gfn = current_gfn;
 
     return 0;
 }
@@ -79,6 +99,7 @@ int policy_choose_victim(xc_interface *x
 void policy_notify_paged_out(domid_t domain_id, unsigned long gfn)
 {
     set_bit(gfn, bitmap);
+    clear_bit(gfn, unconsumed);
 }
 
 void policy_notify_paged_in(domid_t domain_id, unsigned long gfn)
--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -446,7 +446,8 @@ static int evict_victim(xc_interface *xc
         ret = policy_choose_victim(xch, paging, domain_id, victim);
         if ( ret != 0 )
         {
-            ERROR("Error choosing victim");
+            if ( ret != -ENOSPC )
+                ERROR("Error choosing victim");
             goto out;
         }
 
@@ -525,7 +526,9 @@ int main(int argc, char *argv[])
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )
     {
-        evict_victim(xch, paging, domain_id, &victims[i], fd, i);
+        rc = evict_victim(xch, paging, domain_id, &victims[i], fd, i);
+        if ( rc == -ENOSPC )
+            break;
         if ( i % 100 == 0 )
             DPRINTF("%d pages evicted\n", i);
     }

[PATCH 02/21] xenpaging: Open paging file only if xenpaging_init() succeeds

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.pagefile.patch --]
[-- Type: text/plain, Size: 1408 bytes --]

Open paging file only if xenpaging_init() succeeds. It can fail if the host
does not support the required virtualization features such as EPT or if
xenpaging was already started for this domain_id.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
Already-Acked-by: Patrick Colp <pjcolp@cs.ubc.ca>
Already-Acked-by: Keir Fraser <keir.fraser@citrix.com>

---
 tools/xenpaging/xenpaging.c |   18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -502,15 +502,6 @@ int main(int argc, char *argv[])
 
     victims = calloc(num_pages, sizeof(xenpaging_victim_t));
 
-    /* Open file */
-    sprintf(filename, "page_cache_%d", domain_id);
-    fd = open(filename, open_flags, open_mode);
-    if ( fd < 0 )
-    {
-        perror("failed to open file");
-        return -1;
-    }
-
     /* Seed random-number generator */
     srand(time(NULL));
 
@@ -522,6 +513,15 @@ int main(int argc, char *argv[])
         goto out;
     }
 
+    /* Open file */
+    sprintf(filename, "page_cache_%d", domain_id);
+    fd = open(filename, open_flags, open_mode);
+    if ( fd < 0 )
+    {
+        perror("failed to open file");
+        return -1;
+    }
+
     /* Evict pages */
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )

[PATCH 03/21] xenpaging: allow only one xenpaging binary per guest

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.xenpaging_init.patch --]
[-- Type: text/plain, Size: 1792 bytes --]

Make sure only one xenpaging binary is active per domain.
Print info when the host lacks the required features for xenpaging.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
Already-Acked-by: Patrick Colp <pjcolp@cs.ubc.ca>
Already-Acked-by: Keir Fraser <keir.fraser@citrix.com>

---
v2: use perror for default case

 tools/xenpaging/xenpaging.c |   12 +++++++++++-
 xen/arch/x86/mm/mem_event.c |    7 +++++++
 2 files changed, 18 insertions(+), 1 deletion(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -123,7 +123,17 @@ xenpaging_t *xenpaging_init(xc_interface
                              paging->mem_event.ring_page);
     if ( rc != 0 )
     {
-        ERROR("Error initialising shared page");
+        switch ( errno ) {
+            case EBUSY:
+                ERROR("xenpaging is (or was) active on this domain");
+                break;
+            case ENODEV:
+                ERROR("EPT not supported for this guest");
+                break;
+            default:
+                perror("Error initialising shared page");
+                break;
+        }
         goto err;
     }
 
--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/mem_event.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/mem_event.c
@@ -214,6 +214,13 @@ int mem_event_domctl(struct domain *d, x
             mfn_t ring_mfn;
             mfn_t shared_mfn;
 
+            /* Only one xenpaging at a time. If xenpaging crashed,
+             * the cache is in an undefined state and so is the guest
+             */
+            rc = -EBUSY;
+            if ( d->mem_event.ring_page )
+                break;
+
             /* Currently only EPT is supported */
             rc = -ENODEV;
             if ( !(hap_enabled(d) &&

[PATCH 04/21] xenpaging: populate paged-out pages unconditionally

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.populate_only_if_paged.patch --]
[-- Type: text/plain, Size: 958 bytes --]

Populate a page unconditionally to avoid missing a page-in request.
If the page is already in the process of being paged-in, the this vcpu
will be stopped and later resumed once the page content is usable again.

This matches other p2m_mem_paging_populate usage in the source tree.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 xen/common/grant_table.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/xen/common/grant_table.c
+++ xen-unstable.hg-4.1.22433/xen/common/grant_table.c
@@ -156,10 +156,11 @@ static int __get_paged_frame(unsigned lo
 
     if ( p2m_is_valid(p2mt) ) {
         *frame = mfn_x(mfn);
-        if ( p2m_is_paged(p2mt) )
-            p2m_mem_paging_populate(p2m, gfn);
         if ( p2m_is_paging(p2mt) )
+        {
+            p2m_mem_paging_populate(p2m, gfn);
             rc = GNTST_eagain;
+        }
     } else {
        *frame = INVALID_MFN;
        rc = GNTST_bad_page;

[PATCH 05/21] xenpaging: add signal handling

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.signal_handling.patch --]
[-- Type: text/plain, Size: 4575 bytes --]

Leave paging loop if xenpaging gets a signal.
Remove paging file on exit.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
v2:
 unlink pagefile in signal handler to avoid stale pagefiles if xenpaging is
 stuck in some loop

 tools/xenpaging/xenpaging.c |   42 +++++++++++++++++++++++++++++++++---------
 1 file changed, 33 insertions(+), 9 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -25,6 +25,7 @@
 #include <stdlib.h>
 #include <stdarg.h>
 #include <time.h>
+#include <signal.h>
 #include <xc_private.h>
 
 #include <xen/mem_event.h>
@@ -43,6 +44,14 @@
 #define DPRINTF(...) ((void)0)
 #endif
 
+static char filename[80];
+static int interrupted;
+static void close_handler(int sig)
+{
+    interrupted = sig;
+    if ( filename[0] )
+        unlink(filename);
+}
 
 static void *init_page(void)
 {
@@ -248,7 +257,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error tearing down domain paging in xen");
-        goto err;
     }
 
     /* Unbind VIRQ */
@@ -256,7 +264,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error unbinding event port");
-        goto err;
     }
     paging->mem_event.port = -1;
 
@@ -265,7 +272,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error closing event channel");
-        goto err;
     }
     paging->mem_event.xce_handle = -1;
     
@@ -274,7 +280,6 @@ int xenpaging_teardown(xc_interface *xch
     if ( rc != 0 )
     {
         ERROR("Error closing connection to xen");
-        goto err;
     }
     paging->xc_handle = NULL;
 
@@ -380,7 +385,7 @@ int xenpaging_evict_page(xc_interface *x
     return ret;
 }
 
-int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
+static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
 {
     int ret;
 
@@ -461,6 +466,11 @@ static int evict_victim(xc_interface *xc
             goto out;
         }
 
+        if ( interrupted )
+        {
+            ret = -EINTR;
+            goto out;
+        }
         ret = xc_mem_paging_nominate(paging->xc_handle,
                                      paging->mem_event.domain_id, victim->gfn);
         if ( ret == 0 )
@@ -485,6 +495,7 @@ static int evict_victim(xc_interface *xc
 
 int main(int argc, char *argv[])
 {
+    struct sigaction act;
     domid_t domain_id;
     int num_pages;
     xenpaging_t *paging;
@@ -498,7 +509,6 @@ int main(int argc, char *argv[])
 
     int open_flags = O_CREAT | O_TRUNC | O_RDWR;
     mode_t open_mode = S_IRUSR | S_IRGRP | S_IROTH | S_IWUSR | S_IWGRP | S_IWOTH;
-    char filename[80];
     int fd;
 
     if ( argc != 3 )
@@ -520,7 +530,7 @@ int main(int argc, char *argv[])
     if ( paging == NULL )
     {
         ERROR("Error initialising paging");
-        goto out;
+        return 1;
     }
 
     /* Open file */
@@ -529,9 +539,18 @@ int main(int argc, char *argv[])
     if ( fd < 0 )
     {
         perror("failed to open file");
-        return -1;
+        return 2;
     }
 
+    /* ensure that if we get a signal, we'll do cleanup, then exit */
+    act.sa_handler = close_handler;
+    act.sa_flags = 0;
+    sigemptyset(&act.sa_mask);
+    sigaction(SIGHUP,  &act, NULL);
+    sigaction(SIGTERM, &act, NULL);
+    sigaction(SIGINT,  &act, NULL);
+    sigaction(SIGALRM, &act, NULL);
+
     /* Evict pages */
     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
     for ( i = 0; i < num_pages; i++ )
@@ -539,6 +558,8 @@ int main(int argc, char *argv[])
         rc = evict_victim(xch, paging, domain_id, &victims[i], fd, i);
         if ( rc == -ENOSPC )
             break;
+        if ( rc == -EINTR )
+            break;
         if ( i % 100 == 0 )
             DPRINTF("%d pages evicted\n", i);
     }
@@ -546,7 +567,7 @@ int main(int argc, char *argv[])
     DPRINTF("pages evicted\n");
 
     /* Swap pages in and out */
-    while ( 1 )
+    while ( !interrupted )
     {
         /* Wait for Xen to signal that a page needs paged in */
         rc = xc_wait_for_event_or_timeout(xch, paging->mem_event.xce_handle, 100);
@@ -637,8 +658,10 @@ int main(int argc, char *argv[])
             }
         }
     }
+    DPRINTF("xenpaging got signal %d\n", interrupted);
 
  out:
+    close(fd);
     free(victims);
 
     /* Tear down domain paging */
@@ -651,6 +674,7 @@ int main(int argc, char *argv[])
 
     xc_interface_close(xch);
 
+    DPRINTF("xenpaging exit code %d\n", rc);
     return rc;
 }

Re: [PATCH 05/21] xenpaging: add signal handling

[George Dunlap]

Olaf, I haven't been looking at these patches as we've been going
along, but there seem to be two things happening in this patch not
mentioned in the description:

* Making xenpaging_teardown() not skip when a tear-down item fails,
but continue to try to tear down the rest
* Making return values for the program as a whole (1 for initializing
the paging, 2 for a failed file open)

These kinds of things should at least be mentioned in the description;
and I would personally probably pull them out and put them in a
separate patch.

 -George

On Fri, Nov 26, 2010 at 1:49 PM, Olaf Hering <olaf@aepfle.de> wrote:
> Leave paging loop if xenpaging gets a signal.
> Remove paging file on exit.
>
> Signed-off-by: Olaf Hering <olaf@aepfle.de>
>
> ---
> v2:
>  unlink pagefile in signal handler to avoid stale pagefiles if xenpaging is
>  stuck in some loop
>
>  tools/xenpaging/xenpaging.c |   42 +++++++++++++++++++++++++++++++++---------
>  1 file changed, 33 insertions(+), 9 deletions(-)
>
> --- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
> +++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
> @@ -25,6 +25,7 @@
>  #include <stdlib.h>
>  #include <stdarg.h>
>  #include <time.h>
> +#include <signal.h>
>  #include <xc_private.h>
>
>  #include <xen/mem_event.h>
> @@ -43,6 +44,14 @@
>  #define DPRINTF(...) ((void)0)
>  #endif
>
> +static char filename[80];
> +static int interrupted;
> +static void close_handler(int sig)
> +{
> +    interrupted = sig;
> +    if ( filename[0] )
> +        unlink(filename);
> +}
>
>  static void *init_page(void)
>  {
> @@ -248,7 +257,6 @@ int xenpaging_teardown(xc_interface *xch
>     if ( rc != 0 )
>     {
>         ERROR("Error tearing down domain paging in xen");
> -        goto err;
>     }
>
>     /* Unbind VIRQ */
> @@ -256,7 +264,6 @@ int xenpaging_teardown(xc_interface *xch
>     if ( rc != 0 )
>     {
>         ERROR("Error unbinding event port");
> -        goto err;
>     }
>     paging->mem_event.port = -1;
>
> @@ -265,7 +272,6 @@ int xenpaging_teardown(xc_interface *xch
>     if ( rc != 0 )
>     {
>         ERROR("Error closing event channel");
> -        goto err;
>     }
>     paging->mem_event.xce_handle = -1;
>
> @@ -274,7 +280,6 @@ int xenpaging_teardown(xc_interface *xch
>     if ( rc != 0 )
>     {
>         ERROR("Error closing connection to xen");
> -        goto err;
>     }
>     paging->xc_handle = NULL;
>
> @@ -380,7 +385,7 @@ int xenpaging_evict_page(xc_interface *x
>     return ret;
>  }
>
> -int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
> +static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
>  {
>     int ret;
>
> @@ -461,6 +466,11 @@ static int evict_victim(xc_interface *xc
>             goto out;
>         }
>
> +        if ( interrupted )
> +        {
> +            ret = -EINTR;
> +            goto out;
> +        }
>         ret = xc_mem_paging_nominate(paging->xc_handle,
>                                      paging->mem_event.domain_id, victim->gfn);
>         if ( ret == 0 )
> @@ -485,6 +495,7 @@ static int evict_victim(xc_interface *xc
>
>  int main(int argc, char *argv[])
>  {
> +    struct sigaction act;
>     domid_t domain_id;
>     int num_pages;
>     xenpaging_t *paging;
> @@ -498,7 +509,6 @@ int main(int argc, char *argv[])
>
>     int open_flags = O_CREAT | O_TRUNC | O_RDWR;
>     mode_t open_mode = S_IRUSR | S_IRGRP | S_IROTH | S_IWUSR | S_IWGRP | S_IWOTH;
> -    char filename[80];
>     int fd;
>
>     if ( argc != 3 )
> @@ -520,7 +530,7 @@ int main(int argc, char *argv[])
>     if ( paging == NULL )
>     {
>         ERROR("Error initialising paging");
> -        goto out;
> +        return 1;
>     }
>
>     /* Open file */
> @@ -529,9 +539,18 @@ int main(int argc, char *argv[])
>     if ( fd < 0 )
>     {
>         perror("failed to open file");
> -        return -1;
> +        return 2;
>     }
>
> +    /* ensure that if we get a signal, we'll do cleanup, then exit */
> +    act.sa_handler = close_handler;
> +    act.sa_flags = 0;
> +    sigemptyset(&act.sa_mask);
> +    sigaction(SIGHUP,  &act, NULL);
> +    sigaction(SIGTERM, &act, NULL);
> +    sigaction(SIGINT,  &act, NULL);
> +    sigaction(SIGALRM, &act, NULL);
> +
>     /* Evict pages */
>     memset(victims, 0, sizeof(xenpaging_victim_t) * num_pages);
>     for ( i = 0; i < num_pages; i++ )
> @@ -539,6 +558,8 @@ int main(int argc, char *argv[])
>         rc = evict_victim(xch, paging, domain_id, &victims[i], fd, i);
>         if ( rc == -ENOSPC )
>             break;
> +        if ( rc == -EINTR )
> +            break;
>         if ( i % 100 == 0 )
>             DPRINTF("%d pages evicted\n", i);
>     }
> @@ -546,7 +567,7 @@ int main(int argc, char *argv[])
>     DPRINTF("pages evicted\n");
>
>     /* Swap pages in and out */
> -    while ( 1 )
> +    while ( !interrupted )
>     {
>         /* Wait for Xen to signal that a page needs paged in */
>         rc = xc_wait_for_event_or_timeout(xch, paging->mem_event.xce_handle, 100);
> @@ -637,8 +658,10 @@ int main(int argc, char *argv[])
>             }
>         }
>     }
> +    DPRINTF("xenpaging got signal %d\n", interrupted);
>
>  out:
> +    close(fd);
>     free(victims);
>
>     /* Tear down domain paging */
> @@ -651,6 +674,7 @@ int main(int argc, char *argv[])
>
>     xc_interface_close(xch);
>
> +    DPRINTF("xenpaging exit code %d\n", rc);
>     return rc;
>  }
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: [PATCH 05/21] xenpaging: add signal handling

[Olaf Hering]

On Fri, Nov 26, George Dunlap wrote:

> Olaf, I haven't been looking at these patches as we've been going
> along, but there seem to be two things happening in this patch not
> mentioned in the description:

This was a "grown" patch.

> * Making xenpaging_teardown() not skip when a tear-down item fails,
> but continue to try to tear down the rest

If a domain is shutting down, xc_mem_event_disable will always fail
because d->is_dying is checked. Thats why I removed the bail_out part.

> * Making return values for the program as a whole (1 for initializing
> the paging, 2 for a failed file open)

return codes are currently not perfect, sometimes -1 is leaked.
I think xenpaging should either return 0 or 1.

> These kinds of things should at least be mentioned in the description;
> and I would personally probably pull them out and put them in a
> separate patch.

Will do better next time.
Thanks for the comment.


Olaf

Re: [PATCH 05/21] xenpaging: add signal handling

[George Dunlap]

Yeah, they certainly seem like reasonable changes.  They just need to
be pointed out, so that people have an idea the actual impact of the
patch (both now when accepting it, and later when going back trying to
figure out where something broke).

FYI, I normally do a bunch of work (involving a large number of
changes) first, then do "hg diff > working.diff ; hg update -C" and
then go through working.diff to see where the individual changes are
best suited, whether in a new patch, or in a modification to an
existing patch.  (emacs diff-mode is really helpful here.)  That helps
me review my own code, and promotes good patch hygiene. :-)

Peace,
 -George

On Thu, Dec 2, 2010 at 9:59 AM, Olaf Hering <olaf@aepfle.de> wrote:
> On Fri, Nov 26, George Dunlap wrote:
>
>> Olaf, I haven't been looking at these patches as we've been going
>> along, but there seem to be two things happening in this patch not
>> mentioned in the description:
>
> This was a "grown" patch.
>
>> * Making xenpaging_teardown() not skip when a tear-down item fails,
>> but continue to try to tear down the rest
>
> If a domain is shutting down, xc_mem_event_disable will always fail
> because d->is_dying is checked. Thats why I removed the bail_out part.
>
>> * Making return values for the program as a whole (1 for initializing
>> the paging, 2 for a failed file open)
>
> return codes are currently not perfect, sometimes -1 is leaked.
> I think xenpaging should either return 0 or 1.
>
>> These kinds of things should at least be mentioned in the description;
>> and I would personally probably pull them out and put them in a
>> separate patch.
>
> Will do better next time.
> Thanks for the comment.
>
>
> Olaf
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[PATCH 06/21] xenpaging: print info when free request slots drop below 2

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.mem_event_check_ring-free_requests.patch --]
[-- Type: text/plain, Size: 842 bytes --]

Add debugging aid to free request slots in the ring buffer.
It should not happen that the ring gets full, print info anyway if it happens. 

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 xen/arch/x86/mm/mem_event.c |    5 +++++
 1 file changed, 5 insertions(+)

--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/mem_event.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/mem_event.c
@@ -152,6 +152,11 @@ int mem_event_check_ring(struct domain *
     mem_event_ring_lock(d);
 
     free_requests = RING_FREE_REQUESTS(&d->mem_event.front_ring);
+    if ( unlikely(free_requests < 2) )
+    {
+        gdprintk(XENLOG_INFO, "free request slots: %d\n", free_requests);
+        WARN_ON(free_requests == 0);
+    }
     ring_full = free_requests < MEM_EVENT_RING_THRESHOLD;
 
     if ( (curr->domain->domain_id == d->domain_id) && ring_full )

[PATCH 07/21] xenpaging: print p2mt for already paged-in pages

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.page_already_populated.patch --]
[-- Type: text/plain, Size: 824 bytes --]

Add more debug output, print p2mt for pages which were requested more than once.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/xenpaging.c |    2 ++
 1 file changed, 2 insertions(+)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -634,8 +634,10 @@ int main(int argc, char *argv[])
             else
             {
                 DPRINTF("page already populated (domain = %d; vcpu = %d;"
+                        " p2mt = %x;"
                         " gfn = %"PRIx64"; paused = %"PRId64")\n",
                         paging->mem_event.domain_id, req.vcpu_id,
+                        req.p2mt,
                         req.gfn, req.flags & MEM_EVENT_FLAG_VCPU_PAUSED);
 
                 /* Tell Xen to resume the vcpu */

[PATCH 08/21] xenpaging: notify policy only on resume

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.notify_policy_only_once.patch --]
[-- Type: text/plain, Size: 2142 bytes --]

If a page is requested more than once, the policy is also notified more
than once about the page-in. However, a page-in happens only once. Any
further resume will only unpause the other vcpu. The multiple notify
will put the page into the mru list multiple times and it will unlock
other already resumed pages too early. In the worst case, a page that
was just resumed can be evicted right away, causing a deadlock in the
guest.  

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/xenpaging.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -385,7 +385,7 @@ int xenpaging_evict_page(xc_interface *x
     return ret;
 }
 
-static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp)
+static int xenpaging_resume_page(xenpaging_t *paging, mem_event_response_t *rsp, int notify_policy)
 {
     int ret;
 
@@ -395,7 +395,8 @@ static int xenpaging_resume_page(xenpagi
         goto out;
 
     /* Notify policy of page being paged in */
-    policy_notify_paged_in(paging->mem_event.domain_id, rsp->gfn);
+    if ( notify_policy )
+        policy_notify_paged_in(paging->mem_event.domain_id, rsp->gfn);
 
     /* Tell Xen page is ready */
     ret = xc_mem_paging_resume(paging->xc_handle, paging->mem_event.domain_id,
@@ -621,7 +622,7 @@ int main(int argc, char *argv[])
                 rsp.vcpu_id = req.vcpu_id;
                 rsp.flags = req.flags;
 
-                rc = xenpaging_resume_page(paging, &rsp);
+                rc = xenpaging_resume_page(paging, &rsp, 1);
                 if ( rc != 0 )
                 {
                     ERROR("Error resuming page");
@@ -650,7 +651,7 @@ int main(int argc, char *argv[])
                     rsp.vcpu_id = req.vcpu_id;
                     rsp.flags = req.flags;
 
-                    rc = xenpaging_resume_page(paging, &rsp);
+                    rc = xenpaging_resume_page(paging, &rsp, 0);
                     if ( rc != 0 )
                     {
                         ERROR("Error resuming");

[PATCH 09/21] xenpaging: allow negative num_pages and limit num_pages

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.num_pages_equal_max_pages.patch --]
[-- Type: text/plain, Size: 1186 bytes --]

Simplify paging size argument. If a negative number is specified, it
means the entire guest memory should be paged out. This is useful for
debugging. Also limit num_pages to the guests max_pages.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/xenpaging.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -521,8 +521,6 @@ int main(int argc, char *argv[])
     domain_id = atoi(argv[1]);
     num_pages = atoi(argv[2]);
 
-    victims = calloc(num_pages, sizeof(xenpaging_victim_t));
-
     /* Seed random-number generator */
     srand(time(NULL));
 
@@ -543,6 +541,13 @@ int main(int argc, char *argv[])
         return 2;
     }
 
+    if ( num_pages < 0 || num_pages > paging->domain_info->max_pages )
+    {
+        num_pages = paging->domain_info->max_pages;
+        DPRINTF("setting num_pages to %d\n", num_pages);
+    }
+    victims = calloc(num_pages, sizeof(xenpaging_victim_t));
+
     /* ensure that if we get a signal, we'll do cleanup, then exit */
     act.sa_handler = close_handler;
     act.sa_flags = 0;

[PATCH 10/21] xenpaging: when populating a page, check if populating is already in progress

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.p2m_mem_paging_populate_if_p2m_ram_paged.patch --]
[-- Type: text/plain, Size: 1149 bytes --]

p2m_mem_paging_populate can be called serveral times from different
vcpus. If the page is already in state p2m_ram_paging_in and has a new
valid mfn, invalidating this new mfn will cause trouble later if
p2m_mem_paging_resume will set the new gfn/mfn pair back to state
p2m_ram_rw.
Detect this situation and keep p2m state if the page is in the process
of being still paged-out or already paged-in.
In fact, p2m state p2m_ram_paged is the only state where the mfn type
can be invalidated.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 xen/arch/x86/mm/p2m.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/p2m.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/p2m.c
@@ -2767,7 +2767,7 @@ void p2m_mem_paging_populate(struct p2m_
     /* XXX: It seems inefficient to have this here, as it's only needed
      *      in one case (ept guest accessing paging out page) */
     gfn_to_mfn(p2m, gfn, &p2mt);
-    if ( p2mt != p2m_ram_paging_out )
+    if ( p2mt == p2m_ram_paged )
     {
         p2m_lock(p2m);
         set_p2m_entry(p2m, gfn, _mfn(PAGING_MFN), 0, p2m_ram_paging_in_start);

[PATCH 11/21] xenpaging: optimize p2m_mem_paging_populate

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.optimize_p2m_mem_paging_populate.patch --]
[-- Type: text/plain, Size: 1490 bytes --]

p2m_mem_paging_populate will always put another request in the ring.  To
reduce pressure on the ring, place only required requests in the ring.
If the gfn was already processed by another thread, and the current vcpu
does not need to be paused, p2m_mem_paging_resume will do nothing with
the request.  And also xenpaging will drop the request if the vcpu does
not need a wakeup.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 xen/arch/x86/mm/p2m.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/p2m.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/p2m.c
@@ -2757,12 +2757,12 @@ void p2m_mem_paging_populate(struct p2m_
     p2m_type_t p2mt;
     struct domain *d = p2m->domain;
 
-    memset(&req, 0, sizeof(req));
-
     /* Check that there's space on the ring for this request */
     if ( mem_event_check_ring(d) )
         return;
 
+    memset(&req, 0, sizeof(req));
+
     /* Fix p2m mapping */
     /* XXX: It seems inefficient to have this here, as it's only needed
      *      in one case (ept guest accessing paging out page) */
@@ -2781,6 +2781,11 @@ void p2m_mem_paging_populate(struct p2m_
         vcpu_pause_nosync(v);
         req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
     }
+    else if ( p2mt != p2m_ram_paging_out && p2mt != p2m_ram_paged )
+    {
+        /* gfn is already on its way back and vcpu is not paused */
+        return;
+    }
 
     /* Send request to pager */
     req.gfn = gfn;

[PATCH 12/21] xenpaging: print xenpaging cmdline options

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.print-arguments.patch --]
[-- Type: text/plain, Size: 638 bytes --]

Print xenpaging arguments to simplify domain_id mapping from xenpaging
logfile to other logfiles and Xen console output.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/xenpaging.c |    2 ++
 1 file changed, 2 insertions(+)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -532,6 +532,8 @@ int main(int argc, char *argv[])
         return 1;
     }
 
+    DPRINTF("starting %s %u %d\n", argv[0], domain_id, num_pages);
+
     /* Open file */
     sprintf(filename, "page_cache_%d", domain_id);
     fd = open(filename, open_flags, open_mode);

[PATCH 13/21] xenpaging: handle temporary out-of-memory conditions during page-in

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.paging_prep_enomem.patch --]
[-- Type: text/plain, Size: 2436 bytes --]

p2m_mem_paging_prep() should return -ENOMEM if a new page could not be
allocated. This can be handled in xenpaging to retry the page-in. Right
now such condition would stall the guest because the requested page will
not come back, xenpaging simply exits. So xenpaging could very well
retry the allocation forever to rescue the guest.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/xenpaging.c |   27 ++++++++++++++++++++-------
 xen/arch/x86/mm/p2m.c       |    2 +-
 2 files changed, 21 insertions(+), 8 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -26,6 +26,7 @@
 #include <stdarg.h>
 #include <time.h>
 #include <signal.h>
+#include <unistd.h>
 #include <xc_private.h>
 
 #include <xen/mem_event.h>
@@ -415,19 +416,31 @@ static int xenpaging_populate_page(
     unsigned long _gfn;
     void *page;
     int ret;
+    unsigned char oom = 0;
 
-    /* Tell Xen to allocate a page for the domain */
-    ret = xc_mem_paging_prep(paging->xc_handle, paging->mem_event.domain_id,
-                             *gfn);
-    if ( ret != 0 )
+    _gfn = *gfn;
+    do
     {
-        ERROR("Error preparing for page in");
-        goto out_map;
+        /* Tell Xen to allocate a page for the domain */
+        ret = xc_mem_paging_prep(paging->xc_handle, paging->mem_event.domain_id,
+                                 _gfn);
+        if ( ret != 0 )
+        {
+            if ( errno == ENOMEM )
+            {
+                if ( oom++ == 0 )
+                    DPRINTF("ENOMEM while preparing gfn %lx\n", _gfn);
+                sleep(1);
+                continue;
+            }
+            ERROR("Error preparing for page in");
+            goto out_map;
+        }
     }
+    while ( ret && !interrupted );
 
     /* Map page */
     ret = -EFAULT;
-    _gfn = *gfn;
     page = xc_map_foreign_pages(paging->xc_handle, paging->mem_event.domain_id,
                                 PROT_READ | PROT_WRITE, &_gfn, 1);
     *gfn = _gfn;
--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/p2m.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/p2m.c
@@ -2802,7 +2802,7 @@ int p2m_mem_paging_prep(struct p2m_domai
     /* Get a free page */
     page = alloc_domheap_page(p2m->domain, 0);
     if ( unlikely(page == NULL) )
-        return -EINVAL;
+        return -ENOMEM;
 
     /* Fix p2m mapping */
     p2m_lock(p2m);

[PATCH 14/21] xenpaging: increase recently used pages from 4MB to 64MB

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.MRU_SIZE.patch --]
[-- Type: text/plain, Size: 575 bytes --]

Increase recently used pages from 4MB to 64MB.
Keeping more pages in memory allows the guest to make more progress if the
paging file spans the entire guest memory.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/policy_default.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/policy_default.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/policy_default.c
@@ -26,7 +26,7 @@
 #include "policy.h"
 
 
-#define MRU_SIZE 1024
+#define MRU_SIZE (1024 * 16)
 
 
 static unsigned long mru[MRU_SIZE];

[PATCH 15/21] xenpaging: update machine_to_phys_mapping during page-in and page deallocation

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.machine_to_phys_mapping.patch --]
[-- Type: text/plain, Size: 3095 bytes --]

The machine_to_phys_mapping array needs updating during page-in, and
during page deallocation.  If a page is gone, a call to
get_gpfn_from_mfn will still return the old gfn for an already paged-out
page.  This happens when the entire guest ram is paged-out before
xen_vga_populate_vram() runs.  Then XENMEM_populate_physmap is called
with gfn 0xff000.  A new page is allocated with alloc_domheap_pages.
This new page does not have a gfn yet.  However, in
guest_physmap_add_entry() the passed mfn maps still to an old gfn
(perhaps from another old guest).  This old gfn is in paged-out state in
this guests context and has no mfn anymore.  As a result, the ASSERT()
triggers because p2m_is_ram() is true for p2m_ram_paging* types.

If the machine_to_phys_mapping array is updated properly, both loops in
guest_physmap_add_entry() turn into no-ops for the new page and the
mfn/gfn mapping will be done at the end of the function.

The same thing needs to happen dring a page-in, the current gfn must be
written for the page.

If XENMEM_add_to_physmap is used with XENMAPSPACE_gmfn,
get_gpfn_from_mfn() will return an appearently valid gfn.  As a result,
guest_physmap_remove_page() is called.  The ASSERT in p2m_remove_page
triggers because the passed mfn does not match the old mfn for the
passed gfn.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
v3:
  invalidate machine_to_phys_mapping[] during page deallocation
v2:
  call set_gpfn_from_mfn only if mfn is valid
---
 xen/arch/x86/mm/p2m.c   |   15 +++++++++++----
 xen/common/page_alloc.c |    9 +++++++++
 2 files changed, 20 insertions(+), 4 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/p2m.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/p2m.c
@@ -2825,10 +2825,17 @@ void p2m_mem_paging_resume(struct p2m_do
 
     /* Fix p2m entry */
     mfn = gfn_to_mfn(p2m, rsp.gfn, &p2mt);
-    p2m_lock(p2m);
-    set_p2m_entry(p2m, rsp.gfn, mfn, 0, p2m_ram_rw);
-    audit_p2m(p2m, 1);
-    p2m_unlock(p2m);
+    if ( mfn_valid(mfn) )
+    {
+        p2m_lock(p2m);
+        set_p2m_entry(p2m, rsp.gfn, mfn, 0, p2m_ram_rw);
+        set_gpfn_from_mfn(mfn_x(mfn), rsp.gfn);
+        audit_p2m(p2m, 1);
+        p2m_unlock(p2m);
+    } else {
+        gdprintk(XENLOG_ERR, "invalid mfn %lx for gfn %lx p2mt %x flags %lx\n",
+            mfn_x(mfn), rsp.gfn, p2mt, (unsigned long)rsp.flags);
+    }
 
     /* Unpause domain */
     if ( rsp.flags & MEM_EVENT_FLAG_VCPU_PAUSED )
--- xen-unstable.hg-4.1.22433.orig/xen/common/page_alloc.c
+++ xen-unstable.hg-4.1.22433/xen/common/page_alloc.c
@@ -1199,9 +1199,18 @@ void free_domheap_pages(struct page_info
 {
     int            i, drop_dom_ref;
     struct domain *d = page_get_owner(pg);
+    unsigned long mfn;
 
     ASSERT(!in_irq());
 
+    /* this page is not a gfn anymore */
+    mfn = page_to_mfn(pg);
+    if ( mfn_valid(mfn) )
+    {
+        for ( i = 0; i < (1 << order); i++ )
+            set_gpfn_from_mfn(mfn + i, INVALID_M2P_ENTRY);
+    }
+
     if ( unlikely(is_xen_heap_page(pg)) )
     {
         /* NB. May recursively lock from relinquish_memory(). */

[PATCH 16/21] xenpaging: drop paged pages in guest_remove_page

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.guest_remove_page.patch --]
[-- Type: text/plain, Size: 6529 bytes --]

Simply drop paged-pages in guest_remove_page(), and notify xenpaging to
drop its reference to the gfn.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
v2:
 resume dropped page to unpause vcpus

 tools/xenpaging/xenpaging.c    |   17 +++++++---
 xen/arch/x86/mm/p2m.c          |   68 +++++++++++++++++++++++++++++++----------
 xen/common/memory.c            |    6 +++
 xen/include/asm-x86/p2m.h      |    4 ++
 xen/include/public/mem_event.h |    1 
 5 files changed, 75 insertions(+), 21 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/xenpaging.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/xenpaging.c
@@ -628,12 +628,19 @@ int main(int argc, char *argv[])
                     goto out;
                 }
                 
-                /* Populate the page */
-                rc = xenpaging_populate_page(xch, paging, &req.gfn, fd, i);
-                if ( rc != 0 )
+                if ( req.flags & MEM_EVENT_FLAG_DROP_PAGE )
                 {
-                    ERROR("Error populating page");
-                    goto out;
+                    DPRINTF("Dropping page %"PRIx64" p2mt %x\n", req.gfn, req.p2mt);
+                }
+                else
+                {
+                    /* Populate the page */
+                    rc = xenpaging_populate_page(xch, paging, &req.gfn, fd, i);
+                    if ( rc != 0 )
+                    {
+                        ERROR("Error populating page");
+                        goto out;
+                    }
                 }
 
                 /* Prepare the response */
--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/mm/p2m.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/mm/p2m.c
@@ -2194,12 +2194,15 @@ p2m_remove_page(struct p2m_domain *p2m,
 
     P2M_DEBUG("removing gfn=%#lx mfn=%#lx\n", gfn, mfn);
 
-    for ( i = 0; i < (1UL << page_order); i++ )
+    if ( mfn_valid(_mfn(mfn)) )
     {
-        mfn_return = p2m->get_entry(p2m, gfn + i, &t, p2m_query);
-        if ( !p2m_is_grant(t) )
-            set_gpfn_from_mfn(mfn+i, INVALID_M2P_ENTRY);
-        ASSERT( !p2m_is_valid(t) || mfn + i == mfn_x(mfn_return) );
+        for ( i = 0; i < (1UL << page_order); i++ )
+        {
+            mfn_return = p2m->get_entry(p2m, gfn + i, &t, p2m_query);
+            if ( !p2m_is_grant(t) )
+                set_gpfn_from_mfn(mfn+i, INVALID_M2P_ENTRY);
+            ASSERT( !p2m_is_valid(t) || mfn + i == mfn_x(mfn_return) );
+        }
     }
     set_p2m_entry(p2m, gfn, _mfn(INVALID_MFN), page_order, p2m_invalid);
 }
@@ -2750,6 +2753,36 @@ int p2m_mem_paging_evict(struct p2m_doma
     return 0;
 }
 
+void p2m_mem_paging_drop_page(struct p2m_domain *p2m, unsigned long gfn)
+{
+    struct vcpu *v = current;
+    mem_event_request_t req;
+    p2m_type_t p2mt;
+    struct domain *d = p2m->domain;
+
+    memset(&req, 0, sizeof(req));
+
+    /* Check that there's space on the ring for this request */
+    if ( mem_event_check_ring(d) )
+        return;
+
+    gfn_to_mfn(p2m, gfn, &p2mt);
+    /* Pause domain */
+    if ( v->domain->domain_id == d->domain_id )
+    {
+        vcpu_pause_nosync(v);
+        req.flags |= MEM_EVENT_FLAG_VCPU_PAUSED;
+    }
+
+    /* Send request to pager */
+    req.flags |= MEM_EVENT_FLAG_DROP_PAGE;
+    req.gfn = gfn;
+    req.p2mt = p2mt;
+    req.vcpu_id = v->vcpu_id;
+
+    mem_event_put_request(d, &req);
+}
+
 void p2m_mem_paging_populate(struct p2m_domain *p2m, unsigned long gfn)
 {
     struct vcpu *v = current;
@@ -2823,18 +2856,21 @@ void p2m_mem_paging_resume(struct p2m_do
     /* Pull the response off the ring */
     mem_event_get_response(d, &rsp);
 
-    /* Fix p2m entry */
-    mfn = gfn_to_mfn(p2m, rsp.gfn, &p2mt);
-    if ( mfn_valid(mfn) )
+    if ( !( rsp.flags & MEM_EVENT_FLAG_DROP_PAGE ) )
     {
-        p2m_lock(p2m);
-        set_p2m_entry(p2m, rsp.gfn, mfn, 0, p2m_ram_rw);
-        set_gpfn_from_mfn(mfn_x(mfn), rsp.gfn);
-        audit_p2m(p2m, 1);
-        p2m_unlock(p2m);
-    } else {
-        gdprintk(XENLOG_ERR, "invalid mfn %lx for gfn %lx p2mt %x flags %lx\n",
-            mfn_x(mfn), rsp.gfn, p2mt, (unsigned long)rsp.flags);
+        /* Fix p2m entry */
+        mfn = gfn_to_mfn(p2m, rsp.gfn, &p2mt);
+        if ( mfn_valid(mfn) )
+        {
+            p2m_lock(p2m);
+            set_p2m_entry(p2m, rsp.gfn, mfn, 0, p2m_ram_rw);
+            set_gpfn_from_mfn(mfn_x(mfn), rsp.gfn);
+            audit_p2m(p2m, 1);
+            p2m_unlock(p2m);
+        } else {
+            gdprintk(XENLOG_ERR, "invalid mfn %lx for gfn %lx p2mt %x flags %lx\n",
+                mfn_x(mfn), rsp.gfn, p2mt, (unsigned long)rsp.flags);
+        }
     }
 
     /* Unpause domain */
--- xen-unstable.hg-4.1.22433.orig/xen/common/memory.c
+++ xen-unstable.hg-4.1.22433/xen/common/memory.c
@@ -162,6 +162,12 @@ int guest_remove_page(struct domain *d,
 
 #ifdef CONFIG_X86
     mfn = mfn_x(gfn_to_mfn(p2m_get_hostp2m(d), gmfn, &p2mt)); 
+    if ( unlikely(p2m_is_paging(p2mt)) )
+    {
+        guest_physmap_remove_page(d, gmfn, mfn, 0);
+        p2m_mem_paging_drop_page(p2m_get_hostp2m(d), gmfn);
+        return 1;
+    }
 #else
     mfn = gmfn_to_mfn(d, gmfn);
 #endif
--- xen-unstable.hg-4.1.22433.orig/xen/include/asm-x86/p2m.h
+++ xen-unstable.hg-4.1.22433/xen/include/asm-x86/p2m.h
@@ -471,6 +471,8 @@ int set_shared_p2m_entry(struct p2m_doma
 int p2m_mem_paging_nominate(struct p2m_domain *p2m, unsigned long gfn);
 /* Evict a frame */
 int p2m_mem_paging_evict(struct p2m_domain *p2m, unsigned long gfn);
+/* Tell xenpaging to drop a paged out frame */
+void p2m_mem_paging_drop_page(struct p2m_domain *p2m, unsigned long gfn);
 /* Start populating a paged out frame */
 void p2m_mem_paging_populate(struct p2m_domain *p2m, unsigned long gfn);
 /* Prepare the p2m for paging a frame in */
@@ -478,6 +480,8 @@ int p2m_mem_paging_prep(struct p2m_domai
 /* Resume normal operation (in case a domain was paused) */
 void p2m_mem_paging_resume(struct p2m_domain *p2m);
 #else
+static inline void p2m_mem_paging_drop_page(struct p2m_domain *p2m, unsigned long gfn)
+{ }
 static inline void p2m_mem_paging_populate(struct p2m_domain *p2m, unsigned long gfn)
 { }
 #endif
--- xen-unstable.hg-4.1.22433.orig/xen/include/public/mem_event.h
+++ xen-unstable.hg-4.1.22433/xen/include/public/mem_event.h
@@ -37,6 +37,7 @@
 #define MEM_EVENT_FLAG_VCPU_PAUSED  (1 << 0)
 #define MEM_EVENT_FLAG_DOM_PAUSED   (1 << 1)
 #define MEM_EVENT_FLAG_OUT_OF_MEM   (1 << 2)
+#define MEM_EVENT_FLAG_DROP_PAGE    (1 << 3)
 
 
 typedef struct mem_event_shared_page {

[PATCH 17/21] xenpaging: handle HVMCOPY_gfn_paged_out in copy_from/to_user

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.HVMCOPY_gfn_paged_out.patch --]
[-- Type: text/plain, Size: 5607 bytes --]

This is a quick fix to keep my guests running. 
The proper fix will use the recently added waitqueue API.


copy_from_user_hvm can fail when __hvm_copy returns
HVMCOPY_gfn_paged_out for a referenced gfn, for example during guests
pagetable walk.  This has to be handled in some way.  One hypercall that
failed was do_memory_op/XENMEM_decrease_reservation which lead to a
BUG_ON balloon.c.  Since do_memory_op already has restart support for
the hypercall, copy_from_guest uses this existing retry code.  In
addition, cleanup on error was added to increase_reservation and
populate_physmap.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 xen/arch/x86/hvm/hvm.c |    4 ++++
 xen/common/memory.c    |   44 +++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 43 insertions(+), 5 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/xen/arch/x86/hvm/hvm.c
+++ xen-unstable.hg-4.1.22433/xen/arch/x86/hvm/hvm.c
@@ -2066,6 +2066,8 @@ unsigned long copy_to_user_hvm(void *to,
 
     rc = hvm_copy_to_guest_virt_nofault((unsigned long)to, (void *)from,
                                         len, 0);
+    if ( rc == HVMCOPY_gfn_paged_out )
+        return -EAGAIN;
     return rc ? len : 0; /* fake a copy_to_user() return code */
 }
 
@@ -2083,6 +2085,8 @@ unsigned long copy_from_user_hvm(void *t
 #endif
 
     rc = hvm_copy_from_guest_virt_nofault(to, (unsigned long)from, len, 0);
+    if ( rc == HVMCOPY_gfn_paged_out )
+        return -EAGAIN;
     return rc ? len : 0; /* fake a copy_from_user() return code */
 }
 
--- xen-unstable.hg-4.1.22433.orig/xen/common/memory.c
+++ xen-unstable.hg-4.1.22433/xen/common/memory.c
@@ -47,6 +47,7 @@ static void increase_reservation(struct
 {
     struct page_info *page;
     unsigned long i;
+    unsigned long ctg_ret;
     xen_pfn_t mfn;
     struct domain *d = a->domain;
 
@@ -80,8 +81,14 @@ static void increase_reservation(struct
         if ( !guest_handle_is_null(a->extent_list) )
         {
             mfn = page_to_mfn(page);
-            if ( unlikely(__copy_to_guest_offset(a->extent_list, i, &mfn, 1)) )
+            ctg_ret = __copy_to_guest_offset(a->extent_list, i, &mfn, 1);
+            if ( unlikely(ctg_ret) )
+            {
+                free_domheap_pages(page, a->extent_order);
+                if ( (long)ctg_ret == -EAGAIN )
+                    a->preempted = 1;
                 goto out;
+            }
         }
     }
 
@@ -93,6 +100,7 @@ static void populate_physmap(struct memo
 {
     struct page_info *page;
     unsigned long i, j;
+    unsigned long cftg_ret;
     xen_pfn_t gpfn, mfn;
     struct domain *d = a->domain;
 
@@ -111,8 +119,13 @@ static void populate_physmap(struct memo
             goto out;
         }
 
-        if ( unlikely(__copy_from_guest_offset(&gpfn, a->extent_list, i, 1)) )
+        cftg_ret = __copy_from_guest_offset(&gpfn, a->extent_list, i, 1);
+        if ( unlikely(cftg_ret) )
+        {
+            if ( (long)cftg_ret == -EAGAIN )
+                a->preempted = 1;
             goto out;
+        }
 
         if ( a->memflags & MEMF_populate_on_demand )
         {
@@ -142,8 +155,17 @@ static void populate_physmap(struct memo
                     set_gpfn_from_mfn(mfn + j, gpfn + j);
 
                 /* Inform the domain of the new page's machine address. */ 
-                if ( unlikely(__copy_to_guest_offset(a->extent_list, i, &mfn, 1)) )
+                cftg_ret = __copy_to_guest_offset(a->extent_list, i, &mfn, 1);
+                if ( unlikely(cftg_ret) )
+                {
+                    for ( j = 0; j < (1 << a->extent_order); j++ )
+                        set_gpfn_from_mfn(mfn + j, INVALID_M2P_ENTRY);
+                    guest_physmap_remove_page(d, gpfn, mfn, a->extent_order);
+                    free_domheap_pages(page, a->extent_order);
+                    if ( (long)cftg_ret == -EAGAIN )
+                        a->preempted = 1;
                     goto out;
+                }
             }
         }
     }
@@ -212,6 +234,7 @@ int guest_remove_page(struct domain *d,
 static void decrease_reservation(struct memop_args *a)
 {
     unsigned long i, j;
+    unsigned long cfg_ret;
     xen_pfn_t gmfn;
 
     if ( !guest_handle_subrange_okay(a->extent_list, a->nr_done,
@@ -226,8 +249,13 @@ static void decrease_reservation(struct
             goto out;
         }
 
-        if ( unlikely(__copy_from_guest_offset(&gmfn, a->extent_list, i, 1)) )
+        cfg_ret = __copy_from_guest_offset(&gmfn, a->extent_list, i, 1);
+        if ( unlikely(cfg_ret) )
+        {
+            if ( (long)cfg_ret == -EAGAIN )
+                a->preempted = 1;
             goto out;
+        }
 
         if ( tb_init_done )
         {
@@ -508,6 +536,7 @@ long do_memory_op(unsigned long cmd, XEN
     int rc, op;
     unsigned int address_bits;
     unsigned long start_extent;
+    unsigned long cfg_ret;
     struct xen_memory_reservation reservation;
     struct memop_args args;
     domid_t domid;
@@ -521,8 +550,13 @@ long do_memory_op(unsigned long cmd, XEN
     case XENMEM_populate_physmap:
         start_extent = cmd >> MEMOP_EXTENT_SHIFT;
 
-        if ( copy_from_guest(&reservation, arg, 1) )
+        cfg_ret = copy_from_guest(&reservation, arg, 1);
+        if ( unlikely(cfg_ret) )
+        {
+            if ( (long)cfg_ret == -EAGAIN )
+                return hypercall_create_continuation(__HYPERVISOR_memory_op, "lh", cmd, arg);
             return start_extent;
+        }
 
         /* Is size too large for us to encode a continuation? */
         if ( reservation.nr_extents > (ULONG_MAX >> MEMOP_EXTENT_SHIFT) )

[PATCH 18/21] xenpaging: prevent page-out of first 16MB

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.blacklist.patch --]
[-- Type: text/plain, Size: 814 bytes --]

This is more a workaround than a bugfix:
Don't page out first 16MB of memory.
When the BIOS does its initialization process and xenpaging removes pages,
crashes will occour due to lack of support of xenpaging.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/policy_default.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--- xen-unstable.hg-4.1.22433.orig/tools/xenpaging/policy_default.c
+++ xen-unstable.hg-4.1.22433/tools/xenpaging/policy_default.c
@@ -60,8 +60,9 @@ int policy_init(xenpaging_t *paging)
     for ( i = 0; i < MRU_SIZE; i++ )
         mru[i] = INVALID_MFN;
 
-    /* Don't page out page 0 */
-    set_bit(0, bitmap);
+    /* Don't page out first 16MB */
+    for ( i = 0; i < ((16*1024*1024)/4096); i++ )
+        set_bit(i, bitmap);
 
  out:
     return rc;

Re: [PATCH 18/21] xenpaging: prevent page-out of first 16MB

[Olaf Hering]

On Fri, Nov 26, Olaf Hering wrote:

> This is more a workaround than a bugfix:
> Don't page out first 16MB of memory.
> When the BIOS does its initialization process and xenpaging removes pages,
> crashes will occour due to lack of support of xenpaging.

While looking at this again, I came up with this change. Any idea whats
at 512K during BIOS startup?


 Subject: xenpaging: prevent page-out of gfn 0x80

Add a workaround for missing handling of paged-out pages during BIOS startup.
For some reason, only gfn 0x80 is affected.

(XEN) HVM3: HVM Loader
(XEN) traps.c:649:d3 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000
(XEN) HVM3: Detected Xen v4.1
(XEN) HVM3: HVMLoader bug at util.c:604
(XEN) hvm.c:1085:d3 Triple fault on VCPU0 - invoking HVM system reset.

(XEN) HVM4: HVM Loader
(XEN) traps.c:649:d4 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000
(XEN) io.c:194:d4 MMIO emulation failed @ 0018:9ffff: 00 6a 10 80 c4 82
(XEN) hvm.c:1085:d4 Triple fault on VCPU0 - invoking HVM system reset.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/xenpaging/policy_default.c |    3 +++
 1 file changed, 3 insertions(+)

--- xen-unstable.hg-4.1.22548.orig/tools/xenpaging/policy_default.c
+++ xen-unstable.hg-4.1.22548/tools/xenpaging/policy_default.c
@@ -76,6 +76,9 @@ int policy_init(xenpaging_t *paging)
     /* Don't page out page 0 */
     set_bit(0, bitmap);
 
+    /* this is the trap page, disabled for the time being */
+    set_bit(0x80, bitmap);
+
  out:
     return rc;
 }

Re: [PATCH 18/21] xenpaging: prevent page-out of first 16MB

[Jan Beulich]

>>> On 16.12.10 at 17:59, Olaf Hering <olaf@aepfle.de> wrote:
> On Fri, Nov 26, Olaf Hering wrote:
> 
>> This is more a workaround than a bugfix:
>> Don't page out first 16MB of memory.
>> When the BIOS does its initialization process and xenpaging removes pages,
>> crashes will occour due to lack of support of xenpaging.
> 
> While looking at this again, I came up with this change. Any idea whats
> at 512K during BIOS startup?
> 
> 
>  Subject: xenpaging: prevent page-out of gfn 0x80
> 
> Add a workaround for missing handling of paged-out pages during BIOS 
> startup.
> For some reason, only gfn 0x80 is affected.
> 
> (XEN) HVM3: HVM Loader
> (XEN) traps.c:649:d3 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000

This clearly is another place where page-in needs to be triggered.
Isn't it the case that *any* gmfn_to_mfn() needs this special
treatment? Couldn't therefore the translation function itself deal
with requesting the page-in (and waiting for its completion, now
that this can be done)?

Jan

Re: [PATCH 18/21] xenpaging: prevent page-out of first 16MB

[Olaf Hering]

On Fri, Dec 17, Jan Beulich wrote:

> >>> On 16.12.10 at 17:59, Olaf Hering <olaf@aepfle.de> wrote:
> > On Fri, Nov 26, Olaf Hering wrote:
> > 
> >> This is more a workaround than a bugfix:
> >> Don't page out first 16MB of memory.
> >> When the BIOS does its initialization process and xenpaging removes pages,
> >> crashes will occour due to lack of support of xenpaging.
> > 
> > While looking at this again, I came up with this change. Any idea whats
> > at 512K during BIOS startup?
> > 
> > 
> >  Subject: xenpaging: prevent page-out of gfn 0x80
> > 
> > Add a workaround for missing handling of paged-out pages during BIOS 
> > startup.
> > For some reason, only gfn 0x80 is affected.
> > 
> > (XEN) HVM3: HVM Loader
> > (XEN) traps.c:649:d3 Bad GMFN 80 (MFN ffffffffff) to MSR 40000000
> 
> This clearly is another place where page-in needs to be triggered.

Yes, thats true.
I had a printk in gfn_to_mfn for some months, and it did not trigger
during my testing. But then, I also had the patch which keeps the first
16MB in memory.

So as it stands, gfn_to_mfn() should call p2m_mem_paging_populate() at
least, even if that does not fix this crash.
I think the few gfn_to_mfn* variants should not return the paging types
anymore, and just sleep until the page is back. Thats something for next
year, as time runs out for me.

And: I think the various redefines of those functions/macros can go as well.


Olaf

[PATCH 19/21] xenpaging: start xenpaging via config option

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.autostart.patch --]
[-- Type: text/plain, Size: 9650 bytes --]

Start xenpaging via config option.

TODO: add config option for different pagefile directory
TODO: add libxl support
TODO: parse config values like 42K, 42M, 42G, 42%

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
v2:
  unlink logfile instead of truncating it.
  allows hardlinking for further inspection

 tools/examples/xmexample.hvm            |    3 +
 tools/python/README.XendConfig          |    1 
 tools/python/README.sxpcfg              |    1 
 tools/python/xen/xend/XendConfig.py     |    3 +
 tools/python/xen/xend/XendDomainInfo.py |    6 ++
 tools/python/xen/xend/image.py          |   91 ++++++++++++++++++++++++++++++++
 tools/python/xen/xm/create.py           |    5 +
 tools/python/xen/xm/xenapi_create.py    |    1 
 8 files changed, 111 insertions(+)

--- xen-unstable.hg-4.1.22433.orig/tools/examples/xmexample.hvm
+++ xen-unstable.hg-4.1.22433/tools/examples/xmexample.hvm
@@ -127,6 +127,9 @@ disk = [ 'file:/var/images/min-el3-i386.
 # Device Model to be used
 device_model = 'qemu-dm'
 
+# xenpaging, number of pages
+xenpaging = 42
+
 #-----------------------------------------------------------------------------
 # boot on floppy (a), hard disk (c), Network (n) or CD-ROM (d) 
 # default: hard disk, cd-rom, floppy
--- xen-unstable.hg-4.1.22433.orig/tools/python/README.XendConfig
+++ xen-unstable.hg-4.1.22433/tools/python/README.XendConfig
@@ -120,6 +120,7 @@ otherConfig
                                 image.vncdisplay
                                 image.vncunused
                                 image.hvm.device_model
+                                image.hvm.xenpaging
                                 image.hvm.display
                                 image.hvm.xauthority
                                 image.hvm.vncconsole
--- xen-unstable.hg-4.1.22433.orig/tools/python/README.sxpcfg
+++ xen-unstable.hg-4.1.22433/tools/python/README.sxpcfg
@@ -51,6 +51,7 @@ image
   - vncunused
   (HVM)
   - device_model
+  - xenpaging
   - display
   - xauthority
   - vncconsole
--- xen-unstable.hg-4.1.22433.orig/tools/python/xen/xend/XendConfig.py
+++ xen-unstable.hg-4.1.22433/tools/python/xen/xend/XendConfig.py
@@ -147,6 +147,7 @@ XENAPI_PLATFORM_CFG_TYPES = {
     'apic': int,
     'boot': str,
     'device_model': str,
+    'xenpaging': int,
     'loader': str,
     'display' : str,
     'fda': str,
@@ -508,6 +509,8 @@ class XendConfig(dict):
             self['platform']['nomigrate'] = 0
 
         if self.is_hvm():
+            if 'xenpaging' not in self['platform']:
+                self['platform']['xenpaging'] = None
             if 'timer_mode' not in self['platform']:
                 self['platform']['timer_mode'] = 1
             if 'viridian' not in self['platform']:
--- xen-unstable.hg-4.1.22433.orig/tools/python/xen/xend/XendDomainInfo.py
+++ xen-unstable.hg-4.1.22433/tools/python/xen/xend/XendDomainInfo.py
@@ -2390,6 +2390,7 @@ class XendDomainInfo:
 
         if self.image:
             self.image.createDeviceModel()
+            self.image.createXenPaging()
 
         #if have pass-through devs, need the virtual pci slots info from qemu
         self.pci_device_configure_boot()
@@ -2402,6 +2403,11 @@ class XendDomainInfo:
                 self.image.destroyDeviceModel()
             except Exception, e:
                 log.exception("Device model destroy failed %s" % str(e))
+            try:
+                log.debug("stopping xenpaging")
+                self.image.destroyXenPaging()
+            except Exception, e:
+                log.exception("stopping xenpaging failed %s" % str(e))
         else:
             log.debug("No device model")
 
--- xen-unstable.hg-4.1.22433.orig/tools/python/xen/xend/image.py
+++ xen-unstable.hg-4.1.22433/tools/python/xen/xend/image.py
@@ -122,12 +122,14 @@ class ImageHandler:
         self.vm.permissionsVm("image/cmdline", { 'dom': self.vm.getDomid(), 'read': True } )
 
         self.device_model = vmConfig['platform'].get('device_model')
+        self.xenpaging = vmConfig['platform'].get('xenpaging')
 
         self.display = vmConfig['platform'].get('display')
         self.xauthority = vmConfig['platform'].get('xauthority')
         self.vncconsole = int(vmConfig['platform'].get('vncconsole', 0))
         self.dmargs = self.parseDeviceModelArgs(vmConfig)
         self.pid = None
+        self.xenpaging_pid = None
         rtc_timeoffset = int(vmConfig['platform'].get('rtc_timeoffset', 0))
         if int(vmConfig['platform'].get('localtime', 0)):
             if time.localtime(time.time())[8]:
@@ -392,6 +394,95 @@ class ImageHandler:
         sentinel_fifos_inuse[sentinel_path_fifo] = 1
         self.sentinel_path_fifo = sentinel_path_fifo
 
+    def createXenPaging(self):
+        if self.xenpaging is None:
+            return
+        if self.xenpaging == 0:
+            return
+        if self.xenpaging_pid:
+            return
+        xenpaging_bin = auxbin.pathTo("xenpaging")
+        args = [xenpaging_bin]
+        args = args + ([ "%d" % self.vm.getDomid()])
+        args = args + ([ "%s" % self.xenpaging])
+        env = dict(os.environ)
+        self.xenpaging_logfile = "/var/log/xen/xenpaging-%s.log" %  str(self.vm.info['name_label'])
+        logfile_mode = os.O_WRONLY|os.O_CREAT|os.O_APPEND|os.O_TRUNC
+        null = os.open("/dev/null", os.O_RDONLY)
+        try:
+            os.unlink(self.xenpaging_logfile)
+        except:
+            pass
+        logfd = os.open(self.xenpaging_logfile, logfile_mode, 0644)
+        sys.stderr.flush()
+        contract = osdep.prefork("%s:%d" % (self.vm.getName(), self.vm.getDomid()))
+        xenpaging_pid = os.fork()
+        if xenpaging_pid == 0: #child
+            try:
+                xenpaging_dir = "/var/lib/xen/xenpaging"
+                osdep.postfork(contract)
+                os.dup2(null, 0)
+                os.dup2(logfd, 1)
+                os.dup2(logfd, 2)
+                try:
+                    os.mkdir(xenpaging_dir)
+                except:
+                    log.info("mkdir %s failed" % xenpaging_dir)
+                    pass
+                try:
+                    os.chdir(xenpaging_dir)
+                except:
+                    log.warn("chdir %s failed" % xenpaging_dir)
+                try:
+                    log.info("starting %s" % args)
+                    os.execve(xenpaging_bin, args, env)
+                except Exception, e:
+                    print >>sys.stderr, (
+                        'failed to execute xenpaging: %s: %s' %
+                        xenpaging_bin, utils.exception_string(e))
+                    os._exit(126)
+            except Exception, e:
+                log.warn("staring xenpaging in %s failed" % xenpaging_dir)
+                os._exit(127)
+        else:
+            osdep.postfork(contract, abandon=True)
+            self.xenpaging_pid = xenpaging_pid
+            os.close(null)
+            os.close(logfd)
+
+    def destroyXenPaging(self):
+        if self.xenpaging is None:
+            return
+        if self.xenpaging_pid:
+            try:
+                os.kill(self.xenpaging_pid, signal.SIGHUP)
+            except OSError, exn:
+                log.exception(exn)
+            for i in xrange(100):
+                try:
+                    (p, rv) = os.waitpid(self.xenpaging_pid, os.WNOHANG)
+                    if p == self.xenpaging_pid:
+                        break
+                except OSError:
+                    # This is expected if Xend has been restarted within
+                    # the life of this domain.  In this case, we can kill
+                    # the process, but we can't wait for it because it's
+                    # not our child. We continue this loop, and after it is
+                    # terminated make really sure the process is going away
+                    # (SIGKILL).
+                    pass
+                time.sleep(0.1)
+            else:
+                log.warning("xenpaging %d took more than 10s "
+                            "to terminate: sending SIGKILL" % self.xenpaging_pid)
+                try:
+                    os.kill(self.xenpaging_pid, signal.SIGKILL)
+                    os.waitpid(self.xenpaging_pid, 0)
+                except OSError:
+                    # This happens if the process doesn't exist.
+                    pass
+        self.xenpaging_pid = None
+
     def createDeviceModel(self, restore = False):
         if self.device_model is None:
             return
--- xen-unstable.hg-4.1.22433.orig/tools/python/xen/xm/create.py
+++ xen-unstable.hg-4.1.22433/tools/python/xen/xm/create.py
@@ -491,6 +491,10 @@ gopts.var('nfs_root', val="PATH",
           fn=set_value, default=None,
           use="Set the path of the root NFS directory.")
 
+gopts.var('xenpaging', val='NUM',
+          fn=set_int, default=None,
+          use="Number of pages to swap.")
+
 gopts.var('device_model', val='FILE',
           fn=set_value, default=None,
           use="Path to device model program.")
@@ -1076,6 +1080,7 @@ def configure_hvm(config_image, vals):
     args = [ 'acpi', 'apic',
              'boot',
              'cpuid', 'cpuid_check',
+             'xenpaging',
              'device_model', 'display',
              'fda', 'fdb',
              'gfx_passthru', 'guest_os_type',
--- xen-unstable.hg-4.1.22433.orig/tools/python/xen/xm/xenapi_create.py
+++ xen-unstable.hg-4.1.22433/tools/python/xen/xm/xenapi_create.py
@@ -1085,6 +1085,7 @@ class sxp2xml:
             'acpi',
             'apic',
             'boot',
+            'xenpaging',
             'device_model',
             'loader',
             'fda',

[PATCH 20/21] xenpaging: add dynamic startup delay for xenpaging

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.autostart_delay.patch --]
[-- Type: text/plain, Size: 4588 bytes --]

This is a debug helper. Since the xenpaging support is still fragile, run
xenpaging at different stages in the bootprocess. Different delays will trigger
more bugs. This implementation starts without delay for 5 reboots, then
increments the delay by 0.1 seconds It uses xenstore for presistant storage of
delay values

TODO: find the correct place to remove the xenstore directory when the guest is shutdown or crashed

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 tools/python/xen/xend/image.py |   32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

--- xen-unstable.hg-4.1.22433.orig/tools/python/xen/xend/image.py
+++ xen-unstable.hg-4.1.22433/tools/python/xen/xend/image.py
@@ -123,6 +123,18 @@ class ImageHandler:
 
         self.device_model = vmConfig['platform'].get('device_model')
         self.xenpaging = vmConfig['platform'].get('xenpaging')
+        self.xenpaging_delay = xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay" % self.vm.info['name_label'])
+        if self.xenpaging_delay == None:
+            log.warn("XXX creating /local/domain/0/xenpaging/%s" % self.vm.info['name_label'])
+            xstransact.Mkdir("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'])
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay', '0.0'))
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_inc', '0.1'))
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_use', '5'))
+            xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_used', '0'))
+        self.xenpaging_delay = float(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay" % self.vm.info['name_label']))
+        self.xenpaging_delay_inc = float(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay_inc" % self.vm.info['name_label']))
+        self.xenpaging_delay_use = int(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay_use" % self.vm.info['name_label']))
+        self.xenpaging_delay_used = int(xstransact.Read("/local/domain/0/xenpaging/%s/xenpaging_delay_used" % self.vm.info['name_label']))
 
         self.display = vmConfig['platform'].get('display')
         self.xauthority = vmConfig['platform'].get('xauthority')
@@ -401,6 +413,17 @@ class ImageHandler:
             return
         if self.xenpaging_pid:
             return
+        if self.xenpaging_delay_used < self.xenpaging_delay_use:
+            self.xenpaging_delay_used += 1
+        else:
+            self.xenpaging_delay_used = 0
+            self.xenpaging_delay += self.xenpaging_delay_inc
+        log.info("delay_used %s" % self.xenpaging_delay_used)
+        log.info("delay_use %s" % self.xenpaging_delay_use)
+        log.info("delay %s" % self.xenpaging_delay)
+        log.info("delay_inc %s" % self.xenpaging_delay_inc)
+        xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay', self.xenpaging_delay))
+        xstransact.Store("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'], ('xenpaging_delay_used', self.xenpaging_delay_used))
         xenpaging_bin = auxbin.pathTo("xenpaging")
         args = [xenpaging_bin]
         args = args + ([ "%d" % self.vm.getDomid()])
@@ -434,6 +457,9 @@ class ImageHandler:
                 except:
                     log.warn("chdir %s failed" % xenpaging_dir)
                 try:
+                    if self.xenpaging_delay != 0.0:
+                        log.info("delaying xenpaging startup %s seconds ..." % self.xenpaging_delay)
+                        time.sleep(self.xenpaging_delay)
                     log.info("starting %s" % args)
                     os.execve(xenpaging_bin, args, env)
                 except Exception, e:
@@ -449,10 +475,16 @@ class ImageHandler:
             self.xenpaging_pid = xenpaging_pid
             os.close(null)
             os.close(logfd)
+            if self.xenpaging_delay == 0.0:
+                log.warn("waiting for xenpaging ...")
+                time.sleep(22)
+                log.warn("waiting for xenpaging done.")
 
     def destroyXenPaging(self):
         if self.xenpaging is None:
             return
+        # FIXME find correct place for guest shutdown or crash
+        #xstransact.Remove("/local/domain/0/xenpaging/%s" % self.vm.info['name_label'])
         if self.xenpaging_pid:
             try:
                 os.kill(self.xenpaging_pid, signal.SIGHUP)

[PATCH 21/21] xenpaging: (sparse) documenation

[Olaf Hering]

[-- Attachment #1: xen-unstable.xenpaging.doc.patch --]
[-- Type: text/plain, Size: 1775 bytes --]

Write up some sparse documentation about xenpaging usage.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

---
 docs/misc/xenpaging.txt |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

--- /dev/null
+++ xen-unstable.hg-4.1.22433/docs/misc/xenpaging.txt
@@ -0,0 +1,48 @@
+Warning:
+
+The xenpaging code is new and not fully debugged.
+Usage of xenpaging can crash Xen or cause severe data corruption in the
+guest memory and its filesystems!
+
+Description:
+
+xenpaging writes memory pages of a given guest to a file and moves the
+pages back to the pool of available memory.  Once the guests wants to
+access the paged-out memory, the page is read from disk and placed into
+memory.  This allows the sum of all running guests to use more memory
+than physically available on the host.
+
+Usage:
+
+Once the guest is running, run xenpaging with the guest_id and the
+number of pages to page-out:
+
+  chdir /var/lib/xen/xenpaging
+  xenpaging <guest_id>  <number_of_pages>
+
+To obtain the guest_id, run 'xm list'.
+xenpaging will write the pagefile to the current directory.
+Example with 128MB pagefile on guest 1:
+
+  xenpaging 1 32768
+
+Caution: stopping xenpaging manually will cause the guest to stall or
+crash because the paged-out memory is not written back into the guest!
+
+After a reboot of a guest, its guest_id changes, the current xenpaging
+binary has no target anymore. To automate restarting of xenpaging after
+guest reboot, specify the number if pages in the guest configuration
+file /etc/xen/vm/<guest_name>:
+
+xenpaging=32768
+
+Redo the guest with 'xm create /etc/xen/vm/<guest_name>' to activate the
+changes.
+
+
+Todo:
+- implement stopping of xenpaging
+- implement/test live migration
+
+
+# vim: tw=72

Re: [PATCH 21/21] xenpaging: (sparse) documenation

[Ian Jackson]

Olaf Hering writes ("[Xen-devel] [PATCH 21/21] xenpaging: (sparse) documenation"):
> Write up some sparse documentation about xenpaging usage.
> 
> Signed-off-by: Olaf Hering <olaf@aepfle.de>

Marvellous, thanks.  I think this should be applied right away.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ian.

Re: [PATCH 00/21] xenpaging changes for xen-unstable

[Ian Jackson]

Olaf Hering writes ("[Xen-devel] [PATCH 00/21] xenpaging changes for xen-unstable"):
> Here are some changes for xenpaging in xen-unstable.
> 
> Patches 1 to 14 are likely non-controversial and could be applied.

Having eyeballed them (but not in depth) I'd like to give notice that
I intend to apply the tools patches in 1-14 unless someone objects.

Thanks,
Ian.

Re: [PATCH 00/21] xenpaging changes for xen-unstable

[Keir Fraser]

On 07/12/2010 17:56, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:

> Olaf Hering writes ("[Xen-devel] [PATCH 00/21] xenpaging changes for
> xen-unstable"):
>> Here are some changes for xenpaging in xen-unstable.
>> 
>> Patches 1 to 14 are likely non-controversial and could be applied.
> 
> Having eyeballed them (but not in depth) I'd like to give notice that
> I intend to apply the tools patches in 1-14 unless someone objects.

I think you are looking at a stale patch series. Olaf's latest runs 00-17,
with patches 1-7 only considered by him suitable for immediate application.

 K.

> Thanks,
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 00/21] xenpaging changes for xen-unstable

[Keir Fraser]

On 07/12/2010 18:05, "Keir Fraser" <keir@xen.org> wrote:

> On 07/12/2010 17:56, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:
> 
>> Olaf Hering writes ("[Xen-devel] [PATCH 00/21] xenpaging changes for
>> xen-unstable"):
>>> Here are some changes for xenpaging in xen-unstable.
>>> 
>>> Patches 1 to 14 are likely non-controversial and could be applied.
>> 
>> Having eyeballed them (but not in depth) I'd like to give notice that
>> I intend to apply the tools patches in 1-14 unless someone objects.
> 
> I think you are looking at a stale patch series. Olaf's latest runs 00-17,
> with patches 1-7 only considered by him suitable for immediate application.

All of those patches are to tools/xenpaging, so I'll flush them from my
queue and leave them to you.

 -- Keir

>  K.
> 
>> Thanks,
>> Ian.
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 

Re: [PATCH 00/21] xenpaging changes for xen-unstable

[Ian Jackson]

Keir Fraser writes ("Re: [Xen-devel] [PATCH 00/21] xenpaging changes for xen-unstable"):
> I think you are looking at a stale patch series. Olaf's latest runs 00-17,
> with patches 1-7 only considered by him suitable for immediate application.

Oh, sorry.  I hadn't seen the second series and I'm rather behind on
the list.

Ian.