[PATCH 00 of 20] tools/xenpaging changes

[PATCH 00 of 20] tools/xenpaging changes

[Olaf Hering]

The following changes contains non-controversial changes for xenpaging.
They were sent out twice earlier already.

Please apply.


Olaf


 Config.mk                        |    2 
 config/StdGNU.mk                 |    2 
 tools/libxc/xc_bitops.h          |    6 
 tools/libxl/libxl.h              |    1 
 tools/libxl/libxl_paths.c        |    5 
 tools/xenpaging/Makefile         |    6 
 tools/xenpaging/file_ops.c       |   30 --
 tools/xenpaging/policy.h         |    2 
 tools/xenpaging/policy_default.c |   51 +++-
 tools/xenpaging/xenpaging.c      |  463 +++++++++++++++++++++++++++------------
 tools/xenpaging/xenpaging.h      |    8 
 11 files changed, 405 insertions(+), 171 deletions(-)

[PATCH 01 of 20] xenpaging: remove filename from comment

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804875 -3600
# Node ID b80e53a0d6f06ac6084765961127319990371dca
# Parent  335e8273a3f34a5e2972643a028f83684609f1c1
xenpaging: remove filename from comment

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 335e8273a3f3 -r b80e53a0d6f0 tools/xenpaging/file_ops.c
--- a/tools/xenpaging/file_ops.c
+++ b/tools/xenpaging/file_ops.c
@@ -1,5 +1,4 @@
 /******************************************************************************
- * tools/xenpaging/file_ops.c
  *
  * Common file operations.
  *
diff -r 335e8273a3f3 -r b80e53a0d6f0 tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -1,5 +1,4 @@
 /******************************************************************************
- * tools/xenpaging/policy.c
  *
  * Xen domain paging default policy.
  *
diff -r 335e8273a3f3 -r b80e53a0d6f0 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -1,5 +1,4 @@
 /******************************************************************************
- * tools/xenpaging/xenpaging.c
  *
  * Domain paging. 
  * Copyright (c) 2009 by Citrix Systems, Inc. (Patrick Colp)

[PATCH 02 of 20] xenpaging: remove obsolete comment in resume path

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804880 -3600
# Node ID 00f6e1294ae25a13820af1582170305b27eb69aa
# Parent  b80e53a0d6f06ac6084765961127319990371dca
xenpaging: remove obsolete comment in resume path

Remove stale comment.
If a page was populated several times the vcpu is paused and
xenpaging has to unpause it again.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r b80e53a0d6f0 -r 00f6e1294ae2 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -744,7 +744,6 @@ int main(int argc, char *argv[])
                         !!(req.flags & MEM_EVENT_FLAG_EVICT_FAIL) );
 
                 /* Tell Xen to resume the vcpu */
-                /* XXX: Maybe just check if the vcpu was paused? */
                 if ( req.flags & MEM_EVENT_FLAG_VCPU_PAUSED )
                 {
                     /* Prepare the response */

[PATCH 03 of 20] xenpaging: use PERROR to print errno

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804892 -3600
# Node ID b6c4a6f4f1f77b2471e4aa5fd23b375761691f79
# Parent  00f6e1294ae25a13820af1582170305b27eb69aa
xenpaging: use PERROR to print errno

v3:
 - adjust arguments for xc_mem_paging_enable() failures
v2:
 - move changes to file_op() to different patch

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 00f6e1294ae2 -r b6c4a6f4f1f7 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -90,7 +90,7 @@ static int xenpaging_wait_for_event_or_t
         if (errno == EINTR)
             return 0;
 
-        ERROR("Poll exited with an error");
+        PERROR("Poll exited with an error");
         return -errno;
     }
 
@@ -121,7 +121,7 @@ static int xenpaging_wait_for_event_or_t
         port = xc_evtchn_pending(xce);
         if ( port == -1 )
         {
-            ERROR("Failed to read port from event channel");
+            PERROR("Failed to read port from event channel");
             rc = -1;
             goto err;
         }
@@ -129,7 +129,7 @@ static int xenpaging_wait_for_event_or_t
         rc = xc_evtchn_unmask(xce, port);
         if ( rc < 0 )
         {
-            ERROR("Failed to unmask event channel port");
+            PERROR("Failed to unmask event channel port");
         }
     }
 err:
@@ -185,7 +185,7 @@ static xenpaging_t *xenpaging_init(domid
     paging->xs_handle = xs_open(0);
     if ( paging->xs_handle == NULL )
     {
-        ERROR("Error initialising xenstore connection");
+        PERROR("Error initialising xenstore connection");
         goto err;
     }
 
@@ -193,7 +193,7 @@ static xenpaging_t *xenpaging_init(domid
     snprintf(watch_token, sizeof(watch_token), "%u", domain_id);
     if ( xs_watch(paging->xs_handle, "@releaseDomain", watch_token) == false )
     {
-        ERROR("Could not bind to shutdown watch\n");
+        PERROR("Could not bind to shutdown watch\n");
         goto err;
     }
 
@@ -214,7 +214,7 @@ static xenpaging_t *xenpaging_init(domid
     paging->mem_event.shared_page = init_page();
     if ( paging->mem_event.shared_page == NULL )
     {
-        ERROR("Error initialising shared page");
+        PERROR("Error initialising shared page");
         goto err;
     }
 
@@ -222,7 +222,7 @@ static xenpaging_t *xenpaging_init(domid
     paging->mem_event.ring_page = init_page();
     if ( paging->mem_event.ring_page == NULL )
     {
-        ERROR("Error initialising ring page");
+        PERROR("Error initialising ring page");
         goto err;
     }
 
@@ -249,7 +249,7 @@ static xenpaging_t *xenpaging_init(domid
                 ERROR("xenpaging not supported in a PoD guest");
                 break;
             default:
-                ERROR("Error initialising shared page: %s", strerror(errno));
+                PERROR("Error initialising shared page");
                 break;
         }
         goto err;
@@ -259,7 +259,7 @@ static xenpaging_t *xenpaging_init(domid
     paging->mem_event.xce_handle = xc_evtchn_open(NULL, 0);
     if ( paging->mem_event.xce_handle == NULL )
     {
-        ERROR("Failed to open event channel");
+        PERROR("Failed to open event channel");
         goto err;
     }
 
@@ -269,7 +269,7 @@ static xenpaging_t *xenpaging_init(domid
                                     paging->mem_event.shared_page->port);
     if ( rc < 0 )
     {
-        ERROR("Failed to bind event channel");
+        PERROR("Failed to bind event channel");
         goto err;
     }
 
@@ -279,7 +279,7 @@ static xenpaging_t *xenpaging_init(domid
     paging->domain_info = malloc(sizeof(xc_domaininfo_t));
     if ( paging->domain_info == NULL )
     {
-        ERROR("Error allocating memory for domain info");
+        PERROR("Error allocating memory for domain info");
         goto err;
     }
 
@@ -287,7 +287,7 @@ static xenpaging_t *xenpaging_init(domid
                                paging->domain_info);
     if ( rc != 1 )
     {
-        ERROR("Error getting domain info");
+        PERROR("Error getting domain info");
         goto err;
     }
 
@@ -295,7 +295,7 @@ static xenpaging_t *xenpaging_init(domid
     paging->bitmap = bitmap_alloc(paging->domain_info->max_pages);
     if ( !paging->bitmap )
     {
-        ERROR("Error allocating bitmap");
+        PERROR("Error allocating bitmap");
         goto err;
     }
     DPRINTF("max_pages = %"PRIx64"\n", paging->domain_info->max_pages);
@@ -311,7 +311,7 @@ static xenpaging_t *xenpaging_init(domid
     rc = policy_init(paging);
     if ( rc != 0 )
     {
-        ERROR("Error initialising policy");
+        PERROR("Error initialising policy");
         goto err;
     }
 
@@ -358,14 +358,14 @@ static int xenpaging_teardown(xenpaging_
     rc = xc_mem_paging_disable(xch, paging->mem_event.domain_id);
     if ( rc != 0 )
     {
-        ERROR("Error tearing down domain paging in xen");
+        PERROR("Error tearing down domain paging in xen");
     }
 
     /* Unbind VIRQ */
     rc = xc_evtchn_unbind(paging->mem_event.xce_handle, paging->mem_event.port);
     if ( rc != 0 )
     {
-        ERROR("Error unbinding event port");
+        PERROR("Error unbinding event port");
     }
     paging->mem_event.port = -1;
 
@@ -373,7 +373,7 @@ static int xenpaging_teardown(xenpaging_
     rc = xc_evtchn_close(paging->mem_event.xce_handle);
     if ( rc != 0 )
     {
-        ERROR("Error closing event channel");
+        PERROR("Error closing event channel");
     }
     paging->mem_event.xce_handle = NULL;
     
@@ -384,7 +384,7 @@ static int xenpaging_teardown(xenpaging_
     rc = xc_interface_close(xch);
     if ( rc != 0 )
     {
-        ERROR("Error closing connection to xen");
+        PERROR("Error closing connection to xen");
     }
 
     return 0;
@@ -444,7 +444,7 @@ static int xenpaging_evict_page(xenpagin
                                 PROT_READ | PROT_WRITE, &gfn, 1);
     if ( page == NULL )
     {
-        ERROR("Error mapping page");
+        PERROR("Error mapping page");
         goto out;
     }
 
@@ -452,8 +452,8 @@ static int xenpaging_evict_page(xenpagin
     ret = write_page(fd, page, i);
     if ( ret != 0 )
     {
+        PERROR("Error copying page");
         munmap(page, PAGE_SIZE);
-        ERROR("Error copying page");
         goto out;
     }
 
@@ -464,7 +464,7 @@ static int xenpaging_evict_page(xenpagin
                               victim->gfn);
     if ( ret != 0 )
     {
-        ERROR("Error evicting page");
+        PERROR("Error evicting page");
         goto out;
     }
 
@@ -520,7 +520,7 @@ static int xenpaging_populate_page(xenpa
                 sleep(1);
                 continue;
             }
-            ERROR("Error preparing for page in");
+            PERROR("Error preparing for page in");
             goto out_map;
         }
     }
@@ -532,7 +532,7 @@ static int xenpaging_populate_page(xenpa
                                 PROT_READ | PROT_WRITE, &gfn, 1);
     if ( page == NULL )
     {
-        ERROR("Error mapping page: page is null");
+        PERROR("Error mapping page: page is null");
         goto out_map;
     }
 
@@ -540,7 +540,7 @@ static int xenpaging_populate_page(xenpa
     ret = read_page(fd, page, i);
     if ( ret != 0 )
     {
-        ERROR("Error reading page");
+        PERROR("Error reading page");
         goto out;
     }
 
@@ -579,7 +579,7 @@ static int evict_victim(xenpaging_t *pag
         {
             if ( j++ % 1000 == 0 )
                 if ( xenpaging_mem_paging_flush_ioemu_cache(paging) )
-                    ERROR("Error flushing ioemu cache");
+                    PERROR("Error flushing ioemu cache");
         }
     }
     while ( ret );
@@ -670,7 +670,7 @@ int main(int argc, char *argv[])
         rc = xenpaging_wait_for_event_or_timeout(paging);
         if ( rc < 0 )
         {
-            ERROR("Error getting event");
+            PERROR("Error getting event");
             goto out;
         }
         else if ( rc != 0 )
@@ -710,7 +710,7 @@ int main(int argc, char *argv[])
                     rc = xenpaging_populate_page(paging, req.gfn, fd, i);
                     if ( rc != 0 )
                     {
-                        ERROR("Error populating page");
+                        PERROR("Error populating page");
                         goto out;
                     }
                 }
@@ -723,7 +723,7 @@ int main(int argc, char *argv[])
                 rc = xenpaging_resume_page(paging, &rsp, 1);
                 if ( rc != 0 )
                 {
-                    ERROR("Error resuming page");
+                    PERROR("Error resuming page");
                     goto out;
                 }
 
@@ -754,7 +754,7 @@ int main(int argc, char *argv[])
                     rc = xenpaging_resume_page(paging, &rsp, 0);
                     if ( rc != 0 )
                     {
-                        ERROR("Error resuming");
+                        PERROR("Error resuming");
                         goto out;
                     }
                 }

[PATCH 04 of 20] xenpaging: simplify file_op

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804899 -3600
# Node ID 0bc6eb35c8bf6d09dc13f1fedb70874c1962f95d
# Parent  b6c4a6f4f1f77b2471e4aa5fd23b375761691f79
xenpaging: simplify file_op

Catch lseek() errors.
Use -1 as return value and let caller read errno.
Remove const casts from buffer pointers, the page is writeable.
Use wrapper for write() which matches the read() prototype.
Remove unused stdarg.h inclusion.
Remove unused macro.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r b6c4a6f4f1f7 -r 0bc6eb35c8bf tools/xenpaging/file_ops.c
--- a/tools/xenpaging/file_ops.c
+++ b/tools/xenpaging/file_ops.c
@@ -21,55 +21,44 @@
 
 
 #include <unistd.h>
-#include <stdarg.h>
 #include <xc_private.h>
 
-
-#define page_offset(_pfn)     (((off_t)(_pfn)) << PAGE_SHIFT)
-
-
 static int file_op(int fd, void *page, int i,
-                   ssize_t (*fn)(int, const void *, size_t))
+                   ssize_t (*fn)(int, void *, size_t))
 {
     off_t seek_ret;
-    int total;
+    int total = 0;
     int bytes;
-    int ret;
 
     seek_ret = lseek(fd, i << PAGE_SHIFT, SEEK_SET);
+    if ( seek_ret == (off_t)-1 )
+        return -1;
 
-    total = 0;
     while ( total < PAGE_SIZE )
     {
         bytes = fn(fd, page + total, PAGE_SIZE - total);
         if ( bytes <= 0 )
-        {
-            ret = -errno;
-            goto err;
-        }
+            return -1;
 
         total += bytes;
     }
 
     return 0;
-
- err:
-    return ret;
 }
 
-static ssize_t my_read(int fd, const void *buf, size_t count)
+static ssize_t my_write(int fd, void *buf, size_t count)
 {
-    return read(fd, (void *)buf, count);
+    return write(fd, buf, count);
 }
 
 int read_page(int fd, void *page, int i)
 {
-    return file_op(fd, page, i, &my_read);
+    return file_op(fd, page, i, &read);
 }
 
 int write_page(int fd, void *page, int i)
 {
-    return file_op(fd, page, i, &write);
+    return file_op(fd, page, i, &my_write);
 }

[PATCH 05 of 20] xenpaging: print gfn in failure case

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804901 -3600
# Node ID c0c01de70558deca53a338b8c0956b56900c2257
# Parent  0bc6eb35c8bf6d09dc13f1fedb70874c1962f95d
xenpaging: print gfn in failure case

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 0bc6eb35c8bf -r c0c01de70558 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -444,7 +444,7 @@ static int xenpaging_evict_page(xenpagin
                                 PROT_READ | PROT_WRITE, &gfn, 1);
     if ( page == NULL )
     {
-        PERROR("Error mapping page");
+        PERROR("Error mapping page %lx", victim->gfn);
         goto out;
     }
 
@@ -452,7 +452,7 @@ static int xenpaging_evict_page(xenpagin
     ret = write_page(fd, page, i);
     if ( ret != 0 )
     {
-        PERROR("Error copying page");
+        PERROR("Error copying page %lx", victim->gfn);
         munmap(page, PAGE_SIZE);
         goto out;
     }
@@ -464,7 +464,7 @@ static int xenpaging_evict_page(xenpagin
                               victim->gfn);
     if ( ret != 0 )
     {
-        PERROR("Error evicting page");
+        PERROR("Error evicting page %lx", victim->gfn);
         goto out;
     }
 
@@ -520,7 +520,7 @@ static int xenpaging_populate_page(xenpa
                 sleep(1);
                 continue;
             }
-            PERROR("Error preparing for page in");
+            PERROR("Error preparing %"PRI_xen_pfn" for page-in", gfn);
             goto out_map;
         }
     }
@@ -532,7 +532,7 @@ static int xenpaging_populate_page(xenpa
                                 PROT_READ | PROT_WRITE, &gfn, 1);
     if ( page == NULL )
     {
-        PERROR("Error mapping page: page is null");
+        PERROR("Error mapping page %"PRI_xen_pfn": page is null", gfn);
         goto out_map;
     }
 
@@ -540,7 +540,7 @@ static int xenpaging_populate_page(xenpa
     ret = read_page(fd, page, i);
     if ( ret != 0 )
     {
-        PERROR("Error reading page");
+        PERROR("Error reading page %"PRI_xen_pfn"", gfn);
         goto out;
     }
 
@@ -710,7 +710,7 @@ int main(int argc, char *argv[])
                     rc = xenpaging_populate_page(paging, req.gfn, fd, i);
                     if ( rc != 0 )
                     {
-                        PERROR("Error populating page");
+                        PERROR("Error populating page %"PRIx64"", req.gfn);
                         goto out;
                     }
                 }
@@ -723,7 +723,7 @@ int main(int argc, char *argv[])
                 rc = xenpaging_resume_page(paging, &rsp, 1);
                 if ( rc != 0 )
                 {
-                    PERROR("Error resuming page");
+                    PERROR("Error resuming page %"PRIx64"", req.gfn);
                     goto out;
                 }
 
@@ -754,7 +754,7 @@ int main(int argc, char *argv[])
                     rc = xenpaging_resume_page(paging, &rsp, 0);
                     if ( rc != 0 )
                     {
-                        PERROR("Error resuming");
+                        PERROR("Error resuming page %"PRIx64"", req.gfn);
                         goto out;
                     }
                 }

[PATCH 06 of 20] xenpaging: update xenpaging_init

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804942 -3600
# Node ID 3c4e8b05da8b15d399f6813cf5f9f365dd0b9a17
# Parent  c0c01de70558deca53a338b8c0956b56900c2257
xenpaging: update xenpaging_init

Move comment about xc_handle to the right place.
Allocate paging early and use calloc.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r c0c01de70558 -r 3c4e8b05da8b tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -169,18 +169,21 @@ static xenpaging_t *xenpaging_init(domid
     char *p;
     int rc;
 
+    /* Allocate memory */
+    paging = calloc(1, sizeof(xenpaging_t));
+    if ( !paging )
+        goto err;
+
     if ( getenv("XENPAGING_DEBUG") )
         dbg = (xentoollog_logger *)xtl_createlogger_stdiostream(stderr, XTL_DEBUG, 0);
-    xch = xc_interface_open(dbg, NULL, 0);
+
+    /* Open connection to xen */
+    paging->xc_handle = xch = xc_interface_open(dbg, NULL, 0);
     if ( !xch )
-        goto err_iface;
+        goto err;
 
     DPRINTF("xenpaging init\n");
 
-    /* Allocate memory */
-    paging = malloc(sizeof(xenpaging_t));
-    memset(paging, 0, sizeof(xenpaging_t));
-
     /* Open connection to xenstore */
     paging->xs_handle = xs_open(0);
     if ( paging->xs_handle == NULL )
@@ -204,9 +207,6 @@ static xenpaging_t *xenpaging_init(domid
          DPRINTF("Setting policy mru_size to %d\n", paging->policy_mru_size);
     }
 
-    /* Open connection to xen */
-    paging->xc_handle = xch;
-
     /* Set domain id */
     paging->mem_event.domain_id = domain_id;
 
@@ -322,7 +322,8 @@ static xenpaging_t *xenpaging_init(domid
     {
         if ( paging->xs_handle )
             xs_close(paging->xs_handle);
-        xc_interface_close(xch);
+        if ( xch )
+            xc_interface_close(xch);
         if ( paging->mem_event.shared_page )
         {
             munlock(paging->mem_event.shared_page, PAGE_SIZE);
@@ -340,7 +341,6 @@ static xenpaging_t *xenpaging_init(domid
         free(paging);
     }
 
- err_iface: 
     return NULL;
 }

[PATCH 07 of 20] xenpaging: remove xc_dominfo_t from paging_t

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804956 -3600
# Node ID 98c0a058f05634833ccb9fb6671ee94cde444fa1
# Parent  3c4e8b05da8b15d399f6813cf5f9f365dd0b9a17
xenpaging: remove xc_dominfo_t from paging_t

Remove xc_dominfo_t from paging_t, record only max_pages.
This value is used to setup internal data structures.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 3c4e8b05da8b -r 98c0a058f056 tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -41,17 +41,17 @@ int policy_init(xenpaging_t *paging)
     int i;
     int rc = -ENOMEM;
 
+    max_pages = paging->max_pages;
+
     /* Allocate bitmap for pages not to page out */
-    bitmap = bitmap_alloc(paging->domain_info->max_pages);
+    bitmap = bitmap_alloc(max_pages);
     if ( !bitmap )
         goto out;
     /* Allocate bitmap to track unusable pages */
-    unconsumed = bitmap_alloc(paging->domain_info->max_pages);
+    unconsumed = bitmap_alloc(max_pages);
     if ( !unconsumed )
         goto out;
 
-    max_pages = paging->domain_info->max_pages;
-
     /* Initialise MRU list of paged in pages */
     if ( paging->policy_mru_size > 0 )
         mru_size = paging->policy_mru_size;
diff -r 3c4e8b05da8b -r 98c0a058f056 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -164,6 +164,7 @@ static void *init_page(void)
 static xenpaging_t *xenpaging_init(domid_t domain_id, int num_pages)
 {
     xenpaging_t *paging;
+    xc_domaininfo_t domain_info;
     xc_interface *xch;
     xentoollog_logger *dbg = NULL;
     char *p;
@@ -275,34 +276,29 @@ static xenpaging_t *xenpaging_init(domid
 
     paging->mem_event.port = rc;
 
-    /* Get domaininfo */
-    paging->domain_info = malloc(sizeof(xc_domaininfo_t));
-    if ( paging->domain_info == NULL )
-    {
-        PERROR("Error allocating memory for domain info");
-        goto err;
-    }
-
     rc = xc_domain_getinfolist(xch, paging->mem_event.domain_id, 1,
-                               paging->domain_info);
+                               &domain_info);
     if ( rc != 1 )
     {
         PERROR("Error getting domain info");
         goto err;
     }
 
+    /* Record number of max_pages */
+    paging->max_pages = domain_info.max_pages;
+
     /* Allocate bitmap for tracking pages that have been paged out */
-    paging->bitmap = bitmap_alloc(paging->domain_info->max_pages);
+    paging->bitmap = bitmap_alloc(paging->max_pages);
     if ( !paging->bitmap )
     {
         PERROR("Error allocating bitmap");
         goto err;
     }
-    DPRINTF("max_pages = %"PRIx64"\n", paging->domain_info->max_pages);
+    DPRINTF("max_pages = %d\n", paging->max_pages);
 
-    if ( num_pages < 0 || num_pages > paging->domain_info->max_pages )
+    if ( num_pages < 0 || num_pages > paging->max_pages )
     {
-        num_pages = paging->domain_info->max_pages;
+        num_pages = paging->max_pages;
         DPRINTF("setting num_pages to %d\n", num_pages);
     }
     paging->num_pages = num_pages;
@@ -337,7 +333,6 @@ static xenpaging_t *xenpaging_init(domid
         }
 
         free(paging->bitmap);
-        free(paging->domain_info);
         free(paging);
     }
 
@@ -765,7 +760,7 @@ int main(int argc, char *argv[])
         if ( interrupted == SIGTERM || interrupted == SIGINT )
         {
             int num = 0;
-            for ( i = 0; i < paging->domain_info->max_pages; i++ )
+            for ( i = 0; i < paging->max_pages; i++ )
             {
                 if ( test_bit(i, paging->bitmap) )
                 {
@@ -781,7 +776,7 @@ int main(int argc, char *argv[])
              */
             if ( num )
                 page_in_trigger();
-            else if ( i == paging->domain_info->max_pages )
+            else if ( i == paging->max_pages )
                 break;
         }
         else
diff -r 3c4e8b05da8b -r 98c0a058f056 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -44,11 +44,11 @@ typedef struct xenpaging {
     xc_interface *xc_handle;
     struct xs_handle *xs_handle;
 
-    xc_domaininfo_t    *domain_info;
-
     unsigned long *bitmap;
 
     mem_event_t mem_event;
+    /* number of pages for which data structures were allocated */
+    int max_pages;
     int num_pages;
     int policy_mru_size;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];

[PATCH 08 of 20] xenpaging: track the number of paged-out pages

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804958 -3600
# Node ID 927af61e2c89bf70c3e873140750aa897fb575e9
# Parent  98c0a058f05634833ccb9fb6671ee94cde444fa1
xenpaging: track the number of paged-out pages

This change is required by subsequent changes.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 98c0a058f056 -r 927af61e2c89 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -467,6 +467,9 @@ static int xenpaging_evict_page(xenpagin
     /* Notify policy of page being paged out */
     policy_notify_paged_out(victim->gfn);
 
+    /* Record number of evicted pages */
+    paging->num_paged_out++;
+
  out:
     return ret;
 }
@@ -480,8 +483,13 @@ static int xenpaging_resume_page(xenpagi
 
     /* Notify policy of page being paged in */
     if ( notify_policy )
+    {
         policy_notify_paged_in(rsp->gfn);
 
+       /* Record number of resumed pages */
+       paging->num_paged_out--;
+    }
+
     /* Tell Xen page is ready */
     ret = xc_mem_paging_resume(paging->xc_handle, paging->mem_event.domain_id,
                                rsp->gfn);
diff -r 98c0a058f056 -r 927af61e2c89 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -49,6 +49,7 @@ typedef struct xenpaging {
     mem_event_t mem_event;
     /* number of pages for which data structures were allocated */
     int max_pages;
+    int num_paged_out;
     int num_pages;
     int policy_mru_size;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];

[PATCH 09 of 20] xenpaging: move page add/resume loops into its own function

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804959 -3600
# Node ID 35a65cd961e7ba04350ce0fe539f677dd4722b93
# Parent  927af61e2c89bf70c3e873140750aa897fb575e9
xenpaging: move page add/resume loops into its own function.

Move page resume loop into its own function.
Move page eviction loop into its own function.
Allocate all possible slots in a paging file to allow growing and
shrinking of the number of paged-out pages. Adjust other places to
iterate over all slots.

This change is required by subsequent patches.

v2:
 - check if victims allocation succeeded

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 927af61e2c89 -r 35a65cd961e7 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -553,6 +553,27 @@ static int xenpaging_populate_page(xenpa
     return ret;
 }
 
+/* Trigger a page-in for a batch of pages */
+static void resume_pages(xenpaging_t *paging, int num_pages)
+{
+    xc_interface *xch = paging->xc_handle;
+    int i, num = 0;
+
+    for ( i = 0; i < paging->max_pages && num < num_pages; i++ )
+    {
+        if ( test_bit(i, paging->bitmap) )
+        {
+            paging->pagein_queue[num] = i;
+            num++;
+            if ( num == XENPAGING_PAGEIN_QUEUE_SIZE )
+                break;
+        }
+    }
+    /* num may be less than num_pages, caller has to try again */
+    if ( num )
+        page_in_trigger();
+}
+
 static int evict_victim(xenpaging_t *paging,
                         xenpaging_victim_t *victim, int fd, int i)
 {
@@ -596,6 +617,30 @@ static int evict_victim(xenpaging_t *pag
     return ret;
 }
 
+/* Evict a batch of pages and write them to a free slot in the paging file */
+static int evict_pages(xenpaging_t *paging, int fd, xenpaging_victim_t *victims, int num_pages)
+{
+    xc_interface *xch = paging->xc_handle;
+    int rc, slot, num = 0;
+
+    for ( slot = 0; slot < paging->max_pages && num < num_pages; slot++ )
+    {
+        /* Slot is allocated */
+        if ( victims[slot].gfn != INVALID_MFN )
+            continue;
+
+        rc = evict_victim(paging, &victims[slot], fd, slot);
+        if ( rc == -ENOSPC )
+            break;
+        if ( rc == -EINTR )
+            break;
+        if ( num && num % 100 == 0 )
+            DPRINTF("%d pages evicted\n", num);
+        num++;
+    }
+    return num;
+}
+
 int main(int argc, char *argv[])
 {
     struct sigaction act;
@@ -638,7 +683,14 @@ int main(int argc, char *argv[])
         return 2;
     }
 
-    victims = calloc(paging->num_pages, sizeof(xenpaging_victim_t));
+    /* Allocate upper limit of pages to allow growing and shrinking */
+    victims = calloc(paging->max_pages, sizeof(xenpaging_victim_t));
+    if ( !victims )
+        goto out;
+
+    /* Mark all slots as unallocated */
+    for ( i = 0; i < paging->max_pages; i++ )
+        victims[i].gfn = INVALID_MFN;
 
     /* ensure that if we get a signal, we'll do cleanup, then exit */
     act.sa_handler = close_handler;
@@ -652,18 +704,7 @@ int main(int argc, char *argv[])
     /* listen for page-in events to stop pager */
     create_page_in_thread(paging);
 
-    /* Evict pages */
-    for ( i = 0; i < paging->num_pages; i++ )
-    {
-        rc = evict_victim(paging, &victims[i], fd, i);
-        if ( rc == -ENOSPC )
-            break;
-        if ( rc == -EINTR )
-            break;
-        if ( i % 100 == 0 )
-            DPRINTF("%d pages evicted\n", i);
-    }
-
+    i = evict_pages(paging, fd, victims, paging->num_pages);
     DPRINTF("%d pages evicted. Done.\n", i);
 
     /* Swap pages in and out */
@@ -689,13 +730,13 @@ int main(int argc, char *argv[])
             if ( test_and_clear_bit(req.gfn, paging->bitmap) )
             {
                 /* Find where in the paging file to read from */
-                for ( i = 0; i < paging->num_pages; i++ )
+                for ( i = 0; i < paging->max_pages; i++ )
                 {
                     if ( victims[i].gfn == req.gfn )
                         break;
                 }
     
-                if ( i >= paging->num_pages )
+                if ( i >= paging->max_pages )
                 {
                     DPRINTF("Couldn't find page %"PRIx64"\n", req.gfn);
                     goto out;
@@ -767,25 +808,12 @@ int main(int argc, char *argv[])
         /* Write all pages back into the guest */
         if ( interrupted == SIGTERM || interrupted == SIGINT )
         {
-            int num = 0;
-            for ( i = 0; i < paging->max_pages; i++ )
-            {
-                if ( test_bit(i, paging->bitmap) )
-                {
-                    paging->pagein_queue[num] = i;
-                    num++;
-                    if ( num == XENPAGING_PAGEIN_QUEUE_SIZE )
-                        break;
-                }
-            }
-            /*
-             * One more round if there are still pages to process.
-             * If no more pages to process, exit loop.
-             */
-            if ( num )
-                page_in_trigger();
-            else if ( i == paging->max_pages )
+            /* If no more pages to process, exit loop. */
+            if ( !paging->num_paged_out )
                 break;
+            
+            /* One more round if there are still pages to process. */
+            resume_pages(paging, paging->num_paged_out);
         }
         else
         {

[PATCH 10 of 20] xenpaging: improve mainloop exit handling

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804959 -3600
# Node ID cc72e36fa9c3a6a467ad12188551f6f233185fc9
# Parent  35a65cd961e7ba04350ce0fe539f677dd4722b93
xenpaging: improve mainloop exit handling

Remove the if/else logic to exit from the in case a signal arrives.
Update comments.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 35a65cd961e7 -r cc72e36fa9c3 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -805,7 +805,7 @@ int main(int argc, char *argv[])
             }
         }
 
-        /* Write all pages back into the guest */
+        /* If interrupted, write all pages back into the guest */
         if ( interrupted == SIGTERM || interrupted == SIGINT )
         {
             /* If no more pages to process, exit loop. */
@@ -814,13 +814,15 @@ int main(int argc, char *argv[])
             
             /* One more round if there are still pages to process. */
             resume_pages(paging, paging->num_paged_out);
+
+            /* Resume main loop */
+            continue;
         }
-        else
-        {
-            /* Exit on any other signal */
-            if ( interrupted )
-                break;
-        }
+
+        /* Exit main loop on any other signal */
+        if ( interrupted )
+            break;
+
     }
     DPRINTF("xenpaging got signal %d\n", interrupted);

[PATCH 11 of 20] libxc: add bitmap_clear function

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804960 -3600
# Node ID e74c938d38518f6f6344c9bcc6b7c8916a073594
# Parent  cc72e36fa9c3a6a467ad12188551f6f233185fc9
libxc: add bitmap_clear function

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r cc72e36fa9c3 -r e74c938d3851 tools/libxc/xc_bitops.h
--- a/tools/libxc/xc_bitops.h
+++ b/tools/libxc/xc_bitops.h
@@ -4,6 +4,7 @@
 /* bitmap operations for single threaded access */
 
 #include <stdlib.h>
+#include <string.h>
 
 #define BITS_PER_LONG (sizeof(unsigned long) * 8)
 #define ORDER_LONG (sizeof(unsigned long) == 4 ? 5 : 6)
@@ -25,6 +26,11 @@ static inline unsigned long *bitmap_allo
     return calloc(1, bitmap_size(nr_bits));
 }
 
+static inline void bitmap_clear(unsigned long *addr, int nr_bits)
+{
+    memset(addr, 0, bitmap_size(nr_bits));
+}
+
 static inline int test_bit(int nr, volatile unsigned long *addr)
 {
     return (BITMAP_ENTRY(nr, addr) >> BITMAP_SHIFT(nr)) & 1;

[PATCH 12 of 20] xenpaging: retry unpageable gfns

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804961 -3600
# Node ID 1e343a57860ea0d75d862076eec695b81219092e
# Parent  e74c938d38518f6f6344c9bcc6b7c8916a073594
xenpaging: retry unpageable gfns

Nomination of gfns can fail, but may succeed later.
Thats the case for a guest that starts ballooned.

v2:
 - print debug when clearing uncosumed happens

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r e74c938d3851 -r 1e343a57860e tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -32,6 +32,7 @@ static unsigned int i_mru;
 static unsigned int mru_size;
 static unsigned long *bitmap;
 static unsigned long *unconsumed;
+static unsigned int unconsumed_cleared;
 static unsigned long current_gfn;
 static unsigned long max_pages;
 
@@ -87,8 +88,21 @@ int policy_choose_victim(xenpaging_t *pa
         current_gfn++;
         if ( current_gfn >= max_pages )
             current_gfn = 0;
+        /* Could not nominate any gfn */
         if ( wrap == current_gfn )
         {
+            /* Count wrap arounds */
+            unconsumed_cleared++;
+            /* Force retry every few seconds (depends on poll() timeout) */
+            if ( unconsumed_cleared > 123)
+            {
+                /* Force retry of unconsumed gfns */
+                bitmap_clear(unconsumed, max_pages);
+                unconsumed_cleared = 0;
+                DPRINTF("clearing unconsumed, wrap %lx", wrap);
+                /* One more round before returning ENOSPC */
+                continue;
+            }
             victim->gfn = INVALID_MFN;
             return -ENOSPC;
         }

[PATCH 13 of 20] xenpaging: install into LIBEXEC dir

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804962 -3600
# Node ID 3fe020b22882c369855dab3f3ebb88c02568e917
# Parent  1e343a57860ea0d75d862076eec695b81219092e
xenpaging: install into LIBEXEC dir

In preparation of upcoming libxl integration,
move xenpaging binary from /usr/sbin/ to /usr/lib/xen/bin/

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 1e343a57860e -r 3fe020b22882 tools/xenpaging/Makefile
--- a/tools/xenpaging/Makefile
+++ b/tools/xenpaging/Makefile
@@ -24,8 +24,8 @@ xenpaging: $(OBJS)
 
 install: all
 	$(INSTALL_DIR) $(DESTDIR)/var/lib/xen/xenpaging
-	$(INSTALL_DIR) $(DESTDIR)$(SBINDIR)
-	$(INSTALL_PROG) $(IBINS) $(DESTDIR)$(SBINDIR)
+	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC)
+	$(INSTALL_PROG) $(IBINS) $(DESTDIR)$(LIBEXEC)
 
 clean:
 	rm -f *.o *~ $(DEPS) xen TAGS $(IBINS) $(LIB)

[PATCH 14 of 20] xenpaging: add XEN_PAGING_DIR / libxl_xenpaging_dir_path()

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804963 -3600
# Node ID 303d1521463bda6431cc0001d8755dd681e67a0d
# Parent  3fe020b22882c369855dab3f3ebb88c02568e917
xenpaging: add XEN_PAGING_DIR / libxl_xenpaging_dir_path()

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 3fe020b22882 -r 303d1521463b Config.mk
--- a/Config.mk
+++ b/Config.mk
@@ -142,7 +142,7 @@ define buildmakevars2file-closure
 	$(foreach var,                                                      \
 	          SBINDIR BINDIR LIBEXEC LIBDIR SHAREDIR PRIVATE_BINDIR     \
 	          XENFIRMWAREDIR XEN_CONFIG_DIR XEN_SCRIPT_DIR XEN_LOCK_DIR \
-	          XEN_RUN_DIR,                                              \
+	          XEN_RUN_DIR XEN_PAGING_DIR,                               \
 	          echo "$(var)=\"$($(var))\"" >>$(1).tmp;)        \
 	$(call move-if-changed,$(1).tmp,$(1))
 endef
diff -r 3fe020b22882 -r 303d1521463b config/StdGNU.mk
--- a/config/StdGNU.mk
+++ b/config/StdGNU.mk
@@ -53,10 +53,12 @@ ifeq ($(PREFIX),/usr)
 CONFIG_DIR = /etc
 XEN_LOCK_DIR = /var/lock
 XEN_RUN_DIR = /var/run/xen
+XEN_PAGING_DIR = /var/lib/xen/xenpaging
 else
 CONFIG_DIR = $(PREFIX)/etc
 XEN_LOCK_DIR = $(PREFIX)/var/lock
 XEN_RUN_DIR = $(PREFIX)/var/run/xen
+XEN_PAGING_DIR = $(PREFIX)/var/lib/xen/xenpaging
 endif
 
 SYSCONFIG_DIR = $(CONFIG_DIR)/$(CONFIG_LEAF_DIR)
diff -r 3fe020b22882 -r 303d1521463b tools/libxl/libxl.h
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -633,6 +633,7 @@ const char *libxl_xen_config_dir_path(vo
 const char *libxl_xen_script_dir_path(void);
 const char *libxl_lock_dir_path(void);
 const char *libxl_run_dir_path(void);
+const char *libxl_xenpaging_dir_path(void);
 
 #endif /* LIBXL_H */
 
diff -r 3fe020b22882 -r 303d1521463b tools/libxl/libxl_paths.c
--- a/tools/libxl/libxl_paths.c
+++ b/tools/libxl/libxl_paths.c
@@ -70,6 +70,11 @@ const char *libxl_run_dir_path(void)
     return XEN_RUN_DIR;
 }
 
+const char *libxl_xenpaging_dir_path(void)
+{
+    return XEN_PAGING_DIR;
+}
+
 /*
  * Local variables:
  * mode: C
diff -r 3fe020b22882 -r 303d1521463b tools/xenpaging/Makefile
--- a/tools/xenpaging/Makefile
+++ b/tools/xenpaging/Makefile
@@ -23,7 +23,7 @@ xenpaging: $(OBJS)
 	$(CC) $(LDFLAGS) -o $@ $^ $(LDLIBS) $(APPEND_LDFLAGS)
 
 install: all
-	$(INSTALL_DIR) $(DESTDIR)/var/lib/xen/xenpaging
+	$(INSTALL_DIR) $(DESTDIR)$(XEN_PAGING_DIR)
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC)
 	$(INSTALL_PROG) $(IBINS) $(DESTDIR)$(LIBEXEC)

[PATCH 15 of 20] xenpaging: use guests tot_pages as working target

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804965 -3600
# Node ID 827fcaf0d610dd0c54d3f1f305b2b8f12bd4240a
# Parent  303d1521463bda6431cc0001d8755dd681e67a0d
xenpaging: use guests tot_pages as working target

This change reverses the task of xenpaging. Before this change a fixed number
of pages was paged out. With this change the guest will not have access to
more than the given number of pages at the same time.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 303d1521463b -r 827fcaf0d610 tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -71,7 +71,6 @@ int policy_init(xenpaging_t *paging)
 
     /* Start in the middle to avoid paging during BIOS startup */
     current_gfn = max_pages / 2;
-    current_gfn -= paging->num_pages / 2;
 
     rc = 0;
  out:
diff -r 303d1521463b -r 827fcaf0d610 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -136,6 +136,21 @@ err:
     return rc;
 }
 
+static int xenpaging_get_tot_pages(xenpaging_t *paging)
+{
+    xc_interface *xch = paging->xc_handle;
+    xc_domaininfo_t domain_info;
+    int rc;
+
+    rc = xc_domain_getinfolist(xch, paging->mem_event.domain_id, 1, &domain_info);
+    if ( rc != 1 )
+    {
+        PERROR("Error getting domain info");
+        return -1;
+    }
+    return domain_info.tot_pages;
+}
+
 static void *init_page(void)
 {
     void *buffer;
@@ -161,7 +176,7 @@ static void *init_page(void)
     return NULL;
 }
 
-static xenpaging_t *xenpaging_init(domid_t domain_id, int num_pages)
+static xenpaging_t *xenpaging_init(domid_t domain_id, int target_tot_pages)
 {
     xenpaging_t *paging;
     xc_domaininfo_t domain_info;
@@ -296,12 +311,7 @@ static xenpaging_t *xenpaging_init(domid
     }
     DPRINTF("max_pages = %d\n", paging->max_pages);
 
-    if ( num_pages < 0 || num_pages > paging->max_pages )
-    {
-        num_pages = paging->max_pages;
-        DPRINTF("setting num_pages to %d\n", num_pages);
-    }
-    paging->num_pages = num_pages;
+    paging->target_tot_pages = target_tot_pages;
 
     /* Initialise policy */
     rc = policy_init(paging);
@@ -648,7 +658,9 @@ int main(int argc, char *argv[])
     xenpaging_victim_t *victims;
     mem_event_request_t req;
     mem_event_response_t rsp;
+    int num, prev_num = 0;
     int i;
+    int tot_pages;
     int rc = -1;
     int rc1;
     xc_interface *xch;
@@ -659,7 +671,7 @@ int main(int argc, char *argv[])
 
     if ( argc != 3 )
     {
-        fprintf(stderr, "Usage: %s <domain_id> <num_pages>\n", argv[0]);
+        fprintf(stderr, "Usage: %s <domain_id> <tot_pages>\n", argv[0]);
         return -1;
     }
 
@@ -672,7 +684,7 @@ int main(int argc, char *argv[])
     }
     xch = paging->xc_handle;
 
-    DPRINTF("starting %s %u %d\n", argv[0], paging->mem_event.domain_id, paging->num_pages);
+    DPRINTF("starting %s %u %d\n", argv[0], paging->mem_event.domain_id, paging->target_tot_pages);
 
     /* Open file */
     sprintf(filename, "page_cache_%u", paging->mem_event.domain_id);
@@ -704,9 +716,6 @@ int main(int argc, char *argv[])
     /* listen for page-in events to stop pager */
     create_page_in_thread(paging);
 
-    i = evict_pages(paging, fd, victims, paging->num_pages);
-    DPRINTF("%d pages evicted. Done.\n", i);
-
     /* Swap pages in and out */
     while ( 1 )
     {
@@ -771,12 +780,8 @@ int main(int argc, char *argv[])
                     goto out;
                 }
 
-                /* Evict a new page to replace the one we just paged in,
-                 * or clear this pagefile slot on exit */
-                if ( interrupted )
-                    victims[i].gfn = INVALID_MFN;
-                else
-                    evict_victim(paging, &victims[i], fd, i);
+                /* Clear this pagefile slot */
+                victims[i].gfn = INVALID_MFN;
             }
             else
             {
@@ -823,6 +828,43 @@ int main(int argc, char *argv[])
         if ( interrupted )
             break;
 
+        /* Check if the target has been reached already */
+        tot_pages = xenpaging_get_tot_pages(paging);
+        if ( tot_pages < 0 )
+            goto out;
+
+        /* Resume all pages if paging is disabled or no target was set */
+        if ( paging->target_tot_pages == 0 )
+        {
+            if ( paging->num_paged_out )
+                resume_pages(paging, paging->num_paged_out);
+        }
+        /* Evict more pages if target not reached */
+        else if ( tot_pages > paging->target_tot_pages )
+        {
+            num = tot_pages - paging->target_tot_pages;
+            if ( num != prev_num )
+            {
+                DPRINTF("Need to evict %d pages to reach %d target_tot_pages\n", num, paging->target_tot_pages);
+                prev_num = num;
+            }
+            /* Limit the number of evicts to be able to process page-in requests */
+            if ( num > 42 )
+                num = 42;
+            evict_pages(paging, fd, victims, num);
+        }
+        /* Resume some pages if target not reached */
+        else if ( tot_pages < paging->target_tot_pages && paging->num_paged_out )
+        {
+            num = paging->target_tot_pages - tot_pages;
+            if ( num != prev_num )
+            {
+                DPRINTF("Need to resume %d pages to reach %d target_tot_pages\n", num, paging->target_tot_pages);
+                prev_num = num;
+            }
+            resume_pages(paging, num);
+        }
+
     }
     DPRINTF("xenpaging got signal %d\n", interrupted);
 
diff -r 303d1521463b -r 827fcaf0d610 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -50,7 +50,7 @@ typedef struct xenpaging {
     /* number of pages for which data structures were allocated */
     int max_pages;
     int num_paged_out;
-    int num_pages;
+    int target_tot_pages;
     int policy_mru_size;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];
 } xenpaging_t;

[PATCH 16 of 20] xenpaging: watch the guests memory/target-tot_pages xenstore value

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804967 -3600
# Node ID bbe679f21208fd8f3341e2d8556469c7b1e29d57
# Parent  827fcaf0d610dd0c54d3f1f305b2b8f12bd4240a
xenpaging: watch the guests memory/target-tot_pages xenstore value

Subsequent patches will use xenstored to store the numbers of pages
xenpaging is suppose to page-out.
Remove num_pages and use target_pages instead.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 827fcaf0d610 -r bbe679f21208 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -19,8 +19,10 @@
  */
 
 #define _XOPEN_SOURCE	600
+#define _GNU_SOURCE
 
 #include <inttypes.h>
+#include <stdio.h>
 #include <stdlib.h>
 #include <stdarg.h>
 #include <time.h>
@@ -35,6 +37,10 @@
 #include "policy.h"
 #include "xenpaging.h"
 
+/* Defines number of mfns a guest should use at a time, in KiB */
+#define WATCH_TARGETPAGES "memory/target-tot_pages"
+static char *watch_target_tot_pages;
+static char *dom_path;
 static char watch_token[16];
 static char filename[80];
 static int interrupted;
@@ -72,7 +78,7 @@ static int xenpaging_wait_for_event_or_t
 {
     xc_interface *xch = paging->xc_handle;
     xc_evtchn *xce = paging->mem_event.xce_handle;
-    char **vec;
+    char **vec, *val;
     unsigned int num;
     struct pollfd fd[2];
     int port;
@@ -111,6 +117,25 @@ static int xenpaging_wait_for_event_or_t
                     rc = 0;
                 }
             }
+            else if ( strcmp(vec[XS_WATCH_PATH], watch_target_tot_pages) == 0 )
+            {
+                int ret, target_tot_pages;
+                val = xs_read(paging->xs_handle, XBT_NULL, vec[XS_WATCH_PATH], NULL);
+                if ( val )
+                {
+                    ret = sscanf(val, "%d", &target_tot_pages);
+                    if ( ret > 0 )
+                    {
+                        /* KiB to pages */
+                        target_tot_pages >>= 2;
+                        if ( target_tot_pages < 0 || target_tot_pages > paging->max_pages )
+                            target_tot_pages = paging->max_pages;
+                        paging->target_tot_pages = target_tot_pages;
+                        DPRINTF("new target_tot_pages %d\n", target_tot_pages);
+                    }
+                    free(val);
+                }
+            }
             free(vec);
         }
     }
@@ -216,6 +241,25 @@ static xenpaging_t *xenpaging_init(domid
         goto err;
     }
 
+    /* Watch xenpagings working target */
+    dom_path = xs_get_domain_path(paging->xs_handle, domain_id);
+    if ( !dom_path )
+    {
+        PERROR("Could not find domain path\n");
+        goto err;
+    }
+    if ( asprintf(&watch_target_tot_pages, "%s/%s", dom_path, WATCH_TARGETPAGES) < 0 )
+    {
+        PERROR("Could not alloc watch path\n");
+        goto err;
+    }
+    DPRINTF("watching '%s'\n", watch_target_tot_pages);
+    if ( xs_watch(paging->xs_handle, watch_target_tot_pages, "") == false )
+    {
+        PERROR("Could not bind to xenpaging watch\n");
+        goto err;
+    }
+
     p = getenv("XENPAGING_POLICY_MRU_SIZE");
     if ( p && *p )
     {
@@ -342,6 +386,8 @@ static xenpaging_t *xenpaging_init(domid
             free(paging->mem_event.ring_page);
         }
 
+        free(dom_path);
+        free(watch_target_tot_pages);
         free(paging->bitmap);
         free(paging);
     }
@@ -357,6 +403,9 @@ static int xenpaging_teardown(xenpaging_
     if ( paging == NULL )
         return 0;
 
+    xs_unwatch(paging->xs_handle, watch_target_tot_pages, "");
+    xs_unwatch(paging->xs_handle, "@releaseDomain", watch_token);
+
     xch = paging->xc_handle;
     paging->xc_handle = NULL;
     /* Tear down domain paging in Xen */

[PATCH 17 of 20] xenpaging: add cmdline interface for pager

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804968 -3600
# Node ID deb016f351e55e924274b4ccfbb80cb71543fc3e
# Parent  bbe679f21208fd8f3341e2d8556469c7b1e29d57
xenpaging: add cmdline interface for pager

Introduce a cmdline handling for the pager. This simplifies libxl support,
debug and mru_size are not passed via the environment anymore.
The new interface looks like this:

xenpaging [options] -f <pagefile> -d <domain_id>
options:
 -d <domid>     --domain=<domid>         numerical domain_id of guest. This option is required.
 -f <file>      --pagefile=<file>        pagefile to use. This option is required.
 -m <max_memkb> --max_memkb=<max_memkb>  maximum amount of memory to handle.
 -r <num>       --mru_size=<num>         number of paged-in pages to keep in memory.
 -d             --debug                  enable debug output.
 -h             --help                   this output.


Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r bbe679f21208 -r deb016f351e5 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -31,6 +31,7 @@
 #include <poll.h>
 #include <xc_private.h>
 #include <xs.h>
+#include <getopt.h>
 
 #include "xc_bitops.h"
 #include "file_ops.h"
@@ -42,12 +43,12 @@
 static char *watch_target_tot_pages;
 static char *dom_path;
 static char watch_token[16];
-static char filename[80];
+static char *filename;
 static int interrupted;
 
 static void unlink_pagefile(void)
 {
-    if ( filename[0] )
+    if ( filename && filename[0] )
     {
         unlink(filename);
         filename[0] = '\0';
@@ -201,11 +202,85 @@ static void *init_page(void)
     return NULL;
 }
 
-static xenpaging_t *xenpaging_init(domid_t domain_id, int target_tot_pages)
+static void usage(void)
+{
+    printf("usage:\n\n");
+
+    printf("  xenpaging [options] -f <pagefile> -d <domain_id>\n\n");
+
+    printf("options:\n");
+    printf(" -d <domid>     --domain=<domid>         numerical domain_id of guest. This option is required.\n");
+    printf(" -f <file>      --pagefile=<file>        pagefile to use. This option is required.\n");
+    printf(" -m <max_memkb> --max_memkb=<max_memkb>  maximum amount of memory to handle.\n");
+    printf(" -r <num>       --mru_size=<num>         number of paged-in pages to keep in memory.\n");
+    printf(" -v             --verbose                enable debug output.\n");
+    printf(" -h             --help                   this output.\n");
+}
+
+static int xenpaging_getopts(xenpaging_t *paging, int argc, char *argv[])
+{
+    int ch;
+    static const char sopts[] = "hvd:f:m:r:";
+    static const struct option lopts[] = {
+        {"help", 0, NULL, 'h'},
+        {"verbose", 0, NULL, 'v'},
+        {"domain", 1, NULL, 'd'},
+        {"pagefile", 1, NULL, 'f'},
+        {"mru_size", 1, NULL, 'm'},
+        { }
+    };
+
+    while ((ch = getopt_long(argc, argv, sopts, lopts, NULL)) != -1)
+    {
+        switch(ch) {
+        case 'd':
+            paging->mem_event.domain_id = atoi(optarg);
+            break;
+        case 'f':
+            filename = strdup(optarg);
+            break;
+        case 'm':
+            /* KiB to pages */
+            paging->max_pages = atoi(optarg) >> 2;
+            break;
+        case 'r':
+            paging->policy_mru_size = atoi(optarg);
+            break;
+        case 'v':
+            paging->debug = 1;
+            break;
+        case 'h':
+        case '?':
+            usage();
+            return 1;
+        }
+    }
+
+    argv += optind; argc -= optind;
+    
+    /* Path to pagefile is required */
+    if ( !filename )
+    {
+        printf("Filename for pagefile missing!\n");
+        usage();
+        return 1;
+    }
+
+    /* Set domain id */
+    if ( !paging->mem_event.domain_id )
+    {
+        printf("Numerical <domain_id> missing!\n");
+        return 1;
+    }
+
+    return 0;
+}
+
+static xenpaging_t *xenpaging_init(int argc, char *argv[])
 {
     xenpaging_t *paging;
     xc_domaininfo_t domain_info;
-    xc_interface *xch;
+    xc_interface *xch = NULL;
     xentoollog_logger *dbg = NULL;
     char *p;
     int rc;
@@ -215,7 +290,12 @@ static xenpaging_t *xenpaging_init(domid
     if ( !paging )
         goto err;
 
-    if ( getenv("XENPAGING_DEBUG") )
+    /* Get cmdline options and domain_id */
+    if ( xenpaging_getopts(paging, argc, argv) )
+        goto err;
+
+    /* Enable debug output */
+    if ( paging->debug )
         dbg = (xentoollog_logger *)xtl_createlogger_stdiostream(stderr, XTL_DEBUG, 0);
 
     /* Open connection to xen */
@@ -234,7 +314,7 @@ static xenpaging_t *xenpaging_init(domid
     }
 
     /* write domain ID to watch so we can ignore other domain shutdowns */
-    snprintf(watch_token, sizeof(watch_token), "%u", domain_id);
+    snprintf(watch_token, sizeof(watch_token), "%u", paging->mem_event.domain_id);
     if ( xs_watch(paging->xs_handle, "@releaseDomain", watch_token) == false )
     {
         PERROR("Could not bind to shutdown watch\n");
@@ -242,7 +322,7 @@ static xenpaging_t *xenpaging_init(domid
     }
 
     /* Watch xenpagings working target */
-    dom_path = xs_get_domain_path(paging->xs_handle, domain_id);
+    dom_path = xs_get_domain_path(paging->xs_handle, paging->mem_event.domain_id);
     if ( !dom_path )
     {
         PERROR("Could not find domain path\n");
@@ -260,16 +340,6 @@ static xenpaging_t *xenpaging_init(domid
         goto err;
     }
 
-    p = getenv("XENPAGING_POLICY_MRU_SIZE");
-    if ( p && *p )
-    {
-         paging->policy_mru_size = atoi(p);
-         DPRINTF("Setting policy mru_size to %d\n", paging->policy_mru_size);
-    }
-
-    /* Set domain id */
-    paging->mem_event.domain_id = domain_id;
-
     /* Initialise shared page */
     paging->mem_event.shared_page = init_page();
     if ( paging->mem_event.shared_page == NULL )
@@ -335,17 +405,21 @@ static xenpaging_t *xenpaging_init(domid
 
     paging->mem_event.port = rc;
 
-    rc = xc_domain_getinfolist(xch, paging->mem_event.domain_id, 1,
-                               &domain_info);
-    if ( rc != 1 )
+    /* Get max_pages from guest if not provided via cmdline */
+    if ( !paging->max_pages )
     {
-        PERROR("Error getting domain info");
-        goto err;
+        rc = xc_domain_getinfolist(xch, paging->mem_event.domain_id, 1,
+                                   &domain_info);
+        if ( rc != 1 )
+        {
+            PERROR("Error getting domain info");
+            goto err;
+        }
+
+        /* Record number of max_pages */
+        paging->max_pages = domain_info.max_pages;
     }
 
-    /* Record number of max_pages */
-    paging->max_pages = domain_info.max_pages;
-
     /* Allocate bitmap for tracking pages that have been paged out */
     paging->bitmap = bitmap_alloc(paging->max_pages);
     if ( !paging->bitmap )
@@ -355,8 +429,6 @@ static xenpaging_t *xenpaging_init(domid
     }
     DPRINTF("max_pages = %d\n", paging->max_pages);
 
-    paging->target_tot_pages = target_tot_pages;
-
     /* Initialise policy */
     rc = policy_init(paging);
     if ( rc != 0 )
@@ -718,25 +790,18 @@ int main(int argc, char *argv[])
     mode_t open_mode = S_IRUSR | S_IRGRP | S_IROTH | S_IWUSR | S_IWGRP | S_IWOTH;
     int fd;
 
-    if ( argc != 3 )
-    {
-        fprintf(stderr, "Usage: %s <domain_id> <tot_pages>\n", argv[0]);
-        return -1;
-    }
-
     /* Initialise domain paging */
-    paging = xenpaging_init(atoi(argv[1]), atoi(argv[2]));
+    paging = xenpaging_init(argc, argv);
     if ( paging == NULL )
     {
-        fprintf(stderr, "Error initialising paging");
+        fprintf(stderr, "Error initialising paging\n");
         return 1;
     }
     xch = paging->xc_handle;
 
-    DPRINTF("starting %s %u %d\n", argv[0], paging->mem_event.domain_id, paging->target_tot_pages);
+    DPRINTF("starting %s for domain_id %u with pagefile %s\n", argv[0], paging->mem_event.domain_id, filename);
 
     /* Open file */
-    sprintf(filename, "page_cache_%u", paging->mem_event.domain_id);
     fd = open(filename, open_flags, open_mode);
     if ( fd < 0 )
     {
diff -r bbe679f21208 -r deb016f351e5 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -52,6 +52,7 @@ typedef struct xenpaging {
     int num_paged_out;
     int target_tot_pages;
     int policy_mru_size;
+    int debug;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];
 } xenpaging_t;

[PATCH 18 of 20] xenpaging: improve policy mru list handling

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321804970 -3600
# Node ID ba84a323f2dfbdfd2430d3897e616a0e1dd1e4d7
# Parent  deb016f351e55e924274b4ccfbb80cb71543fc3e
xenpaging: improve policy mru list handling

Without this change it is not possible to page-out all guest pages, then
trigger a page-in for all pages, and then page-out everything once
again. All pages in the mru list can not be paged out because they
remain active in the internal bitmap of paged pages.

Use the mru list only if the number of paged-out pages is larger than
the mru list. If the number is smaller, start to clear the mru list. In
case the number of paged-out pages drops to zero the mru list and the
internal bitmap will be empty as well.

Also add a new interface for dropped pages. If a gfn was dropped there
is no need to adjust the mru list because dropping a page is not usage
of a page.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r deb016f351e5 -r ba84a323f2df tools/xenpaging/policy.h
--- a/tools/xenpaging/policy.h
+++ b/tools/xenpaging/policy.h
@@ -32,6 +32,8 @@ int policy_init(xenpaging_t *paging);
 int policy_choose_victim(xenpaging_t *paging, xenpaging_victim_t *victim);
 void policy_notify_paged_out(unsigned long gfn);
 void policy_notify_paged_in(unsigned long gfn);
+void policy_notify_paged_in_nomru(unsigned long gfn);
+void policy_notify_dropped(unsigned long gfn);
 
 #endif // __XEN_PAGING_POLICY_H__
 
diff -r deb016f351e5 -r ba84a323f2df tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -57,7 +57,7 @@ int policy_init(xenpaging_t *paging)
     if ( paging->policy_mru_size > 0 )
         mru_size = paging->policy_mru_size;
     else
-        mru_size = DEFAULT_MRU_SIZE;
+        mru_size = paging->policy_mru_size = DEFAULT_MRU_SIZE;
 
     mru = malloc(sizeof(*mru) * mru_size);
     if ( mru == NULL )
@@ -120,17 +120,38 @@ void policy_notify_paged_out(unsigned lo
     clear_bit(gfn, unconsumed);
 }
 
-void policy_notify_paged_in(unsigned long gfn)
+static void policy_handle_paged_in(unsigned long gfn, int do_mru)
 {
     unsigned long old_gfn = mru[i_mru & (mru_size - 1)];
 
     if ( old_gfn != INVALID_MFN )
         clear_bit(old_gfn, bitmap);
     
-    mru[i_mru & (mru_size - 1)] = gfn;
+    if (do_mru) {
+        mru[i_mru & (mru_size - 1)] = gfn;
+    } else {
+        clear_bit(gfn, bitmap);
+        mru[i_mru & (mru_size - 1)] = INVALID_MFN;
+    }
+
     i_mru++;
 }
 
+void policy_notify_paged_in(unsigned long gfn)
+{
+    policy_handle_paged_in(gfn, 1);
+}
+
+void policy_notify_paged_in_nomru(unsigned long gfn)
+{
+    policy_handle_paged_in(gfn, 0);
+}
+
+void policy_notify_dropped(unsigned long gfn)
+{
+    clear_bit(gfn, bitmap);
+}
+
 
 /*
  * Local variables:
diff -r deb016f351e5 -r ba84a323f2df tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -615,7 +615,14 @@ static int xenpaging_resume_page(xenpagi
     /* Notify policy of page being paged in */
     if ( notify_policy )
     {
-        policy_notify_paged_in(rsp->gfn);
+        /*
+         * Do not add gfn to mru list if the target is lower than mru size.
+         * This allows page-out of these gfns if the target grows again.
+         */
+        if (paging->num_paged_out > paging->policy_mru_size)
+            policy_notify_paged_in(rsp->gfn);
+        else
+            policy_notify_paged_in_nomru(rsp->gfn);
 
        /* Record number of resumed pages */
        paging->num_paged_out--;
@@ -869,7 +876,7 @@ int main(int argc, char *argv[])
                 {
                     DPRINTF("drop_page ^ gfn %"PRIx64" pageslot %d\n", req.gfn, i);
                     /* Notify policy of page being dropped */
-                    policy_notify_paged_in(req.gfn);
+                    policy_notify_dropped(req.gfn);
                 }
                 else
                 {

[PATCH 19 of 20] xenpaging: add debug to show received watch event

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321809975 -3600
# Node ID 47fd37efc3738f6c75af73ccbed0e87eb12ab31f
# Parent  ba84a323f2dfbdfd2430d3897e616a0e1dd1e4d7
xenpaging: add debug to show received watch event.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r ba84a323f2df -r 47fd37efc373 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -108,6 +108,7 @@ static int xenpaging_wait_for_event_or_t
         vec = xs_read_watch(paging->xs_handle, &num);
         if ( vec )
         {
+            DPRINTF("path '%s' token '%s'\n", vec[XS_WATCH_PATH], vec[XS_WATCH_TOKEN]);
             if ( strcmp(vec[XS_WATCH_TOKEN], watch_token) == 0 )
             {
                 /* If our guest disappeared, set interrupt flag and fall through */

[PATCH 20 of 20] xenpaging: restrict pagefile permissions

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1321809976 -3600
# Node ID 9f0156c725381ee25d96637aa5b486443b2641e5
# Parent  47fd37efc3738f6c75af73ccbed0e87eb12ab31f
xenpaging: restrict pagefile permissions

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 47fd37efc373 -r 9f0156c72538 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -795,7 +795,7 @@ int main(int argc, char *argv[])
     xc_interface *xch;
 
     int open_flags = O_CREAT | O_TRUNC | O_RDWR;
-    mode_t open_mode = S_IRUSR | S_IRGRP | S_IROTH | S_IWUSR | S_IWGRP | S_IWOTH;
+    mode_t open_mode = S_IRUSR | S_IWUSR;
     int fd;
 
     /* Initialise domain paging */

Re: [PATCH 00 of 20] tools/xenpaging changes

[Ian Jackson]

Olaf Hering writes ("[Xen-devel] [PATCH 00 of 20] tools/xenpaging changes"):
> The following changes contains non-controversial changes for xenpaging.
> They were sent out twice earlier already.

Thanks.  Sorry for the delay.  I have been very busy moving all the
Xen.org services out of our old colo; that task suddenly became
critical and unfortunately members of the Xen.org team including
myself had to pick it up.  So sorry to you and to everyone else who
has been waiting for patches to go in.

I have been catching up on the list in the past few days and applying
small and obvious patches.  With the larger series I have been waiting
until I was fully caught up, to avoid applying superseded versions,
etc.

Anyway, I have now applied all 20 of your patches.

To others who are waiting: I hope to start clearing the larger queues
from the backlog over the next few days.

Thanks,
Ian.