[PATCH] remus: proper cleanup on checkpoint failure

[PATCH] remus: proper cleanup on checkpoint failure

[Shriram Rajagopalan]

# HG changeset patch
# User Shriram Rajagopalan <rshriram@cs.ubc.ca>
# Date 1302204999 25200
# Node ID a73514445065390ae70c44e1708971dd6fa2a6f0
# Parent  97763efc41f9b664cf6f7db653c9c3f51e50b358
remus: proper cleanup on checkpoint failure.

While running remus, when an error occurs during checkpointing
(e.g., timeouts on primary, failing to checkpoint network buffer
or disk or even communication failure) the domU is sometimes
left in suspended state on primary. Instead of blindly closing
the checkpoint file handle, attempt to resume the domain before
the close.

Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>

diff -r 97763efc41f9 -r a73514445065 tools/python/xen/lowlevel/checkpoint/checkpoint.c
--- a/tools/python/xen/lowlevel/checkpoint/checkpoint.c	Tue Apr 05 18:23:54 2011 +0100
+++ b/tools/python/xen/lowlevel/checkpoint/checkpoint.c	Thu Apr 07 12:36:39 2011 -0700
@@ -80,6 +80,9 @@
 {
   CheckpointObject* self = (CheckpointObject*)obj;
 
+  if (checkpoint_resume(&self->cps) < 0)
+    fprintf(stderr, "%s\n", checkpoint_error(&self->cps));
+
   checkpoint_close(&self->cps);
 
   Py_XDECREF(self->suspend_cb);
diff -r 97763efc41f9 -r a73514445065 tools/python/xen/remus/save.py
--- a/tools/python/xen/remus/save.py	Tue Apr 05 18:23:54 2011 +0100
+++ b/tools/python/xen/remus/save.py	Thu Apr 07 12:36:39 2011 -0700
@@ -158,9 +158,13 @@
             self.checkpointer.open(self.vm.domid)
             self.checkpointer.start(self.fd, self.suspendcb, self.resumecb,
                                     self.checkpointcb, self.interval)
-            self.checkpointer.close()
         except xen.lowlevel.checkpoint.error, e:
             raise CheckpointError(e)
+        finally:
+            try: #errors in checkpoint close are not critical atm.
+                self.checkpointer.close()
+            except:
+                pass
 
     def _resume(self):
         """low-overhead version of XendDomainInfo.resumeDomain"""

[SPAM] [PATCH] remus: blackhole replication target

[Shriram Rajagopalan]

# HG changeset patch
# User Shriram Rajagopalan <rshriram@cs.ubc.ca>
# Date 1302206497 25200
# Node ID d7b7aef658dd962e13d29c851a28fde89b690e2e
# Parent  a73514445065390ae70c44e1708971dd6fa2a6f0
remus: blackhole replication target

The new --null option allows one to test and play with just the
memory checkpointing and network buffering aspect of remus, without
the need for a second host. The disk is not replicated. All replication
data is sent to /dev/null. This option is pretty handy when a user
wants to see the page churn for his workload or observe the latency hit
though the latter will not be accurate.

Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>

diff -r a73514445065 -r d7b7aef658dd tools/python/xen/remus/save.py
--- a/tools/python/xen/remus/save.py	Thu Apr 07 12:36:39 2011 -0700
+++ b/tools/python/xen/remus/save.py	Thu Apr 07 13:01:37 2011 -0700
@@ -95,6 +95,12 @@
         self.sock = sock
         super(MigrationSocket, self).__init__(fd)
 
+class NullSocket(_proxy):
+    def __init__(self, address):
+        filedesc = os.open('/dev/null', 0666)
+        fd = os.fdopen(filedesc, 'w+')
+        super(NullSocket, self).__init__(fd)
+
 class Keepalive(object):
     "Call a keepalive method at intervals"
     def __init__(self, method, interval=0.1):
diff -r a73514445065 -r d7b7aef658dd tools/remus/remus
--- a/tools/remus/remus	Thu Apr 07 12:36:39 2011 -0700
+++ b/tools/remus/remus	Thu Apr 07 13:01:37 2011 -0700
@@ -21,6 +21,7 @@
         self.domid = 0
 
         self.host = 'localhost'
+        self.nullremus = False
         self.port = XendOptions.instance().get_xend_relocation_port()
         self.interval = 200
         self.netbuffer = True
@@ -33,6 +34,8 @@
                           help='checkpoint every MS milliseconds')
         parser.add_option('-p', '--port', dest='port', type='int',
                           help='send stream to port PORT', metavar='PORT')
+        parser.add_option('', '--blackhole', dest='nullremus', action='store_true',
+                          help='replicate to /dev/null (no disk checkpoints, only memory & net buffering)')
         parser.add_option('', '--no-net', dest='nonet', action='store_true',
                           help='run without net buffering (benchmark option)')
         parser.add_option('', '--timer', dest='timer', action='store_true',
@@ -49,6 +52,8 @@
             self.interval = opts.interval
         if opts.port:
             self.port = opts.port
+        if opts.nullremus:
+            self.nullremus = True
         if opts.nonet:
             self.netbuffer = False
         if opts.timer:
@@ -107,18 +112,22 @@
     bufs = []
 
     # disks must commit before network can be released
-    for disk in dom.disks:
-        try:
-            bufs.append(ReplicatedDisk(disk))
-        except ReplicatedDiskException, e:
-            print e
-            continue
+    if not cfg.nullremus:
+        for disk in dom.disks:
+            try:
+                bufs.append(ReplicatedDisk(disk))
+            except ReplicatedDiskException, e:
+                print e
+                continue
 
     if cfg.netbuffer:
         for vif in dom.vifs:
             bufs.append(BufferedNIC(vif))
 
-    fd = save.MigrationSocket((cfg.host, cfg.port))
+    if cfg.nullremus:
+        fd = save.NullSocket((cfg.host, cfg.port))
+    else:
+        fd = save.MigrationSocket((cfg.host, cfg.port))
 
     def postsuspend():
         'Begin external checkpointing after domain has paused'

Re: [PATCH] remus: proper cleanup on checkpoint failure [and 1 more messages]

[Ian Jackson]

Shriram Rajagopalan writes ("[Xen-devel] [PATCH] remus: proper cleanup on checkpoint failure"):
> remus: proper cleanup on checkpoint failure.

Shriram Rajagopalan writes ("[Xen-devel] [SPAM]  [PATCH] remus: blackhole replication target"):
> remus: blackhole replication target

Thanks, I have applied both.

Ian.