[PATCH 0 of 5] Make checkpointing backwards compatible with older kernels

[PATCH 0 of 5] Make checkpointing backwards compatible with older kernels

[Brendan Cully]

This set of patches makes checkpointing backwards compatible with
older kernels. Checkpointable kernels now create a xenstore entry
called "control/feature-resumable" which xc save -c checks before
attempting to checkpoint a domain.

It also writes a new SIF_RESUMABLE flag into the suspend record,
which xc_linux_restore uses to decide whether it is safe to set the
hypercall return value. Although pre-checkpoint linux images ignore
the return value, I've heard that solaris bails out if the result is
anything but 0. Another possibly simpler strategy would be to flip the
return values, so 1 means the call is resuming in the source domain
and 0 means it is in a new domain and must reconnect. This is what
older kernels (that are only ever restored in a new domain) expect
anyway.

[PATCH 1 of 5] Advertise resumability feature in guest kernel

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1168642819 28800
# Node ID 34a989337172b3df09a8b10089d4471a026bb613
# Parent  8c14bd989fba459154b266396067a8dd1e1fe829
Advertise resumability feature in guest kernel.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r 8c14bd989fba -r 34a989337172 linux-2.6-xen-sparse/drivers/xen/core/reboot.c
--- a/linux-2.6-xen-sparse/drivers/xen/core/reboot.c	Fri Jan 12 15:00:19 2007 -0800
+++ b/linux-2.6-xen-sparse/drivers/xen/core/reboot.c	Fri Jan 12 15:00:19 2007 -0800
@@ -204,6 +204,8 @@ static int setup_shutdown_watcher(struct
 	else
 		xenbus_write(XBT_NIL, "control", "feature-sysrq", "1");
 
+	xenbus_write(XBT_NIL, "control", "feature-resumable", "1");
+
 	return NOTIFY_DONE;
 }

[PATCH 2 of 5] Disallow xm save -c unless the guest advertises that it is resumable

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1168642934 28800
# Node ID fda11d9e5241eccd3c103e9b6a42ea33dd236174
# Parent  34a989337172b3df09a8b10089d4471a026bb613
Disallow xm save -c unless the guest advertises that it is resumable.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r 34a989337172 -r fda11d9e5241 tools/python/xen/xend/XendDomain.py
--- a/tools/python/xen/xend/XendDomain.py	Fri Jan 12 15:00:19 2007 -0800
+++ b/tools/python/xen/xend/XendDomain.py	Fri Jan 12 15:02:14 2007 -0800
@@ -1196,6 +1196,10 @@ class XendDomain:
             if dominfo.getDomid() == DOM0_ID:
                 raise XendError("Cannot save privileged domain %i" % domid)
 
+            if checkpoint:
+                resumable = dominfo.readDom("control/feature-resumable")
+                if not resumable:
+                    raise XendError("Guest does not support checkpointing")
             fd = os.open(dst, os.O_WRONLY | os.O_CREAT | os.O_TRUNC)
             try:
                 XendCheckpoint.save(fd, dominfo, False, False, dst,

[PATCH 3 of 5] Add SIF_RESUMABLE flag to tell restore that a guest is resumable

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1168642944 28800
# Node ID ebdc01d032e8656c130ed945ce54f08108f42c73
# Parent  fda11d9e5241eccd3c103e9b6a42ea33dd236174
Add SIF_RESUMABLE flag to tell restore that a guest is resumable.

Unlike the other start_info flags, the flow of information here is
from the kernel to the tools.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r fda11d9e5241 -r ebdc01d032e8 xen/include/public/xen.h
--- a/xen/include/public/xen.h	Fri Jan 12 15:02:14 2007 -0800
+++ b/xen/include/public/xen.h	Fri Jan 12 15:02:24 2007 -0800
@@ -535,6 +535,7 @@ typedef struct start_info start_info_t;
 /* These flags are passed in the 'flags' field of start_info_t. */
 #define SIF_PRIVILEGED    (1<<0)  /* Is the domain privileged? */
 #define SIF_INITDOMAIN    (1<<1)  /* Is this the initial control domain? */
+#define SIF_RESUMABLE     (1<<2)  /* does the domain support checkpointing? */
 
 typedef struct dom0_vga_console_info {
     uint8_t video_type; /* DOM0_VGA_CONSOLE_??? */

[PATCH 4 of 5] Set SIF_RESUMABLE flag in suspend

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1168642944 28800
# Node ID 8c3139592f4d99d7ce1ca4e4e00792b00fc323b4
# Parent  ebdc01d032e8656c130ed945ce54f08108f42c73
Set SIF_RESUMABLE flag in suspend.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r ebdc01d032e8 -r 8c3139592f4d linux-2.6-xen-sparse/drivers/xen/core/machine_reboot.c
--- a/linux-2.6-xen-sparse/drivers/xen/core/machine_reboot.c	Fri Jan 12 15:02:24 2007 -0800
+++ b/linux-2.6-xen-sparse/drivers/xen/core/machine_reboot.c	Fri Jan 12 15:02:24 2007 -0800
@@ -83,6 +83,7 @@ static void pre_suspend(void)
 	xen_start_info->store_mfn = mfn_to_pfn(xen_start_info->store_mfn);
 	xen_start_info->console.domU.mfn =
 		mfn_to_pfn(xen_start_info->console.domU.mfn);
+	xen_start_info->flags |= SIF_RESUMABLE;
 }
 
 static void post_suspend(int reconnect)

[PATCH 5 of 5] Do not set suspend return value unless SIF_RESUMABLE is set

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1168642944 28800
# Node ID 4fc2fde1b3737554edd795e65701292d8320405d
# Parent  8c3139592f4d99d7ce1ca4e4e00792b00fc323b4
Do not set suspend return value unless SIF_RESUMABLE is set.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r 8c3139592f4d -r 4fc2fde1b373 tools/libxc/xc_linux_restore.c
--- a/tools/libxc/xc_linux_restore.c	Fri Jan 12 15:02:24 2007 -0800
+++ b/tools/libxc/xc_linux_restore.c	Fri Jan 12 15:02:24 2007 -0800
@@ -690,11 +690,13 @@ int xc_linux_restore(int xc_handle, int 
         ERROR("Suspend record frame number is bad");
         goto out;
     }
-    /* HYPERVISOR_suspend returns 1 to let guest know it should reconnect */
-    ctxt.user_regs.eax = 1;
     ctxt.user_regs.edx = mfn = p2m[pfn];
     start_info = xc_map_foreign_range(
         xc_handle, dom, PAGE_SIZE, PROT_READ | PROT_WRITE, mfn);
+    if (start_info->flags & SIF_RESUMABLE) {
+	/* HYPERVISOR_suspend returns 1 to let guest know it should reconnect */
+	ctxt.user_regs.eax = 1;
+    }
     start_info->nr_pages    = max_pfn;
     start_info->shared_info = shared_info_frame << PAGE_SHIFT;
     start_info->flags       = 0;

Re: [PATCH 0 of 5] Make checkpointing backwards compatible with older kernels

[Keir Fraser]

On 12/1/07 10:03 pm, "Brendan Cully" <brendan@cs.ubc.ca> wrote:

> This set of patches makes checkpointing backwards compatible with
> older kernels. Checkpointable kernels now create a xenstore entry
> called "control/feature-resumable" which xc save -c checks before
> attempting to checkpoint a domain.
> 
> It also writes a new SIF_RESUMABLE flag into the suspend record,
> which xc_linux_restore uses to decide whether it is safe to set the
> hypercall return value.

A single unified new elfnote would probably be better. We would like a new
libxc function to pull image metadata (inc. Elfnotes) out of a kernel image
and provide a comma-separated key-value list, turned into a dictionary for
Python consumption. This would then be persisted by xend (e.g., be added to
save/restore stream format) and can be used to decide whether 'xm save -c'
is okay, and would allow info to be passed to xc_save/xc_restore about
whether suspend is resumable. We already have another intended use for this
mechanism, which is to provide a pte mask to use to decide whether or not a
not-present pte contains an MFN (needing conversion to PFN on xc_save).
Unfortunately although this method is cleanest there is some tedious
plumbing to be done and I'm not sure how to represent the Elf notes in a
save image format (perhaps it can be merged into the image sxpr).

Yes -- returning 1 on resume (rather than restore) will allow new guest
kernels to run properly on old Xen and tools which do not understand resume,
so that would be the better way round.

 -- Keir

Re: [PATCH 0 of 5] Make checkpointing backwards compatible with older kernels

[Keir Fraser]

On 12/1/07 10:03 pm, "Brendan Cully" <brendan@cs.ubc.ca> wrote:

> This set of patches

You could have sent this one as a single patch. The previous patchset was
more useful to split up, but even there a smaller set would have done (e.g.,
merge patches which implement a hypercall, plumb it into libxc and plumb it
into python). Large patchsets make me think I have a depressing amount of
review work to do. :-)

 -- Keir