[PATCH 0 of 4] Enable domain checkpointing via xm save --checkpoint

[PATCH 0 of 4] Enable domain checkpointing via xm save --checkpoint

[Brendan Cully]

Here's take two of xm save --checkpoint. It puts suspend_cancel into a
standalone elfnote. I noticed an off-by-one in my xc.c code to import
elfnotes when I did this; the fix is included here.

>From the last post:

The attached patches create a --checkpoint flag for xm save:
 
xc_domain_resume lets the caller alert the guest that it is being
resumed. Callers should not do this unless they know the guest
supports the operation.
 
When xend wants to resume a domain, it checks for the suspend_cancel
elfnote from the notes it has recorded from domain build or resume. If
the flag is present, it simply lets the guest handle resuming,
otherwise it tears down and rebuilds the domain and lets the guest act
as if it is resuming in a new domain.
 
A new suspend_cancel elfnote is added with value 1.

[PATCH 1 of 4] Fix off-by-one preventing the last elfnote from being read in xc.c

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1172644688 28800
# Node ID a7afd4050ce3201044ac8fec1b4469d758a0fc80
# Parent  1c5e6239a8d0381fdbf56d4926f986d7f0ec07c0
Fix off-by-one preventing the last elfnote from being read in xc.c.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r 1c5e6239a8d0 -r a7afd4050ce3 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c	Sun Feb 25 23:58:33 2007 -0600
+++ b/tools/python/xen/lowlevel/xc/xc.c	Tue Feb 27 22:38:08 2007 -0800
@@ -411,7 +411,7 @@ static PyObject *pyxc_linux_build(XcObje
 
     if (!(elfnote_dict = PyDict_New()))
 	goto out;
-    for (i = 0; i < XEN_ELFNOTE_MAX; i++) {
+    for (i = 0; i <= XEN_ELFNOTE_MAX; i++) {
 	switch (dom->parms.elf_notes[i].type) {
 	case XEN_ENT_NONE:
 	    continue;

Re: [PATCH 1 of 4] Fix off-by-one preventing the last elfnote from being read in xc.c

[Ian Campbell]

On Tue, 2007-02-27 at 22:38 -0800, Brendan Cully wrote:
> # HG changeset patch
> # User Brendan Cully <brendan@cs.ubc.ca>
> # Date 1172644688 28800
> # Node ID a7afd4050ce3201044ac8fec1b4469d758a0fc80
> # Parent  1c5e6239a8d0381fdbf56d4926f986d7f0ec07c0
> Fix off-by-one preventing the last elfnote from being read in xc.c.
> 
> Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>
> 
> diff -r 1c5e6239a8d0 -r a7afd4050ce3 tools/python/xen/lowlevel/xc/xc.c
> --- a/tools/python/xen/lowlevel/xc/xc.c	Sun Feb 25 23:58:33 2007 -0600
> +++ b/tools/python/xen/lowlevel/xc/xc.c	Tue Feb 27 22:38:08 2007 -0800
> @@ -411,7 +411,7 @@ static PyObject *pyxc_linux_build(XcObje
>  
>      if (!(elfnote_dict = PyDict_New()))
>  	goto out;
> -    for (i = 0; i < XEN_ELFNOTE_MAX; i++) {
> +    for (i = 0; i <= XEN_ELFNOTE_MAX; i++) {
>  	switch (dom->parms.elf_notes[i].type) {
>  	case XEN_ENT_NONE:
>  	    continue;

I think we should rename XEN_ELFNOTE_MAX to XEN_ELFNOTE_NR (and +1)
which would be clearer IMHO. The only other place it used is
"XEN_ELFNOTE_MAX + 1" anyway:

--- 
Change XEN_ELFNOTE_MAX to XEN_ELFNOTE_NR and make it the number of ELF
notes rather than the index of the last ELF note.

Signed-off-by: Ian Campbell <ian.campbell@xensource.com>
diff -r 5217185f7588 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c	Tue Feb 27 20:27:18 2007 +0000
+++ b/tools/python/xen/lowlevel/xc/xc.c	Wed Feb 28 08:59:41 2007 +0000
@@ -411,7 +411,7 @@ static PyObject *pyxc_linux_build(XcObje
 
     if (!(elfnote_dict = PyDict_New()))
 	goto out;
-    for (i = 0; i < XEN_ELFNOTE_MAX; i++) {
+    for (i = 0; i < XEN_ELFNOTE_NR; i++) {
 	switch (dom->parms.elf_notes[i].type) {
 	case XEN_ENT_NONE:
 	    continue;
diff -r 5217185f7588 xen/include/public/elfnote.h
--- a/xen/include/public/elfnote.h	Tue Feb 27 20:27:18 2007 +0000
+++ b/xen/include/public/elfnote.h	Wed Feb 28 09:03:30 2007 +0000
@@ -157,9 +157,9 @@
 #define XEN_ELFNOTE_L1_MFN_VALID  13
 
 /*
- * The number of the highest elfnote defined.
- */
-#define XEN_ELFNOTE_MAX XEN_ELFNOTE_L1_MFN_VALID
+ * The number of elfnotes defined.
+ */
+#define XEN_ELFNOTE_NR (XEN_ELFNOTE_L1_MFN_VALID + 1)
 
 /*
  * System information exported through crash notes.
diff -r 5217185f7588 xen/include/public/libelf.h
--- a/xen/include/public/libelf.h	Tue Feb 27 20:27:18 2007 +0000
+++ b/xen/include/public/libelf.h	Wed Feb 28 08:59:23 2007 +0000
@@ -195,7 +195,7 @@ struct elf_dom_parms {
     const char *guest_info;
     const void *elf_note_start;
     const void *elf_note_end;
-    struct xen_elfnote elf_notes[XEN_ELFNOTE_MAX + 1];
+    struct xen_elfnote elf_notes[XEN_ELFNOTE_NR];
   
     /* parsed */
     char guest_os[16];

Re: [PATCH 1 of 4] Fix off-by-one preventing the last elfnote from being read in xc.c

[Brendan Cully]

On Wednesday, 28 February 2007 at 09:06, Ian Campbell wrote:
> On Tue, 2007-02-27 at 22:38 -0800, Brendan Cully wrote:
> > # HG changeset patch
> > # User Brendan Cully <brendan@cs.ubc.ca>
> > # Date 1172644688 28800
> > # Node ID a7afd4050ce3201044ac8fec1b4469d758a0fc80
> > # Parent  1c5e6239a8d0381fdbf56d4926f986d7f0ec07c0
> > Fix off-by-one preventing the last elfnote from being read in xc.c.
> > 
> > Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>
> > 
> > diff -r 1c5e6239a8d0 -r a7afd4050ce3 tools/python/xen/lowlevel/xc/xc.c
> > +++ b/tools/python/xen/lowlevel/xc/xc.c	Tue Feb 27 22:38:08 2007 -0800
> > @@ -411,7 +411,7 @@ static PyObject *pyxc_linux_build(XcObje
> >  
> >      if (!(elfnote_dict = PyDict_New()))
> >  	goto out;
> > -    for (i = 0; i < XEN_ELFNOTE_MAX; i++) {
> > +    for (i = 0; i <= XEN_ELFNOTE_MAX; i++) {
> >  	switch (dom->parms.elf_notes[i].type) {
> >  	case XEN_ENT_NONE:
> >  	    continue;
> 
> I think we should rename XEN_ELFNOTE_MAX to XEN_ELFNOTE_NR (and +1)
> which would be clearer IMHO. The only other place it used is
> "XEN_ELFNOTE_MAX + 1" anyway:
> 
> Change XEN_ELFNOTE_MAX to XEN_ELFNOTE_NR and make it the number of ELF
> notes rather than the index of the last ELF note.

Sounds good to me.

[PATCH 2 of 4] Let xend choose to do hard or soft domain resumption depending on

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1172644688 28800
# Node ID 6b127078519f0c318b0d212ce901491eb1c155c7
# Parent  a7afd4050ce3201044ac8fec1b4469d758a0fc80
Let xend choose to do hard or soft domain resumption depending on
whether the domain advertises support for soft resumption in its
elfnotes.

Signed-off-by: Brendan Cully <brendan@kublai.com>

diff -r a7afd4050ce3 -r 6b127078519f tools/libxc/xc_resume.c
--- a/tools/libxc/xc_resume.c	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/libxc/xc_resume.c	Tue Feb 27 22:38:08 2007 -0800
@@ -169,13 +169,9 @@ static int xc_domain_resume_any(int xc_h
  * (2) should be used only for guests which cannot handle the special
  * new return code. (1) is always safe (but slower).
  */
-int xc_domain_resume(int xc_handle, uint32_t domid)
+int xc_domain_resume(int xc_handle, uint32_t domid, int fast)
 {
-    /*
-     * XXX: Implement a way to select between options (1) and (2).
-     * Or expose the options as two different methods to Python.
-     */
-    return (0
+    return (fast
             ? xc_domain_resume_cooperative(xc_handle, domid)
             : xc_domain_resume_any(xc_handle, domid));
 }
diff -r a7afd4050ce3 -r 6b127078519f tools/libxc/xenctrl.h
--- a/tools/libxc/xenctrl.h	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/libxc/xenctrl.h	Tue Feb 27 22:38:08 2007 -0800
@@ -243,10 +243,12 @@ int xc_domain_destroy(int xc_handle,
  *
  * @parm xc_handle a handle to an open hypervisor interface
  * @parm domid the domain id to resume
+ * @parm fast use cooperative resume (guest must support this)
  * return 0 on success, -1 on failure
  */
 int xc_domain_resume(int xc_handle,
-                      uint32_t domid);
+		     uint32_t domid,
+		     int fast);
 
 /**
  * This function will shutdown a domain. This is intended for use in
diff -r a7afd4050ce3 -r 6b127078519f tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/python/xen/lowlevel/xc/xc.c	Tue Feb 27 22:38:08 2007 -0800
@@ -178,7 +178,17 @@ static PyObject *pyxc_domain_shutdown(Xc
 
 static PyObject *pyxc_domain_resume(XcObject *self, PyObject *args)
 {
-    return dom_op(self, args, xc_domain_resume);
+    uint32_t dom;
+    int fast;
+
+    if (!PyArg_ParseTuple(args, "ii", &dom, &fast))
+        return NULL;
+
+    if (xc_domain_resume(self->xc_handle, dom, fast) != 0)
+        return pyxc_error_to_exception();
+
+    Py_INCREF(zero);
+    return zero;
 }
 
 static PyObject *pyxc_vcpu_setaffinity(XcObject *self,
@@ -1124,7 +1134,8 @@ static PyMethodDef pyxc_methods[] = {
       (PyCFunction)pyxc_domain_resume,
       METH_VARARGS, "\n"
       "Resume execution of a suspended domain.\n"
-      " dom [int]: Identifier of domain to be resumed.\n\n"
+      " dom [int]: Identifier of domain to be resumed.\n"
+      " fast [int]: Use cooperative resume.\n\n"
       "Returns: [int] 0 on success; -1 on error.\n" },
 
     { "domain_shutdown", 
diff -r a7afd4050ce3 -r 6b127078519f tools/python/xen/xend/XendCheckpoint.py
--- a/tools/python/xen/xend/XendCheckpoint.py	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/python/xen/xend/XendCheckpoint.py	Tue Feb 27 22:38:08 2007 -0800
@@ -137,23 +137,6 @@ def save(fd, dominfo, network, live, dst
         log.exception("Save failed on domain %s (%s).", domain_name,
                       dominfo.getDomid())
 
-        dominfo._releaseDevices()
-        dominfo.testDeviceComplete()
-        dominfo.testvifsComplete()
-        log.debug("XendCheckpoint.save: devices released")
-
-        dominfo._resetChannels()
-
-        dominfo._removeDom('control/shutdown')
-        dominfo._removeDom('device-misc/vif/nextDeviceID')
-
-        dominfo._createChannels()
-        dominfo._introduceDomain()
-        dominfo._storeDomDetails()
-
-        dominfo._createDevices()
-        log.debug("XendCheckpoint.save: devices created")
-
         dominfo.resumeDomain()
         log.debug("XendCheckpoint.save: resumeDomain")
 
diff -r a7afd4050ce3 -r 6b127078519f tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/python/xen/xend/XendDomainInfo.py	Tue Feb 27 22:38:08 2007 -0800
@@ -1662,10 +1662,31 @@ class XendDomainInfo:
     def resumeDomain(self):
         log.debug("XendDomainInfo.resumeDomain(%s)", str(self.domid))
 
+        if self.domid is None:
+            return
         try:
-            if self.domid is not None:
-                xc.domain_resume(self.domid)
-                ResumeDomain(self.domid)
+            # could also fetch a parsed note from xenstore
+            fast = self.info.get_notes().get('SUSPEND_CANCEL') and 1 or 0
+            if not fast:
+                self._releaseDevices()
+                self.testDeviceComplete()
+                self.testvifsComplete()
+                log.debug("XendDomainInfo.resumeDomain: devices released")
+
+                self._resetChannels()
+
+                self._removeDom('control/shutdown')
+                self._removeDom('device-misc/vif/nextDeviceID')
+
+                self._createChannels()
+                self._introduceDomain()
+                self._storeDomDetails()
+
+                self._createDevices()
+                log.debug("XendDomainInfo.resumeDomain: devices created")
+
+            xc.domain_resume(self.domid, fast)
+            ResumeDomain(self.domid)
         except:
             log.exception("XendDomainInfo.resume: xc.domain_resume failed on domain %s." % (str(self.domid)))

[PATCH 3 of 4] Add xm save -c/--checkpoint option

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1172644688 28800
# Node ID c61bfd68ca422003ea6c0ed6e13e26dc881b98a8
# Parent  6b127078519f0c318b0d212ce901491eb1c155c7
Add xm save -c/--checkpoint option

xm save --checkpoint leaves the domain running after creating the
snapshot.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r 6b127078519f -r c61bfd68ca42 tools/python/xen/xend/XendCheckpoint.py
--- a/tools/python/xen/xend/XendCheckpoint.py	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/python/xen/xend/XendCheckpoint.py	Tue Feb 27 22:38:08 2007 -0800
@@ -54,7 +54,7 @@ def read_exact(fd, size, errmsg):
     return buf
 
 
-def save(fd, dominfo, network, live, dst):
+def save(fd, dominfo, network, live, dst, checkpoint=False):
     write_exact(fd, SIGNATURE, "could not write guest state file: signature")
 
     config = sxp.to_string(dominfo.sxpr())
@@ -121,9 +121,11 @@ def save(fd, dominfo, network, live, dst
             os.close(qemu_fd)
             os.remove("/tmp/xen.qemu-dm.%d" % dominfo.getDomid())
 
-        dominfo.destroyDomain()
-        dominfo.testDeviceComplete()
-
+        if checkpoint:
+            dominfo.resumeDomain()
+        else:
+            dominfo.destroyDomain()
+            dominfo.testDeviceComplete()
         try:
             dominfo.setName(domain_name)
         except VmError:
diff -r 6b127078519f -r c61bfd68ca42 tools/python/xen/xend/XendDomain.py
--- a/tools/python/xen/xend/XendDomain.py	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/python/xen/xend/XendDomain.py	Tue Feb 27 22:38:08 2007 -0800
@@ -1172,7 +1172,7 @@ class XendDomain:
         XendCheckpoint.save(sock.fileno(), dominfo, True, live, dst)
         sock.close()
 
-    def domain_save(self, domid, dst):
+    def domain_save(self, domid, dst, checkpoint):
         """Start saving a domain to file.
 
         @param domid: Domain ID or Name
@@ -1196,8 +1196,8 @@ class XendDomain:
                 oflags |= os.O_LARGEFILE
             fd = os.open(dst, oflags)
             try:
-                # For now we don't support 'live checkpoint' 
-                XendCheckpoint.save(fd, dominfo, False, False, dst)
+                XendCheckpoint.save(fd, dominfo, False, False, dst,
+                                    checkpoint=checkpoint)
             finally:
                 os.close(fd)
         except OSError, ex:
diff -r 6b127078519f -r c61bfd68ca42 tools/python/xen/xm/main.py
--- a/tools/python/xen/xm/main.py	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/python/xen/xm/main.py	Tue Feb 27 22:38:08 2007 -0800
@@ -102,7 +102,7 @@ SUBCOMMAND_HELP = {
     'reboot'      : ('<Domain> [-wa]', 'Reboot a domain.'),
     'restore'     : ('<CheckpointFile> [-p]',
                      'Restore a domain from a saved state.'),
-    'save'        : ('<Domain> <CheckpointFile>',
+    'save'        : ('[-c] <Domain> <CheckpointFile>',
                      'Save a domain state to restore later.'),
     'shutdown'    : ('<Domain> [-waRH]', 'Shutdown a domain.'),
     'top'         : ('', 'Monitor a host and the domains in real time.'),
@@ -232,6 +232,9 @@ SUBCOMMAND_OPTIONS = {
     'resume': (
       ('-p', '--paused', 'Do not unpause domain after resuming it'),
     ),
+    'save': (
+      ('-c', '--checkpoint', 'Leave domain running after creating snapshot'),
+    ),
    'restore': (
       ('-p', '--paused', 'Do not unpause domain after restoring it'),
     ),
@@ -604,21 +607,37 @@ def xm_shell(args):
 #########################################################################
 
 def xm_save(args):
-    arg_check(args, "save", 2)
-
-    try:
-        dominfo = parse_doms_info(server.xend.domain(args[0]))
+    arg_check(args, "save", 2, 3)
+
+    try:
+        (options, params) = getopt.gnu_getopt(args, 'c', ['checkpoint'])
+    except getopt.GetoptError, opterr:
+        err(opterr)
+        sys.exit(1)
+
+    checkpoint = False
+    for (k, v) in options:
+        if k in ['-c', '--checkpoint']:
+            checkpoint = True
+
+    if len(params) != 2:
+        err("Wrong number of parameters")
+        usage('save')
+        sys.exit(1)
+
+    try:
+        dominfo = parse_doms_info(server.xend.domain(params[0]))
     except xmlrpclib.Fault, ex:
         raise ex
     
     domid = dominfo['domid']
-    savefile = os.path.abspath(args[1])
+    savefile = os.path.abspath(params[1])
 
     if not os.access(os.path.dirname(savefile), os.W_OK):
         err("xm save: Unable to create file %s" % savefile)
         sys.exit(1)
     
-    server.xend.domain.save(domid, savefile)
+    server.xend.domain.save(domid, savefile, checkpoint)
     
 def xm_restore(args):
     arg_check(args, "restore", 1, 2)

[PATCH 4 of 4] Add suspend_cancel flag to linux elf notes

[Brendan Cully]

# HG changeset patch
# User Brendan Cully <brendan@cs.ubc.ca>
# Date 1172644689 28800
# Node ID c80136cd1ecfd5f351f48bb4d18b80f1a6fbfb5b
# Parent  c61bfd68ca422003ea6c0ed6e13e26dc881b98a8
Add suspend_cancel flag to linux elf notes.

Signed-off-by: Brendan Cully <brendan@cs.ubc.ca>

diff -r c61bfd68ca42 -r c80136cd1ecf linux-2.6-xen-sparse/arch/i386/kernel/head-xen.S
--- a/linux-2.6-xen-sparse/arch/i386/kernel/head-xen.S	Tue Feb 27 22:38:08 2007 -0800
+++ b/linux-2.6-xen-sparse/arch/i386/kernel/head-xen.S	Tue Feb 27 22:38:09 2007 -0800
@@ -206,3 +206,4 @@ ENTRY(cpu_gdt_table)
 	ELFNOTE(Xen, XEN_ELFNOTE_L1_MFN_VALID,   .long,  _PAGE_PRESENT,_PAGE_PRESENT)
 #endif
 	ELFNOTE(Xen, XEN_ELFNOTE_LOADER,         .asciz, "generic")
+	ELFNOTE(Xen, XEN_ELFNOTE_SUSPEND_CANCEL, .long,  1)
diff -r c61bfd68ca42 -r c80136cd1ecf linux-2.6-xen-sparse/arch/x86_64/kernel/head-xen.S
--- a/linux-2.6-xen-sparse/arch/x86_64/kernel/head-xen.S	Tue Feb 27 22:38:08 2007 -0800
+++ b/linux-2.6-xen-sparse/arch/x86_64/kernel/head-xen.S	Tue Feb 27 22:38:09 2007 -0800
@@ -186,3 +186,4 @@ ENTRY(empty_zero_page)
 	ELFNOTE(Xen, XEN_ELFNOTE_L1_MFN_VALID,   .quad,  _PAGE_PRESENT,_PAGE_PRESENT)
 	ELFNOTE(Xen, XEN_ELFNOTE_FEATURES,       .asciz, "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel")
 	ELFNOTE(Xen, XEN_ELFNOTE_LOADER,         .asciz, "generic")
+	ELFNOTE(Xen, XEN_ELFNOTE_SUSPEND_CANCEL, .long,  1)
diff -r c61bfd68ca42 -r c80136cd1ecf tools/xcutils/readnotes.c
--- a/tools/xcutils/readnotes.c	Tue Feb 27 22:38:08 2007 -0800
+++ b/tools/xcutils/readnotes.c	Tue Feb 27 22:38:09 2007 -0800
@@ -80,6 +80,9 @@ static int print_notes(struct elf_binary
 			break;
 		case XEN_ELFNOTE_HV_START_LOW:
 			print_numeric_note("HV_START_LOW", elf, note);
+			break;
+		case XEN_ELFNOTE_SUSPEND_CANCEL:
+			print_numeric_note("SUSPEND_CANCEL", elf, note);
 			break;
 		default:
 			printf("unknown note type %#x\n",
diff -r c61bfd68ca42 -r c80136cd1ecf xen/common/libelf/libelf-dominfo.c
--- a/xen/common/libelf/libelf-dominfo.c	Tue Feb 27 22:38:08 2007 -0800
+++ b/xen/common/libelf/libelf-dominfo.c	Tue Feb 27 22:38:09 2007 -0800
@@ -99,6 +99,7 @@ int elf_xen_parse_note(struct elf_binary
         [XEN_ELFNOTE_PAE_MODE] = { "PAE_MODE", 1},
         [XEN_ELFNOTE_FEATURES] = { "FEATURES", 1},
         [XEN_ELFNOTE_BSD_SYMTAB] = { "BSD_SYMTAB", 1},
+        [XEN_ELFNOTE_SUSPEND_CANCEL] = { "SUSPEND_CANCEL", 0 },
     };
 /* *INDENT-ON* */
 
diff -r c61bfd68ca42 -r c80136cd1ecf xen/include/public/elfnote.h
--- a/xen/include/public/elfnote.h	Tue Feb 27 22:38:08 2007 -0800
+++ b/xen/include/public/elfnote.h	Tue Feb 27 22:38:09 2007 -0800
@@ -157,9 +157,14 @@
 #define XEN_ELFNOTE_L1_MFN_VALID  13
 
 /*
+ * Whether or not the guest supports cooperative suspend cancellation.
+ */
+#define XEN_ELFNOTE_SUSPEND_CANCEL 14
+
+/*
  * The number of the highest elfnote defined.
  */
-#define XEN_ELFNOTE_MAX XEN_ELFNOTE_L1_MFN_VALID
+#define XEN_ELFNOTE_MAX XEN_ELFNOTE_SUSPEND_CANCEL
 
 /*
  * System information exported through crash notes.