[PATCH] Recover from corrupt tdb on reboot

[Markus Armbruster <armbru@redhat.com>]

This patch was created for 3.1.0.  Based on inspection, I believe the
current version has the same problem, but I did not actually try it.
I hope you'll find the patch useful anyway.

Xen cannot work when xenstored's tdb is corrupt.  When that happens
somehow (and we've seen it happen), even reboot doesn't recover from
it.  It could: there is no state in tdb that needs to be persisted
across reboots.

The appended patch arranges that tdb is removed before xenstored is
started, provided it doesn't already run.  This is safe, because:

* xenstored cannot be restarted.  If it dies, Xen's screwed until
  reboot.

* /usr/sbin/xend always starts xenstored anyway.

* xenstored locks its pid-file (see write_pidfile() in
  tools/xenstore/xenstored_core.c), and refuses to start when it
  can't.

* My patch makes /usr/sbin/xend remove tdb iff it can lock the
  pid-file.  In other words, it removes tdb only when xenstored is not
  running, and locks it out until it is done.

  Bonus fix: it also removes stale copies of the tdb xenstored tends
  to leave behind when it exits uncleanly.

Signed-off-by: Markus Armbruster <armbru@redhat.com>



diff -r 3191627e5ad6 tools/misc/xend
--- a/tools/misc/xend	Wed Oct 31 16:21:18 2007 +0000
+++ b/tools/misc/xend	Mon Nov 05 18:23:11 2007 +0100
@@ -23,6 +23,8 @@
    On Solaris, the daemons are SMF managed, and you should not attempt
    to start xend by hand.
 """
+import fcntl
+import glob
 import os
 import os.path
 import sys
@@ -76,6 +78,23 @@ def check_user():
         raise CheckError("invalid user")
 
 def start_xenstored():
+    pidfname = "/var/run/xenstore.pid"
+    try:
+        f = open(pidfname, "a")
+        try:
+            fcntl.lockf(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
+            rootdir = os.getenv("XENSTORED_ROOTDIR") or "/var/lib/xenstored"
+            for i in glob.glob(rootdir + "/tdb*"):
+                try:
+                    os.unlink(i)
+                except:
+                    pass
+            os.unlink(pidfname)
+        except:
+            pass
+        f.close()
+    except:
+        pass
     XENSTORED_TRACE = os.getenv("XENSTORED_TRACE")
     cmd = "xenstored --pid-file /var/run/xenstore.pid"
     if XENSTORED_TRACE: