RFC: making the xen startup integrate better with distros

RFC: making the xen startup integrate better with distros

[Daniel P. Berrange]

[-- Attachment #1: Type: text/plain, Size: 5013 bytes --]

What follows below is an outline of what we're changing in Xen startup
for Fedora 8. I know historically when getting into initscripts you tend
to hit distro specific issues. So I'm posting this to see if there's any
interest from others in having these changes integrated upstream. I'd
like to thing that with a few minor tweaks this is general enough to be 
useful to other Linux distros at least - the init script is basically
making use of LSB defined functions...


The way xen init scripts currently work is that they basically call out
to /usr/bin/xend which re-implements ideas already provided to all init
scripts via /etc/init.d/functions. Only the re-implementation is in python
and is slower & doesn't integrate well with standard distro configuration
procedure. eg we've no way to config stuff in /etc/sysconfig/xend to give
users flexibility. The plan is thus:

 * The auto-restart angel process should go since it merely serves to
   fork-bomb Xend when something goes wrong during startup. We've never
   seen XenD crash anyway so its not really buying us much except more
   complexity. Its also not monitoring the other Xen related daemons
   which have have seen crash! No other system daemons typically have
   auto crash restart built it - there are plenty of monitoring tools
   which add this system wide.

 * The individual daemons should each be started from the init scripts
   directly. User can see exactly what's started & what's running. We
   can then easily pass custom command line args to each one.

 * Drop a file in /etc/sysconfig/xen allowing user to customize the
   startup options for individual daemons.

 * Drop a file in /etc/sysconfig/modules/xen.modules to trigger loading
   of the backend driver modules.


So we have a config file  in /etc/sysconfig/modules containing user config
options

  [root@celery console]# cat /etc/sysconfig/xend
  #XENSTORED_PID="/var/run/xenstore.pid"
  #XENSTORED_ARGS=

  # Log all hypervisor messages (cf xm dmesg)
  #XENCONSOLED_LOG_HYPERVISOR=yes

  # Log all guest console output (cf xm console)
  #XENCONSOLED_LOG_GUESTS=yes

  # Location to store guest & hypervisor logs
  #XENCONSOLED_LOG_DIR=/var/log/xen/console

  #XENCONSOLED_ARGS=

  #BLKTAPCTRL_ARGS=

(This would be /etc/default/xend instead on Debian I believe)

When starting, we run all four currently provided Xen daemon processes,
list out all daemons being started along the way...

  [root@celery console]# service xend start
  Starting xen daemons: xenstored blktapctrl xenconsoled xend[  OK  ]

And you can see they're each now running

  [root@celery console]# service xend status
  xenstored (pid 2924) is running...
  blktapctrl (pid 2934) is running...
  xenconsoled (pid 32595) is running...
  xend (pid  32599) is running...
                                                           [  OK  ]

See the processes running and that they picked up config from settings
the user provided in /etc/sysconfig/xend

root      2924  1.0  0.0   8412   988 ?        S    May24 105:31 xenstored --pid-file /var/run/xenstore.pid
root     32595  0.0  0.0  20516   632 ?        Sl   18:11   0:00 /usr/sbin/xenconsoled --log=all --log-dir=/var/log/xen/console
root     32599  4.9  0.3 346952 13732 ?        Sl   18:11   0:09 python /usr/sbin/xend


We can also trigger a reload by directly sending each reloadable daemon
a SIGHUP. This makes xend re-read its config file, and make xenconsoled 
re-open its logfiles

  [root@celery console]# service xend reload
  Reloading xen daemons: xenconsoled xend                    [  OK  ]

Both Xend & XenConsoleD are capable of being stopped & restarted. It isn't
safe to stop blktapctrl or xenstored though. So init script only stops
the former two processes. eg:

  [root@celery console]# service xend stop
  Stopping xen daemons: xenconsoled xend                     [  OK  ]
 
Looking again at status of individual daemons, notice how we only stopped
xend & xenconsoled:
 
  [root@celery console]# service xend status
  xenstored (pid 2924) is running...
  blktapctrl (pid 2934) is running...
  xenconsoled is stopped
  xend is stopped
                                                           [FAILED]
 

The attached patch shows the changes to the init script to do this, as well
as killing off most of tools/misc/xend (what becomes /usr/sbin/xend). It
could probably go a bit further a kill off alot of stuff in the python
class tools/python/xen/xend/server/SrvDaemon.py since much of that isn't
needed now that its leveraging standard LSB init functions to do most of
the PID file handling for shutdown, reload, status commands.

   Signed-off-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

[-- Attachment #2: xen-initscript.patch --]
[-- Type: text/plain, Size: 8877 bytes --]

diff -rup xen-3.1.0-src/tools/examples/init.d/xend xen-3.1.0-src.new/tools/examples/init.d/xend
--- xen-3.1.0-src/tools/examples/init.d/xend	2007-05-31 12:14:19.000000000 -0400
+++ xen-3.1.0-src.new/tools/examples/init.d/xend	2007-05-31 17:53:29.000000000 -0400
@@ -19,48 +19,161 @@
 # Description:       Starts and stops the Xen control daemon.
 ### END INIT INFO
 
+# Source function library.
+. /etc/rc.d/init.d/functions
+
+if [ ! -d /proc/xen ]; then
+	exit 0
+fi
 if ! grep -q "control_d" /proc/xen/capabilities ; then
 	exit 0
 fi
 
-# Wait for Xend to be up
-function await_daemons_up
-{
-	i=1
-	rets=10
-	xend status
-	while [ $? -ne 0 -a $i -lt $rets ]; do
-	    sleep 1
-	    echo -n .
-	    i=$(($i + 1))
-	    xend status
-	done
+# Default config params
+XENSTORED_PID="/var/run/xenstore.pid"
+XENSTORED_ARGS=
+
+XENCONSOLED_LOG_HYPERVISOR=yes
+XENCONSOLED_LOG_GUESTS=yes
+XENCONSOLED_LOG_DIR=/var/log/xen/console
+XENCONSOLED_ARGS=
+
+BLKTAPCTRL_ARGS=
+
+# User customized params
+test -f /etc/sysconfig/xend && . /etc/sysconfig/xend
+
+XENCONSOLED_LOG=none
+if [ "$XENCONSOLED_LOG_HYPERVISOR" = "yes" ]
+then
+        if [ "$XENCONSOLED_LOG_GUESTS" = "yes" ]
+        then
+                XENCONSOLED_LOG=all
+        else
+                XENCONSOLED_LOG=hv
+        fi
+else
+        if [ "$XENCONSOLED_LOG_GUESTS" = "yes" ]
+        then
+                XENCONSOLED_LOG=guest
+        fi
+fi
+
+start() {
+	echo -n $"Starting xen daemons: "
+
+	echo -n "xenstored "
+        /usr/sbin/xenstored --pid-file $XENSTORED_PID $XENSTORED_ARGS
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	echo -n "blktapctrl "
+        /usr/sbin/blktapctrl $BLKTAPCTRL_ARGS
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	echo -n "xenconsoled "
+        /usr/sbin/xenconsoled --log=$XENCONSOLED_LOG --log-dir=$XENCONSOLED_LOG_DIR $XENCONSOLED_ARGS
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+        echo -n "xend"
+	/usr/sbin/xend
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	test $RETVAL = 0 && echo_success || echo_failure
+        echo
+        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/xend
 }
 
+stop() {
+	echo -n $"Stopping xen daemons: "
+
+	# NB not safe to stop xenstored, or blktapctrl if guests are active
+	# or backend driver modules are loaded, so we skip them
+
+	echo -n "xenconsoled "
+	killproc xenconsoled > /dev/null
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	echo -n "xend "
+	killproc xend > /dev/null
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	test $RETVAL = 0 && echo_success || echo_failure
+        echo
+        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/xend
+}
+
+rcstatus() {
+        status xenstored
+        rc=$?
+        test $rc != 0 && RETVAL=$rc
+
+        status blktapctrl
+        rc=$?
+        test $rc != 0 && RETVAL=$rc
+
+        status xenconsoled
+        rc=$?
+        test $rc != 0 && RETVAL=$rc
+
+        status xend
+        rc=$?
+        test $rc != 0 && RETVAL=$rc
+	test $RETVAL = 0 && echo_success || echo_failure
+	echo
+}
+
+reload() {
+	echo -n $"Reloading xen daemons: "
+
+	echo -n "xenconsoled "
+	killproc xenconsoled -HUP > /dev/null
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	echo -n "xend "
+	killproc xend -HUP > /dev/null
+	rc=$?
+	test $rc = 0 || RETVAL=$rc
+
+	test $RETVAL = 0 && echo_success || echo_failure
+        echo
+}
+
+RETVAL=0
 case "$1" in
   start)
-	xend start
-	await_daemons_up
+        start
 	;;
   stop)
-	xend stop
+        stop
 	;;
   status)
-	xend status
+        rcstatus
 	;;
   reload)
-        xend reload
+	reload
         ;;
   restart|force-reload)
-	xend restart
-	await_daemons_up
+	stop
+        start
 	;;
+  condrestart)
+        if [ -f /var/lock/subsys/xend ]
+        then
+                stop
+                start
+        fi
+        ;;
   *)
-	# do not advertise unreasonable commands that there is no reason
-	# to use with this device
-	echo $"Usage: $0 {start|stop|status|restart|reload|force-reload}"
+	echo $"Usage: $0 {start|stop|status|restart|condrestart|reload|force-reload}"
 	exit 1
 esac
 
-exit $?
+exit $RETVAL
 
diff -rup xen-3.1.0-src/tools/misc/xend xen-3.1.0-src.new/tools/misc/xend
--- xen-3.1.0-src/tools/misc/xend	2007-05-31 13:10:49.000000000 -0400
+++ xen-3.1.0-src.new/tools/misc/xend	2007-05-31 13:44:14.000000000 -0400
@@ -8,123 +8,16 @@
 """Xen management daemon.
    Provides console server and HTTP management api.
 
-   Run:
-   xend start
-
-   Restart:
-   xend restart
-
-   The daemon is stopped with:
-   xend stop
-
    The daemon should reconnect to device control interfaces
    and recover its state when restarted.
-
-   On Solaris, the daemons are SMF managed, and you should not attempt
-   to start xend by hand.
 """
-import os
-import os.path
 import sys
-import socket
-import signal
-import time
-import commands
-
-result = commands.getstatusoutput(os.path.join(os.path.dirname(sys.argv[0]),
-                                               'xen-python-path'))
-if result[0] != 0:
-    print >>sys.stderr, result[1]
-    sys.exit(1)
-
-sys.path.append(result[1])
 
 from xen.xend.server import SrvDaemon
 
-class CheckError(ValueError):
-    pass
-
-def hline():
-    print >>sys.stderr, "*" * 70
-
-def msg(message):
-    print >>sys.stderr, "*" * 3, message
-
-def check_logging():
-    """Check python logging is installed and raise an error if not.
-    Logging is standard from Python 2.3 on.
-    """
-    try:
-        import logging
-    except ImportError:
-        hline()
-        msg("Python logging is not installed.")
-        msg("Use 'make install-logging' at the xen root to install.")
-        msg("")
-        msg("Alternatively download and install from")
-        msg("http://www.red-dove.com/python_logging.html")
-        hline()
-        raise CheckError("logging is not installed")
-
-def check_user():
-    """Check that the effective user id is 0 (root).
-    """
-    if os.geteuid() != 0:
-        hline()
-        msg("Xend must be run as root.")
-        hline()
-        raise CheckError("invalid user")
-
-def start_xenstored():
-    XENSTORED_TRACE = os.getenv("XENSTORED_TRACE")
-    cmd = "xenstored --pid-file /var/run/xenstore.pid"
-    if XENSTORED_TRACE:
-        cmd += " -T /var/log/xen/xenstored-trace.log"
-    s,o = commands.getstatusoutput(cmd)
-
-def start_consoled():
-    if os.fork() == 0:
-        os.execvp('xenconsoled', ['xenconsoled'])
-
-def start_blktapctrl():
-    if os.fork() == 0:
-        os.execvp('blktapctrl', ['blktapctrl'])
-            
 def main():
-    try:
-        check_logging()
-        check_user()
-    except CheckError:
-        sys.exit(1)
-    
     daemon = SrvDaemon.instance()
-    if not sys.argv[1:]:
-        print 'usage: %s {start|stop|reload|restart}' % sys.argv[0]
-    elif sys.argv[1] == 'start':
-        if os.uname()[0] != "SunOS":
-            start_xenstored()
-            start_consoled()
-            start_blktapctrl()
-        return daemon.start()
-    elif sys.argv[1] == 'trace_start':
-        start_xenstored()
-        start_consoled()
-        start_blktapctrl()
-        return daemon.start(trace=1)
-    elif sys.argv[1] == 'stop':
-        return daemon.stop()
-    elif sys.argv[1] == 'reload':
-        return daemon.reloadConfig()
-    elif sys.argv[1] == 'restart':
-        start_xenstored()
-        start_consoled()
-        start_blktapctrl()
-        return daemon.stop() or daemon.start()
-    elif sys.argv[1] == 'status':
-        return daemon.status()
-    else:
-        print 'not an option:', sys.argv[1]
-    return 1
+    return daemon.start()
 
 if __name__ == '__main__':
     sys.exit(main())
diff -rup xen-3.1.0-src/tools/python/xen/xend/osdep.py xen-3.1.0-src.new/tools/python/xen/xend/osdep.py
--- xen-3.1.0-src/tools/python/xen/xend/osdep.py	2007-05-18 10:45:21.000000000 -0400
+++ xen-3.1.0-src.new/tools/python/xen/xend/osdep.py	2007-05-31 13:13:07.000000000 -0400
@@ -25,7 +25,7 @@ _scripts_dir = {
 }
 
 _xend_autorestart = {
-    "Linux": True,
+    "Linux": False,
     "SunOS": False,
 }
 
diff -rup xen-3.1.0-src/tools/python/xen/xend/server/SrvDaemon.py xen-3.1.0-src.new/tools/python/xen/xend/server/SrvDaemon.py
--- xen-3.1.0-src/tools/python/xen/xend/server/SrvDaemon.py	2007-05-18 10:45:21.000000000 -0400
+++ xen-3.1.0-src.new/tools/python/xen/xend/server/SrvDaemon.py	2007-05-31 14:36:23.000000000 -0400
@@ -109,7 +109,14 @@ class Daemon:
         # Fork, this allows the group leader to exit,
         # which means the child can never again regain control of the
         # terminal
-        if os.fork():
+        child = os.fork()
+        if child:
+            if not osdep.xend_autorestart:
+                pidfile = open(XEND_PID_FILE, 'w')
+                try:
+                    pidfile.write(str(child))
+                finally:
+                    pidfile.close()
             os._exit(0)
 
         # Detach from standard file descriptors, and redirect them to

[-- Attachment #3: xen.modules --]
[-- Type: text/plain, Size: 182 bytes --]

#!/bin/sh

test -f /proc/xen/capabilities || exit 0
grep -q "control_d" /proc/xen/capabilities || exit 0

for m in blkbk netbk blktap xenblktap
do
  modprobe $m >/dev/null 2>&1
done

[-- Attachment #4: xen.sysconfig --]
[-- Type: text/plain, Size: 336 bytes --]


#XENSTORED_PID="/var/run/xenstore.pid"
#XENSTORED_ARGS=

# Log all hypervisor messages (cf xm dmesg)
#XENCONSOLED_LOG_HYPERVISOR=yes

# Log all guest console output (cf xm console)
#XENCONSOLED_LOG_GUESTS=yes

# Location to store guest & hypervisor logs
#XENCONSOLED_LOG_DIR=/var/log/xen/console

#XENCONSOLED_ARGS=

#BLKTAPCTRL_ARGS=

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: RFC: making the xen startup integrate better with distros

[Nate Carlson]

On Fri, 1 Jun 2007, Daniel P. Berrange wrote:
> What follows below is an outline of what we're changing in Xen startup 
> for Fedora 8. I know historically when getting into initscripts you tend 
> to hit distro specific issues. So I'm posting this to see if there's any 
> interest from others in having these changes integrated upstream. I'd 
> like to thing that with a few minor tweaks this is general enough to be 
> useful to other Linux distros at least - the init script is basically 
> making use of LSB defined functions...

Sounds like an excellent idea. Debian and Ubuntu don't ship with a 
'functions' file, but I think this could easily be made to work on both.

> * The individual daemons should each be started from the init scripts
>   directly. User can see exactly what's started & what's running. We
>   can then easily pass custom command line args to each one.

Agreed. The other advantage of this is that it would be easy to launch 
said programs from a daemon monitoring app (ie, daemontools, etc).

> * Drop a file in /etc/sysconfig/xen allowing user to customize the
>   startup options for individual daemons.

Like you said, also /etc/default/xen on Debian. Debian does already have 
an /etc/default/xendomains which control the way that domains are 
started/stopped.

> We can also trigger a reload by directly sending each reloadable daemon 
> a SIGHUP. This makes xend re-read its config file, and make xenconsoled 
> re-open its logfiles

Beautiful!

> The attached patch shows the changes to the init script to do this, as 
> well as killing off most of tools/misc/xend (what becomes 
> /usr/sbin/xend). It could probably go a bit further a kill off alot of 
> stuff in the python class tools/python/xen/xend/server/SrvDaemon.py 
> since much of that isn't needed now that its leveraging standard LSB 
> init functions to do most of the PID file handling for shutdown, reload, 
> status commands.

The Debian packaging for Xen also includes a pile of patches to allow 
python paths and such to be more configurable - this is because Debian 
allows you to have multiple versions of the tools and hypervisor 
installed. It'd be really nice to see these integrated upstream; they are 
the primary thing that takes awhile to patch up with each new Xen release 
to get Debian packages built. If you're interested in seeing what they 
are, I can post them, or convince one of the regular Deb developers to 
upload them.  :)

------------------------------------------------------------------------
| nate carlson | natecars@natecarlson.com | http://www.natecarlson.com |
|       depriving some poor village of its idiot since 1981            |
------------------------------------------------------------------------