Re: Suggestion for merging xl save/restore/migrate/migrate-receive

[Zhigang Wang <zhigang.x.wang@oracle.com>]

On 09/16/2013 06:04 AM, George Dunlap wrote:
> On Fri, Sep 13, 2013 at 5:04 PM, Zhigang Wang <zhigang.x.wang@oracle.com> wrote:
>> Hi,
>>
>> As we talked in
>> http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a
>> suggestion: what about merging xl save/restore/migrate/migrate-receive?
>>
>> Here is the description: xl-migrate.rst
> 
> Thanks for bringing this up, but this thing with attaching files that
> actually contain your proposal isn't really going to work.  Can you
> please resend this with your proposals in-line, so that 1) the entire
> discussion can happen in the mail reader, rather than having to switch
> back and forth between an editor and a reader, 2) people can comment
> in-line on the proposals?
> 
> Thanks,
>  -George
> 

Thanks George for the comment. Here it is:

As we talked in
http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a
suggestion: what about merging xl save/restore/migrate/migrate-receive?

Here is the description: xl-migrate.rst

Also there is a workaround for the xl migration daemon example:
xl-migrate-socat.rst

Example patch for the workaround: xen-xl-migrate-socat.patch

(I don't have enough knowledge to implement the merge.)

Thanks,

Zhigang

---- xl-migrate.rst ----

==========
XL Migrate
==========

:Date: 2013-09-16

Current Status
==============

* xl migrate leverages ssh/sshd.

* In order to migrate a VM without user interactive, we have to configure ssh
  keys for all Servers in a pool. Key management brings complexity.

* In certain cases, customers need non-ssl migrate, which greatly improves the
  migration speed. There's no way to do it with ssh.

* Current xl migrate command is not intuitive, especially the `-s` option::

      # xl migrate
      Usage: xl [-v] migrate [options] <Domain> <host>
      
      Save a domain state to restore later.
      
      Options:
      
      -h              Print this help.
      -C <config>     Send <config> instead of config file from creation.
      -s <sshcommand> Use <sshcommand> instead of ssh.  String will be passed
                      to sh. If empty, run <host> instead of ssh <host> xl
                      migrate-receive [-d -e]
      -e              Do not wait in the background (on <host>) for the death
                      of the domain.

  It's a little hard to adapt other tools as transport.

* We have differnt implementation for `xl save/restore` and 
  `xl migrate/migrate-receive`. Can we merge them?


Proposal
========

* Implement dedicated daemons for ssl and non-ssl migration receive 
  (`socat <http://www.dest-unreach.org/socat/>`_ can be used).

  Example patch for dedicated migrate receive daemon:
  xen-xl-migrate-socat.patch

  `socat` will call `xl restore` instead of `xl migrate-receive` after the
  following change.

* Merge `xl migrate/migrate-receive` to `xl save/restore`:

  - To save a VM::

      # xl save [-c] <domain> -f vm.chk

    Or::

      # xl save [-c] <domain> >vm.chk

  - To restore a VM::

      # xl restore -f vm.chk

    Or::

      # cat vm.chk | xl restore

  - To migrate a VM using ssh/sshd::

      # xl save -c <domain> | ssh root@<remote-host> xl restore

    We can implement a wrapper to make `xl migrate <domain> <remote-host>` to
    call the above command. 

  - To migrate a VM using dedicated migrate receive daemon::

      # xl save -c <domain> | socat - TCP:<remote-host>:8004"

    Or with SSL::

      # xl save -c <domain> | socat - OPENSSL:<remote-host>:8005,verify=0

  - Localhost migration::

      # xl save <domain> | xl restrore

  - Localhost live migration::

      # xl save -c <domain> | xl restrore

  Patch: I don't have the capability and time to implement it yet.


---- xl-migrate-socat.rst ----

==========
XL Migrate
==========

:Date: 2013-09-16

Current Status
==============

* xl migrate leverages ssh/sshd::

      xl migrate <domain> <host>

* In order to migrate a VM without user interactive, we have to configure ssh
  keys for all Servers in a pool. Key management with dynamic Server Pools is
  error prone.

* In certain cases, customers need non-ssl migrate, which greatly improves the
  migration speed. There's no way to do it with ssh.


Proposal
========

* Implement dedicated daemons for ssl and non-ssl migration receive.

* `socat <http://www.dest-unreach.org/socat/>`_ can be used.

* Patch: xen-xl-migrate-socat.patch


Testing
=======

* Install the new Xen.

* Start the services::

      service xl-migrate-received start
      service xl-ssl-migrate-received start

* Start a VM::

      xl create vm.cfg

* Migrate::

      xl migrate -s xl-migrate-command <domid> localhost

* SSL migrate::

      xl migrate -s xl-ssl-migrate-command <domid> localhost

* Speed for PV guest with 1024M memory:

  - ssh: 45s
  - socat: 9s
  - socat ssl: 57s


---- xen-xl-migrate-socat.patch ----

diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 47655f6..586728b 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -55,6 +55,8 @@ endif
 	$(INSTALL_PROG) $(XENCOMMONS_INITD) $(DESTDIR)$(INITD_DIR)
 	$(INSTALL_DATA) $(XENCOMMONS_SYSCONFIG) $(DESTDIR)$(SYSCONFIG_DIR)/xencommons
 	$(INSTALL_PROG) init.d/xen-watchdog $(DESTDIR)$(INITD_DIR)
+	$(INSTALL_PROG) init.d/xl-migrate-received $(DESTDIR)$(INITD_DIR)
+	$(INSTALL_PROG) init.d/xl-ssl-migrate-received $(DESTDIR)$(INITD_DIR)
 
 .PHONY: install-scripts
 install-scripts:
diff --git a/tools/hotplug/Linux/init.d/xl-migrate-received b/tools/hotplug/Linux/init.d/xl-migrate-received
new file mode 100644
index 0000000..d67cf7d
--- /dev/null
+++ b/tools/hotplug/Linux/init.d/xl-migrate-received
@@ -0,0 +1,106 @@
+#!/bin/bash
+#
+# xl-migrate-received: xl migrate receive daemon
+#
+# chkconfig: 2345 80 20
+# description: xl migrate receive daemon
+
+. /etc/rc.d/init.d/functions
+
+start() {
+    socat -ly TCP-LISTEN:8004,reuseaddr,fork EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null &
+    pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        action $"Starting xl migrate receive daemon: " true
+        touch /var/lock/subsys/xl-migrate-received
+    else
+        action $"Starting xl migrate receive daemon: " false
+    fi
+}
+
+stop() {
+    pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk '{print $1}')
+    action $"Stopping xl migrate receive daemon: " kill -9 $pid
+    [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received
+}
+
+status() {
+    pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        echo $"xl migrate receive daemon is running"
+    else
+        echo $"xl migrate receive daemon is stopped"
+    fi
+}
+
+case "$1" in
+    start)
+        start
+        ;;
+    stop)
+        stop
+        ;;
+    restart)
+        stop
+        start
+        ;;
+    status)
+        status
+        ;;
+    *)
+        echo $"Usage: $0 {start|stop|restart|status}"
+        exit 1
+esac
+#!/bin/bash
+#
+# xl-migrate-received: xl migrate receive daemon
+#
+# chkconfig: 2345 80 20
+# description: xl migrate receive daemon
+
+. /etc/rc.d/init.d/functions
+
+start() {
+    socat -ly TCP-LISTEN:8004,reuseaddr,fork EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null &
+    pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        action $"Starting xl migrate receive daemon: " true
+        touch /var/lock/subsys/xl-migrate-received
+    else
+        action $"Starting xl migrate receive daemon: " false
+    fi
+}
+
+stop() {
+    pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk '{print $1}')
+    action $"Stopping xl migrate receive daemon: " kill -9 $pid
+    [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received
+}
+
+status() {
+    pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        echo $"xl migrate receive daemon is running"
+    else
+        echo $"xl migrate receive daemon is stopped"
+    fi
+}
+
+case "$1" in
+    start)
+        start
+        ;;
+    stop)
+        stop
+        ;;
+    restart)
+        stop
+        start
+        ;;
+    status)
+        status
+        ;;
+    *)
+        echo $"Usage: $0 {start|stop|restart|status}"
+        exit 1
+esac
diff --git a/tools/hotplug/Linux/init.d/xl-ssl-migrate-received b/tools/hotplug/Linux/init.d/xl-ssl-migrate-received
new file mode 100644
index 0000000..272ed9c
--- /dev/null
+++ b/tools/hotplug/Linux/init.d/xl-ssl-migrate-received
@@ -0,0 +1,106 @@
+#!/bin/bash
+#
+# xl-ssl-migrate-received: xl ssl migrate receive daemon
+#
+# chkconfig: 2345 80 20
+# description: xl ssl migrate receive daemon
+
+. /etc/rc.d/init.d/functions
+
+start() {
+    socat -ly OPENSSL-LISTEN:8005,reuseaddr,fork,key=/etc/ovs-agent/cert/key.pem,cert=/etc/ovs-agent/cert/certificate.pem,verify=0 EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null &
+    pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        action $"Starting xl ssl migrate receive daemon: " true
+        touch /var/lock/subsys/xl-migrate-received
+    else
+        action $"Starting xl ssl migrate receive daemon: " false
+    fi
+}
+
+stop() {
+    pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk '{print $1}')
+    action $"Stopping xl ssl migrate receive daemon: " kill -9 $pid
+    [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received
+}
+
+status() {
+    pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        echo $"xl ssl migrate receive daemon is running"
+    else
+        echo $"xl ssl migrate receive daemon is stopped"
+    fi
+}
+
+case "$1" in
+    start)
+        start
+        ;;
+    stop)
+        stop
+        ;;
+    restart)
+        stop
+        start
+        ;;
+    status)
+        status
+        ;;
+    *)
+        echo $"Usage: $0 {start|stop|restart|status}"
+        exit 1
+esac
+#!/bin/bash
+#
+# xl-ssl-migrate-received: xl ssl migrate receive daemon
+#
+# chkconfig: 2345 80 20
+# description: xl ssl migrate receive daemon
+
+. /etc/rc.d/init.d/functions
+
+start() {
+    socat -ly OPENSSL-LISTEN:8005,reuseaddr,fork,key=/etc/ovs-agent/cert/key.pem,cert=/etc/ovs-agent/cert/certificate.pem,verify=0 EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null &
+    pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        action $"Starting xl ssl migrate receive daemon: " true
+        touch /var/lock/subsys/xl-migrate-received
+    else
+        action $"Starting xl ssl migrate receive daemon: " false
+    fi
+}
+
+stop() {
+    pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk '{print $1}')
+    action $"Stopping xl ssl migrate receive daemon: " kill -9 $pid
+    [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received
+}
+
+status() {
+    pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk '{print $1}')
+    if [ -n "$pid" ]; then
+        echo $"xl ssl migrate receive daemon is running"
+    else
+        echo $"xl ssl migrate receive daemon is stopped"
+    fi
+}
+
+case "$1" in
+    start)
+        start
+        ;;
+    stop)
+        stop
+        ;;
+    restart)
+        stop
+        start
+        ;;
+    status)
+        status
+        ;;
+    *)
+        echo $"Usage: $0 {start|stop|restart|status}"
+        exit 1
+esac
diff --git a/tools/misc/Makefile b/tools/misc/Makefile
index 59def7a..88aaf20 100644
--- a/tools/misc/Makefile
+++ b/tools/misc/Makefile
@@ -16,7 +16,7 @@ TARGETS := $(TARGETS-y)
 
 SUBDIRS := $(SUBDIRS-y)
 
-INSTALL_BIN-y := xencons xencov_split
+INSTALL_BIN-y := xencons xencov_split xl-migrate-command xl-ssl-migrate-command
 INSTALL_BIN-$(CONFIG_X86) += xen-detect
 INSTALL_BIN := $(INSTALL_BIN-y)
 
diff --git a/tools/misc/xl-migrate-command b/tools/misc/xl-migrate-command
new file mode 100644
index 0000000..0558ff9
--- /dev/null
+++ b/tools/misc/xl-migrate-command
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+socat - TCP:$1:8004
diff --git a/tools/misc/xl-ssl-migrate-command b/tools/misc/xl-ssl-migrate-command
new file mode 100644
index 0000000..628c554
--- /dev/null
+++ b/tools/misc/xl-ssl-migrate-command
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+socat - OPENSSL:$1:8005,verify=0