Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

Hey,

I launch a bunch of guests at the same time or in parallel and 
the scripts end up timing out with:


Parsing config from //g-vm8.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:53 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/5632
libxl: error: libxl_aoutils.c:539:async_exec_timeout: killing execution of /etc/xen/scripts/block add because of timeout
libxl: error: libxl_create.c:1157:domcreate_launch_dm: unable to add disk devices
libxl: error: libxl_dm.c:1955:kill_device_model: unable to find device model pid in /local/domain/13/image/device-model-pid
libxl: error: libxl.c:1606:libxl__destroy_domid: libxl__destroy_device_model failed for 13
Jul 28 19:21:03 tst036 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/13/5632
Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error xenstore-read backend/vbd/13/5632/node failed. backend/vbd/13/5632/hotplug-status error to xenstore.
Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: xenstore-read backend/vbd/13/5632/node failed.
Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/13/5632/hotplug-status error to xenstore.
Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10344] exited with error status 1
libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl.c:1569:libxl__destroy_domid: non-existant domain 13
libxl: error: libxl.c:1527:domain_destroy_callback: unable to destroy guest with domid 13
libxl: error: libxl.c:1454:domain_destroy_cb: destruction of domain 13 failed

And I cannot start the guest.

While if I revert the mentioned commit everything works peachy.

What is interesting is that if I have the revert I can see that the

Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/physical-device 7:d to xenstore.
Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/hotplug-status connected to xenstore.

or often done much much later after xl create has started.

Attached is the bad log and the good log.

The only difference between the two runs is the revert (also attached)


[-- Attachment #2: 0001-Revert-libxl-Remove-linux-udev-rules.patch --]
[-- Type: text/plain, Size: 12723 bytes --]

>From fb78e678d9371ef9d2ce8b89a44d50e37f69c18a Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 28 Jul 2015 16:26:21 -0400
Subject: [PATCH] Revert "libxl: Remove linux udev rules"

This reverts commit 2ba368d13893402b2f1fb3c283ddcc714659dd9b.

Conflicts:

	tools/configure
---
 tools/configure                              |  3 ++-
 tools/configure.ac                           |  1 +
 tools/examples/README                        |  1 +
 tools/hotplug/Linux/Makefile                 | 14 +++++++++++++-
 tools/hotplug/Linux/xen-backend.rules.in     | 13 +++++++++++++
 tools/hotplug/Linux/xen-hotplug-common.sh.in |  7 +++++++
 tools/libxl/libxl.c                          | 13 +++++++++++++
 tools/libxl/libxl_create.c                   | 27 +++++++++++++++++++++++++++
 tools/libxl/libxl_internal.c                 | 19 +++++++++++++++++++
 tools/libxl/libxl_internal.h                 |  1 +
 tools/libxl/libxl_linux.c                    |  7 +++++++
 tools/libxl/libxl_netbsd.c                   |  7 +++++++
 12 files changed, 111 insertions(+), 2 deletions(-)
 create mode 100644 tools/hotplug/Linux/xen-backend.rules.in

diff --git a/tools/configure b/tools/configure
index d90db47..54dbddf 100755
--- a/tools/configure
+++ b/tools/configure
@@ -2403,7 +2403,7 @@ ac_compiler_gnu=$ac_cv_c_compiler_gnu
 
 
 
-ac_config_files="$ac_config_files ../config/Tools.mk hotplug/FreeBSD/rc.d/xencommons hotplug/FreeBSD/rc.d/xendriverdomain hotplug/Linux/init.d/sysconfig.xencommons hotplug/Linux/init.d/xen-watchdog hotplug/Linux/init.d/xencommons hotplug/Linux/init.d/xendomains hotplug/Linux/init.d/xendriverdomain hotplug/Linux/vif-setup hotplug/Linux/xen-hotplug-common.sh hotplug/Linux/xendomains hotplug/NetBSD/rc.d/xencommons hotplug/NetBSD/rc.d/xendriverdomain libxl/xenlight.pc.in libxl/xlutil.pc.in"
+ac_config_files="$ac_config_files ../config/Tools.mk hotplug/FreeBSD/rc.d/xencommons hotplug/FreeBSD/rc.d/xendriverdomain hotplug/Linux/init.d/sysconfig.xencommons hotplug/Linux/init.d/xen-watchdog hotplug/Linux/xen-backend.rules hotplug/Linux/init.d/xencommons hotplug/Linux/init.d/xendomains hotplug/Linux/init.d/xendriverdomain hotplug/Linux/vif-setup hotplug/Linux/xen-hotplug-common.sh hotplug/Linux/xendomains hotplug/NetBSD/rc.d/xencommons hotplug/NetBSD/rc.d/xendriverdomain libxl/xenlight.pc.in libxl/xlutil.pc.in"
 
 ac_config_headers="$ac_config_headers config.h"
 
@@ -10052,6 +10052,7 @@ do
     "hotplug/Linux/init.d/xendomains") CONFIG_FILES="$CONFIG_FILES hotplug/Linux/init.d/xendomains" ;;
     "hotplug/Linux/init.d/xendriverdomain") CONFIG_FILES="$CONFIG_FILES hotplug/Linux/init.d/xendriverdomain" ;;
     "hotplug/Linux/vif-setup") CONFIG_FILES="$CONFIG_FILES hotplug/Linux/vif-setup" ;;
+    "hotplug/Linux/xen-backend.rules") CONFIG_FILES="$CONFIG_FILES hotplug/Linux/xen-backend.rules" ;;
     "hotplug/Linux/xen-hotplug-common.sh") CONFIG_FILES="$CONFIG_FILES hotplug/Linux/xen-hotplug-common.sh" ;;
     "hotplug/Linux/xendomains") CONFIG_FILES="$CONFIG_FILES hotplug/Linux/xendomains" ;;
     "hotplug/NetBSD/rc.d/xencommons") CONFIG_FILES="$CONFIG_FILES hotplug/NetBSD/rc.d/xencommons" ;;
diff --git a/tools/configure.ac b/tools/configure.ac
index f6a79c2..30decb2 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -15,6 +15,7 @@ hotplug/Linux/init.d/xencommons
 hotplug/Linux/init.d/xendomains
 hotplug/Linux/init.d/xendriverdomain
 hotplug/Linux/vif-setup
+hotplug/Linux/xen-backend.rules
 hotplug/Linux/xen-hotplug-common.sh
 hotplug/Linux/xendomains
 hotplug/NetBSD/rc.d/xencommons
diff --git a/tools/examples/README b/tools/examples/README
index 13380a4..115ca02 100644
--- a/tools/examples/README
+++ b/tools/examples/README
@@ -24,6 +24,7 @@ vif-nat             - xen virtual network start/stop script in NAT mode
 vif-route           - xen virtual network start/stop script in routed mode
 xen-backend.agent   - calls block, vif-* scripts to add, remove, hotplug
                       devices  
+xen-backend.rules   - hotplug script rules
 xen-hotplug-common.sh - sourced by vif-common.sh
 xen-network-common.sh - sourced by vif-common.sh
 xen-script-common.sh  - sourced by xen-hotplug-common.sh
diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 6e10118..01ee48d 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -35,6 +35,9 @@ XEN_SCRIPT_DATA = xen-script-common.sh locking.sh logging.sh
 XEN_SCRIPT_DATA += xen-hotplug-common.sh xen-network-common.sh vif-common.sh
 XEN_SCRIPT_DATA += block-common.sh
 
+UDEV_RULES_DIR = $(CONFIG_DIR)/udev
+UDEV_RULES = xen-backend.rules $(UDEV_RULES-y)
+
 .PHONY: all
 all: subdirs-all
 
@@ -42,7 +45,7 @@ all: subdirs-all
 build:
 
 .PHONY: install
-install: install-initd install-scripts subdirs-install
+install: install-initd install-scripts install-udev subdirs-install
 
 # See docs/misc/distro_mapping.txt for INITD_DIR location
 .PHONY: install-initd
@@ -71,6 +74,15 @@ install-scripts:
 	    $(INSTALL_DATA) $$i $(DESTDIR)$(XEN_SCRIPT_DIR); \
 	done
 
+.PHONY: install-udev
+install-udev:
+	[ -d $(DESTDIR)$(UDEV_RULES_DIR) ] || \
+		$(INSTALL_DIR) $(DESTDIR)$(UDEV_RULES_DIR)/rules.d
+	set -e; for i in $(UDEV_RULES); \
+	    do \
+	    $(INSTALL_DATA) $$i $(DESTDIR)$(UDEV_RULES_DIR)/rules.d; \
+	done
+
 .PHONY: clean
 clean: subdirs-clean
 
diff --git a/tools/hotplug/Linux/xen-backend.rules.in b/tools/hotplug/Linux/xen-backend.rules.in
new file mode 100644
index 0000000..ee107af
--- /dev/null
+++ b/tools/hotplug/Linux/xen-backend.rules.in
@@ -0,0 +1,13 @@
+SUBSYSTEM=="xen-backend", KERNEL=="vbd*", ENV{UDEV_CALL}="1", RUN+="@XEN_SCRIPT_DIR@/block $env{ACTION}"
+SUBSYSTEM=="xen-backend", KERNEL=="vif2-*", RUN+="@XEN_SCRIPT_DIR@/vif2 $env{ACTION}"
+SUBSYSTEM=="xen-backend", KERNEL=="vif-*", ENV{UDEV_CALL}="1", ACTION=="online", RUN+="@XEN_SCRIPT_DIR@/vif-setup online type_if=vif"
+SUBSYSTEM=="xen-backend", KERNEL=="vif-*", ENV{UDEV_CALL}="1", ACTION=="offline", RUN+="@XEN_SCRIPT_DIR@/vif-setup offline type_if=vif"
+SUBSYSTEM=="xen-backend", KERNEL=="vscsi*", RUN+="@XEN_SCRIPT_DIR@/vscsi $env{ACTION}"
+SUBSYSTEM=="xen-backend", ACTION=="remove", ENV{UDEV_CALL}="1", RUN+="@XEN_SCRIPT_DIR@/xen-hotplug-cleanup"
+KERNEL=="evtchn", NAME="xen/%k"
+SUBSYSTEM=="blktap2", KERNEL=="blktap[0-9]*", NAME="xen/blktap-2/%k", MODE="0600"
+KERNEL=="blktap-control", NAME="xen/blktap-2/control", MODE="0600"
+KERNEL=="gntdev", NAME="xen/%k", MODE="0600"
+KERNEL=="pci_iomul", NAME="xen/%k", MODE="0600"
+KERNEL=="tapdev[a-z]*", NAME="xen/blktap-2/tapdev%m", MODE="0600"
+SUBSYSTEM=="net", KERNEL=="vif*-emu", ACTION=="add", ENV{UDEV_CALL}="1", RUN+="@XEN_SCRIPT_DIR@/vif-setup $env{ACTION} type_if=tap"
diff --git a/tools/hotplug/Linux/xen-hotplug-common.sh.in b/tools/hotplug/Linux/xen-hotplug-common.sh.in
index afb240f..40b933e 100644
--- a/tools/hotplug/Linux/xen-hotplug-common.sh.in
+++ b/tools/hotplug/Linux/xen-hotplug-common.sh.in
@@ -15,6 +15,13 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 #
 
+# Hack to prevent the execution of hotplug scripts from udev if the domain
+# has been launched from libxl
+if [ -n "${UDEV_CALL}" ] && \
+   xenstore-read "libxl/disable_udev" >/dev/null 2>&1; then
+    exit 0
+fi
+
 dir=$(dirname "$0")
 . "$dir/hotplugpath.sh"
 . "$dir/logging.sh"
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index ff0d616..029c8cd 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3194,6 +3194,7 @@ out:
 int libxl__device_nic_setdefault(libxl__gc *gc, libxl_device_nic *nic,
                                  uint32_t domid)
 {
+    int run_hotplug_scripts;
     int rc;
 
     if (!nic->mtu)
@@ -3224,6 +3225,12 @@ int libxl__device_nic_setdefault(libxl__gc *gc, libxl_device_nic *nic,
                                   libxl__xen_script_dir_path()) < 0 )
         return ERROR_FAIL;
 
+    run_hotplug_scripts = libxl__hotplug_settings(gc, XBT_NULL);
+    if (run_hotplug_scripts < 0) {
+        LOG(ERROR, "unable to get current hotplug scripts execution setting");
+        return run_hotplug_scripts;
+    }
+
     rc = libxl__resolve_domid(gc, nic->backend_domname, &nic->backend_domid);
     if (rc < 0) return rc;
 
@@ -4526,6 +4533,12 @@ int libxl_device_events_handler(libxl_ctx *ctx,
         goto out;
     }
 
+    rc = libxl__xs_write_checked(gc, XBT_NULL, DISABLE_UDEV_PATH, "1");
+    if (rc) {
+        LOGE(ERROR, "unable to write %s = 1", DISABLE_UDEV_PATH);
+        goto out;
+    }
+
     /*
      * We use absolute paths because we want xswatch to also return
      * absolute paths that can be parsed by libxl__parse_backend_path.
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 855b42c..4621d88 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -646,6 +646,33 @@ retry_transaction:
         goto out;
     }
     libxl_vminfo_list_free(vm_list, nb_vm);
+    int hotplug_setting = libxl__hotplug_settings(gc, t);
+    if (hotplug_setting < 0) {
+        LOG(ERROR, "unable to get current hotplug scripts execution setting");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    if (libxl_defbool_val(info->run_hotplug_scripts) != hotplug_setting &&
+        (nb_vm - 1)) {
+        LOG(ERROR, "cannot change hotplug execution option once set, "
+                    "please shutdown all guests before changing it");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (libxl_defbool_val(info->run_hotplug_scripts)) {
+        rc = libxl__xs_write_checked(gc, t, DISABLE_UDEV_PATH, "1");
+        if (rc) {
+            LOGE(ERROR, "unable to write %s = 1", DISABLE_UDEV_PATH);
+            goto out;
+        }
+    } else {
+        rc = libxl__xs_rm_checked(gc, t, DISABLE_UDEV_PATH);
+        if (rc) {
+            LOGE(ERROR, "unable to delete %s", DISABLE_UDEV_PATH);
+            goto out;
+        }
+    }
 
     xs_write(ctx->xsh, t, libxl__sprintf(gc, "%s/uuid", vm_path), uuid_string, strlen(uuid_string));
     xs_write(ctx->xsh, t, libxl__sprintf(gc, "%s/name", vm_path), info->name, strlen(info->name));
diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index 23fd751..3d8a593 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -383,6 +383,25 @@ int libxl__device_model_version_running(libxl__gc *gc, uint32_t domid)
     return value;
 }
 
+int libxl__hotplug_settings(libxl__gc *gc, xs_transaction_t t)
+{
+    int rc = 0;
+    char *val;
+
+    val = libxl__xs_read(gc, t, DISABLE_UDEV_PATH);
+    if (!val && errno != ENOENT) {
+        LOGE(ERROR, "cannot read %s from xenstore", DISABLE_UDEV_PATH);
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    if (!val) val = "0";
+
+    rc = !!atoi(val);
+
+out:
+    return rc;
+}
+
 /* Portability note: this lock utilises flock(2) so a proper implementation of
  * flock(2) is required.
  */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index f2466dc..a9c5cc4 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -108,6 +108,7 @@
 #define STUBDOM_CONSOLE_SERIAL 3
 #define STUBDOM_SPECIAL_CONSOLES 3
 #define TAP_DEVICE_SUFFIX "-emu"
+#define DISABLE_UDEV_PATH "libxl/disable_udev"
 #define DOMID_XS_PATH "domid"
 
 #define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))
diff --git a/tools/libxl/libxl_linux.c b/tools/libxl/libxl_linux.c
index 4fbcba1..1a780ee 100644
--- a/tools/libxl/libxl_linux.c
+++ b/tools/libxl/libxl_linux.c
@@ -238,8 +238,15 @@ int libxl__get_hotplug_script_info(libxl__gc *gc, libxl__device *dev,
                                    libxl__device_action action,
                                    int num_exec)
 {
+    char *disable_udev = libxl__xs_read(gc, XBT_NULL, DISABLE_UDEV_PATH);
     int rc;
 
+    /* Check if we have to run hotplug scripts */
+    if (!disable_udev) {
+        rc = 0;
+        goto out;
+    }
+
     switch (dev->backend_kind) {
     case LIBXL__DEVICE_KIND_VBD:
         if (num_exec != 0) {
diff --git a/tools/libxl/libxl_netbsd.c b/tools/libxl/libxl_netbsd.c
index 096c057..a2a962e 100644
--- a/tools/libxl/libxl_netbsd.c
+++ b/tools/libxl/libxl_netbsd.c
@@ -64,8 +64,15 @@ int libxl__get_hotplug_script_info(libxl__gc *gc, libxl__device *dev,
                                    libxl__device_action action,
                                    int num_exec)
 {
+    char *disable_udev = libxl__xs_read(gc, XBT_NULL, DISABLE_UDEV_PATH);
     int rc;
 
+    /* Check if we have to run hotplug scripts */
+    if (!disable_udev || num_exec > 0) {
+        rc = 0;
+        goto out;
+    }
+
     switch (dev->backend_kind) {
     case LIBXL__DEVICE_KIND_VBD:
     case LIBXL__DEVICE_KIND_VIF:
-- 
1.8.4.2


[-- Attachment #3: bad.txt --]
[-- Type: text/plain, Size: 13901 bytes --]

-bash-4.1# for a in `ls /*.cfg`; do  cat /$a.cfg;  echo $a.cfg;  xl create /$a.cfg; done\b\b\b\b\b\b\b^[[1P\b^[[1P\b^[[1P\b^[[1P\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[1P\b^[[1P\b^[[1P\b^[[1P;  echo $a^[[1P^[[1P^[[1P^[[1P
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm0'
on_crash='preserve'
# IP:192.168.102.90
# MAC:00:0f:4b:01:01:00
pci= [ '0000:0a:10.1' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm0.cfg
Parsing config from //g-vm0.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:26 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/5632
Jul 28 19:20:26 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/1/5632/node /dev/loop0 to xenstore.
Jul 28 19:20:26 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/1/5632/physical-device 7:0 to xenstore.
Jul 28 19:20:26 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/1/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm1'
on_crash='preserve'
# IP:192.168.102.91
# MAC:00:0f:4b:01:01:01
pci= [ '0000:0a:10.3' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm1.cfg
Parsing config from //g-vm1.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:27 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/2/5632
Jul 28 19:20:27 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/node /dev/loop1 to xenstore.
Jul 28 19:20:27 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/physical-device 7:1 to xenstore.
Jul 28 19:20:27 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm10'
on_crash='preserve'
# IP:192.168.102.100
# MAC:00:0f:4b:01:01:0a
pci= [ '0000:0a:10.6' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm10.cfg
Parsing config from //g-vm10.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:28 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/5632
Jul 28 19:20:28 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/3/5632/node /dev/loop2 to xenstore.
Jul 28 19:20:28 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/3/5632/physical-device 7:2 to xenstore.
Jul 28 19:20:28 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/3/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm11'
on_crash='preserve'
# IP:192.168.102.101
# MAC:00:0f:4b:01:01:0b
pci= [ '0000:0a:11.0' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm11.cfg
Parsing config from //g-vm11.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:29 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/4/5632
Jul 28 19:20:29 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/4/5632/node /dev/loop3 to xenstore.
Jul 28 19:20:29 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/4/5632/physical-device 7:3 to xenstore.
Jul 28 19:20:29 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/4/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm12'
on_crash='preserve'
# IP:192.168.102.102
# MAC:00:0f:4b:01:01:0c
pci= [ '0000:0a:11.2' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm12.cfg
Parsing config from //g-vm12.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:30 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/5/5632
Jul 28 19:20:31 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/5/5632/node /dev/loop4 to xenstore.
Jul 28 19:20:31 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/5/5632/physical-device 7:4 to xenstore.
Jul 28 19:20:31 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/5/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm13'
on_crash='preserve'
# IP:192.168.102.103
# MAC:00:0f:4b:01:01:0d
pci= [ '0000:0a:11.4' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm13.cfg
Parsing config from //g-vm13.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:31 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/6/5632
Jul 28 19:20:32 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/6/5632/node /dev/loop5 to xenstore.
Jul 28 19:20:32 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/6/5632/physical-device 7:5 to xenstore.
Jul 28 19:20:32 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/6/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm2'
on_crash='preserve'
# IP:192.168.102.92
# MAC:00:0f:4b:01:01:02
pci= [ '0000:0a:10.5' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm2.cfg
Parsing config from //g-vm2.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:33 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/7/5632
Jul 28 19:20:34 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/7/5632/node /dev/loop6 to xenstore.
Jul 28 19:20:34 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/7/5632/physical-device 7:6 to xenstore.
Jul 28 19:20:34 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/7/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm3'
on_crash='preserve'
# IP:192.168.102.93
# MAC:00:0f:4b:01:01:03
pci= [ '0000:0a:10.7' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm3.cfg
Parsing config from //g-vm3.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:34 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/8/5632
Jul 28 19:20:35 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/8/5632/node /dev/loop7 to xenstore.
Jul 28 19:20:36 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/8/5632/physical-device 7:7 to xenstore.
Jul 28 19:20:36 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/8/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm4'
on_crash='preserve'
# IP:192.168.102.94
# MAC:00:0f:4b:01:01:04
pci= [ '0000:0a:11.1' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm4.cfg
Parsing config from //g-vm4.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:36 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/9/5632
Jul 28 19:20:38 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/9/5632/node /dev/loop8 to xenstore.
Jul 28 19:20:38 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/9/5632/physical-device 7:8 to xenstore.
Jul 28 19:20:38 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/9/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm5'
on_crash='preserve'
# IP:192.168.102.95
# MAC:00:0f:4b:01:01:05
pci= [ '0000:0a:11.3' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm5.cfg
Parsing config from //g-vm5.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:39 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/10/5632
Jul 28 19:20:41 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/10/5632/node /dev/loop9 to xenstore.
Jul 28 19:20:41 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/10/5632/physical-device 7:9 to xenstore.
Jul 28 19:20:41 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/10/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm6'
on_crash='preserve'
# IP:192.168.102.96
# MAC:00:0f:4b:01:01:06
pci= [ '0000:0a:11.5' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm6.cfg
Parsing config from //g-vm6.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:41 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/11/5632
Jul 28 19:20:44 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/11/5632/node /dev/loop10 to xenstore.
Jul 28 19:20:44 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/11/5632/physical-device 7:a to xenstore.
Jul 28 19:20:44 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/11/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm7'
on_crash='preserve'
# IP:192.168.102.97
# MAC:00:0f:4b:01:01:07
pci= [ '0000:0a:10.0' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm7.cfg
Parsing config from //g-vm7.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:45 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/12/5632
Jul 28 19:20:51 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/12/5632/node /dev/loop11 to xenstore.
Jul 28 19:20:51 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/12/5632/physical-device 7:b to xenstore.
Jul 28 19:20:51 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/12/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm8'
on_crash='preserve'
# IP:192.168.102.98
# MAC:00:0f:4b:01:01:08
pci= [ '0000:0a:10.2' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
/g-vm8.cfg
Parsing config from //g-vm8.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:20:53 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/5632
libxl: error: libxl_aoutils.c:539:async_exec_timeout: killing execution of /etc/xen/scripts/block add because of timeout
libxl: error: libxl_create.c:1157:domcreate_launch_dm: unable to add disk devices
libxl: error: libxl_dm.c:1955:kill_device_model: unable to find device model pid in /local/domain/13/image/device-model-pid
libxl: error: libxl.c:1606:libxl__destroy_domid: libxl__destroy_device_model failed for 13
Jul 28 19:21:03 tst036 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/13/5632
Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error xenstore-read backend/vbd/13/5632/node failed. backend/vbd/13/5632/hotplug-status error to xenstore.
Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: xenstore-read backend/vbd/13/5632/node failed.
Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/13/5632/hotplug-status error to xenstore.
Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10344] exited with error status 1
libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
libxl: error: libxl.c:1569:libxl__destroy_domid: non-existant domain 13
libxl: error: libxl.c:1527:domain_destroy_callback: unable to destroy guest with domid 13
libxl: error: libxl.c:1454:domain_destroy_cb: destruction of domain 13 failed

[-- Attachment #4: good.txt --]
[-- Type: text/plain, Size: 16728 bytes --]

-bash-4.1# tail -f /var/log/messages 
Jul 28 19:28:02 tst036 xenstored: Checking store complete.
Jul 28 19:28:03 tst036 iscsid: transport class version 2.0-870. iscsid version 2.0-872
Jul 28 19:28:03 tst036 iscsid: iSCSI daemon with pid=4277 started!
Jul 28 19:28:06 tst036 iscsid: Connection1:0 to [target: iqn.2003-01.org.linux-iscsi.target:sn.bd5777dc54e541e0bd772761999275e6, portal: 192.168.102.1,3260] through [iface: default] is operational now
Jul 28 19:28:13 tst036 init: starting pid 4575, tty '/dev/tty1': '/bin/sh'
Jul 28 19:28:15 tst036 init: reloading /etc/inittab
Jul 28 19:28:15 tst036 init: starting pid 4579, tty '/dev/hvc0': '/bin/sh'
Jul 28 19:36:46 tst036 sshd[4581]: WARNING: /etc/ssh/moduli does not exist, using fixed modulus
Jul 28 19:36:46 tst036 sshd[4581]: Accepted publickey for root from 192.168.102.1 port 54524 ssh2
Jul 28 19:36:46 tst036 sshd[4583]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
^Z
[1]+  Stopped                 tail -f /var/log/messages
-bash-4.1# bg
[1]+ tail -f /var/log/messages &
-bash-4.1# Jul 28 19:36:46 tst036 sshd[4583]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
for ai n`\b\b\b\b\b\b\b\b\b^[[Kfor a in `ls /*.cfg`
> do
>  cat /$a
>  xl create /$a
> done
ls: cannot access /*.cfg: No such file or directory
-bash-4.1# /mnt/lab/ru\an_guests
We have 14 VFs.
Creating 14 HVM(qemu-xen) guests.
-bash-4.1# /mnt/lab/run_guests\r-bash-4.1# for a in `ls /*.cfg`; do  cat /$a;  xl create /$a; done
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm0'
on_crash='preserve'
# IP:192.168.102.90
# MAC:00:0f:4b:01:01:00
pci= [ '0000:0a:10.1' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm0.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/5632
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/1/5632/node /dev/loop0 to xenstore.
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/1/5632/physical-device 7:0 to xenstore.
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/1/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm1'
on_crash='preserve'
# IP:192.168.102.91
# MAC:00:0f:4b:01:01:01
pci= [ '0000:0a:10.3' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm1.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/2/5632
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/node /dev/loop1 to xenstore.
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/physical-device 7:1 to xenstore.
Jul 28 19:37:58 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm10'
on_crash='preserve'
# IP:192.168.102.100
# MAC:00:0f:4b:01:01:0a
pci= [ '0000:0a:10.6' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm10.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:37:59 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/5632
Jul 28 19:37:59 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/3/5632/node /dev/loop2 to xenstore.
Jul 28 19:37:59 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/3/5632/physical-device 7:2 to xenstore.
Jul 28 19:37:59 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/3/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm11'
on_crash='preserve'
# IP:192.168.102.101
# MAC:00:0f:4b:01:01:0b
pci= [ '0000:0a:11.0' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm11.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:00 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/4/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm12'
on_crash='preserve'
# IP:192.168.102.102
# MAC:00:0f:4b:01:01:0c
pci= [ '0000:0a:11.2' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm12.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:00 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/4/5632/node /dev/loop3 to xenstore.
Jul 28 19:38:00 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/4/5632/physical-device 7:3 to xenstore.
Jul 28 19:38:00 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/4/5632/hotplug-status connected to xenstore.
Jul 28 19:38:00 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/5/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm13'
on_crash='preserve'
# IP:192.168.102.103
# MAC:00:0f:4b:01:01:0d
pci= [ '0000:0a:11.4' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm13.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:01 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/5/5632/node /dev/loop4 to xenstore.
Jul 28 19:38:01 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/5/5632/physical-device 7:4 to xenstore.
Jul 28 19:38:01 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/5/5632/hotplug-status connected to xenstore.
Jul 28 19:38:01 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/6/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm2'
on_crash='preserve'
# IP:192.168.102.92
# MAC:00:0f:4b:01:01:02
pci= [ '0000:0a:10.5' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm2.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:02 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/7/5632
Jul 28 19:38:02 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/6/5632/node /dev/loop5 to xenstore.
Jul 28 19:38:02 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/6/5632/physical-device 7:5 to xenstore.
Jul 28 19:38:02 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/6/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm3'
on_crash='preserve'
# IP:192.168.102.93
# MAC:00:0f:4b:01:01:03
pci= [ '0000:0a:10.7' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm3.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:03 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/8/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm4'
on_crash='preserve'
# IP:192.168.102.94
# MAC:00:0f:4b:01:01:04
pci= [ '0000:0a:11.1' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm4.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/7/5632/node /dev/loop6 to xenstore.
Jul 28 19:38:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/7/5632/physical-device 7:6 to xenstore.
Jul 28 19:38:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/7/5632/hotplug-status connected to xenstore.
Jul 28 19:38:04 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/9/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm5'
on_crash='preserve'
# IP:192.168.102.95
# MAC:00:0f:4b:01:01:05
pci= [ '0000:0a:11.3' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm5.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:06 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/10/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm6'
on_crash='preserve'
# IP:192.168.102.96
# MAC:00:0f:4b:01:01:06
pci= [ '0000:0a:11.5' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm6.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:06 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/8/5632/node /dev/loop7 to xenstore.
Jul 28 19:38:06 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/8/5632/physical-device 7:7 to xenstore.
Jul 28 19:38:06 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/8/5632/hotplug-status connected to xenstore.
Jul 28 19:38:07 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/11/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm7'
on_crash='preserve'
# IP:192.168.102.97
# MAC:00:0f:4b:01:01:07
pci= [ '0000:0a:10.0' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm7.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:09 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/9/5632/node /dev/loop8 to xenstore.
Jul 28 19:38:09 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/9/5632/physical-device 7:8 to xenstore.
Jul 28 19:38:09 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/9/5632/hotplug-status connected to xenstore.
Jul 28 19:38:09 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/12/5632
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm8'
on_crash='preserve'
# IP:192.168.102.98
# MAC:00:0f:4b:01:01:08
pci= [ '0000:0a:10.2' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm8.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:12 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/5632
Jul 28 19:38:12 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/10/5632/node /dev/loop9 to xenstore.
Jul 28 19:38:12 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/10/5632/physical-device 7:9 to xenstore.
Jul 28 19:38:12 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/10/5632/hotplug-status connected to xenstore.
memory=768
boot='d'
vcpus=2
vnclisten='0.0.0.0'
name='g-vm9'
on_crash='preserve'
# IP:192.168.102.99
# MAC:00:0f:4b:01:01:09
pci= [ '0000:0a:10.4' ]
builder='hvm'
disk = [ 'file:/mnt/lab/bootstrap-x86_64/root_image.iso,hdc:cdrom,r']
kernel = 'hvmloader'
serial='pty'
Parsing config from //g-vm9.cfg
WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
Jul 28 19:38:15 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/12/5632/node /dev/loop10 to xenstore.
Jul 28 19:38:15 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/12/5632/physical-device 7:a to xenstore.
Jul 28 19:38:15 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/12/5632/hotplug-status connected to xenstore.
Jul 28 19:38:16 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/14/5632
-bash-4.1# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   999     3     r-----      77.0
g-vm0                                        1   768     1     ------       0.9
g-vm1                                        2   768     1     ------       1.0
g-vm10                                       3   768     1     ------       1.0
g-vm11                                       4   768     1     ------       0.8
g-vm12                                       5   768     1     ------       0.9
g-vm13                                       6   768     1     ------       0.8
g-vm2                                        7   768     1     ------       0.8
g-vm3                                        8   768     1     ------       0.7
g-vm4                                        9   768     1     -b----       0.6
g-vm5                                       10   768     1     -b----       0.6
g-vm6                                       11   768     1     -b----       0.6
g-vm7                                       12   768     1     -b----       0.5
g-vm8                                       13   768     1     ------       0.5
g-vm9                                       14   768     1     -b----       0.3
-bash-4.1# 
-bash-4.1# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   999     3     r-----     159.1
g-vm0                                        1   768     1     ------       1.4
g-vm1                                        2   768     1     ------       1.5
g-vm10                                       3   768     1     ------       1.3
g-vm11                                       4   768     1     ------       1.4
g-vm12                                       5   768     1     ------       1.3
g-vm13                                       6   768     1     ------       1.1
g-vm2                                        7   768     1     ------       1.2
g-vm3                                        8   768     1     ------       1.1
g-vm4                                        9   768     1     ------       1.2
g-vm5                                       10   768     1     ------       1.0
g-vm6                                       11   768     1     ------       1.0
g-vm7                                       12   768     1     ------       1.0
g-vm8                                       13   768     1     ------       1.0
g-vm9                                       14   768     1     ------       0.8
-bash-4.1# Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/node /dev/loop13 to xenstore.
Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/physical-device 7:d to xenstore.
Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/hotplug-status connected to xenstore.

[-- Attachment #5: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Ian Campbell]

On Tue, 2015-07-28 at 15:47 -0400, Konrad Rzeszutek Wilk wrote:
> Hey,
> 
> I launch a bunch of guests at the same time or in parallel and 
> the scripts end up timing out with:

Are you sure you have cleaned out all the old udev .rules files? If any of
those are still present then you will get both sets competing to drive
things and they will conflict and cause this sort of breakage.

Perhaps we should put back the hacks which nobble the udev case for another
release? i.e. the thing which writes the path (but unconditional in
xencommons) and the bit in the hotplug scripts which gates on it, but still
remove the .rules files. That's only delaying the inevitable though, since
upgrades to 4.7 will have the same issue.

Perhaps in the scripts themselves:

if [ -n "${UDEV_CALL}" ] ; then
	error "called through udev, please remove stale udev rules files"
fi

relying on the (stale) 4.5 rules file having the UDEV_CALL=1 in them.

Ian.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 29/07/15 a les 11.03, Ian Campbell ha escrit:
> On Tue, 2015-07-28 at 15:47 -0400, Konrad Rzeszutek Wilk wrote:
>> Hey,
>>
>> I launch a bunch of guests at the same time or in parallel and 
>> the scripts end up timing out with:
> 
> Are you sure you have cleaned out all the old udev .rules files? If any of
> those are still present then you will get both sets competing to drive
> things and they will conflict and cause this sort of breakage.
> 
> Perhaps we should put back the hacks which nobble the udev case for another
> release? i.e. the thing which writes the path (but unconditional in
> xencommons) and the bit in the hotplug scripts which gates on it, but still
> remove the .rules files. That's only delaying the inevitable though, since
> upgrades to 4.7 will have the same issue.
> 
> Perhaps in the scripts themselves:
> 
> if [ -n "${UDEV_CALL}" ] ; then
> 	error "called through udev, please remove stale udev rules files"
> fi
> 
> relying on the (stale) 4.5 rules file having the UDEV_CALL=1 in them.

Another option would be to install an empty xen-backend.rules for the
4.6 release, and then remove it for 4.7.

I've also been able to trigger this by using a similar loop. AFAICT the
hotplug scripts are running correctly, the problem seems to be that the
check_sharing function that's executed to check *every* loop device that
points to the same file is scanning xenstore in order to find if the
loop device is also used by another guest. When 20 guests are launched
in parallel, the CPU consumption in Dom0 is quite high because of all
the Qemu processes, and the xenstore daemon is basically starving to get
some CPU time.

IMHO, we should remove this checks and allow the users to shoot on their
feet if they want to, and in fact that's what I did on FreeBSD.

What I still don't understand is why this only triggers with 2ba368
applied. My best guess is that you still have a stale xen-backend.rules
file so you are actually calling the hotplug scripts twice, creating x2
loop devices for each guest, which of course also slows down things even
more.

Roger.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Konrad Rzeszutek Wilk]

On Wed, Jul 29, 2015 at 12:52:37PM +0200, Roger Pau Monné wrote:
> El 29/07/15 a les 11.03, Ian Campbell ha escrit:
> > On Tue, 2015-07-28 at 15:47 -0400, Konrad Rzeszutek Wilk wrote:
> >> Hey,
> >>
> >> I launch a bunch of guests at the same time or in parallel and 
> >> the scripts end up timing out with:
> > 
> > Are you sure you have cleaned out all the old udev .rules files? If any of
> > those are still present then you will get both sets competing to drive
> > things and they will conflict and cause this sort of breakage.

This is what I have in udev without the revert.

-bash-4.1# find /etc/udev/
/udev/
/etc/udev/udev.conf
/etc/udev/makedev.d
/etc/udev/rules.d
/etc/udev/rules.d/70-persistent-net.rules
/etc/udev/rules.d/dhclient.rules


> > 
> > Perhaps we should put back the hacks which nobble the udev case for another
> > release? i.e. the thing which writes the path (but unconditional in
> > xencommons) and the bit in the hotplug scripts which gates on it, but still
> > remove the .rules files. That's only delaying the inevitable though, since
> > upgrades to 4.7 will have the same issue.
> > 
> > Perhaps in the scripts themselves:
> > 
> > if [ -n "${UDEV_CALL}" ] ; then
> > 	error "called through udev, please remove stale udev rules files"
> > fi
> > 
> > relying on the (stale) 4.5 rules file having the UDEV_CALL=1 in them.

I don't exactly understand how the hotplug scripts are invoked via 'xl'.
With udev it was pretty clear and easy to me.

And the invocation of the scripts were driven by the backend - that is 
xen-blkback advertising whenever it was ready - and then udev rules
triggering the scripts.

In my case I seem to have xen-blkback taking its time and coming up with
the disk much much later - and the block scripts have already run
and timed out.

We could add some code in the xl (where it executes the hotplug scripts)
to monitor for kernel udev events from the backend - and only then
execute the hotplug scripts. But that is a bit of custom code to deal
with xen-blkback, xen-netback, xen-pciback, etc.

And that would still neccessity an udev rules to funnel them to the 'xl devd'
daemon which would understand udev format and such (via a socket).

Note that I see this problem regardless of me having 'xl devd' running or not.
> 
> Another option would be to install an empty xen-backend.rules for the
> 4.6 release, and then remove it for 4.7.

Or trim down the udev rules ?

> 
> I've also been able to trigger this by using a similar loop. AFAICT the
> hotplug scripts are running correctly, the problem seems to be that the
> check_sharing function that's executed to check *every* loop device that
> points to the same file is scanning xenstore in order to find if the
> loop device is also used by another guest. When 20 guests are launched
> in parallel, the CPU consumption in Dom0 is quite high because of all
> the Qemu processes, and the xenstore daemon is basically starving to get
> some CPU time.
> 
> IMHO, we should remove this checks and allow the users to shoot on their
> feet if they want to, and in fact that's what I did on FreeBSD.
> 
> What I still don't understand is why this only triggers with 2ba368
> applied. My best guess is that you still have a stale xen-backend.rules
> file so you are actually calling the hotplug scripts twice, creating x2
> loop devices for each guest, which of course also slows down things even
> more.

Sadly no. No stable xen-backend.rules - this was a fresh install.

> 
> Roger.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Ian Campbell]

On Wed, 2015-07-29 at 11:45 -0400, Konrad Rzeszutek Wilk wrote:
> relying on the (stale) 4.5 rules file having the UDEV_CALL=1 in them.
> 
> I don't exactly understand how the hotplug scripts are invoked via 'xl'.

They are called when the backend gets to (or passes through)
XenbusStateInitWait (attach) or XenbusStateClosed (detach).

> With udev it was pretty clear and easy to me.

It was also, unfortunately, racy.

In particular on tear down there was no interlock between the scripts
(executed asynchronously by udev) and the toolstack. Some backends have
interlock between the backend and the script, but that's not the
same/sufficient.

This race means that the xenstore dir could be removed before the script
runs, and the script may need information from xenstore in order to do the
tear down.

This was a particular problem for detaching a vif on a vswitch system,
since vswitch (unlike Linux bridge) does not automatically remove a port
when the device disappears, so we need xenstore info (specifically the
bridge node) to clean up.

I believe there were also similar issues with block-iscsi (to logout of the
target) and even regular block devices where loopback was in use (to see
the type and know whether to losetup -d or not, this was the reason why we
didn't do loopback for file:// devices with libxl for quite a while).

It was also completely different for each backend platform (Linux,
BSD,etc), which was problematic from a support PoV.

> Note that I see this problem regardless of me having 'xl devd' running or 
> not.

You definitely do _not_ want to run xl devd in dom0 (or more precisely in
your toolstack domain). Having both xl and devd doing this operations will
not result in anything you want.

Might be worth having some interlock on that, if we don't already.

> > Another option would be to install an empty xen-backend.rules for the
> > 4.6 release, and then remove it for 4.7.
> 
> Or trim down the udev rules ?

The udev scripts should have been unused since 4.5.0-rc1, where they were
by default gated from running in dom0 in favour of the libxl version. In
the default configuration the scripts detected when they were called via
udev and exited immediately without doing anything, leaving them to do the
real work when called directly from the toolstack.

Have you been seeing this issue since then and "fixing" it by manually
reverting to the udev behaviour in /etc/xen/xl.conf (or elsewhere for other
libxl clients)?

If not then there is some unintentional change in 2ba368d138934 as well as
the unintentional removal of the udev scripts. There really should have
been no semantic change compared with the default behaviour from 4.5.0-rc1.

Putting back the udev rules (even a trimmed down version, whatever that
means) is just papering over the underlying issue, whatever that is. Only
once we have understood the underlying issue can we consider whether the
appropriate remedial action for 4.6 is to put udev back (i.e. if the real
fix is too intrusive etc)

I think the next thing to try should be to revert only the tools/libxl
portion of 2ba368d138934, i.e. return to the old toolstack code without
putting the udev scripts back (being careful to clear up any remnants of
the previous larger revert from the installed system). 

That should also be a change with no functional difference. So it will, I
think, help rule in/out any unintentional change in behaviour in (lib)xl as
opposed to some weird interaction with the inactive udev scripts.

Ian.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 30/07/15 a les 10.17, Ian Campbell ha escrit:
[...]
> The udev scripts should have been unused since 4.5.0-rc1, where they were

udev scripts have been unused since 4.2 on Dom0, not 4.5. See:

b24dc64ef34437c958b40a71f510f404e0c4bbe4
57ad6afe2a08a03c40bcd336bfb27e008e1d3e53

Roger.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 28/07/15 a les 21.47, Konrad Rzeszutek Wilk ha escrit:
> Hey,
> 
> I launch a bunch of guests at the same time or in parallel and 
> the scripts end up timing out with:
> 
> 
> Parsing config from //g-vm8.cfg
> WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
> Jul 28 19:20:53 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/5632
> libxl: error: libxl_aoutils.c:539:async_exec_timeout: killing execution of /etc/xen/scripts/block add because of timeout
> libxl: error: libxl_create.c:1157:domcreate_launch_dm: unable to add disk devices
> libxl: error: libxl_dm.c:1955:kill_device_model: unable to find device model pid in /local/domain/13/image/device-model-pid
> libxl: error: libxl.c:1606:libxl__destroy_domid: libxl__destroy_device_model failed for 13
> Jul 28 19:21:03 tst036 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/13/5632
> Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error xenstore-read backend/vbd/13/5632/node failed. backend/vbd/13/5632/hotplug-status error to xenstore.
> Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: xenstore-read backend/vbd/13/5632/node failed.
> Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/13/5632/hotplug-status error to xenstore.
> Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10344] exited with error status 1
> libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
> libxl: error: libxl.c:1569:libxl__destroy_domid: non-existant domain 13
> libxl: error: libxl.c:1527:domain_destroy_callback: unable to destroy guest with domid 13
> libxl: error: libxl.c:1454:domain_destroy_cb: destruction of domain 13 failed
> 
> And I cannot start the guest.
> 
> While if I revert the mentioned commit everything works peachy.
> 
> What is interesting is that if I have the revert I can see that the
> 
> Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/physical-device 7:d to xenstore.
> Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/hotplug-status connected to xenstore.
> 
> or often done much much later after xl create has started.
> 
> Attached is the bad log and the good log.

Can you do the same test with xl -vvv and the following patch applied 
(with and without 2ba368 reverted):

diff --git a/tools/hotplug/Linux/block b/tools/hotplug/Linux/block
index 8d2ee9d..d6f1b58 100644
--- a/tools/hotplug/Linux/block
+++ b/tools/hotplug/Linux/block
@@ -283,13 +283,19 @@ mount it read-write in a guest domain."
 
           shared_list=$(losetup -a |
                 sed -n -e "s@^\([^:]\+\)\(:[[:blank:]]\[0*${dev}\]:${inode}[[:blank:]](.*)\)@\1@p" )
+          echo "Starting sharing checks"
+          start=`date +%s`
           for dev in $shared_list
           do
             if [ -n "$dev" ]
             then
+              echo "Checking sharing of $file $dev $mode"
               check_file_sharing "$file" "$dev" "$mode"
             fi
           done
+          end=`date +%s`
+          runtime=$((end-start))
+          echo "Checks took: $runtime"
         fi
 
         loopdev=$(losetup -f 2>/dev/null || find_free_loopback_dev)

Roger. 

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Ian Campbell]

On Thu, 2015-07-30 at 10:43 +0200, Roger Pau Monné wrote:
> El 30/07/15 a les 10.17, Ian Campbell ha escrit:
> [...]
> > The udev scripts should have been unused since 4.5.0-rc1, where they 
> > were
> 
> udev scripts have been unused since 4.2 on Dom0, not 4.5. See:
> 
> b24dc64ef34437c958b40a71f510f404e0c4bbe4
> 57ad6afe2a08a03c40bcd336bfb27e008e1d3e53

My mistake.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 30/07/15 a les 10.53, Roger Pau Monné ha escrit:
> El 28/07/15 a les 21.47, Konrad Rzeszutek Wilk ha escrit:
>> Hey,
>>
>> I launch a bunch of guests at the same time or in parallel and 
>> the scripts end up timing out with:
>>
>>
>> Parsing config from //g-vm8.cfg
>> WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
>> Jul 28 19:20:53 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/5632
>> libxl: error: libxl_aoutils.c:539:async_exec_timeout: killing execution of /etc/xen/scripts/block add because of timeout
>> libxl: error: libxl_create.c:1157:domcreate_launch_dm: unable to add disk devices
>> libxl: error: libxl_dm.c:1955:kill_device_model: unable to find device model pid in /local/domain/13/image/device-model-pid
>> libxl: error: libxl.c:1606:libxl__destroy_domid: libxl__destroy_device_model failed for 13
>> Jul 28 19:21:03 tst036 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/13/5632
>> Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error xenstore-read backend/vbd/13/5632/node failed. backend/vbd/13/5632/hotplug-status error to xenstore.
>> Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: xenstore-read backend/vbd/13/5632/node failed.
>> Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/13/5632/hotplug-status error to xenstore.
>> Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
>> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10344] exited with error status 1
>> libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
>> libxl: error: libxl.c:1569:libxl__destroy_domid: non-existant domain 13
>> libxl: error: libxl.c:1527:domain_destroy_callback: unable to destroy guest with domid 13
>> libxl: error: libxl.c:1454:domain_destroy_cb: destruction of domain 13 failed
>>
>> And I cannot start the guest.
>>
>> While if I revert the mentioned commit everything works peachy.
>>
>> What is interesting is that if I have the revert I can see that the
>>
>> Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/physical-device 7:d to xenstore.
>> Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/hotplug-status connected to xenstore.
>>
>> or often done much much later after xl create has started.
>>
>> Attached is the bad log and the good log.
> 
> Can you do the same test with xl -vvv and the following patch applied 
> (with and without 2ba368 reverted):

Ping?

I've looked into this, and AFAICT you were probably using the udev 
rules (you have run_hotplug_scripts=0 in xl.conf?) before 2ba368, and 
now you are forcefully switched to launching hotplug scripts from libxl.

The issue is that you have multiple guests all using the same image 
file, so the time to execute the block hotplug script is O(n), where n 
is the number of times the same image is used:

shared_list=$(losetup -a |
      sed -n -e "s@^\([^:]\+\)\(:[[:blank:]]\[0*${dev}\]:${inode}[[:blank:]](.*)\)@\1@p" )
for dev in $shared_list
do
  if [ -n "$dev" ]
  then
    check_file_sharing "$file" "$dev" "$mode"
  fi
done

This was not a problem when using udev, because there's no timeout, but 
libxl has a hard timeout (10s) regarding hotplug script execution. The 
only way I see to solve this is to remove the checks done in the block 
hotplug script, or to increase the timeout (but since the execution 
time is not bounded this is doomed to fail if enough guests are using 
the same image).

Roger.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Ian Campbell]

On Tue, 2015-08-04 at 10:14 +0200, Roger Pau Monné wrote:
> This was not a problem when using udev, because there's no timeout, but 
> libxl has a hard timeout (10s) regarding hotplug script execution. The 
> only way I see to solve this is to remove the checks done in the block 
> hotplug script, or to increase the timeout (but since the execution 
> time is not bounded this is doomed to fail if enough guests are using 
> the same image).

If we have evidence (as we appear to) that the timeout is not sufficient
for starting large numbers of guests in // then increasing the timeout
would seem reasonable.

Perhaps someone could set an insane timeout and measure the maximum time
they see in practice in this test case. Then we could decide what a
reasonable timeout should be? Maybe we could even find a linear pattern in
the number of guests and use that to extrapolate a reasonable timeout?

That would be a small self contained fix which could be made for 4.6 I
think.

For 4.7 we could consider other options. I don't think we want to remove
the timeout altogether, but perhaps the mechanisms used for doing and/or
checking the level sharing could be optimised somewhat (perhaps by reusing
the loopback device?).

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 04/08/15 a les 10.32, Ian Campbell ha escrit:
> On Tue, 2015-08-04 at 10:14 +0200, Roger Pau Monné wrote:
>> This was not a problem when using udev, because there's no timeout, but 
>> libxl has a hard timeout (10s) regarding hotplug script execution. The 
>> only way I see to solve this is to remove the checks done in the block 
>> hotplug script, or to increase the timeout (but since the execution 
>> time is not bounded this is doomed to fail if enough guests are using 
>> the same image).
> 
> If we have evidence (as we appear to) that the timeout is not sufficient
> for starting large numbers of guests in // then increasing the timeout
> would seem reasonable.
> 
> Perhaps someone could set an insane timeout and measure the maximum time
> they see in practice in this test case. Then we could decide what a
> reasonable timeout should be? Maybe we could even find a linear pattern in
> the number of guests and use that to extrapolate a reasonable timeout?

I've created 40 HVM guest in parallel, all of them using the same file
as CD-ROM image. The highest block hotplug execution time I've measured
is 23s, so IMHO setting the timeout to 40s (or maybe 60s to really be on
the safe side?) seems reasonable. I will send a patch to do that ASAP.

Doing it based on the number of guests is not completely right, ideally
we should do it based on the number of guests using the same image file.

Here is the trace of execution times:

Time to execute: 0.00
Time to execute: 0.00
Time to execute: 0.00
Time to execute: 1.00
Time to execute: 0.00
Time to execute: 0.00
Time to execute: 1.00
Time to execute: 2.00
Time to execute: 1.00
Time to execute: 2.00
Time to execute: 2.00
Time to execute: 3.00
Time to execute: 4.00
Time to execute: 5.00
Time to execute: 7.00
Time to execute: 7.00
Time to execute: 9.00
Time to execute: 11.00
Time to execute: 13.00
Time to execute: 14.00
Time to execute: 16.00
Time to execute: 16.00
Time to execute: 12.00
Time to execute: 12.00
Time to execute: 11.00
Time to execute: 12.00
Time to execute: 12.00
Time to execute: 12.00
Time to execute: 14.00
Time to execute: 15.00
Time to execute: 15.00
Time to execute: 14.00
Time to execute: 19.00
Time to execute: 20.00
Time to execute: 19.00
Time to execute: 19.00
Time to execute: 21.00
Time to execute: 23.00
Time to execute: 21.00
Time to execute: 23.00

Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Konrad Rzeszutek Wilk]

On Tue, Aug 04, 2015 at 10:14:32AM +0200, Roger Pau Monné wrote:
> El 30/07/15 a les 10.53, Roger Pau Monné ha escrit:
> > El 28/07/15 a les 21.47, Konrad Rzeszutek Wilk ha escrit:
> >> Hey,
> >>
> >> I launch a bunch of guests at the same time or in parallel and 
> >> the scripts end up timing out with:
> >>
> >>
> >> Parsing config from //g-vm8.cfg
> >> WARNING: you seem to be using "kernel" directive to override HVM guest firmware. Ignore that. Use "firmware_override" instead if you really want a non-default firmware
> >> Jul 28 19:20:53 tst036 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/13/5632
> >> libxl: error: libxl_aoutils.c:539:async_exec_timeout: killing execution of /etc/xen/scripts/block add because of timeout
> >> libxl: error: libxl_create.c:1157:domcreate_launch_dm: unable to add disk devices
> >> libxl: error: libxl_dm.c:1955:kill_device_model: unable to find device model pid in /local/domain/13/image/device-model-pid
> >> libxl: error: libxl.c:1606:libxl__destroy_domid: libxl__destroy_device_model failed for 13
> >> Jul 28 19:21:03 tst036 logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/13/5632
> >> Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error xenstore-read backend/vbd/13/5632/node failed. backend/vbd/13/5632/hotplug-status error to xenstore.
> >> Jul 28 19:21:04 tst036 logger: /etc/xen/scripts/block: xenstore-read backend/vbd/13/5632/node failed.
> >> Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/13/5632/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/13/5632/hotplug-status error to xenstore.
> >> Jul 28 19:21:05 tst036 logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
> >> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [10344] exited with error status 1
> >> libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected.
> >> libxl: error: libxl.c:1569:libxl__destroy_domid: non-existant domain 13
> >> libxl: error: libxl.c:1527:domain_destroy_callback: unable to destroy guest with domid 13
> >> libxl: error: libxl.c:1454:domain_destroy_cb: destruction of domain 13 failed
> >>
> >> And I cannot start the guest.
> >>
> >> While if I revert the mentioned commit everything works peachy.
> >>
> >> What is interesting is that if I have the revert I can see that the
> >>
> >> Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/physical-device 7:d to xenstore.
> >> Jul 28 19:39:03 tst036 logger: /etc/xen/scripts/block: Writing backend/vbd/14/5632/hotplug-status connected to xenstore.
> >>
> >> or often done much much later after xl create has started.
> >>
> >> Attached is the bad log and the good log.
> > 
> > Can you do the same test with xl -vvv and the following patch applied 
> > (with and without 2ba368 reverted):
> 
> Ping?

Hey!
> 
> I've looked into this, and AFAICT you were probably using the udev 
> rules (you have run_hotplug_scripts=0 in xl.conf?) before 2ba368, and 

Correct. I think I needed that for driver domains and had left it in there.
> now you are forcefully switched to launching hotplug scripts from libxl.

OK.
> 
> The issue is that you have multiple guests all using the same image 
> file, so the time to execute the block hotplug script is O(n), where n 
> is the number of times the same image is used:
> 
> shared_list=$(losetup -a |
>       sed -n -e "s@^\([^:]\+\)\(:[[:blank:]]\[0*${dev}\]:${inode}[[:blank:]](.*)\)@\1@p" )
> for dev in $shared_list
> do
>   if [ -n "$dev" ]
>   then
>     check_file_sharing "$file" "$dev" "$mode"
>   fi
> done
> 
> This was not a problem when using udev, because there's no timeout, but 
> libxl has a hard timeout (10s) regarding hotplug script execution. The 
> only way I see to solve this is to remove the checks done in the block 
> hotplug script, or to increase the timeout (but since the execution 
> time is not bounded this is doomed to fail if enough guests are using 
> the same image).

Ok. I hadn't run your patch yet. Do you want me to run the latest staging
instead once more with my test-case?
> 
> Roger.
> 

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 07/08/15 a les 16.54, Konrad Rzeszutek Wilk ha escrit:
> Ok. I hadn't run your patch yet. Do you want me to run the latest staging
> instead once more with my test-case?

Yes please, 40s in my test case seemed to be fine.

Roger.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Ian Campbell]

On Fri, 2015-08-07 at 10:54 -0400, Konrad Rzeszutek Wilk wrote:
> 
> > I've looked into this, and AFAICT you were probably using the udev 
> > rules (you have run_hotplug_scripts=0 in xl.conf?) before 2ba368, and 
> 
> Correct. I think I needed that for driver domains and had left it in 
> there.

The intention was that "xl devd" would be run in the driver domain too, I
added an initscript for that purpose last week (or was it two weeks ago?)
but you could also just arrange for it to happen in /etc/rc or something.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Konrad Rzeszutek Wilk]

On Fri, Aug 07, 2015 at 04:58:57PM +0200, Roger Pau Monné wrote:
> El 07/08/15 a les 16.54, Konrad Rzeszutek Wilk ha escrit:
> > Ok. I hadn't run your patch yet. Do you want me to run the latest staging
> > instead once more with my test-case?
> 
> Yes please, 40s in my test case seemed to be fine.

I get inconsistent values. Sometimes it works, sometimes it does not.

I will try 120 on my test-bed :-(
> 
> Roger.
> 

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Roger Pau Monné]

El 12/08/15 a les 16.09, Konrad Rzeszutek Wilk ha escrit:
> On Fri, Aug 07, 2015 at 04:58:57PM +0200, Roger Pau Monné wrote:
>> El 07/08/15 a les 16.54, Konrad Rzeszutek Wilk ha escrit:
>>> Ok. I hadn't run your patch yet. Do you want me to run the latest staging
>>> instead once more with my test-case?
>>
>> Yes please, 40s in my test case seemed to be fine.
> 
> I get inconsistent values. Sometimes it works, sometimes it does not.
> 
> I will try 120 on my test-bed :-(

Thanks, this is not unexpected. IMHO the best way to solve this would be
to make the hotplug timeout settable by the user by adding a new option
to xl.conf:

hotplug_timeout=40

Allowing the user to set it to -1 in order to remove the timeout if desired.

This however seems too intrusive IMHO at this point in the release, so I
would argue for just increasing the default timeout to a suitable value.
Konrad do you have any suggestions?

Roger.

Re: Second regression due to libxl: Remove linux udev rules (2ba368d13893402b2f1fb3c283ddcc714659dd9b)

[Konrad Rzeszutek Wilk]

On Tue, Aug 18, 2015 at 09:49:43AM +0200, Roger Pau Monné wrote:
> El 12/08/15 a les 16.09, Konrad Rzeszutek Wilk ha escrit:
> > On Fri, Aug 07, 2015 at 04:58:57PM +0200, Roger Pau Monné wrote:
> >> El 07/08/15 a les 16.54, Konrad Rzeszutek Wilk ha escrit:
> >>> Ok. I hadn't run your patch yet. Do you want me to run the latest staging
> >>> instead once more with my test-case?
> >>
> >> Yes please, 40s in my test case seemed to be fine.
> > 
> > I get inconsistent values. Sometimes it works, sometimes it does not.
> > 
> > I will try 120 on my test-bed :-(
> 
> Thanks, this is not unexpected. IMHO the best way to solve this would be
> to make the hotplug timeout settable by the user by adding a new option
> to xl.conf:
> 
> hotplug_timeout=40
> 
> Allowing the user to set it to -1 in order to remove the timeout if desired.
> 
> This however seems too intrusive IMHO at this point in the release, so I
> would argue for just increasing the default timeout to a suitable value.
> Konrad do you have any suggestions?

I've been running with staging for months now and I don't see this problem.

Going forward I would think that making this work as you explained
with the 'hotplug_timeout=-1' would be nice. But then I seem to be the
only one doing it - and I don't see the normal user re-using the same
file or phy to launch tons of guests in parallel.

In my mind this is very much a feature for a developer and as such I
would say less important as a developer I can always just modify the
source code with a higher timeout.

> 
> Roger.
>