[PATCH OSSTEST v3 0/8] Add support to examine the needed memdisk flags for each hos

[PATCH OSSTEST v3 0/8] Add support to examine the needed memdisk flags for each hos

[Roger Pau Monne]

Hello,

This is the remaining memdisk examine series (ie: starting with the
first non-acked patch) and it's rebased on top of the previous FreeBSD
host install series.

The series expands the examine flight in order to test which memdisk
options should be used for each host. Hopefully all of this will be
automatic upon running a examine flight. The required options are
detected by ts-memdisk-try-append and stored into the database by
ts-examine-hostprops-save.

This has been tested except for ts-examine-hostprops-save because I
haven't run it in a flight with "real" blessing.

The series can be found at:

git://xenbits.xen.org/people/royger/osstest.git freebsd_v11

Which as said is already rebased on top of the previous FreeBSD
series.

Note that before running the FreeBSD branch flights an examine flight
with "real" intended blessing must be run, so that the proper memdisk
append options are set for each host. After that FreeBSD flights can
be run as normal.

The output of an examine flight with this series can be found at:

http://osstest.xs.citrite.net/~osstest/testlogs/logs/71931/

List of patches:

A = Acked

  1/8 HostDB: introduce set_property
A 2/8 mfi-common: move set_freebsd_runvars to
A 3/8 TestSupport: introduce
  4/8 ts-freebsd-host-install: add arguments to test
A 5/8 ts-memdisk-try-append: introduce a script to
  6/8 ts-examine-hostprops-save: introduce a script
A 7/8 make-hosts-flight: set runvars for FreeBSD
  8/8 sg-run-job: hook the memdisk test into examine

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 1/8] HostDB: introduce set_property

[Roger Pau Monne]

And provide a helper in TestSupport to use it. This allows osstest to
set host properties from test script themselves (instead of using
the mg-hosts clu).

Note that the setting of host properties is limited to flights with
intended blessing real, and it will fail for any other blessing.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Die if attempting to modify a host prop with intended blessing !=
   real.
---
 Osstest/HostDB/Executive.pm | 23 +++++++++++++++++++++++
 Osstest/HostDB/Static.pm    |  7 +++++++
 Osstest/TestSupport.pm      |  8 +++++++-
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/Osstest/HostDB/Executive.pm b/Osstest/HostDB/Executive.pm
index 300178bb..2a961b6d 100644
--- a/Osstest/HostDB/Executive.pm
+++ b/Osstest/HostDB/Executive.pm
@@ -51,6 +51,29 @@ END
     }
 }
 
+sub set_property($$$$) {
+    my ($hd, $ho, $prop, $val) = @_;
+    my $rmq = $dbh_tests->prepare(<<END);
+        DELETE FROM resource_properties
+               WHERE restype='host' and resname=? AND name=?
+END
+    my $addq = $dbh_tests->prepare(<<END);
+        INSERT INTO resource_properties (restype,resname,name,val)
+               VALUES ('host', ?,?,?)
+END
+    my $blessing = intended_blessing();
+
+    die "Attempting to modify host props with blessing $blessing != real"
+        if $blessing ne "real";
+
+    db_retry($dbh_tests, [qw(resources)], sub {
+        $rmq->execute($ho->{Name}, $prop);
+        if (length $val) {
+            $addq->execute($ho->{Name}, $prop, $val);
+       }
+    });
+}
+
 sub get_flags ($$) {
     my ($hd, $ho) = @_;
 
diff --git a/Osstest/HostDB/Static.pm b/Osstest/HostDB/Static.pm
index 60f5d3c2..3191c565 100644
--- a/Osstest/HostDB/Static.pm
+++ b/Osstest/HostDB/Static.pm
@@ -40,6 +40,13 @@ sub get_properties ($$$) { #method
     my ($hd, $name, $hp) = @_;
 }
 
+sub set_property($$$$) {
+    my ($hd, $ho, $prop, $val) = @_;
+
+    die
+    "Cannot set property in standalone mode for $ho->{Name} $prop = $val\n";
+}
+
 sub get_flags ($$) { #method
     my ($hd, $ho) = @_;
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 0af55555..27b2342c 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -80,7 +80,7 @@ BEGIN {
                       get_target_property get_host_native_linux_console
                       hostnamepath hostnamepath_list set_runtime_hostflag
                       power_state power_cycle power_cycle_sleep
-                      serial_fetch_logs
+                      serial_fetch_logs set_host_property
                       propname_massage propname_check
          
                       get_stashed open_unique_stashfile compress_stashed
@@ -1183,6 +1183,12 @@ sub get_host_property ($$;$) {
     return defined($val) ? $val : $defval;
 }
 
+sub set_host_property ($$$) {
+    my ($ho,$prop,$val) = @_;
+
+    $mhostdb->set_property($ho, $prop, $val);
+}
+
 sub get_target_property ($$;$);
 sub get_target_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 2/8] mfi-common: move set_freebsd_runvars to mfi-common

[Roger Pau Monne]

So that it can also be used by make-hosts-flight. No functional change
intended.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 make-freebsd-flight | 31 -------------------------------
 mfi-common          | 31 +++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/make-freebsd-flight b/make-freebsd-flight
index 64dfe9a6..72695742 100755
--- a/make-freebsd-flight
+++ b/make-freebsd-flight
@@ -36,37 +36,6 @@ job_create_build_filter_callback () {
     :
 }
 
-set_freebsd_runvars () {
-    # Caller should have done if required:
-    # local freebsd_runvars
-    #
-    # Figure out where are the installer binaries. The order is the
-    # following:
-    #
-    # 1. Env variable FREEBSD_<arch>_BUILDJOB: use the output from a
-    # previous build-<arch>-freebsd.
-    #
-    # 2. Env variables FREEBSD_DIST, FREEBSD_VERSION: set before calling
-    # into make-flight, provide the path to the installer image, the sets
-    # to install and the version being installed.
-    #
-    # 3. Config file FreeBSDDist, FreeBSDVersion: same as 2. except that
-    # they are set on the config file.
-    #
-    envvar="FREEBSD_${arch^^}_BUILDJOB"
-    if [ -n "${!envvar}" ]; then
-        freebsd_runvars="freebsdbuildjob=${!envvar}"
-    elif [ -n "$FREEBSD_DIST" ] && [ -n "$FREEBSD_VERSION" ]; then
-        freebsd_runvars="freebsd_distpath=$FREEBSD_DIST/$arch \
-                         freebsd_version=$FREEBSD_VERSION"
-    else
-        distpath=`getconfig "FreeBSDDist"`
-        version=`getconfig "FreeBSDVersion"`
-        freebsd_runvars="freebsd_distpath=$distpath/$arch \
-                         freebsd_version=$version"
-    fi
-}
-
 for arch in "$arches"; do
     set_freebsd_runvars
     job_create_build build-$arch-freebsd build-freebsd          \
diff --git a/mfi-common b/mfi-common
index 4827c827..8a9546ab 100644
--- a/mfi-common
+++ b/mfi-common
@@ -113,6 +113,37 @@ set_hostos_runvars () {
   esac
 }
 
+set_freebsd_runvars () {
+    # Caller should have done if required:
+    # local freebsd_runvars
+    #
+    # Figure out where are the installer binaries. The order is the
+    # following:
+    #
+    # 1. Env variable FREEBSD_<arch>_BUILDJOB: use the output from a
+    # previous build-<arch>-freebsd.
+    #
+    # 2. Env variables FREEBSD_DIST, FREEBSD_VERSION: set before calling
+    # into make-flight, provide the path to the installer image, the sets
+    # to install and the version being installed.
+    #
+    # 3. Config file FreeBSDDist, FreeBSDVersion: same as 2. except that
+    # they are set on the config file.
+    #
+    local envvar="FREEBSD_${arch^^}_BUILDJOB"
+    if [ -n "${!envvar}" ]; then
+        freebsd_runvars="freebsdbuildjob=${!envvar}"
+    elif [ -n "$FREEBSD_DIST" ] && [ -n "$FREEBSD_VERSION" ]; then
+        freebsd_runvars="freebsd_distpath=$FREEBSD_DIST/$arch \
+                         freebsd_version=$FREEBSD_VERSION"
+    else
+        local distpath=`getconfig "FreeBSDDist"`
+        local version=`getconfig "FreeBSDVersion"`
+        freebsd_runvars="freebsd_distpath=$distpath/$arch \
+                         freebsd_version=$version"
+    fi
+}
+
 create_build_jobs () {
 
   local arch
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 3/8] TestSupport: introduce hostprop_putative_record

[Roger Pau Monne]

This is used to store tentative host properties in the runvars of a
job, with the expectation that at some point (ie: at the end of the
job) they will be turned into real properties stored in the database.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v2:
 - Use the following runvar format to store the putative host props:
   hostprop/$ident/$prop=$val.
---
 Osstest/TestSupport.pm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 27b2342c..a5cca391 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -82,6 +82,7 @@ BEGIN {
                       power_state power_cycle power_cycle_sleep
                       serial_fetch_logs set_host_property
                       propname_massage propname_check
+                      hostprop_putative_record
          
                       get_stashed open_unique_stashfile compress_stashed
                       dir_identify_vcs
@@ -1189,6 +1190,12 @@ sub set_host_property ($$$) {
     $mhostdb->set_property($ho, $prop, $val);
 }
 
+sub hostprop_putative_record ($$$) {
+    my ($ho, $prop, $val) = @_;
+
+    store_runvar("hostprop/$ho->{Ident}/$prop", $val);
+}
+
 sub get_target_property ($$;$);
 sub get_target_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 4/8] ts-freebsd-host-install: add arguments to test memdisk append options

[Roger Pau Monne]

This is needed in order to figure out which memdisk options should be
used to boot the images on each specific box.

Note that when passed the --recordappend argument upon success the
script stores the tentative host property in the runvars.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Fix commit message.

Changes since v1:
 - Provide a --recordappend argument to force the recording the
   memdisk parameters.
 - Exit gracefully if a bootonly test is attempted against a
   non-supported architecture.
 - Use NONE instead of an empty string when calling
   setup_netboot_memdisk if nothing should be appended.
 - Do not perform any arch test in ts-freebsd-host-install.
---
 ts-freebsd-host-install | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
index 483b9aec..50af5dd3 100755
--- a/ts-freebsd-host-install
+++ b/ts-freebsd-host-install
@@ -41,6 +41,23 @@ use Osstest::TestSupport;
 
 tsreadconfig();
 
+our $bootonly;
+our $memdisk_append;
+our $record_append;
+while (@ARGV && $ARGV[0] =~ m/^-/g) {
+    if ($ARGV[0] =~ m/^--memdiskappend=(.*)/) {
+        $memdisk_append = $1;
+    } elsif ($ARGV[0] eq "--testboot") {
+        $memdisk_append //= "NONE";
+        $bootonly = 1;
+    } elsif ($ARGV[0] eq "--recordappend") {
+        $record_append = 1;
+    } else {
+        die "Unknown argument $ARGV[0]";
+    }
+    shift @ARGV;
+}
+
 our ($whhost) = @ARGV;
 $whhost ||= 'host';
 our $ho= selecthost($whhost);
@@ -95,7 +112,7 @@ END
 
     # Setup the pxelinux config file
     logm("Booting from installer image at $pxeimg");
-    setup_netboot_memdisk($ho, $pxeimg);
+    setup_netboot_memdisk($ho, $pxeimg, $memdisk_append);
 }
 
 sub install () {
@@ -247,6 +264,12 @@ power_state($ho, 1);
 logm("Waiting for the installer to boot");
 await_tcp(get_timeout($ho,'reboot',$timeout), 5, $ho);
 
+if ($bootonly) {
+    hostprop_putative_record($ho, "MemdiskAppend", $memdisk_append)
+        if $record_append;
+    exit 0;
+}
+
 # Next boot will be from local disk
 setup_netboot_local($ho);
 
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 5/8] ts-memdisk-try-append: introduce a script to test memdisk options

[Roger Pau Monne]

The intended usage is to run this script against every host in order
to record the possible needed memdisk flags.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v1:
 - Get the arch of the job and exit with 0 if it's not supported.
 - Pass the --recordappend argument to ts-memdisk-try-append.
---
 ts-memdisk-try-append | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100755 ts-memdisk-try-append

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
new file mode 100755
index 00000000..86c6ee41
--- /dev/null
+++ b/ts-memdisk-try-append
@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -xe -o posix
+
+arch=`perl -e '
+                use Osstest;
+                use Osstest::TestSupport;
+
+                tsreadconfig();
+                print $r{"arch"} or die $!;
+              '`
+
+case "$arch" in
+amd64)
+    ;;
+*)
+    echo "Arch $arch not supported for memdisk tests"
+    exit 0
+    ;;
+esac
+
+if ./ts-freebsd-host-install --testboot --recordappend $@; then
+    exit 0
+elif ./ts-freebsd-host-install --testboot --recordappend \
+                               --memdiskappend="raw" $@; then
+    exit 0
+fi
+
+exit 1
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 6/8] ts-examine-hostprops-save: introduce a script to save properties

[Roger Pau Monne]

This script turns the properties stored in the runvars using the
format hostprop/$ident/$prop=$val into host properties stored in the
database.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Call selecthost based on the idents passed in the putative
   hostprops runvar.
 - Fix commit message.
 - Use '/' instead of '_' in the runvars.
 - Do a dry run if flight blessing != real.
 - Fix parentheses indentation.

Changes since v1:
 - Select a host for setting the properties.
 - Print a message before exiting if blessing != real.
 - Skip properties that don't contain the selected host.
---
 ts-examine-hostprops-save | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 ts-examine-hostprops-save

diff --git a/ts-examine-hostprops-save b/ts-examine-hostprops-save
new file mode 100755
index 00000000..99a4627b
--- /dev/null
+++ b/ts-examine-hostprops-save
@@ -0,0 +1,46 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our $blessing = intended_blessing();
+
+logm("setting host properties");
+
+foreach my $k (sort keys %r) {
+    next unless $k =~ m/^hostprop\/([^\/]*)\/([^\/]*)$/;
+    my $ho = selecthost($1);
+    my $prop = $2;
+
+    logm("recording for $ho->{Name} $prop=$r{$k}");
+
+    # NB: in order to aid debug only attempt to save the host props
+    # on flights with real blessing, for the rest just do a dry run.
+    if ($blessing eq "real") {
+        set_host_property($ho, $prop, $r{$k});
+    } else {
+        logm("not saving host prop with blessing $blessing != real");
+    }
+}
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 7/8] make-hosts-flight: set runvars for FreeBSD test

[Roger Pau Monne]

This is needed in order to run the memdisk test.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 make-hosts-flight | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/make-hosts-flight b/make-hosts-flight
index 0152dfe1..d5670857 100755
--- a/make-hosts-flight
+++ b/make-hosts-flight
@@ -69,10 +69,13 @@ hosts_iterate () {
     case $kern in
       xen|linux)
         local di_version=`getconfig_TftpDiVersion_suite $suite`
+        local freebsd_runvars
+        set_freebsd_runvars
         runvars+=" 
                    kernkind=pvops
                    all_host_di_version=$di_version
                    all_host_suite=$suite
+                   $freebsd_runvars
                  "
         ;;
     esac
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH OSSTEST v3 8/8] sg-run-job: hook the memdisk test into examine

[Roger Pau Monne]

Hook the memdisk parameter detection and the saving of the host
properties into the examine jobs.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Do not pass a host ident to ts-examine-hostprops-save.
 - Use .- for ts-memdisk-try-append so that the rest of the job will
   run even if this step fails.

Changes since v1:
 - Run the memdisk test first (so that we don't leave the host in a
   weird state).
 - Pass a host to the examine-hostprops-save.
---
 sg-run-job | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index ed1ed3c8..de6e3f76 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -655,6 +655,7 @@ proc examine-host-install-xen {} {
 proc examine-host-examine {install} {
     global ok
     catching-otherwise fail {
+	run-ts -.  =            ts-memdisk-try-append + host
 	examine-host-install-$install
 	run-ts .   =            ts-examine-serial-pre + host
 	run-ts .   reboot       ts-host-reboot        + host
@@ -663,6 +664,7 @@ proc examine-host-examine {install} {
     if {$ok} {
 	run-ts -.  =           ts-examine-serial-post + host
 	run-ts .   =           ts-examine-logs-save   + host
+	run-ts .   =           ts-examine-hostprops-save
     }
 }
 
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel