[PATCH v5 00/17] osstest: initial FreeBSD support

[PATCH v5 00/17] osstest: initial FreeBSD support

[Roger Pau Monne]

Hello,

This series introduces initial FreeBSD host support to osstest. The
current series allow installing a bare-metal host with FreeBSD and
building FreeBSD on it in order to generate new install media that can
be fed into the installer script.

This is still very limited, since no Xen testing is done on those
hosts, however it sets the base to add a Xen build test for FreeBSD.

Note that it should be quite easy to add more steps to the build
script so that FreeBSD VM images are also generated, that could be
used by osstest.

I've tried to add a detailed commit log at each relevant patch, so not
much more to add here in the cover letter.

The series can also be found on my git repo:

git://xenbits.xen.org/people/royger/osstest.git freebsd_v5

List of patches:

A = Acked
N = New in this version

A 01/17 osstest: make built_stash_file store a path_ runvar
A 02/17 osstest: move known_hosts generation to TestSupport
A 03/17 osstest: introduce helper to get per-host tftp
A 04/17 osstest: introduce a helper to calculate the sha256
A 05/17 osstest: introduce a helper to setup a host to boot
N 06/17 osstest: add executive prefix to
N 07/17 osstest: introduce rename_shared_mark_ready
  08/17 osstest: add a FreeBSD host install script
A 09/17 osstest: introduce build helpers for FreeBSD
  10/17 osstest: add support for the FreeBSD package manager
  11/17 osstest: introduce a FreeBSD build script
  12/17 osstest: add support for runtime_IDENT_hostflags
  13/17 osstest: introduce a script to set the runtime
  14/17 osstest: change the meaning of need_build_host
  15/17 osstest: add support for FreeBSD buildjobs to
  16/17 osstest: introduce a script to create a FreeBSD
  17/17 osstest: hook FreeBSD flight into cr-daily-branch

The results of the FreeBSD flight can be seen at:

http://osstest.xs.citrite.net/~osstest/testlogs/logs/71669/

And the results of a xen-unstable flight with this changes:

http://osstest.xs.citrite.net/~osstest/testlogs/logs/71671/

(Sorry this is not available outside of the Citrix network)

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 01/17] osstest: make built_stash_file store a path_ runvar for each file

[Roger Pau Monne]

And introduce built_stash_debugfile in order the keep the previous
behavior of built_stash_file.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 14 ++++++++++++--
 ts-kernel-build        |  4 ++--
 ts-xen-build           |  8 ++++----
 3 files changed, 18 insertions(+), 8 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 6e19b28a..83f24005 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -85,7 +85,7 @@ BEGIN {
                       get_stashed open_unique_stashfile compress_stashed
                       dir_identify_vcs
                       build_url_vcs build_clone
-                      built_stash built_stash_file
+                      built_stash built_stash_file built_stash_debugfile
                       built_compress_stashed
                       hg_dir_revision git_dir_revision vcs_dir_revision
                       store_revision store_vcs_revision
@@ -1495,7 +1495,7 @@ END
     store_runvar("path_$item", $stashleaf);
 }
 
-sub built_stash_file ($$$$;$) {
+sub built_stash_debugfile ($$$$;$) {
     my ($ho, $builddir, $item, $fname, $optional) = @_;
     my $build= "build";
     my $stashleaf= "$build/$item";
@@ -1508,6 +1508,16 @@ sub built_stash_file ($$$$;$) {
                    "$stash/$stashleaf");
 }
 
+sub built_stash_file ($$$$;$) {
+    my ($ho, $builddir, $item, $fname, $optional) = @_;
+    my $build= "build";
+    my $stashleaf= "$build/$item";
+
+    built_stash_debugfile($ho, $builddir, $item, $fname, $optional);
+    store_runvar("path_$item", $stashleaf);
+}
+
+
 sub built_compress_stashed($) {
     my ($path) = @_;
     compress_stashed("build/$path");
diff --git a/ts-kernel-build b/ts-kernel-build
index 94e67a47..5b87f5a7 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -438,9 +438,9 @@ if ($r{tree_linuxfirmware}) {
     fwinstall();
 }
 built_stash($ho, $builddir, 'dist', 'kerndist');
-built_stash_file($ho, $builddir, 'vmlinux', 'linux/vmlinux');
+built_stash_debugfile($ho, $builddir, 'vmlinux', 'linux/vmlinux');
 built_compress_stashed('vmlinux');
-built_stash_file($ho, $builddir, 'config', 'linux/.config');
+built_stash_debugfile($ho, $builddir, 'config', 'linux/.config');
 
 sub enable_xen_config () {
     return <<'END';
diff --git a/ts-xen-build b/ts-xen-build
index 097ac0a5..3a4bcfa3 100755
--- a/ts-xen-build
+++ b/ts-xen-build
@@ -232,10 +232,10 @@ sub stash () {
     }
     built_stash($ho, $builddir, "xen/dist/xenlpt", "xenlptdist")
 	if $enable_livepatch;
-    built_stash_file($ho, $builddir, "xen-syms", "xen/xen/xen-syms", 1);
-    built_stash_file($ho, $builddir, "xen-config", "xen/.config", 1);
-    built_stash_file($ho, $builddir, "xen-hv-config", "xen/xen/.config", 1);
-    built_stash_file($ho, $builddir, "seabios-config",
+    built_stash_debugfile($ho, $builddir, "xen-syms", "xen/xen/xen-syms", 1);
+    built_stash_debugfile($ho, $builddir, "xen-config", "xen/.config", 1);
+    built_stash_debugfile($ho, $builddir, "xen-hv-config", "xen/xen/.config", 1);
+    built_stash_debugfile($ho, $builddir, "seabios-config",
 		     "xen/tools/firmware/seabios-dir-remote/.config", 1);
     built_compress_stashed("xen-syms");
 }
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 02/17] osstest: move known_hosts generation to TestSupport

[Roger Pau Monne]

This is equivalent to the already existing authorized_keys function,
and generates the contents of the known_hosts file that should be
installed on targets.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm      | 36 +-----------------------------------
 Osstest/TestSupport.pm | 41 ++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 41 insertions(+), 36 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 845027a3..f3e7582b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -714,41 +714,7 @@ sub preseed_ssh ($$) {
     my ($ho,$sfx) = @_;
 
     my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
-
-    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
-    my $hostkey= get_filecontents($hostkeyfile);
-    chomp($hostkey); $hostkey.="\n";
-    my $knownhosts= '';
-
-    my $hostsq= $dbh_tests->prepare(<<END);
-        SELECT val FROM runvars
-         WHERE flight=? AND name LIKE '%host'
-         GROUP BY val
-END
-    $hostsq->execute($flight);
-    while (my ($node) = $hostsq->fetchrow_array()) {
-        my $defaultfqdn = $node;
-        $defaultfqdn .= ".$c{TestHostDomain}" unless $defaultfqdn =~ m/\./;
-
-        my %props;
-        $mhostdb->get_properties($node, \%props);
-
-        my $longname= $props{Fqdn} // $defaultfqdn;
-        my (@hostent)= gethostbyname($longname);
-        if (!@hostent) {
-            logm("skipping host key for nonexistent host $longname");
-            next;
-        }
-        my $specs= join ',', $longname, $node, map {
-            join '.', unpack 'W4', $_;
-        } @hostent[4..$#hostent];
-        logm("adding host key for $specs");
-        $knownhosts.= "$specs ".$hostkey;
-    }
-    $hostsq->finish();
-
-    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
-    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
+    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", known_hosts());
 
     preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 83f24005..88606b04 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -91,7 +91,7 @@ BEGIN {
                       store_revision store_vcs_revision
                       git_massage_url
 
-                      sshopts authorized_keys
+                      sshopts authorized_keys known_hosts
                       cfg_tftp_di_version controller_ipaddr
                       remote_perl_script_open remote_perl_script_done
                       host_reboot target_reboot target_reboot_hard            
@@ -2415,6 +2415,45 @@ sub authorized_keys () {
     return $authkeys;
 }
 
+sub known_hosts () {
+    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
+    my $hostkey= get_filecontents($hostkeyfile);
+    chomp($hostkey); $hostkey.="\n";
+
+    my $knownhosts= '';
+
+    my $hostsq= $dbh_tests->prepare(<<END);
+        SELECT val FROM runvars
+         WHERE flight=? AND name LIKE '%host'
+         GROUP BY val
+END
+    $hostsq->execute($flight);
+    while (my ($node) = $hostsq->fetchrow_array()) {
+        my $defaultfqdn = $node;
+        $defaultfqdn .= ".$c{TestHostDomain}" unless $defaultfqdn =~ m/\./;
+
+        my %props;
+        $mhostdb->get_properties($node, \%props);
+
+        my $longname= $props{Fqdn} // $defaultfqdn;
+        my (@hostent)= gethostbyname($longname);
+        if (!@hostent) {
+            logm("skipping host key for nonexistent host $longname");
+            next;
+        }
+        my $specs= join ',', $longname, $node, map {
+            join '.', unpack 'W4', $_;
+        } @hostent[4..$#hostent];
+        logm("adding host key for $specs");
+        $knownhosts.= "$specs ".$hostkey;
+    }
+    $hostsq->finish();
+
+    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
+
+    return $knownhosts;
+}
+
 sub cfg_tftp_di_version ($) {
     my ($suite) = @_;
     $suite //= 'x def suite'; # will not find $c{...}
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 03/17] osstest: introduce helper to get per-host tftp prefix

[Roger Pau Monne]

This is used in order to get the per-host tftp prefix, used to store
the host initrd file.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v1:
 - New in this version.
---
 Osstest/TestSupport.pm | 8 +++++++-
 ts-host-install        | 2 +-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 88606b04..e30e5759 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -69,7 +69,7 @@ BEGIN {
                       target_run_apt
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
-                      target_extract_jobdistpath
+                      target_extract_jobdistpath target_tftp_prefix
                       lv_create lv_dev_mapper
 
                       poll_loop tcpconnect await_tcp
@@ -2337,6 +2337,12 @@ sub target_extract_jobdistpath ($$$$$) {
     target_cmd_root($ho, "cd / && tar -hzxf $distcopy", 300);
 }
 
+sub target_tftp_prefix ($) {
+    my ($ho) = @_;
+
+    return "$ho->{Tftp}{TmpDir}/" . hostnamepath($ho);
+}
+
 sub guest_find_domid ($$) {
     my ($ho,$gho) = @_;
     return if defined $gho->{Domid};
diff --git a/ts-host-install b/ts-host-install
index 299eae48..9f9c84fd 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -253,7 +253,7 @@ END
     push @initrds, "$initrd_overlay.cpio.gz";
 
     logm("using initrds: @initrds");
-    my $initrd= "$ho->{Tftp}{TmpDir}".hostnamepath($ho)."--initrd.gz";
+    my $initrd= target_tftp_prefix($ho) . "--initrd.gz";
     system_checked("cat -- @initrds >$ho->{Tftp}{Path}$initrd");
     
     push @dicmdline, "domain=$c{TestHostDomain}";
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 04/17] osstest: introduce a helper to calculate the sha256 of a given file

[Roger Pau Monne]

The helper is going to be used to calculate the sha256 of the FreeBSD
installer images.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v3:
 - New in this version.
---
 Osstest/TestSupport.pm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e30e5759..12ebc32d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -32,6 +32,7 @@ use File::Copy;
 use File::Basename;
 use IO::Handle;
 use Carp;
+use Digest::SHA;
 
 BEGIN {
     use Exporter ();
@@ -132,6 +133,7 @@ BEGIN {
                       guest_editconfig_nocd
                       host_install_postboot_complete
                       target_core_dump_setup
+                      sha256file
                       );
     %EXPORT_TAGS = ( );
 
@@ -2769,4 +2771,15 @@ sub target_https_mitm_proxy_setup ($) {
     target_cmd_root($ho, 'update-ca-certificates', 300);
 }
 
+sub sha256file ($;$) {
+    my ($file,$truncate) = @_;
+    my $sha = Digest::SHA->new(256);
+    my $digest;
+
+    $sha->addfile($file);
+    $digest = $sha->hexdigest;
+
+    return $truncate ? substr($digest, 0, $truncate) : $digest;
+}
+
 1;
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 05/17] osstest: introduce a helper to setup a host to boot using memdisk

[Roger Pau Monne]

The helper allows booting a host from pxelinux using an image
compatible with memdisk.

Note that this requires having a copy of the memdisk binary (provided
as part of pxelinux) in the root of the tftp server.

Deployment note: A copy of the memdisk binary (from syslinux) must be
placed in the root of the tftp server, on Debian this is accomplished
using;

ln -s /usr/lib/syslinux/memdisk /tftp/root/memdisk

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v4:
 - Add a deployment note to the commit message.

Changes since v3:
 - New in this version (split from install patch).
---
 Osstest/TestSupport.pm | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 12ebc32d..66790575 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -122,7 +122,7 @@ BEGIN {
                       await_webspace_fetch_byleaf create_webfile
                       file_link_contents get_timeout
                       setup_netboot_di setup_netboot_local host_netboot_file
-		      subst_netboot_template
+                      subst_netboot_template setup_netboot_memdisk
 
                       ether_prefix
 
@@ -2629,6 +2629,26 @@ default local
 END
 }
 
+sub setup_netboot_memdisk ($$) {
+    my ($ho, $img) = @_;
+    setup_netboot_bootcfg($ho, <<END);
+serial 0 $c{Baud}
+timeout 5
+label overwrite
+        menu label ^Overwrite
+        menu default
+        kernel memdisk
+        initrd $img
+        # NB: according to the memdisk syslinux wikipage [0]
+        # adding "append raw" is required in order to boot on
+        # some boxes, and in fact some hardware will not boot
+        # without it.
+        # [0] http://www.syslinux.org/wiki/index.php?title=MEMDISK#Memory_access_method
+        append raw
+default overwrite
+END
+}
+
 # uboot emulates pxelinux, so reuse BIOS stuff
 sub setup_netboot_di_uboot ($$$$$;%) { return &setup_netboot_di_bios; }
 sub setup_netboot_local_uboot ($) { return &setup_netboot_local_bios; }
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 06/17] osstest: add executive prefix to resource_shared_mark_ready

[Roger Pau Monne]

This is a non-functional change in preparation for introducing a
resource_shared_mark_ready in TestSupport.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - New in this version.
---
 Osstest/Executive.pm       | 5 +++--
 Osstest/JobDB/Executive.pm | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 635e5dde..217c2b08 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -53,7 +53,8 @@ BEGIN {
                       tcpconnect_queuedaemon plan_search
                       manual_allocation_base_jobinfo
                       alloc_resources alloc_resources_rollback_begin_work
-                      resource_check_allocated resource_shared_mark_ready
+                      resource_check_allocated
+                      executive_resource_shared_mark_ready
                       duration_estimator
                       db_pg_dsn opendb opendb_state
                       db_schema_updates_applied db_schema_updates_intree
@@ -994,7 +995,7 @@ END
     return $shared;
 }
 
-sub resource_shared_mark_ready ($$$) {
+sub executive_resource_shared_mark_ready ($$$) {
     my ($restype, $resname, $sharetype) = @_;
     # must run outside transaction
 
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index a7a6696f..ed087b78 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -353,7 +353,7 @@ sub gen_ether_offset ($$) { #method
 
 sub jobdb_resource_shared_mark_ready { #method
     my ($mo, $restype, $resname, $sharetype) = @_;
-    resource_shared_mark_ready($restype, $resname, $sharetype);
+    executive_resource_shared_mark_ready($restype, $resname, $sharetype);
 }
 
 sub jobdb_check_other_job { #method
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 07/17] osstest: introduce rename_shared_mark_ready

[Roger Pau Monne]

That allows marking a host as ready to be shared. Replace the current
caller that open-codes it.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - New in this version.
---
 Osstest/TestSupport.pm | 9 ++++++++-
 ts-xen-build-prep      | 4 ++--
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 66790575..77585562 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -133,7 +133,7 @@ BEGIN {
                       guest_editconfig_nocd
                       host_install_postboot_complete
                       target_core_dump_setup
-                      sha256file
+                      sha256file resource_shared_mark_ready
                       );
     %EXPORT_TAGS = ( );
 
@@ -2802,4 +2802,11 @@ sub sha256file ($;$) {
     return $truncate ? substr($digest, 0, $truncate) : $digest;
 }
 
+sub resource_shared_mark_ready($$) {
+    my ($ho,$resource) = @_;
+
+    $mjobdb->jobdb_resource_shared_mark_ready($ho->{Ident}, $ho->{Name},
+                                              $resource);
+}
+
 1;
diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 37736b7f..2e9ae6ff 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -273,5 +273,5 @@ if (!$ho->{Flags}{'no-reinstall'}) {
     ccache_setup();
     gitcache_setup();
 }
-$mjobdb->jobdb_resource_shared_mark_ready
-   ($ho->{Ident}, $ho->{Name}, "build-".$ho->{Suite}."-".$r{arch});
+
+resource_shared_mark_ready($ho, "build-".$ho->{Suite}."-".$r{arch});
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 08/17] osstest: add a FreeBSD host install script

[Roger Pau Monne]

The installation is performed using the bsdinstall tool, which is part
of the FreeBSD base system. The installer image is setup with the
osstest ssh keys and sshd enabled by default, which allows the test
harness to just ssh into the box, create the install config file and
launch the scripted install.

Currently the installation is done with ZFS only, in stripe mode, and
a single disk.

In order to support the FreeBSD installer the setup_netboot_memdisk
helper is used, that allows setting the pxe boot of a host using a
memdisk.

The install script either picks the binary images from the output of a
previous FreeBSD buildjob (yet to be introduced), or from the folder
pointed by freebsd_distpath. This folder should contain the installer
image (install.img) and the compressed install sets (kernel.txz,
base.txz) together with the MANIFEST file that contains the checksums.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Remove get_sets_path routine and xopts variable.
 - Call resource_shared_mark_ready when the host is ready to be used.

Changes since v3:
 - Use sha256file instead of the sha256sum clu.
 - Shift parameters passed to the lock and copy script, so that
   parameter 0 is ignored.
 - Copy to .tmp and then move the FreeBSD image.
 - Use a single-line find rune to cleanup old images.
 - Add some missing "set -e".
 - Don't check the output of target_cmd_output_root against undefined,
   that's never going to be true.
 - Split the setup_netboot_memdisk helper code into a pre-patch.

Changes since v2:
 - Use sysrc instead of echo to modify rc.conf.

Changes since v1:
 - Use pkg bootstrap instead of pkg update.
 - Use freebsdbuildjob instead of freebsd_buildjob.
 - add "append raw" to the memdisk command line.
 - Use with-lock-ex instead of flock.
 - Use system_checked and an array to hold the locked command to
   execute.
 - Use an array to store the list of possible disk devices.
 - Use target_putfilecontents_root_stash in order to upload the
   install script to the target.
 - Remove the usage of TftpFreeBSDBase.
 - Use target_tftp_prefix in order to get the prefix for the tftp
   target image path.
 - Add a missing linewrap.
 - Replace the split used to get the runvar paths with a regexp.
 - Place the install sets at /root/osstest_sets instead of
   /tmp/osstest_sets.
 - Pick the sets/installer image from the freebsdbuildjob
   path_freebsdist or from the runvar variable freebsd_distpath.
 - Add a comment about which FreeBSD specific inputs the script
   consumes.
---
 ts-freebsd-host-install | 256 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 256 insertions(+)
 create mode 100755 ts-freebsd-host-install

diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
new file mode 100755
index 00000000..321763b0
--- /dev/null
+++ b/ts-freebsd-host-install
@@ -0,0 +1,256 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The FreeBSD installer script consumes either one of the following
+# runvars:
+#
+# freebsd_distpath: points to a folder accessible by the controller that
+# should contain the installer image and the compressed install sets,
+# together with the MANIFEST file.
+#
+# freebsdbuildjob: points to a previous FreeBSD build job, that has
+# produced an installer image and compressed sets. The only runvar used
+# form the job pointed to by freebsdbuildjob is the path_freebsddist one,
+# that points to the folder containing the files described above.
+#
+# The installer image must be named 'install.img', and the sets
+# 'kernel.txz', 'base.txz' and finally the 'MANIFEST' file that contains
+# the checksums.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+exit 0 if $ho->{Flags}{'no-reinstall'};
+exit 0 if $ho->{SharedReady};
+
+our $timeout = 1000;
+
+our @sets = qw(base.txz kernel.txz);
+
+our $path_prefix = $r{"freebsd_distpath"} ||
+                   get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
+
+sub create_ssh_overlay () {
+    my $url = create_webfile($ho, "ssh.tar", sub {
+        my ($fh) = @_;
+        contents_make_cpio($fh, 'ustar',  "$c{OverlayLocal}/etc/ssh/");
+    });
+
+    return $url;
+}
+
+sub setup_netboot_installer () {
+    my $image = "$path_prefix/install.img";
+    my $pxeimg = target_tftp_prefix($ho) . "--freebsd.img";
+    my $hash = sha256file($image, 16);
+    my $tftp_freebsd = "$ho->{Tftp}{Path}/$ho->{Tftp}{TmpDir}/freebsd-images/";
+    my $script = <<'END';
+basedir=$1
+imagepath=$2
+sharedpath=$3
+targetpath=$4
+cd $basedir
+mkdir -p `dirname $sharedpath`
+if [ ! -f $sharedpath ]; then
+    cp $imagepath $sharedpath.tmp
+    mv $sharedpath.tmp $sharedpath
+fi
+rm -f $targetpath
+ln $sharedpath $targetpath
+# Prune old images not used anymore
+find `dirname $sharedpath` -links 1 -ctime +7 -delete
+END
+    my @cmd = ( "with-lock-ex", "-w", "$tftp_freebsd/lock",
+                "bash", "-exc", "$script", "x",
+                "$tftp_freebsd", "$image", "by-hash/$hash.img",
+                "$ho->{Tftp}{Path}/$pxeimg" );
+
+    ensuredir($tftp_freebsd);
+    system_checked(@cmd);
+
+    # Setup the pxelinux config file
+    logm("Booting from installer image at $pxeimg");
+    setup_netboot_memdisk($ho, $pxeimg);
+}
+
+sub install () {
+    my $authkeys = authorized_keys();
+    my $knownhosts = known_hosts();
+    my $sshd_keys_url = create_ssh_overlay();
+    my @disk_names = qw(ada0 da0 ad0);
+    my $target_sets = "/root/osstest_sets";
+    my $disk;
+    my $nic;
+
+    target_cmd_root($ho, 'chsh -s /bin/sh', 10);
+
+    logm("Trying to find a disk to install to");
+    $disk = target_cmd_output_root($ho, <<END, 30);
+set -e
+for disk in @disk_names; do
+    if [ -c "/dev/\$disk" ]; then
+        echo \$disk
+        exit 0
+    fi
+done
+exit 1
+END
+    logm("Using $disk as destination disk device");
+
+    logm("Trying to figure out primary nic device name");
+    $nic = target_cmd_output_root($ho, <<END, 30);
+set -e
+nics=`ifconfig -l`
+for nic in \$nics; do
+    addr=`ifconfig \$nic inet|grep inet|awk {'print \$2'}`
+    if [ "\$addr" = "$ho->{Ip}" ]; then
+        echo \$nic
+        exit 0
+    fi
+done
+exit 1
+END
+    logm("Using $nic as primary network interface");
+
+    logm("Uploading the install sets to the system");
+    target_cmd_root($ho, <<END, 30);
+set -e
+mkdir -p $target_sets
+mount -o size=1G -t tmpfs tmpfs $target_sets
+END
+
+    foreach my $set (@sets, "MANIFEST") {
+        target_putfile_root($ho, 600, "$path_prefix/$set",
+                            "$target_sets/$set");
+    }
+
+    logm("Creating the installer script");
+    target_putfilecontents_root_stash($ho, 10, <<END, '~/installscript');
+set -a
+BSDINSTALL_DISTDIR="$target_sets"
+ZFSBOOT_DISKS="$disk"
+DISTRIBUTIONS="@sets"
+nonInteractive=1
+
+#!/bin/sh
+set -ex
+
+# Setup nic and sshd
+sysrc ifconfig_$nic=DHCP
+sysrc sshd_enable=YES
+
+# Use syncronious DHCP in case the server is slow
+sysrc synchronous_dhclient=YES
+
+# Disable sendmail
+sysrc sendmail_enable=NO
+sysrc sendmail_submit_enable=NO
+sysrc sendmail_outbound_enable=NO
+sysrc sendmail_msp_queue_enable=NO
+
+# Set proxy for the pkg manager
+mkdir -p /usr/local/etc/
+cat << ENDPKG >> /usr/local/etc/pkg.conf
+pkg_env: { http_proxy = $c{HttpProxy} }
+default_always_yes: true
+assume_always_yes: true
+ENDPKG
+
+# Bootstap the package manager
+export HTTP_PROXY=$c{HttpProxy}
+export ASSUME_ALWAYS_YES=yes
+pkg bootstrap
+
+# Allow root user login and setup ssh keys
+chsh -s /bin/sh root
+echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
+mkdir -p /root/.ssh
+cat << ENDKEYS > /root/.ssh/authorized_keys
+$authkeys
+ENDKEYS
+cat << ENDHOSTS > /root/.ssh/known_hosts
+$knownhosts
+ENDHOSTS
+
+# Fetch host keys
+fetch $sshd_keys_url -o - | tar -xf - -C /etc/ssh/
+# Set correct permissions
+chown root:wheel /etc/ssh/ssh_host_*_key*
+chmod 0600 /etc/ssh/ssh_host_*_key
+chmod 0644 /etc/ssh/ssh_host_*_key.pub
+
+# Add a osstest user
+pw useradd osstest -m
+chsh -s /bin/sh osstest
+mkdir -p /home/osstest/.ssh
+cat << ENDKEYS > /home/osstest/.ssh/authorized_keys
+$authkeys
+ENDKEYS
+cat << ENDHOSTS > /home/osstest/.ssh/known_hosts
+$knownhosts
+ENDHOSTS
+
+# Setup serial console
+printf "%s" "-h -S$c{Baud}" >> /boot.config
+cat << ENDBOOT >> /boot/loader.conf
+boot_serial="YES"
+comconsole_speed="$c{Baud}"
+console="comconsole"
+boot_verbose="YES"
+beastie_disable="YES"
+ENDBOOT
+END
+
+    logm("Launch the installer");
+    target_cmd_root($ho, 'bsdinstall script installscript', 1200);
+
+    target_reboot($ho);
+
+    logm("Waiting for the host to boot");
+    await_tcp(get_timeout($ho,'reboot',$timeout), 5, $ho);
+
+    logm("FreeBSD installed succesfully");
+}
+
+# Switch off, setup PXE and switch on to the installer
+power_state($ho, 0);
+setup_netboot_installer();
+power_cycle_sleep($ho);
+power_state($ho, 1);
+
+# Wait for the host to finish booting
+logm("Waiting for the installer to boot");
+await_tcp(get_timeout($ho,'reboot',$timeout), 5, $ho);
+
+# Next boot will be from local disk
+setup_netboot_local($ho);
+
+# Proceed with the install
+install();
+
+resource_shared_mark_ready($ho, "build-freebsd-".
+                                sha256file("$path_prefix/install.img", 16));
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 09/17] osstest: introduce build helpers for FreeBSD

[Roger Pau Monne]

Some steps of the FreeBSD build process need to be done as root, so
introduce some helpers to perform those steps:
buildcmd_stamped_logged_root and target_cmd_build_root. They behave
exactly the same as the non-root counterparts.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v3:
 - New in this version.
---
 Osstest/BuildSupport.pm | 26 ++++++++++++++++++++++----
 Osstest/TestSupport.pm  | 27 +++++++++++++++++++++++----
 2 files changed, 45 insertions(+), 8 deletions(-)

diff --git a/Osstest/BuildSupport.pm b/Osstest/BuildSupport.pm
index 4c2b658b..b96ce766 100644
--- a/Osstest/BuildSupport.pm
+++ b/Osstest/BuildSupport.pm
@@ -36,7 +36,7 @@ BEGIN {
                       $whhost $ho
 
                       builddirsprops
-                      buildcmd_stamped_logged
+                      buildcmd_stamped_logged buildcmd_stamped_logged_root
                       $builddir $makeflags
 		      $buildcmd_global_prefix
 
@@ -60,9 +60,9 @@ our ($xendist);
 
 our $buildcmd_global_prefix //= '';
 
-sub buildcmd_stamped_logged ($$$$$$) {
-    my ($timeout, $component, $stampname, $prefix, $cmd, $suffix) = @_;
-    target_cmd_build($ho, $timeout, $builddir, <<END);
+sub buildcmd_stamped_logged_script ($$$$$) {
+    my ($component, $stampname, $prefix, $cmd, $suffix) = @_;
+    my $script = <<END;
         cd $component
         $buildcmd_global_prefix
         $prefix
@@ -73,6 +73,24 @@ sub buildcmd_stamped_logged ($$$$$$) {
         echo ok.
 END
 #/;
+
+    return $script;
+}
+
+sub buildcmd_stamped_logged_root ($$$$$$) {
+    my ($timeout, $component, $stampname, $prefix, $cmd, $suffix) = @_;
+    my $script = buildcmd_stamped_logged_script($component, $stampname,
+                                                $prefix, $cmd, $suffix);
+
+    target_cmd_build_root($ho, $timeout, $builddir, $script);
+}
+
+sub buildcmd_stamped_logged ($$$$$$) {
+    my ($timeout, $component, $stampname, $prefix, $cmd, $suffix) = @_;
+    my $script = buildcmd_stamped_logged_script($component, $stampname,
+                                                $prefix, $cmd, $suffix);
+
+    target_cmd_build($ho, $timeout, $builddir, $script);
 }
 
 sub selectbuildhost {
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 77585562..ab8495a9 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -56,6 +56,7 @@ BEGIN {
 
                       target_cmd_root_status target_cmd_output_root_status
                       target_cmd_root target_cmd target_cmd_build
+                      target_cmd_build_root
                       target_cmd_output_root target_cmd_output
                       target_cmd_inputfh_root sshuho
                       target_getfile target_getfile_root
@@ -669,8 +670,8 @@ sub target_editfile      ($$$;$$) { teditfileex('osstest',@_); }
     # my ($ho,$rfile, $lleaf,$rdest) = @_;
     #                 ^^^^^^^^^^^^^ optional
 
-sub target_cmd_build ($$$$) {
-    my ($ho,$timeout,$builddir,$script) = @_;
+sub target_cmd_build_script ($$$) {
+    my ($ho, $builddir, $script) = @_;
 
     my $distcc_hosts = get_host_property($ho,'DistccHosts',undef);
     my $distcc = defined($distcc_hosts) ? <<END : "";
@@ -682,8 +683,8 @@ END
 
     my $httpproxy = http_proxy_envsettings($ho);
 
-    target_cmd($ho, <<END.$distcc.<<END.$script, $timeout);
-	set -xe
+    my $cmd = <<END.$distcc.<<END.$script;
+        set -xe
         LC_ALL=C; export LC_ALL
         PATH=/usr/lib/ccache:\$PATH:/usr/lib/git-core
         $httpproxy
@@ -692,6 +693,24 @@ END
         cd $builddir
         rm -f build-ok-stamp
 END
+
+    return $cmd;
+}
+
+sub target_cmd_build ($$$$) {
+    my ($ho,$timeout,$builddir,$script) = @_;
+
+    my $cmd = target_cmd_build_script($ho, $builddir, $script);
+
+    target_cmd($ho, $cmd, $timeout);
+}
+
+sub target_cmd_build_root ($$$$) {
+    my ($ho,$timeout,$builddir,$script) = @_;
+
+    my $cmd = target_cmd_build_script($ho, $builddir, $script);
+
+    target_cmd_root($ho, $cmd, $timeout);
 }
 
 sub target_ping_check_core {
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 10/17] osstest: add support for the FreeBSD package manager

[Roger Pau Monne]

FreeBSD support is added to target_install_packages and
target_install_packages_norec, although there's no equivalent to the
--no-install-recommends in the FreeBSD package manager.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Unify more code between FreeBSD and Debian: introduce an
   OS-agnostic package_install_cmd to generate the install cmd.

Changes since v3:
 - New in this version.
---
 Osstest/TestSupport.pm | 30 ++++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ab8495a9..2928a6fa 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -68,7 +68,6 @@ BEGIN {
                       http_proxy_envsettings
                       target_editfile_root target_file_exists
                       target_editfile_kvp_replace
-                      target_run_apt
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
                       target_extract_jobdistpath target_tftp_prefix
@@ -523,19 +522,34 @@ sub target_putfile ($$$$;$) {
 sub target_putfile_root ($$$$;$) {
     tputfileex('root', @_);
 }
-sub target_run_apt {
-    my ($ho, @aptopts) = @_;
-    target_cmd_root($ho,
-        "DEBIAN_PRIORITY=critical UCF_FORCE_CONFFOLD=y \\
-            with-lock-ex -w /var/lock/osstest-apt apt-get @aptopts", 3000);
+
+sub package_install_cmd {
+    my ($norec) = @_;
+    my @hostflags = get_hostflags('host');
+    my @cmd;
+
+    if (grep /^freebsd\b/i, @hostflags) {
+        push @cmd, qw(lockf /var/run/osstest-pkg-lock pkg-static install);
+    } else {
+        push @cmd, qw(DEBIAN_PRIORITY=critical UCF_FORCE_CONFFOLD=y
+                      with-lock-ex -w /var/lock/osstest-apt apt-get);
+        push @cmd, qw(--no-install-recommends) if $norec;
+        push @cmd, qw(-y install);
+    }
+
+    return @cmd;
 }
 sub target_install_packages {
     my ($ho, @packages) = @_;
-    target_run_apt($ho, qw(-y install), @packages);
+    my @cmd = package_install_cmd();
+
+    target_cmd_root($ho,"@cmd @packages", 3000);
 }
 sub target_install_packages_norec {
     my ($ho, @packages) = @_;
-    target_run_apt($ho, qw(--no-install-recommends -y install), @packages);
+    my @cmd = package_install_cmd(1);
+
+    target_cmd_root($ho,"@cmd @packages", 3000);
 }
 
 sub target_somefile_getleaf ($$$) {
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 11/17] osstest: introduce a FreeBSD build script

[Roger Pau Monne]

In order to generate the FreeBSD installer image and the install
media.

The install sets are the vanilla ones generated by the 'ftp' release
target.  The installer image is handcrafted based on the filesystem
created by the 'bootonly' target, which is then populated with the ssh
host keys, and setup in order to use the serial console. The other
difference from upstream FreeBSD installer images is that the one
built by osstest uses a ramdisk instead of relying on the installer
media to be somehow attached, either on a CD or USB drive. This is
required in order to boot the image from pxelinux (where no CD or USB
is actually attached to the host, and everything is fetched from
tftp).

Due to the nature of the FreeBSD build, the outputs are different from
what osstest expects from a buildjob, more specifically
path_freebsddist points to a folder that contains the several outputs
form this buildjob.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Remove some useless comments.
 - Use qw(git) instead of 'git'.
 - Replace unicode nonbreaking space.
 - Remove trail ','.
 - Improve the way to fetch the version number.

Changes since v3:
 - Introduce two helpers to perform the build process.
 - Install packages using target_install_packages.

Changes since v2:
 - Increase build target timeouts.
 - Use sysrc instead echo to set rc.conf options.

Changes since v1:
 - Remove the ts-build-check FreeBSD hack.
 - Use pkg-static instead of pkg.
 - Introduce buildcmd_stamped_logged_root and target_cmd_build_root.
 - Use target_cmd_build_root and buildcmd_stamped_logged_root in the
   ts-freebsd-build script.
 - Fix the script snippets to use <<END.<<'END' in order to avoid
   escaping the shell variables.
 - Set path_freebsddist runvar to point to the folder where the build
   files are stashed.
 - Add a comment at the top of the file describing what runvars are
   consumed/produced by the build script.
---
 ts-freebsd-build | 239 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 239 insertions(+)
 create mode 100755 ts-freebsd-build

diff --git a/ts-freebsd-build b/ts-freebsd-build
new file mode 100755
index 00000000..00885636
--- /dev/null
+++ b/ts-freebsd-build
@@ -0,0 +1,239 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Consumes the following input runvars:
+#
+# revision_freebsd: FreeBSD commit ID to generate the install media from.
+# tree_freebsd: FreeBSD git tree to fetch the source code from.
+#
+# Produces the following output:
+#
+# Compressed install sets: kernel.txz, base.txz
+# Compressed debug symbols for the kernel: kernel-dbg.txz
+# Manifest file (checkums): MANIFEST
+# Installer image: install.img
+#
+# Sets the following runvar:
+#
+# path_freebsddist: points to the folder where the above files are stored.
+# freebsd_buildversion: version of FreeBSD built.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+
+selectbuildhost(\@ARGV);
+builddirsprops();
+
+sub install_deps () {
+    target_install_packages($ho, qw(git));
+}
+
+sub checkout () {
+    prepbuilddirs();
+
+    # Remove the directory as root, there might be files owned by root
+    target_cmd_build_root($ho, 300, $builddir, <<'END');
+# chflags will return error if the directory doesn't exist
+chflags -fR noschg freebsd || true
+rm -rf freebsd
+END
+
+    logm("Checkout the FreeBSD source tree");
+    build_clone($ho, 'freebsd', $builddir, 'freebsd');
+}
+
+sub build_target($$$) {
+    my ($target, $prefix, $time) = @_;
+
+    buildcmd_stamped_logged($time, 'freebsd', $target,
+                            $prefix, <<END, '');
+make $makeflags $target
+END
+}
+
+sub build_release($$$) {
+    my ($target, $prefix, $time) = @_;
+
+    buildcmd_stamped_logged_root($time, 'freebsd', "release-$target",
+                                 $prefix, <<END, '');
+make -C release $target
+END
+}
+
+sub build () {
+    my $authkeys = authorized_keys();
+    my $target = "bootonly";
+    my $prefix = <<END;
+export MAKEOBJDIRPREFIX=$builddir/obj
+export TARGET=$r{arch}
+END
+
+    # Build process as documented in the handbook:
+    # https://www.freebsd.org/doc/handbook/updating-src.html
+
+    logm("Cleaning up previous builds");
+    build_target('cleanworld', $prefix, 300);
+
+    logm("Building world");
+    build_target('buildworld', $prefix, 25200);
+
+    logm("Building kernel");
+    build_target('buildkernel', $prefix, 3600);
+
+    # NB: the steps below need to be done as root or the permissions
+    # of the files won't be properly set (and the target will fail).
+    logm("Creating the install sets");
+    build_release('ftp', $prefix, 3600);
+
+    logm("Populating the installer image");
+    build_release($target, $prefix, 3600);
+
+    logm("Placing ssh host keys");
+    foreach my $file (<$c{OverlayLocal}/etc/ssh/ssh_host_*_key*>) {
+        target_putfile_root($ho, 30, $file,
+                            "$builddir/freebsd/release/$target/etc/ssh/");
+    }
+
+    logm("Configuring the installer image");
+    target_cmd_build_root($ho, 30, $builddir, <<END.<<'END');
+authkeys="$authkeys"
+bauds="$c{Baud}"
+cd freebsd/release/$target
+END
+sysrc -f etc/rc.conf sshd_enable=YES
+
+echo 'PermitRootLogin yes' >> etc/ssh/sshd_config
+mkdir -p root/.ssh
+cat << ENDKEYS > root/.ssh/authorized_keys
+$authkeys
+ENDKEYS
+
+chown root:wheel etc/ssh/ssh_host_*_key*
+chmod 0600 etc/ssh/ssh_host_*_key
+chmod 0644 etc/ssh/ssh_host_*_key.pub
+
+# Setup serial console output for stage1
+printf "%s" "-h -S$bauds" >> boot.config
+cat << ENDBOOT >> boot/loader.conf
+# Serial console configuration
+boot_serial="YES"
+comconsole_speed="$bauds"
+console="comconsole"
+boot_verbose="YES"
+beastie_disable="YES"
+
+# mfs boot parameters
+mfs_load="YES"
+mfs_type="mfs_root"
+mfs_name="/mfsroot"
+vfs.root.mountfrom="ufs:/dev/ufs/FreeBSD_Install"
+ENDBOOT
+
+sysrc -f etc/rc.conf ifconfig_DEFAULT=DHCP
+
+# Remove the local script that launches the installer by default
+rm -rf etc/rc.local
+
+# Create a temporary fstab with the root dir
+echo '/dev/ufs/FreeBSD_Install / ufs rw 1 1' > etc/fstab
+
+# Remove the linked resolv.conf
+rm -rf etc/resolv.conf
+END
+
+    logm("Create the installer");
+    target_cmd_build_root($ho, 900, $builddir, <<END.<<'END');
+target="freebsd/release/$target"
+output="install.img"
+END
+mkdir -p $output.tmp
+
+# Do some pruning
+rm -rf $target/usr/share/man
+rm -rf $target/usr/share/examples
+rm -rf $target/usr/share/doc
+rm -rf $target/usr/share/dtrace
+
+# Create a mfs root image
+makefs -b 10% -B little -o label=FreeBSD_Install $output.tmp/mfsroot $target
+gzip $output.tmp/mfsroot
+
+# Copy boot to the staging dir
+cp -r $target/boot $output.tmp/
+cp $target/boot.config $output.tmp/
+
+# The loader doesn't need any modules in order to boot into the mfsroot.
+# The rest of the modules can be loaded from the mfs root itself.
+rm -f $output.tmp/boot/kernel/*.ko
+
+gzip $output.tmp/boot/kernel/kernel
+
+makefs -B little $output.part $output.tmp
+
+mkimg -s gpt -b $target/boot/pmbr -p efi:=$target/boot/boot1.efifat \
+    -p freebsd-boot:=$target/boot/gptboot -p freebsd-ufs:=$output.part \
+    -p freebsd-swap::1M -o $output
+
+rm $output.part
+rm -rf $output.tmp
+END
+}
+
+sub stash () {
+    my @sets = qw(MANIFEST base.txz kernel.txz);
+    my @symbols = qw(kernel-dbg.txz);
+
+    logm("Stashing FreeBSD build output");
+    foreach my $set (@sets) {
+        built_stash_file($ho, $builddir, $set,
+                         "freebsd/release/ftp/$set", 0);
+    }
+    foreach my $symbol (@symbols) {
+        built_stash_debugfile($ho, $builddir, $symbol,
+                              "freebsd/release/ftp/$symbol", 0);
+    }
+    built_stash_file($ho, $builddir, "install.img", "install.img", 0);
+
+    my $srcversion = target_cmd_output_root($ho, <<END, 30);
+set -e
+cd $builddir/freebsd
+eval `make buildenvvars`
+test -n "\$SRCRELDATE"
+expr "\$SRCRELDATE" / 100000
+END
+    store_runvar("freebsd_buildversion", "$srcversion");
+
+    # Set path_freebsddist to point to the build output folder
+    # in order to make ts-build-check happy.
+    store_runvar("path_freebsddist", "build/");
+}
+
+install_deps();
+checkout();
+build();
+stash();
+
+logm("FreeBSD build successful");
+
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 12/17] osstest: add support for runtime_IDENT_hostflags

[Roger Pau Monne]

This is required for FreeBSD, that will need to set some of the
hostflags at runtime. The current IDENT_hostflags will be keep as-is,
and they should only be set at job creation time.

Also introduce a helper to set the runtime hostflags.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Allow runtime flags to be appended instead of overwritten.

Changes since v3:
 - New in this version.
---
 Osstest/TestSupport.pm | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2928a6fa..16cf1fe3 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -78,7 +78,7 @@ BEGIN {
 
                       selecthost get_hostflags get_host_property
                       get_target_property get_host_native_linux_console
-                      hostnamepath hostnamepath_list
+                      hostnamepath hostnamepath_list set_runtime_hostflag
                       power_state power_cycle power_cycle_sleep
                       serial_fetch_logs
                       propname_massage propname_check
@@ -1584,10 +1584,20 @@ sub get_hostflags ($) {
     my ($ident) = @_;
     # may be run outside transaction, or with flights locked
     my $flags= get_runvar_default('all_hostflags',     $job, '').','.
-               get_runvar_default("${ident}_hostflags", $job, '');
+               get_runvar_default("${ident}_hostflags", $job, '').','.
+               get_runvar_default("runtime_${ident}_hostflags", $job, '');
     return grep /./, split /\,/, $flags;
 }
 
+sub set_runtime_hostflag ($$) {
+    my ($ident,$value) = @_;
+    my @current = split /,/, get_runvar_default("runtime_${ident}_hostflags",
+                                                $job, '');
+
+    push @current, (split /,/, $value);
+    store_runvar("runtime_${ident}_hostflags", (join ',', @current));
+}
+
 sub host_involves_pcipassthrough ($) {
     my ($ho) = @_;
     return !!grep m/^pcipassthrough\-/, get_hostflags($ho->{Ident});
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 13/17] osstest: introduce a script to set the runtime hostflags runvar for FreeBSD jobs

[Roger Pau Monne]

Due to the nature of the FreeBSD install media, which is
self-generated from the ts-freebsd-build script, the hostflags runvar
set to FreeBSD jobs are related to the current version under test.

The following hostflags might need to be fetched from the runvars of a
previous build-$arch-freebsd job:

 - share-build-freebsd-$arch-$hash: the $hash used here is calculated
   from the checksum of the installer image used by this specific job.
   This allows osstest to share FreeBSD build hosts, and be sure the
   exact desired FreeBSD version is used.
 - freebsd-$version: $version contains the major FreeBSD version under
   test. This version is obtained from the FreeBSD source code used to
   create the install media.

Note that the share-build-* hostflag is optional, and will only be set
for build jobs.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Make the share-build- flag optional.

Changes since v3:
 - Make use of the set_runtime_hostflag helper.
 - Make use of the sha256file helper.

Changes since v2:
 - New in this version.
---
 ts-freebsd-set-hostflags | 72 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100755 ts-freebsd-set-hostflags

diff --git a/ts-freebsd-set-hostflags b/ts-freebsd-set-hostflags
new file mode 100755
index 00000000..c5ba41ef
--- /dev/null
+++ b/ts-freebsd-set-hostflags
@@ -0,0 +1,72 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This script sets the host_hostflags for a FreeBSD job based on the runvars
+# provided and the contents of the extra_hostflags runvar.
+#
+# If the freebsd_distpath runvar is set the installer image will be retrieved
+# from "freebsd_distpath"/install.img, and the FreeBSD version from the
+# freebsd_version runvar. Note that both those runvars should be set on the
+# current job.
+#
+# If freebsd_distpath is not set, it is assumed that freebsdbuildjob runvar is
+# set and the installer image will be retrieved from the path pointed to by
+# "path_freebsdddist"/install.img, and the FreeBSD version will be obtained
+# from the "freebsd_buildversion" runvar. Both of those runvars belong to the
+# flight and job pointed to by freebsdbuildjob.
+#
+# As output upon successful completion this script will set the host_hostflags
+# runvar for the current job. Note that this _must_ be done before running
+# ts-host-allocate.
+#
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our $share;
+if (@ARGV && $ARGV[0] eq "--share") {
+    $share = 1;
+    shift @ARGV;
+}
+
+sub get_freebsd_image_hash() {
+    my $distpath =  $r{"freebsd_distpath"} ||
+                    get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
+
+    return sha256file("$distpath/install.img", 16);
+}
+
+sub get_freebsd_version() {
+    return $r{"freebsd_version"} ||
+           get_runvar("freebsd_buildversion", $r{"freebsdbuildjob"});
+}
+
+my $version = get_freebsd_version();
+set_runtime_hostflag("host", "freebsd-$version");
+
+if ($share) {
+    my $hash = get_freebsd_image_hash();
+
+    set_runtime_hostflag("host", "share-build-freebsd-$hash");
+}
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 14/17] osstest: change the meaning of need_build_host

[Roger Pau Monne]

Make need_build_host store a string instead of a boolean. This is
later going to be expanded to handle the FreeBSD build jobs.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Fix the usage of the catching-otherwise blocks.

Changes since v3:
 - New in this version (split from patch).
---
 sg-run-job | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index b1f94f4d..65e0a834 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -52,12 +52,12 @@ proc run-job {job} {
     set skip_globs     [jobdb::read-runvar $flight $job skip_testids]
 
     set nh [need-hosts/$jobinfo(recipe)]
-    if {![string compare $nh BUILD]} {
+    if {[string match BUILD_* $nh]} {
         set need_xen_hosts {}
-        set need_build_host 1
+        set need_build_host [string range $nh [expr [string first _ $nh] + 1] end]
     } else {
         set need_xen_hosts $nh
-        set need_build_host 0
+        set need_build_host {}
     }
     set nested_layers_hosts {}
 
@@ -68,7 +68,9 @@ proc run-job {job} {
         eval run-ts broken  =             ts-hosts-allocate + $need_xen_hosts
     }
 
-    if {$need_build_host} { catching-otherwise broken allocate-build-host }
+    if {[llength $need_build_host]} {
+        catching-otherwise broken { allocate-build-host $need_build_host }
+    }
 
     if {$ok} { setstatus running                                          }
 
@@ -77,7 +79,11 @@ proc run-job {job} {
 
     if {$ok} { set syslog [spawn-ts broken = | ts-syslog-server]          }
 
-    if {$need_build_host} { catching-otherwise broken prepare-build-host }
+    if {[llength $need_build_host]} {
+        catching-otherwise broken {
+            prepare-build-host-[string tolower $need_build_host]
+        }
+    }
 
     per-host-ts broken  host-install/@(*) ts-host-install-twice
 
@@ -95,7 +101,7 @@ proc run-job {job} {
         set need_xen_hosts [lunappend nested_layers_hosts]
     }
 
-    if {$need_build_host && !$ok} {
+    if {[llength $need_build_host] && !$ok} {
 	run-ts  !broken capture-logs      ts-logs-capture + host
     }
 
@@ -116,7 +122,7 @@ proc run-job {job} {
 
     if {$ok} { setstatus pass                                             }
 
-    if {$need_build_host && $ok} { jobdb::preserve-task 90 }
+    if {[llength $need_build_host] && $ok} { jobdb::preserve-task 90 }
 
     if {!$ok} {
         jobdb::logputs stdout "job not ok"
@@ -671,11 +677,11 @@ proc need-hosts/host-examine-linux {} { examine-host-examine debian }
 
 #---------- builds ----------
 
-proc need-hosts/build {} { return BUILD }
-proc need-hosts/build-kern {} { return BUILD }
-proc need-hosts/build-libvirt {} { return BUILD }
-proc need-hosts/build-rumprun {} { return BUILD }
-proc need-hosts/build-xtf {} { return BUILD }
+proc need-hosts/build {}                { return BUILD_LINUX }
+proc need-hosts/build-kern {}           { return BUILD_LINUX }
+proc need-hosts/build-libvirt {}        { return BUILD_LINUX }
+proc need-hosts/build-rumprun {}        { return BUILD_LINUX }
+proc need-hosts/build-xtf {}            { return BUILD_LINUX }
 
 proc run-job/build {} {
     run-ts . = ts-xen-build
@@ -702,11 +708,11 @@ proc run-job/build-xtf {} {
     run-ts . = ts-xtf-build
 }
 
-proc allocate-build-host {} {
+proc allocate-build-host {ostype} {
     global jobinfo
     run-ts broken = ts-hosts-allocate + host
 }
-proc prepare-build-host {} {
+proc prepare-build-host-linux {} {
     global jobinfo
     run-ts broken host-install(*) ts-host-install-twice
     run-ts . host-build-prep ts-xen-build-prep
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 15/17] osstest: add support for FreeBSD buildjobs to sg-run-job

[Roger Pau Monne]

Add support and introduce a FreeBSD build job to sg-run-job.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Use a switch in allocate-build-host.

Changes since v3:
 - New in this version (split from existing patch).
---
 sg-run-job | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 65e0a834..28bed621 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -682,6 +682,7 @@ proc need-hosts/build-kern {}           { return BUILD_LINUX }
 proc need-hosts/build-libvirt {}        { return BUILD_LINUX }
 proc need-hosts/build-rumprun {}        { return BUILD_LINUX }
 proc need-hosts/build-xtf {}            { return BUILD_LINUX }
+proc need-hosts/build-freebsd {}        { return BUILD_FREEBSD }
 
 proc run-job/build {} {
     run-ts . = ts-xen-build
@@ -708,8 +709,16 @@ proc run-job/build-xtf {} {
     run-ts . = ts-xtf-build
 }
 
+proc run-job/build-freebsd {} {
+    run-ts . = ts-freebsd-build
+}
+
 proc allocate-build-host {ostype} {
     global jobinfo
+    switch -exact $ostype {
+        FREEBSD { run-ts broken = ts-freebsd-set-hostflags --share }
+        default {}
+    }
     run-ts broken = ts-hosts-allocate + host
 }
 proc prepare-build-host-linux {} {
@@ -718,6 +727,11 @@ proc prepare-build-host-linux {} {
     run-ts . host-build-prep ts-xen-build-prep
 }
 
+proc prepare-build-host-freebsd {} {
+    global jobinfo
+    run-ts broken host-install(*) ts-freebsd-host-install
+}
+
 proc need-hosts/coverity {} { return BUILD }
 proc run-job/coverity {} {
     run-ts . = ts-coverity-build + host
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 16/17] osstest: introduce a script to create a FreeBSD flight

[Roger Pau Monne]

The logic to create a FreeBSD build job is added to
make-freebsd-flight. This includes creating a FreeBSD build job, and
also testing the output of that build job (by creating another build
job that depends on the output of the first).

Note that the FreeBSD build job needs some input in order to setup a
FreeBSD host, and that can be fetched from different places:

1. Env variable FREEBSD_BUILDJOB: use the output from a previous
build-<arch>-freebsd.

2. Env variables FREEBSD_DIST and FREEBSD_VERSION: set before calling
into make-flight, provide the path to the installer image and sets
folder and the version being installed.

3. Config file FreeBSDDist and FreeBSDVersion: same as 2. except that
they are set on the config file.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v4:
 - Convert get_freebsdjob_runvars into set_freebsd_runvars, mimic the
   behavior of set_hostos_runvars.

Changes since v3:
 - Split the sg-run-job code into pre-patches.

Changes since v1:
 - Replace freebsd_buildjob with freebsdbuildjob.
 - Replace FREEBSD_SETS/IMAGE with a single FREEBSD_DIST that points
   to a folder that should contain both things.
 - Document the FreeBSDDist and FreeBSDVersion config file options in
   the README file.
---
 README              | 11 +++++++
 ap-common           |  4 +++
 make-freebsd-flight | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 102 insertions(+)
 create mode 100755 make-freebsd-flight

diff --git a/README b/README
index b45058da..51243974 100644
--- a/README
+++ b/README
@@ -423,6 +423,17 @@ HostGroupFlags_<group>
 DebianPreseed
    Text to add to the debian-installer preseed file.  Optional.
 
+FreeBSDDist
+   Path to the folder that contains the FreeBSD install image and
+   the FreeBSD compressed install sets, together with the MANIFEST
+   file that holds the checksums. This is required in order to run
+   a FreeBSD host install if no previous FreeBSD buildjob is
+   available (ie: for example when running in standalone mode).
+
+FreeBSDVersion
+   Numeric value holding the major FreeBSD version of the media
+   provided in FreeBSDDist (ie: 12).
+
 ========================================
 
 Config settings relevant only to standalone mode
diff --git a/ap-common b/ap-common
index cbb815ce..d4fa7aef 100644
--- a/ap-common
+++ b/ap-common
@@ -37,6 +37,10 @@
 : ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/xtf.git}
 : ${BASE_TREE_XTF:=git://xenbits.xen.org/xtf.git}
 
+: ${TREE_FREEBSD:=git://github.com/freebsd/freebsd.git}
+: ${PUSH_TREE_FREEBSD:=$XENBITS:/home/xen/git/freebsd.git}
+: ${BASE_TREE_FREEBSD:=git://xenbits.xen.org/freebsd.git}
+
 : ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git}
 : ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git}
 : ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git}
diff --git a/make-freebsd-flight b/make-freebsd-flight
new file mode 100755
index 00000000..5f31d8c7
--- /dev/null
+++ b/make-freebsd-flight
@@ -0,0 +1,87 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -e -o posix
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ./cri-common
+. ./ap-common
+. ./mfi-common
+
+arch=amd64
+
+job_create_build_filter_callback () {
+    :
+}
+
+set_freebsd_runvars () {
+    # Caller should have done if required:
+    # local freebsd_runvars
+    #
+    # Figure out where are the installer binaries. The order is the
+    # following:
+    #
+    # 1. Env variable FREEBSD_BUILDJOB: use the output from a previous
+    # build-<arch>-freebsd.
+    #
+    # 2. Env variables FREEBSD_DIST, FREEBSD_VERSION: set before calling
+    # into make-flight, provide the path to the installer image, the sets
+    # to install and the version being installed.
+    #
+    # 3. Config file FreeBSDDist, FreeBSDVersion: same as 2. except that
+    # they are set on the config file.
+    #
+    if [ -n "$FREEBSD_BUILDJOB" ]; then
+        freebsd_runvars="freebsdbuildjob=$FREEBSD_BUILDJOB"
+    elif [ -n "$FREEBSD_DIST" ] && [ -n "$FREEBSD_VERSION" ]; then
+        freebsd_runvars="freebsd_distpath=$FREEBSD_DIST \
+                         freebsd_version=$FREEBSD_VERSION"
+    else
+        distpath=`getconfig "FreeBSDDist"`
+        version=`getconfig "FreeBSDVersion"`
+        freebsd_runvars="freebsd_distpath=$distpath \
+                         freebsd_version=$version"
+    fi
+}
+
+set_freebsd_runvars
+job_create_build build-$arch-freebsd build-freebsd                       \
+            arch=$arch                                                   \
+            $RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
+            tree_freebsd=$TREE_FREEBSD                                   \
+            revision_freebsd=$REVISION_FREEBSD                           \
+            host_hostflags=arch-$arch,purpose-build                      \
+            $freebsd_runvars
+
+# Create an identical job that's going to use the build output from
+# the previous one.
+job_create_build build-$arch-freebsd-again build-freebsd                 \
+            arch=$arch                                                   \
+            $RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
+            host_hostflags=arch-$arch,purpose-build                      \
+            tree_freebsd=$TREE_FREEBSD                                   \
+            revision_freebsd=$REVISION_FREEBSD                           \
+            freebsdbuildjob=build-$arch-freebsd
+
+echo $flight
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v5 17/17] osstest: hook FreeBSD flight into cr-daily-branch

[Roger Pau Monne]

Add the missing pieces so that cr-daily-branch knows how to create a
FreeBSD flight. This has been tested by calling `./cr-daily-branch
freebsd`.

Note that there are two missing pieces. The first one is correctly
setting the FREEBSD_BUILDJOB environment variable when calling
make-freebsd-flight, without it the installer media will always be
picked from the configuration file. This requires keeping track of
previous FreeBSD successful build jobs, marking them as permanent and
fetching them from the database in order to pass them on the env
variable.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v3:
 - Fix the daily-cron-email-* settings for FreeBSD branch flights.
---
 ap-common                       | 1 +
 ap-fetch-version                | 4 ++++
 ap-fetch-version-old            | 5 +++++
 ap-print-url                    | 3 +++
 ap-push                         | 5 +++++
 cr-daily-branch                 | 9 +++++++++
 cr-for-branches                 | 2 +-
 cri-common                      | 1 +
 daily-cron-email-adhoc--freebsd | 1 +
 daily-cron-email-play--freebsd  | 1 +
 daily-cron-email-real--freebsd  | 4 ++++
 11 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100644 daily-cron-email-adhoc--freebsd
 create mode 100644 daily-cron-email-play--freebsd
 create mode 100644 daily-cron-email-real--freebsd

diff --git a/ap-common b/ap-common
index d4fa7aef..0d8a667a 100644
--- a/ap-common
+++ b/ap-common
@@ -86,6 +86,7 @@ fi
 : ${LOCALREV_SEABIOS:=daily-cron.$branch}
 : ${LOCALREV_OVMF:=daily-cron.$branch}
 : ${LOCALREV_XTF:=daily-cron.$branch}
+: ${LOCALREV_FREEBSD:=daily-cron.$branch}
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index a107c93d..365ae018 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -106,6 +106,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
 	;;
+freebsd)
+	repo_tree_rev_fetch_git freebsd \
+		$TREE_FREEBSD master $LOCALREV_FREEBSD
+	;;
 osstest)
         if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
 	    git update-ref -m "Arranging to test HEAD" \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 3cbc1768..9b086401 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -34,6 +34,7 @@ check_ap_fetch_placeholders
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_XTF:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
+: ${BASE_LOCALREV_FREEBSD:=daily-cron.$branch.old}
 : ${BASE_TAG_LIBVIRT:=xen-tested-master}
 
 if info_linux_tree "$branch"; then
@@ -114,6 +115,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$BASE_TREE_OVMF xen-tested-master $BASE_LOCALREV_OVMF
 	;;
+freebsd)
+	repo_tree_rev_fetch_git freebsd \
+		$BASE_TREE_FREEBSD xen-tested-master $BASE_LOCALREV_FREEBSD
+	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
 	    git fetch -f $HOME/testing.git production:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index 93c14b35..57126292 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -64,6 +64,9 @@ xtf)
 ovmf)
 	echo $TREE_OVMF_UPSTREAM
 	;;
+freebsd)
+	echo $TREE_FREEBSD
+	;;
 osstest)
 	echo none:;
 	;;
diff --git a/ap-push b/ap-push
index a27ccc22..5b6ce7fc 100755
--- a/ap-push
+++ b/ap-push
@@ -41,6 +41,7 @@ TREE_RUMPRUN=$PUSH_TREE_RUMPRUN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
 TREE_XTF=$PUSH_TREE_XTF
+TREE_FREEBSD=$PUSH_TREE_FREEBSD
 
 if info_linux_tree "$branch"; then
 	cd $repos/linux
@@ -125,6 +126,10 @@ xtf)
 	cd $repos/xtf
 	git push $TREE_XTF $revision:refs/heads/xen-tested-master
 	;;
+freebsd)
+	cd $repos/freebsd
+	git push $TREE_FREEBSD $revision:refs/heads/xen-tested-master
+	;;
 ovmf)
 	cd $repos/ovmf
 	git push $TREE_OVMF $revision:refs/heads/xen-tested-master
diff --git a/cr-daily-branch b/cr-daily-branch
index 39483cda..7bc2c1da 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -207,6 +207,10 @@ if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
 	determine_version REVISION_LINUXFIRMWARE linuxfirmware LINUXFIRMWARE
         export REVISION_LINUXFIRMWARE
 fi
+if [ "x$REVISION_FREEBSD" = x ]; then
+	determine_version REVISION_FREEBSD freebsd FREEBSD
+	export REVISION_FREEBSD
+fi
 
 case "$tree" in
 xen)
@@ -255,6 +259,10 @@ ovmf)
 	realtree=ovmf
 	NEW_REVISION=$REVISION_OVMF
 	;;
+freebsd)
+	realtree=freebsd
+	NEW_REVISION=$REVISION_FREEBSD
+	;;
 *)
 	NEW_REVISION=''
 	wantpush=false
@@ -274,6 +282,7 @@ examine)	makeflight=make_hosts_flight
 			./make-hosts-flight $1 $2 blessed-$3 $3 $4
 		}
 		;;
+freebsd)	makeflight=./make-freebsd-flight ;;
 *)         	makeflight=./make-flight ;;
 esac
 
diff --git a/cr-for-branches b/cr-for-branches
index a6912735..23e4ca84 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
 LOGFILE=tmp/cr-for-branches.log
 export LOGFILE
 
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-4.8-testing xen-4.9-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing qemu-upstream-4.8-testing qemu-upstream-4.9-testing linux-linus linux-4.9 linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-4.8-testing xen-4.9-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing qemu-upstream-4.8-testing qemu-upstream-4.9-testing linux-linus linux-4.9 linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf freebsd ${EXTRA_BRANCHES}}
 export BRANCHES
 
 fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index 903fb4e2..a3e679cf 100644
--- a/cri-common
+++ b/cri-common
@@ -80,6 +80,7 @@ select_xenbranch () {
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	xtf)			tree=xtf;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
+	freebsd)		tree=freebsd;	xenbranch=xen-unstable ;;
 	distros-*|examine)	tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
 	esac
diff --git a/daily-cron-email-adhoc--freebsd b/daily-cron-email-adhoc--freebsd
new file mode 100644
index 00000000..b64821af
--- /dev/null
+++ b/daily-cron-email-adhoc--freebsd
@@ -0,0 +1 @@
+To: royger@FreeBSD.org
diff --git a/daily-cron-email-play--freebsd b/daily-cron-email-play--freebsd
new file mode 100644
index 00000000..b64821af
--- /dev/null
+++ b/daily-cron-email-play--freebsd
@@ -0,0 +1 @@
+To: royger@FreeBSD.org
diff --git a/daily-cron-email-real--freebsd b/daily-cron-email-real--freebsd
new file mode 100644
index 00000000..6ee2fa4d
--- /dev/null
+++ b/daily-cron-email-real--freebsd
@@ -0,0 +1,4 @@
+To: xen-devel@lists.xenproject.org,
+    osstest-admin@xenproject.org,
+    royger@FreeBSD.org
+Bcc: osstest-output@lists.xenproject.org
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 06/17] osstest: add executive prefix to resource_shared_mark_ready

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 06/17] osstest: add executive prefix to resource_shared_mark_ready"):
> This is a non-functional change in preparation for introducing a
> resource_shared_mark_ready in TestSupport.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 07/17] osstest: introduce rename_shared_mark_ready

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 07/17] osstest: introduce rename_shared_mark_ready"):
> That allows marking a host as ready to be shared. Replace the current
> caller that open-codes it.

You got the Subject wrong.  "rename" for "resource".  :-)

With that fixed:

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 08/17] osstest: add a FreeBSD host install script

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 08/17] osstest: add a FreeBSD host install script"):
> The installation is performed using the bsdinstall tool, which is part
> of the FreeBSD base system. The installer image is setup with the
> osstest ssh keys and sshd enabled by default, which allows the test
> harness to just ssh into the box, create the install config file and
> launch the scripted install.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 10/17] osstest: add support for the FreeBSD package manager

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 10/17] osstest: add support for the FreeBSD package manager"):
> FreeBSD support is added to target_install_packages and
> target_install_packages_norec, although there's no equivalent to the
> --no-install-recommends in the FreeBSD package manager.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
...
> +sub package_install_cmd {
                          ^
                           (;$)

Should have a prototype.  Sorry for not spotting this before.

With that fixed,

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 11/17] osstest: introduce a FreeBSD build script

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 11/17] osstest: introduce a FreeBSD build script"):
> In order to generate the FreeBSD installer image and the install
> media.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 12/17] osstest: add support for runtime_IDENT_hostflags

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 12/17] osstest: add support for runtime_IDENT_hostflags"):
> This is required for FreeBSD, that will need to set some of the
> hostflags at runtime. The current IDENT_hostflags will be keep as-is,
> and they should only be set at job creation time.
> 
> Also introduce a helper to set the runtime hostflags.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 13/17] osstest: introduce a script to set the runtime hostflags runvar for FreeBSD jobs

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 13/17] osstest: introduce a script to set the runtime hostflags runvar for FreeBSD jobs"):
> Due to the nature of the FreeBSD install media, which is
> self-generated from the ts-freebsd-build script, the hostflags runvar
> set to FreeBSD jobs are related to the current version under test.
> 
> The following hostflags might need to be fetched from the runvars of a
> previous build-$arch-freebsd job:
...
> +our $share;
> +if (@ARGV && $ARGV[0] eq "--share") {
> +    $share = 1;
> +    shift @ARGV;
> +}

I think the remaining arguments should be host idents.

Also you should check that the first ident doesn't start with -.
(simply calling die if it does is fine).

> +my $version = get_freebsd_version();
> +set_runtime_hostflag("host", "freebsd-$version");

Specifically, you should iterate that, and this ...

> +if ($share) {
> +    my $hash = get_freebsd_image_hash();
> +
> +    set_runtime_hostflag("host", "share-build-freebsd-$hash");
> +}

for each entry in @ARGV.

That way this script can be used for pair tests etc.

ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 14/17] osstest: change the meaning of need_build_host

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 14/17] osstest: change the meaning of need_build_host"):
> Make need_build_host store a string instead of a boolean. This is
> later going to be expanded to handle the FreeBSD build jobs.

This is all fine, but I have two style comments:

> +    if {[string match BUILD_* $nh]} {
>          set need_xen_hosts {}
> -        set need_build_host 1
> +        set need_build_host [string range $nh [expr [string first _ $nh] + 1] end]

This string range stuff is rather clunky.  How about

   if {[regsub {^BUILD_(.*)} $nh need_build_host]} {

?

> -    if {$need_build_host} { catching-otherwise broken prepare-build-host }
> +    if {[llength $need_build_host]} {
> +        catching-otherwise broken {
> +            prepare-build-host-[string tolower $need_build_host]

I might be tempted to not bother with the `string tolower' and simply
let the functions have SHOUTING in their names.  Up to you.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 15/17] osstest: add support for FreeBSD buildjobs to sg-run-job

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 15/17] osstest: add support for FreeBSD buildjobs to sg-run-job"):
> Add support and introduce a FreeBSD build job to sg-run-job.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v5 15/17] osstest: add support for FreeBSD buildjobs to sg-run-job

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v5 15/17] osstest: add support for FreeBSD buildjobs to sg-run-job"):
> Add support and introduce a FreeBSD build job to sg-run-job.
...
> +    switch -exact $ostype {
> +        FREEBSD { run-ts broken = ts-freebsd-set-hostflags --share }
                                                             ^ +
Actually, I just acked this, but I think you want to
add a plus there to make the testid not contain --share.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Missing bits for production usage (was: [PATCH v5 00...)

[Roger Pau Monne]

Hello,

This is mostly relevant for Ian, but I'm writing it here in case
anyone else is interested.

So far the content of this series works fine in standalone mode, and
in a limited way in production mode: new flights always use the same
base image in order to do the base install of FreeBSD.

Instead osstest should use the output of previous successful flights
in order to setup new FreeBSD hosts, so that osstest is self-updating
the FreeBSD version it uses for tests.

In order to do that, I would need the following set of scripts, that
Ian has kindly agreed to provide:

A script to mark/unmark a $flight.$job as blessed. Marking a
$flight.$job as blessed means that it's build output is never deleted
from osstest storage, a proposed command line would be:

./mg-freebsd-modify-blessed <$flight.$job> <arch> <branch> <bless/unbless>

A script to fetch the latest current blessed flight:

./mg-freebsd-get-blessed <arch> <branch>

This should return a flight number.

In cr-daily-branch upon successful completion the following will be
executed in order to changed the blessed build:

previous=./mg-freebsd-get-blessed <arch> <branch>
./mg-freebsd-modify-blessed $flight.build-freebsd-$arch $branch bless
./mg-freebsd-modify-blessed $previous.build-freebsd-$arch $branch unbless

FWIW, I'm not 100% sure the unblessing of the previous flight should
be done here, maybe it would be better to perform it in one of the
cleanup scripts?

I'm certainly up for suggestions on the interface.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Missing bits for production usage (was: [PATCH v5 00...)

[Ian Jackson]

Roger Pau Monne writes ("Missing bits for production usage (was: [PATCH v5 00...)"):
> A script to mark/unmark a $flight.$job as blessed. Marking a
> $flight.$job as blessed means that it's build output is never deleted
> from osstest storage, a proposed command line would be:

We can't call it "blessed" because that's something else.  Best I
could come with with is "anointed" but maybe we will think of
something better.

Suggested design:

Each flavour of anointment is represented by a row in tasks:

 type      anoint
 refkey    <scope> <details...>
   the flavour, eg   freebsd build <bsdbranch> <arch>
 username  NULL
 comment   <for human consumption>
 live      t
 refinfo   keep=<number to keep>

Each anointment is represented by a row in resources:

 restype    'share-flight'
 resname    <flight>
 shareix    <incrementing integer>
 owntaskid  <the anointment flavour taskid>
 subtask    <job>

So originally we do something like:

  ./mg-anoint prepare --keep=3 'freebsd build master amd64' 'FreeBSD builds'

which I guess should be done by mg-branch-setup.

> ./mg-freebsd-modify-blessed <$flight.$job> <arch> <branch> <bless/unbless>

  ./mg-anoint anoint "freebsd build $bsdbranch $arch" $flight $job

This would clean out older anointments, in a single db transaction.

> A script to fetch the latest current blessed flight:
> 
> ./mg-freebsd-get-blessed <arch> <branch>

How about

  ./mg-anoint retrieve "freebsd build $bsdbranch $arch"

which prints "$flight $job".

> In cr-daily-branch upon successful completion the following will be
> executed in order to changed the blessed build:
> 
> previous=./mg-freebsd-get-blessed <arch> <branch>
> ./mg-freebsd-modify-blessed $flight.build-freebsd-$arch $branch bless
> ./mg-freebsd-modify-blessed $previous.build-freebsd-$arch $branch unbless
> 
> FWIW, I'm not 100% sure the unblessing of the previous flight should
> be done here, maybe it would be better to perform it in one of the
> cleanup scripts?

mg-anoint anoint should do that based on the keep value.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Missing bits for production usage (was: [PATCH v5 00...)

[Roger Pau Monne]

On Thu, Jul 20, 2017 at 04:57:50PM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("Missing bits for production usage (was: [PATCH v5 00...)"):
> > A script to mark/unmark a $flight.$job as blessed. Marking a
> > $flight.$job as blessed means that it's build output is never deleted
> > from osstest storage, a proposed command line would be:
> 
> We can't call it "blessed" because that's something else.  Best I
> could come with with is "anointed" but maybe we will think of
> something better.

besaint? Not being a native speaker I'm not sure I can be more
helpful than searching for thesaurus of blessed online :).

> Suggested design:
> 
> Each flavour of anointment is represented by a row in tasks:
> 
>  type      anoint
>  refkey    <scope> <details...>
>    the flavour, eg   freebsd build <bsdbranch> <arch>
>  username  NULL
>  comment   <for human consumption>
>  live      t
>  refinfo   keep=<number to keep>
> 
> Each anointment is represented by a row in resources:
> 
>  restype    'share-flight'
>  resname    <flight>
>  shareix    <incrementing integer>
>  owntaskid  <the anointment flavour taskid>
>  subtask    <job>
> 
> So originally we do something like:
> 
>   ./mg-anoint prepare --keep=3 'freebsd build master amd64' 'FreeBSD builds'
> 
> which I guess should be done by mg-branch-setup.
> 
> > ./mg-freebsd-modify-blessed <$flight.$job> <arch> <branch> <bless/unbless>
> 
>   ./mg-anoint anoint "freebsd build $bsdbranch $arch" $flight $job
> 
> This would clean out older anointments, in a single db transaction.
> 
> > A script to fetch the latest current blessed flight:
> > 
> > ./mg-freebsd-get-blessed <arch> <branch>
> 
> How about
> 
>   ./mg-anoint retrieve "freebsd build $bsdbranch $arch"

That seems all fine to me, but I cannot really comment on the database
part of it.

> which prints "$flight $job".

For consumption it might be easier to simply return $flight.$job, but
that's a nit.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 0/3] Introduce anointments

[Ian Jackson]

3/3 of this series introduces the "anointment" concept which we will
use to record the FreeBSD flight and job which other things should
reuse.

I have tested this script with a test database (made with
mg-schema-test-database) and it seems to WFM.

I have not introduced anything to hook this into mg-branch-setup.  I
suggest that initially, we (you, Roger, I think) add a Deployment Note
to the appropriate FreeBSD patches asking for `mg-anoint prepare' to
be run and stating the right parameters.

Roger Pau Monne writes ("Re: Missing bits for production usage (was: [PATCH v5 00...)"):
> On Thu, Jul 20, 2017 at 04:57:50PM +0100, Ian Jackson wrote:
> > How about
> >   ./mg-anoint retrieve "freebsd build $bsdbranch $arch"
> > which prints "$flight $job".
> 
> For consumption it might be easier to simply return $flight.$job, but
> that's a nit.

Since mg-anoint consumes $flight $job, I think I'd prefer it to
produce that, as well.

Regards,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 1/3] README.planner: Improve internals documentation a bit

[Ian Jackson]

* share-flight resources may end up owned by a different task to their
  shareix, perhaps as a result of test database operations or perhaps
  as a result of donation with mg-allocate.  This should not be a
  problem.

* Document the xdbref task type.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.planner | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/README.planner b/README.planner
index f3cab53..b3b41a9 100644
--- a/README.planner
+++ b/README.planner
@@ -133,6 +133,10 @@ Types of task
    mg-execute-flight).  They are automatically created and destroyed -
    see above.
 
+ * `xdbref' tasks.  These are used to own resources whose allocation
+   authority has been transferred to a separate database, eg a test
+   database.  The refkey is an indication of the other database.
+
  * magic task numbers with special meanings:
 
      magic/allocatable
@@ -211,10 +215,11 @@ Flights can be protected (preserved) by allocating them with
 
 Flights are represented by restype='share-flight' entries in the
 resources table.  Conventionally, the shareix is the owning taskid.
-This allows multiple tasks to lock a single flight.  There is no
-corresponding entry with restype='flight', nor a resource_sharing
-entry.  mg-allocate will create and clean up share-flight entries as
-needed.
+(This is not a constraint, because the convention can be violated by
+transfer of ownerships.)  This allows multiple tasks to lock a single
+flight.  There is no corresponding entry with restype='flight', nor a
+resource_sharing entry.  mg-allocate (and other tools) will create and
+clean up share-flight entries as needed.
 
 
 DETAILED PROTOCOL NOTES
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/3] schema updates: mg-schema-test-database: Cope with breaking updates

[Ian Jackson]

We need to pass -ff, since we want to replicate the existing database,
regardless of its compatibility properties.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mg-schema-test-database | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index 641aadb..9be0625 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -411,7 +411,7 @@ END
 END
 	withtest ./mg-schema-create -q --no-updates
 
-	withtest ./mg-schema-update -q apply $wantupdates
+	withtest ./mg-schema-update -qff apply $wantupdates
 
         printf " (seqs)"
         seq_alters=""
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 3/3] anointments: Introduce anointment scheme, and mg-anoint script

[Ian Jackson]

CC: Roger Pau Monne <roger.pau@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.planner |   4 +
 mg-anoint      | 347 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 351 insertions(+)
 create mode 100755 mg-anoint

diff --git a/README.planner b/README.planner
index b3b41a9..255460b 100644
--- a/README.planner
+++ b/README.planner
@@ -133,6 +133,10 @@ Types of task
    mg-execute-flight).  They are automatically created and destroyed -
    see above.
 
+ * `anoint' tasks.  These are not really ongoing tasks; but they
+   represent something that can `own' a flight.  Each anointment kind
+   is represented by an `anoint' task.
+
  * `xdbref' tasks.  These are used to own resources whose allocation
    authority has been transferred to a separate database, eg a test
    database.  The refkey is an indication of the other database.
diff --git a/mg-anoint b/mg-anoint
new file mode 100755
index 0000000..4bb8ab0
--- /dev/null
+++ b/mg-anoint
@@ -0,0 +1,347 @@
+#!/usr/bin/perl -w
+#
+# usage:
+#
+#  ./mg-anoint prepare [PARAM...] REFKEY DESCRIPTION
+#        REFKEY should be 'SCOPE DETAILS...'
+#     PARAMs are:
+#        --keep=<number to keep>                 default is 3
+#        --blessings=<allowable-blessing>,...    default is just "real"
+#
+#  ./mg-anoint destroy REFKEY
+#
+#  ./mg-anoint anoint [ANOINT-OPTION...] REFKEY FLIGHT JOB
+#     ANOINT-OPTIONs are:
+#        --allow-blessed=BLESSING,...       default is from `prepare'
+#        --allow-job-status=STATUS,...      default is only `pass'
+#
+#  ./mg-anoint retrieve REFKEY
+#      => FLIGHT JOB
+#         if nothing anointed yet, prints nothing and exits 0
+#         if anointment not prepared, fails
+#
+#  ./mg-anoint list
+#      => human-readable output
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013,2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Each flavour of anointment is represented by a row in tasks:
+#
+# type      anoint
+# refkey    <scope> <details...>
+#   the flavour, eg   freebsd build <bsdbranch> <arch>
+# username  NULL
+# comment   <for human consumption>
+# live      t
+# refinfo   keep=<number-to-keep> blessings=<blessing>,...
+#
+# Each anointment is represented by a row in resources:
+#
+# restype    'share-flight'
+# resname    <flight>
+# owntaskid  <the anointment flavour taskid>
+# shareix    <initially, the anointment flavour taskid>
+# subtask    <job>
+
+
+use strict qw(vars refs);
+use DBI;
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::Executive;
+use IO::Handle;
+
+csreadconfig();
+
+our @o;
+sub pr_o () { print $_, "\n" or die $! foreach @o; }
+
+our $anointments_qtxt = <<END;
+      anointments AS (
+        SELECT f.flight   flight,
+               f.started  started,
+               r.shareix  shareix,
+               r.subtask  job
+          FROM resources r
+          JOIN flights   f
+            ON r.restype   = 'share-flight'
+           AND r.resname   = text(f.flight)
+           AND r.owntaskid = ?
+           AND f.started IS NOT NULL
+      )
+END
+
+our $task_q;
+our $mostrecent_q;
+
+sub prep_queries {
+    $task_q = $dbh_tests->prepare(<<END);
+        SELECT taskid, refinfo FROM tasks WHERE type='anoint' AND refkey=?
+END
+
+    $mostrecent_q = $dbh_tests->prepare(<<END);
+	    WITH $anointments_qtxt
+          SELECT * FROM anointments
+	ORDER BY started DESC LIMIT 1
+END
+}
+
+sub cmd_prepare {
+    my %params = (keep => 3,
+		  blessings => 'real');
+    while (@ARGV && $ARGV[0] =~ m/^-/) {
+	$_= shift @ARGV;
+	last if m/^\-\-?$/;
+	if (m/^--(keep)=(\d+)$/) {
+	    $params{$1} = $2 + 0;
+	} elsif (m/^--(blessings)=([-+0-9a-z,]+)$/) {
+	    $params{$1} = $2;
+	} else {
+	    die "unknown option $_ ?";
+	}
+    }
+    die "usage: mg-anoint prepare [OPTIONS] 'SCOPE DETAILS...' DESCRIPTION"
+	unless @ARGV==2 || $ARGV[0] =~ m/ /;
+    my ($refkey, $description) = @ARGV;
+    my $dq = $dbh_tests->prepare(<<END);
+        DELETE FROM tasks WHERE type='anoint' AND refkey=?
+END
+    my $iq = $dbh_tests->prepare(<<END);
+        INSERT INTO tasks (type,live, refkey,comment,refinfo)
+                    VALUES ('anoint','t', ?,?,?)
+END
+    my $refinfo = join ' ', map { "$_=$params{$_}" } sort keys %params;
+    db_retry($dbh_tests, [], sub {
+        @o = ();
+        $dq->execute($refkey);
+        $iq->execute($refkey, $description, $refinfo);
+    });
+    pr_o();
+}
+
+sub cmd_destroy {
+    die unless @ARGV==1;
+    die if $ARGV[0] =~ m/^-/;
+    my ($refkey) = @ARGV;
+    my $rdq = $dbh_tests->prepare(<<END);
+        DELETE FROM resources
+              WHERE restype='share-flight'
+                AND owntaskid = (SELECT taskid
+				   FROM tasks
+				  WHERE type='anoint'
+				    AND refkey=?)
+END
+    my $tdq = $dbh_tests->prepare(<<END);
+        DELETE FROM tasks WHERE type='anoint' AND refkey=?
+END
+    db_retry($dbh_tests, [], sub {
+        @o = ();
+        $rdq->execute($refkey);
+        $tdq->execute($refkey);
+    });
+    pr_o();
+}
+
+sub cmd_anoint {
+    my $allow_blessed = '';
+    my $allow_jobstatus = '';
+    while (@ARGV && $ARGV[0] =~ m/^-/) {
+	$_= shift @ARGV;
+	last if m/^\-\-?$/;
+	if (m/^--allow-blessed=/) {
+	    $allow_blessed = $';	
+	} elsif (m/^--allow-job-status=/) {
+	    $allow_jobstatus = $';	
+	} else {
+	    die "unknown option $_ ?";
+	}
+    }
+    die unless @ARGV==3;
+    my ($refkey, $flight, $job) = @ARGV;
+
+    prep_queries();
+
+    my $newflight_q = $dbh_tests->prepare(<<END);
+        SELECT * FROM flights WHERE flight=?
+END
+    my $newjob_q = $dbh_tests->prepare(<<END);
+        SELECT status FROM jobs WHERE flight=? AND job=?
+END
+
+    my $count_q = $dbh_tests->prepare(<<END);
+	    WITH $anointments_qtxt
+          SELECT count(*) FROM anointments
+END
+    my $todelete_q = $dbh_tests->prepare(<<END);
+	     WITH $anointments_qtxt
+           SELECT *
+             FROM anointments
+         ORDER BY started ASC
+            LIMIT ?
+END
+    my $delete_res_q = $dbh_tests->prepare(<<END);
+            DELETE FROM resources
+             WHERE restype   = 'share-flight'
+               AND owntaskid = ?
+               AND resname   = ?
+               AND shareix   = ?
+END
+    my $insert_q = $dbh_tests->prepare(<<END);
+       INSERT INTO resources
+                   (restype, resname,owntaskid,shareix,subtask)
+            VALUES ('share-flight', ?,?,?,?)
+END
+
+    db_retry($dbh_tests, [], sub {
+	@o = ();
+        $task_q->execute($refkey);
+
+	# find the task row (ie, the anointment kind)
+	my ($task, $refinfo) = $task_q->fetchrow_array();
+	die "no such anointment kind \`$refkey' (no prepare?)\n"
+	    unless defined $task;
+	my %params;
+	foreach (split /\s+/, $refinfo) {
+	    die unless m/=/;
+	    $params{$`} = $';
+	}
+	my %blessings;
+	$blessings{$_}++ foreach
+	    grep /./,
+	    (split /,/, $params{blessings}),
+	    (split /,/, $allow_blessed);
+
+	my %jobstatus;
+	$jobstatus{pass}++;
+	$jobstatus{$_}++ foreach grep /./, split /,/, $allow_jobstatus;
+
+	# check the to-be-anointed flight's blessing
+	$newflight_q->execute($flight);
+	my $frow = $newflight_q->fetchrow_hashref();
+	die "flight $flight missing" unless $frow;
+	die "flight $flight not started" unless defined $frow->{started};
+
+	# check the job status
+	$newjob_q->execute($flight, $job);
+	my ($jstatus) = $newjob_q->fetchrow_array();
+	die "job $flight.$job missing" unless defined $jstatus;
+	die "job $flight.$job status $jstatus" unless $jobstatus{$jstatus};
+
+	push @o, "flight $flight blessed $frow->{blessing}".
+	         " started ".show_abs_time($frow->{started});
+
+	die "flight $flight blessing $frow->{blessing}".
+	    " (not $params{blessings} / $allow_blessed)"
+	    unless $blessings{ $frow->{blessing} };
+
+	# check to-be-annointed flight is most recent
+	$mostrecent_q->execute($task);
+	my $mostrecent = $mostrecent_q->fetchrow_hashref();
+	die "flight $flight not newer than $mostrecent->{flight}"
+	    unless $frow->{started} > ($mostrecent->{started} // 0);
+
+	# expire old anointments
+	$count_q->execute($task);
+	my ($current) = $count_q->fetchrow_array();
+	my $want_delete = ($current+1) - $params{keep};
+	push @o, "anointment $refkey: currently $current anointed";
+	if ($want_delete > 0) {
+	    $todelete_q->execute($task, $want_delete);
+	    while (my $d = $todelete_q->fetchrow_hashref()) {
+		push @o, " expiring $d->{flight}.$d->{job} [/$d->{shareix}]".
+		    " started ".show_abs_time($d->{started});
+		$delete_res_q->execute($task, $d->{flight}, $d->{shareix});
+	    }
+	}
+
+	# at last!
+	$insert_q->execute($flight,$task,$task,$job);
+	push @o, "anointed $flight.$job";
+    });
+    pr_o();
+}    
+
+sub cmd_retrieve {
+    die unless @ARGV==1;
+    die if $ARGV[0] =~ m/^-/;
+    my ($refkey) = @ARGV;
+
+    prep_queries();
+
+    db_retry($dbh_tests, [], sub {
+        @o = ();
+        $task_q->execute($refkey);
+	my ($task) = $task_q->fetchrow_array();
+	die "no such anointment kind \`$refkey'" unless defined $task;
+
+	$mostrecent_q->execute($task);
+	my $row = $mostrecent_q->fetchrow_hashref();
+	if ($row) {
+	    push @o, "$row->{flight} $row->{job}";
+	} else {
+	    print STDERR "warning: nothing anointed $refkey\n";
+	}
+    });
+    pr_o();
+};
+
+sub cmd_list {
+    die "no options to list" if @ARGV;
+
+    my $tq = $dbh_tests->prepare(<<END);
+        SELECT *
+          FROM tasks
+         WHERE type = 'anoint'
+      ORDER BY refkey
+END
+    my $rq = $dbh_tests->prepare(<<END);
+        SELECT *
+          FROM resources
+         WHERE owntaskid = ?
+END
+
+    db_retry($dbh_tests, [], sub {
+        @o = ();
+        db_readonly_report();
+	$tq->execute();
+	while (my $t = $tq->fetchrow_hashref()) {
+	    push @o, sprintf "%s [%s] (%s)",
+		$t->{refkey}, $t->{refinfo}, $t->{comment};
+	    push @o, " ! NOT LIVE" unless $t->{live};
+	    push @o, " ! username=$t->{username}" if defined $t->{username};
+	    $rq->execute($t->{taskid});
+	    while (my $r = $rq->fetchrow_hashref()) {
+		if ($r->{restype} eq 'share-flight') {
+		    push @o, "  $r->{resname}.$r->{subtask}";
+		} else {
+		    push @o, "  ! $r->{restype}/$r->{resname}/$r->{subtask}";
+		}
+	    }
+	}
+    });
+    pr_o();
+}
+
+die 'need operation' unless @ARGV;
+die 'no global options understood' if $ARGV[0] =~ m/^-/;
+
+my $subcmd= shift @ARGV;
+$subcmd =~ s/-/_/g;
+my $subcmdproc = ${*::}{"cmd_$subcmd"};
+die "unknown subcommand" unless $subcmdproc;
+$subcmdproc->();
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel