[OSSTEST PATCH 0/7] Nested pathname and logging fixes

[OSSTEST PATCH 0/7] Nested pathname and logging fixes

[Ian Jackson]

While looking at the webspace non-unique filename issue, I ended up
peering at some of the logs generated by the new Nested HVM test, and
saw some things that could be improved.

Ian.

[OSSTEST PATCH 1/7] Nested hosts: Print L0 as well as L1 parent name in selectguest

[Ian Jackson]

Now that we have hostnamepath_list, we can remove the ad-hoc
expansion "$gn on $...{Name}" with a recipe which ascends through the
applicable nesting levels.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a66f8a9..38386d4 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1453,7 +1453,9 @@ sub selectguest ($$) {
     foreach my $opt (guest_var_commalist($gho,'options')) {
         $gho->{Options}{$opt}++;
     }
-    logm("guest: using $gn on $gho->{Host}{Name}");
+    my $m = "guest: using $gn";
+    $m .= " on $_" foreach hostnamepath_list($gho->{Host});
+    logm($m);
     guest_find_lv($gho);
     guest_find_diskimg($gho);
     guest_find_ether($gho);
-- 
1.7.10.4

Re: [OSSTEST PATCH 1/7] Nested hosts: Print L0 as well as L1 parent name in selectguest

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> Now that we have hostnamepath_list, we can remove the ad-hoc
> expansion "$gn on $...{Name}" with a recipe which ascends through the
> applicable nesting levels.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 2/7] Nested hosts: Use full hostnamepath in various paths

[Ian Jackson]

Replace some (but not all) interpolations of $ho->{Name} in filenames
with hostnamepath($ho).

I searched the tree for `{Host}' and `{Name}', looking for paths which
need (or might need) to be qualified with an L0 hostname in an L1 or
L2 case.

I have left ts-logs-capture for now; that will come in a moment.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm       |    8 ++++----
 Osstest/Toolstack/libvirt.pm |    2 +-
 ts-host-install              |    4 ++--
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 38386d4..d0acdc6 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -469,7 +469,7 @@ sub target_somefile_getleaf ($$$) {
         $$lleaf_ref= $rdest;
         $$lleaf_ref =~ s,.*/,,;
     }
-    $$lleaf_ref= "$ho->{Name}--$$lleaf_ref";
+    $$lleaf_ref= hostnamepath($ho)."--$$lleaf_ref";
 }
 
 sub tpfcs_core {
@@ -2345,7 +2345,7 @@ sub setup_pxelinux_bootcfg ($$) {
     my ($ho, $bootfile) = @_;
     my $f= host_pxefile($ho);
     file_link_contents("$ho->{Tftp}{Path}$ho->{Tftp}{PxeDir}$f", $bootfile,
-	"$ho->{Name}-pxelinux.cfg");
+	hostnamepath($ho)."-pxelinux.cfg");
 }
 
 # Systems using BIOS are configured to use pxelinux
@@ -2386,14 +2386,14 @@ sub setup_grub_efi_bootcfg ($$) {
     my $f = "grub.cfg-$ho->{Ether}";
     my $grub= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{GrubBase}.'/'.
 	$c{TftpGrubVersion}."/pxegrub-$r{arch}.efi";
-    my $pxe=$ho->{Tftp}{Path}.'/'.$ho->{Name}.'/pxe.img';
+    my $pxe=$ho->{Tftp}{Path}.'/'.hostnamepath($ho).'/pxe.img';
 
     logm("Copy $grub => $pxe");
     copy($grub, $pxe) or die "Copy $grub to $pxe failed: $!";
 
     logm("grub_efi bootcfg into $f");
     file_link_contents("$ho->{Tftp}{Path}$ho->{Tftp}{TmpDir}$f",
-		       $bootfile,  "$ho->{Name}-pxegrub.cfg");
+		       $bootfile,  hostnamepath($ho)."-pxegrub.cfg");
 }
 
 # UEFI systems PXE boot using grub.efi
diff --git a/Osstest/Toolstack/libvirt.pm b/Osstest/Toolstack/libvirt.pm
index 776432d..69ff0bb 100644
--- a/Osstest/Toolstack/libvirt.pm
+++ b/Osstest/Toolstack/libvirt.pm
@@ -48,7 +48,7 @@ sub create ($$) {
     my $cfg = $gho->{CfgPath};
     my $lcfg = $cfg;
     $lcfg =~ s,/,-,g;
-    $lcfg = "$ho->{Name}--$lcfg";
+    $lcfg = hostnamepath($ho)."--$lcfg";
     target_cmd_root($ho, "virsh domxml-from-native xen-xl $cfg > $cfg.xml", 30);
     target_getfile_root($ho,60,"$cfg.xml", "$stash/$lcfg");
     target_cmd_root($ho, "virsh create --file $cfg.xml", 100);
diff --git a/ts-host-install b/ts-host-install
index 1cacd07..cfa6044 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -208,7 +208,7 @@ sub setup_pxeboot_firstboot($) {
 
     $kernel = "/$d_i/linux" unless $kernel;
 
-    my $initrd_overlay= "tmp/t.$ho->{Name}.initrd";
+    my $initrd_overlay= "tmp/t.".hostnamepath($ho).".initrd";
     system qw(rm -rf --),"$initrd_overlay.d";
     mkdir "$initrd_overlay.d" or die "$initrd_overlay.d: $!";
 
@@ -253,7 +253,7 @@ END
     push @initrds, "$initrd_overlay.cpio.gz";
 
     logm("using initrds: @initrds");
-    my $initrd= "$ho->{Tftp}{TmpDir}$ho->{Name}--initrd.gz";
+    my $initrd= "$ho->{Tftp}{TmpDir}".hostnamepath($ho)."--initrd.gz";
     system_checked("cat -- @initrds >$ho->{Tftp}{Path}$initrd");
     
     push @dicmdline, "domain=$c{TestHostDomain}";
-- 
1.7.10.4

Re: [OSSTEST PATCH 2/7] Nested hosts: Use full hostnamepath in various paths

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> Replace some (but not all) interpolations of $ho->{Name} in filenames
> with hostnamepath($ho).
> 
> I searched the tree for `{Host}' and `{Name}', looking for paths which
> need (or might need) to be qualified with an L0 hostname in an L1 or
> L2 case.
> 
> I have left ts-logs-capture for now; that will come in a moment.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 3/7] Log capture pathnames: Include host name (and nested L0 name)

[Ian Jackson]

Rather than using $gho->{Name} (and, sometimes, separately
$ho->{Name}), use hostnamepath().

This means that the hostname is always included in the standard way.
The filename structure for xenctx and console snapshots changes
slightly.

In two cases this means that the hostname is now included where it
wasn't previously.  That is helpful if due to some insanity the same
guest is present on more than one host.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ts-logs-capture |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/ts-logs-capture b/ts-logs-capture
index cb0d371..86fad93 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -85,7 +85,7 @@ END
         foreach my $logfile (split / /, $logfileslist) {
             my $llogfile= $logfile;
             $llogfile =~ s,/,-,g;
-            $llogfile= "$lho->{Name}--$llogfile";
+            $llogfile= hostnamepath($lho)."--$llogfile";
             logm("fetching $logfile to $llogfile");
             if (!eval {
                 target_getfile_root($lho,60, $logfile,"$stash/$llogfile");
@@ -113,7 +113,7 @@ sub try_cmd_output_save ($;$) {
         $dfn =~ s,-,-,g;
         $dfn =~ s/ /_/g;
         $dfn =~ s/[^-+._,=0-9a-zA-Z]/#/g;
-        $dfn= $ho->{Name}."-output-$dfn";
+        $dfn= hostnamepath($ho)."-output-$dfn";
     }
     my $output;
     if (!eval {
@@ -219,7 +219,7 @@ sub fetch_xenctx_guest ($) {
 		$kernpath !~ m,/vmlinuz-, ? die "$kernpath ?" :
 		"-s $`/System.map-$'";
 	    foreach my $iter (0..1) {
-		my $dfn = "$ho->{Name}-xenctx-$gho->{Name},vcpu$vcpu,i$iter";
+		my $dfn = hostnamepath($ho)."-xenctx,vcpu$vcpu,i$iter";
 		try_cmd_output_save(
 		    'PATH=/usr/local/lib/xen/bin:/usr/lib/xen/bin:$PATH; '.
 		    "xenctx $sysmaparg $gho->{Domid} $vcpu",
@@ -236,11 +236,12 @@ sub fetch_logs_guest ($) {
 
     my $consolecmd= toolstack($ho)->consolecmd($gho);
     try_cmd_output_save("sleep 1 | $consolecmd | cat",
-                        "guest-$gho->{Name}-console");
+                        "guest-".hostnamepath($gho)."-console");
 
     if (!eval {
         my $vnc= guest_vncsnapshot_begin($ho,$gho);
-        guest_vncsnapshot_stash($ho,$gho,$vnc, "$gho->{Name}--vnc.jpeg");
+        guest_vncsnapshot_stash($ho,$gho,$vnc,
+				hostnamepath($gho)."-vnc.jpeg");
         1;
     }) {
         logm("cannot get vnc snapshot: $@");
-- 
1.7.10.4

Re: [OSSTEST PATCH 3/7] Log capture pathnames: Include host name (and nested L0 name)

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> Rather than using $gho->{Name} (and, sometimes, separately
> $ho->{Name}), use hostnamepath().
> 
> This means that the hostname is always included in the standard way.
> The filename structure for xenctx and console snapshots changes
> slightly.
> 
> In two cases this means that the hostname is now included where it
> wasn't previously.  That is helpful if due to some insanity the same
> guest is present on more than one host.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 4/7] logm: Introduce $logm_prefix, and annotate nested guest messages

[Ian Jackson]

This allows code elsewhere to annotate log messages which might
otherwise be confusing.  The variable should be localised, and the
value should always start with a space, if not empty.

Use this to annotate the calls to selecthost and selectguest from
within selecthost-for-an-L1.  Otherwise some of the log messages can
be very confusing.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index d0acdc6..6c08c6d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -41,7 +41,8 @@ BEGIN {
                       tsreadconfig %r $flight $job $stash
                       ts_get_host_guest
 
-                      fail broken logm $logm_handle get_filecontents
+                      fail broken logm $logm_handle $logm_prefix
+                      get_filecontents
                       report_once
 
                       store_runvar get_runvar get_runvar_maybe
@@ -132,6 +133,7 @@ our %timeout= qw(RebootDown   100
                  HardRebootUp 600);
 
 our $logm_handle= new IO::File ">& STDERR" or die $!;
+our $logm_prefix= '';
 
 #---------- test script startup ----------
 
@@ -194,7 +196,7 @@ sub ts_get_host_guest { # pass this @ARGV
 sub logm ($) {
     my ($m) = @_;
     my @t = gmtime;
-    my $fm = (show_abs_time time)." $m\n";
+    my $fm = (show_abs_time time)."$logm_prefix $m \n";
     foreach my $h ((ref($logm_handle) eq 'ARRAY')
 		   ? @$logm_handle : $logm_handle) {
 	print $h $fm or die $!;
@@ -866,6 +868,7 @@ sub selecthost ($) {
 
     if ($name =~ s/^(.*)://) {
 	my $parentname = $1;
+	local $logm_prefix = $logm_prefix." [host $ident]";
 	my $parent = selecthost($parentname);
 	my $child = selectguest($name,$parent);
 	$child->{Ident} = $ho->{Ident};
-- 
1.7.10.4

Re: [OSSTEST PATCH 4/7] logm: Introduce $logm_prefix, and annotate nested guest messages

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> This allows code elsewhere to annotate log messages which might
> otherwise be confusing.  The variable should be localised, and the
> value should always start with a space, if not empty.
> 
> Use this to annotate the calls to selecthost and selectguest from
> within selecthost-for-an-L1.  Otherwise some of the log messages can
> be very confusing.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 5/7] Nested: Do not attempt to look up nested host ownership in DB

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 6ba3d37..2572f5f 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -125,6 +125,14 @@ END
 
 sub host_check_allocated ($$) { #method
     my ($jd, $ho) = @_;
+
+    if ($ho->{Host}) {
+	# This is a nested guest; check allocation of the parent host.
+	# We don't attempt to set Shared, SharedMaybeOthers, etc.
+	# (since nested hosts aren't shared between jobs).
+	return $jd->host_check_allocated($ho->{Host});
+    }
+
     $ho->{Shared}= resource_check_allocated('host', $ho->{Name});
     $ho->{SharedMaybeOthers}=
         $ho->{Shared} &&
-- 
1.7.10.4

Re: [OSSTEST PATCH 5/7] Nested: Do not attempt to look up nested host ownership in DB

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 6/7] Nested: Make Osstest::PDU::guest idempotent

[Ian Jackson]

These methods are supposed to succeed, silently, if the host is
already in the required power state.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/PDU/guest.pm   |   13 +++++++++++--
 Osstest/TestSupport.pm |    3 ++-
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/Osstest/PDU/guest.pm b/Osstest/PDU/guest.pm
index b6bf9a1..b708af5 100755
--- a/Osstest/PDU/guest.pm
+++ b/Osstest/PDU/guest.pm
@@ -49,10 +49,19 @@ sub pdu_power_state {
     die "$child->{Name} ?" unless $parent;
 
     logm("power $child->{Name} nested on $parent->{Name} ".($on+0));
+
+    my $st = guest_get_state($parent, $child);
     if ($on) {
-	toolstack($parent)->create($child);
+	if ($st =~ m/^$guest_state_running_re$/o) {
+	} elsif (length($st)) { # exists but crashed or something ?
+	    toolstack($parent)->destroy($child);
+	} else { # does not exist
+	    toolstack($parent)->create($child);
+	}
     } else {
-	toolstack($parent)->destroy($child);
+	if (length($st)) {
+	    toolstack($parent)->destroy($child);
+	}
     }
 }
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 6c08c6d..d1f7d36 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -97,7 +97,8 @@ BEGIN {
                       prepareguest_part_lvmdisk prepareguest_part_diskimg
                       prepareguest_part_xencfg
                       guest_umount_lv guest_await guest_await_dhcp_tcp
-                      guest_checkrunning target_check_ip guest_find_ether
+                      guest_checkrunning $guest_state_running_re
+                      target_check_ip guest_find_ether
                       guest_find_domid guest_check_up guest_check_up_quick
                       guest_get_state guest_await_reboot
                       guest_await_shutdown guest_await_destroy guest_destroy
-- 
1.7.10.4

Re: [OSSTEST PATCH 6/7] Nested: Make Osstest::PDU::guest idempotent

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> These methods are supposed to succeed, silently, if the host is
> already in the required power state.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 7/7] ts-logs-capture: Attempt to power up a host before capturing its logs

[Ian Jackson]

If for any reason it's powered down, this is useful.

(In theory for a nested host, we could get the logs by grobbling in
its parent's view of its filesystem, but that is too much work.)

We do not trap the errors from this, so that if the PDU is broken (or
the L1 has not been created), we can fail more quickly.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ts-logs-capture |    1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-logs-capture b/ts-logs-capture
index 86fad93..73b7c5e 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -248,6 +248,7 @@ sub fetch_logs_guest ($) {
     }
 }
 
+power_state($ho,1);
 find_guests();
 fetch_xenctx_guest($_) foreach @guests;
 serial_fetch_logs($ho);
-- 
1.7.10.4

Re: [OSSTEST PATCH 7/7] ts-logs-capture: Attempt to power up a host before capturing its logs

[Ian Campbell]

On Mon, 2015-11-16 at 17:54 +0000, Ian Jackson wrote:
> If for any reason it's powered down, this is useful.
> 
> (In theory for a nested host, we could get the logs by grobbling in
> its parent's view of its filesystem, but that is too much work.)
> 
> We do not trap the errors from this, so that if the PDU is broken (or
> the L1 has not been created), we can fail more quickly.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>