[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes

[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes

[Ian Campbell]

The marilith boxes use a conserver (http://www.conserver.com/) setup for
serial access. Our installation exports the logs via http allowing us to
grab them with wget.

Sending debug keys with is handled separately via xenuse. xenuse
ultimately speaks to the conserver too but it abstracts away the IP and
port to use so this is preferred.

With these changes the correct Serial hostprop for a marilith box is:
        xenuse;http http://conserver.uk.xensource.com/consoles/

[PATCH OSSTEST 1/6] Standalone: Collect logs modified in the last hour

[Ian Campbell]

On failure, when the console may have been silent for a considerable time, 1s is not long enough ago.
---
 Osstest/JobDB/Standalone.pm |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/JobDB/Standalone.pm b/Osstest/JobDB/Standalone.pm
index da35376..d3ff1df 100644
--- a/Osstest/JobDB/Standalone.pm
+++ b/Osstest/JobDB/Standalone.pm
@@ -101,7 +101,8 @@ sub jobdb_check_other_job { } #method
 
 sub jobdb_flight_started_for_log_capture ($$) { #method
     my ($mo, $flight) = @_;
-    return time - 1; # just the most recent serial log then
+    return time - 60*60; # just the most recent serial log then,
+                         # modified in the last hour
 }
 
 sub jobdb_enable_log_capture ($) { #method
-- 
1.7.10.4

[PATCH OSSTEST 2/6] TestSupport: allow multiple host serial methods

[Ian Campbell]

---
 Osstest/TestSupport.pm |   25 ++++++++++++++++++++++---
 ts-logs-capture        |    2 +-
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e5343c8..75fb7cc 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -63,6 +63,7 @@ BEGIN {
 
                       selecthost get_hostflags get_host_property
                       power_state power_cycle power_cycle_time
+                      serial_fetch_logs
                       propname_massage
          
                       get_stashed open_unique_stashfile
@@ -620,6 +621,26 @@ sub guest_check_ip ($) {
     $gho->{DhcpWatch}->check_ip($gho);
 }
 
+#-------------- serial -------------
+
+sub serial_host_setup ($) {
+    my ($ho) = @_;
+    my $methobjs = [ ];
+    my $serialmeth = get_host_property($ho,'serial','noop');
+    foreach my $meth (split /\;\s*/, $serialmeth) {
+	push @$methobjs, get_host_method_object($ho,'Serial',$meth);
+    }
+    $ho->{SerialMethobjs} = $methobjs;
+}
+
+sub serial_fetch_logs ($) {
+    my ($ho) = @_;
+    logm("serial: collecting logs for $ho->{Name}");
+    foreach my $mo (@{ $ho->{SerialMethobjs} }) {
+	$mo->fetch_logs();
+    }
+}
+
 #---------- power cycling ----------
 
 sub power_cycle_host_setup ($) {
@@ -721,9 +742,7 @@ sub selecthost ($) {
 
     dhcp_watch_setup($ho,$ho);
     power_cycle_host_setup($ho);
-
-    my $serialmeth = get_host_property($ho,'serial','noop');
-    $ho->{SerialMethobj} = get_host_method_object($ho,'Serial',$serialmeth);
+    serial_host_setup($ho);
 
     $ho->{IpStatic} = get_host_property($ho,'ip-addr');
     if (!defined $ho->{IpStatic}) {
diff --git a/ts-logs-capture b/ts-logs-capture
index 9a71db0..2dd95ea 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -211,6 +211,6 @@ sub fetch_logs_guest ($) {
     }
 }
 
-$ho->{SerialMethobj}->fetch_logs();
+serial_fetch_logs($ho);
 fetch_logs_host_guests();
 logm("logs captured to $stash");
-- 
1.7.10.4

[PATCH OSSTEST 3/6] Serial: Refactor debug key sending into separate request_debug function

[Ian Campbell]

This puts the list of keys to send into common code.
---
 Osstest/Serial/noop.pm     |    4 ++++
 Osstest/Serial/sympathy.pm |   42 +++++++++++++++++++++++++-----------------
 Osstest/TestSupport.pm     |   12 ++++++++++++
 3 files changed, 41 insertions(+), 17 deletions(-)

diff --git a/Osstest/Serial/noop.pm b/Osstest/Serial/noop.pm
index 1b980c2..d0de9cd 100644
--- a/Osstest/Serial/noop.pm
+++ b/Osstest/Serial/noop.pm
@@ -39,6 +39,10 @@ sub new {
     return bless { }, $class;
 }
 
+sub request_debug {
+    return 0;
+}
+
 sub fetch_logs {
     my ($mo) = @_;
     logm("serial access method \`noop',".
diff --git a/Osstest/Serial/sympathy.pm b/Osstest/Serial/sympathy.pm
index 3128b68..d6bf425 100644
--- a/Osstest/Serial/sympathy.pm
+++ b/Osstest/Serial/sympathy.pm
@@ -57,21 +57,15 @@ sub new {
     return bless $mo, $class;
 }
 
-sub fetch_logs {
-    my ($mo) = @_;
-
-    my $started= $mjobdb->jobdb_flight_started_for_log_capture($flight);
+sub request_debug {
+    my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
 
-    my $ho = $mo->{Host};
-    my $logpat = $mo->{Pattern};
     my $targhost= $mo->{Server};
 
-    logm("requesting debug information");
-
     my ($sshopts) = sshopts();
     my $sympwrite= sub {
         my ($what,$str,$pause) = @_;
-        logm("sending $what");
+        logm("sympathy sending $what");
         if (!eval {
             local ($SIG{'PIPE'}) = 'IGNORE';
             my $sock= $mo->{Socket};
@@ -89,18 +83,32 @@ sub fetch_logs {
         }
         return 1;
     };
+
     my $debugkeys= sub {
-        my ($what, $keys) = @_;
-        foreach my $k (split //, $keys) {
-            $sympwrite->("$what debug info request, debug key $k", $k, 2);
-        }
+	my ($what, $keys) = @_;
+	foreach my $k (split //, $keys) {
+	    $sympwrite->("$what debug info request, debug key $k", $k, 2);
+	}
     };
-    $sympwrite->('request for input to Xen',"\x18\x18\x18",1);
-    $debugkeys->('Xen',"0HMQacdegimnrstuvz");
+
+    $sympwrite->('request for input to Xen', $conswitch, 1);
+    $debugkeys->('Xen', $xenkeys);
     sleep(10);
-    $debugkeys->('guest',"q");
+    $debugkeys->('guest', $guestkeys);
     sleep(10);
-    $sympwrite->("RET to dom0","\x18\x18\x18\r", 5);
+    $sympwrite->("RET to dom0","$conswitch\r", 5);
+
+    return 1;
+}
+
+sub fetch_logs {
+    my ($mo) = @_;
+
+    my $started= $mjobdb->jobdb_flight_started_for_log_capture($flight);
+
+    my $ho = $mo->{Host};
+    my $logpat = $mo->{Pattern};
+    my $targhost= $mo->{Server};
 
     logm("collecting serial logs since $started from $targhost");
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 75fb7cc..f4ec055 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -635,7 +635,19 @@ sub serial_host_setup ($) {
 
 sub serial_fetch_logs ($) {
     my ($ho) = @_;
+
+    logm("serial: requesting debug information from $ho->{Name}");
+
+    foreach my $mo (@{ $ho->{SerialMethobjs} }) {
+	$mo->request_debug("\x18\x18\x18",
+			   "0HMQacdegimnrstuvz",
+			   "q") or next;
+	# use the first method which supports ->request_debug.
+	last;
+    }
+
     logm("serial: collecting logs for $ho->{Name}");
+
     foreach my $mo (@{ $ho->{SerialMethobjs} }) {
 	$mo->fetch_logs();
     }
-- 
1.7.10.4

[PATCH OSSTEST 4/6] Serial: collect serial logs from an http server

[Ian Campbell]

This relies on wget and httpd colluding to set the mtimes on the downloaded
files, which works with at least the one server I care about (the apache on
the conserver handling the marilith boxes)
---
 Osstest/Serial/http.pm |  114 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 114 insertions(+)
 create mode 100644 Osstest/Serial/http.pm

diff --git a/Osstest/Serial/http.pm b/Osstest/Serial/http.pm
new file mode 100644
index 0000000..cb36f1c
--- /dev/null
+++ b/Osstest/Serial/http.pm
@@ -0,0 +1,114 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Fetch logs from a web directory export.
+
+package Osstest::Serial::http;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+
+use File::Temp;
+use File::Copy;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho, $methname, @args) = @_;
+    my $mo = { Host => $ho, Name => $ho->{Name} };
+    die if @args<1;
+    push @args, "$ho->{Name}.txt*" if @args < 2;
+
+    logm("serial method $methname $mo->{Host}{Name}: @args");
+    ($mo->{Server}, $mo->{Pattern}) = @args;
+    return bless $mo, $class;
+}
+
+sub request_debug {
+    return 0;
+}
+
+sub fetch_logs {
+    my ($mo) = @_;
+
+    my $started= $mjobdb->jobdb_flight_started_for_log_capture($flight);
+
+    my $ho = $mo->{Host};
+    my $logpat = $mo->{Pattern};
+    my $targhost= $mo->{Server};
+
+    logm("serial http from $mo->{Name} fetching $mo->{Pattern} from $mo->{Server}");
+
+    my $dir = File::Temp->newdir();
+    my $tdir = $dir->dirname;
+
+    my $lrf = "$tdir/log-retrieval.log";
+
+    system_checked(qw(wget -nH --cut-dirs=1 -r -l1 --no-parent),
+                   '-o', "$lrf",
+                   '-A', $mo->{Pattern},
+                   '-P', $tdir,
+                   $mo->{Server});
+
+    my $sr = "serial-retrieval.log";
+    my $lr = open_unique_stashfile(\$sr);
+    File::Copy::copy($lrf, $lr);
+
+    my %done;
+    foreach my $logfile (glob "$tdir/$mo->{Pattern}") {
+	my $lh= new IO::File $logfile, 'r';
+	if (!defined $lh) {
+	    $!==&ENOENT or warn "$logfile $!";
+	    next;
+	}
+	stat $lh or die "$logfile $!";
+	my $inum= (stat _)[1];
+	my $lfage= (stat _)[9];
+        my $df= $logfile;
+        $df =~ s,.*/,,;
+	if ($lfage < $started) {
+	    next if $done{$inum};
+	    logm("$df modified $lfage, skipping")
+		unless $done{$inum};
+	    $done{$inum}= 1;
+	    next;
+	}
+	next if defined $done{$inum} and $done{$inum} >= 2;
+	$done{$inum}= 2;
+
+        $df = "serial-$df";
+        logm("stashing $df");
+
+        my $dh= open_unique_stashfile(\$df);
+        File::Copy::copy($logfile, $dh);
+    }
+    return;
+
+}
+
+1;
-- 
1.7.10.4

[PATCH OSSTEST 5/6] Serial: new module to send debug keys via xenuse

[Ian Campbell]

No log retreival here, that is handled via a separate conserver.
---
 Osstest/Serial/xenuse.pm |  100 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)
 create mode 100644 Osstest/Serial/xenuse.pm

diff --git a/Osstest/Serial/xenuse.pm b/Osstest/Serial/xenuse.pm
new file mode 100644
index 0000000..6b9d955
--- /dev/null
+++ b/Osstest/Serial/xenuse.pm
@@ -0,0 +1,100 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Send debug keys via xenuse.
+
+package Osstest::Serial::xenuse;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+
+use File::Temp;
+use File::Copy;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho, $methname, @args) = @_;
+    my $mo = { Host => $ho, Name => $ho->{Name} };
+
+    logm("serial method $methname $mo->{Host}{Name}: @args");
+    return bless $mo, $class;
+}
+
+sub request_debug {
+    my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
+    my $xenuse= $c{XenUsePath} || "xenuse";
+
+    my $ho= $mo->{Host};
+
+    my $writer= sub {
+        my ($what,$str,$pause) = @_;
+        logm("xenuse sending $what");
+        if (!eval {
+            print XENUSEWRITE $str or die $!;
+            sleep($pause);
+            1;
+        }) {
+            warn "failed to send $what: $@\n";
+            return 0;
+        }
+        return 1;
+    };
+
+    my $debugkeys= sub {
+	my ($what, $keys) = @_;
+	foreach my $k (split //, $keys) {
+	    $writer->("$what debug info request, debug key $k", $k, 2);
+	}
+    };
+
+    local ($SIG{'PIPE'}) = 'IGNORE';
+    open XENUSEWRITE, "|$xenuse -t $ho->{Name}" or die $!;
+    autoflush XENUSEWRITE 1;
+
+    $writer->('force attach', "\x05cf", 1); # ^E c f == force attach
+
+    $writer->('request for input to Xen', $conswitch, 1);
+    $debugkeys->('Xen', $xenkeys);
+    sleep(10);
+    $debugkeys->('guest', $guestkeys);
+    sleep(10);
+    $writer->("RET to dom0","$conswitch\r", 5);
+
+    $writer->('dettach', "\x05c.", 1); # ^E c . == disconnect
+
+    close XENUSEWRITE or die "$? $!";
+
+    return 1;
+}
+
+sub fetch_logs {
+    return;
+}
+
+1;
-- 
1.7.10.4

[PATCH OSSTEST 6/6] Serial/http: use mkdtemp instead of File::Temp->newdir

[Ian Campbell]

The latter is not available in Squeeze's File::Temp.
---
 Osstest/Serial/http.pm |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/Osstest/Serial/http.pm b/Osstest/Serial/http.pm
index cb36f1c..3094baa 100644
--- a/Osstest/Serial/http.pm
+++ b/Osstest/Serial/http.pm
@@ -24,7 +24,8 @@ use warnings;
 use Osstest;
 use Osstest::TestSupport;
 
-use File::Temp;
+use File::Path qw / rmtree /;
+use File::Temp qw / mkdtemp /;
 use File::Copy;
 
 BEGIN {
@@ -64,8 +65,7 @@ sub fetch_logs {
 
     logm("serial http from $mo->{Name} fetching $mo->{Pattern} from $mo->{Server}");
 
-    my $dir = File::Temp->newdir();
-    my $tdir = $dir->dirname;
+    my $tdir = mkdtemp("osstest.log.XXXXX") or die $!;
 
     my $lrf = "$tdir/log-retrieval.log";
 
@@ -93,7 +93,7 @@ sub fetch_logs {
         $df =~ s,.*/,,;
 	if ($lfage < $started) {
 	    next if $done{$inum};
-	    logm("$df modified $lfage, skipping")
+	    logm("$df modified $lfage ($started), skipping")
 		unless $done{$inum};
 	    $done{$inum}= 1;
 	    next;
@@ -107,8 +107,9 @@ sub fetch_logs {
         my $dh= open_unique_stashfile(\$df);
         File::Copy::copy($logfile, $dh);
     }
-    return;
 
+    rmtree($tdir);
+    return;
 }
 
 1;
-- 
1.7.10.4

Re: [PATCH OSSTEST 0/6] Support for serial logs from marilith boxes

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes"):
> The marilith boxes use a conserver (http://www.conserver.com/) setup for
> serial access. Our installation exports the logs via http allowing us to
> grab them with wget.
> 
> Sending debug keys with is handled separately via xenuse. xenuse
> ultimately speaks to the conserver too but it abstracts away the IP and
> port to use so this is preferred.
> 
> With these changes the correct Serial hostprop for a marilith box is:
>         xenuse;http http://conserver.uk.xensource.com/consoles/

These all look plausible to me (as do your previous to); please go
ahead and push them.

Ian.

Re: [PATCH OSSTEST 0/6] Support for serial logs from marilith boxes

[Ian Campbell]

On Fri, 2013-10-11 at 18:22 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes"):
> > The marilith boxes use a conserver (http://www.conserver.com/) setup for
> > serial access. Our installation exports the logs via http allowing us to
> > grab them with wget.
> > 
> > Sending debug keys with is handled separately via xenuse. xenuse
> > ultimately speaks to the conserver too but it abstracts away the IP and
> > port to use so this is preferred.
> > 
> > With these changes the correct Serial hostprop for a marilith box is:
> >         xenuse;http http://conserver.uk.xensource.com/consoles/
> 
> These all look plausible to me (as do your previous to); please go
> ahead and push them.

I assumed s/to/two/ and pushed all eight:
fbcda5f Serial/http: use mkdtemp instead of File::Temp->newdir
5caaf09 Serial: new module to send debug keys via xenuse
660efb2 Serial: collect serial logs from an http server
514d05e Serial: Refactor debug key sending into separate request_debug function
acecbdc TestSupport: allow multiple host serial methods
cc60e34 Standalone: Collect logs modified in the last hour
5ce9690 Debian: set FANCYTTY=0 so serial logs aren't full of escape chars on Wheezy
a2164c3 TestSupport: fix warning about cacheing_git when !no-reinstall

Ian.

Re: [PATCH OSSTEST 0/6] Support for serial logs from marilith boxes

[Ian Campbell]

On Sun, 2013-10-13 at 10:19 +0100, Ian Campbell wrote:
> On Fri, 2013-10-11 at 18:22 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST 0/6] Support for serial logs from marilith boxes"):
> > > The marilith boxes use a conserver (http://www.conserver.com/) setup for
> > > serial access. Our installation exports the logs via http allowing us to
> > > grab them with wget.
> > > 
> > > Sending debug keys with is handled separately via xenuse. xenuse
> > > ultimately speaks to the conserver too but it abstracts away the IP and
> > > port to use so this is preferred.
> > > 
> > > With these changes the correct Serial hostprop for a marilith box is:
> > >         xenuse;http http://conserver.uk.xensource.com/consoles/
> > 
> > These all look plausible to me (as do your previous to); please go
> > ahead and push them.
> 
> I assumed s/to/two/ and pushed all eight:

I also ran:
        
        ./mg-hosts setprops marilith-n5 -- serial 'xenuse;http http://conserver.uk.xensource.com/consoles/'

Ian.