[OSSTEST PATCH 0/8] Log publication from Citrix Cambridge

[OSSTEST PATCH 0/8] Log publication from Citrix Cambridge

[Ian Jackson]

From: Ian Jackson <Ian.Jackson@eu.citrix.com>

We finally have a VM host for publishing the test logs from the Citrix
instance of osstest.  It needs an ssh bounce to get to the actual
public webserver, so we add support for that.

With this, and appropriate ssh keys, osstest@osstest.xs can run
cr-publish-flight-logs, cr-ensure-disk-space

Ian Jackson (8):
  cr-ensure-disk-space: With -D, print check_space command
  Publish: Default LogsPublish and ResultsPublish from Publish
  cr-publish-flight-logs: Refactor copydir args
  cr-publish-flight-logs: Refactor rsync -e option construction
  Publish: Introduce publish_ssh_opts
  Publish: Introduce {Logs,Results}PublishSshOpts
  Publish: Cambridge: Publish to new public host
  Publish: Cambridge: Get the public URL right

 Osstest.pm                  |  5 +++--
 Osstest/Management.pm       | 10 ++++++++--
 cr-ensure-disk-space        |  5 ++++-
 cr-publish-flight-logs      | 14 +++++++++-----
 production-config           |  3 ---
 production-config-cambridge |  5 ++++-
 6 files changed, 28 insertions(+), 14 deletions(-)

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 1/8] cr-ensure-disk-space: With -D, print check_space command

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 7091314..3e0288f 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -25,6 +25,7 @@ BEGIN { unshift @INC, qw(.); }
 use Osstest;
 use Osstest::Management qw(:logs);
 use Fcntl qw(:flock);
+use Data::Dumper;
 
 our $dryrun= 0;
 our $force;
@@ -56,7 +57,9 @@ csreadconfig();
 logs_select $cfgbase or exit 0;
 
 sub check_space () {
-    open P, "-|", onloghost "df --block-size=1M -P $logdir" or die $!;
+    my @cmd = onloghost "df --block-size=1M -P $logdir";
+    print DEBUG "check_space: ", Dumper(\@cmd);
+    open P, "-|", @cmd or die $!;
     $_= <P>;
     m/^filesystem/i or die "$_ ?";
     $_= <P>;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 2/8] Publish: Default LogsPublish and ResultsPublish from Publish

[Ian Jackson]

And delete the explicit settings from production-config.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest.pm        | 4 ++--
 production-config | 3 ---
 2 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 3377ea3..738ed4f 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -255,8 +255,8 @@ END
     my $pubbaseprefix = $c{PubBaseDir} ? "$c{PubBaseDir}/" : "";
     foreach my $l (qw(logs results)) {
 	my $u = ucfirst $l;
-	next if defined $c{$u};
-	$c{"${u}"} = "$pubbaseprefix$l";
+	$c{"${u}"} //= "$pubbaseprefix$l";
+	$c{"${u}Publish"} //= "$c{Publish}/$l" if defined $c{Publish};
     }
 
     $c{Stash} //= $c{Logs};
diff --git a/production-config b/production-config
index 48b1e8e..df02cd3 100644
--- a/production-config
+++ b/production-config
@@ -59,9 +59,6 @@ ReportHtmlUnpubBaseUrl="http://osstest/~osstest/pub/logs/"
 Publish osstest@www:/var/www/osstest
 GlobalLockDir /home/osstest/testing.git
 
-LogsPublish= "$c{Publish}/logs"
-ResultsPublish= "$c{Publish}/results"
-
 HarnessPublishGitUserHost osstest@xenbits.xen.org
 HarnessPublishGitRepoDir ext/osstest-massachusetts.git
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 3/8] cr-publish-flight-logs: Refactor copydir args

[Ian Jackson]

Have it take $cfgbase and $subdir instead.  This allows us to lift the
config test into it.  And it will be even more useful in a moment.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-publish-flight-logs | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/cr-publish-flight-logs b/cr-publish-flight-logs
index 1e5a49d..539318d 100755
--- a/cr-publish-flight-logs
+++ b/cr-publish-flight-logs
@@ -57,7 +57,10 @@ if ($push_harness) {
 }
 
 sub copydir ($$) {
-    my ($src,$dst) = @_;
+    my ($cfgbase,$subdir) = @_;
+    return unless $c{"${cfgbase}Publish"};
+    my $src = $c{$cfgbase}.$subdir."/";
+    my $dst = $c{"${cfgbase}Publish"}.$subdir;
     my @cmd= qw(rsync --compress --compress-level=9 --stats --delete -auH);
     push @cmd, '-e', 'ssh -o batchmode=yes';
     #--bwlimit=50
@@ -66,6 +69,5 @@ sub copydir ($$) {
     $!=0; $?=0; system @cmd; die "rsync $? $!" if $? or $!;
 }
 
-copydir("$c{Logs}/$flight/", "$c{LogsPublish}/$flight")
-    if $flight && $c{LogsPublish};
-copydir("$c{Results}/", "$c{ResultsPublish}") if $c{ResultsPublish};
+copydir(qw(Logs), "/$flight") if $flight;
+copydir(qw(Results), '');
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 4/8] cr-publish-flight-logs: Refactor rsync -e option construction

[Ian Jackson]

Previously this was hardcoded.  Now we make a variable @ssh, and use
rsync's quoting scheme to transform it into a value suitable for -e.

No overall functional change, although now the rsync command contains
additional quotes in the -e option.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-publish-flight-logs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cr-publish-flight-logs b/cr-publish-flight-logs
index 539318d..9e13652 100755
--- a/cr-publish-flight-logs
+++ b/cr-publish-flight-logs
@@ -61,8 +61,9 @@ sub copydir ($$) {
     return unless $c{"${cfgbase}Publish"};
     my $src = $c{$cfgbase}.$subdir."/";
     my $dst = $c{"${cfgbase}Publish"}.$subdir;
+    my @ssh = qw(ssh -o batchmode=yes);
     my @cmd= qw(rsync --compress --compress-level=9 --stats --delete -auH);
-    push @cmd, '-e', 'ssh -o batchmode=yes';
+    push @cmd, '-e', join(' ', map { s/\'/''/g; "'$_'"; } @ssh);
     #--bwlimit=50
     push @cmd, $src, $dst;
     print "+ @cmd\n";
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 5/8] Publish: Introduce publish_ssh_opts

[Ian Jackson]

We need to pass it $cfgbase because this will soon be configurable.

No functional change right now.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/Management.pm  | 10 ++++++++--
 cr-publish-flight-logs |  3 ++-
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/Osstest/Management.pm b/Osstest/Management.pm
index 2c875a7..c1ed2ac 100644
--- a/Osstest/Management.pm
+++ b/Osstest/Management.pm
@@ -27,7 +27,7 @@ BEGIN {
     our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
     $VERSION     = 1.00;
     @ISA         = qw(Exporter);
-    @EXPORT      = qw(
+    @EXPORT      = qw(publish_ssh_opts
                       );
     %EXPORT_TAGS = (
 	'logs' => [qw(logs_select onloghost logcfg
@@ -40,7 +40,12 @@ BEGIN {
 }
 
 our ($logcfgbase, $loghost, $logdir);
-our @logsshopts= qw(-o batchmode=yes);
+our @logsshopts;
+
+sub publish_ssh_opts ($) {
+    my ($cfgbase) = @_; # LogsPublish or ResultsPublish
+    return (qw(-o batchmode=yes));
+}
 
 sub logs_select ($) {
     ($logcfgbase) = @_;
@@ -48,6 +53,7 @@ sub logs_select ($) {
     return 0 unless $cfgvalue;
     if ($cfgvalue =~ m/\:/) {
 	($loghost, $logdir) = ($`,$'); #');
+	@logsshopts = publish_ssh_opts($logcfgbase);
     } else {
 	($loghost, $logdir) = (undef, $cfgvalue);
     }
diff --git a/cr-publish-flight-logs b/cr-publish-flight-logs
index 9e13652..7249d03 100755
--- a/cr-publish-flight-logs
+++ b/cr-publish-flight-logs
@@ -21,6 +21,7 @@ use strict qw(refs vars);
 use Fcntl qw(:flock);
 BEGIN { unshift @INC, qw(.); }
 use Osstest;
+use Osstest::Management;
 
 our %c;
 
@@ -61,7 +62,7 @@ sub copydir ($$) {
     return unless $c{"${cfgbase}Publish"};
     my $src = $c{$cfgbase}.$subdir."/";
     my $dst = $c{"${cfgbase}Publish"}.$subdir;
-    my @ssh = qw(ssh -o batchmode=yes);
+    my @ssh = (qw(ssh), publish_ssh_opts("${cfgbase}Publish"));
     my @cmd= qw(rsync --compress --compress-level=9 --stats --delete -auH);
     push @cmd, '-e', join(' ', map { s/\'/''/g; "'$_'"; } @ssh);
     #--bwlimit=50
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 6/8] Publish: Introduce {Logs, Results}PublishSshOpts

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest.pm            | 1 +
 Osstest/Management.pm | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index 738ed4f..85a6e78 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -257,6 +257,7 @@ END
 	my $u = ucfirst $l;
 	$c{"${u}"} //= "$pubbaseprefix$l";
 	$c{"${u}Publish"} //= "$c{Publish}/$l" if defined $c{Publish};
+	$c{"${u}PublishSshOpts"} //= $c{PublishSshOpts} // [];
     }
 
     $c{Stash} //= $c{Logs};
diff --git a/Osstest/Management.pm b/Osstest/Management.pm
index c1ed2ac..4edb361 100644
--- a/Osstest/Management.pm
+++ b/Osstest/Management.pm
@@ -44,7 +44,7 @@ our @logsshopts;
 
 sub publish_ssh_opts ($) {
     my ($cfgbase) = @_; # LogsPublish or ResultsPublish
-    return (qw(-o batchmode=yes));
+    return (qw(-o batchmode=yes), @{$c{"${cfgbase}SshOpts"}});
 }
 
 sub logs_select ($) {
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 7/8] Publish: Cambridge: Publish to new public host

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 production-config-cambridge | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/production-config-cambridge b/production-config-cambridge
index f557614..b1360ba 100644
--- a/production-config-cambridge
+++ b/production-config-cambridge
@@ -41,6 +41,9 @@ OverlayLocal /export/home/osstest/overlay-local
 LogsMinSpaceMby= 10*1e3
 LogsMinExpireAge= 86400*4
 
+Publish osstest@10.59.48.19:/var/www/osstest
+PublishSshOpts= ['-o','ProxyCommand ssh -W 10.59.48.19:22 osstest@10.59.128.11']
+
 TestHostKeypairPath /export/home/osstest/.ssh/id_rsa_osstest
 
 GitCacheProxy git://git-cache.daemon.osstest.xs.citrite.net:9419/
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 8/8] Publish: Cambridge: Get the public URL right

[Ian Jackson]

This URL is now accessible, although there are some webserver tweaks
remaining to do.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 production-config-cambridge | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/production-config-cambridge b/production-config-cambridge
index b1360ba..8647feb 100644
--- a/production-config-cambridge
+++ b/production-config-cambridge
@@ -48,7 +48,7 @@ TestHostKeypairPath /export/home/osstest/.ssh/id_rsa_osstest
 
 GitCacheProxy git://git-cache.daemon.osstest.xs.citrite.net:9419/
 
-PubBaseUrl http://osstest.xs.citrite.net/~osstest/testlogs
+PubBaseUrl http://osstest.xensource.com/osstest
 ReportHtmlPubBaseUrl="$c{PubBaseUrl}/logs"
 ResultsHtmlPubBaseUrl="$c{PubBaseUrl}/results"
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel