[OSSTEST PATCH v16 00/22] Nested HVM

[OSSTEST PATCH v16 00/22] Nested HVM

[Ian Jackson]

This series is also available at:
  git://xenbits.xen.org/people/iwj/osstest.git
  http://xenbits.xen.org/git-http/people/iwj/osstest.git
in wip.nested-hvm.v16..wip.nested-hvm.v16

The first three of these patches are IMO ready to go in whenever.

The rest of them are ready, but are part of the Nested HVM support and
should wait for the whole series, or they're just dead code in-tree.

There are only three patches with substantial changes not yet seen on
xen-devel:

 + 20/22 HVM guests: Use qemu "pipe:" for serial output
 + 21/22 Serial: Factor out Osstest::Serial::keys_real
 * 22/22 Serial: Add new serial method object for

The first two are entirely new cleanup patches.  I have (fairly
lightly) tested them (and if this series goes into pretest will take
responsibility for fixing them up).

The final patch - for nested HVM L1 debug keys - is completely
untested.  I'm expecting Robert to test it and report back.

Ian: I'm not sure whether or not you want to review these.  About half
of the patches are mine; the other half are Robert's but have been
reviewed by me.  All but the last three have been tested by Robert.

Thanks,
Ian.

[OSSTEST PATCH 01/22] cs-adjust-flight: Add some missing doc comment info

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v14: New patch
---
 cs-adjust-flight |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/cs-adjust-flight b/cs-adjust-flight
index 4bfef48..d70abf7 100755
--- a/cs-adjust-flight
+++ b/cs-adjust-flight
@@ -24,6 +24,14 @@
 #   .                 means all jobs
 #   ^<pcre>           means $foo =~ m/^<pcre>/
 #   /<pcre>           means $foo =~ m/<pcre>/
+#
+# <dst-flight>:
+#   <flight>
+#   new:<intended-blessing>
+#
+# options:
+#   -v              verbose - list changes to stderr
+#   -D              debugging output to stderr
 
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
-- 
1.7.10.4

Re: [OSSTEST PATCH 01/22] cs-adjust-flight: Add some missing doc comment info

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 02/22] cs-adjust-flight: Allow adjusting "this" flight

[Ian Jackson]

This allows cs-adjust-flight to be run by hand to adjust runvars, in a
flight being used with hand-invocation of ./ts-* scripts.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v14: New patch
---
 cs-adjust-flight |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/cs-adjust-flight b/cs-adjust-flight
index d70abf7..7c2c384 100755
--- a/cs-adjust-flight
+++ b/cs-adjust-flight
@@ -28,6 +28,7 @@
 # <dst-flight>:
 #   <flight>
 #   new:<intended-blessing>
+#   running:                 uses OSSTEST_FLIGHT (may be `constructing' too)
 #
 # options:
 #   -v              verbose - list changes to stderr
@@ -395,6 +396,13 @@ sub main () {
             verbose_discard();
             changes();
         });
+    } elsif ($dstflightspec =~ m/^running:$/) {
+        $dstflight = $ENV{OSSTEST_FLIGHT} // die;
+        db_retry($dstflight,[qw(constructing running)],
+                 $dbh_tests, [qw(flights)], sub {
+            verbose_discard();
+            changes();
+        });
     } elsif ($dstflightspec =~ m/^new:/) {
         my $intended = $'; #';
         db_retry($dbh_tests, [qw(flights)], sub {
-- 
1.7.10.4

Re: [OSSTEST PATCH 02/22] cs-adjust-flight: Allow adjusting "this" flight

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> This allows cs-adjust-flight to be run by hand to adjust runvars, in a
> flight being used with hand-invocation of ./ts-* scripts.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
> v14: New patch
> ---
>  cs-adjust-flight |    8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/cs-adjust-flight b/cs-adjust-flight
> index d70abf7..7c2c384 100755
> --- a/cs-adjust-flight
> +++ b/cs-adjust-flight
> @@ -28,6 +28,7 @@
>  # <dst-flight>:
>  #   <flight>
>  #   new:<intended-blessing>
> +#   running:                 uses OSSTEST_FLIGHT (may be `constructing' too)

I understanding the parenthetical to mean that the flight referenced by
$OSSTEST_FLIGHT is allowed to have a current (not intended) blessing of
`constructing' as well as the `running' implied by the option name, but I
suspect that other than you I'm one of the few people familiar enough with
osstest to reach that conclusion.

It's a bit verbose but I think I would make it say something like "(the
referenced flight must currently be blessed either `running' or
`constructing')"

>  #
>  # options:
>  #   -v              verbose - list changes to stderr
> @@ -395,6 +396,13 @@ sub main () {
>              verbose_discard();
>              changes();
>          });
> +    } elsif ($dstflightspec =~ m/^running:$/) {
> +        $dstflight = $ENV{OSSTEST_FLIGHT} // die;
> +        db_retry($dstflight,[qw(constructing running)],
> +                 $dbh_tests, [qw(flights)], sub {
> +            verbose_discard();
> +            changes();
> +        });

The actual code looks fine to me.

>      } elsif ($dstflightspec =~ m/^new:/) {
>          my $intended = $'; #';
>          db_retry($dbh_tests, [qw(flights)], sub {
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 03/22] selecthost: Minor cleanups

[Ian Jackson]

Document the syntax for $ident.

Log the ident as well as the selected hostname.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: New patch
---
 Osstest/TestSupport.pm |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index aa41952..5e88add 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -810,6 +810,20 @@ sub power_state ($$) {
 sub selecthost ($) {
     my ($ident) = @_;
     # must be run outside transaction
+
+    # $ident is <identspec>
+    #
+    # <identspec> can be <ident>, typically "host" or "xxx_host"
+    # which means look up the runvar $r{$ident} which
+    # contains <hostspec>
+    # OR
+    # <identspec> can be <ident>=<hostspec>
+    # which means ignore <ident> except for logging purposes etc.
+    # and use <hostspec>
+    #
+    # <hostspec> uis <hostname> which means use that host (and all
+    # its flags and properties from the configuration and database)
+
     my $name;
     if ($ident =~ m/=/) {
         $ident= $`;
@@ -917,7 +931,7 @@ sub selecthost ($) {
 
     $mjobdb->host_check_allocated($ho);
 
-    logm("host: selected $ho->{Name} ".
+    logm("host $ho->{Ident}: selected $ho->{Name} ".
 	 (defined $ho->{Ether} ? $ho->{Ether} : '<unknown-ether>').
 	 " $ho->{Ip}".
          (!$ho->{Shared} ? '' :
-- 
1.7.10.4

Re: [OSSTEST PATCH 03/22] selecthost: Minor cleanups

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> Document the syntax for $ident.
> 
> Log the ident as well as the selected hostname.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>
> ---
> v14: New patch
> ---
>  Osstest/TestSupport.pm |   16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
> index aa41952..5e88add 100644
> --- a/Osstest/TestSupport.pm
> +++ b/Osstest/TestSupport.pm
> @@ -810,6 +810,20 @@ sub power_state ($$) {
>  sub selecthost ($) {
>      my ($ident) = @_;
>      # must be run outside transaction
> +
> +    # $ident is <identspec>
> +    #
> +    # <identspec> can be <ident>, typically "host" or "xxx_host"
> +    # which means look up the runvar $r{$ident} which
> +    # contains <hostspec>
> +    # OR
> +    # <identspec> can be <ident>=<hostspec>
> +    # which means ignore <ident> except for logging purposes etc.
> +    # and use <hostspec>
> +    #
> +    # <hostspec> uis <hostname> which means use that host (and all

                    ^is

o/w: Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 04/22] selecthost: Support nested hosts (guests which are also hosts)

[Ian Jackson]

We introduce a new syntax: instead of a hostname (which might appear
in a command line argument to a ts-* script and hence be passed to
selecthost, or which might be in a runvar), we now support
<hostspec>:<domname>.

Such `hosts' (let us refer to such a thing as an L1, although in
principle further nesting may be possible) are expected to be
dynamically created.  So they do not have flags and properties in the
configuration (or in an Executive instance's database).

The IP address is determined dynamically from the leases file.  If the
L1 is not running, then no IP address may be found.  This is not an
error.  Users of this facility will need to make sure that ts-*
scripts which are unaware of the L1's special status are only invoked
when it is known that the L1 is up and has obtained its IP address.

`Power cycling' the L1 will be done by VM control operations in the
L0; this will come in a subsequent patch.

`Serial access' to the L1 guest will likewise need to be done via the
console arrangements in L0.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: New patch
---
 Osstest/TestSupport.pm |   37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 5e88add..3145040 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -807,6 +807,7 @@ sub power_state ($$) {
 
 #---------- host selection and properties ----------
 
+sub selecthost ($);
 sub selecthost ($) {
     my ($ident) = @_;
     # must be run outside transaction
@@ -821,8 +822,12 @@ sub selecthost ($) {
     # which means ignore <ident> except for logging purposes etc.
     # and use <hostspec>
     #
-    # <hostspec> uis <hostname> which means use that host (and all
+    # <hostspec> can be <hostname> which means use that host (and all
     # its flags and properties from the configuration and database)
+    # OR
+    # <hostspec> can be <parent_identspec>:<domname> meaning use the
+    # Xen domain name <domname> on the host specified by
+    # <parent_identspec>, which is an <identspec> as above.
 
     my $name;
     if ($ident =~ m/=/) {
@@ -838,6 +843,7 @@ sub selecthost ($) {
         Ident => $ident,
         Name => $name,
         TcpCheckPort => 22,
+        NestingLevel => 0,
         Info => [],
     };
     if (defined $job) {
@@ -845,6 +851,35 @@ sub selecthost ($) {
 					  $c{DebianSuite});
     }
 
+    #----- handle hosts which are themselves guests (nested) -----
+
+    if ($name =~ s/^(.*)://) {
+	my $parentname = $1;
+	my $parent = selecthost($parentname);
+	my $child = selectguest($name,$parent);
+	$child->{Ident} = $ho->{Ident};
+	$child->{Info} = [ "in", $parent->{Name}, @{ $parent->{Info} } ];
+	$child->{NestingLevel} = $parent->{NestingLevel}+1;
+
+	# $child->{Power} = 'guest';   todo
+	power_cycle_host_setup($child);
+
+	$child->{Properties}{Serial} = 'noop'; # todo
+	serial_host_setup($child);
+
+	my $msg = "L$child->{NestingLevel} host $child->{Ident}:";
+	$msg .= " guest $child->{Guest} (@{ $child->{Info} })";
+	$msg .= " $child->{Ether}";
+
+	my $err = guest_check_ip($child);
+	$msg .= " ".(defined $err ? "<no-ip> $err" : $child->{Ip});
+
+	logm($msg);
+
+	# all the rest of selecthost is wrong for this case
+	return $child;
+    }
+
     #----- calculation of the host's properties -----
 
     $ho->{Properties} = { };
-- 
1.7.10.4

Re: [OSSTEST PATCH 04/22] selecthost: Support nested hosts (guests which are also hosts)

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> We introduce a new syntax: instead of a hostname (which might appear
> in a command line argument to a ts-* script and hence be passed to
> selecthost, or which might be in a runvar), we now support
> <hostspec>:<domname>.
> 
> Such `hosts' (let us refer to such a thing as an L1, although in
> principle further nesting may be possible) are expected to be
> dynamically created.  So they do not have flags and properties in the
> configuration (or in an Executive instance's database).
> 
> The IP address is determined dynamically from the leases file.  If the
> L1 is not running, then no IP address may be found.  This is not an
> error.  Users of this facility will need to make sure that ts-*
> scripts which are unaware of the L1's special status are only invoked
> when it is known that the L1 is up and has obtained its IP address.
> 
> `Power cycling' the L1 will be done by VM control operations in the
> L0; this will come in a subsequent patch.
> 
> `Serial access' to the L1 guest will likewise need to be done via the
> console arrangements in L0.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 05/22] Nested hosts: Provide PDU power method

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

This `guest' power method uses VM create/destroy.  It is automatically
used for nested hosts.  It would not make much sense to configure it
manually.

For nested host/guest, its power on/off method shall be
its host invoke $(toolstack)->create/destroy method.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: Mostly rewritten by iwj
---
 Osstest/PDU/guest.pm   |   59 ++++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm |    2 +-
 2 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100755 Osstest/PDU/guest.pm

diff --git a/Osstest/PDU/guest.pm b/Osstest/PDU/guest.pm
new file mode 100755
index 0000000..b6bf9a1
--- /dev/null
+++ b/Osstest/PDU/guest.pm
@@ -0,0 +1,59 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::PDU::guest;
+
+use strict;
+use warnings;
+use Switch;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho) = @_;
+    return bless { Target => $ho }, $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+
+    my $child = $mo->{Target};
+    my $parent = $child->{Host};
+    die "$child->{Name} ?" unless $parent;
+
+    logm("power $child->{Name} nested on $parent->{Name} ".($on+0));
+    if ($on) {
+	toolstack($parent)->create($child);
+    } else {
+	toolstack($parent)->destroy($child);
+    }
+}
+
+1;
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 3145040..9a256a9 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -861,7 +861,7 @@ sub selecthost ($) {
 	$child->{Info} = [ "in", $parent->{Name}, @{ $parent->{Info} } ];
 	$child->{NestingLevel} = $parent->{NestingLevel}+1;
 
-	# $child->{Power} = 'guest';   todo
+	$child->{Power} = 'guest';
 	power_cycle_host_setup($child);
 
 	$child->{Properties}{Serial} = 'noop'; # todo
-- 
1.7.10.4

Re: [OSSTEST PATCH 05/22] Nested hosts: Provide PDU power method

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> This `guest' power method uses VM create/destroy.  It is automatically
> used for nested hosts.  It would not make much sense to configure it
> manually.
> 
> For nested host/guest, its power on/off method shall be
> its host invoke $(toolstack)->create/destroy method.
> 
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 06/22] DhcpWatch::leases: Fix a reporting message

[Ian Jackson]

This talks about `guest_check_ip', but this code is now factored out
into a method.  Use the correct method name in reporting.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: New patch
---
 Osstest/DhcpWatch/leases.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/DhcpWatch/leases.pm b/Osstest/DhcpWatch/leases.pm
index 9a74c40..b74ebf0 100644
--- a/Osstest/DhcpWatch/leases.pm
+++ b/Osstest/DhcpWatch/leases.pm
@@ -171,7 +171,7 @@ sub check_ip ($$) {
     }
     $gho->{Ip}= $best->{' addr'};
 
-    report_once($gho, 'guest_check_ip', 
+    report_once($gho, 'leases::check_ip', 
 		"guest $gho->{Name}: $gho->{Ether} $gho->{Ip}");
     return undef;
 }
-- 
1.7.10.4

Re: [OSSTEST PATCH 06/22] DhcpWatch::leases: Fix a reporting message

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> This talks about `guest_check_ip', but this code is now factored out
> into a method.  Use the correct method name in reporting.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 07/22] target_check_ip: Rename and improve from guest_check_ip

[Ian Jackson]

Make this function suitable for running on targets with static IP
addresses.  (Ie, on physical hosts.)  Accordingly, rename it and
adjust all call sites.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: New patch
---
 Osstest/TestSupport.pm |   11 ++++++-----
 ts-guest-localmigrate  |    2 +-
 ts-guest-migrate       |    2 +-
 ts-guest-saverestore   |    2 +-
 4 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 9a256a9..325e2e3 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -95,7 +95,7 @@ BEGIN {
                       prepareguest_part_lvmdisk prepareguest_part_diskimg
                       prepareguest_part_xencfg
                       guest_umount_lv guest_await guest_await_dhcp_tcp
-                      guest_checkrunning guest_check_ip guest_find_ether
+                      guest_checkrunning target_check_ip guest_find_ether
                       guest_find_domid guest_check_up guest_check_up_quick
                       guest_get_state guest_await_reboot
                       guest_await_shutdown guest_await_destroy guest_destroy
@@ -730,8 +730,9 @@ sub dhcp_watch_setup ($$) {
     $gho->{DhcpWatch} = get_host_method_object($ho, 'DhcpWatch', $meth);
 }
 
-sub guest_check_ip ($) {
-    my ($gho) = @_;
+sub target_check_ip ($) {
+    my ($gho) = @_; # returns error message or undef
+    return undef if $gho->{IpStatic};
     guest_find_ether($gho);
     $gho->{DhcpWatch}->check_ip($gho);
 }
@@ -871,7 +872,7 @@ sub selecthost ($) {
 	$msg .= " guest $child->{Guest} (@{ $child->{Info} })";
 	$msg .= " $child->{Ether}";
 
-	my $err = guest_check_ip($child);
+	my $err = target_check_ip($child);
 	$msg .= " ".(defined $err ? "<no-ip> $err" : $child->{Ip});
 
 	logm($msg);
@@ -1956,7 +1957,7 @@ sub guest_await_dhcp_tcp ($$) {
 	      " $gho->{TcpCheckPort}".
               " link/ip/tcp",
               sub {
-        my $err= guest_check_ip($gho);
+        my $err= target_check_ip($gho);
         return $err if defined $err;
 
         return
diff --git a/ts-guest-localmigrate b/ts-guest-localmigrate
index 8fe986d..85a0887 100755
--- a/ts-guest-localmigrate
+++ b/ts-guest-localmigrate
@@ -40,7 +40,7 @@ sub migrate () {
 guest_await_dhcp_tcp($gho, 5);
 guest_check_up($gho);
 
-my $err= guest_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
+my $err= target_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
 
 for (my $rep=1; $rep<=$reps; $rep++) {
     logm("========== rep $rep ==========");
diff --git a/ts-guest-migrate b/ts-guest-migrate
index b77d0de..505fab2 100755
--- a/ts-guest-migrate
+++ b/ts-guest-migrate
@@ -30,7 +30,7 @@ our $gho = selectguest($ARGV[2],$sho);
 
 sub migrate () {
     guest_checkrunning($sho,$gho) or die $gho->{Name};
-    my $err= guest_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
+    my $err= target_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
     toolstack($sho)->migrate($gho, $dho, $timeout{Migrate});
 }
 
diff --git a/ts-guest-saverestore b/ts-guest-saverestore
index 7e13d90..73883ef 100755
--- a/ts-guest-saverestore
+++ b/ts-guest-saverestore
@@ -26,7 +26,7 @@ our ($ho,$gho) = ts_get_host_guest(@ARGV);
 
 sub save () {
     guest_checkrunning($ho,$gho) or die $gho->{Name};
-    my $err= guest_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
+    my $err= target_check_ip($gho);  die "$err $gho->{Name}" if defined $err;
     toolstack($ho)->save($gho,"image",200);
     target_ping_check_down($gho);
 }
-- 
1.7.10.4

Re: [OSSTEST PATCH 07/22] target_check_ip: Rename and improve from guest_check_ip

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> Make this function suitable for running on targets with static IP
> addresses.  (Ie, on physical hosts.)  Accordingly, rename it and
> adjust all call sites.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 08/22] await_tcp(): Run check_ip on each loop iteration

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

await_tcp is often invoked after a reboot.

In this situation the target's IP address may change.  If this happens
while await_tcp is running, we would continue to poll the old IP address.
Fix this by running target_check_ip on each iteration.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: Dropped change to selecthost, which was in code which is no
      longer present in this version of the series.
     Rewritten to use target_check_ip.
     Dropped IMO-unnecessary comment.
---
 Osstest/TestSupport.pm |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 325e2e3..b70f19f 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -2029,9 +2029,10 @@ sub await_tcp ($$$) {
     my ($maxwait,$interval,$ho) = @_;
     target_adjust_timeout($ho,\$maxwait);
     poll_loop($maxwait,$interval,
-              "await tcp $ho->{Name} $ho->{TcpCheckPort}",
+              "await tcp $ho->{Name} $ho->{Ip} $ho->{TcpCheckPort}",
               sub {
-        return target_tcp_check($ho,$interval);
+	return target_check_ip($ho) //
+               target_tcp_check($ho,$interval);
     });
 }
 
-- 
1.7.10.4

Re: [OSSTEST PATCH 08/22] await_tcp(): Run check_ip on each loop iteration

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> await_tcp is often invoked after a reboot.
> 
> In this situation the target's IP address may change.  If this happens
> while await_tcp is running, we would continue to poll the old IP address.
> Fix this by running target_check_ip on each iteration.
> 
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 09/22] LVM: Break out lv_create

[Ian Jackson]

We are going to want to reuse this.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: New patch
v15: Change some trivial typo, so to resolve conflicts with
     production tree.
---
 Osstest/TestSupport.pm |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index b70f19f..b507ada 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -63,7 +63,7 @@ BEGIN {
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
                       target_extract_jobdistpath
-                      lv_dev_mapper
+                      lv_create lv_dev_mapper
 
                       poll_loop tcpconnect await_tcp
                       contents_make_cpio file_simple_write_contents
@@ -713,6 +713,15 @@ sub poll_loop ($$$&) {
     logm("$what: ok. (${waited}s)");
 }
 
+sub lv_create ($$$$) {
+    my ($ho, $vg, $lv, $mb) = @_;
+    my $lvdev = "/dev/$vg/$lv";
+    target_cmd_root($ho, "lvremove -f $lvdev ||:");
+    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
+    return $lvdev;
+}
+
 sub lv_dev_mapper ($$) {
     my ($vg,$lv) = @_;
     $vg =~ s/-/--/g;
@@ -1692,9 +1701,7 @@ sub prepareguest ($$$$$$) {
 
 sub prepareguest_part_lvmdisk ($$$) {
     my ($ho, $gho, $disk_mb) = @_;
-    target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
-    target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho->{Vg}");
-    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
+    lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb);
 }
 
 sub make_vhd ($$$) {
-- 
1.7.10.4

Re: [OSSTEST PATCH 09/22] LVM: Break out lv_create

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> 
> +sub lv_create ($$$$) {
> +    my ($ho, $vg, $lv, $mb) = @_;
> +    my $lvdev = "/dev/$vg/$lv";

In the original code it was using $gho->{Lvdev}, is this semantic change
deliberate or a rebase-o? If the former then I think it warrants a comment
in the commit message.

> +    target_cmd_root($ho, "lvremove -f $lvdev ||:");
> +    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
> +    target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
> +    return $lvdev;
> +}
> +
>  sub lv_dev_mapper ($$) {
>      my ($vg,$lv) = @_;
>      $vg =~ s/-/--/g;
> @@ -1692,9 +1701,7 @@ sub prepareguest ($$$$$$) {
>  
>  sub prepareguest_part_lvmdisk ($$$) {
>      my ($ho, $gho, $disk_mb) = @_;
> -    target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
> -    target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho-
> >{Vg}");
> -    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
> +    lv_create($ho, $gho->{Vg}, $gho->{Lv}, $disk_mb);
>  }
>  
>  sub make_vhd ($$$) {

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 09/22] LVM: Break out lv_create

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 09/22] LVM: Break out lv_create"):
> On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> > +sub lv_create ($$$$) {
> > +    my ($ho, $vg, $lv, $mb) = @_;
> > +    my $lvdev = "/dev/$vg/$lv";
> 
> In the original code it was using $gho->{Lvdev}, is this semantic change
> deliberate or a rebase-o? If the former then I think it warrants a comment
> in the commit message.

It was deliberate.  I have added this to the commit message:

 lv_create doesn't (want to) take a $gho, but the $vg and $lv names
 directly (so that callers can use it when they don't have a suitable
 $gho whose $gho->{Lvdev} they want to use).

 In the one existing call site we pass $gho->{Vg} and $gho->{Lv} so
 that the effect is the same.

 There is a minor functional change: $gho->{Lvdev} has been put through
 lv_dev_mapper.  But we don't care about that in lv_create (since the
 LVM operations, and dd, are perfectly happy to use the `real',
 non-/dev/mapper, names).  So we can just use /dev/$vg/$lv.

Ian.

Re: [OSSTEST PATCH 09/22] LVM: Break out lv_create

[Ian Campbell]

On Wed, 2015-11-11 at 16:30 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 09/22] LVM: Break out
> lv_create"):
> > On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> > > +sub lv_create ($$$$) {
> > > +    my ($ho, $vg, $lv, $mb) = @_;
> > > +    my $lvdev = "/dev/$vg/$lv";
> > 
> > In the original code it was using $gho->{Lvdev}, is this semantic
> > change
> > deliberate or a rebase-o? If the former then I think it warrants a
> > comment
> > in the commit message.
> 
> It was deliberate.  I have added this to the commit message:
> 
>  lv_create doesn't (want to) take a $gho, but the $vg and $lv names
>  directly (so that callers can use it when they don't have a suitable
>  $gho whose $gho->{Lvdev} they want to use).
> 
>  In the one existing call site we pass $gho->{Vg} and $gho->{Lv} so
>  that the effect is the same.
> 
>  There is a minor functional change: $gho->{Lvdev} has been put through
>  lv_dev_mapper.  But we don't care about that in lv_create (since the
>  LVM operations, and dd, are perfectly happy to use the `real',
>  non-/dev/mapper, names).  So we can just use /dev/$vg/$lv.

That looks good. The "return $lvdev" from this new function would slightly
encourage using this path in other situations which are not lv* commands
and which might want the mapper version. I think this is a remote enough
possibility to not worry about, although maybe you want to add a comment.

In any case:

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 09/22] LVM: Break out lv_create

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 09/22] LVM: Break out lv_create"):
> On Wed, 2015-11-11 at 16:30 +0000, Ian Jackson wrote:
> >  There is a minor functional change: $gho->{Lvdev} has been put through
> >  lv_dev_mapper.  But we don't care about that in lv_create (since the
> >  LVM operations, and dd, are perfectly happy to use the `real',
> >  non-/dev/mapper, names).  So we can just use /dev/$vg/$lv.
> 
> That looks good. The "return $lvdev" from this new function would slightly
> encourage using this path in other situations which are not lv* commands
> and which might want the mapper version. I think this is a remote enough
> possibility to not worry about, although maybe you want to add a comment.

Yes, I guess, although if it breaks it will be obvious.  I don't think
I'll add a comment on that; it's obvious enough and the comment would
be clutter.

> In any case:
> 
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,
Ian.

[OSSTEST PATCH 10/22] Toolstack::xl: Provide block_attach method

[Ian Jackson]

It is possible that this may work some of the time with xm, so I have
taken no measures to prevent it running then.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Signed-off-by: Robert Hu <robert.hu@intel.com>
Tested-by: Robert Hu <robert.hu@intel.com>
----
v14: New patch
v15: Fix missing $ho assignment in block-attach method
---
 Osstest/Toolstack/xl.pm |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/Osstest/Toolstack/xl.pm b/Osstest/Toolstack/xl.pm
index 0f8abed..d956bcd 100644
--- a/Osstest/Toolstack/xl.pm
+++ b/Osstest/Toolstack/xl.pm
@@ -109,4 +109,13 @@ sub restore ($$$$) {
 		    ." $f", $timeout);
 }
 
+sub block_attach ($$$$) {
+    my ($self,$gho,$xldiskspec) = @_;
+    die "quotes in $xldiskspec ?" if $xldiskspec =~ m/'/;
+    my $ho = $self->{Host};
+    my $gn = $gho->{Name};
+    my $cmd = $self->{_VerboseCommand}." block-attach $gn '$xldiskspec'";
+    target_cmd_root($ho, $cmd, 100);
+}
+
 1;
-- 
1.7.10.4

Re: [OSSTEST PATCH 10/22] Toolstack::xl: Provide block_attach method

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> It is possible that this may work some of the time with xm, so I have
> taken no measures to prevent it running then.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Signed-off-by: Robert Hu <robert.hu@intel.com>
> Tested-by: Robert Hu <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 11/22] sg-run-job: Break out per-host-prep and per-host-finish

[Ian Jackson]

No functional change.

We now call the per-host-ts finish steps unconditionally, rather than
only if !$need_build_host, per-host-ts is (complicated) no-op if
$need_build_host, since in that case $need_xen_hosts is {}.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested by: Robert Ho <robert.hu@intel.com>
---
v14: Squash typo fix from Robert into this patch
---
 sg-run-job |   31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index 66145b8..884a21d 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -23,6 +23,20 @@ source ./tcl/osstestlib.tcl
 readconfig
 source-method JobDB
 
+proc per-host-prep {} {
+    per-host-ts .       host-ping-check-native/@ ts-host-ping-check
+    per-host-ts .       xen-install/@     ts-xen-install
+    per-host-ts .       xen-boot/@        ts-host-reboot
+
+    per-host-ts .       host-ping-check-xen/@ ts-host-ping-check
+    per-host-ts .       =(*)             {ts-leak-check basis}
+}
+
+proc per-host-finish {} {
+    per-host-ts .       =                {ts-leak-check check}
+    per-host-ts !broken capture-logs/@(*) ts-logs-capture
+}
+
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
 
@@ -51,22 +65,15 @@ proc run-job {job} {
     if {$ok} { setstatus running                                          }
 
     per-host-ts broken  host-install/@(*) ts-host-install-twice
-    per-host-ts .       host-ping-check-native/@ ts-host-ping-check
-    per-host-ts .       xen-install/@     ts-xen-install
-    per-host-ts .       xen-boot/@        ts-host-reboot
-    per-host-ts .       host-ping-check-xen/@ ts-host-ping-check
 
-    per-host-ts .       =(*)             {ts-leak-check basis}
+    per-host-prep
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
-    per-host-ts .       =                {ts-leak-check check}
 
-    if {!$need_build_host} {
-        per-host-ts !broken capture-logs/@(*) ts-logs-capture
-    } else {
-        if {$anyfailed} {
-            run-ts  !broken capture-logs      ts-logs-capture + host
-        }
+    per-host-finish
+
+    if {$need_build_host && $anyfailed} {
+	run-ts  !broken capture-logs      ts-logs-capture + host
     }
 
     if {$ok} { setstatus pass                                             }
-- 
1.7.10.4

Re: [OSSTEST PATCH 11/22] sg-run-job: Break out per-host-prep and per-host-finish

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> No functional change.
> 
> We now call the per-host-ts finish steps unconditionally, rather than
> only if !$need_build_host, per-host-ts is (complicated) no-op if
> $need_build_host, since in that case $need_xen_hosts is {}.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 12/22] sg-run-job: Provide infrastructure for layers of nesting

[Ian Jackson]

Provides nested-layer-descend, which can be called in an individual
test job at the appropriate point (after the L1 has been set up).

The inner host is a guest of the outer host; powering it off means
destroying it.  Putting the poweroff at this point in the loop, rather
than in per-host-finish, avoids powering off physical servers.  The
use of `.'  rather than `!.' for iffail means we do not power off
after failures (as we might want to preserve the state for debugging
etc).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v14: Squash syntax fix from Robert Ho into this patch
v15: Remove spurious "=" from final-poweroff step invocation
---
 sg-run-job |   21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/sg-run-job b/sg-run-job
index 884a21d..172c87e 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -39,6 +39,7 @@ proc per-host-finish {} {
 
 proc run-job {job} {
     global jobinfo builds flight ok need_xen_hosts anyfailed
+    global nested_layers_hosts
 
     set ok 1
     set anyfailed 0
@@ -52,6 +53,7 @@ proc run-job {job} {
         set need_xen_hosts $nh
         set need_build_host 0
     }
+    set nested_layers_hosts {}
 
     catching-otherwise blocked            check-not-blocked
     if {!$ok} return
@@ -70,7 +72,15 @@ proc run-job {job} {
 
     if {$ok} { catching-otherwise fail      run-job/$jobinfo(recipe)      }
 
-    per-host-finish
+    while 1 {
+	per-host-finish
+	
+	if {![llength $nested_layers_hosts]} break
+
+	per-host-ts . final-poweroff {ts-host-powercycle --power=0}
+
+        set need_xen_hosts [lunappend nested_layers_hosts]
+    }
 
     if {$need_build_host && $anyfailed} {
 	run-ts  !broken capture-logs      ts-logs-capture + host
@@ -247,6 +257,15 @@ proc per-host-ts {iffail ident script args} {
     }
 }
 
+proc nested-layer-descend {nested_hosts} {
+    # We save need_xen_hosts on a stack in nested_layers_hosts
+    # It gets popped again during the cleanup part of run-job
+    global nested_layers_hosts need_xen_hosts
+    lappend nested_layers_hosts $need_xen_hosts
+    set need_xen_hosts $nested_hosts
+    per-host-prep
+}
+
 #---------- test recipes ----------
 
 proc need-hosts/test-debian-nomigr {} { return host }
-- 
1.7.10.4

[OSSTEST PATCH 13/22] Nested HVM: Provide ts-nested-setup to help make L1 usable as a host

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

* Provide the L1 with some storage for its own guests' disks
* Install some packages in the L1
* Optionally, set a runvar defining the L1 for the rest of the job

The recipe is going to run ts-xen-install etc.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: Use target_check_ip (renamed, earlier, in a new patch)
     ts-nested-setup now has a default gueststorage_size of 20G
      and this is implicitly used by make-flight.
     Adjusted for new selecthost nested host syntax and correspondingly
      completely changed invocation syntax.
     Only optionally sets the runvar, if you pass --define.  (This
      will make it easier to play around with interactively.)
     Broken the pieces of work out into subroutines for clarity.
     Comment about guest storage slightly edited and rewrapped.
     Guest storage runvars and perl variable names etc. renamed to
      `gueststorage' rather than `guest_storage'.
     LVM and VG names and perl variable names changed to be clearer.
     Install `ed' too.
     Use lv_create and toolstack()->block_attach.
     Dropped ack from Ian Campbell.
---
 ts-nested-setup |   99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 0000000..f7e0ebd
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,99 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our $dodefine;
+if (@ARGV && $ARGV[0] eq '--define') { $dodefine=1; shift @ARGV; }
+
+our ($l1_identspec) = @ARGV;
+
+my $l1 = selecthost($l1_identspec);
+my $l0 = $l1->{Host};
+
+die unless $l0;
+
+our $def_gueststorage_size = 20000; # Mby
+
+sub packages () {
+    # Faster to install packages while L1 is still running PVHVM
+    target_install_packages_norec($l1, qw(lvm2 rsync ed genisoimage));
+}
+
+sub guest_storage () {
+    # We need to attach an extra disk to the L1 guest to be used as L2
+    # guest storage:
+    #
+    # When running in a nested HVM environment the L1 domain is acting
+    # as both a guest to L0 and a host to L2 guests and therefore
+    # potentially sees connections to two independent xenstore
+    # instances, one provided by the L0 host and one which is provided
+    # by the L1 instance of xenstore.
+    #
+    # Unfortunately the kernel is not capable of dealing with this and
+    # is only able to cope with a single xenstore connection. Since
+    # the L1 toolstack and L2 guests absolutely require xenstore to
+    # function we therefore cannot use the L0 xenstore and therefore
+    # cannot use PV devices (xvdX etc) in the L1 guest and must use
+    # emulated devices (sdX etc).
+    #
+    # However at the moment we have not yet rebooted L1 into Xen and
+    # so it does have PV devices available and sdb actually appears as
+    # xvdb.  We could disable the Xen platform device and use emulated
+    # devices for the install phase too but that would be needlessly
+    # slow.
+    # 
+    # Instead, to avoid needing to even mention the name of xvdb or
+    # sdb, we create the vg in l0.  When the l1 reboots it will
+    # automatically find the empty vg we have created for it, and
+    # target_choose_vg on l1 (which is used by all the guest creation
+    # ts-* scripts) will use it since it has plenty of space.
+
+    my $size = guest_var($l1,'gueststorage_size',$def_gueststorage_size);
+    die "gueststorage_size is undefined" unless $size;
+
+    my $outer_vg = target_choose_vg($l0, $size);
+    my $outer_lv = "$l1->{Ident}_gueststorage_outer_lv";
+    my $inner_vg = "$l1->{Ident}_gueststorage_vg";
+
+    target_cmd_root($l0, "vgremove -f $inner_vg ||:");
+    my $outer_lvdev = lv_create($l0, $outer_vg, $outer_lv, $size);
+
+    target_cmd_root($l0, <<END);
+        pvcreate $outer_lvdev
+	vgcreate $inner_vg $outer_lvdev
+END
+
+    toolstack($l0)->block_attach($l1, "$outer_lvdev,raw,sdb,rw");
+    # NB this does not update the l1 guest config so if the l1 is shut
+    # down and recreated in the l0, this will vanish.
+}
+
+packages();
+guest_storage();
+host_install_postboot_complete($l1);
+
+if ($dodefine) {
+    $l1_identspec =~ m/^(.*)=/ or die "$l1_identspec ?";
+    store_runvar($1, $'); #';
+}
-- 
1.7.10.4

Re: [OSSTEST PATCH 13/22] Nested HVM: Provide ts-nested-setup to help make L1 usable as a host

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> * Provide the L1 with some storage for its own guests' disks
> * Install some packages in the L1
> * Optionally, set a runvar defining the L1 for the rest of the job
> 
> The recipe is going to run ts-xen-install etc.
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 14/22] Nested HVM: Provide test-nested recipe

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: ts-nested-setup command line syntax updated.
v15: (Robert Ho) remove the unnecessary l1 destroy; as it will
     implicitly powered off by framework as a nested host.
     This change hasn't been confirmed by Ian Jackson yet; I
     plan to separate it as fix patch but by mistake squashed
     it in. Ian Jackson may want to revert this if he dosn't
     agree.
---
 sg-run-job |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 172c87e..a2527ca 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -348,6 +348,15 @@ proc run-job/test-pair-oneway {} {
     run-ts . =              ts-guest-stop          dst_host          + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+    run-ts . = ts-debian-hvm-install + host l1
+    run-ts . = ts-nested-setup + --define l1=host:l1
+    nested-layer-descend l1
+    run-ts . = ts-debian-hvm-install l1 l2
+    run-ts . = ts-guest-stop l1 l2
+}
+
 proc test-guest-migr {g} {
     set to_reap [spawn-ts . = ts-migrate-support-check + host $g 1]
     set can_migrate [reap-ts $to_reap]
-- 
1.7.10.4

Re: [OSSTEST PATCH 14/22] Nested HVM: Provide test-nested recipe

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 15/22] Nested HVM: Add test job to appropriate flights

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: Use default gueststorage_size, rather than setting runvar.
     Dropped acked from Ian Campbell.
---
 make-flight |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/make-flight b/make-flight
index 28820f5..3dfe5dc 100755
--- a/make-flight
+++ b/make-flight
@@ -249,6 +249,33 @@ do_hvm_win7_x64_tests () {
             all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  bios=$1
+
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+      -o "x$qemuu_suffix" != "x-qemuu" ]; then
+    return
+  fi
+
+  case $xenbranch in
+    xen-3.*-testing)      return;;
+    xen-4.0-testing)      return;;
+    xen-4.1-testing)      return;;
+    xen-4.2-testing)      return;;
+    xen-4.3-testing)      return;;
+  esac
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+            test-nested xl $xenarch $dom0arch $qemuu_runvar        \
+            l1_image=debian-7.2.0-amd64-CD-1.iso                   \
+            l1_vifmodel='e1000'                                    \
+            l1_memsize='3072'                                      \
+            l1_enable_nestedhvm='true'                             \
+            l2_image=debian-7.2.0-amd64-CD-1.iso                   \
+            bios=$bios                                             \
+            all_hostflags=$most_hostflags,hvm
+}
+
 branch_debianhvm_arch () {
   case $branch in
     xen-unstable-smoke) echo i386;;
@@ -592,6 +619,7 @@ test_matrix_do_one () {
     do_hvm_rhel6_tests
 
     do_hvm_debian_tests
+    do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
-- 
1.7.10.4

Re: [OSSTEST PATCH 15/22] Nested HVM: Add test job to appropriate flights

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

+            l1_image=debian-7.2.0-amd64-CD-1.iso                   \
> 
> [...]
> +            l2_image=debian-7.2.0-amd64-CD-1.iso                   \

We should remember to update "make-flight: Use Jessie ISO image for Debian
HVM tests" when this goes in.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 15/22] Nested HVM: Add test job to appropriate flights

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 15/22] Nested HVM: Add test job to appropriate flights"):
> On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> +            l1_image=debian-7.2.0-amd64-CD-1.iso                   \
> > 
> > [...]
> > +            l2_image=debian-7.2.0-amd64-CD-1.iso                   \
> 
> We should remember to update "make-flight: Use Jessie ISO image for Debian
> HVM tests" when this goes in.

That you need to ask this demonstrates that there is something wrong:
we have failed to break out this common setting.

I will fix this by inventing a shell function `debianhvm_usual_image'
in mfi-common.

Ian.

[OSSTEST PATCH 16/22] ts-xen-install: Properly handle hosts without a static IP address

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

Check IpStatic, and if it is not set, provide a dhcp stanza in
/etc/network/interfaces, rather than an `inet static' one.

This is necessary for L1 nested hosts, because they don't have a
static IP address.

In principle this makes matters more correct for physical hosts
without static IP addresses, but these are currently not supported
by selecthost().

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: Only use `inet dhcp' if !$ho->{IpStatic}.
---
 ts-xen-install |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ts-xen-install b/ts-xen-install
index b511e2b..d9aa694 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -320,11 +320,14 @@ END
                 /auto $iface\n/x;
             if (m/^\s* iface \s+ (?: $physif | xenbr0 ) \s+ inet \s /x) {
                 $suppress= 1;
-                print EO <<END;
+                print EO ($ho->{IpStatic} ? <<END : <<END), <<END;
 iface $iface inet static
     address $ho->{Ip}
     netmask $netmask
     gateway $gateway
+END
+iface $iface inet dhcp
+END
 $bridgex
 END
             }
-- 
1.7.10.4

Re: [OSSTEST PATCH 16/22] ts-xen-install: Properly handle hosts without a static IP address

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> Check IpStatic, and if it is not set, provide a dhcp stanza in
> /etc/network/interfaces, rather than an `inet static' one.
> 
> This is necessary for L1 nested hosts, because they don't have a
> static IP address.
> 
> In principle this makes matters more correct for physical hosts
> without static IP addresses, but these are currently not supported
> by selecthost().
> 
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 17/22] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

[Ian Jackson]

This function does not (now) always undo the DHCP configuration.
Sometimes it leaves it.  Its main function is to ensure that we have
a bridge for use by guests.

So rename the function.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v14: This patch was previously 4/4 of a miniature series containing
      a different way of dealing with the Nested HVM L1 DHCP problem.
---
 ts-xen-install |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ts-xen-install b/ts-xen-install
index d9aa694..3d0f394 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -247,7 +247,7 @@ sub hosts () {
     });
 }
 
-sub nodhcp () {
+sub ensurebridge () {
     target_editfile_root($ho, "/etc/network/interfaces",
                          "etc-network-interfaces", sub {
         my $physif= get_host_property($ho,'interface force',undef);
@@ -370,6 +370,6 @@ if ($checkmode) {
     adjustconfig();
     setupboot();
     setupinitd();
-    nodhcp();
+    ensurebridge();
     hosts();
 }
-- 
1.7.10.4

Re: [OSSTEST PATCH 17/22] ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> This function does not (now) always undo the DHCP configuration.
> Sometimes it leaves it.  Its main function is to ensure that we have
> a bridge for use by guests.
> 
> So rename the function.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 18/22] Osstest/Testsupport.pm: change target's default kernkind to 'pvops'

[Ian Jackson]

This is safe only if no existing flights would be affected.  (That is,
the meaning of no existing sets of runvars would be changed.)

To check whether this would make any difference I did some database
searches.  Since any time target_kernkind_check is called it sets a
corresponding `console' runvar, I can search for `console' without a
corresponding `kernkind'.  I ran this query:

  select * from (select *, (select name from runvars r2 where
  r2.flight=r1.flight and r2.job=r1.job and r2.name=
  replace(r1.name,'console','kernkind')) kk from runvars r1 where
  r1.name like '%console') iq where kk is null order by flight desc;

and it found nothing since flight 7682.  So I think we can change the
default.

Signed-off-by: 	Ian Jackson <ian.jackson@eu.citrix.com>
Signed-off-by: 	Robert Ho <robert.hu@intel.com>
Tested-by: 	Robert Ho <robert.hu@intel.com>
---
v15: New patch
---
 Osstest/TestSupport.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index b507ada..cb496d2 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -2065,7 +2065,7 @@ sub target_var ($$) {
 sub target_kernkind_check ($) {
     my ($gho) = @_;
     my $pfx= target_var_prefix($gho);
-    my $kernkind= $r{$pfx."kernkind"};
+    my $kernkind= $r{$pfx."kernkind"} // 'pvops';
     my $isguest= exists $gho->{Guest};
     if ($kernkind eq 'pvops') {
         store_runvar($pfx."rootdev", 'xvda') if $isguest;
-- 
1.7.10.4

Re: [OSSTEST PATCH 18/22] Osstest/Testsupport.pm: change target's default kernkind to 'pvops'

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> This is safe only if no existing flights would be affected.  (That is,
> the meaning of no existing sets of runvars would be changed.)
> 
> To check whether this would make any difference I did some database
> searches.  Since any time target_kernkind_check is called it sets a
> corresponding `console' runvar, I can search for `console' without a
> corresponding `kernkind'.  I ran this query:
> 
>   select * from (select *, (select name from runvars r2 where
>   r2.flight=r1.flight and r2.job=r1.job and r2.name=
>   replace(r1.name,'console','kernkind')) kk from runvars r1 where
>   r1.name like '%console') iq where kk is null order by flight desc;
> 
> and it found nothing since flight 7682.  So I think we can change the
> default.
> 
> Signed-off-by: 	Ian Jackson <ian.jackson@eu.citrix.com>
> Signed-off-by: 	Robert Ho <robert.hu@intel.com>
> Tested-by: 	Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 19/22] Osstest/Testsupport.pm: use get_target_property() for some host setup

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

For nested cases, nested host can inherit its host's property for
dhcp watch setup and ether_prefix property setup.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v15: New patch
---
 Osstest/TestSupport.pm |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index cb496d2..32f2956 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -735,7 +735,7 @@ sub lv_dev_mapper ($$) {
 sub dhcp_watch_setup ($$) {
     my ($ho,$gho) = @_;
 
-    my $meth = get_host_property($ho,'dhcp-watch-method',undef);
+    my $meth = get_target_property($ho,'dhcp-watch-method',undef);
     $gho->{DhcpWatch} = get_host_method_object($ho, 'DhcpWatch', $meth);
 }
 
@@ -1583,7 +1583,7 @@ sub target_choose_vg ($$) {
 
 sub ether_prefix($) {
     my ($ho) = @_;
-    my $prefix = get_host_property($ho, 'gen-ether-prefix-base');
+    my $prefix = get_target_property($ho, 'gen-ether-prefix-base');
     $prefix =~ m/^(\w+:\w+):(\w+):(\w+)$/ or die "$prefix ?";
     my $lhs = $1;
     my $pv = (hex($2)<<8) | (hex($3));
-- 
1.7.10.4

Re: [OSSTEST PATCH 19/22] Osstest/Testsupport.pm: use get_target_property() for some host setup

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> From: Robert Ho <robert.hu@intel.com>
> 
> For nested cases, nested host can inherit its host's property for
> dhcp watch setup and ether_prefix property setup.
> 
> Signed-off-by: Robert Ho <robert.hu@intel.com>
> Tested-by: Robert Ho <robert.hu@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 20/22] HVM guests: Use qemu "pipe:" for serial output logging

[Ian Jackson]

Modern qemu has the "pipe:/PATH" character driver.  This opens
/PATH.in for reading and /PATH.out for writing.  In my tests, I found
that:
  - contrary to the documentation, they do not need to be pipes
    (at least, /PATH.out can be a file)
  - but they must both already exist
  - qemu will follow symlinks, so /PATH.out can be a symlink to
    a file
  - if /PATH.in is a fifo, qemu will tolerate other processes opening
    it for writing, and writing things, only occasionally.  (Probably,
    qemu opens it O_RDWR; or perhaps it reopens it after EOF.)

Use this feature to achieve the following:
  - guest serial output ends up in /var/log/xen/osstest-serial-GUEST.log
    (which is already captured by ts-logs-capture) rather than
    interleaved with qemu's stderr output (in the libxl-created logfile)
  - guest serial input comes from a pipe in /root which we can open
    and write to if we want to talk to the guest

We are mostly interested in the final bullet point, because that will
allow us to send debug keys to the emulated serial port of an L1
nested HVM guest.

Looking at the source code of qemu in 4.2 and 4.6 I think the above
approach will work with all relevant qemu-xen's.

If the device model version is qemu-xen-traditional, `pipe:' is not
supported.  If device_model_version is not set, we will be using
whatever the xen.git we used defaults to.  For Xen 4.1 and earlier
that is qemu-xen-traditional, and I'm slightly loathe to break osstest
for those earlier versions.  There doesn't seem to be anything else in
the runvars that would clue us in.  So be cautious and do not use the
new feature unless device_model_version is explicitly set.

The nested tests are all -qemuu so set device_model_version.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v16: New patch
---
 Osstest/TestSupport.pm |   12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 32f2956..0a47b52 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1884,8 +1884,18 @@ END
 	# fd pointing to mini-os's console. IOW any such path used
 	# here ends up in the host logs in /var/log/xen/qemu-dm-$guest.log
 	$cfg .= "serial='file:/var/log/dm-serial.log'\n";
-    } else {
+    } elsif (!(defined $devmodel) || $devmodel =~ /traditional/) {
 	$cfg .= "serial='file:/dev/stderr'\n";
+    } else {
+	my $logpath = "/var/log/xen/osstest-serial-$gho->{Name}.log";
+	my $basepath = "/root/$flight.$job.$gho->{Name}.serial";
+	target_cmd_root($ho, <<END);
+            set -ex
+            test -e $logpath      || touch $logpath
+            test -e $basepath.out || ln -s $logpath $basepath.out
+            test -e $basepath.in  || mkfifo -m700 $basepath.in
+END
+	$cfg .= "serial='pipe:$basepath'\n";
     }
 
     $xopts{VifType} ||= "ioemu";
-- 
1.7.10.4

Re: [OSSTEST PATCH 20/22] HVM guests: Use qemu "pipe:" for serial output logging

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> Modern qemu has the "pipe:/PATH" character driver.  This opens
> /PATH.in for reading and /PATH.out for writing.

Coo...

> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 21/22] Serial: Factor out Osstest::Serial::keys_real

[Ian Jackson]

The sympathy and xenuse serial modules had too much in common.  Factor
out the common code, which is now responsible for
  - knowledge of the Xen console switch
  - splitting strings up into individual keys
  - timing decisions
  - error trapping and logging

This new class is an abstract base class for the concrete serial
method classes, and calls back to its derived class to prepare, send
each actual key, and shut down.

There is some functional change: notably, after failure to send the
first debug key, sending the remainder will not be attempted.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/Serial/keys_real.pm |   69 +++++++++++++++++++++++++++++++++++++++++++
 Osstest/Serial/sympathy.pm  |   55 +++++++++++-----------------------
 Osstest/Serial/xenuse.pm    |   51 ++++++++++----------------------
 3 files changed, 102 insertions(+), 73 deletions(-)
 create mode 100644 Osstest/Serial/keys_real.pm

diff --git a/Osstest/Serial/keys_real.pm b/Osstest/Serial/keys_real.pm
new file mode 100644
index 0000000..5c0cabf
--- /dev/null
+++ b/Osstest/Serial/keys_real.pm
@@ -0,0 +1,69 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::Serial::keys_real;
+
+# Base class providing debug keys for real serial ports.
+# Derived class is expected to provide:
+#   $mo->keys_prepare();
+#   $mo->keys_prepare($what,$str,$pause);
+#   $mo->keys_shutdown();
+
+
+use strict;
+use warnings;
+
+use Osstest::TestSupport;
+
+sub request_debug {
+    my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
+
+    if (!eval {
+	local ($SIG{'PIPE'}) = 'IGNORE';
+
+	$mo->keys_prepare();
+
+	$mo->keys_write('force attach', "\x05cf", 1); # ^E c f == force attach
+
+	my $debugkeys= sub {
+	    my ($what, $keys) = @_;
+	    foreach my $k (split //, $keys) {
+		$mo->keys_write("$what debug info request, debug key $k",
+				$k, 2);
+	    }
+	};
+
+	$mo->keys_write('request for input to Xen', $conswitch, 1);
+	$debugkeys->('Xen', $xenkeys);
+	sleep(10);
+	$debugkeys->('guest', $guestkeys);
+	sleep(10);
+	$mo->keys_write("RET to dom0","$conswitch\r", 5);
+
+	$mo->keys_write('dettach', "\x05c.", 1); # ^E c . == disconnect
+
+	$mo->keys_shutdown();
+
+	1;
+    }) {
+	warn "failed to send debug key(s): $@\n";
+	return 0;
+    }
+    return 1;
+}
+
+1;
diff --git a/Osstest/Serial/sympathy.pm b/Osstest/Serial/sympathy.pm
index d6bf425..fa34143 100644
--- a/Osstest/Serial/sympathy.pm
+++ b/Osstest/Serial/sympathy.pm
@@ -22,12 +22,13 @@ use warnings;
 
 use Osstest;
 use Osstest::TestSupport;
+use Osstest::Serial::keys_real;
 
 BEGIN {
     use Exporter ();
     our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
     $VERSION     = 1.00;
-    @ISA         = qw(Exporter);
+    @ISA         = qw(Exporter Osstest::Serial::keys_real);
     @EXPORT      = qw();
     %EXPORT_TAGS = ( );
 
@@ -57,48 +58,28 @@ sub new {
     return bless $mo, $class;
 }
 
-sub request_debug {
-    my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
+sub keys_prepare {
+}
 
-    my $targhost= $mo->{Server};
+sub keys_write {
+    my ($mo, $what,$str,$pause) = @_;
 
+    my $targhost= $mo->{Server};
     my ($sshopts) = sshopts();
-    my $sympwrite= sub {
-        my ($what,$str,$pause) = @_;
-        logm("sympathy sending $what");
-        if (!eval {
-            local ($SIG{'PIPE'}) = 'IGNORE';
-            my $sock= $mo->{Socket};
-            my $rcmd= "sympathy -c -k $sock -N >/dev/null";
-            $rcmd= "alarm 5 $rcmd";
-            open SYMPWRITE, "|ssh @$sshopts root\@$targhost '$rcmd'" or die $!;
-            autoflush SYMPWRITE 1;
-            print SYMPWRITE $str or die $!;
-            sleep($pause);
-            close SYMPWRITE or die "$? $!";
-            1;
-        }) {
-            warn "failed to send $what: $@\n";
-            return 0;
-        }
-        return 1;
-    };
 
-    my $debugkeys= sub {
-	my ($what, $keys) = @_;
-	foreach my $k (split //, $keys) {
-	    $sympwrite->("$what debug info request, debug key $k", $k, 2);
-	}
-    };
+    logm("sympathy sending $what");
 
-    $sympwrite->('request for input to Xen', $conswitch, 1);
-    $debugkeys->('Xen', $xenkeys);
-    sleep(10);
-    $debugkeys->('guest', $guestkeys);
-    sleep(10);
-    $sympwrite->("RET to dom0","$conswitch\r", 5);
+    my $sock= $mo->{Socket};
+    my $rcmd= "sympathy -c -k $sock -N >/dev/null";
+    $rcmd= "alarm 5 $rcmd";
+    open SYMPWRITE, "|ssh @$sshopts root\@$targhost '$rcmd'" or die $!;
+    autoflush SYMPWRITE 1;
+    print SYMPWRITE $str or die $!;
+    sleep($pause);
+    close SYMPWRITE or die "$? $!";
+}
 
-    return 1;
+sub keys_shutdown {
 }
 
 sub fetch_logs {
diff --git a/Osstest/Serial/xenuse.pm b/Osstest/Serial/xenuse.pm
index 6b9d955..7db2c28 100644
--- a/Osstest/Serial/xenuse.pm
+++ b/Osstest/Serial/xenuse.pm
@@ -23,6 +23,7 @@ use warnings;
 
 use Osstest;
 use Osstest::TestSupport;
+use Osstest::Serial::keys_real;
 
 use File::Temp;
 use File::Copy;
@@ -31,7 +32,7 @@ BEGIN {
     use Exporter ();
     our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
     $VERSION     = 1.00;
-    @ISA         = qw(Exporter);
+    @ISA         = qw(Exporter Osstest::Serial::keys_real);
     @EXPORT      = qw();
     %EXPORT_TAGS = ( );
 
@@ -46,51 +47,29 @@ sub new {
     return bless $mo, $class;
 }
 
-sub request_debug {
-    my ($mo,$conswitch,$xenkeys,$guestkeys) = @_;
-    my $xenuse= $c{XenUsePath} || "xenuse";
+sub keys_prepare {
+    my ($mo) = @_;
 
     my $ho= $mo->{Host};
 
-    my $writer= sub {
-        my ($what,$str,$pause) = @_;
-        logm("xenuse sending $what");
-        if (!eval {
-            print XENUSEWRITE $str or die $!;
-            sleep($pause);
-            1;
-        }) {
-            warn "failed to send $what: $@\n";
-            return 0;
-        }
-        return 1;
-    };
-
-    my $debugkeys= sub {
-	my ($what, $keys) = @_;
-	foreach my $k (split //, $keys) {
-	    $writer->("$what debug info request, debug key $k", $k, 2);
-	}
-    };
-
-    local ($SIG{'PIPE'}) = 'IGNORE';
+    my $xenuse= $c{XenUsePath} || "xenuse";
+
     open XENUSEWRITE, "|$xenuse -t $ho->{Name}" or die $!;
     autoflush XENUSEWRITE 1;
+}
 
-    $writer->('force attach', "\x05cf", 1); # ^E c f == force attach
+sub keys_write {
+    my ($mo, $what,$str,$pause) = @_;
+    logm("xenuse sending $what");
 
-    $writer->('request for input to Xen', $conswitch, 1);
-    $debugkeys->('Xen', $xenkeys);
-    sleep(10);
-    $debugkeys->('guest', $guestkeys);
-    sleep(10);
-    $writer->("RET to dom0","$conswitch\r", 5);
+    print XENUSEWRITE $str or die $!;
+    sleep($pause);
+}
 
-    $writer->('dettach', "\x05c.", 1); # ^E c . == disconnect
+sub keys_shutdown {
+    my ($mo) = @_;
 
     close XENUSEWRITE or die "$? $!";
-
-    return 1;
 }
 
 sub fetch_logs {
-- 
1.7.10.4

Re: [OSSTEST PATCH 21/22] Serial: Factor out Osstest::Serial::keys_real

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> The sympathy and xenuse serial modules had too much in common.  Factor
> out the common code, which is now responsible for
>   - knowledge of the Xen console switch
>   - splitting strings up into individual keys
>   - timing decisions
>   - error trapping and logging
> 
> This new class is an abstract base class for the concrete serial
> method classes, and calls back to its derived class to prepare, send
> each actual key, and shut down.
> 
> There is some functional change: notably, after failure to send the
> first debug key, sending the remainder will not be attempted.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 22/22] Serial: Add new serial method object for `guest' type

[Ian Jackson]

From: Robert Ho <robert.hu@intel.com>

L1 guests' serial ports are owned by qemu in L0.  We can send them
debug keys by writing to the qemu pipe.

(xl debug-key looks like it would be useful but it actually sends
debug keys to the hypervisor of the host it is running on.  We want to
send the debug keys to the hypervisor and kernel from the outside.)

Log fetching is not needed because from the POV of the L0 the L1 is a
guest, so the L0's log capture will already fetch the L1's serial
console output.

XXX THIS PATCH IS TOTALLY UNTESTED

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

---
v16: Mostly rewritten: now uses new keys_real base class, and
     uses the qemu pipe rather than xl debug-keys.
v15: New patch.
---
 Osstest/Serial/guest.pm |   93 +++++++++++++++++++++++++++++++++++++++++++++++
 Osstest/TestSupport.pm  |    2 +-
 2 files changed, 94 insertions(+), 1 deletion(-)
 create mode 100644 Osstest/Serial/guest.pm

diff --git a/Osstest/Serial/guest.pm b/Osstest/Serial/guest.pm
new file mode 100644
index 0000000..e2902ee
--- /dev/null
+++ b/Osstest/Serial/guest.pm
@@ -0,0 +1,93 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Send debug keys to nested host (L1).
+
+package Osstest::Serial::guest;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::Serial::keys_real;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter Osstest::Serial::keys_real);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $l1ho, $methname, @args) = @_;
+    my $mo = { Target => $l1ho, Parent => $l1ho->{Host} };
+
+    logm("serial method $methname $mo->{Target}{Name}: @args");
+    return bless $mo, $class;
+}
+
+sub keys_prepare {
+    my ($mo) = @_;
+
+    my $gho = $mo->{Target};
+    my $pho = $mo->{Parent};
+    my $puho = sshuho('root',$pho);
+
+    my $fifo = "/root/$flight.$job.$gho->{Name}.serial.in";
+
+    # NB this by-hand construction and execution of an
+    # ssh command line bypasses the usual timeout arrangements.
+    my ($sshopts) = sshopts();
+
+    # timeouts: open will carry on regardless even if the command hangs
+    open SERIALWRITE, "|ssh @$sshopts $puho cat >$fifo" or die $!;
+    autoflush SERIALWRITE 1;
+}
+
+sub keys_write {
+    my ($mo, $what,$str,$pause) = @_;
+    logm("xenuse sending $what");
+
+    # timeouts: we are going to write much less than any plausible
+    # PIPE_MAX so there is no risk that we will block on write
+    print SERIALWRITE $str or die $!;
+    sleep($pause);
+}
+
+sub keys_shutdown {
+    my ($mo) = @_;
+
+    # timeouts: close waits for the child to exit, so set an alarm
+    alarm(15);
+    $!=0; $?=0; close SERIALWRITE or die "$? $!";
+    alarm(0);
+}
+
+sub fetch_logs {
+    my ($mo);
+
+    logm("$mo->{Target}{Name} (nested host) serial console logs".
+	 " will be found in guest logs from $mo->{Parent}{Name} (parent)");
+    return;
+}
+
+1;
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 0a47b52..bfc5652 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -874,7 +874,7 @@ sub selecthost ($) {
 	$child->{Power} = 'guest';
 	power_cycle_host_setup($child);
 
-	$child->{Properties}{Serial} = 'noop'; # todo
+	$child->{Properties}{Serial} = 'guest';
 	serial_host_setup($child);
 
 	my $msg = "L$child->{NestingLevel} host $child->{Ident}:";
-- 
1.7.10.4

Re: [OSSTEST PATCH v16 00/22] Nested HVM

[Ian Campbell]

On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> 
> Ian: I'm not sure whether or not you want to review these.  About half
> of the patches are mine; the other half are Robert's but have been
> reviewed by me.  All but the last three have been tested by Robert.

I'll have a skim.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v16 00/22] Nested HVM

[Ian Campbell]

On Wed, 2015-11-11 at 09:45 +0000, Ian Campbell wrote:
> On Tue, 2015-11-10 at 19:53 +0000, Ian Jackson wrote:
> >  
> > Ian: I'm not sure whether or not you want to review these.  About half
> > of the patches are mine; the other half are Robert's but have been
> > reviewed by me.  All but the last three have been tested by Robert.
> 
> I'll have a skim.

It took us a long while to get the model and the semantics for nesting (e.g
. the host spec etc) down, but it was worth it, since it has all fallen out
quite naturally in this series.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel