[OSSTEST PATCH 0/8] Support diverse-CLASS hostflag

[OSSTEST PATCH 0/8] Support diverse-CLASS hostflag

[Ian Jackson]

Here is a quick series for osstest which provides support for a new
kind of hostflag diverse-CLASS.  The host allocator will not consider
any host for such an allocation, which has been allocated for any
other such allocation (with the same CLASS) in the same flight.

This is going to be useful because we want to run several identical
XTF test jobs but run each one on a different host (since there is
probably little value on running two jobs on the same host in the same
flight).

Some of the state has to be threaded through the planner daemon.
Without the planner daemon patches, the projection will be wrong,
slightly messing up planning estimates etc., but the actual
allocations will, I think, be right.

It is fine to run the new host allocator against an old planner daemon
(even a production one), at least for testing.


These patches can be found in my personal xenbits repo
  xenbits.xen.org:/home/iwj/ext/osstest.git
  base.diverse-class.v1 .. wip.diverse-class.v1


These three

 [OSSTEST PATCH 1/8] Tcl: Use tclsh8.5
 [OSSTEST PATCH 2/8] ms-flights-summary: Remove spurious \ in keys \%{
 [OSSTEST PATCH 3/8] mg-schema-test-database: Direct logs to local

are preparatory patches which are probably going to be helpful if
someone wants to test the new planner (as I did, using the test
database facility).

Then there is the meat:

 [OSSTEST PATCH 4/8] ms-planner: Support ClientNotes
 [OSSTEST PATCH 5/8] ts-hosts-allocate-Executive: Replace some odd
 [OSSTEST PATCH 6/8] ts-hosts-allocate-Executive: pass $plan to
 [OSSTEST PATCH 7/8] ts-hosts-allocate-Executive: Support

And one helpful test script:

 [OSSTEST PATCH 8/8] DO NOT APPLY make-flight-diverse-test: test case


Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 1/8] Tcl: Use tclsh8.5

[Ian Jackson]

I have checked that tclsh8.5 and TclX work on osstest.test-lab (and
also osstest.xs.citrite.net).  TclX seems to be provided by tcl8.4 but
work with tcl8.5 (at least on wheezy and jessie).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 README            | 2 +-
 ms-ownerdaemon    | 2 +-
 ms-queuedaemon    | 2 +-
 ms-reportuptime   | 2 +-
 sg-execute-flight | 2 +-
 sg-run-job        | 2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/README b/README
index 0a346dc..3aa35d2 100644
--- a/README
+++ b/README
@@ -246,7 +246,7 @@ To run osstest in standalone mode:
 
  - You need to install
      sqlite3
-     tcl8.4 tclx8.4 libsqlite3-tcl
+     tcl8.5 tclx8.4 libsqlite3-tcl
      libdbi-perl libdbd-sqlite3-perl
      pax rsync
      curl
diff --git a/ms-ownerdaemon b/ms-ownerdaemon
index 33ee238..5b52339 100755
--- a/ms-ownerdaemon
+++ b/ms-ownerdaemon
@@ -1,4 +1,4 @@
-#!/usr/bin/tclsh8.4
+#!/usr/bin/tclsh8.5
 # -*- Tcl -*- 
 # usage: ./ms-ownerdaemon  ... | logger
 
diff --git a/ms-queuedaemon b/ms-queuedaemon
index 2b8d621..8affacc 100755
--- a/ms-queuedaemon
+++ b/ms-queuedaemon
@@ -1,4 +1,4 @@
-#!/usr/bin/tclsh8.4
+#!/usr/bin/tclsh8.5
 # -*- Tcl -*- 
 # usage: ./ms-queuedaemon  ... | logger
 
diff --git a/ms-reportuptime b/ms-reportuptime
index aeac483..fe72b78 100755
--- a/ms-reportuptime
+++ b/ms-reportuptime
@@ -1,4 +1,4 @@
-#!/usr/bin/tclsh8.4
+#!/usr/bin/tclsh8.5
 # -*- Tcl -*- 
 # usage: ./ms-reportuptime
 
diff --git a/sg-execute-flight b/sg-execute-flight
index 4e3fcf2..27fc04c 100755
--- a/sg-execute-flight
+++ b/sg-execute-flight
@@ -1,4 +1,4 @@
-#!/usr/bin/tclsh8.4
+#!/usr/bin/tclsh8.5
 # -*- Tcl -*- 
 # usage: ./sg-execute-flight FLIGHT BLESSING
 
diff --git a/sg-run-job b/sg-run-job
index 8b2d5e1..920834a 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -1,4 +1,4 @@
-#!/usr/bin/tclsh8.4
+#!/usr/bin/tclsh8.5
 
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/8] ms-flights-summary: Remove spurious \ in keys \%{ something }

[Ian Jackson]

With some versions of Perl this generates a warning which causes
ms-flights-summary to fail.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 ms-flights-summary | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ms-flights-summary b/ms-flights-summary
index ea72088..c703d81 100755
--- a/ms-flights-summary
+++ b/ms-flights-summary
@@ -359,7 +359,7 @@ printf("<p>%d flight(s) consisting of %s job(s)<br />%s<br />%s anonymous/rogue
 
 my %summarycounts;
 foreach my $fi (values %flights) {
-    $summarycounts{$_} += $fi->{Stats}{$_} foreach (keys \%{ $fi->{Stats} });
+    $summarycounts{$_} += $fi->{Stats}{$_} foreach (keys %{ $fi->{Stats} });
 }
 my @summarycounts = sort_stats \%summarycounts;
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 3/8] mg-schema-test-database: Direct logs to local directory

[Ian Jackson]

Do not pollute a shared log area with logs of flights whose numbers
are valid only in the context of our test database.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 mg-schema-test-database | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index 4e0ee68..88a75cf 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -338,6 +338,14 @@ QueueDaemonPort ${ctrlports#*,}
 ExecutiveDbOwningRoleRegexp .*
 QueueDaemonHoldoff 3
 QueueDaemonRetry 5
+Logs $PWD/logs
+Stash $PWD/logs
+Results $PWD/results
+Publish=undef
+LogsPublish=undef
+ResultsPublish=undef
+HarnessPublishGitUserHost=undef
+HarnessPublishGitRepoDir=undef
 END
 	mv -f $tcfg.tmp $tcfg
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 4/8] ms-planner: Support ClientNotes

[Ian Jackson]

No users yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 README.planner | 22 ++++++++++++++++++++++
 ms-planner     | 23 +++++++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/README.planner b/README.planner
index daccef5..80c2506 100644
--- a/README.planner
+++ b/README.planner
@@ -286,6 +286,13 @@ Plan is:
 						Live
 			Job	flight.job which made the allocation
 
+	ClientNotes
+		map from note key
+		to one of
+			scalar
+			array of values supplied by clients
+			hash of keys/values supplied by clients
+
 	* = internal to plan
 	** = computed by launder_check_plan
 	+ = as shown to clients
@@ -302,6 +309,12 @@ Booking list is:
 			Share		optional struct containing
 						Type
 						Shares
+	[ClientNotes]
+		as for Plan; merged by
+		for scalar, updating from Booking if specified
+		for array, appending Booking to Plan
+		for hash, updating individual values from Booking
+		type must not have changed!
 
 Sharing resources:
 
@@ -313,3 +326,12 @@ Bookings which do not create the share do not mention the master.
 
 Note that whether a resource is free, or simply nonexistent, is not
 represented.
+
+
+Note keys
+---------
+
+Note key name (key in ClientNotes):
+	Note value format:
+
+(none yet defined)
diff --git a/ms-planner b/ms-planner
index abf5fac..bce6e13 100755
--- a/ms-planner
+++ b/ms-planner
@@ -245,6 +245,7 @@ sub cmd_reset () {
     $plan->{Start}= time;
     $plan->{Events}= { };
     $plan->{Unprocessed}= [ ];
+    $plan->{ClientNotes}= { };
 
     my %magictask;
     foreach my $taskrefkey (qw(preparing shared)) {
@@ -415,6 +416,9 @@ sub cmd_get_plan () {
 	}
 	$jplan->{Events}{$reso}= \@jevts;
     }
+
+    $jplan->{ClientNotes} = $plan->{ClientNotes};
+
     print to_json($jplan),"\n" or die $!;
 }
 
@@ -529,6 +533,25 @@ sub cmd_book_resources () {
 		  );
     }
 
+    my $jnotes = $jbookings->{ClientNotes} // { };
+    foreach my $k (sort keys %$jnotes) {
+	my $v = $jnotes->{$k};
+	my $newt = ref $v // '(none)';
+	if (exists $plan->{ClientNotes}{$k}) {
+	    my $oldt = ref $plan->{ClientNotes}{$k} // '(none)';
+	    die "$k $oldt -> $newt" unless $oldt eq $newt;
+	}
+	if (!ref $v) {
+	    $plan->{ClientNotes}{$k} = $v;
+	} elsif (ref $v eq 'HASH') {
+	    $plan->{ClientNotes}{$k}{$_} = $v->{$_} foreach keys %$v;
+	} elsif (ref $v eq 'ARRAY') {
+	    push @{ $plan->{ClientNotes}{$_} }, @{ $jnotes->{$_} };
+	} else {
+	    die "$k $newt";
+	}
+    }
+
     check_write_new_plan();
 }
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 5/8] ts-hosts-allocate-Executive: Replace some odd commas with semicolons

[Ian Jackson]

There does not seem to be any reason for this.  I think it must be a
leftover from a previous use of { A => X, B => Y, ... } syntax.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 ts-hosts-allocate-Executive | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 13a0652..69f1cf6 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -709,8 +709,8 @@ sub compute_booking_list () {
 	$book->{Job}= "$flight.$job";
 	$book->{Reso}= "$sel->{restype} $sel->{resname}";
 	$book->{Xinfo}= $hid->{Ident};
-	$book->{Start}= $best->{Start},
-	$book->{End}= $best->{Start} + $best->{Duration},
+	$book->{Start}= $best->{Start};
+	$book->{End}= $best->{Start} + $best->{Duration};
 	push @bookings, $book;
     }
     return { Bookings => \@bookings };
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 6/8] ts-hosts-allocate-Executive: pass $plan to hid_find_possibilities

[Ian Jackson]

We are going to want this to implement the new diverse-CLASS hostflag.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 ts-hosts-allocate-Executive | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 69f1cf6..d4a96d5 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -272,8 +272,8 @@ sub compute_hids () {
     }
 }
 
-sub hid_find_possibilities ($) {
-    my ($hid) = @_;
+sub hid_find_possibilities ($$) {
+    my ($hid, $plan) = @_;
 
     delete $hid->{Candidates};
 
@@ -647,7 +647,7 @@ sub attempt_allocation {
 
     foreach my $hid (@hids) {
 	delete $hid->{Allocated};
-	hid_find_possibilities($hid);
+	hid_find_possibilities($hid, $plan);
     }
     optimally_order_hids();
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 7/8] ts-hosts-allocate-Executive: Support diverse-CLASS hostflag

[Ian Jackson]

Specifically:
 * Parse it out of the hostflags when constructing the hid
 * Look for the `hostalloc-diverse-FLIGHT-CLASS' ClientNote in
   the resource plan, to avoid inappropriately planning to reuse hosts.
 * Look for the `diversehosts_CLASS' runvar in other jobs in this flight,
   to find out who might have allocated with the same CLASS.  (This
   sort of duplicates information in *hostflags and *host, but digging
   the information out of the latter two would be very tiresome.)
 * Check each of the above for each candidate host.
 * Set the ClientNote when we are preparing a booking.
 * Set the runvar when we do the allocation.
 * Document the ClientNote.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 README.planner              |  3 +++
 ts-hosts-allocate-Executive | 59 +++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 60 insertions(+), 2 deletions(-)

diff --git a/README.planner b/README.planner
index 80c2506..bb18432 100644
--- a/README.planner
+++ b/README.planner
@@ -335,3 +335,6 @@ Note key name (key in ClientNotes):
 	Note value format:
 
 (none yet defined)
+
+hostalloc-diverse-<flight>-<class>
+	{ <hostname> => 'info for debug', ... }
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index d4a96d5..72283de 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -213,7 +213,7 @@ sub compute_hids () {
     our %equivs;
 
     foreach my $ident (@ARGV) {
-        my $hid= { Conds => { host => [] } };
+        my $hid= { Conds => { host => [] }, Diverse => [] };
         my $override_use;
         if ($ident =~ m/\=/) {
             $hid->{OverrideUse}= $'; #'
@@ -251,6 +251,11 @@ sub compute_hids () {
                 my $equiv= $hid->{Equiv}= $equivs{$formalclass};
                 print DEBUG "HID $ident FLAG $flag EQUIV $equiv->{Wanted}\n";
                 next;
+            } elsif ($flag =~ m/^diverse-/) {
+                my $dclass= $'; #'
+		push @{ $hid->{Diverse} }, $dclass;
+		print DEBUG "HID $ident FLAG $flag DIVERSE $dclass\n";
+                next;
             } elsif ($flag =~ m/^\w+:/) {
 		my (@c) = split /:/, $flag;
 		my $o;
@@ -322,9 +327,40 @@ END
 	     LIMIT 1;
 END
 
+    # we are looking for other jobs where the runvar diversehosts_CLASS
+    # (a space-separated list of hostnames) has been set
+    my $diverseq = $dbh_tests->prepare(<<END);
+            SELECT r.job, r.val
+              FROM runvars r
+             WHERE r.flight = ?
+               AND r.name = ?
+END
+
     my @candidates;
     my $any=0;
 
+    my %diverse_excluded;
+    # $diverse_excluded{CLASS}{HOST} = 1
+    foreach my $dclass (@{ $hid->{Diverse} }) {
+	print DEBUG "HID $hid->{Ident} DIVERSE $dclass:";
+	# look for allocated
+	$diverseq->execute($flight, "diversehosts_$dclass");
+	my ($djob, $dhosts) = $diverseq->fetchrow_array();
+	my $excl = sub {
+	    my ($dh,$why) = @_;
+	    print DEBUG " $dh ($why)";
+	    $diverse_excluded{$dclass}{$dh} = $why;
+	};
+	if (defined $dhosts) {
+	    $excl->($_, "alloc'd job $job") foreach split / /, $dhosts;
+	}
+	# look for booked
+	my $notekey = "hostalloc-diverse-$flight-$dclass";
+	my $booked = $plan->{ClientNotes}{$notekey};
+	$excl->($_, "boooked ".$booked->{$_}) foreach sort keys %$booked;
+	print DEBUG ".\n";
+    }
+
   CANDIDATE:
     while (my $candrow= $findhostsq->fetchrow_hashref()) {
         $candrow->{Warnings}= [ ];
@@ -375,6 +411,13 @@ END
                 "specified host lacks flags @missingflags";
         }
 
+	foreach my $dclass (@{ $hid->{Diverse} }) {
+	    my $excl = $diverse_excluded{$dclass}{ $candrow->{resname} };
+	    next unless defined $excl;
+	    print DEBUG "$dbg excluded by diverse-$dclass ($excl)\n";
+	    next CANDIDATE unless defined $use;
+	}
+
 	foreach my $c (@{ $hid->{Conds}{$candrow->{restype}} }) {
 	    if (!$c->check($candrow->{restype}, $candrow->{resname})) {
 		print DEBUG "$dbg failed $c condition\n";
@@ -629,6 +672,13 @@ sub alloc_hosts () {
         my $use= $r{ $hid->{Ident} };
         next if defined $use;
         store_runvar($hid->{Ident}, $sel->{resname});
+	foreach my $dclass (@{ $hid->{Diverse} }) {
+	    my $drk = "diversehosts_$dclass";
+	    my %dused;
+	    $dused{$_} = 1 foreach split / /, ($r{$drk} // '');
+	    $dused{$sel->{resname}} = 1;
+	    store_runvar($drk, join ' ', sort keys %dused);
+	}
     }
 
     logm("host allocation: all successful and recorded.");
@@ -697,6 +747,7 @@ sub attempt_allocation {
 
 sub compute_booking_list () {
     my @bookings;
+    my %cnotes;
     foreach my $hid (@hids) {
 	my $sel= $hid->{Selected};
 	my $alloc= $hid->{Allocated};
@@ -712,8 +763,12 @@ sub compute_booking_list () {
 	$book->{Start}= $best->{Start};
 	$book->{End}= $best->{Start} + $best->{Duration};
 	push @bookings, $book;
+	foreach my $dclass (@{ $hid->{Diverse} }) {
+	    my $notekey = "hostalloc-diverse-$flight-$dclass";
+	    $cnotes{$notekey}{$sel->{resname}} = "$job $hid->{Ident}";
+	}
     }
-    return { Bookings => \@bookings };
+    return { Bookings => \@bookings, ClientNotes => \%cnotes };
 }
 
 #---------- actually allocate things ----------
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 8/8] DO NOT APPLY make-flight-diverse-test: test case for diverse-CLASS hostflag

[Ian Jackson]

This generates a flight with two jobs, test-1 and test-2.  They can be
run, and are supposed to allocate different hosts, and will bomb out
after host allocation.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
 make-flight-diverse-test | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 make-flight-diverse-test

diff --git a/make-flight-diverse-test b/make-flight-diverse-test
new file mode 100755
index 0000000..dffd23b
--- /dev/null
+++ b/make-flight-diverse-test
@@ -0,0 +1,11 @@
+#!/bin/sh
+set -e
+
+flight=`./cs-flight-create play xen-unstable`
+
+for j in test-1 test-2; do
+    ./cs-job-create $flight $j test-debian \
+        all_hostflags="blessed-adhoc,diverse-d"
+done
+
+echo $flight
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel