[OSSTEST PATCH 1/6] cr-ensure-disk-space: Do not take out the lock in dry run mode

[OSSTEST PATCH 1/6] cr-ensure-disk-space: Do not take out the lock in dry run mode

[Ian Jackson]

This makes testing a bit easier, and is of course fine.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index ff6b01b..bfdbcc5 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -72,9 +72,11 @@ check_space if check_space;
 
 print "taking lock...";
 
-my $lock = "$c{GlobalLockDir}/publish-lock";
-open LOCK, "> $lock" or die "$lock $!";
-flock LOCK, LOCK_EX or die $!;
+if (!$dryrun) {
+    my $lock = "$c{GlobalLockDir}/publish-lock";
+    open LOCK, "> $lock" or die "$lock $!";
+    flock LOCK, LOCK_EX or die $!;
+}
 
 print "proceeding...\n";
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/6] cr-ensure-disk-space: -F option

[Ian Jackson]

May be repeated (cuddled with itself) or given a number.  Forces
deletion, even if there is enough space.  Normally clean up one less
flight than specified, since cr-ensure-disk-space reruns its check
after acquiring the lock.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index bfdbcc5..c65423a 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -26,6 +26,7 @@ use Osstest::Management qw(:logs);
 use Fcntl qw(:flock);
 
 our $dryrun= 0;
+our $force;
 
 open DEBUG, ">/dev/null" or die $!;
 
@@ -34,6 +35,10 @@ while (@ARGV && $ARGV[0] =~ m/^\-/) {
     last if $_ eq '--';
     if (m/^-n$/) {
         $dryrun= 1;
+    } elsif (m/^-(F+)$/) {
+        $force += length $1;
+    } elsif (m/^-F(\d+)$/) {
+        $force += $1;
     } elsif (m/^-D$/) {
         open DEBUG, ">&2" or die $!;
     } else {
@@ -58,6 +63,11 @@ sub check_space () {
     $!=0; $?=0; close P or die "$! $?";
     my $space= $1;
     printf "space: %8d, wanted: %8d ", $space, logcfg('MinSpaceMby');
+    if ($force) {
+	$force--;
+	printf "FORCING ";
+	return 0;
+    }
     return $space >= logcfg('MinSpaceMby');
 }
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 3/6] cr-ensure-disk-space: Dry run produces better `die' message.

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index c65423a..83372ca 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -153,7 +153,7 @@ sub iteration_proceed () {
 
     printf "...";
 
-    die if $dryrun;
+    die "dry run" if $dryrun;
 
     my $spawn= sub {
         $!=0; my $r= system @_; die "@_ $r $!" if $r;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 4/6] mg-allocate: Tiny refactoring

[Ian Jackson]

Break out $shareix assignment from $4.  (We are going to want to put
some code just after this point which will want to do regexp matching,
which would trash $4.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mg-allocate | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mg-allocate b/mg-allocate
index 9b66114..0317229 100755
--- a/mg-allocate
+++ b/mg-allocate
@@ -111,7 +111,8 @@ sub parse_1res ($) {
     my $restype= defined($2) ? $2 : 'host';
     $restype= 'share-host' if $restype eq 'S';
     my $resname= $3;
-    my $shareix= defined($4) ? $4+0 : '*';
+    my $shareix= $4;
+    $shareix= defined($shareix) ? $shareix+0 : '*';
     my $shareixcond = $shareix eq '*' ? '' : "AND shareix = $shareix";
 
     my @resnames;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 5/6] flight preservation: Honour flight allocation during expiry

[Ian Jackson]

Look in the resources and tasks table for a resources table entry
corresponding to each flight, owned by a live task.  Such flights are
not deleted.

Specifically:
 * At the start, we get a list of all the preserved flights, and
   also print the information to stdout.
 * Whenever we compare flight numbers for inequality (as a proxy
   for flight age), we first compare where the flights are allocated.
   (When there are references, the effect is that an allocated
   referring flight counts as very late, so $latestref will contain it.)
 * Before actually deleting the selected flight we check it's not
   allocated.  (Strictly, we check it's "latest" reference is not
   allocated.)

Currently there is nothing which creates such resources table entries
so there is no overall functional change.  Also, as a result, the doc
reads rather oddly.  This will be fixed in the next patch.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.planner       | 11 +++++++++++
 cr-ensure-disk-space | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 61 insertions(+), 2 deletions(-)

diff --git a/README.planner b/README.planner
index bb18432..9d35b40 100644
--- a/README.planner
+++ b/README.planner
@@ -199,6 +199,17 @@ that shared systems get regrooved occasionally even if nothing decides
 to unshare them.
 
 
+Flights
+-------
+
+Flight logs and build artefacts are normally expired based on their
+start time, or the start time of the most recent flight which refers
+to them.
+
+Flights can be protected (preserved) by allocating them somehow:
+Flights are represented by restype='share-flight' entries in the
+resources table.
+
 
 DETAILED PROTOCOL NOTES
 =======================
diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 83372ca..7846580 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -100,9 +100,45 @@ my $refq= db_prepare(<<END);
        AND flight >= ?
 END
 
+my $allocqtxt = <<END;
+    SELECT resname AS flight,
+           shareix,
+           (tasks. taskid  ||  ' ' ||
+            tasks.type     ||  ' ' ||
+            tasks.refkey   || ': ' ||
+   COALESCE(tasks.username,'[no username]') ||  ' ' ||
+   COALESCE(tasks.comment, '[no comment]')
+           ) AS info
+      FROM resources
+      JOIN tasks
+        ON owntaskid = taskid
+     WHERE live
+       AND owntaskid != (SELECT taskid FROM tasks
+                         WHERE type='magic' AND refkey='allocatable')
+       AND restype='share-flight'
+END
+
+my $allocq = db_prepare(<<END);
+$allocqtxt
+       AND resname ::int >= ?
+END
+
+my $allocchkq = db_prepare(<<END);
+$allocqtxt
+       AND resname = ?
+     LIMIT 1
+END
+
 our @flights;
+our %allocd;
 our %latestref;
 
+sub flight_num_cmp ($$) {
+    my ($x,$y) = @_;
+    !!$allocd{$x} <=> !!$allocd{$y} or
+            $x    <=>           $y;
+}
+
 sub iteration_proceed () {
     if (!@flights) {
 	%latestref = ();
@@ -119,20 +155,27 @@ sub iteration_proceed () {
 
 	print DEBUG "MINFLIGHT $minflight\n";
 
+	%allocd = ();
+	$allocq->execute($minflight);
+	while (my $ar= $allocq->fetchrow_hashref) {
+	    print DEBUG "preserving allocated $ar->{flight}: $ar->{info}\n";
+	    push @{ $allocd{ $ar->{flight} } }, $ar->{info};
+	}
+
 	$refq->execute($minflight);
 	while (my $rr= $refq->fetchrow_hashref) {
 	    my $testflight = $rr->{flight};
 	    next unless $rr->{val} =~ m/^(\d+)\./;
 	    my $buildflight = $1;
 	    next unless exists $latestref{$buildflight};
-	    if ($testflight > $latestref{$buildflight}) {
+	    if (flight_num_cmp($testflight, $latestref{$buildflight}) > 0) {
 		print DEBUG "REF $buildflight <- $testflight\n";
 		$latestref{$buildflight} = $testflight;
 	    }
 	}
 
 	@flights =
-	    sort { $latestref{$b} <=> $latestref{$a} }
+	    sort { flight_num_cmp($latestref{$b}, $latestref{$a}) }
 	    keys %latestref;
         printf "(%d flights) ", scalar @flights;
         die unless @flights;
@@ -151,6 +194,11 @@ sub iteration_proceed () {
 
     die "age $age" if $age < logcfg('MinExpireAge');
 
+    $allocchkq->execute($latestref);
+    my $arow= $allocchkq->fetchrow_hashref();
+    $allocchkq->finish();
+    die "alloc $arow->{info} !" if $arow;
+
     printf "...";
 
     die "dry run" if $dryrun;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 6/6] flight preservation: Provide a way to allocate flights

[Ian Jackson]

Teach mg-allocate to create and delete resources table entries for
flights, as necessary.  And, provide a conveniance alias F/<flight>.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.planner | 10 ++++++++--
 mg-allocate    | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/README.planner b/README.planner
index 9d35b40..f3cab53 100644
--- a/README.planner
+++ b/README.planner
@@ -206,9 +206,15 @@ Flight logs and build artefacts are normally expired based on their
 start time, or the start time of the most recent flight which refers
 to them.
 
-Flights can be protected (preserved) by allocating them somehow:
+Flights can be protected (preserved) by allocating them with
+./mg-allocate F/<flight>.
+
 Flights are represented by restype='share-flight' entries in the
-resources table.
+resources table.  Conventionally, the shareix is the owning taskid.
+This allows multiple tasks to lock a single flight.  There is no
+corresponding entry with restype='flight', nor a resource_sharing
+entry.  mg-allocate will create and clean up share-flight entries as
+needed.
 
 
 DETAILED PROTOCOL NOTES
diff --git a/mg-allocate b/mg-allocate
index 0317229..ef57bb8 100755
--- a/mg-allocate
+++ b/mg-allocate
@@ -6,6 +6,7 @@
 # <resource-spec> syntax:
 #   [!][<type>/]<name>[/<share>]      type defaults to 'host'
 #                                     type=='S' means 'shared-host'
+#                                     type=='F' means 'shared-flight'
 #                                     share defaults to *
 #                                     "!" prefix means deallocate
 #                                     name=option|option|... means
@@ -112,6 +113,17 @@ sub parse_1res ($) {
     $restype= 'share-host' if $restype eq 'S';
     my $resname= $3;
     my $shareix= $4;
+
+    if ($restype eq 'F') {
+	die unless $resname =~ m/^\d+$/;
+	die unless $resname eq $resname+0;
+	die if defined $shareix;
+	die if $donate_spec;
+	die if @steal_specs;
+	$restype = 'share-flight';
+	$shareix = $tid;
+    }
+
     $shareix= defined($shareix) ? $shareix+0 : '*';
     my $shareixcond = $shareix eq '*' ? '' : "AND shareix = $shareix";
 
@@ -147,6 +159,23 @@ sub alloc_1rescand ($$) {
     my ($res, $rescand) = @_;
     my ($allocate, $restype, $resname, $shareix, $shareixcond) = @$rescand;
 
+    if ($allocate && $restype eq 'share-flight' && $shareix == $tid) {
+	$dbh_tests->do(<<END,{},
+            INSERT INTO resources
+                 (SELECT ? AS restype,
+                         ? AS resname,
+                         ? AS shareix,
+                         ? AS owntaskid
+                   WHERE NOT EXISTS
+                    (SELECT 1 FROM resources
+                         WHERE restype=?
+                           AND resname=?
+                           AND shareix=?))
+END
+		       $restype,$resname,$shareix, $magictask{idle},
+		       $restype,$resname,$shareix);
+    }
+
     my $resq= $dbh_tests->prepare(<<END);
                 SELECT * FROM resources r
                          JOIN tasks t
@@ -257,6 +286,16 @@ END
                 logm("$desc: freeing");
             }
 	    $setres->($donate_taskid // $magictask{idle});
+	    if ($restype eq 'share-flight' && $shareix == $tid) {
+		$dbh_tests->do(<<END,{},
+                    DELETE FROM resources
+                          WHERE restype = ?
+                            AND resname = ?
+                            AND shareix = ?
+                            AND owntaskid = ?
+END
+			       $restype,$resname,$shareix, $magictask{idle});
+	    }
         }
 
         if ($isshared) {
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel