[OSSTEST PATCH 0/3] test linux-next

[OSSTEST PATCH 0/3] test linux-next

[Ian Jackson]

We need a different approach to selecting a baseline for testing
linux-next.  I have tested this series and it appeared to work - the
results can be found in this email on xen-devel:

 Subject: [linux-next test] 25862: tolerable FAIL
 Date: Sat, 12 Apr 2014 00:28:06 +0100

(The test results were a complete disaster because the linux tip on
which this linux-next was based cannot boot under Xen.  See flight
25840.  I.e. the failures are a bug in Linux, not in the tester.)

 1/3 sg-check-tested: add "use strict"
 2/3 sg-check-tested: support --git-ancestor-of
 3/3 cr-*, ap-*: Facilities for testing linux-next

[OSSTEST PATCH 1/3] sg-check-tested: add "use strict"

[Ian Jackson]

This revealed no bugs so no other changes needed.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-check-tested |    1 +
 1 file changed, 1 insertion(+)

diff --git a/sg-check-tested b/sg-check-tested
index 206447a..9e78448 100755
--- a/sg-check-tested
+++ b/sg-check-tested
@@ -16,6 +16,7 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+use strict;
 
 use DBI;
 use Osstest;
-- 
1.7.10.4

[OSSTEST PATCH 2/3] sg-check-tested: support --git-ancestor-of

[Ian Jackson]

This will be used for the linux-next branch, whose baseline is the
most recently tested linux-linus which is an ancestor of the
being-tested linux-next.

We do not provide this for any VCS other than git.  Hopefully we will
never need to.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-check-tested |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/sg-check-tested b/sg-check-tested
index 9e78448..8cdc8a9 100755
--- a/sg-check-tested
+++ b/sg-check-tested
@@ -20,10 +20,13 @@ use strict;
 
 use DBI;
 use Osstest;
+use IO::Handle;
 
 our @conds;
 our @conds_vars;
 our $prrev;
+our $ancestorof;
+our %ancestors;
 
 open DEBUG, ">/dev/null" or die $!;
 
@@ -84,6 +87,16 @@ END
     } elsif (m/^--print-revision=(.*)$/) {
         die if $prrev;
         $prrev= $1;
+    } elsif (m/^--git-ancestor-of=(.*)$/) {
+	# set GIT_DIR appropriately when using this
+	$ancestorof=$1;
+	open GA, "git rev-list $ancestorof |" or die $!;
+	while (<GA>) {
+	    chomp or die;
+	    $ancestors{$_} = 1;
+	}
+	GA->error and die $!;
+	$!=$?=0; close GA or die "$? $!";
     } elsif (m/^--debug$/) {
         open DEBUG, ">&2" or die $!;
     } else {
@@ -93,6 +106,7 @@ END
 
 die if @ARGV;
 die unless @conds;
+die if defined($ancestorof) && !$prrev;
 
 csreadconfig();
 
@@ -162,6 +176,12 @@ END
 	    print DEBUG "also $val2\n";
 	    next FLIGHT;
 	}
+	if (defined $ancestorof) {
+	    if (!$ancestors{$val1}) {
+		print DEBUG "not an ancestor of $ancestorof\n";
+		next FLIGHT;
+	    }
+	}
         print "$val1\n";
         exit 0;
     }
-- 
1.7.10.4

[OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next

[Ian Jackson]

* Define the new branch's git source repo in ap-common.

* Provide a branch-settings file which disables running additional
  baseline tests, listing changesets in flight reports, pushing,
  and which arranges for the baseline revision for the flight
  report to be determined "later", ie after the to-be-tested
  versions of everything have been decided.

* When the baseline revision is to be determined "later", run
  the new script ap-fetch-version-baseline-later.  This is another
  collection of per-branch rules.  We introduce currently one
  per-branch rule, for linux-next:

* The baseline version for a test of linux-next is
    - the version tested by the most recent flight on the
      linux-linus branch;
    - provided that said version is an ancestor of the linux-next
      version under test;
    - otherwise we try an earlier linux-linus flight.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ap-common                      |    5 +++++
 ap-fetch-version-baseline-late |   44 ++++++++++++++++++++++++++++++++++++++++
 branch-settings.linux-next     |    4 ++++
 cr-daily-branch                |    4 ++++
 4 files changed, 57 insertions(+)
 create mode 100755 ap-fetch-version-baseline-late
 create mode 100644 branch-settings.linux-next

diff --git a/ap-common b/ap-common
index cb0cfff..254eaf4 100644
--- a/ap-common
+++ b/ap-common
@@ -87,6 +87,11 @@ info_linux_tree () {
 		: ${GITFORCEFLAG_TREE_LINUX_THIS:=-f}
 		;;
 
+	linux-next)
+		: ${TREE_LINUX_THIS:=${KERNEL_SCM}/next/linux-next.git}
+		: ${GITFORCEFLAG_TREE_LINUX_THIS:=-f}
+		;;
+
 	*)
 		return 1
 		;;
diff --git a/ap-fetch-version-baseline-late b/ap-fetch-version-baseline-late
new file mode 100755
index 0000000..ef1a8b1
--- /dev/null
+++ b/ap-fetch-version-baseline-late
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+new=$2
+
+. cri-lock-repos
+. cri-common
+select_xenbranch
+. ap-common
+
+case "$branch" in
+
+linux-next)
+	GIT_DIR=$repos/linux \
+	./sg-check-tested --branch=linux-linus \
+		--blessings=${OSSTEST_BLESSING-real} \
+		--git-ancestor-of=$new --print-revision=linux
+	exit 0
+	;;
+
+*)
+	echo >&2 "branch $branch ?"
+	exit 1
+	;;
+esac
diff --git a/branch-settings.linux-next b/branch-settings.linux-next
new file mode 100644
index 0000000..e9bf926
--- /dev/null
+++ b/branch-settings.linux-next
@@ -0,0 +1,4 @@
+OSSTEST_NO_BASELINE=y
+OSSTEST_PUSH=false
+OLD_REVISION=determine-late
+GITFORCEFLAG=--fail
diff --git a/cr-daily-branch b/cr-daily-branch
index 02fef15..e2b3664 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -171,6 +171,10 @@ linuxfirmware)
 	;;
 esac
 
+if [ "x$OLD_REVISION" = xdetermine-late ]; then
+	OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
+fi
+
 if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
         wantpush=false
 	for checkbranch in x $BRANCHES_ALWAYS; do
-- 
1.7.10.4

Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next

[Ian Campbell]

On Mon, 2014-04-14 at 11:33 +0100, Ian Jackson wrote:
> * Define the new branch's git source repo in ap-common.
> 
> * Provide a branch-settings file which disables running additional
>   baseline tests, listing changesets in flight reports, pushing,
>   and which arranges for the baseline revision for the flight
>   report to be determined "later", ie after the to-be-tested
>   versions of everything have been decided.
> 
> * When the baseline revision is to be determined "later", run
>   the new script ap-fetch-version-baseline-later.  This is another
>   collection of per-branch rules.  We introduce currently one
>   per-branch rule, for linux-next:

FWIW (not much) you seem to have named the script -late not -later.

Did you push these changes already? I thought I'd seen some results
emails.

Anyway, it all looks sensible so far as I am able to tell about these
things.

> * The baseline version for a test of linux-next is
>     - the version tested by the most recent flight on the
>       linux-linus branch;
>     - provided that said version is an ancestor of the linux-next
>       version under test;
>     - otherwise we try an earlier linux-linus flight.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  ap-common                      |    5 +++++
>  ap-fetch-version-baseline-late |   44 ++++++++++++++++++++++++++++++++++++++++
>  branch-settings.linux-next     |    4 ++++
>  cr-daily-branch                |    4 ++++
>  4 files changed, 57 insertions(+)
>  create mode 100755 ap-fetch-version-baseline-late
>  create mode 100644 branch-settings.linux-next
> 
> diff --git a/ap-common b/ap-common
> index cb0cfff..254eaf4 100644
> --- a/ap-common
> +++ b/ap-common
> @@ -87,6 +87,11 @@ info_linux_tree () {
>  		: ${GITFORCEFLAG_TREE_LINUX_THIS:=-f}
>  		;;
>  
> +	linux-next)
> +		: ${TREE_LINUX_THIS:=${KERNEL_SCM}/next/linux-next.git}
> +		: ${GITFORCEFLAG_TREE_LINUX_THIS:=-f}
> +		;;
> +
>  	*)
>  		return 1
>  		;;
> diff --git a/ap-fetch-version-baseline-late b/ap-fetch-version-baseline-late
> new file mode 100755
> index 0000000..ef1a8b1
> --- /dev/null
> +++ b/ap-fetch-version-baseline-late
> @@ -0,0 +1,44 @@
> +#!/bin/bash
> +
> +# This is part of "osstest", an automated testing framework for Xen.
> +# Copyright (C) 2009-2013 Citrix Inc.
> +# 
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +# 
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU Affero General Public License for more details.
> +# 
> +# You should have received a copy of the GNU Affero General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +
> +set -e
> +
> +branch=$1
> +new=$2
> +
> +. cri-lock-repos
> +. cri-common
> +select_xenbranch
> +. ap-common
> +
> +case "$branch" in
> +
> +linux-next)
> +	GIT_DIR=$repos/linux \
> +	./sg-check-tested --branch=linux-linus \
> +		--blessings=${OSSTEST_BLESSING-real} \
> +		--git-ancestor-of=$new --print-revision=linux
> +	exit 0
> +	;;
> +
> +*)
> +	echo >&2 "branch $branch ?"
> +	exit 1
> +	;;
> +esac
> diff --git a/branch-settings.linux-next b/branch-settings.linux-next
> new file mode 100644
> index 0000000..e9bf926
> --- /dev/null
> +++ b/branch-settings.linux-next
> @@ -0,0 +1,4 @@
> +OSSTEST_NO_BASELINE=y
> +OSSTEST_PUSH=false
> +OLD_REVISION=determine-late
> +GITFORCEFLAG=--fail
> diff --git a/cr-daily-branch b/cr-daily-branch
> index 02fef15..e2b3664 100755
> --- a/cr-daily-branch
> +++ b/cr-daily-branch
> @@ -171,6 +171,10 @@ linuxfirmware)
>  	;;
>  esac
>  
> +if [ "x$OLD_REVISION" = xdetermine-late ]; then
> +	OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
> +fi
> +
>  if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
>          wantpush=false
>  	for checkbranch in x $BRANCHES_ALWAYS; do

Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next"):
> On Mon, 2014-04-14 at 11:33 +0100, Ian Jackson wrote:
> > * Define the new branch's git source repo in ap-common.
> > 
> > * Provide a branch-settings file which disables running additional
> >   baseline tests, listing changesets in flight reports, pushing,
> >   and which arranges for the baseline revision for the flight
> >   report to be determined "later", ie after the to-be-tested
> >   versions of everything have been decided.
> > 
> > * When the baseline revision is to be determined "later", run
> >   the new script ap-fetch-version-baseline-later.  This is another
> >   collection of per-branch rules.  We introduce currently one
> >   per-branch rule, for linux-next:
> 
> FWIW (not much) you seem to have named the script -late not -later.

Would you like me to rename it (or reword the commit message) ?

> Did you push these changes already? I thought I'd seen some results
> emails.

No.  I created the relevant tree and arranged by hand for it to have a
version of these patches, and ran it as osstest, but I haven't wired
the tree into cron or pushed the code.

Thanks,
Ian.

Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next

[Ian Campbell]

On Mon, 2014-04-14 at 12:04 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next"):
> > On Mon, 2014-04-14 at 11:33 +0100, Ian Jackson wrote:
> > > * Define the new branch's git source repo in ap-common.
> > > 
> > > * Provide a branch-settings file which disables running additional
> > >   baseline tests, listing changesets in flight reports, pushing,
> > >   and which arranges for the baseline revision for the flight
> > >   report to be determined "later", ie after the to-be-tested
> > >   versions of everything have been decided.
> > > 
> > > * When the baseline revision is to be determined "later", run
> > >   the new script ap-fetch-version-baseline-later.  This is another
> > >   collection of per-branch rules.  We introduce currently one
> > >   per-branch rule, for linux-next:
> > 
> > FWIW (not much) you seem to have named the script -late not -later.
> 
> Would you like me to rename it (or reword the commit message) ?

Either one as you wish.

> > Did you push these changes already? I thought I'd seen some results
> > emails.
> 
> No.  I created the relevant tree and arranged by hand for it to have a
> version of these patches, and ran it as osstest, but I haven't wired
> the tree into cron or pushed the code.

Ah OK.

Ian.

Re: [OSSTEST PATCH 0/3] test linux-next

[David Vrabel]

On 14/04/14 11:33, Ian Jackson wrote:
> We need a different approach to selecting a baseline for testing
> linux-next.  I have tested this series and it appeared to work - the
> results can be found in this email on xen-devel:
> 
>  Subject: [linux-next test] 25862: tolerable FAIL
>  Date: Sat, 12 Apr 2014 00:28:06 +0100
> 
> (The test results were a complete disaster because the linux tip on
> which this linux-next was based cannot boot under Xen.  See flight
> 25840.  I.e. the failures are a bug in Linux, not in the tester.)
> 
>  1/3 sg-check-tested: add "use strict"
>  2/3 sg-check-tested: support --git-ancestor-of
>  3/3 cr-*, ap-*: Facilities for testing linux-next

The first run of this was a disaster.  Unless someone can allocate time
(not me, I have no spare time) to triage/improve this run I don't think
this is going to be useful for regression testing.

David

Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 3/3] cr-*, ap-*: Facilities for testing linux-next"):
> On Mon, 2014-04-14 at 12:04 +0100, Ian Jackson wrote:
> > Would you like me to rename it (or reword the commit message) ?
> 
> Either one as you wish.

I have changed the commit message.  I'll wait with pushing this until
the backlog of other osstest branches is a bit smaller.

Thanks,
Ian.

Re: [OSSTEST PATCH 0/3] test linux-next

[Ian Jackson]

David Vrabel writes ("Re: [Xen-devel] [OSSTEST PATCH 0/3] test linux-next"):
> On 14/04/14 11:33, Ian Jackson wrote:
> > (The test results were a complete disaster because the linux tip on
> > which this linux-next was based cannot boot under Xen.  See flight
> > 25840.  I.e. the failures are a bug in Linux, not in the tester.)
> > 
> >  1/3 sg-check-tested: add "use strict"
> >  2/3 sg-check-tested: support --git-ancestor-of
> >  3/3 cr-*, ap-*: Facilities for testing linux-next
> 
> The first run of this was a disaster.  Unless someone can allocate time
> (not me, I have no spare time) to triage/improve this run I don't think
> this is going to be useful for regression testing.

It was a disaster because linux-linus was a disaster.  I think we
should see what it looks like if linux-linus is better.

Ian.

Re: [OSSTEST PATCH 0/3] test linux-next

[Ian Campbell]

On Mon, 2014-04-14 at 11:33 +0100, Ian Jackson wrote:
>  1/3 sg-check-tested: add "use strict"
>  2/3 sg-check-tested: support --git-ancestor-of
>  3/3 cr-*, ap-*: Facilities for testing linux-next

These have now pushed the push gate, but the branch is not yet enabled
in the crontab or cr-for-branches.

Since linux-next is published daily I think a daily cronjob would be
more appropriate than adding to cr-for-branches which runs multiple
times per day.

Looking at
http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/refs/
and the dates of the various next-XXXXXX tags it seems that there are
new trees Monday to Friday and that they are constructed in the early
hours of the Morning GMT (Stephen who runs Linux next is in Australia I
think).

So I have made the following change to the production crontab and also
pushed it to osstest's pretest branch.

8<-----

>From 382311ca045ac02398b3c2b4937700f3afe9bce9 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 25 Apr 2014 09:37:13 +0100
Subject: [PATCH] crontab: Enable linux-next branch

linux-next is published week days in the early hours GMT.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 crontab | 1 +
 1 file changed, 1 insertion(+)

diff --git a/crontab b/crontab
index c9786af..77c7c20 100644
--- a/crontab
+++ b/crontab
@@ -3,6 +3,7 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 # m		h	dom mon dow	command
 49		1	* * *		cd testing.git && BRANCHES_ALWAYS=xen-unstable	./cr-for-branches branches -w "./cr-daily-branch --real"
 4-59/30		*	* * *		cd testing.git &&				./cr-for-branches branches -q "./cr-daily-branch --real"
+18		9	* * 1-5		cd testing.git && BRANCHES=linux-next		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		10	* * *		cd testing.git && BRANCHES=linux-linus		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		11	* * *		cd testing.git && BRANCHES=linux-mingo-tip-master ./cr-for-branches branches -w "./cr-daily-branch --real"
 18		12	* * *		cd testing.git && BRANCHES=linux-3.0		./cr-for-branches branches -w "./cr-daily-branch --real"
-- 
1.9.0

Re: [OSSTEST PATCH 0/3] test linux-next

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 0/3] test linux-next"):
> On Mon, 2014-04-14 at 11:33 +0100, Ian Jackson wrote:
> >  1/3 sg-check-tested: add "use strict"
> >  2/3 sg-check-tested: support --git-ancestor-of
> >  3/3 cr-*, ap-*: Facilities for testing linux-next
> 
> These have now pushed the push gate, but the branch is not yet enabled
> in the crontab or cr-for-branches.
> 
> Since linux-next is published daily I think a daily cronjob would be
> more appropriate than adding to cr-for-branches which runs multiple
> times per day.
> 
> Looking at
> http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/refs/
> and the dates of the various next-XXXXXX tags it seems that there are
> new trees Monday to Friday and that they are constructed in the early
> hours of the Morning GMT (Stephen who runs Linux next is in Australia I
> think).
> 
> So I have made the following change to the production crontab and also
> pushed it to osstest's pretest branch.

That looks plausible, thanks.

Ian.