[xen-unstable test] 10135: regressions - FAIL

[xen-unstable test] 10135: regressions - FAIL

[xen.org]

flight 10135 xen-unstable real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/10135/

Regressions :-(

Tests which did not succeed and are blocking:
 test-amd64-i386-rhel6hvm-intel  7 redhat-install           fail REGR. vs. 9956
 test-amd64-i386-rhel6hvm-amd  7 redhat-install             fail REGR. vs. 9956
 test-amd64-amd64-xl-win       7 windows-install            fail REGR. vs. 9956
 test-amd64-i386-xl-win-vcpus1  7 windows-install           fail REGR. vs. 9956
 test-i386-i386-xl-win         7 windows-install            fail REGR. vs. 9956

Tests which did not succeed, but are not blocking,
including regressions (tests previously passed) regarded as allowable:
 test-amd64-amd64-xl-pcipt-intel  9 guest-start                 fail never pass
 test-amd64-i386-win-vcpus1   16 leak-check/check             fail   never pass
 test-amd64-i386-win          16 leak-check/check             fail   never pass
 test-i386-i386-win           16 leak-check/check             fail   never pass
 test-amd64-amd64-win         16 leak-check/check             fail   never pass

version targeted for testing:
 xen                  95d4e2e0bed3
baseline version:
 xen                  0a0c02a61676

------------------------------------------------------------
People who touched revisions under test:
  Adin Scannell <adin@scannell.ca>
  Andres Lagar-Cavilla <andres@lagarcavilla.org>
  Anil Madhavapeddy <anil@recoil.org>
  Daniel De Graaf <dgdegra@tycho.nsa.gov>
  George Dunlap <george.dunlap@eu.citrix.com>
  Ian Campbell <ian.campbell@citrix.com>
  Ian Jackson <ian.jackson.citrix.com>
  Ian Jackson <ian.jackson@eu.citrix.com>
  Jan Beulich <jbeulich@suse.com>
  Jean Guyader <jean.guyader@eu.citrix.com>
  Keir Fraser <keir@xen.org>
  Olaf Hering <olaf@aepfle.de>
  Paul Durrant <paul.durrant@citrix.com>
  Stefano Stabellini <stefano.stabellini@eu.citrix.com>
  Tim Deegan <tim@xen.org>
  Wei Wang <wei.wang2@amd.com>
------------------------------------------------------------

jobs:
 build-amd64                                                  pass    
 build-i386                                                   pass    
 build-amd64-oldkern                                          pass    
 build-i386-oldkern                                           pass    
 build-amd64-pvops                                            pass    
 build-i386-pvops                                             pass    
 test-amd64-amd64-xl                                          pass    
 test-amd64-i386-xl                                           pass    
 test-i386-i386-xl                                            pass    
 test-amd64-i386-rhel6hvm-amd                                 fail    
 test-amd64-i386-xl-credit2                                   pass    
 test-amd64-amd64-xl-pcipt-intel                              fail    
 test-amd64-i386-rhel6hvm-intel                               fail    
 test-amd64-i386-xl-multivcpu                                 pass    
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         pass    
 test-i386-i386-pair                                          pass    
 test-amd64-amd64-pv                                          pass    
 test-amd64-i386-pv                                           pass    
 test-i386-i386-pv                                            pass    
 test-amd64-amd64-xl-sedf                                     pass    
 test-amd64-i386-win-vcpus1                                   fail    
 test-amd64-i386-xl-win-vcpus1                                fail    
 test-amd64-amd64-win                                         fail    
 test-amd64-i386-win                                          fail    
 test-i386-i386-win                                           fail    
 test-amd64-amd64-xl-win                                      fail    
 test-i386-i386-xl-win                                        fail    


------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Not pushing.

(No revision log; it would be 906 lines long.)

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Jackson]

xen.org writes ("[xen-unstable test] 10135: regressions - FAIL"):
> flight 10135 xen-unstable real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/10135/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking:
>  test-amd64-i386-rhel6hvm-intel  7 redhat-install       fail REGR. vs. 9956

My bisector wasn't able to finger an exact changeset due to a blocking
bug in the vicinity, but:

  pass      24183:53bec838bb08  Merge
  blocked   24184:4ecd3615e726  tools: use system installed libaio by default.
  blocked   24185:f88c745575bb  docs: remove some fatally out of date ...
  fail      24186:7aa5838499d1  tools: use system libaio for blktap1 as well.

Here the "system installed libaio" is that from Debian squeeze.

So for now I have applied the changeset below in the hope of getting a
pass and push.

Ian.

# HG changeset patch
# User Ian Jackson <Ian.Jackson@eu.citrix.com>
# Date 1322481443 0
# Node ID a9c67c2daf4b0181ef2581471ea920eecb495616
# Parent  95d4e2e0bed374602b5a78ee004b057ad8715d65
tools: Revert to our built-in aio

These two changesets:
   24184:4ecd3615e726  tools: use system installed libaio by default.
   24186:7aa5838499d1  tools: use system libaio for blktap1 as well.
cause HVM guest installs (both Windows and Redhat) to fail on Debian
squeeze with xl.  So change the default for now.

Committed-by: Ian Jackson <ian.jackson@eu.citrix.com>

diff -r 95d4e2e0bed3 -r a9c67c2daf4b Config.mk
--- a/Config.mk	Fri Nov 25 20:32:05 2011 +0000
+++ b/Config.mk	Mon Nov 28 11:57:23 2011 +0000
@@ -232,7 +232,7 @@ PYTHON_TOOLS       ?= y
 OCAML_TOOLS        ?= y
 CONFIG_MINITERM    ?= n
 CONFIG_LOMOUNT     ?= n
-CONFIG_SYSTEM_LIBAIO ?= y
+CONFIG_SYSTEM_LIBAIO ?= n
 
 ifeq ($(OCAML_TOOLS),y)
 OCAML_TOOLS := $(shell ocamlopt -v > /dev/null 2>&1 && echo "y" || echo "n")

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Campbell]

On Mon, 2011-11-28 at 11:58 +0000, Ian Jackson wrote:
> xen.org writes ("[xen-unstable test] 10135: regressions - FAIL"):
> > flight 10135 xen-unstable real [real]
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/10135/
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking:
> >  test-amd64-i386-rhel6hvm-intel  7 redhat-install       fail REGR. vs. 9956
> 
> My bisector wasn't able to finger an exact changeset due to a blocking
> bug in the vicinity, but:
> 
>   pass      24183:53bec838bb08  Merge
>   blocked   24184:4ecd3615e726  tools: use system installed libaio by default.
>   blocked   24185:f88c745575bb  docs: remove some fatally out of date ...
>   fail      24186:7aa5838499d1  tools: use system libaio for blktap1 as well.
> 
> Here the "system installed libaio" is that from Debian squeeze.

This is really weird since nothing in the failings tests even exercises
this stuff. Only blktap (1 and 2) uses aio and the the tests don't use
it and neither does qemu-dm which is the only other thing I can think
of. I'm setting up a repro to see if I can figure out what is going
wrong.

> So for now I have applied the changeset below in the hope of getting a
> pass and push.

I think that's best for now.

Ian.

> 
> Ian.
> 
> # HG changeset patch
> # User Ian Jackson <Ian.Jackson@eu.citrix.com>
> # Date 1322481443 0
> # Node ID a9c67c2daf4b0181ef2581471ea920eecb495616
> # Parent  95d4e2e0bed374602b5a78ee004b057ad8715d65
> tools: Revert to our built-in aio
> 
> These two changesets:
>    24184:4ecd3615e726  tools: use system installed libaio by default.
>    24186:7aa5838499d1  tools: use system libaio for blktap1 as well.
> cause HVM guest installs (both Windows and Redhat) to fail on Debian
> squeeze with xl.  So change the default for now.
> 
> Committed-by: Ian Jackson <ian.jackson@eu.citrix.com>
> 
> diff -r 95d4e2e0bed3 -r a9c67c2daf4b Config.mk
> --- a/Config.mk	Fri Nov 25 20:32:05 2011 +0000
> +++ b/Config.mk	Mon Nov 28 11:57:23 2011 +0000
> @@ -232,7 +232,7 @@ PYTHON_TOOLS       ?= y
>  OCAML_TOOLS        ?= y
>  CONFIG_MINITERM    ?= n
>  CONFIG_LOMOUNT     ?= n
> -CONFIG_SYSTEM_LIBAIO ?= y
> +CONFIG_SYSTEM_LIBAIO ?= n
>  
>  ifeq ($(OCAML_TOOLS),y)
>  OCAML_TOOLS := $(shell ocamlopt -v > /dev/null 2>&1 && echo "y" || echo "n")
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Campbell]

On Mon, 2011-11-28 at 13:38 +0000, Ian Campbell wrote:
> On Mon, 2011-11-28 at 11:58 +0000, Ian Jackson wrote:
> > xen.org writes ("[xen-unstable test] 10135: regressions - FAIL"):
> > > flight 10135 xen-unstable real [real]
> > > http://www.chiark.greenend.org.uk/~xensrcts/logs/10135/
> > > 
> > > Regressions :-(
> > > 
> > > Tests which did not succeed and are blocking:
> > >  test-amd64-i386-rhel6hvm-intel  7 redhat-install       fail REGR. vs. 9956
> > 
> > My bisector wasn't able to finger an exact changeset due to a blocking
> > bug in the vicinity, but:
> > 
> >   pass      24183:53bec838bb08  Merge
> >   blocked   24184:4ecd3615e726  tools: use system installed libaio by default.
> >   blocked   24185:f88c745575bb  docs: remove some fatally out of date ...
> >   fail      24186:7aa5838499d1  tools: use system libaio for blktap1 as well.
> > 
> > Here the "system installed libaio" is that from Debian squeeze.
> 
> This is really weird since nothing in the failings tests even exercises
> this stuff. Only blktap (1 and 2) uses aio and the the tests don't use
> it and neither does qemu-dm which is the only other thing I can think
> of.

Turns out the test system is using the 2.6.32 kernel from xen.git and
hence is using tapdisk2 for the guest cdrom drive.

>  I'm setting up a repro to see if I can figure out what is going
> wrong.

I didn't get as far a reproing but I realised that the issue was that
libaio was not installed on the target and so tapdisk was failing to run
but error handling was broken so we didn't notice apart from some
messages deep in the logs. The following fixes the error handling:

8>-------------------------------------------------------------

# HG changeset patch
# User Ian Campbell <ian.campbell@citrix.com>
# Date 1322493402 0
# Node ID 35883c0b285c8dfce341bdd04deda53ab6db03cf
# Parent  f688f1c8205f90bbfbd6900ccd410382c676b47f
libxl: propagate error from tap_ctl_spawn.

Failure here means that a disk will not be correctly setup. I briefly
scanned tools/blktap2/control.c for other goto constructs which did not
set their err variable but didn't see any others.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff -r f688f1c8205f -r 35883c0b285c tools/blktap2/control/tap-ctl-create.c
--- a/tools/blktap2/control/tap-ctl-create.c	Mon Nov 28 12:59:15 2011 +0000
+++ b/tools/blktap2/control/tap-ctl-create.c	Mon Nov 28 15:16:42 2011 +0000
@@ -44,8 +44,10 @@ tap_ctl_create(const char *params, char 
 		return err;
 
 	id = tap_ctl_spawn();
-	if (id < 0)
+	if (id < 0) {
+		err = id;
 		goto destroy;
+	}
 
 	err = tap_ctl_attach(id, minor);
 	if (err)

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [xen-unstable test] 10135: regressions - FAIL"):
> On Mon, 2011-11-28 at 13:38 +0000, Ian Campbell wrote:
> >  I'm setting up a repro to see if I can figure out what is going
> > wrong.
> 
> I didn't get as far a reproing but I realised that the issue was that
> libaio was not installed on the target and so tapdisk was failing to run
> but error handling was broken so we didn't notice apart from some
> messages deep in the logs. The following fixes the error handling:

Thanks, applied.  (I have also fixed the tester.)

Ian.

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Campbell]

On Mon, 2011-11-28 at 11:58 +0000, Ian Jackson wrote:
> # HG changeset patch
> # User Ian Jackson <Ian.Jackson@eu.citrix.com>
> # Date 1322481443 0
> # Node ID a9c67c2daf4b0181ef2581471ea920eecb495616
> # Parent  95d4e2e0bed374602b5a78ee004b057ad8715d65
> tools: Revert to our built-in aio
> 
> These two changesets:
>    24184:4ecd3615e726  tools: use system installed libaio by default.
>    24186:7aa5838499d1  tools: use system libaio for blktap1 as well.
> cause HVM guest installs (both Windows and Redhat) to fail on Debian
> squeeze with xl.  So change the default for now.
> 
> Committed-by: Ian Jackson <ian.jackson@eu.citrix.com>

AIUI the tester is now installs libaio1 so we can revert this one?

According to
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-amd64-i386-rhel6hvm-intel.redhat-install.html
changeset 24227:1027e7d13d02 was tested and failed. That would have
included
24206:05dd94652d8d tools/check: Add files missing from 24205:5c88358164cc
24205:5c88358164cc tools: check for libaio unless user has configured CONFIG_SYSTEM_LIBAIO=n

Now I don't think the bisector would ever have given us reason to look
at this commits but if they had gone it at the same time as the new
dependency this could have saved a lot of trouble tracking down the
problem.

In
http://www.chiark.greenend.org.uk/~xensrcts/logs/10032/build-i386/4.ts-xen-build.log (one of the runs finger by the bisector which included the above commits) I can see the CHECK-BUILD runes getting run but not the CHECK-INSTALL ones.

The call to ./chk install in the install target seems to have been
deliberately removed in 16906:f4ee7e5793cf for reasons I don't quite
understand (I suspect because of the install-into-a-staging-dir property
of install). Since you install on a different host to the build host
it's possible that the tester might need to jump through some extra
hoops to cause this stuff to run there anyway. Perhaps the tester could
copy tools/check over and execute "./chk install" or "make
check-install" itself? The top-level install.sh does something like
this, but I'm not sure you are (or should be) using it.

Ian.

> 
> diff -r 95d4e2e0bed3 -r a9c67c2daf4b Config.mk
> --- a/Config.mk	Fri Nov 25 20:32:05 2011 +0000
> +++ b/Config.mk	Mon Nov 28 11:57:23 2011 +0000
> @@ -232,7 +232,7 @@ PYTHON_TOOLS       ?= y
>  OCAML_TOOLS        ?= y
>  CONFIG_MINITERM    ?= n
>  CONFIG_LOMOUNT     ?= n
> -CONFIG_SYSTEM_LIBAIO ?= y
> +CONFIG_SYSTEM_LIBAIO ?= n
>  
>  ifeq ($(OCAML_TOOLS),y)
>  OCAML_TOOLS := $(shell ocamlopt -v > /dev/null 2>&1 && echo "y" || echo "n")
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Jackson]

~Ian Campbell writes ("Re: [Xen-devel] [xen-unstable test] 10135: regressions - FAIL"):
>> According to
> http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-amd64-i386-rhel6hvm-intel.redhat-install.html
> changeset 24227:1027e7d13d02 was tested and failed. That would have
> included
> 24206:05dd94652d8d tools/check: Add files missing from 24205:5c88358164cc
> 24205:5c88358164cc tools: check for libaio unless user has configured CONFIG_SYSTEM_LIBAIO=n
> 
> Now I don't think the bisector would ever have given us reason to look
> at this commits but if they had gone it at the same time as the new
> dependency this could have saved a lot of trouble tracking down the
> problem.

"gone in" I guess.  I spent a while staring that trying to make sense
of it assuming it was meant to say "done it".

But, no, because the check would have passed because it's run on the
build host and the build host has the runtime lib installed because
it's a dependency of the dev lib.

> In
> http://www.chiark.greenend.org.uk/~xensrcts/logs/10032/build-i386/4.ts-xen-build.log (one of the runs finger by the bisector which included the above commits) I can see the CHECK-BUILD runes getting run but not the CHECK-INSTALL ones.
> 
> The call to ./chk install in the install target seems to have been
> deliberately removed in 16906:f4ee7e5793cf for reasons I don't quite
> understand (I suspect because of the install-into-a-staging-dir property
> of install). Since you install on a different host to the build host
> it's possible that the tester might need to jump through some extra
> hoops to cause this stuff to run there anyway. Perhaps the tester could
> copy tools/check over and execute "./chk install" or "make
> check-install" itself? The top-level install.sh does something like
> this, but I'm not sure you are (or should be) using it.

I could have the tester run ./chk install on the install host and
see.  But I don't think we promise that this will work.

Running ./chk install on the build host during "make install" is a
good idea but wouldn't have found this problem more quickly.

Ian.

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Campbell]

On Wed, 2011-11-30 at 14:35 +0000, Ian Jackson wrote:
> ~Ian Campbell writes ("Re: [Xen-devel] [xen-unstable test] 10135: regressions - FAIL"):
> >> According to
> > http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-unstable.test-amd64-i386-rhel6hvm-intel.redhat-install.html
> > changeset 24227:1027e7d13d02 was tested and failed. That would have
> > included
> > 24206:05dd94652d8d tools/check: Add files missing from 24205:5c88358164cc
> > 24205:5c88358164cc tools: check for libaio unless user has configured CONFIG_SYSTEM_LIBAIO=n
> > 
> > Now I don't think the bisector would ever have given us reason to look
> > at this commits but if they had gone it at the same time as the new
> > dependency this could have saved a lot of trouble tracking down the
> > problem.
> 
> "gone in" I guess.  I spent a while staring that trying to make sense
> of it assuming it was meant to say "done it".

Yes, sorry. "... if they had gone in at the ... "

> But, no, because the check would have passed because it's run on the
> build host and the build host has the runtime lib installed because
> it's a dependency of the dev lib.
> 
> > In
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/10032/build-i386/4.ts-xen-build.log (one of the runs finger by the bisector which included the above commits) I can see the CHECK-BUILD runes getting run but not the CHECK-INSTALL ones.
> > 
> > The call to ./chk install in the install target seems to have been
> > deliberately removed in 16906:f4ee7e5793cf for reasons I don't quite
> > understand (I suspect because of the install-into-a-staging-dir property
> > of install). Since you install on a different host to the build host
> > it's possible that the tester might need to jump through some extra
> > hoops to cause this stuff to run there anyway. Perhaps the tester could
> > copy tools/check over and execute "./chk install" or "make
> > check-install" itself? The top-level install.sh does something like
> > this, but I'm not sure you are (or should be) using it.
> 
> I could have the tester run ./chk install on the install host and
> see.  But I don't think we promise that this will work.

We could decide to promise it going forward if it is useful.

> Running ./chk install on the build host during "make install" is a
> good idea but wouldn't have found this problem more quickly.

Right, if they are running on the same host I don't think ./chk install
will find much which is not found by ./chk build in practice.

Ian.

Re: [xen-unstable test] 10135: regressions - FAIL

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [xen-unstable test] 10135: regressions - FAIL"):
> AIUI the tester is now installs libaio1 so we can revert this one?

Yes, I've done so.

# HG changeset patch
# User Ian Jackson <Ian.Jackson@eu.citrix.com>
# Date 1322756338 0
# Node ID a07e45d2aa7f792674b6325b11a48d297fd4d075
# Parent  7b180380681db3382f7c2f38e096bc551aa301d2
tools: Switch to system libaio (again)

The test system problems which prompted 24233:a9c67c2daf4b (itself a
tiny partial revert of 24184:4ecd3615e726 / 24186:7aa5838499d1) have
now been resolved, we think.  So revert 24233.

Committed-by: Ian Jackson <ian.jackson@eu.citrix.com>

diff -r 7b180380681d -r a07e45d2aa7f Config.mk
--- a/Config.mk	Thu Dec 01 16:05:51 2011 +0000
+++ b/Config.mk	Thu Dec 01 16:18:58 2011 +0000
@@ -232,7 +232,7 @@ PYTHON_TOOLS       ?= y
 OCAML_TOOLS        ?= y
 CONFIG_MINITERM    ?= n
 CONFIG_LOMOUNT     ?= n
-CONFIG_SYSTEM_LIBAIO ?= n
+CONFIG_SYSTEM_LIBAIO ?= y
 
 ifeq ($(OCAML_TOOLS),y)
 OCAML_TOOLS := $(shell ocamlopt -v > /dev/null 2>&1 && echo "y" || echo "n")