[rumpuserxen test] 33416: regressions - FAIL

[rumpuserxen test] 33416: regressions - FAIL

[xen.org]

flight 33416 rumpuserxen real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-amd64-rumpuserxen-amd64  8 guest-start         fail REGR. vs. 33401

version targeted for testing:
 rumpuserxen          ec3b62146d52e5deab349b00c6e768937d39a01f
baseline version:
 rumpuserxen          1e71553faec32af113f1d91c8331341eed320ea5

------------------------------------------------------------
People who touched revisions under test:
  Antti Kantee <pooka@iki.fi>
  Martin Lucina <martin@lucina.net>
------------------------------------------------------------

jobs:
 build-amd64                                                  pass    
 build-i386                                                   pass    
 build-amd64-pvops                                            pass    
 build-i386-pvops                                             pass    
 build-amd64-rumpuserxen                                      pass    
 build-i386-rumpuserxen                                       pass    
 test-amd64-amd64-rumpuserxen-amd64                           fail    
 test-amd64-i386-rumpuserxen-i386                             pass    


------------------------------------------------------------
sg-report-flight on osstest.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit ec3b62146d52e5deab349b00c6e768937d39a01f
Author: Antti Kantee <pooka@iki.fi>
Date:   Tue Jan 13 18:26:15 2015 +0000

    pull in new buildrump.sh and rumpsrc (openssl 1.0.1k)

commit e858a76fe81e8fda2efcba025494b59747143a64
Author: Martin Lucina <martin@lucina.net>
Date:   Tue Jan 13 19:05:09 2015 +0100

    rumpconfig: Enable deconfig for network interfaces
    
    Enable deconfig of network interaces, including dhcp. Current tests show
    that issue #14 appears to be fixed, so this should be good to go.
    
    Signed-off-by: Martin Lucina <martin@lucina.net>

commit 5c716e22bdb0d6f62e22b6ad97dd7048c29db4ca
Author: Martin Lucina <martin@lucina.net>
Date:   Tue Jan 13 18:20:32 2015 +0100

    Make initial stack setup consistent with per-thread stack setup
    
    Push an "end of backtrace" marker onto the stack and zero BP. For x86_64
    additionally push an extra 0 to ensure correct stack alignment as per
    the ABI.
    
    Signed-off-by: Martin Lucina <martin@lucina.net>

commit 97afb0013eeb4eb28bcd173bcc83c9c9163c5bdd
Author: Martin Lucina <martin@lucina.net>
Date:   Mon Jan 12 15:20:00 2015 +0100

    Correct x86-64 stack alignment at thread entry (fixes #18)
    
    The x86-64 ABI mandates in section 3.2.2 that (%rsp + 8) is always a
    multiple of 16 when control is transferred to a function entry point. We
    did not correctly implement this convention which resulted in broken
    alignment for stack variables, and, following from that, SSE
    instructions generated by GCC faulting due to bad alignment.
    
    For the x86-32 version the alignment constraint appears to be[1] that
    %esp must be a multiple of 16, this is already the case so no change is
    made to x86-32.s.
    
    [1] http://sourceforge.net/p/fbc/bugs/659/,
        https://gcc.gnu.org/bugzilla/show_bug.cgi?id=38496
    
    Signed-off-by: Martin Lucina <martin@lucina.net>

Re: [rumpuserxen test] 33416: regressions - FAIL

[Martin Lucina]

Ian.Jackson@eu.citrix.com said:
> flight 33416 rumpuserxen real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/
> 
> Regressions :-(

I presume this is the console log:

http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/test-amd64-amd64-rumpuserxen-amd64/guest-rump-console

---
GPF rip: 0x6018b, error_code=0
Thread: main
RIP: e030:[<000000000006018b>] 
RSP: e02b:000000000042f4d0  EFLAGS: 00010216
RAX: 0000000000000018 RBX: 00000000000001ee RCX: 00000000002a2e60
RDX: 00000000000001ee RSI: 000000000042f4e0 RDI: 48d10148000003ff
RBP: 000000000042f4e0 R08: 0000000000000001 R09: 00000000005cd010
R10: 0000000000681000 R11: 000000000042f270 R12: 000000000000e02b
R13: 0000000000000000 R14: 0000000000299e90 R15: 0000000000000000
base is 0x42f4e0 caller is 0x6023e
base is 0x42fe88 caller is 0x42ff70
---

Ian, do you have the build output (rump-kernel) around somewhere? Would be
useful to get a symbol table to see where 0x6018b lies.

Martin

Re: [rumpuserxen test] 33416: regressions - FAIL

[Ian Jackson]

Martin Lucina writes ("Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL"):
> http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/test-amd64-amd64-rumpuserxen-amd64/guest-rump-console

Yes.

> ---
> GPF rip: 0x6018b, error_code=0
...
> ---
> 
> Ian, do you have the build output (rump-kernel) around somewhere? Would be
> useful to get a symbol table to see where 0x6018b lies.

The build was this
  http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/info.html
and the output files are here:
  http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/build/
of which the relevant part is this:
  http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/build/rumpuserxendist.tar.gz
which contains
  usr/local/lib/xen/rump-kernel/rump-kernel

Looking at the output of `objdump -d .../rump-kernel' it's in
__sflush, called by fflush.  The alleged caller of fflush, `0x42ff70',
isn't in the text segment.

Ian.

Re: [rumpuserxen test] 33416: regressions - FAIL

[Martin Lucina]

Ian.Jackson@eu.citrix.com said:
> Martin Lucina writes ("Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL"):
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/test-amd64-amd64-rumpuserxen-amd64/guest-rump-console
> 
> Yes.
> 
> > ---
> > GPF rip: 0x6018b, error_code=0
> ...
> > ---
> > 
> > Ian, do you have the build output (rump-kernel) around somewhere? Would be
> > useful to get a symbol table to see where 0x6018b lies.
> 
> The build was this
>   http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/info.html
> and the output files are here:
>   http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/build/
> of which the relevant part is this:
>   http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/build/rumpuserxendist.tar.gz
> which contains
>   usr/local/lib/xen/rump-kernel/rump-kernel
> 
> Looking at the output of `objdump -d .../rump-kernel' it's in
> __sflush, called by fflush.  The alleged caller of fflush, `0x42ff70',
> isn't in the text segment.

Ok, that makes no sense. Can you re-run the tests to see if it's
repeatable?

Martin

Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL

[Martin Lucina]

martin@lucina.net said:
> > The build was this
> >   http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/info.html
> > and the output files are here:
> >   http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/build/
> > of which the relevant part is this:
> >   http://www.chiark.greenend.org.uk/~xensrcts/logs/33416/build-amd64-rumpuserxen/build/rumpuserxendist.tar.gz
> > which contains
> >   usr/local/lib/xen/rump-kernel/rump-kernel
> > 
> > Looking at the output of `objdump -d .../rump-kernel' it's in
> > __sflush, called by fflush.  The alleged caller of fflush, `0x42ff70',
> > isn't in the text segment.
> 
> Ok, that makes no sense. Can you re-run the tests to see if it's
> repeatable?

Just did this myself against the rumpuserxendist.tar.gz from the build and
it does not crash here (Debian jessie, Xen 4.4.1). The httpd is also
working fine.

Martin

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet

Re: [rumpuserxen test] 33416: regressions - FAIL

[Ian Jackson]

Martin Lucina writes ("Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL"):
> Ian.Jackson@eu.citrix.com said:
> > Looking at the output of `objdump -d .../rump-kernel' it's in
> > __sflush, called by fflush.  The alleged caller of fflush, `0x42ff70',
> > isn't in the text segment.
> 
> Ok, that makes no sense. Can you re-run the tests to see if it's
> repeatable?

I can't easily rerun it by hand, but the cron job is rerunning it
already.  Also our automatic bisector is working on it.

(Neither of these will report very quickly because our test farm is
rather overloaded right now.)

Ian.

[rumpuserxen test] 33434: tolerable FAIL - PUSHED

[xen.org]

flight 33434 rumpuserxen real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/33434/

Failures :-/ but no regressions.

Tests which are failing intermittently (not blocking):
 test-amd64-i386-rumpuserxen-i386  5 xen-boot                fail pass in 33416
 test-amd64-amd64-rumpuserxen-amd64  8 guest-start  fail in 33416 pass in 33434

version targeted for testing:
 rumpuserxen          ec3b62146d52e5deab349b00c6e768937d39a01f
baseline version:
 rumpuserxen          1e71553faec32af113f1d91c8331341eed320ea5

------------------------------------------------------------
People who touched revisions under test:
  Antti Kantee <pooka@iki.fi>
  Martin Lucina <martin@lucina.net>
------------------------------------------------------------

jobs:
 build-amd64                                                  pass    
 build-i386                                                   pass    
 build-amd64-pvops                                            pass    
 build-i386-pvops                                             pass    
 build-amd64-rumpuserxen                                      pass    
 build-i386-rumpuserxen                                       pass    
 test-amd64-amd64-rumpuserxen-amd64                           pass    
 test-amd64-i386-rumpuserxen-i386                             fail    


------------------------------------------------------------
sg-report-flight on osstest.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Pushing revision :

+ branch=rumpuserxen
+ revision=ec3b62146d52e5deab349b00c6e768937d39a01f
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"Repos"} or die $!;
        '
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x '!=' x/export/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/export/home/osstest/repos/lock
++ exec with-lock-ex -w /export/home/osstest/repos/lock ./ap-push rumpuserxen ec3b62146d52e5deab349b00c6e768937d39a01f
+ branch=rumpuserxen
+ revision=ec3b62146d52e5deab349b00c6e768937d39a01f
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"Repos"} or die $!;
        '
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x/export/home/osstest/repos/lock '!=' x/export/home/osstest/repos/lock ']'
+ . cri-common
++ . cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=rumpuserxen
+ xenbranch=xen-unstable
+ '[' xrumpuserxen = xlinux ']'
+ linuxbranch=
+ '[' x = x ']'
+ qemuubranch=qemu-upstream-unstable
+ : tested/2.6.39.x
+ . ap-common
++ : osstest@xenbits.xensource.com
++ : git://xenbits.xen.org/xen.git
++ : osstest@xenbits.xensource.com:/home/xen/git/xen.git
++ : git://xenbits.xen.org/staging/qemu-xen-unstable.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/libvirt.git
++ : osstest@xenbits.xensource.com:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : https://github.com/rumpkernel/rumprun-xen
++ : git
++ : git://xenbits.xen.org/rumpuser-xen.git
++ : osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+++ besteffort_repo https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ cached_repo https://github.com/rumpkernel/rumpkernel-netbsd-src '[fetch=try]'
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local 'options=[fetch=try]'
++++ getconfig GitCacheProxy
++++ perl -e '
                use Osstest;
                readglobalconfig();
                print $c{"GitCacheProxy"} or die $!;
        '
+++ local cache=git://drall.uk.xensource.com:9419/
+++ '[' xgit://drall.uk.xensource.com:9419/ '!=' x ']'
+++ echo 'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : 'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : git
++ : git://git.seabios.org/seabios.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/linux-firmware.git
++ : osstest@xenbits.xensource.com:/home/osstest/ext/linux-firmware.git
++ : git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
++ : osstest@xenbits.xensource.com:/home/xen/git/linux-pvops.git
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-3.14
++ : tested/linux-arm-xen
++ '[' xgit://xenbits.xen.org/linux-pvops.git = x ']'
++ '[' x = x ']'
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-arm-xen
++ : git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
++ : tested/2.6.39.x
++ : daily-cron.rumpuserxen
++ : daily-cron.rumpuserxen
++ : daily-cron.rumpuserxen
++ : daily-cron.rumpuserxen
++ : daily-cron.rumpuserxen
++ : http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27
++ : git://xenbits.xen.org/staging/qemu-upstream-unstable.git
++ : daily-cron.rumpuserxen
++ : git://git.qemu.org/qemu.git
++ : git://xenbits.xen.org/osstest/qemu.git
++ : osstest@xenbits.xensource.com:/home/xen/git/osstest/qemu.git
++ : daily-cron.rumpuserxen
+ TREE_LINUX=osstest@xenbits.xensource.com:/home/xen/git/linux-pvops.git
+ TREE_QEMU_MAINLINE=osstest@xenbits.xensource.com:/home/xen/git/osstest/qemu.git
+ TREE_QEMU_UPSTREAM=osstest@xenbits.xensource.com:/home/xen/git/qemu-upstream-unstable.git
+ TREE_XEN=osstest@xenbits.xensource.com:/home/xen/git/xen.git
+ TREE_LIBVIRT=osstest@xenbits.xensource.com:/home/xen/git/libvirt.git
+ TREE_RUMPUSERXEN=osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+ TREE_SEABIOS=osstest@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
+ info_linux_tree rumpuserxen
+ case $1 in
+ return 1
+ case "$branch" in
+ cd /export/home/osstest/repos/rumpuserxen
+ git push osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git ec3b62146d52e5deab349b00c6e768937d39a01f:xen-tested-master
To osstest@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
   1e71553..ec3b621  ec3b62146d52e5deab349b00c6e768937d39a01f -> xen-tested-master

Re: [rumpuserxen test] 33416: regressions - FAIL [and 1 more messages]

[Ian Jackson]

Ian Jackson writes ("Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL"):
...
> I can't easily rerun it by hand, but the cron job is rerunning it
> already.  Also our automatic bisector is working on it.

xen.org writes ("[rumpuserxen test] 33434: tolerable FAIL - PUSHED"):
> flight 33434 rumpuserxen real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/33434/
> 
> Failures :-/ but no regressions.
> 
> Tests which are failing intermittently (not blocking):
>  test-amd64-i386-rumpuserxen-i386  5 xen-boot            fail pass in 33416
>  test-amd64-amd64-rumpuserxen-amd64  8 guest-start  fail in 33416 pass in 33434

The former is nothing to do with the rump kernel code and is due to an
infrastructure problem in the test setup.

The latter is the GPF turning out to be unreproducible.

I guess we should keep an eye out in case it turns out to be a race
rather than a cosmic ray.

Ian.

Re: [rumpuserxen test] 33416: regressions - FAIL [and 1 more messages]

[Antti Kantee]

On 16/01/15 11:17, Ian Jackson wrote:
> The latter is the GPF turning out to be unreproducible.
>
> I guess we should keep an eye out in case it turns out to be a race
> rather than a cosmic ray.

I do not believe in cosmic rays.

Some notes in case it resurfaces:

1) the topmost caller address appears to be on the stack (??)

2) "calling main()" seems to be output properly, the next line ("running 
demos") doesn't.  there is exactly one function call (~20 instructions) 
between these two lines

3) the first printf goes to stderr, while the second, supposedly failing 
one, goes to stdout (no idea why)

4) it's crashing calling fp->_swrite().  notably, the execution of 
__sflush() might or might not go that far if fp upon entry points to 
garbage (existing garbage, not page faulting garbage)

5) the call is made as:
    6018b:       41 ff 54 24 50          callq  *0x50(%r12)

from the output:
R12: 000000000000e02b
(e02b is somewhere in the middle of xenbus_xb_write, highly unlikely to 
be relevant, more likely to be some random garbage)

from the image:
rump-kernel$ nm rump-kernel  | grep __sF
0000000000299d60 D __sF
(stdout is __sF[1])

There are no (user) threads active yet.  I currently have no theories 
how the failure is reachable, at least assuming that the console output 
is accurate.  I still don't believe in cosmic rays, though ;)