[xen-4.4-testing test] 52629: regressions - FAIL

[xen-4.4-testing test] 52629: regressions - FAIL

[osstest service user]

flight 52629 xen-4.4-testing real [real]
http://logs.test-lab.xenproject.org/osstest/logs/52629/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 build-armhf-libvirt           5 libvirt-build             fail REGR. vs. 50429
 test-amd64-amd64-xl-credit2  17 guest-localmigrate/x10    fail REGR. vs. 50429
 test-amd64-amd64-xl-win7-amd64 12 guest-localmigrate      fail REGR. vs. 50429
 test-amd64-i386-qemut-rhel6hvm-amd 11 guest-start.2       fail REGR. vs. 50429
 test-amd64-i386-qemuu-rhel6hvm-intel 12 guest-start/redhat.repeat fail REGR. vs. 50429
 test-amd64-amd64-pv          20 guest-start/debian.repeat fail REGR. vs. 50429
 build-armhf-pvops             5 kernel-build              fail REGR. vs. 50429

Regressions which are regarded as allowable (not blocking):
 test-amd64-amd64-xl-sedf     17 guest-localmigrate/x10    fail REGR. vs. 50429
 test-amd64-amd64-xl-qemuu-ovmf-amd64 12 guest-localmigrate fail REGR. vs. 50429
 test-amd64-amd64-xl-qemuu-winxpsp3 15 guest-localmigrate/x10   fail like 50316
 test-amd64-i386-pair        21 guest-migrate/src_host/dst_host fail like 50429

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-rumpuserxen-amd64  1 build-check(1)               blocked n/a
 test-amd64-i386-rumpuserxen-i386  1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl           1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-sedf      1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-sedf-pin  1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-multivcpu  1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-arndale   1 build-check(1)               blocked  n/a
 test-armhf-armhf-libvirt      1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-credit2   1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-cubietruck  1 build-check(1)               blocked  n/a
 test-amd64-amd64-libvirt     12 migrate-support-check        fail   never pass
 test-amd64-i386-libvirt      12 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop             fail never pass
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 16 guest-stop         fail never pass
 test-amd64-i386-xl-win7-amd64 16 guest-stop                   fail  never pass
 test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop             fail never pass
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop              fail never pass
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop              fail never pass
 test-amd64-i386-xend-winxpsp3 20 leak-check/check             fail  never pass
 build-i386-rumpuserxen        6 xen-build                    fail   never pass
 build-amd64-rumpuserxen       6 xen-build                    fail   never pass
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1 16 guest-stop         fail never pass
 test-amd64-i386-xl-winxpsp3-vcpus1 16 guest-stop               fail never pass
 test-amd64-amd64-xl-winxpsp3 16 guest-stop                   fail   never pass
 test-amd64-amd64-xl-qemut-winxpsp3 16 guest-stop               fail never pass
 test-amd64-i386-xend-qemut-winxpsp3 20 leak-check/check        fail never pass

version targeted for testing:
 xen                  cb296dd08587df2c59ef7ebc6d600cdace468bf2
baseline version:
 xen                  6b09a29ced2e7fc449a39f513e1d8c2b10d2af6d

------------------------------------------------------------
People who touched revisions under test:
  Andrew Cooper <andrew.cooper3@citrix.com>
  Ian Campbell <ian.campbell@citrix.com>
  Jan Beulich <jbeulich@suse.com>
  JeHyeon Yeon <tom.yeon@windriver.com>
  Stefano Stabellini <stefano.stabellini@eu.citrix.com>
  Tim Deegan <tim@xen.org>
  Yang Zhang <yang.z.zhang@intel.com>
------------------------------------------------------------

jobs:
 build-amd64-xend                                             pass    
 build-i386-xend                                              pass    
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-i386                                                   pass    
 build-amd64-libvirt                                          pass    
 build-armhf-libvirt                                          fail    
 build-i386-libvirt                                           pass    
 build-amd64-pvops                                            pass    
 build-armhf-pvops                                            fail    
 build-i386-pvops                                             pass    
 build-amd64-rumpuserxen                                      fail    
 build-i386-rumpuserxen                                       fail    
 test-amd64-amd64-xl                                          pass    
 test-armhf-armhf-xl                                          blocked 
 test-amd64-i386-xl                                           pass    
 test-amd64-i386-rhel6hvm-amd                                 pass    
 test-amd64-i386-qemut-rhel6hvm-amd                           fail    
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64                     pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     pass    
 test-amd64-i386-freebsd10-amd64                              pass    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         fail    
 test-amd64-i386-xl-qemuu-ovmf-amd64                          pass    
 test-amd64-amd64-rumpuserxen-amd64                           blocked 
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-amd64-amd64-xl-win7-amd64                               fail    
 test-amd64-i386-xl-win7-amd64                                fail    
 test-armhf-armhf-xl-arndale                                  blocked 
 test-amd64-amd64-xl-credit2                                  fail    
 test-armhf-armhf-xl-credit2                                  blocked 
 test-armhf-armhf-xl-cubietruck                               blocked 
 test-amd64-i386-freebsd10-i386                               pass    
 test-amd64-i386-rumpuserxen-i386                             blocked 
 test-amd64-i386-rhel6hvm-intel                               pass    
 test-amd64-i386-qemut-rhel6hvm-intel                         pass    
 test-amd64-i386-qemuu-rhel6hvm-intel                         fail    
 test-amd64-amd64-libvirt                                     pass    
 test-armhf-armhf-libvirt                                     blocked 
 test-amd64-i386-libvirt                                      pass    
 test-amd64-amd64-xl-multivcpu                                pass    
 test-armhf-armhf-xl-multivcpu                                blocked 
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         fail    
 test-amd64-amd64-xl-sedf-pin                                 pass    
 test-armhf-armhf-xl-sedf-pin                                 blocked 
 test-amd64-amd64-pv                                          fail    
 test-amd64-i386-pv                                           pass    
 test-amd64-amd64-xl-sedf                                     fail    
 test-armhf-armhf-xl-sedf                                     blocked 
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-winxpsp3-vcpus1                           fail    
 test-amd64-i386-xend-qemut-winxpsp3                          fail    
 test-amd64-amd64-xl-qemut-winxpsp3                           fail    
 test-amd64-amd64-xl-qemuu-winxpsp3                           fail    
 test-amd64-i386-xend-winxpsp3                                fail    
 test-amd64-amd64-xl-winxpsp3                                 fail    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/osstest/pub/logs
images: /home/osstest/pub/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit cb296dd08587df2c59ef7ebc6d600cdace468bf2
Author: Tim Deegan <tim@xen.org>
Date:   Tue Apr 21 09:22:56 2015 +0200

    x86_emulate: split the {reg,mem} union in struct operand
    
    In the hopes of making any future errors along the lines of XSA-123
    into clean crashes instead of memory corruption bugs.
    
    Signed-off-by: Tim Deegan <tim@xen.org>
    Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
    master commit: dd766684e7c97641bbaf16ee2b0e9add388199b7
    master date: 2015-03-27 16:13:07 +0100

commit 19ae8c19539159b132cd63dada4c18b17cf71a4a
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Apr 21 09:22:17 2015 +0200

    VT-d: improve fault info logging
    
    I got repeatedly annoyed by there not getting anything logged by
    default on VT-d faults (and hence having to tell people to add extra
    command line options), and hence I think it is time to redo this code:
    Log basic fault information at guest-warning level (rate limited by
    default), and show the page walk in verbose rather than only in debug
    mode. Break up multi-line message so that each gets a proper log level
    attached, at once splitting out the common part. Also don't log
    "unknown" faults as interrupt-remapping ones.
    
    As a minor cleanup fix the type of the involved "fault_type" variables.
    
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
    Acked-by: Yang Zhang <yang.z.zhang@intel.com>
    master commit: f0250f4b4818f5f4230995407ea2501de3485934
    master date: 2015-03-27 15:23:25 +0100

commit 8f2d240fc097c519ac1b4aabf28dff706843486c
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Apr 21 09:21:41 2015 +0200

    x86/MSI: fix error handling
    
    __setup_msi_irq() needs to undo what it did before calling
    write_msi_msg() in case that returned an error.
    
    map_domain_pirq() needs to get rid of the MSI descriptor it
    (implicitly) allocated. The case of a setup_msi_irq() failure on a
    non-initial multi-vector-MSI interrupt needs special handling: While
    the initial IRQ will get freed by the caller (who also passed it to
    us), we need to undo the effect setup_msi_irq() had on it. (As a
    benefit from the added call to msi_free_irq() we no longer need to
    explicitly call destroy_irq() on the non-initial slots.)
    
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
    master commit: 29c1b7886c36d4e6aa03a779b2251b829d9689c3
    master date: 2015-03-26 11:19:57 +0100

commit ce516e8880b3fedf7a87438534e5df5fdb7613d4
Author: JeHyeon Yeon <tom.yeon@windriver.com>
Date:   Tue Apr 21 09:21:05 2015 +0200

    LZ4 : fix the data abort issue
    
    If the part of the compression data are corrupted, or the compression
    data is totally fake, the memory access over the limit is possible.
    
    This is the log from my system usning lz4 decompression.
       [6502]data abort, halting
       [6503]r0  0x00000000 r1  0x00000000 r2  0xdcea0ffc r3  0xdcea0ffc
       [6509]r4  0xb9ab0bfd r5  0xdcea0ffc r6  0xdcea0ff8 r7  0xdce80000
       [6515]r8  0x00000000 r9  0x00000000 r10 0x00000000 r11 0xb9a98000
       [6522]r12 0xdcea1000 usp 0x00000000 ulr 0x00000000 pc  0x820149bc
       [6528]spsr 0x400001f3
    and the memory addresses of some variables at the moment are
        ref:0xdcea0ffc, op:0xdcea0ffc, oend:0xdcea1000
    
    As you can see, COPYLENGH is 8bytes, so @ref and @op can access the momory
    over @oend.
    
    Signed-off-by: JeHyeon Yeon <tom.yeon@windriver.com>
    Reviewed-by: David Sterba <dsterba@suse.cz>
    [Linux commit d5e7cafd69da24e6d6cc988fab6ea313a2577efc]
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Acked-by: Ian Campbell <ian.campbell@citrix.com>
    master commit: fcc17f96c2776eb220e3dee79fd0ad6a624ffcd9
    master date: 2015-03-26 11:19:10 +0100

commit 1af109570552b4f7b2aaa3a335b447c50ebb676c
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Apr 21 09:20:34 2015 +0200

    hvmloader: don't treat ROM BAR like other BARs
    
    Its low 11 bits have different meaning.
    
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
    Acked-by: Ian Campbell <ian.campbell@citrix.com>
    Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
    master commit: 73aa7fc2926c5ae30d8ebd049beadbb48e24d6c6
    master date: 2015-03-26 11:17:51 +0100

commit 5fea6a3db9ad60e956a5182e2bd8184bdf992fa8
Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Date:   Tue Apr 21 09:19:37 2015 +0200

    QEMU_UPSTREAM_REVISION = master again
    
    Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

commit 5365c7b98240acdaac2a332fe2070d5cc5aa596f
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date:   Tue Apr 21 09:18:39 2015 +0200

    domctl/sysctl: don't leak hypervisor stack to toolstacks
    
    This is CVE-2015-3340 / XSA-132.
    
    Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
    Reviewed-by: Jan Beulich <jbeulich@suse.com>
    Acked-by: Ian Campbell <ian.campbell@citrix.com>
    master commit: 4ff3449f0e9d175ceb9551d3f2aecb59273f639d
    master date: 2015-04-21 09:03:15 +0200
========================================

Re: [xen-4.4-testing test] 52629: regressions - FAIL

[Ian Campbell]

On Thu, 2015-04-30 at 11:27 +0000, osstest service user wrote:
> flight 52629 xen-4.4-testing real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/52629/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  build-armhf-libvirt           5 libvirt-build             fail REGR. vs. 50429
>  test-amd64-amd64-xl-credit2  17 guest-localmigrate/x10    fail REGR. vs. 50429
>  test-amd64-amd64-xl-win7-amd64 12 guest-localmigrate      fail REGR. vs. 50429
>  test-amd64-i386-qemut-rhel6hvm-amd 11 guest-start.2       fail REGR. vs. 50429
>  test-amd64-i386-qemuu-rhel6hvm-intel 12 guest-start/redhat.repeat fail REGR. vs. 50429
>  test-amd64-amd64-pv          20 guest-start/debian.repeat fail REGR. vs. 50429
>  build-armhf-pvops             5 kernel-build              fail REGR. vs. 50429

Several of the non-build ones here seem to be ssh timeouts.
http://logs.test-lab.xenproject.org/osstest/logs/52629/test-amd64-i386-qemut-rhel6hvm-amd/11.ts-guest-start.log
status (timed out) at Osstest/TestSupport.pm line 397.
debug1: Unspecified GSS failure.  Minor code may provide more information
Cannot determine realm for numeric host address

http://logs.test-lab.xenproject.org/osstest/logs/52629/test-amd64-amd64-xl-credit2/17.ts-guest-localmigrate.log
has things just appearing to timeout after:
        debug1: Sending command: date

http://logs.test-lab.xenproject.org/osstest/logs/52629/test-amd64-i386-qemuu-rhel6hvm-intel/12.ts-repeat-test.log
has the same gss failure as the first.

http://logs.test-lab.xenproject.org/osstest/logs/52629/test-amd64-amd64-pv/20.ts-repeat-test.log
has timeouts with nothing very suspicious looking in the (verbose) logs.

Ian.

Re: [xen-4.4-testing test] 52629: regressions - FAIL

[Ian Campbell]

On Fri, 2015-05-01 at 12:36 +0100, Ian Campbell wrote:
> On Thu, 2015-04-30 at 11:27 +0000, osstest service user wrote:
> > flight 52629 xen-4.4-testing real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/52629/
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  build-armhf-libvirt           5 libvirt-build             fail REGR. vs. 50429
> >  test-amd64-amd64-xl-credit2  17 guest-localmigrate/x10    fail REGR. vs. 50429
> >  test-amd64-amd64-xl-win7-amd64 12 guest-localmigrate      fail REGR. vs. 50429
> >  test-amd64-i386-qemut-rhel6hvm-amd 11 guest-start.2       fail REGR. vs. 50429
> >  test-amd64-i386-qemuu-rhel6hvm-intel 12 guest-start/redhat.repeat fail REGR. vs. 50429
> >  test-amd64-amd64-pv          20 guest-start/debian.repeat fail REGR. vs. 50429
> >  build-armhf-pvops             5 kernel-build              fail REGR. vs. 50429
> 
> Several of the non-build ones here seem to be ssh timeouts.
> http://logs.test-lab.xenproject.org/osstest/logs/52629/test-amd64-i386-qemut-rhel6hvm-amd/11.ts-guest-start.log
> status (timed out) at Osstest/TestSupport.pm line 397.
> debug1: Unspecified GSS failure.  Minor code may provide more information
> Cannot determine realm for numeric host address

Going through other recent flight failures I'm seeing all of these a
fair bit. Searching for this particular message led me to
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/416264

That suggests "-o GSSAPIAuthentication=no" as a workaround.

Seems like the root cause might be to do with either reverse DNS or
perhaps AVAHI. Shall we just nix GSSAPIAuthentication?

Ian.

Re: [xen-4.4-testing test] 52629: regressions - FAIL

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [xen-4.4-testing test] 52629: regressions - FAIL"):
> Going through other recent flight failures I'm seeing all of these a
> fair bit. Searching for this particular message led me to
> https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/416264
> 
> That suggests "-o GSSAPIAuthentication=no" as a workaround.

I think this is a red herring.  Our reverse DNS is not set up, but we
do have empty zones for all the relevant reverse ranges, so reverse
DNS lookups will fail quickly.

As I said on IRC:

13:49 <Diziet> Yesterday I found a copy of `convert' using 3G of RAM and the 
               osstest VM had a load of 30
13:51 <Diziet> http://logs.test-lab.xenproject.org/osstest/logs/52629/test-amd64-amd64-pv/20.ts-repeat-test.log
               shows ssh taking >10s to get to the point of 
               printing the remote protocol version, and getting timed out 
               halfway through enumerating its local private keys.
13:51 <Diziet> I sent it a SIGSTOP.  Let me find it...
13:53 <Diziet> Looks like that particular convert started on the 30th but a 
               previous run of the same bisection attempt (for that is what it 
               was) could explain a fair few spurious timeouts.
13:53 <Diziet> I have a patch queued up to change cs-bisection-step to use pbm 
               instead.
13:54 <Diziet> (And also fix a performance problem with some of the sql in 
               sg-report-job-history)

I'm waiting for the C600/TG3 bugfix to make it into osstest production
(because that's holding up the deployment of 4-6 test hosts) and then
I will probably force push those.

I'll email those patches later today

Ian.

[OSSTEST PATCH 0/6] Urgent performance improvements

[Ian Jackson]

This series contains these two important performance improvements:

 1/6 cs-bisection-step: Use pbm tools, not graphicsmagick/imagemagick
 2/6 sg-report-job-history: Avoid full runvars table scan (!)

The use of `convert' seems to have suffered from a severe performance
problem which made the osstest VM so slow that we had quite a few
spurious test failures.

sg-report-job-history runs with important locks held, so that (for
example) flight construction is blocked.

I'm also bundling some improvementse to sg-report-flight's HTML
generator.  They are in 4 small patches for ease of review.

None of the patches in this series would be tested in any effective
way by the osstest self-push-gate.  Subject to acks, I therefore
intend to force push them right after the oustanding queue goes in.

Ian.

[OSSTEST PATCH 1/6] cs-bisection-step: Use pbm tools, not graphicsmagick/imagemagick

[Ian Jackson]

Graphicsmagick / imagemagick have very poor performance with images
with large pixel sizes.  The bisector can generate some very large
images.

In an example I have seen, a 21595x21048 png, occupying only 2.6Mby of
disk space.  An invocation of `convert' to resize this was using 3Gby
of RAM and lots of CPU.  Whereas, the pbm utilities can process this
with much less memory and a tiny fraction of the cpu time.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cs-bisection-step |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/cs-bisection-step b/cs-bisection-step
index d5f8cb8..c9c8bb6 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -992,8 +992,13 @@ END
         }
         1;
     }) {
-        system_checked("convert -resize $c{BisectionRevisonGraphSize}".
-                       " $graphfile.png $graphfile.mini.png");
+	my $gsize = $c{BisectionRevisonGraphSize};
+        system_checked("pngtopnm <$graphfile.png".
+		       " | pnmscale -xysize ".
+		       ($gsize =~ m/^(\d+)x(\d+)$/ ? "$1 $2" :
+			$gsize =~ m/^(\d+)$/ ? "$1 $1" :
+			die "$gsize ?").
+		       " | pnmtopng >$graphfile.mini.png");
         print HTML <<END or die $!;
 <h2>Revision graph overview</h2>
 <img src="$href.mini.png">
-- 
1.7.10.4

[OSSTEST PATCH 2/6] sg-report-job-history: Avoid full runvars table scan (!)

[Ian Jackson]

sg-report-job-history wnats to know the potential names of runvars
relating to hosts.  To do this it tries to find a list of distinct
runvar names which exist in the flights it's processing.

However, it fails to limit the runvar query appropriately, and as a
result postgresql must scan almost the complete runvars table to
produce an answer.  This is very slow if the table is bigger than the
database server's RAM.

Fix this by limiting the runvars table query to relevant flights.

Specifically:

 * Break the `100' from the LIMIT clause on the flights search
   into a local variable $limit.
 * Break the bulk of the flights search sql statement text into
   a local variable $fromstuff.
 * In the runvars statement, add a condition on flights which uses
   LIMIT and OFFSET, based on results of the the flights query.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-job-history |   17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index e7052a3..d59717f 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -163,12 +163,17 @@ sub processjobbranch ($$) {
 END
         push @params, $bra;
     }
-    my $flightsq= $dbh_tests->prepare(<<END);
-        SELECT *
+    my $limit= 100;
+    my $offset= $limit-1;
+
+    my $fromstuff= <<END;
           FROM jobs JOIN flights USING (flight)
          WHERE ($cond)
       ORDER BY flight DESC
-         LIMIT 100
+END
+    my $flightsq= $dbh_tests->prepare(<<END);
+        SELECT * $fromstuff
+         LIMIT $limit
 END
     $flightsq->execute(@params);
 
@@ -177,9 +182,13 @@ END
 	 FROM runvars
 	 JOIN flights USING (flight)
 	WHERE ($cond)
+	  AND flight >= (
+	      SELECT flight $fromstuff
+	      LIMIT 1 OFFSET $offset
+          )
      ORDER BY name;
 END
-    $hostsq->execute(@params);
+    $hostsq->execute(@params, @params);
     my @hostvarcols;
     while (my ($hostvar) = $hostsq->fetchrow_array()) {
 	next unless $hostvar =~ m/(^|_)host$/;
-- 
1.7.10.4

Re: [OSSTEST PATCH 2/6] sg-report-job-history: Avoid full runvars table scan (!)

[Ian Campbell]

On Fri, 2015-05-01 at 16:02 +0100, Ian Jackson wrote:
> sg-report-job-history wnats to know the potential names of runvars

Typo "wnats".

[OSSTEST PATCH 3/6] sg-report-flight: Break colour of out $cell_html->() results

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-flight |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/sg-report-flight b/sg-report-flight
index 18a5afa..29b5ef6 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -970,8 +970,8 @@ END
 		? ('#ff8888',400)
 		: ('#ffff00',250);
 	}
-	my $h = " bgcolor=\"$bgc\">";
-	$h .= "<a href=\"".encode_entities($s->{job}).
+	my $ch = "bgcolor=\"$bgc\"";
+	my $h = "<a href=\"".encode_entities($s->{job}).
 	    "/$s->{stepno}.".encode_entities($s->{step}).".log\">";
 	#print H "[$s->{stepno}] ";
 	$h .= encode_entities($sum_core);
@@ -980,7 +980,7 @@ END
 	    $h .= " <a href=\"../$sum_flight/\">".
 		displayflightnum($sum_flight)."</a>";
 	}
-	return ($h,$priority);
+	return ($h,$priority,$ch);
     };
 
     foreach my $col (@cols) {
@@ -992,7 +992,7 @@ END
 	    next if $this[1] < $worst[1];
 	    @worst=@this;
 	}
-        print H "<td ",$worst[0],"</td>";
+	print H "<td ",$worst[2],">",$worst[0],"</td>";
     }
     print H "\n";
     # Create a gutter row after the headers.
@@ -1038,8 +1038,8 @@ END
             if ($ei) {
                 $deferred_cells->(1);
                 my $s= $ei->{Step};
-		my ($h,$priority) = $cell_html->($ei);
-                print H "<td", $h, "</td>";
+		my ($h,$priority,$ch) = $cell_html->($ei);
+                print H "<td ",$ch,">",$h,"</td>";
             } else {
                 $ndeferred++;
             }
-- 
1.7.10.4

[OSSTEST PATCH 4/6] sg-report-flight: Better choice of column-head cell

[Ian Jackson]

We are more interested the first worst failure.  But if everything
passed, then the last pass.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-flight |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sg-report-flight b/sg-report-flight
index 29b5ef6..7c9e52e 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -949,7 +949,7 @@ END
     }
     print H "\n";
     print H "<tr><th>";
-    print H "Worst/last";
+    print H "Last pass / First worst fail";
     print H "</th>";
 
     my $cell_html = sub {
@@ -990,6 +990,7 @@ END
 	    next unless $ei;
 	    my @this = $cell_html->($ei);
 	    next if $this[1] < $worst[1];
+	    next if $this[1] == $worst[1] && $ei->{Step}{status} ne 'pass';
 	    @worst=@this;
 	}
 	print H "<td ",$worst[2],">",$worst[0],"</td>";
-- 
1.7.10.4

Re: [OSSTEST PATCH 4/6] sg-report-flight: Better choice of column-head cell

[Ian Campbell]

On Fri, 2015-05-01 at 16:02 +0100, Ian Jackson wrote:
> We are more interested the first worst failure.  But if everything
                       ^in ?

> passed, then the last pass.

> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  sg-report-flight |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/sg-report-flight b/sg-report-flight
> index 29b5ef6..7c9e52e 100755
> --- a/sg-report-flight
> +++ b/sg-report-flight
> @@ -949,7 +949,7 @@ END
>      }
>      print H "\n";
>      print H "<tr><th>";
> -    print H "Worst/last";
> +    print H "Last pass / First worst fail";
>      print H "</th>";
>  
>      my $cell_html = sub {
> @@ -990,6 +990,7 @@ END
>  	    next unless $ei;
>  	    my @this = $cell_html->($ei);
>  	    next if $this[1] < $worst[1];
> +	    next if $this[1] == $worst[1] && $ei->{Step}{status} ne 'pass';
>  	    @worst=@this;
>  	}
>  	print H "<td ",$worst[2],">",$worst[0],"</td>";

[OSSTEST PATCH 5/6] sg-report-flight: Include actual status in column-head cells

[Ian Jackson]

Adjust the $cell_html anonymous sub to include the actual step status
unless it is told not to (by passing it a new $core_only argument).

Change the main body call site to pass this option.  As a result the
column headings now include the step status.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-flight |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/sg-report-flight b/sg-report-flight
index 7c9e52e..016f8d7 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -953,7 +953,7 @@ END
     print H "</th>";
 
     my $cell_html = sub {
-	my ($ei) = @_;
+	my ($ei, $core_only) = @_;
 	# => ($h, $priority);
 	my $s= $ei->{Step};
 	my $sum_core= $ei->{SummaryCore};
@@ -965,6 +965,7 @@ END
 	    $sum_core= $s->{status};
 	    ($bgc,$priority)= html_status2_colour_priority($s->{status});
 	} else {
+	    $sum_core= "$s->{status} $sum_core" unless $core_only;
 	    ($bgc,$priority) =
 		$ei->{Blocker} eq 'regression'
 		? ('#ff8888',400)
@@ -1039,7 +1040,7 @@ END
             if ($ei) {
                 $deferred_cells->(1);
                 my $s= $ei->{Step};
-		my ($h,$priority,$ch) = $cell_html->($ei);
+		my ($h,$priority,$ch) = $cell_html->($ei, 1);
                 print H "<td ",$ch,">",$h,"</td>";
             } else {
                 $ndeferred++;
-- 
1.7.10.4

[OSSTEST PATCH 6/6] sg-report-flight: Report stepno and testid of first worst fail

[Ian Jackson]

This makes reading the scoreboard considerably easier.

We abuse the local variable @worst slight, pushing the extra info we
are going to print onto the end of it.

We also have to defer printing the cells, because we compute the cell
to duplicate in column order but we have to output them in row order.
For symmetry we accumulate both rows rather than only the second row.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-flight |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/sg-report-flight b/sg-report-flight
index 016f8d7..80777af 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -948,7 +948,7 @@ END
         print H "</th>";
     }
     print H "\n";
-    print H "<tr><th>";
+    print H "<tr><th rowspan=2>";
     print H "Last pass / First worst fail";
     print H "</th>";
 
@@ -984,6 +984,8 @@ END
 	return ($h,$priority,$ch);
     };
 
+    my @worstrow1;
+    my @worstrow2;
     foreach my $col (@cols) {
 	my @worst=(' bgcolor="#444444">',0);
 	foreach my $rowix (0..$#{ $fi->{GridRows} }) {
@@ -993,9 +995,13 @@ END
 	    next if $this[1] < $worst[1];
 	    next if $this[1] == $worst[1] && $ei->{Step}{status} ne 'pass';
 	    @worst=@this;
+	    push @worst,
+	        encode_entities("$ei->{Step}{stepno}. $ei->{Step}{testid}");
 	}
-	print H "<td ",$worst[2],">",$worst[0],"</td>";
+	push @worstrow1, "<td ",$worst[2],">",$worst[3],"</td>";
+	push @worstrow2, "<td ",$worst[2],">",$worst[0],"</td>";
     }
+    print H "\n", @worstrow1, "\n<tr>\n", @worstrow2;
     print H "\n";
     # Create a gutter row after the headers.
     print H "<tr><td>&nbsp;</td></tr>\n";
-- 
1.7.10.4

Re: [OSSTEST PATCH 6/6] sg-report-flight: Report stepno and testid of first worst fail

[Ian Campbell]

On Fri, 2015-05-01 at 16:02 +0100, Ian Jackson wrote:
> This makes reading the scoreboard considerably easier.
> 
> We abuse the local variable @worst slight, pushing the extra info we
                                          ^ly

> @@ -993,9 +995,13 @@ END
>  	    next if $this[1] < $worst[1];
>  	    next if $this[1] == $worst[1] && $ei->{Step}{status} ne 'pass';
>  	    @worst=@this;
> +	    push @worst,
> +	        encode_entities("$ei->{Step}{stepno}. $ei->{Step}{testid}");
>  	}
> -	print H "<td ",$worst[2],">",$worst[0],"</td>";
> +	push @worstrow1, "<td ",$worst[2],">",$worst[3],"</td>";
> +	push @worstrow2, "<td ",$worst[2],">",$worst[0],"</td>";
>      }
> +    print H "\n", @worstrow1, "\n<tr>\n", @worstrow2;

Does this need a </tr> in the middle?

I think there was likely no </tr> in the old code either, and I suppose
web standards being what they are <tr>...<tr>...<tr>... probably works.

That being the case you could reasonably defer shaving this yakk for
another time.

>      print H "\n";
>      # Create a gutter row after the headers.
>      print H "<tr><td>&nbsp;</td></tr>\n";

Re: [OSSTEST PATCH 6/6] sg-report-flight: Report stepno and testid of first worst fail

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 6/6] sg-report-flight: Report stepno and testid of first worst fail"):
> On Fri, 2015-05-01 at 16:02 +0100, Ian Jackson wrote:
> > This makes reading the scoreboard considerably easier.
> > 
> > We abuse the local variable @worst slight, pushing the extra info we
>                                           ^ly

I have fixed this and the other commit message bugs.

> > +    print H "\n", @worstrow1, "\n<tr>\n", @worstrow2;
> 
> Does this need a </tr> in the middle?

Lack of </tr> is OK in trad HTML because it can be inferred.

> That being the case you could reasonably defer shaving this yakk for
> another time.

:-).

Ian.

Re: [OSSTEST PATCH 0/6] Urgent performance improvements

[Ian Campbell]

On Fri, 2015-05-01 at 16:02 +0100, Ian Jackson wrote:
> Subject to acks,

I spotted a few minor typos and had one not so minor (but still not very
important) query on the last one. So:
        Acked-by: Ian Campbell <ian.campbell@citrix.com>