Re: next/master baseline: 368 runs, 20 regressions (next-20200825)

public inbox for kernelci@lists.linux.dev
 help / color / mirror / Atom feed

* Re: next/master baseline: 368 runs, 20 regressions (next-20200825)
       [not found] <5f450985.1c69fb81.d0e74.7478@mx.google.com>
@ 2020-08-25 17:58 ` Kevin Hilman
  2020-09-01 10:57   ` Guillaume Tucker
       [not found]   ` <1630A3B4244BEF6E.8371@groups.io>
  0 siblings, 2 replies; 5+ messages in thread
From: Kevin Hilman @ 2020-08-25 17:58 UTC (permalink / raw)
  To: kernelci; +Cc: kernelci-results

[ -linux-next, +kernelci dev ]

"kernelci.org bot" <bot@kernelci.org> writes:

> next/master baseline: 368 runs, 20 regressions (next-20200825)
>
> Regressions Summary
> -------------------
>
> platform              | arch  | lab           | compiler | defconfig                    | results
> ----------------------+-------+---------------+----------+------------------------------+--------
> at91-sama5d4_xplained | arm   | lab-baylibre  | gcc-8    | sama5_defconfig              | 0/1    
> bcm2837-rpi-3-b       | arm64 | lab-baylibre  | gcc-8    | defconfig                    | 3/4    
> exynos5422-odroidxu3  | arm   | lab-collabora | gcc-8    | exynos_defconfig             | 0/1    
> exynos5422-odroidxu3  | arm   | lab-collabora | gcc-8    | multi_v7_defc...G_ARM_LPAE=y | 0/1    
> exynos5422-odroidxu3  | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 0/1    
> exynos5422-odroidxu3  | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 0/1    
> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 4/5    
> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 4/5    
> omap4-panda           | arm   | lab-collabora | gcc-8    | omap2plus_defconfig          | 0/1    
> rk3399-gru-kevin      | arm64 | lab-collabora | clang-10 | defconfig                    | 83/88  
> rk3399-gru-kevin      | arm64 | lab-collabora | gcc-8    | defconfig+CON...OMIZE_BASE=y | 83/88  
> rk3399-gru-kevin      | arm64 | lab-collabora | gcc-8    | defconfig                    | 0/1    
> sun7i-a20-cubieboard2 | arm   | lab-baylibre  | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 0/1    
> sun7i-a20-cubieboard2 | arm   | lab-baylibre  | gcc-8    | multi_v7_defconfig           | 0/1    
> sun7i-a20-cubieboard2 | arm   | lab-clabbe    | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 0/1    
> sun7i-a20-cubieboard2 | arm   | lab-clabbe    | gcc-8    | multi_v7_defconfig           | 0/1    
> sun7i-a20-cubieboard2 | arm   | lab-baylibre  | gcc-8    | sunxi_defconfig              | 0/1    
> sun7i-a20-cubieboard2 | arm   | lab-clabbe    | gcc-8    | sunxi_defconfig              | 0/1    
>
>   Details:  https://kernelci.org/test/job/next/branch/master/kernel/next-20200825/plan/baseline/
>
>   Test:     baseline
>   Tree:     next
>   Branch:   master
>   Describe: next-20200825
>   URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
>   SHA:      3a00d3dfd4b68b208ecd5405e676d06c8ad6bb63 
>
>
> Test Regressions
> ---------------- 
>
>
> platform              | arch  | lab           | compiler | defconfig                    | results
> ----------------------+-------+---------------+----------+------------------------------+--------
> at91-sama5d4_xplained | arm   | lab-baylibre  | gcc-8    | sama5_defconfig              | 0/1    
>
>   Details:     https://kernelci.org/test/plan/id/5f44d1f560dfd32ca09fb433
>
>   Results:     0 PASS, 1 FAIL, 0 SKIP
>   Full config: sama5_defconfig
>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/sama5_defconfig/gcc-8/lab-baylibre/baseline-at91-sama5d4_xplained.txt
>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/sama5_defconfig/gcc-8/lab-baylibre/baseline-at91-sama5d4_xplained.html
>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>
>   * baseline.login: https://kernelci.org/test/case/id/5f44d1f560dfd32ca09fb434
>       failing since 118 days (last pass: next-20200424, first fail: next-20200428)  
>
>
> platform              | arch  | lab           | compiler | defconfig                    | results
> ----------------------+-------+---------------+----------+------------------------------+--------
> bcm2837-rpi-3-b       | arm64 | lab-baylibre  | gcc-8    | defconfig                    | 3/4    
>
>   Details:     https://kernelci.org/test/plan/id/5f44d354974cc7a9749fb42b
>
>   Results:     3 PASS, 1 FAIL, 0 SKIP
>   Full config: defconfig
>   Compiler:    gcc-8 (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0)
>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.txt
>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.html
>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/arm64/baseline/rootfs.cpio.gz 
>
>   * baseline.dmesg.crit: https://kernelci.org/test/case/id/5f44d354974cc7a9749fb42f
>       new failure (last pass: next-20200824)
>       1 lines
>
>     2020-08-25 08:59:02.167000  Connected to bcm2837-rpi-3-b console [channel connected] (~$quit to exit)
>     2020-08-25 08:59:02.167000  (user:khilman) is already connected
>     2020-08-25 08:59:18.764000  .
>     2020-08-25 08:59:18.765000  
>     2020-08-25 08:59:18.765000  U-Boot 2018.11 (Dec 04 2018 - 10:54:32 -0800)
>     2020-08-25 08:59:18.765000  
>     2020-08-25 08:59:18.765000  DRAM:  948 MiB
>     2020-08-25 08:59:18.780000  RPI 3 Model B (0xa02082)
>     2020-08-25 08:59:18.866000  MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
>     2020-08-25 08:59:18.899000  Loading Environment from FAT... *** Warning - bad CRC, using default environment
>     ... (379 line(s) more)

Shouldn't we show the last few lines of the log instead of the first
few lines?

> platform              | arch  | lab           | compiler | defconfig                    | results
> ----------------------+-------+---------------+----------+------------------------------+--------
> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 4/5    
>
>   Details:     https://kernelci.org/test/plan/id/5f44d4448493d6f73c9fb42b
>
>   Results:     4 PASS, 1 FAIL, 0 SKIP
>   Full config: multi_v7_defconfig+CONFIG_SMP=n
>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.txt
>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.html
>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>
>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d4448493d6f73c9fb431
>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>       60 lines
>
>     2020-08-25 09:05:03.232000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c802
>     2020-08-25 09:05:03.238000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c803
>     2020-08-25 09:05:03.244000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c804
>     2020-08-25 09:05:03.249000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c805
>     2020-08-25 09:05:03.255000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c806
>     2020-08-25 09:05:03.261000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c807
>     2020-08-25 09:05:03.267000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c808
>     2020-08-25 09:05:03.273000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c809
>     2020-08-25 09:05:03.278000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80a
>     2020-08-25 09:05:03.284000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80b
>     ... (49 line(s) more)

... similar here ...

> platform              | arch  | lab           | compiler | defconfig                    | results
> ----------------------+-------+---------------+----------+------------------------------+--------
> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 4/5    
>
>   Details:     https://kernelci.org/test/plan/id/5f44d6605371b404fc9fb42b
>
>   Results:     4 PASS, 1 FAIL, 0 SKIP
>   Full config: multi_v7_defconfig
>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.txt
>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.html
>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>
>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>       60 lines
>
>     2020-08-25 09:14:02.674000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c802
>     2020-08-25 09:14:02.680000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c803
>     2020-08-25 09:14:02.686000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c804
>     2020-08-25 09:14:02.692000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c805
>     2020-08-25 09:14:02.698000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c806
>     2020-08-25 09:14:02.704000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c807
>     2020-08-25 09:14:02.710000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c808
>     2020-08-25 09:14:02.716000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c809
>     2020-08-25 09:14:02.722000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80a
>     2020-08-25 09:14:02.728000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80b
>     ... (49 line(s) more)
>       
... and here.

It's not always true, but I think typically the last part of the log
will be more interesting to spotting an issue.

Kevin

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: next/master baseline: 368 runs, 20 regressions (next-20200825)
  2020-08-25 17:58 ` next/master baseline: 368 runs, 20 regressions (next-20200825) Kevin Hilman
@ 2020-09-01 10:57   ` Guillaume Tucker
  2020-09-01 13:22     ` Guenter Roeck
       [not found]   ` <1630A3B4244BEF6E.8371@groups.io>
  1 sibling, 1 reply; 5+ messages in thread
From: Guillaume Tucker @ 2020-09-01 10:57 UTC (permalink / raw)
  To: khilman; +Cc: kernelci, kernelci-results

On 25/08/2020 18:58, Kevin Hilman wrote:
> [ -linux-next, +kernelci dev ]
> 
> "kernelci.org bot" <bot@kernelci.org> writes:
> 
>> next/master baseline: 368 runs, 20 regressions (next-20200825)

[...]

>> platform              | arch  | lab           | compiler | defconfig                    | results
>> ----------------------+-------+---------------+----------+------------------------------+--------
>> bcm2837-rpi-3-b       | arm64 | lab-baylibre  | gcc-8    | defconfig                    | 3/4    
>>
>>   Details:     https://kernelci.org/test/plan/id/5f44d354974cc7a9749fb42b
>>
>>   Results:     3 PASS, 1 FAIL, 0 SKIP
>>   Full config: defconfig
>>   Compiler:    gcc-8 (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0)
>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.txt
>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.html
>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/arm64/baseline/rootfs.cpio.gz 
>>
>>   * baseline.dmesg.crit: https://kernelci.org/test/case/id/5f44d354974cc7a9749fb42f
>>       new failure (last pass: next-20200824)
>>       1 lines
>>
>>     2020-08-25 08:59:02.167000  Connected to bcm2837-rpi-3-b console [channel connected] (~$quit to exit)
>>     2020-08-25 08:59:02.167000  (user:khilman) is already connected
>>     2020-08-25 08:59:18.764000  .
>>     2020-08-25 08:59:18.765000  
>>     2020-08-25 08:59:18.765000  U-Boot 2018.11 (Dec 04 2018 - 10:54:32 -0800)
>>     2020-08-25 08:59:18.765000  
>>     2020-08-25 08:59:18.765000  DRAM:  948 MiB
>>     2020-08-25 08:59:18.780000  RPI 3 Model B (0xa02082)
>>     2020-08-25 08:59:18.866000  MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
>>     2020-08-25 08:59:18.899000  Loading Environment from FAT... *** Warning - bad CRC, using default environment
>>     ... (379 line(s) more)
> 
> Shouldn't we show the last few lines of the log instead of the first
> few lines?

In this case, it's the span of log lines that was wrong.  The log
lines should have started after login, not at the bootloader
stage.  The "critical" error is:

[   17.957677] hwmon hwmon1: Undervoltage detected!

This is a recurring error and I believe previous runs showed it
correctly, so it's worth investigating why the start line was not
set correctly in this case.

By the way, it only seems to be happening in lab-baylibre and I
wonder if it could have anything to do with an unstable power
supply.

>> platform              | arch  | lab           | compiler | defconfig                    | results
>> ----------------------+-------+---------------+----------+------------------------------+--------
>> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 4/5    
>>
>>   Details:     https://kernelci.org/test/plan/id/5f44d4448493d6f73c9fb42b
>>
>>   Results:     4 PASS, 1 FAIL, 0 SKIP
>>   Full config: multi_v7_defconfig+CONFIG_SMP=n
>>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.txt
>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.html
>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>>
>>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d4448493d6f73c9fb431
>>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>>       60 lines
>>
>>     2020-08-25 09:05:03.232000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c802
>>     2020-08-25 09:05:03.238000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c803
>>     2020-08-25 09:05:03.244000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c804
>>     2020-08-25 09:05:03.249000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c805
>>     2020-08-25 09:05:03.255000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c806
>>     2020-08-25 09:05:03.261000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c807
>>     2020-08-25 09:05:03.267000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c808
>>     2020-08-25 09:05:03.273000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c809
>>     2020-08-25 09:05:03.278000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80a
>>     2020-08-25 09:05:03.284000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80b
>>     ... (49 line(s) more)
> 
> ... similar here ...

In this case, all the lines are actually the same.

>> platform              | arch  | lab           | compiler | defconfig                    | results
>> ----------------------+-------+---------------+----------+------------------------------+--------
>> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 4/5    
>>
>>   Details:     https://kernelci.org/test/plan/id/5f44d6605371b404fc9fb42b
>>
>>   Results:     4 PASS, 1 FAIL, 0 SKIP
>>   Full config: multi_v7_defconfig
>>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.txt
>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.html
>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>>
>>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
>>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>>       60 lines
>>
>>     2020-08-25 09:14:02.674000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c802
>>     2020-08-25 09:14:02.680000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c803
>>     2020-08-25 09:14:02.686000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c804
>>     2020-08-25 09:14:02.692000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c805
>>     2020-08-25 09:14:02.698000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c806
>>     2020-08-25 09:14:02.704000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c807
>>     2020-08-25 09:14:02.710000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c808
>>     2020-08-25 09:14:02.716000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c809
>>     2020-08-25 09:14:02.722000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80a
>>     2020-08-25 09:14:02.728000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80b
>>     ... (49 line(s) more)
>>       
> ... and here.
> 
> It's not always true, but I think typically the last part of the log
> will be more interesting to spotting an issue.

In the examples above, the top part of the log lines was
completely relevant.  For things like Oops and other kernel
errors, the issue is often shown first and then there's a stack
dump.  Ultimately, each test suite will be doing things
differently.  So it's really hard to tell whether the last lines
are more likely to be useful than the first ones.  There's also a
link to the details on the frontend where all the lines are
visible:

  https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431

Maybe we could increase the limit and show up to say, 16 lines
instead of 10 but then when there are 17 or more show the first
and last 3 lines?  It's all kind of heuristics at this point.  It
might also be something to tune on a per-test-suite basis when we
have kci_email and some YAML configuration.

Guillaume

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: next/master baseline: 368 runs, 20 regressions (next-20200825)
  2020-09-01 10:57   ` Guillaume Tucker
@ 2020-09-01 13:22     ` Guenter Roeck
  2020-09-01 13:42       ` Guillaume Tucker
  0 siblings, 1 reply; 5+ messages in thread
From: Guenter Roeck @ 2020-09-01 13:22 UTC (permalink / raw)
  To: kernelci, Guillaume Tucker; +Cc: Kevin Hilman, kernelci-results

On Tue, Sep 1, 2020 at 3:57 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:
>
> On 25/08/2020 18:58, Kevin Hilman wrote:
> > [ -linux-next, +kernelci dev ]
> >
> > "kernelci.org bot" <bot@kernelci.org> writes:
> >
> >> next/master baseline: 368 runs, 20 regressions (next-20200825)
>
> [...]
>
> >> platform              | arch  | lab           | compiler | defconfig                    | results
> >> ----------------------+-------+---------------+----------+------------------------------+--------
> >> bcm2837-rpi-3-b       | arm64 | lab-baylibre  | gcc-8    | defconfig                    | 3/4
> >>
> >>   Details:     https://kernelci.org/test/plan/id/5f44d354974cc7a9749fb42b
> >>
> >>   Results:     3 PASS, 1 FAIL, 0 SKIP
> >>   Full config: defconfig
> >>   Compiler:    gcc-8 (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0)
> >>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.txt
> >>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.html
> >>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/arm64/baseline/rootfs.cpio.gz
> >>
> >>   * baseline.dmesg.crit: https://kernelci.org/test/case/id/5f44d354974cc7a9749fb42f
> >>       new failure (last pass: next-20200824)
> >>       1 lines
> >>
> >>     2020-08-25 08:59:02.167000  Connected to bcm2837-rpi-3-b console [channel connected] (~$quit to exit)
> >>     2020-08-25 08:59:02.167000  (user:khilman) is already connected
> >>     2020-08-25 08:59:18.764000  .
> >>     2020-08-25 08:59:18.765000
> >>     2020-08-25 08:59:18.765000  U-Boot 2018.11 (Dec 04 2018 - 10:54:32 -0800)
> >>     2020-08-25 08:59:18.765000
> >>     2020-08-25 08:59:18.765000  DRAM:  948 MiB
> >>     2020-08-25 08:59:18.780000  RPI 3 Model B (0xa02082)
> >>     2020-08-25 08:59:18.866000  MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
> >>     2020-08-25 08:59:18.899000  Loading Environment from FAT... *** Warning - bad CRC, using default environment
> >>     ... (379 line(s) more)
> >
> > Shouldn't we show the last few lines of the log instead of the first
> > few lines?
>
> In this case, it's the span of log lines that was wrong.  The log
> lines should have started after login, not at the bootloader
> stage.  The "critical" error is:
>
> [   17.957677] hwmon hwmon1: Undervoltage detected!
>
That is just a log message, though, that triggers a sysfs_notify call
to in0_lcrit_alarm. By itself it doesn't result in a crash/reboot.

Guenter

> This is a recurring error and I believe previous runs showed it
> correctly, so it's worth investigating why the start line was not
> set correctly in this case.
>
> By the way, it only seems to be happening in lab-baylibre and I
> wonder if it could have anything to do with an unstable power
> supply.
>
> >> platform              | arch  | lab           | compiler | defconfig                    | results
> >> ----------------------+-------+---------------+----------+------------------------------+--------
> >> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 4/5
> >>
> >>   Details:     https://kernelci.org/test/plan/id/5f44d4448493d6f73c9fb42b
> >>
> >>   Results:     4 PASS, 1 FAIL, 0 SKIP
> >>   Full config: multi_v7_defconfig+CONFIG_SMP=n
> >>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
> >>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.txt
> >>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.html
> >>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz
> >>
> >>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d4448493d6f73c9fb431
> >>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
> >>       60 lines
> >>
> >>     2020-08-25 09:05:03.232000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c802
> >>     2020-08-25 09:05:03.238000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c803
> >>     2020-08-25 09:05:03.244000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c804
> >>     2020-08-25 09:05:03.249000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c805
> >>     2020-08-25 09:05:03.255000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c806
> >>     2020-08-25 09:05:03.261000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c807
> >>     2020-08-25 09:05:03.267000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c808
> >>     2020-08-25 09:05:03.273000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c809
> >>     2020-08-25 09:05:03.278000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80a
> >>     2020-08-25 09:05:03.284000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80b
> >>     ... (49 line(s) more)
> >
> > ... similar here ...
>
> In this case, all the lines are actually the same.
>
> >> platform              | arch  | lab           | compiler | defconfig                    | results
> >> ----------------------+-------+---------------+----------+------------------------------+--------
> >> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 4/5
> >>
> >>   Details:     https://kernelci.org/test/plan/id/5f44d6605371b404fc9fb42b
> >>
> >>   Results:     4 PASS, 1 FAIL, 0 SKIP
> >>   Full config: multi_v7_defconfig
> >>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
> >>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.txt
> >>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.html
> >>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz
> >>
> >>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
> >>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
> >>       60 lines
> >>
> >>     2020-08-25 09:14:02.674000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c802
> >>     2020-08-25 09:14:02.680000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c803
> >>     2020-08-25 09:14:02.686000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c804
> >>     2020-08-25 09:14:02.692000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c805
> >>     2020-08-25 09:14:02.698000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c806
> >>     2020-08-25 09:14:02.704000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c807
> >>     2020-08-25 09:14:02.710000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c808
> >>     2020-08-25 09:14:02.716000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c809
> >>     2020-08-25 09:14:02.722000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80a
> >>     2020-08-25 09:14:02.728000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80b
> >>     ... (49 line(s) more)
> >>
> > ... and here.
> >
> > It's not always true, but I think typically the last part of the log
> > will be more interesting to spotting an issue.
>
> In the examples above, the top part of the log lines was
> completely relevant.  For things like Oops and other kernel
> errors, the issue is often shown first and then there's a stack
> dump.  Ultimately, each test suite will be doing things
> differently.  So it's really hard to tell whether the last lines
> are more likely to be useful than the first ones.  There's also a
> link to the details on the frontend where all the lines are
> visible:
>
>   https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
>
> Maybe we could increase the limit and show up to say, 16 lines
> instead of 10 but then when there are 17 or more show the first
> and last 3 lines?  It's all kind of heuristics at this point.  It
> might also be something to tune on a per-test-suite basis when we
> have kci_email and some YAML configuration.
>
> Guillaume
>
> 
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: next/master baseline: 368 runs, 20 regressions (next-20200825)
  2020-09-01 13:22     ` Guenter Roeck
@ 2020-09-01 13:42       ` Guillaume Tucker
  0 siblings, 0 replies; 5+ messages in thread
From: Guillaume Tucker @ 2020-09-01 13:42 UTC (permalink / raw)
  To: Guenter Roeck; +Cc: kernelci, Kevin Hilman, kernelci-results

On 01/09/2020 14:22, Guenter Roeck wrote:
> On Tue, Sep 1, 2020 at 3:57 AM Guillaume Tucker
> <guillaume.tucker@collabora.com> wrote:
>>
>> On 25/08/2020 18:58, Kevin Hilman wrote:
>>> [ -linux-next, +kernelci dev ]
>>>
>>> "kernelci.org bot" <bot@kernelci.org> writes:
>>>
>>>> next/master baseline: 368 runs, 20 regressions (next-20200825)
>>
>> [...]
>>
>>>> platform              | arch  | lab           | compiler | defconfig                    | results
>>>> ----------------------+-------+---------------+----------+------------------------------+--------
>>>> bcm2837-rpi-3-b       | arm64 | lab-baylibre  | gcc-8    | defconfig                    | 3/4
>>>>
>>>>   Details:     https://kernelci.org/test/plan/id/5f44d354974cc7a9749fb42b
>>>>
>>>>   Results:     3 PASS, 1 FAIL, 0 SKIP
>>>>   Full config: defconfig
>>>>   Compiler:    gcc-8 (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0)
>>>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.txt
>>>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.html
>>>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/arm64/baseline/rootfs.cpio.gz
>>>>
>>>>   * baseline.dmesg.crit: https://kernelci.org/test/case/id/5f44d354974cc7a9749fb42f
>>>>       new failure (last pass: next-20200824)
>>>>       1 lines
>>>>
>>>>     2020-08-25 08:59:02.167000  Connected to bcm2837-rpi-3-b console [channel connected] (~$quit to exit)
>>>>     2020-08-25 08:59:02.167000  (user:khilman) is already connected
>>>>     2020-08-25 08:59:18.764000  .
>>>>     2020-08-25 08:59:18.765000
>>>>     2020-08-25 08:59:18.765000  U-Boot 2018.11 (Dec 04 2018 - 10:54:32 -0800)
>>>>     2020-08-25 08:59:18.765000
>>>>     2020-08-25 08:59:18.765000  DRAM:  948 MiB
>>>>     2020-08-25 08:59:18.780000  RPI 3 Model B (0xa02082)
>>>>     2020-08-25 08:59:18.866000  MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
>>>>     2020-08-25 08:59:18.899000  Loading Environment from FAT... *** Warning - bad CRC, using default environment
>>>>     ... (379 line(s) more)
>>>
>>> Shouldn't we show the last few lines of the log instead of the first
>>> few lines?
>>
>> In this case, it's the span of log lines that was wrong.  The log
>> lines should have started after login, not at the bootloader
>> stage.  The "critical" error is:
>>
>> [   17.957677] hwmon hwmon1: Undervoltage detected!
>>
> That is just a log message, though, that triggers a sysfs_notify call
> to in0_lcrit_alarm. By itself it doesn't result in a crash/reboot.

That's right, the test case is "baseline.dmesg.crit" which looks
for "critical" kernel error messages.  Early boot crashes are
normally caught by the "baseline.login" test case.

New kernel errors at the "emerg", "crit" or "alert" levels are
detected as regressions and reported as such.  The hwmon issue in
this case is an intermittent one, so it regularly comes back as a
regression.

Best wishes,
Guillaume

>> This is a recurring error and I believe previous runs showed it
>> correctly, so it's worth investigating why the start line was not
>> set correctly in this case.
>>
>> By the way, it only seems to be happening in lab-baylibre and I
>> wonder if it could have anything to do with an unstable power
>> supply.
>>
>>>> platform              | arch  | lab           | compiler | defconfig                    | results
>>>> ----------------------+-------+---------------+----------+------------------------------+--------
>>>> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 4/5
>>>>
>>>>   Details:     https://kernelci.org/test/plan/id/5f44d4448493d6f73c9fb42b
>>>>
>>>>   Results:     4 PASS, 1 FAIL, 0 SKIP
>>>>   Full config: multi_v7_defconfig+CONFIG_SMP=n
>>>>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>>>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.txt
>>>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.html
>>>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz
>>>>
>>>>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d4448493d6f73c9fb431
>>>>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>>>>       60 lines
>>>>
>>>>     2020-08-25 09:05:03.232000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c802
>>>>     2020-08-25 09:05:03.238000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c803
>>>>     2020-08-25 09:05:03.244000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c804
>>>>     2020-08-25 09:05:03.249000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c805
>>>>     2020-08-25 09:05:03.255000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c806
>>>>     2020-08-25 09:05:03.261000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c807
>>>>     2020-08-25 09:05:03.267000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c808
>>>>     2020-08-25 09:05:03.273000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c809
>>>>     2020-08-25 09:05:03.278000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80a
>>>>     2020-08-25 09:05:03.284000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80b
>>>>     ... (49 line(s) more)
>>>
>>> ... similar here ...
>>
>> In this case, all the lines are actually the same.
>>
>>>> platform              | arch  | lab           | compiler | defconfig                    | results
>>>> ----------------------+-------+---------------+----------+------------------------------+--------
>>>> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 4/5
>>>>
>>>>   Details:     https://kernelci.org/test/plan/id/5f44d6605371b404fc9fb42b
>>>>
>>>>   Results:     4 PASS, 1 FAIL, 0 SKIP
>>>>   Full config: multi_v7_defconfig
>>>>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>>>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.txt
>>>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.html
>>>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz
>>>>
>>>>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
>>>>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>>>>       60 lines
>>>>
>>>>     2020-08-25 09:14:02.674000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c802
>>>>     2020-08-25 09:14:02.680000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c803
>>>>     2020-08-25 09:14:02.686000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c804
>>>>     2020-08-25 09:14:02.692000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c805
>>>>     2020-08-25 09:14:02.698000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c806
>>>>     2020-08-25 09:14:02.704000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c807
>>>>     2020-08-25 09:14:02.710000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c808
>>>>     2020-08-25 09:14:02.716000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c809
>>>>     2020-08-25 09:14:02.722000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80a
>>>>     2020-08-25 09:14:02.728000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80b
>>>>     ... (49 line(s) more)
>>>>
>>> ... and here.
>>>
>>> It's not always true, but I think typically the last part of the log
>>> will be more interesting to spotting an issue.
>>
>> In the examples above, the top part of the log lines was
>> completely relevant.  For things like Oops and other kernel
>> errors, the issue is often shown first and then there's a stack
>> dump.  Ultimately, each test suite will be doing things
>> differently.  So it's really hard to tell whether the last lines
>> are more likely to be useful than the first ones.  There's also a
>> link to the details on the frontend where all the lines are
>> visible:
>>
>>   https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
>>
>> Maybe we could increase the limit and show up to say, 16 lines
>> instead of 10 but then when there are 17 or more show the first
>> and last 3 lines?  It's all kind of heuristics at this point.  It
>> might also be something to tune on a per-test-suite basis when we
>> have kci_email and some YAML configuration.
>>
>> Guillaume
>>
>> 
>>


^ permalink raw reply	[flat|nested] 5+ messages in thread

[parent not found: <1630A3B4244BEF6E.8371@groups.io>]

* Re: next/master baseline: 368 runs, 20 regressions (next-20200825)
       [not found]   ` <1630A3B4244BEF6E.8371@groups.io>
@ 2020-09-01 12:41     ` Guillaume Tucker
  0 siblings, 0 replies; 5+ messages in thread
From: Guillaume Tucker @ 2020-09-01 12:41 UTC (permalink / raw)
  To: khilman; +Cc: kernelci, kernelci-results

On 01/09/2020 11:57, Guillaume Tucker wrote:
> On 25/08/2020 18:58, Kevin Hilman wrote:
>> [ -linux-next, +kernelci dev ]
>>
>> "kernelci.org bot" <bot@kernelci.org> writes:
>>
>>> next/master baseline: 368 runs, 20 regressions (next-20200825)
> 
> [...]
> 
>>> platform              | arch  | lab           | compiler | defconfig                    | results
>>> ----------------------+-------+---------------+----------+------------------------------+--------
>>> bcm2837-rpi-3-b       | arm64 | lab-baylibre  | gcc-8    | defconfig                    | 3/4    
>>>
>>>   Details:     https://kernelci.org/test/plan/id/5f44d354974cc7a9749fb42b
>>>
>>>   Results:     3 PASS, 1 FAIL, 0 SKIP
>>>   Full config: defconfig
>>>   Compiler:    gcc-8 (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0)
>>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.txt
>>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm64/defconfig/gcc-8/lab-baylibre/baseline-bcm2837-rpi-3-b.html
>>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/arm64/baseline/rootfs.cpio.gz 
>>>
>>>   * baseline.dmesg.crit: https://kernelci.org/test/case/id/5f44d354974cc7a9749fb42f
>>>       new failure (last pass: next-20200824)
>>>       1 lines
>>>
>>>     2020-08-25 08:59:02.167000  Connected to bcm2837-rpi-3-b console [channel connected] (~$quit to exit)
>>>     2020-08-25 08:59:02.167000  (user:khilman) is already connected
>>>     2020-08-25 08:59:18.764000  .
>>>     2020-08-25 08:59:18.765000  
>>>     2020-08-25 08:59:18.765000  U-Boot 2018.11 (Dec 04 2018 - 10:54:32 -0800)
>>>     2020-08-25 08:59:18.765000  
>>>     2020-08-25 08:59:18.765000  DRAM:  948 MiB
>>>     2020-08-25 08:59:18.780000  RPI 3 Model B (0xa02082)
>>>     2020-08-25 08:59:18.866000  MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
>>>     2020-08-25 08:59:18.899000  Loading Environment from FAT... *** Warning - bad CRC, using default environment
>>>     ... (379 line(s) more)
>>
>> Shouldn't we show the last few lines of the log instead of the first
>> few lines?
> 
> In this case, it's the span of log lines that was wrong.  The log
> lines should have started after login, not at the bootloader
> stage.  The "critical" error is:
> 
> [   17.957677] hwmon hwmon1: Undervoltage detected!
> 
> This is a recurring error and I believe previous runs showed it
> correctly, so it's worth investigating why the start line was not
> set correctly in this case.

In fact, the baseline.login test case doesn't appear here:

  https://kernelci.org/test/plan/id/5f456c171cabe7cb219fb493/

So that would explain why all the boot log was included in this
test case.  I guess that's the known issue of the unreliable LAVA
auto-login step result.

Guillaume

> By the way, it only seems to be happening in lab-baylibre and I
> wonder if it could have anything to do with an unstable power
> supply.
> 
>>> platform              | arch  | lab           | compiler | defconfig                    | results
>>> ----------------------+-------+---------------+----------+------------------------------+--------
>>> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defc...CONFIG_SMP=n | 4/5    
>>>
>>>   Details:     https://kernelci.org/test/plan/id/5f44d4448493d6f73c9fb42b
>>>
>>>   Results:     4 PASS, 1 FAIL, 0 SKIP
>>>   Full config: multi_v7_defconfig+CONFIG_SMP=n
>>>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.txt
>>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-8/lab-collabora/baseline-omap4-panda.html
>>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>>>
>>>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d4448493d6f73c9fb431
>>>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>>>       60 lines
>>>
>>>     2020-08-25 09:05:03.232000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c802
>>>     2020-08-25 09:05:03.238000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c803
>>>     2020-08-25 09:05:03.244000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c804
>>>     2020-08-25 09:05:03.249000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c805
>>>     2020-08-25 09:05:03.255000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c806
>>>     2020-08-25 09:05:03.261000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c807
>>>     2020-08-25 09:05:03.267000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c808
>>>     2020-08-25 09:05:03.273000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c809
>>>     2020-08-25 09:05:03.278000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80a
>>>     2020-08-25 09:05:03.284000  kern  :alert : BUG: Bad page state in process swapper  pfn:9c80b
>>>     ... (49 line(s) more)
>>
>> ... similar here ...
> 
> In this case, all the lines are actually the same.
> 
>>> platform              | arch  | lab           | compiler | defconfig                    | results
>>> ----------------------+-------+---------------+----------+------------------------------+--------
>>> omap4-panda           | arm   | lab-collabora | gcc-8    | multi_v7_defconfig           | 4/5    
>>>
>>>   Details:     https://kernelci.org/test/plan/id/5f44d6605371b404fc9fb42b
>>>
>>>   Results:     4 PASS, 1 FAIL, 0 SKIP
>>>   Full config: multi_v7_defconfig
>>>   Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
>>>   Plain log:   https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.txt
>>>   HTML log:    https://storage.kernelci.org//next/master/next-20200825/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-omap4-panda.html
>>>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 
>>>
>>>   * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
>>>       failing since 40 days (last pass: next-20200706, first fail: next-20200715)
>>>       60 lines
>>>
>>>     2020-08-25 09:14:02.674000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c802
>>>     2020-08-25 09:14:02.680000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c803
>>>     2020-08-25 09:14:02.686000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c804
>>>     2020-08-25 09:14:02.692000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c805
>>>     2020-08-25 09:14:02.698000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c806
>>>     2020-08-25 09:14:02.704000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c807
>>>     2020-08-25 09:14:02.710000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c808
>>>     2020-08-25 09:14:02.716000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c809
>>>     2020-08-25 09:14:02.722000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80a
>>>     2020-08-25 09:14:02.728000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c80b
>>>     ... (49 line(s) more)
>>>       
>> ... and here.
>>
>> It's not always true, but I think typically the last part of the log
>> will be more interesting to spotting an issue.
> 
> In the examples above, the top part of the log lines was
> completely relevant.  For things like Oops and other kernel
> errors, the issue is often shown first and then there's a stack
> dump.  Ultimately, each test suite will be doing things
> differently.  So it's really hard to tell whether the last lines
> are more likely to be useful than the first ones.  There's also a
> link to the details on the frontend where all the lines are
> visible:
> 
>   https://kernelci.org/test/case/id/5f44d6605371b404fc9fb431
> 
> Maybe we could increase the limit and show up to say, 16 lines
> instead of 10 but then when there are 17 or more show the first
> and last 3 lines?  It's all kind of heuristics at this point.  It
> might also be something to tune on a per-test-suite basis when we
> have kci_email and some YAML configuration.
> 
> Guillaume
> 
> 
> 


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2020-09-01 13:42 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <5f450985.1c69fb81.d0e74.7478@mx.google.com>
2020-08-25 17:58 ` next/master baseline: 368 runs, 20 regressions (next-20200825) Kevin Hilman
2020-09-01 10:57   ` Guillaume Tucker
2020-09-01 13:22     ` Guenter Roeck
2020-09-01 13:42       ` Guillaume Tucker
     [not found]   ` <1630A3B4244BEF6E.8371@groups.io>
2020-09-01 12:41     ` Guillaume Tucker

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox