All of lore.kernel.org
 help / color / mirror / Atom feed
From: "kernelci.org bot" <bot@kernelci.org>
To: stable@vger.kernel.org, kernel-build-reports@lists.linaro.org,
	kernelci-results@groups.io
Subject: stable-rc/linux-4.19.y baseline: 160 runs, 28 regressions (v4.19.272-62-g70e81ae774a8)
Date: Mon, 13 Feb 2023 13:39:53 -0800 (PST)	[thread overview]
Message-ID: <63eaae29.170a0220.a4396.a1bc@mx.google.com> (raw)

stable-rc/linux-4.19.y baseline: 160 runs, 28 regressions (v4.19.272-62-g70e81ae774a8)

Regressions Summary
-------------------

platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
at91sam9g20ek              | arm   | lab-broonie   | gcc-10   | multi_v5_defconfig         | 1          
beaglebone-black           | arm   | lab-broonie   | gcc-10   | omap2plus_defconfig        | 1          
cubietruck                 | arm   | lab-baylibre  | gcc-10   | multi_v7_defconfig         | 1          
qemu_arm64-virt-gicv2      | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv2      | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv2      | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv2      | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv2      | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv2      | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          
r8a7743-iwg20d-q7          | arm   | lab-cip       | gcc-10   | shmobile_defconfig         | 1          

  Details:  https://kernelci.org/test/job/stable-rc/branch/linux-4.19.y/kernel/v4.19.272-62-g70e81ae774a8/plan/baseline/

  Test:     baseline
  Tree:     stable-rc
  Branch:   linux-4.19.y
  Describe: v4.19.272-62-g70e81ae774a8
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
  SHA:      70e81ae774a8c3b06cfb2cabc1051261062fa059 


Test Regressions
---------------- 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
at91sam9g20ek              | arm   | lab-broonie   | gcc-10   | multi_v5_defconfig         | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7bc5d1fcfad4de8c8664

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v5_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/multi_v5_defconfig/gcc-10/lab-broonie/baseline-at91sam9g20ek.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/multi_v5_defconfig/gcc-10/lab-broonie/baseline-at91sam9g20ek.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/armel/rootfs.cpio.gz 


  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63ea7bc5d1fcfad4de8c866d
        failing since 21 days (last pass: v4.19.269-516-gf9f90bbcdb21, first fail: v4.19.269-516-gde8db55b044b)

    2023-02-13T18:04:34.774300  + set +x
    2023-02-13T18:04:34.779321  <8><LAVA_SIGNAL_ENDRUN 0_dmesg 4729_1.5.2.4.1>
    2023-02-13T18:04:34.892809  / # #
    2023-02-13T18:04:34.995826  export SHELL=/bin/sh
    2023-02-13T18:04:34.996587  #
    2023-02-13T18:04:35.098533  / # export SHELL=/bin/sh. /lava-4729/environment
    2023-02-13T18:04:35.099280  
    2023-02-13T18:04:35.201258  / # /lava-4729/bin/lava-test-runner /lava-4729/1
    2023-02-13T18:04:35.202548  
    2023-02-13T18:04:35.208965  / # /lava-4729/bin/lava-test-runner /lava-4729/1 
    ... (12 line(s) more)  
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
beaglebone-black           | arm   | lab-broonie   | gcc-10   | omap2plus_defconfig        | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7c608b552190078c8641

  Results:     4 PASS, 2 FAIL, 1 SKIP
  Full config: omap2plus_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/omap2plus_defconfig/gcc-10/lab-broonie/baseline-beaglebone-black.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/omap2plus_defconfig/gcc-10/lab-broonie/baseline-beaglebone-black.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/armel/rootfs.cpio.gz 


  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63ea7c608b552190078c864a
        new failure (last pass: v4.19.272)

    2023-02-13T18:07:05.787086  + set +x<8>[   16.463733] <LAVA_SIGNAL_ENDRUN 0_dmesg 4771_1.5.2.4.1>
    2023-02-13T18:07:05.787535  
    2023-02-13T18:07:05.899609  / # #
    2023-02-13T18:07:06.002550  export SHELL=/bin/sh
    2023-02-13T18:07:06.003270  #
    2023-02-13T18:07:06.105966  / # export SHELL=/bin/sh. /lava-4771/environment
    2023-02-13T18:07:06.106685  
    2023-02-13T18:07:06.209154  / # . /lava-4771/environment/lava-4771/bin/lava-test-runner /lava-4771/1
    2023-02-13T18:07:06.210401  
    2023-02-13T18:07:06.215088  / # /lava-4771/bin/lava-test-runner /lava-4771/1 
    ... (12 line(s) more)  
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
cubietruck                 | arm   | lab-baylibre  | gcc-10   | multi_v7_defconfig         | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7c17d0614a95858c8669

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-cubietruck.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-cubietruck.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/armel/rootfs.cpio.gz 


  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63ea7c17d0614a95858c8672
        failing since 27 days (last pass: v4.19.268-50-gbf741d1d7e6d, first fail: v4.19.269-522-gc75d2b5524ab)

    2023-02-13T18:05:52.983072  <8>[    7.330048] <LAVA_SIGNAL_ENDRUN 0_dmesg 3332991_1.5.2.4.1>
    2023-02-13T18:05:53.095135  / # #
    2023-02-13T18:05:53.198725  export SHELL=/bin/sh
    2023-02-13T18:05:53.199827  #
    2023-02-13T18:05:53.302448  / # export SHELL=/bin/sh. /lava-3332991/environment
    2023-02-13T18:05:53.303629  
    2023-02-13T18:05:53.405880  / # . /lava-3332991/environment/lava-3332991/bin/lava-test-runner /lava-3332991/1
    2023-02-13T18:05:53.407665  
    2023-02-13T18:05:53.412971  / # /lava-3332991/bin/lava-test-runner /lava-3332991/1
    2023-02-13T18:05:53.487224  + export 'TESTRUN_ID=1_bootrr' 
    ... (11 line(s) more)  
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e50c8e8209bae8c8631

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e50c8e8209bae8c8632
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-191-gab9c8d4442969) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e63c8e8209bae8c864e

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e63c8e8209bae8c864f
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-215-g4373264025937) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e1ad0f50c03ce8c863d

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e1ad0f50c03ce8c863e
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-191-gab9c8d4442969) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7f8175a98179b48c863d

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7f8175a98179b48c863e
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-215-g4373264025937) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e0f3a36dd00df8c865d

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e0f3a36dd00df8c865e
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-191-gab9c8d4442969) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e38e8c20eeaf58c8635

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e38e8c20eeaf58c8636
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-215-g4373264025937) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e4d3616a525d58c8630

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e4d3616a525d58c8631
        failing since 168 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.256-37-g34c3cf0160a8) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e61c8e8209bae8c8648

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e61c8e8209bae8c8649
        failing since 174 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-288-g9901269a16d1) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e18d43888c1498c863a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e18d43888c1498c863b
        failing since 168 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.256-37-g34c3cf0160a8) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e40e8c20eeaf58c8643

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e40e8c20eeaf58c8644
        failing since 174 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-288-g9901269a16d1) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e0f162b07202b8c864f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e0f162b07202b8c8650
        failing since 168 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.256-37-g34c3cf0160a8) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e3708cb096bea8c8631

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e3708cb096bea8c8632
        failing since 174 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-288-g9901269a16d1) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e4e3616a525d58c8635

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e4e3616a525d58c8636
        failing since 194 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.254) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e62c8e8209bae8c864b

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e62c8e8209bae8c864c
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-191-gab9c8d4442969) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e19d0f50c03ce8c863a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e19d0f50c03ce8c863b
        failing since 194 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.254) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7f807c04342d7c8c8641

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7f807c04342d7c8c8642
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-191-gab9c8d4442969) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e1061263e59468c8635

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e1061263e59468c8636
        failing since 194 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.254) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e3908cb096bea8c8637

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e3908cb096bea8c8638
        failing since 182 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-191-gab9c8d4442969) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e51c8e8209bae8c8634

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e51c8e8209bae8c8635
        failing since 180 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-baylibre  | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e64c8e8209bae8c8654

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e64c8e8209bae8c8655
        failing since 174 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-288-g9901269a16d1) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie   | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e1bd0f50c03ce8c8640

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e1bd0f50c03ce8c8641
        failing since 180 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie   | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7f8275a98179b48c8642

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7f8275a98179b48c8643
        failing since 174 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-288-g9901269a16d1) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-collabora | gcc-10   | defconfig                  | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e1261263e59468c8643

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e1261263e59468c8644
        failing since 180 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-collabora | gcc-10   | defconfig+arm64-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63ea7e4b08cb096bea8c867e

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea7e4b08cb096bea8c867f
        failing since 174 days (last pass: v4.19.230-41-g73351b9c55d9, first fail: v4.19.255-288-g9901269a16d1) 
 


platform                   | arch  | lab           | compiler | defconfig                  | regressions
---------------------------+-------+---------------+----------+----------------------------+------------
r8a7743-iwg20d-q7          | arm   | lab-cip       | gcc-10   | shmobile_defconfig         | 1          

  Details:     https://kernelci.org/test/plan/id/63ea79ed7db0c784548c862f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: shmobile_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/shmobile_defconfig/gcc-10/lab-cip/baseline-r8a7743-iwg20d-q7.txt
  HTML log:    https://storage.kernelci.org//stable-rc/linux-4.19.y/v4.19.272-62-g70e81ae774a8/arm/shmobile_defconfig/gcc-10/lab-cip/baseline-r8a7743-iwg20d-q7.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230211.0/armel/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/63ea79ed7db0c784548c8630
        failing since 6 days (last pass: v4.19.271-121-g282d872fb7c4, first fail: v4.19.271-122-g3e6dcdab73ec) 
  

                 reply	other threads:[~2023-02-13 21:40 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=63eaae29.170a0220.a4396.a1bc@mx.google.com \
    --to=bot@kernelci.org \
    --cc=kernel-build-reports@lists.linaro.org \
    --cc=kernelci-results@groups.io \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.