arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

[kernelci.org bot]

arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

Regressions Summary
-------------------

platform                   | arch  | lab         | compiler | defconfig | regressions
---------------------------+-------+-------------+----------+-----------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-broonie | gcc-12   | defconfig | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie | gcc-12   | defconfig | 1          
qemu_arm64-virt-gicv3      | arm64 | lab-broonie | gcc-12   | defconfig | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie | gcc-12   | defconfig | 1          

  Details:  https://kernelci.org/test/job/arm64/branch/for-kernelci/kernel/v6.12-rc3-67-gb9e20acb61ba/plan/baseline/

  Test:     baseline
  Tree:     arm64
  Branch:   for-kernelci
  Describe: v6.12-rc3-67-gb9e20acb61ba
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
  SHA:      b9e20acb61ba6342ea4c89b587883209defc755f 


Test Regressions
---------------- 


platform                   | arch  | lab         | compiler | defconfig | regressions
---------------------------+-------+-------------+----------+-----------+------------
qemu_arm64-virt-gicv2      | arm64 | lab-broonie | gcc-12   | defconfig | 1          

  Details:     https://kernelci.org/test/plan/id/671002a4ff09627193c86863

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
  Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/671002a4ff09627193c86864
        new failure (last pass: v6.12-rc3-58-g3d363ba041e1) 
 


platform                   | arch  | lab         | compiler | defconfig | regressions
---------------------------+-------+-------------+----------+-----------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie | gcc-12   | defconfig | 1          

  Details:     https://kernelci.org/test/plan/id/6710028f7bd86327ebc86856

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
  Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/6710028f7bd86327ebc86857
        new failure (last pass: v6.12-rc3-58-g3d363ba041e1) 
 


platform                   | arch  | lab         | compiler | defconfig | regressions
---------------------------+-------+-------------+----------+-----------+------------
qemu_arm64-virt-gicv3      | arm64 | lab-broonie | gcc-12   | defconfig | 1          

  Details:     https://kernelci.org/test/plan/id/6710027ae7cc9d3225c86857

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
  Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/6710027ae7cc9d3225c86858
        new failure (last pass: v6.12-rc3-58-g3d363ba041e1) 
 


platform                   | arch  | lab         | compiler | defconfig | regressions
---------------------------+-------+-------------+----------+-----------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie | gcc-12   | defconfig | 1          

  Details:     https://kernelci.org/test/plan/id/671002a3ff09627193c8685d

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
  Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 


  * baseline.login: https://kernelci.org/test/case/id/671002a3ff09627193c8685e
        new failure (last pass: v6.12-rc3-58-g3d363ba041e1) 
  

Re: arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

[Catalin Marinas]

On Wed, Oct 16, 2024 at 11:59:02AM -0700, kernelci.org bot wrote:
> Test Regressions
> ---------------- 
> 
> 
> platform                   | arch  | lab         | compiler | defconfig | regressions
> ---------------------------+-------+-------------+----------+-----------+------------
> qemu_arm64-virt-gicv2      | arm64 | lab-broonie | gcc-12   | defconfig | 1          
> 
>   Details:     https://kernelci.org/test/plan/id/671002a4ff09627193c86863

This page is still 404 (it happens quite a lot even with the logs in
these reports, not really helpful).

> 
>   Results:     0 PASS, 1 FAIL, 0 SKIP
>   Full config: defconfig
>   Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
>   Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.txt
>   HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.html
>   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

At least the boot log is shown (for this email, others point to missing
pages). However, it looks more like a Qemu bug, something about SME it
doesn't like:

ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
Bail out! ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached

I tried Qemu 7.2.0 and 8.2.4, both from Debian (stable and unstable).
Adding Peter, maybe he has any idea.

-- 
Catalin

Re: arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

[Peter Maydell]

On Thu, 17 Oct 2024 at 12:34, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Oct 16, 2024 at 11:59:02AM -0700, kernelci.org bot wrote:
> > Test Regressions
> > ----------------
> >
> >
> > platform                   | arch  | lab         | compiler | defconfig | regressions
> > ---------------------------+-------+-------------+----------+-----------+------------
> > qemu_arm64-virt-gicv2      | arm64 | lab-broonie | gcc-12   | defconfig | 1
> >
> >   Details:     https://kernelci.org/test/plan/id/671002a4ff09627193c86863
>
> This page is still 404 (it happens quite a lot even with the logs in
> these reports, not really helpful).
>
> >
> >   Results:     0 PASS, 1 FAIL, 0 SKIP
> >   Full config: defconfig
> >   Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
> >   Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.txt
> >   HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.html
> >   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz
>
> At least the boot log is shown (for this email, others point to missing
> pages). However, it looks more like a Qemu bug, something about SME it
> doesn't like:
>
> ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
> Bail out! ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
>
> I tried Qemu 7.2.0 and 8.2.4, both from Debian (stable and unstable).
> Adding Peter, maybe he has any idea.

Yeah, we shouldn't be hitting asserts (to state the obvious).

If you can provide the QEMU command line and the kernel/etc
files needed to run it, I can have a look.

-- PMM

Re: arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

[Catalin Marinas]

On Thu, Oct 17, 2024 at 12:50:42PM +0100, Peter Maydell wrote:
> On Thu, 17 Oct 2024 at 12:34, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Wed, Oct 16, 2024 at 11:59:02AM -0700, kernelci.org bot wrote:
> > > Test Regressions
> > > ----------------
> > >
> > >
> > > platform                   | arch  | lab         | compiler | defconfig | regressions
> > > ---------------------------+-------+-------------+----------+-----------+------------
> > > qemu_arm64-virt-gicv2      | arm64 | lab-broonie | gcc-12   | defconfig | 1
> > >
> > >   Details:     https://kernelci.org/test/plan/id/671002a4ff09627193c86863
> >
> > This page is still 404 (it happens quite a lot even with the logs in
> > these reports, not really helpful).
> >
> > >
> > >   Results:     0 PASS, 1 FAIL, 0 SKIP
> > >   Full config: defconfig
> > >   Compiler:    gcc-12 (aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0)
> > >   Plain log:   https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.txt
> > >   HTML log:    https://storage.kernelci.org//arm64/for-kernelci/v6.12-rc3-67-gb9e20acb61ba/arm64/defconfig/gcc-12/lab-broonie/baseline-qemu_arm64-virt-gicv2.html
> > >   Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz
> >
> > At least the boot log is shown (for this email, others point to missing
> > pages). However, it looks more like a Qemu bug, something about SME it
> > doesn't like:
> >
> > ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
> > Bail out! ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
> >
> > I tried Qemu 7.2.0 and 8.2.4, both from Debian (stable and unstable).
> > Adding Peter, maybe he has any idea.
> 
> Yeah, we shouldn't be hitting asserts (to state the obvious).
> 
> If you can provide the QEMU command line and the kernel/etc
> files needed to run it, I can have a look.

The kernel defconfig should be sufficient to trigger it. It must be the
for-kernelci branch from here:

https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/

My qemu command line is:

qemu-system-aarch64 \
	-machine virt,gic_version=3,mte=on \
	-cpu max \
	-smp 2 -m 2G \
	-kernel </path/to/Image> \
	-semihosting -nographic \
	-serial mon:stdio \
	-monitor tcp:0.0.0.0:4000,server,nowait \
	-netdev user,id=net0,hostfwd=tcp:0.0.0.0:4022-:22 \
	-device virtio-net-device,netdev=net0 \
	-object rng-random,filename=/dev/urandom,id=rng0 \
	-device virtio-rng-pci,rng=rng0 \
	-append 'console=ttyAMA0 earlycon=pl011,0x9000000 loglevel=9 ip=dhcp root=/dev/nfs rw nfsroot=<ip-addr>:/srv/nfs/debian-arm64,tcp,v4 kasan.mode=sync'

You don't need a filesystem, it fails before reaching that point. I
guess all it matters is '-cpu max'.

Thanks.

-- 
Catalin

Re: arm64/for-kernelci baseline: 14 runs, 4 regressions (v6.12-rc3-67-gb9e20acb61ba)

[Peter Maydell]

On Thu, 17 Oct 2024 at 13:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Oct 17, 2024 at 12:50:42PM +0100, Peter Maydell wrote:
> > On Thu, 17 Oct 2024 at 12:34, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > At least the boot log is shown (for this email, others point to missing
> > > pages). However, it looks more like a Qemu bug, something about SME it
> > > doesn't like:
> > >
> > > ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
> > > Bail out! ERROR:target/arm/internals.h:923:regime_is_user: code should not be reached
> > >
> > > I tried Qemu 7.2.0 and 8.2.4, both from Debian (stable and unstable).
> > > Adding Peter, maybe he has any idea.
> >
> > Yeah, we shouldn't be hitting asserts (to state the obvious).
> >
> > If you can provide the QEMU command line and the kernel/etc
> > files needed to run it, I can have a look.
>
> The kernel defconfig should be sufficient to trigger it. It must be the
> for-kernelci branch from here:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/
>
> My qemu command line is:
>
> qemu-system-aarch64 \
>         -machine virt,gic_version=3,mte=on \
>         -cpu max \
>         -smp 2 -m 2G \
>         -kernel </path/to/Image> \
>         -semihosting -nographic \
>         -serial mon:stdio \
>         -monitor tcp:0.0.0.0:4000,server,nowait \
>         -netdev user,id=net0,hostfwd=tcp:0.0.0.0:4022-:22 \
>         -device virtio-net-device,netdev=net0 \
>         -object rng-random,filename=/dev/urandom,id=rng0 \
>         -device virtio-rng-pci,rng=rng0 \
>         -append 'console=ttyAMA0 earlycon=pl011,0x9000000 loglevel=9 ip=dhcp root=/dev/nfs rw nfsroot=<ip-addr>:/srv/nfs/debian-arm64,tcp,v4 kasan.mode=sync'

Thanks. This turns out to be a bug in our FEAT_E0PD
emulation which you'll hit if the guest enables
E0PD by setting th TCR_ELx bits. The fix is
straightforward and I've just sent a patch to qemu-devel:

https://lore.kernel.org/qemu-devel/20241017172331.822587-1-peter.maydell@linaro.org/T/#u

-- PMM