Re: ❌ FAIL: Test report for kernel 5.13.0-rc4 (arm-next, 8124c8a6)

From: Will Deacon <will@kernel.org>
To: Ard Biesheuvel <ardb@kernel.org>
Cc: CKI Project <cki-project@redhat.com>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Anshuman Khandual <anshuman.khandual@arm.com>,
	Mark Rutland <mark.rutland@arm.com>,
	Marc Zyngier <maz@kernel.org>,
	skt-results-master@redhat.com,
	Catalin Marinas <catalin.marinas@arm.com>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>,
	Memory Management <mm-qe@redhat.com>,
	Jan Stancek <jstancek@redhat.com>,
	Fendy Tjahjadi <ftjahjad@redhat.com>,
	Jeff Bastian <jbastian@redhat.com>
Subject: Re: ❌ FAIL: Test report for kernel 5.13.0-rc4 (arm-next, 8124c8a6)
Date: Wed, 2 Jun 2021 11:51:36 +0100	[thread overview]
Message-ID: <20210602105135.GC30593@willie-the-truck> (raw)
In-Reply-To: <CAMj1kXENsUScpTg294HDiQUUXuUBz438hysEM9M4N-Jcq+q2fA@mail.gmail.com>

On Wed, Jun 02, 2021 at 12:40:07PM +0200, Ard Biesheuvel wrote:
> On Wed, 2 Jun 2021 at 12:12, Will Deacon <will@kernel.org> wrote:
> >
> > Hi all,
> >
> > On Wed, Jun 02, 2021 at 01:35:01AM -0000, CKI Project wrote:
> > > We ran automated tests on a recent commit from this kernel tree:
> > >
> > >        Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
> > >             Commit: 8124c8a6b353 - Linux 5.13-rc4
> >
> > As above, this is a run against plain -rc4 in preparation for testing the
> > 5.14 queue on top to detect any regressions.
> >
> > > The results of these automated tests are provided below.
> > >
> > >     Overall result: FAILED (see details below)
> > >              Merge: OK
> > >            Compile: OK
> > >              Tests: FAILED
> > >
> > > All kernel binaries, config files, and logs are available for download here:
> > >
> > >   https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/06/01/313156257
> > >
> > > One or more kernel tests failed:
> > >
> > >     aarch64:
> > >      ❌ LTP
> >
> > The two tests failing here are 'statx04' and 'hugeshmat04':
> >
> > https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/06/01/313156257/build_aarch64_redhat%3A1310052388/tests/LTP/10079826_aarch64_1_resultoutputfile.log
> >
> > (grep for FAIL). I've not had a chance to dig into them yet.
> >
> > >      stress: stress-ng
> >
> > This explodes pretty badly. Some CPUs detect RCU stalls when trying to use
> > the EFI "efi_read_time" service, which eventually fails but soon after we
> > explode trying to access memory which I think is mapped by
> > acpi_os_ioremap(), so it looks like the f/w might be the culprit here. Is
> > the "HPE Apollo 70" machine known to have bad EFI firmware?
> >
> > https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/06/01/313156257/build_aarch64_redhat%3A1310052388/tests/stress_stress_ng/10079827_aarch64_2_dmesg.log
> >
> > (scroll to the end for the fireworks)
> >
> 
> Wow that looks pretty horrible. I take it this tree has your MAIR changes?

Nope, this is just vanilla -rc4! I'm trying to get a "known good" base
before I throw all the new things at it :)

> Would be useful to have a log with efi=debug, to see what the EFI
> memory map looks like.

Veronika -- please could you help us with that?

> Note that efi_read_time() kicks a separate worker thread to perform
> the actual call into the firmware, so the CPU stalling on
> efi_read_time() is kind of expected and not as interesting as the one
> doing the actual call.

Fair enough, although the _dev_err() in the backtrace implies the call
failed so I became suspicious.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel