From: Will Deacon <will@kernel.org>
To: Bjorn Andersson <bjorn.andersson@linaro.org>
Cc: Mark Brown <broonie@kernel.org>, Andy Gross <agross@kernel.org>,
kernel-build-reports@lists.linaro.org,
linux-next@vger.kernel.org, linux-arm-msm@vger.kernel.org,
linux-arm-kernel@lists.infradead.org,
Catalin Marinas <catalin.marinas@arm.com>
Subject: Re: next/master boot: 257 boots: 8 failed, 237 passed with 8 offline, 2 untried/unknown, 2 conflicts (next-20191028)
Date: Mon, 28 Oct 2019 20:14:18 +0000 [thread overview]
Message-ID: <20191028201418.GA8317@willie-the-truck> (raw)
In-Reply-To: <20191028200219.GS571@minitux>
On Mon, Oct 28, 2019 at 01:02:19PM -0700, Bjorn Andersson wrote:
> On Mon 28 Oct 12:11 PDT 2019, Mark Brown wrote:
>
> > On Mon, Oct 28, 2019 at 11:40:19AM -0700, Bjorn Andersson wrote:
> > > On Mon 28 Oct 10:48 PDT 2019, Mark Brown wrote:
> > > > On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
> >
> > > > Today's -next (anf Friday's) fails to boot on db820c:
> >
> > > > > defconfig:
> > > > > gcc-8:
> > > > > apq8096-db820c: 1 failed lab
> >
> > > > It looks like it deadlocks somewhere, the last things in the log are a
> > > > failure to start ufshcd-qcom and then an RCU stall some time later:
> >
> > > db820c has been failing intermittently for a while now, it seems that
> > > booting with kpti enabled causes something to go wrong. There are
> > > nothing strange in the kernel logs and ftrace seems to indicate that all
> > > the CPUs are idling nicely.
> >
> > Oh dear. Adding Catalin and Will. Is it definitely KPTI that's
> > triggering stuff? It did turn up some bugs on other systems, though
> > it's a bit strange it's only manifesting in KernelCI...
>
> I did a test recently where I booted my db820c 100 times with kpti=yes
> and 100 times with kpti=no on the kernel command line, and the result
> was 90% failure to reach console vs 0%. Going back and looking at the
> logs for the 10% indicated that the boot CPU was fine, but I had stalls
> reported on other CPUs.
>
> In an effort to rule out driver bugs I reduced the DT to CPUs, the core
> clocks, gic, timers and serial driver, and I still saw the problem.
>
> I have not looked at this with jtag and hence do not know what secure
> world is doing.
Hmm. Is this a recent thing? Neither kpti nor the snapdragon 820 are
particular new. Might be worth checking that CONFIG_QCOM_FALKOR_ERRATUM_1003
is enabled and getting patched in at runtime -- we had hardware issues
during kpti development with this CPU.
Will
WARNING: multiple messages have this Message-ID (diff)
From: Will Deacon <will@kernel.org>
To: Bjorn Andersson <bjorn.andersson@linaro.org>
Cc: kernel-build-reports@lists.linaro.org,
linux-arm-msm@vger.kernel.org, Mark Brown <broonie@kernel.org>,
Andy Gross <agross@kernel.org>,
linux-next@vger.kernel.org,
Catalin Marinas <catalin.marinas@arm.com>,
linux-arm-kernel@lists.infradead.org
Subject: Re: next/master boot: 257 boots: 8 failed, 237 passed with 8 offline, 2 untried/unknown, 2 conflicts (next-20191028)
Date: Mon, 28 Oct 2019 20:14:18 +0000 [thread overview]
Message-ID: <20191028201418.GA8317@willie-the-truck> (raw)
In-Reply-To: <20191028200219.GS571@minitux>
On Mon, Oct 28, 2019 at 01:02:19PM -0700, Bjorn Andersson wrote:
> On Mon 28 Oct 12:11 PDT 2019, Mark Brown wrote:
>
> > On Mon, Oct 28, 2019 at 11:40:19AM -0700, Bjorn Andersson wrote:
> > > On Mon 28 Oct 10:48 PDT 2019, Mark Brown wrote:
> > > > On Mon, Oct 28, 2019 at 08:03:08AM -0700, kernelci.org bot wrote:
> >
> > > > Today's -next (anf Friday's) fails to boot on db820c:
> >
> > > > > defconfig:
> > > > > gcc-8:
> > > > > apq8096-db820c: 1 failed lab
> >
> > > > It looks like it deadlocks somewhere, the last things in the log are a
> > > > failure to start ufshcd-qcom and then an RCU stall some time later:
> >
> > > db820c has been failing intermittently for a while now, it seems that
> > > booting with kpti enabled causes something to go wrong. There are
> > > nothing strange in the kernel logs and ftrace seems to indicate that all
> > > the CPUs are idling nicely.
> >
> > Oh dear. Adding Catalin and Will. Is it definitely KPTI that's
> > triggering stuff? It did turn up some bugs on other systems, though
> > it's a bit strange it's only manifesting in KernelCI...
>
> I did a test recently where I booted my db820c 100 times with kpti=yes
> and 100 times with kpti=no on the kernel command line, and the result
> was 90% failure to reach console vs 0%. Going back and looking at the
> logs for the 10% indicated that the boot CPU was fine, but I had stalls
> reported on other CPUs.
>
> In an effort to rule out driver bugs I reduced the DT to CPUs, the core
> clocks, gic, timers and serial driver, and I still saw the problem.
>
> I have not looked at this with jtag and hence do not know what secure
> world is doing.
Hmm. Is this a recent thing? Neither kpti nor the snapdragon 820 are
particular new. Might be worth checking that CONFIG_QCOM_FALKOR_ERRATUM_1003
is enabled and getting patched in at runtime -- we had hardware issues
during kpti development with this CPU.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2019-10-28 20:14 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <5db7032c.1c69fb81.888b0.b521@mx.google.com>
2019-10-28 17:48 ` next/master boot: 257 boots: 8 failed, 237 passed with 8 offline, 2 untried/unknown, 2 conflicts (next-20191028) Mark Brown
2019-10-28 17:48 ` Mark Brown
2019-10-28 18:40 ` Bjorn Andersson
2019-10-28 18:40 ` Bjorn Andersson
2019-10-28 19:11 ` Mark Brown
2019-10-28 19:11 ` Mark Brown
2019-10-28 20:02 ` Bjorn Andersson
2019-10-28 20:02 ` Bjorn Andersson
2019-10-28 20:14 ` Will Deacon [this message]
2019-10-28 20:14 ` Will Deacon
2019-10-28 20:23 ` Mark Brown
2019-10-28 20:23 ` Mark Brown
2019-10-28 15:03 kernelci.org bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191028201418.GA8317@willie-the-truck \
--to=will@kernel.org \
--cc=agross@kernel.org \
--cc=bjorn.andersson@linaro.org \
--cc=broonie@kernel.org \
--cc=catalin.marinas@arm.com \
--cc=kernel-build-reports@lists.linaro.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-arm-msm@vger.kernel.org \
--cc=linux-next@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.