Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Bjorn Helgaas]

[+cc Sherry, author of 610fa91d9863, Richard, Mani, linux-pci]

On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> Hello,
> 
> We are seeing a regression on Apalis iMX6 where the kernel doesn't boot in the
> newest v7.2-rc1 (it was working before, in v7.1). The device tree being used is the imx6q-apalis-eval.dtb. The kernel
> configuration used is the one shown below:
> 
>     https://gist.github.com/lcosta37/53efdb2fb6e6e0fc05437c7e53b47737
> 
> The kernel logs stop almost immediately as the board starts to boot, and I 
> don't notice any difference in the logs that points to the cause.
> 
> Is this known? We are seeing this behavior on all Apalis iMX6 modules, though
> we don't see it on Colibri iMX6, so it is not SoC-specific.

This regression is listed on
https://linux-regtracking.leemhuis.info/regzbot/mainline/ as being
caused by
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=master&id=610fa91d9863
("PCI: imx6: Assert PERST# before enabling regulators") but I can't
find any bisect or other report that specifically points to
610fa91d9863.

Can somebody confirm that reverting 610fa91d9863 fixes this
regression, or at least that 610fa91d9863^ works and 610fa91d9863
fails?

Bjorn

RE: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Hongxing Zhu (OSS)]

> -----Original Message-----
> From: Bjorn Helgaas <helgaas@kernel.org>
> Sent: Thursday, July 16, 2026 7:48 AM
> To: Leonardo Costa <leoreis.costa@gmail.com>
> Cc: robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org; Frank Li
> <frank.li@nxp.com>; s.hauer@pengutronix.de; kernel@pengutronix.de;
> festevam@gmail.com; leonardo.costa@toradex.com;
> devicetree@vger.kernel.org; imx@lists.linux.dev; linux-arm-
> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> regressions@lists.linux.dev; Sherry Sun <sherry.sun@nxp.com>; Manivannan
> Sadhasivam <mani@kernel.org>; Hongxing Zhu <hongxing.zhu@nxp.com>; linux-
> pci@vger.kernel.org
> Subject: Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots
>
> [+cc Sherry, author of 610fa91d9863, Richard, Mani, linux-pci]
>
> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> > Hello,
> >
> > We are seeing a regression on Apalis iMX6 where the kernel doesn't
> > boot in the newest v7.2-rc1 (it was working before, in v7.1). The
> > device tree being used is the imx6q-apalis-eval.dtb. The kernel configuration
> used is the one shown below:
> >
> >
> > https://gist/
> > .github.com%2Flcosta37%2F53efdb2fb6e6e0fc05437c7e53b47737&data=05%7
> C02
> > %7Chongxing.zhu%40nxp.com%7C3d58575d22384d61d15608dee2cb7471%7C
> 686ea1d
> >
> 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C639197560620871110%7CUnknow
> n%7CTWF
> >
> pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi
> IsI
> >
> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=KtTOaEVJI3aq6Bt
> V7Js
> > Y7r%2F9B5LY%2F%2BVj3xEMk592ghk%3D&reserved=0
> >
> > The kernel logs stop almost immediately as the board starts to boot,
> > and I don't notice any difference in the logs that points to the cause.
> >
> > Is this known? We are seeing this behavior on all Apalis iMX6 modules,
> > though we don't see it on Colibri iMX6, so it is not SoC-specific.
>
> This regression is listed on
> https://linux-/
> regtracking.leemhuis.info%2Fregzbot%2Fmainline%2F&data=05%7C02%7Chongx
> ing.zhu%40nxp.com%7C3d58575d22384d61d15608dee2cb7471%7C686ea1d3bc
> 2b4c6fa92cd99c5c301635%7C0%7C0%7C639197560620886501%7CUnknown%
> 7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXa
> W4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=K%2FBs
> xg56w4Jv4Phf%2B7i4CePTQA1M45xNH%2BxcrjuA4r0%3D&reserved=0 as being
> caused by
> https://git.kernel/.
> org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Fcommit
> %2F%3Fh%3Dmaster%26id%3D610fa91d9863&data=05%7C02%7Chongxing.zhu
> %40nxp.com%7C3d58575d22384d61d15608dee2cb7471%7C686ea1d3bc2b4c6f
> a92cd99c5c301635%7C0%7C0%7C639197560620895764%7CUnknown%7CTWF
> pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi
> IsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZT0p2CaXc8EJ
> YM2mlEAJegvmDqObaa%2FFiZtbCvM3PHw%3D&reserved=0
> ("PCI: imx6: Assert PERST# before enabling regulators") but I can't find any bisect
> or other report that specifically points to 610fa91d9863.
>
> Can somebody confirm that reverting 610fa91d9863 fixes this regression, or at
> least that 610fa91d9863^ works and 610fa91d9863 fails?
Sorry for this severe regression. I have posted a fix that has been tested on
my i.MX6Q SabreSD board:

https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/

Hi Leonardo,

Would you be able to test this patch on your affected boards? Your testing
would help expedite the merge process and ensure the fix resolves the boot
issue you're experiencing.

Best Regards
Richard Zhu
>
> Bjorn

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Bjorn Helgaas]

On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
> > On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> > > We are seeing a regression on Apalis iMX6 where the kernel
> > > doesn't boot in the newest v7.2-rc1 (it was working before, in
> > > v7.1). The device tree being used is the imx6q-apalis-eval.dtb.
> > > The kernel configuration
> > > used is the one shown below:
> ...

> > Can somebody confirm that reverting 610fa91d9863 fixes this
> > regression, or at least that 610fa91d9863^ works and 610fa91d9863
> > fails?
>
> Sorry for this severe regression. I have posted a fix that has been
> tested on my i.MX6Q SabreSD board:
> 
> https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/

My question was whether somebody had actually bisected to 610fa91d9863
("PCI: imx6: Assert PERST# before enabling regulators").  I don't see
anything specific on the mailing list, so I don't know how
610fa91d9863 got blamed for the regression.

https://linux-regtracking.leemhuis.info/regzbot/mainline/ lists this
as a regression caused by 610fa91d9863, which means we need (1) the
details to prove that 610fa91d9863 actually caused it and (2) a tested
revert or fix.

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Thorsten Leemhuis]

On 7/16/26 18:40, Bjorn Helgaas wrote:
> On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
>>> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
>>>> We are seeing a regression on Apalis iMX6 where the kernel
>>>> doesn't boot in the newest v7.2-rc1 (it was working before, in
>>>> v7.1). The device tree being used is the imx6q-apalis-eval.dtb.
>>>> The kernel configuration
>>>> used is the one shown below:
>> ...
> 
>>> Can somebody confirm that reverting 610fa91d9863 fixes this
>>> regression, or at least that 610fa91d9863^ works and 610fa91d9863
>>> fails?
>>
>> Sorry for this severe regression. I have posted a fix that has been
>> tested on my i.MX6Q SabreSD board:
>>
>> https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/
> 
> My question was whether somebody had actually bisected to 610fa91d9863
> ("PCI: imx6: Assert PERST# before enabling regulators").  I don't see
> anything specific on the mailing list, so I don't know how
> 610fa91d9863 got blamed for the regression.

That was me and might have been a stupid idea. I normally wouldn't have
added this, but a earlier message from Hongxing Zhu sounded a pretty
convincingly like something along the lines of  "this is the problem I
already have a fix for there" -- and then I took the commit-id from there.

Sorry.

Ciao, Thorsten

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Bjorn Helgaas]

On Thu, Jul 16, 2026 at 07:16:31PM +0200, Thorsten Leemhuis wrote:
> On 7/16/26 18:40, Bjorn Helgaas wrote:
> > On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
> >>> On Mon, Jun 29, 2026 at 11:34:32AM -0300, Leonardo Costa wrote:
> >>>> We are seeing a regression on Apalis iMX6 where the kernel
> >>>> doesn't boot in the newest v7.2-rc1 (it was working before, in
> >>>> v7.1). The device tree being used is the imx6q-apalis-eval.dtb.
> >>>> The kernel configuration
> >>>> used is the one shown below:
> >> ...
> > 
> >>> Can somebody confirm that reverting 610fa91d9863 fixes this
> >>> regression, or at least that 610fa91d9863^ works and 610fa91d9863
> >>> fails?
> >>
> >> Sorry for this severe regression. I have posted a fix that has been
> >> tested on my i.MX6Q SabreSD board:
> >>
> >> https://lore.kernel.org/imx/20260708035928.580236-2-hongxing.zhu@oss.nxp.com/
> > 
> > My question was whether somebody had actually bisected to 610fa91d9863
> > ("PCI: imx6: Assert PERST# before enabling regulators").  I don't see
> > anything specific on the mailing list, so I don't know how
> > 610fa91d9863 got blamed for the regression.
> 
> That was me and might have been a stupid idea. I normally wouldn't have
> added this, but a earlier message from Hongxing Zhu sounded a pretty
> convincingly like something along the lines of  "this is the problem I
> already have a fix for there" -- and then I took the commit-id from there.

Ah, ok, that explains it, thanks!  Hopefully we can get some clear
connection in the patch that eventually gets merged.

Re: [REGRESSION] mainline/master: Apalis iMX6 no longer boots

[Leonardo Costa]

On Thu, Jul 16, 2026 at 02:15:42AM +0000, Hongxing Zhu (OSS) wrote:
> Hi Leonardo,
> 
> Would you be able to test this patch on your affected boards? Your testing
> would help expedite the merge process and ensure the fix resolves the boot
> issue you're experiencing.
> 
> Best Regards
> Richard Zhu

Hi Richard,

Sorry for taking long to respond, I was caught up with something. I
tested the patch and it solved the issue. I sent a Tested-by tag there.
Thanks for looking into this!

Best Regards
Leonardo