bcm4708-smartrg-sr400ac boot failures in next-20150918+

bcm4708-smartrg-sr400ac boot failures in next-20150918+

[Hauke Mehrtens]

Adding the arm List.

On 10/01/2015 11:46 PM, Hauke Mehrtens wrote:
> On 10/01/2015 07:50 PM, Tyler Baker wrote:
>> Hi Hauke, Rafal,
>>
>> The kernelci.org bot has been reporting complete boot failures[1]
>> since next-20150918. I've bisected[2] the issue and found "ARM:
>> 8422/1: enable imprecise aborts during early kernel startup" is the
>> offending patch. Reverting this on top of -next fixes the boot issue.
>>
>> Apologies for sending this off list, but I want to make sure you can
>> confirm this issue before we raise it.
>>
>> Cheers,
>>
>> Tyler
>>
>> [1] http://kernelci.org/boot/bcm4708-smartrg-sr400ac/
>> [2] http://hastebin.com/mirunerobi.vhdl
>>
> Hi Tyler,
> 
> Thanks for the info, the broken bootloader causes some problems when
> booting the kernel, we already had to ignore some exceptions. I will
> look into this at the weekend.
> 
> Hauke
> 
Hi,

The commit "ARM: 8422/1: enable imprecise aborts during early kernel
startup" breaks bcm53xx devices with a broken boot loader, this is a
regression. There are probably millions of these devices shipped with
this boot loader called CFE.
For BCM53xx we install a fault handler in the init_early callback of
DT_MACHINE_START to ignore the faults produced by the boot loader in
http://lxr.free-electrons.com/source/arch/arm/mach-bcm/bcm_5301x.c#L40 ,
but init_early is called after local_abt_enable() gets called and this
causes a kernel panic because of any unhandled CPU fault. The reserve()
callback in DT_MACHINE_START was the only callback I could find which
gets called before local_abt_enable(), should I move the code which
installs the fault handler there?

This is causing the problems in kernelci with linux next kernels on this
board: http://kernelci.org/boot/bcm4708-smartrg-sr400ac/

Hauke

bcm4708-smartrg-sr400ac boot failures in next-20150918+

[Tyler Baker]

On 11 October 2015 at 15:49, Hauke Mehrtens <hauke@hauke-m.de> wrote:
> Adding the arm List.
>
> On 10/01/2015 11:46 PM, Hauke Mehrtens wrote:
>> On 10/01/2015 07:50 PM, Tyler Baker wrote:
>>> Hi Hauke, Rafal,
>>>
>>> The kernelci.org bot has been reporting complete boot failures[1]
>>> since next-20150918. I've bisected[2] the issue and found "ARM:
>>> 8422/1: enable imprecise aborts during early kernel startup" is the
>>> offending patch. Reverting this on top of -next fixes the boot issue.
>>>
>>> Apologies for sending this off list, but I want to make sure you can
>>> confirm this issue before we raise it.
>>>
>>> Cheers,
>>>
>>> Tyler
>>>
>>> [1] http://kernelci.org/boot/bcm4708-smartrg-sr400ac/
>>> [2] http://hastebin.com/mirunerobi.vhdl
>>>
>> Hi Tyler,
>>
>> Thanks for the info, the broken bootloader causes some problems when
>> booting the kernel, we already had to ignore some exceptions. I will
>> look into this at the weekend.
>>
>> Hauke
>>
> Hi,
>
> The commit "ARM: 8422/1: enable imprecise aborts during early kernel
> startup" breaks bcm53xx devices with a broken boot loader, this is a
> regression. There are probably millions of these devices shipped with
> this boot loader called CFE.
> For BCM53xx we install a fault handler in the init_early callback of
> DT_MACHINE_START to ignore the faults produced by the boot loader in
> http://lxr.free-electrons.com/source/arch/arm/mach-bcm/bcm_5301x.c#L40 ,
> but init_early is called after local_abt_enable() gets called and this
> causes a kernel panic because of any unhandled CPU fault. The reserve()
> callback in DT_MACHINE_START was the only callback I could find which
> gets called before local_abt_enable(), should I move the code which
> installs the fault handler there?
>
> This is causing the problems in kernelci with linux next kernels on this
> board: http://kernelci.org/boot/bcm4708-smartrg-sr400ac/

Adding Russell, and Lucas to CC as they may be interested in this thread.

Cheers,

Tyler

bcm4708-smartrg-sr400ac boot failures in next-20150918+

[Lucas Stach]

Hi Tyler,

Am Dienstag, den 13.10.2015, 11:49 -0700 schrieb Tyler Baker:
> On 11 October 2015 at 15:49, Hauke Mehrtens <hauke@hauke-m.de> wrote:
> > Adding the arm List.
> >
> > On 10/01/2015 11:46 PM, Hauke Mehrtens wrote:
> >> On 10/01/2015 07:50 PM, Tyler Baker wrote:
> >>> Hi Hauke, Rafal,
> >>>
> >>> The kernelci.org bot has been reporting complete boot failures[1]
> >>> since next-20150918. I've bisected[2] the issue and found "ARM:
> >>> 8422/1: enable imprecise aborts during early kernel startup" is the
> >>> offending patch. Reverting this on top of -next fixes the boot issue.
> >>>
> >>> Apologies for sending this off list, but I want to make sure you can
> >>> confirm this issue before we raise it.
> >>>
> >>> Cheers,
> >>>
> >>> Tyler
> >>>
> >>> [1] http://kernelci.org/boot/bcm4708-smartrg-sr400ac/
> >>> [2] http://hastebin.com/mirunerobi.vhdl
> >>>
> >> Hi Tyler,
> >>
> >> Thanks for the info, the broken bootloader causes some problems when
> >> booting the kernel, we already had to ignore some exceptions. I will
> >> look into this at the weekend.
> >>
> >> Hauke
> >>
> > Hi,
> >
> > The commit "ARM: 8422/1: enable imprecise aborts during early kernel
> > startup" breaks bcm53xx devices with a broken boot loader, this is a
> > regression. There are probably millions of these devices shipped with
> > this boot loader called CFE.
> > For BCM53xx we install a fault handler in the init_early callback of
> > DT_MACHINE_START to ignore the faults produced by the boot loader in
> > http://lxr.free-electrons.com/source/arch/arm/mach-bcm/bcm_5301x.c#L40 ,
> > but init_early is called after local_abt_enable() gets called and this
> > causes a kernel panic because of any unhandled CPU fault. The reserve()
> > callback in DT_MACHINE_START was the only callback I could find which
> > gets called before local_abt_enable(), should I move the code which
> > installs the fault handler there?
> >
> > This is causing the problems in kernelci with linux next kernels on this
> > board: http://kernelci.org/boot/bcm4708-smartrg-sr400ac/
> 
> Adding Russell, and Lucas to CC as they may be interested in this thread.

Thanks for looping me in and sorry about the regression.

I think we really want to keep enabling the imprecise aborts as early as
possible to catch them near the spot where they did happen.

So I think the right thing to work around your broken bootloader is to
install the fault handler earlier. Russell, do you have any objections
to that?

I think there are some other SoCs that need a similar workaround, if we
agree to move the fault handler installation to an earlier stage I'll
audit other places and send patches if needed, so we can fix this up
before this commit ends up in a pull to Linus.

Regards,
Lucas

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

bcm4708-smartrg-sr400ac boot failures in next-20150918+

[Russell King - ARM Linux]

On Wed, Oct 14, 2015 at 11:48:45AM +0200, Lucas Stach wrote:
> I think we really want to keep enabling the imprecise aborts as early as
> possible to catch them near the spot where they did happen.

Agreed.

> So I think the right thing to work around your broken bootloader is to
> install the fault handler earlier. Russell, do you have any objections
> to that?

How about installing the fault handler early before enabling the imprecise
aborts, and have the fault handler print a complaint about the broken
boot loader, and have the fault handler uninstall itself after the
imprecise abort has been enabled?

If there's a pending imprecise abort, it should be delivered as soon as
they're unmasked.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

bcm4708-smartrg-sr400ac boot failures in next-20150918+

[Lucas Stach]

Am Mittwoch, den 14.10.2015, 10:56 +0100 schrieb Russell King - ARM
Linux:
> On Wed, Oct 14, 2015 at 11:48:45AM +0200, Lucas Stach wrote:
> > I think we really want to keep enabling the imprecise aborts as early as
> > possible to catch them near the spot where they did happen.
> 
> Agreed.
> 
> > So I think the right thing to work around your broken bootloader is to
> > install the fault handler earlier. Russell, do you have any objections
> > to that?
> 
> How about installing the fault handler early before enabling the imprecise
> aborts, and have the fault handler print a complaint about the broken
> boot loader, and have the fault handler uninstall itself after the
> imprecise abort has been enabled?
> 
We might point at the bootloader a bit too often this way, as there is
quite a bit of kernel code running before we can enable imprecise
aborts. But I like the idea in general as it would allow us to drop
those workaround from the individual architectures and move things into
common code.

> If there's a pending imprecise abort, it should be delivered as soon as
> they're unmasked.
> 
Right.

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

bcm4708-smartrg-sr400ac boot failures in next-20150918+

[Hauke Mehrtens]

On 10/14/2015 12:04 PM, Lucas Stach wrote:
> Am Mittwoch, den 14.10.2015, 10:56 +0100 schrieb Russell King - ARM
> Linux:
>> On Wed, Oct 14, 2015 at 11:48:45AM +0200, Lucas Stach wrote:
>>> I think we really want to keep enabling the imprecise aborts as early as
>>> possible to catch them near the spot where they did happen.
>>
>> Agreed.

I was never against this patch I just wanted to point out that it caused
a regression and we should fix it, I was just a little bit frightened to
add some early init code into a callback which gets called before early
init.

>>> So I think the right thing to work around your broken bootloader is to
>>> install the fault handler earlier. Russell, do you have any objections
>>> to that?
>>
>> How about installing the fault handler early before enabling the imprecise
>> aborts, and have the fault handler print a complaint about the broken
>> boot loader, and have the fault handler uninstall itself after the
>> imprecise abort has been enabled?
>>
> We might point at the bootloader a bit too often this way, as there is
> quite a bit of kernel code running before we can enable imprecise
> aborts. But I like the idea in general as it would allow us to drop
> those workaround from the individual architectures and move things into
> common code.

Lucas patches fixed the problem for me on a bcm4708 based device with a
broken boot loader. I assume Lucas patches will make it into mainline
and this issue is now solved for me.

>> If there's a pending imprecise abort, it should be delivered as soon as
>> they're unmasked.
>>
> Right.

Hauke