PROBLEM: boot hang on Indy R4400SC (regression)

PROBLEM: boot hang on Indy R4400SC (regression)

[Nick Bowler]

Hi,

After a recent 6.1.y stable kernel update, my Indy (mips64 R4400SC) now
just stops booting early, just before when I would normally see the
kernel messages about mounting the root filesystem.

There are no further messages of any kind, and the boot process does not
appear to ever complete.  However, the kernel is not fully crashed, as
it does respond to sysrq commands from the keyboard (and I do get output
on the console from these).

I bisected to the following:
  
    794b679a28bb59a4533ae39a7cf945b9d5bbe336 is the first bad commit
    commit 794b679a28bb59a4533ae39a7cf945b9d5bbe336
    Author: Jiaxun Yang <jiaxun.yang@flygoat.com>
    Date:   Sat Jun 7 13:43:56 2025 +0100
    
        MIPS: mm: tlb-r4k: Uniquify TLB entries on init
    
        commit 35ad7e181541aa5757f9f316768d3e64403ec843 upstream.

This reverts cleanly on top of 6.1.158 and the resulting kernel boots
normally.  I then reproduced this failure on 6.18-rc4.  Reverting
35ad7e181541 on top of 6.18-rc4 also results in a normal boot.

Let me know if you need any more info!

Thanks,
  Nick

Re: PROBLEM: boot hang on Indy R4400SC (regression)

[Jiaxun Yang]

On Fri, 7 Nov 2025, at 7:04 AM, Nick Bowler wrote:
> Hi,

Hi Nick,

Many thanks for the issue report! It's pretty rare to get reports from
hardware that old.

Unfortunately my Indy won't go over ARCS prom so I'm not in a position
to debug this on my side. I have inspected the code again and I can't
see anything preventing it to work on R4000 family.

Maybe we can revert this for non-MIPS64R1 system only so we can get something
working for both old and new systems.

#regzbot introduced: 35ad7e181541aa5757f9f316768d3e64403ec843

Thanks

>
> After a recent 6.1.y stable kernel update, my Indy (mips64 R4400SC) now
> just stops booting early, just before when I would normally see the
> kernel messages about mounting the root filesystem.
>
> There are no further messages of any kind, and the boot process does not
> appear to ever complete.  However, the kernel is not fully crashed, as
> it does respond to sysrq commands from the keyboard (and I do get output
> on the console from these).
>
> I bisected to the following:
>  
>     794b679a28bb59a4533ae39a7cf945b9d5bbe336 is the first bad commit
>     commit 794b679a28bb59a4533ae39a7cf945b9d5bbe336
>     Author: Jiaxun Yang <jiaxun.yang@flygoat.com>
>     Date:   Sat Jun 7 13:43:56 2025 +0100
>    
>         MIPS: mm: tlb-r4k: Uniquify TLB entries on init
>    
>         commit 35ad7e181541aa5757f9f316768d3e64403ec843 upstream.
>
> This reverts cleanly on top of 6.1.158 and the resulting kernel boots
> normally.  I then reproduced this failure on 6.18-rc4.  Reverting
> 35ad7e181541 on top of 6.18-rc4 also results in a normal boot.
>
> Let me know if you need any more info!
>
> Thanks,
>   Nick

-- 
- Jiaxun

Re: PROBLEM: boot hang on Indy R4400SC (regression)

[Nick Bowler]

On Fri, Nov 07, 2025 at 07:29:25PM +0000, Jiaxun Yang wrote:
> Unfortunately my Indy won't go over ARCS prom so I'm not in a position
> to debug this on my side. I have inspected the code again and I can't
> see anything preventing it to work on R4000 family.

I'll try adding some extra prints to at least figure out where it is
actually hanging.

Thanks,
  Nick

Re: PROBLEM: boot hang on Indy R4400SC (regression)

[Nick Bowler]

On Fri, Nov 07, 2025 at 03:12:31PM -0500, Nick Bowler wrote:
> On Fri, Nov 07, 2025 at 07:29:25PM +0000, Jiaxun Yang wrote:
> > Unfortunately my Indy won't go over ARCS prom so I'm not in a position
> > to debug this on my side. I have inspected the code again and I can't
> > see anything preventing it to work on R4000 family.
> 
> I'll try adding some extra prints to at least figure out where it is
> actually hanging.

I did not have much success with adding prints, but looking more closely
at the console output it seems that what is ultimately failing is the
SCSI bus enumeration which does not complete unless I revert commit
35ad7e181541.

So I presume that is why I also don't see the messages about mounting
the root filesystem (I suppose it is just waiting for a disk).

I see the drivers printing the usual info about each device, but not
everything.  Specifically, the lines that are missing are all of these
ones that would normally be printed:

    sda: sda1 sda2 sda9 sda11
   sd 0:0:1:0: [sda] Attached SCSI disk

    sdb: sdb1 sdb9 sdb11
   sd 0:0:2:0: [sdb] Attached SCSI disk

   sr 0:0:5:0: Attached scsi CD-ROM sr0

Other than that everything else seems alive.  Several other drivers
go through their initialization during the time the SCSI stuff is not
completing.  The 'random: crng init done' message is printed after a
while too.

I tried enabling CONFIG_SOFTLOCKUP_DETECTOR and CONFIG_WQ_WATCHDOG to
get some more information out but these options do not seem to do
anything in this scenario, nothing is printed even after ~10 minutes.

Thanks,
  Nick

Re: PROBLEM: boot hang on Indy R4400SC (regression)

[Thomas Bogendoerfer]

On Sat, Nov 08, 2025 at 03:02:17PM -0500, Nick Bowler wrote:
> On Fri, Nov 07, 2025 at 03:12:31PM -0500, Nick Bowler wrote:
> > On Fri, Nov 07, 2025 at 07:29:25PM +0000, Jiaxun Yang wrote:
> > > Unfortunately my Indy won't go over ARCS prom so I'm not in a position
> > > to debug this on my side. I have inspected the code again and I can't
> > > see anything preventing it to work on R4000 family.
> > 
> > I'll try adding some extra prints to at least figure out where it is
> > actually hanging.
> 
> I did not have much success with adding prints, but looking more closely
> at the console output it seems that what is ultimately failing is the
> SCSI bus enumeration which does not complete unless I revert commit
> 35ad7e181541.

the problem is the tlb_probe() r4k_tlb_uniquify(), if the TLB contains

Index:  0 pgmask=4kb va=80000000 asid=00
	[pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
Index:  1 pgmask=4kb va=80000000 asid=00
	[pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
Index:  2 pgmask=4kb va=80000000 asid=00
	[pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
Index:  3 pgmask=4kb va=80000000 asid=00
	[pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
Index:  4 pgmask=4kb va=80000000 asid=00
	[pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
[...]

which is coming from the Indy PROM. I only tested this patch on an R5k
Indy and the R5000 CPU doesn't have a problem with this.

I'm working on a fix for this.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a
good idea.                                                [ RFC1925, 2.3 ]