PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Nick Bowler]

[-- Attachment #1: Type: text/plain, Size: 1426 bytes --]

Hi,

Linux 6.8 (and 6.8.1) are hanging on boot on my Ultra 60, right after
mounting the root filesystem.  This is a regression from 6.7, which
boots without any such problem.

Everything looks pretty normal, up to:

  [...]
  Run /sbin/init as init process
  INIT:

and the abnormal behaviour is that the boot appears to be stuck at
this point with no further output on the console.

I bisected to this commit:

  commit 45a3a8ef81291b63a2b50a1a145857dd9fc05e89
  Author: Tony Lindgren <tony@atomide.com>
  Date:   Mon Nov 13 10:07:53 2023 +0200
  
      serial: core: Revert checks for tx runtime PM state

Reverting this commit on top of Linux 6.8 is sufficient to get the
system booting again.

The console output related to the serial port (or anything else) does
not appear to be any different on a failing versus working kernel:

  f0061e88: ttyS0 at MMIO 0x1fff1400000 (irq = 7, base_baud = 115200) is a SAB82532 V3.2
  Console: ttyS0 (SAB82532)
  printk: legacy console [ttyS0] enabled
  f0061e88: ttyS1 at MMIO 0x1fff1400040 (irq = 7, base_baud = 115200) is a SAB82532 V3.2
  Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
  serial 0000:00:05.0: enabling device (0000 -> 0001)
  0000:00:05.0: ttyS2 at I/O 0x1fe02010c00 (irq = 17, base_baud = 115200) is a 16550A

In case it helps, I attached the complete console output from a failing
boot (gzipped).

Let me know if you need any more info!

Thanks,
  Nick

[-- Attachment #2: console-boot.log.gz --]
[-- Type: application/gzip, Size: 4693 bytes --]

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Tony Lindgren]

Hi,

* Nick Bowler <nbowler@draconx.ca> [240322 04:29]:
> I bisected to this commit:
> 
>   commit 45a3a8ef81291b63a2b50a1a145857dd9fc05e89
>   Author: Tony Lindgren <tony@atomide.com>
>   Date:   Mon Nov 13 10:07:53 2023 +0200
>   
>       serial: core: Revert checks for tx runtime PM state
> 
> Reverting this commit on top of Linux 6.8 is sufficient to get the
> system booting again.

Thanks a lot for bisecting and reporting the issue.

Can you please test if the following change to add back the check for
!pm_runtime_active() is enough to fix the issue?

Regards,

Tony

8< -----------------------
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -156,7 +156,7 @@ static void __uart_start(struct uart_state *state)
 	 * enabled, serial_port_runtime_resume() calls start_tx() again
 	 * after enabling the device.
 	 */
-	if (pm_runtime_active(&port_dev->dev))
+	if (!pm_runtime_enabled(&port_dev->dev) || pm_runtime_active(&port_dev->dev))
 		port->ops->start_tx(port);
 	pm_runtime_mark_last_busy(&port_dev->dev);
 	pm_runtime_put_autosuspend(&port_dev->dev);
-- 
2.44.0

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Nick Bowler]

On 2024-03-22 01:15, Tony Lindgren wrote:
> * Nick Bowler <nbowler@draconx.ca> [240322 04:29]:
>> I bisected to this commit:
>> 
>>   commit 45a3a8ef81291b63a2b50a1a145857dd9fc05e89
>>   Author: Tony Lindgren <tony@atomide.com>
>>   Date:   Mon Nov 13 10:07:53 2023 +0200
>>   
>>       serial: core: Revert checks for tx runtime PM state
>> 
>> Reverting this commit on top of Linux 6.8 is sufficient to get the
>> system booting again.
> 
> Thanks a lot for bisecting and reporting the issue.
> 
> Can you please test if the following change to add back the check for
> !pm_runtime_active() is enough to fix the issue?

I applied the below patch on top of 6.8 and unfortunately it does _not_
fix the problem (no obvious change in behaviour).

Thanks,
  Nick

> 8< -----------------------
> diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
> --- a/drivers/tty/serial/serial_core.c
> +++ b/drivers/tty/serial/serial_core.c
> @@ -156,7 +156,7 @@ static void __uart_start(struct uart_state *state)
>  	 * enabled, serial_port_runtime_resume() calls start_tx() again
>  	 * after enabling the device.
>  	 */
> -	if (pm_runtime_active(&port_dev->dev))
> +	if (!pm_runtime_enabled(&port_dev->dev) || pm_runtime_active(&port_dev->dev))
>  		port->ops->start_tx(port);
>  	pm_runtime_mark_last_busy(&port_dev->dev);
>  	pm_runtime_put_autosuspend(&port_dev->dev);

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Tony Lindgren]

* Nick Bowler <nbowler@draconx.ca> [240322 06:36]:
> On 2024-03-22 01:15, Tony Lindgren wrote:
> > Can you please test if the following change to add back the check for
> > !pm_runtime_active() is enough to fix the issue?
> 
> I applied the below patch on top of 6.8 and unfortunately it does _not_
> fix the problem (no obvious change in behaviour).

Hmm OK thanks for testing. I'll take a look and see if I can debug this
with qemu sparc as I no longer have any sparc boxes around.

Regards,

Tony

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Tony Lindgren]

* Tony Lindgren <tony@atomide.com> [240322 08:48]:
> * Nick Bowler <nbowler@draconx.ca> [240322 06:36]:
> > On 2024-03-22 01:15, Tony Lindgren wrote:
> > > Can you please test if the following change to add back the check for
> > > !pm_runtime_active() is enough to fix the issue?
> > 
> > I applied the below patch on top of 6.8 and unfortunately it does _not_
> > fix the problem (no obvious change in behaviour).
> 
> Hmm OK thanks for testing. I'll take a look and see if I can debug this
> with qemu sparc as I no longer have any sparc boxes around.

I can't reproduce this on qemu-system-sparc64, probably as it does not use
the sunsab driver.

I noticed something though, I think we need to test for the port device
instead for being runtime PM enabled.

Can you please test if the updated patch below make things work again?

Regards,

Tony

8< -------------------
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -156,7 +156,7 @@ static void __uart_start(struct uart_state *state)
 	 * enabled, serial_port_runtime_resume() calls start_tx() again
 	 * after enabling the device.
 	 */
-	if (pm_runtime_active(&port_dev->dev))
+	if (!pm_runtime_enabled(port->dev) || pm_runtime_active(&port_dev->dev))
 		port->ops->start_tx(port);
 	pm_runtime_mark_last_busy(&port_dev->dev);
 	pm_runtime_put_autosuspend(&port_dev->dev);
-- 
2.44.0

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Nick Bowler]

On 2024-03-22 05:06, Tony Lindgren wrote:
[...]
> I can't reproduce this on qemu-system-sparc64, probably as it does not use
> the sunsab driver.
> 
> I noticed something though, I think we need to test for the port device
> instead for being runtime PM enabled.
> 
> Can you please test if the updated patch below make things work again?

Yes, with the below patch applied on top of 6.8 things are working.

Thanks,
  Nick

> 8< -------------------
> diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
> --- a/drivers/tty/serial/serial_core.c
> +++ b/drivers/tty/serial/serial_core.c
> @@ -156,7 +156,7 @@ static void __uart_start(struct uart_state *state)
>  	 * enabled, serial_port_runtime_resume() calls start_tx() again
>  	 * after enabling the device.
>  	 */
> -	if (pm_runtime_active(&port_dev->dev))
> +	if (!pm_runtime_enabled(port->dev) || pm_runtime_active(&port_dev->dev))
>  		port->ops->start_tx(port);
>  	pm_runtime_mark_last_busy(&port_dev->dev);
>  	pm_runtime_put_autosuspend(&port_dev->dev)

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Tony Lindgren]

* Nick Bowler <nbowler@draconx.ca> [240322 14:12]:
> On 2024-03-22 05:06, Tony Lindgren wrote:
> [...]
> > I can't reproduce this on qemu-system-sparc64, probably as it does not use
> > the sunsab driver.
> > 
> > I noticed something though, I think we need to test for the port device
> > instead for being runtime PM enabled.
> > 
> > Can you please test if the updated patch below make things work again?
> 
> Yes, with the below patch applied on top of 6.8 things are working.

OK great thanks for testing, I'll send out a proper patch.

Regards,

Tony

> > 8< -------------------
> > diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
> > --- a/drivers/tty/serial/serial_core.c
> > +++ b/drivers/tty/serial/serial_core.c
> > @@ -156,7 +156,7 @@ static void __uart_start(struct uart_state *state)
> >  	 * enabled, serial_port_runtime_resume() calls start_tx() again
> >  	 * after enabling the device.
> >  	 */
> > -	if (pm_runtime_active(&port_dev->dev))
> > +	if (!pm_runtime_enabled(port->dev) || pm_runtime_active(&port_dev->dev))
> >  		port->ops->start_tx(port);
> >  	pm_runtime_mark_last_busy(&port_dev->dev);
> >  	pm_runtime_put_autosuspend(&port_dev->dev)

Re: PROBLEM: Sun Ultra 60 hangs on boot since Linux 6.8

[Tony Lindgren]

* Tony Lindgren <tony@atomide.com> [240323 08:49]:
> * Nick Bowler <nbowler@draconx.ca> [240322 14:12]:
> > Yes, with the below patch applied on top of 6.8 things are working.
> 
> OK great thanks for testing, I'll send out a proper patch.

For reference, patch posted now at [0] below.

Regards,

Tony

[0] https://lore.kernel.org/linux-serial/20240325071649.27040-1-tony@atomide.com/T/#u