[BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[Sherry Sun]

Hi all,

Since the latest i.MX UART switch to nbcon console with commit 70acca67bdd3 ("serial: imx: Switch to nbcon console"), I'm encountering a suspend issue on an i.MX8MP EVK board while testing with Linux 6.18-rc4.
The system fails to suspend when both nbcon is enabled and `pm_debug_messages` is turned on. When `pm_debug_messages` is disabled, suspend works normally. Additionally, if I revert the patch that adds nbcon support for the i.MX UART, suspend also works fine even with `pm_debug_messages` enabled.

The error message during suspend is:
    echo: write error: Operation not supported

This seems to indicate that nbcon's interaction with the suspend flow-especially when debug messages are printed-might be preventing certain threads or devices from freezing properly.
Let me know if more logs or testing details would help. I'd be happy to assist in debugging or validating any proposed fixes.

Suppose this is not i.MX UART specific issue, currently the following 3 UART drivers also support nbcon console.
Hi Toshiyuki / Kartik / Ryo, if it's convenient, can you please help check if this issue also exists on your UARTs? Thanks!

drivers/tty/serial/sifive.c:907:        .flags          = CON_PRINTBUFFER | CON_NBCON,
drivers/tty/serial/tegra-utc.c:512:     tup->console.flags              = CON_PRINTBUFFER | CON_NBCON;
drivers/tty/serial/amba-pl011.c:2616:   .flags          = CON_PRINTBUFFER | CON_ANYTIME | CON_NBCON,

The reproduce steps are quite simple:
root@imx8mpevk:~# echo 1 > /sys/power/pm_debug_messages
root@imx8mpevk:~# cat /sys/power/pm_debug_messages
1
root@imx8mpevk:~# echo mem > /sys/power/state
[   46.434401] PM: suspend entry (deep)
[   46.439474] Filesystems sync: 0.005 seconds
[   46.444008] Freezing user space processes
[   46.451855] Freezing user space processes completed (elapsed 0.007 seconds)
[   46.451876] OOM killer disabled.
[   46.451884] Freezing remaining freezable tasks
[   46.453106] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[   46.498122] PM: suspend devices took 0.048 seconds
[   46.505390] Disabling non-boot CPUs ...
[   46.511169] psci: CPU3 killed (polled 0 ms)
[   46.517174] psci: CPU2 killed (polled 0 ms)
[   46.523472] psci: CPU1 killed (polled 0 ms)
[   46.528669] Enabling non-boot CPUs ...
[   46.532891] Detected VIPT I-cache on CPU1
[   46.532920] GICv3: CPU1: found redistributor 1 region 0:0x00000000388a0000
[   46.532947] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[   46.551156] CPU1 is up
[   46.553907] Detected VIPT I-cache on CPU2
[   46.553934] GICv3: CPU2: found redistributor 2 region 0:0x00000000388c0000
[   46.553959] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[   46.554629] CPU2 is up
[   46.554999] Detected VIPT I-cache on CPU3
[   46.555022] GICv3: CPU3: found redistributor 3 region 0:0x00000000388e0000
[   46.555044] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[   46.555748] CPU3 is up
[   46.602411] imx-dwmac 30bf0000.ethernet eth1: No Safety Features support found
[   46.604199] imx-dwmac 30bf0000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
[   46.606020] imx-dwmac 30bf0000.ethernet eth1: configuring for phy/rgmii-id link mode
[   46.713935] caam 30900000.crypto: registering rng-caam
[   46.714857] xhci-hcd xhci-hcd.1.auto: xHC error in resume, USBSTS 0x401, Reinit
[   46.714867] usb usb1: root hub lost power or was reset
[   46.714871] usb usb2: root hub lost power or was reset
[   46.818976] PM: resume devices took 0.260 seconds
-sh: echo: write error: Operation not supported
[   46.819307] OOM killer enabled.
[   46.819311] Restarting tasks: Starting
[   46.820385] Restarting tasks: Done
[   46.820450] random: crng reseeded on system resumption
[   46.820553] PM: suspend exit
root@imx8mpevk:~#

Best Regards
Sherry

Re: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[John Ogness]

Hi Sherry,

On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
> Since the latest i.MX UART switch to nbcon console with commit
> 70acca67bdd3 ("serial: imx: Switch to nbcon console"), I'm
> encountering a suspend issue on an i.MX8MP EVK board while testing
> with Linux 6.18-rc4.
>
> The system fails to suspend when both nbcon is enabled and
> `pm_debug_messages` is turned on. When `pm_debug_messages` is
> disabled, suspend works normally. Additionally, if I revert the patch
> that adds nbcon support for the i.MX UART, suspend also works fine
> even with `pm_debug_messages` enabled.

Thanks for reporting! I believe this is the same issue reported by
NVIDIA [0] when we tried to switch the 8250 UART driver to nbcon.

We have been working with NVIDIA recently to address the issue. There is
a patch [1] we are currently testing that looks good so far. It is based
on 6.17 but should work fine for 6.18-rc4 as well. Can you give it a
spin?

John Ogness

[0] https://lore.kernel.org/lkml/80b020fc-c18a-4da4-b222-16da1cab2f4c@nvidia.com

[1] https://github.com/Linutronix/linux/commit/ae173249d9028ef159fba040bdab260d80dda43f

RE: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[Sherry Sun]

> -----Original Message-----
> From: John Ogness <john.ogness@linutronix.de>
> Sent: Tuesday, November 4, 2025 4:29 PM
> To: Sherry Sun <sherry.sun@nxp.com>; esben@geanix.com;
> pmladek@suse.com; senozhatsky@chromium.org; tglx@linutronix.de; Greg
> KH <gregkh@linuxfoundation.org>; Jiri Slaby <jirislaby@kernel.org>; Shawn
> Guo <shawnguo@kernel.org>; Sascha Hauer <s.hauer@pengutronix.de>;
> ryotkkr98@gmail.com; kkartik@nvidia.com; fj6611ie@aa.jp.fujitsu.com
> Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-serial
> <linux-serial@vger.kernel.org>; imx@lists.linux.dev; Pengutronix Kernel Team
> <kernel@pengutronix.de>; linux-pm@vger.kernel.org
> Subject: Re: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux
> 6.18-rc4)
>
> Hi Sherry,
>
> On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
> > Since the latest i.MX UART switch to nbcon console with commit
> > 70acca67bdd3 ("serial: imx: Switch to nbcon console"), I'm
> > encountering a suspend issue on an i.MX8MP EVK board while testing
> > with Linux 6.18-rc4.
> >
> > The system fails to suspend when both nbcon is enabled and
> > `pm_debug_messages` is turned on. When `pm_debug_messages` is
> > disabled, suspend works normally. Additionally, if I revert the patch
> > that adds nbcon support for the i.MX UART, suspend also works fine
> > even with `pm_debug_messages` enabled.
>
> Thanks for reporting! I believe this is the same issue reported by NVIDIA [0]
> when we tried to switch the 8250 UART driver to nbcon.
>
> We have been working with NVIDIA recently to address the issue. There is a
> patch [1] we are currently testing that looks good so far. It is based on 6.17
> but should work fine for 6.18-rc4 as well. Can you give it a spin?
>

Hi John,

Thanks a lot for your quick reply, I just tried the patch [1] on my i.MX8MP EVK with L6.18-rc4, it does fix the suspend issue.
Now i.MX UART nbcon can enter suspend with `pm_debug_messages` is turned on.
May I know what is the upstream plan for the patch [1]?

Best Regards
Sherry

> [0]
> https://lore.ke/
> rnel.org%2Flkml%2F80b020fc-c18a-4da4-b222-
> 16da1cab2f4c%40nvidia.com&data=05%7C02%7Csherry.sun%40nxp.com%7C
> 223190eafeb045a646fb08de1b7c2741%7C686ea1d3bc2b4c6fa92cd99c5c3016
> 35%7C0%7C0%7C638978417234027264%7CUnknown%7CTWFpbGZsb3d8eyJ
> FbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiT
> WFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuIijva%2FcH8LBrPhAS
> PopGShQDJdoEBreIc8qGhR63E%3D&reserved=0
>
> [1]
> https://github/.
> com%2FLinutronix%2Flinux%2Fcommit%2Fae173249d9028ef159fba040bdab
> 260d80dda43f&data=05%7C02%7Csherry.sun%40nxp.com%7C223190eafeb0
> 45a646fb08de1b7c2741%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
> %7C638978417234052653%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcG
> kiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj
> oyfQ%3D%3D%7C0%7C%7C%7C&sdata=oyqo8zAnUA%2FuAc5RuTtOCVRJMZc
> k3IexrBNvI5NpjzQ%3D&reserved=0

RE: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[John Ogness]

On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
>> Thanks for reporting! I believe this is the same issue reported by NVIDIA [0]
>> when we tried to switch the 8250 UART driver to nbcon.
>>
>> We have been working with NVIDIA recently to address the issue. There is a
>> patch [1] we are currently testing that looks good so far. It is based on 6.17
>> but should work fine for 6.18-rc4 as well. Can you give it a spin?
>
> Thanks a lot for your quick reply, I just tried the patch [1] on my
> i.MX8MP EVK with L6.18-rc4, it does fix the suspend issue.  Now i.MX
> UART nbcon can enter suspend with `pm_debug_messages` is turned on.
> May I know what is the upstream plan for the patch [1]?

Thanks for confirming the fix. I will make an official post on LKML with
the patch today. Since the i.MX nbcon-driver is already mainline, I will
CC stable. I will CC you as well.

Note that the patch still needs a formal review from the printk folks
once it hits LKML.

John Ogness

>> [1] https://github.com/Linutronix/linux/commit/ae173249d9028ef159fba040bdab260d80dda43f

RE: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[Sherry Sun]

> On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
> >> Thanks for reporting! I believe this is the same issue reported by
> >> NVIDIA [0] when we tried to switch the 8250 UART driver to nbcon.
> >>
> >> We have been working with NVIDIA recently to address the issue. There
> >> is a patch [1] we are currently testing that looks good so far. It is
> >> based on 6.17 but should work fine for 6.18-rc4 as well. Can you give it a
> spin?
> >
> > Thanks a lot for your quick reply, I just tried the patch [1] on my
> > i.MX8MP EVK with L6.18-rc4, it does fix the suspend issue.  Now i.MX
> > UART nbcon can enter suspend with `pm_debug_messages` is turned on.
> > May I know what is the upstream plan for the patch [1]?
> 
> Thanks for confirming the fix. I will make an official post on LKML with the
> patch today. Since the i.MX nbcon-driver is already mainline, I will CC stable. I
> will CC you as well.
> 
> Note that the patch still needs a formal review from the printk folks once it
> hits LKML.
> 

Hi John, got it.
I'll keep an eye on any fixes you send later and verify them if necessary. Hope this fix can be applied to the mainline and the stable tree ASAP.

Best Regards
Sherry

RE: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[John Ogness]

Hi Sherry,

On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
>> Thanks for confirming the fix. I will make an official post on LKML
>> with the patch today. Since the i.MX nbcon-driver is already
>> mainline, I will CC stable. I will CC you as well.
>
> I'll keep an eye on any fixes you send later and verify them if
> necessary. Hope this fix can be applied to the mainline and the stable
> tree ASAP.

Even with this patch [1] I am able to reproduce the problem on one of my
test boards. I am using the following extra boot arguments:

no_console_suspend loglevel=8 initcall_debug

And then:

# echo 1 > /sys/power/pm_debug_messages
# echo mem > /sys/power/state

This generates a lot of output and leads to the same suspend
failure. This may be due to the out-of-tree nbcon-8250 driver I am
using, which introduces extra irq_work usage within the driver. Could
you please try the above boot arguments with your i.MX board?

I am wondering if blocking the queueing of irq_work during suspend
should be generally implemented rather than just printk avoiding it. It
also seems to only be a problem for ARM32 so perhaps it needs to be
addressed at the architecture level.

John Ogness

>> [1] https://github.com/Linutronix/linux/commit/ae173249d9028ef159fba040bdab260d80dda43f

RE: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[Sherry Sun]

> -----Original Message-----
> From: John Ogness <john.ogness@linutronix.de>
> Sent: Wednesday, November 5, 2025 7:20 AM
> To: Sherry Sun <sherry.sun@nxp.com>; esben@geanix.com;
> pmladek@suse.com; senozhatsky@chromium.org; tglx@linutronix.de; Greg
> KH <gregkh@linuxfoundation.org>; Jiri Slaby <jirislaby@kernel.org>; Shawn
> Guo <shawnguo@kernel.org>; Sascha Hauer <s.hauer@pengutronix.de>;
> ryotkkr98@gmail.com; kkartik@nvidia.com; fj6611ie@aa.jp.fujitsu.com
> Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>; linux-serial
> <linux-serial@vger.kernel.org>; imx@lists.linux.dev; Pengutronix Kernel Team
> <kernel@pengutronix.de>; linux-pm@vger.kernel.org
> Subject: RE: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux
> 6.18-rc4)
> 
> Hi Sherry,
> 
> On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
> >> Thanks for confirming the fix. I will make an official post on LKML
> >> with the patch today. Since the i.MX nbcon-driver is already
> >> mainline, I will CC stable. I will CC you as well.
> >
> > I'll keep an eye on any fixes you send later and verify them if
> > necessary. Hope this fix can be applied to the mainline and the stable
> > tree ASAP.
> 
> Even with this patch [1] I am able to reproduce the problem on one of my test
> boards. I am using the following extra boot arguments:
> 
> no_console_suspend loglevel=8 initcall_debug
> 
> And then:
> 
> # echo 1 > /sys/power/pm_debug_messages
> # echo mem > /sys/power/state
> 
> This generates a lot of output and leads to the same suspend failure. This
> may be due to the out-of-tree nbcon-8250 driver I am using, which introduces
> extra irq_work usage within the driver. Could you please try the above boot
> arguments with your i.MX board?
> 
> I am wondering if blocking the queueing of irq_work during suspend should
> be generally implemented rather than just printk avoiding it. It also seems to
> only be a problem for ARM32 so perhaps it needs to be addressed at the
> architecture level.
> 

Hi John,

I tried to add "no_console_suspend loglevel=8 initcall_debug" boot arguments and enable pm_debug_messages, i.MX8MP suspend is still ok  with your patch [1].
And please note, i.MX8MP is ARM64 arch. 
I also tested the patch [1] with the same settings on i.MX6ULL(ARM32), same observation with i.MX8MP, suspend also works fine.
Seems the issue you observed here cannot be reproduced on i.MX UART as it doesn't use the irq_work, and at least per my testing, the fix should be addressed for both ARM64 and ARM32.

Best Regards
Sherry

> 
> >> [1]
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> >>
> hub.com%2FLinutronix%2Flinux%2Fcommit%2Fae173249d9028ef159fba040b
> dab2
> >>
> 60d80dda43f&data=05%7C02%7Csherry.sun%40nxp.com%7C7988cfdc84f64e
> 9cd1d
> >>
> d08de1bf8ab21%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6389
> 789520
> >>
> 47428481%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYi
> OiIwLjAu
> >>
> MDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%
> 7C%7C
> >>
> &sdata=yPKOGyd5wAECK8HFz3bD09vLva6jCPAQyLZPNbnG%2Buk%3D&reserv
> ed=0

Re: [BUG] Suspend failure with nbcon + pm_debug_messages (Linux 6.18-rc4)

[Petr Mladek]

On Wed 2025-11-05 00:25:55, John Ogness wrote:
> Hi Sherry,
> 
> On 2025-11-04, Sherry Sun <sherry.sun@nxp.com> wrote:
> >> Thanks for confirming the fix. I will make an official post on LKML
> >> with the patch today. Since the i.MX nbcon-driver is already
> >> mainline, I will CC stable. I will CC you as well.
> >
> > I'll keep an eye on any fixes you send later and verify them if
> > necessary. Hope this fix can be applied to the mainline and the stable
> > tree ASAP.
> 
> Even with this patch [1] I am able to reproduce the problem on one of my
> test boards. I am using the following extra boot arguments:
> 
> no_console_suspend loglevel=8 initcall_debug
> 
> And then:
> 
> # echo 1 > /sys/power/pm_debug_messages
> # echo mem > /sys/power/state
> 
> This generates a lot of output and leads to the same suspend
> failure. This may be due to the out-of-tree nbcon-8250 driver I am
> using, which introduces extra irq_work usage within the driver.

Great catch!

> Could you please try the above boot arguments with your i.MX board?
>
> I am wondering if blocking the queueing of irq_work during suspend
> should be generally implemented rather than just printk avoiding it. It
> also seems to only be a problem for ARM32 so perhaps it needs to be
> addressed at the architecture level.

IMHO, we could not _quietly_ block irq_work during the suspend a
generic way. We would need to warn about it so that the callers are
aware of it. Then the callers would need to explicitly block it
to avoid the warning.

I want to say that explicitly blocking the irq_work in printk() looks
like the right way to go. I think that it would be needed anyway.

Best Regards,
Petr