arm64: pseudo NMI bootconfig question

arm64: pseudo NMI bootconfig question

[Breno Leitao]

Hello,

In most major Linux distributions, it’s common to have a separate debug kernel
build. This variant is compiled with additional debug configuration options to
provide deeper observability and easier debugging, typically at the expense of
some performance.

This approach is also used in several companies, including mine. The debug
flavor is usually a drop-in replacement deployed when investigating
issues, allowing quicker identification before more detailed manual
debugging.

I’m currently debugging some hung tasks using the debug kernel flavor and
noticed that backtraces weren’t printed correctly because PSEUDO NMI isn’t
enabled.

AFAIK, to enable PSEUDO NMI on ARM, the following are required:

 cmdline: irqchip.gicv3_pseudo_nmi=1
 config: CONFIG_ARM64_PSEUDO_NMI=y

I wanted to have pseudo NMI enabled by default in the debug kernel
(without relying on kernel command-line parameters), but this isn’t
possible today. The reason is that `irqchip.gicv3_pseudo_nmi` is an early
parameter, which can’t be set through bootconfig, so, independent of my
.config, I need to hack cmdline arguments to have the debuggability
I need.

Question:

Would it make sense to provide an option to enable pseudo NMI in certain
kernel configuration without requiring an extra command-line parameter?

Re: arm64: pseudo NMI bootconfig question

[Marc Zyngier]

On Wed, 12 Nov 2025 14:05:43 +0000,
Breno Leitao <leitao@debian.org> wrote:
> 
> Hello,
> 
> In most major Linux distributions, it’s common to have a separate debug kernel
> build. This variant is compiled with additional debug configuration options to
> provide deeper observability and easier debugging, typically at the expense of
> some performance.
> 
> This approach is also used in several companies, including mine. The debug
> flavor is usually a drop-in replacement deployed when investigating
> issues, allowing quicker identification before more detailed manual
> debugging.
> 
> I’m currently debugging some hung tasks using the debug kernel flavor and
> noticed that backtraces weren’t printed correctly because PSEUDO NMI isn’t
> enabled.
> 
> AFAIK, to enable PSEUDO NMI on ARM, the following are required:
> 
>  cmdline: irqchip.gicv3_pseudo_nmi=1
>  config: CONFIG_ARM64_PSEUDO_NMI=y
> 
> I wanted to have pseudo NMI enabled by default in the debug kernel
> (without relying on kernel command-line parameters), but this isn’t
> possible today. The reason is that `irqchip.gicv3_pseudo_nmi` is an early
> parameter, which can’t be set through bootconfig, so, independent of my
> .config, I need to hack cmdline arguments to have the debuggability
> I need.
> 
> Question:
> 
> Would it make sense to provide an option to enable pseudo NMI in certain
> kernel configuration without requiring an extra command-line parameter?

[I wasn't asked, but I'll give my answer anyway]

The short answer is no. The long answer is that there is so much
broken HW out there that dies a painful death when enabling pseudo-NMI
that is isn't practical to do so. It also brings a measurable
overhead to some of the most frequent operations (interrupt masking).

Until someone weeds out the broken machines (in some cases, by prying
them from my cold dead hands), the requirement for the option is
likely to stick.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: arm64: pseudo NMI bootconfig question

[Masami Hiramatsu]

On Wed, 12 Nov 2025 06:05:43 -0800
Breno Leitao <leitao@debian.org> wrote:

> Hello,
> 
> In most major Linux distributions, it’s common to have a separate debug kernel
> build. This variant is compiled with additional debug configuration options to
> provide deeper observability and easier debugging, typically at the expense of
> some performance.
> 
> This approach is also used in several companies, including mine. The debug
> flavor is usually a drop-in replacement deployed when investigating
> issues, allowing quicker identification before more detailed manual
> debugging.
> 
> I’m currently debugging some hung tasks using the debug kernel flavor and
> noticed that backtraces weren’t printed correctly because PSEUDO NMI isn’t
> enabled.
> 
> AFAIK, to enable PSEUDO NMI on ARM, the following are required:
> 
>  cmdline: irqchip.gicv3_pseudo_nmi=1
>  config: CONFIG_ARM64_PSEUDO_NMI=y
> 
> I wanted to have pseudo NMI enabled by default in the debug kernel
> (without relying on kernel command-line parameters), but this isn’t
> possible today. The reason is that `irqchip.gicv3_pseudo_nmi` is an early
> parameter, which can’t be set through bootconfig, so, independent of my
> .config, I need to hack cmdline arguments to have the debuggability
> I need.

Yeah, that parameter is for early architecture initialization,
which is done before parsing bootconfig.

Can you try building kernel with:

(arm)
CONFIG_CMDLINE="irqchip.gicv3_pseudo_nmi=1"
CONFIG_CMDLINE_EXTEND=y

(arm64)
cmdline=$(cat /proc/cmdline)
CONFIG_CMDLINE="irqchip.gicv3_pseudo_nmi=1 ${cmdline}"
CMDLINE_FORCE=y

BTW I wonder why you can not configure your bootloader to pass it.

Thank you,

> 
> Question:
> 
> Would it make sense to provide an option to enable pseudo NMI in certain
> kernel configuration without requiring an extra command-line parameter?

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: arm64: pseudo NMI bootconfig question

[Breno Leitao]

Helo Marc,

On Wed, Nov 12, 2025 at 02:21:22PM +0000, Marc Zyngier wrote:
> On Wed, 12 Nov 2025 14:05:43 +0000,
> Breno Leitao <leitao@debian.org> wrote:
...
> > Question:
> > 
> > Would it make sense to provide an option to enable pseudo NMI in certain
> > kernel configuration without requiring an extra command-line parameter?
> 
> [I wasn't asked, but I'll give my answer anyway]
> 
> The short answer is no. The long answer is that there is so much
> broken HW out there that dies a painful death when enabling pseudo-NMI
> that is isn't practical to do so.

That’s helpful to know. I hadn’t realized there were potential issues
with hardware implementation, which explains why it isn’t
straightforward to enable. Thanks for the additional context.

> It also brings a measurable overhead to some of the most frequent
> operations (interrupt masking).

For debug kernel variants, performance overhead isn’t a concern since
they’re only used in debugging scenarios—not in production. So, it’s
fine if things run slower when the debug configuration is active, at
least from my perspective.

Thanks for jumping in!
--breno

Re: arm64: pseudo NMI bootconfig question

[Breno Leitao]

Hello Masami,

On Thu, Nov 13, 2025 at 01:34:03PM +0900, Masami Hiramatsu wrote:
> > I wanted to have pseudo NMI enabled by default in the debug kernel
> > (without relying on kernel command-line parameters), but this isn’t
> > possible today. The reason is that `irqchip.gicv3_pseudo_nmi` is an early
> > parameter, which can’t be set through bootconfig, so, independent of my
> > .config, I need to hack cmdline arguments to have the debuggability
> > I need.
> 
> Yeah, that parameter is for early architecture initialization,
> which is done before parsing bootconfig.
> 
> Can you try building kernel with:
> 
> (arm64)
> cmdline=$(cat /proc/cmdline)
> CONFIG_CMDLINE="irqchip.gicv3_pseudo_nmi=1 ${cmdline}"
> CMDLINE_FORCE=y

This option isn’t practical for my use case since I maintain a generic
debug kernel that needs to support multiple workloads and services, each
with their own kernel command line. I’d need something similar to the
old CMDLINE_EXTEND feature (which was removed some time ago), or perhaps
a mechanism like CMDLINE_PREPEND or CMDLINE_APPEND, which were never
introduced.

Another possible solution would be to load bootconfig earlier in the
boot process so that early parameters can be defined within bootconfig.
Petr suggested this approach some time back, but it doesn’t appear to
have made it upstream.

https://lore.kernel.org/all/20231123194106.08f5832f558fe806b1fd8098@kernel.org/

I’m not fully up to speed on the details of this change, so I want to
ask directly: Would Petr’s approach—allowing early parameters to be set
via bootconfig make sense from a bootconfig design perspective?

> BTW I wonder why you can not configure your bootloader to pass it.

The main issue is that changing cmdline parameters increases the
confusion for users, and updating the kernel command line isn’t simple
in a heterogeneous environment.

My objective is to provide a drop-in kernel replacement, so the user
only needs to swap the kernel binary, gaining additional debugging
capabilities without modifying anything else.

Additionally, it’s common for the same kernel version to be used across
different bootloaders, which makes distributing and installing debug
kernels even more complicated if additional changes (such as appending
configurations to the cmdline) are required.

Thanks for your help!
--breno

Re: arm64: pseudo NMI bootconfig question

[Will Deacon]

On Thu, Nov 13, 2025 at 01:36:48AM -0800, Breno Leitao wrote:
> Helo Marc,
> 
> On Wed, Nov 12, 2025 at 02:21:22PM +0000, Marc Zyngier wrote:
> > On Wed, 12 Nov 2025 14:05:43 +0000,
> > Breno Leitao <leitao@debian.org> wrote:
> ...
> > > Question:
> > > 
> > > Would it make sense to provide an option to enable pseudo NMI in certain
> > > kernel configuration without requiring an extra command-line parameter?
> > 
> > [I wasn't asked, but I'll give my answer anyway]
> > 
> > The short answer is no. The long answer is that there is so much
> > broken HW out there that dies a painful death when enabling pseudo-NMI
> > that is isn't practical to do so.
> 
> That’s helpful to know. I hadn’t realized there were potential issues
> with hardware implementation, which explains why it isn’t
> straightforward to enable. Thanks for the additional context.

FWIW, I've also run into platforms where the _hardware_ is fine but the
TZ integration can't handle Linux using priority masking properly.

Will