[PATCH 0/2] Add HPET NMI Watchdog support

[PATCH 0/2] Add HPET NMI Watchdog support

[Alexander Graf]

The current NMI watchdog relies on performance counters and consistently
occupies one on each CPU. When running virtual machines, we want to pass
performance counters to virtual machines so they can make use of them.
In addition the host system wants to use performance counters to check
the system to identify when anything looks abnormal, such as split
locks.

That makes PMCs a precious resource. So any PMC we can free up is a PMC
we can use for something useful. That made me look at the NMI watchdog.

The PMC based NMI watchdog implementation does not actually need any
performance counting. It just needs a per-CPU NMI timer source. X86
systems can make anything that emits an interrupt descriptor (IOAPIC,
MSI(-X), etc) become an NMI source. So any time goes. Including the
HPET. And while they can't really operate per-CPU, in almost all cases
you only really want the NMI on *all* CPUs, rather than per-CPU.

So I took a stab at building an HPET based NMI watchdog. In my (QEMU
based) testing, it's fully functional and can successfully detect when
CPUs get stuck. It even survives suspend/resume cycles.

For now, its enablement is a config time option because the hardlockup
framework does not support dynamic switching of multiple detectors.
That's ok for our use case. But maybe something for the interested
reader to tackle eventually :).

You can enable the HPET watchdog by default by setting

  CONFIG_HARDLOCKUP_DETECTOR_HPET_DEFAULT=y

or passing "hpet=watchdog" to the kernel command line. When active, it
will emit a kernel log message to indicate it works:

  [    0.179176] hpet: HPET watchdog initialized on timer 0, GSI 2

The HPET can only be in either watchdog or generic mode. I am a bit
worried about IO-APIC pin allocation logic, so I opted to reuse the
generic timer pin. And that means I'm effectively breaking the normal
interrupt delivery path. so the easy way out was to say when watchdog is
active, PIT and HPET are not available as timer sources. Which is ok on
modern systems. There are way too many (unreliable) timer sources on x86
already. Trimming a few surely won't hurt.

I'm open to inputs on how to make the HPET multi-purpose though, in case
anyone feels strongly about it.

Alex

Alexander Graf (2):
  x86/ioapic: Add NMI delivery configuration helper
  hpet: Add HPET-based NMI watchdog support

 .../admin-guide/kernel-parameters.txt         |   5 +-
 arch/x86/Kconfig                              |  19 ++
 arch/x86/include/asm/io_apic.h                |   2 +
 arch/x86/kernel/apic/io_apic.c                |  32 ++++
 arch/x86/kernel/hpet.c                        | 172 ++++++++++++++++++
 arch/x86/kernel/i8253.c                       |   9 +
 drivers/char/hpet.c                           |   3 +
 include/linux/hpet.h                          |  14 ++
 8 files changed, 255 insertions(+), 1 deletion(-)

-- 
2.47.1




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

[PATCH 1/2] x86/ioapic: Add NMI delivery configuration helper

[Alexander Graf]

To implement an HPET based NMI watchdog, the HPET code will need to
reconfigure an IOAPIC pin to NMI mode. Add a function that allows driver
code to configure an IOAPIC pin for NMI delivery mode.

(Disclaimer: Some of this code was written with the help of Kiro, an AI
coding assistant)

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 arch/x86/include/asm/io_apic.h |  2 ++
 arch/x86/kernel/apic/io_apic.c | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 0d806513c4b3..58cfb338bf39 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -158,6 +158,8 @@ extern void mp_save_irq(struct mpc_intsrc *m);
 
 extern void disable_ioapic_support(void);
 
+extern int ioapic_set_nmi(u32 gsi, bool broadcast);
+
 extern void __init io_apic_init_mappings(void);
 extern unsigned int native_io_apic_read(unsigned int apic, unsigned int reg);
 extern void native_restore_boot_irq_mode(void);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 28f934f05a85..006f328929cd 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2951,6 +2951,38 @@ int mp_irqdomain_ioapic_idx(struct irq_domain *domain)
 	return (int)(long)domain->host_data;
 }
 
+/**
+ * ioapic_set_nmi - Configure an IOAPIC pin for NMI delivery
+ * @gsi: Global System Interrupt number
+ * @broadcast: true to broadcast to all CPUs, false to send to CPU 0 only
+ *
+ * Configures the specified GSI for NMI delivery mode.
+ *
+ * Returns 0 on success, negative error code on failure.
+ */
+int ioapic_set_nmi(u32 gsi, bool broadcast)
+{
+	struct IO_APIC_route_entry entry = { };
+	int ioapic_idx, pin;
+
+	ioapic_idx = mp_find_ioapic(gsi);
+	if (ioapic_idx < 0)
+		return -ENODEV;
+
+	pin = mp_find_ioapic_pin(ioapic_idx, gsi);
+	if (pin < 0)
+		return -ENODEV;
+
+	entry.delivery_mode = APIC_DELIVERY_MODE_NMI;
+	entry.destid_0_7 = broadcast ? 0xFF : 0;
+	entry.dest_mode_logical = 0;
+	entry.masked = 0;
+
+	ioapic_write_entry(ioapic_idx, pin, entry);
+
+	return 0;
+}
+
 const struct irq_domain_ops mp_ioapic_irqdomain_ops = {
 	.alloc		= mp_irqdomain_alloc,
 	.free		= mp_irqdomain_free,
-- 
2.47.1




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

[PATCH 1/2] x86/ioapic: Add NMI delivery configuration helper

[Alexander Graf]

To implement an HPET based NMI watchdog, the HPET code will need to
reconfigure an IOAPIC pin to NMI mode. Add a function that allows driver
code to configure an IOAPIC pin for NMI delivery mode.

The caller can choose whether to invoke NMIs on the BSP or broadcast on
all CPUs in the system.

(Disclaimer: Some of this code was written with the help of Kiro, an AI
coding assistant)

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 arch/x86/include/asm/io_apic.h |  2 ++
 arch/x86/kernel/apic/io_apic.c | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 0d806513c4b3..58cfb338bf39 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -158,6 +158,8 @@ extern void mp_save_irq(struct mpc_intsrc *m);
 
 extern void disable_ioapic_support(void);
 
+extern int ioapic_set_nmi(u32 gsi, bool broadcast);
+
 extern void __init io_apic_init_mappings(void);
 extern unsigned int native_io_apic_read(unsigned int apic, unsigned int reg);
 extern void native_restore_boot_irq_mode(void);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 28f934f05a85..5b303e5d2f3f 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2951,6 +2951,38 @@ int mp_irqdomain_ioapic_idx(struct irq_domain *domain)
 	return (int)(long)domain->host_data;
 }
 
+/**
+ * ioapic_set_nmi - Configure an IOAPIC pin for NMI delivery
+ * @gsi: Global System Interrupt number
+ * @broadcast: true to broadcast to all CPUs, false to send to CPU 0 only
+ *
+ * Configures the specified GSI for NMI delivery mode.
+ *
+ * Returns 0 on success, negative error code on failure.
+ */
+int ioapic_set_nmi(u32 gsi, bool broadcast)
+{
+	struct IO_APIC_route_entry entry = { };
+	int ioapic_idx, pin;
+
+	ioapic_idx = mp_find_ioapic(gsi);
+	if (ioapic_idx < 0)
+		return -ENODEV;
+
+	pin = mp_find_ioapic_pin(ioapic_idx, gsi);
+	if (pin < 0)
+		return -ENODEV;
+
+	entry.delivery_mode = APIC_DELIVERY_MODE_NMI;
+	entry.destid_0_7 = broadcast ? 0xFF : boot_cpu_physical_apicid;
+	entry.dest_mode_logical = 0;
+	entry.masked = 0;
+
+	ioapic_write_entry(ioapic_idx, pin, entry);
+
+	return 0;
+}
+
 const struct irq_domain_ops mp_ioapic_irqdomain_ops = {
 	.alloc		= mp_irqdomain_alloc,
 	.free		= mp_irqdomain_free,
-- 
2.47.1




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH 0/2] Add HPET NMI Watchdog support

[Alexander Graf]

On 02.02.26 18:43, Alexander Graf wrote:
> The current NMI watchdog relies on performance counters and consistently
> occupies one on each CPU. When running virtual machines, we want to pass
> performance counters to virtual machines so they can make use of them.
> In addition the host system wants to use performance counters to check
> the system to identify when anything looks abnormal, such as split
> locks.
>
> That makes PMCs a precious resource. So any PMC we can free up is a PMC
> we can use for something useful. That made me look at the NMI watchdog.
>
> The PMC based NMI watchdog implementation does not actually need any
> performance counting. It just needs a per-CPU NMI timer source. X86
> systems can make anything that emits an interrupt descriptor (IOAPIC,
> MSI(-X), etc) become an NMI source. So any time goes. Including the
> HPET. And while they can't really operate per-CPU, in almost all cases
> you only really want the NMI on *all* CPUs, rather than per-CPU.
>
> So I took a stab at building an HPET based NMI watchdog. In my (QEMU
> based) testing, it's fully functional and can successfully detect when
> CPUs get stuck. It even survives suspend/resume cycles.
>
> For now, its enablement is a config time option because the hardlockup
> framework does not support dynamic switching of multiple detectors.
> That's ok for our use case. But maybe something for the interested
> reader to tackle eventually :).
>
> You can enable the HPET watchdog by default by setting
>
>    CONFIG_HARDLOCKUP_DETECTOR_HPET_DEFAULT=y
>
> or passing "hpet=watchdog" to the kernel command line. When active, it
> will emit a kernel log message to indicate it works:
>
>    [    0.179176] hpet: HPET watchdog initialized on timer 0, GSI 2
>
> The HPET can only be in either watchdog or generic mode. I am a bit
> worried about IO-APIC pin allocation logic, so I opted to reuse the
> generic timer pin. And that means I'm effectively breaking the normal
> interrupt delivery path. so the easy way out was to say when watchdog is
> active, PIT and HPET are not available as timer sources. Which is ok on
> modern systems. There are way too many (unreliable) timer sources on x86
> already. Trimming a few surely won't hurt.
>
> I'm open to inputs on how to make the HPET multi-purpose though, in case
> anyone feels strongly about it.


Sorry for the resend. I caught an issue while sending out the series, 
hit ctrl-c before thinking and suddenly had a half sent series. Discard 
this one. Happy review on the real, full one :)


Alex




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH 1/2] x86/ioapic: Add NMI delivery configuration helper

[Thomas Gleixner]

On Mon, Feb 02 2026 at 17:48, Alexander Graf wrote:
> To implement an HPET based NMI watchdog, the HPET code will need to
> reconfigure an IOAPIC pin to NMI mode. Add a function that allows driver
> code to configure an IOAPIC pin for NMI delivery mode.

A function which violates all layering of the interrupt hierarchy...

> +/**
> + * ioapic_set_nmi - Configure an IOAPIC pin for NMI delivery
> + * @gsi: Global System Interrupt number
> + * @broadcast: true to broadcast to all CPUs, false to send to CPU 0 only
> + *
> + * Configures the specified GSI for NMI delivery mode.
> + *
> + * Returns 0 on success, negative error code on failure.
> + */
> +int ioapic_set_nmi(u32 gsi, bool broadcast)
> +{
> +	struct IO_APIC_route_entry entry = { };
> +	int ioapic_idx, pin;
> +
> +	ioapic_idx = mp_find_ioapic(gsi);
> +	if (ioapic_idx < 0)
> +		return -ENODEV;
> +
> +	pin = mp_find_ioapic_pin(ioapic_idx, gsi);
> +	if (pin < 0)
> +		return -ENODEV;
> +
> +	entry.delivery_mode = APIC_DELIVERY_MODE_NMI;
> +	entry.destid_0_7 = broadcast ? 0xFF : boot_cpu_physical_apicid;
> +	entry.dest_mode_logical = 0;
> +	entry.masked = 0;
> +
> +	ioapic_write_entry(ioapic_idx, pin, entry);

Q: How is that supposed to work with interrupt remapping?
A: Not at all.

Thanks,

        tglx

Re: [PATCH 1/2] x86/ioapic: Add NMI delivery configuration helper

[Alexander Graf]

On 03.02.26 11:08, Thomas Gleixner wrote:
> On Mon, Feb 02 2026 at 17:48, Alexander Graf wrote:
>> To implement an HPET based NMI watchdog, the HPET code will need to
>> reconfigure an IOAPIC pin to NMI mode. Add a function that allows driver
>> code to configure an IOAPIC pin for NMI delivery mode.
> A function which violates all layering of the interrupt hierarchy...


Yes, just like the device itself :). The HPET is magical.

Let me try and see whether I can just make the HPET logic require MSI 
(FSB) mode, so it can generate the NMI MSI message itself and post it 
without going through the IOAPIC in the first place. That's probably 
cleaner, more self contained and hence creates less layering violations 
and complexity in the long run.


>
>> +/**
>> + * ioapic_set_nmi - Configure an IOAPIC pin for NMI delivery
>> + * @gsi: Global System Interrupt number
>> + * @broadcast: true to broadcast to all CPUs, false to send to CPU 0 only
>> + *
>> + * Configures the specified GSI for NMI delivery mode.
>> + *
>> + * Returns 0 on success, negative error code on failure.
>> + */
>> +int ioapic_set_nmi(u32 gsi, bool broadcast)
>> +{
>> +     struct IO_APIC_route_entry entry = { };
>> +     int ioapic_idx, pin;
>> +
>> +     ioapic_idx = mp_find_ioapic(gsi);
>> +     if (ioapic_idx < 0)
>> +             return -ENODEV;
>> +
>> +     pin = mp_find_ioapic_pin(ioapic_idx, gsi);
>> +     if (pin < 0)
>> +             return -ENODEV;
>> +
>> +     entry.delivery_mode = APIC_DELIVERY_MODE_NMI;
>> +     entry.destid_0_7 = broadcast ? 0xFF : boot_cpu_physical_apicid;
>> +     entry.dest_mode_logical = 0;
>> +     entry.masked = 0;
>> +
>> +     ioapic_write_entry(ioapic_idx, pin, entry);
> Q: How is that supposed to work with interrupt remapping?
> A: Not at all.


... and yes, hopefully also gets us support for INTR if I manage to find 
the right abstraction.

Thanks a lot for the review!


Alex




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH 1/2] x86/ioapic: Add NMI delivery configuration helper

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

On Tue, 2026-02-03 at 11:08 +0100, Thomas Gleixner wrote:
> On Mon, Feb 02 2026 at 17:48, Alexander Graf wrote:
> > To implement an HPET based NMI watchdog, the HPET code will need to
> > reconfigure an IOAPIC pin to NMI mode. Add a function that allows driver
> > code to configure an IOAPIC pin for NMI delivery mode.
> 
> A function which violates all layering of the interrupt hierarchy...

I think you mean that this should be done by composing an MSI message
accordingly, and letting ioapic_setup_msg_from_msi() convert that into
an RTE for the I/O APIC without messing with the content? None of this
part should be specific to the I/O APIC?

And of course, if you're generating the MSI message you could just have
the HPET raise that directly instead of using a line interrupt, right?

> > +/**
> > + * ioapic_set_nmi - Configure an IOAPIC pin for NMI delivery
> > + * @gsi: Global System Interrupt number
> > + * @broadcast: true to broadcast to all CPUs, false to send to CPU 0 only
> > + *
> > + * Configures the specified GSI for NMI delivery mode.
> > + *
> > + * Returns 0 on success, negative error code on failure.
> > + */
> > +int ioapic_set_nmi(u32 gsi, bool broadcast)
> > +{
> > +	struct IO_APIC_route_entry entry = { };
> > +	int ioapic_idx, pin;
> > +
> > +	ioapic_idx = mp_find_ioapic(gsi);
> > +	if (ioapic_idx < 0)
> > +		return -ENODEV;
> > +
> > +	pin = mp_find_ioapic_pin(ioapic_idx, gsi);
> > +	if (pin < 0)
> > +		return -ENODEV;
> > +
> > +	entry.delivery_mode = APIC_DELIVERY_MODE_NMI;
> > +	entry.destid_0_7 = broadcast ? 0xFF : boot_cpu_physical_apicid;

How does that work in x2apic mode? Broadcast isn't 0xff there, is it?
And for systems with 15-bit MSI support you would also want to fill in
the extra 7 bits? But the MSI message composition function should
handle that for you anyway, right?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]