System Secondary HPETs

System Secondary HPETs

[Andrew Cooper]

Recently, I debugged an issue with a failure to kexec because the kdump
kernel could not connect up the timer interrupt through the IOAPIC.

In reality, the problem was not the interrupt routing, but that the
timer itself was not generating interrupts. , yet advertising it in the
DSDT as an HPET device.

The root problem is that the system is not advertising the primary HPET
in an HPET ACPI table...

[root@bear ~]# xl dmesg | grep -i hpet
(XEN) [    2.662941] CPUIDLE: disabled due to no HPET. Force enable with
'cpuidle'.

yet advertising it in the DSDT to dom0:

[root@bear ~]# dmesg | grep -i hpet
[    0.038884] pci 0000:00:1f.0: Force enabled HPET at 0xfed00000
[    0.053055] hpet clockevent registered
[    0.053055] HPET: 3 timers in total, 0 timers will be used for
per-cpu timer
[    0.053055] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.053055] hpet0: 3 comparators, 64-bit 14.318180 MHz counter
[    0.155730] rtc_cmos 00:03: alarms up to one month, y3k, 114 bytes
nvram, hpet irqs

Device 00:1f.0 is "ISA bridge: Intel Corporation 631xESB/632xESB/3100
Chipset LPC Interface Controller (rev 09)"

Dom0 then configures the HPET for its own use, and disables legacy
broadcast mode.  When crashing, the kdump kernel no longer gets legacy
interrupts, as the HPET is still configured in oneshot mode for dom0. 
Booting dom0 with "hpet=disable" causes it to never reprogram the HPET
out of legacy broadcast mode, and allows the kdump kernel to function
correctly.

Clearly, one problem here is the firmware not writing a primary HPET
ACPI table, but this does highlight another problem.

There is no possible way dom0 could use any HPETs it finds; dom0 does
not control the IDTs or interrupt remapping tables.  Allowing dom0 to
program any of the HPETs can, at best, only result in spurious
interrupts from Xens point of view, and at worst, interrupt remapping
faults.

I believe that dom0 should never attempt to use any HPET it can find. 
It is less clear whether forwarding the details to Xen would be useful
or not.  The range itself cannot be protected as it is only 1K long, but
Xen might be able to retroactively enable cpuidle.  Having said that,
Xen's ability to use this particular HPET appears to be stunted
somewhat, given an AML enable bit which starts disabled.

~Andrew

Re: System Secondary HPETs

[Jan Beulich]

>>> On 02.06.14 at 19:08, <andrew.cooper3@citrix.com> wrote:
> Recently, I debugged an issue with a failure to kexec because the kdump
> kernel could not connect up the timer interrupt through the IOAPIC.
> 
> In reality, the problem was not the interrupt routing, but that the
> timer itself was not generating interrupts. , yet advertising it in the
> DSDT as an HPET device.
> 
> The root problem is that the system is not advertising the primary HPET
> in an HPET ACPI table...
> 
> [root@bear ~]# xl dmesg | grep -i hpet
> (XEN) [    2.662941] CPUIDLE: disabled due to no HPET. Force enable with
> 'cpuidle'.
> 
> yet advertising it in the DSDT to dom0:

Not exactly:

> [root@bear ~]# dmesg | grep -i hpet
> [    0.038884] pci 0000:00:1f.0: Force enabled HPET at 0xfed00000

This line indicates that the chipset was recognized without any help
from the ACPI tables - see the various *_force_enable_hpet() in
arch/x86/kernel/quirks.c. This may want to be overridden by Xen
specific initialization code (short of hiding the HPET_TIMER config
option as we do in the legacy kernel).

Jan