MTRR init sequence in Xen

MTRR init sequence in Xen

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 771 bytes --]

Just as a heads up: a hardware partner of SUSE has seen hard lockups
of the Linux kernel during boot on a new machine. This machine has
8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
attempts in MTRR initialization of the APs.

I have sent a small patch series to LKML which seems to fix the problem:
https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/

As Xen MTRR handling is taken from the Linux kernel, I guess the same
problem could happen in Xen, too.

As the hang always occurred while waiting for the lock, which is
serializing the single CPUs doing MTRR initialization, my solution was
to eliminate the lock, allowing all APs to init MTRRs in parallel.

Maybe we want to do the same in Xen.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: MTRR init sequence in Xen

[Roger Pau Monné]

On Thu, Jan 22, 2026 at 04:56:24PM +0100, Jürgen Groß wrote:
> Just as a heads up: a hardware partner of SUSE has seen hard lockups
> of the Linux kernel during boot on a new machine. This machine has
> 8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
> attempts in MTRR initialization of the APs.

Do you know why you get hard lockups?  Is some watchdog triggering on
Linux?  Otherwise I think it should just be slow, but ultimately
succeed?

> I have sent a small patch series to LKML which seems to fix the problem:
> https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/
> 
> As Xen MTRR handling is taken from the Linux kernel, I guess the same
> problem could happen in Xen, too.
> 
> As the hang always occurred while waiting for the lock, which is
> serializing the single CPUs doing MTRR initialization, my solution was
> to eliminate the lock, allowing all APs to init MTRRs in parallel.
> 
> Maybe we want to do the same in Xen.

Hm, yes, I think Xen would be equally affected with regards to being
contented on a lock while updating MTRRs.  The MTRR initialization is
deferred until all APs are up, and serialized on the
set_atomicity_lock lock.

Regards, Roger.

Re: MTRR init sequence in Xen

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 1358 bytes --]

On 22.01.26 18:18, Roger Pau Monné wrote:
> On Thu, Jan 22, 2026 at 04:56:24PM +0100, Jürgen Groß wrote:
>> Just as a heads up: a hardware partner of SUSE has seen hard lockups
>> of the Linux kernel during boot on a new machine. This machine has
>> 8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
>> attempts in MTRR initialization of the APs.
> 
> Do you know why you get hard lockups?  Is some watchdog triggering on
> Linux?  Otherwise I think it should just be slow, but ultimately
> succeed?

The NMI watchdog triggered.

> 
>> I have sent a small patch series to LKML which seems to fix the problem:
>> https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/
>>
>> As Xen MTRR handling is taken from the Linux kernel, I guess the same
>> problem could happen in Xen, too.
>>
>> As the hang always occurred while waiting for the lock, which is
>> serializing the single CPUs doing MTRR initialization, my solution was
>> to eliminate the lock, allowing all APs to init MTRRs in parallel.
>>
>> Maybe we want to do the same in Xen.
> 
> Hm, yes, I think Xen would be equally affected with regards to being
> contented on a lock while updating MTRRs.  The MTRR initialization is
> deferred until all APs are up, and serialized on the
> set_atomicity_lock lock.

Right.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: MTRR init sequence in Xen

[Andrew Cooper]

On 22/01/2026 3:56 pm, Jürgen Groß wrote:
> Just as a heads up: a hardware partner of SUSE has seen hard lockups
> of the Linux kernel during boot on a new machine. This machine has
> 8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
> attempts in MTRR initialization of the APs.
>
> I have sent a small patch series to LKML which seems to fix the problem:
> https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/
>
> As Xen MTRR handling is taken from the Linux kernel, I guess the same
> problem could happen in Xen, too.
>
> As the hang always occurred while waiting for the lock, which is
> serializing the single CPUs doing MTRR initialization, my solution was
> to eliminate the lock, allowing all APs to init MTRRs in parallel.
>
> Maybe we want to do the same in Xen.

I suspect Xen might be insulated by the fact that we don't have parallel
AP start (yet), so we don't have the whole system competing on the
spinlock at once.

Nevertheless, there's a lot of improvement available.  We still have a
lot of pre-64bit logic that we haven't purged yet.

~Andrew

Re: MTRR init sequence in Xen

[Roger Pau Monné]

On Thu, Jan 22, 2026 at 05:21:12PM +0000, Andrew Cooper wrote:
> On 22/01/2026 3:56 pm, Jürgen Groß wrote:
> > Just as a heads up: a hardware partner of SUSE has seen hard lockups
> > of the Linux kernel during boot on a new machine. This machine has
> > 8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
> > attempts in MTRR initialization of the APs.
> >
> > I have sent a small patch series to LKML which seems to fix the problem:
> > https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/
> >
> > As Xen MTRR handling is taken from the Linux kernel, I guess the same
> > problem could happen in Xen, too.
> >
> > As the hang always occurred while waiting for the lock, which is
> > serializing the single CPUs doing MTRR initialization, my solution was
> > to eliminate the lock, allowing all APs to init MTRRs in parallel.
> >
> > Maybe we want to do the same in Xen.
> 
> I suspect Xen might be insulated by the fact that we don't have parallel
> AP start (yet), so we don't have the whole system competing on the
> spinlock at once.

Oh, I think I've misunderstood the issue.  Linux is doing MTRR init in
the AP startup path, and so if it takes too long Linux will report
that the AP has failed to start.

This is not an issue on Xen because MTRR initialization is deferred
until all APs are up, and hence is not part of the timed AP start
path.  This optimization was done in:

0d22c8d92c6c x86: CPU synchronization while doing MTRR register update

So even if we did parallel AP startup we won't likely be affected,
because we would still defer the MTRR setup until all APs are up.

Regards, Roger.

Re: MTRR init sequence in Xen

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 1935 bytes --]

On 22.01.26 18:36, Roger Pau Monné wrote:
> On Thu, Jan 22, 2026 at 05:21:12PM +0000, Andrew Cooper wrote:
>> On 22/01/2026 3:56 pm, Jürgen Groß wrote:
>>> Just as a heads up: a hardware partner of SUSE has seen hard lockups
>>> of the Linux kernel during boot on a new machine. This machine has
>>> 8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
>>> attempts in MTRR initialization of the APs.
>>>
>>> I have sent a small patch series to LKML which seems to fix the problem:
>>> https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/
>>>
>>> As Xen MTRR handling is taken from the Linux kernel, I guess the same
>>> problem could happen in Xen, too.
>>>
>>> As the hang always occurred while waiting for the lock, which is
>>> serializing the single CPUs doing MTRR initialization, my solution was
>>> to eliminate the lock, allowing all APs to init MTRRs in parallel.
>>>
>>> Maybe we want to do the same in Xen.
>>
>> I suspect Xen might be insulated by the fact that we don't have parallel
>> AP start (yet), so we don't have the whole system competing on the
>> spinlock at once.
> 
> Oh, I think I've misunderstood the issue.  Linux is doing MTRR init in
> the AP startup path, and so if it takes too long Linux will report
> that the AP has failed to start.

No, Linux is deferring the MTRRs until all APs are up, just like Xen
(or Xen does it like Linux).

> 
> This is not an issue on Xen because MTRR initialization is deferred
> until all APs are up, and hence is not part of the timed AP start
> path.  This optimization was done in:
> 
> 0d22c8d92c6c x86: CPU synchronization while doing MTRR register update
> 
> So even if we did parallel AP startup we won't likely be affected,
> because we would still defer the MTRR setup until all APs are up.

We will be affected, as its the deferred MTRR setup which is the
problem.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: MTRR init sequence in Xen

[Roger Pau Monné]

On Thu, Jan 22, 2026 at 08:24:11PM +0100, Jürgen Groß wrote:
> On 22.01.26 18:36, Roger Pau Monné wrote:
> > On Thu, Jan 22, 2026 at 05:21:12PM +0000, Andrew Cooper wrote:
> > > On 22/01/2026 3:56 pm, Jürgen Groß wrote:
> > > > Just as a heads up: a hardware partner of SUSE has seen hard lockups
> > > > of the Linux kernel during boot on a new machine. This machine has
> > > > 8 NUMA nodes and 960 CPUs. The hang occurs in roughly 1.5% of the boot
> > > > attempts in MTRR initialization of the APs.
> > > > 
> > > > I have sent a small patch series to LKML which seems to fix the problem:
> > > > https://lore.kernel.org/lkml/20260121141106.755458-1-jgross@suse.com/
> > > > 
> > > > As Xen MTRR handling is taken from the Linux kernel, I guess the same
> > > > problem could happen in Xen, too.
> > > > 
> > > > As the hang always occurred while waiting for the lock, which is
> > > > serializing the single CPUs doing MTRR initialization, my solution was
> > > > to eliminate the lock, allowing all APs to init MTRRs in parallel.
> > > > 
> > > > Maybe we want to do the same in Xen.
> > > 
> > > I suspect Xen might be insulated by the fact that we don't have parallel
> > > AP start (yet), so we don't have the whole system competing on the
> > > spinlock at once.
> > 
> > Oh, I think I've misunderstood the issue.  Linux is doing MTRR init in
> > the AP startup path, and so if it takes too long Linux will report
> > that the AP has failed to start.
> 
> No, Linux is deferring the MTRRs until all APs are up, just like Xen
> (or Xen does it like Linux).
> 
> > 
> > This is not an issue on Xen because MTRR initialization is deferred
> > until all APs are up, and hence is not part of the timed AP start
> > path.  This optimization was done in:
> > 
> > 0d22c8d92c6c x86: CPU synchronization while doing MTRR register update
> > 
> > So even if we did parallel AP startup we won't likely be affected,
> > because we would still defer the MTRR setup until all APs are up.
> 
> We will be affected, as its the deferred MTRR setup which is the
> problem.

If it's the watchdog NMI then than won't be possible on Xen, as the
watchdog is setup after the MTRR synchronization step.  We should
however fix it even if it's not a fatal issue on Xen.  I assume the
avoidance of locking will make a very noticeable performance
difference in boot times.

Thanks, Roger.