Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Linux regression tracking (Thorsten Leemhuis)]

H! Lyude, Thomas, what's the status here? From here it looks like we
were close to a fix, but then it turned out to be a bad fix -- and
afterwards nothing much seems to have happened. Did it fall through the
cracks, or was this already fixed and I just missed that?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

On 02.05.24 12:33, Mario Limonciello wrote:
> On 4/25/2024 16:42, Thomas Gleixner wrote:
>> On Thu, Apr 25 2024 at 11:56, Lyude Paul wrote:
>>> On Thu, 2024-04-25 at 04:11 +0200, Thomas Gleixner wrote:
>>>>
>>>> Can you please boot a kernel with the commit in question reverted and
>>>> add 'possible_cpus=8' to the kernel command line?
>>>>
>>>> In theory this should fail too.
>>>
>>> Yep - tried booting a kernel with f0551af0213 reverted and
>>> possible_cpus=8, it definitely looks like that crashes things as well
>>> in the same way.
>>
>> Good. That means it's a problem which existed before but went unnoticed.
>>
>>> Also - it scrolled off the screen before I had a chance to write it
>>> down, but I'm -fairly- sure I saw some sort of complaint about "16 [or
>>> some double digit number] processors exceeds max number of 8". Which
>>> is quite interesting, as this is definitely just a quad core ryzen
>>> processor with hyperthreading - so there should only be 8 threads.
>>
>> Right, that's what we saw with the debug patch. The ACPI/MADT table
>> is clearly bonkers. The effect of it is that it pretends that the system
>> has 16 possible CPUs:
>>
>>      [    0.089381] CPU topo: Allowing 8 present CPUs plus 8 hotplug CPUs
>>
>> Which in turn changes the sizing of the per CPU data and affects some
>> other details which depend on the number of possible CPUs.
> 
> At least this aspect of this I suspect is caused by commit
> fed8d8773b8ea68ad99d9eee8c8343bef9da2c2c.
> 
> If you try reverting that I expect the "hotplug CPUs" disappear.
> 
>>
>> But that should not matter at all because the system scaling should be
>> sufficient with 8 CPUs, but it does not for some completely non-obvious
>> reasons.
>>
>> Can you please try to increase possible_cpus=N on the command line one
>> by one and check when it actually starts to "work" again.
>>
>> One other thing to try is to boot with 'possible_cpus=8' and
>> 'intremap=off' and see whether that makes a difference.
>>
>> I really have no idea where to look and not having the early boot
>> messages in case of the fail is not helpful as I can't add meaningful
>> debug to it.
>>
>> I just checked: the motherboard has a serial port, so it would be
>> extremly helpful to hook up a serial cable to this thing and enable
>> serial console on the kernel command line. That way we might eventually
>> see information which is emitted before it fails to validate the timer
>> interrupt.
>>

--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten Leemhuis) wrote:
> H! Lyude, Thomas, what's the status here? From here it looks like we
> were close to a fix, but then it turned out to be a bad fix -- and
> afterwards nothing much seems to have happened. Did it fall through the
> cracks, or was this already fixed and I just missed that?

I'm waiting for more data still.

Thanks,

        tglx

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

Yes sorry - I was out of work for a bit but I'm back now and can get
you the info today - thanks for the patience ♥

On Wed, 2024-05-08 at 12:30 +0200, Thomas Gleixner wrote:
> On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten
> Leemhuis) wrote:
> > H! Lyude, Thomas, what's the status here? From here it looks like
> > we
> > were close to a fix, but then it turned out to be a bad fix -- and
> > afterwards nothing much seems to have happened. Did it fall through
> > the
> > cracks, or was this already fixed and I just missed that?
> 
> I'm waiting for more data still.
> 
> Thanks,
> 
>         tglx
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

Ok - so re the serial port comment (which I missed the first time I
read your message, oops!): I can totally get a serial console on this
machine, I happen to already have a serial port hooked up to the
motherboard. I'm not sure where my USB to RS232 adapter to connect to
it on my laptop is though, so I put in an order for a new one and it
should arrive tomorrow.

Regarding the test results - I tried possible_cpus all the way up to 17
and nothing got the machine to boot. However, possible_cpus=8
intremap=off did get the machine to boot successfully

On Wed, 2024-05-08 at 17:02 -0400, Lyude Paul wrote:
> Yes sorry - I was out of work for a bit but I'm back now and can get
> you the info today - thanks for the patience ♥
> 
> On Wed, 2024-05-08 at 12:30 +0200, Thomas Gleixner wrote:
> > On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten
> > Leemhuis) wrote:
> > > H! Lyude, Thomas, what's the status here? From here it looks like
> > > we
> > > were close to a fix, but then it turned out to be a bad fix --
> > > and
> > > afterwards nothing much seems to have happened. Did it fall
> > > through
> > > the
> > > cracks, or was this already fixed and I just missed that?
> > 
> > I'm waiting for more data still.
> > 
> > Thanks,
> > 
> >         tglx
> > 
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

On Wed, May 08 2024 at 19:21, Lyude Paul wrote:
> Regarding the test results - I tried possible_cpus all the way up to 17
> and nothing got the machine to boot. However, possible_cpus=8
> intremap=off did get the machine to boot successfully

Oh. That's interesting.

Does v6.9 (released yesterday) boot with just 'intremap=off' too?

Thanks,

        tglx

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

On Mon, 2024-05-13 at 16:08 +0200, Thomas Gleixner wrote:
> On Wed, May 08 2024 at 19:21, Lyude Paul wrote:
> > Regarding the test results - I tried possible_cpus all the way up to 17
> > and nothing got the machine to boot. However, possible_cpus=8
> > intremap=off did get the machine to boot successfully
> 
> Oh. That's interesting.
> 
> Does v6.9 (released yesterday) boot with just 'intremap=off' too?

Yes - it still boots. As well I finally got the serial console adapter in, but
I don't see any additional output:

[    0.146896] printk: legacy console [tty0] enabled
[    0.147433] printk: legacy console [ttyS0] enabled
[    1.951453] ACPI: Core revision 20230628
[    1.955466] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
[    1.964698] APIC: Switch to symmetric I/O mode setup
[    1.970973] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
[    1.978491] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    1.996945] Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
[    2.005376] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.9.0Lyude-Test+ #7
[    2.012248] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.I0 01/22/2019
[    2.020160] Call Trace:
[    2.022584]  <TASK>
[    2.024666]  dump_stack_lvl+0x5d/0x80
[    2.028499]  panic+0x118/0x2c8
[    2.031522]  ? timer_irq_works+0x54/0xf0
[    2.035408]  panic_if_irq_remap.cold+0x5/0x9
[    2.039631]  setup_IO_APIC+0x398/0x860
[    2.043475]  ? _raw_spin_unlock_irqrestore+0x1d/0x40
[    2.048393]  ? clear_IO_APIC_pin+0x125/0x1e0
[    2.052617]  apic_intr_mode_init+0xcc/0xe0
[    2.056672]  x86_late_time_init+0x24/0x40
[    2.060820]  start_kernel+0x8be/0x960
[    2.064446]  x86_64_start_reservations+0x24/0x30
[    2.069015]  x86_64_start_kernel+0x95/0xa0
[    2.073069]  common_startup_64+0x13e/0x141
[    2.077280]  </TASK>
[    2.079447] ---[ end Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC ]---


> 
> Thanks,
> 
>         tglx
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

Oh! I am not sure what changed, but I realized that you might also want to see
the serial output from the debugging patch that you had sent me a while ago.
So I just built that against the 6.9 kernel and it actually seems to boot
without needing me to pass intremap=off. So it seems like that might actually
be a fix!

On Mon, 2024-05-13 at 19:18 -0400, Lyude Paul wrote:
> On Mon, 2024-05-13 at 16:08 +0200, Thomas Gleixner wrote:
> > On Wed, May 08 2024 at 19:21, Lyude Paul wrote:
> > > Regarding the test results - I tried possible_cpus all the way up to 17
> > > and nothing got the machine to boot. However, possible_cpus=8
> > > intremap=off did get the machine to boot successfully
> > 
> > Oh. That's interesting.
> > 
> > Does v6.9 (released yesterday) boot with just 'intremap=off' too?
> 
> Yes - it still boots. As well I finally got the serial console adapter in, but
> I don't see any additional output:
> 
> [    0.146896] printk: legacy console [tty0] enabled
> [    0.147433] printk: legacy console [ttyS0] enabled
> [    1.951453] ACPI: Core revision 20230628
> [    1.955466] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
> [    1.964698] APIC: Switch to symmetric I/O mode setup
> [    1.970973] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
> [    1.978491] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> [    1.996945] Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
> [    2.005376] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.9.0Lyude-Test+ #7
> [    2.012248] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.I0 01/22/2019
> [    2.020160] Call Trace:
> [    2.022584]  <TASK>
> [    2.024666]  dump_stack_lvl+0x5d/0x80
> [    2.028499]  panic+0x118/0x2c8
> [    2.031522]  ? timer_irq_works+0x54/0xf0
> [    2.035408]  panic_if_irq_remap.cold+0x5/0x9
> [    2.039631]  setup_IO_APIC+0x398/0x860
> [    2.043475]  ? _raw_spin_unlock_irqrestore+0x1d/0x40
> [    2.048393]  ? clear_IO_APIC_pin+0x125/0x1e0
> [    2.052617]  apic_intr_mode_init+0xcc/0xe0
> [    2.056672]  x86_late_time_init+0x24/0x40
> [    2.060820]  start_kernel+0x8be/0x960
> [    2.064446]  x86_64_start_reservations+0x24/0x30
> [    2.069015]  x86_64_start_kernel+0x95/0xa0
> [    2.073069]  common_startup_64+0x13e/0x141
> [    2.077280]  </TASK>
> [    2.079447] ---[ end Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC ]---
> 
> 
> > 
> > Thanks,
> > 
> >         tglx
> > 
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

Lyude!

On Mon, May 13 2024 at 19:32, Lyude Paul wrote:
> Oh! I am not sure what changed, but I realized that you might also want to see
> the serial output from the debugging patch that you had sent me a while ago.
> So I just built that against the 6.9 kernel and it actually seems to boot
> without needing me to pass intremap=off. So it seems like that might actually
> be a fix!

Which one of the debug patches did you use?

>> Yes - it still boots. As well I finally got the serial console adapter in, but
>> I don't see any additional output:

That's fine, but now I can provide you debug patches which dump
information during early boot.

Can you please provide the output of 'cpuid -r' ?

Thanks,

        tglx

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

On Tue, 2024-05-14 at 10:25 +0200, Thomas Gleixner wrote:
> Lyude!
> 
> Which one of the debug patches did you use?

The one you sent on 4/18, when you also asked me for the output of
/sys/kernel/debug/x86/topo/

(I thought I remembered that patch not booting on previous kernels but maybe
I'm misremembering)

> 
> > > Yes - it still boots. As well I finally got the serial console adapter in, but
> > > I don't see any additional output:
> 
> That's fine, but now I can provide you debug patches which dump
> information during early boot.
> 
> Can you please provide the output of 'cpuid -r' ?
> 

CPU 0:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x00080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x00000000 ebx=0x00000100 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 1:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x01080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x00000001 ebx=0x00000100 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 2:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x02080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x00000002 ebx=0x00000101 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 3:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x03080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x00000003 ebx=0x00000101 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 4:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x08080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x00000008 ebx=0x00000104 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 5:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x09080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x00000009 ebx=0x00000104 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 6:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x0a080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x0000000a ebx=0x00000105 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 7:
   0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x00000001 0x00: eax=0x00800f11 ebx=0x0b080800 ecx=0x7ed8320b edx=0x178bfbff
   0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
   0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
   0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
   0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
   0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
   0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
   0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
   0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
   0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
   0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
   0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
   0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
   0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
   0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
   0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
   0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
   0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
   0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
   0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
   0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
   0x8000001e 0x00: eax=0x0000000b ebx=0x00000105 ecx=0x00000000 edx=0x00000000
   0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
   0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
   0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000


> Thanks,
> 
>         tglx
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

Lyude!

On Wed, May 15 2024 at 19:15, Lyude Paul wrote:
> On Tue, 2024-05-14 at 10:25 +0200, Thomas Gleixner wrote:
>> 
>> Which one of the debug patches did you use?
>
> The one you sent on 4/18, when you also asked me for the output of
> /sys/kernel/debug/x86/topo/
>
> (I thought I remembered that patch not booting on previous kernels but maybe
> I'm misremembering)

It booted with the -rc kernel too. :)

I found an interesting and probably related difference in the boot logs
though. Up to 8 possible CPUs the kernel uses logical destination mode
for the APIC. With more than 8 it uses physical destination mode.

Can you please test the patch below on top of 6.9 and validate that it
boots w/o any magic command line parameter?

Thanks,

        tglx
---
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 66fd4b2a37a3..db5e93a7c194 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1883,6 +1883,8 @@ static inline void try_to_enable_x2apic(int remap_mode) { }
 static inline void __x2apic_enable(void) { }
 #endif /* !CONFIG_X86_X2APIC */
 
+int irq_remap_mode = -1;
+
 void __init enable_IR_x2apic(void)
 {
 	unsigned long flags;
@@ -1915,6 +1917,8 @@ void __init enable_IR_x2apic(void)
 
 	if (ir_stat < 0)
 		restore_ioapic_entries();
+	else
+		irq_remap_mode = ir_stat;
 	legacy_pic->restore_mask();
 	local_irq_restore(flags);
 }
diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
index f37ad3392fec..5cc386db3557 100644
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -12,6 +12,7 @@
 #include <linux/export.h>
 #include <linux/acpi.h>
 
+#include <asm/irq_remapping.h>
 #include <asm/jailhouse_para.h>
 #include <asm/apic.h>
 
@@ -130,7 +131,8 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
 
 static int physflat_probe(void)
 {
-	return apic == &apic_physflat || num_possible_cpus() > 8 || jailhouse_paravirt();
+	return apic == &apic_physflat || irq_remap_mode == IRQ_REMAP_XAPIC_MODE ||
+		num_possible_cpus() > 8 || jailhouse_paravirt();
 }
 
 static struct apic apic_physflat __ro_after_init = {
diff --git a/arch/x86/kernel/apic/local.h b/arch/x86/kernel/apic/local.h
index 842fe28496be..f633ab6dfa9f 100644
--- a/arch/x86/kernel/apic/local.h
+++ b/arch/x86/kernel/apic/local.h
@@ -22,6 +22,8 @@ void x2apic_send_IPI_allbutself(int vector);
 void x2apic_send_IPI_self(int vector);
 extern u32 x2apic_max_apicid;
 
+extern int irq_remap_mode;
+
 /* IPI */
 
 DECLARE_STATIC_KEY_FALSE(apic_use_ipi_shorthand);

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

Awesome! This patch does seem to make the system boot, thank you for your help
<3

Tested-by: Lyude Paul <lyude@redhat.com>

On Thu, 2024-05-16 at 15:38 +0200, Thomas Gleixner wrote:
> Lyude!
> 
> On Wed, May 15 2024 at 19:15, Lyude Paul wrote:
> > On Tue, 2024-05-14 at 10:25 +0200, Thomas Gleixner wrote:
> > > 
> > > Which one of the debug patches did you use?
> > 
> > The one you sent on 4/18, when you also asked me for the output of
> > /sys/kernel/debug/x86/topo/
> > 
> > (I thought I remembered that patch not booting on previous kernels but maybe
> > I'm misremembering)
> 
> It booted with the -rc kernel too. :)
> 
> I found an interesting and probably related difference in the boot logs
> though. Up to 8 possible CPUs the kernel uses logical destination mode
> for the APIC. With more than 8 it uses physical destination mode.
> 
> Can you please test the patch below on top of 6.9 and validate that it
> boots w/o any magic command line parameter?
> 
> Thanks,
> 
>         tglx
> ---
> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
> index 66fd4b2a37a3..db5e93a7c194 100644
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -1883,6 +1883,8 @@ static inline void try_to_enable_x2apic(int remap_mode) { }
>  static inline void __x2apic_enable(void) { }
>  #endif /* !CONFIG_X86_X2APIC */
>  
> +int irq_remap_mode = -1;
> +
>  void __init enable_IR_x2apic(void)
>  {
>  	unsigned long flags;
> @@ -1915,6 +1917,8 @@ void __init enable_IR_x2apic(void)
>  
>  	if (ir_stat < 0)
>  		restore_ioapic_entries();
> +	else
> +		irq_remap_mode = ir_stat;
>  	legacy_pic->restore_mask();
>  	local_irq_restore(flags);
>  }
> diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
> index f37ad3392fec..5cc386db3557 100644
> --- a/arch/x86/kernel/apic/apic_flat_64.c
> +++ b/arch/x86/kernel/apic/apic_flat_64.c
> @@ -12,6 +12,7 @@
>  #include <linux/export.h>
>  #include <linux/acpi.h>
>  
> +#include <asm/irq_remapping.h>
>  #include <asm/jailhouse_para.h>
>  #include <asm/apic.h>
>  
> @@ -130,7 +131,8 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
>  
>  static int physflat_probe(void)
>  {
> -	return apic == &apic_physflat || num_possible_cpus() > 8 || jailhouse_paravirt();
> +	return apic == &apic_physflat || irq_remap_mode == IRQ_REMAP_XAPIC_MODE ||
> +		num_possible_cpus() > 8 || jailhouse_paravirt();
>  }
>  
>  static struct apic apic_physflat __ro_after_init = {
> diff --git a/arch/x86/kernel/apic/local.h b/arch/x86/kernel/apic/local.h
> index 842fe28496be..f633ab6dfa9f 100644
> --- a/arch/x86/kernel/apic/local.h
> +++ b/arch/x86/kernel/apic/local.h
> @@ -22,6 +22,8 @@ void x2apic_send_IPI_allbutself(int vector);
>  void x2apic_send_IPI_self(int vector);
>  extern u32 x2apic_max_apicid;
>  
> +extern int irq_remap_mode;
> +
>  /* IPI */
>  
>  DECLARE_STATIC_KEY_FALSE(apic_use_ipi_shorthand);
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

Lyude!

On Wed, May 22 2024 at 15:35, Lyude Paul wrote:

Thank for testing!

> Awesome! This patch does seem to make the system boot, thank you for
> your help

The only thing what's awesome here is that it confirms my analysis of
the underlying problem. I offered Borislav a bet on that, but he
politely declined :(

The not so awesome part is the question what to do with that insight.

The first issue is that we don't know whether that's only a problem on
your particular system or if there is an underlying systematic problem
on that particular CPU variant (model/stepping).

Unless the AMD folks can give an authoritative answer we have three options:

  1) Targeted via quirk

     As you are so far the only one complaining about this, it might be
     sufficient to enforce the physical flat mode for your particular
     machine via a DMI quirk or on the actual CPU model/stepping.

  2) Tie it to interrupt remapping

     That's the patch I provided you for testing

  3) Remove the default logical destination mode on 64bit completely

     My favourite

#1 is stupid IMO because it's likely that other systems are affected by
   this nonsense and I don't want to end up adding quirks over and over

#2 is silly because it effectively enforces physical destination mode on
   any system which has interrupt remapping available in hardware.

   That's pretty much everything halfways modern.

#3 makes a lot of sense because:

   - it reduces the amount of code

     Given the trend of the last decade this actually removes code which
     will be used less frequently as the number of logical CPUs keeps
     increasing.

   - the only benefit of logical destination mode over physical
     destination mode is the ability to send IPIs to multiple CPUs in
     one operation.

     The question is whether this still matters.

     IMO it does not matter because anything which is IPI sensitive is
     running on machines which have more than 8 CPUs today. The time
     where 8 CPU (threads) workstations and servers were state of the
     art are long gone.

   - physical destination mode is guaranteed to work because it's the
     only way to get a CPU up and running via the INIT/INIT/STARTUP
     sequence, while obvioulsy logical destination mode has its issues
     not only on the system at hand (see physflat_acpi_madt_oem_check()).

   Patch for this below.

Thanks,

        tglx
---
 arch/x86/kernel/apic/apic_flat_64.c |  116 ------------------------------------
 1 file changed, 3 insertions(+), 113 deletions(-)

--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -18,126 +18,19 @@
 #include "local.h"
 
 static struct apic apic_physflat;
-static struct apic apic_flat;
 
-struct apic *apic __ro_after_init = &apic_flat;
+struct apic *apic __ro_after_init = &apic_phys_flat;
 EXPORT_SYMBOL_GPL(apic);
 
-static int flat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
-{
-	return 1;
-}
-
-static void _flat_send_IPI_mask(unsigned long mask, int vector)
-{
-	unsigned long flags;
-
-	local_irq_save(flags);
-	__default_send_IPI_dest_field(mask, vector, APIC_DEST_LOGICAL);
-	local_irq_restore(flags);
-}
-
-static void flat_send_IPI_mask(const struct cpumask *cpumask, int vector)
-{
-	unsigned long mask = cpumask_bits(cpumask)[0];
-
-	_flat_send_IPI_mask(mask, vector);
-}
-
-static void
-flat_send_IPI_mask_allbutself(const struct cpumask *cpumask, int vector)
-{
-	unsigned long mask = cpumask_bits(cpumask)[0];
-	int cpu = smp_processor_id();
-
-	if (cpu < BITS_PER_LONG)
-		__clear_bit(cpu, &mask);
-
-	_flat_send_IPI_mask(mask, vector);
-}
-
-static u32 flat_get_apic_id(u32 x)
-{
-	return (x >> 24) & 0xFF;
-}
-
-static int flat_probe(void)
-{
-	return 1;
-}
-
-static struct apic apic_flat __ro_after_init = {
-	.name				= "flat",
-	.probe				= flat_probe,
-	.acpi_madt_oem_check		= flat_acpi_madt_oem_check,
-
-	.dest_mode_logical		= true,
-
-	.disable_esr			= 0,
-
-	.init_apic_ldr			= default_init_apic_ldr,
-	.cpu_present_to_apicid		= default_cpu_present_to_apicid,
-
-	.max_apic_id			= 0xFE,
-	.get_apic_id			= flat_get_apic_id,
-
-	.calc_dest_apicid		= apic_flat_calc_apicid,
-
-	.send_IPI			= default_send_IPI_single,
-	.send_IPI_mask			= flat_send_IPI_mask,
-	.send_IPI_mask_allbutself	= flat_send_IPI_mask_allbutself,
-	.send_IPI_allbutself		= default_send_IPI_allbutself,
-	.send_IPI_all			= default_send_IPI_all,
-	.send_IPI_self			= default_send_IPI_self,
-	.nmi_to_offline_cpu		= true,
-
-	.read				= native_apic_mem_read,
-	.write				= native_apic_mem_write,
-	.eoi				= native_apic_mem_eoi,
-	.icr_read			= native_apic_icr_read,
-	.icr_write			= native_apic_icr_write,
-	.wait_icr_idle			= apic_mem_wait_icr_idle,
-	.safe_wait_icr_idle		= apic_mem_wait_icr_idle_timeout,
-};
-
-/*
- * Physflat mode is used when there are more than 8 CPUs on a system.
- * We cannot use logical delivery in this case because the mask
- * overflows, so use physical mode.
- */
-static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
-{
-#ifdef CONFIG_ACPI
-	/*
-	 * Quirk: some x86_64 machines can only use physical APIC mode
-	 * regardless of how many processors are present (x86_64 ES7000
-	 * is an example).
-	 */
-	if (acpi_gbl_FADT.header.revision >= FADT2_REVISION_ID &&
-		(acpi_gbl_FADT.flags & ACPI_FADT_APIC_PHYSICAL)) {
-		printk(KERN_DEBUG "system APIC only can use physical flat");
-		return 1;
-	}
-
-	if (!strncmp(oem_id, "IBM", 3) && !strncmp(oem_table_id, "EXA", 3)) {
-		printk(KERN_DEBUG "IBM Summit detected, will use apic physical");
-		return 1;
-	}
-#endif
-
-	return 0;
-}
-
 static int physflat_probe(void)
 {
-	return apic == &apic_physflat || num_possible_cpus() > 8 || jailhouse_paravirt();
+	return 1;
 }
 
 static struct apic apic_physflat __ro_after_init = {
 
 	.name				= "physical flat",
 	.probe				= physflat_probe,
-	.acpi_madt_oem_check		= physflat_acpi_madt_oem_check,
 
 	.dest_mode_logical		= false,
 
@@ -167,7 +60,4 @@ static struct apic apic_physflat __ro_af
 	.safe_wait_icr_idle		= apic_mem_wait_icr_idle_timeout,
 };
 
-/*
- * We need to check for physflat first, so this order is important.
- */
-apic_drivers(apic_physflat, apic_flat);
+apic_drivers(apic_physflat);

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Linux regression tracking (Thorsten Leemhuis)]

On 23.05.24 00:12, Thomas Gleixner wrote:
> On Wed, May 22 2024 at 15:35, Lyude Paul wrote:
>
>> Awesome! This patch does seem to make the system boot, thank you for
>> your help

+1

> The only thing what's awesome here is that it confirms my analysis of
> the underlying problem. I offered Borislav a bet on that, but he
> politely declined :(
> 
> The not so awesome part is the question what to do with that insight.
> 
> The first issue is that we don't know whether that's only a problem on
> your particular system or if there is an underlying systematic problem
> on that particular CPU variant (model/stepping).

FWIW, Arch Linux and openSUSE Tumbleweed switched to 6.9.1 in the past
few days (Fedora will likely follow in ~two weeks) and I have not yet
seen any reports about problems like this, which is a first small
indicator that this might be a system specific problem; if any such
reports come up I'll mention it here.

Ciao, Thorsten

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

Lyude!

On Thu, May 23 2024 at 00:12, Thomas Gleixner wrote:
> The first issue is that we don't know whether that's only a problem on
> your particular system or if there is an underlying systematic problem
> on that particular CPU variant (model/stepping).

Something I wanted to ask before. Is the BIOS of that machine up to
date?

Thanks,

        tglx

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

Lyude!

On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
> Something I wanted to ask before. Is the BIOS of that machine up to
> date?

Any update on this? I really don't want to add magic workarounds if
there is a BIOS update which fixes it.

Thanks,

        tglx

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

Ah whoops! Sorry I totally missed this question, thank you for poking
again
I'm not sure it is but I will double check tomorrow and let you know.

On Wed, 2024-05-29 at 00:43 +0200, Thomas Gleixner wrote:
> Lyude!
> 
> On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
> > Something I wanted to ask before. Is the BIOS of that machine up to
> > date?
> 
> Any update on this? I really don't want to add magic workarounds if
> there is a BIOS update which fixes it.
> 
> Thanks,
> 
>         tglx
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

On Wed, 2024-05-29 at 00:43 +0200, Thomas Gleixner wrote:
> Lyude!
> 
> On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
> > Something I wanted to ask before. Is the BIOS of that machine up to
> > date?
> 
> Any update on this? I really don't want to add magic workarounds if
> there is a BIOS update which fixes it.

OK - just booted a normal kernel up after updating the BIOS and you're
right - that does appear to solve the problem :). Thank you!

> 
> Thanks,
> 
>         tglx
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Thomas Gleixner]

On Wed, Jun 05 2024 at 19:15, Lyude Paul wrote:
> On Wed, 2024-05-29 at 00:43 +0200, Thomas Gleixner wrote:
>> Lyude!
>> 
>> On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
>> > Something I wanted to ask before. Is the BIOS of that machine up to
>> > date?
>> 
>> Any update on this? I really don't want to add magic workarounds if
>> there is a BIOS update which fixes it.
>
> OK - just booted a normal kernel up after updating the BIOS and you're
> right - that does appear to solve the problem :). Thank you!

Perfect!

Thanks

        Thomas

Re: Early boot regression from f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")

[Lyude Paul]

So uh, apparently amazon decided that instead of using the shipping
address I have in their address book that is marked as "Default
Shipping Address" that it makes much more sense to sort all addresses
in the address book alphabetically, and then surprise me by sending it
to the first address in that list. which means the adapter got shipped
half way across the country and it's probably going to take me until at
least tomorrow to get a different one here, sorry :(.

Will get back to you with this info as soon as I can

On Wed, 2024-05-08 at 12:30 +0200, Thomas Gleixner wrote:
> On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten
> Leemhuis) wrote:
> > H! Lyude, Thomas, what's the status here? From here it looks like
> > we
> > were close to a fix, but then it turned out to be a bad fix -- and
> > afterwards nothing much seems to have happened. Did it fall through
> > the
> > cracks, or was this already fixed and I just missed that?
> 
> I'm waiting for more data still.
> 
> Thanks,
> 
>         tglx
> 

-- 
Cheers,
 Lyude Paul (she/her)
 Software Engineer at Red Hat