Re: [PATCH v6] i386/cpu: fixup number of addressable IDs for logical processors in the physical package

[Xiaoyao Li <xiaoyao.li@intel.com>]

On 10/14/2024 11:36 AM, Zhao Liu wrote:
>>>> On 10/9/2024 11:56 AM, Chuang Xu wrote:
>>>>> When QEMU is started with:
>>>>> -cpu host,migratable=on,host-cache-info=on,l3-cache=off
>>>>> -smp 180,sockets=2,dies=1,cores=45,threads=2
>>>>>
>>>>> On Intel platform:
>>>>> CPUID.01H.EBX[23:16] is defined as "max number of addressable IDs for
>>>>> logical processors in the physical package".
>>>>>
>>>>> When executing "cpuid -1 -l 1 -r" in the guest, we obtain a
>>>>> value of 90 for
>>>>> CPUID.01H.EBX[23:16], whereas the expected value is 128. Additionally,
>>>>> executing "cpuid -1 -l 4 -r" in the guest yields a value of 63 for
>>>>> CPUID.04H.EAX[31:26], which matches the expected result.
>>>>>
>>>>> As (1+CPUID.04H.EAX[31:26]) rounds up to the nearest power-of-2 integer,
>>>>> we'd beter round up CPUID.01H.EBX[23:16] to the nearest power-of-2
>>>>> integer too. Otherwise we may encounter unexpected results in guest.
>>>>>
>>>>> For example, when QEMU is started with CLI above and xtopology
>>>>> is disabled,
>>>>> guest kernel 5.15.120 uses CPUID.01H.EBX[23:16]/
>>>>> (1+CPUID.04H.EAX[31:26]) to
>>>>> calculate threads-per-core in detect_ht(). Then guest will get
>>>>> "90/ (1+63)=1"
>>>>> as the result, even though threads-per-core should actually be 2.
>>>>
>>>> It's kernel's bug instead.
>>>>
>>>> In 1.5.3 "Sub ID Extraction Parameters for initial APIC ID" of
>>>> "Intel 64 Architecture Processor Topology Enumeration" [1], it is
>>>>
>>>>    - SMT_Mask_Width = Log2(RoundToNearestPof2(CPUID.1:EBX[23:16])/
>>>> (CPUID.(EAX=4,ECX=0):EAX[31:26]) + 1))
>>>>
>>>> The value of CPUID.1:EBX[23:16] needs to be *rounded* to the
>>>> neartest power-of-two integer instead of itself being the
>>>> power-of-two.
>>>>
>>>> This also is consistency with the SDM, where the comment for bit
>>>> 23-16 of CPUID.1:EBX is:
>>>>
>>>>    The nearest power-of-2 integer that is not smaller than EBX[23:16] is
>>>>    the number of unique initial APIC IDs reserved for addressing
>>>>    different logical processors in a physical package.
>>>>
>>>> What I read from this is, the nearest power-of-2 integer that is not
>>>> smaller than EBX[23:16] is a different thing than EBX[23:16]. i.e.,
>>>
>>> Yes, when I read sdm, I also thought it was a kernel bug. But on my
>>> 192ht spr host, the value of CPUID.1:EBX[23:16] was indeed rounded up
>>>
>>> to the nearest power of 2 by the hardware. After communicating with
>>> Intel technical support staff, we thought that perhaps the description
>>> in sdm
>>>
>>> is not so accurate, and rounding up CPUID.1:EBX[23:16] to the power of 2
>>> in qemu may be more in line with the hardware behavior.
>>
>> I think above justification is important. We need to justify our changes
>> with the fact and correct reason.
>>
>> I somehow agree to set EBX[23:16] to a value of power-of-2, because the
>> 1.5.3 "Sub ID Extraction Parameters for initial APIC ID" of "Intel 64
>> Architecture Processor Topology Enumeration" spec says
>>
>>      CPUID.1:EBX[23:16] represents the maximum number of addressable IDs
>>      (initial APIC ID) that can be assigned to logical processors in a
>>      physical package. The value may not be the same as the number of
>>      logical processors that are present in the hardware of a physical
>>      package.
>>
>> It uses the word "may not".
> 
> IMO, I don't quite understand your confusion regarding this. I've already
> explained the meaning of addressable ID, and the spec you referenced also
> clarifies its significance. The reason for this modification is not
> because of the two words "may not".
> 
> Whether it is "be" or "not be" the same as the number of logical
> processors, the essence is that due to topology, the actual number of
> initial IDs that can be accommodated in the APIC ID may exceed the number
> of logical processors.

I have the confusion because no matter from SDM:

   Bit 23-16: Maximum number of addressable IDs for logical processors in
              this physical package*

   * The nearest power-of-2 integer that is not smaller than EBX[23:16]
     is the number of unique initial APIC IDs reserved for addressing
     different logical processors in a physical package.

or from "Intel 64 Architecture Processor Topology Enumeration" spec，(Jan 
2018, revision 1.1), 1.5.3 "sub ID Extraction Parameters for Inital APIC ID"

   RoundToNearestPof2(CPUID.1:EBX[23:16])

or from "Intel 64 Architecture Processor Topology Enumeration" 
spec，(April 2023, revision 2.0), 1.6.1 Legacy Extraction Algorithm

https://cdrdv2-public.intel.com/775917/intel-64-architecture-processor-topology-enumeration.pdf

   "MaximumLogicalProcessorIDsPerPackage" is calculated by rounding
    CPUID.01H.EBX[23:16] to nearest power of 2.

what I read from them is that EBX[23:16] is a different thing than 
pow2ceil(EBX[23:16]) and EBX[23:16] doesn't need to be power-of-2, but 
the patch are trying to make it power-of-2.

Then I consult it with Intel internal architect. I was told that 
EBX[23:16] used to be that software was to round to the next power of 2. 
However, software had issues a long time ago because applications could 
then compute the wrong power of 2 based on APIC ID holes or some 
applications would use it directly (without round it up to power-of-2).
So intel became to report exact power-of-2 and this behavior is not 
documented.