Two questions about the MADT in ACPI table

Two questions about the MADT in ACPI table

[Dou Liyang]

Hi, Rafael, Lv

May I ask for your help.

Recently, I am very confused about the generation of the MADT
in ACPI table.

1: Why we put all the Apic Ids which flag==1 in front of the
one which flag==0 in MADT?

Just like that:

-----orginal-----------------MADT-------
Apic Id	Enable Flag	Apic Id	Enable Flag
0x00	1		0x00	1
0x02	1		0x02	1
0x04	0		0x10	1
0x06	0		0x12	1
0x10	1		0x04	0
0x12	1		0x06	0
0x14	0		0x14	0
0x16	0		0x16	0

If the flag is the same, It will not change the order.

-----orginal-----------------MADT---------
Apic Id	Enable Flag	Apic Id	Enable Flag
0x00	0		0x00	0
0x02	0		0x02	0
0x04	0		0x04	0
0x06	0		0x06	0
0x10	0		0x10	0
0x12	0		0x12	0
0x14	0		0x14	0
0x16	0		0x16	0


2: If the order rule can be changed ?
If it can be done, What should I do?

Thanks,
Liyang.

Re: Two questions about the MADT in ACPI table

[Rafael J. Wysocki]

On Mon, Feb 13, 2017 at 2:31 PM, Dou Liyang <douly.fnst@cn.fujitsu.com> wrote:
> Hi, Rafael, Lv
>
> May I ask for your help.
>
> Recently, I am very confused about the generation of the MADT
> in ACPI table.
>
> 1: Why we put all the Apic Ids which flag==1 in front of the
> one which flag==0 in MADT?
>
> Just like that:
>
> -----orginal-----------------MADT-------
> Apic Id Enable Flag     Apic Id Enable Flag
> 0x00    1               0x00    1
> 0x02    1               0x02    1
> 0x04    0               0x10    1
> 0x06    0               0x12    1
> 0x10    1               0x04    0
> 0x12    1               0x06    0
> 0x14    0               0x14    0
> 0x16    0               0x16    0
>
> If the flag is the same, It will not change the order.
>
> -----orginal-----------------MADT---------
> Apic Id Enable Flag     Apic Id Enable Flag
> 0x00    0               0x00    0
> 0x02    0               0x02    0
> 0x04    0               0x04    0
> 0x06    0               0x06    0
> 0x10    0               0x10    0
> 0x12    0               0x12    0
> 0x14    0               0x14    0
> 0x16    0               0x16    0
>
>
> 2: If the order rule can be changed ?
> If it can be done, What should I do?

Are you asking about Local APIC IDs?

Thanks,
Rafael

Re: Two questions about the MADT in ACPI table

[Hanjun Guo]

On 2017/2/13 21:31, Dou Liyang wrote:
> Hi, Rafael, Lv
>
> May I ask for your help.
>
> Recently, I am very confused about the generation of the MADT
> in ACPI table.
>
> 1: Why we put all the Apic Ids which flag==1 in front of the
> one which flag==0 in MADT?
>
> Just like that:
>
> -----orginal-----------------MADT-------
> Apic Id    Enable Flag    Apic Id    Enable Flag
> 0x00    1        0x00    1
> 0x02    1        0x02    1
> 0x04    0        0x10    1
> 0x06    0        0x12    1
> 0x10    1        0x04    0
> 0x12    1        0x06    0
> 0x14    0        0x14    0
> 0x16    0        0x16    0
>
> If the flag is the same, It will not change the order.
>
> -----orginal-----------------MADT---------
> Apic Id    Enable Flag    Apic Id    Enable Flag
> 0x00    0        0x00    0
> 0x02    0        0x02    0
> 0x04    0        0x04    0
> 0x06    0        0x06    0
> 0x10    0        0x10    0
> 0x12    0        0x12    0
> 0x14    0        0x14    0
> 0x16    0        0x16    0

Sorry, a little bit confused, local APIC entries are listed in
MADT by firmware, and MADT is a static table which means the OS
will not update the orders.

>
>
> 2: If the order rule can be changed ?
> If it can be done, What should I do?

What's the issue you are facing? any problem here?

Thanks
Hanjun

Re: Two questions about the MADT in ACPI table

[Dou Liyang]

Dear, Rafael

At 02/14/2017 06:18 AM, Rafael J. Wysocki wrote:
> On Mon, Feb 13, 2017 at 2:31 PM, Dou Liyang <douly.fnst@cn.fujitsu.com> wrote:
>> Hi, Rafael, Lv
>>
>> May I ask for your help.
>>
>> Recently, I am very confused about the generation of the MADT
>> in ACPI table.
>>
>> 1: Why we put all the Apic Ids which flag==1 in front of the
>> one which flag==0 in MADT?
>>
>> Just like that:
>>
>> -----orginal----------|-------MADT-------
>> Apic Id | Enable Flag |   Apic Id | Enable Flag
>> 0x00    	1               0x00    1
>> 0x02    	1               0x02    1
>> 0x04    	0               0x10    1
>> 0x06    	0               0x12    1
>> 0x10    	1               0x04    0
>> 0x12    	1               0x06    0
>> 0x14   	0               0x14    0
>> 0x16    	0               0x16    0
>>
>> If the flag is the same, It will not change the order.
>>
>> -----orginal----------|-------MADT---------
>> Apic Id | Enable Flag |  Apic Id | Enable Flag
>> 0x00    	0               0x00    0
>> 0x02    	0               0x02    0
>> 0x04    	0               0x04    0
>> 0x06    	0               0x06    0
>> 0x10    	0               0x10    0
>> 0x12    	0               0x12    0
>> 0x14    	0               0x14    0
>> 0x16    	0               0x16    0
>>
>>
>> 2: If the order rule can be changed ?
>> If it can be done, What should I do?
>
> Are you asking about Local APIC IDs?
>

Yes, it is. Something about *Processor Local APIC Structure*.

In MADT, Their arrangement may be different between the usable
processors and unusable processors(Processor Enabled is 1 or 0
in MADT).

The rule of the arrangement seems like that it makes the usable
processors in front of the unusable processors, when we generate
the MADT at the booting time. And it seems do not care about the
arrangement of the unusable processors.

Unfortunately, the unusable processors will be used in CPU hotplug.
And we also make the cpu_id <-> apic_id fixed according to the MADT
after linux-4.9. So the arrangement may be influence on the value of
CPUIds.

I am investigating this. So I hope to know:

1. How the MADT is generated, Where can I find the Method/Function of
the generation?

2. Is there a rule used for the order of the Local APIC/x2APIC IDs?

Thanks,
  Liyang

Re: Two questions about the MADT in ACPI table

[Dou Liyang]

ccing Hanjun

At 02/15/2017 10:53 AM, Dou Liyang wrote:
> Dear, Rafael
>
> At 02/14/2017 06:18 AM, Rafael J. Wysocki wrote:
>> On Mon, Feb 13, 2017 at 2:31 PM, Dou Liyang
>> <douly.fnst@cn.fujitsu.com> wrote:
>>> Hi, Rafael, Lv
>>>
>>> May I ask for your help.
>>>
>>> Recently, I am very confused about the generation of the MADT
>>> in ACPI table.
>>>
>>> 1: Why we put all the Apic Ids which flag==1 in front of the
>>> one which flag==0 in MADT?
>>>
>>> Just like that:
>>>
>>> -----orginal----------|-------MADT-------
>>> Apic Id | Enable Flag |   Apic Id | Enable Flag
>>> 0x00        1               0x00    1
>>> 0x02        1               0x02    1
>>> 0x04        0               0x10    1
>>> 0x06        0               0x12    1
>>> 0x10        1               0x04    0
>>> 0x12        1               0x06    0
>>> 0x14       0               0x14    0
>>> 0x16        0               0x16    0
>>>
>>> If the flag is the same, It will not change the order.
>>>
>>> -----orginal----------|-------MADT---------
>>> Apic Id | Enable Flag |  Apic Id | Enable Flag
>>> 0x00        0               0x00    0
>>> 0x02        0               0x02    0
>>> 0x04        0               0x04    0
>>> 0x06        0               0x06    0
>>> 0x10        0               0x10    0
>>> 0x12        0               0x12    0
>>> 0x14        0               0x14    0
>>> 0x16        0               0x16    0
>>>
>>>
>>> 2: If the order rule can be changed ?
>>> If it can be done, What should I do?
>>
>> Are you asking about Local APIC IDs?
>>
>
> Yes, it is. Something about *Processor Local APIC Structure*.
>
> In MADT, Their arrangement may be different between the usable
> processors and unusable processors(Processor Enabled is 1 or 0
> in MADT).
>
> The rule of the arrangement seems like that it makes the usable
> processors in front of the unusable processors, when we generate
> the MADT at the booting time. And it seems do not care about the
> arrangement of the unusable processors.
>
> Unfortunately, the unusable processors will be used in CPU hotplug.
> And we also make the cpu_id <-> apic_id fixed according to the MADT
> after linux-4.9. So the arrangement may be influence on the value of
> CPUIds.
>
> I am investigating this. So I hope to know:
>
> 1. How the MADT is generated, Where can I find the Method/Function of
> the generation?
>
> 2. Is there a rule used for the order of the Local APIC/x2APIC IDs?
>
> Thanks,
>  Liyang

Re: Two questions about the MADT in ACPI table

[Dou Liyang]

Hi, Hanjun

Thank you for your help!

[...]
>
> Sorry, a little bit confused, local APIC entries are listed in
> MADT by firmware, and MADT is a static table which means the OS
> will not update the orders.
>

Yes, MADT will never be updated after booting time.

My concern is how do the firmware arrange the order of the local
APIC entries in MADT.

I found the local APIC entries which its processor enable flag was
1 would be sorted close to the front of MADT.

I startup the computer twice,

1. I don't use the local APIC IDs with 0x10 and 0x12.
There are just 2 online CPUs in the computer in the booting time.
The MADT likes:

-------MADT---------
Apic Id | Enable Flag
0x00        1
0x02        1
0x04        0
0x06        0
0x10        0
0x12        0
0x14        0
0x16        0
-------------------------------------

2. I use the local APIC IDs with 0x10 and 0x12. Then, startup the
computer. There are 4 online CPUs.
The MADT likes:
------MADT-------
Apic Id | Enable Flag
0x00        1
0x02        1
0x10        1
0x12        1
0x04        0
0x16        0
0x14        0
0x16        0

The position of the 0x10 and 0x12 is changed.

>>
>>
>> 2: If the order rule can be changed ?
>> If it can be done, What should I do?
>
> What's the issue you are facing? any problem here?

Sorry, I do not describe it clearly.
please see the email, which i have cc-ed to you.

Thanks,
Liyang.

>
> Thanks
> Hanjun
>
>
>

Re: Two questions about the MADT in ACPI table

[Dou Liyang]

Adding a simple example:

I startup a pc twice.

*1.* There are just 2 online CPUs in the pc at booting time.
The MADT likes:

-------MADT---------
Apic Id | Enable Flag
0x00        1
0x02        1
0x04        0
0x06        0
*0x10*        0
*0x12*        0
0x14        0
0x16        0
-------------------------------------

*2.*  There are 4 online CPUs in the pc. Then, startup it.
The new CPUs use the local APIC IDs with 0x10 and 0x12.
The MADT likes:
------MADT-------
Apic Id | Enable Flag
0x00        1
0x02        1
*0x10*       1
*0x12*       1
0x04        0
0x16        0
0x14        0
0x16        0

The position of the 0x10 and 0x12 is changed.

[...]

>> Are you asking about Local APIC IDs?
>>
>
> Yes, it is. Something about *Processor Local APIC Structure*.
>
> In MADT, Their arrangement may be different between the usable
> processors and unusable processors(Processor Enabled is 1 or 0
> in MADT).
>
> The rule of the arrangement seems like that it makes the usable
> processors in front of the unusable processors, when we generate
> the MADT at the booting time. And it seems do not care about the
> arrangement of the unusable processors.
>
> Unfortunately, the unusable processors will be used in CPU hotplug.
> And we also make the cpu_id <-> apic_id fixed according to the MADT
> after linux-4.9. So the arrangement may be influence on the value of
> CPUIds.
>
> I am investigating this. So I hope to know:
>
> 1. How the MADT is generated, Where can I find the Method/Function of
> the generation?
>
> 2. Is there a rule used for the order of the Local APIC/x2APIC IDs?
>
> Thanks,
>  Liyang

Re: Two questions about the MADT in ACPI table

[Hanjun Guo]

On 2017/2/15 11:18, Dou Liyang wrote:
> ccing Hanjun

Thank you for adding me in the loop.

>
> At 02/15/2017 10:53 AM, Dou Liyang wrote:
>> Dear, Rafael
>>
>> At 02/14/2017 06:18 AM, Rafael J. Wysocki wrote:
>>> On Mon, Feb 13, 2017 at 2:31 PM, Dou Liyang
>>> <douly.fnst@cn.fujitsu.com> wrote:
>>>> Hi, Rafael, Lv
>>>>
>>>> May I ask for your help.
>>>>
>>>> Recently, I am very confused about the generation of the MADT
>>>> in ACPI table.
>>>>
>>>> 1: Why we put all the Apic Ids which flag==1 in front of the
>>>> one which flag==0 in MADT?
>>>>
>>>> Just like that:
>>>>
>>>> -----orginal----------|-------MADT-------
>>>> Apic Id | Enable Flag |   Apic Id | Enable Flag
>>>> 0x00        1               0x00    1
>>>> 0x02        1               0x02    1
>>>> 0x04        0               0x10    1
>>>> 0x06        0               0x12    1
>>>> 0x10        1               0x04    0
>>>> 0x12        1               0x06    0
>>>> 0x14       0               0x14    0
>>>> 0x16        0               0x16    0
>>>>
>>>> If the flag is the same, It will not change the order.
>>>>
>>>> -----orginal----------|-------MADT---------
>>>> Apic Id | Enable Flag |  Apic Id | Enable Flag
>>>> 0x00        0               0x00    0
>>>> 0x02        0               0x02    0
>>>> 0x04        0               0x04    0
>>>> 0x06        0               0x06    0
>>>> 0x10        0               0x10    0
>>>> 0x12        0               0x12    0
>>>> 0x14        0               0x14    0
>>>> 0x16        0               0x16    0
>>>>
>>>>
>>>> 2: If the order rule can be changed ?
>>>> If it can be done, What should I do?
>>>
>>> Are you asking about Local APIC IDs?
>>>
>>
>> Yes, it is. Something about *Processor Local APIC Structure*.
>>
>> In MADT, Their arrangement may be different between the usable
>> processors and unusable processors(Processor Enabled is 1 or 0
>> in MADT).
>>
>> The rule of the arrangement seems like that it makes the usable
>> processors in front of the unusable processors, when we generate
>> the MADT at the booting time. And it seems do not care about the
>> arrangement of the unusable processors.
>>
>> Unfortunately, the unusable processors will be used in CPU hotplug.
>> And we also make the cpu_id <-> apic_id fixed according to the MADT
>> after linux-4.9. So the arrangement may be influence on the value of
>> CPUIds.

Can you specify the issues? maybe the logical CPU IDs are not
continuous in a numa node even two CPUs are hyper-threadings
in one core?

>>
>> I am investigating this. So I hope to know:
>>
>> 1. How the MADT is generated, Where can I find the Method/Function of
>> the generation?

It depends on the firmware I think and lots of them are not open source,
you may try the UEFI tianocore project (www.tianocore.org).

>>
>> 2. Is there a rule used for the order of the Local APIC/x2APIC IDs?

I think not, there is no rules in the ACPI spec for the ordering
of processor entries (local apic/x2apic/gicc).

Thanks
Hanjun

Re: Two questions about the MADT in ACPI table

[Hanjun Guo]

On 2017/2/15 11:46, Dou Liyang wrote:
> Hi, Hanjun
>
> Thank you for your help!
>
> [...]
>>
>> Sorry, a little bit confused, local APIC entries are listed in
>> MADT by firmware, and MADT is a static table which means the OS
>> will not update the orders.
>>
>
> Yes, MADT will never be updated after booting time.
>
> My concern is how do the firmware arrange the order of the local
> APIC entries in MADT.
>
> I found the local APIC entries which its processor enable flag was
> 1 would be sorted close to the front of MADT.
>
> I startup the computer twice,
>
> 1. I don't use the local APIC IDs with 0x10 and 0x12.
> There are just 2 online CPUs in the computer in the booting time.
> The MADT likes:
>
> -------MADT---------
> Apic Id | Enable Flag
> 0x00        1
> 0x02        1

Digital order.

> 0x04        0
> 0x06        0
> 0x10        0
> 0x12        0
> 0x14        0
> 0x16        0

Same here...

> -------------------------------------
>
> 2. I use the local APIC IDs with 0x10 and 0x12. Then, startup the
> computer. There are 4 online CPUs.
> The MADT likes:
> ------MADT-------
> Apic Id | Enable Flag
> 0x00        1
> 0x02        1
> 0x10        1
> 0x12        1

And digital order...

> 0x04        0
> 0x16        0

0x06?

> 0x14        0
> 0x16        0
>
> The position of the 0x10 and 0x12 is changed.

So your firmware will list enabled CPUs first and
them disabled one, both in digital order, and lot of
firmware (but not all of them) doing the same thing.

Thanks
Hanjun

Re: Two questions about the MADT in ACPI table

[Dou Liyang]

Hi, Hanjun

[...]
>> -------MADT---------
>> Apic Id | Enable Flag
>> 0x00        1
>> 0x02        1
>
> Digital order.
>
>> 0x04        0
>> 0x06        0
>> 0x10        0
>> 0x12        0
>> 0x14        0
>> 0x16        0
>
> Same here...
>
>> -------------------------------------
>>
>> 2. I use the local APIC IDs with 0x10 and 0x12. Then, startup the
>> computer. There are 4 online CPUs.
>> The MADT likes:
>> ------MADT-------
>> Apic Id | Enable Flag
>> 0x00        1
>> 0x02        1
>> 0x10        1
>> 0x12        1
>
> And digital order...
>
>> 0x04        0
>> 0x16        0
>
> 0x06?

oops! yes.

>
>> 0x14        0
>> 0x16        0
>>
>> The position of the 0x10 and 0x12 is changed.
>
> So your firmware will list enabled CPUs first and
> them disabled one, both in digital order, and lot of
> firmware (but not all of them) doing the same thing.

It seems reasonable, I understood! :)

Thanks,
Liyang

>
> Thanks
> Hanjun
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

Re: Two questions about the MADT in ACPI table

[Dou Liyang]

Hi,

>>>>
>>>> Are you asking about Local APIC IDs?
>>>>
>>>
>>> Yes, it is. Something about *Processor Local APIC Structure*.
>>>
>>> In MADT, Their arrangement may be different between the usable
>>> processors and unusable processors(Processor Enabled is 1 or 0
>>> in MADT).
>>>
>>> The rule of the arrangement seems like that it makes the usable
>>> processors in front of the unusable processors, when we generate
>>> the MADT at the booting time. And it seems do not care about the
>>> arrangement of the unusable processors.
>>>
>>> Unfortunately, the unusable processors will be used in CPU hotplug.
>>> And we also make the cpu_id <-> apic_id fixed according to the MADT
>>> after linux-4.9. So the arrangement may be influence on the value of
>>> CPUIds.
>
> Can you specify the issues?

OK,

maybe the logical CPU IDs are not
> continuous in a numa node even two CPUs are hyper-threadings
> in one core?
>

Yes, it is. The logical CPU IDs may be not serial, when we
use "lscpu" in the OS. it may like that:

NUMA node0 CPU：    0-23,256-279
NUMA node1 CPU：    24-47,280-303

Or even like

Node2: 64-69, 72-77, 80-85, 88-93, 320-325, 328-333, 336-341 344-349
Node3: 96-101, 104-109, 112-117, 120-125, 352-357, 360-365, 368-373, 376-381

The reason :

1. We make the cpu_id <-> apic_id fixed according to the MADT. details
see in commit f7c28833c252031bc in kernel upstream tree.
2. We may have a Local APIC entry in MADT, but it has not Corresponding
processor device in DSDT. and the Local APIC entry also occupy an
logical CPU ID.

So, the seqence of logical CPU IDs will skip it. and will not be serial.

As the ACPI table is not reliable, if we fix it, we may need do in
software level.

And now, I don't have a good way to fix it.
So, I guess I need more knowledge about the ACPI table.

>>>
>>> I am investigating this. So I hope to know:
>>>
>>> 1. How the MADT is generated, Where can I find the Method/Function of
>>> the generation?
>
> It depends on the firmware I think and lots of them are not open source,
> you may try the UEFI tianocore project (www.tianocore.org).
>

I will have a look.

>>>
>>> 2. Is there a rule used for the order of the Local APIC/x2APIC IDs?
>
> I think not, there is no rules in the ACPI spec for the ordering
> of processor entries (local apic/x2apic/gicc).
>

As you say, normally, by digital order default.

Thanks
Liyang.