nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[zhiyi]

Hi,
I have just found the value of nr_cpu_ids is 32, instead of 16 on my  
new Dell PowerEdge R905 which has 16 cpus (4 quad-core Opteron 8380).  
However, /proc/cpuinfo displays the right number (16).
I have searched the archive but found no related subject.  Is there  
any patch already available to fix the problem?
I am using 2.6.31-4 and verified this incorrect value (32) using a  
module.
Please CC responses to my personal email.
Cheers,
Zhiyi

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[Borislav Petkov]

On Thu, Jan 28, 2010 at 12:16:44PM +1300, zhiyi wrote:
> Hi,
> I have just found the value of nr_cpu_ids is 32, instead of 16 on my
> new Dell PowerEdge R905 which has 16 cpus (4 quad-core Opteron
> 8380). However, /proc/cpuinfo displays the right number (16).
> I have searched the archive but found no related subject.  Is there
> any patch already available to fix the problem?
> I am using 2.6.31-4 and verified this incorrect value (32) using a
> module.

nr_cpu_ids is set to CONFIG_NR_CPUS on SMP kernels and denotes the max
CPUs supported by your kernel. This is not a bug but a configurable
option in the kernel for saving memory.

You can still set CONFIG_NR_CPUS in "-> Processor type and features" to
the max number of cores N you have on your machine if you want to save
approx 8*N KB. But guessing from your quadsocket configuration, memory
shouldn't be that scarce on that machine to go the trouble :).

-- 
Regards/Gruss,
    Boris.

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[zhiyi]

Hi Boris,
Thanks for your answer.
However, your answer doesn't fully resolve my puzzle.
My module worked for a kernel 2.6.29 with CONFIG_NR_CPUS=64 on my dual  
quad-core AMD Opteron. The value of nr_cpu_ids I got from the same  
module was correct (8). It seems nr_cpu_ids has nothing to do with  
CONFIG_NR_CPUS.
Cheers,
Zhiyi

On 31/01/2010, at 6:18 AM, Borislav Petkov wrote:

> On Thu, Jan 28, 2010 at 12:16:44PM +1300, zhiyi wrote:
>> Hi,
>> I have just found the value of nr_cpu_ids is 32, instead of 16 on my
>> new Dell PowerEdge R905 which has 16 cpus (4 quad-core Opteron
>> 8380). However, /proc/cpuinfo displays the right number (16).
>> I have searched the archive but found no related subject.  Is there
>> any patch already available to fix the problem?
>> I am using 2.6.31-4 and verified this incorrect value (32) using a
>> module.
>
> nr_cpu_ids is set to CONFIG_NR_CPUS on SMP kernels and denotes the max
> CPUs supported by your kernel. This is not a bug but a configurable
> option in the kernel for saving memory.
>
> You can still set CONFIG_NR_CPUS in "-> Processor type and features"  
> to
> the max number of cores N you have on your machine if you want to save
> approx 8*N KB. But guessing from your quadsocket configuration, memory
> shouldn't be that scarce on that machine to go the trouble :).
>
> -- 
> Regards/Gruss,
>    Boris.
>

=======================
Zhiyi Huang
Dept of Computer Science
University of Otago
Email: hzy@cs.otago.ac.nz
Phone: 0064-3-4795680
Fax: 0064-3-4798529

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[Borislav Petkov]

On Sun, Jan 31, 2010 at 11:59 PM, zhiyi <zhuang@cs.otago.ac.nz> wrote:
> My module worked for a kernel 2.6.29 with CONFIG_NR_CPUS=64 on my dual
> quad-core AMD Opteron. The value of nr_cpu_ids I got from the same module
> was correct (8).

This could mean that your include/generated/autoconf.h, which is used by
external modules, is not updated and contains CONFIG_NR_CPUS=8. Do

make mrproper

in your kernel directory by moving your .config out of the way first and
then rebuild your kernel and external module(s). nr_cpu_ids should pick
up the updated CONFIG_NR_CPUS value from your .config.

-- 
Regards/Gruss,
Boris

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[Borislav Petkov]

On Mon, Feb 1, 2010 at 2:57 PM, Borislav Petkov <petkovbb@googlemail.com> wrote:
> On Sun, Jan 31, 2010 at 11:59 PM, zhiyi <zhuang@cs.otago.ac.nz> wrote:
>> My module worked for a kernel 2.6.29 with CONFIG_NR_CPUS=64 on my dual
>> quad-core AMD Opteron. The value of nr_cpu_ids I got from the same module
>> was correct (8).
>
> This could mean that your include/generated/autoconf.h, which is used by
> external modules, is not updated and contains CONFIG_NR_CPUS=8. Do
>
> make mrproper
>
> in your kernel directory by moving your .config out of the way first and
> then rebuild your kernel and external module(s). nr_cpu_ids should pick
> up the updated CONFIG_NR_CPUS value from your .config.

That's actually not necessary - the nr_cpu_ids thing is set at compile
time to CONFIG_NR_CPUS but then capped to a possibly lower value upon
boot depending on the info in the ACPI mptables. Can you please send the
.config and the full dmesg of the quadsocket machine?

Thanks.

-- 
Regards/Gruss,
Boris

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[zhiyi]

On 2/02/2010, at 4:04 AM, Borislav Petkov wrote:

> On Mon, Feb 1, 2010 at 2:57 PM, Borislav Petkov <petkovbb@googlemail.com 
> > wrote:
>> On Sun, Jan 31, 2010 at 11:59 PM, zhiyi <zhuang@cs.otago.ac.nz>  
>> wrote:
>>> My module worked for a kernel 2.6.29 with CONFIG_NR_CPUS=64 on my  
>>> dual
>>> quad-core AMD Opteron. The value of nr_cpu_ids I got from the same  
>>> module
>>> was correct (8).
>>
>> This could mean that your include/generated/autoconf.h, which is  
>> used by
>> external modules, is not updated and contains CONFIG_NR_CPUS=8. Do
>>
>> make mrproper
>>
>> in your kernel directory by moving your .config out of the way  
>> first and
>> then rebuild your kernel and external module(s). nr_cpu_ids should  
>> pick
>> up the updated CONFIG_NR_CPUS value from your .config.
>
> That's actually not necessary - the nr_cpu_ids thing is set at compile
> time to CONFIG_NR_CPUS but then capped to a possibly lower value upon
> boot depending on the info in the ACPI mptables. Can you please send  
> the
> .config and the full dmesg of the quadsocket machine?

The dmesg relevant to nr_cpu_ids of the quad socket is:

[    0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:32 nr_node_ids:4

For comparison purposes, the dmesg for the dual socket is:

[    0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:8 nr_node_ids:1

It seems nr_node_ids:1 for the dual socket is not quite right though.

Cheers,
Zhiyi




>
>
> Thanks.
>
> -- 
> Regards/Gruss,
> Boris
>

=======================
Zhiyi Huang
Dept of Computer Science
University of Otago
Email: hzy@cs.otago.ac.nz
Phone: 0064-3-4795680
Fax: 0064-3-4798529

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[Borislav Petkov]

On Tue, Feb 02, 2010 at 01:49:06PM +1300, zhiyi wrote:
> The dmesg relevant to nr_cpu_ids of the quad socket is:
> 
> [    0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:32 nr_node_ids:4
> 

No, this is not the relevant info. I was actually looking for the MADT ACPI
table and whether there are disabled entries like so:

[    0.000000] ACPI: LAPIC (acpi-id[0x03] lapic-id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi-id[0x04] lapic-id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi-id[0x05] lapic-id[0x84] disabled)
[    0.000000] ACPI: LAPIC (acpi-id[0x06] lapic-id[0x85] disabled)

because if there are, nr_cpu_ids will include those when you don't boot
with "possible_cpus=N".

Anyhow, you can read this for more info: http://www.pubbs.net/kernel/200912/64310/

-- 
Regards/Gruss,
    Boris.

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[zhiyi]

On 2/02/2010, at 8:07 PM, Borislav Petkov wrote:

> On Tue, Feb 02, 2010 at 01:49:06PM +1300, zhiyi wrote:
>> The dmesg relevant to nr_cpu_ids of the quad socket is:
>>
>> [    0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:32  
>> nr_node_ids:4
>>
>
> No, this is not the relevant info. I was actually looking for the  
> MADT ACPI
> table and whether there are disabled entries like so:
>
> [    0.000000] ACPI: LAPIC (acpi-id[0x03] lapic-id[0x02] enabled)
> [    0.000000] ACPI: LAPIC (acpi-id[0x04] lapic-id[0x03] enabled)
> [    0.000000] ACPI: LAPIC (acpi-id[0x05] lapic-id[0x84] disabled)
> [    0.000000] ACPI: LAPIC (acpi-id[0x06] lapic-id[0x85] disabled)
>
> because if there are, nr_cpu_ids will include those when you don't  
> boot
> with "possible_cpus=N".

Thanks. I understand the problem now. Below is the related message on  
my quad-socket. Not surprisingly nr_cpu_ids should be 32 since the  
machine can potentially accommodate 32 cores.

The good thing is that I realized I should use num_present_cpus()  
instead of nr_cpu_ids for my purpose (get the number of real cores on  
the machine)

Many thanks,
Zhiyi


[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x0c] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x08] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0d] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x09] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x05] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0e] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0a] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0f] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0b] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x07] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x20] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x21] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x22] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x23] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x24] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x25] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x26] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x27] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x28] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x29] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x2a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x2b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x2c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x2d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x2e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x2f] disabled)




>
>
> Anyhow, you can read this for more info: http://www.pubbs.net/kernel/200912/64310/
>
> -- 
> Regards/Gruss,
>    Boris.
>

=======================
Zhiyi Huang
Dept of Computer Science
University of Otago
Email: hzy@cs.otago.ac.nz
Phone: 0064-3-4795680
Fax: 0064-3-4798529

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[Borislav Petkov]

On Wed, Feb 03, 2010 at 10:33:46AM +1300, zhiyi wrote:
> The good thing is that I realized I should use num_present_cpus()
> instead of nr_cpu_ids for my purpose (get the number of real cores
> on the machine)

Just a minor thing: there could be a subtlety with hotplug when
using num_present_cpus() since this includes all CPUs, even the
hotplug-offlined ones. Depending on your case, you might want to use
num_online_cpus() instead. Look at the comment at the beginning of
<include/linux/cpumask.h> which explains all the different masks to
figure out which one fits your needs best.

-- 
Regards/Gruss,
    Boris.

Re: nr_cpu_ids incorrect on AMD Quad-Core Opteron 8380

[zhiyi]

On 2/02/2010, at 2:57 AM, Borislav Petkov wrote:

> On Sun, Jan 31, 2010 at 11:59 PM, zhiyi <zhuang@cs.otago.ac.nz> wrote:
>> My module worked for a kernel 2.6.29 with CONFIG_NR_CPUS=64 on my  
>> dual
>> quad-core AMD Opteron. The value of nr_cpu_ids I got from the same  
>> module
>> was correct (8).
>
> This could mean that your include/generated/autoconf.h, which is  
> used by
> external modules, is not updated and contains CONFIG_NR_CPUS=8.

I have checked include/linux/autoconf.h in my linux src and found
#define CONFIG_NR_CPUS 64

but I don't see any dir like "generated" under include/

By the way, I compiled my module independent of the compilation of  
Linux tree, i.e. after the kernel is compiled and booted.

Cheers,
Zhiyi


> Do
>
> make mrproper
>
> in your kernel directory by moving your .config out of the way first  
> and
> then rebuild your kernel and external module(s). nr_cpu_ids should  
> pick
> up the updated CONFIG_NR_CPUS value from your .config.
>
> -- 
> Regards/Gruss,
> Boris
>