A problem with percpu variable cpu_number

A problem with percpu variable cpu_number

[Tao Jiang]

Hi:

I've asked the question on kernelnewbies, no replies.
So forward it here.
Thank you.


---------- Forwarded message ----------
From: tao jiang <jiangtao.jit@gmail.com>
Date: 2012/2/16
Subject: A problem with percpu variable cpu_number
To: kernelnewbies <kernelnewbies@kernelnewbies.org>


Hi :

At the beginning of start_kernel() -- init/main.c
boot_cpu_init() is called

and in boot_cpu_init() will call smp_processor_id()
it's a macro
and it will be expanded as (percpu_read(cpu_number)) and so on

but i noticed that it's before setup_per_cpu_areas() in start_kernel()
is that mean the percpu areas hadn't be initialized yet
but why smp_processor_id() could be called before setup_per_cpu_areas()


Thank you.

---------------
jiangtao

Re: A problem with percpu variable cpu_number

[Cong Wang]

On 02/19/2012 08:20 PM, Tao Jiang wrote:
> Hi :
>
> At the beginning of start_kernel() -- init/main.c
> boot_cpu_init() is called
>
> and in boot_cpu_init() will call smp_processor_id()
> it's a macro
> and it will be expanded as (percpu_read(cpu_number)) and so on
>
> but i noticed that it's before setup_per_cpu_areas() in start_kernel()
> is that mean the percpu areas hadn't be initialized yet
> but why smp_processor_id() could be called before setup_per_cpu_areas()

It doesn't matter, as the percpu var 'cpu_number' is defined statically: 
DECLARE_PER_CPU(int, cpu_number);.

Thanks.

Re: A problem with percpu variable cpu_number

[Tao Jiang]

Cong Wang:

Thanks for you reply.
But I think it's not the truth.

The percpu variable cpu_number is defined in arch/x86/kernel/setup_percpu.c
use DEFINE_PER_CPU(int, cpu_number);
and declared in head file arch/x86/include/asm/smp.h
use DECLARE_PER_CPU(int, cpu_number);

I read the macor DEFINE_PER_CPU
it's decorated by some percpu attributes
it will be put in section .data..percpu in init section
and init section will be free after the kernel had been initialized
so at the beginning of start_kernel()
what smp_processor_id() read is not initialized yet
am I right ?
Let me know if i made some misunderstanding of the code.
Thank you.




2012/2/19 Cong Wang <xiyou.wangcong@gmail.com>:
> On 02/19/2012 08:20 PM, Tao Jiang wrote:
>>
>> Hi :
>>
>> At the beginning of start_kernel() -- init/main.c
>> boot_cpu_init() is called
>>
>> and in boot_cpu_init() will call smp_processor_id()
>> it's a macro
>> and it will be expanded as (percpu_read(cpu_number)) and so on
>>
>> but i noticed that it's before setup_per_cpu_areas() in start_kernel()
>> is that mean the percpu areas hadn't be initialized yet
>> but why smp_processor_id() could be called before setup_per_cpu_areas()
>
>
> It doesn't matter, as the percpu var 'cpu_number' is defined statically:
> DECLARE_PER_CPU(int, cpu_number);.
>
> Thanks.

Re: A problem with percpu variable cpu_number

[Cong Wang]

On 02/20/2012 07:04 PM, Tao Jiang wrote:
>
> I read the macor DEFINE_PER_CPU
> it's decorated by some percpu attributes
> it will be put in section .data..percpu in init section

.data..percpu is not .init section, the .init sections should all have 
".init.*" names.

Re: A problem with percpu variable cpu_number

[Tao Jiang]

Hi Cong Wang:

I read the file vmlinux.lds.S in arch/x86/kernel
section .data..percpu is between .init.data and .init.end
Is that means these percpu variables will be freed after init?

Thank you.


2012/2/20 Cong Wang <xiyou.wangcong@gmail.com>:
> On 02/20/2012 07:04 PM, Tao Jiang wrote:
>>
>>
>> I read the macor DEFINE_PER_CPU
>> it's decorated by some percpu attributes
>> it will be put in section .data..percpu in init section
>
>
> .data..percpu is not .init section, the .init sections should all have
> ".init.*" names.

Re: A problem with percpu variable cpu_number

[Cong Wang]

On 02/21/2012 08:41 PM, Tao Jiang wrote:
> Hi Cong Wang:
>
> I read the file vmlinux.lds.S in arch/x86/kernel
> section .data..percpu is between .init.data and .init.end
> Is that means these percpu variables will be freed after init?

% grep -e __init_begin -e __init_end -e __per_cpu_start -e __per_cpu_end 
/boot/System.map
0000000000000000 D __per_cpu_start
0000000000014bc0 D __per_cpu_end
ffffffff81cf3000 D __init_begin
ffffffff81dfc000 R __init_end
% objdump -d -j .data..percpu vmlinux | grep cpu_number
000000000000dc38 <cpu_number>:

Re: A problem with percpu variable cpu_number

[Brian Gerst]

On Tue, Feb 21, 2012 at 9:29 AM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> On 02/21/2012 08:41 PM, Tao Jiang wrote:
>>
>> Hi Cong Wang:
>>
>> I read the file vmlinux.lds.S in arch/x86/kernel
>> section .data..percpu is between .init.data and .init.end
>> Is that means these percpu variables will be freed after init?
>
>
> % grep -e __init_begin -e __init_end -e __per_cpu_start -e __per_cpu_end
> /boot/System.map
> 0000000000000000 D __per_cpu_start
> 0000000000014bc0 D __per_cpu_end
> ffffffff81cf3000 D __init_begin
> ffffffff81dfc000 R __init_end
> % objdump -d -j .data..percpu vmlinux | grep cpu_number
> 000000000000dc38 <cpu_number>:

The .data..percpu section is placed in the init section, but x86-64 is
a special case as noted below.  The boot cpu is pointed to the init
percpu section until setup_per_cpu_areas() is called, when it switches
to the regular percpu area.  The init percpu data is then freed with
all other init data.

The reason the percpu symbols start at virtual address 0 on x86-64 is
because of the requirement that gs_base must be a canonical address
(it cannot be a simple offset like x86-32).  But, the data is still
loaded in the init section in memory.  See
arch/x86/kernel/vmlinux.lds.S for the explanation of how the linker
changes the program headers to set the virtual address to zero but
keeps the load address in the init section.

To answer the original question. in the case of cpu_number, it is set
to zero in the init section because it doesn't have an explicit
initializer.  Therefore the boot cpu will always read zero for the
cpu_number, even before setup_per_cpu_areas() is called.


--
Brian Gerst

Re: A problem with percpu variable cpu_number

[Tao Jiang]

Hi:

Thank you all.

So in boot_cpu_init(), it will always set bit 0 to these masks.
If the boot cpu is the first processor, it's the right case.
And if the BP is not the first one, is it wrong?
But can it happen that the BP is not cpu0?
Thank you.


2012/2/22 Brian Gerst <brgerst@gmail.com>:
> On Tue, Feb 21, 2012 at 9:29 AM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>> On 02/21/2012 08:41 PM, Tao Jiang wrote:
>>>
>>> Hi Cong Wang:
>>>
>>> I read the file vmlinux.lds.S in arch/x86/kernel
>>> section .data..percpu is between .init.data and .init.end
>>> Is that means these percpu variables will be freed after init?
>>
>>
>> % grep -e __init_begin -e __init_end -e __per_cpu_start -e __per_cpu_end
>> /boot/System.map
>> 0000000000000000 D __per_cpu_start
>> 0000000000014bc0 D __per_cpu_end
>> ffffffff81cf3000 D __init_begin
>> ffffffff81dfc000 R __init_end
>> % objdump -d -j .data..percpu vmlinux | grep cpu_number
>> 000000000000dc38 <cpu_number>:
>
> The .data..percpu section is placed in the init section, but x86-64 is
> a special case as noted below.  The boot cpu is pointed to the init
> percpu section until setup_per_cpu_areas() is called, when it switches
> to the regular percpu area.  The init percpu data is then freed with
> all other init data.
>
> The reason the percpu symbols start at virtual address 0 on x86-64 is
> because of the requirement that gs_base must be a canonical address
> (it cannot be a simple offset like x86-32).  But, the data is still
> loaded in the init section in memory.  See
> arch/x86/kernel/vmlinux.lds.S for the explanation of how the linker
> changes the program headers to set the virtual address to zero but
> keeps the load address in the init section.
>
> To answer the original question. in the case of cpu_number, it is set
> to zero in the init section because it doesn't have an explicit
> initializer.  Therefore the boot cpu will always read zero for the
> cpu_number, even before setup_per_cpu_areas() is called.
>
>
> --
> Brian Gerst

Re: A problem with percpu variable cpu_number

[Hillf Danton]

On Wed, Feb 22, 2012 at 7:13 PM, Tao Jiang <jiangtao.jit@gmail.com> wrote:
> Hi:
>
> Thank you all.
>
> So in boot_cpu_init(), it will always set bit 0 to these masks.
> If the boot cpu is the first processor, it's the right case.
> And if the BP is not the first one, is it wrong?
> But can it happen that the BP is not cpu0?

BP could be not "the first one".
Due to that current code boots fine, I guess, you mess
logic CPU with hard CPU.

btw, replying messages in this way looks not fine.

> Thank you.
>
>
> 2012/2/22 Brian Gerst <brgerst@gmail.com>:
>> On Tue, Feb 21, 2012 at 9:29 AM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>>> On 02/21/2012 08:41 PM, Tao Jiang wrote:
>>>>
>>>> Hi Cong Wang:
>>>>
>>>> I read the file vmlinux.lds.S in arch/x86/kernel
>>>> section .data..percpu is between .init.data and .init.end
>>>> Is that means these percpu variables will be freed after init?
>>>
>>>
>>> % grep -e __init_begin -e __init_end -e __per_cpu_start -e __per_cpu_end
>>> /boot/System.map
>>> 0000000000000000 D __per_cpu_start
>>> 0000000000014bc0 D __per_cpu_end
>>> ffffffff81cf3000 D __init_begin
>>> ffffffff81dfc000 R __init_end
>>> % objdump -d -j .data..percpu vmlinux | grep cpu_number
>>> 000000000000dc38 <cpu_number>:
>>
>> The .data..percpu section is placed in the init section, but x86-64 is
>> a special case as noted below.  The boot cpu is pointed to the init
>> percpu section until setup_per_cpu_areas() is called, when it switches
>> to the regular percpu area.  The init percpu data is then freed with
>> all other init data.
>>
>> The reason the percpu symbols start at virtual address 0 on x86-64 is
>> because of the requirement that gs_base must be a canonical address
>> (it cannot be a simple offset like x86-32).  But, the data is still
>> loaded in the init section in memory.  See
>> arch/x86/kernel/vmlinux.lds.S for the explanation of how the linker
>> changes the program headers to set the virtual address to zero but
>> keeps the load address in the init section.
>>
>> To answer the original question. in the case of cpu_number, it is set
>> to zero in the init section because it doesn't have an explicit
>> initializer.  Therefore the boot cpu will always read zero for the
>> cpu_number, even before setup_per_cpu_areas() is called.
>>
>>
>> --
>> Brian Gerst
> --

Re: A problem with percpu variable cpu_number

[Tao Jiang]

Hillf Danton:

Thanks for your remind
and sorry for my poor English.


2012/2/22 Hillf Danton <dhillf@gmail.com>:
> On Wed, Feb 22, 2012 at 7:13 PM, Tao Jiang <jiangtao.jit@gmail.com> wrote:
>> Hi:
>>
>> Thank you all.
>>
>> So in boot_cpu_init(), it will always set bit 0 to these masks.
>> If the boot cpu is the first processor, it's the right case.
>> And if the BP is not the first one, is it wrong?
>> But can it happen that the BP is not cpu0?
>
> BP could be not "the first one".
> Due to that current code boots fine, I guess, you mess
> logic CPU with hard CPU.
>
> btw, replying messages in this way looks not fine.
>
>> Thank you.
>>
>>
>> 2012/2/22 Brian Gerst <brgerst@gmail.com>:
>>> On Tue, Feb 21, 2012 at 9:29 AM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
>>>> On 02/21/2012 08:41 PM, Tao Jiang wrote:
>>>>>
>>>>> Hi Cong Wang:
>>>>>
>>>>> I read the file vmlinux.lds.S in arch/x86/kernel
>>>>> section .data..percpu is between .init.data and .init.end
>>>>> Is that means these percpu variables will be freed after init?
>>>>
>>>>
>>>> % grep -e __init_begin -e __init_end -e __per_cpu_start -e __per_cpu_end
>>>> /boot/System.map
>>>> 0000000000000000 D __per_cpu_start
>>>> 0000000000014bc0 D __per_cpu_end
>>>> ffffffff81cf3000 D __init_begin
>>>> ffffffff81dfc000 R __init_end
>>>> % objdump -d -j .data..percpu vmlinux | grep cpu_number
>>>> 000000000000dc38 <cpu_number>:
>>>
>>> The .data..percpu section is placed in the init section, but x86-64 is
>>> a special case as noted below.  The boot cpu is pointed to the init
>>> percpu section until setup_per_cpu_areas() is called, when it switches
>>> to the regular percpu area.  The init percpu data is then freed with
>>> all other init data.
>>>
>>> The reason the percpu symbols start at virtual address 0 on x86-64 is
>>> because of the requirement that gs_base must be a canonical address
>>> (it cannot be a simple offset like x86-32).  But, the data is still
>>> loaded in the init section in memory.  See
>>> arch/x86/kernel/vmlinux.lds.S for the explanation of how the linker
>>> changes the program headers to set the virtual address to zero but
>>> keeps the load address in the init section.
>>>
>>> To answer the original question. in the case of cpu_number, it is set
>>> to zero in the init section because it doesn't have an explicit
>>> initializer.  Therefore the boot cpu will always read zero for the
>>> cpu_number, even before setup_per_cpu_areas() is called.
>>>
>>>
>>> --
>>> Brian Gerst
>> --