PER-CPU data

PER-CPU data

[Rajasekhar Pulluru]

Hi,

I would like to know how per-cpu data are stored internally?
And how are they protected from other cores?

Thanks & Regards,
Rajasekhar

PER-CPU data

[Dave Hylands]

Hi Rajasekhar,

On Thu, Mar 29, 2012 at 11:00 PM, Rajasekhar Pulluru
<pullururajasekhar@gmail.com> wrote:
> Hi,
>
> I would like to know how per-cpu data are stored internally?
> And how are they protected from other cores?

I believe that they're just kmalloc'd like other kernel data. At the
kernel level there is no protection, just like all the rest of the
memory accessible to the kernel.
http://lxr.linux.no/#linux+v3.3/include/asm-generic/percpu.h#L8
http://lxr.linux.no/#linux+v3.3/mm/percpu.c

When you declare a per-cpu variable, it goes into a special section,
and what you're really doing is figuring out the offset within a
per_cpu region of memory.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

PER-CPU data

[Srivatsa S. Bhat]

On 03/30/2012 12:05 PM, Dave Hylands wrote:

> Hi Rajasekhar,
> 
> On Thu, Mar 29, 2012 at 11:00 PM, Rajasekhar Pulluru
> <pullururajasekhar@gmail.com> wrote:
>> Hi,
>>
>> I would like to know how per-cpu data are stored internally?
>> And how are they protected from other cores?
> 


To put it in very simplistic terms, per-cpu data is nothing but having
NR_CPUS copies of the data, like an array, something like:

int data[NR_CPUS];

And accessing this per-cpu data will essentially boil down to finding
out the id of the processor you are running on, and indexing this array
using that, something like:

int val, cpu;

cpu = smp_processor_id();
val = data[cpu];

So you automatically read/write the copy that belongs to your processor.
That's it. However, this is an over-simplified view of per-cpu data,
but you get the general idea...

> I believe that they're just kmalloc'd like other kernel data. At the
> kernel level there is no protection, just like all the rest of the
> memory accessible to the kernel.
> http://lxr.linux.no/#linux+v3.3/include/asm-generic/percpu.h#L8
> http://lxr.linux.no/#linux+v3.3/mm/percpu.c
> 
> When you declare a per-cpu variable, it goes into a special section,
> and what you're really doing is figuring out the offset within a
> per_cpu region of memory.
> 

 
Regards,
Srivatsa S. Bhat

PER-CPU data

[Rajasekhar Pulluru]

Hi Srivatsa,

Thanks for the response. I have used per-CPU vars and I know about how
to creating/using per-CPU vars: DECLARE_PER_CPU(type, name) for
creating per-cpu at compile time and use alloc_percpu(type) for
creating them dynamically.

I intended to ask how they are stored internally (.percpu section) and
its protection mechanism if it has any.

Thanks & Regards,
Rajasekhar


On Fri, Mar 30, 2012 at 12:24 PM, Srivatsa S. Bhat
<srivatsa.bhat@linux.vnet.ibm.com> wrote:
> On 03/30/2012 12:05 PM, Dave Hylands wrote:
>
>> Hi Rajasekhar,
>>
>> On Thu, Mar 29, 2012 at 11:00 PM, Rajasekhar Pulluru
>> <pullururajasekhar@gmail.com> wrote:
>>> Hi,
>>>
>>> I would like to know how per-cpu data are stored internally?
>>> And how are they protected from other cores?
>>
>
>
> To put it in very simplistic terms, per-cpu data is nothing but having
> NR_CPUS copies of the data, like an array, something like:
>
> int data[NR_CPUS];
>
> And accessing this per-cpu data will essentially boil down to finding
> out the id of the processor you are running on, and indexing this array
> using that, something like:
>
> int val, cpu;
>
> cpu = smp_processor_id();
> val = data[cpu];
>
> So you automatically read/write the copy that belongs to your processor.
> That's it. However, this is an over-simplified view of per-cpu data,
> but you get the general idea...
>
>> I believe that they're just kmalloc'd like other kernel data. At the
>> kernel level there is no protection, just like all the rest of the
>> memory accessible to the kernel.
>> http://lxr.linux.no/#linux+v3.3/include/asm-generic/percpu.h#L8
>> http://lxr.linux.no/#linux+v3.3/mm/percpu.c
>>
>> When you declare a per-cpu variable, it goes into a special section,
>> and what you're really doing is figuring out the offset within a
>> per_cpu region of memory.
>>
>
>
> Regards,
> Srivatsa S. Bhat
>

PER-CPU data

[Srivatsa S. Bhat]

On 03/30/2012 12:05 PM, Dave Hylands wrote:

> Hi Rajasekhar,
> 
> On Thu, Mar 29, 2012 at 11:00 PM, Rajasekhar Pulluru
> <pullururajasekhar@gmail.com> wrote:
>> Hi,
>>
>> I would like to know how per-cpu data are stored internally?
>> And how are they protected from other cores?
> 


To put it in very simplistic terms, per-cpu data is nothing but having
NR_CPUS copies of the data, like an array, something like:

int data[NR_CPUS];

And accessing this per-cpu data will essentially boil down to finding
out the id of the processor you are running on, and indexing this array
using that, something like:

int val, cpu;

cpu = smp_processor_id();
val = data[cpu];

So you automatically read/write the copy that belongs to your processor.
That's it. However, this is an over-simplified view of per-cpu data,
but you get the general idea...

> I believe that they're just kmalloc'd like other kernel data. At the
> kernel level there is no protection, just like all the rest of the
> memory accessible to the kernel.
> http://lxr.linux.no/#linux+v3.3/include/asm-generic/percpu.h#L8
> http://lxr.linux.no/#linux+v3.3/mm/percpu.c
> 
> When you declare a per-cpu variable, it goes into a special section,
> and what you're really doing is figuring out the offset within a
> per_cpu region of memory.
> 

 
Regards,
Srivatsa S. Bhat

per-cpu data...

[Rusty Russell]

IIUC, __per_cpu_data is insufficient for Alpha as stands, to use
thread-specific gcc tricks (__thread prepended to the decl, even
though they had a perfectly good __attribute__ extension already).

So, I guess we're stuck with something like:

DECLARE_PER_CPU(int x);

If we're going to do this, we can also mangle the name as well to
avoid accidental "direct" accesses:

eg:
	#define DECLARE_PER_CPU(var) 
		var##__percpu __attribute__((section(".percpu")))

	/* If not SMP: */
	#define per_cpu(var) var##__percpu

(From my reading, ## on "int x" and "__per_cpu" is well-defined).

Thoughts?
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: per-cpu data...

[Richard Henderson]

On Fri, Jul 12, 2002 at 04:01:52PM +1000, Rusty Russell wrote:
> (__thread prepended to the decl, even
> though they had a perfectly good __attribute__ extension already).

Sorry, not our extension.  I just copied it.

> (From my reading, ## on "int x" and "__per_cpu" is well-defined).

Yep.

> Thoughts?

Seems ok, though you really ought to differentiate between
DECLARE and DEFINE.  Otherwise I can see getting screwed again.


r~

Re: per-cpu data...

[Jamie Lokier]

Rusty Russell wrote:
> (From my reading, ## on "int x" and "__per_cpu" is well-defined).

  DECLARE_PER_CPU (int x[3]);

doesn't work, although you can always do

  typedef int three_ints_t[3];
  DECLARE_PER_CPU (three_ints_t x);

I encountered the same thing while doing a user-space
`MAKE_THREAD_SPECIFIC' macro.  The solution I went for looks like this:

  #define DECLARE_PER_CPU(type, name) \
    __attribute__ ((__section (".percpu"))) __typeof__ (type) name##__per_cpu

enjoy,
-- Jamie

Re: per-cpu data...

[Rusty Russell]

In message <20020724133128.A7192@kushida.apsleyroad.org> you write:
> Rusty Russell wrote:
> > (From my reading, ## on "int x" and "__per_cpu" is well-defined).
> 
>   DECLARE_PER_CPU (int x[3]);
> 
> doesn't work, although you can always do
> 
>   typedef int three_ints_t[3];
>   DECLARE_PER_CPU (three_ints_t x);
> 
> I encountered the same thing while doing a user-space
> `MAKE_THREAD_SPECIFIC' macro.  The solution I went for looks like this:
> 
>   #define DECLARE_PER_CPU(type, name) \
>     __attribute__ ((__section (".percpu"))) __typeof__ (type) name##__per_cpu

Hmmm.... Yeah, might as well go the whole way.

Thanks!
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.