cacheline alignment and per-cpu data

cacheline alignment and per-cpu data

[Chris Friesen]

Someone asked me a question that I couldn't answer, so I thought I'd 
pass it on to here.

Suppose I declare an array of a struct type, where the size of the 
struct is not a multiple of the cacheline size.  Each element in the 
array is used by a different cpu.

If I understand it, this would mean that the last member in the data 
belonging to one cpu shares a cacheline with the first member in the 
data belonging to the next cpu.

Will this cause cacheline pingpong?  If I do this sort of thing do I 
need to ensure that the struct is a multiple of cacheline size (or 
specify cacheline alignement)?

Thanks,

Chris

Re: cacheline alignment and per-cpu data

[Chase Venters]

On Fri, 9 Jun 2006, Chris Friesen wrote:

>
> Someone asked me a question that I couldn't answer, so I thought I'd pass it 
> on to here.
>
> Suppose I declare an array of a struct type, where the size of the struct is 
> not a multiple of the cacheline size.  Each element in the array is used by a 
> different cpu.
>
> If I understand it, this would mean that the last member in the data 
> belonging to one cpu shares a cacheline with the first member in the data 
> belonging to the next cpu.

Yes. Since an array element position is essentially 
base_ptr + sizeof(each_element) * index, if sizeof(each_element) is not a 
multiple of the cache line, the next element will start in the middle of a 
cache line.

> Will this cause cacheline pingpong?  If I do this sort of thing do I need to 
> ensure that the struct is a multiple of cacheline size (or specify cacheline 
> alignement)?

Yes. If CPU 2 tries to write to struct member 1 of array element 2, and 
array element 1 is in CPU 1's cache, it must be invalidated.

GCC (and kernel macros) provide good support for enforced cacheline 
alignment, but it's also possible to pad your structures.

> Thanks,
>
> Chris

Chase

Re: cacheline alignment and per-cpu data

[Jes Sorensen]

>>>>> "Chase" == Chase Venters <chase.venters@clientec.com> writes:

Chase> On Fri, 9 Jun 2006, Chris Friesen wrote:
>> Will this cause cacheline pingpong?  If I do this sort of thing do
>> I need to ensure that the struct is a multiple of cacheline size
>> (or specify cacheline alignement)?

Chase> Yes. If CPU 2 tries to write to struct member 1 of array
Chase> element 2, and array element 1 is in CPU 1's cache, it must be
Chase> invalidated.

Chase> GCC (and kernel macros) provide good support for enforced
Chase> cacheline alignment, but it's also possible to pad your
Chase> structures.

Best way to go is to use __cacheline_aligned rather than try to get
the padding right manually. Someone will add a field somewhere in the
middle and mess it up and nobody notices.

Cheers,
Jes