Inter processor synchronization

Inter processor synchronization

[Nilanjan Roychowdhury]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

Hi,

I have a scenario where two images of the same Linux kernel are running
on two MIPS cores. One is 24K and another is 4KEC. What is the best way
to achieve inter processor synchronization between them?

I guess the locks for LL/SC are local to a particular core and can not
be extended across a multi core system. 

 

Will it be easier for me if both of them becomes same core ( like both
24k) and I run the SMP version of Linux.

Please throw some light.

 

Thanks,

Nilanjan

 

 


[-- Attachment #2: Type: text/html, Size: 3026 bytes --]

Inter processor synchronization

[Nilanjan Roychowdhury]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

Hi,

I have a scenario where two images of the same Linux kernel are running
on two MIPS cores. One is 24K and another is 4KEC. What is the best way
to achieve inter processor synchronization between them?

I guess the locks for LL/SC are local to a particular core and can not
be extended across a multi core system. 

 

Will it be easier for me if both of them becomes same core ( like both
24k) and I run the SMP version of Linux.

Please throw some light.

 

Thanks,

Nilanjan

 

 


[-- Attachment #2: Type: text/html, Size: 3026 bytes --]

Re: Inter processor synchronization

[Ralf Baechle]

On Thu, Dec 13, 2007 at 09:07:20AM +0530, Nilanjan Roychowdhury wrote:

> I have a scenario where two images of the same Linux kernel are running
> on two MIPS cores. One is 24K and another is 4KEC. What is the best way
> to achieve inter processor synchronization between them?
> 
> I guess the locks for LL/SC are local to a particular core and can not
> be extended across a multi core system. 

4K and 24K cores don't support cache coherency.  So SMP is out of question.
This is a _total_ showstopper for SMP, don't waste your time thinking on
possible workarounds.

The you could do is some sort of clusting, running two OS images, one
on the 4K and one on the 24K which would communicate through a carefully
cache managed or even uncached shared memory region.

> Will it be easier for me if both of them becomes same core ( like both
> 24k) and I run the SMP version of Linux.


Within limits Linux supports mixing different CPU types such as R4000MC /
R4400MC and R10000 / R12000 / R14000 mixes because those processors are
similar enough

  Ralf

RE: Inter processor synchronization

[Nilanjan Roychowdhury]

on Thursday, December 13, 2007 6:29 PM:, Ralf Baechle wrote:
> On Thu, Dec 13, 2007 at 09:07:20AM +0530, Nilanjan Roychowdhury wrote:
> 
>> I have a scenario where two images of the same Linux kernel are
>> running on two MIPS cores. One is 24K and another is 4KEC. What is
>> the best way to achieve inter processor synchronization between them?
>> 
>> I guess the locks for LL/SC are local to a particular core and can
>> not be extended across a multi core system.

 
> 4K and 24K cores don't support cache coherency.  So SMP is out of
> question. 
> This is a _total_ showstopper for SMP, don't waste your time thinking
> on possible workarounds. 
> 
> The you could do is some sort of clusting, running two OS images, one
> on the 4K and one on the 24K which would communicate through a
> carefully cache managed or even uncached shared memory region.  

I guess I am left with only this option. Can you please throw some more
lights
On the IPC you are mentioning?



>> Will it be easier for me if both of them becomes same core ( like
>> both 24k) and I run the SMP version of Linux.
> 
> 
> Within limits Linux supports mixing different CPU types such as
> R4000MC / R4400MC and R10000 / R12000 / R14000 mixes because those
> processors are similar enough  
> 
>   Ralf

Thanks,
Nilanjan

RE: Inter processor synchronization

[Nilanjan Roychowdhury]

on Thursday, December 13, 2007 6:29 PM:, Ralf Baechle wrote:
> On Thu, Dec 13, 2007 at 09:07:20AM +0530, Nilanjan Roychowdhury wrote:
> 
>> I have a scenario where two images of the same Linux kernel are
>> running on two MIPS cores. One is 24K and another is 4KEC. What is
>> the best way to achieve inter processor synchronization between them?
>> 
>> I guess the locks for LL/SC are local to a particular core and can
>> not be extended across a multi core system.

 
> 4K and 24K cores don't support cache coherency.  So SMP is out of
> question. 
> This is a _total_ showstopper for SMP, don't waste your time thinking
> on possible workarounds. 
> 
> The you could do is some sort of clusting, running two OS images, one
> on the 4K and one on the 24K which would communicate through a
> carefully cache managed or even uncached shared memory region.  

I guess I am left with only this option. Can you please throw some more
lights
On the IPC you are mentioning?



>> Will it be easier for me if both of them becomes same core ( like
>> both 24k) and I run the SMP version of Linux.
> 
> 
> Within limits Linux supports mixing different CPU types such as
> R4000MC / R4400MC and R10000 / R12000 / R14000 mixes because those
> processors are similar enough  
> 
>   Ralf

Thanks,
Nilanjan

Re: Inter processor synchronization

[Kevin D. Kissell]

> >> I have a scenario where two images of the same Linux kernel are
> >> running on two MIPS cores. One is 24K and another is 4KEC. What is
> >> the best way to achieve inter processor synchronization between them?
> >> 
> >> I guess the locks for LL/SC are local to a particular core and can
> >> not be extended across a multi core system.

Just to be clear,  LL/SC are indeed local to a particular core *but*, 
in a cache coherent multiprocessor system, they provide multiprocessor
synchronization - the fact that another core has referenced the coherent
location will clear the link bit so that the SC will fail locally.

> > 4K and 24K cores don't support cache coherency.  So SMP is out of
> > question. 
> > This is a _total_ showstopper for SMP, don't waste your time thinking
> > on possible workarounds. 
> > 
> > The you could do is some sort of clusting, running two OS images, one
> > on the 4K and one on the 24K which would communicate through a
> > carefully cache managed or even uncached shared memory region.  
> 
> I guess I am left with only this option. Can you please throw some more
> lights On the IPC you are mentioning?

Unless one has special-purpose hardware that implements atomic operations
(e.g. a hardware semaphore device), one must use algorithms that do not
require atomic read-modify-write.  Most classically, one uses mailboxes 
where each memory location has a single reader and a single writer.  There 
are other, more general but less efficient algorithms (e.g. Decker's algorithm)
out there as well.  If one is doing this in cacheable memory, one needs
to take care that (a) an explicit forced cache writeback operation is done
to complete each update to the shared memory array, and (b) the "ownership" 
is at a granularity of a cache line, and not a memory word.  If the memory
is mapped uncached, and one has a message queue "next" pointer that
is written by CPU A and a "last-read" pointer that is written by B, those two
pointers can be in consecutive memory locations.  But if the memory is cached,
they must be in separate cache lines to avoid the writebacks of one CPU
destroying the writebacks of another.

            Regards,

            Kevin K.

Re: Inter processor synchronization

[Kevin D. Kissell]

> >> I have a scenario where two images of the same Linux kernel are
> >> running on two MIPS cores. One is 24K and another is 4KEC. What is
> >> the best way to achieve inter processor synchronization between them?
> >> 
> >> I guess the locks for LL/SC are local to a particular core and can
> >> not be extended across a multi core system.

Just to be clear,  LL/SC are indeed local to a particular core *but*, 
in a cache coherent multiprocessor system, they provide multiprocessor
synchronization - the fact that another core has referenced the coherent
location will clear the link bit so that the SC will fail locally.

> > 4K and 24K cores don't support cache coherency.  So SMP is out of
> > question. 
> > This is a _total_ showstopper for SMP, don't waste your time thinking
> > on possible workarounds. 
> > 
> > The you could do is some sort of clusting, running two OS images, one
> > on the 4K and one on the 24K which would communicate through a
> > carefully cache managed or even uncached shared memory region.  
> 
> I guess I am left with only this option. Can you please throw some more
> lights On the IPC you are mentioning?

Unless one has special-purpose hardware that implements atomic operations
(e.g. a hardware semaphore device), one must use algorithms that do not
require atomic read-modify-write.  Most classically, one uses mailboxes 
where each memory location has a single reader and a single writer.  There 
are other, more general but less efficient algorithms (e.g. Decker's algorithm)
out there as well.  If one is doing this in cacheable memory, one needs
to take care that (a) an explicit forced cache writeback operation is done
to complete each update to the shared memory array, and (b) the "ownership" 
is at a granularity of a cache line, and not a memory word.  If the memory
is mapped uncached, and one has a message queue "next" pointer that
is written by CPU A and a "last-read" pointer that is written by B, those two
pointers can be in consecutive memory locations.  But if the memory is cached,
they must be in separate cache lines to avoid the writebacks of one CPU
destroying the writebacks of another.

            Regards,

            Kevin K.