SMP questions

SMP questions

[Shreyas Kejariwal]

Hi everyone,

I am new to this arena and I have few questions so please help me answer
them.

1. Looks like most of the new systems employ SMPs. But do we still have
ASMPs systems ?

2. How can we control the tasks, threads to run on a particular processor ?
Is there a way we can specify ? Take the scenario where there are two
multi-threaded applications. I want all the threads of the App1 to run on
Processor, P1 and the threads of the App2 on Processor, P2. Not like some
threads of App1 would run on P2 depending on the scheduler.

3. How can processors access each other's memory / addr space?

4. Digging into the scheduler code, I found that we can either set scheduler
policy to be either Round Robin or FIFO. Are there other schedulers
available which use different policy.

5. I read about Shared and Non-shared Cache. How does L2 gets shared. Its on
the die. So does that mean that you have to buy a shared cache board ?

I apologize if some of the questions are stupid and outside the scope of
this forum.

Moreover, pointers to web-resources would be helpful.

Thanks,
Shreyas

Re: SMP questions

[Robert M. Hyatt]

On Wed, 18 Jun 2003, Shreyas Kejariwal wrote:

> Hi everyone,
> 
> I am new to this arena and I have few questions so please help me answer
> them.
> 
> 1. Looks like most of the new systems employ SMPs. But do we still have
> ASMPs systems ?


If you mean "asymmetric multiprocessing" the answer is generally "no".  
That has pretty much disappeared.  If you mean something like clusters
and so forth, then they are going strong but require different 
programming approaches.


> 
> 2. How can we control the tasks, threads to run on a particular processor ?

Why would you want to?  This is something the operating system should 
handle correctly.


> Is there a way we can specify ? Take the scenario where there are two
> multi-threaded applications. I want all the threads of the App1 to run on
> Processor, P1 and the threads of the App2 on Processor, P2. Not like some
> threads of App1 would run on P2 depending on the scheduler.
> 


Most systems support some sort of "processor affinity" stuff, but it is 
a bad idea in general.  Leave the O/S stuff to the O/S.  It _really_ 
does know better...



> 3. How can processors access each other's memory / addr space?
> 

Both processors are connected to the memory sub-system with a common
bus.



> 4. Digging into the scheduler code, I found that we can either set scheduler
> policy to be either Round Robin or FIFO. Are there other schedulers
> available which use different policy.
> 

What scheduler are you looking at?



> 5. I read about Shared and Non-shared Cache. How does L2 gets shared. Its on
> the die. So does that mean that you have to buy a shared cache board ?
> 

L2 isn't shared on Intel platforms. Each processor has its own L2 cache
on chip.  Otherwise, it is shared just like memory is shared, with a 
common bus connecting it to more than one processor.




> I apologize if some of the questions are stupid and outside the scope of
> this forum.
> 
> Moreover, pointers to web-resources would be helpful.
> 
> Thanks,
> Shreyas
> 
> 
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Re: SMP questions

[Shreyas Kejariwal]

Thanks Robert.

Getting back to Q3, as you mentioned, it uses shared bus. So that makes it
hardware specific.  However, I read that Posix threads can achieve the same
w/o sharing the bus. Is that correct?

Can anyone point which dual/SMP processors share memory bus ?


Shreyas



--

-----------------------------------------------
Shreyas Kejariwal
Perigee LLC
Tel: 315-453 7842 x 42



"Robert M. Hyatt" <hyatt@cis.uab.edu> wrote in message
news:Pine.LNX.4.44.0306191057030.5678-100000@crafty.cis.uab.edu...
> On Wed, 18 Jun 2003, Shreyas Kejariwal wrote:
>
> > Hi everyone,
> >
> > I am new to this arena and I have few questions so please help me answer
> > them.
> >
> > 1. Looks like most of the new systems employ SMPs. But do we still have
> > ASMPs systems ?
>
>
> If you mean "asymmetric multiprocessing" the answer is generally "no".
> That has pretty much disappeared.  If you mean something like clusters
> and so forth, then they are going strong but require different
> programming approaches.
>
>
> >
> > 2. How can we control the tasks, threads to run on a particular
processor ?
>
> Why would you want to?  This is something the operating system should
> handle correctly.
>
>
> > Is there a way we can specify ? Take the scenario where there are two
> > multi-threaded applications. I want all the threads of the App1 to run
on
> > Processor, P1 and the threads of the App2 on Processor, P2. Not like
some
> > threads of App1 would run on P2 depending on the scheduler.
> >
>
>
> Most systems support some sort of "processor affinity" stuff, but it is
> a bad idea in general.  Leave the O/S stuff to the O/S.  It _really_
> does know better...
>
>
>
> > 3. How can processors access each other's memory / addr space?
> >
>
> Both processors are connected to the memory sub-system with a common
> bus.
>
>
>
> > 4. Digging into the scheduler code, I found that we can either set
scheduler
> > policy to be either Round Robin or FIFO. Are there other schedulers
> > available which use different policy.
> >
>
> What scheduler are you looking at?
>
>
>
> > 5. I read about Shared and Non-shared Cache. How does L2 gets shared.
Its on
> > the die. So does that mean that you have to buy a shared cache board ?
> >
>
> L2 isn't shared on Intel platforms. Each processor has its own L2 cache
> on chip.  Otherwise, it is shared just like memory is shared, with a
> common bus connecting it to more than one processor.
>
>
>
>
> > I apologize if some of the questions are stupid and outside the scope of
> > this forum.
> >
> > Moreover, pointers to web-resources would be helpful.
> >
> > Thanks,
> > Shreyas
> >
> >
> >
> >
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> --
> Robert Hyatt                    Computer and Information Sciences
> hyatt@cis.uab.edu               University of Alabama at Birmingham
> (205) 934-2213                  115A Campbell Hall, UAB Station
> (205) 934-5473 FAX              Birmingham, AL 35294-1170
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: SMP questions

[Robert M. Hyatt]

On Thu, 19 Jun 2003, Shreyas Kejariwal wrote:

> Thanks Robert.
> 
> Getting back to Q3, as you mentioned, it uses shared bus. So that makes it
> hardware specific.  However, I read that Posix threads can achieve the same
> w/o sharing the bus. Is that correct?

no.  The two concepts are not related.  Posix threads _uses_ shared
memory, with no regard to how it is shared.  IE on a PC the shared bus
connects multiple processors to memory.  On a Cray, it is a crossbar 
that connects processors to memory.  But "threads" have no idea about
the underlying hardware details.




> 
> Can anyone point which dual/SMP processors share memory bus ?


_any_ intel/AMD machine with more than one processor does this.



> 
> 
> Shreyas
> 
> 
> 
> --
> 
> -----------------------------------------------
> Shreyas Kejariwal
> Perigee LLC
> Tel: 315-453 7842 x 42
> 
> 
> 
> "Robert M. Hyatt" <hyatt@cis.uab.edu> wrote in message
> news:Pine.LNX.4.44.0306191057030.5678-100000@crafty.cis.uab.edu...
> > On Wed, 18 Jun 2003, Shreyas Kejariwal wrote:
> >
> > > Hi everyone,
> > >
> > > I am new to this arena and I have few questions so please help me answer
> > > them.
> > >
> > > 1. Looks like most of the new systems employ SMPs. But do we still have
> > > ASMPs systems ?
> >
> >
> > If you mean "asymmetric multiprocessing" the answer is generally "no".
> > That has pretty much disappeared.  If you mean something like clusters
> > and so forth, then they are going strong but require different
> > programming approaches.
> >
> >
> > >
> > > 2. How can we control the tasks, threads to run on a particular
> processor ?
> >
> > Why would you want to?  This is something the operating system should
> > handle correctly.
> >
> >
> > > Is there a way we can specify ? Take the scenario where there are two
> > > multi-threaded applications. I want all the threads of the App1 to run
> on
> > > Processor, P1 and the threads of the App2 on Processor, P2. Not like
> some
> > > threads of App1 would run on P2 depending on the scheduler.
> > >
> >
> >
> > Most systems support some sort of "processor affinity" stuff, but it is
> > a bad idea in general.  Leave the O/S stuff to the O/S.  It _really_
> > does know better...
> >
> >
> >
> > > 3. How can processors access each other's memory / addr space?
> > >
> >
> > Both processors are connected to the memory sub-system with a common
> > bus.
> >
> >
> >
> > > 4. Digging into the scheduler code, I found that we can either set
> scheduler
> > > policy to be either Round Robin or FIFO. Are there other schedulers
> > > available which use different policy.
> > >
> >
> > What scheduler are you looking at?
> >
> >
> >
> > > 5. I read about Shared and Non-shared Cache. How does L2 gets shared.
> Its on
> > > the die. So does that mean that you have to buy a shared cache board ?
> > >
> >
> > L2 isn't shared on Intel platforms. Each processor has its own L2 cache
> > on chip.  Otherwise, it is shared just like memory is shared, with a
> > common bus connecting it to more than one processor.
> >
> >
> >
> >
> > > I apologize if some of the questions are stupid and outside the scope of
> > > this forum.
> > >
> > > Moreover, pointers to web-resources would be helpful.
> > >
> > > Thanks,
> > > Shreyas
> > >
> > >
> > >
> > >
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> >
> > --
> > Robert Hyatt                    Computer and Information Sciences
> > hyatt@cis.uab.edu               University of Alabama at Birmingham
> > (205) 934-2213                  115A Campbell Hall, UAB Station
> > (205) 934-5473 FAX              Birmingham, AL 35294-1170
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Re: SMP questions

[Eckhard Rüggeberg]

Robert M. Hyatt schrieb:
> > 2. How can we control the tasks, threads to run on a particular
> > processor ?
>
> Why would you want to?  This is something the operating system should
> handle correctly.

Unfortunately, it seems not to in all cases. I have for instance a 
severe performance problem with an application within Win4Lin. One 
instance of it running on a dual CPU server is SLOWER than on the same 
machine, same kernel with only one CPU...
-- 
Eckhard Rüggeberg
E.Rueggeberg@t-online.de

 "One OS to rule them all, One OS to find them,
  One OS to bring them all and in the darkness bind them
  In the Land of Redmond, where the Shadows lie."

Re: SMP questions

[Earle R. Nietzel]

On Thu, 2003-06-19 at 18:55, Shreyas Kejariwal wrote:
> Thanks Robert.
> 
> Getting back to Q3, as you mentioned, it uses shared bus. So that makes it
> hardware specific.  However, I read that Posix threads can achieve the same
> w/o sharing the bus. Is that correct?
> 
> Can anyone point which dual/SMP processors share memory bus ?
> 

There are some systems that have TLC that are shared between a group fo
processors. This how the Unisys ES7000 system works. It has a 32MB TLC
shared between each group of 4 processors. This allows SMP to scale
better.

Unisys has it where you can assign a group (4 processors) to a
particular process which will make sure that a hit occurs in the TLC and
improves performance for that particular process.

Another tweak they allow is to assign I/O devices to a group so that
I/O's are occuring in the same processor group as well.

> 
> Shreyas
> 
> 
> 
> --
> 
> -----------------------------------------------
> Shreyas Kejariwal
> Perigee LLC
> Tel: 315-453 7842 x 42
> 
> 
> 
> "Robert M. Hyatt" <hyatt@cis.uab.edu> wrote in message
> news:Pine.LNX.4.44.0306191057030.5678-100000@crafty.cis.uab.edu...
> > On Wed, 18 Jun 2003, Shreyas Kejariwal wrote:
> >
> > > Hi everyone,
> > >
> > > I am new to this arena and I have few questions so please help me answer
> > > them.
> > >
> > > 1. Looks like most of the new systems employ SMPs. But do we still have
> > > ASMPs systems ?
> >
> >
> > If you mean "asymmetric multiprocessing" the answer is generally "no".
> > That has pretty much disappeared.  If you mean something like clusters
> > and so forth, then they are going strong but require different
> > programming approaches.
> >
> >
> > >
> > > 2. How can we control the tasks, threads to run on a particular
> processor ?
> >
> > Why would you want to?  This is something the operating system should
> > handle correctly.
> >
> >
> > > Is there a way we can specify ? Take the scenario where there are two
> > > multi-threaded applications. I want all the threads of the App1 to run
> on
> > > Processor, P1 and the threads of the App2 on Processor, P2. Not like
> some
> > > threads of App1 would run on P2 depending on the scheduler.
> > >
> >
> >
> > Most systems support some sort of "processor affinity" stuff, but it is
> > a bad idea in general.  Leave the O/S stuff to the O/S.  It _really_
> > does know better...
> >
> >
> >
> > > 3. How can processors access each other's memory / addr space?
> > >
> >
> > Both processors are connected to the memory sub-system with a common
> > bus.
> >
> >
> >
> > > 4. Digging into the scheduler code, I found that we can either set
> scheduler
> > > policy to be either Round Robin or FIFO. Are there other schedulers
> > > available which use different policy.
> > >
> >
> > What scheduler are you looking at?
> >
> >
> >
> > > 5. I read about Shared and Non-shared Cache. How does L2 gets shared.
> Its on
> > > the die. So does that mean that you have to buy a shared cache board ?
> > >
> >
> > L2 isn't shared on Intel platforms. Each processor has its own L2 cache
> > on chip.  Otherwise, it is shared just like memory is shared, with a
> > common bus connecting it to more than one processor.
> >
> >
> >
> >
> > > I apologize if some of the questions are stupid and outside the scope of
> > > this forum.
> > >
> > > Moreover, pointers to web-resources would be helpful.
> > >
> > > Thanks,
> > > Shreyas
> > >
> > >
> > >
> > >
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> >
> > --
> > Robert Hyatt                    Computer and Information Sciences
> > hyatt@cis.uab.edu               University of Alabama at Birmingham
> > (205) 934-2213                  115A Campbell Hall, UAB Station
> > (205) 934-5473 FAX              Birmingham, AL 35294-1170
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Earle R. Nietzel <nietzel@rhinobox.org>

Re: SMP questions

[Robert M. Hyatt]

On Thu, 19 Jun 2003, Eckhard Rüggeberg wrote:

> Robert M. Hyatt schrieb:
> > > 2. How can we control the tasks, threads to run on a particular
> > > processor ?
> >
> > Why would you want to?  This is something the operating system should
> > handle correctly.
> 
> Unfortunately, it seems not to in all cases. I have for instance a 
> severe performance problem with an application within Win4Lin. One 
> instance of it running on a dual CPU server is SLOWER than on the same 
> machine, same kernel with only one CPU...
> 

That isn't necessarily a cpu affinity problem.  I've seen plenty of
parallel algorithms that occasionally run slower.  I have a chess 
program that can do this in rare cases as well...

I think trying to hamstring the O/S into holding a process (thread) 
on a specific CPU is going over the "portability line" a good ways,
in addition to introducing other problems...





-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Re: SMP questions

[Earle R. Nietzel]

> Why would you want to?  This is something the operating system should 
> handle correctly.
Agreed that the OS should be able to handle it the best it can. 

Though somtimes you have processes that are more important than others and you want to maintain a level of performance for that (application) process.

An example might be if other (applications) processes hog the cpu's, therby lowering the level of performace of the more important process.

This might be unacceptable which is why processor affinity is required.

These utilities are very valuable on scale up boxes versus scale out where problems like this are frequent.

-- 
Earle R. Nietzel <nietzel@rhinobox.org>

Re: SMP questions

[Robert M. Hyatt]

On 19 Jun 2003, Earle R. Nietzel wrote:

> 
> > Why would you want to?  This is something the operating system should 
> > handle correctly.
> Agreed that the OS should be able to handle it the best it can. 
> 
> Though somtimes you have processes that are more important than others and you want to maintain a level of performance for that (application) process.
> 
> An example might be if other (applications) processes hog the cpu's, therby lowering the level of performace of the more important process.

Wouldn't changing the priority be a better solution?  Or else "nice" 
down those compute-hogs so they don't slow down the "important"
applications?



> 
> This might be unacceptable which is why processor affinity is required.
> 
> These utilities are very valuable on scale up boxes versus scale out where problems like this are frequent.
> 
> 

-- 
Robert Hyatt                    Computer and Information Sciences
hyatt@cis.uab.edu               University of Alabama at Birmingham
(205) 934-2213                  115A Campbell Hall, UAB Station 
(205) 934-5473 FAX              Birmingham, AL 35294-1170

Re: SMP questions

[Earle R. Nietzel]

On Fri, 2003-06-20 at 02:45, Robert M. Hyatt wrote:
> On 19 Jun 2003, Earle R. Nietzel wrote:
> 
> > 
> > > Why would you want to?  This is something the operating system should 
> > > handle correctly.
> > Agreed that the OS should be able to handle it the best it can. 
> > 
> > Though somtimes you have processes that are more important than others and you want to maintain a level of performance for that (application) process.
> > 
> > An example might be if other (applications) processes hog the cpu's, therby lowering the level of performace of the more important process.
> 
> Wouldn't changing the priority be a better solution?  Or else "nice" 
> down those compute-hogs so they don't slow down the "important"
> applications?
> 

I think were getting to a case by case basis. In one situation yes maybe
prioritizing the processes would be enough but the in others the only
sure way is using affinity.

Since it is clear that the future is going back to a scale up
environment these utilities will become more important.


> 
> 
> > 
> > This might be unacceptable which is why processor affinity is required.
> > 
> > These utilities are very valuable on scale up boxes versus scale out where problems like this are frequent.
> > 
> >