Linux 2.6 context switching and posix threads performance question

Linux 2.6 context switching and posix threads performance question

[Mateusz Berezecki]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]


Hello List Readers,

I would really appreciate any comment on the overall performance of task
switching with 25 000 threads running on the Linux system. I was asked to work
on some software which spawns 25 000 threads and I am really worried if
it will ever work on 2 CPU HP Blade. The kernel was modified to support
bigger threads amount running (I have no idea how it was done, probably
just changing hardcoded limits) What is the performance impact of
so much threads on the overall system performance? Is there any ?
Wouldn't it be that such application would spend all of its time
switching contexts ? I'm asking for some kind of an authoritative answer
quite urgently. What is the optimum thread amount on 2 CPU SMP system
running Linux ?


Thank you very much in advance
Mateusz

-- 
  @..@   Mateusz Berezecki 
 (----)  mateuszb@gmail.com http://mateusz.agrest.org
( >__< ) PGP: 5F1C 86DF 89DB BFE9 899E 8CBE EB60 B7A7 43F9 5808 
^^ ~~ ^^

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux 2.6 context switching and posix threads performance question

[Peter Zijlstra]

On Sat, 2005-08-27 at 14:11 +0200, Mateusz Berezecki wrote:
> Hello List Readers,
> 
> I would really appreciate any comment on the overall performance of task
> switching with 25 000 threads running on the Linux system. I was asked to work
> on some software which spawns 25 000 threads and I am really worried if
> it will ever work on 2 CPU HP Blade. The kernel was modified to support
> bigger threads amount running (I have no idea how it was done, probably
> just changing hardcoded limits) What is the performance impact of
> so much threads on the overall system performance? Is there any ?
> Wouldn't it be that such application would spend all of its time
> switching contexts ? I'm asking for some kind of an authoritative answer
> quite urgently. What is the optimum thread amount on 2 CPU SMP system
> running Linux ?
> 
Well the obvious question is: what kernel version and which thread
library?

2.4 with LinuxThreads might have severe problems. However 2.6 with NPTL
should be able to handle it, IIRC Igno once did a million threads with
that combination just to show that it worked ;-).


-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: Linux 2.6 context switching and posix threads performance question

[Peter Zijlstra]

Group replies work best with mailing lists.

On Sat, 2005-08-27 at 14:30 +0200, Mateusz Berezecki wrote:
> On 8/27/05, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > Well the obvious question is: what kernel version and which thread
> > library?
> 
>   Ah, sorry I just forgot to put this info 
> 
> Linux blade1 2.6.12 #1 SMP Tue Jul 26 08:43:57 GMT 2005 i686 Intel(R)
> Xeon(TM) CPU 3.06GHz GenuineIntel GNU/Linux
> 
> the cpu is with HT so Linux thinks there are 4 cpus on board
>   
> > 
> > 2.4 with LinuxThreads might have severe problems. However 2.6 with NPTL
> > should be able to handle it, IIRC Igno once did a million threads with
_Ingo_ Molnar that is; I really should have gotten those 4 letters in
the right order by now, humble appologies.

> > that combination just to show that it worked ;-).
> 
>   Yes, but how about switching contexts and the performance impact?
> Will it slowdown the whole system or just result in low or medium
> overhead?.
> These threads are meant to be running an intensive network connection
> and possibly analysing the dataflow and maybe filtering some stuff.
> Each thread doing the same task.

I think it should work; however I'm not the most qualified to answer
this.



-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: Linux 2.6 context switching and posix threads performance question

[Arjan van de Ven]

>  I'm asking for some kind of an authoritative answer
> quite urgently. What is the optimum thread amount on 2 CPU SMP system
> running Linux ?

context switching in linux isn't THAT expensive compared to some other
operating systems, but it's not free either.
The optimum is obviously 2 threads, one for each cpu that processes your
network service in a state machine like way. This is why thttpd beats
apache by 10x if not more.

Re: Linux 2.6 context switching and posix threads performance question

[Con Kolivas]

On Sat, 27 Aug 2005 22:58, Arjan van de Ven wrote:
> >  I'm asking for some kind of an authoritative answer
> > quite urgently. What is the optimum thread amount on 2 CPU SMP system
> > running Linux ?
>
> context switching in linux isn't THAT expensive compared to some other
> operating systems, but it's not free either.
> The optimum is obviously 2 threads, one for each cpu that processes your
> network service in a state machine like way. This is why thttpd beats
> apache by 10x if not more.

On a current model processor (P4 3Ghz) the current 2.6 kernel can do about 
700,000 context switches per second with processes if they do nothing but 
switch, and perhaps slightly faster with threads. Each context switch, 
therefore, is quite cheap to perform. However you're unlikely to perform more 
than 10,000 context switches per second with real workloads and the switch 
itself contributes a measurable, but not performance limiting, impact. The 
more cpu bound your threads are the less context switches you'll perform. 
Fork is quite a bit more expensive. I don't have current figures on fork, but 
if you only fork once it shouldn't be a problem.

Cheers,
Con

Re: Linux 2.6 context switching and posix threads performance question

[Alan Cox]

On Sad, 2005-08-27 at 14:11 +0200, Mateusz Berezecki wrote:
> switching contexts ? I'm asking for some kind of an authoritative answer
> quite urgently. What is the optimum thread amount on 2 CPU SMP system
> running Linux ?

For doing what ? How often is a given thread woken up, do you need
latency or throughput.

A good rule of thumb is about 1.5 to 2 threads per core. But it is only
that and without context it is hard to assess the real needs. As you add
more threads you generally decrease cache effectiveness and that rather
than switching cost may well be the biggest hurt you experience. The
memory usage may also hurt.

Now if you have 25,000 threads and 24995 of them wake once every few
minutes that will have no real impact but if you are randomly flipping
between 25,000 threads all with different stacks and data areas at high
speed your cache utilisation will go down.

Equally if you have a lot of shared objects being written you avoid that
but can get into contention of cached memory. That however is more a
problem of number of processors than threads - ie a 25000 thread app
with a lot of shared objects may run *horribly* on a 64 CPU system and
really well on a dual.

So in essence you are asking "how long is a piece of string". Linux
2.6.x with NPTL will handle large numbers of threads. 25,000 is
excessive for most situations but how it behaves is more a question of
the application than the OS here.

Re: Linux 2.6 context switching and posix threads performance question

[James Courtier-Dutton]

Mateusz Berezecki wrote:
> Hello List Readers,
> 
> I would really appreciate any comment on the overall performance of task
> switching with 25 000 threads running on the Linux system. I was asked to work
> on some software which spawns 25 000 threads and I am really worried if
> it will ever work on 2 CPU HP Blade. The kernel was modified to support
> bigger threads amount running (I have no idea how it was done, probably
> just changing hardcoded limits) What is the performance impact of
> so much threads on the overall system performance? Is there any ?
> Wouldn't it be that such application would spend all of its time
> switching contexts ? I'm asking for some kind of an authoritative answer
> quite urgently. What is the optimum thread amount on 2 CPU SMP system
> running Linux ?
> 
> 
> Thank you very much in advance
> Mateusz
> 

You would get much better performance by switching the application to 
use a thread pool using a fixed low number ( about CPU*2 = 4) of worker 
threads.

James

Re: Linux 2.6 context switching and posix threads performance question

[Christoph Lameter]

On Sat, 27 Aug 2005, Mateusz Berezecki wrote:

> I would really appreciate any comment on the overall performance of task
> switching with 25 000 threads running on the Linux system. I was asked to work
> on some software which spawns 25 000 threads and I am really worried if
> it will ever work on 2 CPU HP Blade. The kernel was modified to support
> bigger threads amount running (I have no idea how it was done, probably
> just changing hardcoded limits) What is the performance impact of
> so much threads on the overall system performance? Is there any ?
> Wouldn't it be that such application would spend all of its time
> switching contexts ? I'm asking for some kind of an authoritative answer
> quite urgently. What is the optimum thread amount on 2 CPU SMP system
> running Linux ?

At 32k threads you run into the limit of waiters for RW semaphores. So 
make sure that the number stays lower than that or use a 64 bit platform 
that supports more threads.