Maximum number of processes in Linux

Maximum number of processes in Linux

[Irfan Habib]

Hi,

What is the maximum number of process which can run simultaneously in
linux? I need to create an application which requires 40,000 threads.
I was testing with far fewer numbers than that, I was getting
exceptions in pthread_create

Regards
Irfan

Re: Maximum number of processes in Linux

[Stephen Hemminger]

On Tue, 15 Aug 2006 22:59:37 +0500
"Irfan Habib" <irfan.habib@gmail.com> wrote:

> Hi,
> 
> What is the maximum number of process which can run simultaneously in
> linux? I need to create an application which requires 40,000 threads.
> I was testing with far fewer numbers than that, I was getting
> exceptions in pthread_create

What kernel version, and glibc version?

Re: Maximum number of processes in Linux

[linux-os (Dick Johnson)]

On Tue, 15 Aug 2006, Irfan Habib wrote:

> Hi,
>
> What is the maximum number of process which can run simultaneously in
> linux? I need to create an application which requires 40,000 threads.
> I was testing with far fewer numbers than that, I was getting
> exceptions in pthread_create
>
> Regards
> Irfan

#include <stdio.h>
int main(){
     unsigned long i;
     while(fork() != -1)
         i++;
     printf("%u\n", i);
     return 0;
}
$ gcc -o xxx xxx.c
$ ./xxx

1251392833         <<---- At least this number
1251392834
1251392834
1251392834
1251392834
1251392833
1251392833
1251392834
1251392834
1251392834
^C
$ killall xxx

BYW 40,000 threads? 40,000 tasks all sharing the same address space?
Hopefully this is just a training exercise to see if it's possible.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Maximum number of processes in Linux

[Willy Tarreau]

On Tue, Aug 15, 2006 at 02:22:02PM -0400, linux-os (Dick Johnson) wrote:
> 
> On Tue, 15 Aug 2006, Irfan Habib wrote:
> 
> > Hi,
> >
> > What is the maximum number of process which can run simultaneously in
> > linux? I need to create an application which requires 40,000 threads.
> > I was testing with far fewer numbers than that, I was getting
> > exceptions in pthread_create
> >
> > Regards
> > Irfan
> 
> #include <stdio.h>
> int main(){
>      unsigned long i;
       ^^^^^^^^^^^^^^^^

>      while(fork() != -1)
>          i++;
>      printf("%u\n", i);
>      return 0;
> }
> $ gcc -o xxx xxx.c
> $ ./xxx
> 
> 1251392833         <<---- At least this number

Dick, would you please initialize your local variables when you send
examples like this ? You should have been amazed by one billion processes
on your box, at least.

> 1251392834
> 1251392834
> 1251392834
> 1251392834
> 1251392833
> 1251392833
> 1251392834
> 1251392834
> 1251392834
> ^C
> $ killall xxx
> 
> BYW 40,000 threads? 40,000 tasks all sharing the same address space?
> Hopefully this is just a training exercise to see if it's possible.
> 
> Cheers,
> Dick Johnson
> Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
> New book: http://www.AbominableFirebug.com/

Regards,
Willy

Re: Maximum number of processes in Linux

[linux-os (Dick Johnson)]

On Tue, 15 Aug 2006, Willy Tarreau wrote:

> On Tue, Aug 15, 2006 at 02:22:02PM -0400, linux-os (Dick Johnson) wrote:
>>
>> On Tue, 15 Aug 2006, Irfan Habib wrote:
>>
>>> Hi,
>>>
>>> What is the maximum number of process which can run simultaneously in
>>> linux? I need to create an application which requires 40,000 threads.
>>> I was testing with far fewer numbers than that, I was getting
>>> exceptions in pthread_create
>>>
>>> Regards
>>> Irfan
>>
>> #include <stdio.h>
>> int main(){
>>      unsigned long i;
>       ^^^^^^^^^^^^^^^^
>
>>      while(fork() != -1)
>>          i++;
>>      printf("%u\n", i);
>>      return 0;
>> }
>> $ gcc -o xxx xxx.c
>> $ ./xxx
>>
>> 1251392833         <<---- At least this number
>
> Dick, would you please initialize your local variables when you send
> examples like this ? You should have been amazed by one billion processes
> on your box, at least.

Yep. I discovered it just as I hit the ^X button!


>
>> 1251392834
>> 1251392834
>> 1251392834
>> 1251392834
>> 1251392833
>> 1251392833
>> 1251392834
>> 1251392834
>> 1251392834
>> ^C
>> $ killall xxx
>>
>> BYW 40,000 threads? 40,000 tasks all sharing the same address space?
>> Hopefully this is just a training exercise to see if it's possible.
>>
>> Cheers,
>> Dick Johnson
>> Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
>> New book: http://www.AbominableFirebug.com/
>
> Regards,
> Willy
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Maximum number of processes in Linux

[linux-os (Dick Johnson)]

On Tue, 15 Aug 2006, Willy Tarreau wrote:

> On Tue, Aug 15, 2006 at 02:22:02PM -0400, linux-os (Dick Johnson) wrote:
>>
>> On Tue, 15 Aug 2006, Irfan Habib wrote:
>>
>>> Hi,
>>>
>>> What is the maximum number of process which can run simultaneously in
>>> linux? I need to create an application which requires 40,000 threads.
>>> I was testing with far fewer numbers than that, I was getting
>>> exceptions in pthread_create
>>>
>>> Regards
>>> Irfan

[SNIPPED bad stuff]

>
> Dick, would you please initialize your local variables when you send
> examples like this ? You should have been amazed by one billion processes
> on your box, at least.
>


Yep....

#include <stdio.h>
#include <signal.h>
int main()
{
     unsigned long i;
     for(i = 0; ; i++)
     {
         switch(fork())
         {
         case 0:		// kid
 	pause();
         break;
         case -1:	// Failed
         printf("%lu\n", i);
             kill(0, SIGTERM);
             exit(0);
         default:
             break;
         }
     }
     return 0;
}

Shows a consistent 6140.

>
> Regards,
> Willy
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Maximum number of processes in Linux

[Willy Tarreau]

On Tue, Aug 15, 2006 at 03:13:35PM -0400, linux-os (Dick Johnson) wrote:
> 
> On Tue, 15 Aug 2006, Willy Tarreau wrote:
> 
> > On Tue, Aug 15, 2006 at 02:22:02PM -0400, linux-os (Dick Johnson) wrote:
> >>
> >> On Tue, 15 Aug 2006, Irfan Habib wrote:
> >>
> >>> Hi,
> >>>
> >>> What is the maximum number of process which can run simultaneously in
> >>> linux? I need to create an application which requires 40,000 threads.
> >>> I was testing with far fewer numbers than that, I was getting
> >>> exceptions in pthread_create
> >>>
> >>> Regards
> >>> Irfan
> 
> [SNIPPED bad stuff]
> 
> >
> > Dick, would you please initialize your local variables when you send
> > examples like this ? You should have been amazed by one billion processes
> > on your box, at least.
> >
> 
> 
> Yep....
> 
> #include <stdio.h>
> #include <signal.h>
> int main()
> {
>      unsigned long i;
>      for(i = 0; ; i++)
>      {
>          switch(fork())
>          {
>          case 0:		// kid
>  	pause();
>          break;
>          case -1:	// Failed
>          printf("%lu\n", i);
>              kill(0, SIGTERM);
>              exit(0);
>          default:
>              break;
>          }
>      }
>      return 0;
> }
> 
> Shows a consistent 6140.

Better ! :-)

1) how much memory do you have ?
2) Would you try with clone() instead of fork(), you should get more because
   everything will be shared.

Regards,
Willy

Re: Maximum number of processes in Linux

[Arjan van de Ven]

> Shows a consistent 6140.
> 

the default limit in proc scales with memory (to avoid really bad
stuff), you can oversize it to 2^16 if you want. 

Going over 2^16 is not too good an idea (16 bit counters overflow),
especially if you have hostile users (read: students) on the machine,
since this is the kind of scenario you can trigger on purpose.

Re: Maximum number of processes in Linux

[Irfan Habib]

I have 1GB RAM and a p4 HT 3 GHz.


On 8/16/06, Arjan van de Ven <arjan@infradead.org> wrote:
>
> > Shows a consistent 6140.
> >
>
> the default limit in proc scales with memory (to avoid really bad
> stuff), you can oversize it to 2^16 if you want.
>
> Going over 2^16 is not too good an idea (16 bit counters overflow),
> especially if you have hostile users (read: students) on the machine,
> since this is the kind of scenario you can trigger on purpose.
>
>
>

Re: Maximum number of processes in Linux

[Avi Kivity]

Irfan Habib wrote:
>
> I have 1GB RAM and a p4 HT 3 GHz.
>
>
Use pthread_attr_setstacksize() to set a small stack size, and you'll 
get many more threads running.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Maximum number of processes in Linux

[Helge Hafting]

linux-os (Dick Johnson) wrote:
> Yep....
>
> #include <stdio.h>
> #include <signal.h>
> int main()
> {
>      unsigned long i;
>      for(i = 0; ; i++)
>      {
>          switch(fork())
>          {
>          case 0:		// kid
>  	pause();
>          break;
>          case -1:	// Failed
>          printf("%lu\n", i);
>              kill(0, SIGTERM);
>              exit(0);
>          default:
>              break;
>          }
>      }
>      return 0;
> }
>
> Shows a consistent 6140.
>   
Doesn't work here.  Without ulimit, I wasn't surprised
about the resulting OOM mess. 

Problem was, it never stopped.  I expected OOM to kill
this program, and quite possibly lots of other running programs
as well.  What I got, was ever-rolling OOM messages
with stack traces inbetween. 
2.6.18-rc4-mm1 never recovered and had to be killed by sysrq.

Helge Hafting

Re: Maximum number of processes in Linux

[linux-os (Dick Johnson)]

On Wed, 16 Aug 2006, Helge Hafting wrote:

> linux-os (Dick Johnson) wrote:
>> Yep....
>>
>> #include <stdio.h>
>> #include <signal.h>
>> int main()
>> {
>>      unsigned long i;
>>      for(i = 0; ; i++)
>>      {
>>          switch(fork())
>>          {
>>          case 0:		// kid
>>  	pause();
>>          break;
>>          case -1:	// Failed
>>          printf("%lu\n", i);
>>              kill(0, SIGTERM);
>>              exit(0);
>>          default:
>>              break;
>>          }
>>      }
>>      return 0;
>> }
>>
>> Shows a consistent 6140.
>>
> Doesn't work here.  Without ulimit, I wasn't surprised
> about the resulting OOM mess.
>
> Problem was, it never stopped.  I expected OOM to kill
> this program, and quite possibly lots of other running programs
> as well.  What I got, was ever-rolling OOM messages
> with stack traces inbetween.
> 2.6.18-rc4-mm1 never recovered and had to be killed by sysrq.
>
> Helge Hafting

Script started on Wed 16 Aug 2006 07:18:48 AM EDT
LINUX> gcc -o xxx xxx.c
LINUX> ./xxx
6138
Terminated
LINUX> ./xxx
6138
Terminated
LINUX> ./xxx
6138
Terminated
LINUX> ulimit
unlimited
LINUX> uname -r
2.6.16.24
LINUX> cat /proc/meminfo
MemTotal:       774572 kB
MemFree:        381984 kB
Buffers:        154120 kB
Cached:          31000 kB
SwapCached:          0 kB
Active:          74908 kB
Inactive:       130816 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       774572 kB
LowFree:        381984 kB
SwapTotal:      907664 kB
SwapFree:       907660 kB
Dirty:              28 kB
Writeback:           0 kB
Mapped:          34100 kB
Slab:           177948 kB
CommitLimit:   1294948 kB
Committed_AS:    31080 kB
PageTables:        804 kB
VmallocTotal:   515796 kB
VmallocUsed:      3524 kB
VmallocChunk:   511568 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB
LINUX> exit
Script done on Wed 16 Aug 2006 07:19:48 AM EDT

Runs fine here. I'm using 2.6.14.24. Maybe your kernel version still
has an OEM bug???

Since the forked process never touches any of its memory, it
shouldn't use anything except space in the kernel for a new task-
structure and space in user-space for stack. COW wouldn't have
happened yet. I don't see how you get out of memory before you
run out of PIDs!

The first instance of the fork failing should cause a signal
to be sent to all the children, killing them:

         case -1:	// Failed
         printf("%lu\n", i);
             kill(0, SIGTERM);
             exit(0);


I can set /proc/sys/vm/overcommit_memory to either 1 or 0 with
the same effect, no out-of-memory errors. Maybe your kernel
version has a bug?



Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Maximum number of processes in Linux

[Mike Galbraith]

On Wed, 2006-08-16 at 07:33 -0400, linux-os (Dick Johnson) wrote:
> On Wed, 16 Aug 2006, Helge Hafting wrote:
> > Doesn't work here.  Without ulimit, I wasn't surprised
> > about the resulting OOM mess.
> >
> > Problem was, it never stopped.  I expected OOM to kill
> > this program, and quite possibly lots of other running programs
> > as well.  What I got, was ever-rolling OOM messages
> > with stack traces inbetween.
> > 2.6.18-rc4-mm1 never recovered and had to be killed by sysrq.
> >
> > Helge Hafting
> 

> Runs fine here. I'm using 2.6.14.24. Maybe your kernel version still
> has an OEM bug???

Hmm.  For grins, I ran it as root in 2.6.18-rc4-mm1, and didn't even get
an oom.  I did a SysRq-E, ran it again, and watched it fork off 18600
kids, then hang the box again.

SysRq : Terminate All Tasks
SysRq : Changing Loglevel
Loglevel set to 9

(start forkbomb... generates ~18600 kids, I thumb twiddle then...)

SysRq : Show Memory
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:56
cpu 0 cold: high 62, batch 15 used:59
cpu 1 hot: high 186, batch 31 used:61
cpu 1 cold: high 62, batch 15 used:49
HighMem per-cpu:
cpu 0 hot: high 42, batch 7 used:6
cpu 0 cold: high 14, batch 3 used:13
cpu 1 hot: high 42, batch 7 used:1
cpu 1 cold: high 14, batch 3 used:11
Active:130198 inactive:1205 dirty:0 writeback:0 unstable:0 free:3009 slab:64587 mapped:518 pagetables:55775
DMA free:4096kB min:68kB low:84kB high:100kB active:3652kB inactive:4656kB present:16384kB pages_scanned:27 all_unreclaimable? no
lowmem_reserve[]: 0 880 1007
Normal free:7836kB min:3756kB low:4692kB high:5632kB active:426716kB inactive:156kB present:901120kB pages_scanned:44713 all_unreclaimable? no
lowmem_reserve[]: 0 0 1023
HighMem free:104kB min:128kB low:264kB high:400kB active:90424kB inactive:8kB present:131008kB pages_scanned:46493 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 4096kB
Normal: 1*4kB 1*8kB 1*16kB 2*32kB 1*64kB 0*128kB 4*256kB 5*512kB 0*1024kB 0*2048kB 1*4096kB = 7836kB
HighMem: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 104kB
Swap cache: add 280, delete 69, find 0/0, race 0+0
Free swap  = 1027032kB
Total swap = 1028152kB
Free swap:       1027032kB
262128 pages of RAM
32752 pages of HIGHMEM
5276 reserved pages
521412 pages shared
211 pages swap cached
0 pages dirty
0 pages writeback
518 pages mapped
64587 pages slab
55775 pages pagetables
SysRq : Terminate All Tasks

> Since the forked process never touches any of its memory, it
> shouldn't use anything except space in the kernel for a new task-
> structure and space in user-space for stack. COW wouldn't have
> happened yet. I don't see how you get out of memory before you
> run out of PIDs!
> 
> The first instance of the fork failing should cause a signal
> to be sent to all the children, killing them:
> 
>          case -1:	// Failed
>          printf("%lu\n", i);
>              kill(0, SIGTERM);
>              exit(0);

No oom-kill action here.

	-Mike

Re: Maximum number of processes in Linux

[Alistair John Strachan]

On Wednesday 16 August 2006 11:01, Helge Hafting wrote:
> linux-os (Dick Johnson) wrote:
> > Yep....
> >
> > #include <stdio.h>
> > #include <signal.h>
> > int main()
> > {
> >      unsigned long i;
> >      for(i = 0; ; i++)
> >      {
> >          switch(fork())
> >          {
> >          case 0:		// kid
> >  	pause();
> >          break;
> >          case -1:	// Failed
> >          printf("%lu\n", i);
> >              kill(0, SIGTERM);
> >              exit(0);
> >          default:
> >              break;
> >          }
> >      }
> >      return 0;
> > }
> >
> > Shows a consistent 6140.
>
> Doesn't work here.  Without ulimit, I wasn't surprised
> about the resulting OOM mess.
>
> Problem was, it never stopped.  I expected OOM to kill
> this program, and quite possibly lots of other running programs
> as well.  What I got, was ever-rolling OOM messages
> with stack traces inbetween.
> 2.6.18-rc4-mm1 never recovered and had to be killed by sysrq.

It took 4.5 minutes to recover on my X2 3800+, 2GB RAM, 512MB swap, when I 
tried without ulimit on 2.6.18-rc4. However, the OOM killer did call all of 
the offending processes and I was able to use the machine for many hours 
afterwards. The VM didn't even mind after a swapoff -a.

Maybe an -mm patch?

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Maximum number of processes in Linux

[linux-os (Dick Johnson)]

On Tue, 15 Aug 2006, linux-os (Dick Johnson) wrote:

>
> On Tue, 15 Aug 2006, Irfan Habib wrote:
>
>> Hi,
>>
>> What is the maximum number of process which can run simultaneously in
>> linux? I need to create an application which requires 40,000 threads.
>> I was testing with far fewer numbers than that, I was getting
>> exceptions in pthread_create
>>
>> Regards
>> Irfan
>
> #include <stdio.h>
> int main(){
>     unsigned long i;
>     while(fork() != -1)
>         i++;
>     printf("%u\n", i);
>     return 0;
> }
> $ gcc -o xxx xxx.c
> $ ./xxx
>
> 1251392833         <<---- At least this number
> 1251392834
> 1251392834
> 1251392834
> 1251392834
> 1251392833
> 1251392833
> 1251392834
> 1251392834
> 1251392834
> ^C
> $ killall xxx
>
> BYW 40,000 threads? 40,000 tasks all sharing the same address space?
> Hopefully this is just a training exercise to see if it's possible.
>
> Cheers,
> Dick Johnson
> Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
> New book: http://www.AbominableFirebug.com/
> _
> \x1a\x04


I blew it here...
     unsigned long i = 0;
...required.



Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Maximum number of processes in Linux

[Alan Cox]

Ar Maw, 2006-08-15 am 22:59 +0500, ysgrifennodd Irfan Habib:
> What is the maximum number of process which can run simultaneously in
> linux? I need to create an application which requires 40,000 threads.
> I was testing with far fewer numbers than that, I was getting
> exceptions in pthread_create

On the usual default configuration far less. If you have lots of memory
and adjust the pid limits you can create 40,000 threads. Its not a very
good idea unless you are working on a system with several thousand
processors and usually means your program design is wrong, but you can
do it.

Alan

Re: Maximum number of processes in Linux

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

On Tue, 15 Aug 2006 22:59:37 +0500, Irfan Habib said:
> Hi,
> 
> What is the maximum number of process which can run simultaneously in
> linux? I need to create an application which requires 40,000 threads.
> I was testing with far fewer numbers than that, I was getting
> exceptions in pthread_create

There's some a<<FOO funkiness in the /proc file system that will explode
on 32-bit machines if the process ID goes over 128K.  Of course, with 40K
threads, you're probably either on a 64-bit NUMA box or doing things in an
incredibly ugly way....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Maximum number of processes in Linux

[Robert Hancock]

Irfan Habib wrote:
> Hi,
> 
> What is the maximum number of process which can run simultaneously in
> linux? I need to create an application which requires 40,000 threads.
> I was testing with far fewer numbers than that, I was getting
> exceptions in pthread_create

What architecture is this? On a 32-bit architecture with a 2MB stack 
size (which I think is the default) you couldn't possibly create more 
than 2048 threads just because of stack space requirements. Reducing the 
stack size would get you more.

I should also point out that any design that requires 40,000 threads is 
probably quite flawed unless you are running on a very large machine..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Maximum number of processes in Linux

[Michael Buesch]

On Wednesday 16 August 2006 01:12, Robert Hancock wrote:
> Irfan Habib wrote:
> > Hi,
> > 
> > What is the maximum number of process which can run simultaneously in
> > linux? I need to create an application which requires 40,000 threads.
> > I was testing with far fewer numbers than that, I was getting
> > exceptions in pthread_create
> 
> What architecture is this? On a 32-bit architecture with a 2MB stack 
> size (which I think is the default) you couldn't possibly create more 
> than 2048 threads just because of stack space requirements. Reducing the 
> stack size would get you more.

Hm, I'm on a 4way PPC64 machine with 2.5G RAM.
It can only create 509 pthreads and fails with ENOMEM
on the 510th.
That's not a really big machine, but I expected it to be able
to create somewhere around 8000 threads or so, at least. Especially
as it has a 64bit kernel and lots of memory.

Well...

That's my test app:

#include <stdio.h>
#include <pthread.h>
#include <string.h>
#include <errno.h>


static void * thread(void *arg)
{
        while (1)
                sleep(10);
}

int main(void)
{
        int err = 0;
        unsigned long i = 0;
        pthread_t t;

        while (!err) {
                err = pthread_create(&t, NULL, thread, NULL);
                i++;
                if (err) {
                        printf("Creating pthread %lu failed with \"%s\"\n",
                                i, strerror(errno));
                        break;
                }
                printf("%lu pthreads created\n", i);
        }

        return 0;
}

-- 
Greetings Michael.

Re: Maximum number of processes in Linux

[Lee Revell]

On Tue, 2006-08-15 at 17:12 -0600, Robert Hancock wrote:
> Irfan Habib wrote:
> > Hi,
> > 
> > What is the maximum number of process which can run simultaneously in
> > linux? I need to create an application which requires 40,000 threads.
> > I was testing with far fewer numbers than that, I was getting
> > exceptions in pthread_create
> 
> What architecture is this? On a 32-bit architecture with a 2MB stack 
> size (which I think is the default) you couldn't possibly create more 
> than 2048 threads just because of stack space requirements. Reducing the 
> stack size would get you more.
> 
> I should also point out that any design that requires 40,000 threads is 
> probably quite flawed unless you are running on a very large machine..
> 

Thread stack size defaults to whatever your distro sets RLIMIT_STACK to.
It's 8MB here.

Lee