NPTL: stack limit limiting number of threads

NPTL: stack limit limiting number of threads

[Ritesh Kumar]

Hi everybody,
     I recently had a chance to benchmark Linux/NPTL vs FreeBSD to
compare thread creation performance on both the systems. Let me not
disclose the results as I am sure my methodology is not the best one
;-). However, I did notice something queer which I would like to
discuss.
    I noticed that on my system (which has a 3G:1G split linux-2.6.1
NPTL gentoo), with the (possibly default) stack size limit of 8M, I
couldn't create more than around 380 threads in a process. On
searching the web, I found that a reason for this could be that in 3G
of userspace, there is space for exactly 384 8M chunks of memory.
Thus, because it is not possible to create more than 384 thread stacks
in the user space, the NPTL disallows creation of threads greater than
around 380. When I reduced the stack limit (using ulimit in bash), I
could create enough number of threads for my benchmarks to run
properly.
However, I was most amazed to see that the limit on stack size on
FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is
FreeBSD able to create about a 1000 threads with that kind of a stack
limit. Also, is there anything specific in its implementation which
makes it difficult to incorporate in Linux? Wouldn't it be a good idea
to remove this "trade-off" between stack limit and number of threads
and fail thread creation only when we have run out of address space
being *actually used* in the stacks in a process.

Ritesh
-- 
http://www.cs.unc.edu/~ritesh/

Re: NPTL: stack limit limiting number of threads

[Jakub Jelinek]

On Fri, May 13, 2005 at 03:49:48PM -0400, Ritesh Kumar wrote:
> However, I was most amazed to see that the limit on stack size on
> FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is
> FreeBSD able to create about a 1000 threads with that kind of a stack
> limit. Also, is there anything specific in its implementation which
> makes it difficult to incorporate in Linux? Wouldn't it be a good idea
> to remove this "trade-off" between stack limit and number of threads
> and fail thread creation only when we have run out of address space
> being *actually used* in the stacks in a process.

On FreeBSD the default thread stack size is not computed from ulimit -s,
but is constant.  They apparently only recently increased it to 1MB
(resp. 2MB on 64-bit arches), from 64K.

On Linux, the default thread stack size (except with fixed stack LinuxThreads)
is determined from ulimit -s (with a constant default if ulimit -s is
unlimited).

If your threaded application has specific needs for stack sizes, it can
always pthread_attr_setstacksize to whatever you find appropriate.

The thread library needs to know the stack size limit before creating
the thread, that can't be changed dynamically.

	Jakub

Re: NPTL: stack limit limiting number of threads

[Ritesh Kumar]

On 5/13/05, Jakub Jelinek <jakub@redhat.com> wrote:
> On Fri, May 13, 2005 at 03:49:48PM -0400, Ritesh Kumar wrote:
> > However, I was most amazed to see that the limit on stack size on
> > FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is
> > FreeBSD able to create about a 1000 threads with that kind of a stack
> > limit. Also, is there anything specific in its implementation which
> > makes it difficult to incorporate in Linux? Wouldn't it be a good idea
> > to remove this "trade-off" between stack limit and number of threads
> > and fail thread creation only when we have run out of address space
> > being *actually used* in the stacks in a process.
> 
> On FreeBSD the default thread stack size is not computed from ulimit -s,
> but is constant.  They apparently only recently increased it to 1MB
> (resp. 2MB on 64-bit arches), from 64K.
> 
> On Linux, the default thread stack size (except with fixed stack LinuxThreads)
> is determined from ulimit -s (with a constant default if ulimit -s is
> unlimited).
> 
> If your threaded application has specific needs for stack sizes, it can
> always pthread_attr_setstacksize to whatever you find appropriate.
> 
> The thread library needs to know the stack size limit before creating
> the thread, that can't be changed dynamically.
> 
>         Jakub
> 

Hi Jakub,
    Thanks for your reply. I actually went ahead after getting your
mail and coded up a small program to check the stack limit
deliberately. The program is shown inline.

#include <stdio.h>

#define BUF_SIZE 1024000

void recurse(int n){
	char ch[BUF_SIZE];
	if(n<=0)
		return;
	else
		recurse(n-1);
}

int main(argc, argv)
	int argc;
	char **argv;
{
	if(argc!=2){
		printf("Usage: %s <n (megabytes)>\n", argv[0]);
		return 1;
	}
	printf("Checking for %dMB\n", atoi(argv[1]));
	recurse(atoi(argv[1]));
}

Its a fairly crude way to find out the actual stack limit. Basically,
the resurse function recurses each time allocating ~1MB of space on
the stack. The program segfaults exactly at the ulimit -s value of
stack size on both linux and freebsd. So it does seem that the ulimit
-s is the value of stack limit used on FreeBSD.


Ritesh
-- 
http://www.cs.unc.edu/~ritesh/

Re: NPTL: stack limit limiting number of threads

[Jakub Jelinek]

On Fri, May 13, 2005 at 08:02:48PM -0400, Ritesh Kumar wrote:
>     Thanks for your reply. I actually went ahead after getting your
> mail and coded up a small program to check the stack limit
> deliberately. The program is shown inline.
> 
> #include <stdio.h>
> 
> #define BUF_SIZE 1024000
> 
> void recurse(int n){
> 	char ch[BUF_SIZE];
> 	if(n<=0)
> 		return;
> 	else
> 		recurse(n-1);
> }
> 
> int main(argc, argv)
> 	int argc;
> 	char **argv;
> {
> 	if(argc!=2){
> 		printf("Usage: %s <n (megabytes)>\n", argv[0]);
> 		return 1;
> 	}
> 	printf("Checking for %dMB\n", atoi(argv[1]));
> 	recurse(atoi(argv[1]));
> }
> 
> Its a fairly crude way to find out the actual stack limit. Basically,
> the resurse function recurses each time allocating ~1MB of space on
> the stack. The program segfaults exactly at the ulimit -s value of
> stack size on both linux and freebsd. So it does seem that the ulimit
> -s is the value of stack limit used on FreeBSD.

For the main stack sure.  But now try to call that recurse in
some other thread.

	Jakub

Re: NPTL: stack limit limiting number of threads

[Ritesh Kumar]

On 5/13/05, Jakub Jelinek <jakub@redhat.com> wrote:
> On Fri, May 13, 2005 at 08:02:48PM -0400, Ritesh Kumar wrote:
> >     Thanks for your reply. I actually went ahead after getting your
> > mail and coded up a small program to check the stack limit
> > deliberately. The program is shown inline.
> >
> > #include <stdio.h>
> >
> > #define BUF_SIZE 1024000
> >
> > void recurse(int n){
> >       char ch[BUF_SIZE];
> >       if(n<=0)
> >               return;
> >       else
> >               recurse(n-1);
> > }
> >
> > int main(argc, argv)
> >       int argc;
> >       char **argv;
> > {
> >       if(argc!=2){
> >               printf("Usage: %s <n (megabytes)>\n", argv[0]);
> >               return 1;
> >       }
> >       printf("Checking for %dMB\n", atoi(argv[1]));
> >       recurse(atoi(argv[1]));
> > }
> >
> > Its a fairly crude way to find out the actual stack limit. Basically,
> > the resurse function recurses each time allocating ~1MB of space on
> > the stack. The program segfaults exactly at the ulimit -s value of
> > stack size on both linux and freebsd. So it does seem that the ulimit
> > -s is the value of stack limit used on FreeBSD.
> 
> For the main stack sure.  But now try to call that recurse in
> some other thread.
> 
>         Jakub
> 

Oh great... I see your point now. I called the same recurse in an
alternative thread and found the limit to be 8M on Linux ( and Mac OS
X (Panther)) but around 1MB on FreeBSD. Thanks for the clarification!

Ritesh 

-- 
http://www.cs.unc.edu/~ritesh/