NPTL: stack limit limiting number of threads

[Ritesh Kumar <digitalove@gmail.com>]

Hi everybody,
     I recently had a chance to benchmark Linux/NPTL vs FreeBSD to
compare thread creation performance on both the systems. Let me not
disclose the results as I am sure my methodology is not the best one
;-). However, I did notice something queer which I would like to
discuss.
    I noticed that on my system (which has a 3G:1G split linux-2.6.1
NPTL gentoo), with the (possibly default) stack size limit of 8M, I
couldn't create more than around 380 threads in a process. On
searching the web, I found that a reason for this could be that in 3G
of userspace, there is space for exactly 384 8M chunks of memory.
Thus, because it is not possible to create more than 384 thread stacks
in the user space, the NPTL disallows creation of threads greater than
around 380. When I reduced the stack limit (using ulimit in bash), I
could create enough number of threads for my benchmarks to run
properly.
However, I was most amazed to see that the limit on stack size on
FreeBSD (5.3 Release) was 64M by default! I was just wondering, how is
FreeBSD able to create about a 1000 threads with that kind of a stack
limit. Also, is there anything specific in its implementation which
makes it difficult to incorporate in Linux? Wouldn't it be a good idea
to remove this "trade-off" between stack limit and number of threads
and fail thread creation only when we have run out of address space
being *actually used* in the stacks in a process.

Ritesh
-- 
http://www.cs.unc.edu/~ritesh/