Re: Poll and Select not scaling

Re: Poll and Select not scaling

[Dan Kegel]

Micah wrote:
> I have been trying to increase the scalabilty of
> an email server that has been ported to Linux. It
> was originally written for Netware, and there we
> are able to provide over 30,000 connections at any
> given time. On Linux however select stops working
> after the first 1024 connections. I have changed
> include/linux/fs.h and updated NR_FILE to be 81920. 

On modern kernels, you no longer need to change NR_FILE;
all you need to do is increase ulimit -n and a few proc 
entries; see 
http://www.kegel.com/c10k.html#limits.filehandles

> In test applications I have been able to create well
> over 30,000 connections but I am unable to do either
> a select or a poll on them. 

select() is usually limited to 1024 file descriptors, so
poll() is a slightly better choice.  However, although
it can handle 30000 file descriptors, the performance sucks;
see http://www.kegel.com/dkftpbench/Poller_bench.html#results

You may wish to use an alternative to poll().
See http://www.kegel.com/c10k.html#strategies for a list of possibilities.

The only one that performs well on Linux at 30000 fds would be
the real time signal stuff.  That's a little tricky to use.
Or you could recode your app to use one thread or process
per 1000 fd's.
Solaris and FreeBSD both offer friendlier poll alternatives
that perform quite well.  You might consider using one of those
operating systems if all else fails.

- Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Poll and Select not scaling

[dean gaudet]

On Wed, 10 Jan 2001, Dan Kegel wrote:

> select() is usually limited to 1024 file descriptors

oh hey, this limit is only a libc limit these days.  you can do this:

#define MY_FD_SETSIZE (16384)
typedef struct {
        __fd_mask __fds_bits[MY_FD_SETSIZE / __NFDBITS];
} my_fd_set;
#define MY_FD_ZERO(_f)  (memset((_f), 0, sizeof(my_fd_set)))

and do select()s of 16384 descriptors.

> poll() is a slightly better choice.  However, although
> it can handle 30000 file descriptors, the performance sucks;
> see http://www.kegel.com/dkftpbench/Poller_bench.html#results

poll() stops working at 16384 file descriptors (as of 2.2.14-foo original
redhat 6.2 kernel).  at least that's where i think it is, maybe it's
32768.  it's limited by the maximum kmalloc() size of 128k.  this is
somewhat unfortunate.  even though we know it'll stop performing well.

-dean

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Poll and Select not scaling

[Micah Gorrell]

I would just like to thank every one for your help.  I have found the
problem and it is now working wonderfuly.

Micah

-----Original Message-----
From: "David Schwartz" <davids@webmaster.com>
To: "Micah Gorrell" <angelcode@myrealbox.com>;
<linux-kernel@vger.kernel.org>
Date: Thursday, January 11, 2001 3:42 PM
Subject: RE: Poll and Select not scaling


>
>> I have been trying to increase the scalabilty of an email server that has
>> been ported to Linux.  It was originally written for Netware, and there
we
>> are able to provide over 30,000 connections at any given time.  On Linux
>> however select stops working after the first 1024 connections.  I have
>> changed include/linux/fs.h and updated NR_FILE to be 81920.  In test
>> applications I have been able to create well over 30,000 connections but
I
>> am unable to do either a select or a poll on them.  Does any one
>> know what I
>> can do to fix this?
>
> Multithread.
>
> DS
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>Please read the FAQ at http://www.tux.org/lkml/
>
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Poll and Select not scaling

[Micah Gorrell]

If I am asking this in the wrong place I apoligize in advace.  Please let me
know where the correct forum would be.


I have been trying to increase the scalabilty of an email server that has
been ported to Linux.  It was originally written for Netware, and there we
are able to provide over 30,000 connections at any given time.  On Linux
however select stops working after the first 1024 connections.  I have
changed include/linux/fs.h and updated NR_FILE to be 81920.  In test
applications I have been able to create well over 30,000 connections but I
am unable to do either a select or a poll on them.  Does any one know what I
can do to fix this?

Micah

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Poll and Select not scaling

[Chris Wedgwood]

On Wed, Jan 10, 2001 at 04:21:17PM -0700, Micah Gorrell wrote:

    I have been trying to increase the scalabilty of an email server
    that has been ported to Linux.  It was originally written for
    Netware, and there we are able to provide over 30,000 connections
    at any given time.  On Linux however select stops working after
    the first 1024 connections.  I have changed include/linux/fs.h
    and updated NR_FILE to be 81920.  In test applications I have
    been able to create well over 30,000 connections but I am unable
    to do either a select or a poll on them.  Does any one know what
    I can do to fix this?

Which verion of linux and what libc are you using? For some time now
linux has supported the ability to select and poll on more than 102
FDs and several applications do indeed use this (squid for example).

For large numbers of FDs you probably want poll, which shouldn't give
you any problems. As you point out with sleect you need to redefine
NR_FILE or somesuch -- but hacking the kernel headers shouldn't be
necessary and may not help as libc will stull potentially get the
wrong value.

See where libc gets it from, I think recent libc version may allow
you to do something like:


#define NR_FILE	 <large-number>
#include <blah.h>
#include <blem.h>


but I could be wrong. Using poll you you have no problems.




  --cw
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Poll and Select not scaling

[Bill Wendling]

Also sprach Micah Gorrell:
} If I am asking this in the wrong place I apoligize in advace.  Please let me
} know where the correct forum would be.
} 
} 
} I have been trying to increase the scalabilty of an email server that has
} been ported to Linux.  It was originally written for Netware, and there we
} are able to provide over 30,000 connections at any given time.  On Linux
} however select stops working after the first 1024 connections.  I have
} changed include/linux/fs.h and updated NR_FILE to be 81920.  In test
} applications I have been able to create well over 30,000 connections but I
} am unable to do either a select or a poll on them.  Does any one know what I
} can do to fix this?
} 
Hi Micah,

Which kernel version are you using?

-- 
|| Bill Wendling			wendling@ganymede.isdn.uiuc.edu
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: Poll and Select not scaling

[David Schwartz]

> I have been trying to increase the scalabilty of an email server that has
> been ported to Linux.  It was originally written for Netware, and there we
> are able to provide over 30,000 connections at any given time.  On Linux
> however select stops working after the first 1024 connections.  I have
> changed include/linux/fs.h and updated NR_FILE to be 81920.  In test
> applications I have been able to create well over 30,000 connections but I
> am unable to do either a select or a poll on them.  Does any one
> know what I
> can do to fix this?

	Multithread.

	DS

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/