More than 1M open file descriptors

More than 1M open file descriptors

[John Masinter]

I'm a software engineer and work with a fortune-500 company. We have a
line of linux-powered network security appliances and I am responsible
for Linux / OS development.
Our multi-core appliance handles a large number of low-bandwith tcp
connections, and after years of performance streamlining, we have hit
the kernel's million tcp connection limit.
Is there a kernel hack to increase the file descriptor limit beyond 1M
to that we may have more than 1M open tcp connections? If this is
discussed elsewhere, please point me in the right direction. Many
thanks to everyone that contributes to open source.

Re: More than 1M open file descriptors

[Eric Dumazet]

Le mercredi 19 mai 2010 à 10:52 -0400, John Masinter a écrit :
> I'm a software engineer and work with a fortune-500 company. We have a
> line of linux-powered network security appliances and I am responsible
> for Linux / OS development.
> Our multi-core appliance handles a large number of low-bandwith tcp
> connections, and after years of performance streamlining, we have hit
> the kernel's million tcp connection limit.
> Is there a kernel hack to increase the file descriptor limit beyond 1M
> to that we may have more than 1M open tcp connections? If this is
> discussed elsewhere, please point me in the right direction. Many
> thanks to everyone that contributes to open source.
> --

Its not a problem with a recent kernel (2.6.25)

commit 9cfe015aa424b3c003baba3841a60dd9b5ad319b
Author: Eric Dumazet <dada1@cosmosbay.com>
Date:   Wed Feb 6 01:37:16 2008 -0800

    get rid of NR_OPEN and introduce a sysctl_nr_open
    
    NR_OPEN (historically set to 1024*1024) actually forbids processes to open
    more than 1024*1024 handles.
    
    Unfortunatly some production servers hit the not so 'ridiculously high
    value' of 1024*1024 file descriptors per process.
    
    Changing NR_OPEN is not considered safe because of vmalloc space potential
    exhaust.
    
    This patch introduces a new sysctl (/proc/sys/fs/nr_open) wich defaults to
    1024*1024, so that admins can decide to change this limit if their workload
    needs it.
    

Re: More than 1M open file descriptors

[Felipe Balbi]

On Wed, May 19, 2010 at 04:52:56PM +0200, ext John Masinter wrote:
>I'm a software engineer and work with a fortune-500 company. We have a
>line of linux-powered network security appliances and I am responsible
>for Linux / OS development.
>Our multi-core appliance handles a large number of low-bandwith tcp
>connections, and after years of performance streamlining, we have hit
>the kernel's million tcp connection limit.
>Is there a kernel hack to increase the file descriptor limit beyond 1M
>to that we may have more than 1M open tcp connections? If this is
>discussed elsewhere, please point me in the right direction. Many
>thanks to everyone that contributes to open source.

look into fs/file.c
increasing sysctl_nr_open like below should help:

diff --git a/fs/file.c b/fs/file.c
index 87e1290..23c83df 100644
--- a/fs/file.c
+++ b/fs/file.c
@@ -27,9 +27,9 @@ struct fdtable_defer {
  	struct fdtable *next;
  };
  
-int sysctl_nr_open __read_mostly = 1024*1024;
+int sysctl_nr_open __read_mostly = 4*1024*1024;
  int sysctl_nr_open_min = BITS_PER_LONG;
-int sysctl_nr_open_max = 1024 * 1024; /* raised later */
+int sysctl_nr_open_max = 4 * 1024 * 1024; /* raised later */
  
  /*
   * We use this list to defer free fdtables that have vmalloced

you can also do it via /proc AFAICT.

-- 
balbi

DefectiveByDesign.org