file-nr, file-max, and ReiserFS

file-nr, file-max, and ReiserFS

[JP Howard]

We're currently testing ReiserFS on one of our servers. It's holding up
pretty well so far, except that we're seeing one perculiarity...

The ReiserFS server (running Cyrus and Postfix) has been hitting our max
FDs (/proc/sys/fs/file-max) a few times, which is set at 90,000. However,
we have an Ext3 server with almost the same # of users and overall load,
and it has only hit 20,000 open files (according to
/proc/sys/fs/file-nr).

Does ReiserFS use more file handles for some reason? Is there any
downside to setting file-max to something really big to avoid running out
of FDs? Do you think that this may actually indicate some problem with
our configuration, or this normal for ReiserFS?

Also, I checked with `lsof | wc -l` to see how many files it reported as
open. On the ReiserFS server it only showed 45,000 files as open (when
the system had actually used all 90,000 handles). On the Ext3 server is
showed 30,000 files as open (when the system had actually only used
20,000 handles according to file-nr). I guess I'm wrong to be thinking
that lsof and file-nr should show the same number of open files, but
could someone explain why?

TIA,
  Jeremy

Re: file-nr, file-max, and ReiserFS

[Oleg Drokin]

Hello!

On Tue, Oct 22, 2002 at 04:41:17AM +0000, JP Howard wrote:

> The ReiserFS server (running Cyrus and Postfix) has been hitting our max
> FDs (/proc/sys/fs/file-max) a few times, which is set at 90,000. However,
> we have an Ext3 server with almost the same # of users and overall load,
> and it has only hit 20,000 open files (according to
> /proc/sys/fs/file-nr).

Filesystems itself does not consume file descriptors.

> Does ReiserFS use more file handles for some reason? Is there any

Not likely.

> downside to setting file-max to something really big to avoid running out
> of FDs? Do you think that this may actually indicate some problem with

More fds you use, more RAM they consume ;)

> our configuration, or this normal for ReiserFS?

There must be the difference between your ext3 based and reiserfs based systems
that will explain FD usage difference. Something in userspace most probably.

> Also, I checked with `lsof | wc -l` to see how many files it reported as
> open. On the ReiserFS server it only showed 45,000 files as open (when
> the system had actually used all 90,000 handles). On the Ext3 server is

If the system is really used all 90000  handles, you would not be able to look
at /proc/sys/fs/file-nr content, so perhaps the peak was over already at a time
you reviewed it?

> showed 30,000 files as open (when the system had actually only used
> 20,000 handles according to file-nr). I guess I'm wrong to be thinking
> that lsof and file-nr should show the same number of open files, but
> could someone explain why?

Well, file descriptors might be shared between processes/threads, but
lsof does not know about this

Bye,
    Oleg

Re: file-nr, file-max, and ReiserFS

[JP Howard]

On Tue, 22 Oct 2002 10:33:49 +0400, "Oleg Drokin"
<green@namesys.com> said:
> > The ReiserFS server (running Cyrus and Postfix) has been hitting our
> > max FDs (/proc/sys/fs/file-max) a few times, which is set at 90,000.
> > However, we have an Ext3 server with almost the same # of users and
> > overall load, and it has only hit 20,000 open files (according to
> > /proc/sys/fs/file-nr).
>
> There must be the difference between your ext3 based and reiserfs based
> systems that will explain FD usage difference. Something in userspace
> most probably.
<...>

Sorry to waste your time--we've identified the problem and it wasn't
ReiserFS related. Turns out we introduced a bug into our LMTP proxy that
was only triggered by a rare event--it so happened this event was related
to an email sent to this particular server. 

<...>
> > showed 30,000 files as open (when the system had actually only used
> > 20,000 handles according to file-nr). I guess I'm wrong to be
> > thinking that lsof and file-nr should show the same number of open
> > files, but could someone explain why?
>
> Well, file descriptors might be shared between processes/threads, but
> lsof does not know about this
>
Me being dumb again... Well, kinda... turns out that the docs for file-nr
in the Linux source tree are wrong. For the record, the correct
information is here:  http://www.cs.tau.ac.il/~didi/file-nr.html

When interpreted correctly, the counts are pretty close.

Re: file-nr, file-max, and ReiserFS

[JP Howard]

On Tue, 22 Oct 2002 08:02:10 -0600, "Eric Whiting"
<eric_whiting@amis.com> said:
> JP,
> 
> Were both boxes running cyrus/postfix? Postfix relies heavily on several
> different mailqueue dirs.
> 
The source of the problem was actually a Postfix bug, which Wietse had
provided a patch for on the mailing list, but I hadn't yet applied. The
bug caused an infinite loop that opened thousands of LMTP processes on
our backend, causing it to run out of file handles. It was nothing to do
with ReiserFS.
> 
> Oleg Drokin wrote:
> > 
> > Hello!
> > 
> > On Tue, Oct 22, 2002 at 04:41:17AM +0000, JP Howard wrote:
> > 
> > > The ReiserFS server (running Cyrus and Postfix) has been hitting our max
> > > FDs (/proc/sys/fs/file-max) a few times, which is set at 90,000. However,
> > > we have an Ext3 server with almost the same # of users and overall load,
> > > and it has only hit 20,000 open files (according to
> > > /proc/sys/fs/file-nr).
> > 
> > Filesystems itself does not consume file descriptors.
> > 
> > > Does ReiserFS use more file handles for some reason? Is there any
> > 
> > Not likely.
> > 
> > > downside to setting file-max to something really big to avoid running out
> > > of FDs? Do you think that this may actually indicate some problem with
> > 
> > More fds you use, more RAM they consume ;)
> > 
> > > our configuration, or this normal for ReiserFS?
> > 
> > There must be the difference between your ext3 based and reiserfs based systems
> > that will explain FD usage difference. Something in userspace most probably.
> > 
> > > Also, I checked with `lsof | wc -l` to see how many files it reported as
> > > open. On the ReiserFS server it only showed 45,000 files as open (when
> > > the system had actually used all 90,000 handles). On the Ext3 server is
> > 
> > If the system is really used all 90000  handles, you would not be able to look
> > at /proc/sys/fs/file-nr content, so perhaps the peak was over already at a time
> > you reviewed it?
> > 
> > > showed 30,000 files as open (when the system had actually only used
> > > 20,000 handles according to file-nr). I guess I'm wrong to be thinking
> > > that lsof and file-nr should show the same number of open files, but
> > > could someone explain why?
> > 
> > Well, file descriptors might be shared between processes/threads, but
> > lsof does not know about this
> > 
> > Bye,
> >     Oleg
>