rsync and memory leak on linux 2.2?

rsync and memory leak on linux 2.2?

[Patrick O'Rourke]

We have two machines running Linux 2.2 with ReiserFS v3.5.32 and we're
seeing a memory leak which we feel is related to reiser and some interaction
with rsync.

The configuration is such that system A maintains a set of log files and
these files are mirrored via rsync on system B for redundancy.

What happens is that we put both systems under a fixed work load and after
many hours system B will start consuming lots of swap and suffer degraded
performance.  Through some kernel debugging we discovered that the 4K kmalloc
slab is slowly growing to the point where there is enough memory pressure to
start swapping.  We observe that one of the heaviest users of the 4K slab is
reiserfs_kmalloc(), and in particular, the calls issued from
get_mem_for_virtual_node() and reiserfs_file_write().  As an experiment we
ran the same work load, but this time disabled the rsync'ing of the log files
and we no longer see the growth in the 4k slab.

This leads us to believe that we have a memory leak somewhere in the reiserfs
and was wondering if anyone else has seen this, and if so, if a patch exists.

One question I have is that get_mem_for_virtual_node() will first attempt to
allocate memory atomically, and if this fails, will re-try non atomically.
In this case we return SCHEDULE_OCCURRED which results in fix_nodes()
will also return to its caller.  Is it possible for buffer allocated by
get_mem_for_virtual_node() to be lost in this case?  I did not see any
path out of reiserfs_file_write() in which we could return w/o freeing
the buffer.  Perhaps this problem is triggered via memory pressure?

I scanned the 2.2.20 reiser patch and did not see any fix that was apparent
to me.  Any suggestions / recommendations would be appreciated.

Thanks,

Pat

p.s. Please CC me on any reply, I am not subscribed to this list.

-- 
Patrick O'Rourke
porourke@egenera.com

Re: rsync and memory leak on linux 2.2?

[Oleg Drokin]

Hello!

On Thu, Feb 13, 2003 at 12:11:44PM -0500, Patrick O'Rourke wrote:
> What happens is that we put both systems under a fixed work load and after
> many hours system B will start consuming lots of swap and suffer degraded
> performance.  Through some kernel debugging we discovered that the 4K kmalloc
> slab is slowly growing to the point where there is enough memory pressure to
> start swapping.  We observe that one of the heaviest users of the 4K slab is
> reiserfs_kmalloc(), and in particular, the calls issued from
> get_mem_for_virtual_node() and reiserfs_file_write().  As an experiment we
> ran the same work load, but this time disabled the rsync'ing of the log files
> and we no longer see the growth in the 4k slab.
> This leads us to believe that we have a memory leak somewhere in the reiserfs
> and was wondering if anyone else has seen this, and if so, if a patch exists.
> One question I have is that get_mem_for_virtual_node() will first attempt to
> allocate memory atomically, and if this fails, will re-try non atomically.
> In this case we return SCHEDULE_OCCURRED which results in fix_nodes()
> will also return to its caller.  Is it possible for buffer allocated by
> get_mem_for_virtual_node() to be lost in this case?  I did not see any
> path out of reiserfs_file_write() in which we could return w/o freeing
> the buffer.  Perhaps this problem is triggered via memory pressure?

Thanks for the report. I will investigate it tomorrow, when it is day again
in Russia.
Meanwile there is easy way to find if there is some memory leaked through
reiserfs_kmalloc. If you enable CONFIG_REISERFS_CHECK kernel option,
then there is code in reiserfs_kmalloc, that cheks if we alloc, but not free
the memory.
It will print a warning once in a while.
But please note that CONFIG_REISERFS_CHECK will impose some (substantial?)
slowdown to reiserfs operations, so you might just unconditionally
enable the check for memleak in there if you cannot afford the slowdown.

Bye,
    Oleg

Re: rsync and memory leak on linux 2.2?

[Adrian Phillips]

>>>>> "Patrick" == Patrick O'Rourke <porourke@egenera.com> writes:

    Patrick> We have two machines running Linux 2.2 with ReiserFS
    Patrick> v3.5.32 and we're seeing a memory leak which we feel is
    Patrick> related to reiser and some interaction with rsync.

Well, there is a 3.5.34 which has, obviously, a number of bug
fixes. You could try that. I've been running 2.2.20 with this version
for a good number of months on several servers with no problems.

Sincerely,

Adrian Phillips

-- 
Your mouse has moved.
Windows NT must be restarted for the change to take effect.
Reboot now?  [OK]

Re: rsync and memory leak on linux 2.2?

[Oleg Drokin]

Hello!

On Fri, Feb 14, 2003 at 07:38:41AM +0100, Adrian Phillips wrote:
>     Patrick> We have two machines running Linux 2.2 with ReiserFS
>     Patrick> v3.5.32 and we're seeing a memory leak which we feel is
>     Patrick> related to reiser and some interaction with rsync.
> Well, there is a 3.5.34 which has, obviously, a number of bug

There are 3.5.35 and 3.5.36-beta that are even better ;)

Bye,
    Oleg

Re: rsync and memory leak on linux 2.2?

[Adrian Phillips]

>>>>> "Oleg" == Oleg Drokin <green@namesys.com> writes:

    Oleg> There are 3.5.35 and 3.5.36-beta that are even better ;)

I'd forgotten 35, even though I had saved the announcement. To busy
testing 2.4 last year when it came out. That and the fact that all the
servers were running rock solid. Thanks to the reiserfs team :-)

Sincerely,

Adrian Phillips

-- 
Your mouse has moved.
Windows NT must be restarted for the change to take effect.
Reboot now?  [OK]

Re: rsync and memory leak on linux 2.2?

[Patrick O'Rourke]

On Fri, 2003-02-14 at 01:38, Adrian Phillips wrote:

> Well, there is a 3.5.34 which has, obviously, a number of bug
> fixes. You could try that. I've been running 2.2.20 with this version
> for a good number of months on several servers with no problems.

That's a thought.  We've been running w/ 3.5.32 (+ a variety of fixes)
for over a year and this is the only issue we have thus far and it only
occurs in an isolated scenario (i.e. rsync).

Thanks for the suggestion.

Pat

-- 
Patrick O'Rourke
porourke@egenera.com

Re: rsync and memory leak on linux 2.2?

[Patrick O'Rourke]

On Fri, 2003-02-14 at 05:01, Oleg Drokin wrote:

> On Fri, Feb 14, 2003 at 07:38:41AM +0100, Adrian Phillips wrote:
> >     Patrick> We have two machines running Linux 2.2 with ReiserFS
> >     Patrick> v3.5.32 and we're seeing a memory leak which we feel is
> >     Patrick> related to reiser and some interaction with rsync.
> > Well, there is a 3.5.34 which has, obviously, a number of bug

I just thought I would follow up with a note that the bug turned
out NOT to be in the reiser code.  The rsync operation I described
turned out to be a red herring.

Thanks for the suggestions,

Pat

-- 
Patrick O'Rourke
porourke@egenera.com

Re: rsync and memory leak on linux 2.2?

[Hans Reiser]

Patrick O'Rourke wrote:

>On Fri, 2003-02-14 at 05:01, Oleg Drokin wrote:
>
>  
>
>>On Fri, Feb 14, 2003 at 07:38:41AM +0100, Adrian Phillips wrote:
>>    
>>
>>>    Patrick> We have two machines running Linux 2.2 with ReiserFS
>>>    Patrick> v3.5.32 and we're seeing a memory leak which we feel is
>>>    Patrick> related to reiser and some interaction with rsync.
>>>Well, there is a 3.5.34 which has, obviously, a number of bug
>>>      
>>>
>
>I just thought I would follow up with a note that the bug turned
>out NOT to be in the reiser code.  The rsync operation I described
>turned out to be a red herring.
>
>Thanks for the suggestions,
>
>Pat
>
>  
>
Thanks Patrick.  This leaves us with one unsolved reiserfs V3 bug that I 
know of (unlink related one, vs is working on it, and he is not 
completely sure it is reiserfs not vfs....).  Oleg, am I right in 
thinking that the samba+reiserfs bug looks like a samba bug?

-- 
Hans

Re: rsync and memory leak on linux 2.2?

[Oleg Drokin]

Hello!

On Thu, Feb 20, 2003 at 01:18:24AM +0300, Hans Reiser wrote:

> Thanks Patrick.  This leaves us with one unsolved reiserfs V3 bug that I 
> know of (unlink related one, vs is working on it, and he is not 

Vladimir tracked that down yesterday.
At least we think so. (and even if the fix we discussed yesterday won't help,
that'd mean we have another bug in addition to what Vladimir found.
And this is indeed a race between two iget(). I am not sure if VFS
is flawed, but I think at least partially it is. Because in 2.5
that piece of code redesigned right way.)

> completely sure it is reiserfs not vfs....).  Oleg, am I right in 
> thinking that the samba+reiserfs bug looks like a samba bug?

That turned out to be fs corruption of unknown source. Hardware is
the main suspect for now.

Bye,
    Oleg