Serious allocating/freeing memory or ReiserFS problem in 2.4.10

Serious allocating/freeing memory or ReiserFS problem in 2.4.10

[Veit Wahlich]

[-- Attachment #1: Type: text/plain, Size: 4191 bytes --]

Hi!

When I upgraded my kernel from 2.4.9 to 2.4.10 I experienced the problem
described in this posting. Because I lack in time, I am not able to read
this mailing list regularly, so I do not know whether this problem is
known or not. Because I am short on time and this caused crashes on my
system I can not apply any screenshots/hardcopys but I will try to help
fetching information about this wherever I can. For this please contact
cru@zodia.de .

I discovered this while using donkey (www.edonkey2000.com), a
non-opensource P2P solution applying a Linux client. Contact is
info@edonkey2000.com , I will forward this email to them.

As donkey is a MFTP implementation it begins to "hash" all incomplete
files in the download queue when run. This means it reads the whole file
and builds a kind of checksum (as I suppose) of all segments that are
already downloaded. During this process, the amount of memory in use
steadily grows, while the memory used by all processes (including
donkey) stays nearly unchanged. Because I am running ReiserFS and donkey
is reading huge files, this could also be a ReiserFS problem. I have no
experience with donkey and kernels <2.4.9.


Using 384Mb RAM on Linux 2.4.9 with CONFIG_NOHIGHMEM:

When memory usage is up to 95%, the system begins to push all other
processes to swap. This includes some daemons, XF86 4.0.3, some killer
applications (Mozilla, Evolution) and even parts of the donkey itself.
Working becomes nearly impossible but the system keeps working.
When "hashing" has been completed, memory usage stays at 99.9% but all
processes can be used again, pulling them from swap. Opening and closing
bigger applications also "frees" some memory, closing the donkey frees
not more memory than the few Mb the donkey processes really used. But in
this state working with the machine is possible as every day.


Using 384Mb RAM on Linux 2.4.10 with CONFIG_NOHIGHMEM:

The first time I ran the donkey I discovered that it was much slower and
took twice the time it usually takes for hashing, but work on the
machine after hashing was fine. I decided to buy more RAM.


Using 1023Mb RAM on Linux 2.4.10 with CONFIG_HIGHMEM4G:

As the kernel told me on boot, I enabled CONFIG_HIGHMEM4G to use more
than 896Mb. That gave me 896+127Mb.
Now running donkey ended with X to hang, when the system began to swap
at 99% mem usage. I was able to ssh to the machine, ps told me X was
using 1160% of MEM and killing it was impossible, kill gave no msg on
STDOUT/-ERR, the process stayed in state R. Shutting it down failed,
shutdown became a zombie.

On next run I started watching the donkey hashing and inspected the
memory usage with top. When mem usage reached 99% and swap 72Mb, several
processes collapsed with a segfault (some Gnome applets and Evolution),
top got some "/proc/*: out of memory" messages on the screen, than also
top segfaulted. Donkey stopped working without completing the hashing
because one of its own processes collapsed. Still running processes
looked fine with ps but shutting down failed as described above.

The third run exactly was like run #2.


Using 896Mb RAM on Linux 2.4.10 with CONFIG_NOHIGHMEM:

When donkey was hashing and swap reached about 70Mb, X collapsed (and
got restarted by init), donkey was killed by this. Gracefully shutting
down the machine was possible.


Using 896Mb RAM on Linux 2.4.9 with CONFIG_NOHIGHMEM:

Run #2: I booted the 2.4.9 kernel without highmem support and donkey
runs fine, I am even able to work a bit while it is hashing (and
swapping).


I was unable to test 2.4.9 with CONFIG_HIGHMEM4G because I deleted the
source tree backup after 2.4.10 seemed to run fine and I had no time to
download and build it again.

The kernel is Linux 2.4.9/2.4.10 without third-party patches. I am
running nVidia's NVdriver/GLX v1.0-1512 but this module was disabled in
one unsuccessful run, gcc is 2.95.2.1. The new RAM has been successfully
tested for 8h by memtest86 on this machine.


Please CC any comments, answers, questions, solutions regarding this
posting to cru@zodia.de . 

Thanks.
// Veit Wahlich


[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: Serious allocating/freeing memory or ReiserFS problem in 2.4.10

[Olivier Sessink]

On  0, Veit Wahlich <cru@zodia.de> wrote:
> Hi!
> 
> When I upgraded my kernel from 2.4.9 to 2.4.10 I experienced the problem
> described in this posting. Because I lack in time, I am not able to read
> this mailing list regularly, so I do not know whether this problem is
> known or not. Because I am short on time and this caused crashes on my
> system I can not apply any screenshots/hardcopys but I will try to help
> fetching information about this wherever I can. For this please contact
> cru@zodia.de .
> 
> I discovered this while using donkey (www.edonkey2000.com), a
> non-opensource P2P solution applying a Linux client. Contact is
> info@edonkey2000.com , I will forward this email to them.
> 
> As donkey is a MFTP implementation it begins to "hash" all incomplete
> files in the download queue when run. This means it reads the whole file
> and builds a kind of checksum (as I suppose) of all segments that are
> already downloaded. During this process, the amount of memory in use
> steadily grows, while the memory used by all processes (including
> donkey) stays nearly unchanged. Because I am running ReiserFS and donkey
> is reading huge files, this could also be a ReiserFS problem. I have no
> experience with donkey and kernels <2.4.9.

this is a problem specific to 2.4.10, I have the same problem (also using
edonkey), 2.4.10pre8 doesn't have this problem. I don't know what changed,
but 2.4.10 is not usable.

regards,
	Olivier

-- 
begin signature_virus.TXT.vbs
Hi, I'm a signature virus. plz set me as your signature and help me
spread :)
end