Re: linux-2.4.13 high SWAP

Re: linux-2.4.13 high SWAP

[Marcelo Tosatti]

On Wed, 24 Oct 2001, Andre Margis wrote:

> Em Qua 24 Out 2001 15:05, Andre Margis escreveu:
> 
> Mor minutes later the machine "froze".

Could you please redo the tests without tmpfs? 

I'm not sure if its the problem, just want to make sure.

Thanks.

linux-2.4.13 high SWAP

[Andre Margis]

I'm testing 2.4.13, and something  is wrong.....


Them machine for test is a DELL 8450 4xPIII 4 GBram, running 4 setiathome, 5 
cp on tmpfs and 1 cpio.

After minutes running the machine eat all my swap area. like "top" sample 
bellow :
Mem:  4118212K av, 3693728K used,  424484K free,       0K shrd,     956K buff
                    615928K actv,       0K in_d,       0K in_c,       0Ktarget
Swap: 1048568K av,  957456K used,   91112K free                 2420888Kcached

I'm using highmem.


Some kernel tunning to adjust that?


Thank's


Andre

Re: linux-2.4.13 high SWAP

[Marcelo Tosatti]

Ok, 

Have you checked if the amount of data you copied to the tmpfs device is
not way too big to fit in memory ?

Remember: Everything copied to tmpfs will be kept in memory, so if you
simply copy way too much data to tmpfs thats your problem :)

On Wed, 24 Oct 2001, Andre Margis wrote:

> Em Qua 24 Out 2001 14:44, Marcelo Tosatti escreveu:
> Marcelo,
> 
> I restart the test using the same programs, but now I'm using the "cp" on a 
> normal filesystem. At this time everything is OK.
> 
> In the last run we Nedd 30 minutes to the disaster.
> 
> 
> Andre
> 
> > On Wed, 24 Oct 2001, Andre Margis wrote:
> > > Em Qua 24 Out 2001 15:05, Andre Margis escreveu:
> > >
> > > Mor minutes later the machine "froze".
> >
> > Could you please redo the tests without tmpfs?
> >
> > I'm not sure if its the problem, just want to make sure.
> >
> > Thanks.
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: linux-2.4.13 high SWAP

[Andre Margis]

Em Qua 24 Out 2001 15:05, Andre Margis escreveu:

Mor minutes later the machine "froze".


Andre

> I'm testing 2.4.13, and something  is wrong.....
>
>
> Them machine for test is a DELL 8450 4xPIII 4 GBram, running 4 setiathome,
> 5 cp on tmpfs and 1 cpio.
>
> After minutes running the machine eat all my swap area. like "top" sample
> bellow :
> Mem:  4118212K av, 3693728K used,  424484K free,       0K shrd,     956K
> buff 615928K actv,       0K in_d,       0K in_c,       0Ktarget Swap:
> 1048568K av,  957456K used,   91112K free                 2420888Kcached
>
> I'm using highmem.
>
>
> Some kernel tunning to adjust that?
>
>
> Thank's
>
>
> Andre
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-2.4.13 high SWAP

[Andre Margis]

Em Qua 24 Out 2001 14:44, Marcelo Tosatti escreveu:
Marcelo,

I restart the test using the same programs, but now I'm using the "cp" on a 
normal filesystem. At this time everything is OK.

In the last run we Nedd 30 minutes to the disaster.


Andre

> On Wed, 24 Oct 2001, Andre Margis wrote:
> > Em Qua 24 Out 2001 15:05, Andre Margis escreveu:
> >
> > Mor minutes later the machine "froze".
>
> Could you please redo the tests without tmpfs?
>
> I'm not sure if its the problem, just want to make sure.
>
> Thanks.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-2.4.13 high SWAP

[Andre Margis]

Em Qua 24 Out 2001 15:10, Marcelo Tosatti escreveu:
The tmpfs is configured with 2GB. I'm copying 6 file of 200M, total 1.2GB.

df -k

 Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda3              4096440   3342964    753476  82% /
/dev/sda2                31111     12264     17241  42% /boot
/dev/root/fs01         3145628    465992   2679636  15% /u
/dev/root/fs02         9330396   1475848   7854548  16% /prod
tmpfs                  2097152    204836   1892316  10% /tmp

Without use the tmpfs, appears to be OK!!!!!!!!!!

Using 2.4.10-ac7, the same test run OK!!!!!!!!


Andre


> Ok,
>
> Have you checked if the amount of data you copied to the tmpfs device is
> not way too big to fit in memory ?
>
> Remember: Everything copied to tmpfs will be kept in memory, so if you
> simply copy way too much data to tmpfs thats your problem :)
>
> On Wed, 24 Oct 2001, Andre Margis wrote:
> > Em Qua 24 Out 2001 14:44, Marcelo Tosatti escreveu:
> > Marcelo,
> >
> > I restart the test using the same programs, but now I'm using the "cp" on
> > a normal filesystem. At this time everything is OK.
> >
> > In the last run we Nedd 30 minutes to the disaster.
> >
> >
> > Andre
> >
> > > On Wed, 24 Oct 2001, Andre Margis wrote:
> > > > Em Qua 24 Out 2001 15:05, Andre Margis escreveu:
> > > >
> > > > Mor minutes later the machine "froze".
> > >
> > > Could you please redo the tests without tmpfs?
> > >
> > > I'm not sure if its the problem, just want to make sure.
> > >
> > > Thanks.
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> > > in the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-2.4.13 high SWAP

[Linus Torvalds]

In article <200110241936.RAA04632@inter.lojasrenner.com.br>,
Andre Margis  <andre@sam.com.br> wrote:
>
>Without use the tmpfs, appears to be OK!!!!!!!!!!

Ok, the problem appears to be that tmpfs stuff just stays on the
inactive list, and because it cannot be written out it eventually
totally clogs the system.

Suggested fix appended (from Andrea),

		Linus

-----
diff -u --recursive --new-file v2.4.13/linux/drivers/block/rd.c linux/drivers/block/rd.c
--- v2.4.13/linux/drivers/block/rd.c	Tue Oct 23 22:48:50 2001
+++ linux/drivers/block/rd.c	Wed Oct 24 08:59:21 2001
@@ -209,6 +209,7 @@
  */
 static int ramdisk_writepage(struct page *page)
 {
+	activate_page(page);
 	SetPageDirty(page);
 	UnlockPage(page);
 	return 0;
diff -u --recursive --new-file v2.4.13/linux/fs/ramfs/inode.c linux/fs/ramfs/inode.c
--- v2.4.13/linux/fs/ramfs/inode.c	Tue Oct  9 17:06:53 2001
+++ linux/fs/ramfs/inode.c	Wed Oct 24 08:59:21 2001
@@ -81,6 +81,7 @@
  */
 static int ramfs_writepage(struct page *page)
 {
+	activate_page(page);
 	SetPageDirty(page);
 	UnlockPage(page);
 	return 0;

Re: linux-2.4.13 high SWAP

[Lukasz Trabinski]

In article <9r73pv$8h1$1@penguin.transmeta.com> you wrote:
>>
>>Without use the tmpfs, appears to be OK!!!!!!!!!!

> Ok, the problem appears to be that tmpfs stuff just stays on the
> inactive list, and because it cannot be written out it eventually
> totally clogs the system.

> Suggested fix appended (from Andrea),

What about that:

 10:42pm  up 10:09,  2 users,  load average: 1.40, 1.31, 1.28
166 processes: 163 sleeping, 3 running, 0 zombie, 0 stopped
CPU0 states:  0.0% user, 97.0% system,  0.0% nice,  2.0% idle
CPU1 states:  0.1% user, 21.0% system,  0.0% nice, 77.0% idle
Mem:  2061632K av, 2057024K used,    4608K free,       0K shrd,   55412K buff
Swap: 1911528K av,    3060K used, 1908468K free                 1513964K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
    5 root      16   0     0    0     0 RW   99.9  0.0 157:00 kswapd

It's looks strange and danger. On this machine squid and INN running.
Swap is on still level, but 99.9% for CPU? System without tmpfs, but
with resierfs (50GB of squid spool on 5 partitions).


-- 
*[ Łukasz Trąbiński ]*
SysAdmin @wsisiz.edu.pl

Re: linux-2.4.13 high SWAP

[toon]

On Wed, Oct 24, 2001 at 07:11:59PM +0000, Linus Torvalds wrote:
> In article <200110241936.RAA04632@inter.lojasrenner.com.br>,
> Andre Margis  <andre@sam.com.br> wrote:
> >
> >Without use the tmpfs, appears to be OK!!!!!!!!!!
> 
> Ok, the problem appears to be that tmpfs stuff just stays on the
> inactive list, and because it cannot be written out it eventually
> totally clogs the system.
> 
> Suggested fix appended (from Andrea),
> 
> 		Linus

I started out with a clean 2.4.13 source tree, applied the
activate_page patch, and compile as follows:

make dep clean bzImage
make modules modules_install

The command `make modules_install' results in the following output:

if [ -r System.map ]; then /sbin/depmod -ae -F System.map  2.4.13; fi
depmod: *** Unresolved symbols in /lib/modules/2.4.13/kernel/fs/ramfs/ramfs.o
depmod: 	activate_page

Maybe an #include of some header file is missing somewhere?

Regards,
Toon.
-- 
 /"\                             |   Windows XP:
 \ /     ASCII RIBBON CAMPAIGN   |        "I'm sorry Dave...
  X        AGAINST HTML MAIL     |         I'm afraid I can't do that."
 / \

Re: linux-2.4.13 high SWAP

[David S. Miller]

   From: toon@vdpas.hobby.nl
   Date: Wed, 24 Oct 2001 23:48:26 +0200

   The command `make modules_install' results in the following output:
   
   if [ -r System.map ]; then /sbin/depmod -ae -F System.map  2.4.13; fi
   depmod: *** Unresolved symbols in /lib/modules/2.4.13/kernel/fs/ramfs/ramfs.o
   depmod: 	activate_page
   
   Maybe an #include of some header file is missing somewhere?

No, the fix is even simpler:

--- ../vanilla/linux/kernel/ksyms.c	Wed Oct 17 14:32:50 2001
+++ kernel/ksyms.c	Wed Oct 24 14:45:31 2001
@@ -116,6 +116,7 @@
 EXPORT_SYMBOL(get_unmapped_area);
 EXPORT_SYMBOL(init_mm);
 EXPORT_SYMBOL(deactivate_page);
+EXPORT_SYMBOL(activate_page);
 #ifdef CONFIG_HIGHMEM
 EXPORT_SYMBOL(kmap_high);
 EXPORT_SYMBOL(kunmap_high);

Re: linux-2.4.13 high SWAP

[Christoph Rohland]

Hi Marcelo,

On Wed, 24 Oct 2001, Marcelo Tosatti wrote:
> Remember: Everything copied to tmpfs will be kept in memory, so if
> you simply copy way too much data to tmpfs thats your problem :)

Nope, it will swap it out.

On Wed, 24 Oct 2001, Linus Torvalds wrote:
> Ok, the problem appears to be that tmpfs stuff just stays on the
> inactive list, and because it cannot be written out it eventually
> totally clogs the system.
> 
> Suggested fix appended (from Andrea),

> --- v2.4.13/linux/fs/ramfs/inode.c	Tue Oct  9 17:06:53 2001
> +++ linux/fs/ramfs/inode.c	Wed Oct 24 08:59:21 2001

tmpfs != ramfs. So either the patch is not complete or fixes another
problem...

Greetings
		Christoph

Re: linux-2.4.13 high SWAP

[Robert Scussel]

Just thought that I would add our experience.

We have experienced the same kind of swap symptoms described, however we
have no mounted tmpfs, or ramfs partitions. We have, in fact,
experienced the same symptoms on the 2.4.2,2.4.5,2.4.7 and 2.4.12
kernel, haven't yet tried the 2.4.13 kernel.  The symptoms include hung
processes which can not be killed, system cannot right to disk, and
files accessed during this time are filled with binary zeros.  As sync
does not work as well, the only resolution is to do a reboot -f -n.

All systems are comprised of exclusively SGI XFS partitions, with dual
pentium II/III processors.

Any insight would be helpful,

Robert Scussel
--
Robert Scussel
1024D/BAF70959/0036 B19E 86CE 181D 0912  5FCC 92D8 1EA1 BAF7 0959

Re: linux-2.4.13 high SWAP

[Pablo Ninja]

Hi Robert,

I'm just a regular user of sgi xfs on my desktop and I noted It eats up all memory (maybe cos it caches too much). Don't know if it matters but have you ever tried to umount/mount these partitions ?

[]'s
Pablo

On Mon, 29 Oct 2001 23:49:56 -0500
Robert Scussel <rscuss@omniti.com> wrote:

> Just thought that I would add our experience.
> 
> We have experienced the same kind of swap symptoms described, however we
> have no mounted tmpfs, or ramfs partitions. We have, in fact,
> experienced the same symptoms on the 2.4.2,2.4.5,2.4.7 and 2.4.12
> kernel, haven't yet tried the 2.4.13 kernel.  The symptoms include hung
> processes which can not be killed, system cannot right to disk, and
> files accessed during this time are filled with binary zeros.  As sync
> does not work as well, the only resolution is to do a reboot -f -n.
> 
> All systems are comprised of exclusively SGI XFS partitions, with dual
> pentium II/III processors.
> 
> Any insight would be helpful,
> 
> Robert Scussel
> --
> Robert Scussel
> 1024D/BAF70959/0036 B19E 86CE 181D 0912  5FCC 92D8 1EA1 BAF7 0959
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 



=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Pablo Borges                                pablo.borges@uol.com.br
-------------------------------------------------------------------
  ____                                               Tecnologia UOL
 /    \    Debian:
 |  =_/      The 100% suck free linux distro.
  \
    \      SETI is lame. http://www.distributed.net
                                                     Dnetc is XNUG!

Re: linux-2.4.13 high SWAP

[Ulrich Wiederhold]

Hi,
* Pablo Ninja <pablo.ninja@uol.com.br> [011030 11:28]:
> I'm just a regular user of sgi xfs on my desktop and I noted It eats up all memory (maybe cos it caches too much). Don't know if it matters but have you ever tried to umount/mount these partitions ?
> 
No problems here with sgi xfs on a lvm-volume and 2.4.13, 192MB RAM,
K6-2/400.

Uli

> 
> On Mon, 29 Oct 2001 23:49:56 -0500
> Robert Scussel <rscuss@omniti.com> wrote:
> 
> > Just thought that I would add our experience.
> > 
> > We have experienced the same kind of swap symptoms described, however we
> > have no mounted tmpfs, or ramfs partitions. We have, in fact,
> > experienced the same symptoms on the 2.4.2,2.4.5,2.4.7 and 2.4.12
> > kernel, haven't yet tried the 2.4.13 kernel.  The symptoms include hung
> > processes which can not be killed, system cannot right to disk, and
> > files accessed during this time are filled with binary zeros.  As sync
> > does not work as well, the only resolution is to do a reboot -f -n.
> > 
> > All systems are comprised of exclusively SGI XFS partitions, with dual
> > pentium II/III processors.
> > 
> > Any insight would be helpful,
> > 
> > Robert Scussel
> > --
> > Robert Scussel
> > 1024D/BAF70959/0036 B19E 86CE 181D 0912  5FCC 92D8 1EA1 BAF7 0959
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> > in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> > 
> 
> 
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Pablo Borges                                pablo.borges@uol.com.br
> -------------------------------------------------------------------
>   ____                                               Tecnologia UOL
>  /    \    Debian:
>  |  =_/      The 100% suck free linux distro.
>   \
>     \      SETI is lame. http://www.distributed.net
>                                                      Dnetc is XNUG!
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
'The box said, 'Requires Windows 95 or better', so i installed Linux - TKK 5

Re: linux-2.4.13 high SWAP

[Andre Margis]

I test 2.4.9 , 2.4.10-ac7, 2.4.13 and all have this problem, I'm not using 
XFS, but reiserfs with LVM and 4 GB RAM. I detected if use tmpfs the kswapd 
eat my all CPU's, in 2.4.13 the system hang after a time. Now I'm testing 
2.4.13-ac3 without tmpfs and he is very better than the others versions. But 
a nice test is disable the HIGHMEM support. I have a machine with 1GB RAM and 
the system is very fine and stable, running 2.4.10-ac7.

 


Em Ter 30 Out 2001 02:49, Robert Scussel escreveu:
> Just thought that I would add our experience.
>
> We have experienced the same kind of swap symptoms described, however we
> have no mounted tmpfs, or ramfs partitions. We have, in fact,
> experienced the same symptoms on the 2.4.2,2.4.5,2.4.7 and 2.4.12
> kernel, haven't yet tried the 2.4.13 kernel.  The symptoms include hung
> processes which can not be killed, system cannot right to disk, and
> files accessed during this time are filled with binary zeros.  As sync
> does not work as well, the only resolution is to do a reboot -f -n.
>
> All systems are comprised of exclusively SGI XFS partitions, with dual
> pentium II/III processors.
>
> Any insight would be helpful,
>
> Robert Scussel

Re: linux-2.4.13 high SWAP

[Robert Scussel]

Pablo Ninja wrote:
> 
> Hi Robert,
> 
> I'm just a regular user of sgi xfs on my desktop and I noted It eats up all memory (maybe cos it caches too much). Don't know if it matters but have you ever tried to umount/mount these partitions ?
>

Yes, I have tried to unmount the partition, however, it is impossible
once the machine gets into this state. 

One thing that I have noticed is that when the load starts to increase,
a manual sync, although it takes a long time, appears to keep off an
immediate hang of the system. Once the load gets above 25 however, the
machines spiral out of control.

Robert


> []'s
> Pablo
> 
> On Mon, 29 Oct 2001 23:49:56 -0500
> Robert Scussel <rscuss@omniti.com> wrote:
> 
> > Just thought that I would add our experience.
> >
> > We have experienced the same kind of swap symptoms described, however we
> > have no mounted tmpfs, or ramfs partitions. We have, in fact,
> > experienced the same symptoms on the 2.4.2,2.4.5,2.4.7 and 2.4.12
> > kernel, haven't yet tried the 2.4.13 kernel.  The symptoms include hung
> > processes which can not be killed, system cannot right to disk, and
> > files accessed during this time are filled with binary zeros.  As sync
> > does not work as well, the only resolution is to do a reboot -f -n.
> >
> > All systems are comprised of exclusively SGI XFS partitions, with dual
> > pentium II/III processors.
> >
> > Any insight would be helpful,
> >
> > Robert Scussel
> > --
> > Robert Scussel
> > 1024D/BAF70959/0036 B19E 86CE 181D 0912  5FCC 92D8 1EA1 BAF7 0959
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> > in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Pablo Borges                                pablo.borges@uol.com.br
> -------------------------------------------------------------------
>   ____                                               Tecnologia UOL
>  /    \    Debian:
>  |  =_/      The 100% suck free linux distro.
>   \
>     \      SETI is lame. http://www.distributed.net
>                                                      Dnetc is XNUG!
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
Robert Scussel
1024D/BAF70959/0036 B19E 86CE 181D 0912  5FCC 92D8 1EA1 BAF7 0959

Re: linux-2.4.13 high SWAP

[Theo Schlossnagle]

On Tuesday, October 30, 2001, at 10:10  AM, Andre Margis wrote:
> I test 2.4.9 , 2.4.10-ac7, 2.4.13 and all have this problem, I'm not 
> using
> XFS, but reiserfs with LVM and 4 GB RAM. I detected if use tmpfs the 
> kswapd
> eat my all CPU's, in 2.4.13 the system hang after a time. Now I'm 
> testing
> 2.4.13-ac3 without tmpfs and he is very better than the others 
> versions. But
> a nice test is disable the HIGHMEM support. I have a machine with 1GB 
> RAM and
> the system is very fine and stable, running 2.4.10-ac7.

To contrast.  We have had the best success with Linux 2.4.7 with HIGHMEM 
support enabled.

I see several people having similar symptoms and everyone is point 
fingers at tmpfs or ramfs or xfs (I pointed my finger there 
originally).  It seems to me that it is something more fundamental and 
that particular usage patterns are triggering this.  I have around 20 
different dual processor machines that run 2.4.7-xfs, 2.4.2-xfs, or 
2.4.12-xfs and I can only replicate it when I put heavy filesystem 
activity on the machine.

Every time I replicate the problem, I become less certain of the actual 
cause!  I have kdb and SysReq enabled on all these boxes and it doesn't 
cause a console freeze, so if some extra info would be helpful, let me 
know and I will gather it during the next "glitch."


I am confident that the problem, on triggered, directly effects all 
filesystem code.  Once I have an "unkillable" process scheduled, it is 
CPU bound and all filesystem sync operations fail.  The sync command 
never returns, reboot won't work without (-n) and all my mailers (exim 
in this case) freeze on their fsync calls.

The _REALLY_ bad thing here (other than suspended disk I/O and the cruel 
reboot) is that all file blocks that are modified after this "glitch" 
are full of \000's upon reboot.  A good portion of my mail spool is 
corrupted as well as other files.  I don't know if this is specific to 
only journalled file systems as I cannot afford the downtime of 
replicating this problem on a non-journalled fs.

--
Theo Schlossnagle
1024D/82844984/95FD 30F1 489E 4613 F22E  491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA  3D 90 B9 9F BE 27 24 E7