excessive swapping

excessive swapping

[Florin Andrei]

I am running 2.6.8-rc4 with Ingo's voluntary preempt patch O5, on Fedora
2.
I'm using the default Gnome environment, reading mail with Evolution,
browsing with Firefox, etc. I've 512MB of RAM.

At the same time, i'm processing some DVDs that i made - i'm extracting
titles from a DVD to a dedicated hard-drive, saving audio and video
tracks, etc with transcode-0.6.12 ( http://www.transcoding.org ). All
that means reading/writing from/to large files on /dev/dvd and /dev/hde
at high speeds.

The system is swapping excessively. There's no way the total size of the
applications exceeds the size of RAM. There's plenty of room to spare,
yet 16% of the 530MB of swap is used.
The system's responsiveness as a desktop is very poor. If i touch an
application, the drive thrashes raising it from the dead^H^H^H^H swap.

With the Fedora 2 kernel, this never happens. I did exactly the same
things, and there was no excessive swapping. The system feels very
responsive and fast.

I'll try to add Con Kolivas' hard swappiness patch to my customized
kernel and see if that improves things.

In any case, the current behaviour is unacceptable.

-- 
Florin Andrei

http://florin.myip.org/

Re: excessive swapping

[Florin Andrei]

On Thu, 2004-08-12 at 23:40, Florin Andrei wrote:

> The system is swapping excessively. There's no way the total size of the
> applications exceeds the size of RAM. There's plenty of room to spare,
> yet 16% of the 530MB of swap is used.

Now it's 22% and counting. Way to go. :-(

> With the Fedora 2 kernel, this never happens. I did exactly the same
> things, and there was no excessive swapping. The system feels very
> responsive and fast.

In such situations, there is 0% swapping with the FC2 kernel.

-- 
Florin Andrei

http://florin.myip.org/

Re: excessive swapping

[Florin Andrei]

On Thu, 2004-08-12 at 23:44, Florin Andrei wrote:
> On Thu, 2004-08-12 at 23:40, Florin Andrei wrote:
> 
> > The system is swapping excessively. There's no way the total size of the
> > applications exceeds the size of RAM. There's plenty of room to spare,
> > yet 16% of the 530MB of swap is used.
> 
> Now it's 22% and counting. Way to go. :-(

Now it's 27%. You get the picture.

Anyway, out of the 512MB of RAM, like 390MB are disk cache. No wonder
that the useful pages are swaped out.

It seems like the kernel believes that the disk cache has some
miraculous properties w.r.t. the system performance, and desperately
tries to grow it as much as possible.
This is wrong religion. The reality is opposite. The system is much
slower, because applications are thrown out in the swap, then sucked
back in, which is a very slow process.

The efficiency of increasing the disk cache decreases exponentially with
size, like any other cache. Then what's the point of sacrificing useful
memory just to increase some hypothetical "useful" cache?

Even on a server, the same universal laws still apply, the efficiency of
increasing cache still decreases exponentially. There's still precious
time wasted when an application is sucked back in from swap, at the
price of an immeasurably small performance gain due to the disk cache
being larger.

I'm sorry for rambling, but to me the current swapping policy is so
blatantly wrong. Besides the space occupied by the apps themselves,
there's a lot of room to _spare_ - then why swap?

-- 
Florin Andrei

http://florin.myip.org/

dvd ripping causing: Re: excessive swapping

[bert hubert]

On Fri, Aug 13, 2004 at 12:10:37AM -0700, Florin Andrei wrote:

> I'm sorry for rambling, but to me the current swapping policy is so
> blatantly wrong. Besides the space occupied by the apps themselves,
> there's a lot of room to _spare_ - then why swap?

It is a bug - calm down. It has been reported before but now quite hammered
down yet.

Can you report the files I asked you for? Could you also try:
echo 0 > /proc/sys/vm/swappiness 

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: excessive swapping

[Matt Heler]

Can you try using Nick Piggin's np patchset ? 

Matt H.

On Friday 13 August 2004 12:10 am, Florin Andrei wrote:
> On Thu, 2004-08-12 at 23:44, Florin Andrei wrote:
> > On Thu, 2004-08-12 at 23:40, Florin Andrei wrote:
> > > The system is swapping excessively. There's no way the total size of
> > > the applications exceeds the size of RAM. There's plenty of room to
> > > spare, yet 16% of the 530MB of swap is used.
> >
> > Now it's 22% and counting. Way to go. :-(
>
> Now it's 27%. You get the picture.
>
> Anyway, out of the 512MB of RAM, like 390MB are disk cache. No wonder
> that the useful pages are swaped out.
>
> It seems like the kernel believes that the disk cache has some
> miraculous properties w.r.t. the system performance, and desperately
> tries to grow it as much as possible.
> This is wrong religion. The reality is opposite. The system is much
> slower, because applications are thrown out in the swap, then sucked
> back in, which is a very slow process.
>
> The efficiency of increasing the disk cache decreases exponentially with
> size, like any other cache. Then what's the point of sacrificing useful
> memory just to increase some hypothetical "useful" cache?
>
> Even on a server, the same universal laws still apply, the efficiency of
> increasing cache still decreases exponentially. There's still precious
> time wasted when an application is sucked back in from swap, at the
> price of an immeasurably small performance gain due to the disk cache
> being larger.
>
> I'm sorry for rambling, but to me the current swapping policy is so
> blatantly wrong. Besides the space occupied by the apps themselves,
> there's a lot of room to _spare_ - then why swap?

Re: excessive swapping

[Alan Cox]

On Gwe, 2004-08-13 at 08:10, Florin Andrei wrote:
> The efficiency of increasing the disk cache decreases exponentially with
> size, like any other cache. Then what's the point of sacrificing useful
> memory just to increase some hypothetical "useful" cache?

The problem is defining "useful". Pieces of applications not being
run at the moment are also not useful. Balancing them isn't easy 
because as you say the cache behaviour is exponential _and_ the two
caches have different behaviours when you get blocking (program paging
is generally random so 30+ times slower than streaming bits). Plus there
is user perception.

The latest code is clearly wrong but it isn't simple to get the balance
right either.

Re: excessive swapping

[bert hubert]

On Thu, Aug 12, 2004 at 11:40:51PM -0700, Florin Andrei wrote:
> I am running 2.6.8-rc4 with Ingo's voluntary preempt patch O5, on Fedora
> 2.
> I'm using the default Gnome environment, reading mail with Evolution,
> browsing with Firefox, etc. I've 512MB of RAM.

Please see if you can reproduce this behaviour without any additional
patches. Furthermore, when this has happened, please show the output of cat
/proc/meminfo and cat /proc/slabinfo and your .config.

> The system is swapping excessively. There's no way the total size of the
> applications exceeds the size of RAM. There's plenty of room to spare,
> yet 16% of the 530MB of swap is used.

Please show a few lines of vmstat 1 when this happens.

Thanks!

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: excessive swapping

[Florin Andrei]

On Thu, 2004-08-12 at 23:40, Florin Andrei wrote:
> I am running 2.6.8-rc4 with Ingo's voluntary preempt patch O5, on Fedora
> 2.
> At the same time, i'm processing some DVDs that i made - i'm extracting
> titles from a DVD to a dedicated hard-drive, saving audio and video
> tracks, etc with transcode-0.6.12 ( http://www.transcoding.org ). All
> that means reading/writing from/to large files on /dev/dvd and /dev/hde
> at high speeds.
> The system is swapping excessively. There's no way the total size of the
> applications exceeds the size of RAM. There's plenty of room to spare,
> yet 16% of the 530MB of swap is used.

I am running now 2.6.8.1 with Ingo's O8 and Con Kolivas' hard swappiness
patch (an old version, sorry). Under the same conditions described
above, there is 0% swap usage.

The CK hard swappiness patch solved the issue for me. The system works
without any problems whatsoever.

-- 
Florin Andrei

http://florin.myip.org/

Re: excessive swapping

[Marcelo Tosatti]

Florin,

You should get much better behaviour in 2.6.8-rc2-mm4 due
to Nick's kswapd "aggressiveness" fix.

Please try it!

On Wed, Sep 01, 2004 at 11:02:32PM -0700, Florin Andrei wrote:
> On Thu, 2004-08-12 at 23:40, Florin Andrei wrote:
> > I am running 2.6.8-rc4 with Ingo's voluntary preempt patch O5, on Fedora
> > 2.
> > At the same time, i'm processing some DVDs that i made - i'm extracting
> > titles from a DVD to a dedicated hard-drive, saving audio and video
> > tracks, etc with transcode-0.6.12 ( http://www.transcoding.org ). All
> > that means reading/writing from/to large files on /dev/dvd and /dev/hde
> > at high speeds.
> > The system is swapping excessively. There's no way the total size of the
> > applications exceeds the size of RAM. There's plenty of room to spare,
> > yet 16% of the 530MB of swap is used.
> 
> I am running now 2.6.8.1 with Ingo's O8 and Con Kolivas' hard swappiness
> patch (an old version, sorry). Under the same conditions described
> above, there is 0% swap usage.
> 
> The CK hard swappiness patch solved the issue for me. The system works
> without any problems whatsoever.