Re: why swap at all?

Re: why swap at all?

[Andy Lutomirski]

Nick Piggin wrote:
> Tom Felker wrote:
> 
>> On Wednesday 26 May 2004 7:37 am, Matthias Schniedermeyer wrote:
>>
>>
>>> program to kernel: "i read ONCE though this file caching not useful".
>>
>>
>>
>> Very true.  The system is based on the assumption that just-used pages 
>> are more useful that older pages, and it slows when this isn't true.  
>> We need ways to tell the kernel whether the assumption holds.
>>
> 
> A streaming flag is great, but we usually do OK without it. There
> is a "used once" heuristic that often gets it right as far as I
> know. Basically, new pages that are only used once put almost zero
> pressure on the rest of the memory.

(Disclaimer: I don't know all that much about the current scheme.)

First, I don't believe this works.  A couple weeks ago I did

# cp -a <~100GB> <different physical disk>

and my system was nearly unusable for a few hours.  This is Athlon 64 
3200+, 512MB RAM, DMA on on both drives, iowait time around 90%.  So this 
was an io/pagecache problem.

The benchmark involved was ls.  It took several seconds.  If I ran it again 
in 5 seconds or so, it was fine.  Much longer and it would take several 
seconds again.  Sounds like pages getting evicted in LRU order.

I have this problem not only on every linux kernel I've ever tried (on 
different computers) but on other OS's as well.  It's not an easy one to solve.

For kicks, I checked out vmstat 1 (I don't have a copy right of the 
output).  It looked like cp -a dirtied pages as long as it could get them, 
and they got written out as quickly as they could.  And, for whatever 
reason, the writes lag behind the reads by an amount comparable to the size 
of my physical memory.

It seems like some kind of limiting/balancing of what gets to use the cache 
is needed.  I bet that most workloads that touch data much larger than RAM 
don't benefit that much from caching it all.  (Yes, that kernel-tree-grep 
from cache is nice, but having glibc in cache is also nice.)

Should there be something like a (small) limit to how many dirty, 
non-mmaped pages a task can have?  I have no objection to a program taking 
longer to finish because the 100MB it writes need to mostly hit the platter 
before it returns, since, in return, I get a usable system while it's 
running and it's not taking any more CPU time.

Second (IMHO) a "used once" heuristic has a fundamental problem:

If there are more pages "used more than once" _in roughly sequential order_ 
than fit in memory, then trying to cache them all is absurd.  That is, if 
some program makes _multiple passes_ over that 100GB (mkisofs?), the system 
should never try to cache it all.  It would be better off taking a guess 
(even a wild-ass-guess) of which 200MB to cache plus a few MB for 
readahead, leaving pages from other programs in cache for more than a few 
seconds, and probably getting better performance (i.e. those 200MB are at 
least cached next time around).


Is any of this reasonable?

--Andy

Re: why swap at all?

[Denis Vlasenko]

On Thursday 27 May 2004 14:39, Andy Lutomirski wrote:
> Nick Piggin wrote:
> > Tom Felker wrote:
> >> On Wednesday 26 May 2004 7:37 am, Matthias Schniedermeyer wrote:
> >>> program to kernel: "i read ONCE though this file caching not useful".
> >>
> >> Very true.  The system is based on the assumption that just-used pages
> >> are more useful that older pages, and it slows when this isn't true.
> >> We need ways to tell the kernel whether the assumption holds.
> >
> > A streaming flag is great, but we usually do OK without it. There
> > is a "used once" heuristic that often gets it right as far as I
> > know. Basically, new pages that are only used once put almost zero
> > pressure on the rest of the memory.
>
> (Disclaimer: I don't know all that much about the current scheme.)
>
> First, I don't believe this works.  A couple weeks ago I did
>
> # cp -a <~100GB> <different physical disk>
>
> and my system was nearly unusable for a few hours.  This is Athlon 64
> 3200+, 512MB RAM, DMA on on both drives, iowait time around 90%.  So this
> was an io/pagecache problem.
>
> The benchmark involved was ls.  It took several seconds.  If I ran it again
> in 5 seconds or so, it was fine.  Much longer and it would take several
> seconds again.  Sounds like pages getting evicted in LRU order.

By what magic system can know that you are going to do ls again
in 2 minutes?

Does is happen if you do ls several times in a row (to make needed pages
not-once-used), then wait a bit and do ls again?

> I have this problem not only on every linux kernel I've ever tried (on
> different computers) but on other OS's as well.  It's not an easy one to
> solve.
>
> For kicks, I checked out vmstat 1 (I don't have a copy right of the
> output).  It looked like cp -a dirtied pages as long as it could get them,
> and they got written out as quickly as they could.  And, for whatever
> reason, the writes lag behind the reads by an amount comparable to the size
> of my physical memory.

cp should use fadvise() and say that it _really_ does not need those pages.

> It seems like some kind of limiting/balancing of what gets to use the cache
> is needed.  I bet that most workloads that touch data much larger than RAM
> don't benefit that much from caching it all.  (Yes, that kernel-tree-grep
> from cache is nice, but having glibc in cache is also nice.)
>
> Should there be something like a (small) limit to how many dirty,
> non-mmaped pages a task can have?  I have no objection to a program taking
> longer to finish because the 100MB it writes need to mostly hit the platter
> before it returns, since, in return, I get a usable system while it's
> running and it's not taking any more CPU time.
>
> Second (IMHO) a "used once" heuristic has a fundamental problem:
>
> If there are more pages "used more than once" _in roughly sequential order_
> than fit in memory, then trying to cache them all is absurd.  That is, if
> some program makes _multiple passes_ over that 100GB (mkisofs?), the system
> should never try to cache it all.  It would be better off taking a guess
> (even a wild-ass-guess) of which 200MB to cache plus a few MB for
> readahead, leaving pages from other programs in cache for more than a few
> seconds, and probably getting better performance (i.e. those 200MB are at
> least cached next time around).

Easier said than done... Why 200MB and not 400? etc...

> Is any of this reasonable?

If you think you see VM misbehavior,
1) verify that it is indeed MISbehaving
2) produce useful bug report
3) report it and track it until fixed

Apps take ages to start after cache being trashed because they are bloated.
Fight bloat. Join uclibc/dietlibc/etc efforts.

Random example: why on earth ntpd daemon have RSS of ~1.5 Mb???!
--
vda

Re: why swap at all?

[Bernd Eckenfels]

In article <200405290037.17775.vda@port.imtp.ilyichevsk.odessa.ua> you wrote:
>> The benchmark involved was ls.  It took several seconds.  If I ran it again
>> in 5 seconds or so, it was fine.  Much longer and it would take several
>> seconds again.  Sounds like pages getting evicted in LRU order.
> 
> By what magic system can know that you are going to do ls again
> in 2 minutes?

The problem is more about the blocks cp touches, less  about predicting the ls workload.

> cp should use fadvise() and say that it _really_ does not need those pages.

Yes, indeed. On the other hand the sequential read could be detected by the kernel, too.

Greetings
Bernd
-- 
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Re: why swap at all?

[Denis Vlasenko]

On Saturday 29 May 2004 01:28, Bernd Eckenfels wrote:
> In article <200405290037.17775.vda@port.imtp.ilyichevsk.odessa.ua> you wrote:
> >> The benchmark involved was ls.  It took several seconds.  If I ran it
> >> again in 5 seconds or so, it was fine.  Much longer and it would take
> >> several seconds again.  Sounds like pages getting evicted in LRU order.
> >
> > By what magic system can know that you are going to do ls again
> > in 2 minutes?
>
> The problem is more about the blocks cp touches, less  about predicting the
> ls workload.
>
> > cp should use fadvise() and say that it _really_ does not need those
> > pages.
>
> Yes, indeed. On the other hand the sequential read could be detected by the
> kernel, too.

Looks like it was. ls' read was sequential, too, so it did not get any
advantage. If you can definitely show that streaming io
(say, cat hugefile >/dev/null) flushes _non_ sequentially read data
(pages with program/library code, data of e.g. your Mozilla, etc),
please submit a report to lkml. VM gurus said more than once
that they _want_ to fix things, but need to know how to reproduce.
--
vda

MM patches (was Re: why swap at all?)

[Nick Piggin]

Denis Vlasenko wrote:

> (pages with program/library code, data of e.g. your Mozilla, etc),
> please submit a report to lkml. VM gurus said more than once
> that they _want_ to fix things, but need to know how to reproduce.

Yep.

Thanks to everyone's input I was able to test and adapt my mm work.
It is hopefully at a stage where it can have wider testing now. It
is stable on my SMP system under very heavy swapping, but the usual
caution applies.

Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X,
with xterms and mozilla open browsing and grepping kernel tree, etc.

Plain 2.6.7-rc1-mm1 swapped 200MB then completely froze up the system
within 9 seconds of starting the read load. Things remained fairly
responsive with my patch applied. A bit of swap out, but very little
swap in, which is good. The entire 32GB went through the pagecache no
problem.

A couple of concurrent mkisofs's writing 4 GB isos don't seem to be
any problem either with the patched kernel. Haven't tried plain -mm
yet.

http://www.kerneltrap.org/~npiggin/nickvm-267r1m1.gz

It is a cocktail of cleanups, simplification, and enhancements. The
main ones that applie here is my split active lists patch (search
archives for details), and explicit use-once logic.

Known issue: page reclaim can get a little bit lumpy (ie lots of
memory freed up at once), but that is just a matter of teaching
things not to bite off massive chunks at a time when it starts
hitting memory pressure.

Re: MM patches (was Re: why swap at all?)

[Nick Piggin]

Nick Piggin wrote:
> Denis Vlasenko wrote:
> 
>> (pages with program/library code, data of e.g. your Mozilla, etc),
>> please submit a report to lkml. VM gurus said more than once
>> that they _want_ to fix things, but need to know how to reproduce.
> 
> 
> Yep.
> 
> Thanks to everyone's input I was able to test and adapt my mm work.
> It is hopefully at a stage where it can have wider testing now. It
> is stable on my SMP system under very heavy swapping, but the usual
> caution applies.
> 
> Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X,
> with xterms and mozilla open browsing and grepping kernel tree, etc.
> 

This isn't the "very heavy swapping" load, BTW :)

The very heavy swapping load is make -j15 in 64MB of memory.

Re: MM patches (was Re: why swap at all?)

[Nick Piggin]

Matt wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Nick,
> 
> Your patch has worked great, and has significantly made my desktop pc more 
> responsive . 
> 
> If you happen to have any newer versions of your patch , I can test them out 
> and send feedback to you. Or I can provide you with a server to use to 
> benchmark and toy with. 
> 

Thanks for testing Matt. The patch I posted is the newest rollup
I have. I haven't run into any significant problems with it yet,
but I will notify lkml if I have found a bad bug or made significant
changes.

Nick

Re: MM patches (was Re: why swap at all?)

[Tvrtko A. Uršulin]

On Saturday 29 May 2004 10:40, Nick Piggin wrote:

> It is a cocktail of cleanups, simplification, and enhancements. The
> main ones that applie here is my split active lists patch (search
> archives for details), and explicit use-once logic.

I didn't have time to personally test it but just want to share some thoughts. 
This kind of improvement is absolutely necessary for 2.6 to be usefull on the 
desktop. It continues to escape me how come that this kind of, almost, bugfix 
arrives so late during stable period.

Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM from 
now on) which I am currently testing. AM also does an excellent job in 
preventing swap trashing while doing a lot of filesystem reading.

I am curious on how does your patch technically relates to Con's AM, and is it 
possible to merge the two? I know that they do their job on completely 
different ways, so the real question would be: Is there a need for something 
like AM if using your patch, or it completely obsoletes it?

Re: MM patches (was Re: why swap at all?)

[Con Kolivas]

On Mon, 31 May 2004 23:13, Tvrtko A. Uršulin wrote:
> On Saturday 29 May 2004 10:40, Nick Piggin wrote:
> > It is a cocktail of cleanups, simplification, and enhancements. The
> > main ones that applie here is my split active lists patch (search
> > archives for details), and explicit use-once logic.
>
> I didn't have time to personally test it but just want to share some
> thoughts. This kind of improvement is absolutely necessary for 2.6 to be
> usefull on the desktop. It continues to escape me how come that this kind
> of, almost, bugfix arrives so late during stable period.
>
> Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM
> from now on) which I am currently testing. AM also does an excellent job in
> preventing swap trashing while doing a lot of filesystem reading.
>
> I am curious on how does your patch technically relates to Con's AM, and is
> it possible to merge the two? I know that they do their job on completely
> different ways, so the real question would be: Is there a need for
> something like AM if using your patch, or it completely obsoletes it?

I had a quick look at Nick's patches to see for you and it appears that Nick's 
work completely removes the swappiness dial and introduces a different metric 
called vm_mapped_page_cost which is not interchangeable with the swappiness 
value. ie they cannot work together, sorry.

Con

Re: MM patches (was Re: why swap at all?)

[Andy Lutomirski]

Nick Piggin wrote:

> Yep.
> 
> Thanks to everyone's input I was able to test and adapt my mm work.
> It is hopefully at a stage where it can have wider testing now. It
> is stable on my SMP system under very heavy swapping, but the usual
> caution applies.

Thanks!

This feels quite a bit better on my system.  I'll try and stress it a bit 
more later today or tomorrow, but my system is now usable under heavy io load.

--Andy

Re: why swap at all?

[jlnance]

> > cp should use fadvise() and say that it _really_ does not need those pages.
> 
> Yes, indeed. On the other hand the sequential read could be detected by the kernel, too.

I'm not sure.  Copying a file is a pretty good indication that you
are about to do something with either the new or the old file.

Thanks,

Jim

Re: why swap at all?

[Lenar Lõhmus]

jlnance@unity.ncsu.edu wrote:

>I'm not sure.  Copying a file is a pretty good indication that you
>are about to do something with either the new or the old file.
>
>  
>
Like taking the new file with me on USB dongle and deleting old one? 
Caching the file really doesn't help in this case.

Lenar

Re: why swap at all?

[Robin Rosenberg]

On Tuesday 01 June 2004 13.57, Lenar Lõhmus wrote:
> jlnance@unity.ncsu.edu wrote:
> >I'm not sure.  Copying a file is a pretty good indication that you
> >are about to do something with either the new or the old file.
>
> Like taking the new file with me on USB dongle and deleting old one?
> Caching the file really doesn't help in this case.

No, and most file copies are not to be used in the "near" future. I.e. on
my machine. Caching on the second read (close in time) is ok, or if there
are unused ram, but paging out things in use is bad. It's much more likely
that the page allocated to a program will be used than a newly read or written 
file.

Ofcourse your milega may vary. I'm thinking of my desktop now.

-- robin

Re: why swap at all?

[jlnance]

On Tue, Jun 01, 2004 at 02:57:00PM +0300, Lenar L?hmus wrote:
> jlnance@unity.ncsu.edu wrote:
> 
> >I'm not sure.  Copying a file is a pretty good indication that you
> >are about to do something with either the new or the old file.
> >
> Like taking the new file with me on USB dongle and deleting old one? 
> Caching the file really doesn't help in this case.

No, it does not help in this case.

Not putting things in cache is a solution for the problem of
having useful stuff pushed out of the cache.  However, fixing
the problem this way may create other problems if it causes
us to fail to put useful things into the cache.

The point I was trying (perhaps unsuccessfully) to make, is
that we should be careful about not caching things.  We are
likely to break other corner cases by fixing the ones we
are discussing.

Thanks,

Jim

Re: why swap at all? (what the user feels)

[Martin Olsson]

Hi,

 From a pure *users* perspective the most evil part of my computer is 
the harddrive, because it has three bad properties;

-- its slowing stuff down
-- its very noisy
-- its sucks my battery dry (on laptops)

I dont care as much about how fast or efficient the swapping system 
really is, what does annoy me is when it lets me down. Say I do some 
action repeatedly every once in a while and it always takes about X 
milliseconds, when after a while that same action takes 2*X 
milliseconds. Now I'm DISAPPOINTED and I got a grudge with kswapd.

Its *emotionally* more 'okay' for the harddrive to its thing when I 
initiate it through some action (launching program, copying files) 
because then I expect it.

 From what I've read previously in this thread, it seems to me that the 
only major problem with swapping that not all users want file system 
cache to swap out actual applications (thus making that somewhat aged 
mozilla window abit laggy).

Maybe we could just have a "Allow file system cache to swap out 
applications checkbox somewhere"?

Or, Am I missing something?




Sincerly,
/m

jlnance@unity.ncsu.edu wrote:
> On Tue, Jun 01, 2004 at 02:57:00PM +0300, Lenar L?hmus wrote:
> 
>>jlnance@unity.ncsu.edu wrote:
>>
>>
>>>I'm not sure.  Copying a file is a pretty good indication that you
>>>are about to do something with either the new or the old file.
>>>
>>
>>Like taking the new file with me on USB dongle and deleting old one? 
>>Caching the file really doesn't help in this case.
> 
> 
> No, it does not help in this case.
> 
> Not putting things in cache is a solution for the problem of
> having useful stuff pushed out of the cache.  However, fixing
> the problem this way may create other problems if it causes
> us to fail to put useful things into the cache.
> 
> The point I was trying (perhaps unsuccessfully) to make, is
> that we should be careful about not caching things.  We are
> likely to break other corner cases by fixing the ones we
> are discussing.
> 
> Thanks,
> 
> Jim
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: why swap at all? (what the user feels)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

On Tue, 01 Jun 2004 19:38:54 +0200, Martin Olsson <mnemo@minimum.se>  said:

> Maybe we could just have a "Allow file system cache to swap out 
> applications checkbox somewhere"?
> 
> Or, Am I missing something?

Isn't that what /proc/sys/vm/swappiness is for? (And yes, I know that it's not
a total solution to the whole range of conflicting requirements that we have -
I suspect that it's really difficult to make it work much better than it is
without adding a lot more history-tracking instrumentation).


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

RE: why swap at all? (what the user feels)

[David Schwartz]

>  From what I've read previously in this thread, it seems to me that the
> only major problem with swapping that not all users want file system
> cache to swap out actual applications (thus making that somewhat aged
> mozilla window abit laggy).
>
> Maybe we could just have a "Allow file system cache to swap out
> applications checkbox somewhere"?
>
> Or, Am I missing something?

	In practice, that would make no difference at all. Once physical memory is
full (and it pretty much will always be so), every memory request (whether
due to the file system cache or application usage) will require discarding
some page or other. So even if all memory requests due to file system cache
usage were prohibited from forcing out application pages, you're launching
enough other application that need pages that application pages will still
be evicted.

	Now, if you make the rule "don't ever swap out application pages", what
exactly is the swap going to do? Swap is for dirty pages. Dirty file pages
get written back to their ultimate home, not swap.

	DS

Re: why swap at all? (what the user feels)

[Robin Rosenberg]

On Tuesday 01 June 2004 20.01, David Schwartz wrote:
> >  From what I've read previously in this thread, it seems to me that the
> > only major problem with swapping that not all users want file system
> > cache to swap out actual applications (thus making that somewhat aged
> > mozilla window abit laggy).
> >
> > Maybe we could just have a "Allow file system cache to swap out
> > applications checkbox somewhere"?
> >
> > Or, Am I missing something?
>
> 	In practice, that would make no difference at all. Once physical memory is
> full (and it pretty much will always be so), every memory request (whether

No. 

Many people have machines with plenty of RAM (512MB or more is pretty much 
standard on new machines), much of which is only used to cache files. The 
file cache is the reason the memory is full.

-- robin

RE: why swap at all? (what the user feels)

[David Schwartz]

> On Tuesday 01 June 2004 20.01, David Schwartz wrote:
> > >  From what I've read previously in this thread, it seems to
> > > me that the
> > > only major problem with swapping that not all users want file system
> > > cache to swap out actual applications (thus making that somewhat aged
> > > mozilla window abit laggy).
> > >
> > > Maybe we could just have a "Allow file system cache to swap out
> > > applications checkbox somewhere"?
> > >
> > > Or, Am I missing something?

> > 	In practice, that would make no difference at all. Once
> > physical memory is
> > full (and it pretty much will always be so), every memory
> request (whether

> No.

	Huh?

> Many people have machines with plenty of RAM (512MB or more is
> pretty much
> standard on new machines), much of which is only used to cache files. The
> file cache is the reason the memory is full.

	Of course. That's why I said, "once physical memory is full (and it pretty
much will always be so)". Physical memory is always full, so every memory
request requires that a page be evicted.

	DS

Re: why swap at all?

[John Hendrikx]

jlnance@unity.ncsu.edu wrote:

>On Tue, Jun 01, 2004 at 02:57:00PM +0300, Lenar L?hmus wrote:
>  
>
>>jlnance@unity.ncsu.edu wrote:
>>
>>    
>>
>>>I'm not sure.  Copying a file is a pretty good indication that you
>>>are about to do something with either the new or the old file.
>>>
>>>      
>>>
>>Like taking the new file with me on USB dongle and deleting old one? 
>>Caching the file really doesn't help in this case.
>>    
>>
>
>No, it does not help in this case.
>
>Not putting things in cache is a solution for the problem of
>having useful stuff pushed out of the cache.  However, fixing
>the problem this way may create other problems if it causes
>us to fail to put useful things into the cache.
>
>The point I was trying (perhaps unsuccessfully) to make, is
>that we should be careful about not caching things.  We are
>likely to break other corner cases by fixing the ones we
>are discussing.
>  
>
I've experienced the problem where applications need to be swapped back 
in.  It's mainly caused by the dual role my machine has (desktop machine 
when I'm using it, server when it is serving files).   Whenever my 
machine has been sitting idly serving files for a while, when I get 
back, the desktop is slow.  However, there is no need for that, as the 
files are served at low speeds -- there's no real point in caching them 
apart from maybe preventing harddisk wear... the harddisk itself can 
serve these files again faster than they will be needed.

So perhaps it is possible to reduce caching of data that is simply not 
putting stress on the system (the harddisk in this case).  If the 
harddisk is not the bottleneck, it is probably not worth caching.  
Typical examples are letting a box play music all day (and then trying 
to read your mail...), having a webserver on a slow connection or 
watching a large movie file.  None of these really require much caching 
beyond a bit of read-ahead. 

I'm not sure how best to distinguish when something is fast I/O that 
would benefit from caching and when something is slow I/O that the 
harddisk can handle well enough on its own.

--John

Re: why swap at all?

[David B. Stevens]

jlnance@unity.ncsu.edu wrote:
>>>cp should use fadvise() and say that it _really_ does not need those pages.
>>
>>Yes, indeed. On the other hand the sequential read could be detected by the kernel, too.
> 
> 
> I'm not sure.  Copying a file is a pretty good indication that you
> are about to do something with either the new or the old file.
> 
> Thanks,
> 
> Jim

It is?

Sorry for butting in folks, but I've been reading this thread hoping to 
see some possible solutions.  Seems that a survey of best practices 
might have been suggested, however, I haven't seen such a suggestion.

So here goes, might it not be of some benefit to see how other operating 
systems (there are rather large number) handle the use  of memory.  For 
just one example you could look at MVS, where the application can 
request through various means how and how much memory it uses.  Most 
defaults can be overridden by the scripting language used to run the 
application.  This also true of other operating systems.

I would be more willing to say that the folks setting up the running of 
systems should have far more control over the use or non use of cache 
backed I/O data.

Now that I've said that you have to consider how and where this control 
should be based.

<soapbox>

SWAP is a solution for the age old whine, I caused the system to run out 
of memory and the big mean operating system terminated my application.

These days it allows the performance of the system to degrade to the 
point that the whine goes, The big mean operating system is taking 
forever to run my 10 TB backup and, by the way, it takes 3 days to wake 
up my openoffice application that I started a week ago.

Ain't progress grand ;-)

</soapbox>

Cheers,
   Dave