* Re: why swap at all? [not found] ` <fa.bqpvcrs.u648jq@ifi.uio.no> @ 2004-05-27 11:39 ` Andy Lutomirski 2004-05-28 21:37 ` Denis Vlasenko 0 siblings, 1 reply; 22+ messages in thread From: Andy Lutomirski @ 2004-05-27 11:39 UTC (permalink / raw) To: Nick Piggin; +Cc: Tom Felker, Matthias Schniedermeyer, linux-kernel Nick Piggin wrote: > Tom Felker wrote: > >> On Wednesday 26 May 2004 7:37 am, Matthias Schniedermeyer wrote: >> >> >>> program to kernel: "i read ONCE though this file caching not useful". >> >> >> >> Very true. The system is based on the assumption that just-used pages >> are more useful that older pages, and it slows when this isn't true. >> We need ways to tell the kernel whether the assumption holds. >> > > A streaming flag is great, but we usually do OK without it. There > is a "used once" heuristic that often gets it right as far as I > know. Basically, new pages that are only used once put almost zero > pressure on the rest of the memory. (Disclaimer: I don't know all that much about the current scheme.) First, I don't believe this works. A couple weeks ago I did # cp -a <~100GB> <different physical disk> and my system was nearly unusable for a few hours. This is Athlon 64 3200+, 512MB RAM, DMA on on both drives, iowait time around 90%. So this was an io/pagecache problem. The benchmark involved was ls. It took several seconds. If I ran it again in 5 seconds or so, it was fine. Much longer and it would take several seconds again. Sounds like pages getting evicted in LRU order. I have this problem not only on every linux kernel I've ever tried (on different computers) but on other OS's as well. It's not an easy one to solve. For kicks, I checked out vmstat 1 (I don't have a copy right of the output). It looked like cp -a dirtied pages as long as it could get them, and they got written out as quickly as they could. And, for whatever reason, the writes lag behind the reads by an amount comparable to the size of my physical memory. It seems like some kind of limiting/balancing of what gets to use the cache is needed. I bet that most workloads that touch data much larger than RAM don't benefit that much from caching it all. (Yes, that kernel-tree-grep from cache is nice, but having glibc in cache is also nice.) Should there be something like a (small) limit to how many dirty, non-mmaped pages a task can have? I have no objection to a program taking longer to finish because the 100MB it writes need to mostly hit the platter before it returns, since, in return, I get a usable system while it's running and it's not taking any more CPU time. Second (IMHO) a "used once" heuristic has a fundamental problem: If there are more pages "used more than once" _in roughly sequential order_ than fit in memory, then trying to cache them all is absurd. That is, if some program makes _multiple passes_ over that 100GB (mkisofs?), the system should never try to cache it all. It would be better off taking a guess (even a wild-ass-guess) of which 200MB to cache plus a few MB for readahead, leaving pages from other programs in cache for more than a few seconds, and probably getting better performance (i.e. those 200MB are at least cached next time around). Is any of this reasonable? --Andy ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-05-27 11:39 ` why swap at all? Andy Lutomirski @ 2004-05-28 21:37 ` Denis Vlasenko 2004-05-28 22:28 ` Bernd Eckenfels 0 siblings, 1 reply; 22+ messages in thread From: Denis Vlasenko @ 2004-05-28 21:37 UTC (permalink / raw) To: Andy Lutomirski, Nick Piggin Cc: Tom Felker, Matthias Schniedermeyer, linux-kernel On Thursday 27 May 2004 14:39, Andy Lutomirski wrote: > Nick Piggin wrote: > > Tom Felker wrote: > >> On Wednesday 26 May 2004 7:37 am, Matthias Schniedermeyer wrote: > >>> program to kernel: "i read ONCE though this file caching not useful". > >> > >> Very true. The system is based on the assumption that just-used pages > >> are more useful that older pages, and it slows when this isn't true. > >> We need ways to tell the kernel whether the assumption holds. > > > > A streaming flag is great, but we usually do OK without it. There > > is a "used once" heuristic that often gets it right as far as I > > know. Basically, new pages that are only used once put almost zero > > pressure on the rest of the memory. > > (Disclaimer: I don't know all that much about the current scheme.) > > First, I don't believe this works. A couple weeks ago I did > > # cp -a <~100GB> <different physical disk> > > and my system was nearly unusable for a few hours. This is Athlon 64 > 3200+, 512MB RAM, DMA on on both drives, iowait time around 90%. So this > was an io/pagecache problem. > > The benchmark involved was ls. It took several seconds. If I ran it again > in 5 seconds or so, it was fine. Much longer and it would take several > seconds again. Sounds like pages getting evicted in LRU order. By what magic system can know that you are going to do ls again in 2 minutes? Does is happen if you do ls several times in a row (to make needed pages not-once-used), then wait a bit and do ls again? > I have this problem not only on every linux kernel I've ever tried (on > different computers) but on other OS's as well. It's not an easy one to > solve. > > For kicks, I checked out vmstat 1 (I don't have a copy right of the > output). It looked like cp -a dirtied pages as long as it could get them, > and they got written out as quickly as they could. And, for whatever > reason, the writes lag behind the reads by an amount comparable to the size > of my physical memory. cp should use fadvise() and say that it _really_ does not need those pages. > It seems like some kind of limiting/balancing of what gets to use the cache > is needed. I bet that most workloads that touch data much larger than RAM > don't benefit that much from caching it all. (Yes, that kernel-tree-grep > from cache is nice, but having glibc in cache is also nice.) > > Should there be something like a (small) limit to how many dirty, > non-mmaped pages a task can have? I have no objection to a program taking > longer to finish because the 100MB it writes need to mostly hit the platter > before it returns, since, in return, I get a usable system while it's > running and it's not taking any more CPU time. > > Second (IMHO) a "used once" heuristic has a fundamental problem: > > If there are more pages "used more than once" _in roughly sequential order_ > than fit in memory, then trying to cache them all is absurd. That is, if > some program makes _multiple passes_ over that 100GB (mkisofs?), the system > should never try to cache it all. It would be better off taking a guess > (even a wild-ass-guess) of which 200MB to cache plus a few MB for > readahead, leaving pages from other programs in cache for more than a few > seconds, and probably getting better performance (i.e. those 200MB are at > least cached next time around). Easier said than done... Why 200MB and not 400? etc... > Is any of this reasonable? If you think you see VM misbehavior, 1) verify that it is indeed MISbehaving 2) produce useful bug report 3) report it and track it until fixed Apps take ages to start after cache being trashed because they are bloated. Fight bloat. Join uclibc/dietlibc/etc efforts. Random example: why on earth ntpd daemon have RSS of ~1.5 Mb???! -- vda ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-05-28 21:37 ` Denis Vlasenko @ 2004-05-28 22:28 ` Bernd Eckenfels 2004-05-29 7:31 ` Denis Vlasenko 2004-05-31 10:49 ` why swap at all? jlnance 0 siblings, 2 replies; 22+ messages in thread From: Bernd Eckenfels @ 2004-05-28 22:28 UTC (permalink / raw) To: linux-kernel In article <200405290037.17775.vda@port.imtp.ilyichevsk.odessa.ua> you wrote: >> The benchmark involved was ls. It took several seconds. If I ran it again >> in 5 seconds or so, it was fine. Much longer and it would take several >> seconds again. Sounds like pages getting evicted in LRU order. > > By what magic system can know that you are going to do ls again > in 2 minutes? The problem is more about the blocks cp touches, less about predicting the ls workload. > cp should use fadvise() and say that it _really_ does not need those pages. Yes, indeed. On the other hand the sequential read could be detected by the kernel, too. Greetings Bernd -- eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-05-28 22:28 ` Bernd Eckenfels @ 2004-05-29 7:31 ` Denis Vlasenko 2004-05-29 8:40 ` MM patches (was Re: why swap at all?) Nick Piggin 2004-05-31 10:49 ` why swap at all? jlnance 1 sibling, 1 reply; 22+ messages in thread From: Denis Vlasenko @ 2004-05-29 7:31 UTC (permalink / raw) To: Bernd Eckenfels, linux-kernel On Saturday 29 May 2004 01:28, Bernd Eckenfels wrote: > In article <200405290037.17775.vda@port.imtp.ilyichevsk.odessa.ua> you wrote: > >> The benchmark involved was ls. It took several seconds. If I ran it > >> again in 5 seconds or so, it was fine. Much longer and it would take > >> several seconds again. Sounds like pages getting evicted in LRU order. > > > > By what magic system can know that you are going to do ls again > > in 2 minutes? > > The problem is more about the blocks cp touches, less about predicting the > ls workload. > > > cp should use fadvise() and say that it _really_ does not need those > > pages. > > Yes, indeed. On the other hand the sequential read could be detected by the > kernel, too. Looks like it was. ls' read was sequential, too, so it did not get any advantage. If you can definitely show that streaming io (say, cat hugefile >/dev/null) flushes _non_ sequentially read data (pages with program/library code, data of e.g. your Mozilla, etc), please submit a report to lkml. VM gurus said more than once that they _want_ to fix things, but need to know how to reproduce. -- vda ^ permalink raw reply [flat|nested] 22+ messages in thread
* MM patches (was Re: why swap at all?) 2004-05-29 7:31 ` Denis Vlasenko @ 2004-05-29 8:40 ` Nick Piggin 2004-05-29 8:46 ` Nick Piggin ` (3 more replies) 0 siblings, 4 replies; 22+ messages in thread From: Nick Piggin @ 2004-05-29 8:40 UTC (permalink / raw) To: Denis Vlasenko, Matthias Schniedermeyer; +Cc: Bernd Eckenfels, linux-kernel Denis Vlasenko wrote: > (pages with program/library code, data of e.g. your Mozilla, etc), > please submit a report to lkml. VM gurus said more than once > that they _want_ to fix things, but need to know how to reproduce. Yep. Thanks to everyone's input I was able to test and adapt my mm work. It is hopefully at a stage where it can have wider testing now. It is stable on my SMP system under very heavy swapping, but the usual caution applies. Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X, with xterms and mozilla open browsing and grepping kernel tree, etc. Plain 2.6.7-rc1-mm1 swapped 200MB then completely froze up the system within 9 seconds of starting the read load. Things remained fairly responsive with my patch applied. A bit of swap out, but very little swap in, which is good. The entire 32GB went through the pagecache no problem. A couple of concurrent mkisofs's writing 4 GB isos don't seem to be any problem either with the patched kernel. Haven't tried plain -mm yet. http://www.kerneltrap.org/~npiggin/nickvm-267r1m1.gz It is a cocktail of cleanups, simplification, and enhancements. The main ones that applie here is my split active lists patch (search archives for details), and explicit use-once logic. Known issue: page reclaim can get a little bit lumpy (ie lots of memory freed up at once), but that is just a matter of teaching things not to bite off massive chunks at a time when it starts hitting memory pressure. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: MM patches (was Re: why swap at all?) 2004-05-29 8:40 ` MM patches (was Re: why swap at all?) Nick Piggin @ 2004-05-29 8:46 ` Nick Piggin [not found] ` <200405292014.23559.matt@lpbproductions.com> ` (2 subsequent siblings) 3 siblings, 0 replies; 22+ messages in thread From: Nick Piggin @ 2004-05-29 8:46 UTC (permalink / raw) To: Denis Vlasenko, Matthias Schniedermeyer; +Cc: Bernd Eckenfels, linux-kernel Nick Piggin wrote: > Denis Vlasenko wrote: > >> (pages with program/library code, data of e.g. your Mozilla, etc), >> please submit a report to lkml. VM gurus said more than once >> that they _want_ to fix things, but need to know how to reproduce. > > > Yep. > > Thanks to everyone's input I was able to test and adapt my mm work. > It is hopefully at a stage where it can have wider testing now. It > is stable on my SMP system under very heavy swapping, but the usual > caution applies. > > Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X, > with xterms and mozilla open browsing and grepping kernel tree, etc. > This isn't the "very heavy swapping" load, BTW :) The very heavy swapping load is make -j15 in 64MB of memory. ^ permalink raw reply [flat|nested] 22+ messages in thread
[parent not found: <200405292014.23559.matt@lpbproductions.com>]
* Re: MM patches (was Re: why swap at all?) [not found] ` <200405292014.23559.matt@lpbproductions.com> @ 2004-05-30 3:31 ` Nick Piggin 0 siblings, 0 replies; 22+ messages in thread From: Nick Piggin @ 2004-05-30 3:31 UTC (permalink / raw) To: matt; +Cc: linux-kernel Matt wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Nick, > > Your patch has worked great, and has significantly made my desktop pc more > responsive . > > If you happen to have any newer versions of your patch , I can test them out > and send feedback to you. Or I can provide you with a server to use to > benchmark and toy with. > Thanks for testing Matt. The patch I posted is the newest rollup I have. I haven't run into any significant problems with it yet, but I will notify lkml if I have found a bad bug or made significant changes. Nick ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: MM patches (was Re: why swap at all?) 2004-05-29 8:40 ` MM patches (was Re: why swap at all?) Nick Piggin 2004-05-29 8:46 ` Nick Piggin [not found] ` <200405292014.23559.matt@lpbproductions.com> @ 2004-05-31 13:13 ` Tvrtko A. Uršulin 2004-05-31 13:33 ` Con Kolivas 2004-05-31 17:34 ` Andy Lutomirski 3 siblings, 1 reply; 22+ messages in thread From: Tvrtko A. Uršulin @ 2004-05-31 13:13 UTC (permalink / raw) To: linux-kernel; +Cc: Nick Piggin On Saturday 29 May 2004 10:40, Nick Piggin wrote: > It is a cocktail of cleanups, simplification, and enhancements. The > main ones that applie here is my split active lists patch (search > archives for details), and explicit use-once logic. I didn't have time to personally test it but just want to share some thoughts. This kind of improvement is absolutely necessary for 2.6 to be usefull on the desktop. It continues to escape me how come that this kind of, almost, bugfix arrives so late during stable period. Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM from now on) which I am currently testing. AM also does an excellent job in preventing swap trashing while doing a lot of filesystem reading. I am curious on how does your patch technically relates to Con's AM, and is it possible to merge the two? I know that they do their job on completely different ways, so the real question would be: Is there a need for something like AM if using your patch, or it completely obsoletes it? ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: MM patches (was Re: why swap at all?) 2004-05-31 13:13 ` Tvrtko A. Uršulin @ 2004-05-31 13:33 ` Con Kolivas 0 siblings, 0 replies; 22+ messages in thread From: Con Kolivas @ 2004-05-31 13:33 UTC (permalink / raw) To: Tvrtko A. Uršulin; +Cc: linux-kernel, Nick Piggin On Mon, 31 May 2004 23:13, Tvrtko A. Uršulin wrote: > On Saturday 29 May 2004 10:40, Nick Piggin wrote: > > It is a cocktail of cleanups, simplification, and enhancements. The > > main ones that applie here is my split active lists patch (search > > archives for details), and explicit use-once logic. > > I didn't have time to personally test it but just want to share some > thoughts. This kind of improvement is absolutely necessary for 2.6 to be > usefull on the desktop. It continues to escape me how come that this kind > of, almost, bugfix arrives so late during stable period. > > Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM > from now on) which I am currently testing. AM also does an excellent job in > preventing swap trashing while doing a lot of filesystem reading. > > I am curious on how does your patch technically relates to Con's AM, and is > it possible to merge the two? I know that they do their job on completely > different ways, so the real question would be: Is there a need for > something like AM if using your patch, or it completely obsoletes it? I had a quick look at Nick's patches to see for you and it appears that Nick's work completely removes the swappiness dial and introduces a different metric called vm_mapped_page_cost which is not interchangeable with the swappiness value. ie they cannot work together, sorry. Con ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: MM patches (was Re: why swap at all?) 2004-05-29 8:40 ` MM patches (was Re: why swap at all?) Nick Piggin ` (2 preceding siblings ...) 2004-05-31 13:13 ` Tvrtko A. Uršulin @ 2004-05-31 17:34 ` Andy Lutomirski 3 siblings, 0 replies; 22+ messages in thread From: Andy Lutomirski @ 2004-05-31 17:34 UTC (permalink / raw) To: Nick Piggin; +Cc: Bernd Eckenfels, linux-kernel Nick Piggin wrote: > Yep. > > Thanks to everyone's input I was able to test and adapt my mm work. > It is hopefully at a stage where it can have wider testing now. It > is stable on my SMP system under very heavy swapping, but the usual > caution applies. Thanks! This feels quite a bit better on my system. I'll try and stress it a bit more later today or tomorrow, but my system is now usable under heavy io load. --Andy ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-05-28 22:28 ` Bernd Eckenfels 2004-05-29 7:31 ` Denis Vlasenko @ 2004-05-31 10:49 ` jlnance 2004-06-01 11:57 ` Lenar Lõhmus 2004-06-01 12:21 ` David B. Stevens 1 sibling, 2 replies; 22+ messages in thread From: jlnance @ 2004-05-31 10:49 UTC (permalink / raw) To: linux-kernel > > cp should use fadvise() and say that it _really_ does not need those pages. > > Yes, indeed. On the other hand the sequential read could be detected by the kernel, too. I'm not sure. Copying a file is a pretty good indication that you are about to do something with either the new or the old file. Thanks, Jim ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-05-31 10:49 ` why swap at all? jlnance @ 2004-06-01 11:57 ` Lenar Lõhmus 2004-06-01 12:27 ` Robin Rosenberg 2004-06-01 16:49 ` jlnance 2004-06-01 12:21 ` David B. Stevens 1 sibling, 2 replies; 22+ messages in thread From: Lenar Lõhmus @ 2004-06-01 11:57 UTC (permalink / raw) To: Linux Kernel Mailinglist jlnance@unity.ncsu.edu wrote: >I'm not sure. Copying a file is a pretty good indication that you >are about to do something with either the new or the old file. > > > Like taking the new file with me on USB dongle and deleting old one? Caching the file really doesn't help in this case. Lenar ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-06-01 11:57 ` Lenar Lõhmus @ 2004-06-01 12:27 ` Robin Rosenberg 2004-06-01 16:49 ` jlnance 1 sibling, 0 replies; 22+ messages in thread From: Robin Rosenberg @ 2004-06-01 12:27 UTC (permalink / raw) To: Lenar Lõhmus; +Cc: Linux Kernel Mailinglist On Tuesday 01 June 2004 13.57, Lenar Lõhmus wrote: > jlnance@unity.ncsu.edu wrote: > >I'm not sure. Copying a file is a pretty good indication that you > >are about to do something with either the new or the old file. > > Like taking the new file with me on USB dongle and deleting old one? > Caching the file really doesn't help in this case. No, and most file copies are not to be used in the "near" future. I.e. on my machine. Caching on the second read (close in time) is ok, or if there are unused ram, but paging out things in use is bad. It's much more likely that the page allocated to a program will be used than a newly read or written file. Ofcourse your milega may vary. I'm thinking of my desktop now. -- robin ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-06-01 11:57 ` Lenar Lõhmus 2004-06-01 12:27 ` Robin Rosenberg @ 2004-06-01 16:49 ` jlnance 2004-06-01 17:38 ` why swap at all? (what the user feels) Martin Olsson 2004-06-02 18:38 ` why swap at all? John Hendrikx 1 sibling, 2 replies; 22+ messages in thread From: jlnance @ 2004-06-01 16:49 UTC (permalink / raw) To: Lenar =?unknown-8bit?Q?L=F5hmus?=; +Cc: Linux Kernel Mailinglist On Tue, Jun 01, 2004 at 02:57:00PM +0300, Lenar L?hmus wrote: > jlnance@unity.ncsu.edu wrote: > > >I'm not sure. Copying a file is a pretty good indication that you > >are about to do something with either the new or the old file. > > > Like taking the new file with me on USB dongle and deleting old one? > Caching the file really doesn't help in this case. No, it does not help in this case. Not putting things in cache is a solution for the problem of having useful stuff pushed out of the cache. However, fixing the problem this way may create other problems if it causes us to fail to put useful things into the cache. The point I was trying (perhaps unsuccessfully) to make, is that we should be careful about not caching things. We are likely to break other corner cases by fixing the ones we are discussing. Thanks, Jim ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? (what the user feels) 2004-06-01 16:49 ` jlnance @ 2004-06-01 17:38 ` Martin Olsson 2004-06-01 17:57 ` Valdis.Kletnieks 2004-06-01 18:01 ` David Schwartz 2004-06-02 18:38 ` why swap at all? John Hendrikx 1 sibling, 2 replies; 22+ messages in thread From: Martin Olsson @ 2004-06-01 17:38 UTC (permalink / raw) To: linux-kernel Hi, From a pure *users* perspective the most evil part of my computer is the harddrive, because it has three bad properties; -- its slowing stuff down -- its very noisy -- its sucks my battery dry (on laptops) I dont care as much about how fast or efficient the swapping system really is, what does annoy me is when it lets me down. Say I do some action repeatedly every once in a while and it always takes about X milliseconds, when after a while that same action takes 2*X milliseconds. Now I'm DISAPPOINTED and I got a grudge with kswapd. Its *emotionally* more 'okay' for the harddrive to its thing when I initiate it through some action (launching program, copying files) because then I expect it. From what I've read previously in this thread, it seems to me that the only major problem with swapping that not all users want file system cache to swap out actual applications (thus making that somewhat aged mozilla window abit laggy). Maybe we could just have a "Allow file system cache to swap out applications checkbox somewhere"? Or, Am I missing something? Sincerly, /m jlnance@unity.ncsu.edu wrote: > On Tue, Jun 01, 2004 at 02:57:00PM +0300, Lenar L?hmus wrote: > >>jlnance@unity.ncsu.edu wrote: >> >> >>>I'm not sure. Copying a file is a pretty good indication that you >>>are about to do something with either the new or the old file. >>> >> >>Like taking the new file with me on USB dongle and deleting old one? >>Caching the file really doesn't help in this case. > > > No, it does not help in this case. > > Not putting things in cache is a solution for the problem of > having useful stuff pushed out of the cache. However, fixing > the problem this way may create other problems if it causes > us to fail to put useful things into the cache. > > The point I was trying (perhaps unsuccessfully) to make, is > that we should be careful about not caching things. We are > likely to break other corner cases by fixing the ones we > are discussing. > > Thanks, > > Jim > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? (what the user feels) 2004-06-01 17:38 ` why swap at all? (what the user feels) Martin Olsson @ 2004-06-01 17:57 ` Valdis.Kletnieks 2004-06-01 18:01 ` David Schwartz 1 sibling, 0 replies; 22+ messages in thread From: Valdis.Kletnieks @ 2004-06-01 17:57 UTC (permalink / raw) To: Martin Olsson; +Cc: linux-kernel [-- Attachment #1: Type: text/plain, Size: 510 bytes --] On Tue, 01 Jun 2004 19:38:54 +0200, Martin Olsson <mnemo@minimum.se> said: > Maybe we could just have a "Allow file system cache to swap out > applications checkbox somewhere"? > > Or, Am I missing something? Isn't that what /proc/sys/vm/swappiness is for? (And yes, I know that it's not a total solution to the whole range of conflicting requirements that we have - I suspect that it's really difficult to make it work much better than it is without adding a lot more history-tracking instrumentation). [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: why swap at all? (what the user feels) 2004-06-01 17:38 ` why swap at all? (what the user feels) Martin Olsson 2004-06-01 17:57 ` Valdis.Kletnieks @ 2004-06-01 18:01 ` David Schwartz 2004-06-01 19:01 ` Robin Rosenberg 1 sibling, 1 reply; 22+ messages in thread From: David Schwartz @ 2004-06-01 18:01 UTC (permalink / raw) To: linux-kernel > From what I've read previously in this thread, it seems to me that the > only major problem with swapping that not all users want file system > cache to swap out actual applications (thus making that somewhat aged > mozilla window abit laggy). > > Maybe we could just have a "Allow file system cache to swap out > applications checkbox somewhere"? > > Or, Am I missing something? In practice, that would make no difference at all. Once physical memory is full (and it pretty much will always be so), every memory request (whether due to the file system cache or application usage) will require discarding some page or other. So even if all memory requests due to file system cache usage were prohibited from forcing out application pages, you're launching enough other application that need pages that application pages will still be evicted. Now, if you make the rule "don't ever swap out application pages", what exactly is the swap going to do? Swap is for dirty pages. Dirty file pages get written back to their ultimate home, not swap. DS ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? (what the user feels) 2004-06-01 18:01 ` David Schwartz @ 2004-06-01 19:01 ` Robin Rosenberg 2004-06-01 19:04 ` David Schwartz 0 siblings, 1 reply; 22+ messages in thread From: Robin Rosenberg @ 2004-06-01 19:01 UTC (permalink / raw) To: davids; +Cc: linux-kernel On Tuesday 01 June 2004 20.01, David Schwartz wrote: > > From what I've read previously in this thread, it seems to me that the > > only major problem with swapping that not all users want file system > > cache to swap out actual applications (thus making that somewhat aged > > mozilla window abit laggy). > > > > Maybe we could just have a "Allow file system cache to swap out > > applications checkbox somewhere"? > > > > Or, Am I missing something? > > In practice, that would make no difference at all. Once physical memory is > full (and it pretty much will always be so), every memory request (whether No. Many people have machines with plenty of RAM (512MB or more is pretty much standard on new machines), much of which is only used to cache files. The file cache is the reason the memory is full. -- robin ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: why swap at all? (what the user feels) 2004-06-01 19:01 ` Robin Rosenberg @ 2004-06-01 19:04 ` David Schwartz 0 siblings, 0 replies; 22+ messages in thread From: David Schwartz @ 2004-06-01 19:04 UTC (permalink / raw) To: robin.rosenberg.lists; +Cc: linux-kernel > On Tuesday 01 June 2004 20.01, David Schwartz wrote: > > > From what I've read previously in this thread, it seems to > > > me that the > > > only major problem with swapping that not all users want file system > > > cache to swap out actual applications (thus making that somewhat aged > > > mozilla window abit laggy). > > > > > > Maybe we could just have a "Allow file system cache to swap out > > > applications checkbox somewhere"? > > > > > > Or, Am I missing something? > > In practice, that would make no difference at all. Once > > physical memory is > > full (and it pretty much will always be so), every memory > request (whether > No. Huh? > Many people have machines with plenty of RAM (512MB or more is > pretty much > standard on new machines), much of which is only used to cache files. The > file cache is the reason the memory is full. Of course. That's why I said, "once physical memory is full (and it pretty much will always be so)". Physical memory is always full, so every memory request requires that a page be evicted. DS ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-06-01 16:49 ` jlnance 2004-06-01 17:38 ` why swap at all? (what the user feels) Martin Olsson @ 2004-06-02 18:38 ` John Hendrikx 1 sibling, 0 replies; 22+ messages in thread From: John Hendrikx @ 2004-06-02 18:38 UTC (permalink / raw) To: Linux Kernel Mailinglist jlnance@unity.ncsu.edu wrote: >On Tue, Jun 01, 2004 at 02:57:00PM +0300, Lenar L?hmus wrote: > > >>jlnance@unity.ncsu.edu wrote: >> >> >> >>>I'm not sure. Copying a file is a pretty good indication that you >>>are about to do something with either the new or the old file. >>> >>> >>> >>Like taking the new file with me on USB dongle and deleting old one? >>Caching the file really doesn't help in this case. >> >> > >No, it does not help in this case. > >Not putting things in cache is a solution for the problem of >having useful stuff pushed out of the cache. However, fixing >the problem this way may create other problems if it causes >us to fail to put useful things into the cache. > >The point I was trying (perhaps unsuccessfully) to make, is >that we should be careful about not caching things. We are >likely to break other corner cases by fixing the ones we >are discussing. > > I've experienced the problem where applications need to be swapped back in. It's mainly caused by the dual role my machine has (desktop machine when I'm using it, server when it is serving files). Whenever my machine has been sitting idly serving files for a while, when I get back, the desktop is slow. However, there is no need for that, as the files are served at low speeds -- there's no real point in caching them apart from maybe preventing harddisk wear... the harddisk itself can serve these files again faster than they will be needed. So perhaps it is possible to reduce caching of data that is simply not putting stress on the system (the harddisk in this case). If the harddisk is not the bottleneck, it is probably not worth caching. Typical examples are letting a box play music all day (and then trying to read your mail...), having a webserver on a slow connection or watching a large movie file. None of these really require much caching beyond a bit of read-ahead. I'm not sure how best to distinguish when something is fast I/O that would benefit from caching and when something is slow I/O that the harddisk can handle well enough on its own. --John ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: why swap at all? 2004-05-31 10:49 ` why swap at all? jlnance 2004-06-01 11:57 ` Lenar Lõhmus @ 2004-06-01 12:21 ` David B. Stevens 1 sibling, 0 replies; 22+ messages in thread From: David B. Stevens @ 2004-06-01 12:21 UTC (permalink / raw) To: jlnance; +Cc: linux-kernel jlnance@unity.ncsu.edu wrote: >>>cp should use fadvise() and say that it _really_ does not need those pages. >> >>Yes, indeed. On the other hand the sequential read could be detected by the kernel, too. > > > I'm not sure. Copying a file is a pretty good indication that you > are about to do something with either the new or the old file. > > Thanks, > > Jim It is? Sorry for butting in folks, but I've been reading this thread hoping to see some possible solutions. Seems that a survey of best practices might have been suggested, however, I haven't seen such a suggestion. So here goes, might it not be of some benefit to see how other operating systems (there are rather large number) handle the use of memory. For just one example you could look at MVS, where the application can request through various means how and how much memory it uses. Most defaults can be overridden by the scripting language used to run the application. This also true of other operating systems. I would be more willing to say that the folks setting up the running of systems should have far more control over the use or non use of cache backed I/O data. Now that I've said that you have to consider how and where this control should be based. <soapbox> SWAP is a solution for the age old whine, I caused the system to run out of memory and the big mean operating system terminated my application. These days it allows the performance of the system to degrade to the point that the whine goes, The big mean operating system is taking forever to run my 10 TB backup and, by the way, it takes 3 days to wake up my openoffice application that I started a week ago. Ain't progress grand ;-) </soapbox> Cheers, Dave ^ permalink raw reply [flat|nested] 22+ messages in thread
* MM patches (was Re: why swap at all?)
@ 2004-05-31 15:48 Zoltan Boszormenyi
0 siblings, 0 replies; 22+ messages in thread
From: Zoltan Boszormenyi @ 2004-05-31 15:48 UTC (permalink / raw)
To: linux-kernel
> http://www.kerneltrap.org/~npiggin/nickvm-267r1m1.gz
>
> It is a cocktail of cleanups, simplification, and enhancements. The
> main ones that applie here is my split active lists patch (search
> archives for details), and explicit use-once logic.
Works good, a vmware session (WinME on a simulated 128MB machine)
does not disturb two open mozillas for two different logged in users,
both using GNOME-2.4 on FC1 system.
Kernel is 2.6.7-rc1-mm1 + your patch + linuxconsole.sf.net ruby.
The machine used about 230KB swap. 512MB DRAM.
Best regards,
Zoltán Böszörményi
^ permalink raw reply [flat|nested] 22+ messages in threadend of thread, other threads:[~2004-06-02 18:44 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <fa.fegqf9v.kmidof@ifi.uio.no>
[not found] ` <fa.bqpvcrs.u648jq@ifi.uio.no>
2004-05-27 11:39 ` why swap at all? Andy Lutomirski
2004-05-28 21:37 ` Denis Vlasenko
2004-05-28 22:28 ` Bernd Eckenfels
2004-05-29 7:31 ` Denis Vlasenko
2004-05-29 8:40 ` MM patches (was Re: why swap at all?) Nick Piggin
2004-05-29 8:46 ` Nick Piggin
[not found] ` <200405292014.23559.matt@lpbproductions.com>
2004-05-30 3:31 ` Nick Piggin
2004-05-31 13:13 ` Tvrtko A. Uršulin
2004-05-31 13:33 ` Con Kolivas
2004-05-31 17:34 ` Andy Lutomirski
2004-05-31 10:49 ` why swap at all? jlnance
2004-06-01 11:57 ` Lenar Lõhmus
2004-06-01 12:27 ` Robin Rosenberg
2004-06-01 16:49 ` jlnance
2004-06-01 17:38 ` why swap at all? (what the user feels) Martin Olsson
2004-06-01 17:57 ` Valdis.Kletnieks
2004-06-01 18:01 ` David Schwartz
2004-06-01 19:01 ` Robin Rosenberg
2004-06-01 19:04 ` David Schwartz
2004-06-02 18:38 ` why swap at all? John Hendrikx
2004-06-01 12:21 ` David B. Stevens
2004-05-31 15:48 MM patches (was Re: why swap at all?) Zoltan Boszormenyi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox