MM patches (was Re: why swap at all?)

MM patches (was Re: why swap at all?)

[Zoltan Boszormenyi]

> http://www.kerneltrap.org/~npiggin/nickvm-267r1m1.gz
> 
> It is a cocktail of cleanups, simplification, and enhancements. The
> main ones that applie here is my split active lists patch (search
> archives for details), and explicit use-once logic.

Works good, a vmware session (WinME on a simulated 128MB machine)
does not disturb two open mozillas for two different logged in users,
both using GNOME-2.4 on FC1 system.

Kernel is 2.6.7-rc1-mm1 + your patch + linuxconsole.sf.net ruby.
The machine used about 230KB swap. 512MB DRAM.

Best regards,
Zoltán Böszörményi

Re: why swap at all?

[Bernd Eckenfels]

In article <200405290037.17775.vda@port.imtp.ilyichevsk.odessa.ua> you wrote:
>> The benchmark involved was ls.  It took several seconds.  If I ran it again
>> in 5 seconds or so, it was fine.  Much longer and it would take several
>> seconds again.  Sounds like pages getting evicted in LRU order.
> 
> By what magic system can know that you are going to do ls again
> in 2 minutes?

The problem is more about the blocks cp touches, less  about predicting the ls workload.

> cp should use fadvise() and say that it _really_ does not need those pages.

Yes, indeed. On the other hand the sequential read could be detected by the kernel, too.

Greetings
Bernd
-- 
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Re: why swap at all?

[Denis Vlasenko]

On Saturday 29 May 2004 01:28, Bernd Eckenfels wrote:
> In article <200405290037.17775.vda@port.imtp.ilyichevsk.odessa.ua> you wrote:
> >> The benchmark involved was ls.  It took several seconds.  If I ran it
> >> again in 5 seconds or so, it was fine.  Much longer and it would take
> >> several seconds again.  Sounds like pages getting evicted in LRU order.
> >
> > By what magic system can know that you are going to do ls again
> > in 2 minutes?
>
> The problem is more about the blocks cp touches, less  about predicting the
> ls workload.
>
> > cp should use fadvise() and say that it _really_ does not need those
> > pages.
>
> Yes, indeed. On the other hand the sequential read could be detected by the
> kernel, too.

Looks like it was. ls' read was sequential, too, so it did not get any
advantage. If you can definitely show that streaming io
(say, cat hugefile >/dev/null) flushes _non_ sequentially read data
(pages with program/library code, data of e.g. your Mozilla, etc),
please submit a report to lkml. VM gurus said more than once
that they _want_ to fix things, but need to know how to reproduce.
--
vda

MM patches (was Re: why swap at all?)

[Nick Piggin]

Denis Vlasenko wrote:

> (pages with program/library code, data of e.g. your Mozilla, etc),
> please submit a report to lkml. VM gurus said more than once
> that they _want_ to fix things, but need to know how to reproduce.

Yep.

Thanks to everyone's input I was able to test and adapt my mm work.
It is hopefully at a stage where it can have wider testing now. It
is stable on my SMP system under very heavy swapping, but the usual
caution applies.

Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X,
with xterms and mozilla open browsing and grepping kernel tree, etc.

Plain 2.6.7-rc1-mm1 swapped 200MB then completely froze up the system
within 9 seconds of starting the read load. Things remained fairly
responsive with my patch applied. A bit of swap out, but very little
swap in, which is good. The entire 32GB went through the pagecache no
problem.

A couple of concurrent mkisofs's writing 4 GB isos don't seem to be
any problem either with the patched kernel. Haven't tried plain -mm
yet.

http://www.kerneltrap.org/~npiggin/nickvm-267r1m1.gz

It is a cocktail of cleanups, simplification, and enhancements. The
main ones that applie here is my split active lists patch (search
archives for details), and explicit use-once logic.

Known issue: page reclaim can get a little bit lumpy (ie lots of
memory freed up at once), but that is just a matter of teaching
things not to bite off massive chunks at a time when it starts
hitting memory pressure.

Re: MM patches (was Re: why swap at all?)

[Nick Piggin]

Nick Piggin wrote:
> Denis Vlasenko wrote:
> 
>> (pages with program/library code, data of e.g. your Mozilla, etc),
>> please submit a report to lkml. VM gurus said more than once
>> that they _want_ to fix things, but need to know how to reproduce.
> 
> 
> Yep.
> 
> Thanks to everyone's input I was able to test and adapt my mm work.
> It is hopefully at a stage where it can have wider testing now. It
> is stable on my SMP system under very heavy swapping, but the usual
> caution applies.
> 
> Test is 4 x cat 8GB > /dev/null (aggregate 100-200MB/s!) while in X,
> with xterms and mozilla open browsing and grepping kernel tree, etc.
> 

This isn't the "very heavy swapping" load, BTW :)

The very heavy swapping load is make -j15 in 64MB of memory.

Re: MM patches (was Re: why swap at all?)

[Nick Piggin]

Matt wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Nick,
> 
> Your patch has worked great, and has significantly made my desktop pc more 
> responsive . 
> 
> If you happen to have any newer versions of your patch , I can test them out 
> and send feedback to you. Or I can provide you with a server to use to 
> benchmark and toy with. 
> 

Thanks for testing Matt. The patch I posted is the newest rollup
I have. I haven't run into any significant problems with it yet,
but I will notify lkml if I have found a bad bug or made significant
changes.

Nick

Re: MM patches (was Re: why swap at all?)

[Tvrtko A. Uršulin]

On Saturday 29 May 2004 10:40, Nick Piggin wrote:

> It is a cocktail of cleanups, simplification, and enhancements. The
> main ones that applie here is my split active lists patch (search
> archives for details), and explicit use-once logic.

I didn't have time to personally test it but just want to share some thoughts. 
This kind of improvement is absolutely necessary for 2.6 to be usefull on the 
desktop. It continues to escape me how come that this kind of, almost, bugfix 
arrives so late during stable period.

Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM from 
now on) which I am currently testing. AM also does an excellent job in 
preventing swap trashing while doing a lot of filesystem reading.

I am curious on how does your patch technically relates to Con's AM, and is it 
possible to merge the two? I know that they do their job on completely 
different ways, so the real question would be: Is there a need for something 
like AM if using your patch, or it completely obsoletes it?

Re: MM patches (was Re: why swap at all?)

[Con Kolivas]

On Mon, 31 May 2004 23:13, Tvrtko A. Uršulin wrote:
> On Saturday 29 May 2004 10:40, Nick Piggin wrote:
> > It is a cocktail of cleanups, simplification, and enhancements. The
> > main ones that applie here is my split active lists patch (search
> > archives for details), and explicit use-once logic.
>
> I didn't have time to personally test it but just want to share some
> thoughts. This kind of improvement is absolutely necessary for 2.6 to be
> usefull on the desktop. It continues to escape me how come that this kind
> of, almost, bugfix arrives so late during stable period.
>
> Unfortunately it doesn't apply on top of Con's autoregulate swappines (AM
> from now on) which I am currently testing. AM also does an excellent job in
> preventing swap trashing while doing a lot of filesystem reading.
>
> I am curious on how does your patch technically relates to Con's AM, and is
> it possible to merge the two? I know that they do their job on completely
> different ways, so the real question would be: Is there a need for
> something like AM if using your patch, or it completely obsoletes it?

I had a quick look at Nick's patches to see for you and it appears that Nick's 
work completely removes the swappiness dial and introduces a different metric 
called vm_mapped_page_cost which is not interchangeable with the swappiness 
value. ie they cannot work together, sorry.

Con

Re: MM patches (was Re: why swap at all?)

[Andy Lutomirski]

Nick Piggin wrote:

> Yep.
> 
> Thanks to everyone's input I was able to test and adapt my mm work.
> It is hopefully at a stage where it can have wider testing now. It
> is stable on my SMP system under very heavy swapping, but the usual
> caution applies.

Thanks!

This feels quite a bit better on my system.  I'll try and stress it a bit 
more later today or tomorrow, but my system is now usable under heavy io load.

--Andy