How do I make this thing stop laging? Reboot? Sounds like Windows!

How do I make this thing stop laging? Reboot? Sounds like Windows!

[rmoser]

Figured this one out.  I had the issue where xmms was skipping a
lot and my system was lagging wasy too much.  Rebooted to fix it
(WTF?!) whenever it happened.  Couldn't understand why.  Hard disk
light flashing, but no programs that read from the HDD all that much
(xmms doesn't do it THAT much).

I got rid of htdig.  It stopped.

So I ran GIMP recently and caused it again.  Clicked in the wrong
place and BOOM!  Gradient needs too much RAM and CPU and time.
Killed gimp.  Suddenly, my system is lagging.

Ten minutes later I get the brains to run top.  It seems I have about
50 MB in swap, and 54 MB free memory.  So I wait ten minutes more.

No change.

% swapoff -a; swapon -a

Fixes all my problems.

Now this long story shows something:  The kernel appears to be unable
to intelligently pull swap back into RAM.  What gives?

I'm poking around in linux/mm and can't find the code to control this.  I
want to make it swap back in any page that it reads, even if it has to
swap out another page (preferably one which hasn't been used for very
long).  Also, a more aggressive thing, kswapd should have freepages.max
in there, to force it to pull in pages from swap aggressively if there's a lot
of free RAM and a lot of swap used.

Uhh, I'm lost... how does this stuff work?  I'm... really lost.  Should I be
doing this?  Tell me where to start maybe?

--Bluefox Icy

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Helge Hafting]

rmoser wrote:
[...]
> Ten minutes later I get the brains to run top.  It seems I have about
> 50 MB in swap, and 54 MB free memory.  So I wait ten minutes more.
> 
> No change.
> 
> % swapoff -a; swapon -a
> 
> Fixes all my problems.
> 
> Now this long story shows something:  The kernel appears to be unable
> to intelligently pull swap back into RAM.  What gives?
> 
Because the problem _is_ unsolvable.  You want the kernel
to go "oh, lots of free memory showed up, lets pull
everything in from swap just in case someone might need it."


That would solve _your_ problem.  But lots of other people
would get another problem - much _more_ swapping:

Whenever they quit one big app to run another big one,
everything is pulled in from swap before the next
big app start.  Then it starts, and push everything out
again.  The current system lets you quit one app,
the stuff in swap remains there until someone actually use it,
and lots of free memory remain in case it is needed.


The "intelligent" thing is to leave stuff in swap until
some app needs it, and pull it in then.  Perhaps with
some read-ahead/clustering to minimize io load.

It is simply impossible to know "what" the
next thing we will need from swap will be, and what
stuff won't ever be needed from swap.  The memory
might be putr to better uses, such as:
1. New programs/allocations can start without
    having to push something out first
2. file cache for io-intensive apps.

Note that reading from swap is very much like reading
from executable files - it is done when needed.
We donæ't normally pre-read every executable
on the system when there is free memory just
in case someone might want to run a program,
the same applies to swap.

Helge Hafting

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Karl Vogel]

On 18 Jun 2003, you wrote in linux.kernel:

> rmoser wrote:
> [...]
>> Ten minutes later I get the brains to run top.  It seems I have about
>> 50 MB in swap, and 54 MB free memory.  So I wait ten minutes more.
>> 
>> No change.
>> 
>> % swapoff -a; swapon -a
>> 
>> Fixes all my problems.
>> 
>> Now this long story shows something:  The kernel appears to be unable
>> to intelligently pull swap back into RAM.  What gives?
>> 
> Because the problem _is_ unsolvable.  You want the kernel
> to go "oh, lots of free memory showed up, lets pull
> everything in from swap just in case someone might need it."


You might want to try Con Kolivas' patches on:
   http://members.optusnet.com.au/ckolivas/kernel/

More specifically the 'swap prefetch' patch. From this FAQ:

--
Swap prefetching? If you have >10% free physical ram and any used swap it 
will start swapping pages back into physical ram. Probably not of real 
benefit but many people like this idea. I have a soft spot for it and like 
using it.
--

The disadvantage is ofcourse that you will be using up more RAM than is 
really necessary.

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Oliver Neukum]

> Swap prefetching? If you have >10% free physical ram and any used swap it
> will start swapping pages back into physical ram. Probably not of real
> benefit but many people like this idea. I have a soft spot for it and like
> using it.
> --
>
> The disadvantage is ofcourse that you will be using up more RAM than is
> really necessary.

No, free RAM is wasted RAM.
You will start wasting RAM once you refuse to free up
pages you just read from swap space for other more
important needs. But that's a general VM problem.

You'll waste IO bandwidth and CPU power to read in pages
that will be evicted unused, but you may get hits and save
page faults. It's just a matter of selecting pages to be preswapped.

But those preswapped pages are either clean and thus immediately
evictable or you were right to preswap them and saved a page fault.

	Regards
		Oliver

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Anders Karlsson]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

On Wed, 2003-06-18 at 10:22, Karl Vogel wrote:
> On 18 Jun 2003, you wrote in linux.kernel:
> 
> > rmoser wrote:
> > [...]
[...]
> > Because the problem _is_ unsolvable.  You want the kernel
> > to go "oh, lots of free memory showed up, lets pull
> > everything in from swap just in case someone might need it."
> 
> 
> You might want to try Con Kolivas' patches on:
>    http://members.optusnet.com.au/ckolivas/kernel/
> 
> More specifically the 'swap prefetch' patch. From this FAQ:
> 
> --
> Swap prefetching? If you have >10% free physical ram and any used swap it 
> will start swapping pages back into physical ram. Probably not of real 
> benefit but many people like this idea. I have a soft spot for it and like 
> using it.
> --
> 
> The disadvantage is ofcourse that you will be using up more RAM than is 
> really necessary.

Sorry for breaking in, but this is an interesting discussion. :-)

I find that the Linux VM tend to push things out in to swap-space when
it does not need it. This is fine. However, I was once told something
about AIX that has lodged itself in the back of my mind.

AIX uses (or used to use) the exact same way of reading/writing data
from/to disk for all I/O. AIX also makes a distinction between code and
data. If code in RAM is unused, it simply gets flushed. If it is needed
again at a later time, it is paged in from disk where it was originally
loaded from. Only dirty data is paged out into swap.

Is it feasible to tweak the Linux VM to behave in the same fashion? If
Linux already does it this way, I'll just shut up. :)

/A


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Gianni Tedesco]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Wed, 2003-06-18 at 11:23, Anders Karlsson wrote:
> AIX uses (or used to use) the exact same way of reading/writing data
> from/to disk for all I/O. AIX also makes a distinction between code and
> data. If code in RAM is unused, it simply gets flushed. If it is needed
> again at a later time, it is paged in from disk where it was originally
> loaded from. Only dirty data is paged out into swap.
> 
> Is it feasible to tweak the Linux VM to behave in the same fashion? If
> Linux already does it this way, I'll just shut up. :)

The distinction in Linux is between anonymous and file-backed mappings.
Executables are file mappings so the pages can just be dropped and they
will be read in from the backing file when needed again. This also holds
true for memory mapped files.

When a mapping has no file (anonymous) then the swap is used. This is
one of the reasons I like mmap() it means you dont end up using swap for
storing buffers containing bits of files which are on disk anyway.

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Joe]

Karl Vogel wrote:

>On 18 Jun 2003, you wrote in linux.kernel:
>
>  
>
>>rmoser wrote:
>>[...]
>>    
>>
>>>Ten minutes later I get the brains to run top.  It seems I have about
>>>50 MB in swap, and 54 MB free memory.  So I wait ten minutes more.
>>>
>>>No change.
>>>
>>>% swapoff -a; swapon -a
>>>
>>>Fixes all my problems.
>>>
>>>Now this long story shows something:  The kernel appears to be unable
>>>to intelligently pull swap back into RAM.  What gives?
>>>
>>>      
>>>
>>Because the problem _is_ unsolvable.  You want the kernel
>>to go "oh, lots of free memory showed up, lets pull
>>everything in from swap just in case someone might need it."
>>    
>>
>
>
>You might want to try Con Kolivas' patches on:
>   http://members.optusnet.com.au/ckolivas/kernel/
>  
>
Unfortunately he provided no information as to
what kernel and/or distro he is running - for all
we know he's running 2.4.9 - so the con man's
patches probably won't apply.

Joe

>  
>

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Yaroslav Rastrigin]

Hi !
>
> Because the problem _is_ unsolvable.  You want the kernel
> to go "oh, lots of free memory showed up, lets pull
> everything in from swap just in case someone might need it."
> ...
>
> It is simply impossible to know "what" the
> next thing we will need from swap will be, and what
> stuff won't ever be needed from swap.  The memory
> might be putr to better uses, such as:
> 1. New programs/allocations can start without
>     having to push something out first
> 2. file cache for io-intensive apps.
> ...
> Note that reading from swap is very much like reading
> from executable files - it is done when needed.
> We donæ't normally pre-read every executable
> on the system when there is free memory just
> in case someone might want to run a program,
> the same applies to swap.
Well, the problem is probably unsolvable on kernel level (kernel is unaware of 
user's habits in app/mem usage), but I think it's pretty solvable on user 
level - give us a knob to tune VM's behavior. We mere mortals often know 
better how we will use our system's memory, and which apps we will be 
running. I, for myself, like laptop-mode patch (basically, it groups disk 
writes to do them once in 5-10 minutes, thus allowing hdd to sleep a lot) 
very much - when I'm on AC, most probably I'm in office , and turning it off 
is reasonable. When I'm on battery, though, chances are I won't be compiling 
the kernel and/or do other heavy disk IO, instead, I most likely will be 
coding, so echo 1 >/proc/sys/vm/laptop_mode seems appropriate, reasonable and 
useful. 
Could something like this be done with VM/swap policy ? 

-- 
With all the best, yarick at relex dot ru.

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Helge Hafting]

On Wed, Jun 18, 2003 at 01:30:48PM +0400, Yaroslav Rastrigin wrote:
[...]
> Well, the problem is probably unsolvable on kernel level (kernel is unaware of 
> user's habits in app/mem usage), but I think it's pretty solvable on user 
> level - give us a knob to tune VM's behavior. We mere mortals often know 
> better how we will use our system's memory, and which apps we will be 
> running. 

There are some knobs.  There's the mlock call that disables
paging for whatever memory you want.  

xmms could easily stop skipping if it mlocks its own code
and data.  Running it at elevated priority might also
be a good idea, so cpu hogs don't starve it.

Both of these needs root pribileges, or at least a suid
binary.  (The priority stuff _can_ be done without extra
privileges by nicing every _other_ process instead.)

> I, for myself, like laptop-mode patch (basically, it groups disk 
> writes to do them once in 5-10 minutes, thus allowing hdd to sleep a lot) 
> very much - when I'm on AC, most probably I'm in office , and turning it off 
> is reasonable. When I'm on battery, though, chances are I won't be compiling 
> the kernel and/or do other heavy disk IO, instead, I most likely will be 
> coding, so echo 1 >/proc/sys/vm/laptop_mode seems appropriate, reasonable and 
> useful. 
> Could something like this be done with VM/swap policy ? 
> 
Sure.  Take a look at /proc/sys/vm/swappiness for example.
More stuff like this can be made - by those interested.

The original poster also fixed the problems by doing
swapoff -a ; swapon -a 
after quitting a memory hog.  That can be automated
with a little cron script that parses the output
of free or vmstat and do this trick whenever
free memory exceed the amount in swap.

Helge Hafting

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Mike Galbraith]

At 01:02 PM 6/18/2003 +0200, Helge Hafting wrote:
>On Wed, Jun 18, 2003 at 01:30:48PM +0400, Yaroslav Rastrigin wrote:
>[...]
> > Well, the problem is probably unsolvable on kernel level (kernel is 
> unaware of
> > user's habits in app/mem usage), but I think it's pretty solvable on user
> > level - give us a knob to tune VM's behavior. We mere mortals often know
> > better how we will use our system's memory, and which apps we will be
> > running.
>
>There are some knobs.  There's the mlock call that disables
>paging for whatever memory you want.
>
>xmms could easily stop skipping if it mlocks its own code
>and data.  Running it at elevated priority might also
>be a good idea, so cpu hogs don't starve it.

Unless it's having trouble getting music into ram fast enough.  It doesn't 
skip here unless I'm pushing hard on the pagecache.

>Both of these needs root pribileges, or at least a suid
>binary.  (The priority stuff _can_ be done without extra
>privileges by nicing every _other_ process instead.)
>
> > I, for myself, like laptop-mode patch (basically, it groups disk
> > writes to do them once in 5-10 minutes, thus allowing hdd to sleep a lot)
> > very much - when I'm on AC, most probably I'm in office , and turning 
> it off
> > is reasonable. When I'm on battery, though, chances are I won't be 
> compiling
> > the kernel and/or do other heavy disk IO, instead, I most likely will be
> > coding, so echo 1 >/proc/sys/vm/laptop_mode seems appropriate, 
> reasonable and
> > useful.
> > Could something like this be done with VM/swap policy ?
> >
>Sure.  Take a look at /proc/sys/vm/swappiness for example.
>More stuff like this can be made - by those interested.
>
>The original poster also fixed the problems by doing
>swapoff -a ; swapon -a...

I don't understand that.  Here, I can swap heftily and not skip.

         -Mike 

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[H. Peter Anvin]

Followup to:  <3EF0214A.3000103@aitel.hist.no>
By author:    Helge Hafting <helgehaf@aitel.hist.no>
In newsgroup: linux.dev.kernel
>
> rmoser wrote:
> [...]
> > Ten minutes later I get the brains to run top.  It seems I have about
> > 50 MB in swap, and 54 MB free memory.  So I wait ten minutes more.
> > 
> > No change.
> > 
> > % swapoff -a; swapon -a
> > 
> > Fixes all my problems.
> > 
> > Now this long story shows something:  The kernel appears to be unable
> > to intelligently pull swap back into RAM.  What gives?
> > 
> Because the problem _is_ unsolvable.  You want the kernel
> to go "oh, lots of free memory showed up, lets pull
> everything in from swap just in case someone might need it."
> 
> 
> That would solve _your_ problem.  But lots of other people
> would get another problem - much _more_ swapping:
> 
> Whenever they quit one big app to run another big one,
> everything is pulled in from swap before the next
> big app start.  Then it starts, and push everything out
> again.  The current system lets you quit one app,
> the stuff in swap remains there until someone actually use it,
> and lots of free memory remain in case it is needed.
> 
> The "intelligent" thing is to leave stuff in swap until
> some app needs it, and pull it in then.  Perhaps with
> some read-ahead/clustering to minimize io load.
> 

This is why you pull things in from swap, but keep tabs on the fact
that it's clean against swap and therefore can be culled at will if
you don't need it.  In other words -- it's present *both* in swap and
RAM.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Helge Hafting]

H. Peter Anvin wrote:
[...]
>>Whenever they quit one big app to run another big one,
>>everything is pulled in from swap before the next
>>big app start.  Then it starts, and push everything out
>>again.  The current system lets you quit one app,
>>the stuff in swap remains there until someone actually use it,
>>and lots of free memory remain in case it is needed.
>>
>>The "intelligent" thing is to leave stuff in swap until
>>some app needs it, and pull it in then.  Perhaps with
>>some read-ahead/clustering to minimize io load.
>>
> 
> 
> This is why you pull things in from swap, but keep tabs on the fact
> that it's clean against swap and therefore can be culled at will if
> you don't need it.  In other words -- it's present *both* in swap and
> RAM.


Good point.
The question is still what to pull in.  Stuff in swap
is one option.  It has been used before, and might
be needed again.

Contents of memory mapped files (executables and others) are another.
We can't know what we will need next, but at least the already opened
files ought to be as likely as swap.

Pulling other files into cache is a third option.  Going for open
files (readahead) or recently used ones might be smart.

Helge Hafting

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Nick Piggin]

Helge Hafting wrote:

> H. Peter Anvin wrote:
> [...]
>
>>> Whenever they quit one big app to run another big one,
>>> everything is pulled in from swap before the next
>>> big app start.  Then it starts, and push everything out
>>> again.  The current system lets you quit one app,
>>> the stuff in swap remains there until someone actually use it,
>>> and lots of free memory remain in case it is needed.
>>>
>>> The "intelligent" thing is to leave stuff in swap until
>>> some app needs it, and pull it in then.  Perhaps with
>>> some read-ahead/clustering to minimize io load.
>>>
>>
>>
>> This is why you pull things in from swap, but keep tabs on the fact
>> that it's clean against swap and therefore can be culled at will if
>> you don't need it.  In other words -- it's present *both* in swap and
>> RAM.
>
>
>
> Good point.
> The question is still what to pull in.  Stuff in swap
> is one option.  It has been used before, and might
> be needed again.
>
> Contents of memory mapped files (executables and others) are another.
> We can't know what we will need next, but at least the already opened
> files ought to be as likely as swap.
>
> Pulling other files into cache is a third option.  Going for open
> files (readahead) or recently used ones might be smart.
>
I think the pauses that desktop people notice and disagree to
is when mapped memory is paged out. If it is paged back in when
there is plenty of memory free, and ide disk, it might be useful
for a desktop load.

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Daniel Gryniewicz]

On Thu, 2003-06-19 at 06:00, Helge Hafting wrote:
[...]
> Good point.
> The question is still what to pull in.  Stuff in swap
> is one option.  It has been used before, and might
> be needed again.
> 
> Contents of memory mapped files (executables and others) are another.
> We can't know what we will need next, but at least the already opened
> files ought to be as likely as swap.
> 
> Pulling other files into cache is a third option.  Going for open
> files (readahead) or recently used ones might be smart.
> 

How about a hint from userspace?  A window manager could say "this is my
working set", and you could try to pull files/maps/swap in for that
working set first.  The window manager could keep an LRU based on
windows getting focus, or something like that, to keep the working set
up to date.  The hint is, of course, open to abuse, so care would have
to be taken, but my window manager could get most of this correct just
based on window input.  Things like xmms would be harder, since I rarely
actually interact with it, but it's also less likely to be swapped out,
because it's always running.

Or, you could start swapping in based on interactive bonus in the
scheduler, but that requires sharing the information with the MM and
trusting the bonus' are correct.

Daniel
-- 
Daniel Gryniewicz <dang@fprintf.net>

RE: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Karl Vogel]

> > Swap prefetching? If you have >10% free physical ram and 
> any used swap it
> > will start swapping pages back into physical ram. Probably 
> not of real
> > benefit but many people like this idea. I have a soft spot 
> for it and like
> > using it.
> > --
> >
> > The disadvantage is ofcourse that you will be using up more 
> RAM than is
> > really necessary.
> 
> No, free RAM is wasted RAM.

But the point is, it's not really free RAM. It's being used for
I/O caching. So while swap prefetching might be suited for
desktop systems, it certainly isn't for servers.

Re: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Oliver Neukum]

Am Mittwoch, 18. Juni 2003 11:56 schrieb Karl Vogel:
> > > Swap prefetching? If you have >10% free physical ram and
> >
> > any used swap it
> >
> > > will start swapping pages back into physical ram. Probably
> >
> > not of real
> >
> > > benefit but many people like this idea. I have a soft spot
> >
> > for it and like
> >
> > > using it.
> > > --
> > >
> > > The disadvantage is ofcourse that you will be using up more
> >
> > RAM than is
> >
> > > really necessary.
> >
> > No, free RAM is wasted RAM.
>
> But the point is, it's not really free RAM. It's being used for
> I/O caching. So while swap prefetching might be suited for
> desktop systems, it certainly isn't for servers.

Not entirely. Have a look:

oliver@vermuden:~> free
             total       used       free     shared    buffers     cached
Mem:        514844     501652      13192          0     130124     216476
-/+ buffers/cache:     155052     359792
Swap:      1036152       6120    1030032
oliver@vermuden:~> free
             total       used       free     shared    buffers     cached
Mem:        514844     500500      14344          0     119940     224772
-/+ buffers/cache:     155788     359056
Swap:      1036152       6584    1029568
oliver@vermuden:~>

This is before starting OpenOffice and after quitting it.
As you can see the amount free RAM really goes up. This RAM is wasted.
It will eventually be put to use, but there's nothing wrong with speeding
this up. Reading back swap is almost certainly not the optimum way to
use it, but it's better than nothing at all, provided the RAM is not tied up.
The question is whether it's worth it in terms of IO.

	Regards
		Oliver

RE: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Karl Vogel]

> Is it feasible to tweak the Linux VM to behave in the same fashion? If
> Linux already does it this way, I'll just shut up. :)

I think it's time to shut up :-)

RE: How do I make this thing stop laging? Reboot? Sounds like Windows!

[Karl Vogel]

> I find that the Linux VM tend to push things out in to swap-space when
> it does not need it. This is fine. However, I was once told something
> about AIX that has lodged itself in the back of my mind.
> 
> AIX uses (or used to use) the exact same way of reading/writing data
> from/to disk for all I/O. AIX also makes a distinction 
> between code and
> data. If code in RAM is unused, it simply gets flushed. If it 
> is needed
> again at a later time, it is paged in from disk where it was 
> originally
> loaded from. Only dirty data is paged out into swap.
> 
> Is it feasible to tweak the Linux VM to behave in the same fashion? If
> Linux already does it this way, I'll just shut up. :)

EUhm now a bit more constructive than my last reply (sorry, couldn't help
myself :)


Referring to Mel Gorman's excellent v2.4 kernel VM documentation:

Understanding The
Linux Virtual Memory Manager

"
12. Swap Management

Just as Linux uses free memory for purposes such as buffering data from
disk, there eventually is a need to free up private or anonymous pages used
by a process. These pages, unlike those backed by a file on disk, cannot be
simply discarded to be read in later. Instead they have to be carefully
copied to backing storage, sometimes called the swap area. This chapter
details how Linux uses and manages its backing storage.
"

http://www.csn.ul.ie/~mel/projects/vm/guide/html/understand/node73.html