VM balancing problems under 2.4.2-ac1

VM balancing problems under 2.4.2-ac1

[Adam Sampson]

Hiya.

The VM balancing updates in the recent ac kernels seem to have caused
some interesting performance problems on my desktop machine. I've got
160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large
amounts of it for buffers and cache while pushing stuff out to
swap. This means that Mozilla, for instance, runs significantly worse
than under 2.4.0, since bits of it are being swapped in and out.

After the machine had been sitting for a while not doing very much:
   procs                      memory    swap          io     system
     cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in   cs
us  sy  id
 1  0  0  97184   2116  12844 111768   5   6    15    11  154  791
29   4  67

After some heavy reiserfs disk IO (deleting lots of small files):
   procs                      memory    swap          io     system
     cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs
us  sy  id
 1  0  0 102620   1796  85836  43880 100   0    25     0  190   587
12   3  85

-- 

Adam Sampson
azz@gnu.org

Re: VM balancing problems under 2.4.2-ac1

[Rik van Riel]

On 23 Feb 2001, Adam Sampson wrote:

> The VM balancing updates in the recent ac kernels seem to have caused
> some interesting performance problems on my desktop machine. I've got
> 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large
> amounts of it for buffers and cache while pushing stuff out to
> swap. This means that Mozilla, for instance, runs significantly worse
> than under 2.4.0, since bits of it are being swapped in and out.

This is a known problem which I'll fix as soon as I have a
solution.

The problem is that we still have no good way to balance
how much memory we take from the cache and how much memory
we take from processes.

This means that for some workloads we'll be evicting too
much cache while for other workloads we'll be evicting too
much process pages...

If anybody as a good idea to make this code auto-balancing,
please let me know.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: VM balancing problems under 2.4.2-ac1

[Joseph Bueno]

Rik van Riel a écrit :
> 
> On 23 Feb 2001, Adam Sampson wrote:
> 
> > The VM balancing updates in the recent ac kernels seem to have caused
> > some interesting performance problems on my desktop machine. I've got
> > 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large
> > amounts of it for buffers and cache while pushing stuff out to
> > swap. This means that Mozilla, for instance, runs significantly worse
> > than under 2.4.0, since bits of it are being swapped in and out.
> 
> This is a known problem which I'll fix as soon as I have a
> solution.
> 
> The problem is that we still have no good way to balance
> how much memory we take from the cache and how much memory
> we take from processes.
> 
> This means that for some workloads we'll be evicting too
> much cache while for other workloads we'll be evicting too
> much process pages...
> 
> If anybody as a good idea to make this code auto-balancing,
> please let me know.
> 
> regards,
> 
> Rik
> --
> Virtual memory is like a game you can't win;
> However, without VM there's truly nothing to lose...
> 
>                 http://www.surriel.com/
> http://www.conectiva.com/       http://distro.conectiva.com.br/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Hi Rik,

I understand that auto-balancing code that deals with all
situations is very hard to design; so let me share my experience
on other Unix systems (from a user/administrator point of view):

I have used Unix systems (mainly HPUX) for several years as personal
workstations or servers and buffer cache usage were very differents:

On workstations, you are mainly looking for fast interactive response
time and  you want to dedicate as much memory as possible to running
processes so limiting buffer cache to 10% of physical memory (these
workstations had typically 32 - 64 Mb of RAM) was good.

On file servers, interactive response time is much less important than
file/network througput. In this case, having 80% of RAM used for buffer
cache is good and you may even want to not let it go below 50% even if
it slows down some batch processes.

Both cases were easily handled by 2 HPUX kernel tunable parameters that
defined minimum and maximum number of pages that could be used by the
buffer cache.
This could be implemented on Linux via /proc. I know it is already done
for minimum limit (in 2.2, I have no experience with 2.4 yet).
I have found some situations where not being able to force a maximum
limit was a problem.

You could argue that with a good load balancing algorithm user
defined limits are useless. Believe me, my experience on HPUX
workstations showed that lowering its max. limit from 50% (default
value) to 10% turned some sluggish machines into speed daemons !

Just my 0.02$
Hope this helps
--
Joseph Bueno

Re: VM balancing problems under 2.4.2-ac1

[Rik van Riel]

On Sat, 24 Feb 2001, Joseph Bueno wrote:
> Rik van Riel a écrit :
> > On 23 Feb 2001, Adam Sampson wrote:
> >
> > > The VM balancing updates in the recent ac kernels seem to have caused
> > > some interesting performance problems on my desktop machine. I've got
> > > 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large
> > > amounts of it for buffers and cache while pushing stuff out to
> > > swap. This means that Mozilla, for instance, runs significantly worse
> > > than under 2.4.0, since bits of it are being swapped in and out.
> >
> > This is a known problem which I'll fix as soon as I have a
> > solution.
> >
> > The problem is that we still have no good way to balance
> > how much memory we take from the cache and how much memory
> > we take from processes.

> I understand that auto-balancing code that deals with all
> situations is very hard to design; so let me share my experience
> on other Unix systems (from a user/administrator point of view):
>
> I have used Unix systems (mainly HPUX) for several years as personal
> workstations or servers and buffer cache usage were very differents:
>
> On workstations, you are mainly looking for fast interactive response
> time and  you want to dedicate as much memory as possible to running
> processes so limiting buffer cache to 10% of physical memory (these
> workstations had typically 32 - 64 Mb of RAM) was good.

"Unfortunately" the cache also contains _process memory_ in
Linux. Limiting the cache to 10% also means limiting the
code size of all your processes to something smaller than
that.

Also, read-in swap pages are in the so-called swap cache,
which is also part of the page cache.

This means that simple limits on cache size probably won't do
much good on Linux.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: VM balancing problems under 2.4.2-ac1

[Mike Galbraith]

On Fri, 23 Feb 2001, Rik van Riel wrote:

> If anybody as a good idea to make this code auto-balancing,
> please let me know.

(I haven't seen even one suggestion posted.. must be a real bugger)

I haven't found any silver bullets ;) but the one line bend-adjust
below does help the cache problem a little on my wimpy 128mb box.

Worth trying out on other boxen with other loads?  It doesn't get
swap in/out to a 1:1 ratio, but does improve it considerably by
scanning a larger portion of active pages prior to swapout.

(the other two lines are there only because it seemed reasonable;)

	-Mike

against 2.4.2-ac4

--- mm/vmscan.c.org	Mon Feb 26 09:31:46 2001
+++ mm/vmscan.c	Mon Feb 26 16:32:46 2001
@@ -278,6 +278,8 @@
 	/* Always start by trying to penalize the process that is allocating memory */
 	if (mm)
 		retval = swap_out_mm(mm, swap_amount(mm));
+	if (retval)
+		return retval;

 	/* Then, look at the other mm's */
 	counter = (mmlist_nr << SWAP_SHIFT) >> priority;
@@ -846,7 +848,7 @@
  * continue with its real work sooner. It also helps balancing when we
  * have multiple processes in try_to_free_pages simultaneously.
  */
-#define DEF_PRIORITY (6)
+#define DEF_PRIORITY (2)
 static int refill_inactive(unsigned int gfp_mask, int user)
 {
 	int count, start_count, maxtry;

Re: VM balancing problems under 2.4.2-ac1

[Adrian Bunk]

On Fri, 23 Feb 2001, Rik van Riel wrote:

> On 23 Feb 2001, Adam Sampson wrote:
>
> > The VM balancing updates in the recent ac kernels seem to have caused
> > some interesting performance problems on my desktop machine. I've got
> > 160Mb of RAM, and 2.4.2-ac1 appears to be using excessively large
> > amounts of it for buffers and cache while pushing stuff out to
> > swap. This means that Mozilla, for instance, runs significantly worse
> > than under 2.4.0, since bits of it are being swapped in and out.
>
> This is a known problem which I'll fix as soon as I have a
> solution.
>
> The problem is that we still have no good way to balance
> how much memory we take from the cache and how much memory
> we take from processes.

I have the same problem Adam has: I'm running 3-5 applications on my
computer. I have 64 MB of RAM and I use usually less than 50 MB. I have
swap for the rare cases where I need more RAM than I have. But with
2.4.x-acyz kernels I do often have to wait several seconds after I
switched to another running application before it's swapped in again
because it seems this application was swapped out to cache some MP3 I
surely won't listen to before the next reboot...

> This means that for some workloads we'll be evicting too
> much cache while for other workloads we'll be evicting too
> much process pages...
>
> If anybody as a good idea to make this code auto-balancing,
> please let me know.

I have no idea for auto-balancing but another idea: It's one possibility
to let the user choose when doing "make *config" what he wants:

- A VM optimized for servers that swaps out applications in favor of
  caching.
or
- A VM optimized for workstations that won't swap out applications in
  favor of caching.


I know that's not a perfect solution but it would make the situation much
better.


> regards,
>
> Rik

cu
Adrian

-- 

Nicht weil die Dinge schwierig sind wagen wir sie nicht,
sondern weil wir sie nicht wagen sind sie schwierig.

Re: VM balancing problems under 2.4.2-ac1

[Ingo Oeser]

On Sat, Mar 03, 2001 at 01:03:26AM +0100, Adrian Bunk wrote:
> > If anybody as a good idea to make this code auto-balancing,
> > please let me know.
> 
> I have no idea for auto-balancing but another idea: It's one possibility
> to let the user choose when doing "make *config" what he wants:
> 
> - A VM optimized for servers that swaps out applications in favor of
>   caching.
> or
> - A VM optimized for workstations that won't swap out applications in
>   favor of caching.

I thought about the same thing sometimes (but for other troughput
vs. latency decisions, too).

But I realized, that my very own workstation is also a server,
since it runs an httpd, mysqld, smbd, ftpd etc.

And somtimes the servers become very busy in our LAN[1].

IF we want that tuning, we should have it as a sysctl. Most of it
is already possible with /proc/sys/vm/*, but balancing decisions
are still missing.

And even for servers we need to reduce caching sometimes. Think
of an httpd serving _very_ dynamic content. Or any other
application (e.g. DMBS), that doesn't rely on file system
caching.

A anonymous/file-backed[2] ratio would be VERY handy ;-)

But maybe this will be implemented one day along the lines of QoS
in the VM...

Regards

Ingo Oeser

[1] >1500 possible clients for these servers.
[2] Not counting swaps as file backed. We have a special inode
   for the swapper anyway, right?
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
         <<<<<<<<<<<<       come and join the fun       >>>>>>>>>>>>

Re: VM balancing problems under 2.4.2-ac1

[Mike Galbraith]

On Sun, 4 Mar 2001, Ingo Oeser wrote:

> On Sat, Mar 03, 2001 at 01:03:26AM +0100, Adrian Bunk wrote:
> > > If anybody as a good idea to make this code auto-balancing,
> > > please let me know.
> >
> > I have no idea for auto-balancing but another idea: It's one possibility
> > to let the user choose when doing "make *config" what he wants:
> >
> > - A VM optimized for servers that swaps out applications in favor of
> >   caching.
> > or
> > - A VM optimized for workstations that won't swap out applications in
> >   favor of caching.
>
> I thought about the same thing sometimes (but for other troughput
> vs. latency decisions, too).
>
> But I realized, that my very own workstation is also a server,
> since it runs an httpd, mysqld, smbd, ftpd etc.
>
> And somtimes the servers become very busy in our LAN[1].
>
> IF we want that tuning, we should have it as a sysctl. Most of it
> is already possible with /proc/sys/vm/*, but balancing decisions
> are still missing.

I think sysctls for balancing knobs is a great idea.  The VM has no
clue concerning the cost of rebuilding cache eg but a human may.

Automatic tuning would be wonderful, but it requires information
which the VM flat doesn't have.. so it should ask the boss for help.

Three handy knobs I can think of off the top of my head are swap_size,
flush_size [for page_launder().. bdflush has that] and cache_stickiness.

	-Mike

Re: VM balancing problems under 2.4.2-ac1

[Benjamin Redelings I]

 
Content-Type: text/plain; charset=big5
Content-Transfer-Encoding: 7bit

Rik van Riel wrote:
> In 2.4.1-pre<something> the kernel swaps out cache 32 times more
> agressively than it scans pages in processes. Until we find a way
> to auto-balance these things, expect them to be wrong for at least
> some workloads ;(

and elsewhere,

> That's because your problem requires a change to the
> balancing between swap_out() and refill_inactive_scan()
> in refill_inactive()...

Rik, can you explain why we still need to "balance" things, instead of
just swapping out the least used pages?  Is this only a problem with the
2.4 implementation (e.g. will refill_inactive_scan eventually do
swap_out in 2.5, or something?), or is it a generic VM issue that has to
be solved?

I'm not quite sure what is going on, but is there perhaps some way that
we can get better information about recent use for pages?  Or do we have
to treat age pages of different type differently based on their usage
pattern, which depends on the type of workload?

Regarding better information on recent page accesses, one thing that
seems strange to me is aging pages down to 0 before making them
inactive.  In order to have the most amount of information in the page
ages, we don't want more than 1/PAGE_AGE_MAX pages to have the same
age.  (Well, there are problems to that view, )  If we only consider
pages with age 0 to be inactive, then if we want a lot of inactive
pages, we make all those pages age zero.  Instead, it seems like a good
thing to do, would be to have a variable "inactive_level", so that a
page is inactive if the age is less than inactive_level.  If we want
more pages to be inactive, we increase inactive_level.  I guess I can
thing of some problems for this approach, but maybe it can be rescued.

In any case, is there a file in /proc/ that displays the number of pages
at each age?  It could be interesting to calculate Shannon Entropy from
this and see how many bits of information we get - a better entropy
could indicate a better aging algorithm.  I guess "6" is the best we
could get right now.

Also, somebody posted a paper reference to linux-mm a few months ago,
about how the optimal page aging strategy aged pages according to how
rapidly referenced bits were being set.  I guess the background page
aging tries to do something like this.  I don't know how well it
succeeds though...

	Thanks for any response/explanation!

-BenRI
-- 
"...assisted of course by pride, for we teach them to describe the
 Creeping Death, as Good Sense, or Maturity, or Experience." 
- "The Screwtape Letters"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: VM balancing problems under 2.4.2-ac1

[Rik van Riel]

On Sat, 24 Feb 2001, Benjamin Redelings I wrote:
> Rik van Riel wrote:
> > In 2.4.1-pre<something> the kernel swaps out cache 32 times more
> > agressively than it scans pages in processes. Until we find a way
> > to auto-balance these things, expect them to be wrong for at least
> > some workloads ;(
>
> and elsewhere,
>
> > That's because your problem requires a change to the
> > balancing between swap_out() and refill_inactive_scan()
> > in refill_inactive()...
>
> Rik, can you explain why we still need to "balance" things,
> instead of just swapping out the least used pages?  Is this only
> a problem with the 2.4 implementation (e.g. will
> refill_inactive_scan eventually do swap_out in 2.5, or
> something?), or is it a generic VM issue that has to be solved?

The problem is that we scan processes by _virtual address_
and the cache by physical page. Furthermore, there are LOTS
of pages which are present in both the cache AND in processes.

Say, for example, that on a 64MB system you have 32MB cache but
half of that cache is cached executable text and swap cache (which
is also mapped into processes).

Now how would you adjust the scanning rate of processes and cache
to make sure we age each page in the system at approximately the
same rate ?

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/