Re: The pagecache unloved in zone NORMAL?

[Zlatko Calusic <zcalusic@bitsync.net>]

[-- Attachment #1: Type: text/plain, Size: 3911 bytes --]

On 11.04.2013 22:30, Zlatko Calusic wrote:
> This is something that I've been chasing for months, and I'm getting
> tired of it. :(
>
> The issue has been observed on 4GB RAM x86_64 machines (one server, one
> desktop) without swap subsystem (not even compiled in). The important
> thing to remember about a 4GB x86_64 machine is that the NORMAL zone is
> about 6 times smaller than the DMA32 zone.
>
> As picture is 10000 words, I've attached two graphs that nicely show
> what I've observed. As memory usage slowly rises, the MM subsystem
> gradually evicts pagecache pages from the NORMAL zone, trying to
> eventually get rid of all of them! This process takes days, typically
> more than 5 on this particular server. Of course, this means that
> eventually the zone will be choke full of anon pages, and without swap,
> the kernel can't do much about it. But as it tries to balance the zone,
> various bad things will happen. On the server I've seen sudden freeing
> of hundreds of MB of pagecache, on the desktop there's a general
> slowdown, sound dropouts (HTTP streaming) and so...

An excellent Konstantin's patch better described here 
http://marc.info/?l=linux-mm&m=136731974301311 is already giving some 
useful additional insight into this problem, just as I expected. Here's 
the data after 31h of server uptime (also see the attached graph):

Node 0, zone    DMA32
     nr_inactive_file 443705
   avg_age_inactive_file: 362800
Node 0, zone   Normal
     nr_inactive_file 32832
   avg_age_inactive_file: 38760

I reckon that only aging of the inactive LRU lists is of the interest at 
the moment, because there's currently a streaming I/O of about 8MB/s 
that can be seen on the graphs. Here's how I decipher the numbers:

DMA32: 443705 pages * 4k ~ 1733MB, 362800 ms = 362.8 seconds to go 
through the LRU and replace each page in it, which finally gives: 
1733/362.8 ~ 4.78 MB/s (approx speed at which the reclaim is goin' on)

Normal zone: 32832*4/1024/38.76 ~ 3.31 MB/s

Check: 4.78 + 3.31 ~ 8 MB/s (just about the rate of the read I/O from 
the disk)

So, if my calculations are right and my model makes sense (Konstantin, 
chime in if I got something wrong!), the reclaim is going through the 
pages in those 2 zones at a very similar speed, although there's already 
13 times less pages in the Normal zone available for streaming I/O 
caching. If this behavior continues when Normal zone get practically 
washed out of file pages (guaranteed in a few days), then we will 
measure TTL of pages in the Normal zone by milliseconds. Not a very 
useful cache, you'll agree. Of course, it's not a problem for streaming 
reads, but dirty pages that end there will be written out practically 
synchronously, and then it's no wonder that the desktop at those moments 
starts behaving worse than a trusty old 486DX2 with 16MB of RAM once was. :(

The only question I have is, is this a design mistake, or a plain bug?

I strongly believe that pages should be reclaimed at speed appropriate 
to the LRU size. After all, all those pages are the same as far as I/O 
is concerned, so there's no reason to throw out some pages after only 38 
seconds, while others are privileged to spend 6 minutes in the memory? 
Those are the numbers from the data above, and we'll see by the end of 
the following week how bad it can really get.

This imbalance is possibly the main reasons why file pages are pushed 
out from the Normal zone too aggresively in the first place. Probably, 
if we can balance the reclaim speed, the whole problem would disappear. 
It looks like faster reclaim in the smaller zone manages to throw out 
more file pages from it (anon pages replace them easier), which in turn 
makes the file LRU's even smaller, which produces even faster reclaim, 
which... you get the idea, kind of a positive feedback loop that feeds 
on itself. The kind that always ends up with a bang. ;)

-- 
Zlatko

[-- Attachment #2: screenshot12.png --]
[-- Type: image/png, Size: 51090 bytes --]