Re: balance_pgdat(): where is total_scanned ever updated?

[Nikita Danilov <nikita@clusterfs.com>]

Marcelo Tosatti writes:

[...]

 > 
 > Another related thing I noted this afternoon is that right now kswapd will
 > always block on full queues:
 > 
 > static int may_write_to_queue(struct backing_dev_info *bdi)
 > {
 >         if (current_is_kswapd())
 >                 return 1;
 >         if (current_is_pdflush())       /* This is unlikely, but why not... */
 >                 return 1;
 >         if (!bdi_write_congested(bdi))
 >                 return 1;
 >         if (bdi == current->backing_dev_info)
 >                 return 1;
 >         return 0;
 > }
 > 
 > We should make kswapd use the "bdi_write_congested" information and avoid
 > blocking on full queues. It should improve performance on multi-device 
 > systems with intense VM loads.

This will have following undesirable side effect: if
may_write_to_queue() returns false, page is not paged out, instead it is
thrown to the head of the inactive queue, thus destroying "LRU
ordering", shrink_list() will dive deeper into inactive list, reclaiming
hotter pages.

It's OK to accidentially skip pageout in direct reclaim path, because

 - we hope most pageout is done by kswapd, and

 - we don't want __alloc_pages() to stall

but _something_ in the kernel should take a pain of actually writing
pages out in LRU order.

 > 
 > Maybe something along the lines 
 > 
 > "if the reclaim ratio is high, do not writepage"
 > "if the reclaim ratio is below high, writepage but not block"
 > "if the reclaim ratio is low, writepage and block"

If kswapd blocking is a concern, inactive list scanning should be
decoupled from actual page-out (a la Solaris): kswapd queues pages to
the yet another kernel thread that calls pageout().

I played with this idea (see
http://nikita.w3.to/code/patches/2-6-10-rc1/async-writepage.txt note
that async_writepage() has to be adjusted to work for kswapd), but while
in some cases (large concurrent builds) it does provide a benefit, in
other cases (heavy write through mmap) it makes throughput slightly
worse.

Besides, this doesn't completely avoid the problem of destroying LRU
ordering, as kswapd still proceeds further through inactive list while
pages are sent out asynchronously.

Nikita.