Re: [RFC PATCH 0/6] Do not call ->writepage[s] from direct reclaim and use a_ops->writepages() where possible

Mel Gorman
2010-06-15T14:51:54Z

On Tue, Jun 15, 2010 at 04:00:11PM +0200, Andrea Arcangeli wrote:
> Hi Mel,
>
> I know lots of people doesn't like direct reclaim,

It's not direct reclaim that is the problem per-se, it's direct reclaim
calling writepage and splicing two potentially deep call chains
together.

> but I personally do
> and I think if memory pressure is hard enough we should eventually
> enter direct reclaim full force including ->writepage to avoid false
> positive OOM failures.

Be that as it may, filesystems that have deep call paths for their
->writepage are ignoring both kswapd and direct reclaim so on XFS and
btrfs for example, this "full force" effect is not being reached.

> Transparent hugepage allocation in fact won't
> even wakeup kswapd that would be insist to create hugepages and shrink
> an excessive amount of memory (especially before memory compaction was
> merged, it shall be tried again but if memory compaction fails in
> kswapd context, definitely kswapd should immediately stop and not go
> ahead trying the create hugepages the blind way, kswapd
> order-awareness the blind way is surely detrimental and pointless).
>

kswapd does end up freeing a lot of memory in response to lumpy reclaim
because it also tries to restore watermarks for a high-order page. This
is disruptive to the system and something I'm going to revisit but it's
a separate topic for another discussion. I can see why transparent
hugepage support would not want this disruptive effect to occur where as
it might make sense when resizing the hugepage pool.

> When memory pressure is low, not going into ->writepage may be
> beneficial from latency prospective too. (but again it depends how
> much it matters to go in LRU and how beneficial is the cache, to know
> if it's worth taking clean cache away even if hotter than dirty cache)
>
> About the stack overflow did you ever got any stack-debug error?

Not an error. Got a report from Dave Chinner though and it's what kicked
off this whole routine in the first place. I've been recording stack
usage figures but not reporting them. In reclaim I'm getting to about 5K
deep but this was on simple storage and XFS was ignoring attempts for
reclaim to writeback.

http://lkml.org/lkml/2010/4/13/121

Here is one my my own stack traces though

Depth Size Location (49 entries)
----- ---- --------
0) 5064 304 get_page_from_freelist+0x2e4/0x722
1) 4760 240 __alloc_pages_nodemask+0x15f/0x6a7
2) 4520 48 kmem_getpages+0x61/0x12c
3) 4472 96 cache_grow+0xca/0x272
4) 4376 80 cache_alloc_refill+0x1d4/0x226
5) 4296 64 kmem_cache_alloc+0x129/0x1bc
6) 4232 16 mempool_alloc_slab+0x16/0x18
7) 4216 144 mempool_alloc+0x56/0x104
8) 4072 16 scsi_sg_alloc+0x48/0x4a [scsi_mod]
9) 4056 96 __sg_alloc_table+0x58/0xf8
10) 3960 32 scsi_init_sgtable+0x37/0x8f [scsi_mod]
11) 3928 32 scsi_init_io+0x24/0xce [scsi_mod]
12) 3896 48 scsi_setup_fs_cmnd+0xbc/0xc4 [scsi_mod]
13) 3848 144 sd_prep_fn+0x1d3/0xc13 [sd_mod]
14) 3704 64 blk_peek_request+0xe2/0x1a6
15) 3640 96 scsi_request_fn+0x87/0x522 [scsi_mod]
16) 3544 32 __blk_run_queue+0x88/0x14b
17) 3512 48 elv_insert+0xb7/0x254
18) 3464 48 __elv_add_request+0x9f/0xa7
19) 3416 128 __make_request+0x3f4/0x476
20) 3288 192 generic_make_request+0x332/0x3a4
21) 3096 64 submit_bio+0xc4/0xcd
22) 3032 80 _xfs_buf_ioapply+0x222/0x252 [xfs]
23) 2952 48 xfs_buf_iorequest+0x84/0xa1 [xfs]
24) 2904 32 xlog_bdstrat+0x47/0x4d [xfs]
25) 2872 64 xlog_sync+0x21a/0x329 [xfs]
26) 2808 48 xlog_state_release_iclog+0x9b/0xa8 [xfs]
27) 2760 176 xlog_write+0x356/0x506 [xfs]
28) 2584 96 xfs_log_write+0x5a/0x86 [xfs]
29) 2488 368 xfs_trans_commit_iclog+0x165/0x2c3 [xfs]
30) 2120 80 _xfs_trans_commit+0xd8/0x20d [xfs]
31) 2040 240 xfs_iomap_write_allocate+0x247/0x336 [xfs]
32) 1800 144 xfs_iomap+0x31a/0x345 [xfs]
33) 1656 48 xfs_map_blocks+0x3c/0x40 [xfs]
34) 1608 256 xfs_page_state_convert+0x2c4/0x597 [xfs]
35) 1352 64 xfs_vm_writepage+0xf5/0x12f [xfs]
36) 1288 32 __writepage+0x17/0x34
37) 1256 288 write_cache_pages+0x1f3/0x2f8
38) 968 16 generic_writepages+0x24/0x2a
39) 952 64 xfs_vm_writepages+0x4f/0x5c [xfs]
40) 888 16 do_writepages+0x21/0x2a
41) 872 48 writeback_single_inode+0xd8/0x2f4
42) 824 112 writeback_inodes_wb+0x41a/0x51e
43) 712 176 wb_writeback+0x13d/0x1b7
44) 536 128 wb_do_writeback+0x150/0x167
45) 408 80 bdi_writeback_task+0x43/0x117
46) 328 48 bdi_start_fn+0x76/0xd5
47) 280 96 kthread+0x82/0x8a
48) 184 184 kernel_thread_helper+0x4/0x10

XFS as you can see is quite deep there. Now consider if
get_page_from_freelist() there had entered direct reclaim and then tried
to writeback a page. That's the problem that is being worried about.

> We've
> plenty of instrumentation and ->writepage definitely runs with irq
> enable, so if there's any issue, it can't possibly be unnoticed. The
> worry about stack overflow shall be backed by numbers.
>
> You posted lots of latency numbers (surely latency will improve but
> it's only safe approach on light memory pressure, on heavy pressure
> it'll early-oom not to call ->writepage, and if cache is very
> important and system has little ram, not going in lru order may also
> screw fs-cache performance),

I also haven't been able to trigger a new OOM as a result of the patch
but maybe I'm missing something. To trigger an OOM, the bulk of the LRU
would have to be dirty and the direct reclaimer making no further
progress but if the bulk of the LRU has been dirtied like this, are we
not already in trouble?

We could have it that direct reclaimers kick the flusher threads when it
counters dirty pages and goes to sleep but this will increase latency
and considering the number of dirty pages direct reclaimers should be
seeing, I'm not sure it's necessary.

> but I didn't see any max-stack usage hard
> numbers, to back the claim that we're going to overflow.
>

I hadn't posted them because they had been posted previously and I
didn't think they were that interesting as such because it wasn't being
disputed.

> In any case I'd prefer to be able to still call ->writepage if memory
> pressure is high (at some point when priority going down and
> collecting clean cache doesn't still satisfy the allocation),

Well, kswapd is still writing pages if the pressure is high enough that
the flusher threads are not doing it and a direct reclaimer will wait on
congestion_wait() if the pressure gets high enough (PRIORITY < 2).

> during
> allocations in direct reclaim and increase the THREAD_SIZE than doing
> this purely for stack reasons as the VM will lose reliability if we
> forbid ->writepage at all in direct reclaim.

Well, we've lost that particular reliability already on btrfs and xfs
because they are ignoring the VM and increasing THREAD_SIZE would
increase the order used for stack allocations which causes problems of
its own.

The VM would lose a lot of reliability if we weren't throttling on pages
being dirtied in the fault path but because we are doing that, I don't
currently believe we are losing reliability by not writing back pages in
direct reclaim.

> Throttling on kswapd is
> possible but it's probably less efficient and on the stack we know
> exactly which kind of memory we should allocate, kswapd doesn't and it
> works global.
>

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab