linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/6] writeback: moving expire targets for background/kupdate works v2
@ 2011-04-20  8:03 Wu Fengguang
  2011-04-20  8:03 ` [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes() Wu Fengguang
                   ` (5 more replies)
  0 siblings, 6 replies; 37+ messages in thread
From: Wu Fengguang @ 2011-04-20  8:03 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Jan Kara, Mel Gorman, Dave Chinner, Itaru Kitayama, Minchan Kim,
	Linux Memory Management List, linux-fsdevel, Wu Fengguang, LKML

Andrew,

This aims to reduce possible pageout() calls by making the flusher
concentrate a bit more on old/expired dirty inodes.

Rationals and benchmark numbers are added in patches 05, 06.

It runs fine on simple workloads over ext3/4, xfs, btrfs and NFS.

code refactor
	[PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes()

loop condition fixes
	[PATCH 2/6] writeback: introduce writeback_control.inodes_cleaned
	[PATCH 3/6] writeback: try more writeback as long as something was written

make dirty expire time a moving target
	[PATCH 4/6] writeback: the kupdate expire timestamp should be a moving target
	[PATCH 5/6] writeback: sync expired inodes first in background writeback

consistent requeue policy
it's not an integral part of this patchset, however do depends on patch 03
	[PATCH 6/6] writeback: refill b_io iff empty


Thanks,
Fengguang


^ permalink raw reply	[flat|nested] 37+ messages in thread
* [PATCH 0/6] writeback: moving expire targets for background/kupdate works
@ 2011-04-19  3:00 Wu Fengguang
  2011-04-19  3:00 ` [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes() Wu Fengguang
  0 siblings, 1 reply; 37+ messages in thread
From: Wu Fengguang @ 2011-04-19  3:00 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Jan Kara, Mel Gorman, Dave Chinner, Trond Myklebust,
	Itaru Kitayama, Minchan Kim, Wu Fengguang, LKML, linux-fsdevel,
	Linux Memory Management List


Andrew,

This aims to reduce possible pageout() calls by making the flusher
concentrate a bit more on old/expired dirty inodes.

Patches 04, 05 have been updated since last post, please review.
The concerns from last review have been addressed.

It runs fine on simple workloads over ext3/4, xfs, btrfs and NFS.

Trond, will you take the last patch? The fixed "bug" has no real impact for now.

make dirty expire time a moving target
        [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes()
        [PATCH 2/6] writeback: the kupdate expire timestamp should be a moving target
        [PATCH 3/6] writeback: sync expired inodes first in background writeback

loop condition fixes (the most tricky part)
        [PATCH 4/6] writeback: introduce writeback_control.inodes_cleaned
        [PATCH 5/6] writeback: try more writeback as long as something was written

NFS fix
        [PATCH 6/6] NFS: return -EAGAIN when skipped commit in nfs_commit_unstable_pages()

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [PATCH 0/6] [RFC] writeback: try to write older pages first
@ 2010-07-22  5:09 Wu Fengguang
  2010-07-22  5:09 ` [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes() Wu Fengguang
  0 siblings, 1 reply; 37+ messages in thread
From: Wu Fengguang @ 2010-07-22  5:09 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Dave Chinner, Christoph Hellwig, Mel Gorman, Chris Mason,
	Jens Axboe, Wu Fengguang, LKML, linux-fsdevel, linux-mm

Andrew,

The basic way of avoiding pageout() is to make the flusher sync inodes in the
right order. Oldest dirty inodes contains oldest pages. The smaller inode it
is, the more correlation between inode dirty time and its pages' dirty time.
So for small dirty inodes, syncing in the order of inode dirty time is able to
avoid pageout(). If pageout() is still triggered frequently in this case, the
30s dirty expire time may be too long and could be shrinked adaptively; or it
may be a stressed memcg list whose dirty inodes/pages are more hard to track.

For a large dirty inode, it may flush lots of newly dirtied pages _after_
syncing the expired pages. This is the normal case for a single-stream
sequential dirtier, where older pages are in lower offsets.  In this case we
shall not insist on syncing the whole large dirty inode before considering the
other small dirty inodes. This risks wasting time syncing 1GB freshly dirtied
pages before syncing the other N*1MB expired dirty pages who are approaching
the end of the LRU list and hence pageout().

For a large dirty inode, it may also flush lots of newly dirtied pages _before_
hitting the desired old ones, in which case it helps for pageout() to do some
clustered writeback, and/or set mapping->writeback_index to help the flusher
focus on old pages.

For a large dirty inode, it may also have intermixed old and new dirty pages.
In this case we need to make sure the inode is queued for IO before some of
its pages hit pageout(). Adaptive dirty expire time helps here.

OK, end of the vapour ideas. As for this patchset, it fixes the current
kupdate/background writeback priority:

- the kupdate/background writeback shall include newly expired inodes at each
  queue_io() time, as the large inodes left over from previous writeback rounds
  are likely to have less density of old pages.

- the background writeback shall consider expired inodes first, just like the
  kupdate writeback

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 37+ messages in thread

end of thread, other threads:[~2011-05-10  4:53 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-04-20  8:03 [PATCH 0/6] writeback: moving expire targets for background/kupdate works v2 Wu Fengguang
2011-04-20  8:03 ` [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes() Wu Fengguang
2011-05-04 11:04   ` Christoph Hellwig
2011-05-04 11:13     ` Wu Fengguang
2011-04-20  8:03 ` [PATCH 2/6] writeback: introduce writeback_control.inodes_cleaned Wu Fengguang
2011-05-04 11:05   ` Christoph Hellwig
2011-05-04 11:11     ` Wu Fengguang
2011-05-04 11:16       ` Christoph Hellwig
2011-05-04 11:32         ` Wu Fengguang
2011-04-20  8:03 ` [PATCH 3/6] writeback: try more writeback as long as something was written Wu Fengguang
2011-04-20  8:03 ` [PATCH 4/6] writeback: the kupdate expire timestamp should be a moving target Wu Fengguang
2011-04-20  8:03 ` [PATCH 5/6] writeback: sync expired inodes first in background writeback Wu Fengguang
2011-04-20 23:40   ` Andrew Morton
2011-04-21  1:14     ` Wu Fengguang
2011-04-21  1:21       ` Wu Fengguang
2011-04-24  3:15     ` Wu Fengguang
2011-04-26 12:17       ` Jan Kara
2011-04-26 13:51         ` Wu Fengguang
2011-04-26 13:59           ` Wu Fengguang
2011-04-26 14:05           ` Wu Fengguang
2011-04-27 11:15           ` Wu Fengguang
2011-04-20  8:03 ` [PATCH 6/6] writeback: refill b_io iff empty Wu Fengguang
2011-05-04  7:39   ` Wu Fengguang
2011-05-05 16:37     ` Jan Kara
2011-05-05 16:47       ` Wu Fengguang
2011-05-06  5:29       ` Wu Fengguang
2011-05-06  8:42         ` [RFC][PATCH] writeback: limit number of moved inodes in queue_io() Wu Fengguang
2011-05-06 10:06           ` [RFC][PATCH v2] " Wu Fengguang
2011-05-06 23:06             ` Dave Chinner
2011-05-06 14:21         ` [PATCH 6/6] writeback: refill b_io iff empty Jan Kara
2011-05-10  4:31           ` Wu Fengguang
2011-05-10  4:53             ` Dave Chinner
  -- strict thread matches above, loose matches on Subject: below --
2011-04-19  3:00 [PATCH 0/6] writeback: moving expire targets for background/kupdate works Wu Fengguang
2011-04-19  3:00 ` [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes() Wu Fengguang
2010-07-22  5:09 [PATCH 0/6] [RFC] writeback: try to write older pages first Wu Fengguang
2010-07-22  5:09 ` [PATCH 1/6] writeback: pass writeback_control down to move_expired_inodes() Wu Fengguang
2010-07-23 18:16   ` Jan Kara
2010-07-26 10:44   ` Mel Gorman
2010-08-01 15:23   ` Minchan Kim

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).