From: Wu Fengguang <fengguang.wu@intel.com>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Theodore Tso <tytso@mit.edu>,
Christoph Hellwig <hch@infradead.org>,
Dave Chinner <david@fromorbit.com>,
Chris Mason <chris.mason@oracle.com>,
"Li, Shaohua" <shaohua.li@intel.com>,
Myklebust Trond <Trond.Myklebust@netapp.com>,
"jens.axboe@oracle.com" <jens.axboe@oracle.com>,
Jan Kara <jack@suse.cz>, Nick Piggin <npiggin@suse.de>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
Richard Kennedy <richard@rsk.demon.co.uk>
Subject: Re: [PATCH 06/45] writeback: use larger ratelimit when dirty_exceeded
Date: Wed, 7 Oct 2009 17:17:53 +0800 [thread overview]
Message-ID: <20091007091753.GB5646@localhost> (raw)
In-Reply-To: <1254905599.26976.217.camel@twins>
On Wed, Oct 07, 2009 at 04:53:19PM +0800, Peter Zijlstra wrote:
> On Wed, 2009-10-07 at 15:38 +0800, Wu Fengguang wrote:
> > plain text document attachment
> > (writeback-ratelimit-on-dirty-exceeded.patch)
> > When dirty_exceeded, use ratelimit = ratelimit_pages/8, allowing it to
> > scale up to 512KB for memory bounty systems. This is more efficient than
> > the original 8 pages, and won't risk exceeding the dirty limit too much.
> >
> > Given the larger ratelimit value, we can safely ignore the low bound
> > check in sync_writeback_pages.
> >
> > dirty_exceeded is more likely to be seen when there are multiple dirty
> > processes. In which case the lowered ratelimit will help reduce their
> > overall wait time (latency) in the throttled queue.
>
> Don't forget that ratelimit_pages is a per-cpu limit. So the total error
> on the dirty limit scales with the number of cpus.
Ah yes! Given that a typical NUMA configuration would be to equip each CPU
with >1GB memory, 512KB is not a big error :)
> Other than that, I guess this patch needs numbers ;-)
Lots of numbers, hehe. Basically I see ~25ms wait time for ext2/3/4
and btrfs (with this patch and the dirty exceed timestamp patch).
One cp process:
[ 3404.554478] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.570407] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 3404.601166] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.632872] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.663633] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.694198] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.724879] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.756655] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.787350] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.817950] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.848640] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3404.880556] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 3404.911372] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.942006] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3404.972712] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.002672] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.034480] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.065114] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.095870] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.126453] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.158043] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.188801] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.219327] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.249922] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.281869] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.312533] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.343194] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.373761] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.410828] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 3405.441993] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.473266] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.505746] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.537103] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.568327] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 3405.600730] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.632039] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.663326] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.695674] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.727031] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.758146] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.790743] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 3405.822129] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.853427] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 3405.885889] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.917116] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 3405.948411] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
Two:
[ 9393.138529] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9393.171042] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9393.344709] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9393.377045] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9393.555611] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9393.586298] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9393.747218] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9393.777825] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9393.808456] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9393.964086] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.120223] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.273854] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9394.304842] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.334474] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.480031] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.527769] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9394.557919] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.590690] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9394.621626] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.654424] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9394.678289] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9394.709837] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.742912] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.774576] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.806149] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.822272] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 9394.852609] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.886215] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.915804] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9394.947044] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9394.979504] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9395.010944] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9395.042221] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9395.074640] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
[ 9395.105938] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9395.137157] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9395.169668] write_bandwidth: comm=btrfs-endio-wri pages=192 time=28ms
Four:
[ 9435.829736] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.702953] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9436.719957] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.753598] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms
[ 9436.770935] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.804367] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9436.818913] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.854302] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms
[ 9436.870489] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9436.903449] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9443.140282] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9443.155674] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 9443.180418] write_bandwidth: comm=btrfs-endio-wri pages=192 time=24ms
[ 9443.697286] write_bandwidth: comm=btrfs-endio-wri pages=192 time=20ms
[ 9443.714321] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9443.748455] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9443.763764] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9443.796788] write_bandwidth: comm=btrfs-endio-wri pages=192 time=36ms
[ 9443.814309] write_bandwidth: comm=btrfs-endio-wri pages=192 time=16ms
[ 9446.493680] write_bandwidth: comm=btrfs-endio-wri pages=193 time=16ms
[ 9446.528486] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
[ 9446.888774] write_bandwidth: comm=btrfs-endio-wri pages=192 time=40ms
[ 9446.903518] write_bandwidth: comm=btrfs-endio-wri pages=192 time=12ms
[ 9446.933610] write_bandwidth: comm=btrfs-endio-wri pages=192 time=32ms
Thanks,
Fengguang
next prev parent reply other threads:[~2009-10-07 9:18 UTC|newest]
Thread overview: 116+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-10-07 7:38 [PATCH 00/45] some writeback experiments Wu Fengguang
2009-10-07 7:38 ` [PATCH 01/45] writeback: reduce calls to global_page_state in balance_dirty_pages() Wu Fengguang
2009-10-09 15:12 ` Jan Kara
2009-10-09 15:18 ` Peter Zijlstra
2009-10-09 15:47 ` Jan Kara
2009-10-11 2:28 ` Wu Fengguang
2009-10-11 7:44 ` Peter Zijlstra
2009-10-11 10:50 ` Wu Fengguang
2009-10-11 10:58 ` Peter Zijlstra
2009-10-11 11:25 ` Peter Zijlstra
2009-10-12 1:26 ` Wu Fengguang
2009-10-12 9:07 ` Peter Zijlstra
2009-10-12 9:24 ` Wu Fengguang
2009-10-10 21:33 ` Wu Fengguang
2009-10-12 21:18 ` Jan Kara
2009-10-13 3:24 ` Wu Fengguang
2009-10-13 8:41 ` Peter Zijlstra
2009-10-13 18:12 ` Jan Kara
2009-10-13 18:28 ` Peter Zijlstra
2009-10-14 1:38 ` Wu Fengguang
2009-10-14 11:22 ` Peter Zijlstra
2009-10-17 5:30 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 02/45] writeback: reduce calculation of bdi dirty thresholds Wu Fengguang
2009-10-07 7:38 ` [PATCH 03/45] ext4: remove unused parameter wbc from __ext4_journalled_writepage() Wu Fengguang
2009-10-07 7:38 ` [PATCH 04/45] writeback: remove unused nonblocking and congestion checks Wu Fengguang
2009-10-09 15:26 ` Jan Kara
2009-10-10 13:47 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 05/45] writeback: remove the always false bdi_cap_writeback_dirty() test Wu Fengguang
2009-10-07 7:38 ` [PATCH 06/45] writeback: use larger ratelimit when dirty_exceeded Wu Fengguang
2009-10-07 8:53 ` Peter Zijlstra
2009-10-07 9:17 ` Wu Fengguang [this message]
2009-10-07 7:38 ` [PATCH 07/45] writeback: dont redirty tail an inode with dirty pages Wu Fengguang
2009-10-09 15:45 ` Jan Kara
2009-10-07 7:38 ` [PATCH 08/45] writeback: quit on wrap for .range_cyclic (write_cache_pages) Wu Fengguang
2009-10-07 7:38 ` [PATCH 09/45] writeback: quit on wrap for .range_cyclic (pohmelfs) Wu Fengguang
2009-10-07 12:32 ` Evgeniy Polyakov
2009-10-07 14:23 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 10/45] writeback: quit on wrap for .range_cyclic (btrfs) Wu Fengguang
2009-10-07 7:38 ` [PATCH 11/45] writeback: quit on wrap for .range_cyclic (cifs) Wu Fengguang
2009-10-07 7:38 ` [PATCH 12/45] writeback: quit on wrap for .range_cyclic (ext4) Wu Fengguang
2009-10-07 7:38 ` [PATCH 13/45] writeback: quit on wrap for .range_cyclic (gfs2) Wu Fengguang
2009-10-07 7:38 ` [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs) Wu Fengguang
2009-10-07 7:38 ` [PATCH 15/45] writeback: fix queue_io() ordering Wu Fengguang
2009-10-07 7:38 ` [PATCH 16/45] writeback: merge for_kupdate and !for_kupdate cases Wu Fengguang
2009-10-07 7:38 ` [PATCH 17/45] writeback: only allow two background writeback works Wu Fengguang
2009-10-07 7:38 ` [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages() Wu Fengguang
2009-10-08 1:01 ` KAMEZAWA Hiroyuki
2009-10-08 1:58 ` Wu Fengguang
2009-10-08 2:40 ` KAMEZAWA Hiroyuki
2009-10-08 4:01 ` Wu Fengguang
2009-10-08 5:59 ` KAMEZAWA Hiroyuki
2009-10-08 6:07 ` Wu Fengguang
2009-10-08 6:28 ` Wu Fengguang
2009-10-08 6:39 ` KAMEZAWA Hiroyuki
2009-10-08 8:08 ` Peter Zijlstra
2009-10-08 8:11 ` KAMEZAWA Hiroyuki
2009-10-08 8:36 ` Jens Axboe
2009-10-09 2:52 ` [PATCH] writeback: account IO throttling wait as iowait Wu Fengguang
2009-10-09 10:41 ` Jens Axboe
2009-10-09 10:58 ` Wu Fengguang
2009-10-09 11:01 ` Jens Axboe
2009-10-08 8:05 ` [PATCH 18/45] writeback: introduce wait queue for balance_dirty_pages() Peter Zijlstra
2009-10-07 7:38 ` [PATCH 19/45] writeback: remove the loop in balance_dirty_pages() Wu Fengguang
2009-10-07 7:38 ` [PATCH 20/45] NFS: introduce writeback wait queue Wu Fengguang
2009-10-07 8:53 ` Peter Zijlstra
2009-10-07 9:07 ` Wu Fengguang
2009-10-07 9:15 ` Peter Zijlstra
2009-10-07 9:19 ` Wu Fengguang
2009-10-07 9:17 ` Nick Piggin
2009-10-07 9:52 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 21/45] writeback: estimate bdi write bandwidth Wu Fengguang
2009-10-07 8:53 ` Peter Zijlstra
2009-10-07 9:39 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 22/45] writeback: show bdi write bandwidth in debugfs Wu Fengguang
2009-10-07 7:38 ` [PATCH 23/45] writeback: kill space in debugfs item name Wu Fengguang
2009-10-07 7:38 ` [PATCH 24/45] writeback: remove global nr_to_write and use timeout instead Wu Fengguang
2009-10-07 7:38 ` [PATCH 25/45] writeback: convert wbc.nr_to_write to per-file parameter Wu Fengguang
2009-10-07 7:38 ` [PATCH 26/45] block: pass the non-rotational queue flag to backing_dev_info Wu Fengguang
2009-10-07 7:38 ` [PATCH 27/45] writeback: introduce wbc.for_background Wu Fengguang
2009-10-07 7:38 ` [PATCH 28/45] writeback: introduce wbc.nr_segments Wu Fengguang
2009-10-07 7:38 ` [PATCH 29/45] writeback: fix the shmem AOP_WRITEPAGE_ACTIVATE case Wu Fengguang
2009-10-07 11:57 ` Hugh Dickins
2009-10-07 14:00 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 30/45] vmscan: lumpy pageout Wu Fengguang
2009-10-07 7:38 ` [PATCH 31/45] writeback: sync old inodes first in background writeback Wu Fengguang
2010-07-12 3:01 ` Christoph Hellwig
2010-07-12 15:24 ` Wu Fengguang
2009-10-07 7:38 ` [PATCH 32/45] writeback: update kupdate expire timestamp on each scan of b_io Wu Fengguang
2009-10-07 7:38 ` [PATCH 34/45] writeback: sync livelock - kick background writeback Wu Fengguang
2009-10-07 7:38 ` [PATCH 35/45] writeback: sync livelock - use single timestamp for whole sync work Wu Fengguang
2009-10-07 7:38 ` [PATCH 36/45] writeback: sync livelock - curb dirty speed for inodes to be synced Wu Fengguang
2009-10-07 7:38 ` [PATCH 37/45] writeback: use timestamp to indicate dirty exceeded Wu Fengguang
2009-10-07 7:38 ` [PATCH 38/45] writeback: introduce queue b_more_io_wait Wu Fengguang
2009-10-07 7:38 ` [PATCH 39/45] writeback: remove wbc.more_io Wu Fengguang
2009-10-07 7:38 ` [PATCH 40/45] writeback: requeue_io_wait() on I_SYNC locked inode Wu Fengguang
2009-10-07 7:38 ` [PATCH 41/45] writeback: requeue_io_wait() on pages_skipped inode Wu Fengguang
2009-10-07 7:39 ` [PATCH 42/45] writeback: requeue_io_wait() on blocked inode Wu Fengguang
2009-10-07 7:39 ` [PATCH 43/45] writeback: requeue_io_wait() on fs redirtied inode Wu Fengguang
2009-10-07 7:39 ` [PATCH 44/45] NFS: remove NFS_INO_FLUSHING lock Wu Fengguang
2009-10-07 13:11 ` Peter Staubach
2009-10-07 13:32 ` Wu Fengguang
2009-10-07 13:59 ` Peter Staubach
2009-10-08 1:44 ` Wu Fengguang
2009-10-07 7:39 ` [PATCH 45/45] btrfs: fix race on syncing the btree inode Wu Fengguang
2009-10-07 8:53 ` [PATCH 00/45] some writeback experiments Peter Zijlstra
2009-10-07 10:17 ` [PATCH 14/45] writeback: quit on wrap for .range_cyclic (afs) David Howells
2009-10-07 10:21 ` Nick Piggin
2009-10-07 10:47 ` Wu Fengguang
2009-10-07 11:23 ` Nick Piggin
2009-10-07 12:21 ` Wu Fengguang
2009-10-07 13:47 ` [PATCH 00/45] some writeback experiments Peter Staubach
2009-10-07 15:18 ` Wu Fengguang
2009-10-08 5:33 ` Wu Fengguang
2009-10-08 5:44 ` Wu Fengguang
2009-10-07 14:26 ` Theodore Tso
2009-10-07 14:45 ` Wu Fengguang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20091007091753.GB5646@localhost \
--to=fengguang.wu@intel.com \
--cc=Trond.Myklebust@netapp.com \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=chris.mason@oracle.com \
--cc=david@fromorbit.com \
--cc=hch@infradead.org \
--cc=jack@suse.cz \
--cc=jens.axboe@oracle.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=npiggin@suse.de \
--cc=richard@rsk.demon.co.uk \
--cc=shaohua.li@intel.com \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).