Re: [RFC] page-writeback: move indoes from one superblock together

[Shaohua Li <shaohua.li@intel.com>]

On Thu, Sep 24, 2009 at 03:14:15PM +0800, Wu, Fengguang wrote:
> On Thu, Sep 24, 2009 at 02:54:20PM +0800, Li, Shaohua wrote:
> > __mark_inode_dirty adds inode to wb dirty list in random order. If a disk has
> > several partitions, writeback might keep spindle moving between partitions.
> > To reduce the move, better write big chunk of one partition and then move to
> > another. Inodes from one fs usually are in one partion, so idealy move indoes
> > from one fs together should reduce spindle move. This patch tries to address
> > this. Before per-bdi writeback is added, the behavior is write indoes
> > from one fs first and then another, so the patch restores previous behavior.
> > The loop in the patch is a bit ugly, should we add a dirty list for each
> > superblock in bdi_writeback?
> > 
> > Test in a two partition disk with attached fio script shows about 3% ~ 6%
> > improvement.
> 
> Reviewed-by: Wu Fengguang <fengguang.wu@intel.com>
> 
> Good idea! The optimization looks good to me, it addresses one
> weakness of per-bdi writeback.
> 
> But one problem is, Jan Kara and me are planning to remove b_io and
> hence this move_expired_inodes() function. Not sure how to do this
> optimization without b_io.
> 
> > Signed-off-by: Shaohua Li <shaohua.li@intel.com>
> > 
> > diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> > index 8e1e5e1..fc87730 100644
> > --- a/fs/fs-writeback.c
> > +++ b/fs/fs-writeback.c
> > @@ -324,13 +324,29 @@ static void move_expired_inodes(struct list_head *delaying_queue,
> >  			       struct list_head *dispatch_queue,
> >  				unsigned long *older_than_this)
> >  {
> > +	LIST_HEAD(tmp);
> > +	struct list_head *pos, *node;
> > +	struct super_block *sb;
> > +	struct inode *inode;
> > +
> >  	while (!list_empty(delaying_queue)) {
> > -		struct inode *inode = list_entry(delaying_queue->prev,
> > -						struct inode, i_list);
> > +		inode = list_entry(delaying_queue->prev, struct inode, i_list);
> >  		if (older_than_this &&
> >  		    inode_dirtied_after(inode, *older_than_this))
> >  			break;
> > -		list_move(&inode->i_list, dispatch_queue);
> > +		list_move(&inode->i_list, &tmp);
> > +	}
> > +
> > +	/* Move indoes from one superblock together */
> > +	while (!list_empty(&tmp)) {
> > +		inode = list_entry(tmp.prev, struct inode, i_list);
> > +		sb = inode->i_sb;
> > +		list_for_each_prev_safe(pos, node, &tmp) {
> 
> We are in spin lock, so not necessary to use the safe version?
it's to protect list delete.
 
> > +			struct inode *inode = list_entry(pos,
> 
> Could just reuse inode.
oops, forgot to remove it when moveing inode to global.


__mark_inode_dirty adds inode to wb dirty list in random order. If a disk has
several partitions, writeback might keep spindle moving between partitions.
To reduce the move, better write big chunk of one partition and then move to
another. Inodes from one fs usually are in one partion, so idealy move indoes
from one fs together should reduce spindle move. This patch tries to address
this. Before per-bdi writeback is added, the behavior is write indoes
from one fs first and then another, so the patch restores previous behavior.
The loop in the patch is a bit ugly, should we add a dirty list for each
superblock in bdi_writeback?

Test in a two partition disk with attached fio script shows about 3% ~ 6%
improvement.

Signed-off-by: Shaohua Li <shaohua.li@intel.com>
Reviewed-by: Wu Fengguang <fengguang.wu@intel.com>

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 8e1e5e1..303a1c5 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -324,13 +324,28 @@ static void move_expired_inodes(struct list_head *delaying_queue,
 			       struct list_head *dispatch_queue,
 				unsigned long *older_than_this)
 {
+	LIST_HEAD(tmp);
+	struct list_head *pos, *node;
+	struct super_block *sb;
+	struct inode *inode;
+
 	while (!list_empty(delaying_queue)) {
-		struct inode *inode = list_entry(delaying_queue->prev,
-						struct inode, i_list);
+		inode = list_entry(delaying_queue->prev, struct inode, i_list);
 		if (older_than_this &&
 		    inode_dirtied_after(inode, *older_than_this))
 			break;
-		list_move(&inode->i_list, dispatch_queue);
+		list_move(&inode->i_list, &tmp);
+	}
+
+	/* Move indoes from one superblock together */
+	while (!list_empty(&tmp)) {
+		inode = list_entry(tmp.prev, struct inode, i_list);
+		sb = inode->i_sb;
+		list_for_each_prev_safe(pos, node, &tmp) {
+			inode = list_entry(pos, struct inode, i_list);
+			if (inode->i_sb == sb)
+				list_move(&inode->i_list, dispatch_queue);
+		}
 	}
 }