Re: [PATCH 2/7] writeback: switch to per-bdi threads for flushing data

[Jens Axboe <jens.axboe@oracle.com>]

On Mon, Mar 16 2009, Dave Chinner wrote:
> On Fri, Mar 13, 2009 at 11:54:46AM +0100, Jens Axboe wrote:
> > On Thu, Mar 12 2009, Andrew Morton wrote:
> > > On Thu, 12 Mar 2009 15:33:43 +0100 Jens Axboe <jens.axboe@oracle.com> wrote:
> > > Bear in mind that the XFS guys found that one thread per fs had
> > > insufficient CPU power to keep up with fast devices.
> > 
> > Yes, I definitely want to experiment with > 1 thread per device in the
> > near future.
> 
> The question here is how to do this efficiently. Even if XFS is
> operating on a single device, it is not optimal just to throw
> multiple threads at the bdi. Ideally we want a thread per region
> (allocation group) of the filesystem as each allocation group has
> it's own inode cache (radix tree) to traverse. These traversals can
> be done completely in parallel and won't contend either at the
> traversal level or in the IO hardware....
> 
> i.e. what I'd like to see is the ability so any new flushing
> mechanism to be able to offload responsibility of tracking,
> traversing and flushing of dirty inodes to the filesystem.
> Filesystems that don't do such things could use a generic
> bdi-based implementation.
> 
> FWIW, we also want to avoid the current pattern of flushing
> data, then the inode, then data, then the inode, ....
> By offloading into the filesystem, this writeback ordering can
> be done as efficiently as possible for each given filesystem.
> XFs already has all the hooks to be able to do this
> effectively....
> 
> I know that Christoph was doing some work towards this end;
> perhaps he can throw his 2c worth in here...

This is very useful feedback, thanks Dave. So on the filesystem vs bdi
side, XFS could register a bdi per allocation group. Then set the proper
inode->mapping->backing_dev_info from sb->s_op->alloc_inode and
__mark_inode_dirty() should get the placement right. For private
traverse and flush, provide some address_space op to override
generic_sync_bdi_inodes().

It sounds like I should move the bdi flushing bits separate from the bdi
itself. Embed one in the bdi, but allow outside registration of others.
Will fit better with the need for more than one flusher per backing
device.

-- 
Jens Axboe