Re: merging the per-bdi writeback patchset

[Jens Axboe <jens.axboe@oracle.com>]

On Tue, Jun 23 2009, Andrew Morton wrote:
> On Tue, 23 Jun 2009 10:55:05 +0200 Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Tue, Jun 23 2009, Andrew Morton wrote:
> > > On Tue, 23 Jun 2009 10:11:56 +0200 Jens Axboe <jens.axboe@oracle.com> wrote:
> > > 
> > > > Things are looking good for this patchset and it's been in -next for
> > > > almost a week without any reports of problems. So I'd like to merge it
> > > > for 2.6.31 if at all possible. Any objections?
> > > 
> > > erk.  I was rather expecting I'd have time to have a look at it all.
> > 
> > OK, we can wait if we have to, just trying to avoid having to keep this
> > fresh for one full cycle. I have posted this patchset 11 times though
> > over months, so it's not like it's a new piece of work :-)
> 
> Yeah, sorry.  
> 
> > > It's unclear to me actually _why_ the performance changes which were
> > > observed have actually occurred.  In fact it's a bit unclear (to me)
> > > why the patchset was written and what it sets out to achieve :(
> > 
> > It started out trying to get rid of the pdflush uneven writeout. If you
> > look at various pdflush intensive workloads, even on a single disk you
> > often have 5 or more pdflush threads working the same device. It's just
> > not optimal.
> 
> That's a bug, isn't it?  This
> 
> 		/* Is another pdflush already flushing this queue? */
> 		if (current_is_pdflush() && !writeback_acquire(bdi))
> 			break;
> 
> isn't working.

But that's on a per-inode basis. I didn't look further into the problem
to be honest, just noticed that you very quickly get a handful of
pdflush threads ticking along.

> > Another issue was starvation with request allocation. Given
> > that pdflush does non-blocking writes (it has to, by design), pdflush
> > can potentially be starved if someone else is working the device.
> 
> hm, true.  100% starved, or just "slowed down"?  The latter I trust -
> otherwise there are still failure modes?

Just slowed down, I'm suspecting this is where the lumpiness comes from
as well. At least in the cases I have seen, in theory you could starve
the pdflush thread indefinitely.

> > > A long time ago the XFS guys (Dave Chinner iirc) said that XFS needs
> > > more than one thread per device to keep the device saturated.  Did that
> > > get addressed?
> > 
> > It supports up to 32-threads per device, but Chinner et all have been
> > silent. So the support is there and there's a
> > super_operations->inode_get_wb() to map a dirty inode to a writeback
> > device. Nobody is doing that yet though.
> 
> OK.
> 
> How many kernel threads do the 1000-spindle people end up with?

If all 1000 spindles are exposed and flushing dirty data, you get 1000
threads. Realistically, you'll likely use some sort of dm/md frontend
though. And then you only get 1 thread per dm/md device.

-- 
Jens Axboe