Re: [PATCH UPDATED] workqueue: add documentation

[Dave Chinner <david@fromorbit.com>]

Hi Tejun,

A couple more queustions on cmwq.

On Fri, Sep 10, 2010 at 04:55:21PM +0200, Tejun Heo wrote:
.....
> +  WQ_HIGHPRI
> +
> +	Work items of a highpri wq are queued at the head of the
> +	worklist of the target gcwq and start execution regardless of
> +	the current concurrency level.  In other words, highpri work
> +	items will always start execution as soon as execution
> +	resource is available.
> +
> +	Ordering among highpri work items is preserved - a highpri
> +	work item queued after another highpri work item will start
> +	execution after the earlier highpri work item starts.
> +
> +	Although highpri work items are not held back by other
> +	runnable work items, they still contribute to the concurrency
> +	level.  Highpri work items in runnable state will prevent
> +	non-highpri work items from starting execution.
> +
> +	This flag is meaningless for unbound wq.

We talked about this for XFS w.r.t. the xfslogd IO completion
work items to be promoted ahead of data IO completion items and
that has worked fine. This appears to gives us only two
levels of priority, or from an user point of view, two levels of
dependency between workqueue item execution.

Thinking about the XFS situation more, we actually have three levels
of dependency: xfslogd -> xfsdatad -> xfsconvertd. That is, we defer
long running, blocking items from xfsdatad to xfsconvertd so we
don't block the xfsdatad from continuing to process data IO
completion items. How do we guarantee that the xfsconvertd work
items won't prevent/excessively delay processing of xfsdatad items?


> +@max_active determines the maximum number of execution contexts per
> +CPU which can be assigned to the work items of a wq.  For example,
> +with @max_active of 16, at most 16 work items of the wq can be
> +executing at the same time per CPU.

I think the reason you were seeing XFS blow this out of the water is
that every IO completion for a write beyond EOF (i.e. every single
one for an extending streaming write) will require inode locking to
update file size. If the inode is locked, then the item will
delay(1), and the cmwq controller will run the next item in a new
worker. That will then block in delay(1) 'cause it can't get the
inode lock, as so on....

As such, I can't see that increasing the max_active count for XFS is
a good thing - all it will do is cause larger blockages to occur....

> +6. Guidelines
> +
> +* Do not forget to use WQ_RESCUER if a wq may process work items which
> +  are used during memory reclaim.  Each wq with WQ_RESCUER set has one
> +  rescuer thread reserved for it.  If there is dependency among
> +  multiple work items used during memory reclaim, they should be
> +  queued to separate wq each with WQ_RESCUER.
> +
> +* Unless strict ordering is required, there is no need to use ST wq.
> +
> +* Unless there is a specific need, using 0 for @nr_active is
                                                  max_active?


Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com