Re: [PATCH 18/43] workqueue: reimplement workqueue flushing using color coded works

[Oleg Nesterov <oleg@redhat.com>]

On 03/02, Tejun Heo wrote:
>
> > Very naive question: why do we need cwq->nr_in_flight[] at all? Can't
> > cwq_dec_nr_in_flight(color) do
> >
> > 	if (WORK_NO_COLOR)
> > 		return;
> >
> > 	if (cwq->flush_color == -1)
> > 		return;
> >
> > 	BUG_ON((cwq->flush_color != color);
> > 	if (next_work && work_to_color(next_work) == color)
> > 		return;
> >
> > 	cwq->flush_color = -1;
> > 	if (atomic_dec_and_test(nr_cwqs_to_flush))
> > 		complete(first_flusher);
> >
> > ?
>
> Issues I can think of off the top of my head are...
>
> 1. Works are removed from the worklist before the callback is called
>    and inaccessible once the callback starts running, so completion
>    order can't be tracked with work structure themselves.
>
> 2. Even if it could be tracked that way, with later unified worklist,
>    it will require walking the worklist looking for works with
>    matching cwq.
>
> 3. Transfer of works to work->scheduled list and the linked works.
>    This ends up fragmenting the worklist before execution starts.

Yes, thanks. Not that I fully understand this right now, but I see
that the next patches make my naive suggestion impossible.

> >>  void flush_workqueue(struct workqueue_struct *wq)
> >>  {
> >> ...
> >> +	 * First flushers are responsible for cascading flushes and
> >> +	 * handling overflow.  Non-first flushers can simply return.
> >> +	 */
> >> +	if (wq->first_flusher != &this_flusher)
> >> +		return;
> >> +
> >> +	mutex_lock(&wq->flush_mutex);
> >> +
> >> +	wq->first_flusher = NULL;
> >> +
> >> +	BUG_ON(!list_empty(&this_flusher.list));
> >> +	BUG_ON(wq->flush_color != this_flusher.flush_color);
> >> +
> >> +	while (true) {
> >> +		struct wq_flusher *next, *tmp;
> >> +
> >> +		/* complete all the flushers sharing the current flush color */
> >> +		list_for_each_entry_safe(next, tmp, &wq->flusher_queue, list) {
> >> +			if (next->flush_color != wq->flush_color)
> >> +				break;
> >> +			list_del_init(&next->list);
> >> +			complete(&next->done);
> >
> > Is it really possible that "next" can ever have the same flush_color?
>
> Yeap, when all the colors are consumed, works wait in the overflow
> queue.  On the next flush completion, all works on the overflow list
> get the same color

Ah. I see.

Thanks Tejun,

Oleg.