Re: Is it really safe to use workqueues to drive expedited grace periods?

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Fri, Mar 03, 2017 at 11:30:49AM -0800, Tejun Heo wrote:
> Hello, Paul.
> 
> Sorry about the long delay.  Travel + sickness.  Just starting to
> catch up with things now.

No problem, I have been distracted by other projects in any case.  ;-)

> On Mon, Feb 13, 2017 at 04:16:00PM -0800, Paul E. McKenney wrote:
> > Thank you for the information!  So if I am to continue using workqueues
> > for expedited RCU grace periods, I believe that need to do the following:
> > 
> > 1.	Use alloc_workqueue() to create my own WQ_MEM_RECLAIM
> > 	workqueue.
> 
> This is only necessary if RCU can be in a dependency chain in the
> memory reclaim path - e.g. somebody doing synchronize_expedited_rcu()
> in some obscure writeback path; however, given how wildly RCU is used,
> I don't think this is a bad idea.  The only extra overhead which comes
> from it is memory used for an extra workqueue and a rescuer thread
> after all.

I suspect that SLAB_DESTROY_BY_RCU qualifies -- such a slab allocator
could have a great many slabs waiting for an RCU grace period.

> > 2.	Rework my workqueue handler to avoid blocking waiting for
> > 	the expedited grace period to complete.  I should be able
> > 	to do a small number of timed wait, but if I actually
> > 	wait for the grace period to complete, I might end up
> > 	hogging the reserved items.  (Or does my workqueue supply
> > 	them for me?  If so, so much the better!)
> 
> So, what the dedicated workqueue w/ WQ_MEMRECLAIM guarantees is that
> there will always be at least one worker thread which is executing
> work items from the workqueue.
> 
> As long as your workqueue usage guarantees forward progress - that is,
> as long as one work item in the workqueue won't block indefinitely on
> another work item on the same workqueue, you shouldn't need to reword
> the workqueue handler.
> 
> If there can be dependency chain among work items of the same
> WQ_MEMRECLAIM workqueue, often the best approach is breaking up the
> chain and putting them into their own WQ_MEMRECLAIM workqueues.

Thank you, good to know!

> > 3.	Concurrency would not be a problem -- there can be no more
> > 	four work elements in flight across both possible flavors
> > 	of expedited grace periods.
> 
> You usually don't have to worry about concurrency all that much with
> workqueues.  They'll provide the maximum the system can as long as
> that's possible.
> 
> If the four work items can depend on each other, it'd be best to put
> them in separate workqueues.  If not, there's nothing to worry about.

For a given pair, one can be acquiring a mutext that the other holds.
Ah, so if only one task was running, that would fail to make forward
progress.  That said, splitting would be a bit weird in this case,
because alternating grace periods for a given RCU flavor would need
to schedule work on a different workqueue.  Might be easier to do
mutex_trylock() and reschedule...

And thank you for the info, very helpful!

							Thanx, Paul