Re: Does it make sense to have the hdev workqueue serialized?

[Marcel Holtmann <marcel@holtmann.org>]

Hi Mat,

> The benefit would be in having no need to keep track of which context 
> functions are executing in.  It's been a big headache with the ERTM 
> and AMP changes, and there is a bunch of code that could work better 
> in process context if it didn't have to also handle calls from 
> tasklets.
> 
> That said, after learning more about how workqueues are implemented 
> now, it may be worthwhile to change the "use one single-threaded 
> workqueue for everything" assumption.  alloc_workqueue() has a 
> max_active parameter, and it is possible to have many work items 
> running concurrently.  Some of those threads could be suspended, like 
> Andre does in his patch.  Workqueues created with the old 
> create_workqueue() or create_singlethread_workqueue() have max_active 
> == 1, which enforces serialization on each processor.
> 
> 
> So there are two big questions:  Do we want to keep pushing everything 
> on the hdev workqueue, since workqueues are not as heavyweight as they 
> used to be?  And does it make sense to keep our workqueues serialized?
> 
> 
> Advantages of serialization:
>   * An HCI device is serialized by the transport anyway, so it makes 
> sense to match that model.
>   * Ordering is maintained.  The order of incoming HCI events may queue 
> work in a particular order and need to assume the work will be 
> executed in that order.
>   * Simplicity.
>   * No lock contention between multiple workers.
> 
> Advantages of not serializing:
>   * Takes advantage of SMP
>   * Workers can block without affecting the rest of the queue, enabling 
> workers to be long-lived and use state on the thread stack instead of 
> complicated lists of pending operations and dynamic allocation.
>   * We need to have proper locking to deal with user processes anyway, 
> so why not allow more concurrency internally.
>   * Some work can proceed while waiting for locks in other workers.
>   * Can use WQ_HIGHPRI to keep tx/rx data moving even when workers are 
> waiting for locks
> 
> 
> I think what's called for is a hybrid approach that serializes where 
> necessary, but uses multiple workqueues.  How about this:
> 
>   * Use the serialized hdev workqueue for rx/tx only.  This could use 
> WQ_HIGHPRI to help performance.

I think this is what we have to do to ensure that our event, cmd and ACL
and also SCO processing is not affected by anything else.

>   * Have a serialized workqueue for each L2CAP channel to handle 
> per-channel timeouts.

Why per channel? Wouldn't be one per ACL connection by enough?

>   * Have a global, non-serialized workqueue for stuff like sysfs and 
> mgmt to use.

And yes, one global workqueue for random tasks we have to do seems like
a good idea as well.

Regards

Marcel