Re: [PATCH net-next v1 2/2] net/rds: Give each connection its own workqueue

[Allison Henderson <allison.henderson@oracle.com>]

On Tue, 2025-11-04 at 15:57 +0100, Paolo Abeni wrote:
> On 10/29/25 6:46 PM, Allison Henderson wrote:
> > From: Allison Henderson <allison.henderson@oracle.com>
> > 
> > RDS was written to require ordered workqueues for "cp->cp_wq":
> > Work is executed in the order scheduled, one item at a time.
> > 
> > If these workqueues are shared across connections,
> > then work executed on behalf of one connection blocks work
> > scheduled for a different and unrelated connection.
> > 
> > Luckily we don't need to share these workqueues.
> > While it obviously makes sense to limit the number of
> > workers (processes) that ought to be allocated on a system,
> > a workqueue that doesn't have a rescue worker attached,
> > has a tiny footprint compared to the connection as a whole:
> > A workqueue costs ~800 bytes, while an RDS/IB connection
> > totals ~5 MBytes.
> 
> Still a workqueue per connection feels overkill. Have you considered
> moving to WQ_PERCPU for rds_wq? Why does not fit?
> 
> Thanks,
> 
> Paolo
> 
Hi Paolo

I hadnt thought of WQ_PERCPU before, so I did some digging on it.  In our case though, we need FIFO behavior per-
connection, so if we switched to queues per cpu, we'd have to pin a CPU to a connection to get the right behavior.  And
then that brings back head of the line blocking since now all the items on that queue have to share that CPU even if the
other CPUs are idle.  So it wouldn't quite be a synonymous solution for what we're trying to do in this case.  I hope
that made sense?  Let me know what you think.

Thank you,
Allison