Re: [QUESTION] xfs: Can we remove flush_workqueue from xlog_cil_push_now?

[Tejun Heo <tj@kernel.org>]

Hello,

On Wed, Sep 02, 2026 at 10:23:14AM -0400, Chuck Lever wrote:
> 
> On Wed, Sep 02, 2026 at 09:35:56AM +0200, Christoph Hellwig wrote:
> > On Mon, Aug 31, 2026 at 07:56:27AM +1000, Dave Chinner wrote:
> > > However, behavioural changes like this seem like an infrastructure
> > > regression, and trying to modify callers to hide that regression is
> > > a bandaid. This affects all UNBOUND workqueues, not just this one,
> > > and we do not need per-cpu scheduling for workqueues with such low
> > > work and concurrency counts - there can only be 4 pushs scheduled
> > > and running concurrently at any time on this work queue. Hence we
> > > don't even need NUMA queuing - a single global queue would work just
> > > fine...
> >
> > Yeah, a lot of the recent workqueue changes had a lot of adverse effects.
> > We really need to come up with a way to queue work to task context (or a
> > different task context) that doesn't require reverse engineering "smart"
> > behavior in the backend.  This has become a never ending whack-a-mole
> > unfortunately.
> 
> Agreed.

Well, the changes aren't gratuituous. They're mostly driven by the
increasing number of cpus in systems. Hardware has been scaling pretty fast
and workqueue needs to keep up. Sure, there may be things that could have
gone smoother but you'd need to solve the same problems no matter what.

> So, I've been chasing the other end of this trade-off. On the NFS client,
> rpciod, nfsiod, and xprtiod see sustained completion traffic from many
> CPUs, and with the default WQ_AFFN_CACHE_SHARD scope they lose a large
> fraction of their cycles to the pool lock on enqueue and dequeue. The
> fix there is a finer affinity scope (WQ_AFFN_SMT), so more pools and
> fewer CPUs per lock:
> 
>   https://lore.kernel.org/linux-nfs/20260831-performance-v1-0-8d9fd9b67f96@kernel.org/

And this was worse before because we had a single pool per node for all
unbound workqueues. You're using exactly the new feature that's made
available to solve scalability problems.

> That is the converse of Ran's problem. flush_workqueue_prep_pwqs()
> walks every pwq and takes the pool lock once per pool change along the
> way, so a finer scope makes each flush take more locks, and a coarser
> scope makes the producers contend more. The pwq walk itself is
> per-CPU at every scope, which is the part no scope setting removes.
> 
> For Ran's immediate purpose, patch 5/8 of that series may be useful:
> 
>   https://lore.kernel.org/linux-nfs/20260831-performance-v1-5-8d9fd9b67f96@kernel.org/
> 
> It exports alloc_workqueue_attrs(), apply_workqueue_attrs(), and
> free_workqueue_attrs(), so a module can set affn_scope on its own
> workqueue after alloc_workqueue().
> 
> With that, xfs_cil could set WQ_AFFN_SYSTEM on xc_push_wq and get the
> single global pool Dave describes, without touching the boot-time
> default or relying on WQ_SYSFS, which XFS_WQFLAGS() adds only in
> DEBUG builds. That would let Ran measure the flush cost against a
> one-pool walk directly rather than through the per-NUMA proxy in the
> table.

Try http://lkml.kernel.org/r/20260901210929.3092513-1-tj@kernel.org first?

Thanks.

-- 
tejun