[PATCH RFC 0/3] Refactor the workqueue allocations

[PATCH RFC 0/3] Refactor the workqueue allocations

[Breno Leitao]

POC for Tejun's idea to unify the per-cpu and unbound workqueues, as
suggested in http://lkml.kernel.org/r/ak569WYSm3ygKl1-@slm.duckdns.org

This series only unifies the allocation part (alloc_pwq()). It makes
alloc_pwq() a single allocator that returns a pwq backed by either a static
per-cpu pool (get_percpu_pool) or an unbound pool (get_unbound_pool), reusing
the existing static per-cpu pools. The install/link path is unchanged and all
patches are behavior-neutral. This sets things up to unify the install side
later.

Is this the right direction?

Questions to follow up:

1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects
   non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well,
   so the unification can happen later (apply_workqueue_attrs_locked())?

2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be
   treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node
   accounting (wq_node_nr_active) and lose percpu's per-cpu max_active
   (pwq->nr_active).

3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
   unified internals (single install path, per-cpu accounting special-cased)
   with WQ_UNBOUND staying for now -- or something more radical (the flags gone
   entirely, percpu purely an affinity value)?

Tests: 

 1) I've tested this on x86 and arm64, with regular tests
 2) I've hacked up a workqueue test suite, which has a bunch of tests,
    and this is what I am using to test these changes:
	https://github.com/leitao/wqtest

Thanks,
-breno

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (3):
      workqueue: introduce alloc_pwq()
      workqueue: allocate percpu pwqs through alloc_pwq()
      workqueue: factor out alloc_and_link_percpu_pwqs()

 kernel/workqueue.c | 96 ++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 60 insertions(+), 36 deletions(-)
---
base-commit: b9810cd75b9fb56a3425d391cba3f608502bd474
change-id: 20260709-tejun1-c2aaf36f54a4

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH RFC 1/3] workqueue: introduce alloc_pwq()

[Breno Leitao]

Factor the static per-cpu pool lookup out of alloc_and_link_pwqs() into
get_percpu_pool(), and add alloc_pwq() -- a common pwq allocator that
picks the backing pool by workqueue type: a percpu workqueue uses the
static per-cpu pool for @cpu (get_percpu_pool()), an unbound workqueue
uses a pool matching @attrs (get_unbound_pool()).  The paired
put_pwq_pool() releases only unbound pools; static per-cpu pools are
permanent.

alloc_pwq() replaces alloc_unbound_pwq() and gives the workqueue core a
single entry point that can produce a pwq pointing at either kind of pool
-- the building block for unifying percpu and unbound workqueues.

No functional change: the current alloc_pwq() callers only operate on
unbound workqueues, so its percpu branch is not exercised yet.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 kernel/workqueue.c | 61 +++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 19 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 94f37ea762365..c32e173af2335 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -5354,8 +5354,38 @@ static void link_pwq(struct pool_workqueue *pwq)
 	list_add_tail_rcu(&pwq->pwqs_node, &wq->pwqs);
 }
 
-/* obtain a pool matching @attr and create a pwq associating the pool and @wq */
-static struct pool_workqueue *alloc_unbound_pwq(struct workqueue_struct *wq,
+/* Return the static per-cpu worker_pool that backs @wq on @cpu. */
+static struct worker_pool *get_percpu_pool(struct workqueue_struct *wq, int cpu)
+{
+	struct worker_pool __percpu *pools;
+	bool highpri;
+
+	WARN_ON(wq->flags & WQ_UNBOUND);
+
+	highpri = wq->flags & WQ_HIGHPRI;
+	if (wq->flags & WQ_BH)
+		pools = bh_worker_pools;
+	else
+		pools = cpu_worker_pools;
+
+	return &per_cpu_ptr(pools, cpu)[highpri];
+}
+
+/* release the pool obtained for @wq's pwq; only unbound pools are refcounted */
+static void put_pwq_pool(struct workqueue_struct *wq, struct worker_pool *pool)
+{
+	if (!(wq->flags & WQ_UNBOUND))
+		return;
+
+	put_unbound_pool(pool);
+}
+
+/*
+ * Obtain the pool backing @wq on @cpu and create a pwq associating it with @wq.
+ * A percpu @wq uses the static per-cpu pool for @cpu; an unbound @wq uses a
+ * pool matching @attrs.
+ */
+static struct pool_workqueue *alloc_pwq(struct workqueue_struct *wq, int cpu,
 					const struct workqueue_attrs *attrs)
 {
 	struct worker_pool *pool;
@@ -5363,13 +5393,16 @@ static struct pool_workqueue *alloc_unbound_pwq(struct workqueue_struct *wq,
 
 	lockdep_assert_held(&wq_pool_mutex);
 
-	pool = get_unbound_pool(attrs);
+	if (wq->flags & WQ_UNBOUND)
+		pool = get_unbound_pool(attrs);
+	else
+		pool = get_percpu_pool(wq, cpu);
 	if (!pool)
 		return NULL;
 
 	pwq = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL, pool->node);
 	if (!pwq) {
-		put_unbound_pool(pool);
+		put_pwq_pool(wq, pool);
 		return NULL;
 	}
 
@@ -5478,7 +5511,7 @@ apply_wqattrs_prepare(struct workqueue_struct *wq,
 	copy_workqueue_attrs(new_attrs, attrs);
 	wqattrs_actualize_cpumask(new_attrs, unbound_cpumask);
 	cpumask_copy(new_attrs->__pod_cpumask, new_attrs->cpumask);
-	ctx->dfl_pwq = alloc_unbound_pwq(wq, new_attrs);
+	ctx->dfl_pwq = alloc_pwq(wq, -1, new_attrs);
 	if (!ctx->dfl_pwq)
 		goto out_free;
 
@@ -5488,7 +5521,7 @@ apply_wqattrs_prepare(struct workqueue_struct *wq,
 			ctx->pwq_tbl[cpu] = ctx->dfl_pwq;
 		} else {
 			wq_calc_pod_cpumask(new_attrs, cpu);
-			ctx->pwq_tbl[cpu] = alloc_unbound_pwq(wq, new_attrs);
+			ctx->pwq_tbl[cpu] = alloc_pwq(wq, cpu, new_attrs);
 			if (!ctx->pwq_tbl[cpu])
 				goto out_free;
 		}
@@ -5632,7 +5665,7 @@ static void unbound_wq_update_pwq(struct workqueue_struct *wq, int cpu)
 		return;
 
 	/* create a new pwq */
-	pwq = alloc_unbound_pwq(wq, target_attrs);
+	pwq = alloc_pwq(wq, cpu, target_attrs);
 	if (!pwq) {
 		pr_warn("workqueue: allocation failed while updating CPU pod affinity of \"%s\"\n",
 			wq->name);
@@ -5668,19 +5701,9 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 		goto enomem;
 
 	if (!(wq->flags & WQ_UNBOUND)) {
-		struct worker_pool __percpu *pools;
-
-		if (wq->flags & WQ_BH)
-			pools = bh_worker_pools;
-		else
-			pools = cpu_worker_pools;
-
 		for_each_possible_cpu(cpu) {
-			struct pool_workqueue **pwq_p;
-			struct worker_pool *pool;
-
-			pool = &(per_cpu_ptr(pools, cpu)[highpri]);
-			pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
+			struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
+			struct worker_pool *pool = get_percpu_pool(wq, cpu);
 
 			*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,
 						       pool->node);

-- 
2.53.0-Meta

[PATCH RFC 2/3] workqueue: allocate percpu pwqs through alloc_pwq()

[Breno Leitao]

Route the per-cpu pwq allocation loop in alloc_and_link_pwqs() through the
common alloc_pwq() instead of open-coding get_percpu_pool() + pwq
allocation + init_pwq().  alloc_pwq() is now the single entry point for
both percpu and unbound pwq allocation, and its percpu branch is exercised
for every percpu workqueue created.

No functional change: alloc_pwq(wq, cpu, NULL) on a percpu workqueue
selects the static per-cpu pool for @cpu, exactly as the open-coded path
did.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 kernel/workqueue.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index c32e173af2335..50baf4fe5fcf5 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -5703,15 +5703,11 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 	if (!(wq->flags & WQ_UNBOUND)) {
 		for_each_possible_cpu(cpu) {
 			struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
-			struct worker_pool *pool = get_percpu_pool(wq, cpu);
 
-			*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,
-						       pool->node);
+			*pwq_p = alloc_pwq(wq, cpu, NULL);
 			if (!*pwq_p)
 				goto enomem;
 
-			init_pwq(*pwq_p, wq, pool);
-
 			mutex_lock(&wq->mutex);
 			link_pwq(*pwq_p);
 			mutex_unlock(&wq->mutex);

-- 
2.53.0-Meta

[PATCH RFC 3/3] workqueue: factor out alloc_and_link_percpu_pwqs()

[Breno Leitao]

Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a
helper.  The inner allocation-failure path now returns -ENOMEM and the
caller jumps to the existing enomem cleanup, equivalent to the previous
goto.

Now that the per-cpu branch returns a value through the helper, the
per-cpu, ordered and unbound cases can share a single error-handling
tail.  Turn the separate per-cpu block and the ordered/unbound if/else
into one if/else-if/else chain, dropping the early return and the
duplicated goto.

No functional change.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 kernel/workqueue.c | 35 ++++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 15 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 50baf4fe5fcf5..411e4bd105576 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -5689,6 +5689,24 @@ static void unbound_wq_update_pwq(struct workqueue_struct *wq, int cpu)
 	put_pwq_unlocked(old_pwq);
 }
 
+static int alloc_and_link_percpu_pwqs(struct workqueue_struct *wq)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
+
+		*pwq_p = alloc_pwq(wq, cpu, NULL);
+		if (!*pwq_p)
+			return -ENOMEM;
+
+		mutex_lock(&wq->mutex);
+		link_pwq(*pwq_p);
+		mutex_unlock(&wq->mutex);
+	}
+	return 0;
+}
+
 static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 {
 	bool highpri = wq->flags & WQ_HIGHPRI;
@@ -5701,21 +5719,8 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq)
 		goto enomem;
 
 	if (!(wq->flags & WQ_UNBOUND)) {
-		for_each_possible_cpu(cpu) {
-			struct pool_workqueue **pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
-
-			*pwq_p = alloc_pwq(wq, cpu, NULL);
-			if (!*pwq_p)
-				goto enomem;
-
-			mutex_lock(&wq->mutex);
-			link_pwq(*pwq_p);
-			mutex_unlock(&wq->mutex);
-		}
-		return 0;
-	}
-
-	if (wq->flags & __WQ_ORDERED) {
+		ret = alloc_and_link_percpu_pwqs(wq);
+	} else if (wq->flags & __WQ_ORDERED) {
 		struct pool_workqueue *dfl_pwq;
 
 		ret = apply_workqueue_attrs_locked(wq, ordered_wq_attrs[highpri]);

-- 
2.53.0-Meta

Re: [PATCH RFC 0/3] Refactor the workqueue allocations

[Tejun Heo]

Hello,

On Tue, Jul 14, 2026 at 04:41:46AM -0700, Breno Leitao wrote:
...
> 1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects
>    non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well,
>    so the unification can happen later (apply_workqueue_attrs_locked())?

I'm not sure adding wqattrs to percpu workqueues makes sense. Wouldn't the
shape more be like unbound workqueue subsuming percpu workqueue?

> 2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be
>    treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node
>    accounting (wq_node_nr_active) and lose percpu's per-cpu max_active
>    (pwq->nr_active).

I think it probably would be better to introduce a separate affinity scope
than modifying WQ_AFFN_CPU. Something which indicates that concurrency
management is in effect and max_active is per-cpu.

> 3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
>    unified internals (single install path, per-cpu accounting special-cased)
>    with WQ_UNBOUND staying for now -- or something more radical (the flags gone
>    entirely, percpu purely an affinity value)?

Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense
to me.

Thanks.

-- 
tejun

Re: [PATCH RFC 0/3] Refactor the workqueue allocations

[Tejun Heo]

On Thu, Jul 16, 2026 at 09:22:50AM -1000, Tejun Heo wrote:
...
> > 3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
> >    unified internals (single install path, per-cpu accounting special-cased)
> >    with WQ_UNBOUND staying for now -- or something more radical (the flags gone
> >    entirely, percpu purely an affinity value)?
> 
> Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense
> to me.

Oh, and please note that if a workqueue specifies WQ_PERCPU, it should be
confined to strict cpu affinity mode.

Thanks.

-- 
tejun

Re: [PATCH RFC 0/3] Refactor the workqueue allocations

[Breno Leitao]

On Thu, Jul 16, 2026 at 09:22:50AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Tue, Jul 14, 2026 at 04:41:46AM -0700, Breno Leitao wrote:
> ...
> > 1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects
> >    non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well,
> >    so the unification can happen later (apply_workqueue_attrs_locked())?
> 
> I'm not sure adding wqattrs to percpu workqueues makes sense. Wouldn't the
> shape more be like unbound workqueue subsuming percpu workqueue?

> > 2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be
> >    treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node
> >    accounting (wq_node_nr_active) and lose percpu's per-cpu max_active
> >    (pwq->nr_active).
> 
> I think it probably would be better to introduce a separate affinity scope
> than modifying WQ_AFFN_CPU. Something which indicates that concurrency
> management is in effect and max_active is per-cpu.
> 
> > 3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
> >    unified internals (single install path, per-cpu accounting special-cased)
> >    with WQ_UNBOUND staying for now -- or something more radical (the flags gone
> >    entirely, percpu purely an affinity value)?
> 
> Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense
> to me.

Oh, now I see what you mean, I was heading the wrong way. Thanks for the
clarification, that makes total sense.

So: add a PERCPU wq_affn_scope and back it strictly per-CPU, rather than
reusing WQ_AFFN_CPU. Something like:

    enum wq_affn_scope {
            ...
  +         WQ_AFFN_PERCPU,         /* one pod per CPU, backed by the per-cpu pool */

and move the per-cpu workqueue users onto WQ_AFFN_PERCPU. With that, the
unbound install path (apply_wqattrs and the per-cpu, replaceable pwqs) can
point a pwq at a per-cpu pool, so one mechanism serves both. Then move
all the WQ_PERCPU users to WQ_AFFN_PERCPU, and eventually deprecate
WQ_PERCPU ?

I have this working as a prototype: WQ_PERCPU selects the scope and forces
strict affinity, and it boots with every percpu wq created through the
new path. 

A few things I'd like your read on:

    1) Percpu workqueues keep the WQ_PERCPU flag (I don't switch them to
       WQ_UNBOUND when they move onto the WQ_AFFN_PERCPU scope), so per-cpu
       accounting falls out of the existing !WQ_UNBOUND checks. do you
       have any preference here, or should percpu become purely an
       affn_scope value with accounting decoupled from the flag?

    2) What about WQ_BH? Can we keep it on the direct per-cpu path (softirq
       context) for now?

    3) Routing percpu through apply_wqattrs pulls in unbound-only assumptions
       (unbound_attrs allocation, and the CPU-hotplug fixups in
       workqueue_online_cpu()/workqueue_offline_cpu() that gate on
       unbound_attrs) that now have to learn about the percpu scope.

       Do you prefer teaching that shared path about WQ_AFFN_PERCPU, or would
       you rather percpu keep a lighter install path (closer to the current
       WQ_PERCPU direct path), if that's feasible?

Thanks for the guidance,
--breno

Re: [PATCH RFC 0/3] Refactor the workqueue allocations

[Tejun Heo]

Hello, Breno.

Sorry about the delay.

On Fri, Jul 17, 2026 at 09:25:09AM -0700, Breno Leitao wrote:
> So: add a PERCPU wq_affn_scope and back it strictly per-CPU, rather than
> reusing WQ_AFFN_CPU. Something like:
> 
>     enum wq_affn_scope {
>             ...
>   +         WQ_AFFN_PERCPU,         /* one pod per CPU, backed by the per-cpu pool */
> 
> and move the per-cpu workqueue users onto WQ_AFFN_PERCPU. With that, the
> unbound install path (apply_wqattrs and the per-cpu, replaceable pwqs) can
> point a pwq at a per-cpu pool, so one mechanism serves both. Then move
> all the WQ_PERCPU users to WQ_AFFN_PERCPU, and eventually deprecate
> WQ_PERCPU ?

I don't think we'd deprecate WQ_PERCPU. It'd remain the way to specify on
creation that the workqueue has to be WQ_AFFN_PERCPU.

Also, maybe WQ_AFFN_PERCPU can be more descriptive - WQ_AFFN_CPU_CM for
concurrency-managed CPU scope? Or maybe this shouldn't be packaged into
WQ_AFFN but rather become its own mode field.

> I have this working as a prototype: WQ_PERCPU selects the scope and forces
> strict affinity, and it boots with every percpu wq created through the
> new path. 
> 
> A few things I'd like your read on:
> 
>     1) Percpu workqueues keep the WQ_PERCPU flag (I don't switch them to
>        WQ_UNBOUND when they move onto the WQ_AFFN_PERCPU scope), so per-cpu
>        accounting falls out of the existing !WQ_UNBOUND checks. do you
>        have any preference here, or should percpu become purely an
>        affn_scope value with accounting decoupled from the flag?

For concurrency management to work, it would need separate accounting (of
max_active, right?). WQ_PERCPU would indicate that the wq must stay per-cpu
for correctness, and we'd also want to allow concurrency management to
workqueues which want to be percpu for performance reasons but can be
switched into other affinity scopes for isolation, so it should move
together with whether the backend needs concurrency management or not
instead of WQ_PERCPU expressed at creation time.

It kinda sucks that max_active's meaning is different across the boundary
tho. Maybe percpu max_active should be separate into its own field, idk.

>     2) What about WQ_BH? Can we keep it on the direct per-cpu path (softirq
>        context) for now?

We can't switch workqueues on / off WQ_BH, but it'd also look a bit silly if
this becomes its own path. Code-shape-wise, I suspect it'd be cleaner if
this also becomes one of the pwq backends like the other two cases but
that's just a gut feel.

>     3) Routing percpu through apply_wqattrs pulls in unbound-only assumptions
>        (unbound_attrs allocation, and the CPU-hotplug fixups in
>        workqueue_online_cpu()/workqueue_offline_cpu() that gate on
>        unbound_attrs) that now have to learn about the percpu scope.
> 
>        Do you prefer teaching that shared path about WQ_AFFN_PERCPU, or would
>        you rather percpu keep a lighter install path (closer to the current
>        WQ_PERCPU direct path), if that's feasible?

Again, without thinking too deeploy about it, I think the code would look
better if we unify everything we can.

Thanks.

-- 
tejun

Re: [PATCH RFC 0/3] Refactor the workqueue allocations

[Breno Leitao]

Hello Tejun,

On Wed, Jul 22, 2026 at 08:56:10AM -1000, Tejun Heo wrote:
>
> Again, without thinking too deeploy about it, I think the code would look
> better if we unify everything we can.

I believe I have what I need to proceed. I'm planning a two-phase approach:

First, refactor the code to improve modularity where needed keeping
as-is behavior, then introduce the substantive changes once the initial
refactoring is done.

The goal is to make the code more managable and easy to digest.

Please let me know if you prefer otherwise.
--breno