[PATCH 0/2] s390: replace wq users and add WQ_PERCPU to alloc_workqueue() users

[PATCH 0/2] s390: replace wq users and add WQ_PERCPU to alloc_workqueue() users

[Marco Crivellari]

Below is a summary of a discussion about the Workqueue API and cpu isolation
considerations. Details and more information are available here:

        "workqueue: Always use wq_select_unbound_cpu() for WORK_CPU_UNBOUND."
        https://lore.kernel.org/all/20250221112003.1dSuoGyc@linutronix.de/

=== Current situation: problems ===

Let's consider a nohz_full system with isolated CPUs: wq_unbound_cpumask is
set to the housekeeping CPUs, for !WQ_UNBOUND the local CPU is selected.

This leads to different scenarios if a work item is scheduled on an isolated
CPU where "delay" value is 0 or greater then 0:
        schedule_delayed_work(, 0);

This will be handled by __queue_work() that will queue the work item on the
current local (isolated) CPU, while:

        schedule_delayed_work(, 1);

Will move the timer on an housekeeping CPU, and schedule the work there.

Currently if a user enqueue a work item using schedule_delayed_work() the
used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
schedule_work() that is using system_wq and queue_work(), that makes use
again of WORK_CPU_UNBOUND.

This lack of consistentcy cannot be addressed without refactoring the API.

=== Plan and future plans ===

This patchset is the first stone on a refactoring needed in order to
address the points aforementioned; it will have a positive impact also
on the cpu isolation, in the long term, moving away percpu workqueue in
favor to an unbound model.

These are the main steps:
1)  API refactoring (that this patch is introducing)
    -   Make more clear and uniform the system wq names, both per-cpu and
        unbound. This to avoid any possible confusion on what should be
        used.

    -   Introduction of WQ_PERCPU: this flag is the complement of WQ_UNBOUND,
        introduced in this patchset and used on all the callers that are not
        currently using WQ_UNBOUND.

        WQ_UNBOUND will be removed in a future release cycle.

        Most users don't need to be per-cpu, because they don't have
        locality requirements, because of that, a next future step will be
        make "unbound" the default behavior.

2)  Check who really needs to be per-cpu
    -   Remove the WQ_PERCPU flag when is not strictly required.

3)  Add a new API (prefer local cpu)
    -   There are users that don't require a local execution, like mentioned
        above; despite that, local execution yeld to performance gain.

        This new API will prefer the local execution, without requiring it.

=== Introduced Changes by this series ===

1) [P 1] Replace use of system_wq

        system_wq is a per-CPU workqueue, but his name is not clear.

        Because of that, system_wq has been renamed in system_percpu_wq.

2) [P 2] add WQ_PERCPU to remaining alloc_workqueue() users

        Every alloc_workqueue() caller should use one among WQ_PERCPU or
        WQ_UNBOUND. This is actually enforced warning if both or none of them
        are present at the same time.

        WQ_UNBOUND will be removed in a next release cycle.

=== For Maintainers ===

There are prerequisites for this series, already merged in the master branch.
The commits are:

128ea9f6ccfb6960293ae4212f4f97165e42222d ("workqueue: Add system_percpu_wq and
system_dfl_wq")

930c2ea566aff59e962c50b2421d5fcc3b98b8be ("workqueue: Add new WQ_PERCPU flag")


Thanks!

Marco Crivellari (2):
  drivers/s390: WQ_PERCPU added to alloc_workqueue users
  s390: replace use of system_wq with system_percpu_wq

 arch/s390/kernel/diag/diag324.c  | 4 ++--
 arch/s390/kernel/hiperdispatch.c | 2 +-
 drivers/s390/char/tape_3590.c    | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

-- 
2.51.0

[PATCH 1/2] drivers/s390: WQ_PERCPU added to alloc_workqueue users

[Marco Crivellari]

Currently if a user enqueue a work item using schedule_delayed_work() the
used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
schedule_work() that is using system_wq and queue_work(), that makes use
again of WORK_CPU_UNBOUND.
This lack of consistentcy cannot be addressed without refactoring the API.

alloc_workqueue() treats all queues as per-CPU by default, while unbound
workqueues must opt-in via WQ_UNBOUND.

This default is suboptimal: most workloads benefit from unbound queues,
allowing the scheduler to place worker threads where they’re needed and
reducing noise when CPUs are isolated.

This default is suboptimal: most workloads benefit from unbound queues,
allowing the scheduler to place worker threads where they’re needed and
reducing noise when CPUs are isolated.

This patch adds a new WQ_PERCPU flag to explicitly request the use of
the per-CPU behavior. Both flags coexist for one release cycle to allow
callers to transition their calls.

Once migration is complete, WQ_UNBOUND can be removed and unbound will
become the implicit default.

With the introduction of the WQ_PERCPU flag (equivalent to !WQ_UNBOUND),
any alloc_workqueue() caller that doesn’t explicitly specify WQ_UNBOUND
must now use WQ_PERCPU.

All existing users have been updated accordingly.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 drivers/s390/char/tape_3590.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/s390/char/tape_3590.c b/drivers/s390/char/tape_3590.c
index 0d484fe43d7e..aee11fece701 100644
--- a/drivers/s390/char/tape_3590.c
+++ b/drivers/s390/char/tape_3590.c
@@ -1670,7 +1670,7 @@ tape_3590_init(void)
 
 	DBF_EVENT(3, "3590 init\n");
 
-	tape_3590_wq = alloc_workqueue("tape_3590", 0, 0);
+	tape_3590_wq = alloc_workqueue("tape_3590", WQ_PERCPU, 0);
 	if (!tape_3590_wq)
 		return -ENOMEM;
 
-- 
2.51.0

[PATCH 2/2] s390: replace use of system_wq with system_percpu_wq

[Marco Crivellari]

Currently if a user enqueue a work item using schedule_delayed_work() the
used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
schedule_work() that is using system_wq and queue_work(), that makes use
again of WORK_CPU_UNBOUND.

This lack of consistentcy cannot be addressed without refactoring the API.

system_wq is a per-CPU worqueue, yet nothing in its name tells about that
CPU affinity constraint, which is very often not required by users. Make
it clear by adding a system_percpu_wq.

queue_work() / queue_delayed_work() mod_delayed_work() will now use the
new per-cpu wq: whether the user still stick on the old name a warn will
be printed along a wq redirect to the new one.

This patch add the new system_percpu_wq except for mm, fs and net
subsystem, whom are handled in separated patches.

The old wq will be kept for a few release cylces.

Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
---
 arch/s390/kernel/diag/diag324.c  | 4 ++--
 arch/s390/kernel/hiperdispatch.c | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kernel/diag/diag324.c b/arch/s390/kernel/diag/diag324.c
index 7fa4c0b7eb6c..f0a8b4841fb9 100644
--- a/arch/s390/kernel/diag/diag324.c
+++ b/arch/s390/kernel/diag/diag324.c
@@ -116,7 +116,7 @@ static void pibwork_handler(struct work_struct *work)
 	mutex_lock(&pibmutex);
 	timedout = ktime_add_ns(data->expire, PIBWORK_DELAY);
 	if (ktime_before(ktime_get(), timedout)) {
-		mod_delayed_work(system_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
+		mod_delayed_work(system_percpu_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
 		goto out;
 	}
 	vfree(data->pib);
@@ -174,7 +174,7 @@ long diag324_pibbuf(unsigned long arg)
 		pib_update(data);
 		data->sequence++;
 		data->expire = ktime_add_ns(ktime_get(), tod_to_ns(data->pib->intv));
-		mod_delayed_work(system_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
+		mod_delayed_work(system_percpu_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
 		first = false;
 	}
 	rc = data->rc;
diff --git a/arch/s390/kernel/hiperdispatch.c b/arch/s390/kernel/hiperdispatch.c
index e7b66d046e8d..85b5508ab62c 100644
--- a/arch/s390/kernel/hiperdispatch.c
+++ b/arch/s390/kernel/hiperdispatch.c
@@ -191,7 +191,7 @@ int hd_enable_hiperdispatch(void)
 		return 0;
 	if (hd_online_cores <= hd_entitled_cores)
 		return 0;
-	mod_delayed_work(system_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
+	mod_delayed_work(system_percpu_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
 	hd_update_capacities();
 	return 1;
 }
-- 
2.51.0

Re: [PATCH 2/2] s390: replace use of system_wq with system_percpu_wq

[Mete Durlu]

On 9/5/25 11:08 AM, Marco Crivellari wrote:
> Currently if a user enqueue a work item using schedule_delayed_work() the
> used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
> WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
> schedule_work() that is using system_wq and queue_work(), that makes use
> again of WORK_CPU_UNBOUND.
> 
> This lack of consistentcy cannot be addressed without refactoring the API.
> 
> system_wq is a per-CPU worqueue, yet nothing in its name tells about that
> CPU affinity constraint, which is very often not required by users. Make
> it clear by adding a system_percpu_wq.
> 
> queue_work() / queue_delayed_work() mod_delayed_work() will now use the
> new per-cpu wq: whether the user still stick on the old name a warn will
> be printed along a wq redirect to the new one.
> 
> This patch add the new system_percpu_wq except for mm, fs and net
> subsystem, whom are handled in separated patches.
> 
> The old wq will be kept for a few release cylces.
> 
> Suggested-by: Tejun Heo <tj@kernel.org>
> Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>

If I get this correctly system_wq will be obsolete and users will get
system_percpu_wq instead, which means local cpu gets to deal with the
delayed work and its timer and it has an affinity to that cpu via per
cpu workqueue. In that case;


> diff --git a/arch/s390/kernel/hiperdispatch.c b/arch/s390/kernel/hiperdispatch.c
> index e7b66d046e8d..85b5508ab62c 100644
> --- a/arch/s390/kernel/hiperdispatch.c
> +++ b/arch/s390/kernel/hiperdispatch.c
> @@ -191,7 +191,7 @@ int hd_enable_hiperdispatch(void)
>   		return 0;
>   	if (hd_online_cores <= hd_entitled_cores)
>   		return 0;
> -	mod_delayed_work(system_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
> +	mod_delayed_work(system_percpu_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
>   	hd_update_capacities();

Hiperdispatch's delayed work wouldn't get a noticeable benefit from
utilizing a per-cpu workqueue. We probably settled on system_wq to
utilize the global work queue at the time. Would system_unbound_wq
make more sense here?

Thanks.

Re: [PATCH 1/2] drivers/s390: WQ_PERCPU added to alloc_workqueue users

[Alexander Gordeev]

On Fri, Sep 05, 2025 at 11:08:56AM +0200, Marco Crivellari wrote:
> Currently if a user enqueue a work item using schedule_delayed_work() the
> used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
> WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
> schedule_work() that is using system_wq and queue_work(), that makes use
> again of WORK_CPU_UNBOUND.
> This lack of consistentcy cannot be addressed without refactoring the API.
> 
> alloc_workqueue() treats all queues as per-CPU by default, while unbound
> workqueues must opt-in via WQ_UNBOUND.
> 
> This default is suboptimal: most workloads benefit from unbound queues,
> allowing the scheduler to place worker threads where they’re needed and
> reducing noise when CPUs are isolated.
> 
> This default is suboptimal: most workloads benefit from unbound queues,
> allowing the scheduler to place worker threads where they’re needed and
> reducing noise when CPUs are isolated.

A duplicate paragraph.

> This patch adds a new WQ_PERCPU flag to explicitly request the use of
> the per-CPU behavior. Both flags coexist for one release cycle to allow
> callers to transition their calls.
> 
> Once migration is complete, WQ_UNBOUND can be removed and unbound will
> become the implicit default.
> 
> With the introduction of the WQ_PERCPU flag (equivalent to !WQ_UNBOUND),
> any alloc_workqueue() caller that doesn’t explicitly specify WQ_UNBOUND
> must now use WQ_PERCPU.
> 
> All existing users have been updated accordingly.
> 
> Suggested-by: Tejun Heo <tj@kernel.org>
> Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
> ---
>  drivers/s390/char/tape_3590.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/s390/char/tape_3590.c b/drivers/s390/char/tape_3590.c
> index 0d484fe43d7e..aee11fece701 100644
> --- a/drivers/s390/char/tape_3590.c
> +++ b/drivers/s390/char/tape_3590.c
> @@ -1670,7 +1670,7 @@ tape_3590_init(void)
>  
>  	DBF_EVENT(3, "3590 init\n");
>  
> -	tape_3590_wq = alloc_workqueue("tape_3590", 0, 0);
> +	tape_3590_wq = alloc_workqueue("tape_3590", WQ_PERCPU, 0);
>  	if (!tape_3590_wq)
>  		return -ENOMEM;

Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>

Thanks!

Re: [PATCH 2/2] s390: replace use of system_wq with system_percpu_wq

[Alexander Gordeev]

On Fri, Sep 05, 2025 at 11:08:57AM +0200, Marco Crivellari wrote:

Hi Marco,

> Currently if a user enqueue a work item using schedule_delayed_work() the
> used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
> WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
> schedule_work() that is using system_wq and queue_work(), that makes use
> again of WORK_CPU_UNBOUND.
> 
> This lack of consistentcy cannot be addressed without refactoring the API.
> 
> system_wq is a per-CPU worqueue, yet nothing in its name tells about that
> CPU affinity constraint, which is very often not required by users. Make
> it clear by adding a system_percpu_wq.

This paragraph is not exactly correct. You switch from system_wq to
system_percpu_wq - which are two different queues with the same
characteristics:

	system_wq = alloc_workqueue("events", 0, 0);                            
	system_percpu_wq = alloc_workqueue("events", 0, 0);                     

> queue_work() / queue_delayed_work() mod_delayed_work() will now use the
> new per-cpu wq: whether the user still stick on the old name a warn will
> be printed along a wq redirect to the new one.
> 
> This patch add the new system_percpu_wq except for mm, fs and net
> subsystem, whom are handled in separated patches.

I do not see this patch does anything like that.

> The old wq will be kept for a few release cylces.
> 
> Suggested-by: Tejun Heo <tj@kernel.org>
> Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
> ---
>  arch/s390/kernel/diag/diag324.c  | 4 ++--
>  arch/s390/kernel/hiperdispatch.c | 2 +-
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/s390/kernel/diag/diag324.c b/arch/s390/kernel/diag/diag324.c
> index 7fa4c0b7eb6c..f0a8b4841fb9 100644
> --- a/arch/s390/kernel/diag/diag324.c
> +++ b/arch/s390/kernel/diag/diag324.c
> @@ -116,7 +116,7 @@ static void pibwork_handler(struct work_struct *work)
>  	mutex_lock(&pibmutex);
>  	timedout = ktime_add_ns(data->expire, PIBWORK_DELAY);
>  	if (ktime_before(ktime_get(), timedout)) {
> -		mod_delayed_work(system_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
> +		mod_delayed_work(system_percpu_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
>  		goto out;
>  	}
>  	vfree(data->pib);
> @@ -174,7 +174,7 @@ long diag324_pibbuf(unsigned long arg)
>  		pib_update(data);
>  		data->sequence++;
>  		data->expire = ktime_add_ns(ktime_get(), tod_to_ns(data->pib->intv));
> -		mod_delayed_work(system_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
> +		mod_delayed_work(system_percpu_wq, &pibwork, nsecs_to_jiffies(PIBWORK_DELAY));
>  		first = false;
>  	}
>  	rc = data->rc;
> diff --git a/arch/s390/kernel/hiperdispatch.c b/arch/s390/kernel/hiperdispatch.c
> index e7b66d046e8d..85b5508ab62c 100644
> --- a/arch/s390/kernel/hiperdispatch.c
> +++ b/arch/s390/kernel/hiperdispatch.c
> @@ -191,7 +191,7 @@ int hd_enable_hiperdispatch(void)
>  		return 0;
>  	if (hd_online_cores <= hd_entitled_cores)
>  		return 0;
> -	mod_delayed_work(system_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
> +	mod_delayed_work(system_percpu_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
>  	hd_update_capacities();
>  	return 1;
>  }
> -- 
> 2.51.0
> 

Re: [PATCH 1/2] drivers/s390: WQ_PERCPU added to alloc_workqueue users

[Marco Crivellari]

On Mon, Sep 8, 2025 at 1:38 PM Alexander Gordeev <agordeev@linux.ibm.com> wrote:
> > This default is suboptimal: most workloads benefit from unbound queues,
> > allowing the scheduler to place worker threads where they’re needed and
> > reducing noise when CPUs are isolated.
>
> A duplicate paragraph.
>

> Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>

Thank you, Alexander!

-- 

Marco Crivellari

L3 Support Engineer, Technology & Product




marco.crivellari@suse.com

Re: [PATCH 2/2] s390: replace use of system_wq with system_percpu_wq

[Marco Crivellari]

On Mon, Sep 8, 2025 at 10:33 AM Mete Durlu <meted@linux.ibm.com> wrote:
>
> If I get this correctly system_wq will be obsolete and users will get
> system_percpu_wq instead, which means local cpu gets to deal with the
> delayed work and its timer and it has an affinity to that cpu via per
> cpu workqueue. In that case;
>
> > diff --git a/arch/s390/kernel/hiperdispatch.c b/arch/s390/kernel/hiperdispatch.c
> > index e7b66d046e8d..85b5508ab62c 100644
> > --- a/arch/s390/kernel/hiperdispatch.c
> > +++ b/arch/s390/kernel/hiperdispatch.c
> > @@ -191,7 +191,7 @@ int hd_enable_hiperdispatch(void)
> >               return 0;
> >       if (hd_online_cores <= hd_entitled_cores)
> >               return 0;
> > -     mod_delayed_work(system_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
> > +     mod_delayed_work(system_percpu_wq, &hd_capacity_work, HD_DELAY_INTERVAL * hd_delay_factor);
> >       hd_update_capacities();
>
> Hiperdispatch's delayed work wouldn't get a noticeable benefit from
> utilizing a per-cpu workqueue. We probably settled on system_wq to
> utilize the global work queue at the time. Would system_unbound_wq
> make more sense here?
>
> Thanks.

Hello,

I will check the code and if it is possible, I will send the v2 with
system_dfl_wq (eg. the current/old system_unbound_wq).

Thanks!
-- 

Marco Crivellari

L3 Support Engineer, Technology & Product

marco.crivellari@suse.com

Re: [PATCH 2/2] s390: replace use of system_wq with system_percpu_wq

[Marco Crivellari]

On Mon, Sep 8, 2025 at 1:44 PM Alexander Gordeev <agordeev@linux.ibm.com> wrote:
>
> On Fri, Sep 05, 2025 at 11:08:57AM +0200, Marco Crivellari wrote:
>
> Hi Marco,
>
> > Currently if a user enqueue a work item using schedule_delayed_work() the
> > used wq is "system_wq" (per-cpu wq) while queue_delayed_work() use
> > WORK_CPU_UNBOUND (used when a cpu is not specified). The same applies to
> > schedule_work() that is using system_wq and queue_work(), that makes use
> > again of WORK_CPU_UNBOUND.
> >
> > This lack of consistentcy cannot be addressed without refactoring the API.
> >
> > system_wq is a per-CPU worqueue, yet nothing in its name tells about that
> > CPU affinity constraint, which is very often not required by users. Make
> > it clear by adding a system_percpu_wq.
>
> This paragraph is not exactly correct. You switch from system_wq to
> system_percpu_wq - which are two different queues with the same
> characteristics:
>
>         system_wq = alloc_workqueue("events", 0, 0);
>         system_percpu_wq = alloc_workqueue("events", 0, 0);

Hi Alexander,

Yes, system_percpu_wq will be in future the replacement of system_wq.

> > queue_work() / queue_delayed_work() mod_delayed_work() will now use the
> > new per-cpu wq: whether the user still stick on the old name a warn will
> > be printed along a wq redirect to the new one.
> >
> > This patch add the new system_percpu_wq except for mm, fs and net
> > subsystem, whom are handled in separated patches.
>
> I do not see this patch does anything like that.
>

I'm sorry for the confusion, I forgot to update the log after the
previous change.
There are not, indeed, the changes you mentioned.

Thanks!
-- 

Marco Crivellari

L3 Support Engineer, Technology & Product

marco.crivellari@suse.com