[PATCH] usb: hub: Make usb_hub_wq type depend on isolcpus/nohz_full setting

[PATCH] usb: hub: Make usb_hub_wq type depend on isolcpus/nohz_full setting

[Waiman Long]

A Red Hat customer reports a kernel stability problem where hung tasks
are reported with occasional kernel panics. Analysis of the core dump
indicates that USB work items are running on isolcpus+nohz_full cores
competing with RT-class tasks running on those core while holding
usb_hub device mutex transitively blocking other kworkers waiting for
the same mutex leading to hung_task reports.

As the usb_hub_wq uses the WQ_PERCPU flag, it will run the work items on
the same CPU that queues them. For many use cases, it is a more efficient
setup leading to higher throughput as it reduces cacheline bouncing.

It is a different story if the system needs to run latency sensitive RT
workload on dedicated isolated CPUs. Having the kworkers processing work
items on the same set of isolated CPUs will likely break the low latency
requirements of the RT tasks. As the RT tasks have higher priority,
not much CPU time will be left running the kworkers to process work
items which, in turn, will block other tasks that have dependency on
the completion of those work items. In this case, using a WQ_UNBOUND
workqueue to avoid running on isolated CPUs will be more beneficial.

One solution to get the best of both worlds is to make the workqueue
type depending on whether the "isolcpus" or "nohz_full" boot command
line options have been specified. If at least one of those options are
present, usb_hub_wq will be created as an unbound workqueue. Otherwise,
it will remain as a percpu workqueue.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 drivers/usb/core/hub.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
index 24960ba9caa9..f79e5edd627a 100644
--- a/drivers/usb/core/hub.c
+++ b/drivers/usb/core/hub.c
@@ -33,6 +33,7 @@
 #include <linux/random.h>
 #include <linux/pm_qos.h>
 #include <linux/kobject.h>
+#include <linux/sched/isolation.h>
 
 #include <linux/bitfield.h>
 #include <linux/uaccess.h>
@@ -6066,6 +6067,8 @@ static struct usb_driver hub_driver = {
 
 int usb_hub_init(void)
 {
+	unsigned int wq_flags;
+
 	if (usb_register(&hub_driver) < 0) {
 		printk(KERN_ERR "%s: can't register hub driver\n",
 			usbcore_name);
@@ -6077,8 +6080,17 @@ int usb_hub_init(void)
 	 * USB-PERSIST port handover. Otherwise it might see that a full-speed
 	 * device was gone before the EHCI controller had handed its port
 	 * over to the companion full-speed controller.
+	 *
+	 * Create WQ_UNBOUND workqueue instead of WQ_PERCPU if either isolcpus
+	 * or nohz_full boot option is specified.
 	 */
-	hub_wq = alloc_workqueue("usb_hub_wq", WQ_FREEZABLE | WQ_PERCPU, 0);
+	if (housekeeping_enabled(HK_TYPE_DOMAIN) ||
+	    housekeeping_enabled(HK_TYPE_KERNEL_NOISE))
+		wq_flags = WQ_UNBOUND;
+	else
+		wq_flags = WQ_PERCPU;
+
+	hub_wq = alloc_workqueue("usb_hub_wq", WQ_FREEZABLE | wq_flags, 0);
 	if (hub_wq)
 		return 0;
 
-- 
2.54.0

Re: [PATCH] usb: hub: Make usb_hub_wq type depend on isolcpus/nohz_full setting

[Frederic Weisbecker]

Le Thu, May 21, 2026 at 01:06:59PM -0400, Waiman Long a écrit :
> A Red Hat customer reports a kernel stability problem where hung tasks
> are reported with occasional kernel panics. Analysis of the core dump
> indicates that USB work items are running on isolcpus+nohz_full cores
> competing with RT-class tasks running on those core while holding
> usb_hub device mutex transitively blocking other kworkers waiting for
> the same mutex leading to hung_task reports.
> 
> As the usb_hub_wq uses the WQ_PERCPU flag, it will run the work items on
> the same CPU that queues them. For many use cases, it is a more efficient
> setup leading to higher throughput as it reduces cacheline bouncing.
> 
> It is a different story if the system needs to run latency sensitive RT
> workload on dedicated isolated CPUs. Having the kworkers processing work
> items on the same set of isolated CPUs will likely break the low latency
> requirements of the RT tasks. As the RT tasks have higher priority,
> not much CPU time will be left running the kworkers to process work
> items which, in turn, will block other tasks that have dependency on
> the completion of those work items. In this case, using a WQ_UNBOUND
> workqueue to avoid running on isolated CPUs will be more beneficial.
> 
> One solution to get the best of both worlds is to make the workqueue
> type depending on whether the "isolcpus" or "nohz_full" boot command
> line options have been specified. If at least one of those options are
> present, usb_hub_wq will be created as an unbound workqueue. Otherwise,
> it will remain as a percpu workqueue.
> 
> Signed-off-by: Waiman Long <longman@redhat.com>
> ---
>  drivers/usb/core/hub.c | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
> index 24960ba9caa9..f79e5edd627a 100644
> --- a/drivers/usb/core/hub.c
> +++ b/drivers/usb/core/hub.c
> @@ -33,6 +33,7 @@
>  #include <linux/random.h>
>  #include <linux/pm_qos.h>
>  #include <linux/kobject.h>
> +#include <linux/sched/isolation.h>
>  
>  #include <linux/bitfield.h>
>  #include <linux/uaccess.h>
> @@ -6066,6 +6067,8 @@ static struct usb_driver hub_driver = {
>  
>  int usb_hub_init(void)
>  {
> +	unsigned int wq_flags;
> +
>  	if (usb_register(&hub_driver) < 0) {
>  		printk(KERN_ERR "%s: can't register hub driver\n",
>  			usbcore_name);
> @@ -6077,8 +6080,17 @@ int usb_hub_init(void)
>  	 * USB-PERSIST port handover. Otherwise it might see that a full-speed
>  	 * device was gone before the EHCI controller had handed its port
>  	 * over to the companion full-speed controller.
> +	 *
> +	 * Create WQ_UNBOUND workqueue instead of WQ_PERCPU if either isolcpus
> +	 * or nohz_full boot option is specified.
>  	 */
> -	hub_wq = alloc_workqueue("usb_hub_wq", WQ_FREEZABLE | WQ_PERCPU, 0);
> +	if (housekeeping_enabled(HK_TYPE_DOMAIN) ||
> +	    housekeeping_enabled(HK_TYPE_KERNEL_NOISE))

HK_TYPE_DOMAIN is supposed to be a subset of HK_TYPE_KERNEL_NOISE anyway so
the first should be enough.

> +		wq_flags = WQ_UNBOUND;
> +	else
> +		wq_flags = WQ_PERCPU;
> +
> +	hub_wq = alloc_workqueue("usb_hub_wq", WQ_FREEZABLE | wq_flags, 0);

But then what happens if no isolcpus= is passed but later cpuset creates
an isolated partition?

Tejun and Marco thought about introducing a WQ_PREFER_PERCPU flag that would
do what you want above. And the workqueue code should also handle dynamic
isolation, that is switch from per-cpu workqueues to unbound ones or vice-versa
dynamically.

Marco is working on it.

Thanks.

>  	if (hub_wq)
>  		return 0;
>  
> -- 
> 2.54.0
> 
> 

-- 
Frederic Weisbecker
SUSE Labs