[RFC PATCH] usb: core: defer leaf-device resume off the S3 critical path

[RFC PATCH] usb: core: defer leaf-device resume off the S3 critical path

[Hongyu Xie]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 13864 bytes --]

Problem:
System resume (dpm_resume) invokes each device's resume callback
synchronously on the wake-up critical path.  For a USB leaf device
this means port resume and, when reset_resume is needed, a port
reset that, for persistent devices, waits up to 2000 ms for the
device connection (CCS; on SuperSpeed this includes link training)
to be re-established (wait_for_connected()).  Slow link training of
some U-disks and webcams after S3 power-on therefore stalls the
entire system wake-up: on an
arm64 test machine with four leaf devices (bluetooth, UVC camera,
two U-disks) dpm_resume took 2508 ms, about 2.1 s of it caused by a
single SanDisk U-disk whose link training stayed in Rx.Detect for
2000 ms (worst case: -ENODEV fallback to disconnect +
re-enumeration).  A device compatibility issue should not lengthen
the system-wide wake-up.

Fix:
The patch addresses this on four levels:

1. Defer leaf-device recovery off the critical path.  Leaf devices
   (udev->parent && !udev->maxchild) that are port-suspended on
   system resume have their real port resume moved to a per-device
   work item queued from usb_resume_complete(), after all hubs have
   finished resume.  The work reuses the same udev (in-place reset,
   devnum preserved); drivers without reset_resume get unbind+rebind.

2. Safe abandonment.  If a new suspend or a disconnect arrives before
   the work item runs, the reset is abandoned, not flushed: the
   device is still port-suspended, which is exactly what the new
   suspend needs, and the work item then only releases the runtime
   reference it took.  The abandon decision is serialized with the
   work item by the device lock, so no cancel_work_sync() (which
   would deadlock against the work item's device_lock()) is needed
   on the suspend path.

3. Truthful runtime state with lazy resume.  The deferred branch of
   usb_resume() reports the device as suspended
   (pm_runtime_set_suspended) until recovery actually completes; the
   work item switches it back to active only after usb_resume_both()
   succeeds.  Accesses made in between trigger
   usb_autoresume_device() -> usb_port_resume() and resume the
   device synchronously, so a successful operation is a real
   confirmation of readiness.  A failed recovery leaves the device
   suspended: later accesses fail or the device is disconnected,
   both visible to user space.

4. Port-state gating.  A device is only deferred when its port is
   not connected (link training still pending, or the device was
   removed while asleep) -- the case where the long waits happen.  A
   connected port resumes in bounded time and keeps the synchronous
   path with unchanged semantics.

Results:
With four leaf devices (bluetooth, UVC camera, Kingston U-disk,
SanDisk U-disk), repeated S3 measurements (14 rounds):
- without a 2000 ms event: dpm_resume drops from ~498-502 ms to
  ~219-223 ms (-56%);
- with the occasional 2000 ms link-training wait (or the fixed
  2000 ms CONNECT timeout for a device removed during sleep):
  dpm_resume drops from ~2508-2813 ms to ~219 ms (-91%), the wait
  running in the background workqueue without affecting
  system-wide resume.
Deferred recovery completes in 0-20 ms in the common case.

Known limitations (no practical impact):
- The system resume notification (PM_POST_SYSTEM_RESUME) is
  system-wide and carries no device information; between the
  notification and deferred recovery completion an application
  accessing the device may fail.  Drivers using usb_autopm trigger
  synchronous recovery instead (lazy resume), and the window is
  milliseconds in practice.
- usb_resume() returns success before the device is actually
  recovered; the runtime PM state is truthful (suspended), but the
  PM core counts the callback as successful.  No kernel path
  depends on that accounting.
- A recovery failure in the workqueue cannot be reported through
  the dpm error path; it surfaces as device disconnect, which user
  space already handles.

Gated by module parameter usbcore.defer_resume (default off).

Signed-off-by: Hongyu Xie <xiehongyu1@kylinos.cn>
---
 drivers/usb/core/driver.c | 139 +++++++++++++++++++++++++++++++++++++-
 drivers/usb/core/hub.c    |  16 +++++
 drivers/usb/core/usb.c    |   3 +
 drivers/usb/core/usb.h    |   1 +
 include/linux/usb.h       |  13 ++++
 5 files changed, 169 insertions(+), 3 deletions(-)

diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c
index f63004417058..286392104330 100644
--- a/drivers/usb/core/driver.c
+++ b/drivers/usb/core/driver.c
@@ -32,7 +32,7 @@
 #include <linux/usb/quirks.h>
 #include <linux/usb/hcd.h>
 
-#include "usb.h"
+#include "hub.h"	/* includes "usb.h" */
 
 
 /*
@@ -1220,6 +1220,78 @@ void usb_unbind_and_rebind_marked_interfaces(struct usb_device *udev)
 
 #ifdef CONFIG_PM
 
+/*
+ * Defer slow leaf-device resume off the S3 critical path.  The per-device
+ * port reset (~1-2 s for webcams, USB NICs, etc.) is moved to a work item
+ * queued from usb_resume_complete(), after all hubs have finished resume.
+ * The deferred work reuses the same udev (in-place reset, devnum
+ * preserved).  Drivers such as ax_usb_nic lack reset_resume; the work
+ * performs unbind+rebind after reset.
+ *
+ * If a new suspend or a disconnect arrives before the work item runs, the
+ * reset is abandoned, not flushed: the device is still port-suspended,
+ * which is exactly what the new suspend needs, and the work item then only
+ * releases the runtime reference taken by the deferred resume.  The
+ * abandon decision is serialized with the work item by the device lock, so
+ * no cancel_work_sync() (which would deadlock against the work item's
+ * device_lock()) is needed on the suspend path.
+ */
+bool usb_defer_resume_enabled;
+module_param_named(defer_resume, usb_defer_resume_enabled, bool, 0644);
+MODULE_PARM_DESC(defer_resume,
+	"defer leaf USB device resume to a workqueue (off the S3 critical path)");
+
+static int usb_resume_both(struct usb_device *udev, pm_message_t msg);
+
+static bool usb_should_defer_leaf_resume(struct usb_device *udev,
+		pm_message_t msg)
+{
+	struct usb_hub *hub;
+	u16 portstatus, portchange;
+
+	if (!usb_defer_resume_enabled || PMSG_IS_AUTO(msg))
+		return false;
+	if (!udev->parent || udev->maxchild)
+		return false;
+	if (udev->state != USB_STATE_SUSPENDED)
+		return false;
+
+	/*
+	 * Only defer when the port is not connected: link training is
+	 * either still pending or the device was removed while asleep,
+	 * which is where the long waits (the 2000 ms CONNECT timeout in
+	 * wait_for_connected()) happen.  A connected port resumes in a
+	 * bounded time, so keep the synchronous path -- and its exact
+	 * "resume complete means device usable" semantics -- for it.
+	 */
+	hub = usb_hub_to_struct_hub(udev->parent);
+	if (!hub)
+		return false;
+	if (usb_hub_port_status(hub, udev->portnum, &portstatus, &portchange))
+		return false;	/* cannot read port status: stay synchronous */
+	if (portstatus & USB_PORT_STAT_CONNECTION)
+		return false;	/* connected: resume is fast, keep sync */
+
+	return true;
+}
+
+/* Finish a deferred resume synchronously.  The caller holds the device lock. */
+static int usb_defer_flush_resume(struct usb_device *udev)
+{
+	pm_message_t msg = udev->defer_resume_msg;
+	int status;
+
+	status = usb_resume_both(udev, msg);
+	if (status == 0) {
+		unbind_marked_interfaces(udev);
+		rebind_marked_interfaces(udev);
+	} else if (status != -ENODEV && status != -ESHUTDOWN) {
+		/* Don't log when the device is being disconnected */
+		dev_err(&udev->dev, "deferred resume failed: %d\n", status);
+	}
+	return status;
+}
+
 /* Unbind drivers for @udev's interfaces that don't support suspend/resume
  * There is no check for reset_resume here because it can be determined
  * only during resume whether reset_resume is needed.
@@ -1587,6 +1659,30 @@ static int usb_resume_both(struct usb_device *udev, pm_message_t msg)
 	return status;
 }
 
+void usb_defer_resume_workfn(struct work_struct *work)
+{
+	struct usb_device *udev =
+		container_of(work, struct usb_device, defer_resume_work);
+	int status = -ENODEV;
+
+	device_lock(&udev->dev);
+	if (!udev->defer_resume_cancel) {
+		if (udev->state == USB_STATE_SUSPENDED)
+			status = usb_defer_flush_resume(udev);
+		else if (udev->state != USB_STATE_NOTATTACHED)
+			/* Already resumed via the runtime PM path. */
+			status = 0;
+	}
+	device_unlock(&udev->dev);
+
+	if (status == 0) {
+		pm_runtime_disable(&udev->dev);
+		pm_runtime_set_active(&udev->dev);
+		pm_runtime_enable(&udev->dev);
+	}
+	pm_runtime_put(&udev->dev);
+}
+
 static void choose_wakeup(struct usb_device *udev, pm_message_t msg)
 {
 	int	w;
@@ -1621,6 +1717,21 @@ int usb_suspend(struct device *dev, pm_message_t msg)
 	struct usb_device	*udev = to_usb_device(dev);
 	int r;
 
+	/* A deferred leaf resume may still be outstanding (queued or
+	 * running).  The device is still port-suspended, which is exactly
+	 * what this suspend needs, so abandon the deferred port reset:
+	 * if the work item was never queued, drop the runtime reference
+	 * taken by the deferred resume here; otherwise the flag makes
+	 * the work item skip the reset and just release the reference.
+	 * Serialized with the work item by the device lock.
+	 */
+	if (udev->defer_resume_pending) {
+		udev->defer_resume_pending = 0;
+		pm_runtime_put(&udev->dev);
+	} else {
+		udev->defer_resume_cancel = 1;
+	}
+
 	unbind_no_pm_drivers_interfaces(udev);
 
 	/* From now on we are sure all drivers support suspend/resume
@@ -1643,11 +1754,22 @@ int usb_resume_complete(struct device *dev)
 {
 	struct usb_device *udev = to_usb_device(dev);
 
+	if (udev->state == USB_STATE_NOTATTACHED)
+		return 0;
+
+	/* Queue the real port reset after dpm_complete, once every hub in
+	 * the tree has finished resume and downstream ports are settled.
+	 */
+	if (udev->defer_resume_pending) {
+		udev->defer_resume_pending = 0;
+		queue_work(system_unbound_wq, &udev->defer_resume_work);
+		return 0;
+	}
+
 	/* For PM complete calls, all we do is rebind interfaces
 	 * whose needs_binding flag is set
 	 */
-	if (udev->state != USB_STATE_NOTATTACHED)
-		rebind_marked_interfaces(udev);
+	rebind_marked_interfaces(udev);
 	return 0;
 }
 
@@ -1657,6 +1779,17 @@ int usb_resume(struct device *dev, pm_message_t msg)
 	struct usb_device	*udev = to_usb_device(dev);
 	int			status;
 
+	if (usb_should_defer_leaf_resume(udev, msg)) {
+		udev->defer_resume_msg = msg;
+		udev->defer_resume_pending = 1;
+		udev->defer_resume_cancel = 0;
+		pm_runtime_get_noresume(dev);
+		pm_runtime_disable(dev);
+		pm_runtime_set_suspended(dev);
+		pm_runtime_enable(dev);
+		return 0;
+	}
+
 	/* For all calls, take the device back to full power and
 	 * tell the PM core in case it was autosuspended previously.
 	 * Unbind the interfaces that will need rebinding later,
diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
index 5262e11c12cd..43a72ebefca7 100644
--- a/drivers/usb/core/hub.c
+++ b/drivers/usb/core/hub.c
@@ -2327,6 +2327,22 @@ void usb_disconnect(struct usb_device **pdev)
 	dev_info(&udev->dev, "USB disconnect, device number %d\n",
 			udev->devnum);
 
+#ifdef CONFIG_PM
+	/* A deferred leaf resume may still be outstanding.  If the work
+	 * item was never queued, drop the runtime reference taken by the
+	 * deferred resume here; otherwise cancel the work item so it
+	 * cannot run after teardown begins — it will release the
+	 * reference itself.  No device lock is held here, so the sync
+	 * cancel cannot deadlock against the work item.
+	 */
+	if (udev->defer_resume_pending) {
+		udev->defer_resume_pending = 0;
+		pm_runtime_put(&udev->dev);
+	}
+	if (cancel_work_sync(&udev->defer_resume_work))
+		pm_runtime_put(&udev->dev);
+#endif
+
 	/*
 	 * Ensure that the pm runtime code knows that the USB device
 	 * is in the process of being disconnected.
diff --git a/drivers/usb/core/usb.c b/drivers/usb/core/usb.c
index df166cafe106..2005c468ea14 100644
--- a/drivers/usb/core/usb.c
+++ b/drivers/usb/core/usb.c
@@ -681,6 +681,9 @@ struct usb_device *usb_alloc_dev(struct usb_device *parent,
 	/* ep0 maxpacket comes later, from device descriptor */
 	usb_enable_endpoint(dev, &dev->ep0, false);
 	dev->can_submit = 1;
+#ifdef CONFIG_PM
+	INIT_WORK(&dev->defer_resume_work, usb_defer_resume_workfn);
+#endif
 
 	/* Save readable and stable topology id, distinguishing devices
 	 * by location for diagnostics, tools, driver model, etc.  The
diff --git a/drivers/usb/core/usb.h b/drivers/usb/core/usb.h
index a9b37aeb515b..3dbeede2db62 100644
--- a/drivers/usb/core/usb.h
+++ b/drivers/usb/core/usb.h
@@ -95,6 +95,7 @@ extern int usb_port_disable(struct usb_device *udev);
 extern int usb_suspend(struct device *dev, pm_message_t msg);
 extern int usb_resume(struct device *dev, pm_message_t msg);
 extern int usb_resume_complete(struct device *dev);
+extern void usb_defer_resume_workfn(struct work_struct *work);
 
 extern int usb_port_suspend(struct usb_device *dev, pm_message_t msg);
 extern int usb_port_resume(struct usb_device *dev, pm_message_t msg);
diff --git a/include/linux/usb.h b/include/linux/usb.h
index 1da4ad1610bc..be69e9318b98 100644
--- a/include/linux/usb.h
+++ b/include/linux/usb.h
@@ -743,6 +743,19 @@ struct usb_device {
 
 	u16 hub_delay;
 	unsigned use_generic_driver:1;
+
+#ifdef CONFIG_PM
+	/* Deferred leaf-device resume: the slow port reset runs in this
+	 * work item after dpm_complete, reusing the same udev (devnum
+	 * preserved).  defer_resume_cancel is set by a new suspend to
+	 * abandon the reset; the work item then only drops the runtime
+	 * reference.
+	 */
+	struct work_struct defer_resume_work;
+	pm_message_t defer_resume_msg;
+	unsigned defer_resume_pending:1;
+	unsigned defer_resume_cancel:1;
+#endif
 };
 
 #define to_usb_device(__dev)	container_of_const(__dev, struct usb_device, dev)
-- 
2.25.1

Re: [RFC PATCH] usb: core: defer leaf-device resume off the S3 critical path

[Greg KH]

On Wed, Aug 05, 2026 at 05:36:41PM +0800, Hongyu Xie wrote:
> Gated by module parameter usbcore.defer_resume (default off).

This is not the 1990's, we don't add module parameters for core
functionality that should "just work".  Please make this dynamic and
work properly for all systems, without a boot option, and without the
#ifdef stuff.

thanks,

greg k-h

Re: [RFC PATCH] usb: core: defer leaf-device resume off the S3 critical path

[Marco Crivellari]

Hello,

On Wed, Aug 5, 2026 at 11:37 AM Hongyu Xie <xiehongyu1@kylinos.cn> wrote:
> [...]
> @@ -1643,11 +1754,22 @@ int usb_resume_complete(struct device *dev)
>  {
>         struct usb_device *udev = to_usb_device(dev);
>
> +       if (udev->state == USB_STATE_NOTATTACHED)
> +               return 0;
> +
> +       /* Queue the real port reset after dpm_complete, once every hub in
> +        * the tree has finished resume and downstream ports are settled.
> +        */
> +       if (udev->defer_resume_pending) {
> +               udev->defer_resume_pending = 0;
> +               queue_work(system_unbound_wq, &udev->defer_resume_work);
> +               return 0;
> +       }
> +

Please note that `system_unbound_wq` is the old unbound Workqueue. You
should use the newer, `system_dfl_wq`.

Thanks!

-- 

Marco Crivellari

SUSE Labs

Re: [RFC PATCH] usb: core: defer leaf-device resume off the S3 critical path

[Oliver Neukum]

On 05.08.26 11:36, Hongyu Xie wrote:
> Problem:
> System resume (dpm_resume) invokes each device's resume callback
> synchronously on the wake-up critical path.  For a USB leaf device
> this means port resume and, when reset_resume is needed, a port
> reset that, for persistent devices, waits up to 2000 ms for the
> device connection (CCS; on SuperSpeed this includes link training)
> to be re-established (wait_for_connected()).  Slow link training of
> some U-disks and webcams after S3 power-on therefore stalls the
> entire system wake-up: on an
> arm64 test machine with four leaf devices (bluetooth, UVC camera,
> two U-disks) dpm_resume took 2508 ms, about 2.1 s of it caused by a
> single SanDisk U-disk whose link training stayed in Rx.Detect for
> 2000 ms (worst case: -ENODEV fallback to disconnect +
> re-enumeration).  A device compatibility issue should not lengthen
> the system-wide wake-up.
> 
> Fix:
> The patch addresses this on four levels:
> 
> 1. Defer leaf-device recovery off the critical path.  Leaf devices
>     (udev->parent && !udev->maxchild) that are port-suspended on
>     system resume have their real port resume moved to a per-device
>     work item queued from usb_resume_complete(), after all hubs have
>     finished resume.  The work reuses the same udev (in-place reset,
>     devnum preserved); drivers without reset_resume get unbind+rebind.

If you do this, you may have drivers which do not runtime PM
face devices that are effectively runtime suspended. This will
not work.
  
> 2. Safe abandonment.  If a new suspend or a disconnect arrives before
>     the work item runs, the reset is abandoned, not flushed: the
>     device is still port-suspended, which is exactly what the new
>     suspend needs, and the work item then only releases the runtime
>     reference it took.  The abandon decision is serialized with the
>     work item by the device lock, so no cancel_work_sync() (which
>     would deadlock against the work item's device_lock()) is needed
>     on the suspend path.

You cannot do this without checking whether remote_wakeup is correctly
set.

Furthermore you cannot just use the system work queue for this.
You must not have any memory allocations with GFP_KERNEL on that
queue.

	Regards
		Oliver

Re: [RFC PATCH] usb: core: defer leaf-device resume off the S3 critical path

[Alan Stern]

On Wed, Aug 05, 2026 at 05:36:41PM +0800, Hongyu Xie wrote:
> Problem:
> System resume (dpm_resume) invokes each device's resume callback
> synchronously on the wake-up critical path.

I thought the PM core used asynchronous wakeups for most devices, 
including all USB devices.  They can delay the end of the wakeup 
transition, but the delays do not add; they can occur concurrently.

>  For a USB leaf device
> this means port resume and, when reset_resume is needed, a port
> reset that, for persistent devices, waits up to 2000 ms for the
> device connection (CCS; on SuperSpeed this includes link training)
> to be re-established (wait_for_connected()).  Slow link training of
> some U-disks and webcams after S3 power-on therefore stalls the
> entire system wake-up: on an
> arm64 test machine with four leaf devices (bluetooth, UVC camera,
> two U-disks) dpm_resume took 2508 ms, about 2.1 s of it caused by a
> single SanDisk U-disk whose link training stayed in Rx.Detect for
> 2000 ms (worst case: -ENODEV fallback to disconnect +
> re-enumeration).  A device compatibility issue should not lengthen
> the system-wide wake-up.

On the other hand, you don't want the system to be awake and running at 
a time when a disk drive is still suspended and unusable.  What will 
happens if a process needs to read from or write to that drive?

> Fix:
> The patch addresses this on four levels:
> 
> 1. Defer leaf-device recovery off the critical path.  Leaf devices
>    (udev->parent && !udev->maxchild) that are port-suspended on
>    system resume have their real port resume moved to a per-device
>    work item queued from usb_resume_complete(), after all hubs have
>    finished resume.  The work reuses the same udev (in-place reset,
>    devnum preserved); drivers without reset_resume get unbind+rebind.

And what happens if a hub needs a 2-second-long reset-resume?

> 2. Safe abandonment.  If a new suspend or a disconnect arrives before
>    the work item runs, the reset is abandoned, not flushed: the
>    device is still port-suspended, which is exactly what the new
>    suspend needs, and the work item then only releases the runtime
>    reference it took.  The abandon decision is serialized with the
>    work item by the device lock, so no cancel_work_sync() (which
>    would deadlock against the work item's device_lock()) is needed
>    on the suspend path.
> 
> 3. Truthful runtime state with lazy resume.  The deferred branch of
>    usb_resume() reports the device as suspended
>    (pm_runtime_set_suspended) until recovery actually completes; the
>    work item switches it back to active only after usb_resume_both()
>    succeeds.  Accesses made in between trigger
>    usb_autoresume_device() -> usb_port_resume() and resume the
>    device synchronously, so a successful operation is a real
>    confirmation of readiness.  A failed recovery leaves the device
>    suspended: later accesses fail or the device is disconnected,
>    both visible to user space.

Runtime-suspend and system-suspend are not the same thing.  In 
particular, they can have different wakeup settings.  You cannot simply 
substitute one for the other.

> 4. Port-state gating.  A device is only deferred when its port is
>    not connected (link training still pending, or the device was
>    removed while asleep) -- the case where the long waits happen.  A
>    connected port resumes in bounded time and keeps the synchronous
>    path with unchanged semantics.

What happens if other, non-USB, devices lie below the USB leaf device in 
the device tree?  How will they resume?  What will happen if the system 
needs to use them as soon as the system resume is finished?

> Results:
> With four leaf devices (bluetooth, UVC camera, Kingston U-disk,
> SanDisk U-disk), repeated S3 measurements (14 rounds):
> - without a 2000 ms event: dpm_resume drops from ~498-502 ms to
>   ~219-223 ms (-56%);
> - with the occasional 2000 ms link-training wait (or the fixed
>   2000 ms CONNECT timeout for a device removed during sleep):
>   dpm_resume drops from ~2508-2813 ms to ~219 ms (-91%), the wait
>   running in the background workqueue without affecting
>   system-wide resume.
> Deferred recovery completes in 0-20 ms in the common case.
> 
> Known limitations (no practical impact):
> - The system resume notification (PM_POST_SYSTEM_RESUME) is
>   system-wide and carries no device information; between the
>   notification and deferred recovery completion an application
>   accessing the device may fail.  Drivers using usb_autopm trigger
>   synchronous recovery instead (lazy resume), and the window is
>   milliseconds in practice.
> - usb_resume() returns success before the device is actually
>   recovered; the runtime PM state is truthful (suspended), but the
>   PM core counts the callback as successful.  No kernel path
>   depends on that accounting.
> - A recovery failure in the workqueue cannot be reported through
>   the dpm error path; it surfaces as device disconnect, which user
>   space already handles.
> 
> Gated by module parameter usbcore.defer_resume (default off).

Overall this does not sound like a good idea.  At the very least, there 
are several implications you have not considered.

Also, have you looked into the smart_suspend and may_skip_resume flags 
in struct dev_pm_info?  Can some combination of them accomplish what you 
want with minimal code changes?

Alan Stern