s2idle resume hangs in ohci/ehci on RK3588: HC registers touched before the USB2 PHY is powered back on

s2idle resume hangs in ohci/ehci on RK3588: HC registers touched before the USB2 PHY is powered back on

[Igor Paunovic]

Hi,

on an Orange Pi 5 Plus (RK3588, mainline 7.3-rc1 based tree, generic-ohci/generic-ehci with
phy-rockchip-inno-usb2) resume from suspend-to-idle reliably hangs in the USB 2.0 host
controller resume path. The CPU that runs ohci_platform_resume() never returns (RCU stall,
"CPUs still haven't responded to the NMI"), everything that waits on it in dpm_resume()
stalls behind it, and the board needs a cold reset. With the four USB 2.0 host controllers
unbound before suspend the same s2idle cycle completes every time (RTC alarm wake, full
resume, 3 s), so the rest of the platform is fine.

Wake-up itself works: the board is woken by the hym8563 RTC alarm (PM: Triggering wakeup
from IRQ 52) - that needed a separate dts fix which I sent yesterday [1].

To find where it stops I put kprobes (with tp_printk) on the resume path. For the two
OHCI controllers, same kernel, same cycle:

  fc8c0000.usb (host1, survives):
    ohci_platform_resume -> ohci_platform_power_on (clocks only) -> 0
    ohci_resume  entered        @142.838
    usb_hcd_resume_root_hub     @142.859   (the "powerup ports" + msleep(20) branch)
    ohci_resume  returned 0
    ... later, root hub resume: usb_phy_roothub_resume -> rockchip_usb2phy_init/power_on
        -> ohci_rh_resume
  fc840000.usb (host0, hangs):
    ohci_platform_resume -> ohci_platform_power_on (clocks only) -> 0
    ohci_resume  entered        @142.994
    (nothing else, ever)

So the hang is at the first HC register access in ohci_resume() (the ohci_readl() of
HcControl / the port power / intrenable writes), which happens before the USB2 PHY is
powered on again: since the PHY handling moved into the HCD core (usb_phy_roothub), the
PHYs are powered off in hcd_bus_suspend() and powered on in hcd_bus_resume() (both only for
system sleep, !PMSG_IS_AUTO), i.e. at root hub level, and the root hub resumes after its
parent controller. At probe time the order is the opposite: usb_add_hcd() does
usb_phy_roothub_power_on() before hcd->driver->reset(). That matches what I see: after a
"hosts unbound" s2idle cycle, binding the drivers again (probe path) reliably works, while
the resume path hangs.

On RK3588 the inno-usb2 PHY powers down its PLL/refclk/bias blocks while suspended
(phy-rockchip-inno-usb2.c, comment in rockchip_usb2phy_power_on() about common_on_n and the
reset done on power-on), and the OHCI/EHCI controllers take one of their clocks from that
PHY (clocks = <&cru HCLK_HOST0>, ..., <&u2phy2>). So an AHB access to the controller while
its PHY is still suspended has no clock to complete on, and the access never returns - which
would explain the "no reaction to NMI" symptom (this is my best explanation so far, not
confirmed with a bus or clock trace). The Rockchip vendor tree avoids this by giving the PHY
driver system PM ops that reset and re-tune the PHY on resume, before the consumers run
(rockchip-linux/kernel, develop-6.1, phy-rockchip-inno-usb2.c, rockchip_usb2phy_pm_resume():
"PHY lost power in suspend, it needs to reset PHY to recovery clock to usb controller").

Data points (all dvfs test kernel, 7.3.0-rc1 based; every "hang" needed a cold reset):
  - all 4 USB2 hosts bound, s2idle:            hang (ehci x2 + ohci calling, none returned)
  - EHCI unbound, OHCI bound:                  hang in ohci_resume of fc840000 (2/2)
  - all 4 unbound:                             OK (4/4)
  - all bound, cpuidle limited to WFI:         OK (1/1)   <- timing dependent, see below
  - the surviving controller (fc8c0000) took the "HC state retained" branch of
    ohci_resume(); the hanging one (fc840000) did not get past the first register access.

(The OHCI platform devices resume synchronously from the main dpm_resume() thread - power/async
is disabled for them - which is why nothing else in the resume sequence is printed after the
hang; the EHCI ones are async, so in the all-bound case two ehci and one ohci resume were in
flight when everything stopped.)

Why host1 survives and host0 does not (identical PHY port configs) - reading the PHY GRF
status and the clk enable counts right before suspend, in the test configuration (EHCI
unbound, OHCI bound): the u2phy2 port (host0, nothing plugged in) is already in PHY suspend
(GRF status phy_sus set, no line state) and its usb480m clock has the OHCI as its only user;
the u2phy3 port (host1, a HID dongle plugged in) is not suspended and its clock has more than
one user.

So the port without a device is already put into suspend by the PHY driver's host-port state
machine, and its 480 MHz clock has a single user (the OHCI): ohci_platform_suspend() drops it
to zero and the clock output is gated; ohci_platform_resume() re-enables the clock, but the PHY
itself stays suspended (PLL down) until the root hub resume powers it on, so the first HC
register access would have no clock to complete on. The port with a device connected keeps its
PHY awake and its clock never reaches zero, so the same code path survives there. With the EHCI
siblings bound the outcome depends on timing (the async EHCI root hub may power the shared PHY
on before the synchronous OHCI resume touches its registers), which would match the mixed
results. Also: after suspend the OHCI ends up in the RCU-stall/no-NMI-response state, i.e. the
CPU is stuck in the bus access, not in a software wait.

Questions:
  1. Is the intended fix to power the roothub PHYs on before the controller's own resume
     touches the hardware (e.g. in ohci_resume()/ehci_resume(), or a usb_phy_roothub_resume()
     call from the platform glue before ohci_resume()), or should this be handled in the
     Rockchip PHY driver with system PM ops as the vendor tree does?
  2. Has anyone got s2idle + USB2 host working on RK3588 mainline? I could not find a report
     on lore.

I can test patches on this board (UART console logging is in place), or try one myself
for whichever layer you think is right - I have not attempted a fix yet because that choice
is the question. Per Documentation/process/coding-assistants.rst: the kprobe placement and
the log triage above were done with the help of an LLM assistant; all measurements are from
the board and the reproducer is the unbind/bind matrix above.

[1] https://lore.kernel.org/all/20260906181622.11991-1-royalnet026@gmail.com/

Kernel: 7.3.0-rc1 based (drm-misc-next + accel/rocket DVFS series), BL31 v2.12.0-10-g70d814213
(v2.12.0 plus one local cherry-pick unrelated to USB), Orange Pi 5 Plus.

Igor

Re: s2idle resume hangs in ohci/ehci on RK3588: HC registers touched before the USB2 PHY is powered back on

[Sebastian Reichel]

[-- Attachment #1: Type: text/plain, Size: 7910 bytes --]

Hello Igor,

On Mon, Sep 07, 2026 at 07:33:47PM +0200, Igor Paunovic wrote:
> on an Orange Pi 5 Plus (RK3588, mainline 7.3-rc1 based tree, generic-ohci/generic-ehci with
> phy-rockchip-inno-usb2) resume from suspend-to-idle reliably hangs in the USB 2.0 host
> controller resume path. The CPU that runs ohci_platform_resume() never returns (RCU stall,
> "CPUs still haven't responded to the NMI"), everything that waits on it in dpm_resume()
> stalls behind it, and the board needs a cold reset. With the four USB 2.0 host controllers
> unbound before suspend the same s2idle cycle completes every time (RTC alarm wake, full
> resume, 3 s), so the rest of the platform is fine.
> 
> Wake-up itself works: the board is woken by the hym8563 RTC alarm (PM: Triggering wakeup
> from IRQ 52) - that needed a separate dts fix which I sent yesterday [1].
> 
> To find where it stops I put kprobes (with tp_printk) on the resume path. For the two
> OHCI controllers, same kernel, same cycle:
> 
>   fc8c0000.usb (host1, survives):
>     ohci_platform_resume -> ohci_platform_power_on (clocks only) -> 0
>     ohci_resume  entered        @142.838
>     usb_hcd_resume_root_hub     @142.859   (the "powerup ports" + msleep(20) branch)
>     ohci_resume  returned 0
>     ... later, root hub resume: usb_phy_roothub_resume -> rockchip_usb2phy_init/power_on
>         -> ohci_rh_resume
>   fc840000.usb (host0, hangs):
>     ohci_platform_resume -> ohci_platform_power_on (clocks only) -> 0
>     ohci_resume  entered        @142.994
>     (nothing else, ever)
> 
> So the hang is at the first HC register access in ohci_resume() (the ohci_readl() of
> HcControl / the port power / intrenable writes), which happens before the USB2 PHY is
> powered on again: since the PHY handling moved into the HCD core (usb_phy_roothub), the
> PHYs are powered off in hcd_bus_suspend() and powered on in hcd_bus_resume() (both only for
> system sleep, !PMSG_IS_AUTO), i.e. at root hub level, and the root hub resumes after its
> parent controller. At probe time the order is the opposite: usb_add_hcd() does
> usb_phy_roothub_power_on() before hcd->driver->reset(). That matches what I see: after a
> "hosts unbound" s2idle cycle, binding the drivers again (probe path) reliably works, while
> the resume path hangs.
> 
> On RK3588 the inno-usb2 PHY powers down its PLL/refclk/bias blocks while suspended
> (phy-rockchip-inno-usb2.c, comment in rockchip_usb2phy_power_on() about common_on_n and the
> reset done on power-on), and the OHCI/EHCI controllers take one of their clocks from that
> PHY (clocks = <&cru HCLK_HOST0>, ..., <&u2phy2>). So an AHB access to the controller while
> its PHY is still suspended has no clock to complete on, and the access never returns - which
> would explain the "no reaction to NMI" symptom (this is my best explanation so far, not
> confirmed with a bus or clock trace). The Rockchip vendor tree avoids this by giving the PHY
> driver system PM ops that reset and re-tune the PHY on resume, before the consumers run
> (rockchip-linux/kernel, develop-6.1, phy-rockchip-inno-usb2.c, rockchip_usb2phy_pm_resume():
> "PHY lost power in suspend, it needs to reset PHY to recovery clock to usb controller").
> 
> Data points (all dvfs test kernel, 7.3.0-rc1 based; every "hang" needed a cold reset):
>   - all 4 USB2 hosts bound, s2idle:            hang (ehci x2 + ohci calling, none returned)
>   - EHCI unbound, OHCI bound:                  hang in ohci_resume of fc840000 (2/2)
>   - all 4 unbound:                             OK (4/4)
>   - all bound, cpuidle limited to WFI:         OK (1/1)   <- timing dependent, see below
>   - the surviving controller (fc8c0000) took the "HC state retained" branch of
>     ohci_resume(); the hanging one (fc840000) did not get past the first register access.
> 
> (The OHCI platform devices resume synchronously from the main dpm_resume() thread - power/async
> is disabled for them - which is why nothing else in the resume sequence is printed after the
> hang; the EHCI ones are async, so in the all-bound case two ehci and one ohci resume were in
> flight when everything stopped.)
> 
> Why host1 survives and host0 does not (identical PHY port configs) - reading the PHY GRF
> status and the clk enable counts right before suspend, in the test configuration (EHCI
> unbound, OHCI bound): the u2phy2 port (host0, nothing plugged in) is already in PHY suspend
> (GRF status phy_sus set, no line state) and its usb480m clock has the OHCI as its only user;
> the u2phy3 port (host1, a HID dongle plugged in) is not suspended and its clock has more than
> one user.
> 
> So the port without a device is already put into suspend by the PHY driver's host-port state
> machine, and its 480 MHz clock has a single user (the OHCI): ohci_platform_suspend() drops it
> to zero and the clock output is gated; ohci_platform_resume() re-enables the clock, but the PHY
> itself stays suspended (PLL down) until the root hub resume powers it on, so the first HC
> register access would have no clock to complete on. The port with a device connected keeps its
> PHY awake and its clock never reaches zero, so the same code path survives there. With the EHCI
> siblings bound the outcome depends on timing (the async EHCI root hub may power the shared PHY
> on before the synchronous OHCI resume touches its registers), which would match the mixed
> results. Also: after suspend the OHCI ends up in the RCU-stall/no-NMI-response state, i.e. the
> CPU is stuck in the bus access, not in a software wait.
> 
> Questions:
>   1. Is the intended fix to power the roothub PHYs on before the controller's own resume
>      touches the hardware (e.g. in ohci_resume()/ehci_resume(), or a usb_phy_roothub_resume()
>      call from the platform glue before ohci_resume()), or should this be handled in the
>      Rockchip PHY driver with system PM ops as the vendor tree does?

Coincidentally I looked into this issue last week when testing my
PCIe patches on RK3588 EVB1 instead of RK3576 and fixed it up from
the clock path. From my perspective the root cause is with the clock
registered by the PHY driver. The clock consumer expects it to be
running when the enable function succeeds and in case of the PHY
that is not true for the suspend resume path. The PHY just ungates
the clock, but does not take care of resuming the "parent" PLL.

I've not yet send out the fix, but you you can find v0 here (part
of the rockchip-devel branch):

https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commit/53014abcf948b7afcb2150ea690a8f6062cf66a6

>   2. Has anyone got s2idle + USB2 host working on RK3588 mainline?
>      I could not find a report on lore.

mainline still lacks my RK3588/RK3576 PCIe suspend patches (I plan
to send a new version this week). Thus with pure mainline the PCIe
driver will block the RK3588 from going into suspend in the first
place.

FWIW There is also a bunch of other bugs around system suspend on
RK3588; most of them are less critical though. Also some BL31
firmwares seem to be broken.

> I can test patches on this board (UART console logging is in place), or try one myself
> for whichever layer you think is right - I have not attempted a fix yet because that choice
> is the question. Per Documentation/process/coding-assistants.rst: the kprobe placement and
> the log triage above were done with the help of an LLM assistant; all measurements are from
> the board and the reproducer is the unbind/bind matrix above.
> 
> [1] https://lore.kernel.org/all/20260906181622.11991-1-royalnet026@gmail.com/
> 
> Kernel: 7.3.0-rc1 based (drm-misc-next + accel/rocket DVFS series), BL31 v2.12.0-10-g70d814213
> (v2.12.0 plus one local cherry-pick unrelated to USB), Orange Pi 5 Plus.

Greetings,

-- Sebastian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: s2idle resume hangs in ohci/ehci on RK3588: HC registers touched before the USB2 PHY is powered back on

[Igor Paunovic]

Hello Sebastian,

Thank you, that is a much better place to fix it than either of the two
options I asked about, and it matches what the board shows: the OHCI
that hung was the one whose 480 MHz clock had it as the only user, so
the clock was gated on suspend and the PHY was left in suspend when the
controller came back and touched its registers.

I read 53014abc in the rockchip-devel branch. One data point in case it
is useful for the version you send: on RK3588 the early return does not
get in the way.

    /* Limit to single port; it's unclear how multi-port should be handled */
    if (rphy->phy_cfg->num_ports > 1)
            return 0;

All four entries in rk3588_phy_cfgs[] have num_ports = 1 (0x0000 and
0x4000 as OTG, 0x8000 and 0xc000 as HOST), so the new code does run on
RK3588, including u2phy2 at 0x8000, which is the port whose controller
(fc840000, host0) hung for me. The num_ports = 2 configs are the older
SoCs, so as far as RK3588 is concerned that guard costs nothing.

I will build the patch for my test kernel and report back. My setup
should be a reasonable test case for it: with all four USB 2.0 hosts
bound, s2idle resume hung every time and needed a cold reset, and with
the four hosts unbound the same cycle completed 4 out of 4, so there is
a clear before/after to measure rather than an intermittent one. The
driver is a module in my configs, so this is a single module rebuild.
If it holds up I will send a Tested-by.

One observation on your second point. On this board s2idle is entered:
the machine goes down, the hym8563 RTC alarm wakes it ("PM: Triggering
wakeup from IRQ 52"), and only then does the resume stall in the USB2
hosts. So PCIe did not block suspend here, even though the board has
NVMe, two r8169 NICs and an rtw89 card on PCIe. I do not know why my
case differs from yours; if that is interesting for your PCIe suspend
series I can dump whatever state you want from before the suspend.

The RTC pin fix I mentioned is now v2 on the list [1], with megi's
Reviewed-by. Without it there is no wake source on this board at all,
so it is a prerequisite for anyone else trying to reproduce this.

If you would rather I wait for the version you post this week instead of
testing the branch commit, just say so and I will wait.

[1] https://lore.kernel.org/all/20260908111908.19371-1-royalnet026@gmail.com/

Igor

Re: s2idle resume hangs in ohci/ehci on RK3588: HC registers touched before the USB2 PHY is powered back on

[Igor Paunovic]

Hello Sebastian,

I tested 53014abc on the Orange Pi 5 Plus and it fixes the hang here.

Setup: same kernel (7.3.0-rc1 based), same dts, same procedure as the
report, all four USB 2.0 hosts bound, RTC alarm as the wakeup source.
The only change is that commit in phy-rockchip-inno-usb2.

Result: 4 full s2idle cycles, all of them resumed. The controller that
used to hang, fc840000 (the OHCI on u2phy2, the port with nothing
plugged in whose 480 MHz clock had it as its only user), went from
"enters ohci_platform_resume and never returns" to:

  ohci-platform fc840000.usb: PM: ohci_platform_resume returned 0 after 20649 usecs
  ohci-platform fc840000.usb: PM: ohci_platform_resume returned 0 after 20773 usecs
  ...

8 out of 8 resume passes in that boot returned 0 (4 real s2idle cycles
plus the pm_test device/platform phases), each around 20.7 ms, and the
same for fc8c0000. For comparison, before the patch the all-bound
configuration hung every single time and needed a cold reset, while the
control with the four hosts unbound completed 4 out of 4.

Tested-by: Igor Paunovic <royalnet026@gmail.com> # Orange Pi 5 Plus (RK3588)

Two things about how the module was built, so the tag is not read as
more than it is. The kernel was built with aarch64-linux-gnu-gcc 13.3.0
and I built the module with gcc 15.2.0 on the board itself, against the
installed linux-headers package (that package is arm64, so its host
tools cannot run on my x86 build machine). vermagic and the modversions
CRCs match, and as a control I first rebuilt the unmodified file from
the same commit and got exactly the srcversion of the stock module, so
the difference in the tested module comes from your patch and nothing
else. The module is also unsigned, since the headers package has no
private key, so the kernel is tainted O and E.

Two log entries I do not think are related, mentioned so you know I read
the whole log rather than only the part I was looking for. On one resume
there is an rcu_preempt stall report, but RCU marks the CPUs
"(false positive?)", the NMI is answered, and the backtraces show them in
cpuidle_enter_s2idle, i.e. asleep where they should be - the opposite of
the original failure, where the CPUs did not answer the NMI at all. And
on one of the three resumes there is a WARNING in
drm_crtc_wait_one_vblank ("vblank wait timed out on crtc 0"). It is
display side, and it happened on one resume only rather than on all of
them; one of the monitors on this board goes into its own power saving
after a while, which would leave the CRTC with no vblank to wait for
while keeping HPD asserted, so nothing about it appears in the log. I
cannot prove that from the log either way, but it is not the USB path.

Also, the network check right after resume fails for about a second
before r8169 reports "Link is Up - 2.5Gbps/Full"; that is my script
asking too early, not a regression.

Happy to re-run this on the version you post this week, and to test the
PCIe suspend series on this board as well if that is useful - it has
NVMe, two r8169 NICs and an rtw89 card on PCIe.

Igor