Re: [PATCH 0/4] Verify devices transition from D3cold to D0

[Mika Westerberg <mika.westerberg@linux.intel.com>]

On Tue, Jun 25, 2024 at 10:43:20AM -0500, Mario Limonciello wrote:
> On 6/19/2024 13:50, Mario Limonciello wrote:
> > On 6/19/2024 00:29, Mika Westerberg wrote:
> > > On Tue, Jun 18, 2024 at 11:56:50AM -0500, Mario Limonciello wrote:
> > > > On 6/18/2024 08:14, Mika Westerberg wrote:
> > > > > Hi Mario,
> > > > > 
> > > > > On Thu, Jun 13, 2024 at 12:42:00AM -0500, Mario Limonciello wrote:
> > > > > > Gary has reported that when a dock is plugged into a
> > > > > > system at the same
> > > > > > time the autosuspend delay has tripped that the USB4
> > > > > > stack malfunctions.
> > > > > > 
> > > > > > Messages show up like this:
> > > > > > 
> > > > > > ```
> > > > > > thunderbolt 0000:e5:00.6: ring_interrupt_active:
> > > > > > interrupt for TX ring 0 is already enabled
> > > > > > ```
> > > > > > 
> > > > > > Furthermore the USB4 router is non-functional at this point.
> > > > > 
> > > > > Once the USB4 domain starts the sleep transition, it cannot be
> > > > > interrupted by anything so it always should go through full sleep
> > > > > transition and only then back from sleep.
> > > > > 
> > > > > > Those messages happen because the device is still in
> > > > > > D3cold at the time
> > > > > > that the PCI core handed control back to the USB4 connection manager
> > > > > > (thunderbolt).
> > > > > 
> > > > > This is weird. Yes we should be getting the wake from the hotplug but
> > > > > that should happen only after the domain is fully in sleep
> > > > > (D3cold). The
> > > > > BIOS ACPI code is supposed to deal with this.
> > > > 
> > > > Is that from from experience or do you mean there is a spec behavior?
> > > > 
> > > > IE I'm wondering if we have different "expectations" from different
> > > > company's hardware designers.
> > > 
> > > The spec and the CM guide "imply" this behaviour as far as I can tell,
> > > so that the "sleep event" is done completely once started. I guess this
> > > can be interpreted differently too because it is not explicitly said
> > > there.
> > > 
> > > Can you ask AMD HW folks if this is their interpretation too? Basically
> > > when we get "Sleep Ready" bit set for all the routers in the domain and
> > > turn off power (send PERST) there cannot be wake events until that is
> > > fully completed.
> > > 
> > > There is typically a timeout mechanism in the BIOS side (part of the
> > > power off method) that waits for the PCIe links to enter L2 before it
> > > triggers PERST. We have seen an issue on our side that if this L2
> > > transition is not completed in time a wake event triggered but that was
> > > a BIOS issue.
> > 
> > Sure thing.  I'll discuss it with them and get back with the results.
> 
> From the hardware team they describe this as an abnormal state that they
> don't expect.  I don't believe there is anything in the BIOS to prevent it
> though.

Okay thanks for checking.

> 
> I could discuss options for this with the BIOS team in the future for the
> USB4 router ACPI device, but as this "seems" to be the same problem as XHCI
> controllers going back at least 5 generations with those quirks I put
> reverts in this series I think a general kernel solution to make "sure" that
> devices have transitioned is the better way to go.

Agreed.