Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[rick]

Hi all,

I am having the exact same issue. 

linux-lts-6.6.28-1 works, anything above doesn’t.

When kernel above linux-lts-6.6.28-1:
- Boltctl does not show anything
- thunderbolt.host_reset=0 had no impact
- triggers following errors:
  [   50.627948] ucsi_acpi USBC000:00: unknown error 0
  [   50.627957] ucsi_acpi USBC000:00: UCSI_GET_PDOS failed (-5)

Gists:
- https://gist.github.com/ricklahaye/83695df8c8273c30d2403da97a353e15 dmesg
with “Linux system 6.11.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 17 Oct 2024
20:53:41 +0000 x86_64 GNU/Linux” where thunderbolt dock does not work
- https://gist.github.com/ricklahaye/79e4040abcd368524633e86addec1833 dmesg
with “Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024
10:11:09 +0000 x86_64 GNU/Linux” where thunderbolt does work
- https://gist.github.com/ricklahaye/c9a7b4a7eeba5e7900194eecf9fce454
boltctl with “Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr
2024 10:11:09 +0000 x86_64 GNU/Linux” where thunderbolt does work


Kind regards,
Rick

Ps: sorry for resend; this time with plain text format

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mario Limonciello]

On 10/22/2024 07:13, rick@581238.xyz wrote:
> Hi all,
> 
> I am having the exact same issue.
> 
> linux-lts-6.6.28-1 works, anything above doesn't.
> 
> When kernel above linux-lts-6.6.28-1:
> - Boltctl does not show anything
> - thunderbolt.host_reset=0 had no impact
> - triggers following errors:
>    [   50.627948] ucsi_acpi USBC000:00: unknown error 0
>    [   50.627957] ucsi_acpi USBC000:00: UCSI_GET_PDOS failed (-5)
> 
> Gists:
> - https://gist.github.com/ricklahaye/83695df8c8273c30d2403da97a353e15 dmesg
> with "Linux system 6.11.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 17 Oct 2024
> 20:53:41 +0000 x86_64 GNU/Linux" where thunderbolt dock does not work
> - https://gist.github.com/ricklahaye/79e4040abcd368524633e86addec1833 dmesg
> with "Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024
> 10:11:09 +0000 x86_64 GNU/Linux" where thunderbolt does work
> - https://gist.github.com/ricklahaye/c9a7b4a7eeba5e7900194eecf9fce454
> boltctl with "Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr
> 2024 10:11:09 +0000 x86_64 GNU/Linux" where thunderbolt does work
> 
> 
> Kind regards,
> Rick
> 
> Ps: sorry for resend; this time with plain text format
> 
> 

Can you please share a log with 'thunderbolt.host_reset=0 
thunderbolt.dyndbg' on the kernel command line in a kernel that it 
doesn't work?  This should make the behavior match 6.6.28 and we can 
compare.

Maybe the best thing would be:
* 6.6.28 w/ thunderbolt.dyndbg
* 6.6.29 w/ thunderbolt.dyndbg thunderbolt.host_reset=0

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mario,

I apologize. I think I mixed up the versions between linux-lts and linux 
kernel.

linux-6.6.28-1-lts works: 
https://gist.github.com/ricklahaye/610d137b4816370cd6c4062d391e9df5
linux-6.6.57-1-lts works: 
https://gist.github.com/ricklahaye/48d5a44467fc29abe2b4fd04050309d7

linux-6.11.4-arch2-1 doesn't work: 
https://gist.github.com/ricklahaye/3b13a093e707acd0882203a56e184d3f
linux-6.11.4-arch2-1 with host_reset on 0 also doesn't work: 
https://gist.github.com/ricklahaye/ea2f4a04f7b9bedcbcce885df09a0388

Kind regards,

Rick


On 22-10-2024 14:55, Mario Limonciello wrote:
> On 10/22/2024 07:13, rick@581238.xyz wrote:
>> Hi all,
>>
>> I am having the exact same issue.
>>
>> linux-lts-6.6.28-1 works, anything above doesn't.
>>
>> When kernel above linux-lts-6.6.28-1:
>> - Boltctl does not show anything
>> - thunderbolt.host_reset=0 had no impact
>> - triggers following errors:
>>    [   50.627948] ucsi_acpi USBC000:00: unknown error 0
>>    [   50.627957] ucsi_acpi USBC000:00: UCSI_GET_PDOS failed (-5)
>>
>> Gists:
>> - https://gist.github.com/ricklahaye/83695df8c8273c30d2403da97a353e15 
>> dmesg
>> with "Linux system 6.11.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 17 Oct 
>> 2024
>> 20:53:41 +0000 x86_64 GNU/Linux" where thunderbolt dock does not work
>> - https://gist.github.com/ricklahaye/79e4040abcd368524633e86addec1833 
>> dmesg
>> with "Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024
>> 10:11:09 +0000 x86_64 GNU/Linux" where thunderbolt does work
>> - https://gist.github.com/ricklahaye/c9a7b4a7eeba5e7900194eecf9fce454
>> boltctl with "Linux system 6.6.28-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 
>> 17 Apr
>> 2024 10:11:09 +0000 x86_64 GNU/Linux" where thunderbolt does work
>>
>>
>> Kind regards,
>> Rick
>>
>> Ps: sorry for resend; this time with plain text format
>>
>>
>
> Can you please share a log with 'thunderbolt.host_reset=0 
> thunderbolt.dyndbg' on the kernel command line in a kernel that it 
> doesn't work?  This should make the behavior match 6.6.28 and we can 
> compare.
>
> Maybe the best thing would be:
> * 6.6.28 w/ thunderbolt.dyndbg
> * 6.6.29 w/ thunderbolt.dyndbg thunderbolt.host_reset=0
>

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

Hi,

On Tue, Oct 22, 2024 at 05:44:18PM +0200, Rick wrote:
> Hi Mario,
> 
> I apologize. I think I mixed up the versions between linux-lts and linux
> kernel.
> 
> linux-6.6.28-1-lts works:
> https://gist.github.com/ricklahaye/610d137b4816370cd6c4062d391e9df5
> linux-6.6.57-1-lts works:
> https://gist.github.com/ricklahaye/48d5a44467fc29abe2b4fd04050309d7
> 
> linux-6.11.4-arch2-1 doesn't work:
> https://gist.github.com/ricklahaye/3b13a093e707acd0882203a56e184d3f
> linux-6.11.4-arch2-1 with host_reset on 0 also doesn't work:
> https://gist.github.com/ricklahaye/ea2f4a04f7b9bedcbcce885df09a0388

Looks like some sort of connectivity issue to me.

However, can you first drop the "pcie_aspm=force" from the command line
and see if that has any affect. Probably does not but I suggest not to
keep it unless you really know that you want force ASPM on all PCIe
links (this may cause issues with some devices).

Anyways, even the working -lts ones you see the link goes down and up
which is not expected (unless you unplugged and plugged it back). Some
devices that are not Thunderbolt certified have this kinds of issues.
The cable could make difference too. Is it Thunderbolt 4 cable or
regular one?

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mario Limonciello]

On 10/22/2024 11:10, Mika Westerberg wrote:
> Hi,
> 
> On Tue, Oct 22, 2024 at 05:44:18PM +0200, Rick wrote:
>> Hi Mario,
>>
>> I apologize. I think I mixed up the versions between linux-lts and linux
>> kernel.
>>
>> linux-6.6.28-1-lts works:
>> https://gist.github.com/ricklahaye/610d137b4816370cd6c4062d391e9df5
>> linux-6.6.57-1-lts works:
>> https://gist.github.com/ricklahaye/48d5a44467fc29abe2b4fd04050309d7
>>
>> linux-6.11.4-arch2-1 doesn't work:
>> https://gist.github.com/ricklahaye/3b13a093e707acd0882203a56e184d3f
>> linux-6.11.4-arch2-1 with host_reset on 0 also doesn't work:
>> https://gist.github.com/ricklahaye/ea2f4a04f7b9bedcbcce885df09a0388
> 
> Looks like some sort of connectivity issue to me.
> 
> However, can you first drop the "pcie_aspm=force" from the command line
> and see if that has any affect. Probably does not but I suggest not to
> keep it unless you really know that you want force ASPM on all PCIe
> links (this may cause issues with some devices).
> 
> Anyways, even the working -lts ones you see the link goes down and up
> which is not expected (unless you unplugged and plugged it back). Some
> devices that are not Thunderbolt certified have this kinds of issues.
> The cable could make difference too. Is it Thunderbolt 4 cable or
> regular one?

And if it's really starting to fail consistently 6.6.29, the number of 
commits is very small, it should be a pretty quick bisect.

❯ git log --oneline v6.6.28..v6.6.29 | wc -l
158

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mika,

I have removed pcie_asm=force as kernel parameter but still not working 
on latest non LTS kernel.

In regards to the disconnect; sorry I think I might have turned of the 
docking station myself during that test. I have taken another dmesg 
without me disconnecting the docking station: 
https://gist.github.com/ricklahaye/9798b7de573d0f29b3ada6a5d99b69f1

The cable is the original Thunderbolt 4 cable that came with the docking 
station. I have used it on this laptop using Windows (dualboot) without 
any issues. Also on another Windows laptop also without issues. It was 
used in 40Gbit mode.

So right now all linux-lts versions work with the dock, but linux non 
LTS does not.

Kind regards,
Rick


On 22-10-2024 18:10, Mika Westerberg wrote:
> Hi,
>
> On Tue, Oct 22, 2024 at 05:44:18PM +0200, Rick wrote:
>> Hi Mario,
>>
>> I apologize. I think I mixed up the versions between linux-lts and linux
>> kernel.
>>
>> linux-6.6.28-1-lts works:
>> https://gist.github.com/ricklahaye/610d137b4816370cd6c4062d391e9df5
>> linux-6.6.57-1-lts works:
>> https://gist.github.com/ricklahaye/48d5a44467fc29abe2b4fd04050309d7
>>
>> linux-6.11.4-arch2-1 doesn't work:
>> https://gist.github.com/ricklahaye/3b13a093e707acd0882203a56e184d3f
>> linux-6.11.4-arch2-1 with host_reset on 0 also doesn't work:
>> https://gist.github.com/ricklahaye/ea2f4a04f7b9bedcbcce885df09a0388
> Looks like some sort of connectivity issue to me.
>
> However, can you first drop the "pcie_aspm=force" from the command line
> and see if that has any affect. Probably does not but I suggest not to
> keep it unless you really know that you want force ASPM on all PCIe
> links (this may cause issues with some devices).
>
> Anyways, even the working -lts ones you see the link goes down and up
> which is not expected (unless you unplugged and plugged it back). Some
> devices that are not Thunderbolt certified have this kinds of issues.
> The cable could make difference too. Is it Thunderbolt 4 cable or
> regular one?

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

Hi,

On Tue, Oct 22, 2024 at 07:06:50PM +0200, Rick wrote:
> Hi Mika,
> 
> I have removed pcie_asm=force as kernel parameter but still not working on
> latest non LTS kernel.

Okay, I still suggest not having that unless you absolutely know that
you need it.

> In regards to the disconnect; sorry I think I might have turned of the
> docking station myself during that test. I have taken another dmesg without
> me disconnecting the docking station:
> https://gist.github.com/ricklahaye/9798b7de573d0f29b3ada6a5d99b69f1
> 
> The cable is the original Thunderbolt 4 cable that came with the docking
> station. I have used it on this laptop using Windows (dualboot) without any
> issues. Also on another Windows laptop also without issues. It was used in
> 40Gbit mode.

In the dmesg you shared above, there are still unplug and USB tunnel
creation fails so you only get USB 2.x connection with all the USB
devices on the dock.

How do you determine if it "works"? I guess keyboard and mouse (both
USB 2.x devices) and display (tunneled over USB4 link) all are working
right? However, if you plug in USB 3.x device to the dock it enumerates
as FullSpeed instead of SuperSpeed. There is definitely something wrong
here. I asked from our TB validation folks if they have any experience
with this dock but did not receive any reply yet.

What you mean by 40Gbit mode? The dock exposes two lanes both at 20G so
it should always be 40G since we bind the lanes, also in Windows.

Also In Windows, do you see if the all USB devices on the dock are
enumerated as FullSpeed or SuperSpeed? I suspect it's the former there
too but can you check? Keyboard and mouse should be FullSpeed but there
is some audio device that may be USB 3.x (SuperSpeed), or alternatively
if you have USB 3.x memory stick (or any other device) you can plug that
to the dock and see how it enumerates.

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mika

On 23-10-2024 08:10, Mika Westerberg wrote:
> Hi,
> 
> On Tue, Oct 22, 2024 at 07:06:50PM +0200, Rick wrote:
>> Hi Mika,
>>
>> I have removed pcie_asm=force as kernel parameter but still not working on
>> latest non LTS kernel.
> 
> Okay, I still suggest not having that unless you absolutely know that
> you need it.
> 

Noted thank you!

>> In regards to the disconnect; sorry I think I might have turned of the
>> docking station myself during that test. I have taken another dmesg without
>> me disconnecting the docking station:
>> https://gist.github.com/ricklahaye/9798b7de573d0f29b3ada6a5d99b69f1
>>
>> The cable is the original Thunderbolt 4 cable that came with the docking
>> station. I have used it on this laptop using Windows (dualboot) without any
>> issues. Also on another Windows laptop also without issues. It was used in
>> 40Gbit mode.
> 
> In the dmesg you shared above, there are still unplug and USB tunnel
> creation fails so you only get USB 2.x connection with all the USB
> devices on the dock.
> 

Yes you are right. I removed all attached USB devices from the dock, but 
still see "3:3: USB3 tunnel activation failed, aborting"

> How do you determine if it "works"? I guess keyboard and mouse (both
> USB 2.x devices) and display (tunneled over USB4 link) all are working
> right? However, if you plug in USB 3.x device to the dock it enumerates
> as FullSpeed instead of SuperSpeed. There is definitely something wrong
> here. I asked from our TB validation folks if they have any experience
> with this dock but did not receive any reply yet.
> 
> What you mean by 40Gbit mode? The dock exposes two lanes both at 20G so
> it should always be 40G since we bind the lanes, also in Windows.

2 lanes of 20G indeed.

> 
> Also In Windows, do you see if the all USB devices on the dock are
> enumerated as FullSpeed or SuperSpeed? I suspect it's the former there
> too but can you check? Keyboard and mouse should be FullSpeed but there
> is some audio device that may be USB 3.x (SuperSpeed), or alternatively
> if you have USB 3.x memory stick (or any other device) you can plug that
> to the dock and see how it enumerates.

I checked on Windows with some 3.1 USB devices, and they were properly 
seen as 3.1 Superspeed+/10Gbps when attached to dock (using USBView from 
Windows SDK).

I also tried some Linux kernels, and it seems that 6.9 works, and 6.10 
doesn't.

6.9: https://gist.github.com/ricklahaye/da8c63edb0c27dc55bef351f9f4dd035
6.10: https://gist.github.com/ricklahaye/c2f314f74ecadcc4a2bd358d5d07e97b

Thank you for the help you have provided.

Kind regards,
Rick Lahaye

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

Hi,

On Fri, Oct 25, 2024 at 12:20:55PM +0200, Rick wrote:
> Hi Mika
> 
> On 23-10-2024 08:10, Mika Westerberg wrote:
> > Hi,
> > 
> > On Tue, Oct 22, 2024 at 07:06:50PM +0200, Rick wrote:
> > > Hi Mika,
> > > 
> > > I have removed pcie_asm=force as kernel parameter but still not working on
> > > latest non LTS kernel.
> > 
> > Okay, I still suggest not having that unless you absolutely know that
> > you need it.
> > 
> 
> Noted thank you!
> 
> > > In regards to the disconnect; sorry I think I might have turned of the
> > > docking station myself during that test. I have taken another dmesg without
> > > me disconnecting the docking station:
> > > https://gist.github.com/ricklahaye/9798b7de573d0f29b3ada6a5d99b69f1
> > > 
> > > The cable is the original Thunderbolt 4 cable that came with the docking
> > > station. I have used it on this laptop using Windows (dualboot) without any
> > > issues. Also on another Windows laptop also without issues. It was used in
> > > 40Gbit mode.
> > 
> > In the dmesg you shared above, there are still unplug and USB tunnel
> > creation fails so you only get USB 2.x connection with all the USB
> > devices on the dock.
> > 
> 
> Yes you are right. I removed all attached USB devices from the dock, but
> still see "3:3: USB3 tunnel activation failed, aborting"
> 
> > How do you determine if it "works"? I guess keyboard and mouse (both
> > USB 2.x devices) and display (tunneled over USB4 link) all are working
> > right? However, if you plug in USB 3.x device to the dock it enumerates
> > as FullSpeed instead of SuperSpeed. There is definitely something wrong
> > here. I asked from our TB validation folks if they have any experience
> > with this dock but did not receive any reply yet.
> > 
> > What you mean by 40Gbit mode? The dock exposes two lanes both at 20G so
> > it should always be 40G since we bind the lanes, also in Windows.
> 
> 2 lanes of 20G indeed.
> 
> > 
> > Also In Windows, do you see if the all USB devices on the dock are
> > enumerated as FullSpeed or SuperSpeed? I suspect it's the former there
> > too but can you check? Keyboard and mouse should be FullSpeed but there
> > is some audio device that may be USB 3.x (SuperSpeed), or alternatively
> > if you have USB 3.x memory stick (or any other device) you can plug that
> > to the dock and see how it enumerates.
> 
> I checked on Windows with some 3.1 USB devices, and they were properly seen
> as 3.1 Superspeed+/10Gbps when attached to dock (using USBView from Windows
> SDK).
> 
> I also tried some Linux kernels, and it seems that 6.9 works, and 6.10
> doesn't.
> 
> 6.9: https://gist.github.com/ricklahaye/da8c63edb0c27dc55bef351f9f4dd035

I still see similar issue even with the v6.9 kernel. The link goes up an
down unexpectly.

I wonder if you could try to take traces of the control channel traffic?
I suggest to use v6.11 kernel because it should have all the tracing
bits added then install tbtools [1] and follow the steps here:

  https://github.com/intel/tbtools?tab=readme-ov-file#tracing

Then provide both full dmesg and the trace output. That hopefully shows
if some of the access we are doing in the Linux side is causing the link
to to drop. Let me know if you need more detailed instructions.

Also please drop the "thunderbolt.host_reset=0" from the command line as
that did not help, so it is not needed.

[1] https://github.com/intel/tbtools

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mika,

Thank you for your email.

On 28-10-2024 09:18, Mika Westerberg wrote:
> 
> I still see similar issue even with the v6.9 kernel. The link goes up an
> down unexpectly.
> 
> I wonder if you could try to take traces of the control channel traffic?
> I suggest to use v6.11 kernel because it should have all the tracing
> bits added then install tbtools [1] and follow the steps here:
> 
>    https://github.com/intel/tbtools?tab=readme-ov-file#tracing
> 
> Then provide both full dmesg and the trace output. That hopefully shows
> if some of the access we are doing in the Linux side is causing the link
> to to drop. Let me know if you need more detailed instructions.
> 
> Also please drop the "thunderbolt.host_reset=0" from the command line as
> that did not help, so it is not needed.

Dropped thank you.

> 
> [1] https://github.com/intel/tbtools

tbtrace on 6.11.5: 
https://gist.github.com/ricklahaye/69776e9c39fd30a80e2adb6156bdb42d
dmesg on 6.11.5: 
https://gist.github.com/ricklahaye/8588450725695a0bd45799d3d66c7aff

Kind regards,
Rick Lahaye

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

Hi Rick,

On Wed, Oct 30, 2024 at 08:11:30AM +0100, Rick wrote:
> Hi Mika,
> 
> Thank you for your email.
> 
> On 28-10-2024 09:18, Mika Westerberg wrote:
> > 
> > I still see similar issue even with the v6.9 kernel. The link goes up an
> > down unexpectly.
> > 
> > I wonder if you could try to take traces of the control channel traffic?
> > I suggest to use v6.11 kernel because it should have all the tracing
> > bits added then install tbtools [1] and follow the steps here:
> > 
> >    https://github.com/intel/tbtools?tab=readme-ov-file#tracing
> > 
> > Then provide both full dmesg and the trace output. That hopefully shows
> > if some of the access we are doing in the Linux side is causing the link
> > to to drop. Let me know if you need more detailed instructions.
> > 
> > Also please drop the "thunderbolt.host_reset=0" from the command line as
> > that did not help, so it is not needed.
> 
> Dropped thank you.
> 
> > 
> > [1] https://github.com/intel/tbtools
> 
> tbtrace on 6.11.5:
> https://gist.github.com/ricklahaye/69776e9c39fd30a80e2adb6156bdb42d
> dmesg on 6.11.5:
> https://gist.github.com/ricklahaye/8588450725695a0bd45799d3d66c7aff

Thanks! I suspect there is something we do when we read the sideband
that makes the device router to "timeout" and retry the link
establishment. There is also the failure when USB 3.x tunnel is created
but we can look that after we figure out the connection issue.

Looking at the trace we are still polling for retimers when we see the
unplug:

[   48.684078] tb_tx Read Request Domain 0 Route 0 Adapter 3 / Lane
               0x00/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String High
               0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String Low
               0x02/---- 0x02182091 0b00000010 00011000 00100000 10010001 .... 
                 [00:12]       0x91 Address
                 [13:18]        0x1 Read Size
                 [19:24]        0x3 Adapter Num
                 [25:26]        0x1 Configuration Space (CS) → Adapter Configuration Space
                 [27:28]        0x0 Sequence Number (SN)
[   48.684339] tb_rx Read Response Domain 0 Route 0 Adapter 3 / Lane
               0x00/---- 0x80000000 0b10000000 00000000 00000000 00000000 .... Route String High
               0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String Low
               0x02/---- 0x02182091 0b00000010 00011000 00100000 10010001 .... 
                 [00:12]       0x91 Address
                 [13:18]        0x1 Read Size
                 [19:24]        0x3 Adapter Num
                 [25:26]        0x1 Configuration Space (CS) → Adapter Configuration Space
                 [27:28]        0x0 Sequence Number (SN)
               0x03/0091 0x81320408 0b10000001 00110010 00000100 00001000 .2.. PORT_CS_1
                 [00:07]        0x8 Address
                 [08:15]        0x4 Length
                 [16:18]        0x2 Target
                 [20:23]        0x3 Re-timer Index
                 [24:24]        0x1 WnR
                 [25:25]        0x0 No Response (NR)
                 [26:26]        0x0 Result Code (RC)
                 [31:31]        0x1 Pending (PND)
[   48.691410] tb_event Hot Plug Event Packet Domain 0 Route 0 Adapter 3 / Lane
               0x00/---- 0x80000000 0b10000000 00000000 00000000 00000000 .... Route String High
               0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000 .... Route String Low
               0x02/---- 0x80000003 0b10000000 00000000 00000000 00000011 .... 
                 [00:05]        0x3 Adapter Num
                 [31:31]        0x1 UPG
[   48.691414] thunderbolt 0000:00:0d.2: acking hot unplug event on 0:3

Taking this into account and also the fact that your previous email you
say that v6.9 works and v6.10 does not, I wonder if you could first try
just to revert:

  c6ca1ac9f472 ("thunderbolt: Increase sideband access polling delay")

and see if that helps with the connection issue? If it does then can you
take full dmesg and the trace again and with me so I can look at the USB
tunnel issue too?

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mika

On 30-10-2024 10:06, Mika Westerberg wrote:
>> tbtrace on 6.11.5:
>> https://gist.github.com/ricklahaye/69776e9c39fd30a80e2adb6156bdb42d
>> dmesg on 6.11.5:
>> https://gist.github.com/ricklahaye/8588450725695a0bd45799d3d66c7aff
> 
> Thanks! I suspect there is something we do when we read the sideband
> that makes the device router to "timeout" and retry the link
> establishment. There is also the failure when USB 3.x tunnel is created
> but we can look that after we figure out the connection issue.
> 
> Looking at the trace we are still polling for retimers when we see the
> unplug:
> 
> [   48.684078] tb_tx Read Request Domain 0 Route 0 Adapter 3 / Lane
>                 0x00/---- 0x00000000 0b00000000 00000000 00000000 00000000
> .... Route String High
>                 0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000
> .... Route String Low
>                 0x02/---- 0x02182091 0b00000010 00011000 00100000 10010001
> ....
>                   [00:12]       0x91 Address
>                   [13:18]        0x1 Read Size
>                   [19:24]        0x3 Adapter Num
>                   [25:26]        0x1 Configuration Space (CS) → Adapter
> Configuration Space
>                   [27:28]        0x0 Sequence Number (SN)
> [   48.684339] tb_rx Read Response Domain 0 Route 0 Adapter 3 / Lane
>                 0x00/---- 0x80000000 0b10000000 00000000 00000000 00000000
> .... Route String High
>                 0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000
> .... Route String Low
>                 0x02/---- 0x02182091 0b00000010 00011000 00100000 10010001
> ....
>                   [00:12]       0x91 Address
>                   [13:18]        0x1 Read Size
>                   [19:24]        0x3 Adapter Num
>                   [25:26]        0x1 Configuration Space (CS) → Adapter
> Configuration Space
>                   [27:28]        0x0 Sequence Number (SN)
>                 0x03/0091 0x81320408 0b10000001 00110010 00000100 00001000
> .2.. PORT_CS_1
>                   [00:07]        0x8 Address
>                   [08:15]        0x4 Length
>                   [16:18]        0x2 Target
>                   [20:23]        0x3 Re-timer Index
>                   [24:24]        0x1 WnR
>                   [25:25]        0x0 No Response (NR)
>                   [26:26]        0x0 Result Code (RC)
>                   [31:31]        0x1 Pending (PND)
> [   48.691410] tb_event Hot Plug Event Packet Domain 0 Route 0 Adapter 3 /
> Lane
>                 0x00/---- 0x80000000 0b10000000 00000000 00000000 00000000
> .... Route String High
>                 0x01/---- 0x00000000 0b00000000 00000000 00000000 00000000
> .... Route String Low
>                 0x02/---- 0x80000003 0b10000000 00000000 00000000 00000011
> ....
>                   [00:05]        0x3 Adapter Num
>                   [31:31]        0x1 UPG
> [   48.691414] thunderbolt 0000:00:0d.2: acking hot unplug event on 0:3
> 
> Taking this into account and also the fact that your previous email you
> say that v6.9 works and v6.10 does not, I wonder if you could first try
> just to revert:
> 
>    c6ca1ac9f472 ("thunderbolt: Increase sideband access polling delay")

I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty resulting in docking 
station not working.

Then I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty without commit 
c6ca1ac9f472 (reverted), and now the docking station works correctly (as 
in screen output + USBs + Ethernet)

So it seems c6ca1ac9f472 is causing issues for my setup.

> 
> and see if that helps with the connection issue? If it does then can you
> take full dmesg and the trace again and with me so I can look at the USB
> tunnel issue too?

Below files are generated with commit c6ca1ac9f472 reverted!

Tbtrace: https://gist.github.com/ricklahaye/05e54f12c974d3ed3e15527af7f67ed2
Dmesg: https://gist.github.com/ricklahaye/f50ad55159dec2b5265dd20bcebe4a10

Thank you,
Kind regards,
Rick Lahaye

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

Hi Rick,

On Fri, Nov 01, 2024 at 01:57:50PM +0100, Rick wrote:
> I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty resulting in docking station
> not working.
> 
> Then I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty without commit
> c6ca1ac9f472 (reverted), and now the docking station works correctly (as in
> screen output + USBs + Ethernet)
> 
> So it seems c6ca1ac9f472 is causing issues for my setup.

Okay, thanks for testing!

It indeed looks like there is no any kind of link issues anymore with
that one reverted. So my suspect is that we are taking too long before
we enumerate the device router which makes it to reset the link.

Can you try the below patch too on top of v6.12-rcX (without the revert)
and see if that still keeps it working? This one cuts down the delay to
1ms which I'm hoping is sufficient for the device. Can you share
dmesg+trace from that test as well?

diff --git a/drivers/thunderbolt/usb4.c b/drivers/thunderbolt/usb4.c
index c6dcc23e8c16..1b740d7fc7da 100644
--- a/drivers/thunderbolt/usb4.c
+++ b/drivers/thunderbolt/usb4.c
@@ -48,7 +48,7 @@ enum usb4_ba_index {
 
 /* Delays in us used with usb4_port_wait_for_bit() */
 #define USB4_PORT_DELAY			50
-#define USB4_PORT_SB_DELAY		5000
+#define USB4_PORT_SB_DELAY		1000
 
 static int usb4_native_switch_op(struct tb_switch *sw, u16 opcode,
 				 u32 *metadata, u8 *status,

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mika

On 04-11-2024 07:36, Mika Westerberg wrote:
> Hi Rick,
> 
> On Fri, Nov 01, 2024 at 01:57:50PM +0100, Rick wrote:
>> I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty resulting in docking station
>> not working.
>>
>> Then I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty without commit
>> c6ca1ac9f472 (reverted), and now the docking station works correctly (as in
>> screen output + USBs + Ethernet)
>>
>> So it seems c6ca1ac9f472 is causing issues for my setup.
> 
> Okay, thanks for testing!
> 
> It indeed looks like there is no any kind of link issues anymore with
> that one reverted. So my suspect is that we are taking too long before
> we enumerate the device router which makes it to reset the link.
> 
> Can you try the below patch too on top of v6.12-rcX (without the revert)
> and see if that still keeps it working? This one cuts down the delay to
> 1ms which I'm hoping is sufficient for the device. Can you share
> dmesg+trace from that test as well?
> 
> diff --git a/drivers/thunderbolt/usb4.c b/drivers/thunderbolt/usb4.c
> index c6dcc23e8c16..1b740d7fc7da 100644
> --- a/drivers/thunderbolt/usb4.c
> +++ b/drivers/thunderbolt/usb4.c
> @@ -48,7 +48,7 @@ enum usb4_ba_index {
>   
>   /* Delays in us used with usb4_port_wait_for_bit() */
>   #define USB4_PORT_DELAY			50
> -#define USB4_PORT_SB_DELAY		5000
> +#define USB4_PORT_SB_DELAY		1000
>   
>   static int usb4_native_switch_op(struct tb_switch *sw, u16 opcode,
>   				 u32 *metadata, u8 *status,

See below pasts without the revert, and with the above provided patch.

dmesg with patch (and without the revert): 
https://gist.github.com/ricklahaye/8412af228063546dd8375ca796fffeef
tbtrace with patch (and without the revert): 
https://gist.github.com/ricklahaye/4b9cbeeb36b546c6686ce79a044a2d61

Seems to be working correctly with the provided patch.
Thank you!

Kind regards,
Rick

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

Hi Rick,

On Mon, Nov 04, 2024 at 07:04:08PM +0100, Rick wrote:
> Hi Mika
> 
> On 04-11-2024 07:36, Mika Westerberg wrote:
> > Hi Rick,
> > 
> > On Fri, Nov 01, 2024 at 01:57:50PM +0100, Rick wrote:
> > > I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty resulting in docking station
> > > not working.
> > > 
> > > Then I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty without commit
> > > c6ca1ac9f472 (reverted), and now the docking station works correctly (as in
> > > screen output + USBs + Ethernet)
> > > 
> > > So it seems c6ca1ac9f472 is causing issues for my setup.
> > 
> > Okay, thanks for testing!
> > 
> > It indeed looks like there is no any kind of link issues anymore with
> > that one reverted. So my suspect is that we are taking too long before
> > we enumerate the device router which makes it to reset the link.
> > 
> > Can you try the below patch too on top of v6.12-rcX (without the revert)
> > and see if that still keeps it working? This one cuts down the delay to
> > 1ms which I'm hoping is sufficient for the device. Can you share
> > dmesg+trace from that test as well?
> > 
> > diff --git a/drivers/thunderbolt/usb4.c b/drivers/thunderbolt/usb4.c
> > index c6dcc23e8c16..1b740d7fc7da 100644
> > --- a/drivers/thunderbolt/usb4.c
> > +++ b/drivers/thunderbolt/usb4.c
> > @@ -48,7 +48,7 @@ enum usb4_ba_index {
> >   /* Delays in us used with usb4_port_wait_for_bit() */
> >   #define USB4_PORT_DELAY			50
> > -#define USB4_PORT_SB_DELAY		5000
> > +#define USB4_PORT_SB_DELAY		1000
> >   static int usb4_native_switch_op(struct tb_switch *sw, u16 opcode,
> >   				 u32 *metadata, u8 *status,
> 
> See below pasts without the revert, and with the above provided patch.
> 
> dmesg with patch (and without the revert):
> https://gist.github.com/ricklahaye/8412af228063546dd8375ca796fffeef
> tbtrace with patch (and without the revert):
> https://gist.github.com/ricklahaye/4b9cbeeb36b546c6686ce79a044a2d61
> 
> Seems to be working correctly with the provided patch.
> Thank you!

Thanks for testing! Yes, logs look good now.

I will submit this fix upstream today then. Do you mind providing me your
full name and email so that I can add tag like

Reported-by: Rick ...

in the commit message?

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Rick]

Hi Mika

On 05-11-2024 07:31, Mika Westerberg wrote:
> Hi Rick,
> 
> On Mon, Nov 04, 2024 at 07:04:08PM +0100, Rick wrote:
>> Hi Mika
>>
>> On 04-11-2024 07:36, Mika Westerberg wrote:
>>> Hi Rick,
>>>
>>> On Fri, Nov 01, 2024 at 01:57:50PM +0100, Rick wrote:
>>>> I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty resulting in docking station
>>>> not working.
>>>>
>>>> Then I compiled 6.12.0-rc5-00181-g6c52d4da1c74-dirty without commit
>>>> c6ca1ac9f472 (reverted), and now the docking station works correctly (as in
>>>> screen output + USBs + Ethernet)
>>>>
>>>> So it seems c6ca1ac9f472 is causing issues for my setup.
>>>
>>> Okay, thanks for testing!
>>>
>>> It indeed looks like there is no any kind of link issues anymore with
>>> that one reverted. So my suspect is that we are taking too long before
>>> we enumerate the device router which makes it to reset the link.
>>>
>>> Can you try the below patch too on top of v6.12-rcX (without the revert)
>>> and see if that still keeps it working? This one cuts down the delay to
>>> 1ms which I'm hoping is sufficient for the device. Can you share
>>> dmesg+trace from that test as well?
>>>
>>> diff --git a/drivers/thunderbolt/usb4.c b/drivers/thunderbolt/usb4.c
>>> index c6dcc23e8c16..1b740d7fc7da 100644
>>> --- a/drivers/thunderbolt/usb4.c
>>> +++ b/drivers/thunderbolt/usb4.c
>>> @@ -48,7 +48,7 @@ enum usb4_ba_index {
>>>    /* Delays in us used with usb4_port_wait_for_bit() */
>>>    #define USB4_PORT_DELAY			50
>>> -#define USB4_PORT_SB_DELAY		5000
>>> +#define USB4_PORT_SB_DELAY		1000
>>>    static int usb4_native_switch_op(struct tb_switch *sw, u16 opcode,
>>>    				 u32 *metadata, u8 *status,
>>
>> See below pasts without the revert, and with the above provided patch.
>>
>> dmesg with patch (and without the revert):
>> https://gist.github.com/ricklahaye/8412af228063546dd8375ca796fffeef
>> tbtrace with patch (and without the revert):
>> https://gist.github.com/ricklahaye/4b9cbeeb36b546c6686ce79a044a2d61
>>
>> Seems to be working correctly with the provided patch.
>> Thank you!
> 
> Thanks for testing! Yes, logs look good now.
> 
> I will submit this fix upstream today then. Do you mind providing me your
> full name and email so that I can add tag like
> 
> Reported-by: Rick ...
> 
> in the commit message?

Thank you that's nice.
You can note Rick Lahaye - rick@581238.xyz

Thank you for the help.

Kind regards,
Rick Lahaye

Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Fabian Stäber]

Hi,

I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
Linux for years without issues. However, updating to
linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
latest non-LTS kernel doesn't help, it also breaks the USB ports.

Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
working version.

I opened a thread on the Arch Linux forum
https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
output. However, it sounds like this is a regression in the Linux
kernel, so I'm posting this here as well.

Let me know if you need any more info.

Thanks a lot

Fabian

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Greg KH]

On Mon, Sep 23, 2024 at 08:34:23AM +0200, Fabian Stäber wrote:
> Hi,

Adding the linux-usb list.

> I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
> Linux for years without issues. However, updating to
> linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
> latest non-LTS kernel doesn't help, it also breaks the USB ports.
> 
> Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
> working version.
> 
> I opened a thread on the Arch Linux forum
> https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
> output. However, it sounds like this is a regression in the Linux
> kernel, so I'm posting this here as well.
> 
> Let me know if you need any more info.

Is there any way you can use 'git bisect' to test inbetween kernel
versions/commits to find the offending change?

Does the non-lts arch kernel work properly?

thanks,

greg k-h

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Fabian Stäber]

Hi Greg,

I can reproduce the issue with the upstream Linux kernel: I compiled
6.6.28 and 6.6.29 from source: 6.6.28 works, 6.6.29 doesn't.

I'll learn how to do 'git bisect' to narrow it down to the offending commit.

The non-lts kernel is also broken.

Fabian

On Mon, Sep 23, 2024 at 8:45 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Mon, Sep 23, 2024 at 08:34:23AM +0200, Fabian Stäber wrote:
> > Hi,
>
> Adding the linux-usb list.
>
> > I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
> > Linux for years without issues. However, updating to
> > linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
> > latest non-LTS kernel doesn't help, it also breaks the USB ports.
> >
> > Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
> > working version.
> >
> > I opened a thread on the Arch Linux forum
> > https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
> > output. However, it sounds like this is a regression in the Linux
> > kernel, so I'm posting this here as well.
> >
> > Let me know if you need any more info.
>
> Is there any way you can use 'git bisect' to test inbetween kernel
> versions/commits to find the offending change?
>
> Does the non-lts arch kernel work properly?
>
> thanks,
>
> greg k-h

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Fabian Stäber]

Hi,

sorry for the delay, I ran git bisect, here's the output. If you need
any additional info please let me know.

3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 is the first bad commit
commit 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 (HEAD)
Author: Sanath S <Sanath.S@amd.com>
Date:   Sat Jan 13 10:52:48 2024

    thunderbolt: Reset topology created by the boot firmware

    commit 59a54c5f3dbde00b8ad30aef27fe35b1fe07bf5c upstream.

    Boot firmware (typically BIOS) might have created tunnels of its own.
    The tunnel configuration that it does might be sub-optimal. For instance
    it may only support HBR2 monitors so the DisplayPort tunnels it created
    may limit Linux graphics drivers. In addition there is an issue on some
    AMD based systems where the BIOS does not allocate enough PCIe resources
    for future topology extension. By resetting the USB4 topology the PCIe
    links will be reset as well allowing Linux to re-allocate.

    This aligns the behavior with Windows Connection Manager.

    We already issued host router reset for USB4 v2 routers, now extend it
    to USB4 v1 routers as well. For pre-USB4 (that's Apple systems) we leave
    it as is and continue to discover the existing tunnels.

    Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
    Signed-off-by: Sanath S <Sanath.S@amd.com>
    Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

 drivers/thunderbolt/domain.c |  5 +++--
 drivers/thunderbolt/icm.c    |  2 +-
 drivers/thunderbolt/nhi.c    | 19 +++++++++++++------
 drivers/thunderbolt/tb.c     | 26 +++++++++++++++++++-------
 drivers/thunderbolt/tb.h     |  4 ++--
 5 files changed, 38 insertions(+), 18 deletions(-)

On Tue, Sep 24, 2024 at 8:58 AM Fabian Stäber <fabian@fstab.de> wrote:
>
> Hi Greg,
>
> I can reproduce the issue with the upstream Linux kernel: I compiled
> 6.6.28 and 6.6.29 from source: 6.6.28 works, 6.6.29 doesn't.
>
> I'll learn how to do 'git bisect' to narrow it down to the offending commit.
>
> The non-lts kernel is also broken.
>
> Fabian
>
> On Mon, Sep 23, 2024 at 8:45 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Mon, Sep 23, 2024 at 08:34:23AM +0200, Fabian Stäber wrote:
> > > Hi,
> >
> > Adding the linux-usb list.
> >
> > > I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
> > > Linux for years without issues. However, updating to
> > > linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
> > > latest non-LTS kernel doesn't help, it also breaks the USB ports.
> > >
> > > Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
> > > working version.
> > >
> > > I opened a thread on the Arch Linux forum
> > > https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
> > > output. However, it sounds like this is a regression in the Linux
> > > kernel, so I'm posting this here as well.
> > >
> > > Let me know if you need any more info.
> >
> > Is there any way you can use 'git bisect' to test inbetween kernel
> > versions/commits to find the offending change?
> >
> > Does the non-lts arch kernel work properly?
> >
> > thanks,
> >
> > greg k-h

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Linux regression tracking (Thorsten Leemhuis)]

On 07.10.24 18:49, Fabian Stäber wrote:
> 
> sorry for the delay, I ran git bisect, here's the output. If you need
> any additional info please let me know.

Many thx, I CCed those that authored and handled the change. But there
is one thing that is not really clear to me.

Earlier you wrote: "The non-lts kernel is also broken." -- which version
exactly did you mean here? But whatever version it was: if you haven't
tried 6.12-rc1 or 6.12-rc2, it would be best if you could check and
report back if that's affected as well.

Ciao, Thorsten

> 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 is the first bad commit
> commit 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 (HEAD)
> Author: Sanath S <Sanath.S@amd.com>
> Date:   Sat Jan 13 10:52:48 2024
> 
>     thunderbolt: Reset topology created by the boot firmware
> 
>     commit 59a54c5f3dbde00b8ad30aef27fe35b1fe07bf5c upstream.
> 
>     Boot firmware (typically BIOS) might have created tunnels of its own.
>     The tunnel configuration that it does might be sub-optimal. For instance
>     it may only support HBR2 monitors so the DisplayPort tunnels it created
>     may limit Linux graphics drivers. In addition there is an issue on some
>     AMD based systems where the BIOS does not allocate enough PCIe resources
>     for future topology extension. By resetting the USB4 topology the PCIe
>     links will be reset as well allowing Linux to re-allocate.
> 
>     This aligns the behavior with Windows Connection Manager.
> 
>     We already issued host router reset for USB4 v2 routers, now extend it
>     to USB4 v1 routers as well. For pre-USB4 (that's Apple systems) we leave
>     it as is and continue to discover the existing tunnels.
> 
>     Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
>     Signed-off-by: Sanath S <Sanath.S@amd.com>
>     Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
>  drivers/thunderbolt/domain.c |  5 +++--
>  drivers/thunderbolt/icm.c    |  2 +-
>  drivers/thunderbolt/nhi.c    | 19 +++++++++++++------
>  drivers/thunderbolt/tb.c     | 26 +++++++++++++++++++-------
>  drivers/thunderbolt/tb.h     |  4 ++--
>  5 files changed, 38 insertions(+), 18 deletions(-)
> 
> On Tue, Sep 24, 2024 at 8:58 AM Fabian Stäber <fabian@fstab.de> wrote:
>>
>> Hi Greg,
>>
>> I can reproduce the issue with the upstream Linux kernel: I compiled
>> 6.6.28 and 6.6.29 from source: 6.6.28 works, 6.6.29 doesn't.
>>
>> I'll learn how to do 'git bisect' to narrow it down to the offending commit.
>>
>> The non-lts kernel is also broken.
>>
>> Fabian
>>
>> On Mon, Sep 23, 2024 at 8:45 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>>>
>>> On Mon, Sep 23, 2024 at 08:34:23AM +0200, Fabian Stäber wrote:
>>>> Hi,
>>>
>>> Adding the linux-usb list.
>>>
>>>> I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
>>>> Linux for years without issues. However, updating to
>>>> linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
>>>> latest non-LTS kernel doesn't help, it also breaks the USB ports.
>>>>
>>>> Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
>>>> working version.
>>>>
>>>> I opened a thread on the Arch Linux forum
>>>> https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
>>>> output. However, it sounds like this is a regression in the Linux
>>>> kernel, so I'm posting this here as well.
>>>>
>>>> Let me know if you need any more info.
>>>
>>> Is there any way you can use 'git bisect' to test inbetween kernel
>>> versions/commits to find the offending change?
>>>
>>> Does the non-lts arch kernel work properly?
>>>
>>> thanks,
>>>
>>> greg k-h

#regzbot introduced: 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Christian Heusel]

[-- Attachment #1: Type: text/plain, Size: 3487 bytes --]

On 24/10/07 06:49PM, Fabian Stäber wrote:
> Hi,

Hey Fabian,

> sorry for the delay, I ran git bisect, here's the output. If you need
> any additional info please let me know.
> 
> 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 is the first bad commit
> commit 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 (HEAD)
> Author: Sanath S <Sanath.S@amd.com>
> Date:   Sat Jan 13 10:52:48 2024
> 
>     thunderbolt: Reset topology created by the boot firmware
> 
>     commit 59a54c5f3dbde00b8ad30aef27fe35b1fe07bf5c upstream.

So there is a commit c67f926ec870 ("thunderbolt: Reset only non-USB4
host routers in resume") that carries a fixes tag for the commit that
you have bisected to. The commits should both be in v6.6.29 and onwards,
so in the same release that's causing you problems. Maybe the fix is
incomplete or has a missing dependency 🤔

>     [...]
>     Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
>     Signed-off-by: Sanath S <Sanath.S@amd.com>
>     Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

I have added Mika, Mario and Sanath to the recipients, maybe they have
inputs on what would be useful debugging output.

In the meantime maybe also test if the issue is present with the latest
stable kernel ("linux" in the Arch packages) and with the latest release
candidate (you can find a precompiled version [here][0].

Cheers,
Chris

[0]: https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.12rc2-1-x86_64.pkg.tar.zst

> 
>  drivers/thunderbolt/domain.c |  5 +++--
>  drivers/thunderbolt/icm.c    |  2 +-
>  drivers/thunderbolt/nhi.c    | 19 +++++++++++++------
>  drivers/thunderbolt/tb.c     | 26 +++++++++++++++++++-------
>  drivers/thunderbolt/tb.h     |  4 ++--
>  5 files changed, 38 insertions(+), 18 deletions(-)
> 
> On Tue, Sep 24, 2024 at 8:58 AM Fabian Stäber <fabian@fstab.de> wrote:
> >
> > Hi Greg,
> >
> > I can reproduce the issue with the upstream Linux kernel: I compiled
> > 6.6.28 and 6.6.29 from source: 6.6.28 works, 6.6.29 doesn't.
> >
> > I'll learn how to do 'git bisect' to narrow it down to the offending commit.
> >
> > The non-lts kernel is also broken.
> >
> > Fabian
> >
> > On Mon, Sep 23, 2024 at 8:45 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Mon, Sep 23, 2024 at 08:34:23AM +0200, Fabian Stäber wrote:
> > > > Hi,
> > >
> > > Adding the linux-usb list.
> > >
> > > > I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
> > > > Linux for years without issues. However, updating to
> > > > linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
> > > > latest non-LTS kernel doesn't help, it also breaks the USB ports.
> > > >
> > > > Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
> > > > working version.
> > > >
> > > > I opened a thread on the Arch Linux forum
> > > > https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
> > > > output. However, it sounds like this is a regression in the Linux
> > > > kernel, so I'm posting this here as well.
> > > >
> > > > Let me know if you need any more info.
> > >
> > > Is there any way you can use 'git bisect' to test inbetween kernel
> > > versions/commits to find the offending change?
> > >
> > > Does the non-lts arch kernel work properly?
> > >
> > > thanks,
> > >
> > > greg k-h
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mario Limonciello]

On 10/7/2024 12:21, Christian Heusel wrote:
> On 24/10/07 06:49PM, Fabian Stäber wrote:
>> Hi,
> 
> Hey Fabian,
> 
>> sorry for the delay, I ran git bisect, here's the output. If you need
>> any additional info please let me know.
>>
>> 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 is the first bad commit
>> commit 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 (HEAD)
>> Author: Sanath S <Sanath.S@amd.com>
>> Date:   Sat Jan 13 10:52:48 2024
>>
>>      thunderbolt: Reset topology created by the boot firmware
>>
>>      commit 59a54c5f3dbde00b8ad30aef27fe35b1fe07bf5c upstream.
> 
> So there is a commit c67f926ec870 ("thunderbolt: Reset only non-USB4
> host routers in resume") that carries a fixes tag for the commit that
> you have bisected to. The commits should both be in v6.6.29 and onwards,
> so in the same release that's causing you problems. Maybe the fix is
> incomplete or has a missing dependency 🤔

You mean mainline commit 8cf9926c537c ("thunderbolt: Reset only non-USB4 
host routers in resume").

> 
>>      [...]
>>      Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
>>      Signed-off-by: Sanath S <Sanath.S@amd.com>
>>      Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
>>      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
> I have added Mika, Mario and Sanath to the recipients, maybe they have
> inputs on what would be useful debugging output.
> 
> In the meantime maybe also test if the issue is present with the latest
> stable kernel ("linux" in the Arch packages) and with the latest release
> candidate (you can find a precompiled version [here][0].

To double confirm, does thunderbolt.host_reset=0 on the kernel command 
line help your issue?  Based on the bisect I would expect it should 
help.  Yes; comments on both 6.6.y as well as 6.12-rc2 would be ideal.

Also assuming it helps can you please post your dmesg from 6.12-rc2 both 
with thunderbolt.host_reset=0 and without?  A github gist or a new 
kernel bugzilla are good places to post it.

> 
> Cheers,
> Chris
> 
> [0]: https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.12rc2-1-x86_64.pkg.tar.zst
> 
>>
>>   drivers/thunderbolt/domain.c |  5 +++--
>>   drivers/thunderbolt/icm.c    |  2 +-
>>   drivers/thunderbolt/nhi.c    | 19 +++++++++++++------
>>   drivers/thunderbolt/tb.c     | 26 +++++++++++++++++++-------
>>   drivers/thunderbolt/tb.h     |  4 ++--
>>   5 files changed, 38 insertions(+), 18 deletions(-)
>>
>> On Tue, Sep 24, 2024 at 8:58 AM Fabian Stäber <fabian@fstab.de> wrote:
>>>
>>> Hi Greg,
>>>
>>> I can reproduce the issue with the upstream Linux kernel: I compiled
>>> 6.6.28 and 6.6.29 from source: 6.6.28 works, 6.6.29 doesn't.
>>>
>>> I'll learn how to do 'git bisect' to narrow it down to the offending commit.
>>>
>>> The non-lts kernel is also broken.
>>>
>>> Fabian
>>>
>>> On Mon, Sep 23, 2024 at 8:45 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>>>>
>>>> On Mon, Sep 23, 2024 at 08:34:23AM +0200, Fabian Stäber wrote:
>>>>> Hi,
>>>>
>>>> Adding the linux-usb list.
>>>>
>>>>> I got a Dell WD19TBS Thunderbolt Dock, and it has been working with
>>>>> Linux for years without issues. However, updating to
>>>>> linux-lts-6.6.29-1 or newer breaks the USB ports on my Dock. Using the
>>>>> latest non-LTS kernel doesn't help, it also breaks the USB ports.
>>>>>
>>>>> Downgrading the kernel to linux-lts-6.6.28-1 works. This is the last
>>>>> working version.
>>>>>
>>>>> I opened a thread on the Arch Linux forum
>>>>> https://bbs.archlinux.org/viewtopic.php?id=299604 with some dmesg
>>>>> output. However, it sounds like this is a regression in the Linux
>>>>> kernel, so I'm posting this here as well.
>>>>>
>>>>> Let me know if you need any more info.
>>>>
>>>> Is there any way you can use 'git bisect' to test inbetween kernel
>>>> versions/commits to find the offending change?
>>>>
>>>> Does the non-lts arch kernel work properly?
>>>>
>>>> thanks,
>>>>
>>>> greg k-h
>>

Re: Dell WD19TB Thunderbolt Dock not working with kernel > 6.6.28-1

[Mika Westerberg]

On Mon, Oct 07, 2024 at 12:33:54PM -0500, Mario Limonciello wrote:
> On 10/7/2024 12:21, Christian Heusel wrote:
> > On 24/10/07 06:49PM, Fabian Stäber wrote:
> > > Hi,
> > 
> > Hey Fabian,
> > 
> > > sorry for the delay, I ran git bisect, here's the output. If you need
> > > any additional info please let me know.
> > > 
> > > 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 is the first bad commit
> > > commit 3c1d704d9266741fc5a9a0a287a5c6b72ddbea55 (HEAD)
> > > Author: Sanath S <Sanath.S@amd.com>
> > > Date:   Sat Jan 13 10:52:48 2024
> > > 
> > >      thunderbolt: Reset topology created by the boot firmware
> > > 
> > >      commit 59a54c5f3dbde00b8ad30aef27fe35b1fe07bf5c upstream.
> > 
> > So there is a commit c67f926ec870 ("thunderbolt: Reset only non-USB4
> > host routers in resume") that carries a fixes tag for the commit that
> > you have bisected to. The commits should both be in v6.6.29 and onwards,
> > so in the same release that's causing you problems. Maybe the fix is
> > incomplete or has a missing dependency 🤔
> 
> You mean mainline commit 8cf9926c537c ("thunderbolt: Reset only non-USB4
> host routers in resume").
> 
> > 
> > >      [...]
> > >      Suggested-by: Mario Limonciello <mario.limonciello@amd.com>
> > >      Signed-off-by: Sanath S <Sanath.S@amd.com>
> > >      Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
> > >      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > 
> > I have added Mika, Mario and Sanath to the recipients, maybe they have
> > inputs on what would be useful debugging output.
> > 
> > In the meantime maybe also test if the issue is present with the latest
> > stable kernel ("linux" in the Arch packages) and with the latest release
> > candidate (you can find a precompiled version [here][0].
> 
> To double confirm, does thunderbolt.host_reset=0 on the kernel command line
> help your issue?  Based on the bisect I would expect it should help.  Yes;
> comments on both 6.6.y as well as 6.12-rc2 would be ideal.
> 
> Also assuming it helps can you please post your dmesg from 6.12-rc2 both
> with thunderbolt.host_reset=0 and without?  A github gist or a new kernel
> bugzilla are good places to post it.

Also to understand the flow, you are booting with the dock connected
right?

Can you see also what:

  $ boltctl

outputs (after you have booted up, and the problem is reproduced)? It
should list the dock and show it as "authorized" but I'm not familiar
with Arch Linux so it could be that they are not using bolt and that
explains why things do not appear working as nobody is going to
re-create that PCIe tunnel that was torn down during host router reset.