Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Chris Murphy]

crossposting linux-usb@ and linux-pci@

I filed a bug https://bugzilla.kernel.org/show_bug.cgi?id=191681 but
was told I need to post to the list first. I'm not actually sure this
is a USB bug, as filed in the bug report, because I think this
USB-C/Thunderbolt port is really a PCIe port that needs to become a
USB port only once the adapter is connected.

Anyway, the gist of the problem is that if the USB-C to USB 3 adapter
is connected at boot time, the kernel doesn't instantiate a USB port
at all. So if for example I have a USB flash drive connected to the
adapter with a Linux live OS on it, the firmware sees the stick, finds
the bootloader, the bootloader finds and loads the kernel and
initramfs - so clearly up to this point it's working properly. But
then I end up at a dracut shell because there's not root fs to be
found. And there's no rootfs because there's no USB bus at all as far
as the kernel is concerned.

If the adapter is not connected at boot time, is connected later on,
it all works as I'd expect it to work. I can plug it in, unplug it,
plug it back in, and it always works fine. So weirdly it's just a "if
connected at boot time" something is becoming deeply confused.

I've tested 4.10-rc1 from kernel.org; but it's been a problem since
kernel 4.7.7 at least. The bug report has lspci, dmidecode, and two
dmesg outputs from boots with the adapter connected at boot time, and
not at boot time (but connected later). That bug report is 4.10-rc1
based with CONFIG_PCIEASPM_DEBUG=y
CONFIG_PCI_DEBUG=y but I'm not really seeing much additional
information that helps figure out what the source of the problem is.


Thanks,


-- 
Chris Murphy

Re: Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Alan Stern]

On Mon, 2 Jan 2017, Chris Murphy wrote:

> crossposting linux-usb@ and linux-pci@
> 
> I filed a bug https://bugzilla.kernel.org/show_bug.cgi?id=191681 but
> was told I need to post to the list first. I'm not actually sure this
> is a USB bug, as filed in the bug report, because I think this
> USB-C/Thunderbolt port is really a PCIe port that needs to become a
> USB port only once the adapter is connected.
> 
> Anyway, the gist of the problem is that if the USB-C to USB 3 adapter
> is connected at boot time, the kernel doesn't instantiate a USB port
> at all. So if for example I have a USB flash drive connected to the
> adapter with a Linux live OS on it, the firmware sees the stick, finds
> the bootloader, the bootloader finds and loads the kernel and
> initramfs - so clearly up to this point it's working properly. But
> then I end up at a dracut shell because there's not root fs to be
> found. And there's no rootfs because there's no USB bus at all as far
> as the kernel is concerned.
> 
> If the adapter is not connected at boot time, is connected later on,
> it all works as I'd expect it to work. I can plug it in, unplug it,
> plug it back in, and it always works fine. So weirdly it's just a "if
> connected at boot time" something is becoming deeply confused.
> 
> I've tested 4.10-rc1 from kernel.org; but it's been a problem since
> kernel 4.7.7 at least. The bug report has lspci, dmidecode, and two
> dmesg outputs from boots with the adapter connected at boot time, and
> not at boot time (but connected later). That bug report is 4.10-rc1
> based with CONFIG_PCIEASPM_DEBUG=y
> CONFIG_PCI_DEBUG=y but I'm not really seeing much additional
> information that helps figure out what the source of the problem is.

Considering that the failed boot log contains no USB messages at all, 
and no messages referring to PCI bus 0000:37 (the bus associated with 
the adapter), this certainly looks like a PCI problem.

Alan Stern

Re: Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Chris Murphy]

On Mon, Jan 2, 2017 at 2:28 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Mon, 2 Jan 2017, Chris Murphy wrote:
>
>> crossposting linux-usb@ and linux-pci@
>>
>> I filed a bug https://bugzilla.kernel.org/show_bug.cgi?id=191681 but
>> was told I need to post to the list first. I'm not actually sure this
>> is a USB bug, as filed in the bug report, because I think this
>> USB-C/Thunderbolt port is really a PCIe port that needs to become a
>> USB port only once the adapter is connected.
>>
>> Anyway, the gist of the problem is that if the USB-C to USB 3 adapter
>> is connected at boot time, the kernel doesn't instantiate a USB port
>> at all. So if for example I have a USB flash drive connected to the
>> adapter with a Linux live OS on it, the firmware sees the stick, finds
>> the bootloader, the bootloader finds and loads the kernel and
>> initramfs - so clearly up to this point it's working properly. But
>> then I end up at a dracut shell because there's not root fs to be
>> found. And there's no rootfs because there's no USB bus at all as far
>> as the kernel is concerned.
>>
>> If the adapter is not connected at boot time, is connected later on,
>> it all works as I'd expect it to work. I can plug it in, unplug it,
>> plug it back in, and it always works fine. So weirdly it's just a "if
>> connected at boot time" something is becoming deeply confused.
>>
>> I've tested 4.10-rc1 from kernel.org; but it's been a problem since
>> kernel 4.7.7 at least. The bug report has lspci, dmidecode, and two
>> dmesg outputs from boots with the adapter connected at boot time, and
>> not at boot time (but connected later). That bug report is 4.10-rc1
>> based with CONFIG_PCIEASPM_DEBUG=y
>> CONFIG_PCI_DEBUG=y but I'm not really seeing much additional
>> information that helps figure out what the source of the problem is.
>
> Considering that the failed boot log contains no USB messages at all,
> and no messages referring to PCI bus 0000:37 (the bus associated with
> the adapter), this certainly looks like a PCI problem.

I see these lines in the problem case, which don't occur in the working case.

[    0.561046] pci_bus 0000:37: busn_res: [bus 37-ff] end is updated to 37
[    0.561047] pci_bus 0000:37: busn_res: can not insert [bus 37]
under domain [bus 00-ff] (conflicts with (null) [bus 00-fe])

[    0.579446] pci 0000:37:00.0: can't claim BAR 0 [mem
0xb0000000-0xb000ffff]: address conflict with PCI Bus 0000:00 [mem
0x40000000-0xdfffffff window]

[    0.602277] pci 0000:37:00.0: can't derive routing for PCI INT A
[    0.602280] pci 0000:37:00.0: PCI INT A: not connected
[    0.612304] pci 0000:37:00.0: xHCI HW not ready after 5 sec (HC
bug?) status = 0xffffffff




-- 
Chris Murphy

Re: Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Oliver Neukum]

On Mon, 2017-01-02 at 14:45 -0700, Chris Murphy wrote:
> I see these lines in the problem case, which don't occur in the
> working case.
> 
> [    0.561046] pci_bus 0000:37: busn_res: [bus 37-ff] end is updated
> to 37
> [    0.561047] pci_bus 0000:37: busn_res: can not insert [bus 37]
> under domain [bus 00-ff] (conflicts with (null) [bus 00-fe])
> 
> [    0.579446] pci 0000:37:00.0: can't claim BAR 0 [mem
> 0xb0000000-0xb000ffff]: address conflict with PCI Bus 0000:00 [mem
> 0x40000000-0xdfffffff window]
> 
> [    0.602277] pci 0000:37:00.0: can't derive routing for PCI INT A
> [    0.602280] pci 0000:37:00.0: PCI INT A: not connected
> [    0.612304] pci 0000:37:00.0: xHCI HW not ready after 5 sec (HC
> bug?) status = 0xffffffff

Have you tried "pci=realloc"?

	Regards
		Oliver

Re: Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Chris Murphy]

On Tue, Jan 3, 2017 at 2:28 AM, Oliver Neukum <oneukum@suse.com> wrote:
> On Mon, 2017-01-02 at 14:45 -0700, Chris Murphy wrote:
>> I see these lines in the problem case, which don't occur in the
>> working case.
>>
>> [    0.561046] pci_bus 0000:37: busn_res: [bus 37-ff] end is updated
>> to 37
>> [    0.561047] pci_bus 0000:37: busn_res: can not insert [bus 37]
>> under domain [bus 00-ff] (conflicts with (null) [bus 00-fe])
>>
>> [    0.579446] pci 0000:37:00.0: can't claim BAR 0 [mem
>> 0xb0000000-0xb000ffff]: address conflict with PCI Bus 0000:00 [mem
>> 0x40000000-0xdfffffff window]
>>
>> [    0.602277] pci 0000:37:00.0: can't derive routing for PCI INT A
>> [    0.602280] pci 0000:37:00.0: PCI INT A: not connected
>> [    0.612304] pci 0000:37:00.0: xHCI HW not ready after 5 sec (HC
>> bug?) status = 0xffffffff
>
> Have you tried "pci=realloc"?

No effect. Attached that dmesg to the bug report anyway though.

-- 
Chris Murphy

Re: Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Lukas Wunner]

On Mon, Jan 02, 2017 at 02:45:00PM -0700, Chris Murphy wrote:
> On Mon, Jan 2, 2017 at 2:28 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
> > Considering that the failed boot log contains no USB messages at all,
> > and no messages referring to PCI bus 0000:37 (the bus associated with
> > the adapter), this certainly looks like a PCI problem.
> 
> I see these lines in the problem case, which don't occur in the working case.
> 
> [    0.561046] pci_bus 0000:37: busn_res: [bus 37-ff] end is updated to 37
> [    0.561047] pci_bus 0000:37: busn_res: can not insert [bus 37]
> under domain [bus 00-ff] (conflicts with (null) [bus 00-fe])

In the "failure dmesg" (with the adapter plugged in on boot), bus 37 is
considered a root bus:

[    0.560662] PCI: Discovered peer bus 37
[    0.560667] PCI: root bus 37: using default resources
[    0.560668] PCI: Probing PCI hardware (bus 37)
[    0.560695] ACPI: \: failed to evaluate _DSM (0x1001)
[    0.560698] PCI host bridge to bus 0000:37

Whereas in the "normal behaviour" dmesg (with the adapter plugged in
after booting Linux), bus 37 is below downstream port 0000:02:02.0
(this belongs to the PCIe switch of the Thunderbolt controller):

[   59.247680] pci 0000:02:02.0: scanning [bus 37-37] behind bridge, pass 0
[   59.247915] pci_bus 0000:37: scanning bus
[   59.247963] pci 0000:37:00.0: [8086:15b6] type 00 class 0x0c0330
[   59.248014] pci 0000:37:00.0: reg 0x10: [mem 0xc7f00000-0xc7f0ffff]
[   59.248360] pci 0000:37:00.0: supports D1 D2
[   59.248366] pci 0000:37:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   59.248379] pci 0000:37:00.0: PME# disabled
[   59.248951] pci_bus 0000:37: fixups for bus
[   59.248959] pci 0000:02:02.0: PCI bridge to [bus 37]
[   59.249117] pci 0000:02:02.0:   bridge window [mem 0xc7f00000-0xc7ffffff]
[   59.249135] pci_bus 0000:37: bus scan returning with max=37

Could be an error in the ACPI tables, could be a misconfiguration in
the EEPROM of the Thunderbolt controller.

Thanks,

Lukas

Re: Thunderbolt adapter fails to instantiate USB and device enumeration if already connected at boot time

[Chris Murphy]

On Tue, Jan 3, 2017 at 3:23 AM, Lukas Wunner <lukas@wunner.de> wrote:
> On Mon, Jan 02, 2017 at 02:45:00PM -0700, Chris Murphy wrote:
>> On Mon, Jan 2, 2017 at 2:28 PM, Alan Stern <stern@rowland.harvard.edu> wrote:
>> > Considering that the failed boot log contains no USB messages at all,
>> > and no messages referring to PCI bus 0000:37 (the bus associated with
>> > the adapter), this certainly looks like a PCI problem.
>>
>> I see these lines in the problem case, which don't occur in the working case.
>>
>> [    0.561046] pci_bus 0000:37: busn_res: [bus 37-ff] end is updated to 37
>> [    0.561047] pci_bus 0000:37: busn_res: can not insert [bus 37]
>> under domain [bus 00-ff] (conflicts with (null) [bus 00-fe])
>
> In the "failure dmesg" (with the adapter plugged in on boot), bus 37 is
> considered a root bus:
>
> [    0.560662] PCI: Discovered peer bus 37
> [    0.560667] PCI: root bus 37: using default resources
> [    0.560668] PCI: Probing PCI hardware (bus 37)
> [    0.560695] ACPI: \: failed to evaluate _DSM (0x1001)
> [    0.560698] PCI host bridge to bus 0000:37
>
> Whereas in the "normal behaviour" dmesg (with the adapter plugged in
> after booting Linux), bus 37 is below downstream port 0000:02:02.0
> (this belongs to the PCIe switch of the Thunderbolt controller):
>
> [   59.247680] pci 0000:02:02.0: scanning [bus 37-37] behind bridge, pass 0
> [   59.247915] pci_bus 0000:37: scanning bus
> [   59.247963] pci 0000:37:00.0: [8086:15b6] type 00 class 0x0c0330
> [   59.248014] pci 0000:37:00.0: reg 0x10: [mem 0xc7f00000-0xc7f0ffff]
> [   59.248360] pci 0000:37:00.0: supports D1 D2
> [   59.248366] pci 0000:37:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [   59.248379] pci 0000:37:00.0: PME# disabled
> [   59.248951] pci_bus 0000:37: fixups for bus
> [   59.248959] pci 0000:02:02.0: PCI bridge to [bus 37]
> [   59.249117] pci 0000:02:02.0:   bridge window [mem 0xc7f00000-0xc7ffffff]
> [   59.249135] pci_bus 0000:37: bus scan returning with max=37
>
> Could be an error in the ACPI tables, could be a misconfiguration in
> the EEPROM of the Thunderbolt controller.

Attached acpidump to the bug report.

The problem doesn't affect GRUB. It's able to find USB devices on this
adapter in the pre-boot environment, load kernel and initramfs. And
also Windows 10 can boot from USB stick media with this adapter on
these two gen 2 ports.

The non-Thunderbolt, USC-B 3.1 gen1 port can be used to boot Linux,
however. The catch there is this is also the power port. So the choice
is to either use power or boot from that port.

-- 
Chris Murphy