i.MX8M Plus PCIe link regression

i.MX8M Plus PCIe link regression

[Alexander Stein]

Hi,

these days I noticed that there is a PCIe link regression on my i.MX8MP
platform (TQMa8MPxL) running next-20260227.
I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip link up 
workaround for newer platforms"). I always get the following errors:

imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110

Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2 device is
detected without issues.

Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform, both Gen 1 and
Gen 2 devices are detect. Commit 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only
required for conflict free revert.

Here is a summary with outputs:
Gen 1 device
> 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 01)

Gen 2 device
> 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6
> Gb/s RAID controller (rev 20)

output of "dmesg | grep imx6q-pcie". Common part for all tests:
> imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> 0x0000000000
> imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> 0x0018000000
> imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G

next-20260227
Gen 1
> imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
> imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110

Gen 2
> imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
> imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00

next-20260227 + reverts
Gen 1
> imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00

Gen 2
> imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
> imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00

What can we do here? I'm wondering why Gen 2 trains correctly, while Gen 1
doesn't.

Best regards,
Alexander
-- 
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
Amtsgericht München, HRB 105018
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
http://www.tq-group.com/

Re: i.MX8M Plus PCIe link regression

[Tim Harvey]

On Tue, Mar 3, 2026 at 4:38 AM Alexander Stein
<alexander.stein@ew.tq-group.com> wrote:
>
> Hi,
>
> these days I noticed that there is a PCIe link regression on my i.MX8MP
> platform (TQMa8MPxL) running next-20260227.
> I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip link up
> workaround for newer platforms"). I always get the following errors:
>
> imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
> imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110
>
> Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2 device is
> detected without issues.
>
> Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
> b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform, both Gen 1 and
> Gen 2 devices are detect. Commit 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only
> required for conflict free revert.
>
> Here is a summary with outputs:
> Gen 1 device
> > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 01)
>
> Gen 2 device
> > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6
> > Gb/s RAID controller (rev 20)
>
> output of "dmesg | grep imx6q-pcie". Common part for all tests:
> > imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> > imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> > 0x0000000000
> > imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> > 0x0018000000
> > imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> > imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
>
> next-20260227
> Gen 1
> > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
> > imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110
>
> Gen 2
> > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
> > imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
>
> next-20260227 + reverts
> Gen 1
> > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
>
> Gen 2
> > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
> > imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
>
> What can we do here? I'm wondering why Gen 2 trains correctly, while Gen 1
> doesn't.
>
> Best regards,
> Alexander

Hi Alexander,

Does your link partner perhaps suffer from some errata?

Ironically I have had PCI link issues that required the patch you are
needing to revert on IMX8MP with specific switches.

Best Regards,

Tim

Re: i.MX8M Plus PCIe link regression

[Manivannan Sadhasivam]

On Tue, Mar 03, 2026 at 01:38:13PM +0100, Alexander Stein wrote:
> Hi,
> 
> these days I noticed that there is a PCIe link regression on my i.MX8MP
> platform (TQMa8MPxL) running next-20260227.
> I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip link up 
> workaround for newer platforms"). I always get the following errors:
> 
> imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
> imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110
> 
> Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2 device is
> detected without issues.
> 
> Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
> b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform, both Gen 1 and
> Gen 2 devices are detect. Commit 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only
> required for conflict free revert.
> 
> Here is a summary with outputs:
> Gen 1 device
> > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 01)
> 
> Gen 2 device
> > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6
> > Gb/s RAID controller (rev 20)
> 
> output of "dmesg | grep imx6q-pcie". Common part for all tests:
> > imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> > imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> > 0x0000000000
> > imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> > 0x0018000000
> > imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> > imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
> 
> next-20260227
> Gen 1
> > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
> > imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110
> 
> Gen 2
> > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
> > imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
> 
> next-20260227 + reverts
> Gen 1
> > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
> 
> Gen 2
> > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
> > imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
> 
> What can we do here? I'm wondering why Gen 2 trains correctly, while Gen 1
> doesn't.
> 

Thanks for the report and sorry for the breakage. Commit 9c03e30e3ade3, mentions
that the workaround is only needed for the i.MX6 platforms. So I'm not sure why
the patch breaks i.MX8M and that too only for Gen 1 speed.

Also, before 9c03e30e3ade3, LTSSM was always started in Gen 1, but as per your
finding, only Gen 1 devices fail to work. So the code till
imx_pcie_wait_for_speed_change() shouldn't have an impact. Maybe the issue is
due to skipping imx_pcie_wait_for_speed_change()?

Richard, thoughts?

- Mani

-- 
மணிவண்ணன் சதாசிவம்

RE: i.MX8M Plus PCIe link regression

[Hongxing Zhu]

> -----Original Message-----
> From: Manivannan Sadhasivam <mani@kernel.org>
> Sent: 2026年3月4日 0:42
> To: Hongxing Zhu <hongxing.zhu@nxp.com>; Alexander Stein
> <alexander.stein@ew.tq-group.com>
> Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> imx@lists.linux.dev
> Subject: Re: i.MX8M Plus PCIe link regression
> 
> On Tue, Mar 03, 2026 at 01:38:13PM +0100, Alexander Stein wrote:
> > Hi,
> >
> > these days I noticed that there is a PCIe link regression on my
> > i.MX8MP platform (TQMa8MPxL) running next-20260227.
> > I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip link
> > up workaround for newer platforms"). I always get the following errors:
> >
> > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > imx6q-pcie failed with error -110
> >
> > Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2
> > device is detected without issues.
> >
> > Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
> > b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform, both
> Gen
> > 1 and Gen 2 devices are detect. Commit
> > 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only required for conflict
> free revert.
> >
> > Here is a summary with outputs:
> > Gen 1 device
> > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev
> > > 01)
> >
> > Gen 2 device
> > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > > 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe
> > > SATA 6 Gb/s RAID controller (rev 20)
> >
> > output of "dmesg | grep imx6q-pcie". Common part for all tests:
> > > imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> > > imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> > > 0x0000000000
> > > imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> > > 0x0018000000
> > > imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> > > imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K,
> > > limit 16G
> >
> > next-20260227
> > Gen 1
> > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > imx6q-pcie failed with error -110
> >
> > Gen 2
> > > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie
> > > 33800000.pcie: PCI host bridge to bus 0000:00
> >
> > next-20260227 + reverts
> > Gen 1
> > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie 33800000.pcie: PCI
> > > host bridge to bus 0000:00
> >
> > Gen 2
> > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie 33800000.pcie: PCI
> > > host bridge to bus 0000:00
> >
> > What can we do here? I'm wondering why Gen 2 trains correctly, while
> > Gen 1 doesn't.
> >
> 
> Thanks for the report and sorry for the breakage. Commit 9c03e30e3ade3,
> mentions that the workaround is only needed for the i.MX6 platforms. So
> I'm not sure why the patch breaks i.MX8M and that too only for Gen 1
> speed.
> 
> Also, before 9c03e30e3ade3, LTSSM was always started in Gen 1, but as per
> your finding, only Gen 1 devices fail to work. So the code till
> imx_pcie_wait_for_speed_change() shouldn't have an impact. Maybe the
> issue is due to skipping imx_pcie_wait_for_speed_change()?
> 
> Richard, thoughts?
Hi Maini:
Prior to calling imx_pcie_wait_for_speed_change(), the PCIe link must
 already be established. Before commit 9c03e30e3ade3, the speed change
 procedure was only initiated after the initial Gen1 link was successfully
 brought up.

Hi Alexander:
Sorry for the breakage.
Before commit 9c03e30e3ade3, i.MX8MP PCIe forced Gen1 link training
 initially, then switched to Gen3 after link-up by triggering a speed
 change. After commit 9c03e30e3ade3, link training starts directly at
 the Gen3 capability level.

The two link training methods differ only in the i.MX8MP Root Complex (RC)
 link capability configuration prior to link training: one method forces
 Gen1 speed, while the other retains the default Gen3 setting in your tests.

Fortunately, I also have a RTL8168E Gigabit network card on hand.
Based on v7.0-rc1, here is the tests log on my i.MX8MP EVK board and one M.2
Key E to PCIe standard slot exchange adaptor.

Logs:
root@imx8mpevk:~# dmesg | grep pci
[    2.172420] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
[    2.172516] imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
[    2.172545] imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff -> 0x0018000000
[    2.172631] imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
[    2.380065] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
[    2.579604] imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
[    2.580020] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
<snipped>
[    2.583653] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000 PCIe Endpoint
[    2.583858] pci 0000:01:00.0: BAR 0 [io  0x0000-0x00ff]
[    2.583885] pci 0000:01:00.0: BAR 2 [mem 0x00000000-0x00000fff 64bit]
[    2.583902] pci 0000:01:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]
[    2.583914] pci 0000:01:00.0: ROM [mem 0x00000000-0x0001ffff pref]
[    2.584280] pci 0000:01:00.0: supports D1 D2
[    2.584320] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.595615] pci 0000:01:00.0: ASPM: default states L0s L1
root@imx8mpevk:~#
root@imx8mpevk:~# lspci
00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 06)

Best Regards
Richard
> 
> - Mani
> 
> --
> மணிவண்ணன் சதாசிவம்

Re: i.MX8M Plus PCIe link regression

[Manivannan Sadhasivam]

On Wed, Mar 04, 2026 at 02:55:51AM +0000, Hongxing Zhu wrote:
> > -----Original Message-----
> > From: Manivannan Sadhasivam <mani@kernel.org>
> > Sent: 2026年3月4日 0:42
> > To: Hongxing Zhu <hongxing.zhu@nxp.com>; Alexander Stein
> > <alexander.stein@ew.tq-group.com>
> > Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> > imx@lists.linux.dev
> > Subject: Re: i.MX8M Plus PCIe link regression
> > 
> > On Tue, Mar 03, 2026 at 01:38:13PM +0100, Alexander Stein wrote:
> > > Hi,
> > >
> > > these days I noticed that there is a PCIe link regression on my
> > > i.MX8MP platform (TQMa8MPxL) running next-20260227.
> > > I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip link
> > > up workaround for newer platforms"). I always get the following errors:
> > >
> > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > imx6q-pcie failed with error -110
> > >
> > > Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2
> > > device is detected without issues.
> > >
> > > Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
> > > b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform, both
> > Gen
> > > 1 and Gen 2 devices are detect. Commit
> > > 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only required for conflict
> > free revert.
> > >
> > > Here is a summary with outputs:
> > > Gen 1 device
> > > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > > > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > > > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev
> > > > 01)
> > >
> > > Gen 2 device
> > > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > > > 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe
> > > > SATA 6 Gb/s RAID controller (rev 20)
> > >
> > > output of "dmesg | grep imx6q-pcie". Common part for all tests:
> > > > imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> > > > imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> > > > 0x0000000000
> > > > imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> > > > 0x0018000000
> > > > imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> > > > imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K,
> > > > limit 16G
> > >
> > > next-20260227
> > > Gen 1
> > > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > > imx6q-pcie failed with error -110
> > >
> > > Gen 2
> > > > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie
> > > > 33800000.pcie: PCI host bridge to bus 0000:00
> > >
> > > next-20260227 + reverts
> > > Gen 1
> > > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > > 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie 33800000.pcie: PCI
> > > > host bridge to bus 0000:00
> > >
> > > Gen 2
> > > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > > 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie 33800000.pcie: PCI
> > > > host bridge to bus 0000:00
> > >
> > > What can we do here? I'm wondering why Gen 2 trains correctly, while
> > > Gen 1 doesn't.
> > >
> > 
> > Thanks for the report and sorry for the breakage. Commit 9c03e30e3ade3,
> > mentions that the workaround is only needed for the i.MX6 platforms. So
> > I'm not sure why the patch breaks i.MX8M and that too only for Gen 1
> > speed.
> > 
> > Also, before 9c03e30e3ade3, LTSSM was always started in Gen 1, but as per
> > your finding, only Gen 1 devices fail to work. So the code till
> > imx_pcie_wait_for_speed_change() shouldn't have an impact. Maybe the
> > issue is due to skipping imx_pcie_wait_for_speed_change()?
> > 
> > Richard, thoughts?
> Hi Maini:
> Prior to calling imx_pcie_wait_for_speed_change(), the PCIe link must
>  already be established. Before commit 9c03e30e3ade3, the speed change
>  procedure was only initiated after the initial Gen1 link was successfully
>  brought up.
> 
> Hi Alexander:
> Sorry for the breakage.
> Before commit 9c03e30e3ade3, i.MX8MP PCIe forced Gen1 link training
>  initially, then switched to Gen3 after link-up by triggering a speed
>  change. After commit 9c03e30e3ade3, link training starts directly at
>  the Gen3 capability level.
> 
> The two link training methods differ only in the i.MX8MP Root Complex (RC)
>  link capability configuration prior to link training: one method forces
>  Gen1 speed, while the other retains the default Gen3 setting in your tests.
> 

I still don't understand why the link fails to train at Gen 1.

- Mani

> Fortunately, I also have a RTL8168E Gigabit network card on hand.
> Based on v7.0-rc1, here is the tests log on my i.MX8MP EVK board and one M.2
> Key E to PCIe standard slot exchange adaptor.
> 
> Logs:
> root@imx8mpevk:~# dmesg | grep pci
> [    2.172420] imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> [    2.172516] imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff -> 0x0000000000
> [    2.172545] imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff -> 0x0018000000
> [    2.172631] imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> [    2.380065] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 16G
> [    2.579604] imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> [    2.580020] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
> <snipped>
> [    2.583653] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000 PCIe Endpoint
> [    2.583858] pci 0000:01:00.0: BAR 0 [io  0x0000-0x00ff]
> [    2.583885] pci 0000:01:00.0: BAR 2 [mem 0x00000000-0x00000fff 64bit]
> [    2.583902] pci 0000:01:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]
> [    2.583914] pci 0000:01:00.0: ROM [mem 0x00000000-0x0001ffff pref]
> [    2.584280] pci 0000:01:00.0: supports D1 D2
> [    2.584320] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    2.595615] pci 0000:01:00.0: ASPM: default states L0s L1
> root@imx8mpevk:~#
> root@imx8mpevk:~# lspci
> 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 06)
> 
> Best Regards
> Richard
> > 
> > - Mani
> > 
> > --
> > மணிவண்ணன் சதாசிவம்

-- 
மணிவண்ணன் சதாசிவம்

RE: i.MX8M Plus PCIe link regression

[Hongxing Zhu]

> -----Original Message-----
> From: Manivannan Sadhasivam <mani@kernel.org>
> Sent: 2026年3月4日 14:33
> To: Hongxing Zhu <hongxing.zhu@nxp.com>
> Cc: Alexander Stein <alexander.stein@ew.tq-group.com>;
> linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> imx@lists.linux.dev
> Subject: Re: i.MX8M Plus PCIe link regression
> 
> On Wed, Mar 04, 2026 at 02:55:51AM +0000, Hongxing Zhu wrote:
> > > -----Original Message-----
> > > From: Manivannan Sadhasivam <mani@kernel.org>
> > > Sent: 2026年3月4日 0:42
> > > To: Hongxing Zhu <hongxing.zhu@nxp.com>; Alexander Stein
> > > <alexander.stein@ew.tq-group.com>
> > > Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> > > imx@lists.linux.dev
> > > Subject: Re: i.MX8M Plus PCIe link regression
> > >
> > > On Tue, Mar 03, 2026 at 01:38:13PM +0100, Alexander Stein wrote:
> > > > Hi,
> > > >
> > > > these days I noticed that there is a PCIe link regression on my
> > > > i.MX8MP platform (TQMa8MPxL) running next-20260227.
> > > > I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip
> > > > link up workaround for newer platforms"). I always get the following
> errors:
> > > >
> > > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > > imx6q-pcie failed with error -110
> > > >
> > > > Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2
> > > > device is detected without issues.
> > > >
> > > > Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
> > > > b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform,
> both
> > > Gen
> > > > 1 and Gen 2 devices are detect. Commit
> > > > 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only required for
> > > > conflict
> > > free revert.
> > > >
> > > > Here is a summary with outputs:
> > > > Gen 1 device
> > > > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev
> > > > > 01)
> > > > > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > > > > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
> > > > > (rev
> > > > > 01)
> > > >
> > > > Gen 2 device
> > > > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev
> > > > > 01)
> > > > > 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128
> > > > > PCIe SATA 6 Gb/s RAID controller (rev 20)
> > > >
> > > > output of "dmesg | grep imx6q-pcie". Common part for all tests:
> > > > > imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000
> ranges:
> > > > > imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> > > > > 0x0000000000
> > > > > imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> > > > > 0x0018000000
> > > > > imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu
> > > > > 0x1ff00000 imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib,
> > > > > align 64K, limit 16G
> > > >
> > > > next-20260227
> > > > Gen 1
> > > > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > > > imx6q-pcie failed with error -110
> > > >
> > > > Gen 2
> > > > > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie
> > > > > 33800000.pcie: PCI host bridge to bus 0000:00
> > > >
> > > > next-20260227 + reverts
> > > > Gen 1
> > > > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > > > 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie 33800000.pcie:
> > > > > PCI host bridge to bus 0000:00
> > > >
> > > > Gen 2
> > > > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > > > 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie 33800000.pcie:
> > > > > PCI host bridge to bus 0000:00
> > > >
> > > > What can we do here? I'm wondering why Gen 2 trains correctly,
> > > > while Gen 1 doesn't.
> > > >
> > >
> > > Thanks for the report and sorry for the breakage. Commit
> > > 9c03e30e3ade3, mentions that the workaround is only needed for the
> > > i.MX6 platforms. So I'm not sure why the patch breaks i.MX8M and
> > > that too only for Gen 1 speed.
> > >
> > > Also, before 9c03e30e3ade3, LTSSM was always started in Gen 1, but
> > > as per your finding, only Gen 1 devices fail to work. So the code
> > > till
> > > imx_pcie_wait_for_speed_change() shouldn't have an impact. Maybe the
> > > issue is due to skipping imx_pcie_wait_for_speed_change()?
> > >
> > > Richard, thoughts?
> > Hi Maini:
> > Prior to calling imx_pcie_wait_for_speed_change(), the PCIe link must
> > already be established. Before commit 9c03e30e3ade3, the speed change
> > procedure was only initiated after the initial Gen1 link was
> > successfully  brought up.
> >
> > Hi Alexander:
> > Sorry for the breakage.
> > Before commit 9c03e30e3ade3, i.MX8MP PCIe forced Gen1 link training
> > initially, then switched to Gen3 after link-up by triggering a speed
> > change. After commit 9c03e30e3ade3, link training starts directly at
> > the Gen3 capability level.
> >
> > The two link training methods differ only in the i.MX8MP Root Complex
> > (RC)  link capability configuration prior to link training: one method
> > forces
> >  Gen1 speed, while the other retains the default Gen3 setting in your
> tests.
> >
> 
> I still don't understand why the link fails to train at Gen 1.

Hi Mani:
Yes, it is.
IMHO, I'm confused too.

Hi Alexander:
Tim used report one issue on i.MX8MP.
Here is the history. Don't know it's helpful or not. FYI.
https://lore.kernel.org/all/20250709033722.2924372-2-hongxing.zhu@nxp.com/

Best Regards
Richard Zhu
> 
> - Mani
> 
> > Fortunately, I also have a RTL8168E Gigabit network card on hand.
> > Based on v7.0-rc1, here is the tests log on my i.MX8MP EVK board and
> > one M.2 Key E to PCIe standard slot exchange adaptor.
> >
> > Logs:
> > root@imx8mpevk:~# dmesg | grep pci
> > [    2.172420] imx6q-pcie 33800000.pcie: host bridge
> /soc@0/pcie@33800000 ranges:
> > [    2.172516] imx6q-pcie 33800000.pcie:       IO
> 0x001ff80000..0x001ff8ffff -> 0x0000000000
> > [    2.172545] imx6q-pcie 33800000.pcie:      MEM
> 0x0018000000..0x001fefffff -> 0x0018000000
> > [    2.172631] imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu
> 0x1ff00000
> > [    2.380065] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align
> 64K, limit 16G
> > [    2.579604] imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
> > [    2.580020] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
> > <snipped>
> > [    2.583653] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000 PCIe
> Endpoint
> > [    2.583858] pci 0000:01:00.0: BAR 0 [io  0x0000-0x00ff]
> > [    2.583885] pci 0000:01:00.0: BAR 2 [mem 0x00000000-0x00000fff
> 64bit]
> > [    2.583902] pci 0000:01:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit
> pref]
> > [    2.583914] pci 0000:01:00.0: ROM [mem 0x00000000-0x0001ffff pref]
> > [    2.584280] pci 0000:01:00.0: supports D1 D2
> > [    2.584320] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot
> D3cold
> > [    2.595615] pci 0000:01:00.0: ASPM: default states L0s L1
> > root@imx8mpevk:~#
> > root@imx8mpevk:~# lspci
> > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev
> > 06)
> >
> > Best Regards
> > Richard
> > >
> > > - Mani
> > >
> > > --
> > > மணிவண்ணன் சதாசிவம்
> 
> --
> மணிவண்ணன் சதாசிவம்

Re: i.MX8M Plus PCIe link regression

[Alexander Stein]

Hi,

Am Mittwoch, 4. März 2026, 07:32:48 CET schrieb Manivannan Sadhasivam:
> On Wed, Mar 04, 2026 at 02:55:51AM +0000, Hongxing Zhu wrote:
> > > -----Original Message-----
> > > From: Manivannan Sadhasivam <mani@kernel.org>
> > > Sent: 2026年3月4日 0:42
> > > To: Hongxing Zhu <hongxing.zhu@nxp.com>; Alexander Stein
> > > <alexander.stein@ew.tq-group.com>
> > > Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> > > imx@lists.linux.dev
> > > Subject: Re: i.MX8M Plus PCIe link regression
> > > 
> > > On Tue, Mar 03, 2026 at 01:38:13PM +0100, Alexander Stein wrote:
> > > > Hi,
> > > >
> > > > these days I noticed that there is a PCIe link regression on my
> > > > i.MX8MP platform (TQMa8MPxL) running next-20260227.
> > > > I could bisect it back to commit 9c03e30e3ade3 ("PCI: imx6: Skip link
> > > > up workaround for newer platforms"). I always get the following errors:
> > > >
> > > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > > imx6q-pcie failed with error -110
> > > >
> > > > Connected is a Gen1 PCIe -> Ethernet adapter. Interestingly a Gen2
> > > > device is detected without issues.
> > > >
> > > > Reverting 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc and
> > > > b9a8d28ebbf118bb3eac953f4a37abbd341257ab "fixes" my platform, both
> > > Gen
> > > > 1 and Gen 2 devices are detect. Commit
> > > > 3c96a61dd2e098dda8dcac3dce3d38a3c87afbfc is only required for conflict
> > > free revert.
> > > >
> > > > Here is a summary with outputs:
> > > > Gen 1 device
> > > > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > > > > 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > > > > RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev
> > > > > 01)
> > > >
> > > > Gen 2 device
> > > > > 00:00.0 PCI bridge: Synopsys, Inc. DWC_usb3 / PCIe bridge (rev 01)
> > > > > 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe
> > > > > SATA 6 Gb/s RAID controller (rev 20)
> > > >
> > > > output of "dmesg | grep imx6q-pcie". Common part for all tests:
> > > > > imx6q-pcie 33800000.pcie: host bridge /soc@0/pcie@33800000 ranges:
> > > > > imx6q-pcie 33800000.pcie:       IO 0x001ff80000..0x001ff8ffff ->
> > > > > 0x0000000000
> > > > > imx6q-pcie 33800000.pcie:      MEM 0x0018000000..0x001fefffff ->
> > > > > 0x0018000000
> > > > > imx6q-pcie 33800000.pcie: config reg[1] 0x1ff00000 == cpu 0x1ff00000
> > > > > imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K,
> > > > > limit 16G
> > > >
> > > > next-20260227
> > > > Gen 1
> > > > > imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM:
> > > > > CFG_LINKWD_START imx6q-pcie 33800000.pcie: probe with driver
> > > > > imx6q-pcie failed with error -110
> > > >
> > > > Gen 2
> > > > > imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie
> > > > > 33800000.pcie: PCI host bridge to bus 0000:00
> > > >
> > > > next-20260227 + reverts
> > > > Gen 1
> > > > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > > > 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie 33800000.pcie: PCI
> > > > > host bridge to bus 0000:00
> > > >
> > > > Gen 2
> > > > > imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up imx6q-pcie
> > > > > 33800000.pcie: PCIe Gen.2 x1 link up imx6q-pcie 33800000.pcie: PCI
> > > > > host bridge to bus 0000:00
> > > >
> > > > What can we do here? I'm wondering why Gen 2 trains correctly, while
> > > > Gen 1 doesn't.
> > > >
> > > 
> > > Thanks for the report and sorry for the breakage. Commit 9c03e30e3ade3,
> > > mentions that the workaround is only needed for the i.MX6 platforms. So
> > > I'm not sure why the patch breaks i.MX8M and that too only for Gen 1
> > > speed.
> > > 
> > > Also, before 9c03e30e3ade3, LTSSM was always started in Gen 1, but as per
> > > your finding, only Gen 1 devices fail to work. So the code till
> > > imx_pcie_wait_for_speed_change() shouldn't have an impact. Maybe the
> > > issue is due to skipping imx_pcie_wait_for_speed_change()?
> > > 
> > > Richard, thoughts?
> > Hi Maini:
> > Prior to calling imx_pcie_wait_for_speed_change(), the PCIe link must
> >  already be established. Before commit 9c03e30e3ade3, the speed change
> >  procedure was only initiated after the initial Gen1 link was successfully
> >  brought up.
> > 
> > Hi Alexander:
> > Sorry for the breakage.
> > Before commit 9c03e30e3ade3, i.MX8MP PCIe forced Gen1 link training
> >  initially, then switched to Gen3 after link-up by triggering a speed
> >  change. After commit 9c03e30e3ade3, link training starts directly at
> >  the Gen3 capability level.
> > 
> > The two link training methods differ only in the i.MX8MP Root Complex (RC)
> >  link capability configuration prior to link training: one method forces
> >  Gen1 speed, while the other retains the default Gen3 setting in your tests.
> > 
> 
> I still don't understand why the link fails to train at Gen 1.

Independently from the configured maximum link speed, shouldn't the initial
link be Gen1 anyway and upgraded to Gen 2 later on?

I added some debug output to dw_pcie_link_up()

imx6q-pcie 33800000.pcie: dw_pcie_link_set_max_speed: current pci->max_link_speed: 3
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f704
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x28000000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f722
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110


imx6q-pcie 33800000.pcie: dw_pcie_link_set_max_speed: current pci->max_link_speed: 2
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x28000000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x28000000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x2800f702
imx6q-pcie 33800000.pcie: Link failed to come up. LTSSM: CFG_LINKWD_START
imx6q-pcie 33800000.pcie: probe with driver imx6q-pcie failed with error -110


imx6q-pcie 33800000.pcie: dw_pcie_link_set_max_speed: current pci->max_link_speed: 1
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08000010
imx6q-pcie 33800000.pcie: PCIe Gen.1 x1 link up
imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00

So for any maximum speed above Gen1 it seems that LTSSM is stuck at link
training.
Unfortunately only link training and link up bits are defined and RM doesn't
show anything, so I don't know what the other bits are indicating.

For a Gen2 device the link training succeeds right away.

imx6q-pcie 33800000.pcie: dw_pcie_link_set_max_speed: current pci->max_link_speed: 2
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08200000
imx6q-pcie 33800000.pcie: dw_pcie_link_up: PCIE_PORT_DEBUG1: 0x08000010
imx6q-pcie 33800000.pcie: PCIe Gen.2 x1 link up
imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00

Best regards,
Alexander
-- 
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
Amtsgericht München, HRB 105018
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
http://www.tq-group.com/