From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Date: Mon, 6 Jan 2014 15:26:21 -0500 Message-ID: <20140106202621.GA30667@phenom.dumpdata.com> References: <387b2f80a3866e53ec471421558cf4de@mail.shatteredsilicon.net> <52308E1402000078000F2748@nat28.tlf.novell.com> <20131211183233.GA2760@phenom.dumpdata.com> <52A8D5E5.2030902@bobich.net> <20131211213025.GA8283@phenom.dumpdata.com> <52AAF9D7020000780010CF2C@nat28.tlf.novell.com> <20131213144319.GK2923@phenom.dumpdata.com> <52AB2E18020000780010D13B@nat28.tlf.novell.com> <52AB275D.2010401@bobich.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W0Gl0-0005gJ-Eu for xen-devel@lists.xenproject.org; Mon, 06 Jan 2014 20:26:42 +0000 Content-Disposition: inline In-Reply-To: <52AB275D.2010401@bobich.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Gordan Bobic Cc: Yang Z Zhang , "xen-devel@lists.xenproject.org" , Jan Beulich List-Id: xen-devel@lists.xenproject.org On Fri, Dec 13, 2013 at 03:27:25PM +0000, Gordan Bobic wrote: > On 12/13/2013 02:56 PM, Jan Beulich wrote: > >>>>On 13.12.13 at 15:43, Konrad Rzeszutek Wilk wrote: > >>-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller > >> +-01.0-[01]--+-00.0 Intel Corporation 82571EB Gigabit Ethernet Controller > >> | \-00.1 Intel Corporation 82571EB Gigabit Ethernet Controller > >> +-01.1-[02]----00.0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] > >> +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller > >> +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller > >> +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI > >> +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 > >> +-16.3 Intel Corporation 8 Series/C220 Series Chipset Family KT Controller > >> +-19.0 Intel Corporation Ethernet Connection I217-LM > >> +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 > >> +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller > >> +-1c.0-[03]----00.0 Intel Corporation 82574L Gigabit Network Connection > >> +-1c.1-[04]----00.0 Intel Corporation 82574L Gigabit Network Connection > >> +-1c.3-[05]----00.0 Intel Corporation I210 Gigabit Network Connection > >> +-1c.4-[06]----00.0 Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) > >> +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree Corporation Bt878 Video Capture > >> | | +-08.1 Brooktree Corporation Bt878 Audio Capture > >> | | +-09.0 Brooktree Corporation Bt878 Video Capture > >> | | +-09.1 Brooktree Corporation Bt878 Audio Capture > >> | | +-0a.0 Brooktree Corporation Bt878 Video Capture > >> | | +-0a.1 Brooktree Corporation Bt878 Audio Capture > >> | | +-0b.0 Brooktree Corporation Bt878 Video Capture > >> | | \-0b.1 Brooktree Corporation Bt878 Audio Capture > > > >So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are > >PCI-PCI bridges, the capture devices are ordinary ones. The only > >possibly unexpected aspect I can see here is that there are two > >intermediate PCI-PCI bridges, but iirc this should be handled by > >the VT-d code. > > Not sure about multiple intermediate PCI-PCI bridges, but I can > confirm that multiuple intermediate PCIe bridge setup works fine, > e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690) To double-check whether this is a problem with the card itself, I plucked the card in an AMD box (TA890FXE) and an Intel box (ThinkServer TS130) and they work great there. It only has one PCIe->PCI bridge for the PCI cards. TA890FXE: 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) Flags: bus master, 66MHz, medium devsel, latency 64 Bus: primary=00, secondary=02, subordinate=02, sec-latency=64 I/O behind bridge: 0000c000-0000dfff 02:05.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent mode) (rev 11) (prog-if 00 [Normal decode]) Flags: bus master, medium devsel, latency 64 Bus: primary=02, secondary=03, subordinate=03, sec-latency=64 Prefetchable memory behind bridge: fde00000-fdefffff Capabilities: [80] Power Management version 2 Capabilities: [90] CompactPCI hot-swap +-14.4-[02-03]--+-05.0-[03]--+-08.0 Brooktree Corporation Bt878 Video Capture | | +-08.1 Brooktree Corporation Bt878 Audio Capture | | +-09.0 Brooktree Corporation Bt878 Video Capture | | +-09.1 Brooktree Corporation Bt878 Audio Capture | | +-0a.0 Brooktree Corporation Bt878 Video Capture | | +-0a.1 Brooktree Corporation Bt878 Audio Capture | | +-0b.0 Brooktree Corporation Bt878 Video Capture | | \-0b.1 Brooktree Corporation Bt878 Audio Capture | \-06.0 NetMos Technology PCI 9835 Multi-I/O Controller ThinkServer TS130: 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) (prog-if 01 [Subtractive decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=02, subordinate=03, sec-latency=32 Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff Capabilities: [50] Subsystem: Lenovo Device 1025 00:00.0 Host bridge: Intel Corporation Device 0108 (rev 09) Subsystem: Lenovo Device 1025 Flags: bus master, fast devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c +-1e.0-[02-03]----00.0-[03]--+-08.0 Brooktree Corporation Bt878 Video Capture | +-08.1 Brooktree Corporation Bt878 Audio Capture | +-09.0 Brooktree Corporation Bt878 Video Capture | +-09.1 Brooktree Corporation Bt878 Audio Capture | +-0a.0 Brooktree Corporation Bt878 Video Capture | +-0a.1 Brooktree Corporation Bt878 Audio Capture | +-0b.0 Brooktree Corporation Bt878 Video Capture | \-0b.1 Brooktree Corporation Bt878 Audio Capture SuperMicro (two PCI bridgeS): +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree Corporation Bt878 Video Capture | | +-08.1 Brooktree Corporation Bt878 Audio Capture | | +-09.0 Brooktree Corporation Bt878 Video Capture | | +-09.1 Brooktree Corporation Bt878 Audio Capture | | +-0a.0 Brooktree Corporation Bt878 Video Capture | | +-0a.1 Brooktree Corporation Bt878 Audio Capture | | +-0b.0 Brooktree Corporation Bt878 Video Capture | | \-0b.1 Brooktree Corporation Bt878 Audio Capture | \-03.0 Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=07, subordinate=09, sec-latency=0 Memory behind bridge: f0400000-f05fffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805 Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: f0d00000-f0dfffff Capabilities: [88] Subsystem: Intel Corporation Device 2010 Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [a0] Express Root Port (Slot+), MSI 00 Capabilities: [100] Virtual Channel Capabilities: [140] Root Complex Link Capabilities: [d94] #19 Kernel driver in use: pcieport 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01 [Subtractive decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=07, secondary=08, subordinate=09, sec-latency=32 Memory behind bridge: f0400000-f05fffff Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805 Capabilities: [a0] Power Management version 3 Which would look like this: C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card \--------------> IEEE-1394a I am actually wondering if this 07:00.0 device is the one that reports itself as 08:00.0 (which I think is what you alluding to Jan) Gordan (in earlier emails) reported that the VT-d does work for him and he has much more complex setup than I do on this motherboard: This is on http://www.newegg.com/Product/Product.aspx?Item=N82E16813188070 \-[0000:00]-+-00.0 Intel Corporation 5520 I/O Hub to ESI Port [8086:3406] +-07.0-[05-08]----00.0-[06-08]--+-00.0-[08]--+-00.0 NVIDIA Corporation GF100GL [Quadro 6000] [10de:06d8] | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5] | \-02.0-[07]--+-00.0 NVIDIA Corporation GF100GL [Quadro 5000] [10de:06d9] | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5] 00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 22) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR- TAbort- SERR- NF200 ---> Quadro 6000 (GPU and audio card) \---> NF200 ---> Quadro 5000 (GPU and audio card). Which is similar setup as mine - an intermediate bridge. So I think the code is OK - this is likely a firmware/motherboard bug.