From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Gordan Bobic <gordan@bobich.net>
Cc: Yang Z Zhang <yang.z.zhang@intel.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
Jan Beulich <JBeulich@suse.com>
Subject: Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
Date: Mon, 6 Jan 2014 15:26:21 -0500 [thread overview]
Message-ID: <20140106202621.GA30667@phenom.dumpdata.com> (raw)
In-Reply-To: <52AB275D.2010401@bobich.net>
On Fri, Dec 13, 2013 at 03:27:25PM +0000, Gordan Bobic wrote:
> On 12/13/2013 02:56 PM, Jan Beulich wrote:
> >>>>On 13.12.13 at 15:43, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> >>-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller
> >> +-01.0-[01]--+-00.0 Intel Corporation 82571EB Gigabit Ethernet Controller
> >> | \-00.1 Intel Corporation 82571EB Gigabit Ethernet Controller
> >> +-01.1-[02]----00.0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
> >> +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
> >> +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
> >> +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
> >> +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
> >> +-16.3 Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
> >> +-19.0 Intel Corporation Ethernet Connection I217-LM
> >> +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
> >> +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
> >> +-1c.0-[03]----00.0 Intel Corporation 82574L Gigabit Network Connection
> >> +-1c.1-[04]----00.0 Intel Corporation 82574L Gigabit Network Connection
> >> +-1c.3-[05]----00.0 Intel Corporation I210 Gigabit Network Connection
> >> +-1c.4-[06]----00.0 Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
> >> +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree Corporation Bt878 Video Capture
> >> | | +-08.1 Brooktree Corporation Bt878 Audio Capture
> >> | | +-09.0 Brooktree Corporation Bt878 Video Capture
> >> | | +-09.1 Brooktree Corporation Bt878 Audio Capture
> >> | | +-0a.0 Brooktree Corporation Bt878 Video Capture
> >> | | +-0a.1 Brooktree Corporation Bt878 Audio Capture
> >> | | +-0b.0 Brooktree Corporation Bt878 Video Capture
> >> | | \-0b.1 Brooktree Corporation Bt878 Audio Capture
> >
> >So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
> >PCI-PCI bridges, the capture devices are ordinary ones. The only
> >possibly unexpected aspect I can see here is that there are two
> >intermediate PCI-PCI bridges, but iirc this should be handled by
> >the VT-d code.
>
> Not sure about multiple intermediate PCI-PCI bridges, but I can
> confirm that multiuple intermediate PCIe bridge setup works fine,
> e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690)
To double-check whether this is a problem with the card itself,
I plucked the card in an AMD box (TA890FXE) and an Intel box (ThinkServer
TS130) and they work great there. It only has one PCIe->PCI bridge
for the PCI cards.
TA890FXE:
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
Flags: bus master, 66MHz, medium devsel, latency 64
Bus: primary=00, secondary=02, subordinate=02, sec-latency=64
I/O behind bridge: 0000c000-0000dfff
02:05.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent mode) (rev 11) (prog-if 00 [Normal decode])
Flags: bus master, medium devsel, latency 64
Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
Prefetchable memory behind bridge: fde00000-fdefffff
Capabilities: [80] Power Management version 2
Capabilities: [90] CompactPCI hot-swap <?>
+-14.4-[02-03]--+-05.0-[03]--+-08.0 Brooktree Corporation Bt878 Video Capture
| | +-08.1 Brooktree Corporation Bt878 Audio Capture
| | +-09.0 Brooktree Corporation Bt878 Video Capture
| | +-09.1 Brooktree Corporation Bt878 Audio Capture
| | +-0a.0 Brooktree Corporation Bt878 Video Capture
| | +-0a.1 Brooktree Corporation Bt878 Audio Capture
| | +-0b.0 Brooktree Corporation Bt878 Video Capture
| | \-0b.1 Brooktree Corporation Bt878 Audio Capture
| \-06.0 NetMos Technology PCI 9835 Multi-I/O Controller
ThinkServer TS130:
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=03, sec-latency=32
Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff
Capabilities: [50] Subsystem: Lenovo Device 1025
00:00.0 Host bridge: Intel Corporation Device 0108 (rev 09)
Subsystem: Lenovo Device 1025
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
+-1e.0-[02-03]----00.0-[03]--+-08.0 Brooktree Corporation Bt878 Video Capture
| +-08.1 Brooktree Corporation Bt878 Audio Capture
| +-09.0 Brooktree Corporation Bt878 Video Capture
| +-09.1 Brooktree Corporation Bt878 Audio Capture
| +-0a.0 Brooktree Corporation Bt878 Video Capture
| +-0a.1 Brooktree Corporation Bt878 Audio Capture
| +-0b.0 Brooktree Corporation Bt878 Video Capture
| \-0b.1 Brooktree Corporation Bt878 Audio Capture
SuperMicro (two PCI bridgeS):
+-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree Corporation Bt878 Video Capture
| | +-08.1 Brooktree Corporation Bt878 Audio Capture
| | +-09.0 Brooktree Corporation Bt878 Video Capture
| | +-09.1 Brooktree Corporation Bt878 Audio Capture
| | +-0a.0 Brooktree Corporation Bt878 Video Capture
| | +-0a.1 Brooktree Corporation Bt878 Audio Capture
| | +-0b.0 Brooktree Corporation Bt878 Video Capture
| | \-0b.1 Brooktree Corporation Bt878 Audio Capture
| \-03.0 Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
Memory behind bridge: f0400000-f05fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: f0d00000-f0dfffff
Capabilities: [88] Subsystem: Intel Corporation Device 2010
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] #19
Kernel driver in use: pcieport
07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=08, subordinate=09, sec-latency=32
Memory behind bridge: f0400000-f05fffff
Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
Capabilities: [a0] Power Management version 3
Which would look like this:
C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card
\--------------> IEEE-1394a
I am actually wondering if this 07:00.0 device is the one that
reports itself as 08:00.0 (which I think is what you alluding to Jan)
Gordan (in earlier emails) reported that the VT-d does work for him and
he has much more complex setup than I do on this motherboard:
This is on http://www.newegg.com/Product/Product.aspx?Item=N82E16813188070
\-[0000:00]-+-00.0 Intel Corporation 5520 I/O Hub to ESI Port [8086:3406]
+-07.0-[05-08]----00.0-[06-08]--+-00.0-[08]--+-00.0 NVIDIA Corporation GF100GL [Quadro 6000] [10de:06d8]
| | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
| \-02.0-[07]--+-00.0 NVIDIA Corporation GF100GL [Quadro 5000] [10de:06d9]
| \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 22) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: f4000000-fbdfffff
05:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: f4000000-fbdfffff
Prefetchable memory behind bridge: 00000000a8000000-00000000bfffffff
06:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: f8000000-fbdfffff
Prefetchable memory behind bridge: 00000000b4000000-00000000bfffffff
Thought I am having a hard time parsing the 'lspci -vt' I think it is:
X58---> NF200 ---> Quadro 6000 (GPU and audio card)
\---> NF200 ---> Quadro 5000 (GPU and audio card).
Which is similar setup as mine - an intermediate bridge.
So I think the code is OK - this is likely a firmware/motherboard
bug.
next prev parent reply other threads:[~2014-01-06 20:26 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-11 11:05 Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Gordan Bobic
2013-09-11 11:25 ` Gordan Bobic
2013-09-11 11:44 ` Gordan Bobic
2013-09-11 11:57 ` Jan Beulich
2013-09-11 12:19 ` Gordan Bobic
2013-09-11 12:56 ` Pasi Kärkkäinen
2013-09-11 11:53 ` Jan Beulich
2013-09-11 12:14 ` Gordan Bobic
2013-09-11 12:31 ` Jan Beulich
2013-09-11 12:45 ` Gordan Bobic
2013-09-11 13:03 ` Jan Beulich
2013-09-11 13:10 ` Gordan Bobic
2013-09-11 13:22 ` Jan Beulich
2013-09-11 13:26 ` Gordan Bobic
2013-09-11 13:36 ` Jan Beulich
2013-09-12 6:20 ` Zhang, Yang Z
2013-12-11 18:32 ` Konrad Rzeszutek Wilk
2013-12-11 21:15 ` Gordan Bobic
2013-12-11 21:30 ` Konrad Rzeszutek Wilk
2013-12-13 11:13 ` Jan Beulich
2013-12-13 14:43 ` Konrad Rzeszutek Wilk
2013-12-13 14:56 ` Jan Beulich
2013-12-13 15:27 ` Gordan Bobic
2014-01-06 20:26 ` Konrad Rzeszutek Wilk [this message]
2014-01-06 21:45 ` Konrad Rzeszutek Wilk
2014-01-07 3:17 ` Zhang, Yang Z
2014-01-07 10:35 ` Gordan Bobic
2014-01-07 10:38 ` Andrew Cooper
2014-01-07 10:44 ` Gordan Bobic
2014-02-21 19:08 ` Konrad Rzeszutek Wilk
2014-02-24 10:14 ` Jan Beulich
2013-09-11 13:23 ` Gordan Bobic
2013-09-11 13:34 ` Jan Beulich
-- strict thread matches above, loose matches on Subject: below --
2014-01-07 11:26 Wu, Feng
2014-01-07 11:35 ` Gordan Bobic
2014-01-07 12:15 ` Jan Beulich
2014-01-07 12:42 ` Gordan Bobic
2014-01-07 14:38 ` Konrad Rzeszutek Wilk
2014-01-07 14:47 ` Jan Beulich
2014-01-07 15:40 ` Konrad Rzeszutek Wilk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140106202621.GA30667@phenom.dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=JBeulich@suse.com \
--cc=gordan@bobich.net \
--cc=xen-devel@lists.xenproject.org \
--cc=yang.z.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).