From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Gordan Bobic <gordan@bobich.net>
Cc: Yang Z Zhang <yang.z.zhang@intel.com>,
"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
Jan Beulich <JBeulich@suse.com>
Subject: Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
Date: Mon, 6 Jan 2014 15:26:21 -0500 [thread overview]
Message-ID: <20140106202621.GA30667@phenom.dumpdata.com> (raw)
In-Reply-To: <52AB275D.2010401@bobich.net>
On Fri, Dec 13, 2013 at 03:27:25PM +0000, Gordan Bobic wrote:
> On 12/13/2013 02:56 PM, Jan Beulich wrote:
> >>>>On 13.12.13 at 15:43, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> >>-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller
> >> +-01.0-[01]--+-00.0 Intel Corporation 82571EB Gigabit Ethernet Controller
> >> | \-00.1 Intel Corporation 82571EB Gigabit Ethernet Controller
> >> +-01.1-[02]----00.0 LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
> >> +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
> >> +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
> >> +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
> >> +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
> >> +-16.3 Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
> >> +-19.0 Intel Corporation Ethernet Connection I217-LM
> >> +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
> >> +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
> >> +-1c.0-[03]----00.0 Intel Corporation 82574L Gigabit Network Connection
> >> +-1c.1-[04]----00.0 Intel Corporation 82574L Gigabit Network Connection
> >> +-1c.3-[05]----00.0 Intel Corporation I210 Gigabit Network Connection
> >> +-1c.4-[06]----00.0 Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
> >> +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree Corporation Bt878 Video Capture
> >> | | +-08.1 Brooktree Corporation Bt878 Audio Capture
> >> | | +-09.0 Brooktree Corporation Bt878 Video Capture
> >> | | +-09.1 Brooktree Corporation Bt878 Audio Capture
> >> | | +-0a.0 Brooktree Corporation Bt878 Video Capture
> >> | | +-0a.1 Brooktree Corporation Bt878 Audio Capture
> >> | | +-0b.0 Brooktree Corporation Bt878 Video Capture
> >> | | \-0b.1 Brooktree Corporation Bt878 Audio Capture
> >
> >So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
> >PCI-PCI bridges, the capture devices are ordinary ones. The only
> >possibly unexpected aspect I can see here is that there are two
> >intermediate PCI-PCI bridges, but iirc this should be handled by
> >the VT-d code.
>
> Not sure about multiple intermediate PCI-PCI bridges, but I can
> confirm that multiuple intermediate PCIe bridge setup works fine,
> e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690)
To double-check whether this is a problem with the card itself,
I plucked the card in an AMD box (TA890FXE) and an Intel box (ThinkServer
TS130) and they work great there. It only has one PCIe->PCI bridge
for the PCI cards.
TA890FXE:
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
Flags: bus master, 66MHz, medium devsel, latency 64
Bus: primary=00, secondary=02, subordinate=02, sec-latency=64
I/O behind bridge: 0000c000-0000dfff
02:05.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent mode) (rev 11) (prog-if 00 [Normal decode])
Flags: bus master, medium devsel, latency 64
Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
Prefetchable memory behind bridge: fde00000-fdefffff
Capabilities: [80] Power Management version 2
Capabilities: [90] CompactPCI hot-swap <?>
+-14.4-[02-03]--+-05.0-[03]--+-08.0 Brooktree Corporation Bt878 Video Capture
| | +-08.1 Brooktree Corporation Bt878 Audio Capture
| | +-09.0 Brooktree Corporation Bt878 Video Capture
| | +-09.1 Brooktree Corporation Bt878 Audio Capture
| | +-0a.0 Brooktree Corporation Bt878 Video Capture
| | +-0a.1 Brooktree Corporation Bt878 Audio Capture
| | +-0b.0 Brooktree Corporation Bt878 Video Capture
| | \-0b.1 Brooktree Corporation Bt878 Audio Capture
| \-06.0 NetMos Technology PCI 9835 Multi-I/O Controller
ThinkServer TS130:
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=03, sec-latency=32
Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff
Capabilities: [50] Subsystem: Lenovo Device 1025
00:00.0 Host bridge: Intel Corporation Device 0108 (rev 09)
Subsystem: Lenovo Device 1025
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
+-1e.0-[02-03]----00.0-[03]--+-08.0 Brooktree Corporation Bt878 Video Capture
| +-08.1 Brooktree Corporation Bt878 Audio Capture
| +-09.0 Brooktree Corporation Bt878 Video Capture
| +-09.1 Brooktree Corporation Bt878 Audio Capture
| +-0a.0 Brooktree Corporation Bt878 Video Capture
| +-0a.1 Brooktree Corporation Bt878 Audio Capture
| +-0b.0 Brooktree Corporation Bt878 Video Capture
| \-0b.1 Brooktree Corporation Bt878 Audio Capture
SuperMicro (two PCI bridgeS):
+-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree Corporation Bt878 Video Capture
| | +-08.1 Brooktree Corporation Bt878 Audio Capture
| | +-09.0 Brooktree Corporation Bt878 Video Capture
| | +-09.1 Brooktree Corporation Bt878 Audio Capture
| | +-0a.0 Brooktree Corporation Bt878 Video Capture
| | +-0a.1 Brooktree Corporation Bt878 Audio Capture
| | +-0b.0 Brooktree Corporation Bt878 Video Capture
| | \-0b.1 Brooktree Corporation Bt878 Audio Capture
| \-03.0 Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
Memory behind bridge: f0400000-f05fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: f0d00000-f0dfffff
Capabilities: [88] Subsystem: Intel Corporation Device 2010
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] #19
Kernel driver in use: pcieport
07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=08, subordinate=09, sec-latency=32
Memory behind bridge: f0400000-f05fffff
Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
Capabilities: [a0] Power Management version 3
Which would look like this:
C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card
\--------------> IEEE-1394a
I am actually wondering if this 07:00.0 device is the one that
reports itself as 08:00.0 (which I think is what you alluding to Jan)
Gordan (in earlier emails) reported that the VT-d does work for him and
he has much more complex setup than I do on this motherboard:
This is on http://www.newegg.com/Product/Product.aspx?Item=N82E16813188070
\-[0000:00]-+-00.0 Intel Corporation 5520 I/O Hub to ESI Port [8086:3406]
+-07.0-[05-08]----00.0-[06-08]--+-00.0-[08]--+-00.0 NVIDIA Corporation GF100GL [Quadro 6000] [10de:06d8]
| | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
| \-02.0-[07]--+-00.0 NVIDIA Corporation GF100GL [Quadro 5000] [10de:06d9]
| \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 22) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: f4000000-fbdfffff
05:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: f4000000-fbdfffff
Prefetchable memory behind bridge: 00000000a8000000-00000000bfffffff
06:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: f8000000-fbdfffff
Prefetchable memory behind bridge: 00000000b4000000-00000000bfffffff
Thought I am having a hard time parsing the 'lspci -vt' I think it is:
X58---> NF200 ---> Quadro 6000 (GPU and audio card)
\---> NF200 ---> Quadro 5000 (GPU and audio card).
Which is similar setup as mine - an intermediate bridge.
So I think the code is OK - this is likely a firmware/motherboard
bug.
next prev parent reply other threads:[~2014-01-06 20:26 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-11 11:05 Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Gordan Bobic
2013-09-11 11:25 ` Gordan Bobic
2013-09-11 11:44 ` Gordan Bobic
2013-09-11 11:57 ` Jan Beulich
2013-09-11 12:19 ` Gordan Bobic
2013-09-11 12:56 ` Pasi Kärkkäinen
2013-09-11 11:53 ` Jan Beulich
2013-09-11 12:14 ` Gordan Bobic
2013-09-11 12:31 ` Jan Beulich
2013-09-11 12:45 ` Gordan Bobic
2013-09-11 13:03 ` Jan Beulich
2013-09-11 13:10 ` Gordan Bobic
2013-09-11 13:22 ` Jan Beulich
2013-09-11 13:26 ` Gordan Bobic
2013-09-11 13:36 ` Jan Beulich
2013-09-12 6:20 ` Zhang, Yang Z
2013-12-11 18:32 ` Konrad Rzeszutek Wilk
2013-12-11 21:15 ` Gordan Bobic
2013-12-11 21:30 ` Konrad Rzeszutek Wilk
2013-12-13 11:13 ` Jan Beulich
2013-12-13 14:43 ` Konrad Rzeszutek Wilk
2013-12-13 14:56 ` Jan Beulich
2013-12-13 15:27 ` Gordan Bobic
2014-01-06 20:26 ` Konrad Rzeszutek Wilk [this message]
2014-01-06 21:45 ` Konrad Rzeszutek Wilk
2014-01-07 3:17 ` Zhang, Yang Z
2014-01-07 10:35 ` Gordan Bobic
2014-01-07 10:38 ` Andrew Cooper
2014-01-07 10:44 ` Gordan Bobic
2014-02-21 19:08 ` Konrad Rzeszutek Wilk
2014-02-24 10:14 ` Jan Beulich
2013-09-11 13:23 ` Gordan Bobic
2013-09-11 13:34 ` Jan Beulich
-- strict thread matches above, loose matches on Subject: below --
2014-01-07 11:26 Wu, Feng
2014-01-07 11:35 ` Gordan Bobic
2014-01-07 12:15 ` Jan Beulich
2014-01-07 12:42 ` Gordan Bobic
2014-01-07 14:38 ` Konrad Rzeszutek Wilk
2014-01-07 14:47 ` Jan Beulich
2014-01-07 15:40 ` Konrad Rzeszutek Wilk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140106202621.GA30667@phenom.dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=JBeulich@suse.com \
--cc=gordan@bobich.net \
--cc=xen-devel@lists.xenproject.org \
--cc=yang.z.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.