Cabling for NTB between Intel Xeon E5 v4 servers

Cabling for NTB between Intel Xeon E5 v4 servers

[Ed Swierk]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

I'm trying to set up a PCIe link between Intel Xeon E5 v4 (S2600WT) servers
running Linux 4.10.17, as a testbed for performance measurements. My goal
is to evaluate feasibility NTB on a custom Xeon D-based plug-in card, but
as a first step I'm sticking with systems on which NTB is already known to
(or supposed to) work.

I read https://github.com/davejiang/linux/wiki/Intel-NTB-Startup-Guide and
every other document I could find on Intel NTB, but I'm stuck on a basic
question: what kind of cable should I use to connect the PCIe slots?

I found some card-and-cable kits at
http://www.onestopsystems.com/expansion-kits, but these are quite expensive
($thousands).

Another idea is to use so-called mini-SAS HD aka SFF-8643, which I
understand can carry PCIe x4, along with a pair of PCIe-to-SF-8643 adapters
(e.g.
https://www.startech.com/HDD/Adapters/pcie-nvme-u2-ssd-adapter~PEX4SFF8643).
The TX-to-RX wiring seems like it should work, but I have no idea about
signal integrity through such a cable.

There's also the SuperMicro AOC-SLG3-2E4R, which is similar to the StarTech
card but claims to have redriver circuitry.

What cabling have folks successfully used for NTB between Intel Xeon-based
servers?

--Ed

[-- Attachment #2: Type: text/html, Size: 1653 bytes --]

Re: Cabling for NTB between Intel Xeon E5 v4 servers

[Dave Jiang]

On 05/26/2017 07:00 PM, 'Ed Swierk' via linux-ntb wrote:
> I'm trying to set up a PCIe link between Intel Xeon E5 v4 (S2600WT)
> servers running Linux 4.10.17, as a testbed for performance
> measurements. My goal is to evaluate feasibility NTB on a custom Xeon
> D-based plug-in card, but as a first step I'm sticking with systems on
> which NTB is already known to (or supposed to) work.

So you'll need PCIe retimer cards. Not redrivers. The hw people
recommended using iDT based card pair. Others may work but have not
tried. I don't know what to recommend because we have an in-house
cabling set that's not commercially available for testing. Also, make
sure your BIOS support NTB and BAR size change and also it turns off SSC
when NTB is on.

> 
> I read https://github.com/davejiang/linux/wiki/Intel-NTB-Startup-Guide
> and every other document I could find on Intel NTB, but I'm stuck on a
> basic question: what kind of cable should I use to connect the PCIe slots?
> 
> I found some card-and-cable kits
> at http://www.onestopsystems.com/expansion-kits, but these are quite
> expensive ($thousands).
> 
> Another idea is to use so-called mini-SAS HD aka SFF-8643, which I
> understand can carry PCIe x4, along with a pair of PCIe-to-SF-8643
> adapters
> (e.g. https://www.startech.com/HDD/Adapters/pcie-nvme-u2-ssd-adapter~PEX4SFF8643).
> The TX-to-RX wiring seems like it should work, but I have no idea about
> signal integrity through such a cable.
> 
> There's also the SuperMicro AOC-SLG3-2E4R, which is similar to the
> StarTech card but claims to have redriver circuitry.
> 
> What cabling have folks successfully used for NTB between Intel
> Xeon-based servers?
> 
> --Ed
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to linux-ntb+unsubscribe@googlegroups.com
> <mailto:linux-ntb+unsubscribe@googlegroups.com>.
> To post to this group, send email to linux-ntb@googlegroups.com
> <mailto:linux-ntb@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/linux-ntb/CAO_EM_n81HfXZY-X%2BOuQWX0T%3D0E5BYpfJJEf%2BWQVC1DzyyWkLA%40mail.gmail.com
> <https://groups.google.com/d/msgid/linux-ntb/CAO_EM_n81HfXZY-X%2BOuQWX0T%3D0E5BYpfJJEf%2BWQVC1DzyyWkLA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

Re: Cabling for NTB between Intel Xeon E5 v4 servers

[Ed Swierk]

[-- Attachment #1: Type: text/plain, Size: 2882 bytes --]

I went ahead and connected two PCIe slots on the same system (Intel
S2600WT) in NTB B2B mode, with just an SFF-8643 cable and dumb passive
adapters, and managed to get 4 lanes at gen 2 speed. Next I'll try between
two systems.

--Ed


On Tue, May 30, 2017 at 9:34 AM, Dave Jiang <dave.jiang@intel.com> wrote:

>
>
> On 05/26/2017 07:00 PM, 'Ed Swierk' via linux-ntb wrote:
> > I'm trying to set up a PCIe link between Intel Xeon E5 v4 (S2600WT)
> > servers running Linux 4.10.17, as a testbed for performance
> > measurements. My goal is to evaluate feasibility NTB on a custom Xeon
> > D-based plug-in card, but as a first step I'm sticking with systems on
> > which NTB is already known to (or supposed to) work.
>
> So you'll need PCIe retimer cards. Not redrivers. The hw people
> recommended using iDT based card pair. Others may work but have not
> tried. I don't know what to recommend because we have an in-house
> cabling set that's not commercially available for testing. Also, make
> sure your BIOS support NTB and BAR size change and also it turns off SSC
> when NTB is on.
>
> >
> > I read https://github.com/davejiang/linux/wiki/Intel-NTB-Startup-Guide
> > and every other document I could find on Intel NTB, but I'm stuck on a
> > basic question: what kind of cable should I use to connect the PCIe
> slots?
> >
> > I found some card-and-cable kits
> > at http://www.onestopsystems.com/expansion-kits, but these are quite
> > expensive ($thousands).
> >
> > Another idea is to use so-called mini-SAS HD aka SFF-8643, which I
> > understand can carry PCIe x4, along with a pair of PCIe-to-SF-8643
> > adapters
> > (e.g. https://www.startech.com/HDD/Adapters/pcie-nvme-u2-ssd-
> adapter~PEX4SFF8643).
> > The TX-to-RX wiring seems like it should work, but I have no idea about
> > signal integrity through such a cable.
> >
> > There's also the SuperMicro AOC-SLG3-2E4R, which is similar to the
> > StarTech card but claims to have redriver circuitry.
> >
> > What cabling have folks successfully used for NTB between Intel
> > Xeon-based servers?
> >
> > --Ed
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "linux-ntb" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to linux-ntb+unsubscribe@googlegroups.com
> > <mailto:linux-ntb+unsubscribe@googlegroups.com>.
> > To post to this group, send email to linux-ntb@googlegroups.com
> > <mailto:linux-ntb@googlegroups.com>.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/linux-ntb/CAO_EM_n81HfXZY-X%2BOuQWX0T%
> 3D0E5BYpfJJEf%2BWQVC1DzyyWkLA%40mail.gmail.com
> > <https://groups.google.com/d/msgid/linux-ntb/CAO_EM_
> n81HfXZY-X%2BOuQWX0T%3D0E5BYpfJJEf%2BWQVC1DzyyWkLA%
> 40mail.gmail.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>

[-- Attachment #2: Type: text/html, Size: 4769 bytes --]