Fwd: IPoIB and STP

Fwd: IPoIB and STP

[Gandalf Corvotempesta]

Hi,
i'm trying to configure a redundant IPoIB network.
Obviously, IB switches doesn't talk IP and so doesn't have STP support.

How can I interconnect two different switches with multiple cables and
avoid loops?

For example:
http://pastebin.com/raw.php?i=w8ySRibG

To have a fully redundant network, I have to interconnect switch1 and
switch2 at least with 2 cables and enable STP to avoit loops

Is OpenSM smart enough to detect a multiple link and shutdown one port
automatically (doing the same as STP) ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IPoIB and STP

[Gennadiy Nerubayev]

On Sat, Dec 28, 2013 at 6:08 AM, Gandalf Corvotempesta
<gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi,
> i'm trying to configure a redundant IPoIB network.
> Obviously, IB switches doesn't talk IP and so doesn't have STP support.
>
> How can I interconnect two different switches with multiple cables and
> avoid loops?
>
> For example:
> http://pastebin.com/raw.php?i=w8ySRibG
>
> To have a fully redundant network, I have to interconnect switch1 and
> switch2 at least with 2 cables and enable STP to avoit loops
>
> Is OpenSM smart enough to detect a multiple link and shutdown one port
> automatically (doing the same as STP) ?

Infiniband uses a switched fabric topology, so regular Ethernet STP
topology rules do not apply. In this scenario here each link, direct
or via the interconnects to the other switch, would just be an
alternate path to the target. If I recall correctly, running OpenSM
only binds to one port, so this is why you'd technically want those
links between the switches to have a single fabric. Of course, losing
the link to that port also means you will have no subnet manager, so
you might want to consider having it running on one more host as
well..

-Gennadiy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IPoIB and STP

[Gandalf Corvotempesta]

OpenSM will be running on both nodes and both node ports will be
bonded together in active-failover (or active-active, if possible). My
issue is with interconnection links between switches.
Last time that i've tried I had issue due to the interface bonding in
active-active.
When using active-active i've seen IPoIB running and just a couple of
KBytes/s and in active-failover was running at 10-11Gbps/s

So, are you suggesting:
OpenSM running on both, node1 and node2
Both nodes connected to both switches (1 port per switch)
Both switches interconnected with 2 or more links to have aggregated bandwidth

This would'nt require any special configuration? How can I ensure the
both link on switches are used together? Any command to run with
OpenSM ?

2013/12/29 Gennadiy Nerubayev <parakie-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> On Sat, Dec 28, 2013 at 6:08 AM, Gandalf Corvotempesta
> <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>
>> Hi,
>> i'm trying to configure a redundant IPoIB network.
>> Obviously, IB switches doesn't talk IP and so doesn't have STP support.
>>
>> How can I interconnect two different switches with multiple cables and
>> avoid loops?
>>
>> For example:
>> http://pastebin.com/raw.php?i=w8ySRibG
>>
>> To have a fully redundant network, I have to interconnect switch1 and
>> switch2 at least with 2 cables and enable STP to avoit loops
>>
>> Is OpenSM smart enough to detect a multiple link and shutdown one port
>> automatically (doing the same as STP) ?
>
> Infiniband uses a switched fabric topology, so regular Ethernet STP
> topology rules do not apply. In this scenario here each link, direct
> or via the interconnects to the other switch, would just be an
> alternate path to the target. If I recall correctly, running OpenSM
> only binds to one port, so this is why you'd technically want those
> links between the switches to have a single fabric. Of course, losing
> the link to that port also means you will have no subnet manager, so
> you might want to consider having it running on one more host as
> well..
>
> -Gennadiy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IPoIB and STP

[Gennadiy Nerubayev]

On Sun, Dec 29, 2013 at 5:43 AM, Gandalf Corvotempesta
<gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> OpenSM will be running on both nodes and both node ports will be
> bonded together in active-failover (or active-active, if possible). My
> issue is with interconnection links between switches.
> Last time that i've tried I had issue due to the interface bonding in
> active-active.
> When using active-active i've seen IPoIB running and just a couple of
> KBytes/s and in active-failover was running at 10-11Gbps/s
>
> So, are you suggesting:
> OpenSM running on both, node1 and node2
> Both nodes connected to both switches (1 port per switch)
> Both switches interconnected with 2 or more links to have aggregated bandwidth
>
> This would'nt require any special configuration? How can I ensure the
> both link on switches are used together? Any command to run with
> OpenSM ?

I haven't had much success and/or advantages with a active/active
bond, and used active/passive instead. One of the reasons for that is
the app I used at a time only did one network connection worth of
traffic, which doesn't gain any speed benefits from active/active. I
don't recall seeing such drastic speed issues with active/active, but
granted it's been a bit since I last worked with it and Infiniband in
general.

-Gennadiy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: IPoIB and STP

[Hal Rosenstock]

On 12/28/2013 6:08 AM, Gandalf Corvotempesta wrote:
> Hi,
> i'm trying to configure a redundant IPoIB network.
> Obviously, IB switches doesn't talk IP and so doesn't have STP support.
> 
> How can I interconnect two different switches with multiple cables and
> avoid loops?
> 
> For example:
> http://pastebin.com/raw.php?i=w8ySRibG
> 
> To have a fully redundant network, I have to interconnect switch1 and
> switch2 at least with 2 cables and enable STP to avoit loops
> 
> Is OpenSM smart enough to detect a multiple link and shutdown one port
> automatically (doing the same as STP) ?

SM doesn't need to shut down redundant links. It can use multiple links
as long as it is required to ensure that there are no cyclic flow
control dependencies. The SM configures this via the switch forwarding
tables and the SL to VL mapping on the ports in the subnet.

-- Hal

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IPoIB and STP

[Hal Rosenstock]

On 12/29/2013 5:43 AM, Gandalf Corvotempesta wrote:
> OpenSM will be running on both nodes and both node ports will be
> bonded together in active-failover (or active-active, if possible). My
> issue is with interconnection links between switches.
> Last time that i've tried I had issue due to the interface bonding in
> active-active.
> When using active-active i've seen IPoIB running and just a couple of
> KBytes/s and in active-failover was running at 10-11Gbps/s

AFAIK IPoIB bonding only supports active/backup and not active/active
(load sharing) mode.

-- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: IPoIB and STP

[Yann Droneaud]

Hi,

Le samedi 28 décembre 2013 à 12:08 +0100, Gandalf Corvotempesta a
écrit :
> Hi,
> i'm trying to configure a redundant IPoIB network.
> Obviously, IB switches doesn't talk IP and so doesn't have STP support.
> 

Beware, you're mixing L3 and L2.

STP, Spanning Tree Protocol (IEEE 802.1D), is a component of Ethernet,
which is Layer-2/Link layer. It does not involve IP, which is
Layer-3/Network layer. IP is not bound to Ethernet, indeed.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: IPoIB and STP

[Gandalf Corvotempesta]

2014/1/2 Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>:
> STP, Spanning Tree Protocol (IEEE 802.1D), is a component of Ethernet,
> which is Layer-2/Link layer. It does not involve IP, which is
> Layer-3/Network layer. IP is not bound to Ethernet, indeed.

You are right, mine was just an example to explain what I'm trying to do.

So, I can safely link both switches with multiple cables and OpenSM
will automatically use both link for aggregated bandwidth and fault
tollerance without any additional configuration?

In this case, a simple IPoIB network , with active-standby bonded
interfaces and multiple links between switches gives me HA, Load
Sharing and IP
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html