rsockets addressing

rsockets addressing

[Gandalf Corvotempesta]

Hi to all,
which kind of system is used by rsockets to address the remote host?
Is using IPoIB ?

Will I be able to support two redundant fabrics with failover managed
by OpenSM with rsockets ?
For example, two nodes connected (with 2 HBA on each) to two different
IB fabrics.
In case of an HBA failure, will rsocket be able to re-establish a
connected using the second fabric ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets addressing

[Hefty, Sean]

> which kind of system is used by rsockets to address the remote host?
> Is using IPoIB ?

rsockets is built over the RDMA CM.  The rdma cm uses ipoib indirectly when connecting over IB, by mapping the IP addresses to GIDs.

rsockets can also support using native IB addresses, though this is a new feature which requires the latest kernel and upstream, but unreleased rsockets code.
 
> Will I be able to support two redundant fabrics with failover managed
> by OpenSM with rsockets ?
> For example, two nodes connected (with 2 HBA on each) to two different
> IB fabrics.
> In case of an HBA failure, will rsocket be able to re-establish a
> connected using the second fabric ?

rsockets does not implement failover.  An application would need to reestablish a connection in the case of a failure.  I have not looked to see what it would take to implement failover inside rsockets, and that's not something I would have time to implement anytime soon.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Gandalf Corvotempesta]

2013/9/16 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>:
> rsockets does not implement failover.  An application would need to reestablish a connection in the case of a failure.  I have not looked to see what it would take to implement failover inside rsockets, and that's not something I would have time to implement anytime soon.

Connection restablishement is ok, but in case of port failure, ipoib
will remap the remote IP against the new port/fabric ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Yann Droneaud]

Le 16.09.2013 17:59, Gandalf Corvotempesta a écrit :
> 2013/9/16 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>:
>> rsockets does not implement failover.  An application would need to 
>> reestablish a connection in the case of a failure.  I have not looked 
>> to see what it would take to implement failover inside rsockets, and 
>> that's not something I would have time to implement anytime soon.
> 
> Connection restablishement is ok, but in case of port failure, ipoib
> will remap the remote IP against the new port/fabric ?

You have to use InfiniBand port bonding to benefit from fail-over 
(through APM: Automatic Path Migration ?)

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt

Regards.

-- 
Yann Droneaud
OPTEYA

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Gandalf Corvotempesta]

2013/9/16 Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>:
> You have to use InfiniBand port bonding to benefit from fail-over (through
> APM: Automatic Path Migration ?)
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt

IB boinding is exactly what I would like to do but I have some questions:

1. is the ISL supported? I've seen that is not suggested to
interconnect two IB switches (and I don't know why)
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt#n1967
Without ISL and using two different IB switches, one port failure will
result in the whole node down because there is no other active path to
the primary switch
This would be good in my environment by I prefere to get a real failover.

2. can I use active-active mode like standard ethernet bonding with
some sort of balancing?
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt#n2127
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Yann Droneaud]

Hi,

Le lundi 16 septembre 2013 à 18:31 +0200, Gandalf Corvotempesta a
écrit :
> 2013/9/16 Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>:
> > You have to use InfiniBand port bonding to benefit from fail-over (through
> > APM: Automatic Path Migration ?)
> >
> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt
> 
> IB boinding is exactly what I would like to do but I have some questions:
> 
> 1. is the ISL supported? I've seen that is not suggested to
> interconnect two IB switches (and I don't know why)
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt#n1967
> Without ISL and using two different IB switches, one port failure will
> result in the whole node down because there is no other active path to
> the primary switch
> This would be good in my environment by I prefere to get a real failover.
> 

The InfiniBand fabrics support "ISL" ... indeed.

InfiniBand fabrics support more complicated network topology than
(plain) Ethernet: that's why there's (at least) one Subnet Manager,
while Ethernet struggle with Spanning Tree ...

You definitely need link between switches if you want to use a "high
availability" fabric topology with HCA ports connected to differents
switches. (additionally, a switch should have a link to two others
switches ... then it's starting to be complicated, since you have to
design your fabric topology to match the communication pattern / data
locality used by your application ...).

> 2. can I use active-active mode like standard ethernet bonding with
> some sort of balancing?
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/bonding.txt#n2127

AFAIK, IPoIB bonding doesn't support "active-active" bonding.
Only high availability is provided. Link aggregate is not supported.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Gandalf Corvotempesta]

2013/9/16 Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>:
> The InfiniBand fabrics support "ISL" ... indeed.
[cut]
> You definitely need link between switches if you want to use a "high
> availability" fabric topology with HCA ports connected to differents
> switches. (additionally, a switch should have a link to two others
> switches ... then it's starting to be complicated, since you have to
> design your fabric topology to match the communication pattern / data
> locality used by your application ...).

I've read somewhere that this is not suggested because issues on one
switch could affect also the second switch, but doing so will allow me
to use both port in hot-standby failover.

In one port fails, IPoIB bonding will switch on the second port and at
the same time , traffic will be routed trough the ISL link

Without the ISL, a port failure will bring down the whole node.

So:
 - switch1 conntected to switch2
 - node1 connected to both switches
 - node2 connected to both switches
 - IPoIB on each node with active-passive bonding for each IB port.

How can I create a redundant ISL ? Should I connect two or more port and
the subnet manager will automatically take care of this or I have to configure
something like STP on plain ethernet networks.
Will the ISL be used in loadbalancing in case of 2 or more cables connected ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Yann Droneaud]

Le 16.09.2013 22:41, Gandalf Corvotempesta a écrit :
> 2013/9/16 Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>:
>> The InfiniBand fabrics support "ISL" ... indeed.
> [cut]
>> You definitely need link between switches if you want to use a "high
>> availability" fabric topology with HCA ports connected to differents
>> switches. (additionally, a switch should have a link to two others
>> switches ... then it's starting to be complicated, since you have to
>> design your fabric topology to match the communication pattern / data
>> locality used by your application ...).
> 
> I've read somewhere that this is not suggested because issues on one
> switch could affect also the second switch, but doing so will allow me
> to use both port in hot-standby failover.
> 

Are talking about Ethernet network or InfiniBand Fabric ?

> In one port fails, IPoIB bonding will switch on the second port and at
> the same time , traffic will be routed trough the ISL link
> 

It's "routed" through the fabric.

> Without the ISL, a port failure will bring down the whole node.
> 
> So:
>  - switch1 conntected to switch2
>  - node1 connected to both switches
>  - node2 connected to both switches
>  - IPoIB on each node with active-passive bonding for each IB port.
> 
> How can I create a redundant ISL ? Should I connect two or more port 
> and
> the subnet manager will automatically take care of this or I have to 
> configure
> something like STP on plain ethernet networks.

In case of InfiniBand, the subnet manager should take care of it.

> Will the ISL be used in loadbalancing in case of 2 or more cables 
> connected ?

It should depend on the topology chosen for the fabric. You might ask to 
someone else for this.

Regards.

-- 
Yann Droneaud
OPTEYA

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets addressing

[Gandalf Corvotempesta]

2013/9/17 Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org>:
> Are talking about Ethernet network or InfiniBand Fabric ?

IB fabric

> In case of InfiniBand, the subnet manager should take care of it.

So, interconnecting 2 IB switches is good.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html