Dual star topology

Dual star topology

[Gandalf Corvotempesta]

Hi to all
i'm probably OT but I don't know where to ask
I'm searching for some advice creating a dual-star topology to get
full path redundancy.
I have one dual-port DDR card on each server and two switches.
I'll connect one port to each switch, but should I also interconnect
both switches like in a standard ethernet network ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Hal Rosenstock]

Hi Gandalf,

On 7/24/2013 11:20 AM, Gandalf Corvotempesta wrote:
> Hi to all
> i'm probably OT but I don't know where to ask
> I'm searching for some advice creating a dual-star topology to get
> full path redundancy.
> I have one dual-port DDR card on each server and two switches.
> I'll connect one port to each switch, but should I also interconnect
> both switches like in a standard ethernet network ?

The most fully redundant model would be to _not_ interconnect the 2
switches so it's 2 IB subnets rather than a single subnet. This means
you need at least 1 SM on each subnet (with different subnet prefix) but
you might want more if you're worried about some SM failing and not
having an SM on that subnet.

-- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Fwd: Dual star topology

[Gandalf Corvotempesta]

in this way all services should be dual stack with at least two
difrent addresses, one for each infiniband subnets.

I have to configure ceph on these subnets and ceph doesn't allow to
set multiple addresses for each service.

Il giorno 24/lug/2013 20:04, "Hal Rosenstock" <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
ha scritto:

> Hi Gandalf,
>
> On 7/24/2013 11:20 AM, Gandalf Corvotempesta wrote:
> > Hi to all
> > i'm probably OT but I don't know where to ask
> > I'm searching for some advice creating a dual-star topology to get
> > full path redundancy.
> > I have one dual-port DDR card on each server and two switches.
> > I'll connect one port to each switch, but should I also interconnect
> > both switches like in a standard ethernet network ?
>
> The most fully redundant model would be to _not_ interconnect the 2
> switches so it's 2 IB subnets rather than a single subnet. This means
> you need at least 1 SM on each subnet (with different subnet prefix) but
> you might want more if you're worried about some SM failing and not
> having an SM on that subnet.
>
> -- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Gandalf Corvotempesta]

2013/7/24 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> I have to configure ceph on these subnets and ceph doesn't allow to
> set multiple addresses for each service.

Let me try to explain in a better way.
I would like to create a ceph cluster over an infiniband network.
Each server has a signle dual-port HBA.
Ceph is running with a *single* IP address on each server.

In a standard IP network, I have to interconnect both switches or i'll
loose some traffic in case of a single port failure:

server1.port1      ib switch 1 <--> server2.port1
server1.port2 <-> ib switch 2


in this case, server1 will not be able to reach server2, because of split brain.
An interconnection between both switches will solve this in a standard
IP network.

How can I archieve this in an infiniband network?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Hal Rosenstock]

On 7/24/2013 4:56 PM, Gandalf Corvotempesta wrote:
> 2013/7/24 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> I have to configure ceph on these subnets and ceph doesn't allow to
>> set multiple addresses for each service.
> 
> Let me try to explain in a better way.
> I would like to create a ceph cluster over an infiniband network.
> Each server has a signle dual-port HBA.
> Ceph is running with a *single* IP address on each server.
> 
> In a standard IP network, I have to interconnect both switches or i'll
> loose some traffic in case of a single port failure:
> 
> server1.port1      ib switch 1 <--> server2.port1
> server1.port2 <-> ib switch 2
> 
> 
> in this case, server1 will not be able to reach server2, because of split brain.
> An interconnection between both switches will solve this in a standard
> IP network.

So does ceph run on top of IP ? If so, could you use IPoIB bonding (and
interconnect the switches with some number of links) ?

-- Hal

> How can I archieve this in an infiniband network?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Fwd: Dual star topology

[Gandalf Corvotempesta]

IPoIB is very slow, I prefere to try with rsockets and the preloader library.

Il giorno 24/lug/2013 23:18, "Hal Rosenstock" <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
ha scritto:

> On 7/24/2013 4:56 PM, Gandalf Corvotempesta wrote:
> > 2013/7/24 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> >> I have to configure ceph on these subnets and ceph doesn't allow to
> >> set multiple addresses for each service.
> >
> > Let me try to explain in a better way.
> > I would like to create a ceph cluster over an infiniband network.
> > Each server has a signle dual-port HBA.
> > Ceph is running with a *single* IP address on each server.
> >
> > In a standard IP network, I have to interconnect both switches or i'll
> > loose some traffic in case of a single port failure:
> >
> > server1.port1      ib switch 1 <--> server2.port1
> > server1.port2 <-> ib switch 2
> >
> >
> > in this case, server1 will not be able to reach server2, because of split brain.
> > An interconnection between both switches will solve this in a standard
> > IP network.
>
> So does ceph run on top of IP ? If so, could you use IPoIB bonding (and
> interconnect the switches with some number of links) ?
>
> -- Hal
>
> > How can I archieve this in an infiniband network?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Dual star topology

[Hefty, Sean]

> IPoIB is very slow, I prefere to try with rsockets and the preloader library.

Currently, rsockets cannot fail over between local HCA ports.  This is an implementation restriction resulting from no automatic path migration support.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Gandalf Corvotempesta]

2013/7/25 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>:
> Currently, rsockets cannot fail over between local HCA ports.  This is an implementation restriction resulting from no automatic path migration support.

So, currently there is not way to have a fully redundant IB network
with dual-port HCA ?

In case of IPoIB, how can I use multiple ports for interconnecting
both switches? In a standard IP network I have to bond them toghether
to avoid loops. In IB should I do the same ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Hal Rosenstock]

On 7/26/2013 3:22 AM, Gandalf Corvotempesta wrote:
> 2013/7/25 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>:
>> Currently, rsockets cannot fail over between local HCA ports.  This is an implementation restriction resulting from no automatic path migration support.
> 
> So, currently there is not way to have a fully redundant IB network
> with dual-port HCA ?
> 
> In case of IPoIB, how can I use multiple ports for interconnecting
> both switches? In a standard IP network I have to bond them toghether
> to avoid loops. In IB should I do the same ?

There is support for IPoIB bonding.

-- Hal

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Gandalf Corvotempesta]

2013/7/26 Hal Rosenstock <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>:
> There is support for IPoIB bonding.

I'm talking about switch interconnection. How can I create bonding on switches?
Any recent benchmark about IPoIB ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Hal Rosenstock]

On 7/26/2013 8:20 AM, Gandalf Corvotempesta wrote:
> 2013/7/26 Hal Rosenstock <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>:
>> There is support for IPoIB bonding.
> 
> I'm talking about switch interconnection. How can I create bonding on switches?

The SM reroutes based on links coming and going so if there are multiple
links between switches, this addresses the internal link failover scenario.

-- Hal

> Any recent benchmark about IPoIB ?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Gandalf Corvotempesta]

2013/7/26 Hal Rosenstock <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>:
> The SM reroutes based on links coming and going so if there are multiple
> links between switches, this addresses the internal link failover scenario.

Ok, so, IPoIB across bonded ports on each host.
Multiple links between switches managed automatically by SM.
Will these multiple links be used in parallel by aggregating bandwidth
or just in case of failover ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Hal Rosenstock]

On 7/26/2013 9:51 AM, Gandalf Corvotempesta wrote:
> Ok, so, IPoIB across bonded ports on each host.
> Multiple links between switches managed automatically by SM.
> Will these multiple links be used in parallel by aggregating bandwidth
> or just in case of failover ?

How paths are routed is an internal policy of the SM. In terms of
OpenSM, paths are balanced across multiple internal switch to switch
links so they're used in parallel in that sense.

Note that in terms of IPoIB bonding, I think it's an active/standby
rather than active/active model.

-- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Gandalf Corvotempesta]

2013/7/26 Hal Rosenstock <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>:
> Note that in terms of IPoIB bonding, I think it's an active/standby
> rather than active/active model.

Why? Can't I create an active/active bonded interface ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dual star topology

[Gandalf Corvotempesta]

2013/7/25 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>:
> Currently, rsockets cannot fail over between local HCA ports.  This is an implementation restriction resulting from no automatic path migration support.

This is still unclear.
>From the README (
http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=blob;f=README;h=e1f222740144ed8f7d8bc935ea6643355b077bcd;hb=refs/heads/master
) I can read:
"Using multiple interfaces
  The librdmacm does support multiple interfaces."

so, multiple interfaces are supported. Why I can't use both port? Each
port is threathed as a single interface.

Next question: rsockets addressing is done by IPoIB ? How can I reach
a remote IP address without IPoIB ?
If yes, I think that I could use a bonded IPoIB interface to have an
"always-on" path.

English is not my primary language, so, i repeat: today, is not
possible to use librdma_cm in a fault tollerance architecture ? I
don't need bandwidth aggregation, but just a fault tollerance across 2
switches.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Dual star topology

[Hefty, Sean]

> This is still unclear.
> From the README (
> http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=blob;f=README;h=e1f222
> 740144ed8f7d8bc935ea6643355b077bcd;hb=refs/heads/master
> ) I can read:
> "Using multiple interfaces
>   The librdmacm does support multiple interfaces."
> 
> so, multiple interfaces are supported. Why I can't use both port? Each
> port is threathed as a single interface.

You can use both ports, but you need 2 different connections to do so.
 
> Next question: rsockets addressing is done by IPoIB ? How can I reach
> a remote IP address without IPoIB ?

You cannot with the current release.  I have patches that will allow this, which will be available within a week or so.  But those will not be in a release until later.

> If yes, I think that I could use a bonded IPoIB interface to have an
> "always-on" path.
> 
> English is not my primary language, so, i repeat: today, is not
> possible to use librdma_cm in a fault tollerance architecture ? I
> don't need bandwidth aggregation, but just a fault tollerance across 2
> switches.

You can establish a single connection.  If a switch to switch port fails, the SM may (should?) be able to fix the routes, so that the connection continues to work.  The issue is whether the SM can reroute around the issue faster than the connection can detect the failure.  The connection itself will not have failover capabilities -- meaning that automatic path migration (APM - see the IB spec) is not supported.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html