APM support for IPoIB

APM support for IPoIB

[Bang Nguyen]

  Hi,

Adding APM support for IPoIB seems like something worthwhile and we're 
thinking about it..but are wondering why has hasn't been done 
already..any comments/suggestions?

Thanks, --Bang.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Jason Gunthorpe]

On Mon, Oct 01, 2012 at 03:23:38PM -0700, Bang Nguyen wrote:

> Adding APM support for IPoIB seems like something worthwhile and
> we're thinking about it..but are wondering why has hasn't been done
> already..any comments/suggestions?

I've been quite supportive of APM in the past, IMHO I think it is
something unique to IB that should be exploited as often as possible.

Improving the RDMA-CM to setup APM at all would be a great first step,
but some SM work is also ultimately needed to get sufficiently
independent/redundant paths.

I would like to see the full APM support eventually implemented - this
would be fail over across adaptor ports, where each port is on a
seperate subnet - so you get full redundancy of cabling, switching and
management software.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Bang Nguyen]

  On 10/1/2012 4:53 PM, Jason Gunthorpe wrote:
> On Mon, Oct 01, 2012 at 03:23:38PM -0700, Bang Nguyen wrote:
>
>> Adding APM support for IPoIB seems like something worthwhile and
>> we're thinking about it..but are wondering why has hasn't been done
>> already..any comments/suggestions?
> I've been quite supportive of APM in the past, IMHO I think it is
> something unique to IB that should be exploited as often as possible.
>
> Improving the RDMA-CM to setup APM at all would be a great first step,
> but some SM work is also ultimately needed to get sufficiently
> independent/redundant paths.
Our version of OFED has full support of APM in RDMA-CM and it's being 
used by SDP and RDS..
IPoIB is using CM instead of RDMA CM so we can not leverage it..IPoIB 
APM will need to have its own implementation..
> I would like to see the full APM support eventually implemented - this
> would be fail over across adaptor ports, where each port is on a
> seperate subnet - so you get full redundancy of cabling, switching and
> management software.
>
> Jason

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Roland Dreier]

On Mon, Oct 1, 2012 at 5:02 PM, Bang Nguyen <bang.nguyen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> Our version of OFED has full support of APM in RDMA-CM and it's being used
> by SDP and RDS..

Perhaps working on getting that merged upstream would be a better first step.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: APM support for IPoIB

[Tziporet Koren]

> Perhaps working on getting that merged upstream would be a better first step.

We have it on our plans to submit RDMA-CM APM to upstream.
Since we are in holidays season here it will take us some time to come back with plan.

In any case as Bang wrote this is not related to IPoIB-CM, that requires a different implementation, and any one is welcome to contribute here :-)

Tziporet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Bang Nguyen]

  On 10/2/2012 1:12 AM, Tziporet Koren wrote:
>> Perhaps working on getting that merged upstream would be a better first step.
> We have it on our plans to submit RDMA-CM APM to upstream.
> Since we are in holidays season here it will take us some time to come back with plan.
>
> In any case as Bang wrote this is not related to IPoIB-CM, that requires a different implementation, and any one is welcome to contribute here :-)
>
> Tziporet
>
Just a crazy thought but IPoIB could use RDMA-CM for all of its 
connection management stuff..this way it can leverage the APM support in 
RDMA-CM..

comments?...

Thanks, --Bang.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: APM support for IPoIB

[Hefty, Sean]

> Just a crazy thought but IPoIB could use RDMA-CM for all of its
> connection management stuff..this way it can leverage the APM support in
> RDMA-CM..
> 
> comments?...

ipoib predates the rdma cm, which is why it's written to the ib_cm directly.  Communication setup with the rdma cm follows these 3 steps.

1. address resolution - The rdma cm indirectly ends up using ipoib for this.  With ipoib, this step isn't necessary.  I think having AF_IB support would help this by avoiding this step, plus it restricts the rdma cm to using IB devices.
2. route resolution (aka path record query) - Ipoib maintains its own path record cache.  It can set the rdma cm paths manually, or there would need to be some sort of rework.
3. connection setup - formats the ib cm private data and exchanges messages.  The rdma cm would need to know the ipoib private data format.

Sooo... ipoib wouldn't use address resolution.  It maintains its own path record cache, which is a significant piece of supporting APM.  Ipoib probably doesn't need to maintain its own cache, but there's no decent alternative for kernel clients currently.  And connection setup using the rdma cm is only slightly easier than using the ib cm.  (I have no idea what APM support in the rdma cm looks like.)

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Bang Nguyen]

  Here is the presentation on RDMA-CM APM

https://www.openfabrics.org/ofa-documents/presentations/doc_download/516-rdmacm-amp.html

On 10/2/2012 8:03 PM, Hefty, Sean wrote:
>> Just a crazy thought but IPoIB could use RDMA-CM for all of its
>> connection management stuff..this way it can leverage the APM support in
>> RDMA-CM..
>>
>> comments?...
> ipoib predates the rdma cm, which is why it's written to the ib_cm directly.  Communication setup with the rdma cm follows these 3 steps.
>
> 1. address resolution - The rdma cm indirectly ends up using ipoib for this.  With ipoib, this step isn't necessary.  I think having AF_IB support would help this by avoiding this step, plus it restricts the rdma cm to using IB devices.
> 2. route resolution (aka path record query) - Ipoib maintains its own path record cache.  It can set the rdma cm paths manually, or there would need to be some sort of rework.
> 3. connection setup - formats the ib cm private data and exchanges messages.  The rdma cm would need to know the ipoib private data format.
>
> Sooo... ipoib wouldn't use address resolution.  It maintains its own path record cache, which is a significant piece of supporting APM.  Ipoib probably doesn't need to maintain its own cache, but there's no decent alternative for kernel clients currently.  And connection setup using the rdma cm is only slightly easier than using the ib cm.  (I have no idea what APM support in the rdma cm looks like.)
>
> - Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Bang Nguyen]

  I'm experimenting with trying to model IPoIB-CM APM after RDMA-CM APM, 
particularly the protocol for loading/re-loading of the alternate path 
and will see how it goes..

Thanks, --Bang.

On 10/2/2012 9:24 PM, Bang Nguyen wrote:
>  Here is the presentation on RDMA-CM APM
>
> https://www.openfabrics.org/ofa-documents/presentations/doc_download/516-rdmacm-amp.html 
>
>
> On 10/2/2012 8:03 PM, Hefty, Sean wrote:
>>> Just a crazy thought but IPoIB could use RDMA-CM for all of its
>>> connection management stuff..this way it can leverage the APM 
>>> support in
>>> RDMA-CM..
>>>
>>> comments?...
>> ipoib predates the rdma cm, which is why it's written to the ib_cm 
>> directly.  Communication setup with the rdma cm follows these 3 steps.
>>
>> 1. address resolution - The rdma cm indirectly ends up using ipoib 
>> for this.  With ipoib, this step isn't necessary.  I think having 
>> AF_IB support would help this by avoiding this step, plus it 
>> restricts the rdma cm to using IB devices.
>> 2. route resolution (aka path record query) - Ipoib maintains its own 
>> path record cache.  It can set the rdma cm paths manually, or there 
>> would need to be some sort of rework.
>> 3. connection setup - formats the ib cm private data and exchanges 
>> messages.  The rdma cm would need to know the ipoib private data format.
>>
>> Sooo... ipoib wouldn't use address resolution.  It maintains its own 
>> path record cache, which is a significant piece of supporting APM.  
>> Ipoib probably doesn't need to maintain its own cache, but there's no 
>> decent alternative for kernel clients currently.  And connection 
>> setup using the rdma cm is only slightly easier than using the ib 
>> cm.  (I have no idea what APM support in the rdma cm looks like.)
>>
>> - Sean
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: APM support for IPoIB

[Paul Grun]

Jason - Like your idea for full APM support especially across subnets, but
wouldn't that require some sort capability that lives above the subnet level
and is able to query both SMs/SAs?  Sounds vaguely router-ish?
-Paul

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jason Gunthorpe
> Sent: Monday, October 01, 2012 4:53 PM
> To: Bang Nguyen
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Richard Frank
> Subject: Re: APM support for IPoIB
> 
> On Mon, Oct 01, 2012 at 03:23:38PM -0700, Bang Nguyen wrote:
> 
> > Adding APM support for IPoIB seems like something worthwhile and we're
> > thinking about it..but are wondering why has hasn't been done
> > already..any comments/suggestions?
> 
> I've been quite supportive of APM in the past, IMHO I think it is
> something unique to IB that should be exploited as often as possible.
> 
> Improving the RDMA-CM to setup APM at all would be a great first step, but
> some SM work is also ultimately needed to get sufficiently
> independent/redundant paths.
> 
> I would like to see the full APM support eventually implemented - this
> would be fail over across adaptor ports, where each port is on a seperate
> subnet - so you get full redundancy of cabling, switching and management
> software.
> 
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: APM support for IPoIB

[Jason Gunthorpe]

On Tue, Oct 02, 2012 at 09:53:17AM -0700, Paul Grun wrote:

> Jason - Like your idea for full APM support especially across
> subnets, but wouldn't that require some sort capability that lives
> above the subnet level and is able to query both SMs/SAs?  Sounds
> vaguely router-ish?

Well, in terms of the current spec, PRs are fully specified enough
that a sufficiently informed SA can return PRs for a non-requesting
port on the same node. I've long advocated for an improved PR
mechanism that would be more APM 'aware' and router compatible
(Connected Path Query)

There are a few good ways to get that sort of information into the
SA..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: APM support for IPoIB

[Hefty, Sean]

> Jason - Like your idea for full APM support especially across subnets, but
> wouldn't that require some sort capability that lives above the subnet level
> and is able to query both SMs/SAs?  Sounds vaguely router-ish?

I'm guessing that it would require that the ports involved have different GIDs and coordinate the partition keys, but I'm not sure anything else is needed.  The GIDs and PKeys could be managed manually by an administrator.

A service could query both SAs for PRs, then pass them in a single CM message to the remote side.  Since the SAs are on different subnets, no matter which paths were selected, they would automatically by disjoint, so I don't believe any other coordination is required.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html