[PATCH 0/20 v2] rdma: Add XRC support

[PATCH 0/20 v2] rdma: Add XRC support

[Hefty, Sean]

XRC provides a scalability enhancement when a process on one node
must communicate with multiple processes on another node.  This
is commonly the case when running MPI applications on multi-core
systems in a cluster.

An XRC connection consists of an initiator (XRC INI) qp and a target
(XRC TGT) qp.  A single xrc ini -> xrc tgt qp connection enables a
process to communicate with all remote processes on a single node.
Security is enforced through the use of a special type of protection
domain and shared receive queues.  An xrc srq is allocated
on an xrc domain.  The xrc tgt qp is also allocated on an xrc domain.
A process may receive data on an xrc srq from a tgt qp if the xrc
domains are the same.

The following patch series adds support for XRC.  The patches used
the patches found in Roland's xrc branch as a base, plus added in
patches found in the OFED 1.5 linux git tree.  The latter were mainly
used for updating the mlx4 driver.  I broke the patches up, so that
I could follow each change better.

This set differs from Roland's xrc branch in that XRC initiator and
target QP types are separated, and a type is specified when creating
an SRQ.  The patches differ from the OFED 1.5 patches by using the
existing QP APIs and ABI to create and manage XRC QPs.  XRCD's may
still be shared among multiple processes, and XRC QPs may be connected
out of band.

There are corresponding patches to libibverbs, libmlx4, and the
librdmacm to support XRC.

This patch sets are also available on the ofa server:

	git://git.openfabrics.org/~shefty/rdma-dev.git   xrc
	git://git.openfabrics.org/~shefty/libibverbs.git xrc
	git://git.openfabrics.org/~shefty/libmlx4.git    xrc
	git://git.openfabrics.org/~shefty/librdmacm.git  xrc

Changes from v1:
- Add ability for any process using an xrcd to open an existing tgt qp
- Provided update to ib_cm changes based on proposed errata.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/20 v2] rdma: Add XRC support

[Roland Dreier]

Great stuff!  I'd really like to get this into kernel 3.2... do you have any
open issues or is it all good from your POV?

 - R.

On Thu, Aug 18, 2011 at 7:14 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> XRC provides a scalability enhancement when a process on one node
> must communicate with multiple processes on another node.  This
> is commonly the case when running MPI applications on multi-core
> systems in a cluster.
>
> An XRC connection consists of an initiator (XRC INI) qp and a target
> (XRC TGT) qp.  A single xrc ini -> xrc tgt qp connection enables a
> process to communicate with all remote processes on a single node.
> Security is enforced through the use of a special type of protection
> domain and shared receive queues.  An xrc srq is allocated
> on an xrc domain.  The xrc tgt qp is also allocated on an xrc domain.
> A process may receive data on an xrc srq from a tgt qp if the xrc
> domains are the same.
>
> The following patch series adds support for XRC.  The patches used
> the patches found in Roland's xrc branch as a base, plus added in
> patches found in the OFED 1.5 linux git tree.  The latter were mainly
> used for updating the mlx4 driver.  I broke the patches up, so that
> I could follow each change better.
>
> This set differs from Roland's xrc branch in that XRC initiator and
> target QP types are separated, and a type is specified when creating
> an SRQ.  The patches differ from the OFED 1.5 patches by using the
> existing QP APIs and ABI to create and manage XRC QPs.  XRCD's may
> still be shared among multiple processes, and XRC QPs may be connected
> out of band.
>
> There are corresponding patches to libibverbs, libmlx4, and the
> librdmacm to support XRC.
>
> This patch sets are also available on the ofa server:
>
>        git://git.openfabrics.org/~shefty/rdma-dev.git   xrc
>        git://git.openfabrics.org/~shefty/libibverbs.git xrc
>        git://git.openfabrics.org/~shefty/libmlx4.git    xrc
>        git://git.openfabrics.org/~shefty/librdmacm.git  xrc
>
> Changes from v1:
> - Add ability for any process using an xrcd to open an existing tgt qp
> - Provided update to ib_cm changes based on proposed errata.
>
> Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/20 v2] rdma: Add XRC support

[Hefty, Sean]

> Great stuff!  I'd really like to get this into kernel 3.2... do you have any
> open issues or is it all good from your POV?

The only issue from my viewpoint is whether the change referenced by patch 11 (ib/cm: Update xrc support based on xrc annex errata) has been approved by the IBTA.  I'm waiting on a response on that.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/20 v2] rdma: Add XRC support

[Hefty, Sean]

> The only issue from my viewpoint is whether the change referenced by patch 11
> (ib/cm: Update xrc support based on xrc annex errata) has been approved by the
> IBTA.  I'm waiting on a response on that.

I received confirmation that the errata to the XRC Annex was approved.  Both patch 10 (ib/cm: Update protocol to support XRC) and 11 (ib/cm: Update xrc support based on xrc annex errata) should be merged.  You can either keep them separate, combine them into a single patch, or I can resubmit a combined patch.

Let me know if you see any other issues with the series.

Thanks,
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/20 v2] rdma: Add XRC support

[Hefty, Sean]

Roland,

Can I get a quick status update regarding this series?

Thanks,
Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/20 v2] rdma: Add XRC support

[Roland Dreier]

Sorry, haven't been very productive, partially due to patchwork being
down (along with the rest of kernel.org).
So not much to report...

On Wed, Sep 14, 2011 at 2:00 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Roland,
>
> Can I get a quick status update regarding this series?
>
> Thanks,
> Sean
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/20 v2] rdma: Add XRC support

[Steve Wise]

On 09/14/2011 05:01 PM, Roland Dreier wrote:
> Sorry, haven't been very productive, partially due to patchwork being
> down (along with the rest of kernel.org).
> So not much to report...
>

Is there an ETA on kernel.org getting healthy again?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/20 v2] rdma: Add XRC support

[Roland Dreier]

On Wed, Sep 14, 2011 at 3:15 PM, Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> wrote:
> Is there an ETA on kernel.org getting healthy again?

None that I've heard.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/20 v2] rdma: Add XRC support

[Or Gerlitz]

On 9/15/2011 1:01 AM, Roland Dreier wrote:
> Sorry, haven't been very productive, partially due to patchwork being
> down (along with the rest of kernel.org). So not much to report...
>

Roland, Sean was saying (SB) that he has them on the ofa git, so in case 
you have
a local clone, maybe you can start get things going even before 
kernel.org is back?

Or.

> There are corresponding patches to libibverbs, libmlx4, and the
> librdmacm to support XRC.
>
> This patch sets are also available on the ofa server:
>
> 	git://git.openfabrics.org/~shefty/rdma-dev.git   xrc
> 	git://git.openfabrics.org/~shefty/libibverbs.git xrc
> 	git://git.openfabrics.org/~shefty/libmlx4.git    xrc
> 	git://git.openfabrics.org/~shefty/librdmacm.git  xrc
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html