[PATCH] rdma/core: Really export ib_open_qp() to user space

[PATCH] rdma/core: Really export ib_open_qp() to user space

[Hefty, Sean]

We need to add an entry into the uverbs and device command
tables to allow user space to actually call ib_open_qp.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
If possible, this should just be merged with the last patch in the XRC
series.

In my previous tests, this was not getting called.  I either did not
have enough MPI processes or the right MPI ranks running on a single
node to invoke this routine.  Once I bumped up the number of MPI
processes, I did see ib_open_qp() being called, failing, and MPI
aborting.  (The OCU mvapich2 alltoall benchmark with 4 processes all on
the same node can test this.)

I pulled the xrc patches in your for-next branch and ran some simple
tests against it.  Between the last time I tested XRC and now, I'm now
seeing mvapich2 hang during MPI finalize, which I'm debugging.  I can
use ctrl-c to abort the MPI job, and the system appears fine after.  The
librdmacm and ibverbs XRC tests are working fine.

 drivers/infiniband/core/uverbs_main.c |    3 ++-
 drivers/infiniband/hw/mlx4/main.c     |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index 9c877e2..485ea6c 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -110,7 +110,8 @@ static ssize_t (*uverbs_cmd_table[])(struct ib_uverbs_file *file,
 	[IB_USER_VERBS_CMD_DESTROY_SRQ]		= ib_uverbs_destroy_srq,
 	[IB_USER_VERBS_CMD_OPEN_XRCD]		= ib_uverbs_open_xrcd,
 	[IB_USER_VERBS_CMD_CLOSE_XRCD]		= ib_uverbs_close_xrcd,
-	[IB_USER_VERBS_CMD_CREATE_XSRQ]		= ib_uverbs_create_xsrq
+	[IB_USER_VERBS_CMD_CREATE_XSRQ]		= ib_uverbs_create_xsrq,
+	[IB_USER_VERBS_CMD_OPEN_QP]		= ib_uverbs_open_qp
 };
 
 static void ib_uverbs_add_one(struct ib_device *device);
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 9cfa6f9..bc47d17 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -1098,7 +1098,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev)
 		(1ull << IB_USER_VERBS_CMD_MODIFY_SRQ)		|
 		(1ull << IB_USER_VERBS_CMD_QUERY_SRQ)		|
 		(1ull << IB_USER_VERBS_CMD_DESTROY_SRQ)		|
-		(1ull << IB_USER_VERBS_CMD_CREATE_XSRQ);	
+		(1ull << IB_USER_VERBS_CMD_CREATE_XSRQ)		|
+		(1ull << IB_USER_VERBS_CMD_OPEN_QP);
 
 	ibdev->ib_dev.query_device	= mlx4_ib_query_device;
 	ibdev->ib_dev.query_port	= mlx4_ib_query_port;


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] rdma/core: Really export ib_open_qp() to user space

[Hefty, Sean]

> I pulled the xrc patches in your for-next branch and ran some simple
> tests against it.  Between the last time I tested XRC and now, I'm now
> seeing mvapich2 hang during MPI finalize, which I'm debugging.

Just an update: the issues that I was seeing were caused by missing patches in my libraries (one in libmlx4 and the other in the ofed compatibility layer).  The XRC patches in for-next are testing out fine for me, though it would be good if someone from Mellanox could try running some tests.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma/core: Really export ib_open_qp() to user space

[Or Gerlitz]

On 10/14/2011 10:22 PM, Hefty, Sean wrote:
> Just an update: the issues that I was seeing were caused by missing 
> patches in my libraries (one in libmlx4 and the other in the ofed 
> compatibility layer). The XRC patches in for-next are testing out fine 
> for me, though it would be good if someone from Mellanox could try 
> running some tests

Hi Sean,

I pulled for-next and made some basic IB tests with it, looks okay, 
specifically I used
ipoib connected mode and ibv_srq_pingpoing (libibverbs' srq test) to 
make sure non XRC
SRQs are working fine.

So what else would you suggest for further testing, is that pulling
the xrc branch of your ofa hosted librdmacm/libibverbs/libmlx4 trees
and run librdmacm's rdma_{xclient,xserver} example? I was a bit confused
since I see this example both in the master and the xrc brach.

BTW the master branch misses a tag for v1.0.15...

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] rdma/core: Really export ib_open_qp() to user space

[Hefty, Sean]

> So what else would you suggest for further testing, is that pulling
> the xrc branch of your ofa hosted librdmacm/libibverbs/libmlx4 trees
> and run librdmacm's rdma_{xclient,xserver} example? I was a bit confused
> since I see this example both in the master and the xrc brach.

I wanted to keep the rdma_client / rdma_server tests as simple as possible, but I still wanted a more complete client / server test where I could add RDMA reads / writes / etc.  The rdma_xclient / rdma_xserver tests are a place for those.  They support RC and XRC, so they're in the master branch for the RC support.  I have extensions for them in a private branch which aren't quite ready yet.

For my testing, I pulled mvapich2 and run with and without xrc support, plus the normal verbs tests.  For the xrc support I have to build with the OFED compatibility APIs added into verbs.

> BTW the master branch misses a tag for v1.0.15...

thanks - I'll add.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma/core: Really export ib_open_qp() to user space

[Or Gerlitz]

Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> The rdma_xclient / rdma_xserver tests are a place for those.  They support RC and
> XRC, so they're in the master branch for the RC support.  I have extensions for them in
> a private branch which aren't quite ready yet.

again, and just to make sure I got it - for basic XRC testing which
doesn't go to MPI nor to the OFED compatability APIs, what env/test
would you recommend - is that the xrc branch on the three libraries
and rdma_x{client, server}?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] rdma/core: Really export ib_open_qp() to user space

[Hefty, Sean]

> again, and just to make sure I got it - for basic XRC testing which doesn't go
> to MPI nor to
> the OFED compatability APIs, what env/test would you recommend - is that the
> xrc branch on the three libraries and rdma_x{client, server}?

Yes - please make sure you have pulled those branches recently (since last Friday).  The rdma_cm tests are fairly simple and just transfer one direction.  The ibv_xsrq_pingpong tests will connect and transfer data both ways.

- Sean 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html