IBV Test Program

IBV Test Program

[Greg Kerr]

Hi, 

I've tried asking this question in a few different places with no luck, so I'm 
hoping you folks can help me. I'm an undergraduate research assistant learning 
how to use verbs and I had a question.

I want to modify rc_pingpong.c to loop and basically send data between the two 
nodes multiple times, instead of just once. The first time through, it 
successfuly completes the transmission. On the next iteration the call to 
ibv_post_recv fails.

The source code is here: 
http://pastebin.com/1mEu9x4f (server)
http://pastebin.com/k1NKdCSn (client)

Please forgive the debug code scattered in there.

I'm pretty sure it must be a simple problem, I just don't know what the 
problem is.  I can't imagine the queue overflowing after a second 
ibv_post_recv.

The relevant changes are around line 769.

-- 
Greg Kerr
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IBV Test Program

[Roland Dreier]

On Tue, Mar 29, 2011 at 9:48 AM, Greg Kerr <kerr.g-movQPkccWJngpn9g0Uvcdg@public.gmane.org> wrote:
> I want to modify rc_pingpong.c to loop and basically send data between the two
> nodes multiple times, instead of just once. The first time through, it
> successfuly completes the transmission. On the next iteration the call to
> ibv_post_recv fails.

Not sure I follow... the pingpong code already sends multiple times in a loop?

Instead of a link to a pastebin, can you send a diff to show what you changed?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IBV Test Program

[Jason Gunthorpe]

On Tue, Mar 29, 2011 at 12:48:42PM -0400, Greg Kerr wrote:

> I've tried asking this question in a few different places with no
> luck, so I'm hoping you folks can help me. I'm an undergraduate
> research assistant learning how to use verbs and I had a question.
 
> I want to modify rc_pingpong.c to loop and basically send data
> between the two nodes multiple times, instead of just once. The
> first time through, it successfuly completes the transmission. On
> the next iteration the call to ibv_post_recv fails.

This will post into a RC QP without a recv buffer on the far side
because the way you have the loop around ibv_poll_cq is not right.

You need to loop around ibv_poll_cq until you get both completions,
then repost your recv buffer and then sleep. Otherwise you will end up
posting a send into a QP with no recv buffer and send the QP to the
err state which mucks everything up.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html