ib_write_bw hanging when using max max_inline value

ib_write_bw hanging when using max max_inline value

[Håkon Bugge]

Hi,


I modified ib_write_bw to support inline sizes beyond the default value of 400 (not sure why the program does not allow larger values though).

When using the largest max_inline value supported by ConnectX (i.e. the creation of QP goes OK), the test program hangs when exchanging 920 bytes.

Client command line: ib_write_bw -I 928 -s 920 <server>

I am running on CentOS 5.3, Intel E5540, OFED 1.4.1, and ConnectX w/fw 2.6.000.


Thanks, Håkon

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib_write_bw hanging when using max max_inline value

[Or Gerlitz]

Håkon Bugge wrote:
> the test program hangs when exchanging 920 bytes [...]  the creation of QP goes OK
>   
attaching a debugger is typically helpful to see where a program talking 
directly to the hardware hangs. If it happens on the slow pass, strace 
can be useful as well.  Did you take a look on the actual values set for 
this qp, that it as suggested by ibv_create_qp(3) look on the init 
attributes after the function returns.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib_write_bw hanging when using max max_inline value

[Håkon Bugge]

On Jan 24, 2010, at 6:26 , Or Gerlitz wrote:
> attaching a debugger is typically helpful to see where a program talking directly to the hardware hangs. If it happens on the slow pass, strace can be useful as well.  Did you take a look on the actual values set for this qp, that it as suggested by ibv_create_qp(3) look on the init attributes after the function returns.


The capabilities in qp_init_attr used as input to ibv_create_qp() are:

max_send_wr = 100, max_recv_wr = 1, max_send_sge = 1, max_recv_sge = 1, max_inline_data = 928

Upon return from ibv_create_qp, the capabilities are modified to the following (note, max_inline_data is not changed);

max_send_wr = 125, max_recv_wr = 1, max_send_sge = 32, max_recv_sge = 1, max_inline_data = 928

All WRs have IBV_SEND_SIGNALED set. The program does not get any completions, hence it is running in the while loop surrounding the call to ibv_poll_cq().

Note decreasing the size of the RDMA to 912 bytes, the program works.


-h

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib_write_bw hanging when using max max_inline value

[Or Gerlitz]

Håkon Bugge wrote:
> The capabilities in qp_init_attr used as input to ibv_create_qp() are:
> max_send_sge = 1, max_recv_sge = 1, max_inline_data = 928
> Upon return the capabilities are modified to the following 
> max_send_sge = 32, max_recv_sge = 1, max_inline_data = 928
> Note decreasing the size of the RDMA to 912 bytes, the program works
Jack, sounds like this use case hits the bug/s you were attempting to 
solve with the patch set we were discussing @ 
http://marc.info/?l=linux-rdma&m=126330119309593 which that never made 
it upstream, correct?

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html