Documentation for libiverbs?

Documentation for libiverbs?

[Werther Pirani]

    Hello,

I've been redirected to this list from OpenFabrics' web site, so let me apologise 
right away if this is not the right place for this question.

We are writing an application that requires high throughput and low latency. More 
specifically, we need to take advantage of features like RDMA, possibly through 
the ibv_ API in libiverbs, but the documentation available is really too sketchy 
to be of any help (and even the OpenFabrics' web site isn't of any use in this 
respect).

At the moment all I have to go by is the RDMA_RC_example in the "RDMA Ware Network 
Programming User's Manual" from Mellanox, but it's really not quite enough. Since 
even searching the web isn't turning out any documentation, I was wondering if 
anyone here has any to share.



Many thanks in advance,

Werther Pirani
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documentation for libiverbs?

[Tziporet Koren]

On 1/13/2010 12:06 PM, Werther Pirani wrote:
>    Hello,
>
> I've been redirected to this list from OpenFabrics' web site, so let 
> me apologise right away if this is not the right place for this question.
>
> We are writing an application that requires high throughput and low 
> latency. More specifically, we need to take advantage of features like 
> RDMA, possibly through the ibv_ API in libiverbs, but the 
> documentation available is really too sketchy to be of any help (and 
> even the OpenFabrics' web site isn't of any use in this respect).
>
> At the moment all I have to go by is the RDMA_RC_example in the "RDMA 
> Ware Network Programming User's Manual" from Mellanox, but it's really 
> not quite enough. Since even searching the web isn't turning out any 
> documentation, I was wondering if anyone here has any to share.
>
>
You can look at the ping-pong examples that come with libibverbs. 
http://www.openfabrics.org/downloads/libibverbs/
Also you can look at the performance tests (ib_write_lat, ib_write_bw, 
etc). http://www.openfabrics.org/downloads/perftest/

Tziporet


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documentation for libiverbs?

[Werther Pirani]

Tziporet Koren wrote:

 > You can look at the ping-pong examples that come with libibverbs. 
http://www.openfabrics.org/downloads/libibverbs/
 > Also you can look at the performance tests (ib_write_lat, ib_write_bw, etc). 
http://www.openfabrics.org/downloads/perftest/

Thank you for the prompt reply! Not quite what I'm looking for (a proper 
programmer's guide) but better than nothing and probably enough to get started.

However, I have a question: our application needs multicast which, if I understand 
things correctly, means the only option is Unreliable Datagrams.

I'm experimenting with ud_pingpong.c and it works, but as soon as a try to 
exchange messages bigger than 2048 bytes I have the following error:

Failed status local length error (1) for wr_id 2

A table in Mellanox' "RDMA Aware Networks Programming" shows that for UDs the max 
messages size is limited by the MTU. That really being the case, how can I 
check/change the it?


Many thanks in advance,

Werther Pirani
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documentation for libiverbs?

[Roland Dreier]

 > I understand things correctly, means the only option is Unreliable
 > Datagrams.

Yes.

 > I'm experimenting with ud_pingpong.c and it works, but as soon as a
 > try to exchange messages bigger than 2048 bytes I have the following
 > error:
 > 
 > Failed status local length error (1) for wr_id 2
 > 
 > A table in Mellanox' "RDMA Aware Networks Programming" shows that for
 > UDs the max messages size is limited by the MTU. That really being the
 > case, how can I check/change the it?

UD messages are limited to a single packet.  The packet MTU is a
hardware/fabric property, and it is quite likely that 2048 is the
maximum for your fabric.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documentation for libiverbs?

[Bryan Green]

Werther Pirani writes:
>     Hello,
> 
> I've been redirected to this list from OpenFabrics' web site, so let me apo=
> logise=20
> right away if this is not the right place for this question.
> 
> We are writing an application that requires high throughput and low latency=
> . More=20
> specifically, we need to take advantage of features like RDMA, possibly thr=
> ough=20
> the ibv_ API in libiverbs, but the documentation available is really too sk=
> etchy=20
> to be of any help (and even the OpenFabrics' web site isn't of any use in t=
> his=20
> respect).
> 
> At the moment all I have to go by is the RDMA_RC_example in the "RDMA Ware =
> Network=20
> Programming User's Manual" from Mellanox, but it's really not quite enough.=
>  Since=20
> even searching the web isn't turning out any documentation, I was wondering=
>  if=20
> anyone here has any to share.

You should read Chapters 10 (Software Transport Interface) and 11 (Software
Transport Verbs) of the Infiniband Architecture Specification, Volume 1.

Then, besides the example code such as ib_write_bw and ib_rdma_bw, there are the man pages for libibverbs
and librdmacm.

-Bryan

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html