Re: [openib-general] Re: 2.6.16-rc6-mm2: new RDMA CM EXPORT_SYMBOL's

[Sean Hefty <mshefty@ichips.intel.com>]

Matthew Frost wrote:
> To the point.  I, insightful betimes, but a non-user of the technology,
> can grep TFM's and find out what the names could mean, but we're left
> guessing at what some of these *do*.  Translating names falls into the
> "any idiot can" category of data mining, but if you code for them, we can
> see context.  If you named them more transparently, we might even use
> them right.  Maybe just comment well?

Documentation in the form of comments are provided in the header files.  I can 
clarify the calls if needed.  To help understand the structure of this patch, 
three modules were submitted:

ib_addr - maps IP addresses to an RDMA device.
rdma_cm - Adds connection management over Infiniband using IP addressing.  This 
exports most of the symbols below.
rdma_ucm - Exports the rdma_cm functionality to userspace.  This uses most of 
the exported symbols (starting at rdma_create_id and below).

> +EXPORT_SYMBOL(rdma_wq); Work Queue (do what to it?)

This is used by ib_addr and rdma_cm modules to invoke user callbacks. 
Additional code not yet ready for merging will also make use of this work queue. 
  The intent is to re-use this work queue, rather than each module creating 
their own.

> +EXPORT_SYMBOL(rdma_translate_ip); Translate IP Address
> +EXPORT_SYMBOL(rdma_resolve_ip); Resolve IP Address
> +EXPORT_SYMBOL(rdma_addr_cancel); Address Cancel (memory?)

These exports are from the ib_addr module.  The routines are called by the 
rdma_cm module.  The first two map an IP address to a local Infiniband device 
address.  Rdma_resolve_ip is an asynchronous call, so rdma_addr_cancel is used 
to cancel its operation.  'rdma_cancel_resolve_ip' might have been a clearer name.

> +EXPORT_SYMBOL(rdma_create_id); Create (?) ID

The rdma_cm_id created by this call is required for the calls below. 
Conceptually, it may help to think of an rdma_cm_id as somewhat like a socket.

> +EXPORT_SYMBOL(rdma_create_qp); Create Queue Pair (WQ,CQ)
> +EXPORT_SYMBOL(rdma_destroy_qp); Destroy Queue Pair (WQ,CQ)

'rdma_create_qp' associates a QP with an rdma_cm_id, so that the rdma_cm can 
perform the QP transitions for the user during connection establishment.

> +EXPORT_SYMBOL(rdma_init_qp_attr); Set Initial Queue Pair Attributes (?)

This initializes the QP attributes for a user that wants to manually perform QP 
transitions.  It is provided mainly for userspace support.

> +EXPORT_SYMBOL(rdma_destroy_id); Destroy (?) ID
> +EXPORT_SYMBOL(rdma_listen); Listen (to ... socket, port, pipe, what?)

Listens across RDMA devices for connection requests.  The listen is on an 
rdma_cm_id.

> +EXPORT_SYMBOL(rdma_resolve_route); Resolve Route (datagram path?)

In Infiniband terms, this obtains a path record from the subnet manager.  The 
path record specifies the route through the subnet that packets between two 
connected queue pairs will take.

> +EXPORT_SYMBOL(rdma_resolve_addr); Resolve Address (memory?)

This converts struct sockaddr to RDMA addresses.  It ends up calling 
rdma_translate_ip and rdma_resolve_ip, but performs some additional work to 
handle device hotplug events.

> +EXPORT_SYMBOL(rdma_bind_addr); Bind Address (memory?)

Associates an rdma_cm_id to a specific struct sockaddr.

> +EXPORT_SYMBOL(rdma_connect); Connect
> +EXPORT_SYMBOL(rdma_accept); Accept
> +EXPORT_SYMBOL(rdma_reject); Reject
> +EXPORT_SYMBOL(rdma_disconnect); Disconnect
> 
> Address vs. IP - I know we're talking about a net/dma kluge here, but the
> twin usage is bugging me.  I'm intuiting the _addr as memory addresses,
> rather than IP addresses, which seem to be _ip, but my poor gray goo
> suffers pointer overload.

Maybe the naming is off here.  I used _ip when referring specifically to an IP 
address, and _addr when using a struct sockaddr.  In some cases, such as 
rdma_bind_addr, the 'address' may be nothing more than a port number.

> +EXPORT_SYMBOL(ib_get_rmpp_segment); Reliable MultiPacket Protocol

This is from a separate patch.  It is exported by the ib_mad module, and used by 
the ib_umad module.  There is at least one out of tree module (not yet ready for 
merging) that will make use of it.

> +EXPORT_SYMBOL(ib_copy_qp_attr_to_user); Push Queue Pair Attribute
> +EXPORT_SYMBOL(ib_copy_path_rec_to_user); Push Path Record
> +EXPORT_SYMBOL(ib_copy_path_rec_from_user); Retrieve Path Record

These are used by ib_uverbs, ib_ucm, and rdma_ucm modules.

> +EXPORT_SYMBOL(ib_modify_qp_is_ok); Yes, Modify Queue Pair, or "QP is
> OK", or "QP was Modified OK"?

This is from a separate patch.  It should be exported by ib_verbs, and used by 
ib_mthca.  The call verifies that the settings to modify a QP from one state to 
the next are valid.  I believe that the check used to be part of the ib_mthca 
driver itself, but additional kernel drivers, such as the ipath driver recently 
submitted for inclusion, now make use of this routine.

> +EXPORT_SYMBOL(ip_dev_find); Find IP device (sub(/ip/, "ib")? find the
> network interface device?)

This is the network call in fib_frontend.c.  It is being re-exported (the export 
was removed a couple of versions ago) for use by ib_addr.

>>Please explain the thinking behind the choice of a non-GPL export. 
>>(Yes, we discussed this when inifiniband was first merged, but it
>>doesn't hurt to reiterate).

The agreement made within the OpenIB community, from where this code originates, 
is that all source code be licensed under a dual license of BSD/GPL.  I am not a 
lawyer, so I don't know the implications of changing the exports to be GPL only, 
given the OpenIB license.  But my understanding is that makes using those 
functions less attractive.

- Sean