Re: Further thoughts on uAPI

Jason Gunthorpe
2016-04-26T14:33:23Z

On Tue, Apr 26, 2016 at 09:18:04AM -0400, Doug Ledford wrote:

> > So, a sketch would look like this:
> >
> > struct msg
> > {
> > uint16_t length;
> > uint16_t class_id;
> > uint32_t object_id; // in/out
> > struct qp_base_attr
> > {
> > uint16_t length;
> > uint16_t attribute_id;
> >
> > uint16_t qpn; //in/out
> > uint16_t qp_flags;
> > uint16_t max_send_wr,max_recv_qr,max_send_sge,////
> > };
> > // Option to piggy back what ibv_modify_qp does:
> > struct qp_addr_ib
> > {
> > uint16_t length;
> > uint16_t attribute_id;
> >
> > uint16_t dlid,slid,sl,pkey,etc;
> > }
> > }
> >
> > msg.length = sizeof(msg);
> > msg.class_id = RDMA_OBJ_QP_UD;
> > msg.base.legnth = sizeof(msg.base);
> > msg.base.attribute_id = RDMA_ATTR_QP_BASE;
> > msg.base.qp_flags = XX
> > [..]
> > ioctl(fd,RDMA_CREATE_OBJECT,&msg);
> > [..]
> > ioctl(fd,RDMA_MODIFY_OBJECT,&msg2);
>
> I think I would do it slightly differently. In this example, the
> class_id covers the entire list of commands.

It isn't a list of commands, it is a list of attributes - this was
specifically exploring the idea Liran has talked about where create_qp
and modify_qp *as a special case* can be combined. This is not
chaining commands but atomically creating a qp with the full set of
attributes.

> if each command in the linked list was fully self-contained and
> complete. For the example of the cmtime program that's part of
> librdamcm, when you run it with 1000s of connections as the test, being
> able to pipeline 10, or 50, or 100 different commands would be useful to
> the test. Likewise, a verbs 2.0 application might want to build up a
> chain of commands and pass the whole chain down in one ioctl.

It would not be too hard to provide a linked list execution ioctl:

struct chain_hdr
{
uint32_t command;
struct chain_hdr *next;
uint64_t msg[];
}
ioctl(fd,RDMA_EXECUTE_CHAINED_COMMANDS,&chain_hdr);

I think the kernel side would be fairly trivial.

Someone would have to figure out a way to use that from userspace of
course..

Jason