Re: [PATCH] RDMA/nes: IB_QPT_RAW_PACKET QP type support for nes driver

Or Gerlitz
2010-07-06T08:50:13Z

miroslaw.walukiewicz-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org wrote:
> adds a IB_QPT_RAW_PACKET QP type implementation for nes driver

> +++ b/drivers/infiniband/hw/nes/nes_ud.c
> +static const struct file_operations nes_ud_sksq_fops = {
> + .owner = THIS_MODULE,
> + .open = nes_ud_sksq_open,
> + .release = nes_ud_sksq_close,
> + .write = nes_ud_sksq_write,
> + .read = nes_ud_sksq_read,
> + .mmap = nes_ud_sksq_mmap,
> +};
> +
> +
> +static struct miscdevice nes_ud_sksq_misc = {
> + .minor = MISC_DYNAMIC_MINOR,
> + .name = "nes_ud_sksq",
> + .fops = &nes_ud_sksq_fops,
> +};

Reading through the May 2010 "RDMA/nes: IB_QPT_RAW_PACKET QP type support for nes driver" email thread, e.g at the below links, you say

> The non-bypass post_send/recv channel (using /dev/infiniband/rdma_cm) is shared with
> all other user-kernel communication and it is quite complex. It is a perfect path
> for QP/CQ/PD/mem management but for me it is too complex for traffic acceleration.
> The user<->kernel path through additional driver, shared page for lkey/vaddr/len
> passing and SW memory translation in kernel is much more effective.

http://marc.info/?l=linux-rdma&m=127299659017928
http://marc.info/?l=linux-rdma&m=127306694704653

I still don't see what is the performance issue with the uverbs post_send/post_recv and if there is such why it can't be fixed, to avoid introducing lib/driver nes special char device. Could you explain it with some more details? You were mention the rdma-cm device file, but the uverbs cmd api is used by libibverbs / uverbs and not by librdmacm / rdma-ucm, which is anyway a slow path.

Also, I understand that .read (.write) entry maps to posting a receive (send) buffer, what is the use case for .mmap entry

> --- a/drivers/infiniband/hw/nes/nes_verbs.c
> +++ b/drivers/infiniband/hw/nes/nes_verbs.c

> @@ -1139,7 +1141,6 @@ static struct ib_qp *nes_create_qp(struct ib_pd *ibpd,
[...]
> - atomic_inc(&qps_created);
> @@ -1405,10 +1406,122 @@ static struct ib_qp *nes_create_qp(struct ib_pd *ibpd,
[...]
> + /* moved here to be sure that QP is really created */
> + /*(now it counted a number of QP creation trials */
> + atomic_inc(&qps_created);

best if this change and couple more of its such will be placed in a clean-up patch to nes_verbs.c, such that the amount of RAW QP related changes to review is minimized.

> @@ -2939,6 +3130,9 @@ int nes_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
> nesqp->hwqp.qp_id, attr->qp_state, nesqp->ibqp_state,
> nesqp->iwarp_state, atomic_read(&nesqp->refcount));
>
> + if (ibqp->qp_type == IB_QPT_RAW_PACKET)
> + return 0;

isn't a raw qp associated with a specific port of the device?