Re: [PATCH] RDMA/nes: IB_QPT_RAW_PACKET QP type support for nes driver

Or Gerlitz
2010-07-07T06:45:28Z

Walukiewicz, Miroslaw wrote:
> From my measuremnts it looks like the problem is related to memory allocation in the user-space and kernel path, that is a very, very expesive operation. Look for the tx path (rx is very similar). Ibv_post_send():
> post_send_wrapper_1_0
> for (w = wr; w; w = w->next) {
> real_wr = alloca(sizeof *real_wr); <- 1. dyn alloc
> real_wr->wr_id = w->wr_id;
> next the call to HW specific part
> and prepare message to send
> cmd = alloca(cmd_size); <- 2. dyn allocation

Hi Mirek,

I don't think there are applications around which would use raw qp AND
are linked against libibverbs-1.0, such that they would exercise the 1_0
wrapper, so we can ignore the 1st allocation, the one at the wrapper code.

As for the 2nd allocation, since a WQE --posting-- is synchronous,
using the maximal values specified during the creation of the QP, I
believe that this allocation can be done once per QP and used later.

> dive to kernel:
> ib_uverbs_post_send()
> user_wr = kmalloc(cmd.wqe_size, GFP_KERNEL); <- 3. dyn alloc
> next = kmalloc(ALIGN(sizeof *next, sizeof (struct ib_sge)) +
> user_wr->num_sge * sizeof (struct ib_sge),
> GFP_KERNEL); <- 4. dyn alloc
> And now there is finel call to driver.
~same here for #4 you can compute/allocate once the maximal possible
size for "next" per qp and use it later. As for #3, this need further
thinking.

But before diving to all this design changes, what was the penalty
introduced by these allocations? is it in packets-per-second, latency?

> Diving to kernel is treated as a something like passing signal to kernel that there is prepared information to post_send/post_recv. The information about buffers are passed through shared page (available to userspace through mmap) to avoid copying of data. Write() ops is used to passing signal about post_send. Read() ops is used to pass information about post_recv(). We avoid additional copying of the data that way.
thanks for the heads-up, I took a look and this user/kernel shared
memory page is used to hold the work-request, nothing to do with data.

As for the work request, you still have to copy it in user space from
the user work request to the library mmaped buffer. So the only
difference would be the copy_from_user done by uverbs, for few tens of
bytes, can you tell if/what is the extra penalty introduced by this copy?

> struct nes_ud_send_wr {
> u32 wr_cnt;
> u32 qpn;
> u32 flags;
> u32 resv[1];
> struct ib_sge sg_list[64];
> };
>
> struct nes_ud_recv_wr {
> u32 wr_cnt;
> u32 qpn;
> u32 resv[2];
> struct ib_sge sg_list[64];
> };
Looking on struct nes_ud_send/recv_wr, I wasn't sure to follow, the same
instance can be used to post list of work requests, where is work
request is limited to use one SGE, am I correct?

I don't think there a need to support posting 64 --send-- requests, for
recv it might makes sense, but it could be done in a "batch/background"
flow, thoughts?