Re: [PATCH rdma-core 0/7] Add mlx5 direct verbs

Sagi Grimberg
2017-02-02T08:52:36Z

> This patchset from Leon adds direct access to mlx5 devices.
>
> The libibverbs API is an abstract API. It is agnostic to any underlying
> provider specific implementation. While this abstraction has the advantage of
> user applications portability it has a performance penalty. For some
> applications optimizing performance is more important than portability.
>
> The mlx5 direct verbs API introduced in this patchset is intended for such
> applications. It exposes mlx5 specific low level data path
> (send/receive/completion) operations, allowing the application to bypass the
> libibverbs data path API.

Yishai and Leon this is good stuff !!

Couple of comments:

- The user of this is pretty intimate with the driver, which makes a
somewhat strange combo of having both ibv_ and mlx5dv_ API. I
understand that it can inflate the the implementation, but perhaps
for completeness we can have also the control plane API as well (even
as a trampoline to the generic equivalent where it's useless).

The rational here is that extended features that needs allocation
flags can be contained to mlx5 and not pollute/inflate the generic
verbs implementation.

- This brings me to the next step of this. What is still missing is
how we allow the user to have more control and information about how
his db records (namely UARs) are allocated. How does the user
know if he needs to protect the DB write? Is it BF? How can the user
ask for more UARs than originally allocated to it by the kernel?
The user of this can really use an API that allows more control
of this area.

Hopefully we can come up with something that is abstract enough to
not expose how exactly UARs are strided on pages and the exact
interface with the kernel, but still let the user choose if to use
a dedicated db_rec and let the driver manage all the internal details.

Thoughts?