Re: bifurcated driver

[Nicolas Dichtel <nicolas.dichtel-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>]

Also CC netdev, this thread may interest network folks.

Le 06/11/2014 09:13, Alex Markuze a écrit :
> Danny sums up the issue perfectly IMHO.
> While both verbs and DPDK aim to provide generic user space networking, the
> similarities end there.
> verbs and RDMA HW are closely coupled and behave differently then standard
> eth nics and are not related to netdev mechanisms.
>
> Or, welcome to this discussion.
>
> Those interested can read the IB spec's (+1K pages) available from
> openfabrics*.
> *https://www.openfabrics.org/index.php
>
>
>
>
> On Thu, Nov 6, 2014 at 6:45 AM, Zhou, Danny <danny.zhou-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>
>> I roughly read libibverbs related code and relevant infiniband/rdma
>> documents, and found though
>> many concepts in libibverbs looks similar to bifurcated driver, but there
>> are still lots of differences as
>> illustrated below based on my understanding:
>>
>> 1) Queue pair defined in RDMA specification are abstract concept, where
>> the queue pairs term used in
>>    bifurcated driver are rx/tx queue pairs in the NIC.
>> 2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver
>> (no NIC control), while libibverbs
>>    as a user space library rather than driver offloads certain operations
>> to kernel driver and NIC by invoking
>>    "verbs" APIs.
>> 3) Libibverbs invokes infiniband specific system calls to allow
>> user/kernel space communication based on
>>    "verbs" defined in infiniband/RDMA spec, while bifurcated driver build
>> on top of af_packet module
>>    and new socket options to do things like hw queue split-off , map
>> certain pages on I/O space to user space
>>    operations, etc.
>> 4) There is a specific embedded MMU unit in Infiniband/RDMA to provides
>> memory protection, while
>>    bifurcated driver uses IOMMU rather than NIC to provide memory
>> protection.
>>
>> IMHO, libibverbs and corresponding kernel modules/drivers are specifically
>> designed and implemented for
>> direct access to RDMA hardware from userspace, and it highly depends on
>> "verbs" related system calls
>> supported by infiniband/rdma mechanism in kernel, rather than netdev
>> mechanism that bifurcated driver
>> solution depends on.
>>
>>> -----Original Message-----
>>> From: Vincent JARDIN [mailto:vincent.jardin-pdR9zngts4EAvxtiuMwx3w@public.gmane.org]
>>> Sent: Thursday, November 06, 2014 9:31 AM
>>> To: Zhou, Danny
>>> Cc: Thomas Monjalon; dev-VfR2kkLFssw@public.gmane.org; Fastabend, John R; Or Gerlitz
>>> Subject: Re: [dpdk-dev] bifurcated driver
>>>
>>> +Or
>>>
>>> On 05/11/2014 23:48, Zhou, Danny wrote:
>>>> Hi Thomas,
>>>>
>>>> Thanks for sharing the links to ibverbs, I will take a close look at
>> it and compare it to bifurcated driver. My take
>>>> after a rough review is that idea is very much similar, but bifurcated
>> driver implementation is generic for any
>>>> Ethernet device based on existing af_packet mechanism, with extension
>> of exchanging the messages between
>>>> user space and kernel space driver.
>>>>
>>>> I have an internal document to summary the pros and cons of below
>> solutions, except for ibvers, but
>>>> will be adding it shortly.
>>>>
>>>> - igb_uio
>>>> - uio_pci_generic
>>>> - VFIO
>>>> - bifurcated driver
>>>>
>>>> Short answers to your questions:
>>>>>     - upstream status
>>>> Adding IOMMU based memory protection and generic descriptor
>> description support now, into version 2
>>>> kernel patches.
>>>>
>>>>>     - usable with kernel netdev
>>>> af_packet based, and relevant patchset will be submitted to netdev for
>> sure.
>>>>
>>>>>     - usable in a vm
>>>> No, it does no coexist with SRIOV for number of reasons. but if you
>> pass-through a PF to a VM, it works perfect.
>>>>
>>>>>     - usable for Ethernet
>>>> It could work with all Ethernet NICs, as flow director is available
>> and NIC driver support new net_ops to split off
>>>> queue pairs for user space.
>>>>
>>>>>     - hardware requirements
>>>> No specific hardware requirements. All mainstream NICs have multiple
>> qpairs and flow director support.
>>>>
>>>>>     - security protection
>>>> Leverage IOMMU to provide memory protection on Intel platform. Other
>> archs provide similar memory protection
>>>> mechanism, so we only use arch-agnostic DMA memory allocation APIs in
>> kernel to support memory protection.
>>>>
>>>>>     - performance
>>>> DPDK native performance on user space queues, as long as drop_en is
>> enabled to avoid head-of-line blocking.
>>>>
>>>> -Danny
>>>>
>>>>> -----Original Message-----
>>>>> From: Thomas Monjalon [mailto:thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org]
>>>>> Sent: Wednesday, November 05, 2014 9:01 PM
>>>>> To: Zhou, Danny
>>>>> Cc: dev-VfR2kkLFssw@public.gmane.org; Fastabend, John R
>>>>> Subject: Re: [dpdk-dev] bifurcated driver
>>>>>
>>>>> Hi Danny,
>>>>>
>>>>> 2014-10-31 17:36, O'driscoll, Tim:
>>>>>> Bifurcated Driver (Danny.Zhou-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org)
>>>>>
>>>>> Thanks for the presentation of bifurcated driver during the community
>> call.
>>>>> I asked if you looked at ibverbs and you wanted a link to check.
>>>>> The kernel module is here:
>>>>>
>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
>>>>> The userspace library:
>>>>>     http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
>>>>>
>>>>> Extract from Kconfig:
>>>>> "
>>>>> config INFINIBAND_USER_ACCESS
>>>>>     tristate "InfiniBand userspace access (verbs and CM)"
>>>>>     select ANON_INODES
>>>>>     ---help---
>>>>>       Userspace InfiniBand access support.  This enables the
>>>>>       kernel side of userspace verbs and the userspace
>>>>>       communication manager (CM).  This allows userspace processes
>>>>>       to set up connections and directly access InfiniBand
>>>>>       hardware for fast-path operations.  You will also need
>>>>>       libibverbs, libibcm and a hardware driver library from
>>>>>       <http://www.openfabrics.org/git/>.
>>>>> "
>>>>>
>>>>> It seems to be close to the bifurcated driver needs.
>>>>> Not sure if it can solve the security issues if there is no dedicated
>> MMU
>>>>> in the NIC.
>>>>>
>>>>> I feel we should sum up pros and cons of
>>>>>     - igb_uio
>>>>>     - uio_pci_generic
>>>>>     - VFIO
>>>>>     - ibverbs
>>>>>     - bifurcated driver
>>>>> I suggest to consider these criterias:
>>>>>     - upstream status
>>>>>     - usable with kernel netdev
>>>>>     - usable in a vm
>>>>>     - usable for ethernet
>>>>>     - hardware requirements
>>>>>     - security protection
>>>>>     - performance
>>>>>
>>>>> --
>>>>> Thomas
>>
>>