* [RFC] Generic InfiniBand transport done in software
@ 2015-12-22 12:38 Moni Shoua
[not found] ` <CAG9sBKO=fP2Xd=JXK7hvzGFRLQmh1dF+mWDxdnhU+TqFg7YD1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Moni Shoua @ 2015-12-22 12:38 UTC (permalink / raw)
To: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib
Cc: linux-rdma
Hi,
In the past months the need for a kernel module that implements the InfiniBand
transport in software and unify all the InfiniBand software drivers has
been raised. Since then, nobody has submitted any design proposal that satisfy
the initial thoughts and can serve various back-ends.
The following is a RFC that presents a solution made of a single
generic InfiniBand
driver and many hardware specific back-end drivers. The RFC defines
the requirements
and the interfaces that have to be part of any generic InfiniBand driver.
A generic InfiniBand driver that is not compliant with this RFC wouldn't be able
to serve different back-ends and therefore would miss its target.
================================================================================
A. Introduction
--------------------------------------------------------------------------------
In Linux kernel, the responsibility to implement the InfiniBand protocol is
roughly divided between 2 elements. The first are the core drivers which expose
an abstract verbs interface to the upper layer as well as interfaces to some
common IB services like MAD, SA and CM. The second are vendor drivers and
hardware which implement the abstract verbs interface.
A high level view of the model is in Figure A1
+-----------------+
| |
| IB core |
| drivers |
| |
+--------+--------+
| Common
--------------------------------------------
| Vendor
+--------+--------+
| |
| Hardware |
| drivers |
| |
+--------+--------+
| |
| Hardware |
| |
+-----------------+
A1 - IB implementation model in Linux kernel
In the vendor part of the model, the devision of work between software and
hardware is undefined and is usually one of the two below
- Context and logic are managed in software. Hardware role is limited to
lower layer protocols (depending on the link layer) and maybe some offloads
- Context and logic are managed in hardware while software role is to create
or destroy a context in the hardware and gets notified when hardware reports
about a completions tasks.
The following examples demonstrates the difference between the approaches above.
- Send flow: application calls post_send() with QP and a WR. In the software
based approach the QP context is retrieved, the WR is parsed and a proper IB
packet is formed to be put in hardware buffers. In hardware based approach the
driver puts the WR in hardware buffers with a handle to the QP and hardware
does the rest.
- Receive flow: a data packet is received and put in hardware buffers (assume
that the destination is a RC QP). In software based approach the packet is
handed to the driver. The driver retrieves the context by QPN, decides if
ACK/NACK packet is required (if so, generates it) and decides if CQE is
required on the CQ of the QP. In the other approach the hardware compares
the packet to the state in the context, generates the ACK/NACK and raises an
event to the driver that CQE is ready for reading.
Figure A2 illustrates hardware/software relationship in the implementation of a
IB transport solution in the software based approach.
+----------------------------+
| IB transport |
| capable driver |
| |
| +----------------------+ |
| |Context (deep) | |
| | | |
| +----------------------+ |
| |
| +----------------------+ |
| | Logic (IB transport) | |
| | | |
| +----------------------+ |
| |
| +----------------------+ |
| | Interface | |
| | | |
| +----------------------+ |
+----------------------------+
|
|
+----------------------------+
| +----------------------+ |
| | Interface | |
| | | |
| +----------------------+ |
| |
| +----------------------+ |
| | Context (shallow) | |
| | | |
| +----------------------+ |
| |
| +----------------------+ |
| | Logic (link layer) | |
| | | |
| +----------------------+ |
| |
| IB transport |
| incapable NIC |
+----------------------------+
A2 - software based approach for IB transport implementation
The rest of this paper is focused on IB transport solutions that are based on
software
B. Common software based IB transport solution
--------------------------------------------------------------------------------
In hardware based solutions the software driver implementation is tightly tied
to the hardware. Therefore, for different hardware types there is hardly a
common part to all software driver. In software based solutions things are
different. For such, since hardware capabilities are limited and implement only
the lower layer(s) of the OSI model and transport layer is done in the software.
In that case the common part between all drivers is quite large and is actually
the implementation of the InfiniBand transport specification.
The immediate conclusion is that we can take one software driver and
theoretically match it to any hardware with some (obvious and less obvious)
adjustments.
Taking this one step forward yields another conclusion that only one
IB transport software driver can be designed to work with all types of
hardware back-ends, if we supply a well defined abstract interface
between the IB
transport driver and a theoretical model of hardware that all real hardware can
fit to.
The structure of the generic InfiniBand driver and its placement in the IB
stack layer model is as Figure B1 describes
+-----------------------------------------------+
| |
| IB core drivers |
| |
+------------^-----------+--+--+----------------+
| | | |
ib_register_device() | | | Abstract interface
| | | |
+------------+-----------v--v--v----------------+
| Gneric IB +--------------+ |
| driver |Abstract | |
| |interface | |
| +--------------+ |implementation| |
| | Context & | +--------------+ |
| | Logic | |
| | | |
| +--------------+ |
+-----^-----------------------------^--------+--+
| | | |
| | API | | Abstract interface
| | | |
| | | |
| | | |
+-----+--------v------+ +---------+--------v--+
| HW device driver A | | HW device driver B |
| (specific) | | (specific) |
| | | |
+---------------------+ +---------------------+
B1 - Generic InfiniBand driver structure & layout
To the layer above it, ib_core, the generic driver registers a new
ib_device and implements all the mandatory verbs(as in
ib_device_check_mandatory()). To the layer below, the thin hardware
specific driver, the generic driver declares an abstract interface that
supplies API functions as we will describe later in this document.
In addition, all SW providers may share a single common user-space provider,
since there is no direct interaction between user-space and HW.
C. Requirements from the generic InfiniBand driver
-----------------------------------------------------------------------------
To achieve its goal, be a single IB transport implementation for any back-end
driver, the generic driver is required to the following
- Offer in implementation to the InfiniBand Architecture Specification as
described in chapters 9 (TRANSPORT LAYER) and 10 (SOFTWARE TRANSPORT
INTERFACE)
- Offer an implementation to the to the ib_core abstract interface as
described in InfiniBand Architecture Specification chapter 11 (SOFTWARE
TRANSPORT VERBS)
- Have the ability to build and validate network and link layer headers as
described in the InfiniBand Architecture Specification chapters 7 (LINK LAYER)
and 8 (NETWORK LAYER) and as described in Annex A16 (RoCE) and Annex A17
(RoCEv2)
- Define an abstract interface that a hardware driver needs to implement. This
interface needs to be general enough in the functions it requires all to
implement but flexible enough in the functions it allows to implement.
- Supply an interface for hardware drivers for the opposite direction
interaction
Note: at the time when this RFC was written there are 3 software based IB
transport solutions: QIB, hfi1 and SoftRoCE.
D. The abstract back-end driver
-----------------------------------------------------------------------------
Before we proceed to define the interface between the generic driver and the
hardware driver we should understand what the hardware is capable of and
what it is not. In fact, since hardware can offer many offloading features
that we can't predict now we need categorize the functions in the abstract
interface to two, those that must be implemented by the back-end drive and
those that may be implemented. The most simple back-end driver, the one that
doesn't have any offloading capabilities, will implement only the set of
mandatory functions. Back-end driver that have some kind of offloads or require
more complex flows can implement some of the optional functions and notify the
generic driver about that.
For example, if a network driver has an optimization for sending small packets
it can implement a set of functions for a small packet flow (we leave the
question what this flow is made of unanswered here) that the generic driver will
use. Of course, any function in the abstract interface, even if it is optional,
should be as general as can be so it would fit as many as other
hardware drivers.
Among the common capabilities that a hardware driver must implement we find
- Detect new hardware on the PCI bus
- Put a packet on the wire from a buffer
- Receive a packet from the wire to a buffer
- Capture hardware events
E. The generic InfiniBand driver lower interface
--------------------------------------------------------------------------------
After discussing the general model of the back-end driver we are ready to define
the interface. From the generic driver to the hardware driver the
interface should
be abstracti, i.e. the generic driver defines what he expects from the driver to
implement,mandatory or optional.
>From the hardware driver to the generic driver the relation is many to one and
therefore the interface should have the form of an API. Tablas E1 and E2 lists
the interfaces
+------------------+---------------------------------------------------+
| Name | Description |
|==================|===================================================|
|register() |tell generic driver that new hardware is present |
|------------------|---------------------------------------------------|
|unregister() |tell generic driver that hardware was removed |
|------------------|---------------------------------------------------|
|receive() |receive a packet from the wire |
|------------------|---------------------------------------------------|
|handle_event() |notify about hardware event |
+----------------------------------------------------------------------+
Table E1 - The back-end to generic driver interface
+------------------+---------------------------------------------------+
| Name | Description |
|==================|===================================================|
|send() |put packet on the wire |
|------------------|---------------------------------------------------|
|query_link() |report link state and properties |
+----------------------------------------------------------------------+
|node_guid() |get GUID of node |
+-----------------------------------------------------------------------
|port_guid() |get GUID of port |
+-----------------------------------------------------------------------
|mcast_add() |register multicast address |
+-----------------------------------------------------------------------
|mcast_del() |unregister multicast address |
+-----------------------------------------------------------------------
|alloc_qpn() |allocate a new QP number |
+-----------------------------------------------------------------------
|get_send_buffer() |allocate a buffer for send operation |
+-----------------------------------------------------------------------
Table E2 - The generic driver to back-end driver interface
Following is a detailed description for the functions above
+---------------------------------------------------------------------------------+
|Name: |register
|
|---------------------------------------------------------------------------------|
|Description: |notify about a new instance of the hardware.
|
|---------------------------------------------------------------------------------|
|Input: |- pointer to hardware description struct
|
| |- pointers to abstract interface implementation
struct |
|---------------------------------------------------------------------------------|
|Output |- registration handle
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name: |unregister
|
|---------------------------------------------------------------------------------|
|Description: |notify that hardware instance is gone
|
|---------------------------------------------------------------------------------|
|Input: |registration handle
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name: |receive
|
|---------------------------------------------------------------------------------|
|Description: |get a packet from the wire. When done notify the
caller |
|---------------------------------------------------------------------------------|
|Input: |-buffer for the packet
|
| |-buffer length
|
| |-pointer to receive done function
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name: |handle_event
|
|---------------------------------------------------------------------------------|
|Description: |pass hardware event for processing
|
|---------------------------------------------------------------------------------|
|Input: |-registration handle of the device
|
| |-event type
|
| |-event data struct
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name: |send
|
|---------------------------------------------------------------------------------|
|Description: |transmit one packet. When done, notify the caller
|
|---------------------------------------------------------------------------------|
|Input: |- pointer to headers buffer
|
| |- headers buffer length
|
| |- pointer to scatter gather struct
|
| |- pointer to xmit_done() function
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name: |query_link
|
|---------------------------------------------------------------------------------|
|Description: |query link state
|
|---------------------------------------------------------------------------------|
|Input: |- device identifier
|
| |- port index
|
|---------------------------------------------------------------------------------|
|Output |- pointer to port state struct
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name |node_guid()
|
|---------------------------------------------------------------------------------|
|Description |get 64 bit node guid
|
|---------------------------------------------------------------------------------|
|Input: |-device identifier
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |node GUID
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name |port_guid
|
|---------------------------------------------------------------------------------|
|Description |get 64 bit
|
|---------------------------------------------------------------------------------|
|Input: |-device identifier
|
| |-port index
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |port GUID
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name |mcast_add
|
|---------------------------------------------------------------------------------|
|Description |open device for multicast address
|
|---------------------------------------------------------------------------------|
|Input: |-device identifier
|
| |-address
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name |mcast_del
|
|---------------------------------------------------------------------------------|
|Description |close device for multicast address
|
|---------------------------------------------------------------------------------|
|Input: |-device identifier
|
| |-address
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |success/failure
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name |alloc_qpn
|
|---------------------------------------------------------------------------------|
|Description |get unique QP number
|
|---------------------------------------------------------------------------------|
|Input: |
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |QP number
|
+---------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------+
|Name |get_send_buffer
|
|---------------------------------------------------------------------------------|
|Description |allocate buffer that will be used to send a patcket
|
|---------------------------------------------------------------------------------|
|Input: |-length
|
|---------------------------------------------------------------------------------|
|Output |
|
|---------------------------------------------------------------------------------|
|Return |pointer to a buffer
|
+---------------------------------------------------------------------------------+
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKO=fP2Xd=JXK7hvzGFRLQmh1dF+mWDxdnhU+TqFg7YD1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-22 18:19 ` Dennis Dalessandro
[not found] ` <20151222181905.GA742-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-23 23:15 ` ira.weiny
1 sibling, 1 reply; 16+ messages in thread
From: Dennis Dalessandro @ 2015-12-22 18:19 UTC (permalink / raw)
To: Moni Shoua
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
On Tue, Dec 22, 2015 at 02:38:10PM +0200, Moni Shoua wrote:
>Hi,
>
>In the past months the need for a kernel module that implements the InfiniBand
>transport in software and unify all the InfiniBand software drivers has
>been raised. Since then, nobody has submitted any design proposal that satisfy
>the initial thoughts and can serve various back-ends.
>
>The following is a RFC that presents a solution made of a single
>generic InfiniBand
>driver and many hardware specific back-end drivers. The RFC defines
>the requirements
>and the interfaces that have to be part of any generic InfiniBand driver.
>A generic InfiniBand driver that is not compliant with this RFC wouldn't be able
>to serve different back-ends and therefore would miss its target.
This makes no mention of the already posted work which aims to consolidate
the qib and hfi1 drivers verbs implementation. However it does seem to be
mostly in line with what we have already presented for rdmavt and the
direction the next set of patches is going in. Have you seen something in
rdmavt that contradicts this? I have not seen the need to write a detailed
RFC because it's pretty straight forward what needs to happen. Take the
common parts of qib and hfi and move to another kmod. Don't do anything that
prevents soft-roce from using the same infrastructure. I think we are
accomplishing that and have been very open during the process.
Specifically this RFC does not capture much more detail beyond what has
already been posted. There are also a number of details regarding hardware
specifics that are not dealt with here but are in the current rdmavt patch
series, as well as patches that are yet to be posted. The bottom line is we
can't sacrifice performance for a perfect API.
Previous discussion threads:
http://marc.info/?l=linux-rdma&m=144353141420065&w=2
http://marc.info/?l=linux-rdma&m=144563342718705&w=2
http://marc.info/?l=linux-rdma&m=144614769419528&w=2
Rdmavt & Qib code submissions:
http://marc.info/?l=linux-rdma&m=144968107508268&w=2
http://marc.info/?l=linux-rdma&m=144952133926970&w=2
Feedback has been received on these patches and unless otherwise noted in
the relevant threads will be incorporated into the V2 of the patch sets.
hfi1 RFC sent to staging list:
http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2015-December/082896.html
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <20151222181905.GA742-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
@ 2015-12-23 16:28 ` Moni Shoua
[not found] ` <CAG9sBKNOu+d-5f6FYTXCyTGjJU-md4dW3MFMB2-UDgRFafRdBw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Moni Shoua @ 2015-12-23 16:28 UTC (permalink / raw)
To: Dennis Dalessandro
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
>
>
> This makes no mention of the already posted work which aims to consolidate
> the qib and hfi1 drivers verbs implementation. However it does seem to be
> mostly in line with what we have already presented for rdmavt and the
> direction the next set of patches is going in. Have you seen something in
> rdmavt that contradicts this? I have not seen the need to write a detailed
> RFC because it's pretty straight forward what needs to happen. Take the
> common parts of qib and hfi and move to another kmod. Don't do anything that
> prevents soft-roce from using the same infrastructure. I think we are
> accomplishing that and have been very open during the process.
>
> Specifically this RFC does not capture much more detail beyond what has
> already been posted. There are also a number of details regarding hardware
> specifics that are not dealt with here but are in the current rdmavt patch
> series, as well as patches that are yet to be posted. The bottom line is we
> can't sacrifice performance for a perfect API.
>
> Previous discussion threads:
> http://marc.info/?l=linux-rdma&m=144353141420065&w=2
> http://marc.info/?l=linux-rdma&m=144563342718705&w=2
> http://marc.info/?l=linux-rdma&m=144614769419528&w=2
>
> Rdmavt & Qib code submissions:
> http://marc.info/?l=linux-rdma&m=144968107508268&w=2
> http://marc.info/?l=linux-rdma&m=144952133926970&w=2
> Feedback has been received on these patches and unless otherwise noted in
> the relevant threads will be incorporated into the V2 of the patch sets.
>
Hi Denny
http://marc.info/?l=linux-rdma&m=144614769419528&w=2 introduced a
concept (RVT) which was never followed by a discussion and never
agreed upon. Moreover,
http://marc.info/?l=linux-rdma&m=144952098726776&w=2 presents a work
that besides keeping the name RVT is far from the immature concept I
mentioned earlier and its scope was changed from general purpose
solution to Intel and HFI/QIB only. I therefore conclude that the
concept of RVT, as it was supposed to be, was abandoned.
Moni
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKNOu+d-5f6FYTXCyTGjJU-md4dW3MFMB2-UDgRFafRdBw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-23 20:07 ` Dennis Dalessandro
[not found] ` <20151223200727.GA6886-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Dennis Dalessandro @ 2015-12-23 20:07 UTC (permalink / raw)
To: Moni Shoua
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
On Wed, Dec 23, 2015 at 06:28:08PM +0200, Moni Shoua wrote:
>>
>>
>> This makes no mention of the already posted work which aims to consolidate
>> the qib and hfi1 drivers verbs implementation. However it does seem to be
>> mostly in line with what we have already presented for rdmavt and the
>> direction the next set of patches is going in. Have you seen something in
>> rdmavt that contradicts this? I have not seen the need to write a detailed
>> RFC because it's pretty straight forward what needs to happen. Take the
>> common parts of qib and hfi and move to another kmod. Don't do anything that
>> prevents soft-roce from using the same infrastructure. I think we are
>> accomplishing that and have been very open during the process.
>>
>> Specifically this RFC does not capture much more detail beyond what has
>> already been posted. There are also a number of details regarding hardware
>> specifics that are not dealt with here but are in the current rdmavt patch
>> series, as well as patches that are yet to be posted. The bottom line is we
>> can't sacrifice performance for a perfect API.
>>
>> Previous discussion threads:
>> http://marc.info/?l=linux-rdma&m=144353141420065&w=2
>> http://marc.info/?l=linux-rdma&m=144563342718705&w=2
>> http://marc.info/?l=linux-rdma&m=144614769419528&w=2
>>
>> Rdmavt & Qib code submissions:
>> http://marc.info/?l=linux-rdma&m=144968107508268&w=2
>> http://marc.info/?l=linux-rdma&m=144952133926970&w=2
>> Feedback has been received on these patches and unless otherwise noted in
>> the relevant threads will be incorporated into the V2 of the patch sets.
>>
>
>Hi Denny
>
>http://marc.info/?l=linux-rdma&m=144614769419528&w=2 introduced a
>concept (RVT) which was never followed by a discussion and never
>agreed upon. Moreover
There were discussions, and Mellanox even contributed code to the effort.
See Kamal's patches in the patch set I provided.
>http://marc.info/?l=linux-rdma&m=144952098726776&w=2 presents a work
>that besides keeping the name RVT is far from the immature concept I
>mentioned earlier and its scope was changed from general purpose
>solution to Intel and HFI/QIB only.
The scope has never changed. Our goal is, and has always been to remove the
code duplication between qib and hfi1. We are doing that by way of rdmavt.
It is limited in scope to Intel's drivers currently for what I hope are
obvious reasons.
I think it makes sense that soft-roce be added as well and hope that
Mellanox decides to contribute rather than reinventing the wheel.
Is there something in rdmavt that would not work for soft-roce, or is
something fundamental missing? I have asked this a number of times and
nothing has been raised so I assume there are no issues. If there are lets
discuss them.
Reading through your RFC here, perhaps something like the multicast add
and delete is concerning? This is something that is not really needed by
qib and hfi1 but may be for soft-roce. All that means is soft-roce needs to
provide it and it would be optional for qib and hfi1. The rdmavt
architecture is flexible and allows exactly this.
>I therefore conclude that the
>concept of RVT, as it was supposed to be, was abandoned.
This is absolutely incorrect. As mentioned above, nothing has changed.
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKO=fP2Xd=JXK7hvzGFRLQmh1dF+mWDxdnhU+TqFg7YD1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-22 18:19 ` Dennis Dalessandro
@ 2015-12-23 23:15 ` ira.weiny
[not found] ` <20151223231525.GA22759-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
1 sibling, 1 reply; 16+ messages in thread
From: ira.weiny @ 2015-12-23 23:15 UTC (permalink / raw)
To: Moni Shoua
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
On Tue, Dec 22, 2015 at 02:38:10PM +0200, Moni Shoua wrote:
> Hi,
>
> In the past months the need for a kernel module that implements the InfiniBand
> transport in software and unify all the InfiniBand software drivers has
> been raised. Since then, nobody has submitted any design proposal that satisfy
> the initial thoughts and can serve various back-ends.
I'm not sure I understand what you mean. Denny has posted several high level
emails similar to this one and has asked for public feedback. We have been
addressing all the feedback as it has come in and we continue to work toward
this common layer.
To that end we have been asking for you to identify any place you see an
incompatibility with the rdmavt layer for SoftRoCE. Patches and comments to
rdmavt are encouraged and we have already incorporated patches from outside
Intel.
We are very sensitive to the fact that other drivers will want to use this
layer in the future. But I don't think this layer is set in stone. To
that end, any functionality which is specific to SoftRoCE should be added when
SoftRoCE is submitted to the kernel. To date I have not seen any show stopper
objections except Sagi's comment on the lack of IB_WR_LOCAL_INV which we will
drop in the next submission.
Also, please be aware that we are being very careful with this work to not
sacrifice the performance of either qib or hfi1. There are a couple of items
you mention below which seem to indicate you would like a more "pure"
separation of the driver. I hope you understand that any work in this area
which affects our performance must be accounted for and may not result in as
"pure" a separation as you may like. If that is a show stopper for SoftRoCE
lets work through the specific examples.
More inline.
>
> The following is a RFC that presents a solution made of a single
> generic InfiniBand
> driver and many hardware specific back-end drivers. The RFC defines
> the requirements
> and the interfaces that have to be part of any generic InfiniBand driver.
> A generic InfiniBand driver that is not compliant with this RFC wouldn't be able
> to serve different back-ends and therefore would miss its target.
>
> ================================================================================
>
> A. Introduction
> --------------------------------------------------------------------------------
> In Linux kernel, the responsibility to implement the InfiniBand protocol is
> roughly divided between 2 elements. The first are the core drivers which expose
> an abstract verbs interface to the upper layer as well as interfaces to some
> common IB services like MAD, SA and CM. The second are vendor drivers and
> hardware which implement the abstract verbs interface.
>
> A high level view of the model is in Figure A1
>
> +-----------------+
> | |
> | IB core |
> | drivers |
> | |
> +--------+--------+
> | Common
> --------------------------------------------
> | Vendor
> +--------+--------+
> | |
> | Hardware |
> | drivers |
> | |
> +--------+--------+
> | |
> | Hardware |
> | |
> +-----------------+
>
> A1 - IB implementation model in Linux kernel
>
> In the vendor part of the model, the devision of work between software and
> hardware is undefined and is usually one of the two below
>
> - Context and logic are managed in software. Hardware role is limited to
> lower layer protocols (depending on the link layer) and maybe some offloads
> - Context and logic are managed in hardware while software role is to create
> or destroy a context in the hardware and gets notified when hardware reports
> about a completions tasks.
>
> The following examples demonstrates the difference between the approaches above.
>
> - Send flow: application calls post_send() with QP and a WR. In the software
> based approach the QP context is retrieved, the WR is parsed and a proper IB
> packet is formed to be put in hardware buffers. In hardware based approach the
> driver puts the WR in hardware buffers with a handle to the QP and hardware
> does the rest.
I'm glad you brought this up but I'm a bit confused. For anything that sends
more than a single packet I think this is an oversimplification.[1] Do you
perhaps mean WQE rather than packet?
If you are referring to a WQE we were thinking the same thing but have not
gotten the send code posted yet.
[1] For example a 1MB RDMA write will require many packets.
> - Receive flow: a data packet is received and put in hardware buffers (assume
> that the destination is a RC QP). In software based approach the packet is
> handed to the driver. The driver retrieves the context by QPN, decides if
> ACK/NACK packet is required (if so, generates it) and decides if CQE is
> required on the CQ of the QP. In the other approach the hardware compares
> the packet to the state in the context, generates the ACK/NACK and raises an
> event to the driver that CQE is ready for reading.
Again I wonder if you are speaking of WQE's rather than packets. If you are
speaking of WQEs then I think we are exactly on the same page. The low level
drivers need to handle processing of the WQEs to allow for any hardware assist
which may be available.
>
> Figure A2 illustrates hardware/software relationship in the implementation of a
> IB transport solution in the software based approach.
>
> +----------------------------+
> | IB transport |
> | capable driver |
> | |
> | +----------------------+ |
> | |Context (deep) | |
> | | | |
> | +----------------------+ |
> | |
> | +----------------------+ |
> | | Logic (IB transport) | |
> | | | |
> | +----------------------+ |
> | |
> | +----------------------+ |
> | | Interface | |
> | | | |
> | +----------------------+ |
> +----------------------------+
> |
> |
> +----------------------------+
> | +----------------------+ |
> | | Interface | |
> | | | |
> | +----------------------+ |
> | |
> | +----------------------+ |
> | | Context (shallow) | |
> | | | |
> | +----------------------+ |
> | |
> | +----------------------+ |
> | | Logic (link layer) | |
> | | | |
> | +----------------------+ |
> | |
> | IB transport |
> | incapable NIC |
> +----------------------------+
> A2 - software based approach for IB transport implementation
>
> The rest of this paper is focused on IB transport solutions that are based on
> software
>
> B. Common software based IB transport solution
> --------------------------------------------------------------------------------
> In hardware based solutions the software driver implementation is tightly tied
> to the hardware. Therefore, for different hardware types there is hardly a
> common part to all software driver. In software based solutions things are
> different. For such, since hardware capabilities are limited and implement only
> the lower layer(s) of the OSI model and transport layer is done in the software.
Again this is an oversimplification. There are significant differences in the
hardware capabilities within the qib and hfi1 hardware (as I would expect for
SoftRoCE device as well). As you say elsewhere in this mail the ability for
the drivers to override functionality is necessary and is being accounted for
in rdmavt.
> In that case the common part between all drivers is quite large and is actually
> the implementation of the InfiniBand transport specification.
> The immediate conclusion is that we can take one software driver and
> theoretically match it to any hardware with some (obvious and less obvious)
> adjustments.
I agree that there is a lot of shared code here. And that is exactly what we
are working toward with rdmavt.
> Taking this one step forward yields another conclusion that only one
> IB transport software driver can be designed to work with all types of
> hardware back-ends, if we supply a well defined abstract interface
> between the IB
> transport driver and a theoretical model of hardware that all real hardware can
> fit to.
The idea of making a "theoretical model" is nice but we can not be asked to
sacrifice the performance of our hardware for a "theoretical model of
hardware". One only has to look at a comparison of qib and hfi1 to see that
hardware differences affect performance. We will not sacrifice performance to
support a nice "theoretical model" of hardware.
>
> The structure of the generic InfiniBand driver and its placement in the IB
> stack layer model is as Figure B1 describes
>
> +-----------------------------------------------+
> | |
> | IB core drivers |
> | |
> +------------^-----------+--+--+----------------+
> | | | |
> ib_register_device() | | | Abstract interface
> | | | |
> +------------+-----------v--v--v----------------+
> | Gneric IB +--------------+ |
> | driver |Abstract | |
> | |interface | |
> | +--------------+ |implementation| |
> | | Context & | +--------------+ |
> | | Logic | |
> | | | |
> | +--------------+ |
> +-----^-----------------------------^--------+--+
> | | | |
> | | API | | Abstract interface
> | | | |
> | | | |
> | | | |
> +-----+--------v------+ +---------+--------v--+
> | HW device driver A | | HW device driver B |
> | (specific) | | (specific) |
> | | | |
> +---------------------+ +---------------------+
>
> B1 - Generic InfiniBand driver structure & layout
>
> To the layer above it, ib_core, the generic driver registers a new
> ib_device and implements all the mandatory verbs(as in
> ib_device_check_mandatory()). To the layer below, the thin hardware
> specific driver, the generic driver declares an abstract interface that
> supplies API functions as we will describe later in this document.
I agree. rdmavt does this.
>
> In addition, all SW providers may share a single common user-space provider,
> since there is no direct interaction between user-space and HW.
I agree, libipathverbs and libhfi1verbs should be merged. Is there a lib
"softroce" as well? At this point however, there is no reason we have to do
this merge.
>
> C. Requirements from the generic InfiniBand driver
> -----------------------------------------------------------------------------
> To achieve its goal, be a single IB transport implementation for any back-end
> driver, the generic driver is required to the following
>
> - Offer in implementation to the InfiniBand Architecture Specification as
> described in chapters 9 (TRANSPORT LAYER) and 10 (SOFTWARE TRANSPORT
> INTERFACE)
> - Offer an implementation to the to the ib_core abstract interface as
> described in InfiniBand Architecture Specification chapter 11 (SOFTWARE
> TRANSPORT VERBS)
Agree... and again rdmavt does this.
> - Have the ability to build and validate network and link layer headers as
> described in the InfiniBand Architecture Specification chapters 7 (LINK LAYER)
> and 8 (NETWORK LAYER) and as described in Annex A16 (RoCE) and Annex A17
> (RoCEv2)
I disagree. The specifics of the link layer should be in the hardware driver.
Why would you want this to be in the software driver? Each hardware layer has
specific packet formatting requirements. Which is exactly why there is a low
level hardware driver.
> - Define an abstract interface that a hardware driver needs to implement. This
> interface needs to be general enough in the functions it requires all to
> implement but flexible enough in the functions it allows to implement.
Agree and again rdmavt allows for functions to be overridden by the hardware
driver.
> - Supply an interface for hardware drivers for the opposite direction
> interaction
Agree and again rdmavt allows for this.
>
> Note: at the time when this RFC was written there are 3 software based IB
> transport solutions: QIB, hfi1 and SoftRoCE.
Agree and each one has different packet formatting requirements.
>
> D. The abstract back-end driver
> -----------------------------------------------------------------------------
> Before we proceed to define the interface between the generic driver and the
> hardware driver we should understand what the hardware is capable of and
> what it is not. In fact, since hardware can offer many offloading features
> that we can't predict now we need categorize the functions in the abstract
> interface to two, those that must be implemented by the back-end drive and
> those that may be implemented. The most simple back-end driver, the one that
> doesn't have any offloading capabilities, will implement only the set of
> mandatory functions. Back-end driver that have some kind of offloads or require
> more complex flows can implement some of the optional functions and notify the
> generic driver about that.
Agree and again rdmavt allows for the low level drivers to override various
functions to allow for this. Of course no one can predict everything which may
be needed so we do anticipate additional functions or hooks to be implemented
as those features become available.
> For example, if a network driver has an optimization for sending small packets
> it can implement a set of functions for a small packet flow (we leave the
> question what this flow is made of unanswered here) that the generic driver will
> use. Of course, any function in the abstract interface, even if it is optional,
> should be as general as can be so it would fit as many as other
> hardware drivers.
Agree. But how is the architecture you are presenting different from what
rdmavt is already implementing? Every function has the ability to be
overridden. So each of the verbs calls could be overridden with optimized
versions for the hardware in question.
>
> Among the common capabilities that a hardware driver must implement we find
> - Detect new hardware on the PCI bus
> - Put a packet on the wire from a buffer
> - Receive a packet from the wire to a buffer
> - Capture hardware events
- processing MAD packets to monitor/configure the hardware
(How does softroce handle QP0/1 and MADs? Does it?)
>
> E. The generic InfiniBand driver lower interface
> --------------------------------------------------------------------------------
> After discussing the general model of the back-end driver we are ready to define
> the interface. From the generic driver to the hardware driver the
> interface should
> be abstracti, i.e. the generic driver defines what he expects from the driver to
> implement,mandatory or optional.
> From the hardware driver to the generic driver the relation is many to one and
> therefore the interface should have the form of an API. Tablas E1 and E2 lists
> the interfaces
>
> +------------------+---------------------------------------------------+
> | Name | Description |
> |==================|===================================================|
> |register() |tell generic driver that new hardware is present |
rvt_register_device
> |------------------|---------------------------------------------------|
> |unregister() |tell generic driver that hardware was removed |
rvt_unregister_device
> |------------------|---------------------------------------------------|
> |receive() |receive a packet from the wire |
Again I am confused by what you mean by packet. Do you mean WQE?
I think the confusion here is that what has been discussed is still very high
level and Denny has not posted the recv code yet. We absolutely agree that the
low level driver needs to notify the upper level that something has arrived.
But how this is done needs to be in a performant manner and a simple function
may be insufficient.
> |------------------|---------------------------------------------------|
> |handle_event() |notify about hardware event |
I'm not sure if this is completely necessary. If we need to isolate the low
level drivers from the verbs core we can, but currently it seems reasonable to
use the existing event infrastructure which is already defined.
> +----------------------------------------------------------------------+
>
>
> Table E1 - The back-end to generic driver interface
>
> +------------------+---------------------------------------------------+
> | Name | Description |
> |==================|===================================================|
> |send() |put packet on the wire |
Struct rvt_dev_info -> struct rvt_driver_provided -> do_send
> |------------------|---------------------------------------------------|
> |query_link() |report link state and properties |
What are the properties here? Originally we also thought this was necessary
but so far we have not needed this functionality. So it has not been added.
Does SoftRoCE report the same information about the link as IB? Are you saying
that for SoftRoCE the rdmavt layer will need to "manufacture" link information
to make it compatible with the IB core? If so then yes we can surely add this
when SoftRoCE is added and we see what those details are.
> +----------------------------------------------------------------------+
> |node_guid() |get GUID of node |
> +-----------------------------------------------------------------------
> |port_guid() |get GUID of port |
I agree this information needs to be available to the upper layer but with
rdmavt we have chosen to simply have the low level drivers fill in these
details in the rdmavt structure. This is similar to the discussion on the list
right now about removing the query_device call. If we are removing that call
why would we need calls like this in this layer?
> +-----------------------------------------------------------------------
> |mcast_add() |register multicast address |
> +-----------------------------------------------------------------------
> |mcast_del() |unregister multicast address |
rdmavt will have to handle these differently for for SoftRoCE as both hfi1 and
qib join multicast groups the "Verbs Way", in that they post MAD to a QP which
gets sent to the SM. So the joining of multicast groups is no different than
sending a UD Verbs message. I see no reason to have rdmavt nor the low level
drivers handle any of that special from what is currently done by the ib_sa
module.
How does SoftRoCE handle mcast joins now?
It sounds like you should patch rdmavt to add this function and have qib/hfi1
set this to NULL so that rdmavt ignores that functionality just like any other
call can be overridden.
> +-----------------------------------------------------------------------
> |alloc_qpn() |allocate a new QP number |
Again rdmavt is flexible enough to let you over ride this if you want...
> +-----------------------------------------------------------------------
> |get_send_buffer() |allocate a buffer for send operation |
Qib and hfi1 have DMA engines to send the data directly out of the supplied
buffer. It sounds like you need this to get an sk_buff? If so, this is yet
another difference which will need to be added when SoftRoCE is added. But it
will not be implemented or used by qib/hfi1. So at this time I don't see a
need for this function.
Thanks for the feedback,
Ira
> +-----------------------------------------------------------------------
>
> Table E2 - The generic driver to back-end driver interface
>
> Following is a detailed description for the functions above
>
> +---------------------------------------------------------------------------------+
> |Name: |register
> |
> |---------------------------------------------------------------------------------|
> |Description: |notify about a new instance of the hardware.
> |
> |---------------------------------------------------------------------------------|
> |Input: |- pointer to hardware description struct
> |
> | |- pointers to abstract interface implementation
> struct |
> |---------------------------------------------------------------------------------|
> |Output |- registration handle
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name: |unregister
> |
> |---------------------------------------------------------------------------------|
> |Description: |notify that hardware instance is gone
> |
> |---------------------------------------------------------------------------------|
> |Input: |registration handle
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name: |receive
> |
> |---------------------------------------------------------------------------------|
> |Description: |get a packet from the wire. When done notify the
> caller |
> |---------------------------------------------------------------------------------|
> |Input: |-buffer for the packet
> |
> | |-buffer length
> |
> | |-pointer to receive done function
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name: |handle_event
> |
> |---------------------------------------------------------------------------------|
> |Description: |pass hardware event for processing
> |
> |---------------------------------------------------------------------------------|
> |Input: |-registration handle of the device
> |
> | |-event type
> |
> | |-event data struct
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name: |send
> |
> |---------------------------------------------------------------------------------|
> |Description: |transmit one packet. When done, notify the caller
> |
> |---------------------------------------------------------------------------------|
> |Input: |- pointer to headers buffer
> |
> | |- headers buffer length
> |
> | |- pointer to scatter gather struct
> |
> | |- pointer to xmit_done() function
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name: |query_link
> |
> |---------------------------------------------------------------------------------|
> |Description: |query link state
> |
> |---------------------------------------------------------------------------------|
> |Input: |- device identifier
> |
> | |- port index
> |
> |---------------------------------------------------------------------------------|
> |Output |- pointer to port state struct
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name |node_guid()
> |
> |---------------------------------------------------------------------------------|
> |Description |get 64 bit node guid
> |
> |---------------------------------------------------------------------------------|
> |Input: |-device identifier
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |node GUID
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name |port_guid
> |
> |---------------------------------------------------------------------------------|
> |Description |get 64 bit
> |
> |---------------------------------------------------------------------------------|
> |Input: |-device identifier
> |
> | |-port index
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |port GUID
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name |mcast_add
> |
> |---------------------------------------------------------------------------------|
> |Description |open device for multicast address
> |
> |---------------------------------------------------------------------------------|
> |Input: |-device identifier
> |
> | |-address
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name |mcast_del
> |
> |---------------------------------------------------------------------------------|
> |Description |close device for multicast address
> |
> |---------------------------------------------------------------------------------|
> |Input: |-device identifier
> |
> | |-address
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |success/failure
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name |alloc_qpn
> |
> |---------------------------------------------------------------------------------|
> |Description |get unique QP number
> |
> |---------------------------------------------------------------------------------|
> |Input: |
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |QP number
> |
> +---------------------------------------------------------------------------------+
>
> +---------------------------------------------------------------------------------+
> |Name |get_send_buffer
> |
> |---------------------------------------------------------------------------------|
> |Description |allocate buffer that will be used to send a patcket
> |
> |---------------------------------------------------------------------------------|
> |Input: |-length
> |
> |---------------------------------------------------------------------------------|
> |Output |
> |
> |---------------------------------------------------------------------------------|
> |Return |pointer to a buffer
> |
> +---------------------------------------------------------------------------------+
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <20151223200727.GA6886-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
@ 2015-12-24 15:43 ` Moni Shoua
[not found] ` <CAG9sBKNkeA2Op-OTJ_MQ=pC3BQ7=_tB4g6oV=fwDOJ7gs_y+fQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Moni Shoua @ 2015-12-24 15:43 UTC (permalink / raw)
To: Dennis Dalessandro
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
>
>
> There were discussions, and Mellanox even contributed code to the effort.
> See Kamal's patches in the patch set I provided.
>
As far as I see it discussions were shallow and never produced an
agreement. Kamal's patches should not be considered as as such.
>> http://marc.info/?l=linux-rdma&m=144952098726776&w=2 presents a work
>> that besides keeping the name RVT is far from the immature concept I
>> mentioned earlier and its scope was changed from general purpose
>> solution to Intel and HFI/QIB only.
>
>
> The scope has never changed. Our goal is, and has always been to remove the
> code duplication between qib and hfi1. We are doing that by way of rdmavt.
> It is limited in scope to Intel's drivers currently for what I hope are
> obvious reasons.
>
So you actually agree that rdmavt was intended to be a solution to
Intel's specific drivers.
Fair, but IMO this is not what we aimed for.
In fact, if this is an Intel specific solution then why put it in
drivers/infiniband/sw and why publish it when it is not ready?
> I think it makes sense that soft-roce be added as well and hope that
> Mellanox decides to contribute rather than reinventing the wheel.
>
> Is there something in rdmavt that would not work for soft-roce, or is
> something fundamental missing? I have asked this a number of times and
> nothing has been raised so I assume there are no issues. If there are lets
> discuss them.
>
Interfaces between rdmavt and its backends are missing. I consider
this as fundamental.
Concerns were raised but answers were not provided, at least not
satisfying answers.
> Reading through your RFC here, perhaps something like the multicast add
> and delete is concerning? This is something that is not really needed by
> qib and hfi1 but may be for soft-roce. All that means is soft-roce needs to
> provide it and it would be optional for qib and hfi1. The rdmavt
> architecture is flexible and allows exactly this.
>
>> I therefore conclude that the
>> concept of RVT, as it was supposed to be, was abandoned.
>
>
> This is absolutely incorrect. As mentioned above, nothing has changed.
>
>
> -Denny
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <20151223231525.GA22759-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
@ 2015-12-24 16:07 ` Moni Shoua
0 siblings, 0 replies; 16+ messages in thread
From: Moni Shoua @ 2015-12-24 16:07 UTC (permalink / raw)
To: ira.weiny
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
> I'm not sure I understand what you mean. Denny has posted several high level
> emails similar to this one and has asked for public feedback. We have been
> addressing all the feedback as it has come in and we continue to work toward
> this common layer.
>
> To that end we have been asking for you to identify any place you see an
> incompatibility with the rdmavt layer for SoftRoCE. Patches and comments to
> rdmavt are encouraged and we have already incorporated patches from outside
> Intel.
>
I really don't know how to answer to that because I don't expect that
the stub interface to last and if it does, how the problem of code
duplication is going to be solved. When I know how interfaces look
like plus some other documentation I will be able to answer this.
> We are very sensitive to the fact that other drivers will want to use this
> layer in the future. But I don't think this layer is set in stone. To
> that end, any functionality which is specific to SoftRoCE should be added when
> SoftRoCE is submitted to the kernel. To date I have not seen any show stopper
> objections except Sagi's comment on the lack of IB_WR_LOCAL_INV which we will
> drop in the next submission.
>
SoftRoCE was submitted today to staging. Now, if I want to take it out
of staging by using rdmavt I actually can't unless I'll use the stubs
and watch for rdmavt changes. This is unacceptable. Missing final
interfaces and documentation of how to use them is a show stopper IMO
> Also, please be aware that we are being very careful with this work to not
> sacrifice the performance of either qib or hfi1. There are a couple of items
> you mention below which seem to indicate you would like a more "pure"
> separation of the driver. I hope you understand that any work in this area
> which affects our performance must be accounted for and may not result in as
> "pure" a separation as you may like. If that is a show stopper for SoftRoCE
> lets work through the specific examples.
Eventually you'll have to break WR to packets of MTU size, wouldn't you?
Anyway, this is what we had to finalize in the early discussion and
not after code posting
Ira,
I've read your comments to the RFC and I find them worth a thought
before I answer.
In general, finding a good abstraction of the back-end is the hardest
thing in this project and affects how the interface looks like.
I think that most if not all your comments fall in the category of abstraction.
>
> More inline.
>
>>
>> The following is a RFC that presents a solution made of a single
>> generic InfiniBand
>> driver and many hardware specific back-end drivers. The RFC defines
>> the requirements
>> and the interfaces that have to be part of any generic InfiniBand driver.
>> A generic InfiniBand driver that is not compliant with this RFC wouldn't be able
>> to serve different back-ends and therefore would miss its target.
>>
>> ================================================================================
>>
>> A. Introduction
>> --------------------------------------------------------------------------------
>> In Linux kernel, the responsibility to implement the InfiniBand protocol is
>> roughly divided between 2 elements. The first are the core drivers which expose
>> an abstract verbs interface to the upper layer as well as interfaces to some
>> common IB services like MAD, SA and CM. The second are vendor drivers and
>> hardware which implement the abstract verbs interface.
>>
>> A high level view of the model is in Figure A1
>>
>> +-----------------+
>> | |
>> | IB core |
>> | drivers |
>> | |
>> +--------+--------+
>> | Common
>> --------------------------------------------
>> | Vendor
>> +--------+--------+
>> | |
>> | Hardware |
>> | drivers |
>> | |
>> +--------+--------+
>> | |
>> | Hardware |
>> | |
>> +-----------------+
>>
>> A1 - IB implementation model in Linux kernel
>>
>> In the vendor part of the model, the devision of work between software and
>> hardware is undefined and is usually one of the two below
>>
>> - Context and logic are managed in software. Hardware role is limited to
>> lower layer protocols (depending on the link layer) and maybe some offloads
>> - Context and logic are managed in hardware while software role is to create
>> or destroy a context in the hardware and gets notified when hardware reports
>> about a completions tasks.
>>
>> The following examples demonstrates the difference between the approaches above.
>>
>> - Send flow: application calls post_send() with QP and a WR. In the software
>> based approach the QP context is retrieved, the WR is parsed and a proper IB
>> packet is formed to be put in hardware buffers. In hardware based approach the
>> driver puts the WR in hardware buffers with a handle to the QP and hardware
>> does the rest.
>
> I'm glad you brought this up but I'm a bit confused. For anything that sends
> more than a single packet I think this is an oversimplification.[1] Do you
> perhaps mean WQE rather than packet?
>
> If you are referring to a WQE we were thinking the same thing but have not
> gotten the send code posted yet.
>
> [1] For example a 1MB RDMA write will require many packets.
>
>
>> - Receive flow: a data packet is received and put in hardware buffers (assume
>> that the destination is a RC QP). In software based approach the packet is
>> handed to the driver. The driver retrieves the context by QPN, decides if
>> ACK/NACK packet is required (if so, generates it) and decides if CQE is
>> required on the CQ of the QP. In the other approach the hardware compares
>> the packet to the state in the context, generates the ACK/NACK and raises an
>> event to the driver that CQE is ready for reading.
>
> Again I wonder if you are speaking of WQE's rather than packets. If you are
> speaking of WQEs then I think we are exactly on the same page. The low level
> drivers need to handle processing of the WQEs to allow for any hardware assist
> which may be available.
>
>
>>
>> Figure A2 illustrates hardware/software relationship in the implementation of a
>> IB transport solution in the software based approach.
>>
>> +----------------------------+
>> | IB transport |
>> | capable driver |
>> | |
>> | +----------------------+ |
>> | |Context (deep) | |
>> | | | |
>> | +----------------------+ |
>> | |
>> | +----------------------+ |
>> | | Logic (IB transport) | |
>> | | | |
>> | +----------------------+ |
>> | |
>> | +----------------------+ |
>> | | Interface | |
>> | | | |
>> | +----------------------+ |
>> +----------------------------+
>> |
>> |
>> +----------------------------+
>> | +----------------------+ |
>> | | Interface | |
>> | | | |
>> | +----------------------+ |
>> | |
>> | +----------------------+ |
>> | | Context (shallow) | |
>> | | | |
>> | +----------------------+ |
>> | |
>> | +----------------------+ |
>> | | Logic (link layer) | |
>> | | | |
>> | +----------------------+ |
>> | |
>> | IB transport |
>> | incapable NIC |
>> +----------------------------+
>> A2 - software based approach for IB transport implementation
>>
>> The rest of this paper is focused on IB transport solutions that are based on
>> software
>>
>> B. Common software based IB transport solution
>> --------------------------------------------------------------------------------
>> In hardware based solutions the software driver implementation is tightly tied
>> to the hardware. Therefore, for different hardware types there is hardly a
>> common part to all software driver. In software based solutions things are
>> different. For such, since hardware capabilities are limited and implement only
>> the lower layer(s) of the OSI model and transport layer is done in the software.
>
> Again this is an oversimplification. There are significant differences in the
> hardware capabilities within the qib and hfi1 hardware (as I would expect for
> SoftRoCE device as well). As you say elsewhere in this mail the ability for
> the drivers to override functionality is necessary and is being accounted for
> in rdmavt.
>
>> In that case the common part between all drivers is quite large and is actually
>> the implementation of the InfiniBand transport specification.
>> The immediate conclusion is that we can take one software driver and
>> theoretically match it to any hardware with some (obvious and less obvious)
>> adjustments.
>
> I agree that there is a lot of shared code here. And that is exactly what we
> are working toward with rdmavt.
>
>> Taking this one step forward yields another conclusion that only one
>> IB transport software driver can be designed to work with all types of
>> hardware back-ends, if we supply a well defined abstract interface
>> between the IB
>> transport driver and a theoretical model of hardware that all real hardware can
>> fit to.
>
>
> The idea of making a "theoretical model" is nice but we can not be asked to
> sacrifice the performance of our hardware for a "theoretical model of
> hardware". One only has to look at a comparison of qib and hfi1 to see that
> hardware differences affect performance. We will not sacrifice performance to
> support a nice "theoretical model" of hardware.
>
>>
>> The structure of the generic InfiniBand driver and its placement in the IB
>> stack layer model is as Figure B1 describes
>>
>> +-----------------------------------------------+
>> | |
>> | IB core drivers |
>> | |
>> +------------^-----------+--+--+----------------+
>> | | | |
>> ib_register_device() | | | Abstract interface
>> | | | |
>> +------------+-----------v--v--v----------------+
>> | Gneric IB +--------------+ |
>> | driver |Abstract | |
>> | |interface | |
>> | +--------------+ |implementation| |
>> | | Context & | +--------------+ |
>> | | Logic | |
>> | | | |
>> | +--------------+ |
>> +-----^-----------------------------^--------+--+
>> | | | |
>> | | API | | Abstract interface
>> | | | |
>> | | | |
>> | | | |
>> +-----+--------v------+ +---------+--------v--+
>> | HW device driver A | | HW device driver B |
>> | (specific) | | (specific) |
>> | | | |
>> +---------------------+ +---------------------+
>>
>> B1 - Generic InfiniBand driver structure & layout
>>
>> To the layer above it, ib_core, the generic driver registers a new
>> ib_device and implements all the mandatory verbs(as in
>> ib_device_check_mandatory()). To the layer below, the thin hardware
>> specific driver, the generic driver declares an abstract interface that
>> supplies API functions as we will describe later in this document.
>
> I agree. rdmavt does this.
>
>>
>> In addition, all SW providers may share a single common user-space provider,
>> since there is no direct interaction between user-space and HW.
>
> I agree, libipathverbs and libhfi1verbs should be merged. Is there a lib
> "softroce" as well? At this point however, there is no reason we have to do
> this merge.
>
>>
>> C. Requirements from the generic InfiniBand driver
>> -----------------------------------------------------------------------------
>> To achieve its goal, be a single IB transport implementation for any back-end
>> driver, the generic driver is required to the following
>>
>> - Offer in implementation to the InfiniBand Architecture Specification as
>> described in chapters 9 (TRANSPORT LAYER) and 10 (SOFTWARE TRANSPORT
>> INTERFACE)
>> - Offer an implementation to the to the ib_core abstract interface as
>> described in InfiniBand Architecture Specification chapter 11 (SOFTWARE
>> TRANSPORT VERBS)
>
> Agree... and again rdmavt does this.
>
>> - Have the ability to build and validate network and link layer headers as
>> described in the InfiniBand Architecture Specification chapters 7 (LINK LAYER)
>> and 8 (NETWORK LAYER) and as described in Annex A16 (RoCE) and Annex A17
>> (RoCEv2)
>
> I disagree. The specifics of the link layer should be in the hardware driver.
> Why would you want this to be in the software driver? Each hardware layer has
> specific packet formatting requirements. Which is exactly why there is a low
> level hardware driver.
>
>> - Define an abstract interface that a hardware driver needs to implement. This
>> interface needs to be general enough in the functions it requires all to
>> implement but flexible enough in the functions it allows to implement.
>
> Agree and again rdmavt allows for functions to be overridden by the hardware
> driver.
>
>> - Supply an interface for hardware drivers for the opposite direction
>> interaction
>
> Agree and again rdmavt allows for this.
>
>>
>> Note: at the time when this RFC was written there are 3 software based IB
>> transport solutions: QIB, hfi1 and SoftRoCE.
>
> Agree and each one has different packet formatting requirements.
>
>>
>> D. The abstract back-end driver
>> -----------------------------------------------------------------------------
>> Before we proceed to define the interface between the generic driver and the
>> hardware driver we should understand what the hardware is capable of and
>> what it is not. In fact, since hardware can offer many offloading features
>> that we can't predict now we need categorize the functions in the abstract
>> interface to two, those that must be implemented by the back-end drive and
>> those that may be implemented. The most simple back-end driver, the one that
>> doesn't have any offloading capabilities, will implement only the set of
>> mandatory functions. Back-end driver that have some kind of offloads or require
>> more complex flows can implement some of the optional functions and notify the
>> generic driver about that.
>
> Agree and again rdmavt allows for the low level drivers to override various
> functions to allow for this. Of course no one can predict everything which may
> be needed so we do anticipate additional functions or hooks to be implemented
> as those features become available.
>
>> For example, if a network driver has an optimization for sending small packets
>> it can implement a set of functions for a small packet flow (we leave the
>> question what this flow is made of unanswered here) that the generic driver will
>> use. Of course, any function in the abstract interface, even if it is optional,
>> should be as general as can be so it would fit as many as other
>> hardware drivers.
>
> Agree. But how is the architecture you are presenting different from what
> rdmavt is already implementing? Every function has the ability to be
> overridden. So each of the verbs calls could be overridden with optimized
> versions for the hardware in question.
>
>>
>> Among the common capabilities that a hardware driver must implement we find
>> - Detect new hardware on the PCI bus
>> - Put a packet on the wire from a buffer
>> - Receive a packet from the wire to a buffer
>> - Capture hardware events
>
> - processing MAD packets to monitor/configure the hardware
> (How does softroce handle QP0/1 and MADs? Does it?)
>
>>
>> E. The generic InfiniBand driver lower interface
>> --------------------------------------------------------------------------------
>> After discussing the general model of the back-end driver we are ready to define
>> the interface. From the generic driver to the hardware driver the
>> interface should
>> be abstracti, i.e. the generic driver defines what he expects from the driver to
>> implement,mandatory or optional.
>> From the hardware driver to the generic driver the relation is many to one and
>> therefore the interface should have the form of an API. Tablas E1 and E2 lists
>> the interfaces
>>
>> +------------------+---------------------------------------------------+
>> | Name | Description |
>> |==================|===================================================|
>> |register() |tell generic driver that new hardware is present |
>
> rvt_register_device
>
>> |------------------|---------------------------------------------------|
>> |unregister() |tell generic driver that hardware was removed |
>
> rvt_unregister_device
>
>> |------------------|---------------------------------------------------|
>> |receive() |receive a packet from the wire |
>
> Again I am confused by what you mean by packet. Do you mean WQE?
>
> I think the confusion here is that what has been discussed is still very high
> level and Denny has not posted the recv code yet. We absolutely agree that the
> low level driver needs to notify the upper level that something has arrived.
> But how this is done needs to be in a performant manner and a simple function
> may be insufficient.
>
>> |------------------|---------------------------------------------------|
>> |handle_event() |notify about hardware event |
>
> I'm not sure if this is completely necessary. If we need to isolate the low
> level drivers from the verbs core we can, but currently it seems reasonable to
> use the existing event infrastructure which is already defined.
>
>> +----------------------------------------------------------------------+
>>
>>
>> Table E1 - The back-end to generic driver interface
>>
>> +------------------+---------------------------------------------------+
>> | Name | Description |
>> |==================|===================================================|
>> |send() |put packet on the wire |
>
> Struct rvt_dev_info -> struct rvt_driver_provided -> do_send
>
>> |------------------|---------------------------------------------------|
>> |query_link() |report link state and properties |
>
> What are the properties here? Originally we also thought this was necessary
> but so far we have not needed this functionality. So it has not been added.
> Does SoftRoCE report the same information about the link as IB? Are you saying
> that for SoftRoCE the rdmavt layer will need to "manufacture" link information
> to make it compatible with the IB core? If so then yes we can surely add this
> when SoftRoCE is added and we see what those details are.
>
>> +----------------------------------------------------------------------+
>> |node_guid() |get GUID of node |
>> +-----------------------------------------------------------------------
>> |port_guid() |get GUID of port |
>
> I agree this information needs to be available to the upper layer but with
> rdmavt we have chosen to simply have the low level drivers fill in these
> details in the rdmavt structure. This is similar to the discussion on the list
> right now about removing the query_device call. If we are removing that call
> why would we need calls like this in this layer?
>
>> +-----------------------------------------------------------------------
>> |mcast_add() |register multicast address |
>> +-----------------------------------------------------------------------
>> |mcast_del() |unregister multicast address |
>
> rdmavt will have to handle these differently for for SoftRoCE as both hfi1 and
> qib join multicast groups the "Verbs Way", in that they post MAD to a QP which
> gets sent to the SM. So the joining of multicast groups is no different than
> sending a UD Verbs message. I see no reason to have rdmavt nor the low level
> drivers handle any of that special from what is currently done by the ib_sa
> module.
>
> How does SoftRoCE handle mcast joins now?
>
> It sounds like you should patch rdmavt to add this function and have qib/hfi1
> set this to NULL so that rdmavt ignores that functionality just like any other
> call can be overridden.
>
>
>> +-----------------------------------------------------------------------
>> |alloc_qpn() |allocate a new QP number |
>
> Again rdmavt is flexible enough to let you over ride this if you want...
>
>> +-----------------------------------------------------------------------
>> |get_send_buffer() |allocate a buffer for send operation |
>
> Qib and hfi1 have DMA engines to send the data directly out of the supplied
> buffer. It sounds like you need this to get an sk_buff? If so, this is yet
> another difference which will need to be added when SoftRoCE is added. But it
> will not be implemented or used by qib/hfi1. So at this time I don't see a
> need for this function.
>
> Thanks for the feedback,
> Ira
>
>
>> +-----------------------------------------------------------------------
>>
>> Table E2 - The generic driver to back-end driver interface
>>
>> Following is a detailed description for the functions above
>>
>> +---------------------------------------------------------------------------------+
>> |Name: |register
>> |
>> |---------------------------------------------------------------------------------|
>> |Description: |notify about a new instance of the hardware.
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |- pointer to hardware description struct
>> |
>> | |- pointers to abstract interface implementation
>> struct |
>> |---------------------------------------------------------------------------------|
>> |Output |- registration handle
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name: |unregister
>> |
>> |---------------------------------------------------------------------------------|
>> |Description: |notify that hardware instance is gone
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |registration handle
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name: |receive
>> |
>> |---------------------------------------------------------------------------------|
>> |Description: |get a packet from the wire. When done notify the
>> caller |
>> |---------------------------------------------------------------------------------|
>> |Input: |-buffer for the packet
>> |
>> | |-buffer length
>> |
>> | |-pointer to receive done function
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name: |handle_event
>> |
>> |---------------------------------------------------------------------------------|
>> |Description: |pass hardware event for processing
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |-registration handle of the device
>> |
>> | |-event type
>> |
>> | |-event data struct
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name: |send
>> |
>> |---------------------------------------------------------------------------------|
>> |Description: |transmit one packet. When done, notify the caller
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |- pointer to headers buffer
>> |
>> | |- headers buffer length
>> |
>> | |- pointer to scatter gather struct
>> |
>> | |- pointer to xmit_done() function
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name: |query_link
>> |
>> |---------------------------------------------------------------------------------|
>> |Description: |query link state
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |- device identifier
>> |
>> | |- port index
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |- pointer to port state struct
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name |node_guid()
>> |
>> |---------------------------------------------------------------------------------|
>> |Description |get 64 bit node guid
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |-device identifier
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |node GUID
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name |port_guid
>> |
>> |---------------------------------------------------------------------------------|
>> |Description |get 64 bit
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |-device identifier
>> |
>> | |-port index
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |port GUID
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name |mcast_add
>> |
>> |---------------------------------------------------------------------------------|
>> |Description |open device for multicast address
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |-device identifier
>> |
>> | |-address
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name |mcast_del
>> |
>> |---------------------------------------------------------------------------------|
>> |Description |close device for multicast address
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |-device identifier
>> |
>> | |-address
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |success/failure
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name |alloc_qpn
>> |
>> |---------------------------------------------------------------------------------|
>> |Description |get unique QP number
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |QP number
>> |
>> +---------------------------------------------------------------------------------+
>>
>> +---------------------------------------------------------------------------------+
>> |Name |get_send_buffer
>> |
>> |---------------------------------------------------------------------------------|
>> |Description |allocate buffer that will be used to send a patcket
>> |
>> |---------------------------------------------------------------------------------|
>> |Input: |-length
>> |
>> |---------------------------------------------------------------------------------|
>> |Output |
>> |
>> |---------------------------------------------------------------------------------|
>> |Return |pointer to a buffer
>> |
>> +---------------------------------------------------------------------------------+
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKNkeA2Op-OTJ_MQ=pC3BQ7=_tB4g6oV=fwDOJ7gs_y+fQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-24 16:14 ` Dennis Dalessandro
[not found] ` <20151224161415.GA674-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Dennis Dalessandro @ 2015-12-24 16:14 UTC (permalink / raw)
To: Moni Shoua
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
On Thu, Dec 24, 2015 at 05:43:11PM +0200, Moni Shoua wrote:
>>
>>
>> There were discussions, and Mellanox even contributed code to the effort.
>> See Kamal's patches in the patch set I provided.
>>
>As far as I see it discussions were shallow and never produced an
>agreement. Kamal's patches should not be considered as as such.
Point is others have looked at the code. No issues have been called out to
date as to why what is there won't work for everyone.
>>> http://marc.info/?l=linux-rdma&m=144952098726776&w=2 presents a work
>>> that besides keeping the name RVT is far from the immature concept I
>>> mentioned earlier and its scope was changed from general purpose
>>> solution to Intel and HFI/QIB only.
>>
>>
>> The scope has never changed. Our goal is, and has always been to remove the
>> code duplication between qib and hfi1. We are doing that by way of rdmavt.
>> It is limited in scope to Intel's drivers currently for what I hope are
>> obvious reasons.
>>
>So you actually agree that rdmavt was intended to be a solution to
>Intel's specific drivers.
>Fair, but IMO this is not what we aimed for.
>In fact, if this is an Intel specific solution then why put it in
>drivers/infiniband/sw and why publish it when it is not ready?
Yes it is specific to Intel *now*, that doesn't mean it should stay that
way. Rdmavt could, and in my opinion should, be extended to support
soft-roce. I don't think replicating the same thing is a great idea.
As to the location, where do you think it should go. drivers/infiniband/sw
makes the most sense to me, but open to suggestions.
And for the question of why publish when it's not ready, the better question
is why not? Is it not good to see the work in progress as it evolves so the
community can provide feedback?
>> I think it makes sense that soft-roce be added as well and hope that
>> Mellanox decides to contribute rather than reinventing the wheel.
>>
>> Is there something in rdmavt that would not work for soft-roce, or is
>> something fundamental missing? I have asked this a number of times and
>> nothing has been raised so I assume there are no issues. If there are lets
>> discuss them.
>>
>Interfaces between rdmavt and its backends are missing. I consider
>this as fundamental.
>Concerns were raised but answers were not provided, at least not
>satisfying answers.
No one is arguing that. It is a work in progress and will get there. More
details are in in Ira's response.
http://marc.info/?l=linux-rdma&m=145091253118395&w=2
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <20151224161415.GA674-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
@ 2015-12-27 17:54 ` Moni Shoua
[not found] ` <CAG9sBKNBDC=YymoYMLjW+RceChQXfNMXcc8UzJJVhqBuW5jGAQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Moni Shoua @ 2015-12-27 17:54 UTC (permalink / raw)
To: Dennis Dalessandro
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
>
> Point is others have looked at the code. No issues have been called out to
> date as to why what is there won't work for everyone.
>
http://marc.info/?l=linux-rdma&m=144968107508268&w=2
Your answer to the send() issue is first an agreement with the comment
and later says that it can't be done because of how PIO and SDMA
(Intel specific implementation)
This is an example for a discussion that never ended with an agreement.
>
> Yes it is specific to Intel *now*, that doesn't mean it should stay that
> way. Rdmavt could, and in my opinion should, be extended to support
> soft-roce. I don't think replicating the same thing is a great idea.
>
But you post *now* a so called generic driver so it must now fit any
possible driver (including Soft RoCE)
> As to the location, where do you think it should go. drivers/infiniband/sw
> makes the most sense to me, but open to suggestions.
>
> And for the question of why publish when it's not ready, the better question
> is why not? Is it not good to see the work in progress as it evolves so the
> community can provide feedback?
>
What kind of a feedback you expect when I don't have an idea about
your plans for rdmavt
Interfaces, flows, data structures... all is missing from the
documentation to rdmavt.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKNBDC=YymoYMLjW+RceChQXfNMXcc8UzJJVhqBuW5jGAQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-28 20:32 ` Dennis Dalessandro
[not found] ` <20151228203249.GA4213-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-29 9:21 ` Christoph Hellwig
2015-12-29 16:37 ` Doug Ledford
2 siblings, 1 reply; 16+ messages in thread
From: Dennis Dalessandro @ 2015-12-28 20:32 UTC (permalink / raw)
To: Moni Shoua
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
On Sun, Dec 27, 2015 at 07:54:46PM +0200, Moni Shoua wrote:
>>
>> Point is others have looked at the code. No issues have been called out to
>> date as to why what is there won't work for everyone.
>>
>http://marc.info/?l=linux-rdma&m=144968107508268&w=2
>Your answer to the send() issue is first an agreement with the comment
>and later says that it can't be done because of how PIO and SDMA
>(Intel specific implementation)
>This is an example for a discussion that never ended with an agreement.
No. PIO and SDMA is driver specific and lives in the driver. Rdmavt has no
concept of this. I'm agreeing that the send will be generic and have no hw
specific stuff.
>> Yes it is specific to Intel *now*, that doesn't mean it should stay that
>> way. Rdmavt could, and in my opinion should, be extended to support
>> soft-roce. I don't think replicating the same thing is a great idea.
>>
>But you post *now* a so called generic driver so it must now fit any
>possible driver (including Soft RoCE)
As I've stated a number of times across multiple threads: It must not do
anything that would prevent another driver from using it.
>> As to the location, where do you think it should go. drivers/infiniband/sw
>> makes the most sense to me, but open to suggestions.
>>
>> And for the question of why publish when it's not ready, the better question
>> is why not? Is it not good to see the work in progress as it evolves so the
>> community can provide feedback?
>>
>What kind of a feedback you expect when I don't have an idea about
>your plans for rdmavt
>Interfaces, flows, data structures... all is missing from the
>documentation to rdmavt.
I expect feedback based on the code submissions. More will be coming
shortly. I have taken all the feedback from the first post and will be
sending a v2 shortly.
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKNBDC=YymoYMLjW+RceChQXfNMXcc8UzJJVhqBuW5jGAQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-28 20:32 ` Dennis Dalessandro
@ 2015-12-29 9:21 ` Christoph Hellwig
[not found] ` <20151229092126.GB8445-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-12-29 16:37 ` Doug Ledford
2 siblings, 1 reply; 16+ messages in thread
From: Christoph Hellwig @ 2015-12-29 9:21 UTC (permalink / raw)
To: Moni Shoua
Cc: Dennis Dalessandro, Doug Ledford, Liran Liss, Haggai Eran,
Majd Dibbiny, Kamal Heib, linux-rdma
Hi Moni,
On Sun, Dec 27, 2015 at 07:54:46PM +0200, Moni Shoua wrote:
> But you post *now* a so called generic driver so it must now fit any
> possible driver (including Soft RoCE)
it's never going to fit any possible future driver. Dennis and folks
have done great work to move code outside the drivers into a shared
library. So far it's been driven just by the Intel drivers as that's
the only thing they were interested in.
If you are interested in supporting SoftROCE please work with them
by adjusting the code towards your requirements. In Linux we have
great results with iterative appoaches and I'd suggest you try it
as well.
> What kind of a feedback you expect when I don't have an idea about
> your plans for rdmavt
> Interfaces, flows, data structures... all is missing from the
> documentation to rdmavt.
You've got the code, so let's work based on that.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <20151229092126.GB8445-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
@ 2015-12-29 13:21 ` Moni Shoua
0 siblings, 0 replies; 16+ messages in thread
From: Moni Shoua @ 2015-12-29 13:21 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Dennis Dalessandro, Doug Ledford, Liran Liss, Haggai Eran,
Majd Dibbiny, Kamal Heib, linux-rdma
> it's never going to fit any possible future driver. Dennis and folks
> have done great work to move code outside the drivers into a shared
> library. So far it's been driven just by the Intel drivers as that's
> the only thing they were interested in.
>
If it's not going to be a solution for anything else but Intel then
why declare it as such?
Where is that shared library? There amount of shared code in rdmavt
that can be considered as shared is very little.
> If you are interested in supporting SoftROCE please work with them
> by adjusting the code towards your requirements. In Linux we have
> great results with iterative appoaches and I'd suggest you try it
> as well.
>
Exactly. All you asked for is in the RFC I posted.
>
> You've got the code, so let's work based on that.
> --
I say let's agree on the interfaces and start writing code.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <20151228203249.GA4213-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
@ 2015-12-29 14:03 ` Moni Shoua
0 siblings, 0 replies; 16+ messages in thread
From: Moni Shoua @ 2015-12-29 14:03 UTC (permalink / raw)
To: Dennis Dalessandro
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
> No. PIO and SDMA is driver specific and lives in the driver. Rdmavt has no
> concept of this. I'm agreeing that the send will be generic and have no hw
> specific stuff.
>
I understand that PIO/SDMA are not a concept of RVT. However, making
the send from RVT to driver exactly as the interface from ib_core to
RVT raises the question: What exactly do we achieve by this?
>
>
> As I've stated a number of times across multiple threads: It must not do
> anything that would prevent another driver from using it.
>
The question is not how Soft RoCE fits into this framework but how
does this framework achieve its goals.
>
>
> I expect feedback based on the code submissions. More will be coming
> shortly. I have taken all the feedback from the first post and will be
> sending a v2 shortly.
>
Again, I have no idea about the complete interfaces between both
pieces of the suggested solution.
- If you have them then please publish
- if you don't but plan to have them then why did you submit a half baked idea
- If you say that final interface is what we see now then I say that
the problem of code duplication isn't going to be resolved
So, what it is from the 3?
>
> -Denny
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKNBDC=YymoYMLjW+RceChQXfNMXcc8UzJJVhqBuW5jGAQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-28 20:32 ` Dennis Dalessandro
2015-12-29 9:21 ` Christoph Hellwig
@ 2015-12-29 16:37 ` Doug Ledford
[not found] ` <5682B6B4.60306-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2 siblings, 1 reply; 16+ messages in thread
From: Doug Ledford @ 2015-12-29 16:37 UTC (permalink / raw)
To: Moni Shoua, Dennis Dalessandro
Cc: Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib, linux-rdma
[-- Attachment #1: Type: text/plain, Size: 2357 bytes --]
On 12/27/2015 12:54 PM, Moni Shoua wrote:
>> Yes it is specific to Intel *now*, that doesn't mean it should stay that
>> way. Rdmavt could, and in my opinion should, be extended to support
>> soft-roce. I don't think replicating the same thing is a great idea.
>>
> But you post *now* a so called generic driver so it must now fit any
> possible driver (including Soft RoCE)
This is incorrect. This isn't some public API that we are exporting to
user space. Nor is it an API that out of tree drivers are using. This
is a purely kernel internal API for use by a limited number of drivers.
As such, it need not be finalized before it is submitted or used. It
can be taken one piece at a time, and if, at some point, it is
determined that there are shortcomings to the API, it can be updated in
place with all of the drivers that use it in a single patch or patch
series. So a finalized design prior to putting code in place is
specifically *not* needed.
>> As to the location, where do you think it should go. drivers/infiniband/sw
>> makes the most sense to me, but open to suggestions.
>>
>> And for the question of why publish when it's not ready, the better question
>> is why not? Is it not good to see the work in progress as it evolves so the
>> community can provide feedback?
>>
> What kind of a feedback you expect when I don't have an idea about
> your plans for rdmavt
> Interfaces, flows, data structures... all is missing from the
> documentation to rdmavt.
They released it so that you can start hooking SoftRoCE into it. As you
hook it in, if it needs changes to work with SoftRoCE, simply make the
changes needed and move on.
I think Dennis' point, and I agree with him, is that you are over
complicating the issue here. This need not be a highly designed item,
it needs to be a functional item, and we can build it as we go. If you
have to make changes to rdmavt in order to hook up SoftRoCE, that's
fine, post them to the list, they will get reviewed. As long as the
change doesn't break or otherwise negatively impact qib and/or hfi1,
then it should be fine. If it does, then I'm sure Intel will work with
you to find a solution that doesn't negatively impact them.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <5682B6B4.60306-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2015-12-29 17:38 ` Moni Shoua
[not found] ` <CAG9sBKMoXneqQJvMWKXkTXx-D2WrLuabW=5hCH_5V=TePe4WYQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 16+ messages in thread
From: Moni Shoua @ 2015-12-29 17:38 UTC (permalink / raw)
To: Doug Ledford
Cc: Dennis Dalessandro, Liran Liss, Haggai Eran, Majd Dibbiny,
Kamal Heib, linux-rdma
I think that my point is missed. See my answers inline
> This is incorrect. This isn't some public API that we are exporting to
> user space. Nor is it an API that out of tree drivers are using. This
> is a purely kernel internal API for use by a limited number of drivers.
> As such, it need not be finalized before it is submitted or used. It
> can be taken one piece at a time, and if, at some point, it is
> determined that there are shortcomings to the API, it can be updated in
> place with all of the drivers that use it in a single patch or patch
> series. So a finalized design prior to putting code in place is
> specifically *not* needed.
>
This is not a question of future backward comparability where
interfaces must be kept forever. I agree that kernel interfaces may be
changed with kernel moving forward. However, this is not what I'm
arguing against.
When one submits a RFC for a generic Infrastructure he must state what
are the interfaces between blocks of the design.
Soft RoCE block can't start until I know how the final interfaces look
like. This is an unacceptable method of work.
>
> They released it so that you can start hooking SoftRoCE into it. As you
> hook it in, if it needs changes to work with SoftRoCE, simply make the
> changes needed and move on.
This is not a question if I can hook Soft RoCE driver into this framework.
In fact, I can't think of an IB driver that can't use this framework. What this
framework offers is just another hop from ib_core the real driver.
Where is the removal of duplicated code? This is a list of functions
that for now
must be implemented in the low level driver.
create_cq
destroy_cq
poll_cq
req_notify_cq
resize_cq
create_srq
modify_srq
destroy_srq
query_srq
create_qp
query_device
query_gid
alloc_ucontext
modify_device
modify_qp
dealloc_ucontext
query_port
destroy_qp
get_port_immutable
modify_port
query_qp
post_send
post_recv
post_srq_recv
Most if not all of them have common part in all drivers.
What are the plans to get rid of them? When?
Don't you think that this should be known in advance?
I already asked and never been answered seriously: what was
the purpose of the submission in this premature state of the code
It can't be for feedback because what kind of feedback can you provide
for just a skeleton? Moreover, today they submitted V2 with a changelog
that is almost 100% cosmetic changes. I really don't understand this kind
of work.
>
> I think Dennis' point, and I agree with him, is that you are over
> complicating the issue here. This need not be a highly designed item,
> it needs to be a functional item, and we can build it as we go. If you
> have to make changes to rdmavt in order to hook up SoftRoCE, that's
> fine, post them to the list, they will get reviewed. As long as the
> change doesn't break or otherwise negatively impact qib and/or hfi1,
> then it should be fine. If it does, then I'm sure Intel will work with
> you to find a solution that doesn't negatively impact them.
A reminder of what the initial goal was - remove code duplicates between
all IB transport drivers. This goal is complicated and in my RFC I explained
why. So, for start, I am not complicating anything that was simple before.
Second, what you are saying here is actually: "this is a project to serves
Intel's needs". So why treat it as a generic infrastructure? I'm not aiming to
hurt performance but Intel should aim for achieving the goals we agreed on
in the begging.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] Generic InfiniBand transport done in software
[not found] ` <CAG9sBKMoXneqQJvMWKXkTXx-D2WrLuabW=5hCH_5V=TePe4WYQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-29 17:55 ` Dennis Dalessandro
0 siblings, 0 replies; 16+ messages in thread
From: Dennis Dalessandro @ 2015-12-29 17:55 UTC (permalink / raw)
To: Moni Shoua
Cc: Doug Ledford, Liran Liss, Haggai Eran, Majd Dibbiny, Kamal Heib,
linux-rdma
On Tue, Dec 29, 2015 at 07:38:30PM +0200, Moni Shoua wrote:
>This is not a question if I can hook Soft RoCE driver into this framework.
>In fact, I can't think of an IB driver that can't use this framework. What
>this
>framework offers is just another hop from ib_core the real driver.
>Where is the removal of duplicated code? This is a list of functions
>that for now
>must be implemented in the low level driver.
>
>create_cq
>destroy_cq
>poll_cq
>req_notify_cq
>resize_cq
>create_srq
>modify_srq
>destroy_srq
>query_srq
>create_qp
>query_device
>query_gid
>alloc_ucontext
>modify_device
>modify_qp
>dealloc_ucontext
>query_port
>destroy_qp
>get_port_immutable
>modify_port
>query_qp
>post_send
>post_recv
>post_srq_recv
>
>Most if not all of them have common part in all drivers.
>What are the plans to get rid of them? When?
>Don't you think that this should be known in advance?
We have patch sets that implement all of these which will be posted soon.
With the holidays things have just been a bit slow to come out.
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2015-12-29 17:55 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-12-22 12:38 [RFC] Generic InfiniBand transport done in software Moni Shoua
[not found] ` <CAG9sBKO=fP2Xd=JXK7hvzGFRLQmh1dF+mWDxdnhU+TqFg7YD1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-22 18:19 ` Dennis Dalessandro
[not found] ` <20151222181905.GA742-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-23 16:28 ` Moni Shoua
[not found] ` <CAG9sBKNOu+d-5f6FYTXCyTGjJU-md4dW3MFMB2-UDgRFafRdBw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-23 20:07 ` Dennis Dalessandro
[not found] ` <20151223200727.GA6886-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-24 15:43 ` Moni Shoua
[not found] ` <CAG9sBKNkeA2Op-OTJ_MQ=pC3BQ7=_tB4g6oV=fwDOJ7gs_y+fQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-24 16:14 ` Dennis Dalessandro
[not found] ` <20151224161415.GA674-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-27 17:54 ` Moni Shoua
[not found] ` <CAG9sBKNBDC=YymoYMLjW+RceChQXfNMXcc8UzJJVhqBuW5jGAQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-28 20:32 ` Dennis Dalessandro
[not found] ` <20151228203249.GA4213-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-29 14:03 ` Moni Shoua
2015-12-29 9:21 ` Christoph Hellwig
[not found] ` <20151229092126.GB8445-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2015-12-29 13:21 ` Moni Shoua
2015-12-29 16:37 ` Doug Ledford
[not found] ` <5682B6B4.60306-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2015-12-29 17:38 ` Moni Shoua
[not found] ` <CAG9sBKMoXneqQJvMWKXkTXx-D2WrLuabW=5hCH_5V=TePe4WYQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-29 17:55 ` Dennis Dalessandro
2015-12-23 23:15 ` ira.weiny
[not found] ` <20151223231525.GA22759-W4f6Xiosr+yv7QzWx2u06xL4W9x8LtSr@public.gmane.org>
2015-12-24 16:07 ` Moni Shoua
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox