[RFC] RDMA verbs transport design notes

[Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>]

Hi Folks,

I had previously posted a notice about the very beginnings of the rdmavt 
driver which is the software verbs consolidation for multiple drivers [1].  
I have  now pushed another set of updates to a GitHub repo [2] which 
contains more details.  What this latest batch of patches entail is a 
stubbed out version with annotations in the comments as to what the 
interaction between drivers and rdmavt will look like. Look for lines like:

VT-DRIVER-API

The following is a summary of the current posted code and the direction 
which we are thinking of going based on knowledge of qib and hfi1 drivers.  
Feedback and suggestions are welcome.

Design Goals
------------
- Remove duplication of software verbs code present in multiple drivers.
- Do not regress performance.

Registration and general code flow
----------------------------------
Instead of registering directly with the IB core like they do now, drivers 
will register with the rdmavt, referred to as rvt in the code. Drivers will 
build up the ib_device_attr and pass in to the registration by way of the 
rvt_dev_info struct. This will also contain any other driver specific 
settings that rvt will need to know about.

Currently allocation of the ib_device is done by the driver. This is merely 
a stepping stone, and eventually the allocation will move up to rvt. The 
driver should not need to know about the ib_device structure eventually, 
other than for those functions it chooses to override

In addition to describing its properties, drivers will supply a mapping of 
function pointers for use by rvt. The idea is that most of the verbs code 
lives in rvt, but there are some device specific functions which drivers 
will need to perform, such as pushing packets to the wire. Rvt will 
accomplish its tasks by calling into the drivers for these.

There are also times when drivers will need to call back into rvt. We 
should aim to limit this as much as possible. For things like a packet 
arriving from the wire we have no choice but for the driver to initiate the 
processing and call into (or signal in some way) rvt.

Driver override
---------------
Drivers need to be able to override functions that would normally be done 
by the rvt. In the current set of patches this is accomplished by filling 
in a value in the ib_device_attr function pointer map. If the value is NULL 
then rvt uses its function, otherwise rvt is bypassed by the core and the 
driver is called directly.  Performance optimizations could be one reason, 
another is incremental development. We can work on moving a driver over to 
rvt in stages.

Driver provided functionality
-----------------------------
This list will likely grow as the code evolves but as a first pass through 
these are the things which I see as needing to be provided by the driver:

query_port_state()
	Returns pretty much what is in ib_port_attr
	Will differ based on driver

set_link_state()
	For rvt to have the driver set the state of the link

get_lid()
	Provides the LID

qp_mtu()
	Using the SL determines the MTU (this varies per VL in OPA)

make_qpn()
	QPN ranges differ for drivers

flush_qp()
	Flush out all pending operations for a QP that have not made it the 
	wire, and wait for that flush to finish.

do_send()
	Take a fully constructed packet and place on the wire

Other functions for things like maintaining MAD counters perhaps.

Driver notification or upcall to rvt
------------------------------------
Certain things will require the driver to notify the rvt or execute some 
function. For instance, the driver needs to hand the packet to rvt after it 
pulls it off the wire.

There are also event which the driver needs to let rvt know have happened.  
Things that currently generate IB_EVENT_PORT_ERROR, or 
IB_EVENT_PORT_ACTIVE, etc. There are likely other events as well.

Next steps
----------
We will continue posting code to GitHub [2] while we field feedback. Note
the repo has been moved from my previous announcement [1]. I have placed it
under my GitHub. Once there is more significant development and folks are
generally happy with the design we will begin posting to this mailing list
(linux-rdma).

The current branch on [2] is rdmavt-v1. I'll bump this whenever a rebase
is needed.

[1] http://marc.info/?l=linux-rdma&m=144563342718705&w=2
[2] https://github.com/ddalessa/kernel/tree/rdmavt-v1

Thanks

-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html