Re: [PATCH V4 for-next 1/5] IB/core: Add RSS and TSS QP groups - suggesting BOF during OFA conf to further discuss that

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Thu, Apr 25, 2013 at 11:56:16PM +0300, Or Gerlitz wrote:

> > Ah, this seems contrary to the IPoIB specification? Someone should
> > probably talk about how sending from the wrong QPN is acceptable..

> AFAIK the IPoIB specification doesn't mandate the QPN of the sender

I'd have to read it again very carefully.. However, checking the src
QPN of every UD packet is the only way to detect if the packet was
generated by the authentic kernel or from an unprivileged user space
process, so there is a certainly importance in the value.

But I don't follow why the send QPNs have to be sequential for
IPoIB. It looks like this is being motivated by RSS and RSS QPNs are
just being reused for TSS?

> > As I said, that is ugly. 'TSS' that changes the on-the-wire packet
> > is not TSS. It is just ganging QPs together.
> >
> > Allocating sequential TSS QPNs is an awful hack, what we really
> > need is a way to force a UD QP's outgoing QPN.
> 
> INDEED, but this must be supported by the HW. The patch set is already
> supporting the case of HW the knows to do that forcing, quoting  --->
> IB_DEVICE_UD_TSS which is set to indicate that the device supports "HW
> TSS" which means that the HW is capable of over-riding the source UD
> QPN present in sent IB datagram header (DTH) with the parent's QPN
> <--- where over such HW the on-the-wire IPoIB header isn't touched.

For the TSS case, I'd say just allocate normal QPs and provide
something like ibv_override_ud_src_qpn(). This is very general and
broadly useful for any application using UD QPs.

> BUT for the sake of improving performance and being competitive with
> tons of Linux Ethernet drivers that support TSS/MQ we still need IPoIB
> to support MQ/TSS before such HW is introduced, and as such the chosen
> solution was to use reserved fields of the wire header.

You've lost me again, what reserved bits?

If a new uverb is introduced the on-the-wire behaviour needs to be
fully documented..

> How about we discuss RSS 1st? for RSS no wire change is introduced,
> lets see if/how we can come to an agreement how the RSS related verbs
> should look like and we'll take it from there to TSS.

Well, to me, TSS is pretty simple. RSS is where things got really
complicated..

As Sean said earlier, please think about a single QP, multiple RQ/SQ
style API - that seems much more general to me and also could
reasonably be defined for other transport types.

For instance, someday supporting multiple RQ on a RC transport, with
content-based steering, is a limited form of tag matching.. From a
longer-term user space API design standpoint the concept seems to have
more longevity.

Also, I feel what happens inside the kernel is more flexable API
wise, so dropping the uverbs component may also be something you want
to look at.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html