Re: [PATCH V4 libibverbs 2/7] Add member functions to poll an extended CQ

Doug Ledford
2016-05-30T13:50:07Z

[-- Attachment #1: Type: text/plain, Size: 3204 bytes --]

Resend because of my mailer...

On 05/30/2016 09:46 AM, Doug Ledford wrote:
> On 05/30/2016 03:47 AM, Matan Barak wrote:
>> On 30/05/2016 02:47, Doug Ledford wrote:
>
>>> Today's version has a data copy. What I'm suggesting here eliminates
>>> that entirely and all of the branches can be optimized away at compile
>>> time, leaving a load from a hot cache line of an offset value and then
>>> a direct load from that offset in the hcqe. I would think that because
>>> of the elimination of the data copy and the compile time elimination
>>> of branches, this has the potential to meet or exceed today's
>>> implementation. I would certainly like to see the numbers.
>>>
>>
>> Not necessarily, smart compiler could fetch the data straight from the
>> hardware's CQE and pass it through a register.
>
> Not while still maintaining multi-device capability. What I laid out
> works without having to compile for a specific piece of hardware.
>
>>>> This is a whole picture optimization and the user side is only one
>>>> part of the equation - the function pointer scheme is also optimizing
>>>> the driver path quite heavily, which is why it is showing positively
>>>> in benchmarks.
>>>
>>> Passing an opaque hcqe pointer to the user and then only getting the
>>> data they want out of it should similarly help optimize the driver's
>>> path I would think.
>>>
>>
>> What if the driver's CQE format has a field which is split between two
>> different offsets?
>
> I have an idea for that as well.
>
>> We looked at storing offsets and masks before implementing this and
>> supplying inline functions to get the various fields, but IMHO, that's a
>> no go as it's not generic enough.
>>
>> Anyway, if you really want to tie yourself to one vendor, maybe we could
>> allow static linking with the user-space driver and by using lto you
>> could get inline-like function behavior.
>
> No, what I'm outlining works with multi-vendor support. It's opaque to
> the user application.
>
> Now, as to how to handle complex fields.
>
> I was thinking of making the wc_elements enum encode some complex
> information. Right now, it's just the offset number. It would be
> possible to set each particular element as either A) simple offset, B)
> possibly complex (which means some vendors can do a simple offset and
> others need a complex function), or C) all complex. Then the #define
> can be made smart enough to know the difference and on simple items
> return the data, on possibly complex items you have an if statement and
> branch so we avoid this when possible but allows different hardware
> types to be handled, and on always complex items we are able to optimize
> down to just the call in the compiler. Although, I freely admit that if
> you have more than just a handful of these complex items then it likely
> becomes cleaner to just do the "function call per wc element" way of
> doing things like your patch has. But I doubt there are that many
> complex items in your CQE. Do you have a count?
>
>
>

--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]