Re: [PATCH for-next v5 0/6] Replace AV by AH in UD sends

[Bob Pearson <rpearsonhpe@gmail.com>]

On 10/7/21 2:05 PM, Jason Gunthorpe wrote:
> On Thu, Oct 07, 2021 at 01:53:27PM -0500, Bob Pearson wrote:
> 
>> On looking, Rao's patch is not in for-next. Last one was
>> January. Which branch are you looking at?
> 
> Oh, it is still in the wip branch, try now
> 
> Jason
> 

I see the issue. Rao is asking for 2^20 objects max by default which will
require 128KiB of memory in the index reservation bit mask for each of them.
There are 4 indexed objects QP by qpn, SRQ by srqn, MR by rkey and MW by rkey.
That's 512KiB of memory which seems excessive to me for many use cases where the
number of objects is fairly small.

The bit mask is used to allocate and free the indices and there is also a red black
tree that is used to look up objects by their index (or key if they use keys instead.)

If there is a usual way to address these kinds of issues in Linux maybe we should
consider that. If not there are a couple of approaches we could take. First would
be to get rid of the index bit mask and just hand out randomly selected indices in
(a bigger range) and detect collisions when we insert the object into the red black
tree and retry. This is basically what happens with 'keys' for example mgids for
multicast group elements. Alternatively we could leave the max big but limit the
allocated indices to a smaller amount until the total number of allocated indices
reached some threshold and then extend the bit mask table. Then only the use cases
that really needed the big index range would pay the price for the memory.

Random indices would slightly reduce some of the security issues that have been
pointed out about the InfiniBand transport.

I am looking for suggestions on how to go forward here.

Bob