Re: [PATCH for-next v5 0/6] Replace AV by AH in UD sends

[Bob Pearson <rpearsonhpe@gmail.com>]

On 10/7/21 3:40 PM, Shoaib Rao wrote:
> 
> On 10/7/21 12:57 PM, Jason Gunthorpe wrote:
>> On Thu, Oct 07, 2021 at 02:51:11PM -0500, Bob Pearson wrote:
>>> On 10/7/21 2:05 PM, Jason Gunthorpe wrote:
>>>> On Thu, Oct 07, 2021 at 01:53:27PM -0500, Bob Pearson wrote:
>>>>
>>>>> On looking, Rao's patch is not in for-next. Last one was
>>>>> January. Which branch are you looking at?
>>>> Oh, it is still in the wip branch, try now
>>>>
>>>> Jason
>>>>
>>> I see the issue. Rao is asking for 2^20 objects max by default which will
>>> require 128KiB of memory in the index reservation bit mask for each of them.
>>> There are 4 indexed objects QP by qpn, SRQ by srqn, MR by rkey and MW by rkey.
>>> That's 512KiB of memory which seems excessive to me for many use cases where the
>>> number of objects is fairly small.
>>>
>>> The bit mask is used to allocate and free the indices and there is also a red black
>>> tree that is used to look up objects by their index (or key if they use keys instead.)
>>>
>>> If there is a usual way to address these kinds of issues in Linux maybe we should
>>> consider that.
>> Use an allocating xarray
>>
>> But for these AV patches just fix the merge conflict to something sane
>> and go ahead
>>
>> Jason
> 
> I did not want to increase the values too high but we discussed it so I did. Let me know if I need to modify the patch and reduce the values.
> 
> Shoaib
> 

If we convert the rxe_pools to use xarrays as Jason suggests it looks like this issue
goes away. I'm looking at that.

Bob