Re: [PATCH net] ionic: Fix allocation of q/cq info structures from device local node

[Jakub Kicinski <kuba@kernel.org>]

On Wed, 12 Apr 2023 19:58:16 +0300 Leon Romanovsky wrote:
> > > I'm not sure about it as you are running kernel thread which is
> > > triggered directly by device and most likely will run on same node as
> > > PCI device.  
> > 
> > Isn't that true only for bus-side probing?
> > If you bind/unbind via sysfs does it still try to move to the right
> > node? Same for resources allocated during ifup?  
> 
> Kernel threads are more interesting case, as they are not controlled
> through mempolicy (maybe it is not true in 2023, I'm not sure).
> 
> User triggered threads are subjected to mempolicy and all allocations
> are expected to follow it. So users, who wants specific memory behaviour
> should use it.
> 
> https://docs.kernel.org/6.1/admin-guide/mm/numa_memory_policy.html
> 
> There is a huge chance that fallback mechanisms proposed here in ionic
> and implemented in ENA are "break" this interface.

Ack, that's what I would have answered while working for a vendor
myself, 5 years ago. Now, after seeing how NICs get configured in
practice, and all the random tools which may decide to tweak some
random param and forget to pin themselves - I'm not as sure.

Having a policy configured per netdev and maybe netdev helpers for
memory allocation could be an option. We already link netdev to 
the struct device.

> > > vzalloc_node() doesn't do fallback, but vzalloc will find the right node
> > > for you.  
> > 
> > Sounds like we may want a vzalloc_node_with_fallback or some GFP flag?
> > All the _node() helpers which don't fall back lead to unpleasant code
> > in the users.  
> 
> I would challenge the whole idea of having *_node() allocations in
> driver code at the first place. Even in RDMA, where we super focused
> on performance and allocation of memory in right place is super
> critical, we rely on general kzalloc().
> 
> There is one exception in RDMA world (hfi1), but it is more because of
> legacy implementation and not because of specific need, at least Intel
> folks didn't success to convince me with real data.

Yes, but RDMA is much more heavy on the application side, much more
tightly integrated in general.