Re: [PATCH rdma-next] IB/core: Add option to limit user mad receive list

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Apr 04, 2024 at 07:51:03PM +0300, Leon Romanovsky wrote:
> On Thu, Apr 04, 2024 at 11:01:13AM -0300, Jason Gunthorpe wrote:
> > On Tue, Apr 02, 2024 at 12:50:21PM +0300, Leon Romanovsky wrote:
> > > From: Michael Guralnik <michaelgur@nvidia.com>
> > > 
> > > ib_umad is keeping the received MAD packets in a list that is not
> > > limited in size. As the extraction of packets from this list is done
> > > from user-space application, there is no way to guarantee the extraction
> > > rate to be faster than the rate of incoming packets. This can cause to
> > > the list to grow uncontrollably.
> > > 
> > > As a solution, let's add new ysfs control knob for the users to limit
> > > the number of received MAD packets in the list.
> > > 
> > > Packets received when the list is full would be dropped. Sent packets
> > > that are queued on the receive list for whatever reason, like timed out
> > > sends, are not dropped even when the list is full.
> > > 
> > > Signed-off-by: Michael Guralnik <michaelgur@nvidia.com>
> > > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> > > ---
> > >  .../ABI/stable/sysfs-class-infiniband         | 12 ++++
> > >  drivers/infiniband/core/user_mad.c            | 63 ++++++++++++++++++-
> > >  2 files changed, 72 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/Documentation/ABI/stable/sysfs-class-infiniband b/Documentation/ABI/stable/sysfs-class-infiniband
> > > index 694f23a03a28..0ea9d590ab0e 100644
> > > --- a/Documentation/ABI/stable/sysfs-class-infiniband
> > > +++ b/Documentation/ABI/stable/sysfs-class-infiniband
> > > @@ -275,6 +275,18 @@ Description:
> > >  		=============== ===========================================
> > >  
> > >  
> > > +What:		/sys/class/infiniband_mad/umad<N>/max_recv_list_size
> > > +Date:		January, 2024
> > > +KernelVersion:	v6.9
> > > +Contact:	linux-rdma@vger.kernel.org
> > > +Description:
> > > +		(RW) Limit the size of the list of MAD packets waiting to be
> > > +		     read by the user-space agent.
> > > +		     The default value is 0, which means unlimited list size.
> > > +		     Packets received when the list is full will be silently
> > > +		     dropped.
> > 
> > I'm really not keen on this as a tunable, when we get to future
> > designs it may be hard to retain this specific behavior.
> > 
> > Why do we need a tunable? Can we just set it to something large and be
> > done with it?
> 
> I don't know which value to set to be large enough from one side and
> small enough to do not cause to OOM while host gets MAD packets.

I'd say something like 32M worth of packets is probably good from both
directions? Or 1% of system memory, or something like that.

Jason