Re: [PATCH net-next] net/mlx5: Make ASO poll CQ usable in atomic context

[Leon Romanovsky <leon@kernel.org>]

On Sat, Sep 24, 2022 at 01:11:31PM -0700, Saeed Mahameed wrote:
> On 24 Sep 21:51, Leon Romanovsky wrote:
> > On Sat, Sep 24, 2022 at 10:24:25AM -0700, Saeed Mahameed wrote:
> > > On 24 Sep 14:17, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@nvidia.com>
> > > >
> > > > Poll CQ functions shouldn't sleep as they are called in atomic context.
> > > > The following splat appears once the mlx5_aso_poll_cq() is used in such
> > > > flow.
> > > >
> > > > BUG: scheduling while atomic: swapper/17/0/0x00000100
> > > 
> > > ...
> > > 
> > > > 	/* With newer FW, the wait for the first ASO WQE is more than 2us, put the wait 10ms. */
> > > > -	err = mlx5_aso_poll_cq(aso, true, 10);
> > > > +	expires = jiffies + msecs_to_jiffies(10);
> > > > +	do {
> > > > +		err = mlx5_aso_poll_cq(aso, true);
> > > > +	} while (err && time_is_after_jiffies(expires));
> > > > 	mutex_unlock(&flow_meters->aso_lock);
> > >         ^^^^
> > > busy poll won't work, this mutex is held and can sleep anyway.
> > > Let's discuss internally and solve this by design.
> > 
> > This is TC code, it doesn't need atomic context and had mutex + sleep
> > from the beginning.
> > 
> 
> then let's bring back the sleep for TC use-case, we don't need to burn the
> CPU!

Again, this is exactly how TC was implemented. The difference is that
before it burnt cycles in poll_cq and now it does it inside TC code.

> But still the change has bigger effect on other aso use cases, see below.

I have serious doubts about it.

> 
> > My change cleans mlx5_aso_poll_cq() from busy loop for the IPsec path,
> > so why do you plan to change in the design?
> > 
> 
> a typical use case for aso is
> 
> post_aso_wqe();
> poll_aso_cqe();
> 
> The HW needs time to consume and excute the wqe then generate the CQE.
> This is why the driver needs to wait a bit when polling for the cqe,
> your change will break others. Let's discuss internally.

IPsec was always implemented without any time delays, and I'm sure that
MACsec doesn't need too, it is probably copy/paste.

More generally, post_wqe->poll_cq is very basic paradigm in RDMA and
delays were never needed.

But if you insist, let's discuss internally.

Thanks

> 
> > Thanks