Re: Inconsistent lock state warning for rxe_poll_cq()

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Feb 15, 2022 at 04:45:01PM +0000, Pearson, Robert B wrote:
> 
> 
> From: Jason Gunthorpe <jgg@nvidia.com> 
> Sent: Tuesday, February 15, 2022 8:42 AM
> To: Bob Pearson <rpearsonhpe@gmail.com>
> Cc: Bart Van Assche <bvanassche@acm.org>; Guoqing Jiang <guoqing.jiang@linux.dev>; linux-rdma@vger.kernel.org
> Subject: Re: Inconsistent lock state warning for rxe_poll_cq()
> 
> On Mon, Feb 14, 2022 at 11:43:40PM -0600, Bob Pearson wrote:
> 
> >> in the write_unlock_bh() call. This appears to complain if hardirqs 
> >> are not enabled on the current cpu. This only happens if 
> >> CONFIG_PROVE_LOCKING=y.
> 
> > The trace shows this context is called within a irq disabled spinlock region. Ie this is trying to do
> 
> >   spinlock_irqsave()
> >   write_lock_bh()
> >   write_unlock_bh()
> >   spinunlock_irqrestore()
> 
> > Which is illegal locking.
> 
> > Jason
> 
> Yes. I figured that. But I don't know why, or why
> 
> spin_lock_irqsave(lock_a)
> spin_lock_irqsave(lock_b)
> spin_unlock_irqrestore(lock_b)
> spin_unlock_irqrestore(lock_a)
> 
> Is clearly better which has been suggested as a fix.

Right

> Apparently someone above me is taking an irqsave lock and then
> calling

Yes, the CM code

> Down to rxe_create_ah().

Yep

> Calling the verbs APIs while holding a spinlock seems likely to
> cause problems.

Indeed, I hate it, but it is hard to fix

> This looks to be in response to receiving a MAD so possibly is a
> call to ibv_create_ah_from_wc().

Yes something like that

Jason