[PATCH for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[PATCH for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Adit Ranadive]

From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>

There is a chance of a race between arming the CQ and receiving
completions. By reporting CQ missed events any ULPs should poll
again to get the completions.

Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
index 69bda61..f4ab5fa 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
@@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
 	struct pvrdma_dev *dev = to_vdev(ibcq->device);
 	struct pvrdma_cq *cq = to_vcq(ibcq);
 	u32 val = cq->cq_handle;
+	unsigned long flags;
 
 	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
 		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
 
+	spin_lock_irqsave(&cq->cq_lock, flags);
+
 	pvrdma_write_uar_cq(dev, val);
 
+	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
+		unsigned int head;
+
+		if (pvrdma_idx_ring_has_data(&cq->ring_state->rx,
+					     cq->ibcq.cqe, &head)) {
+			spin_unlock_irqrestore(&cq->cq_lock, flags);
+			return 1;
+		}
+	}
+
+	spin_unlock_irqrestore(&cq->cq_lock, flags);
+
 	return 0;
 }
 
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote:
> From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
>
> There is a chance of a race between arming the CQ and receiving
> completions. By reporting CQ missed events any ULPs should poll
> again to get the completions.
>
> Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> ---
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
>
> diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> index 69bda61..f4ab5fa 100644
> --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
>  	struct pvrdma_dev *dev = to_vdev(ibcq->device);
>  	struct pvrdma_cq *cq = to_vcq(ibcq);
>  	u32 val = cq->cq_handle;
> +	unsigned long flags;
>
>  	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
>  		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
>
> +	spin_lock_irqsave(&cq->cq_lock, flags);
> +
>  	pvrdma_write_uar_cq(dev, val);
>
> +	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> +		unsigned int head;
> +
> +		if (pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> +					     cq->ibcq.cqe, &head)) {

The pvrdma_idx_ring_has_data function can return PVRDMA_INVALID_IDX (-1).

> +			spin_unlock_irqrestore(&cq->cq_lock, flags);
> +			return 1;
> +		}
> +	}
> +
> +	spin_unlock_irqrestore(&cq->cq_lock, flags);
> +
>  	return 0;
>  }
>
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Yuval Shaia]

On Thu, Aug 10, 2017 at 02:15:57PM +0300, Leon Romanovsky wrote:
> On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote:
> > From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> >
> > There is a chance of a race between arming the CQ and receiving
> > completions. By reporting CQ missed events any ULPs should poll
> > again to get the completions.
> >
> > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> > Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > ---
> >  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> >
> > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > index 69bda61..f4ab5fa 100644
> > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
> >  	struct pvrdma_dev *dev = to_vdev(ibcq->device);
> >  	struct pvrdma_cq *cq = to_vcq(ibcq);
> >  	u32 val = cq->cq_handle;
> > +	unsigned long flags;
> >
> >  	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
> >  		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
> >
> > +	spin_lock_irqsave(&cq->cq_lock, flags);
> > +
> >  	pvrdma_write_uar_cq(dev, val);
> >
> > +	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> > +		unsigned int head;
> > +
> > +		if (pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> > +					     cq->ibcq.cqe, &head)) {
> 
> The pvrdma_idx_ring_has_data function can return PVRDMA_INVALID_IDX (-1).

I believe that the case of PVRDMA_INVALID_IDX is only due to some bug. I do
not think it is an expected behaviour, it means that ring is corrupted, am
i right Adit?

> 
> > +			spin_unlock_irqrestore(&cq->cq_lock, flags);
> > +			return 1;
> > +		}
> > +	}
> > +
> > +	spin_unlock_irqrestore(&cq->cq_lock, flags);
> > +
> >  	return 0;
> >  }
> >
> > --
> > 2.7.4
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Adit Ranadive]

> On Thu, Aug 10, 2017 at 02:15:57PM +0300, Leon Romanovsky wrote:
> > On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote:
> > > From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > >
> > > There is a chance of a race between arming the CQ and receiving
> > > completions. By reporting CQ missed events any ULPs should poll
> > > again to get the completions.
> > >
> > > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> > > Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > ---
> > >  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++
> > >  1 file changed, 15 insertions(+)
> > >
> > > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > > index 69bda61..f4ab5fa 100644
> > > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > > @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
> > >      struct pvrdma_dev *dev = to_vdev(ibcq->device);
> > >      struct pvrdma_cq *cq = to_vcq(ibcq);
> > >      u32 val = cq->cq_handle;
> > > +   unsigned long flags;
> > >
> > >      val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
> > >              PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
> > >
> > > +   spin_lock_irqsave(&cq->cq_lock, flags);
> > > +
> > >      pvrdma_write_uar_cq(dev, val);
> > >
> > > +   if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> > > +           unsigned int head;
> > > +
> > > +           if (pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> > > +                                        cq->ibcq.cqe, &head)) {
> >
> > The pvrdma_idx_ring_has_data function can return PVRDMA_INVALID_IDX (-1).
>
> I believe that the case of PVRDMA_INVALID_IDX is only due to some bug. I do
> not think it is an expected behaviour, it means that ring is corrupted, am
> i right Adit?

Thanks Leon, Yuval. While hitting the INVALID_IDX is really rare (we haven't hit it ever in our internal testing), we should check this for consistency purposes. Look for a v1.

- Adit--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Yuval Shaia]

On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote:
> From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> 
> There is a chance of a race between arming the CQ and receiving
> completions. By reporting CQ missed events any ULPs should poll
> again to get the completions.
> 
> Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> ---
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> index 69bda61..f4ab5fa 100644
> --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
>  	struct pvrdma_dev *dev = to_vdev(ibcq->device);
>  	struct pvrdma_cq *cq = to_vcq(ibcq);
>  	u32 val = cq->cq_handle;
> +	unsigned long flags;
>  
>  	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
>  		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
>  
> +	spin_lock_irqsave(&cq->cq_lock, flags);
> +
>  	pvrdma_write_uar_cq(dev, val);
>  
> +	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> +		unsigned int head;
> +
> +		if (pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> +					     cq->ibcq.cqe, &head)) {
> +			spin_unlock_irqrestore(&cq->cq_lock, flags);
> +			return 1;
> +		}
> +	}
> +
> +	spin_unlock_irqrestore(&cq->cq_lock, flags);
> +

Reviewed-by: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

>  	return 0;
>  }
>  
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html