[PATCH] net: mana: Batch ringing RX queue doorbell on receiving packets

[PATCH] net: mana: Batch ringing RX queue doorbell on receiving packets

[longli]

From: Long Li <longli@microsoft.com>

It's inefficient to ring the doorbell page every time a WQE is posted to
the received queue.

Move the code for ringing doorbell page to where after we have posted all
WQEs to the receive queue during a callback from napi_poll().

Tests showed no regression in network latency benchmarks.

Cc: stable@vger.kernel.org
Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")
Signed-off-by: Long Li <longli@microsoft.com>
---
 drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index cd4d5ceb9f2d..ef1f0ce8e44d 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -1383,8 +1383,8 @@ static void mana_post_pkt_rxq(struct mana_rxq *rxq)
 
 	recv_buf_oob = &rxq->rx_oobs[curr_index];
 
-	err = mana_gd_post_and_ring(rxq->gdma_rq, &recv_buf_oob->wqe_req,
-				    &recv_buf_oob->wqe_inf);
+	err = mana_gd_post_work_request(rxq->gdma_rq, &recv_buf_oob->wqe_req,
+					&recv_buf_oob->wqe_inf);
 	if (WARN_ON_ONCE(err))
 		return;
 
@@ -1654,6 +1654,12 @@ static void mana_poll_rx_cq(struct mana_cq *cq)
 		mana_process_rx_cqe(rxq, cq, &comp[i]);
 	}
 
+	if (comp_read) {
+		struct gdma_context *gc = rxq->gdma_rq->gdma_dev->gdma_context;
+
+		mana_gd_wq_ring_doorbell(gc, rxq->gdma_rq);
+	}
+
 	if (rxq->xdp_flush)
 		xdp_do_flush();
 }
-- 
2.34.1

RE: [PATCH] net: mana: Batch ringing RX queue doorbell on receiving packets

[Haiyang Zhang]

> -----Original Message-----
> From: longli@linuxonhyperv.com <longli@linuxonhyperv.com>
> Sent: Thursday, June 15, 2023 7:28 PM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Wei Liu <wei.liu@kernel.org>; Dexuan Cui
> <decui@microsoft.com>; David S. Miller <davem@davemloft.net>; Eric
> Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo
> Abeni <pabeni@redhat.com>; Leon Romanovsky <leon@kernel.org>; Shradha
> Gupta <shradhagupta@linux.microsoft.com>; Ajay Sharma
> <sharmaajay@microsoft.com>; Shachar Raindel <shacharr@microsoft.com>;
> Stephen Hemminger <stephen@networkplumber.org>; linux-
> hyperv@vger.kernel.org; netdev@vger.kernel.org; linux-
> kernel@vger.kernel.org
> Cc: linux-rdma@vger.kernel.org; Long Li <longli@microsoft.com>;
> stable@vger.kernel.org
> Subject: [PATCH] net: mana: Batch ringing RX queue doorbell on receiving
> packets
> 
> From: Long Li <longli@microsoft.com>
> 
> It's inefficient to ring the doorbell page every time a WQE is posted to
> the received queue.
> 
> Move the code for ringing doorbell page to where after we have posted all
> WQEs to the receive queue during a callback from napi_poll().
> 
> Tests showed no regression in network latency benchmarks.
> 
> Cc: stable@vger.kernel.org
> Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network
> Adapter (MANA)")
> Signed-off-by: Long Li <longli@microsoft.com>
> ---
>  drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c
> b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index cd4d5ceb9f2d..ef1f0ce8e44d 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> @@ -1383,8 +1383,8 @@ static void mana_post_pkt_rxq(struct mana_rxq
> *rxq)
> 
>  	recv_buf_oob = &rxq->rx_oobs[curr_index];
> 
> -	err = mana_gd_post_and_ring(rxq->gdma_rq, &recv_buf_oob-
> >wqe_req,
> -				    &recv_buf_oob->wqe_inf);
> +	err = mana_gd_post_work_request(rxq->gdma_rq, &recv_buf_oob-
> >wqe_req,
> +					&recv_buf_oob->wqe_inf);
>  	if (WARN_ON_ONCE(err))
>  		return;
> 
> @@ -1654,6 +1654,12 @@ static void mana_poll_rx_cq(struct mana_cq *cq)
>  		mana_process_rx_cqe(rxq, cq, &comp[i]);
>  	}
> 
> +	if (comp_read) {
> +		struct gdma_context *gc = rxq->gdma_rq->gdma_dev-
> >gdma_context;
> +
> +		mana_gd_wq_ring_doorbell(gc, rxq->gdma_rq);
> +	}
> +

Thank you!

Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>

RE: [PATCH] net: mana: Batch ringing RX queue doorbell on receiving packets

[Long Li]

Hi,

I'm sending v2 to address some corner cases discovered during tests.

Thanks,
Long

> -----Original Message-----
> From: Haiyang Zhang <haiyangz@microsoft.com>
> Sent: Friday, June 16, 2023 9:49 AM
> To: longli@linuxonhyperv.com; KY Srinivasan <kys@microsoft.com>; Wei Liu
> <wei.liu@kernel.org>; Dexuan Cui <decui@microsoft.com>; David S. Miller
> <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub
> Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Leon
> Romanovsky <leon@kernel.org>; Shradha Gupta
> <shradhagupta@linux.microsoft.com>; Ajay Sharma
> <sharmaajay@microsoft.com>; Shachar Raindel <shacharr@microsoft.com>;
> Stephen Hemminger <stephen@networkplumber.org>; linux-
> hyperv@vger.kernel.org; netdev@vger.kernel.org; linux-
> kernel@vger.kernel.org
> Cc: linux-rdma@vger.kernel.org; Long Li <longli@microsoft.com>;
> stable@vger.kernel.org
> Subject: RE: [PATCH] net: mana: Batch ringing RX queue doorbell on receiving
> packets
> 
> 
> 
> > -----Original Message-----
> > From: longli@linuxonhyperv.com <longli@linuxonhyperv.com>
> > Sent: Thursday, June 15, 2023 7:28 PM
> > To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> > <haiyangz@microsoft.com>; Wei Liu <wei.liu@kernel.org>; Dexuan Cui
> > <decui@microsoft.com>; David S. Miller <davem@davemloft.net>; Eric
> > Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>;
> Paolo
> > Abeni <pabeni@redhat.com>; Leon Romanovsky <leon@kernel.org>;
> Shradha
> > Gupta <shradhagupta@linux.microsoft.com>; Ajay Sharma
> > <sharmaajay@microsoft.com>; Shachar Raindel <shacharr@microsoft.com>;
> > Stephen Hemminger <stephen@networkplumber.org>; linux-
> > hyperv@vger.kernel.org; netdev@vger.kernel.org; linux-
> > kernel@vger.kernel.org
> > Cc: linux-rdma@vger.kernel.org; Long Li <longli@microsoft.com>;
> > stable@vger.kernel.org
> > Subject: [PATCH] net: mana: Batch ringing RX queue doorbell on
> > receiving packets
> >
> > From: Long Li <longli@microsoft.com>
> >
> > It's inefficient to ring the doorbell page every time a WQE is posted
> > to the received queue.
> >
> > Move the code for ringing doorbell page to where after we have posted
> > all WQEs to the receive queue during a callback from napi_poll().
> >
> > Tests showed no regression in network latency benchmarks.
> >
> > Cc: stable@vger.kernel.org
> > Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure
> > Network Adapter (MANA)")
> > Signed-off-by: Long Li <longli@microsoft.com>
> > ---
> >  drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++--
> >  1 file changed, 8 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c
> > b/drivers/net/ethernet/microsoft/mana/mana_en.c
> > index cd4d5ceb9f2d..ef1f0ce8e44d 100644
> > --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> > +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
> > @@ -1383,8 +1383,8 @@ static void mana_post_pkt_rxq(struct mana_rxq
> > *rxq)
> >
> >  	recv_buf_oob = &rxq->rx_oobs[curr_index];
> >
> > -	err = mana_gd_post_and_ring(rxq->gdma_rq, &recv_buf_oob-
> > >wqe_req,
> > -				    &recv_buf_oob->wqe_inf);
> > +	err = mana_gd_post_work_request(rxq->gdma_rq, &recv_buf_oob-
> > >wqe_req,
> > +					&recv_buf_oob->wqe_inf);
> >  	if (WARN_ON_ONCE(err))
> >  		return;
> >
> > @@ -1654,6 +1654,12 @@ static void mana_poll_rx_cq(struct mana_cq
> *cq)
> >  		mana_process_rx_cqe(rxq, cq, &comp[i]);
> >  	}
> >
> > +	if (comp_read) {
> > +		struct gdma_context *gc = rxq->gdma_rq->gdma_dev-
> > >gdma_context;
> > +
> > +		mana_gd_wq_ring_doorbell(gc, rxq->gdma_rq);
> > +	}
> > +
> 
> Thank you!
> 
> Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>