[net PATCH] octeon_ep: update BQL sent bytes before ringing doorbell

[net PATCH] octeon_ep: update BQL sent bytes before ringing doorbell

[Shinas Rasheed]

Sometimes Tx is completed immediately after doorbell is updated, which
causes Tx completion routing to update completion bytes before the
same packet bytes are updated in sent bytes in transmit function, hence
hitting BUG_ON() in dql_completed(). To avoid this, update BQL
sent bytes before ringing doorbell.

Fixes: 37d79d059606 ("octeon_ep: add Tx/Rx processing and interrupt support")
Signed-off-by: Shinas Rasheed <srasheed@marvell.com>
---
 drivers/net/ethernet/marvell/octeon_ep/octep_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/octeon_ep/octep_main.c b/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
index dbc518ff8276..314f9c661f93 100644
--- a/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
+++ b/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
@@ -718,6 +718,7 @@ static netdev_tx_t octep_start_xmit(struct sk_buff *skb,
 	/* Flush the hw descriptor before writing to doorbell */
 	wmb();
 
+	netdev_tx_sent_queue(iq->netdev_q, skb->len);
 	/* Ring Doorbell to notify the NIC there is a new packet */
 	writel(1, iq->doorbell_reg);
 	atomic_inc(&iq->instr_pending);
@@ -726,7 +727,6 @@ static netdev_tx_t octep_start_xmit(struct sk_buff *skb,
 		wi = 0;
 	iq->host_write_index = wi;
 
-	netdev_tx_sent_queue(iq->netdev_q, skb->len);
 	iq->stats.instr_posted++;
 	skb_tx_timestamp(skb);
 	return NETDEV_TX_OK;
-- 
2.25.1

Re: [net PATCH] octeon_ep: update BQL sent bytes before ringing doorbell

[Paolo Abeni]

On Tue, 2023-10-10 at 04:50 -0700, Shinas Rasheed wrote:
> Sometimes Tx is completed immediately after doorbell is updated, which
> causes Tx completion routing to update completion bytes before the
> same packet bytes are updated in sent bytes in transmit function, hence
> hitting BUG_ON() in dql_completed(). To avoid this, update BQL
> sent bytes before ringing doorbell.
> 
> Fixes: 37d79d059606 ("octeon_ep: add Tx/Rx processing and interrupt support")
> Signed-off-by: Shinas Rasheed <srasheed@marvell.com>
> ---
>  drivers/net/ethernet/marvell/octeon_ep/octep_main.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/marvell/octeon_ep/octep_main.c b/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
> index dbc518ff8276..314f9c661f93 100644
> --- a/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
> +++ b/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
> @@ -718,6 +718,7 @@ static netdev_tx_t octep_start_xmit(struct sk_buff *skb,
>  	/* Flush the hw descriptor before writing to doorbell */
>  	wmb();
>  
> +	netdev_tx_sent_queue(iq->netdev_q, skb->len);

If tx completion and start_xmit happen on 2 different CPUs, how do you
ensure that xmit_completion will observe the values written here?

Specifically, don't you need to move netdev_tx_sent_queue() before the
above memory barrier?

Thanks,

Paolo

Re: [EXT] Re: [net PATCH] octeon_ep: update BQL sent bytes before ringing doorbell

[Shinas Rasheed]

Hi,

If tx completion and start_xmit happen on 2 different CPUs, how do you
ensure that xmit_completion will observe the values written here?

Specifically, don't you need to move netdev_tx_sent_queue() before the
above memory barrier?

>>> Yes, you are correct. I'll update the changeset. Thank you.

[net PATCH v2] octeon_ep: update BQL sent bytes before ringing doorbell

[Shinas Rasheed]

Sometimes Tx is completed immediately after doorbell is updated, which
causes Tx completion routing to update completion bytes before the
same packet bytes are updated in sent bytes in transmit function, hence
hitting BUG_ON() in dql_completed(). To avoid this, update BQL
sent bytes before ringing doorbell.

Fixes: 37d79d059606 ("octeon_ep: add Tx/Rx processing and interrupt support")
Signed-off-by: Shinas Rasheed <srasheed@marvell.com>
---
V1 -> V2: Call netdev_tx_sent_queue before memory barrier

 drivers/net/ethernet/marvell/octeon_ep/octep_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/marvell/octeon_ep/octep_main.c b/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
index dbc518ff8276..15420325aef3 100644
--- a/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
+++ b/drivers/net/ethernet/marvell/octeon_ep/octep_main.c
@@ -715,6 +715,7 @@ static netdev_tx_t octep_start_xmit(struct sk_buff *skb,
 		hw_desc->dptr = tx_buffer->sglist_dma;
 	}
 
+	netdev_tx_sent_queue(iq->netdev_q, skb->len);
 	/* Flush the hw descriptor before writing to doorbell */
 	wmb();
 
@@ -726,7 +727,6 @@ static netdev_tx_t octep_start_xmit(struct sk_buff *skb,
 		wi = 0;
 	iq->host_write_index = wi;
 
-	netdev_tx_sent_queue(iq->netdev_q, skb->len);
 	iq->stats.instr_posted++;
 	skb_tx_timestamp(skb);
 	return NETDEV_TX_OK;
-- 
2.25.1

Re: [net PATCH v2] octeon_ep: update BQL sent bytes before ringing doorbell

[Jakub Kicinski]

On Thu, 12 Oct 2023 03:17:06 -0700 Shinas Rasheed wrote:
> Sometimes Tx is completed immediately after doorbell is updated, which
> causes Tx completion routing to update completion bytes before the
> same packet bytes are updated in sent bytes in transmit function, hence
> hitting BUG_ON() in dql_completed(). To avoid this, update BQL
> sent bytes before ringing doorbell.

Please read this:

https://www.kernel.org/doc/html/next/process/maintainer-netdev.html#resending-after-review

And also when you reply to people please use sane quoting.
This: >>>
is used to indicate three levels of quoting.

Re: [EXT] Re: [net PATCH v2] octeon_ep: update BQL sent bytes before ringing doorbell

[Shinas Rasheed]

Hi Jakub,

Apologizing for the format errors on my part. Should I send the v2 patch again separately in a new thread, or will this be enough for now to avoid the clutter?

Re: [EXT] Re: [net PATCH v2] octeon_ep: update BQL sent bytes before ringing doorbell

[Jakub Kicinski]

On Fri, 13 Oct 2023 09:04:07 +0000 Shinas Rasheed wrote:
> Apologizing for the format errors on my part. Should I send the v2
> patch again separately in a new thread, or will this be enough for
> now to avoid the clutter?

Sorry, I meant to get to this on Friday but run out of hours.

No need to repost, my comments were just for future reference
since you have sent multiple patches already and will likely
send more :)

Re: [net PATCH v2] octeon_ep: update BQL sent bytes before ringing doorbell

[Jakub Kicinski]

On Thu, 12 Oct 2023 03:17:06 -0700 Shinas Rasheed wrote:
> -	netdev_tx_sent_queue(iq->netdev_q, skb->len);
>  	iq->stats.instr_posted++;
>  	skb_tx_timestamp(skb);

The skb_tx_timestamp() here will do the same exact UAF, no?
I think you should move them both.
-- 
pw-bot: cr