[PATCH net] ibmvnic: Clean RX pools only during a hard reset

[PATCH net] ibmvnic: Clean RX pools only during a hard reset

[Thomas Falcon]

Sorry, this fixes a bug in commit d0869c0071e4. The cause of the
bug is that "stale" RX buffers containing packet data are returned
to the driver after device close and open. While most buffers will be
returned with an error and handled by the polling routine, some buffers
will be returned as containing valid data. Unfortunately, the socket
buffers allocated were already freed when the device was closed, so
attempts to process them result in a panic.

RX pools still need to be cleaned in some cases, such as during
a fatal reset. In all other cases, the socket buffers will either
be freed in the polling routine or processed by the kernel.

Fixes: d0869c0071e4 ("ibmvnic: Clean RX pool buffers during device close")
Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 996f475..6710313 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -1179,7 +1179,9 @@ static int __ibmvnic_close(struct net_device *netdev)
 			}
 		}
 	}
-	clean_rx_pools(adapter);
+	if (unlikely(adapter->resetting &&
+		     adapter->reset_reason != VNIC_RESET_NON_FATAL))
+		clean_rx_pools(adapter);
 	clean_tx_pools(adapter);
 	adapter->state = VNIC_CLOSED;
 	return rc;
-- 
2.7.5

[PATCH net-next] ibmvnic: Keep track of supplementary TX descriptors

[Thomas Falcon]

Supplementary TX descriptors were not being accounted for, which
was resulting in an overflow of the hardware device's transmit
queue. Keep track of those descriptors now when determining
how many entries remain on the TX queue.

Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 8 ++++++--
 drivers/net/ethernet/ibm/ibmvnic.h | 1 +
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index a3865ff5a48e..7f5488695fd5 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -1469,6 +1469,7 @@ static int ibmvnic_xmit(struct sk_buff *skb, struct net_device *netdev)
 	if ((*hdrs >> 7) & 1) {
 		build_hdr_descs_arr(tx_buff, &num_entries, *hdrs);
 		tx_crq.v1.n_crq_elem = num_entries;
+		tx_buff->num_entries = num_entries;
 		tx_buff->indir_arr[0] = tx_crq;
 		tx_buff->indir_dma = dma_map_single(dev, tx_buff->indir_arr,
 						    sizeof(tx_buff->indir_arr),
@@ -1517,7 +1518,7 @@ static int ibmvnic_xmit(struct sk_buff *skb, struct net_device *netdev)
 		goto out;
 	}
 
-	if (atomic_inc_return(&tx_scrq->used)
+	if (atomic_add_return(num_entries, &tx_scrq->used)
 					>= adapter->req_tx_entries_per_subcrq) {
 		netdev_info(netdev, "Stopping queue %d\n", queue_num);
 		netif_stop_subqueue(netdev, queue_num);
@@ -2470,6 +2471,7 @@ static int ibmvnic_complete_tx(struct ibmvnic_adapter *adapter,
 restart_loop:
 	while (pending_scrq(adapter, scrq)) {
 		unsigned int pool = scrq->pool_index;
+		int num_entries = 0;
 
 		next = ibmvnic_next_scrq(adapter, scrq);
 		for (i = 0; i < next->tx_comp.num_comps; i++) {
@@ -2500,6 +2502,8 @@ static int ibmvnic_complete_tx(struct ibmvnic_adapter *adapter,
 				txbuff->skb = NULL;
 			}
 
+			num_entries += txbuff->num_entries;
+
 			adapter->tx_pool[pool].free_map[adapter->tx_pool[pool].
 						     producer_index] = index;
 			adapter->tx_pool[pool].producer_index =
@@ -2509,7 +2513,7 @@ static int ibmvnic_complete_tx(struct ibmvnic_adapter *adapter,
 		/* remove tx_comp scrq*/
 		next->tx_comp.first = 0;
 
-		if (atomic_sub_return(next->tx_comp.num_comps, &scrq->used) <=
+		if (atomic_sub_return(num_entries, &scrq->used) <=
 		    (adapter->req_tx_entries_per_subcrq / 2) &&
 		    __netif_subqueue_stopped(adapter->netdev,
 					     scrq->pool_index)) {
diff --git a/drivers/net/ethernet/ibm/ibmvnic.h b/drivers/net/ethernet/ibm/ibmvnic.h
index fe21a6e2ddae..2f51458ccdc3 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.h
+++ b/drivers/net/ethernet/ibm/ibmvnic.h
@@ -909,6 +909,7 @@ struct ibmvnic_tx_buff {
 	union sub_crq indir_arr[6];
 	u8 hdr_data[140];
 	dma_addr_t indir_dma;
+	int num_entries;
 };
 
 struct ibmvnic_tx_pool {
-- 
2.15.0

Re: [PATCH net] ibmvnic: Clean RX pools only during a hard reset

[David Miller]

From: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
Date: Sun, 18 Feb 2018 10:08:40 -0600

> Sorry, this fixes a bug in commit d0869c0071e4. The cause of the
> bug is that "stale" RX buffers containing packet data are returned
> to the driver after device close and open. While most buffers will be
> returned with an error and handled by the polling routine, some buffers
> will be returned as containing valid data. Unfortunately, the socket
> buffers allocated were already freed when the device was closed, so
> attempts to process them result in a panic.
> 
> RX pools still need to be cleaned in some cases, such as during
> a fatal reset. In all other cases, the socket buffers will either
> be freed in the polling routine or processed by the kernel.
> 
> Fixes: d0869c0071e4 ("ibmvnic: Clean RX pool buffers during device close")
> Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>

It really shouldn't matter who, or how many times, clear_rx_pools() is
called.

Anyone who calls it and frees the SKBs will mark the SKB slots as NULL,
so any subsequent call cannot possibly double free the buffers.

At best you need to explain the problem better in the commit message.

Re: [PATCH net] ibmvnic: Clean RX pools only during a hard reset

[Thomas Falcon]

On 02/19/2018 10:37 AM, David Miller wrote:
> From: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
> Date: Sun, 18 Feb 2018 10:08:40 -0600
>
>> Sorry, this fixes a bug in commit d0869c0071e4. The cause of the
>> bug is that "stale" RX buffers containing packet data are returned
>> to the driver after device close and open. While most buffers will be
>> returned with an error and handled by the polling routine, some buffers
>> will be returned as containing valid data. Unfortunately, the socket
>> buffers allocated were already freed when the device was closed, so
>> attempts to process them result in a panic.
>>
>> RX pools still need to be cleaned in some cases, such as during
>> a fatal reset. In all other cases, the socket buffers will either
>> be freed in the polling routine or processed by the kernel.
>>
>> Fixes: d0869c0071e4 ("ibmvnic: Clean RX pool buffers during device close")
>> Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
> It really shouldn't matter who, or how many times, clear_rx_pools() is
> called.
>
> Anyone who calls it and frees the SKBs will mark the SKB slots as NULL,
> so any subsequent call cannot possibly double free the buffers.
>
> At best you need to explain the problem better in the commit message.

Sorry, I should explain it better. It's not there is a double free.  It's that the driver is receiving RX descriptors from the previous session for which socket buffers have been freed. The driver's polling routine tries to copy data to the socket buffer, but it's been freed, so it's trying to copy to a NULL pointer.

Tom

Re: [PATCH net] ibmvnic: Clean RX pools only during a hard reset

[David Miller]

From: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
Date: Mon, 19 Feb 2018 13:24:52 -0600

> Sorry, I should explain it better. It's not there is a double free.
> It's that the driver is receiving RX descriptors from the previous
> session for which socket buffers have been freed. The driver's
> polling routine tries to copy data to the socket buffer, but it's
> been freed, so it's trying to copy to a NULL pointer.

That's kinda hairy, is this resend of the old descriptors guaranteed
to always happen in this situation?

Maybe it's better to have some way for the RX descriptor receiving
path to detect this situation (is SKB slot NULL?) to handle the
problem there.

Thanks.

Re: [PATCH net] ibmvnic: Clean RX pools only during a hard reset

[Thomas Falcon]

On 02/19/2018 01:30 PM, David Miller wrote:
> From: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
> Date: Mon, 19 Feb 2018 13:24:52 -0600
>
>> Sorry, I should explain it better. It's not there is a double free.
>> It's that the driver is receiving RX descriptors from the previous
>> session for which socket buffers have been freed. The driver's
>> polling routine tries to copy data to the socket buffer, but it's
>> been freed, so it's trying to copy to a NULL pointer.
> That's kinda hairy, is this resend of the old descriptors guaranteed
> to always happen in this situation?
>
> Maybe it's better to have some way for the RX descriptor receiving
> path to detect this situation (is SKB slot NULL?) to handle the
> problem there.

It is something we can expect to happen in this situation.  Thanks for the suggestion.  That way the driver can free up that memory when it closes.  I'll try to get a v2 out soon.

Thanks again.

> Thanks.
>

[PATCH net v2] ibmvnic: Check for NULL skb's in NAPI poll routine

[Thomas Falcon]

After introduction of commit d0869c0071e4, there were some instances of
RX queue entries from a previous session (before the device was closed
and reopened) returned to the NAPI polling routine. Since the corresponding
socket buffers were freed, this resulted in a panic on reopen. Include
a check for a NULL skb here to avoid this.

Fixes: d0869c0071e4 ("ibmvnic: Clean RX pool buffers during device close")
Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
---
v2: Followed David Miller's suggestion to check for a NULL socket buffer
    Originally sent as "ibmvnic: Clean RX pools only during a hard reset"
---
 drivers/net/ethernet/ibm/ibmvnic.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 996f475..1495cb9 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -1901,6 +1901,11 @@ static int ibmvnic_poll(struct napi_struct *napi, int budget)
 			dev_kfree_skb_any(rx_buff->skb);
 			remove_buff_from_pool(adapter, rx_buff);
 			continue;
+		} else if (!rx_buff->skb) {
+			/* free the entry */
+			next->rx_comp.first = 0;
+			remove_buff_from_pool(adapter, rx_buff);
+			continue;
 		}
 
 		length = be32_to_cpu(next->rx_comp.len);
-- 
2.7.5

Re: [PATCH net-next] ibmvnic: Keep track of supplementary TX descriptors

[David Miller]

From: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
Date: Sun, 18 Feb 2018 10:08:41 -0600

> Supplementary TX descriptors were not being accounted for, which
> was resulting in an overflow of the hardware device's transmit
> queue. Keep track of those descriptors now when determining
> how many entries remain on the TX queue.
> 
> Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>

Applied, thanks Thomas.