[PATCH net-next v1 0/6] ibmvnic: honour flexible in-range MTU values

[PATCH net-next v1 0/6] ibmvnic: honour flexible in-range MTU values

[Mingming Cao]

ibmvnic currently only keeps an MTU of 1500 or 9000. Any other in-range
value is accepted by ip(8) and then silently reverted: the vnicserver
answers REQ_MTU with PARTIALSUCCESS and the size the backing device
settled on, and the driver treats that as rejection and falls back.

Customers want flexible in-range MTUs (for example 1414, or values in
the 2000..8000 range) rather than being forced onto the 1500/9000
buckets. Today those requests either snap back quietly or, in the
mid-range case, can fail pool allocation and leave the interface down.

This series makes those requests stick, hardens the reset path an MTU
change goes through, and (patch 6) skips the reset when the backing
device and current buffers already cover the new MTU.

Posted as a single net-next series so the flexible-MTU change and the
reset-path hardening can be reviewed together. Patches 1-5 address
user-visible bugs and carry Fixes: tags; patch 6 is the follow-on
optimization. Happy to resplit 1-5 to net and leave 6 for net-next if
that is preferred.

  Patch 2 is the functional fix: keep the requested MTU when
  PARTIALSUCCESS covers it, and converge on the device size when it
  does not.

  Patch 1 is a prerequisite. Once req_mtu may sit below the backing
  buffer size, the existing pool-entry guard (which budgets with
  req_mtu) overshoots and mid-range MTU changes can fail allocation.
  It must not be backported without patch 2, or separated from it.

  Patches 3-5 clean up side effects of the MTU reset: skip unmaps for
  buffers the VIOS already forgot across a CRQ reconnect, and allocate
  new pools / long term buffers before releasing the old ones so a
  failed realloc refuses the MTU change instead of stranding the
  interface.

  Patch 6 records the covering backing size and skips wait_for_reset()
  when the new MTU fits in both that size and the current buffers.

Tested on PowerVM ibmvnic with MTU sweeps including 1414 and the
2000..9000 range that previously failed to allocate.

Mingming Cao (6):
  ibmvnic: cap rx pool entries against the real buffer size
  ibmvnic: honour the requested mtu instead of reverting to a fallback
  ibmvnic: do not unmap long term buffers across a crq reconnect
  ibmvnic: allocate new buffer pools before releasing the old ones
  ibmvnic: allocate a new long term buffer before freeing the old one
  ibmvnic: change the mtu without a reset where the buffers allow it

 drivers/net/ethernet/ibm/ibmvnic.c | 188 ++++++++++++++++++++---------
 drivers/net/ethernet/ibm/ibmvnic.h |   3 +
 2 files changed, 133 insertions(+), 58 deletions(-)

-- 
2.50.1 (Apple Git-155)

[PATCH net-next v1 1/6] ibmvnic: cap rx pool entries against the real buffer size

[Mingming Cao]

Changing the mtu from 1500 to a value in the 2000..8000 range can fail
pool allocation and leave the interface down:

  ibmvnic 30000008: Couldn't alloc long term buffer
  __alloc_pages: ... order:N, mode:0x...

(9000 happens to succeed, which is how the middle of the range stood out.)

That is the non-bucket case this series exists to allow: the guest mtu is
not one of the sizes the backing buffers were sized for. Until the next
patch, req_mtu was always forced back to a size the device carries, so
the pool guard happened to measure the right buffer. Once arbitrary
in-range mtus are honoured, that luck goes away.

send_request_cap() limits how many entries a pool may hold so the whole
pool fits in one long term buffer set:

  max_entries = IBMVNIC_LTB_SET_SIZE /
                (adapter->req_mtu + IBMVNIC_BUFFER_HLEN);

The rx buffers are not sized from req_mtu. init_rx_pools() sizes them
from cur_rx_buf_sz, which comes from the login response and describes
the backing device. When the VIOS has the device at jumbo (cur_rx_buf_sz
around 9014) but the guest mtu is still smaller, the guard measures a
buffer much smaller than the one really allocated. The entry count
passes through untouched and the pool asks for several times the space
the guard assumed - enough that dma_alloc_coherent() / __alloc_pages()
refuses the request on kernels whose per-pool LTB budget is a single
allocation rather than a large set.

With a backing size of 9014 the buffers are ALIGN(9014, L1_CACHE_BYTES)
= 9088 bytes, while an mtu of 2000 leaves the guard working from 2514.
The overshoot only shrinks as req_mtu approaches the backing size, so
the largest mtus are the safe ones and everything below drifts.

Cap the count against the size the buffers are really allocated at.
reuse_rx_pools() has to compare against the same number or it would
see a difference on every reset and reallocate pools that are already
the right shape. clean_rx_pools() must walk rx_pool->size for the same
reason, or a clamped pool would be cleaned past its end.

Comes ahead of the next patch, which is what lets req_mtu sit below
the backing size. Keep the two together when backporting.

Fixes: c26eba03e407 ("ibmvnic: Update reset infrastructure to support tunable parameters")
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Vaishnavi Bhat <vaishnavi@linux.ibm.com>
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 5a510eed335e..86e643ee6b3b 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -1021,6 +1021,24 @@ static void release_rx_pools(struct ibmvnic_adapter *adapter)
 	adapter->prev_rx_pool_size = 0;
 }
 
+/**
+ * rx_pool_entries() - Number of buffers one rx pool may hold
+ * @adapter: ibmvnic adapter
+ *
+ * send_request_cap() budgets the entry count with req_mtu, but the
+ * buffers are sized from cur_rx_buf_sz. Cap against that size here.
+ */
+static u64 rx_pool_entries(struct ibmvnic_adapter *adapter)
+{
+	u64 buff_size = ALIGN(adapter->cur_rx_buf_sz, L1_CACHE_BYTES);
+
+	if (!buff_size)
+		return adapter->req_rx_add_entries_per_subcrq;
+
+	return min_t(u64, adapter->req_rx_add_entries_per_subcrq,
+		     IBMVNIC_LTB_SET_SIZE / buff_size);
+}
+
 /**
  * reuse_rx_pools() - Check if the existing rx pools can be reused.
  * @adapter: ibmvnic adapter
@@ -1048,7 +1066,7 @@ static bool reuse_rx_pools(struct ibmvnic_adapter *adapter)
 	new_num_pools = adapter->req_rx_queues;
 
 	old_pool_size = adapter->prev_rx_pool_size;
-	new_pool_size = adapter->req_rx_add_entries_per_subcrq;
+	new_pool_size = rx_pool_entries(adapter);
 
 	old_buff_size = adapter->prev_rx_buf_sz;
 	new_buff_size = adapter->cur_rx_buf_sz;
@@ -1082,7 +1100,7 @@ static int init_rx_pools(struct net_device *netdev)
 	u64 buff_size;
 	int i, j, rc;
 
-	pool_size = adapter->req_rx_add_entries_per_subcrq;
+	pool_size = rx_pool_entries(adapter);
 	num_pools = adapter->req_rx_queues;
 	buff_size = adapter->cur_rx_buf_sz;
 
@@ -2000,7 +2018,6 @@ static void clean_rx_pools(struct ibmvnic_adapter *adapter)
 {
 	struct ibmvnic_rx_pool *rx_pool;
 	struct ibmvnic_rx_buff *rx_buff;
-	u64 rx_entries;
 	int rx_scrqs;
 	int i, j;
 
@@ -2008,7 +2025,6 @@ static void clean_rx_pools(struct ibmvnic_adapter *adapter)
 		return;
 
 	rx_scrqs = adapter->num_active_rx_pools;
-	rx_entries = adapter->req_rx_add_entries_per_subcrq;
 
 	/* Free any remaining skbs in the rx buffer pools */
 	for (i = 0; i < rx_scrqs; i++) {
@@ -2017,7 +2033,7 @@ static void clean_rx_pools(struct ibmvnic_adapter *adapter)
 			continue;
 
 		netdev_dbg(adapter->netdev, "Cleaning rx_pool[%d]\n", i);
-		for (j = 0; j < rx_entries; j++) {
+		for (j = 0; j < rx_pool->size; j++) {
 			rx_buff = &rx_pool->rx_buff[j];
 			if (rx_buff && rx_buff->skb) {
 				dev_kfree_skb_any(rx_buff->skb);
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v1 2/6] ibmvnic: honour the requested mtu instead of reverting to a fallback

[Mingming Cao]

No mtu other than 1500 or 9000 can be set on an ibmvnic interface. Any
other value is accepted by ip(8) but silently reverts:

  # ip link set mtu 1414 env8
  ibmvnic 30000008: req=1428, rsp=1514 in mtu queue, retrying.
  mtu of 1428 is not supported. Reverting.
  ibmvnic 30000008: req=1514, rsp=1514 in mtu queue, retrying.
  mtu of 1514 is not supported. Reverting.
  # ip link show env8 | grep -o 'mtu [0-9]*'
  mtu 1500

The backing device runs at one of a small set of fixed sizes, so the
vnicserver answers every REQ_MTU between the advertised minimum and
maximum with PARTIALSUCCESS and the size it settled on: 1514 for
anything up to 1514, 9014 for anything above that. It is reporting what
the backing device carries. Requests outside the advertised range never
get here, since dev_set_mtu() rejects them against netdev->min_mtu and
netdev->max_mtu.

The driver read that as a rejection. It reverted req_mtu to
fallback.mtu and re-ran the exchange, which is why the log above shows
the revert twice: the fallback is in range as well, so it partially
succeeds too and the mtu settles on whatever the fallback was.

Treat a PARTIALSUCCESS that covers the request as the confirmation it
is, and keep the requested value.

The response can also come back below the request. The advertised
maximum is the protocol ceiling rather than a promise about the current
backing configuration, so a device that is not set up for the larger
size answers with the smaller one it does carry. That does not cover
the request and must not be published, so it keeps the existing retry,
which now also stores the response value rather than reverting to a
fallback. It therefore converges on a size the device can carry instead
of re-requesting the same number. Other capabilities are unchanged.

Later patches in this series make the resets that follow a real mtu
change safe (and, on net-next, skip them when the buffers already fit).

Fixes: e79138034068 ("ibmvnic: Revert to previous mtu when unsupported value requested")
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Vaishnavi Bhat <vaishnavi@linux.ibm.com>
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 34 ++++++++++++++++++------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 86e643ee6b3b..f5f9c0d5b4e6 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -5502,12 +5502,17 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq,
 {
 	struct device *dev = &adapter->vdev->dev;
 	u64 *req_value;
+	u64 rsp_value;
 	char *name;
+	u16 cap;
 
 	atomic_dec(&adapter->running_cap_crqs);
 	netdev_dbg(adapter->netdev, "Outstanding request-caps: %d\n",
 		   atomic_read(&adapter->running_cap_crqs));
-	switch (be16_to_cpu(crq->request_capability_rsp.capability)) {
+
+	cap = be16_to_cpu(crq->request_capability_rsp.capability);
+
+	switch (cap) {
 	case REQ_TX_QUEUES:
 		req_value = &adapter->req_tx_queues;
 		name = "tx";
@@ -5546,21 +5551,22 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq,
 	case SUCCESS:
 		break;
 	case PARTIALSUCCESS:
-		dev_info(dev, "req=%lld, rsp=%ld in %s queue, retrying.\n",
-			 *req_value,
-			 (long)be64_to_cpu(crq->request_capability_rsp.number),
-			 name);
-
-		if (be16_to_cpu(crq->request_capability_rsp.capability) ==
-		    REQ_MTU) {
-			pr_err("mtu of %llu is not supported. Reverting.\n",
-			       *req_value);
-			*req_value = adapter->fallback.mtu;
-		} else {
-			*req_value =
-				be64_to_cpu(crq->request_capability_rsp.number);
+		rsp_value = be64_to_cpu(crq->request_capability_rsp.number);
+
+		/* Covering PARTIALSUCCESS: keep the request. Otherwise
+		 * retry with rsp_value.
+		 */
+		if (cap == REQ_MTU && rsp_value >= *req_value) {
+			netdev_dbg(adapter->netdev,
+				   "backing mtu %llu covers requested %llu\n",
+				   rsp_value, *req_value);
+			break;
 		}
 
+		dev_info(dev, "req=%lld, rsp=%lld in %s queue, retrying.\n",
+			 *req_value, rsp_value, name);
+		*req_value = rsp_value;
+
 		send_request_cap(adapter, 1);
 		return;
 	default:
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v1 3/6] ibmvnic: do not unmap long term buffers across a crq reconnect

[Mingming Cao]

Once in-range mtus are honoured, changing the mtu - especially up to
jumbo - becomes a normal admin action rather than a rare failover path.
Those changes can make the VIOS reconfigure the backing device and
re-establish the crq connection. Every long term buffer mapping belongs
to the connection it was registered on, so the VIOS drops all of them,
and the driver then asks it to unmap buffers it no longer knows about:

  ibmvnic 30000003 env3: MTU change 1400->9000: slow path (reset required)
  ibmvnic 30000003: Partner initialization complete
  ibmvnic 30000003: Partner protocol version is 1
  ibmvnic 30000003: Error 4 in REQUEST_UNMAP_RSP
  ibmvnic 30000003: Error 4 in REQUEST_UNMAP_RSP
  ...

Error 4 is H_PARAMETER, one per buffer still on the books. Traffic keeps
flowing; the damage is to the log. A single jumbo mtu change prints a
flood of these lines, which drowns out real failures and makes every
reset look broken.

free_long_term_buff() decides from reset_reason alone, listing the
resets after which the VIOS is known to have unmapped everything. That
cannot describe this case. The connection is re-established by the
partner partway through the reset, so whether an unmap is still valid
depends on when it is sent rather than on why the reset was started,
which is why the errors come and go between otherwise identical runs.

Give the connection a generation, bump it whenever the crq goes away,
and record it in each long term buffer as that buffer is mapped. One
whose generation no longer matches was mapped on a connection that has
since gone, so release it locally and leave the VIOS alone. The field
fits in the padding that already followed map_id, so the struct stays
32 bytes and the few hundred mappings an adapter can hold cost nothing
extra.

The reset_reason tests are now redundant, since all three tear the
connection down, but leave them for the moment.

Fixes: 7d3a7b9ea59d ("ibmvnic: skip send_request_unmap for timeout reset")
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Vaishnavi Bhat <vaishnavi@linux.ibm.com>
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 30 +++++++++++++++++++++++-------
 drivers/net/ethernet/ibm/ibmvnic.h |  2 ++
 2 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index f5f9c0d5b4e6..e875f43a1ea1 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -495,6 +495,9 @@ static int alloc_long_term_buff(struct ibmvnic_adapter *adapter,
 	adapter->fw_done_rc = 0;
 	reinit_completion(&adapter->fw_done);
 
+	/* Snapshot gen before the map wait so a reconnect mid-wait is stale. */
+	ltb->crq_gen = adapter->crq.gen;
+
 	rc = send_request_map(adapter, ltb->addr, ltb->size, ltb->map_id);
 	if (rc) {
 		dev_err(dev, "send_request_map failed, rc = %d\n", rc);
@@ -529,11 +532,12 @@ static void free_long_term_buff(struct ibmvnic_adapter *adapter,
 	if (!ltb->buff)
 		return;
 
-	/* VIOS automatically unmaps the long term buffer at remote
-	 * end for the following resets:
-	 * FAILOVER, MOBILITY, TIMEOUT.
+	/* Skip unmap if mapped on a prior crq generation, or after resets
+	 * where the VIOS has already dropped mappings (FAILOVER/MOBILITY/
+	 * TIMEOUT).
 	 */
-	if (adapter->reset_reason != VNIC_RESET_FAILOVER &&
+	if (ltb->crq_gen == adapter->crq.gen &&
+	    adapter->reset_reason != VNIC_RESET_FAILOVER &&
 	    adapter->reset_reason != VNIC_RESET_MOBILITY &&
 	    adapter->reset_reason != VNIC_RESET_TIMEOUT)
 		send_request_unmap(adapter, ltb->map_id);
@@ -5974,6 +5978,18 @@ static int handle_query_phys_parms_rsp(union ibmvnic_crq *crq,
 	return rc;
 }
 
+/**
+ * ibmvnic_crq_deactivate() - Mark the crq connection inactive
+ * @crq: crq queue
+ *
+ * Bump gen so LTB mappings from the old connection can be freed locally.
+ */
+static void ibmvnic_crq_deactivate(struct ibmvnic_crq_queue *crq)
+{
+	crq->active = false;
+	crq->gen++;
+}
+
 static void ibmvnic_handle_crq(union ibmvnic_crq *crq,
 			       struct ibmvnic_adapter *adapter)
 {
@@ -6036,7 +6052,7 @@ static void ibmvnic_handle_crq(union ibmvnic_crq *crq,
 		return;
 	case IBMVNIC_CRQ_XPORT_EVENT:
 		netif_carrier_off(netdev);
-		adapter->crq.active = false;
+		ibmvnic_crq_deactivate(&adapter->crq);
 		/* terminate any thread waiting for a response
 		 * from the device
 		 */
@@ -6241,7 +6257,7 @@ static int ibmvnic_reset_crq(struct ibmvnic_adapter *adapter)
 
 	memset(crq->msgs, 0, PAGE_SIZE);
 	crq->cur = 0;
-	crq->active = false;
+	ibmvnic_crq_deactivate(crq);
 
 	/* And re-open it again */
 	rc = plpar_hcall_norets(H_REG_CRQ, vdev->unit_address,
@@ -6276,7 +6292,7 @@ static void release_crq_queue(struct ibmvnic_adapter *adapter)
 			 DMA_BIDIRECTIONAL);
 	free_page((unsigned long)crq->msgs);
 	crq->msgs = NULL;
-	crq->active = false;
+	ibmvnic_crq_deactivate(crq);
 }
 
 static int init_crq_queue(struct ibmvnic_adapter *adapter)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.h b/drivers/net/ethernet/ibm/ibmvnic.h
index 480dc587078f..4cfedae5d89d 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.h
+++ b/drivers/net/ethernet/ibm/ibmvnic.h
@@ -794,6 +794,7 @@ struct ibmvnic_crq_queue {
 	/* Used for serialization of msgs, cur */
 	spinlock_t lock;
 	bool active;
+	u32 gen; /* bumped when the crq connection drops */
 	char name[32];
 };
 
@@ -839,6 +840,7 @@ struct ibmvnic_long_term_buff {
 	dma_addr_t addr;
 	u64 size;
 	u8 map_id;
+	u32 crq_gen; /* crq.gen when this buffer was mapped */
 };
 
 struct ibmvnic_ltb_set {
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v1 4/6] ibmvnic: allocate new buffer pools before releasing the old ones

[Mingming Cao]

An mtu change goes through wait_for_reset() and then init_rx_pools() /
init_tx_pools(). Those helpers release the existing pools before
allocating their replacements:

  release_rx_pools(adapter);

  adapter->rx_pool = kzalloc_objs(struct ibmvnic_rx_pool, num_pools);
  if (!adapter->rx_pool) {
          dev_err(dev, "Failed to allocate rx pools\n");
          return -ENOMEM;
  }

If that allocation fails mid-reset the interface is left with no pools
and stays down after what should have been a refused mtu change, when
it could have kept running with the pools it still had.

Allocate first and only release once the allocation has succeeded. The
tx side has to swap ->tx_pool and ->tso_pool in together, so that
release_tx_pools() never sees one set without the other.

These are small allocations and failing them is unlikely, so this is
about the ordering rather than about any failure seen in the field.
alloc_long_term_buff() has the same shape for a much larger allocation
on the mtu-grow path and is dealt with next.

Fixes: 489de956e7a2 ("ibmvnic: Reuse rx pools when possible")
Fixes: bbd809305bc7 ("ibmvnic: Reuse tx pools when possible")
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Vaishnavi Bhat <vaishnavi@linux.ibm.com>
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 37 ++++++++++++++++--------------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index e875f43a1ea1..24c8acc42a93 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -1098,6 +1098,7 @@ static int init_rx_pools(struct net_device *netdev)
 {
 	struct ibmvnic_adapter *adapter = netdev_priv(netdev);
 	struct device *dev = &adapter->vdev->dev;
+	struct ibmvnic_rx_pool *new_rx_pool;
 	struct ibmvnic_rx_pool *rx_pool;
 	u64 num_pools;
 	u64 pool_size;		/* # of buffers in one pool */
@@ -1113,15 +1114,16 @@ static int init_rx_pools(struct net_device *netdev)
 		goto update_ltb;
 	}
 
-	/* Allocate/populate the pools. */
-	release_rx_pools(adapter);
-
-	adapter->rx_pool = kzalloc_objs(struct ibmvnic_rx_pool, num_pools);
-	if (!adapter->rx_pool) {
+	/* Allocate before release so a failure keeps the old pools. */
+	new_rx_pool = kzalloc_objs(struct ibmvnic_rx_pool, num_pools);
+	if (!new_rx_pool) {
 		dev_err(dev, "Failed to allocate rx pools\n");
 		return -ENOMEM;
 	}
 
+	release_rx_pools(adapter);
+	adapter->rx_pool = new_rx_pool;
+
 	/* Set num_active_rx_pools early. If we fail below after partial
 	 * allocation, release_rx_pools() will know how many to look for.
 	 */
@@ -1333,6 +1335,8 @@ static int init_tx_pools(struct net_device *netdev)
 {
 	struct ibmvnic_adapter *adapter = netdev_priv(netdev);
 	struct device *dev = &adapter->vdev->dev;
+	struct ibmvnic_tx_pool *new_tso_pool;
+	struct ibmvnic_tx_pool *new_tx_pool;
 	int num_pools;
 	u64 pool_size;		/* # of buffers in pool */
 	u64 buff_size;
@@ -1349,26 +1353,25 @@ static int init_tx_pools(struct net_device *netdev)
 		goto update_ltb;
 	}
 
-	/* Allocate/populate the pools. */
-	release_tx_pools(adapter);
-
+	/* Allocate before release so a failure keeps the old pools. */
 	pool_size = adapter->req_tx_entries_per_subcrq;
 	num_pools = adapter->num_active_tx_scrqs;
 
-	adapter->tx_pool = kzalloc_objs(struct ibmvnic_tx_pool, num_pools);
-	if (!adapter->tx_pool)
+	new_tx_pool = kzalloc_objs(struct ibmvnic_tx_pool, num_pools);
+	if (!new_tx_pool)
 		return -ENOMEM;
 
-	adapter->tso_pool = kzalloc_objs(struct ibmvnic_tx_pool, num_pools);
-	/* To simplify release_tx_pools() ensure that ->tx_pool and
-	 * ->tso_pool are either both NULL or both non-NULL.
-	 */
-	if (!adapter->tso_pool) {
-		kfree(adapter->tx_pool);
-		adapter->tx_pool = NULL;
+	new_tso_pool = kzalloc_objs(struct ibmvnic_tx_pool, num_pools);
+	if (!new_tso_pool) {
+		kfree(new_tx_pool);
 		return -ENOMEM;
 	}
 
+	/* Swap both in together for release_tx_pools(). */
+	release_tx_pools(adapter);
+	adapter->tx_pool = new_tx_pool;
+	adapter->tso_pool = new_tso_pool;
+
 	/* Set num_active_tx_pools early. If we fail below after partial
 	 * allocation, release_tx_pools() will know how many to look for.
 	 */
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v1 5/6] ibmvnic: allocate a new long term buffer before freeing the old one

[Mingming Cao]

When an mtu change grows the pools, alloc_long_term_buff() releases the
existing buffer as soon as it knows the size has changed, and only then
asks for the replacement:

  if (!reuse_ltb(ltb, size)) {
          prev = ltb->size;
          free_long_term_buff(adapter, ltb);
  }

  if (ltb->buff) {
          ...
  } else {
          ltb->buff = dma_alloc_coherent(dev, size, &ltb->addr,
                                         GFP_KERNEL);
          if (!ltb->buff) {
                  dev_err(dev, "Couldn't alloc long term buffer\n");
                  return -ENOMEM;
          }

This one is megabytes of physically contiguous memory, so it can fail
while there is plenty free. By then the old buffer is gone and the pool
it belongs to has nothing, which takes the interface down.

Allocate into a local buffer and only commit it once the allocation has
succeeded. A failure now returns with the old buffer still in place, so
the mtu change is refused and the interface keeps running at the size
it had.

Both buffers are live across the swap, so an mtu increase now needs the
old and the new at once. That makes the allocation somewhat more likely
to fail in the low memory conditions this is meant to survive, but
failing with the old buffer intact is still the better outcome: the
caller can decline the change instead of losing the interface.

The new map id is taken before the old one is released, so the two
differ across the swap. That is harmless with 255 of them.

Fixes: f8ac0bfa7d7a ("ibmvnic: Reuse LTB when possible")
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Vaishnavi Bhat <vaishnavi@linux.ibm.com>
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 30 ++++++++++++++++--------------
 1 file changed, 16 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 24c8acc42a93..88d0c231a74f 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -455,6 +455,7 @@ static bool reuse_ltb(struct ibmvnic_long_term_buff *ltb, int size)
 static int alloc_long_term_buff(struct ibmvnic_adapter *adapter,
 				struct ibmvnic_long_term_buff *ltb, int size)
 {
+	struct ibmvnic_long_term_buff new_ltb = {};
 	struct device *dev = &adapter->vdev->dev;
 	u64 prev = 0;
 	int rc;
@@ -464,28 +465,29 @@ static int alloc_long_term_buff(struct ibmvnic_adapter *adapter,
 			"LTB size changed from 0x%llx to 0x%x, reallocating\n",
 			 ltb->size, size);
 		prev = ltb->size;
-		free_long_term_buff(adapter, ltb);
-	}
 
-	if (ltb->buff) {
-		dev_dbg(dev, "Reusing LTB [map %d, size 0x%llx]\n",
-			ltb->map_id, ltb->size);
-	} else {
-		ltb->buff = dma_alloc_coherent(dev, size, &ltb->addr,
-					       GFP_KERNEL);
-		if (!ltb->buff) {
+		/* Allocate first so failure leaves the old buffer in place. */
+		new_ltb.buff = dma_alloc_coherent(dev, size, &new_ltb.addr,
+						  GFP_KERNEL);
+		if (!new_ltb.buff) {
 			dev_err(dev, "Couldn't alloc long term buffer\n");
 			return -ENOMEM;
 		}
-		ltb->size = size;
+		new_ltb.size = size;
 
-		ltb->map_id = find_first_zero_bit(adapter->map_ids,
-						  MAX_MAP_ID);
-		bitmap_set(adapter->map_ids, ltb->map_id, 1);
+		new_ltb.map_id = find_first_zero_bit(adapter->map_ids,
+						     MAX_MAP_ID);
+		bitmap_set(adapter->map_ids, new_ltb.map_id, 1);
 
 		dev_dbg(dev,
 			"Allocated new LTB [map %d, size 0x%llx was 0x%llx]\n",
-			 ltb->map_id, ltb->size, prev);
+			 new_ltb.map_id, new_ltb.size, prev);
+
+		free_long_term_buff(adapter, ltb);
+		*ltb = new_ltb;
+	} else {
+		dev_dbg(dev, "Reusing LTB [map %d, size 0x%llx]\n",
+			ltb->map_id, ltb->size);
 	}
 
 	/* Ensure ltb is zeroed - specially when reusing it. */
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v1 6/6] ibmvnic: change the mtu without a reset where the buffers allow it

[Mingming Cao]

With the earlier patches in this series, in-range mtus are honoured
instead of snapped to 1500 or 9000. Every such change still cycles the
adapter through wait_for_reset(), which tears down the CRQ, logs in
again and rebuilds every queue. The link goes down for around two
seconds each time, even when the change needs nothing from the VIOS.

Most of them do not. The backing device runs at one of a small set of
fixed sizes and the vnicserver reports that size when it answers
REQ_MTU, so any mtu within it is already carried end to end. Record it
as adapter->backing_mtu (set when the covering PARTIALSUCCESS path from
the honour-mtu patch accepts a request) and skip the reset when the new
mtu is within it and the tx buffers are large enough as they are, which
covers every decrease and the increases that stay inside the current
buffer size.

Both conditions are needed. Buffer sizes are rounded up to a cache
line, so buffers alone would also admit an mtu somewhat past what the
backing device carries: with a 1514 backing mtu, everything up to 1532
aligns to the same 1536-byte buffer. Those frames would leave the
partition and be dropped by a backing device still configured for the
smaller size.

An increase past backing_mtu still resets, since it has to be put to
the vnicserver rather than assumed. Whatever comes back becomes the new
backing_mtu, so if the device does move up, later changes within the
larger size settle without a reset, and if it does not, the mtu is
renegotiated down to what the device carries and the fast path keeps
measuring against the truth.

Buffers are left at their existing size on a decrease rather than
shrunk. reuse_tx_pools() compares prev_mtu against req_mtu, so the next
reset for any reason reallocates them.

Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Vaishnavi Bhat <vaishnavi@linux.ibm.com>
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
---
 drivers/net/ethernet/ibm/ibmvnic.c | 35 +++++++++++++++++++++++++++---
 drivers/net/ethernet/ibm/ibmvnic.h |  1 +
 2 files changed, 33 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 88d0c231a74f..1d18a0e13cad 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -3705,8 +3705,34 @@ out:
 static int ibmvnic_change_mtu(struct net_device *netdev, int new_mtu)
 {
 	struct ibmvnic_adapter *adapter = netdev_priv(netdev);
+	u64 new_mtu_with_hdr = new_mtu + ETH_HLEN;
+	u64 old_buff_size, new_buff_size;
+
+	if (adapter->req_mtu == new_mtu_with_hdr)
+		return 0;
+
+	old_buff_size = ALIGN(adapter->prev_mtu + VLAN_HLEN, L1_CACHE_BYTES);
+	new_buff_size = ALIGN(new_mtu_with_hdr + VLAN_HLEN, L1_CACHE_BYTES);
+
+	/* Skip the reset when backing_mtu and the current buffers already
+	 * cover the new mtu. Keep desired.mtu in sync with req_mtu.
+	 */
+	if (new_mtu_with_hdr <= adapter->backing_mtu &&
+	    new_buff_size <= old_buff_size) {
+		netdev_dbg(netdev, "mtu %u->%d without reset\n",
+			   netdev->mtu, new_mtu);
+
+		WRITE_ONCE(netdev->mtu, new_mtu);
+		adapter->req_mtu = new_mtu_with_hdr;
+		adapter->desired.mtu = new_mtu_with_hdr;
+
+		return 0;
+	}
+
+	netdev_dbg(netdev, "mtu %u->%d needs larger buffers, resetting\n",
+		   netdev->mtu, new_mtu);
 
-	adapter->desired.mtu = new_mtu + ETH_HLEN;
+	adapter->desired.mtu = new_mtu_with_hdr;
 
 	return wait_for_reset(adapter);
 }
@@ -5558,14 +5584,17 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq,
 
 	switch (crq->request_capability_rsp.rc.code) {
 	case SUCCESS:
+		if (cap == REQ_MTU)
+			adapter->backing_mtu = *req_value;
 		break;
 	case PARTIALSUCCESS:
 		rsp_value = be64_to_cpu(crq->request_capability_rsp.number);
 
-		/* Covering PARTIALSUCCESS: keep the request. Otherwise
-		 * retry with rsp_value.
+		/* Covering PARTIALSUCCESS: keep the request and record
+		 * backing_mtu. Otherwise retry with rsp_value.
 		 */
 		if (cap == REQ_MTU && rsp_value >= *req_value) {
+			adapter->backing_mtu = rsp_value;
 			netdev_dbg(adapter->netdev,
 				   "backing mtu %llu covers requested %llu\n",
 				   rsp_value, *req_value);
diff --git a/drivers/net/ethernet/ibm/ibmvnic.h b/drivers/net/ethernet/ibm/ibmvnic.h
index 4cfedae5d89d..80e1b2f6ede5 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.h
+++ b/drivers/net/ethernet/ibm/ibmvnic.h
@@ -1020,6 +1020,7 @@ struct ibmvnic_adapter {
 	u64 max_mtu;
 	u64 req_mtu;
 	u64 prev_mtu;
+	u64 backing_mtu; /* mtu the backing device currently carries */
 	u64 max_multicast_filters;
 	u64 vlan_header_insertion;
 	u64 rx_vlan_header_insertion;
-- 
2.50.1 (Apple Git-155)

Re: [PATCH net-next v1 0/6] ibmvnic: honour flexible in-range MTU values

[Jakub Kicinski]

On Wed,  5 Aug 2026 15:43:55 -0700 Mingming Cao wrote:
> Subject: [PATCH net-next v1 0/6] ibmvnic: honour flexible in-range MTU values

Warning: drivers/net/ethernet/ibm/ibmvnic.c:1031 No description found for return value of 'rx_pool_entries'


please DO NOT repost this until your previous series is reviewed and
merged. Please read:
https://www.kernel.org/doc/html/latest/process/maintainer-netdev.html
There aren't supposed to be more than 15 outstanding patches form one
person or company to one tree. We are flooded with AI-assisted code
so reviews are slow. Your place in review queue depends on your
credibility with the community so please help review other patches
or just wait your turn.

Re: [PATCH net-next v1 0/6] ibmvnic: honour flexible in-range MTU values

[mingming cao]

On 8/5/26 7:38 PM, Jakub Kicinski wrote:
> On Wed,  5 Aug 2026 15:43:55 -0700 Mingming Cao wrote:
>> Subject: [PATCH net-next v1 0/6] ibmvnic: honour flexible in-range MTU values
> Warning: drivers/net/ethernet/ibm/ibmvnic.c:1031 No description found for return value of 'rx_pool_entries'
>
>
> please DO NOT repost this until your previous series is reviewed and
> merged. Please read:
> https://www.kernel.org/doc/html/latest/process/maintainer-netdev.html
> There aren't supposed to be more than 15 outstanding patches form one
> person or company to one tree. We are flooded with AI-assisted code
> so reviews are slow. Your place in review queue depends on your
> credibility with the community so please help review other patches
> or just wait your turn.
Hi Jakub,

Thanks for the feedback.


Sorry for violating the outstanding patch limit. I wasn't aware of that 
rule, so thanks for pointing me to it.
I understand the review queue is already quite busy. I'll wait for the 
ibmveth MQ support series (v4, 14 patches) to progress through review 
before reposting this one.
In the meantime, I'm working through Sakashi's review comments on the 
ibmveth VQ v4 series.

For the ibmvnic MTU support series, I'll hold off on posting a v2 for 
now and will address the kernel-doc warning in the next revision.

Thanks again for the guidance.
Mingming