* [PATCH net-next v8 01/15] ibmveth: Add MQ RX hypercall wrappers and call definitions
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 02/15] ibmveth: Prepare MQ RX adapter data structures Mingming Cao
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1,
linux-perf-users, acme, namhyung, peterz, mingo, mark.rutland,
alexander.shishkin, jolsa, irogers, adrian.hunter, james.clark,
shivani, ssrish, nayna, gautam, linux-kernel
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=true, Size: 11678 bytes --]
Single-queue ibmveth only needs h_register_logical_lan() plus legacy
buffer add/free calls. MQ RX uses per-queue handles, so the driver must
also be able to register/deregister subordinate queues and post
buffers against a specific queue handle.
Add the PHYP call IDs for:
H_REG_LOGICAL_LAN_QUEUE (0x49C)
H_ADD_LOGICAL_LAN_BUFFERS_QUEUE (0x4A0)
H_FREE_LOGICAL_LAN_QUEUE (0x4A8)
as defined in PAPR 11.20.00. The gaps at 0x498 (reserved) and 0x4A4
(reserved for H_FREE_LOGICAL_LAN_BUFFER_QUEUE) are intentional; this
series tears queues down with H_FREE_LOGICAL_LAN_QUEUE and does not add
the buffer-queue free hcall.
Raising MAX_HCALL_OPCODE for these IDs also widens KVM's
kvm_arch.enabled_hcalls bitmap and the range KVM_CAP_PPC_ENABLE_HCALL
accepts. Neither is user-visible: KVM implements none of the three, so
the ioctl still rejects them, and the bitmap rounds to the same five
unsigned longs, so struct kvm_arch does not change size. The pseries
hcall_inst array is also sized from MAX_HCALL_OPCODE, so those
opcodes can be counted. A debugfs row appears only after the
opcode has been called.
Add ibmveth.h wrapper helpers (h_register_logical_lan_queue(),
h_add_logical_lan_buffers_queue(), h_free_logical_lan_queue()) with
argument ordering matching the existing ibmveth hcall wrappers.
h_free_logical_lan_queue() uses plpar_hcall_norets() like
h_free_logical_lan(). Its kdoc lists the hcall's H_PARAMETER meaning
from the PHYP interface: the unit has no logical LAN, or the handle
names no registered queue (already freed). Also add
h_register_logical_lan_with_handle() so queue 0 can capture the PHYP
queue handle in MQ mode. Both new registration wrappers use
plpar_hcall() rather than plpar_hcall9(), so they do not read unwritten
stack slots, and store their out-parameters only on H_SUCCESS.
Name the three opcodes in tools/perf/python/powerpc-hcalls.py
so perf hcall traces show them. The two reserved slots stay unnamed.
This patch is intentionally plumbing only: no runtime behavior change
yet. Legacy firmware keeps H_REGISTER_LOGICAL_LAN and the existing
buffer hcalls. The new wrappers are used only when a later commit sets
multi_queue from H_ILLAN_ATTRIBUTES.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- H_FREE_LOGICAL_LAN_QUEUE kdoc: subordinate handles only (queue 0 goes
with H_FREE_LOGICAL_LAN); H_PARAMETER means no logical LAN or no such
registered queue; the with_handle wrapper's R4 handle exists only on
firmware that reports MQ RX support
- commit message: out-params are stored only on H_SUCCESS; hcall_inst
counts the new opcodes
Changes in v7:
- Drop the duplicated "Both new registration wrappers use
plpar_hcall()…" sentence
- Register and free-queue Return: also H_FUNCTION. Add-buffers
already had it in v6; all three PAPR 11.20.00 opcodes return that
status on older firmware
- h_register_logical_lan_queue Return: also
H_BUSY / H_LONG_BUSY_*; the caller retries those,
and free-queue already documented them
- tools/perf powerpc-hcalls.py: name 0x49C / 0x4A0 / 0x4A8.
0x498 and 0x4A4 stay unnamed (reserved holes)
Changes in v6:
- both new registration wrappers use plpar_hcall() instead of
plpar_hcall9(). They do not need nine args, and plpar_hcall9() was
reading three unwritten stack slots
- Widen add-buffers-queue and free-queue @queue_handle kdoc: handle may
come from h_register_logical_lan_queue() or
h_register_logical_lan_with_handle() (queue 0)
- add-buffers-queue Return: also H_FUNCTION (MQ firmware missing that
hcall)
Changes in v5:
- Cite PAPR 11.20.00 for MQ hcall IDs 0x49C / 0x4A0 / 0x4A8 (v4 named
the opcodes only)
- Call out reserved gaps at 0x498 and 0x4A4 (0x4A4 =
H_FREE_LOGICAL_LAN_BUFFER_QUEUE); decision: do not add that hcall -
queue teardown uses H_FREE_LOGICAL_LAN_QUEUE
- Note MAX_HCALL_OPCODE raise also widens KVM enabled_hcalls /
KVM_CAP_PPC_ENABLE_HCALL accepted range (cross-subsystem)
- Rename h_reg_logical_lan_queue() -> h_register_logical_lan_queue() to
match h_register_logical_lan() / h_free_logical_lan()
- Unify queue_handle out-params as unsigned long * on both
h_register_logical_lan_queue() and h_register_logical_lan_with_handle()
(v4 used u64 * on with_handle)
- h_free_logical_lan_queue() uses plpar_hcall_norets() like
h_free_logical_lan() (v4 used plpar_hcall9 with unused retbuf)
Changes in v4:
- Document @queue_handle and @irq in h_reg_logical_lan_queue() kdoc.
- Wrap h_register_logical_lan_with_handle() prototype for readability /
checkpatch.
- Drop unused H_FREE_LOGICAL_LAN_BUFFER_QUEUE wrapper/opcode (no caller;
buffer return is local harvest + queue free).
arch/powerpc/include/asm/hvcall.h | 6 +-
drivers/net/ethernet/ibm/ibmveth.h | 146 ++++++++++++++++++++++++++++
tools/perf/python/powerpc-hcalls.py | 4 +
3 files changed, 155 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index dff90a7d7f70..cb0ea53491e6 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -362,7 +362,11 @@
#define H_GUEST_DELETE 0x488
#define H_PKS_WRAP_OBJECT 0x490
#define H_PKS_UNWRAP_OBJECT 0x494
-#define MAX_HCALL_OPCODE H_PKS_UNWRAP_OBJECT
+/* 0x498 reserved; 0x4A4 = H_FREE_LOGICAL_LAN_BUFFER_QUEUE (unused here) */
+#define H_REG_LOGICAL_LAN_QUEUE 0x49C
+#define H_ADD_LOGICAL_LAN_BUFFERS_QUEUE 0x4A0
+#define H_FREE_LOGICAL_LAN_QUEUE 0x4A8
+#define MAX_HCALL_OPCODE H_FREE_LOGICAL_LAN_QUEUE
/* Scope args for H_SCM_UNBIND_ALL */
#define H_UNBIND_SCOPE_ALL (0x1)
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index be0939caa328..82f22de16394 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -68,6 +68,152 @@ static inline long h_add_logical_lan_buffers(unsigned long unit_address,
desc5, desc6, desc7, desc8);
}
+/**
+ * h_register_logical_lan_queue - Register a subordinate receive queue
+ * @unit_address: Device unit address
+ * @buffer_list: DMA address of 4KB page for tracking registered buffers
+ * @rec_queue: Buffer descriptor of receive queue
+ * @queue_handle: Output queue handle on success (may be NULL)
+ * @irq: Output hypervisor IRQ number on success (may be NULL)
+ *
+ * Registers a subordinate receive queue with the hypervisor.
+ *
+ * Return:
+ * H_SUCCESS (0) on success
+ * H_PARAMETER if parameters are invalid
+ * H_BUSY / H_LONG_BUSY_* if the resource is busy; retry
+ * H_FUNCTION if firmware does not support this hcall
+ *
+ * On success, hypervisor returns:
+ * R3: H_SUCCESS
+ * R4: Queue handle
+ * R5: IRQ number for this queue
+ */
+static inline long
+h_register_logical_lan_queue(unsigned long unit_address,
+ unsigned long buffer_list,
+ unsigned long rec_queue,
+ unsigned long *queue_handle,
+ unsigned long *irq)
+{
+ unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
+ long rc;
+
+ rc = plpar_hcall(H_REG_LOGICAL_LAN_QUEUE,
+ retbuf, unit_address,
+ buffer_list, rec_queue);
+
+ if (rc == H_SUCCESS) {
+ if (queue_handle)
+ *queue_handle = retbuf[0];
+ if (irq)
+ *irq = retbuf[1];
+ }
+
+ return rc;
+}
+
+/**
+ * h_add_logical_lan_buffers_queue - Add buffers to an RX queue by handle
+ * @unit_address: Device unit address
+ * @queue_handle: Queue handle from h_register_logical_lan_queue() or
+ * h_register_logical_lan_with_handle() (queue 0)
+ * @buffersznum: Buffer size (upper 32 bits) | count (lower 32 bits)
+ * @ioba12: Buffer addresses 1 and 2 packed ((addr1 << 32) | addr2)
+ * @ioba34: Buffer addresses 3 and 4 packed
+ * @ioba56: Buffer addresses 5 and 6 packed
+ * @ioba78: Buffer addresses 7 and 8 packed
+ * @ioba910: Buffer addresses 9 and 10 packed
+ * @ioba1112: Buffer addresses 11 and 12 packed
+ *
+ * Return:
+ * H_SUCCESS - All buffers added successfully
+ * H_PARAMETER - Invalid parameters
+ * H_HARDWARE - Hardware error
+ * H_FUNCTION - Firmware does not support this hcall
+ */
+static inline long h_add_logical_lan_buffers_queue(unsigned long unit_address,
+ unsigned long queue_handle,
+ unsigned long buffersznum,
+ unsigned long ioba12,
+ unsigned long ioba34,
+ unsigned long ioba56,
+ unsigned long ioba78,
+ unsigned long ioba910,
+ unsigned long ioba1112)
+{
+ unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+
+ return plpar_hcall9(H_ADD_LOGICAL_LAN_BUFFERS_QUEUE,
+ retbuf, unit_address,
+ queue_handle, buffersznum,
+ ioba12, ioba34, ioba56,
+ ioba78, ioba910, ioba1112);
+}
+
+/**
+ * h_free_logical_lan_queue - Deregister a subordinate receive queue
+ * @unit_address: Device unit address
+ * @queue_handle: Queue handle from h_register_logical_lan_queue()
+ *
+ * Deregisters and frees all structures associated with the subordinate
+ * queue. Queue 0 is torn down with the whole logical LAN by
+ * H_FREE_LOGICAL_LAN.
+ *
+ * Return:
+ * H_SUCCESS - Queue freed successfully
+ * H_PARAMETER - The unit address has no logical LAN, or the handle
+ * names no registered queue (already freed)
+ * H_HARDWARE - Hardware error
+ * H_STATE - VIOA not in valid state
+ * H_BUSY / H_LONG_BUSY_* - Resource busy, retry
+ * H_FUNCTION - Firmware does not support this hcall
+ */
+static inline long h_free_logical_lan_queue(unsigned long unit_address,
+ unsigned long queue_handle)
+{
+ return plpar_hcall_norets(H_FREE_LOGICAL_LAN_QUEUE,
+ unit_address, queue_handle);
+}
+
+/**
+ * h_register_logical_lan_with_handle - Register primary queue and get handle
+ * @unit_address: Device unit address
+ * @buffer_list: DMA address of buffer list
+ * @rec_queue: Buffer descriptor of receive queue
+ * @filter_list: DMA address of filter list
+ * @mac_address: MAC address
+ * @queue_handle: Output parameter for queue handle (may be NULL)
+ *
+ * Registers the primary receive queue (queue 0) with the hypervisor and
+ * returns the queue handle. This is needed in multi-queue mode to use
+ * h_add_logical_lan_buffers_queue() for all queues including queue 0.
+ * R4 carries a handle only on firmware that reports MQ RX support in
+ * H_ILLAN_ATTRIBUTES; use h_register_logical_lan() otherwise.
+ *
+ * Return: H_SUCCESS (0) on success, error code otherwise
+ */
+static inline long
+h_register_logical_lan_with_handle(unsigned long unit_address,
+ unsigned long buffer_list,
+ unsigned long rec_queue,
+ unsigned long filter_list,
+ unsigned long mac_address,
+ unsigned long *queue_handle)
+{
+ unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
+ long rc;
+
+ rc = plpar_hcall(H_REGISTER_LOGICAL_LAN, retbuf,
+ unit_address, buffer_list, rec_queue,
+ filter_list, mac_address);
+
+ if (rc == H_SUCCESS && queue_handle)
+ *queue_handle = retbuf[0];
+
+ return rc;
+}
+
/* FW allows us to send 6 descriptors but we only use one so mark
* the other 5 as unused (0)
*/
diff --git a/tools/perf/python/powerpc-hcalls.py b/tools/perf/python/powerpc-hcalls.py
index ab5867c05d3e..f743e5751b60 100755
--- a/tools/perf/python/powerpc-hcalls.py
+++ b/tools/perf/python/powerpc-hcalls.py
@@ -197,6 +197,10 @@ HCALL_TABLE = {
# Key wrapping hcalls
1168: 'H_PKS_WRAP_OBJECT',
1172: 'H_PKS_UNWRAP_OBJECT',
+ # Logical LAN multi-queue RX (PAPR 11.20.00)
+ 1180: 'H_REG_LOGICAL_LAN_QUEUE',
+ 1184: 'H_ADD_LOGICAL_LAN_BUFFERS_QUEUE',
+ 1192: 'H_FREE_LOGICAL_LAN_QUEUE',
# Platform-specific hcalls used by the Ultravisor
61184: 'H_SVM_PAGE_IN',
61188: 'H_SVM_PAGE_OUT',
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 02/15] ibmveth: Prepare MQ RX adapter data structures
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 01/15] ibmveth: Add MQ RX hypercall wrappers and call definitions Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 03/15] ibmveth: Refactor RX resource allocation for MQ RX bring-up Mingming Cao
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
MQ RX needs per-queue state for NAPI, queue handles/IRQs, RX rings,
buffer-list DMA mappings, and buffer pools. The current driver stores
most of this as single instances tied to queue 0.
Convert those fields to queue-indexed layouts sized by
IBMVETH_MAX_RX_QUEUES, adding queue_handle[] / queue_irq[]:
rx_queue[]
napi[]
buffer_list_addr[] / buffer_list_dma[]
rx_buff_pool[queue][pool]
and add multi_queue / num_rx_queues to track MQ capability and how
many RX queues are active. Keep IBMVETH_MAX_RX_QUEUES at 1 for now so
this remains a structural preparation patch; later enablement raises
the limit when multi-queue RX is actually turned on.
This patch keeps behavior unchanged by mechanically switching existing
references to index 0 (for example rx_queue -> rx_queue[0],
rx_buff_pool[pool] -> rx_buff_pool[0][pool], napi -> napi[0]).
open/poll/close still drive a single RX queue only.
First use of the new fields: queue_irq[] and multi_queue in the IRQ
control patch; queue_handle[] in the register-helpers patch, which
captures queue 0's handle from H_REGISTER_LOGICAL_LAN; num_rx_queues
in the RX resource-allocation patch. multi_queue only begins
selecting between code paths once enablement raises
IBMVETH_MAX_RX_QUEUES above 1.
Queue-0 pool sysfs (poolN/) remains the shared geometry template for
all RX queues: later patches clone that metadata per queue. Per-queue
runtime visibility is added via debugfs later, not per-queue sysfs.
Per-queue statistics structs are introduced later with their first use
(statistics collection); the replenish_* counters stay adapter-wide
plain u64 here and move into them at that point.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- poll oversize check uses rx_buff_pool[0][pool], so patches 2-7 build
- rebased onto net-next: the merged synchronize_net() after
napi_disable() and correlator accessors stay on queue 0 until the
queue-aware patches
Changes in v7:
- no P02 code change
Changes in v5:
- Drop unrelated style nits that v4 mixed into the index-0 conversion
(ibmveth_rxq_get_buffer() prototype reflow, get_desired_dma
"Return:" kdoc, blank line after rx_large_packets)
- Document queue-0 poolN/ sysfs as the shared geometry/template for all
RX queues (buff_size/size/active); queues 1..N get no separate pool
sysfs in this series
- Decision: keep full ibmveth_buff_pool per queue for this series
(unused embedded kobjects on rows 1..N); a config-vs-runtime layout
split is out of scope here
Changes in v4:
- Keep IBMVETH_MAX_RX_QUEUES at 1 until MQ enablement (same idea as v3,
but v3 also planted unused stats types here).
- Layout-only: queue-indexed adapter fields only. Do not introduce
the rx/tx qstats here (first-use); they land with the per-queue
statistics patch.
- Subject: "Prepare MQ RX adapter data structures" (was "...and
statistics structures" in earlier drafts).
drivers/net/ethernet/ibm/ibmveth.c | 240 ++++++++++++++++-------------
drivers/net/ethernet/ibm/ibmveth.h | 17 +-
2 files changed, 141 insertions(+), 116 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index e9874dfedf37..ecfa7f737dbb 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -101,7 +101,9 @@ static struct ibmveth_stat ibmveth_stats[] = {
/* simple methods of getting data from the current rxq entry */
static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter)
{
- return be32_to_cpu(adapter->rx_queue.queue_addr[adapter->rx_queue.index].flags_off);
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+
+ return be32_to_cpu(rxq->queue_addr[rxq->index].flags_off);
}
static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
@@ -112,7 +114,7 @@ static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter)
{
- return ibmveth_rxq_toggle(adapter) == adapter->rx_queue.toggle;
+ return ibmveth_rxq_toggle(adapter) == adapter->rx_queue[0].toggle;
}
static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter)
@@ -132,14 +134,17 @@ static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter)
static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter)
{
- return be32_to_cpu(adapter->rx_queue.queue_addr[adapter->rx_queue.index].length);
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+
+ return be32_to_cpu(rxq->queue_addr[rxq->index].length);
}
static u64 ibmveth_rxq_correlator(struct ibmveth_adapter *adapter)
{
- unsigned int idx = adapter->rx_queue.index;
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+ unsigned int idx = rxq->index;
- return READ_ONCE(adapter->rx_queue.queue_addr[idx].correlator);
+ return READ_ONCE(rxq->queue_addr[idx].correlator);
}
static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter)
@@ -393,7 +398,7 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
*/
static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
{
- __be64 *p = adapter->buffer_list_addr + 4096 - 8;
+ __be64 *p = adapter->buffer_list_addr[0] + 4096 - 8;
adapter->rx_no_buffer = be64_to_cpup(p);
}
@@ -406,7 +411,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)
adapter->replenish_task_cycles++;
for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
- struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[i];
+ struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[0][i];
if (pool->active &&
(atomic_read(&pool->available) < pool->threshold))
@@ -459,8 +464,8 @@ static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
/* An inactive pool keeps its size but has no skbuff array. */
if (pool < IBMVETH_NUM_BUFF_POOLS &&
- index < adapter->rx_buff_pool[pool].size &&
- adapter->rx_buff_pool[pool].skbuff)
+ index < adapter->rx_buff_pool[0][pool].size &&
+ adapter->rx_buff_pool[0][pool].skbuff)
return true;
if (net_ratelimit())
@@ -504,7 +509,7 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
if (!ibmveth_rxq_correlator_valid(adapter, correlator))
return -EINVAL;
- skb = adapter->rx_buff_pool[pool].skbuff[index];
+ skb = adapter->rx_buff_pool[0][pool].skbuff[index];
if (!skb) {
ibmveth_rxq_no_skb(adapter, correlator);
return -EFAULT;
@@ -518,24 +523,24 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
/* remove the skb pointer to mark free. actual freeing is done
* by upper level networking after gro_receive
*/
- adapter->rx_buff_pool[pool].skbuff[index] = NULL;
+ adapter->rx_buff_pool[0][pool].skbuff[index] = NULL;
dma_unmap_single(&adapter->vdev->dev,
- adapter->rx_buff_pool[pool].dma_addr[index],
- adapter->rx_buff_pool[pool].buff_size,
+ adapter->rx_buff_pool[0][pool].dma_addr[index],
+ adapter->rx_buff_pool[0][pool].buff_size,
DMA_FROM_DEVICE);
}
- free_index = adapter->rx_buff_pool[pool].producer_index;
- adapter->rx_buff_pool[pool].producer_index++;
- if (adapter->rx_buff_pool[pool].producer_index >=
- adapter->rx_buff_pool[pool].size)
- adapter->rx_buff_pool[pool].producer_index = 0;
- adapter->rx_buff_pool[pool].free_map[free_index] = index;
+ free_index = adapter->rx_buff_pool[0][pool].producer_index;
+ adapter->rx_buff_pool[0][pool].producer_index++;
+ if (adapter->rx_buff_pool[0][pool].producer_index >=
+ adapter->rx_buff_pool[0][pool].size)
+ adapter->rx_buff_pool[0][pool].producer_index = 0;
+ adapter->rx_buff_pool[0][pool].free_map[free_index] = index;
mb();
- atomic_dec(&(adapter->rx_buff_pool[pool].available));
+ atomic_dec(&adapter->rx_buff_pool[0][pool].available);
return 0;
}
@@ -551,7 +556,7 @@ static struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
if (!ibmveth_rxq_correlator_valid(adapter, correlator))
return NULL;
- skb = adapter->rx_buff_pool[pool].skbuff[index];
+ skb = adapter->rx_buff_pool[0][pool].skbuff[index];
if (!skb)
ibmveth_rxq_no_skb(adapter, correlator);
return skb;
@@ -559,9 +564,11 @@ static struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
static void ibmveth_rxq_advance(struct ibmveth_adapter *adapter)
{
- if (++adapter->rx_queue.index == adapter->rx_queue.num_slots) {
- adapter->rx_queue.index = 0;
- adapter->rx_queue.toggle = !adapter->rx_queue.toggle;
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+
+ if (++rxq->index == rxq->num_slots) {
+ rxq->index = 0;
+ rxq->toggle = !rxq->toggle;
}
}
@@ -641,7 +648,7 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
*/
retry:
rc = h_register_logical_lan(adapter->vdev->unit_address,
- adapter->buffer_list_dma, rxq_desc.desc,
+ adapter->buffer_list_dma[0], rxq_desc.desc,
adapter->filter_list_dma, mac_address);
if (rc != H_SUCCESS && try_again) {
@@ -670,11 +677,11 @@ static int ibmveth_open(struct net_device *netdev)
netdev_dbg(netdev, "open starting\n");
for(i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- rxq_entries += adapter->rx_buff_pool[i].size;
+ rxq_entries += adapter->rx_buff_pool[0][i].size;
rc = -ENOMEM;
- adapter->buffer_list_addr = (void*) get_zeroed_page(GFP_KERNEL);
- if (!adapter->buffer_list_addr) {
+ adapter->buffer_list_addr[0] = (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->buffer_list_addr[0]) {
netdev_err(netdev, "unable to allocate list pages\n");
goto out;
}
@@ -687,17 +694,18 @@ static int ibmveth_open(struct net_device *netdev)
dev = &adapter->vdev->dev;
- adapter->rx_queue.queue_len = sizeof(struct ibmveth_rx_q_entry) *
+ adapter->rx_queue[0].queue_len = sizeof(struct ibmveth_rx_q_entry) *
rxq_entries;
- adapter->rx_queue.queue_addr =
- dma_alloc_coherent(dev, adapter->rx_queue.queue_len,
- &adapter->rx_queue.queue_dma, GFP_KERNEL);
- if (!adapter->rx_queue.queue_addr)
+ adapter->rx_queue[0].queue_addr =
+ dma_alloc_coherent(dev, adapter->rx_queue[0].queue_len,
+ &adapter->rx_queue[0].queue_dma, GFP_KERNEL);
+ if (!adapter->rx_queue[0].queue_addr)
goto out_free_filter_list;
- adapter->buffer_list_dma = dma_map_single(dev,
- adapter->buffer_list_addr, 4096, DMA_BIDIRECTIONAL);
- if (dma_mapping_error(dev, adapter->buffer_list_dma)) {
+ adapter->buffer_list_dma[0] =
+ dma_map_single(dev, adapter->buffer_list_addr[0],
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->buffer_list_dma[0])) {
netdev_err(netdev, "unable to map buffer list pages\n");
goto out_free_queue_mem;
}
@@ -714,19 +722,21 @@ static int ibmveth_open(struct net_device *netdev)
goto out_unmap_filter_list;
}
- adapter->rx_queue.index = 0;
- adapter->rx_queue.num_slots = rxq_entries;
- adapter->rx_queue.toggle = 1;
+ adapter->rx_queue[0].index = 0;
+ adapter->rx_queue[0].num_slots = rxq_entries;
+ adapter->rx_queue[0].toggle = 1;
mac_address = ether_addr_to_u64(netdev->dev_addr);
rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
- adapter->rx_queue.queue_len;
- rxq_desc.fields.address = adapter->rx_queue.queue_dma;
+ adapter->rx_queue[0].queue_len;
+ rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
- netdev_dbg(netdev, "buffer list @ 0x%p\n", adapter->buffer_list_addr);
+ netdev_dbg(netdev, "buffer list @ 0x%p\n",
+ adapter->buffer_list_addr[0]);
netdev_dbg(netdev, "filter list @ 0x%p\n", adapter->filter_list_addr);
- netdev_dbg(netdev, "receive q @ 0x%p\n", adapter->rx_queue.queue_addr);
+ netdev_dbg(netdev, "receive q @ 0x%p\n",
+ adapter->rx_queue[0].queue_addr);
h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
@@ -737,7 +747,7 @@ static int ibmveth_open(struct net_device *netdev)
lpar_rc);
netdev_err(netdev, "buffer TCE:0x%llx filter TCE:0x%llx rxq "
"desc:0x%llx MAC:0x%llx\n",
- adapter->buffer_list_dma,
+ adapter->buffer_list_dma[0],
adapter->filter_list_dma,
rxq_desc.desc,
mac_address);
@@ -746,11 +756,11 @@ static int ibmveth_open(struct net_device *netdev)
}
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- if (!adapter->rx_buff_pool[i].active)
+ if (!adapter->rx_buff_pool[0][i].active)
continue;
- if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[i])) {
+ if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[0][i])) {
netdev_err(netdev, "unable to alloc pool\n");
- adapter->rx_buff_pool[i].active = 0;
+ adapter->rx_buff_pool[0][i].active = 0;
rc = -ENOMEM;
goto out_free_buffer_pools;
}
@@ -761,13 +771,13 @@ static int ibmveth_open(struct net_device *netdev)
* call, so enable it only once everything ibmveth_poll() touches
* exists.
*/
- napi_enable(&adapter->napi);
+ napi_enable(&adapter->napi[0]);
netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
netdev);
if (rc != 0) {
- napi_disable(&adapter->napi);
+ napi_disable(&adapter->napi[0]);
synchronize_net();
netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
netdev->irq, rc);
@@ -791,9 +801,9 @@ static int ibmveth_open(struct net_device *netdev)
lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
while (--i >= 0) {
- if (adapter->rx_buff_pool[i].active)
+ if (adapter->rx_buff_pool[0][i].active)
ibmveth_free_buffer_pool(adapter,
- &adapter->rx_buff_pool[i]);
+ &adapter->rx_buff_pool[0][i]);
}
out_unmap_filter_list:
dma_unmap_single(dev, adapter->filter_list_dma, 4096,
@@ -803,16 +813,16 @@ static int ibmveth_open(struct net_device *netdev)
ibmveth_free_tx_ltb(adapter, i);
out_unmap_buffer_list:
- dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
+ dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
DMA_BIDIRECTIONAL);
out_free_queue_mem:
- dma_free_coherent(dev, adapter->rx_queue.queue_len,
- adapter->rx_queue.queue_addr,
- adapter->rx_queue.queue_dma);
+ dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
+ adapter->rx_queue[0].queue_addr,
+ adapter->rx_queue[0].queue_dma);
out_free_filter_list:
free_page((unsigned long)adapter->filter_list_addr);
out_free_buffer_list:
- free_page((unsigned long)adapter->buffer_list_addr);
+ free_page((unsigned long)adapter->buffer_list_addr[0]);
out:
return rc;
}
@@ -834,7 +844,7 @@ static int ibmveth_close(struct net_device *netdev)
netdev_dbg(netdev, "close starting\n");
- napi_disable(&adapter->napi);
+ napi_disable(&adapter->napi[0]);
/* napi_disable() returns once ibmveth_poll() has called
* napi_complete_done(), but the poll still re-enables the
* interrupt and reads the RX queue after that.
@@ -858,22 +868,22 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_update_rx_no_buffer(adapter);
- dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
+ dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
DMA_BIDIRECTIONAL);
- free_page((unsigned long)adapter->buffer_list_addr);
+ free_page((unsigned long)adapter->buffer_list_addr[0]);
dma_unmap_single(dev, adapter->filter_list_dma, 4096,
DMA_BIDIRECTIONAL);
free_page((unsigned long)adapter->filter_list_addr);
- dma_free_coherent(dev, adapter->rx_queue.queue_len,
- adapter->rx_queue.queue_addr,
- adapter->rx_queue.queue_dma);
+ dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
+ adapter->rx_queue[0].queue_addr,
+ adapter->rx_queue[0].queue_dma);
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- if (adapter->rx_buff_pool[i].active)
+ if (adapter->rx_buff_pool[0][i].active)
ibmveth_free_buffer_pool(adapter,
- &adapter->rx_buff_pool[i]);
+ &adapter->rx_buff_pool[0][i]);
for (i = 0; i < netdev->real_num_tx_queues; i++)
ibmveth_free_tx_ltb(adapter, i);
@@ -1554,7 +1564,7 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
static int ibmveth_poll(struct napi_struct *napi, int budget)
{
struct ibmveth_adapter *adapter =
- container_of(napi, struct ibmveth_adapter, napi);
+ container_of(napi, struct ibmveth_adapter, napi[0]);
struct net_device *netdev = adapter->netdev;
int frames_processed = 0;
unsigned long lpar_rc;
@@ -1597,7 +1607,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
pool = correlator >> 32;
room = min_t(unsigned int, skb_tailroom(skb),
- adapter->rx_buff_pool[pool].buff_size);
+ adapter->rx_buff_pool[0][pool].buff_size);
off = offset;
len = length;
if (unlikely(len < ETH_HLEN || off >= room ||
@@ -1730,11 +1740,11 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
struct ibmveth_adapter *adapter = netdev_priv(netdev);
unsigned long lpar_rc;
- if (napi_schedule_prep(&adapter->napi)) {
+ if (napi_schedule_prep(&adapter->napi[0])) {
lpar_rc = h_vio_signal(adapter->vdev->unit_address,
VIO_IRQ_DISABLE);
WARN_ON(lpar_rc != H_SUCCESS);
- __napi_schedule(&adapter->napi);
+ __napi_schedule(&adapter->napi[0]);
}
return IRQ_HANDLED;
}
@@ -1802,7 +1812,7 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
int need_restart = 0;
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- if (new_mtu_oh <= adapter->rx_buff_pool[i].buff_size)
+ if (new_mtu_oh <= adapter->rx_buff_pool[0][i].buff_size)
break;
if (i == IBMVETH_NUM_BUFF_POOLS)
@@ -1817,9 +1827,9 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
/* Look for an active buffer pool that can hold the new MTU */
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- adapter->rx_buff_pool[i].active = 1;
+ adapter->rx_buff_pool[0][i].active = 1;
- if (new_mtu_oh <= adapter->rx_buff_pool[i].buff_size) {
+ if (new_mtu_oh <= adapter->rx_buff_pool[0][i].buff_size) {
WRITE_ONCE(dev->mtu, new_mtu);
vio_cmo_set_dev_desired(viodev,
ibmveth_get_desired_dma
@@ -1869,12 +1879,12 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
/* add the size of the active receive buffers */
- if (adapter->rx_buff_pool[i].active)
+ if (adapter->rx_buff_pool[0][i].active)
ret +=
- adapter->rx_buff_pool[i].size *
- IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[i].
+ adapter->rx_buff_pool[0][i].size *
+ IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[0][i].
buff_size, tbl);
- rxqentries += adapter->rx_buff_pool[i].size;
+ rxqentries += adapter->rx_buff_pool[0][i].size;
}
/* add the size of the receive queue entries */
ret += IOMMU_PAGE_ALIGN(
@@ -1969,9 +1979,9 @@ static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter)
int i;
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- kobject_put(&adapter->rx_buff_pool[i].kobj);
+ kobject_put(&adapter->rx_buff_pool[0][i].kobj);
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- wait_for_completion(&adapter->rx_buff_pool[i].released);
+ wait_for_completion(&adapter->rx_buff_pool[0][i].released);
}
static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
@@ -2024,7 +2034,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
ibmveth_init_link_settings(netdev);
- netif_napi_add_weight(netdev, &adapter->napi, ibmveth_poll, 16);
+ netif_napi_add_weight(netdev, &adapter->napi[0], ibmveth_poll, 16);
netdev->irq = dev->irq;
netdev->netdev_ops = &ibmveth_netdev_ops;
@@ -2056,6 +2066,10 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
netdev->features |= NETIF_F_FRAGLIST;
}
+ /* Initialize queue count - always 1 for now */
+ adapter->multi_queue = 0;
+ adapter->num_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;
+
if (ret == H_SUCCESS &&
(ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_PER_HCALL;
@@ -2078,13 +2092,13 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
memcpy(pool_count, pool_count_cmo, sizeof(pool_count));
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- struct kobject *kobj = &adapter->rx_buff_pool[i].kobj;
+ struct kobject *kobj = &adapter->rx_buff_pool[0][i].kobj;
int error;
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- init_completion(&adapter->rx_buff_pool[i].released);
+ init_completion(&adapter->rx_buff_pool[0][i].released);
error = kobject_init_and_add(kobj, &ktype_veth_pool,
&dev->dev.kobj, "pool%d", i);
if (!error)
@@ -2217,11 +2231,12 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
/* Make sure there is a buffer pool with buffers that
can hold a packet of the size of the MTU */
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- if (pool == &adapter->rx_buff_pool[i])
+ if (pool == &adapter->rx_buff_pool[0][i])
continue;
- if (!adapter->rx_buff_pool[i].active)
+ if (!adapter->rx_buff_pool[0][i].active)
continue;
- if (mtu <= adapter->rx_buff_pool[i].buff_size)
+ if (mtu <=
+ adapter->rx_buff_pool[0][i].buff_size)
break;
}
@@ -2395,11 +2410,11 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
/* Set sane values for buffer pools */
for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- pool = &adapter->rx_buff_pool[0];
+ pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
@@ -2407,7 +2422,7 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
- correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0].size;
+ correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0][0].size;
KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
@@ -2447,40 +2462,43 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
INIT_WORK(&adapter->work, ibmveth_reset_kunit);
- adapter->rx_queue.queue_len = 1;
- adapter->rx_queue.index = 0;
- adapter->rx_queue.queue_addr = kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),
- GFP_KERNEL);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue.queue_addr);
+ adapter->rx_queue[0].queue_len = 1;
+ adapter->rx_queue[0].index = 0;
+ adapter->rx_queue[0].queue_addr =
+ kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue[0].queue_addr);
/* Set sane values for buffer pools */
for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- pool = &adapter->rx_buff_pool[0];
+ pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
- adapter->rx_queue.queue_addr[0].correlator = (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
+ adapter->rx_queue[0].queue_addr[0].correlator =
+ (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
cor = ibmveth_rxq_correlator(adapter);
KUNIT_EXPECT_PTR_EQ(test, NULL,
ibmveth_rxq_get_buffer(adapter, cor));
- adapter->rx_queue.queue_addr[0].correlator = (u64)0 << 32 | adapter->rx_buff_pool[0].size;
+ adapter->rx_queue[0].queue_addr[0].correlator =
+ (u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
cor = ibmveth_rxq_correlator(adapter);
KUNIT_EXPECT_PTR_EQ(test, NULL,
ibmveth_rxq_get_buffer(adapter, cor));
/* Pool 2 is in range but has no skbuff array, like an inactive pool. */
- adapter->rx_queue.queue_addr[0].correlator = (u64)2 << 32 | 0;
+ adapter->rx_queue[0].queue_addr[0].correlator = (u64)2 << 32 | 0;
cor = ibmveth_rxq_correlator(adapter);
KUNIT_EXPECT_PTR_EQ(test, NULL,
ibmveth_rxq_get_buffer(adapter, cor));
pool->skbuff[0] = skb;
- adapter->rx_queue.queue_addr[0].correlator = (u64)0 << 32 | 0;
+ adapter->rx_queue[0].queue_addr[0].correlator = (u64)0 << 32 | 0;
cor = ibmveth_rxq_correlator(adapter);
KUNIT_EXPECT_PTR_EQ(test, skb,
ibmveth_rxq_get_buffer(adapter, cor));
@@ -2510,41 +2528,41 @@ static void ibmveth_rxq_harvest_buffer_test(struct kunit *test)
INIT_WORK(&adapter->work, ibmveth_reset_kunit);
- adapter->rx_queue.num_slots = 2;
- adapter->rx_queue.index = 0;
- adapter->rx_queue.toggle = 1;
- adapter->rx_queue.queue_addr =
+ adapter->rx_queue[0].num_slots = 2;
+ adapter->rx_queue[0].index = 0;
+ adapter->rx_queue[0].toggle = 1;
+ adapter->rx_queue[0].queue_addr =
kunit_kcalloc(test, 2, sizeof(struct ibmveth_rx_q_entry),
GFP_KERNEL);
- KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue.queue_addr);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue[0].queue_addr);
/* Set sane values for buffer pools */
for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+ ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
- pool = &adapter->rx_buff_pool[0];
+ pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *),
GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
/* Slot 0: pool out of range. Slot 1: valid, but no skb. */
- adapter->rx_queue.queue_addr[0].correlator =
+ adapter->rx_queue[0].queue_addr[0].correlator =
(u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
- adapter->rx_queue.queue_addr[1].correlator = (u64)0 << 32 | 0;
+ adapter->rx_queue[0].queue_addr[1].correlator = (u64)0 << 32 | 0;
cor = ibmveth_rxq_correlator(adapter);
ret = ibmveth_rxq_harvest_buffer(adapter, cor, true);
KUNIT_EXPECT_EQ(test, -EINVAL, ret);
- KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue.index);
- KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue.toggle);
+ KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue[0].index);
+ KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue[0].toggle);
cor = ibmveth_rxq_correlator(adapter);
ret = ibmveth_rxq_harvest_buffer(adapter, cor, true);
KUNIT_EXPECT_EQ(test, -EFAULT, ret);
- KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue.index);
- KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue.toggle);
+ KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue[0].index);
+ KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue[0].toggle);
flush_work(&adapter->work);
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 82f22de16394..df5a3b69a157 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -269,6 +269,8 @@ static inline long h_illan_attributes(unsigned long unit_address,
#define IBMVETH_MAX_TX_BUF_SIZE (1024 * 64)
#define IBMVETH_MAX_QUEUES 16U
#define IBMVETH_DEFAULT_QUEUES 8U
+#define IBMVETH_MAX_RX_QUEUES 1U
+#define IBMVETH_DEFAULT_RX_QUEUES 1U
#define IBMVETH_MAX_RX_PER_HCALL 8U
static int pool_size[] = { 512, 1024 * 2, 1024 * 16, 1024 * 32, 1024 * 64 };
@@ -306,18 +308,23 @@ struct ibmveth_rx_q {
struct ibmveth_adapter {
struct vio_dev *vdev;
struct net_device *netdev;
- struct napi_struct napi;
+ struct napi_struct napi[IBMVETH_MAX_RX_QUEUES];
struct work_struct work;
unsigned int mcastFilterSize;
- void *buffer_list_addr;
+ void *buffer_list_addr[IBMVETH_MAX_RX_QUEUES];
void *filter_list_addr;
void *tx_ltb_ptr[IBMVETH_MAX_QUEUES];
unsigned int tx_ltb_size;
dma_addr_t tx_ltb_dma[IBMVETH_MAX_QUEUES];
- dma_addr_t buffer_list_dma;
+ dma_addr_t buffer_list_dma[IBMVETH_MAX_RX_QUEUES];
dma_addr_t filter_list_dma;
- struct ibmveth_buff_pool rx_buff_pool[IBMVETH_NUM_BUFF_POOLS];
- struct ibmveth_rx_q rx_queue;
+ struct ibmveth_buff_pool
+ rx_buff_pool[IBMVETH_MAX_RX_QUEUES][IBMVETH_NUM_BUFF_POOLS];
+ struct ibmveth_rx_q rx_queue[IBMVETH_MAX_RX_QUEUES];
+ u64 queue_handle[IBMVETH_MAX_RX_QUEUES];
+ unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
+ int multi_queue;
+ unsigned int num_rx_queues;
int rx_csum;
int large_send;
bool is_active_trunk;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 03/15] ibmveth: Refactor RX resource allocation for MQ RX bring-up
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 01/15] ibmveth: Add MQ RX hypercall wrappers and call definitions Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 02/15] ibmveth: Prepare MQ RX adapter data structures Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 04/15] ibmveth: Refactor buffer pool management for per-queue MQ RX Mingming Cao
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
ibmveth_open() allocates the filter list and every RX queue inline.
That is already a long sequence and would get uglier once we loop over
num_rx_queues, especially on error unwind.
Pull the RX bits into helpers and wire them into open()/close() in the
same patch:
ibmveth_alloc_filter_list() / ibmveth_free_filter_list()
- shared multicast filter list (one per adapter, not per queue)
ibmveth_alloc_rx_queues() / ibmveth_cleanup_rx_resources()
- per-queue buffer lists and RX rings, looping [0, num_rx_queues)
alloc_rx_queues() rolls back on failure so open() does not need nested
goto chains for every queue index. open-failure and close release the
same resources through the same helpers.
Cleanup NULLs each slot as it frees. A filter-list map error clears
both filter_list_addr and filter_list_dma, so no stale handle stays
in the adapter; unmap is gated on the CPU page because SPAPR can
return DMA address 0. ibmveth_update_rx_no_buffer() returns when
the buffer list is not allocated yet.
Runtime behavior stays single-queue (num_rx_queues is still 1). Buffer
pools, IRQ, TX LTB, and PHYP registration remain inline for later
helper patches.
Also set rc = -ENOMEM before the TX LTB allocation loop so a failed
ibmveth_allocate_tx_ltb() still returns a useful errno after the RX
allocation blocks move into helpers, and drop the dead rc = -ENOMEM
after request_irq().
open() now allocates the filter list first and initialises the RX
ring before the TX LTB loop. The list-page error names the queue, a
failed RX ring allocation gets its own error, and the per-buffer
netdev_dbg() lines are replaced by per-helper ones.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- rebased onto net-next, which carries the open-fail pair and the nine
merged fixes; the TX unwind keeps net-next's real_num_tx_queues loop
- commit message lists the open() order and log string changes, and
the dead rc after request_irq() this patch drops
This patch adds the filter-list DMA sentinel (filter_list_dma = 0
on map error) and the update_rx_no_buffer() NULL guard on
buffer_list_addr[0]; those are not in the merged fixes.
Changes in v7:
- NULL-check buffer_list_addr[0] in update_rx_no_buffer()
- reword err_cleanup comment for the mixed map-failure state
- drop the failed-reopen then ndo_stop double-free claim from the body
- noted: no Fixes: peel of the DMA-handle checks; standalone after
this series
- noted: TX LTB stale handle predates this; P06 (pointer check, zero dma)
- noted: shared-i TX LTB leak predates this; P06 (own counter, TX
after pools)
- noted: pool-fail without h_free predates this; P06 adds it, P07
allocates pools first
- noted: napi imbalance on failed reopen predates this; P05
- noted: replenish_lock stays in P08; poll_controller !opened
stays in P15; leftover netpoll/irqsave after this series
Changes in v6:
- on RX-queue dma_alloc or map failure, free the buffer-list page
before cleanup so the addr-based unmap does not dma_unmap an
address that was never mapped
- unmap RX filter/buffer lists by CPU pointer, not dma_addr==0
- use %u for the two num_rx_queues dbg prints
- noted: NULL buffer_list_addr[0] window: close() in P05, netpoll
through replenish_task() until the per-queue guard in P10
- noted: shared-i TX LTB leak on pool-fail predates this; P04/P06
- noted: pool-fail after register without h_free predates this; P06/P07
Changes in v5:
- On filter_list DMA map failure: free_page and zero filter_list_dma so
a later free path cannot dma_unmap the DMA_MAPPING_ERROR sentinel
(match buffer_list_dma convention). Closes reopen->ifdown WARN path
after set_csum/set_tso/change_mtu/pool_store close+open while running
- v4 left the sentinel
- Call out rc = -ENOMEM before the TX LTB loop after RX helper extract
(already in v4; still required so allocate_tx_ltb failure returns a
useful errno)
Changes in v4:
- Introduce RX/filter allocation helpers in the same patch that wires
their first open/close callers; v3 left unused statics ahead of the
old open/close pipeline patch.
- Preserve correct -ENOMEM return on TX LTB allocation failure after
the RX helper extract.
- Drop reliance on v3's separate "open/close pipeline" patch for this
wiring.
drivers/net/ethernet/ibm/ibmveth.c | 284 +++++++++++++++++++++--------
1 file changed, 206 insertions(+), 78 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index ecfa7f737dbb..68791402cd53 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -159,6 +159,194 @@ static unsigned int ibmveth_real_max_tx_queues(void)
return min(n_cpu, IBMVETH_MAX_QUEUES);
}
+/**
+ * ibmveth_alloc_filter_list - Allocate and map filter list
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_filter_list(struct ibmveth_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
+
+ adapter->filter_list_addr = (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->filter_list_addr) {
+ netdev_err(netdev, "unable to allocate filter pages\n");
+ return -ENOMEM;
+ }
+
+ adapter->filter_list_dma = dma_map_single(dev,
+ adapter->filter_list_addr,
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->filter_list_dma)) {
+ netdev_err(netdev, "unable to map filter list pages\n");
+ free_page((unsigned long)adapter->filter_list_addr);
+ adapter->filter_list_addr = NULL;
+ /* Do not leave DMA_MAPPING_ERROR in the adapter. */
+ adapter->filter_list_dma = 0;
+ return -ENOMEM;
+ }
+
+ netdev_dbg(netdev, "filter list @ 0x%p (DMA: 0x%llx)\n",
+ adapter->filter_list_addr,
+ (unsigned long long)adapter->filter_list_dma);
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_filter_list - Free filter list resources
+ * @adapter: ibmveth adapter structure
+ */
+static void
+ibmveth_free_filter_list(struct ibmveth_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+
+ /* Unmap by CPU pointer: SPAPR can return DMA address 0. */
+ if (adapter->filter_list_addr) {
+ dma_unmap_single(dev, adapter->filter_list_dma, 4096,
+ DMA_BIDIRECTIONAL);
+ adapter->filter_list_dma = 0;
+ free_page((unsigned long)adapter->filter_list_addr);
+ adapter->filter_list_addr = NULL;
+ }
+}
+
+/**
+ * ibmveth_alloc_rx_queues - Allocate per-queue RX resources
+ * @adapter: ibmveth adapter structure
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ adapter->buffer_list_addr[i] =
+ (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->buffer_list_addr[i]) {
+ netdev_err(netdev,
+ "unable to allocate buffer list for queue %d\n",
+ i);
+ goto err_cleanup;
+ }
+
+ adapter->rx_queue[i].queue_len =
+ sizeof(struct ibmveth_rx_q_entry) * rxq_entries;
+ adapter->rx_queue[i].queue_addr =
+ dma_alloc_coherent(dev, adapter->rx_queue[i].queue_len,
+ &adapter->rx_queue[i].queue_dma,
+ GFP_KERNEL);
+ if (!adapter->rx_queue[i].queue_addr) {
+ netdev_err(netdev,
+ "unable to allocate RX queue for queue %d\n",
+ i);
+ free_page((unsigned long)adapter->buffer_list_addr[i]);
+ adapter->buffer_list_addr[i] = NULL;
+ goto err_cleanup;
+ }
+
+ adapter->buffer_list_dma[i] =
+ dma_map_single(dev, adapter->buffer_list_addr[i],
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->buffer_list_dma[i])) {
+ netdev_err(netdev,
+ "unable to map buffer list for queue %d\n",
+ i);
+ free_page((unsigned long)adapter->buffer_list_addr[i]);
+ adapter->buffer_list_addr[i] = NULL;
+ adapter->buffer_list_dma[i] = 0;
+ goto err_cleanup;
+ }
+
+ adapter->rx_queue[i].index = 0;
+ adapter->rx_queue[i].num_slots = rxq_entries;
+ adapter->rx_queue[i].toggle = 1;
+
+ netdev_dbg(netdev, "queue %d: buffer_list @ 0x%p (DMA: 0x%llx), rx_queue @ 0x%p (DMA: 0x%llx), %llu entries\n",
+ i, adapter->buffer_list_addr[i],
+ (unsigned long long)adapter->buffer_list_dma[i],
+ adapter->rx_queue[i].queue_addr,
+ (unsigned long long)adapter->rx_queue[i].queue_dma,
+ (unsigned long long)rxq_entries);
+ }
+
+ netdev_dbg(netdev, "allocated %u RX queue(s) with %d entries each\n",
+ adapter->num_rx_queues, rxq_entries);
+
+ return 0;
+
+err_cleanup:
+ /*
+ * Free each leftover resource by pointer presence. A
+ * dma_mapping_error on queue i has already dropped
+ * buffer_list_addr[i] but can leave rx_queue[i].queue_addr
+ * live, so this index may be mixed. Do not collapse the
+ * buffer_list and queue_addr checks.
+ */
+ for (; i >= 0; i--) {
+ if (adapter->buffer_list_addr[i]) {
+ dma_unmap_single(dev, adapter->buffer_list_dma[i],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[i] = 0;
+ }
+ if (adapter->rx_queue[i].queue_addr) {
+ dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
+ adapter->rx_queue[i].queue_addr,
+ adapter->rx_queue[i].queue_dma);
+ adapter->rx_queue[i].queue_addr = NULL;
+ }
+ if (adapter->buffer_list_addr[i]) {
+ free_page((unsigned long)adapter->buffer_list_addr[i]);
+ adapter->buffer_list_addr[i] = NULL;
+ }
+ }
+
+ return -ENOMEM;
+}
+
+/**
+ * ibmveth_cleanup_rx_resources - Free all RX queue resources
+ * @adapter: ibmveth adapter structure
+ */
+static void
+ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ int i;
+
+ netdev_dbg(adapter->netdev, "cleaning up %u RX queue(s)\n",
+ adapter->num_rx_queues);
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (adapter->buffer_list_addr[i]) {
+ dma_unmap_single(dev, adapter->buffer_list_dma[i],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[i] = 0;
+ }
+
+ if (adapter->rx_queue[i].queue_addr) {
+ dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
+ adapter->rx_queue[i].queue_addr,
+ adapter->rx_queue[i].queue_dma);
+ adapter->rx_queue[i].queue_addr = NULL;
+ }
+
+ if (adapter->buffer_list_addr[i]) {
+ free_page((unsigned long)adapter->buffer_list_addr[i]);
+ adapter->buffer_list_addr[i] = NULL;
+ }
+ }
+}
+
/* setup the initial settings for a buffer pool */
static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,
u32 pool_index, u32 pool_size,
@@ -398,8 +586,12 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
*/
static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
{
- __be64 *p = adapter->buffer_list_addr[0] + 4096 - 8;
+ __be64 *p;
+
+ if (!adapter->buffer_list_addr[0])
+ return;
+ p = adapter->buffer_list_addr[0] + 4096 - 8;
adapter->rx_no_buffer = be64_to_cpup(p);
}
@@ -672,72 +864,32 @@ static int ibmveth_open(struct net_device *netdev)
int rc;
union ibmveth_buf_desc rxq_desc;
int i;
- struct device *dev;
netdev_dbg(netdev, "open starting\n");
- for(i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
rxq_entries += adapter->rx_buff_pool[0][i].size;
- rc = -ENOMEM;
- adapter->buffer_list_addr[0] = (void *)get_zeroed_page(GFP_KERNEL);
- if (!adapter->buffer_list_addr[0]) {
- netdev_err(netdev, "unable to allocate list pages\n");
+ rc = ibmveth_alloc_filter_list(adapter);
+ if (rc)
goto out;
- }
- adapter->filter_list_addr = (void*) get_zeroed_page(GFP_KERNEL);
- if (!adapter->filter_list_addr) {
- netdev_err(netdev, "unable to allocate filter pages\n");
- goto out_free_buffer_list;
- }
-
- dev = &adapter->vdev->dev;
-
- adapter->rx_queue[0].queue_len = sizeof(struct ibmveth_rx_q_entry) *
- rxq_entries;
- adapter->rx_queue[0].queue_addr =
- dma_alloc_coherent(dev, adapter->rx_queue[0].queue_len,
- &adapter->rx_queue[0].queue_dma, GFP_KERNEL);
- if (!adapter->rx_queue[0].queue_addr)
+ rc = ibmveth_alloc_rx_queues(adapter, rxq_entries);
+ if (rc)
goto out_free_filter_list;
- adapter->buffer_list_dma[0] =
- dma_map_single(dev, adapter->buffer_list_addr[0],
- 4096, DMA_BIDIRECTIONAL);
- if (dma_mapping_error(dev, adapter->buffer_list_dma[0])) {
- netdev_err(netdev, "unable to map buffer list pages\n");
- goto out_free_queue_mem;
- }
-
- adapter->filter_list_dma = dma_map_single(dev,
- adapter->filter_list_addr, 4096, DMA_BIDIRECTIONAL);
- if (dma_mapping_error(dev, adapter->filter_list_dma)) {
- netdev_err(netdev, "unable to map filter list pages\n");
- goto out_unmap_buffer_list;
- }
-
+ rc = -ENOMEM;
for (i = 0; i < netdev->real_num_tx_queues; i++) {
if (ibmveth_allocate_tx_ltb(adapter, i))
- goto out_unmap_filter_list;
+ goto out_free_tx_ltb;
}
- adapter->rx_queue[0].index = 0;
- adapter->rx_queue[0].num_slots = rxq_entries;
- adapter->rx_queue[0].toggle = 1;
-
mac_address = ether_addr_to_u64(netdev->dev_addr);
rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
adapter->rx_queue[0].queue_len;
rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
- netdev_dbg(netdev, "buffer list @ 0x%p\n",
- adapter->buffer_list_addr[0]);
- netdev_dbg(netdev, "filter list @ 0x%p\n", adapter->filter_list_addr);
- netdev_dbg(netdev, "receive q @ 0x%p\n",
- adapter->rx_queue[0].queue_addr);
-
h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
@@ -752,7 +904,7 @@ static int ibmveth_open(struct net_device *netdev)
rxq_desc.desc,
mac_address);
rc = -ENONET;
- goto out_unmap_filter_list;
+ goto out_free_tx_ltb;
}
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
@@ -784,8 +936,6 @@ static int ibmveth_open(struct net_device *netdev)
goto out_free_buffer_pools;
}
- rc = -ENOMEM;
-
netdev_dbg(netdev, "initial replenish cycle\n");
ibmveth_interrupt(netdev->irq, netdev);
@@ -805,24 +955,12 @@ static int ibmveth_open(struct net_device *netdev)
ibmveth_free_buffer_pool(adapter,
&adapter->rx_buff_pool[0][i]);
}
-out_unmap_filter_list:
- dma_unmap_single(dev, adapter->filter_list_dma, 4096,
- DMA_BIDIRECTIONAL);
-
+out_free_tx_ltb:
for (i = netdev->real_num_tx_queues - 1; i >= 0; i--)
ibmveth_free_tx_ltb(adapter, i);
-
-out_unmap_buffer_list:
- dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
- DMA_BIDIRECTIONAL);
-out_free_queue_mem:
- dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
- adapter->rx_queue[0].queue_addr,
- adapter->rx_queue[0].queue_dma);
+ ibmveth_cleanup_rx_resources(adapter);
out_free_filter_list:
- free_page((unsigned long)adapter->filter_list_addr);
-out_free_buffer_list:
- free_page((unsigned long)adapter->buffer_list_addr[0]);
+ ibmveth_free_filter_list(adapter);
out:
return rc;
}
@@ -830,7 +968,6 @@ static int ibmveth_open(struct net_device *netdev)
static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- struct device *dev = &adapter->vdev->dev;
long lpar_rc;
int i;
@@ -868,17 +1005,8 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_update_rx_no_buffer(adapter);
- dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
- DMA_BIDIRECTIONAL);
- free_page((unsigned long)adapter->buffer_list_addr[0]);
-
- dma_unmap_single(dev, adapter->filter_list_dma, 4096,
- DMA_BIDIRECTIONAL);
- free_page((unsigned long)adapter->filter_list_addr);
-
- dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
- adapter->rx_queue[0].queue_addr,
- adapter->rx_queue[0].queue_dma);
+ ibmveth_cleanup_rx_resources(adapter);
+ ibmveth_free_filter_list(adapter);
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
if (adapter->rx_buff_pool[0][i].active)
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 04/15] ibmveth: Refactor buffer pool management for per-queue MQ RX
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (2 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 03/15] ibmveth: Refactor RX resource allocation for MQ RX bring-up Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 05/15] ibmveth: Refactor RX interrupt control for MQ RX queues Mingming Cao
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
Legacy ibmveth uses five adapter-level RX buffer pools (512 B through
64 KiB). pool_active[] enables the standard-MTU pools by default;
larger pools activate when MTU requires them. With single-queue RX
that set is shared on one completion path.
MQ requires the same pool model per queue: buffers post with
H_ADD_LOGICAL_LAN_BUFFERS_QUEUE against a queue handle and completions
return on that queue. Sharing pools across queues would mix ownership
and break queue-local replenish/drain/teardown.
Refactor around queue-local pools:
rx_buff_pool[queue][pool]
ibmveth_alloc_queue_buffer_pools()
ibmveth_free_queue_buffer_pools()
ibmveth_alloc_buffer_pools() / ibmveth_free_buffer_pools()
Queue 0 remains the template for pool geometry and activation policy
(size, buff_size, threshold, index, active). Queues 1..N copy that
metadata from queue 0, then allocate backing arrays/skbs per queue.
The existing poolN/ sysfs knobs stay adapter-wide on queue 0: writing
a pool size multiplies real memory by num_rx_queues (still 1 here).
Per-queue runtime state is exposed later via debugfs, not separate
per-queue pool sysfs nodes.
Wire the helpers into open()/close() in the same patch. Runtime
remains single-queue (num_rx_queues is still 1).
A pool allocation failure still unwinds through out_free_buffer_pools,
so the logical LAN registered just before is freed with
h_free_logical_lan() as on the other post-register failures.
Error handling is queue-safe: allocation failure unwinds only what
that queue allocated (then prior queues in the caller).
alloc_buffer_pool() already undoes its own partials, so this open-fail
slot is empty. Free paths release by real allocations
(free_map/dma_addr/skbuff) rather than pool->active, so a pool is
freed whether or not its active flag still matches its memory.
close() also reorders pool teardown ahead of cleanup_rx_resources()
and filter-list free. napi_disable() and free_irq() have already run,
so NAPI no longer references the pool buffers or RX completion queue,
and PHYP does not once h_free_logical_lan() has succeeded. A failed
free is logged and the memory is still released, as close() did
before this patch; the order of the frees does not change that.
Keep the legacy 64 KiB pool enabled by default at standard MTU (same
as single-queue policy).
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- a pool allocation failure in open() unwinds through
h_free_logical_lan(), as the merged open-fail fix does
Dropped from this series (merged fixes own them):
- commit 84bec0bf0352: commit-message shared-TX-index unwind paragraph
(base owns the unwind)
Changes in v7:
- no P04 code change
- commit message: request_irq() failure also reached
out_free_tx_ltb with i spent
- noted: err_cleanup comment reworded in P03
- noted: no Fixes: peel of the shared-i TX LTB leak (named
in the v6 body after v5); standalone [PATCH net] after
this series
- noted: no pool-before-register reorder here; P06 issues h_free
on the pool-fail path, P07 allocates pools before register
Changes in v6:
- retarget the free-by-presence comment to the later resize paths;
alloc_buffer_pool() already undoes its own partials, so this
open-fail slot is empty
- use %u for the two num_rx_queues dbg prints
- noted: pool-fail after register without h_free_logical_lan predates
this patch; the inline pool loop reached the same labels. P06
issues h_free before the pool DMA; P07 moves pools ahead of
register_rx_queues
Changes in v5:
- vs mailed v4: keep pool_active[] = {1,1,0,0,1} (v4 set the 64 KiB
pool inactive). No code delta vs prior patches here; SQ large-receive /
CMO stay at the historical baseline. change_mtu does not re-enable
pool4 at MTU 1500; MQ memory pressure belongs at scale-up, not SQ
default
- Unwind / free pools by real allocations (free_map/dma_addr/skbuff),
not only pool->active, so open-fail cannot leak partially allocated
pools (v4 fail path freed by active and skipped the failing pool)
- Clarify kdoc: free paths use allocation presence (not "all active");
queue 1..N metadata copy from queue 0 (v4 said "queues 1-15" while
MAX is still 1 here)
- Document poolN sysfs size multiplies by num_rx_queues (queue 0 is the
shared template; no per-queue pool sysfs in this series)
- Call out close() pool-free before cleanup_rx_resources, after LAN/
NAPI/IRQ teardown (order already in v4; spell it for later drain)
Changes in v4:
- Introduce the pool helpers in the same patch that wires their first
open/close callers, instead of leaving unused statics.
- Copy pool->index when cloning queue-0 geometry to later queues
(needed for correlators; also required by incremental resize).
drivers/net/ethernet/ibm/ibmveth.c | 159 +++++++++++++++++++++++++----
1 file changed, 139 insertions(+), 20 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 68791402cd53..31fad7713dc7 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -678,6 +678,140 @@ static void ibmveth_rxq_no_skb(struct ibmveth_adapter *adapter,
schedule_work(&adapter->work);
}
+/**
+ * ibmveth_free_queue_buffer_pools - Free buffer pools for a single queue
+ * @adapter: ibmveth adapter structure
+ * @queue: queue index
+ *
+ * Frees buffer pools that still hold allocations for the specified
+ * queue (by free_map / dma_addr / skbuff presence), regardless of the
+ * active flag.
+ */
+static void ibmveth_free_queue_buffer_pools(struct ibmveth_adapter *adapter,
+ int queue)
+{
+ int i;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[queue][i];
+
+ /* Free by allocation presence, not by the active flag,
+ * so a pool's memory is released whatever active says.
+ */
+ if (pool->free_map || pool->dma_addr || pool->skbuff)
+ ibmveth_free_buffer_pool(adapter, pool);
+ }
+}
+
+/**
+ * ibmveth_alloc_queue_buffer_pools - Allocate buffer pools for a single queue
+ * @adapter: ibmveth adapter structure
+ * @queue: queue index
+ *
+ * Allocates backing storage for each active pool on @queue.
+ * Inactive pools (!active) are skipped. Pool metadata must be
+ * initialized before calling this function.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int ibmveth_alloc_queue_buffer_pools(struct ibmveth_adapter *adapter,
+ int queue)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[queue][i];
+
+ if (!bpool->active)
+ continue;
+
+ if (ibmveth_alloc_buffer_pool(bpool)) {
+ netdev_err(netdev,
+ "queue %d pool %d alloc failed (size=%u count=%u)\n",
+ queue, i,
+ bpool->buff_size,
+ bpool->size);
+ bpool->active = 0;
+ /* Undo the pools this queue already allocated; the
+ * failed one freed its own partial allocation.
+ */
+ ibmveth_free_queue_buffer_pools(adapter, queue);
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * ibmveth_alloc_buffer_pools - Allocate buffer pools for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Initializes pool metadata for queues 1-N from queue 0 settings,
+ * then allocates buffer pools for all queues using the helper function.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i, q, rc;
+
+ /* Initialize pool metadata for queues 1..N from queue 0 settings */
+ for (q = 1; q < adapter->num_rx_queues; q++) {
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *src =
+ &adapter->rx_buff_pool[0][i];
+ struct ibmveth_buff_pool *dst =
+ &adapter->rx_buff_pool[q][i];
+
+ dst->size = src->size;
+ dst->index = src->index;
+ dst->buff_size = src->buff_size;
+ dst->threshold = src->threshold;
+ dst->active = src->active;
+ }
+ }
+
+ /* Allocate actual buffers for all queues */
+ for (q = 0; q < adapter->num_rx_queues; q++) {
+ rc = ibmveth_alloc_queue_buffer_pools(adapter, q);
+ if (rc) {
+ /* Free pools for all previous queues */
+ while (--q >= 0)
+ ibmveth_free_queue_buffer_pools(adapter, q);
+ return rc;
+ }
+ }
+
+ netdev_dbg(netdev, "allocated buffer pools for %u queue(s)\n",
+ adapter->num_rx_queues);
+ return 0;
+}
+
+/**
+ * ibmveth_free_buffer_pools - Free buffer pools for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Frees buffer pools for all queues using the helper function.
+ */
+static void
+ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
+{
+ int q;
+
+ /* Free buffer pools for all queues */
+ for (q = 0; q < adapter->num_rx_queues; q++)
+ ibmveth_free_queue_buffer_pools(adapter, q);
+
+ netdev_dbg(adapter->netdev, "freed buffer pools for %u queue(s)\n",
+ adapter->num_rx_queues);
+}
+
/**
* ibmveth_remove_buffer_from_pool - remove a buffer from a pool
* @adapter: adapter instance
@@ -907,16 +1041,9 @@ static int ibmveth_open(struct net_device *netdev)
goto out_free_tx_ltb;
}
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- if (!adapter->rx_buff_pool[0][i].active)
- continue;
- if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[0][i])) {
- netdev_err(netdev, "unable to alloc pool\n");
- adapter->rx_buff_pool[0][i].active = 0;
- rc = -ENOMEM;
- goto out_free_buffer_pools;
- }
- }
+ rc = ibmveth_alloc_buffer_pools(adapter);
+ if (rc)
+ goto out_free_buffer_pools;
/* NAPI can run as soon as it is enabled, from netpoll during the
* direct close()/open() pairs or from a direct ibmveth_interrupt()
@@ -950,11 +1077,7 @@ static int ibmveth_open(struct net_device *netdev)
do {
lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
- while (--i >= 0) {
- if (adapter->rx_buff_pool[0][i].active)
- ibmveth_free_buffer_pool(adapter,
- &adapter->rx_buff_pool[0][i]);
- }
+ ibmveth_free_buffer_pools(adapter);
out_free_tx_ltb:
for (i = netdev->real_num_tx_queues - 1; i >= 0; i--)
ibmveth_free_tx_ltb(adapter, i);
@@ -1005,14 +1128,10 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_update_rx_no_buffer(adapter);
+ ibmveth_free_buffer_pools(adapter);
ibmveth_cleanup_rx_resources(adapter);
ibmveth_free_filter_list(adapter);
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
- if (adapter->rx_buff_pool[0][i].active)
- ibmveth_free_buffer_pool(adapter,
- &adapter->rx_buff_pool[0][i]);
-
for (i = 0; i < netdev->real_num_tx_queues; i++)
ibmveth_free_tx_ltb(adapter, i);
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 05/15] ibmveth: Refactor RX interrupt control for MQ RX queues
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (3 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 04/15] ibmveth: Refactor buffer pool management for per-queue MQ RX Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 06/15] ibmveth: Refactor TX resource allocation in open/close paths Mingming Cao
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
Queue 0 and subordinate RX queues use different interrupt control
interfaces in PHYP:
- queue 0: h_vio_signal() after h_register_logical_lan()
- queue N: H_VIOCTL against the queue's mapped hwirq
The current code is single-queue oriented and cannot safely scale to
multiple RX queues in poll completion and open/close IRQ setup.
Introduce queue-indexed interrupt helpers and wire them into
open()/close()/poll()/interrupt in the same patch:
ibmveth_toggle_irq() / enable_irq() / disable_irq()
ibmveth_setup_rx_interrupts() / ibmveth_cleanup_rx_interrupts()
ibmveth_schedule_rx_queue()
These helpers centralize queue0-vs-subordinate dispatch.
request_irq() uses &adapter->napi[i] as the per-queue cookie so the
handler can resolve the queue index.
Add ibmveth_dispose_subordinate_irq_mapping() and a bulk 1..N
variant; queue 0 keeps netdev->irq and its mapping is left
alone. The interrupt handler returns IRQ_NONE for an
out-of-range queue index instead of warning.
Move napi_enable() into setup_rx_interrupts(). It still runs only
after LAN registration and buffer-pool allocation, so everything
ibmveth_poll() touches exists, but now after request_irq() rather than
just before it: PHYP delivery is still masked at that point. In this
single-queue tree, setup does not yet unmask PHYP; schedule_rx_queue()
masks queue 0 and schedules NAPI, and ibmveth_poll() is what unmasks
it on completion. That order matches the later scale-up rule (NAPI
live before PHYP unmask), not an inverted window relative to it.
Factor process-context RX kicks (open, resume, pool sysfs)
into ibmveth_schedule_rx_queue(); keep ibmveth_interrupt() as a thin
IRQ-only wrapper.
cleanup_rx_interrupts() masks PHYP and synchronizes IRQs before
napi_disable, waits for the poll tail, then remasks and
synchronizes again before free_irq. napi_disable waits for SCHED. The
synchronize_net() after it covers enable_irq after complete, so
remask/free_irq cannot race that tail, and close() needs no wait of
its own. Close then proceeds to h_free_logical_lan(): free_irq
before free_lan is intentional once PHYP delivery is masked.
close() now runs netif_tx_disable() first; net-next ran it after
napi_disable() and synchronize_net(). Its h_free_logical_lan
failure message is shortened.
rx_irq_setup gates whether cleanup walks IRQ/NAPI state. close()
remains gated on adapter->opened from commit 95811c873d1b ("ibmveth:
do not close twice after a failed reopen"); this patch does not
introduce adapter->opened or change that gate.
schedule_rx_queue() masks PHYP only when napi_schedule_prep() succeeds.
Masking on prep failure can race a completing poll that already
re-enabled PHYP and leave NAPI idle with the queue masked (TX OK, RX
stalled until reload). Teardown storm control stays on STOP
(ibmveth_disable_irq() + synchronize_irq() before napi_disable).
Poll still enables after complete; cleanup's wait covers that tail
versus free_irq. The RX poll helper patch adds the stopping check.
IRQ helpers return 0 or negative errno only (never raw H_* to
ethtool/resize). On subordinate queues (H_VIOCTL), H_PARAMETER on
disable is folded to success (idempotent mask). Queue 0 uses
h_vio_signal() and any failure is -EIO. On enable, H_PARAMETER does
not mean already enabled (enabling an enabled interrupt succeeds); it
means the interrupt number is out of range for the unit address. Keep
it an error so a missed LISN stays visible to poll/resize recovery.
A failed h_vio_signal() in poll or the interrupt handler is now
reported by a rate-limited dev_err() in ibmveth_toggle_irq() instead
of WARN_ON(); poll still schedules a reset on an enable failure.
Runtime remains single-queue (num_rx_queues is still 1).
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- teardown: synchronize_net() after napi_disable(), then remask and
free_irq()
- rebased onto net-next's poll tail (mask then schedule, no
restart_poll)
- close() keeps net-next's order and samples the no_buffer counter
after h_free_logical_lan()
- ibmveth_schedule_rx_queue() returns void
- commit message: the H_PARAMETER fold applies to the subordinate
H_VIOCTL disable only
Dropped from this series (merged fixes own them):
- commit 8d9ecac6b96e: poll_controller() netpoll kick (ndo removed)
- commit 95811c873d1b: adapter->opened close gate (in base; not re-added)
- commit 3f872b34368c: close/request_irq synchronize_net() hunks; helpers
carry the close wait; request_irq fails before napi_enable
- commit af0524bf4ce1: setup-fail inline h_free; out_free_buffer_pools
already calls h_free_logical_lan()
Changes in v7:
- reword schedule_rx_queue() kdoc: out-of-range is
WARN_ON, then false
- commit message: remask only catches a re-arm that lands
before it; name harvest before h_free and synchronize_net()
after teardown
- noted: no Fixes: peel of the opened-gate hang; stays in
this patch, not the unwind standalone
Changes in v6:
- setup_rx enable-fail remasks queues 0..i, including the one that
failed (the rollback used while (--i) and left queue i unmasked
across napi_disable); err_disable_napi remasks after napi_disable
- drop WARN_ON() on enable/disable_irq() returns (schedule_rx and
poll); the helper already logs the hcall rc, and rate-limit that
print
- reword schedule_rx_queue() kdoc: true means NAPI was scheduled and
the mask attempted; a failed disable_irq() does not change the
return, and an out-of-range qindex returns false
- noted: pool-fail without h_free predates this; the opened gate
also drops the accidental h_free a later ndo_stop used to give.
P06/P07 close it
- noted: set_channels() IFF_UP vs opened TX LTB window closes in P14/P15
- noted: poll() takes queue_index in P08; range check / skip helpers
in P09
Changes in v5:
- Remask+sync after napi_disable in cleanup (in-flight poll can re-arm)
- Interrupt: quiet IRQ_NONE on out-of-range qindex (no WARN storm)
- Opened / rx_irq_setup gate cleanup so close after a failed open
cannot napi_disable / free_irq without a prior enable/request; set
opened on successful open; rx_irq_setup only on full setup success
- H_PARAMETER fold disable-only; enable stays error (stuck-masked visible)
- IRQ helpers return 0 / negative errno only (never raw H_* to ethtool)
- Decision: keep open IRQ order request_irq -> napi_enable while PHYP
stays masked until schedule/enable (coherent with later scale-up)
- schedule_rx_queue returns bool (napi_schedule_prep success)
- Keep mask-only-on-prep-success (no else-mask; avoids idle+masked race)
- Call out free_irq-before-free_lan as intentional once PHYP is masked
- Poll re-arm during teardown lands with the RX poll helper patch
(not claimed here)
- synchronize_net() after RX IRQ/NAPI teardown in close
- Add subordinate IRQ dispose helpers (per-queue + bulk 1..N; bound to MAX)
Changes in v4:
- Include irq.h / irqdomain.h with first irq_dispose_mapping() use.
- Introduce IRQ helpers in the same patch that wires open/close/poll
callers, instead of leaving unused statics.
- Factor process-context RX kicks into ibmveth_schedule_rx_queue();
keep ibmveth_interrupt() as the IRQ-only wrapper.
- On cleanup, mask PHYP and synchronize_irq before napi_disable
(storm-safety; not fully behavior-preserving vs classic close).
- Leave queue_irq[0] set after cleanup (queue 0 uses netdev->irq;
next open reuses it). Only subordinate virqs are disposed.
drivers/net/ethernet/ibm/ibmveth.c | 404 +++++++++++++++++++++++++----
drivers/net/ethernet/ibm/ibmveth.h | 2 +
2 files changed, 357 insertions(+), 49 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 31fad7713dc7..62e6c9a9f5e9 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -21,6 +21,8 @@
#include <linux/skbuff.h>
#include <linux/init.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
#include <linux/mm.h>
#include <linux/pm.h>
#include <linux/ethtool.h>
@@ -347,6 +349,325 @@ ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
}
}
+/**
+ * ibmveth_toggle_irq - Common helper to enable/disable queue interrupts
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ * @enable: true to enable, false to disable
+ *
+ * For queue 0 (primary), uses h_vio_signal() as it's registered via
+ * h_register_logical_lan(). For subordinate queues (1+), uses H_VIOCTL
+ * with H_ENABLE/DISABLE_VIO_INTERRUPT for per-queue interrupt control.
+ *
+ * Return: 0 on success, negative errno on failure (never raw H_*).
+ */
+static int
+ibmveth_toggle_irq(struct ibmveth_adapter *adapter, int queue_index,
+ bool enable)
+{
+ unsigned long h_rc;
+ unsigned long irq = adapter->queue_irq[queue_index];
+ const char *action = enable ? "enable" : "disable";
+
+ if (queue_index == 0) {
+ /* Primary queue: use h_vio_signal() */
+ h_rc = h_vio_signal(adapter->vdev->unit_address,
+ enable ? VIO_IRQ_ENABLE : VIO_IRQ_DISABLE);
+ } else {
+ /* Subordinate queues: use H_VIOCTL with hardware IRQ */
+ struct irq_data *irq_data = irq_get_irq_data(irq);
+ irq_hw_number_t hwirq;
+ u64 vioctl_cmd = enable ? H_ENABLE_VIO_INTERRUPT :
+ H_DISABLE_VIO_INTERRUPT;
+
+ if (!irq_data) {
+ netdev_err(adapter->netdev,
+ "Failed to get IRQ data for queue %d (virq=%lu)\n",
+ queue_index, irq);
+ return -EINVAL;
+ }
+
+ hwirq = irqd_to_hwirq(irq_data);
+ h_rc = plpar_hcall_norets(H_VIOCTL,
+ adapter->vdev->unit_address,
+ vioctl_cmd,
+ hwirq, 0, 0);
+
+ /*
+ * Fold H_PARAMETER only on disable (idempotent mask). On
+ * enable it is not already-enabled: a trivial enable
+ * returns success. H_PARAMETER means the interrupt
+ * number was not in the valid range for that unit
+ * address. Keep that an error so a missed LISN stays
+ * visible to poll/resize recovery.
+ */
+ if (h_rc == H_PARAMETER && !enable) {
+ dev_warn_ratelimited(&adapter->netdev->dev,
+ "H_VIOCTL %s IRQ returned H_PARAMETER for queue %d (hwirq=%lu)\n",
+ action, queue_index, hwirq);
+ return 0;
+ }
+ }
+
+ if (h_rc) {
+ dev_err_ratelimited(&adapter->netdev->dev,
+ "Failed to %s IRQ for queue %d, rc=0x%lx\n",
+ action, queue_index, h_rc);
+ return -EIO;
+ }
+ return 0;
+}
+
+/**
+ * ibmveth_disable_irq - Disable interrupt for a specific queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int
+ibmveth_disable_irq(struct ibmveth_adapter *adapter, int queue_index)
+{
+ return ibmveth_toggle_irq(adapter, queue_index, false);
+}
+
+/**
+ * ibmveth_enable_irq - Enable interrupt for a specific queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int
+ibmveth_enable_irq(struct ibmveth_adapter *adapter, int queue_index)
+{
+ return ibmveth_toggle_irq(adapter, queue_index, true);
+}
+
+/**
+ * ibmveth_dispose_subordinate_irq_mapping - Drop one subordinate virq mapping
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: RX queue index (1..N)
+ *
+ * Subordinate queues get mappings from irq_create_mapping() during PHYP
+ * registration. Queue 0 uses netdev->irq from device tree and is left alone.
+ *
+ * Bound against IBMVETH_MAX_RX_QUEUES, not num_rx_queues: a caller may
+ * dispose a queue that is no longer in the published live set but still
+ * owns a virq in queue_irq[]. Contrast with the bulk helper, which only
+ * walks 1..num_rx_queues-1 (close / open-fail cleanup of the live set).
+ *
+ * Linux virq lifetime is owned by interrupt cleanup helpers. Call this only
+ * after free_irq() when a handler was installed, or from registration failure
+ * cleanup before request_irq().
+ */
+static void
+ibmveth_dispose_subordinate_irq_mapping(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ if (queue_idx <= 0 || queue_idx >= IBMVETH_MAX_RX_QUEUES)
+ return;
+
+ if (adapter->queue_irq[queue_idx]) {
+ irq_dispose_mapping(adapter->queue_irq[queue_idx]);
+ adapter->queue_irq[queue_idx] = 0;
+ }
+}
+
+/**
+ * ibmveth_dispose_subordinate_irq_mappings - Drop virq mappings for queues 1..N
+ * @adapter: ibmveth adapter structure
+ *
+ * Bulk helper for close / open-fail cleanup of the published live set
+ * (queues 1..num_rx_queues-1). Paths that need a retired or not-yet-published
+ * queue must call ibmveth_dispose_subordinate_irq_mapping() directly.
+ */
+static void
+ibmveth_dispose_subordinate_irq_mappings(struct ibmveth_adapter *adapter)
+{
+ int i;
+
+ for (i = 1; i < adapter->num_rx_queues; i++)
+ ibmveth_dispose_subordinate_irq_mapping(adapter, i);
+}
+
+/**
+ * ibmveth_setup_rx_interrupts - Register IRQs and enable NAPI
+ * @adapter: ibmveth adapter structure
+ *
+ * Registers interrupt handlers for all RX queues, enables NAPI, then
+ * enables hypervisor interrupt delivery for multi-queue mode after
+ * every queue has a Linux handler installed. For multi-queue open the
+ * caller should replenish RX buffers before this helper so traffic
+ * during open is not dropped (PHYP only interrupts after a successful
+ * enqueue, which needs buffers). Single-queue open leaves PHYP masked
+ * here and kicks NAPI afterward (classic path: first poll posts then
+ * enables).
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i, rc, num = adapter->num_rx_queues;
+
+ for (i = 0; i < num; i++) {
+ if (!adapter->queue_irq[i]) {
+ netdev_err(netdev, "queue %d has invalid IRQ (0)\n", i);
+ rc = -EINVAL;
+ goto err_free_irqs;
+ }
+
+ rc = request_irq(adapter->queue_irq[i], ibmveth_interrupt,
+ 0, netdev->name, &adapter->napi[i]);
+ if (rc) {
+ netdev_err(netdev,
+ "request_irq() failed for irq 0x%x queue %d: %d\n",
+ adapter->queue_irq[i], i, rc);
+ goto err_free_irqs;
+ }
+ }
+
+ for (i = 0; i < num; i++)
+ napi_enable(&adapter->napi[i]);
+
+ if (adapter->multi_queue && num > 1) {
+ for (i = 0; i < num; i++) {
+ rc = ibmveth_enable_irq(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to enable IRQ for queue %d, rc=%d\n",
+ i, rc);
+ for (; i >= 0; i--) {
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+ rc = -EIO;
+ goto err_disable_napi;
+ }
+ }
+ }
+
+ /* Set only on full success; fail paths leave this false so a later
+ * close() / cleanup is a no-op.
+ */
+ adapter->rx_irq_setup = true;
+ return 0;
+
+err_disable_napi:
+ /* STOP: wait for the poll tail, then remask; an in-flight poll
+ * can re-arm after napi_complete_done().
+ */
+ for (i = 0; i < num; i++)
+ napi_disable(&adapter->napi[i]);
+ synchronize_net();
+ for (i = 0; i < num; i++) {
+ if (!adapter->queue_irq[i])
+ continue;
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+ for (i = 0; i < num; i++) {
+ if (adapter->queue_irq[i])
+ free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+ }
+ goto err_dispose_mappings;
+
+err_free_irqs:
+ while (--i >= 0)
+ free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+err_dispose_mappings:
+ /* Both setup failure paths own subordinate virq disposal. */
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+ return rc;
+}
+
+/**
+ * ibmveth_cleanup_rx_interrupts - Mask PHYP IRQs, stop NAPI, and free IRQs
+ * @adapter: ibmveth adapter structure
+ *
+ * Mask and synchronize each queue IRQ before napi_disable() so the handler
+ * cannot miss a PHYP mask while NAPI is already dead. napi_disable() only
+ * waits for SCHED to clear at napi_complete_done(), so wait for the poll
+ * tail after it, then remask and synchronize again before free_irq(). The
+ * remask alone does not wait for poll to return. Safe for close and for
+ * open failure after setup_rx_interrupts() already unmasked PHYP. No-op if
+ * setup never succeeded (avoids double napi_disable / free_irq after a
+ * failed close+open while IFF_UP remains set).
+ */
+static void
+ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
+{
+ int i;
+
+ if (!adapter->rx_irq_setup)
+ return;
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (!adapter->queue_irq[i])
+ continue;
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ napi_disable(&adapter->napi[i]);
+
+ synchronize_net();
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (!adapter->queue_irq[i])
+ continue;
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (adapter->queue_irq[i])
+ free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+ }
+
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+
+ /* Queue 0 uses netdev->irq; leave queue_irq[0] for next open. */
+ adapter->rx_irq_setup = false;
+}
+
+/**
+ * ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue
+ * @adapter: ibmveth adapter structure
+ * @qindex: RX queue index
+ *
+ * Shared by the IRQ handler and process-context kick sites (open, resume,
+ * pool sysfs).
+ *
+ * Does nothing if the index is out of range (WARN_ON) or
+ * napi_schedule_prep() fails (NAPI already scheduled, or disabled
+ * because the device is closed). A failed
+ * disable_irq() is logged by the helper; the queue may stay unmasked.
+ */
+static void ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
+ int qindex)
+{
+ struct napi_struct *napi = &adapter->napi[qindex];
+
+ if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
+ return;
+
+ /*
+ * Only mask PHYP when NAPI will run. Masking on prep failure can
+ * race a completing poll that already re-enabled the queue, leaving
+ * NAPI idle with the IRQ masked (TX works, RX stalls) until reload.
+ * Storm prevention on teardown remains in cleanup/disable paths.
+ */
+ if (napi_schedule_prep(napi)) {
+ /* Failure is already logged with the hcall rc by the helper. */
+ ibmveth_disable_irq(adapter, qindex);
+ __napi_schedule(napi);
+ }
+}
+
/* setup the initial settings for a buffer pool */
static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,
u32 pool_index, u32 pool_size,
@@ -1024,7 +1345,8 @@ static int ibmveth_open(struct net_device *netdev)
adapter->rx_queue[0].queue_len;
rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
- h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
+ adapter->queue_irq[0] = netdev->irq;
+ ibmveth_disable_irq(adapter, 0);
lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
@@ -1045,26 +1367,16 @@ static int ibmveth_open(struct net_device *netdev)
if (rc)
goto out_free_buffer_pools;
- /* NAPI can run as soon as it is enabled, from netpoll during the
- * direct close()/open() pairs or from a direct ibmveth_interrupt()
- * call, so enable it only once everything ibmveth_poll() touches
- * exists.
+ /* Enables NAPI, so everything ibmveth_poll() touches must exist:
+ * netpoll during the direct close()/open() pairs, or a direct
+ * kick, can run it as soon as it is enabled.
*/
- napi_enable(&adapter->napi[0]);
-
- netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
- rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
- netdev);
- if (rc != 0) {
- napi_disable(&adapter->napi[0]);
- synchronize_net();
- netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
- netdev->irq, rc);
+ rc = ibmveth_setup_rx_interrupts(adapter);
+ if (rc)
goto out_free_buffer_pools;
- }
netdev_dbg(netdev, "initial replenish cycle\n");
- ibmveth_interrupt(netdev->irq, netdev);
+ ibmveth_schedule_rx_queue(adapter, 0);
netif_tx_start_all_queues(netdev);
@@ -1104,30 +1416,20 @@ static int ibmveth_close(struct net_device *netdev)
netdev_dbg(netdev, "close starting\n");
- napi_disable(&adapter->napi[0]);
- /* napi_disable() returns once ibmveth_poll() has called
- * napi_complete_done(), but the poll still re-enables the
- * interrupt and reads the RX queue after that.
- */
- synchronize_net();
-
netif_tx_disable(netdev);
- h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
+ ibmveth_cleanup_rx_interrupts(adapter);
+ /* Full LAN teardown (subordinates arrive with register helpers). */
do {
lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-
if (lpar_rc != H_SUCCESS) {
- netdev_err(netdev, "h_free_logical_lan failed with %lx, "
- "continuing with close\n", lpar_rc);
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan failed with %lx, continuing\n",
+ lpar_rc);
}
-
- free_irq(netdev->irq, netdev);
-
ibmveth_update_rx_no_buffer(adapter);
-
ibmveth_free_buffer_pools(adapter);
ibmveth_cleanup_rx_resources(adapter);
ibmveth_free_filter_list(adapter);
@@ -1814,7 +2116,6 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
container_of(napi, struct ibmveth_adapter, napi[0]);
struct net_device *netdev = adapter->netdev;
int frames_processed = 0;
- unsigned long lpar_rc;
u64 correlator;
u16 mss = 0;
int rc;
@@ -1962,18 +2263,17 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
/* We think we are done - reenable interrupts,
* then check once more to make sure we are done.
*/
- lpar_rc = h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_ENABLE);
- if (WARN_ON(lpar_rc != H_SUCCESS)) {
+ rc = ibmveth_enable_irq(adapter, 0);
+ if (rc) {
schedule_work(&adapter->work);
goto out;
}
- /* Mask before queueing, as ibmveth_interrupt() does: a threaded
+ /* Mask before queueing, as schedule_rx_queue() does: a threaded
* NAPI can start polling, and unmask, as soon as it is queued.
*/
if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule_prep(napi)) {
- lpar_rc = h_vio_signal(adapter->vdev->unit_address,
- VIO_IRQ_DISABLE);
+ ibmveth_disable_irq(adapter, 0);
__napi_schedule(napi);
}
@@ -1983,16 +2283,20 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
{
- struct net_device *netdev = dev_instance;
+ struct napi_struct *napi = dev_instance;
+ struct net_device *netdev = napi->dev;
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- unsigned long lpar_rc;
+ int qindex;
- if (napi_schedule_prep(&adapter->napi[0])) {
- lpar_rc = h_vio_signal(adapter->vdev->unit_address,
- VIO_IRQ_DISABLE);
- WARN_ON(lpar_rc != H_SUCCESS);
- __napi_schedule(&adapter->napi[0]);
- }
+ qindex = napi - adapter->napi;
+ /*
+ * Quiet on out-of-range: teardown can leave a residual IRQ after the
+ * live count drops. Do not WARN-storm; return IRQ_NONE until free_irq.
+ */
+ if (qindex < 0 || qindex >= adapter->num_rx_queues)
+ return IRQ_NONE;
+
+ ibmveth_schedule_rx_queue(adapter, qindex);
return IRQ_HANDLED;
}
@@ -2536,8 +2840,8 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
}
rtnl_unlock();
- /* kick the interrupt handler to allocate/deallocate pools */
- ibmveth_interrupt(netdev->irq, netdev);
+ /* kick RX processing to allocate/deallocate pools */
+ ibmveth_schedule_rx_queue(adapter, 0);
return count;
unlock_err:
@@ -2577,7 +2881,9 @@ static struct kobj_type ktype_veth_pool = {
static int ibmveth_resume(struct device *dev)
{
struct net_device *netdev = dev_get_drvdata(dev);
- ibmveth_interrupt(netdev->irq, netdev);
+ struct ibmveth_adapter *adapter = netdev_priv(netdev);
+
+ ibmveth_schedule_rx_queue(adapter, 0);
return 0;
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index df5a3b69a157..50be6c95457e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -325,6 +325,8 @@ struct ibmveth_adapter {
unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
int multi_queue;
unsigned int num_rx_queues;
+ /* Lifetime: true while RX IRQ handlers / NAPI are installed. */
+ bool rx_irq_setup;
int rx_csum;
int large_send;
bool is_active_trunk;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 06/15] ibmveth: Refactor TX resource allocation in open/close paths
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (4 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 05/15] ibmveth: Refactor RX interrupt control for MQ RX queues Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 07/15] ibmveth: Add RX queue register helpers for MQ Mingming Cao
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
As with the RX refactor, pull TX LTB alloc/free out of open/close
into helpers and wire them in this patch.
ibmveth_alloc_tx_resources()
ibmveth_free_tx_resources()
They wrap the existing per-queue allocate_tx_ltb() / free_tx_ltb()
primitives. alloc_tx_resources() allocates every TX queue and unwinds
partial failure itself; free_tx_resources() walks real_num_tx_queues.
The helpers remove dependence on shared open/close loop indices and
match the RX helper structure. TX was already multi-queue capable via
ethtool -L.
free_tx_ltb() already returns early on a NULL slot and clears
tx_ltb_ptr[] before unmapping. Clear tx_ltb_dma[] there too, and on
the allocate_tx_ltb() DMA-map failure path, so a freed slot carries no
stale DMA address.
Stop TX at the start of open() with netif_tx_disable() so TX cannot
run while TX LTBs are still being allocated. That covers first-open
generic XDP_TX, whose queues are not DRV_XOFF.
netif_tx_start_all_queues() stays after alloc_tx_resources(), under
netif_tx_lock_bh(): RX NAPI is already live there, and generic
XDP_TX and netpoll test DRV_XOFF under the queue's xmit lock before
they read the LTB, so the lock orders the LTB stores before any
sender that sees the queues started.
Move TX LTB allocation to the end of open(), after LAN registration,
RX pools, RX interrupt setup, and the initial replenish kick. A late
alloc_tx_resources() failure jumps to out_cleanup_rx_interrupts and
must not call free_tx_resources() again: alloc already freed any
partial TX LTBs. RX is live by then; cleanup_rx_interrupts() waits
for the poll tail before the RX ring and pools are freed, as it
does for close(). start_xmit() bails if
tx_ltb_ptr[] is gone, counting the drop in tx_dropped and falling
into the existing out: label like the function's other drop paths.
That guard catches a sender that picked its queue before an
ethtool -L shrink and arrives after the free; a sender already in
xmit during the free is not covered, as before this series. The
close path is quiesced by netif_tx_disable().
Every failure after LAN registration still issues
h_free_logical_lan() before the RX pools are freed. As in close(),
the unwind does not check its result, and the late TX failure now
reaches it with RX buffers already posted; a later patch makes the
multi-queue paths keep that memory mapped on a hard failure.
close() keeps netif_tx_disable() and frees the LTBs through
free_tx_resources() after h_free_logical_lan().
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- netif_tx_disable() at the start of open(); the TX queues start under
netif_tx_lock_bh(), since NAPI (and generic XDP_TX) is already live
- a late TX allocation failure unwinds through cleanup_rx_interrupts()
(patch 5), which already waits for the poll
- start_xmit LTB guard comment is about a sender that picked its queue
before a shrink
- commit message: the late TX failure unwind frees posted RX after an
unchecked H_FREE, as close() does (MQ keeps it mapped from patch 10)
Dropped from this series (merged fixes own them):
- commit c807573ea58e: close() netif_tx_stop_all_queues() ->
netif_tx_disable() (in base; this patch adds open()-start disable)
- commit 84bec0bf0352: free_tx_ltb() NULL check and NULL-first clear
(in base; MQ refactor keeps the v6 behavior, not a duplicate fix hunk)
- commit 95811c873d1b: set_channels IFF_UP vs opened commit-message claim
(base gates set_channels() on opened)
Changes in v7:
- late alloc_tx_resources() failure: synchronize_net()
after cleanup_rx_interrupts(), as close() does, before
the RX ring and pools are freed
- noted: no Fixes: peel of the pool-fail h_free; stay
at 15; one standalone [PATCH net] after this series
(Fixes: d43732ce021f)
Changes in v6:
- free_tx_ltb(): clear tx_ltb_ptr[idx] then tx_ltb_dma[idx] before
unmap (NULL-first narrows the race; close still uses netif_tx_disable())
- start_xmit(): tx_dropped++ and goto out: when the LTB is missing
- allocate_tx_ltb(): comment that caller must leave tx_ltb_ptr[idx] NULL
- alloc/free_tx_resources kdoc uses real_num_tx_queues
- noted: ethtool -L TX shrink still stop-then-free; cover leftovers
Changes in v5:
- Quiesce TX with netif_tx_disable before free (stop_all_queues does not
wait for in-flight xmit); free LTBs after h_free_logical_lan - direct
close() callers bypass synchronize_net()
- Guard start_xmit if tx_ltb_ptr gone so open can leave RX live while TX
LTB alloc still runs (also covers close/failed-reopen with IFF_UP set)
- Drop fake mid-open TX-leak / Fixes: motivation; reword as helper
extraction matching RX (shared loop-index independence)
- Free TX LTB by pointer presence (drop dma==0 sentinel; dma_mapping_error
already cleared the slot on map failure)
- Document intentional open-fail LAN-first unwind (free_lan before RX
pool/DMA teardown) rather than leaving it silent in a TX-only refactor
- Drop drive-by blank-line cosmetics (header / start_xmit)
Changes in v4:
- Introduce the TX resource helpers in the same patch that wires their
first open/close callers.
- Do not free TX LTBs again after a failed alloc_tx_resources();
harden free_tx_ltb() against unset slots.
- Move TX allocation after RX IRQ setup / replenish kick so open()
failure unwind no longer depends on a shared loop index (also fixes
a mid-open TX LTB leak).
drivers/net/ethernet/ibm/ibmveth.c | 111 +++++++++++++++++++++++------
1 file changed, 89 insertions(+), 22 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 62e6c9a9f5e9..cab895cc970c 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1248,17 +1248,27 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
{
- void *ptr = adapter->tx_ltb_ptr[idx];
+ void *ltb = adapter->tx_ltb_ptr[idx];
+ dma_addr_t dma = adapter->tx_ltb_dma[idx];
- if (!ptr)
+ if (!ltb)
return;
+ /*
+ * Clear the slot before releasing it. start_xmit() tests
+ * tx_ltb_ptr[idx] to decide whether the LTB is usable.
+ */
adapter->tx_ltb_ptr[idx] = NULL;
- dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
- adapter->tx_ltb_size, DMA_TO_DEVICE);
- kfree(ptr);
+ adapter->tx_ltb_dma[idx] = 0;
+
+ dma_unmap_single(&adapter->vdev->dev, dma, adapter->tx_ltb_size,
+ DMA_TO_DEVICE);
+ kfree(ltb);
}
+/* Caller must ensure tx_ltb_ptr[idx] is NULL. open() runs on
+ * probe-zeroed slots; set_channels() skips populated indices.
+ */
static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
{
adapter->tx_ltb_ptr[idx] = kzalloc(adapter->tx_ltb_size,
@@ -1277,12 +1287,54 @@ static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
"unable to DMA map tx long term buffer\n");
kfree(adapter->tx_ltb_ptr[idx]);
adapter->tx_ltb_ptr[idx] = NULL;
+ adapter->tx_ltb_dma[idx] = 0;
return -ENOMEM;
}
return 0;
}
+/**
+ * ibmveth_alloc_tx_resources - Allocate TX LTBs for real_num_tx_queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Allocates TX Long Term Buffers (LTBs) for real_num_tx_queues.
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_resources(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < netdev->real_num_tx_queues; i++) {
+ if (ibmveth_allocate_tx_ltb(adapter, i))
+ goto err_free_ltbs;
+ }
+
+ return 0;
+
+err_free_ltbs:
+ while (--i >= 0)
+ ibmveth_free_tx_ltb(adapter, i);
+ return -ENOMEM;
+}
+
+/**
+ * ibmveth_free_tx_resources - Free TX LTBs for real_num_tx_queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Frees TX Long Term Buffers (LTBs) for real_num_tx_queues.
+ */
+static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ int i;
+
+ for (i = 0; i < netdev->real_num_tx_queues; i++)
+ ibmveth_free_tx_ltb(adapter, i);
+}
+
static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
union ibmveth_buf_desc rxq_desc, u64 mac_address)
{
@@ -1322,6 +1374,11 @@ static int ibmveth_open(struct net_device *netdev)
netdev_dbg(netdev, "open starting\n");
+ /* Stop TX before RX is live. First-open queues are not DRV_XOFF,
+ * so generic XDP_TX can run during TX LTB alloc.
+ */
+ netif_tx_disable(netdev);
+
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
rxq_entries += adapter->rx_buff_pool[0][i].size;
@@ -1333,12 +1390,6 @@ static int ibmveth_open(struct net_device *netdev)
if (rc)
goto out_free_filter_list;
- rc = -ENOMEM;
- for (i = 0; i < netdev->real_num_tx_queues; i++) {
- if (ibmveth_allocate_tx_ltb(adapter, i))
- goto out_free_tx_ltb;
- }
-
mac_address = ether_addr_to_u64(netdev->dev_addr);
rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
@@ -1360,12 +1411,12 @@ static int ibmveth_open(struct net_device *netdev)
rxq_desc.desc,
mac_address);
rc = -ENONET;
- goto out_free_tx_ltb;
+ goto out_free_queue_mem;
}
rc = ibmveth_alloc_buffer_pools(adapter);
if (rc)
- goto out_free_buffer_pools;
+ goto out_unregister_lan;
/* Enables NAPI, so everything ibmveth_poll() touches must exist:
* netpoll during the direct close()/open() pairs, or a direct
@@ -1373,26 +1424,36 @@ static int ibmveth_open(struct net_device *netdev)
*/
rc = ibmveth_setup_rx_interrupts(adapter);
if (rc)
- goto out_free_buffer_pools;
+ goto out_unregister_lan;
netdev_dbg(netdev, "initial replenish cycle\n");
ibmveth_schedule_rx_queue(adapter, 0);
+ rc = ibmveth_alloc_tx_resources(adapter);
+ if (rc)
+ goto out_cleanup_rx_interrupts;
+
+ /* NAPI is live, so generic XDP_TX can transmit as soon as it sees
+ * the queues started. Senders test DRV_XOFF under the xmit lock;
+ * starting the queues under it orders the LTB stores before them.
+ */
+ netif_tx_lock_bh(netdev);
netif_tx_start_all_queues(netdev);
+ netif_tx_unlock_bh(netdev);
adapter->opened = true;
netdev_dbg(netdev, "open complete\n");
return 0;
-out_free_buffer_pools:
+out_cleanup_rx_interrupts:
+ ibmveth_cleanup_rx_interrupts(adapter);
+out_unregister_lan:
do {
lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
ibmveth_free_buffer_pools(adapter);
-out_free_tx_ltb:
- for (i = netdev->real_num_tx_queues - 1; i >= 0; i--)
- ibmveth_free_tx_ltb(adapter, i);
+out_free_queue_mem:
ibmveth_cleanup_rx_resources(adapter);
out_free_filter_list:
ibmveth_free_filter_list(adapter);
@@ -1404,7 +1465,6 @@ static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
long lpar_rc;
- int i;
/* change_mtu, pool sysfs, set_csum and set_tso call close() and
* open() directly. If that open() fails, IFF_UP stays set and
@@ -1430,13 +1490,12 @@ static int ibmveth_close(struct net_device *netdev)
lpar_rc);
}
ibmveth_update_rx_no_buffer(adapter);
+ /* netif_tx_disable() above keeps xmit off the TX LTBs. */
+ ibmveth_free_tx_resources(adapter);
ibmveth_free_buffer_pools(adapter);
ibmveth_cleanup_rx_resources(adapter);
ibmveth_free_filter_list(adapter);
- for (i = 0; i < netdev->real_num_tx_queues; i++)
- ibmveth_free_tx_ltb(adapter, i);
-
netdev_dbg(netdev, "close complete\n");
return 0;
@@ -1864,6 +1923,14 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
int i, queue_num = skb_get_queue_mapping(skb);
unsigned long mss = 0;
+ /* TX LTBs are freed on shrink; a sender that picked its queue
+ * before the shrink can get here after the free.
+ */
+ if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
+ netdev->stats.tx_dropped++;
+ goto out;
+ }
+
if (ibmveth_is_packet_unsupported(skb, netdev))
goto out;
/* veth can't checksum offload UDP */
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 07/15] ibmveth: Add RX queue register helpers for MQ
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (5 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 06/15] ibmveth: Refactor TX resource allocation in open/close paths Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 08/15] ibmveth: Add queue-aware RX buffer submit helper " Mingming Cao
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
MQ RX changes the RX queue lifecycle from one adapter-level
register/free pair into a mixed model: queue 0 is registered through
h_register_logical_lan*(), queues 1..N through
H_REG_LOGICAL_LAN_QUEUE. The queue-0 control plane that open() and
close() use today has to come out into helpers first, so the
subordinate paths have somewhere to attach when MQ RX is enabled.
Extract logical LAN registration and queue teardown into dedicated
helpers:
ibmveth_register_logical_lan()
ibmveth_register_rx_queues()
ibmveth_free_all_queues()
Runtime stays single-queue here. adapter->multi_queue is still false
until MQ RX is enabled, so queue 0 keeps the legacy
h_register_logical_lan() flow, the with_handle arm is not yet
reachable, and no subordinate queue is registered.
ibmveth_register_logical_lan() registers queue 0 with PHYP. When
multi-queue mode is active it uses h_register_logical_lan_with_handle()
and stores the handle only on H_SUCCESS, so a failed attempt leaves no
stale handle behind.
ibmveth_register_rx_queues() is the open()-side entry point: it builds
queue 0's buffer descriptor, records queue 0's virq in queue_irq[0],
masks that IRQ before registration, and calls
ibmveth_register_logical_lan(). It registers queue 0 only.
ibmveth_free_all_queues() issues one H_FREE_LOGICAL_LAN and clears all
queue handles. One hypercall is enough because H_FREE_LOGICAL_LAN is a
full teardown: per PAPR/PHYP it drops the primary LAN and any
subordinate queues registered under it. Close and the open-fail unwind
both rely on that. Incremental scale-down cannot, and issues
H_FREE_LOGICAL_LAN_QUEUE per queue instead; that path arrives with
resize.
Handle 0 is a live PHYP cookie. queue_registered[] is the slot-is-live
flag; do not treat a zero handle as unregistered. Single-queue
registration returns no handle and leaves queue_registered[0] clear.
open() now allocates buffer pools before PHYP registration
(net-next registered first), so a pool-fail
path never has a LAN this kernel registered. A LAN left registered
across a kexec is a different matter: the pool failure used to come
after registration and free it, so issue one H_FREE_LOGICAL_LAN on
that path, ignoring its result (H_PARAMETER when nothing is
registered). Post-register errors still unwind through
ibmveth_free_all_queues().
open() also calls netif_set_real_num_rx_queues() with num_rx_queues
after registration (one queue until multi-queue is enabled); a failure
there unwinds the same way.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- queue_registered[] marks a live slot; handle 0 can be a valid cookie
- a buffer pool allocation failure in open() still calls
H_FREE_LOGICAL_LAN once, for a LAN a kexec'd kernel left registered
- ibmveth_register_logical_lan() returns long, the hypercall status
type
Changes in v7:
- no P07 code change
- commit message: name netif_set_real_num_rx_queues()
in open()
Changes in v6:
- drop the eight hypercall counters, struct ibmveth_hcall_stats, and
the adapter field. Only reg_lan and free_lan had writers here;
none of the eight had a reader. The ethtool -S keys v5 added in
patch 11 are gone with them. This patch no longer touches ibmveth.h
- trace LAN registration and the free retry with netdev_dbg() (open/
close/resize, not per packet). Teardown already uses netdev_err()
- document free_all_queues() log-and-continue in its kdoc, including
that queue_handle[] is cleared whatever the hypercall returned
- noted: queue 0 handle is stored only on H_SUCCESS
- noted: body drops the out_unregister_queues walk and the
unconditional PHYP-vs-pools guarantee
Changes in v5:
- Document H_FREE_LOGICAL_LAN full-teardown contract in kdoc/changelog:
one free_lan drops primary + any subordinate queues
(H_FREE_LOGICAL_LAN_QUEUE is for incremental scale-down only) - v4
helper text did not spell the subordinate semantics
- Call out close() update_rx_no_buffer() before free_lan as intentional
last-glimpse accounting while the LAN is still registered (order
already in v4; v4 step list omitted it); NULL-safe per-queue form
lands with MQ enablement
- opened / rx_irq_setup idempotent close is owned by the IRQ-helper
patch (same failed-reopen / second napi_disable hang raised here)
- Poll re-arm during teardown not claimed here (same race as IRQ patch;
lands with poll harden)
Changes in v4:
- Introduce register/free helpers in the same patch that wires their
first open/close callers; keep subordinate-only helpers deferred
until MQ enablement.
- Introduce adapter->hcall_stats here (first use); not in patch 2.
- Correct open/close unwind so free_all_queues() precedes
free_buffer_pools().
- Drop the orphaned big-bang "open/close pipeline" patch from v3; that
wiring is incremental across helper patches 3-7 instead.
drivers/net/ethernet/ibm/ibmveth.c | 201 ++++++++++++++++++++++-------
drivers/net/ethernet/ibm/ibmveth.h | 4 +
2 files changed, 159 insertions(+), 46 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index cab895cc970c..e2aa10865567 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1335,10 +1335,31 @@ static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)
ibmveth_free_tx_ltb(adapter, i);
}
-static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
- union ibmveth_buf_desc rxq_desc, u64 mac_address)
+static void ibmveth_set_rx_queue_handle(struct ibmveth_adapter *adapter,
+ int queue_index, u64 handle)
{
- int rc, try_again = 1;
+ /* PHYP handle; 0 can be valid. queue_registered[] is the sentinel. */
+ adapter->queue_handle[queue_index] = handle;
+ adapter->queue_registered[queue_index] = true;
+}
+
+static void ibmveth_clear_rx_queue_handle(struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ adapter->queue_registered[queue_index] = false;
+ adapter->queue_handle[queue_index] = 0;
+}
+
+static long ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
+ union ibmveth_buf_desc rxq_desc,
+ u64 mac_address)
+{
+ long rc;
+ int try_again = 1;
+ unsigned long ua = adapter->vdev->unit_address;
+ unsigned long buf_dma = adapter->buffer_list_dma[0];
+ unsigned long filter_dma = adapter->filter_list_dma;
+ unsigned long qh0;
/*
* After a kexec the adapter will still be open, so our attempt to
@@ -1346,14 +1367,29 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
* try again, but only once.
*/
retry:
- rc = h_register_logical_lan(adapter->vdev->unit_address,
- adapter->buffer_list_dma[0], rxq_desc.desc,
- adapter->filter_list_dma, mac_address);
+ /* In multi-queue mode, obtain a queue handle for queue 0 so all RX
+ * queues can use the same per-queue buffer hypercalls.
+ */
+ if (adapter->multi_queue) {
+ rc = h_register_logical_lan_with_handle(ua, buf_dma,
+ rxq_desc.desc,
+ filter_dma,
+ mac_address,
+ &qh0);
+ if (rc == H_SUCCESS)
+ ibmveth_set_rx_queue_handle(adapter, 0, qh0);
+ } else {
+ rc = h_register_logical_lan(ua, buf_dma, rxq_desc.desc,
+ filter_dma, mac_address);
+ }
+ netdev_dbg(adapter->netdev, "h_register_logical_lan%s rc=%ld\n",
+ adapter->multi_queue ? "_with_handle" : "", rc);
if (rc != H_SUCCESS && try_again) {
do {
rc = h_free_logical_lan(adapter->vdev->unit_address);
} while (H_IS_LONG_BUSY(rc) || (rc == H_BUSY));
+ netdev_dbg(adapter->netdev, "h_free_logical_lan rc=%ld\n", rc);
try_again = 0;
goto retry;
@@ -1362,14 +1398,97 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
return rc;
}
+/**
+ * ibmveth_free_all_queues - Free all RX queues at once
+ * @adapter: ibmveth adapter structure
+ *
+ * Issues one H_FREE_LOGICAL_LAN for full adapter teardown. Per PAPR/PHYP,
+ * that drops the primary LAN and any subordinate queues registered under
+ * it. Incremental scale-down uses H_FREE_LOGICAL_LAN_QUEUE per queue
+ * instead; do not use this helper for partial live-set shrink.
+ *
+ * Used by close() and the post-register open() unwind.
+ *
+ * Retries only H_BUSY and H_IS_LONG_BUSY. On other failures, logs and
+ * returns; callers cannot observe hypercall status. Registered slots
+ * are cleared regardless. Callers still run RX pool and DMA teardown
+ * afterward (same as pre-helper close()).
+ *
+ * Clears registered flags and cookies of queues [0, num_rx_queues);
+ * queue_irq[] is released by ibmveth_cleanup_rx_interrupts().
+ */
+static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
+{
+ unsigned long lpar_rc;
+ int i;
+
+ netdev_dbg(adapter->netdev, "freeing all RX queues at once\n");
+
+ do {
+ lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan failed: %ld\n", lpar_rc);
+ }
+
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ ibmveth_clear_rx_queue_handle(adapter, i);
+}
+
+/**
+ * ibmveth_register_rx_queues - Register RX queues with hypervisor
+ * @adapter: ibmveth adapter structure
+ * @mac_address: MAC address for device registration
+ *
+ * Registers queue 0 via ibmveth_register_logical_lan(). Subordinate queue
+ * registration is added when multi-queue RX is enabled.
+ *
+ * Return: 0 on success, -ENONET if queue 0 registration fails
+ */
+static int
+ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
+{
+ struct net_device *netdev = adapter->netdev;
+ union ibmveth_buf_desc rxq_desc;
+ unsigned long lpar_rc;
+ int rc;
+
+ rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
+ adapter->rx_queue[0].queue_len;
+ rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
+ adapter->queue_irq[0] = netdev->irq;
+
+ rc = ibmveth_disable_irq(adapter, 0);
+ if (rc)
+ netdev_dbg(netdev,
+ "Failed to disable IRQ for queue 0 before registration, rc=%d\n",
+ rc);
+
+ lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(netdev,
+ "h_register_logical_lan failed: %ld\n", lpar_rc);
+ netdev_err(netdev,
+ "buffer TCE:0x%llx filter TCE:0x%llx rxq desc:0x%llx MAC:0x%llx\n",
+ adapter->buffer_list_dma[0],
+ adapter->filter_list_dma,
+ rxq_desc.desc, mac_address);
+ return -ENONET;
+ }
+
+ netdev_dbg(netdev,
+ "registered 1 RX queue with hypervisor (single-queue mode)\n");
+ return 0;
+}
+
static int ibmveth_open(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- u64 mac_address;
+ u64 mac_address = ether_addr_to_u64(netdev->dev_addr);
int rxq_entries = 1;
- unsigned long lpar_rc;
int rc;
- union ibmveth_buf_desc rxq_desc;
int i;
netdev_dbg(netdev, "open starting\n");
@@ -1390,33 +1509,30 @@ static int ibmveth_open(struct net_device *netdev)
if (rc)
goto out_free_filter_list;
- mac_address = ether_addr_to_u64(netdev->dev_addr);
-
- rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
- adapter->rx_queue[0].queue_len;
- rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
-
- adapter->queue_irq[0] = netdev->irq;
- ibmveth_disable_irq(adapter, 0);
-
- lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
+ rc = ibmveth_alloc_buffer_pools(adapter);
+ if (rc) {
+ unsigned long ua = adapter->vdev->unit_address;
+ unsigned long lpar_rc;
- if (lpar_rc != H_SUCCESS) {
- netdev_err(netdev, "h_register_logical_lan failed with %ld\n",
- lpar_rc);
- netdev_err(netdev, "buffer TCE:0x%llx filter TCE:0x%llx rxq "
- "desc:0x%llx MAC:0x%llx\n",
- adapter->buffer_list_dma[0],
- adapter->filter_list_dma,
- rxq_desc.desc,
- mac_address);
- rc = -ENONET;
+ /* A kexec can leave the LAN registered by the old kernel
+ * (see ibmveth_register_logical_lan()); do not leave its
+ * buffers live in PHYP. H_PARAMETER if none is registered.
+ */
+ do {
+ lpar_rc = h_free_logical_lan(ua);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
goto out_free_queue_mem;
}
- rc = ibmveth_alloc_buffer_pools(adapter);
+ rc = ibmveth_register_rx_queues(adapter, mac_address);
if (rc)
- goto out_unregister_lan;
+ goto out_free_buffer_pools;
+
+ rc = netif_set_real_num_rx_queues(netdev, adapter->num_rx_queues);
+ if (rc) {
+ netdev_err(netdev, "failed to set number of rx queues\n");
+ goto out_unregister_queues;
+ }
/* Enables NAPI, so everything ibmveth_poll() touches must exist:
* netpoll during the direct close()/open() pairs, or a direct
@@ -1424,7 +1540,7 @@ static int ibmveth_open(struct net_device *netdev)
*/
rc = ibmveth_setup_rx_interrupts(adapter);
if (rc)
- goto out_unregister_lan;
+ goto out_free_all_queues; /* setup already disposed IRQs */
netdev_dbg(netdev, "initial replenish cycle\n");
ibmveth_schedule_rx_queue(adapter, 0);
@@ -1448,10 +1564,12 @@ static int ibmveth_open(struct net_device *netdev)
out_cleanup_rx_interrupts:
ibmveth_cleanup_rx_interrupts(adapter);
-out_unregister_lan:
- do {
- lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
- } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+ goto out_free_all_queues; /* cleanup already disposed IRQs */
+out_unregister_queues:
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+out_free_all_queues:
+ ibmveth_free_all_queues(adapter);
+out_free_buffer_pools:
ibmveth_free_buffer_pools(adapter);
out_free_queue_mem:
ibmveth_cleanup_rx_resources(adapter);
@@ -1464,7 +1582,6 @@ static int ibmveth_open(struct net_device *netdev)
static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- long lpar_rc;
/* change_mtu, pool sysfs, set_csum and set_tso call close() and
* open() directly. If that open() fails, IFF_UP stays set and
@@ -1480,15 +1597,7 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_cleanup_rx_interrupts(adapter);
- /* Full LAN teardown (subordinates arrive with register helpers). */
- do {
- lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
- } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
- if (lpar_rc != H_SUCCESS) {
- netdev_err(adapter->netdev,
- "h_free_logical_lan failed with %lx, continuing\n",
- lpar_rc);
- }
+ ibmveth_free_all_queues(adapter);
ibmveth_update_rx_no_buffer(adapter);
/* netif_tx_disable() above keeps xmit off the TX LTBs. */
ibmveth_free_tx_resources(adapter);
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 50be6c95457e..1433f62aaaca 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -321,7 +321,11 @@ struct ibmveth_adapter {
struct ibmveth_buff_pool
rx_buff_pool[IBMVETH_MAX_RX_QUEUES][IBMVETH_NUM_BUFF_POOLS];
struct ibmveth_rx_q rx_queue[IBMVETH_MAX_RX_QUEUES];
+ /* Opaque PHYP cookie; 0 can be live. queue_registered[] is slot-live,
+ * tracked only for multi_queue: SQ registration sets no handle.
+ */
u64 queue_handle[IBMVETH_MAX_RX_QUEUES];
+ bool queue_registered[IBMVETH_MAX_RX_QUEUES];
unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
int multi_queue;
unsigned int num_rx_queues;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 08/15] ibmveth: Add queue-aware RX buffer submit helper for MQ
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (6 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 07/15] ibmveth: Add RX queue register helpers for MQ Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 09/15] ibmveth: Split RX poll into per-slot helpers " Mingming Cao
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
This patch queue-indexes RX replenish and pool access ahead of MQ
enable:
- hcall: ibmveth_add_logical_lan_buffers() (MQ queue hcall vs legacy)
- concurrency: per-queue replenish_lock; replenish outcomes logged
after unlock
- poll/pool: rxq_* take queue_index; INVALID_MAP and correlator
errors refactored
- limits: IBMVETH_MAX_RX_PER_HCALL 12, IBMVETH_MAX_RX_REGULAR 8
Runtime stays single-queue until the MQ enable patch; correlator,
lock and replenish return paths apply on queue 0 immediately.
Replenish is the last open-path hypercall that still assumes queue 0.
Registration is already queue-aware, and multi-queue posts buffers
through H_ADD_LOGICAL_LAN_BUFFERS_QUEUE against adapter->queue_handle[],
but ibmveth_replenish_buffer_pool() still calls
h_add_logical_lan_buffer() or h_add_logical_lan_buffers() directly.
Add ibmveth_add_logical_lan_buffers() to route RX buffer pool
replenishment through H_ADD_LOGICAL_LAN_BUFFERS_QUEUE in multi-queue
mode, falling back to the legacy 8-buffer and single-buffer hypercalls
in single-queue mode.
Supporting that helper means the RX side can no longer assume queue 0.
The queue index is 0 everywhere until multi-queue RX is enabled, so
the queue plumbing below does not change behaviour yet. The correlator
and locking changes do take effect immediately, on the single queue.
Parameterise the RX accessors by queue. The eight ibmveth_rxq_*
helpers took only the adapter and hardcoded rx_queue[0]; they now take
a queue_index, as do ibmveth_remove_buffer_from_pool() and
ibmveth_rxq_get_buffer(). ibmveth_rxq_correlator() becomes
ibmveth_rxq_read_correlator(); poll still reads the correlator once
per slot, as net-next does, and now passes the queue index with it
to rxq_get_buffer() and rxq_harvest_buffer().
Add a per-queue replenish_lock to struct ibmveth_rx_q, initialised in
probe. Replenish is not the only writer of a pool's free_map: the
harvest and remove consumer runs from that queue's NAPI instance. At
this commit replenish also runs only from NAPI (netpoll's budget-0
poll skips it), but later patches (MQ open and the queue resize)
replenish from process context while that queue's NAPI may run, so
producer and consumer can run at once. Its users are NAPI and process
context, so it is taken with spin_lock_bh().
Give ibmveth_replenish_buffer_pool() a return value. It was void and
logged from inside what is now a critical section. It now returns one
of IBMVETH_REPLENISH_OK, _RESET_MAP, _RESET_MQ, _HCALL_FAIL or
_BATCH_FALLBACK, and ibmveth_replenish_task() does the logging and any
schedule_work() after dropping replenish_lock. Replenish map paths use
DMA_ATTR_NO_WARN; failures are counted, and the next poll's replenish
retries. As before this patch, a queue whose pools are left with
nothing posted gets no further poll and waits for a reopen. The
hcall-failure warning now logs against the netdev rather than the vio
device, the two batch-fallback info messages become one ratelimited
warning, and DMA_ATTR_NO_WARN silences the DMA layer's map-failure
warning. In multi-queue mode an H_FUNCTION from the buffer-add hcall
schedules a reset instead of the legacy single-buffer fallback;
that arm is unreachable until MQ RX is enabled.
An IBM_VETH_INVALID_MAP free_map entry no longer WARNs. It now drops
the partly filled batch (those skbs stay in the pool for reuse),
skips the remaining pools and schedules a reset, where the old code
posted the partial batch and moved on to the next pool.
replenish_task() also returns early for a queue index at or past the
live count, and ibmveth_poll() logs an IRQ enable failure before it
schedules the reset.
Update replenish_task_cycles under replenish_lock; the statistics patch
moves that counter per queue.
Snapshot the first BATCH_FALLBACK and HCALL_FAIL into their own
copies and log those after unlock, so a later pool cannot overwrite
the values those two messages print.
Make the correlator check per queue. ibmveth_rxq_correlator_valid()
takes the queue index and also requires the pool to be active with a
free_map. It no longer logs or schedules the reset, because
remove_buffer_from_pool() now calls it under replenish_lock:
ibmveth_rxq_bad_correlator() and ibmveth_rxq_no_skb() do the
ratelimited log and the reset, called by rxq_harvest_buffer() after
remove_buffer_from_pool() returns and by rxq_get_buffer() directly.
The ring still advances on every error. The KUnit tests pass the queue
index and set up free_map and replenish_lock.
Redefine IBMVETH_MAX_RX_PER_HCALL from 8 to 12, the argument-list
capacity of H_ADD_LOGICAL_LAN_BUFFERS_QUEUE, and add
IBMVETH_MAX_RX_REGULAR (8) for the legacy hypercall. Probe sets
rx_buffers_per_hcall to IBMVETH_MAX_RX_REGULAR, so the legacy batch
stays at 8 until multi-queue RX is enabled.
free_buffer_pool() keeps probe/sysfs geometry (active/size/threshold)
and zeroes pool->available so teardown cannot leave a stale count.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- poll reads each RX-ring correlator once (READ_ONCE) and passes it to
lookup and harvest, as net-next does
- harvest logs and resets on a bad correlator or a missing skb
- replenish_lock is spin_lock_bh(); replenish_task_cycles is counted
under it; the first batch-fallback and hcall failures are both kept
for the log
Dropped from this series (merged fixes own them):
- commit 8d9ecac6b96e: the poll_controller() replenish_task(adapter, 0)
change. ndo_poll_controller is gone. poll keeps commit 8d9ecac6b96e's
if (budget) around the per-queue replenish. The commit message drops
that reason, and the DMA_ATTR_NO_WARN, lock-init and
remove_buffer_from_pool comments no longer cite netpoll
- commit c9b95adc755f: the correlator WARN_ON() -> correlator_valid()
conversion and poll's advance on a NULL buffer. The commit message
drops that WARN_ON paragraph
Changes in v7:
- extend remove_buffer_from_pool Return: -EINVAL also
inactive / not live (skbuff or free_map NULL)
- replenish fail log names the wrapper from
filled, not batch
- advance past slot on NULL buffer so poll does not spin
at this commit; P09 skip_bad refactors error
escalation into helper
- noted: irqsave CS and free_buffer_pool vs lock stay
cover leftovers (lock-protocol rewrite, not
two conditions)
Changes in v6:
- drop inline on the queue-index rxq_* accessors (plain static)
- initialise the replenish locks once in probe; open() no longer
reinitialises them in alloc_rx_queues(), where it could reset a
lock another CPU was holding through poll_controller()
- pass DMA_ATTR_NO_WARN to dma_map_single_attrs() in the replenish
loop. The iommu warning could re-enter via netconsole and deadlock
on the replenish_lock this CPU already holds
- static_assert IBMVETH_MAX_RX_REGULAR == 8 and
IBMVETH_MAX_RX_PER_HCALL == 12 against the hand-written
legacy descs[0..7] and MQ ioba[0..5] argument lists
- add clarifying comment on the legacy 8-buffer hcall batch bound
- drop the three buffer-submit hypercall counters along with the rest
of hcall_stats; see patch 7
- fix the KUnit fixtures to allocate a dummy free_map
- noted: irqsave CS and free_buffer_pool vs replenish stay cover
- noted: skip_bad_correlator lands in P09
Changes in v5:
- On MQ buffer-add H_FUNCTION: schedule adapter reset after dropping
replenish_lock (v4 logged/broke with no recovery; can permanently dry
the pool)
- Move replenish fail logging / reset scheduling out from under
replenish_lock so netconsole cannot deadlock re-entering replenish
- Serialize harvest/remove with per-queue replenish_lock (netpoll
replenish vs NAPI consumer; v4 locked producer only)
- Fail logs use real wrapper names: h_add_logical_lan_buffers[_queue] /
h_add_logical_lan_buffer (v4 interpolated broken lan[_queue] strings)
- Document replenish_lock + no-printk-under-lock for netconsole (first
lock use); outcomes enum + ibmveth_replenish_fail defined here
- Defer adapter-global counter atomics and irqsave critical-section
shorten to cover follow-up
- Bad queue_index poll path: napi_complete before return lands with
poll harden (not claimed fully here)
- Keep pool active/size/threshold across free_buffer_pool (probe/sysfs
geometry); only clear runtime allocations + available (ifdown/up
reopen must still see active pools)
- get_buffer: use correlator_valid (drop WARN_ON; keep schedule_work
until poll skip owns reset)
- Introduce ibmveth_rxq_correlator_valid / ibmveth_rxq_advance at first
remove/harvest use; init replenish_lock in remove_buffer KUnit
- On RESET_MAP/RESET_MQ, stop remaining pool walks (goto unlock)
Changes in v4:
- Introduce queue-aware replenish/poll helpers with their first callers
in the same patch; do not leave a 2-arg replenish call ahead of the
signature change.
- Restore the pre-MQ LPM H_FUNCTION break instead of continue; do not
loop forever on a stale local batch size.
- Fold per-queue replenish_lock into this patch.
- Update kdoc for MQ parameters on remove_buffer_from_pool /
rxq_harvest_buffer.
drivers/net/ethernet/ibm/ibmveth.c | 632 +++++++++++++++++++++--------
drivers/net/ethernet/ibm/ibmveth.h | 5 +-
2 files changed, 459 insertions(+), 178 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index e2aa10865567..a1f286b37eff 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -30,6 +30,7 @@
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/slab.h>
+#include <linux/spinlock.h>
#include <asm/hvcall.h>
#include <linux/atomic.h>
#include <asm/vio.h>
@@ -101,57 +102,58 @@ static struct ibmveth_stat ibmveth_stats[] = {
};
/* simple methods of getting data from the current rxq entry */
-static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter)
+static u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
return be32_to_cpu(rxq->queue_addr[rxq->index].flags_off);
}
-static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return (ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_TOGGLE) >>
- IBMVETH_RXQ_TOGGLE_SHIFT;
+ return (ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_TOGGLE) >>
+ IBMVETH_RXQ_TOGGLE_SHIFT;
}
-static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_toggle(adapter) == adapter->rx_queue[0].toggle;
+ return ibmveth_rxq_toggle(adapter, queue_index) ==
+ adapter->rx_queue[queue_index].toggle;
}
-static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_VALID;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_VALID;
}
-static inline int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_OFF_MASK;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_OFF_MASK;
}
-static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_LRG_PKT;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_LRG_PKT;
}
-static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
return be32_to_cpu(rxq->queue_addr[rxq->index].length);
}
-static u64 ibmveth_rxq_correlator(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
- unsigned int idx = rxq->index;
-
- return READ_ONCE(rxq->queue_addr[idx].correlator);
-}
-
-static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter)
-{
- return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_CSUM_GOOD;
+ return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_CSUM_GOOD;
}
static unsigned int ibmveth_real_max_tx_queues(void)
@@ -726,11 +728,103 @@ static inline void ibmveth_flush_buffer(void *addr, unsigned long length)
asm("dcbf %0,%1,1" :: "b" (addr), "r" (offset));
}
-/* replenish the buffers for a pool. note that we don't need to
- * skb_reserve these since they are used for incoming...
+/**
+ * ibmveth_add_logical_lan_buffers - Add receive buffers to hypervisor
+ * @adapter: ibmveth adapter structure
+ * @descs: array of buffer descriptors to add
+ * @filled: number of valid descriptors in the array
+ * @buff_size: size of each buffer (multi-queue mode only)
+ * @queue_index: RX queue index
+ *
+ * Return: hypervisor return code
+ */
+static long ibmveth_add_logical_lan_buffers(struct ibmveth_adapter *adapter,
+ union ibmveth_buf_desc *descs,
+ int filled,
+ unsigned long buff_size,
+ int queue_index)
+{
+ struct vio_dev *vdev = adapter->vdev;
+ unsigned long rc;
+
+ /*
+ * The MQ hcall takes six ioba words (12 packed addresses). The
+ * legacy hcall takes eight descriptors. The argument lists below
+ * are written out by hand; keep the defines matched to those lists.
+ */
+ static_assert(IBMVETH_MAX_RX_PER_HCALL == 12);
+ static_assert(IBMVETH_MAX_RX_REGULAR == 8);
+
+ if (adapter->multi_queue) {
+ unsigned long buffersznum = (buff_size << 32) | filled;
+ unsigned long ioba[IBMVETH_MAX_RX_PER_HCALL / 2] = {0};
+ unsigned long handle = adapter->queue_handle[queue_index];
+ int i;
+
+ /* Pack descriptor addresses into ioba pairs.
+ * Each ioba holds two 32-bit addresses packed into 64 bits:
+ * - Even descriptors (0,2,4...) go in high 32 bits
+ * - Odd descriptors (1,3,5...) go in low 32 bits
+ */
+ for (i = 0; i < filled && i < IBMVETH_MAX_RX_PER_HCALL; i++) {
+ int pair_idx = i / 2;
+ int is_high = (i % 2 == 0);
+
+ if (is_high)
+ ioba[pair_idx] = (unsigned long)
+ descs[i].fields.address << 32;
+ else
+ ioba[pair_idx] |= descs[i].fields.address;
+ }
+
+ rc = h_add_logical_lan_buffers_queue(vdev->unit_address,
+ handle,
+ buffersznum,
+ ioba[0], ioba[1], ioba[2],
+ ioba[3], ioba[4], ioba[5]);
+ } else if (filled == 1) {
+ rc = h_add_logical_lan_buffer(vdev->unit_address,
+ descs[0].desc);
+ } else {
+ /* Legacy 8-desc hcall; probe/mq_fallback keep batch <=
+ * IBMVETH_MAX_RX_REGULAR.
+ */
+ rc = h_add_logical_lan_buffers(vdev->unit_address,
+ descs[0].desc, descs[1].desc,
+ descs[2].desc, descs[3].desc,
+ descs[4].desc, descs[5].desc,
+ descs[6].desc, descs[7].desc);
+ }
+
+ return rc;
+}
+
+/* Outcomes for ibmveth_replenish_buffer_pool(); logged after unlock. */
+enum {
+ IBMVETH_REPLENISH_OK = 0,
+ IBMVETH_REPLENISH_RESET_MAP,
+ IBMVETH_REPLENISH_RESET_MQ,
+ IBMVETH_REPLENISH_HCALL_FAIL,
+ IBMVETH_REPLENISH_BATCH_FALLBACK,
+};
+
+struct ibmveth_replenish_fail {
+ unsigned long lpar_rc;
+ u32 filled;
+ u32 batch;
+};
+
+/* Replenish the buffers for a pool. The skbs are not skb_reserve()d:
+ * PHYP reports the offset of each received frame.
+ * Caller must hold the per-queue replenish_lock.
+ * Do not log here: failures are returned for replenish_task to log
+ * after it drops the lock, so console output does not extend that
+ * section. A failed GFP_ATOMIC skb allocation can still warn.
*/
-static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
- struct ibmveth_buff_pool *pool)
+static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
+ struct ibmveth_buff_pool *pool,
+ int queue_index,
+ struct ibmveth_replenish_fail *fail)
{
union ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};
u32 remaining = pool->size - atomic_read(&pool->available);
@@ -742,6 +836,7 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
dma_addr_t dma_addr;
struct device *dev;
u32 index;
+ int outcome = IBMVETH_REPLENISH_OK;
vdev = adapter->vdev;
dev = &vdev->dev;
@@ -756,12 +851,9 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
/* Fill a batch of descriptors */
for (filled = 0; filled < min(remaining, batch); filled++) {
index = pool->free_map[free_index];
- if (WARN_ON(index == IBM_VETH_INVALID_MAP)) {
+ if (index == IBM_VETH_INVALID_MAP) {
adapter->replenish_add_buff_failure++;
- netdev_info(adapter->netdev,
- "Invalid map index %u, reset\n",
- index);
- schedule_work(&adapter->work);
+ outcome = IBMVETH_REPLENISH_RESET_MAP;
break;
}
@@ -776,9 +868,16 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
break;
}
- dma_addr = dma_map_single(dev, skb->data,
- pool->buff_size,
- DMA_FROM_DEVICE);
+ /* NO_WARN: a map failure under TCE pressure
+ * is expected; it is counted below and the
+ * next poll's replenish retries.
+ */
+ dma_addr =
+ dma_map_single_attrs(dev,
+ skb->data,
+ pool->buff_size,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_NO_WARN);
if (dma_mapping_error(dev, dma_addr)) {
dev_kfree_skb_any(skb);
adapter->replenish_add_buff_failure++;
@@ -813,28 +912,21 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
free_index = 0;
}
+ if (outcome != IBMVETH_REPLENISH_OK)
+ break;
+
if (!filled)
break;
- /* single buffer case*/
- if (filled == 1)
- lpar_rc = h_add_logical_lan_buffer(vdev->unit_address,
- descs[0].desc);
- else
- /* Multi-buffer hcall */
- lpar_rc = h_add_logical_lan_buffers(vdev->unit_address,
- descs[0].desc,
- descs[1].desc,
- descs[2].desc,
- descs[3].desc,
- descs[4].desc,
- descs[5].desc,
- descs[6].desc,
- descs[7].desc);
+ lpar_rc = ibmveth_add_logical_lan_buffers(adapter, descs,
+ filled,
+ pool->buff_size,
+ queue_index);
+
if (lpar_rc != H_SUCCESS) {
- dev_warn_ratelimited(dev,
- "RX h_add_logical_lan failed: filled=%u, rc=%lu, batch=%u\n",
- filled, lpar_rc, batch);
+ fail->lpar_rc = lpar_rc;
+ fail->filled = filled;
+ fail->batch = batch;
goto hcall_failure;
}
@@ -874,30 +966,35 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
}
adapter->replenish_add_buff_failure += filled;
- /*
- * If multi rx buffers hcall is no longer supported by FW
- * e.g. in the case of Live Partition Migration
- */
- if (batch > 1 && lpar_rc == H_FUNCTION) {
- /*
- * Instead of retry submit single buffer individually
- * here just set the max rx buffer per hcall to 1
- * buffers will be respleshed next time
- * when ibmveth_replenish_buffer_pool() is called again
- * with single-buffer case
- */
- netdev_info(adapter->netdev,
- "RX Multi buffers not supported by FW, rc=%lu\n",
- lpar_rc);
- adapter->rx_buffers_per_hcall = 1;
- netdev_info(adapter->netdev,
- "Next rx replesh will fall back to single-buffer hcall\n");
+ if (lpar_rc == H_FUNCTION) {
+ if (adapter->multi_queue) {
+ /*
+ * LPM / firmware may drop MQ buffer hcalls.
+ * Schedule reset so we do not sit forever in
+ * no-buffer with the link still up.
+ */
+ outcome = IBMVETH_REPLENISH_RESET_MQ;
+ } else if (batch > 1) {
+ /*
+ * Live Partition Migration may drop multi-
+ * buffer support. Fall back to single-buffer
+ * on the next replenish; do not continue with
+ * a stale local batch size (infinite loop).
+ */
+ adapter->rx_buffers_per_hcall = 1;
+ outcome = IBMVETH_REPLENISH_BATCH_FALLBACK;
+ } else {
+ outcome = IBMVETH_REPLENISH_HCALL_FAIL;
+ }
+ } else {
+ outcome = IBMVETH_REPLENISH_HCALL_FAIL;
}
break;
}
mb();
atomic_add(buffers_added, &(pool->available));
+ return outcome;
}
/*
@@ -917,21 +1014,91 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
}
/* replenish routine */
-static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)
+static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
+ int queue_index)
{
- int i;
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+ struct ibmveth_replenish_fail fail = {};
+ struct ibmveth_replenish_fail fail_fallback = {};
+ struct ibmveth_replenish_fail fail_hcall = {};
+ int i, rc;
+ int need_reset = 0;
+ int batch_fallback = 0;
+ int hcall_fail = 0;
+
+ if (queue_index >= adapter->num_rx_queues) {
+ netdev_dbg(adapter->netdev,
+ "Skipping replenish for freed queue %d (num_queues=%u)\n",
+ queue_index, adapter->num_rx_queues);
+ return;
+ }
+
+ spin_lock_bh(&rxq->replenish_lock);
adapter->replenish_task_cycles++;
for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
- struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[0][i];
-
- if (pool->active &&
- (atomic_read(&pool->available) < pool->threshold))
- ibmveth_replenish_buffer_pool(adapter, pool);
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[queue_index][i];
+
+ if (pool->active && pool->free_map &&
+ (atomic_read(&pool->available) < pool->threshold)) {
+ rc = ibmveth_replenish_buffer_pool(adapter, pool,
+ queue_index, &fail);
+ switch (rc) {
+ case IBMVETH_REPLENISH_RESET_MAP:
+ case IBMVETH_REPLENISH_RESET_MQ:
+ need_reset = rc;
+ goto out_unlock;
+ case IBMVETH_REPLENISH_BATCH_FALLBACK:
+ if (!batch_fallback)
+ fail_fallback = fail;
+ batch_fallback = 1;
+ break;
+ case IBMVETH_REPLENISH_HCALL_FAIL:
+ if (!hcall_fail)
+ fail_hcall = fail;
+ hcall_fail = 1;
+ break;
+ default:
+ break;
+ }
+ }
}
+out_unlock:
ibmveth_update_rx_no_buffer(adapter);
+
+ spin_unlock_bh(&rxq->replenish_lock);
+
+ /* Log and schedule reset only after dropping replenish_lock. */
+ if (need_reset == IBMVETH_REPLENISH_RESET_MAP) {
+ netdev_info(adapter->netdev,
+ "Invalid RX free_map entry on queue %d, reset\n",
+ queue_index);
+ schedule_work(&adapter->work);
+ } else if (need_reset == IBMVETH_REPLENISH_RESET_MQ) {
+ dev_err_ratelimited(&adapter->netdev->dev,
+ "MQ buffer add H_FUNCTION (q=%d, batch=%u), reset\n",
+ queue_index, fail.batch);
+ schedule_work(&adapter->work);
+ }
+
+ if (batch_fallback)
+ dev_warn_ratelimited(&adapter->netdev->dev,
+ "Legacy batch add H_FUNCTION (batch=%u), fallback\n",
+ fail_fallback.batch);
+
+ if (hcall_fail)
+ dev_warn_ratelimited(&adapter->netdev->dev,
+ "RX %s failed: filled=%u, rc=%lu, batch=%u\n",
+ adapter->multi_queue ?
+ "h_add_logical_lan_buffers_queue" :
+ (fail_hcall.filled == 1 ?
+ "h_add_logical_lan_buffer" :
+ "h_add_logical_lan_buffers"),
+ fail_hcall.filled, fail_hcall.lpar_rc,
+ fail_hcall.batch);
}
/* empty and free ana buffer pool - also used to do cleanup in error paths */
@@ -966,27 +1133,25 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
kfree(pool->skbuff);
pool->skbuff = NULL;
}
+
+ /*
+ * Keep probe/sysfs geometry (active, size, buff_size, threshold).
+ * Only tear down runtime allocations; open reuses active pools.
+ */
+ atomic_set(&pool->available, 0);
}
-/* The correlator comes back from PHYP; a bad one schedules a reset. */
-static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
- u64 correlator)
+/* A bad correlator loses the pool slot behind it; schedule a reset.
+ * Logs, so call it only after dropping replenish_lock.
+ */
+static void ibmveth_rxq_bad_correlator(struct ibmveth_adapter *adapter,
+ u64 correlator)
{
- unsigned int index = correlator & 0xffffffffUL;
- unsigned int pool = correlator >> 32;
-
- /* An inactive pool keeps its size but has no skbuff array. */
- if (pool < IBMVETH_NUM_BUFF_POOLS &&
- index < adapter->rx_buff_pool[0][pool].size &&
- adapter->rx_buff_pool[0][pool].skbuff)
- return true;
-
if (net_ratelimit())
netdev_err(adapter->netdev,
"invalid RX correlator %llx, resetting\n",
correlator);
schedule_work(&adapter->work);
- return false;
}
static void ibmveth_rxq_no_skb(struct ibmveth_adapter *adapter,
@@ -1133,33 +1298,77 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
adapter->num_rx_queues);
}
+/* The correlator comes back from PHYP, so check it rather than trust it.
+ * Does not log: remove_buffer_from_pool() calls it under replenish_lock.
+ */
+static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
+ int queue_index, u64 correlator)
+{
+ unsigned int pool = correlator >> 32;
+ unsigned int index = correlator & 0xffffffffUL;
+ struct ibmveth_buff_pool *bpool;
+
+ if (pool >= IBMVETH_NUM_BUFF_POOLS)
+ return false;
+
+ bpool = &adapter->rx_buff_pool[queue_index][pool];
+
+ /* Require a live pool with allocated arrays before indexing.
+ * Inactive pools still have size from init; free clears skbuff.
+ */
+ if (!bpool->active || !bpool->skbuff || !bpool->free_map)
+ return false;
+
+ return index < bpool->size;
+}
+
+static void ibmveth_rxq_advance(struct ibmveth_rx_q *rxq)
+{
+ if (++rxq->index == rxq->num_slots) {
+ rxq->index = 0;
+ rxq->toggle = !rxq->toggle;
+ }
+}
+
/**
* ibmveth_remove_buffer_from_pool - remove a buffer from a pool
* @adapter: adapter instance
* @correlator: identifies pool and index
+ * @queue_index: RX queue index (0..num_rx_queues-1)
* @reuse: whether to reuse buffer
*
+ * Context: may run concurrently with a process-context replenish_task
+ * on the same queue; takes per-queue replenish_lock to serialize
+ * free_map / producer_index / available against the producer.
+ *
* Return:
* * %0 - success
- * * %-EINVAL - correlator maps to pool or index out of range, or to an
- * inactive pool
+ * * %-EINVAL - pool or index out of range, or the pool is
+ * inactive / not live (skbuff or free_map NULL)
* * %-EFAULT - pool and index map to null skb
*/
static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
- u64 correlator, bool reuse)
+ u64 correlator, int queue_index,
+ bool reuse)
{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
unsigned int pool = correlator >> 32;
unsigned int index = correlator & 0xffffffffUL;
unsigned int free_index;
struct sk_buff *skb;
+ int rc = 0;
- if (!ibmveth_rxq_correlator_valid(adapter, correlator))
- return -EINVAL;
+ spin_lock_bh(&rxq->replenish_lock);
- skb = adapter->rx_buff_pool[0][pool].skbuff[index];
+ if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
+ rc = -EINVAL;
+ goto out_unlock;
+ }
+
+ skb = adapter->rx_buff_pool[queue_index][pool].skbuff[index];
if (!skb) {
- ibmveth_rxq_no_skb(adapter, correlator);
- return -EFAULT;
+ rc = -EFAULT;
+ goto out_unlock;
}
/* if we are going to reuse the buffer then keep the pointers around
@@ -1170,78 +1379,95 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
/* remove the skb pointer to mark free. actual freeing is done
* by upper level networking after gro_receive
*/
- adapter->rx_buff_pool[0][pool].skbuff[index] = NULL;
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[queue_index][pool];
+
+ bpool->skbuff[index] = NULL;
dma_unmap_single(&adapter->vdev->dev,
- adapter->rx_buff_pool[0][pool].dma_addr[index],
- adapter->rx_buff_pool[0][pool].buff_size,
+ bpool->dma_addr[index],
+ bpool->buff_size,
DMA_FROM_DEVICE);
}
- free_index = adapter->rx_buff_pool[0][pool].producer_index;
- adapter->rx_buff_pool[0][pool].producer_index++;
- if (adapter->rx_buff_pool[0][pool].producer_index >=
- adapter->rx_buff_pool[0][pool].size)
- adapter->rx_buff_pool[0][pool].producer_index = 0;
- adapter->rx_buff_pool[0][pool].free_map[free_index] = index;
+ free_index = adapter->rx_buff_pool[queue_index][pool].producer_index;
+ adapter->rx_buff_pool[queue_index][pool].producer_index++;
+ if (adapter->rx_buff_pool[queue_index][pool].producer_index >=
+ adapter->rx_buff_pool[queue_index][pool].size)
+ adapter->rx_buff_pool[queue_index][pool].producer_index = 0;
+ adapter->rx_buff_pool[queue_index][pool].free_map[free_index] = index;
mb();
- atomic_dec(&adapter->rx_buff_pool[0][pool].available);
+ atomic_dec(&adapter->rx_buff_pool[queue_index][pool].available);
- return 0;
+out_unlock:
+ spin_unlock_bh(&rxq->replenish_lock);
+ return rc;
+}
+
+static u64 ibmveth_rxq_read_correlator(struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+
+ return READ_ONCE(rxq->queue_addr[rxq->index].correlator);
}
/* get the buffer for @correlator, read once from the current rx queue entry */
-static struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
- u64 correlator)
+static struct sk_buff *
+ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
+ int queue_index, u64 correlator)
{
unsigned int pool = correlator >> 32;
unsigned int index = correlator & 0xffffffffUL;
struct sk_buff *skb;
- if (!ibmveth_rxq_correlator_valid(adapter, correlator))
+ if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
+ ibmveth_rxq_bad_correlator(adapter, correlator);
return NULL;
+ }
- skb = adapter->rx_buff_pool[0][pool].skbuff[index];
+ skb = adapter->rx_buff_pool[queue_index][pool].skbuff[index];
if (!skb)
ibmveth_rxq_no_skb(adapter, correlator);
return skb;
}
-static void ibmveth_rxq_advance(struct ibmveth_adapter *adapter)
-{
- struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
-
- if (++rxq->index == rxq->num_slots) {
- rxq->index = 0;
- rxq->toggle = !rxq->toggle;
- }
-}
-
/**
* ibmveth_rxq_harvest_buffer - Harvest buffer from pool
*
- * @adapter: pointer to adapter
+ * @adapter: pointer to adapter
+ * @queue_index: RX queue index to harvest from
* @correlator: correlator of the current rx queue entry, as read by poll
- * @reuse: whether to reuse buffer
+ * @reuse: whether to reuse buffer
*
* Context: called from ibmveth_poll
*
* The ring advances even on error, so poll does not return to a bad
- * slot before the scheduled reset can run.
+ * slot before the scheduled reset can run. The error is still returned:
+ * callers must not treat it as a successful take from the pool (especially
+ * reuse=false, which would hand the SKB to the stack while it remains
+ * pool-owned).
*
* Return:
- * * %0 - success
- * * other - non-zero return from ibmveth_remove_buffer_from_pool
+ * * %0 - buffer removed from pool (or marked for reuse) and ring advanced
+ * * other - non-zero return from ibmveth_remove_buffer_from_pool; ring has
+ * still been advanced for -EINVAL/-EFAULT
*/
static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
- u64 correlator, bool reuse)
+ int queue_index, u64 correlator,
+ bool reuse)
{
int rc;
- rc = ibmveth_remove_buffer_from_pool(adapter, correlator, reuse);
- ibmveth_rxq_advance(adapter);
+ rc = ibmveth_remove_buffer_from_pool(adapter, correlator, queue_index,
+ reuse);
+ ibmveth_rxq_advance(&adapter->rx_queue[queue_index]);
+ if (unlikely(rc == -EFAULT))
+ ibmveth_rxq_no_skb(adapter, correlator);
+ else if (unlikely(rc))
+ ibmveth_rxq_bad_correlator(adapter, correlator);
return rc;
}
@@ -2288,25 +2514,28 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
static int ibmveth_poll(struct napi_struct *napi, int budget)
{
- struct ibmveth_adapter *adapter =
- container_of(napi, struct ibmveth_adapter, napi[0]);
- struct net_device *netdev = adapter->netdev;
+ struct net_device *netdev = napi->dev;
+ struct ibmveth_adapter *adapter = netdev_priv(netdev);
int frames_processed = 0;
+ int queue_index;
u64 correlator;
u16 mss = 0;
int rc;
+ queue_index = napi - adapter->napi;
+
while (frames_processed < budget) {
- if (!ibmveth_rxq_pending_buffer(adapter))
+ if (!ibmveth_rxq_pending_buffer(adapter, queue_index))
break;
smp_rmb();
- correlator = ibmveth_rxq_correlator(adapter);
- if (!ibmveth_rxq_buffer_valid(adapter)) {
+ correlator = ibmveth_rxq_read_correlator(adapter, queue_index);
+ if (!ibmveth_rxq_buffer_valid(adapter, queue_index)) {
wmb(); /* suggested by larson1 */
adapter->rx_invalid_buffer++;
netdev_dbg(netdev, "recycling invalid buffer\n");
- rc = ibmveth_rxq_harvest_buffer(adapter, correlator, true);
+ rc = ibmveth_rxq_harvest_buffer(adapter, queue_index,
+ correlator, true);
if (unlikely(rc)) {
netdev->stats.rx_dropped++;
frames_processed++;
@@ -2314,24 +2543,33 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
}
} else {
struct sk_buff *skb, *new_skb;
- int length = ibmveth_rxq_frame_length(adapter);
- int offset = ibmveth_rxq_frame_offset(adapter);
- int csum_good = ibmveth_rxq_csum_good(adapter);
- int lrg_pkt = ibmveth_rxq_large_packet(adapter);
+ int length = ibmveth_rxq_frame_length(adapter,
+ queue_index);
+ int offset = ibmveth_rxq_frame_offset(adapter,
+ queue_index);
+ int csum_good = ibmveth_rxq_csum_good(adapter,
+ queue_index);
+ int lrg_pkt = ibmveth_rxq_large_packet(adapter,
+ queue_index);
unsigned int pool, room, off, len;
+ struct ibmveth_buff_pool *bpool;
+ struct ibmveth_rx_q *rxq;
__sum16 iph_check = 0;
- skb = ibmveth_rxq_get_buffer(adapter, correlator);
+ skb = ibmveth_rxq_get_buffer(adapter, queue_index,
+ correlator);
if (unlikely(!skb)) {
- ibmveth_rxq_advance(adapter);
+ rxq = &adapter->rx_queue[queue_index];
+ ibmveth_rxq_advance(rxq);
netdev->stats.rx_dropped++;
frames_processed++;
continue;
}
pool = correlator >> 32;
+ bpool = &adapter->rx_buff_pool[queue_index][pool];
room = min_t(unsigned int, skb_tailroom(skb),
- adapter->rx_buff_pool[0][pool].buff_size);
+ bpool->buff_size);
off = offset;
len = length;
if (unlikely(len < ETH_HLEN || off >= room ||
@@ -2340,8 +2578,8 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
netdev_err(netdev,
"bad RX frame offset %u length %u (buffer %u), dropping\n",
off, len, room);
- ibmveth_rxq_harvest_buffer(adapter, correlator,
- true);
+ ibmveth_rxq_harvest_buffer(adapter, queue_index,
+ correlator, true);
netdev->stats.rx_dropped++;
frames_processed++;
continue;
@@ -2368,8 +2606,9 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
length);
if (rx_flush)
ibmveth_flush_buffer(skb->data,
- length + offset);
+ length + offset);
rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index,
correlator,
true);
if (unlikely(rc)) {
@@ -2381,6 +2620,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
skb = new_skb;
} else {
rc = ibmveth_rxq_harvest_buffer(adapter,
+ queue_index,
correlator,
false);
if (unlikely(rc)) {
@@ -2428,7 +2668,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
* with a NAPI poll that was already running when it was set up
*/
if (budget)
- ibmveth_replenish_task(adapter);
+ ibmveth_replenish_task(adapter, queue_index);
if (frames_processed == budget)
goto out;
@@ -2439,8 +2679,11 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
/* We think we are done - reenable interrupts,
* then check once more to make sure we are done.
*/
- rc = ibmveth_enable_irq(adapter, 0);
+ rc = ibmveth_enable_irq(adapter, queue_index);
if (rc) {
+ netdev_err(netdev,
+ "Failed to enable IRQ for queue %d (rc=%d), scheduling reset\n",
+ queue_index, rc);
schedule_work(&adapter->work);
goto out;
}
@@ -2448,8 +2691,9 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
/* Mask before queueing, as schedule_rx_queue() does: a threaded
* NAPI can start polling, and unmask, as soon as it is queued.
*/
- if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule_prep(napi)) {
- ibmveth_disable_irq(adapter, 0);
+ if (ibmveth_rxq_pending_buffer(adapter, queue_index) &&
+ napi_schedule_prep(napi)) {
+ ibmveth_disable_irq(adapter, queue_index);
__napi_schedule(napi);
}
@@ -2758,6 +3002,15 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
adapter->vdev = dev;
adapter->netdev = netdev;
INIT_WORK(&adapter->work, ibmveth_reset);
+
+ /*
+ * Initialise the replenish locks once. open() is re-entered on
+ * MTU and offload changes, so a lock set up there could be
+ * reinitialised while still held.
+ */
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+ spin_lock_init(&adapter->rx_queue[i].replenish_lock);
+
adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
ibmveth_init_link_settings(netdev);
@@ -2799,7 +3052,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
if (ret == H_SUCCESS &&
(ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
- adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_PER_HCALL;
+ adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
netdev_dbg(netdev,
"RX Multi-buffer hcall supported by FW, batch set to %u\n",
adapter->rx_buffers_per_hcall);
@@ -3122,7 +3375,7 @@ static void ibmveth_reset_kunit(struct work_struct *w)
* @test: pointer to kunit structure
*
* Tests the error returns from ibmveth_remove_buffer_from_pool.
- * Each error also logs a ratelimited netdev_err.
+ * Bad correlators return -EINVAL/-EFAULT (no WARN_ON).
*
* Return: void
*/
@@ -3137,6 +3390,8 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
INIT_WORK(&adapter->work, ibmveth_reset_kunit);
+ spin_lock_init(&adapter->rx_queue[0].replenish_lock);
+
/* Set sane values for buffer pools */
for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
@@ -3146,26 +3401,41 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
+ pool->free_map = kunit_kcalloc(test, pool->size, sizeof(u16),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->free_map);
correlator = ((u64)IBMVETH_NUM_BUFF_POOLS << 32) | 0;
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, false));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, true));
correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0][0].size;
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
- KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, false));
+ KUNIT_EXPECT_EQ(test, -EINVAL,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, true));
/* Pool 2 is in range but has no skbuff array, like an inactive pool. */
correlator = ((u64)2 << 32) | 0;
- ret = ibmveth_remove_buffer_from_pool(adapter, correlator, false);
+ ret = ibmveth_remove_buffer_from_pool(adapter, correlator, 0, false);
KUNIT_EXPECT_EQ(test, -EINVAL, ret);
- ret = ibmveth_remove_buffer_from_pool(adapter, correlator, true);
+ ret = ibmveth_remove_buffer_from_pool(adapter, correlator, 0, true);
KUNIT_EXPECT_EQ(test, -EINVAL, ret);
correlator = (u64)0 | 0;
pool->skbuff[0] = NULL;
- KUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
- KUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+ KUNIT_EXPECT_EQ(test, -EFAULT,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, false));
+ KUNIT_EXPECT_EQ(test, -EFAULT,
+ ibmveth_remove_buffer_from_pool(adapter,
+ correlator, 0, true));
flush_work(&adapter->work);
}
@@ -3207,30 +3477,33 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
pool = &adapter->rx_buff_pool[0][0];
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
+ pool->free_map = kunit_kcalloc(test, pool->size, sizeof(u16),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->free_map);
adapter->rx_queue[0].queue_addr[0].correlator =
(u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
- cor = ibmveth_rxq_correlator(adapter);
+ cor = ibmveth_rxq_read_correlator(adapter, 0);
KUNIT_EXPECT_PTR_EQ(test, NULL,
- ibmveth_rxq_get_buffer(adapter, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, cor));
adapter->rx_queue[0].queue_addr[0].correlator =
(u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
- cor = ibmveth_rxq_correlator(adapter);
+ cor = ibmveth_rxq_read_correlator(adapter, 0);
KUNIT_EXPECT_PTR_EQ(test, NULL,
- ibmveth_rxq_get_buffer(adapter, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, cor));
/* Pool 2 is in range but has no skbuff array, like an inactive pool. */
adapter->rx_queue[0].queue_addr[0].correlator = (u64)2 << 32 | 0;
- cor = ibmveth_rxq_correlator(adapter);
+ cor = ibmveth_rxq_read_correlator(adapter, 0);
KUNIT_EXPECT_PTR_EQ(test, NULL,
- ibmveth_rxq_get_buffer(adapter, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, cor));
pool->skbuff[0] = skb;
adapter->rx_queue[0].queue_addr[0].correlator = (u64)0 << 32 | 0;
- cor = ibmveth_rxq_correlator(adapter);
+ cor = ibmveth_rxq_read_correlator(adapter, 0);
KUNIT_EXPECT_PTR_EQ(test, skb,
- ibmveth_rxq_get_buffer(adapter, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, cor));
flush_work(&adapter->work);
}
@@ -3257,6 +3530,8 @@ static void ibmveth_rxq_harvest_buffer_test(struct kunit *test)
INIT_WORK(&adapter->work, ibmveth_reset_kunit);
+ spin_lock_init(&adapter->rx_queue[0].replenish_lock);
+
adapter->rx_queue[0].num_slots = 2;
adapter->rx_queue[0].index = 0;
adapter->rx_queue[0].toggle = 1;
@@ -3275,20 +3550,23 @@ static void ibmveth_rxq_harvest_buffer_test(struct kunit *test)
pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *),
GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
+ pool->free_map = kunit_kcalloc(test, pool->size, sizeof(u16),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->free_map);
/* Slot 0: pool out of range. Slot 1: valid, but no skb. */
adapter->rx_queue[0].queue_addr[0].correlator =
(u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
adapter->rx_queue[0].queue_addr[1].correlator = (u64)0 << 32 | 0;
- cor = ibmveth_rxq_correlator(adapter);
- ret = ibmveth_rxq_harvest_buffer(adapter, cor, true);
+ cor = ibmveth_rxq_read_correlator(adapter, 0);
+ ret = ibmveth_rxq_harvest_buffer(adapter, 0, cor, true);
KUNIT_EXPECT_EQ(test, -EINVAL, ret);
KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue[0].index);
KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue[0].toggle);
- cor = ibmveth_rxq_correlator(adapter);
- ret = ibmveth_rxq_harvest_buffer(adapter, cor, true);
+ cor = ibmveth_rxq_read_correlator(adapter, 0);
+ ret = ibmveth_rxq_harvest_buffer(adapter, 0, cor, true);
KUNIT_EXPECT_EQ(test, -EFAULT, ret);
KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue[0].index);
KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue[0].toggle);
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 1433f62aaaca..bdc8ea7069ec 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -15,6 +15,7 @@
#define _IBMVETH_H
#include <linux/completion.h>
+#include <linux/spinlock_types.h>
/* constants for H_MULTICAST_CTRL */
#define IbmVethMcastReceptionModifyBit 0x80000UL
@@ -271,7 +272,8 @@ static inline long h_illan_attributes(unsigned long unit_address,
#define IBMVETH_DEFAULT_QUEUES 8U
#define IBMVETH_MAX_RX_QUEUES 1U
#define IBMVETH_DEFAULT_RX_QUEUES 1U
-#define IBMVETH_MAX_RX_PER_HCALL 8U
+#define IBMVETH_MAX_RX_REGULAR 8U
+#define IBMVETH_MAX_RX_PER_HCALL 12U
static int pool_size[] = { 512, 1024 * 2, 1024 * 16, 1024 * 32, 1024 * 64 };
static int pool_count[] = { 256, 512, 256, 256, 256 };
@@ -303,6 +305,7 @@ struct ibmveth_rx_q {
dma_addr_t queue_dma;
u32 queue_len;
struct ibmveth_rx_q_entry *queue_addr;
+ spinlock_t replenish_lock; /* per-queue buffer replenish */
};
struct ibmveth_adapter {
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 09/15] ibmveth: Split RX poll into per-slot helpers for MQ
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (7 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 08/15] ibmveth: Add queue-aware RX buffer submit helper " Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 10/15] ibmveth: Enable multi-queue RX receive path Mingming Cao
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
Multi-queue RX needs two things from ibmveth_poll() that
single-queue never did. Resize disables one queue's NAPI while
the device stays up, so poll must stop consuming the ring and
not re-arm PHYP once a disable is pending. A NAPI whose index is
past the queue count must never be polled (napi_disable() keeps
netpoll away from it), so poll warns, completes it and returns
rather than touch a queue that may be gone.
Adding those checks inline would make the NAPI callback a deeply
nested switchyard. Split each slot outcome into a small helper,
moving the checks net-next already has (correlator bound, frame
bound, header checks) with it, so ibmveth_poll() stays a thin
budget loop:
ibmveth_poll_stopping()
ibmveth_poll_recycle_invalid() / skip_bad_correlator()
ibmveth_poll_drop_oversize()
ibmveth_poll_deliver_frame()
ibmveth_poll_bump_invalid()
Not pure motion. ibmveth_poll_bump_invalid() still counts only the
slots PHYP marks invalid, so rx_invalid_buffer keeps its meaning;
bad-correlator and oversize frames count in rx_dropped. Deliver keeps
the frame bound from commit c9b95adc755f ("ibmveth: step past bad RX
correlators instead of spinning or oopsing"): a PHYP
offset+length beyond min(skb_tailroom, pool->buff_size) is dropped;
skb_tailroom is the skb_put bound, the DMA map is pool->buff_size.
A slot skipped for a bad correlator or an oversize frame counts against
the NAPI budget like a delivered frame, so a run of them cannot keep
one poll going. Recycling a slot PHYP marked invalid stays outside the
budget unless its harvest fails, as before.
deliver_frame() and recycle_invalid() each read the RX-ring correlator
once with ibmveth_rxq_read_correlator() (previous patch) and pass that
value through get_buffer(), the room check and harvest, as poll did
before.
mss becomes a local of deliver_frame() rather than living across
ibmveth_poll()'s whole budget loop; it was never read stale, because
ibmveth_rx_mss_helper() only uses it under lrg_pkt, so that part is
scoping hygiene and not a fix.
rxq_get_buffer() becomes a plain lookup on that correlator and no
longer reports. skip_bad_correlator escalates to reset, and
rxq_harvest_correlator() (the previous patch's rxq_harvest_buffer(),
renamed) still logs and resets on a bad correlator, so recycle_invalid
does too. Skipped slots and failed harvests still count in rx_dropped,
and a failed copybreak harvest still frees new_skb. A bad-correlator
or oversize slot does not end the loop: poll advances and continues.
A failed harvest of a frame being delivered does end it, without
consuming budget. The lookup has just found that buffer and nothing
changes the pool before the harvest at this commit, so the check is
defensive; the harvest has already advanced the ring. The poll tail
masks then schedules NAPI like schedule_rx_queue().
Poll stops consuming the ring once the device is down or a
napi_disable() is pending (close today; live resize removes a queue
in a later patch), so frames still pending when the device goes down
are not delivered.
The check after napi_complete_done() narrows a PHYP re-arm during
close; teardown does not rely on it, since cleanup_rx_interrupts()
waits for the poll tail and masks again.
If poll is stopping with the budget used up, it returns the budget
without completing NAPI, which the core still owns at that point.
Runtime is still single-queue: helpers are defined and called from the
existing SQ poll path in this same patch (first use). The next patch
turns on MQ and reuses this loop; that MQ enable work stays in the
following commit.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- bad-correlator and oversize slots count rx_dropped (not
rx_invalid_buffer) and consume budget
- a failed invalid-slot recycle consumes budget, as in net-next
- a poll that stops with its budget used returns budget without
completing NAPI
- rxq_harvest_buffer() is renamed rxq_harvest_correlator() and takes
the correlator poll read
- deliver_frame() adopts net-next's frame bound and header checks
(ETH_HLEN, skb_headlen(), mss helper result)
- deliver_frame() kdoc: 0 means the slot was consumed without delivery
(skipped or dropped); drop a dead flush_work() in the get_buffer
KUnit test
Dropped from this series (merged fixes own them):
- commit c9b95adc755f: duplicate copybreak/NULL-skb hunks and message
paragraphs (in base; deliver_frame keeps the frame bound)
Changes in v7:
- bound oversize against min(skb_tailroom, pool->buff_size);
name that bound in the commit message
- noted: poll_stopping stays mask-first; skip vs budget stays;
cancel_work stays in P10
Changes in v6:
- read the IPv4 header check through skb->data, as mainline does.
v5 used ip_hdr(), which reads the network header offset that
eth_type_trans() does not set on this path
- do not complete NAPI or consume budget when budget is 0. That is
the netpoll case, where v5 completed and returned budget - 1,
underflowing to -1
- noted: the split is not pure motion; the body lists what changed
Changes in v5:
- Wrap poll_stopping return after complete so it stays under 80 cols
- On poll_stopping after the budget loop: return < budget after
napi_complete_done (was frames_processed-1 even when under budget)
- New in v5: peel SQ poll harden + helpers before MQ enable so tip P10
stays bring-up focused (mailed v4 09/14 MQ enable -> tip P10; also
14->15). Kitchen-sink / enable-path poll fixes land here on the live
single-queue path first.
- After napi_complete_done, check poll_stopping again before enable_irq
(avoid re-arm while resize/close waits on napi_disable)
drivers/net/ethernet/ibm/ibmveth.c | 435 +++++++++++++++++------------
1 file changed, 259 insertions(+), 176 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index a1f286b37eff..b12bfa1e485e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1414,33 +1414,29 @@ static u64 ibmveth_rxq_read_correlator(struct ibmveth_adapter *adapter,
return READ_ONCE(rxq->queue_addr[rxq->index].correlator);
}
-/* get the buffer for @correlator, read once from the current rx queue entry */
+/* Get the buffer behind @correlator. Does not report a bad correlator:
+ * the harvest of that slot does, after re-checking it under the lock.
+ */
static struct sk_buff *
ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
int queue_index, u64 correlator)
{
unsigned int pool = correlator >> 32;
unsigned int index = correlator & 0xffffffffUL;
- struct sk_buff *skb;
- if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
- ibmveth_rxq_bad_correlator(adapter, correlator);
+ if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator))
return NULL;
- }
- skb = adapter->rx_buff_pool[queue_index][pool].skbuff[index];
- if (!skb)
- ibmveth_rxq_no_skb(adapter, correlator);
- return skb;
+ return adapter->rx_buff_pool[queue_index][pool].skbuff[index];
}
/**
- * ibmveth_rxq_harvest_buffer - Harvest buffer from pool
+ * ibmveth_rxq_harvest_correlator - Harvest buffer from a known correlator
*
* @adapter: pointer to adapter
* @queue_index: RX queue index to harvest from
- * @correlator: correlator of the current rx queue entry, as read by poll
* @reuse: whether to reuse buffer
+ * @correlator: pool and index already read from this slot
*
* Context: called from ibmveth_poll
*
@@ -1455,15 +1451,16 @@ ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
* * other - non-zero return from ibmveth_remove_buffer_from_pool; ring has
* still been advanced for -EINVAL/-EFAULT
*/
-static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
- int queue_index, u64 correlator,
- bool reuse)
+static int ibmveth_rxq_harvest_correlator(struct ibmveth_adapter *adapter,
+ int queue_index, bool reuse,
+ u64 correlator)
{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
int rc;
rc = ibmveth_remove_buffer_from_pool(adapter, correlator, queue_index,
reuse);
- ibmveth_rxq_advance(&adapter->rx_queue[queue_index]);
+ ibmveth_rxq_advance(rxq);
if (unlikely(rc == -EFAULT))
ibmveth_rxq_no_skb(adapter, correlator);
else if (unlikely(rc))
@@ -2512,154 +2509,226 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
skb->ip_summed = CHECKSUM_NONE;
}
+static void ibmveth_poll_bump_invalid(struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ adapter->rx_invalid_buffer++;
+}
+
+static bool ibmveth_poll_stopping(struct net_device *netdev,
+ struct napi_struct *napi)
+{
+ return !netif_running(netdev) || napi_disable_pending(napi);
+}
+
+/* Return: true if the harvest failed; that slot then consumes budget. */
+static bool ibmveth_poll_recycle_invalid(struct net_device *netdev,
+ struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ u64 correlator = ibmveth_rxq_read_correlator(adapter, queue_index);
+ int rc;
+
+ netdev_dbg(netdev, "recycling invalid buffer\n");
+ ibmveth_poll_bump_invalid(adapter, queue_index);
+ rc = ibmveth_rxq_harvest_correlator(adapter, queue_index, true,
+ correlator);
+ if (unlikely(rc))
+ netdev->stats.rx_dropped++;
+ return rc;
+}
+
+static void ibmveth_poll_skip_bad_correlator(struct net_device *netdev,
+ struct ibmveth_adapter *adapter,
+ int queue_index, u64 correlator)
+{
+ /* Stale or corrupt slot: recover via reset rather than spinning
+ * forever. Always escalate, even if the harvest below succeeds;
+ * the harvest logs the correlator when it fails.
+ */
+ schedule_work(&adapter->work);
+ netdev->stats.rx_dropped++;
+ ibmveth_rxq_harvest_correlator(adapter, queue_index, true, correlator);
+}
+
+static void ibmveth_poll_drop_oversize(struct net_device *netdev,
+ struct ibmveth_adapter *adapter,
+ int queue_index, unsigned int off,
+ unsigned int len, unsigned int room,
+ u64 correlator)
+{
+ if (net_ratelimit())
+ netdev_err(netdev,
+ "RX frame %u+%u exceeds buffer %u on queue %d, dropping\n",
+ off, len, room, queue_index);
+ netdev->stats.rx_dropped++;
+ ibmveth_rxq_harvest_correlator(adapter, queue_index, true, correlator);
+}
+
+/**
+ * ibmveth_poll_deliver_frame - Build SKB from one valid RX slot and GRO it
+ * @napi: NAPI context for this RX queue
+ * @adapter: ibmveth adapter
+ * @netdev: net_device for @adapter
+ * @queue_index: RX queue index
+ *
+ * Return: 1 frame delivered, 0 if slot consumed without delivery (skipped
+ * or dropped), -1 on error.
+ */
+static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
+ struct ibmveth_adapter *adapter,
+ struct net_device *netdev,
+ int queue_index)
+{
+ struct sk_buff *skb, *new_skb;
+ unsigned int room, off, len, pool;
+ struct ibmveth_buff_pool *bpool;
+ int length, offset, csum_good, lrg_pkt;
+ __sum16 iph_check = 0;
+ u16 mss = 0;
+ u64 correlator;
+ int rc;
+
+ length = ibmveth_rxq_frame_length(adapter, queue_index);
+ offset = ibmveth_rxq_frame_offset(adapter, queue_index);
+ csum_good = ibmveth_rxq_csum_good(adapter, queue_index);
+ lrg_pkt = ibmveth_rxq_large_packet(adapter, queue_index);
+
+ correlator = ibmveth_rxq_read_correlator(adapter, queue_index);
+ skb = ibmveth_rxq_get_buffer(adapter, queue_index, correlator);
+ if (unlikely(!skb)) {
+ ibmveth_poll_skip_bad_correlator(netdev, adapter, queue_index,
+ correlator);
+ return 0;
+ }
+
+ pool = correlator >> 32;
+ bpool = &adapter->rx_buff_pool[queue_index][pool];
+ room = min_t(unsigned int, skb_tailroom(skb),
+ bpool->buff_size);
+ off = offset;
+ len = length;
+ if (unlikely(len < ETH_HLEN || off >= room ||
+ len > room - off)) {
+ ibmveth_poll_drop_oversize(netdev, adapter, queue_index,
+ off, len, room, correlator);
+ return 0;
+ }
+
+ if (lrg_pkt) {
+ /* if the large packet bit is set in the rx queue
+ * descriptor, the mss will be written by PHYP eight
+ * bytes from the start of the rx buffer, which is
+ * skb->data at this stage
+ */
+ __be64 *rxmss = (__be64 *)(skb->data + 8);
+
+ mss = (u16)be64_to_cpu(*rxmss);
+ }
+
+ new_skb = NULL;
+ if (length < rx_copybreak)
+ new_skb = netdev_alloc_skb(netdev, length);
+
+ if (new_skb) {
+ skb_copy_to_linear_data(new_skb, skb->data + offset, length);
+ if (rx_flush)
+ ibmveth_flush_buffer(skb->data, length + offset);
+ rc = ibmveth_rxq_harvest_correlator(adapter, queue_index, true,
+ correlator);
+ if (unlikely(rc)) {
+ dev_kfree_skb_any(new_skb);
+ netdev->stats.rx_dropped++;
+ return -1;
+ }
+ skb = new_skb;
+ } else {
+ rc = ibmveth_rxq_harvest_correlator(adapter, queue_index,
+ false, correlator);
+ if (unlikely(rc)) {
+ netdev->stats.rx_dropped++;
+ return -1;
+ }
+ skb_reserve(skb, offset);
+ }
+
+ skb_put(skb, length);
+ skb->protocol = eth_type_trans(skb, netdev);
+
+ if (skb->protocol == cpu_to_be16(ETH_P_IP) &&
+ skb_headlen(skb) >= sizeof(struct iphdr)) {
+ struct iphdr *iph = (struct iphdr *)skb->data;
+
+ iph_check = iph->check;
+ }
+
+ /* PHYP without PLSO support places a -1 in the ip
+ * checksum for large send frames.
+ */
+ if ((length > netdev->mtu + ETH_HLEN) || lrg_pkt ||
+ iph_check == 0xffff) {
+ if (!ibmveth_rx_mss_helper(skb, mss, lrg_pkt))
+ csum_good = 0;
+ adapter->rx_large_packets++;
+ }
+
+ if (csum_good) {
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ ibmveth_rx_csum_helper(skb, adapter);
+ }
+
+ napi_gro_receive(napi, skb);
+
+ netdev->stats.rx_packets++;
+ netdev->stats.rx_bytes += length;
+
+ return 1;
+}
+
static int ibmveth_poll(struct napi_struct *napi, int budget)
{
struct net_device *netdev = napi->dev;
struct ibmveth_adapter *adapter = netdev_priv(netdev);
int frames_processed = 0;
- int queue_index;
- u64 correlator;
- u16 mss = 0;
- int rc;
+ int queue_index, rc;
queue_index = napi - adapter->napi;
+ if (WARN_ON(queue_index < 0 ||
+ queue_index >= adapter->num_rx_queues)) {
+ if (budget)
+ napi_complete_done(napi, 0);
+ return 0;
+ }
+
+ if (ibmveth_poll_stopping(netdev, napi)) {
+ if (budget)
+ napi_complete_done(napi, 0);
+ return 0;
+ }
+
while (frames_processed < budget) {
+ if (ibmveth_poll_stopping(netdev, napi))
+ break;
+
if (!ibmveth_rxq_pending_buffer(adapter, queue_index))
break;
smp_rmb();
- correlator = ibmveth_rxq_read_correlator(adapter, queue_index);
if (!ibmveth_rxq_buffer_valid(adapter, queue_index)) {
wmb(); /* suggested by larson1 */
- adapter->rx_invalid_buffer++;
- netdev_dbg(netdev, "recycling invalid buffer\n");
- rc = ibmveth_rxq_harvest_buffer(adapter, queue_index,
- correlator, true);
- if (unlikely(rc)) {
- netdev->stats.rx_dropped++;
+ if (ibmveth_poll_recycle_invalid(netdev, adapter,
+ queue_index))
frames_processed++;
- continue;
- }
} else {
- struct sk_buff *skb, *new_skb;
- int length = ibmveth_rxq_frame_length(adapter,
- queue_index);
- int offset = ibmveth_rxq_frame_offset(adapter,
- queue_index);
- int csum_good = ibmveth_rxq_csum_good(adapter,
- queue_index);
- int lrg_pkt = ibmveth_rxq_large_packet(adapter,
- queue_index);
- unsigned int pool, room, off, len;
- struct ibmveth_buff_pool *bpool;
- struct ibmveth_rx_q *rxq;
- __sum16 iph_check = 0;
-
- skb = ibmveth_rxq_get_buffer(adapter, queue_index,
- correlator);
- if (unlikely(!skb)) {
- rxq = &adapter->rx_queue[queue_index];
- ibmveth_rxq_advance(rxq);
- netdev->stats.rx_dropped++;
- frames_processed++;
- continue;
- }
-
- pool = correlator >> 32;
- bpool = &adapter->rx_buff_pool[queue_index][pool];
- room = min_t(unsigned int, skb_tailroom(skb),
- bpool->buff_size);
- off = offset;
- len = length;
- if (unlikely(len < ETH_HLEN || off >= room ||
- len > room - off)) {
- if (net_ratelimit())
- netdev_err(netdev,
- "bad RX frame offset %u length %u (buffer %u), dropping\n",
- off, len, room);
- ibmveth_rxq_harvest_buffer(adapter, queue_index,
- correlator, true);
- netdev->stats.rx_dropped++;
- frames_processed++;
- continue;
- }
-
- /* if the large packet bit is set in the rx queue
- * descriptor, the mss will be written by PHYP eight
- * bytes from the start of the rx buffer, which is
- * skb->data at this stage
- */
- if (lrg_pkt) {
- __be64 *rxmss = (__be64 *)(skb->data + 8);
-
- mss = (u16)be64_to_cpu(*rxmss);
- }
-
- new_skb = NULL;
- if (length < rx_copybreak)
- new_skb = netdev_alloc_skb(netdev, length);
-
- if (new_skb) {
- skb_copy_to_linear_data(new_skb,
- skb->data + offset,
- length);
- if (rx_flush)
- ibmveth_flush_buffer(skb->data,
- length + offset);
- rc = ibmveth_rxq_harvest_buffer(adapter,
- queue_index,
- correlator,
- true);
- if (unlikely(rc)) {
- dev_kfree_skb_any(new_skb);
- netdev->stats.rx_dropped++;
- frames_processed++;
- continue;
- }
- skb = new_skb;
- } else {
- rc = ibmveth_rxq_harvest_buffer(adapter,
- queue_index,
- correlator,
- false);
- if (unlikely(rc)) {
- netdev->stats.rx_dropped++;
- frames_processed++;
- continue;
- }
- skb_reserve(skb, offset);
- }
-
- skb_put(skb, length);
- skb->protocol = eth_type_trans(skb, netdev);
-
- /* PHYP without PLSO support places a -1 in the ip
- * checksum for large send frames.
+ rc = ibmveth_poll_deliver_frame(napi, adapter, netdev,
+ queue_index);
+ if (rc < 0)
+ break;
+ /* A slot skipped for a bad correlator or an oversize
+ * frame consumes budget too, so a run of them cannot
+ * keep this poll going.
*/
- if (skb->protocol == cpu_to_be16(ETH_P_IP) &&
- skb_headlen(skb) >= sizeof(struct iphdr)) {
- struct iphdr *iph = (struct iphdr *)skb->data;
-
- iph_check = iph->check;
- }
-
- if ((length > netdev->mtu + ETH_HLEN) ||
- lrg_pkt || iph_check == 0xffff) {
- if (!ibmveth_rx_mss_helper(skb, mss, lrg_pkt))
- csum_good = 0;
- adapter->rx_large_packets++;
- }
-
- if (csum_good) {
- skb->ip_summed = CHECKSUM_UNNECESSARY;
- ibmveth_rx_csum_helper(skb, adapter);
- }
-
- napi_gro_receive(napi, skb); /* send it up */
-
- netdev->stats.rx_packets++;
- netdev->stats.rx_bytes += length;
frames_processed++;
}
}
@@ -2670,15 +2739,30 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
if (budget)
ibmveth_replenish_task(adapter, queue_index);
+ /* At full budget the NAPI core still owns the instance and
+ * completes it itself if a disable is pending.
+ */
+ if (ibmveth_poll_stopping(netdev, napi)) {
+ if (budget && frames_processed < budget)
+ napi_complete_done(napi, frames_processed);
+ return frames_processed;
+ }
+
if (frames_processed == budget)
goto out;
if (!napi_complete_done(napi, frames_processed))
goto out;
- /* We think we are done - reenable interrupts,
- * then check once more to make sure we are done.
+ /*
+ * Narrows the window in which enable_irq() re-arms PHYP for a
+ * queue that close() is tearing down. It is not a barrier:
+ * napi_disable() does not wait for this tail, so teardown waits
+ * for it with synchronize_net() and then remasks.
*/
+ if (ibmveth_poll_stopping(netdev, napi))
+ goto out;
+
rc = ibmveth_enable_irq(adapter, queue_index);
if (rc) {
netdev_err(netdev,
@@ -2688,8 +2772,9 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
goto out;
}
- /* Mask before queueing, as schedule_rx_queue() does: a threaded
- * NAPI can start polling, and unmask, as soon as it is queued.
+ /* Mask before queueing, as ibmveth_schedule_rx_queue() does: a
+ * threaded NAPI can start polling, and unmask, as soon as it
+ * is queued.
*/
if (ibmveth_rxq_pending_buffer(adapter, queue_index) &&
napi_schedule_prep(napi)) {
@@ -3444,8 +3529,8 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
* ibmveth_rxq_get_buffer_test - unit test for ibmveth_rxq_get_buffer
* @test: pointer to kunit structure
*
- * Tests ibmveth_rxq_get_buffer. Each NULL return also logs a ratelimited
- * netdev_err.
+ * Tests ibmveth_rxq_get_buffer. A NULL return does not log; the harvest
+ * of that slot reports it.
*
* Return: void
*/
@@ -3454,7 +3539,6 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
struct ibmveth_adapter *adapter = kunit_kzalloc(test, sizeof(*adapter), GFP_KERNEL);
struct sk_buff *skb = kunit_kzalloc(test, sizeof(*skb), GFP_KERNEL);
struct ibmveth_buff_pool *pool;
- u64 cor;
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
@@ -3481,49 +3565,48 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->free_map);
- adapter->rx_queue[0].queue_addr[0].correlator =
- (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
- cor = ibmveth_rxq_read_correlator(adapter, 0);
+ u64 correlator;
+
+ correlator = (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
+ adapter->rx_queue[0].queue_addr[0].correlator = correlator;
KUNIT_EXPECT_PTR_EQ(test, NULL,
- ibmveth_rxq_get_buffer(adapter, 0, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, correlator));
- adapter->rx_queue[0].queue_addr[0].correlator =
- (u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
- cor = ibmveth_rxq_read_correlator(adapter, 0);
+ correlator = (u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
+ adapter->rx_queue[0].queue_addr[0].correlator = correlator;
KUNIT_EXPECT_PTR_EQ(test, NULL,
- ibmveth_rxq_get_buffer(adapter, 0, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, correlator));
/* Pool 2 is in range but has no skbuff array, like an inactive pool. */
- adapter->rx_queue[0].queue_addr[0].correlator = (u64)2 << 32 | 0;
- cor = ibmveth_rxq_read_correlator(adapter, 0);
+ correlator = (u64)2 << 32 | 0;
+ adapter->rx_queue[0].queue_addr[0].correlator = correlator;
KUNIT_EXPECT_PTR_EQ(test, NULL,
- ibmveth_rxq_get_buffer(adapter, 0, cor));
+ ibmveth_rxq_get_buffer(adapter, 0, correlator));
pool->skbuff[0] = skb;
- adapter->rx_queue[0].queue_addr[0].correlator = (u64)0 << 32 | 0;
- cor = ibmveth_rxq_read_correlator(adapter, 0);
+ correlator = (u64)0 << 32 | 0;
+ adapter->rx_queue[0].queue_addr[0].correlator = correlator;
KUNIT_EXPECT_PTR_EQ(test, skb,
- ibmveth_rxq_get_buffer(adapter, 0, cor));
-
- flush_work(&adapter->work);
+ ibmveth_rxq_get_buffer(adapter, 0, correlator));
}
/**
- * ibmveth_rxq_harvest_buffer_test - unit test for ibmveth_rxq_harvest_buffer
+ * ibmveth_rxq_harvest_correlator_test - unit test for
+ * ibmveth_rxq_harvest_correlator
* @test: pointer to kunit structure
*
* A bad correlator must still advance the RX ring, wrapping and flipping
- * the toggle at the end. This covers the harvest path; the advance after
- * ibmveth_rxq_get_buffer() fails in ibmveth_poll() is not tested here.
+ * the toggle at the end. Poll's skip of a slot whose
+ * ibmveth_rxq_get_buffer() fails advances through the same function.
*
* Return: void
*/
-static void ibmveth_rxq_harvest_buffer_test(struct kunit *test)
+static void ibmveth_rxq_harvest_correlator_test(struct kunit *test)
{
struct ibmveth_adapter *adapter;
struct ibmveth_buff_pool *pool;
- u64 cor;
int ret;
+ u64 cor;
adapter = kunit_kzalloc(test, sizeof(*adapter), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter);
@@ -3560,13 +3643,13 @@ static void ibmveth_rxq_harvest_buffer_test(struct kunit *test)
adapter->rx_queue[0].queue_addr[1].correlator = (u64)0 << 32 | 0;
cor = ibmveth_rxq_read_correlator(adapter, 0);
- ret = ibmveth_rxq_harvest_buffer(adapter, 0, cor, true);
+ ret = ibmveth_rxq_harvest_correlator(adapter, 0, true, cor);
KUNIT_EXPECT_EQ(test, -EINVAL, ret);
KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue[0].index);
KUNIT_EXPECT_EQ(test, 1ULL, adapter->rx_queue[0].toggle);
cor = ibmveth_rxq_read_correlator(adapter, 0);
- ret = ibmveth_rxq_harvest_buffer(adapter, 0, cor, true);
+ ret = ibmveth_rxq_harvest_correlator(adapter, 0, true, cor);
KUNIT_EXPECT_EQ(test, -EFAULT, ret);
KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue[0].index);
KUNIT_EXPECT_EQ(test, 0ULL, adapter->rx_queue[0].toggle);
@@ -3802,7 +3885,7 @@ static void ibmveth_rx_csum_helper_test(struct kunit *test)
static struct kunit_case ibmveth_test_cases[] = {
KUNIT_CASE(ibmveth_remove_buffer_from_pool_test),
KUNIT_CASE(ibmveth_rxq_get_buffer_test),
- KUNIT_CASE(ibmveth_rxq_harvest_buffer_test),
+ KUNIT_CASE(ibmveth_rxq_harvest_correlator_test),
KUNIT_CASE(ibmveth_rx_mss_helper_test),
KUNIT_CASE(ibmveth_rx_csum_helper_test),
{}
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 10/15] ibmveth: Enable multi-queue RX receive path
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (8 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 09/15] ibmveth: Split RX poll into per-slot helpers " Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 11/15] ibmveth: Add per-queue RX and TX statistics collection Mingming Cao
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
Turn on multi-queue RX when firmware sets
IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT in H_ILLAN_ATTRIBUTES. The
earlier patches made every RX path take a queue index; this one
supplies queues 1..N:
- probe: set multi_queue and publish min(num_online_cpus(),
IBMVETH_DEFAULT_QUEUES) live queues, matching the TX default
(cap 8). IBMVETH_MAX_RX_QUEUES goes from 1 to 16, so every array
sized by it grows; the netdev is allocated with that many RX
queues and one NAPI each, and real_num_rx_queues is set before
register_netdev(). MQ firmware also raises the replenish batch
from 8 to 12 buffers per hcall (IBMVETH_MAX_RX_PER_HCALL).
- open: register queue 0 and subordinates 1..N, set real_num after
registration, post buffers to every queue, request IRQs and enable
NAPI; MQ setup with more than one live queue unmasks PHYP before
kicking each queue, while SQ
leaves PHYP masked until poll or restart kicks NAPI (per queue).
- receive: each queue's interrupt schedules only its own NAPI;
poll uses that queue's ring, pools and replenish_lock, and tags
each skb with skb_record_rx_queue() for RPS/RFS.
- teardown: one H_FREE_LOGICAL_LAN drops every queue.
- fallback: H_FUNCTION from subordinate register or buffer add
latches mq_fallback, and the next open comes up single-queue.
Without the firmware bit, behaviour stays single-queue.
num_rx_queues is read through ibmveth_get_num_rx_queues()
(smp_load_acquire) and written through ibmveth_publish_num_rx_queues()
(smp_store_release), so a lockless IRQ, poll or replenish reader that
sees a count also sees the per-queue state behind it. multi_queue
becomes a bool, now that it is a flag and not a count.
Queue 0 keeps netdev->irq and netdev->name in /proc/interrupts and
is never disposed with the subordinates; queue N is named
ibmveth-<unit address>-rxN, as ibmvnic names its sub-CRQs.
Each RX queue gets its own copy of the active buffer pools, so RX
buffer memory and IOMMU mappings scale linearly with num_rx_queues:
N queues use N times the single-queue footprint, eight times by
default on a partition with eight or more CPUs. ethtool -L lowers
it. get_desired_dma() accounts for the mappings (see below).
This commit rejects an ethtool -L that changes rx_count. Later patches
implement live RX set, the down-path publish/rollback, and the max_rx
cap once mq_fallback latches.
Restate open() here:
... register_rx_queues / set_real_num_rx_queues ...
replenish_task() for every live RX queue (before IRQ setup)
setup_rx_interrupts() (MQ with num>1 unmasks PHYP)
restart_rx_queue() for every live queue (schedule NAPI; a poll
already queued unmasks)
... alloc_tx_resources / tx_start / opened ...
That path is the same for SQ and MQ; it replaces the prior
setup-then-schedule_rx_queue(0) kick. resume() likewise kicks every
live RX queue rather than only queue 0, and each delivered skb is
tagged with skb_record_rx_queue() so RPS/RFS see the queue it
arrived on. close() keeps its order but skips the frees when
free_all_queues() fails (below).
H_FUNCTION means firmware withdrew MQ. Both the subordinate-register
and the buffer-add path latch mq_fallback, so the next open comes up
single-queue (apply_mq_fallback at open entry). That helper publishes
1 and sets real_num to match, caps rx_buffers_per_hcall back to
IBMVETH_MAX_RX_REGULAR (8) for the legacy hcall, so a failed open
after it does not leave real_num at N. The two H_FUNCTION sites
differ in what happens to the open in progress. Register fails it
outright rather than dropping to
single-queue silently mid-open, and buffer-add schedules a reset.
Reset does not retry if that close/open fails.
Both setup failure paths dispose subordinate virq mappings before
unwinding registration.
A poolN kobject that fails kobject_init_and_add() now fails probe;
the base driver ignored the error and probed without that sysfs node.
The failed kobject is put and its release() waited for on the spot,
and ibmveth_put_pool_kobjs() takes the number of pools added so far.
get_desired_dma() sizes RX from queue-0 pool metadata across
num_rx_queues. CMO (Power9 and earlier) and MQ firmware (Power11+)
do not coexist, so probe does not refresh CMO desired and a CMO
partition keeps its single-queue default.
remove() already disables the reset work before unregister_netdev().
Per-queue RX and TX statistics land in the next patch. Until then MQ
reuses the adapter-wide counter fields used on the single-queue path.
Those fields now have one writer per queue, so the RX and replenish
counters can lose increments when several NAPI polls run at once,
and rx_no_buffer reports the buffer-list page sampled last.
On subordinate register failure, honor H_FREE_LOGICAL_LAN: if that
hard-fails, ibmveth_free_all_queues() returns -EIO,
register_rx_queues() reports it through @skip_unmap, and open() skips
the unmap even when queue_handle[0] is 0, since handle 0 is a valid
PHYP cookie. The buffer pool arrays are still freed on that path: no
buffer is posted before registration, so PHYP never saw them.
The same skip applies after buffers are posted: open() post-register
unwind and close() skip DMA unmap when free_all_queues() returns
-EIO, which it does before clearing queue_registered[]. ndo_stop
still returns 0. The next open() and remove() retry
H_FREE_LOGICAL_LAN before a new allocation or free_netdev(), and
release the old memory once it succeeds. If the retry in remove()
fails too, the memory is leaked rather than unmapped under PHYP, and
remove() says so. That relies on CMO and MQ firmware not coexisting
(above): on a CMO partition the leaked entitlement would BUG() at
remove.
That deferred free walks queue 0's pools with pool->size and
pool->buff_size, so a poolN sysfs write returns -EBUSY while the old
pools are still mapped, after reopening a running device so the free
is retried with the old geometry; if that reopen fails, its error is
returned instead. A write whose reopen with the new geometry fails
keeps that geometry if the new pools stay mapped, for the same
reason. free_tx_resources() walks every TX LTB slot rather
than real_num_tx_queues, since an ethtool -L while down can lower
real_num_tx_queues under LTBs that are still mapped.
Only MQ adapters get -EIO, and not for H_PARAMETER (already free).
Single-queue adapters keep what close() did before this series: log
the failure and unmap. Keeping that memory mapped instead is not a
drop-in change there; one reason is that on a CMO partition
vio_cmo_bus_remove() BUG()s on outstanding IO entitlement at remove.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- MQ hard H_FREE_LOGICAL_LAN failure (not H_PARAMETER) keeps the LAN
memory mapped: register failure, the open() unwind and close() skip
the unmap; the next open() and remove() retry and then free it
(remove() leaves it mapped if that fails); poolN writes return -EBUSY
meanwhile, and a write whose reopen fails keeps the new geometry
if its pools stay mapped. Single queue keeps the base log-and-unmap
- free_tx_resources() walks every TX LTB slot
- apply_mq_fallback() sets real_num to 1; set_channels rejects an
rx_count change until the resize patch
- restart_rx_queue() does not unmask PHYP when napi_schedule_prep()
fails
- subordinate IRQs are named ibmveth-<unit>-rxN; queue 0 keeps
netdev->name
- a probe kobject_init_and_add() failure puts only the pool kobjects
already added (pools_ready)
- commit message: RX counters stay adapter-wide until the next patch;
remove() relies on CMO and MQ firmware not coexisting
Dropped from this series (merged fixes own them):
- commit 8d9ecac6b96e: poll_controller per-queue loop (ndo removed)
- commit 141223544786: remove unregister/cancel reorder and reset gate
text (in base)
- commit fd052c6823b3: pool kobject release/wait hunks and probe
message paragraph (in base; pools_ready unwind added here)
- commit 3f872b34368c: extra open() poll wait (err_disable_napi in P05)
Changes in v7:
- register_single_rx_queue logs lpar_rc with %ld
- setup_rx_interrupts kdoc: open posts first in both modes
- wait for pool kobject release before free_netdev
- remove-path unregister/cancel reorder and reset
reg_state gate land here alongside the reset producer
- drop the probe CMO desired refresh: CMO (Power9 and
earlier) and MQ (Power11+) do not coexist; say so in
the get_desired_dma() sizing comment
- commit message: drop the stale net sentence
- noted: restart enable_irq on prep fail stays;
counters stay in P11; poll_controller
!opened stays in P15
- noted: no CMO+MQ partition (CMO is P9, MQ is P11+);
default-8 is the TX match, not a CMO tradeoff;
get_desired_dma TX term is CMO desired-hint only;
real TX LTBs already scale; those two CMO leftovers
drop from the v7 cover
Changes in v6:
- ibmveth_get_num_rx_queues / ibmveth_publish_num_rx_queues are
static, not static inline
- wrap the rx_buffers_per_hcall assignment (81 cols)
- smp_store_release / smp_load_acquire for num_rx_queues, not
smp_wmb() + WRITE_ONCE()
- skb_record_rx_queue() before GRO
- drop the IBMVETH_MAX_RX_QUEUE alias (duplicated MAX_RX_PER_HCALL)
- drop WARN_ON on restart enable_irq; the helper already logs.
Enabling on prep failure is deliberate
- drop unused kick_rx_queue_if_pending() (WERROR); patch 14 uses
restart_rx_queue() at both sites
- register_rx_queues() kdoc: this function latches mq_fallback on
-EOPNOTSUPP
- noted: pool kobj vs DEBUG_KOBJECT_RELEASE stays cover leftovers
Changes in v5:
- RTNL/serialized readers use get_num_rx_queues() once the helper exists
(open/close/register/apply_mq_fallback/alloc/cleanup/IRQ setup)
- Interrupt: quiet IRQ_NONE on out-of-range qindex vs published live
count (scale-down residual IRQ; no WARN storm)
- Series renumber: mailed v4 09/14 MQ enable -> tip P10 (P09 peel; 14->15)
- Open: replenish all queues before setup; restart_rx_queue after setup
for SQ and MQ - replaces mailed MQ-only replenish + kick_if_pending vs
SQ schedule_rx_queue(0)
- update_rx_no_buffer(queue_index): NULL-safe, no all-queue walk under one
lock (fixes the resize/race class). Still stores adapter->rx_no_buffer
for now; per-queue slot + monotonic sum lands with qstats next
- H_FUNCTION: subordinate register fails open + stash mq_fallback;
buffer-add schedules reset + mq_fallback so next open drops to SQ
- Introduce get_num_rx_queues / publish_num_rx_queues (READ/WRITE_ONCE)
at first use; lockless IRQ/poll/replenish/interrupt bounds use the getter
- Introduce kick_rx_queue_if_pending() for pending-after-unmask; open uses
restart_rx_queue, resize calls the helper later
- resume() schedules every live RX queue (not only queue 0)
- Call out enable_irq-fail synchronize_irq and unused mac register-arg
cleanup (mailed enable-path review; code already earlier in tip)
Changes in v4:
- Fold subordinate register helpers and their review fixes into the MQ
enablement patch that first uses them.
- Prefer request_irq -> napi_enable -> PHYP enable on MQ open.
- Preserve open unwind so set_real_num_rx / IRQ failures free LAN
before buffer pools.
- MQ open replenishes every queue before setup_rx_interrupts() unmasks
PHYP (drop avoidance during open; PHYP only interrupts after a
successful enqueue). SQ keeps classic setup-then-schedule kick.
- Dispose subordinate virq mappings on setup_rx_interrupts()
request_irq failure (err_free_irqs), matching err_disable_napi.
- Open unwind: setup/cleanup own subordinate dispose; skip duplicate
dispose on those paths.
- H_FUNCTION on subordinate register is a hard open failure (no blind
retry / no fake single-queue fallback).
- Note: hot-path netdev->stats accounting moves to the next patch (qstats).
- Put already-created pool kobjects on probe kobject_init_and_add /
set_real_num_tx_queues / register_netdev failure (bisect-safe).
drivers/net/ethernet/ibm/ibmveth.c | 681 ++++++++++++++++++++++++-----
drivers/net/ethernet/ibm/ibmveth.h | 7 +-
2 files changed, 579 insertions(+), 109 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index b12bfa1e485e..a0c3c06d2b9d 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -156,6 +156,28 @@ static int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter,
return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_CSUM_GOOD;
}
+/* Lockless IRQ/poll readers vs resize publishers. */
+static unsigned int
+ibmveth_get_num_rx_queues(const struct ibmveth_adapter *adapter)
+{
+ /*
+ * Pairs with the release in ibmveth_publish_num_rx_queues(): a reader
+ * that sees the new count also sees the per-queue state behind it.
+ */
+ return smp_load_acquire(&adapter->num_rx_queues);
+}
+
+static void
+ibmveth_publish_num_rx_queues(struct ibmveth_adapter *adapter,
+ unsigned int num)
+{
+ /*
+ * Pairs with the acquire in ibmveth_get_num_rx_queues(): per-queue
+ * state must be visible to a reader before it observes the new count.
+ */
+ smp_store_release(&adapter->num_rx_queues, num);
+}
+
static unsigned int ibmveth_real_max_tx_queues(void)
{
unsigned int n_cpu = num_online_cpus();
@@ -233,7 +255,7 @@ ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
struct net_device *netdev = adapter->netdev;
int i;
- for (i = 0; i < adapter->num_rx_queues; i++) {
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
adapter->buffer_list_addr[i] =
(void *)get_zeroed_page(GFP_KERNEL);
if (!adapter->buffer_list_addr[i]) {
@@ -284,7 +306,7 @@ ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
}
netdev_dbg(netdev, "allocated %u RX queue(s) with %d entries each\n",
- adapter->num_rx_queues, rxq_entries);
+ ibmveth_get_num_rx_queues(adapter), rxq_entries);
return 0;
@@ -328,9 +350,9 @@ ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
int i;
netdev_dbg(adapter->netdev, "cleaning up %u RX queue(s)\n",
- adapter->num_rx_queues);
+ ibmveth_get_num_rx_queues(adapter));
- for (i = 0; i < adapter->num_rx_queues; i++) {
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
if (adapter->buffer_list_addr[i]) {
dma_unmap_single(dev, adapter->buffer_list_dma[i],
4096, DMA_BIDIRECTIONAL);
@@ -489,22 +511,45 @@ ibmveth_dispose_subordinate_irq_mappings(struct ibmveth_adapter *adapter)
{
int i;
- for (i = 1; i < adapter->num_rx_queues; i++)
+ for (i = 1; i < ibmveth_get_num_rx_queues(adapter); i++)
ibmveth_dispose_subordinate_irq_mapping(adapter, i);
}
+/**
+ * ibmveth_queue_irq_name - Name for one RX queue's Linux IRQ
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: RX queue index
+ *
+ * Queue 0 keeps netdev->name, as before multi-queue. Subordinate queues
+ * use "ibmveth-<unit address>-rx<N>", which survives a rename.
+ * request_irq() keeps the pointer, so the string lives in the adapter.
+ *
+ * Return: the name to pass to request_irq()
+ */
+static const char *ibmveth_queue_irq_name(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ char *name = adapter->queue_irq_name[queue_idx];
+
+ if (!queue_idx)
+ return adapter->netdev->name;
+
+ snprintf(name, sizeof(adapter->queue_irq_name[0]), "ibmveth-%x-rx%d",
+ adapter->vdev->unit_address, queue_idx);
+ return name;
+}
+
/**
* ibmveth_setup_rx_interrupts - Register IRQs and enable NAPI
* @adapter: ibmveth adapter structure
*
- * Registers interrupt handlers for all RX queues, enables NAPI, then
- * enables hypervisor interrupt delivery for multi-queue mode after
- * every queue has a Linux handler installed. For multi-queue open the
- * caller should replenish RX buffers before this helper so traffic
- * during open is not dropped (PHYP only interrupts after a successful
- * enqueue, which needs buffers). Single-queue open leaves PHYP masked
- * here and kicks NAPI afterward (classic path: first poll posts then
- * enables).
+ * Registers interrupt handlers for all RX queues, enables NAPI, then,
+ * in multi-queue mode with more than one live queue, enables
+ * hypervisor interrupt delivery after every queue has a Linux handler
+ * installed. The caller replenishes every live queue before this
+ * helper so traffic during open is not dropped (PHYP only interrupts
+ * after a successful enqueue, which needs buffers). With one queue
+ * PHYP stays masked here; restart_rx_queue then kicks NAPI.
*
* Return: 0 on success, negative error code on failure
*/
@@ -512,7 +557,7 @@ static int
ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
- int i, rc, num = adapter->num_rx_queues;
+ int i, rc, num = ibmveth_get_num_rx_queues(adapter);
for (i = 0; i < num; i++) {
if (!adapter->queue_irq[i]) {
@@ -522,7 +567,8 @@ ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
}
rc = request_irq(adapter->queue_irq[i], ibmveth_interrupt,
- 0, netdev->name, &adapter->napi[i]);
+ 0, ibmveth_queue_irq_name(adapter, i),
+ &adapter->napi[i]);
if (rc) {
netdev_err(netdev,
"request_irq() failed for irq 0x%x queue %d: %d\n",
@@ -606,26 +652,26 @@ ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
if (!adapter->rx_irq_setup)
return;
- for (i = 0; i < adapter->num_rx_queues; i++) {
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
if (!adapter->queue_irq[i])
continue;
ibmveth_disable_irq(adapter, i);
synchronize_irq(adapter->queue_irq[i]);
}
- for (i = 0; i < adapter->num_rx_queues; i++)
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
napi_disable(&adapter->napi[i]);
synchronize_net();
- for (i = 0; i < adapter->num_rx_queues; i++) {
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
if (!adapter->queue_irq[i])
continue;
ibmveth_disable_irq(adapter, i);
synchronize_irq(adapter->queue_irq[i]);
}
- for (i = 0; i < adapter->num_rx_queues; i++) {
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
if (adapter->queue_irq[i])
free_irq(adapter->queue_irq[i], &adapter->napi[i]);
}
@@ -654,7 +700,7 @@ static void ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
{
struct napi_struct *napi = &adapter->napi[qindex];
- if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
+ if (WARN_ON(qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter)))
return;
/*
@@ -758,6 +804,7 @@ static long ibmveth_add_logical_lan_buffers(struct ibmveth_adapter *adapter,
if (adapter->multi_queue) {
unsigned long buffersznum = (buff_size << 32) | filled;
unsigned long ioba[IBMVETH_MAX_RX_PER_HCALL / 2] = {0};
+ /* PHYP queue handle; 0 can be valid. */
unsigned long handle = adapter->queue_handle[queue_index];
int i;
@@ -1002,15 +1049,21 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
* because there was not a buffer in the buffer list capable of holding
* the frame.
*/
-static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
+static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
{
__be64 *p;
+ u64 drops;
- if (!adapter->buffer_list_addr[0])
+ if (queue_index < 0 ||
+ queue_index >= ibmveth_get_num_rx_queues(adapter) ||
+ !adapter->buffer_list_addr[queue_index])
return;
- p = adapter->buffer_list_addr[0] + 4096 - 8;
- adapter->rx_no_buffer = be64_to_cpup(p);
+ p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
+ drops = be64_to_cpup(p);
+
+ adapter->rx_no_buffer = drops;
}
/* replenish routine */
@@ -1026,10 +1079,10 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
int batch_fallback = 0;
int hcall_fail = 0;
- if (queue_index >= adapter->num_rx_queues) {
+ if (queue_index >= ibmveth_get_num_rx_queues(adapter)) {
netdev_dbg(adapter->netdev,
"Skipping replenish for freed queue %d (num_queues=%u)\n",
- queue_index, adapter->num_rx_queues);
+ queue_index, ibmveth_get_num_rx_queues(adapter));
return;
}
@@ -1067,7 +1120,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
}
out_unlock:
- ibmveth_update_rx_no_buffer(adapter);
+ ibmveth_update_rx_no_buffer(adapter, queue_index);
spin_unlock_bh(&rxq->replenish_lock);
@@ -1081,6 +1134,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
dev_err_ratelimited(&adapter->netdev->dev,
"MQ buffer add H_FUNCTION (q=%d, batch=%u), reset\n",
queue_index, fail.batch);
+ adapter->mq_fallback = true;
schedule_work(&adapter->work);
}
@@ -1101,6 +1155,27 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
fail_hcall.batch);
}
+/**
+ * ibmveth_restart_rx_queue - Post buffers and ensure Q can take RX
+ * @adapter: ibmveth adapter
+ * @qindex: RX queue index
+ *
+ * SQ open leaves PHYP masked until the first poll, so every queue needs
+ * a NAPI run. Callers enable NAPI first, so napi_schedule_prep() fails
+ * only when a poll is already scheduled. That poll unmasks PHYP when it
+ * completes, so do not unmask here: while NAPI is scheduled the handler
+ * cannot remask, and PHYP would interrupt once per frame. Replenish
+ * first so the queue has buffers to deliver into.
+ * Also used after every open (SQ and MQ) so a queue is not left
+ * idle+masked.
+ */
+static void ibmveth_restart_rx_queue(struct ibmveth_adapter *adapter,
+ int qindex)
+{
+ ibmveth_replenish_task(adapter, qindex);
+ ibmveth_schedule_rx_queue(adapter, qindex);
+}
+
/* empty and free ana buffer pool - also used to do cleanup in error paths */
static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
struct ibmveth_buff_pool *pool)
@@ -1248,7 +1323,7 @@ ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
int i, q, rc;
/* Initialize pool metadata for queues 1..N from queue 0 settings */
- for (q = 1; q < adapter->num_rx_queues; q++) {
+ for (q = 1; q < ibmveth_get_num_rx_queues(adapter); q++) {
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
struct ibmveth_buff_pool *src =
&adapter->rx_buff_pool[0][i];
@@ -1264,7 +1339,7 @@ ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
}
/* Allocate actual buffers for all queues */
- for (q = 0; q < adapter->num_rx_queues; q++) {
+ for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
rc = ibmveth_alloc_queue_buffer_pools(adapter, q);
if (rc) {
/* Free pools for all previous queues */
@@ -1275,7 +1350,7 @@ ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
}
netdev_dbg(netdev, "allocated buffer pools for %u queue(s)\n",
- adapter->num_rx_queues);
+ ibmveth_get_num_rx_queues(adapter));
return 0;
}
@@ -1291,11 +1366,11 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
int q;
/* Free buffer pools for all queues */
- for (q = 0; q < adapter->num_rx_queues; q++)
+ for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++)
ibmveth_free_queue_buffer_pools(adapter, q);
netdev_dbg(adapter->netdev, "freed buffer pools for %u queue(s)\n",
- adapter->num_rx_queues);
+ ibmveth_get_num_rx_queues(adapter));
}
/* The correlator comes back from PHYP, so check it rather than trust it.
@@ -1544,17 +1619,19 @@ static int ibmveth_alloc_tx_resources(struct ibmveth_adapter *adapter)
}
/**
- * ibmveth_free_tx_resources - Free TX LTBs for real_num_tx_queues
+ * ibmveth_free_tx_resources - Free every allocated TX LTB
* @adapter: ibmveth adapter structure
*
- * Frees TX Long Term Buffers (LTBs) for real_num_tx_queues.
+ * Walks IBMVETH_MAX_QUEUES, not real_num_tx_queues. An ethtool -L
+ * while down can shrink real_num while skip-unmap close left bounce
+ * buffers mapped; stopping the walk at real_num would skip those
+ * slots. ibmveth_free_tx_ltb() is a no-op on a NULL slot.
*/
static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)
{
- struct net_device *netdev = adapter->netdev;
int i;
- for (i = 0; i < netdev->real_num_tx_queues; i++)
+ for (i = 0; i < IBMVETH_MAX_QUEUES; i++)
ibmveth_free_tx_ltb(adapter, i);
}
@@ -1621,6 +1698,163 @@ static long ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
return rc;
}
+/**
+ * ibmveth_register_logical_lan_queue - Register subordinate queue with
+ * hypervisor
+ * @adapter: ibmveth adapter structure
+ * @rxq_desc: Receive queue descriptor
+ * @queue_index: RX queue index (1..N for subordinate queues)
+ *
+ * Registers a subordinate receive queue using H_REG_LOGICAL_LAN_QUEUE.
+ * On success, stores the queue handle and virtual IRQ in the adapter.
+ * If IRQ mapping fails after a successful hypervisor registration, the
+ * queue is freed before returning.
+ *
+ * Return: H_SUCCESS on success, negative errno on IRQ mapping failure,
+ * hypervisor error code otherwise
+ */
+static int
+ibmveth_register_logical_lan_queue(struct ibmveth_adapter *adapter,
+ union ibmveth_buf_desc rxq_desc,
+ int queue_index)
+{
+ unsigned long handle, hwirq;
+ unsigned int virq;
+ long lpar_rc;
+ unsigned long ua = adapter->vdev->unit_address;
+ unsigned long bl = adapter->buffer_list_dma[queue_index];
+
+ netdev_dbg(adapter->netdev,
+ "register queue %d: ua=0x%lx bl=0x%lx rxq=0x%llx\n",
+ queue_index, ua, bl, rxq_desc.desc);
+ do {
+ lpar_rc = h_register_logical_lan_queue(ua, bl,
+ rxq_desc.desc, &handle,
+ &hwirq);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+ netdev_dbg(adapter->netdev,
+ "h_register_logical_lan_queue queue %d rc=%ld\n",
+ queue_index, lpar_rc);
+
+ if (lpar_rc == H_SUCCESS) {
+ virq = irq_create_mapping(NULL, hwirq);
+ if (!virq) {
+ unsigned long free_rc;
+
+ netdev_err(adapter->netdev,
+ "Failed to map IRQ for queue %d (hwirq=%lu)\n",
+ queue_index, hwirq);
+ do {
+ free_rc = h_free_logical_lan_queue(ua, handle);
+ } while (H_IS_LONG_BUSY(free_rc) ||
+ (free_rc == H_BUSY));
+ if (free_rc != H_SUCCESS)
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan_queue failed for queue %d after IRQ map failure: rc=0x%lx\n",
+ queue_index, free_rc);
+ return -EINVAL;
+ }
+
+ ibmveth_set_rx_queue_handle(adapter, queue_index, handle);
+ adapter->queue_irq[queue_index] = virq;
+
+ netdev_dbg(adapter->netdev,
+ "queue %d registered: handle=0x%llx irq=%u\n",
+ queue_index, adapter->queue_handle[queue_index],
+ adapter->queue_irq[queue_index]);
+ return H_SUCCESS;
+ }
+
+ /*
+ * H_FUNCTION means firmware rejected this subordinate register
+ * (MQ unsupported / dropped after LPM). Caller fails this open and
+ * latches mq_fallback so the next open applies SQ; keep a specific
+ * log then the generic failure lines below.
+ */
+ if (lpar_rc == H_FUNCTION)
+ netdev_err(adapter->netdev,
+ "h_register_logical_lan_queue H_FUNCTION for queue %d (firmware MQ unsupported)\n",
+ queue_index);
+
+ netdev_err(adapter->netdev,
+ "h_register_logical_lan_queue failed for queue %d with %ld\n",
+ queue_index, lpar_rc);
+ netdev_err(adapter->netdev,
+ "queue %d params: unit_addr=0x%x buffer_list_dma=0x%llx rxq_desc=0x%llx\n",
+ queue_index, adapter->vdev->unit_address,
+ adapter->buffer_list_dma[queue_index],
+ rxq_desc.desc);
+
+ return lpar_rc;
+}
+
+/**
+ * ibmveth_register_single_rx_queue - Register one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to register (1..N)
+ *
+ * Builds the queue descriptor and registers with the hypervisor via
+ * ibmveth_register_logical_lan_queue().
+ *
+ * Return: 0 on success, -EINVAL if @queue_idx is invalid, -EOPNOTSUPP if
+ * firmware rejects MQ (H_FUNCTION), -EIO on other failures
+ */
+static int
+ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ struct net_device *netdev = adapter->netdev;
+ union ibmveth_buf_desc rxq_desc;
+ long lpar_rc;
+
+ if (WARN_ON(queue_idx < 1 || queue_idx >= IBMVETH_MAX_RX_QUEUES))
+ return -EINVAL;
+
+ rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
+ adapter->rx_queue[queue_idx].queue_len;
+ rxq_desc.fields.address = adapter->rx_queue[queue_idx].queue_dma;
+
+ lpar_rc = ibmveth_register_logical_lan_queue(adapter, rxq_desc,
+ queue_idx);
+ if (lpar_rc != H_SUCCESS) {
+ netdev_err(netdev, "Failed to register queue %d: rc=%ld\n",
+ queue_idx, lpar_rc);
+ if (lpar_rc == H_FUNCTION)
+ return -EOPNOTSUPP;
+ return -EIO;
+ }
+
+ netdev_dbg(netdev, "Registered queue %d with handle 0x%llx\n",
+ queue_idx, adapter->queue_handle[queue_idx]);
+
+ return 0;
+}
+
+static bool ibmveth_lan_resources_live(struct ibmveth_adapter *adapter)
+{
+ int i;
+
+ if (adapter->filter_list_addr)
+ return true;
+ if (adapter->tx_ltb_ptr[0])
+ return true;
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ if (adapter->queue_registered[i])
+ return true;
+ if (adapter->buffer_list_addr[i])
+ return true;
+ }
+ return false;
+}
+
+static void ibmveth_release_lan_resources(struct ibmveth_adapter *adapter)
+{
+ ibmveth_free_tx_resources(adapter);
+ ibmveth_free_buffer_pools(adapter);
+ ibmveth_cleanup_rx_resources(adapter);
+ ibmveth_free_filter_list(adapter);
+}
+
/**
* ibmveth_free_all_queues - Free all RX queues at once
* @adapter: ibmveth adapter structure
@@ -1630,17 +1864,25 @@ static long ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
* it. Incremental scale-down uses H_FREE_LOGICAL_LAN_QUEUE per queue
* instead; do not use this helper for partial live-set shrink.
*
- * Used by close() and the post-register open() unwind.
+ * Used by close(), the open() unwind and retry, and remove().
*
- * Retries only H_BUSY and H_IS_LONG_BUSY. On other failures, logs and
- * returns; callers cannot observe hypercall status. Registered slots
- * are cleared regardless. Callers still run RX pool and DMA teardown
- * afterward (same as pre-helper close()).
+ * Retries only H_BUSY and H_IS_LONG_BUSY. Other failures are logged.
+ * In multi-queue mode a failure other than H_PARAMETER (already free)
+ * is fail-before-start: PHYP may still DMA into the LAN memory, so
+ * return -EIO with the slots still registered. The caller skips the
+ * unmap, and the next open() or remove() retries. Single-queue mode
+ * keeps what close() always did, log and let the caller unmap; one
+ * reason it cannot simply keep the memory mapped is that on a CMO
+ * partition vio_cmo_bus_remove() BUG()s on an outstanding entitlement.
*
- * Clears registered flags and cookies of queues [0, num_rx_queues);
- * queue_irq[] is released by ibmveth_cleanup_rx_interrupts().
+ * On return 0, clears registered flags and cookies of queues
+ * [0, num_rx_queues); queue_irq[] is released by
+ * ibmveth_cleanup_rx_interrupts() on close, or by
+ * ibmveth_dispose_subordinate_irq_mappings() on partial register failure.
+ *
+ * Return: 0 if the caller may unmap, -EIO if it must not
*/
-static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
+static int ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
{
unsigned long lpar_rc;
int i;
@@ -1654,29 +1896,45 @@ static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
if (lpar_rc != H_SUCCESS) {
netdev_err(adapter->netdev,
"h_free_logical_lan failed: %ld\n", lpar_rc);
+ if (lpar_rc != H_PARAMETER && adapter->multi_queue)
+ return -EIO;
}
- for (i = 0; i < adapter->num_rx_queues; i++)
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
ibmveth_clear_rx_queue_handle(adapter, i);
+
+ return 0;
}
/**
* ibmveth_register_rx_queues - Register RX queues with hypervisor
* @adapter: ibmveth adapter structure
* @mac_address: MAC address for device registration
+ * @skip_unmap: set when H_FREE_LOGICAL_LAN hard-fails after a
+ * subordinate register error so open() will not unmap
+ * pages PHYP still owns (handle 0 is a valid cookie;
+ * queue_registered[] is the live-slot flag)
*
- * Registers queue 0 via ibmveth_register_logical_lan(). Subordinate queue
- * registration is added when multi-queue RX is enabled.
+ * Registers queue 0 via ibmveth_register_logical_lan(), then subordinate
+ * queues 1..N when multi-queue mode is enabled.
*
- * Return: 0 on success, -ENONET if queue 0 registration fails
+ * Return: 0 on success, -ENONET if queue 0 registration fails,
+ * -EOPNOTSUPP if firmware rejects a subordinate queue (H_FUNCTION;
+ * this function latches mq_fallback), -EIO on other subordinate
+ * failures
*/
static int
-ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
+ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address,
+ bool *skip_unmap)
{
struct net_device *netdev = adapter->netdev;
union ibmveth_buf_desc rxq_desc;
unsigned long lpar_rc;
- int rc;
+ unsigned int num;
+ int i, rc;
+
+ if (skip_unmap)
+ *skip_unmap = false;
rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
adapter->rx_queue[0].queue_len;
@@ -1701,9 +1959,77 @@ ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
return -ENONET;
}
+ num = ibmveth_get_num_rx_queues(adapter);
+ if (num == 1 || !adapter->multi_queue) {
+ netdev_dbg(netdev,
+ "registered 1 RX queue with hypervisor (single-queue mode)\n");
+ return 0;
+ }
+
+ netdev_dbg(netdev, "Registering %u subordinate queues (1-%u)\n",
+ num - 1, num - 1);
+
+ for (i = 1; i < num; i++) {
+ rc = ibmveth_register_single_rx_queue(adapter, i);
+ if (rc) {
+ /* Firmware MQ gone: fall back to SQ on next open. */
+ if (rc == -EOPNOTSUPP)
+ adapter->mq_fallback = true;
+ goto err_unregister;
+ }
+ }
+
netdev_dbg(netdev,
- "registered 1 RX queue with hypervisor (single-queue mode)\n");
+ "registered %u RX queues with hypervisor (multi-queue mode)\n",
+ num);
+
return 0;
+
+err_unregister:
+ ibmveth_dispose_subordinate_irq_mappings(adapter);
+ /*
+ * Honor H_FREE. A hard-fail is fail-before-start: open()
+ * must skip unmap even if queue 0's handle is 0.
+ */
+ if (ibmveth_free_all_queues(adapter)) {
+ if (skip_unmap)
+ *skip_unmap = true;
+ return rc;
+ }
+ return rc;
+}
+
+/**
+ * ibmveth_apply_mq_fallback - Drop multi-queue mode after firmware rejection
+ * @adapter: ibmveth adapter
+ *
+ * mq_fallback is set when firmware rejects MQ (subordinate register or
+ * buffer-add H_FUNCTION). Apply only at the start of open after teardown so
+ * num_rx_queues is not shrunk while IRQ/NAPI still reference higher queues.
+ * Consumes the flag and clears multi_queue, which is what makes the
+ * single-queue decision permanent for this device. Publish 1 and set
+ * real_num to match so a later alloc fail cannot leave real_num at N.
+ */
+static void ibmveth_apply_mq_fallback(struct ibmveth_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+
+ if (!adapter->mq_fallback)
+ return;
+
+ adapter->mq_fallback = false;
+
+ if (!adapter->multi_queue && ibmveth_get_num_rx_queues(adapter) == 1)
+ return;
+
+ netdev_warn(netdev,
+ "Falling back to single RX queue (firmware MQ unavailable)\n");
+ adapter->multi_queue = false;
+ ibmveth_publish_num_rx_queues(adapter, 1);
+ if (netif_set_real_num_rx_queues(netdev, 1))
+ netdev_err(netdev, "failed to set number of rx queues\n");
+ if (adapter->rx_buffers_per_hcall > IBMVETH_MAX_RX_REGULAR)
+ adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
}
static int ibmveth_open(struct net_device *netdev)
@@ -1711,6 +2037,7 @@ static int ibmveth_open(struct net_device *netdev)
struct ibmveth_adapter *adapter = netdev_priv(netdev);
u64 mac_address = ether_addr_to_u64(netdev->dev_addr);
int rxq_entries = 1;
+ bool skip_unmap = false;
int rc;
int i;
@@ -1721,6 +2048,23 @@ static int ibmveth_open(struct net_device *netdev)
*/
netif_tx_disable(netdev);
+ /*
+ * close() may have skipped unmap on H_FREE fail-before-start.
+ * Retry before apply_mq_fallback so the live count still
+ * matches the mapped set, and before a new alloc overwrites
+ * those pages.
+ */
+ if (ibmveth_lan_resources_live(adapter)) {
+ if (ibmveth_free_all_queues(adapter)) {
+ netdev_err(netdev,
+ "h_free_logical_lan still failing; LAN remains registered\n");
+ return -EIO;
+ }
+ ibmveth_release_lan_resources(adapter);
+ }
+
+ ibmveth_apply_mq_fallback(adapter);
+
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
rxq_entries += adapter->rx_buff_pool[0][i].size;
@@ -1747,16 +2091,38 @@ static int ibmveth_open(struct net_device *netdev)
goto out_free_queue_mem;
}
- rc = ibmveth_register_rx_queues(adapter, mac_address);
- if (rc)
+ rc = ibmveth_register_rx_queues(adapter, mac_address, &skip_unmap);
+ if (rc) {
+ if (skip_unmap) {
+ /* Nothing is posted yet: PHYP never saw the pools. */
+ ibmveth_free_buffer_pools(adapter);
+ goto out;
+ }
goto out_free_buffer_pools;
+ }
+
+ rc = netif_set_real_num_rx_queues(netdev,
+ ibmveth_get_num_rx_queues(adapter));
- rc = netif_set_real_num_rx_queues(netdev, adapter->num_rx_queues);
if (rc) {
netdev_err(netdev, "failed to set number of rx queues\n");
goto out_unregister_queues;
}
+ /*
+ * Post buffers before setup_rx_interrupts(). With more than one
+ * live queue, MQ setup then unmasks PHYP; otherwise PHYP stays
+ * masked. Scheduling NAPI only when a descriptor is already
+ * pending is not enough: after ifdown/up
+ * (RX=8, no -L) NAPI can be idle with nothing pending and the
+ * queue stays dead (TX OK, ARP/RX fail).
+ * restart_rx_queue() replenishes and schedules NAPI.
+ */
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
+ netdev_dbg(netdev, "initial replenish cycle for queue %d\n", i);
+ ibmveth_replenish_task(adapter, i);
+ }
+
/* Enables NAPI, so everything ibmveth_poll() touches must exist:
* netpoll during the direct close()/open() pairs, or a direct
* kick, can run it as soon as it is enabled.
@@ -1765,8 +2131,8 @@ static int ibmveth_open(struct net_device *netdev)
if (rc)
goto out_free_all_queues; /* setup already disposed IRQs */
- netdev_dbg(netdev, "initial replenish cycle\n");
- ibmveth_schedule_rx_queue(adapter, 0);
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+ ibmveth_restart_rx_queue(adapter, i);
rc = ibmveth_alloc_tx_resources(adapter);
if (rc)
@@ -1791,7 +2157,8 @@ static int ibmveth_open(struct net_device *netdev)
out_unregister_queues:
ibmveth_dispose_subordinate_irq_mappings(adapter);
out_free_all_queues:
- ibmveth_free_all_queues(adapter);
+ if (ibmveth_free_all_queues(adapter))
+ goto out;
out_free_buffer_pools:
ibmveth_free_buffer_pools(adapter);
out_free_queue_mem:
@@ -1805,6 +2172,7 @@ static int ibmveth_open(struct net_device *netdev)
static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
+ int i;
/* change_mtu, pool sysfs, set_csum and set_tso call close() and
* open() directly. If that open() fails, IFF_UP stays set and
@@ -1820,13 +2188,12 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_cleanup_rx_interrupts(adapter);
- ibmveth_free_all_queues(adapter);
- ibmveth_update_rx_no_buffer(adapter);
- /* netif_tx_disable() above keeps xmit off the TX LTBs. */
- ibmveth_free_tx_resources(adapter);
- ibmveth_free_buffer_pools(adapter);
- ibmveth_cleanup_rx_resources(adapter);
- ibmveth_free_filter_list(adapter);
+ if (!ibmveth_free_all_queues(adapter)) {
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+ ibmveth_update_rx_no_buffer(adapter, i);
+ /* netif_tx_disable() above keeps xmit off the TX LTBs. */
+ ibmveth_release_lan_resources(adapter);
+ }
netdev_dbg(netdev, "close complete\n");
@@ -2135,6 +2502,14 @@ static int ibmveth_set_channels(struct net_device *netdev,
goal = channels->tx_count;
int rc, i, alloc_rc = 0;
+ /*
+ * RX channel resize is implemented later; reject any request
+ * that changes rx_count. Read-modify-write TX adjustments
+ * submit the current rx_count and proceed.
+ */
+ if (channels->rx_count != ibmveth_get_num_rx_queues(adapter))
+ return -EOPNOTSUPP;
+
/* If the device is not open (including a failed close/open with
* IFF_UP still set) then don't allocate, just set desired
* netdev_queue's and return
@@ -2677,6 +3052,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
ibmveth_rx_csum_helper(skb, adapter);
}
+ skb_record_rx_queue(skb, queue_index);
napi_gro_receive(napi, skb);
netdev->stats.rx_packets++;
@@ -2695,7 +3071,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
queue_index = napi - adapter->napi;
if (WARN_ON(queue_index < 0 ||
- queue_index >= adapter->num_rx_queues)) {
+ queue_index >= ibmveth_get_num_rx_queues(adapter))) {
if (budget)
napi_complete_done(napi, 0);
return 0;
@@ -2795,10 +3171,11 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
qindex = napi - adapter->napi;
/*
- * Quiet on out-of-range: teardown can leave a residual IRQ after the
- * live count drops. Do not WARN-storm; return IRQ_NONE until free_irq.
+ * Quiet on out-of-range: a live-count shrink publishes the lower
+ * count before free_irq(). A residual IRQ must not WARN-storm;
+ * return IRQ_NONE until the handler is removed.
*/
- if (qindex < 0 || qindex >= adapter->num_rx_queues)
+ if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
return IRQ_NONE;
ibmveth_schedule_rx_queue(adapter, qindex);
@@ -2917,8 +3294,7 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
struct ibmveth_adapter *adapter;
struct iommu_table *tbl;
unsigned long ret;
- int i;
- int rxqentries = 1;
+ int i, q;
tbl = get_iommu_table_base(&vdev->dev);
@@ -2928,23 +3304,38 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
adapter = netdev_priv(netdev);
- ret = IBMVETH_BUFF_LIST_SIZE + IBMVETH_FILT_LIST_SIZE;
+ /* One buffer list page per RX queue; filter list is shared. */
+ ret = IBMVETH_BUFF_LIST_SIZE * ibmveth_get_num_rx_queues(adapter) +
+ IBMVETH_FILT_LIST_SIZE;
ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
/* add size of mapped tx buffers */
ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- /* add the size of the active receive buffers */
- if (adapter->rx_buff_pool[0][i].active)
- ret +=
- adapter->rx_buff_pool[0][i].size *
- IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[0][i].
- buff_size, tbl);
- rxqentries += adapter->rx_buff_pool[0][i].size;
- }
- /* add the size of the receive queue entries */
- ret += IOMMU_PAGE_ALIGN(
- rxqentries * sizeof(struct ibmveth_rx_q_entry), tbl);
+ /*
+ * Pool metadata for queues 1+ is copied from queue 0 at open.
+ * Always size from pool 0 x num_rx_queues.
+ *
+ * CMO (Power9 and earlier) and MQ firmware (Power11+) do not
+ * coexist, so a CMO partition always sizes one queue here. The
+ * MQ terms are defensive only.
+ */
+ for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
+ int rxqentries = 1;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[0][i];
+
+ if (bpool->active)
+ ret += bpool->size *
+ IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
+ rxqentries += bpool->size;
+ }
+
+ /* add the size of the receive queue entries */
+ ret += IOMMU_PAGE_ALIGN(rxqentries *
+ sizeof(struct ibmveth_rx_q_entry), tbl);
+ }
return ret;
}
@@ -3025,24 +3416,26 @@ static void ibmveth_pool_kobj_release(struct kobject *kobj)
/**
* ibmveth_put_pool_kobjs - Drop the pool kobjects and wait for release
* @adapter: ibmveth adapter
+ * @pools_ready: number of queue-0 pool kobjects that were added
*
* With CONFIG_DEBUG_KOBJECT_RELEASE the cleanup, including removing the
* sysfs files, runs later from a work item in the kobject; wait for it
* so free_netdev() cannot free the pools first.
*/
-static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter)
+static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,
+ int pools_ready)
{
int i;
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
+ for (i = 0; i < pools_ready; i++)
kobject_put(&adapter->rx_buff_pool[0][i].kobj);
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
+ for (i = 0; i < pools_ready; i++)
wait_for_completion(&adapter->rx_buff_pool[0][i].released);
}
static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
{
- int rc, i, mac_len;
+ int rc, i, mac_len, pools_ready = 0;
struct net_device *netdev;
struct ibmveth_adapter *adapter;
unsigned char *mac_addr_p;
@@ -3077,7 +3470,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
return -EINVAL;
}
- netdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter), IBMVETH_MAX_QUEUES, 1);
+ netdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter),
+ IBMVETH_MAX_QUEUES, IBMVETH_MAX_RX_QUEUES);
if (!netdev)
return -ENOMEM;
@@ -3099,7 +3493,9 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
ibmveth_init_link_settings(netdev);
- netif_napi_add_weight(netdev, &adapter->napi[0], ibmveth_poll, 16);
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+ netif_napi_add_weight(netdev, &adapter->napi[i],
+ ibmveth_poll, 16);
netdev->irq = dev->irq;
netdev->netdev_ops = &ibmveth_netdev_ops;
@@ -3131,16 +3527,30 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
netdev->features |= NETIF_F_FRAGLIST;
}
- /* Initialize queue count - always 1 for now */
- adapter->multi_queue = 0;
- adapter->num_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;
+ if (ret == H_SUCCESS &&
+ (ret_attr & IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT)) {
+ adapter->multi_queue = true;
+ ibmveth_publish_num_rx_queues(adapter,
+ min(num_online_cpus(),
+ IBMVETH_DEFAULT_QUEUES));
+ netdev_dbg(netdev, "RX multi queue mode enabled: %u queues\n",
+ ibmveth_get_num_rx_queues(adapter));
+ } else {
+ adapter->multi_queue = false;
+ ibmveth_publish_num_rx_queues(adapter,
+ IBMVETH_DEFAULT_RX_QUEUES);
+ }
if (ret == H_SUCCESS &&
(ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
- adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
+ if (adapter->multi_queue)
+ adapter->rx_buffers_per_hcall =
+ IBMVETH_MAX_RX_PER_HCALL;
+ else
+ adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
netdev_dbg(netdev,
"RX Multi-buffer hcall supported by FW, batch set to %u\n",
- adapter->rx_buffers_per_hcall);
+ adapter->rx_buffers_per_hcall);
} else {
adapter->rx_buffers_per_hcall = 1;
netdev_dbg(netdev,
@@ -3158,16 +3568,27 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
struct kobject *kobj = &adapter->rx_buff_pool[0][i].kobj;
- int error;
ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
pool_count[i], pool_size[i],
pool_active[i]);
init_completion(&adapter->rx_buff_pool[0][i].released);
- error = kobject_init_and_add(kobj, &ktype_veth_pool,
- &dev->dev.kobj, "pool%d", i);
- if (!error)
- kobject_uevent(kobj, KOBJ_ADD);
+ rc = kobject_init_and_add(kobj, &ktype_veth_pool,
+ &dev->dev.kobj, "pool%d", i);
+ if (rc) {
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[0][i];
+
+ dev_err(&dev->dev,
+ "failed to create pool%d kobject: %d\n", i, rc);
+ /* init_and_add takes a ref even on failure */
+ kobject_put(kobj);
+ wait_for_completion(&pool->released);
+ goto err_put_pools;
+ }
+
+ pools_ready++;
+ kobject_uevent(kobj, KOBJ_ADD);
}
rc = netif_set_real_num_tx_queues(netdev, min(num_online_cpus(),
@@ -3177,6 +3598,21 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
rc);
goto err_put_pools;
}
+
+ /*
+ * alloc_etherdev_mqs() sized real_num_rx_queues to MAX. Match the
+ * advertised default (or SQ 1) before register_netdev so down-state
+ * readers agree with adapter->num_rx_queues / ethtool -l.
+ */
+ rc = netif_set_real_num_rx_queues(netdev,
+ ibmveth_get_num_rx_queues(adapter));
+
+ if (rc) {
+ netdev_dbg(netdev, "failed to set number of rx queues rc=%d\n",
+ rc);
+ goto err_put_pools;
+ }
+
adapter->tx_ltb_size = PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE);
for (i = 0; i < IBMVETH_MAX_QUEUES; i++)
adapter->tx_ltb_ptr[i] = NULL;
@@ -3198,7 +3634,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
return 0;
err_put_pools:
- ibmveth_put_pool_kobjs(adapter);
+ ibmveth_put_pool_kobjs(adapter, pools_ready);
free_netdev(netdev);
return rc;
}
@@ -3210,10 +3646,18 @@ static void ibmveth_remove(struct vio_dev *dev)
disable_work_sync(&adapter->work);
- ibmveth_put_pool_kobjs(adapter);
+ ibmveth_put_pool_kobjs(adapter, IBMVETH_NUM_BUFF_POOLS);
unregister_netdev(netdev);
+ if (ibmveth_lan_resources_live(adapter)) {
+ if (ibmveth_free_all_queues(adapter))
+ dev_err(&dev->dev,
+ "h_free_logical_lan still failing at remove; leaking LAN buffers\n");
+ else
+ ibmveth_release_lan_resources(adapter);
+ }
+
free_netdev(netdev);
dev_set_drvdata(&dev->dev, NULL);
}
@@ -3255,8 +3699,11 @@ static ssize_t veth_pool_show(struct kobject *kobj,
* Return:
* * %-EPERM - Not allowed to disabled all MTU-sized buffer pools
* * %-EINVAL - New pool size or buffer size is out of range
+ * * %-EBUSY - The old pools are still mapped after a failed
+ * H_FREE_LOGICAL_LAN; nothing was changed
* * count - Return count for success
- * * other - Return value from a failed ibmveth_open call
+ * * other - Return value from a failed ibmveth_open call; the new
+ * geometry stays if that left its pools mapped
*/
static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
const char *buf, size_t count)
@@ -3338,6 +3785,18 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
if (netif_running(netdev))
ibmveth_close(netdev);
+ /* close() keeps the pools mapped when H_FREE_LOGICAL_LAN
+ * fails; open() or remove() frees them later with
+ * pool->size and pool->buff_size, so the geometry must not
+ * change until they are gone.
+ */
+ if (ibmveth_lan_resources_live(adapter)) {
+ rc = -EBUSY;
+ if (netif_running(netdev))
+ rc = ibmveth_open(netdev) ?: -EBUSY;
+ goto unlock_err;
+ }
+
pool->active = newactive;
pool->buff_size = newbuff_size;
pool->size = newsize;
@@ -3345,9 +3804,14 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
if (netif_running(netdev)) {
rc = ibmveth_open(netdev);
if (rc) {
- pool->active = oldactive;
- pool->buff_size = oldbuff_size;
- pool->size = oldsize;
+ /* A failed open() can keep the new pools
+ * mapped; their free needs this geometry.
+ */
+ if (!ibmveth_lan_resources_live(adapter)) {
+ pool->active = oldactive;
+ pool->buff_size = oldbuff_size;
+ pool->size = oldsize;
+ }
goto unlock_err;
}
}
@@ -3363,7 +3827,6 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
return rc;
}
-
#define ATTR(_name, _mode) \
struct attribute veth_##_name##_attr = { \
.name = __stringify(_name), .mode = _mode, \
@@ -3396,8 +3859,12 @@ static int ibmveth_resume(struct device *dev)
{
struct net_device *netdev = dev_get_drvdata(dev);
struct ibmveth_adapter *adapter = netdev_priv(netdev);
+ unsigned int num = ibmveth_get_num_rx_queues(adapter);
+ int i;
+
+ for (i = 0; i < num; i++)
+ ibmveth_schedule_rx_queue(adapter, i);
- ibmveth_schedule_rx_queue(adapter, 0);
return 0;
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index bdc8ea7069ec..393d37add10c 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -31,6 +31,7 @@
#define IbmVethMcastRemoveFilter 0x2UL
#define IbmVethMcastClearFilterTable 0x3UL
+#define IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT 0x0000000000080000UL
#define IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT 0x0000000000040000UL
#define IBMVETH_ILLAN_LRG_SR_ENABLED 0x0000000000010000UL
#define IBMVETH_ILLAN_LRG_SND_SUPPORT 0x0000000000008000UL
@@ -270,7 +271,7 @@ static inline long h_illan_attributes(unsigned long unit_address,
#define IBMVETH_MAX_TX_BUF_SIZE (1024 * 64)
#define IBMVETH_MAX_QUEUES 16U
#define IBMVETH_DEFAULT_QUEUES 8U
-#define IBMVETH_MAX_RX_QUEUES 1U
+#define IBMVETH_MAX_RX_QUEUES 16U
#define IBMVETH_DEFAULT_RX_QUEUES 1U
#define IBMVETH_MAX_RX_REGULAR 8U
#define IBMVETH_MAX_RX_PER_HCALL 12U
@@ -330,8 +331,10 @@ struct ibmveth_adapter {
u64 queue_handle[IBMVETH_MAX_RX_QUEUES];
bool queue_registered[IBMVETH_MAX_RX_QUEUES];
unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
- int multi_queue;
+ char queue_irq_name[IBMVETH_MAX_RX_QUEUES][32];
+ bool multi_queue;
unsigned int num_rx_queues;
+ bool mq_fallback;
/* Lifetime: true while RX IRQ handlers / NAPI are installed. */
bool rx_irq_setup;
int rx_csum;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 11/15] ibmveth: Add per-queue RX and TX statistics collection
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (9 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 10/15] ibmveth: Enable multi-queue RX receive path Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 12/15] ibmveth: Report MQ-aware RX counts in ethtool get_channels Mingming Cao
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
MQ RX points several queues at the same adapter-wide counters, which
races the updates and leaves no way to attribute a count to a queue.
The patch has three parts:
- storage: every counter with more than one writer moves into
per-queue structs, allocated at probe and freed at remove, with
a single writer per slot.
- export: packets, bytes and hardware drops are reported per queue
through netdev_stat_ops; ndo_get_stats64() sums every allocated
slot (live and retired after shrink) into the device totals;
ethtool -S keeps its thirteen keys (summed on read) and adds a few
driver-specific per-queue keys.
- rx_no_buffer: each queue carries the drops from buffer-list
pages it has already released, so the count is cumulative across
a reopen and never goes backwards.
The per-queue structs are:
struct ibmveth_rx_queue_stats
struct ibmveth_tx_queue_stats
Each slot has a single writer -- replenish_* under that queue's
replenish_lock, interrupts from that queue's IRQ handler, the
other RX fields from its NAPI (polls is not counted for
netpoll's budget-0 call), TX under
the stack's per-queue TX lock -- so u64 needs no atomic and no
u64_stats_sync on this PPC64-only driver. Readers run locklessly on
other CPUs, so writers go through WRITE_ONCE() (IBMVETH_STAT_INC/ADD)
and readers use READ_ONCE(), which keeps KCSAN quiet and stops the
compiler from splitting or refetching the accesses.
packets, bytes and hardware RX drops go through struct
netdev_stat_ops (.get_queue_stats_rx, .get_queue_stats_tx,
.get_base_stats). rx-hw-drops is what PHYP dropped: its no-buffer
count and the slots it hands back marked invalid. The no-buffer count
alone is also rx-hw-drop-overruns. rx-alloc-fail is the replenish
skb allocation failures (replenish_no_mem). Driver software drops (bad
correlator, failed harvest, oversize, recycle-invalid) are not separate
netdev_stat_ops or ethtool -S keys; each queue's NAPI increments
rx_qstats[].dropped and ndo_get_stats64() sums that into rtnl
rx_dropped; the poll paths no longer bump netdev->stats.rx_dropped.
TX large sends stay in the existing tx_large_packets key only, with
no per-queue key and no tx-hw-gso-packets: whether anything segments
a large send depends on where the frame goes.
TX hw_drops is left unset: yaml tx-hw-drops is packets that
arrived at the device, and dropped_packets also counts
software xmit rejects. ndo_get_stats64 still sums those
into tx_dropped, same as SQ.
.ndo_get_stats64() sums the per-queue packet and byte counters into
64-bit device totals. ethtool -S keeps the driver-specific keys with
no qstats field of their own: interrupts, polls, large_packets and
invalid_buffers per RX queue (rx-hw-drops folds invalid slots in with
the no-buffer count); send_failures per TX queue.
ETH_SS_STATS becomes variable-length because that block
scales with the live queue count. Hypercall counters and pool%d_ keys
are not added: which hcall a batch picks is not ABI, and size/active
already have sysfs.
The thirteen existing ethtool -S keys keep their exact names, their
order and their adapter-wide values, summed from the per-queue slots
on read. The storage moved; that ABI did not, except that
rx_no_buffer becomes cumulative across a reopen (below). The four
replenish_* counters get per-queue storage but no per-queue key of
their own.
Holding that ABI while the storage moves needs the ethtool -S table
to record where each key lives. IBMVETH_STAT_OFF() could only express
an offset into struct ibmveth_adapter. Tag every entry with an enum
ibmveth_stat_src naming the struct it indexes: adapter-wide keys are
read directly, per-queue keys are summed across the slots by one pair
of offset-keyed helpers. That is what lets the field names change
while the key names do not (rx_invalid_buffer now reads
invalid_buffers, tx_send_failed reads send_failures, and the two
large_packets fields live in different structs). tx_map_failed still
reads from the adapter -- it has no writer, here or in mainline --
and the three fw_enabled_* keys are capability flags, not counters.
The per-queue keys come from their own tables with the counts derived
by ARRAY_SIZE(), so get_strings(), get_ethtool_stats() and
get_sset_count() cannot drift apart.
ndo_get_stats64() walks every allocated slot rather than only the live
queues, so device totals cannot go backwards when ethtool -L shrinks
the queue count. get_base_stats() therefore reports the retired-queue
remainder rather than zero; the core sums it with the live queues it
iterates itself. Zeroing would assert that the live-queue sum is
already complete. Every field the per-queue callbacks fill is also
initialised there, because netdev_nl_stats_add() drops a field from
the device total unless both sides set it.
Give every queue a no_buffer_retired carry. PHYP's drop counter is
absolute for the buffer-list page currently mapped, so a reopen or a
queue reuse restarts it near zero. Storing only the newest absolute in
adapter->rx_no_buffer meant whichever queue ran last won, and the
value could go backwards. The carry sits beside the no_buffer_drops it
accumulates from, so both belong to one queue. Whenever
free_all_queues() lets the caller unmap, fold the last sample into
retired and clear drops while the page is still mapped; as close()
did before, that includes a single-queue H_FREE failure. Live
replenish still stores the page absolute; if the reading drops, the
previous absolute is folded into retired first. Readers snapshot the pair
under that queue's replenish_lock so a page-reset cannot be counted
twice. Every exported no_buffer value sums the two: the adapter-wide
rx_no_buffer key, rx_missed_errors, and per-queue rx-hw-drops and
rx-hw-drop-overruns, so none goes backwards across a close.
remove() frees the per-queue statistics arrays after
unregister_netdev() and after its H_FREE_LOGICAL_LAN retry, which
retires into them; the reset work is already disabled at the top of
remove().
ibmveth_probe_cleanup() also cancels the reset work and clears the
vio drvdata before free_netdev(), so drvdata does not outlive the
netdev on a probe failure, which never reaches ibmveth_remove().
Readers do not test the arrays for NULL: both exist from before
register_netdev() until after unregister_netdev(), with the reset
work already disabled, and probe fails -ENOMEM if either allocation
does.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- the per-queue no_buffer carry is folded by free_all_queues() before
every unmap, not only when a lower sample is seen, and read under
replenish_lock, so no exported value goes backwards across a close
- counters use WRITE_ONCE()/READ_ONCE() (IBMVETH_STAT_INC/ADD)
- ndo_get_stats64() fills rx_dropped and rx_missed_errors; rx_qstats
dropped counts bad-correlator, failed-harvest, recycle-invalid and
oversize slots
- qstats: rx-hw-drop-overruns filled; TX hw_drops unset; rx%d_polls
skips budget 0
- ethtool -S drops tx%d_checksum_offload (counter removed),
rx%d_no_buffer_drops (in qstats) and tx%d_large_packets (the
summed tx_large_packets stays)
- remove() frees the qstats arrays after its H_FREE retry
Changes in v7:
- close() harvests no_buffer under replenish_lock
- remove() uses put_pool_kobjs so the wait lands
before free_netdev
- remove-path unregister/cancel reorder and reset gate
moved to P10 with the reset producer
- noted: -S / get_stats64 / hw_drops mapping stays;
irqsave / free_buffer_pool / netpoll_poll_disable
stay leftover
Changes in v6:
- wrap the qstats local in replenish (81 cols)
- replenish_* per-queue u64, summed on the existing adapter-wide keys;
no per-queue replenish key; no atomics
- netdev_stat_ops for packets/bytes/drops, not private -S strings
- enum ibmveth_stat_src so existing -S keys keep names while storage
moves; ARRAY_SIZE() for the per-queue key counts
- no hcall_* keys (buffer-submit and H_SEND_LOGICAL_LAN)
- drop the three pool%d_ keys (15 ethtool entries)
- drop the fifteen qstats NULL checks outside the allocators
(five in the RX hot path)
- per-queue no_buffer_retired carry
- get_base_stats() reports the retired-queue remainder
- gate reset on NETREG_REGISTERED
- noted: harvest no_buffer on -L shrink is patch 14
Changes in v5:
- Series renumber: mailed v4 10/14 stats -> tip P11 (P09 peel;
get_channels -> P12)
- rx_no_buffer_retired + sum MAX_* slots so adapter no-buffer / qstat
totals stay monotonic across reopen and channel shrink
- probe_cleanup: clear vio drvdata before free_netdev (CMO cannot see a
freed netdev on rebind)
- remove: unregister_netdev then cancel_work_sync (no UAF reset worker)
Changes in v4:
- Merge v3's separate RX and TX stats commits into one patch.
- Introduce rx_queue_stats / tx_qstats / NUM macros here (first use).
- Allocate/free qstats at probe/remove instead of open/close.
- Report adapter-level ethtool strings by summing per-queue counters on
read; drop aggregate_* helpers.
- Sum global rx_no_buffer across MQ queues into this statistics patch.
- Cacheline-align per-queue stats; derive field counts with offsetof so
alignment padding is not counted as a statistic.
- probe_cleanup() cancels reset work, puts pool kobjects via helper from
the prior patch, and frees qstats on probe failure paths.
- Keep plain u64 qstats like existing ibmveth / ibmvnic (PPC_PSERIES).
drivers/net/ethernet/ibm/ibmveth.c | 549 ++++++++++++++++++++++++++---
drivers/net/ethernet/ibm/ibmveth.h | 60 +++-
2 files changed, 542 insertions(+), 67 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index a0c3c06d2b9d..45adf6e9be09 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -38,6 +38,7 @@
#include <asm/firmware.h>
#include <net/tcp.h>
#include <net/ip6_checksum.h>
+#include <net/netdev_queues.h>
#include "ibmveth.h"
@@ -75,32 +76,101 @@ module_param(old_large_send, bool, 0444);
MODULE_PARM_DESC(old_large_send,
"Use old large send method on firmware that supports the new method");
+/**
+ * enum ibmveth_stat_src - where an ethtool -S counter is stored
+ * @IBMVETH_STAT_ADAPTER: plain u64 in struct ibmveth_adapter
+ * @IBMVETH_STAT_RX_QSUM: per-queue u64, summed over rx_qstats[]
+ * @IBMVETH_STAT_TX_QSUM: per-queue u64, summed over tx_qstats[]
+ * @IBMVETH_STAT_RX_NO_BUFFER: rx_qstats[] live-page absolute plus the
+ * absolutes carried over from pages the queue has already retired
+ *
+ * Counters live per-queue so multi-queue writers never share a field.
+ * The adapter is only ever read from ethtool, so summing there is free.
+ */
+enum ibmveth_stat_src {
+ IBMVETH_STAT_ADAPTER,
+ IBMVETH_STAT_RX_QSUM,
+ IBMVETH_STAT_TX_QSUM,
+ IBMVETH_STAT_RX_NO_BUFFER,
+};
+
struct ibmveth_stat {
char name[ETH_GSTRING_LEN];
- int offset;
+ enum ibmveth_stat_src src;
+ /* Offset into the struct named by @src. */
+ size_t off;
};
#define IBMVETH_STAT_OFF(stat) offsetof(struct ibmveth_adapter, stat)
+#define IBMVETH_RXQ_OFF(stat) offsetof(struct ibmveth_rx_queue_stats, stat)
+#define IBMVETH_TXQ_OFF(stat) offsetof(struct ibmveth_tx_queue_stats, stat)
+
+/* Single writer per counter; lockless readers use READ_ONCE(). */
+#define IBMVETH_STAT_ADD(stat, n) WRITE_ONCE((stat), (stat) + (n))
+#define IBMVETH_STAT_INC(stat) IBMVETH_STAT_ADD(stat, 1)
#define IBMVETH_GET_STAT(a, off) *((u64 *)(((unsigned long)(a)) + off))
+#define IBMVETH_ADAPTER_STAT(key, field) \
+ { key, IBMVETH_STAT_ADAPTER, IBMVETH_STAT_OFF(field) }
+#define IBMVETH_RXQ_STAT(key, field) \
+ { key, IBMVETH_STAT_RX_QSUM, IBMVETH_RXQ_OFF(field) }
+#define IBMVETH_TXQ_STAT(key, field) \
+ { key, IBMVETH_STAT_TX_QSUM, IBMVETH_TXQ_OFF(field) }
+
+/*
+ * Key names and their order are ABI. Do not reorder or rename; append
+ * only, and only when the counter is worth a permanent interface.
+ */
static struct ibmveth_stat ibmveth_stats[] = {
- { "replenish_task_cycles", IBMVETH_STAT_OFF(replenish_task_cycles) },
- { "replenish_no_mem", IBMVETH_STAT_OFF(replenish_no_mem) },
- { "replenish_add_buff_failure",
- IBMVETH_STAT_OFF(replenish_add_buff_failure) },
- { "replenish_add_buff_success",
- IBMVETH_STAT_OFF(replenish_add_buff_success) },
- { "rx_invalid_buffer", IBMVETH_STAT_OFF(rx_invalid_buffer) },
- { "rx_no_buffer", IBMVETH_STAT_OFF(rx_no_buffer) },
- { "tx_map_failed", IBMVETH_STAT_OFF(tx_map_failed) },
- { "tx_send_failed", IBMVETH_STAT_OFF(tx_send_failed) },
- { "fw_enabled_ipv4_csum", IBMVETH_STAT_OFF(fw_ipv4_csum_support) },
- { "fw_enabled_ipv6_csum", IBMVETH_STAT_OFF(fw_ipv6_csum_support) },
- { "tx_large_packets", IBMVETH_STAT_OFF(tx_large_packets) },
- { "rx_large_packets", IBMVETH_STAT_OFF(rx_large_packets) },
- { "fw_enabled_large_send", IBMVETH_STAT_OFF(fw_large_send_support) }
+ IBMVETH_RXQ_STAT("replenish_task_cycles", replenish_task_cycles),
+ IBMVETH_RXQ_STAT("replenish_no_mem", replenish_no_mem),
+ IBMVETH_RXQ_STAT("replenish_add_buff_failure",
+ replenish_add_buff_failure),
+ IBMVETH_RXQ_STAT("replenish_add_buff_success",
+ replenish_add_buff_success),
+ IBMVETH_RXQ_STAT("rx_invalid_buffer", invalid_buffers),
+ { "rx_no_buffer", IBMVETH_STAT_RX_NO_BUFFER, 0 },
+ IBMVETH_ADAPTER_STAT("tx_map_failed", tx_map_failed),
+ IBMVETH_TXQ_STAT("tx_send_failed", send_failures),
+ IBMVETH_ADAPTER_STAT("fw_enabled_ipv4_csum", fw_ipv4_csum_support),
+ IBMVETH_ADAPTER_STAT("fw_enabled_ipv6_csum", fw_ipv6_csum_support),
+ IBMVETH_TXQ_STAT("tx_large_packets", large_packets),
+ IBMVETH_RXQ_STAT("rx_large_packets", large_packets),
+ IBMVETH_ADAPTER_STAT("fw_enabled_large_send", fw_large_send_support),
+};
+
+/**
+ * struct ibmveth_qstat - a per-queue counter exposed through ethtool -S
+ * @fmt: key name, taking the queue index as its only argument
+ * @off: offset into the matching per-queue stats struct
+ *
+ * Driving the strings and the values from one table keeps the two in
+ * step; get_sset_count() derives its length from ARRAY_SIZE() so the
+ * three cannot drift apart.
+ */
+struct ibmveth_qstat {
+ const char *fmt;
+ size_t off;
+};
+
+/*
+ * packets, bytes, drops and no-buffer overruns are reported through
+ * netdev_stat_ops. The keys below stay driver-specific.
+ */
+static const struct ibmveth_qstat ibmveth_rx_qstat_keys[] = {
+ { "rx%d_interrupts", IBMVETH_RXQ_OFF(interrupts) },
+ { "rx%d_polls", IBMVETH_RXQ_OFF(polls) },
+ { "rx%d_large_packets", IBMVETH_RXQ_OFF(large_packets) },
+ { "rx%d_invalid_buffers", IBMVETH_RXQ_OFF(invalid_buffers) },
};
+static const struct ibmveth_qstat ibmveth_tx_qstat_keys[] = {
+ { "tx%d_send_failures", IBMVETH_TXQ_OFF(send_failures) },
+};
+
+#define IBMVETH_NUM_RX_QSTATS ARRAY_SIZE(ibmveth_rx_qstat_keys)
+#define IBMVETH_NUM_TX_QSTATS ARRAY_SIZE(ibmveth_tx_qstat_keys)
+
/* simple methods of getting data from the current rxq entry */
static u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,
int queue_index)
@@ -241,6 +311,60 @@ ibmveth_free_filter_list(struct ibmveth_adapter *adapter)
}
}
+/**
+ * ibmveth_alloc_rx_qstats - Allocate per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_rx_qstats(struct ibmveth_adapter *adapter)
+{
+ adapter->rx_qstats = kcalloc(IBMVETH_MAX_RX_QUEUES,
+ sizeof(*adapter->rx_qstats),
+ GFP_KERNEL);
+ if (!adapter->rx_qstats)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_rx_qstats - Free per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_rx_qstats(struct ibmveth_adapter *adapter)
+{
+ kfree(adapter->rx_qstats);
+ adapter->rx_qstats = NULL;
+}
+
+/**
+ * ibmveth_alloc_tx_qstats - Allocate per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_qstats(struct ibmveth_adapter *adapter)
+{
+ adapter->tx_qstats = kcalloc(IBMVETH_MAX_QUEUES,
+ sizeof(*adapter->tx_qstats),
+ GFP_KERNEL);
+ if (!adapter->tx_qstats)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_tx_qstats - Free per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_tx_qstats(struct ibmveth_adapter *adapter)
+{
+ kfree(adapter->tx_qstats);
+ adapter->tx_qstats = NULL;
+}
+
/**
* ibmveth_alloc_rx_queues - Allocate per-queue RX resources
* @adapter: ibmveth adapter structure
@@ -861,6 +985,14 @@ struct ibmveth_replenish_fail {
u32 batch;
};
+static void ibmveth_count_add_fail(struct ibmveth_rx_queue_stats *qstats,
+ bool no_mem)
+{
+ if (no_mem)
+ IBMVETH_STAT_INC(qstats->replenish_no_mem);
+ IBMVETH_STAT_INC(qstats->replenish_add_buff_failure);
+}
+
/* Replenish the buffers for a pool. The skbs are not skb_reserve()d:
* PHYP reports the offset of each received frame.
* Caller must hold the per-queue replenish_lock.
@@ -873,6 +1005,8 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
int queue_index,
struct ibmveth_replenish_fail *fail)
{
+ struct ibmveth_rx_queue_stats *qstats =
+ &adapter->rx_qstats[queue_index];
union ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};
u32 remaining = pool->size - atomic_read(&pool->available);
u64 correlators[IBMVETH_MAX_RX_PER_HCALL] = {0};
@@ -899,7 +1033,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
for (filled = 0; filled < min(remaining, batch); filled++) {
index = pool->free_map[free_index];
if (index == IBM_VETH_INVALID_MAP) {
- adapter->replenish_add_buff_failure++;
+ ibmveth_count_add_fail(qstats, false);
outcome = IBMVETH_REPLENISH_RESET_MAP;
break;
}
@@ -910,8 +1044,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
skb = netdev_alloc_skb(adapter->netdev,
pool->buff_size);
if (!skb) {
- adapter->replenish_no_mem++;
- adapter->replenish_add_buff_failure++;
+ ibmveth_count_add_fail(qstats, true);
break;
}
@@ -927,7 +1060,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
DMA_ATTR_NO_WARN);
if (dma_mapping_error(dev, dma_addr)) {
dev_kfree_skb_any(skb);
- adapter->replenish_add_buff_failure++;
+ ibmveth_count_add_fail(qstats, false);
break;
}
@@ -988,7 +1121,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
}
buffers_added += filled;
- adapter->replenish_add_buff_success += filled;
+ IBMVETH_STAT_ADD(qstats->replenish_add_buff_success, filled);
remaining -= filled;
memset(&descs, 0, sizeof(descs));
@@ -1011,7 +1144,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
pool->skbuff[index] = NULL;
}
}
- adapter->replenish_add_buff_failure += filled;
+ IBMVETH_STAT_ADD(qstats->replenish_add_buff_failure, filled);
if (lpar_rc == H_FUNCTION) {
if (adapter->multi_queue) {
@@ -1052,6 +1185,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
int queue_index)
{
+ struct ibmveth_rx_queue_stats *qstats;
__be64 *p;
u64 drops;
@@ -1063,7 +1197,45 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
drops = be64_to_cpup(p);
- adapter->rx_no_buffer = drops;
+ qstats = &adapter->rx_qstats[queue_index];
+
+ if (drops < qstats->no_buffer_drops)
+ IBMVETH_STAT_ADD(qstats->no_buffer_retired,
+ qstats->no_buffer_drops);
+ WRITE_ONCE(qstats->no_buffer_drops, drops);
+}
+
+/* Snapshot live+retired under replenish_lock; see retire_rx_no_buffer(). */
+static u64 ibmveth_rx_no_buffer_pair(struct ibmveth_adapter *adapter, int i)
+{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[i];
+ struct ibmveth_rx_queue_stats *qstats = &adapter->rx_qstats[i];
+ u64 drops, retired;
+
+ spin_lock_bh(&rxq->replenish_lock);
+ drops = qstats->no_buffer_drops;
+ retired = qstats->no_buffer_retired;
+ spin_unlock_bh(&rxq->replenish_lock);
+ return drops + retired;
+}
+
+/*
+ * Call whenever the caller is about to unmap the buffer-list page.
+ * Take the last sample, then fold it into retired so the next page
+ * cannot overwrite the total.
+ */
+static void ibmveth_retire_rx_no_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+ struct ibmveth_rx_queue_stats *qstats;
+
+ spin_lock_bh(&rxq->replenish_lock);
+ ibmveth_update_rx_no_buffer(adapter, queue_index);
+ qstats = &adapter->rx_qstats[queue_index];
+ IBMVETH_STAT_ADD(qstats->no_buffer_retired, qstats->no_buffer_drops);
+ WRITE_ONCE(qstats->no_buffer_drops, 0);
+ spin_unlock_bh(&rxq->replenish_lock);
}
/* replenish routine */
@@ -1088,7 +1260,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
spin_lock_bh(&rxq->replenish_lock);
- adapter->replenish_task_cycles++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].replenish_task_cycles);
for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
struct ibmveth_buff_pool *pool =
@@ -1875,6 +2047,9 @@ static void ibmveth_release_lan_resources(struct ibmveth_adapter *adapter)
* reason it cannot simply keep the memory mapped is that on a CMO
* partition vio_cmo_bus_remove() BUG()s on an outstanding entitlement.
*
+ * Whenever the caller will unmap, retire each queue's no_buffer
+ * sample first, while the page is still mapped.
+ *
* On return 0, clears registered flags and cookies of queues
* [0, num_rx_queues); queue_irq[] is released by
* ibmveth_cleanup_rx_interrupts() on close, or by
@@ -1900,6 +2075,9 @@ static int ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
return -EIO;
}
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+ ibmveth_retire_rx_no_buffer(adapter, i);
+
for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
ibmveth_clear_rx_queue_handle(adapter, i);
@@ -2172,7 +2350,6 @@ static int ibmveth_open(struct net_device *netdev)
static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- int i;
/* change_mtu, pool sysfs, set_csum and set_tso call close() and
* open() directly. If that open() fails, IFF_UP stays set and
@@ -2189,8 +2366,6 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_cleanup_rx_interrupts(adapter);
if (!ibmveth_free_all_queues(adapter)) {
- for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
- ibmveth_update_rx_no_buffer(adapter, i);
/* netif_tx_disable() above keeps xmit off the TX LTBs. */
ibmveth_release_lan_resources(adapter);
}
@@ -2453,22 +2628,100 @@ static int ibmveth_set_features(struct net_device *dev,
return rc1 ? rc1 : rc2;
}
-static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8 *data)
+/*
+ * Sum per-queue counters for rare ethtool reads. The hot paths only ever
+ * touch their own queue's slot, so nothing here needs an atomic; the cost
+ * of aggregation is paid by the reader instead (ibmvnic-style).
+ *
+ * Every slot is summed, not just the live ones, so that shrinking the
+ * queue count with ethtool -L cannot make a counter go backwards.
+ */
+static u64 ibmveth_sum_rx_qstat(struct ibmveth_adapter *adapter, size_t off)
{
+ u64 total = 0;
int i;
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ const u8 *q = (const u8 *)&adapter->rx_qstats[i];
+
+ total += READ_ONCE(*(const u64 *)(q + off));
+ }
+
+ return total;
+}
+
+static u64 ibmveth_sum_tx_qstat(struct ibmveth_adapter *adapter, size_t off)
+{
+ u64 total = 0;
+ int i;
+
+ for (i = 0; i < IBMVETH_MAX_QUEUES; i++) {
+ const u8 *q = (const u8 *)&adapter->tx_qstats[i];
+
+ total += READ_ONCE(*(const u64 *)(q + off));
+ }
+
+ return total;
+}
+
+static u64 ibmveth_ethtool_adapter_stat(struct ibmveth_adapter *adapter,
+ int index)
+{
+ const struct ibmveth_stat *stat = &ibmveth_stats[index];
+
+ switch (stat->src) {
+ case IBMVETH_STAT_RX_QSUM:
+ return ibmveth_sum_rx_qstat(adapter, stat->off);
+ case IBMVETH_STAT_TX_QSUM:
+ return ibmveth_sum_tx_qstat(adapter, stat->off);
+ case IBMVETH_STAT_RX_NO_BUFFER: {
+ u64 total = 0;
+ int i;
+
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+ total += ibmveth_rx_no_buffer_pair(adapter, i);
+ return total;
+ }
+ case IBMVETH_STAT_ADAPTER:
+ break;
+ }
+
+ return IBMVETH_GET_STAT(adapter, stat->off);
+}
+
+static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8 *data)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ u8 *p = data;
+ int i, j;
+
if (stringset != ETH_SS_STATS)
return;
- for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++, data += ETH_GSTRING_LEN)
- memcpy(data, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+ for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++) {
+ memcpy(p, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
+ }
+
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+ for (j = 0; j < IBMVETH_NUM_RX_QSTATS; j++)
+ ethtool_sprintf(&p, ibmveth_rx_qstat_keys[j].fmt, i);
+
+ for (i = 0; i < dev->real_num_tx_queues; i++)
+ for (j = 0; j < IBMVETH_NUM_TX_QSTATS; j++)
+ ethtool_sprintf(&p, ibmveth_tx_qstat_keys[j].fmt, i);
}
static int ibmveth_get_sset_count(struct net_device *dev, int sset)
{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+
switch (sset) {
case ETH_SS_STATS:
- return ARRAY_SIZE(ibmveth_stats);
+ return ARRAY_SIZE(ibmveth_stats) +
+ ibmveth_get_num_rx_queues(adapter) *
+ IBMVETH_NUM_RX_QSTATS +
+ dev->real_num_tx_queues * IBMVETH_NUM_TX_QSTATS;
default:
return -EOPNOTSUPP;
}
@@ -2477,11 +2730,27 @@ static int ibmveth_get_sset_count(struct net_device *dev, int sset)
static void ibmveth_get_ethtool_stats(struct net_device *dev,
struct ethtool_stats *stats, u64 *data)
{
- int i;
struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i, j, k;
for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++)
- data[i] = IBMVETH_GET_STAT(adapter, ibmveth_stats[i].offset);
+ data[i] = ibmveth_ethtool_adapter_stat(adapter, i);
+
+ for (j = 0; j < ibmveth_get_num_rx_queues(adapter); j++) {
+ const u8 *q = (const u8 *)&adapter->rx_qstats[j];
+
+ for (k = 0; k < IBMVETH_NUM_RX_QSTATS; k++)
+ data[i++] = READ_ONCE(*(const u64 *)
+ (q + ibmveth_rx_qstat_keys[k].off));
+ }
+
+ for (j = 0; j < dev->real_num_tx_queues; j++) {
+ const u8 *q = (const u8 *)&adapter->tx_qstats[j];
+
+ for (k = 0; k < IBMVETH_NUM_TX_QSTATS; k++)
+ data[i++] = READ_ONCE(*(const u64 *)
+ (q + ibmveth_tx_qstat_keys[k].off));
+ }
}
static void ibmveth_get_channels(struct net_device *netdev,
@@ -2605,8 +2874,10 @@ static int ibmveth_send(struct ibmveth_adapter *adapter,
}
static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
- struct net_device *netdev)
+ struct ibmveth_adapter *adapter,
+ int queue_num)
{
+ struct net_device *netdev = adapter->netdev;
struct ethhdr *ether_header;
int ret = 0;
@@ -2614,7 +2885,7 @@ static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
if (ether_addr_equal(ether_header->h_dest, netdev->dev_addr)) {
netdev_dbg(netdev, "veth doesn't support loopback packets, dropping packet.\n");
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(adapter->tx_qstats[queue_num].dropped_packets);
ret = -EOPNOTSUPP;
}
@@ -2625,20 +2896,23 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
+ struct ibmveth_tx_queue_stats *txs;
unsigned int desc_flags, total_bytes;
union ibmveth_buf_desc desc;
int i, queue_num = skb_get_queue_mapping(skb);
unsigned long mss = 0;
+ txs = &adapter->tx_qstats[queue_num];
+
/* TX LTBs are freed on shrink; a sender that picked its queue
* before the shrink can get here after the free.
*/
if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
- if (ibmveth_is_packet_unsupported(skb, netdev))
+ if (ibmveth_is_packet_unsupported(skb, adapter, queue_num))
goto out;
/* veth can't checksum offload UDP */
if (skb->ip_summed == CHECKSUM_PARTIAL &&
@@ -2649,7 +2923,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
skb_checksum_help(skb)) {
netdev_err(netdev, "tx: failed to checksum packet\n");
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
@@ -2672,7 +2946,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (skb->ip_summed == CHECKSUM_PARTIAL && skb_is_gso(skb)) {
if (adapter->fw_large_send_support) {
mss = (unsigned long)skb_shinfo(skb)->gso_size;
- adapter->tx_large_packets++;
+ IBMVETH_STAT_INC(txs->large_packets);
} else if (!skb_is_gso_v6(skb)) {
/* Put -1 in the IP checksum to tell phyp it
* is a largesend packet. Put the mss in
@@ -2681,7 +2955,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
ip_hdr(skb)->check = 0xffff;
tcp_hdr(skb)->check =
cpu_to_be16(skb_shinfo(skb)->gso_size);
- adapter->tx_large_packets++;
+ IBMVETH_STAT_INC(txs->large_packets);
}
}
@@ -2689,7 +2963,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (unlikely(skb->len > adapter->tx_ltb_size)) {
netdev_err(adapter->netdev, "tx: packet size (%u) exceeds ltb (%u)\n",
skb->len, adapter->tx_ltb_size);
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
memcpy(adapter->tx_ltb_ptr[queue_num], skb->data, skb_headlen(skb));
@@ -2706,7 +2980,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (unlikely(total_bytes != skb->len)) {
netdev_err(adapter->netdev, "tx: incorrect packet len copied into ltb (%u != %u)\n",
skb->len, total_bytes);
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
desc.fields.flags_len = desc_flags | skb->len;
@@ -2715,11 +2989,11 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
dma_wmb();
if (ibmveth_send(adapter, desc.desc, mss)) {
- adapter->tx_send_failed++;
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->send_failures);
+ IBMVETH_STAT_INC(txs->dropped_packets);
} else {
- netdev->stats.tx_packets++;
- netdev->stats.tx_bytes += skb->len;
+ IBMVETH_STAT_INC(txs->packets);
+ IBMVETH_STAT_ADD(txs->bytes, skb->len);
}
out:
@@ -2887,7 +3161,7 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
static void ibmveth_poll_bump_invalid(struct ibmveth_adapter *adapter,
int queue_index)
{
- adapter->rx_invalid_buffer++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].invalid_buffers);
}
static bool ibmveth_poll_stopping(struct net_device *netdev,
@@ -2909,7 +3183,7 @@ static bool ibmveth_poll_recycle_invalid(struct net_device *netdev,
rc = ibmveth_rxq_harvest_correlator(adapter, queue_index, true,
correlator);
if (unlikely(rc))
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].dropped);
return rc;
}
@@ -2922,7 +3196,7 @@ static void ibmveth_poll_skip_bad_correlator(struct net_device *netdev,
* the harvest logs the correlator when it fails.
*/
schedule_work(&adapter->work);
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].dropped);
ibmveth_rxq_harvest_correlator(adapter, queue_index, true, correlator);
}
@@ -2936,7 +3210,7 @@ static void ibmveth_poll_drop_oversize(struct net_device *netdev,
netdev_err(netdev,
"RX frame %u+%u exceeds buffer %u on queue %d, dropping\n",
off, len, room, queue_index);
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].dropped);
ibmveth_rxq_harvest_correlator(adapter, queue_index, true, correlator);
}
@@ -2955,6 +3229,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
struct net_device *netdev,
int queue_index)
{
+ struct ibmveth_rx_queue_stats *rxs = &adapter->rx_qstats[queue_index];
struct sk_buff *skb, *new_skb;
unsigned int room, off, len, pool;
struct ibmveth_buff_pool *bpool;
@@ -3013,7 +3288,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
correlator);
if (unlikely(rc)) {
dev_kfree_skb_any(new_skb);
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(rxs->dropped);
return -1;
}
skb = new_skb;
@@ -3021,7 +3296,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
rc = ibmveth_rxq_harvest_correlator(adapter, queue_index,
false, correlator);
if (unlikely(rc)) {
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(rxs->dropped);
return -1;
}
skb_reserve(skb, offset);
@@ -3044,7 +3319,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
iph_check == 0xffff) {
if (!ibmveth_rx_mss_helper(skb, mss, lrg_pkt))
csum_good = 0;
- adapter->rx_large_packets++;
+ IBMVETH_STAT_INC(rxs->large_packets);
}
if (csum_good) {
@@ -3055,8 +3330,8 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
skb_record_rx_queue(skb, queue_index);
napi_gro_receive(napi, skb);
- netdev->stats.rx_packets++;
- netdev->stats.rx_bytes += length;
+ IBMVETH_STAT_INC(rxs->packets);
+ IBMVETH_STAT_ADD(rxs->bytes, length);
return 1;
}
@@ -3083,6 +3358,10 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
return 0;
}
+ /* budget 0 is netpoll; polls stays single-writer. */
+ if (budget)
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].polls);
+
while (frames_processed < budget) {
if (ibmveth_poll_stopping(netdev, napi))
break;
@@ -3178,6 +3457,8 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
return IRQ_NONE;
+ IBMVETH_STAT_INC(adapter->rx_qstats[qindex].interrupts);
+
ibmveth_schedule_rx_queue(adapter, qindex);
return IRQ_HANDLED;
}
@@ -3383,6 +3664,133 @@ static netdev_features_t ibmveth_features_check(struct sk_buff *skb,
return vlan_features_check(skb, features);
}
+/**
+ * ibmveth_get_stats64 - Return aggregated per-queue statistics
+ * @dev: network device
+ * @stats: rtnl link statistics storage
+ *
+ * Sums per-queue rx_qstats and tx_qstats into the rtnl counters.
+ * rx_missed_errors is the PHYP no-buffer pair (live + retired),
+ * which if_link.h counts separately from rx_dropped.
+ * Walk the full allocated arrays (not the live queue count) so shrinking
+ * channels cannot make the totals go backwards.
+ */
+static void ibmveth_get_stats64(struct net_device *dev,
+ struct rtnl_link_stats64 *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i;
+
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ stats->rx_packets += READ_ONCE(adapter->rx_qstats[i].packets);
+ stats->rx_bytes += READ_ONCE(adapter->rx_qstats[i].bytes);
+ stats->rx_dropped += READ_ONCE(adapter->rx_qstats[i].dropped);
+ stats->rx_missed_errors +=
+ ibmveth_rx_no_buffer_pair(adapter, i);
+ }
+
+ for (i = 0; i < IBMVETH_MAX_QUEUES; i++) {
+ stats->tx_packets += READ_ONCE(adapter->tx_qstats[i].packets);
+ stats->tx_bytes += READ_ONCE(adapter->tx_qstats[i].bytes);
+ stats->tx_dropped +=
+ READ_ONCE(adapter->tx_qstats[i].dropped_packets);
+ }
+}
+
+static void ibmveth_get_queue_stats_rx(struct net_device *dev, int idx,
+ struct netdev_queue_stats_rx *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ u64 no_buffer;
+
+ stats->packets = READ_ONCE(adapter->rx_qstats[idx].packets);
+ stats->bytes = READ_ONCE(adapter->rx_qstats[idx].bytes);
+ /*
+ * Only frames PHYP itself dropped: the no-buffer pair (live page
+ * plus pages this queue already released) and slots PHYP handed
+ * back marked invalid. Frames the driver rejects count in
+ * rx_dropped instead.
+ */
+ no_buffer = ibmveth_rx_no_buffer_pair(adapter, idx);
+ stats->hw_drops = no_buffer +
+ READ_ONCE(adapter->rx_qstats[idx].invalid_buffers);
+ stats->hw_drop_overruns = no_buffer;
+ stats->alloc_fail = READ_ONCE(adapter->rx_qstats[idx].replenish_no_mem);
+}
+
+static void ibmveth_get_queue_stats_tx(struct net_device *dev, int idx,
+ struct netdev_queue_stats_tx *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+
+ stats->packets = READ_ONCE(adapter->tx_qstats[idx].packets);
+ stats->bytes = READ_ONCE(adapter->tx_qstats[idx].bytes);
+ /* TX hw_drops left unset: dropped_packets includes
+ * software xmit rejects that never reached PHYP.
+ */
+}
+
+/**
+ * ibmveth_get_base_stats - account for traffic not on a live queue
+ * @dev: network device
+ * @rx: RX base statistics storage
+ * @tx: TX base statistics storage
+ *
+ * get_queue_stats_{rx,tx}() only report queues the core still iterates,
+ * i.e. below real_num_{rx,tx}_queues, while ibmveth_get_stats64() walks
+ * the full arrays so device totals stay monotonic across a shrink.
+ * Report the retired-queue remainder here, otherwise qstats and
+ * rtnl_link_stats64 disagree by a delta that grows with every shrink.
+ * Zeroing would not be neutral: per netdev_stat_ops it asserts the
+ * per-queue sum is already exact.
+ *
+ * Bound the live side with real_num_*_queues rather than the adapter's
+ * own count, so the split lines up with the core's iteration exactly.
+ *
+ * Every field the per-queue callbacks fill must also be initialised
+ * here: netdev_nl_stats_add() starts the sum at NETDEV_STAT_NOT_SET and
+ * only accumulates while both sides are set, so a field left unset here
+ * is dropped from the device total even though the queues report it.
+ */
+static void ibmveth_get_base_stats(struct net_device *dev,
+ struct netdev_queue_stats_rx *rx,
+ struct netdev_queue_stats_tx *tx)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ unsigned int i;
+
+ rx->packets = 0;
+ rx->bytes = 0;
+ rx->alloc_fail = 0;
+ rx->hw_drops = 0;
+ rx->hw_drop_overruns = 0;
+ tx->packets = 0;
+ tx->bytes = 0;
+
+ for (i = dev->real_num_rx_queues; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ u64 no_buffer = ibmveth_rx_no_buffer_pair(adapter, i);
+
+ rx->packets += READ_ONCE(adapter->rx_qstats[i].packets);
+ rx->bytes += READ_ONCE(adapter->rx_qstats[i].bytes);
+ rx->hw_drops += no_buffer +
+ READ_ONCE(adapter->rx_qstats[i].invalid_buffers);
+ rx->hw_drop_overruns += no_buffer;
+ rx->alloc_fail +=
+ READ_ONCE(adapter->rx_qstats[i].replenish_no_mem);
+ }
+
+ for (i = dev->real_num_tx_queues; i < IBMVETH_MAX_QUEUES; i++) {
+ tx->packets += READ_ONCE(adapter->tx_qstats[i].packets);
+ tx->bytes += READ_ONCE(adapter->tx_qstats[i].bytes);
+ }
+}
+
+static const struct netdev_stat_ops ibmveth_stat_ops = {
+ .get_queue_stats_rx = ibmveth_get_queue_stats_rx,
+ .get_queue_stats_tx = ibmveth_get_queue_stats_tx,
+ .get_base_stats = ibmveth_get_base_stats,
+};
+
static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_open = ibmveth_open,
.ndo_stop = ibmveth_close,
@@ -3395,6 +3803,7 @@ static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_validate_addr = eth_validate_addr,
.ndo_set_mac_address = ibmveth_set_mac_addr,
.ndo_features_check = ibmveth_features_check,
+ .ndo_get_stats64 = ibmveth_get_stats64,
};
/**
@@ -3433,6 +3842,23 @@ static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,
wait_for_completion(&adapter->rx_buff_pool[0][i].released);
}
+static void ibmveth_probe_cleanup(struct ibmveth_adapter *adapter,
+ int pools_ready)
+{
+ struct net_device *netdev = adapter->netdev;
+
+ cancel_work_sync(&adapter->work);
+ ibmveth_put_pool_kobjs(adapter, pools_ready);
+
+ ibmveth_free_tx_qstats(adapter);
+ ibmveth_free_rx_qstats(adapter);
+ /* Probe failure never reaches ibmveth_remove(); clear before free
+ * so drvdata does not outlive the netdev.
+ */
+ dev_set_drvdata(&adapter->vdev->dev, NULL);
+ free_netdev(netdev);
+}
+
static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
{
int rc, i, mac_len, pools_ready = 0;
@@ -3497,9 +3923,16 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
netif_napi_add_weight(netdev, &adapter->napi[i],
ibmveth_poll, 16);
+ if (ibmveth_alloc_rx_qstats(adapter) ||
+ ibmveth_alloc_tx_qstats(adapter)) {
+ ibmveth_probe_cleanup(adapter, 0);
+ return -ENOMEM;
+ }
+
netdev->irq = dev->irq;
netdev->netdev_ops = &ibmveth_netdev_ops;
netdev->ethtool_ops = &netdev_ethtool_ops;
+ netdev->stat_ops = &ibmveth_stat_ops;
SET_NETDEV_DEV(netdev, &dev->dev);
netdev->hw_features = NETIF_F_SG;
if (vio_get_attribute(dev, "ibm,illan-options", NULL) != NULL) {
@@ -3634,8 +4067,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
return 0;
err_put_pools:
- ibmveth_put_pool_kobjs(adapter, pools_ready);
- free_netdev(netdev);
+ ibmveth_probe_cleanup(adapter, pools_ready);
return rc;
}
@@ -3658,6 +4090,9 @@ static void ibmveth_remove(struct vio_dev *dev)
ibmveth_release_lan_resources(adapter);
}
+ ibmveth_free_tx_qstats(adapter);
+ ibmveth_free_rx_qstats(adapter);
+
free_netdev(netdev);
dev_set_drvdata(&dev->dev, NULL);
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 393d37add10c..76640bd5598e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -283,6 +283,46 @@ static int pool_active[] = { 1, 1, 0, 0, 1};
#define IBM_VETH_INVALID_MAP ((u16)0xffff)
+/*
+ * Per-queue RX counters. No field has two concurrent writers:
+ * interrupts is written only from this queue's IRQ handler; polls,
+ * packets, bytes, large_packets, invalid_buffers and dropped only from
+ * its NAPI poll (polls not from netpoll's budget-0 call);
+ * replenish_* only under its replenish_lock; and no_buffer_drops and
+ * no_buffer_retired only under that lock. Readers snapshot that pair
+ * under the same lock. No atomic or u64_stats_sync is therefore
+ * needed: the driver is PPC64-only, so 64-bit loads and stores do not
+ * tear. Readers still run locklessly on other CPUs, so writers use
+ * IBMVETH_STAT_INC()/IBMVETH_STAT_ADD() and readers READ_ONCE(), which
+ * keeps the compiler from splitting or refetching the accesses.
+ */
+struct ibmveth_rx_queue_stats {
+ u64 packets;
+ u64 bytes;
+ u64 interrupts;
+ u64 polls;
+ u64 large_packets;
+ u64 invalid_buffers;
+ u64 dropped;
+ /* PHYP's per-page absolute drop count for the live page. */
+ u64 no_buffer_drops;
+ /* Absolutes from pages this queue has already retired. */
+ u64 no_buffer_retired;
+ u64 replenish_task_cycles;
+ u64 replenish_no_mem;
+ u64 replenish_add_buff_failure;
+ u64 replenish_add_buff_success;
+} ____cacheline_aligned_in_smp;
+
+/* Per-queue TX counters; serialized by the stack's per-queue TX lock. */
+struct ibmveth_tx_queue_stats {
+ u64 packets;
+ u64 bytes;
+ u64 large_packets;
+ u64 dropped_packets;
+ u64 send_failures;
+} ____cacheline_aligned_in_smp;
+
struct ibmveth_buff_pool {
u32 size;
u32 index;
@@ -347,17 +387,17 @@ struct ibmveth_adapter {
u64 fw_ipv6_csum_support;
u64 fw_ipv4_csum_support;
u64 fw_large_send_support;
- /* adapter specific stats */
- u64 replenish_task_cycles;
- u64 replenish_no_mem;
- u64 replenish_add_buff_failure;
- u64 replenish_add_buff_success;
- u64 rx_invalid_buffer;
- u64 rx_no_buffer;
+ /*
+ * Every other ethtool -S counter lives in rx_qstats/tx_qstats and is
+ * summed on read. tx_map_failed predates multi-queue, has never been
+ * updated by any code path, and is kept only so the key keeps
+ * reporting the zero userspace already sees.
+ */
u64 tx_map_failed;
- u64 tx_send_failed;
- u64 tx_large_packets;
- u64 rx_large_packets;
+
+ struct ibmveth_rx_queue_stats *rx_qstats;
+ struct ibmveth_tx_queue_stats *tx_qstats;
+
/* Ethtool settings */
u8 duplex;
u32 speed;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 12/15] ibmveth: Report MQ-aware RX counts in ethtool get_channels
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (10 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 11/15] ibmveth: Add per-queue RX and TX statistics collection Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 13/15] ibmveth: Expose per-queue buffer pool details via debugfs Mingming Cao
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
Report MQ-aware channel limits and counts from get_channels():
- max_rx is IBMVETH_MAX_RX_QUEUES when MQ firmware is enabled and
mq_fallback is not set, else the live rx_count
- rx_count tracks the driver's published num_rx_queues
Previously get_channels() mirrored netdev->real_num_rx_queues for both
fields, which under-reports max_rx on MQ firmware. Here every publish
of num_rx_queues is paired with a matching real_num, so rx_count does
not change yet; reading the published count keeps it right once the
resize patches can publish a count that real_num fails to reach.
This patch only reports. The set_channels() rx_count reject lives in
the MQ enable patch; the live RX set lands with the incremental resize
patch. Once mq_fallback is set but not yet applied, rx_count is still N
until the next open; max_rx follows rx_count rather than dropping
to 1, because the ethtool core rejects a TX-only -L that
resubmits rx_count above max_rx.
mq_fallback is set from the replenish path without RTNL and is now
read under RTNL here, so its accesses use READ_ONCE()/WRITE_ONCE(),
including the two writers the MQ enable patch added.
Keep this out of the stats patch: channel reporting is ethtool -l ABI,
independent of per-queue counters, so it can be reviewed and blamed on
its own.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- get_channels max_rx is the live rx_count when single queue or once
mq_fallback is set, not a fixed 1 (the v7 patch 15 cap moves here)
- mq_fallback uses READ_ONCE()/WRITE_ONCE(): replenish writes it
without RTNL
- the set_channels rx_count reject is now in patch 10
Changes in v7:
- reject rx_count change with -EOPNOTSUPP in
set_channels() until patch 14 implements resize
- noted: TX-only read-modify-write still passes;
max_rx stays here (reject, not reorder);
get_channels uses the adapter count,
get_base_stats uses real_num; RX max is
MAX_RX_QUEUES, not the TX CPU cap; P14
wires live rx_count resize; P15 is the
tip ABI (live rx_count, max_rx cap on
mq_fallback, max_tx at least the live tx_count)
Changes in v5:
- get_channels rx_count via get_num_rx_queues() (READ_ONCE consistency)
- New peel: get_channels from v4 stats -> tip P12 (14->15)
- get_channels: max_rx = MAX_RX_QUEUES when MQ else 1; rx_count =
get_num_rx_queues() (was mirroring real_num_rx_queues for both)
drivers/net/ethernet/ibm/ibmveth.c | 17 +++++++++++------
drivers/net/ethernet/ibm/ibmveth.h | 1 +
2 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 45adf6e9be09..1e2e845f8011 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1306,7 +1306,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
dev_err_ratelimited(&adapter->netdev->dev,
"MQ buffer add H_FUNCTION (q=%d, batch=%u), reset\n",
queue_index, fail.batch);
- adapter->mq_fallback = true;
+ WRITE_ONCE(adapter->mq_fallback, true);
schedule_work(&adapter->work);
}
@@ -2152,7 +2152,7 @@ ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address,
if (rc) {
/* Firmware MQ gone: fall back to SQ on next open. */
if (rc == -EOPNOTSUPP)
- adapter->mq_fallback = true;
+ WRITE_ONCE(adapter->mq_fallback, true);
goto err_unregister;
}
}
@@ -2192,10 +2192,10 @@ static void ibmveth_apply_mq_fallback(struct ibmveth_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
- if (!adapter->mq_fallback)
+ if (!READ_ONCE(adapter->mq_fallback))
return;
- adapter->mq_fallback = false;
+ WRITE_ONCE(adapter->mq_fallback, false);
if (!adapter->multi_queue && ibmveth_get_num_rx_queues(adapter) == 1)
return;
@@ -2756,11 +2756,16 @@ static void ibmveth_get_ethtool_stats(struct net_device *dev,
static void ibmveth_get_channels(struct net_device *netdev,
struct ethtool_channels *channels)
{
+ struct ibmveth_adapter *adapter = netdev_priv(netdev);
+
channels->max_tx = ibmveth_real_max_tx_queues();
channels->tx_count = netdev->real_num_tx_queues;
- channels->max_rx = netdev->real_num_rx_queues;
- channels->rx_count = netdev->real_num_rx_queues;
+ channels->rx_count = ibmveth_get_num_rx_queues(adapter);
+ if (adapter->multi_queue && !READ_ONCE(adapter->mq_fallback))
+ channels->max_rx = IBMVETH_MAX_RX_QUEUES;
+ else
+ channels->max_rx = channels->rx_count;
}
static int ibmveth_set_channels(struct net_device *netdev,
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 76640bd5598e..66288f8e8349 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -374,6 +374,7 @@ struct ibmveth_adapter {
char queue_irq_name[IBMVETH_MAX_RX_QUEUES][32];
bool multi_queue;
unsigned int num_rx_queues;
+ /* Set from replenish without RTNL: READ_ONCE()/WRITE_ONCE(). */
bool mq_fallback;
/* Lifetime: true while RX IRQ handlers / NAPI are installed. */
bool rx_irq_setup;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 13/15] ibmveth: Expose per-queue buffer pool details via debugfs
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (11 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 12/15] ibmveth: Report MQ-aware RX counts in ethtool get_channels Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 14/15] ibmveth: Implement incremental MQ RX queue resize Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 15/15] ibmveth: Complete set_channels down-path Mingming Cao
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
With multi-queue RX each queue owns its own set of five buffer pools,
so a 16-queue adapter has 80 of them. Nothing reports their runtime
state: sysfs exposes queue 0 only, and only as configuration, and no
ethtool key is per-pool. When RX drops under load, qstats
rx-hw-drop-overruns names the queue but not which of its pools ran
dry, nor how close the others are.
Add a read-only buffer_pools debugfs file, one row per RX queue and
buffer pool:
/sys/kernel/debug/ibmveth/<dev_name>/buffer_pools
(e.g. /sys/kernel/debug/ibmveth/30000002/buffer_pools)
Queue Pool Count BuffSize Active Available
Active is the poolN/active configuration flag. Active and Available
are shown only for a pool that is allocated (skbuff && free_map);
an unallocated pool shows 0 for both.
The root is driver-owned so each adapter directory can use its stable
vio name rather than the mutable netdev->name. It is created in
module_init() and unwound if vio_register_driver() fails.
A multi-line table does not belong in sysfs, so the historical queue-0
ABI is left alone:
.../poolN/{active,num,size}
Those stay one-value configuration for queue-0 pool classes. Open
copies that geometry to queues 1..N. This series does not add
per-queue pool sysfs dirs.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- no code change beyond the rebase
- commit message: per-queue drops are in qstats rx-hw-drop-overruns;
the Active column is the configured flag
Changes in v7:
- no P13 code change
Changes in v6:
- create the debugfs root in module_init() instead of lazily on
first probe, which raced concurrent probes and could orphan the
directory on ERR_PTR(-EEXIST)
- rename the pool buffer-count column from Size to Count, so the
debugfs table stops reusing the word sysfs poolN/size spells as a
byte length on the same pool object
- widen the down banner: geometry above queue 0 is only populated
once open copies the queue-0 template
- scope the dump RTNL comment to geometry/pool->active; available is
atomic_read
Changes in v5:
- debugfs buffer_pools_show walks get_num_rx_queues()
- Series renumber: mailed v4 11/14 debugfs -> tip P13 (14->15)
- Path uses stable vio dev_name under a driver-owned root (not netdev
name - avoids rename/collide)
- rtnl_lock around dump (writers are under RTNL)
- Show Active/Available as 0 when pool !live (debugfs view; free-path
available clear already in the buffer-submit patch)
Changes in v4:
- Move the all-queue buffer_pools diagnostic from sysfs to debugfs;
subject updated to match.
- Keep historical queue-0 poolN/{active,num,size} sysfs as one-value
config (template for MQ); do not add per-queue pool sysfs dirs.
drivers/net/ethernet/ibm/ibmveth.c | 80 +++++++++++++++++++++++++++++-
drivers/net/ethernet/ibm/ibmveth.h | 2 +
2 files changed, 81 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 1e2e845f8011..e805e7b1d2fe 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -31,6 +31,7 @@
#include <linux/ipv6.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
+#include <linux/debugfs.h>
#include <asm/hvcall.h>
#include <linux/atomic.h>
#include <asm/vio.h>
@@ -3811,6 +3812,67 @@ static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_get_stats64 = ibmveth_get_stats64,
};
+static int ibmveth_buffer_pools_show(struct seq_file *m, void *v)
+{
+ struct ibmveth_adapter *adapter = m->private;
+ int i, j;
+
+ /*
+ * size / buff_size / pool->active are written under RTNL
+ * (veth_pool_store, open template copy). Take the same lock so
+ * those columns are not a torn snapshot. available is updated
+ * from NAPI/softirq; only atomic_read() keeps it from tearing.
+ * Not required for memory safety; embedded arrays only.
+ */
+ rtnl_lock();
+
+ seq_puts(m, "Queue Pool Count BuffSize Active Available\n");
+ seq_puts(m, "----- ---- ----- -------- ------ ---------\n");
+ if (!adapter->opened) {
+ seq_puts(m, "# down: Active/Available 0 unless allocated\n");
+ seq_puts(m, "# down: geometry above queue 0 set at open\n");
+ }
+
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
+ for (j = 0; j < IBMVETH_NUM_BUFF_POOLS; j++) {
+ struct ibmveth_buff_pool *pool =
+ &adapter->rx_buff_pool[i][j];
+ bool live = pool->skbuff && pool->free_map;
+ int active = live ? pool->active : 0;
+ int available = live ? atomic_read(&pool->available)
+ : 0;
+
+ seq_printf(m, "%5d %4d %5u %8u %6d %9d\n",
+ i, j, pool->size, pool->buff_size,
+ active, available);
+ }
+ }
+
+ rtnl_unlock();
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(ibmveth_buffer_pools);
+
+/* Driver-owned root so per-adapter dirs use a stable vio name, not the
+ * mutable netdev->name (avoids stale names / eth0 collisions after rename).
+ */
+static struct dentry *ibmveth_dbg_root;
+
+static void ibmveth_debugfs_init(struct ibmveth_adapter *adapter)
+{
+ adapter->debugfs_dir =
+ debugfs_create_dir(dev_name(&adapter->vdev->dev),
+ ibmveth_dbg_root);
+ debugfs_create_file("buffer_pools", 0400, adapter->debugfs_dir,
+ adapter, &ibmveth_buffer_pools_fops);
+}
+
+static void ibmveth_debugfs_exit(struct ibmveth_adapter *adapter)
+{
+ debugfs_remove_recursive(adapter->debugfs_dir);
+ adapter->debugfs_dir = NULL;
+}
+
/**
* ibmveth_pool_kobj_release - Mark a pool kobject finished
* @kobj: kobject embedded in the pool
@@ -4069,6 +4131,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
netdev_dbg(netdev, "registered\n");
+ ibmveth_debugfs_init(adapter);
+
return 0;
err_put_pools:
@@ -4083,6 +4147,8 @@ static void ibmveth_remove(struct vio_dev *dev)
disable_work_sync(&adapter->work);
+ ibmveth_debugfs_exit(adapter);
+
ibmveth_put_pool_kobjs(adapter, IBMVETH_NUM_BUFF_POOLS);
unregister_netdev(netdev);
@@ -4329,15 +4395,27 @@ static struct vio_driver ibmveth_driver = {
static int __init ibmveth_module_init(void)
{
+ int rc;
+
printk(KERN_DEBUG "%s: %s %s\n", ibmveth_driver_name,
ibmveth_driver_string, ibmveth_driver_version);
- return vio_register_driver(&ibmveth_driver);
+ ibmveth_dbg_root = debugfs_create_dir(ibmveth_driver_name, NULL);
+
+ rc = vio_register_driver(&ibmveth_driver);
+ if (rc) {
+ debugfs_remove_recursive(ibmveth_dbg_root);
+ ibmveth_dbg_root = NULL;
+ }
+
+ return rc;
}
static void __exit ibmveth_module_exit(void)
{
vio_unregister_driver(&ibmveth_driver);
+ debugfs_remove_recursive(ibmveth_dbg_root);
+ ibmveth_dbg_root = NULL;
}
module_init(ibmveth_module_init);
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 66288f8e8349..31c2c72b2cff 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -399,6 +399,8 @@ struct ibmveth_adapter {
struct ibmveth_rx_queue_stats *rx_qstats;
struct ibmveth_tx_queue_stats *tx_qstats;
+ struct dentry *debugfs_dir;
+
/* Ethtool settings */
u8 duplex;
u32 speed;
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 14/15] ibmveth: Implement incremental MQ RX queue resize
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (12 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 13/15] ibmveth: Expose per-queue buffer pool details via debugfs Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
2026-11-09 23:55 ` [PATCH net-next v8 15/15] ibmveth: Complete set_channels down-path Mingming Cao
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
ethtool -L can now change the RX queue count while the device is up,
one queue at a time, without a full close/open:
- scale-up: for each new queue, allocate it, register it with PHYP,
set up its IRQ, then publish the count, replenish, enable NAPI
and unmask PHYP.
- scale-down: quiesce the retiring queues, drain them, publish the
smaller count, then deregister each with H_FREE_LOGICAL_LAN_QUEUE
and free it.
- failure: either direction unwinds to a consistent live count. A
queue that PHYP will not release stays mapped, either kept in the
live set or parked above it until the next H_FREE_LOGICAL_LAN.
- entry: set_channels() calls ibmveth_resize_rx_channels(), which
validates the request; a combined -L resizes RX first and rolls
RX back if TX fails. A change while down is still rejected until
the next patch.
Add ibmveth_resize_rx_queues_incremental() so the live RX channel
count can change without a full device tear-down, and
ibmveth_resize_rx_channels() as the entry point set_channels() calls.
The two directions live in ibmveth_scale_up_rx_queues() and
ibmveth_scale_down_rx_queues(); the incremental helper validates the
request, dispatches to one of them, and refreshes CMO on success.
Scale-up works one new queue at a time: allocate its DMA buffer list
and pool memory, register it with PHYP via H_REG_LOGICAL_LAN_QUEUE,
map its subordinate IRQ and request it, then bring it up: publish the
count, replenish, enable NAPI, and only then unmask PHYP, so the handler
cannot run on an unpublished, empty or NAPI-disabled queue. It
finishes with the same restart helper open() uses, so a total
replenish miss can still get a poll to retry.
Widen netif_set_real_num_rx_queues() to the target count before any
new queue is unmasked, so a frame on a new queue cannot carry an rx
queue index above real_num_rx_queues. Scale-up failure restores
real_num to the surviving count.
Scale-down masks PHYP on the retiring queues first, then disables NAPI,
waits for the poll tail with synchronize_net(), then masks and
synchronises again: napi_disable() does not wait for a poll past
napi_complete_done(), which can still re-arm PHYP; napi_disable() also
keeps netpoll's poll_napi() off the retiring queues. The scale-up
unwind does the same. Scale-down then drains the ring,
publishes the surviving count with smp_store_release() and calls
synchronize_net(), then netif_set_real_num_rx_queues().
Once real_num is lowered, it deregisters via
H_FREE_LOGICAL_LAN_QUEUE, takes the queue's final no_buffer sample
under its replenish_lock while the page is still mapped, releases
the IRQ mapping and frees the DMA and pool memory. That last sample
skips the live-count bound the replenish path uses, since the count
was already lowered. Deregistration has to precede the unmap so PHYP
releases ownership of the buffers while the queue metadata its
correlators index is still valid. If the free
hcall fails, leave the handle set and skip the unmap.
H_PARAMETER from H_FREE_LOGICAL_LAN_QUEUE means the unit has no
logical LAN or the handle names no registered queue (already freed),
as the PHYP interface defines it and the wrapper kdoc records: clear
the handle and unmap. Destroy walks high-to-low so a failed free keeps
0..i live, publishes that count,
and restores NAPI/IRQ on the queues it could not remove.
Scale-up register failure matches that skip-unmap rule: if IRQ
mapping fails after H_REG and the matching H_FREE fails, keep the
handle so the caller does not unmap. Scale-up enable_irq failure
that cannot destroy the new queue keeps that queue in the live set
with the queues this -L already added, instead of publishing below
it and orphaning the mapping, and sizes CMO desired for that count.
A scale-up register or IRQ-setup failure whose H_FREE also fails
leaves that queue above the live count with queue_registered[]
set (the cookie may be 0). ibmveth_free_stranded_rx_queues()
frees such slots after a successful H_FREE_LOGICAL_LAN, which drops
every subordinate queue, retiring each one's no_buffer sample first:
in close(), and in the open() and remove() H_FREE retry that the MQ
enable patch added for a hard H_FREE_LOGICAL_LAN failure. Until then
a scale-up that reaches that index returns -EBUSY.
Whenever the caller is going to unmap a live queue, its final
no_buffer sample is taken first. The scale-up register and IRQ-setup
unwinds free a queue they registered moments earlier without that
sample, so drops PHYP counted in its page in that window are not
added.
As in the MQ enable patch, a hard H_FREE_LOGICAL_LAN failure keeps
the LAN memory mapped only in multi-queue mode; single-queue keeps
the base close() behaviour of logging and unmapping (one reason it
cannot simply keep the memory mapped is the vio_cmo_bus_remove()
BUG() on outstanding IO entitlement on a CMO partition). The new
scale-up H_FUNCTION fallback below latches mq_fallback, and the next
open() then clears multi_queue, which puts the adapter on that
single-queue rule.
Both directions unwind. Scale-up failures jump to cleanup_new_queues:,
which restores real_num to the surviving count and destroys only the
queues this call created. Scale-down failures republish the remaining
count, then replenish, napi_enable and unmask each retiring queue that
stays (and restart it if the IRQ comes back). If the IRQ cannot be
re-enabled they schedule a reset.
If growing real_num back to the surviving count fails after it
was lowered, skip the unmask; still napi_enable so close/reset
is not a second napi_disable hang. Reset is already queued. The
scale-down path that failed to lower real_num still re-enables:
those indexes stay in range.
Use buffer-list allocation presence, not DMA-address truthiness, to
decide whether an incrementally removed queue needs
dma_unmap_single(): DMA address zero is valid.
Scale-up register failure returning -EOPNOTSUPP latches
mq_fallback so future scale-ups are blocked immediately. Reject
rx > 1 with -EOPNOTSUPP when firmware lacks MQ support or
mq_fallback is set. That rejection lives here rather than in
set_channels(), and it sits below the no-op check: ethtool -L is
read-modify-write, so a TX-only request arrives carrying the current
RX count and must not be rejected once mq_fallback is set. An rx_count
outside 1..IBMVETH_MAX_RX_QUEUES is rejected with -EINVAL ahead of both
checks; the ethtool core already range-checks against the max_rx that
get_channels() reports, and this is the driver's own guard.
Refresh CMO entitlement across the resize. ibmveth_desired_dma_for_rxqs()
is factored out of ibmveth_get_desired_dma() so the same math can be
applied to a prospective queue count before the driver commits to it.
A combined ethtool -L resizes RX first. If TX does not reach
the request, restore the previous RX count (best effort: a failed restore
is logged) and return rc ? rc : -ENOMEM so the call cannot report
success with RX committed and TX unchanged. An RX count change
while not opened is
rejected with -EOPNOTSUPP here rather than reported as success and
discarded; the following commit publishes the down-path count and
lifts the rejection.
The keep path can leave queues above real_num_rx_queues with NAPI
still enabled and PHYP masked. Bound every path that could reach them
by real_num: poll records an RX queue index only below it, and poll
completion does not enable_irq above it (that would undo the skipped
unmask); interrupt() returns IRQ_NONE above it rather than scheduling
NAPI; resume() walks min(num_rx_queues, real_num_rx_queues).
ibmveth_schedule_rx_queue() skips an index at or past the live count
without a WARN, since a shrink can race resume()'s walk.
Deregister and the scale-up -EBUSY check key off
queue_registered[], not a zero handle. Handle 0 can still be
a live cookie.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- scale-down and the scale-up unwinds: synchronize_net() after
napi_disable(), then remask; if set_real(keep) fails after real_num
was lowered, skip the unmask but still napi_enable()
- a combined ethtool -L rolls RX back, best effort, if TX fails (moved
here from patch 15)
- H_FREE_LOGICAL_LAN_QUEUE H_PARAMETER (already free) clears the slot
and unmaps
- a slot stranded by a scale-up failure is freed (no_buffer retired
first) after the next successful H_FREE_LOGICAL_LAN in close(),
open() or remove()
- deregister and the scale-up -EBUSY check key on queue_registered[],
not a zero handle; scale-down retires no_buffer after a successful
H_FREE_LOGICAL_LAN_QUEUE
- keep path: poll, completion, interrupt and resume honour real_num;
CMO desired is refreshed before -EIO
- scale-up unmasks only through enable_irq(), as restart_rx_queue() no
longer unmasks (patch 10)
Dropped from this series (merged fixes own them):
- commit 8d9ecac6b96e: comments that cited ndo_poll_controller
- commit 95811c873d1b: set_channels IFF_UP -> adapter->opened. It
already gates set_channels() on opened
Changes in v7:
- drain_rx_queue kdoc and debug log: slots
processed, not buffers recycled
- skip unmap when H_FREE_LOGICAL_LAN_QUEUE fails;
keep the handle and restore remaining queues
- widen set_real before scale-up unmask;
restore real_num on scale-up failure
- scale-up register fail: keep handle and skip
unmap if H_FREE after IRQ map fails
- scale-up enable_irq fail: if destroy fails,
keep that queue in the live set
- scale-up register / IRQ-setup failure with a failed
H_FREE: close() frees that queue after successful
H_FREE_LOGICAL_LAN; a scale-up to it returns -EBUSY
- scale-up register site latches mq_fallback
on -EOPNOTSUPP
- scale-up enable_irq failure: remask PHYP after
napi_disable(), as scale-down does
- CMO desired sites share ibmveth_refresh_cmo_desired();
desired_dma_for_rxqs comment notes CMO and MQ do not
coexist
- free_single_rx_queue() keys buffer-list unmap on
allocation presence because DMA address zero is valid
- reject an RX count change while down with -EOPNOTSUPP
until patch 15 publishes the down-path count
- split resize_rx_queues_incremental() into
scale_up_rx_queues() / scale_down_rx_queues() plus a
dispatcher; code motion only
- schedule_rx_queue kdoc: past the live count is quiet
- noted: mask-then-napi_disable stays;
no PHYP mask ack; no reset on failed
H_FREE (skip unmap); shrinks to 2..live-1
stay -EOPNOTSUPP once mq_fallback
Changes in v6:
- harvest no_buffer on retiring queues under that queue's
replenish_lock before publish: netpoll still reaches them until
the surviving count is stored, and update_rx_no_buffer() ignores
indexes at or above the live count
- publish num_rx_queues, then synchronize_net(), then destroy.
Scale-down runs set_real_num_rx_queues() between sync and destroy
(destroy only if that succeeds); cleanup_new_queues does not harvest
- napi_disable() before publishing the lowered count on scale-up
enable_irq failure, matching scale-down
- restart_rx_queue() after the checked enable_irq() on scale-up and
on scale-down rollback; kick_rx_queue_if_pending() cannot retry a
queue whose replenish posted nothing
- drop the scale-down survivor restart_rx_queue() loop: those queues
were never quiesced, and enable_irq() on prep-fail unmasks behind
a live poll (subordinate H_PARAMETER on enable is -EIO and reset)
- schedule_rx_queue() returns quietly when only the upper bound
trips: netpoll walks a snapshot of the old count, so an ethtool -L
shrink raced it into a WARN userspace can trigger at will
- drop leftover spin_lock_init() from alloc_single_rx_queue(); scale-up
runs on a live adapter and probe already inits the lock
- zero buffer_list_dma[] on the alloc_single_rx_queue() map-failure
path; free_single_rx_queue() keys its unmap on that field
- skip PHYP mask and remask on scale-down when queue_irq[i] is 0
- put the rx > 1 / mq_fallback gate below the no-op check, and refuse
any rx > 1 once mq_fallback is set, so a TX-only read-modify-write
is not rejected
- noted: down-path stash/CMO and RX rollback on TX fail are patch 15
- noted: KUnit free_map fixtures are patch 8
Changes in v5:
- resize / get_desired_dma read live count via get_num_rx_queues()
- Scale-down / scale-up-fail: remask+sync after napi_disable (poll may
re-arm while disable waits)
- Series renumber: mailed v4 12/14 resize -> tip P14 (14->15)
- Absorb kitchen-sink resize ownership: teardown-first scale-down
(drain/deregister before unmap) + live-pool correlator_valid /
schedule_work (cover has peel map)
- Ordered num_rx_queues publish + publish-before-destroy
- After unmask: kick if pending; enable_irq errno + rollback check;
raise CMO before scale-up allocs
- Scale-up enable_irq-fail: publish-down / napi_disable / drain / destroy
- After scale-down destroy, restart remaining RX queues so Q0 cannot stay
masked with NAPI idle
- resize_rx_channels: reject rx>1 without MQ / validate 1..MAX; call
before !opened early return (stash still tip P15)
Changes in v4:
- Copy pool->index when cloning buffer pools for incrementally added
queues.
- Scale-up order: publish -> replenish -> napi_enable -> enable_irq
(vs older enable-before-publish drafts); mirror NAPI-before-unmask on
set_real_num_rx rollback.
- Linux-owned subordinate virq disposal; deregister is PHYP-only;
dispose bound to MAX_RX_QUEUES; dispose on request_irq failure.
- Drain-path smp_rmb() before harvest.
- Mask PHYP before napi_disable/drain on scale-down and scale-up fail
cleanup.
- Replenish before re-enabling IRQ/NAPI on set_real_num_rx scale-down
rollback.
- Keep set_channels wiring as the following patch (same split as v3)
but call resize_rx_channels() here when IFF_UP so the helper is not
an unused static.
- wrap close() TX-LTB free comment to 80 cols (two-step)
drivers/net/ethernet/ibm/ibmveth.c | 1152 ++++++++++++++++++++++++++--
1 file changed, 1078 insertions(+), 74 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index e805e7b1d2fe..510d9d80e2df 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -807,25 +807,90 @@ ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
adapter->rx_irq_setup = false;
}
+/**
+ * ibmveth_setup_single_rx_interrupt - Setup interrupt for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to setup
+ *
+ * Registers the IRQ handler for one queue. Used during incremental
+ * scale-up when adding new RX queues. The caller publishes the queue,
+ * replenishes buffers, enables NAPI, then unmasks PHYP delivery.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_setup_single_rx_interrupt(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ struct net_device *netdev = adapter->netdev;
+ int rc;
+
+ rc = request_irq(adapter->queue_irq[queue_idx], ibmveth_interrupt,
+ 0, ibmveth_queue_irq_name(adapter, queue_idx),
+ &adapter->napi[queue_idx]);
+ if (rc) {
+ netdev_err(netdev, "request_irq() failed for queue %d: %d\n",
+ queue_idx, rc);
+ return rc;
+ }
+
+ netdev_dbg(netdev, "Setup IRQ %d for queue %d\n",
+ adapter->queue_irq[queue_idx], queue_idx);
+ return 0;
+}
+
+/**
+ * ibmveth_cleanup_single_rx_interrupt - Cleanup interrupt for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to cleanup
+ *
+ * Frees the IRQ handler for one queue and releases the subordinate virq
+ * mapping. Used during incremental scale-down and on scale-up
+ * destroy via ibmveth_destroy_subordinate_rx_queue().
+ */
+static void
+ibmveth_cleanup_single_rx_interrupt(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ if (adapter->queue_irq[queue_idx]) {
+ free_irq(adapter->queue_irq[queue_idx],
+ &adapter->napi[queue_idx]);
+ ibmveth_dispose_subordinate_irq_mapping(adapter, queue_idx);
+ netdev_dbg(adapter->netdev,
+ "Freed IRQ for queue %d\n", queue_idx);
+ }
+}
+
/**
* ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue
* @adapter: ibmveth adapter structure
* @qindex: RX queue index
*
* Shared by the IRQ handler and process-context kick sites (open, resume,
- * pool sysfs).
+ * pool sysfs, and ethtool -L scale-up, keep and rollback through
+ * ibmveth_restart_rx_queue()).
*
- * Does nothing if the index is out of range (WARN_ON) or
- * napi_schedule_prep() fails (NAPI already scheduled, or disabled
- * because the device is closed). A failed
- * disable_irq() is logged by the helper; the queue may stay unmasked.
+ * Does nothing if the index is negative (WARN_ON), past the live count
+ * (quiet: a shrink can race resume()), or napi_schedule_prep() fails
+ * (NAPI already scheduled, or disabled because the device is closed).
+ * A failed disable_irq() is logged by the helper; the queue may stay
+ * unmasked.
*/
static void ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
int qindex)
{
struct napi_struct *napi = &adapter->napi[qindex];
- if (WARN_ON(qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter)))
+ if (WARN_ON(qindex < 0))
+ return;
+
+ /*
+ * A live shrink can publish a lower count while resume() walks a
+ * snapshot of the old one, so an index past the end is expected
+ * here and must not splat. ibmveth_replenish_task() skips the
+ * same way.
+ */
+ if (qindex >= ibmveth_get_num_rx_queues(adapter))
return;
/*
@@ -1181,18 +1246,17 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
/*
* The final 8 bytes of the buffer list is a counter of frames dropped
* because there was not a buffer in the buffer list capable of holding
- * the frame.
+ * the frame. No live-count bound: retire samples a queue scale-down has
+ * already unpublished but not yet freed.
*/
-static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
+static void ibmveth_sample_rx_no_buffer(struct ibmveth_adapter *adapter,
int queue_index)
{
struct ibmveth_rx_queue_stats *qstats;
__be64 *p;
u64 drops;
- if (queue_index < 0 ||
- queue_index >= ibmveth_get_num_rx_queues(adapter) ||
- !adapter->buffer_list_addr[queue_index])
+ if (!adapter->buffer_list_addr[queue_index])
return;
p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
@@ -1206,6 +1270,15 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
WRITE_ONCE(qstats->no_buffer_drops, drops);
}
+static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ if (queue_index < 0 ||
+ queue_index >= ibmveth_get_num_rx_queues(adapter))
+ return;
+ ibmveth_sample_rx_no_buffer(adapter, queue_index);
+}
+
/* Snapshot live+retired under replenish_lock; see retire_rx_no_buffer(). */
static u64 ibmveth_rx_no_buffer_pair(struct ibmveth_adapter *adapter, int i)
{
@@ -1220,10 +1293,16 @@ static u64 ibmveth_rx_no_buffer_pair(struct ibmveth_adapter *adapter, int i)
return drops + retired;
}
-/*
- * Call whenever the caller is about to unmap the buffer-list page.
- * Take the last sample, then fold it into retired so the next page
- * cannot overwrite the total.
+/**
+ * ibmveth_retire_rx_no_buffer - Fold the page drop counter before unmap
+ * @adapter: ibmveth adapter
+ * @queue_index: RX queue index
+ *
+ * Sample the buffer-list tail while the page is still mapped, then move
+ * the live absolute into retired. Call whenever the caller is about to
+ * unmap the buffer-list page.
+ * ibmveth_sample_rx_no_buffer() has no live-count bound so scale-down
+ * can sample queues already unpublished.
*/
static void ibmveth_retire_rx_no_buffer(struct ibmveth_adapter *adapter,
int queue_index)
@@ -1232,7 +1311,7 @@ static void ibmveth_retire_rx_no_buffer(struct ibmveth_adapter *adapter,
struct ibmveth_rx_queue_stats *qstats;
spin_lock_bh(&rxq->replenish_lock);
- ibmveth_update_rx_no_buffer(adapter, queue_index);
+ ibmveth_sample_rx_no_buffer(adapter, queue_index);
qstats = &adapter->rx_qstats[queue_index];
IBMVETH_STAT_ADD(qstats->no_buffer_retired, qstats->no_buffer_drops);
WRITE_ONCE(qstats->no_buffer_drops, 0);
@@ -1339,8 +1418,9 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
* completes, so do not unmask here: while NAPI is scheduled the handler
* cannot remask, and PHYP would interrupt once per frame. Replenish
* first so the queue has buffers to deliver into.
- * Also used after every open (SQ and MQ) so a queue is not left
- * idle+masked.
+ * Also used after every open (SQ and MQ), after each successful
+ * scale-up queue bring-up, and on the scale-up keep and
+ * scale-down rollback paths.
*/
static void ibmveth_restart_rx_queue(struct ibmveth_adapter *adapter,
int qindex)
@@ -1546,6 +1626,142 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
ibmveth_get_num_rx_queues(adapter));
}
+/**
+ * ibmveth_alloc_single_rx_queue - Allocate resources for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to allocate
+ * @rxq_entries: Number of RX queue entries
+ *
+ * Allocates buffer list, RX queue, and per-queue buffer pools for one queue.
+ * Used during incremental scale-up without affecting existing queues.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx,
+ int rxq_entries)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
+ int i, rc = -ENOMEM;
+
+ adapter->buffer_list_addr[queue_idx] =
+ (void *)get_zeroed_page(GFP_KERNEL);
+ if (!adapter->buffer_list_addr[queue_idx]) {
+ netdev_err(netdev, "unable to allocate buffer list for queue %d\n",
+ queue_idx);
+ return -ENOMEM;
+ }
+
+ adapter->rx_queue[queue_idx].queue_len =
+ sizeof(struct ibmveth_rx_q_entry) * rxq_entries;
+ adapter->rx_queue[queue_idx].queue_addr =
+ dma_alloc_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+ &adapter->rx_queue[queue_idx].queue_dma,
+ GFP_KERNEL);
+ if (!adapter->rx_queue[queue_idx].queue_addr) {
+ netdev_err(netdev, "unable to allocate RX queue for queue %d\n",
+ queue_idx);
+ goto out_free_buflist;
+ }
+
+ adapter->buffer_list_dma[queue_idx] =
+ dma_map_single(dev, adapter->buffer_list_addr[queue_idx],
+ 4096, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, adapter->buffer_list_dma[queue_idx])) {
+ netdev_err(netdev, "unable to map buffer list for queue %d\n",
+ queue_idx);
+ adapter->buffer_list_dma[queue_idx] = 0;
+ goto out_free_rxq;
+ }
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *src =
+ &adapter->rx_buff_pool[0][i];
+ struct ibmveth_buff_pool *dst =
+ &adapter->rx_buff_pool[queue_idx][i];
+
+ dst->size = src->size;
+ dst->index = src->index;
+ dst->buff_size = src->buff_size;
+ dst->threshold = src->threshold;
+ dst->active = src->active;
+ }
+
+ rc = ibmveth_alloc_queue_buffer_pools(adapter, queue_idx);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to allocate buffer pools for queue %d\n",
+ queue_idx);
+ goto out_unmap_buflist;
+ }
+
+ adapter->rx_queue[queue_idx].index = 0;
+ adapter->rx_queue[queue_idx].num_slots = rxq_entries;
+ adapter->rx_queue[queue_idx].toggle = 1;
+
+ netdev_dbg(netdev,
+ "Allocated queue %d: buffer_list @ %p (DMA: 0x%llx), rx_queue @ %p (DMA: 0x%llx), %d entries\n",
+ queue_idx, adapter->buffer_list_addr[queue_idx],
+ (unsigned long long)adapter->buffer_list_dma[queue_idx],
+ adapter->rx_queue[queue_idx].queue_addr,
+ (unsigned long long)adapter->rx_queue[queue_idx].queue_dma,
+ rxq_entries);
+
+ return 0;
+
+out_unmap_buflist:
+ dma_unmap_single(dev, adapter->buffer_list_dma[queue_idx],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[queue_idx] = 0;
+out_free_rxq:
+ dma_free_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+ adapter->rx_queue[queue_idx].queue_addr,
+ adapter->rx_queue[queue_idx].queue_dma);
+ adapter->rx_queue[queue_idx].queue_addr = NULL;
+out_free_buflist:
+ free_page((unsigned long)adapter->buffer_list_addr[queue_idx]);
+ adapter->buffer_list_addr[queue_idx] = NULL;
+ return rc;
+}
+
+/**
+ * ibmveth_free_single_rx_queue - Free resources for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to free
+ *
+ * Frees buffer list, RX queue, and per-queue buffer pools for one queue.
+ * Used by scale-down, the scale-up unwind and the stranded-queue
+ * cleanup; other queues are not touched.
+ */
+static void
+ibmveth_free_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx)
+{
+ struct device *dev = &adapter->vdev->dev;
+
+ ibmveth_free_queue_buffer_pools(adapter, queue_idx);
+
+ if (adapter->buffer_list_addr[queue_idx]) {
+ dma_unmap_single(dev, adapter->buffer_list_dma[queue_idx],
+ 4096, DMA_BIDIRECTIONAL);
+ adapter->buffer_list_dma[queue_idx] = 0;
+ }
+
+ if (adapter->rx_queue[queue_idx].queue_addr) {
+ dma_free_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+ adapter->rx_queue[queue_idx].queue_addr,
+ adapter->rx_queue[queue_idx].queue_dma);
+ adapter->rx_queue[queue_idx].queue_addr = NULL;
+ }
+
+ if (adapter->buffer_list_addr[queue_idx]) {
+ free_page((unsigned long)adapter->buffer_list_addr[queue_idx]);
+ adapter->buffer_list_addr[queue_idx] = NULL;
+ }
+
+ netdev_dbg(adapter->netdev, "Freed queue %d resources\n", queue_idx);
+}
+
/* The correlator comes back from PHYP, so check it rather than trust it.
* Does not log: remove_buffer_from_pool() calls it under replenish_lock.
*/
@@ -1717,6 +1933,65 @@ static int ibmveth_rxq_harvest_correlator(struct ibmveth_adapter *adapter,
return rc;
}
+static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
+ int queue_index, bool reuse)
+{
+ return ibmveth_rxq_harvest_correlator(adapter, queue_index, reuse,
+ ibmveth_rxq_read_correlator(adapter, queue_index));
+}
+
+/**
+ * ibmveth_drain_rx_queue - Drain pending buffers from an RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Queue index to drain
+ *
+ * Harvests pending completions back to the per-queue buffer pools.
+ * A corrupt slot still advances the ring, so the return is slots
+ * processed, not buffers recycled.
+ * Must be called with NAPI disabled for this queue.
+ *
+ * Return: number of slots processed
+ */
+static int
+ibmveth_drain_rx_queue(struct ibmveth_adapter *adapter, int queue_index)
+{
+ struct net_device *netdev = adapter->netdev;
+ int drained = 0;
+ int limit = adapter->rx_queue[queue_index].num_slots;
+ int rc;
+
+ netdev_dbg(netdev, "Draining RX queue %d (limit: %d slots)\n",
+ queue_index, limit);
+
+ while (drained < limit &&
+ ibmveth_rxq_pending_buffer(adapter, queue_index)) {
+ /* Match poll-side order before harvesting completion state. */
+ smp_rmb();
+ rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);
+ if (rc) {
+ /* -EINVAL/-EFAULT already advanced past the slot. */
+ if (rc == -EINVAL || rc == -EFAULT) {
+ drained++;
+ continue;
+ }
+ netdev_err(netdev,
+ "Failed to harvest buffer from queue %d during drain: %d\n",
+ queue_index, rc);
+ break;
+ }
+ drained++;
+ }
+
+ if (drained > 0)
+ netdev_dbg(netdev, "Drained %d slot(s) from RX queue %d\n",
+ drained, queue_index);
+ else
+ netdev_dbg(netdev, "No slots to drain from RX queue %d\n",
+ queue_index);
+
+ return drained;
+}
+
static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
{
void *ltb = adapter->tx_ltb_ptr[idx];
@@ -1881,7 +2156,8 @@ static long ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
* Registers a subordinate receive queue using H_REG_LOGICAL_LAN_QUEUE.
* On success, stores the queue handle and virtual IRQ in the adapter.
* If IRQ mapping fails after a successful hypervisor registration, the
- * queue is freed before returning.
+ * queue is freed before returning. If that free fails, the handle is
+ * kept so the caller can skip unmap.
*
* Return: H_SUCCESS on success, negative errno on IRQ mapping failure,
* hypervisor error code otherwise
@@ -1921,10 +2197,20 @@ ibmveth_register_logical_lan_queue(struct ibmveth_adapter *adapter,
free_rc = h_free_logical_lan_queue(ua, handle);
} while (H_IS_LONG_BUSY(free_rc) ||
(free_rc == H_BUSY));
- if (free_rc != H_SUCCESS)
+ if (free_rc != H_SUCCESS &&
+ free_rc != H_PARAMETER) {
netdev_err(adapter->netdev,
"h_free_logical_lan_queue failed for queue %d after IRQ map failure: rc=0x%lx\n",
queue_index, free_rc);
+ /*
+ * PHYP still owns this queue. Keep the
+ * handle so the caller skips unmap.
+ */
+ ibmveth_set_rx_queue_handle(adapter,
+ queue_index,
+ handle);
+ return -EIO;
+ }
return -EINVAL;
}
@@ -1940,9 +2226,10 @@ ibmveth_register_logical_lan_queue(struct ibmveth_adapter *adapter,
/*
* H_FUNCTION means firmware rejected this subordinate register
- * (MQ unsupported / dropped after LPM). Caller fails this open and
- * latches mq_fallback so the next open applies SQ; keep a specific
- * log then the generic failure lines below.
+ * (MQ unsupported / dropped after LPM). The caller (open or
+ * ethtool -L scale-up) fails and latches mq_fallback so the next
+ * open applies SQ; keep a specific log then the generic failure
+ * lines below.
*/
if (lpar_rc == H_FUNCTION)
netdev_err(adapter->netdev,
@@ -2028,6 +2315,644 @@ static void ibmveth_release_lan_resources(struct ibmveth_adapter *adapter)
ibmveth_free_filter_list(adapter);
}
+/**
+ * ibmveth_deregister_single_rx_queue - Deregister one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to deregister (1..N)
+ *
+ * Deregisters a single queue via H_FREE_LOGICAL_LAN_QUEUE. Linux IRQ handler
+ * teardown and subordinate virq mapping disposal are owned by interrupt
+ * cleanup helpers; queue 0 is freed only through ibmveth_free_all_queues()
+ * (H_FREE_LOGICAL_LAN).
+ *
+ * Return: 0 if the queue was freed or was not registered, -EIO if
+ * H_FREE_LOGICAL_LAN_QUEUE failed. On -EIO the slot stays
+ * registered so the caller can skip unmap. Handle 0 is a
+ * live cookie; do not use it as "unregistered".
+ *
+ * H_PARAMETER is no-adapter or already-free, same as
+ * H_FREE_LOGICAL_LAN: clear the registered flag and return 0 so
+ * the caller can unmap.
+ */
+static int
+ibmveth_deregister_single_rx_queue(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ unsigned long lpar_rc;
+ unsigned long ua = adapter->vdev->unit_address;
+ unsigned long qh = adapter->queue_handle[queue_idx];
+
+ if (!adapter->queue_registered[queue_idx])
+ return 0;
+
+ do {
+ lpar_rc = h_free_logical_lan_queue(ua, qh);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+
+ if (lpar_rc != H_SUCCESS) {
+ if (lpar_rc == H_PARAMETER) {
+ ibmveth_clear_rx_queue_handle(adapter, queue_idx);
+ return 0;
+ }
+ netdev_err(adapter->netdev,
+ "h_free_logical_lan_queue failed, queue %d rc=%ld\n",
+ queue_idx, lpar_rc);
+ return -EIO;
+ }
+
+ ibmveth_clear_rx_queue_handle(adapter, queue_idx);
+
+ netdev_dbg(adapter->netdev, "Deregistered queue %d\n", queue_idx);
+ return 0;
+}
+
+/**
+ * ibmveth_destroy_subordinate_rx_queue - Tear down one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to destroy (1..N)
+ *
+ * Deregister with PHYP before unmapping buffer pools so hypervisor buffer
+ * ownership is released while queue metadata is still valid. After a
+ * successful H_FREE, fold the last no_buffer sample into retired while
+ * the page is still mapped. If the hcall fails, leave the queue mapped,
+ * keep the handle, and skip retire.
+ *
+ * Return: 0 on success, -EIO if deregister failed
+ */
+static int
+ibmveth_destroy_subordinate_rx_queue(struct ibmveth_adapter *adapter,
+ int queue_idx)
+{
+ int rc;
+
+ rc = ibmveth_deregister_single_rx_queue(adapter, queue_idx);
+ if (rc)
+ return rc;
+ ibmveth_retire_rx_no_buffer(adapter, queue_idx);
+ ibmveth_cleanup_single_rx_interrupt(adapter, queue_idx);
+ ibmveth_free_single_rx_queue(adapter, queue_idx);
+ return 0;
+}
+
+/**
+ * ibmveth_desired_dma_for_rxqs - CMO entitlement for a given RX queue count
+ * @adapter: ibmveth adapter
+ * @rxqs: number of RX queues to size for
+ *
+ * Same math as ibmveth_get_desired_dma(), but uses @rxqs instead of the
+ * live adapter->num_rx_queues. Scale-up raises desired for the *target*
+ * count before allocating. CMO rebalances asynchronously, so an
+ * allocation can still fail; the scale-up unwind handles that.
+ *
+ * Return: bytes of IO memory desired for @rxqs RX queues
+ */
+static unsigned long
+ibmveth_desired_dma_for_rxqs(struct ibmveth_adapter *adapter,
+ unsigned int rxqs)
+{
+ struct net_device *netdev = adapter->netdev;
+ struct iommu_table *tbl;
+ unsigned long ret;
+ int i, q;
+
+ tbl = get_iommu_table_base(&adapter->vdev->dev);
+
+ ret = IBMVETH_BUFF_LIST_SIZE * rxqs + IBMVETH_FILT_LIST_SIZE;
+ ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
+ ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
+
+ /*
+ * Pool metadata for queues 1+ is copied from queue 0 at open.
+ * Always size from pool 0 x @rxqs.
+ *
+ * CMO (Power9 and earlier) and MQ firmware (Power11+) do not
+ * coexist, so a CMO partition always sizes one queue here. The
+ * MQ terms and the ethtool -L CMO refreshes are defensive only.
+ */
+ for (q = 0; q < rxqs; q++) {
+ int rxqentries = 1;
+
+ for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+ struct ibmveth_buff_pool *bpool =
+ &adapter->rx_buff_pool[0][i];
+
+ if (bpool->active)
+ ret += bpool->size *
+ IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
+ rxqentries += bpool->size;
+ }
+
+ ret += IOMMU_PAGE_ALIGN(rxqentries *
+ sizeof(struct ibmveth_rx_q_entry), tbl);
+ }
+
+ return ret;
+}
+
+/**
+ * ibmveth_free_stranded_rx_queues - Free queues stranded above the live count
+ * @adapter: ibmveth adapter structure
+ *
+ * A scale-up register or IRQ-setup failure whose H_FREE_LOGICAL_LAN_QUEUE
+ * also fails leaves that queue out of the live set with its slot still
+ * registered and its memory still mapped. Slots below the live count,
+ * including those a skip-unmap open fail left mapped, are released by
+ * ibmveth_release_lan_resources() instead. Call only after successful
+ * H_FREE_LOGICAL_LAN, which drops every subordinate queue, so the memory
+ * can be released. Retire each queue's no_buffer sample first, as
+ * free_all_queues() does for the live set.
+ */
+static void ibmveth_free_stranded_rx_queues(struct ibmveth_adapter *adapter)
+{
+ int i;
+
+ for (i = ibmveth_get_num_rx_queues(adapter);
+ i < IBMVETH_MAX_RX_QUEUES; i++) {
+ if (!adapter->buffer_list_addr[i] &&
+ !adapter->rx_queue[i].queue_addr)
+ continue;
+ ibmveth_retire_rx_no_buffer(adapter, i);
+ ibmveth_clear_rx_queue_handle(adapter, i);
+ ibmveth_free_single_rx_queue(adapter, i);
+ }
+}
+
+/* Set CMO desired entitlement from the live RX queue count. */
+static void ibmveth_refresh_cmo_desired(struct ibmveth_adapter *adapter)
+{
+ if (firmware_has_feature(FW_FEATURE_CMO))
+ vio_cmo_set_dev_desired(adapter->vdev,
+ ibmveth_get_desired_dma(adapter->vdev));
+}
+
+/**
+ * ibmveth_scale_up_rx_queues - Add RX queues to a live adapter
+ * @adapter: ibmveth adapter structure
+ * @old_count: Current number of RX queues
+ * @new_count: Target number of RX queues, above @old_count
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Raises CMO entitlement and widens real_num_rx_queues before any new
+ * queue is unmasked, then brings each queue up in turn. A failure
+ * unwinds only the queues this call added; a queue PHYP refuses to
+ * release stays in the live set instead.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_scale_up_rx_queues(struct ibmveth_adapter *adapter, int old_count,
+ int new_count, int rxq_entries)
+{
+ struct net_device *netdev = adapter->netdev;
+ int failed_queue;
+ int rc, i;
+
+ netdev_dbg(netdev, "Scale-up: adding queues %d-%d\n",
+ old_count, new_count - 1);
+
+ /*
+ * Raise CMO desired for the target count before dma_map /
+ * dma_alloc_coherent / replenish (same order as change_mtu).
+ * Do not bump live num_rx_queues here, only entitlement.
+ */
+ if (firmware_has_feature(FW_FEATURE_CMO)) {
+ unsigned long dma;
+
+ dma = ibmveth_desired_dma_for_rxqs(adapter, new_count);
+ vio_cmo_set_dev_desired(adapter->vdev, dma);
+ }
+
+ /*
+ * Widen real_num before any new queue is unmasked so
+ * skb_record_rx_queue() cannot trip get_rps_cpu()
+ * WARN_ONCE. Probe already allocated num_rx_queues.
+ */
+ rc = netif_set_real_num_rx_queues(netdev, new_count);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to set real RX queues to %d: %d\n",
+ new_count, rc);
+ ibmveth_refresh_cmo_desired(adapter);
+ return rc;
+ }
+
+ for (i = old_count; i < new_count; i++) {
+ if (adapter->queue_registered[i]) {
+ /*
+ * Stranded by a failed deregister, which queued a
+ * reset. Its close frees the queue after H_FREE;
+ * if that fails, the next open retries.
+ */
+ netdev_err(netdev,
+ "RX queue %d still held by PHYP, reset pending\n",
+ i);
+ rc = -EBUSY;
+ goto cleanup_new_queues;
+ }
+ rc = ibmveth_alloc_single_rx_queue(adapter, i,
+ rxq_entries);
+ if (rc) {
+ netdev_err(netdev, "Failed to allocate queue %d: %d\n",
+ i, rc);
+ goto cleanup_new_queues;
+ }
+
+ rc = ibmveth_register_single_rx_queue(adapter, i);
+ if (rc) {
+ netdev_err(netdev, "Failed to register queue %d: %d\n",
+ i, rc);
+ if (rc == -EOPNOTSUPP)
+ WRITE_ONCE(adapter->mq_fallback, true);
+ if (!ibmveth_deregister_single_rx_queue(adapter,
+ i))
+ ibmveth_free_single_rx_queue(adapter,
+ i);
+ else
+ schedule_work(&adapter->work);
+ goto cleanup_new_queues;
+ }
+
+ rc = ibmveth_setup_single_rx_interrupt(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to setup IRQ for queue %d: %d\n",
+ i, rc);
+ /* request_irq failed: mapped but no handler */
+ ibmveth_dispose_subordinate_irq_mapping(adapter,
+ i);
+ if (!ibmveth_deregister_single_rx_queue(adapter,
+ i))
+ ibmveth_free_single_rx_queue(adapter,
+ i);
+ else
+ schedule_work(&adapter->work);
+ goto cleanup_new_queues;
+ }
+
+ /*
+ * Fully ready before PHYP delivery, matching open():
+ * publish -> replenish -> napi_enable -> enable_irq.
+ * That way ibmveth_interrupt() cannot run on an
+ * unpublished, empty, or NAPI-disabled queue.
+ */
+ ibmveth_publish_num_rx_queues(adapter, i + 1);
+ ibmveth_replenish_task(adapter, i);
+ napi_enable(&adapter->napi[i]);
+
+ rc = ibmveth_enable_irq(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "Failed to enable IRQ for queue %d: %d\n",
+ i, rc);
+ /*
+ * Published, replenished, and NAPI-enabled,
+ * but PHYP never unmasked. Match scale-down /
+ * shared cleanup: drain posted buffers, then
+ * deregister before unmap via
+ * destroy_subordinate.
+ *
+ * napi_disable() must come BEFORE the count
+ * is lowered, matching scale-down and
+ * cleanup_new_queues. Lowering it first does
+ * not hide queue i from netpoll:
+ * netpoll_poll_dev() calls poll_napi(),
+ * which walks dev->napi_list
+ * unbounded by the queue count and skips a NAPI
+ * only once NAPI_STATE_NPSVC is set. Queue i is
+ * enabled here, so ibmveth_poll() would run and
+ * trip its queue_index >= num_rx_queues
+ * WARN_ON. napi_disable() sets NPSVC, so
+ * poll_napi() skips the queue instead.
+ */
+ napi_disable(&adapter->napi[i]);
+ /* Wait for the poll tail, then remask. */
+ synchronize_net();
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ ibmveth_publish_num_rx_queues(adapter, i);
+ ibmveth_drain_rx_queue(adapter, i);
+ synchronize_net();
+ }
+ if (rc) {
+ if (ibmveth_destroy_subordinate_rx_queue(adapter,
+ i)) {
+ int keep = i + 1;
+
+ /*
+ * PHYP still owns queue i. Keep it in
+ * the live set with the queues this
+ * -L already added.
+ * cleanup_new_queues would publish
+ * below i and orphan this mapping.
+ */
+ ibmveth_publish_num_rx_queues(adapter,
+ keep);
+ if (netif_set_real_num_rx_queues(netdev,
+ keep))
+ schedule_work(&adapter->work);
+ ibmveth_replenish_task(adapter, i);
+ napi_enable(&adapter->napi[i]);
+ rc = ibmveth_enable_irq(adapter, i);
+ if (rc) {
+ netdev_err(netdev,
+ "queue %d enable_irq rc=%d\n",
+ i, rc);
+ schedule_work(&adapter->work);
+ } else {
+ ibmveth_restart_rx_queue(adapter,
+ i);
+ }
+ ibmveth_refresh_cmo_desired(adapter);
+ return -EIO;
+ }
+ }
+ if (rc) {
+ /* enable_irq errno; keep -EIO. */
+ rc = -EIO;
+ goto cleanup_new_queues;
+ }
+ ibmveth_restart_rx_queue(adapter, i);
+ }
+
+ return 0;
+
+cleanup_new_queues:
+ failed_queue = i;
+ if (failed_queue > old_count)
+ netdev_err(netdev,
+ "Scale-up failed at queue %d, cleaning up queues %d-%d\n",
+ failed_queue, old_count, failed_queue - 1);
+ else
+ netdev_err(netdev,
+ "Scale-up failed at queue %d, nothing to clean up\n",
+ failed_queue);
+
+ for (i = old_count; i < failed_queue; i++) {
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = old_count; i < failed_queue; i++)
+ napi_disable(&adapter->napi[i]);
+
+ /* As scale-down: wait for the poll tail, then remask. */
+ synchronize_net();
+ for (i = old_count; i < failed_queue; i++) {
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = old_count; i < failed_queue; i++)
+ ibmveth_drain_rx_queue(adapter, i);
+
+ /* Drop the live count before freeing the half-added queues. */
+ ibmveth_publish_num_rx_queues(adapter, old_count);
+ if (netif_set_real_num_rx_queues(netdev, old_count))
+ schedule_work(&adapter->work);
+ synchronize_net();
+
+ for (i = failed_queue - 1; i >= old_count; i--) {
+ int keep;
+
+ if (!ibmveth_destroy_subordinate_rx_queue(adapter, i))
+ continue;
+
+ keep = i + 1;
+ ibmveth_publish_num_rx_queues(adapter, keep);
+ if (netif_set_real_num_rx_queues(netdev, keep)) {
+ schedule_work(&adapter->work);
+ /*
+ * Grow to keep failed; real_num stayed
+ * lowered. Leave PHYP masked. Reset
+ * already queued.
+ */
+ }
+ for (i = old_count; i < keep; i++) {
+ int irq_rc;
+
+ ibmveth_replenish_task(adapter, i);
+ /* START: NAPI before PHYP unmask. */
+ napi_enable(&adapter->napi[i]);
+ /*
+ * Skip unmask while real_num is below
+ * keep (RPS WARN). NAPI is still
+ * enabled so close/reset is not a
+ * second napi_disable hang.
+ */
+ if (netdev->real_num_rx_queues < keep)
+ continue;
+ irq_rc = ibmveth_enable_irq(adapter, i);
+ if (irq_rc) {
+ netdev_err(netdev,
+ "queue %d enable_irq rc=%d\n",
+ i, irq_rc);
+ schedule_work(&adapter->work);
+ continue;
+ }
+ ibmveth_restart_rx_queue(adapter, i);
+ }
+ ibmveth_refresh_cmo_desired(adapter);
+ netdev_warn(netdev,
+ "Keeping %d queues after scale-up failure\n",
+ keep);
+ return rc;
+ }
+
+ /* Roll CMO desired back to the surviving queue count. */
+ ibmveth_refresh_cmo_desired(adapter);
+
+ netdev_warn(netdev, "Keeping %d queues after scale-up failure\n",
+ old_count);
+ return rc;
+}
+
+/**
+ * ibmveth_scale_down_rx_queues - Remove RX queues from a live adapter
+ * @adapter: ibmveth adapter structure
+ * @old_count: Current number of RX queues
+ * @new_count: Target number of RX queues, below @old_count
+ *
+ * Quiesces the retiring queues, drains them, publishes the surviving
+ * count, and only then deregisters them. Successful H_FREE folds the
+ * last no_buffer sample into retired. Walks high to low so a failed
+ * free leaves 0..i live and republishes that count.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_scale_down_rx_queues(struct ibmveth_adapter *adapter, int old_count,
+ int new_count)
+{
+ struct net_device *netdev = adapter->netdev;
+ int rc, i;
+
+ netdev_dbg(netdev, "Scale-down: removing queues %d-%d\n",
+ new_count, old_count - 1);
+
+ /*
+ * Mask PHYP before napi_disable so the handler cannot miss
+ * a mask while NAPI is already dead. napi_disable() does not
+ * wait for the poll tail, which can still re-arm PHYP, so
+ * wait for it, then remask and sync again. Then drain, publish
+ * the surviving count, and synchronize_net() before destroy
+ * so NAPI/softirq cannot walk dying queues (handler may still
+ * be registered until destroy). napi_disable() also keeps
+ * netpoll's poll_napi() off the retiring queues.
+ * Destroy retires no_buffer after successful H_FREE, while
+ * the page is still mapped.
+ */
+ for (i = new_count; i < old_count; i++) {
+ if (!adapter->queue_irq[i])
+ continue;
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = new_count; i < old_count; i++)
+ napi_disable(&adapter->napi[i]);
+ synchronize_net();
+
+ for (i = new_count; i < old_count; i++) {
+ if (!adapter->queue_irq[i])
+ continue;
+ ibmveth_disable_irq(adapter, i);
+ synchronize_irq(adapter->queue_irq[i]);
+ }
+
+ for (i = new_count; i < old_count; i++)
+ ibmveth_drain_rx_queue(adapter, i);
+
+ ibmveth_publish_num_rx_queues(adapter, new_count);
+ synchronize_net();
+
+ rc = netif_set_real_num_rx_queues(netdev, new_count);
+ if (rc) {
+ netdev_err(netdev, "Failed to set real RX queues to %d: %d\n",
+ new_count, rc);
+ ibmveth_publish_num_rx_queues(adapter, old_count);
+ for (i = new_count; i < old_count; i++) {
+ int irq_rc;
+
+ ibmveth_replenish_task(adapter, i);
+ /* START: NAPI before PHYP unmask. */
+ napi_enable(&adapter->napi[i]);
+ irq_rc = ibmveth_enable_irq(adapter, i);
+ if (irq_rc) {
+ netdev_err(netdev,
+ "Failed to re-enable IRQ for queue %d during scale-down rollback (rc=%d), scheduling reset\n",
+ i, irq_rc);
+ schedule_work(&adapter->work);
+ continue;
+ }
+ ibmveth_restart_rx_queue(adapter, i);
+ }
+ return rc;
+ }
+
+ for (i = old_count - 1; i >= new_count; i--) {
+ int keep;
+
+ rc = ibmveth_destroy_subordinate_rx_queue(adapter, i);
+ if (!rc)
+ continue;
+
+ /*
+ * High-to-low so a failed free leaves 0..i live.
+ * Queues i+1..old_count-1 are already gone.
+ */
+ keep = i + 1;
+ ibmveth_publish_num_rx_queues(adapter, keep);
+ if (netif_set_real_num_rx_queues(netdev, keep)) {
+ schedule_work(&adapter->work);
+ /*
+ * Grow to keep failed; real_num stayed
+ * lowered. Leave PHYP masked. Reset
+ * already queued.
+ */
+ }
+ for (i = new_count; i < keep; i++) {
+ int irq_rc;
+
+ ibmveth_replenish_task(adapter, i);
+ /* START: NAPI before PHYP unmask. */
+ napi_enable(&adapter->napi[i]);
+ /*
+ * Skip unmask while real_num is below
+ * keep (RPS WARN). NAPI is still
+ * enabled so close/reset is not a
+ * second napi_disable hang.
+ */
+ if (netdev->real_num_rx_queues < keep)
+ continue;
+ irq_rc = ibmveth_enable_irq(adapter, i);
+ if (irq_rc) {
+ netdev_err(netdev,
+ "queue %d enable_irq rc=%d\n",
+ i, irq_rc);
+ schedule_work(&adapter->work);
+ continue;
+ }
+ ibmveth_restart_rx_queue(adapter, i);
+ }
+ return rc;
+ }
+
+ return 0;
+}
+
+/**
+ * ibmveth_resize_rx_queues_incremental - Resize RX queue count incrementally
+ * @adapter: ibmveth adapter structure
+ * @new_count: Target number of RX queues
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Adds or removes RX queues without tearing down the entire adapter.
+ * Active queues continue receiving during scale-up. Scale-up widens
+ * real_num_rx_queues before unmasking a new queue. Scale-down drains
+ * excess queues before deregistering them with the hypervisor.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_resize_rx_queues_incremental(struct ibmveth_adapter *adapter,
+ int new_count, int rxq_entries)
+{
+ struct net_device *netdev = adapter->netdev;
+ int old_count = ibmveth_get_num_rx_queues(adapter);
+ int rc;
+
+ if (old_count == new_count) {
+ netdev_dbg(netdev, "RX queue count unchanged (%d), nothing to do\n",
+ old_count);
+ return 0;
+ }
+
+ if (new_count < 1 || new_count > IBMVETH_MAX_RX_QUEUES) {
+ netdev_err(netdev, "Invalid RX queue count %d (must be 1-%d)\n",
+ new_count, IBMVETH_MAX_RX_QUEUES);
+ return -EINVAL;
+ }
+
+ netdev_info(netdev, "Incrementally resizing RX queues: %d to %d\n",
+ old_count, new_count);
+
+ if (new_count > old_count)
+ rc = ibmveth_scale_up_rx_queues(adapter, old_count, new_count,
+ rxq_entries);
+ else
+ rc = ibmveth_scale_down_rx_queues(adapter, old_count,
+ new_count);
+ if (rc)
+ return rc;
+
+ netdev_info(netdev, "Successfully resized to %u RX queues (incremental)\n",
+ ibmveth_get_num_rx_queues(adapter));
+
+ ibmveth_refresh_cmo_desired(adapter);
+
+ return 0;
+}
+
/**
* ibmveth_free_all_queues - Free all RX queues at once
* @adapter: ibmveth adapter structure
@@ -2056,7 +2981,9 @@ static void ibmveth_release_lan_resources(struct ibmveth_adapter *adapter)
* ibmveth_cleanup_rx_interrupts() on close, or by
* ibmveth_dispose_subordinate_irq_mappings() on partial register failure.
*
- * Return: 0 if the caller may unmap, -EIO if it must not
+ * Return: 0 if the caller may unmap (the LAN is off, or single-queue
+ * mode after any failure), -EIO if H_FREE_LOGICAL_LAN failed
+ * before start in multi-queue mode
*/
static int ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
{
@@ -2239,6 +3166,7 @@ static int ibmveth_open(struct net_device *netdev)
"h_free_logical_lan still failing; LAN remains registered\n");
return -EIO;
}
+ ibmveth_free_stranded_rx_queues(adapter);
ibmveth_release_lan_resources(adapter);
}
@@ -2367,6 +3295,7 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_cleanup_rx_interrupts(adapter);
if (!ibmveth_free_all_queues(adapter)) {
+ ibmveth_free_stranded_rx_queues(adapter);
/* netif_tx_disable() above keeps xmit off the TX LTBs. */
ibmveth_release_lan_resources(adapter);
}
@@ -2769,21 +3698,96 @@ static void ibmveth_get_channels(struct net_device *netdev,
channels->max_rx = channels->rx_count;
}
+/**
+ * ibmveth_resize_rx_channels - Validate and apply a new RX queue count
+ * @adapter: ibmveth adapter
+ * @goal_rx: requested RX queue count
+ *
+ * Rejects rx > 1 without MQ firmware (-EOPNOTSUPP) and rx outside
+ * 1..IBMVETH_MAX_RX_QUEUES (-EINVAL). An RX count change while the
+ * device is down is rejected (-EOPNOTSUPP); publishing it without
+ * allocating arrives in the next patch. When up, apply via
+ * ibmveth_resize_rx_queues_incremental().
+ *
+ * Return: 0 or negative errno
+ */
+static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,
+ unsigned int goal_rx)
+{
+ struct net_device *netdev = adapter->netdev;
+ unsigned int old_rx = ibmveth_get_num_rx_queues(adapter);
+ int rxq_entries;
+ int rc;
+
+ if (goal_rx < 1 || goal_rx > IBMVETH_MAX_RX_QUEUES) {
+ netdev_err(netdev,
+ "Invalid RX queue count %u (must be 1-%d)\n",
+ goal_rx, IBMVETH_MAX_RX_QUEUES);
+ return -EINVAL;
+ }
+
+ /*
+ * Check for a no-op before the capability gate. ethtool -L is
+ * read-modify-write, so a TX-only request arrives carrying the
+ * current RX count; gating first would fail those with
+ * -EOPNOTSUPP once mq_fallback is set.
+ */
+ if (goal_rx == old_rx)
+ return 0;
+
+ /*
+ * Refuse any rx > 1, not just growth: once mq_fallback is set the
+ * next open comes up single-queue, so an intermediate count could
+ * not be honoured either, and accepting it would only repeat the
+ * silent clamp at open. max_rx stays at the live count so that
+ * read-modify-write TX-only requests still clear the core.
+ */
+ if (goal_rx > 1 &&
+ (!adapter->multi_queue || READ_ONCE(adapter->mq_fallback))) {
+ netdev_err(netdev,
+ "Cannot resize to %u RX queues: multi-queue mode not supported by firmware\n",
+ goal_rx);
+ return -EOPNOTSUPP;
+ }
+
+ /*
+ * Down / failed-open: there is nothing to resize, and publishing
+ * the desired count without allocating arrives in the next
+ * patch. Refuse rather than report success for a request that
+ * would be discarded.
+ */
+ if (!adapter->opened)
+ return -EOPNOTSUPP;
+
+ rxq_entries = adapter->rx_queue[0].num_slots;
+ rc = ibmveth_resize_rx_queues_incremental(adapter, goal_rx,
+ rxq_entries);
+ if (rc)
+ netdev_err(netdev, "Failed to resize RX queues: %d\n", rc);
+ return rc;
+}
+
static int ibmveth_set_channels(struct net_device *netdev,
struct ethtool_channels *channels)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
unsigned int old = netdev->real_num_tx_queues,
goal = channels->tx_count;
+ unsigned int old_rx = ibmveth_get_num_rx_queues(adapter);
+ unsigned int want_tx = goal;
+ bool rx_changed = false;
int rc, i, alloc_rc = 0;
- /*
- * RX channel resize is implemented later; reject any request
- * that changes rx_count. Read-modify-write TX adjustments
- * submit the current rx_count and proceed.
+ /* Validate RX (and resize when opened) before the down-path
+ * early return so MQ/range errors are reported here. Publishing
+ * the desired RX count and CMO while down is the next patch.
*/
- if (channels->rx_count != ibmveth_get_num_rx_queues(adapter))
- return -EOPNOTSUPP;
+ rc = ibmveth_resize_rx_channels(adapter, channels->rx_count);
+ if (rc)
+ return rc;
+
+ if (channels->rx_count != old_rx)
+ rx_changed = true;
/* If the device is not open (including a failed close/open with
* IFF_UP still set) then don't allocate, just set desired
@@ -2831,6 +3835,28 @@ static int ibmveth_set_channels(struct net_device *netdev,
netif_tx_wake_all_queues(netdev);
+ if (netdev->real_num_tx_queues != want_tx) {
+ if (rx_changed) {
+ /*
+ * Best-effort restore of the RX count from before
+ * this -L; a failure is logged and the TX error
+ * returned. num_slots is the live size after that
+ * resize.
+ */
+ int rxq_entries = adapter->rx_queue[0].num_slots;
+ int rb;
+
+ rb = ibmveth_resize_rx_queues_incremental(adapter,
+ old_rx,
+ rxq_entries);
+ if (rb)
+ netdev_err(netdev,
+ "Failed to roll back RX queues to %u after TX failure: %d\n",
+ old_rx, rb);
+ }
+ return rc ? rc : -ENOMEM;
+ }
+
return rc;
}
@@ -3333,7 +4359,10 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
ibmveth_rx_csum_helper(skb, adapter);
}
- skb_record_rx_queue(skb, queue_index);
+ /* Past real_num on the resize keep-path: deliver untagged. */
+ if ((unsigned int)queue_index <
+ READ_ONCE(netdev->real_num_rx_queues))
+ skb_record_rx_queue(skb, queue_index);
napi_gro_receive(napi, skb);
IBMVETH_STAT_INC(rxs->packets);
@@ -3424,6 +4453,9 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
if (ibmveth_poll_stopping(netdev, napi))
goto out;
+ if (queue_index >= READ_ONCE(netdev->real_num_rx_queues))
+ goto out; /* keep-path: skip enable_irq past real_num */
+
rc = ibmveth_enable_irq(adapter, queue_index);
if (rc) {
netdev_err(netdev,
@@ -3456,11 +4488,11 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
qindex = napi - adapter->napi;
/*
- * Quiet on out-of-range: a live-count shrink publishes the lower
- * count before free_irq(). A residual IRQ must not WARN-storm;
- * return IRQ_NONE until the handler is removed.
+ * IRQ_NONE when out of range: scale-down before free_irq(), or
+ * qindex >= real_num on resize keep-path (do not schedule).
*/
- if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
+ if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter) ||
+ qindex >= READ_ONCE(netdev->real_num_rx_queues))
return IRQ_NONE;
IBMVETH_STAT_INC(adapter->rx_qstats[qindex].interrupts);
@@ -3580,8 +4612,6 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
struct net_device *netdev = dev_get_drvdata(&vdev->dev);
struct ibmveth_adapter *adapter;
struct iommu_table *tbl;
- unsigned long ret;
- int i, q;
tbl = get_iommu_table_base(&vdev->dev);
@@ -3590,41 +4620,8 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
return IOMMU_PAGE_ALIGN(IBMVETH_IO_ENTITLEMENT_DEFAULT, tbl);
adapter = netdev_priv(netdev);
-
- /* One buffer list page per RX queue; filter list is shared. */
- ret = IBMVETH_BUFF_LIST_SIZE * ibmveth_get_num_rx_queues(adapter) +
- IBMVETH_FILT_LIST_SIZE;
- ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
- /* add size of mapped tx buffers */
- ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
-
- /*
- * Pool metadata for queues 1+ is copied from queue 0 at open.
- * Always size from pool 0 x num_rx_queues.
- *
- * CMO (Power9 and earlier) and MQ firmware (Power11+) do not
- * coexist, so a CMO partition always sizes one queue here. The
- * MQ terms are defensive only.
- */
- for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
- int rxqentries = 1;
-
- for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
- struct ibmveth_buff_pool *bpool =
- &adapter->rx_buff_pool[0][i];
-
- if (bpool->active)
- ret += bpool->size *
- IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
- rxqentries += bpool->size;
- }
-
- /* add the size of the receive queue entries */
- ret += IOMMU_PAGE_ALIGN(rxqentries *
- sizeof(struct ibmveth_rx_q_entry), tbl);
- }
-
- return ret;
+ return ibmveth_desired_dma_for_rxqs(adapter,
+ ibmveth_get_num_rx_queues(adapter));
}
static int ibmveth_set_mac_addr(struct net_device *dev, void *p)
@@ -4154,11 +5151,13 @@ static void ibmveth_remove(struct vio_dev *dev)
unregister_netdev(netdev);
if (ibmveth_lan_resources_live(adapter)) {
- if (ibmveth_free_all_queues(adapter))
+ if (ibmveth_free_all_queues(adapter)) {
dev_err(&dev->dev,
"h_free_logical_lan still failing at remove; leaking LAN buffers\n");
- else
+ } else {
+ ibmveth_free_stranded_rx_queues(adapter);
ibmveth_release_lan_resources(adapter);
+ }
}
ibmveth_free_tx_qstats(adapter);
@@ -4366,8 +5365,13 @@ static int ibmveth_resume(struct device *dev)
struct net_device *netdev = dev_get_drvdata(dev);
struct ibmveth_adapter *adapter = netdev_priv(netdev);
unsigned int num = ibmveth_get_num_rx_queues(adapter);
+ unsigned int live;
int i;
+ live = READ_ONCE(netdev->real_num_rx_queues);
+ if (num > live)
+ num = live; /* same as interrupt() */
+
for (i = 0; i < num; i++)
ibmveth_schedule_rx_queue(adapter, i);
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH net-next v8 15/15] ibmveth: Complete set_channels down-path
2026-11-09 23:55 [PATCH net-next v8 00/15] ibmveth: Add multi-queue RX support Mingming Cao
` (13 preceding siblings ...)
2026-11-09 23:55 ` [PATCH net-next v8 14/15] ibmveth: Implement incremental MQ RX queue resize Mingming Cao
@ 2026-11-09 23:55 ` Mingming Cao
14 siblings, 0 replies; 16+ messages in thread
From: Mingming Cao @ 2026-11-09 23:55 UTC (permalink / raw)
To: netdev
Cc: Mingming Cao, davem, kuba, horms, edumazet, pabeni, andrew+netdev,
nnac123, maddy, mpe, linuxppc-dev, npiggin, chleroy, ritesh.list,
sshegde, haren, ricklind, davemarq, bjking1, shaik.abdulla1
The previous commit, "ibmveth: Implement incremental MQ RX queue
resize", wires live ethtool -L rx. This patch completes the
down-path publish, replacing that commit's temporary -EOPNOTSUPP
for an RX count change while down.
This patch finishes ethtool -L for ibmveth:
- while down: publish desired RX and TX counts (and CMO/sysfs);
no PHYP register, buffer pools, or IRQ setup; -EBUSY if a new
queue slot is still registered; never unmap on this path
- while up: validate tx_count before RX resize; skip TX LTB work
when tx_count is unchanged (the previous commit already resizes
RX then TX)
- get/set_channels: max_tx is at least the live tx_count so CPU
offline cannot block an RX-only -L via max_tx below real_num
When down: if the RX count grows and a queue in [old, goal) is
still registered (queue_registered[]), return
-EBUSY before changing TX, same as live scale-up. Then set TX
queues, then publish the desired RX count in
adapter->num_rx_queues and netdev->real_num_rx_queues (rx-N sysfs
and CMO desired change immediately). Do not allocate RX mappings,
buffers or IRQs while down. If RX set_real fails, restore the old
TX real_num. set_channels never unmaps while down: a queue left
registered by a failed H_FREE is still owned by PHYP, and the next
open() retries the free.
When up: the previous patch already resizes RX then TX and rolls RX
back, best effort, if TX cannot reach the request. This patch skips
the TX LTB path when the TX count is unchanged.
Validate tx_count before touching RX so a request the driver will
reject does not resize RX first. The ethtool core already range-checks
against the max_tx get_channels() reports; this is the driver's own
guard.
get_channels() ("ibmveth: Report MQ-aware RX counts in ethtool
get_channels") already reports the live rx_count
and caps max_rx at that count once mq_fallback. This patch does not
change that getter ABI. It adds the max_tx floor: max_tx is at least
the live tx_count. After CPU offline, ibmveth_real_max_tx_queues()
can drop below real_num_tx_queues; ethtool -L resubmits that count
and the core rejects tx_count > max_tx, which also blocked an
RX-only request. set_channels uses the same ceiling (live count or
the online-CPU cap, whichever is larger) and refuses only a request
above that.
Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---
Changes in v8:
- down-path: -EBUSY if a queue in the new range is still registered,
checked before the TX change
- subject drops "and mq_fallback max_rx cap" (now patch 12); the TX
failure RX rollback moved to patch 14
Dropped from this series (merged fixes own them):
- commit 8d9ecac6b96e: the poll_controller !opened guard.
ndo_poll_controller is gone
Changes in v7:
- refresh the TX-fail RX rollback comment
- get_channels max_tx is at least the live
tx_count; set_channels uses that same
ceiling and refuses only a request above it
- lift patch 14's temporary -EOPNOTSUPP for an RX
count change while down
- noted: get_channels keeps the live rx_count;
mq_fallback caps max_rx at that count; do
not advertise max_rx = 1 and do not clamp
rx_count
Changes in v6:
- get_channels() caps max_rx at the live rx_count when mq_fallback
is set; rx_count stays live
- roll back TX real_num if down-path RX set_real fails
- poll_controller() returns if !opened
- noted: rx > 1 reject once mq_fallback is set is patch 14
Changes in v5:
- set_channels / resize_rx_channels read via get_num_rx_queues()
- Series renumber: mailed v4 13/14 set_channels -> tip P15 (end; 14->15)
- set_channels: gate on adapter->opened (not IFF_UP) for live RX resize
vs while-down stash
- Down-path RX stash also refreshes CMO (publish + set_real_num_rx)
- When up: resize RX then TX; roll RX back if TX cannot reach goal
Changes in v4:
- On !IFF_UP, stash num_rx_queues only after TX set succeeds; do not
allocate live subordinate IRQs/buffers while down.
- Initialize i = old_tx on the TX adjust path.
- Always return rc from set_channels().
- Split from the resize-helper patch (same split as v3) while keeping
a live caller of resize_rx_channels() in the previous patch.
drivers/net/ethernet/ibm/ibmveth.c | 144 +++++++++++++++++++++--------
1 file changed, 105 insertions(+), 39 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 510d9d80e2df..3ced6f5bad2d 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -3687,15 +3687,32 @@ static void ibmveth_get_channels(struct net_device *netdev,
struct ethtool_channels *channels)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
+ unsigned int rx_count = ibmveth_get_num_rx_queues(adapter);
- channels->max_tx = ibmveth_real_max_tx_queues();
channels->tx_count = netdev->real_num_tx_queues;
+ /*
+ * Never advertise max_tx below the live count. After CPU
+ * offline, real_max can drop below real_num_tx_queues;
+ * ethtool -L is read-modify-write and the core rejects
+ * tx_count > max_tx, which would also block an RX-only
+ * request. set_channels uses this same ceiling.
+ */
+ channels->max_tx = max_t(unsigned int, channels->tx_count,
+ ibmveth_real_max_tx_queues());
- channels->rx_count = ibmveth_get_num_rx_queues(adapter);
+ /*
+ * Always report the live RX count. ethtool -L is read-modify-
+ * write, so a TX-only request echoes rx_count back at us; an
+ * understated value would be applied as a silent RX shrink.
+ * mq_fallback instead caps max_rx at the live count, which
+ * blocks growth in the core without misreporting what is
+ * currently configured.
+ */
+ channels->rx_count = rx_count;
if (adapter->multi_queue && !READ_ONCE(adapter->mq_fallback))
channels->max_rx = IBMVETH_MAX_RX_QUEUES;
else
- channels->max_rx = channels->rx_count;
+ channels->max_rx = rx_count;
}
/**
@@ -3704,9 +3721,9 @@ static void ibmveth_get_channels(struct net_device *netdev,
* @goal_rx: requested RX queue count
*
* Rejects rx > 1 without MQ firmware (-EOPNOTSUPP) and rx outside
- * 1..IBMVETH_MAX_RX_QUEUES (-EINVAL). An RX count change while the
- * device is down is rejected (-EOPNOTSUPP); publishing it without
- * allocating arrives in the next patch. When up, apply via
+ * 1..IBMVETH_MAX_RX_QUEUES (-EINVAL). While down, return 0 here;
+ * ibmveth_set_channels() publishes the desired counts without
+ * allocating. When up, apply via
* ibmveth_resize_rx_queues_incremental().
*
* Return: 0 or negative errno
@@ -3750,14 +3767,9 @@ static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,
return -EOPNOTSUPP;
}
- /*
- * Down / failed-open: there is nothing to resize, and publishing
- * the desired count without allocating arrives in the next
- * patch. Refuse rather than report success for a request that
- * would be discarded.
- */
+ /* Down / failed-open: do not allocate. */
if (!adapter->opened)
- return -EOPNOTSUPP;
+ return 0;
rxq_entries = adapter->rx_queue[0].num_slots;
rc = ibmveth_resize_rx_queues_incremental(adapter, goal_rx,
@@ -3771,30 +3783,84 @@ static int ibmveth_set_channels(struct net_device *netdev,
struct ethtool_channels *channels)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- unsigned int old = netdev->real_num_tx_queues,
- goal = channels->tx_count;
unsigned int old_rx = ibmveth_get_num_rx_queues(adapter);
- unsigned int want_tx = goal;
+ unsigned int goal_rx = channels->rx_count;
+ unsigned int old_tx = netdev->real_num_tx_queues;
+ unsigned int goal_tx = channels->tx_count;
+ unsigned int want_tx = goal_tx;
+ unsigned int max_tx;
bool rx_changed = false;
int rc, i, alloc_rc = 0;
- /* Validate RX (and resize when opened) before the down-path
- * early return so MQ/range errors are reported here. Publishing
- * the desired RX count and CMO while down is the next patch.
+ /*
+ * Same ceiling get_channels reports: live count or the
+ * online-CPU cap, whichever is larger.
*/
- rc = ibmveth_resize_rx_channels(adapter, channels->rx_count);
+ max_tx = max_t(unsigned int, old_tx,
+ ibmveth_real_max_tx_queues());
+ if (goal_tx < 1 || goal_tx > max_tx) {
+ netdev_err(netdev,
+ "Invalid TX queue count %u (must be 1-%u)\n",
+ goal_tx, max_tx);
+ return -EINVAL;
+ }
+
+ /* RX range / MQ checks live in ibmveth_resize_rx_channels(). */
+ rc = ibmveth_resize_rx_channels(adapter, goal_rx);
if (rc)
return rc;
- if (channels->rx_count != old_rx)
+ /* If not opened (never opened, close+open failed while IFF_UP
+ * stayed set, or close() kept the LAN memory after a failed
+ * H_FREE_LOGICAL_LAN), publish desired queue counts without
+ * allocating.
+ */
+ if (!adapter->opened) {
+ if (goal_rx > old_rx) {
+ for (i = old_rx; i < goal_rx; i++) {
+ if (adapter->queue_registered[i]) {
+ netdev_err(netdev,
+ "RX queue %d still held by PHYP, free retried on next open\n",
+ i);
+ return -EBUSY;
+ }
+ }
+ }
+ /* Apply TX first so a failure leaves the published RX
+ * count unchanged.
+ */
+ rc = netif_set_real_num_tx_queues(netdev, goal_tx);
+ if (rc)
+ return rc;
+
+ /* Publish desired RX count for next open() and refresh CMO;
+ * do not allocate while down.
+ */
+ if (goal_rx != ibmveth_get_num_rx_queues(adapter)) {
+ ibmveth_publish_num_rx_queues(adapter, goal_rx);
+ rc = netif_set_real_num_rx_queues(netdev, goal_rx);
+ if (rc) {
+ int tx_rc;
+
+ ibmveth_publish_num_rx_queues(adapter, old_rx);
+ tx_rc = netif_set_real_num_tx_queues(netdev,
+ old_tx);
+ if (tx_rc)
+ netdev_err(netdev,
+ "Failed to restore TX queues to %u after RX failure: %d\n",
+ old_tx, tx_rc);
+ return rc;
+ }
+ ibmveth_refresh_cmo_desired(adapter);
+ }
+ return 0;
+ }
+
+ if (goal_rx != old_rx)
rx_changed = true;
- /* If the device is not open (including a failed close/open with
- * IFF_UP still set) then don't allocate, just set desired
- * netdev_queue's and return
- */
- if (!adapter->opened)
- return netif_set_real_num_tx_queues(netdev, goal);
+ if (goal_tx == old_tx)
+ return 0;
/* We have IBMVETH_MAX_QUEUES netdev_queue's allocated
* but we may need to alloc/free the ltb's.
@@ -3802,7 +3868,7 @@ static int ibmveth_set_channels(struct net_device *netdev,
netif_tx_stop_all_queues(netdev);
/* Allocate any queue that we need */
- for (i = old; i < goal; i++) {
+ for (i = old_tx; i < goal_tx; i++) {
if (adapter->tx_ltb_ptr[i])
continue;
@@ -3811,24 +3877,24 @@ static int ibmveth_set_channels(struct net_device *netdev,
continue;
/* if something goes wrong, free everything we just allocated */
- netdev_err(netdev, "Failed to allocate more tx queues, returning to %d queues\n",
- old);
+ netdev_err(netdev, "Failed to allocate more tx queues, returning to %u queues\n",
+ old_tx);
alloc_rc = rc;
- goal = old;
- old = i;
+ goal_tx = old_tx;
+ old_tx = i;
break;
}
- rc = netif_set_real_num_tx_queues(netdev, goal);
+ rc = netif_set_real_num_tx_queues(netdev, goal_tx);
if (rc) {
- netdev_err(netdev, "Failed to set real tx queues, returning to %d queues\n",
- old);
- goal = old;
- old = i;
+ netdev_err(netdev, "Failed to set real tx queues, returning to %u queues\n",
+ old_tx);
+ goal_tx = old_tx;
+ old_tx = i;
} else if (alloc_rc) {
rc = alloc_rc;
}
/* Free any that are no longer needed */
- for (i = old; i > goal; i--) {
+ for (i = old_tx; i > goal_tx; i--) {
if (adapter->tx_ltb_ptr[i - 1])
ibmveth_free_tx_ltb(adapter, i - 1);
}
@@ -3857,7 +3923,7 @@ static int ibmveth_set_channels(struct net_device *netdev,
return rc ? rc : -ENOMEM;
}
- return rc;
+ return 0;
}
static const struct ethtool_ops netdev_ethtool_ops = {
--
2.50.1 (Apple Git-155)
^ permalink raw reply related [flat|nested] 16+ messages in thread