[PATCH net-next v6 00/15] ibmveth: Add multi-queue RX support

[PATCH net-next v6 00/15] ibmveth: Add multi-queue RX support

[Mingming Cao]

Hi,

Power11 PHYP adds Virtual Ethernet multi-queue (MQ) RX: multiple
logical-LAN RX queues, per-queue buffer posting, and completion
delivery. Guest Linux did not use that; ibmveth still registered one
RX queue even when PHYP was MQ-capable.

This series adds the ibmveth MQ client for net-next. When PHYP
advertises IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT via H_ILLAN_ATTRIBUTES,
probe enables MQ with a default RX count of min(num_online_cpus(), 8)
(same cap as TX today); ethtool -L can raise RX up to 16. Packets are
received on per-queue NAPI. Older firmware without the bit is unchanged.
Queue selection remains firmware-defined (PHYP hash). Ethtool RSS hash
get/set for that algorithm is deferred to a follow-up series so this
one stays MQ datapath only.

User-visible bits: ethtool -l/-L (channels); standard per-queue
packets/bytes/drops via netdev_stat_ops (ethtool -S keeps only
driver-specific counters; ndo_get_stats64 is the aggregate, including
retired-queue history); and a read-only debugfs buffer_pools dump
(v3's multi-line sysfs dump moved to debugfs; the historical queue-0
poolN/ sysfs ABI is unchanged).

Background:

ibmveth today uses one logical LAN, one set of buffer pools, and one
NAPI context. PHYP MQ mode gives each RX queue its own handle (post via
H_ADD_LOGICAL_LAN_BUFFERS_QUEUE, subordinate register via
H_REG_LOGICAL_LAN_QUEUE); traffic can land on any active queue. The
driver needs per-queue pools, IRQs, and NAPI to match. Legacy firmware
keeps the original hcall path.

Series layout (15 patches):

  1-2   Hypercall wrappers; MQ adapter layout (MAX_RX_QUEUES stays 1)
  3-9   Queue-aware helpers (still SQ runtime): RX, per-queue pools,
        IRQ, TX, PHYP, buffer submit (open/close 3-8); poll harden (9)
  10    Enable MQ datapath at probe/open (subordinate register helpers
        land here with first use)
  11-13 Per-queue RX/TX stats; get_channels MQ counts; debugfs buffer_pools
  14    Incremental RX resize; live ethtool -L rx
  15    Down-path rollback and mq_fallback max_rx cap

- Helper patches (3-8) reshape ibmveth_open()/close() into
  queue-aware helpers. Patch 9 hardens the SQ poll path with the same
  queue-index helpers; it does not change open/close. MQ stays off
  through 3-9: num_rx_queues stays 1 and multi_queue is false until
  patch 10. The live single-queue path still changes where the review
  required it (open/close unwind, IRQ remask, replenish lock, poll
  harden).
- Patch 10 is the switch: probe sets multi_queue from firmware, raises
  num_rx_queues, registers subordinates, and replenishes every active
  queue.
- Patch 11 moves counters per-queue and exports packets/bytes/drops
  through netdev_stat_ops. The thirteen existing -S keys stay; no
  hcall_* or pool%d_ keys.

Testing:

ppc64le PowerVM LPAR, MQ-capable firmware:
* ethtool -L cycling (16/1/8/11/1/3/16/8/1) with ping - no hangs
* ethtool -L under iperf3; link down/up during traffic
* ifdown/ifup under iperf3 RX+TX (MQ and ethtool -L rx 1)
* Legacy firmware (no MQ bit): open/close/stress on helper path
* W=1 clean at every commit (15/15 PASS, 0 warnings per-patch and
  in aggregate, ARCH=powerpc ibmveth.o)

Changes in v6:

Same 15 patches as v5. Jakub v5 review folded in; per-patch detail is
below --- on each commit.

* Both new registration wrappers use plpar_hcall(), not plpar_hcall9().
* Poll: IPv4 check through skb->data; budget 0 does not complete NAPI.
* Scale-down: publish the surviving count, then synchronize_net(),
  then destroy. num_rx_queues uses smp_store_release / smp_load_acquire.
* packets/bytes/drops through netdev_stat_ops, not private -S strings.
  Thirteen existing -S keys kept. No hcall_* or pool%d_ keys.
  replenish_* are per-queue u64; no atomics. get_base_stats() is the
  retired-queue remainder.
* Reset worker gated on NETREG_REGISTERED (cannot reopen after
  unregister).
* get_channels() keeps the live rx_count; mq_fallback caps max_rx so
  a TX-only ethtool -L is not a silent RX shrink.
* Open-fail double-free (d43732ce021f) rides in patches 3 and 6;
  standalone fix to net follows this series.

Known limitations (not this series):

* h_free_logical_lan[_queue] still log-and-continue on non-busy
  failure; fixing requires status propagation through all teardown
  callers. Pre-existing; incremental shrink copies the same path.
* Internal close+open restarts (pool_store, change_mtu) do not call
  netpoll_poll_disable(); pre-existing single-queue behaviour,
  unchanged by this series.
* CMO desired is not recomputed when a down-path set_channels publish
  is never realised (mq_fallback or failed reopen); fixing requires
  recomputing on every path that changes the realised queue count.
* Pool kobject .release is NULL; put then free_netdev() is unsafe
  under CONFIG_DEBUG_KOBJECT_RELEASE. Requires a proper release
  callback; pre-existing pattern.
* ethtool -L TX shrink uses netif_tx_stop_all_queues(), not
  netif_tx_disable(); close() already uses disable. Pre-existing;
  needs its own patch with a Fixes: tag.
* get_desired_dma() TX term is one LTB regardless of TX queue count;
  should scale with real_num_tx_queues. Pre-existing.
* max_tx from num_online_cpus() can fall below a configured tx_count
  after CPU hotplug; pre-existing.

Changes in v5:

* Restack mailed v4 (14 patches) to v5 (15):

    v4 1-8  helpers                -> v5 1-8
    (new)   SQ poll harden         -> v5 9   (before MQ enable)
    v4 9    MQ enable              -> v5 10
    v4 10   stats                  -> v5 11
    (new)   get_channels           -> v5 12  (peeled from stats)
    v4 11   debugfs                -> v5 13
    v4 12   resize                 -> v5 14
    v4 13   set_channels           -> v5 15
    v4 14   trailing poll/shutdown -> folded into v5 5/9/10/14
            (mailed "P14" was that trailer, not v5 14)
* Teardown-first resize after aggressive ethtool -L; thin defensive
  poll skip remains; no correlator generation field this series
* opened / rx_irq_setup; set_channels keys on opened (not IFF_UP)
* filter_list_dma=0 on map error; restore default-active 64 KiB pool;
  unwind pools by allocation presence; probe_cleanup clears vio
  drvdata; remove: unregister then cancel_work
* TX quiesce before freeing bounce buffers; guard start_xmit if LTB gone
* MQ H_FUNCTION recovery (reset + SQ fallback); no printk under
  replenish_lock; lock harvest with replenish; resume kicks all queues
* Per-queue update_rx_no_buffer; publish-before-free on resize;
  CMO refresh; IRQ helpers return errno
* Harvest abort (no fake GRO / UAF); poll refuses PHYP re-arm on close;
  wrap-safe skb_put; atomic set_channels; monotonic stats across shrink
* Keep mask -> sync -> napi_disable on teardown; open stays
  request_irq -> napi_enable while PHYP masked; scale-up/recovery keep
  napi_enable before enable_irq
* Pool geometry kept on free; restart_rx_queue after open/scale-down;
  remask after napi_disable; schedule_rx_queue masks only when
  napi_schedule_prep succeeds (STOP + poll no-rearm for storms)

Changes in v4:

Addresses Simon's v3 review and related fixes:
* First-use helpers/includes (irqdomain.h with first dispose); no
  unused statics; dropped orphan open/close pipeline patch
* Open/close unwind (free LAN before RX pools); no double TX teardown
* MQ open: replenish all queues before PHYP unmask; H_FUNCTION on
  subordinate register is a hard open failure
* Resize/set_channels hardenings; stats probe-lifetime + sum-on-read;
  buffer_pools diagnostic on debugfs
* Patch 9: put already-created pool kobjects on probe failure paths
* Patch 14: correlator skip, skb tailroom check, napi_complete_done
  shutdown return < budget
* Bisect-friendly restack (helpers with first use)

Changes in v3:

* Dropped RFC; addressed style / DMA feedback from earlier revisions
* Early MQ enablement iterations (see lore links below)

Comments welcome.

---
v5 lore:
  https://lore.kernel.org/r/20260814073642.24630-1-mmc@linux.ibm.com
v5 review (Jakub Kicinski):
  https://lore.kernel.org/r/20260818014710.3853684-1-kuba@kernel.org
Sashiko Gemini (sashiko.dev):
  https://sashiko.dev/#/patchset/20260814073642.24630-1-mmc@linux.ibm.com
Sashiko NIPA (netdev-ai):
  https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260814073642.24630-1-mmc@linux.ibm.com

Previous versions
v5: https://lore.kernel.org/r/20260814073642.24630-1-mmc@linux.ibm.com
v4: https://lore.kernel.org/r/cover.1785457143.git.mmc@linux.ibm.com
v3: https://lore.kernel.org/r/20260706193603.8039-1-mmc@linux.ibm.com
v2: https://lore.kernel.org/r/20260701222327.61325-1-mmc@linux.ibm.com
v1: https://lore.kernel.org/r/cover.1782758799.git.mmc@linux.ibm.com
v4 review (Jakub Kicinski):
  https://lore.kernel.org/r/20260806183614.3171785-1-kuba@kernel.org
v3 review (Simon Horman):
  https://lore.kernel.org/r/20260714124327.GJ1364329@horms.kernel.org

Mingming Cao (15):
  ibmveth: Add MQ RX hypercall wrappers and call definitions
  ibmveth: Prepare MQ RX adapter data structures
  ibmveth: Refactor RX resource allocation for MQ RX bring-up
  ibmveth: Refactor buffer pool management for per-queue MQ RX
  ibmveth: Refactor RX interrupt control for MQ RX queues
  ibmveth: Refactor TX resource allocation in open/close paths
  ibmveth: Add RX queue register helpers for MQ
  ibmveth: Add queue-aware RX buffer submit helper for MQ
  ibmveth: Harden RX poll path with helpers
  ibmveth: Enable multi-queue RX receive path
  ibmveth: Add per-queue RX and TX statistics collection
  ibmveth: Report MQ-aware RX counts in ethtool get_channels
  ibmveth: Expose per-queue buffer pool details via debugfs
  ibmveth: Implement incremental MQ RX queue resize
  ibmveth: Complete set_channels down-path and mq_fallback max_rx cap

 arch/powerpc/include/asm/hvcall.h  |    6 +-
 drivers/net/ethernet/ibm/ibmveth.c | 4159 +++++++++++++++++++++++-----
 drivers/net/ethernet/ibm/ibmveth.h |  227 +-
 3 files changed, 3630 insertions(+), 762 deletions(-)

base-commit: 1b78070aaef63512688aebfbc82365ef9d6660f1
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 01/15] ibmveth: Add MQ RX hypercall wrappers and call definitions

[Mingming Cao]

Single-queue ibmveth only needs h_register_logical_lan() plus legacy
buffer add/free calls. MQ RX uses per-queue handles, so the driver must
also be able to register/deregister subordinate queues and post
buffers against a specific queue handle.

Add the PHYP call IDs for:

  H_REG_LOGICAL_LAN_QUEUE (0x49C)
  H_ADD_LOGICAL_LAN_BUFFERS_QUEUE (0x4A0)
  H_FREE_LOGICAL_LAN_QUEUE (0x4A8)

as defined in PAPR 11.20.00. The gaps at 0x498 (reserved) and 0x4A4
(reserved for H_FREE_LOGICAL_LAN_BUFFER_QUEUE) are intentional; this
series tears queues down with H_FREE_LOGICAL_LAN_QUEUE and does not add
the buffer-queue free hcall.

Raising MAX_HCALL_OPCODE for these IDs also widens KVM's
kvm_arch.enabled_hcalls bitmap and the range KVM_CAP_PPC_ENABLE_HCALL
accepts. Neither is user-visible: KVM implements none of the three, so
the ioctl still rejects them, and the bitmap rounds to the same five
unsigned longs, so struct kvm_arch does not change size.

Add ibmveth.h wrapper helpers (h_register_logical_lan_queue(),
h_add_logical_lan_buffers_queue(), h_free_logical_lan_queue()) with
argument ordering and return semantics matching the existing ibmveth
hcall wrappers. h_free_logical_lan_queue() uses plpar_hcall_norets()
like h_free_logical_lan(). Also add h_register_logical_lan_with_handle()
so queue 0 can capture the PHYP queue handle in MQ mode. Both new
registration wrappers use plpar_hcall() rather than plpar_hcall9(), so
they do not read unwritten stack slots. Both new
registration wrappers use plpar_hcall() rather than plpar_hcall9(), so
they do not read unwritten stack slots.

This patch is intentionally plumbing only: no runtime behavior change
yet. Legacy firmware keeps H_REGISTER_LOGICAL_LAN and the existing
buffer hcalls. The new wrappers are used only when a later commit sets
multi_queue from H_ILLAN_ATTRIBUTES.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- both new registration wrappers use plpar_hcall() instead of
  plpar_hcall9(). They do not need nine args, and plpar_hcall9() was
  reading three unwritten stack slots
- Widen add-buffers-queue and free-queue @queue_handle kdoc: handle may
  come from h_register_logical_lan_queue() or
  h_register_logical_lan_with_handle() (queue 0)
- add-buffers-queue Return: also H_FUNCTION (MQ firmware missing that
  hcall)

Changes in v5:
- Cite PAPR 11.20.00 for MQ hcall IDs 0x49C / 0x4A0 / 0x4A8 (v4 named
  the opcodes only)
- Call out reserved gaps at 0x498 and 0x4A4 (0x4A4 =
  H_FREE_LOGICAL_LAN_BUFFER_QUEUE); decision: do not add that hcall -
  queue teardown uses H_FREE_LOGICAL_LAN_QUEUE
- Note MAX_HCALL_OPCODE raise also widens KVM enabled_hcalls /
  KVM_CAP_PPC_ENABLE_HCALL accepted range (cross-subsystem)
- Rename h_reg_logical_lan_queue() -> h_register_logical_lan_queue() to
  match h_register_logical_lan() / h_free_logical_lan()
- Unify queue_handle out-params as unsigned long * on both
  h_register_logical_lan_queue() and h_register_logical_lan_with_handle()
  (v4 used u64 * on with_handle)
- h_free_logical_lan_queue() uses plpar_hcall_norets() like
  h_free_logical_lan() (v4 used plpar_hcall9 with unused retbuf)

Changes in v4:
- Document @queue_handle and @irq in h_reg_logical_lan_queue() kdoc.
- Wrap h_register_logical_lan_with_handle() prototype for readability /
  checkpatch.
- Drop unused H_FREE_LOGICAL_LAN_BUFFER_QUEUE wrapper/opcode (no caller;
  buffer return is local harvest + queue free).

 arch/powerpc/include/asm/hvcall.h  |   6 +-
 drivers/net/ethernet/ibm/ibmveth.h | 139 +++++++++++++++++++++++++++++
 2 files changed, 144 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index dff90a7d7f70..cb0ea53491e6 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -362,7 +362,11 @@
 #define H_GUEST_DELETE		0x488
 #define H_PKS_WRAP_OBJECT	0x490
 #define H_PKS_UNWRAP_OBJECT	0x494
-#define MAX_HCALL_OPCODE	H_PKS_UNWRAP_OBJECT
+/* 0x498 reserved; 0x4A4 = H_FREE_LOGICAL_LAN_BUFFER_QUEUE (unused here) */
+#define H_REG_LOGICAL_LAN_QUEUE 0x49C
+#define H_ADD_LOGICAL_LAN_BUFFERS_QUEUE 0x4A0
+#define H_FREE_LOGICAL_LAN_QUEUE 0x4A8
+#define MAX_HCALL_OPCODE	H_FREE_LOGICAL_LAN_QUEUE
 
 /* Scope args for H_SCM_UNBIND_ALL */
 #define H_UNBIND_SCOPE_ALL (0x1)
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index d87713668ed3..08504d1cafd5 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -66,6 +66,145 @@ static inline long h_add_logical_lan_buffers(unsigned long unit_address,
 			    desc5, desc6, desc7, desc8);
 }
 
+/**
+ * h_register_logical_lan_queue - Register a subordinate receive queue
+ * @unit_address: Device unit address
+ * @buffer_list: DMA address of 4KB page for tracking registered buffers
+ * @rec_queue: Buffer descriptor of receive queue
+ * @queue_handle: Output queue handle on success (may be NULL)
+ * @irq: Output hypervisor IRQ number on success (may be NULL)
+ *
+ * Registers a subordinate receive queue with the hypervisor.
+ *
+ * Return:
+ *   H_SUCCESS (0) on success
+ *   H_PARAMETER if parameters are invalid
+ *
+ * On success, hypervisor returns:
+ *   R3: H_SUCCESS
+ *   R4: Queue handle
+ *   R5: IRQ number for this queue
+ */
+static inline long
+h_register_logical_lan_queue(unsigned long unit_address,
+			     unsigned long buffer_list,
+			     unsigned long rec_queue,
+			     unsigned long *queue_handle,
+			     unsigned long *irq)
+{
+	unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
+	long rc;
+
+	rc = plpar_hcall(H_REG_LOGICAL_LAN_QUEUE,
+			 retbuf, unit_address,
+			 buffer_list, rec_queue);
+
+	if (rc == H_SUCCESS) {
+		if (queue_handle)
+			*queue_handle = retbuf[0];
+		if (irq)
+			*irq = retbuf[1];
+	}
+
+	return rc;
+}
+
+/**
+ * h_add_logical_lan_buffers_queue - Add buffers to subordinate queue
+ * @unit_address: Device unit address
+ * @queue_handle: Queue handle from h_register_logical_lan_queue() or
+ *		  h_register_logical_lan_with_handle() (queue 0)
+ * @buffersznum: Buffer size (upper 32 bits) | count (lower 32 bits)
+ * @ioba12: Buffer addresses 1 and 2 packed ((addr1 << 32) | addr2)
+ * @ioba34: Buffer addresses 3 and 4 packed
+ * @ioba56: Buffer addresses 5 and 6 packed
+ * @ioba78: Buffer addresses 7 and 8 packed
+ * @ioba910: Buffer addresses 9 and 10 packed
+ * @ioba1112: Buffer addresses 11 and 12 packed
+ *
+ * Return:
+ *   H_SUCCESS - All buffers added successfully
+ *   H_PARAMETER - Invalid parameters
+ *   H_HARDWARE - Hardware error
+ *   H_FUNCTION - Firmware does not support this hcall
+ */
+static inline long h_add_logical_lan_buffers_queue(unsigned long unit_address,
+						   unsigned long queue_handle,
+						   unsigned long buffersznum,
+						   unsigned long ioba12,
+						   unsigned long ioba34,
+						   unsigned long ioba56,
+						   unsigned long ioba78,
+						   unsigned long ioba910,
+						   unsigned long ioba1112)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+
+	return plpar_hcall9(H_ADD_LOGICAL_LAN_BUFFERS_QUEUE,
+			    retbuf, unit_address,
+			    queue_handle, buffersznum,
+			    ioba12, ioba34, ioba56,
+			    ioba78, ioba910, ioba1112);
+}
+
+/**
+ * h_free_logical_lan_queue - Deregister subordinate receive queue
+ * @unit_address: Device unit address
+ * @queue_handle: Queue handle from h_register_logical_lan_queue() or
+ *		  h_register_logical_lan_with_handle() (queue 0)
+ *
+ * Deregisters and frees all structures associated with the subordinate queue.
+ *
+ * Return:
+ *   H_SUCCESS - Queue freed successfully
+ *   H_PARAMETER - Invalid parameters
+ *   H_HARDWARE - Hardware error
+ *   H_STATE - VIOA not in valid state
+ *   H_BUSY / H_LONG_BUSY_* - Resource busy, retry
+ */
+static inline long h_free_logical_lan_queue(unsigned long unit_address,
+					    unsigned long queue_handle)
+{
+	return plpar_hcall_norets(H_FREE_LOGICAL_LAN_QUEUE,
+				  unit_address, queue_handle);
+}
+
+/**
+ * h_register_logical_lan_with_handle - Register primary queue and get handle
+ * @unit_address: Device unit address
+ * @buffer_list: DMA address of buffer list
+ * @rec_queue: Buffer descriptor of receive queue
+ * @filter_list: DMA address of filter list
+ * @mac_address: MAC address
+ * @queue_handle: Output parameter for queue handle (may be NULL)
+ *
+ * Registers the primary receive queue (queue 0) with the hypervisor and
+ * returns the queue handle. This is needed in multi-queue mode to use
+ * h_add_logical_lan_buffers_queue() for all queues including queue 0.
+ *
+ * Return: H_SUCCESS (0) on success, error code otherwise
+ */
+static inline long
+h_register_logical_lan_with_handle(unsigned long unit_address,
+				   unsigned long buffer_list,
+				   unsigned long rec_queue,
+				   unsigned long filter_list,
+				   unsigned long mac_address,
+				   unsigned long *queue_handle)
+{
+	unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
+	long rc;
+
+	rc = plpar_hcall(H_REGISTER_LOGICAL_LAN, retbuf,
+			 unit_address, buffer_list, rec_queue,
+			 filter_list, mac_address);
+
+	if (rc == H_SUCCESS && queue_handle)
+		*queue_handle = retbuf[0];
+
+	return rc;
+}
+
 /* FW allows us to send 6 descriptors but we only use one so mark
  * the other 5 as unused (0)
  */
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 02/15] ibmveth: Prepare MQ RX adapter data structures

[Mingming Cao]

MQ RX needs per-queue state for NAPI, queue handles/IRQs, RX rings,
buffer-list DMA mappings, and buffer pools. The current driver stores
most of this as single instances tied to queue 0.

Convert those fields to queue-indexed layouts sized by
IBMVETH_MAX_RX_QUEUES:

  rx_queue[]
  napi[]
  queue_handle[] / queue_irq[]
  buffer_list_addr[] / buffer_list_dma[]
  rx_buff_pool[queue][pool]

and add multi_queue / num_rx_queues to track MQ capability and how
many RX queues are active. Keep IBMVETH_MAX_RX_QUEUES at 1 for now so
this remains a structural preparation patch; later enablement raises
the limit when multi-queue RX is actually turned on.

This patch keeps behavior unchanged by mechanically switching existing
references to index 0 (for example rx_queue -> rx_queue[0],
rx_buff_pool[pool] -> rx_buff_pool[0][pool], napi -> napi[0]).
open/poll/close still drive a single RX queue only.

First use of the new fields: queue_irq[] and multi_queue in the IRQ
control patch; queue_handle[] in the register-helpers patch, which
captures queue 0's handle from H_REGISTER_LOGICAL_LAN; num_rx_queues
in the RX resource-allocation patch. multi_queue only begins
selecting between code paths once enablement raises
IBMVETH_MAX_RX_QUEUES above 1.

Probe kobject / drvdata cleanup on register failure is pre-existing.
Inline kobject_put lands in the MQ enablement patch;
ibmveth_probe_cleanup() in the statistics patch.

Queue-0 pool sysfs (poolN/) remains the shared geometry template for
all RX queues: later patches clone that metadata per queue. Per-queue
runtime visibility is added via debugfs later, not per-queue sysfs.

Per-queue statistics structs are introduced later with their first use
(statistics collection); the replenish_* counters stay adapter-wide
plain u64 here and move into them at that point.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v5:
- Drop unrelated style nits that v4 mixed into the index-0 conversion
  (ibmveth_rxq_get_buffer() prototype reflow, get_desired_dma
  "Return:" kdoc, blank line after rx_large_packets)
- Document queue-0 poolN/ sysfs as the shared geometry/template for all
  RX queues (buff_size/size/active); queues 1..N get no separate pool
  sysfs in this series
- Decision: keep full ibmveth_buff_pool per queue for this series
  (unused embedded kobjects on rows 1..N); a config-vs-runtime layout
  split is out of scope here

Changes in v4:
- Keep IBMVETH_MAX_RX_QUEUES at 1 until MQ enablement (same idea as v3,
  but v3 also planted unused stats types here).
- Layout-only: queue-indexed adapter fields only. Do not introduce
  the rx/tx qstats here (first-use); they land with the per-queue
  statistics patch.
- Subject: "Prepare MQ RX adapter data structures" (was "...and
  statistics structures" in earlier drafts).

 drivers/net/ethernet/ibm/ibmveth.c | 206 ++++++++++++++++-------------
 drivers/net/ethernet/ibm/ibmveth.h |  17 ++-
 2 files changed, 124 insertions(+), 99 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 73e051d26b9d..7cb828b476c1 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -101,7 +101,9 @@ static struct ibmveth_stat ibmveth_stats[] = {
 /* simple methods of getting data from the current rxq entry */
 static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter)
 {
-	return be32_to_cpu(adapter->rx_queue.queue_addr[adapter->rx_queue.index].flags_off);
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+
+	return be32_to_cpu(rxq->queue_addr[rxq->index].flags_off);
 }
 
 static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
@@ -112,7 +114,7 @@ static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
 
 static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter)
 {
-	return ibmveth_rxq_toggle(adapter) == adapter->rx_queue.toggle;
+	return ibmveth_rxq_toggle(adapter) == adapter->rx_queue[0].toggle;
 }
 
 static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter)
@@ -132,7 +134,9 @@ static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter)
 
 static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter)
 {
-	return be32_to_cpu(adapter->rx_queue.queue_addr[adapter->rx_queue.index].length);
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+
+	return be32_to_cpu(rxq->queue_addr[rxq->index].length);
 }
 
 static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter)
@@ -386,7 +390,7 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
  */
 static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
 {
-	__be64 *p = adapter->buffer_list_addr + 4096 - 8;
+	__be64 *p = adapter->buffer_list_addr[0] + 4096 - 8;
 
 	adapter->rx_no_buffer = be64_to_cpup(p);
 }
@@ -399,7 +403,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)
 	adapter->replenish_task_cycles++;
 
 	for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
-		struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[i];
+		struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[0][i];
 
 		if (pool->active &&
 		    (atomic_read(&pool->available) < pool->threshold))
@@ -463,12 +467,12 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
 	struct sk_buff *skb;
 
 	if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
-	    WARN_ON(index >= adapter->rx_buff_pool[pool].size)) {
+	    WARN_ON(index >= adapter->rx_buff_pool[0][pool].size)) {
 		schedule_work(&adapter->work);
 		return -EINVAL;
 	}
 
-	skb = adapter->rx_buff_pool[pool].skbuff[index];
+	skb = adapter->rx_buff_pool[0][pool].skbuff[index];
 	if (WARN_ON(!skb)) {
 		schedule_work(&adapter->work);
 		return -EFAULT;
@@ -482,24 +486,24 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
 		/* remove the skb pointer to mark free. actual freeing is done
 		 * by upper level networking after gro_receive
 		 */
-		adapter->rx_buff_pool[pool].skbuff[index] = NULL;
+		adapter->rx_buff_pool[0][pool].skbuff[index] = NULL;
 
 		dma_unmap_single(&adapter->vdev->dev,
-				 adapter->rx_buff_pool[pool].dma_addr[index],
-				 adapter->rx_buff_pool[pool].buff_size,
+				 adapter->rx_buff_pool[0][pool].dma_addr[index],
+				 adapter->rx_buff_pool[0][pool].buff_size,
 				 DMA_FROM_DEVICE);
 	}
 
-	free_index = adapter->rx_buff_pool[pool].producer_index;
-	adapter->rx_buff_pool[pool].producer_index++;
-	if (adapter->rx_buff_pool[pool].producer_index >=
-	    adapter->rx_buff_pool[pool].size)
-		adapter->rx_buff_pool[pool].producer_index = 0;
-	adapter->rx_buff_pool[pool].free_map[free_index] = index;
+	free_index = adapter->rx_buff_pool[0][pool].producer_index;
+	adapter->rx_buff_pool[0][pool].producer_index++;
+	if (adapter->rx_buff_pool[0][pool].producer_index >=
+	    adapter->rx_buff_pool[0][pool].size)
+		adapter->rx_buff_pool[0][pool].producer_index = 0;
+	adapter->rx_buff_pool[0][pool].free_map[free_index] = index;
 
 	mb();
 
-	atomic_dec(&(adapter->rx_buff_pool[pool].available));
+	atomic_dec(&adapter->rx_buff_pool[0][pool].available);
 
 	return 0;
 }
@@ -507,17 +511,18 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
 /* get the current buffer on the rx queue */
 static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter)
 {
-	u64 correlator = adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator;
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+	u64 correlator = rxq->queue_addr[rxq->index].correlator;
 	unsigned int pool = correlator >> 32;
 	unsigned int index = correlator & 0xffffffffUL;
 
 	if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
-	    WARN_ON(index >= adapter->rx_buff_pool[pool].size)) {
+	    WARN_ON(index >= adapter->rx_buff_pool[0][pool].size)) {
 		schedule_work(&adapter->work);
 		return NULL;
 	}
 
-	return adapter->rx_buff_pool[pool].skbuff[index];
+	return adapter->rx_buff_pool[0][pool].skbuff[index];
 }
 
 /**
@@ -538,14 +543,16 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
 	u64 cor;
 	int rc;
 
-	cor = adapter->rx_queue.queue_addr[adapter->rx_queue.index].correlator;
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+
+	cor = rxq->queue_addr[rxq->index].correlator;
 	rc = ibmveth_remove_buffer_from_pool(adapter, cor, reuse);
 	if (unlikely(rc))
 		return rc;
 
-	if (++adapter->rx_queue.index == adapter->rx_queue.num_slots) {
-		adapter->rx_queue.index = 0;
-		adapter->rx_queue.toggle = !adapter->rx_queue.toggle;
+	if (++adapter->rx_queue[0].index == adapter->rx_queue[0].num_slots) {
+		adapter->rx_queue[0].index = 0;
+		adapter->rx_queue[0].toggle = !adapter->rx_queue[0].toggle;
 	}
 
 	return 0;
@@ -595,7 +602,7 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
 	 */
 retry:
 	rc = h_register_logical_lan(adapter->vdev->unit_address,
-				    adapter->buffer_list_dma, rxq_desc.desc,
+				    adapter->buffer_list_dma[0], rxq_desc.desc,
 				    adapter->filter_list_dma, mac_address);
 
 	if (rc != H_SUCCESS && try_again) {
@@ -623,14 +630,14 @@ static int ibmveth_open(struct net_device *netdev)
 
 	netdev_dbg(netdev, "open starting\n");
 
-	napi_enable(&adapter->napi);
+	napi_enable(&adapter->napi[0]);
 
 	for(i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		rxq_entries += adapter->rx_buff_pool[i].size;
+		rxq_entries += adapter->rx_buff_pool[0][i].size;
 
 	rc = -ENOMEM;
-	adapter->buffer_list_addr = (void*) get_zeroed_page(GFP_KERNEL);
-	if (!adapter->buffer_list_addr) {
+	adapter->buffer_list_addr[0] = (void *)get_zeroed_page(GFP_KERNEL);
+	if (!adapter->buffer_list_addr[0]) {
 		netdev_err(netdev, "unable to allocate list pages\n");
 		goto out;
 	}
@@ -643,17 +650,18 @@ static int ibmveth_open(struct net_device *netdev)
 
 	dev = &adapter->vdev->dev;
 
-	adapter->rx_queue.queue_len = sizeof(struct ibmveth_rx_q_entry) *
+	adapter->rx_queue[0].queue_len = sizeof(struct ibmveth_rx_q_entry) *
 						rxq_entries;
-	adapter->rx_queue.queue_addr =
-		dma_alloc_coherent(dev, adapter->rx_queue.queue_len,
-				   &adapter->rx_queue.queue_dma, GFP_KERNEL);
-	if (!adapter->rx_queue.queue_addr)
+	adapter->rx_queue[0].queue_addr =
+		dma_alloc_coherent(dev, adapter->rx_queue[0].queue_len,
+				   &adapter->rx_queue[0].queue_dma, GFP_KERNEL);
+	if (!adapter->rx_queue[0].queue_addr)
 		goto out_free_filter_list;
 
-	adapter->buffer_list_dma = dma_map_single(dev,
-			adapter->buffer_list_addr, 4096, DMA_BIDIRECTIONAL);
-	if (dma_mapping_error(dev, adapter->buffer_list_dma)) {
+	adapter->buffer_list_dma[0] =
+		dma_map_single(dev, adapter->buffer_list_addr[0],
+			       4096, DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(dev, adapter->buffer_list_dma[0])) {
 		netdev_err(netdev, "unable to map buffer list pages\n");
 		goto out_free_queue_mem;
 	}
@@ -670,19 +678,21 @@ static int ibmveth_open(struct net_device *netdev)
 			goto out_free_tx_ltb;
 	}
 
-	adapter->rx_queue.index = 0;
-	adapter->rx_queue.num_slots = rxq_entries;
-	adapter->rx_queue.toggle = 1;
+	adapter->rx_queue[0].index = 0;
+	adapter->rx_queue[0].num_slots = rxq_entries;
+	adapter->rx_queue[0].toggle = 1;
 
 	mac_address = ether_addr_to_u64(netdev->dev_addr);
 
 	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
-					adapter->rx_queue.queue_len;
-	rxq_desc.fields.address = adapter->rx_queue.queue_dma;
+					adapter->rx_queue[0].queue_len;
+	rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
 
-	netdev_dbg(netdev, "buffer list @ 0x%p\n", adapter->buffer_list_addr);
+	netdev_dbg(netdev, "buffer list @ 0x%p\n",
+		   adapter->buffer_list_addr[0]);
 	netdev_dbg(netdev, "filter list @ 0x%p\n", adapter->filter_list_addr);
-	netdev_dbg(netdev, "receive q   @ 0x%p\n", adapter->rx_queue.queue_addr);
+	netdev_dbg(netdev, "receive q   @ 0x%p\n",
+		   adapter->rx_queue[0].queue_addr);
 
 	h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
 
@@ -693,7 +703,7 @@ static int ibmveth_open(struct net_device *netdev)
 			   lpar_rc);
 		netdev_err(netdev, "buffer TCE:0x%llx filter TCE:0x%llx rxq "
 			   "desc:0x%llx MAC:0x%llx\n",
-				     adapter->buffer_list_dma,
+				     adapter->buffer_list_dma[0],
 				     adapter->filter_list_dma,
 				     rxq_desc.desc,
 				     mac_address);
@@ -702,11 +712,11 @@ static int ibmveth_open(struct net_device *netdev)
 	}
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-		if (!adapter->rx_buff_pool[i].active)
+		if (!adapter->rx_buff_pool[0][i].active)
 			continue;
-		if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[i])) {
+		if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[0][i])) {
 			netdev_err(netdev, "unable to alloc pool\n");
-			adapter->rx_buff_pool[i].active = 0;
+			adapter->rx_buff_pool[0][i].active = 0;
 			rc = -ENOMEM;
 			goto out_free_buffer_pools;
 		}
@@ -738,9 +748,9 @@ static int ibmveth_open(struct net_device *netdev)
 
 out_free_buffer_pools:
 	while (--i >= 0) {
-		if (adapter->rx_buff_pool[i].active)
+		if (adapter->rx_buff_pool[0][i].active)
 			ibmveth_free_buffer_pool(adapter,
-						 &adapter->rx_buff_pool[i]);
+						 &adapter->rx_buff_pool[0][i]);
 	}
 out_unmap_filter_list:
 	dma_unmap_single(dev, adapter->filter_list_dma, 4096,
@@ -752,18 +762,18 @@ static int ibmveth_open(struct net_device *netdev)
 	}
 
 out_unmap_buffer_list:
-	dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
+	dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
 			 DMA_BIDIRECTIONAL);
 out_free_queue_mem:
-	dma_free_coherent(dev, adapter->rx_queue.queue_len,
-			  adapter->rx_queue.queue_addr,
-			  adapter->rx_queue.queue_dma);
+	dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
+			  adapter->rx_queue[0].queue_addr,
+			  adapter->rx_queue[0].queue_dma);
 out_free_filter_list:
 	free_page((unsigned long)adapter->filter_list_addr);
 out_free_buffer_list:
-	free_page((unsigned long)adapter->buffer_list_addr);
+	free_page((unsigned long)adapter->buffer_list_addr[0]);
 out:
-	napi_disable(&adapter->napi);
+	napi_disable(&adapter->napi[0]);
 	return rc;
 }
 
@@ -776,7 +786,7 @@ static int ibmveth_close(struct net_device *netdev)
 
 	netdev_dbg(netdev, "close starting\n");
 
-	napi_disable(&adapter->napi);
+	napi_disable(&adapter->napi[0]);
 
 	netif_tx_stop_all_queues(netdev);
 
@@ -795,22 +805,22 @@ static int ibmveth_close(struct net_device *netdev)
 
 	ibmveth_update_rx_no_buffer(adapter);
 
-	dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
+	dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
 			 DMA_BIDIRECTIONAL);
-	free_page((unsigned long)adapter->buffer_list_addr);
+	free_page((unsigned long)adapter->buffer_list_addr[0]);
 
 	dma_unmap_single(dev, adapter->filter_list_dma, 4096,
 			 DMA_BIDIRECTIONAL);
 	free_page((unsigned long)adapter->filter_list_addr);
 
-	dma_free_coherent(dev, adapter->rx_queue.queue_len,
-			  adapter->rx_queue.queue_addr,
-			  adapter->rx_queue.queue_dma);
+	dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
+			  adapter->rx_queue[0].queue_addr,
+			  adapter->rx_queue[0].queue_dma);
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		if (adapter->rx_buff_pool[i].active)
+		if (adapter->rx_buff_pool[0][i].active)
 			ibmveth_free_buffer_pool(adapter,
-						 &adapter->rx_buff_pool[i]);
+						 &adapter->rx_buff_pool[0][i]);
 
 	for (i = 0; i < netdev->real_num_tx_queues; i++)
 		ibmveth_free_tx_ltb(adapter, i);
@@ -1448,7 +1458,7 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
 static int ibmveth_poll(struct napi_struct *napi, int budget)
 {
 	struct ibmveth_adapter *adapter =
-			container_of(napi, struct ibmveth_adapter, napi);
+			container_of(napi, struct ibmveth_adapter, napi[0]);
 	struct net_device *netdev = adapter->netdev;
 	int frames_processed = 0;
 	unsigned long lpar_rc;
@@ -1573,11 +1583,11 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
 	unsigned long lpar_rc;
 
-	if (napi_schedule_prep(&adapter->napi)) {
+	if (napi_schedule_prep(&adapter->napi[0])) {
 		lpar_rc = h_vio_signal(adapter->vdev->unit_address,
 				       VIO_IRQ_DISABLE);
 		WARN_ON(lpar_rc != H_SUCCESS);
-		__napi_schedule(&adapter->napi);
+		__napi_schedule(&adapter->napi[0]);
 	}
 	return IRQ_HANDLED;
 }
@@ -1645,7 +1655,7 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
 	int need_restart = 0;
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		if (new_mtu_oh <= adapter->rx_buff_pool[i].buff_size)
+		if (new_mtu_oh <= adapter->rx_buff_pool[0][i].buff_size)
 			break;
 
 	if (i == IBMVETH_NUM_BUFF_POOLS)
@@ -1660,9 +1670,9 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
 
 	/* Look for an active buffer pool that can hold the new MTU */
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-		adapter->rx_buff_pool[i].active = 1;
+		adapter->rx_buff_pool[0][i].active = 1;
 
-		if (new_mtu_oh <= adapter->rx_buff_pool[i].buff_size) {
+		if (new_mtu_oh <= adapter->rx_buff_pool[0][i].buff_size) {
 			WRITE_ONCE(dev->mtu, new_mtu);
 			vio_cmo_set_dev_desired(viodev,
 						ibmveth_get_desired_dma
@@ -1720,12 +1730,12 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
 		/* add the size of the active receive buffers */
-		if (adapter->rx_buff_pool[i].active)
+		if (adapter->rx_buff_pool[0][i].active)
 			ret +=
-			    adapter->rx_buff_pool[i].size *
-			    IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[i].
+			    adapter->rx_buff_pool[0][i].size *
+			    IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[0][i].
 					     buff_size, tbl);
-		rxqentries += adapter->rx_buff_pool[i].size;
+		rxqentries += adapter->rx_buff_pool[0][i].size;
 	}
 	/* add the size of the receive queue entries */
 	ret += IOMMU_PAGE_ALIGN(
@@ -1844,7 +1854,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 	adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
 	ibmveth_init_link_settings(netdev);
 
-	netif_napi_add_weight(netdev, &adapter->napi, ibmveth_poll, 16);
+	netif_napi_add_weight(netdev, &adapter->napi[0], ibmveth_poll, 16);
 
 	netdev->irq = dev->irq;
 	netdev->netdev_ops = &ibmveth_netdev_ops;
@@ -1876,6 +1886,10 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 		netdev->features |= NETIF_F_FRAGLIST;
 	}
 
+	/* Initialize queue count - always 1 for now */
+	adapter->multi_queue = 0;
+	adapter->num_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;
+
 	if (ret == H_SUCCESS &&
 	    (ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
 		adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_PER_HCALL;
@@ -1898,10 +1912,10 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 		memcpy(pool_count, pool_count_cmo, sizeof(pool_count));
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-		struct kobject *kobj = &adapter->rx_buff_pool[i].kobj;
+		struct kobject *kobj = &adapter->rx_buff_pool[0][i].kobj;
 		int error;
 
-		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
 					 pool_count[i], pool_size[i],
 					 pool_active[i]);
 		error = kobject_init_and_add(kobj, &ktype_veth_pool,
@@ -1949,7 +1963,7 @@ static void ibmveth_remove(struct vio_dev *dev)
 	cancel_work_sync(&adapter->work);
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		kobject_put(&adapter->rx_buff_pool[i].kobj);
+		kobject_put(&adapter->rx_buff_pool[0][i].kobj);
 
 	unregister_netdev(netdev);
 
@@ -2035,11 +2049,12 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
 			/* Make sure there is a buffer pool with buffers that
 			   can hold a packet of the size of the MTU */
 			for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-				if (pool == &adapter->rx_buff_pool[i])
+				if (pool == &adapter->rx_buff_pool[0][i])
 					continue;
-				if (!adapter->rx_buff_pool[i].active)
+				if (!adapter->rx_buff_pool[0][i].active)
 					continue;
-				if (mtu <= adapter->rx_buff_pool[i].buff_size)
+				if (mtu <=
+				    adapter->rx_buff_pool[0][i].buff_size)
 					break;
 			}
 
@@ -2213,11 +2228,11 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
 
 	/* Set sane values for buffer pools */
 	for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
 					 pool_count[i], pool_size[i],
 					 pool_active[i]);
 
-	pool = &adapter->rx_buff_pool[0];
+	pool = &adapter->rx_buff_pool[0][0];
 	pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
 	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
 
@@ -2225,7 +2240,7 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
 	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
 	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
 
-	correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0].size;
+	correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0][0].size;
 	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
 	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
 
@@ -2258,30 +2273,33 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
 
 	INIT_WORK(&adapter->work, ibmveth_reset_kunit);
 
-	adapter->rx_queue.queue_len = 1;
-	adapter->rx_queue.index = 0;
-	adapter->rx_queue.queue_addr = kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),
-						     GFP_KERNEL);
-	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue.queue_addr);
+	adapter->rx_queue[0].queue_len = 1;
+	adapter->rx_queue[0].index = 0;
+	adapter->rx_queue[0].queue_addr =
+		kunit_kzalloc(test, sizeof(struct ibmveth_rx_q_entry),
+			      GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, adapter->rx_queue[0].queue_addr);
 
 	/* Set sane values for buffer pools */
 	for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[i], i,
+		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
 					 pool_count[i], pool_size[i],
 					 pool_active[i]);
 
-	pool = &adapter->rx_buff_pool[0];
+	pool = &adapter->rx_buff_pool[0][0];
 	pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
 	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
 
-	adapter->rx_queue.queue_addr[0].correlator = (u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
+	adapter->rx_queue[0].queue_addr[0].correlator =
+		(u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
 	KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));
 
-	adapter->rx_queue.queue_addr[0].correlator = (u64)0 << 32 | adapter->rx_buff_pool[0].size;
+	adapter->rx_queue[0].queue_addr[0].correlator =
+		(u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
 	KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));
 
 	pool->skbuff[0] = skb;
-	adapter->rx_queue.queue_addr[0].correlator = (u64)0 << 32 | 0;
+	adapter->rx_queue[0].queue_addr[0].correlator = (u64)0 << 32 | 0;
 	KUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter));
 
 	flush_work(&adapter->work);
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 08504d1cafd5..495269631323 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -260,6 +260,8 @@ static inline long h_illan_attributes(unsigned long unit_address,
 #define IBMVETH_MAX_TX_BUF_SIZE (1024 * 64)
 #define IBMVETH_MAX_QUEUES 16U
 #define IBMVETH_DEFAULT_QUEUES 8U
+#define IBMVETH_MAX_RX_QUEUES 1U
+#define IBMVETH_DEFAULT_RX_QUEUES 1U
 #define IBMVETH_MAX_RX_PER_HCALL 8U
 
 static int pool_size[] = { 512, 1024 * 2, 1024 * 16, 1024 * 32, 1024 * 64 };
@@ -296,18 +298,23 @@ struct ibmveth_rx_q {
 struct ibmveth_adapter {
 	struct vio_dev *vdev;
 	struct net_device *netdev;
-	struct napi_struct napi;
+	struct napi_struct napi[IBMVETH_MAX_RX_QUEUES];
 	struct work_struct work;
 	unsigned int mcastFilterSize;
-	void *buffer_list_addr;
+	void *buffer_list_addr[IBMVETH_MAX_RX_QUEUES];
 	void *filter_list_addr;
 	void *tx_ltb_ptr[IBMVETH_MAX_QUEUES];
 	unsigned int tx_ltb_size;
 	dma_addr_t tx_ltb_dma[IBMVETH_MAX_QUEUES];
-	dma_addr_t buffer_list_dma;
+	dma_addr_t buffer_list_dma[IBMVETH_MAX_RX_QUEUES];
 	dma_addr_t filter_list_dma;
-	struct ibmveth_buff_pool rx_buff_pool[IBMVETH_NUM_BUFF_POOLS];
-	struct ibmveth_rx_q rx_queue;
+	struct ibmveth_buff_pool
+		rx_buff_pool[IBMVETH_MAX_RX_QUEUES][IBMVETH_NUM_BUFF_POOLS];
+	struct ibmveth_rx_q rx_queue[IBMVETH_MAX_RX_QUEUES];
+	u64 queue_handle[IBMVETH_MAX_RX_QUEUES];
+	unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
+	int multi_queue;
+	unsigned int num_rx_queues;
 	int rx_csum;
 	int large_send;
 	bool is_active_trunk;
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 03/15] ibmveth: Refactor RX resource allocation for MQ RX bring-up

[Mingming Cao]

ibmveth_open() allocates the filter list and every RX queue inline.
That is already a long sequence and would get uglier once we loop over
num_rx_queues, especially on error unwind.

Pull the RX bits into helpers and wire them into open()/close() in the
same patch:

  ibmveth_alloc_filter_list() / ibmveth_free_filter_list()
    - shared multicast filter list (one per adapter, not per queue)

  ibmveth_alloc_rx_queues() / ibmveth_cleanup_rx_resources()
    - per-queue buffer lists and RX rings, looping [0, num_rx_queues)

alloc_rx_queues() rolls back on failure so open() does not need nested
goto chains for every queue index. open-failure and close release the
same resources through the same helpers.

The pointer and DMA-handle guards also close a baseline double
free_page() and stale dma_unmap after a failed reopen followed by
ndo_stop: a filter-list map error zeros filter_list_dma so a later
free path cannot unmap DMA_MAPPING_ERROR, and unmap is gated on the
CPU page because SPAPR can return DMA address 0. Cleanup NULLs each
slot as it frees.

Runtime behavior stays single-queue (num_rx_queues is still 1). Buffer
pools, IRQ, TX LTB, and PHYP registration remain inline for later
helper patches.

Also set rc = -ENOMEM before the TX LTB allocation loop so a failed
ibmveth_allocate_tx_ltb() still returns a useful errno after the RX
allocation blocks move into helpers.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- on RX-queue dma_alloc or map failure, free the buffer-list page
  before cleanup so the addr-based unmap does not dma_unmap an
  address that was never mapped
- unmap RX filter/buffer lists by CPU pointer, not dma_addr==0
- use %u for the two num_rx_queues dbg prints
- noted: NULL buffer_list_addr[0] window: close() in P05, netpoll
  through replenish_task() until the per-queue guard in P10
- noted: shared-i TX LTB leak on pool-fail predates this; P04/P06
- noted: pool-fail after register without h_free predates this; P06/P07

Changes in v5:
- On filter_list DMA map failure: free_page and zero filter_list_dma so
  a later free path cannot dma_unmap the DMA_MAPPING_ERROR sentinel
  (match buffer_list_dma convention). Closes reopen->ifdown WARN path
  after set_csum/set_tso/change_mtu/pool_store close+open while running
  - v4 left the sentinel
- Call out rc = -ENOMEM before the TX LTB loop after RX helper extract
  (already in v4; still required so allocate_tx_ltb failure returns a
  useful errno)

Changes in v4:
- Introduce RX/filter allocation helpers in the same patch that wires
  their first open/close callers; v3 left unused statics ahead of the
  old open/close pipeline patch.
- Preserve correct -ENOMEM return on TX LTB allocation failure after
  the RX helper extract.
- Drop reliance on v3's separate "open/close pipeline" patch for this
  wiring.

 drivers/net/ethernet/ibm/ibmveth.c | 277 +++++++++++++++++++++--------
 1 file changed, 199 insertions(+), 78 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 7cb828b476c1..d2dad9e6068e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -151,6 +151,193 @@ static unsigned int ibmveth_real_max_tx_queues(void)
 	return min(n_cpu, IBMVETH_MAX_QUEUES);
 }
 
+/**
+ * ibmveth_alloc_filter_list - Allocate and map filter list
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_filter_list(struct ibmveth_adapter *adapter)
+{
+	struct device *dev = &adapter->vdev->dev;
+	struct net_device *netdev = adapter->netdev;
+
+	adapter->filter_list_addr = (void *)get_zeroed_page(GFP_KERNEL);
+	if (!adapter->filter_list_addr) {
+		netdev_err(netdev, "unable to allocate filter pages\n");
+		return -ENOMEM;
+	}
+
+	adapter->filter_list_dma = dma_map_single(dev,
+						  adapter->filter_list_addr,
+						  4096, DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(dev, adapter->filter_list_dma)) {
+		netdev_err(netdev, "unable to map filter list pages\n");
+		free_page((unsigned long)adapter->filter_list_addr);
+		adapter->filter_list_addr = NULL;
+		/* Do not leave DMA_MAPPING_ERROR for free_filter_list(). */
+		adapter->filter_list_dma = 0;
+		return -ENOMEM;
+	}
+
+	netdev_dbg(netdev, "filter list @ 0x%p (DMA: 0x%llx)\n",
+		   adapter->filter_list_addr,
+		   (unsigned long long)adapter->filter_list_dma);
+
+	return 0;
+}
+
+/**
+ * ibmveth_free_filter_list - Free filter list resources
+ * @adapter: ibmveth adapter structure
+ */
+static void
+ibmveth_free_filter_list(struct ibmveth_adapter *adapter)
+{
+	struct device *dev = &adapter->vdev->dev;
+
+	/* Unmap by CPU pointer: SPAPR can return DMA address 0. */
+	if (adapter->filter_list_addr) {
+		dma_unmap_single(dev, adapter->filter_list_dma, 4096,
+				 DMA_BIDIRECTIONAL);
+		adapter->filter_list_dma = 0;
+		free_page((unsigned long)adapter->filter_list_addr);
+		adapter->filter_list_addr = NULL;
+	}
+}
+
+/**
+ * ibmveth_alloc_rx_queues - Allocate per-queue RX resources
+ * @adapter: ibmveth adapter structure
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
+{
+	struct device *dev = &adapter->vdev->dev;
+	struct net_device *netdev = adapter->netdev;
+	int i;
+
+	for (i = 0; i < adapter->num_rx_queues; i++) {
+		adapter->buffer_list_addr[i] =
+			(void *)get_zeroed_page(GFP_KERNEL);
+		if (!adapter->buffer_list_addr[i]) {
+			netdev_err(netdev,
+				   "unable to allocate buffer list for queue %d\n",
+				   i);
+			goto err_cleanup;
+		}
+
+		adapter->rx_queue[i].queue_len =
+			sizeof(struct ibmveth_rx_q_entry) * rxq_entries;
+		adapter->rx_queue[i].queue_addr =
+			dma_alloc_coherent(dev, adapter->rx_queue[i].queue_len,
+					   &adapter->rx_queue[i].queue_dma,
+					   GFP_KERNEL);
+		if (!adapter->rx_queue[i].queue_addr) {
+			netdev_err(netdev,
+				   "unable to allocate RX queue for queue %d\n",
+				   i);
+			free_page((unsigned long)adapter->buffer_list_addr[i]);
+			adapter->buffer_list_addr[i] = NULL;
+			goto err_cleanup;
+		}
+
+		adapter->buffer_list_dma[i] =
+			dma_map_single(dev, adapter->buffer_list_addr[i],
+				       4096, DMA_BIDIRECTIONAL);
+		if (dma_mapping_error(dev, adapter->buffer_list_dma[i])) {
+			netdev_err(netdev,
+				   "unable to map buffer list for queue %d\n",
+				   i);
+			free_page((unsigned long)adapter->buffer_list_addr[i]);
+			adapter->buffer_list_addr[i] = NULL;
+			adapter->buffer_list_dma[i] = 0;
+			goto err_cleanup;
+		}
+
+		adapter->rx_queue[i].index = 0;
+		adapter->rx_queue[i].num_slots = rxq_entries;
+		adapter->rx_queue[i].toggle = 1;
+
+		netdev_dbg(netdev, "queue %d: buffer_list @ 0x%p (DMA: 0x%llx), rx_queue @ 0x%p (DMA: 0x%llx), %llu entries\n",
+			   i, adapter->buffer_list_addr[i],
+			   (unsigned long long)adapter->buffer_list_dma[i],
+			   adapter->rx_queue[i].queue_addr,
+			   (unsigned long long)adapter->rx_queue[i].queue_dma,
+			   (unsigned long long)rxq_entries);
+	}
+
+	netdev_dbg(netdev, "allocated %u RX queue(s) with %d entries each\n",
+		   adapter->num_rx_queues, rxq_entries);
+
+	return 0;
+
+err_cleanup:
+	/*
+	 * Every failure path above releases what it had already allocated
+	 * for queue i, so each index here is either fully constructed or
+	 * fully empty. Do not unmap buffer_list_dma[] without the matching
+	 * buffer_list_addr[] check: the two are only ever set together.
+	 */
+	for (; i >= 0; i--) {
+		if (adapter->buffer_list_addr[i]) {
+			dma_unmap_single(dev, adapter->buffer_list_dma[i],
+					 4096, DMA_BIDIRECTIONAL);
+			adapter->buffer_list_dma[i] = 0;
+		}
+		if (adapter->rx_queue[i].queue_addr) {
+			dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
+					  adapter->rx_queue[i].queue_addr,
+					  adapter->rx_queue[i].queue_dma);
+			adapter->rx_queue[i].queue_addr = NULL;
+		}
+		if (adapter->buffer_list_addr[i]) {
+			free_page((unsigned long)adapter->buffer_list_addr[i]);
+			adapter->buffer_list_addr[i] = NULL;
+		}
+	}
+
+	return -ENOMEM;
+}
+
+/**
+ * ibmveth_cleanup_rx_resources - Free all RX queue resources
+ * @adapter: ibmveth adapter structure
+ */
+static void
+ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
+{
+	struct device *dev = &adapter->vdev->dev;
+	int i;
+
+	netdev_dbg(adapter->netdev, "cleaning up %u RX queue(s)\n",
+		   adapter->num_rx_queues);
+
+	for (i = 0; i < adapter->num_rx_queues; i++) {
+		if (adapter->buffer_list_addr[i]) {
+			dma_unmap_single(dev, adapter->buffer_list_dma[i],
+					 4096, DMA_BIDIRECTIONAL);
+			adapter->buffer_list_dma[i] = 0;
+		}
+
+		if (adapter->rx_queue[i].queue_addr) {
+			dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
+					  adapter->rx_queue[i].queue_addr,
+					  adapter->rx_queue[i].queue_dma);
+			adapter->rx_queue[i].queue_addr = NULL;
+		}
+
+		if (adapter->buffer_list_addr[i]) {
+			free_page((unsigned long)adapter->buffer_list_addr[i]);
+			adapter->buffer_list_addr[i] = NULL;
+		}
+	}
+}
+
 /* setup the initial settings for a buffer pool */
 static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,
 				     u32 pool_index, u32 pool_size,
@@ -626,74 +813,34 @@ static int ibmveth_open(struct net_device *netdev)
 	int rc;
 	union ibmveth_buf_desc rxq_desc;
 	int i;
-	struct device *dev;
 
 	netdev_dbg(netdev, "open starting\n");
 
 	napi_enable(&adapter->napi[0]);
 
-	for(i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
+	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		rxq_entries += adapter->rx_buff_pool[0][i].size;
 
-	rc = -ENOMEM;
-	adapter->buffer_list_addr[0] = (void *)get_zeroed_page(GFP_KERNEL);
-	if (!adapter->buffer_list_addr[0]) {
-		netdev_err(netdev, "unable to allocate list pages\n");
+	rc = ibmveth_alloc_filter_list(adapter);
+	if (rc)
 		goto out;
-	}
 
-	adapter->filter_list_addr = (void*) get_zeroed_page(GFP_KERNEL);
-	if (!adapter->filter_list_addr) {
-		netdev_err(netdev, "unable to allocate filter pages\n");
-		goto out_free_buffer_list;
-	}
-
-	dev = &adapter->vdev->dev;
-
-	adapter->rx_queue[0].queue_len = sizeof(struct ibmveth_rx_q_entry) *
-						rxq_entries;
-	adapter->rx_queue[0].queue_addr =
-		dma_alloc_coherent(dev, adapter->rx_queue[0].queue_len,
-				   &adapter->rx_queue[0].queue_dma, GFP_KERNEL);
-	if (!adapter->rx_queue[0].queue_addr)
+	rc = ibmveth_alloc_rx_queues(adapter, rxq_entries);
+	if (rc)
 		goto out_free_filter_list;
 
-	adapter->buffer_list_dma[0] =
-		dma_map_single(dev, adapter->buffer_list_addr[0],
-			       4096, DMA_BIDIRECTIONAL);
-	if (dma_mapping_error(dev, adapter->buffer_list_dma[0])) {
-		netdev_err(netdev, "unable to map buffer list pages\n");
-		goto out_free_queue_mem;
-	}
-
-	adapter->filter_list_dma = dma_map_single(dev,
-			adapter->filter_list_addr, 4096, DMA_BIDIRECTIONAL);
-	if (dma_mapping_error(dev, adapter->filter_list_dma)) {
-		netdev_err(netdev, "unable to map filter list pages\n");
-		goto out_unmap_buffer_list;
-	}
-
+	rc = -ENOMEM;
 	for (i = 0; i < netdev->real_num_tx_queues; i++) {
 		if (ibmveth_allocate_tx_ltb(adapter, i))
 			goto out_free_tx_ltb;
 	}
 
-	adapter->rx_queue[0].index = 0;
-	adapter->rx_queue[0].num_slots = rxq_entries;
-	adapter->rx_queue[0].toggle = 1;
-
 	mac_address = ether_addr_to_u64(netdev->dev_addr);
 
 	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
 					adapter->rx_queue[0].queue_len;
 	rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
 
-	netdev_dbg(netdev, "buffer list @ 0x%p\n",
-		   adapter->buffer_list_addr[0]);
-	netdev_dbg(netdev, "filter list @ 0x%p\n", adapter->filter_list_addr);
-	netdev_dbg(netdev, "receive q   @ 0x%p\n",
-		   adapter->rx_queue[0].queue_addr);
-
 	h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
 
 	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
@@ -708,7 +855,7 @@ static int ibmveth_open(struct net_device *netdev)
 				     rxq_desc.desc,
 				     mac_address);
 		rc = -ENONET;
-		goto out_unmap_filter_list;
+		goto out_free_tx_ltb;
 	}
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
@@ -735,8 +882,6 @@ static int ibmveth_open(struct net_device *netdev)
 		goto out_free_buffer_pools;
 	}
 
-	rc = -ENOMEM;
-
 	netdev_dbg(netdev, "initial replenish cycle\n");
 	ibmveth_interrupt(netdev->irq, netdev);
 
@@ -752,26 +897,12 @@ static int ibmveth_open(struct net_device *netdev)
 			ibmveth_free_buffer_pool(adapter,
 						 &adapter->rx_buff_pool[0][i]);
 	}
-out_unmap_filter_list:
-	dma_unmap_single(dev, adapter->filter_list_dma, 4096,
-			 DMA_BIDIRECTIONAL);
-
 out_free_tx_ltb:
-	while (--i >= 0) {
+	while (--i >= 0)
 		ibmveth_free_tx_ltb(adapter, i);
-	}
-
-out_unmap_buffer_list:
-	dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
-			 DMA_BIDIRECTIONAL);
-out_free_queue_mem:
-	dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
-			  adapter->rx_queue[0].queue_addr,
-			  adapter->rx_queue[0].queue_dma);
+	ibmveth_cleanup_rx_resources(adapter);
 out_free_filter_list:
-	free_page((unsigned long)adapter->filter_list_addr);
-out_free_buffer_list:
-	free_page((unsigned long)adapter->buffer_list_addr[0]);
+	ibmveth_free_filter_list(adapter);
 out:
 	napi_disable(&adapter->napi[0]);
 	return rc;
@@ -780,7 +911,6 @@ static int ibmveth_open(struct net_device *netdev)
 static int ibmveth_close(struct net_device *netdev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
-	struct device *dev = &adapter->vdev->dev;
 	long lpar_rc;
 	int i;
 
@@ -805,17 +935,8 @@ static int ibmveth_close(struct net_device *netdev)
 
 	ibmveth_update_rx_no_buffer(adapter);
 
-	dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
-			 DMA_BIDIRECTIONAL);
-	free_page((unsigned long)adapter->buffer_list_addr[0]);
-
-	dma_unmap_single(dev, adapter->filter_list_dma, 4096,
-			 DMA_BIDIRECTIONAL);
-	free_page((unsigned long)adapter->filter_list_addr);
-
-	dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
-			  adapter->rx_queue[0].queue_addr,
-			  adapter->rx_queue[0].queue_dma);
+	ibmveth_cleanup_rx_resources(adapter);
+	ibmveth_free_filter_list(adapter);
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		if (adapter->rx_buff_pool[0][i].active)
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 04/15] ibmveth: Refactor buffer pool management for per-queue MQ RX

[Mingming Cao]

Legacy ibmveth uses five adapter-level RX buffer pools (512 B through
64 KiB). pool_active[] enables the standard-MTU pools by default;
larger pools activate when MTU requires them. With single-queue RX
that set is shared on one completion path.

MQ requires the same pool model per queue: buffers post with
H_ADD_LOGICAL_LAN_BUFFERS_QUEUE against a queue handle and completions
return on that queue. Sharing pools across queues would mix ownership
and break queue-local replenish/drain/teardown.

Refactor around queue-local pools:

  rx_buff_pool[queue][pool]
  ibmveth_alloc_queue_buffer_pools()
  ibmveth_free_queue_buffer_pools()
  ibmveth_alloc_buffer_pools() / ibmveth_free_buffer_pools()

Queue 0 remains the template for pool geometry and activation policy
(size, buff_size, threshold, index, active). Queues 1..N copy that
metadata from queue 0, then allocate backing arrays/skbs per queue.
The existing poolN/ sysfs knobs stay adapter-wide on queue 0: writing
a pool size multiplies real memory by num_rx_queues (still 1 here).
Per-queue runtime state is exposed later via debugfs, not separate
per-queue pool sysfs nodes.

Wire the helpers into open()/close() in the same patch. Runtime
remains single-queue (num_rx_queues is still 1).

Pulling the pool loop out has one side effect worth naming: it no
longer consumes open()'s loop index, so a pool failure reaches
out_free_tx_ltb with i still at real_num_tx_queues and the TX LTBs
actually get freed. The shared index that swallowed them was
pre-existing, so there is no standalone Fixes: tag; the TX side gets
its own unwind two patches later.

Error handling is queue-safe: allocation failure unwinds only what
that queue allocated (then prior queues in the caller).
alloc_buffer_pool() already undoes its own partials, so this open-fail
slot is empty. Free paths still release by real allocations
(free_map/dma_addr/skbuff), not only pool->active, which later resize
needs when a pool can hold memory after active was cleared.

close() also reorders pool teardown ahead of cleanup_rx_resources()
and filter-list free. That is safe here because h_free_logical_lan(),
napi_disable(), and free_irq() have already run, so neither PHYP nor
NAPI still reference the pool buffers or RX completion queue.

Keep the legacy 64 KiB pool enabled by default at standard MTU (same
as single-queue policy).

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- retarget the free-by-presence comment to the later resize paths;
  alloc_buffer_pool() already undoes its own partials, so this
  open-fail slot is empty
- use %u for the two num_rx_queues dbg prints
- noted: pool-fail after register without h_free_logical_lan predates
  this patch; the inline pool loop reached the same labels. P06
  issues h_free before the pool DMA; P07 moves pools ahead of
  register_rx_queues

Changes in v5:
- vs mailed v4: keep pool_active[] = {1,1,0,0,1} (v4 set the 64 KiB
  pool inactive). No code delta vs prior patches here; SQ large-receive /
  CMO stay at the historical baseline. change_mtu does not re-enable
  pool4 at MTU 1500; MQ memory pressure belongs at scale-up, not SQ
  default
- Unwind / free pools by real allocations (free_map/dma_addr/skbuff),
  not only pool->active, so open-fail cannot leak partially allocated
  pools (v4 fail path freed by active and skipped the failing pool)
- Clarify kdoc: free paths use allocation presence (not "all active");
  queue 1..N metadata copy from queue 0 (v4 said "queues 1-15" while
  MAX is still 1 here)
- Document poolN sysfs size multiplies by num_rx_queues (queue 0 is the
  shared template; no per-queue pool sysfs in this series)
- Call out close() pool-free before cleanup_rx_resources, after LAN/
  NAPI/IRQ teardown (order already in v4; spell it for later drain)

Changes in v4:
- Introduce the pool helpers in the same patch that wires their first
  open/close callers, instead of leaving unused statics.
- Copy pool->index when cloning queue-0 geometry to later queues
  (needed for correlators; also required by incremental resize).

 drivers/net/ethernet/ibm/ibmveth.c | 163 +++++++++++++++++++++++++----
 1 file changed, 143 insertions(+), 20 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index d2dad9e6068e..37a6d13e603e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -634,6 +634,144 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
 	}
 }
 
+/**
+ * ibmveth_free_queue_buffer_pools - Free buffer pools for a single queue
+ * @adapter: ibmveth adapter structure
+ * @queue: queue index
+ *
+ * Frees buffer pools that still hold allocations for the specified
+ * queue (by free_map / dma_addr / skbuff presence), regardless of the
+ * active flag.
+ */
+static void ibmveth_free_queue_buffer_pools(struct ibmveth_adapter *adapter,
+					    int queue)
+{
+	int i;
+
+	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+		struct ibmveth_buff_pool *pool =
+			&adapter->rx_buff_pool[queue][i];
+
+		/* Free pool if it has allocated memory, regardless of
+		 * active flag. Allocation and active can diverge on failure
+		 * paths, so check for actual allocations.
+		 */
+		if (pool->free_map || pool->dma_addr || pool->skbuff)
+			ibmveth_free_buffer_pool(adapter, pool);
+	}
+}
+
+/**
+ * ibmveth_alloc_queue_buffer_pools - Allocate buffer pools for a single queue
+ * @adapter: ibmveth adapter structure
+ * @queue: queue index
+ *
+ * Allocates backing storage for each active pool on @queue.
+ * Inactive pools (!active) are skipped. Pool metadata must be
+ * initialized before calling this function.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int ibmveth_alloc_queue_buffer_pools(struct ibmveth_adapter *adapter,
+					    int queue)
+{
+	struct net_device *netdev = adapter->netdev;
+	int i;
+
+	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+		struct ibmveth_buff_pool *bpool =
+			&adapter->rx_buff_pool[queue][i];
+
+		if (!bpool->active)
+			continue;
+
+		if (ibmveth_alloc_buffer_pool(bpool)) {
+			netdev_err(netdev,
+				   "queue %d pool %d alloc failed (size=%u count=%u)\n",
+				   queue, i,
+				   bpool->buff_size,
+				   bpool->size);
+			bpool->active = 0;
+			/* Free by allocation presence, not pool->active.
+			 * Needed on later resize paths (a pool can still
+			 * hold memory after active is cleared). This
+			 * open-fail slot is already empty:
+			 * alloc_buffer_pool() undoes its own partials.
+			 */
+			ibmveth_free_queue_buffer_pools(adapter, queue);
+			return -ENOMEM;
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * ibmveth_alloc_buffer_pools - Allocate buffer pools for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Initializes pool metadata for queues 1-N from queue 0 settings,
+ * then allocates buffer pools for all queues using the helper function.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
+{
+	struct net_device *netdev = adapter->netdev;
+	int i, q, rc;
+
+	/* Initialize pool metadata for queues 1..N from queue 0 settings */
+	for (q = 1; q < adapter->num_rx_queues; q++) {
+		for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+			struct ibmveth_buff_pool *src =
+				&adapter->rx_buff_pool[0][i];
+			struct ibmveth_buff_pool *dst =
+				&adapter->rx_buff_pool[q][i];
+
+			dst->size = src->size;
+			dst->index = src->index;
+			dst->buff_size = src->buff_size;
+			dst->threshold = src->threshold;
+			dst->active = src->active;
+		}
+	}
+
+	/* Allocate actual buffers for all queues */
+	for (q = 0; q < adapter->num_rx_queues; q++) {
+		rc = ibmveth_alloc_queue_buffer_pools(adapter, q);
+		if (rc) {
+			/* Free pools for all previous queues */
+			while (--q >= 0)
+				ibmveth_free_queue_buffer_pools(adapter, q);
+			return rc;
+		}
+	}
+
+	netdev_dbg(netdev, "allocated buffer pools for %u queue(s)\n",
+		   adapter->num_rx_queues);
+	return 0;
+}
+
+/**
+ * ibmveth_free_buffer_pools - Free buffer pools for all queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Frees buffer pools for all queues using the helper function.
+ */
+static void
+ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
+{
+	int q;
+
+	/* Free buffer pools for all queues */
+	for (q = 0; q < adapter->num_rx_queues; q++)
+		ibmveth_free_queue_buffer_pools(adapter, q);
+
+	netdev_dbg(adapter->netdev, "freed buffer pools for %u queue(s)\n",
+		   adapter->num_rx_queues);
+}
+
 /**
  * ibmveth_remove_buffer_from_pool - remove a buffer from a pool
  * @adapter: adapter instance
@@ -858,16 +996,9 @@ static int ibmveth_open(struct net_device *netdev)
 		goto out_free_tx_ltb;
 	}
 
-	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-		if (!adapter->rx_buff_pool[0][i].active)
-			continue;
-		if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[0][i])) {
-			netdev_err(netdev, "unable to alloc pool\n");
-			adapter->rx_buff_pool[0][i].active = 0;
-			rc = -ENOMEM;
-			goto out_free_buffer_pools;
-		}
-	}
+	rc = ibmveth_alloc_buffer_pools(adapter);
+	if (rc)
+		goto out_free_tx_ltb;
 
 	netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
 	rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
@@ -892,11 +1023,7 @@ static int ibmveth_open(struct net_device *netdev)
 	return 0;
 
 out_free_buffer_pools:
-	while (--i >= 0) {
-		if (adapter->rx_buff_pool[0][i].active)
-			ibmveth_free_buffer_pool(adapter,
-						 &adapter->rx_buff_pool[0][i]);
-	}
+	ibmveth_free_buffer_pools(adapter);
 out_free_tx_ltb:
 	while (--i >= 0)
 		ibmveth_free_tx_ltb(adapter, i);
@@ -935,14 +1062,10 @@ static int ibmveth_close(struct net_device *netdev)
 
 	ibmveth_update_rx_no_buffer(adapter);
 
+	ibmveth_free_buffer_pools(adapter);
 	ibmveth_cleanup_rx_resources(adapter);
 	ibmveth_free_filter_list(adapter);
 
-	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
-		if (adapter->rx_buff_pool[0][i].active)
-			ibmveth_free_buffer_pool(adapter,
-						 &adapter->rx_buff_pool[0][i]);
-
 	for (i = 0; i < netdev->real_num_tx_queues; i++)
 		ibmveth_free_tx_ltb(adapter, i);
 
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 05/15] ibmveth: Refactor RX interrupt control for MQ RX queues

[Mingming Cao]

Queue 0 and subordinate RX queues use different interrupt control
interfaces in PHYP:

  - queue 0: h_vio_signal() after h_register_logical_lan()
  - queue N: H_VIOCTL against the queue's mapped hwirq

The current code is single-queue oriented and cannot safely scale to
multiple RX queues in poll completion and open/close IRQ setup.

Introduce queue-indexed interrupt helpers and wire them into
open()/close()/poll()/interrupt in the same patch:

  ibmveth_toggle_irq() / enable_irq() / disable_irq()
  ibmveth_setup_rx_interrupts() / ibmveth_cleanup_rx_interrupts()
  ibmveth_schedule_rx_queue()

These helpers centralize queue0-vs-subordinate dispatch.
request_irq() uses &adapter->napi[i] as the per-queue cookie so the
handler can resolve the queue index.

Move napi_enable() into setup_rx_interrupts() (after LAN registration
and buffer-pool allocation): request_irq -> napi_enable. In this
single-queue tree, setup does not yet unmask PHYP; schedule_rx_queue()
masks queue 0 and schedules NAPI, and ibmveth_poll() is what unmasks
it on completion. That order matches the later scale-up rule (NAPI
live before PHYP unmask), not an inverted window relative to it.

Factor process-context RX kicks (open, resume, pool sysfs, netpoll)
into ibmveth_schedule_rx_queue(); keep ibmveth_interrupt() as a thin
IRQ-only wrapper.

cleanup_rx_interrupts() masks PHYP and synchronizes IRQs before
napi_disable, remasks and synchronizes again after it because an
in-flight poll can re-arm, then free_irq. Close then proceeds to
h_free_logical_lan(): free_irq before free_lan is intentional once
PHYP delivery is masked.

On setup_rx enable-fail (MQ path), if enable_irq() fails for queue i,
remask+sync queues 0..i, including the one that failed, before
napi_disable/free_irq; the rollback loop used while (--i) and skipped
it. H_PARAMETER stays an error on enable, so PHYP may already be
unmasked; an unmasked queue must not drive schedule_rx, which would
prep-fail without mask during the napi_disable wait (STOP storm).
err_disable_napi mirrors cleanup remask after napi_disable.

opened / rx_irq_setup gate whether cleanup walks IRQ/NAPI state.
Opened / rx_irq_setup also closes a pre-existing hang: after a
failed reopen, a later ndo_stop used to napi_disable and free_irq
a second time (rtnl spin + already-free IRQ). That depends on the
helpers in this patch, so there is no standalone Fixes: tag.

schedule_rx_queue() masks PHYP only when napi_schedule_prep() succeeds.
Masking on prep failure can race a completing poll that already
re-enabled PHYP and leave NAPI idle with the queue masked (TX OK, RX
stalled until reload). Teardown storm control stays on STOP
(disable_irq + synchronize_irq before napi_disable) and the
poll_stopping() re-arm guard added in P09, not on the schedule helper
failure path.

IRQ helpers return 0 or negative errno only (never raw H_* to
ethtool/resize). H_PARAMETER is folded to success only on disable
(idempotent mask). On enable it remains an error so a stuck-masked
queue stays visible to poll/resize recovery.

Runtime remains single-queue (num_rx_queues is still 1).

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- setup_rx enable-fail remasks queues 0..i, including the one that
  failed (the rollback used while (--i) and left queue i unmasked
  across napi_disable); err_disable_napi remasks after napi_disable
- drop WARN_ON() on enable/disable_irq() returns (schedule_rx and
  poll); the helper already logs the hcall rc, and rate-limit that
  print
- reword schedule_rx_queue() kdoc: true means NAPI was scheduled and
  the mask attempted; a failed disable_irq() does not change the
  return, and an out-of-range qindex returns false
- noted: pool-fail without h_free predates this; the opened gate
  also drops the accidental h_free a later ndo_stop used to give.
  P06/P07 close it
- noted: set_channels() IFF_UP vs opened TX LTB window closes in P14/P15
- noted: poll() takes queue_index in P08; range check / skip helpers
  in P09

Changes in v5:
- Remask+sync after napi_disable in cleanup (in-flight poll can re-arm)
- Interrupt: quiet IRQ_NONE on out-of-range qindex (no WARN storm)
- Opened / rx_irq_setup gate cleanup so close after a failed open
  cannot napi_disable / free_irq without a prior enable/request; set
  opened on successful open; rx_irq_setup only on full setup success
- H_PARAMETER fold disable-only; enable stays error (stuck-masked visible)
- IRQ helpers return 0 / negative errno only (never raw H_* to ethtool)
- Decision: keep open IRQ order request_irq -> napi_enable while PHYP
  stays masked until schedule/enable (coherent with later scale-up)
- schedule_rx_queue returns bool (napi_schedule_prep success)
- Keep mask-only-on-prep-success (no else-mask; avoids idle+masked race)
- Call out free_irq-before-free_lan as intentional once PHYP is masked
- Poll re-arm during teardown lands with SQ poll-harden (not claimed here)
- synchronize_net() after RX IRQ/NAPI teardown in close
- Add subordinate IRQ dispose helpers (per-queue + bulk 1..N; bound to MAX)

Changes in v4:
- Include irq.h / irqdomain.h with first irq_dispose_mapping() use.
- Introduce IRQ helpers in the same patch that wires open/close/poll
  callers, instead of leaving unused statics.
- Factor process-context RX kicks into ibmveth_schedule_rx_queue();
  keep ibmveth_interrupt() as the IRQ-only wrapper.
- On cleanup, mask PHYP and synchronize_irq before napi_disable
  (storm-safety; not fully behavior-preserving vs classic close).
- Leave queue_irq[0] set after cleanup (queue 0 uses netdev->irq;
  next open reuses it). Only subordinate virqs are disposed.

 drivers/net/ethernet/ibm/ibmveth.c | 404 ++++++++++++++++++++++++++---
 drivers/net/ethernet/ibm/ibmveth.h |   4 +
 2 files changed, 367 insertions(+), 41 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 37a6d13e603e..335712faaa42 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -21,6 +21,8 @@
 #include <linux/skbuff.h>
 #include <linux/init.h>
 #include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
 #include <linux/mm.h>
 #include <linux/pm.h>
 #include <linux/ethtool.h>
@@ -338,6 +340,320 @@ ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
 	}
 }
 
+/**
+ * ibmveth_toggle_irq - Common helper to enable/disable queue interrupts
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ * @enable: true to enable, false to disable
+ *
+ * For queue 0 (primary), uses h_vio_signal() as it's registered via
+ * h_register_logical_lan(). For subordinate queues (1+), uses H_VIOCTL
+ * with H_ENABLE/DISABLE_VIO_INTERRUPT for per-queue interrupt control.
+ *
+ * Return: 0 on success, negative errno on failure (never raw H_*).
+ */
+static int
+ibmveth_toggle_irq(struct ibmveth_adapter *adapter, int queue_index,
+		   bool enable)
+{
+	unsigned long h_rc;
+	unsigned long irq = adapter->queue_irq[queue_index];
+	const char *action = enable ? "enable" : "disable";
+
+	if (queue_index == 0) {
+		/* Primary queue: use h_vio_signal() */
+		h_rc = h_vio_signal(adapter->vdev->unit_address,
+				    enable ? VIO_IRQ_ENABLE : VIO_IRQ_DISABLE);
+	} else {
+		/* Subordinate queues: use H_VIOCTL with hardware IRQ */
+		struct irq_data *irq_data = irq_get_irq_data(irq);
+		irq_hw_number_t hwirq;
+		u64 vioctl_cmd = enable ? H_ENABLE_VIO_INTERRUPT :
+			H_DISABLE_VIO_INTERRUPT;
+
+		if (!irq_data) {
+			netdev_err(adapter->netdev,
+				   "Failed to get IRQ data for queue %d (virq=%lu)\n",
+				   queue_index, irq);
+			return -EINVAL;
+		}
+
+		hwirq = irqd_to_hwirq(irq_data);
+		h_rc = plpar_hcall_norets(H_VIOCTL,
+					  adapter->vdev->unit_address,
+					  vioctl_cmd,
+					  hwirq, 0, 0);
+
+		/*
+		 * H_PARAMETER is ambiguous (already in requested state vs bad
+		 * args). Fold only on disable as an idempotent mask. On enable
+		 * keep it an error so a stuck-masked queue stays visible to
+		 * poll/resize recovery.
+		 */
+		if (h_rc == H_PARAMETER && !enable) {
+			dev_warn_ratelimited(&adapter->netdev->dev,
+					     "H_VIOCTL %s IRQ returned H_PARAMETER for queue %d (hwirq=%lu)\n",
+					     action, queue_index, hwirq);
+			return 0;
+		}
+	}
+
+	if (h_rc) {
+		dev_err_ratelimited(&adapter->netdev->dev,
+				    "Failed to %s IRQ for queue %d, rc=0x%lx\n",
+				    action, queue_index, h_rc);
+		return -EIO;
+	}
+	return 0;
+}
+
+/**
+ * ibmveth_disable_irq - Disable interrupt for a specific queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int
+ibmveth_disable_irq(struct ibmveth_adapter *adapter, int queue_index)
+{
+	return ibmveth_toggle_irq(adapter, queue_index, false);
+}
+
+/**
+ * ibmveth_enable_irq - Enable interrupt for a specific queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Index of the queue (0 for primary, 1+ for subordinate)
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int
+ibmveth_enable_irq(struct ibmveth_adapter *adapter, int queue_index)
+{
+	return ibmveth_toggle_irq(adapter, queue_index, true);
+}
+
+/**
+ * ibmveth_dispose_subordinate_irq_mapping - Drop one subordinate virq mapping
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: RX queue index (1..N)
+ *
+ * Subordinate queues get mappings from irq_create_mapping() during PHYP
+ * registration. Queue 0 uses netdev->irq from device tree and is left alone.
+ *
+ * Bound against IBMVETH_MAX_RX_QUEUES, not num_rx_queues: a caller may
+ * dispose a queue that is no longer in the published live set but still
+ * owns a virq in queue_irq[]. Contrast with the bulk helper, which only
+ * walks 1..num_rx_queues-1 (close / open-fail cleanup of the live set).
+ *
+ * Linux virq lifetime is owned by interrupt cleanup helpers. Call this only
+ * after free_irq() when a handler was installed, or from registration failure
+ * cleanup before request_irq().
+ */
+static void
+ibmveth_dispose_subordinate_irq_mapping(struct ibmveth_adapter *adapter,
+					int queue_idx)
+{
+	if (queue_idx <= 0 || queue_idx >= IBMVETH_MAX_RX_QUEUES)
+		return;
+
+	if (adapter->queue_irq[queue_idx]) {
+		irq_dispose_mapping(adapter->queue_irq[queue_idx]);
+		adapter->queue_irq[queue_idx] = 0;
+	}
+}
+
+/**
+ * ibmveth_dispose_subordinate_irq_mappings - Drop virq mappings for queues 1..N
+ * @adapter: ibmveth adapter structure
+ *
+ * Bulk helper for close / open-fail cleanup of the published live set
+ * (queues 1..num_rx_queues-1). Paths that need a retired or not-yet-published
+ * queue must call ibmveth_dispose_subordinate_irq_mapping() directly.
+ */
+static void
+ibmveth_dispose_subordinate_irq_mappings(struct ibmveth_adapter *adapter)
+{
+	int i;
+
+	for (i = 1; i < adapter->num_rx_queues; i++)
+		ibmveth_dispose_subordinate_irq_mapping(adapter, i);
+}
+
+/**
+ * ibmveth_setup_rx_interrupts - Register IRQs and enable NAPI
+ * @adapter: ibmveth adapter structure
+ *
+ * Registers interrupt handlers for all RX queues, enables NAPI, then
+ * enables hypervisor interrupt delivery for multi-queue mode after
+ * every queue has a Linux handler installed. For multi-queue open the
+ * caller should replenish RX buffers before this helper so traffic
+ * during open is not dropped (PHYP only interrupts after a successful
+ * enqueue, which needs buffers). Single-queue open leaves PHYP masked
+ * here and kicks NAPI afterward (classic path: first poll posts then
+ * enables).
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
+{
+	struct net_device *netdev = adapter->netdev;
+	int i, rc, num = adapter->num_rx_queues;
+
+	for (i = 0; i < num; i++) {
+		if (!adapter->queue_irq[i]) {
+			netdev_err(netdev, "queue %d has invalid IRQ (0)\n", i);
+			rc = -EINVAL;
+			goto err_free_irqs;
+		}
+
+		rc = request_irq(adapter->queue_irq[i], ibmveth_interrupt,
+				 0, netdev->name, &adapter->napi[i]);
+		if (rc) {
+			netdev_err(netdev,
+				   "request_irq() failed for irq 0x%x queue %d: %d\n",
+				   adapter->queue_irq[i], i, rc);
+			goto err_free_irqs;
+		}
+	}
+
+	for (i = 0; i < num; i++)
+		napi_enable(&adapter->napi[i]);
+
+	if (adapter->multi_queue && num > 1) {
+		for (i = 0; i < num; i++) {
+			rc = ibmveth_enable_irq(adapter, i);
+			if (rc) {
+				netdev_err(netdev,
+					   "Failed to enable IRQ for queue %d, rc=%d\n",
+					   i, rc);
+				for (; i >= 0; i--) {
+					ibmveth_disable_irq(adapter, i);
+					synchronize_irq(adapter->queue_irq[i]);
+				}
+				rc = -EIO;
+				goto err_disable_napi;
+			}
+		}
+	}
+
+	/* Set only on full success; fail paths leave this false so a later
+	 * close() / cleanup is a no-op.
+	 */
+	adapter->rx_irq_setup = true;
+	return 0;
+
+err_disable_napi:
+	/* STOP: remask after napi_disable; an in-flight poll can re-arm. */
+	for (i = 0; i < num; i++)
+		napi_disable(&adapter->napi[i]);
+	for (i = 0; i < num; i++) {
+		if (!adapter->queue_irq[i])
+			continue;
+		ibmveth_disable_irq(adapter, i);
+		synchronize_irq(adapter->queue_irq[i]);
+	}
+	for (i = 0; i < num; i++) {
+		if (adapter->queue_irq[i])
+			free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+	}
+	goto err_dispose_mappings;
+
+err_free_irqs:
+	while (--i >= 0)
+		free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+err_dispose_mappings:
+	/* Both setup failure paths own subordinate virq disposal. */
+	ibmveth_dispose_subordinate_irq_mappings(adapter);
+	return rc;
+}
+
+/**
+ * ibmveth_cleanup_rx_interrupts - Mask PHYP IRQs, stop NAPI, and free IRQs
+ * @adapter: ibmveth adapter structure
+ *
+ * Mask and synchronize each queue IRQ before napi_disable() so the handler
+ * cannot miss a PHYP mask while NAPI is already dead. Remask after
+ * napi_disable() in case an in-flight poll re-armed PHYP while we waited.
+ * free_irq() runs only after that. Safe for close and for open failure after
+ * setup_rx_interrupts() already unmasked PHYP. No-op if setup never
+ * succeeded (avoids double napi_disable / free_irq after a failed close+open
+ * while IFF_UP remains set).
+ */
+static void
+ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
+{
+	int i;
+
+	if (!adapter->rx_irq_setup)
+		return;
+
+	for (i = 0; i < adapter->num_rx_queues; i++) {
+		if (!adapter->queue_irq[i])
+			continue;
+		ibmveth_disable_irq(adapter, i);
+		synchronize_irq(adapter->queue_irq[i]);
+	}
+
+	for (i = 0; i < adapter->num_rx_queues; i++)
+		napi_disable(&adapter->napi[i]);
+
+	for (i = 0; i < adapter->num_rx_queues; i++) {
+		if (!adapter->queue_irq[i])
+			continue;
+		ibmveth_disable_irq(adapter, i);
+		synchronize_irq(adapter->queue_irq[i]);
+	}
+
+	for (i = 0; i < adapter->num_rx_queues; i++) {
+		if (adapter->queue_irq[i])
+			free_irq(adapter->queue_irq[i], &adapter->napi[i]);
+	}
+
+	ibmveth_dispose_subordinate_irq_mappings(adapter);
+
+	/* Queue 0 uses netdev->irq; leave queue_irq[0] for next open. */
+	adapter->rx_irq_setup = false;
+}
+
+/**
+ * ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue
+ * @adapter: ibmveth adapter structure
+ * @qindex: RX queue index
+ *
+ * Shared by the IRQ handler and process-context kick sites (open, resume,
+ * pool sysfs, poll_controller).
+ *
+ * Return: true if napi_schedule_prep() succeeded and NAPI was scheduled.
+ * Mask is attempted in that case; a failed disable_irq() is logged by the
+ * helper and does not change the return (queue may still be unmasked).
+ * false if the index is out of range or prep failed (including NAPI
+ * already scheduled).
+ */
+static bool ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
+				      int qindex)
+{
+	struct napi_struct *napi = &adapter->napi[qindex];
+
+	if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
+		return false;
+
+	/*
+	 * Only mask PHYP when NAPI will run. Masking on prep failure can
+	 * race a completing poll that already re-enabled the queue, leaving
+	 * NAPI idle with the IRQ masked (TX works, RX stalls) until reload.
+	 * Storm prevention on teardown remains in cleanup/disable paths.
+	 */
+	if (napi_schedule_prep(napi)) {
+		/* Failure is already logged with the hcall rc by the helper. */
+		ibmveth_disable_irq(adapter, qindex);
+		__napi_schedule(napi);
+		return true;
+	}
+	return false;
+}
+
 /* setup the initial settings for a buffer pool */
 static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,
 				     u32 pool_index, u32 pool_size,
@@ -954,8 +1270,6 @@ static int ibmveth_open(struct net_device *netdev)
 
 	netdev_dbg(netdev, "open starting\n");
 
-	napi_enable(&adapter->napi[0]);
-
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		rxq_entries += adapter->rx_buff_pool[0][i].size;
 
@@ -979,7 +1293,8 @@ static int ibmveth_open(struct net_device *netdev)
 					adapter->rx_queue[0].queue_len;
 	rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
 
-	h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
+	adapter->queue_irq[0] = netdev->irq;
+	ibmveth_disable_irq(adapter, 0);
 
 	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
 
@@ -1000,24 +1315,20 @@ static int ibmveth_open(struct net_device *netdev)
 	if (rc)
 		goto out_free_tx_ltb;
 
-	netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
-	rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
-			 netdev);
-	if (rc != 0) {
-		netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
-			   netdev->irq, rc);
+	rc = ibmveth_setup_rx_interrupts(adapter);
+	if (rc) {
 		do {
 			lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
 		} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-
 		goto out_free_buffer_pools;
 	}
 
 	netdev_dbg(netdev, "initial replenish cycle\n");
-	ibmveth_interrupt(netdev->irq, netdev);
+	ibmveth_schedule_rx_queue(adapter, 0);
 
 	netif_tx_start_all_queues(netdev);
 
+	adapter->opened = true;
 	netdev_dbg(netdev, "open complete\n");
 
 	return 0;
@@ -1031,7 +1342,6 @@ static int ibmveth_open(struct net_device *netdev)
 out_free_filter_list:
 	ibmveth_free_filter_list(adapter);
 out:
-	napi_disable(&adapter->napi[0]);
 	return rc;
 }
 
@@ -1041,27 +1351,32 @@ static int ibmveth_close(struct net_device *netdev)
 	long lpar_rc;
 	int i;
 
-	netdev_dbg(netdev, "close starting\n");
+	/* Gate on opened, not IFF_UP: pool_store/change_mtu close+open can
+	 * leave IFF_UP set after a failed reopen.
+	 */
+	if (!adapter->opened)
+		return 0;
 
-	napi_disable(&adapter->napi[0]);
+	adapter->opened = false;
+
+	netdev_dbg(netdev, "close starting\n");
 
 	netif_tx_stop_all_queues(netdev);
 
-	h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
+	ibmveth_cleanup_rx_interrupts(adapter);
+	/* Wait for softirq/poll that already passed shutdown checks. */
+	synchronize_net();
 
+	ibmveth_update_rx_no_buffer(adapter);
+	/* Full LAN teardown (subordinates arrive with register helpers). */
 	do {
 		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
 	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-
 	if (lpar_rc != H_SUCCESS) {
-		netdev_err(netdev, "h_free_logical_lan failed with %lx, "
-			   "continuing with close\n", lpar_rc);
+		netdev_err(adapter->netdev,
+			   "h_free_logical_lan failed with %lx, continuing\n",
+			   lpar_rc);
 	}
-
-	free_irq(netdev->irq, netdev);
-
-	ibmveth_update_rx_no_buffer(adapter);
-
 	ibmveth_free_buffer_pools(adapter);
 	ibmveth_cleanup_rx_resources(adapter);
 	ibmveth_free_filter_list(adapter);
@@ -1705,7 +2020,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 			container_of(napi, struct ibmveth_adapter, napi[0]);
 	struct net_device *netdev = adapter->netdev;
 	int frames_processed = 0;
-	unsigned long lpar_rc;
+	int rc;
 	u16 mss = 0;
 
 restart_poll:
@@ -1805,15 +2120,14 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 	/* We think we are done - reenable interrupts,
 	 * then check once more to make sure we are done.
 	 */
-	lpar_rc = h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_ENABLE);
-	if (WARN_ON(lpar_rc != H_SUCCESS)) {
+	rc = ibmveth_enable_irq(adapter, 0);
+	if (rc) {
 		schedule_work(&adapter->work);
 		goto out;
 	}
 
 	if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule(napi)) {
-		lpar_rc = h_vio_signal(adapter->vdev->unit_address,
-				       VIO_IRQ_DISABLE);
+		ibmveth_disable_irq(adapter, 0);
 		goto restart_poll;
 	}
 
@@ -1823,16 +2137,20 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 
 static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
 {
-	struct net_device *netdev = dev_instance;
+	struct napi_struct *napi = dev_instance;
+	struct net_device *netdev = napi->dev;
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
-	unsigned long lpar_rc;
+	int qindex;
 
-	if (napi_schedule_prep(&adapter->napi[0])) {
-		lpar_rc = h_vio_signal(adapter->vdev->unit_address,
-				       VIO_IRQ_DISABLE);
-		WARN_ON(lpar_rc != H_SUCCESS);
-		__napi_schedule(&adapter->napi[0]);
-	}
+	qindex = napi - adapter->napi;
+	/*
+	 * Quiet on out-of-range: teardown can leave a residual IRQ after the
+	 * live count drops. Do not WARN-storm; return IRQ_NONE until free_irq.
+	 */
+	if (qindex < 0 || qindex >= adapter->num_rx_queues)
+		return IRQ_NONE;
+
+	ibmveth_schedule_rx_queue(adapter, qindex);
 	return IRQ_HANDLED;
 }
 
@@ -1937,8 +2255,10 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
 #ifdef CONFIG_NET_POLL_CONTROLLER
 static void ibmveth_poll_controller(struct net_device *dev)
 {
-	ibmveth_replenish_task(netdev_priv(dev));
-	ibmveth_interrupt(dev->irq, dev);
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+
+	ibmveth_replenish_task(adapter);
+	ibmveth_schedule_rx_queue(adapter, 0);
 }
 #endif
 
@@ -2351,8 +2671,8 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
 	}
 	rtnl_unlock();
 
-	/* kick the interrupt handler to allocate/deallocate pools */
-	ibmveth_interrupt(netdev->irq, netdev);
+	/* kick RX processing to allocate/deallocate pools */
+	ibmveth_schedule_rx_queue(adapter, 0);
 	return count;
 
 unlock_err:
@@ -2392,7 +2712,9 @@ static struct kobj_type ktype_veth_pool = {
 static int ibmveth_resume(struct device *dev)
 {
 	struct net_device *netdev = dev_get_drvdata(dev);
-	ibmveth_interrupt(netdev->irq, netdev);
+	struct ibmveth_adapter *adapter = netdev_priv(netdev);
+
+	ibmveth_schedule_rx_queue(adapter, 0);
 	return 0;
 }
 
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 495269631323..c13240f0ea2e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -315,6 +315,10 @@ struct ibmveth_adapter {
 	unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
 	int multi_queue;
 	unsigned int num_rx_queues;
+	/* Lifetime: true after successful ndo_open until close clears it. */
+	bool opened;
+	/* Lifetime: true while RX IRQ handlers / NAPI are installed. */
+	bool rx_irq_setup;
 	int rx_csum;
 	int large_send;
 	bool is_active_trunk;
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 06/15] ibmveth: Refactor TX resource allocation in open/close paths

[Mingming Cao]

Same story as the RX refactor: pull TX LTB alloc/free out of open/close
into helpers and wire them in this patch.

  ibmveth_alloc_tx_resources()
  ibmveth_free_tx_resources()

They wrap the existing per-queue allocate_tx_ltb() / free_tx_ltb()
primitives. alloc_tx_resources() allocates every TX queue and unwinds
partial failure itself; free_tx_resources() walks real_num_tx_queues.
The helpers remove dependence on shared open/close loop indices and
match the RX helper structure. TX was already multi-queue capable via
ethtool -L.

Also tighten TX LTB lifetime: free_tx_ltb() returns early if
tx_ltb_ptr[] is already NULL, then clears both tx_ltb_ptr[] and
tx_ltb_dma[] before unmapping and freeing, so start_xmit() cannot pick
up a slot that is mid-teardown. allocate_tx_ltb() clears tx_ltb_dma[]
on the DMA-map failure path.

Move TX LTB allocation to the end of open(), after LAN registration,
RX pools, RX interrupt setup, and the initial replenish kick. A late
alloc_tx_resources() failure jumps to out_cleanup_rx_interrupts and
must not call free_tx_resources() again: alloc already freed any
partial TX LTBs. start_xmit() bails if tx_ltb_ptr[] is gone, counting
the drop in tx_dropped and falling into the existing out: label like
the function's other drop paths, so RX can be live while TX LTB alloc
still runs and close/failed-reopen cannot race a live mapping. The
close path is quiesced by netif_tx_disable(); NULL-first in
free_tx_ltb() only closes the check-then-use window, it is not itself
a UAF barrier. set_channels() IFF_UP vs opened is later (P14/P15).

After LAN registration, open-fail teardown issues h_free_logical_lan()
before RX pool DMA teardown on the pool-fail path that previously never
issued that hcall (missing deregistration, not a preference reorder).

close() quiesces TX with netif_tx_disable() (stop_all_queues does not
wait for in-flight ndo_start_xmit), then frees LTBs after
h_free_logical_lan() via free_tx_resources() - required because direct
close() callers bypass synchronize_net().

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- NULL tx_ltb_ptr[idx] and zero tx_ltb_dma[idx] before unmap/free, so a
  racing start_xmit() fails the pointer check. Close-path safety is
  still netif_tx_disable()
- the NULL-LTB start_xmit drop increments tx_dropped and falls into
  the existing out: label
- comment on allocate_tx_ltb(): caller must leave tx_ltb_ptr[idx] NULL
- kdoc alloc/free_tx_resources says real_num_tx_queues
- noted: ethtool -L TX shrink still stop-then-free; cover leftovers

Changes in v5:
- Quiesce TX with netif_tx_disable before free (stop_all_queues does not
  wait for in-flight xmit); free LTBs after h_free_logical_lan - direct
  close() callers bypass synchronize_net()
- Guard start_xmit if tx_ltb_ptr gone so open can leave RX live while TX
  LTB alloc still runs (also covers close/failed-reopen with IFF_UP set)
- Drop fake mid-open TX-leak / Fixes: motivation; reword as helper
  extraction matching RX (shared loop-index independence)
- Free TX LTB by pointer presence (drop dma==0 sentinel; dma_mapping_error
  already cleared the slot on map failure)
- Document intentional open-fail LAN-first unwind (free_lan before RX
  pool/DMA teardown) rather than leaving it silent in a TX-only refactor
- Drop drive-by blank-line cosmetics (header / start_xmit)

Changes in v4:
- Introduce the TX resource helpers in the same patch that wires their
  first open/close callers.
- Do not free TX LTBs again after a failed alloc_tx_resources();
  harden free_tx_ltb() against unset slots.
- Move TX allocation after RX IRQ setup / replenish kick so open()
  failure unwind no longer depends on a shared loop index (also fixes
  a mid-open TX LTB leak).

 drivers/net/ethernet/ibm/ibmveth.c | 115 ++++++++++++++++++++++-------
 1 file changed, 89 insertions(+), 26 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 335712faaa42..7a420e1a41d5 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1201,12 +1201,27 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
 
 static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
 {
-	dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
-			 adapter->tx_ltb_size, DMA_TO_DEVICE);
-	kfree(adapter->tx_ltb_ptr[idx]);
+	void *ltb = adapter->tx_ltb_ptr[idx];
+	dma_addr_t dma = adapter->tx_ltb_dma[idx];
+
+	if (!ltb)
+		return;
+
+	/*
+	 * Clear the slot before releasing it. start_xmit() tests
+	 * tx_ltb_ptr[idx] to decide whether the LTB is usable.
+	 */
 	adapter->tx_ltb_ptr[idx] = NULL;
+	adapter->tx_ltb_dma[idx] = 0;
+
+	dma_unmap_single(&adapter->vdev->dev, dma, adapter->tx_ltb_size,
+			 DMA_TO_DEVICE);
+	kfree(ltb);
 }
 
+/* Caller must ensure tx_ltb_ptr[idx] is NULL. open() runs on
+ * probe-zeroed slots; set_channels() skips populated indices.
+ */
 static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
 {
 	adapter->tx_ltb_ptr[idx] = kzalloc(adapter->tx_ltb_size,
@@ -1225,12 +1240,54 @@ static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
 			   "unable to DMA map tx long term buffer\n");
 		kfree(adapter->tx_ltb_ptr[idx]);
 		adapter->tx_ltb_ptr[idx] = NULL;
+		adapter->tx_ltb_dma[idx] = 0;
 		return -ENOMEM;
 	}
 
 	return 0;
 }
 
+/**
+ * ibmveth_alloc_tx_resources - Allocate TX LTBs for real_num_tx_queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Allocates TX Long Term Buffers (LTBs) for real_num_tx_queues.
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_resources(struct ibmveth_adapter *adapter)
+{
+	struct net_device *netdev = adapter->netdev;
+	int i;
+
+	for (i = 0; i < netdev->real_num_tx_queues; i++) {
+		if (ibmveth_allocate_tx_ltb(adapter, i))
+			goto err_free_ltbs;
+	}
+
+	return 0;
+
+err_free_ltbs:
+	while (--i >= 0)
+		ibmveth_free_tx_ltb(adapter, i);
+	return -ENOMEM;
+}
+
+/**
+ * ibmveth_free_tx_resources - Free TX LTBs for real_num_tx_queues
+ * @adapter: ibmveth adapter structure
+ *
+ * Frees TX Long Term Buffers (LTBs) for real_num_tx_queues.
+ */
+static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)
+{
+	struct net_device *netdev = adapter->netdev;
+	int i;
+
+	for (i = 0; i < netdev->real_num_tx_queues; i++)
+		ibmveth_free_tx_ltb(adapter, i);
+}
+
 static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
         union ibmveth_buf_desc rxq_desc, u64 mac_address)
 {
@@ -1281,12 +1338,6 @@ static int ibmveth_open(struct net_device *netdev)
 	if (rc)
 		goto out_free_filter_list;
 
-	rc = -ENOMEM;
-	for (i = 0; i < netdev->real_num_tx_queues; i++) {
-		if (ibmveth_allocate_tx_ltb(adapter, i))
-			goto out_free_tx_ltb;
-	}
-
 	mac_address = ether_addr_to_u64(netdev->dev_addr);
 
 	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
@@ -1308,24 +1359,24 @@ static int ibmveth_open(struct net_device *netdev)
 				     rxq_desc.desc,
 				     mac_address);
 		rc = -ENONET;
-		goto out_free_tx_ltb;
+		goto out_free_queue_mem;
 	}
 
 	rc = ibmveth_alloc_buffer_pools(adapter);
 	if (rc)
-		goto out_free_tx_ltb;
+		goto out_unregister_lan;
 
 	rc = ibmveth_setup_rx_interrupts(adapter);
-	if (rc) {
-		do {
-			lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
-		} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-		goto out_free_buffer_pools;
-	}
+	if (rc)
+		goto out_unregister_lan;
 
 	netdev_dbg(netdev, "initial replenish cycle\n");
 	ibmveth_schedule_rx_queue(adapter, 0);
 
+	rc = ibmveth_alloc_tx_resources(adapter);
+	if (rc)
+		goto out_cleanup_rx_interrupts;
+
 	netif_tx_start_all_queues(netdev);
 
 	adapter->opened = true;
@@ -1333,11 +1384,14 @@ static int ibmveth_open(struct net_device *netdev)
 
 	return 0;
 
-out_free_buffer_pools:
+out_cleanup_rx_interrupts:
+	ibmveth_cleanup_rx_interrupts(adapter);
+out_unregister_lan:
+	do {
+		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
+	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
 	ibmveth_free_buffer_pools(adapter);
-out_free_tx_ltb:
-	while (--i >= 0)
-		ibmveth_free_tx_ltb(adapter, i);
+out_free_queue_mem:
 	ibmveth_cleanup_rx_resources(adapter);
 out_free_filter_list:
 	ibmveth_free_filter_list(adapter);
@@ -1349,7 +1403,6 @@ static int ibmveth_close(struct net_device *netdev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
 	long lpar_rc;
-	int i;
 
 	/* Gate on opened, not IFF_UP: pool_store/change_mtu close+open can
 	 * leave IFF_UP set after a failed reopen.
@@ -1361,7 +1414,10 @@ static int ibmveth_close(struct net_device *netdev)
 
 	netdev_dbg(netdev, "close starting\n");
 
-	netif_tx_stop_all_queues(netdev);
+	/* Disable and wait for in-flight ndo_start_xmit (stop_all_queues
+	 * alone does not). Direct close() callers bypass synchronize_net().
+	 */
+	netif_tx_disable(netdev);
 
 	ibmveth_cleanup_rx_interrupts(adapter);
 	/* Wait for softirq/poll that already passed shutdown checks. */
@@ -1377,13 +1433,14 @@ static int ibmveth_close(struct net_device *netdev)
 			   "h_free_logical_lan failed with %lx, continuing\n",
 			   lpar_rc);
 	}
+	/* Free TX LTBs after quiesce and after H_FREE_LOGICAL_LAN so xmit
+	 * cannot touch unmapped bounce buffers while the LAN is live.
+	 */
+	ibmveth_free_tx_resources(adapter);
 	ibmveth_free_buffer_pools(adapter);
 	ibmveth_cleanup_rx_resources(adapter);
 	ibmveth_free_filter_list(adapter);
 
-	for (i = 0; i < netdev->real_num_tx_queues; i++)
-		ibmveth_free_tx_ltb(adapter, i);
-
 	netdev_dbg(netdev, "close complete\n");
 
 	return 0;
@@ -1807,6 +1864,12 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 	int i, queue_num = skb_get_queue_mapping(skb);
 	unsigned long mss = 0;
 
+	/* Close / failed reopen can free LTBs while IFF_UP is still set. */
+	if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
+		netdev->stats.tx_dropped++;
+		goto out;
+	}
+
 	if (ibmveth_is_packet_unsupported(skb, netdev))
 		goto out;
 	/* veth can't checksum offload UDP */
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 07/15] ibmveth: Add RX queue register helpers for MQ

[Mingming Cao]

MQ RX changes the RX queue lifecycle from one adapter-level
register/free pair into a mixed model: queue 0 is registered through
h_register_logical_lan*(), queues 1..N through
H_REG_LOGICAL_LAN_QUEUE. The queue-0 control plane that open() and
close() use today has to come out into helpers first, so the
subordinate paths have somewhere to attach when MQ RX is enabled.

Extract logical LAN registration and queue teardown into dedicated
helpers:

  ibmveth_register_logical_lan()
  ibmveth_register_rx_queues()
  ibmveth_free_all_queues()

Runtime stays single-queue here. adapter->multi_queue is still false
until MQ RX is enabled, so queue 0 keeps the legacy
h_register_logical_lan() flow, the with_handle arm is not yet
reachable, and no subordinate queue is registered.

ibmveth_register_logical_lan() registers queue 0 with PHYP. When
multi-queue mode is active it uses h_register_logical_lan_with_handle()
and stores the handle only on H_SUCCESS, so a failed attempt leaves no
stale handle behind.
ibmveth_register_rx_queues() is the open()-side entry point: it builds
queue 0's buffer descriptor, records queue 0's virq in queue_irq[0],
masks that IRQ before registration, and calls
ibmveth_register_logical_lan(). It registers queue 0 only.
ibmveth_free_all_queues() issues one H_FREE_LOGICAL_LAN and clears all
queue handles. One hypercall is enough because H_FREE_LOGICAL_LAN is a
full teardown: per PAPR/PHYP it drops the primary LAN and any
subordinate queues registered under it. Close and the open-fail unwind
both rely on that. Incremental scale-down cannot, and issues
H_FREE_LOGICAL_LAN_QUEUE per queue instead; that path arrives with
resize.

open() allocates buffer pools before PHYP registration so a pool-fail
path never has a live LAN. Post-register errors still unwind through
ibmveth_free_all_queues().

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- drop the eight hypercall counters, struct ibmveth_hcall_stats, and
  the adapter field. Only reg_lan and free_lan had writers here;
  none of the eight had a reader. The ethtool -S keys v5 added in
  patch 11 are gone with them. This patch no longer touches ibmveth.h
- trace LAN registration and the free retry with netdev_dbg() (open/
  close/resize, not per packet). Teardown already uses netdev_err()
- document free_all_queues() log-and-continue in its kdoc, including
  that queue_handle[] is cleared whatever the hypercall returned
- noted: queue 0 handle is stored only on H_SUCCESS
- noted: body drops the out_unregister_queues walk and the
  unconditional PHYP-vs-pools guarantee

Changes in v5:
- Document H_FREE_LOGICAL_LAN full-teardown contract in kdoc/changelog:
  one free_lan drops primary + any subordinate queues
  (H_FREE_LOGICAL_LAN_QUEUE is for incremental scale-down only) - v4
  helper text did not spell the subordinate semantics
- Call out close() update_rx_no_buffer() before free_lan as intentional
  last-glimpse accounting while the LAN is still registered (order
  already in v4; v4 step list omitted it); NULL-safe per-queue form
  lands with MQ enablement
- opened / rx_irq_setup idempotent close is owned by the IRQ-helper
  patch (same failed-reopen / second napi_disable hang raised here)
- Poll re-arm during teardown not claimed here (same race as IRQ patch;
  lands with poll harden)

Changes in v4:
- Introduce register/free helpers in the same patch that wires their
  first open/close callers; keep subordinate-only helpers deferred
  until MQ enablement.
- Introduce adapter->hcall_stats here (first use); not in patch 2.
- Correct open/close unwind so free_all_queues() precedes
  free_buffer_pools().
- Drop the orphaned big-bang "open/close pipeline" patch from v3; that
  wiring is incremental across helper patches 3-7 instead.

 drivers/net/ethernet/ibm/ibmveth.c | 174 +++++++++++++++++++++--------
 1 file changed, 128 insertions(+), 46 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 7a420e1a41d5..465330464f91 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1289,9 +1289,14 @@ static void ibmveth_free_tx_resources(struct ibmveth_adapter *adapter)
 }
 
 static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
-        union ibmveth_buf_desc rxq_desc, u64 mac_address)
+					union ibmveth_buf_desc rxq_desc,
+					u64 mac_address)
 {
 	int rc, try_again = 1;
+	unsigned long ua = adapter->vdev->unit_address;
+	unsigned long buf_dma = adapter->buffer_list_dma[0];
+	unsigned long filter_dma = adapter->filter_list_dma;
+	unsigned long qh0;
 
 	/*
 	 * After a kexec the adapter will still be open, so our attempt to
@@ -1299,14 +1304,29 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
 	 * try again, but only once.
 	 */
 retry:
-	rc = h_register_logical_lan(adapter->vdev->unit_address,
-				    adapter->buffer_list_dma[0], rxq_desc.desc,
-				    adapter->filter_list_dma, mac_address);
+	/* In multi-queue mode, obtain a queue handle for queue 0 so all RX
+	 * queues can use the same per-queue buffer hypercalls.
+	 */
+	if (adapter->multi_queue) {
+		rc = h_register_logical_lan_with_handle(ua, buf_dma,
+							rxq_desc.desc,
+							filter_dma,
+							mac_address,
+							&qh0);
+		if (rc == H_SUCCESS)
+			adapter->queue_handle[0] = qh0;
+	} else {
+		rc = h_register_logical_lan(ua, buf_dma, rxq_desc.desc,
+					    filter_dma, mac_address);
+	}
+	netdev_dbg(adapter->netdev, "h_register_logical_lan%s rc=%d\n",
+		   adapter->multi_queue ? "_with_handle" : "", rc);
 
 	if (rc != H_SUCCESS && try_again) {
 		do {
 			rc = h_free_logical_lan(adapter->vdev->unit_address);
 		} while (H_IS_LONG_BUSY(rc) || (rc == H_BUSY));
+		netdev_dbg(adapter->netdev, "h_free_logical_lan rc=%d\n", rc);
 
 		try_again = 0;
 		goto retry;
@@ -1315,14 +1335,97 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
 	return rc;
 }
 
+/**
+ * ibmveth_free_all_queues - Free all RX queues at once
+ * @adapter: ibmveth adapter structure
+ *
+ * Issues one H_FREE_LOGICAL_LAN for full adapter teardown. Per PAPR/PHYP,
+ * that drops the primary LAN and any subordinate queues registered under
+ * it. Incremental scale-down uses H_FREE_LOGICAL_LAN_QUEUE per queue
+ * instead; do not use this helper for partial live-set shrink.
+ *
+ * Used during interface close and registration error cleanup.
+ *
+ * Retries only H_BUSY and H_IS_LONG_BUSY. On other failures, logs and
+ * returns; callers cannot observe hypercall status. queue_handle[] is
+ * cleared regardless. Callers still run RX pool and DMA teardown
+ * afterward (same as pre-helper close()).
+ *
+ * Clears queue handles only; queue_irq[] is released by
+ * ibmveth_cleanup_rx_interrupts().
+ */
+static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
+{
+	unsigned long lpar_rc;
+	int i;
+
+	netdev_dbg(adapter->netdev, "freeing all RX queues at once\n");
+
+	do {
+		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
+	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+
+	if (lpar_rc != H_SUCCESS) {
+		netdev_err(adapter->netdev,
+			   "h_free_logical_lan failed: %ld\n", lpar_rc);
+	}
+
+	for (i = 0; i < adapter->num_rx_queues; i++)
+		adapter->queue_handle[i] = 0;
+}
+
+/**
+ * ibmveth_register_rx_queues - Register RX queues with hypervisor
+ * @adapter: ibmveth adapter structure
+ * @mac_address: MAC address for device registration
+ *
+ * Registers queue 0 via ibmveth_register_logical_lan(). Subordinate queue
+ * registration is added when multi-queue RX is enabled.
+ *
+ * Return: 0 on success, -ENONET if queue 0 registration fails
+ */
+static int
+ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
+{
+	struct net_device *netdev = adapter->netdev;
+	union ibmveth_buf_desc rxq_desc;
+	unsigned long lpar_rc;
+	int rc;
+
+	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
+				    adapter->rx_queue[0].queue_len;
+	rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
+	adapter->queue_irq[0] = netdev->irq;
+
+	rc = ibmveth_disable_irq(adapter, 0);
+	if (rc)
+		netdev_dbg(netdev,
+			   "Failed to disable IRQ for queue 0 before registration, rc=%d\n",
+			   rc);
+
+	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
+	if (lpar_rc != H_SUCCESS) {
+		netdev_err(netdev,
+			   "h_register_logical_lan failed: %ld\n", lpar_rc);
+		netdev_err(netdev,
+			   "buffer TCE:0x%llx filter TCE:0x%llx rxq desc:0x%llx MAC:0x%llx\n",
+			   adapter->buffer_list_dma[0],
+			   adapter->filter_list_dma,
+			   rxq_desc.desc, mac_address);
+		return -ENONET;
+	}
+
+	netdev_dbg(netdev,
+		   "registered 1 RX queue with hypervisor (single-queue mode)\n");
+	return 0;
+}
+
 static int ibmveth_open(struct net_device *netdev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
-	u64 mac_address;
+	u64 mac_address = ether_addr_to_u64(netdev->dev_addr);
 	int rxq_entries = 1;
-	unsigned long lpar_rc;
 	int rc;
-	union ibmveth_buf_desc rxq_desc;
 	int i;
 
 	netdev_dbg(netdev, "open starting\n");
@@ -1338,37 +1441,23 @@ static int ibmveth_open(struct net_device *netdev)
 	if (rc)
 		goto out_free_filter_list;
 
-	mac_address = ether_addr_to_u64(netdev->dev_addr);
-
-	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
-					adapter->rx_queue[0].queue_len;
-	rxq_desc.fields.address = adapter->rx_queue[0].queue_dma;
-
-	adapter->queue_irq[0] = netdev->irq;
-	ibmveth_disable_irq(adapter, 0);
-
-	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
-
-	if (lpar_rc != H_SUCCESS) {
-		netdev_err(netdev, "h_register_logical_lan failed with %ld\n",
-			   lpar_rc);
-		netdev_err(netdev, "buffer TCE:0x%llx filter TCE:0x%llx rxq "
-			   "desc:0x%llx MAC:0x%llx\n",
-				     adapter->buffer_list_dma[0],
-				     adapter->filter_list_dma,
-				     rxq_desc.desc,
-				     mac_address);
-		rc = -ENONET;
+	rc = ibmveth_alloc_buffer_pools(adapter);
+	if (rc)
 		goto out_free_queue_mem;
-	}
 
-	rc = ibmveth_alloc_buffer_pools(adapter);
+	rc = ibmveth_register_rx_queues(adapter, mac_address);
 	if (rc)
-		goto out_unregister_lan;
+		goto out_free_buffer_pools;
+
+	rc = netif_set_real_num_rx_queues(netdev, adapter->num_rx_queues);
+	if (rc) {
+		netdev_err(netdev, "failed to set number of rx queues\n");
+		goto out_unregister_queues;
+	}
 
 	rc = ibmveth_setup_rx_interrupts(adapter);
 	if (rc)
-		goto out_unregister_lan;
+		goto out_free_all_queues; /* setup already disposed IRQs */
 
 	netdev_dbg(netdev, "initial replenish cycle\n");
 	ibmveth_schedule_rx_queue(adapter, 0);
@@ -1386,10 +1475,12 @@ static int ibmveth_open(struct net_device *netdev)
 
 out_cleanup_rx_interrupts:
 	ibmveth_cleanup_rx_interrupts(adapter);
-out_unregister_lan:
-	do {
-		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
-	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+	goto out_free_all_queues; /* cleanup already disposed IRQs */
+out_unregister_queues:
+	ibmveth_dispose_subordinate_irq_mappings(adapter);
+out_free_all_queues:
+	ibmveth_free_all_queues(adapter);
+out_free_buffer_pools:
 	ibmveth_free_buffer_pools(adapter);
 out_free_queue_mem:
 	ibmveth_cleanup_rx_resources(adapter);
@@ -1402,7 +1493,6 @@ static int ibmveth_open(struct net_device *netdev)
 static int ibmveth_close(struct net_device *netdev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
-	long lpar_rc;
 
 	/* Gate on opened, not IFF_UP: pool_store/change_mtu close+open can
 	 * leave IFF_UP set after a failed reopen.
@@ -1424,15 +1514,7 @@ static int ibmveth_close(struct net_device *netdev)
 	synchronize_net();
 
 	ibmveth_update_rx_no_buffer(adapter);
-	/* Full LAN teardown (subordinates arrive with register helpers). */
-	do {
-		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
-	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-	if (lpar_rc != H_SUCCESS) {
-		netdev_err(adapter->netdev,
-			   "h_free_logical_lan failed with %lx, continuing\n",
-			   lpar_rc);
-	}
+	ibmveth_free_all_queues(adapter);
 	/* Free TX LTBs after quiesce and after H_FREE_LOGICAL_LAN so xmit
 	 * cannot touch unmapped bounce buffers while the LAN is live.
 	 */
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 08/15] ibmveth: Add queue-aware RX buffer submit helper for MQ

[Mingming Cao]

Replenish is the last open-path hypercall that still assumes queue 0.
Registration is already queue-aware, and multi-queue posts buffers
through H_ADD_LOGICAL_LAN_BUFFERS_QUEUE against adapter->queue_handle[],
but ibmveth_replenish_buffer_pool() still calls
h_add_logical_lan_buffer() or h_add_logical_lan_buffers() directly.

Add ibmveth_add_logical_lan_buffers() to route RX buffer pool
replenishment through H_ADD_LOGICAL_LAN_BUFFERS_QUEUE in multi-queue
mode, falling back to the legacy 8-buffer and single-buffer hypercalls
in single-queue mode.

Supporting that helper means the RX side can no longer assume queue 0.
The queue index is 0 everywhere until multi-queue RX is enabled, so
the queue plumbing below does not change behaviour yet. The correlator
and locking changes do take effect immediately, on the single queue.

Parameterise the RX accessors by queue. The eight ibmveth_rxq_*
helpers took only the adapter and hardcoded rx_queue[0]; they now take
a queue_index, as do ibmveth_remove_buffer_from_pool() and
ibmveth_rxq_get_buffer().

Add a per-queue replenish_lock to struct ibmveth_rx_q, initialised in
probe. Replenish is not the only writer of a pool's free_map: the
harvest and remove consumer runs from that queue's NAPI instance, and
ndo_poll_controller() runs replenish on the same queue from outside
NAPI, so producer and consumer can run at once on one queue.

Give ibmveth_replenish_buffer_pool() a return value. It was void and
logged from inside the critical section, where a printk can re-enter
replenish through netconsole on the same device. It now returns one of
IBMVETH_REPLENISH_OK, _RESET_MAP, _RESET_MQ, _HCALL_FAIL or
_BATCH_FALLBACK, and ibmveth_replenish_task() does the logging and any
schedule_work() after dropping replenish_lock. The replenish map uses
DMA_ATTR_NO_WARN so the iommu path cannot printk under that lock.

Replace the correlator WARN_ON()s with ibmveth_rxq_correlator_valid().
The pool and buffer index come from a hypervisor-supplied correlator,
which is not a kernel invariant, so WARN_ON() was the wrong tool: with
panic_on_warn set, a malformed correlator would take the partition
down. The helper returns false instead, callers propagate -EINVAL or
-EFAULT, and ibmveth_rxq_advance() still advances the ring so poll
makes progress. The KUnit expectations are updated to match.

Redefine IBMVETH_MAX_RX_PER_HCALL from 8 to 12, the argument-list
capacity of H_ADD_LOGICAL_LAN_BUFFERS_QUEUE, and add
IBMVETH_MAX_RX_REGULAR (8) for the legacy hypercall.

free_buffer_pool() keeps probe/sysfs geometry (active/size/threshold)
and zeroes pool->available so teardown cannot leave a stale count.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- drop inline on the queue-index rxq_* accessors (plain static)
- initialise the replenish locks once in probe; open() no longer
  reinitialises them in alloc_rx_queues(), where it could reset a
  lock another CPU was holding through poll_controller()
- pass DMA_ATTR_NO_WARN to dma_map_single_attrs() in the replenish
  loop. The iommu warning could re-enter via netconsole and deadlock
  on the replenish_lock this CPU already holds
- static_assert IBMVETH_MAX_RX_REGULAR == 8 and
  IBMVETH_MAX_RX_PER_HCALL == 12 against the hand-written
  legacy descs[0..7] and MQ ioba[0..5] argument lists
- add clarifying comment on the legacy 8-buffer hcall batch bound
- drop the three buffer-submit hypercall counters along with the rest
  of hcall_stats; see patch 7
- fix the KUnit fixtures to allocate a dummy free_map
- noted: irqsave CS and free_buffer_pool vs replenish stay cover
- noted: skip_bad_correlator lands in P09

Changes in v5:
- On MQ buffer-add H_FUNCTION: schedule adapter reset after dropping
  replenish_lock (v4 logged/broke with no recovery; can permanently dry
  the pool)
- Move replenish fail logging / reset scheduling out from under
  replenish_lock so netconsole cannot deadlock re-entering replenish
- Serialize harvest/remove with per-queue replenish_lock (netpoll
  replenish vs NAPI consumer; v4 locked producer only)
- Fail logs use real wrapper names: h_add_logical_lan_buffers[_queue] /
  h_add_logical_lan_buffer (v4 interpolated broken lan[_queue] strings)
- Document replenish_lock + no-printk-under-lock for netconsole (first
  lock use); outcomes enum + ibmveth_replenish_fail defined here
- Defer adapter-global counter atomics and irqsave critical-section
  shorten to cover follow-up
- Bad queue_index poll path: napi_complete before return lands with
  poll harden (not claimed fully here)
- Keep pool active/size/threshold across free_buffer_pool (probe/sysfs
  geometry); only clear runtime allocations + available (ifdown/up
  reopen must still see active pools)
- get_buffer: use correlator_valid (drop WARN_ON; keep schedule_work
  until poll skip owns reset)
- Introduce ibmveth_rxq_correlator_valid / ibmveth_rxq_advance at first
  remove/harvest use; init replenish_lock in remove_buffer KUnit
- On RESET_MAP/RESET_MQ, stop remaining pool walks (goto unlock)

Changes in v4:
- Introduce queue-aware replenish/poll helpers with their first callers
  in the same patch; do not leave a 2-arg replenish call ahead of the
  signature change.
- Restore the pre-MQ LPM H_FUNCTION break instead of continue; do not
  loop forever on a stale local batch size.
- Fold per-queue replenish_lock into this patch.
- Update kdoc for MQ parameters on remove_buffer_from_pool /
  rxq_harvest_buffer.

 drivers/net/ethernet/ibm/ibmveth.c | 553 +++++++++++++++++++++--------
 drivers/net/ethernet/ibm/ibmveth.h |   6 +-
 2 files changed, 415 insertions(+), 144 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 465330464f91..46f68f5b52e7 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -30,6 +30,7 @@
 #include <linux/ip.h>
 #include <linux/ipv6.h>
 #include <linux/slab.h>
+#include <linux/spinlock.h>
 #include <asm/hvcall.h>
 #include <linux/atomic.h>
 #include <asm/vio.h>
@@ -101,49 +102,58 @@ static struct ibmveth_stat ibmveth_stats[] = {
 };
 
 /* simple methods of getting data from the current rxq entry */
-static inline u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter)
+static u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,
+			     int queue_index)
 {
-	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
 
 	return be32_to_cpu(rxq->queue_addr[rxq->index].flags_off);
 }
 
-static inline int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_toggle(struct ibmveth_adapter *adapter,
+			      int queue_index)
 {
-	return (ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_TOGGLE) >>
-			IBMVETH_RXQ_TOGGLE_SHIFT;
+	return (ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_TOGGLE) >>
+		IBMVETH_RXQ_TOGGLE_SHIFT;
 }
 
-static inline int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_pending_buffer(struct ibmveth_adapter *adapter,
+				      int queue_index)
 {
-	return ibmveth_rxq_toggle(adapter) == adapter->rx_queue[0].toggle;
+	return ibmveth_rxq_toggle(adapter, queue_index) ==
+		adapter->rx_queue[queue_index].toggle;
 }
 
-static inline int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_buffer_valid(struct ibmveth_adapter *adapter,
+				    int queue_index)
 {
-	return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_VALID;
+	return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_VALID;
 }
 
-static inline int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_frame_offset(struct ibmveth_adapter *adapter,
+				    int queue_index)
 {
-	return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_OFF_MASK;
+	return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_OFF_MASK;
 }
 
-static inline int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_large_packet(struct ibmveth_adapter *adapter,
+				    int queue_index)
 {
-	return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_LRG_PKT;
+	return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_LRG_PKT;
 }
 
-static inline int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_frame_length(struct ibmveth_adapter *adapter,
+				    int queue_index)
 {
-	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
 
 	return be32_to_cpu(rxq->queue_addr[rxq->index].length);
 }
 
-static inline int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter)
+static int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter,
+				 int queue_index)
 {
-	return ibmveth_rxq_flags(adapter) & IBMVETH_RXQ_CSUM_GOOD;
+	return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_CSUM_GOOD;
 }
 
 static unsigned int ibmveth_real_max_tx_queues(void)
@@ -712,11 +722,100 @@ static inline void ibmveth_flush_buffer(void *addr, unsigned long length)
 		asm("dcbf %0,%1,1" :: "b" (addr), "r" (offset));
 }
 
-/* replenish the buffers for a pool.  note that we don't need to
- * skb_reserve these since they are used for incoming...
+/**
+ * ibmveth_add_logical_lan_buffers - Add receive buffers to hypervisor
+ * @adapter: ibmveth adapter structure
+ * @descs: array of buffer descriptors to add
+ * @filled: number of valid descriptors in the array
+ * @buff_size: size of each buffer (multi-queue mode only)
+ * @queue_index: RX queue index
+ *
+ * Return: hypervisor return code
+ */
+static long ibmveth_add_logical_lan_buffers(struct ibmveth_adapter *adapter,
+					    union ibmveth_buf_desc *descs,
+					    int filled,
+					    unsigned long buff_size,
+					    int queue_index)
+{
+	struct vio_dev *vdev = adapter->vdev;
+	unsigned long rc;
+
+	/*
+	 * The MQ hcall takes six ioba words (12 packed addresses). The
+	 * legacy hcall takes eight descriptors. The argument lists below
+	 * are written out by hand; keep the defines matched to those lists.
+	 */
+	static_assert(IBMVETH_MAX_RX_PER_HCALL == 12);
+	static_assert(IBMVETH_MAX_RX_REGULAR == 8);
+
+	if (adapter->multi_queue) {
+		unsigned long buffersznum = (buff_size << 32) | filled;
+		unsigned long ioba[IBMVETH_MAX_RX_PER_HCALL / 2] = {0};
+		unsigned long handle = adapter->queue_handle[queue_index];
+		int i;
+
+		/* Pack descriptor addresses into ioba pairs.
+		 * Each ioba holds two 32-bit addresses packed into 64 bits:
+		 * - Even descriptors (0,2,4...) go in high 32 bits
+		 * - Odd descriptors (1,3,5...) go in low 32 bits
+		 */
+		for (i = 0; i < filled && i < IBMVETH_MAX_RX_PER_HCALL; i++) {
+			int pair_idx = i / 2;
+			int is_high = (i % 2 == 0);
+
+			if (is_high)
+				ioba[pair_idx] = (unsigned long)
+					descs[i].fields.address << 32;
+			else
+				ioba[pair_idx] |= descs[i].fields.address;
+		}
+
+		rc = h_add_logical_lan_buffers_queue(vdev->unit_address,
+						     handle,
+						     buffersznum,
+						     ioba[0], ioba[1], ioba[2],
+						     ioba[3], ioba[4], ioba[5]);
+	} else if (filled == 1) {
+		rc = h_add_logical_lan_buffer(vdev->unit_address,
+					      descs[0].desc);
+	} else {
+		/* Legacy 8-desc hcall; probe/mq_fallback keep batch <=
+		 * IBMVETH_MAX_RX_REGULAR.
+		 */
+		rc = h_add_logical_lan_buffers(vdev->unit_address,
+					       descs[0].desc, descs[1].desc,
+					       descs[2].desc, descs[3].desc,
+					       descs[4].desc, descs[5].desc,
+					       descs[6].desc, descs[7].desc);
+	}
+
+	return rc;
+}
+
+/* Outcomes for ibmveth_replenish_buffer_pool(); logged after unlock. */
+enum {
+	IBMVETH_REPLENISH_OK = 0,
+	IBMVETH_REPLENISH_RESET_MAP,
+	IBMVETH_REPLENISH_RESET_MQ,
+	IBMVETH_REPLENISH_HCALL_FAIL,
+	IBMVETH_REPLENISH_BATCH_FALLBACK,
+};
+
+struct ibmveth_replenish_fail {
+	unsigned long lpar_rc;
+	u32 filled;
+	u32 batch;
+};
+
+/* Replenish the buffers for a pool.
+ * Caller must hold the per-queue replenish_lock. Do not printk here:
+ * netconsole on the same device can re-enter replenish_task.
  */
-static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
-					  struct ibmveth_buff_pool *pool)
+static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
+					 struct ibmveth_buff_pool *pool,
+					 int queue_index,
+					 struct ibmveth_replenish_fail *fail)
 {
 	union ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};
 	u32 remaining = pool->size - atomic_read(&pool->available);
@@ -728,6 +827,7 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 	dma_addr_t dma_addr;
 	struct device *dev;
 	u32 index;
+	int outcome = IBMVETH_REPLENISH_OK;
 
 	vdev = adapter->vdev;
 	dev = &vdev->dev;
@@ -742,12 +842,9 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 		/* Fill a batch of descriptors */
 		for (filled = 0; filled < min(remaining, batch); filled++) {
 			index = pool->free_map[free_index];
-			if (WARN_ON(index == IBM_VETH_INVALID_MAP)) {
+			if (index == IBM_VETH_INVALID_MAP) {
 				adapter->replenish_add_buff_failure++;
-				netdev_info(adapter->netdev,
-					    "Invalid map index %u, reset\n",
-					    index);
-				schedule_work(&adapter->work);
+				outcome = IBMVETH_REPLENISH_RESET_MAP;
 				break;
 			}
 
@@ -762,9 +859,15 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 					break;
 				}
 
-				dma_addr = dma_map_single(dev, skb->data,
-							  pool->buff_size,
-							  DMA_FROM_DEVICE);
+				/* NO_WARN: hold replenish_lock; iommu
+				 * printk can re-enter via netconsole.
+				 */
+				dma_addr =
+					dma_map_single_attrs(dev,
+							     skb->data,
+							     pool->buff_size,
+							     DMA_FROM_DEVICE,
+							     DMA_ATTR_NO_WARN);
 				if (dma_mapping_error(dev, dma_addr)) {
 					dev_kfree_skb_any(skb);
 					adapter->replenish_add_buff_failure++;
@@ -799,28 +902,21 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 				free_index = 0;
 		}
 
+		if (outcome != IBMVETH_REPLENISH_OK)
+			break;
+
 		if (!filled)
 			break;
 
-		/* single buffer case*/
-		if (filled == 1)
-			lpar_rc = h_add_logical_lan_buffer(vdev->unit_address,
-							   descs[0].desc);
-		else
-			/* Multi-buffer hcall */
-			lpar_rc = h_add_logical_lan_buffers(vdev->unit_address,
-							    descs[0].desc,
-							    descs[1].desc,
-							    descs[2].desc,
-							    descs[3].desc,
-							    descs[4].desc,
-							    descs[5].desc,
-							    descs[6].desc,
-							    descs[7].desc);
+		lpar_rc = ibmveth_add_logical_lan_buffers(adapter, descs,
+							  filled,
+							  pool->buff_size,
+							  queue_index);
+
 		if (lpar_rc != H_SUCCESS) {
-			dev_warn_ratelimited(dev,
-					     "RX h_add_logical_lan failed: filled=%u, rc=%lu, batch=%u\n",
-					     filled, lpar_rc, batch);
+			fail->lpar_rc = lpar_rc;
+			fail->filled = filled;
+			fail->batch = batch;
 			goto hcall_failure;
 		}
 
@@ -860,30 +956,35 @@ static void ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 		}
 		adapter->replenish_add_buff_failure += filled;
 
-		/*
-		 * If multi rx buffers hcall is no longer supported by FW
-		 * e.g. in the case of Live Partition Migration
-		 */
-		if (batch > 1 && lpar_rc == H_FUNCTION) {
-			/*
-			 * Instead of retry submit single buffer individually
-			 * here just set the max rx buffer per hcall to 1
-			 * buffers will be respleshed next time
-			 * when ibmveth_replenish_buffer_pool() is called again
-			 * with single-buffer case
-			 */
-			netdev_info(adapter->netdev,
-				    "RX Multi buffers not supported by FW, rc=%lu\n",
-				    lpar_rc);
-			adapter->rx_buffers_per_hcall = 1;
-			netdev_info(adapter->netdev,
-				    "Next rx replesh will fall back to single-buffer hcall\n");
+		if (lpar_rc == H_FUNCTION) {
+			if (adapter->multi_queue) {
+				/*
+				 * LPM / firmware may drop MQ buffer hcalls.
+				 * Schedule reset so we do not sit forever in
+				 * no-buffer with the link still up.
+				 */
+				outcome = IBMVETH_REPLENISH_RESET_MQ;
+			} else if (batch > 1) {
+				/*
+				 * Live Partition Migration may drop multi-
+				 * buffer support. Fall back to single-buffer
+				 * on the next replenish; do not continue with
+				 * a stale local batch size (infinite loop).
+				 */
+				adapter->rx_buffers_per_hcall = 1;
+				outcome = IBMVETH_REPLENISH_BATCH_FALLBACK;
+			} else {
+				outcome = IBMVETH_REPLENISH_HCALL_FAIL;
+			}
+		} else {
+			outcome = IBMVETH_REPLENISH_HCALL_FAIL;
 		}
 		break;
 	}
 
 	mb();
 	atomic_add(buffers_added, &(pool->available));
+	return outcome;
 }
 
 /*
@@ -899,21 +1000,85 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
 }
 
 /* replenish routine */
-static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)
+static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
+				   int queue_index)
 {
-	int i;
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+	struct ibmveth_replenish_fail fail = {};
+	unsigned long flags;
+	int i, rc;
+	int need_reset = 0;
+	int batch_fallback = 0;
+	int hcall_fail = 0;
+
+	if (queue_index >= adapter->num_rx_queues) {
+		netdev_dbg(adapter->netdev,
+			   "Skipping replenish for freed queue %d (num_queues=%u)\n",
+			   queue_index, adapter->num_rx_queues);
+		return;
+	}
 
 	adapter->replenish_task_cycles++;
 
-	for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
-		struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[0][i];
+	spin_lock_irqsave(&rxq->replenish_lock, flags);
 
-		if (pool->active &&
-		    (atomic_read(&pool->available) < pool->threshold))
-			ibmveth_replenish_buffer_pool(adapter, pool);
+	for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
+		struct ibmveth_buff_pool *pool =
+			&adapter->rx_buff_pool[queue_index][i];
+
+		if (pool->active && pool->free_map &&
+		    (atomic_read(&pool->available) < pool->threshold)) {
+			rc = ibmveth_replenish_buffer_pool(adapter, pool,
+							   queue_index, &fail);
+			switch (rc) {
+			case IBMVETH_REPLENISH_RESET_MAP:
+			case IBMVETH_REPLENISH_RESET_MQ:
+				need_reset = rc;
+				goto out_unlock;
+			case IBMVETH_REPLENISH_BATCH_FALLBACK:
+				batch_fallback = 1;
+				break;
+			case IBMVETH_REPLENISH_HCALL_FAIL:
+				hcall_fail = 1;
+				break;
+			default:
+				break;
+			}
+		}
 	}
 
+out_unlock:
 	ibmveth_update_rx_no_buffer(adapter);
+
+	spin_unlock_irqrestore(&rxq->replenish_lock, flags);
+
+	/* Log and schedule reset only after dropping replenish_lock. */
+	if (need_reset == IBMVETH_REPLENISH_RESET_MAP) {
+		netdev_info(adapter->netdev,
+			    "Invalid RX free_map entry on queue %d, reset\n",
+			    queue_index);
+		schedule_work(&adapter->work);
+	} else if (need_reset == IBMVETH_REPLENISH_RESET_MQ) {
+		dev_err_ratelimited(&adapter->netdev->dev,
+				    "MQ buffer add H_FUNCTION (q=%d, batch=%u), reset\n",
+				    queue_index, fail.batch);
+		schedule_work(&adapter->work);
+	}
+
+	if (batch_fallback)
+		dev_warn_ratelimited(&adapter->netdev->dev,
+				     "Legacy batch add H_FUNCTION (batch=%u), fallback\n",
+				     fail.batch);
+
+	if (hcall_fail)
+		dev_warn_ratelimited(&adapter->netdev->dev,
+				     "RX %s failed: filled=%u, rc=%lu, batch=%u\n",
+				     adapter->multi_queue ?
+				     "h_add_logical_lan_buffers_queue" :
+				     (fail.batch == 1 ?
+				      "h_add_logical_lan_buffer" :
+				      "h_add_logical_lan_buffers"),
+				     fail.filled, fail.lpar_rc, fail.batch);
 }
 
 /* empty and free ana buffer pool - also used to do cleanup in error paths */
@@ -948,6 +1113,12 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
 		kfree(pool->skbuff);
 		pool->skbuff = NULL;
 	}
+
+	/*
+	 * Keep probe/sysfs geometry (active, size, buff_size, threshold).
+	 * Only tear down runtime allocations; open reuses active pools.
+	 */
+	atomic_set(&pool->available, 0);
 }
 
 /**
@@ -1088,35 +1259,74 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
 		   adapter->num_rx_queues);
 }
 
+static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
+					 int queue_index, u64 correlator)
+{
+	unsigned int pool = correlator >> 32;
+	unsigned int index = correlator & 0xffffffffUL;
+	struct ibmveth_buff_pool *bpool;
+
+	if (pool >= IBMVETH_NUM_BUFF_POOLS)
+		return false;
+
+	bpool = &adapter->rx_buff_pool[queue_index][pool];
+
+	/* Require a live pool with allocated arrays before indexing.
+	 * Inactive pools still have size from init; free clears skbuff.
+	 */
+	if (!bpool->active || !bpool->skbuff || !bpool->free_map)
+		return false;
+
+	return index < bpool->size;
+}
+
+static void ibmveth_rxq_advance(struct ibmveth_rx_q *rxq)
+{
+	if (++rxq->index == rxq->num_slots) {
+		rxq->index = 0;
+		rxq->toggle = !rxq->toggle;
+	}
+}
+
 /**
  * ibmveth_remove_buffer_from_pool - remove a buffer from a pool
  * @adapter: adapter instance
  * @correlator: identifies pool and index
+ * @queue_index: RX queue index (0..num_rx_queues-1)
  * @reuse: whether to reuse buffer
  *
+ * Context: may run concurrently with netpoll replenish_task on the same
+ * queue; takes per-queue replenish_lock to serialize free_map /
+ * producer_index / available against the producer.
+ *
  * Return:
  * * %0       - success
  * * %-EINVAL - correlator maps to pool or index out of range
  * * %-EFAULT - pool and index map to null skb
  */
 static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
-					   u64 correlator, bool reuse)
+					   u64 correlator, int queue_index,
+					   bool reuse)
 {
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
 	unsigned int pool  = correlator >> 32;
 	unsigned int index = correlator & 0xffffffffUL;
 	unsigned int free_index;
 	struct sk_buff *skb;
+	unsigned long flags;
+	int rc = 0;
 
-	if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
-	    WARN_ON(index >= adapter->rx_buff_pool[0][pool].size)) {
-		schedule_work(&adapter->work);
-		return -EINVAL;
+	spin_lock_irqsave(&rxq->replenish_lock, flags);
+
+	if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
+		rc = -EINVAL;
+		goto out_unlock;
 	}
 
-	skb = adapter->rx_buff_pool[0][pool].skbuff[index];
-	if (WARN_ON(!skb)) {
-		schedule_work(&adapter->work);
-		return -EFAULT;
+	skb = adapter->rx_buff_pool[queue_index][pool].skbuff[index];
+	if (!skb) {
+		rc = -EFAULT;
+		goto out_unlock;
 	}
 
 	/* if we are going to reuse the buffer then keep the pointers around
@@ -1127,75 +1337,88 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
 		/* remove the skb pointer to mark free. actual freeing is done
 		 * by upper level networking after gro_receive
 		 */
-		adapter->rx_buff_pool[0][pool].skbuff[index] = NULL;
+		struct ibmveth_buff_pool *bpool =
+			&adapter->rx_buff_pool[queue_index][pool];
+
+		bpool->skbuff[index] = NULL;
 
 		dma_unmap_single(&adapter->vdev->dev,
-				 adapter->rx_buff_pool[0][pool].dma_addr[index],
-				 adapter->rx_buff_pool[0][pool].buff_size,
+				 bpool->dma_addr[index],
+				 bpool->buff_size,
 				 DMA_FROM_DEVICE);
 	}
 
-	free_index = adapter->rx_buff_pool[0][pool].producer_index;
-	adapter->rx_buff_pool[0][pool].producer_index++;
-	if (adapter->rx_buff_pool[0][pool].producer_index >=
-	    adapter->rx_buff_pool[0][pool].size)
-		adapter->rx_buff_pool[0][pool].producer_index = 0;
-	adapter->rx_buff_pool[0][pool].free_map[free_index] = index;
+	free_index = adapter->rx_buff_pool[queue_index][pool].producer_index;
+	adapter->rx_buff_pool[queue_index][pool].producer_index++;
+	if (adapter->rx_buff_pool[queue_index][pool].producer_index >=
+	    adapter->rx_buff_pool[queue_index][pool].size)
+		adapter->rx_buff_pool[queue_index][pool].producer_index = 0;
+	adapter->rx_buff_pool[queue_index][pool].free_map[free_index] = index;
 
 	mb();
 
-	atomic_dec(&adapter->rx_buff_pool[0][pool].available);
+	atomic_dec(&adapter->rx_buff_pool[queue_index][pool].available);
 
-	return 0;
+out_unlock:
+	spin_unlock_irqrestore(&rxq->replenish_lock, flags);
+	return rc;
 }
 
 /* get the current buffer on the rx queue */
-static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter)
+static struct sk_buff *
+ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
+		       int queue_index)
 {
-	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
 	u64 correlator = rxq->queue_addr[rxq->index].correlator;
 	unsigned int pool = correlator >> 32;
 	unsigned int index = correlator & 0xffffffffUL;
 
-	if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
-	    WARN_ON(index >= adapter->rx_buff_pool[0][pool].size)) {
+	if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
 		schedule_work(&adapter->work);
 		return NULL;
 	}
 
-	return adapter->rx_buff_pool[0][pool].skbuff[index];
+	return adapter->rx_buff_pool[queue_index][pool].skbuff[index];
 }
 
 /**
  * ibmveth_rxq_harvest_buffer - Harvest buffer from pool
  *
  * @adapter: pointer to adapter
+ * @queue_index: RX queue index to harvest from
  * @reuse:   whether to reuse buffer
  *
  * Context: called from ibmveth_poll
  *
+ * On a bad correlator (-EINVAL/-EFAULT) the ring is still advanced so poll
+ * cannot spin forever on one slot. The error is still returned: callers must
+ * not treat it as a successful take from the pool (especially reuse=false,
+ * which would hand the SKB to the stack while it remains pool-owned).
+ *
  * Return:
- * * %0    - success
- * * other - non-zero return from ibmveth_remove_buffer_from_pool
+ * * %0    - buffer removed from pool (or marked for reuse) and ring advanced
+ * * other - non-zero return from ibmveth_remove_buffer_from_pool; ring has
+ *           still been advanced for -EINVAL/-EFAULT
  */
 static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
-				      bool reuse)
+				      int queue_index, bool reuse)
 {
+	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
 	u64 cor;
 	int rc;
 
-	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
-
 	cor = rxq->queue_addr[rxq->index].correlator;
-	rc = ibmveth_remove_buffer_from_pool(adapter, cor, reuse);
-	if (unlikely(rc))
+	rc = ibmveth_remove_buffer_from_pool(adapter, cor, queue_index, reuse);
+	if (unlikely(rc)) {
+		/* Skip a corrupt slot without claiming pool ownership. */
+		if (rc == -EINVAL || rc == -EFAULT)
+			ibmveth_rxq_advance(rxq);
 		return rc;
-
-	if (++adapter->rx_queue[0].index == adapter->rx_queue[0].num_slots) {
-		adapter->rx_queue[0].index = 0;
-		adapter->rx_queue[0].toggle = !adapter->rx_queue[0].toggle;
 	}
 
+	ibmveth_rxq_advance(rxq);
+
 	return 0;
 }
 
@@ -2161,34 +2384,41 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
 
 static int ibmveth_poll(struct napi_struct *napi, int budget)
 {
-	struct ibmveth_adapter *adapter =
-			container_of(napi, struct ibmveth_adapter, napi[0]);
-	struct net_device *netdev = adapter->netdev;
+	struct net_device *netdev = napi->dev;
+	struct ibmveth_adapter *adapter = netdev_priv(netdev);
 	int frames_processed = 0;
-	int rc;
+	int queue_index, rc;
 	u16 mss = 0;
 
+	queue_index = napi - adapter->napi;
+
 restart_poll:
 	while (frames_processed < budget) {
-		if (!ibmveth_rxq_pending_buffer(adapter))
+		if (!ibmveth_rxq_pending_buffer(adapter, queue_index))
 			break;
 
 		smp_rmb();
-		if (!ibmveth_rxq_buffer_valid(adapter)) {
+		if (!ibmveth_rxq_buffer_valid(adapter, queue_index)) {
 			wmb(); /* suggested by larson1 */
 			adapter->rx_invalid_buffer++;
 			netdev_dbg(netdev, "recycling invalid buffer\n");
-			if (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))
+			rc = ibmveth_rxq_harvest_buffer(adapter,
+							queue_index, true);
+			if (unlikely(rc))
 				break;
 		} else {
 			struct sk_buff *skb, *new_skb;
-			int length = ibmveth_rxq_frame_length(adapter);
-			int offset = ibmveth_rxq_frame_offset(adapter);
-			int csum_good = ibmveth_rxq_csum_good(adapter);
-			int lrg_pkt = ibmveth_rxq_large_packet(adapter);
+			int length = ibmveth_rxq_frame_length(adapter,
+							      queue_index);
+			int offset = ibmveth_rxq_frame_offset(adapter,
+							      queue_index);
+			int csum_good = ibmveth_rxq_csum_good(adapter,
+							      queue_index);
+			int lrg_pkt = ibmveth_rxq_large_packet(adapter,
+							       queue_index);
 			__sum16 iph_check = 0;
 
-			skb = ibmveth_rxq_get_buffer(adapter);
+			skb = ibmveth_rxq_get_buffer(adapter, queue_index);
 			if (unlikely(!skb))
 				break;
 
@@ -2213,12 +2443,18 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 							length);
 				if (rx_flush)
 					ibmveth_flush_buffer(skb->data,
-						length + offset);
-				if (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))
+							     length + offset);
+				rc = ibmveth_rxq_harvest_buffer(adapter,
+								queue_index,
+								true);
+				if (unlikely(rc))
 					break;
 				skb = new_skb;
 			} else {
-				if (unlikely(ibmveth_rxq_harvest_buffer(adapter, false)))
+				rc = ibmveth_rxq_harvest_buffer(adapter,
+								queue_index,
+								false);
+				if (unlikely(rc))
 					break;
 				skb_reserve(skb, offset);
 			}
@@ -2254,7 +2490,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 		}
 	}
 
-	ibmveth_replenish_task(adapter);
+	ibmveth_replenish_task(adapter, queue_index);
 
 	if (frames_processed == budget)
 		goto out;
@@ -2265,14 +2501,18 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 	/* We think we are done - reenable interrupts,
 	 * then check once more to make sure we are done.
 	 */
-	rc = ibmveth_enable_irq(adapter, 0);
+	rc = ibmveth_enable_irq(adapter, queue_index);
 	if (rc) {
+		netdev_err(netdev,
+			   "Failed to enable IRQ for queue %d (rc=%d), scheduling reset\n",
+			   queue_index, rc);
 		schedule_work(&adapter->work);
 		goto out;
 	}
 
-	if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule(napi)) {
-		ibmveth_disable_irq(adapter, 0);
+	if (ibmveth_rxq_pending_buffer(adapter, queue_index) &&
+	    napi_schedule(napi)) {
+		ibmveth_disable_irq(adapter, queue_index);
 		goto restart_poll;
 	}
 
@@ -2402,7 +2642,7 @@ static void ibmveth_poll_controller(struct net_device *dev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(dev);
 
-	ibmveth_replenish_task(adapter);
+	ibmveth_replenish_task(adapter, 0);
 	ibmveth_schedule_rx_queue(adapter, 0);
 }
 #endif
@@ -2560,6 +2800,16 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 	adapter->vdev = dev;
 	adapter->netdev = netdev;
 	INIT_WORK(&adapter->work, ibmveth_reset);
+
+	/*
+	 * Initialise the replenish locks once. open() is re-entered on
+	 * MTU and offload changes without netpoll_poll_disable(), so a
+	 * lock set up there could be reinitialised while poll_controller()
+	 * holds it.
+	 */
+	for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+		spin_lock_init(&adapter->rx_queue[i].replenish_lock);
+
 	adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
 	ibmveth_init_link_settings(netdev);
 
@@ -2601,7 +2851,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 
 	if (ret == H_SUCCESS &&
 	    (ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
-		adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_PER_HCALL;
+		adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
 		netdev_dbg(netdev,
 			   "RX Multi-buffer hcall supported by FW, batch set to %u\n",
 			    adapter->rx_buffers_per_hcall);
@@ -2922,8 +3172,7 @@ static void ibmveth_reset_kunit(struct work_struct *w)
  * @test: pointer to kunit structure
  *
  * Tests the error returns from ibmveth_remove_buffer_from_pool.
- * ibmveth_remove_buffer_from_pool also calls WARN_ON, so dmesg should be
- * checked to see that these warnings happened.
+ * Bad correlators return -EINVAL/-EFAULT (no WARN_ON).
  *
  * Return: void
  */
@@ -2937,6 +3186,8 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
 
 	INIT_WORK(&adapter->work, ibmveth_reset_kunit);
 
+	spin_lock_init(&adapter->rx_queue[0].replenish_lock);
+
 	/* Set sane values for buffer pools */
 	for (int i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
@@ -2946,19 +3197,34 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
 	pool = &adapter->rx_buff_pool[0][0];
 	pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
 	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
+	pool->free_map = kunit_kcalloc(test, pool->size, sizeof(u16),
+				       GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->free_map);
 
 	correlator = ((u64)IBMVETH_NUM_BUFF_POOLS << 32) | 0;
-	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
-	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+	KUNIT_EXPECT_EQ(test, -EINVAL,
+			ibmveth_remove_buffer_from_pool(adapter,
+							correlator, 0, false));
+	KUNIT_EXPECT_EQ(test, -EINVAL,
+			ibmveth_remove_buffer_from_pool(adapter,
+							correlator, 0, true));
 
 	correlator = ((u64)0 << 32) | adapter->rx_buff_pool[0][0].size;
-	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
-	KUNIT_EXPECT_EQ(test, -EINVAL, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+	KUNIT_EXPECT_EQ(test, -EINVAL,
+			ibmveth_remove_buffer_from_pool(adapter,
+							correlator, 0, false));
+	KUNIT_EXPECT_EQ(test, -EINVAL,
+			ibmveth_remove_buffer_from_pool(adapter,
+							correlator, 0, true));
 
 	correlator = (u64)0 | 0;
 	pool->skbuff[0] = NULL;
-	KUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, false));
-	KUNIT_EXPECT_EQ(test, -EFAULT, ibmveth_remove_buffer_from_pool(adapter, correlator, true));
+	KUNIT_EXPECT_EQ(test, -EFAULT,
+			ibmveth_remove_buffer_from_pool(adapter,
+							correlator, 0, false));
+	KUNIT_EXPECT_EQ(test, -EFAULT,
+			ibmveth_remove_buffer_from_pool(adapter,
+							correlator, 0, true));
 
 	flush_work(&adapter->work);
 }
@@ -2967,9 +3233,7 @@ static void ibmveth_remove_buffer_from_pool_test(struct kunit *test)
  * ibmveth_rxq_get_buffer_test - unit test for ibmveth_rxq_get_buffer
  * @test: pointer to kunit structure
  *
- * Tests ibmveth_rxq_get_buffer. ibmveth_rxq_get_buffer also calls WARN_ON for
- * the NULL returns, so dmesg should be checked to see that these warnings
- * happened.
+ * Tests ibmveth_rxq_get_buffer invalid correlator returns NULL without WARN.
  *
  * Return: void
  */
@@ -3000,18 +3264,21 @@ static void ibmveth_rxq_get_buffer_test(struct kunit *test)
 	pool = &adapter->rx_buff_pool[0][0];
 	pool->skbuff = kunit_kcalloc(test, pool->size, sizeof(void *), GFP_KERNEL);
 	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->skbuff);
+	pool->free_map = kunit_kcalloc(test, pool->size, sizeof(u16),
+				       GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool->free_map);
 
 	adapter->rx_queue[0].queue_addr[0].correlator =
 		(u64)IBMVETH_NUM_BUFF_POOLS << 32 | 0;
-	KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));
+	KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter, 0));
 
 	adapter->rx_queue[0].queue_addr[0].correlator =
 		(u64)0 << 32 | adapter->rx_buff_pool[0][0].size;
-	KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter));
+	KUNIT_EXPECT_PTR_EQ(test, NULL, ibmveth_rxq_get_buffer(adapter, 0));
 
 	pool->skbuff[0] = skb;
 	adapter->rx_queue[0].queue_addr[0].correlator = (u64)0 << 32 | 0;
-	KUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter));
+	KUNIT_EXPECT_PTR_EQ(test, skb, ibmveth_rxq_get_buffer(adapter, 0));
 
 	flush_work(&adapter->work);
 }
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index c13240f0ea2e..88448e915e86 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -14,6 +14,8 @@
 #ifndef _IBMVETH_H
 #define _IBMVETH_H
 
+#include <linux/spinlock_types.h>
+
 /* constants for H_MULTICAST_CTRL */
 #define IbmVethMcastReceptionModifyBit     0x80000UL
 #define IbmVethMcastReceptionEnableBit     0x20000UL
@@ -262,7 +264,8 @@ static inline long h_illan_attributes(unsigned long unit_address,
 #define IBMVETH_DEFAULT_QUEUES 8U
 #define IBMVETH_MAX_RX_QUEUES 1U
 #define IBMVETH_DEFAULT_RX_QUEUES 1U
-#define IBMVETH_MAX_RX_PER_HCALL 8U
+#define IBMVETH_MAX_RX_REGULAR 8U
+#define IBMVETH_MAX_RX_PER_HCALL 12U
 
 static int pool_size[] = { 512, 1024 * 2, 1024 * 16, 1024 * 32, 1024 * 64 };
 static int pool_count[] = { 256, 512, 256, 256, 256 };
@@ -293,6 +296,7 @@ struct ibmveth_rx_q {
     dma_addr_t queue_dma;
     u32        queue_len;
     struct ibmveth_rx_q_entry *queue_addr;
+	spinlock_t	replenish_lock;	/* per-queue buffer replenish */
 };
 
 struct ibmveth_adapter {
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 09/15] ibmveth: Harden RX poll path with helpers

[Mingming Cao]

ibmveth_poll() must handle several distinct skip/fail outcomes on each
RX slot, not only the happy path:

  - interface close / napi_disable must not re-arm PHYP delivery
  - bad correlators and harvest errors must not look like successful GRO
  - oversize or wrap-around offset+length must not skb_put() past the
    buffer
  - an out-of-range queue index must napi_complete_done rather than
    fall through and keep polling

Doing all of that inline turns the NAPI callback into a deeply nested
switchyard. Split each outcome into a small helper so ibmveth_poll()
stays a thin budget loop:

  ibmveth_poll_stopping()
  ibmveth_poll_harvest_slot() / recycle_invalid / skip_bad_correlator
  ibmveth_poll_drop_oversize()
  ibmveth_poll_deliver_frame()
  ibmveth_poll_bump_invalid()

Not pure motion: ibmveth_poll_bump_invalid() also counts oversize
frames and skipped slots (intentional). skip_bad_correlator escalates
a valid correlator with a NULL skb (-EFAULT) to reset, not only an
out-of-range correlator. Deliver also rejects a PHYP offset+length
that would skb_put() past the buffer. Skipped and dropped slots do not
count against the NAPI budget; only a delivered frame does.

Two further behaviour changes come with the split. On the rx_copybreak
path a harvest failure, after the frame has already been copied into
new_skb, used to break out of the loop and leak that skb;
deliver_frame() kfree_skb()s it before returning an error. And mss
becomes a local of deliver_frame() rather than living across
ibmveth_poll()'s whole budget loop; it was never read stale, because
ibmveth_rx_mss_helper() only uses it under lrg_pkt, so that part is
scoping hygiene and not a fix.

Drop schedule_work from rxq_get_buffer() (skip_bad_correlator owns
reset escalation). ibmveth_poll_stopping() ensures close/napi_disable
does not re-arm PHYP. Runtime is still single-queue: helpers are defined
and called from the existing SQ poll path in this same patch (first
use). The next patch turns on MQ and reuses this loop; subordinate
register helpers stay there.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- read the IPv4 header check through skb->data, as mainline does.
  v5 used ip_hdr(), which reads the network header offset that
  eth_type_trans() does not set on this path
- do not complete NAPI or consume budget when budget is 0. That is
  the netpoll case, where v5 completed and returned budget - 1,
  underflowing to -1
- noted: the split is not pure motion; the body lists what changed

Changes in v5:
- Wrap poll_stopping return after complete so it stays under 80 cols
- On poll_stopping after the budget loop: return < budget after
  napi_complete_done (was frames_processed-1 even when under budget)
- New in v5: peel SQ poll harden + helpers before MQ enable so tip P10
  stays bring-up focused (mailed v4 09/14 MQ enable -> tip P10; also
  14->15). Kitchen-sink / enable-path poll fixes land here on the live
  single-queue path first.
- After napi_complete_done, check poll_stopping again before enable_irq
  (avoid re-arm while resize/close waits on napi_disable)

 drivers/net/ethernet/ibm/ibmveth.c | 289 ++++++++++++++++++++---------
 1 file changed, 199 insertions(+), 90 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 46f68f5b52e7..f726d1e3f0b1 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1374,10 +1374,8 @@ ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
 	unsigned int pool = correlator >> 32;
 	unsigned int index = correlator & 0xffffffffUL;
 
-	if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
-		schedule_work(&adapter->work);
+	if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator))
 		return NULL;
-	}
 
 	return adapter->rx_buff_pool[queue_index][pool].skbuff[index];
 }
@@ -2382,125 +2380,236 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
 	}
 }
 
+static void ibmveth_poll_bump_invalid(struct ibmveth_adapter *adapter,
+				      int queue_index)
+{
+	adapter->rx_invalid_buffer++;
+}
+
+static bool ibmveth_poll_stopping(struct net_device *netdev,
+				  struct napi_struct *napi)
+{
+	return !netif_running(netdev) || napi_disable_pending(napi);
+}
+
+static bool ibmveth_poll_harvest_slot(struct ibmveth_adapter *adapter,
+				      int queue_index, bool reuse)
+{
+	int rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, reuse);
+
+	return !rc || rc == -EINVAL || rc == -EFAULT;
+}
+
+static bool ibmveth_poll_recycle_invalid(struct net_device *netdev,
+					 struct ibmveth_adapter *adapter,
+					 int queue_index)
+{
+	netdev_dbg(netdev, "recycling invalid buffer\n");
+	ibmveth_poll_bump_invalid(adapter, queue_index);
+	return ibmveth_poll_harvest_slot(adapter, queue_index, true);
+}
+
+static bool ibmveth_poll_skip_bad_correlator(struct net_device *netdev,
+					     struct ibmveth_adapter *adapter,
+					     int queue_index)
+{
+	if (net_ratelimit())
+		netdev_err(netdev,
+			   "bad correlator on queue %d, skipping slot\n",
+			   queue_index);
+	/* Residual stale slot after resize: recover via reset rather
+	 * than spinning forever. Always escalate; only the log is
+	 * rate-limited.
+	 */
+	schedule_work(&adapter->work);
+	ibmveth_poll_bump_invalid(adapter, queue_index);
+	return ibmveth_poll_harvest_slot(adapter, queue_index, true);
+}
+
+static bool ibmveth_poll_drop_oversize(struct net_device *netdev,
+				       struct ibmveth_adapter *adapter,
+				     int queue_index, unsigned int off,
+				     unsigned int len, unsigned int room)
+{
+	if (net_ratelimit())
+		netdev_err(netdev,
+			   "RX frame %u+%u exceeds buffer %u on queue %d, dropping\n",
+			   off, len, room, queue_index);
+	ibmveth_poll_bump_invalid(adapter, queue_index);
+	return ibmveth_poll_harvest_slot(adapter, queue_index, true);
+}
+
+/**
+ * ibmveth_poll_deliver_frame - Build SKB from one valid RX slot and GRO it
+ * @napi: NAPI context for this RX queue
+ * @adapter: ibmveth adapter
+ * @netdev: net_device for @adapter
+ * @queue_index: RX queue index
+ *
+ * Return: 1 frame delivered, 0 if the slot was skipped cleanly, -1 on error.
+ */
+static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
+				      struct ibmveth_adapter *adapter,
+				      struct net_device *netdev,
+				      int queue_index)
+{
+	struct sk_buff *skb, *new_skb;
+	unsigned int room, off, len;
+	int length, offset, csum_good, lrg_pkt;
+	__sum16 iph_check = 0;
+	u16 mss = 0;
+	int rc;
+
+	length = ibmveth_rxq_frame_length(adapter, queue_index);
+	offset = ibmveth_rxq_frame_offset(adapter, queue_index);
+	csum_good = ibmveth_rxq_csum_good(adapter, queue_index);
+	lrg_pkt = ibmveth_rxq_large_packet(adapter, queue_index);
+
+	skb = ibmveth_rxq_get_buffer(adapter, queue_index);
+	if (unlikely(!skb)) {
+		if (!ibmveth_poll_skip_bad_correlator(netdev, adapter,
+						      queue_index))
+			return -1;
+		return 0;
+	}
+
+	room = skb_tailroom(skb);
+	off = offset;
+	len = length;
+	if (unlikely(off >= room || len > room - off)) {
+		if (!ibmveth_poll_drop_oversize(netdev, adapter, queue_index,
+						off, len, room))
+			return -1;
+		return 0;
+	}
+
+	if (lrg_pkt) {
+		__be64 *rxmss = (__be64 *)(skb->data + 8);
+
+		mss = (u16)be64_to_cpu(*rxmss);
+	}
+
+	new_skb = NULL;
+	if (length < rx_copybreak)
+		new_skb = netdev_alloc_skb(netdev, length);
+
+	if (new_skb) {
+		skb_copy_to_linear_data(new_skb, skb->data + offset, length);
+		if (rx_flush)
+			ibmveth_flush_buffer(skb->data, length + offset);
+		rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);
+		if (unlikely(rc)) {
+			kfree_skb(new_skb);
+			return -1;
+		}
+		skb = new_skb;
+	} else {
+		rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, false);
+		if (unlikely(rc))
+			return -1;
+		skb_reserve(skb, offset);
+	}
+
+	skb_put(skb, length);
+	skb->protocol = eth_type_trans(skb, netdev);
+
+	if (skb->protocol == cpu_to_be16(ETH_P_IP))
+		iph_check = ((struct iphdr *)skb->data)->check;
+
+	if ((length > netdev->mtu + ETH_HLEN) || lrg_pkt ||
+	    iph_check == 0xffff) {
+		ibmveth_rx_mss_helper(skb, mss, lrg_pkt);
+		adapter->rx_large_packets++;
+	}
+
+	if (csum_good) {
+		skb->ip_summed = CHECKSUM_UNNECESSARY;
+		ibmveth_rx_csum_helper(skb, adapter);
+	}
+
+	napi_gro_receive(napi, skb);
+
+	netdev->stats.rx_packets++;
+	netdev->stats.rx_bytes += length;
+
+	return 1;
+}
+
 static int ibmveth_poll(struct napi_struct *napi, int budget)
 {
 	struct net_device *netdev = napi->dev;
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
 	int frames_processed = 0;
 	int queue_index, rc;
-	u16 mss = 0;
 
 	queue_index = napi - adapter->napi;
 
+	if (WARN_ON(queue_index < 0 ||
+		    queue_index >= adapter->num_rx_queues)) {
+		if (budget)
+			napi_complete_done(napi, 0);
+		return 0;
+	}
+
+	if (ibmveth_poll_stopping(netdev, napi)) {
+		if (budget)
+			napi_complete_done(napi, 0);
+		return 0;
+	}
+
 restart_poll:
 	while (frames_processed < budget) {
+		if (ibmveth_poll_stopping(netdev, napi))
+			break;
+
 		if (!ibmveth_rxq_pending_buffer(adapter, queue_index))
 			break;
 
 		smp_rmb();
 		if (!ibmveth_rxq_buffer_valid(adapter, queue_index)) {
 			wmb(); /* suggested by larson1 */
-			adapter->rx_invalid_buffer++;
-			netdev_dbg(netdev, "recycling invalid buffer\n");
-			rc = ibmveth_rxq_harvest_buffer(adapter,
-							queue_index, true);
-			if (unlikely(rc))
+			if (!ibmveth_poll_recycle_invalid(netdev, adapter,
+							  queue_index))
 				break;
 		} else {
-			struct sk_buff *skb, *new_skb;
-			int length = ibmveth_rxq_frame_length(adapter,
-							      queue_index);
-			int offset = ibmveth_rxq_frame_offset(adapter,
-							      queue_index);
-			int csum_good = ibmveth_rxq_csum_good(adapter,
-							      queue_index);
-			int lrg_pkt = ibmveth_rxq_large_packet(adapter,
-							       queue_index);
-			__sum16 iph_check = 0;
-
-			skb = ibmveth_rxq_get_buffer(adapter, queue_index);
-			if (unlikely(!skb))
+			rc = ibmveth_poll_deliver_frame(napi, adapter, netdev,
+							queue_index);
+			if (rc < 0)
 				break;
-
-			/* if the large packet bit is set in the rx queue
-			 * descriptor, the mss will be written by PHYP eight
-			 * bytes from the start of the rx buffer, which is
-			 * skb->data at this stage
-			 */
-			if (lrg_pkt) {
-				__be64 *rxmss = (__be64 *)(skb->data + 8);
-
-				mss = (u16)be64_to_cpu(*rxmss);
-			}
-
-			new_skb = NULL;
-			if (length < rx_copybreak)
-				new_skb = netdev_alloc_skb(netdev, length);
-
-			if (new_skb) {
-				skb_copy_to_linear_data(new_skb,
-							skb->data + offset,
-							length);
-				if (rx_flush)
-					ibmveth_flush_buffer(skb->data,
-							     length + offset);
-				rc = ibmveth_rxq_harvest_buffer(adapter,
-								queue_index,
-								true);
-				if (unlikely(rc))
-					break;
-				skb = new_skb;
-			} else {
-				rc = ibmveth_rxq_harvest_buffer(adapter,
-								queue_index,
-								false);
-				if (unlikely(rc))
-					break;
-				skb_reserve(skb, offset);
-			}
-
-			skb_put(skb, length);
-			skb->protocol = eth_type_trans(skb, netdev);
-
-			/* PHYP without PLSO support places a -1 in the ip
-			 * checksum for large send frames.
-			 */
-			if (skb->protocol == cpu_to_be16(ETH_P_IP)) {
-				struct iphdr *iph = (struct iphdr *)skb->data;
-
-				iph_check = iph->check;
-			}
-
-			if ((length > netdev->mtu + ETH_HLEN) ||
-			    lrg_pkt || iph_check == 0xffff) {
-				ibmveth_rx_mss_helper(skb, mss, lrg_pkt);
-				adapter->rx_large_packets++;
-			}
-
-			if (csum_good) {
-				skb->ip_summed = CHECKSUM_UNNECESSARY;
-				ibmveth_rx_csum_helper(skb, adapter);
-			}
-
-			napi_gro_receive(napi, skb);	/* send it up */
-
-			netdev->stats.rx_packets++;
-			netdev->stats.rx_bytes += length;
-			frames_processed++;
+			if (rc > 0)
+				frames_processed++;
 		}
 	}
 
 	ibmveth_replenish_task(adapter, queue_index);
 
+	if (ibmveth_poll_stopping(netdev, napi)) {
+		/* budget 0 is netpoll, which must not complete NAPI.
+		 * Otherwise returning budget after completing would ask
+		 * NAPI to reschedule, so cap the return at budget - 1.
+		 */
+		if (budget) {
+			napi_complete_done(napi, frames_processed);
+			return min(frames_processed, budget - 1);
+		}
+		return 0;
+	}
+
 	if (frames_processed == budget)
 		goto out;
 
 	if (!napi_complete_done(napi, frames_processed))
 		goto out;
 
-	/* We think we are done - reenable interrupts,
-	 * then check once more to make sure we are done.
+	/*
+	 * napi_disable() sets DISABLE then waits for this poll. Without a
+	 * second stopping check here, enable_irq() can re-arm PHYP after
+	 * resize already masked the queue; late IRQs then hit the handler
+	 * after num_rx_queues was published lower (lab WARN at interrupt).
 	 */
+	if (ibmveth_poll_stopping(netdev, napi))
+		goto out;
+
 	rc = ibmveth_enable_irq(adapter, queue_index);
 	if (rc) {
 		netdev_err(netdev,
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 10/15] ibmveth: Enable multi-queue RX receive path

[Mingming Cao]

Turn on the live MQ RX path when firmware sets
IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT in H_ILLAN_ATTRIBUTES. Probe sets
multi_queue and num_rx_queues to min(num_online_cpus(),
IBMVETH_DEFAULT_QUEUES), matching the existing TX default (cap 8).
IBMVETH_MAX_RX_QUEUES goes from 1 to 16, which is what actually gives
the adapter more than one RX ring: every rx_queue[], queue_handle[]
and queue_irq[] array sized by it grows to sixteen entries.
multi_queue also becomes a bool, now that it is a flag and not a count.
MQ firmware also raises the replenish batch from eight buffers per
hypercall to twelve, the argument-list capacity of
H_ADD_LOGICAL_LAN_BUFFERS_QUEUE, by setting rx_buffers_per_hcall to
IBMVETH_MAX_RX_PER_HCALL instead of IBMVETH_MAX_RX_REGULAR.
Patch 14 wires live ethtool -L rx (set_channels reads rx_count and
calls resize_rx_channels). Patch 15 completes the down-path
publish/rollback and caps max_rx at the live RX count once mq_fallback
latches.
This commit does not implement set_channels / rx_count. Without the bit,
behaviour stays single-queue.

Wire subordinate queues through H_REG_LOGICAL_LAN_QUEUE (with
irq_create_mapping for subordinate virqs), request_irq/napi_enable,
and PHYP enable_irq when multi_queue && num_rx_queues > 1. Queue 0
continues to use netdev->irq and is never disposed with the
subordinates.

Restate open() here (supersedes the register-helpers patch kick):

  ... register_rx_queues / set_real_num_rx_queues ...
  replenish_task() for every live RX queue   (before IRQ setup)
  setup_rx_interrupts()                      (MQ also unmasks PHYP)
  restart_rx_queue() for every live queue    (schedule NAPI, or
                                             enable_irq if prep fails)
  ... alloc_tx_resources / tx_start / opened ...

That path is the same for SQ and MQ; it replaces the prior
setup-then-schedule_rx_queue(0) kick. Close shape is unchanged.

H_FUNCTION means firmware withdrew MQ. Both the subordinate-register
and the buffer-add path latch mq_fallback, so the next open comes up
single-queue (apply_mq_fallback at open entry); they differ in what
happens to the open in progress. Register fails it outright rather
than dropping to single-queue silently mid-open, and buffer-add
schedules a reset. Reset does not retry if that close/open fails.
Both setup failure paths dispose subordinate virq mappings, so a
request_irq failure after successful registration cannot leak Linux
mappings.

On probe failure after pool kobjects were created, put them before
free_netdev(). The leak is pre-existing and unrelated to multi-queue,
so a standalone version also goes to net; the probe_cleanup helper
lands in patch 11.

Refresh VIO CMO entitlement after MQ probe / pool init, sizing
get_desired_dma() from queue-0 pool metadata across num_rx_queues. TX
entitlement is still one LTB (pre-existing); per-TX queue CMO scaling
is on the cover leftovers list.

Two things are knowingly left for patch 11. RX counters are racy:
adapter->rx_no_buffer is assigned rather than summed from one queue's
buffer-list page, so it reports whichever queue replenished last and
can go backwards, while rx_packets, rx_bytes, rx_invalid_buffer and
rx_large_packets are plain read-modify-writes now reached from several
NAPI instances at once, so they can lose counts; patch 11 moves both
to per-queue storage summed on read. This commit also adds
schedule_work() producers, and the remove-path unregister /
cancel_work_sync reorder and reset reg_state gate that make them safe
land there too.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- ibmveth_get_num_rx_queues / ibmveth_publish_num_rx_queues are
  static, not static inline
- wrap the rx_buffers_per_hcall assignment (81 cols)
- smp_store_release / smp_load_acquire for num_rx_queues, not
  smp_wmb() + WRITE_ONCE()
- skb_record_rx_queue() before GRO
- drop the IBMVETH_MAX_RX_QUEUE alias (duplicated MAX_RX_PER_HCALL)
- drop WARN_ON on restart enable_irq; the helper already logs.
  Enabling on prep failure is deliberate
- drop unused kick_rx_queue_if_pending() (WERROR); patch 14 uses
  restart_rx_queue() at both sites
- register_rx_queues() kdoc: this function latches mq_fallback on
  -EOPNOTSUPP
- noted: pool kobj vs DEBUG_KOBJECT_RELEASE stays cover leftovers

Changes in v5:
- RTNL/serialized readers use get_num_rx_queues() once the helper exists
  (open/close/register/apply_mq_fallback/alloc/cleanup/IRQ setup)
- Interrupt: quiet IRQ_NONE on out-of-range qindex vs published live
  count (scale-down residual IRQ; no WARN storm)
- Series renumber: mailed v4 09/14 MQ enable -> tip P10 (P09 peel; 14->15)
- Open: replenish all queues before setup; restart_rx_queue after setup
  for SQ and MQ - replaces mailed MQ-only replenish + kick_if_pending vs
  SQ schedule_rx_queue(0)
- update_rx_no_buffer(queue_index): NULL-safe, no all-queue walk under one
  lock (fixes the resize/race class). Still stores adapter->rx_no_buffer
  for now; per-queue slot + monotonic sum lands with qstats next
- H_FUNCTION: subordinate register fails open + stash mq_fallback;
  buffer-add schedules reset + mq_fallback so next open drops to SQ
- Introduce get_num_rx_queues / publish_num_rx_queues (READ/WRITE_ONCE)
  at first use; lockless IRQ/poll/replenish/interrupt bounds use the getter
- Introduce kick_rx_queue_if_pending() for pending-after-unmask; open uses
  restart_rx_queue, resize calls the helper later
- resume() schedules every live RX queue (not only queue 0)
- Call out enable_irq-fail synchronize_irq and unused mac register-arg
  cleanup (mailed enable-path review; code already earlier in tip)

Changes in v4:
- Fold subordinate register helpers and their review fixes into the MQ
  enablement patch that first uses them.
- Prefer request_irq -> napi_enable -> PHYP enable on MQ open.
- Preserve open unwind so set_real_num_rx / IRQ failures free LAN
  before buffer pools.
- MQ open replenishes every queue before setup_rx_interrupts() unmasks
  PHYP (drop avoidance during open; PHYP only interrupts after a
  successful enqueue). SQ keeps classic setup-then-schedule kick.
- Dispose subordinate virq mappings on setup_rx_interrupts()
  request_irq failure (err_free_irqs), matching err_disable_napi.
- Open unwind: setup/cleanup own subordinate dispose; skip duplicate
  dispose on those paths.
- H_FUNCTION on subordinate register is a hard open failure (no blind
  retry / no fake single-queue fallback).
- Note: hot-path netdev->stats accounting moves to the next patch (qstats).
- Put already-created pool kobjects on probe kobject_init_and_add /
  set_real_num_tx_queues / register_netdev failure (bisect-safe).

 drivers/net/ethernet/ibm/ibmveth.c | 493 +++++++++++++++++++++++++----
 drivers/net/ethernet/ibm/ibmveth.h |   6 +-
 2 files changed, 428 insertions(+), 71 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index f726d1e3f0b1..2e8896ea5af2 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -156,6 +156,28 @@ static int ibmveth_rxq_csum_good(struct ibmveth_adapter *adapter,
 	return ibmveth_rxq_flags(adapter, queue_index) & IBMVETH_RXQ_CSUM_GOOD;
 }
 
+/* Lockless IRQ/poll readers vs resize publishers. */
+static unsigned int
+ibmveth_get_num_rx_queues(const struct ibmveth_adapter *adapter)
+{
+	/*
+	 * Pairs with the release in ibmveth_publish_num_rx_queues(): a reader
+	 * that sees the new count also sees the per-queue state behind it.
+	 */
+	return smp_load_acquire(&adapter->num_rx_queues);
+}
+
+static void
+ibmveth_publish_num_rx_queues(struct ibmveth_adapter *adapter,
+			      unsigned int num)
+{
+	/*
+	 * Pairs with the acquire in ibmveth_get_num_rx_queues(): per-queue
+	 * state must be visible to a reader before it observes the new count.
+	 */
+	smp_store_release(&adapter->num_rx_queues, num);
+}
+
 static unsigned int ibmveth_real_max_tx_queues(void)
 {
 	unsigned int n_cpu = num_online_cpus();
@@ -233,7 +255,7 @@ ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
 	struct net_device *netdev = adapter->netdev;
 	int i;
 
-	for (i = 0; i < adapter->num_rx_queues; i++) {
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
 		adapter->buffer_list_addr[i] =
 			(void *)get_zeroed_page(GFP_KERNEL);
 		if (!adapter->buffer_list_addr[i]) {
@@ -284,7 +306,7 @@ ibmveth_alloc_rx_queues(struct ibmveth_adapter *adapter, int rxq_entries)
 	}
 
 	netdev_dbg(netdev, "allocated %u RX queue(s) with %d entries each\n",
-		   adapter->num_rx_queues, rxq_entries);
+		   ibmveth_get_num_rx_queues(adapter), rxq_entries);
 
 	return 0;
 
@@ -327,9 +349,9 @@ ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
 	int i;
 
 	netdev_dbg(adapter->netdev, "cleaning up %u RX queue(s)\n",
-		   adapter->num_rx_queues);
+		   ibmveth_get_num_rx_queues(adapter));
 
-	for (i = 0; i < adapter->num_rx_queues; i++) {
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
 		if (adapter->buffer_list_addr[i]) {
 			dma_unmap_single(dev, adapter->buffer_list_dma[i],
 					 4096, DMA_BIDIRECTIONAL);
@@ -486,7 +508,7 @@ ibmveth_dispose_subordinate_irq_mappings(struct ibmveth_adapter *adapter)
 {
 	int i;
 
-	for (i = 1; i < adapter->num_rx_queues; i++)
+	for (i = 1; i < ibmveth_get_num_rx_queues(adapter); i++)
 		ibmveth_dispose_subordinate_irq_mapping(adapter, i);
 }
 
@@ -509,7 +531,7 @@ static int
 ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
 {
 	struct net_device *netdev = adapter->netdev;
-	int i, rc, num = adapter->num_rx_queues;
+	int i, rc, num = ibmveth_get_num_rx_queues(adapter);
 
 	for (i = 0; i < num; i++) {
 		if (!adapter->queue_irq[i]) {
@@ -599,24 +621,24 @@ ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
 	if (!adapter->rx_irq_setup)
 		return;
 
-	for (i = 0; i < adapter->num_rx_queues; i++) {
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
 		if (!adapter->queue_irq[i])
 			continue;
 		ibmveth_disable_irq(adapter, i);
 		synchronize_irq(adapter->queue_irq[i]);
 	}
 
-	for (i = 0; i < adapter->num_rx_queues; i++)
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
 		napi_disable(&adapter->napi[i]);
 
-	for (i = 0; i < adapter->num_rx_queues; i++) {
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
 		if (!adapter->queue_irq[i])
 			continue;
 		ibmveth_disable_irq(adapter, i);
 		synchronize_irq(adapter->queue_irq[i]);
 	}
 
-	for (i = 0; i < adapter->num_rx_queues; i++) {
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
 		if (adapter->queue_irq[i])
 			free_irq(adapter->queue_irq[i], &adapter->napi[i]);
 	}
@@ -646,7 +668,7 @@ static bool ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
 {
 	struct napi_struct *napi = &adapter->napi[qindex];
 
-	if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
+	if (WARN_ON(qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter)))
 		return false;
 
 	/*
@@ -992,11 +1014,21 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
  * because there was not a buffer in the buffer list capable of holding
  * the frame.
  */
-static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
+static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
+					int queue_index)
 {
-	__be64 *p = adapter->buffer_list_addr[0] + 4096 - 8;
+	__be64 *p;
+	u64 drops;
 
-	adapter->rx_no_buffer = be64_to_cpup(p);
+	if (queue_index < 0 ||
+	    queue_index >= ibmveth_get_num_rx_queues(adapter) ||
+	    !adapter->buffer_list_addr[queue_index])
+		return;
+
+	p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
+	drops = be64_to_cpup(p);
+
+	adapter->rx_no_buffer = drops;
 }
 
 /* replenish routine */
@@ -1011,10 +1043,10 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
 	int batch_fallback = 0;
 	int hcall_fail = 0;
 
-	if (queue_index >= adapter->num_rx_queues) {
+	if (queue_index >= ibmveth_get_num_rx_queues(adapter)) {
 		netdev_dbg(adapter->netdev,
 			   "Skipping replenish for freed queue %d (num_queues=%u)\n",
-			   queue_index, adapter->num_rx_queues);
+			   queue_index, ibmveth_get_num_rx_queues(adapter));
 		return;
 	}
 
@@ -1048,7 +1080,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
 	}
 
 out_unlock:
-	ibmveth_update_rx_no_buffer(adapter);
+	ibmveth_update_rx_no_buffer(adapter, queue_index);
 
 	spin_unlock_irqrestore(&rxq->replenish_lock, flags);
 
@@ -1062,6 +1094,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
 		dev_err_ratelimited(&adapter->netdev->dev,
 				    "MQ buffer add H_FUNCTION (q=%d, batch=%u), reset\n",
 				    queue_index, fail.batch);
+		adapter->mq_fallback = true;
 		schedule_work(&adapter->work);
 	}
 
@@ -1081,6 +1114,27 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
 				     fail.filled, fail.lpar_rc, fail.batch);
 }
 
+/**
+ * ibmveth_restart_rx_queue - Post buffers and ensure Q can take RX
+ * @adapter: ibmveth adapter
+ * @qindex: RX queue index
+ *
+ * SQ open leaves PHYP masked until the first poll. If schedule_prep fails,
+ * NAPI never runs and the queue stays masked (TX OK, RX/ARP dead) until
+ * reload. Replenish first so an enable_irq fallback can actually deliver.
+ * Also used after every open (SQ and MQ) and after scale-down so a
+ * queue is not left idle+masked.
+ */
+static void ibmveth_restart_rx_queue(struct ibmveth_adapter *adapter,
+				     int qindex)
+{
+	ibmveth_replenish_task(adapter, qindex);
+	if (ibmveth_schedule_rx_queue(adapter, qindex))
+		return;
+
+	ibmveth_enable_irq(adapter, qindex);
+}
+
 /* empty and free ana buffer pool - also used to do cleanup in error paths */
 static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
 				     struct ibmveth_buff_pool *pool)
@@ -1209,7 +1263,7 @@ ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
 	int i, q, rc;
 
 	/* Initialize pool metadata for queues 1..N from queue 0 settings */
-	for (q = 1; q < adapter->num_rx_queues; q++) {
+	for (q = 1; q < ibmveth_get_num_rx_queues(adapter); q++) {
 		for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
 			struct ibmveth_buff_pool *src =
 				&adapter->rx_buff_pool[0][i];
@@ -1225,7 +1279,7 @@ ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
 	}
 
 	/* Allocate actual buffers for all queues */
-	for (q = 0; q < adapter->num_rx_queues; q++) {
+	for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
 		rc = ibmveth_alloc_queue_buffer_pools(adapter, q);
 		if (rc) {
 			/* Free pools for all previous queues */
@@ -1236,7 +1290,7 @@ ibmveth_alloc_buffer_pools(struct ibmveth_adapter *adapter)
 	}
 
 	netdev_dbg(netdev, "allocated buffer pools for %u queue(s)\n",
-		   adapter->num_rx_queues);
+		   ibmveth_get_num_rx_queues(adapter));
 	return 0;
 }
 
@@ -1252,11 +1306,11 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
 	int q;
 
 	/* Free buffer pools for all queues */
-	for (q = 0; q < adapter->num_rx_queues; q++)
+	for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++)
 		ibmveth_free_queue_buffer_pools(adapter, q);
 
 	netdev_dbg(adapter->netdev, "freed buffer pools for %u queue(s)\n",
-		   adapter->num_rx_queues);
+		   ibmveth_get_num_rx_queues(adapter));
 }
 
 static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
@@ -1556,6 +1610,138 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
 	return rc;
 }
 
+/**
+ * ibmveth_register_logical_lan_queue - Register subordinate queue with
+ * hypervisor
+ * @adapter: ibmveth adapter structure
+ * @rxq_desc: Receive queue descriptor
+ * @queue_index: RX queue index (1..N for subordinate queues)
+ *
+ * Registers a subordinate receive queue using H_REG_LOGICAL_LAN_QUEUE.
+ * On success, stores the queue handle and virtual IRQ in the adapter.
+ * If IRQ mapping fails after a successful hypervisor registration, the
+ * queue is freed before returning.
+ *
+ * Return: H_SUCCESS on success, negative errno on IRQ mapping failure,
+ *         hypervisor error code otherwise
+ */
+static int
+ibmveth_register_logical_lan_queue(struct ibmveth_adapter *adapter,
+				   union ibmveth_buf_desc rxq_desc,
+				   int queue_index)
+{
+	unsigned long handle, hwirq;
+	unsigned int virq;
+	long lpar_rc;
+	unsigned long ua = adapter->vdev->unit_address;
+	unsigned long bl = adapter->buffer_list_dma[queue_index];
+
+	netdev_dbg(adapter->netdev,
+		   "register queue %d: ua=0x%lx bl=0x%lx rxq=0x%llx\n",
+		   queue_index, ua, bl, rxq_desc.desc);
+	do {
+		lpar_rc = h_register_logical_lan_queue(ua, bl,
+						       rxq_desc.desc, &handle,
+						       &hwirq);
+	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+	netdev_dbg(adapter->netdev,
+		   "h_register_logical_lan_queue queue %d rc=%ld\n",
+		   queue_index, lpar_rc);
+
+	if (lpar_rc == H_SUCCESS) {
+		virq = irq_create_mapping(NULL, hwirq);
+		if (!virq) {
+			unsigned long free_rc;
+
+			netdev_err(adapter->netdev,
+				   "Failed to map IRQ for queue %d (hwirq=%lu)\n",
+				   queue_index, hwirq);
+			do {
+				free_rc = h_free_logical_lan_queue(ua, handle);
+			} while (H_IS_LONG_BUSY(free_rc) ||
+				  (free_rc == H_BUSY));
+			if (free_rc != H_SUCCESS)
+				netdev_err(adapter->netdev,
+					   "h_free_logical_lan_queue failed for queue %d after IRQ map failure: rc=0x%lx\n",
+					   queue_index, free_rc);
+			return -EINVAL;
+		}
+
+		adapter->queue_handle[queue_index] = handle;
+		adapter->queue_irq[queue_index] = virq;
+
+		netdev_dbg(adapter->netdev,
+			   "queue %d registered: handle=0x%llx irq=%u\n",
+			   queue_index, adapter->queue_handle[queue_index],
+			   adapter->queue_irq[queue_index]);
+		return H_SUCCESS;
+	}
+
+	/*
+	 * H_FUNCTION means firmware rejected this subordinate register
+	 * (MQ unsupported / dropped after LPM). Caller fails this open and
+	 * latches mq_fallback so the next open applies SQ; keep a specific
+	 * log then the generic failure lines below.
+	 */
+	if (lpar_rc == H_FUNCTION)
+		netdev_err(adapter->netdev,
+			   "h_register_logical_lan_queue H_FUNCTION for queue %d (firmware MQ unsupported)\n",
+			   queue_index);
+
+	netdev_err(adapter->netdev,
+		   "h_register_logical_lan_queue failed for queue %d with %ld\n",
+		   queue_index, lpar_rc);
+	netdev_err(adapter->netdev,
+		   "queue %d params: unit_addr=0x%x buffer_list_dma=0x%llx rxq_desc=0x%llx\n",
+		   queue_index, adapter->vdev->unit_address,
+		   adapter->buffer_list_dma[queue_index],
+		   rxq_desc.desc);
+
+	return lpar_rc;
+}
+
+/**
+ * ibmveth_register_single_rx_queue - Register one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to register (1..N)
+ *
+ * Builds the queue descriptor and registers with the hypervisor via
+ * ibmveth_register_logical_lan_queue().
+ *
+ * Return: 0 on success, -EINVAL if @queue_idx is invalid, -EOPNOTSUPP if
+ *         firmware rejects MQ (H_FUNCTION), -EIO on other failures
+ */
+static int
+ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,
+				 int queue_idx)
+{
+	struct net_device *netdev = adapter->netdev;
+	union ibmveth_buf_desc rxq_desc;
+	long lpar_rc;
+
+	if (WARN_ON(queue_idx < 1 || queue_idx >= IBMVETH_MAX_RX_QUEUES))
+		return -EINVAL;
+
+	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
+				    adapter->rx_queue[queue_idx].queue_len;
+	rxq_desc.fields.address = adapter->rx_queue[queue_idx].queue_dma;
+
+	lpar_rc = ibmveth_register_logical_lan_queue(adapter, rxq_desc,
+						     queue_idx);
+	if (lpar_rc != H_SUCCESS) {
+		netdev_err(netdev, "Failed to register queue %d: rc=0x%lx\n",
+			   queue_idx, lpar_rc);
+		if (lpar_rc == H_FUNCTION)
+			return -EOPNOTSUPP;
+		return -EIO;
+	}
+
+	netdev_dbg(netdev, "Registered queue %d with handle 0x%llx\n",
+		   queue_idx, adapter->queue_handle[queue_idx]);
+
+	return 0;
+}
+
 /**
  * ibmveth_free_all_queues - Free all RX queues at once
  * @adapter: ibmveth adapter structure
@@ -1573,7 +1759,8 @@ static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
  * afterward (same as pre-helper close()).
  *
  * Clears queue handles only; queue_irq[] is released by
- * ibmveth_cleanup_rx_interrupts().
+ * ibmveth_cleanup_rx_interrupts() on close, or by
+ * ibmveth_dispose_subordinate_irq_mappings() on partial register failure.
  */
 static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
 {
@@ -1591,7 +1778,7 @@ static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
 			   "h_free_logical_lan failed: %ld\n", lpar_rc);
 	}
 
-	for (i = 0; i < adapter->num_rx_queues; i++)
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
 		adapter->queue_handle[i] = 0;
 }
 
@@ -1600,10 +1787,13 @@ static void ibmveth_free_all_queues(struct ibmveth_adapter *adapter)
  * @adapter: ibmveth adapter structure
  * @mac_address: MAC address for device registration
  *
- * Registers queue 0 via ibmveth_register_logical_lan(). Subordinate queue
- * registration is added when multi-queue RX is enabled.
+ * Registers queue 0 via ibmveth_register_logical_lan(), then subordinate
+ * queues 1..N when multi-queue mode is enabled.
  *
- * Return: 0 on success, -ENONET if queue 0 registration fails
+ * Return: 0 on success, -ENONET if queue 0 registration fails,
+ *         -EOPNOTSUPP if firmware rejects a subordinate queue (H_FUNCTION;
+ *         this function latches mq_fallback), -EIO on other subordinate
+ *         failures
  */
 static int
 ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
@@ -1611,7 +1801,8 @@ ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
 	struct net_device *netdev = adapter->netdev;
 	union ibmveth_buf_desc rxq_desc;
 	unsigned long lpar_rc;
-	int rc;
+	unsigned int num;
+	int i, rc;
 
 	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
 				    adapter->rx_queue[0].queue_len;
@@ -1636,9 +1827,67 @@ ibmveth_register_rx_queues(struct ibmveth_adapter *adapter, u64 mac_address)
 		return -ENONET;
 	}
 
+	num = ibmveth_get_num_rx_queues(adapter);
+	if (num == 1 || !adapter->multi_queue) {
+		netdev_dbg(netdev,
+			   "registered 1 RX queue with hypervisor (single-queue mode)\n");
+		return 0;
+	}
+
+	netdev_dbg(netdev, "Registering %u subordinate queues (1-%u)\n",
+		   num - 1, num - 1);
+
+	for (i = 1; i < num; i++) {
+		rc = ibmveth_register_single_rx_queue(adapter, i);
+		if (rc) {
+			/* Firmware MQ gone: fall back to SQ on next open. */
+			if (rc == -EOPNOTSUPP)
+				adapter->mq_fallback = true;
+			goto err_unregister;
+		}
+	}
+
 	netdev_dbg(netdev,
-		   "registered 1 RX queue with hypervisor (single-queue mode)\n");
+		   "registered %u RX queues with hypervisor (multi-queue mode)\n",
+		   num);
+
 	return 0;
+
+err_unregister:
+	ibmveth_dispose_subordinate_irq_mappings(adapter);
+	ibmveth_free_all_queues(adapter);
+	return rc;
+}
+
+/**
+ * ibmveth_apply_mq_fallback - Drop multi-queue mode after firmware rejection
+ * @adapter: ibmveth adapter
+ *
+ * mq_fallback is set when firmware rejects MQ (subordinate register or
+ * buffer-add H_FUNCTION). Apply only at the start of open after teardown so
+ * num_rx_queues is not shrunk while IRQ/NAPI still reference higher queues.
+ * Consumes the flag and clears multi_queue, which is what makes the
+ * single-queue decision permanent for this device.
+ */
+static void ibmveth_apply_mq_fallback(struct ibmveth_adapter *adapter)
+{
+	struct net_device *netdev = adapter->netdev;
+
+	if (!adapter->mq_fallback)
+		return;
+
+	adapter->mq_fallback = false;
+
+	if (!adapter->multi_queue && ibmveth_get_num_rx_queues(adapter) == 1)
+		return;
+
+	netdev_warn(netdev,
+		    "Falling back to single RX queue (firmware MQ unavailable)\n");
+	adapter->multi_queue = false;
+	ibmveth_publish_num_rx_queues(adapter, 1);
+	/* real_num_rx_queues is set later in open after resources exist. */
+	if (adapter->rx_buffers_per_hcall > IBMVETH_MAX_RX_REGULAR)
+		adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
 }
 
 static int ibmveth_open(struct net_device *netdev)
@@ -1651,6 +1900,8 @@ static int ibmveth_open(struct net_device *netdev)
 
 	netdev_dbg(netdev, "open starting\n");
 
+	ibmveth_apply_mq_fallback(adapter);
+
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		rxq_entries += adapter->rx_buff_pool[0][i].size;
 
@@ -1670,18 +1921,34 @@ static int ibmveth_open(struct net_device *netdev)
 	if (rc)
 		goto out_free_buffer_pools;
 
-	rc = netif_set_real_num_rx_queues(netdev, adapter->num_rx_queues);
+	rc = netif_set_real_num_rx_queues(netdev,
+					  ibmveth_get_num_rx_queues(adapter));
+
 	if (rc) {
 		netdev_err(netdev, "failed to set number of rx queues\n");
 		goto out_unregister_queues;
 	}
 
+	/*
+	 * Post buffers before setup_rx_interrupts(). MQ setup then unmasks
+	 * PHYP; SQ setup leaves PHYP masked. Scheduling NAPI only when a
+	 * descriptor is already pending is not enough: after ifdown/up
+	 * (RX=8, no -L) NAPI can be idle with nothing pending and the
+	 * queue stays dead (TX OK, ARP/RX fail).
+	 * restart_rx_queue() replenishes, schedules NAPI, and unmasks if
+	 * prep fails.
+	 */
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
+		netdev_dbg(netdev, "initial replenish cycle for queue %d\n", i);
+		ibmveth_replenish_task(adapter, i);
+	}
+
 	rc = ibmveth_setup_rx_interrupts(adapter);
 	if (rc)
 		goto out_free_all_queues; /* setup already disposed IRQs */
 
-	netdev_dbg(netdev, "initial replenish cycle\n");
-	ibmveth_schedule_rx_queue(adapter, 0);
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+		ibmveth_restart_rx_queue(adapter, i);
 
 	rc = ibmveth_alloc_tx_resources(adapter);
 	if (rc)
@@ -1714,6 +1981,7 @@ static int ibmveth_open(struct net_device *netdev)
 static int ibmveth_close(struct net_device *netdev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
+	int i;
 
 	/* Gate on opened, not IFF_UP: pool_store/change_mtu close+open can
 	 * leave IFF_UP set after a failed reopen.
@@ -1734,7 +2002,8 @@ static int ibmveth_close(struct net_device *netdev)
 	/* Wait for softirq/poll that already passed shutdown checks. */
 	synchronize_net();
 
-	ibmveth_update_rx_no_buffer(adapter);
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+		ibmveth_update_rx_no_buffer(adapter, i);
 	ibmveth_free_all_queues(adapter);
 	/* Free TX LTBs after quiesce and after H_FREE_LOGICAL_LAN so xmit
 	 * cannot touch unmapped bounce buffers while the LAN is live.
@@ -2527,6 +2796,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
 		ibmveth_rx_csum_helper(skb, adapter);
 	}
 
+	skb_record_rx_queue(skb, queue_index);
 	napi_gro_receive(napi, skb);
 
 	netdev->stats.rx_packets++;
@@ -2545,7 +2815,7 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 	queue_index = napi - adapter->napi;
 
 	if (WARN_ON(queue_index < 0 ||
-		    queue_index >= adapter->num_rx_queues)) {
+		    queue_index >= ibmveth_get_num_rx_queues(adapter))) {
 		if (budget)
 			napi_complete_done(napi, 0);
 		return 0;
@@ -2638,10 +2908,11 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
 
 	qindex = napi - adapter->napi;
 	/*
-	 * Quiet on out-of-range: teardown can leave a residual IRQ after the
-	 * live count drops. Do not WARN-storm; return IRQ_NONE until free_irq.
+	 * Quiet on out-of-range: scale-down publishes a lower live count
+	 * before free_irq(). A residual IRQ must not WARN-storm; return
+	 * IRQ_NONE until the handler is removed.
 	 */
-	if (qindex < 0 || qindex >= adapter->num_rx_queues)
+	if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
 		return IRQ_NONE;
 
 	ibmveth_schedule_rx_queue(adapter, qindex);
@@ -2750,9 +3021,14 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
 static void ibmveth_poll_controller(struct net_device *dev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(dev);
+	unsigned int num = ibmveth_get_num_rx_queues(adapter);
+	int i;
 
-	ibmveth_replenish_task(adapter, 0);
-	ibmveth_schedule_rx_queue(adapter, 0);
+	for (i = 0; i < num; i++)
+		ibmveth_replenish_task(adapter, i);
+
+	for (i = 0; i < num; i++)
+		ibmveth_schedule_rx_queue(adapter, i);
 }
 #endif
 
@@ -2770,8 +3046,7 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
 	struct ibmveth_adapter *adapter;
 	struct iommu_table *tbl;
 	unsigned long ret;
-	int i;
-	int rxqentries = 1;
+	int i, q;
 
 	tbl = get_iommu_table_base(&vdev->dev);
 
@@ -2781,23 +3056,35 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
 
 	adapter = netdev_priv(netdev);
 
-	ret = IBMVETH_BUFF_LIST_SIZE + IBMVETH_FILT_LIST_SIZE;
+	/* One buffer list page per RX queue; filter list is shared. */
+	ret = IBMVETH_BUFF_LIST_SIZE * ibmveth_get_num_rx_queues(adapter) +
+	      IBMVETH_FILT_LIST_SIZE;
 	ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
 	/* add size of mapped tx buffers */
 	ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
 
-	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-		/* add the size of the active receive buffers */
-		if (adapter->rx_buff_pool[0][i].active)
-			ret +=
-			    adapter->rx_buff_pool[0][i].size *
-			    IOMMU_PAGE_ALIGN(adapter->rx_buff_pool[0][i].
-					     buff_size, tbl);
-		rxqentries += adapter->rx_buff_pool[0][i].size;
-	}
-	/* add the size of the receive queue entries */
-	ret += IOMMU_PAGE_ALIGN(
-		rxqentries * sizeof(struct ibmveth_rx_q_entry), tbl);
+	/*
+	 * Pool metadata for queues 1+ is copied from queue 0 at open.
+	 * Always size from pool 0 x num_rx_queues (valid at probe before that
+	 * copy).
+	 */
+	for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
+		int rxqentries = 1;
+
+		for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+			struct ibmveth_buff_pool *bpool =
+				&adapter->rx_buff_pool[0][i];
+
+			if (bpool->active)
+				ret += bpool->size *
+					IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
+			rxqentries += bpool->size;
+		}
+
+		/* add the size of the receive queue entries */
+		ret += IOMMU_PAGE_ALIGN(rxqentries *
+					sizeof(struct ibmveth_rx_q_entry), tbl);
+	}
 
 	return ret;
 }
@@ -2862,9 +3149,18 @@ static const struct net_device_ops ibmveth_netdev_ops = {
 #endif
 };
 
+static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,
+				   int pools_ready)
+{
+	int i;
+
+	for (i = 0; i < pools_ready; i++)
+		kobject_put(&adapter->rx_buff_pool[0][i].kobj);
+}
+
 static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 {
-	int rc, i, mac_len;
+	int rc, i, mac_len, pools_ready = 0;
 	struct net_device *netdev;
 	struct ibmveth_adapter *adapter;
 	unsigned char *mac_addr_p;
@@ -2899,7 +3195,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 		return -EINVAL;
 	}
 
-	netdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter), IBMVETH_MAX_QUEUES, 1);
+	netdev = alloc_etherdev_mqs(sizeof(struct ibmveth_adapter),
+				    IBMVETH_MAX_QUEUES, IBMVETH_MAX_RX_QUEUES);
 	if (!netdev)
 		return -ENOMEM;
 
@@ -2922,7 +3219,9 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 	adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
 	ibmveth_init_link_settings(netdev);
 
-	netif_napi_add_weight(netdev, &adapter->napi[0], ibmveth_poll, 16);
+	for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+		netif_napi_add_weight(netdev, &adapter->napi[i],
+				      ibmveth_poll, 16);
 
 	netdev->irq = dev->irq;
 	netdev->netdev_ops = &ibmveth_netdev_ops;
@@ -2954,16 +3253,30 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 		netdev->features |= NETIF_F_FRAGLIST;
 	}
 
-	/* Initialize queue count - always 1 for now */
-	adapter->multi_queue = 0;
-	adapter->num_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;
+	if (ret == H_SUCCESS &&
+	    (ret_attr & IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT)) {
+		adapter->multi_queue = true;
+		ibmveth_publish_num_rx_queues(adapter,
+					      min(num_online_cpus(),
+						  IBMVETH_DEFAULT_QUEUES));
+		netdev_dbg(netdev, "RX multi queue mode enabled: %u queues\n",
+			   ibmveth_get_num_rx_queues(adapter));
+	} else {
+		adapter->multi_queue = false;
+		ibmveth_publish_num_rx_queues(adapter,
+					      IBMVETH_DEFAULT_RX_QUEUES);
+	}
 
 	if (ret == H_SUCCESS &&
 	    (ret_attr & IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT)) {
-		adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
+		if (adapter->multi_queue)
+			adapter->rx_buffers_per_hcall =
+				IBMVETH_MAX_RX_PER_HCALL;
+		else
+			adapter->rx_buffers_per_hcall = IBMVETH_MAX_RX_REGULAR;
 		netdev_dbg(netdev,
 			   "RX Multi-buffer hcall supported by FW, batch set to %u\n",
-			    adapter->rx_buffers_per_hcall);
+			   adapter->rx_buffers_per_hcall);
 	} else {
 		adapter->rx_buffers_per_hcall = 1;
 		netdev_dbg(netdev,
@@ -2981,25 +3294,62 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
 		struct kobject *kobj = &adapter->rx_buff_pool[0][i].kobj;
-		int error;
 
 		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
 					 pool_count[i], pool_size[i],
 					 pool_active[i]);
-		error = kobject_init_and_add(kobj, &ktype_veth_pool,
-					     &dev->dev.kobj, "pool%d", i);
-		if (!error)
-			kobject_uevent(kobj, KOBJ_ADD);
+		rc = kobject_init_and_add(kobj, &ktype_veth_pool,
+					  &dev->dev.kobj, "pool%d", i);
+		if (rc) {
+			dev_err(&dev->dev,
+				"failed to create pool%d kobject: %d\n", i, rc);
+			/* init_and_add takes a ref even on failure */
+			kobject_put(kobj);
+			ibmveth_put_pool_kobjs(adapter, pools_ready);
+			dev_set_drvdata(&dev->dev, NULL);
+			free_netdev(netdev);
+			return rc;
+		}
+
+		pools_ready++;
+		kobject_uevent(kobj, KOBJ_ADD);
 	}
 
+	/*
+	 * VIO CMO entitlement was set before probe (netdev NULL, so default).
+	 * Recompute now that num_rx_queues and pool 0 metadata are known.
+	 */
+	if (firmware_has_feature(FW_FEATURE_CMO))
+		vio_cmo_set_dev_desired(dev, ibmveth_get_desired_dma(dev));
+
 	rc = netif_set_real_num_tx_queues(netdev, min(num_online_cpus(),
 						      IBMVETH_DEFAULT_QUEUES));
 	if (rc) {
 		netdev_dbg(netdev, "failed to set number of tx queues rc=%d\n",
 			   rc);
+		ibmveth_put_pool_kobjs(adapter, pools_ready);
+		dev_set_drvdata(&dev->dev, NULL);
+		free_netdev(netdev);
+		return rc;
+	}
+
+	/*
+	 * alloc_etherdev_mqs() sized real_num_rx_queues to MAX. Match the
+	 * advertised default (or SQ 1) before register_netdev so down-state
+	 * readers agree with adapter->num_rx_queues / ethtool -l.
+	 */
+	rc = netif_set_real_num_rx_queues(netdev,
+					  ibmveth_get_num_rx_queues(adapter));
+
+	if (rc) {
+		netdev_dbg(netdev, "failed to set number of rx queues rc=%d\n",
+			   rc);
+		ibmveth_put_pool_kobjs(adapter, pools_ready);
+		dev_set_drvdata(&dev->dev, NULL);
 		free_netdev(netdev);
 		return rc;
 	}
+
 	adapter->tx_ltb_size = PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE);
 	for (i = 0; i < IBMVETH_MAX_QUEUES; i++)
 		adapter->tx_ltb_ptr[i] = NULL;
@@ -3013,6 +3363,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 
 	if (rc) {
 		netdev_dbg(netdev, "failed to register netdev rc=%d\n", rc);
+		ibmveth_put_pool_kobjs(adapter, pools_ready);
+		dev_set_drvdata(&dev->dev, NULL);
 		free_netdev(netdev);
 		return rc;
 	}
@@ -3184,7 +3536,6 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
 	return rc;
 }
 
-
 #define ATTR(_name, _mode)				\
 	struct attribute veth_##_name##_attr = {	\
 	.name = __stringify(_name), .mode = _mode,	\
@@ -3217,8 +3568,12 @@ static int ibmveth_resume(struct device *dev)
 {
 	struct net_device *netdev = dev_get_drvdata(dev);
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
+	unsigned int num = ibmveth_get_num_rx_queues(adapter);
+	int i;
+
+	for (i = 0; i < num; i++)
+		ibmveth_schedule_rx_queue(adapter, i);
 
-	ibmveth_schedule_rx_queue(adapter, 0);
 	return 0;
 }
 
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 88448e915e86..cf9e77fc2190 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -30,6 +30,7 @@
 #define IbmVethMcastRemoveFilter     0x2UL
 #define IbmVethMcastClearFilterTable 0x3UL
 
+#define IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT	0x0000000000080000UL
 #define IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT	0x0000000000040000UL
 #define IBMVETH_ILLAN_LRG_SR_ENABLED	0x0000000000010000UL
 #define IBMVETH_ILLAN_LRG_SND_SUPPORT	0x0000000000008000UL
@@ -262,7 +263,7 @@ static inline long h_illan_attributes(unsigned long unit_address,
 #define IBMVETH_MAX_TX_BUF_SIZE (1024 * 64)
 #define IBMVETH_MAX_QUEUES 16U
 #define IBMVETH_DEFAULT_QUEUES 8U
-#define IBMVETH_MAX_RX_QUEUES 1U
+#define IBMVETH_MAX_RX_QUEUES 16U
 #define IBMVETH_DEFAULT_RX_QUEUES 1U
 #define IBMVETH_MAX_RX_REGULAR 8U
 #define IBMVETH_MAX_RX_PER_HCALL 12U
@@ -317,8 +318,9 @@ struct ibmveth_adapter {
 	struct ibmveth_rx_q rx_queue[IBMVETH_MAX_RX_QUEUES];
 	u64 queue_handle[IBMVETH_MAX_RX_QUEUES];
 	unsigned int queue_irq[IBMVETH_MAX_RX_QUEUES];
-	int multi_queue;
+	bool multi_queue;
 	unsigned int num_rx_queues;
+	bool mq_fallback;
 	/* Lifetime: true after successful ndo_open until close clears it. */
 	bool opened;
 	/* Lifetime: true while RX IRQ handlers / NAPI are installed. */
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 11/15] ibmveth: Add per-queue RX and TX statistics collection

[Mingming Cao]

MQ RX points several queues at the same adapter-wide counters, which
races the updates and leaves no way to attribute a count to a queue.

Move every counter that has more than one writer into per-queue
structs allocated at probe and freed at remove:

  struct ibmveth_rx_queue_stats
  struct ibmveth_tx_queue_stats

Each slot has a single writer — replenish_* under that queue's
replenish_lock, the other RX fields from that queue's NAPI, TX under
the stack's per-queue TX lock — so plain u64 is enough on this
PPC64-only driver. No atomic and no u64_stats_sync.

packets, bytes and drops go through struct netdev_stat_ops
(.get_queue_stats_rx, .get_queue_stats_tx, .get_base_stats).
.ndo_get_stats64() sums the per-queue packet and byte counters into
64-bit device totals. ethtool -S keeps only the driver-specific keys
that have no standard equivalent: interrupts, polls, large_packets,
invalid_buffers and no_buffer_drops per RX queue; large_packets,
send_failures and checksum_offload per TX queue. ETH_SS_STATS becomes
variable-length because that block scales with the live queue count.
Hypercall counters and pool%d_ keys are not added: which hcall a
batch picks is not ABI, and size/active already have sysfs (available
for every queue is patch 13).

The thirteen existing ethtool -S keys keep their exact names, their
order and their adapter-wide values, summed from the per-queue slots
on read. The storage moved; that ABI did not. The four replenish_*
counters get per-queue storage but no per-queue key of their own.

Holding that ABI while the storage moves needs the ethtool -S table
to record where each key lives. IBMVETH_STAT_OFF() could only express
an offset into struct ibmveth_adapter. Tag every entry with an enum
ibmveth_stat_src naming the struct it indexes: adapter-wide keys are
read directly, per-queue keys are summed across the slots by one pair
of offset-keyed helpers. That is what lets the field names change
while the key names do not (rx_invalid_buffer now reads
invalid_buffers, tx_send_failed reads send_failures, and the two
large_packets fields live in different structs). tx_map_failed still
reads from the adapter — it has no writer, here or in mainline —
and the three fw_enabled_* keys are capability flags, not counters.

The per-queue keys come from their own tables with the counts derived
by ARRAY_SIZE(), so get_strings(), get_ethtool_stats() and
get_sset_count() cannot drift apart.

ndo_get_stats64() walks every allocated slot rather than only the live
queues, so device totals cannot go backwards when ethtool -L shrinks
the queue count. get_base_stats() therefore reports the retired-queue
remainder rather than zero; the core sums it with the live queues it
iterates itself. Zeroing would assert that the live-queue sum is
already complete. Every field the per-queue callbacks fill is also
initialised there, because netdev_nl_stats_add() drops a field from
the device total unless both sides set it.

Give every queue a no_buffer_retired carry. PHYP's drop counter is
absolute for the buffer-list page currently mapped, so a reopen or a
queue reuse restarts it near zero. Storing only the newest absolute in
adapter->rx_no_buffer meant whichever queue ran last won, and the
value could go backwards. The carry sits beside the no_buffer_drops it
accumulates from, so both belong to one queue. The rx%d_no_buffer_drops
key reports that live page absolute on its own, so it is the one
exported value that is not monotonic; the adapter-wide rx_no_buffer
sums the two and per-queue rx-hw-drops includes both.

Freeing these arrays in remove() forces the teardown order to be
fixed first. Mainline cancels reset work before unregister_netdev(),
but the RX path stays live until unregister and can re-arm it, so the
worker could run after the cancel and reach memory this patch now
frees. unregister_netdev() therefore moves ahead of cancel_work_sync(),
and ibmveth_reset() returns early unless reg_state is NETREG_REGISTERED.
That reorder is a use-after-free fix in its own right; it is carried
here because this patch depends on it. No Fixes: tag — a stable
backport of a feature patch this size is the wrong vehicle; if the
fix is wanted on its own it should be lifted and tagged separately.

ibmveth_probe_cleanup() also clears the vio drvdata before
free_netdev(). A probe failure never reaches ibmveth_remove(), and
CMO get_desired_dma() reads that pointer on a later rebind.

Readers do not test the arrays for NULL: both exist from before
register_netdev() until after unregister_netdev() and
cancel_work_sync(), and probe fails -ENOMEM if either allocation
does.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- wrap the qstats local in replenish (81 cols)
- replenish_* per-queue u64, summed on the existing adapter-wide keys;
  no per-queue replenish key; no atomics
- netdev_stat_ops for packets/bytes/drops, not private -S strings
- enum ibmveth_stat_src so existing -S keys keep names while storage
  moves; ARRAY_SIZE() for the per-queue key counts
- no hcall_* keys (buffer-submit and H_SEND_LOGICAL_LAN)
- drop the three pool%d_ keys (15 ethtool entries)
- drop the fifteen qstats NULL checks outside the allocators
  (five in the RX hot path)
- per-queue no_buffer_retired carry
- get_base_stats() reports the retired-queue remainder
- gate reset on NETREG_REGISTERED
- noted: harvest no_buffer on -L shrink is patch 14

Changes in v5:
- Series renumber: mailed v4 10/14 stats -> tip P11 (P09 peel;
  get_channels -> P12)
- rx_no_buffer_retired + sum MAX_* slots so adapter no-buffer / qstat
  totals stay monotonic across reopen and channel shrink
- probe_cleanup: clear vio drvdata before free_netdev (CMO cannot see a
  freed netdev on rebind)
- remove: unregister_netdev then cancel_work_sync (no UAF reset worker)

Changes in v4:
- Merge v3's separate RX and TX stats commits into one patch.
- Introduce rx_queue_stats / tx_qstats / NUM macros here (first use).
- Allocate/free qstats at probe/remove instead of open/close.
- Report adapter-level ethtool strings by summing per-queue counters on
  read; drop aggregate_* helpers.
- Sum global rx_no_buffer across MQ queues into this statistics patch.
- Cacheline-align per-queue stats; derive field counts with offsetof so
  alignment padding is not counted as a statistic.
- probe_cleanup() cancels reset work, puts pool kobjects via helper from
  the prior patch, and frees qstats on probe failure paths.
- Keep plain u64 qstats like existing ibmveth / ibmvnic (PPC_PSERIES).

 drivers/net/ethernet/ibm/ibmveth.c | 507 +++++++++++++++++++++++++----
 drivers/net/ethernet/ibm/ibmveth.h |  57 +++-
 2 files changed, 492 insertions(+), 72 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 2e8896ea5af2..f4fddfa56571 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -38,6 +38,7 @@
 #include <asm/firmware.h>
 #include <net/tcp.h>
 #include <net/ip6_checksum.h>
+#include <net/netdev_queues.h>
 
 #include "ibmveth.h"
 
@@ -75,32 +76,101 @@ module_param(old_large_send, bool, 0444);
 MODULE_PARM_DESC(old_large_send,
 	"Use old large send method on firmware that supports the new method");
 
+/**
+ * enum ibmveth_stat_src - where an ethtool -S counter is stored
+ * @IBMVETH_STAT_ADAPTER: plain u64 in struct ibmveth_adapter
+ * @IBMVETH_STAT_RX_QSUM: per-queue u64, summed over rx_qstats[]
+ * @IBMVETH_STAT_TX_QSUM: per-queue u64, summed over tx_qstats[]
+ * @IBMVETH_STAT_RX_NO_BUFFER: rx_qstats[] live-page absolute plus the
+ *	absolutes carried over from pages the queue has already retired
+ *
+ * Counters live per-queue so multi-queue writers never share a field.
+ * The adapter is only ever read from ethtool, so summing there is free.
+ */
+enum ibmveth_stat_src {
+	IBMVETH_STAT_ADAPTER,
+	IBMVETH_STAT_RX_QSUM,
+	IBMVETH_STAT_TX_QSUM,
+	IBMVETH_STAT_RX_NO_BUFFER,
+};
+
 struct ibmveth_stat {
 	char name[ETH_GSTRING_LEN];
-	int offset;
+	enum ibmveth_stat_src src;
+	/* Offset into the struct named by @src. */
+	size_t off;
 };
 
 #define IBMVETH_STAT_OFF(stat) offsetof(struct ibmveth_adapter, stat)
+#define IBMVETH_RXQ_OFF(stat) offsetof(struct ibmveth_rx_queue_stats, stat)
+#define IBMVETH_TXQ_OFF(stat) offsetof(struct ibmveth_tx_queue_stats, stat)
 #define IBMVETH_GET_STAT(a, off) *((u64 *)(((unsigned long)(a)) + off))
 
+#define IBMVETH_ADAPTER_STAT(key, field) \
+	{ key, IBMVETH_STAT_ADAPTER, IBMVETH_STAT_OFF(field) }
+#define IBMVETH_RXQ_STAT(key, field) \
+	{ key, IBMVETH_STAT_RX_QSUM, IBMVETH_RXQ_OFF(field) }
+#define IBMVETH_TXQ_STAT(key, field) \
+	{ key, IBMVETH_STAT_TX_QSUM, IBMVETH_TXQ_OFF(field) }
+
+/*
+ * Key names and their order are ABI. Do not reorder or rename; append
+ * only, and only when the counter is worth a permanent interface.
+ */
 static struct ibmveth_stat ibmveth_stats[] = {
-	{ "replenish_task_cycles", IBMVETH_STAT_OFF(replenish_task_cycles) },
-	{ "replenish_no_mem", IBMVETH_STAT_OFF(replenish_no_mem) },
-	{ "replenish_add_buff_failure",
-			IBMVETH_STAT_OFF(replenish_add_buff_failure) },
-	{ "replenish_add_buff_success",
-			IBMVETH_STAT_OFF(replenish_add_buff_success) },
-	{ "rx_invalid_buffer", IBMVETH_STAT_OFF(rx_invalid_buffer) },
-	{ "rx_no_buffer", IBMVETH_STAT_OFF(rx_no_buffer) },
-	{ "tx_map_failed", IBMVETH_STAT_OFF(tx_map_failed) },
-	{ "tx_send_failed", IBMVETH_STAT_OFF(tx_send_failed) },
-	{ "fw_enabled_ipv4_csum", IBMVETH_STAT_OFF(fw_ipv4_csum_support) },
-	{ "fw_enabled_ipv6_csum", IBMVETH_STAT_OFF(fw_ipv6_csum_support) },
-	{ "tx_large_packets", IBMVETH_STAT_OFF(tx_large_packets) },
-	{ "rx_large_packets", IBMVETH_STAT_OFF(rx_large_packets) },
-	{ "fw_enabled_large_send", IBMVETH_STAT_OFF(fw_large_send_support) }
+	IBMVETH_RXQ_STAT("replenish_task_cycles", replenish_task_cycles),
+	IBMVETH_RXQ_STAT("replenish_no_mem", replenish_no_mem),
+	IBMVETH_RXQ_STAT("replenish_add_buff_failure",
+			 replenish_add_buff_failure),
+	IBMVETH_RXQ_STAT("replenish_add_buff_success",
+			 replenish_add_buff_success),
+	IBMVETH_RXQ_STAT("rx_invalid_buffer", invalid_buffers),
+	{ "rx_no_buffer", IBMVETH_STAT_RX_NO_BUFFER,
+	  IBMVETH_RXQ_OFF(no_buffer_drops) },
+	IBMVETH_ADAPTER_STAT("tx_map_failed", tx_map_failed),
+	IBMVETH_TXQ_STAT("tx_send_failed", send_failures),
+	IBMVETH_ADAPTER_STAT("fw_enabled_ipv4_csum", fw_ipv4_csum_support),
+	IBMVETH_ADAPTER_STAT("fw_enabled_ipv6_csum", fw_ipv6_csum_support),
+	IBMVETH_TXQ_STAT("tx_large_packets", large_packets),
+	IBMVETH_RXQ_STAT("rx_large_packets", large_packets),
+	IBMVETH_ADAPTER_STAT("fw_enabled_large_send", fw_large_send_support),
 };
 
+/**
+ * struct ibmveth_qstat - a per-queue counter exposed through ethtool -S
+ * @fmt: key name, taking the queue index as its only argument
+ * @off: offset into the matching per-queue stats struct
+ *
+ * Driving the strings and the values from one table keeps the two in
+ * step; get_sset_count() derives its length from ARRAY_SIZE() so the
+ * three cannot drift apart.
+ */
+struct ibmveth_qstat {
+	const char *fmt;
+	size_t off;
+};
+
+/*
+ * Only counters with no home in the standard interfaces belong here.
+ * packets, bytes and drops are reported through netdev_stat_ops.
+ */
+static const struct ibmveth_qstat ibmveth_rx_qstat_keys[] = {
+	{ "rx%d_interrupts", IBMVETH_RXQ_OFF(interrupts) },
+	{ "rx%d_polls", IBMVETH_RXQ_OFF(polls) },
+	{ "rx%d_large_packets", IBMVETH_RXQ_OFF(large_packets) },
+	{ "rx%d_invalid_buffers", IBMVETH_RXQ_OFF(invalid_buffers) },
+	{ "rx%d_no_buffer_drops", IBMVETH_RXQ_OFF(no_buffer_drops) },
+};
+
+static const struct ibmveth_qstat ibmveth_tx_qstat_keys[] = {
+	{ "tx%d_large_packets", IBMVETH_TXQ_OFF(large_packets) },
+	{ "tx%d_send_failures", IBMVETH_TXQ_OFF(send_failures) },
+	{ "tx%d_checksum_offload", IBMVETH_TXQ_OFF(checksum_offload) },
+};
+
+#define IBMVETH_NUM_RX_QSTATS ARRAY_SIZE(ibmveth_rx_qstat_keys)
+#define IBMVETH_NUM_TX_QSTATS ARRAY_SIZE(ibmveth_tx_qstat_keys)
+
 /* simple methods of getting data from the current rxq entry */
 static u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,
 			     int queue_index)
@@ -241,6 +311,60 @@ ibmveth_free_filter_list(struct ibmveth_adapter *adapter)
 	}
 }
 
+/**
+ * ibmveth_alloc_rx_qstats - Allocate per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_rx_qstats(struct ibmveth_adapter *adapter)
+{
+	adapter->rx_qstats = kcalloc(IBMVETH_MAX_RX_QUEUES,
+				     sizeof(*adapter->rx_qstats),
+				     GFP_KERNEL);
+	if (!adapter->rx_qstats)
+		return -ENOMEM;
+
+	return 0;
+}
+
+/**
+ * ibmveth_free_rx_qstats - Free per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_rx_qstats(struct ibmveth_adapter *adapter)
+{
+	kfree(adapter->rx_qstats);
+	adapter->rx_qstats = NULL;
+}
+
+/**
+ * ibmveth_alloc_tx_qstats - Allocate per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_qstats(struct ibmveth_adapter *adapter)
+{
+	adapter->tx_qstats = kcalloc(IBMVETH_MAX_QUEUES,
+				     sizeof(*adapter->tx_qstats),
+				     GFP_KERNEL);
+	if (!adapter->tx_qstats)
+		return -ENOMEM;
+
+	return 0;
+}
+
+/**
+ * ibmveth_free_tx_qstats - Free per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_tx_qstats(struct ibmveth_adapter *adapter)
+{
+	kfree(adapter->tx_qstats);
+	adapter->tx_qstats = NULL;
+}
+
 /**
  * ibmveth_alloc_rx_queues - Allocate per-queue RX resources
  * @adapter: ibmveth adapter structure
@@ -839,6 +963,8 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 					 int queue_index,
 					 struct ibmveth_replenish_fail *fail)
 {
+	struct ibmveth_rx_queue_stats *qstats =
+		&adapter->rx_qstats[queue_index];
 	union ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};
 	u32 remaining = pool->size - atomic_read(&pool->available);
 	u64 correlators[IBMVETH_MAX_RX_PER_HCALL] = {0};
@@ -865,7 +991,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 		for (filled = 0; filled < min(remaining, batch); filled++) {
 			index = pool->free_map[free_index];
 			if (index == IBM_VETH_INVALID_MAP) {
-				adapter->replenish_add_buff_failure++;
+				qstats->replenish_add_buff_failure++;
 				outcome = IBMVETH_REPLENISH_RESET_MAP;
 				break;
 			}
@@ -876,8 +1002,8 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 				skb = netdev_alloc_skb(adapter->netdev,
 						       pool->buff_size);
 				if (!skb) {
-					adapter->replenish_no_mem++;
-					adapter->replenish_add_buff_failure++;
+					qstats->replenish_no_mem++;
+					qstats->replenish_add_buff_failure++;
 					break;
 				}
 
@@ -892,7 +1018,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 							     DMA_ATTR_NO_WARN);
 				if (dma_mapping_error(dev, dma_addr)) {
 					dev_kfree_skb_any(skb);
-					adapter->replenish_add_buff_failure++;
+					qstats->replenish_add_buff_failure++;
 					break;
 				}
 
@@ -953,7 +1079,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 		}
 
 		buffers_added += filled;
-		adapter->replenish_add_buff_success += filled;
+		qstats->replenish_add_buff_success += filled;
 		remaining -= filled;
 
 		memset(&descs, 0, sizeof(descs));
@@ -976,7 +1102,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 				pool->skbuff[index] = NULL;
 			}
 		}
-		adapter->replenish_add_buff_failure += filled;
+		qstats->replenish_add_buff_failure += filled;
 
 		if (lpar_rc == H_FUNCTION) {
 			if (adapter->multi_queue) {
@@ -1017,6 +1143,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
 static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
 					int queue_index)
 {
+	struct ibmveth_rx_queue_stats *qstats;
 	__be64 *p;
 	u64 drops;
 
@@ -1028,7 +1155,18 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
 	p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
 	drops = be64_to_cpup(p);
 
-	adapter->rx_no_buffer = drops;
+	/*
+	 * PHYP's buffer-list page counter is absolute for that page. A new
+	 * page (reopen / queue reuse after -L) starts near zero; fold the
+	 * previous absolute into this queue's retired carry so sums stay
+	 * monotonic. Both fields belong to the queue being updated, so this
+	 * stays single-writer under the queue's replenish_lock.
+	 */
+	qstats = &adapter->rx_qstats[queue_index];
+
+	if (drops < qstats->no_buffer_drops)
+		qstats->no_buffer_retired += qstats->no_buffer_drops;
+	qstats->no_buffer_drops = drops;
 }
 
 /* replenish routine */
@@ -1050,10 +1188,10 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
 		return;
 	}
 
-	adapter->replenish_task_cycles++;
-
 	spin_lock_irqsave(&rxq->replenish_lock, flags);
 
+	adapter->rx_qstats[queue_index].replenish_task_cycles++;
+
 	for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
 		struct ibmveth_buff_pool *pool =
 			&adapter->rx_buff_pool[queue_index][i];
@@ -2038,6 +2176,10 @@ static void ibmveth_reset(struct work_struct *w)
 	netdev_dbg(netdev, "reset starting\n");
 
 	rtnl_lock();
+	if (netdev->reg_state != NETREG_REGISTERED) {
+		rtnl_unlock();
+		return;
+	}
 
 	dev_close(adapter->netdev);
 	dev_open(adapter->netdev, NULL);
@@ -2271,22 +2413,96 @@ static int ibmveth_set_features(struct net_device *dev,
 	return rc1 ? rc1 : rc2;
 }
 
-static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8 *data)
+/*
+ * Sum per-queue counters for rare ethtool reads. The hot paths only ever
+ * touch their own queue's slot, so nothing here needs an atomic; the cost
+ * of aggregation is paid by the reader instead (ibmvnic-style).
+ *
+ * Every slot is summed, not just the live ones, so that shrinking the
+ * queue count with ethtool -L cannot make a counter go backwards.
+ */
+static u64 ibmveth_sum_rx_qstat(struct ibmveth_adapter *adapter, size_t off)
+{
+	u64 total = 0;
+	int i;
+
+	for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+		total += *(u64 *)((u8 *)&adapter->rx_qstats[i] + off);
+
+	return total;
+}
+
+static u64 ibmveth_sum_tx_qstat(struct ibmveth_adapter *adapter, size_t off)
 {
+	u64 total = 0;
 	int i;
 
+	for (i = 0; i < IBMVETH_MAX_QUEUES; i++)
+		total += *(u64 *)((u8 *)&adapter->tx_qstats[i] + off);
+
+	return total;
+}
+
+static u64 ibmveth_ethtool_adapter_stat(struct ibmveth_adapter *adapter,
+					int index)
+{
+	const struct ibmveth_stat *stat = &ibmveth_stats[index];
+
+	switch (stat->src) {
+	case IBMVETH_STAT_RX_QSUM:
+		return ibmveth_sum_rx_qstat(adapter, stat->off);
+	case IBMVETH_STAT_TX_QSUM:
+		return ibmveth_sum_tx_qstat(adapter, stat->off);
+	case IBMVETH_STAT_RX_NO_BUFFER:
+		/*
+		 * PHYP's page counter is absolute for the page currently
+		 * mapped, so a reopen or queue reuse restarts it near zero.
+		 * ibmveth_update_rx_no_buffer() folds each decrease into the
+		 * queue's retired carry; add both back to stay monotonic.
+		 */
+		return ibmveth_sum_rx_qstat(adapter, stat->off) +
+		       ibmveth_sum_rx_qstat(adapter,
+					    IBMVETH_RXQ_OFF(no_buffer_retired));
+	case IBMVETH_STAT_ADAPTER:
+		break;
+	}
+
+	return IBMVETH_GET_STAT(adapter, stat->off);
+}
+
+static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8 *data)
+{
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+	u8 *p = data;
+	int i, j;
+
 	if (stringset != ETH_SS_STATS)
 		return;
 
-	for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++, data += ETH_GSTRING_LEN)
-		memcpy(data, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+	for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++) {
+		memcpy(p, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+		p += ETH_GSTRING_LEN;
+	}
+
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+		for (j = 0; j < IBMVETH_NUM_RX_QSTATS; j++)
+			ethtool_sprintf(&p, ibmveth_rx_qstat_keys[j].fmt, i);
+
+	for (i = 0; i < dev->real_num_tx_queues; i++)
+		for (j = 0; j < IBMVETH_NUM_TX_QSTATS; j++)
+			ethtool_sprintf(&p, ibmveth_tx_qstat_keys[j].fmt, i);
 }
 
 static int ibmveth_get_sset_count(struct net_device *dev, int sset)
 {
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+
 	switch (sset) {
 	case ETH_SS_STATS:
-		return ARRAY_SIZE(ibmveth_stats);
+		return ARRAY_SIZE(ibmveth_stats) +
+		       ibmveth_get_num_rx_queues(adapter) *
+		       IBMVETH_NUM_RX_QSTATS +
+		       dev->real_num_tx_queues * IBMVETH_NUM_TX_QSTATS;
 	default:
 		return -EOPNOTSUPP;
 	}
@@ -2295,11 +2511,27 @@ static int ibmveth_get_sset_count(struct net_device *dev, int sset)
 static void ibmveth_get_ethtool_stats(struct net_device *dev,
 				      struct ethtool_stats *stats, u64 *data)
 {
-	int i;
 	struct ibmveth_adapter *adapter = netdev_priv(dev);
+	int i, j, k;
 
 	for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++)
-		data[i] = IBMVETH_GET_STAT(adapter, ibmveth_stats[i].offset);
+		data[i] = ibmveth_ethtool_adapter_stat(adapter, i);
+
+	for (j = 0; j < ibmveth_get_num_rx_queues(adapter); j++) {
+		const u8 *q = (const u8 *)&adapter->rx_qstats[j];
+
+		for (k = 0; k < IBMVETH_NUM_RX_QSTATS; k++)
+			data[i++] = *(const u64 *)
+				(q + ibmveth_rx_qstat_keys[k].off);
+	}
+
+	for (j = 0; j < dev->real_num_tx_queues; j++) {
+		const u8 *q = (const u8 *)&adapter->tx_qstats[j];
+
+		for (k = 0; k < IBMVETH_NUM_TX_QSTATS; k++)
+			data[i++] = *(const u64 *)
+				(q + ibmveth_tx_qstat_keys[k].off);
+	}
 }
 
 static void ibmveth_get_channels(struct net_device *netdev,
@@ -2411,8 +2643,10 @@ static int ibmveth_send(struct ibmveth_adapter *adapter,
 }
 
 static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
-					 struct net_device *netdev)
+					 struct ibmveth_adapter *adapter,
+					 int queue_num)
 {
+	struct net_device *netdev = adapter->netdev;
 	struct ethhdr *ether_header;
 	int ret = 0;
 
@@ -2420,7 +2654,7 @@ static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
 
 	if (ether_addr_equal(ether_header->h_dest, netdev->dev_addr)) {
 		netdev_dbg(netdev, "veth doesn't support loopback packets, dropping packet.\n");
-		netdev->stats.tx_dropped++;
+		adapter->tx_qstats[queue_num].dropped_packets++;
 		ret = -EOPNOTSUPP;
 	}
 
@@ -2438,11 +2672,11 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 
 	/* Close / failed reopen can free LTBs while IFF_UP is still set. */
 	if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
-		netdev->stats.tx_dropped++;
+		adapter->tx_qstats[queue_num].dropped_packets++;
 		goto out;
 	}
 
-	if (ibmveth_is_packet_unsupported(skb, netdev))
+	if (ibmveth_is_packet_unsupported(skb, adapter, queue_num))
 		goto out;
 	/* veth can't checksum offload UDP */
 	if (skb->ip_summed == CHECKSUM_PARTIAL &&
@@ -2453,7 +2687,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 	    skb_checksum_help(skb)) {
 
 		netdev_err(netdev, "tx: failed to checksum packet\n");
-		netdev->stats.tx_dropped++;
+		adapter->tx_qstats[queue_num].dropped_packets++;
 		goto out;
 	}
 
@@ -2465,6 +2699,8 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 
 		desc_flags |= (IBMVETH_BUF_NO_CSUM | IBMVETH_BUF_CSUM_GOOD);
 
+		adapter->tx_qstats[queue_num].checksum_offload++;
+
 		/* Need to zero out the checksum */
 		buf[0] = 0;
 		buf[1] = 0;
@@ -2476,7 +2712,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 	if (skb->ip_summed == CHECKSUM_PARTIAL && skb_is_gso(skb)) {
 		if (adapter->fw_large_send_support) {
 			mss = (unsigned long)skb_shinfo(skb)->gso_size;
-			adapter->tx_large_packets++;
+			adapter->tx_qstats[queue_num].large_packets++;
 		} else if (!skb_is_gso_v6(skb)) {
 			/* Put -1 in the IP checksum to tell phyp it
 			 * is a largesend packet. Put the mss in
@@ -2485,7 +2721,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 			ip_hdr(skb)->check = 0xffff;
 			tcp_hdr(skb)->check =
 				cpu_to_be16(skb_shinfo(skb)->gso_size);
-			adapter->tx_large_packets++;
+			adapter->tx_qstats[queue_num].large_packets++;
 		}
 	}
 
@@ -2493,7 +2729,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 	if (unlikely(skb->len > adapter->tx_ltb_size)) {
 		netdev_err(adapter->netdev, "tx: packet size (%u) exceeds ltb (%u)\n",
 			   skb->len, adapter->tx_ltb_size);
-		netdev->stats.tx_dropped++;
+		adapter->tx_qstats[queue_num].dropped_packets++;
 		goto out;
 	}
 	memcpy(adapter->tx_ltb_ptr[queue_num], skb->data, skb_headlen(skb));
@@ -2510,7 +2746,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 	if (unlikely(total_bytes != skb->len)) {
 		netdev_err(adapter->netdev, "tx: incorrect packet len copied into ltb (%u != %u)\n",
 			   skb->len, total_bytes);
-		netdev->stats.tx_dropped++;
+		adapter->tx_qstats[queue_num].dropped_packets++;
 		goto out;
 	}
 	desc.fields.flags_len = desc_flags | skb->len;
@@ -2519,11 +2755,11 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
 	dma_wmb();
 
 	if (ibmveth_send(adapter, desc.desc, mss)) {
-		adapter->tx_send_failed++;
-		netdev->stats.tx_dropped++;
+		adapter->tx_qstats[queue_num].send_failures++;
+		adapter->tx_qstats[queue_num].dropped_packets++;
 	} else {
-		netdev->stats.tx_packets++;
-		netdev->stats.tx_bytes += skb->len;
+		adapter->tx_qstats[queue_num].packets++;
+		adapter->tx_qstats[queue_num].bytes += skb->len;
 	}
 
 out:
@@ -2652,7 +2888,7 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
 static void ibmveth_poll_bump_invalid(struct ibmveth_adapter *adapter,
 				      int queue_index)
 {
-	adapter->rx_invalid_buffer++;
+	adapter->rx_qstats[queue_index].invalid_buffers++;
 }
 
 static bool ibmveth_poll_stopping(struct net_device *netdev,
@@ -2788,7 +3024,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
 	if ((length > netdev->mtu + ETH_HLEN) || lrg_pkt ||
 	    iph_check == 0xffff) {
 		ibmveth_rx_mss_helper(skb, mss, lrg_pkt);
-		adapter->rx_large_packets++;
+		adapter->rx_qstats[queue_index].large_packets++;
 	}
 
 	if (csum_good) {
@@ -2799,8 +3035,8 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
 	skb_record_rx_queue(skb, queue_index);
 	napi_gro_receive(napi, skb);
 
-	netdev->stats.rx_packets++;
-	netdev->stats.rx_bytes += length;
+	adapter->rx_qstats[queue_index].packets++;
+	adapter->rx_qstats[queue_index].bytes += length;
 
 	return 1;
 }
@@ -2827,6 +3063,8 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
 		return 0;
 	}
 
+	adapter->rx_qstats[queue_index].polls++;
+
 restart_poll:
 	while (frames_processed < budget) {
 		if (ibmveth_poll_stopping(netdev, napi))
@@ -2915,6 +3153,8 @@ static irqreturn_t ibmveth_interrupt(int irq, void *dev_instance)
 	if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
 		return IRQ_NONE;
 
+	adapter->rx_qstats[qindex].interrupts++;
+
 	ibmveth_schedule_rx_queue(adapter, qindex);
 	return IRQ_HANDLED;
 }
@@ -3132,6 +3372,124 @@ static netdev_features_t ibmveth_features_check(struct sk_buff *skb,
 	return vlan_features_check(skb, features);
 }
 
+/**
+ * ibmveth_get_stats64 - Return aggregated per-queue statistics
+ * @dev: network device
+ * @stats: rtnl link statistics storage
+ *
+ * Sums per-queue rx_qstats and tx_qstats into the rtnl counters.
+ * Walk the full allocated arrays (not the live queue count) so shrinking
+ * channels cannot make the totals go backwards.
+ * Callers use ndo_get_stats64(); avoid updating netdev->stats on the
+ * xmit/poll paths to keep per-queue counters off the hot cache line.
+ */
+static void ibmveth_get_stats64(struct net_device *dev,
+				struct rtnl_link_stats64 *stats)
+{
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+	int i;
+
+	for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
+		stats->rx_packets += adapter->rx_qstats[i].packets;
+		stats->rx_bytes += adapter->rx_qstats[i].bytes;
+	}
+
+	for (i = 0; i < IBMVETH_MAX_QUEUES; i++) {
+		stats->tx_packets += adapter->tx_qstats[i].packets;
+		stats->tx_bytes += adapter->tx_qstats[i].bytes;
+		stats->tx_dropped += adapter->tx_qstats[i].dropped_packets;
+	}
+}
+
+static void ibmveth_get_queue_stats_rx(struct net_device *dev, int idx,
+				       struct netdev_queue_stats_rx *stats)
+{
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+
+	stats->packets = adapter->rx_qstats[idx].packets;
+	stats->bytes = adapter->rx_qstats[idx].bytes;
+	/*
+	 * All three are frames that entered the device and never left it,
+	 * which is what rx-hw-drops is specified to cover: no_buffer_drops
+	 * is PHYP dropping for lack of buffer space on the page mapped now,
+	 * no_buffer_retired the same for pages this queue has already
+	 * released, and invalid_buffers is a processing error.
+	 */
+	stats->hw_drops = adapter->rx_qstats[idx].no_buffer_drops +
+			  adapter->rx_qstats[idx].no_buffer_retired +
+			  adapter->rx_qstats[idx].invalid_buffers;
+	stats->alloc_fail = adapter->rx_qstats[idx].replenish_no_mem;
+}
+
+static void ibmveth_get_queue_stats_tx(struct net_device *dev, int idx,
+				       struct netdev_queue_stats_tx *stats)
+{
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+
+	stats->packets = adapter->tx_qstats[idx].packets;
+	stats->bytes = adapter->tx_qstats[idx].bytes;
+	stats->hw_drops = adapter->tx_qstats[idx].dropped_packets;
+}
+
+/**
+ * ibmveth_get_base_stats - account for traffic not on a live queue
+ * @dev: network device
+ * @rx: RX base statistics storage
+ * @tx: TX base statistics storage
+ *
+ * get_queue_stats_{rx,tx}() only report queues the core still iterates,
+ * i.e. below real_num_{rx,tx}_queues, while ibmveth_get_stats64() walks
+ * the full arrays so device totals stay monotonic across a shrink.
+ * Report the retired-queue remainder here, otherwise qstats and
+ * rtnl_link_stats64 disagree by a delta that grows with every shrink.
+ * Zeroing would not be neutral: per netdev_stat_ops it asserts the
+ * per-queue sum is already exact.
+ *
+ * Bound the live side with real_num_*_queues rather than the adapter's
+ * own count, so the split lines up with the core's iteration exactly.
+ *
+ * Every field the per-queue callbacks fill must also be initialised
+ * here: netdev_nl_stats_add() starts the sum at NETDEV_STAT_NOT_SET and
+ * only accumulates while both sides are set, so a field left unset here
+ * is dropped from the device total even though the queues report it.
+ */
+static void ibmveth_get_base_stats(struct net_device *dev,
+				   struct netdev_queue_stats_rx *rx,
+				   struct netdev_queue_stats_tx *tx)
+{
+	struct ibmveth_adapter *adapter = netdev_priv(dev);
+	unsigned int i;
+
+	rx->packets = 0;
+	rx->bytes = 0;
+	rx->alloc_fail = 0;
+	rx->hw_drops = 0;
+	tx->packets = 0;
+	tx->bytes = 0;
+	tx->hw_drops = 0;
+
+	for (i = dev->real_num_rx_queues; i < IBMVETH_MAX_RX_QUEUES; i++) {
+		rx->packets += adapter->rx_qstats[i].packets;
+		rx->bytes += adapter->rx_qstats[i].bytes;
+		rx->hw_drops += adapter->rx_qstats[i].no_buffer_drops +
+				adapter->rx_qstats[i].no_buffer_retired +
+				adapter->rx_qstats[i].invalid_buffers;
+		rx->alloc_fail += adapter->rx_qstats[i].replenish_no_mem;
+	}
+
+	for (i = dev->real_num_tx_queues; i < IBMVETH_MAX_QUEUES; i++) {
+		tx->packets += adapter->tx_qstats[i].packets;
+		tx->bytes += adapter->tx_qstats[i].bytes;
+		tx->hw_drops += adapter->tx_qstats[i].dropped_packets;
+	}
+}
+
+static const struct netdev_stat_ops ibmveth_stat_ops = {
+	.get_queue_stats_rx	= ibmveth_get_queue_stats_rx,
+	.get_queue_stats_tx	= ibmveth_get_queue_stats_tx,
+	.get_base_stats		= ibmveth_get_base_stats,
+};
+
 static const struct net_device_ops ibmveth_netdev_ops = {
 	.ndo_open		= ibmveth_open,
 	.ndo_stop		= ibmveth_close,
@@ -3144,6 +3502,7 @@ static const struct net_device_ops ibmveth_netdev_ops = {
 	.ndo_validate_addr	= eth_validate_addr,
 	.ndo_set_mac_address    = ibmveth_set_mac_addr,
 	.ndo_features_check	= ibmveth_features_check,
+	.ndo_get_stats64	= ibmveth_get_stats64,
 #ifdef CONFIG_NET_POLL_CONTROLLER
 	.ndo_poll_controller	= ibmveth_poll_controller,
 #endif
@@ -3158,6 +3517,23 @@ static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,
 		kobject_put(&adapter->rx_buff_pool[0][i].kobj);
 }
 
+static void ibmveth_probe_cleanup(struct ibmveth_adapter *adapter,
+				  int pools_ready)
+{
+	struct net_device *netdev = adapter->netdev;
+
+	cancel_work_sync(&adapter->work);
+	ibmveth_put_pool_kobjs(adapter, pools_ready);
+
+	ibmveth_free_tx_qstats(adapter);
+	ibmveth_free_rx_qstats(adapter);
+	/* Probe failure never reaches ibmveth_remove(); clear before free so
+	 * CMO get_desired_dma() cannot see a freed netdev on rebind.
+	 */
+	dev_set_drvdata(&adapter->vdev->dev, NULL);
+	free_netdev(netdev);
+}
+
 static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 {
 	int rc, i, mac_len, pools_ready = 0;
@@ -3223,9 +3599,16 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 		netif_napi_add_weight(netdev, &adapter->napi[i],
 				      ibmveth_poll, 16);
 
+	if (ibmveth_alloc_rx_qstats(adapter) ||
+	    ibmveth_alloc_tx_qstats(adapter)) {
+		ibmveth_probe_cleanup(adapter, 0);
+		return -ENOMEM;
+	}
+
 	netdev->irq = dev->irq;
 	netdev->netdev_ops = &ibmveth_netdev_ops;
 	netdev->ethtool_ops = &netdev_ethtool_ops;
+	netdev->stat_ops = &ibmveth_stat_ops;
 	SET_NETDEV_DEV(netdev, &dev->dev);
 	netdev->hw_features = NETIF_F_SG;
 	if (vio_get_attribute(dev, "ibm,illan-options", NULL) != NULL) {
@@ -3305,9 +3688,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 				"failed to create pool%d kobject: %d\n", i, rc);
 			/* init_and_add takes a ref even on failure */
 			kobject_put(kobj);
-			ibmveth_put_pool_kobjs(adapter, pools_ready);
-			dev_set_drvdata(&dev->dev, NULL);
-			free_netdev(netdev);
+			ibmveth_probe_cleanup(adapter, pools_ready);
 			return rc;
 		}
 
@@ -3327,9 +3708,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 	if (rc) {
 		netdev_dbg(netdev, "failed to set number of tx queues rc=%d\n",
 			   rc);
-		ibmveth_put_pool_kobjs(adapter, pools_ready);
-		dev_set_drvdata(&dev->dev, NULL);
-		free_netdev(netdev);
+		ibmveth_probe_cleanup(adapter, pools_ready);
 		return rc;
 	}
 
@@ -3344,9 +3723,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 	if (rc) {
 		netdev_dbg(netdev, "failed to set number of rx queues rc=%d\n",
 			   rc);
-		ibmveth_put_pool_kobjs(adapter, pools_ready);
-		dev_set_drvdata(&dev->dev, NULL);
-		free_netdev(netdev);
+		ibmveth_probe_cleanup(adapter, pools_ready);
 		return rc;
 	}
 
@@ -3363,9 +3740,7 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 
 	if (rc) {
 		netdev_dbg(netdev, "failed to register netdev rc=%d\n", rc);
-		ibmveth_put_pool_kobjs(adapter, pools_ready);
-		dev_set_drvdata(&dev->dev, NULL);
-		free_netdev(netdev);
+		ibmveth_probe_cleanup(adapter, pools_ready);
 		return rc;
 	}
 
@@ -3380,12 +3755,20 @@ static void ibmveth_remove(struct vio_dev *dev)
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
 	int i;
 
-	cancel_work_sync(&adapter->work);
-
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		kobject_put(&adapter->rx_buff_pool[0][i].kobj);
 
+	/*
+	 * Unregister first so NAPI/xmit cannot re-arm reset work after we
+	 * cancel it. cancel_work_sync() before unregister left a window
+	 * where poll could schedule_work() and the worker ran after
+	 * free_netdev().
+	 */
 	unregister_netdev(netdev);
+	cancel_work_sync(&adapter->work);
+
+	ibmveth_free_tx_qstats(adapter);
+	ibmveth_free_rx_qstats(adapter);
 
 	free_netdev(netdev);
 	dev_set_drvdata(&dev->dev, NULL);
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index cf9e77fc2190..0f2971c8627a 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -275,6 +275,43 @@ static int pool_active[] = { 1, 1, 0, 0, 1};
 
 #define IBM_VETH_INVALID_MAP ((u16)0xffff)
 
+/*
+ * Per-queue RX counters. No field has two concurrent writers:
+ * interrupts is written only from this queue's IRQ handler; polls,
+ * packets, bytes, large_packets and invalid_buffers only from its NAPI
+ * poll; replenish_* only under its replenish_lock; and no_buffer_drops
+ * and no_buffer_retired under that lock or from a teardown path already
+ * quiesced by napi_disable()/synchronize_irq(). Plain u64 is therefore
+ * sufficient and no atomic or u64_stats_sync is needed: the driver is
+ * PPC64-only, so 64-bit loads and stores do not tear.
+ */
+struct ibmveth_rx_queue_stats {
+	u64 packets;
+	u64 bytes;
+	u64 interrupts;
+	u64 polls;
+	u64 large_packets;
+	u64 invalid_buffers;
+	/* PHYP's per-page absolute drop count for the live page. */
+	u64 no_buffer_drops;
+	/* Absolutes from pages this queue has already retired. */
+	u64 no_buffer_retired;
+	u64 replenish_task_cycles;
+	u64 replenish_no_mem;
+	u64 replenish_add_buff_failure;
+	u64 replenish_add_buff_success;
+} ____cacheline_aligned_in_smp;
+
+/* Per-queue TX counters; serialized by the stack's per-queue TX lock. */
+struct ibmveth_tx_queue_stats {
+	u64 packets;
+	u64 bytes;
+	u64 large_packets;
+	u64 dropped_packets;
+	u64 send_failures;
+	u64 checksum_offload;
+} ____cacheline_aligned_in_smp;
+
 struct ibmveth_buff_pool {
     u32 size;
     u32 index;
@@ -333,17 +370,17 @@ struct ibmveth_adapter {
 	u64 fw_ipv6_csum_support;
 	u64 fw_ipv4_csum_support;
 	u64 fw_large_send_support;
-	/* adapter specific stats */
-	u64 replenish_task_cycles;
-	u64 replenish_no_mem;
-	u64 replenish_add_buff_failure;
-	u64 replenish_add_buff_success;
-	u64 rx_invalid_buffer;
-	u64 rx_no_buffer;
+	/*
+	 * Every other ethtool -S counter lives in rx_qstats/tx_qstats and is
+	 * summed on read. tx_map_failed predates multi-queue, has never been
+	 * updated by any code path, and is kept only so the key keeps
+	 * reporting the zero userspace already sees.
+	 */
 	u64 tx_map_failed;
-	u64 tx_send_failed;
-	u64 tx_large_packets;
-	u64 rx_large_packets;
+
+	struct ibmveth_rx_queue_stats *rx_qstats;
+	struct ibmveth_tx_queue_stats *tx_qstats;
+
 	/* Ethtool settings */
 	u8 duplex;
 	u32 speed;
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 12/15] ibmveth: Report MQ-aware RX counts in ethtool get_channels

[Mingming Cao]

Report MQ-aware channel limits and counts from get_channels():

  - max_rx is IBMVETH_MAX_RX_QUEUES when MQ firmware is enabled, else 1
  - rx_count tracks the driver's published num_rx_queues

Previously get_channels() mirrored netdev->real_num_rx_queues for both
fields, which under-reports max_rx on MQ firmware and can disagree with
the driver's private RX count before the next open.

This patch only reports. set_channels() still reads only tx_count, so
an RX channel request returns success without changing anything;
raising max_rx widens the range of requests that behave that way from
the live count to IBMVETH_MAX_RX_QUEUES. Patch 14 closes it, with
set_channels() reading rx_count and calling
ibmveth_resize_rx_channels(), and patch 15 completes the down-path
publish/rollback and caps max_rx once mq_fallback latches.

Keep this out of the stats patch: channel reporting is ethtool -l ABI,
independent of per-queue counters, so it can be reviewed and blamed on
its own.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v5:
- get_channels rx_count via get_num_rx_queues() (READ_ONCE consistency)
- New peel: get_channels from v4 stats -> tip P12 (14->15)
- get_channels: max_rx = MAX_RX_QUEUES when MQ else 1; rx_count =
  get_num_rx_queues() (was mirroring real_num_rx_queues for both)

 drivers/net/ethernet/ibm/ibmveth.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index f4fddfa56571..4f2d956b4c89 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -2537,11 +2537,16 @@ static void ibmveth_get_ethtool_stats(struct net_device *dev,
 static void ibmveth_get_channels(struct net_device *netdev,
 				 struct ethtool_channels *channels)
 {
+	struct ibmveth_adapter *adapter = netdev_priv(netdev);
+
 	channels->max_tx = ibmveth_real_max_tx_queues();
 	channels->tx_count = netdev->real_num_tx_queues;
 
-	channels->max_rx = netdev->real_num_rx_queues;
-	channels->rx_count = netdev->real_num_rx_queues;
+	if (adapter->multi_queue)
+		channels->max_rx = IBMVETH_MAX_RX_QUEUES;
+	else
+		channels->max_rx = 1;
+	channels->rx_count = ibmveth_get_num_rx_queues(adapter);
 }
 
 static int ibmveth_set_channels(struct net_device *netdev,
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 13/15] ibmveth: Expose per-queue buffer pool details via debugfs

[Mingming Cao]

With multi-queue RX each queue owns its own set of five buffer pools,
so a 16-queue adapter has 80 of them. Nothing reports their runtime
state: sysfs exposes queue 0 only, and only as configuration, and no
ethtool key is per-pool. When RX drops under load, rx%d_no_buffer_drops
names the queue but not which of its pools ran dry, nor how close the
others are.

Add a read-only buffer_pools debugfs file, one row per RX queue and
buffer pool:

  /sys/kernel/debug/ibmveth/<dev_name>/buffer_pools
  (e.g. /sys/kernel/debug/ibmveth/30000002/buffer_pools)

  Queue  Pool  Count  BuffSize  Active  Available

Active is live allocation (skbuff && free_map), not the sysfs
poolN/active configuration flag.

The root is driver-owned so each adapter directory can use its stable
vio name rather than the mutable netdev->name. It is created in
module_init() and unwound if vio_register_driver() fails.

A multi-line table does not belong in sysfs, so the historical queue-0
ABI is left alone:

  .../poolN/{active,num,size}

Those stay one-value configuration for queue-0 pool classes. Open
copies that geometry to queues 1..N. This series does not add
per-queue pool sysfs dirs.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- create the debugfs root in module_init() instead of lazily on
  first probe, which raced concurrent probes and could orphan the
  directory on ERR_PTR(-EEXIST)
- rename the pool buffer-count column from Size to Count, so the
  debugfs table stops reusing the word sysfs poolN/size spells as a
  byte length on the same pool object
- widen the down banner: geometry above queue 0 is only populated
  once open copies the queue-0 template
- scope the dump RTNL comment to geometry/pool->active; available is
  atomic_read

Changes in v5:
- debugfs buffer_pools_show walks get_num_rx_queues()
- Series renumber: mailed v4 11/14 debugfs -> tip P13 (14->15)
- Path uses stable vio dev_name under a driver-owned root (not netdev
  name - avoids rename/collide)
- rtnl_lock around dump (writers are under RTNL)
- Show Active/Available as 0 when pool !live (debugfs view; free-path
  available clear already in the buffer-submit patch)

Changes in v4:
- Move the all-queue buffer_pools diagnostic from sysfs to debugfs;
  subject updated to match.
- Keep historical queue-0 poolN/{active,num,size} sysfs as one-value
  config (template for MQ); do not add per-queue pool sysfs dirs.

 drivers/net/ethernet/ibm/ibmveth.c | 80 +++++++++++++++++++++++++++++-
 drivers/net/ethernet/ibm/ibmveth.h |  2 +
 2 files changed, 81 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 4f2d956b4c89..954846c9ec7b 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -31,6 +31,7 @@
 #include <linux/ipv6.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
+#include <linux/debugfs.h>
 #include <asm/hvcall.h>
 #include <linux/atomic.h>
 #include <asm/vio.h>
@@ -3513,6 +3514,67 @@ static const struct net_device_ops ibmveth_netdev_ops = {
 #endif
 };
 
+static int ibmveth_buffer_pools_show(struct seq_file *m, void *v)
+{
+	struct ibmveth_adapter *adapter = m->private;
+	int i, j;
+
+	/*
+	 * size / buff_size / pool->active are written under RTNL
+	 * (veth_pool_store, open template copy). Take the same lock so
+	 * those columns are not a torn snapshot. available is updated
+	 * from NAPI/softirq; only atomic_read() keeps it from tearing.
+	 * Not required for memory safety; embedded arrays only.
+	 */
+	rtnl_lock();
+
+	seq_puts(m, "Queue  Pool  Count  BuffSize  Active  Available\n");
+	seq_puts(m, "-----  ----  -----  --------  ------  ---------\n");
+	if (!adapter->opened) {
+		seq_puts(m, "# down: Active/Available 0 unless allocated\n");
+		seq_puts(m, "# down: geometry above queue 0 set at open\n");
+	}
+
+	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
+		for (j = 0; j < IBMVETH_NUM_BUFF_POOLS; j++) {
+			struct ibmveth_buff_pool *pool =
+				&adapter->rx_buff_pool[i][j];
+			bool live = pool->skbuff && pool->free_map;
+			int active = live ? pool->active : 0;
+			int available = live ? atomic_read(&pool->available)
+					     : 0;
+
+			seq_printf(m, "%5d  %4d  %5u  %8u  %6d  %9d\n",
+				   i, j, pool->size, pool->buff_size,
+				   active, available);
+		}
+	}
+
+	rtnl_unlock();
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(ibmveth_buffer_pools);
+
+/* Driver-owned root so per-adapter dirs use a stable vio name, not the
+ * mutable netdev->name (avoids stale names / eth0 collisions after rename).
+ */
+static struct dentry *ibmveth_dbg_root;
+
+static void ibmveth_debugfs_init(struct ibmveth_adapter *adapter)
+{
+	adapter->debugfs_dir =
+		debugfs_create_dir(dev_name(&adapter->vdev->dev),
+				   ibmveth_dbg_root);
+	debugfs_create_file("buffer_pools", 0400, adapter->debugfs_dir,
+			    adapter, &ibmveth_buffer_pools_fops);
+}
+
+static void ibmveth_debugfs_exit(struct ibmveth_adapter *adapter)
+{
+	debugfs_remove_recursive(adapter->debugfs_dir);
+	adapter->debugfs_dir = NULL;
+}
+
 static void ibmveth_put_pool_kobjs(struct ibmveth_adapter *adapter,
 				   int pools_ready)
 {
@@ -3751,6 +3813,8 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
 
 	netdev_dbg(netdev, "registered\n");
 
+	ibmveth_debugfs_init(adapter);
+
 	return 0;
 }
 
@@ -3760,6 +3824,8 @@ static void ibmveth_remove(struct vio_dev *dev)
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
 	int i;
 
+	ibmveth_debugfs_exit(adapter);
+
 	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
 		kobject_put(&adapter->rx_buff_pool[0][i].kobj);
 
@@ -3986,15 +4052,27 @@ static struct vio_driver ibmveth_driver = {
 
 static int __init ibmveth_module_init(void)
 {
+	int rc;
+
 	printk(KERN_DEBUG "%s: %s %s\n", ibmveth_driver_name,
 	       ibmveth_driver_string, ibmveth_driver_version);
 
-	return vio_register_driver(&ibmveth_driver);
+	ibmveth_dbg_root = debugfs_create_dir(ibmveth_driver_name, NULL);
+
+	rc = vio_register_driver(&ibmveth_driver);
+	if (rc) {
+		debugfs_remove_recursive(ibmveth_dbg_root);
+		ibmveth_dbg_root = NULL;
+	}
+
+	return rc;
 }
 
 static void __exit ibmveth_module_exit(void)
 {
 	vio_unregister_driver(&ibmveth_driver);
+	debugfs_remove_recursive(ibmveth_dbg_root);
+	ibmveth_dbg_root = NULL;
 }
 
 module_init(ibmveth_module_init);
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 0f2971c8627a..1276b3669f2c 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -381,6 +381,8 @@ struct ibmveth_adapter {
 	struct ibmveth_rx_queue_stats *rx_qstats;
 	struct ibmveth_tx_queue_stats *tx_qstats;
 
+	struct dentry *debugfs_dir;
+
 	/* Ethtool settings */
 	u8 duplex;
 	u32 speed;
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 14/15] ibmveth: Implement incremental MQ RX queue resize

[Mingming Cao]

Add ibmveth_resize_rx_queues_incremental() so the live RX channel
count can change without a full device tear-down, and
ibmveth_resize_rx_channels() as the entry point set_channels() calls.

Scale-up allocates DMA buffer lists and pool memory for the new
queues, registers them with PHYP via H_REG_LOGICAL_LAN_QUEUE and maps
subordinate IRQs, then brings each queue up: publish the count,
replenish, enable NAPI, and only then unmask PHYP, so the handler
cannot run on an unpublished, empty or NAPI-disabled queue. It
finishes with the same restart helper open() uses, so a total
replenish miss can still get a poll to retry.

Queues come up before netif_set_real_num_rx_queues() widens, so a frame
on a new queue can briefly carry an rx queue index above
real_num_rx_queues. RPS then skips steering and takes the default path,
tripping the WARN_ONCE in get_rps_cpu() if the device already had more
than one queue. Transient, and delivery is unaffected.

Scale-down masks PHYP on the retiring queues first, then disables NAPI,
then masks and synchronises again, because an in-flight poll can re-arm
PHYP while napi_disable() is waiting. It then drains the ring, harvests
the queue's final no_buffer count under that queue's replenish_lock,
since netpoll can still reach it until the count is lowered, publishes
the surviving count with smp_store_release() and calls
synchronize_net(), then netif_set_real_num_rx_queues(). On success
it then deregisters via H_FREE_LOGICAL_LAN_QUEUE, releases the IRQ
mapping and frees the DMA and pool memory. Deregistration has to
precede the unmap so PHYP releases ownership of the buffers while the
queue metadata its correlators index is still valid.

Both directions unwind. Scale-up failures jump to cleanup_new_queues:,
which destroys only the queues this call created and leaves the
previous set live and running. Scale-down failures republish the old
count, then replenish, napi_enable and unmask each retiring queue
(and restart if the IRQ comes back). If the IRQ cannot be re-enabled
they schedule a reset.

Reject rx > 1 with -EOPNOTSUPP when firmware lacks MQ support or
mq_fallback is set. That rejection lives here rather than in
set_channels(), and it sits below the no-op check: ethtool -L is
read-modify-write, so a TX-only request arrives carrying the current
RX count and must not be rejected once mq_fallback is set. An rx_count
outside 1..IBMVETH_MAX_RX_QUEUES is rejected with -EINVAL ahead of both
checks; the ethtool core already range-checks against the max_rx that
get_channels() reports, and this is the driver's own guard.

Refresh CMO entitlement across the resize. ibmveth_desired_dma_for_rxqs()
is factored out of ibmveth_get_desired_dma() so the same math can be
applied to a prospective queue count before the driver commits to it.

Key the resize path's early return on adapter->opened rather than
IFF_UP: RX buffers, mappings and IRQs exist only after a successful
open, so IFF_UP alone does not mean there is anything to resize.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- harvest no_buffer on retiring queues under that queue's
  replenish_lock before publish: netpoll still reaches them until
  the surviving count is stored, and update_rx_no_buffer() ignores
  indexes at or above the live count
- publish num_rx_queues, then synchronize_net(), then destroy.
  Scale-down runs set_real_num_rx_queues() between sync and destroy
  (destroy only if that succeeds); cleanup_new_queues does not harvest
- napi_disable() before publishing the lowered count on scale-up
  enable_irq failure, matching scale-down
- restart_rx_queue() after the checked enable_irq() on scale-up and
  on scale-down rollback; kick_rx_queue_if_pending() cannot retry a
  queue whose replenish posted nothing
- drop the scale-down survivor restart_rx_queue() loop: those queues
  were never quiesced, and enable_irq() on prep-fail unmasks behind
  a live poll (subordinate H_PARAMETER on enable is -EIO and reset)
- schedule_rx_queue() returns quietly when only the upper bound
  trips: netpoll walks a snapshot of the old count, so an ethtool -L
  shrink raced it into a WARN userspace can trigger at will
- drop leftover spin_lock_init() from alloc_single_rx_queue(); scale-up
  runs on a live adapter and probe already inits the lock
- zero buffer_list_dma[] on the alloc_single_rx_queue() map-failure
  path; free_single_rx_queue() keys its unmap on that field
- skip PHYP mask and remask on scale-down when queue_irq[i] is 0
- put the rx > 1 / mq_fallback gate below the no-op check, and refuse
  any rx > 1 once mq_fallback is set, so a TX-only read-modify-write
  is not rejected
- noted: down-path stash/CMO and RX rollback on TX fail are patch 15
- noted: KUnit free_map fixtures are patch 8

Changes in v5:
- resize / get_desired_dma read live count via get_num_rx_queues()
- Scale-down / scale-up-fail: remask+sync after napi_disable (poll may
  re-arm while disable waits)
- Series renumber: mailed v4 12/14 resize -> tip P14 (14->15)
- Absorb kitchen-sink resize ownership: teardown-first scale-down
  (drain/deregister before unmap) + live-pool correlator_valid /
  schedule_work (cover has peel map)
- Ordered num_rx_queues publish + publish-before-destroy
- After unmask: kick if pending; enable_irq errno + rollback check;
  raise CMO before scale-up allocs
- Scale-up enable_irq-fail: publish-down / napi_disable / drain / destroy
- After scale-down destroy, restart remaining RX queues so Q0 cannot stay
  masked with NAPI idle
- resize_rx_channels: reject rx>1 without MQ / validate 1..MAX; call
  before !opened early return (stash still tip P15)

Changes in v4:
- Copy pool->index when cloning buffer pools for incrementally added
  queues.
- Scale-up order: publish -> replenish -> napi_enable -> enable_irq
  (vs older enable-before-publish drafts); mirror NAPI-before-unmask on
  set_real_num_rx rollback.
- Linux-owned subordinate virq disposal; deregister is PHYP-only;
  dispose bound to MAX_RX_QUEUES; dispose on request_irq failure.
- Drain-path smp_rmb() before harvest.
- Mask PHYP before napi_disable/drain on scale-down and scale-up fail
  cleanup.
- Replenish before re-enabling IRQ/NAPI on set_real_num_rx scale-down
  rollback.
- Keep set_channels wiring as the following patch (same split as v3)
  but call resize_rx_channels() here when IFF_UP so the helper is not
  an unused static.

 drivers/net/ethernet/ibm/ibmveth.c | 732 +++++++++++++++++++++++++++--
 1 file changed, 692 insertions(+), 40 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 954846c9ec7b..5aef8a1f2c23 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -774,6 +774,58 @@ ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
 	adapter->rx_irq_setup = false;
 }
 
+/**
+ * ibmveth_setup_single_rx_interrupt - Setup interrupt for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to setup
+ *
+ * Registers the IRQ handler for one queue. Used during incremental
+ * scale-up when adding new RX queues. The caller publishes the queue,
+ * replenishes buffers, enables NAPI, then unmasks PHYP delivery.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_setup_single_rx_interrupt(struct ibmveth_adapter *adapter,
+				  int queue_idx)
+{
+	struct net_device *netdev = adapter->netdev;
+	int rc;
+
+	rc = request_irq(adapter->queue_irq[queue_idx], ibmveth_interrupt,
+			 0, netdev->name, &adapter->napi[queue_idx]);
+	if (rc) {
+		netdev_err(netdev, "request_irq() failed for queue %d: %d\n",
+			   queue_idx, rc);
+		return rc;
+	}
+
+	netdev_dbg(netdev, "Setup IRQ %d for queue %d\n",
+		   adapter->queue_irq[queue_idx], queue_idx);
+	return 0;
+}
+
+/**
+ * ibmveth_cleanup_single_rx_interrupt - Cleanup interrupt for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to cleanup
+ *
+ * Frees the IRQ handler for one queue and releases the subordinate virq
+ * mapping. Used during incremental scale-down.
+ */
+static void
+ibmveth_cleanup_single_rx_interrupt(struct ibmveth_adapter *adapter,
+				    int queue_idx)
+{
+	if (adapter->queue_irq[queue_idx]) {
+		free_irq(adapter->queue_irq[queue_idx],
+			 &adapter->napi[queue_idx]);
+		ibmveth_dispose_subordinate_irq_mapping(adapter, queue_idx);
+		netdev_dbg(adapter->netdev,
+			   "Freed IRQ for queue %d\n", queue_idx);
+	}
+}
+
 /**
  * ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue
  * @adapter: ibmveth adapter structure
@@ -793,7 +845,16 @@ static bool ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
 {
 	struct napi_struct *napi = &adapter->napi[qindex];
 
-	if (WARN_ON(qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter)))
+	if (WARN_ON(qindex < 0))
+		return false;
+
+	/*
+	 * A live shrink can publish a lower count while netpoll walks a
+	 * snapshot of the old one, so an index past the end is expected
+	 * here and must not splat. ibmveth_replenish_task() skips the
+	 * same way; callers already treat false as "queue is gone".
+	 */
+	if (qindex >= ibmveth_get_num_rx_queues(adapter))
 		return false;
 
 	/*
@@ -1261,8 +1322,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
  * SQ open leaves PHYP masked until the first poll. If schedule_prep fails,
  * NAPI never runs and the queue stays masked (TX OK, RX/ARP dead) until
  * reload. Replenish first so an enable_irq fallback can actually deliver.
- * Also used after every open (SQ and MQ) and after scale-down so a
- * queue is not left idle+masked.
+ * Also used after every open (SQ and MQ) and after scale-down rollback.
  */
 static void ibmveth_restart_rx_queue(struct ibmveth_adapter *adapter,
 				     int qindex)
@@ -1452,6 +1512,141 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
 		   ibmveth_get_num_rx_queues(adapter));
 }
 
+/**
+ * ibmveth_alloc_single_rx_queue - Allocate resources for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to allocate
+ * @rxq_entries: Number of RX queue entries
+ *
+ * Allocates buffer list, RX queue, and per-queue buffer pools for one queue.
+ * Used during incremental scale-up without affecting existing queues.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_alloc_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx,
+			      int rxq_entries)
+{
+	struct device *dev = &adapter->vdev->dev;
+	struct net_device *netdev = adapter->netdev;
+	int i, rc = -ENOMEM;
+
+	adapter->buffer_list_addr[queue_idx] =
+		(void *)get_zeroed_page(GFP_KERNEL);
+	if (!adapter->buffer_list_addr[queue_idx]) {
+		netdev_err(netdev, "unable to allocate buffer list for queue %d\n",
+			   queue_idx);
+		return -ENOMEM;
+	}
+
+	adapter->rx_queue[queue_idx].queue_len =
+		sizeof(struct ibmveth_rx_q_entry) * rxq_entries;
+	adapter->rx_queue[queue_idx].queue_addr =
+		dma_alloc_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+				   &adapter->rx_queue[queue_idx].queue_dma,
+				   GFP_KERNEL);
+	if (!adapter->rx_queue[queue_idx].queue_addr) {
+		netdev_err(netdev, "unable to allocate RX queue for queue %d\n",
+			   queue_idx);
+		goto out_free_buflist;
+	}
+
+	adapter->buffer_list_dma[queue_idx] =
+		dma_map_single(dev, adapter->buffer_list_addr[queue_idx],
+			       4096, DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(dev, adapter->buffer_list_dma[queue_idx])) {
+		netdev_err(netdev, "unable to map buffer list for queue %d\n",
+			   queue_idx);
+		adapter->buffer_list_dma[queue_idx] = 0;
+		goto out_free_rxq;
+	}
+
+	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+		struct ibmveth_buff_pool *src =
+			&adapter->rx_buff_pool[0][i];
+		struct ibmveth_buff_pool *dst =
+			&adapter->rx_buff_pool[queue_idx][i];
+
+		dst->size = src->size;
+		dst->index = src->index;
+		dst->buff_size = src->buff_size;
+		dst->threshold = src->threshold;
+		dst->active = src->active;
+	}
+
+	rc = ibmveth_alloc_queue_buffer_pools(adapter, queue_idx);
+	if (rc) {
+		netdev_err(netdev,
+			   "Failed to allocate buffer pools for queue %d\n",
+			   queue_idx);
+		goto out_unmap_buflist;
+	}
+
+	adapter->rx_queue[queue_idx].index = 0;
+	adapter->rx_queue[queue_idx].num_slots = rxq_entries;
+	adapter->rx_queue[queue_idx].toggle = 1;
+
+	netdev_dbg(netdev,
+		   "Allocated queue %d: buffer_list @ %p (DMA: 0x%llx), rx_queue @ %p (DMA: 0x%llx), %d entries\n",
+		   queue_idx, adapter->buffer_list_addr[queue_idx],
+		   (unsigned long long)adapter->buffer_list_dma[queue_idx],
+		   adapter->rx_queue[queue_idx].queue_addr,
+		   (unsigned long long)adapter->rx_queue[queue_idx].queue_dma,
+		   rxq_entries);
+
+	return 0;
+
+out_unmap_buflist:
+	dma_unmap_single(dev, adapter->buffer_list_dma[queue_idx],
+			 4096, DMA_BIDIRECTIONAL);
+	adapter->buffer_list_dma[queue_idx] = 0;
+out_free_rxq:
+	dma_free_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+			  adapter->rx_queue[queue_idx].queue_addr,
+			  adapter->rx_queue[queue_idx].queue_dma);
+	adapter->rx_queue[queue_idx].queue_addr = NULL;
+out_free_buflist:
+	free_page((unsigned long)adapter->buffer_list_addr[queue_idx]);
+	adapter->buffer_list_addr[queue_idx] = NULL;
+	return rc;
+}
+
+/**
+ * ibmveth_free_single_rx_queue - Free resources for a single RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to free
+ *
+ * Frees buffer list, RX queue, and per-queue buffer pools for one queue.
+ * Used during incremental scale-down without affecting remaining queues.
+ */
+static void
+ibmveth_free_single_rx_queue(struct ibmveth_adapter *adapter, int queue_idx)
+{
+	struct device *dev = &adapter->vdev->dev;
+
+	ibmveth_free_queue_buffer_pools(adapter, queue_idx);
+
+	if (adapter->buffer_list_dma[queue_idx]) {
+		dma_unmap_single(dev, adapter->buffer_list_dma[queue_idx],
+				 4096, DMA_BIDIRECTIONAL);
+		adapter->buffer_list_dma[queue_idx] = 0;
+	}
+
+	if (adapter->rx_queue[queue_idx].queue_addr) {
+		dma_free_coherent(dev, adapter->rx_queue[queue_idx].queue_len,
+				  adapter->rx_queue[queue_idx].queue_addr,
+				  adapter->rx_queue[queue_idx].queue_dma);
+		adapter->rx_queue[queue_idx].queue_addr = NULL;
+	}
+
+	if (adapter->buffer_list_addr[queue_idx]) {
+		free_page((unsigned long)adapter->buffer_list_addr[queue_idx]);
+		adapter->buffer_list_addr[queue_idx] = NULL;
+	}
+
+	netdev_dbg(adapter->netdev, "Freed queue %d resources\n", queue_idx);
+}
+
 static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
 					 int queue_index, u64 correlator)
 {
@@ -1613,6 +1808,56 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
 	return 0;
 }
 
+/**
+ * ibmveth_drain_rx_queue - Drain pending buffers from an RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_index: Queue index to drain
+ *
+ * Recycles all pending buffers back to the per-queue buffer pools.
+ * Must be called with NAPI disabled for this queue.
+ *
+ * Return: Number of buffers drained
+ */
+static int
+ibmveth_drain_rx_queue(struct ibmveth_adapter *adapter, int queue_index)
+{
+	struct net_device *netdev = adapter->netdev;
+	int drained = 0;
+	int limit = adapter->rx_queue[queue_index].num_slots;
+	int rc;
+
+	netdev_dbg(netdev, "Draining RX queue %d (limit: %d slots)\n",
+		   queue_index, limit);
+
+	while (drained < limit &&
+	       ibmveth_rxq_pending_buffer(adapter, queue_index)) {
+		/* Match poll-side order before harvesting completion state. */
+		smp_rmb();
+		rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);
+		if (rc) {
+			/* -EINVAL/-EFAULT already advanced past the slot. */
+			if (rc == -EINVAL || rc == -EFAULT) {
+				drained++;
+				continue;
+			}
+			netdev_err(netdev,
+				   "Failed to harvest buffer from queue %d during drain: %d\n",
+				   queue_index, rc);
+			break;
+		}
+		drained++;
+	}
+
+	if (drained > 0)
+		netdev_dbg(netdev, "Drained %d buffer(s) from RX queue %d\n",
+			   drained, queue_index);
+	else
+		netdev_dbg(netdev, "No buffers to drain from RX queue %d\n",
+			   queue_index);
+
+	return drained;
+}
+
 static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
 {
 	void *ltb = adapter->tx_ltb_ptr[idx];
@@ -1881,6 +2126,378 @@ ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,
 	return 0;
 }
 
+/**
+ * ibmveth_deregister_single_rx_queue - Deregister one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to deregister (1..N)
+ *
+ * Deregisters a single queue via H_FREE_LOGICAL_LAN_QUEUE. Linux IRQ handler
+ * teardown and subordinate virq mapping disposal are owned by interrupt
+ * cleanup helpers; queue 0 is freed only through ibmveth_free_all_queues()
+ * (H_FREE_LOGICAL_LAN).
+ */
+static void
+ibmveth_deregister_single_rx_queue(struct ibmveth_adapter *adapter,
+				   int queue_idx)
+{
+	unsigned long lpar_rc;
+	unsigned long ua = adapter->vdev->unit_address;
+	unsigned long qh = adapter->queue_handle[queue_idx];
+
+	if (!qh)
+		return;
+
+	do {
+		lpar_rc = h_free_logical_lan_queue(ua, qh);
+	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
+
+	if (lpar_rc != H_SUCCESS) {
+		netdev_err(adapter->netdev,
+			   "h_free_logical_lan_queue failed for queue %d: rc=0x%lx\n",
+			   queue_idx, lpar_rc);
+	}
+
+	adapter->queue_handle[queue_idx] = 0;
+
+	netdev_dbg(adapter->netdev, "Deregistered queue %d\n", queue_idx);
+}
+
+/**
+ * ibmveth_destroy_subordinate_rx_queue - Tear down one subordinate RX queue
+ * @adapter: ibmveth adapter structure
+ * @queue_idx: Queue index to destroy (1..N)
+ *
+ * Deregister with PHYP before unmapping buffer pools so hypervisor buffer
+ * ownership is released while queue metadata is still valid.
+ */
+static void
+ibmveth_destroy_subordinate_rx_queue(struct ibmveth_adapter *adapter,
+				     int queue_idx)
+{
+	ibmveth_deregister_single_rx_queue(adapter, queue_idx);
+	ibmveth_cleanup_single_rx_interrupt(adapter, queue_idx);
+	ibmveth_free_single_rx_queue(adapter, queue_idx);
+}
+
+/**
+ * ibmveth_desired_dma_for_rxqs - CMO entitlement for a given RX queue count
+ * @adapter: ibmveth adapter
+ * @rxqs: number of RX queues to size for
+ *
+ * Same math as ibmveth_get_desired_dma(), but uses @rxqs instead of the
+ * live adapter->num_rx_queues. Scale-up raises desired for the *target*
+ * count before allocating so vio_cmo_alloc cannot fail mid-resize.
+ *
+ * Return: bytes of IO memory desired for @rxqs RX queues
+ */
+static unsigned long
+ibmveth_desired_dma_for_rxqs(struct ibmveth_adapter *adapter,
+			     unsigned int rxqs)
+{
+	struct net_device *netdev = adapter->netdev;
+	struct iommu_table *tbl;
+	unsigned long ret;
+	int i, q;
+
+	tbl = get_iommu_table_base(&adapter->vdev->dev);
+
+	ret = IBMVETH_BUFF_LIST_SIZE * rxqs + IBMVETH_FILT_LIST_SIZE;
+	ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
+	ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
+
+	/*
+	 * Pool metadata for queues 1+ is copied from queue 0 at open.
+	 * Always size from pool 0 x @rxqs (valid at probe before that copy).
+	 */
+	for (q = 0; q < rxqs; q++) {
+		int rxqentries = 1;
+
+		for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
+			struct ibmveth_buff_pool *bpool =
+				&adapter->rx_buff_pool[0][i];
+
+			if (bpool->active)
+				ret += bpool->size *
+					IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
+			rxqentries += bpool->size;
+		}
+
+		ret += IOMMU_PAGE_ALIGN(rxqentries *
+					sizeof(struct ibmveth_rx_q_entry), tbl);
+	}
+
+	return ret;
+}
+
+/**
+ * ibmveth_resize_rx_queues_incremental - Resize RX queue count incrementally
+ * @adapter: ibmveth adapter structure
+ * @new_count: Target number of RX queues
+ * @rxq_entries: Number of entries per RX queue
+ *
+ * Adds or removes RX queues without tearing down the entire adapter.
+ * Active queues continue receiving during scale-up; scale-down drains
+ * excess queues before deregistering them with the hypervisor.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+ibmveth_resize_rx_queues_incremental(struct ibmveth_adapter *adapter,
+				     int new_count, int rxq_entries)
+{
+	struct net_device *netdev = adapter->netdev;
+	int old_count = ibmveth_get_num_rx_queues(adapter);
+	int failed_queue;
+	int rc, i;
+
+	if (old_count == new_count) {
+		netdev_dbg(netdev, "RX queue count unchanged (%d), nothing to do\n",
+			   old_count);
+		return 0;
+	}
+
+	if (new_count < 1 || new_count > IBMVETH_MAX_RX_QUEUES) {
+		netdev_err(netdev, "Invalid RX queue count %d (must be 1-%d)\n",
+			   new_count, IBMVETH_MAX_RX_QUEUES);
+		return -EINVAL;
+	}
+
+	netdev_info(netdev, "Incrementally resizing RX queues: %d to %d\n",
+		    old_count, new_count);
+
+	if (new_count > old_count) {
+		netdev_dbg(netdev, "Scale-up: adding queues %d-%d\n",
+			   old_count, new_count - 1);
+
+		/*
+		 * Raise CMO desired for the target count before dma_map /
+		 * dma_alloc_coherent / replenish (same order as change_mtu).
+		 * Do not bump live num_rx_queues here, only entitlement.
+		 */
+		if (firmware_has_feature(FW_FEATURE_CMO)) {
+			unsigned long dma;
+
+			dma = ibmveth_desired_dma_for_rxqs(adapter, new_count);
+			vio_cmo_set_dev_desired(adapter->vdev, dma);
+		}
+
+		for (i = old_count; i < new_count; i++) {
+			rc = ibmveth_alloc_single_rx_queue(adapter, i,
+							   rxq_entries);
+			if (rc) {
+				netdev_err(netdev, "Failed to allocate queue %d: %d\n",
+					   i, rc);
+				goto cleanup_new_queues;
+			}
+
+			rc = ibmveth_register_single_rx_queue(adapter, i);
+			if (rc) {
+				netdev_err(netdev, "Failed to register queue %d: %d\n",
+					   i, rc);
+				ibmveth_free_single_rx_queue(adapter, i);
+				goto cleanup_new_queues;
+			}
+
+			rc = ibmveth_setup_single_rx_interrupt(adapter, i);
+			if (rc) {
+				netdev_err(netdev,
+					   "Failed to setup IRQ for queue %d: %d\n",
+					   i, rc);
+				/* request_irq failed: mapped but no handler */
+				ibmveth_dispose_subordinate_irq_mapping(adapter,
+									i);
+				ibmveth_deregister_single_rx_queue(adapter, i);
+				ibmveth_free_single_rx_queue(adapter, i);
+				goto cleanup_new_queues;
+			}
+
+			/*
+			 * Fully ready before PHYP delivery, matching open():
+			 * publish -> replenish -> napi_enable -> enable_irq.
+			 * That way ibmveth_interrupt() cannot run on an
+			 * unpublished, empty, or NAPI-disabled queue.
+			 */
+			ibmveth_publish_num_rx_queues(adapter, i + 1);
+			ibmveth_replenish_task(adapter, i);
+			napi_enable(&adapter->napi[i]);
+
+			rc = ibmveth_enable_irq(adapter, i);
+			if (rc) {
+				netdev_err(netdev,
+					   "Failed to enable IRQ for queue %d: %d\n",
+					   i, rc);
+				/*
+				 * Published, replenished, and NAPI-enabled,
+				 * but PHYP never unmasked. Match scale-down /
+				 * shared cleanup: drain posted buffers, then
+				 * deregister before unmap via
+				 * destroy_subordinate.
+				 *
+				 * napi_disable() must come BEFORE the count
+				 * is lowered, matching scale-down and
+				 * cleanup_new_queues. Lowering it first does
+				 * not hide queue i from netpoll: after
+				 * ndo_poll_controller, netpoll_poll_dev()
+				 * calls poll_napi(), which walks dev->napi_list
+				 * unbounded by the queue count and skips a NAPI
+				 * only once NAPI_STATE_NPSVC is set. Queue i is
+				 * enabled here, so ibmveth_poll() would run and
+				 * trip its queue_index >= num_rx_queues
+				 * WARN_ON. napi_disable() sets NPSVC, so
+				 * poll_napi() skips the queue instead.
+				 */
+				napi_disable(&adapter->napi[i]);
+				ibmveth_publish_num_rx_queues(adapter, i);
+				ibmveth_drain_rx_queue(adapter, i);
+				synchronize_net();
+				ibmveth_destroy_subordinate_rx_queue(adapter,
+								     i);
+				/* enable_irq returns errno; keep -EIO. */
+				rc = -EIO;
+				goto cleanup_new_queues;
+			}
+			ibmveth_restart_rx_queue(adapter, i);
+		}
+
+		rc = netif_set_real_num_rx_queues(netdev, new_count);
+		if (rc) {
+			netdev_err(netdev, "Failed to set real RX queues to %d: %d\n",
+				   new_count, rc);
+			goto cleanup_new_queues;
+		}
+	} else {
+		netdev_dbg(netdev, "Scale-down: removing queues %d-%d\n",
+			   new_count, old_count - 1);
+
+		/*
+		 * Mask PHYP before napi_disable so the handler cannot miss
+		 * a mask while NAPI is already dead. An in-flight poll can
+		 * still re-arm PHYP while napi_disable() waits, so remask
+		 * and sync again after NAPI is stopped. Then drain, harvest
+		 * final no_buffer stats, publish the surviving count, and
+		 * synchronize_net() before destroy so netpoll cannot walk
+		 * dying queues (handler may still be registered until destroy).
+		 */
+		for (i = new_count; i < old_count; i++) {
+			if (!adapter->queue_irq[i])
+				continue;
+			ibmveth_disable_irq(adapter, i);
+			synchronize_irq(adapter->queue_irq[i]);
+		}
+
+		for (i = new_count; i < old_count; i++)
+			napi_disable(&adapter->napi[i]);
+
+		for (i = new_count; i < old_count; i++) {
+			if (!adapter->queue_irq[i])
+				continue;
+			ibmveth_disable_irq(adapter, i);
+			synchronize_irq(adapter->queue_irq[i]);
+		}
+
+		for (i = new_count; i < old_count; i++)
+			ibmveth_drain_rx_queue(adapter, i);
+
+		/*
+		 * netpoll still reaches these queues until the publish
+		 * below, and ibmveth_replenish_task() folds the same
+		 * counters under the queue's replenish_lock. Take it here
+		 * so the harvest stays the single writer it documents.
+		 */
+		for (i = new_count; i < old_count; i++) {
+			struct ibmveth_rx_q *rxq = &adapter->rx_queue[i];
+			unsigned long flags;
+
+			spin_lock_irqsave(&rxq->replenish_lock, flags);
+			ibmveth_update_rx_no_buffer(adapter, i);
+			spin_unlock_irqrestore(&rxq->replenish_lock, flags);
+		}
+
+		ibmveth_publish_num_rx_queues(adapter, new_count);
+		synchronize_net();
+
+		rc = netif_set_real_num_rx_queues(netdev, new_count);
+		if (rc) {
+			netdev_err(netdev, "Failed to set real RX queues to %d: %d\n",
+				   new_count, rc);
+			ibmveth_publish_num_rx_queues(adapter, old_count);
+			for (i = new_count; i < old_count; i++) {
+				int irq_rc;
+
+				ibmveth_replenish_task(adapter, i);
+				/* START: NAPI before PHYP unmask. */
+				napi_enable(&adapter->napi[i]);
+				irq_rc = ibmveth_enable_irq(adapter, i);
+				if (irq_rc) {
+					netdev_err(netdev,
+						   "Failed to re-enable IRQ for queue %d during scale-down rollback (rc=%d), scheduling reset\n",
+						   i, irq_rc);
+					schedule_work(&adapter->work);
+					continue;
+				}
+				ibmveth_restart_rx_queue(adapter, i);
+			}
+			return rc;
+		}
+
+		for (i = new_count; i < old_count; i++)
+			ibmveth_destroy_subordinate_rx_queue(adapter, i);
+	}
+
+	netdev_info(netdev, "Successfully resized to %u RX queues (incremental)\n",
+		    ibmveth_get_num_rx_queues(adapter));
+
+	if (firmware_has_feature(FW_FEATURE_CMO))
+		vio_cmo_set_dev_desired(adapter->vdev,
+					ibmveth_get_desired_dma(adapter->vdev));
+
+	return 0;
+
+cleanup_new_queues:
+	failed_queue = i;
+	if (failed_queue > old_count)
+		netdev_err(netdev,
+			   "Scale-up failed at queue %d, cleaning up queues %d-%d\n",
+			   failed_queue, old_count, failed_queue - 1);
+	else
+		netdev_err(netdev,
+			   "Scale-up failed at queue %d, nothing to clean up\n",
+			   failed_queue);
+
+	for (i = old_count; i < failed_queue; i++) {
+		ibmveth_disable_irq(adapter, i);
+		synchronize_irq(adapter->queue_irq[i]);
+	}
+
+	for (i = old_count; i < failed_queue; i++)
+		napi_disable(&adapter->napi[i]);
+
+	/* Same remask as scale-down: poll may have re-armed during disable. */
+	for (i = old_count; i < failed_queue; i++) {
+		ibmveth_disable_irq(adapter, i);
+		synchronize_irq(adapter->queue_irq[i]);
+	}
+
+	for (i = old_count; i < failed_queue; i++)
+		ibmveth_drain_rx_queue(adapter, i);
+
+	/* Drop the live count before freeing the half-added queues. */
+	ibmveth_publish_num_rx_queues(adapter, old_count);
+	synchronize_net();
+
+	for (i = old_count; i < failed_queue; i++)
+		ibmveth_destroy_subordinate_rx_queue(adapter, i);
+
+	/* Roll CMO desired back to the surviving queue count. */
+	if (firmware_has_feature(FW_FEATURE_CMO))
+		vio_cmo_set_dev_desired(adapter->vdev,
+					ibmveth_get_desired_dma(adapter->vdev));
+
+	netdev_warn(netdev, "Keeping %d queues after scale-up failure\n",
+		    old_count);
+	return rc;
+}
+
 /**
  * ibmveth_free_all_queues - Free all RX queues at once
  * @adapter: ibmveth adapter structure
@@ -2550,6 +3167,68 @@ static void ibmveth_get_channels(struct net_device *netdev,
 	channels->rx_count = ibmveth_get_num_rx_queues(adapter);
 }
 
+/**
+ * ibmveth_resize_rx_channels - Validate and apply a new RX queue count
+ * @adapter: ibmveth adapter
+ * @goal_rx: requested RX queue count
+ *
+ * Rejects rx > 1 without MQ firmware (-EOPNOTSUPP) and rx outside
+ * 1..IBMVETH_MAX_RX_QUEUES (-EINVAL). When RX resources are not live
+ * (!opened), only validate; do not allocate. When up, apply via
+ * ibmveth_resize_rx_queues_incremental().
+ *
+ * Return: 0 or negative errno
+ */
+static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,
+				      unsigned int goal_rx)
+{
+	struct net_device *netdev = adapter->netdev;
+	unsigned int old_rx = ibmveth_get_num_rx_queues(adapter);
+	int rxq_entries;
+	int rc;
+
+	if (goal_rx < 1 || goal_rx > IBMVETH_MAX_RX_QUEUES) {
+		netdev_err(netdev,
+			   "Invalid RX queue count %u (must be 1-%d)\n",
+			   goal_rx, IBMVETH_MAX_RX_QUEUES);
+		return -EINVAL;
+	}
+
+	/*
+	 * Check for a no-op before the capability gate. ethtool -L is
+	 * read-modify-write, so a TX-only request arrives carrying the
+	 * current RX count; gating first would fail those with
+	 * -EOPNOTSUPP once mq_fallback is set.
+	 */
+	if (goal_rx == old_rx)
+		return 0;
+
+	/*
+	 * Refuse any rx > 1, not just growth: once mq_fallback is set the
+	 * next open comes up single-queue, so an intermediate count could
+	 * not be honoured either, and accepting it would only repeat the
+	 * silent clamp at open. max_rx stays at the live count so that
+	 * read-modify-write TX-only requests still clear the core.
+	 */
+	if (goal_rx > 1 && (!adapter->multi_queue || adapter->mq_fallback)) {
+		netdev_err(netdev,
+			   "Cannot resize to %u RX queues: multi-queue mode not supported by firmware\n",
+			   goal_rx);
+		return -EOPNOTSUPP;
+	}
+
+	/* Down / failed-open: do not allocate. */
+	if (!adapter->opened)
+		return 0;
+
+	rxq_entries = adapter->rx_queue[0].num_slots;
+	rc = ibmveth_resize_rx_queues_incremental(adapter, goal_rx,
+						  rxq_entries);
+	if (rc)
+		netdev_err(netdev, "Failed to resize RX queues: %d\n", rc);
+	return rc;
+}
+
 static int ibmveth_set_channels(struct net_device *netdev,
 				struct ethtool_channels *channels)
 {
@@ -2558,10 +3237,15 @@ static int ibmveth_set_channels(struct net_device *netdev,
 		     goal = channels->tx_count;
 	int rc, i;
 
-	/* If ndo_open has not been called yet then don't allocate, just set
-	 * desired netdev_queue's and return
+	/* Validate RX (and resize when opened) before the down-path
+	 * early return so MQ/range errors are reported here. Publishing
+	 * the desired RX count and CMO while down is the next patch.
 	 */
-	if (!(netdev->flags & IFF_UP))
+	rc = ibmveth_resize_rx_channels(adapter, channels->rx_count);
+	if (rc)
+		return rc;
+
+	if (!adapter->opened)
 		return netif_set_real_num_tx_queues(netdev, goal);
 
 	/* We have IBMVETH_MAX_QUEUES netdev_queue's allocated
@@ -3291,8 +3975,6 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
 	struct net_device *netdev = dev_get_drvdata(&vdev->dev);
 	struct ibmveth_adapter *adapter;
 	struct iommu_table *tbl;
-	unsigned long ret;
-	int i, q;
 
 	tbl = get_iommu_table_base(&vdev->dev);
 
@@ -3301,38 +3983,8 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
 		return IOMMU_PAGE_ALIGN(IBMVETH_IO_ENTITLEMENT_DEFAULT, tbl);
 
 	adapter = netdev_priv(netdev);
-
-	/* One buffer list page per RX queue; filter list is shared. */
-	ret = IBMVETH_BUFF_LIST_SIZE * ibmveth_get_num_rx_queues(adapter) +
-	      IBMVETH_FILT_LIST_SIZE;
-	ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
-	/* add size of mapped tx buffers */
-	ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
-
-	/*
-	 * Pool metadata for queues 1+ is copied from queue 0 at open.
-	 * Always size from pool 0 x num_rx_queues (valid at probe before that
-	 * copy).
-	 */
-	for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
-		int rxqentries = 1;
-
-		for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
-			struct ibmveth_buff_pool *bpool =
-				&adapter->rx_buff_pool[0][i];
-
-			if (bpool->active)
-				ret += bpool->size *
-					IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
-			rxqentries += bpool->size;
-		}
-
-		/* add the size of the receive queue entries */
-		ret += IOMMU_PAGE_ALIGN(rxqentries *
-					sizeof(struct ibmveth_rx_q_entry), tbl);
-	}
-
-	return ret;
+	return ibmveth_desired_dma_for_rxqs(adapter,
+					    ibmveth_get_num_rx_queues(adapter));
 }
 
 static int ibmveth_set_mac_addr(struct net_device *dev, void *p)
-- 
2.50.1 (Apple Git-155)

[PATCH net-next v6 15/15] ibmveth: Complete set_channels down-path and mq_fallback max_rx cap

[Mingming Cao]

Patch 14 wires live ethtool -L rx. This patch completes the down-path
publish/rollback and get_channels() once mq_fallback is set.

When down: set TX queues first, then publish the desired RX count in
adapter->num_rx_queues and netdev->real_num_rx_queues (rx-N sysfs and
CMO desired change immediately). Do not allocate RX mappings, buffers
or IRQs while down. If RX set_real fails, restore the old TX real_num.

When up: resize RX, then the existing TX LTB
stop/alloc/set_real_num_tx/free/wake path. Skip that path when the TX
count is unchanged. If TX cannot reach the requested count, roll RX
back; that rollback is best-effort.

Validate tx_count before touching RX so a request the driver will
reject does not resize RX first. The ethtool core already range-checks
against the max_tx get_channels() reports; this is the driver's own
guard.

get_channels() always reports the live rx_count. When mq_fallback is
set it caps max_rx at that count. Understating rx_count would turn a
TX-only ethtool -L into a silent RX shrink. Capping max_rx blocks
growth in the core without misreporting what is configured.

i = old_tx before the TX alloc loop is readability; the old
for-initializer already defined i for the free walk.

Guard poll_controller() with adapter->opened so netpoll cannot walk
unallocated queue state while closed.

Signed-off-by: Mingming Cao <mmc@linux.ibm.com>
Reviewed-by: Dave Marquardt <davemarq@linux.ibm.com>
Tested-by: Shaik Abdulla <shaik.abdulla1@ibm.com>
---

Changes in v6:
- get_channels() caps max_rx at the live rx_count when mq_fallback
  is set; rx_count stays live
- roll back TX real_num if down-path RX set_real fails
- poll_controller() returns if !opened
- noted: rx > 1 reject once mq_fallback is set is patch 14

Changes in v5:
- set_channels / resize_rx_channels read via get_num_rx_queues()
- Series renumber: mailed v4 13/14 set_channels -> tip P15 (end; 14->15)
- set_channels: gate on adapter->opened (not IFF_UP) for live RX resize
  vs while-down stash
- Down-path RX stash also refreshes CMO (publish + set_real_num_rx)
- When up: resize RX then TX; roll RX back if TX cannot reach goal

Changes in v4:
- On !IFF_UP, stash num_rx_queues only after TX set succeeds; do not
  allocate live subordinate IRQs/buffers while down.
- Initialize i = old_tx on the TX adjust path.
- Always return rc from set_channels().
- Split from the resize-helper patch (same split as v3) while keeping
  a live caller of resize_rx_channels() in the previous patch.

 drivers/net/ethernet/ibm/ibmveth.c | 143 +++++++++++++++++++++++------
 1 file changed, 117 insertions(+), 26 deletions(-)

diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 5aef8a1f2c23..4cd00ff3d43e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -3156,15 +3156,24 @@ static void ibmveth_get_channels(struct net_device *netdev,
 				 struct ethtool_channels *channels)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
+	unsigned int rx_count = ibmveth_get_num_rx_queues(adapter);
 
 	channels->max_tx = ibmveth_real_max_tx_queues();
 	channels->tx_count = netdev->real_num_tx_queues;
 
-	if (adapter->multi_queue)
+	/*
+	 * Always report the live RX count. ethtool -L is read-modify-
+	 * write, so a TX-only request echoes rx_count back at us; an
+	 * understated value would be applied as a silent RX shrink.
+	 * mq_fallback instead caps max_rx at the live count, which
+	 * blocks growth in the core without misreporting what is
+	 * currently configured.
+	 */
+	channels->rx_count = rx_count;
+	if (adapter->multi_queue && !adapter->mq_fallback)
 		channels->max_rx = IBMVETH_MAX_RX_QUEUES;
 	else
-		channels->max_rx = 1;
-	channels->rx_count = ibmveth_get_num_rx_queues(adapter);
+		channels->max_rx = rx_count;
 }
 
 /**
@@ -3233,28 +3242,83 @@ static int ibmveth_set_channels(struct net_device *netdev,
 				struct ethtool_channels *channels)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(netdev);
-	unsigned int old = netdev->real_num_tx_queues,
-		     goal = channels->tx_count;
+	unsigned int old_rx = ibmveth_get_num_rx_queues(adapter);
+	unsigned int goal_rx = channels->rx_count;
+	unsigned int old_tx = netdev->real_num_tx_queues;
+	unsigned int goal_tx = channels->tx_count;
+	unsigned int want_tx = goal_tx;
+	bool rx_changed = false;
 	int rc, i;
 
-	/* Validate RX (and resize when opened) before the down-path
-	 * early return so MQ/range errors are reported here. Publishing
-	 * the desired RX count and CMO while down is the next patch.
-	 */
-	rc = ibmveth_resize_rx_channels(adapter, channels->rx_count);
+	if (goal_tx < 1 || goal_tx > ibmveth_real_max_tx_queues()) {
+		netdev_err(netdev,
+			   "Invalid TX queue count %u (must be 1-%u)\n",
+			   goal_tx, ibmveth_real_max_tx_queues());
+		return -EINVAL;
+	}
+
+	/* RX range / MQ checks live in ibmveth_resize_rx_channels(). */
+	rc = ibmveth_resize_rx_channels(adapter, goal_rx);
 	if (rc)
 		return rc;
 
-	if (!adapter->opened)
-		return netif_set_real_num_tx_queues(netdev, goal);
+	/* If RX resources are not live (never opened, or close+open failed
+	 * while IFF_UP stayed set), publish desired queue counts without
+	 * allocating.
+	 */
+	if (!adapter->opened) {
+		/* Apply TX first so a failure leaves the published RX
+		 * count unchanged.
+		 */
+		rc = netif_set_real_num_tx_queues(netdev, goal_tx);
+		if (rc)
+			return rc;
+
+		/* Publish desired RX count for next open() and refresh CMO;
+		 * do not allocate while down.
+		 */
+		if (goal_rx != ibmveth_get_num_rx_queues(adapter)) {
+			ibmveth_publish_num_rx_queues(adapter, goal_rx);
+			rc = netif_set_real_num_rx_queues(netdev, goal_rx);
+			if (rc) {
+				int tx_rc;
+
+				ibmveth_publish_num_rx_queues(adapter, old_rx);
+				tx_rc = netif_set_real_num_tx_queues(netdev,
+								     old_tx);
+				if (tx_rc)
+					netdev_err(netdev,
+						   "Failed to restore TX queues to %u after RX failure: %d\n",
+						   old_tx, tx_rc);
+				return rc;
+			}
+			if (firmware_has_feature(FW_FEATURE_CMO)) {
+				unsigned long dma;
+
+				dma = ibmveth_get_desired_dma(adapter->vdev);
+				vio_cmo_set_dev_desired(adapter->vdev, dma);
+			}
+		}
+		return 0;
+	}
+
+	if (goal_rx != old_rx)
+		rx_changed = true;
 
 	/* We have IBMVETH_MAX_QUEUES netdev_queue's allocated
 	 * but we may need to alloc/free the ltb's.
 	 */
+	if (goal_tx == old_tx)
+		return 0;
+
 	netif_tx_stop_all_queues(netdev);
 
-	/* Allocate any queue that we need */
-	for (i = old; i < goal; i++) {
+	/* Allocate any new TX LTBs. i starts at old_tx for the free walk
+	 * below when this loop body never runs (goal_tx == old_tx already
+	 * returned; goal_tx < old_tx is scale-down).
+	 */
+	i = old_tx;
+	for (; i < goal_tx; i++) {
 		if (adapter->tx_ltb_ptr[i])
 			continue;
 
@@ -3263,28 +3327,50 @@ static int ibmveth_set_channels(struct net_device *netdev,
 			continue;
 
 		/* if something goes wrong, free everything we just allocated */
-		netdev_err(netdev, "Failed to allocate more tx queues, returning to %d queues\n",
-			   old);
-		goal = old;
-		old = i;
+		netdev_err(netdev, "Failed to allocate more tx queues, returning to %u queues\n",
+			   old_tx);
+		goal_tx = old_tx;
+		old_tx = i;
 		break;
 	}
-	rc = netif_set_real_num_tx_queues(netdev, goal);
+	rc = netif_set_real_num_tx_queues(netdev, goal_tx);
 	if (rc) {
-		netdev_err(netdev, "Failed to set real tx queues, returning to %d queues\n",
-			   old);
-		goal = old;
-		old = i;
+		netdev_err(netdev, "Failed to set real tx queues, returning to %u queues\n",
+			   old_tx);
+		goal_tx = old_tx;
+		old_tx = i;
 	}
 	/* Free any that are no longer needed */
-	for (i = old; i > goal; i--) {
+	for (i = old_tx; i > goal_tx; i--) {
 		if (adapter->tx_ltb_ptr[i - 1])
 			ibmveth_free_tx_ltb(adapter, i - 1);
 	}
 
 	netif_tx_wake_all_queues(netdev);
 
-	return rc;
+	if (netdev->real_num_tx_queues != want_tx) {
+		if (rx_changed) {
+			/*
+			 * Only meaningful once RX is live. num_slots is
+			 * embedded in the adapter and outlives the DMA ring,
+			 * so reading it at function entry is safe but can
+			 * return a stale geometry from before the resize.
+			 */
+			int rxq_entries = adapter->rx_queue[0].num_slots;
+			int rb;
+
+			rb = ibmveth_resize_rx_queues_incremental(adapter,
+								  old_rx,
+								  rxq_entries);
+			if (rb)
+				netdev_err(netdev,
+					   "Failed to roll back RX queues to %u after TX failure: %d\n",
+					   old_rx, rb);
+		}
+		return rc ? rc : -ENOMEM;
+	}
+
+	return 0;
 }
 
 static const struct ethtool_ops netdev_ethtool_ops = {
@@ -3951,9 +4037,14 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
 static void ibmveth_poll_controller(struct net_device *dev)
 {
 	struct ibmveth_adapter *adapter = netdev_priv(dev);
-	unsigned int num = ibmveth_get_num_rx_queues(adapter);
+	unsigned int num;
 	int i;
 
+	if (!adapter->opened)
+		return;
+
+	num = ibmveth_get_num_rx_queues(adapter);
+
 	for (i = 0; i < num; i++)
 		ibmveth_replenish_task(adapter, i);
 
-- 
2.50.1 (Apple Git-155)

Re: [net-next,v6,01/15] ibmveth: Add MQ RX hypercall wrappers and call definitions

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 3 potential
issue(s) to consider:

Critical: 0 · High: 0 · Medium: 0 · Low: 3

- [Low] The commit message body contains a verbatim duplicated sentence.
- [Low] kernel-doc for the three new MQ hcall wrappers added by this
  patch is inconsistent about the H_FUNCTION (unsupported-hcall) return.
- [Low] The in-tree perf hypercall decoder is not updated for the three
  opcodes added by this patch.

--

ibmveth: Add MQ RX hypercall wrappers and call definitions

This adds the PAPR 11.20.00 hcall IDs H_REG_LOGICAL_LAN_QUEUE (0x49C),
H_ADD_LOGICAL_LAN_BUFFERS_QUEUE (0x4A0) and H_FREE_LOGICAL_LAN_QUEUE
(0x4A8), raising MAX_HCALL_OPCODE, and adds matching static inline
wrappers in ibmveth.h plus h_register_logical_lan_with_handle() so
queue 0 can capture the PHYP queue handle.

A few questions below, all on the changelog text and the kernel-doc.

>     Add ibmveth.h wrapper helpers (h_register_logical_lan_queue(),
>     h_add_logical_lan_buffers_queue(), h_free_logical_lan_queue()) with
>     argument ordering and return semantics matching the existing ibmveth
>     hcall wrappers. h_free_logical_lan_queue() uses plpar_hcall_norets()
>     like h_free_logical_lan(). Also add h_register_logical_lan_with_handle()
>     so queue 0 can capture the PHYP queue handle in MQ mode. Both new
>     registration wrappers use plpar_hcall() rather than plpar_hcall9(), so
>     they do not read unwritten stack slots. Both new
>     registration wrappers use plpar_hcall() rather than plpar_hcall9(), so
>     they do not read unwritten stack slots.

[Severity: Low]
This isn't a bug, but the sentence "Both new registration wrappers use
plpar_hcall() rather than plpar_hcall9(), so they do not read unwritten
stack slots." appears twice in a row here.  Was the second copy meant to
be dropped when the v6 rationale was folded into the changelog body?

> diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
> index dff90a7d7f70b..cb0ea53491e6c 100644
> --- a/arch/powerpc/include/asm/hvcall.h
> +++ b/arch/powerpc/include/asm/hvcall.h
> @@ -362,7 +362,11 @@
>  #define H_GUEST_DELETE		0x488
>  #define H_PKS_WRAP_OBJECT	0x490
>  #define H_PKS_UNWRAP_OBJECT	0x494
> -#define MAX_HCALL_OPCODE	H_PKS_UNWRAP_OBJECT
> +/* 0x498 reserved; 0x4A4 = H_FREE_LOGICAL_LAN_BUFFER_QUEUE (unused here) */
> +#define H_REG_LOGICAL_LAN_QUEUE 0x49C
> +#define H_ADD_LOGICAL_LAN_BUFFERS_QUEUE 0x4A0
> +#define H_FREE_LOGICAL_LAN_QUEUE 0x4A8
> +#define MAX_HCALL_OPCODE	H_FREE_LOGICAL_LAN_QUEUE

[Severity: Low]
Should tools/perf/scripts/python/powerpc-hcalls.py get entries for these
three opcodes too?

Its hcall_table currently tracks hvcall.h up to the previous
MAX_HCALL_OPCODE:

	1168: 'H_PKS_WRAP_OBJECT',
	1172: 'H_PKS_UNWRAP_OBJECT',
	# Platform-specific hcalls used by the Ultravisor
	61184: 'H_SVM_PAGE_IN',

There is nothing for 1180 (0x49C), 1184 (0x4A0) or 1192 (0x4A8), so
tracing the new MQ RX path with that script prints the raw numbers rather
than the names.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
> index d87713668ed30..08504d1cafd55 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.h
> +++ b/drivers/net/ethernet/ibm/ibmveth.h
> @@ -66,6 +66,145 @@ static inline long h_add_logical_lan_buffers(unsigned long unit_address,
>  			    desc5, desc6, desc7, desc8);
>  }
>  
> +/**
> + * h_register_logical_lan_queue - Register a subordinate receive queue
> + * @unit_address: Device unit address
> + * @buffer_list: DMA address of 4KB page for tracking registered buffers
> + * @rec_queue: Buffer descriptor of receive queue
> + * @queue_handle: Output queue handle on success (may be NULL)
> + * @irq: Output hypervisor IRQ number on success (may be NULL)
> + *
> + * Registers a subordinate receive queue with the hypervisor.
> + *
> + * Return:
> + *   H_SUCCESS (0) on success
> + *   H_PARAMETER if parameters are invalid

[Severity: Low]
Should H_FUNCTION be listed here as well?

H_REG_LOGICAL_LAN_QUEUE (0x49C) is new in PAPR 11.20.00, so firmware
predating it returns H_FUNCTION, and h_register_logical_lan_queue()
passes the hypervisor status straight back.  The later commit in this
series ("ibmveth: Add RX queue register helpers for MQ") does distinguish
that case:

	if (lpar_rc == H_FUNCTION)
		netdev_err(adapter->netdev,
			   "h_register_logical_lan_queue H_FUNCTION for queue %d (firmware MQ unsupported)\n",

so a caller reading only this Return: section has no documented way to
tell "firmware has no MQ support, fall back to single queue" from a hard
error.

> +static inline long
> +h_register_logical_lan_queue(unsigned long unit_address,
> +			     unsigned long buffer_list,
> +			     unsigned long rec_queue,
> +			     unsigned long *queue_handle,
> +			     unsigned long *irq)
> +{

[ ... ]

> + * Return:
> + *   H_SUCCESS - All buffers added successfully
> + *   H_PARAMETER - Invalid parameters
> + *   H_HARDWARE - Hardware error
> + *   H_FUNCTION - Firmware does not support this hcall
> + */
> +static inline long h_add_logical_lan_buffers_queue(unsigned long unit_address,

[ ... ]

> + * Deregisters and frees all structures associated with the subordinate queue.
> + *
> + * Return:
> + *   H_SUCCESS - Queue freed successfully
> + *   H_PARAMETER - Invalid parameters
> + *   H_HARDWARE - Hardware error
> + *   H_STATE - VIOA not in valid state
> + *   H_BUSY / H_LONG_BUSY_* - Resource busy, retry
> + */
> +static inline long h_free_logical_lan_queue(unsigned long unit_address,
> +					    unsigned long queue_handle)
> +{
> +	return plpar_hcall_norets(H_FREE_LOGICAL_LAN_QUEUE,
> +				  unit_address, queue_handle);
> +}

Same question for h_free_logical_lan_queue(): 0x4A8 is equally new, and
plpar_hcall_norets() returns H_FUNCTION unchanged on firmware that lacks
it, but only h_add_logical_lan_buffers_queue() documents that status.  Was
the difference between the three Return: sections intentional?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,03/15] ibmveth: Refactor RX resource allocation for MQ RX bring-up

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 3 potential
issue(s) to consider:

Critical: 0 · High: 1 (2) · Medium: 0 (3) · Low: 2

- [High] ibmveth_cleanup_rx_resources() and ibmveth_alloc_rx_queues()'s
  rollback newly store NULL into adapter->buffer_list_addr[i] (and…
- [Low] The patch is presented as a mechanical refactor ("Runtime
  behavior stays single-queue") but bundles a claimed memory-corruption…
- [Low] The comment added above the err_cleanup: rollback loop in
  ibmveth_alloc_rx_queues() states an invariant that the function's own…

Pre-existing issues:
- [High] ibmveth_open() calls napi_enable(&adapter->napi[0]) at entry and
  napi_disable() on every failure at the out: label (ibmveth.c:907),…
- [High] On the ibmveth_open() error path taken after
  h_register_logical_lan() has already succeeded (buffer-pool allocation…
- [Medium] TX long-term buffers are leaked when ibmveth_open() fails in
  the buffer-pool allocation loop or in request_irq(): the loop counter…
- [Medium] The TX long-term buffers keep the stale-DMA-handle pattern
  that this patch fixes for the RX/filter lists: ibmveth_free_tx_ltb()…
- [Medium] ibmveth_poll_controller() (ibmveth.c:1815-1819) calls
  ibmveth_replenish_task() directly from netpoll context, bypassing…

--

ibmveth: Refactor RX resource allocation for MQ RX bring-up

This moves the multicast filter list and the per-queue RX buffer lists and
RX rings out of ibmveth_open() into ibmveth_alloc_filter_list(),
ibmveth_alloc_rx_queues() and ibmveth_cleanup_rx_resources(), and wires
those helpers into open()/close().  Runtime behaviour stays single-queue,
and the helpers now NULL each pointer slot and zero each DMA handle as
they free.

(no Link: tags present in the commit header)

Before the diff, a question about the changelog itself.

[Severity: Low]
The commit message says:

  "The pointer and DMA-handle guards also close a baseline double
   free_page() and stale dma_unmap after a failed reopen followed by
   ndo_stop"

Is that baseline sequence actually reachable?  ibmveth_open() calls
napi_enable(&adapter->napi[0]) on entry and napi_disable() at its out:
label, so after a failed reopen NAPI is already disabled while
netif_running() is still true.  The following ndo_stop enters
ibmveth_close(), whose first action is another napi_disable():

drivers/net/ethernet/ibm/ibmveth.c:ibmveth_close() {
	napi_disable(&adapter->napi[0]);

	netif_tx_stop_all_queues(netdev);
	...
}

napi_disable_locked() leaves NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC set on
exit, and only napi_enable()/napi_complete() clear SCHED, so the second
call sits in:

net/core/dev.c:napi_disable_locked() {
	while (val & (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC)) {
		usleep_range(20, 200);
		val = READ_ONCE(n->state);
	}
	...
}

That is ahead of every free path in close(), so the double free_page()
described in the changelog does not appear to be reachable as written.

Separately, if these guards do fix memory corruption on stable kernels,
would it be better to split them into their own patch with a Fixes: tag
ahead of the refactor, rather than bundling them into a 200 line
restructuring?

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 7cb828b476c1b..d2dad9e6068e0 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c
> @@ -151,6 +151,193 @@ static unsigned int ibmveth_real_max_tx_queues(void)
>  	return min(n_cpu, IBMVETH_MAX_QUEUES);
>  }
>  

[ ... ]

> +		adapter->buffer_list_dma[i] =
> +			dma_map_single(dev, adapter->buffer_list_addr[i],
> +				       4096, DMA_BIDIRECTIONAL);
> +		if (dma_mapping_error(dev, adapter->buffer_list_dma[i])) {
> +			netdev_err(netdev,
> +				   "unable to map buffer list for queue %d\n",
> +				   i);
> +			free_page((unsigned long)adapter->buffer_list_addr[i]);
> +			adapter->buffer_list_addr[i] = NULL;
> +			adapter->buffer_list_dma[i] = 0;
> +			goto err_cleanup;
> +		}

[ ... ]

> +err_cleanup:
> +	/*
> +	 * Every failure path above releases what it had already allocated
> +	 * for queue i, so each index here is either fully constructed or
> +	 * fully empty. Do not unmap buffer_list_dma[] without the matching
> +	 * buffer_list_addr[] check: the two are only ever set together.
> +	 */

[Severity: Low]
Does this comment match what the dma_mapping_error() branch just above
leaves behind?  On that path buffer_list_addr[i] is freed and NULLed and
buffer_list_dma[i] is zeroed, but rx_queue[i].queue_addr from
dma_alloc_coherent() is still live, so index i arrives here partially
constructed rather than "fully constructed or fully empty".

The loop below is written with independent checks so it still frees the
ring, but a later reader who trusts "the two are only ever set together"
could collapse the two tests into one buffer_list_addr[] guard and leak
the coherent RX ring on exactly that failure path.  Could the comment be
reworded to describe the mixed state?

> +	for (; i >= 0; i--) {
> +		if (adapter->buffer_list_addr[i]) {
> +			dma_unmap_single(dev, adapter->buffer_list_dma[i],
> +					 4096, DMA_BIDIRECTIONAL);
> +			adapter->buffer_list_dma[i] = 0;
> +		}
> +		if (adapter->rx_queue[i].queue_addr) {
> +			dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
> +					  adapter->rx_queue[i].queue_addr,
> +					  adapter->rx_queue[i].queue_dma);
> +			adapter->rx_queue[i].queue_addr = NULL;
> +		}
> +		if (adapter->buffer_list_addr[i]) {
> +			free_page((unsigned long)adapter->buffer_list_addr[i]);
> +			adapter->buffer_list_addr[i] = NULL;
> +		}
> +	}
> +
> +	return -ENOMEM;
> +}

[ ... ]

> +static void
> +ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
> +{
> +	struct device *dev = &adapter->vdev->dev;
> +	int i;
> +
> +	netdev_dbg(adapter->netdev, "cleaning up %u RX queue(s)\n",
> +		   adapter->num_rx_queues);
> +
> +	for (i = 0; i < adapter->num_rx_queues; i++) {
> +		if (adapter->buffer_list_addr[i]) {
> +			dma_unmap_single(dev, adapter->buffer_list_dma[i],
> +					 4096, DMA_BIDIRECTIONAL);
> +			adapter->buffer_list_dma[i] = 0;
> +		}
> +
> +		if (adapter->rx_queue[i].queue_addr) {
> +			dma_free_coherent(dev, adapter->rx_queue[i].queue_len,
> +					  adapter->rx_queue[i].queue_addr,
> +					  adapter->rx_queue[i].queue_dma);
> +			adapter->rx_queue[i].queue_addr = NULL;
> +		}
> +
> +		if (adapter->buffer_list_addr[i]) {
> +			free_page((unsigned long)adapter->buffer_list_addr[i]);
> +			adapter->buffer_list_addr[i] = NULL;
> +		}
> +	}
> +}

[Severity: High]
This helper now stores NULL into buffer_list_addr[i], but
ibmveth_update_rx_no_buffer() still dereferences that slot with no check:

drivers/net/ethernet/ibm/ibmveth.c:ibmveth_update_rx_no_buffer() {
	__be64 *p = adapter->buffer_list_addr[0] + 4096 - 8;

	adapter->rx_no_buffer = be64_to_cpup(p);
}

Can that read fault after a failed reopen?  ibmveth_set_csum_offload(),
ibmveth_set_tso(), __ibmveth_change_mtu() and veth_pool_store() all do
ibmveth_close() followed by ibmveth_open() and only return the error if
the reopen fails, so netif_running() stays true while
ibmveth_open()'s unwind has already run ibmveth_cleanup_rx_resources()
and left buffer_list_addr[0] NULL.

The next netpoll poll then reaches the reader without going through NAPI:

ibmveth_poll_controller()
  ibmveth_replenish_task()
    ibmveth_update_rx_no_buffer()   /* NULL + 4088 */

ibmveth_free_buffer_pool() does not reset pool->available, so after a
close the "available < threshold" test in ibmveth_replenish_task() is
false for the fully posted pools and the loop falls through to
ibmveth_update_rx_no_buffer().

ibmveth_close() has the same ordering problem: it calls
ibmveth_update_rx_no_buffer() before the new cleanup helpers, so the
"failed reopen followed by ndo_stop" sequence in the changelog reaches
the unguarded reader.

In the baseline the same pointer was stale but still mapped, so this
turns a silent read into a NULL dereference.  A matching
"if (!adapter->buffer_list_addr[queue_index]) return;" guard, together
with an "if (!adapter->opened) return;" gate in
ibmveth_poll_controller(), only shows up in a later patch of this series
- could that guard come in this patch instead, so the NULL state is never
introduced without it?

> +
>  /* setup the initial settings for a buffer pool */
>  static void ibmveth_init_buffer_pool(struct ibmveth_buff_pool *pool,
>  				     u32 pool_index, u32 pool_size,

[ ... ]

> @@ -708,7 +855,7 @@ static int ibmveth_open(struct net_device *netdev)
>  				     rxq_desc.desc,
>  				     mac_address);
>  		rc = -ENONET;
> -		goto out_unmap_filter_list;
> +		goto out_free_tx_ltb;
>  	}

[Severity: Medium]
This isn't a bug introduced by this patch, but the TX long term buffers
keep the stale DMA handle pattern that the RX and filter helpers here
now avoid:

drivers/net/ethernet/ibm/ibmveth.c:ibmveth_free_tx_ltb() {
	dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
			 adapter->tx_ltb_size, DMA_TO_DEVICE);
	kfree(adapter->tx_ltb_ptr[idx]);
	adapter->tx_ltb_ptr[idx] = NULL;
}

tx_ltb_dma[idx] is never zeroed and the unmap is not gated on
tx_ltb_ptr[idx].  When the h_register_logical_lan() failure above takes
out_free_tx_ltb with the TX LTBs already allocated, a later
ibmveth_close() loops over all real_num_tx_queues and calls
ibmveth_free_tx_ltb() again, repeating dma_unmap_single() on a handle the
driver no longer owns.  Can that tear down a recycled TCE?

The changelog notes this is deferred, and the later "ibmveth: Refactor TX
resource allocation in open/close paths" patch adds the pointer check and
zeroes tx_ltb_dma[idx].

>  
>  	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
> @@ -735,8 +882,6 @@ static int ibmveth_open(struct net_device *netdev)
>  		goto out_free_buffer_pools;
>  	}
>  
> -	rc = -ENOMEM;
> -
>  	netdev_dbg(netdev, "initial replenish cycle\n");
>  	ibmveth_interrupt(netdev->irq, netdev);
>  
> @@ -752,26 +897,12 @@ static int ibmveth_open(struct net_device *netdev)
>  			ibmveth_free_buffer_pool(adapter,
>  						 &adapter->rx_buff_pool[0][i]);
>  	}

[Severity: High]
This is a pre-existing ordering issue, but the fall-through from
out_free_buffer_pools now releases more state than before.  The buffer
pool loop runs after h_register_logical_lan() has succeeded, and on
ibmveth_alloc_buffer_pool() failure this label falls through to
out_free_tx_ltb and out_free_filter_list, which now call
ibmveth_cleanup_rx_resources() and ibmveth_free_filter_list() with no
h_free_logical_lan() anywhere on the path.

The adjacent request_irq() failure path does perform the
deregistration first:

		do {
			lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
		} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));

Can PHYP keep DMA-writing the no-buffer counter in the last eight bytes
of the buffer list page after those pages have been unmapped and returned
to the page allocator?

The changelog acknowledges this predates the patch, and at the end of the
series ibmveth_open() allocates the pools before registering and every
post-registration failure goes through ibmveth_free_all_queues(), which
issues h_free_logical_lan().

> -out_unmap_filter_list:
> -	dma_unmap_single(dev, adapter->filter_list_dma, 4096,
> -			 DMA_BIDIRECTIONAL);
> -
>  out_free_tx_ltb:
> -	while (--i >= 0) {
> +	while (--i >= 0)
>  		ibmveth_free_tx_ltb(adapter, i);
> -	}

[Severity: Medium]
This is a pre-existing problem rather than something this patch adds, but
does this loop ever run when the failure came from the buffer pool loop
or from request_irq()?  Both out_free_buffer_pools and out_free_tx_ltb
consume the same counter i, and i has been reused as the buffer pool
index by then:

out_free_buffer_pools:
	while (--i >= 0) {		/* drains i to -1 */
		...
	}
out_free_tx_ltb:
	while (--i >= 0)		/* starts at -2, never runs */
		ibmveth_free_tx_ltb(adapter, i);

So every tx_ltb_ptr[]/tx_ltb_dma[] allocated in this open() stays
allocated and mapped.  The changelog mentions the shared i leak, and the
later TX refactor patch gives the rollback its own counter.

> -
> -out_unmap_buffer_list:
> -	dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
> -			 DMA_BIDIRECTIONAL);
> -out_free_queue_mem:
> -	dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
> -			  adapter->rx_queue[0].queue_addr,
> -			  adapter->rx_queue[0].queue_dma);
> +	ibmveth_cleanup_rx_resources(adapter);
>  out_free_filter_list:
> -	free_page((unsigned long)adapter->filter_list_addr);
> -out_free_buffer_list:
> -	free_page((unsigned long)adapter->buffer_list_addr[0]);
> +	ibmveth_free_filter_list(adapter);
>  out:
>  	napi_disable(&adapter->napi[0]);
>  	return rc;

[Severity: High]
This is a pre-existing imbalance and not introduced here, but it is the
path the changelog leans on, so it seems worth raising.

Every open failure leaves NAPI disabled while the device is still
administratively up, because ibmveth_set_csum_offload(), ibmveth_set_tso(),
__ibmveth_change_mtu() and veth_pool_store() just return the reopen error.
ibmveth_close() then calls napi_disable() a second time with no
intervening napi_enable(), and napi_disable_locked() spins:

net/core/dev.c:napi_disable_locked() {
	while (val & (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC)) {
		usleep_range(20, 200);
		val = READ_ONCE(n->state);
	}
	...
}

Nothing clears SCHED at that point, and the loop runs inside dev_close()
with rtnl_lock() and the netdev lock held.  Can an ifdown after a failed
reopen wedge every later RTNL operation?

The series later drops this by returning early from close() when
adapter->opened is false and handling NAPI in
ibmveth_cleanup_rx_interrupts().

> @@ -780,7 +911,6 @@ static int ibmveth_open(struct net_device *netdev)
>  static int ibmveth_close(struct net_device *netdev)
>  {
>  	struct ibmveth_adapter *adapter = netdev_priv(netdev);
> -	struct device *dev = &adapter->vdev->dev;
>  	long lpar_rc;
>  	int i;
>  
> @@ -805,17 +935,8 @@ static int ibmveth_close(struct net_device *netdev)
>  
>  	ibmveth_update_rx_no_buffer(adapter);
>  
> -	dma_unmap_single(dev, adapter->buffer_list_dma[0], 4096,
> -			 DMA_BIDIRECTIONAL);
> -	free_page((unsigned long)adapter->buffer_list_addr[0]);
> -
> -	dma_unmap_single(dev, adapter->filter_list_dma, 4096,
> -			 DMA_BIDIRECTIONAL);
> -	free_page((unsigned long)adapter->filter_list_addr);
> -
> -	dma_free_coherent(dev, adapter->rx_queue[0].queue_len,
> -			  adapter->rx_queue[0].queue_addr,
> -			  adapter->rx_queue[0].queue_dma);
> +	ibmveth_cleanup_rx_resources(adapter);
> +	ibmveth_free_filter_list(adapter);

[Severity: Medium]
This isn't a bug introduced by the patch either, but the failure mode
changes here.  What serializes these frees against netpoll?

ibmveth_poll_controller() calls ibmveth_replenish_task() directly, so it
never goes through napi_schedule()/poll and the napi_disable() at the top
of ibmveth_close() does not exclude it:

static void ibmveth_poll_controller(struct net_device *dev)
{
	ibmveth_replenish_task(netdev_priv(dev));
	ibmveth_interrupt(dev->irq, dev);
}

The teardown path holds only rtnl and the netdev lock, which netpoll does
not take, and the internal close()/open() callers (ethtool csum and tso,
change_mtu, veth_pool_store) never call netpoll_poll_disable(), which only
dev_close()/dev_open() do.  With the slots now NULLed, a racing netpoll
poll dereferences NULL rather than reading a stale but mapped page.

The series later adds the adapter->opened gate in
ibmveth_poll_controller(), the per-queue replenish_lock, and the NULL
check in ibmveth_update_rx_no_buffer().

>  
>  	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
>  		if (adapter->rx_buff_pool[0][i].active)

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,04/15] ibmveth: Refactor buffer pool management for per-queue MQ RX

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider:

Critical: 0 · High: 0 (1) · Medium: 0 · Low: 2

- [Low] The refactor silently repairs a pre-existing DMA-coherent memory
  leak in ibmveth_open() without a Fixes: tag.
- [Low] The err_cleanup comment in ibmveth_alloc_rx_queues()
  (drivers/net/ethernet/ibm/ibmveth.c:280-285) asserts an invariant the…

Pre-existing issues:
- [High] In ibmveth_open() at this commit, the new failure branch for
  ibmveth_alloc_buffer_pools() (`goto out_free_tx_ltb`,…

--

ibmveth: Refactor buffer pool management for per-queue MQ RX

This moves the five RX buffer pools from adapter scope to
rx_buff_pool[queue][pool] and adds ibmveth_alloc_buffer_pools() /
ibmveth_free_buffer_pools() plus per-queue helpers, wiring them into
open() and close(). Queue 0 stays the template for pool geometry and
activation; runtime is still single-queue.

A few questions below.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index d2dad9e6068e0..37a6d13e603ea 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... helper additions snipped ... ]

> @@ -858,16 +996,9 @@ static int ibmveth_open(struct net_device *netdev)
>  		goto out_free_tx_ltb;
>  	}
>  
> -	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
> -		if (!adapter->rx_buff_pool[0][i].active)
> -			continue;
> -		if (ibmveth_alloc_buffer_pool(&adapter->rx_buff_pool[0][i])) {
> -			netdev_err(netdev, "unable to alloc pool\n");
> -			adapter->rx_buff_pool[0][i].active = 0;
> -			rc = -ENOMEM;
> -			goto out_free_buffer_pools;
> -		}
> -	}
> +	rc = ibmveth_alloc_buffer_pools(adapter);
> +	if (rc)
> +		goto out_free_tx_ltb;

[Severity: High]

This isn't a bug introduced by this patch, but the rewritten jump lands on
an unwind that frees memory PHYP was just told about, without any
deregistration hypercall.

At this point ibmveth_register_logical_lan() has already returned
H_SUCCESS. The chosen label then does:

out_free_tx_ltb:
	while (--i >= 0)
		ibmveth_free_tx_ltb(adapter, i);
	ibmveth_cleanup_rx_resources(adapter);
out_free_filter_list:
	ibmveth_free_filter_list(adapter);

and ibmveth_cleanup_rx_resources() dma_unmap_single()s buffer_list_dma[0],
free_page()s buffer_list_addr[0] and dma_free_coherent()s
rx_queue[0].queue_addr, while ibmveth_free_filter_list() releases the
filter page. Those are exactly the three regions handed to the hypervisor
by the registration.

The neighbouring request_irq() failure path does the opposite order:

		do {
			lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
		} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));

		goto out_free_buffer_pools;

Can PHYP write into the buffer-list page after it has gone back to the page
allocator on the pool-failure path? It stays an independent writer of that
page for the per-page no-buffer drop count that
ibmveth_update_rx_no_buffer() reads back.

The same ordering exists in the parent commit, and the rest of this series
does address it: the later patch that refactors TX resource allocation
issues h_free_logical_lan() before the RX DMA teardown, and the patch
adding the RX queue register helpers moves ibmveth_alloc_buffer_pools()
ahead of registration so this window disappears. Would it be worth
ordering the pool allocation before registration here instead, so the
intermediate commits in the series do not carry the window?

>  
>  	netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
>  	rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
>  			 netdev);
> @@ -892,11 +1023,7 @@ static int ibmveth_open(struct net_device *netdev)
>  	return 0;
>  
>  out_free_buffer_pools:
> -	while (--i >= 0) {
> -		if (adapter->rx_buff_pool[0][i].active)
> -			ibmveth_free_buffer_pool(adapter,
> -						 &adapter->rx_buff_pool[0][i]);
> -	}
> +	ibmveth_free_buffer_pools(adapter);
>  out_free_tx_ltb:
>  	while (--i >= 0)
>  		ibmveth_free_tx_ltb(adapter, i);

[Severity: Low]

The commit message describes the repair this hunk performs and then
declines a tag:

    Pulling the pool loop out has one side effect worth naming: it no
    longer consumes open()'s loop index, so a pool failure reaches
    out_free_tx_ltb with i still at real_num_tx_queues and the TX LTBs
    actually get freed. The shared index that swallowed them was
    pre-existing, so there is no standalone Fixes: tag;

Should this carry a Fixes: tag for the commit that added out_free_tx_ltb
and its second walk over the same index?

Before this patch the old loop at out_free_buffer_pools drove i down to -1,
so the following while (--i >= 0) at out_free_tx_ltb evaluated -2 and freed
nothing. Every failed ndo_open after a successful TX LTB allocation
therefore leaked real_num_tx_queues buffers, one dma_alloc_coherent() of
adapter->tx_ltb_size each, reachable both from an
ibmveth_alloc_buffer_pool() failure and from a request_irq() failure.

Without a tag, stable trees keep the leak and the repair is only reachable
by picking up this refactor.

One further note on code this patch does not touch, but which the
immediately preceding patch in the series added:

[Severity: Low]

Does the err_cleanup comment in ibmveth_alloc_rx_queues() state an
invariant the function actually holds?

	/*
	 * Every failure path above releases what it had already allocated
	 * for queue i, so each index here is either fully constructed or
	 * fully empty. Do not unmap buffer_list_dma[] without the matching
	 * buffer_list_addr[] check: the two are only ever set together.
	 */

The buffer-list mapping failure path leaves rx_queue[i].queue_addr
allocated:

		if (dma_mapping_error(dev, adapter->buffer_list_dma[i])) {
			...
			free_page((unsigned long)adapter->buffer_list_addr[i]);
			adapter->buffer_list_addr[i] = NULL;
			adapter->buffer_list_dma[i] = 0;
			goto err_cleanup;
		}

so index i arrives at err_cleanup partially constructed, and it is the
per-pointer if (adapter->rx_queue[i].queue_addr) check that frees the ring.
Nothing leaks today, but the first half of the comment contradicts the
second half. Could the wording be changed to say the cleanup loop frees by
pointer presence rather than claiming each index is all-or-nothing?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,05/15] ibmveth: Refactor RX interrupt control for MQ RX queues

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 5 potential
issue(s) to consider:

Critical: 0 · High: 0 (1) · Medium: 3 · Low: 2

- [Medium] A user-triggerable rtnl hang fix is bundled into a 400-line
  refactor with an incorrect justification for omitting a Fixes: tag.
- [Medium] ibmveth_poll() re-arms the PHYP interrupt
  (ibmveth_enable_irq(), ibmveth.c:2123) only *after*…
- [Medium] Teardown frees Linux IRQ handlers (and, on the MQ paths,
  disposes subordinate virq mappings) without ensuring PHYP delivery was…
- [Low] Two permanent behavioural changes to ibmveth_close() are absent
  from the commit message: (a) ibmveth_update_rx_no_buffer() is moved…
- [Low] Claimed: ibmveth_schedule_rx_queue()'s kernel-doc documents an
  out-of-range index as an ordinary `return false`, while the body…

Pre-existing issues:
- [High] Claimed: on the ibmveth_alloc_buffer_pools() failure path,
  ibmveth_open() jumps to out_free_tx_ltb without h_free_logical_lan(),…

--

ibmveth: Refactor RX interrupt control for MQ RX queues

This patch introduces queue-indexed RX interrupt helpers (toggle/enable/
disable, setup/cleanup, schedule) and wires them into open(), close(),
poll(), the IRQ handler and the process-context RX kick sites. It also adds
two new adapter flags, opened and rx_irq_setup, that gate close/cleanup
work. Runtime behaviour stays single-queue.

A few questions below.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 37a6d13e603ea..335712faaa42a 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... ]

> @@ -338,6 +340,320 @@ ibmveth_cleanup_rx_resources(struct ibmveth_adapter *adapter)
>  	}
>  }
>  
> +/**
> + * ibmveth_toggle_irq - Common helper to enable/disable queue interrupts

[ ... ]

> +		/*
> +		 * H_PARAMETER is ambiguous (already in requested state vs bad
> +		 * args). Fold only on disable as an idempotent mask. On enable
> +		 * keep it an error so a stuck-masked queue stays visible to
> +		 * poll/resize recovery.
> +		 */
> +		if (h_rc == H_PARAMETER && !enable) {
> +			dev_warn_ratelimited(&adapter->netdev->dev,
> +					     "H_VIOCTL %s IRQ returned H_PARAMETER for queue %d (hwirq=%lu)\n",
> +					     action, queue_index, hwirq);
> +			return 0;
> +		}
> +	}
> +
> +	if (h_rc) {
> +		dev_err_ratelimited(&adapter->netdev->dev,
> +				    "Failed to %s IRQ for queue %d, rc=0x%lx\n",
> +				    action, queue_index, h_rc);
> +		return -EIO;
> +	}
> +	return 0;
> +}

[ ... ]

> +static int
> +ibmveth_setup_rx_interrupts(struct ibmveth_adapter *adapter)
> +{

[ ... ]

> +	if (adapter->multi_queue && num > 1) {
> +		for (i = 0; i < num; i++) {
> +			rc = ibmveth_enable_irq(adapter, i);
> +			if (rc) {
> +				netdev_err(netdev,
> +					   "Failed to enable IRQ for queue %d, rc=%d\n",
> +					   i, rc);
> +				for (; i >= 0; i--) {
> +					ibmveth_disable_irq(adapter, i);
> +					synchronize_irq(adapter->queue_irq[i]);
> +				}
> +				rc = -EIO;
> +				goto err_disable_napi;
> +			}
> +		}
> +	}

[ ... ]

> +err_disable_napi:
> +	/* STOP: remask after napi_disable; an in-flight poll can re-arm. */
> +	for (i = 0; i < num; i++)
> +		napi_disable(&adapter->napi[i]);
> +	for (i = 0; i < num; i++) {
> +		if (!adapter->queue_irq[i])
> +			continue;
> +		ibmveth_disable_irq(adapter, i);
> +		synchronize_irq(adapter->queue_irq[i]);
> +	}
> +	for (i = 0; i < num; i++) {
> +		if (adapter->queue_irq[i])
> +			free_irq(adapter->queue_irq[i], &adapter->napi[i]);
> +	}
> +	goto err_dispose_mappings;

[ ... ]

> +static void
> +ibmveth_cleanup_rx_interrupts(struct ibmveth_adapter *adapter)
> +{
> +	int i;
> +
> +	if (!adapter->rx_irq_setup)
> +		return;
> +
> +	for (i = 0; i < adapter->num_rx_queues; i++) {
> +		if (!adapter->queue_irq[i])
> +			continue;
> +		ibmveth_disable_irq(adapter, i);
> +		synchronize_irq(adapter->queue_irq[i]);
> +	}
> +
> +	for (i = 0; i < adapter->num_rx_queues; i++)
> +		napi_disable(&adapter->napi[i]);
> +
> +	for (i = 0; i < adapter->num_rx_queues; i++) {
> +		if (!adapter->queue_irq[i])
> +			continue;
> +		ibmveth_disable_irq(adapter, i);
> +		synchronize_irq(adapter->queue_irq[i]);
> +	}
> +
> +	for (i = 0; i < adapter->num_rx_queues; i++) {
> +		if (adapter->queue_irq[i])
> +			free_irq(adapter->queue_irq[i], &adapter->napi[i]);
> +	}

[Severity: Medium]
The commit message says cleanup "remasks and synchronizes again after it
because an in-flight poll can re-arm". Does the second remask really close
that window?

ibmveth_poll() re-arms only after NAPI ownership has already been dropped:

	if (!napi_complete_done(napi, frames_processed))
		goto out;
	...
	rc = ibmveth_enable_irq(adapter, 0);

napi_disable() waits for NAPIF_STATE_SCHED / NAPIF_STATE_NPSVC to clear, not
for the poll function to return, so this interleaving looks possible:

CPU1 (softirq)                    CPU0 (ibmveth_close)
ibmveth_poll()
  napi_complete_done()
    /* SCHED cleared */
  <delayed>
                                  ibmveth_cleanup_rx_interrupts()
                                    disable_irq + synchronize_irq
                                    napi_disable()  /* returns at once */
                                    disable_irq + synchronize_irq
                                    free_irq()
  ibmveth_enable_irq(adapter, 0)

That leaves PHYP delivery unmasked on a queue with no handler installed
until h_free_logical_lan() runs. The synchronize_net() added in close()
runs after free_irq(), so it cannot cover this either, and the
err_disable_napi path above has the same shape.

The ordering is unchanged at the end of the series; the poll_stopping()
check added later is passed by CPU1 before CPU0 sets the teardown state.
Would arming the interrupt before napi_complete_done(), or gating the arm
on a teardown flag ordered with the mask, work better here?

[Severity: Medium]
Both ibmveth_disable_irq() calls in this function discard their return
value, and free_irq() below then runs unconditionally. In close(),
h_free_logical_lan() now runs after free_irq(), where the baseline freed
the LAN first and only then removed the handler.

If the mask hcall did not take effect, can PHYP deliver an interrupt in the
interval between free_irq() and h_free_logical_lan(), with no handler
installed? synchronize_irq() only drains a handler that is already running;
it does not stop the hypervisor source.

Related: ibmveth_toggle_irq() folds H_PARAMETER into success on the disable
path, while its own comment says H_PARAMETER may mean bad arguments rather
than "already in requested state", so teardown can treat a rejected mask as
a completed one. The MQ rollback loop in ibmveth_setup_rx_interrupts() has
the same unchecked-remask-then-free_irq pattern. I could not demonstrate
these hcalls failing in practice, so this one may be theoretical.

> +
> +	ibmveth_dispose_subordinate_irq_mappings(adapter);
> +
> +	/* Queue 0 uses netdev->irq; leave queue_irq[0] for next open. */
> +	adapter->rx_irq_setup = false;
> +}
> +
> +/**
> + * ibmveth_schedule_rx_queue - Mask PHYP IRQ and schedule NAPI for one RX queue
> + * @adapter: ibmveth adapter structure
> + * @qindex: RX queue index
> + *
> + * Shared by the IRQ handler and process-context kick sites (open, resume,
> + * pool sysfs, poll_controller).
> + *
> + * Return: true if napi_schedule_prep() succeeded and NAPI was scheduled.
> + * Mask is attempted in that case; a failed disable_irq() is logged by the
> + * helper and does not change the return (queue may still be unmasked).
> + * false if the index is out of range or prep failed (including NAPI
> + * already scheduled).
> + */
> +static bool ibmveth_schedule_rx_queue(struct ibmveth_adapter *adapter,
> +				      int qindex)
> +{
> +	struct napi_struct *napi = &adapter->napi[qindex];
> +
> +	if (WARN_ON(qindex < 0 || qindex >= adapter->num_rx_queues))
> +		return false;

[Severity: Low]
This isn't a bug, but the kdoc and the body disagree. The kdoc describes an
out-of-range index as an ordinary return:

	 * false if the index is out of range or prep failed (including NAPI
	 * already scheduled).

while the body splats on it. ibmveth_interrupt() documents the opposite for
the same condition:

	/*
	 * Quiet on out-of-range: teardown can leave a residual IRQ after the
	 * live count drops. Do not WARN-storm; return IRQ_NONE until free_irq.
	 */

A later patch in the series changes this helper to WARN only on qindex < 0
and to return false quietly when qindex is past the live queue count. Could
that form be used from the start?

The napi pointer is also formed from qindex before the range check; it is
never dereferenced before the check, but computing it afterwards would read
more clearly.

> +
> +	/*
> +	 * Only mask PHYP when NAPI will run. Masking on prep failure can
> +	 * race a completing poll that already re-enabled the queue, leaving
> +	 * NAPI idle with the IRQ masked (TX works, RX stalls) until reload.
> +	 * Storm prevention on teardown remains in cleanup/disable paths.
> +	 */
> +	if (napi_schedule_prep(napi)) {
> +		/* Failure is already logged with the hcall rc by the helper. */
> +		ibmveth_disable_irq(adapter, qindex);
> +		__napi_schedule(napi);
> +		return true;
> +	}
> +	return false;
> +}
> +

[ ... ]

> @@ -1000,24 +1315,20 @@ static int ibmveth_open(struct net_device *netdev)
>  	if (rc)
>  		goto out_free_tx_ltb;
>  

[Severity: High]
This isn't a bug introduced by this patch, but the buffer-pool allocation
failure just above (ibmveth_alloc_buffer_pools() -> goto out_free_tx_ltb)
still skips h_free_logical_lan(), unlike the setup_rx_interrupts() failure
right below it.

The out_free_tx_ltb path continues into ibmveth_cleanup_rx_resources(),
which unmaps and frees the buffer list, the filter list and the RX ring
while the logical LAN registration is still live, so PHYP keeps DMA
addresses for freed pages.

Does the new opened gate in ibmveth_close() also remove the last site that
could have deregistered that residue? A later ndo_stop now returns 0
without calling h_free_logical_lan().

Two later patches in this series rework this ladder, first routing the pool
failure through a path that loops on h_free_logical_lan() and then moving
pool allocation ahead of registration, so the window does not survive the
series. Would it be worth ordering those ahead of this one so this commit
does not carry it?

> -	netdev_dbg(netdev, "registering irq 0x%x\n", netdev->irq);
> -	rc = request_irq(netdev->irq, ibmveth_interrupt, 0, netdev->name,
> -			 netdev);
> -	if (rc != 0) {
> -		netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
> -			   netdev->irq, rc);
> +	rc = ibmveth_setup_rx_interrupts(adapter);
> +	if (rc) {
>  		do {
>  			lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
>  		} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
> -
>  		goto out_free_buffer_pools;
>  	}
>  
>  	netdev_dbg(netdev, "initial replenish cycle\n");
> -	ibmveth_interrupt(netdev->irq, netdev);
> +	ibmveth_schedule_rx_queue(adapter, 0);
>  
>  	netif_tx_start_all_queues(netdev);
>  
> +	adapter->opened = true;
>  	netdev_dbg(netdev, "open complete\n");
>  
>  	return 0;

[ ... ]

> @@ -1041,27 +1351,32 @@ static int ibmveth_close(struct net_device *netdev)
>  	long lpar_rc;
>  	int i;
>  
> -	netdev_dbg(netdev, "close starting\n");
> +	/* Gate on opened, not IFF_UP: pool_store/change_mtu close+open can
> +	 * leave IFF_UP set after a failed reopen.
> +	 */
> +	if (!adapter->opened)
> +		return 0;

[Severity: Medium]
The commit message says:

  "Opened / rx_irq_setup also closes a pre-existing hang: after a
   failed reopen, a later ndo_stop used to napi_disable and free_irq
   a second time (rtnl spin + already-free IRQ). That depends on the
   helpers in this patch, so there is no standalone Fixes: tag."

Is that dependency real? This gate reads only adapter->opened, and the
matching "adapter->opened = true" at the end of ibmveth_open() does not
reference ibmveth_toggle_irq(), ibmveth_setup_rx_interrupts() or
ibmveth_cleanup_rx_interrupts() either.

The hang itself is reachable in the baseline: ibmveth_open() calls
napi_enable() on entry and napi_disable() on its out: path, and
ibmveth_close() starts with napi_disable(). veth_pool_store(),
ibmveth_change_mtu(), ibmveth_set_csum_offload() and ibmveth_set_tso() call
ibmveth_close() then ibmveth_open() directly and do not clear IFF_UP when
the reopen fails, so a later "ip link set dev X down" enters
ibmveth_close() and the second napi_disable() spin-waits forever with rtnl
held.

Could the opened flag and this gate be split into their own patch with a
Fixes: tag, so the fix can be backported without the refactor?

>  
> -	napi_disable(&adapter->napi[0]);
> +	adapter->opened = false;
> +
> +	netdev_dbg(netdev, "close starting\n");
>  
>  	netif_tx_stop_all_queues(netdev);
>  
> -	h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
> +	ibmveth_cleanup_rx_interrupts(adapter);
> +	/* Wait for softirq/poll that already passed shutdown checks. */
> +	synchronize_net();
>  
> +	ibmveth_update_rx_no_buffer(adapter);
> +	/* Full LAN teardown (subordinates arrive with register helpers). */
>  	do {
>  		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
>  	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));

[Severity: Low]
Two permanent changes to ibmveth_close() are not described in the commit
message.

ibmveth_update_rx_no_buffer() moves from after h_free_logical_lan() and
free_irq() to before h_free_logical_lan(). It reads the absolute counter
that the hypervisor writes at the end of the buffer list page, and the
hypervisor still owns that page until h_free_logical_lan() returns, so the
final rx_no_buffer / rx_errors value can now miss drops recorded after the
read. Was the reordering intentional?

The new synchronize_net() in the ndo_stop path is mentioned only in the v5
changelog below the --- separator, which does not land in git history.
Could it be described in the commit message as well?

> -
>  	if (lpar_rc != H_SUCCESS) {
> -		netdev_err(netdev, "h_free_logical_lan failed with %lx, "
> -			   "continuing with close\n", lpar_rc);
> +		netdev_err(adapter->netdev,
> +			   "h_free_logical_lan failed with %lx, continuing\n",
> +			   lpar_rc);
>  	}
> -
> -	free_irq(netdev->irq, netdev);
> -
> -	ibmveth_update_rx_no_buffer(adapter);
> -
>  	ibmveth_free_buffer_pools(adapter);
>  	ibmveth_cleanup_rx_resources(adapter);
>  	ibmveth_free_filter_list(adapter);

[ ... ]

> @@ -1805,15 +2120,14 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)
>  	/* We think we are done - reenable interrupts,
>  	 * then check once more to make sure we are done.
>  	 */
> -	lpar_rc = h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_ENABLE);
> -	if (WARN_ON(lpar_rc != H_SUCCESS)) {
> +	rc = ibmveth_enable_irq(adapter, 0);
> +	if (rc) {
>  		schedule_work(&adapter->work);
>  		goto out;
>  	}

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,06/15] ibmveth: Refactor TX resource allocation in open/close paths

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider:

Critical: 0 · High: 0 · Medium: 1 · Low: 0 (2)

- [Medium] The patch silently repairs a live mainline bug while
  presenting itself as a TX refactor.

Pre-existing issues:
- [Low] Concern (assessed as a FALSE POSITIVE): the new `if
  (unlikely(!adapter->tx_ltb_ptr[queue_num]))` guard in…
- [Low] Concern (assessed as a FALSE POSITIVE for the final series
  state): ibmveth_alloc_tx_resources() calls ibmveth_allocate_tx_ltb()…

--

ibmveth: Refactor TX resource allocation in open/close paths

This moves the per-queue TX LTB allocation and free out of ibmveth_open()
and ibmveth_close() into ibmveth_alloc_tx_resources() and
ibmveth_free_tx_resources(), moves the TX allocation to the end of open()
after RX interrupt setup, and adds a tx_ltb_ptr[] check to
ibmveth_start_xmit(). It also reworks the open() failure unwind labels.

A few questions below.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 335712faaa42a..7a420e1a41d52 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c
> @@ -1201,12 +1201,27 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
>  
>  static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
>  {
> -	dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
> -			 adapter->tx_ltb_size, DMA_TO_DEVICE);
> -	kfree(adapter->tx_ltb_ptr[idx]);
> +	void *ltb = adapter->tx_ltb_ptr[idx];
> +	dma_addr_t dma = adapter->tx_ltb_dma[idx];
> +
> +	if (!ltb)
> +		return;
> +
> +	/*
> +	 * Clear the slot before releasing it. start_xmit() tests
> +	 * tx_ltb_ptr[idx] to decide whether the LTB is usable.
> +	 */
>  	adapter->tx_ltb_ptr[idx] = NULL;
> +	adapter->tx_ltb_dma[idx] = 0;
> +
> +	dma_unmap_single(&adapter->vdev->dev, dma, adapter->tx_ltb_size,
> +			 DMA_TO_DEVICE);
> +	kfree(ltb);
>  }
>  
> +/* Caller must ensure tx_ltb_ptr[idx] is NULL. open() runs on
> + * probe-zeroed slots; set_channels() skips populated indices.
> + */
>  static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
>  {
>  	adapter->tx_ltb_ptr[idx] = kzalloc(adapter->tx_ltb_size,

[Severity: Low]
This is a pre-existing style point rather than something this patch breaks,
but the new kdoc on ibmveth_allocate_tx_ltb() states an invariant that the
helper itself never checks:

  /* Caller must ensure tx_ltb_ptr[idx] is NULL. ...

and ibmveth_alloc_tx_resources() below calls it unconditionally for every
index in [0, real_num_tx_queues). If a slot were ever populated on entry,
the kzalloc() would overwrite the old pointer and leak both the 64 KiB
buffer and its streaming DMA mapping.

Tracing the callers, the only way to reach open() with a populated slot was
ibmveth_set_channels() gating on IFF_UP instead of adapter->opened, and a
later patch in this series makes set_channels() return early on !opened
before it touches the LTBs, so no reachable caller can present a populated
index. Would a WARN_ON_ONCE(adapter->tx_ltb_ptr[idx]) (or just an early
return) be worth adding so the documented precondition is enforced rather
than only described?

[ ... ]

> @@ -1308,24 +1359,24 @@ static int ibmveth_open(struct net_device *netdev)
>  				     rxq_desc.desc,
>  				     mac_address);
>  		rc = -ENONET;
> -		goto out_free_tx_ltb;
> +		goto out_free_queue_mem;
>  	}
>  
>  	rc = ibmveth_alloc_buffer_pools(adapter);
>  	if (rc)
> -		goto out_free_tx_ltb;
> +		goto out_unregister_lan;
>  
>  	rc = ibmveth_setup_rx_interrupts(adapter);
> -	if (rc) {
> -		do {
> -			lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
> -		} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
> -		goto out_free_buffer_pools;
> -	}
> +	if (rc)
> +		goto out_unregister_lan;
>  
>  	netdev_dbg(netdev, "initial replenish cycle\n");
>  	ibmveth_schedule_rx_queue(adapter, 0);
>  
> +	rc = ibmveth_alloc_tx_resources(adapter);
> +	if (rc)
> +		goto out_cleanup_rx_interrupts;
> +
>  	netif_tx_start_all_queues(netdev);
>  
>  	adapter->opened = true;
> @@ -1333,11 +1384,14 @@ static int ibmveth_open(struct net_device *netdev)
>  
>  	return 0;
>  
> -out_free_buffer_pools:
> +out_cleanup_rx_interrupts:
> +	ibmveth_cleanup_rx_interrupts(adapter);
> +out_unregister_lan:
> +	do {
> +		lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
> +	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
>  	ibmveth_free_buffer_pools(adapter);
> -out_free_tx_ltb:
> -	while (--i >= 0)
> -		ibmveth_free_tx_ltb(adapter, i);
> +out_free_queue_mem:
>  	ibmveth_cleanup_rx_resources(adapter);

[Severity: Medium]
Should the h_free_logical_lan() part of this hunk be split out as its own
patch with a Fixes: tag and Cc: stable?

Rerouting the ibmveth_alloc_buffer_pools() failure to out_unregister_lan
looks like a real bug fix, not just a label rename. At the series merge base
(1b78070aaef6) that failure path in ibmveth_open() runs:

  rc = ibmveth_alloc_buffer_pools(adapter);
  if (rc)
          goto out_free_buffer_pools;
  ...
  out_free_buffer_pools:
          while (--i >= 0) { ... }
  out_unmap_filter_list:
          dma_unmap_single(dev, adapter->filter_list_dma, 4096,
                           DMA_BIDIRECTIONAL);

so the filter list, the buffer list and the RX completion ring are unmapped
and freed while the logical LAN registered by ibmveth_register_logical_lan()
is still live. Only the request_irq() failure path issued the hcall. This
patch is the first place that path deregisters, which matches the commit
message note:

  "After LAN registration, open-fail teardown issues h_free_logical_lan()
  before RX pool DMA teardown on the pool-fail path that previously never
  issued that hcall (missing deregistration, not a preference reorder)."

The concern is packaging rather than correctness of the code here. As
written, the fix for an ifup-under-memory-pressure path that leaves PHYP
registered against freed DMA memory is folded into a TX helper refactor with
no Fixes: tag, no Cc: stable and no statement of the failure mode.

The next patch in the series ("ibmveth: Add RX queue register helpers for
MQ") then moves pool allocation ahead of registration, so this arrangement
is transient inside the series and no single commit is backportable for the
mainline/stable bug. Could the deregistration fix be lifted to a standalone
patch at the front of the series?

[ ... ]

> @@ -1807,6 +1864,12 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
>  	int i, queue_num = skb_get_queue_mapping(skb);
>  	unsigned long mss = 0;
>  
> +	/* Close / failed reopen can free LTBs while IFF_UP is still set. */
> +	if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
> +		netdev->stats.tx_dropped++;
> +		goto out;
> +	}
> +
>  	if (ibmveth_is_packet_unsupported(skb, netdev))
>  		goto out;
>  	/* veth can't checksum offload UDP */

[Severity: Low]
This isn't a bug introduced by the patch, and after checking it appears to
be safe, but the shape of the new guard invites the question. The check
reads adapter->tx_ltb_ptr[queue_num] once, then the body re-reads the same
slots later:

  memcpy(adapter->tx_ltb_ptr[queue_num], skb->data, skb_headlen(skb));
  ...
  desc.fields.address = adapter->tx_ltb_dma[queue_num];

Since ibmveth_free_tx_ltb() now NULLs the slot before the unmap, is the
guard intended as a race barrier, or purely as a check for a slot that is
already known to be quiesced?

Every free site does look quiesced independently of this check:
ibmveth_close() calls netif_tx_disable() plus synchronize_net() before
ibmveth_free_tx_resources(); the ibmveth_set_channels() shrink frees only
after netif_set_real_num_tx_queues(), which does synchronize_net() on the
disabling path; and the open() unwind runs with all TX queues still XOFF.

The commit message already says as much:

  "NULL-first in free_tx_ltb() only closes the check-then-use window, it is
  not itself a UAF barrier."

Would it be clearer to state next to the guard that TX is quiesced at all
free sites, so the single read is not load-bearing?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,08/15] ibmveth: Add queue-aware RX buffer submit helper for MQ

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider:

Critical: 0 · High: 1 · Medium: 2 (1) · Low: 3

- [High] (Discarded — resolved later in this series.) In ibmveth_poll()
  at this commit, when ibmveth_rxq_get_buffer() rejects a correlator it…
- [Medium] Removing the WARN_ON()+schedule_work() recovery from
  ibmveth_remove_buffer_from_pool() leaves one live RX path with no…
- [Medium] ibmveth_replenish_task() wraps the entire multi-pool replenish
  loop in spin_lock_irqsave(&rxq->replenish_lock, flags), so up to…
- [Low] The patch contains a user-visible robustness fix (replacing the
  correlator WARN_ON()s that could take the partition down under…
- [Low] The hcall-failure log in ibmveth_replenish_task() reconstructs
  the wrapper name from fail.batch (`fail.batch == 1 ?
- [Low] The kernel-doc Return block of ibmveth_remove_buffer_from_pool()
  still documents only '-EINVAL - correlator maps to pool or index out…

Pre-existing issues:
- [Medium] (Discarded — reachability closed later in this series.) The
  new per-queue replenish_lock is taken by the replenish producer and by…

--

ibmveth: Add queue-aware RX buffer submit helper for MQ

This adds ibmveth_add_logical_lan_buffers() so RX pool replenishment can
go through H_ADD_LOGICAL_LAN_BUFFERS_QUEUE in multi-queue mode, and
parameterises the RX accessors, harvest and replenish paths by queue
index. It also adds a per-queue replenish_lock, converts
ibmveth_replenish_buffer_pool() to return an outcome that is logged after
the unlock, and replaces the correlator WARN_ON()s with
ibmveth_rxq_correlator_valid().

A few questions on the correlator error handling, the replenish critical
section and the failure log below.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 465330464f916..46f68f5b52e70 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... ]

> @@ -712,11 +722,100 @@ static inline void ibmveth_flush_buffer(void *addr, unsigned long length)

[ ... ]

> +	} else if (filled == 1) {
> +		rc = h_add_logical_lan_buffer(vdev->unit_address,
> +					      descs[0].desc);
> +	} else {
> +		/* Legacy 8-desc hcall; probe/mq_fallback keep batch <=
> +		 * IBMVETH_MAX_RX_REGULAR.
> +		 */
> +		rc = h_add_logical_lan_buffers(vdev->unit_address,
> +					       descs[0].desc, descs[1].desc,
> +					       descs[2].desc, descs[3].desc,
> +					       descs[4].desc, descs[5].desc,
> +					       descs[6].desc, descs[7].desc);
> +	}

[ ... ]

> @@ -899,21 +1000,85 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
>  }
>  
>  /* replenish routine */
> -static void ibmveth_replenish_task(struct ibmveth_adapter *adapter)
> +static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
> +				   int queue_index)
>  {
> -	int i;
> +	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
> +	struct ibmveth_replenish_fail fail = {};
> +	unsigned long flags;
> +	int i, rc;
> +	int need_reset = 0;
> +	int batch_fallback = 0;
> +	int hcall_fail = 0;
> +
> +	if (queue_index >= adapter->num_rx_queues) {
> +		netdev_dbg(adapter->netdev,
> +			   "Skipping replenish for freed queue %d (num_queues=%u)\n",
> +			   queue_index, adapter->num_rx_queues);
> +		return;
> +	}
>  
>  	adapter->replenish_task_cycles++;
>  
> -	for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
> -		struct ibmveth_buff_pool *pool = &adapter->rx_buff_pool[0][i];
> +	spin_lock_irqsave(&rxq->replenish_lock, flags);

[Severity: Medium]
Is it intended that the whole multi-pool replenish loop now runs with
hardware interrupts disabled?

The lock is taken here and released only after all five pools have been
walked, and ibmveth_replenish_buffer_pool() allocates and maps every
buffer inside that region:

	skb = netdev_alloc_skb(adapter->netdev, pool->buff_size);
	...
	dma_addr = dma_map_single_attrs(dev, skb->data, pool->buff_size,
					DMA_FROM_DEVICE, DMA_ATTR_NO_WARN);

With pool_count[] = { 256, 512, 256, 256, 256 } that is up to roughly
1536 allocations plus iommu mappings with interrupts masked on a cold
refill, for example ibmveth_open() -> ibmveth_restart_rx_queue() ->
ibmveth_replenish_task() where available is 0 for every pool. Before this
patch replenish ran from NAPI with interrupts enabled.

There is also a second-order concern about the netconsole rationale in
the commit message. ibmveth_poll_controller() calls
ibmveth_replenish_task(adapter, 0), and netpoll_send_skb() has already
done local_irq_save() by then, so this CPU can enter the region from
netpoll. DMA_ATTR_NO_WARN and moving the logging out of the lock cover
the iommu and the driver printks, but netdev_alloc_skb() for the large
pools takes the __alloc_skb()/slab path, which is not blanket
__GFP_NOWARN. If that allocator warning is emitted while netconsole is
bound to this device, can it re-enter ndo_poll_controller ->
ibmveth_replenish_task() -> spin_lock_irqsave() on the lock this CPU
already holds?

>  
> -		if (pool->active &&
> -		    (atomic_read(&pool->available) < pool->threshold))
> -			ibmveth_replenish_buffer_pool(adapter, pool);
> +	for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
> +		struct ibmveth_buff_pool *pool =
> +			&adapter->rx_buff_pool[queue_index][i];
> +
> +		if (pool->active && pool->free_map &&
> +		    (atomic_read(&pool->available) < pool->threshold)) {
> +			rc = ibmveth_replenish_buffer_pool(adapter, pool,
> +							   queue_index, &fail);
> +			switch (rc) {
> +			case IBMVETH_REPLENISH_RESET_MAP:
> +			case IBMVETH_REPLENISH_RESET_MQ:
> +				need_reset = rc;
> +				goto out_unlock;
> +			case IBMVETH_REPLENISH_BATCH_FALLBACK:
> +				batch_fallback = 1;
> +				break;
> +			case IBMVETH_REPLENISH_HCALL_FAIL:
> +				hcall_fail = 1;
> +				break;
> +			default:
> +				break;
> +			}
> +		}
>  	}
>  
> +out_unlock:
>  	ibmveth_update_rx_no_buffer(adapter);
> +
> +	spin_unlock_irqrestore(&rxq->replenish_lock, flags);

[ ... ]

> +	if (hcall_fail)
> +		dev_warn_ratelimited(&adapter->netdev->dev,
> +				     "RX %s failed: filled=%u, rc=%lu, batch=%u\n",
> +				     adapter->multi_queue ?
> +				     "h_add_logical_lan_buffers_queue" :
> +				     (fail.batch == 1 ?
> +				      "h_add_logical_lan_buffer" :
> +				      "h_add_logical_lan_buffers"),
> +				     fail.filled, fail.lpar_rc, fail.batch);

[Severity: Low]
Should the wrapper name here be selected from fail.filled rather than
fail.batch?

ibmveth_add_logical_lan_buffers() dispatches on filled:

	} else if (filled == 1) {
		rc = h_add_logical_lan_buffer(vdev->unit_address,
					      descs[0].desc);

So when adapter->rx_buffers_per_hcall is 8 (firmware with
IBMVETH_ILLAN_RX_MULTI_BUFF_SUPPORT) and the fill loop stops at
filled == 1 because remaining was 1, netdev_alloc_skb() failed or
dma_mapping_error() hit after the first descriptor, the call that failed
is h_add_logical_lan_buffer() but the log prints
h_add_logical_lan_buffers. fail.filled already records the right
discriminator.

Related: fail is a single struct reused across the pool loop while
batch_fallback and hcall_fail are sticky flags, so if two pools fail in
one invocation, do the printed filled/rc/batch values only describe the
last failure?

This is still the case at the end of the series.

>  }
>  
>  /* empty and free ana buffer pool - also used to do cleanup in error paths */
> @@ -948,6 +1113,12 @@ static void ibmveth_free_buffer_pool(struct ibmveth_adapter *adapter,
>  		kfree(pool->skbuff);
>  		pool->skbuff = NULL;
>  	}
> +
> +	/*
> +	 * Keep probe/sysfs geometry (active, size, buff_size, threshold).
> +	 * Only tear down runtime allocations; open reuses active pools.
> +	 */
> +	atomic_set(&pool->available, 0);
>  }

[Severity: Medium]
This is a pre-existing pattern and not introduced by this patch, but with
the new replenish_lock it is worth a note.

ibmveth_free_buffer_pool() kfree()s free_map, skbuff and dma_addr and
does atomic_set(&pool->available, 0) without taking
rxq->replenish_lock, while the new tests in ibmveth_replenish_task()
and ibmveth_rxq_correlator_valid():

	if (pool->active && pool->free_map &&
	...
	if (!bpool->active || !bpool->skbuff || !bpool->free_map)

are check-then-use against that teardown.

Tracing it through, the window looks closed at the end of the series:
ibmveth_poll_controller() returns early on !adapter->opened,
ibmveth_close() clears opened first, then calls
ibmveth_cleanup_rx_interrupts() and synchronize_net() before
ibmveth_free_buffer_pools(), and netpoll runs ndo_poll_controller with
hard interrupts disabled, so a netpoll section that observed
opened == true is waited out by that synchronize_net(). Would it be worth
documenting that free_buffer_pool() relies on the opened flag plus
synchronize_net() rather than on replenish_lock, since it writes state
the lock otherwise protects?

>  
>  /**
> @@ -1088,35 +1259,74 @@ ibmveth_free_buffer_pools(struct ibmveth_adapter *adapter)
>  		   adapter->num_rx_queues);
>  }
>  
> +static bool ibmveth_rxq_correlator_valid(struct ibmveth_adapter *adapter,
> +					 int queue_index, u64 correlator)
> +{
> +	unsigned int pool = correlator >> 32;
> +	unsigned int index = correlator & 0xffffffffUL;
> +	struct ibmveth_buff_pool *bpool;
> +
> +	if (pool >= IBMVETH_NUM_BUFF_POOLS)
> +		return false;
> +
> +	bpool = &adapter->rx_buff_pool[queue_index][pool];
> +
> +	/* Require a live pool with allocated arrays before indexing.
> +	 * Inactive pools still have size from init; free clears skbuff.
> +	 */
> +	if (!bpool->active || !bpool->skbuff || !bpool->free_map)
> +		return false;
> +
> +	return index < bpool->size;
> +}

[Severity: Low]
This isn't a bug, but this helper is a standalone robustness fix bundled
into an MQ feature patch. The commit message says a malformed correlator
with panic_on_warn set "would take the partition down", which is a
user-visible fix that stable trees would want, and the WARN_ON()s being
replaced here and in ibmveth_rxq_get_buffer() came from commit
2c91e2319ed9 ("net: ibmveth: Reset the adapter when unexpected states are
detected").

Could the correlator validation be split into its own patch with a
Fixes: 2c91e2319ed9 tag, so it can be identified independently of the
queue plumbing?

> +
> +static void ibmveth_rxq_advance(struct ibmveth_rx_q *rxq)
> +{
> +	if (++rxq->index == rxq->num_slots) {
> +		rxq->index = 0;
> +		rxq->toggle = !rxq->toggle;
> +	}
> +}
> +
>  /**
>   * ibmveth_remove_buffer_from_pool - remove a buffer from a pool
>   * @adapter: adapter instance
>   * @correlator: identifies pool and index
> + * @queue_index: RX queue index (0..num_rx_queues-1)
>   * @reuse: whether to reuse buffer
>   *
> + * Context: may run concurrently with netpoll replenish_task on the same
> + * queue; takes per-queue replenish_lock to serialize free_map /
> + * producer_index / available against the producer.
> + *
>   * Return:
>   * * %0       - success
>   * * %-EINVAL - correlator maps to pool or index out of range
>   * * %-EFAULT - pool and index map to null skb

[Severity: Low]
Does this Return block still describe all the -EINVAL cases?

ibmveth_rxq_correlator_valid() also returns false, mapped to -EINVAL
here, when bpool->active is 0 or when bpool->skbuff or bpool->free_map
are NULL. That is a deactivated or torn-down pool, not a correlator
"out of range", and callers react to -EINVAL by advancing the ring as
if the slot were corrupt, so the distinction is visible in behaviour.

The two updated tests, ibmveth_remove_buffer_from_pool_test() and
ibmveth_rxq_get_buffer_test(), also exercise none of the three new
predicates: they allocate free_map and rely on the pool being active, so
they pass identically with or without the new checks. Would it be worth
adding cases for active == 0 and for skbuff/free_map being NULL?

The doc block is unchanged at the end of the series.

>   */
>  static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,
> -					   u64 correlator, bool reuse)
> +					   u64 correlator, int queue_index,
> +					   bool reuse)
>  {
> +	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
>  	unsigned int pool  = correlator >> 32;
>  	unsigned int index = correlator & 0xffffffffUL;
>  	unsigned int free_index;
>  	struct sk_buff *skb;
> +	unsigned long flags;
> +	int rc = 0;
>  
> -	if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
> -	    WARN_ON(index >= adapter->rx_buff_pool[0][pool].size)) {
> -		schedule_work(&adapter->work);
> -		return -EINVAL;
> +	spin_lock_irqsave(&rxq->replenish_lock, flags);
> +
> +	if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
> +		rc = -EINVAL;
> +		goto out_unlock;
>  	}
>  
> -	skb = adapter->rx_buff_pool[0][pool].skbuff[index];
> -	if (WARN_ON(!skb)) {
> -		schedule_work(&adapter->work);
> -		return -EFAULT;
> +	skb = adapter->rx_buff_pool[queue_index][pool].skbuff[index];
> +	if (!skb) {
> +		rc = -EFAULT;
> +		goto out_unlock;
>  	}

[Severity: Medium]
Can this leak RX pool capacity permanently on a bad correlator?

The goto out_unlock skips both the free_map push and the
atomic_dec(&pool->available) that the success path performs, and the
schedule_work(&adapter->work) that used to accompany the WARN_ON()s is
gone from this function.

The callers treat -EINVAL/-EFAULT as "slot skipped, keep going".
ibmveth_rxq_harvest_buffer() advances the ring anyway:

	if (unlikely(rc)) {
		/* Skip a corrupt slot without claiming pool ownership. */
		if (rc == -EINVAL || rc == -EFAULT)
			ibmveth_rxq_advance(rxq);
		return rc;
	}

and ibmveth_drain_rx_queue() counts it as drained:

		/* -EINVAL/-EFAULT already advanced past the slot. */
		if (rc == -EINVAL || rc == -EFAULT) {
			drained++;
			continue;
		}

The invalid-buffer recycle branch of ibmveth_poll() is the concrete live
path, since it never calls ibmveth_rxq_get_buffer() and so nothing
schedules a reset there.

Given that ibmveth_replenish_buffer_pool() computes

	u32 remaining = pool->size - atomic_read(&pool->available);

and ibmveth_replenish_task() only refills while available < threshold,
does each occurrence permanently shrink the usable ring, until replenish
stops firing for that pool altogether? For the -EINVAL case the
DMA-mapped skb also stays in pool->skbuff[] with its free_map entry left
at IBM_VETH_INVALID_MAP until teardown.

The pre-patch code performed a close/open reset here, which reconciled
the driver bookkeeping with PHYP. Should these paths still request a
reset, or correct available, when the correlator is rejected?

At the end of the series this is unchanged in effect:
ibmveth_poll_harvest_slot() returns true for -EINVAL/-EFAULT and
ibmveth_poll_recycle_invalid() schedules nothing, while only
ibmveth_poll_skip_bad_correlator() schedules the reset.

>  
>  	/* if we are going to reuse the buffer then keep the pointers around
> @@ -1127,75 +1337,88 @@ static int ibmveth_remove_buffer_from_pool(struct ibmveth_adapter *adapter,

[ ... ]

> -	free_index = adapter->rx_buff_pool[0][pool].producer_index;
> -	adapter->rx_buff_pool[0][pool].producer_index++;
> -	if (adapter->rx_buff_pool[0][pool].producer_index >=
> -	    adapter->rx_buff_pool[0][pool].size)
> -		adapter->rx_buff_pool[0][pool].producer_index = 0;
> -	adapter->rx_buff_pool[0][pool].free_map[free_index] = index;
> +	free_index = adapter->rx_buff_pool[queue_index][pool].producer_index;
> +	adapter->rx_buff_pool[queue_index][pool].producer_index++;
> +	if (adapter->rx_buff_pool[queue_index][pool].producer_index >=
> +	    adapter->rx_buff_pool[queue_index][pool].size)
> +		adapter->rx_buff_pool[queue_index][pool].producer_index = 0;
> +	adapter->rx_buff_pool[queue_index][pool].free_map[free_index] = index;
>  
>  	mb();
>  
> -	atomic_dec(&adapter->rx_buff_pool[0][pool].available);
> +	atomic_dec(&adapter->rx_buff_pool[queue_index][pool].available);
>  
> -	return 0;
> +out_unlock:
> +	spin_unlock_irqrestore(&rxq->replenish_lock, flags);
> +	return rc;
>  }
>  
>  /* get the current buffer on the rx queue */
> -static inline struct sk_buff *ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter)
> +static struct sk_buff *
> +ibmveth_rxq_get_buffer(struct ibmveth_adapter *adapter,
> +		       int queue_index)
>  {
> -	struct ibmveth_rx_q *rxq = &adapter->rx_queue[0];
> +	struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
>  	u64 correlator = rxq->queue_addr[rxq->index].correlator;
>  	unsigned int pool = correlator >> 32;
>  	unsigned int index = correlator & 0xffffffffUL;
>  
> -	if (WARN_ON(pool >= IBMVETH_NUM_BUFF_POOLS) ||
> -	    WARN_ON(index >= adapter->rx_buff_pool[0][pool].size)) {
> +	if (!ibmveth_rxq_correlator_valid(adapter, queue_index, correlator)) {
>  		schedule_work(&adapter->work);
>  		return NULL;
>  	}

[ ... ]

> @@ -2161,34 +2384,41 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,

[ ... ]

> -			skb = ibmveth_rxq_get_buffer(adapter);
> +			skb = ibmveth_rxq_get_buffer(adapter, queue_index);
>  			if (unlikely(!skb))
>  				break;

[ ... ]

> @@ -2265,14 +2501,18 @@ static int ibmveth_poll(struct napi_struct *napi, int budget)

[ ... ]

> -	if (ibmveth_rxq_pending_buffer(adapter) && napi_schedule(napi)) {
> -		ibmveth_disable_irq(adapter, 0);
> +	if (ibmveth_rxq_pending_buffer(adapter, queue_index) &&
> +	    napi_schedule(napi)) {
> +		ibmveth_disable_irq(adapter, queue_index);
>  		goto restart_poll;
>  	}

[Severity: High]
At this commit, can ibmveth_poll() spin here forever after
ibmveth_rxq_get_buffer() rejects a correlator?

Unlike ibmveth_rxq_harvest_buffer(), ibmveth_rxq_get_buffer() returns
NULL without advancing the ring, and the caller just breaks:

			skb = ibmveth_rxq_get_buffer(adapter, queue_index);
			if (unlikely(!skb))
				break;

rxq->index and rxq->toggle are untouched, so
ibmveth_rxq_pending_buffer() is still true, napi_schedule() succeeds and
poll jumps back to restart_poll on the same slot. The reset queued by
ibmveth_rxq_get_buffer() cannot break the cycle either, because
ibmveth_reset() -> dev_close() -> napi_disable() waits for this poll to
finish.

The following patch in the series, "ibmveth: Harden RX poll path with
helpers", replaces this bare break with
ibmveth_poll_skip_bad_correlator(), which schedules the reset and calls
ibmveth_poll_harvest_slot() so the slot is advanced, so the loop is gone
by the end of the series. Would it be better to fold that handling into
this patch, so the intermediate commit is not bisectable into a softirq
loop?

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,09/15] ibmveth: Harden RX poll path with helpers

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 4 potential
issue(s) to consider:

Critical: 0 · High: 2 (1) · Medium: 2 · Low: 0

- [High] The new ibmveth_poll_stopping() check placed between
  napi_complete_done() and ibmveth_enable_irq() cannot prevent a…
- [High] The new oversize guard in ibmveth_poll_deliver_frame() bounds
  the PHYP-reported offset+length against skb_tailroom(skb) instead of…
- [Medium] A real, user-visible sk_buff leak fix is buried inside a
  199-line helper-extraction patch with no Fixes: tag and no Cc: stable.
- [Medium] All 'skip the slot' outcomes (invalid descriptor, bad
  correlator, oversize offset+length) neither increment frames_processed…

Pre-existing issues:
- [High] ibmveth_remove() calls cancel_work_sync(&adapter->work) while
  the netdev is still registered and RX NAPI/IRQ paths are live, then…

--

ibmveth: Harden RX poll path with helpers

This splits the RX slot handling in ibmveth_poll() into a set of small
helpers (poll_stopping, harvest_slot, recycle_invalid, skip_bad_correlator,
drop_oversize, deliver_frame, bump_invalid) so the NAPI callback becomes a
thin budget loop.  Along with the motion it adds an offset+length bounds
check before skb_put(), frees the copybreak skb on a harvest error, and
adds stopping checks so close/napi_disable does not re-arm PHYP delivery.

A few questions below.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 46f68f5b52e70..f726d1e3f0b11 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... ]

> @@ -2382,125 +2380,236 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
>  	}
>  }
>  

[ ... ]

> +static bool ibmveth_poll_harvest_slot(struct ibmveth_adapter *adapter,
> +				      int queue_index, bool reuse)
> +{
> +	int rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, reuse);
> +
> +	return !rc || rc == -EINVAL || rc == -EFAULT;
> +}

[Severity: Medium]
Can this helper ever return false?  ibmveth_rxq_harvest_buffer() documents
and returns only 0, -EINVAL or -EFAULT:

	rc = ibmveth_remove_buffer_from_pool(adapter, cor, queue_index, reuse);
	if (unlikely(rc)) {
		/* Skip a corrupt slot without claiming pool ownership. */
		if (rc == -EINVAL || rc == -EFAULT)
			ibmveth_rxq_advance(rxq);
		return rc;
	}

so every "break" in ibmveth_poll_recycle_invalid(),
ibmveth_poll_skip_bad_correlator() and ibmveth_poll_drop_oversize() looks
unreachable.

Combined with the budget accounting in the loop below, is there anything
left that bounds one ibmveth_poll() invocation?  A skipped slot returns 0
from ibmveth_poll_deliver_frame(), so neither the break nor
frames_processed++ runs:

		rc = ibmveth_poll_deliver_frame(napi, adapter, netdev,
						queue_index);
		if (rc < 0)
			break;
		if (rc > 0)
			frames_processed++;

and each skip recycles the slot with reuse=true, after which
ibmveth_replenish_task() re-posts it.  The tail of ibmveth_poll() then does:

	if (ibmveth_rxq_pending_buffer(adapter, queue_index) &&
	    napi_schedule(napi)) {
		ibmveth_disable_irq(adapter, queue_index);
		goto restart_poll;
	}

which re-enters the loop in the same invocation with frames_processed
unchanged.  If PHYP keeps publishing skippable slots (stale ring contents,
malformed completions), does the frames_processed == budget exit ever
become reachable, and does this poll ever return?

The reset escalation in ibmveth_poll_skip_bad_correlator() uses
schedule_work(), which queues on the current CPU via system_percpu_wq, so
would the worker be able to run while that CPU is stuck in the poll?  The
ibmveth_poll_drop_oversize() path escalates nothing at all.

The commit message states:

    Skipped and dropped slots do not count against the NAPI budget; only
    a delivered frame does.

Is that intentional given it removes the only bound on the loop?

> +
> +static bool ibmveth_poll_recycle_invalid(struct net_device *netdev,
> +					 struct ibmveth_adapter *adapter,
> +					 int queue_index)
> +{
> +	netdev_dbg(netdev, "recycling invalid buffer\n");
> +	ibmveth_poll_bump_invalid(adapter, queue_index);
> +	return ibmveth_poll_harvest_slot(adapter, queue_index, true);
> +}
> +
> +static bool ibmveth_poll_skip_bad_correlator(struct net_device *netdev,
> +					     struct ibmveth_adapter *adapter,
> +					     int queue_index)
> +{
> +	if (net_ratelimit())
> +		netdev_err(netdev,
> +			   "bad correlator on queue %d, skipping slot\n",
> +			   queue_index);
> +	/* Residual stale slot after resize: recover via reset rather
> +	 * than spinning forever. Always escalate; only the log is
> +	 * rate-limited.
> +	 */
> +	schedule_work(&adapter->work);

[Severity: High]
This isn't a bug introduced by this patch, but this adds another NAPI-side
producer of adapter->work while ibmveth_remove() still cancels that work
before the device is quiesced:

	cancel_work_sync(&adapter->work);

	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
		kobject_put(&adapter->rx_buff_pool[0][i].kobj);

	unregister_netdev(netdev);

	free_netdev(netdev);

Can a poll running on another CPU call schedule_work(&adapter->work) from
here (or from the ibmveth_enable_irq() failure path in ibmveth_poll())
after cancel_work_sync() has returned?  Nothing cancels or flushes it
again, and adapter is netdev_priv(netdev), so the queued work_struct and
everything ibmveth_reset() touches would be freed by free_netdev().

For the record, the ordering appears to be repaired later in this series:
at the series tip ibmveth_remove() moves the cancellation after
unregister_netdev(), with a comment that cancel_work_sync() before
unregister left a window where poll could schedule_work() and the worker
ran after free_netdev().

> +	ibmveth_poll_bump_invalid(adapter, queue_index);
> +	return ibmveth_poll_harvest_slot(adapter, queue_index, true);
> +}
> +

[ ... ]

> +static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
> +				      struct ibmveth_adapter *adapter,
> +				      struct net_device *netdev,
> +				      int queue_index)
> +{

[ ... ]

> +	room = skb_tailroom(skb);
> +	off = offset;
> +	len = length;
> +	if (unlikely(off >= room || len > room - off)) {
> +		if (!ibmveth_poll_drop_oversize(netdev, adapter, queue_index,
> +						off, len, room))
> +			return -1;
> +		return 0;
> +	}

[Severity: High]
Should the bound here be pool->buff_size rather than skb_tailroom(skb)?

ibmveth_replenish_buffer_pool() allocates, maps and advertises exactly
pool->buff_size to the hypervisor:

	skb = netdev_alloc_skb(adapter->netdev, pool->buff_size);
	...
	dma_addr = dma_map_single_attrs(dev, skb->data, pool->buff_size,
					DMA_FROM_DEVICE, DMA_ATTR_NO_WARN);
	...
	descs[filled].fields.flags_len = IBMVETH_BUF_VALID | pool->buff_size;

netdev_alloc_skb() rounds up to an allocator bucket, so skb_tailroom() on
an untouched pool buffer is larger than buff_size (a 16 KiB pool buffer
lands in a 32 KiB allocation).  Does a PHYP-reported off+len in the
(buff_size, tailroom] gap pass this check and then get committed and
delivered?

	skb_put(skb, length);
	skb->protocol = eth_type_trans(skb, netdev);
	...
	napi_gro_receive(napi, skb);

That would hand uninitialised slab bytes outside the DMA mapping to the
stack, and on the copybreak path skb_copy_to_linear_data() plus
ibmveth_flush_buffer(skb->data, length + offset) would touch addresses
past the mapped window.  The commit message says deliver "rejects a PHYP
offset+length that would skb_put() past the buffer" - is the buffer here
meant to be the hardware-visible one?  The correlator has already been
validated by ibmveth_rxq_correlator_valid(), so
adapter->rx_buff_pool[queue_index][correlator >> 32].buff_size is
available as the bound.

> +
> +	if (lrg_pkt) {
> +		__be64 *rxmss = (__be64 *)(skb->data + 8);
> +
> +		mss = (u16)be64_to_cpu(*rxmss);
> +	}
> +
> +	new_skb = NULL;
> +	if (length < rx_copybreak)
> +		new_skb = netdev_alloc_skb(netdev, length);
> +
> +	if (new_skb) {
> +		skb_copy_to_linear_data(new_skb, skb->data + offset, length);
> +		if (rx_flush)
> +			ibmveth_flush_buffer(skb->data, length + offset);
> +		rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);
> +		if (unlikely(rc)) {
> +			kfree_skb(new_skb);
> +			return -1;
> +		}

[Severity: Medium]
This kfree_skb(new_skb) is a real fix for an sk_buff leak that exists at
the baseline in ibmveth_poll():

	new_skb = netdev_alloc_skb(netdev, length);
	...
	skb_copy_to_linear_data(new_skb, skb->data + offset, length);
	...
	if (unlikely(ibmveth_rxq_harvest_buffer(adapter, true)))
		break;   /* new_skb is neither freed nor handed to the stack */

Could this be split into its own patch with a Fixes: tag (and Cc: stable)
so it can be routed to net and picked up by older trees?  Buried in a
199-insertion helper extraction it is not separable for backporting.  The
same applies to the new offset+length validation, which the commit message
describes as preventing an skb_put() past the buffer.

> +		skb = new_skb;
> +	} else {
> +		rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, false);
> +		if (unlikely(rc))
> +			return -1;
> +		skb_reserve(skb, offset);
> +	}

[ ... ]

> +	return 1;
> +}
> +
>  static int ibmveth_poll(struct napi_struct *napi, int budget)
>  {

[ ... ]

>  	if (frames_processed == budget)
>  		goto out;
>  
>  	if (!napi_complete_done(napi, frames_processed))
>  		goto out;
>  
> -	/* We think we are done - reenable interrupts,
> -	 * then check once more to make sure we are done.
> +	/*
> +	 * napi_disable() sets DISABLE then waits for this poll. Without a
> +	 * second stopping check here, enable_irq() can re-arm PHYP after
> +	 * resize already masked the queue; late IRQs then hit the handler
> +	 * after num_rx_queues was published lower (lab WARN at interrupt).
>  	 */
> +	if (ibmveth_poll_stopping(netdev, napi))
> +		goto out;
> +
>  	rc = ibmveth_enable_irq(adapter, queue_index);

[Severity: High]
Does this check actually close the window it describes?  It is a
check-then-act pair, and both predicates in ibmveth_poll_stopping() go
false exactly when teardown is finishing:

	return !netif_running(netdev) || napi_disable_pending(napi);

napi_disable_locked() clears the DISABLE bit before returning:

	clear_bit(NAPI_STATE_DISABLE, &n->state);

so napi_disable_pending() is false once close is past its wait.  And
netif_running() stays true for the driver's internal close callers -
ibmveth_change_mtu(), ibmveth_set_mac_addr(), the features paths and
veth_pool_store() all do "if (netif_running(dev)) ibmveth_close(dev);".

Sequence:

CPU0 ibmveth_poll()
	napi_complete_done()		/* clears SCHED */
					/* delayed: hard IRQ, or vCPU
					   dispatch preemption on a
					   shared-processor LPAR */

CPU1 ibmveth_change_mtu() -> ibmveth_close() -> ibmveth_cleanup_rx_interrupts()
	napi_disable()			/* clears DISABLE on return */
	ibmveth_disable_irq(adapter, i);
	synchronize_irq(adapter->queue_irq[i]);
	free_irq(...)

CPU0 resumes:
	if (ibmveth_poll_stopping(netdev, napi))	/* false */
		goto out;
	rc = ibmveth_enable_irq(adapter, queue_index);	/* re-arms PHYP */

Can the queue end up unmasked after the final remask and after free_irq()
removed the handler?  That is the case the commit message claims to close:

    ibmveth_poll_stopping() ensures close/napi_disable does not re-arm PHYP.

Would moving the unmask before napi_complete_done(), or moving close's
final remask after synchronize_net(), be a more reliable ordering than
adding another check here?  This looks unchanged at the series tip.

>  	if (rc) {
>  		netdev_err(netdev,

Thanks for looking at these.

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,10/15] ibmveth: Enable multi-queue RX receive path

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 11 potential
issue(s) to consider:

Critical: 0 · High: 1 (2) · Medium: 4 · Low: 6

- [High] In multi-queue mode ibmveth_setup_rx_interrupts() already
  unmasks the PHYP interrupt for every live RX queue, and ibmveth_open()…
- [Medium] Adapter-wide RX counters are not converted to per-queue
  storage while this commit enables up to min(nr_cpus,8) live RX queues:…
- [Medium] ibmveth_apply_mq_fallback() publishes adapter->num_rx_queues =
  1 at the top of ibmveth_open() but deliberately leaves…
- [Medium] ibmveth_open() now runs ibmveth_replenish_task() for every
  queue, and that function can queue reset work…
- [Medium] The new probe error paths call kobject_put() on the pool
  kobjects, which are embedded in struct ibmveth_adapter inside…
- [Low] ibmveth_get_channels() begins advertising max_rx/rx_count > 1 (up
  to min(num_online_cpus(), IBMVETH_DEFAULT_QUEUES)) while its peer…
- [Low] Probe enables min(num_online_cpus(), IBMVETH_DEFAULT_QUEUES) RX
  queues whenever the firmware MQ bit is present; each queue clones…
- [Low] The commit message itself states that this commit ships known-
  wrong statistics and new schedule_work() producers whose remove-path…
- [Low] The pool sysfs kobject creation loop in ibmveth_probe() changes
  from 'ignore kobject_init_and_add() failure and continue probing' to…
- [Low] The ibmveth_setup_rx_interrupts() kernel-doc still describes the
  pre-MQ open sequence: "Single-queue open leaves PHYP masked here and…
- [Low] ibmveth_register_single_rx_queue() logs a signed `long lpar_rc`
  with the unsigned conversion "rc=0x%lx".

Pre-existing issues:
- [High] This commit makes the MQ reset producer live
  (adapter->multi_queue was hardcoded 0 before) and adds a new…
- [High] ibmveth_poll_controller() now replenishes and schedules every
  published RX queue instead of only queue 0, multiplying the exposure…

--

ibmveth: Enable multi-queue RX receive path

This turns on the multi-queue RX path when firmware advertises
IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT, raises IBMVETH_MAX_RX_QUEUES from 1
to 16, and wires subordinate queue registration, IRQ setup and per-queue
replenish into open()/close().  Probe now selects min(num_online_cpus(),
IBMVETH_DEFAULT_QUEUES) RX queues and refreshes the VIO CMO entitlement.

A few questions on the commit message first.

[Severity: Low]
The changelog says:

    Two things are knowingly left for patch 11. RX counters are racy:
    adapter->rx_no_buffer is assigned rather than summed from one queue's
    buffer-list page, so it reports whichever queue replenished last and
    can go backwards, while rx_packets, rx_bytes, rx_invalid_buffer and
    rx_large_packets are plain read-modify-writes now reached from several
    NAPI instances at once, so they can lose counts

and:

    This commit also adds schedule_work() producers, and the remove-path
    unregister / cancel_work_sync reorder and reset reg_state gate that
    make them safe land there too.

Would it be preferable to reorder so the counter conversion and the
remove-path fix precede the commit that makes the MQ path live, rather
than shipping an intermediate commit that is documented as incorrect in
those two respects?

The same paragraph documents a third, unrelated change:

    On probe failure after pool kobjects were created, put them before
    free_netdev(). The leak is pre-existing and unrelated to multi-queue,
    so a standalone version also goes to net; the probe_cleanup helper
    lands in patch 11.

Since this is described as unrelated and is being posted separately, could
it be dropped from this patch instead of open-coded on four error paths
here and then replaced by a helper later in the series?

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index f726d1e3f0b11..2e8896ea5af2e 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... ]

> @@ -992,11 +1014,21 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
>   * because there was not a buffer in the buffer list capable of holding
>   * the frame.
>   */
> -static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter)
> +static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
> +					int queue_index)
>  {
> -	__be64 *p = adapter->buffer_list_addr[0] + 4096 - 8;
> +	__be64 *p;
> +	u64 drops;
>  
> -	adapter->rx_no_buffer = be64_to_cpup(p);
> +	if (queue_index < 0 ||
> +	    queue_index >= ibmveth_get_num_rx_queues(adapter) ||
> +	    !adapter->buffer_list_addr[queue_index])
> +		return;
> +
> +	p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
> +	drops = be64_to_cpup(p);
> +
> +	adapter->rx_no_buffer = drops;
>  }

[Severity: Medium]
ibmveth_update_rx_no_buffer() now takes a queue index but still ends in a
plain assignment to the single adapter-wide field:

	adapter->rx_no_buffer = drops;

With more than one live queue, does each call simply overwrite the value
harvested for the previous queue?  In ibmveth_close() the new loop calls it
for every live queue back to back:

	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
		ibmveth_update_rx_no_buffer(adapter, i);

so the reported counter ends up being whichever queue was visited last, and
can move backwards between reads.

Similarly, ibmveth_poll_deliver_frame() does unlocked read-modify-writes:

	netdev->stats.rx_packets++;
	netdev->stats.rx_bytes += length;

and the replenish counters (replenish_task_cycles, replenish_add_buff_success
/ _failure, replenish_no_mem) are serialized only by the per-queue
rx_queue[i].replenish_lock, which does not serialize different queues against
each other.  Can these lose counts once several NAPI instances run
concurrently?

The changelog notes that the following patch moves these counters to
per-queue storage summed on read, which does resolve it; the question is
whether this intermediate state is worth carrying.

> @@ -1062,6 +1094,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
>  		dev_err_ratelimited(&adapter->netdev->dev,
>  				    "MQ buffer add H_FUNCTION (q=%d, batch=%u), reset\n",
>  				    queue_index, fail.batch);
> +		adapter->mq_fallback = true;
>  		schedule_work(&adapter->work);
>  	}

[Severity: High]
This isn't a bug introduced by this patch, but this commit makes the reset
producer reachable for the first time (adapter->multi_queue was hardcoded to
0 before) and adds a second schedule_work() site here.

At this commit ibmveth_reset() has no registration-state check:

	rtnl_lock();
	dev_close(adapter->netdev);
	dev_open(adapter->netdev, NULL);
	rtnl_unlock();

and ibmveth_remove() still runs cancel_work_sync(&adapter->work) before
unregister_netdev().  If a live queue hits MQ buffer-add H_FUNCTION after
the cancel, can the requeued work then run dev_close()/dev_open() against a
netdev that unregister_netdev()/free_netdev() is tearing down?

Both halves are fixed later in this series (ibmveth_remove() is reordered to
unregister_netdev() before cancel_work_sync(), and ibmveth_reset() gains the
netdev->reg_state != NETREG_REGISTERED gate), so the exposure is limited to
this intermediate commit.

> @@ -1081,6 +1114,27 @@ static void ibmveth_replenish_task(struct ibmveth_adapter *adapter,
>  				     fail.filled, fail.lpar_rc, fail.batch);
>  }
>  
> +/**
> + * ibmveth_restart_rx_queue - Post buffers and ensure Q can take RX
> + * @adapter: ibmveth adapter
> + * @qindex: RX queue index
> + *
> + * SQ open leaves PHYP masked until the first poll. If schedule_prep fails,
> + * NAPI never runs and the queue stays masked (TX OK, RX/ARP dead) until
> + * reload. Replenish first so an enable_irq fallback can actually deliver.
> + * Also used after every open (SQ and MQ) and after scale-down so a
> + * queue is not left idle+masked.
> + */
> +static void ibmveth_restart_rx_queue(struct ibmveth_adapter *adapter,
> +				     int qindex)
> +{
> +	ibmveth_replenish_task(adapter, qindex);
> +	if (ibmveth_schedule_rx_queue(adapter, qindex))
> +		return;
> +
> +	ibmveth_enable_irq(adapter, qindex);
> +}

[Severity: High]
Is the unconditional ibmveth_enable_irq() here safe in multi-queue mode?

ibmveth_schedule_rx_queue() returns false in two different situations:

	if (napi_schedule_prep(napi)) {
		ibmveth_disable_irq(adapter, qindex);
		__napi_schedule(napi);
		return true;
	}
	return false;

The second case is "NAPI already claimed", which is exactly what the IRQ
handler does after it has masked PHYP.  In MQ mode
ibmveth_setup_rx_interrupts() has already unmasked every queue:

	if (adapter->multi_queue && num > 1) {
		for (i = 0; i < num; i++) {
			rc = ibmveth_enable_irq(adapter, i);

so by the time open() runs its restart loop an interrupt may already have
claimed NAPI and masked the queue.  restart then re-unmasks it under the
in-flight poll, and when that poll finishes ibmveth_poll() calls
ibmveth_enable_irq() again on an already-enabled subordinate interrupt.

ibmveth_toggle_irq() folds H_PARAMETER only on the disable side:

		if (h_rc == H_PARAMETER && !enable) {
			dev_warn_ratelimited(...);
			return 0;
		}

so the redundant enable returns -EIO, and ibmveth_poll() escalates that to
schedule_work(&adapter->work), i.e. a full dev_close()/dev_open() of an
otherwise healthy adapter.

Since the PHYP mask/unmask hcalls are not reference counted, should restart
skip the enable when schedule_rx_queue() failed because NAPI was already
scheduled, rather than when prep failed for lack of a pending descriptor?
The same enable-then-restart sequence appears later in the ethtool -L
scale-up path of ibmveth_resize_rx_queues_incremental():

	rc = ibmveth_enable_irq(adapter, i);
	...
	ibmveth_restart_rx_queue(adapter, i);

[ ... ]

> +static int
> +ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,
> +				 int queue_idx)
> +{
> +	struct net_device *netdev = adapter->netdev;
> +	union ibmveth_buf_desc rxq_desc;
> +	long lpar_rc;
> +
> +	if (WARN_ON(queue_idx < 1 || queue_idx >= IBMVETH_MAX_RX_QUEUES))
> +		return -EINVAL;
> +
> +	rxq_desc.fields.flags_len = IBMVETH_BUF_VALID |
> +				    adapter->rx_queue[queue_idx].queue_len;
> +	rxq_desc.fields.address = adapter->rx_queue[queue_idx].queue_dma;
> +
> +	lpar_rc = ibmveth_register_logical_lan_queue(adapter, rxq_desc,
> +						     queue_idx);
> +	if (lpar_rc != H_SUCCESS) {
> +		netdev_err(netdev, "Failed to register queue %d: rc=0x%lx\n",
> +			   queue_idx, lpar_rc);

[Severity: Low]
lpar_rc is a signed long here, and ibmveth_register_logical_lan_queue()
returns -EINVAL when irq_create_mapping() fails, so that case prints as
rc=0xffffffffffffffea.  Would %ld (or splitting the errno and hypervisor
cases) read better?  Note this will not produce a build warning, since the
signed/unsigned format mismatch is only diagnosed under
-Wformat-signedness, which the kernel does not enable.

[ ... ]

> +static void ibmveth_apply_mq_fallback(struct ibmveth_adapter *adapter)
> +{
> +	struct net_device *netdev = adapter->netdev;
> +
> +	if (!adapter->mq_fallback)
> +		return;
> +
> +	adapter->mq_fallback = false;
> +
> +	if (!adapter->multi_queue && ibmveth_get_num_rx_queues(adapter) == 1)
> +		return;
> +
> +	netdev_warn(netdev,
> +		    "Falling back to single RX queue (firmware MQ unavailable)\n");
> +	adapter->multi_queue = false;
> +	ibmveth_publish_num_rx_queues(adapter, 1);
> +	/* real_num_rx_queues is set later in open after resources exist. */

[Severity: Medium]
Can the two counts be left permanently out of sync here?

ibmveth_apply_mq_fallback() runs at the very top of ibmveth_open() and
publishes adapter->num_rx_queues = 1, but netdev->real_num_rx_queues is only
updated further down.  Four failure exits sit in between:

	ibmveth_alloc_filter_list()
	ibmveth_alloc_rx_queues()
	ibmveth_alloc_buffer_pools()
	ibmveth_register_rx_queues()
	...
	netif_set_real_num_rx_queues(netdev, ibmveth_get_num_rx_queues(adapter))

If the fallback open fails at any of those (for example -ENOMEM from pool
allocation, or -ENONET from queue-0 registration), the device stays down with
adapter->num_rx_queues == 1 while netdev->real_num_rx_queues is still the old
multi-queue value, so /sys/class/net/ethX/queues/rx-1..7 remain present for
queues the driver will never drive.

That contradicts the invariant the probe path documents for exactly this
case:

	 * alloc_etherdev_mqs() sized real_num_rx_queues to MAX. Match the
	 * advertised default (or SQ 1) before register_netdev so down-state
	 * readers agree with adapter->num_rx_queues / ethtool -l.

Would it be better to lower real_num_rx_queues in the fallback path itself,
or to restore the published count on the open failure paths?

> @@ -1651,6 +1900,8 @@ static int ibmveth_open(struct net_device *netdev)
>  
>  	netdev_dbg(netdev, "open starting\n");
>  
> +	ibmveth_apply_mq_fallback(adapter);
> +
>  	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
>  		rxq_entries += adapter->rx_buff_pool[0][i].size;
>  
> @@ -1670,18 +1921,34 @@ static int ibmveth_open(struct net_device *netdev)
>  	if (rc)
>  		goto out_free_buffer_pools;
>  
> -	rc = netif_set_real_num_rx_queues(netdev, adapter->num_rx_queues);
> +	rc = netif_set_real_num_rx_queues(netdev,
> +					  ibmveth_get_num_rx_queues(adapter));
> +
>  	if (rc) {
>  		netdev_err(netdev, "failed to set number of rx queues\n");
>  		goto out_unregister_queues;
>  	}
>  
> +	/*
> +	 * Post buffers before setup_rx_interrupts(). MQ setup then unmasks
> +	 * PHYP; SQ setup leaves PHYP masked. Scheduling NAPI only when a
> +	 * descriptor is already pending is not enough: after ifdown/up
> +	 * (RX=8, no -L) NAPI can be idle with nothing pending and the
> +	 * queue stays dead (TX OK, ARP/RX fail).
> +	 * restart_rx_queue() replenishes, schedules NAPI, and unmasks if
> +	 * prep fails.
> +	 */
> +	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++) {
> +		netdev_dbg(netdev, "initial replenish cycle for queue %d\n", i);
> +		ibmveth_replenish_task(adapter, i);
> +	}
> +
>  	rc = ibmveth_setup_rx_interrupts(adapter);
>  	if (rc)
>  		goto out_free_all_queues; /* setup already disposed IRQs */
>  
> -	netdev_dbg(netdev, "initial replenish cycle\n");
> -	ibmveth_schedule_rx_queue(adapter, 0);
> +	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
> +		ibmveth_restart_rx_queue(adapter, i);

[Severity: Medium]
This replenish loop can queue reset work, and ibmveth_replenish_task()
returns void, so open() cannot see it:

	} else if (need_reset == IBMVETH_REPLENISH_RESET_MQ) {
		...
		adapter->mq_fallback = true;
		schedule_work(&adapter->work);
	}

If a buffer-add hcall returns H_FUNCTION here and the open then fails at
ibmveth_setup_rx_interrupts() or ibmveth_alloc_tx_resources(), userspace gets
an error back from ndo_open and the device is left down, but the pending
ibmveth_reset() only checks registration state:

	rtnl_lock();
	if (netdev->reg_state != NETREG_REGISTERED) {
		rtnl_unlock();
		return;
	}

	dev_close(adapter->netdev);
	dev_open(adapter->netdev, NULL);

so it brings the interface up anyway a moment later.  Should the reset worker
also check that the device was successfully opened before reopening it?

[Severity: Low]
This isn't a bug, but the kernel-doc on ibmveth_setup_rx_interrupts() still
describes the pre-MQ ordering:

 * Single-queue open leaves PHYP masked
 * here and kicks NAPI afterward (classic path: first poll posts then
 * enables).

After this change open() posts buffers for every live queue (queue 0
included, in single-queue mode) before calling setup_rx_interrupts(), and
ibmveth_restart_rx_queue() posts again, so the "first poll posts" behaviour no
longer applies to either mode.  Could that comment be refreshed?

> @@ -1714,6 +1981,7 @@ static int ibmveth_open(struct net_device *netdev)
>  static int ibmveth_close(struct net_device *netdev)
>  {
>  	struct ibmveth_adapter *adapter = netdev_priv(netdev);
> +	int i;
>  
>  	/* Gate on opened, not IFF_UP: pool_store/change_mtu close+open can
>  	 * leave IFF_UP set after a failed reopen.
> @@ -1734,7 +2002,8 @@ static int ibmveth_close(struct net_device *netdev)
>  	/* Wait for softirq/poll that already passed shutdown checks. */
>  	synchronize_net();
>  
> -	ibmveth_update_rx_no_buffer(adapter);
> +	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
> +		ibmveth_update_rx_no_buffer(adapter, i);
>  	ibmveth_free_all_queues(adapter);

[ ... ]

> @@ -2527,6 +2796,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct *napi,
>  		ibmveth_rx_csum_helper(skb, adapter);
>  	}
>  
> +	skb_record_rx_queue(skb, queue_index);
>  	napi_gro_receive(napi, skb);
>  
>  	netdev->stats.rx_packets++;

[ ... ]

> @@ -2750,9 +3021,14 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
>  static void ibmveth_poll_controller(struct net_device *dev)
>  {
>  	struct ibmveth_adapter *adapter = netdev_priv(dev);
> +	unsigned int num = ibmveth_get_num_rx_queues(adapter);
> +	int i;
>  
> -	ibmveth_replenish_task(adapter, 0);
> -	ibmveth_schedule_rx_queue(adapter, 0);
> +	for (i = 0; i < num; i++)
> +		ibmveth_replenish_task(adapter, i);
> +
> +	for (i = 0; i < num; i++)
> +		ibmveth_schedule_rx_queue(adapter, i);
>  }
>  #endif

[Severity: High]
This is a pre-existing race, not one introduced here, but walking every
published queue widens it considerably.

The in-driver callers of ibmveth_close() (veth_pool_store(),
ibmveth_set_csum_offload(), ibmveth_set_tso(), ibmveth_change_mtu()) call
ndo_stop directly rather than through dev_close(), so
netpoll_poll_disable() is never called and netif_running() stays true.
netpoll_poll_dev() can therefore call ndo_poll_controller() while close is
freeing the pools.

The replenish side samples the pool and then dereferences it:

		if (pool->active && pool->free_map &&
		    (atomic_read(&pool->available) < pool->threshold)) {
			rc = ibmveth_replenish_buffer_pool(adapter, pool,
							   queue_index, &fail);

while the freeing side takes no replenish_lock at all:

	kfree(pool->free_map);
	pool->free_map = NULL;

Can this access freed free_map[]/skbuff[]/dma_addr[] arrays, and the freed
buffer-list page in ibmveth_update_rx_no_buffer()?

A later patch in the series adds an "if (!adapter->opened) return;" check to
ibmveth_poll_controller(), but ibmveth_close() clears opened before it frees
anything and does not wait for an in-flight poll_controller, so the window
appears to remain.

> @@ -2781,23 +3056,35 @@ static unsigned long ibmveth_get_desired_dma(struct vio_dev *vdev)
>  
>  	adapter = netdev_priv(netdev);
>  
> -	ret = IBMVETH_BUFF_LIST_SIZE + IBMVETH_FILT_LIST_SIZE;
> +	/* One buffer list page per RX queue; filter list is shared. */
> +	ret = IBMVETH_BUFF_LIST_SIZE * ibmveth_get_num_rx_queues(adapter) +
> +	      IBMVETH_FILT_LIST_SIZE;
>  	ret += IOMMU_PAGE_ALIGN(netdev->mtu, tbl);
>  	/* add size of mapped tx buffers */
>  	ret += IOMMU_PAGE_ALIGN(IBMVETH_MAX_TX_BUF_SIZE, tbl);
>  
> +	for (q = 0; q < ibmveth_get_num_rx_queues(adapter); q++) {
> +		int rxqentries = 1;
> +
> +		for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
> +			struct ibmveth_buff_pool *bpool =
> +				&adapter->rx_buff_pool[0][i];
> +
> +			if (bpool->active)
> +				ret += bpool->size *
> +					IOMMU_PAGE_ALIGN(bpool->buff_size, tbl);
> +			rxqentries += bpool->size;
> +		}

[ ... ]

> @@ -2954,16 +3253,30 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
>  		netdev->features |= NETIF_F_FRAGLIST;
>  	}
>  
> -	/* Initialize queue count - always 1 for now */
> -	adapter->multi_queue = 0;
> -	adapter->num_rx_queues = IBMVETH_DEFAULT_RX_QUEUES;
> +	if (ret == H_SUCCESS &&
> +	    (ret_attr & IBMVETH_ILLAN_RX_MULTI_QUEUE_SUPPORT)) {
> +		adapter->multi_queue = true;
> +		ibmveth_publish_num_rx_queues(adapter,
> +					      min(num_online_cpus(),
> +						  IBMVETH_DEFAULT_QUEUES));

[ ... ]

> @@ -2981,25 +3294,62 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
>  
>  	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++) {
>  		struct kobject *kobj = &adapter->rx_buff_pool[0][i].kobj;
> -		int error;
>  
>  		ibmveth_init_buffer_pool(&adapter->rx_buff_pool[0][i], i,
>  					 pool_count[i], pool_size[i],
>  					 pool_active[i]);
> -		error = kobject_init_and_add(kobj, &ktype_veth_pool,
> -					     &dev->dev.kobj, "pool%d", i);
> -		if (!error)
> -			kobject_uevent(kobj, KOBJ_ADD);
> +		rc = kobject_init_and_add(kobj, &ktype_veth_pool,
> +					  &dev->dev.kobj, "pool%d", i);
> +		if (rc) {
> +			dev_err(&dev->dev,
> +				"failed to create pool%d kobject: %d\n", i, rc);
> +			/* init_and_add takes a ref even on failure */
> +			kobject_put(kobj);
> +			ibmveth_put_pool_kobjs(adapter, pools_ready);
> +			dev_set_drvdata(&dev->dev, NULL);
> +			free_netdev(netdev);
> +			return rc;
> +		}

[Severity: Medium]
Can this free the memory the kobjects live in before their cleanup runs?

The pool kobjects are embedded in struct ibmveth_adapter, which is
netdev_priv(netdev), and ktype_veth_pool has .release = NULL.  With
CONFIG_DEBUG_KOBJECT_RELEASE=y the last kobject_put() goes to
kobject_release(), which defers the cleanup:

lib/kobject.c:kobject_release() {
	...
#ifdef CONFIG_DEBUG_KOBJECT_RELEASE
	INIT_DELAYED_WORK(&kobj->release, kobject_delayed_cleanup);
	schedule_delayed_work(&kobj->release, delay);
	...
}

The delayed_work lives inside the kobject, i.e. inside the storage that
free_netdev() releases immediately on this path (reg_state is
NETREG_UNINITIALIZED), and the pool%d directories stay published against it
until the work fires a few seconds later.

The same put-then-free sequence is repeated on the
netif_set_real_num_tx_queues(), netif_set_real_num_rx_queues() and
register_netdev() failure paths below, and survives into the later
ibmveth_probe_cleanup() helper.  The changelog lists "pool kobj vs
DEBUG_KOBJECT_RELEASE" as a leftover - would adding a real release() with a
container that outlives the kobjects, or creating them only after the last
failure point, be the fix?

[Severity: Low]
This isn't a bug, but the policy here changes from "ignore
kobject_init_and_add() failure and keep probing" to "fail the probe", so a
kernfs name collision or -ENOMEM now leaves the interface unavailable
instead of merely missing the pool%d tuning knobs.  Checking the return
value is the right thing to do; could the changelog mention the behaviour
change, since it currently only describes adding cleanup?

> +	/*
> +	 * VIO CMO entitlement was set before probe (netdev NULL, so default).
> +	 * Recompute now that num_rx_queues and pool 0 metadata are known.
> +	 */
> +	if (firmware_has_feature(FW_FEATURE_CMO))
> +		vio_cmo_set_dev_desired(dev, ibmveth_get_desired_dma(dev));
> +

[Severity: Low]
This isn't a bug, but probe unconditionally selects min(num_online_cpus(),
IBMVETH_DEFAULT_QUEUES) RX queues whenever the firmware bit is set, and each
queue clones queue 0's full pool geometry plus its own buffer-list page and
RX ring.  ibmveth_get_desired_dma() therefore multiplies the desired DMA
entitlement by num_rx_queues, and vio_cmo_set_dev_desired() returns void, so
an entitlement request that cannot be granted is not visible here.

On a CMO partition with many CPUs, is defaulting to eight RX queues the
behaviour you want out of the box, or would defaulting to one and letting
ethtool -L opt in be safer?  The series does add the runtime opt-out in a
later patch.

>  	rc = netif_set_real_num_tx_queues(netdev, min(num_online_cpus(),
>  						      IBMVETH_DEFAULT_QUEUES));
>  	if (rc) {
>  		netdev_dbg(netdev, "failed to set number of tx queues rc=%d\n",
>  			   rc);
> +		ibmveth_put_pool_kobjs(adapter, pools_ready);
> +		dev_set_drvdata(&dev->dev, NULL);
> +		free_netdev(netdev);
> +		return rc;
> +	}
> +
> +	/*
> +	 * alloc_etherdev_mqs() sized real_num_rx_queues to MAX. Match the
> +	 * advertised default (or SQ 1) before register_netdev so down-state
> +	 * readers agree with adapter->num_rx_queues / ethtool -l.
> +	 */
> +	rc = netif_set_real_num_rx_queues(netdev,
> +					  ibmveth_get_num_rx_queues(adapter));

[Severity: Low]
With real_num_rx_queues now published as up to eight before
register_netdev(), ibmveth_get_channels() starts reporting max_rx and
rx_count greater than one:

	channels->max_rx = netdev->real_num_rx_queues;
	channels->rx_count = netdev->real_num_rx_queues;

but ibmveth_set_channels() at this commit only reads channels->tx_count.
Does "ethtool -L ethX rx 4" then pass the core's max_rx check, get silently
discarded by the driver and report success?

The changelog acknowledges this ("This commit does not implement
set_channels / rx_count") and patch 14 wires rx_count up, so the question is
whether the advertisement should wait for the implementation.

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,11/15] ibmveth: Add per-queue RX and TX statistics collection

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider:

Critical: 0 · High: 0 · Medium: 6 · Low: 0

- [Medium] The patch bundles an independently applicable use-after-free
  fix for the reset worker into a statistics feature patch and…
- [Medium] The new ibmveth_get_stats64() populates rx_packets/rx_bytes
  and tx_packets/tx_bytes/tx_dropped but never sets rx_missed_errors or…
- [Medium] ibmveth_get_queue_stats_tx() (and the retired-queue part of
  ibmveth_get_base_stats()) exports the driver's software TX drop…
- [Medium] The new per-queue ethtool -S key tables add driver-private
  strings for counters that already have standard uAPI fields, while the…
- [Medium] ibmveth_update_rx_no_buffer() detects a buffer-list page
  replacement only by the heuristic `if (drops <…
- [Medium] ibmveth_close() calls ibmveth_update_rx_no_buffer() for every
  live queue without holding that queue's replenish_lock, while netpoll…

--

Below is an automated review report for this commit.

ibmveth: Add per-queue RX and TX statistics collection

This moves the ibmveth RX and TX counters out of struct ibmveth_adapter
into per-queue arrays allocated at probe and freed at remove, adds
netdev_stat_ops and ndo_get_stats64, and makes the ethtool -S block
variable length with new per-queue keys.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 2e8896ea5af2e..f4fddfa56571d 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... ]

> @@ -75,32 +76,101 @@ module_param(old_large_send, bool, 0444);

[ ... ]

> +/*
> + * Only counters with no home in the standard interfaces belong here.
> + * packets, bytes and drops are reported through netdev_stat_ops.
> + */
> +static const struct ibmveth_qstat ibmveth_rx_qstat_keys[] = {
> +	{ "rx%d_interrupts", IBMVETH_RXQ_OFF(interrupts) },
> +	{ "rx%d_polls", IBMVETH_RXQ_OFF(polls) },
> +	{ "rx%d_large_packets", IBMVETH_RXQ_OFF(large_packets) },
> +	{ "rx%d_invalid_buffers", IBMVETH_RXQ_OFF(invalid_buffers) },
> +	{ "rx%d_no_buffer_drops", IBMVETH_RXQ_OFF(no_buffer_drops) },
> +};
> +
> +static const struct ibmveth_qstat ibmveth_tx_qstat_keys[] = {
> +	{ "tx%d_large_packets", IBMVETH_TXQ_OFF(large_packets) },
> +	{ "tx%d_send_failures", IBMVETH_TXQ_OFF(send_failures) },
> +	{ "tx%d_checksum_offload", IBMVETH_TXQ_OFF(checksum_offload) },
> +};

[Severity: Medium]
Do some of these new ethtool -S keys duplicate counters that already have
a standard netlink representation?

The comment above the tables says only counters with no home in the
standard interfaces belong here, and the changelog says ethtool -S keeps
"only the driver-specific keys that have no standard equivalent". Looking
at Documentation/netlink/specs/netdev.yaml:

  tx-hw-gso-packets: Number of packets that necessitated segmentation
                     into smaller packets by the device
  tx-needs-csum:     Number of packets that required the device to
                     calculate the checksum
  rx-hw-drop-overruns: Number of packets dropped due to transient lack of
                     resources, such as buffer space, host descriptors etc.

tx%d_large_packets is the large-send/GSO packet count, tx%d_checksum_offload
is incremented exactly where the driver asks the device to compute the
checksum, and rx%d_no_buffer_drops is a buffer-exhaustion drop count.

Meanwhile the new callbacks added in this same patch fill only packets,
bytes, hw_drops and alloc_fail, leaving hw_gso_packets, needs_csum and
hw_drop_overruns unset. Could those standard fields be filled in
ibmveth_get_queue_stats_rx()/ibmveth_get_queue_stats_tx() instead of
adding private strings, so the values are not exported through two
different interfaces?

rx%d_invalid_buffers is also summed into hw_drops by
ibmveth_get_queue_stats_rx(), so it appears in both places as well.

[ ... ]

> @@ -1017,6 +1143,7 @@ static int ibmveth_replenish_buffer_pool(struct ibmveth_adapter *adapter,
>  static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
>  					int queue_index)
>  {
> +	struct ibmveth_rx_queue_stats *qstats;
>  	__be64 *p;
>  	u64 drops;
>  
> @@ -1028,7 +1155,18 @@ static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
>  	p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
>  	drops = be64_to_cpup(p);
>  
> -	adapter->rx_no_buffer = drops;
> +	/*
> +	 * PHYP's buffer-list page counter is absolute for that page. A new
> +	 * page (reopen / queue reuse after -L) starts near zero; fold the
> +	 * previous absolute into this queue's retired carry so sums stay
> +	 * monotonic. Both fields belong to the queue being updated, so this
> +	 * stays single-writer under the queue's replenish_lock.
> +	 */
> +	qstats = &adapter->rx_qstats[queue_index];
> +
> +	if (drops < qstats->no_buffer_drops)
> +		qstats->no_buffer_retired += qstats->no_buffer_drops;
> +	qstats->no_buffer_drops = drops;
>  }

[Severity: Medium]
Can the retired page's count be lost when the new page is already ahead
of the old value at the first sample?

The only signal that the buffer-list page was replaced is the numeric
decrease. ibmveth_close() samples the counter and leaves it in
no_buffer_drops:

	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
		ibmveth_update_rx_no_buffer(adapter, i);
	ibmveth_free_all_queues(adapter);

and the next ibmveth_open() gets a fresh zeroed page from
ibmveth_alloc_rx_queues():

		adapter->buffer_list_addr[i] =
			(void *)get_zeroed_page(GFP_KERNEL);

Between ibmveth_register_rx_queues() and the first per-queue
ibmveth_replenish_task() in ibmveth_open() the LAN is live with no buffers
posted, so PHYP can push the new page's counter up before the driver reads
it. If the previous session ended at 50 and the new page reads 500 on the
first sample, the "drops < qstats->no_buffer_drops" test is false and those
50 drops disappear from the adapter-wide rx_no_buffer key, from per-queue
rx-hw-drops and from ibmveth_get_base_stats().

Would recording the retirement at the point the page is actually released
(close and queue reuse) be more reliable than inferring it from a decrease?

[Severity: Medium]
This turns a single idempotent store into a two-field accumulating
read-modify-write. Is every caller holding that queue's replenish_lock?

ibmveth_close() calls it without the lock:

	for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
		ibmveth_update_rx_no_buffer(adapter, i);

while netpoll can run the same update concurrently under the lock:

  netpoll_poll_dev() -> ndo_poll_controller
    ibmveth_poll_controller()
      ibmveth_replenish_task(adapter, i)
        spin_lock_irqsave(&rxq->replenish_lock, flags)
        ...
        ibmveth_update_rx_no_buffer(adapter, queue_index)

ibmveth_close() is also called directly from ibmveth_set_csum_offload(),
ibmveth_set_tso(), ibmveth_change_mtu() and veth_pool_store() while
netif_running() is still true, so netpoll_poll_disable() has not run.
netpoll_poll_dev() only bails on a missing netpoll info, a held dev_lock,
!netif_running() or local xmit activity, and ndo_poll_controller is not
gated by NAPI state.

With the old single store the worst case was a stale snapshot; with the
new "no_buffer_retired += no_buffer_drops; no_buffer_drops = drops"
sequence an interleaving can double the carry or lose it permanently.

The RX shrink path later in the series wraps the identical call:

	spin_lock_irqsave(&rxq->replenish_lock, flags);
	ibmveth_update_rx_no_buffer(adapter, i);
	spin_unlock_irqrestore(&rxq->replenish_lock, flags);

with the comment that netpoll still reaches these queues. Should the close
path take the same lock?

[ ... ]

> @@ -2038,6 +2176,10 @@ static void ibmveth_reset(struct work_struct *w)
>  	netdev_dbg(netdev, "reset starting\n");
>  
>  	rtnl_lock();
> +	if (netdev->reg_state != NETREG_REGISTERED) {
> +		rtnl_unlock();
> +		return;
> +	}
>  
>  	dev_close(adapter->netdev);
>  	dev_open(adapter->netdev, NULL);

[Severity: Medium]
Could this reg_state gate, together with the ibmveth_remove() reorder
further down, be split out as its own patch with a Fixes: tag?

The changelog acknowledges it: "That reorder is a use-after-free fix in
its own right; it is carried here because this patch depends on it. No
Fixes: tag".

At the baseline the ordering defect is real. ibmveth_remove() has:

	cancel_work_sync(&adapter->work);

	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
		kobject_put(&adapter->rx_buff_pool[i].kobj);

	unregister_netdev(netdev);

and the RX path can re-arm the work after that cancel:

  ibmveth_poll_skip_bad_correlator() -> schedule_work(&adapter->work)
  ibmveth_replenish_task()           -> schedule_work(&adapter->work)

so the worker can run after free_netdev(). Both halves of that fix are
here inside a roughly 500 line feature commit with no Fixes: tag, which
makes the fix hard to identify or backport on its own.

[ ... ]

> @@ -3132,6 +3372,124 @@ static netdev_features_t ibmveth_features_check(struct sk_buff *skb,
>  	return vlan_features_check(skb, features);
>  }
>  
> +/**
> + * ibmveth_get_stats64 - Return aggregated per-queue statistics
> + * @dev: network device
> + * @stats: rtnl link statistics storage
> + *
> + * Sums per-queue rx_qstats and tx_qstats into the rtnl counters.
> + * Walk the full allocated arrays (not the live queue count) so shrinking
> + * channels cannot make the totals go backwards.
> + * Callers use ndo_get_stats64(); avoid updating netdev->stats on the
> + * xmit/poll paths to keep per-queue counters off the hot cache line.
> + */
> +static void ibmveth_get_stats64(struct net_device *dev,
> +				struct rtnl_link_stats64 *stats)
> +{
> +	struct ibmveth_adapter *adapter = netdev_priv(dev);
> +	int i;
> +
> +	for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
> +		stats->rx_packets += adapter->rx_qstats[i].packets;
> +		stats->rx_bytes += adapter->rx_qstats[i].bytes;
> +	}
> +
> +	for (i = 0; i < IBMVETH_MAX_QUEUES; i++) {
> +		stats->tx_packets += adapter->tx_qstats[i].packets;
> +		stats->tx_bytes += adapter->tx_qstats[i].bytes;
> +		stats->tx_dropped += adapter->tx_qstats[i].dropped_packets;
> +	}
> +}

[Severity: Medium]
Should this also fill rx_missed_errors or rx_dropped?

The RX loop reports only packets and bytes, while the TX loop does report
tx_dropped, so the RX drops the driver already tracks are not visible in
rtnl_link_stats64 at all.

include/uapi/linux/if_link.h describes the matching field:

 * @rx_missed_errors: Count of packets missed by the host.
 *   Counts number of packets dropped by the device due to lack
 *   of buffer space.

which is exactly what PHYP's buffer-list page counter records and what
ibmveth_get_queue_stats_rx() below exports as rx-hw-drops.

Documentation/networking/statistics.rst says:

  Drivers should report all statistics which have a matching member in
  struct rtnl_link_stats64 exclusively via `.ndo_get_stats64`.

dev_get_stats() does not derive rtnl stats from netdev_stat_ops, so
"ip -s -s link" shows zero RX drops on a device whose qstats and
ethtool -S rx_no_buffer key are both climbing. Is that divergence
intended?

> +
> +static void ibmveth_get_queue_stats_rx(struct net_device *dev, int idx,
> +				       struct netdev_queue_stats_rx *stats)
> +{

[ ... ]

> +static void ibmveth_get_queue_stats_tx(struct net_device *dev, int idx,
> +				       struct netdev_queue_stats_tx *stats)
> +{
> +	struct ibmveth_adapter *adapter = netdev_priv(dev);
> +
> +	stats->packets = adapter->tx_qstats[idx].packets;
> +	stats->bytes = adapter->tx_qstats[idx].bytes;
> +	stats->hw_drops = adapter->tx_qstats[idx].dropped_packets;
> +}

[Severity: Medium]
Does dropped_packets match what tx-hw-drops is specified to count?

Documentation/netlink/specs/netdev.yaml:

        name: tx-hw-drops
        doc: |
          Number of packets that arrived at the device but never left it,

In ibmveth_start_xmit() the counter is bumped for several conditions that
happen entirely in the driver, before ibmveth_send() issues any hypercall:

	if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
		adapter->tx_qstats[queue_num].dropped_packets++;
		goto out;
	}

plus the ibmveth_is_packet_unsupported() loopback rejection, the
skb_checksum_help() failure, skb->len > adapter->tx_ltb_size, and
total_bytes != skb->len. None of those packets reached the device.

The loopback case is reachable by any local sender addressing the
device's own MAC. Would rtnl_link_stats64::tx_dropped alone (which this
patch already populates) be the right home for these, keeping hw_drops
for the ibmveth_send() failure only?

The same mapping is used for the retired queues in
ibmveth_get_base_stats():

	tx->hw_drops += adapter->tx_qstats[i].dropped_packets;

[ ... ]

> @@ -3380,12 +3755,20 @@ static void ibmveth_remove(struct vio_dev *dev)
>  	struct ibmveth_adapter *adapter = netdev_priv(netdev);
>  	int i;
>  
> -	cancel_work_sync(&adapter->work);
> -
>  	for (i = 0; i < IBMVETH_NUM_BUFF_POOLS; i++)
>  		kobject_put(&adapter->rx_buff_pool[0][i].kobj);
>  
> +	/*
> +	 * Unregister first so NAPI/xmit cannot re-arm reset work after we
> +	 * cancel it. cancel_work_sync() before unregister left a window
> +	 * where poll could schedule_work() and the worker ran after
> +	 * free_netdev().
> +	 */
>  	unregister_netdev(netdev);
> +	cancel_work_sync(&adapter->work);
> +
> +	ibmveth_free_tx_qstats(adapter);
> +	ibmveth_free_rx_qstats(adapter);
>  
>  	free_netdev(netdev);
>  	dev_set_drvdata(&dev->dev, NULL);

This is the second half of the reordering discussed above at
ibmveth_reset(); the same question about lifting it into its own
Fixes:-tagged patch applies here.

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,12/15] ibmveth: Report MQ-aware RX counts in ethtool get_channels

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 3 potential
issue(s) to consider:

Critical: 0 · High: 0 · Medium: 1 · Low: 2

- [Medium] Intermediate-state ethtool channel ABI mismatch:
  ibmveth_get_channels() (drivers/net/ethernet/ibm/ibmveth.c:2537-2549)…
- [Low] Advertised RX maximum not derived from any driver-side capability
  check: ibmveth_get_channels() (ibmveth.c:2545-2548) reports a flat…
- [Low] ibmveth_get_channels()
  (drivers/net/ethernet/ibm/ibmveth.c:2540-2549) now sources…

--

ibmveth: Report MQ-aware RX counts in ethtool get_channels

get_channels() now reports max_rx as IBMVETH_MAX_RX_QUEUES when MQ firmware
is enabled (1 otherwise) and reports rx_count from the driver's published
num_rx_queues instead of netdev->real_num_rx_queues.  The paired
set_channels() is left unchanged in this patch.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index f4fddfa56571d..4f2d956b4c89e 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c
> @@ -2537,11 +2537,16 @@ static void ibmveth_get_ethtool_stats(struct net_device *dev,
>  static void ibmveth_get_channels(struct net_device *netdev,
>  				 struct ethtool_channels *channels)
>  {
> +	struct ibmveth_adapter *adapter = netdev_priv(netdev);
> +
>  	channels->max_tx = ibmveth_real_max_tx_queues();
>  	channels->tx_count = netdev->real_num_tx_queues;
>  
> -	channels->max_rx = netdev->real_num_rx_queues;
> -	channels->rx_count = netdev->real_num_rx_queues;
> +	if (adapter->multi_queue)
> +		channels->max_rx = IBMVETH_MAX_RX_QUEUES;
> +	else
> +		channels->max_rx = 1;

[Severity: Medium]
Does raising max_rx here make the kernel admit RX channel requests that the
driver then silently drops on the floor?

The core only bounds an RX request against the value this callback returns,
in ethtool_set_channels():

	if (channels.rx_count > curr.max_rx ||
	    channels.tx_count > curr.max_tx ||

and ethnl_set_channels() does the same check on the netlink path.
ibmveth_set_channels() at this commit only consumes tx_count:

	unsigned int old = netdev->real_num_tx_queues,
		     goal = channels->tx_count;

channels->rx_count is neither read nor rejected.  Since ethtool -L is a
read-modify-write, tx_count arrives unchanged and the TX grow/shrink loops
do not run, so "ethtool -L ethN rx 4" on an MQ adapter returns 0, generates
an ETHTOOL_MSG_CHANNELS_NTF, and changes nothing; a following "ethtool -l"
still shows the old rx_count.

Before this patch max_rx equalled the live count, so the same request was
rejected with -EINVAL by the core, which is why this reads as a behaviour
change rather than a pre-existing gap.

The commit message does acknowledge this:

  "This patch only reports. set_channels() still reads only tx_count, so
  an RX channel request returns success without changing anything;
  raising max_rx widens the range of requests that behave that way from
  the live count to IBMVETH_MAX_RX_QUEUES."

and the later patches do close it: by the end of the series
ibmveth_set_channels() reads channels->rx_count and calls
ibmveth_resize_rx_channels(), which validates the range:

	if (goal_rx < 1 || goal_rx > IBMVETH_MAX_RX_QUEUES) {
		...
		return -EINVAL;
	}

Would it be possible to order the widening of max_rx after the setter
learns to consume rx_count, so no bisect point reports success for a
request it ignores?

[Severity: Low]
Should this ceiling be derived from a capability check rather than being a
flat 16?

The TX side bounds its advertised maximum by the CPU count:

	static unsigned int ibmveth_real_max_tx_queues(void)
	{
		unsigned int n_cpu = num_online_cpus();

		return min(n_cpu, IBMVETH_MAX_QUEUES);
	}

while the RX side reports IBMVETH_MAX_RX_QUEUES (16) purely on
adapter->multi_queue, even though the driver's own MQ default is
min(num_online_cpus(), IBMVETH_DEFAULT_QUEUES), which is at most 8.

The test also ignores adapter->mq_fallback.  After firmware rejects MQ,
multi_queue stays true until the next open runs
ibmveth_apply_mq_fallback():

	adapter->multi_queue = false;
	ibmveth_publish_num_rx_queues(adapter, 1);
	/* real_num_rx_queues is set later in open after resources exist. */

so in that window ethtool -l advertises max_rx = 16 for a device the driver
has already decided will come up single-queue.

The mq_fallback half is handled later in the series, where get_channels()
becomes:

	if (adapter->multi_queue && !adapter->mq_fallback)
		channels->max_rx = IBMVETH_MAX_RX_QUEUES;
	else
		channels->max_rx = rx_count;

The flat 16 does match the range the end-of-series setter accepts, so this
is mostly a question about whether the commit message should say why the RX
maximum intentionally skips the num_online_cpus() cap that TX applies.

> +	channels->rx_count = ibmveth_get_num_rx_queues(adapter);

[Severity: Low]
Can this disagree with the RX queue count the netdev core enumerates?

The queue-stats side of the same driver deliberately keys on the core's
count, in ibmveth_get_base_stats():

	for (i = dev->real_num_rx_queues; i < IBMVETH_MAX_RX_QUEUES; i++) {

with a comment saying the split is bound by real_num_*_queues so it lines up
with the core's iteration exactly.  After this patch, ethtool -l reports the
adapter's private count instead.

The two can drift.  ibmveth_apply_mq_fallback() publishes num_rx_queues = 1
at the very start of open, and netif_set_real_num_rx_queues() is only
reached much later:

	rc = ibmveth_register_rx_queues(adapter, mac_address);
	if (rc)
		goto out_free_buffer_pools;

	rc = netif_set_real_num_rx_queues(netdev,
					  ibmveth_get_num_rx_queues(adapter));

If ibmveth_alloc_filter_list(), ibmveth_alloc_rx_queues(),
ibmveth_alloc_buffer_pools() or ibmveth_register_rx_queues() fails, open
returns an error with adapter->num_rx_queues == 1 while
netdev->real_num_rx_queues still holds the previous value, and
ibmveth_close() never lowers it.

In that state ethtool -l reports rx_count = 1, the core still enumerates the
old number of RX queues for netlink per-queue stats, and the per-queue
ethtool -S strings (which use the adapter count) list only queue 0.

No out-of-bounds access results, since rx_qstats[] is sized
IBMVETH_MAX_RX_QUEUES and the live/retired split stays non-overlapping, and
the state self-heals on the next successful open.  This divergence is still
present at the end of the series, where get_channels() uses the adapter
count and get_base_stats() uses real_num_rx_queues.  Would keying both on
the same counter be preferable?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,14/15] ibmveth: Implement incremental MQ RX queue resize

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 9 potential
issue(s) to consider:

Critical: 0 · High: 0 · Medium: 5 · Low: 4

- [Medium] On scale-up, ibmveth_resize_rx_queues_incremental() publishes,
  replenishes, NAPI-enables, unmasks PHYP and restarts each new queue…
- [Medium] The new scale-up call site of
  ibmveth_register_single_rx_queue() in…
- [Medium] ibmveth_deregister_single_rx_queue() ignores a failing
  H_FREE_LOGICAL_LAN_QUEUE: the retry loop covers only…
- [Medium] The scale-down and cleanup_new_queues quiesce sequences assume
  napi_disable() plus a second ibmveth_disable_irq() + synchronize_irq()…
- [Medium] RX queue teardown proceeds without confirming that PHYP
  interrupt delivery is actually masked.
- [Low] ibmveth_set_channels() commits the RX resize before any TX work
  and, in this patch, has no RX rollback if the TX side then fails, so a…
- [Low] When the interface is not opened, ibmveth_resize_rx_channels()
  validates the requested RX count and returns 0 without publishing it,…
- [Low] The advertised ethtool channel range and the set path disagree
  once adapter->mq_fallback is latched: ibmveth_get_channels() reports…
- [Low] ibmveth_drain_rx_queue()'s kernel-doc says it "Recycles all
  pending buffers back to the per-queue buffer pools" and "Return:…

--

ibmveth: Implement incremental MQ RX queue resize

This adds ibmveth_resize_rx_queues_incremental() so the live RX channel
count can change without a full device tear-down, plus
ibmveth_resize_rx_channels() as the entry point set_channels() uses. CMO
entitlement math is factored into ibmveth_desired_dma_for_rxqs() so a
prospective queue count can be sized before the driver commits to it.

A few questions about the new resize paths below.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 954846c9ec7b3..5aef8a1f2c23d 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c

[ ... ]

> @@ -1613,6 +1808,56 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
>  	return 0;
>  }
>  
> +/**
> + * ibmveth_drain_rx_queue - Drain pending buffers from an RX queue
> + * @adapter: ibmveth adapter structure
> + * @queue_index: Queue index to drain
> + *
> + * Recycles all pending buffers back to the per-queue buffer pools.
> + * Must be called with NAPI disabled for this queue.
> + *
> + * Return: Number of buffers drained
> + */
> +static int
> +ibmveth_drain_rx_queue(struct ibmveth_adapter *adapter, int queue_index)
> +{
> +	struct net_device *netdev = adapter->netdev;
> +	int drained = 0;
> +	int limit = adapter->rx_queue[queue_index].num_slots;
> +	int rc;
> +
> +	netdev_dbg(netdev, "Draining RX queue %d (limit: %d slots)\n",
> +		   queue_index, limit);
> +
> +	while (drained < limit &&
> +	       ibmveth_rxq_pending_buffer(adapter, queue_index)) {
> +		/* Match poll-side order before harvesting completion state. */
> +		smp_rmb();
> +		rc = ibmveth_rxq_harvest_buffer(adapter, queue_index, true);
> +		if (rc) {
> +			/* -EINVAL/-EFAULT already advanced past the slot. */
> +			if (rc == -EINVAL || rc == -EFAULT) {
> +				drained++;
> +				continue;
> +			}

[Severity: Low]

This isn't a bug, but does the documented return value of
ibmveth_drain_rx_queue() match what is counted here?

The kernel-doc says "Recycles all pending buffers back to the per-queue
buffer pools" and "Return: Number of buffers drained", yet this branch
bumps drained for entries where nothing was recycled.
ibmveth_rxq_harvest_buffer() only advances the ring for -EINVAL/-EFAULT:

	if (unlikely(rc)) {
		/* Skip a corrupt slot without claiming pool ownership. */
		if (rc == -EINVAL || rc == -EFAULT)
			ibmveth_rxq_advance(rxq);
		return rc;
	}

So the value returned is the number of ring slots consumed, not buffers
recycled. Would wording the kernel-doc as "slots processed" be more
accurate, given the value only feeds netdev_dbg()?

> +			netdev_err(netdev,
> +				   "Failed to harvest buffer from queue %d during drain: %d\n",
> +				   queue_index, rc);
> +			break;
> +		}
> +		drained++;
> +	}

[ ... ]

> @@ -1881,6 +2126,378 @@ ibmveth_register_single_rx_queue(struct ibmveth_adapter *adapter,
>  	return 0;
>  }
>  
> +/**
> + * ibmveth_deregister_single_rx_queue - Deregister one subordinate RX queue
> + * @adapter: ibmveth adapter structure
> + * @queue_idx: Queue index to deregister (1..N)
> + *
> + * Deregisters a single queue via H_FREE_LOGICAL_LAN_QUEUE. Linux IRQ handler
> + * teardown and subordinate virq mapping disposal are owned by interrupt
> + * cleanup helpers; queue 0 is freed only through ibmveth_free_all_queues()
> + * (H_FREE_LOGICAL_LAN).
> + */
> +static void
> +ibmveth_deregister_single_rx_queue(struct ibmveth_adapter *adapter,
> +				   int queue_idx)
> +{
> +	unsigned long lpar_rc;
> +	unsigned long ua = adapter->vdev->unit_address;
> +	unsigned long qh = adapter->queue_handle[queue_idx];
> +
> +	if (!qh)
> +		return;
> +
> +	do {
> +		lpar_rc = h_free_logical_lan_queue(ua, qh);
> +	} while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
> +
> +	if (lpar_rc != H_SUCCESS) {
> +		netdev_err(adapter->netdev,
> +			   "h_free_logical_lan_queue failed for queue %d: rc=0x%lx\n",
> +			   queue_idx, lpar_rc);
> +	}
> +
> +	adapter->queue_handle[queue_idx] = 0;

[Severity: Medium]

Can a failing H_FREE_LOGICAL_LAN_QUEUE here lead to PHYP writing into
memory that has already been released?

The retry loop only covers H_BUSY/H_LONG_BUSY. Any other status is
logged, queue_handle[queue_idx] is cleared anyway, and the function
returns void, so no caller can observe or retry it.
ibmveth_destroy_subordinate_rx_queue() then continues unconditionally:

	ibmveth_deregister_single_rx_queue(adapter, queue_idx);
	ibmveth_cleanup_single_rx_interrupt(adapter, queue_idx);
	ibmveth_free_single_rx_queue(adapter, queue_idx);

ibmveth_free_single_rx_queue() dma_unmap_single()s the buffer-list page,
dma_free_coherent()s the RX ring and releases every RX skb of that queue,
all on a live adapter, while ibmveth_resize_rx_queues_incremental()
reports success to ethtool.

The kernel-doc for ibmveth_destroy_subordinate_rx_queue() states the
ordering rationale as "Deregister with PHYP before unmapping buffer pools
so hypervisor buffer ownership is released while queue metadata is still
valid", which relies on the hcall having succeeded.

The driver already treats this class of hcall failure as unrecoverable
elsewhere, for example the scale-down rollback and ibmveth_poll() both do
schedule_work(&adapter->work). Should this helper return the status so
the caller can keep the queue and schedule a reset instead of freeing?

> +
> +	netdev_dbg(adapter->netdev, "Deregistered queue %d\n", queue_idx);
> +}
> +
> +/**
> + * ibmveth_destroy_subordinate_rx_queue - Tear down one subordinate RX queue
> + * @adapter: ibmveth adapter structure
> + * @queue_idx: Queue index to destroy (1..N)
> + *
> + * Deregister with PHYP before unmapping buffer pools so hypervisor buffer
> + * ownership is released while queue metadata is still valid.
> + */
> +static void
> +ibmveth_destroy_subordinate_rx_queue(struct ibmveth_adapter *adapter,
> +				     int queue_idx)
> +{
> +	ibmveth_deregister_single_rx_queue(adapter, queue_idx);
> +	ibmveth_cleanup_single_rx_interrupt(adapter, queue_idx);
> +	ibmveth_free_single_rx_queue(adapter, queue_idx);
> +}

[ ... ]

> +static int
> +ibmveth_resize_rx_queues_incremental(struct ibmveth_adapter *adapter,
> +				     int new_count, int rxq_entries)
> +{

[ ... ]

> +		for (i = old_count; i < new_count; i++) {
> +			rc = ibmveth_alloc_single_rx_queue(adapter, i,
> +							   rxq_entries);
> +			if (rc) {
> +				netdev_err(netdev, "Failed to allocate queue %d: %d\n",
> +					   i, rc);
> +				goto cleanup_new_queues;
> +			}
> +
> +			rc = ibmveth_register_single_rx_queue(adapter, i);
> +			if (rc) {
> +				netdev_err(netdev, "Failed to register queue %d: %d\n",
> +					   i, rc);
> +				ibmveth_free_single_rx_queue(adapter, i);
> +				goto cleanup_new_queues;
> +			}

[Severity: Medium]

Should this call site latch adapter->mq_fallback when
ibmveth_register_single_rx_queue() returns -EOPNOTSUPP?

The helper maps a firmware H_FUNCTION to -EOPNOTSUPP, and the other
caller, ibmveth_register_rx_queues(), does:

	/* Firmware MQ gone: fall back to SQ on next open. */
	if (rc == -EOPNOTSUPP)
		adapter->mq_fallback = true;
	goto err_unregister;

ibmveth_register_logical_lan_queue() documents the same obligation:
"Caller fails this open and latches mq_fallback so the next open applies
SQ".

Here the errno is only logged and propagated, so after a firmware-refused
scale-up the driver still believes MQ is available: get_channels() keeps
advertising growth, ibmveth_resize_rx_channels() keeps accepting rx > 1,
and the next ndo_open() re-attempts subordinate registration, fails the
whole open and only then sets the flag. Does that turn into an avoidable
failed "ip link set up" before the device comes back single-queue?

> +
> +			rc = ibmveth_setup_single_rx_interrupt(adapter, i);
> +			if (rc) {
> +				netdev_err(netdev,
> +					   "Failed to setup IRQ for queue %d: %d\n",
> +					   i, rc);
> +				/* request_irq failed: mapped but no handler */
> +				ibmveth_dispose_subordinate_irq_mapping(adapter,
> +									i);
> +				ibmveth_deregister_single_rx_queue(adapter, i);
> +				ibmveth_free_single_rx_queue(adapter, i);
> +				goto cleanup_new_queues;
> +			}
> +
> +			/*
> +			 * Fully ready before PHYP delivery, matching open():
> +			 * publish -> replenish -> napi_enable -> enable_irq.
> +			 * That way ibmveth_interrupt() cannot run on an
> +			 * unpublished, empty, or NAPI-disabled queue.
> +			 */
> +			ibmveth_publish_num_rx_queues(adapter, i + 1);
> +			ibmveth_replenish_task(adapter, i);
> +			napi_enable(&adapter->napi[i]);
> +
> +			rc = ibmveth_enable_irq(adapter, i);
> +			if (rc) {
> +				netdev_err(netdev,
> +					   "Failed to enable IRQ for queue %d: %d\n",
> +					   i, rc);
> +				/*
> +				 * Published, replenished, and NAPI-enabled,
> +				 * but PHYP never unmasked. Match scale-down /
> +				 * shared cleanup: drain posted buffers, then
> +				 * deregister before unmap via
> +				 * destroy_subordinate.

[Severity: Medium]

Is the comment's "but PHYP never unmasked" guaranteed on this path?

ibmveth_toggle_irq() documents the enable side as ambiguous:

	/*
	 * H_PARAMETER is ambiguous (already in requested state vs bad
	 * args). Fold only on disable as an idempotent mask. On enable
	 * keep it an error so a stuck-masked queue stays visible to
	 * poll/resize recovery.
	 */

So an enable returning -EIO can also mean the interrupt was already
enabled. The queue has been replenished and NAPI-enabled at this point,
so PHYP can deliver, yet this branch goes straight to napi_disable(),
publish-down, drain and destroy with no ibmveth_disable_irq() +
synchronize_irq() for queue i.

The scale-down and cleanup_new_queues paths do issue the mask, but
discard its -EIO result and drain/destroy regardless, and
synchronize_irq() only waits for handlers already running rather than
masking the PHYP source. With the index at or above the published count
ibmveth_interrupt() also just returns IRQ_NONE without masking:

	if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
		return IRQ_NONE;

Should a failed mask be treated the way ibmveth_poll() and the
scale-down rollback treat a failed enable, with
schedule_work(&adapter->work) instead of proceeding?

> +				 *
> +				 * napi_disable() must come BEFORE the count
> +				 * is lowered, matching scale-down and
> +				 * cleanup_new_queues. Lowering it first does
> +				 * not hide queue i from netpoll: after
> +				 * ndo_poll_controller, netpoll_poll_dev()
> +				 * calls poll_napi(), which walks dev->napi_list
> +				 * unbounded by the queue count and skips a NAPI
> +				 * only once NAPI_STATE_NPSVC is set. Queue i is
> +				 * enabled here, so ibmveth_poll() would run and
> +				 * trip its queue_index >= num_rx_queues
> +				 * WARN_ON. napi_disable() sets NPSVC, so
> +				 * poll_napi() skips the queue instead.
> +				 */
> +				napi_disable(&adapter->napi[i]);
> +				ibmveth_publish_num_rx_queues(adapter, i);
> +				ibmveth_drain_rx_queue(adapter, i);
> +				synchronize_net();
> +				ibmveth_destroy_subordinate_rx_queue(adapter,
> +								     i);
> +				/* enable_irq returns errno; keep -EIO. */
> +				rc = -EIO;
> +				goto cleanup_new_queues;
> +			}
> +			ibmveth_restart_rx_queue(adapter, i);
> +		}
> +
> +		rc = netif_set_real_num_rx_queues(netdev, new_count);

[Severity: Medium]

Can this ordering be inverted so the WARN window does not exist at all?

Every new queue is published, replenished, NAPI-enabled, unmasked at PHYP
and restarted inside the loop above, and real_num_rx_queues is only
widened after the loop finishes. A frame arriving on a new queue in that
window is tagged by ibmveth_poll_deliver_frame():

	skb_record_rx_queue(skb, queue_index);
	napi_gro_receive(napi, skb);

and net/core/dev.c get_rps_cpu() then does:

	if (unlikely(index >= dev->real_num_rx_queues)) {
		WARN_ONCE(dev->real_num_rx_queues > 1,
			  "%s received packet on queue %u, but number "
			  "of RX queues is %u\n", ...);
		goto done;
	}

netif_get_rxqueue() has the same WARN_ONCE for the generic XDP path, so a
device that already had more than one RX queue splats when RPS/RFS or
generic XDP is configured, and panic_on_warn systems go down.

netif_set_real_num_rx_queues() only needs dev->num_rx_queues, which is
sized at probe, so could it be called before PHYP delivery is unmasked
and narrowed again on the cleanup path? The commit message describes the
splat as expected:

    "RPS then skips steering and takes the default path, tripping the
    WARN_ONCE in get_rps_cpu() if the device already had more than one
    queue. Transient, and delivery is unaffected."

Is leaving a userspace-reachable WARN in place preferable to widening the
count first?

> +		if (rc) {
> +			netdev_err(netdev, "Failed to set real RX queues to %d: %d\n",
> +				   new_count, rc);
> +			goto cleanup_new_queues;
> +		}
> +	} else {
> +		netdev_dbg(netdev, "Scale-down: removing queues %d-%d\n",
> +			   new_count, old_count - 1);
> +
> +		/*
> +		 * Mask PHYP before napi_disable so the handler cannot miss
> +		 * a mask while NAPI is already dead. An in-flight poll can
> +		 * still re-arm PHYP while napi_disable() waits, so remask
> +		 * and sync again after NAPI is stopped. Then drain, harvest
> +		 * final no_buffer stats, publish the surviving count, and
> +		 * synchronize_net() before destroy so netpoll cannot walk
> +		 * dying queues (handler may still be registered until destroy).
> +		 */
> +		for (i = new_count; i < old_count; i++) {
> +			if (!adapter->queue_irq[i])
> +				continue;
> +			ibmveth_disable_irq(adapter, i);
> +			synchronize_irq(adapter->queue_irq[i]);
> +		}
> +
> +		for (i = new_count; i < old_count; i++)
> +			napi_disable(&adapter->napi[i]);
> +
> +		for (i = new_count; i < old_count; i++) {
> +			if (!adapter->queue_irq[i])
> +				continue;
> +			ibmveth_disable_irq(adapter, i);
> +			synchronize_irq(adapter->queue_irq[i]);
> +		}

[Severity: Medium]

Does this second mask pass really close the re-arm window, or can the
poll tail unmask PHYP after it?

napi_disable_locked() in net/core/dev.c returns once SCHED/NPSVC are
observed clear and clears the DISABLE bit before returning:

	clear_bit(NAPI_STATE_DISABLE, &n->state);

Meanwhile ibmveth_poll() is still running its tail:

	if (!napi_complete_done(napi, frames_processed))
		goto out;
	...
	if (ibmveth_poll_stopping(netdev, napi))
		goto out;

	rc = ibmveth_enable_irq(adapter, queue_index);

napi_complete_done() is what let napi_disable() finish, and
ibmveth_poll_stopping() is !netif_running(netdev) ||
napi_disable_pending(napi). netif_running() is still true during a live
ethtool resize and DISABLE has already been cleared, so the guard passes
and the enable hcall can land after the mask + synchronize_irq() above.
synchronize_irq() waits for hard-IRQ handlers, not for a softirq poll
function.

The retiring queue would then stay unmasked at PHYP for the whole drain,
stat-harvest, publish, synchronize_net() and
netif_set_real_num_rx_queues() window: before the publish
ibmveth_schedule_rx_queue() fails napi_schedule_prep() and deliberately
does not mask, and after the publish ibmveth_interrupt() returns
IRQ_NONE, so the interrupts are swallowed without remasking.

The same idiom is repeated in cleanup_new_queues for queues whose NAPI
may already have been scheduled. Would a per-queue flag checked by the
poll tail, or masking after the poll can no longer run, be needed here?

The commit message says:

    "Scale-down masks PHYP on the retiring queues first, then disables
    NAPI, then masks and synchronises again, because an in-flight poll
    can re-arm PHYP while napi_disable() is waiting."

Is that accurate if the re-arm can also happen after napi_disable() has
returned?

> +
> +		for (i = new_count; i < old_count; i++)
> +			ibmveth_drain_rx_queue(adapter, i);

[ ... ]

> +		ibmveth_publish_num_rx_queues(adapter, new_count);
> +		synchronize_net();
> +
> +		rc = netif_set_real_num_rx_queues(netdev, new_count);

[ ... ]

> +		for (i = new_count; i < old_count; i++)
> +			ibmveth_destroy_subordinate_rx_queue(adapter, i);
> +	}

[ ... ]

> @@ -2550,6 +3167,68 @@ static void ibmveth_get_channels(struct net_device *netdev,
>  	channels->rx_count = ibmveth_get_num_rx_queues(adapter);
>  }
>  
> +/**
> + * ibmveth_resize_rx_channels - Validate and apply a new RX queue count
> + * @adapter: ibmveth adapter
> + * @goal_rx: requested RX queue count
> + *
> + * Rejects rx > 1 without MQ firmware (-EOPNOTSUPP) and rx outside
> + * 1..IBMVETH_MAX_RX_QUEUES (-EINVAL). When RX resources are not live
> + * (!opened), only validate; do not allocate. When up, apply via
> + * ibmveth_resize_rx_queues_incremental().
> + *
> + * Return: 0 or negative errno
> + */
> +static int ibmveth_resize_rx_channels(struct ibmveth_adapter *adapter,
> +				      unsigned int goal_rx)
> +{

[ ... ]

> +	/*
> +	 * Refuse any rx > 1, not just growth: once mq_fallback is set the
> +	 * next open comes up single-queue, so an intermediate count could
> +	 * not be honoured either, and accepting it would only repeat the
> +	 * silent clamp at open. max_rx stays at the live count so that
> +	 * read-modify-write TX-only requests still clear the core.
> +	 */
> +	if (goal_rx > 1 && (!adapter->multi_queue || adapter->mq_fallback)) {
> +		netdev_err(netdev,
> +			   "Cannot resize to %u RX queues: multi-queue mode not supported by firmware\n",
> +			   goal_rx);
> +		return -EOPNOTSUPP;
> +	}

[Severity: Low]

Does the advertised channel range still match what this gate accepts once
adapter->mq_fallback is latched?

ibmveth_replenish_task() can set the flag at runtime while several RX
queues are still live and published:

	adapter->mq_fallback = true;
	schedule_work(&adapter->work);

and ibmveth_apply_mq_fallback() only consumes it at the start of the next
open. In that window ibmveth_get_channels() reports max_rx as
IBMVETH_MAX_RX_QUEUES (the live count with the follow-up patch, still
greater than 1), so the core forwards rx_count = 2 on a four-queue
device, and this gate refuses it with -EOPNOTSUPP.

A pure shrink needs no firmware MQ capability, it only deregisters
subordinate queues with H_FREE_LOGICAL_LAN_QUEUE, which the scale-down
path does unconditionally. Should shrinks be allowed here, or should
max_rx be reported as 1 while mq_fallback is set so the range and the set
path agree? The errno message "multi-queue mode not supported by
firmware" is also somewhat misleading for a request to reduce queues.

> +
> +	/* Down / failed-open: do not allocate. */
> +	if (!adapter->opened)
> +		return 0;

[Severity: Low]

Should a changed RX count be silently discarded here while success is
reported?

With the interface down, "ethtool -L ethX rx N" validates and returns 0,
nothing is published, and ibmveth_get_channels() keeps reporting the old
live count:

	channels->rx_count = ibmveth_get_num_rx_queues(adapter);

so userspace has no way to see that the request was ignored.

This is closed later in the same series: the patch "ibmveth: Complete
set_channels down-path and mq_fallback max_rx cap" adds a full down path
in ibmveth_set_channels() which calls
ibmveth_publish_num_rx_queues(adapter, goal_rx),
netif_set_real_num_rx_queues() and refreshes CMO. Would it read better to
fold that handling in here rather than leaving an intermediate commit
that accepts and drops the request?

> +
> +	rxq_entries = adapter->rx_queue[0].num_slots;
> +	rc = ibmveth_resize_rx_queues_incremental(adapter, goal_rx,
> +						  rxq_entries);
> +	if (rc)
> +		netdev_err(netdev, "Failed to resize RX queues: %d\n", rc);
> +	return rc;
> +}
> +
>  static int ibmveth_set_channels(struct net_device *netdev,
>  				struct ethtool_channels *channels)
>  {
> @@ -2558,10 +3237,15 @@ static int ibmveth_set_channels(struct net_device *netdev,
>  		     goal = channels->tx_count;
>  	int rc, i;
>  
> -	/* If ndo_open has not been called yet then don't allocate, just set
> -	 * desired netdev_queue's and return
> +	/* Validate RX (and resize when opened) before the down-path
> +	 * early return so MQ/range errors are reported here. Publishing
> +	 * the desired RX count and CMO while down is the next patch.
>  	 */
> -	if (!(netdev->flags & IFF_UP))
> +	rc = ibmveth_resize_rx_channels(adapter, channels->rx_count);
> +	if (rc)
> +		return rc;

[Severity: Low]

Can a failed ethtool -L leave the RX side already resized?

The RX resize is committed here, ahead of the TX LTB allocation and
netif_set_real_num_tx_queues() error paths further down, and this patch
has no RX rollback, so those paths return an error with the RX queue
count already changed.

The rollback arrives later in the series, in "ibmveth: Complete
set_channels down-path and mq_fallback max_rx cap", which records
want_tx/rx_changed and at the tail does:

	if (netdev->real_num_tx_queues != want_tx) {
		if (rx_changed) {
			...
			rb = ibmveth_resize_rx_queues_incremental(adapter,
								  old_rx,
								  rxq_entries);
			...
		}
		return rc ? rc : -ENOMEM;
	}

Would it be cleaner to introduce the RX resize and its rollback in one
patch, so no commit in the series leaves a half-applied ethtool -L?

> +
> +	if (!adapter->opened)
>  		return netif_set_real_num_tx_queues(netdev, goal);
>  
>  	/* We have IBMVETH_MAX_QUEUES netdev_queue's allocated

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com

Re: [net-next,v6,15/15] ibmveth: Complete set_channels down-path and mq_fallback max_rx cap

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 4 potential
issue(s) to consider:

Critical: 0 · High: 0 · Medium: 1 · Low: 3

- [Medium] Advertised RX channel range does not match the enforced range
  once `mq_fallback` latches.
- [Low] In `ibmveth_set_channels()` the comment above the RX rollback
  describes code that no longer exists: it claims…
- [Low] The new up-path TX-failure rollback in `ibmveth_set_channels()`
  calls `ibmveth_resize_rx_queues_incremental(adapter, old_rx,…
- [Low] `adapter->mq_fallback` is written with a plain store from
  softirq/NAPI context — `ibmveth_replenish_task()` does…

--

ibmveth: Complete set_channels down-path and mq_fallback max_rx cap

This completes the ethtool -L path in ibmveth: the down-path publishes the
desired RX count without allocating, the up-path resizes RX then TX and
rolls RX back if TX cannot reach the requested count, and get_channels()
now caps max_rx at the live rx_count once mq_fallback is set.
ibmveth_poll_controller() also returns early when the adapter is closed.

> diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
> index 5aef8a1f2c23d..4cd00ff3d43e4 100644
> --- a/drivers/net/ethernet/ibm/ibmveth.c
> +++ b/drivers/net/ethernet/ibm/ibmveth.c
> @@ -3156,15 +3156,24 @@ static void ibmveth_get_channels(struct net_device *netdev,
>  				 struct ethtool_channels *channels)
>  {
>  	struct ibmveth_adapter *adapter = netdev_priv(netdev);
> +	unsigned int rx_count = ibmveth_get_num_rx_queues(adapter);
>  
>  	channels->max_tx = ibmveth_real_max_tx_queues();
>  	channels->tx_count = netdev->real_num_tx_queues;
>  
> -	if (adapter->multi_queue)
> +	/*
> +	 * Always report the live RX count. ethtool -L is read-modify-
> +	 * write, so a TX-only request echoes rx_count back at us; an
> +	 * understated value would be applied as a silent RX shrink.
> +	 * mq_fallback instead caps max_rx at the live count, which
> +	 * blocks growth in the core without misreporting what is
> +	 * currently configured.
> +	 */
> +	channels->rx_count = rx_count;
> +	if (adapter->multi_queue && !adapter->mq_fallback)
>  		channels->max_rx = IBMVETH_MAX_RX_QUEUES;
>  	else
> -		channels->max_rx = 1;
> -	channels->rx_count = ibmveth_get_num_rx_queues(adapter);
> +		channels->max_rx = rx_count;
>  }

[Severity: Medium]
Once mq_fallback is set, does the advertised max_rx still match what the
driver will accept?

Here max_rx becomes the live rx_count, which can be 8 or 16.  The core only
bounds the request by that value, in ethtool_set_channels():

    if (channels.rx_count > curr.max_rx ||
        channels.tx_count > curr.max_tx ||

So every value in 1..rx_count reaches the driver.  But
ibmveth_resize_rx_channels() refuses any changed count above 1 in that
state, after the no-op shortcut:

    if (goal_rx == old_rx)
            return 0;
    ...
    if (goal_rx > 1 && (!adapter->multi_queue || adapter->mq_fallback)) {
            netdev_err(netdev,
                       "Cannot resize to %u RX queues: multi-queue mode not supported by firmware\n",
                       goal_rx);
            return -EOPNOTSUPP;
    }

With mq_fallback set and a live count of 8, ethtool -l reports max_rx 8 while
ethtool -L ethX rx 2 through rx 7 all fail with -EOPNOTSUPP; only 1 and the
current count are accepted.

This state looks persistent rather than transient, since
ibmveth_register_rx_queues() latches the flag and then fails the open:

    if (rc) {
            /* Firmware MQ gone: fall back to SQ on next open. */
            if (rc == -EOPNOTSUPP)
                    adapter->mq_fallback = true;
            goto err_unregister;
    }

adapter->opened stays false and num_rx_queues keeps its old value above 1
until the next successful open.

Would it be clearer to advertise max_rx = 1 in the fallback state and keep
the no-op shortcut for the read-modify-write case?  The kernel-doc for
ibmveth_resize_rx_channels() ("Rejects rx > 1 without MQ firmware
(-EOPNOTSUPP)") also does not mention that the no-op shortcut precedes the
gate.

[Severity: Low]
Is the read of adapter->mq_fallback here synchronized against its writer?

The flag is stored from softirq/NAPI context in ibmveth_replenish_task(),
after the replenish_lock has already been dropped:

    spin_unlock_irqrestore(&rxq->replenish_lock, flags);
    ...
            adapter->mq_fallback = true;
            schedule_work(&adapter->work);

The new reader added here, and the capability gate in
ibmveth_resize_rx_channels():

    if (goal_rx > 1 && (!adapter->multi_queue || adapter->mq_fallback)) {

run under RTNL / the netdev ops lock, which does not exclude the softirq
writer.  There is no lock, no READ_ONCE()/WRITE_ONCE() and no acquire/release
pairing on this field, while the sibling field num_rx_queues in the same
struct is deliberately published with:

    smp_store_release(&adapter->num_rx_queues, num);

A stale false read here would advertise max_rx = IBMVETH_MAX_RX_QUEUES right
after firmware refused MQ buffer adds, and a stale read in the gate would let
the driver allocate and replenish RX queues on that firmware, re-latching
mq_fallback and queueing another reset.  Would WRITE_ONCE()/READ_ONCE() on
mq_fallback, or folding it into the existing publish/consume helpers, be
appropriate?

> @@ -3233,28 +3242,83 @@ static int ibmveth_set_channels(struct net_device *netdev,
>  				struct ethtool_channels *channels)
>  {
>  	struct ibmveth_adapter *adapter = netdev_priv(netdev);
> -	unsigned int old = netdev->real_num_tx_queues,
> -		     goal = channels->tx_count;
> +	unsigned int old_rx = ibmveth_get_num_rx_queues(adapter);
> +	unsigned int goal_rx = channels->rx_count;
> +	unsigned int old_tx = netdev->real_num_tx_queues;
> +	unsigned int goal_tx = channels->tx_count;
> +	unsigned int want_tx = goal_tx;
> +	bool rx_changed = false;
>  	int rc, i;
>  
> -	/* Validate RX (and resize when opened) before the down-path
> -	 * early return so MQ/range errors are reported here. Publishing
> -	 * the desired RX count and CMO while down is the next patch.
> -	 */
> -	rc = ibmveth_resize_rx_channels(adapter, channels->rx_count);
> +	if (goal_tx < 1 || goal_tx > ibmveth_real_max_tx_queues()) {
> +		netdev_err(netdev,
> +			   "Invalid TX queue count %u (must be 1-%u)\n",
> +			   goal_tx, ibmveth_real_max_tx_queues());
> +		return -EINVAL;
> +	}
> +
> +	/* RX range / MQ checks live in ibmveth_resize_rx_channels(). */
> +	rc = ibmveth_resize_rx_channels(adapter, goal_rx);
>  	if (rc)
>  		return rc;

[ ... down-path publish/rollback block snipped ... ]

> -	return rc;
> +	if (netdev->real_num_tx_queues != want_tx) {
> +		if (rx_changed) {
> +			/*
> +			 * Only meaningful once RX is live. num_slots is
> +			 * embedded in the adapter and outlives the DMA ring,
> +			 * so reading it at function entry is safe but can
> +			 * return a stale geometry from before the resize.
> +			 */
> +			int rxq_entries = adapter->rx_queue[0].num_slots;

[Severity: Low]
This isn't a bug, but does this comment still describe the code below it?

It says num_slots is read "at function entry" and "can return a stale
geometry from before the resize", yet the read it annotates happens right
here, inside the if (rx_changed) block, after ibmveth_resize_rx_channels()
has already changed the RX geometry.  Function entry only reads
num_rx_queues, real_num_tx_queues, channels->rx_count and channels->tx_count.

The comment looks like a leftover from the v5 layout mentioned in the
changelog, and now warns about a hazard that would only exist if the read
were moved back to function entry.

> +			int rb;
> +
> +			rb = ibmveth_resize_rx_queues_incremental(adapter,
> +								  old_rx,
> +								  rxq_entries);
> +			if (rb)
> +				netdev_err(netdev,
> +					   "Failed to roll back RX queues to %u after TX failure: %d\n",
> +					   old_rx, rb);
> +		}
> +		return rc ? rc : -ENOMEM;
> +	}

[Severity: Low]
Can this rollback re-attempt a queue geometry that the forward path would
have refused?

The rollback calls ibmveth_resize_rx_queues_incremental() directly, so the
multi_queue / mq_fallback gate in ibmveth_resize_rx_channels() is not
consulted.  That gate only rejects goal_rx > 1:

    if (goal_rx > 1 && (!adapter->multi_queue || adapter->mq_fallback)) {
            ...
            return -EOPNOTSUPP;
    }

Consider adapter->opened true, mq_fallback already latched from softirq in
ibmveth_replenish_task():

    adapter->mq_fallback = true;
    schedule_work(&adapter->work);

and a live RX count of 8.  A request of rx 1 with a larger tx passes the gate
(goal_rx is 1), RX shrinks 8 -> 1 and rx_changed becomes true.  If the TX
step then fails in ibmveth_allocate_tx_ltb() or
netif_set_real_num_tx_queues(), the rollback runs the scale-up path back to
old_rx = 8 on firmware that has already refused MQ buffer adds, so
H_REG_LOGICAL_LAN_QUEUE / replenish hit the same H_FUNCTION, mq_fallback is
re-latched and another schedule_work(&adapter->work) reset is queued from an
ethtool error path.

Would it be better to route the rollback through
ibmveth_resize_rx_channels(), or to skip it when mq_fallback is set and leave
RX at 1?

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/cover.1788102125.git.mmc%40linux.ibm.com