From: Long Li <longli@microsoft.com>
To: Long Li <longli@microsoft.com>,
Konstantin Taranov <kotaranov@microsoft.com>,
Jakub Kicinski <kuba@kernel.org>,
"David S . Miller" <davem@davemloft.net>,
Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@google.com>,
Andrew Lunn <andrew+netdev@lunn.ch>,
Jason Gunthorpe <jgg@ziepe.ca>, Leon Romanovsky <leon@kernel.org>,
Haiyang Zhang <haiyangz@microsoft.com>,
"K . Y . Srinivasan" <kys@microsoft.com>,
Wei Liu <wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>,
shradhagupta@linux.microsoft.com, Simon Horman <horms@kernel.org>,
ernis@linux.microsoft.com, stephen@networkplumber.org
Cc: netdev@vger.kernel.org, linux-rdma@vger.kernel.org,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH net-next v2 10/13] net: mana: release EQs left idle by a channel-count reduction
Date: Wed, 12 Aug 2026 22:04:15 -0700 [thread overview]
Message-ID: <20260813050418.2906468-11-longli@microsoft.com> (raw)
In-Reply-To: <20260813050418.2906468-1-longli@microsoft.com>
The shared EQ pool only grows, so it sits at the high-water mark of every
channel count the port has ever used. After "ethtool -L ens1 combined 32"
then "combined 4" the port keeps 32 EQs and 32 MSI-X vectors while using
four:
# ethtool -L ens1 combined 4
# grep -c mana /proc/interrupts
33
The pre-swap path recreated every EQ per reconfiguration, so this is new.
Release the EQs above the live queue count once a retiring set has been
torn down. That is the only safe point: a CQ holds the gdma_queue pointer
of its parent EQ, so an EQ may only be destroyed once the set referencing
it is gone.
While here, fix mana_create_eq_debugfs(), which stored the new dentry in
a stack copy rather than in apc->eqs[i].
Signed-off-by: Long Li <longli@microsoft.com>
---
.../net/ethernet/microsoft/mana/mana_bpf.c | 4 +-
drivers/net/ethernet/microsoft/mana/mana_en.c | 73 +++++++++++++++----
.../ethernet/microsoft/mana/mana_ethtool.c | 15 ++--
include/net/mana/mana.h | 3 +-
4 files changed, 70 insertions(+), 25 deletions(-)
diff --git a/drivers/net/ethernet/microsoft/mana/mana_bpf.c b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
index 05936453fbbfa59c563fdea50e0b40096c2b46ac..4b29406595b37877e1e5d14cf93d68aa3c4ace02 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_bpf.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
@@ -228,7 +228,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
if (err) {
NL_SET_ERR_MSG_MOD(extack,
"XDP: Re-config failed at publish");
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
/* After the cleanup above: closing destroys the EQ pool
* those queues' CQs were attached to.
*/
@@ -237,7 +237,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
return err;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
mana_qset_scratch_free(scratch);
} else {
/* No queues to rebuild; mana_open() will size the RX buffers
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index be7f9f6626e42c33fc0e749e6897ecc8117222cf..7c43c2f9043ba591b58e4ee2211cf37da9fead36 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -921,16 +921,13 @@ static int mana_change_mtu(struct net_device *ndev, int new_mtu)
err = mana_publish_qset(mpc, &newq, &oldq);
if (err) {
- mana_free_qset(scratch, &newq);
+ mana_free_qset(mpc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(mpc, scratch, &oldq);
free_scratch:
- /* After the caller-side cleanup above, so the EQ pool outlives the
- * CQs that reference it.
- */
mana_publish_close_if_needed(mpc);
mana_qset_scratch_free(scratch);
return err;
@@ -1817,6 +1814,9 @@ void mana_destroy_eq(struct mana_port_context *apc)
msi = eq->eq.msix_index;
mana_gd_destroy_queue(gc, eq);
mana_gd_put_gic(gc, !gc->msi_sharing, msi);
+ apc->eqs[i].eq = NULL;
+ /* Freed with the parent by debugfs_remove_recursive() above. */
+ apc->eqs[i].mana_eq_debugfs = NULL;
}
kfree(apc->eqs);
@@ -1827,15 +1827,16 @@ EXPORT_SYMBOL_NS(mana_destroy_eq, "NET_MANA");
static void mana_create_eq_debugfs(struct mana_port_context *apc, int i)
{
- struct mana_eq eq = apc->eqs[i];
+ struct mana_eq *eq = &apc->eqs[i];
char eqnum[32];
sprintf(eqnum, "eq%d", i);
- eq.mana_eq_debugfs = debugfs_create_dir(eqnum, apc->mana_eqs_debugfs);
- debugfs_create_u32("head", 0400, eq.mana_eq_debugfs, &eq.eq->head);
- debugfs_create_u32("tail", 0400, eq.mana_eq_debugfs, &eq.eq->tail);
- debugfs_create_u32("irq", 0400, eq.mana_eq_debugfs, &eq.eq->eq.irq);
- debugfs_create_file("eq_dump", 0400, eq.mana_eq_debugfs, eq.eq, &mana_dbg_q_fops);
+ eq->mana_eq_debugfs = debugfs_create_dir(eqnum, apc->mana_eqs_debugfs);
+ debugfs_create_u32("head", 0400, eq->mana_eq_debugfs, &eq->eq->head);
+ debugfs_create_u32("tail", 0400, eq->mana_eq_debugfs, &eq->eq->tail);
+ debugfs_create_u32("irq", 0400, eq->mana_eq_debugfs, &eq->eq->eq.irq);
+ debugfs_create_file("eq_dump", 0400, eq->mana_eq_debugfs, eq->eq,
+ &mana_dbg_q_fops);
}
int mana_create_eq(struct mana_port_context *apc)
@@ -1961,6 +1962,37 @@ static int mana_grow_eqs(struct mana_port_context *apc, unsigned int need)
return err;
}
+/* Release EQs above @keep, returning the MSI-X vectors freed. Only safe once
+ * no set references them, i.e. after mana_free_qset(), or a live CQ would
+ * point at a destroyed EQ.
+ */
+static void mana_shrink_eqs(struct mana_port_context *apc, unsigned int keep)
+{
+ struct gdma_context *gc = apc->ac->gdma_dev->gdma_context;
+ struct gdma_queue *eq;
+ unsigned int msi;
+ unsigned int i;
+
+ if (!apc->eqs || keep >= apc->num_eqs)
+ return;
+
+ for (i = keep; i < apc->num_eqs; i++) {
+ eq = apc->eqs[i].eq;
+ if (!eq)
+ continue;
+
+ debugfs_remove_recursive(apc->eqs[i].mana_eq_debugfs);
+ apc->eqs[i].mana_eq_debugfs = NULL;
+
+ msi = eq->eq.msix_index;
+ mana_gd_destroy_queue(gc, eq);
+ mana_gd_put_gic(gc, !gc->msi_sharing, msi);
+ apc->eqs[i].eq = NULL;
+ }
+
+ apc->num_eqs = keep;
+}
+
static int mana_fence_rq(struct mana_port_context *apc, struct mana_rxq *rxq)
{
struct mana_fence_rq_resp resp = {};
@@ -4179,6 +4211,13 @@ int mana_alloc_qset(struct mana_port_context *apc,
kfree(scratch->rxqs);
scratch->rxqs = NULL;
out_err:
+ /* Give back any EQ this attempt added to the shared pool rather than
+ * holding its MSI-X vectors until some later teardown: the live set
+ * still needs only apc->num_queues of them. Safe here because this
+ * set's CQs have already been destroyed above.
+ */
+ mana_shrink_eqs(apc, apc->num_queues);
+
netdev_err(ndev, "mana_alloc_qset(num_queues=%u) failed: %d\n",
num_queues, err);
return err;
@@ -4456,7 +4495,8 @@ static void mana_qset_debugfs_publish(struct mana_port_context *apc)
/* Tear down @qset, no longer installed on @apc, against @scratch so the live
* context never points at queues being freed.
*/
-void mana_free_qset(struct mana_port_context *scratch, struct mana_qset *qset)
+void mana_free_qset(struct mana_port_context *apc,
+ struct mana_port_context *scratch, struct mana_qset *qset)
{
struct bpf_prog *retiring_prog;
unsigned int retiring_queues;
@@ -4551,12 +4591,19 @@ void mana_free_qset(struct mana_port_context *scratch, struct mana_qset *qset)
memset(qset, 0, sizeof(*qset));
+ /* This set is gone, so any EQ above the live queue count is now
+ * unreferenced. Release those vectors instead of holding them at the
+ * high-water mark. Safe here and only here: the retiring set's CQs
+ * have just been destroyed.
+ */
+ mana_shrink_eqs(apc, apc->num_queues);
+
/* Queues built through a scratch context carry no debugfs nodes,
* because both sets are alive during the swap and would collide on
* the same names. The retiring set's nodes are gone now, so the
* published queues can finally take those names.
*/
- mana_qset_debugfs_publish(netdev_priv(scratch->ndev));
+ mana_qset_debugfs_publish(apc);
}
/* --- end of pre-allocate + swap reconfiguration path ---------------------- */
diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
index 08e6fb7785cd3be72f8737083e6424783c5e0d21..024119dd4e353e33d11ccc883b4fe09a99434a26 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
@@ -730,16 +730,13 @@ static int mana_set_channels(struct net_device *ndev,
err = mana_publish_qset(apc, &newq, &oldq);
if (err) {
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
free_scratch:
- /* After the caller-side cleanup above, so the EQ pool outlives the
- * CQs that reference it.
- */
mana_publish_close_if_needed(apc);
mana_qset_scratch_free(scratch);
clear_flag:
@@ -831,11 +828,11 @@ static int mana_set_ringparam(struct net_device *ndev,
if (err) {
NL_SET_ERR_MSG_FMT(extack, "failed to change ring params: %d",
err);
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
free_scratch:
/* After the caller-side cleanup above, so the EQ pool outlives the
@@ -925,11 +922,11 @@ static int mana_set_priv_flags(struct net_device *ndev, u32 priv_flags)
err = mana_publish_qset(apc, &newq, &oldq);
if (err) {
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
free_scratch:
mana_publish_close_if_needed(apc);
diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
index dfb6ba0012fda629192e4fe9cf8aba57fd5bb451..619c66f3c6192dd2988a1bba73e991df2a773923 100644
--- a/include/net/mana/mana.h
+++ b/include/net/mana/mana.h
@@ -756,7 +756,8 @@ int mana_alloc_qset(struct mana_port_context *apc,
int mana_publish_qset(struct mana_port_context *apc, struct mana_qset *newq,
struct mana_qset *out_old);
void mana_publish_close_if_needed(struct mana_port_context *apc);
-void mana_free_qset(struct mana_port_context *scratch, struct mana_qset *qset);
+void mana_free_qset(struct mana_port_context *apc,
+ struct mana_port_context *scratch, struct mana_qset *qset);
void mana_dim_change(struct mana_cq *cq, bool enable);
--
2.43.0
next prev parent reply other threads:[~2026-08-13 5:05 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-13 5:04 [PATCH net-next v2 00/13] net: mana: reconfigure by replacing the queue set Long Li
2026-08-13 5:04 ` [PATCH net-next v2 01/13] net: mana: add queue-set allocation and teardown helpers Long Li
2026-08-13 5:04 ` [PATCH net-next v2 02/13] net: mana: swap queue sets in mana_set_channels Long Li
2026-08-13 5:04 ` [PATCH net-next v2 03/13] net: mana: swap queue sets in mana_set_ringparam Long Li
2026-08-13 5:04 ` [PATCH net-next v2 04/13] net: mana: swap queue sets in mana_set_priv_flags Long Li
2026-08-13 5:04 ` [PATCH net-next v2 05/13] net: mana: swap queue sets in mana_change_mtu Long Li
2026-08-13 5:04 ` [PATCH net-next v2 06/13] net: mana: swap queue sets in mana_xdp_set Long Li
2026-08-13 5:04 ` [PATCH net-next v2 07/13] net: mana: do not bail out of mana_detach on dealloc failure Long Li
2026-08-13 5:04 ` [PATCH net-next v2 08/13] net: mana: keep per-queue statistics in the port context Long Li
2026-08-13 5:04 ` [PATCH net-next v2 09/13] net: mana: share the EQ pool across a queue-set swap Long Li
2026-08-13 5:04 ` Long Li [this message]
2026-08-13 5:04 ` [PATCH net-next v2 11/13] net: mana: keep a user-configured RSS table across a queue rebuild Long Li
2026-08-13 5:04 ` [PATCH net-next v2 12/13] net: mana: keep the surviving queues when the channel count is reduced Long Li
2026-08-13 5:04 ` [PATCH net-next v2 13/13] net: mana: keep the existing queues when the channel count is raised Long Li
-- strict thread matches above, loose matches on Subject: below --
2026-08-11 6:35 [PATCH net-next v2 01/13] net: mana: add queue-set allocation and teardown helpers Long Li
2026-08-11 6:35 ` [PATCH net-next v2 10/13] net: mana: release EQs left idle by a channel-count reduction Long Li
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260813050418.2906468-11-longli@microsoft.com \
--to=longli@microsoft.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=decui@microsoft.com \
--cc=edumazet@google.com \
--cc=ernis@linux.microsoft.com \
--cc=haiyangz@microsoft.com \
--cc=horms@kernel.org \
--cc=jgg@ziepe.ca \
--cc=kotaranov@microsoft.com \
--cc=kuba@kernel.org \
--cc=kys@microsoft.com \
--cc=leon@kernel.org \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=shradhagupta@linux.microsoft.com \
--cc=stephen@networkplumber.org \
--cc=wei.liu@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.