From: Long Li <longli@microsoft.com>
To: Long Li <longli@microsoft.com>,
Konstantin Taranov <kotaranov@microsoft.com>,
Jakub Kicinski <kuba@kernel.org>,
"David S . Miller" <davem@davemloft.net>,
Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@google.com>,
Andrew Lunn <andrew+netdev@lunn.ch>,
Jason Gunthorpe <jgg@ziepe.ca>, Leon Romanovsky <leon@kernel.org>,
Haiyang Zhang <haiyangz@microsoft.com>,
"K . Y . Srinivasan" <kys@microsoft.com>,
Wei Liu <wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>,
shradhagupta@linux.microsoft.com, Simon Horman <horms@kernel.org>,
ernis@linux.microsoft.com, stephen@networkplumber.org
Cc: netdev@vger.kernel.org, linux-rdma@vger.kernel.org,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH net-next v2 10/13] net: mana: release EQs left idle by a channel-count reduction
Date: Wed, 12 Aug 2026 22:04:15 -0700 [thread overview]
Message-ID: <20260813050418.2906468-11-longli@microsoft.com> (raw)
In-Reply-To: <20260813050418.2906468-1-longli@microsoft.com>
The shared EQ pool only grows, so it sits at the high-water mark of every
channel count the port has ever used. After "ethtool -L ens1 combined 32"
then "combined 4" the port keeps 32 EQs and 32 MSI-X vectors while using
four:
# ethtool -L ens1 combined 4
# grep -c mana /proc/interrupts
33
The pre-swap path recreated every EQ per reconfiguration, so this is new.
Release the EQs above the live queue count once a retiring set has been
torn down. That is the only safe point: a CQ holds the gdma_queue pointer
of its parent EQ, so an EQ may only be destroyed once the set referencing
it is gone.
While here, fix mana_create_eq_debugfs(), which stored the new dentry in
a stack copy rather than in apc->eqs[i].
Signed-off-by: Long Li <longli@microsoft.com>
---
.../net/ethernet/microsoft/mana/mana_bpf.c | 4 +-
drivers/net/ethernet/microsoft/mana/mana_en.c | 73 +++++++++++++++----
.../ethernet/microsoft/mana/mana_ethtool.c | 15 ++--
include/net/mana/mana.h | 3 +-
4 files changed, 70 insertions(+), 25 deletions(-)
diff --git a/drivers/net/ethernet/microsoft/mana/mana_bpf.c b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
index 05936453fbbfa59c563fdea50e0b40096c2b46ac..4b29406595b37877e1e5d14cf93d68aa3c4ace02 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_bpf.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_bpf.c
@@ -228,7 +228,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
if (err) {
NL_SET_ERR_MSG_MOD(extack,
"XDP: Re-config failed at publish");
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
/* After the cleanup above: closing destroys the EQ pool
* those queues' CQs were attached to.
*/
@@ -237,7 +237,7 @@ static int mana_xdp_set(struct net_device *ndev, struct bpf_prog *prog,
return err;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
mana_qset_scratch_free(scratch);
} else {
/* No queues to rebuild; mana_open() will size the RX buffers
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index be7f9f6626e42c33fc0e749e6897ecc8117222cf..7c43c2f9043ba591b58e4ee2211cf37da9fead36 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -921,16 +921,13 @@ static int mana_change_mtu(struct net_device *ndev, int new_mtu)
err = mana_publish_qset(mpc, &newq, &oldq);
if (err) {
- mana_free_qset(scratch, &newq);
+ mana_free_qset(mpc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(mpc, scratch, &oldq);
free_scratch:
- /* After the caller-side cleanup above, so the EQ pool outlives the
- * CQs that reference it.
- */
mana_publish_close_if_needed(mpc);
mana_qset_scratch_free(scratch);
return err;
@@ -1817,6 +1814,9 @@ void mana_destroy_eq(struct mana_port_context *apc)
msi = eq->eq.msix_index;
mana_gd_destroy_queue(gc, eq);
mana_gd_put_gic(gc, !gc->msi_sharing, msi);
+ apc->eqs[i].eq = NULL;
+ /* Freed with the parent by debugfs_remove_recursive() above. */
+ apc->eqs[i].mana_eq_debugfs = NULL;
}
kfree(apc->eqs);
@@ -1827,15 +1827,16 @@ EXPORT_SYMBOL_NS(mana_destroy_eq, "NET_MANA");
static void mana_create_eq_debugfs(struct mana_port_context *apc, int i)
{
- struct mana_eq eq = apc->eqs[i];
+ struct mana_eq *eq = &apc->eqs[i];
char eqnum[32];
sprintf(eqnum, "eq%d", i);
- eq.mana_eq_debugfs = debugfs_create_dir(eqnum, apc->mana_eqs_debugfs);
- debugfs_create_u32("head", 0400, eq.mana_eq_debugfs, &eq.eq->head);
- debugfs_create_u32("tail", 0400, eq.mana_eq_debugfs, &eq.eq->tail);
- debugfs_create_u32("irq", 0400, eq.mana_eq_debugfs, &eq.eq->eq.irq);
- debugfs_create_file("eq_dump", 0400, eq.mana_eq_debugfs, eq.eq, &mana_dbg_q_fops);
+ eq->mana_eq_debugfs = debugfs_create_dir(eqnum, apc->mana_eqs_debugfs);
+ debugfs_create_u32("head", 0400, eq->mana_eq_debugfs, &eq->eq->head);
+ debugfs_create_u32("tail", 0400, eq->mana_eq_debugfs, &eq->eq->tail);
+ debugfs_create_u32("irq", 0400, eq->mana_eq_debugfs, &eq->eq->eq.irq);
+ debugfs_create_file("eq_dump", 0400, eq->mana_eq_debugfs, eq->eq,
+ &mana_dbg_q_fops);
}
int mana_create_eq(struct mana_port_context *apc)
@@ -1961,6 +1962,37 @@ static int mana_grow_eqs(struct mana_port_context *apc, unsigned int need)
return err;
}
+/* Release EQs above @keep, returning the MSI-X vectors freed. Only safe once
+ * no set references them, i.e. after mana_free_qset(), or a live CQ would
+ * point at a destroyed EQ.
+ */
+static void mana_shrink_eqs(struct mana_port_context *apc, unsigned int keep)
+{
+ struct gdma_context *gc = apc->ac->gdma_dev->gdma_context;
+ struct gdma_queue *eq;
+ unsigned int msi;
+ unsigned int i;
+
+ if (!apc->eqs || keep >= apc->num_eqs)
+ return;
+
+ for (i = keep; i < apc->num_eqs; i++) {
+ eq = apc->eqs[i].eq;
+ if (!eq)
+ continue;
+
+ debugfs_remove_recursive(apc->eqs[i].mana_eq_debugfs);
+ apc->eqs[i].mana_eq_debugfs = NULL;
+
+ msi = eq->eq.msix_index;
+ mana_gd_destroy_queue(gc, eq);
+ mana_gd_put_gic(gc, !gc->msi_sharing, msi);
+ apc->eqs[i].eq = NULL;
+ }
+
+ apc->num_eqs = keep;
+}
+
static int mana_fence_rq(struct mana_port_context *apc, struct mana_rxq *rxq)
{
struct mana_fence_rq_resp resp = {};
@@ -4179,6 +4211,13 @@ int mana_alloc_qset(struct mana_port_context *apc,
kfree(scratch->rxqs);
scratch->rxqs = NULL;
out_err:
+ /* Give back any EQ this attempt added to the shared pool rather than
+ * holding its MSI-X vectors until some later teardown: the live set
+ * still needs only apc->num_queues of them. Safe here because this
+ * set's CQs have already been destroyed above.
+ */
+ mana_shrink_eqs(apc, apc->num_queues);
+
netdev_err(ndev, "mana_alloc_qset(num_queues=%u) failed: %d\n",
num_queues, err);
return err;
@@ -4456,7 +4495,8 @@ static void mana_qset_debugfs_publish(struct mana_port_context *apc)
/* Tear down @qset, no longer installed on @apc, against @scratch so the live
* context never points at queues being freed.
*/
-void mana_free_qset(struct mana_port_context *scratch, struct mana_qset *qset)
+void mana_free_qset(struct mana_port_context *apc,
+ struct mana_port_context *scratch, struct mana_qset *qset)
{
struct bpf_prog *retiring_prog;
unsigned int retiring_queues;
@@ -4551,12 +4591,19 @@ void mana_free_qset(struct mana_port_context *scratch, struct mana_qset *qset)
memset(qset, 0, sizeof(*qset));
+ /* This set is gone, so any EQ above the live queue count is now
+ * unreferenced. Release those vectors instead of holding them at the
+ * high-water mark. Safe here and only here: the retiring set's CQs
+ * have just been destroyed.
+ */
+ mana_shrink_eqs(apc, apc->num_queues);
+
/* Queues built through a scratch context carry no debugfs nodes,
* because both sets are alive during the swap and would collide on
* the same names. The retiring set's nodes are gone now, so the
* published queues can finally take those names.
*/
- mana_qset_debugfs_publish(netdev_priv(scratch->ndev));
+ mana_qset_debugfs_publish(apc);
}
/* --- end of pre-allocate + swap reconfiguration path ---------------------- */
diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
index 08e6fb7785cd3be72f8737083e6424783c5e0d21..024119dd4e353e33d11ccc883b4fe09a99434a26 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c
@@ -730,16 +730,13 @@ static int mana_set_channels(struct net_device *ndev,
err = mana_publish_qset(apc, &newq, &oldq);
if (err) {
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
free_scratch:
- /* After the caller-side cleanup above, so the EQ pool outlives the
- * CQs that reference it.
- */
mana_publish_close_if_needed(apc);
mana_qset_scratch_free(scratch);
clear_flag:
@@ -831,11 +828,11 @@ static int mana_set_ringparam(struct net_device *ndev,
if (err) {
NL_SET_ERR_MSG_FMT(extack, "failed to change ring params: %d",
err);
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
free_scratch:
/* After the caller-side cleanup above, so the EQ pool outlives the
@@ -925,11 +922,11 @@ static int mana_set_priv_flags(struct net_device *ndev, u32 priv_flags)
err = mana_publish_qset(apc, &newq, &oldq);
if (err) {
- mana_free_qset(scratch, &newq);
+ mana_free_qset(apc, scratch, &newq);
goto free_scratch;
}
- mana_free_qset(scratch, &oldq);
+ mana_free_qset(apc, scratch, &oldq);
free_scratch:
mana_publish_close_if_needed(apc);
diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
index dfb6ba0012fda629192e4fe9cf8aba57fd5bb451..619c66f3c6192dd2988a1bba73e991df2a773923 100644
--- a/include/net/mana/mana.h
+++ b/include/net/mana/mana.h
@@ -756,7 +756,8 @@ int mana_alloc_qset(struct mana_port_context *apc,
int mana_publish_qset(struct mana_port_context *apc, struct mana_qset *newq,
struct mana_qset *out_old);
void mana_publish_close_if_needed(struct mana_port_context *apc);
-void mana_free_qset(struct mana_port_context *scratch, struct mana_qset *qset);
+void mana_free_qset(struct mana_port_context *apc,
+ struct mana_port_context *scratch, struct mana_qset *qset);
void mana_dim_change(struct mana_cq *cq, bool enable);
--
2.43.0
next prev parent reply other threads:[~2026-08-13 5:05 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-13 5:04 [PATCH net-next v2 00/13] net: mana: reconfigure by replacing the queue set Long Li
2026-08-13 5:04 ` [PATCH net-next v2 01/13] net: mana: add queue-set allocation and teardown helpers Long Li
2026-08-13 5:04 ` [PATCH net-next v2 02/13] net: mana: swap queue sets in mana_set_channels Long Li
2026-08-13 5:04 ` [PATCH net-next v2 03/13] net: mana: swap queue sets in mana_set_ringparam Long Li
2026-08-13 5:04 ` [PATCH net-next v2 04/13] net: mana: swap queue sets in mana_set_priv_flags Long Li
2026-08-13 5:04 ` [PATCH net-next v2 05/13] net: mana: swap queue sets in mana_change_mtu Long Li
2026-08-13 5:04 ` [PATCH net-next v2 06/13] net: mana: swap queue sets in mana_xdp_set Long Li
2026-08-13 5:04 ` [PATCH net-next v2 07/13] net: mana: do not bail out of mana_detach on dealloc failure Long Li
2026-08-13 5:04 ` [PATCH net-next v2 08/13] net: mana: keep per-queue statistics in the port context Long Li
2026-08-13 5:04 ` [PATCH net-next v2 09/13] net: mana: share the EQ pool across a queue-set swap Long Li
2026-08-13 5:04 ` Long Li [this message]
2026-08-13 5:04 ` [PATCH net-next v2 11/13] net: mana: keep a user-configured RSS table across a queue rebuild Long Li
2026-08-13 5:04 ` [PATCH net-next v2 12/13] net: mana: keep the surviving queues when the channel count is reduced Long Li
2026-08-13 5:04 ` [PATCH net-next v2 13/13] net: mana: keep the existing queues when the channel count is raised Long Li
-- strict thread matches above, loose matches on Subject: below --
2026-08-11 6:35 [PATCH net-next v2 01/13] net: mana: add queue-set allocation and teardown helpers Long Li
2026-08-11 6:35 ` [PATCH net-next v2 10/13] net: mana: release EQs left idle by a channel-count reduction Long Li
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260813050418.2906468-11-longli@microsoft.com \
--to=longli@microsoft.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=decui@microsoft.com \
--cc=edumazet@google.com \
--cc=ernis@linux.microsoft.com \
--cc=haiyangz@microsoft.com \
--cc=horms@kernel.org \
--cc=jgg@ziepe.ca \
--cc=kotaranov@microsoft.com \
--cc=kuba@kernel.org \
--cc=kys@microsoft.com \
--cc=leon@kernel.org \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=shradhagupta@linux.microsoft.com \
--cc=stephen@networkplumber.org \
--cc=wei.liu@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox