From: Michael Chan <michael.chan@broadcom.com>
To: davem@davemloft.net
Cc: netdev@vger.kernel.org, edumazet@kernel.org, kuba@kernel.org,
pabeni@redhat.com, andrew+netdev@lunn.ch,
pavan.chebbi@broadcom.com, andrew.gospodarek@broadcom.com
Subject: [PATCH net-next v5 8/9] bnxt_en: Fix ring accounting/validation when rings are constrained by FW
Date: Thu, 19 Nov 2026 17:29:20 -0800 [thread overview]
Message-ID: <20261120012921.2537903-9-michael.chan@broadcom.com> (raw)
In-Reply-To: <20261120012921.2537903-1-michael.chan@broadcom.com>
When __bnxt_reserve_rings() reserves fewer TX rings than requested,
and an XDP program is attached, the driver used to blindly subtract
bp->tx_nr_rings_xdp from bp->tx_nr_rings, potentially causing an integer
underflow.
This patch mainly fixes the existing bnxt_adj_tx_rings(),
bnxt_rings_ok(), some ring helpers, and __bnxt_reserve_rings():
1. bnxt_adj_tx_rings() is now renamed bnxt_adj_rings() to reflect that
all rings (tx, rx, cp) may adjust if needed. It will now correctly
scale down TX rings for XDP and TCs evenly. In combined channel mode,
RX and TX rings are now clamped to match each other - required by
XDP's 1:1 TX/RX mapping and now applied to non-XDP mode for
consistency. CP rings will also be reduced if necessary. Any
leftover rings after integer division are intentionally left unused.
2. bnxt_rings_ok() is now updated to robustly check for the absolute
minimum viable ring configuration to support the TC and XDP settings.
We now call bnxt_rings_ok() first before calling bnxt_adj_rings() to
guarantee no math underflow or ring truncation to 0.
3. bnxt_num_cp_to_tx()/bnxt_num_tx_to_cp() helpers are used while
trimming the rings with possible ring shortages. These helpers now
use the passed in tx_xdp (now used as boolean) and rx parameters to
accurately scale and clamp the XDP rings.
4. __bnxt_reserve_rings() now only calls bnxt_adj_rings() if
bp->bnapi has not been allocated. In this context, we may be expanding
and asking for more rings than FW can grant and we need to call
bnxt_adj_rings() for possible adjustments. We now abort
__bnxt_reserve_rings() for all fatal conditions through the common
reserve_rings_exit label. We now leave the ring states unchanged and
the ULP resources reconciled during abort.
We also need to reset the RSS table of the default context as needed
if the RX rings have shrunk. As noted in the cover letter, we have
similar existing issues regarding RSS tables for non-default RSS
contexts and user defined n-tuple filters when RX rings shrink due
to FW or MSI-X constraints. The fixes for these are deferred to a
separate patchset.
The next patch will fix similar issues when MSI-X vectors are
constrained.
These existing issues were detected by Sashiko when reviewing the
new kTLS patchset (patch #3 of 15):
https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260810051358.1244418-7-michael.chan@broadcom.com
Fixes: 1ee581c24dfd ("bnxt_en: Adjust TX rings if reservation is less than requested")
Fixes: 674f50a5b026 ("bnxt_en: Implement new method to reserve rings.")
Reviewed-by: Andy Gospodarek <andrew.gospodarek@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
---
v5:
Split out the MSI-X changes to the new patch #9.
v4:
Fix bnxt_num_cp_to_tx()/bnxt_num_tx_to_cp() helpers to fix the
ring trimming issue reported by Sashiko.
https://lore.kernel.org/netdev/20261016064427.496944-9-michael.chan@broadcom.com/
v3:
No longer have to deal with dropped NITRO A0
Fix the dynamic IRQ code path in bnxt_reserve_rings().
Use the proper __bnxt_num_tx_to_cp() in bnxt_rings_ok().
https://lore.kernel.org/netdev/20261012220827.340998-8-michael.chan@broadcom.com/
v2:
Reduce RX and CP rings if needed when TX rings are reduced to keep it
consistent.
Handle NITRO_A0 special RX requirements to keep it consistent.
Fix all unwind issues in these code paths.
https://lore.kernel.org/netdev/20260928041712.3467803-7-michael.chan@broadcom.com/
v1:
https://lore.kernel.org/netdev/20260831024342.2161156-2-michael.chan@broadcom.com/
---
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 137 +++++++++++++++++-----
1 file changed, 106 insertions(+), 31 deletions(-)
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index d76227d29282..565099088969 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -8086,10 +8086,32 @@ static void bnxt_copy_reserved_rings(struct bnxt *bp, struct bnxt_hw_rings *hwr)
}
}
+static int bnxt_min_tx_rings(struct bnxt *bp)
+{
+ int min_tx = bp->num_tc ? bp->num_tc : 1;
+
+ return bp->tx_nr_rings_xdp ? min_tx + 1 : min_tx;
+}
+
static bool bnxt_rings_ok(struct bnxt *bp, struct bnxt_hw_rings *hwr)
{
- return hwr->tx && hwr->rx && hwr->cp && hwr->grp && hwr->vnic &&
- hwr->stat && (hwr->cp_p5 || !(bp->flags & BNXT_FLAG_CHIP_P5_PLUS));
+ int min_tx, min_tx_xdp = 0;
+ int min_rx = 1;
+ int min_cp;
+ int tx_cp;
+
+ min_tx = bnxt_min_tx_rings(bp);
+ if (bp->tx_nr_rings_xdp)
+ min_tx_xdp = 1;
+
+ tx_cp = bnxt_num_tx_to_cp(bp, min_tx, min_tx_xdp, min_rx);
+ if (bp->flags & BNXT_FLAG_SHARED_RINGS)
+ min_cp = max(tx_cp, min_rx);
+ else
+ min_cp = tx_cp + min_rx;
+ return hwr->tx >= min_tx && hwr->rx >= min_rx && hwr->cp >= min_cp &&
+ hwr->grp && hwr->vnic && hwr->stat &&
+ (hwr->cp_p5 || !(bp->flags & BNXT_FLAG_CHIP_P5_PLUS));
}
/* Check to see if we need to reset the user configured RSS table
@@ -8107,10 +8129,13 @@ static void bnxt_check_rss_tbl_lost(struct bnxt *bp, int old_rx, int new_rx)
}
static int bnxt_get_avail_msix(struct bnxt *bp, int num);
+static void bnxt_adj_rings(struct bnxt *bp);
static int __bnxt_reserve_rings(struct bnxt *bp)
{
struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+ int req_rx_rings = bp->rx_nr_rings;
+ int req_tx_rings = bp->tx_nr_rings;
struct bnxt_hw_rings hwr = {0};
int rx_rings, old_rx_rings, rc;
int cp = bp->cp_nr_rings;
@@ -8153,7 +8178,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
rc = bnxt_hwrm_reserve_rings(bp, &hwr);
if (rc)
- return rc;
+ goto reserve_rings_exit;
bnxt_copy_reserved_rings(bp, &hwr);
@@ -8162,8 +8187,10 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
if (hwr.rx >= 2) {
rx_rings = hwr.rx >> 1;
} else {
- if (netif_running(bp->dev))
- return -ENOMEM;
+ if (netif_running(bp->dev)) {
+ rc = -ENOMEM;
+ goto reserve_rings_exit;
+ }
bp->flags &= ~BNXT_FLAG_AGG_RINGS;
bp->flags |= BNXT_FLAG_NO_AGG_RINGS;
@@ -8178,49 +8205,67 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
hwr.stat -= bnxt_get_ulp_stat_ctxs(bp);
hwr.cp = min_t(int, hwr.cp, hwr.stat);
rc = bnxt_trim_rings(bp, &rx_rings, &hwr.tx, hwr.cp, sh);
+ if (rc)
+ goto reserve_rings_exit;
+
+ if (bp->bnapi && (rx_rings < req_rx_rings ||
+ hwr.tx < req_tx_rings)) {
+ netdev_err(bp->dev, "Unexpected ring shrinkage %d/%d RX/TX to %d/%d\n",
+ req_rx_rings, req_tx_rings, rx_rings, hwr.tx);
+ rc = -ENOSPC;
+ goto reserve_rings_exit;
+ }
if (bp->flags & BNXT_FLAG_AGG_RINGS)
hwr.rx = rx_rings << 1;
tx_cp = bnxt_num_tx_to_cp(bp, hwr.tx, bp->tx_nr_rings_xdp, rx_rings);
hwr.cp = sh ? max_t(int, tx_cp, rx_rings) : tx_cp + rx_rings;
- if (hwr.tx != bp->tx_nr_rings) {
- netdev_warn(bp->dev,
- "Able to reserve only %d out of %d requested TX rings\n",
- hwr.tx, bp->tx_nr_rings);
+
+ if (!bnxt_rings_ok(bp, &hwr)) {
+ rc = -ENOMEM;
+ goto reserve_rings_exit;
}
bp->tx_nr_rings = hwr.tx;
+ bp->rx_nr_rings = rx_rings;
+ bp->cp_nr_rings = hwr.cp;
+ if (!bp->bnapi)
+ bnxt_adj_rings(bp);
+
+ if (req_tx_rings != bp->tx_nr_rings) {
+ netdev_warn(bp->dev,
+ "Able to reserve only %d out of %d requested TX rings\n",
+ bp->tx_nr_rings, req_tx_rings);
+ }
/* If we cannot reserve all the RX rings, reset the RSS map only
* if absolutely necessary
*/
- if (rx_rings != bp->rx_nr_rings) {
+ if (req_rx_rings != bp->rx_nr_rings) {
netdev_warn(bp->dev, "Able to reserve only %d out of %d requested RX rings\n",
- rx_rings, bp->rx_nr_rings);
- bnxt_check_rss_tbl_lost(bp, bp->rx_nr_rings, rx_rings);
+ bp->rx_nr_rings, req_rx_rings);
+ bnxt_check_rss_tbl_lost(bp, req_rx_rings, bp->rx_nr_rings);
}
- bp->rx_nr_rings = rx_rings;
- bp->cp_nr_rings = hwr.cp;
/* Fall back if we cannot reserve enough HW RSS contexts */
if ((bp->rss_cap & BNXT_RSS_CAP_LARGE_RSS_CTX) &&
hwr.rss_ctx < bnxt_get_total_rss_ctxs(bp, &hwr))
bp->rss_cap &= ~BNXT_RSS_CAP_LARGE_RSS_CTX;
- if (!bnxt_rings_ok(bp, &hwr))
- return -ENOMEM;
-
- if (old_rx_rings != bp->hw_resc.resv_rx_rings &&
+ if ((old_rx_rings != bp->hw_resc.resv_rx_rings ||
+ req_rx_rings != bp->rx_nr_rings) &&
!netif_is_rxfh_configured(bp->dev))
bnxt_set_dflt_rss_indir_tbl(bp, NULL);
+reserve_rings_exit:
if (!bnxt_ulp_registered(edev) && BNXT_NEW_RM(bp)) {
int resv_msix, resv_ctx, ulp_ctxs;
struct bnxt_hw_resc *hw_resc;
hw_resc = &bp->hw_resc;
- resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings;
+ resv_msix = max_t(int, hw_resc->resv_irqs - bp->cp_nr_rings, 0);
ulp_msix = min_t(int, resv_msix, ulp_msix);
bnxt_set_ulp_msix_num(bp, ulp_msix);
- resv_ctx = hw_resc->resv_stat_ctxs - bp->cp_nr_rings;
+ resv_ctx = max_t(int, hw_resc->resv_stat_ctxs - bp->cp_nr_rings,
+ 0);
ulp_ctxs = min(resv_ctx, bnxt_get_ulp_stat_ctxs(bp));
bnxt_set_ulp_stat_ctxs(bp, ulp_ctxs);
}
@@ -11317,14 +11362,27 @@ int bnxt_num_tx_to_cp(struct bnxt *bp, int tx, int tx_xdp, int rx)
{
int tcs = bp->num_tc ? bp->num_tc : 1;
+ if (tx_xdp) {
+ tx_xdp = tx / (tcs + 1);
+ tx_xdp = min(tx_xdp, rx);
+ }
return __bnxt_num_tx_to_cp(bp, tx, tcs, tx_xdp);
}
static int bnxt_num_cp_to_tx(struct bnxt *bp, int tx_cp, int tx_xdp, int rx)
{
int tcs = bp->num_tc ? bp->num_tc : 1;
+ int cp_per_tc;
- return (tx_cp - tx_xdp) * tcs + tx_xdp;
+ if (tx_xdp) {
+ /* The CP rings are divided equally between XDP TX rings and
+ * the regular TX rings.
+ */
+ cp_per_tc = tx_cp / 2;
+ cp_per_tc = min(cp_per_tc, rx);
+ return cp_per_tc * (tcs + 1);
+ }
+ return tx_cp * tcs;
}
static int bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max,
@@ -13156,13 +13214,29 @@ static void bnxt_set_xdp_tx_rings(struct bnxt *bp)
bp->tx_nr_rings += bp->tx_nr_rings_xdp;
}
-static void bnxt_adj_tx_rings(struct bnxt *bp)
+static void bnxt_adj_rings(struct bnxt *bp)
{
- /* Make adjustments if reserved TX rings are less than requested */
- bp->tx_nr_rings -= bp->tx_nr_rings_xdp;
- bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp);
- if (bp->tx_nr_rings_xdp)
- bnxt_set_xdp_tx_rings(bp);
+ int tcs = bp->num_tc ? bp->num_tc : 1;
+ int rx = bp->rx_nr_rings;
+
+ /* Make adjustments to rings based on TC/XDP/shared rings policies */
+ if (bp->tx_nr_rings_xdp) {
+ tcs++;
+ bp->tx_nr_rings_per_tc = bp->tx_nr_rings / tcs;
+ bp->tx_nr_rings_xdp = bp->tx_nr_rings_per_tc;
+ } else {
+ bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp);
+ }
+
+ if (bp->flags & BNXT_FLAG_SHARED_RINGS) {
+ bp->tx_nr_rings_per_tc = min(bp->tx_nr_rings_per_tc, rx);
+ bp->rx_nr_rings = bp->tx_nr_rings_per_tc;
+ if (bp->tx_nr_rings_xdp)
+ bp->tx_nr_rings_xdp = bp->tx_nr_rings_per_tc;
+ }
+
+ bp->tx_nr_rings = bp->tx_nr_rings_per_tc * tcs;
+ bnxt_set_cp_rings(bp, bp->flags & BNXT_FLAG_SHARED_RINGS);
}
static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
@@ -13182,7 +13256,8 @@ static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
if (rc)
return rc;
- bnxt_adj_tx_rings(bp);
+ if (irq_re_init)
+ bnxt_adj_rings(bp);
rc = bnxt_alloc_mem(bp, irq_re_init);
if (rc) {
netdev_err(bp->dev, "bnxt_alloc_mem err: %x\n", rc);
@@ -16876,7 +16951,7 @@ static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh)
if (rc && rc != -ECOMM)
netdev_warn(bp->dev, "Unable to reserve tx rings\n");
- bnxt_adj_tx_rings(bp);
+ bnxt_adj_rings(bp);
if (sh)
bnxt_adj_dflt_rings(bp, true);
@@ -16885,7 +16960,7 @@ static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh)
rc = __bnxt_reserve_rings(bp);
if (rc && rc != -ECOMM)
netdev_warn(bp->dev, "2nd rings reservation failed.\n");
- bnxt_adj_tx_rings(bp);
+ bnxt_adj_rings(bp);
}
if (rc) {
bp->tx_nr_rings = 0;
@@ -16915,7 +16990,7 @@ static int bnxt_init_dflt_ring_mode(struct bnxt *bp)
if (rc)
goto init_dflt_ring_err;
- bnxt_adj_tx_rings(bp);
+ bnxt_adj_rings(bp);
bnxt_set_dflt_rfs(bp);
--
2.51.0
next prev parent reply other threads:[~2026-11-20 1:30 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 1:29 [PATCH net-next v5 0/9] bnxt_en: Ring accounting fixes Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 1/9] bnxt_en: Clear bp->total_irqs when disabling IRQs Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 2/9] bnxt_en: Fix bnxt_reinit_features() when irq_re_init is true Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 3/9] bnxt_en: Refactor RSS table check logic Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 4/9] bnxt_en: Refactor IRQs required logic Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 5/9] bnxt_en: Reinit IRQ when configuring LRO/GRO/HDS Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 6/9] bnxt_en: Remove support for the old NITRO A0 chip Michael Chan
2026-11-20 1:29 ` [PATCH net-next v5 7/9] bnxt_en: Refactor bnxt_num_cp_to_tx()/bnxt_num_tx_to_cp() helpers Michael Chan
2026-11-20 1:29 ` Michael Chan [this message]
2026-11-20 1:29 ` [PATCH net-next v5 9/9] bnxt_en: Fix ring accounting/validation when IRQs are constrained Michael Chan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120012921.2537903-9-michael.chan@broadcom.com \
--to=michael.chan@broadcom.com \
--cc=andrew+netdev@lunn.ch \
--cc=andrew.gospodarek@broadcom.com \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pavan.chebbi@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox