* [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX
2026-11-20 20:24 [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic Chris Babroski
@ 2026-11-20 20:24 ` Chris Babroski
2026-11-20 20:24 ` [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic Chris Babroski
2026-11-20 20:27 ` [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic netdev-bot+sinfo
2 siblings, 0 replies; 5+ messages in thread
From: Chris Babroski @ 2026-11-20 20:24 UTC (permalink / raw)
To: netdev, linux-kernel
Cc: andrew+netdev, davem, edumazet, kuba, pabeni, davthompson,
cbabroski
On BlueField, firmware can disable and reset the OOB MAC when it handles
a fatal error or panic. If Linux is still actively transmitting packets
while the MAC is disabled, then it's possible for the HW consumer index
to be reset while the SW completion index in the driver remains the
same. This can cause the two indices to suddenly become far apart, with
a difference that is potentially larger than the TX queue length itself.
Previously, mlxbf_gige_handle_tx_complete() would continue to loop
around the TX queue, counting packets and freeing their memory, until
the two indices were equal. If entries were already freed the first time
through the queue, then a second attempt could cause a NULL pointer
dereference.
Bound the loop over the length of the TX queue so entries are not
handled more than once, and check that the skb at tx_wqe_index is not
NULL before handling.
After the loop, set the SW completion index to the new HW consumer index
to prevent future polls from walking the queue again. In normal
operation the loop already exits with the two indices equal, so there is
no change in behavior.
TX is not recovered after a sudden MAC disable. If firmware has disabled
the MAC then the system is in a state where it will be rebooting
automatically or needs manual intervention to recover.
Signed-off-by: Chris Babroski <cbabroski@nvidia.com>
---
.../mellanox/mlxbf_gige/mlxbf_gige_tx.c | 29 ++++++++++++-------
1 file changed, 19 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
index 04982e888c63..73c3033591d5 100644
--- a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
+++ b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_tx.c
@@ -130,6 +130,7 @@ bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
u64 *tx_wqe_addr;
u64 tx_status;
u16 tx_ci;
+ u16 entry;
tx_status = readq(priv->base + MLXBF_GIGE_TX_STATUS);
if (tx_status & MLXBF_GIGE_TX_STATUS_DATA_FIFO_FULL)
@@ -143,7 +144,8 @@ bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
* to be supported, that is TX consumer index wrapped from 0xFFFF
* to 0 while TX completion index is still < 0xFFFF.
*/
- for (; priv->prev_tx_ci != tx_ci; priv->prev_tx_ci++) {
+ for (entry = 0; priv->prev_tx_ci != tx_ci && entry < priv->tx_q_entries;
+ priv->prev_tx_ci++, entry++) {
tx_wqe_index = priv->prev_tx_ci % priv->tx_q_entries;
/* Each TX WQE is 16 bytes. The 8 MSB store the 2KB TX
* buffer address and the 8 LSB contain information
@@ -152,18 +154,25 @@ bool mlxbf_gige_handle_tx_complete(struct mlxbf_gige *priv)
tx_wqe_addr = priv->tx_wqe_base +
(tx_wqe_index * MLXBF_GIGE_TX_WQE_SZ_QWORDS);
- stats->tx_packets++;
- stats->tx_bytes += MLXBF_GIGE_TX_WQE_PKT_LEN(tx_wqe_addr);
-
- dma_unmap_single(priv->dev, *tx_wqe_addr,
- priv->tx_skb[tx_wqe_index]->len, DMA_TO_DEVICE);
- dev_consume_skb_any(priv->tx_skb[tx_wqe_index]);
- priv->tx_skb[tx_wqe_index] = NULL;
+ if (priv->tx_skb[tx_wqe_index]) {
+ stats->tx_packets++;
+ stats->tx_bytes += MLXBF_GIGE_TX_WQE_PKT_LEN(tx_wqe_addr);
+ dma_unmap_single(priv->dev, *tx_wqe_addr,
+ priv->tx_skb[tx_wqe_index]->len, DMA_TO_DEVICE);
+ dev_consume_skb_any(priv->tx_skb[tx_wqe_index]);
+ priv->tx_skb[tx_wqe_index] = NULL;
- /* Ensure completion of updates across all cores */
- mb();
+ /* Ensure completion of updates across all cores */
+ mb();
+ }
}
+ /* At this point the completion index should be equal to the
+ * consumer index. If they aren't then the driver is out of sync
+ * from the HW's state and needs to be adjusted.
+ */
+ priv->prev_tx_ci = tx_ci;
+
/* Since the TX ring was likely just drained, check if TX queue
* had previously been stopped and now that there are TX buffers
* available the TX queue can be awakened.
--
2.34.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic
2026-11-20 20:24 [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic Chris Babroski
2026-11-20 20:24 ` [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX Chris Babroski
@ 2026-11-20 20:24 ` Chris Babroski
2026-11-20 20:27 ` [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic netdev-bot+sinfo
2 siblings, 0 replies; 5+ messages in thread
From: Chris Babroski @ 2026-11-20 20:24 UTC (permalink / raw)
To: netdev, linux-kernel
Cc: andrew+netdev, davem, edumazet, kuba, pabeni, davthompson,
cbabroski
After a kernel panic the MAC continues to receive packets into the RX
FIFO even though the driver can no longer process them. By default, the
MAC HW is designed to repeatedly send pause frames to its link partner
when the RX FIFO is full. The internal Ethernet switch on BlueField that
is connected to this port is also designed to stop traffic on all ports
when it receives a high number of continuous pause frames from one of
the ports.
What this means is that when the kernel panics on BlueField, it is
possible to reproduce a scenario under steady and high network load
where the network becomes inaccessible for other devices connected to
the 3-port switch, namely the OOB host and DPU-BMC.
Register a panic notifier at highest priority that disables MAC RX and
clears the RX FIFO to ensure that the MAC can no longer receive packets
or send pause frames.
Signed-off-by: Chris Babroski <cbabroski@nvidia.com>
---
.../ethernet/mellanox/mlxbf_gige/mlxbf_gige.h | 2 +
.../mellanox/mlxbf_gige/mlxbf_gige_main.c | 39 +++++++++++++++++++
2 files changed, 41 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
index e7777700ee18..c63bdfae4344 100644
--- a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
+++ b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige.h
@@ -13,6 +13,7 @@
#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/irqreturn.h>
#include <linux/netdevice.h>
+#include <linux/notifier.h>
#include <linux/irq.h>
#include <linux/phy.h>
@@ -130,6 +131,7 @@ struct mlxbf_gige {
u8 hw_version;
struct mlxbf_gige_mdio_gw *mdio_gw;
int prev_speed;
+ struct notifier_block panic_nb;
};
/* Rx Work Queue Element definitions */
diff --git a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
index d1f8a72cae53..76da71108345 100644
--- a/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
+++ b/drivers/net/ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c
@@ -12,6 +12,7 @@
#include <linux/interrupt.h>
#include <linux/iopoll.h>
#include <linux/module.h>
+#include <linux/panic_notifier.h>
#include <linux/phy.h>
#include <linux/platform_device.h>
#include <linux/rtnetlink.h>
@@ -127,6 +128,27 @@ static int mlxbf_gige_clean_port(struct mlxbf_gige *priv)
return err;
}
+static void mlxbf_gige_disable_port(struct mlxbf_gige *priv)
+{
+ u64 control;
+
+ /* Disable RX DMA */
+ control = readq(priv->base + MLXBF_GIGE_RX_DMA);
+ control &= ~MLXBF_GIGE_RX_DMA_EN;
+ writeq(control, priv->base + MLXBF_GIGE_RX_DMA);
+
+ /* Disable port */
+ control = readq(priv->base + MLXBF_GIGE_CONTROL);
+ control &= ~MLXBF_GIGE_CONTROL_PORT_EN;
+ writeq(control, priv->base + MLXBF_GIGE_CONTROL);
+
+ /* Disable interrupts */
+ writeq(0, priv->base + MLXBF_GIGE_INT_EN);
+
+ /* Ensure completion of disable writes before cleaning the port */
+ mb();
+}
+
static int mlxbf_gige_open(struct net_device *netdev)
{
struct mlxbf_gige *priv = netdev_priv(netdev);
@@ -382,6 +404,17 @@ static struct mlxbf_gige_link_cfg mlxbf_gige_link_cfgs[] = {
}
};
+static int mlxbf_gige_on_panic(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct mlxbf_gige *priv;
+
+ priv = container_of(nb, struct mlxbf_gige, panic_nb);
+ mlxbf_gige_disable_port(priv);
+ mlxbf_gige_clean_port(priv);
+ return NOTIFY_DONE;
+}
+
static int mlxbf_gige_probe(struct platform_device *pdev)
{
struct phy_device *phydev;
@@ -446,6 +479,10 @@ static int mlxbf_gige_probe(struct platform_device *pdev)
/* Write initial MAC address to hardware */
mlxbf_gige_initial_mac(priv);
+ priv->panic_nb.priority = INT_MAX;
+ priv->panic_nb.notifier_call = mlxbf_gige_on_panic;
+ atomic_notifier_chain_register(&panic_notifier_list, &priv->panic_nb);
+
err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
if (err) {
dev_err(&pdev->dev, "DMA configuration failed: %pe\n", ERR_PTR(err));
@@ -497,6 +534,7 @@ static int mlxbf_gige_probe(struct platform_device *pdev)
return 0;
out:
+ atomic_notifier_chain_unregister(&panic_notifier_list, &priv->panic_nb);
mlxbf_gige_mdio_remove(priv);
return err;
}
@@ -507,6 +545,7 @@ static void mlxbf_gige_remove(struct platform_device *pdev)
unregister_netdev(priv->netdev);
phy_disconnect(priv->netdev->phydev);
+ atomic_notifier_chain_unregister(&panic_notifier_list, &priv->panic_nb);
mlxbf_gige_mdio_remove(priv);
platform_set_drvdata(pdev, NULL);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic
2026-11-20 20:24 [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic Chris Babroski
2026-11-20 20:24 ` [PATCH net-next v1 1/2] mlxbf_gige: protect against MAC disable during TX Chris Babroski
2026-11-20 20:24 ` [PATCH net-next v1 2/2] mlxbf_gige: disable MAC after kernel panic Chris Babroski
@ 2026-11-20 20:27 ` netdev-bot+sinfo
2026-11-20 20:43 ` Chris Babroski
2 siblings, 1 reply; 5+ messages in thread
From: netdev-bot+sinfo @ 2026-11-20 20:27 UTC (permalink / raw)
To: Chris Babroski
Cc: netdev, linux-kernel, andrew+netdev, davem, edumazet, kuba,
pabeni, davthompson
Hi!
This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:
- How the issue was discovered, e.g. hit in production, hit during
development, syzbot report, manual code inspection, LLM or static
analysis tool scan.
Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account.
*IMPORTANT*: If you do repost (for other reasons) - please include
the relevant info in the cover letter or commit messages.
--
The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic
2026-11-20 20:27 ` [PATCH net-next v1 0/2] mlxbf_gige: disable MAC after panic netdev-bot+sinfo
@ 2026-11-20 20:43 ` Chris Babroski
0 siblings, 0 replies; 5+ messages in thread
From: Chris Babroski @ 2026-11-20 20:43 UTC (permalink / raw)
To: netdev-bot+sinfo
Cc: andrew+netdev, cbabroski, davem, davthompson, edumazet, kuba,
linux-kernel, netdev, pabeni
> This is an automated message. This series looks like a fix, but its
> commit messages seem to be missing some information:
>
> - How the issue was discovered, e.g. hit in production, hit during
> development, syzbot report, manual code inspection, LLM or static
> analysis tool scan.
This is a theoretical issue that has not yet been seen in production. It
can, however, be reproduced manually. It was found because we hit a
similar issue in production last year that caused OOB network traffic to
stop when our UEFI FW could not process packets fast enough during HTTP
boot to keep the RX FIFO drained (see cover letter for more background
about how this affects the network).
After fixing the FW to prevent that case from occurring, we decided it
would be good to add protection to our kernel driver to ensure that a
kernel panic does not leave the OOB MAC in a state where the RX FIFO can
continue to fill while nothing can drain it.
Chris
^ permalink raw reply [flat|nested] 5+ messages in thread