[PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[Wei Fang]

In the current implementation, the enetc_xdp_xmit() always transmits
redirected XDP frames even if the link is down, but the frames cannot
be transmitted from TX BD rings when the link is down, so the frames
are still kept in the TX BD rings. If the XDP program is uninstalled,
users will see the following warning logs.

fsl_enetc 0000:00:00.0 eno0: timeout for tx ring #6 clear

More worse, the TX BD ring cannot work properly anymore, because the
HW PIR and CIR are not the same after the re-initialization of the TX
BD ring. And I see no reasons to transmit the redirected XDP frames
when the link is down, so add a link status check to quickly fix this
issue. However, this solution does not completely solve the problem,
for example, if the link is broken during transmission and the TX BD
ring still has unsent frames. I think this requires another patch to
address this situation, but it will not conflict with the current
solution and can coexist.

Fixes: 9d2b68cc108d ("net: enetc: add support for XDP_REDIRECT")
Signed-off-by: Wei Fang <wei.fang@nxp.com>
---
 drivers/net/ethernet/freescale/enetc/enetc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c
index 0535e92404e3..f410c245ea91 100644
--- a/drivers/net/ethernet/freescale/enetc/enetc.c
+++ b/drivers/net/ethernet/freescale/enetc/enetc.c
@@ -1778,7 +1778,8 @@ int enetc_xdp_xmit(struct net_device *ndev, int num_frames,
 	int xdp_tx_bd_cnt, i, k;
 	int xdp_tx_frm_cnt = 0;
 
-	if (unlikely(test_bit(ENETC_TX_DOWN, &priv->flags)))
+	if (unlikely(test_bit(ENETC_TX_DOWN, &priv->flags) ||
+		     !netif_carrier_ok(ndev)))
 		return -ENETDOWN;
 
 	enetc_lock_mdio();
-- 
2.34.1

Re: [PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[Vladimir Oltean]

On Fri, Dec 05, 2025 at 06:53:07PM +0800, Wei Fang wrote:
> In the current implementation, the enetc_xdp_xmit() always transmits
> redirected XDP frames even if the link is down, but the frames cannot
> be transmitted from TX BD rings when the link is down, so the frames
> are still kept in the TX BD rings. If the XDP program is uninstalled,
> users will see the following warning logs.
> 
> fsl_enetc 0000:00:00.0 eno0: timeout for tx ring #6 clear
> 
> More worse, the TX BD ring cannot work properly anymore, because the
> HW PIR and CIR are not the same after the re-initialization of the TX
> BD ring.

I understand and I don't disagree that the TX BD ring doesn't work
anymore if we disable it while it has pending frames (the TB0MR[EN]
documentation says that this is unsafe too), but:
- I don't understand why the hardware PIR and CIR are not the same after
  the TX ring reinitialization
- I don't understand how the effect and the claimed cause are connected

Could you please give more details what you mean here?

> And I see no reasons to transmit the redirected XDP frames
> when the link is down, so add a link status check to quickly fix this
> issue. However, this solution does not completely solve the problem,
> for example, if the link is broken during transmission and the TX BD
> ring still has unsent frames. I think this requires another patch to
> address this situation, but it will not conflict with the current
> solution and can coexist.
> 
> Fixes: 9d2b68cc108d ("net: enetc: add support for XDP_REDIRECT")
> Signed-off-by: Wei Fang <wei.fang@nxp.com>
> ---
>  drivers/net/ethernet/freescale/enetc/enetc.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c
> index 0535e92404e3..f410c245ea91 100644
> --- a/drivers/net/ethernet/freescale/enetc/enetc.c
> +++ b/drivers/net/ethernet/freescale/enetc/enetc.c
> @@ -1778,7 +1778,8 @@ int enetc_xdp_xmit(struct net_device *ndev, int num_frames,
>  	int xdp_tx_bd_cnt, i, k;
>  	int xdp_tx_frm_cnt = 0;
>  
> -	if (unlikely(test_bit(ENETC_TX_DOWN, &priv->flags)))
> +	if (unlikely(test_bit(ENETC_TX_DOWN, &priv->flags) ||
> +		     !netif_carrier_ok(ndev)))
>  		return -ENETDOWN;
>  
>  	enetc_lock_mdio();
> -- 
> 2.34.1
>

RE: [PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[Wei Fang]

> On Fri, Dec 05, 2025 at 06:53:07PM +0800, Wei Fang wrote:
> > In the current implementation, the enetc_xdp_xmit() always transmits
> > redirected XDP frames even if the link is down, but the frames cannot
> > be transmitted from TX BD rings when the link is down, so the frames
> > are still kept in the TX BD rings. If the XDP program is uninstalled,
> > users will see the following warning logs.
> >
> > fsl_enetc 0000:00:00.0 eno0: timeout for tx ring #6 clear
> >
> > More worse, the TX BD ring cannot work properly anymore, because the
> > HW PIR and CIR are not the same after the re-initialization of the TX
> > BD ring.
> 
> I understand and I don't disagree that the TX BD ring doesn't work
> anymore if we disable it while it has pending frames (the TB0MR[EN]
> documentation says that this is unsafe too), but:
> - I don't understand why the hardware PIR and CIR are not the same after
>   the TX ring reinitialization
> - I don't understand how the effect and the claimed cause are connected
> 
> Could you please give more details what you mean here?

Currently, the hardware PIR and CIR are not initialized by the software
when the TX BD is re-initialized. The driver just reads HW PIR and CIR and
then initializes the SW PIR and CIR. See enetc_setup_txbdr():

/* clearing PI/CI registers for Tx not supported, adjust sw indexes */
tx_ring->next_to_use = enetc_txbdr_rd(hw, idx, ENETC_TBPIR);
tx_ring->next_to_clean = enetc_txbdr_rd(hw, idx, ENETC_TBCIR);

If there are unsent frames on the TX BD ring, the HW PIR and CIR are
not equal when the TX BD ring is disabled. So if the TX BD ring is
re-initialized at that time, the unsent frames will be freed and HW
PIR and CIR are still not equal after the re-initialization. At this point, 
the BDs between CIR and PIR are invalid, which will cause a hardware
malfunction.

Another reason is that there is internal context in the ring prefetch
logic that will retain the state from the first incarnation of the ring
and continue prefetching from the stale location when we re-initialize
the ring. The internal context is only reset by an FLR. That is to say,
for LS1028A ENETC, software cannot set the HW CIR and PIR when
initializing the TX BD ring.

The best solution is to either not initialize the TX BD ring or use FLR
to initialize it when this situation (the TX BD ring still has unsent
frames) occurs. Either approach involves complex modifications,
especially the FLR method. I don't have enough time to fix this issue
for the LS1028A. At least for now, this patch is what I can do, and it
doesn't conflict with subsequent solutions.

Re: [PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[Vladimir Oltean]

On Tue, Dec 09, 2025 at 11:08:08AM +0200, Wei Fang wrote:
> > On Fri, Dec 05, 2025 at 06:53:07PM +0800, Wei Fang wrote:
> > > In the current implementation, the enetc_xdp_xmit() always transmits
> > > redirected XDP frames even if the link is down, but the frames cannot
> > > be transmitted from TX BD rings when the link is down, so the frames
> > > are still kept in the TX BD rings. If the XDP program is uninstalled,
> > > users will see the following warning logs.
> > >
> > > fsl_enetc 0000:00:00.0 eno0: timeout for tx ring #6 clear
> > >
> > > More worse, the TX BD ring cannot work properly anymore, because the
> > > HW PIR and CIR are not the same after the re-initialization of the TX
> > > BD ring.
> > 
> > I understand and I don't disagree that the TX BD ring doesn't work
> > anymore if we disable it while it has pending frames (the TB0MR[EN]
> > documentation says that this is unsafe too), but:
> > - I don't understand why the hardware PIR and CIR are not the same after
> >   the TX ring reinitialization
> > - I don't understand how the effect and the claimed cause are connected
> > 
> > Could you please give more details what you mean here?
> 
> Currently, the hardware PIR and CIR are not initialized by the software
> when the TX BD is re-initialized. The driver just reads HW PIR and CIR and
> then initializes the SW PIR and CIR. See enetc_setup_txbdr():
> 
> /* clearing PI/CI registers for Tx not supported, adjust sw indexes */
> tx_ring->next_to_use = enetc_txbdr_rd(hw, idx, ENETC_TBPIR);
> tx_ring->next_to_clean = enetc_txbdr_rd(hw, idx, ENETC_TBCIR);
> 
> If there are unsent frames on the TX BD ring, the HW PIR and CIR are
> not equal when the TX BD ring is disabled. So if the TX BD ring is
> re-initialized at that time, the unsent frames will be freed and HW
> PIR and CIR are still not equal after the re-initialization. At this point, 
> the BDs between CIR and PIR are invalid, which will cause a hardware
> malfunction.

Ah, ok, I genuinely didn't understand what you meant by "they are not
the same after reinitialization". I thought you're saying that
enetc_reconfigure() runs, and the next_to_use and next_to_clean values
are not what they were before... which they are, according to the code
you pointed out. You meant "they are not the same" in the sense that
they are not equal to one another... I think this really isn't clear.

> 
> Another reason is that there is internal context in the ring prefetch
> logic that will retain the state from the first incarnation of the ring
> and continue prefetching from the stale location when we re-initialize
> the ring. The internal context is only reset by an FLR. That is to say,
> for LS1028A ENETC, software cannot set the HW CIR and PIR when
> initializing the TX BD ring.
> 
> The best solution is to either not initialize the TX BD ring or use FLR
> to initialize it when this situation (the TX BD ring still has unsent
> frames) occurs. Either approach involves complex modifications,
> especially the FLR method. I don't have enough time to fix this issue
> for the LS1028A. At least for now, this patch is what I can do, and it
> doesn't conflict with subsequent solutions.

I'm wondering if this situation can be completely avoided in the first place.
For i.MX9, I did see a "graceful stop" section in the NETC reference
manual, making use of POR[TXDIS]. Would this help? For LS1028A, I'm still
searching, but there's nothing conclusive... I'll experiment with putting
the MAC in loopback via COMMAND_CONFIG[XGLP] and then drop the received
frames somehow.

I think I agree we should try to avoid sending packets during link down
even if we later have to recover from those packets we couldn't avoid
sending. It is just to try and not make the problem worse, and to make
the recovery procedure deal with a bounded amount of packets rather than
a continuous flow.

Could you please resend with an improved commit message where you
integrate the clarifications made here?

RE: [PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[Wei Fang]

> On Tue, Dec 09, 2025 at 11:08:08AM +0200, Wei Fang wrote:
> > > On Fri, Dec 05, 2025 at 06:53:07PM +0800, Wei Fang wrote:
> > > > In the current implementation, the enetc_xdp_xmit() always transmits
> > > > redirected XDP frames even if the link is down, but the frames cannot
> > > > be transmitted from TX BD rings when the link is down, so the frames
> > > > are still kept in the TX BD rings. If the XDP program is uninstalled,
> > > > users will see the following warning logs.
> > > >
> > > > fsl_enetc 0000:00:00.0 eno0: timeout for tx ring #6 clear
> > > >
> > > > More worse, the TX BD ring cannot work properly anymore, because the
> > > > HW PIR and CIR are not the same after the re-initialization of the TX
> > > > BD ring.
> > >
> > > I understand and I don't disagree that the TX BD ring doesn't work
> > > anymore if we disable it while it has pending frames (the TB0MR[EN]
> > > documentation says that this is unsafe too), but:
> > > - I don't understand why the hardware PIR and CIR are not the same after
> > >   the TX ring reinitialization
> > > - I don't understand how the effect and the claimed cause are connected
> > >
> > > Could you please give more details what you mean here?
> >
> > Currently, the hardware PIR and CIR are not initialized by the software
> > when the TX BD is re-initialized. The driver just reads HW PIR and CIR and
> > then initializes the SW PIR and CIR. See enetc_setup_txbdr():
> >
> > /* clearing PI/CI registers for Tx not supported, adjust sw indexes */
> > tx_ring->next_to_use = enetc_txbdr_rd(hw, idx, ENETC_TBPIR);
> > tx_ring->next_to_clean = enetc_txbdr_rd(hw, idx, ENETC_TBCIR);
> >
> > If there are unsent frames on the TX BD ring, the HW PIR and CIR are
> > not equal when the TX BD ring is disabled. So if the TX BD ring is
> > re-initialized at that time, the unsent frames will be freed and HW
> > PIR and CIR are still not equal after the re-initialization. At this point,
> > the BDs between CIR and PIR are invalid, which will cause a hardware
> > malfunction.
> 
> Ah, ok, I genuinely didn't understand what you meant by "they are not
> the same after reinitialization". I thought you're saying that
> enetc_reconfigure() runs, and the next_to_use and next_to_clean values
> are not what they were before... which they are, according to the code
> you pointed out. You meant "they are not the same" in the sense that
> they are not equal to one another... I think this really isn't clear.
> 
> >
> > Another reason is that there is internal context in the ring prefetch
> > logic that will retain the state from the first incarnation of the ring
> > and continue prefetching from the stale location when we re-initialize
> > the ring. The internal context is only reset by an FLR. That is to say,
> > for LS1028A ENETC, software cannot set the HW CIR and PIR when
> > initializing the TX BD ring.
> >
> > The best solution is to either not initialize the TX BD ring or use FLR
> > to initialize it when this situation (the TX BD ring still has unsent
> > frames) occurs. Either approach involves complex modifications,
> > especially the FLR method. I don't have enough time to fix this issue
> > for the LS1028A. At least for now, this patch is what I can do, and it
> > doesn't conflict with subsequent solutions.
> 
> I'm wondering if this situation can be completely avoided in the first place.
> For i.MX9, I did see a "graceful stop" section in the NETC reference
> manual, making use of POR[TXDIS]. Would this help? For LS1028A, I'm still

No, it does not help, but ENETC v4 supports setting HW PIR and CIR by
software, the latest NETC BG has updated this info after I checked with
NETC IP team. So I have planned to add a fix patch for i.MX9 after this
patch is applied.

> searching, but there's nothing conclusive... I'll experiment with putting
> the MAC in loopback via COMMAND_CONFIG[XGLP] and then drop the received
> frames somehow.
> 
> I think I agree we should try to avoid sending packets during link down
> even if we later have to recover from those packets we couldn't avoid
> sending. It is just to try and not make the problem worse, and to make
> the recovery procedure deal with a bounded amount of packets rather than
> a continuous flow.
> 
> Could you please resend with an improved commit message where you
> integrate the clarifications made here?

Yes, I will improve the commit message

RE: [PATCH net] net: enetc: do not transmit redirected XDP frames when the link is down

[Wei Fang]

> searching, but there's nothing conclusive... I'll experiment with putting
> the MAC in loopback via COMMAND_CONFIG[XGLP] and then drop the received
> frames somehow.

That's great! I think this method is feasible. By sending the packets through
loopback, the TX BD ring could be cleared.