hardware flows are left behind when a flowtable is deleted (nf_tables unbinds the offload block at commit time)

hardware flows are left behind when a flowtable is deleted (nf_tables unbinds the offload block at commit time)

[Martino Dell'Ambrogio]

Deleting a flowtable leaves the flows its devices offloaded behind in the
hardware. Reported against stable v6.12.108 on a Banana Pi R4 (MT7988,
mtk_eth_soc PPE) running OpenWrt, and reproduced there; the analysis below is
from that source and from net.git main, where the function is identical.

Symptom

After the flowtable is deleted, the offloaded flows stay BND in the PPE and
keep forwarding. Their conntrack entries are not released either: they stop
being refreshed and expire (60 s for UDP), after which the device is
forwarding flows that exist in hardware only and that conntrack no longer
knows about. On the router this is visible as entries in
/sys/kernel/debug/ppe{1,2}/bind with no matching entry in `conntrack -L`.

OpenWrt's fw4 deletes and recreates its flowtable in one transaction on every
firewall reload, so this happens on every reload rather than in some rare
teardown path.

Why

  - Deleting a flowtable moves FLOW_BLOCK_UNBIND to commit time.
    __nft_unregister_flowtable_net_hooks() asks the driver to unbind, and
    nf_flow_table_block_setup() frees the driver callback there.
  - The flowtable itself is freed later, from the transaction destroy work
    after synchronize_rcu(): nft_commit_release() ->
    nf_tables_flowtable_destroy() -> type->free() -> nf_flow_table_free().
    That last step tears the remaining flows down and queues
    FLOW_CLS_DESTROY for each one.
  - nf_flow_offload_tuple() delivers those commands by walking
    flowtable->flow_block.cb_list, which has been empty since the commit. It
    reports no error for an empty list, so nothing notices.

This is not a race between the last GC run and the unbind: the unbind always
happens first, at commit time, so the hardware entries are never removed.

It is a regression from 13210fc63f35 ("netfilter: nf_tables: imbalance in
flowtable binding"). That commit removed the late FLOW_BLOCK_UNBIND from the
flowtable destroy path, which 5acab91458ce ("netfilter: nf_tables: unbind
callbacks from flowtable destroy path") had placed after
nf_flow_table_free() on purpose:

    Callback unbinding needs to be done after nf_flow_table_free(),
    otherwise entries are not removed from the hardware.

13210fc63f35's thread only checked BIND/UNBIND balance; nothing there is
about hardware entries left behind. It is in 6.12.y as 2e87c203b72f.

All of the above is read from the sources rather than instrumented at run
time: nothing here traces cb_list directly. What the reproducer below shows
is the consequence, and that part is measured.

Reproducer (~2 minutes, one host)

  1. Drive a busy UDP flow (~100 pps) through the flowtable so it binds in
     hardware: confirm a BND entry in /sys/kernel/debug/ppe*/bind and
     [HW_OFFLOAD] in `conntrack -L`.
  2. Delete the flowtable (on OpenWrt: `fw4 reload`, which deletes and
     recreates it).
  3. Keep the same socket sending, at a low rate, for longer than the
     conntrack timeout. The window has to exceed it: a stranded flow still has
     a conntrack entry until that expires, so a shorter window shows nothing.

  Unpatched: the tuple stays BND and keeps forwarding while conntrack has no
  entry for it any more.
  The discriminator has to be per tuple -- bound in hardware AND absent from
  conntrack. An aggregate "no conntrack entries left" test never fires, because
  other hosts on the LAN are talking to the same destination.

Scope

Drivers that offload through TC_SETUP_FT and rely on FLOW_CLS_DESTROY to
release hardware state look affected by inspection: mtk_eth_soc (measured),
mtk_wed, airoha, mt76 npu, mlx5 representor FT. Drivers that flush their own
entries when they remove their callback (mlx5 tc_ct, nfp, sfc) are not. Only
mtk_eth_soc was tested.

A possible fix, and what I measured

Remove the flows using the device before the block is unbound:
nf_flow_table_offload_setup() on FLOW_BLOCK_UNBIND could call
nf_flow_table_gc_cleanup(flowtable, dev) first, so the FLOW_CLS_DESTROY is
delivered while the callback is still on cb_list. That is what the kernel
already does for indirect devices (nf_flow_table_indr_cleanup()) and on
NETDEV_DOWN, and one site covers the commit, update, netdev-event and abort
paths. I am aware that "generalize pending status bit" reworks this area.

I have not sent a patch, but I have built that change and run it on the
hardware: two images differing by that one hunk and nothing else, the same
reproducer on each.

  - Unmodified kernel: the reproducer strands the flows. All 4 flows stay
    bound in the PPE, in 26 of 26 samples taken after the reload -- first
    sustained 22.7 s in and still true at 167 s, past both the 20 s and the
    60 s UDP timeouts.
  - With the change: no flow is left bound with no conntrack entry behind
    it -- 0 of 4, clean in all 25 samples, and the flows re-bind under the
    new flowtable.

An orphan check on the patched image (every bound entry cross-referenced
against conntrack, with a synthetic control entry injected to show the check
can fire) found none at +10 minutes or +1 hour. The change is carried as a
downstream patch in OpenWrt.

  - Martino Dell'Ambrogio <tillo@tillo.ch>

Assisted-by: LLM

Re: hardware flows are left behind when a flowtable is deleted (nf_tables unbinds the offload block at commit time)

[Pablo Neira Ayuso]

Hi,

On Mon, Oct 05, 2026 at 01:48:10PM +0200, Martino Dell'Ambrogio wrote:
> Deleting a flowtable leaves the flows its devices offloaded behind in the
> hardware. Reported against stable v6.12.108 on a Banana Pi R4 (MT7988,
> mtk_eth_soc PPE) running OpenWrt, and reproduced there; the analysis below is
> from that source and from net.git main, where the function is identical.

I have restored my Pi R64 testbed with OpenWrt.

Let me reproduce this issue and review all pending fixes for -stable
kernels too for the flowtable hardware offload.

I will get back to you with updates, thanks for reporting.

> Symptom
> 
> After the flowtable is deleted, the offloaded flows stay BND in the PPE and
> keep forwarding. Their conntrack entries are not released either: they stop
> being refreshed and expire (60 s for UDP), after which the device is
> forwarding flows that exist in hardware only and that conntrack no longer
> knows about. On the router this is visible as entries in
> /sys/kernel/debug/ppe{1,2}/bind with no matching entry in `conntrack -L`.
> 
> OpenWrt's fw4 deletes and recreates its flowtable in one transaction on every
> firewall reload, so this happens on every reload rather than in some rare
> teardown path.
> 
> Why
> 
>   - Deleting a flowtable moves FLOW_BLOCK_UNBIND to commit time.
>     __nft_unregister_flowtable_net_hooks() asks the driver to unbind, and
>     nf_flow_table_block_setup() frees the driver callback there.
>   - The flowtable itself is freed later, from the transaction destroy work
>     after synchronize_rcu(): nft_commit_release() ->
>     nf_tables_flowtable_destroy() -> type->free() -> nf_flow_table_free().
>     That last step tears the remaining flows down and queues
>     FLOW_CLS_DESTROY for each one.
>   - nf_flow_offload_tuple() delivers those commands by walking
>     flowtable->flow_block.cb_list, which has been empty since the commit. It
>     reports no error for an empty list, so nothing notices.
> 
> This is not a race between the last GC run and the unbind: the unbind always
> happens first, at commit time, so the hardware entries are never removed.
> 
> It is a regression from 13210fc63f35 ("netfilter: nf_tables: imbalance in
> flowtable binding"). That commit removed the late FLOW_BLOCK_UNBIND from the
> flowtable destroy path, which 5acab91458ce ("netfilter: nf_tables: unbind
> callbacks from flowtable destroy path") had placed after
> nf_flow_table_free() on purpose:
> 
>     Callback unbinding needs to be done after nf_flow_table_free(),
>     otherwise entries are not removed from the hardware.
> 
> 13210fc63f35's thread only checked BIND/UNBIND balance; nothing there is
> about hardware entries left behind. It is in 6.12.y as 2e87c203b72f.
> 
> All of the above is read from the sources rather than instrumented at run
> time: nothing here traces cb_list directly. What the reproducer below shows
> is the consequence, and that part is measured.
> 
> Reproducer (~2 minutes, one host)
> 
>   1. Drive a busy UDP flow (~100 pps) through the flowtable so it binds in
>      hardware: confirm a BND entry in /sys/kernel/debug/ppe*/bind and
>      [HW_OFFLOAD] in `conntrack -L`.
>   2. Delete the flowtable (on OpenWrt: `fw4 reload`, which deletes and
>      recreates it).
>   3. Keep the same socket sending, at a low rate, for longer than the
>      conntrack timeout. The window has to exceed it: a stranded flow still has
>      a conntrack entry until that expires, so a shorter window shows nothing.
> 
>   Unpatched: the tuple stays BND and keeps forwarding while conntrack has no
>   entry for it any more.
>   The discriminator has to be per tuple -- bound in hardware AND absent from
>   conntrack. An aggregate "no conntrack entries left" test never fires, because
>   other hosts on the LAN are talking to the same destination.
> 
> Scope
> 
> Drivers that offload through TC_SETUP_FT and rely on FLOW_CLS_DESTROY to
> release hardware state look affected by inspection: mtk_eth_soc (measured),
> mtk_wed, airoha, mt76 npu, mlx5 representor FT. Drivers that flush their own
> entries when they remove their callback (mlx5 tc_ct, nfp, sfc) are not. Only
> mtk_eth_soc was tested.
> 
> A possible fix, and what I measured
> 
> Remove the flows using the device before the block is unbound:
> nf_flow_table_offload_setup() on FLOW_BLOCK_UNBIND could call
> nf_flow_table_gc_cleanup(flowtable, dev) first, so the FLOW_CLS_DESTROY is
> delivered while the callback is still on cb_list. That is what the kernel
> already does for indirect devices (nf_flow_table_indr_cleanup()) and on
> NETDEV_DOWN, and one site covers the commit, update, netdev-event and abort
> paths. I am aware that "generalize pending status bit" reworks this area.
> 
> I have not sent a patch, but I have built that change and run it on the
> hardware: two images differing by that one hunk and nothing else, the same
> reproducer on each.
> 
>   - Unmodified kernel: the reproducer strands the flows. All 4 flows stay
>     bound in the PPE, in 26 of 26 samples taken after the reload -- first
>     sustained 22.7 s in and still true at 167 s, past both the 20 s and the
>     60 s UDP timeouts.
>   - With the change: no flow is left bound with no conntrack entry behind
>     it -- 0 of 4, clean in all 25 samples, and the flows re-bind under the
>     new flowtable.
> 
> An orphan check on the patched image (every bound entry cross-referenced
> against conntrack, with a synthetic control entry injected to show the check
> can fire) found none at +10 minutes or +1 hour. The change is carried as a
> downstream patch in OpenWrt.
> 
>   - Martino Dell'Ambrogio <tillo@tillo.ch>
> 
> Assisted-by: LLM
>