[REGRESSION] netfilter: flowtable: HW offloaded flows are not removed from hardware on flowtable deletion since 13210fc63f35

[REGRESSION] netfilter: flowtable: HW offloaded flows are not removed from hardware on flowtable deletion since 13210fc63f35

[Jalen]

Hi,

Since 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable
binding"), deleting a flowtable with "flags offload" leaves its flows
in hardware. FLOW_BLOCK_UNBIND now runs at commit time, before
nf_flow_table_free(), so the FLOW_CLS_DESTROY calls made by the free
find no driver callback. The hardware keeps forwarding (and NATing)
those flows with no flowtable or ruleset left. Current master is
affected. Seen on mtk_eth_soc and mlx5.

This is the ordering that 5acab91458ce ("netfilter: nf_tables: unbind
callbacks from flowtable destroy path") fixed in 2020:

  Callback unbinding needs to be done after nf_flow_table_free(),
  otherwise entries are not removed from the hardware.

In current master the UNBIND is in nft_unregister_flowtable_ops(),
still called from the commit path. Stable trees with 13210fc63f35
(from the source, not runtime tested): v5.15.180, v6.1.125+,
v6.6.72+, v6.12.10+.

Results
-------

BPI-R3 (MT7986A, mtk_eth_soc PPE), vanilla kernels, each run from a
fresh boot. BND = grep -c BND /sys/kernel/debug/ppe*/entries

  A  4f619d518db9  parent of 13210fc63f35
  B  13210fc63f35
  C  5b0764a53235  master, 2026-10-23
  D  C + the patch mentioned below

  step                                  A     B     C     D
  1. no table                           0     0     0     0
  2. load the table below            1000  1000  1000  1000
  3. nft delete table inet t,
     40 s idle                          0     0     0     0
  4. same 1000 5-tuples again           0  1000  1000     0
  5. 1000 new 5-tuples                  0     0     0     0

A/B differ only in nf_tables_api.c and C/D only in
nf_flow_table_offload.c; each pair uses the same .config. In step 2
all four show 1000 [HW_OFFLOAD] conntrack entries. In step 4 B and C
have no flowtable and no [HW_OFFLOAD] entry, yet the PPE forwards the
old flows in hardware.

On-board ConnectX-7 of an NVIDIA DGX Spark (GB10, arm64; PSID
NVD0000000087, fw 28.45.4028; Ubuntu 6.17.0-1022-nvidia), not a
standalone card. mlx5 in switchdev, flowtable on two VF
representors, traffic VF to VF, 1000 UDP flows,
mlx5e_delete_flower() calls counted with kprobes:

                                   stock   patched
  deletes on "nft delete table"        0      4000
  old flows still forwarded          yes        no
  next 1000 flows offloaded          864      1000
  eswitch mode change to legacy     busy        ok

4000 = 1000 flows x 2 directions x 2 representors. With the flowtable
gone and ip_forward=0, the old 5-tuples still reach the other VF with
the masquerade source address. The 136 missing flows get -EEXIST from
leftover driver entries. "patched" is an out-of-tree nf_flow_table.ko
built from the same distro source; an unmodified rebuild gives the
same numbers as stock.

(On this setup mlx5 only hits FT rules once some tc rule exists on
chain 0, so the forwarding check used a dummy flower rule there. It
looks like esw_chains_create() creates the chain 0 root table, and
its miss rule, before mlx5_chains_set_end_ft() sets the FT chain. I
have not looked into it further.)

Reproduction
------------

  table inet t {
    flowtable ft {
      hook ingress priority filter
      devices = { lan0, lan1 }
      flags offload
    }
    chain ft_forward {
      type filter hook forward priority filter; policy accept;
      ip protocol { tcp, udp } flow add @ft
    }
    chain post {
      type nat hook postrouting priority srcnat; policy accept;
      oifname "lan1" masquerade
    }
  }

1000 UDP flows, 10.0.0.100+i:1024 -> 192.168.200.2:7, 64B, 200 kpps,
lan0 -> lan1 (MT7531 DSA ports). Scripts and raw logs on request.

Fix
---

The tested patch is from an OpenWrt fork, carried there as a MediaTek
target patch although the change is generic. It deletes the device's
hardware flows in nf_flow_table_offload_setup() before
FLOW_BLOCK_UNBIND, as nf_flow_table_indr_cleanup() does, and runs the
GC once more for flows whose offload work was still pending:

  https://github.com/plcharriere/openwrt/pull/1 (commit f36e21492a)

It applies to current master. I have not checked that it keeps the
BIND/UNBIND balance for all the cases 13210fc63f35 fixed.

Paul-Louis (Cc'd), would you be willing to send it to netfilter-devel
on its own? If the maintainers prefer a different fix, I can test it
on both setups.

The same symptom on Qualcomm PPE (IPQ9570, hardware table full after
about 10 firewall reloads):
https://github.com/perceival/openwrt-flint3/issues/115

This was investigated with help from an AI coding assistant (Claude
Opus 5.5 in Claude Code). It read the kernel and driver code, ran the
reproducers and kprobe counts on my hardware, and drafted this
report. I reviewed the analysis and the results and can answer
questions about them.

#regzbot introduced: 13210fc63f35

Thanks,
Jalen

[PATCH nf] netfilter: flowtable: remove hardware flows before unbinding

[Paul-Louis Charrière]

Since commit 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable
binding"), FLOW_BLOCK_UNBIND is called when the flowtable hooks are
unregistered, in the commit phase, and nf_flow_table_free() only tears
the flows down later.  Their FLOW_CLS_DESTROY requests then find no
block callback, so the rules stay in hardware: the flows keep being
forwarded, and drivers such as mtk_eth_soc and mlx5 keep their entries
and reject a later flow with the same cookie with -EEXIST.
Commit 5acab91458ce ("netfilter: nf_tables: unbind callbacks from
flowtable destroy path") had placed the unbind after nf_flow_table_free()
for this reason.

Remove the flows of the device from the hardware in
nf_flow_table_offload_setup() before unbinding it, as
nf_flow_table_indr_cleanup() does for indirect blocks.  The GC run that
nf_flow_table_gc_cleanup() waits for may skip flows with pending offload
work, or have started before the teardown, so queue the GC work once
more, wait for it and flush the delete work again.  A direct
nf_flow_table_gc_run() would race with the GC worker.  No BIND or
UNBIND call is added.

Found on a router with mtk_eth_soc flow offloading, where each firewall
reload left the flows of the old flowtable bound in the PPE.  Jalen
reproduced it on mainline with mtk_eth_soc and mlx5 and tested this
change on both, see the report.  Build-tested with W=1 and sparse on
arm64 and x86_64.  In an arm64 guest with lockdep and KASAN, with TCP
flows through an offload flowtable on netdevsim ports, the delete work
of a deleted flowtable now runs before FLOW_BLOCK_UNBIND returns instead
of after it, and nft_offload.sh passes.

Fixes: 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable binding")
Reported-by: Jalen <jalen.lin@gmail.com>
Closes: https://lore.kernel.org/netfilter-devel/CAHqkChbVoGHrWtaFyGFaLQDiAunWw1qOpW-vKSgkKxWEfQ_tCg@mail.gmail.com/
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Paul-Louis Charrière <plcharriere@gmail.com>
---
 net/netfilter/nf_flow_table_offload.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/net/netfilter/nf_flow_table_offload.c b/net/netfilter/nf_flow_table_offload.c
index 4365859220e..3844a23af5b 100644
--- a/net/netfilter/nf_flow_table_offload.c
+++ b/net/netfilter/nf_flow_table_offload.c
@@ -1279,6 +1279,19 @@ int nf_flow_table_offload_setup(struct nf_flowtable *flowtable,
 	if (!nf_flowtable_hw_offload(flowtable))
 		return nf_flow_offload_xdp_setup(flowtable, dev, cmd);
 
+	/* Hardware rules are deleted through the block callbacks, so delete
+	 * the rules of this device's flows before unbinding it.  Run the GC
+	 * twice: the first run may skip flows with pending offload work, or
+	 * have started before the teardown.
+	 */
+	if (cmd == FLOW_BLOCK_UNBIND) {
+		nf_flow_table_gc_cleanup(flowtable, dev);
+		mod_delayed_work(system_power_efficient_wq,
+				 &flowtable->gc_work, 0);
+		flush_delayed_work(&flowtable->gc_work);
+		flush_workqueue(nf_flow_offload_del_wq);
+	}
+
 	if (dev->netdev_ops->ndo_setup_tc)
 		err = nf_flow_table_offload_cmd(&bo, flowtable, dev, cmd,
 						&extack);

base-commit: 734bef548a96e8774d6eef5dfa2026c6edc4e80b
-- 
2.39.5