Linux Documentation
 help / color / mirror / Atom feed
From: Florian Fainelli <florian.fainelli@broadcom.com>
To: netdev@vger.kernel.org
Cc: "Florian Fainelli" <florian.fainelli@broadcom.com>,
	"Rafał Miłecki" <rafal@milecki.pl>,
	"Broadcom internal kernel review list"
	<bcm-kernel-feedback-list@broadcom.com>,
	"Andrew Lunn" <andrew+netdev@lunn.ch>,
	"David S. Miller" <davem@davemloft.net>,
	"Eric Dumazet" <edumazet@kernel.org>,
	"Jakub Kicinski" <kuba@kernel.org>,
	"Paolo Abeni" <pabeni@redhat.com>,
	"Rob Herring" <robh@kernel.org>,
	"Krzysztof Kozlowski" <krzk+dt@kernel.org>,
	"Conor Dooley" <conor+dt@kernel.org>,
	"Simon Horman" <horms@kernel.org>,
	"Jonathan Corbet" <corbet@lwn.net>,
	"Jonas Gorski" <jonas.gorski@gmail.com>,
	"Vladimir Oltean" <olteanv@gmail.com>,
	"Pablo Neira Ayuso" <pablo@netfilter.org>,
	"Florian Westphal" <fw@strlen.de>, "Phil Sutter" <phil@nwl.cc>,
	devicetree@vger.kernel.org (open list:OPEN FIRMWARE AND
	FLATTENED DEVICE TREE BINDINGS),
	linux-kernel@vger.kernel.org (open list),
	linux-doc@vger.kernel.org (open list:DOCUMENTATION),
	netfilter-devel@vger.kernel.org (open list:NETFILTER),
	coreteam@netfilter.org (open list:NETFILTER)
Subject: [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes
Date: Mon, 16 Nov 2026 14:25:13 -0800	[thread overview]
Message-ID: <20261116222519.1416501-4-florian.fainelli@broadcom.com> (raw)
In-Reply-To: <20261116222519.1416501-1-florian.fainelli@broadcom.com>

Teardown clears IPS_OFFLOAD before asynchronous hardware deletion and
hash removal. A new packet can therefore create a replacement flow on
the same conntrack while the old owner still exists. Repeated teardown
can clear the replacement's IPS_OFFLOAD, and the old hardware delete
worker can clear its IPS_HW_OFFLOAD after a new hardware rule was
installed.

The FA route-change reproducer observed four fresh hardware rows but
only one of two connections still carrying HW_OFFLOAD. Two deterministic
KUnit cases also fail before this change: repeated notification releases
a new claim, and hardware retirement admits a new claim before deletion
finishes.

Release IPS_OFFLOAD after both hash entries have been removed. For
hardware flows, GC reaches this point only after hardware deletion is
complete. NF_FLOW_TEARDOWN immediately excludes the old software fast
path. Packets reaching software still use normal conntrack and
forwarding; TCP state and timeout repair run at first teardown. Hardware
deletion may remain pending. Holding the admission bit prevents
overlapping owners.

The delayed release is observable: conntrack dumps may temporarily
report OFFLOAD or HW_OFFLOAD and omit TCP state after a FIN or RST has
returned to the slow path. The status covers ownership pending cleanup
during the GC and asynchronous delete interval, rather than guaranteeing
acceleration. This trades transient reporting for correct ownership
serialization and prevents an old cleanup from clearing its
replacement's status. Document the interval and use packet
captures/counters to identify the packet path.

Include the private lifetime tests and their KUnit configuration here,
independently of the FA series. They cover real hash removal, deferred
hardware retirement, late RCU notification and balanced conntrack
references. The nf_flow_table.config configuration runs these two cases
on its own.

Fixes: e5eaac2beb54 ("netfilter: flowtable: fix TCP flow teardown")

Signed-off-by: Florian Fainelli <florian.fainelli@broadcom.com>
---
 Documentation/networking/nf_flowtable.rst     | 13 +++
 MAINTAINERS                                   |  1 +
 net/netfilter/Kconfig                         | 10 +++
 net/netfilter/nf_flow_table_core.c            |  9 +-
 net/netfilter/nf_flow_table_core_test.c       | 86 +++++++++++++++++++
 .../kunit/configs/nf_flow_table.config        |  9 ++
 6 files changed, 127 insertions(+), 1 deletion(-)
 create mode 100644 net/netfilter/nf_flow_table_core_test.c
 create mode 100644 tools/testing/kunit/configs/nf_flow_table.config

diff --git a/Documentation/networking/nf_flowtable.rst b/Documentation/networking/nf_flowtable.rst
index d757c21c10f2..b9b444435775 100644
--- a/Documentation/networking/nf_flowtable.rst
+++ b/Documentation/networking/nf_flowtable.rst
@@ -205,6 +205,19 @@ refers to the software offload mode, so there is a distinction between [OFFLOAD]
 which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers
 to the hardware offload datapath being used by the flow.
 
+During teardown, software fastpath lookups immediately stop matching the
+flow. Packets reaching software follow the classic forwarding path, while
+hardware deletion may still be pending. The OFFLOAD status
+bit remains set until hash removal and asynchronous hardware deletion are
+complete, preventing a new owner from racing the old owner's cleanup. As a
+result, conntrack dumps can temporarily retain [OFFLOAD] or [HW_OFFLOAD]
+and omit the TCP state after a FIN or RST has returned to the slow path.
+TCP state and timeout repair still run at first teardown. The transient
+status describes ownership pending cleanup, rather than guaranteeing that
+packets are still accelerated. It clears when garbage collection finishes
+the deletion; packet captures and forwarding counters identify the actual
+forwarding path during that interval.
+
 The flowtable hardware offload infrastructure also supports for the DSA
 (Distributed Switch Architecture).
 
diff --git a/MAINTAINERS b/MAINTAINERS
index c583367c88bc..56212dbc9a0d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19396,6 +19396,7 @@ F:	net/*/netfilter.c
 F:	net/*/netfilter/
 F:	net/bridge/br_netfilter*.c
 F:	net/netfilter/
+F:	tools/testing/kunit/configs/nf_flow_table.config
 F:	tools/testing/selftests/net/netfilter/
 
 NETRONIX EMBEDDED CONTROLLER
diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig
index 09874c26fd13..f565f87b3a6c 100644
--- a/net/netfilter/Kconfig
+++ b/net/netfilter/Kconfig
@@ -718,6 +718,16 @@ config NF_FLOW_TABLE
 
 	  To compile it as a module, choose M here.
 
+config NF_FLOW_TABLE_KUNIT_TEST
+	bool "KUnit tests for flowtable ownership" if !KUNIT_ALL_TESTS
+	depends on KUNIT=y && NF_FLOW_TABLE
+	default KUNIT_ALL_TESTS
+	help
+	  Test conntrack offload ownership during flowtable teardown.
+	  Require hash and asynchronous hardware deletion before re-enrolment.
+	  Verify that late notifications cannot release a replacement owner.
+	  These private tests are embedded in the flowtable core.
+
 config NF_FLOW_TABLE_PROCFS
 	bool "Supply flow table statistics in procfs"
 	depends on NF_FLOW_TABLE
diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c
index 2aa2647ed9df..4c82097cd50b 100644
--- a/net/netfilter/nf_flow_table_core.c
+++ b/net/netfilter/nf_flow_table_core.c
@@ -393,12 +393,15 @@ static void flow_offload_del(struct nf_flowtable *flow_table,
 	rhashtable_remove_fast(&flow_table->rhashtable,
 			       &flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].node,
 			       nf_flow_offload_rhash_params);
+	/* Release admission only after both hashes and async hardware state
+	 * have gone. An older owner must not clear its replacement's status.
+	 */
+	clear_bit_unlock(IPS_OFFLOAD_BIT, &flow->ct->status);
 	flow_offload_free(flow);
 }
 
 void flow_offload_teardown(struct flow_offload *flow)
 {
-	clear_bit(IPS_OFFLOAD_BIT, &flow->ct->status);
 	if (!test_and_set_bit(NF_FLOW_TEARDOWN, &flow->flags))
 		flow_offload_fixup_ct(flow);
 }
@@ -876,3 +879,7 @@ module_exit(nf_flow_table_module_exit);
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Pablo Neira Ayuso <pablo@netfilter.org>");
 MODULE_DESCRIPTION("Netfilter flow table module");
+
+#if IS_ENABLED(CONFIG_NF_FLOW_TABLE_KUNIT_TEST)
+#include "nf_flow_table_core_test.c"
+#endif
diff --git a/net/netfilter/nf_flow_table_core_test.c b/net/netfilter/nf_flow_table_core_test.c
new file mode 100644
index 000000000000..378a1c60b50b
--- /dev/null
+++ b/net/netfilter/nf_flow_table_core_test.c
@@ -0,0 +1,86 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <kunit/test.h>
+
+static void nf_flow_test_teardown_repeat(struct kunit *test)
+{
+	struct flow_offload flow = {};
+	struct nf_conn *ct;
+
+	ct = kunit_kzalloc(test, sizeof(*ct), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, ct);
+	ct->tuplehash[0].tuple.dst.protonum = IPPROTO_ICMP;
+	flow.ct = ct;
+	set_bit(IPS_OFFLOAD_BIT, &ct->status);
+	flow_offload_teardown(&flow);
+	KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
+	clear_bit(IPS_OFFLOAD_BIT, &ct->status);
+	/* A late notification for this retired flow cannot clear a new claim. */
+	set_bit(IPS_OFFLOAD_BIT, &ct->status);
+	flow_offload_teardown(&flow);
+	KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
+}
+
+static void nf_flow_test_hardware_retirement(struct kunit *test)
+{
+	struct nf_flowtable table = {};
+	struct flow_offload *flow;
+	struct nf_conn *ct;
+	unsigned int dir;
+	int err;
+
+	ct = kunit_kzalloc(test, sizeof(*ct), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, ct);
+	err = rhashtable_init(&table.rhashtable, &nf_flow_offload_rhash_params);
+	KUNIT_ASSERT_EQ(test, err, 0);
+	flow = kzalloc_obj(*flow);
+	if (!flow) {
+		rhashtable_destroy(&table.rhashtable);
+		KUNIT_FAIL(test, "flow allocation failed");
+		return;
+	}
+	ct->tuplehash[0].tuple.dst.protonum = IPPROTO_ICMP;
+	refcount_set(&ct->ct_general.use, 2);
+	flow->ct = ct;
+	set_bit(NF_FLOW_HW, &flow->flags);
+	set_bit(IPS_OFFLOAD_BIT, &ct->status);
+	set_bit(IPS_HW_OFFLOAD_BIT, &ct->status);
+	for (dir = 0; dir < FLOW_OFFLOAD_DIR_MAX; dir++) {
+		flow->tuplehash[dir].tuple.src_port = htons(10000 + dir);
+		err = rhashtable_insert_fast(&table.rhashtable,
+					     &flow->tuplehash[dir].node,
+					     nf_flow_offload_rhash_params);
+		KUNIT_EXPECT_EQ(test, err, 0);
+	}
+	flow_offload_teardown(flow);
+	/* nft_flow_offload must not admit another owner during async DESTROY. */
+	KUNIT_EXPECT_TRUE(test, test_and_set_bit(IPS_OFFLOAD_BIT, &ct->status));
+	flow_offload_teardown(flow);
+	KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
+	/* Model completed hardware deletion before the GC removes the hashes. */
+	clear_bit(IPS_HW_OFFLOAD_BIT, &ct->status);
+	set_bit(NF_FLOW_HW_DEAD, &flow->flags);
+	rcu_read_lock();
+	flow_offload_del(&table, flow);
+	KUNIT_EXPECT_FALSE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
+	/* Re-enrolment is safe now; an old RCU reader still cannot undo it. */
+	set_bit(IPS_OFFLOAD_BIT, &ct->status);
+	flow_offload_teardown(flow);
+	KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
+	rcu_read_unlock();
+	rcu_barrier();
+	KUNIT_EXPECT_EQ(test, refcount_read(&ct->ct_general.use), 1);
+	rhashtable_destroy(&table.rhashtable);
+}
+
+static struct kunit_case nf_flow_lifetime_cases[] = {
+	KUNIT_CASE(nf_flow_test_teardown_repeat),
+	KUNIT_CASE(nf_flow_test_hardware_retirement),
+	{}
+};
+
+static struct kunit_suite nf_flow_lifetime_suite = {
+	.name = "nf-flow-lifetime",
+	.test_cases = nf_flow_lifetime_cases,
+};
+
+kunit_test_suite(nf_flow_lifetime_suite);
diff --git a/tools/testing/kunit/configs/nf_flow_table.config b/tools/testing/kunit/configs/nf_flow_table.config
new file mode 100644
index 000000000000..7f2aa55190c2
--- /dev/null
+++ b/tools/testing/kunit/configs/nf_flow_table.config
@@ -0,0 +1,9 @@
+CONFIG_KUNIT=y
+CONFIG_NET=y
+CONFIG_INET=y
+CONFIG_NETFILTER=y
+CONFIG_NETFILTER_INGRESS=y
+CONFIG_NF_CONNTRACK=y
+CONFIG_NF_TABLES=y
+CONFIG_NF_FLOW_TABLE=y
+CONFIG_NF_FLOW_TABLE_KUNIT_TEST=y
-- 
2.53.0


  parent reply	other threads:[~2026-11-16 22:26 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 22:25 [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 1/9] net: ethernet: bgmac: validate RX length against the DMA packet capacity Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 2/9] netfilter: flowtable: reject non-host IPv4 packets before lookup Florian Fainelli
2026-11-16 22:25 ` Florian Fainelli [this message]
2026-11-16 22:25 ` [PATCH net-next 4/9] dt-bindings: net: describe the Northstar flow accelerator Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 5/9] net: dsa: add Broadcom FA metadata and IPv4 exception paths Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 6/9] net: flow_offload: allow drivers to validate and invalidate cached paths Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 7/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 8/9] net: add KUnit coverage for FA and flowtable path validation Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 9/9] docs: net: document Northstar FA setup and NAT qualification Florian Fainelli
2026-11-16 22:38 ` [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload netdev-bot+sinfo
2026-11-16 22:45   ` Florian Fainelli

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116222519.1416501-4-florian.fainelli@broadcom.com \
    --to=florian.fainelli@broadcom.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=bcm-kernel-feedback-list@broadcom.com \
    --cc=conor+dt@kernel.org \
    --cc=corbet@lwn.net \
    --cc=coreteam@netfilter.org \
    --cc=davem@davemloft.net \
    --cc=devicetree@vger.kernel.org \
    --cc=edumazet@kernel.org \
    --cc=fw@strlen.de \
    --cc=horms@kernel.org \
    --cc=jonas.gorski@gmail.com \
    --cc=krzk+dt@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=netfilter-devel@vger.kernel.org \
    --cc=olteanv@gmail.com \
    --cc=pabeni@redhat.com \
    --cc=pablo@netfilter.org \
    --cc=phil@nwl.cc \
    --cc=rafal@milecki.pl \
    --cc=robh@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox