From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.215.228 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794867994; b=1LBWANQTMsaoDG7AguOwLbIB9U1ob7ZOoXvpC93RtX4Gy0y/FagamS1tG+DMnnl6XGXG PqC8geBBmAWQw5md+XtpwLLB/CncV9CKb5oUdJHdV7HitrKOntIcJqGayTKuV21/NRDBM ukbFeEiuXJbjIaXfPR2Ygg4o8Zu3QZggOla2/Vac7F7pzO79SF7bc6SW5jzkNxEq9pQJA m7jzv8/Sorh+Sn4fZvIsUsTT+THX8WGUOETORGCiwdxN3OXK0ZNTh8yzpRbpU1/iymQb0 bO/YyShdpuftEOpJCLON3+s3Jly54HUnPx9LCiy4rGY1fibFbv+BB6lAJWBfwA+qiAg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794867994; h=DMARC-Filter:Received:X-Google-DKIM-Signature:X-Gm-Message-State: X-Gm-Gg:X-Received:Received:X-Relaying-Domain:X-CFilter-Loop:Received: DKIM-Signature:X-Received:X-Received:Received:From:To:Cc:Subject:Date: Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:X-DetectorID-Processed; bh=cVMKW7fB8h0SnQR2hp0PSgoEdY8qiS0OoOpSG4SZZnE=; b=ser5ngBvSOjaG/WEU+SwlYiTytYSwcWjbuYwYhygDVymgm5HQbw07ovouf9UU7kWxUnG GMuzDto+8uFK6YJOyaH4TNipvDeHrhJppC+v31lVMNL3nvXjspZo7/26xk/mL1Q4iaTgl ORvy3QAvsm3miqulKM2EFnYy9S+TlMi6u70mkOtSH4wnGKTiR9t4kzE1Ii17D0sNUMtSZ z9U+a9RR0EAj+Yht3EQbTvpUBy1AVmgI8iBLxO6XWhO1zU2cqFKgHF4QzpnQaHoTIlqDg 66/kjrIiJrk+Jfcz+qNwwlovhAocFeLG6pc5xXqCCG9e8OXnnWzjIhRNzAG3ydk1clA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass header.d=broadcom.com header.i=@broadcom.com header.a=rsa-sha256 header.s=google header.b=GDdgrtPS; arc=none smtp.remote-ip=209.85.215.228 Received: from mail-pg1-f228.google.com (mail-pg1-f228.google.com [209.85.215.228]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79735606A8B0 for ; Mon, 16 Nov 2026 22:26:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 79735606A8B0 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.a=rsa-sha256 header.s=google header.b=GDdgrtPS Received: by mail-pg1-f228.google.com with SMTP id 41be03b00d2f7-cdac6d19ddaso1697609a12.2 for ; Mon, 16 Nov 2026 14:26:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794867993; x=1795472793; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=cVMKW7fB8h0SnQR2hp0PSgoEdY8qiS0OoOpSG4SZZnE=; b=HuTeB3RuhZnjpWosVOj6F7/VWwMDXBhF3qIL3k6Stkd+P4UL+cM0YT1STrRMP+oT04 F24/Astse029/4IfklFpPinLapIGUpmFCtNQyQl90Ayvk4o3ZMfnQ3xZ6f92RdLvoGm0 PIBVcSoS+yHI5TvuntZ4skP/fzQGjjfMGRwgLJvGFS1X+K48mndweGtO+EajYK/d5Jo7 gk45Z9nl3yALdvNIJp9tspgI37h6PFdScZrvm/724dyY/2CFvEguBTODbhE/UH6aYQVU qmzlMRKbDej0h7ckNo2M3cKnRYdtVmgX1/k2i8t0GjT/8WsyZjV6NiT1jTDjmkAy+4sK 5R0w== X-Gm-Message-State: APuU4Z9Rq6VLMoPJjUfvuNYaNBOVcVf91j/86bqvjjhwabSXWP02Z3WK +X1ArEaEg0mYzP15FobOPwMM2Cp26hsRIgHRYk5v0yVANXBlR/7o9vVn2bwW6vwTLIwLb9e6XZe s/11Ktoeu9f+rkTFIDG2WLjbjqxmFroEmZRGAn+96gEnTV5rxROqAo+LPEAJDOQvTb80k0gRomC ja/WHmtlKm7WHivMhOxrvfy7T+xf5JBWu+1j7225YlhlM0IeUxlSJOa95rc+KhgP9vFu0AcT2c6 IEfhKY42HAEq23F X-Gm-Gg: AZFdTZchNr10+jSb0hGzyif89WqotTBEbG28EOe5HaEpSD0QPHh+MxkCvYyUZWXR0pl 5RibyZ6JESXIWf8BdS64N+zHjnOpWFWM7M400ZcY0aBn74ncGFm8k24gPt/NSJawEL4TjPj62ZA rkJQ+rZxLPxbthfUI77o42+EvW3E/QFP2XYd7vZwobqNuvvW+k8NG6+df2Pj2utWi8IIkRR5trh jaP5RJWZNK4XcqfJSrbSjcvaJpb359vyqdAFa1g+Vtx17u6Q10fCg2RGMRGEfsN8+HXgI7alqGN lyCJsnagWlw4JkApa5wtchRrCeSVVuB7lRWSJ1McrNrywlOEk7xNwMM/uKsyW3yb7qZwYo7L1zC oG5+NOGauC5NktFUUAUUubR8dZgVc0Us5NUCphPd4bZVjbY9DuLKqjHHV2LKM7qCjwVz3eUlVhf RoRr1/S2DpJ97a7Je33NoYE6PYdovx9YMcyWUPkVnYUSM= X-Received: by 2002:a17:90b:554d:b0:3ac:9a5e:9ce7 with SMTP id 98e67ed59e1d1-3b1b0a4c61bmr13102462a91.49.1794867992706; Mon, 16 Nov 2026 14:26:32 -0800 (PST) Received: from smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com (address-144-49-247-123.dlp.protect.broadcom.com. [144.49.247.123]) by smtp-relay.gmail.com with ESMTPS id 98e67ed59e1d1-3b3abb84fe8sm474262a91.2.2026.11.16.14.26.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 16 Nov 2026 14:26:32 -0800 (PST) X-Relaying-Domain: broadcom.com X-CFilter-Loop: Reflected Received: by mail-dy1-f198.google.com with SMTP id 5a478bee46e88-36ecd5db581so4942985eec.0 for ; Mon, 16 Nov 2026 14:26:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1794867990; x=1795472790; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=cVMKW7fB8h0SnQR2hp0PSgoEdY8qiS0OoOpSG4SZZnE=; b=GDdgrtPSd1A//IcT5eYJN7ELhJmZHj6CaZY8PbDuGhKTQ/Ny1TPsXW3dBR1RqTY/q3 SppgckwO4LBEoMHB0cJEDUXFPEI9d91o1JTXPSogva4ccX8/g5wDzIecehf05QtV3P6F X0cVF3ubcRC0or/0ACDYXFYOHMrwDWmo57mTA= X-Received: by 2002:a05:7300:7952:b0:381:3b6:6c3d with SMTP id 5a478bee46e88-38103b682f6mr849642eec.35.1794867990282; Mon, 16 Nov 2026 14:26:30 -0800 (PST) X-Received: by 2002:a05:7300:7952:b0:381:3b6:6c3d with SMTP id 5a478bee46e88-38103b682f6mr849591eec.35.1794867989331; Mon, 16 Nov 2026 14:26:29 -0800 (PST) Received: from fainelli-desktop.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-379560f5e06sm50176624eec.19.2026.11.16.14.26.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Nov 2026 14:26:28 -0800 (PST) From: Florian Fainelli To: netdev@vger.kernel.org Cc: Florian Fainelli , =?UTF-8?q?Rafa=C5=82=20Mi=C5=82ecki?= , Broadcom internal kernel review list , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Simon Horman , Jonathan Corbet , Jonas Gorski , Vladimir Oltean , Pablo Neira Ayuso , Florian Westphal , Phil Sutter , devicetree@vger.kernel.org (open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS), linux-kernel@vger.kernel.org (open list), linux-doc@vger.kernel.org (open list:DOCUMENTATION), netfilter-devel@vger.kernel.org (open list:NETFILTER), coreteam@netfilter.org (open list:NETFILTER) Subject: [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes Date: Mon, 16 Nov 2026 14:25:13 -0800 Message-ID: <20261116222519.1416501-4-florian.fainelli@broadcom.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20261116222519.1416501-1-florian.fainelli@broadcom.com> References: <20261116222519.1416501-1-florian.fainelli@broadcom.com> X-Mailing-List: netdev@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-DetectorID-Processed: b00c1d49-9d2e-4205-b15f-d015386d3d5e Teardown clears IPS_OFFLOAD before asynchronous hardware deletion and hash removal. A new packet can therefore create a replacement flow on the same conntrack while the old owner still exists. Repeated teardown can clear the replacement's IPS_OFFLOAD, and the old hardware delete worker can clear its IPS_HW_OFFLOAD after a new hardware rule was installed. The FA route-change reproducer observed four fresh hardware rows but only one of two connections still carrying HW_OFFLOAD. Two deterministic KUnit cases also fail before this change: repeated notification releases a new claim, and hardware retirement admits a new claim before deletion finishes. Release IPS_OFFLOAD after both hash entries have been removed. For hardware flows, GC reaches this point only after hardware deletion is complete. NF_FLOW_TEARDOWN immediately excludes the old software fast path. Packets reaching software still use normal conntrack and forwarding; TCP state and timeout repair run at first teardown. Hardware deletion may remain pending. Holding the admission bit prevents overlapping owners. The delayed release is observable: conntrack dumps may temporarily report OFFLOAD or HW_OFFLOAD and omit TCP state after a FIN or RST has returned to the slow path. The status covers ownership pending cleanup during the GC and asynchronous delete interval, rather than guaranteeing acceleration. This trades transient reporting for correct ownership serialization and prevents an old cleanup from clearing its replacement's status. Document the interval and use packet captures/counters to identify the packet path. Include the private lifetime tests and their KUnit configuration here, independently of the FA series. They cover real hash removal, deferred hardware retirement, late RCU notification and balanced conntrack references. The nf_flow_table.config configuration runs these two cases on its own. Fixes: e5eaac2beb54 ("netfilter: flowtable: fix TCP flow teardown") Signed-off-by: Florian Fainelli --- Documentation/networking/nf_flowtable.rst | 13 +++ MAINTAINERS | 1 + net/netfilter/Kconfig | 10 +++ net/netfilter/nf_flow_table_core.c | 9 +- net/netfilter/nf_flow_table_core_test.c | 86 +++++++++++++++++++ .../kunit/configs/nf_flow_table.config | 9 ++ 6 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 net/netfilter/nf_flow_table_core_test.c create mode 100644 tools/testing/kunit/configs/nf_flow_table.config diff --git a/Documentation/networking/nf_flowtable.rst b/Documentation/networking/nf_flowtable.rst index d757c21c10f2..b9b444435775 100644 --- a/Documentation/networking/nf_flowtable.rst +++ b/Documentation/networking/nf_flowtable.rst @@ -205,6 +205,19 @@ refers to the software offload mode, so there is a distinction between [OFFLOAD] which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers to the hardware offload datapath being used by the flow. +During teardown, software fastpath lookups immediately stop matching the +flow. Packets reaching software follow the classic forwarding path, while +hardware deletion may still be pending. The OFFLOAD status +bit remains set until hash removal and asynchronous hardware deletion are +complete, preventing a new owner from racing the old owner's cleanup. As a +result, conntrack dumps can temporarily retain [OFFLOAD] or [HW_OFFLOAD] +and omit the TCP state after a FIN or RST has returned to the slow path. +TCP state and timeout repair still run at first teardown. The transient +status describes ownership pending cleanup, rather than guaranteeing that +packets are still accelerated. It clears when garbage collection finishes +the deletion; packet captures and forwarding counters identify the actual +forwarding path during that interval. + The flowtable hardware offload infrastructure also supports for the DSA (Distributed Switch Architecture). diff --git a/MAINTAINERS b/MAINTAINERS index c583367c88bc..56212dbc9a0d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -19396,6 +19396,7 @@ F: net/*/netfilter.c F: net/*/netfilter/ F: net/bridge/br_netfilter*.c F: net/netfilter/ +F: tools/testing/kunit/configs/nf_flow_table.config F: tools/testing/selftests/net/netfilter/ NETRONIX EMBEDDED CONTROLLER diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig index 09874c26fd13..f565f87b3a6c 100644 --- a/net/netfilter/Kconfig +++ b/net/netfilter/Kconfig @@ -718,6 +718,16 @@ config NF_FLOW_TABLE To compile it as a module, choose M here. +config NF_FLOW_TABLE_KUNIT_TEST + bool "KUnit tests for flowtable ownership" if !KUNIT_ALL_TESTS + depends on KUNIT=y && NF_FLOW_TABLE + default KUNIT_ALL_TESTS + help + Test conntrack offload ownership during flowtable teardown. + Require hash and asynchronous hardware deletion before re-enrolment. + Verify that late notifications cannot release a replacement owner. + These private tests are embedded in the flowtable core. + config NF_FLOW_TABLE_PROCFS bool "Supply flow table statistics in procfs" depends on NF_FLOW_TABLE diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c index 2aa2647ed9df..4c82097cd50b 100644 --- a/net/netfilter/nf_flow_table_core.c +++ b/net/netfilter/nf_flow_table_core.c @@ -393,12 +393,15 @@ static void flow_offload_del(struct nf_flowtable *flow_table, rhashtable_remove_fast(&flow_table->rhashtable, &flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].node, nf_flow_offload_rhash_params); + /* Release admission only after both hashes and async hardware state + * have gone. An older owner must not clear its replacement's status. + */ + clear_bit_unlock(IPS_OFFLOAD_BIT, &flow->ct->status); flow_offload_free(flow); } void flow_offload_teardown(struct flow_offload *flow) { - clear_bit(IPS_OFFLOAD_BIT, &flow->ct->status); if (!test_and_set_bit(NF_FLOW_TEARDOWN, &flow->flags)) flow_offload_fixup_ct(flow); } @@ -876,3 +879,7 @@ module_exit(nf_flow_table_module_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Pablo Neira Ayuso "); MODULE_DESCRIPTION("Netfilter flow table module"); + +#if IS_ENABLED(CONFIG_NF_FLOW_TABLE_KUNIT_TEST) +#include "nf_flow_table_core_test.c" +#endif diff --git a/net/netfilter/nf_flow_table_core_test.c b/net/netfilter/nf_flow_table_core_test.c new file mode 100644 index 000000000000..378a1c60b50b --- /dev/null +++ b/net/netfilter/nf_flow_table_core_test.c @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include + +static void nf_flow_test_teardown_repeat(struct kunit *test) +{ + struct flow_offload flow = {}; + struct nf_conn *ct; + + ct = kunit_kzalloc(test, sizeof(*ct), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, ct); + ct->tuplehash[0].tuple.dst.protonum = IPPROTO_ICMP; + flow.ct = ct; + set_bit(IPS_OFFLOAD_BIT, &ct->status); + flow_offload_teardown(&flow); + KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status)); + clear_bit(IPS_OFFLOAD_BIT, &ct->status); + /* A late notification for this retired flow cannot clear a new claim. */ + set_bit(IPS_OFFLOAD_BIT, &ct->status); + flow_offload_teardown(&flow); + KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status)); +} + +static void nf_flow_test_hardware_retirement(struct kunit *test) +{ + struct nf_flowtable table = {}; + struct flow_offload *flow; + struct nf_conn *ct; + unsigned int dir; + int err; + + ct = kunit_kzalloc(test, sizeof(*ct), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, ct); + err = rhashtable_init(&table.rhashtable, &nf_flow_offload_rhash_params); + KUNIT_ASSERT_EQ(test, err, 0); + flow = kzalloc_obj(*flow); + if (!flow) { + rhashtable_destroy(&table.rhashtable); + KUNIT_FAIL(test, "flow allocation failed"); + return; + } + ct->tuplehash[0].tuple.dst.protonum = IPPROTO_ICMP; + refcount_set(&ct->ct_general.use, 2); + flow->ct = ct; + set_bit(NF_FLOW_HW, &flow->flags); + set_bit(IPS_OFFLOAD_BIT, &ct->status); + set_bit(IPS_HW_OFFLOAD_BIT, &ct->status); + for (dir = 0; dir < FLOW_OFFLOAD_DIR_MAX; dir++) { + flow->tuplehash[dir].tuple.src_port = htons(10000 + dir); + err = rhashtable_insert_fast(&table.rhashtable, + &flow->tuplehash[dir].node, + nf_flow_offload_rhash_params); + KUNIT_EXPECT_EQ(test, err, 0); + } + flow_offload_teardown(flow); + /* nft_flow_offload must not admit another owner during async DESTROY. */ + KUNIT_EXPECT_TRUE(test, test_and_set_bit(IPS_OFFLOAD_BIT, &ct->status)); + flow_offload_teardown(flow); + KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status)); + /* Model completed hardware deletion before the GC removes the hashes. */ + clear_bit(IPS_HW_OFFLOAD_BIT, &ct->status); + set_bit(NF_FLOW_HW_DEAD, &flow->flags); + rcu_read_lock(); + flow_offload_del(&table, flow); + KUNIT_EXPECT_FALSE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status)); + /* Re-enrolment is safe now; an old RCU reader still cannot undo it. */ + set_bit(IPS_OFFLOAD_BIT, &ct->status); + flow_offload_teardown(flow); + KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status)); + rcu_read_unlock(); + rcu_barrier(); + KUNIT_EXPECT_EQ(test, refcount_read(&ct->ct_general.use), 1); + rhashtable_destroy(&table.rhashtable); +} + +static struct kunit_case nf_flow_lifetime_cases[] = { + KUNIT_CASE(nf_flow_test_teardown_repeat), + KUNIT_CASE(nf_flow_test_hardware_retirement), + {} +}; + +static struct kunit_suite nf_flow_lifetime_suite = { + .name = "nf-flow-lifetime", + .test_cases = nf_flow_lifetime_cases, +}; + +kunit_test_suite(nf_flow_lifetime_suite); diff --git a/tools/testing/kunit/configs/nf_flow_table.config b/tools/testing/kunit/configs/nf_flow_table.config new file mode 100644 index 000000000000..7f2aa55190c2 --- /dev/null +++ b/tools/testing/kunit/configs/nf_flow_table.config @@ -0,0 +1,9 @@ +CONFIG_KUNIT=y +CONFIG_NET=y +CONFIG_INET=y +CONFIG_NETFILTER=y +CONFIG_NETFILTER_INGRESS=y +CONFIG_NF_CONNTRACK=y +CONFIG_NF_TABLES=y +CONFIG_NF_FLOW_TABLE=y +CONFIG_NF_FLOW_TABLE_KUNIT_TEST=y -- 2.53.0