Netdev List
 help / color / mirror / Atom feed
From: netdev-bot+sashiko@kernel.org
To: florian.fainelli@broadcom.com
Cc: netdev@vger.kernel.org, rafal@milecki.pl,
	bcm-kernel-feedback-list@broadcom.com, andrew+netdev@lunn.ch,
	davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
	pabeni@redhat.com, robh@kernel.org, krzk+dt@kernel.org,
	conor+dt@kernel.org, horms@kernel.org, corbet@lwn.net,
	jonas.gorski@gmail.com, olteanv@gmail.com, pablo@netfilter.org,
	fw@strlen.de, phil@nwl.cc, devicetree@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
	netfilter-devel@vger.kernel.org, coreteam@netfilter.org
Subject: Re: [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes
Date: Tue, 17 Nov 2026 23:29:19 +0000	[thread overview]
Message-ID: <179495815956.3136.7417407587662864290@kernel.org> (raw)
In-Reply-To: <20261116222519.1416501-4-florian.fainelli@broadcom.com>

Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 4 · Low: 2

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] Moving IPS_OFFLOAD_BIT out of flow_offload_teardown() means
  the lockless check in nf_flow_table_extend_ct_timeout() can no longer…
- [Medium] This patch reverses the central change of the commit in its
  Fixes: tag but does not say so, and it does not mention the later…
- [Medium] The patch's ownership argument ('For hardware flows, GC
  reaches this point only after hardware deletion is complete ...…
- [Medium] The patch now releases IPS_OFFLOAD (re-admission) in
  flow_offload_del() once NF_FLOW_HW_DEAD is set, and the commit…
- [Low] The new KUnit test nf_flow_test_hardware_retirement() and the
  NF_FLOW_TABLE_KUNIT_TEST help text say they check that hash removal…
- [Low] The new Documentation/networking/nf_flowtable.rst text (and the
  commit message) list the visible side effects of keeping IPS_OFFLOAD…

--

netfilter: flowtable: retain offload ownership until deletion completes

This patch moves the clearing of IPS_OFFLOAD_BIT from
flow_offload_teardown() to flow_offload_del(), after both hash entries have
been removed. The goal is that a replacement flow can't be admitted on the
same conntrack while the old owner's cleanup is still pending. It also
documents the transient status and adds KUnit tests for the ownership
lifetime.

> diff --git a/Documentation/networking/nf_flowtable.rst b/Documentation/networking/nf_flowtable.rst
> index d757c21c10f2d..b9b4444357758 100644
> --- a/Documentation/networking/nf_flowtable.rst
> +++ b/Documentation/networking/nf_flowtable.rst
> @@ -205,6 +205,19 @@ refers to the software offload mode, so there is a distinction between [OFFLOAD]
>  which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers
>  to the hardware offload datapath being used by the flow.
>  
> +During teardown, software fastpath lookups immediately stop matching the
> +flow. Packets reaching software follow the classic forwarding path, while
> +hardware deletion may still be pending. The OFFLOAD status
> +bit remains set until hash removal and asynchronous hardware deletion are
> +complete, preventing a new owner from racing the old owner's cleanup. As a
> +result, conntrack dumps can temporarily retain [OFFLOAD] or [HW_OFFLOAD]
> +and omit the TCP state after a FIN or RST has returned to the slow path.

[Severity: Low]
Is this list of side effects complete? Several in-tree users of
IPS_OFFLOAD_BIT now see the bit set for the whole GC and async DESTROY
window.

ctnetlink_dump_info() skips the timeout as well as the protoinfo:

net/netfilter/nf_conntrack_netlink.c:ctnetlink_dump_info() {
    ...
	if (!test_bit(IPS_OFFLOAD_BIT, &ct->status) &&
	    (ctnetlink_dump_timeout(skb, ct, false) < 0 ||
	     ctnetlink_dump_protoinfo(skb, ct, false) < 0))
    ...
}

ct_seq_show() hides the expires field in the same way. As a result, the
repaired timeout mentioned in the next paragraph can't be seen during this
interval.

nf_nat_used_tuple_harder() also reads the bit as "entry is offloaded":

net/netfilter/nf_nat_core.c:nf_nat_used_tuple_harder() {
    ...
	static const unsigned long flags_offload = IPS_OFFLOAD | IPS_HW_OFFLOAD;
    ...
	if (nf_ct_kill(ct))
		taken = flags & flags_offload;
    ...
}

A torn-down TCP entry in CLOSE or TIME_WAIT that has gone back to the slow
path would now be evicted but still reported as taken.

The UAPI comment for IPS_OFFLOAD_BIT in nf_conntrack_common.h still says
"Conntrack has been offloaded to flow table", but this text redefines the
bit as ownership pending cleanup.

Should these effects be documented here too?

[ ... ]

> diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c
> index 2aa2647ed9df0..4c82097cd50bc 100644
> --- a/net/netfilter/nf_flow_table_core.c
> +++ b/net/netfilter/nf_flow_table_core.c
> @@ -393,12 +393,15 @@ static void flow_offload_del(struct nf_flowtable *flow_table,
>  	rhashtable_remove_fast(&flow_table->rhashtable,
>  			       &flow->tuplehash[FLOW_OFFLOAD_DIR_REPLY].node,
>  			       nf_flow_offload_rhash_params);
> +	/* Release admission only after both hashes and async hardware state
> +	 * have gone. An older owner must not clear its replacement's status.
> +	 */
> +	clear_bit_unlock(IPS_OFFLOAD_BIT, &flow->ct->status);

[Severity: Medium]
The commit message says "For hardware flows, GC reaches this point only
after hardware deletion is complete" and "Holding the admission bit
prevents overlapping owners".

Can a datapath reader still queue a FLOW_CLS_REPLACE for this flow after
its DESTROY has completed?

flow_offload_lookup() checks NF_FLOW_TEARDOWN once. The later
flow_offload_refresh() only looks at NF_FLOW_CLOSING and NF_FLOW_HW:

net/netfilter/nf_flow_table_core.c:flow_offload_refresh() {
    ...
	if (likely(!nf_flowtable_hw_offload(flow_table)) ||
	    test_bit(NF_FLOW_CLOSING, &flow->flags))
		return;

	if (test_bit(NF_FLOW_HW, &flow->flags))
		nf_flow_offload_refresh(flow_table, flow);
}

nf_flow_offload_work_alloc() is gated only by NF_FLOW_PENDING.
flow_offload_work_handler() clears that bit after flow_offload_work_del()
has set NF_FLOW_HW_DEAD. One possible sequence:

reader                      gc / offload work
flow_offload_lookup()
  TEARDOWN not set
                            flow_offload_teardown()
                            nf_flow_offload_del()
                            flow_offload_work_del()
                              set_bit(NF_FLOW_HW_DEAD)
                            NF_FLOW_PENDING cleared
flow_offload_refresh()
  nf_flow_offload_refresh()
    REPLACE queued
                            flow_offload_work_add()
                              set_bit(IPS_HW_OFFLOAD_BIT)
                            flow_offload_del()
                              clear_bit_unlock(IPS_OFFLOAD_BIT)
                              flow_offload_free()

After this, the re-installed hardware rule is keyed by a freed cookie, and
IPS_HW_OFFLOAD stays set on the ct. nft_flow_offload_eval() or act_ct can
then admit a replacement owner through test_and_set_bit(IPS_OFFLOAD_BIT).

With PREEMPT_RCU or PREEMPT_RT, the reader's RCU section could span
flow_offload_del(). Could the queued REPLACE work then use the flow after
flow_offload_free_rcu() has freed it?

The lookup/refresh window existed before this patch, but the new ownership
argument depends on it being closed. Later patches in the series don't
touch flow_offload_refresh() or nf_flow_offload_refresh().

[Severity: Medium]
Does NF_FLOW_HW_DEAD really mean the device removed the rule?
flow_offload_work_del() sets it unconditionally:

net/netfilter/nf_flow_table_offload.c:flow_offload_work_del() {
	clear_bit(IPS_HW_OFFLOAD_BIT, &offload->flow->ct->status);
	flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_ORIGINAL);
	if (test_bit(NF_FLOW_HW_BIDIRECTIONAL, &offload->flow->flags))
		flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_REPLY);
	set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags);
}

flow_offload_tuple_del() throws away the return value of
nf_flow_offload_tuple(..., FLOW_CLS_DESTROY, ...), and
nf_flow_offload_tuple() skips callbacks that return errors.

A driver's DESTROY can fail and leave the rule programmed, for example
bnxt_tc_del_flow() after a firmware HWRM failure. In that case
flow_offload_del() still releases IPS_OFFLOAD and frees the flow, and the
flow's tuplehash[dir].tuple is the driver cookie.

Could a replacement owner then program a second rule for the same tuple
while the stale rule, with the old route or NAT, stays in hardware with no
way to delete it?

The ignored DESTROY result predates this patch. However, the ownership
serialization described in the commit message depends on deletion having
completed.

>  	flow_offload_free(flow);
>  }
>  
>  void flow_offload_teardown(struct flow_offload *flow)
>  {
> -	clear_bit(IPS_OFFLOAD_BIT, &flow->ct->status);
>  	if (!test_and_set_bit(NF_FLOW_TEARDOWN, &flow->flags))
>  		flow_offload_fixup_ct(flow);
>  }

[Severity: Medium]
Without this clear_bit(), can the flowtable GC overwrite the timeout that
flow_offload_fixup_ct() just repaired?

nf_flow_offload_gc_step() samples NF_FLOW_TEARDOWN once and doesn't check
it again before calling nf_flow_table_extend_ct_timeout(). The only guard
left in the extend helper is the IPS_OFFLOAD_BIT re-test:

net/netfilter/nf_flow_table_core.c:nf_flow_table_extend_ct_timeout() {
	u32 ct_timeout = READ_ONCE(ct->timeout);
    ...
	if (!refcount_inc_not_zero(&ct->ct_general.use))
		return;

	/* load ct->status after refcount increase */
	smp_acquire__after_ctrl_dep();

	if (nf_ct_is_confirmed(ct) &&
	    test_bit(IPS_OFFLOAD_BIT, &ct->status)) {
    ...
			cmpxchg(&ct->timeout, ct_timeout, new_timeout);
    ...
}

That re-test now always passes until flow_offload_del() runs. Consider:

CPU0                                CPU1
nf_flow_offload_gc_step()
  teardown = false
                                    flow_offload_teardown()
                                      set NF_FLOW_TEARDOWN
                                      flow_offload_fixup_ct()
                                        nf_ct_refresh()
nf_flow_table_extend_ct_timeout()
  READ_ONCE(ct->timeout)
    /* repaired, below 5 min */
  test_bit(IPS_OFFLOAD_BIT)
    /* still set */
  cmpxchg(&ct->timeout, ct_timeout,
          now + NF_CT_DAY)

CPU1 can be the datapath after a failed dst check or a missing outdev, act_ct
on FIN/RST, or nf_flow_table_cleanup() on netdev events.

The refcount, smp_acquire__after_ctrl_dep() and re-test sequence from
commit 03428ca5cee9 depends on teardown clearing the bit before
flow_offload_fixup_ct() repairs ct->timeout.

flow_offload_del() doesn't repair the timeout again. A UDP flow with no
further packets would therefore keep its conntrack entry for NF_CT_DAY
instead of seconds or minutes.

Doesn't this contradict "TCP state and timeout repair run at first
teardown" in the commit message? Later patches in the series leave
flow_offload_teardown(), nf_flow_offload_gc_step() and
nf_flow_table_extend_ct_timeout() unchanged.

[Severity: Medium]
This undoes the main change of the commit named in the Fixes: tag:

    Fixes: e5eaac2beb54 ("netfilter: flowtable: fix TCP flow teardown")

e5eaac2beb54 moved this clear_bit() from flow_offload_del() into
flow_offload_teardown(). The reason was that conntrack gc could race to
undo the packet path's timeout adjustment, leaving the entry with the
one-day offload timeout.

Moving it back looks safe on mainline only because commit 03428ca5cee9
("netfilter: conntrack: rework offload nf_conn timeout extension logic")
removed this block from gc_worker():

	if (test_bit(IPS_OFFLOAD_BIT, &tmp->status)) {
		nf_ct_offload_timeout(tmp);
		...
		continue;
	}

03428ca5cee9 has no Fixes: or stable tag. Suppose this patch is backported
to a stable tree that has e5eaac2beb54 but not 03428ca5cee9.

Wouldn't gc_worker() then reset ct->timeout to now + NF_CT_DAY during the
whole window from teardown to flow_offload_del()? flow_offload_del() would
then clear the bit without another repair. For closed TCP and expired UDP
entries, that brings back the bug e5eaac2beb54 fixed.

Could the commit message say that it reverts part of e5eaac2beb54 and
depends on 03428ca5cee9?

[ ... ]

> diff --git a/net/netfilter/nf_flow_table_core_test.c b/net/netfilter/nf_flow_table_core_test.c
> new file mode 100644
> index 0000000000000..378a1c60b50b9
> --- /dev/null
> +++ b/net/netfilter/nf_flow_table_core_test.c
> @@ -0,0 +1,86 @@

[ ... ]

> +	flow_offload_teardown(flow);
> +	/* nft_flow_offload must not admit another owner during async DESTROY. */
> +	KUNIT_EXPECT_TRUE(test, test_and_set_bit(IPS_OFFLOAD_BIT, &ct->status));
> +	flow_offload_teardown(flow);
> +	KUNIT_EXPECT_TRUE(test, test_bit(IPS_OFFLOAD_BIT, &ct->status));
> +	/* Model completed hardware deletion before the GC removes the hashes. */
> +	clear_bit(IPS_HW_OFFLOAD_BIT, &ct->status);
> +	set_bit(NF_FLOW_HW_DEAD, &flow->flags);
> +	rcu_read_lock();
> +	flow_offload_del(&table, flow);

[Severity: Low]
Does this test cover the ordering it describes? The Kconfig help says
"Require hash and asynchronous hardware deletion before re-enrolment".
However, the only code that makes deletion wait for hardware removal is in
nf_flow_offload_gc_step():

	if (test_bit(NF_FLOW_HW, &flow->flags)) {
		if (!test_bit(NF_FLOW_HW_DYING, &flow->flags))
			nf_flow_offload_del(flow_table, flow);
		else if (test_bit(NF_FLOW_HW_DEAD, &flow->flags))
			flow_offload_del(flow_table, flow);

This test sets NF_FLOW_HW_DEAD by hand and calls flow_offload_del()
directly, and flow_offload_del() reads neither flag. If the
HW_DYING/HW_DEAD gating were removed or broken, neither case would fail.

Both cases also use IPPROTO_ICMP, so flow_offload_fixup_ct() returns
immediately. The TCP state and timeout repair at first teardown is never
exercised. nf_flow_test_teardown_repeat() only checks that
flow_offload_teardown() no longer touches IPS_OFFLOAD_BIT.

Later in the series the Kconfig help is reworded to "Test flowtable
ownership during asynchronous hardware deletion", so the gap seems to
remain in the final state.

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116222519.1416501-1-florian.fainelli%40broadcom.com

  reply	other threads:[~2026-11-17 23:29 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 22:25 [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 1/9] net: ethernet: bgmac: validate RX length against the DMA packet capacity Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-17 23:43     ` Florian Fainelli
2026-11-16 22:25 ` [PATCH net-next 2/9] netfilter: flowtable: reject non-host IPv4 packets before lookup Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 3/9] netfilter: flowtable: retain offload ownership until deletion completes Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko [this message]
2026-11-16 22:25 ` [PATCH net-next 4/9] dt-bindings: net: describe the Northstar flow accelerator Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 5/9] net: dsa: add Broadcom FA metadata and IPv4 exception paths Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 6/9] net: flow_offload: allow drivers to validate and invalidate cached paths Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 7/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 8/9] net: add KUnit coverage for FA and flowtable path validation Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:25 ` [PATCH net-next 9/9] docs: net: document Northstar FA setup and NAT qualification Florian Fainelli
2026-11-17 23:29   ` netdev-bot+sashiko
2026-11-16 22:38 ` [PATCH net-next 0/9] net: bgmac: add Northstar IPv4 TCP/UDP NAT flowtable offload netdev-bot+sinfo
2026-11-16 22:45   ` Florian Fainelli

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=179495815956.3136.7417407587662864290@kernel.org \
    --to=netdev-bot+sashiko@kernel.org \
    --cc=andrew+netdev@lunn.ch \
    --cc=bcm-kernel-feedback-list@broadcom.com \
    --cc=conor+dt@kernel.org \
    --cc=corbet@lwn.net \
    --cc=coreteam@netfilter.org \
    --cc=davem@davemloft.net \
    --cc=devicetree@vger.kernel.org \
    --cc=edumazet@kernel.org \
    --cc=florian.fainelli@broadcom.com \
    --cc=fw@strlen.de \
    --cc=horms@kernel.org \
    --cc=jonas.gorski@gmail.com \
    --cc=krzk+dt@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=netfilter-devel@vger.kernel.org \
    --cc=olteanv@gmail.com \
    --cc=pabeni@redhat.com \
    --cc=pablo@netfilter.org \
    --cc=phil@nwl.cc \
    --cc=rafal@milecki.pl \
    --cc=robh@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox