From: Ralf Lici <ralf@mandelbit.com>
To: Xia Yan <yanxia@cmss.chinamobile.com>
Cc: netdev@vger.kernel.org, David Ahern <dsahern@kernel.org>,
Ido Schimmel <idosch@nvidia.com>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Simon Horman <horms@kernel.org>,
Neal Cardwell <ncardwell@google.com>,
Kuniyuki Iwashima <kuniyu@google.com>,
Steffen Klassert <steffen.klassert@secunet.com>,
Herbert Xu <herbert@gondor.apana.org.au>,
Florian Westphal <fw@strlen.de>,
Sabrina Dubroca <sd@queasysnail.net>,
Antonio Quartulli <antonio@mandelbit.com>,
linux-kernel@vger.kernel.org, yana06070@163.com
Subject: Re: [PATCH net-next v3 2/3] ipv6: add ip6_dst_lookup_flow_cookie
Date: Wed, 18 Nov 2026 15:39:08 +0100 [thread overview]
Message-ID: <20261118143909.115729-1-ralf@mandelbit.com> (raw)
In-Reply-To: <1794991714000.1115021.7710558447771708161@cmss.chinamobile.com>
Hi Xia,
On Wed, 18 Nov 2026 16:48:34 +0800, Xia Yan <yanxia@cmss.chinamobile.com> wrote:
> I traced the race and the fix against current net-next.
>
> On the writer side, xfrm_policy_insert() links the policy and then calls
> rt_genid_bump_ipv6(), which resolves to fib6_flush_trees(). A fresh serial
> is allocated from the namespace counter first, then every node is
> restamped under tb6_lock. A late rt6_get_cookie() can pick up that
> post-insertion serial for a dst whose lookup ran before the insertion,
> and the pair keeps passing dst_check until the next flush. Sampling the
> cookie between two generation reads closes the in-lookup window. A
> policy inserted after the helper returns is still caught later, because
> the restamp changes fn_sernum itself and the stored pre-insertion cookie
> stops matching. fib6_new_sernum() never returns 0, so the fallback
> cookie always fails fib6_check().
>
Yes, that's a pretty good summary of the change with the small
correction that a newly linked node holds fn_sernum 0 until fib6_add
stamps it. So a cookie-0 dst can pass a check in that window, and fails
as soon as the stamp lands.
> 1/3 checks out as well. The old code kept writing -1 into the node, so a
> second failure changed nothing and the namespace generation never moved.
> The fresh allocation under tb6_lock matches the discipline the
> fib6_ifup()/fib6_ifdown() paths already get from __fib6_clean_all().
> The conversion boundary in 3/3 also matches the tree: rawv6 does not
> cache the dst, and the accepted-socket and refill stores in tcp_ipv6.c
> are listed as follow-up.
>
In fact there are many users of cached IPv6 dsts that will need
conversion and this series is mainly here to introduce the new API.
> One thing I could not fully work out is the ordering behind the
> generation checks. When the lookup misses a change, the reader relies on
> the two generation reads to notice it. My understanding is that this
> needs the change visible before the counter advance, and the counter
> advance visible before the node restamps. I could not establish either
> hop, though I may well be missing something.
>
> In xfrm_policy_insert(), the policy is linked into the hash and
> rt_genid_bump_ipv6() runs within the same critical section. As far as I
> can tell, the linking stores and the cmpxchg on fib6_sernum are all
> relaxed, with no barrier between them. Could a reader read the advanced
> generation in gen0, miss the policy during the lookup, sample a
> restamped fn_sernum, and still read the same generation in gen1?
>
Both hops are covered by fib6_new_sernum. It allocates the serial with
atomic_try_cmpxchg, which is fully ordered when it succeeds, so it acts
as an smp_mb before and after the update of fib6_sernum.
For the first hop, the policy (or route) is linked before
fib6_new_sernum runs, which gives message passing against the smp_rmb
after gen0: a reader that reads the advanced generation in gen0 also
finds the policy.
> In fib6_flush_trees(), the serial is allocated before __fib6_clean_all()
> takes tb6_lock, and fib6_clean_node() stamps the nodes with a plain
> WRITE_ONCE(). spin_lock() is an ACQUIRE, and my reading of
> memory-barriers.txt is that operations issued before an ACQUIRE may
> complete after it. So a reader could sample the restamped fn_sernum
For the second hop, every fn_sernum write follows a successful
fib6_new_sernum on the same CPU: fib6_update_sernum,
__fib6_update_sernum_upto_root, and fib6_clean_node in the flush case.
The full barrier orders the counter update before the tb6_lock
acquisition and the stamps, so the ACQUIRE semantics of spin_lock don't
matter here. A reader that samples a restamped fn_sernum reads the
advanced generation in gen1 after the smp_rmb, and the helper returns
cookie 0.
> between two reads that both still return the old generation. The only
> pairing I found written down here is fib6_get_cookie_safe()'s smp_rmb()
> against the smp_wmb() in __fib6_update_sernum_upto_root().
>
Right, this isn't documented. The barrier comments in
ip6_dst_lookup_flow_cookie should name fib6_new_sernum as the pairing
side, and fib6_new_sernum should state that its full ordering is relied
upon.
I also checked both hops with an LKMM litmus test: the outcome you
describe is forbidden with atomic_cmpxchg and becomes reachable with
atomic_cmpxchg_relaxed.
While going through the writers I also noticed that xfrm_policy_insert
unlinks a replaced policy only after rt_genid_bump_ipv6, so a lookup
after the bump can still match the old policy. That is a writer-side
ordering issue independent of this series, and it affects the current
late cookie sampling in the same way. I'll send a separate fix.
Does this explanation address your concern? If so, I'll carry your
Reviewed-by. I'm waiting for the Sashiko review of v3: if it reports
anything that needs a respin, I'll add the barrier comments in v4,
otherwise I'll send them as a follow-up.
Thanks for the careful review!
--
Ralf Lici
Mandelbit Srl
next prev parent reply other threads:[~2026-11-18 14:39 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 9:21 [PATCH net-next v3 0/3] ipv6: tie the dst cookie to the lookup that returned the dst Ralf Lici
2026-11-17 9:21 ` [PATCH net-next v3 1/3] ipv6: stamp the default route node with a fresh serial on link failure Ralf Lici
2026-11-19 9:22 ` netdev-bot+sashiko
2026-11-17 9:21 ` [PATCH net-next v3 2/3] ipv6: add ip6_dst_lookup_flow_cookie Ralf Lici
2026-11-18 8:48 ` Xia Yan
2026-11-18 14:39 ` Ralf Lici [this message]
2026-11-18 15:07 ` Xia Yan
2026-11-19 9:22 ` netdev-bot+sashiko
2026-11-19 12:31 ` Ralf Lici
2026-11-20 2:13 ` David Ahern
2026-11-20 8:16 ` Ralf Lici
2026-11-17 9:21 ` [PATCH net-next v3 3/3] ipv6: use lookup cookies in selected socket dst cache paths Ralf Lici
2026-11-19 9:22 ` netdev-bot+sashiko
2026-11-19 12:34 ` Ralf Lici
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118143909.115729-1-ralf@mandelbit.com \
--to=ralf@mandelbit.com \
--cc=antonio@mandelbit.com \
--cc=davem@davemloft.net \
--cc=dsahern@kernel.org \
--cc=edumazet@kernel.org \
--cc=fw@strlen.de \
--cc=herbert@gondor.apana.org.au \
--cc=horms@kernel.org \
--cc=idosch@nvidia.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=ncardwell@google.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sd@queasysnail.net \
--cc=steffen.klassert@secunet.com \
--cc=yana06070@163.com \
--cc=yanxia@cmss.chinamobile.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.