[PATCH v2 net-next 0/2] ipv6: fix hangup on device removal

[PATCH v2 net-next 0/2] ipv6: fix hangup on device removal

[Paolo Abeni]

This addresses the infamous unregister_netdevice splat in net selftests;
the actual fix is carried by the first patch, while the 2nd one
addresses a related problem in the relevant test that was patially
hiding the problem.

Targeting net-next as the issue is quite old and I feel a little lost
in the fib info/nh jungle.
---
v1 -> v2:
 - drop unintended whitespace change in patch 1/2

Paolo Abeni (2):
  ipv6: release nexthop on device removal
  selftests: net: really check for bg process completion

 net/ipv6/route.c                    | 6 +++---
 tools/testing/selftests/net/pmtu.sh | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

-- 
2.45.2

[PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

[Paolo Abeni]

The CI is hitting some aperiodic hangup at device removal time in the
pmtu.sh self-test:

unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
	dst_init+0x84/0x4a0
	dst_alloc+0x97/0x150
	ip6_dst_alloc+0x23/0x90
	ip6_rt_pcpu_alloc+0x1e6/0x520
	ip6_pol_route+0x56f/0x840
	fib6_rule_lookup+0x334/0x630
	ip6_route_output_flags+0x259/0x480
	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
	ip6_dst_lookup_flow+0x88/0x190
	udp_tunnel6_dst_lookup+0x2a7/0x4c0
	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
	vxlan_xmit+0x9ad/0xf20 [vxlan]
	dev_hard_start_xmit+0x10e/0x360
	__dev_queue_xmit+0xf95/0x18c0
	arp_solicit+0x4a2/0xe00
	neigh_probe+0xaa/0xf0

While the first suspect is the dst_cache, explicitly tracking the dst
owing the last device reference via probes proved such dst is held by
the nexthop in the originating fib6_info.

Similar to commit f5b51fe804ec ("ipv6: route: purge exception on
removal"), we need to explicitly release the originating fib info when
disconnecting a to-be-removed device from a live ipv6 dst: move the
fib6_info cleanup into ip6_dst_ifdown().

Tested running:

./pmtu.sh cleanup_ipv6_exception

in a tight loop for more than 400 iterations with no spat, running an
unpatched kernel  I observed a splat every ~10 iterations.

Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
---
v1 -> v2:
 - dropped unintended whitespace change
---
 net/ipv6/route.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index d7ce5cf2017a..038c1eeef0be 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -374,6 +374,7 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev)
 {
 	struct rt6_info *rt = dst_rt6_info(dst);
 	struct inet6_dev *idev = rt->rt6i_idev;
+	struct fib6_info *from;
 
 	if (idev && idev->dev != blackhole_netdev) {
 		struct inet6_dev *blackhole_idev = in6_dev_get(blackhole_netdev);
@@ -383,6 +384,8 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev)
 			in6_dev_put(idev);
 		}
 	}
+	from = unrcu_pointer(xchg(&rt->from, NULL));
+	fib6_info_release(from);
 }
 
 static bool __rt6_check_expired(const struct rt6_info *rt)
@@ -1455,7 +1458,6 @@ static DEFINE_SPINLOCK(rt6_exception_lock);
 static void rt6_remove_exception(struct rt6_exception_bucket *bucket,
 				 struct rt6_exception *rt6_ex)
 {
-	struct fib6_info *from;
 	struct net *net;
 
 	if (!bucket || !rt6_ex)
@@ -1467,8 +1469,6 @@ static void rt6_remove_exception(struct rt6_exception_bucket *bucket,
 	/* purge completely the exception to allow releasing the held resources:
 	 * some [sk] cache may keep the dst around for unlimited time
 	 */
-	from = unrcu_pointer(xchg(&rt6_ex->rt6i->from, NULL));
-	fib6_info_release(from);
 	dst_dev_put(&rt6_ex->rt6i->dst);
 
 	hlist_del_rcu(&rt6_ex->hlist);
-- 
2.45.2

Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

[Eric Dumazet]

On Tue, Nov 5, 2024 at 7:24 PM Paolo Abeni <pabeni@redhat.com> wrote:
>
> The CI is hitting some aperiodic hangup at device removal time in the
> pmtu.sh self-test:
>
> unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
> ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
>         dst_init+0x84/0x4a0
>         dst_alloc+0x97/0x150
>         ip6_dst_alloc+0x23/0x90
>         ip6_rt_pcpu_alloc+0x1e6/0x520
>         ip6_pol_route+0x56f/0x840
>         fib6_rule_lookup+0x334/0x630
>         ip6_route_output_flags+0x259/0x480
>         ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
>         ip6_dst_lookup_flow+0x88/0x190
>         udp_tunnel6_dst_lookup+0x2a7/0x4c0
>         vxlan_xmit_one+0xbde/0x4a50 [vxlan]
>         vxlan_xmit+0x9ad/0xf20 [vxlan]
>         dev_hard_start_xmit+0x10e/0x360
>         __dev_queue_xmit+0xf95/0x18c0
>         arp_solicit+0x4a2/0xe00
>         neigh_probe+0xaa/0xf0
>
> While the first suspect is the dst_cache, explicitly tracking the dst
> owing the last device reference via probes proved such dst is held by
> the nexthop in the originating fib6_info.
>
> Similar to commit f5b51fe804ec ("ipv6: route: purge exception on
> removal"), we need to explicitly release the originating fib info when
> disconnecting a to-be-removed device from a live ipv6 dst: move the
> fib6_info cleanup into ip6_dst_ifdown().
>
> Tested running:
>
> ./pmtu.sh cleanup_ipv6_exception
>
> in a tight loop for more than 400 iterations with no spat, running an
> unpatched kernel  I observed a splat every ~10 iterations.
>
> Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>

Thanks a lot Paolo

Reviewed-by: Eric Dumazet <edumazet@google.com>

Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

[David Ahern]

On 11/5/24 11:23 AM, Paolo Abeni wrote:
> The CI is hitting some aperiodic hangup at device removal time in the
> pmtu.sh self-test:
> 
> unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
> ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
> 	dst_init+0x84/0x4a0
> 	dst_alloc+0x97/0x150
> 	ip6_dst_alloc+0x23/0x90
> 	ip6_rt_pcpu_alloc+0x1e6/0x520
> 	ip6_pol_route+0x56f/0x840
> 	fib6_rule_lookup+0x334/0x630
> 	ip6_route_output_flags+0x259/0x480
> 	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
> 	ip6_dst_lookup_flow+0x88/0x190
> 	udp_tunnel6_dst_lookup+0x2a7/0x4c0
> 	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
> 	vxlan_xmit+0x9ad/0xf20 [vxlan]
> 	dev_hard_start_xmit+0x10e/0x360
> 	__dev_queue_xmit+0xf95/0x18c0
> 	arp_solicit+0x4a2/0xe00
> 	neigh_probe+0xaa/0xf0
> 
> While the first suspect is the dst_cache, explicitly tracking the dst
> owing the last device reference via probes proved such dst is held by
> the nexthop in the originating fib6_info.
> 
> Similar to commit f5b51fe804ec ("ipv6: route: purge exception on
> removal"), we need to explicitly release the originating fib info when
> disconnecting a to-be-removed device from a live ipv6 dst: move the
> fib6_info cleanup into ip6_dst_ifdown().
> 
> Tested running:
> 
> ./pmtu.sh cleanup_ipv6_exception
> 
> in a tight loop for more than 400 iterations with no spat, running an
> unpatched kernel  I observed a splat every ~10 iterations.
> 
> Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")

are you sure that is the correct Fixes? That commit is June 2019 and
there have been stable periods since then without netdev release problems.

> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> ---
> v1 -> v2:
>  - dropped unintended whitespace change
> ---
>  net/ipv6/route.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

[Paolo Abeni]

On 11/5/24 22:40, David Ahern wrote:
> On 11/5/24 11:23 AM, Paolo Abeni wrote:
>> The CI is hitting some aperiodic hangup at device removal time in the
>> pmtu.sh self-test:
>>
>> unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
>> ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at
>> 	dst_init+0x84/0x4a0
>> 	dst_alloc+0x97/0x150
>> 	ip6_dst_alloc+0x23/0x90
>> 	ip6_rt_pcpu_alloc+0x1e6/0x520
>> 	ip6_pol_route+0x56f/0x840
>> 	fib6_rule_lookup+0x334/0x630
>> 	ip6_route_output_flags+0x259/0x480
>> 	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
>> 	ip6_dst_lookup_flow+0x88/0x190
>> 	udp_tunnel6_dst_lookup+0x2a7/0x4c0
>> 	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
>> 	vxlan_xmit+0x9ad/0xf20 [vxlan]
>> 	dev_hard_start_xmit+0x10e/0x360
>> 	__dev_queue_xmit+0xf95/0x18c0
>> 	arp_solicit+0x4a2/0xe00
>> 	neigh_probe+0xaa/0xf0
>>
>> While the first suspect is the dst_cache, explicitly tracking the dst
>> owing the last device reference via probes proved such dst is held by
>> the nexthop in the originating fib6_info.
>>
>> Similar to commit f5b51fe804ec ("ipv6: route: purge exception on
>> removal"), we need to explicitly release the originating fib info when
>> disconnecting a to-be-removed device from a live ipv6 dst: move the
>> fib6_info cleanup into ip6_dst_ifdown().
>>
>> Tested running:
>>
>> ./pmtu.sh cleanup_ipv6_exception
>>
>> in a tight loop for more than 400 iterations with no spat, running an
>> unpatched kernel  I observed a splat every ~10 iterations.
>>
>> Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
> 
> are you sure that is the correct Fixes? That commit is June 2019 and
> there have been stable periods since then without netdev release problems.

"Sure" is a big word ;) AFAICS the mentioned commit let fib6_info store
indirectly the extra dev reference via nexthop and does not clean it at
device removal time.

Note that the issue is not deterministic - I needed ~30 mptu.sh
iterations in a row to see it, so it could go unnoticed for a long time.

>> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
>> ---
>> v1 -> v2:
>>  - dropped unintended whitespace change
>> ---
>>  net/ipv6/route.c | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
> 
> Reviewed-by: David Ahern <dsahern@kernel.org>

Thanks!

Paolo

[PATCH v2 net-next 2/2] selftests: net: really check for bg process completion

[Paolo Abeni]

A recent refactor transformed the check for process completion
in a true statement, due to a typo.

As a result, the relevant test-case is unable to catch the
regression it was supposed to detect.

Restore the correct condition.

Fixes: 691bb4e49c98 ("selftests: net: avoid just another constant wait")
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
---
 tools/testing/selftests/net/pmtu.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh
index 569bce8b6383..6c651c880fe8 100755
--- a/tools/testing/selftests/net/pmtu.sh
+++ b/tools/testing/selftests/net/pmtu.sh
@@ -2056,7 +2056,7 @@ check_running() {
 	pid=${1}
 	cmd=${2}
 
-	[ "$(cat /proc/${pid}/cmdline 2>/dev/null | tr -d '\0')" = "{cmd}" ]
+	[ "$(cat /proc/${pid}/cmdline 2>/dev/null | tr -d '\0')" = "${cmd}" ]
 }
 
 test_cleanup_vxlanX_exception() {
-- 
2.45.2

Re: [PATCH v2 net-next 2/2] selftests: net: really check for bg process completion

[David Ahern]

On 11/5/24 11:23 AM, Paolo Abeni wrote:
> A recent refactor transformed the check for process completion
> in a true statement, due to a typo.
> 
> As a result, the relevant test-case is unable to catch the
> regression it was supposed to detect.
> 
> Restore the correct condition.
> 
> Fixes: 691bb4e49c98 ("selftests: net: avoid just another constant wait")
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> ---
>  tools/testing/selftests/net/pmtu.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH v2 net-next 0/2] ipv6: fix hangup on device removal

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Tue,  5 Nov 2024 19:23:49 +0100 you wrote:
> This addresses the infamous unregister_netdevice splat in net selftests;
> the actual fix is carried by the first patch, while the 2nd one
> addresses a related problem in the relevant test that was patially
> hiding the problem.
> 
> Targeting net-next as the issue is quite old and I feel a little lost
> in the fib info/nh jungle.
> 
> [...]

Here is the summary with links:
  - [v2,net-next,1/2] ipv6: release nexthop on device removal
    https://git.kernel.org/netdev/net-next/c/eb02688c5c45
  - [v2,net-next,2/2] selftests: net: really check for bg process completion
    https://git.kernel.org/netdev/net-next/c/52ed077aa633

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html