[PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Fernando Fernandez Mancera]

Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.

NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.

As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.

This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.

Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
Reported-by: Chris Adams <linux@cmadams.net>
Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
---
v2: updated commit description to make it more accurate
---
 net/ipv6/addrconf.c                  | 2 +-
 tools/testing/selftests/net/ioam6.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 5476b6536eb7..bb84a78b80f6 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
 	list_for_each(p, &idev->addr_list) {
 		struct inet6_ifaddr *ifa
 			= list_entry(p, struct inet6_ifaddr, if_list);
-		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
+		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
 			break;
 	}
 
diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
index b2b99889942f..845c26dd01a9 100755
--- a/tools/testing/selftests/net/ioam6.sh
+++ b/tools/testing/selftests/net/ioam6.sh
@@ -273,8 +273,8 @@ setup()
   ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
   ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
 
-  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
   ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
+  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
   ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
   ip -netns $ioam_node_alpha link set lo up &>/dev/null
   ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \
-- 
2.54.0

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Stefano Brivio]

On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera <fmancera@suse.de> wrote:

> Chris Adams reported that preserving insertion order for same-scope
> addresses is causing SSH connections to be dropped after stopping a VM
> while running NetworkManager.
> 
> NetworkManager caches the IPv6 address configuration, when a RA arrives,
> it determines the list of addresses to configure and checks if the
> addresses are already in the right order in the kernel. If they aren't,
> NetworkManager removes and re-adds them to achieve the desired order.
> 
> As the order changes, NetworkManager is confused and reconfigures the
> addresses on every update. In addition, this would also affect to cloud
> tooling that relies on IPv6 addresses order to identify primary and
> secondaries addresses.

By the way, I'm still looking into this part, trying to find
"problematic" examples.

And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.

> 
> This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
> 
> Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
> Reported-by: Chris Adams <linux@cmadams.net>
> Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
> Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
> ---
> v2: updated commit description to make it more accurate
> ---
>  net/ipv6/addrconf.c                  | 2 +-
>  tools/testing/selftests/net/ioam6.sh | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
> index 5476b6536eb7..bb84a78b80f6 100644
> --- a/net/ipv6/addrconf.c
> +++ b/net/ipv6/addrconf.c
> @@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
>  	list_for_each(p, &idev->addr_list) {
>  		struct inet6_ifaddr *ifa
>  			= list_entry(p, struct inet6_ifaddr, if_list);
> -		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
> +		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
>  			break;
>  	}
>  
> diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
> index b2b99889942f..845c26dd01a9 100755
> --- a/tools/testing/selftests/net/ioam6.sh
> +++ b/tools/testing/selftests/net/ioam6.sh
> @@ -273,8 +273,8 @@ setup()
>    ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
>    ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
>  
> -  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>    ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
> +  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>    ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
>    ip -netns $ioam_node_alpha link set lo up &>/dev/null
>    ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \

-- 
Stefano

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Fernando Fernandez Mancera]

On 5/29/26 1:41 PM, Stefano Brivio wrote:
> On Fri, 29 May 2026 13:23:57 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
>> Chris Adams reported that preserving insertion order for same-scope
>> addresses is causing SSH connections to be dropped after stopping a VM
>> while running NetworkManager.
>>
>> NetworkManager caches the IPv6 address configuration, when a RA arrives,
>> it determines the list of addresses to configure and checks if the
>> addresses are already in the right order in the kernel. If they aren't,
>> NetworkManager removes and re-adds them to achieve the desired order.
>>
>> As the order changes, NetworkManager is confused and reconfigures the
>> addresses on every update. In addition, this would also affect to cloud
>> tooling that relies on IPv6 addresses order to identify primary and
>> secondaries addresses.
> 
> By the way, I'm still looking into this part, trying to find
> "problematic" examples.
> 
> And I couldn't find any, yet, because it looks like there's always a
> _single_ IPv6 address being used as a secondary for a primary IPv4
> address.
> 

IIRC, Azure cloud should be one of them. I do not have an account there 
to test it.. but some years ago I did some work supporting configuring 
IPV6 primary and secondary addresses via IMDSv2.

If someone with an account could test it I would appreciate it.

>>
>> This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
>>
>> Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>> Reported-by: Chris Adams <linux@cmadams.net>
>> Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
>> Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
>> ---
>> v2: updated commit description to make it more accurate
>> ---
>>   net/ipv6/addrconf.c                  | 2 +-
>>   tools/testing/selftests/net/ioam6.sh | 2 +-
>>   2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
>> index 5476b6536eb7..bb84a78b80f6 100644
>> --- a/net/ipv6/addrconf.c
>> +++ b/net/ipv6/addrconf.c
>> @@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
>>   	list_for_each(p, &idev->addr_list) {
>>   		struct inet6_ifaddr *ifa
>>   			= list_entry(p, struct inet6_ifaddr, if_list);
>> -		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
>> +		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
>>   			break;
>>   	}
>>   
>> diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
>> index b2b99889942f..845c26dd01a9 100755
>> --- a/tools/testing/selftests/net/ioam6.sh
>> +++ b/tools/testing/selftests/net/ioam6.sh
>> @@ -273,8 +273,8 @@ setup()
>>     ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
>>     ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
>>   
>> -  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>>     ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
>> +  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>>     ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
>>     ip -netns $ioam_node_alpha link set lo up &>/dev/null
>>     ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \
> 

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Chris Adams]

Once upon a time, Stefano Brivio <sbrivio@redhat.com> said:
> On Fri, 29 May 2026 13:23:57 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
> > Chris Adams reported that preserving insertion order for same-scope
> > addresses is causing SSH connections to be dropped after stopping a VM
> > while running NetworkManager.
> > 
> > NetworkManager caches the IPv6 address configuration, when a RA arrives,
> > it determines the list of addresses to configure and checks if the
> > addresses are already in the right order in the kernel. If they aren't,
> > NetworkManager removes and re-adds them to achieve the desired order.
> > 
> > As the order changes, NetworkManager is confused and reconfigures the
> > addresses on every update. In addition, this would also affect to cloud
> > tooling that relies on IPv6 addresses order to identify primary and
> > secondaries addresses.
> 
> By the way, I'm still looking into this part, trying to find
> "problematic" examples.
> 
> And I couldn't find any, yet, because it looks like there's always a
> _single_ IPv6 address being used as a secondary for a primary IPv4
> address.

I hit two cases that triggered the behavior:

- after an overnight suspend/resume cycle (not sure if there's a minimum
  time suspended)

- with a bridge NIC, after stopping a VM with a NIC on the bridge (which
  removed an fe80:: link-local address); this is the case I used to
  bisect since it was easy to script

In both cases, the next IPv6 router-advertisement received was what
triggered NetworkManager to replace the current privacy addresses.  This
could be 2-3 minutes after the trigger.

> > This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
> > 
> > Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
> > Reported-by: Chris Adams <linux@cmadams.net>
> > Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
> > Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
> > ---
> > v2: updated commit description to make it more accurate
> > ---
> >  net/ipv6/addrconf.c                  | 2 +-
> >  tools/testing/selftests/net/ioam6.sh | 2 +-
> >  2 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
> > index 5476b6536eb7..bb84a78b80f6 100644
> > --- a/net/ipv6/addrconf.c
> > +++ b/net/ipv6/addrconf.c
> > @@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
> >  	list_for_each(p, &idev->addr_list) {
> >  		struct inet6_ifaddr *ifa
> >  			= list_entry(p, struct inet6_ifaddr, if_list);
> > -		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
> > +		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
> >  			break;
> >  	}
> >  
> > diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
> > index b2b99889942f..845c26dd01a9 100755
> > --- a/tools/testing/selftests/net/ioam6.sh
> > +++ b/tools/testing/selftests/net/ioam6.sh
> > @@ -273,8 +273,8 @@ setup()
> >    ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
> >    ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
> >  
> > -  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
> >    ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
> > +  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
> >    ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
> >    ip -netns $ioam_node_alpha link set lo up &>/dev/null
> >    ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \
> 
> -- 
> Stefano

-- 
Chris Adams <linux@cmadams.net>

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 29 May 2026 13:23:57 +0200 you wrote:
> Chris Adams reported that preserving insertion order for same-scope
> addresses is causing SSH connections to be dropped after stopping a VM
> while running NetworkManager.
> 
> NetworkManager caches the IPv6 address configuration, when a RA arrives,
> it determines the list of addresses to configure and checks if the
> addresses are already in the right order in the kernel. If they aren't,
> NetworkManager removes and re-adds them to achieve the desired order.
> 
> [...]

Here is the summary with links:
  - [net,v2] Revert "ipv6: preserve insertion order for same-scope addresses"
    https://git.kernel.org/netdev/net/c/072aa0f5c3d8

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Matthieu Baerts]

Hi Stefano,

On 29/05/2026 21:41, Stefano Brivio wrote:
> On Fri, 29 May 2026 13:23:57 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
>> Chris Adams reported that preserving insertion order for same-scope
>> addresses is causing SSH connections to be dropped after stopping a VM
>> while running NetworkManager.
>>
>> NetworkManager caches the IPv6 address configuration, when a RA arrives,
>> it determines the list of addresses to configure and checks if the
>> addresses are already in the right order in the kernel. If they aren't,
>> NetworkManager removes and re-adds them to achieve the desired order.
>>
>> As the order changes, NetworkManager is confused and reconfigures the
>> addresses on every update. In addition, this would also affect to cloud
>> tooling that relies on IPv6 addresses order to identify primary and
>> secondaries addresses.
> 
> By the way, I'm still looking into this part, trying to find
> "problematic" examples.
> 
> And I couldn't find any, yet, because it looks like there's always a
> _single_ IPv6 address being used as a secondary for a primary IPv4
> address.

FYI, the order change also affected some specific scripts, e.g. here
with MPTCP and packetdrill:

  https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8

Because the order was not "natural" before, and different from IPv4, a
workaround was needed to keep the same order. I was happy to remove it,
but now it looks like I need to re-apply it :)

It would be nice to get the "natural" order back without breaking the
userspace (or with a way to choose the order).

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Stefano Brivio]

On Mon, 1 Jun 2026 12:03:59 +1000
Matthieu Baerts <matttbe@kernel.org> wrote:

> Hi Stefano,
> 
> On 29/05/2026 21:41, Stefano Brivio wrote:
> > On Fri, 29 May 2026 13:23:57 +0200
> > Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> >   
> >> Chris Adams reported that preserving insertion order for same-scope
> >> addresses is causing SSH connections to be dropped after stopping a VM
> >> while running NetworkManager.
> >>
> >> NetworkManager caches the IPv6 address configuration, when a RA arrives,
> >> it determines the list of addresses to configure and checks if the
> >> addresses are already in the right order in the kernel. If they aren't,
> >> NetworkManager removes and re-adds them to achieve the desired order.
> >>
> >> As the order changes, NetworkManager is confused and reconfigures the
> >> addresses on every update. In addition, this would also affect to cloud
> >> tooling that relies on IPv6 addresses order to identify primary and
> >> secondaries addresses.  
> > 
> > By the way, I'm still looking into this part, trying to find
> > "problematic" examples.
> > 
> > And I couldn't find any, yet, because it looks like there's always a
> > _single_ IPv6 address being used as a secondary for a primary IPv4
> > address.  
> 
> FYI, the order change also affected some specific scripts, e.g. here
> with MPTCP and packetdrill:
> 
>   https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8
> 
> Because the order was not "natural" before, and different from IPv4, a
> workaround was needed to keep the same order. I was happy to remove it,
> but now it looks like I need to re-apply it :)

Ouch. :) We were are also pondering about some kind of workaround like
that (https://bugs.passt.top/show_bug.cgi?id=175#c9) but fixing the
kernel looked simpler and the right thing to do... until last week.

> It would be nice to get the "natural" order back without breaking the
> userspace (or with a way to choose the order).

I was thinking that if we implement a label like NLM_F_INSERT_LAST
(David's proposal for the name), we could patch iproute2 to set it on
'ip address restore' at least, other than using it in pasta(1).

But that wouldn't be enough for your case. At the same time always
adding it for RTM_NEWADDR requests in iproute2 could break somebody
else's scripts. I guess a reasonable solution could be to add an
additional parameter for ip-address... 'insert_last'? 'last'?

-- 
Stefano

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Íñigo Huguet]

On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio <sbrivio@redhat.com> wrote:
> I was thinking that if we implement a label like NLM_F_INSERT_LAST
> (David's proposal for the name), we could patch iproute2 to set it on
> 'ip address restore' at least, other than using it in pasta(1).

I'm not sure whether what I'm going to say is a good idea, or just a
plain stupid proposal:

By reverting the old odd behaviour, UAPI is restored, yes. On the
other side, userspace programs won't have any incentive to start using
NLM_F_INSERT_LAST, which is actually the most desirable behaviour. The
old odd behaviour is very error prone, as we can see with the `ip addr
restore` bug. That's why programs should adopt the "new" behaviour and
this will save them from introducing bugs like that one in the future.
And new programs should get the normal insertion by default, without
needing to read documentation or kernel headers to find
NLM_F_INSERT_LAST.

What if we reapply the patch and add a NLM_F_INSERT_FIRST /
NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
probably not too bad per the conversations in this and the other
thread. Userspace programs affected by the change, like
NetworkManager, will need to be fixed, either by switching to natural
order insertion, or by passing the NLM_F_INSERT_FIRST flag to restore
the old behaviour.

Additionally, a NLM_F_INSERT_FIRST can be implemented for IPv4 too, at
least for consistency (I don't know whether it may be actually useful
for something, likely yes?).

As I said, this solution would break a few programs like
NetworkManager, but the fix is as easy as `if (ipv6) flags |=
NLM_F_INSERT_FIRST`. Would old kernels ignore this unknown flag?
Because, in that case, we don't even need a detection mechanism from
NetworkManager.

>
> But that wouldn't be enough for your case. At the same time always
> adding it for RTM_NEWADDR requests in iproute2 could break somebody
> else's scripts. I guess a reasonable solution could be to add an
> additional parameter for ip-address... 'insert_last'? 'last'?
>
> --
> Stefano
>


--
Íñigo Huguet

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Thorsten Leemhuis]

On 6/1/26 16:01, Íñigo Huguet wrote:
> As I said, this solution would break a few programs like
> NetworkManager

FWIW, that is not allowed in the Linux kernel; see

https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-situations-where-updating-something-in-userspace-can-resolve-regressions

and other sections on that page.

As mentioned earlier, there are nevertheless solutions for this kind of
problems, but they are all somewhat ugly -- like some bit in /proc/ a
fixed NetworkManager could flip at boot time, as then everything will
continue to work for those that update the kernel without updating
NetworkManager.

Ciao, Thorsten