Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Stefano Brivio]

On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera <fmancera@suse.de> wrote:

> Chris Adams reported that preserving insertion order for same-scope
> addresses is causing SSH connections to be dropped after stopping a VM
> while running NetworkManager.
> 
> NetworkManager caches the IPv6 address configuration, when a RA arrives,
> it determines the list of addresses to configure and checks if the
> addresses are already in the right order in the kernel. If they aren't,
> NetworkManager removes and re-adds them to achieve the desired order.
> 
> As the order changes, NetworkManager is confused and reconfigures the
> addresses on every update. In addition, this would also affect to cloud
> tooling that relies on IPv6 addresses order to identify primary and
> secondaries addresses.

By the way, I'm still looking into this part, trying to find
"problematic" examples.

And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.

> 
> This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
> 
> Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
> Reported-by: Chris Adams <linux@cmadams.net>
> Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
> Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
> ---
> v2: updated commit description to make it more accurate
> ---
>  net/ipv6/addrconf.c                  | 2 +-
>  tools/testing/selftests/net/ioam6.sh | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
> index 5476b6536eb7..bb84a78b80f6 100644
> --- a/net/ipv6/addrconf.c
> +++ b/net/ipv6/addrconf.c
> @@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
>  	list_for_each(p, &idev->addr_list) {
>  		struct inet6_ifaddr *ifa
>  			= list_entry(p, struct inet6_ifaddr, if_list);
> -		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
> +		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
>  			break;
>  	}
>  
> diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
> index b2b99889942f..845c26dd01a9 100755
> --- a/tools/testing/selftests/net/ioam6.sh
> +++ b/tools/testing/selftests/net/ioam6.sh
> @@ -273,8 +273,8 @@ setup()
>    ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
>    ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
>  
> -  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>    ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
> +  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>    ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
>    ip -netns $ioam_node_alpha link set lo up &>/dev/null
>    ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \

-- 
Stefano

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Fernando Fernandez Mancera]

On 5/29/26 1:41 PM, Stefano Brivio wrote:
> On Fri, 29 May 2026 13:23:57 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
>> Chris Adams reported that preserving insertion order for same-scope
>> addresses is causing SSH connections to be dropped after stopping a VM
>> while running NetworkManager.
>>
>> NetworkManager caches the IPv6 address configuration, when a RA arrives,
>> it determines the list of addresses to configure and checks if the
>> addresses are already in the right order in the kernel. If they aren't,
>> NetworkManager removes and re-adds them to achieve the desired order.
>>
>> As the order changes, NetworkManager is confused and reconfigures the
>> addresses on every update. In addition, this would also affect to cloud
>> tooling that relies on IPv6 addresses order to identify primary and
>> secondaries addresses.
> 
> By the way, I'm still looking into this part, trying to find
> "problematic" examples.
> 
> And I couldn't find any, yet, because it looks like there's always a
> _single_ IPv6 address being used as a secondary for a primary IPv4
> address.
> 

IIRC, Azure cloud should be one of them. I do not have an account there 
to test it.. but some years ago I did some work supporting configuring 
IPV6 primary and secondary addresses via IMDSv2.

If someone with an account could test it I would appreciate it.

>>
>> This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
>>
>> Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>> Reported-by: Chris Adams <linux@cmadams.net>
>> Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
>> Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
>> ---
>> v2: updated commit description to make it more accurate
>> ---
>>   net/ipv6/addrconf.c                  | 2 +-
>>   tools/testing/selftests/net/ioam6.sh | 2 +-
>>   2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
>> index 5476b6536eb7..bb84a78b80f6 100644
>> --- a/net/ipv6/addrconf.c
>> +++ b/net/ipv6/addrconf.c
>> @@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
>>   	list_for_each(p, &idev->addr_list) {
>>   		struct inet6_ifaddr *ifa
>>   			= list_entry(p, struct inet6_ifaddr, if_list);
>> -		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
>> +		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
>>   			break;
>>   	}
>>   
>> diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
>> index b2b99889942f..845c26dd01a9 100755
>> --- a/tools/testing/selftests/net/ioam6.sh
>> +++ b/tools/testing/selftests/net/ioam6.sh
>> @@ -273,8 +273,8 @@ setup()
>>     ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
>>     ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
>>   
>> -  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>>     ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
>> +  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
>>     ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
>>     ip -netns $ioam_node_alpha link set lo up &>/dev/null
>>     ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \
> 

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Chris Adams]

Once upon a time, Stefano Brivio <sbrivio@redhat.com> said:
> On Fri, 29 May 2026 13:23:57 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
> > Chris Adams reported that preserving insertion order for same-scope
> > addresses is causing SSH connections to be dropped after stopping a VM
> > while running NetworkManager.
> > 
> > NetworkManager caches the IPv6 address configuration, when a RA arrives,
> > it determines the list of addresses to configure and checks if the
> > addresses are already in the right order in the kernel. If they aren't,
> > NetworkManager removes and re-adds them to achieve the desired order.
> > 
> > As the order changes, NetworkManager is confused and reconfigures the
> > addresses on every update. In addition, this would also affect to cloud
> > tooling that relies on IPv6 addresses order to identify primary and
> > secondaries addresses.
> 
> By the way, I'm still looking into this part, trying to find
> "problematic" examples.
> 
> And I couldn't find any, yet, because it looks like there's always a
> _single_ IPv6 address being used as a secondary for a primary IPv4
> address.

I hit two cases that triggered the behavior:

- after an overnight suspend/resume cycle (not sure if there's a minimum
  time suspended)

- with a bridge NIC, after stopping a VM with a NIC on the bridge (which
  removed an fe80:: link-local address); this is the case I used to
  bisect since it was easy to script

In both cases, the next IPv6 router-advertisement received was what
triggered NetworkManager to replace the current privacy addresses.  This
could be 2-3 minutes after the trigger.

> > This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
> > 
> > Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
> > Reported-by: Chris Adams <linux@cmadams.net>
> > Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
> > Signed-off-by: Fernando Fernandez Mancera <fmancera@suse.de>
> > ---
> > v2: updated commit description to make it more accurate
> > ---
> >  net/ipv6/addrconf.c                  | 2 +-
> >  tools/testing/selftests/net/ioam6.sh | 2 +-
> >  2 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
> > index 5476b6536eb7..bb84a78b80f6 100644
> > --- a/net/ipv6/addrconf.c
> > +++ b/net/ipv6/addrconf.c
> > @@ -1013,7 +1013,7 @@ ipv6_link_dev_addr(struct inet6_dev *idev, struct inet6_ifaddr *ifp)
> >  	list_for_each(p, &idev->addr_list) {
> >  		struct inet6_ifaddr *ifa
> >  			= list_entry(p, struct inet6_ifaddr, if_list);
> > -		if (ifp_scope > ipv6_addr_src_scope(&ifa->addr))
> > +		if (ifp_scope >= ipv6_addr_src_scope(&ifa->addr))
> >  			break;
> >  	}
> >  
> > diff --git a/tools/testing/selftests/net/ioam6.sh b/tools/testing/selftests/net/ioam6.sh
> > index b2b99889942f..845c26dd01a9 100755
> > --- a/tools/testing/selftests/net/ioam6.sh
> > +++ b/tools/testing/selftests/net/ioam6.sh
> > @@ -273,8 +273,8 @@ setup()
> >    ip -netns $ioam_node_beta link set ioam-veth-betaR name veth1 &>/dev/null
> >    ip -netns $ioam_node_gamma link set ioam-veth-gamma name veth0 &>/dev/null
> >  
> > -  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
> >    ip -netns $ioam_node_alpha addr add 2001:db8:1::50/64 dev veth0 &>/dev/null
> > +  ip -netns $ioam_node_alpha addr add 2001:db8:1::2/64 dev veth0 &>/dev/null
> >    ip -netns $ioam_node_alpha link set veth0 up &>/dev/null
> >    ip -netns $ioam_node_alpha link set lo up &>/dev/null
> >    ip -netns $ioam_node_alpha route add 2001:db8:2::/64 \
> 
> -- 
> Stefano

-- 
Chris Adams <linux@cmadams.net>

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Matthieu Baerts]

Hi Stefano,

On 29/05/2026 21:41, Stefano Brivio wrote:
> On Fri, 29 May 2026 13:23:57 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
>> Chris Adams reported that preserving insertion order for same-scope
>> addresses is causing SSH connections to be dropped after stopping a VM
>> while running NetworkManager.
>>
>> NetworkManager caches the IPv6 address configuration, when a RA arrives,
>> it determines the list of addresses to configure and checks if the
>> addresses are already in the right order in the kernel. If they aren't,
>> NetworkManager removes and re-adds them to achieve the desired order.
>>
>> As the order changes, NetworkManager is confused and reconfigures the
>> addresses on every update. In addition, this would also affect to cloud
>> tooling that relies on IPv6 addresses order to identify primary and
>> secondaries addresses.
> 
> By the way, I'm still looking into this part, trying to find
> "problematic" examples.
> 
> And I couldn't find any, yet, because it looks like there's always a
> _single_ IPv6 address being used as a secondary for a primary IPv4
> address.

FYI, the order change also affected some specific scripts, e.g. here
with MPTCP and packetdrill:

  https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8

Because the order was not "natural" before, and different from IPv4, a
workaround was needed to keep the same order. I was happy to remove it,
but now it looks like I need to re-apply it :)

It would be nice to get the "natural" order back without breaking the
userspace (or with a way to choose the order).

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Stefano Brivio]

On Mon, 1 Jun 2026 12:03:59 +1000
Matthieu Baerts <matttbe@kernel.org> wrote:

> Hi Stefano,
> 
> On 29/05/2026 21:41, Stefano Brivio wrote:
> > On Fri, 29 May 2026 13:23:57 +0200
> > Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> >   
> >> Chris Adams reported that preserving insertion order for same-scope
> >> addresses is causing SSH connections to be dropped after stopping a VM
> >> while running NetworkManager.
> >>
> >> NetworkManager caches the IPv6 address configuration, when a RA arrives,
> >> it determines the list of addresses to configure and checks if the
> >> addresses are already in the right order in the kernel. If they aren't,
> >> NetworkManager removes and re-adds them to achieve the desired order.
> >>
> >> As the order changes, NetworkManager is confused and reconfigures the
> >> addresses on every update. In addition, this would also affect to cloud
> >> tooling that relies on IPv6 addresses order to identify primary and
> >> secondaries addresses.  
> > 
> > By the way, I'm still looking into this part, trying to find
> > "problematic" examples.
> > 
> > And I couldn't find any, yet, because it looks like there's always a
> > _single_ IPv6 address being used as a secondary for a primary IPv4
> > address.  
> 
> FYI, the order change also affected some specific scripts, e.g. here
> with MPTCP and packetdrill:
> 
>   https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8
> 
> Because the order was not "natural" before, and different from IPv4, a
> workaround was needed to keep the same order. I was happy to remove it,
> but now it looks like I need to re-apply it :)

Ouch. :) We were are also pondering about some kind of workaround like
that (https://bugs.passt.top/show_bug.cgi?id=175#c9) but fixing the
kernel looked simpler and the right thing to do... until last week.

> It would be nice to get the "natural" order back without breaking the
> userspace (or with a way to choose the order).

I was thinking that if we implement a label like NLM_F_INSERT_LAST
(David's proposal for the name), we could patch iproute2 to set it on
'ip address restore' at least, other than using it in pasta(1).

But that wouldn't be enough for your case. At the same time always
adding it for RTM_NEWADDR requests in iproute2 could break somebody
else's scripts. I guess a reasonable solution could be to add an
additional parameter for ip-address... 'insert_last'? 'last'?

-- 
Stefano

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Íñigo Huguet]

On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio <sbrivio@redhat.com> wrote:
> I was thinking that if we implement a label like NLM_F_INSERT_LAST
> (David's proposal for the name), we could patch iproute2 to set it on
> 'ip address restore' at least, other than using it in pasta(1).

I'm not sure whether what I'm going to say is a good idea, or just a
plain stupid proposal:

By reverting the old odd behaviour, UAPI is restored, yes. On the
other side, userspace programs won't have any incentive to start using
NLM_F_INSERT_LAST, which is actually the most desirable behaviour. The
old odd behaviour is very error prone, as we can see with the `ip addr
restore` bug. That's why programs should adopt the "new" behaviour and
this will save them from introducing bugs like that one in the future.
And new programs should get the normal insertion by default, without
needing to read documentation or kernel headers to find
NLM_F_INSERT_LAST.

What if we reapply the patch and add a NLM_F_INSERT_FIRST /
NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
probably not too bad per the conversations in this and the other
thread. Userspace programs affected by the change, like
NetworkManager, will need to be fixed, either by switching to natural
order insertion, or by passing the NLM_F_INSERT_FIRST flag to restore
the old behaviour.

Additionally, a NLM_F_INSERT_FIRST can be implemented for IPv4 too, at
least for consistency (I don't know whether it may be actually useful
for something, likely yes?).

As I said, this solution would break a few programs like
NetworkManager, but the fix is as easy as `if (ipv6) flags |=
NLM_F_INSERT_FIRST`. Would old kernels ignore this unknown flag?
Because, in that case, we don't even need a detection mechanism from
NetworkManager.

>
> But that wouldn't be enough for your case. At the same time always
> adding it for RTM_NEWADDR requests in iproute2 could break somebody
> else's scripts. I guess a reasonable solution could be to add an
> additional parameter for ip-address... 'insert_last'? 'last'?
>
> --
> Stefano
>


--
Íñigo Huguet

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Thorsten Leemhuis]

On 6/1/26 16:01, Íñigo Huguet wrote:
> As I said, this solution would break a few programs like
> NetworkManager

FWIW, that is not allowed in the Linux kernel; see

https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-situations-where-updating-something-in-userspace-can-resolve-regressions

and other sections on that page.

As mentioned earlier, there are nevertheless solutions for this kind of
problems, but they are all somewhat ugly -- like some bit in /proc/ a
fixed NetworkManager could flip at boot time, as then everything will
continue to work for those that update the kernel without updating
NetworkManager.

Ciao, Thorsten

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Matthieu Baerts]

On 02/06/2026 00:01, Íñigo Huguet wrote:
> On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio <sbrivio@redhat.com> wrote:
>> I was thinking that if we implement a label like NLM_F_INSERT_LAST
>> (David's proposal for the name), we could patch iproute2 to set it on
>> 'ip address restore' at least, other than using it in pasta(1).

(...)

> What if we reapply the patch and add a NLM_F_INSERT_FIRST /
> NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
> probably not too bad per the conversations in this and the other
> thread. 

I guess it is safer not to break UAPI and not reapplying the patch,
especially when there is no clear way to know which order is taken by
the kernel.

What about clearly stating that the default order is "random", except
when an additional flag is set to sort them chronologically or the opposite?

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Íñigo Huguet]

On Wed, Jun 3, 2026 at 7:46 AM Matthieu Baerts <matttbe@kernel.org> wrote:
> I guess it is safer not to break UAPI and not reapplying the patch,
> especially when there is no clear way to know which order is taken by
> the kernel.
>
> What about clearly stating that the default order is "random", except
> when an additional flag is set to sort them chronologically or the opposite?

Stating that the order is "random" implies that someone will change
the default insertion order in the future, as the documentation says
that it's not UAPI. So they'll cause the UAPI breakage that we are
trying to avoid now. Moreover, if we consider the default order
"random" and state that userspace programs must not rely on it, why
can't we change it now?

As mentioned in the other thread by David, we have programs broken by
the original "fix" (NetworkManager), and programs that will be broken
(again) by the revert (pasta). But more importantly, I think that,
with the revert, there may be programs that are broken but they don't
know it yet. Again, see how `ip addr restore` will be broken again.
So, both solutions are bad, but doing the revert is worse, IMO.

As I'm not much involved in NetworkManager any more, only Beniamino is
who can truly speak on behalf of the NetworkManager project, but as a
former NetworkManager maintainer, I think that my proposal earlier in
this thread is the way to go:
https://lore.kernel.org/netdev/CACT4oueDv=KRZmJw=1N58EN7eoFz5AqbBBMP8FE4=+rQ99oFEg@mail.gmail.com/.

It's slightly controversial because it may be considered an UAPI
break, but on the other side it may be considered as a fix to a wrong
implementation of the UAPI, instead, as Stefano pointed out:
https://lore.kernel.org/netdev/20260528165320.15b90ded@elisabeth/

--
Íñigo Huguet

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Thorsten Leemhuis]

On 6/3/26 08:53, Íñigo Huguet wrote:
> Stating that the order is "random" implies that someone will change
> the default insertion order in the future, as the documentation says
> that it's not UAPI. So they'll cause the UAPI breakage that we are
> trying to avoid now. Moreover, if we consider the default order
> "random" and state that userspace programs must not rely on it, why
> can't we change it now?

Because the UAPI is only indirectly important, as it's "something that
used to work doesn't anymore" is what matters; you in fact can break the
UAPI all you want, as long as everything continues to work. Or to quote
Linus from:

https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-what-qualifies-as-userspace-interface-abi-api-documented-interfaces-etc

"I just wanted to point out that the argument about whether it's an ABI
change or not is irrelevant. If it turns out that some program - not a
test script, but something with relevance to conscious user expectations
~ depended on the old broken behavior, then it needs to be done some
other way.""

Yes, he talks about ABIs here, but it's the same for APIs.

Ciao, Thorsten

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Fernando Fernandez Mancera]

On 6/3/26 9:17 AM, Thorsten Leemhuis wrote:
> On 6/3/26 08:53, Íñigo Huguet wrote:
>> Stating that the order is "random" implies that someone will change
>> the default insertion order in the future, as the documentation says
>> that it's not UAPI. So they'll cause the UAPI breakage that we are
>> trying to avoid now. Moreover, if we consider the default order
>> "random" and state that userspace programs must not rely on it, why
>> can't we change it now?
> 
> Because the UAPI is only indirectly important, as it's "something that
> used to work doesn't anymore" is what matters; you in fact can break the
> UAPI all you want, as long as everything continues to work. Or to quote
> Linus from:
> 

I agree with Thorsten here. We must not break the UAPI. Is the current 
order kind of weird? Yes. But software that is running on real systems 
depends on it.

AFAICS, it is not only NetworkManager but systemd-networkd, cloud 
tooling and any other script that uses netlink. I think we must stick to 
the rule of not breaking userspace and the argument of consistency with 
IPv4 is not appealing to me because that isn't a goal of IPv6, plenty is 
different when you compared both protocols.

We should document that the order matters, no that it is "random" 
because it isn't.

> https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-what-qualifies-as-userspace-interface-abi-api-documented-interfaces-etc
> 
> "I just wanted to point out that the argument about whether it's an ABI
> change or not is irrelevant. If it turns out that some program - not a
> test script, but something with relevance to conscious user expectations
> ~ depended on the old broken behavior, then it needs to be done some
> other way.""
> 
> Yes, he talks about ABIs here, but it's the same for APIs.
> 
> Ciao, Thorsten

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Ido Schimmel]

On Wed, Jun 03, 2026 at 09:29:55AM +0200, Fernando Fernandez Mancera wrote:
> We should document that the order matters, no that it is "random" because it
> isn't.

Did you consider writing a kernel selftest for this? Unlikely that the
order will change again when we are explicitly testing for a certain
order.

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Fernando Fernandez Mancera]

On 6/3/26 10:00 AM, Ido Schimmel wrote:
> On Wed, Jun 03, 2026 at 09:29:55AM +0200, Fernando Fernandez Mancera wrote:
>> We should document that the order matters, no that it is "random" because it
>> isn't.
> 
> Did you consider writing a kernel selftest for this? Unlikely that the
> order will change again when we are explicitly testing for a certain
> order.
> 

That is a good idea yes, I am adding it to my TO-DO list.

Thanks,
Fernando.

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[Matthieu Baerts]

Hi Íñigo,

On 03/06/2026 16:53, Íñigo Huguet wrote:
> On Wed, Jun 3, 2026 at 7:46 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>> I guess it is safer not to break UAPI and not reapplying the patch,
>> especially when there is no clear way to know which order is taken by
>> the kernel.
>>
>> What about clearly stating that the default order is "random", except
>> when an additional flag is set to sort them chronologically or the opposite?
> 
> Stating that the order is "random" implies that someone will change
> the default insertion order in the future, as the documentation says
> that it's not UAPI. So they'll cause the UAPI breakage that we are
> trying to avoid now. Moreover, if we consider the default order
> "random" and state that userspace programs must not rely on it, why
> can't we change it now?

I was only suggesting "random", simply to force userspace programs to
ask for a specific order when the order matters. But also to avoid
having to describe the current order, which is different in v4 and v6
for entries of the same scope.

I know the order is not random, and even if it is confusing, I don't
think we can change it without breaking stuff.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses"

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On Wed, Jun 03, 2026 at 03:46:14PM +1000, Matthieu Baerts wrote:
> 
> 
> On 02/06/2026 00:01, Íñigo Huguet wrote:
> > On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio <sbrivio@redhat.com> wrote:
> >> I was thinking that if we implement a label like NLM_F_INSERT_LAST
> >> (David's proposal for the name), we could patch iproute2 to set it on
> >> 'ip address restore' at least, other than using it in pasta(1).
> 
> (...)
> 
> > What if we reapply the patch and add a NLM_F_INSERT_FIRST /
> > NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
> > probably not too bad per the conversations in this and the other
> > thread. 
> 
> I guess it is safer not to break UAPI and not reapplying the patch,
> especially when there is no clear way to know which order is taken by
> the kernel.

I think Íñigo wasn't suggesting reapplying the patch as is, but a
modified version that only makes the behavioural change when userspace
sets a new attribute/flag/whatever.

> What about clearly stating that the default order is "random", except
> when an additional flag is set to sort them chronologically or the
> opposite?

We could, but since userspace relying on the existing behaviour is
already out there, I'm not sure what it buys us.  The insertion order
also isn't random - it's sorted by scope at least.  Theoretically
there could be other (existing) sorting criteria for other address
types (I haven't looked beyong IPv4 and IPv6).  The problem arises
because IPv4 and IPv6 have different behaviours when inserting
something where the order isn't defined - IPv4 inserts after existing
entries of the same scope, IPv6 puts it before existing entries of the
same scope - it's literally a < versus a <=.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2570 bytes --]

I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.

Stefano has enough other fires to fight, so I'm taking a look at
implementing that.  Some initial thoughts, that I'm soliciting
feedback on:

1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
   nlmsghdr

ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it.  Plus all its bits are allocated, anyway.

2) Could we re-use NLM_F_APPEND?

The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing.  Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.

So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?

On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.

3) What other things might need changing to take advantage of the
   kernel change

My innitial testing suggests that unknown nlmsg_flags bits are
currently ignored by the kernel.  That means that tools for which the
new behviour is desirable, but not essential may be able to avoid
probing - they can set the bit and hope.  I think that includes
passt/pasta and also iproute's save/restore functionality.

That said, things which might want updating:
 - iproute2 to use this for save/restore
 - netlink(7) man page to detail the new flag / new flag semantics
 - passt/pasta
 - network-manager, here we require the exact behaviour be maintained,
   so it would have to attempt the new flag, then apply the existing
   workaround (reversing the order itself) if that fails.  Maybe more
   trouble that it's worth?

Any others people can think of?

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Andrew Lunn]

On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
> I get the impression there's a rough consensus that the best we can do
> now is revert this change (already done), and make a new patch which
> changes the insertion order to the "correct" one conditional on a new
> flag.
> 
> Stefano has enough other fires to fight, so I'm taking a look at
> implementing that.  Some initial thoughts, that I'm soliciting
> feedback on:

I've only been partially reading along...

Are we talking about RTM_NEWADDR?

I've never worked on the code dealing with addresses. But in general,
if you want to add new functionality to a netlink message, you add a
new attribute to the message.

https://elixir.bootlin.com/linux/v7.0.10/source/include/uapi/linux/if_addr.h#L26

	Andrew

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

On Tue, Jun 02, 2026 at 02:46:22PM +0200, Andrew Lunn wrote:
> On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
> > I get the impression there's a rough consensus that the best we can do
> > now is revert this change (already done), and make a new patch which
> > changes the insertion order to the "correct" one conditional on a new
> > flag.
> > 
> > Stefano has enough other fires to fight, so I'm taking a look at
> > implementing that.  Some initial thoughts, that I'm soliciting
> > feedback on:
> 
> I've only been partially reading along...
> 
> Are we talking about RTM_NEWADDR?

Yes.

> I've never worked on the code dealing with addresses. But in general,
> if you want to add new functionality to a netlink message, you add a
> new attribute to the message.
> 
> https://elixir.bootlin.com/linux/v7.0.10/source/include/uapi/linux/if_addr.h#L26

Ah, good point, that's another option, and avoids using scarce flags
bits.  It is a little bit odd, because generally the attributes are,
well, attributes, _of the new object_ being created (an address in
this case).  Here we're adjusting where / how it is created, not
anything about the address itself.

Stil, definitely a better option that allocating a new flags bit.
Versus NLM_F_APPEND, I'll reply to Ido Schimmel.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Ido Schimmel]

On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
> I get the impression there's a rough consensus that the best we can do
> now is revert this change (already done), and make a new patch which
> changes the insertion order to the "correct" one conditional on a new
> flag.
> 
> Stefano has enough other fires to fight, so I'm taking a look at
> implementing that.  Some initial thoughts, that I'm soliciting
> feedback on:
> 
> 1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
>    nlmsghdr
> 
> ifa_flags in ifaddrmsg would be the other candidate, but it looks like
> it's encoding properties of the address itself, not about the action
> of inserting it.  Plus all its bits are allocated, anyway.
> 
> 2) Could we re-use NLM_F_APPEND?
> 
> The short description of this existing flag in linux/uapi/netlink.h is
> "Add to end of list" which sounds like the right thing.  Looking
> closer, however, it seems like what is' used for so far is things
> where the entity added with the NEW<whatever> operation is itself a
> list, and NLM_F_APPEND causes it to be added to rather than replaced.
> It's not used for addresses at present, AFAICT the list of addresses
> is a semantic level above the address entity itself.
> 
> So maybe re-using it for the thing I tentatively called
> NLM_F_INSERT_LAST would be confusing?
> 
> On the other hand, it's not used for addresses at the moment, so
> AFAICT there's nothing actually preventing us reusing it for this
> purpose. That would save a bit - we only have 2 general and 4 NEW
> specific bits left, by the looks of it.

This is not really viable. Even if the kernel is not using NLM_F_APPEND
for RTM_NEWADDR, but not rejecting its presence either, then we can
create a change in behavior for a user space that is currently setting
it (intentionally or not).

Example:

https://lore.kernel.org/netdev/27c249d80c346a258cfbf32f1d131ad4fe64e77c.camel@debian.org/

> 
> 3) What other things might need changing to take advantage of the
>    kernel change
> 
> My innitial testing suggests that unknown nlmsg_flags bits are
> currently ignored by the kernel.

Which is a problem... From the looks of it, the same is true for
ifa_flags.

> That means that tools for which the new behviour is desirable, but not
> essential may be able to avoid probing - they can set the bit and
> hope.  I think that includes passt/pasta and also iproute's
> save/restore functionality.
> 
> That said, things which might want updating:
>  - iproute2 to use this for save/restore
>  - netlink(7) man page to detail the new flag / new flag semantics
>  - passt/pasta
>  - network-manager, here we require the exact behaviour be maintained,
>    so it would have to attempt the new flag, then apply the existing
>    workaround (reversing the order itself) if that fails.  Maybe more
>    trouble that it's worth?
> 
> Any others people can think of?

The safest route is probably a new attribute and a corresponding keyword
in iproute2. Note that new iproute2 versions need to be able to work
with old kernels, so iproute2 cannot use the new attribute by default
(it will be rejected by old kernels).

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4722 bytes --]

On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
> On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
> > I get the impression there's a rough consensus that the best we can do
> > now is revert this change (already done), and make a new patch which
> > changes the insertion order to the "correct" one conditional on a new
> > flag.
> > 
> > Stefano has enough other fires to fight, so I'm taking a look at
> > implementing that.  Some initial thoughts, that I'm soliciting
> > feedback on:
> > 
> > 1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
> >    nlmsghdr
> > 
> > ifa_flags in ifaddrmsg would be the other candidate, but it looks like
> > it's encoding properties of the address itself, not about the action
> > of inserting it.  Plus all its bits are allocated, anyway.
> > 
> > 2) Could we re-use NLM_F_APPEND?
> > 
> > The short description of this existing flag in linux/uapi/netlink.h is
> > "Add to end of list" which sounds like the right thing.  Looking
> > closer, however, it seems like what is' used for so far is things
> > where the entity added with the NEW<whatever> operation is itself a
> > list, and NLM_F_APPEND causes it to be added to rather than replaced.
> > It's not used for addresses at present, AFAICT the list of addresses
> > is a semantic level above the address entity itself.
> > 
> > So maybe re-using it for the thing I tentatively called
> > NLM_F_INSERT_LAST would be confusing?
> > 
> > On the other hand, it's not used for addresses at the moment, so
> > AFAICT there's nothing actually preventing us reusing it for this
> > purpose. That would save a bit - we only have 2 general and 4 NEW
> > specific bits left, by the looks of it.
> 
> This is not really viable. Even if the kernel is not using NLM_F_APPEND
> for RTM_NEWADDR, but not rejecting its presence either, then we can
> create a change in behavior for a user space that is currently setting
> it (intentionally or not).
>
> Example:
> 
> https://lore.kernel.org/netdev/27c249d80c346a258cfbf32f1d131ad4fe64e77c.camel@debian.org/

Hmm.  So, in this example case we have a known, widely deployed
userspace that was broken by the change.  Similarly with the
original now-reverted "fix" for the ordering, we have a known, widely
deployed userspace that was broken.

That's a different case from a hypothetical userspace that incorrectly
used NLM_F_APPEND on RTM_NEWADDR.  Moreover, to be broken it would
need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
the counterintuitive and inconsistent insertion order for IPv6
addresses.  Absent a concrete example of something meeting both those
conditions, I'm inclined to breaking that hypothetical case when the
payoff is an easier route to get known cases working with the
preferred insertion semantics.

Fwiw, I did look at the most likely candidates: iproute2,
network-manager and libvirt, and I see no signs that they're misusing
NLM_F_APPEND in this way.

> > 3) What other things might need changing to take advantage of the
> >    kernel change
> > 
> > My innitial testing suggests that unknown nlmsg_flags bits are
> > currently ignored by the kernel.
> 
> Which is a problem... From the looks of it, the same is true for
> ifa_flags.

ifa_flags is a no go anyway.

> > That means that tools for which the new behviour is desirable, but not
> > essential may be able to avoid probing - they can set the bit and
> > hope.  I think that includes passt/pasta and also iproute's
> > save/restore functionality.
> > 
> > That said, things which might want updating:
> >  - iproute2 to use this for save/restore
> >  - netlink(7) man page to detail the new flag / new flag semantics
> >  - passt/pasta
> >  - network-manager, here we require the exact behaviour be maintained,
> >    so it would have to attempt the new flag, then apply the existing
> >    workaround (reversing the order itself) if that fails.  Maybe more
> >    trouble that it's worth?
> > 
> > Any others people can think of?
> 
> The safest route is probably a new attribute and a corresponding keyword
> in iproute2.

I guess attribute+keyword does make it pretty straightforward to opt
in to the new behaviour in scripts, not just C code.

> Note that new iproute2 versions need to be able to work
> with old kernels, so iproute2 cannot use the new attribute by default
> (it will be rejected by old kernels).

Yes, I'm aware.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Ido Schimmel]

On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
> On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
> > On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
> > > I get the impression there's a rough consensus that the best we can do
> > > now is revert this change (already done), and make a new patch which
> > > changes the insertion order to the "correct" one conditional on a new
> > > flag.
> > > 
> > > Stefano has enough other fires to fight, so I'm taking a look at
> > > implementing that.  Some initial thoughts, that I'm soliciting
> > > feedback on:
> > > 
> > > 1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
> > >    nlmsghdr
> > > 
> > > ifa_flags in ifaddrmsg would be the other candidate, but it looks like
> > > it's encoding properties of the address itself, not about the action
> > > of inserting it.  Plus all its bits are allocated, anyway.
> > > 
> > > 2) Could we re-use NLM_F_APPEND?
> > > 
> > > The short description of this existing flag in linux/uapi/netlink.h is
> > > "Add to end of list" which sounds like the right thing.  Looking
> > > closer, however, it seems like what is' used for so far is things
> > > where the entity added with the NEW<whatever> operation is itself a
> > > list, and NLM_F_APPEND causes it to be added to rather than replaced.
> > > It's not used for addresses at present, AFAICT the list of addresses
> > > is a semantic level above the address entity itself.
> > > 
> > > So maybe re-using it for the thing I tentatively called
> > > NLM_F_INSERT_LAST would be confusing?
> > > 
> > > On the other hand, it's not used for addresses at the moment, so
> > > AFAICT there's nothing actually preventing us reusing it for this
> > > purpose. That would save a bit - we only have 2 general and 4 NEW
> > > specific bits left, by the looks of it.
> > 
> > This is not really viable. Even if the kernel is not using NLM_F_APPEND
> > for RTM_NEWADDR, but not rejecting its presence either, then we can
> > create a change in behavior for a user space that is currently setting
> > it (intentionally or not).
> >
> > Example:
> > 
> > https://lore.kernel.org/netdev/27c249d80c346a258cfbf32f1d131ad4fe64e77c.camel@debian.org/
> 
> Hmm.  So, in this example case we have a known, widely deployed
> userspace that was broken by the change.  Similarly with the
> original now-reverted "fix" for the ordering, we have a known, widely
> deployed userspace that was broken.

It was also reported over three years after the kernel change went in.
Point is that we have no way of knowing how user space is using these
flags. Suddenly giving them meaning when we simply ignored them before
is risky.

> That's a different case from a hypothetical userspace that incorrectly
> used NLM_F_APPEND on RTM_NEWADDR.  Moreover, to be broken it would
> need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
> the counterintuitive and inconsistent insertion order for IPv6
> addresses.  Absent a concrete example of something meeting both those
> conditions, I'm inclined to breaking that hypothetical case when the
> payoff is an easier route to get known cases working with the
> preferred insertion semantics.
> 
> Fwiw, I did look at the most likely candidates: iproute2,
> network-manager and libvirt, and I see no signs that they're misusing
> NLM_F_APPEND in this way.

See above. I don't like this approach. IMO, it's not worth making it
slightly a bit easier for some user space programs to adopt when the
risk is breaking other programs and repeating this ordeal.

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Stefano Brivio]

On Wed, 3 Jun 2026 10:47:17 +0300
Ido Schimmel <idosch@nvidia.com> wrote:

> On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
> > On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:  
> > > On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:  
> > > > I get the impression there's a rough consensus that the best we can do
> > > > now is revert this change (already done), and make a new patch which
> > > > changes the insertion order to the "correct" one conditional on a new
> > > > flag.
> > > > 
> > > > Stefano has enough other fires to fight, so I'm taking a look at
> > > > implementing that.  Some initial thoughts, that I'm soliciting
> > > > feedback on:
> > > > 
> > > > 1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
> > > >    nlmsghdr
> > > > 
> > > > ifa_flags in ifaddrmsg would be the other candidate, but it looks like
> > > > it's encoding properties of the address itself, not about the action
> > > > of inserting it.  Plus all its bits are allocated, anyway.
> > > > 
> > > > 2) Could we re-use NLM_F_APPEND?
> > > > 
> > > > The short description of this existing flag in linux/uapi/netlink.h is
> > > > "Add to end of list" which sounds like the right thing.  Looking
> > > > closer, however, it seems like what is' used for so far is things
> > > > where the entity added with the NEW<whatever> operation is itself a
> > > > list, and NLM_F_APPEND causes it to be added to rather than replaced.
> > > > It's not used for addresses at present, AFAICT the list of addresses
> > > > is a semantic level above the address entity itself.
> > > > 
> > > > So maybe re-using it for the thing I tentatively called
> > > > NLM_F_INSERT_LAST would be confusing?
> > > > 
> > > > On the other hand, it's not used for addresses at the moment, so
> > > > AFAICT there's nothing actually preventing us reusing it for this
> > > > purpose. That would save a bit - we only have 2 general and 4 NEW
> > > > specific bits left, by the looks of it.  
> > > 
> > > This is not really viable. Even if the kernel is not using NLM_F_APPEND
> > > for RTM_NEWADDR, but not rejecting its presence either, then we can
> > > create a change in behavior for a user space that is currently setting
> > > it (intentionally or not).
> > >
> > > Example:
> > > 
> > > https://lore.kernel.org/netdev/27c249d80c346a258cfbf32f1d131ad4fe64e77c.camel@debian.org/  
> > 
> > Hmm.  So, in this example case we have a known, widely deployed
> > userspace that was broken by the change.  Similarly with the
> > original now-reverted "fix" for the ordering, we have a known, widely
> > deployed userspace that was broken.  
> 
> It was also reported over three years after the kernel change went in.
> Point is that we have no way of knowing how user space is using these
> flags. Suddenly giving them meaning when we simply ignored them before
> is risky.

I think that's a very different type of issue because, there, *another*
existing flag (NLM_F_EXCL) was suddenly given a meaning, as it happened
to have the same value as NLM_F_BULK, and that's what broke libvirt.
Not support for NLM_F_BULK itself.

Here, NLM_F_APPEND doesn't share its value with any other flag, and it
really is documented as "Add to end of list", but we don't do that.
That's a bug.

I think it's actually more likely that some bits of userspace are
currently broken and causing subtle issues because the author expected
NLM_F_APPEND to actually do what it promises, but maybe they only
tested that with IPv4.

Allow me to draw a parallel that looks more fitting to me: in commit 
1e47b4837f3b ("ipv6: Dump route exceptions if requested") I happened to
fix a two-year old issue that made 'ip -6 route list cache' show no
output and 'ip -6 route flush cache' have no effect.

You could take this to the extreme and say that it was risky to fix
that because some userspace application could meanwhile have started
relying on the fact that 'ip -6 route list cache' returned no output.
I guess we agree it was a good idea to fix that, though.

Of course there are several degrees of UAPI expectations in between,
but *not* allowing to use NLM_F_APPEND to append objects because
userspace might rely on NLM_F_APPEND to *not* append objects sounds
a bit like this extreme to me, or at least closer to it than the
NLM_F_BULK kind of breakage.

> > That's a different case from a hypothetical userspace that incorrectly
> > used NLM_F_APPEND on RTM_NEWADDR.  Moreover, to be broken it would
> > need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
> > the counterintuitive and inconsistent insertion order for IPv6
> > addresses.  Absent a concrete example of something meeting both those
> > conditions, I'm inclined to breaking that hypothetical case when the
> > payoff is an easier route to get known cases working with the
> > preferred insertion semantics.
> > 
> > Fwiw, I did look at the most likely candidates: iproute2,
> > network-manager and libvirt, and I see no signs that they're misusing
> > NLM_F_APPEND in this way.  
> 
> See above. I don't like this approach. IMO, it's not worth making it
> slightly a bit easier for some user space programs to adopt when the
> risk is breaking other programs and repeating this ordeal.

Another fact we shouldn't ignore is that, compared to the NLM_F_BULK
incident, we're actively surveying userspace before touching this.

-- 
Stefano

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 6750 bytes --]

On Wed, Jun 03, 2026 at 05:45:39PM +0200, Stefano Brivio wrote:
> On Wed, 3 Jun 2026 10:47:17 +0300
> Ido Schimmel <idosch@nvidia.com> wrote:
> 
> > On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
> > > On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:  
> > > > On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:  
> > > > > I get the impression there's a rough consensus that the best we can do
> > > > > now is revert this change (already done), and make a new patch which
> > > > > changes the insertion order to the "correct" one conditional on a new
> > > > > flag.
> > > > > 
> > > > > Stefano has enough other fires to fight, so I'm taking a look at
> > > > > implementing that.  Some initial thoughts, that I'm soliciting
> > > > > feedback on:
> > > > > 
> > > > > 1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
> > > > >    nlmsghdr
> > > > > 
> > > > > ifa_flags in ifaddrmsg would be the other candidate, but it looks like
> > > > > it's encoding properties of the address itself, not about the action
> > > > > of inserting it.  Plus all its bits are allocated, anyway.
> > > > > 
> > > > > 2) Could we re-use NLM_F_APPEND?
> > > > > 
> > > > > The short description of this existing flag in linux/uapi/netlink.h is
> > > > > "Add to end of list" which sounds like the right thing.  Looking
> > > > > closer, however, it seems like what is' used for so far is things
> > > > > where the entity added with the NEW<whatever> operation is itself a
> > > > > list, and NLM_F_APPEND causes it to be added to rather than replaced.
> > > > > It's not used for addresses at present, AFAICT the list of addresses
> > > > > is a semantic level above the address entity itself.
> > > > > 
> > > > > So maybe re-using it for the thing I tentatively called
> > > > > NLM_F_INSERT_LAST would be confusing?
> > > > > 
> > > > > On the other hand, it's not used for addresses at the moment, so
> > > > > AFAICT there's nothing actually preventing us reusing it for this
> > > > > purpose. That would save a bit - we only have 2 general and 4 NEW
> > > > > specific bits left, by the looks of it.  
> > > > 
> > > > This is not really viable. Even if the kernel is not using NLM_F_APPEND
> > > > for RTM_NEWADDR, but not rejecting its presence either, then we can
> > > > create a change in behavior for a user space that is currently setting
> > > > it (intentionally or not).
> > > >
> > > > Example:
> > > > 
> > > > https://lore.kernel.org/netdev/27c249d80c346a258cfbf32f1d131ad4fe64e77c.camel@debian.org/  
> > > 
> > > Hmm.  So, in this example case we have a known, widely deployed
> > > userspace that was broken by the change.  Similarly with the
> > > original now-reverted "fix" for the ordering, we have a known, widely
> > > deployed userspace that was broken.  
> > 
> > It was also reported over three years after the kernel change went in.
> > Point is that we have no way of knowing how user space is using these
> > flags. Suddenly giving them meaning when we simply ignored them before
> > is risky.
> 
> I think that's a very different type of issue because, there, *another*
> existing flag (NLM_F_EXCL) was suddenly given a meaning, as it happened
> to have the same value as NLM_F_BULK, and that's what broke libvirt.
> Not support for NLM_F_BULK itself.
> 
> Here, NLM_F_APPEND doesn't share its value with any other flag, and it
> really is documented as "Add to end of list", but we don't do that.
> That's a bug.

Eh.. that's the short description in the header.  But looking at how
it's actually used it generally means "append as opposed to replace"
(which is not relevant for addresses) rather than "append as opposed
to prepend".  So in that sense we would be assigning a new meaning.

> I think it's actually more likely that some bits of userspace are
> currently broken and causing subtle issues because the author expected
> NLM_F_APPEND to actually do what it promises, but maybe they only
> tested that with IPv4.

That's possible, although I'd guess far less likely that simply
expecting insert last behaviour without any extra flag.

> Allow me to draw a parallel that looks more fitting to me: in commit 
> 1e47b4837f3b ("ipv6: Dump route exceptions if requested") I happened to
> fix a two-year old issue that made 'ip -6 route list cache' show no
> output and 'ip -6 route flush cache' have no effect.
> 
> You could take this to the extreme and say that it was risky to fix
> that because some userspace application could meanwhile have started
> relying on the fact that 'ip -6 route list cache' returned no output.
> I guess we agree it was a good idea to fix that, though.
> 
> Of course there are several degrees of UAPI expectations in between,
> but *not* allowing to use NLM_F_APPEND to append objects because
> userspace might rely on NLM_F_APPEND to *not* append objects sounds
> a bit like this extreme to me, or at least closer to it than the
> NLM_F_BULK kind of breakage.
> 
> > > That's a different case from a hypothetical userspace that incorrectly
> > > used NLM_F_APPEND on RTM_NEWADDR.  Moreover, to be broken it would
> > > need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
> > > the counterintuitive and inconsistent insertion order for IPv6
> > > addresses.  Absent a concrete example of something meeting both those
> > > conditions, I'm inclined to breaking that hypothetical case when the
> > > payoff is an easier route to get known cases working with the
> > > preferred insertion semantics.
> > > 
> > > Fwiw, I did look at the most likely candidates: iproute2,
> > > network-manager and libvirt, and I see no signs that they're misusing
> > > NLM_F_APPEND in this way.  
> > 
> > See above. I don't like this approach. IMO, it's not worth making it
> > slightly a bit easier for some user space programs to adopt when the
> > risk is breaking other programs and repeating this ordeal.
> 
> Another fact we shouldn't ignore is that, compared to the NLM_F_BULK
> incident, we're actively surveying userspace before touching this.

So, I second Stefano's arguments for the most part, as well as
re-iterating that being broken by this change would require the
intersection of two unlikely conditions (misusing NLM_F_APPEND *and*
expecting the "wrong" order).

That said, Ido, if you're still not convinced I can do this as an
attribute.  It's more hassle, but I can make it work.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Ido Schimmel]

On Thu, Jun 04, 2026 at 11:26:08AM +1000, David Gibson wrote:
> So, I second Stefano's arguments for the most part, as well as
> re-iterating that being broken by this change would require the
> intersection of two unlikely conditions (misusing NLM_F_APPEND *and*
> expecting the "wrong" order).
> 
> That said, Ido, if you're still not convinced I can do this as an
> attribute.  It's more hassle, but I can make it work.

I appreciate the survey that Stefano and you conducted, but there is
still a non-zero chance of causing regressions by suddenly giving
NLM_F_APPEND a meaning in RTM_NEWADDR. We already tried the
"change-and-see-what-happens" methodology once with this feature and it
backfired, so it's going to be quite painful if we miss again.

As I see it, we have three options:

1. Use NLM_F_APPEND. Relatively easy change in both the kernel and user
space, but at the risk of reintroducing regressions.

2. Add a new attribute (e.g., IFA_INSERT_MODE with DEFAULT/APPEND
options). Less risky than #1, at the cost of a bit more code in both the
kernel and user space.

3. Do nothing. As I understand it, any production software (as opposed
to a test script) that cares about the in-scope order will have to
maintain a fallback anyway (e.g., iterating over IPv6 addresses in
reverse). Therefore, the changes in #1 and #2 are not strictly
necessary, yet they are uAPI that the kernel will have to maintain
forever.

Given the above, my preference would be #3 -> #2 -> #1. The first two
options expose the same capability to user space, so #1 doesn't buy us
anything over #2, except a bit less code, but we risk introducing a
regression.

Between #2 and #3, production software can't drop the fallback even if
we implement #2, yet #2 requires us to maintain uAPI forever. I think we
should accept that the divergence between IPv4 and IPv6 is not ideal,
but at least it's predictable and dependable (Fernando is working on a
ksft and documentation).

That being said, you can send an RFC for #1 and see what others think
since at this point it's unclear who is still following the thread.

Thanks

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Jakub Kicinski]

On Thu, 4 Jun 2026 21:39:09 +0300 Ido Schimmel wrote:
> Given the above, my preference would be #3 -> #2 -> #1. The first two
> options expose the same capability to user space, so #1 doesn't buy us
> anything over #2, except a bit less code, but we risk introducing a
> regression.
> 
> Between #2 and #3, production software can't drop the fallback even if
> we implement #2, yet #2 requires us to maintain uAPI forever. I think we
> should accept that the divergence between IPv4 and IPv6 is not ideal,
> but at least it's predictable and dependable (Fernando is working on a
> ksft and documentation).

+1 to this, FWIW

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Stefano Brivio]

On Thu, 4 Jun 2026 21:39:09 +0300
Ido Schimmel <idosch@nvidia.com> wrote:

> On Thu, Jun 04, 2026 at 11:26:08AM +1000, David Gibson wrote:
> > So, I second Stefano's arguments for the most part, as well as
> > re-iterating that being broken by this change would require the
> > intersection of two unlikely conditions (misusing NLM_F_APPEND *and*
> > expecting the "wrong" order).
> > 
> > That said, Ido, if you're still not convinced I can do this as an
> > attribute.  It's more hassle, but I can make it work.  
> 
> I appreciate the survey that Stefano and you conducted, but there is
> still a non-zero chance of causing regressions by suddenly giving
> NLM_F_APPEND a meaning in RTM_NEWADDR. We already tried the
> "change-and-see-what-happens" methodology once with this feature and it
> backfired, so it's going to be quite painful if we miss again.
> 
> As I see it, we have three options:

Thanks for the helpful summary by the way.

> 1. Use NLM_F_APPEND. Relatively easy change in both the kernel and user
> space, but at the risk of reintroducing regressions.
> 
> 2. Add a new attribute (e.g., IFA_INSERT_MODE with DEFAULT/APPEND
> options). Less risky than #1, at the cost of a bit more code in both the
> kernel and user space.
> 
> 3. Do nothing. As I understand it, any production software (as opposed
> to a test script) that cares about the in-scope order will have to
> maintain a fallback anyway (e.g., iterating over IPv6 addresses in
> reverse).

It's not always enough, or needed, or practical, though:

a. 'ip address restore' could be "fixed" to load the addresses in a
   reversed order for IPv6, but what should 'ip address showdump' do at
   that point?

   Also reverse the order? Or not, because that's the order addresses
   were dumped in? I'm fairly sure 'ip address save' shouldn't reverse
   it. It would be more convenient for the other operations, but
   definitely wrong, because the kernel is picking addresses in the
   opposite order.

   All these are doable, some look questionable, but any of the possible
   workarounds looks hard to document, or even remember.

b. as to pasta(1) and passt(1): the regression introduced by the revert
   of the kernel change isn't a big one: after all, we had the right
   behaviour for just a few months.

   Similarly, if the kernel gives us a way to get it right, we would
   just stick to it for the future and be done with it. Adding a
   workaround for older kernels isn't a priority because there was no
   regression.

   Side note: the workaround would be rather impractical for us because
   we don't use dynamic memory allocation, but we certainly can't blame
   the kernel or anybody else for that.

c. regardless of how addresses are dumped to userspace, there would be
   no way to get the kernel to do what one reasonably expects it to do
   (also from established IPv4 practice): *use* addresses in order of
   insertion

By the way of c., you're explicitly excluding test scripts as they are
certainly less important, but does it really make sense to force people
to to maintain the kind of workaround Matthieu mentioned? Reporting it
here for convenience:

  https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8

> Therefore, the changes in #1 and #2 are not strictly
> necessary, yet they are uAPI that the kernel will have to maintain
> forever.
> 
> Given the above, my preference would be #3 -> #2 -> #1. The first two
> options expose the same capability to user space, so #1 doesn't buy us
> anything over #2, except a bit less code, but we risk introducing a
> regression.
> 
> Between #2 and #3, production software can't drop the fallback even if
> we implement #2, yet #2 requires us to maintain uAPI forever. I think we
> should accept that the divergence between IPv4 and IPv6 is not ideal,
> but at least it's predictable and dependable (Fernando is working on a
> ksft and documentation).
> 
> That being said, you can send an RFC for #1 and see what others think
> since at this point it's unclear who is still following the thread.

I think there's general consensus against #1 ("abusing" NLM_F_APPEND),
but still, I think #2 is very much needed and fundamentally harmless.

-- 
Stefano

Re: IPv6 address insertion order (was Re: [PATCH net v2] Revert "ipv6: preserve insertion order for same-scope addresses")

[Nicolas Dichtel]

Le 03/06/2026 à 09:47, Ido Schimmel a écrit :
> On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
>> On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
>>> On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:

[snip]

>>>> 2) Could we re-use NLM_F_APPEND?
>>>>
>>>> The short description of this existing flag in linux/uapi/netlink.h is
>>>> "Add to end of list" which sounds like the right thing.  Looking
>>>> closer, however, it seems like what is' used for so far is things
>>>> where the entity added with the NEW<whatever> operation is itself a
>>>> list, and NLM_F_APPEND causes it to be added to rather than replaced.
>>>> It's not used for addresses at present, AFAICT the list of addresses
>>>> is a semantic level above the address entity itself.
>>>>
>>>> So maybe re-using it for the thing I tentatively called
>>>> NLM_F_INSERT_LAST would be confusing?
>>>>
>>>> On the other hand, it's not used for addresses at the moment, so
>>>> AFAICT there's nothing actually preventing us reusing it for this
>>>> purpose. That would save a bit - we only have 2 general and 4 NEW
>>>> specific bits left, by the looks of it.
>>>
>>> This is not really viable. Even if the kernel is not using NLM_F_APPEND
>>> for RTM_NEWADDR, but not rejecting its presence either, then we can
>>> create a change in behavior for a user space that is currently setting
>>> it (intentionally or not).
>>>
>>> Example:
>>>
>>> https://lore.kernel.org/netdev/27c249d80c346a258cfbf32f1d131ad4fe64e77c.camel@debian.org/
>>
>> Hmm.  So, in this example case we have a known, widely deployed
>> userspace that was broken by the change.  Similarly with the
>> original now-reverted "fix" for the ordering, we have a known, widely
>> deployed userspace that was broken.
> 
> It was also reported over three years after the kernel change went in.
> Point is that we have no way of knowing how user space is using these
> flags. Suddenly giving them meaning when we simply ignored them before
> is risky.
+1