Problem with IPv6 privacy addresses in 7.0

Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

My normal use of my desktop system (Fedora 43) is to suspend at night
and wake in the morning, and then immediately SSH to my remote server,
where I stay logged in.  I use NetworkManager and have ipv6.ip6-privacy
set to prefer-temp-addr, and SSH configured to use SSH keepalives.

When I upgraded to kernel 7.0, I started having an issue with this - my
SSH session gets dropped, usually several times, in the first hour or
so, then I usually don't have any problem the rest of the day.  If I run
an IPv4 session at the same time, that seems to be fine, so I don't
think it's a network issue (I know that dual-stack doesn't always take
the same path).

What seems to be happening is that privacy addresses are removed while
the SSH session is still using them, before the timeout even.  I wrote a
script to watch public v6 addresses being added and removed, and this is
what I've seen so far today (the number at the end is the valid_lft
seconds), with the public prefix masked:

2026-05-21 07:39:17 removed xx::f4f/128 4673
2026-05-21 07:39:20 added xx::f4f/128 5400
2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120

I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d
source address.  That dropped in 2 minutes and I reconnected, which used
the 1d8f address.  That dropped at 08:14:43, I didn't notice right away,
I reconnected at 08:31:28 which used the 94fd address.

-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Thu, 21 May 2026 08:53:10 -0500 Chris Adams wrote:
> My normal use of my desktop system (Fedora 43) is to suspend at night
> and wake in the morning, and then immediately SSH to my remote server,
> where I stay logged in.  I use NetworkManager and have ipv6.ip6-privacy
> set to prefer-temp-addr, and SSH configured to use SSH keepalives.
> 
> When I upgraded to kernel 7.0, I started having an issue with this - my
> SSH session gets dropped, usually several times, in the first hour or
> so, then I usually don't have any problem the rest of the day.  If I run
> an IPv4 session at the same time, that seems to be fine, so I don't
> think it's a network issue (I know that dual-stack doesn't always take
> the same path).
> 
> What seems to be happening is that privacy addresses are removed while
> the SSH session is still using them, before the timeout even.  I wrote a
> script to watch public v6 addresses being added and removed, and this is
> what I've seen so far today (the number at the end is the valid_lft
> seconds), with the public prefix masked:
> 
> 2026-05-21 07:39:17 removed xx::f4f/128 4673
> 2026-05-21 07:39:20 added xx::f4f/128 5400
> 2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
> 2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
> 2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
> 2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
> 2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
> 2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
> 2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
> 2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120
> 
> I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d
> source address.  That dropped in 2 minutes and I reconnected, which used
> the 1d8f address.  That dropped at 08:14:43, I didn't notice right away,
> I reconnected at 08:31:28 which used the 94fd address.

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
or 6.19?

Would you be able to try testing with some commits reverted?
On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
c7dc5b522882 ("ipv6: clean up routes when manually removing address with a lifetime")

Less likely:
5023479627e3 ("ipv6: Switch to higher-level SHA-1 functions")
9e371b0ba7f5 ("ipv6: addrconf: reduce default temp_valid_lft to 2 days")
6af51e9f3133 ("ipv6: Remove permanent routes from tb6_gc_hlist when all exceptions expire.")

Re: Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

Heh, I opened a Fedora BZ on this and JUST updated it... RHBZ 2480928

Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
> On Thu, 21 May 2026 08:53:10 -0500 Chris Adams wrote:
> > My normal use of my desktop system (Fedora 43) is to suspend at night
> > and wake in the morning, and then immediately SSH to my remote server,
> > where I stay logged in.  I use NetworkManager and have ipv6.ip6-privacy
> > set to prefer-temp-addr, and SSH configured to use SSH keepalives.
> > 
> > When I upgraded to kernel 7.0, I started having an issue with this - my
> > SSH session gets dropped, usually several times, in the first hour or
> > so, then I usually don't have any problem the rest of the day.  If I run
> > an IPv4 session at the same time, that seems to be fine, so I don't
> > think it's a network issue (I know that dual-stack doesn't always take
> > the same path).
> > 
> > What seems to be happening is that privacy addresses are removed while
> > the SSH session is still using them, before the timeout even.  I wrote a
> > script to watch public v6 addresses being added and removed, and this is
> > what I've seen so far today (the number at the end is the valid_lft
> > seconds), with the public prefix masked:
> > 
> > 2026-05-21 07:39:17 removed xx::f4f/128 4673
> > 2026-05-21 07:39:20 added xx::f4f/128 5400
> > 2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
> > 2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
> > 2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
> > 2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
> > 2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
> > 2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
> > 2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
> > 2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120
> > 
> > I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d
> > source address.  That dropped in 2 minutes and I reconnected, which used
> > the 1d8f address.  That dropped at 08:14:43, I didn't notice right away,
> > I reconnected at 08:31:28 which used the 94fd address.
> 
> Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> or 6.19?

It was 6.19 to 7.0.

> Would you be able to try testing with some commits reverted?
> On a quick look the candidates would be:
> 
> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

I figured out that it happens after stopping a VM (and I usually
start/stop a VM for a bit in the morning, which is why it happened more
than once).  So I set up a VM with a nested VM, running up-to-date
Fedora 44, and then was able to bisect pretty easily, and it landed on
this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy
address management (right?).  NM didn't change, so maybe this commit is
confusing something in NM?

-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
> > Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> > or 6.19?  
> 
> It was 6.19 to 7.0.
> 
> > Would you be able to try testing with some commits reverted?
> > On a quick look the candidates would be:
> > 
> > cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> 
> It's this one.

Phew, the second one was mine :)

> I figured out that it happens after stopping a VM (and I usually
> start/stop a VM for a bit in the morning, which is why it happened more
> than once).  So I set up a VM with a nested VM, running up-to-date
> Fedora 44, and then was able to bisect pretty easily, and it landed on
> this commit.
> 
> Fedora is using NetworkManager, and IIRC NM does some part of privacy
> address management (right?).  NM didn't change, so maybe this commit is
> confusing something in NM?

Sounds plausible, pretty sure we knew this commit was risky to begin
with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able
to send the revert with your problem description and a Fixes tag
pointing to the reverted commit?

Re: Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
> > > Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> > > or 6.19?  
> > 
> > It was 6.19 to 7.0.
> > 
> > > Would you be able to try testing with some commits reverted?
> > > On a quick look the candidates would be:
> > > 
> > > cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> > 
> > It's this one.
> 
> Phew, the second one was mine :)
> 
> > I figured out that it happens after stopping a VM (and I usually
> > start/stop a VM for a bit in the morning, which is why it happened more
> > than once).  So I set up a VM with a nested VM, running up-to-date
> > Fedora 44, and then was able to bisect pretty easily, and it landed on
> > this commit.
> > 
> > Fedora is using NetworkManager, and IIRC NM does some part of privacy
> > address management (right?).  NM didn't change, so maybe this commit is
> > confusing something in NM?
> 
> Sounds plausible, pretty sure we knew this commit was risky to begin
> with, but we had no direct proof that it'd break real life users.
> 
> Revert is the right course of action here. Would you be willing/able
> to send the revert with your problem description and a Fixes tag
> pointing to the reverted commit?

So... I'm not very git-fluent and haven't submitted a kernel patch in
many years, so really am not sure of the necessary/proper steps.
-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Fernando Fernandez Mancera]

On 5/27/26 11:13 PM, Chris Adams wrote:
> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
>>>> or 6.19?
>>>
>>> It was 6.19 to 7.0.
>>>
>>>> Would you be able to try testing with some commits reverted?
>>>> On a quick look the candidates would be:
>>>>
>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>>>
>>> It's this one.
>>
>> Phew, the second one was mine :)
>>
>>> I figured out that it happens after stopping a VM (and I usually
>>> start/stop a VM for a bit in the morning, which is why it happened more
>>> than once).  So I set up a VM with a nested VM, running up-to-date
>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
>>> this commit.
>>>
>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
>>> address management (right?).  NM didn't change, so maybe this commit is
>>> confusing something in NM?
>>
>> Sounds plausible, pretty sure we knew this commit was risky to begin
>> with, but we had no direct proof that it'd break real life users.
>>
>> Revert is the right course of action here. Would you be willing/able
>> to send the revert with your problem description and a Fixes tag
>> pointing to the reverted commit?
> 
> So... I'm not very git-fluent and haven't submitted a kernel patch in
> many years, so really am not sure of the necessary/proper steps.

I don't mind to handle it. I will CC you, is that fine for you?

Thanks for the report btw!

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Wed, 27 May 2026 16:13:56 -0500 Chris Adams wrote:
> > > I figured out that it happens after stopping a VM (and I usually
> > > start/stop a VM for a bit in the morning, which is why it happened more
> > > than once).  So I set up a VM with a nested VM, running up-to-date
> > > Fedora 44, and then was able to bisect pretty easily, and it landed on
> > > this commit.
> > > 
> > > Fedora is using NetworkManager, and IIRC NM does some part of privacy
> > > address management (right?).  NM didn't change, so maybe this commit is
> > > confusing something in NM?  
> > 
> > Sounds plausible, pretty sure we knew this commit was risky to begin
> > with, but we had no direct proof that it'd break real life users.
> > 
> > Revert is the right course of action here. Would you be willing/able
> > to send the revert with your problem description and a Fixes tag
> > pointing to the reverted commit?  
> 
> So... I'm not very git-fluent and haven't submitted a kernel patch in
> many years, so really am not sure of the necessary/proper steps.

I wonder how you managed to bisect it then :)

git clone ..
git revert cb3de96eea66

you can use db5dadb562cabb6 for inspiration on how to format 
the commit message. You can use an LLM to help, they are plenty
capable enough to handle a revert. For sending either git send-email
or you can use b4 relay. https://b4.docs.kernel.org/en/latest/

If you prefer us to handle this - that's perfectly fine, just let me
know. I'm asking if you _want_ to handle it, as some folks like the
sense of accomplishment of having upstream commits.

Re: Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
> > > Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> > > or 6.19?  
> > 
> > It was 6.19 to 7.0.
> > 
> > > Would you be able to try testing with some commits reverted?
> > > On a quick look the candidates would be:
> > > 
> > > cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> > 
> > It's this one.
> 
> Phew, the second one was mine :)
> 
> > I figured out that it happens after stopping a VM (and I usually
> > start/stop a VM for a bit in the morning, which is why it happened more
> > than once).  So I set up a VM with a nested VM, running up-to-date
> > Fedora 44, and then was able to bisect pretty easily, and it landed on
> > this commit.
> > 
> > Fedora is using NetworkManager, and IIRC NM does some part of privacy
> > address management (right?).  NM didn't change, so maybe this commit is
> > confusing something in NM?
> 
> Sounds plausible, pretty sure we knew this commit was risky to begin
> with, but we had no direct proof that it'd break real life users.
> 
> Revert is the right course of action here. Would you be willing/able
> to send the revert with your problem description and a Fixes tag
> pointing to the reverted commit?

I did want to add a little more test note:

It's definitely an interaction with NetworkManager.  If I stop NM and
run my VM start/stop test, nothing unexpected happens after.  If NM is
running and I do my VM test, when the next router advertisement is
received, NM replaces the privacy addresses.

-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Fernando Fernandez Mancera]

On 5/27/26 11:51 PM, Chris Adams wrote:
> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
>>>> or 6.19?
>>>
>>> It was 6.19 to 7.0.
>>>
>>>> Would you be able to try testing with some commits reverted?
>>>> On a quick look the candidates would be:
>>>>
>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>>>
>>> It's this one.
>>
>> Phew, the second one was mine :)
>>
>>> I figured out that it happens after stopping a VM (and I usually
>>> start/stop a VM for a bit in the morning, which is why it happened more
>>> than once).  So I set up a VM with a nested VM, running up-to-date
>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
>>> this commit.
>>>
>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
>>> address management (right?).  NM didn't change, so maybe this commit is
>>> confusing something in NM?
>>
>> Sounds plausible, pretty sure we knew this commit was risky to begin
>> with, but we had no direct proof that it'd break real life users.
>>
>> Revert is the right course of action here. Would you be willing/able
>> to send the revert with your problem description and a Fixes tag
>> pointing to the reverted commit?
> 
> I did want to add a little more test note:
> 
> It's definitely an interaction with NetworkManager.  If I stop NM and
> run my VM start/stop test, nothing unexpected happens after.  If NM is
> running and I do my VM test, when the next router advertisement is
> received, NM replaces the privacy addresses.
> 

As someone that is experienced in NetworkManager, I can confirm it is 
related. NetworkManager is querying the IPv6 address and when the 
connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), 
NetworkManager creates a route to make the system use the temporary 
address for outgoing connections by default.

If the order is messed up, the address picked will likely be too. One 
could argue that this is partially fault of NetworkManager and that it 
should check the timestamps or preferred times rather than order.. but 
well, the rule is "do not break userspace".

I hope this clarifies things.

Thanks,
Fernando.

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera wrote:
> On 5/27/26 11:51 PM, Chris Adams wrote:
> > Once upon a time, Jakub Kicinski <kuba@kernel.org> said:  
> >> Sounds plausible, pretty sure we knew this commit was risky to begin
> >> with, but we had no direct proof that it'd break real life users.
> >>
> >> Revert is the right course of action here. Would you be willing/able
> >> to send the revert with your problem description and a Fixes tag
> >> pointing to the reverted commit?  
> > 
> > I did want to add a little more test note:
> > 
> > It's definitely an interaction with NetworkManager.  If I stop NM and
> > run my VM start/stop test, nothing unexpected happens after.  If NM is
> > running and I do my VM test, when the next router advertisement is
> > received, NM replaces the privacy addresses.
> >   
> 
> As someone that is experienced in NetworkManager, I can confirm it is 
> related. NetworkManager is querying the IPv6 address and when the 
> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), 
> NetworkManager creates a route to make the system use the temporary 
> address for outgoing connections by default.
> 
> If the order is messed up, the address picked will likely be too. One 
> could argue that this is partially fault of NetworkManager and that it 
> should check the timestamps or preferred times rather than order.. but 
> well, the rule is "do not break userspace".

Right, plus the justification for the change wasn't very strong 
in the first place. My internal compass is still pointing towards
a revert.

> I hope this clarifies things.

Re: Problem with IPv6 privacy addresses in 7.0

[Stefano Brivio]

Hi Fernando, Jakub,

On Wed, 27 May 2026 23:59:47 +0200
Fernando Fernandez Mancera <fmancera@suse.de> wrote:

> On 5/27/26 11:51 PM, Chris Adams wrote:
> > Once upon a time, Jakub Kicinski <kuba@kernel.org> said:  
> >> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:  
> >>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
> >>>> or 6.19?  
> >>>
> >>> It was 6.19 to 7.0.
> >>>  
> >>>> Would you be able to try testing with some commits reverted?
> >>>> On a quick look the candidates would be:
> >>>>
> >>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> >>>
> >>> It's this one.  
> >>
> >> Phew, the second one was mine :)
> >>  
> >>> I figured out that it happens after stopping a VM (and I usually
> >>> start/stop a VM for a bit in the morning, which is why it happened more
> >>> than once).  So I set up a VM with a nested VM, running up-to-date
> >>> Fedora 44, and then was able to bisect pretty easily, and it landed on
> >>> this commit.
> >>>
> >>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
> >>> address management (right?).  NM didn't change, so maybe this commit is
> >>> confusing something in NM?  
> >>
> >> Sounds plausible, pretty sure we knew this commit was risky to begin
> >> with, but we had no direct proof that it'd break real life users.
> >>
> >> Revert is the right course of action here. Would you be willing/able
> >> to send the revert with your problem description and a Fixes tag
> >> pointing to the reverted commit?  
> > 
> > I did want to add a little more test note:
> > 
> > It's definitely an interaction with NetworkManager.  If I stop NM and
> > run my VM start/stop test, nothing unexpected happens after.  If NM is
> > running and I do my VM test, when the next router advertisement is
> > received, NM replaces the privacy addresses.
> >   
> 
> As someone that is experienced in NetworkManager, I can confirm it is 
> related. NetworkManager is querying the IPv6 address and when the 
> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), 
> NetworkManager creates a route to make the system use the temporary 
> address for outgoing connections by default.
> 
> If the order is messed up, the address picked will likely be too. One 
> could argue that this is partially fault of NetworkManager and that it 
> should check the timestamps or preferred times rather than order.. but 
> well, the rule is "do not break userspace".
> 
> I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose
of that commit is to *preserve* the order of addresses as they were
inserted, not to mess it up.

Before that, addresses were stored and returned via netlink *reversed*
compared to the insertion, which is rather surprising, also because
it's the opposite of what we do for IPv4 addresses, and caused issues
with pasta(1) as it copies addresses into containers in the same order
as reported via netlink.

Do you happen to know exactly in which way the order happens to be wrong
now?

Also note that userspace was broken and fixed a couple of times:

  https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too
long and this is causing issues for a lot of people meanwhile of course
it might make sense to revert, but I'd give it a quick try at least.

-- 
Stefano

Re: Problem with IPv6 privacy addresses in 7.0

[Fernando Fernandez Mancera]

On 5/28/26 7:38 AM, Stefano Brivio wrote:
> Hi Fernando, Jakub,
> 
> On Wed, 27 May 2026 23:59:47 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
>> On 5/27/26 11:51 PM, Chris Adams wrote:
>>> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
>>>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
>>>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
>>>>>> or 6.19?
>>>>>
>>>>> It was 6.19 to 7.0.
>>>>>   
>>>>>> Would you be able to try testing with some commits reverted?
>>>>>> On a quick look the candidates would be:
>>>>>>
>>>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>>>>>
>>>>> It's this one.
>>>>
>>>> Phew, the second one was mine :)
>>>>   
>>>>> I figured out that it happens after stopping a VM (and I usually
>>>>> start/stop a VM for a bit in the morning, which is why it happened more
>>>>> than once).  So I set up a VM with a nested VM, running up-to-date
>>>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
>>>>> this commit.
>>>>>
>>>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
>>>>> address management (right?).  NM didn't change, so maybe this commit is
>>>>> confusing something in NM?
>>>>
>>>> Sounds plausible, pretty sure we knew this commit was risky to begin
>>>> with, but we had no direct proof that it'd break real life users.
>>>>
>>>> Revert is the right course of action here. Would you be willing/able
>>>> to send the revert with your problem description and a Fixes tag
>>>> pointing to the reverted commit?
>>>
>>> I did want to add a little more test note:
>>>
>>> It's definitely an interaction with NetworkManager.  If I stop NM and
>>> run my VM start/stop test, nothing unexpected happens after.  If NM is
>>> running and I do my VM test, when the next router advertisement is
>>> received, NM replaces the privacy addresses.
>>>    
>>
>> As someone that is experienced in NetworkManager, I can confirm it is
>> related. NetworkManager is querying the IPv6 address and when the
>> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr),
>> NetworkManager creates a route to make the system use the temporary
>> address for outgoing connections by default.
>>
>> If the order is messed up, the address picked will likely be too. One
>> could argue that this is partially fault of NetworkManager and that it
>> should check the timestamps or preferred times rather than order.. but
>> well, the rule is "do not break userspace".
>>
>> I hope this clarifies things.
> 
> Not entirely. I'm looking into this right now, but note that the purpose
> of that commit is to *preserve* the order of addresses as they were
> inserted, not to mess it up.
> 
> Before that, addresses were stored and returned via netlink *reversed*
> compared to the insertion, which is rather surprising, also because
> it's the opposite of what we do for IPv4 addresses, and caused issues
> with pasta(1) as it copies addresses into containers in the same order
> as reported via netlink.
> 

Right, to make it clear. I agree with you here. The current order is the 
*right* or rather intuitive order.

> Do you happen to know exactly in which way the order happens to be wrong
> now?
> 

It is not wrong, but userspace applications were parsing addresses and 
expecting them in the previous order whether it was correct or not. Now, 
that parsing is broken because they are coming in an unexpected (maybe 
more intuitive) order.

FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a 
NetworkManager test and they were going to report it.

> Also note that userspace was broken and fixed a couple of times:
> 
>    https://bugs.passt.top/show_bug.cgi?id=175#c10
> 
> so I would look into fixing that properly if doable. If it takes too
> long and this is causing issues for a lot of people meanwhile of course
> it might make sense to revert, but I'd give it a quick try at least.
> 

I think the solution shouldn't modify the order of the list by default. 
Maybe we can introduce a flag for the dump operation?

Re: Problem with IPv6 privacy addresses in 7.0

[Stefano Brivio]

On Thu, 28 May 2026 12:46:05 +0200
Fernando Fernandez Mancera <fmancera@suse.de> wrote:

> On 5/28/26 7:38 AM, Stefano Brivio wrote:
> > Hi Fernando, Jakub,
> > 
> > On Wed, 27 May 2026 23:59:47 +0200
> > Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> >   
> >> On 5/27/26 11:51 PM, Chris Adams wrote:  
> >>> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:  
> >>>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:  
> >>>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
> >>>>>> or 6.19?  
> >>>>>
> >>>>> It was 6.19 to 7.0.
> >>>>>     
> >>>>>> Would you be able to try testing with some commits reverted?
> >>>>>> On a quick look the candidates would be:
> >>>>>>
> >>>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> >>>>>
> >>>>> It's this one.  
> >>>>
> >>>> Phew, the second one was mine :)
> >>>>     
> >>>>> I figured out that it happens after stopping a VM (and I usually
> >>>>> start/stop a VM for a bit in the morning, which is why it happened more
> >>>>> than once).  So I set up a VM with a nested VM, running up-to-date
> >>>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
> >>>>> this commit.
> >>>>>
> >>>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
> >>>>> address management (right?).  NM didn't change, so maybe this commit is
> >>>>> confusing something in NM?  
> >>>>
> >>>> Sounds plausible, pretty sure we knew this commit was risky to begin
> >>>> with, but we had no direct proof that it'd break real life users.
> >>>>
> >>>> Revert is the right course of action here. Would you be willing/able
> >>>> to send the revert with your problem description and a Fixes tag
> >>>> pointing to the reverted commit?  
> >>>
> >>> I did want to add a little more test note:
> >>>
> >>> It's definitely an interaction with NetworkManager.  If I stop NM and
> >>> run my VM start/stop test, nothing unexpected happens after.  If NM is
> >>> running and I do my VM test, when the next router advertisement is
> >>> received, NM replaces the privacy addresses.
> >>>      
> >>
> >> As someone that is experienced in NetworkManager, I can confirm it is
> >> related. NetworkManager is querying the IPv6 address and when the
> >> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr),
> >> NetworkManager creates a route to make the system use the temporary
> >> address for outgoing connections by default.
> >>
> >> If the order is messed up, the address picked will likely be too. One
> >> could argue that this is partially fault of NetworkManager and that it
> >> should check the timestamps or preferred times rather than order.. but
> >> well, the rule is "do not break userspace".
> >>
> >> I hope this clarifies things.  
> > 
> > Not entirely. I'm looking into this right now, but note that the purpose
> > of that commit is to *preserve* the order of addresses as they were
> > inserted, not to mess it up.
> > 
> > Before that, addresses were stored and returned via netlink *reversed*
> > compared to the insertion, which is rather surprising, also because
> > it's the opposite of what we do for IPv4 addresses, and caused issues
> > with pasta(1) as it copies addresses into containers in the same order
> > as reported via netlink.
> 
> Right, to make it clear. I agree with you here. The current order is the 
> *right* or rather intuitive order.
> 
> > Do you happen to know exactly in which way the order happens to be wrong
> > now?
> 
> It is not wrong, but userspace applications were parsing addresses and 
> expecting them in the previous order whether it was correct or not. Now, 
> that parsing is broken because they are coming in an unexpected (maybe 
> more intuitive) order.

Ouch, I see.

Two questions:

- could there be other issues coming from the fact that NetworkManager
  relies on a particular order (whether right or wrong) of addresses? If
  yes, I guess NetworkManager would benefit from a fix in any case
  (that is, we broke it a bit harder, but maybe it was already broken?)

- regardless of this, would there be a way to make NetworkManager
  robust to the order?

  I'm asking this because yes, strictly speaking, you might say this
  change breaks userspace (NetworkManager), but at the same time it
  fixes another part of it (pasta), and changing it back would break
  pasta again (in a way we can't really fix).

  Now, I suppose NetworkManager is much more universally used and it's
  definitely a more mature project so it's a bigger userspace breakage,
  but at the same time it's a ton of containers (Podman uses pasta by
  default, Docker/moby optionally via rootlesskit) we might introduce a
  regression for.

So I'm wondering if in practice it wouldn't be better to fix
NetworkManager instead.

> FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a 
> NetworkManager test and they were going to report it.
> 
> > Also note that userspace was broken and fixed a couple of times:
> > 
> >    https://bugs.passt.top/show_bug.cgi?id=175#c10
> > 
> > so I would look into fixing that properly if doable. If it takes too
> > long and this is causing issues for a lot of people meanwhile of course
> > it might make sense to revert, but I'd give it a quick try at least.
> 
> I think the solution shouldn't modify the order of the list by default. 
> Maybe we can introduce a flag for the dump operation?

I'm a bit struggling with the name and how complicated a description
would be. NLM_F_RIGHT_ORDER? I guess you're suggesting to get the
"wrong" / reversed order by default... even though it's the opposite
compared to IPv4? Or NLM_F_REVERSED, otherwise? Is it really worth it?

-- 
Stefano

Re: Problem with IPv6 privacy addresses in 7.0

[Fernando Fernandez Mancera]

On 5/28/26 1:12 PM, Stefano Brivio wrote:
> On Thu, 28 May 2026 12:46:05 +0200
> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
> 
>> On 5/28/26 7:38 AM, Stefano Brivio wrote:
>>> Hi Fernando, Jakub,
>>>
>>> On Wed, 27 May 2026 23:59:47 +0200
>>> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
>>>    
>>>> On 5/27/26 11:51 PM, Chris Adams wrote:
>>>>> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
>>>>>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
>>>>>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
>>>>>>>> or 6.19?
>>>>>>>
>>>>>>> It was 6.19 to 7.0.
>>>>>>>      
>>>>>>>> Would you be able to try testing with some commits reverted?
>>>>>>>> On a quick look the candidates would be:
>>>>>>>>
>>>>>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>>>>>>>
>>>>>>> It's this one.
>>>>>>
>>>>>> Phew, the second one was mine :)
>>>>>>      
>>>>>>> I figured out that it happens after stopping a VM (and I usually
>>>>>>> start/stop a VM for a bit in the morning, which is why it happened more
>>>>>>> than once).  So I set up a VM with a nested VM, running up-to-date
>>>>>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
>>>>>>> this commit.
>>>>>>>
>>>>>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
>>>>>>> address management (right?).  NM didn't change, so maybe this commit is
>>>>>>> confusing something in NM?
>>>>>>
>>>>>> Sounds plausible, pretty sure we knew this commit was risky to begin
>>>>>> with, but we had no direct proof that it'd break real life users.
>>>>>>
>>>>>> Revert is the right course of action here. Would you be willing/able
>>>>>> to send the revert with your problem description and a Fixes tag
>>>>>> pointing to the reverted commit?
>>>>>
>>>>> I did want to add a little more test note:
>>>>>
>>>>> It's definitely an interaction with NetworkManager.  If I stop NM and
>>>>> run my VM start/stop test, nothing unexpected happens after.  If NM is
>>>>> running and I do my VM test, when the next router advertisement is
>>>>> received, NM replaces the privacy addresses.
>>>>>       
>>>>
>>>> As someone that is experienced in NetworkManager, I can confirm it is
>>>> related. NetworkManager is querying the IPv6 address and when the
>>>> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr),
>>>> NetworkManager creates a route to make the system use the temporary
>>>> address for outgoing connections by default.
>>>>
>>>> If the order is messed up, the address picked will likely be too. One
>>>> could argue that this is partially fault of NetworkManager and that it
>>>> should check the timestamps or preferred times rather than order.. but
>>>> well, the rule is "do not break userspace".
>>>>
>>>> I hope this clarifies things.
>>>
>>> Not entirely. I'm looking into this right now, but note that the purpose
>>> of that commit is to *preserve* the order of addresses as they were
>>> inserted, not to mess it up.
>>>
>>> Before that, addresses were stored and returned via netlink *reversed*
>>> compared to the insertion, which is rather surprising, also because
>>> it's the opposite of what we do for IPv4 addresses, and caused issues
>>> with pasta(1) as it copies addresses into containers in the same order
>>> as reported via netlink.
>>
>> Right, to make it clear. I agree with you here. The current order is the
>> *right* or rather intuitive order.
>>
>>> Do you happen to know exactly in which way the order happens to be wrong
>>> now?
>>
>> It is not wrong, but userspace applications were parsing addresses and
>> expecting them in the previous order whether it was correct or not. Now,
>> that parsing is broken because they are coming in an unexpected (maybe
>> more intuitive) order.
> 
> Ouch, I see.
> 
> Two questions:
> 
> - could there be other issues coming from the fact that NetworkManager
>    relies on a particular order (whether right or wrong) of addresses? If
>    yes, I guess NetworkManager would benefit from a fix in any case
>    (that is, we broke it a bit harder, but maybe it was already broken?)
> 

Yes, I believe the right way to pick a temporary address is to use the 
most recent one and the right way to see that is by looking to 
timestamps not looking at the order.

> - regardless of this, would there be a way to make NetworkManager
>    robust to the order?
> 
>    I'm asking this because yes, strictly speaking, you might say this
>    change breaks userspace (NetworkManager), but at the same time it
>    fixes another part of it (pasta), and changing it back would break
>    pasta again (in a way we can't really fix).
> 

Ugh, that is a mess.

>    Now, I suppose NetworkManager is much more universally used and it's
>    definitely a more mature project so it's a bigger userspace breakage,
>    but at the same time it's a ton of containers (Podman uses pasta by
>    default, Docker/moby optionally via rootlesskit) we might introduce a
>    regression for.
> 

I think the main argument here is: while we know it is affecting 
NetworkManager, we cannot know if it is breaking something else. The 
kernel rule is usually "do not break userspace" and this is likely 
considered a regression from my understanding. Cc'ing Thorsten as he is 
an expert on regressions.

> So I'm wondering if in practice it wouldn't be better to fix
> NetworkManager instead.
> 

We can fix future versions of NetworkManager but we cannot fix released 
versions. That is, if someone upgrade kernel but not NetworkManager, it 
would break. I guess the same applies to pasta.. so we must understand 
what has been the most prevalent behavior.

>> FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a
>> NetworkManager test and they were going to report it.
>>
>>> Also note that userspace was broken and fixed a couple of times:
>>>
>>>     https://bugs.passt.top/show_bug.cgi?id=175#c10
>>>
>>> so I would look into fixing that properly if doable. If it takes too
>>> long and this is causing issues for a lot of people meanwhile of course
>>> it might make sense to revert, but I'd give it a quick try at least.
>>
>> I think the solution shouldn't modify the order of the list by default.
>> Maybe we can introduce a flag for the dump operation?
> 
> I'm a bit struggling with the name and how complicated a description
> would be. NLM_F_RIGHT_ORDER? I guess you're suggesting to get the
> "wrong" / reversed order by default... even though it's the opposite
> compared to IPv4? Or NLM_F_REVERSED, otherwise? Is it really worth it?
> 

I don't have a clear idea on how to do it so it is acceptable, right 
now. FWIW; I think consistent behavior between IPv4 and IPv6 isn't a 
goal. They are designed in different ways and implemented in different 
ways too. There are many situations where they behave completely 
different e.g ECMP routing.

I have sent the revert patch but more likely for AI review + CI. I do 
not want it to be merged until we reach a consensus, we can discard it 
if we decide to do so.

Thanks,
Fernando.

Re: Problem with IPv6 privacy addresses in 7.0

[Thorsten Leemhuis]

On 5/28/26 13:29, Fernando Fernandez Mancera wrote:
> On 5/28/26 1:12 PM, Stefano Brivio wrote:
>> On Thu, 28 May 2026 12:46:05 +0200
>> Fernando Fernandez Mancera <fmancera@suse.de> wrote:
>>> On 5/28/26 7:38 AM, Stefano Brivio wrote:
>
>> - regardless of this, would there be a way to make NetworkManager
>>    robust to the order?
>>
>>    I'm asking this because yes, strictly speaking, you might say this
>>    change breaks userspace (NetworkManager), but at the same time it
>>    fixes another part of it (pasta), and changing it back would break
>>    pasta again (in a way we can't really fix).
>
> Ugh, that is a mess.

Well, yes, but one that happens regularly. :-D

The problem here is that it was reported only some months after the
culprit landed, as fixing the regression through a revert or so could
now cause a bigger regression. Do we assume this to be the case here? It
sounds like pasta would "only" be broken about as much as it was before
-- then a revert or something like that is the right solution to get
back to the status quo. Or does pasta already depend on the new behavior
somehow and would now break even more if we reverted the culprit? Then
it's really a mess. :-/

But there are ways to fix even this, it's just that most of them are
ugly. Like adding some bit somewhere to /proc/ or so that a fixed
NetworkManager (if it's the only affected app) could flip by default to
change the things from the old behavior to the new one; and one pasta
could check that bit and warn. Config options are also an option, but
that's even uglier.

>>    Now, I suppose NetworkManager is much more universally used and it's
>>    definitely a more mature project so it's a bigger userspace breakage,
>>    but at the same time it's a ton of containers (Podman uses pasta by
>>    default, Docker/moby optionally via rootlesskit) we might introduce a
>>    regression for.
> 
> I think the main argument here is: while we know it is affecting
> NetworkManager, we cannot know if it is breaking something else. The
> kernel rule is usually "do not break userspace" and this is likely
> considered a regression from my understanding. Cc'ing Thorsten as he is
> an expert on regressions.

From the mail at the top of the thread it definitely sounds like one.

Ciao, Thorsten

Re: Problem with IPv6 privacy addresses in 7.0

[Stefano Brivio]

On Thu, 28 May 2026 14:29:50 +0200
Thorsten Leemhuis <regressions@leemhuis.info> wrote:

> On 5/28/26 13:29, Fernando Fernandez Mancera wrote:
> > On 5/28/26 1:12 PM, Stefano Brivio wrote:  
> >> On Thu, 28 May 2026 12:46:05 +0200
> >> Fernando Fernandez Mancera <fmancera@suse.de> wrote:  
> >>> On 5/28/26 7:38 AM, Stefano Brivio wrote:  
> >  
> >> - regardless of this, would there be a way to make NetworkManager
> >>    robust to the order?
> >>
> >>    I'm asking this because yes, strictly speaking, you might say this
> >>    change breaks userspace (NetworkManager), but at the same time it
> >>    fixes another part of it (pasta), and changing it back would break
> >>    pasta again (in a way we can't really fix).  
> >
> > Ugh, that is a mess.  
> 
> Well, yes, but one that happens regularly. :-D
> 
> The problem here is that it was reported only some months after the
> culprit landed, as fixing the regression through a revert or so could
> now cause a bigger regression. Do we assume this to be the case here?

Not a bigger regression, but a regression definitely, even just for 'ip
address' or 'ip address showdump' which would now go back to display
/ save addresses in the reversed order for IPv6 (only).

> It
> sounds like pasta would "only" be broken about as much as it was before
> -- then a revert or something like that is the right solution to get
> back to the status quo. Or does pasta already depend on the new behavior
> somehow and would now break even more if we reverted the culprit?

It doesn't specifically depend on it. There might be users depending on
it because, if they choose to copy container addresses from the host,
they will be reversed, and also picked by the kernel in the reversed
order (if all other criteria based on scope and prefix length are the
same).

The likelihood of that breaking setups should be relatively small. It's
just not zero so if we had another sane option to keep this "fixed" I
would probably prefer that.

> Then it's really a mess. :-/

I guess not so much, it shouldn't be a drama for pasta and containers.

But then again if we revert this, how do we fix the issue later,
without resorting to any of the ugly things below?

> But there are ways to fix even this, it's just that most of them are
> ugly. Like adding some bit somewhere to /proc/ or so that a fixed
> NetworkManager (if it's the only affected app) could flip by default to
> change the things from the old behavior to the new one; and one pasta
> could check that bit and warn. Config options are also an option, but
> that's even uglier.

Actually, an eventually fixed version of NetworkManager doesn't need to
know the behaviour of the kernel: it can just order addresses by
timestamps instead, as Fernando mentioned.

And I'm not sure how relevant this is, but if we revert the fix,
current combinations of NetworkManager / kernel versions would be
anyway affected.

So at this point the only robust / complete fix would be changing
NetworkManager to sort addresses as needed. Are you suggesting that we
should anyway try to minimise the temporal impact of this with a revert?

> >>    Now, I suppose NetworkManager is much more universally used and it's
> >>    definitely a more mature project so it's a bigger userspace breakage,
> >>    but at the same time it's a ton of containers (Podman uses pasta by
> >>    default, Docker/moby optionally via rootlesskit) we might introduce a
> >>    regression for.  
> > 
> > I think the main argument here is: while we know it is affecting
> > NetworkManager, we cannot know if it is breaking something else. The
> > kernel rule is usually "do not break userspace" and this is likely
> > considered a regression from my understanding. Cc'ing Thorsten as he is
> > an expert on regressions.  
> 
> From the mail at the top of the thread it definitely sounds like one.

-- 
Stefano