Problem with IPv6 privacy addresses in 7.0

Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

My normal use of my desktop system (Fedora 43) is to suspend at night
and wake in the morning, and then immediately SSH to my remote server,
where I stay logged in.  I use NetworkManager and have ipv6.ip6-privacy
set to prefer-temp-addr, and SSH configured to use SSH keepalives.

When I upgraded to kernel 7.0, I started having an issue with this - my
SSH session gets dropped, usually several times, in the first hour or
so, then I usually don't have any problem the rest of the day.  If I run
an IPv4 session at the same time, that seems to be fine, so I don't
think it's a network issue (I know that dual-stack doesn't always take
the same path).

What seems to be happening is that privacy addresses are removed while
the SSH session is still using them, before the timeout even.  I wrote a
script to watch public v6 addresses being added and removed, and this is
what I've seen so far today (the number at the end is the valid_lft
seconds), with the public prefix masked:

2026-05-21 07:39:17 removed xx::f4f/128 4673
2026-05-21 07:39:20 added xx::f4f/128 5400
2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120

I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d
source address.  That dropped in 2 minutes and I reconnected, which used
the 1d8f address.  That dropped at 08:14:43, I didn't notice right away,
I reconnected at 08:31:28 which used the 94fd address.

-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Thu, 21 May 2026 08:53:10 -0500 Chris Adams wrote:
> My normal use of my desktop system (Fedora 43) is to suspend at night
> and wake in the morning, and then immediately SSH to my remote server,
> where I stay logged in.  I use NetworkManager and have ipv6.ip6-privacy
> set to prefer-temp-addr, and SSH configured to use SSH keepalives.
> 
> When I upgraded to kernel 7.0, I started having an issue with this - my
> SSH session gets dropped, usually several times, in the first hour or
> so, then I usually don't have any problem the rest of the day.  If I run
> an IPv4 session at the same time, that seems to be fine, so I don't
> think it's a network issue (I know that dual-stack doesn't always take
> the same path).
> 
> What seems to be happening is that privacy addresses are removed while
> the SSH session is still using them, before the timeout even.  I wrote a
> script to watch public v6 addresses being added and removed, and this is
> what I've seen so far today (the number at the end is the valid_lft
> seconds), with the public prefix masked:
> 
> 2026-05-21 07:39:17 removed xx::f4f/128 4673
> 2026-05-21 07:39:20 added xx::f4f/128 5400
> 2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
> 2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
> 2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
> 2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
> 2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
> 2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
> 2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
> 2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120
> 
> I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d
> source address.  That dropped in 2 minutes and I reconnected, which used
> the 1d8f address.  That dropped at 08:14:43, I didn't notice right away,
> I reconnected at 08:31:28 which used the 94fd address.

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
or 6.19?

Would you be able to try testing with some commits reverted?
On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
c7dc5b522882 ("ipv6: clean up routes when manually removing address with a lifetime")

Less likely:
5023479627e3 ("ipv6: Switch to higher-level SHA-1 functions")
9e371b0ba7f5 ("ipv6: addrconf: reduce default temp_valid_lft to 2 days")
6af51e9f3133 ("ipv6: Remove permanent routes from tb6_gc_hlist when all exceptions expire.")

Re: Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

Heh, I opened a Fedora BZ on this and JUST updated it... RHBZ 2480928

Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
> On Thu, 21 May 2026 08:53:10 -0500 Chris Adams wrote:
> > My normal use of my desktop system (Fedora 43) is to suspend at night
> > and wake in the morning, and then immediately SSH to my remote server,
> > where I stay logged in.  I use NetworkManager and have ipv6.ip6-privacy
> > set to prefer-temp-addr, and SSH configured to use SSH keepalives.
> > 
> > When I upgraded to kernel 7.0, I started having an issue with this - my
> > SSH session gets dropped, usually several times, in the first hour or
> > so, then I usually don't have any problem the rest of the day.  If I run
> > an IPv4 session at the same time, that seems to be fine, so I don't
> > think it's a network issue (I know that dual-stack doesn't always take
> > the same path).
> > 
> > What seems to be happening is that privacy addresses are removed while
> > the SSH session is still using them, before the timeout even.  I wrote a
> > script to watch public v6 addresses being added and removed, and this is
> > what I've seen so far today (the number at the end is the valid_lft
> > seconds), with the public prefix masked:
> > 
> > 2026-05-21 07:39:17 removed xx::f4f/128 4673
> > 2026-05-21 07:39:20 added xx::f4f/128 5400
> > 2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
> > 2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
> > 2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
> > 2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
> > 2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
> > 2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
> > 2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
> > 2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120
> > 
> > I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d
> > source address.  That dropped in 2 minutes and I reconnected, which used
> > the 1d8f address.  That dropped at 08:14:43, I didn't notice right away,
> > I reconnected at 08:31:28 which used the 94fd address.
> 
> Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> or 6.19?

It was 6.19 to 7.0.

> Would you be able to try testing with some commits reverted?
> On a quick look the candidates would be:
> 
> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

I figured out that it happens after stopping a VM (and I usually
start/stop a VM for a bit in the morning, which is why it happened more
than once).  So I set up a VM with a nested VM, running up-to-date
Fedora 44, and then was able to bisect pretty easily, and it landed on
this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy
address management (right?).  NM didn't change, so maybe this commit is
confusing something in NM?

-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
> > Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> > or 6.19?  
> 
> It was 6.19 to 7.0.
> 
> > Would you be able to try testing with some commits reverted?
> > On a quick look the candidates would be:
> > 
> > cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> 
> It's this one.

Phew, the second one was mine :)

> I figured out that it happens after stopping a VM (and I usually
> start/stop a VM for a bit in the morning, which is why it happened more
> than once).  So I set up a VM with a nested VM, running up-to-date
> Fedora 44, and then was able to bisect pretty easily, and it landed on
> this commit.
> 
> Fedora is using NetworkManager, and IIRC NM does some part of privacy
> address management (right?).  NM didn't change, so maybe this commit is
> confusing something in NM?

Sounds plausible, pretty sure we knew this commit was risky to begin
with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able
to send the revert with your problem description and a Fixes tag
pointing to the reverted commit?

Re: Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
> > > Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> > > or 6.19?  
> > 
> > It was 6.19 to 7.0.
> > 
> > > Would you be able to try testing with some commits reverted?
> > > On a quick look the candidates would be:
> > > 
> > > cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> > 
> > It's this one.
> 
> Phew, the second one was mine :)
> 
> > I figured out that it happens after stopping a VM (and I usually
> > start/stop a VM for a bit in the morning, which is why it happened more
> > than once).  So I set up a VM with a nested VM, running up-to-date
> > Fedora 44, and then was able to bisect pretty easily, and it landed on
> > this commit.
> > 
> > Fedora is using NetworkManager, and IIRC NM does some part of privacy
> > address management (right?).  NM didn't change, so maybe this commit is
> > confusing something in NM?
> 
> Sounds plausible, pretty sure we knew this commit was risky to begin
> with, but we had no direct proof that it'd break real life users.
> 
> Revert is the right course of action here. Would you be willing/able
> to send the revert with your problem description and a Fixes tag
> pointing to the reverted commit?

So... I'm not very git-fluent and haven't submitted a kernel patch in
many years, so really am not sure of the necessary/proper steps.
-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Fernando Fernandez Mancera]

On 5/27/26 11:13 PM, Chris Adams wrote:
> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
>>>> or 6.19?
>>>
>>> It was 6.19 to 7.0.
>>>
>>>> Would you be able to try testing with some commits reverted?
>>>> On a quick look the candidates would be:
>>>>
>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>>>
>>> It's this one.
>>
>> Phew, the second one was mine :)
>>
>>> I figured out that it happens after stopping a VM (and I usually
>>> start/stop a VM for a bit in the morning, which is why it happened more
>>> than once).  So I set up a VM with a nested VM, running up-to-date
>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
>>> this commit.
>>>
>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
>>> address management (right?).  NM didn't change, so maybe this commit is
>>> confusing something in NM?
>>
>> Sounds plausible, pretty sure we knew this commit was risky to begin
>> with, but we had no direct proof that it'd break real life users.
>>
>> Revert is the right course of action here. Would you be willing/able
>> to send the revert with your problem description and a Fixes tag
>> pointing to the reverted commit?
> 
> So... I'm not very git-fluent and haven't submitted a kernel patch in
> many years, so really am not sure of the necessary/proper steps.

I don't mind to handle it. I will CC you, is that fine for you?

Thanks for the report btw!

Re: Problem with IPv6 privacy addresses in 7.0

[Chris Adams]

Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
> > > Hi! Adding more people to CC. Do you know if you upgraded from 6.18 
> > > or 6.19?  
> > 
> > It was 6.19 to 7.0.
> > 
> > > Would you be able to try testing with some commits reverted?
> > > On a quick look the candidates would be:
> > > 
> > > cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> > 
> > It's this one.
> 
> Phew, the second one was mine :)
> 
> > I figured out that it happens after stopping a VM (and I usually
> > start/stop a VM for a bit in the morning, which is why it happened more
> > than once).  So I set up a VM with a nested VM, running up-to-date
> > Fedora 44, and then was able to bisect pretty easily, and it landed on
> > this commit.
> > 
> > Fedora is using NetworkManager, and IIRC NM does some part of privacy
> > address management (right?).  NM didn't change, so maybe this commit is
> > confusing something in NM?
> 
> Sounds plausible, pretty sure we knew this commit was risky to begin
> with, but we had no direct proof that it'd break real life users.
> 
> Revert is the right course of action here. Would you be willing/able
> to send the revert with your problem description and a Fixes tag
> pointing to the reverted commit?

I did want to add a little more test note:

It's definitely an interaction with NetworkManager.  If I stop NM and
run my VM start/stop test, nothing unexpected happens after.  If NM is
running and I do my VM test, when the next router advertisement is
received, NM replaces the privacy addresses.

-- 
Chris Adams <linux@cmadams.net>

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Wed, 27 May 2026 16:13:56 -0500 Chris Adams wrote:
> > > I figured out that it happens after stopping a VM (and I usually
> > > start/stop a VM for a bit in the morning, which is why it happened more
> > > than once).  So I set up a VM with a nested VM, running up-to-date
> > > Fedora 44, and then was able to bisect pretty easily, and it landed on
> > > this commit.
> > > 
> > > Fedora is using NetworkManager, and IIRC NM does some part of privacy
> > > address management (right?).  NM didn't change, so maybe this commit is
> > > confusing something in NM?  
> > 
> > Sounds plausible, pretty sure we knew this commit was risky to begin
> > with, but we had no direct proof that it'd break real life users.
> > 
> > Revert is the right course of action here. Would you be willing/able
> > to send the revert with your problem description and a Fixes tag
> > pointing to the reverted commit?  
> 
> So... I'm not very git-fluent and haven't submitted a kernel patch in
> many years, so really am not sure of the necessary/proper steps.

I wonder how you managed to bisect it then :)

git clone ..
git revert cb3de96eea66

you can use db5dadb562cabb6 for inspiration on how to format 
the commit message. You can use an LLM to help, they are plenty
capable enough to handle a revert. For sending either git send-email
or you can use b4 relay. https://b4.docs.kernel.org/en/latest/

If you prefer us to handle this - that's perfectly fine, just let me
know. I'm asking if you _want_ to handle it, as some folks like the
sense of accomplishment of having upstream commits.

Re: Problem with IPv6 privacy addresses in 7.0

[Fernando Fernandez Mancera]

On 5/27/26 11:51 PM, Chris Adams wrote:
> Once upon a time, Jakub Kicinski <kuba@kernel.org> said:
>> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
>>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
>>>> or 6.19?
>>>
>>> It was 6.19 to 7.0.
>>>
>>>> Would you be able to try testing with some commits reverted?
>>>> On a quick look the candidates would be:
>>>>
>>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
>>>
>>> It's this one.
>>
>> Phew, the second one was mine :)
>>
>>> I figured out that it happens after stopping a VM (and I usually
>>> start/stop a VM for a bit in the morning, which is why it happened more
>>> than once).  So I set up a VM with a nested VM, running up-to-date
>>> Fedora 44, and then was able to bisect pretty easily, and it landed on
>>> this commit.
>>>
>>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
>>> address management (right?).  NM didn't change, so maybe this commit is
>>> confusing something in NM?
>>
>> Sounds plausible, pretty sure we knew this commit was risky to begin
>> with, but we had no direct proof that it'd break real life users.
>>
>> Revert is the right course of action here. Would you be willing/able
>> to send the revert with your problem description and a Fixes tag
>> pointing to the reverted commit?
> 
> I did want to add a little more test note:
> 
> It's definitely an interaction with NetworkManager.  If I stop NM and
> run my VM start/stop test, nothing unexpected happens after.  If NM is
> running and I do my VM test, when the next router advertisement is
> received, NM replaces the privacy addresses.
> 

As someone that is experienced in NetworkManager, I can confirm it is 
related. NetworkManager is querying the IPv6 address and when the 
connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), 
NetworkManager creates a route to make the system use the temporary 
address for outgoing connections by default.

If the order is messed up, the address picked will likely be too. One 
could argue that this is partially fault of NetworkManager and that it 
should check the timestamps or preferred times rather than order.. but 
well, the rule is "do not break userspace".

I hope this clarifies things.

Thanks,
Fernando.

Re: Problem with IPv6 privacy addresses in 7.0

[Jakub Kicinski]

On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera wrote:
> On 5/27/26 11:51 PM, Chris Adams wrote:
> > Once upon a time, Jakub Kicinski <kuba@kernel.org> said:  
> >> Sounds plausible, pretty sure we knew this commit was risky to begin
> >> with, but we had no direct proof that it'd break real life users.
> >>
> >> Revert is the right course of action here. Would you be willing/able
> >> to send the revert with your problem description and a Fixes tag
> >> pointing to the reverted commit?  
> > 
> > I did want to add a little more test note:
> > 
> > It's definitely an interaction with NetworkManager.  If I stop NM and
> > run my VM start/stop test, nothing unexpected happens after.  If NM is
> > running and I do my VM test, when the next router advertisement is
> > received, NM replaces the privacy addresses.
> >   
> 
> As someone that is experienced in NetworkManager, I can confirm it is 
> related. NetworkManager is querying the IPv6 address and when the 
> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), 
> NetworkManager creates a route to make the system use the temporary 
> address for outgoing connections by default.
> 
> If the order is messed up, the address picked will likely be too. One 
> could argue that this is partially fault of NetworkManager and that it 
> should check the timestamps or preferred times rather than order.. but 
> well, the rule is "do not break userspace".

Right, plus the justification for the change wasn't very strong 
in the first place. My internal compass is still pointing towards
a revert.

> I hope this clarifies things.

Re: Problem with IPv6 privacy addresses in 7.0

[Stefano Brivio]

Hi Fernando, Jakub,

On Wed, 27 May 2026 23:59:47 +0200
Fernando Fernandez Mancera <fmancera@suse.de> wrote:

> On 5/27/26 11:51 PM, Chris Adams wrote:
> > Once upon a time, Jakub Kicinski <kuba@kernel.org> said:  
> >> On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:  
> >>>> Hi! Adding more people to CC. Do you know if you upgraded from 6.18
> >>>> or 6.19?  
> >>>
> >>> It was 6.19 to 7.0.
> >>>  
> >>>> Would you be able to try testing with some commits reverted?
> >>>> On a quick look the candidates would be:
> >>>>
> >>>> cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")  
> >>>
> >>> It's this one.  
> >>
> >> Phew, the second one was mine :)
> >>  
> >>> I figured out that it happens after stopping a VM (and I usually
> >>> start/stop a VM for a bit in the morning, which is why it happened more
> >>> than once).  So I set up a VM with a nested VM, running up-to-date
> >>> Fedora 44, and then was able to bisect pretty easily, and it landed on
> >>> this commit.
> >>>
> >>> Fedora is using NetworkManager, and IIRC NM does some part of privacy
> >>> address management (right?).  NM didn't change, so maybe this commit is
> >>> confusing something in NM?  
> >>
> >> Sounds plausible, pretty sure we knew this commit was risky to begin
> >> with, but we had no direct proof that it'd break real life users.
> >>
> >> Revert is the right course of action here. Would you be willing/able
> >> to send the revert with your problem description and a Fixes tag
> >> pointing to the reverted commit?  
> > 
> > I did want to add a little more test note:
> > 
> > It's definitely an interaction with NetworkManager.  If I stop NM and
> > run my VM start/stop test, nothing unexpected happens after.  If NM is
> > running and I do my VM test, when the next router advertisement is
> > received, NM replaces the privacy addresses.
> >   
> 
> As someone that is experienced in NetworkManager, I can confirm it is 
> related. NetworkManager is querying the IPv6 address and when the 
> connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), 
> NetworkManager creates a route to make the system use the temporary 
> address for outgoing connections by default.
> 
> If the order is messed up, the address picked will likely be too. One 
> could argue that this is partially fault of NetworkManager and that it 
> should check the timestamps or preferred times rather than order.. but 
> well, the rule is "do not break userspace".
> 
> I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose
of that commit is to *preserve* the order of addresses as they were
inserted, not to mess it up.

Before that, addresses were stored and returned via netlink *reversed*
compared to the insertion, which is rather surprising, also because
it's the opposite of what we do for IPv4 addresses, and caused issues
with pasta(1) as it copies addresses into containers in the same order
as reported via netlink.

Do you happen to know exactly in which way the order happens to be wrong
now?

Also note that userspace was broken and fixed a couple of times:

  https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too
long and this is causing issues for a lot of people meanwhile of course
it might make sense to revert, but I'd give it a quick try at least.

-- 
Stefano