please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Kees Cook]

Hi stable tree maintainers,

Please revert the backports of

44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")

namely:

5.4:  03475167fda50b8511ef620a27409b08365882e1
5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4
5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93
6.1:  b0cde867b80a5e81fcbc0383e138f5845f2005ee
6.6:  1a45994fb218d93dec48a3a86f68283db61e0936

There seems to be a bad interaction between this change and older
PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation
continues to see if this will need to be reverted from 6.6, 6.11,
and mainline. But for now, it's clearly a problem for older kernels with
older qemu.

Thanks!

-Kees

[1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/

-- 
Kees Cook

Re: please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Michael Tokarev]

21.11.2024 17:33, Kees Cook wrote:
> Hi stable tree maintainers,
> 
> Please revert the backports of
> 
> 44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
> 
> namely:
> 
> 5.4:  03475167fda50b8511ef620a27409b08365882e1
> 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4
> 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93
> 6.1:  b0cde867b80a5e81fcbc0383e138f5845f2005ee
> 6.6:  1a45994fb218d93dec48a3a86f68283db61e0936
> 
> There seems to be a bad interaction between this change and older
> PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation
> continues to see if this will need to be reverted from 6.6, 6.11,
> and mainline. But for now, it's clearly a problem for older kernels with
> older qemu.
> 
> Thanks!
> 
> -Kees
> 
> [1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
Unfortunately I haven't seen this thread and this email before now,
when things are already too late.

And it turned out it's entirely my fault with all this.  Let me
explain so things become clear to everyone.

The problem here is entirely in qemu-user.  The fundamental issue
is that qemu-user does not implement an MMU, instead, it implements
just address shift, searching for a memory region for the guest address
space which is hopefully not used by qemu-user itself.

In practice, this is rarely an issue though, when - and this is the
default - qemu is built as a static-pie executable.  This is important:
it's the default mode for the static build - it builds as static-pie
executable, which works around the problem in almost all cases.
This is done for quite a long time, too.

However, I, as qemu maintainer in debian, got a bug report saying
that qemu-user-static isn't "static enough" - because for some tools
used on debian (lintian), static-pie was something unknown and the
tool issued a warning.  And at the time, I just added --disable-pie
flag to the build, without much thinking.  This is where things went
wrong.

Later I reverted this change with a shame, because it causes numerous
configurations to fail randomly, and each of them is very difficult to
debug (especially due to randomness of failures, sometimes it can work
50 times in a row but fail on the 51th).

But unfortunately, I forgot to revert this "de-PIEsation" change in
debian stable, and that's exactly where the original bug report come
from, stating kernel broke builds in qemu.

The same qemu-user-static configuration has been used by some other
distributions too, but hopefully everything's fixed now.  Except of
debian bookworm, and probably also ubuntu jammy (previous LTS).

It is not an "older qemu" anymore (though for a very old qemu this is
true again, that old one can't be used anymore with modern executables
anyway due to other reasons).  It is just my build mistake which is
*still* unfixed on debian stable (bookworm).  And even there, this
issue can trivially be fixed locally, since qemu-user-static is
self-contained and can be installed on older debian releases, and I
always provide up-to-date backports of qemu packages for debian stable.

And yes, qemu had numerous improvements in this area since bookworm
version, which addressed many other issues around this and fixed many
other configurations (which are not related to this kernel change),
but the fundamental issue (lack of full-blown MMU) remains.

Hopefully this clears things up, and it can be seen that this is not
a kernel bug.  And I'm hoping we'll fix this in debian bookworm soon
too.

Thanks, and sorry for all the buzz which caused my 2 mistakes.

/mjt

Re: please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Salvatore Bonaccorso]

Hi all,

On Mon, Nov 25, 2024 at 10:26:12AM +0300, Michael Tokarev wrote:
> 21.11.2024 17:33, Kees Cook wrote:
> > Hi stable tree maintainers,
> > 
> > Please revert the backports of
> > 
> > 44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
> > 
> > namely:
> > 
> > 5.4:  03475167fda50b8511ef620a27409b08365882e1
> > 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4
> > 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93
> > 6.1:  b0cde867b80a5e81fcbc0383e138f5845f2005ee
> > 6.6:  1a45994fb218d93dec48a3a86f68283db61e0936
> > 
> > There seems to be a bad interaction between this change and older
> > PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation
> > continues to see if this will need to be reverted from 6.6, 6.11,
> > and mainline. But for now, it's clearly a problem for older kernels with
> > older qemu.
> > 
> > Thanks!
> > 
> > -Kees
> > 
> > [1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
> Unfortunately I haven't seen this thread and this email before now,
> when things are already too late.
> 
> And it turned out it's entirely my fault with all this.  Let me
> explain so things become clear to everyone.
> 
> The problem here is entirely in qemu-user.  The fundamental issue
> is that qemu-user does not implement an MMU, instead, it implements
> just address shift, searching for a memory region for the guest address
> space which is hopefully not used by qemu-user itself.
> 
> In practice, this is rarely an issue though, when - and this is the
> default - qemu is built as a static-pie executable.  This is important:
> it's the default mode for the static build - it builds as static-pie
> executable, which works around the problem in almost all cases.
> This is done for quite a long time, too.
> 
> However, I, as qemu maintainer in debian, got a bug report saying
> that qemu-user-static isn't "static enough" - because for some tools
> used on debian (lintian), static-pie was something unknown and the
> tool issued a warning.  And at the time, I just added --disable-pie
> flag to the build, without much thinking.  This is where things went
> wrong.
> 
> Later I reverted this change with a shame, because it causes numerous
> configurations to fail randomly, and each of them is very difficult to
> debug (especially due to randomness of failures, sometimes it can work
> 50 times in a row but fail on the 51th).
> 
> But unfortunately, I forgot to revert this "de-PIEsation" change in
> debian stable, and that's exactly where the original bug report come
> from, stating kernel broke builds in qemu.
> 
> The same qemu-user-static configuration has been used by some other
> distributions too, but hopefully everything's fixed now.  Except of
> debian bookworm, and probably also ubuntu jammy (previous LTS).
> 
> It is not an "older qemu" anymore (though for a very old qemu this is
> true again, that old one can't be used anymore with modern executables
> anyway due to other reasons).  It is just my build mistake which is
> *still* unfixed on debian stable (bookworm).  And even there, this
> issue can trivially be fixed locally, since qemu-user-static is
> self-contained and can be installed on older debian releases, and I
> always provide up-to-date backports of qemu packages for debian stable.
> 
> And yes, qemu had numerous improvements in this area since bookworm
> version, which addressed many other issues around this and fixed many
> other configurations (which are not related to this kernel change),
> but the fundamental issue (lack of full-blown MMU) remains.
> 
> Hopefully this clears things up, and it can be seen that this is not
> a kernel bug.  And I'm hoping we'll fix this in debian bookworm soon
> too.
> 
> Thanks, and sorry for all the buzz which caused my 2 mistakes.

So catching up with that as we currently did cherry-pick the revert in
Debian but I defintivelfy would like to align with upstream (and drop
the cherry-pick again if it's not going to be picked for 6.1.y
upstream):

I'm a bit lost here. What are we going to do? Is the commit still
temporarly be applied to the stable series or are we staying at the
status quo and we should solely deal it within Debian on qemu side to
address the issue above and then we are fine? 

Or are there other cases outside Debian making it necessary apply the
above proposed revert to the stable series?

Regards,
Salvatore

Re: please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Greg KH]

On Thu, Dec 05, 2024 at 07:54:40AM +0100, Salvatore Bonaccorso wrote:
> Hi all,
> 
> On Mon, Nov 25, 2024 at 10:26:12AM +0300, Michael Tokarev wrote:
> > 21.11.2024 17:33, Kees Cook wrote:
> > > Hi stable tree maintainers,
> > > 
> > > Please revert the backports of
> > > 
> > > 44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
> > > 
> > > namely:
> > > 
> > > 5.4:  03475167fda50b8511ef620a27409b08365882e1
> > > 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4
> > > 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93
> > > 6.1:  b0cde867b80a5e81fcbc0383e138f5845f2005ee
> > > 6.6:  1a45994fb218d93dec48a3a86f68283db61e0936
> > > 
> > > There seems to be a bad interaction between this change and older
> > > PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation
> > > continues to see if this will need to be reverted from 6.6, 6.11,
> > > and mainline. But for now, it's clearly a problem for older kernels with
> > > older qemu.
> > > 
> > > Thanks!
> > > 
> > > -Kees
> > > 
> > > [1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
> > Unfortunately I haven't seen this thread and this email before now,
> > when things are already too late.
> > 
> > And it turned out it's entirely my fault with all this.  Let me
> > explain so things become clear to everyone.
> > 
> > The problem here is entirely in qemu-user.  The fundamental issue
> > is that qemu-user does not implement an MMU, instead, it implements
> > just address shift, searching for a memory region for the guest address
> > space which is hopefully not used by qemu-user itself.
> > 
> > In practice, this is rarely an issue though, when - and this is the
> > default - qemu is built as a static-pie executable.  This is important:
> > it's the default mode for the static build - it builds as static-pie
> > executable, which works around the problem in almost all cases.
> > This is done for quite a long time, too.
> > 
> > However, I, as qemu maintainer in debian, got a bug report saying
> > that qemu-user-static isn't "static enough" - because for some tools
> > used on debian (lintian), static-pie was something unknown and the
> > tool issued a warning.  And at the time, I just added --disable-pie
> > flag to the build, without much thinking.  This is where things went
> > wrong.
> > 
> > Later I reverted this change with a shame, because it causes numerous
> > configurations to fail randomly, and each of them is very difficult to
> > debug (especially due to randomness of failures, sometimes it can work
> > 50 times in a row but fail on the 51th).
> > 
> > But unfortunately, I forgot to revert this "de-PIEsation" change in
> > debian stable, and that's exactly where the original bug report come
> > from, stating kernel broke builds in qemu.
> > 
> > The same qemu-user-static configuration has been used by some other
> > distributions too, but hopefully everything's fixed now.  Except of
> > debian bookworm, and probably also ubuntu jammy (previous LTS).
> > 
> > It is not an "older qemu" anymore (though for a very old qemu this is
> > true again, that old one can't be used anymore with modern executables
> > anyway due to other reasons).  It is just my build mistake which is
> > *still* unfixed on debian stable (bookworm).  And even there, this
> > issue can trivially be fixed locally, since qemu-user-static is
> > self-contained and can be installed on older debian releases, and I
> > always provide up-to-date backports of qemu packages for debian stable.
> > 
> > And yes, qemu had numerous improvements in this area since bookworm
> > version, which addressed many other issues around this and fixed many
> > other configurations (which are not related to this kernel change),
> > but the fundamental issue (lack of full-blown MMU) remains.
> > 
> > Hopefully this clears things up, and it can be seen that this is not
> > a kernel bug.  And I'm hoping we'll fix this in debian bookworm soon
> > too.
> > 
> > Thanks, and sorry for all the buzz which caused my 2 mistakes.
> 
> So catching up with that as we currently did cherry-pick the revert in
> Debian but I defintivelfy would like to align with upstream (and drop
> the cherry-pick again if it's not going to be picked for 6.1.y
> upstream):
> 
> I'm a bit lost here. What are we going to do? Is the commit still
> temporarly be applied to the stable series or are we staying at the
> status quo and we should solely deal it within Debian on qemu side to
> address the issue above and then we are fine? 

I read this as "oops, we messed up in qemu and will fix it there" and so
I dropped the reverts from the kernel.  If that's not the case, please
let me know.

thanks,

greg k-h

Re: please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Salvatore Bonaccorso]

Hi Greg,

On Thu, Dec 05, 2024 at 08:07:15AM +0100, Greg KH wrote:
> On Thu, Dec 05, 2024 at 07:54:40AM +0100, Salvatore Bonaccorso wrote:
> > Hi all,
> > 
> > On Mon, Nov 25, 2024 at 10:26:12AM +0300, Michael Tokarev wrote:
> > > 21.11.2024 17:33, Kees Cook wrote:
> > > > Hi stable tree maintainers,
> > > > 
> > > > Please revert the backports of
> > > > 
> > > > 44c76825d6ee ("x86: Increase brk randomness entropy for 64-bit systems")
> > > > 
> > > > namely:
> > > > 
> > > > 5.4:  03475167fda50b8511ef620a27409b08365882e1
> > > > 5.10: 25d31baf922c1ee987efd6fcc9c7d4ab539c66b4
> > > > 5.15: 06cb3463aa58906cfff72877eb7f50cb26e9ca93
> > > > 6.1:  b0cde867b80a5e81fcbc0383e138f5845f2005ee
> > > > 6.6:  1a45994fb218d93dec48a3a86f68283db61e0936
> > > > 
> > > > There seems to be a bad interaction between this change and older
> > > > PIE-built qemu-user-static (for aarch64) binaries[1]. Investigation
> > > > continues to see if this will need to be reverted from 6.6, 6.11,
> > > > and mainline. But for now, it's clearly a problem for older kernels with
> > > > older qemu.
> > > > 
> > > > Thanks!
> > > > 
> > > > -Kees
> > > > 
> > > > [1] https://lore.kernel.org/all/202411201000.F3313C02@keescook/
> > > Unfortunately I haven't seen this thread and this email before now,
> > > when things are already too late.
> > > 
> > > And it turned out it's entirely my fault with all this.  Let me
> > > explain so things become clear to everyone.
> > > 
> > > The problem here is entirely in qemu-user.  The fundamental issue
> > > is that qemu-user does not implement an MMU, instead, it implements
> > > just address shift, searching for a memory region for the guest address
> > > space which is hopefully not used by qemu-user itself.
> > > 
> > > In practice, this is rarely an issue though, when - and this is the
> > > default - qemu is built as a static-pie executable.  This is important:
> > > it's the default mode for the static build - it builds as static-pie
> > > executable, which works around the problem in almost all cases.
> > > This is done for quite a long time, too.
> > > 
> > > However, I, as qemu maintainer in debian, got a bug report saying
> > > that qemu-user-static isn't "static enough" - because for some tools
> > > used on debian (lintian), static-pie was something unknown and the
> > > tool issued a warning.  And at the time, I just added --disable-pie
> > > flag to the build, without much thinking.  This is where things went
> > > wrong.
> > > 
> > > Later I reverted this change with a shame, because it causes numerous
> > > configurations to fail randomly, and each of them is very difficult to
> > > debug (especially due to randomness of failures, sometimes it can work
> > > 50 times in a row but fail on the 51th).
> > > 
> > > But unfortunately, I forgot to revert this "de-PIEsation" change in
> > > debian stable, and that's exactly where the original bug report come
> > > from, stating kernel broke builds in qemu.
> > > 
> > > The same qemu-user-static configuration has been used by some other
> > > distributions too, but hopefully everything's fixed now.  Except of
> > > debian bookworm, and probably also ubuntu jammy (previous LTS).
> > > 
> > > It is not an "older qemu" anymore (though for a very old qemu this is
> > > true again, that old one can't be used anymore with modern executables
> > > anyway due to other reasons).  It is just my build mistake which is
> > > *still* unfixed on debian stable (bookworm).  And even there, this
> > > issue can trivially be fixed locally, since qemu-user-static is
> > > self-contained and can be installed on older debian releases, and I
> > > always provide up-to-date backports of qemu packages for debian stable.
> > > 
> > > And yes, qemu had numerous improvements in this area since bookworm
> > > version, which addressed many other issues around this and fixed many
> > > other configurations (which are not related to this kernel change),
> > > but the fundamental issue (lack of full-blown MMU) remains.
> > > 
> > > Hopefully this clears things up, and it can be seen that this is not
> > > a kernel bug.  And I'm hoping we'll fix this in debian bookworm soon
> > > too.
> > > 
> > > Thanks, and sorry for all the buzz which caused my 2 mistakes.
> > 
> > So catching up with that as we currently did cherry-pick the revert in
> > Debian but I defintivelfy would like to align with upstream (and drop
> > the cherry-pick again if it's not going to be picked for 6.1.y
> > upstream):
> > 
> > I'm a bit lost here. What are we going to do? Is the commit still
> > temporarly be applied to the stable series or are we staying at the
> > status quo and we should solely deal it within Debian on qemu side to
> > address the issue above and then we are fine? 
> 
> I read this as "oops, we messed up in qemu and will fix it there" and so
> I dropped the reverts from the kernel.  If that's not the case, please
> let me know.

Thanks for the quick confirmation.

Michael, I will drop the local revert in src:linux then on the next
upload I do. Can you handle the qemu part in Debian accordingly (and
make sure the fix lands in the next Debian point release in January
and ideally maybe via stable-updates earlier to affected people?)

Regards,
Salvatore

Re: please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Michael Tokarev]

05.12.2024 10:22, Salvatore Bonaccorso wrote:

> Michael, I will drop the local revert in src:linux then on the next
> upload I do. Can you handle the qemu part in Debian accordingly (and
> make sure the fix lands in the next Debian point release in January
> and ideally maybe via stable-updates earlier to affected people?)

It would be great to fix this in debian, yes.  It's a debian matter
anyway, for details there's no need to bother kernel people.
Please see #1088273.  Fixed qemu release should come together with
the updated kernel in debian -- if it will be next point release,
so be it, but if the kernel goes before, we should pick qemu too,
because else we'll re-introduce #1087822.

Ubuntu is also affected, it has similar situation, but they have
different schedule and different versions.

Thanks,

/mjt

Re: please revert backport of 44c76825d6eefee9eb7ce06c38e1a6632ac7eb7d

[Dominique Martinet]

Hi mjt,

Michael Tokarev wrote on Thu, Dec 05, 2024 at 10:35:26AM +0300:
> It would be great to fix this in debian, yes.  It's a debian matter
> anyway, for details there's no need to bother kernel people.
> Please see #1088273.  Fixed qemu release should come together with
> the updated kernel in debian -- if it will be next point release,
> so be it, but if the kernel goes before, we should pick qemu too,
> because else we'll re-introduce #1087822.

The kernel aleady shipped in debian a while ago, hence the bug
reports :)

More precisely, the "bad" commit b0cde867b80 (in 6.1 tree) made it in
6.1.107, and debian stable has been shipping it since the 6.1.112 update
on the 3rd of October[1][2], so we've been here for a while now

[1] https://tracker.debian.org/pkg/linux-signed-amd64
[2] https://tracker.debian.org/news/1571617/accepted-linux-signed-amd64-611121-source-into-stable-security/


Cheers,
-- 
Dominique Martinet | Asmadeus