Re: [PATCH v3 0/9] module: Introduce hash-based integrity checking

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Wed, 2025-05-07 at 09:47 +0200, Arnout Engelen wrote:
> On Tue, May 6, 2025, at 15:24, James Bottomley wrote:
> > I'll repeat the key point again: all modern hermetic build systems
> > come with provenance which is usually a signature.
> 
> I'm not sure the 'hermetic build' parallel is so applicable here:
> typically a hermetic build will produce an artifact and a signature,
> and when you embed that result in a larger aggregate, you only embed
> the artifact (not the signature) and sign the aggregate.

That depends whether you want to demonstrate the provenance of the
result to someone consuming your aggregate or not; Some people are OK
with the trust my signature approach, others want tracing to point of
origin.

>  With module signatures, the module *and* their signatures are
> embedded in the aggregate (e.g. ISO, disk image), which is
> where (at least in my case) the friction comes from.

For Linux in particular, most people won't be booting any image unless
the binary is secure boot signed, so this problem doesn't go away if
you strip module signatures.

> > Plus, you've got to remember that a signature is a cryptographic
> > function of the hash over the build minus the signature.  You can't
> > verify a signature unless you know how to get the build minus the
> > signature.  So the process is required to be deterministic.
> 
> Right: there is no friction validating the module signatures, that is
> fine. There is friction validating the aggregate artifact (e.g. ISO,
> disk image), though, because of those signatures embedded into it.

I think we understand the problem with signatures (particularly the
ones which add entropy and can thus change every time the same object
is signed).  However, I don't think we can accept that no signatures
can be on the ISO ... we'll have to have at least secure boot
signatures and if there's a way of doing that then there should be a
way of doing other signatures.

> As you mentioned earlier, of course this is *possible* to do (for
> example by adding the signatures as inputs to the second
> 'independent' build or by creating a hard-to-validate 'check recipe'
> running the build in reverse). Still, checking modules at run time by
> hash instead of by signature would be a much simpler option for such
> scenario's.

Well, my objection was merely to the description saying verifying
reproducibility with signatures was not possible (it is).

However, the problem with distros adopting an immutable hash list for
module loading would be DKMS, but I think the distributions that go
that route have all solved the reproducibility issues with signatures
anyway, so perhaps that's not an issue.

> > > > All current secure build processes (hermetic builds, SLSA and
> > > > the like) are requiring output provenance (i.e. signed
> > > > artifacts).  If you try to stand like Canute against this tide
> > > > saying "no signed builds", you're simply opposing progress for
> > > > the sake of it
> > > 
> > > I don't think anyone is saying 'no signed builds', but we'd enjoy
> > > being able to keep the signatures as detached metadata instead of
> > > having to embed them into the 'actual' artifacts.
> > 
> > We had this debate about 15 years ago when Debian first started
> > reproducible builds for the kernel.  Their initial approach was
> > detached module signatures.  This was the original patch set:
> > 
> > https://lore.kernel.org/linux-modules/20160405001611.GJ21187@decadent.org.uk/
> > 
> > And this is the reason why Debian abandoned it:
> > 
> > https://lists.debian.org/debian-kernel/2016/05/msg00384.html
> 
> That is interesting history, thanks for digging that up. Of the 2
> problems Ben mentions running into there, '1' does not seem universal
> (I think this feature is indeed mainly interesting for systems where
> you don't _want_ anyone to be able to load locally-built modules),
> and '2' is a problem that detached signatures have but module hashes
> don't have.

I think Debian ended up going with 2, but since they also provide DKMS
infrastructure, hash module lists won't work for them anyway.

> > The specific problem is why detached signatures are almost always a
> > problem: after a period of time, particularly if the process for
> > creating updated artifacts gets repeated often matching the output
> > to the right signature becomes increasingly error prone.
> 
> I haven't experienced that issue with the module hashes yet.

Heh, I'll repeat this question after you've done umpteen builds of the
same kernel for debugging purposes. The problem that will bite me is
that I often just rebuild a single module and reinsert to try to chase
a bug down.  With this scheme I can't simply reinsert, I'd have to
rebuild the hash list and reboot the entire vmlinux.

> > Debian was, however, kind enough to attach what they currently do
> > to get reproducible builds to the kernel documentation:
> > 
> > https://docs.kernel.org/kbuild/reproducible-builds.html
> 
> Cool, I was aware of that page but didn't know it was initially
> contributed by Debian.
> 
> > However, if you want to detach the module signatures for packaging,
> > so the modules can go in a reproducible section and the signatures
> > elsewhere, then I think we could accommodate that (the output of
> > the build is actually unsigned modules, they just get signed on
> > install).
> 
> At least I don't really come to this from the packaging perspective,
> but from the "building an independently verifiable ISO/disk image"
> perspective. Separating the modules and the signatures into separate
> packages doesn't help me there, since they'd still both need to be
> present on the image.

So how do you cope with secure boot?  I mean if the object is to
produce an ISO that is demonstrably reproducible but otherwise
unusable, we can certainly script a way to excise all the signatures
including the secure boot one.

Regards,

James