kernel.org tarball/patch signature files

kernel.org tarball/patch signature files

[Jari Ruusu]

I noticed that patch-3.0.7.sign is a detached signature file for
DECOMPRESSED patch-3.0.7.{bz2,gz,xz}. Maybe this is not the best possible
way to sign compressed tarballs/patches. This is because it places hell of
lot of trust on quality/security of decompressor implementation.
Historically decompressor implementations have had bugs and security flaws.
It is stupid to assume that there won't be any more of them.

Wrong order to verify compressed tarball/patch:

(1) Feed potentially maliciously formatted data to decompressor, and exploit
    any undiscovered/unpatched vulnerability in decompressor implementation.
(2) Verify decompressed output.

Much better order would be:

(1) Verify compressed data.
(2) Feed trusted data to decompressor.

So, would it be possible to have multiple signature files like this? Please.

patch-3.X.Y.bz2
patch-3.X.Y.bz2.sign
patch-3.X.Y.gz
patch-3.X.Y.gz.sign
patch-3.X.Y.xz
patch-3.X.Y.xz.sign

-- 
Jari Ruusu  1024R/3A220F51 5B 4B F9 BB D3 3F 52 E9  DB 1D EB E3 24 0E A9 DD

Re: kernel.org tarball/patch signature files

[Greg KH]

On Sun, Oct 23, 2011 at 02:17:20PM +0300, Jari Ruusu wrote:
> I noticed that patch-3.0.7.sign is a detached signature file for
> DECOMPRESSED patch-3.0.7.{bz2,gz,xz}.

That's exactly what I said in my announcement:
	https://lkml.org/lkml/2011/10/23/51
so I'm glad it's working properly :)

> Maybe this is not the best possible
> way to sign compressed tarballs/patches. This is because it places hell of
> lot of trust on quality/security of decompressor implementation.
> Historically decompressor implementations have had bugs and security flaws.
> It is stupid to assume that there won't be any more of them.
> 
> Wrong order to verify compressed tarball/patch:
> 
> (1) Feed potentially maliciously formatted data to decompressor, and exploit
>     any undiscovered/unpatched vulnerability in decompressor implementation.
> (2) Verify decompressed output.
> 
> Much better order would be:
> 
> (1) Verify compressed data.
> (2) Feed trusted data to decompressor.
> 
> So, would it be possible to have multiple signature files like this? Please.
> 
> patch-3.X.Y.bz2
> patch-3.X.Y.bz2.sign
> patch-3.X.Y.gz
> patch-3.X.Y.gz.sign
> patch-3.X.Y.xz
> patch-3.X.Y.xz.sign

Nope, sorry, let's try this way instead.  That way we only have to
generate one signature, not 3.

If you are really worried about decompressor bugs, then run them in a
virtual machine/chroot :)

greg k-h

Re: kernel.org tarball/patch signature files

[Jari Ruusu]

Greg KH wrote:
> On Sun, Oct 23, 2011 at 02:17:20PM +0300, Jari Ruusu wrote:
> > Wrong order to verify compressed tarball/patch:
> >
> > (1) Feed potentially maliciously formatted data to decompressor, and exploit
> >     any undiscovered/unpatched vulnerability in decompressor implementation.
> > (2) Verify decompressed output.
> >
> > Much better order would be:
> >
> > (1) Verify compressed data.
> > (2) Feed trusted data to decompressor.
> >
> > So, would it be possible to have multiple signature files like this? Please.
> >
> > patch-3.X.Y.bz2
> > patch-3.X.Y.bz2.sign
> > patch-3.X.Y.gz
> > patch-3.X.Y.gz.sign
> > patch-3.X.Y.xz
> > patch-3.X.Y.xz.sign
> 
> Nope, sorry, let's try this way instead.  That way we only have to
> generate one signature, not 3.

How about one signed message that contains multiple SHA256 sums or whatever?

  sha256sum patch-3.X.Y.{bz2,gz,xz} | gpg --clearsign -a >patch-3.X.Y.sign

That allows verification before decompression.

> If you are really worried about decompressor bugs, then run them in a
> virtual machine/chroot :)

I am not amused.

Greg, please put your security hat on, and look at this from security point
of view. Decompression after verify removes/closes attacks utilizing yet
unidentified decompressor bugs or security flaws.

If I remember correctly, newer versions of OpenSSH disable compression
before authentication. They do that to pre-emptively close attacks resulting
yet unidentified bugs in decompression code.

-- 
Jari Ruusu  1024R/3A220F51 5B 4B F9 BB D3 3F 52 E9  DB 1D EB E3 24 0E A9 DD

Re: kernel.org tarball/patch signature files

[Greg KH]

On Sun, Oct 23, 2011 at 05:07:51PM +0300, Jari Ruusu wrote:
> Greg KH wrote:
> > On Sun, Oct 23, 2011 at 02:17:20PM +0300, Jari Ruusu wrote:
> > > Wrong order to verify compressed tarball/patch:
> > >
> > > (1) Feed potentially maliciously formatted data to decompressor, and exploit
> > >     any undiscovered/unpatched vulnerability in decompressor implementation.
> > > (2) Verify decompressed output.
> > >
> > > Much better order would be:
> > >
> > > (1) Verify compressed data.
> > > (2) Feed trusted data to decompressor.
> > >
> > > So, would it be possible to have multiple signature files like this? Please.
> > >
> > > patch-3.X.Y.bz2
> > > patch-3.X.Y.bz2.sign
> > > patch-3.X.Y.gz
> > > patch-3.X.Y.gz.sign
> > > patch-3.X.Y.xz
> > > patch-3.X.Y.xz.sign
> > 
> > Nope, sorry, let's try this way instead.  That way we only have to
> > generate one signature, not 3.
> 
> How about one signed message that contains multiple SHA256 sums or whatever?
> 
>   sha256sum patch-3.X.Y.{bz2,gz,xz} | gpg --clearsign -a >patch-3.X.Y.sign
> 
> That allows verification before decompression.
> 
> > If you are really worried about decompressor bugs, then run them in a
> > virtual machine/chroot :)
> 
> I am not amused.
> 
> Greg, please put your security hat on, and look at this from security point
> of view. Decompression after verify removes/closes attacks utilizing yet
> unidentified decompressor bugs or security flaws.

Please realize that we are building this whole thing back up from
scratch here, and some services are not possible to do just yet.

Also realize our constraints.  I don't want to, and in fact it's almost
impossible to, upload the 3 compressed tarballs over a large number of
internet connections that I have access to as I travel around the world.
So, given that you don't want all kernel release maintainers to store
their private keys on the server (remember what just happened), the
ability to generate that signature would be a pretty difficult thing,
right?

So, we are working on a proposal that has a "throw away" signature for
the compressed files, that can be verified by people after they download
the tarball to ensure that they didn't get something "odd".  That will
handle the sha256sum, and allow us to add future compression types with
no need to regenerate the "real" signatures from the original release.

The real check, to verify that this tarball really came from "me" should
be done on the uncompressed tarball, which is what I can sign, and it is
something that you, or anyone else, can reliable duplicate on their own
by just using git and not even downloading the tarball at all.

In other words, we just saved you a MASSIVE bandwidth transation for all
of your future kernel downloads, and you can reliable know that the
tarball you have in your system is what is on the kernel.org servers
without you even having to download it yourself and run those
decompression tools that you don't trus.

And still you complain :)

Hope this helps explain why this is the way it is.  It's as if people
think we are just making it up as we go along...

greg k-h

Re: kernel.org tarball/patch signature files

[Kyle Moffett]

On Mon, Oct 24, 2011 at 21:49, Greg KH <greg@kroah.com> wrote:
> Please realize that we are building this whole thing back up from
> scratch here, and some services are not possible to do just yet.
>
> Also realize our constraints.  I don't want to, and in fact it's almost
> impossible to, upload the 3 compressed tarballs over a large number of
> internet connections that I have access to as I travel around the world.
> So, given that you don't want all kernel release maintainers to store
> their private keys on the server (remember what just happened), the
> ability to generate that signature would be a pretty difficult thing,
> right?
>
> So, we are working on a proposal that has a "throw away" signature for
> the compressed files, that can be verified by people after they download
> the tarball to ensure that they didn't get something "odd".  That will
> handle the sha256sum, and allow us to add future compression types with
> no need to regenerate the "real" signatures from the original release.
>
> The real check, to verify that this tarball really came from "me" should
> be done on the uncompressed tarball, which is what I can sign, and it is
> something that you, or anyone else, can reliable duplicate on their own
> by just using git and not even downloading the tarball at all.
>
> In other words, we just saved you a MASSIVE bandwidth transation for all
> of your future kernel downloads, and you can reliable know that the
> tarball you have in your system is what is on the kernel.org servers
> without you even having to download it yourself and run those
> decompression tools that you don't trus.
>
> And still you complain :)
>
> Hope this helps explain why this is the way it is.  It's as if people
> think we are just making it up as we go along...

Well, in theory it could still be done (albeit not in practice right now).

There is a tool called "pristine-tar" designed to generate a
byte-identical tar.gz given the original source tree.  Obviously with
git-archive you don't need the "tar" portion, but it also includes
"pristine-gz" and "pristine-bz2" tools which tease out and store the
original compression parameters in a small file.  It's commonly used
to be able to "import" a bunch of source files and a tarball into GIT
without storing the entire original tarball as a binary file.

So the kernel developers would locally produce the uncompressed and
compressed tarballs, then digitally sign them and create small
"pristine-{gz,bz2,xz}" archive deltas; the signatures and deltas would
be uploaded.

The kernel.org server would use the original input and compression
parameters to reconstruct the byte-identical compressed archive from
the output of "git archive --format=tar", again with the "pristine-*"
tools.

Unfortunately, while there are "pristine-gz" and "pristine-bz2" tools,
there has not yet been developed a "pristine-xz" tool:
  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499489

So it's not yet reasonably possible for the kernel.org archive, but
sometime in the future it might become so.

Cheers,
Kyle Moffett

-- 
Curious about my work on the Debian powerpcspe port?
I'm keeping a blog here: http://pureperl.blogspot.com/

Re: kernel.org tarball/patch signature files

[H. Peter Anvin]

On 10/25/2011 06:31 AM, Kyle Moffett wrote:
> 
> Unfortunately, while there are "pristine-gz" and "pristine-bz2" tools,
> there has not yet been developed a "pristine-xz" tool:
>   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499489
> 
> So it's not yet reasonably possible for the kernel.org archive, but
> sometime in the future it might become so.
> 

There are a lot of strong reasons to NOT do so, however.

Any time we have something signed by a developer, we have something that
is precious and cannot be replicated by kernel.org staff.

Any time we have something signed by a robot, we have something that
people *will* misinterpret as having security properties that they don't
-- this has unfortunately been amply shown.

Compression formats evolve over time; right now we're concerned with gz,
bz2, and xz, but xz is brand new here and there may very well be new
things in the future.

It would be a very good thing for people to develop tools to run
compressors and decompressors in locked-down boxes.  It should be
possible to run these kinds of programs without access to either network
or filesystem; only read from stdin and out on stdout (and presumably
stderr for errors.)  This would solve problems for much more than just
kernel.org.

	-hpa

Re: kernel.org tarball/patch signature files

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Tue, 25 Oct 2011 10:27:05 +0200, "H. Peter Anvin" said:

> It would be a very good thing for people to develop tools to run
> compressors and decompressors in locked-down boxes.  It should be
> possible to run these kinds of programs without access to either network
> or filesystem; only read from stdin and out on stdout (and presumably
> stderr for errors.)  This would solve problems for much more than just
> kernel.org.

Wasn't there once a kernel hack called seccomp, that only allowed read syscalls
on stdin and writes on stdout and that was it? ;)


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: kernel.org tarball/patch signature files

[H. Peter Anvin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/25/2011 11:13 AM, Valdis.Kletnieks@vt.edu wrote:
> 
> Wasn't there once a kernel hack called seccomp, that only allowed 
> read syscalls on stdin and writes on stdout and that was it? ;)
> 

It's still there (and there are a bunch of other ways to do this in
Linux).  The big thing is to wrapper this up and get people to use it.
 Even better would be to get this into gzip, bzip2 and xz so it
happens automatically!

	-hpa
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOpoIwAAoJEL2gYIVJO6zkzr4P/2NRkQuMXLtaVLLaxGo1CRx+
7PrNNbUqOMCWcLaMJeNQZPabgWzpaUUshtxkUsmDcKuiHzZcXGK8Pp7QmCGVssmR
zPCn/D2bN+JFkqKxAJyTy57OE35ccXGanGd3BtmNe5KQic06SsEZJ/EWFRuTZxDP
t+Ub08/Fq35D8/dw6ZQ3UdFzwGDQt/8Oxs+gVymuYro08Cezz7aL6bYllS+v5Y1m
2EQ9Nk9sBe9wKLLQJqvwTTXl9abhzlugFCcCsK41PZ0fW38s9728NgKru4sn8CmC
O7E3aClUJPEu/58h6WnAavaemEjHPhQn0KqShgn+amgyqRTMbX+uwq5RY49pLWZ2
eEVRHbGkLLHkfQYJuCjM09y5sRGp1HZC3CUPiZoEZTZXTeZ0Fl162v6utyo39dxT
1TJYsYpoYYrjtg5jFBNLh1uQ1ZeI46fOvhP84v9JbSmYBpo/Fb1yb5nj5YflIpUs
XVXVv9mu0Cuz60tuf0ceqAsvoiYaliM56bZjr65pscOPDzTP/qPtY4Q9UfEjeiAO
1N/l3gBJpOcHD3K5E1v3wcfD0PMeAu1W3WfW9Mj4KswPIz3EFQ6h1ja/urGC8GD3
pw19Uv33mDVHAlKFmZ5N2qIAmbmNaXejDmqvZxsvUyvSzqk+EmB33oYzPv+6n76p
IWHwm1SNbBjQI2AlEmmW
=P2oY
-----END PGP SIGNATURE-----

Re: kernel.org tarball/patch signature files

[Jari Ruusu]

Greg KH wrote:
> Also realize our constraints.  I don't want to, and in fact it's almost
> impossible to, upload the 3 compressed tarballs over a large number of
> internet connections that I have access to as I travel around the world.

How about signing just one compressed tarball/patch, and let other
compressed versions be without signature files. Users who want to check
signature will then download the one you signed.

-- 
Jari Ruusu  1024R/3A220F51 5B 4B F9 BB D3 3F 52 E9  DB 1D EB E3 24 0E A9 DD

Re: kernel.org tarball/patch signature files

[Greg KH]

On Tue, Oct 25, 2011 at 09:06:37AM +0300, Jari Ruusu wrote:
> Greg KH wrote:
> > Also realize our constraints.  I don't want to, and in fact it's almost
> > impossible to, upload the 3 compressed tarballs over a large number of
> > internet connections that I have access to as I travel around the world.
> 
> How about signing just one compressed tarball/patch, and let other
> compressed versions be without signature files. Users who want to check
> signature will then download the one you signed.

That's no difference from what we are doing today if you think it
though.

greg k-h

Re: kernel.org tarball/patch signature files

[Jari Ruusu]

Greg KH wrote:
> On Tue, Oct 25, 2011 at 09:06:37AM +0300, Jari Ruusu wrote:
> > How about signing just one compressed tarball/patch, and let other
> > compressed versions be without signature files. Users who want to check
> > signature will then download the one you signed.
> 
> That's no difference from what we are doing today if you think it
> though.

You signing compressed version eliminates exposure to yet undiscovered
decompressor bugs. Pre-emptively removing as many as possible attack
surfaces is the correct securely engineered way of doing things. You signing
decompressed version is opposite of that. You added new attack surface.

Decompressors are complex software. They almost certainly have bugs in
evil-formatted data handling, forever. Reading data from untrusted internet
and feeding that data to decompressor without any authentication, screams
"REMOTE EXPLOIT ME", if you look at it from security point of view.

> Yes, we are working on just that thing, and the foo.tar.sums file will
> be signed with the kernel.org "throwaway" key, so you can check that as
> well.

That sounds much better than current signing practice.

-- 
Jari Ruusu  1024R/3A220F51 5B 4B F9 BB D3 3F 52 E9  DB 1D EB E3 24 0E A9 DD

Re: kernel.org tarball/patch signature files

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On Tue, 25 Oct 2011 03:49:11 +0200, Greg KH said:

> The real check, to verify that this tarball really came from "me" should
> be done on the uncompressed tarball, which is what I can sign, and it is
> something that you, or anyone else, can reliable duplicate on their own
> by just using git and not even downloading the tarball at all.

I'm OK on that part..

> In other words, we just saved you a MASSIVE bandwidth transation for all
> of your future kernel downloads, and you can reliable know that the
> tarball you have in your system is what is on the kernel.org servers
> without you even having to download it yourself and run those
> decompression tools that you don't trus.

If you're building an automated process that will take a just-uploaded foo.tar
and generate foo.tar.{bz2,gz,foozip}, can you add a step that would just do an
'md5sum foo.tar.* > foo.tar.sums'?  Or sha256sum if you're worried about the
crypto weakness issues with MD5. Personally, I'm more interested in the "Did  I
hit a network error that the TCP checksum didn't catch?" case.

No hurry, I know what a beast it can be to redesign systems of this scale.  Just
a would-be-nice...


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: kernel.org tarball/patch signature files

[Greg KH]

On Tue, Oct 25, 2011 at 03:28:37AM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 25 Oct 2011 03:49:11 +0200, Greg KH said:
> 
> > The real check, to verify that this tarball really came from "me" should
> > be done on the uncompressed tarball, which is what I can sign, and it is
> > something that you, or anyone else, can reliable duplicate on their own
> > by just using git and not even downloading the tarball at all.
> 
> I'm OK on that part..
> 
> > In other words, we just saved you a MASSIVE bandwidth transation for all
> > of your future kernel downloads, and you can reliable know that the
> > tarball you have in your system is what is on the kernel.org servers
> > without you even having to download it yourself and run those
> > decompression tools that you don't trus.
> 
> If you're building an automated process that will take a just-uploaded foo.tar
> and generate foo.tar.{bz2,gz,foozip}, can you add a step that would just do an
> 'md5sum foo.tar.* > foo.tar.sums'?  Or sha256sum if you're worried about the
> crypto weakness issues with MD5. Personally, I'm more interested in the "Did  I
> hit a network error that the TCP checksum didn't catch?" case.

Yes, we are working on just that thing, and the foo.tar.sums file will
be signed with the kernel.org "throwaway" key, so you can check that as
well.

greg k-h

Re: kernel.org tarball/patch signature files

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

On Sun, 23 Oct 2011 13:37:27 +0200, Greg KH said:
> If you are really worried about decompressor bugs, then run them in a
> virtual machine/chroot :)

Of more concern than bugs are errors during download.  Yes, TCP has a checksum,
which is a CRC that quite frankly sucks when we're talking the amount of data
that kernel.org moves. So there's a non-zero chance you'll get bad data
downloaded.  And you really want to do a more effective data check  (MD5 or SHA
sum, or a PGP signature) *before* you decompress, in case the corrupted data
causes a spew of gigabytes of trash and fills your filesystem.


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]