checksums situation

checksums situation

[Marcin Juszkiewicz]

Hi

It is nearly two years since conf/checksums.ini was introduced. We 
populated it with over 6000 entries during that time (mostly by 
automatic fetching of all source on CELF and EWI machines). But there 
are problems with it's format.

First problem is how to use it when overlays are used. Not every entry 
of SRC_URI can be provided into public (NDA binaries etc) so people 
starts to switch off checking of checksums which is not good idea. OE 
reads only one copy of file. OK, such users can copy it to their overlay 
and extend with own entries which will work with properly created value 
of BBPATH variable. But this does not sound as solution.

Second problem which I hit few times is using of DEBIAN_MIRROR etc 
variables. We allow users to choose fastest Debian, kernel.org, 
SourceForge mirror but this also mean that URLs for sources will differ 
from ones in checksums.ini. Of course users can submit new entries but 
we will have duplicates then (we have about 700 of them already). There 
were ideas how to solve that which were sent to this mailing list.

One of ideas was to use filename as key (instead of URL). But I can 
imagine few situations when it will be wrong way (similiar names for 
different files, different contents of same name tarballs etc).

Other was to use filename as key and add "url[0-xx]" fields which will 
list alternative locations. This one looks better but still does not 
solve situation when someone use DEBIAN_MIRROR which is not present in 
checksums.ini file.

Possible solution would be going to keeping MD5/SHA256 sums in recipes. 
We used 'md5' parametr in SRC_URI for that in past. We can switch to 
using it for md5 and sha256, but some purists can say that it will make 
SRC_URI entries much longer.

Example:

SRC_URI = "${SOURCEFORGE_MIRROR}/zziplib/zziplib-${PV}.tar.bz2"

will change to:

SRC_URI = 
"${SOURCEFORGE_MIRROR}/zziplib/zziplib-${PV}.tar.bz2;md5=a6538f6c44ceeed0ed7e8e356f444168;sha256=f684397ce39ec400ba3369521892b7c3a8711d3ef1be59115db9f8d57707bbb8"

which is very long line.

This solution also has one nasty part - now we can keep SRC_URI for 
multiple versions in common file, but if we switch to storing it in 
SRC_URI we will have to change that.

Other solution proposed on IRC was to keep checksums in extra file in 
each directory of packages/ subdirectory. I think that it is not best 
but sounds better then one file.

What do you think? Which way we should go? Do you have other ideas?

Regards, 
-- 
JID:      hrw@jabber.org
Website:  http://marcin.juszkiewicz.com.pl/
LinkedIn: http://www.linkedin.com/in/marcinjuszkiewicz

Re: checksums situation

[Otavio Salvador]

Marcin Juszkiewicz <openembedded@haerwu.biz> writes:

<...>
> This solution also has one nasty part - now we can keep SRC_URI for 
> multiple versions in common file, but if we switch to storing it in 
> SRC_URI we will have to change that.
>
> Other solution proposed on IRC was to keep checksums in extra file in 
> each directory of packages/ subdirectory. I think that it is not best 
> but sounds better then one file.
>
> What do you think? Which way we should go? Do you have other ideas?
<...>

What about having a checksums for _each_ recipe?

foo_1.0.bb
foo_1.0.md5sum

This could have the valid one; this also avoid the sorting problem and
like.

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."

Re: checksums situation

[Ihar Hrachyshka]

On Fri, Feb 13, 2009 at 7:08 PM, Otavio Salvador <otavio@debian.org> wrote:
> Marcin Juszkiewicz <openembedded@haerwu.biz> writes:
>
> <...>
>> This solution also has one nasty part - now we can keep SRC_URI for
>> multiple versions in common file, but if we switch to storing it in
>> SRC_URI we will have to change that.
>>
>> Other solution proposed on IRC was to keep checksums in extra file in
>> each directory of packages/ subdirectory. I think that it is not best
>> but sounds better then one file.
>>
>> What do you think? Which way we should go? Do you have other ideas?
> <...>
>
> What about having a checksums for _each_ recipe?
>
> foo_1.0.bb
> foo_1.0.md5sum

This will waste directories and will make tree navigation harder.

>
> This could have the valid one; this also avoid the sorting problem and
> like.
>
> --
>        O T A V I O    S A L V A D O R
> ---------------------------------------------
>  E-mail: otavio@debian.org      UIN: 5906116
>  GNU/Linux User: 239058     GPG ID: 49A5F855
>  Home Page: http://otavio.ossystems.com.br
> ---------------------------------------------
> "Microsoft sells you Windows ... Linux gives
>  you the whole house."
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: checksums situation

[Otavio Salvador]

Ihar Hrachyshka <ihar.hrachyshka@gmail.com> writes:

> On Fri, Feb 13, 2009 at 7:08 PM, Otavio Salvador <otavio@debian.org> wrote:
>> Marcin Juszkiewicz <openembedded@haerwu.biz> writes:
>>
>> <...>
>>> This solution also has one nasty part - now we can keep SRC_URI for
>>> multiple versions in common file, but if we switch to storing it in
>>> SRC_URI we will have to change that.
>>>
>>> Other solution proposed on IRC was to keep checksums in extra file in
>>> each directory of packages/ subdirectory. I think that it is not best
>>> but sounds better then one file.
>>>
>>> What do you think? Which way we should go? Do you have other ideas?
>> <...>
>>
>> What about having a checksums for _each_ recipe?
>>
>> foo_1.0.bb
>> foo_1.0.md5sum
>
> This will waste directories and will make tree navigation harder.

Well; so let's just create a md5sums directory at OE topdir and add the
dirs and files there.

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."

Re: checksums situation

[Leon Woestenberg]

Hello,

On Fri, Feb 13, 2009 at 7:37 PM, Otavio Salvador <otavio@debian.org> wrote:
> Ihar Hrachyshka <ihar.hrachyshka@gmail.com> writes:
>
>> On Fri, Feb 13, 2009 at 7:08 PM, Otavio Salvador <otavio@debian.org> wrote:
>>> Marcin Juszkiewicz <openembedded@haerwu.biz> writes:
>>>
>>>>
>>>> What do you think? Which way we should go? Do you have other ideas?
>>> <...>
>>>

I have thought about this (and our current fetch and checksum) a lot
in the past.

>> Keep the checksums.ini file, but change it's layout a bit.

In all cases, we need to REMOVE the URL of a package from the key, as
we are ONLY interested in its contents.

So, solution 1:

>> Make the key of a package: <filename>.

However, I would opt for solution 2:

>> Make the key of a source package: <filename, sha1sum, md5sum>.

Yes, that's right. The key of a package is it's filename plus it's contents.
Filename because we humans identify it by its name.
The dual checksum because we can guarantee the desired contents.

Next step, is we adapt the fetcher to:

- find URLs for a package given any of it's (sub)key.
- check the package against its dual sums.

I have written a working proof-of-concept of a
package-fetcher/checker/ url-cache/... in C using sqlite (instead of
checksums.ini)

which is public here:

http://www.sidebranch.com/leon/witpa_20080730.tar.bz2

It can import checksums.ini though.


To summarize: remove URL from the package key.

The .bb names the upstream URL and filename.

The filename ALONE is used as the key into the file database.
The file database caches URLs for the package, and has its md5sum and sha1sum.

The fetcher tries to fetch from the cached URLs or can fall back to
Google for the file by either filename or any checksum.
It can verify correctness against the checksums.


Regards,
-- 
Leon

Re: checksums situation

[mike]

I followed you up till:
> The filename ALONE is used as the key into the file database.

I'm having trouble understanding how this will work with names like 
"Main.java" currently in the checksums.  Is there another discussion about
naming collisions?

Cheers!
Mike

Re: checksums situation

[John Willis]

> >> <...>
> >>> This solution also has one nasty part - now we can keep SRC_URI for
> >>> multiple versions in common file, but if we switch to storing it in
> >>> SRC_URI we will have to change that.
> >>>
> >>> Other solution proposed on IRC was to keep checksums in extra file
> in
> >>> each directory of packages/ subdirectory. I think that it is not
> best
> >>> but sounds better then one file.
> >>>
> >>> What do you think? Which way we should go? Do you have other ideas?
> >> <...>
> >>
> >> What about having a checksums for _each_ recipe?
> >>
> >> foo_1.0.bb
> >> foo_1.0.md5sum
> >
> > This will waste directories and will make tree navigation harder.
> 
> Well; so let's just create a md5sums directory at OE topdir and add the
> dirs and files there.

Why not take that a little further and just have a checksums dir at the root
with a 'downloadname'.manifest with MD5 and SHA256 hashes? Take the
formatting from Gentoo if people want to. Plenty of scope to be flexible in
the future then without committing to some name stamped method ;-).

I think the URL pre filename is a bit of a non issue as it stands mind you
and I am not sure that the current checksums.ini approach is as bad as all
that if it dropped dupes and URLs. Maybe a little refinement is needed but a
wholesale change seems a little extreme (but I can see pros and cons for
both). As long as it is kept clean and contained and does not break overlays
to much I guess I am happy.

Regards,

John

Re: checksums situation

[Phil Blundell]

On Fri, 2009-02-13 at 16:37 -0200, Otavio Salvador wrote:
> Ihar Hrachyshka <ihar.hrachyshka@gmail.com> writes:
> 
> > On Fri, Feb 13, 2009 at 7:08 PM, Otavio Salvador <otavio@debian.org> wrote:
> >> Marcin Juszkiewicz <openembedded@haerwu.biz> writes:
> >>
> >> <...>
> >>> This solution also has one nasty part - now we can keep SRC_URI for
> >>> multiple versions in common file, but if we switch to storing it in
> >>> SRC_URI we will have to change that.
> >>>
> >>> Other solution proposed on IRC was to keep checksums in extra file in
> >>> each directory of packages/ subdirectory. I think that it is not best
> >>> but sounds better then one file.
> >>>
> >>> What do you think? Which way we should go? Do you have other ideas?
> >> <...>
> >>
> >> What about having a checksums for _each_ recipe?
> >>
> >> foo_1.0.bb
> >> foo_1.0.md5sum
> >
> > This will waste directories and will make tree navigation harder.
> 
> Well; so let's just create a md5sums directory at OE topdir and add the
> dirs and files there.

That would work, and it'd certainly be better than the single monolithic
checksums.ini, but it would still be a bit of a nuisance for overlay
users. 

Your original suggestion of a .md5sum file sounds reasonable enough to
me; I'm not really sure I buy the argument about wasting directories.
But I still think it would be even better to find a way to put the
md5sums inside the .bb file itself.

For packages where the SRC_URI is defined locally in the .bb file then
there's no real problem: the existing ";md5sum=..." notation will work
fine.  For packages where SRC_URI is inherited from some class or
include file, obviously that doesn't work.  In that case I guess the
easiest solution is just to put the checksums in a separate variable,
something like:

CHECKSUMS = "foo_${PV}.tar.bz/md5=076e48835bee2609a3713b35fb89631c"
CHECKSUMS += "foo_1.0-${PR}.patch/sha1=9bfbeca5db4887b62826da6d0169c62c244f6188"

p.

Re: checksums situation

[Otavio Salvador]

Phil Blundell <pb@reciva.com> writes:

> For packages where the SRC_URI is defined locally in the .bb file then
> there's no real problem: the existing ";md5sum=..." notation will work
> fine.  For packages where SRC_URI is inherited from some class or
> include file, obviously that doesn't work.  In that case I guess the
> easiest solution is just to put the checksums in a separate variable,
> something like:
>
> CHECKSUMS = "foo_${PV}.tar.bz/md5=076e48835bee2609a3713b35fb89631c"
> CHECKSUMS += "foo_1.0-${PR}.patch/sha1=9bfbeca5db4887b62826da6d0169c62c244f6188"

Yes; I like this solution since it allows it to be with the bb
itself. BTW a nice alternative would be 

#v+
MD5SUMS = "foo_${PV}.tar.bz/076e48835bee2609a3713b35fb89631c"
SHA1SUMS += "foo_1.0-${PR}.patch/9bfbeca5db4887b62826da6d0169c62c244f6188"
#v-

This is more or less what you though however avoids the requirement to
have it specified at the file.

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."

Re: checksums situation

[Tom Rini]

On Fri, Feb 13, 2009 at 05:28:08PM +0100, Marcin Juszkiewicz wrote:
> 
> Hi
> 
> It is nearly two years since conf/checksums.ini was introduced. We 
> populated it with over 6000 entries during that time (mostly by 
> automatic fetching of all source on CELF and EWI machines). But there 
> are problems with it's format.
> 
> First problem is how to use it when overlays are used. Not every entry 
> of SRC_URI can be provided into public (NDA binaries etc) so people 
> starts to switch off checking of checksums which is not good idea. OE 
> reads only one copy of file. OK, such users can copy it to their overlay 
> and extend with own entries which will work with properly created value 
> of BBPATH variable. But this does not sound as solution.

People with overlays already need to setup BBPATH correctly, so I don't
think this part is an issue.

[snip]
> Other was to use filename as key and add "url[0-xx]" fields which will 
> list alternative locations. This one looks better but still does not 
> solve situation when someone use DEBIAN_MIRROR which is not present in 
> checksums.ini file.

Could we perhaps have a literal:
['${DEBIAN_MIRROR}/...']
md5sum:...
sha256:...

format instead?

-- 
Tom Rini

Re: checksums situation

[Andrea Adami]

> Other solution proposed on IRC was to keep checksums in extra file in
> each directory of packages/ subdirectory. I think that it is not best
> but sounds better then one file.

I think Gentoo solved the problem very well:

http://devmanual.gentoo.org/general-concepts/digest-and-manifest/index.html


Regards

Andrea

Re: checksums situation

[Andrea Adami]

This is the second step :

http://www.gentoo.org/news/20080204-digest-files-removed.xml

Regards

Andrea

Re: checksums situation

[Koen Kooi]

On 13-02-09 17:28, Marcin Juszkiewicz wrote:

> One of ideas was to use filename as key (instead of URL). But I can
> imagine few situations when it will be wrong way (similiar names for
> different files, different contents of same name tarballs etc).

That's a non-issue since we already plonk everything in DL_DIR without 
looking twice.

regards,

Koen

Re: checksums situation

[Yuri Bushmelev]

Hello!

[skipped]
> What do you think? Which way we should go? Do you have other ideas?

Well, I can suggest scheme that is very like one created in FreeBSD ports
(I just working with it about 8 years).

In recipe directory part and filename part of SRC_URI are stored separate.
E.g. we can continue storing directory part in SRC_URI but introduce new
variable to store filename part (e.g. DISTFILES - we can have more than
one source file per package).

Example:
# Source directories/mirrors:
SRC_URI = "${SOURCEFORGE_MIRROR}/zziplib/ http://somesite/~user/";
# Filename
DISTFILES = "zziplib-${PV}.tar.bz2 zziplib_extra-${PV}.tar.bz2"

Then we can use words in DISTFILES as keys to checking checksums.

But I don't see any good way to store checksums for multiple files in
recipe. We can't use this:

MD5SUM_zziplib-${PV}.tar.bz2="a6538f6c44ceeed0ed7e8e356f444168"

because of possible some special chars in file name.
We can possible use arrays:

MD5SUM["zziplib-${PV}.tar.bz2"]="a6538f6c44ceeed0ed7e8e356f444168"

but seems that no one uses arrays in recipes.

Other way is using some predefined functions via ${@bb.data....}

Or we can store all checksums in separate file per package directory
(like manifest in gentoo or distinfo in FreeBSD).

E.g. distinfo of russian xmms port from FreeBSD:

MD5 (xmms-1.2.11.tar.bz2) = f3e6dbaf0b3f571a532ab575656be506
SHA256 (xmms-1.2.11.tar.bz2) = 7ec15c56632b6c82e61ccddeaefd372359af2f005708a58cdf3951c574b20390
SIZE (xmms-1.2.11.tar.bz2) = 2581032
MD5 (RusXMMS2-csa41.tar.bz2) = 7d89f35c80849dae89b81cbb57026e57
SHA256 (RusXMMS2-csa41.tar.bz2) = e64df1956502e48c09ca60262efb7f1953a76d82a70c801e4797ca81e130e8d0
SIZE (RusXMMS2-csa41.tar.bz2) = 96642

Links:
1. FreeBSD Porter's Handbook: http://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/index.html

-- 
Yuri Bushmelev

Re: checksums situation

[Tom Rini]

On Fri, Feb 13, 2009 at 05:28:08PM +0100, Marcin Juszkiewicz wrote:

> 
> Hi
> 
> It is nearly two years since conf/checksums.ini was introduced. We 
> populated it with over 6000 entries during that time (mostly by 
> automatic fetching of all source on CELF and EWI machines). But there 
> are problems with it's format.

I'm going to make a different suggestion.  Lets just drop it.  Part of
why I say this is that we went back on it being a mandatory thing
because it was a burden.  And we're already back to adding stuff without
checksums again (gstreamer, libdaemon on this build so far..).  And, is
anyone doing more then 'cat tmp/checksums.ini >>
openembedded/conf/checksums.ini && python the-sorter' ?  And by more I
mean check for an upstream .md5 or similar, to make sure this is really
right?

Perhaps we're going about this wrong.  Perhaps we want to see if from
upstream we can get ${URI}.md5, .md5sum .sha1 or any of the other
standard mechanisms and seeing if we get that as a match (and
pre-seeding ourselves with just the hashes for the tools that check
hashes).

Oh, and in the 5 minutes the above took to write, gst-plugins-base
failed to have a checksum too.

-- 
Tom Rini

Re: checksums situation

[Tom Rini]

On Mon, Feb 23, 2009 at 11:46:39PM -0700, Tom Rini wrote:
> On Fri, Feb 13, 2009 at 05:28:08PM +0100, Marcin Juszkiewicz wrote:
> 
> > 
> > Hi
> > 
> > It is nearly two years since conf/checksums.ini was introduced. We 
> > populated it with over 6000 entries during that time (mostly by 
> > automatic fetching of all source on CELF and EWI machines). But there 
> > are problems with it's format.
> 
> I'm going to make a different suggestion.  Lets just drop it.  Part of
> why I say this is that we went back on it being a mandatory thing
> because it was a burden.  And we're already back to adding stuff without
> checksums again (gstreamer, libdaemon on this build so far..).
[snip]
> Oh, and in the 5 minutes the above took to write, gst-plugins-base
> failed to have a checksum too.

OK, this is a me problem.  But the rest of my point stands.  I'm willing
to bet most people aren't making sure the md5/sha1 is right with
upstream, and I think we'd be better off at trying to automatically grab
an upstream verification and use that.

-- 
Tom Rini

Re: checksums situation

[Marcin Juszkiewicz]

On Tuesday 24 of February 2009 07:46:39 Tom Rini wrote:

> And, is anyone doing more then 'cat tmp/checksums.ini >>
> openembedded/conf/checksums.ini && python the-sorter' ?  And by more
> I mean check for an upstream .md5 or similar, to make sure this is
> really right?

commit 67aa127f4b4cb2e4083a02adfddd7d82534751a3
Author: Marcin Juszkiewicz <hrw@openembedded.org>
Date:   Thu Nov 13 18:07:47 2008 +0100

    checksums.ini: add 1212 entries from test fetch build

    This is result of simple shell script and few hours of fetching:
    for recipe in $OEROOT/packages/*/*.bb
    do
      bitbake -b $recipe -cfetch
    done

This was done with not so empty DL_DIR but most of entries were fresh 
fetched. If someone has machine with good network connection (I no 
longer have access to machine where I was running that test) then feel 
free to call that loop with empty DL_DIR.

Regards, 
-- 
JID:      hrw@jabber.org
Website:  http://marcin.juszkiewicz.com.pl/
LinkedIn: http://www.linkedin.com/in/marcinjuszkiewicz

Re: checksums situation

[Tom Rini]

On Tue, Feb 24, 2009 at 09:49:04AM +0100, Marcin Juszkiewicz wrote:
> On Tuesday 24 of February 2009 07:46:39 Tom Rini wrote:
> 
> > And, is anyone doing more then 'cat tmp/checksums.ini >>
> > openembedded/conf/checksums.ini && python the-sorter' ?  And by more
> > I mean check for an upstream .md5 or similar, to make sure this is
> > really right?
> 
> commit 67aa127f4b4cb2e4083a02adfddd7d82534751a3
> Author: Marcin Juszkiewicz <hrw@openembedded.org>
> Date:   Thu Nov 13 18:07:47 2008 +0100
> 
>     checksums.ini: add 1212 entries from test fetch build
> 
>     This is result of simple shell script and few hours of fetching:
>     for recipe in $OEROOT/packages/*/*.bb
>     do
>       bitbake -b $recipe -cfetch
>     done
> 
> This was done with not so empty DL_DIR but most of entries were fresh 
> fetched. If someone has machine with good network connection (I no 
> longer have access to machine where I was running that test) then feel 
> free to call that loop with empty DL_DIR.

IOW, no.  We're just using checksums.ini as a "at one point, it was ..."
and not actually making sure that it's the right checksum.

-- 
Tom Rini

Re: checksums situation

[Michael 'Mickey' Lauer]

Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
> I'm going to make a different suggestion.  Lets just drop it. 

I'm in favour of this. I don't think they give us the safety we want and
they introduce more inconvenience.

Cheers,

-- 
:M:

Re: checksums situation

[Angus Ainslie]

On Tue, 2009-02-24 at 17:13 +0100, Michael 'Mickey' Lauer wrote:
> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
> > I'm going to make a different suggestion.  Lets just drop it. 
> 
> I'm in favour of this. I don't think they give us the safety we want and
> they introduce more inconvenience.
> 
> Cheers,
> 

Couldn't the default be a stern warning if the checksums don't match and
distro's that want it could change it to an error ?

Angus

Re: checksums situation

[Tom Rini]

On Tue, Feb 24, 2009 at 09:25:26AM -0700, Angus Ainslie wrote:
> On Tue, 2009-02-24 at 17:13 +0100, Michael 'Mickey' Lauer wrote:
> > Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
> > > I'm going to make a different suggestion.  Lets just drop it. 
> > 
> > I'm in favour of this. I don't think they give us the safety we want and
> > they introduce more inconvenience.
> > 
> > Cheers,
> > 
> 
> Couldn't the default be a stern warning if the checksums don't match and
> distro's that want it could change it to an error ?

That's not what's causing heartburn.  What's causing it is the far more
sources than checksums we have today.  Which got me thinking that (as I
said just now in another email) we have a system that's fine for
checking if the tarball changed under me, but worthless for "are the
sources what upstream says they should be", ie guarding against trojans.

-- 
Tom Rini

Re: checksums situation

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

Michael 'Mickey' Lauer wrote:
> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>> I'm going to make a different suggestion.  Lets just drop it. 
> 
> I'm in favour of this. I don't think they give us the safety we want and
> they introduce more inconvenience.

Drop checksum checking? I am not in favour of this. The check does 
provide valuable reassurance that the source's are not changing in 
"funny" ways. This is valuable data for many people.

We have ways of disabling the checks for people who are less concerned 
with image integrity.

I agree the current implementation is not perfect, but it is a good 
compromise.

The only thing I would like to see is a way to keep a local checksums 
file for people using overlays and other out of tree sw.

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: checksums situation

[Tom Rini]

On Tue, Feb 24, 2009 at 11:28:46AM -0500, Philip Balister wrote:
> Michael 'Mickey' Lauer wrote:
>> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>>> I'm going to make a different suggestion.  Lets just drop it. 
>>
>> I'm in favour of this. I don't think they give us the safety we want and
>> they introduce more inconvenience.
>
> Drop checksum checking? I am not in favour of this. The check does  
> provide valuable reassurance that the source's are not changing in  
> "funny" ways. This is valuable data for many people.
>
> We have ways of disabling the checks for people who are less concerned  
> with image integrity.
>
> I agree the current implementation is not perfect, but it is a good  
> compromise.
>
> The only thing I would like to see is a way to keep a local checksums  
> file for people using overlays and other out of tree sw.

So in the case of "I care if the file changed under me" what we have now
is OK.  But it's worthless in the case of "I care that the file is what
upstream says it should be".

-- 
Tom Rini

Re: checksums situation

[GNUtoo]

> Michael 'Mickey' Lauer wrote:
>> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>>> I'm going to make a different suggestion.  Lets just drop it.
>>
>> I'm in favour of this. I don't think they give us the safety we want and
>> they introduce more inconvenience.
>
> Drop checksum checking? I am not in favour of this. The check does
> provide valuable reassurance that the source's are not changing in
> "funny" ways. This is valuable data for many people.
>
> We have ways of disabling the checks for people who are less concerned
> with image integrity.
>
> I agree the current implementation is not perfect, but it is a good
> compromise.
I totally agree with keeping the checksums for 2 reasons:
*to be shure that the package stays the same over time: upstream could
have the bad idea of changing the tarball to put a fix into it without
changing the tarball's name
*security: that is also very important as threats are growing:
**nasty threats are growing(at least that's what I heard in the press)
such as computer infections(malware etc..)
**intrusion into citizen's computer by the state is legal in some countries:
http://yro.slashdot.org/article.pl?sid=09/01/04/2042242
**sometimes distributions repository get compromised
http://www.vnunet.com/vnunet/news/2224622/red-hat-admits-getting-hacked
And theses people could be interested in tampering some repositories...
I don't know every possible uses of openembedded but some people could:
*run ssh client on it
*have sensitive things on it(such as phones,pdas)
*run services(such as networking on hardware) where the availability is
important
*run industrial/military applications on it
etc...
Denis.

Re: checksums situation

[Tom Rini]

On Tue, Feb 24, 2009 at 11:10:05PM +0100, GNUtoo wrote:
[snip]
> *security: that is also very important as threats are growing:
> **nasty threats are growing(at least that's what I heard in the press)
> such as computer infections(malware etc..)
> **intrusion into citizen's computer by the state is legal in some countries:
> http://yro.slashdot.org/article.pl?sid=09/01/04/2042242
> **sometimes distributions repository get compromised
> http://www.vnunet.com/vnunet/news/2224622/red-hat-admits-getting-hacked

This is exactly why I think we should drop what we have now.  If we want
security then we need to do something like verify the signed md5 (or
sha or what have you) that projects that care enough to do it provide.
Otherwise it's a false sense of security we've got.

-- 
Tom Rini

Re: checksums situation

[Phil Blundell]

On Tue, 2009-02-24 at 23:10 +0100, GNUtoo wrote:
> *security: that is also very important as threats are growing:
> **nasty threats are growing(at least that's what I heard in the press)
> such as computer infections(malware etc..)
> **intrusion into citizen's computer by the state is legal in some countries:
> http://yro.slashdot.org/article.pl?sid=09/01/04/2042242
> **sometimes distributions repository get compromised
> http://www.vnunet.com/vnunet/news/2224622/red-hat-admits-getting-hacked

I think Tom Rini's point, which is a good one, was that the existing
checksums.ini workflow doesn't actually do anything to protect against
those threats, since there isn't any validation of the checksum against
an authoritative source.  Right now, the checksum that you get in
checksums.ini is just what was computed by the first person to build the
corresponding .bb file: if the file had been compromised before that, we
would never know.

Even in the case where the upstream tarball changes unexpectedly, I
wouldn't be at all surprised if some or other developer just decided
that the checksums.ini entry was wrong and quietly checked in a
"correction" for it.  So I tend to agree with Tom, the checksums in
their current form do not really buy much.

If you want true source code verification then there needs to be some
cross-check against a trusted fingerprint from upstream.  The obvious
way to do that would be for upstreams to start GPG-signing the release
tarballs and then we could just verify that the signatures matched.  

Some upstream developers do mention the md5sum of the distribution
tarballs in their release announcements, and in those cases it would
already be possible to verify that the archives are good with a
reasonable degree of confidence.  (Of course, it's possible that all
accessible copies of the release announcement might also have been
compromised, but the likelihood of this being the case diminishes if
they are obtained by a different route to the tarball itself.)  However,
it doesn't seem like even this level of checking is currently being
applied to checksums.ini's contents.

p.

Re: checksums situation

[GNUtoo]

> On Tue, 2009-02-24 at 23:10 +0100, GNUtoo wrote:
>> *security: that is also very important as threats are growing:
>> **nasty threats are growing(at least that's what I heard in the press)
>> such as computer infections(malware etc..)
>> **intrusion into citizen's computer by the state is legal in some
>> countries:
>> http://yro.slashdot.org/article.pl?sid=09/01/04/2042242
>> **sometimes distributions repository get compromised
>> http://www.vnunet.com/vnunet/news/2224622/red-hat-admits-getting-hacked
>
> I think Tom Rini's point, which is a good one, was that the existing
> checksums.ini workflow doesn't actually do anything to protect against
> those threats, since there isn't any validation of the checksum against
> an authoritative source.  Right now, the checksum that you get in
> checksums.ini is just what was computed by the first person to build the
> corresponding .bb file: if the file had been compromised before that, we
> would never know.
>
> Even in the case where the upstream tarball changes unexpectedly, I
> wouldn't be at all surprised if some or other developer just decided
> that the checksums.ini entry was wrong and quietly checked in a
> "correction" for it.  So I tend to agree with Tom, the checksums in
> their current form do not really buy much.
>
> If you want true source code verification then there needs to be some
> cross-check against a trusted fingerprint from upstream.  The obvious
> way to do that would be for upstreams to start GPG-signing the release
> tarballs and then we could just verify that the signatures matched.
>
> Some upstream developers do mention the md5sum of the distribution
> tarballs in their release announcements, and in those cases it would
> already be possible to verify that the archives are good with a
> reasonable degree of confidence.  (Of course, it's possible that all
> accessible copies of the release announcement might also have been
> compromised, but the likelihood of this being the case diminishes if
> they are obtained by a different route to the tarball itself.)  However,
> it doesn't seem like even this level of checking is currently being
> applied to checksums.ini's contents.
Sorry I misunderstood his point...
so I think that Generating the url in a way or in another is a good idea
Denis.

Re: checksums situation

[Richard Purdie]

On Tue, 2009-02-24 at 22:29 +0000, Phil Blundell wrote:
> I think Tom Rini's point, which is a good one, was that the existing
> checksums.ini workflow doesn't actually do anything to protect against
> those threats, since there isn't any validation of the checksum against
> an authoritative source.  Right now, the checksum that you get in
> checksums.ini is just what was computed by the first person to build the
> corresponding .bb file: if the file had been compromised before that, we
> would never know.
> 
> Even in the case where the upstream tarball changes unexpectedly, I
> wouldn't be at all surprised if some or other developer just decided
> that the checksums.ini entry was wrong and quietly checked in a
> "correction" for it.  So I tend to agree with Tom, the checksums in
> their current form do not really buy much.

I think checksums.ini even in its current form is useful. If the
checksum matches it tells us that your build configuration at least
matches the configuration the original recipe submitter had. It also
spots corrupted downloads and cases where upstream changes and we have
seen those cases. People shouldn't be silently checking in those changes
and if they do, would most likely get spotted by people with the old
version in DL_DIR.

So to say it does buy much isn't really fair although I agree if you
want verification of the sources at every level, its not good enough. If
we want to do better, all it takes is someone to do the work. We could
have a "verified-checksums.ini" file with some policy attached to it
which is used instead of or supplements checksums.ini...

For overlays, I'd suggest just scanning the overlay directories (BBPATH)
for more checksum.ini files like we do with conf/class files...

-- 
RP

Re: checksums situation

[Denys Dmytriyenko]

On Wed, Feb 25, 2009 at 09:09:40AM +0000, Richard Purdie wrote:
> On Tue, 2009-02-24 at 22:29 +0000, Phil Blundell wrote:
> > I think Tom Rini's point, which is a good one, was that the existing
> > checksums.ini workflow doesn't actually do anything to protect against
> > those threats, since there isn't any validation of the checksum against
> > an authoritative source.  Right now, the checksum that you get in
> > checksums.ini is just what was computed by the first person to build the
> > corresponding .bb file: if the file had been compromised before that, we
> > would never know.
> > 
> > Even in the case where the upstream tarball changes unexpectedly, I
> > wouldn't be at all surprised if some or other developer just decided
> > that the checksums.ini entry was wrong and quietly checked in a
> > "correction" for it.  So I tend to agree with Tom, the checksums in
> > their current form do not really buy much.
> 
> I think checksums.ini even in its current form is useful. If the
> checksum matches it tells us that your build configuration at least
> matches the configuration the original recipe submitter had. It also
> spots corrupted downloads and cases where upstream changes and we have
> seen those cases. People shouldn't be silently checking in those changes
> and if they do, would most likely get spotted by people with the old
> version in DL_DIR.
> 
> So to say it does buy much isn't really fair although I agree if you
> want verification of the sources at every level, its not good enough. If
> we want to do better, all it takes is someone to do the work. We could
> have a "verified-checksums.ini" file with some policy attached to it
> which is used instead of or supplements checksums.ini...
> 
> For overlays, I'd suggest just scanning the overlay directories (BBPATH)
> for more checksum.ini files like we do with conf/class files...

Richard,

I'm not in the position to explain how Bitbake works to the Bitbake's core 
developer and maintainer. :) So, below is just my understanding of how it 
works.

So, for overlays it currently works the same way for checksums.ini as it does 
for conf/class files - it uses only one instance of a file with the same name, 
depending on the priority either from overlay or from upstream org.oe.dev. In 
other words - if my overlay has a higher priority, it would use my 
checksums.ini INSTEAD of the upstream one in org.oe.dev. Same with conf/class 
files - if I have a copy of base.bbclass in my overlay, it REPLACES the one 
from upstream org.oe.dev...

Are you suggesting to simply combine checksums.ini files from overlay and 
org.oe.dev? That may work as long as duplicates are handled properly - I guess 
they can be overwritten based on the priorities - i.e. the entry with higher 
priority would replace the duplicate one. But it is not the way it works now. 
And this will definitely not work for conf/class files...

Or maybe I'm just completely missing your point.

-- 
Denys

Re: checksums situation

[Richard Purdie]

On Wed, 2009-02-25 at 18:04 -0500, Denys Dmytriyenko wrote:
> I'm not in the position to explain how Bitbake works to the Bitbake's core 
> developer and maintainer. :) So, below is just my understanding of how it 
> works.
> 
> So, for overlays it currently works the same way for checksums.ini as it does 
> for conf/class files - it uses only one instance of a file with the same name, 
> depending on the priority either from overlay or from upstream org.oe.dev. In 
> other words - if my overlay has a higher priority, it would use my 
> checksums.ini INSTEAD of the upstream one in org.oe.dev. Same with conf/class 
> files - if I have a copy of base.bbclass in my overlay, it REPLACES the one 
> from upstream org.oe.dev...

There are two different mechanisms that bitbake uses to find files. One
is the conf/class mechanism using BBPATH, the other is the BBFILES
mechanism and collections (overlays).

For BBFILES, we have some globs which expand into a list of files.
Depending on some pattern matching, they get given different priorities
so files from a given collection can "win" compared to those from
another. This is what you describe above and as you rightly point out it
wouldn't work for checksums.ini files.

When we need a conf file, say "bitbake.conf" we search through BBPATH,
appending "conf/" to each entry and seeing if it exists. The first match
found wins. Class files work similarly.

For checksum.ini I'm proposing we'd go through BBPATH and append all the
checksum.ini files found together to form one large complete version.
People's overlay/collections can then supplement the checksum data
easily. There certainly isn't a technical reason I can see why this
wouldn't work.

Cheers,

Richard

Re: checksums situation

[Otavio Salvador]

Richard Purdie <rpurdie@rpsys.net> writes:


[...]

> For checksum.ini I'm proposing we'd go through BBPATH and append all the
> checksum.ini files found together to form one large complete version.
> People's overlay/collections can then supplement the checksum data
> easily. There certainly isn't a technical reason I can see why this
> wouldn't work.

I fully agree that this is technically possible and even a desired
feature if we stay with a checksums.ini file however I fail to see how
it could help to solve the total mess we have in current checksums.ini
commits.

Even worse is the lack of a policy to add/change items there.

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: checksums situation

[Otavio Salvador]

Michael 'Mickey' Lauer <mlauer@vanille-media.de> writes:

> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>> I'm going to make a different suggestion.  Lets just drop it. 
>
> I'm in favour of this. I don't think they give us the safety we want and
> they introduce more inconvenience.

After reading the thread I'm also in favour of this.

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: checksums situation

[Ihar Hrachyshka]

On Tue, Feb 24, 2009 at 8:01 PM, Otavio Salvador
<otavio@ossystems.com.br> wrote:
> Michael 'Mickey' Lauer <mlauer@vanille-media.de> writes:
>
>> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>>> I'm going to make a different suggestion.  Lets just drop it.
>>
>> I'm in favour of this. I don't think they give us the safety we want and
>> they introduce more inconvenience.
>
> After reading the thread I'm also in favour of this.

If you guys don't need it then just disable this checksum feature. No problem.

>
> --
> Otavio Salvador                  O.S. Systems
> E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
> Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: checksums situation

[Tom Rini]

On Tue, Feb 24, 2009 at 08:36:35PM +0200, Ihar Hrachyshka wrote:
> On Tue, Feb 24, 2009 at 8:01 PM, Otavio Salvador
> <otavio@ossystems.com.br> wrote:
> > Michael 'Mickey' Lauer <mlauer@vanille-media.de> writes:
> >
> >> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
> >>> I'm going to make a different suggestion.  Lets just drop it.
> >>
> >> I'm in favour of this. I don't think they give us the safety we want and
> >> they introduce more inconvenience.
> >
> > After reading the thread I'm also in favour of this.
> 
> If you guys don't need it then just disable this checksum feature. No problem.

This misses the point.  We're trying to get things to the point (and
keep them there) where the default case is things working well.  I and
others are arguing that the checksum feature is at best a lazy way to
check only for files changing and not their correctness.  I'm not saying
we don't need a feature to check for correctness, I'm saying what we
have now doesn't.

-- 
Tom Rini

Re: checksums situation

[GNUtoo]

> On Tue, Feb 24, 2009 at 08:36:35PM +0200, Ihar Hrachyshka wrote:
>> On Tue, Feb 24, 2009 at 8:01 PM, Otavio Salvador
>> <otavio@ossystems.com.br> wrote:
>> > Michael 'Mickey' Lauer <mlauer@vanille-media.de> writes:
>> >
>> >> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>> >>> I'm going to make a different suggestion.  Lets just drop it.
>> >>
>> >> I'm in favour of this. I don't think they give us the safety we want
>> and
>> >> they introduce more inconvenience.
>> >
>> > After reading the thread I'm also in favour of this.
>>
>> If you guys don't need it then just disable this checksum feature. No
>> problem.
>
> This misses the point.  We're trying to get things to the point (and
> keep them there) where the default case is things working well.  I and
> others are arguing that the checksum feature is at best a lazy way to
> check only for files changing and not their correctness.  I'm not saying
> we don't need a feature to check for correctness, I'm saying what we
> have now doesn't.
Maybe that should go into the manual:
user have to edit their local.conf so why not adding one more line in it
if they wish not to be bothered by checksums.
(you could also tell the same thing for the check of the checksums not
beeing the default...)
Denis.

Re: checksums situation

[Otavio Salvador]

Tom Rini <trini@kernel.crashing.org> writes:

> On Tue, Feb 24, 2009 at 08:36:35PM +0200, Ihar Hrachyshka wrote:
>> On Tue, Feb 24, 2009 at 8:01 PM, Otavio Salvador
>> <otavio@ossystems.com.br> wrote:
>> > Michael 'Mickey' Lauer <mlauer@vanille-media.de> writes:
>> >
>> >> Am Montag, den 23.02.2009, 23:46 -0700 schrieb Tom Rini:
>> >>> I'm going to make a different suggestion.  Lets just drop it.
>> >>
>> >> I'm in favour of this. I don't think they give us the safety we want and
>> >> they introduce more inconvenience.
>> >
>> > After reading the thread I'm also in favour of this.
>> 
>> If you guys don't need it then just disable this checksum feature. No problem.
>
> This misses the point.  We're trying to get things to the point (and
> keep them there) where the default case is things working well.  I and
> others are arguing that the checksum feature is at best a lazy way to
> check only for files changing and not their correctness.  I'm not saying
> we don't need a feature to check for correctness, I'm saying what we
> have now doesn't.

I do belive that the best way to solve it is to have a md5 file together
with the .bb recipe. This solves the problems for forks, derivatives and
also makes harder to just use "cat tmp/checksums.ini >> conf/checksums.ini".

Doing that we'll have a clear way to add the required content, avoid the
mirror and URL issues and also make simple to forget about useless
entries in the metadata repository.

Obviously, it is a little more difficult to add the contents but I
believe that it will enforce more checking by our side before changing a
hash.

My 2c. 

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: checksums situation

[Tom Rini]

On Tue, Feb 24, 2009 at 11:01:05PM -0300, Otavio Salvador wrote:
[snip]
> I do belive that the best way to solve it is to have a md5 file together
> with the .bb recipe. This solves the problems for forks, derivatives and
> also makes harder to just use "cat tmp/checksums.ini >> conf/checksums.ini".

Running a script that will make the .sum file isn't any harder really.
And it's still a "this is the checksum we downloaded" not "this is the
checksum upstream says is correct".

> Doing that we'll have a clear way to add the required content, avoid the
> mirror and URL issues and also make simple to forget about useless
> entries in the metadata repository.
> 
> Obviously, it is a little more difficult to add the contents but I
> believe that it will enforce more checking by our side before changing a
> hash.

I think there's enough eyes on things now such that people don't just
change checksums.ini and not get called out on it.  But that's still an
aside.

-- 
Tom Rini

Re: checksums situation

[Richard Purdie]

On Tue, 2009-02-24 at 19:25 -0700, Tom Rini wrote:
> On Tue, Feb 24, 2009 at 11:01:05PM -0300, Otavio Salvador wrote:
> [snip]
> > I do belive that the best way to solve it is to have a md5 file together
> > with the .bb recipe. This solves the problems for forks, derivatives and
> > also makes harder to just use "cat tmp/checksums.ini >> conf/checksums.ini".
> 
> Running a script that will make the .sum file isn't any harder really.
> And it's still a "this is the checksum we downloaded" not "this is the
> checksum upstream says is correct".

You can specify the md5 as part of the SRC_URI if I remember correctly.
Its just such a pain, nobody does.

-- 
RP

Re: checksums situation

[Vitus Jensen]

Am Tue, 24 Feb 2009 19:25:07 -0700 schrieb Tom Rini:

> On Tue, Feb 24, 2009 at 11:01:05PM -0300, Otavio Salvador wrote: [snip]
>> I do belive that the best way to solve it is to have a md5 file
>> together with the .bb recipe. This solves the problems for forks,
>> derivatives and also makes harder to just use "cat tmp/checksums.ini >>
>> conf/checksums.ini".
> 
> Running a script that will make the .sum file isn't any harder really.
> And it's still a "this is the checksum we downloaded" not "this is the
> checksum upstream says is correct".
...

But "this is the checksum we downloaded" says that's it's the same 
version the author of the .bb receipe downloaded, reviewed and tested on 
his device.  What is the probability that this author downloaded a 
corrupt but working archive last november and you get the same corrupt 
archive now?

If you want better security you have to ask the download source for a GPG 
signature of his files or the like as MD5 isn't really safe.

Bye,
   Vitus

-- 
Vitus Jensen, Hannover, Germany, Earth, Milky Way, Universe (current)

Re: checksums situation

[Tom Rini]

On Wed, Feb 25, 2009 at 09:27:02PM +0000, Vitus Jensen wrote:
> Am Tue, 24 Feb 2009 19:25:07 -0700 schrieb Tom Rini:
> 
> > On Tue, Feb 24, 2009 at 11:01:05PM -0300, Otavio Salvador wrote: [snip]
> >> I do belive that the best way to solve it is to have a md5 file
> >> together with the .bb recipe. This solves the problems for forks,
> >> derivatives and also makes harder to just use "cat tmp/checksums.ini >>
> >> conf/checksums.ini".
> > 
> > Running a script that will make the .sum file isn't any harder really.
> > And it's still a "this is the checksum we downloaded" not "this is the
> > checksum upstream says is correct".
> ...
> 
> But "this is the checksum we downloaded" says that's it's the same 
> version the author of the .bb receipe downloaded, reviewed and tested on 
> his device.  What is the probability that this author downloaded a 
> corrupt but working archive last november and you get the same corrupt 
> archive now?

See hrw's post earlier where he points out how many checksums are a
simple fetch and add? :)

> If you want better security you have to ask the download source for a GPG 
> signature of his files or the like as MD5 isn't really safe.

This is one of my points.  People think we have security from our
current checksum list, but we do not.

> 
> Bye,
>    Vitus
> 
> -- 
> Vitus Jensen, Hannover, Germany, Earth, Milky Way, Universe (current)
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

-- 
Tom Rini

Re: checksums situation

[Vitus Jensen]

Am Wed, 25 Feb 2009 14:35:36 -0700 schrieb Tom Rini:

> On Wed, Feb 25, 2009 at 09:27:02PM +0000, Vitus Jensen wrote:
>> Am Tue, 24 Feb 2009 19:25:07 -0700 schrieb Tom Rini:
>> 
>> > On Tue, Feb 24, 2009 at 11:01:05PM -0300, Otavio Salvador wrote:
>> > [snip]
>> >> I do belive that the best way to solve it is to have a md5 file
>> >> together with the .bb recipe. This solves the problems for forks,
>> >> derivatives and also makes harder to just use "cat tmp/checksums.ini
>> >> >> conf/checksums.ini".
>> > 
>> > Running a script that will make the .sum file isn't any harder
>> > really. And it's still a "this is the checksum we downloaded" not
>> > "this is the checksum upstream says is correct".
>> ...
>> 
>> But "this is the checksum we downloaded" says that's it's the same
>> version the author of the .bb receipe downloaded, reviewed and tested
>> on his device.  What is the probability that this author downloaded a
>> corrupt but working archive last november and you get the same corrupt
>> archive now?
> 
> See hrw's post earlier where he points out how many checksums are a
> simple fetch and add? :)

Very simple ... but getting the commit of MODIFIED checksum.ini entries 
through unrecognized is much harder.  I have seen discussions about such 
patches here.


>> If you want better security you have to ask the download source for a
>> GPG signature of his files or the like as MD5 isn't really safe.
> 
> This is one of my points.  People think we have security from our
> current checksum list, but we do not.

But even a really safe signature is just an entry in some central or 
distributed checksum.ini, so commiting a different entry would be as easy 
as it is now.

If this leads you to abandon checksums alltogether: the current situation 
protects against corrupt downloads and otherwise undetected updates from 
upstream and I'm all for keeping such a protection.

Bye,
  Vitus

-- 
Vitus Jensen, Hannover, Germany, Earth, Milky Way, Universe (current)

Re: checksums situation

[Koen Kooi]

On 25-02-09 23:04, Vitus Jensen wrote:

> If this leads you to abandon checksums alltogether: the current situation
> protects against corrupt downloads and otherwise undetected updates from
> upstream and I'm all for keeping such a protection.

I get reports pretty much every single week about someone asking "what 
does 'checksum failed' mean" when their download is corrupt due to evil 
proxies and bad disks. It's a lot easier and faster to debug than 
seemingly random build failures.

regards,

Koen

Re: checksums situation

[Bernhard Guillon]

Tom Rini wrote:
> This is one of my points.  People think we have security from our
> current checksum list, but we do not.
>
>   
Then we have to make clear that the checksums are for integrity only and 
not for security.
It is impossible for us to do security. E.g. most sourceforge projects 
do not sign their packages. We would need to review the source of every 
package to see if it does stuff it should not do. We would also need to 
track security updates for packages - which we should do anyway.

Best regards
Bernhard Guillon

Re: checksums situation

[Alessandro GARDICH]

Bernhard Guillon wrote:
> Tom Rini wrote:
>> This is one of my points.  People think we have security from our
>> current checksum list, but we do not.
>>
>>   
> Then we have to make clear that the checksums are for integrity only and 
> not for security.
> It is impossible for us to do security. E.g. most sourceforge projects 
> do not sign their packages. We would need to review the source of every 
> package to see if it does stuff it should not do. We would also need to 
> track security updates for packages - which we should do anyway.
> 
> Best regards
> Bernhard Guillon
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


Sincerely I don't feel the need of "security" in OE but that is.

In my opinion the checking of the sources is a feature we can have but 
for sure not in a global checksum.ini file, it's unmanageable.
Every recipe, in which is defined a source can have a checksum, as 
someone else proposed is a better solution.

Talking about security in a strict way, check the sources have in my 
opinion no sense, an "evil" recipe could fetch a well signed source of 
ssh (as example) and apply a patch to add a back door.
Checking can be useful but not for security reason, at most just to be 
sure the source is what expect to be.

How checksum behave is source is a latest revision of a VCS ?

Other point, I completely dislike the current behaviour : if a source 
haven't a checksum  fail do build. No please ... the default could be a 
warning not a fail!

I'm sure 90% or OE users got a failure, ask for help and now have 
OE_STRICT_CHECKSUMS = "" in their local.conf ... have it sense ???

In my opinion the default behaviour have to be a warning, for who is 
sensible to a (false) security they can enforce the behaviour (suck as 
-Werror for gcc) but no more.
A warning at the end of bitbake build could also be useful, something 
like "your final image contain non checked sources", but not a FAIL!


Last but more important : why the hell this feature is in the default 
dev branch ??? why wasn't created a "checksum" branch to test it !!!
One thing make OE UNUSABLE for day to day work is the BAD behaviour :
- think a feature
- start (but not finish) to implement it
- push
- make dev branch fail to build
- start to correct/finish the feature
damn, we got git to be easy to branch to test new features!!!


--
  /-------------------------------------------------------------\
|           Alessandro Gardich : gremlin#gremlin!it             |
  >-------------------------------------------------------------<
|  I never saw a wild thing sorry for itself.                   |
|  A small bird will drop frozen dead from a bough              |
|  without ever having felt sorry for itself.                   |
  \-------------------------------------------------------------/

Re: checksums situation

[Koen Kooi]

On 28-02-09 10:57, Alessandro GARDICH wrote:

> Last but more important : why the hell this feature is in the default
> dev branch ???

It was discussed over a period of weeks and RFC'ed, so bitching about it 
now is just trolling.

Re: checksums situation

[Alessandro GARDICH]

Koen Kooi wrote:
> On 28-02-09 10:57, Alessandro GARDICH wrote:
> 
>> Last but more important : why the hell this feature is in the default
>> dev branch ???
> 
> It was discussed over a period of weeks and RFC'ed, so bitching about it 
> now is just trolling.
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


Ok my fault, I read mailing list to few ...

but a little question remain, why in the main .dev when is so few tested 
  and broke a lot of build ...

Re: checksums situation

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]

Alessandro GARDICH wrote:
> Koen Kooi wrote:
>> On 28-02-09 10:57, Alessandro GARDICH wrote:
>>
>>> Last but more important : why the hell this feature is in the default
>>> dev branch ???
>>
>> It was discussed over a period of weeks and RFC'ed, so bitching about 
>> it now is just trolling.
>>
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 
> 
> Ok my fault, I read mailing list to few ...
> 
> but a little question remain, why in the main .dev when is so few tested 
>  and broke a lot of build ...

I think there are two issues at the moment:

1) Build breakage caused by the addition of new sources without the 
corresponding checksums. This leads to build failures for people 
building with checksum checking enabled. For some reason, these failures 
have increased in recent weeks. People, please try to remember to add 
the checksums, even if you are building with checksums disabled.

2) The monolithic checksums.ini file does not currently provide an easy 
way for people to add checksums for packages that are in 
collections/private branches. I believe there is a proposal to fix this 
problem by allowing people to extend checksums.ini.

For "casual users", you can set:

OE_STRICT_CHECKSUMS = ""

this will not error on missing checksums, but will error if the checksum 
does not match. This detects bad downloads, upstream projects that 
change source without changing file names, and evil.

If people start paying a little more attention to 1 and we get 2 
implemented, much of the recent checksum noise will go away.

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: checksums situation

[Bernhard Guillon]

Marcin Juszkiewicz wrote:
> Other was to use filename as key and add "url[0-xx]" fields which will 
> list alternative locations. This one looks better but still does not 
> solve situation when someone use DEBIAN_MIRROR which is not present in 
> checksums.ini file.
>
>   
What about using the hashes as identifier:

Something like this:

filename=foo-0.13.49.tar.bz2
url[0-xx]
generated_url[0-xx] //or only [0-xx]
md5=5f7b88ebb2bcd7e8044328482d079661
sha256=f57c4e33eb2cdd87a6c2f01bfa4794340fbe61ea1a1cfc7dac3b6671e1dd22af

pseudo code:
download source
check_ok:=false

if SRC_URI exists in url/generated_url + filename
    do normal check
else
    generate md5 and sha256 sums of the source
    look up md5 and sha256 sum
    if both exists in the same entity
       if filename matches filename of SRC_URI
          check_ok:=true
          add url to generated_urls

best regards
Bernhard Guillon

Re: checksums situation

[GNUtoo]

> Marcin Juszkiewicz wrote:
>> Other was to use filename as key and add "url[0-xx]" fields which will
>> list alternative locations. This one looks better but still does not
>> solve situation when someone use DEBIAN_MIRROR which is not present in
>> checksums.ini file.
>>
>>
> What about using the hashes as identifier:
>
> Something like this:
>
> filename=foo-0.13.49.tar.bz2
> url[0-xx]
> generated_url[0-xx] //or only [0-xx]
> md5=5f7b88ebb2bcd7e8044328482d079661
> sha256=f57c4e33eb2cdd87a6c2f01bfa4794340fbe61ea1a1cfc7dac3b6671e1dd22af
>
> pseudo code:
> download source
> check_ok:=false
>
> if SRC_URI exists in url/generated_url + filename
>     do normal check
> else
>     generate md5 and sha256 sums of the source
>     look up md5 and sha256 sum
>     if both exists in the same entity
>        if filename matches filename of SRC_URI
>           check_ok:=true
>           add url to generated_urls
>
> best regards
> Bernhard Guillon
great idea...
but we also need to find out how to handle the commits of suchs generateds
urls?
because some developers forget to commit the checksums,and because they
could have differents repositories
Denis.

Re: checksums situation

[Koen Kooi]

On 13-02-09 17:28, Marcin Juszkiewicz wrote:
> Hi
>
> It is nearly two years since conf/checksums.ini was introduced. We
> populated it with over 6000 entries during that time (mostly by
> automatic fetching of all source on CELF and EWI machines). But there
> are problems with it's format.

I think most issues people have with it can be solved (or worked around) 
by setting

OE_STRICT_CHECKSUMS = ""

in either local.conf or bitbake.conf. It will make the checker still 
abort if the sums changed, but missing sums aren't fatal and get 
reported to the user[1] with a bb.note (as well as being put into 
TMP/checksums.ini).

Another nice fix would be to check only the leafname, not the complete 
uri. Collision isn't a problem since we put all files in a single dir 
after downloading anyway.

regards,

Koen

[1] with bitbake 1.8.x, bitbake 1.9.x hides bb.note

Re: checksums situation

[Frans Meulenbroeks]

Personally I feel the check that the src has not changed under our
fingers valuable, but I admit that adding the checksums is cumbersome.

Can't we automate this one way or another. E.g. if a .bb file is
checked in it is processed and the checksum is automatically added.
This could be done at server side or client side.
Or maybe harvest new checksums at regular intervals (e.g. from tinderbox).
Of course something should also be done in case a SRCREV changes.

FM.