checksums...

checksums...

[Koen Kooi]

Hi,

Could people adding new recipe and people git-am'ing patches adding new 
recipes please add the corresponding checksums to checksums.ini?

It's not hard:

1) bitbake <recipe>
2) watch it fail
3) cd $TMPDIR ; cat /OE/org.openembedded.dev/conf/checksums.ini >> 
checksums.ini ; python 
/OE/org.openembedded.dev/contrib/source-checker/oe-checksums-sorter.py 
checksums.ini > /OE/org.openembedded.dev/conf/checksums.ini
4) git commit --amend /OE/org.openembedded.dev/conf/checksums.ini

regards,

Koen

Re: checksums...

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

Koen Kooi wrote:
> Hi,
> 
> Could people adding new recipe and people git-am'ing patches adding new 
> recipes please add the corresponding checksums to checksums.ini?
> 
> It's not hard:
> 
> 1) bitbake <recipe>
> 2) watch it fail
> 3) cd $TMPDIR ; cat /OE/org.openembedded.dev/conf/checksums.ini >> 
> checksums.ini ; python 
> /OE/org.openembedded.dev/contrib/source-checker/oe-checksums-sorter.py 
> checksums.ini > /OE/org.openembedded.dev/conf/checksums.ini
> 4) git commit --amend /OE/org.openembedded.dev/conf/checksums.ini

I screwed this up once (and committed a recipe without the checksum) 
when I left ALLOW_INSECURE_DOWNLOADS set in my local.conf. Now I have it 
commented out and am really careful only to use it only when hacking.

I also need to look up what "git commit --amend" does.

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

[RFC] Add option for strict checksumming, was: Re: checksums...

[Koen Kooi]

On 21-01-09 20:15, Philip Balister wrote:
> Koen Kooi wrote:
>> Hi,
>>
>> Could people adding new recipe and people git-am'ing patches adding
>> new recipes please add the corresponding checksums to checksums.ini?
>>
>> It's not hard:
>>
>> 1) bitbake <recipe>
>> 2) watch it fail
>> 3) cd $TMPDIR ; cat /OE/org.openembedded.dev/conf/checksums.ini >>
>> checksums.ini ; python
>> /OE/org.openembedded.dev/contrib/source-checker/oe-checksums-sorter.py
>> checksums.ini > /OE/org.openembedded.dev/conf/checksums.ini
>> 4) git commit --amend /OE/org.openembedded.dev/conf/checksums.ini
>
> I screwed this up once (and committed a recipe without the checksum)
> when I left ALLOW_INSECURE_DOWNLOADS set in my local.conf. Now I have it
> commented out and am really careful only to use it only when hacking.

What about this:

 From a48ee59ac6ea569a5f88066d86de4b66d8f606c3 Mon Sep 17 00:00:00 2001
From: Koen Kooi <koen@openembedded.org>
Date: Wed, 21 Jan 2009 20:29:53 +0100
Subject: [PATCH] base.bbclass: only error out on missing checksums when 
OE_STRICT_CHECKSUMS is set
  * we want to error out on non-matching sums, missing ones are less bad

---
  classes/base.bbclass |    8 +++++++-
  1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/classes/base.bbclass b/classes/base.bbclass
index fca0edd..411adc9 100644
--- a/classes/base.bbclass
+++ b/classes/base.bbclass
@@ -71,7 +71,13 @@ def base_chk_file(parser, pn, pv, src_uri, localpath, 
data):

          file.write("[%s]\nmd5=%s\nsha256=%s\n\n" % (src_uri, md5data, 
shadata))
          file.close()
-        return False
+        if not bb.data.getVar("OE_STRICT_CHECKSUMS",data, True):
+            bb.note("This package has no entry in checksums.ini, please 
add one")
+            bb.note("\n[%s]\nmd5=%s\nsha256=%s" % (src_uri, md5data, 
shadata))
+            return True
+        else:
+            bb.note("Missing checksum")
+            return False

      if not md5 == md5data:
          bb.note("The MD5Sums did not match. Wanted: '%s' and Got: 
'%s'" % (md5,md5data))
-- 
1.5.6.3

Non-mangled version at 
http://dominion.thruhere.net/koen/OE/0001-base.bbclass-only-error-out-on-missing-checksums-wh.patch

regards,

Koen

Re: [RFC] Add option for strict checksumming, was: Re: checksums...

[Tom Rini]

On Wed, Jan 21, 2009 at 08:34:51PM +0100, Koen Kooi wrote:
> On 21-01-09 20:15, Philip Balister wrote:
>> Koen Kooi wrote:
>>> Hi,
>>>
>>> Could people adding new recipe and people git-am'ing patches adding
>>> new recipes please add the corresponding checksums to checksums.ini?
>>>
>>> It's not hard:
>>>
>>> 1) bitbake <recipe>
>>> 2) watch it fail
>>> 3) cd $TMPDIR ; cat /OE/org.openembedded.dev/conf/checksums.ini >>
>>> checksums.ini ; python
>>> /OE/org.openembedded.dev/contrib/source-checker/oe-checksums-sorter.py
>>> checksums.ini > /OE/org.openembedded.dev/conf/checksums.ini
>>> 4) git commit --amend /OE/org.openembedded.dev/conf/checksums.ini
>>
>> I screwed this up once (and committed a recipe without the checksum)
>> when I left ALLOW_INSECURE_DOWNLOADS set in my local.conf. Now I have it
>> commented out and am really careful only to use it only when hacking.
>
> What about this:
>
> From a48ee59ac6ea569a5f88066d86de4b66d8f606c3 Mon Sep 17 00:00:00 2001
> From: Koen Kooi <koen@openembedded.org>
> Date: Wed, 21 Jan 2009 20:29:53 +0100
> Subject: [PATCH] base.bbclass: only error out on missing checksums when  
> OE_STRICT_CHECKSUMS is set
>  * we want to error out on non-matching sums, missing ones are less bad

This will bring us back to where we started.  I think we just need to
live with the pain a bit more and get people used to "Oops, no checksum
in your patch" ...

And btw, git commit --amend will let you change things in the last
commit you made.  ONLY do this if you haven't also done a push.

-- 
Tom Rini

checksums

[folkert]

Hi,

Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?

Also: is it planned to add an option to add checksums to the data as
well?


Folkert van Heusden

-- 
www.vanheusden.com/multitail - multitail is tail on steroids. multiple
               windows, filtering, coloring, anything you can think of
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums

[Theodore Ts'o]

On Tue, May 14, 2013 at 11:14:08AM +0200, folkert wrote:
> 
> Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?

When e2fsck does a full check, it verifies the contents of all of the
metadata blocks, and this includes verifying all of the checksums.

> Also: is it planned to add an option to add checksums to the data as
> well?

That's not currently on the development roadmap; I could imagine
someone deciding to design an extension to ext4 that would do this
probably by storing the checksums in the indirect blocks, but no one
is currently working on it.

Regards,

					- Ted

Re: checksums

[folkert]

> > Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?
> 
> When e2fsck does a full check, it verifies the contents of all of the
> metadata blocks, and this includes verifying all of the checksums.

Ok. But that would only when the filesystem is not mounted.
Maybe some on-line functionality for doing so would be nice. I'm not
totally aware of the filesystem structures in memory/on disk, but
reading meta-data from disk which has changes pending in memory/in the
journal would give at worst a verify of old(er) data. I don't think this
(checking occasional old data) is a bad thing - scrubbing a
raid-device/disk doesn't give you the situation for the whole disk(s) in
1 (!) point at time either. If that would be required, then the user
could still unmount the filesystem and do a check.

> > Also: is it planned to add an option to add checksums to the data as
> > well?
> That's not currently on the development roadmap; I could imagine
> someone deciding to design an extension to ext4 that would do this
> probably by storing the checksums in the indirect blocks, but no one
> is currently working on it.

Would crowdfunding such a feature be an option?
E.g. via https://www.catincan.com/ or http://www.kickstarter.com/

On other thing I was thinking of, was a switch that would enable a
read-after-write which would verify that the data is indeed on the
platters/NAND-chips.
(I don't care (too) much about speed - I just want to be really sure my
data doesn't get corrupt/lost without investing huge piles of money on
a san/nas with remote mirroring etc.).


Regards,

Folkert van Heusden

-- 
Winnen in de staatsloterij? Check htp://vanheusden.com/sl.php voor
alle staatsloterij-uitslag statistieken denkbaar en zelfs ook een paar
win-tips!
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums

[Darrick J. Wong]

On Tue, May 14, 2013 at 04:40:34PM +0200, folkert wrote:
> > > Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?
> > 
> > When e2fsck does a full check, it verifies the contents of all of the
> > metadata blocks, and this includes verifying all of the checksums.
> 
> Ok. But that would only when the filesystem is not mounted.
> Maybe some on-line functionality for doing so would be nice. I'm not
> totally aware of the filesystem structures in memory/on disk, but
> reading meta-data from disk which has changes pending in memory/in the
> journal would give at worst a verify of old(er) data. I don't think this
> (checking occasional old data) is a bad thing - scrubbing a
> raid-device/disk doesn't give you the situation for the whole disk(s) in
> 1 (!) point at time either. If that would be required, then the user
> could still unmount the filesystem and do a check.

Well... if you ran filefrag -v on every file on the disk and read all the
xattrs, you'd scrub nearly all the metadata.  The only things you'd miss are
unallocated parts of the disk, most of which e2fsck also skips.

Ofc e2fsck will also catch incorrect values in the metadata, whereas I think
the kernel only notices /insane/ values.

> > > Also: is it planned to add an option to add checksums to the data as
> > > well?
> > That's not currently on the development roadmap; I could imagine
> > someone deciding to design an extension to ext4 that would do this
> > probably by storing the checksums in the indirect blocks, but no one
> > is currently working on it.

sha256sum < file > file.sha256 ? :D

(If only there was disk space and brain-time to do something where you could
*reconstruct* data.)

--D

> Would crowdfunding such a feature be an option?
> E.g. via https://www.catincan.com/ or http://www.kickstarter.com/
> 
> On other thing I was thinking of, was a switch that would enable a
> read-after-write which would verify that the data is indeed on the
> platters/NAND-chips.
> (I don't care (too) much about speed - I just want to be really sure my
> data doesn't get corrupt/lost without investing huge piles of money on
> a san/nas with remote mirroring etc.).
> 
> 
> Regards,
> 
> Folkert van Heusden
> 
> -- 
> Winnen in de staatsloterij? Check htp://vanheusden.com/sl.php voor
> alle staatsloterij-uitslag statistieken denkbaar en zelfs ook een paar
> win-tips!
> ----------------------------------------------------------------------
> Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: checksums

[folkert]

> > Ok. But that would only when the filesystem is not mounted.
> > Maybe some on-line functionality for doing so would be nice. I'm not
> > totally aware of the filesystem structures in memory/on disk, but
> > reading meta-data from disk which has changes pending in memory/in the
> > journal would give at worst a verify of old(er) data. I don't think this
> > (checking occasional old data) is a bad thing - scrubbing a
> > raid-device/disk doesn't give you the situation for the whole disk(s) in
> > 1 (!) point at time either. If that would be required, then the user
> > could still unmount the filesystem and do a check.
> 
> Well... if you ran filefrag -v on every file on the disk and read all the
> xattrs, you'd scrub nearly all the metadata.  The only things you'd miss are
> unallocated parts of the disk, most of which e2fsck also skips.

Yes but that is, imho, a bit dirty method.
Because I assume the result will be a message in dmesg and the
filesystem being remounted r/o?
I think it would be better if a nice message on the user's terminal and
an exit code.

> > > That's not currently on the development roadmap; I could imagine
> > > someone deciding to design an extension to ext4 that would do this
> > > probably by storing the checksums in the indirect blocks, but no one
> > > is currently working on it.
> 
> sha256sum < file > file.sha256 ? :D

Then you would need to read the whole file. I think it would be better
to have this on e.g. block-level. 4KB so CRC32 suffices?

> (If only there was disk space and brain-time to do something where you could
> *reconstruct* data.)

ah yes.
These days everything is done by the gpu, maybe it can help with that :)


Folkert van Heusden

-- 
www.vanheusden.com/multitail - multitail is tail on steroids. multiple
               windows, filtering, coloring, anything you can think of
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums

[Darrick J. Wong]

On Tue, May 14, 2013 at 08:57:54PM +0200, folkert wrote:
> > > Ok. But that would only when the filesystem is not mounted.
> > > Maybe some on-line functionality for doing so would be nice. I'm not
> > > totally aware of the filesystem structures in memory/on disk, but
> > > reading meta-data from disk which has changes pending in memory/in the
> > > journal would give at worst a verify of old(er) data. I don't think this
> > > (checking occasional old data) is a bad thing - scrubbing a
> > > raid-device/disk doesn't give you the situation for the whole disk(s) in
> > > 1 (!) point at time either. If that would be required, then the user
> > > could still unmount the filesystem and do a check.
> > 
> > Well... if you ran filefrag -v on every file on the disk and read all the
> > xattrs, you'd scrub nearly all the metadata.  The only things you'd miss are
> > unallocated parts of the disk, most of which e2fsck also skips.
> 
> Yes but that is, imho, a bit dirty method.
> Because I assume the result will be a message in dmesg and the
> filesystem being remounted r/o?
> I think it would be better if a nice message on the user's terminal and
> an exit code.

You should see "I/O Error" (or whatever -EIO becomes in the message catalog) on
the terminal running filefrag if you hit a checksum error, in addition to a
complaint in dmesg and a ro fs.

> > > > That's not currently on the development roadmap; I could imagine
> > > > someone deciding to design an extension to ext4 that would do this
> > > > probably by storing the checksums in the indirect blocks, but no one
> > > > is currently working on it.
> > 
> > sha256sum < file > file.sha256 ? :D
> 
> Then you would need to read the whole file. I think it would be better
> to have this on e.g. block-level. 4KB so CRC32 suffices?

block or bigalloc-cluster level, I suppose.

--D
> 
> > (If only there was disk space and brain-time to do something where you could
> > *reconstruct* data.)
> 
> ah yes.
> These days everything is done by the gpu, maybe it can help with that :)
> 
> 
> Folkert van Heusden
> 
> -- 
> www.vanheusden.com/multitail - multitail is tail on steroids. multiple
>                windows, filtering, coloring, anything you can think of
> ----------------------------------------------------------------------
> Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com