* [PATCH 0/1] SRC_URI checksum support v2
@ 2010-12-14 10:01 Yu Ke
2010-12-14 7:55 ` [PATCH 1/1] FetchData: add SRC_URI checksum Yu Ke
2010-12-14 12:32 ` [PATCH 0/1] SRC_URI checksum support v2 Koen Kooi
0 siblings, 2 replies; 5+ messages in thread
From: Yu Ke @ 2010-12-14 10:01 UTC (permalink / raw)
To: poky
This patch add SRC_URI checksum support. With this patch,fetcher
can verify the MD5 and SHA256 checksum of download src with the
value defined in recipes SRC_URI.
This is the V2 patch. The major change compared to V1 is:
moving the logic from poky bbclass to bitbake fetcher, per
Richard's comment
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/srcuri-v2
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v2
Thanks,
Yu Ke <ke.yu@intel.com>
---
Yu Ke (1):
FetchData: add SRC_URI checksum
bitbake/lib/bb/fetch/__init__.py | 36 ++++++++++++++++++++++++++++++++++++
1 files changed, 36 insertions(+), 0 deletions(-)
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/1] FetchData: add SRC_URI checksum 2010-12-14 10:01 [PATCH 0/1] SRC_URI checksum support v2 Yu Ke @ 2010-12-14 7:55 ` Yu Ke 2010-12-15 9:12 ` Richard Purdie 2010-12-14 12:32 ` [PATCH 0/1] SRC_URI checksum support v2 Koen Kooi 1 sibling, 1 reply; 5+ messages in thread From: Yu Ke @ 2010-12-14 7:55 UTC (permalink / raw) To: poky This patch add the per-recipe SRC_URI checksum verification. - SRC_URI format The format of SRC_URI checksum follow OE definition: 1. SRC_URI has single src SRC_URI = "http://some.domain/file.tar.gz" SRC_URI[md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" 2. SRC_URI has multiple src, every src need specify name SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \ http://some.domain/file2.tar.gz;name=name2 " SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx" SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx" - SRC_URI checking invocation: the checksum checking is invoked in do_fetch phase, so it can be invoked manually by # bitbake -f -c fetch <recipe_name> if recipes has no SRC_URI checksum item, bitbake will show warning: " WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072" SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768" " thus recipe author can add it to recpie file after SRC_URI Signed-off-by: Yu Ke <ke.yu@intel.com> --- bitbake/lib/bb/fetch/__init__.py | 36 ++++++++++++++++++++++++++++++++++++ 1 files changed, 36 insertions(+), 0 deletions(-) diff --git a/bitbake/lib/bb/fetch/__init__.py b/bitbake/lib/bb/fetch/__init__.py index 50955f1..d3360b1 100644 --- a/bitbake/lib/bb/fetch/__init__.py +++ b/bitbake/lib/bb/fetch/__init__.py @@ -231,6 +231,29 @@ def removefile(f): except: pass +def verify_checksum(d, ud): + """ + verify the MD5 and SHA256 checksum for downloaded src + return True if matched, False if not + """ + md5data = bb.utils.md5_file(ud.localpath) + sha256data = bb.utils.sha256_file(ud.localpath) + + if (ud.md5_expected == None or ud.sha256_expected == None): + bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \ + "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \ + % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data)) + # TODO: change to "return False" once all recpies has checksum entry + return True + + if (ud.md5_expected != md5data or ud.sha256_expected != sha256data): + bb.error("The checksums for '%s' did not match." % ud.localpath) + bb.error("Expected MD5: '%s' and Got: '%s'" % (ud.md5_expected, md5data)) + bb.error("Expected SHA256: '%s' and Got: '%s'" % (ud.sha256_expected, sha256data)) + return False + + return True + def go(d, urls = None): """ Fetch all urls @@ -283,6 +306,9 @@ def go(d, urls = None): else: Fetch.write_md5sum(u, ud, d) + if not verify_checksum(d, ud): + raise FetchError("%s checksum mismatch." % u) + bb.utils.unlockfile(lf) def checkstatus(d, urls = None): @@ -502,6 +528,16 @@ class FetchData(object): if not self.pswd and "pswd" in self.parm: self.pswd = self.parm["pswd"] self.setup = False + + if "name" in self.parm: + self.md5_name = "%s.md5sum" % self.parm["name"] + self.sha256_name = "%s.sha256sum" % self.parm["name"] + else: + self.md5_name = "md5sum" + self.sha256_name = "sha256sum" + self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d) + self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d) + for m in methods: if m.supports(url, self, d): self.method = m -- 1.7.0.4 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 1/1] FetchData: add SRC_URI checksum 2010-12-14 7:55 ` [PATCH 1/1] FetchData: add SRC_URI checksum Yu Ke @ 2010-12-15 9:12 ` Richard Purdie 2010-12-16 9:16 ` Yu Ke 0 siblings, 1 reply; 5+ messages in thread From: Richard Purdie @ 2010-12-15 9:12 UTC (permalink / raw) To: Yu Ke; +Cc: poky Hi Ke, This patch looks good in general, just a couple of minor things: On Tue, 2010-12-14 at 15:55 +0800, Yu Ke wrote: > > +def verify_checksum(d, ud): > + """ > + verify the MD5 and SHA256 checksum for downloaded src > + return True if matched, False if not > + """ This should be more explicit about the missing checksums case. > + md5data = bb.utils.md5_file(ud.localpath) > + sha256data = bb.utils.sha256_file(ud.localpath) > + > + if (ud.md5_expected == None or ud.sha256_expected == None): > + bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \ > + "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \ > + % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data)) > + # TODO: change to "return False" once all recpies has checksum entry > + return True We can't just redefine "policy" like this in bitbake. I think missing checksums will always be warnings and we might just make warnings optionally fatal for bitbake. What does this do for file:// urls or SCM urls? Otherwise I'm good with the patch and if you tweak these things I'll merge it. Cheers, Richard ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/1] FetchData: add SRC_URI checksum 2010-12-15 9:12 ` Richard Purdie @ 2010-12-16 9:16 ` Yu Ke 0 siblings, 0 replies; 5+ messages in thread From: Yu Ke @ 2010-12-16 9:16 UTC (permalink / raw) To: Richard Purdie; +Cc: poky On Dec 15, 09:12, Richard Purdie wrote: > Hi Ke, > > This patch looks good in general, just a couple of minor things: > > On Tue, 2010-12-14 at 15:55 +0800, Yu Ke wrote: > > > > +def verify_checksum(d, ud): > > + """ > > + verify the MD5 and SHA256 checksum for downloaded src > > + return True if matched, False if not > > + """ > > This should be more explicit about the missing checksums case. thanks for the review. and yes, will do that. > > > + md5data = bb.utils.md5_file(ud.localpath) > > + sha256data = bb.utils.sha256_file(ud.localpath) > > + > > + if (ud.md5_expected == None or ud.sha256_expected == None): > > + bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \ > > + "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \ > > + % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data)) > > + # TODO: change to "return False" once all recpies has checksum entry > > + return True > > We can't just redefine "policy" like this in bitbake. I think missing > checksums will always be warnings and we might just make warnings > optionally fatal for bitbake. for the "make warnings optionally fatal in bitbake", do you mean add a variable like "BB_STRICT_CHECKSUM" in config file, and if for example "BB_STRICT_CHECKSUMS" = "1", then fatal, otherwise, just warning? > > What does this do for file:// urls or SCM urls? Good catch. It should only verify for "http/https/ftp/ftps" protocol, other protocol like local file and SCM url should just return. Regards Ke > > Otherwise I'm good with the patch and if you tweak these things I'll > merge it. > > Cheers, > > Richard > > > _______________________________________________ > poky mailing list > poky@yoctoproject.org > https://lists.yoctoproject.org/listinfo/poky ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/1] SRC_URI checksum support v2 2010-12-14 10:01 [PATCH 0/1] SRC_URI checksum support v2 Yu Ke 2010-12-14 7:55 ` [PATCH 1/1] FetchData: add SRC_URI checksum Yu Ke @ 2010-12-14 12:32 ` Koen Kooi 1 sibling, 0 replies; 5+ messages in thread From: Koen Kooi @ 2010-12-14 12:32 UTC (permalink / raw) To: Yu Ke; +Cc: poky Op 14 dec 2010, om 11:01 heeft Yu Ke het volgende geschreven: > This patch add SRC_URI checksum support. With this patch,fetcher > can verify the MD5 and SHA256 checksum of download src with the > value defined in recipes SRC_URI. Awesome, I have been waiting for that! > > This is the V2 patch. The major change compared to V1 is: > moving the logic from poky bbclass to bitbake fetcher, per > Richard's comment > > Pull URL: git://git.pokylinux.org/poky-contrib.git > Branch: kyu3/srcuri-v2 > Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v2 > > Thanks, > Yu Ke <ke.yu@intel.com> > --- > > > Yu Ke (1): > FetchData: add SRC_URI checksum > > bitbake/lib/bb/fetch/__init__.py | 36 ++++++++++++++++++++++++++++++++++++ > 1 files changed, 36 insertions(+), 0 deletions(-) > > _______________________________________________ > poky mailing list > poky@yoctoproject.org > https://lists.yoctoproject.org/listinfo/poky ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2010-12-16 9:11 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-12-14 10:01 [PATCH 0/1] SRC_URI checksum support v2 Yu Ke 2010-12-14 7:55 ` [PATCH 1/1] FetchData: add SRC_URI checksum Yu Ke 2010-12-15 9:12 ` Richard Purdie 2010-12-16 9:16 ` Yu Ke 2010-12-14 12:32 ` [PATCH 0/1] SRC_URI checksum support v2 Koen Kooi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox