[PATCH 0/1] SRC_URI checksum support v3

[PATCH 0/1] SRC_URI checksum support v3

[Yu Ke]

This patch add SRC_URI checksum support. With this patch,fetcher
can verify the MD5 and SHA256 checksum of download src with the
value defined in recipes SRC_URI.

This is the v3 patch with following changes compared with v2:
- add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case
  if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
- add check to only verify checksum for protocol http/https/ftp/ftps, not
  verify checksum for local file and other SCM

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: kyu3/srcuri-v3
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

Thanks,
    Yu Ke <ke.yu@intel.com>
---


Yu Ke (1):
  FetchData: add SRC_URI checksum

 bitbake/lib/bb/fetch/__init__.py |   49 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 49 insertions(+), 0 deletions(-)

[PATCH 1/1] FetchData: add SRC_URI checksum

[Yu Ke]

This patch add the per-recipe SRC_URI checksum verification.

- SRC_URI format
The format of SRC_URI checksum follow OE definition:

1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz"
SRC_URI[md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

2. SRC_URI has multiple src, every src need specify name
SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
           http://some.domain/file2.tar.gz;name=name2 "
SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"

- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase,
so it can be invoked manually by

# bitbake -f -c fetch <recipe_name>

if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add
SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768"
"
thus recipe author can add it to recpie file after SRC_URI

- control variable BB_STRICT_CHECKSUM
when SRC_URI checksum is missing, this variable decide pass or not
if BB_STRICT_CHECKSUM = "1", bitbake should fatal in this case, otherwise bitbake just pass

Signed-off-by: Yu Ke <ke.yu@intel.com>
---
 bitbake/lib/bb/fetch/__init__.py |   49 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 49 insertions(+), 0 deletions(-)

diff --git a/bitbake/lib/bb/fetch/__init__.py b/bitbake/lib/bb/fetch/__init__.py
index 50955f1..387de66 100644
--- a/bitbake/lib/bb/fetch/__init__.py
+++ b/bitbake/lib/bb/fetch/__init__.py
@@ -231,6 +231,42 @@ def removefile(f):
     except:
         pass
 
+def verify_checksum(d, ud):
+    """
+    verify the MD5 and SHA256 checksum for downloaded src
+
+    return value:
+        - True: checksum matched
+        - False: checksum unmatched
+
+    if checksum is missing in recipes file, "BB_STRICT_CHECKSUM" decide the return value.
+    if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
+    matched
+    """
+
+    if not ud.type in ["http", "https", "ftp", "ftps"]:
+        return True
+
+    md5data = bb.utils.md5_file(ud.localpath)
+    sha256data = bb.utils.sha256_file(ud.localpath)
+
+    if (ud.md5_expected == None or ud.sha256_expected == None):
+        bb.warn("Missing SRC_URI checksum for %s, consider to add\n" \
+                "SRC_URI[%s] = \"%s\"\nSRC_URI[%s] = \"%s\"" \
+                % (ud.localpath, ud.md5_name, md5data, ud.sha256_name, sha256data))
+        if bb.data.getVar("BB_STRICT_CHECKSUM", d, True) == "1":
+            return False
+        else:
+            return True
+
+    if (ud.md5_expected != md5data or ud.sha256_expected != sha256data):
+        bb.error("The checksums for '%s' did not match." % ud.localpath)
+        bb.error("Expected MD5: '%s' and Got: '%s'" % (ud.md5_expected, md5data))
+        bb.error("Expected SHA256: '%s' and Got: '%s'" % (ud.sha256_expected, sha256data))
+        return False
+
+    return True
+
 def go(d, urls = None):
     """
     Fetch all urls
@@ -283,6 +319,9 @@ def go(d, urls = None):
         else:
             Fetch.write_md5sum(u, ud, d)
 
+        if not verify_checksum(d, ud):
+            raise FetchError("%s checksum mismatch." % u)
+
         bb.utils.unlockfile(lf)
 
 def checkstatus(d, urls = None):
@@ -502,6 +541,16 @@ class FetchData(object):
         if not self.pswd and "pswd" in self.parm:
             self.pswd = self.parm["pswd"]
         self.setup = False
+
+        if "name" in self.parm:
+            self.md5_name = "%s.md5sum" % self.parm["name"]
+            self.sha256_name = "%s.sha256sum" % self.parm["name"]
+        else:
+            self.md5_name = "md5sum"
+            self.sha256_name = "sha256sum"
+        self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d)
+        self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d)
+
         for m in methods:
             if m.supports(url, self, d):
                 self.method = m
-- 
1.7.0.4

Re: [PATCH 0/1] SRC_URI checksum support v3

[Richard Purdie]

On Fri, 2010-12-17 at 14:33 +0800, Yu Ke wrote:
> This patch add SRC_URI checksum support. With this patch,fetcher
> can verify the MD5 and SHA256 checksum of download src with the
> value defined in recipes SRC_URI.
> 
> This is the v3 patch with following changes compared with v2:
> - add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case
>   if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
> - add check to only verify checksum for protocol http/https/ftp/ftps, not
>   verify checksum for local file and other SCM
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: kyu3/srcuri-v3
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3

I merged this, then realised there was a problem with subsequent builds
proceeding with corrupt files. I've pushed a fix for at least part of
that.

I'm wondering if we should rename the file to something like the
original name + ".corrupt" to make sure we don't use a known broken
file?

Cheers,

Richard

Re: [PATCH 0/1] SRC_URI checksum support v3

[Yu Ke]

On Dec 20, 16:10, Richard Purdie wrote:
> On Fri, 2010-12-17 at 14:33 +0800, Yu Ke wrote:
> > This patch add SRC_URI checksum support. With this patch,fetcher
> > can verify the MD5 and SHA256 checksum of download src with the
> > value defined in recipes SRC_URI.
> > 
> > This is the v3 patch with following changes compared with v2:
> > - add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case
> >   if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
> > - add check to only verify checksum for protocol http/https/ftp/ftps, not
> >   verify checksum for local file and other SCM
> > 
> > Pull URL: git://git.pokylinux.org/poky-contrib.git
> >   Branch: kyu3/srcuri-v3
> >   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3
> 
> I merged this, then realised there was a problem with subsequent builds
> proceeding with corrupt files. I've pushed a fix for at least part of
> that.

Thanks, it is indeed necessary fix.

BTW, I am also thinking if we could merge Fetch.write_md5sum and Fetch.verify_md5sum into verify_checksum, since they are all checksum related, puting them together would be more clean logically.

> 
> I'm wondering if we should rename the file to something like the
> original name + ".corrupt" to make sure we don't use a known broken
> file?

Yes, I prefer to do that. It is more robust, and when meet corrupt file, if bitbake did not clean the downloaded file, user also need to do that. so it is better for bitbake to do that automatically for user.

so combined with the above points, the verify_checksum can be:
"
if ud.md5 file exist
	touch ud.md5 and return

check the checksum, if mismatch, rename download file to *.corrupt and raise exception

write the ud.md5 file
"

Comment?

Regards
Ke

> 
> Cheers,
> 
> Richard
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/1] SRC_URI checksum support v3

[Joshua Lock]

On Tue, 2010-12-21 at 13:32 +0800, Yu Ke wrote:
> On Dec 20, 16:10, Richard Purdie wrote:
> > On Fri, 2010-12-17 at 14:33 +0800, Yu Ke wrote:
> > > This patch add SRC_URI checksum support. With this patch,fetcher
> > > can verify the MD5 and SHA256 checksum of download src with the
> > > value defined in recipes SRC_URI.
> > > 
> > > This is the v3 patch with following changes compared with v2:
> > > - add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case
> > >   if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
> > > - add check to only verify checksum for protocol http/https/ftp/ftps, not
> > >   verify checksum for local file and other SCM
> > > 
> > > Pull URL: git://git.pokylinux.org/poky-contrib.git
> > >   Branch: kyu3/srcuri-v3
> > >   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3
> > 
> > I merged this, then realised there was a problem with subsequent builds
> > proceeding with corrupt files. I've pushed a fix for at least part of
> > that.
> 
> Thanks, it is indeed necessary fix.
> 
> BTW, I am also thinking if we could merge Fetch.write_md5sum and
> Fetch.verify_md5sum into verify_checksum, since they are all checksum
> related, puting them together would be more clean logically.

That makes the assumption that you only want to verify a checksum when
fetching one, which I don't think is correct?

Cheers,
Joshua
-- 
Joshua Lock
        Intel Open Source Technology Centre

Re: [PATCH 0/1] SRC_URI checksum support v3

[Richard Purdie]

On Tue, 2010-12-21 at 10:54 +0000, Joshua Lock wrote:
> On Tue, 2010-12-21 at 13:32 +0800, Yu Ke wrote:
> > BTW, I am also thinking if we could merge Fetch.write_md5sum and
> > Fetch.verify_md5sum into verify_checksum, since they are all checksum
> > related, puting them together would be more clean logically.
> 
> That makes the assumption that you only want to verify a checksum when
> fetching one, which I don't think is correct?

We really only want to take the time to compute the checksums once, then
mark something to say we've checked it and move on.

I think the use of the ".md5" file is misleading and we should probably
find a more logical approach to this but in principle checking after
download should be ok. We can always have something to enable a recheck
if the user really wanted it I guess but most of the time it will be a
waste of time.

Cheers,

Richard

Re: [PATCH 0/1] SRC_URI checksum support v3

[Yu Ke]

On Dec 21, 10:54, Joshua Lock wrote:
> On Tue, 2010-12-21 at 13:32 +0800, Yu Ke wrote:
> > On Dec 20, 16:10, Richard Purdie wrote:
> > > On Fri, 2010-12-17 at 14:33 +0800, Yu Ke wrote:
> > > > This patch add SRC_URI checksum support. With this patch,fetcher
> > > > can verify the MD5 and SHA256 checksum of download src with the
> > > > value defined in recipes SRC_URI.
> > > > 
> > > > This is the v3 patch with following changes compared with v2:
> > > > - add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case
> > > >   if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
> > > > - add check to only verify checksum for protocol http/https/ftp/ftps, not
> > > >   verify checksum for local file and other SCM
> > > > 
> > > > Pull URL: git://git.pokylinux.org/poky-contrib.git
> > > >   Branch: kyu3/srcuri-v3
> > > >   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3
> > > 
> > > I merged this, then realised there was a problem with subsequent builds
> > > proceeding with corrupt files. I've pushed a fix for at least part of
> > > that.
> > 
> > Thanks, it is indeed necessary fix.
> > 
> > BTW, I am also thinking if we could merge Fetch.write_md5sum and
> > Fetch.verify_md5sum into verify_checksum, since they are all checksum
> > related, puting them together would be more clean logically.
> 
> That makes the assumption that you only want to verify a checksum when
> fetching one, which I don't think is correct?

Do you mean other place may also need verify a checksum? If that is the case, it still can call the verify_checksum() to perform verifcation, right?

Regards
Ke

> 
> Cheers,
> Joshua
> -- 
> Joshua Lock
>         Intel Open Source Technology Centre
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 0/1] SRC_URI checksum support v3

[Yu Ke]

On Dec 21, 21:17, Yu Ke wrote:
> On Dec 21, 10:54, Joshua Lock wrote:
> > On Tue, 2010-12-21 at 13:32 +0800, Yu Ke wrote:
> > > On Dec 20, 16:10, Richard Purdie wrote:
> > > > On Fri, 2010-12-17 at 14:33 +0800, Yu Ke wrote:
> > > > > This patch add SRC_URI checksum support. With this patch,fetcher
> > > > > can verify the MD5 and SHA256 checksum of download src with the
> > > > > value defined in recipes SRC_URI.
> > > > > 
> > > > > This is the v3 patch with following changes compared with v2:
> > > > > - add configurable variable BB_STRICT_CHECKSUM to handle checksum missing case
> > > > >   if checksum is missing and BB_STRICT_CHECKSUM = "1", bitbake will fatal
> > > > > - add check to only verify checksum for protocol http/https/ftp/ftps, not
> > > > >   verify checksum for local file and other SCM
> > > > > 
> > > > > Pull URL: git://git.pokylinux.org/poky-contrib.git
> > > > >   Branch: kyu3/srcuri-v3
> > > > >   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/srcuri-v3
> > > > 
> > > > I merged this, then realised there was a problem with subsequent builds
> > > > proceeding with corrupt files. I've pushed a fix for at least part of
> > > > that.
> > > 
> > > Thanks, it is indeed necessary fix.
> > > 
> > > BTW, I am also thinking if we could merge Fetch.write_md5sum and
> > > Fetch.verify_md5sum into verify_checksum, since they are all checksum
> > > related, puting them together would be more clean logically.
> > 
> > That makes the assumption that you only want to verify a checksum when
> > fetching one, which I don't think is correct?
> 
> Do you mean other place may also need verify a checksum? If that is the case, it still can call the verify_checksum() to perform verifcation, right?

Oh, just see Richard has replied the email, it is pretty clear, so no question from me now.

Regards
Ke

> 
> > 
> > Cheers,
> > Joshua
> > -- 
> > Joshua Lock
> >         Intel Open Source Technology Centre
> > 
> > _______________________________________________
> > poky mailing list
> > poky@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/poky