[PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2

[PATCH 0/1] fetcher2: retry mirror if upstream checksum mismatch V2

[Yu Ke]

This is the V2 patch. the main change is introduce new function update_stamp
per Richard's suggestion.

CC: Richard Purdie <richard.purdie@linuxfoundation.org>

The following changes since commit 7354fc9213f27aa1b643dbe88070437f1ee4c063:
  Paul Eggleton (1):
        insane.bbclass: skip rdepends QA checks for kernel / modules

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib kyu3/verify-checksum
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/verify-checksum

Yu Ke (1):
  fetcher2: retry mirror if upstream checksum mismatch

 bitbake/lib/bb/fetch2/__init__.py |   31 ++++++++++++++++++++-----------
 1 files changed, 20 insertions(+), 11 deletions(-)

[PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch

[Yu Ke]

This patch is for [YOCTO #1085] fix.

If the upstream fails a checksum, retry from the MIRROR before giving up.
This will add more robust fetching if an upstream serves a bad file or webpage.

fetching of distcc prior to the move from samba -> googlecode is a good example
of this.

Signed-off-by: Yu Ke <ke.yu@intel.com>
---
 bitbake/lib/bb/fetch2/__init__.py |   31 ++++++++++++++++++++-----------
 1 files changed, 20 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index e9a64c5..d39f094 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -300,6 +300,22 @@ def verify_checksum(u, ud, d):
     if ud.sha256_expected != sha256data:
         raise SHA256SumError(ud.localpath, ud.sha256_expected, sha256data, u)
 
+def update_stamp(u, ud, d):
+    """
+        donestamp is file stamp indicating the whole fetching is done
+        this function update the stamp after verifying the checksum
+    """
+    if os.path.exists(ud.donestamp):
+        # Touch the done stamp file to show active use of the download
+        try:
+            os.utime(ud.donestamp, None)
+        except:
+            # Errors aren't fatal here
+            pass
+    else:
+        verify_checksum(u, ud, d)
+        open(ud.donestamp, 'w').close()
+
 def subprocess_setup():
     import signal
     # Python installs a SIGPIPE handler by default. This is usually not what
@@ -932,6 +948,9 @@ class Fetch(object):
                         if hasattr(m, "build_mirror_data"):
                             m.build_mirror_data(u, ud, self.d)
                         localpath = ud.localpath
+                        # early checksum verify, so that if checksum mismatched,
+                        # fetcher still have chance to fetch from mirror
+                        update_stamp(u, ud, self.d)
 
                     except bb.fetch2.NetworkAccess:
                         raise
@@ -948,17 +967,7 @@ class Fetch(object):
                 if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1):
                     raise FetchError("Unable to fetch URL %s from any source." % u, u)
 
-                if os.path.exists(ud.donestamp):
-                    # Touch the done stamp file to show active use of the download
-                    try:
-                        os.utime(ud.donestamp, None)
-                    except:
-                        # Errors aren't fatal here
-                        pass
-                else:
-                    # Only check the checksums if we've not seen this item before, then create the stamp
-                    verify_checksum(u, ud, self.d)
-                    open(ud.donestamp, 'w').close()
+                update_stamp(u, ud, self.d)
 
             finally:
                 bb.utils.unlockfile(lf)
-- 
1.7.0.4

Re: [PATCH 1/1] fetcher2: retry mirror if upstream checksum mismatch

[Richard Purdie]

On Wed, 2011-07-13 at 17:08 +0800, Yu Ke wrote:
> This patch is for [YOCTO #1085] fix.
> 
> If the upstream fails a checksum, retry from the MIRROR before giving up.
> This will add more robust fetching if an upstream serves a bad file or webpage.
> 
> fetching of distcc prior to the move from samba -> googlecode is a good example
> of this.
> 
> Signed-off-by: Yu Ke <ke.yu@intel.com>
> ---
>  bitbake/lib/bb/fetch2/__init__.py |   31 ++++++++++++++++++++-----------
>  1 files changed, 20 insertions(+), 11 deletions(-)

Merged to master, thanks.

Richard