[PATCH 0/3] sstate fetch improvements

[PATCH 0/3] sstate fetch improvements

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

The following changes since commit b4f208f418d18f2a4e78a56bebacef481061d917:

  tar: don't mv tar for nativesdk (2013-06-10 18:06:33 +0100)

are available in the git repository at:

  https://github.com/kergoth/oe-core.git improve-sstate-fetch

Christopher Larson (3):
  sstate.bbclass: kill MIRRORS and FILESPATH to speed up fetching
  sstate.bbclass: fetch archive+siginfo in a single pstaging_fetch
  sstate.bbclass: brute force silence fetch errors

 meta/classes/sstate.bbclass | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

-- 
1.8.3

[PATCH 1/3] sstate.bbclass: kill MIRRORS and FILESPATH to speed up fetching

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

The default FILESPATH isn't really of use, as we don't expect to find sstate
archives buried in layer recipe directories, and the default MIRRORS is
intended for use for fetching SRC_URI, not sstate.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index e74c642..3aa4371 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -508,6 +508,8 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
     bb.mkdirhier(dldir)
 
+    localdata.delVar('MIRRORS')
+    localdata.delVar('FILESPATH')
     localdata.setVar('DL_DIR', dldir)
     localdata.setVar('PREMIRRORS', mirrors)
     localdata.setVar('SRC_URI', srcuri)
-- 
1.8.3

[PATCH 2/3] sstate.bbclass: fetch archive+siginfo in a single pstaging_fetch

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This avoids unnecessary duplication of setup. The only visible change in
behavior will be the case if siginfo exists and the archive does not, in which
case it'll redownload both, but this doesn't seem unresasonable to me,
particularly since the archive is downloaded first, making this case
particularly unlikely.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sstate.bbclass | 32 +++++++++++++++-----------------
 1 file changed, 15 insertions(+), 17 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 3aa4371..28dc312 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -224,8 +224,6 @@ def sstate_installpkg(ss, d):
 
     if not os.path.exists(sstatepkg):
         pstaging_fetch(sstatefetch, sstatepkg, d)
-    if not os.path.exists(sstatepkg + ".siginfo"):
-        pstaging_fetch(sstatefetch + ".siginfo", sstatepkg + ".siginfo", d)
 
     if not os.path.isfile(sstatepkg):
         bb.note("Staging package %s does not exist" % sstatepkg)
@@ -504,30 +502,30 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
     bb.data.update_data(localdata)
 
     dldir = localdata.expand("${SSTATE_DIR}")
-    srcuri = "file://" + sstatefetch
-
     bb.mkdirhier(dldir)
 
     localdata.delVar('MIRRORS')
     localdata.delVar('FILESPATH')
     localdata.setVar('DL_DIR', dldir)
     localdata.setVar('PREMIRRORS', mirrors)
-    localdata.setVar('SRC_URI', srcuri)
 
     # Try a fetch from the sstate mirror, if it fails just return and
     # we will build the package
-    try:
-        fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
-        fetcher.download()        
-
-        # Need to optimise this, if using file:// urls, the fetcher just changes the local path
-        # For now work around by symlinking
-        localpath = bb.data.expand(fetcher.localpath(srcuri), localdata)
-        if localpath != sstatepkg and os.path.exists(localpath) and not os.path.exists(sstatepkg):
-            os.symlink(localpath, sstatepkg)
-
-    except bb.fetch2.BBFetchException:
-        pass
+    for srcuri in ['file://{0}'.format(sstatefetch),
+                   'file://{0}.siginfo'.format(sstatefetch)]:
+        localdata.setVar('SRC_URI', srcuri)
+        try:
+            fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+            fetcher.download()
+
+            # Need to optimise this, if using file:// urls, the fetcher just changes the local path
+            # For now work around by symlinking
+            localpath = bb.data.expand(fetcher.localpath(srcuri), localdata)
+            if localpath != sstatepkg and os.path.exists(localpath) and not os.path.exists(sstatepkg):
+                os.symlink(localpath, sstatepkg)
+
+        except bb.fetch2.BBFetchException:
+            break
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
-- 
1.8.3

[PATCH 3/3] sstate.bbclass: brute force silence fetch errors

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

bb.fetch2.Fetcher.download() prints any fetch errors that come from the
underlying fetcher, rather than letting the caller do so, which means there's
no way to silence the fetch error messages without manipulating the loggers.
So we do so. The reason for doing this is that the user doesn't care if we
failed to fetch an sstate archive, the failure mode is just to rerun the real
task anyway.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sstate.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 28dc312..5b9d86e 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -491,6 +491,7 @@ def sstate_package(ss, d):
 
 def pstaging_fetch(sstatefetch, sstatepkg, d):
     import bb.fetch2
+    import logging
 
     # Only try and fetch if the user has configured a mirror
     mirrors = d.getVar('SSTATE_MIRRORS', True)
@@ -514,6 +515,8 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
     for srcuri in ['file://{0}'.format(sstatefetch),
                    'file://{0}.siginfo'.format(sstatefetch)]:
         localdata.setVar('SRC_URI', srcuri)
+        oldlevel = bb.fetch2.logger.level
+        bb.fetch2.logger.setLevel(logging.CRITICAL)
         try:
             fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
             fetcher.download()
@@ -526,6 +529,8 @@ def pstaging_fetch(sstatefetch, sstatepkg, d):
 
         except bb.fetch2.BBFetchException:
             break
+        finally:
+            bb.fetch2.logger.setLevel(oldlevel)
 
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
-- 
1.8.3

Re: [PATCH 1/3] sstate.bbclass: kill MIRRORS and FILESPATH to speed up fetching

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

On Mon, Jun 10, 2013 at 5:46 PM, Christopher Larson <kergoth@gmail.com>wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> The default FILESPATH isn't really of use, as we don't expect to find
> sstate
> archives buried in layer recipe directories, and the default MIRRORS is
> intended for use for fetching SRC_URI, not sstate.
>
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>

Reviewed-by: Otavio Salvador <otavio@ossystems.com.br>

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

[-- Attachment #2: Type: text/html, Size: 1326 bytes --]

Re: [PATCH 2/3] sstate.bbclass: fetch archive+siginfo in a single pstaging_fetch

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On Mon, Jun 10, 2013 at 5:46 PM, Christopher Larson <kergoth@gmail.com>wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> This avoids unnecessary duplication of setup. The only visible change in
> behavior will be the case if siginfo exists and the archive does not, in
> which
> case it'll redownload both, but this doesn't seem unresasonable to me,
> particularly since the archive is downloaded first, making this case
> particularly unlikely.
>
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>

Reviewed-by: Otavio Salvador <otavio@ossystems.com.br>

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

[-- Attachment #2: Type: text/html, Size: 1502 bytes --]

Re: [PATCH 3/3] sstate.bbclass: brute force silence fetch errors

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

On Mon, Jun 10, 2013 at 5:46 PM, Christopher Larson <kergoth@gmail.com>wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> bb.fetch2.Fetcher.download() prints any fetch errors that come from the
> underlying fetcher, rather than letting the caller do so, which means
> there's
> no way to silence the fetch error messages without manipulating the
> loggers.
> So we do so. The reason for doing this is that the user doesn't care if we
> failed to fetch an sstate archive, the failure mode is just to rerun the
> real
> task anyway.
>
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>

Reviewed-by: Otavio Salvador <otavio@ossystems.com.br>

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

[-- Attachment #2: Type: text/html, Size: 1585 bytes --]