[PATCH] classes/sstate.bbclass: Improve checkstatus using connection cache.

[PATCH] classes/sstate.bbclass: Improve checkstatus using connection cache.

[Aníbal Limón]

This patch depends on changes did in bitbake [1] and oe/utils.py [2], don't merge
before it.

For complete review exist an integration branch at,

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/fetch2-wget-checkstatus

[1] http://lists.openembedded.org/pipermail/bitbake-devel/2015-June/005912.html
[2] http://lists.openembedded.org/pipermail/openembedded-core/2015-June/106430.html

Aníbal Limón (1):
  classes/sstate.bbclass: Improve checkstatus using connection cache.

 meta/classes/sstate.bbclass | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

-- 
1.9.1

[PATCH] classes/sstate.bbclass: Improve checkstatus using connection cache.

[Aníbal Limón]

Use FetcherConnectionCache to improve times when do checkstatus over
sstate resources.

Add debug message about how many SState objecs are available also
how many time takes to get it.

[YOCTO #7796]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/classes/sstate.bbclass | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 1e2d4f6..0345716 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -739,6 +739,13 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
         if localdata.getVar('BB_NO_NETWORK', True) == "1" and localdata.getVar('SSTATE_MIRROR_ALLOW_NETWORK', True) == "1":
             localdata.delVar('BB_NO_NETWORK')
 
+        from bb.fetch2 import FetchConnectionCache
+        def checkstatus_init(thread_worker):
+            thread_worker.connection_cache = FetchConnectionCache()
+
+        def checkstatus_end(thread_worker):
+            thread_worker.connection_cache.close_connections()
+
         def checkstatus(thread_worker, arg):
             (task, sstatefile) = arg
 
@@ -748,7 +755,8 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
             bb.debug(2, "SState: Attempting to fetch %s" % srcuri)
 
             try:
-                fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2)
+                fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2,
+                            connection_cache=thread_worker.connection_cache)
                 fetcher.checkstatus()
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 ret.append(task)
@@ -768,14 +776,21 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False):
             tasklist.append((task, sstatefile))
 
         if tasklist:
+            from datetime import datetime
             bb.note("Checking sstate mirror object availability (for %s objects)" % len(tasklist))
             import multiprocessing
             nproc = min(multiprocessing.cpu_count(), len(tasklist))
-            pool = oe.utils.ThreadedPool(nproc, len(tasklist))
+
+            timei = datetime.now()
+            pool = oe.utils.ThreadedPool(nproc, len(tasklist),
+                    worker_init=checkstatus_init, worker_end=checkstatus_end)
             for t in tasklist:
                 pool.add_task(checkstatus, t)
             pool.start()
             pool.wait_completion()
+            timee = datetime.now()
+
+            bb.debug(2, "SState mirror objects available %d, run time %s" % (len(ret), (timee - timei)))
 
     inheritlist = d.getVar("INHERIT", True)
     if "toaster" in inheritlist:
-- 
1.9.1

Re: [PATCH] classes/sstate.bbclass: Improve checkstatus using connection cache.

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On Tue, Jun 23, 2015 at 1:02 PM, Aníbal Limón <anibal.limon@linux.intel.com>
wrote:

> Use FetcherConnectionCache to improve times when do checkstatus over
> sstate resources.
>
> Add debug message about how many SState objecs are available also
> how many time takes to get it.


If this is an RFC, it should be marked as such in the subject. If not, and
you actually want it applied, I don’t think including debug messages with
performance metrics is appropriate.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1008 bytes --]

Re: [PATCH] classes/sstate.bbclass: Improve checkstatus using connection cache.

[Aníbal Limón]

On 23/06/15 16:42, Christopher Larson wrote:
> On Tue, Jun 23, 2015 at 1:02 PM, Aníbal Limón <anibal.limon@linux.intel.com>
> wrote:
>
>> Use FetcherConnectionCache to improve times when do checkstatus over
>> sstate resources.
>>
>> Add debug message about how many SState objecs are available also
>> how many time takes to get it.
>
> If this is an RFC, it should be marked as such in the subject. If not, and
> you actually want it applied, I don’t think including debug messages with
> performance metrics is appropriate.
I agree, sending v2 removing debug messages.
     alimon