[PATCH 1/3] gitsm.py: use download cache for git submodules too

[PATCH 1/3] gitsm.py: use download cache for git submodules too

[Mikko Rapeli]

Currently download caches for recipes using gitsm are filled also with
the submodule repositories but when the git submodule tree is reused from
cache, only the base repository is cloned. After a source URL rewrite,
'git submodule update --init --recursive' is called in the tree
which downloads the needed submodules from upstream URL's.
This wastes bandwidth since download cache already had the needed
commits for the submodule repositories.

With this change, also the submodule repositories are copied to the
work space from download cache and the download from upstream can
be avoided. Using plain 'cp -a' since git cloning doesn't work for the
submodule directories.

Note that using BB_NO_NETWORK does not stop the submodule downloads
from 'git submodule update --init --recursive'. Thus tested by
breaking access to remote repository via /etc/hosts line like:

127.0.0.1       localhost github.com

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
---
 bitbake/lib/bb/fetch2/gitsm.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index 0aff100..20a9c40 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -131,5 +131,13 @@ class GitSM(Git):
         Git.unpack(self, ud, destdir, d)
 
         if self.uses_submodules(ud, d, ud.destdir):
+            # Copy also submodule trees from download cache instead of
+            # downloading again from the upstream repository.
+            # For some reason git does not clone them.
+            clone_modules = os.path.join(ud.clonedir, "modules")
+            if os.path.exists(clone_modules):
+                dest_modules = os.path.join(ud.destdir, ".git")
+                runfetchcmd("cp -a " + clone_modules + " " + dest_modules, d)
+
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
             runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
-- 
1.9.1

[PATCH 2/3] gitsm.py: support BB_NO_NETWORK during do_unpack

[Mikko Rapeli]

'git submodule update --init -recursive' will clone remote repositories
if local tree does not have the referenced commits. Add --no-fetch
to git submodule update if BB_NO_NETWORK is defined and only local
download caches should be used.

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
---
 bitbake/lib/bb/fetch2/gitsm.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index 20a9c40..007d06e 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -140,4 +140,7 @@ class GitSM(Git):
                 runfetchcmd("cp -a " + clone_modules + " " + dest_modules, d)
 
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+            if d.getVar("BB_NO_NETWORK", True) == "1":
+                runfetchcmd(ud.basecmd + " submodule update --no-fetch --init --recursive", d, workdir=ud.destdir)
+            else:
+                runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
-- 
1.9.1

[PATCH 3/3] cve-check-tool: change BB_NO_NETWORK error to a warning

[Mikko Rapeli]

It is perfectly fine to execute cve_check tasks against a cached
CVE database during a BB_NO_NETWORK build.

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
---
 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
index 4829b11..7b70daa 100644
--- a/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
+++ b/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb
@@ -30,7 +30,7 @@ CFLAGS_append = " -Wno-error=pedantic"
 
 do_populate_cve_db() {
     if [ "${BB_NO_NETWORK}" = "1" ] ; then
-        bberror "BB_NO_NETWORK is set; Can't update cve-check-tool database, CVEs won't be checked"
+        bbwarn "BB_NO_NETWORK is set; Can't update cve-check-tool database, new CVEs won't be detected"
         return
     fi
 
-- 
1.9.1

✗ patchtest: failure for "gitsm.py: use download cache f..." and 2 more

[Patchwork]

== Series Details ==

Series: "gitsm.py: use download cache f..." and 2 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/8734/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at cc319b6dcc)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for "gitsm.py: use download cache f..." and 2 more

[Mikko.Rapeli]

On Wed, Sep 06, 2017 at 11:34:54AM +0000, Patchwork wrote:
> == Series Details ==
> 
> Series: "gitsm.py: use download cache f..." and 2 more
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/8734/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
>   Suggested fix    Rebase your series on top of targeted branch
>   Targeted branch  master (currently at cc319b6dcc)

Uh, the patch is based on poky master at:

commit 8b4f16a9cbbaf521461f699b7264fac2ac872581 (refs/remotes/upstream/master-ne
Author: Jussi Kukkonen <jussi.kukkonen@intel.com>
Date:   Mon Sep 4 11:39:24 2017 +0300

    mesa-gl: Fix build after recent mesa PACKAGECONFIG changes
    
    48d39cf43b added "opengl" PACKAGECONFIG option to mesa: before that
    the configuration was always enabled. "opengl" should have been added
    to mesa-gl default PACKAGECONFIG but wasn't: do it now.
    
    (From OE-Core rev: cc319b6dcc5b4a5019fb91c9771b12ce17f3c953)
    
    Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

and there are no newer changes available. Should I port the patch to oe-core
directly and submit again?

Maybe this worked in the past because someone else did the applying and
merging manually?

-Mikko

Re: ✗ patchtest: failure for "gitsm.py: use download cache f..." and 2 more

[Mikko.Rapeli]

On Wed, Sep 06, 2017 at 12:14:48PM +0000, Mikko.Rapeli@bmw.de wrote:
> On Wed, Sep 06, 2017 at 11:34:54AM +0000, Patchwork wrote:
> > == Series Details ==
> > 
> > Series: "gitsm.py: use download cache f..." and 2 more
> > Revision: 1
> > URL   : https://patchwork.openembedded.org/series/8734/
> > State : failure
> > 
> > == Summary ==
> > 
> > 
> > Thank you for submitting this patch series to OpenEmbedded Core. This is
> > an automated response. Several tests have been executed on the proposed
> > series by patchtest resulting in the following failures:
> > 
> > 
> > 
> > * Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
> >   Suggested fix    Rebase your series on top of targeted branch
> >   Targeted branch  master (currently at cc319b6dcc)
> 
> Uh, the patch is based on poky master at:
> 
> commit 8b4f16a9cbbaf521461f699b7264fac2ac872581 (refs/remotes/upstream/master-ne
> Author: Jussi Kukkonen <jussi.kukkonen@intel.com>
> Date:   Mon Sep 4 11:39:24 2017 +0300
> 
>     mesa-gl: Fix build after recent mesa PACKAGECONFIG changes
>     
>     48d39cf43b added "opengl" PACKAGECONFIG option to mesa: before that
>     the configuration was always enabled. "opengl" should have been added
>     to mesa-gl default PACKAGECONFIG but wasn't: do it now.
>     
>     (From OE-Core rev: cc319b6dcc5b4a5019fb91c9771b12ce17f3c953)
>     
>     Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
>     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> and there are no newer changes available. Should I port the patch to oe-core
> directly and submit again?
> 
> Maybe this worked in the past because someone else did the applying and
> merging manually?

Ah, sorry. The patch is for bitbake, not oe-core. Will send to correct list.

-Mikko