[bitbake][PATCH 1/3] bitbake: fetch2/repo: add support for projects to be fetched simultaneously

[bitbake][PATCH 1/3] bitbake: fetch2/repo: add support for projects to be fetched simultaneously

[Sourabh Banerjee]

repo tool can fetch projects simultaneously.
As per repo sync documentation:
  Usage: repo sync [<project>...]
  Options:
    -j JOBS, --jobs=JOBS  projects to fetch simultaneously (default 1)

By supplying jobs=<some_number> recipe can choose to use the
simultaneous fetch feature of repo tool.
Example:
  SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;jobs=8"

Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
---
 bitbake/lib/bb/fetch2/repo.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py
index fa4cb81..8547b9e 100644
--- a/bitbake/lib/bb/fetch2/repo.py
+++ b/bitbake/lib/bb/fetch2/repo.py
@@ -62,11 +62,21 @@ class Repo(FetchMethod):
         repodir = os.path.join(codir, "repo")
         bb.utils.mkdirhier(repodir)
         if not os.path.exists(os.path.join(repodir, ".repo")):
+
             bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
             runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir)
 
+        # Projects to fetch simultaneously at time of 'repo sync'.
+        # If SRC_URI provides parallel jobs parameter as follows:
+        #   SRC_URI = "repo://REPOROOT;branch=some_branch;jobs=8"
+        # Sync is run as 'repo sync -j8'
+        # When jobs parameter is not passed with SRC_URI then
+        # repo tool's default behavior is to fetch 1 project at a time.
+        jobs = ud.parm.get('jobs', '')
+        if jobs:
+            jobs = '-j' + jobs
         bb.fetch2.check_network_access(d, "%s sync %s" % (ud.basecmd, ud.url), ud.url)
-        runfetchcmd("%s sync" % ud.basecmd, d, workdir=repodir)
+        runfetchcmd("%s sync %s" % (ud.basecmd, jobs), d, workdir=repodir)
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
-- 
2.7.4



-- 
Regards,
Sourabh

[bitbake][PATCH 2/3] bitbake: fetch2/repo: add support for sync specific projects

[Sourabh Banerjee]

repo tool can fetch projects by name. I.e., if there are
multiple projects listed in the manifest file.

 <manifest>
  <project name="project1"  path="sync path1"  revision="sha256" />
  <project name="project2"  path="sync path2"  revision="sha256" />
 </manifest>

Documented Usage: repo sync [<project>...]

By supplying project=<project_name> recipe can choose to sync a specific
project.
Example:
  SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;project=project1"

This is helpful when recipe needs to sync only a specific project
from a manifest that lists multiple projects.

Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
---
 bitbake/lib/bb/fetch2/repo.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py
index 8547b9e..e7c5255 100644
--- a/bitbake/lib/bb/fetch2/repo.py
+++ b/bitbake/lib/bb/fetch2/repo.py
@@ -41,7 +41,10 @@ class Repo(FetchMethod):
         if not ud.manifest.endswith('.xml'):
             ud.manifest += '.xml'
 
-        ud.localfile = d.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch))
+        project = ud.parm.get('project', '')
+        if len(project):
+            project = '_' + project
+        ud.localfile = d.expand("repo_%s%s_%s_%s%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch, project.replace("/", ".")))
 
     def download(self, ud, d):
         """Fetch url"""
@@ -52,7 +55,8 @@ class Repo(FetchMethod):
 
         repodir = d.getVar("REPODIR") or (d.getVar("DL_DIR") + "/repo")
         gitsrcname = "%s%s" % (ud.host, ud.path.replace("/", "."))
-        codir = os.path.join(repodir, gitsrcname, ud.manifest)
+        project = ud.parm.get('project', '')
+        codir = os.path.join(repodir, gitsrcname, ud.manifest, project.replace("/", "."))
 
         if ud.user:
             username = ud.user + "@"
@@ -76,7 +80,7 @@ class Repo(FetchMethod):
         if jobs:
             jobs = '-j' + jobs
         bb.fetch2.check_network_access(d, "%s sync %s" % (ud.basecmd, ud.url), ud.url)
-        runfetchcmd("%s sync %s" % (ud.basecmd, jobs), d, workdir=repodir)
+        runfetchcmd("%s sync %s %s" % (ud.basecmd, jobs, project), d, workdir=repodir)
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
-- 
2.7.4



-- 
Regards,
Sourabh

[bitbake][PATCH 3/3] bitbake: fetch2/repo: add support for --repo-url and --repo-branch

[Sourabh Banerjee]

repo tool provides following options to use a stable or preferred
version of the tool
  repo Version options:
    --repo-url=URL      repo repository location
    --repo-branch=REVISION
                        repo branch or revision

By supplying repo_url=<uri> and repo_branch=<branch> recipe may choose
to sync a specific version of repo tool.

Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
---
 bitbake/lib/bb/fetch2/repo.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py
index e7c5255..902337f 100644
--- a/bitbake/lib/bb/fetch2/repo.py
+++ b/bitbake/lib/bb/fetch2/repo.py
@@ -66,9 +66,15 @@ class Repo(FetchMethod):
         repodir = os.path.join(codir, "repo")
         bb.utils.mkdirhier(repodir)
         if not os.path.exists(os.path.join(repodir, ".repo")):
+            repo_url = ud.parm.get('repo_url', '')
+            if repo_url:
+                repo_url = '--repo-url=' + repo_url
+            repo_branch = ud.parm.get('repo_branch', '')
+            if repo_branch:
+                repo_branch = '--repo-branch=' + repo_branch
 
             bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
-            runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir)
+            runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s %s %s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path, repo_url, repo_branch), d, workdir=repodir)
 
         # Projects to fetch simultaneously at time of 'repo sync'.
         # If SRC_URI provides parallel jobs parameter as follows:
-- 
2.7.4



-- 
Regards,
Sourabh

Re: [OE-core] [bitbake][PATCH 3/3] bitbake: fetch2/repo: add support for --repo-url and --repo-branch

[Richard Purdie]

On Sun, 2021-05-16 at 16:53 +0530, Sourabh Banerjee wrote:
> repo tool provides following options to use a stable or preferred
> version of the tool
>   repo Version options:
>     --repo-url=URL      repo repository location
>     --repo-branch=REVISION
>                         repo branch or revision
> 
> By supplying repo_url=<uri> and repo_branch=<branch> recipe may choose
> to sync a specific version of repo tool.
> 
> Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
> ---
>  bitbake/lib/bb/fetch2/repo.py | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py
> index e7c5255..902337f 100644
> --- a/bitbake/lib/bb/fetch2/repo.py
> +++ b/bitbake/lib/bb/fetch2/repo.py
> @@ -66,9 +66,15 @@ class Repo(FetchMethod):
>          repodir = os.path.join(codir, "repo")
>          bb.utils.mkdirhier(repodir)
>          if not os.path.exists(os.path.join(repodir, ".repo")):
> +            repo_url = ud.parm.get('repo_url', '')
> +            if repo_url:
> +                repo_url = '--repo-url=' + repo_url
> +            repo_branch = ud.parm.get('repo_branch', '')
> +            if repo_branch:
> +                repo_branch = '--repo-branch=' + repo_branch
>  
> 
> 
> 
>              bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
> -            runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir)
> +            runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s %s %s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path, repo_url, repo_branch), d, workdir=repodir)
>  
> 

In your other patch you make the new parameters update the mirror tarball 
name. I'm wondering if we also need to do something like that here as 
presumably, the fetched output might vary depending on which version 
of repo was used?

Bitbake patches should go to the bitbake-devel list. I'm also
wondering if there should be new fetcher tests these changes should be
adding to lib/bb/test/fetch.py, run using bitbake-selftest?

Cheers,

Richard

Re: [OE-core] [bitbake][PATCH 3/3] bitbake: fetch2/repo: add support for --repo-url and --repo-branch

[Sourabh Banerjee]

On 2021-05-16 18:13, Richard Purdie wrote:
> On Sun, 2021-05-16 at 16:53 +0530, Sourabh Banerjee wrote:
>> repo tool provides following options to use a stable or preferred
>> version of the tool
>>   repo Version options:
>>     --repo-url=URL      repo repository location
>>     --repo-branch=REVISION
>>                         repo branch or revision
>> 
>> By supplying repo_url=<uri> and repo_branch=<branch> recipe may choose
>> to sync a specific version of repo tool.
>> 
>> Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
>> ---
>>  bitbake/lib/bb/fetch2/repo.py | 8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>> 
>> diff --git a/bitbake/lib/bb/fetch2/repo.py 
>> b/bitbake/lib/bb/fetch2/repo.py
>> index e7c5255..902337f 100644
>> --- a/bitbake/lib/bb/fetch2/repo.py
>> +++ b/bitbake/lib/bb/fetch2/repo.py
>> @@ -66,9 +66,15 @@ class Repo(FetchMethod):
>>          repodir = os.path.join(codir, "repo")
>>          bb.utils.mkdirhier(repodir)
>>          if not os.path.exists(os.path.join(repodir, ".repo")):
>> +            repo_url = ud.parm.get('repo_url', '')
>> +            if repo_url:
>> +                repo_url = '--repo-url=' + repo_url
>> +            repo_branch = ud.parm.get('repo_branch', '')
>> +            if repo_branch:
>> +                repo_branch = '--repo-branch=' + repo_branch
>>  
>> 
>> 
>> 
>>              bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u 
>> %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, 
>> username, ud.host, ud.path), ud.url)
>> -            runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % 
>> (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, 
>> ud.path), d, workdir=repodir)
>> +            runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s %s %s" % 
>> (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, 
>> ud.path, repo_url, repo_branch), d, workdir=repodir)
>>  
>> 
> 
> In your other patch you make the new parameters update the mirror 
> tarball 
> name. I'm wondering if we also need to do something like that here as 
> presumably, the fetched output might vary depending on which version 
> of repo was used?

Hi Richard,

Here is what I understand. The 'repo' tool installed on host, let's say 
at /usr/bin/repo
handles the options --repo-url and --repo-branch in 'init' sub-command 
and downloads the
'repo' tool copy from uri-branch under <working-dir>/.repo/repo/repo. 
All subsequent sub-commands
such as 'sync' will work using the tool copy under 
<working-dir>/.repo/repo/repo.
As a recipe developer/maintainer I find --repo-url and --repo-branch 
useful when I know my manifest
has a XML key-value pair that is only supported certain version (& up) 
of 'repo' tool.
At such a point I will code my SRC_URI as
SRC_URI = "\
repo://URI/my-manifest-project.git;\
protocol=git;branch=main;project=platform/my-project;\
repo_url=git://gerrit.googlesource.com/git-repo;repo_branch=stable\
"
Here I thought as my recipe file carries the URI-BRANCH combo to repo 
tool it needs, shall be good?
With this thought I did not code it in mirror tarball name.

> 
> Bitbake patches should go to the bitbake-devel list. I'm also
I have never submitted to bitbake-devel, but I think I will take these 
there then?

> wondering if there should be new fetcher tests these changes should be
> adding to lib/bb/test/fetch.py, run using bitbake-selftest?
Thanks for valuable suggestion, I will look into how tests are developed 
for fetcher.
If possible, please point me to fetcher test, that you consider to be a 
well written one.
I will learn from it.

> 
> Cheers,
> 
> Richard


-- 
Regards,
Sourabh

Re: [OE-core] [bitbake][PATCH 1/3] bitbake: fetch2/repo: add support for projects to be fetched simultaneously

[Richard Purdie]

On Sun, 2021-05-16 at 16:53 +0530, Sourabh Banerjee wrote:
> repo tool can fetch projects simultaneously.
> As per repo sync documentation:
>   Usage: repo sync [<project>...]
>   Options:
>     -j JOBS, --jobs=JOBS  projects to fetch simultaneously (default 1)
> 
> By supplying jobs=<some_number> recipe can choose to use the
> simultaneous fetch feature of repo tool.
> Example:
>   SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;jobs=8"
> 
> Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
> ---
>  bitbake/lib/bb/fetch2/repo.py | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)

Does this value make sense as a per repository setting or an overall 
build setting?

Cheers,

Richard

Re: [OE-core] [bitbake][PATCH 1/3] bitbake: fetch2/repo: add support for projects to be fetched simultaneously

[Sourabh Banerjee]

On 2021-05-16 18:10, Richard Purdie wrote:
> On Sun, 2021-05-16 at 16:53 +0530, Sourabh Banerjee wrote:
>> repo tool can fetch projects simultaneously.
>> As per repo sync documentation:
>>   Usage: repo sync [<project>...]
>>   Options:
>>     -j JOBS, --jobs=JOBS  projects to fetch simultaneously (default 1)
>> 
>> By supplying jobs=<some_number> recipe can choose to use the
>> simultaneous fetch feature of repo tool.
>> Example:
>>   SRC_URI = 
>> "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;jobs=8"
>> 
>> Signed-off-by: Sourabh Banerjee <sbanerje@codeaurora.org>
>> ---
>>  bitbake/lib/bb/fetch2/repo.py | 12 +++++++++++-
>>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> Does this value make sense as a per repository setting or an overall 
> build setting?

Hi Richard,

The '--jobs' option for the repo tool parallelizes fetch w.r.t 
manifest.xml.
For example, if there are 8 projects listed in a manifest.xml then 
passing-j4
will run 4 parallel jobs fetching 4 projects simultaneously at a time.
So it is related to whole manifest being operated upon by the repo tool.

> 
> Cheers,
> 
> Richard


-- 
Regards,
Sourabh