[PATCH 0/2] RFC: Integrating gitpkgv functionality

[PATCH 0/2] RFC: Integrating gitpkgv functionality

[Mike Looijmans]

Follow-up on the discussion here:
http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html

This is a preliminary implementation of gitpkgv in OE-core. It modifies the
fetcher to optionally provide an incremental numbering scheme for GIT
repositories.

These two patches implement this. The change in __init__ is basically ready for
public consumption. The modification to git.py is a prototype that I've used to
test feasibility. It imitates gitpkgv perfectly for the recipes I tried.

To make this ready for production, we should provide better names and likely
a GITPKGV or similarly named variable that can be used in recipes instead of
"${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}".

Mike Looijmans (2):
  fetch2/__init__.py: Make get_srcrev output configurable
  fetch2/git.py: Add gitpkgv_revision alternative version information

 lib/bb/fetch2/__init__.py |    9 ++++++---
 lib/bb/fetch2/git.py      |   25 +++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 3 deletions(-)

-- 
1.7.9.5

[PATCH 1/2] fetch2/__init__.py: Make get_srcrev output configurable

[Mike Looijmans]

From: Mike Looijmans <milo-software@users.sourceforge.net>

The idea here is to support package version numbering similar to gitpkgv in
meta-openembedded. This commit is the first step towards such functionality.

The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
per Richard's suggestion [1]. While writing this, I noticed that it would
become a copy of get_srcrev with only two lines changed. So to create something
more Pythonic than a boolean argument and conditionals around the calls to the
fetcher's sortable_revision, I just made the method to be called on the fetcher
an argument to the method. Defaulting to 'sortable_revision' prevents affecting
existing code.

Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
set the following in a recipe:
  PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
and this would yield the same result as gitpkgv's GITPKGV variable.

See for the discussion leading to this change:
[1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html

Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
---
 lib/bb/fetch2/__init__.py |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 599ea8c..0eb1a23 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -618,7 +618,7 @@ def get_autorev(d):
         d.setVar('__BB_DONT_CACHE', '1')
     return "AUTOINC"
 
-def get_srcrev(d):
+def get_srcrev(d, method_name='sortable_revision'):
     """
     Return the revsion string, usually for use in the version string (PV) of the current package
     Most packages usually only have one SCM so we just pass on the call.
@@ -627,6 +627,9 @@ def get_srcrev(d):
 
     The idea here is that we put the string "AUTOINC+" into return value if the revisions are not 
     incremental, other code is then responsible for turning that into an increasing value (if needed)
+
+    A method_name can be supplied to retrieve an alternatively formatted revision from a fetcher, if
+    that fetcher provides a method with the given name and the same signature as sortable_revision.
     """
 
     scms = []
@@ -640,7 +643,7 @@ def get_srcrev(d):
         raise FetchError("SRCREV was used yet no valid SCM was found in SRC_URI")
 
     if len(scms) == 1 and len(urldata[scms[0]].names) == 1:
-        autoinc, rev = urldata[scms[0]].method.sortable_revision(urldata[scms[0]], d, urldata[scms[0]].names[0])
+        autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0])
         if len(rev) > 10:
             rev = rev[:10]
         if autoinc:
@@ -658,7 +661,7 @@ def get_srcrev(d):
     for scm in scms:
         ud = urldata[scm]
         for name in ud.names:
-            autoinc, rev = ud.method.sortable_revision(ud, d, name)
+            autoinc, rev = getattr(ud.method, method_name)(ud, d, name)
             seenautoinc = seenautoinc or autoinc
             if len(rev) > 10:
                 rev = rev[:10]
-- 
1.7.9.5

Re: [PATCH 1/2] fetch2/__init__.py: Make get_srcrev output configurable

[Otavio Salvador]

On Sat, Jan 31, 2015 at 12:06 PM, Mike Looijmans
<mike.looijmans@topic.nl> wrote:
> From: Mike Looijmans <milo-software@users.sourceforge.net>
>
> The idea here is to support package version numbering similar to gitpkgv in
> meta-openembedded. This commit is the first step towards such functionality.
>
> The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
> per Richard's suggestion [1]. While writing this, I noticed that it would
> become a copy of get_srcrev with only two lines changed. So to create something
> more Pythonic than a boolean argument and conditionals around the calls to the
> fetcher's sortable_revision, I just made the method to be called on the fetcher
> an argument to the method. Defaulting to 'sortable_revision' prevents affecting
> existing code.
>
> Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
> set the following in a recipe:
>   PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
> and this would yield the same result as gitpkgv's GITPKGV variable.
>
> See for the discussion leading to this change:
> [1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
>
> Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>

I do enjoyed all the design but the way to set the PKGV.

Maybe we could have a way to do it more transparently? Any idea how?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 1/2] fetch2/__init__.py: Make get_srcrev output configurable

[Mike Looijmans]

On 02-02-15 21:50, Otavio Salvador wrote:
> On Sat, Jan 31, 2015 at 12:06 PM, Mike Looijmans
> <mike.looijmans@topic.nl> wrote:
>> From: Mike Looijmans <milo-software@users.sourceforge.net>
>>
>> The idea here is to support package version numbering similar to gitpkgv in
>> meta-openembedded. This commit is the first step towards such functionality.
>>
>> The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
>> per Richard's suggestion [1]. While writing this, I noticed that it would
>> become a copy of get_srcrev with only two lines changed. So to create something
>> more Pythonic than a boolean argument and conditionals around the calls to the
>> fetcher's sortable_revision, I just made the method to be called on the fetcher
>> an argument to the method. Defaulting to 'sortable_revision' prevents affecting
>> existing code.
>>
>> Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
>> set the following in a recipe:
>>    PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
>> and this would yield the same result as gitpkgv's GITPKGV variable.
>>
>> See for the discussion leading to this change:
>> [1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
>>
>> Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
>
> I do enjoyed all the design but the way to set the PKGV.
>
> Maybe we could have a way to do it more transparently? Any idea how?

Just setting GITPKGV="${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}" 
globally (bitbake.conf for example) would basically mimic the gitpkgv syntax, 
then a recipe would just say:

PKGV="1.2+${GITPKGV}"

Without the need to inherit gitpkgv.

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at Embedded World 2015 Nuernberg, 24.02.2015 till 26.02.2015, Hall 1, stand number 136.
https://www.embedded-world.de/de/ausstellerprodukte/?focus=edb3exhibitor&focus2=14017667&focus3=embwld15&highlight=topic

Re: [PATCH 1/2] fetch2/__init__.py: Make get_srcrev output configurable

[Otavio Salvador]

On Tue, Feb 3, 2015 at 4:17 AM, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> On 02-02-15 21:50, Otavio Salvador wrote:
>>
>> On Sat, Jan 31, 2015 at 12:06 PM, Mike Looijmans
>> <mike.looijmans@topic.nl> wrote:
>>>
>>> From: Mike Looijmans <milo-software@users.sourceforge.net>
>>>
>>> The idea here is to support package version numbering similar to gitpkgv
>>> in
>>> meta-openembedded. This commit is the first step towards such
>>> functionality.
>>>
>>> The original plan was to add a "get_pretty_srcrev" method to the fetcher,
>>> as
>>> per Richard's suggestion [1]. While writing this, I noticed that it would
>>> become a copy of get_srcrev with only two lines changed. So to create
>>> something
>>> more Pythonic than a boolean argument and conditionals around the calls
>>> to the
>>> fetcher's sortable_revision, I just made the method to be called on the
>>> fetcher
>>> an argument to the method. Defaulting to 'sortable_revision' prevents
>>> affecting
>>> existing code.
>>>
>>> Now if the git fetcher were to implement, say 'gitpkgv_revision' one
>>> could
>>> set the following in a recipe:
>>>    PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
>>> and this would yield the same result as gitpkgv's GITPKGV variable.
>>>
>>> See for the discussion leading to this change:
>>> [1]
>>> http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
>>>
>>> Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
>>
>>
>> I do enjoyed all the design but the way to set the PKGV.
>>
>> Maybe we could have a way to do it more transparently? Any idea how?
>
>
> Just setting GITPKGV="${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
> globally (bitbake.conf for example) would basically mimic the gitpkgv
> syntax, then a recipe would just say:
>
> PKGV="1.2+${GITPKGV}"
>
> Without the need to inherit gitpkgv.

Awesome! Maybe you could add it on a v2?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

[PATCH 2/2] fetch2/git.py: Add gitpkgv_revision alternative version information

[Mike Looijmans]

From: Mike Looijmans <milo-software@users.sourceforge.net>

gitpkgv_revision returns a sortable revision number that can be used
in the PKGV variable for example. To mimic meta-openembedded gitpkgv
behaviour to provide a sortable revision numner, one could set the
following:

PKGV = "1.0+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"

This would yield a package version like "1.0+69+fb5eb80".

Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
---
 lib/bb/fetch2/git.py |   25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 44fc271..3216ef4 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -394,6 +394,31 @@ class Git(FetchMethod):
     def _build_revision(self, ud, d, name):
         return ud.revisions[name]
 
+    def gitpkgv_revision(self, ud, d, name):
+        """
+        Return a sortable revision number by counting commits in the history
+        Based on gitpkgv.bblass in meta-openembedded
+        """
+        rev = self._build_revision(ud, d, name)
+        localpath = ud.localpath
+        rev_file = os.path.join(localpath, "oe-gitpkgv_" + rev)
+        if not os.path.exists(localpath):
+            commits = None
+        else:
+            if not os.path.exists(rev_file) or not os.path.getsize(rev_file):
+                from pipes import quote
+                commits = bb.fetch2.runfetchcmd(
+                        "git rev-list %s -- | wc -l" % (quote(rev)),
+                        d, quiet=True).strip().lstrip('0')
+                if commits:
+                    open(rev_file, "w").write("%d\n" % int(commits))
+            else:
+                commits = open(rev_file, "r").readline(128).strip()
+        if commits:
+            return False, "%s+%s" % (commits, rev[:7])
+        else:
+            return True, str(rev)
+
     def checkstatus(self, ud, d):
         fetchcmd = "%s ls-remote %s" % (ud.basecmd, ud.url)
         try:
-- 
1.7.9.5

Re: [PATCH 0/2] RFC: Integrating gitpkgv functionality

[Mike Looijmans]

On 31-01-15 15:06, Mike Looijmans wrote:
> Follow-up on the discussion here:
> http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
>
> This is a preliminary implementation of gitpkgv in OE-core. It modifies the
> fetcher to optionally provide an incremental numbering scheme for GIT
> repositories.
>
> These two patches implement this. The change in __init__ is basically ready for
> public consumption. The modification to git.py is a prototype that I've used to
> test feasibility. It imitates gitpkgv perfectly for the recipes I tried.
>
> To make this ready for production, we should provide better names and likely
> a GITPKGV or similarly named variable that can be used in recipes instead of
> "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}".
>
> Mike Looijmans (2):
>    fetch2/__init__.py: Make get_srcrev output configurable
>    fetch2/git.py: Add gitpkgv_revision alternative version information
>
>   lib/bb/fetch2/__init__.py |    9 ++++++---
>   lib/bb/fetch2/git.py      |   25 +++++++++++++++++++++++++
>   2 files changed, 31 insertions(+), 3 deletions(-)
>


-- 
Mike Looijmans