Re: [RFH] Support for automatic package versining using git describe

Re: [RFH] Support for automatic package versining using git describe

[Mario Domenech Goulart]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

Hi

On Fri, 13 Feb 2009 00:32:21 -0200 Otavio Salvador <otavio@ossystems.com.br> wrote:

> One of nice features we've been using is the git-describe to get
> versions based on tags; many projects are using it to name the
> development releases and it does give a better view of how diverged it
> from last released version.
>
> So we're looking for a way to use it to generate the PV based on
> that. The current git fetcher already has the required support to get
> tags, so we can use it as SRCREV, but not yet to use git-describe.

Attached is an attempt to implement this idea.  One could use
bb.fetch.describe_revision for, say, AUTOPV generation, like
bb.fetch.get_srcrev is used for AUTOREV.

Example (from bitbake.conf):

  AUTOREV = "${@bb.fetch.get_srcrev(d)}"
  AUTOPV = "${@bb.fetch.describe_revision(d)}"

It would be nice if somebody could review the attached patch, since I
don't know much about bitbake internals.

P.S.: sorry if this message is messing up threading in your email client.
I'm replying to a fake original message, since I don't have the original
one.

Best wishes.
Mario


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: describe-revision.diff --]
[-- Type: text/x-diff, Size: 4817 bytes --]

Index: lib/bb/fetch/__init__.py
===================================================================
--- lib/bb/fetch/__init__.py	(revision 1152)
+++ lib/bb/fetch/__init__.py	(working copy)
@@ -196,6 +196,20 @@
 
 srcrev_internal_call = False
 
+def get_scms(d, urldata, rev_support_only = True):
+    scms = []
+    for u in urldata:
+        ud = urldata[u]
+        if rev_support_only or ud.method.suppports_srcrev():
+            if not ud.setup:
+                ud.setup_localpath(d)
+            scms.append(u)
+
+    if len(scms) == 0:
+        bb.msg.error(bb.msg.domain.Fetcher, "SRCREV was used yet no valid SCM was found in SRC_URI")
+
+    return scms
+
 def get_srcrev(d):
     """
     Return the version string for the current package
@@ -217,21 +231,9 @@
     if bb.fetch.srcrev_internal_call:
         return "SRCREVINACTION"
 
-    scms = []
-
-    # Only call setup_localpath on URIs which suppports_srcrev() 
     urldata = init(bb.data.getVar('SRC_URI', d, 1).split(), d, False)
-    for u in urldata:
-        ud = urldata[u]
-        if ud.method.suppports_srcrev():
-            if not ud.setup:
-                ud.setup_localpath(d)
-            scms.append(u)
+    scms = get_scms(d, urldata)
 
-    if len(scms) == 0:
-        bb.msg.error(bb.msg.domain.Fetcher, "SRCREV was used yet no valid SCM was found in SRC_URI")
-        raise ParameterError
-
     bb.data.setVar('__BB_DONT_CACHE','1', d)
 
     if len(scms) == 1:
@@ -253,6 +255,21 @@
 
     return format
 
+def describe_revision(d):
+    urldata = init(bb.data.getVar('SRC_URI', d, 1).split(), d, False)
+    bb.data.setVar('__BB_DONT_CACHE','1', d)
+    scms = get_scms(d, urldata)
+    scm = ''
+    if len(scms) == 1:
+        scm = scms[0]
+    else:
+        bb.msg.error(bb.msg.domain.Fetcher, "describe_revision does not support multiple SCMs.")
+
+    try: 
+        return urldata[scm].method.describe_revision(scm, urldata[scm], d)
+    except NoMethodError:
+        return "NO_REV_DESCR"
+
 def localpath(url, d, cache = True):
     """
     Called from the parser with cache=False since the cache isn't ready 
Index: lib/bb/fetch/git.py
===================================================================
--- lib/bb/fetch/git.py	(revision 1152)
+++ lib/bb/fetch/git.py	(working copy)
@@ -42,11 +42,12 @@
             ud.proto = ud.parm['protocol']
 
         ud.branch = ud.parm.get("branch", "master")
+        ud.tag = None
 
         tag = Fetch.srcrev_internal_helper(ud, d)
         if tag is True:
             ud.tag = self.latest_revision(url, ud, d)	
-        elif tag:
+        elif tag and tag != "1":
             ud.tag = tag
 
         if not ud.tag or ud.tag == "master":
@@ -141,6 +142,16 @@
     def _want_sortable_revision(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
+    def _chrepo(self, repodir, ud, d):
+        # Check if we have the rev already
+        if not os.path.exists(repodir):
+            print "no repo"
+            self.go(None, ud, d)
+
+        os.chdir(repodir)
+        if not self._contains_ref(ud.tag, d):
+            self.go(None, ud, d)
+
     def _sortable_revision(self, url, ud, d):
         """
         This is only called when _want_sortable_revision called true
@@ -162,15 +173,8 @@
 
         cwd = os.getcwd()
 
-        # Check if we have the rev already
-        if not os.path.exists(repodir):
-            print "no repo"
-            self.go(None, ud, d)
+        self._chrepo(repodir, ud, d)
 
-        os.chdir(repodir)
-        if not self._contains_ref(ud.tag, d):
-            self.go(None, ud, d)
-
         output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
         os.chdir(cwd)
 
@@ -179,3 +183,27 @@
         return sortable_revision
         
 
+    def describe_revision(self, url, ud, d):
+        """ 
+        Return the output of 'git describe' or 'NO_REV_DESCR' in
+        case no tag is found.
+        """
+        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
+        repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+
+        # Runtime warning on wrongly configured sources
+        if ud.tag == "1":
+            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
+            return "0+1"
+
+        cwd = os.getcwd()
+
+        self._chrepo(repodir, ud, d)
+
+        rev_descr = ''
+        try:
+            rev_descr = runfetchcmd("git describe 2>/dev/null", d, quiet=True).strip()
+        except FetchError:
+            rev_descr = "NO_REV_DESCR"
+        os.chdir(cwd)
+        return rev_descr

Re: [RFH] Support for automatic package versining using git describe

[Otavio Salvador]

Mario Domenech Goulart <mario@ossystems.com.br> writes:


[...]

> Index: lib/bb/fetch/git.py
> ===================================================================
> --- lib/bb/fetch/git.py	(revision 1152)
> +++ lib/bb/fetch/git.py	(working copy)
> @@ -42,11 +42,12 @@
>              ud.proto = ud.parm['protocol']
>  
>          ud.branch = ud.parm.get("branch", "master")
> +        ud.tag = None
>  
>          tag = Fetch.srcrev_internal_helper(ud, d)
>          if tag is True:
>              ud.tag = self.latest_revision(url, ud, d)	
> -        elif tag:
> +        elif tag and tag != "1":
>              ud.tag = tag
>  
>          if not ud.tag or ud.tag == "master":

[...]

Please drop this hook since it is our internal workaround for the lack
of the AUTOREV in our recipes.

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: [RFH] Support for automatic package versining using git describe

[Mario Domenech Goulart]

[-- Attachment #1: Type: text/plain, Size: 983 bytes --]

On Thu, 05 Mar 2009 10:43:04 -0300 Otavio Salvador <otavio@ossystems.com.br> wrote:

> Mario Domenech Goulart <mario@ossystems.com.br> writes:
>
>> Index: lib/bb/fetch/git.py
>> ===================================================================
>> --- lib/bb/fetch/git.py	(revision 1152)
>> +++ lib/bb/fetch/git.py	(working copy)
>> @@ -42,11 +42,12 @@
>>              ud.proto = ud.parm['protocol']
>>  
>>          ud.branch = ud.parm.get("branch", "master")
>> +        ud.tag = None
>>  
>>          tag = Fetch.srcrev_internal_helper(ud, d)
>>          if tag is True:
>>              ud.tag = self.latest_revision(url, ud, d)	
>> -        elif tag:
>> +        elif tag and tag != "1":
>>              ud.tag = tag
>>  
>>          if not ud.tag or ud.tag == "master":
>
> [...]
>
> Please drop this hook since it is our internal workaround for the lack
> of the AUTOREV in our recipes.

Sure.  Attached is the new patch.

Best wishes.
Mario


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: describe-revision2.diff --]
[-- Type: text/x-diff, Size: 4384 bytes --]

Index: lib/bb/fetch/__init__.py
===================================================================
--- lib/bb/fetch/__init__.py	(revision 1152)
+++ lib/bb/fetch/__init__.py	(working copy)
@@ -196,6 +196,20 @@
 
 srcrev_internal_call = False
 
+def get_scms(d, urldata, rev_support_only = True):
+    scms = []
+    for u in urldata:
+        ud = urldata[u]
+        if rev_support_only or ud.method.suppports_srcrev():
+            if not ud.setup:
+                ud.setup_localpath(d)
+            scms.append(u)
+
+    if len(scms) == 0:
+        bb.msg.error(bb.msg.domain.Fetcher, "SRCREV was used yet no valid SCM was found in SRC_URI")
+
+    return scms
+
 def get_srcrev(d):
     """
     Return the version string for the current package
@@ -217,21 +231,9 @@
     if bb.fetch.srcrev_internal_call:
         return "SRCREVINACTION"
 
-    scms = []
-
-    # Only call setup_localpath on URIs which suppports_srcrev() 
     urldata = init(bb.data.getVar('SRC_URI', d, 1).split(), d, False)
-    for u in urldata:
-        ud = urldata[u]
-        if ud.method.suppports_srcrev():
-            if not ud.setup:
-                ud.setup_localpath(d)
-            scms.append(u)
+    scms = get_scms(d, urldata)
 
-    if len(scms) == 0:
-        bb.msg.error(bb.msg.domain.Fetcher, "SRCREV was used yet no valid SCM was found in SRC_URI")
-        raise ParameterError
-
     bb.data.setVar('__BB_DONT_CACHE','1', d)
 
     if len(scms) == 1:
@@ -253,6 +255,25 @@
 
     return format
 
+def describe_revision(d):
+    """ 
+    Return the output of 'git describe' or 'NO_REV_DESCR' in case no
+    tag is found.
+    """
+    urldata = init(bb.data.getVar('SRC_URI', d, 1).split(), d, False)
+    bb.data.setVar('__BB_DONT_CACHE','1', d)
+    scms = get_scms(d, urldata)
+    scm = ''
+    if len(scms) == 1:
+        scm = scms[0]
+    else:
+        bb.msg.error(bb.msg.domain.Fetcher, "describe_revision does not support multiple SCMs.")
+
+    try: 
+        return urldata[scm].method.describe_revision(scm, urldata[scm], d)
+    except NoMethodError:
+        return "NO_REV_DESCR"
+
 def localpath(url, d, cache = True):
     """
     Called from the parser with cache=False since the cache isn't ready 
Index: lib/bb/fetch/git.py
===================================================================
--- lib/bb/fetch/git.py	(revision 1152)
+++ lib/bb/fetch/git.py	(working copy)
@@ -141,6 +141,16 @@
     def _want_sortable_revision(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
+    def _chrepo(self, repodir, ud, d):
+        # Check if we have the rev already
+        if not os.path.exists(repodir):
+            print "no repo"
+            self.go(None, ud, d)
+
+        os.chdir(repodir)
+        if not self._contains_ref(ud.tag, d):
+            self.go(None, ud, d)
+
     def _sortable_revision(self, url, ud, d):
         """
         This is only called when _want_sortable_revision called true
@@ -162,15 +172,8 @@
 
         cwd = os.getcwd()
 
-        # Check if we have the rev already
-        if not os.path.exists(repodir):
-            print "no repo"
-            self.go(None, ud, d)
+        self._chrepo(repodir, ud, d)
 
-        os.chdir(repodir)
-        if not self._contains_ref(ud.tag, d):
-            self.go(None, ud, d)
-
         output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
         os.chdir(cwd)
 
@@ -179,3 +182,23 @@
         return sortable_revision
         
 
+    def describe_revision(self, url, ud, d):
+        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
+        repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+
+        # Runtime warning on wrongly configured sources
+        if ud.tag == "1":
+            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
+            return "0+1"
+
+        cwd = os.getcwd()
+
+        self._chrepo(repodir, ud, d)
+
+        rev_descr = ''
+        try:
+            rev_descr = runfetchcmd("git describe 2>/dev/null", d, quiet=True).strip()
+        except FetchError:
+            rev_descr = "NO_REV_DESCR"
+        os.chdir(cwd)
+        return rev_descr

Re: [RFH] Support for automatic package versining using git describe

[Koen Kooi]

Sorry for top-posting, but I can't find a good point to hook into.

I have a few general questions:

1) what happens when DLDIR and TMPDIR are empty (e.g. first build)
2) does it need a netconnections at any point?
3) do you have any examples how output looks before and after your patch?

I'd also like to hear RPs and zeckes opinion in this

regards,

Koen

On 05-03-09 22:31, Mario Domenech Goulart wrote:
> On Thu, 05 Mar 2009 10:43:04 -0300 Otavio Salvador<otavio@ossystems.com.br>  wrote:
>
>> Mario Domenech Goulart<mario@ossystems.com.br>  writes:
>>
>>> Index: lib/bb/fetch/git.py
>>> ===================================================================
>>> --- lib/bb/fetch/git.py	(revision 1152)
>>> +++ lib/bb/fetch/git.py	(working copy)
>>> @@ -42,11 +42,12 @@
>>>               ud.proto = ud.parm['protocol']
>>>
>>>           ud.branch = ud.parm.get("branch", "master")
>>> +        ud.tag = None
>>>
>>>           tag = Fetch.srcrev_internal_helper(ud, d)
>>>           if tag is True:
>>>               ud.tag = self.latest_revision(url, ud, d)	
>>> -        elif tag:
>>> +        elif tag and tag != "1":
>>>               ud.tag = tag
>>>
>>>           if not ud.tag or ud.tag == "master":
>> [...]
>>
>> Please drop this hook since it is our internal workaround for the lack
>> of the AUTOREV in our recipes.
>
> Sure.  Attached is the new patch.
>
> Best wishes.
> Mario
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: [RFH] Support for automatic package versining using git describe

[Mike (mwester)]

Mario Domenech Goulart wrote:
> On Thu, 05 Mar 2009 10:43:04 -0300 Otavio Salvador <otavio@ossystems.com.br> wrote:
>> Please drop this hook since it is our internal workaround for the lack
>> of the AUTOREV in our recipes.
> 
> Sure.  Attached is the new patch.

I don't know the context for these patches, so please forgive me if I
should be reading the sources instead of asking here (perhaps my
question makes no sense at all):

What does this code do if I make a tarball snapshot of the
"openembedded" directory structure, excluding the .git directory, and
attempt to build from that?

-Mike (mwester)

[RFH] Support for automatic package versining using git describe

[Otavio Salvador]

Hello,

One of nice features we've been using is the git-describe to get
versions based on tags; many projects are using it to name the
development releases and it does give a better view of how diverged it
from last released version.

So we're looking for a way to use it to generate the PV based on
that. The current git fetcher already has the required support to get
tags, so we can use it as SRCREV, but not yet to use git-describe.

I've tried to implement it, and failed. Here is the current patch just
to reference.

Index: lib/bb/fetch/git.py
===================================================================
--- lib/bb/fetch/git.py (revisão 1150)
+++ lib/bb/fetch/git.py (cópia de trabalho)
@@ -49,10 +49,7 @@
         elif tag:
             ud.tag = tag
 
-        if not ud.tag:
-            ud.tag = self.latest_revision(url, ud, d)  
-
-        if ud.tag == "master":
+        if not ud.tag or ud.tag == "master":
             ud.tag = self.latest_revision(url, ud, d)
 
         ud.localfile = data.expand('git_%s%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.tag), d)
@@ -179,6 +176,15 @@
 
         sortable_revision = "%s+%s" % (output.split()[0], ud.tag)
         bb.data.setVar(key, sortable_revision, d)
+
+        if bb.data.getVar("GIT_USE_DESCRIBE", d, False):
+            describe = runfetchcmd("git describe 2> /dev/null").read().strip()
+
+            if describe.empty():
+                bb.data.setVar("GIT_DESCRIBED_VERSION", "0.0.0+gitr%s" % (ud.tag))
+            else:
+                bb.data.setVar("GIT_DESCRIBED_VERSION", describe)
+
         return sortable_revision

Do someone has a better idea how to do that?

TIA,

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: [RFH] Support for automatic package versining using git describe

[Koen Kooi]

On 13-02-09 03:32, Otavio Salvador wrote:
> Hello,
>
> One of nice features we've been using is the git-describe to get
> versions based on tags; many projects are using it to name the
> development releases and it does give a better view of how diverged it
> from last released version.
>
> So we're looking for a way to use it to generate the PV based on
> that. The current git fetcher already has the required support to get
> tags, so we can use it as SRCREV, but not yet to use git-describe.

Do you have any examples of the output?

regards,

Koen

Re: [RFH] Support for automatic package versining using git describe

[Otavio Salvador]

Koen Kooi <k.kooi@student.utwente.nl> writes:

> On 13-02-09 03:32, Otavio Salvador wrote:
>> Hello,
>>
>> One of nice features we've been using is the git-describe to get
>> versions based on tags; many projects are using it to name the
>> development releases and it does give a better view of how diverged it
>> from last released version.
>>
>> So we're looking for a way to use it to generate the PV based on
>> that. The current git fetcher already has the required support to get
>> tags, so we can use it as SRCREV, but not yet to use git-describe.
>
> Do you have any examples of the output?

Sure:

otavio@neumann:~/hacking/ossystems/toscoterm$ git describe 
0.1-1-gd834119

Basically this can be read as:

 - last version was 0.1
 - since that we've done 1 commit
 - the sha1sum of it is d834119 (g is always included before it)

Cheers,

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."