Hash based SCMs and OpenEmbedded/Bitbake

Hash based SCMs and OpenEmbedded/Bitbake

[Holger Freyther]

[-- Attachment #1: Type: text/plain, Size: 2344 bytes --]

Hey guys,

we have a couple of issues with hash based SCMs (like git) and how we handle 
them.

1.) sane-srcrev. During stabilisation you want to lock down a rev, so you set 
one in sane-srcrev.

Issue: The PV says something like PV = "1.0.0+gitr${SRCREV}". Now the SRCREV 
is fixed and will not go through the python hook. So the resulting package 
will be PV = "1.0.0+gitrdeadbeef" and if you want to upgrade to the 
ref "affe" you will have a package called "1.0.0+gitraffe" which is not 
upgradable.

Approach at Openmoko: Invent "SRCPV" and put that into the PV line. SRCPV will 
go throuh the bitbake fetcher hook and will generate a string which includes 
an increasing number (like on the autorev's).

This will allow people to set a SRCREV and people will be able to upgrade. 
Afterward you can remove PR from the PV line to allow such things...

This involves a change to conf/bitbake.conf and a fix to the various bb files 
to use SRCPV (which was also used to fix the PV line to say +SCMr..)


2.) The monotonely increasing number is completely random and local. So if two 
people generate packages on two different systems the one having built more 
often (more revs) will have the upgradable package.

Approach at Openmoko: We assume that upstream has a fast forward history (no 
rev is suddenly vanishing) and we will use git-rev-list to count the revs 
(from henryk) to get a monotonic revision number. This means suddenly the 
monotonic number is global. This patch is optional and needs to be enabled 
with a config key.

Issues:
	- Assume history is fast forward, if not packages are not upgraded
	- This forces to have a local git tree (making distributing git-checkouts 
impossible) of everything bitbake will look at during parse.

Werner Almesberger's proposal:
	- For the decision if we should rebuild or not any id is good enough. So we 
could ping/check upstream, check if the last rev was different and then 
schedule it for rebuild (like we do now)
	- When packaging we will use the right revision and put that into the 
package.

Issues:
	+ Only git repos of what we need! Faster parse time
	- tmp/work/foo-PV-PR and tmp/deploy/glibc/ipk/... will not relate..

 

comments, I would be happy to merge some of those bits from Openmoko to 
Openembedded.


	z.

[-- Attachment #2: 0002-Allow-to-conditionally-implement-sortable-revision-i.patch --]
[-- Type: text/x-diff, Size: 1153 bytes --]

From ddc21607f0ac083fcd240e3dc7a186e5e294b2dd Mon Sep 17 00:00:00 2001
From: Holger Freyther <zecke@openmoko.org>
Date: Fri, 16 May 2008 12:12:54 +0200
Subject: [PATCH] Allow to conditionally implement sortable revision in the fetcher

---
 lib/bb/fetch/__init__.py |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index c3bea44..4a0d091 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -510,8 +510,14 @@ class Fetch(object):
         """
         
         """
-        if hasattr(self, "_sortable_revision"):
+        has_want_sortable = hasattr(self, "_want_sortable_revision")
+        has_sortable = hasattr(self, "_sortable_revision")
+
+        if not has_want_sortable and has_sortable:
+            return self._sortable_revision(url, ud, d)
+        elif has_want_sortable and self._want_sortable_revision(url, ud, d) and has_sortable:
             return self._sortable_revision(url, ud, d)
+        
 
         pd = persist_data.PersistData(d)
         key = self._revision_key(url, ud, d)
-- 
1.5.4.3


[-- Attachment #3: 0005-git-Optionally-use-git-rev-list-to-get-a-sortable.patch --]
[-- Type: text/x-diff, Size: 2274 bytes --]

From 6edcc2fa065194fd29fdeb340d15d9cc4b412a0e Mon Sep 17 00:00:00 2001
From: Holger Freyther <zecke@openmoko.org>
Date: Fri, 16 May 2008 12:25:27 +0200
Subject: [PATCH] [git] Optionally use git-rev-list to get a sortable revision
     With setting BB_GIT_CLONE_FOR_SRCREV="1" you can get a sensible
     and global (per repository with only fast forwards) revision. The downsides
     are you will have to have a repository at parse time which means you will
     git-clone certain trees you don't even use. This is also the reason why
     this is optional. This also means you might need to download your git
     checkouts to get this feature working.

---
 lib/bb/fetch/git.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index 3de3529..b256fd0 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -145,3 +145,38 @@ class Git(Fetch):
     def _build_revision(self, url, ud, d):
         return ud.tag
 
+    def _want_sortable_revision(self, url, ud, d):
+        return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
+
+    def _sortable_revision(self, url, ud, d):
+        """
+        This is only called when _want_sortable_revision called true
+
+        We will have to get the updated revision.
+        """
+        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
+        repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+
+
+        # Runtime warning on wrongly configured sources
+        if ud.tag == "1":
+            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
+            return "0+1"
+
+        cwd = os.getcwd()
+
+        # Check if we have the rev already
+        if not os.path.exists(repodir):
+            print "no repo"
+            self.go(None, ud, d)
+
+        os.chdir(repodir)
+        if not self._contains_ref(ud.tag, d):
+            self.go(None, ud, d)
+
+        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
+        os.chdir(cwd)
+
+        return "%s+%s" % (output.split()[0], ud.tag)
+        
+
-- 
1.5.4.3


[-- Attachment #4: 0007-git-Do-not-run-git-rev-list-everytime-to-increase.patch --]
[-- Type: text/x-diff, Size: 1431 bytes --]

From 16740d951cd253a0f1297487b842ac97ef9322cd Mon Sep 17 00:00:00 2001
From: Holger Freyther <zecke@openmoko.org>
Date: Mon, 26 May 2008 16:05:14 +0200
Subject: [PATCH] [git] Do not run git-rev-list everytime to increase the speed
     Cache the result of git-rev-list for a repo and hash. This
     speeds up do_package of the linux kernel tremendously.

---
 lib/bb/fetch/git.py |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index b256fd0..774bd1a 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -157,6 +157,10 @@ class Git(Fetch):
         gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
         repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
 
+        key = "GIT_CACHED_REVISION-%s-%s"  % (gitsrcname, ud.tag)
+        if bb.data.getVar(key, d):
+            return bb.data.getVar(key, d)
+
 
         # Runtime warning on wrongly configured sources
         if ud.tag == "1":
@@ -177,6 +181,8 @@ class Git(Fetch):
         output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
         os.chdir(cwd)
 
-        return "%s+%s" % (output.split()[0], ud.tag)
+        sortable_revision = "%s+%s" % (output.split()[0], ud.tag)
+        bb.data.setVar(key, sortable_revision, d)
+        return sortable_revision
         
 
-- 
1.5.4.3

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Koen Kooi]

Holger Freyther wrote:
> Hey guys,
>
> we have a couple of issues with hash based SCMs (like git) and how we handle
> them.
>
> 1.) sane-srcrev. During stabilisation you want to lock down a rev, so you set
> one in sane-srcrev.
>
> Issue: The PV says something like PV = "1.0.0+gitr${SRCREV}". Now the SRCREV
> is fixed and will not go through the python hook. So the resulting package
> will be PV = "1.0.0+gitrdeadbeef" and if you want to upgrade to the
> ref "affe" you will have a package called "1.0.0+gitraffe" which is not
> upgradable.

The trick I have been lately:

PV = "0.0+${PR}+gitr${SRCREV}"

You increment PR everytime you bump SRCREV. Ugly, but works quite well :)

regards,

Koen

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Holger Freyther]

On Wednesday 03 September 2008 19:39:39 Koen Kooi wrote:

> > Issue: The PV says something like PV = "1.0.0+gitr${SRCREV}". Now the
> > SRCREV is fixed and will not go through the python hook. So the resulting
> > package will be PV = "1.0.0+gitrdeadbeef" and if you want to upgrade to
> > the ref "affe" you will have a package called "1.0.0+gitraffe" which is
> > not upgradable.
>
> The trick I have been lately:
>
> PV = "0.0+${PR}+gitr${SRCREV}"
>
> You increment PR everytime you bump SRCREV. Ugly, but works quite well :)

I have seen that in the neuros recipes and removed that... I don't want to say 
what I thought when I saw this the first time.... :)

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Koen Kooi]

Holger Freyther wrote:
> On Wednesday 03 September 2008 19:39:39 Koen Kooi wrote:
>
>>> Issue: The PV says something like PV = "1.0.0+gitr${SRCREV}". Now the
>>> SRCREV is fixed and will not go through the python hook. So the resulting
>>> package will be PV = "1.0.0+gitrdeadbeef" and if you want to upgrade to
>>> the ref "affe" you will have a package called "1.0.0+gitraffe" which is
>>> not upgradable.
>> The trick I have been lately:
>>
>> PV = "0.0+${PR}+gitr${SRCREV}"
>>
>> You increment PR everytime you bump SRCREV. Ugly, but works quite well :)
>
> I have seen that in the neuros recipes and removed that... I don't want to say
> what I thought when I saw this the first time.... :)

Please don't remove it, it screws up the versioning ('r' > 'g')

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Phil Blundell]

On Wed, 2008-09-03 at 15:54 +0200, Holger Freyther wrote:
> we have a couple of issues with hash based SCMs (like git) and how we handle 
> them.

It's kind of tempting to say that you should just lean on your upstream
to make proper releases and use them, rather than trying to pull some
random bag of bits out of the SCM repository.  However, I guess this was
not the sort of advice you were looking for. :-}

Perhaps more constructively, my preference would be to use "git log" or
some such to extract the datestamp of the revision in question, and then
glue it into PV to make a version string that was sortable and hence
upgradeable.  So, if you had git revision
5a5346dfbec459c10d9e61e16e125b83975a1ef1 or whatever, you'd end up with
PV = "1.0.0+git20080904T123500
+5a5346dfbec459c10d9e61e16e125b83975a1ef1".  Obviously you still have
the possibility of losing if you want to upgrade from one revision to
another one committed in the same second, but I imagine this does not
arise often in practice.

If you don't like that idea then I would suggest the simpler approach of
a manually-maintained version number that you simply prepend to the SCM
revision identifier.  When you change from one SCM revision to another,
you would be responsible for incrementing this number.  This doesn't
seem like an unreasonable thing to require when using a SCM that doesn't
provide its own global versioning, and it's probably the only way of
making multiple branches work out right.

p.

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Holger Freyther]

On Thursday 04 September 2008 11:54:56 Koen Kooi wrote:

> > I have seen that in the neuros recipes and removed that... I don't want
> > to say what I thought when I saw this the first time.... :)
>
> Please don't remove it, it screws up the versioning ('r' > 'g')

I bumped the PE. It is something I would like to see merged to OE. ;)

z.

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Holger Freyther]

On Thursday 04 September 2008 13:43:23 Phil Blundell wrote:
> On Wed, 2008-09-03 at 15:54 +0200, Holger Freyther wrote:
> > we have a couple of issues with hash based SCMs (like git) and how we
> > handle them.
>
> It's kind of tempting to say that you should just lean on your upstream
> to make proper releases and use them, rather than trying to pull some
> random bag of bits out of the SCM repository.  However, I guess this was
> not the sort of advice you were looking for. :-}
>
> Perhaps more constructively, my preference would be to use "git log" or
> some such to extract the datestamp of the revision in question, and then
> glue it into PV to make a version string that was sortable and hence
> upgradeable.  So, if you had git revision
> 5a5346dfbec459c10d9e61e16e125b83975a1ef1 or whatever, you'd end up with
> PV = "1.0.0+git20080904T123500
> +5a5346dfbec459c10d9e61e16e125b83975a1ef1".  Obviously you still have
> the possibility of losing if you want to upgrade from one revision to
> another one committed in the same second, but I imagine this does not
> arise often in practice.

What is happening is you cherry-pick from another branch and then you have an 
older revision (by commit date) on top and would be able to upgrade. Nice 
idea but I think it will not work. There is the author/comitter separation 
but AFAIK no for the commit date.


z.

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Holger Freyther]

On Wednesday 03 September 2008 15:54:52 Holger Freyther wrote:
> Hey guys,

okay, I merged these patches now.

z.

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Richard Purdie]

On Mon, 2008-10-13 at 00:34 +0200, Holger Freyther wrote:
> On Wednesday 03 September 2008 15:54:52 Holger Freyther wrote:
> > Hey guys,
> 
> okay, I merged these patches now.

Hmm, that will teach me to take my eye off the mailing lists. So we do
git checkouts now when parsing?

Richard

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Holger Freyther]

On Monday 13 October 2008 01:30:06 Richard Purdie wrote:
> On Mon, 2008-10-13 at 00:34 +0200, Holger Freyther wrote:
> > On Wednesday 03 September 2008 15:54:52 Holger Freyther wrote:
> > > Hey guys,
> >
> > okay, I merged these patches now.
>
> Hmm, that will teach me to take my eye off the mailing lists. So we do
> git checkouts now when parsing?

No, nothing changes. If you want to use git-rev-list (e.g. I want that for the 
openmoko distro) you put BB_GIT_CLONE_FOR_SRCREV=1 into your local.conf.

I think this is a sane default, the next part will be implementing the idea 
from werner.

z.

Re: Hash based SCMs and OpenEmbedded/Bitbake

[Koen Kooi]

On 13-10-2008 18:52, Holger Freyther wrote:
> On Monday 13 October 2008 01:30:06 Richard Purdie wrote:
>> On Mon, 2008-10-13 at 00:34 +0200, Holger Freyther wrote:
>>> On Wednesday 03 September 2008 15:54:52 Holger Freyther wrote:
>>>> Hey guys,
>>> okay, I merged these patches now.
>> Hmm, that will teach me to take my eye off the mailing lists. So we do
>> git checkouts now when parsing?
>
> No, nothing changes. If you want to use git-rev-list (e.g. I want that for the
> openmoko distro) you put BB_GIT_CLONE_FOR_SRCREV=1 into your local.conf.
>
> I think this is a sane default, the next part will be implementing the idea
> from werner.

But we can't actually use all this, since .dev can only depend on 
released versions of bitbake, right?

regards,

Koen