AUTOREV and SRCPV

AUTOREV and SRCPV

[pieterg]

I have only a handful of (git) packages for which I need to use AUTOREV, but 
when I define

BB_GIT_CLONE_FOR_SRCREV = "1"
BB_LOCALCOUNT_OVERRIDE = ""

the sideeffect is that hundreds of other packages (which I don't need) are 
suddenly having their repositories cloned as well, when their bb files are 
parsed. Even though these packages are not using AUTOREV, they have fixed 
SRCREV's (either in their bb files, or in my distro).

This is obviously taking a long time, consuming a lot of diskspace, 
generating error messages for invalid git urls, and eventually bitbake will 
fail because it encountered 'parsing errors' because of those invalid git 
urls.

I guess this is because all of these packages are using ${SRCPV} in their 
PV, instead of ${SRCREV} (probably in order to find the correct LOCALCOUNT 
for the gitrev).

But argueing that people who don't use BB_GIT_CLONE_FOR_SRCREV will not get 
these LOCALCOUNT's in their PV anyway, and people that do use 
BB_GIT_CLONE_FOR_SRCREV will use SRCREV = ${AUTOREV} for the packages which 
they want to be automatically updated, so SRCREV would equal SRCPV, 
wouldn't it make sense to just use ${SRCREV} instead of ${SRCPV} in PV?

Rgds, Pieter

Re: AUTOREV and SRCPV

[Martin Jansa]

On Wed, Jun 02, 2010 at 11:51:37PM +0200, pieterg wrote:
> I have only a handful of (git) packages for which I need to use AUTOREV, but 
> when I define
> 
> BB_GIT_CLONE_FOR_SRCREV = "1"
> BB_LOCALCOUNT_OVERRIDE = ""
> 
> the sideeffect is that hundreds of other packages (which I don't need) are 
> suddenly having their repositories cloned as well, when their bb files are 
> parsed. Even though these packages are not using AUTOREV, they have fixed 
> SRCREV's (either in their bb files, or in my distro).
> 
> This is obviously taking a long time, consuming a lot of diskspace, 
> generating error messages for invalid git urls, and eventually bitbake will 
> fail because it encountered 'parsing errors' because of those invalid git 
> urls.

If you use bitbake-1.10 it will do it just once for git revision (it
will cache the result of "git list-rev | wc -l" which is used by
BB_GIT_CLONE_FOR_SRCREV.

> I guess this is because all of these packages are using ${SRCPV} in their 
> PV, instead of ${SRCREV} (probably in order to find the correct LOCALCOUNT 
> for the gitrev).
> 
> But argueing that people who don't use BB_GIT_CLONE_FOR_SRCREV will not get 
> these LOCALCOUNT's in their PV anyway, and people that do use 
> BB_GIT_CLONE_FOR_SRCREV will use SRCREV = ${AUTOREV} for the packages which 
> they want to be automatically updated, so SRCREV would equal SRCPV, 
> wouldn't it make sense to just use ${SRCREV} instead of ${SRCPV} in PV?

No they won't be the same. They will have the same format
git${LOCALCOUNT}+${HASH} as when AUTOREV+SRCREV is used.

Check "[oe] SRCPV migration - How SRCPV works!" thread for more info
http://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg01607.html

in short: why do you insist on BB_GIT_CLONE_FOR_SRCREV? autoincremented
LOCALCOUNT works good in most cases (as long as you keep cache dir), only 
case for which we introduced BB_LOCALCOUNT_OVERRIDE is for multiple builders 
sharing feed, but not cache directory.

Regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: AUTOREV and SRCPV

[pieterg]

On Thursday 03 June 2010 06:35:05 Martin Jansa wrote:
> On Wed, Jun 02, 2010 at 11:51:37PM +0200, pieterg wrote:
> > I have only a handful of (git) packages for which I need to use
> > AUTOREV, but when I define
> >
> > BB_GIT_CLONE_FOR_SRCREV = "1"
> > BB_LOCALCOUNT_OVERRIDE = ""
> >
> > the sideeffect is that hundreds of other packages (which I don't need)
> > are suddenly having their repositories cloned as well, when their bb
> > files are parsed. Even though these packages are not using AUTOREV,
> > they have fixed SRCREV's (either in their bb files, or in my distro).
> >
> > This is obviously taking a long time, consuming a lot of diskspace,
> > generating error messages for invalid git urls, and eventually bitbake
> > will fail because it encountered 'parsing errors' because of those
> > invalid git urls.
>
> If you use bitbake-1.10 it will do it just once for git revision (it
> will cache the result of "git list-rev | wc -l" which is used by
> BB_GIT_CLONE_FOR_SRCREV.

Probably, but I don't get that far, bitbake quits at the end, with several 
parsing errors (problems with git repositories which I have no control 
over, and which I don't even want anything to do with)

> > I guess this is because all of these packages are using ${SRCPV} in
> > their PV, instead of ${SRCREV} (probably in order to find the correct
> > LOCALCOUNT for the gitrev).
> >
> > But argueing that people who don't use BB_GIT_CLONE_FOR_SRCREV will not
> > get these LOCALCOUNT's in their PV anyway, and people that do use
> > BB_GIT_CLONE_FOR_SRCREV will use SRCREV = ${AUTOREV} for the packages
> > which they want to be automatically updated, so SRCREV would equal
> > SRCPV, wouldn't it make sense to just use ${SRCREV} instead of ${SRCPV}
> > in PV?
>
> No they won't be the same. They will have the same format
> git${LOCALCOUNT}+${HASH} as when AUTOREV+SRCREV is used.

My point is that I don't think they have to be the same.
There are two sorts of people, those with BB_GIT_CLONE_FOR_SRCREV, and those 
without.
(or in my case, this is controlled by the distro, an 'unstable', 
or 'release').
As long as the versioning is consistent for each group of people (or in my 
case, each distro), I don't need the versioning to stay consistent when 
toggling BB_GIT_CLONE_FOR_SRCREV.

> in short: why do you insist on BB_GIT_CLONE_FOR_SRCREV? autoincremented
> LOCALCOUNT works good in most cases (as long as you keep cache dir), only
> case for which we introduced BB_LOCALCOUNT_OVERRIDE is for multiple
> builders sharing feed, but not cache directory.

And that's indeed what we have, many people are building the same distro, 
and they should end up with the exact same version (or LOCALCOUNT), and I 
don't think there is a way without clearing BB_LOCALCOUNT_OVERRIDE and 
setting BB_GIT_CLONE_FOR_SRCREV.

My point is, before the big SRCPV change, you only suffered for the 
carefully picked number of packages for which you wanted to use AUTOSRC.
But now, you suffer for hundreds of packages, which you might not even want 
to have anything to do with, packages which use fixed SRCREV's, so they 
shouldn't even need dynamically generated PVs.

For now, I've made this change in bitbake.conf, which pretty much reverts 
the behaviour to what it was before, so at least my build will complete
(don't mean to propose this patch of course, just as demonstration how I 
have to work around it at the moment)
Hopefully there is a better way to make AUTOREV work again for our 
situation.

From 30a9c042d65b32dc1d76a757f493659b556c6a62 Mon Sep 17 00:00:00 2001
From: pieterg <pieterg@users.sourceforge.net>
Date: Thu, 3 Jun 2010 00:23:51 +0200
Subject: [PATCH] bitbake.conf: work around AUTOREV problems

As long as many packages are using SRCPV in their PV,
we cannot use the current AUTOREV / SRCPV strategy
---
 conf/bitbake.conf |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index 12a5522..1d2c123 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -576,8 +576,8 @@ UPDATECOMMAND_cvs = "/usr/bin/env 'PATH=${PATH}' 
cvs -d${CVSROOT} update -d -P $
 UPDATECOMMAND_svn = "/usr/bin/env svn update ${SVNCOOPTS}"
 SRCDATE = "${DATE}"
 SRCREV = "1"
-SRCPV = "${@bb.fetch.get_srcrev(d)}"
-AUTOREV = "${SRCPV}"
+SRCPV = "${SRCREV}"
+AUTOREV = "${@bb.fetch.get_srcrev(d)}"
 
 # For now disable autoincrement of revision counter in SRCPV, whoever wants 
it, should enable it in local.conf or distro config
 # Revision counter is incremented only locally (bad for multiple builders 
filling shared feeds), LOCALCOUNT can be used to maintain
-- 
1.6.5.rc1.44.ga1675

Re: AUTOREV and SRCPV

[Martin Jansa]

On Thu, Jun 03, 2010 at 09:15:56AM +0200, pieterg wrote:
> On Thursday 03 June 2010 06:35:05 Martin Jansa wrote:
> > On Wed, Jun 02, 2010 at 11:51:37PM +0200, pieterg wrote:
> > > I have only a handful of (git) packages for which I need to use
> > > AUTOREV, but when I define
> > >
> > > BB_GIT_CLONE_FOR_SRCREV = "1"
> > > BB_LOCALCOUNT_OVERRIDE = ""
> > >
> > > the sideeffect is that hundreds of other packages (which I don't need)
> > > are suddenly having their repositories cloned as well, when their bb
> > > files are parsed. Even though these packages are not using AUTOREV,
> > > they have fixed SRCREV's (either in their bb files, or in my distro).
> > >
> > > This is obviously taking a long time, consuming a lot of diskspace,
> > > generating error messages for invalid git urls, and eventually bitbake
> > > will fail because it encountered 'parsing errors' because of those
> > > invalid git urls.
> >
> > If you use bitbake-1.10 it will do it just once for git revision (it
> > will cache the result of "git list-rev | wc -l" which is used by
> > BB_GIT_CLONE_FOR_SRCREV.
> 
> Probably, but I don't get that far, bitbake quits at the end, with several 
> parsing errors (problems with git repositories which I have no control 
> over, and which I don't even want anything to do with)

I'm aware of this problem and was discussed in that thread half a year
ago, pity that you didn't join the discussion back then :/. But FYI all
recipes with SRCPV builds OK here (without BB_GIT_CLONE_FOR_SRCREV but
git servers are accessible for me).

> > > I guess this is because all of these packages are using ${SRCPV} in
> > > their PV, instead of ${SRCREV} (probably in order to find the correct
> > > LOCALCOUNT for the gitrev).
> > >
> > > But argueing that people who don't use BB_GIT_CLONE_FOR_SRCREV will not
> > > get these LOCALCOUNT's in their PV anyway, and people that do use
> > > BB_GIT_CLONE_FOR_SRCREV will use SRCREV = ${AUTOREV} for the packages
> > > which they want to be automatically updated, so SRCREV would equal
> > > SRCPV, wouldn't it make sense to just use ${SRCREV} instead of ${SRCPV}
> > > in PV?
> >
> > No they won't be the same. They will have the same format
> > git${LOCALCOUNT}+${HASH} as when AUTOREV+SRCREV is used.
> 
> My point is that I don't think they have to be the same.
> There are two sorts of people, those with BB_GIT_CLONE_FOR_SRCREV, and those 
> without.
> (or in my case, this is controlled by the distro, an 'unstable', 
> or 'release').
> As long as the versioning is consistent for each group of people (or in my 
> case, each distro), I don't need the versioning to stay consistent when 
> toggling BB_GIT_CLONE_FOR_SRCREV.

What about toggling AUTOREV and fixed revision? Which is imho more
common change (ie when development gets slower and there is stable
version in git and you don't need/want it AUTOREV). That's exacly the
case when SRCPV is really handy. 

The other is when you change SRCREV in recipe without updating PV. Then
SRCPV will give you newer PV which will upgrade resulting package on
target.

Regarads,
-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: AUTOREV and SRCPV

[pieterg]

On Thursday 03 June 2010 09:37:17 Martin Jansa wrote:
> On Thu, Jun 03, 2010 at 09:15:56AM +0200, pieterg wrote:
> > On Thursday 03 June 2010 06:35:05 Martin Jansa wrote:

<snip>

> > > If you use bitbake-1.10 it will do it just once for git revision (it
> > > will cache the result of "git list-rev | wc -l" which is used by
> > > BB_GIT_CLONE_FOR_SRCREV.
> >
> > Probably, but I don't get that far, bitbake quits at the end, with
> > several parsing errors (problems with git repositories which I have no
> > control over, and which I don't even want anything to do with)
>
> I'm aware of this problem and was discussed in that thread half a year
> ago, pity that you didn't join the discussion back then :/. But FYI all
> recipes with SRCPV builds OK here (without BB_GIT_CLONE_FOR_SRCREV but
> git servers are accessible for me).

Yes, I know I'm too late now ;)
We were way behind with our oe merges back then, and only recently I started 
to get back uptodate again. So I didn't even realise the impact it would 
have for us untill now.

> > There are two sorts of people, those with BB_GIT_CLONE_FOR_SRCREV, and
> > those without.
> > (or in my case, this is controlled by the distro, an 'unstable',
> > or 'release').
> > As long as the versioning is consistent for each group of people (or in
> > my case, each distro), I don't need the versioning to stay consistent
> > when toggling BB_GIT_CLONE_FOR_SRCREV.
>
> What about toggling AUTOREV and fixed revision? Which is imho more
> common change (ie when development gets slower and there is stable
> version in git and you don't need/want it AUTOREV). That's exacly the
> case when SRCPV is really handy.

But if you then decide to remove BB_GIT_CLONE_FOR_SRCREV again because you 
no longer need it, and somebody starts a clean build, the LOCALCOUNT would 
start at 0 again I guess.
So the versioning isn't consistent anyway when toggling 
BB_GIT_CLONE_FOR_SRCREV.

> The other is when you change SRCREV in recipe without updating PV. Then
> SRCPV will give you newer PV which will upgrade resulting package on
> target.

Yes, but only if you stick to the same buildserver, and as soon as the hdd 
crashes for instance and you have to start with a clean build, everybody is 
in trouble.
So personally, I would never even want to use SRCPV, unless it's for 
AUTOREV.

Would it be an idea to be able to only have BB_GIT_CLONE_FOR_SRCREV for 
those packages which have SRCREV = ${AUTOREV}?

Rgds, Pieter

Re: AUTOREV and SRCPV

[Martin Jansa]

On Thu, Jun 03, 2010 at 10:02:31AM +0200, pieterg wrote:
> On Thursday 03 June 2010 09:37:17 Martin Jansa wrote:
> > On Thu, Jun 03, 2010 at 09:15:56AM +0200, pieterg wrote:
> > > On Thursday 03 June 2010 06:35:05 Martin Jansa wrote:
> 
> <snip>
> 
> But if you then decide to remove BB_GIT_CLONE_FOR_SRCREV again because you 
> no longer need it, and somebody starts a clean build, the LOCALCOUNT would 
> start at 0 again I guess.

SRCPV is storing LOCALCOUNT in the same persistent db as AUTOREV with or without
BB_GIT_CLONE_FOR_SRCREV (at least for bitbake-1.10 and higher). The
BB_GIT_CLONE_FOR_SRCREV caching changed that you won't count revisions
if the requested SRCREV (latest for AUTOREV or specified in recipe) is
the same as was for last parsing and so the same LOCALCOUNT can be taken
from persistent db.

I'm not sure if 1.8 was storing LOCALCOUNT from BB_GIT_CLONE_FOR_SRCREV
but then never used it from db (then switching BB_GIT_CLONE_FOR_SRCREV
off would keep same LOCALCOUNTs) or if it didn't even store it (less
likely - but you can easily check).

Either way if you upgrade bitbake first (while keeping
BB_GIT_CLONE_FOR_SRCREV) and then switch BB_GIT_CLONE_FOR_SRCREV off,
you should have same LOCALCOUNTs


> So the versioning isn't consistent anyway when toggling 
> BB_GIT_CLONE_FOR_SRCREV.
> 
> > The other is when you change SRCREV in recipe without updating PV. Then
> > SRCPV will give you newer PV which will upgrade resulting package on
> > target.
> 
> Yes, but only if you stick to the same buildserver, and as soon as the hdd 
> crashes for instance and you have to start with a clean build, everybody is 
> in trouble.

Yes and that's the same for AUTOREV (without BB_GIT_CLONE_FOR_SRCREV).

> So personally, I would never even want to use SRCPV, unless it's for 
> AUTOREV.

So what you say is that you would also never use AUTOREV without
BB_GIT_CLONE_FOR_SRCREV, no matter if SRCPV is used.

> Would it be an idea to be able to only have BB_GIT_CLONE_FOR_SRCREV for 
> those packages which have SRCREV = ${AUTOREV}?

Well then it will be a bit inconsistent, because those AUTOREV recipes
will restore their LOCALCOUNT (ie after hdd breakage) but notAUTOREV
recipes (which usually could be considered more stable) will "downgrade"
to LOCALCOUNT 0 (if you didn't restore cache).

A bit better would be to disable LOCALCOUNT_OVERRIDE only for recipes
you want to be AUTOREV (then git clone will stop for recipes you don't
care about even with BB_GIT_CLONE_FOR_SRCREV and after removing cache
those LOCALCOUNT will stay 0 as it was before).

Best solution would be to ask git devs to implement "git remote-count
HASH" as remote-ls works.

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: AUTOREV and SRCPV

[pieterg]

On Thursday 03 June 2010 10:27:40 Martin Jansa wrote:
> On Thu, Jun 03, 2010 at 10:02:31AM +0200, pieterg wrote:
> > On Thursday 03 June 2010 09:37:17 Martin Jansa wrote:
> > > On Thu, Jun 03, 2010 at 09:15:56AM +0200, pieterg wrote:

<snip>

> > So personally, I would never even want to use SRCPV, unless it's for
> > AUTOREV.
>
> So what you say is that you would also never use AUTOREV without
> BB_GIT_CLONE_FOR_SRCREV, no matter if SRCPV is used.

Right, same reason. I need reproducable versions at all times, on all build 
systems, even when the cache is lost.

> > Would it be an idea to be able to only have BB_GIT_CLONE_FOR_SRCREV for
> > those packages which have SRCREV = ${AUTOREV}?
>
> Well then it will be a bit inconsistent, because those AUTOREV recipes
> will restore their LOCALCOUNT (ie after hdd breakage) but notAUTOREV
> recipes (which usually could be considered more stable) will "downgrade"
> to LOCALCOUNT 0 (if you didn't restore cache).

True, and a bit selfish of course, because I don't care about all those 
SRCPV packages, i cannot even use them because of their inconsistent 
versions.

> A bit better would be to disable LOCALCOUNT_OVERRIDE only for recipes
> you want to be AUTOREV (then git clone will stop for recipes you don't
> care about even with BB_GIT_CLONE_FOR_SRCREV and after removing cache
> those LOCALCOUNT will stay 0 as it was before).

Yes, that would lead to the same workable situation for me.
And seems like an easier fix, because BB_LOCALCOUNT_OVERRIDE is checked for 
each package, where BB_GIT_CLONE_FOR_SRCREV is only checked once, globally.

> Best solution would be to ask git devs to implement "git remote-count
> HASH" as remote-ls works.

Yes, if we could convince them, that would be ideal...

Rgds, Pieter

Re: AUTOREV and SRCPV

[pieterg]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

On Thursday 03 June 2010 10:27:40 Martin Jansa wrote:
> A bit better would be to disable LOCALCOUNT_OVERRIDE only for recipes
> you want to be AUTOREV (then git clone will stop for recipes you don't
> care about even with BB_GIT_CLONE_FOR_SRCREV and after removing cache
> those LOCALCOUNT will stay 0 as it was before).

Check, this would be a good workaround for me at the moment.
Should we propose this patch? (see attachment, patch against bb 1.10)
Are the bitbake people reading along here?

Rgds, Pieter

[-- Attachment #2: 0001-allow-BB_LOCALCOUNT_OVERRIDE-to-be-defined-per-packa.patch --]
[-- Type: text/x-diff, Size: 1658 bytes --]

From 6265ab521627a38b9b0596149e31119b2073f5ba Mon Sep 17 00:00:00 2001
From: pieterg <pieterg@gmx.com>
Date: Thu, 3 Jun 2010 11:13:42 +0200
Subject: [PATCH] allow BB_LOCALCOUNT_OVERRIDE to be defined per package

---
 lib/bb/fetch/__init__.py |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 085f864..a586fda 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -695,7 +695,8 @@ class Fetch(object):
 
         latest_rev = self._build_revision(url, ud, d)
         last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
-        uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
+        pn = bb.data.getVar("PN", d, True)
+        uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE_pn-%s" % pn, d, True) or bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
         count = None
         if uselocalcount:
             count = Fetch.localcount_internal_helper(ud, d)
@@ -705,9 +706,10 @@ class Fetch(object):
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
-        buildindex_provided = hasattr(self, "_sortable_buildindex")
-        if buildindex_provided:
-            count = self._sortable_buildindex(url, ud, d, latest_rev)
+        if not uselocalcount:
+            buildindex_provided = hasattr(self, "_sortable_buildindex")
+            if buildindex_provided:
+                count = self._sortable_buildindex(url, ud, d, latest_rev)
 
         if count is None:
             count = "0"
-- 
1.6.5.rc1.44.ga1675

Re: AUTOREV and SRCPV

[Phil Blundell]

On Thu, 2010-06-03 at 09:37 +0200, Martin Jansa wrote:
> The other is when you change SRCREV in recipe without updating PV. Then
> SRCPV will give you newer PV which will upgrade resulting package on
> target.

If it's just a question of getting the package upgraded on the target
then you can do that with PKGV; there's no need to fiddle with PV at
all.  The only case where PV is important is where you need bitbake to
realise that the package needs to be rebuilt in the first (since the
SRCREVs aren't going to be sortable), but I think that's already fixable
with strategic use of PREFERRED_VERSION and/or DEFAULT_PREFERENCE.

p.

Re: AUTOREV and SRCPV

[Martin Jansa]

On Thu, Jun 03, 2010 at 09:13:37AM +0100, Phil Blundell wrote:
> On Thu, 2010-06-03 at 09:37 +0200, Martin Jansa wrote:
> > The other is when you change SRCREV in recipe without updating PV. Then
> > SRCPV will give you newer PV which will upgrade resulting package on
> > target.
> 
> If it's just a question of getting the package upgraded on the target
> then you can do that with PKGV; there's no need to fiddle with PV at
> all.  The only case where PV is important is where you need bitbake to
> realise that the package needs to be rebuilt in the first (since the
> SRCREVs aren't going to be sortable), 

Ah right, sure you need both PV and PKGV higher.

> but I think that's already fixable
> with strategic use of PREFERRED_VERSION and/or DEFAULT_PREFERENCE.

How? Can you explain in more detail?

Regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: AUTOREV and SRCPV

[Phil Blundell]

On Thu, 2010-06-03 at 10:30 +0200, Martin Jansa wrote:
> On Thu, Jun 03, 2010 at 09:13:37AM +0100, Phil Blundell wrote:
> > but I think that's already fixable
> > with strategic use of PREFERRED_VERSION and/or DEFAULT_PREFERENCE.
> 
> How? Can you explain in more detail?

If you arrange for the autorev'd recipe to be, say,
DEFAULT_PREFERENCE=1, then I think bitbake will treat it as the desired
version to build even if SRCREV - and hence PV - would sort lower than
some other version (irrespective of whether that other version has
already been built or not).  Then you just need to engineer PKGV to be
suitably monotonic in order to get the right things to happen inside
deploy/ and on the target system.

Obviously you will have a problem if PV is actually identical to some
previously-built version, but the whole idea of using hashes as SCM keys
is predicated on the assumption that this isn't going to happen.

p.

Re: AUTOREV and SRCPV

[Martin Jansa]

On Thu, Jun 03, 2010 at 09:37:17AM +0100, Phil Blundell wrote:
> On Thu, 2010-06-03 at 10:30 +0200, Martin Jansa wrote:
> > On Thu, Jun 03, 2010 at 09:13:37AM +0100, Phil Blundell wrote:
> > > but I think that's already fixable
> > > with strategic use of PREFERRED_VERSION and/or DEFAULT_PREFERENCE.
> > 
> > How? Can you explain in more detail?
> 
> If you arrange for the autorev'd recipe to be, say,
> DEFAULT_PREFERENCE=1, then I think bitbake will treat it as the desired
> version to build even if SRCREV - and hence PV - would sort lower than
> some other version (irrespective of whether that other version has
> already been built or not).  Then you just need to engineer PKGV to be
> suitably monotonic in order to get the right things to happen inside
> deploy/ and on the target system.

So will bitbake build again the same recipe (because I've built from the
same one before) with lower PV, just because it _still_ has the highest
DEFAULT_PREFERENCE and it's desired version (but still lower PV because
of hash, then before)?.

Maybe I had not enough coffee today, but I still don't understand it,
sorry, maybe someone else can comment?

Regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: AUTOREV and SRCPV

[Phil Blundell]

On Thu, 2010-06-03 at 10:46 +0200, Martin Jansa wrote:
> So will bitbake build again the same recipe (because I've built from the
> same one before) with lower PV, just because it _still_ has the highest
> DEFAULT_PREFERENCE and it's desired version (but still lower PV because
> of hash, then before)?.

Yes, I think so.  PV, and hence P, will be different and that's about
all that matters to bitbake.  The fact that PF is the same should be
irrelevant; I don't think any of its persistent state is keyed on that.

p.

Re: AUTOREV and SRCPV

[Phil Blundell]

On Thu, 2010-06-03 at 09:55 +0100, Phil Blundell wrote:
> On Thu, 2010-06-03 at 10:46 +0200, Martin Jansa wrote:
> > So will bitbake build again the same recipe (because I've built from the
> > same one before) with lower PV, just because it _still_ has the highest
> > DEFAULT_PREFERENCE and it's desired version (but still lower PV because
> > of hash, then before)?.
> 
> Yes, I think so.  PV, and hence P, will be different and that's about
> all that matters to bitbake.  The fact that PF is the same should be
> irrelevant; I don't think any of its persistent state is keyed on that.

I did a quick test on this and it does indeed seem to work as you would
expect.  So that seems like the easiest way of solving the problem at
hand.

p.

Re: AUTOREV and SRCPV

[pieterg]

On Thursday 03 June 2010 18:35:29 Phil Blundell wrote:
> On Thu, 2010-06-03 at 09:55 +0100, Phil Blundell wrote:
> > On Thu, 2010-06-03 at 10:46 +0200, Martin Jansa wrote:
> > > So will bitbake build again the same recipe (because I've built from
> > > the same one before) with lower PV, just because it _still_ has the
> > > highest DEFAULT_PREFERENCE and it's desired version (but still lower
> > > PV because of hash, then before)?.
> >
> > Yes, I think so.  PV, and hence P, will be different and that's about
> > all that matters to bitbake.  The fact that PF is the same should be
> > irrelevant; I don't think any of its persistent state is keyed on that.
>
> I did a quick test on this and it does indeed seem to work as you would
> expect.  So that seems like the easiest way of solving the problem at
> hand.

I probably don't know enough about the bitbake internals to see how to fix 
my problem with that, could you please point me in the right direction?
I think in case of AUTOREV I would still need some way to automatically 
include a sortable revision count in PKGV, so the actual problem would only 
move from PV to PKGV.
Or do you mean this would allow SRCPV to be removed from those hundreds of 
packages which don't use AUTOREV at the moment, so I can keep using AUTOREV 
for my own packages, without those other packages starting to cause 
problems?

Note that my problem is not 'getting the package upgraded on the target', 
but 'getting automatically incrementing package versions which are 
consistent on several build systems'.

So for me a simple workaround would be allowing BB_LOCALCOUNT_OVERRIDE to be 
defined per package, rather than globally.
Or something similar, which would allow me to use the sortable revision 
count only for those packages for which I really need AUTOREV.

Rgds, Pieter

Re: AUTOREV and SRCPV

[Phil Blundell]

On Fri, 2010-06-04 at 10:24 +0200, pieterg wrote:
> I probably don't know enough about the bitbake internals to see how to fix 
> my problem with that, could you please point me in the right direction?
> I think in case of AUTOREV I would still need some way to automatically 
> include a sortable revision count in PKGV, so the actual problem would only 
> move from PV to PKGV.

That's correct: essentially it is just moving the SRCPV-equivalent thing
from PV to PKGV.  But, PKGV only needs to be evaluated during the
do_package step, not at parse time, so you only incur the cost and risk
of running git rev-list (or whatever) for packages that you are actually
building rather than for all recipes in your source tree.

p.

Re: AUTOREV and SRCPV

[pieterg]

On Friday 04 June 2010 10:30:57 Phil Blundell wrote:
> On Fri, 2010-06-04 at 10:24 +0200, pieterg wrote:
> > I probably don't know enough about the bitbake internals to see how to
> > fix my problem with that, could you please point me in the right
> > direction? I think in case of AUTOREV I would still need some way to
> > automatically include a sortable revision count in PKGV, so the actual
> > problem would only move from PV to PKGV.
>
> That's correct: essentially it is just moving the SRCPV-equivalent thing
> from PV to PKGV.  But, PKGV only needs to be evaluated during the
> do_package step, not at parse time, so you only incur the cost and risk
> of running git rev-list (or whatever) for packages that you are actually
> building rather than for all recipes in your source tree.

I'm starting to see your point. That would really allow a much nicer AUTOREV 
implementation.
But in that case I'm not getting away with a simple workaround I'm afraid...

Rgds, Pieter

Re: AUTOREV and SRCPV

[Phil Blundell]

On Fri, 2010-06-04 at 10:44 +0200, pieterg wrote:
> I'm starting to see your point. That would really allow a much nicer AUTOREV 
> implementation.
> But in that case I'm not getting away with a simple workaround I'm afraid...

I think you're probably right that there is no trivial workaround, but I
don't think it would be very hard to fix this "properly".  You can do it
one recipe at a time; there's no need for a single big bang.

p.

Re: AUTOREV and SRCPV

[pieterg]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

On Friday 04 June 2010 13:03:28 Phil Blundell wrote:
> On Fri, 2010-06-04 at 10:44 +0200, pieterg wrote:
> > I'm starting to see your point. That would really allow a much nicer
> > AUTOREV implementation.
> > But in that case I'm not getting away with a simple workaround I'm
> > afraid...
>
> I think you're probably right that there is no trivial workaround, but I
> don't think it would be very hard to fix this "properly".  You can do it
> one recipe at a time; there's no need for a single big bang.

It turns out not to be that hard at all.
I've created a simple gitpkv.bbclass (see attachment), and I'm able to use 
it as follows, in the recipe:

inherit gitpkgv

PV = "1.0+git${SRCPV}"
PKGV = "1.0+git${GITPKGV}"

And in my distro's unstable-version.inc:

SRCREV_pn-packagename = "${AUTOREV}"

And in my distro's stable-version.inc:

SRCREV_pn-packagename = "5276f059461b6e9015e46a01b1cabfc42dfa600f"

Now I get a nice
1.0+git7348+5276f059461b6e9015e46a01b1cabfc42dfa600f
PKGV, which is the same in both stable/unstable situations, assuming 
the 'stable' rev is the latest rev.
And it is of course the same on all buildservers, and doesn't depend on 
what's in the cache.

And best of all, no more need for BB_LOCALCOUNT_OVERRIDE nor 
BB_GIT_CLONE_FOR_SRCREV, so no more lengthy bb parse sessions.

gitpkg.bbclass could be improved.
It currently does not support SRCREV_FORMAT, instead it assumes the first 
git url is to be used.
To make it obey SRCREV_FORMAT, I would have to duplicate more bitbake code.
Instead, it might be better if bb/lib/fetch/__init__.py would export helpers 
to find the correct src url, in the same way bb.fetch.get_srcrev does.
That would ensure gitpkgv.bbclass would always pick the correct repository.

But for now, I'm really happy with the result.

Rgds, Pieter

[-- Attachment #2: gitpkgv.bbclass --]
[-- Type: text/plain, Size: 1210 bytes --]

# gitpkgv.bbclass provides a GITPKGV variable which is a sortable version
# with the format NN+GITHASH, to be used in PKGV, where
#
# NN equals the total number of revs up to SRCREV
# GITHASH is SRCREV's (full) hash
#
# gitpkgv.bbclass assumes the git repository has been cloned, and contains
# SRCREV. So ${GITPKGV} should never be used in PV, only in PKGV.
# It can handle SRCREV = ${AUTOREV}, as well as SRCREV = "<some fixed git hash>"
#
# use example:
#
# inherit gitpkgv
#
# PV = "1.0+git${SRCPV}"
# PKGV = "1.0+git${GITPKGV}"

GITPKGV = "${@get_git_pkgv(d)}"

def get_git_pkgv(d):
	import os
	import bb

	urls = bb.data.getVar('SRC_URI', d, 1).split()

	for url in urls:
		(type, host, path, user, pswd, parm) = bb.decodeurl(bb.data.expand(url, d))
		if type in ['git']:

			gitsrcname = '%s%s' % (host, path.replace('/', '.'))
			repodir = os.path.join(bb.data.expand('${GITDIR}', d), gitsrcname)
			rev = bb.fetch.get_srcrev(d).split('+')[1]

			cwd = os.getcwd()
			os.chdir(repodir)
			output = bb.fetch.runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % rev, d, quiet=True)
			os.chdir(cwd)

			return "%s+%s" % (output.split()[0], rev)

	return "0+0"

Re: AUTOREV and SRCPV

[Phil Blundell]

On Sat, 2010-06-05 at 12:36 +0200, pieterg wrote:
> Now I get a nice
> 1.0+git7348+5276f059461b6e9015e46a01b1cabfc42dfa600f
> PKGV, which is the same in both stable/unstable situations, assuming 
> the 'stable' rev is the latest rev.
> And it is of course the same on all buildservers, and doesn't depend on 
> what's in the cache.
> 
> And best of all, no more need for BB_LOCALCOUNT_OVERRIDE nor 
> BB_GIT_CLONE_FOR_SRCREV, so no more lengthy bb parse sessions.

Yup, that looks good.

In the longer term I guess it would be nice to make this less
git-specific.  But, as you observe, that would mean either copying a lot
of code from bitbake or waiting for a new bitbake release which exposes
the necessary functionality.  And, in any case, the majority of projects
use only a single SCM which doesn't change very often, so I guess there
will not be too much of an issue in practice.

Thanks for your work, anyway.  I'll go ahead and check this in unless
somebody has an objection.

p.

Re: AUTOREV and SRCPV

[Khem Raj]

On (06/06/10 22:00), Phil Blundell wrote:
> On Sat, 2010-06-05 at 12:36 +0200, pieterg wrote:
> > Now I get a nice
> > 1.0+git7348+5276f059461b6e9015e46a01b1cabfc42dfa600f
> > PKGV, which is the same in both stable/unstable situations, assuming 
> > the 'stable' rev is the latest rev.
> > And it is of course the same on all buildservers, and doesn't depend on 
> > what's in the cache.
> > 
> > And best of all, no more need for BB_LOCALCOUNT_OVERRIDE nor 
> > BB_GIT_CLONE_FOR_SRCREV, so no more lengthy bb parse sessions.
> 
> Yup, that looks good.
> 
> In the longer term I guess it would be nice to make this less
> git-specific.  But, as you observe, that would mean either copying a lot
> of code from bitbake or waiting for a new bitbake release which exposes
> the necessary functionality.  And, in any case, the majority of projects
> use only a single SCM which doesn't change very often, so I guess there
> will not be too much of an issue in practice.
> 
> Thanks for your work, anyway.  I'll go ahead and check this in unless
> somebody has an objection.

Please go ahead. Seems to work ok here.
> 
> p.
> 
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: AUTOREV and SRCPV

[Phil Blundell]

On Sun, 2010-06-06 at 22:00 +0100, Phil Blundell wrote:
> Thanks for your work, anyway.  I'll go ahead and check this in unless
> somebody has an objection.

This is in now.

p.