Use AUTOREV and have revision/commit number in package name?

Use AUTOREV and have revision/commit number in package name?

[Denys Dmytriyenko]

All,

I've been playing with AUTOREV for some time now and also asked this question 
before on the IRC, but so far there seems to be no easy solution to the 
problem in subject.

Usually, when a package is built from an SCM repository, SRCREV is set to a 
specific commit#/tag etc. And SRCREV is also usually referenced from PV (like 
PV = "X.Y.Z-scmr${SRCREV}") to have the revision/commit number in the package 
name, to easily track different builds.

Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python function, 
which does the magic of fetching the latest snapshot from a repository. But 
now if PV contains references to SRCREV, it breaks (undesrandably), since 
SRCREV gets resolved in do_fetch and PV needs to be resolved early on.

Are there any standard ways to work around this limitation?

I was thinking this maybe quite tricky, since the actual revision number used, 
is not known until do_fetch runs the actual checkout, at which time changing 
PV is too late, as temp/work is already created and so on. So, would it be 
possible to hack one of the do_package tasks to add the actual revision/commit 
number to a final package name, bypassing changing PV/PR?

Any thoughts? Feel free to throw ideas. Thanks.

-- 
Denys

Re: Use AUTOREV and have revision/commit number in package name?

[Holger Hans Peter Freyther]

On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
> All,
>
> I've been playing with AUTOREV for some time now and also asked this
> question before on the IRC, but so far there seems to be no easy solution
> to the problem in subject.
>
> Usually, when a package is built from an SCM repository, SRCREV is set to a
> specific commit#/tag etc. And SRCREV is also usually referenced from PV
> (like PV = "X.Y.Z-scmr${SRCREV}") to have the revision/commit number in the
> package name, to easily track different builds.
>
> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
> function, which does the magic of fetching the latest snapshot from a
> repository. But now if PV contains references to SRCREV, it breaks
> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to be
> resolved early on.
>
> Are there any standard ways to work around this limitation?

Check the mailinglist archives... Openmoko has deployed a solution that has 
fixed that about 1 1/2 years ago.

Short answer:
	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
	- Populate SRCPV with something based on the SRCREV..


z.

Re: Use AUTOREV and have revision/commit number in package name?

[Denys Dmytriyenko]

On Thu, Aug 20, 2009 at 07:52:58AM +0200, Holger Hans Peter Freyther wrote:
> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
> > All,
> >
> > I've been playing with AUTOREV for some time now and also asked this
> > question before on the IRC, but so far there seems to be no easy solution
> > to the problem in subject.
> >
> > Usually, when a package is built from an SCM repository, SRCREV is set to a
> > specific commit#/tag etc. And SRCREV is also usually referenced from PV
> > (like PV = "X.Y.Z-scmr${SRCREV}") to have the revision/commit number in the
> > package name, to easily track different builds.
> >
> > Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
> > function, which does the magic of fetching the latest snapshot from a
> > repository. But now if PV contains references to SRCREV, it breaks
> > (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to be
> > resolved early on.
> >
> > Are there any standard ways to work around this limitation?
> 
> Check the mailinglist archives... Openmoko has deployed a solution that has 
> fixed that about 1 1/2 years ago.
> 
> Short answer:
> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
> 	- Populate SRCPV with something based on the SRCREV..

Ah, thanks! I was able to find your RFC message from last year on the Openmoko 
mailing list:
http://markmail.org/message/v3s32gmxxqcaxpph

I'll give it a try. Thanks.

To be clear - it's not used in OE. While there is a single python-phoneutils 
recipe, which references SRCPV, it is not being set anywhere, apparently...

-- 
Denys

Re: Use AUTOREV and have revision/commit number in package name?

[Koen Kooi]

On 20-08-09 09:10, Denys Dmytriyenko wrote:
> On Thu, Aug 20, 2009 at 07:52:58AM +0200, Holger Hans Peter Freyther wrote:
>> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
>>> All,
>>>
>>> I've been playing with AUTOREV for some time now and also asked this
>>> question before on the IRC, but so far there seems to be no easy solution
>>> to the problem in subject.
>>>
>>> Usually, when a package is built from an SCM repository, SRCREV is set to a
>>> specific commit#/tag etc. And SRCREV is also usually referenced from PV
>>> (like PV = "X.Y.Z-scmr${SRCREV}") to have the revision/commit number in the
>>> package name, to easily track different builds.
>>>
>>> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
>>> function, which does the magic of fetching the latest snapshot from a
>>> repository. But now if PV contains references to SRCREV, it breaks
>>> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to be
>>> resolved early on.
>>>
>>> Are there any standard ways to work around this limitation?
>>
>> Check the mailinglist archives... Openmoko has deployed a solution that has
>> fixed that about 1 1/2 years ago.
>>
>> Short answer:
>> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
>> 	- Populate SRCPV with something based on the SRCREV..
>
> Ah, thanks! I was able to find your RFC message from last year on the Openmoko
> mailing list:
> http://markmail.org/message/v3s32gmxxqcaxpph
>
> I'll give it a try. Thanks.
>
> To be clear - it's not used in OE. While there is a single python-phoneutils
> recipe, which references SRCPV, it is not being set anywhere, apparently...

Take a look at 
http://cgit.openembedded.org/cgit.cgi/openembedded/log/?h=xora/angstrom-srcpv 
:)

regards,

Koen

Re: Use AUTOREV and have revision/commit number in package name?

[Denys Dmytriyenko]

On Thu, Aug 20, 2009 at 10:34:58AM +0200, Koen Kooi wrote:
> On 20-08-09 09:10, Denys Dmytriyenko wrote:
>> On Thu, Aug 20, 2009 at 07:52:58AM +0200, Holger Hans Peter Freyther 
>> wrote:
>>> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
>>>> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
>>>> function, which does the magic of fetching the latest snapshot from a
>>>> repository. But now if PV contains references to SRCREV, it breaks
>>>> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to 
>>>> be
>>>> resolved early on.
>>>>
>>>> Are there any standard ways to work around this limitation?
>>>
>>> Check the mailinglist archives... Openmoko has deployed a solution that 
>>> has
>>> fixed that about 1 1/2 years ago.
>>>
>>> Short answer:
>>> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
>>> 	- Populate SRCPV with something based on the SRCREV..
>>
>> Ah, thanks! I was able to find your RFC message from last year on the 
>> Openmoko
>> mailing list:
>> http://markmail.org/message/v3s32gmxxqcaxpph
>>
>> I'll give it a try. Thanks.
>>
>> To be clear - it's not used in OE. While there is a single 
>> python-phoneutils
>> recipe, which references SRCPV, it is not being set anywhere, 
>> apparently...
>
> Take a look at 
> http://cgit.openembedded.org/cgit.cgi/openembedded/log/?h=xora/angstrom-srcpv 
> :)

Thanks! More specifically:

http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?h=xora/angstrom-srcpv&id=453a13c7a68087895f4048410aba1a519ec99897

Any ETA for the merge back to .dev? :)

-- 
Denys

Re: Use AUTOREV and have revision/commit number in package name?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]

On Thu, Aug 20, 2009 at 05:01:36AM -0400, Denys Dmytriyenko wrote:
> On Thu, Aug 20, 2009 at 10:34:58AM +0200, Koen Kooi wrote:
> > On 20-08-09 09:10, Denys Dmytriyenko wrote:
> >> On Thu, Aug 20, 2009 at 07:52:58AM +0200, Holger Hans Peter Freyther 
> >> wrote:
> >>> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
> >>>> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
> >>>> function, which does the magic of fetching the latest snapshot from a
> >>>> repository. But now if PV contains references to SRCREV, it breaks
> >>>> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to 
> >>>> be
> >>>> resolved early on.
> >>>>
> >>>> Are there any standard ways to work around this limitation?
> >>>
> >>> Check the mailinglist archives... Openmoko has deployed a solution that 
> >>> has
> >>> fixed that about 1 1/2 years ago.
> >>>
> >>> Short answer:
> >>> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
> >>> 	- Populate SRCPV with something based on the SRCREV..
> >>
> >> Ah, thanks! I was able to find your RFC message from last year on the 
> >> Openmoko
> >> mailing list:
> >> http://markmail.org/message/v3s32gmxxqcaxpph
> >>
> >> I'll give it a try. Thanks.
> >>
> >> To be clear - it's not used in OE. While there is a single 
> >> python-phoneutils
> >> recipe, which references SRCPV, it is not being set anywhere, 
> >> apparently...
> >
> > Take a look at 
> > http://cgit.openembedded.org/cgit.cgi/openembedded/log/?h=xora/angstrom-srcpv 
> > :)
> 
> Thanks! More specifically:
> 
> http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?h=xora/angstrom-srcpv&id=453a13c7a68087895f4048410aba1a519ec99897
> 
> Any ETA for the merge back to .dev? :)

Just out of my curiosity:

How are PREFERRED_VERSION_pkg supposed to work when I prefer pkg with
autoreved revision.

bbfile will be named like pkg_git.bb
inside will be PV="1.0.0+gitr${SRCPV}" and DEFAULT_PREFERENCE=-1

I would like to force using this _git version even when there will be
pkg_1.1.0.bb with PV="1.1.0"

For fixed revision ie from sane-srcrevs i can bump PREFERRED_VERSION_pkg
with every revision bump in sane-srcrevs, but it also suboptimal :/.

With autorev I cannot set PREFERRED_VERSION_pkg at all, if I understand
it right, because I never know which string will be in ${SRCPV} and
using just PREFERRED_VERSION_pkg="1.0.0+git" or "git" ends with
"preferred version is not available". Setting PREFERRED_VERSION_pkg to
"1.0.0+gitr${SRCPV}" is not expanded and ends with a bit confusing 
bitbake exception about SRC_URI not referencing some SCM.

I've asked on IRC and they said that its common practise to remove
PREFERRED_VERSION_pkg and adjust DEFAULT_PREFERENCE or blacklist all
other versions (like that with PV="1.1.0").

Now I solved my problem with updating PV in recipies which weren't
preferred by bitbake to latest version from all available (moved to
PV="1.1.0+gitr${SRCPV}" when pkg_1.1.0.bb is commited and removed
DEFAULT_PREFERENCE=-1.

Could it be solved better? Something like:
1) use .bb file with exact the same PV
2) use .bb file where PV starts with the same string as is in
PREFERRED_VERSION_pkg if exact is not found and print warning.

Regards

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: Use AUTOREV and have revision/commit number in package name?

[Denys Dmytriyenko]

On Thu, Aug 20, 2009 at 11:46:42AM +0200, Martin Jansa wrote:
> On Thu, Aug 20, 2009 at 05:01:36AM -0400, Denys Dmytriyenko wrote:
> > On Thu, Aug 20, 2009 at 10:34:58AM +0200, Koen Kooi wrote:
> > > On 20-08-09 09:10, Denys Dmytriyenko wrote:
> > >> On Thu, Aug 20, 2009 at 07:52:58AM +0200, Holger Hans Peter Freyther 
> > >> wrote:
> > >>> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
> > >>>> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
> > >>>> function, which does the magic of fetching the latest snapshot from a
> > >>>> repository. But now if PV contains references to SRCREV, it breaks
> > >>>> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to 
> > >>>> be
> > >>>> resolved early on.
> > >>>>
> > >>>> Are there any standard ways to work around this limitation?
> > >>>
> > >>> Check the mailinglist archives... Openmoko has deployed a solution that 
> > >>> has
> > >>> fixed that about 1 1/2 years ago.
> > >>>
> > >>> Short answer:
> > >>> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
> > >>> 	- Populate SRCPV with something based on the SRCREV..
> > >>
> > >> Ah, thanks! I was able to find your RFC message from last year on the 
> > >> Openmoko
> > >> mailing list:
> > >> http://markmail.org/message/v3s32gmxxqcaxpph
> > >>
> > >> I'll give it a try. Thanks.
> > >>
> > >> To be clear - it's not used in OE. While there is a single 
> > >> python-phoneutils
> > >> recipe, which references SRCPV, it is not being set anywhere, 
> > >> apparently...
> > >
> > > Take a look at 
> > > http://cgit.openembedded.org/cgit.cgi/openembedded/log/?h=xora/angstrom-srcpv 
> > > :)
> > 
> > Thanks! More specifically:
> > 
> > http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?h=xora/angstrom-srcpv&id=453a13c7a68087895f4048410aba1a519ec99897
> > 
> > Any ETA for the merge back to .dev? :)
> 
> Just out of my curiosity:
> 
> How are PREFERRED_VERSION_pkg supposed to work when I prefer pkg with
> autoreved revision.
> 
> bbfile will be named like pkg_git.bb
> inside will be PV="1.0.0+gitr${SRCPV}" and DEFAULT_PREFERENCE=-1
> 
> I would like to force using this _git version even when there will be
> pkg_1.1.0.bb with PV="1.1.0"
> 
> For fixed revision ie from sane-srcrevs i can bump PREFERRED_VERSION_pkg
> with every revision bump in sane-srcrevs, but it also suboptimal :/.
> 
> With autorev I cannot set PREFERRED_VERSION_pkg at all, if I understand
> it right, because I never know which string will be in ${SRCPV} and
> using just PREFERRED_VERSION_pkg="1.0.0+git" or "git" ends with
> "preferred version is not available". Setting PREFERRED_VERSION_pkg to
> "1.0.0+gitr${SRCPV}" is not expanded and ends with a bit confusing 
> bitbake exception about SRC_URI not referencing some SCM.
> 
> I've asked on IRC and they said that its common practise to remove
> PREFERRED_VERSION_pkg and adjust DEFAULT_PREFERENCE or blacklist all
> other versions (like that with PV="1.1.0").
> 
> Now I solved my problem with updating PV in recipies which weren't
> preferred by bitbake to latest version from all available (moved to
> PV="1.1.0+gitr${SRCPV}" when pkg_1.1.0.bb is commited and removed
> DEFAULT_PREFERENCE=-1.
> 
> Could it be solved better? Something like:
> 1) use .bb file with exact the same PV
> 2) use .bb file where PV starts with the same string as is in
> PREFERRED_VERSION_pkg if exact is not found and print warning.

Another approach, which may be even less elegant, would be to rename 
pkg_git.bb into pkg-git.bb, let it PROVIDE=pkg and select it by 
PREFERRED_PROVIDER_pkg=pkg-git, then its lower version wouldn't matter.

-- 
Denys

Re: Use AUTOREV and have revision/commit number in package name?

[Martin Jansa]

[-- Attachment #1.1: Type: text/plain, Size: 2197 bytes --]

On Thu, Aug 20, 2009 at 11:46:42AM +0200, Martin Jansa wrote:
> Just out of my curiosity:
> 
> How are PREFERRED_VERSION_pkg supposed to work when I prefer pkg with
> autoreved revision.
> 
> bbfile will be named like pkg_git.bb
> inside will be PV="1.0.0+gitr${SRCPV}" and DEFAULT_PREFERENCE=-1
> 
> I would like to force using this _git version even when there will be
> pkg_1.1.0.bb with PV="1.1.0"
> 
> For fixed revision ie from sane-srcrevs i can bump PREFERRED_VERSION_pkg
> with every revision bump in sane-srcrevs, but it also suboptimal :/.
> 
> With autorev I cannot set PREFERRED_VERSION_pkg at all, if I understand
> it right, because I never know which string will be in ${SRCPV} and
> using just PREFERRED_VERSION_pkg="1.0.0+git" or "git" ends with
> "preferred version is not available". Setting PREFERRED_VERSION_pkg to
> "1.0.0+gitr${SRCPV}" is not expanded and ends with a bit confusing 
> bitbake exception about SRC_URI not referencing some SCM.
> 
> I've asked on IRC and they said that its common practise to remove
> PREFERRED_VERSION_pkg and adjust DEFAULT_PREFERENCE or blacklist all
> other versions (like that with PV="1.1.0").
> 
> Now I solved my problem with updating PV in recipies which weren't
> preferred by bitbake to latest version from all available (moved to
> PV="1.1.0+gitr${SRCPV}" when pkg_1.1.0.bb is commited and removed
> DEFAULT_PREFERENCE=-1.
> 
> Could it be solved better? Something like:
> 1) use .bb file with exact the same PV
> 2) use .bb file where PV starts with the same string as is in
> PREFERRED_VERSION_pkg if exact is not found and print warning.

What about this small patch in attachment?

Is there better character for wildchar ('*')?

For this type of PVs is wildchar usefull only in the end of
PREFERRED_VERSION_pkg, would someone need it somewhere else?

Testing regexp match seems like too much for me, at least because
preferred versions contain '.' now and it would be confusing to use
regexp without escaping them.

Regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #1.2: 0001-Allow-as-wildcard-in-the-end-of-PREFERRED_VERSION_pk.patch --]
[-- Type: text/plain, Size: 1192 bytes --]

From b65bb3e5bf29c59f2f1da297a89af6c02a1a2a4f Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Wed, 26 Aug 2009 21:49:33 +0200
Subject: [PATCH] Allow % as wildcard in the end of PREFERRED_VERSION_pkg

---
 lib/bb/providers.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/bb/providers.py b/lib/bb/providers.py
index 8970fb3..b2a9fea 100644
--- a/lib/bb/providers.py
+++ b/lib/bb/providers.py
@@ -96,7 +96,7 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg_pn = None, item = None):
         for file_set in pkg_pn:
             for f in file_set:
                 pe,pv,pr = dataCache.pkg_pepvpr[f]
-                if preferred_v == pv and (preferred_r == pr or preferred_r == None) and (preferred_e == pe or preferred_e == None):
+                if (pv == preferred_v or (preferred_v != None and preferred_v.endswith('%') and pv.startswith(preferred_v[:len(preferred_v)-1]))) and (pr == preferred_r or preferred_r == None) and (pe == preferred_e or preferred_e == None):
                     preferred_file = f
                     preferred_ver = (pe, pv, pr)
                     break
-- 
1.6.4


[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: Use AUTOREV and have revision/commit number in package name?

[Graeme Gregory]

Denys Dmytriyenko wrote:
> On Thu, Aug 20, 2009 at 10:34:58AM +0200, Koen Kooi wrote:
>   
>> On 20-08-09 09:10, Denys Dmytriyenko wrote:
>>     
>>> On Thu, Aug 20, 2009 at 07:52:58AM +0200, Holger Hans Peter Freyther 
>>> wrote:
>>>       
>>>> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
>>>>         
>>>>> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
>>>>> function, which does the magic of fetching the latest snapshot from a
>>>>> repository. But now if PV contains references to SRCREV, it breaks
>>>>> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to 
>>>>> be
>>>>> resolved early on.
>>>>>
>>>>> Are there any standard ways to work around this limitation?
>>>>>           
>>>> Check the mailinglist archives... Openmoko has deployed a solution that 
>>>> has
>>>> fixed that about 1 1/2 years ago.
>>>>
>>>> Short answer:
>>>> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
>>>> 	- Populate SRCPV with something based on the SRCREV..
>>>>         
>>> Ah, thanks! I was able to find your RFC message from last year on the 
>>> Openmoko
>>> mailing list:
>>> http://markmail.org/message/v3s32gmxxqcaxpph
>>>
>>> I'll give it a try. Thanks.
>>>
>>> To be clear - it's not used in OE. While there is a single 
>>> python-phoneutils
>>> recipe, which references SRCPV, it is not being set anywhere, 
>>> apparently...
>>>       
>> Take a look at 
>> http://cgit.openembedded.org/cgit.cgi/openembedded/log/?h=xora/angstrom-srcpv 
>> :)
>>     
>
> Thanks! More specifically:
>
> http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?h=xora/angstrom-srcpv&id=453a13c7a68087895f4048410aba1a519ec99897
>
> Any ETA for the merge back to .dev? :)
>
>   
Not yet, I am running out of time to actually work on it, I suspect that 
monster commit is the wrong way to do it anyway.

Probably merge bitbake.conf changes. Then merge by package-name as the 
SRCPV packages should always be versioned to be the same or an upgrade 
to the old PV.

Graeme

Re: Use AUTOREV and have revision/commit number in package name?

[Graeme Gregory]

Holger Hans Peter Freyther wrote:
> On Thursday 20 August 2009 05:28:02 Denys Dmytriyenko wrote:
>   
>> All,
>>
>> I've been playing with AUTOREV for some time now and also asked this
>> question before on the IRC, but so far there seems to be no easy solution
>> to the problem in subject.
>>
>> Usually, when a package is built from an SCM repository, SRCREV is set to a
>> specific commit#/tag etc. And SRCREV is also usually referenced from PV
>> (like PV = "X.Y.Z-scmr${SRCREV}") to have the revision/commit number in the
>> package name, to easily track different builds.
>>
>> Now, when SRCREV = "${AUTOREV}", AUTOREV gets resolved into a Python
>> function, which does the magic of fetching the latest snapshot from a
>> repository. But now if PV contains references to SRCREV, it breaks
>> (undesrandably), since SRCREV gets resolved in do_fetch and PV needs to be
>> resolved early on.
>>
>> Are there any standard ways to work around this limitation?
>>     
>
> Check the mailinglist archives... Openmoko has deployed a solution that has 
> fixed that about 1 1/2 years ago.
>
> Short answer:
> 	- Do not put SRCREV into PV, e.g. create a new var called SRCPV
> 	- Populate SRCPV with something based on the SRCREV..
>
>
>   
Holger, I have been trying to maintain a branch with your SRCPV work in 
it, xora/angstrom-srcpv in git. Unfortuneately as I only get to do it in 
spare time I never seem to be able to quite keep up with OE mainline.

But it would be good to start merging this into mainline, probably 
package by package.

Graeme