Request to move "gitpkgv.bbclass" from meta-oe to oe-core

Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Mike Looijmans]

The gitpkgv class is very helpful when you want packages to have a consistent 
incremental version number, when build machines cannot talk to eachother 
(ruling out using PRSERVER for example) other than through repositories.

However, this small gem is present in meta-oe and not in the core. I now have 
several layers that pull in all of meta-oe just for being able to number GIT 
versions incrementally.

Could gitpkgv be moved into oe-core? (I'll be happy to write and submit the 
patch.) Or, is there an alternative that does something similar?

Kind regards,
Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Mike Looijmans]

In partial reply to my own question, this has been discussed 2 years ago, but 
apparently nothing ever came out of it.

http://lists.openembedded.org/pipermail/openembedded-core/2012-February/056050.html

So once more the request to be able to get sane version strings for git repos. 
Can we please just move gitpkgv. That's like five minutes work, and if anyone 
(e.g. me) feels like building something better, he or she is free do submit 
that and get rid of the gitpkgv class once it's obsolete.

Mike.

On 10/08/2014 04:39 PM, Mike Looijmans wrote:
> The gitpkgv class is very helpful when you want packages to have a consistent
> incremental version number, when build machines cannot talk to eachother
> (ruling out using PRSERVER for example) other than through repositories.
>
> However, this small gem is present in meta-oe and not in the core. I now have
> several layers that pull in all of meta-oe just for being able to number GIT
> versions incrementally.
>
> Could gitpkgv be moved into oe-core? (I'll be happy to write and submit the
> patch.) Or, is there an alternative that does something similar?
>
> Kind regards,
> Mike.
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
>
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) (0) 499 33 69 79
> Telefax:  (+31) (0) 499 33 69 70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
>
> Please consider the environment before printing this e-mail
>
> Topic zoekt gedreven (embedded) software specialisten!
> http://topic.nl/vacatures/topic-zoekt-software-engineers/
>



Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Richard Purdie]

On Thu, 2014-10-09 at 10:16 +0200, Mike Looijmans wrote:
> In partial reply to my own question, this has been discussed 2 years ago, but 
> apparently nothing ever came out of it.
> 
> http://lists.openembedded.org/pipermail/openembedded-core/2012-February/056050.html
> 
> So once more the request to be able to get sane version strings for git repos. 
> Can we please just move gitpkgv. That's like five minutes work, and if anyone 
> (e.g. me) feels like building something better, he or she is free do submit 
> that and get rid of the gitpkgv class once it's obsolete.

I do feel quite strongly that this is something which should get
improved in the fetcher itself. That is why I've pushed back on the
patch since if I take it into OE-Core, there is even less incentive to
fix the real problem...

Cheers,

Richard

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Otavio Salvador]

On Thu, Oct 9, 2014 at 4:56 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2014-10-09 at 10:16 +0200, Mike Looijmans wrote:
>> In partial reply to my own question, this has been discussed 2 years ago, but
>> apparently nothing ever came out of it.
>>
>> http://lists.openembedded.org/pipermail/openembedded-core/2012-February/056050.html
>>
>> So once more the request to be able to get sane version strings for git repos.
>> Can we please just move gitpkgv. That's like five minutes work, and if anyone
>> (e.g. me) feels like building something better, he or she is free do submit
>> that and get rid of the gitpkgv class once it's obsolete.
>
> I do feel quite strongly that this is something which should get
> improved in the fetcher itself. That is why I've pushed back on the
> patch since if I take it into OE-Core, there is even less incentive to
> fix the real problem...

Mike I am also an user of gitpkgv but I must to agree with Richard
here. Moving this to OE-Core will just make it worse. It is way better
to prepare a patch and add support for it on the fetcher code of
BitBake.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Mike Looijmans]

On 10/10/2014 12:15 AM, Otavio Salvador wrote:
> On Thu, Oct 9, 2014 at 4:56 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> On Thu, 2014-10-09 at 10:16 +0200, Mike Looijmans wrote:
>>> In partial reply to my own question, this has been discussed 2 years ago, but
>>> apparently nothing ever came out of it.
>>>
>>> http://lists.openembedded.org/pipermail/openembedded-core/2012-February/056050.html
>>>
>>> So once more the request to be able to get sane version strings for git repos.
>>> Can we please just move gitpkgv. That's like five minutes work, and if anyone
>>> (e.g. me) feels like building something better, he or she is free do submit
>>> that and get rid of the gitpkgv class once it's obsolete.
>>
>> I do feel quite strongly that this is something which should get
>> improved in the fetcher itself. That is why I've pushed back on the
>> patch since if I take it into OE-Core, there is even less incentive to
>> fix the real problem...
>
> Mike I am also an user of gitpkgv but I must to agree with Richard
> here. Moving this to OE-Core will just make it worse. It is way better
> to prepare a patch and add support for it on the fetcher code of
> BitBake.

Give me some pointers on what you're talking about here, and I'll be happy to 
try and implement it. Currently I have no idea as to what is "wrong" with the 
gitpkgv approach, and I also have no clue as to what it is you want to have 
implemented in the git fetcher.

Kind regards,
Mike.



Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Paul Eggleton]

Hi Mike,

On Friday 10 October 2014 07:36:25 Mike Looijmans wrote:
> On 10/10/2014 12:15 AM, Otavio Salvador wrote:
> > On Thu, Oct 9, 2014 at 4:56 PM, Richard Purdie
> > 
> > <richard.purdie@linuxfoundation.org> wrote:
> >> On Thu, 2014-10-09 at 10:16 +0200, Mike Looijmans wrote:
> >>> In partial reply to my own question, this has been discussed 2 years
> >>> ago, but apparently nothing ever came out of it.
> >>> 
> >>> http://lists.openembedded.org/pipermail/openembedded-core/2012-February/
> >>> 056050.html
> >>> 
> >>> So once more the request to be able to get sane version strings for git
> >>> repos. Can we please just move gitpkgv. That's like five minutes work,
> >>> and if anyone (e.g. me) feels like building something better, he or she
> >>> is free do submit that and get rid of the gitpkgv class once it's
> >>> obsolete.
> >> 
> >> I do feel quite strongly that this is something which should get
> >> improved in the fetcher itself. That is why I've pushed back on the
> >> patch since if I take it into OE-Core, there is even less incentive to
> >> fix the real problem...
> > 
> > Mike I am also an user of gitpkgv but I must to agree with Richard
> > here. Moving this to OE-Core will just make it worse. It is way better
> > to prepare a patch and add support for it on the fetcher code of
> > BitBake.
> 
> Give me some pointers on what you're talking about here, and I'll be happy
> to try and implement it. Currently I have no idea as to what is "wrong"
> with the gitpkgv approach, and I also have no clue as to what it is you
> want to have implemented in the git fetcher.

The code would go into the fetcher itself i.e. lib/bb/fetch2/git.py and 
lib/bb/fetch2/__init__.py. 

I'm not sure whether it would be a case of setting a variable in order to have 
the value of SRCPV within OE (which calls bb.fetch2.get_srcrev() ) return the 
"git describe" style version instead of the current one, or whether we would 
preserve the separate GITPKGVTAG variable for that purpose. (The GITPKGV case 
is already handled by the current behaviour of SRCPV, right?)

FWIW there is an open bug covering this issue:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=2872

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Mike Looijmans]

On 10/10/2014 05:03 PM, Paul Eggleton wrote:
> Hi Mike,
>
> On Friday 10 October 2014 07:36:25 Mike Looijmans wrote:
>> On 10/10/2014 12:15 AM, Otavio Salvador wrote:
>>> On Thu, Oct 9, 2014 at 4:56 PM, Richard Purdie
>>>
>>> <richard.purdie@linuxfoundation.org> wrote:
>>>> On Thu, 2014-10-09 at 10:16 +0200, Mike Looijmans wrote:
>>>>> In partial reply to my own question, this has been discussed 2 years
>>>>> ago, but apparently nothing ever came out of it.
>>>>>
>>>>> http://lists.openembedded.org/pipermail/openembedded-core/2012-February/
>>>>> 056050.html
>>>>>
>>>>> So once more the request to be able to get sane version strings for git
>>>>> repos. Can we please just move gitpkgv. That's like five minutes work,
>>>>> and if anyone (e.g. me) feels like building something better, he or she
>>>>> is free do submit that and get rid of the gitpkgv class once it's
>>>>> obsolete.
>>>>
>>>> I do feel quite strongly that this is something which should get
>>>> improved in the fetcher itself. That is why I've pushed back on the
>>>> patch since if I take it into OE-Core, there is even less incentive to
>>>> fix the real problem...
>>>
>>> Mike I am also an user of gitpkgv but I must to agree with Richard
>>> here. Moving this to OE-Core will just make it worse. It is way better
>>> to prepare a patch and add support for it on the fetcher code of
>>> BitBake.
>>
>> Give me some pointers on what you're talking about here, and I'll be happy
>> to try and implement it. Currently I have no idea as to what is "wrong"
>> with the gitpkgv approach, and I also have no clue as to what it is you
>> want to have implemented in the git fetcher.
>
> The code would go into the fetcher itself i.e. lib/bb/fetch2/git.py and
> lib/bb/fetch2/__init__.py.
>
> I'm not sure whether it would be a case of setting a variable in order to have
> the value of SRCPV within OE (which calls bb.fetch2.get_srcrev() ) return the
> "git describe" style version instead of the current one, or whether we would
> preserve the separate GITPKGVTAG variable for that purpose. (The GITPKGV case
> is already handled by the current behaviour of SRCPV, right?)
>
> FWIW there is an open bug covering this issue:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=2872

I followed that link a few clicks deeper and noticed that Richard Purdie 
suggested adding another interface method, but I couldn't quite understand 
what he meant here.


I figured that "sortable_revision" was the method that I needed to delve into. 
However, it turned out that this one is called about 50 times when building a 
single package, and i cannot really run any git commands at that spot it 
seems, the current directory is not inside the repo.

So that's a fail (my trial code from git.py with some extra logger output below).



     def sortable_revision(self, ud, d, name):
         """
         Return a sortable revision number by counting commits in the history
         """
		rev = self._build_revision(ud, d, name)
		logger.warn("git.sortable_revision url=%s" % (ud.url))
		localpath = ud.localpath
		rev_file = os.path.join(localpath, "oe-gitpkgv_" + rev)
		if not os.path.exists(localpath):
			logger.warn("git.sortable_revision non-existent: %s" % (localpath))
			commits = None
		else:
			if not os.path.exists(rev_file) or not os.path.getsize(rev_file):
				from pipes import quote
				commits = bb.fetch2.runfetchcmd(
						"git rev-list %s -- 2> /dev/null | wc -l" % (quote(rev)),
						d, quiet=True).strip().lstrip('0')
				if commits:
					os.unlink(rev_file)
					open(rev_file, "w").write("%d\n" % int(commits))
			else:
				commits = open(rev_file, "r").readline(128).strip()
		logger.warn("git.sortable_revision commits=%s\n" % (commits))
		if commits:
			return False, "%s+%s" % (commits, rev[:7])
		else:
			return True, str(rev)




Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Request to move "gitpkgv.bbclass" from meta-oe to oe-core

[Mike Looijmans]

I'm afraid this will go the same path as the previous attempt at getting 
gitpkgv equivalent into oe-core, which is, it'll silently disappear until 
someone with both the internal knowledge and the time to implement it will 
pick it up again...

M.

On 10/20/2014 04:00 PM, Mike Looijmans wrote:
> On 10/10/2014 05:03 PM, Paul Eggleton wrote:
>> Hi Mike,
>>
>> On Friday 10 October 2014 07:36:25 Mike Looijmans wrote:
>>> On 10/10/2014 12:15 AM, Otavio Salvador wrote:
>>>> On Thu, Oct 9, 2014 at 4:56 PM, Richard Purdie
>>>>
>>>> <richard.purdie@linuxfoundation.org> wrote:
>>>>> On Thu, 2014-10-09 at 10:16 +0200, Mike Looijmans wrote:
>>>>>> In partial reply to my own question, this has been discussed 2 years
>>>>>> ago, but apparently nothing ever came out of it.
>>>>>>
>>>>>> http://lists.openembedded.org/pipermail/openembedded-core/2012-February/
>>>>>> 056050.html
>>>>>>
>>>>>> So once more the request to be able to get sane version strings for git
>>>>>> repos. Can we please just move gitpkgv. That's like five minutes work,
>>>>>> and if anyone (e.g. me) feels like building something better, he or she
>>>>>> is free do submit that and get rid of the gitpkgv class once it's
>>>>>> obsolete.
>>>>>
>>>>> I do feel quite strongly that this is something which should get
>>>>> improved in the fetcher itself. That is why I've pushed back on the
>>>>> patch since if I take it into OE-Core, there is even less incentive to
>>>>> fix the real problem...
>>>>
>>>> Mike I am also an user of gitpkgv but I must to agree with Richard
>>>> here. Moving this to OE-Core will just make it worse. It is way better
>>>> to prepare a patch and add support for it on the fetcher code of
>>>> BitBake.
>>>
>>> Give me some pointers on what you're talking about here, and I'll be happy
>>> to try and implement it. Currently I have no idea as to what is "wrong"
>>> with the gitpkgv approach, and I also have no clue as to what it is you
>>> want to have implemented in the git fetcher.
>>
>> The code would go into the fetcher itself i.e. lib/bb/fetch2/git.py and
>> lib/bb/fetch2/__init__.py.
>>
>> I'm not sure whether it would be a case of setting a variable in order to have
>> the value of SRCPV within OE (which calls bb.fetch2.get_srcrev() ) return the
>> "git describe" style version instead of the current one, or whether we would
>> preserve the separate GITPKGVTAG variable for that purpose. (The GITPKGV case
>> is already handled by the current behaviour of SRCPV, right?)
>>
>> FWIW there is an open bug covering this issue:
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=2872
>
> I followed that link a few clicks deeper and noticed that Richard Purdie
> suggested adding another interface method, but I couldn't quite understand
> what he meant here.
>
>
> I figured that "sortable_revision" was the method that I needed to delve into.
> However, it turned out that this one is called about 50 times when building a
> single package, and i cannot really run any git commands at that spot it
> seems, the current directory is not inside the repo.
>
> So that's a fail (my trial code from git.py with some extra logger output below).
>
>
>
>      def sortable_revision(self, ud, d, name):
>          """
>          Return a sortable revision number by counting commits in the history
>          """
>          rev = self._build_revision(ud, d, name)
>          logger.warn("git.sortable_revision url=%s" % (ud.url))
>          localpath = ud.localpath
>          rev_file = os.path.join(localpath, "oe-gitpkgv_" + rev)
>          if not os.path.exists(localpath):
>              logger.warn("git.sortable_revision non-existent: %s" % (localpath))
>              commits = None
>          else:
>              if not os.path.exists(rev_file) or not os.path.getsize(rev_file):
>                  from pipes import quote
>                  commits = bb.fetch2.runfetchcmd(
>                          "git rev-list %s -- 2> /dev/null | wc -l" %
> (quote(rev)),
>                          d, quiet=True).strip().lstrip('0')
>                  if commits:
>                      os.unlink(rev_file)
>                      open(rev_file, "w").write("%d\n" % int(commits))
>              else:
>                  commits = open(rev_file, "r").readline(128).strip()
>          logger.warn("git.sortable_revision commits=%s\n" % (commits))
>          if commits:
>              return False, "%s+%s" % (commits, rev[:7])
>          else:
>              return True, str(rev)



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/