versioning madness

versioning madness

[Tom Walsh]

I would like to know how others are handling this problem.   As you 
already know, versioning can be "interesting" with OE.  It is probably 
how I have approached the issue of keeping my list of files seperate 
from the OE bb filetree.

In my project, I have a number of packages which needed specific changes 
to files so that they reflected the needs of the target.  A simple 
example is that of a list of /dev/devices.  initscripts controls the 
device creation through the device_table list.  So, I have a 
device_table file unique to my target.   No problem.

What is a problem is controlling the changes that I make to the OE 
packages tree.  If I simply create a ${MACHINE_NAME} directory under 
packages/initscripts/initscripts-1.0/, then I have to remember that it 
is there.  As this target may never be in the mainstream OE database, I 
keep the uniqueness of my changes in a seperate bb filetree.  The 
local.conf names the BBFILES location of both the mainstream OE tree and 
my bb tree.

This works, if I copy, in total, the packages/initscripts directory into 
my bb tree, then do the changes inside the copy.  I now know where my 
changes are and can distribute them without having people invasively 
patch the OE tree.  Where this breaks down is the in the "PR" variable 
(PR = "r88").  Bitbake will ignore my initscripts-1.0.bb totally, if the 
OE initscripts has a higher "PR" number than my local copy!

The solution that I have done was to set my copy of initscripts-1.0.bb 
to have a PR="r9999", this forces bitbake to always defer to my copy of 
the bb file and ignore the copy in the OE tree.

Develing further into this, as it would seem to me that while this is a 
solution, it is not "elegant" nor is it preferred.  When others make 
changes to packages, such as initscripts, these changes would not be 
propogated into onto my target.  With forcing bitbake to ONLY use my 
copy, changes are not propogated.

It would seem to me that something like this would work, but it doesn't:

BBFILES/initscripts/files/MACHINE_NAME/device_table (IOW: 
zipitbbfiles/initscripts/files/zipit/device_table  would override 
org.openembedded.dev/packages/initscripts-1.0/files/device_table)

That having named 'zipitbbfiles' in local.conf as part of BBFILES, and, 
naming the package 'initscripts' with a dir of 'files'.  That bitbake 
would logically flow with its search for 
'files/${MACHINE_NAME}/device_table' when choosing 'device_table' ???

You would think so, but it doesn't.  The presence of the secondary 
initscripts directory is ignored.  This may be how it should be as there 
is no overriding "PR" value to cause bitbake to vote-off the OE default 
copy.

Do you see what I am trying to do here?




So, the madness continues with this.  Ok, I'll try to use the 
PREFERRED_VERSION_initscripts = "1.0+zipit" and rename my copy of 
initscripts_1.0.bb to initscripts_1.0+zipit.bb.  That does work...  I 
don't need to edit the bb file and set PR="r9999" anymore.

However, this breaks down as well.  There are bb files inside the OE 
tree that do not have a version!  For example: 
org.openembedded.dev/packages/modutils/modutils-collateral.bb is not 
versioned.

Pulling my hair out at this point.  I have reverted back to PR="r9999" 
to force the bitbake version election.  It looks sloppy though, isn't 
there an easier, cleaner way to keep ${MACHINE_NAME} specific files in a 
seperate tree, or, am I resigned to patching the OE tree whenever I want 
to create a working database for my target?




The major issue, as I see it comes down to changes.  If I put my changes 
into the local copy of the OE tree, how do I archive those changes and 
pass them onto someone else?  I could gain write access to the OE tree, 
but, this target should not (or would not) ever be that widespread to 
require it to be added to the repository.

Otherwise, I will still have to set PR to a fantastic value and keep the 
bb files isolated, or use a tarball to patch whatever OE tree that 
someone installs onto their system?



Regards,

TomW





-- 
Tom Walsh - WN3L - Embedded Systems Consultant
http://openhardware.net http://cyberiansoftware.com http://openzipit.org
"Windows? No thanks, I have work to do..."
----------------------------------------------------

Re: versioning madness

[Marcin Juszkiewicz]

I think that you should ignore fact that your machine is not wide used and 
submit your changes into OE. For example I do not know does anyone other 
then me use ProGear support in OE but this does not stopped me from 
adding it.

You can also create own branch in monotone database where you will 
cherrypick needed changes from .dev tree (or even use .dev tree locally 
with your changes commited into database).

Other way is bb collections where you set which tree is more important 
then others. But here I can't tell how it handle misc PR in both trees.

You can also use PR ="s88" in your tree as "s" > "r" and use same numbers 
as OE ones to get info how synced you are.

There are many methods - I would choose first one.

-- 
JID: hrw-jabber.org
OpenEmbedded developer/consultant

               Don't personalize computers. They hate that.

Re: versioning madness

[Cliff Brake]

On 4/2/07, Tom Walsh <tom@openhardware.net> wrote:
> I would like to know how others are handling this problem.   As you
> already know, versioning can be "interesting" with OE.  It is probably
> how I have approached the issue of keeping my list of files seperate
> from the OE bb filetree.
>
> In my project, I have a number of packages which needed specific changes
> to files so that they reflected the needs of the target.  A simple
> example is that of a list of /dev/devices.  initscripts controls the
> device creation through the device_table list.  So, I have a
> device_table file unique to my target.   No problem.
>
> What is a problem is controlling the changes that I make to the OE
> packages tree.  If I simply create a ${MACHINE_NAME} directory under
> packages/initscripts/initscripts-1.0/, then I have to remember that it
> is there.  As this target may never be in the mainstream OE database, I
> keep the uniqueness of my changes in a seperate bb filetree.  The
> local.conf names the BBFILES location of both the mainstream OE tree and
> my bb tree.
>
> This works, if I copy, in total, the packages/initscripts directory into
> my bb tree, then do the changes inside the copy.  I now know where my
> changes are and can distribute them without having people invasively
> patch the OE tree.  Where this breaks down is the in the "PR" variable
> (PR = "r88").  Bitbake will ignore my initscripts-1.0.bb totally, if the
> OE initscripts has a higher "PR" number than my local copy!

The solution is bbcollections.  If you set up bbcollections properly,
the package from the high priority tree will be used, even if the PR
is greater in the upstream tree.  I just tested this as I have a local
copy of tslib in my custom tree and it works.  I have been using
bbcollections for a number of projects and it works very well.  I can
check all my OE changes into SVN or whatever is used for the rest of
the project, and still track the upstream OE tree as needed.

BBFILES := "${TOPDIR}/openembedded/packages/*/*.bb
${TOPDIR}/openembedded.custom/packages/*/*.bb"
BBFILE_COLLECTIONS = "upstream custom"
BBFILE_PATTERN_upstream = "^${TOPDIR}/openembedded/packages/"
BBFILE_PATTERN_custom = "^${TOPDIR}/openembedded.custom/packages"
BBFILE_PRIORITY_upstream = "5"
BBFILE_PRIORITY_custom = "10"

Does this work for you?

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: versioning madness

[Leon Woestenberg]

Hello Tom, Cliff,

On 4/5/07, Cliff Brake <cliff.brake@gmail.com> wrote:
> On 4/2/07, Tom Walsh <tom@openhardware.net> wrote:
> > I would like to know how others are handling this problem.   As you
> > ...
> The solution is bbcollections.  If you set up bbcollections properly,
> ...
>
I am using the same method as Cliff and this works very well for me. I
put my private tree in a CVS repository. (But I try to commit as much
generic stuff into mainline).

One remaining issue for me is that I now have two seperate version
controlled repositories and I need a way to keep them interoperable.
This is not hard to do for the HEADs, but I have people working with
older branches or snapshots so there is a pretty amount of overhead
involved in keeping track of breaking changes.

Regards,

Leon.

Re: versioning madness

[Tom Walsh]

Leon Woestenberg wrote:
> Hello Tom, Cliff,
>
> On 4/5/07, Cliff Brake <cliff.brake@gmail.com> wrote:
>   
>> On 4/2/07, Tom Walsh <tom@openhardware.net> wrote:
>>     
>>> I would like to know how others are handling this problem.   As you
>>> ...
>>>       
>> The solution is bbcollections.  If you set up bbcollections properly,
>> ...
>>
>>     
> I am using the same method as Cliff and this works very well for me. I
> put my private tree in a CVS repository. (But I try to commit as much
> generic stuff into mainline).
>
>   

AH!  You know, I use svn regularly and I didn't even think about using 
it against the OE tree!  Thanks, that would be the best solution for me.

I have two projects using OE, one is something that can be public, the 
other is for a client.  The svn solution should work for me well in both 
cases.

Regards,

TomW



-- 
Tom Walsh - WN3L - Embedded Systems Consultant
http://openhardware.net http://cyberiansoftware.com http://openzipit.org
"Windows? No thanks, I have work to do..."
----------------------------------------------------

Re: versioning madness

[Tom Walsh]

Tom Walsh wrote:
> Leon Woestenberg wrote:
>   
>> Hello Tom, Cliff,
>>
>> On 4/5/07, Cliff Brake <cliff.brake@gmail.com> wrote:
>>   
>>     
>>> On 4/2/07, Tom Walsh <tom@openhardware.net> wrote:
>>>     
>>>       
>>>> I would like to know how others are handling this problem.   As you
>>>> ...
>>>>       
>>>>         
>>> The solution is bbcollections.  If you set up bbcollections properly,
>>> ...
>>>
>>>     
>>>       
>> I am using the same method as Cliff and this works very well for me. I
>> put my private tree in a CVS repository. (But I try to commit as much
>> generic stuff into mainline).
>>
>>   
>>     
>
> AH!  You know, I use svn regularly and I didn't even think about using 
> it against the OE tree!  Thanks, that would be the best solution for me.
>
>   
Nope, didn't work as well as I had hoped.  Both monotone and subversion 
bit^H^H^H complain about having an existing directory, so, I cannot 
"overlay" into the OE tree with either.

I'll just have to do it the old-fashioned way and edit the stuff with vi.

TomW



-- 
Tom Walsh - WN3L - Embedded Systems Consultant
http://openhardware.net http://cyberiansoftware.com http://openzipit.org
"Windows? No thanks, I have work to do..."
----------------------------------------------------

Re: versioning madness

[Cliff Brake]

On 4/9/07, Tom Walsh <tom@openhardware.net> wrote:
> Tom Walsh wrote:
> > AH!  You know, I use svn regularly and I didn't even think about using
> > it against the OE tree!  Thanks, that would be the best solution for me.
> >
> >
> Nope, didn't work as well as I had hoped.  Both monotone and subversion
> bit^H^H^H complain about having an existing directory, so, I cannot
> "overlay" into the OE tree with either.
>
> I'll just have to do it the old-fashioned way and edit the stuff with vi.

I don't understand?  I have two trees that are completely separate
from each other:

openembedded (HEAD of OE managed with monotone)
openembedded.custom (custom stuff managed with SVN)

The beauty of this setup is the trees do not interfere with each other.

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: versioning madness

[Tom Walsh]

Cliff Brake wrote:
> On 4/9/07, Tom Walsh <tom@openhardware.net> wrote:
>   
>> Tom Walsh wrote:
>>     
>>> AH!  You know, I use svn regularly and I didn't even think about using
>>> it against the OE tree!  Thanks, that would be the best solution for me.
>>>
>>>
>>>       
>> Nope, didn't work as well as I had hoped.  Both monotone and subversion
>> bit^H^H^H complain about having an existing directory, so, I cannot
>> "overlay" into the OE tree with either.
>>
>> I'll just have to do it the old-fashioned way and edit the stuff with vi.
>>     
>
> I don't understand?  I have two trees that are completely separate
> from each other:
>
> openembedded (HEAD of OE managed with monotone)
> openembedded.custom (custom stuff managed with SVN)
>
> The beauty of this setup is the trees do not interfere with each other.
>
>   
True, I was doing it that way, having two source trees.  That works fine 
if your recipes are for something that is not already in the OE tree.  
However, it becomes problematic where you have the same recipe in both 
trees.

For example, sysvinit.  I have changes to sysvinit that I want to make.  
So, you can do that only one of two ways:

1. create a subdir under the OE tree as 
'$projroot/org.openembedded.dev/packages/sysvinit/sysvinit/zipit', where 
'zipit' is the name of my MACHINE.  Copy initab into the 'zipit' dir, 
then edit your changes.  Done.

2. duplicate the recipe for sysvinit into the private tree 
($projroot/zipitbbfiles/sysvinit).  Do the same here with creating the 
new MACHINE dir ($projroot/zipitbbfiles/sysvinit/sysvinit/zipit/), copy 
& edit inittab as before.

When the two recipe trees are specified within 
$projdir/build/conf/local.conf as BBFILES := 
"$projdir/zipitbbfiles/*/*.bb 
$projdir/org.openembedded.dev/packages/*/*.bb", still no problem.

The problem comes up later when someone edits the recipe for 
$projroot/org.openembedded.dev/packages/sysvinit/sysvinit_2.86.bb and 
changes the PR = "rX" to a value larger than the one contained inside 
$projdir/zipitbbfiles/sysvinit_2.86.bb.  Guess what happens now?  The 
undesired recipe now takes precedence over the private recipe copy as PR 
gets asserted in the numerical version vote!

Do you see the problems? 

A) The problem is that my recipe would be outvoted in the second case 
structure of a private tree having a duplicate recipe.

B) There appears no convenient way to manage the addition of an 
extension to a recipe that is kept inside the OE tree (without commiting 
that into the main OE repository).  That is the problem with the first 
case example.


I guess that the issue(s) are not so much "versioning" but "source 
control" as the PV version voting borks private copies.  And, a code 
versioning system like subversion (or another copy of monotone, perhaps) 
is unable to do source control within the org.openembedded.dev OE main 
tree copy.

That's what I would like to resolve.  I could commit my board changes 
into the OE repository, but, as it is a proprietary hardware design, I 
doubt that anyone would encounter one as surplus.

There are some more issues, but, I'll stop here.  heh.

Regards,

TomW


> Cliff
>
>   


-- 
Tom Walsh - WN3L - Embedded Systems Consultant
http://openhardware.net http://cyberiansoftware.com http://openzipit.org
"Windows? No thanks, I have work to do..."
----------------------------------------------------

Re: versioning madness

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 3675 bytes --]

Tom,

Have you looked at bb collections?

http://bitbake.berlios.de/manual/ch04s02.html#id870544

I think they will do what you want. You can keep separate data and chose 
which data takes priority in case of duplicate packages.

Philip

Tom Walsh wrote:
> Cliff Brake wrote:
>> On 4/9/07, Tom Walsh <tom@openhardware.net> wrote:
>>   
>>> Tom Walsh wrote:
>>>     
>>>> AH!  You know, I use svn regularly and I didn't even think about using
>>>> it against the OE tree!  Thanks, that would be the best solution for me.
>>>>
>>>>
>>>>       
>>> Nope, didn't work as well as I had hoped.  Both monotone and subversion
>>> bit^H^H^H complain about having an existing directory, so, I cannot
>>> "overlay" into the OE tree with either.
>>>
>>> I'll just have to do it the old-fashioned way and edit the stuff with vi.
>>>     
>> I don't understand?  I have two trees that are completely separate
>> from each other:
>>
>> openembedded (HEAD of OE managed with monotone)
>> openembedded.custom (custom stuff managed with SVN)
>>
>> The beauty of this setup is the trees do not interfere with each other.
>>
>>   
> True, I was doing it that way, having two source trees.  That works fine 
> if your recipes are for something that is not already in the OE tree.  
> However, it becomes problematic where you have the same recipe in both 
> trees.
> 
> For example, sysvinit.  I have changes to sysvinit that I want to make.  
> So, you can do that only one of two ways:
> 
> 1. create a subdir under the OE tree as 
> '$projroot/org.openembedded.dev/packages/sysvinit/sysvinit/zipit', where 
> 'zipit' is the name of my MACHINE.  Copy initab into the 'zipit' dir, 
> then edit your changes.  Done.
> 
> 2. duplicate the recipe for sysvinit into the private tree 
> ($projroot/zipitbbfiles/sysvinit).  Do the same here with creating the 
> new MACHINE dir ($projroot/zipitbbfiles/sysvinit/sysvinit/zipit/), copy 
> & edit inittab as before.
> 
> When the two recipe trees are specified within 
> $projdir/build/conf/local.conf as BBFILES := 
> "$projdir/zipitbbfiles/*/*.bb 
> $projdir/org.openembedded.dev/packages/*/*.bb", still no problem.
> 
> The problem comes up later when someone edits the recipe for 
> $projroot/org.openembedded.dev/packages/sysvinit/sysvinit_2.86.bb and 
> changes the PR = "rX" to a value larger than the one contained inside 
> $projdir/zipitbbfiles/sysvinit_2.86.bb.  Guess what happens now?  The 
> undesired recipe now takes precedence over the private recipe copy as PR 
> gets asserted in the numerical version vote!
> 
> Do you see the problems? 
> 
> A) The problem is that my recipe would be outvoted in the second case 
> structure of a private tree having a duplicate recipe.
> 
> B) There appears no convenient way to manage the addition of an 
> extension to a recipe that is kept inside the OE tree (without commiting 
> that into the main OE repository).  That is the problem with the first 
> case example.
> 
> 
> I guess that the issue(s) are not so much "versioning" but "source 
> control" as the PV version voting borks private copies.  And, a code 
> versioning system like subversion (or another copy of monotone, perhaps) 
> is unable to do source control within the org.openembedded.dev OE main 
> tree copy.
> 
> That's what I would like to resolve.  I could commit my board changes 
> into the OE repository, but, as it is a proprietary hardware design, I 
> doubt that anyone would encounter one as surplus.
> 
> There are some more issues, but, I'll stop here.  heh.
> 
> Regards,
> 
> TomW
> 
> 
>> Cliff
>>
>>   
> 
> 

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: versioning madness

[Cliff Brake]

On 4/9/07, Tom Walsh <tom@openhardware.net> wrote:

> When the two recipe trees are specified within
> $projdir/build/conf/local.conf as BBFILES :=
> "$projdir/zipitbbfiles/*/*.bb
> $projdir/org.openembedded.dev/packages/*/*.bb", still no problem.
>
> The problem comes up later when someone edits the recipe for
> $projroot/org.openembedded.dev/packages/sysvinit/sysvinit_2.86.bb and
> changes the PR = "rX" to a value larger than the one contained inside
> $projdir/zipitbbfiles/sysvinit_2.86.bb.  Guess what happens now?  The
> undesired recipe now takes precedence over the private recipe copy as PR
> gets asserted in the numerical version vote!

Actually if you use bbcollections, and set the priority higher in the
zipitbbfiles, it will take precedence no matter what the PR is.  Try
something like the following in your local.conf -- replace
openembedded.custom w/ zipitbbfiles ....

BBFILES := "${TOPDIR}/openembedded/packages/*/*.bb
${TOPDIR}/openembedded.custom/packages/*/*.bb"
BBFILE_COLLECTIONS = "upstream custom"
BBFILE_PATTERN_upstream = "^${TOPDIR}/openembedded/packages/"
BBFILE_PATTERN_custom = "^${TOPDIR}/openembedded.custom/packages"
BBFILE_PRIORITY_upstream = "5"
BBFILE_PRIORITY_custom = "10"

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: versioning madness

[Tom Walsh]

Philip Balister wrote:
> Tom,
>
> Have you looked at bb collections?
>
> http://bitbake.berlios.de/manual/ch04s02.html#id870544
>
> I think they will do what you want. You can keep separate data and 
> chose which data takes priority in case of duplicate packages.
>

AH! Thank you!  That is exactly what I need.

TomW


-- 
Tom Walsh - WN3L - Embedded Systems Consultant
http://openhardware.net http://cyberiansoftware.com http://openzipit.org
"Windows? No thanks, I have work to do..."
----------------------------------------------------

Re: versioning madness

[Leon Woestenberg]

Tom,

On 4/11/07, Tom Walsh <tom@openhardware.net> wrote:
> Philip Balister wrote:
> > Tom,
> >
> > Have you looked at bb collections?
> >
> > http://bitbake.berlios.de/manual/ch04s02.html#id870544
> >
> > I think they will do what you want. You can keep separate data and
> > chose which data takes priority in case of duplicate packages.
> >
>
> AH! Thank you!  That is exactly what I need.
>
Yep, bitbake collections to the rescue.

Note that you should carefully make sure you have no "Multiple
providers" and "Consider specifying PREFERRED_PROVIDER" messages left,
as they seem to work undeterministic in that case when using bitbake
collections.

If you specify a PREFERRED_PROVIDER for each such package, it works brilliantly.

Regards,

Leon



-- 
Leon