Overriding package config files in OE

Overriding package config files in OE

[shanevolpe]

I  have quite a few config files I need to override with my own
customized files.  From what I have seen the most common way to do
this is to set up a local collection and use it in an overlay fashion
copying all the packages you wish to modify a config file for into the
local collection.
 I do not like the above solution for several reasons, its a
documentation challenge to keep track of what packages you just
changed config files and what packages you may have added patches or
changed other things (compile/linker/EXTRA_OECONF).  It also presents
a challenge when you want to merge back into a later version of OE.
It also forces you to make duplicate machines with different names and
add in machine directories to the package directory containing the
unique configurations if you have the same hardware that may have
different options (configurations).  Also if you wish to field upgrade
a machine from one version to another (configuration) it would be, I
think, a challenge with the above method.

Since I'm not satisfied with the overlay option I have worked on a
different approach and is the real reason I'm posting this email as I
would like to get feedback, comments  and criticism to my approach,
please keep in mind that I do realize this approach is STILL a hack
but it is what I have been able to come up with in a short period of
time and currently I believe it to be more scalable than the overlay
and also easier to manage and document.

I made a separate package containing all my custom config files, I
will refer to it as configs.bb.  I did NOT flag them as config files
(CONFFILES).

I was then able to make an image.bb file as follows:
##############################################################
EXTRA_IMAGEDEPENDS = "configs"
#RDEPENDS = ${PACKAGE_INSTALL}
RDEPENDS = "task-base"

PACKAGE_INSTALL = "${RDEPENDS} ${EXTRA_IMAGEDEPENDS}"
LICENSE = "LGPL"
PR = "r0"
IMAGE_BASENAME = "my-image"
IMAGE_ROOTFS = "${TMPDIR}/rootfs_myimage"

inherit image
##############################################################

This seems to force my configs.bb to be installed into the rootfs
after all the other packages have been installed and since I did not
specify the files in configs.bb as configuration  files it overrides
all the config files of the already installed packages with out
causing ipkg to spit out questions on how to handle the config file
(update, keep current version, show diffs) and hanging up the image
building.

Also with this method if you do need to update config.bb you just need
to set -force-overwrite and if you update one of the packages you
overwrote a config file to, ipkg should prompt you with what you want
to do with the config file.

Regards,
Shane

Re: Overriding package config files in OE

[Cliff Brake]

The issue described below is a very real problem for anyone doing
one-off embedded work with OE where it does not make sense to push the
changes into the main OE repository.  The basic requirement is to be
able to localize config file changes for a build without duplicating
entire packages in a bbcollections tree, or maintaining a separate OE
mtn branch for each project.

I'm not complaining as OE is clearly the _best_ way to build embedded
Linux systems :-), but this is one area where OE could be improved to
appeal to more embedded developers.

Another approach I've been thinking about is to figure out a way for
bitbake to use <files> from a tree outside the main OE tree.  One
possibility is for bitbake to pull <files> as well as bb files from a
high priority bbcollections tree, even though the bb remains in the
main OE tree.  This probably totally breaks the way bitbake works, but
...

Cliff

On 11/1/07, shanevolpe@gmail.com <shanevolpe@gmail.com> wrote:
> I  have quite a few config files I need to override with my own
> customized files.  From what I have seen the most common way to do
> this is to set up a local collection and use it in an overlay fashion
> copying all the packages you wish to modify a config file for into the
> local collection.
>  I do not like the above solution for several reasons, its a
> documentation challenge to keep track of what packages you just
> changed config files and what packages you may have added patches or
> changed other things (compile/linker/EXTRA_OECONF).  It also presents
> a challenge when you want to merge back into a later version of OE.
> It also forces you to make duplicate machines with different names and
> add in machine directories to the package directory containing the
> unique configurations if you have the same hardware that may have
> different options (configurations).  Also if you wish to field upgrade
> a machine from one version to another (configuration) it would be, I
> think, a challenge with the above method.
>
> Since I'm not satisfied with the overlay option I have worked on a
> different approach and is the real reason I'm posting this email as I
> would like to get feedback, comments  and criticism to my approach,
> please keep in mind that I do realize this approach is STILL a hack
> but it is what I have been able to come up with in a short period of
> time and currently I believe it to be more scalable than the overlay
> and also easier to manage and document.
>
> I made a separate package containing all my custom config files, I
> will refer to it as configs.bb.  I did NOT flag them as config files
> (CONFFILES).
>
> I was then able to make an image.bb file as follows:
> ##############################################################
> EXTRA_IMAGEDEPENDS = "configs"
> #RDEPENDS = ${PACKAGE_INSTALL}
> RDEPENDS = "task-base"
>
> PACKAGE_INSTALL = "${RDEPENDS} ${EXTRA_IMAGEDEPENDS}"
> LICENSE = "LGPL"
> PR = "r0"
> IMAGE_BASENAME = "my-image"
> IMAGE_ROOTFS = "${TMPDIR}/rootfs_myimage"
>
> inherit image
> ##############################################################
>
> This seems to force my configs.bb to be installed into the rootfs
> after all the other packages have been installed and since I did not
> specify the files in configs.bb as configuration  files it overrides
> all the config files of the already installed packages with out
> causing ipkg to spit out questions on how to handle the config file
> (update, keep current version, show diffs) and hanging up the image
> building.
>
> Also with this method if you do need to update config.bb you just need
> to set -force-overwrite and if you update one of the packages you
> overwrote a config file to, ipkg should prompt you with what you want
> to do with the config file.
>
> Regards,
> Shane
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>


-- 
=======================
Cliff Brake
http://bec-systems.com

Re: Overriding package config files in OE

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

Cliff Brake wrote:
> Another approach I've been thinking about is to figure out a way for
> bitbake to use <files> from a tree outside the main OE tree.  One
> possibility is for bitbake to pull <files> as well as bb files from a
> high priority bbcollections tree, even though the bb remains in the
> main OE tree.  This probably totally breaks the way bitbake works, but

I think that having an overlay set up as a high priority collection does 
work. Steve S. uses one for his gumstix project. I like this method 
because I can easily find all the local changes he makes and see if they 
need to move back into .dev.

A local overlay lets you change the version of .dev underneath as 
needed, without moving changes across branches. Combined with sane 
source rev/dates, I believe this gives the people supporting product a 
sane build/support environment.

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: Overriding package config files in OE

[Cliff Brake]

On 11/2/07, Philip Balister <philip@balister.org> wrote:
> Cliff Brake wrote:
> > Another approach I've been thinking about is to figure out a way for
> > bitbake to use <files> from a tree outside the main OE tree.  One
> > possibility is for bitbake to pull <files> as well as bb files from a
> > high priority bbcollections tree, even though the bb remains in the
> > main OE tree.  This probably totally breaks the way bitbake works, but
>
> I think that having an overlay set up as a high priority collection does
> work.

I was referring to the case where I only put the files directory in
the overlay -- not the entire recipe directory.  I have not tried this
lately and will do so the next time I get a chance.

> Steve S. uses one for his gumstix project. I like this method
> because I can easily find all the local changes he makes and see if they
> need to move back into .dev.
>
> A local overlay lets you change the version of .dev underneath as
> needed, without moving changes across branches. Combined with sane
> source rev/dates, I believe this gives the people supporting product a
> sane build/support environment.

Agreed.  I use overlays a lot as well and they work great -- I would
just like to avoid having to copy the entire recipe + the files
directory -- for example just to make a small change in
/etc/network/interfaces.  If you copy the entire recipe directory,
diff'ing is easy initially, but several months down the road it is
very difficult to figure out what you changed.

Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: Overriding package config files in OE

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Cliff Brake schreef:
> On 11/2/07, Philip Balister <philip@balister.org> wrote:
>> Cliff Brake wrote:
>>> Another approach I've been thinking about is to figure out a way for
>>> bitbake to use <files> from a tree outside the main OE tree.  One
>>> possibility is for bitbake to pull <files> as well as bb files from a
>>> high priority bbcollections tree, even though the bb remains in the
>>> main OE tree.  This probably totally breaks the way bitbake works, but
>> I think that having an overlay set up as a high priority collection does
>> work.
> 
> I was referring to the case where I only put the files directory in
> the overlay -- not the entire recipe directory.  I have not tried this
> lately and will do so the next time I get a chance.
> 
>> Steve S. uses one for his gumstix project. I like this method
>> because I can easily find all the local changes he makes and see if they
>> need to move back into .dev.
>>
>> A local overlay lets you change the version of .dev underneath as
>> needed, without moving changes across branches. Combined with sane
>> source rev/dates, I believe this gives the people supporting product a
>> sane build/support environment.
> 
> Agreed.  I use overlays a lot as well and they work great -- I would
> just like to avoid having to copy the entire recipe + the files
> directory -- for example just to make a small change in
> /etc/network/interfaces.  If you copy the entire recipe directory,
> diff'ing is easy initially, but several months down the road it is
> very difficult to figure out what you changed.

You could also do it the openmoko way by defining extra overrides and
using those in FILESPATH.


regards,

Koen

- --
koen@dominion.kabel.utwente.nl will go go away in december 2007, please
use k.kooi@student.utwente.nl instead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFHK1FAMkyGM64RGpERAuFpAJ44e/KObKpHYwvmXrLyqNy4o+9VDgCfRUQk
syve1LMsa+yXbjUdrE4NfsQ=
=eU2C
-----END PGP SIGNATURE-----

Re: Overriding package config files in OE

[Leon Woestenberg]

Hello,


Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Cliff Brake schreef:
>   
>> On 11/2/07, Philip Balister <philip@balister.org> wrote:
>>     
>>> Cliff Brake wrote:
>>>       
>>>> Another approach I've been thinking about is to figure out a way for
>>>> bitbake to use <files> from a tree outside the main OE tree.  One
>>>> possibility is for bitbake to pull <files> as well as bb files from a
>>>> high priority bbcollections tree, even though the bb remains in the
>>>> main OE tree.  This probably totally breaks the way bitbake works, but
>>>>         
>>> I think that having an overlay set up as a high priority collection does
>>> work.
>>>       
>> I was referring to the case where I only put the files directory in
>> the overlay -- not the entire recipe directory.  I have not tried this
>> lately and will do so the next time I get a chance.
>>
>>     
Only copying files, and not the recipes, to an overlay, does not work.

It would be quite nice, but I understand the complexities if every FILE 
has to be
search for in all the collections.

The current behaviour is, as I understand:

If no PREFERED_VERSION, select highest version of a package, else 
PREFERED_VERSION.
Iterate through collections by priority, select the collection that 
provides the "package-version".
Use that collection as the base directory for accessing FILES.

> You could also do it the openmoko way by defining extra overrides and
> using those in FILESPATH.
>   
Thanks, will look into this as well,

Regards,

Leon Woestenberg.