[Buildroot] Project layout : where to put the .config files

[Buildroot] Project layout : where to put the .config files

[Jeremy Rosen]

Hello everybody

Buildroot is currently in the process of cleanly separating files
that are generated by buildroot and are throwaway files (in the 
output directory) from files that are configuration files (the
BR2_EXTERNAL directory) 

However the .config files used by buildroot and other kconfig
enabled software are not dealt with in a way that is consistant.

.config is the main buildroot configuration file. This is, more
than anything else "the project." It is the file we want to 
keep in git, share within a software project, be warned in 
case of conflict. However it is currently savedin the output 
directory. 

I am sending this mail to understand how the community see this
file, if there is something that needs to be changed to follow
the new separation and see what I can do to get this going.

(I will focus only in the buildroot .config file, the kernel,
busybox and uclibc ones can be worked out after that)

so, I have been looking around and there are a couple of
approches I can think of...


* keep the current approch

.config is considered a throw away file, it is logical that 
it is in output/ and it is normal that it is destroyed on a
regular basis.

I don't like this too much because, for a final product, it 
is the most important file, and a file that needs to be 
kept in git. I don't think it is very logical to have such
an important file be a hidden file and be in a directory that
is meant to be removed.

* keep the file as $(CONFIG_DIR)/.config, but document that
CONFIG_DIR can be overridden

This keeps the file as a hidden file, but it allows us to 
move it out of the output directory. Maybe CONFIG_DIR should
default to BR2_EXTERNAL instead of output/ (with some 
backward compatibility layer)

* allow to override the configuration file with env variables

Basically allow overriding BUILDROOT_CONFIG from the command
line. This would solve all the problems I have but I am not 
sure it follows the buildroot philosophy, thus the long mail.



This could be a subject for the buildroot days. If other people
are interested I can add it to the list of topics...


(note that I tried moving the kernel/busybox .config and it's 
more tricky than the buildroot one... maybe we need a generic
infrastructure for kconfig enabled packages...)


    Regards

    J?r?my Rosen

fight key loggers : write some perl using vim


Open Wide Ingenierie

23, rue Daviel
75013 Paris - France
www.openwide.fr

[Buildroot] Project layout : where to put the .config files

[Thomas De Schampheleire]

On Thu, Jan 30, 2014 at 10:53 AM, Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:
> Hello everybody
>
> Buildroot is currently in the process of cleanly separating files
> that are generated by buildroot and are throwaway files (in the
> output directory) from files that are configuration files (the
> BR2_EXTERNAL directory)
>
> However the .config files used by buildroot and other kconfig
> enabled software are not dealt with in a way that is consistant.
>
> .config is the main buildroot configuration file. This is, more
> than anything else "the project." It is the file we want to
> keep in git, share within a software project, be warned in
> case of conflict. However it is currently savedin the output
> directory.
>
> I am sending this mail to understand how the community see this
> file, if there is something that needs to be changed to follow
> the new separation and see what I can do to get this going.
>
> (I will focus only in the buildroot .config file, the kernel,
> busybox and uclibc ones can be worked out after that)
>
> so, I have been looking around and there are a couple of
> approches I can think of...
>
>
> * keep the current approch
>
> .config is considered a throw away file, it is logical that
> it is in output/ and it is normal that it is destroyed on a
> regular basis.
>
> I don't like this too much because, for a final product, it
> is the most important file, and a file that needs to be
> kept in git. I don't think it is very logical to have such
> an important file be a hidden file and be in a directory that
> is meant to be removed.
>
> * keep the file as $(CONFIG_DIR)/.config, but document that
> CONFIG_DIR can be overridden
>
> This keeps the file as a hidden file, but it allows us to
> move it out of the output directory. Maybe CONFIG_DIR should
> default to BR2_EXTERNAL instead of output/ (with some
> backward compatibility layer)
>
> * allow to override the configuration file with env variables
>
> Basically allow overriding BUILDROOT_CONFIG from the command
> line. This would solve all the problems I have but I am not
> sure it follows the buildroot philosophy, thus the long mail.
>
>
>
> This could be a subject for the buildroot days. If other people
> are interested I can add it to the list of topics...
>
>
> (note that I tried moving the kernel/busybox .config and it's
> more tricky than the buildroot one... maybe we need a generic
> infrastructure for kconfig enabled packages...)
>
>

I find it a strange strategy to put .config in git: this is why we
have the configs/ directory, and a target 'make savedefconfig' to
store the .config configuration in a defconfig file. If you don't like
defconfigs, you can still copy .config directly in the configs/
subdirectory.

Best regards,
Thomas

[Buildroot] Project layout : where to put the .config files

[Jeremy Rosen]

Thomas, All

> 
> I find it a strange strategy to put .config in git: this is why we
> have the configs/ directory, and a target 'make savedefconfig' to
> store the .config configuration in a defconfig file. If you don't
> like
> defconfigs, you can still copy .config directly in the configs/
> subdirectory.
> 
> Best regards,
> Thomas
> 

This is my own little project I am building here it is not a new
board I am trying to support. I have been bitten multiple times 
by .config not being saved until I do a savedefconfig. savedefconfig
should not be a "my change are good let's validate" step. That's 
what git commit is for.

Defconfigs are a great way to define templates for a board and
distribute them with buildroot, but when you are building a final
product they are not the right tool. 

In particular the fact that you have to do a savedefconfig before 
any git commit is very error prone. moreover savedefconfig won't
warn you if defconfig has been modified by a pull and will overwrite
it.

This is even worse for the kernel/busybox case where the config
files are hidden in output/build/xxx/.config

[Buildroot] Project layout : where to put the .config files

[Thomas De Schampheleire]

Hi Jeremy,

On Thu, Jan 30, 2014 at 11:42 AM, Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:
> Thomas, All
>
>>
>> I find it a strange strategy to put .config in git: this is why we
>> have the configs/ directory, and a target 'make savedefconfig' to
>> store the .config configuration in a defconfig file. If you don't
>> like
>> defconfigs, you can still copy .config directly in the configs/
>> subdirectory.
>>
>> Best regards,
>> Thomas
>>
>
> This is my own little project I am building here it is not a new
> board I am trying to support. I have been bitten multiple times
> by .config not being saved until I do a savedefconfig. savedefconfig
> should not be a "my change are good let's validate" step. That's
> what git commit is for.
>
> Defconfigs are a great way to define templates for a board and
> distribute them with buildroot, but when you are building a final
> product they are not the right tool.

This is probably personal, but I don't agree. The configs directory is
not limited to 'templates'. This may be the upstream strategy, but in
the repository for a particular project you are free to add one or
more full configs for that project.

>
> In particular the fact that you have to do a savedefconfig before
> any git commit is very error prone. moreover savedefconfig won't
> warn you if defconfig has been modified by a pull and will overwrite
> it.
>
> This is even worse for the kernel/busybox case where the config
> files are hidden in output/build/xxx/.config

You you could make a mistake, but in my experience mistakes the other
way around also happen: you accidentally commit a change to .config
that was just for test. This may be fine for your personal repo, but
in a work environment it's not very nice at all, especially if this is
realized only much later.

You could create commit hooks to verify that the .config files are in
line with the actual version-controlled versions, I guess.

Note again: this is just my opinion...

Best regards,
Thomas

[Buildroot] Project layout : where to put the .config files

[Sagaert Johan]

 
Hi

Besides .config ,I think it could be good to also save the kernel .config and the .config files of busybox and uclibc when doing a
make savedefconfig.
Therefore I always keep the kernel, busybox and uclibc config files in my board directory and use the the path to these my buildroot
config files.

Regards Johan

-----Oorspronkelijk bericht-----
Van: buildroot-bounces at busybox.net [mailto:buildroot-bounces at busybox.net] Namens Thomas De Schampheleire
Verzonden: donderdag 30 januari 2014 10:59
Aan: Jeremy Rosen
CC: buildroot
Onderwerp: Re: [Buildroot] Project layout : where to put the .config files

On Thu, Jan 30, 2014 at 10:53 AM, Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:
> Hello everybody
>
> Buildroot is currently in the process of cleanly separating files that 
> are generated by buildroot and are throwaway files (in the output 
> directory) from files that are configuration files (the BR2_EXTERNAL 
> directory)
>
> However the .config files used by buildroot and other kconfig enabled 
> software are not dealt with in a way that is consistant.
>
> .config is the main buildroot configuration file. This is, more than 
> anything else "the project." It is the file we want to keep in git, 
> share within a software project, be warned in case of conflict. 
> However it is currently savedin the output directory.
>
> I am sending this mail to understand how the community see this file, 
> if there is something that needs to be changed to follow the new 
> separation and see what I can do to get this going.
>
> (I will focus only in the buildroot .config file, the kernel, busybox 
> and uclibc ones can be worked out after that)
>
> so, I have been looking around and there are a couple of approches I 
> can think of...
>
>
> * keep the current approch
>
> .config is considered a throw away file, it is logical that it is in 
> output/ and it is normal that it is destroyed on a regular basis.
>
> I don't like this too much because, for a final product, it is the 
> most important file, and a file that needs to be kept in git. I don't 
> think it is very logical to have such an important file be a hidden 
> file and be in a directory that is meant to be removed.
>
> * keep the file as $(CONFIG_DIR)/.config, but document that CONFIG_DIR 
> can be overridden
>
> This keeps the file as a hidden file, but it allows us to move it out 
> of the output directory. Maybe CONFIG_DIR should default to 
> BR2_EXTERNAL instead of output/ (with some backward compatibility 
> layer)
>
> * allow to override the configuration file with env variables
>
> Basically allow overriding BUILDROOT_CONFIG from the command line. 
> This would solve all the problems I have but I am not sure it follows 
> the buildroot philosophy, thus the long mail.
>
>
>
> This could be a subject for the buildroot days. If other people are 
> interested I can add it to the list of topics...
>
>
> (note that I tried moving the kernel/busybox .config and it's more 
> tricky than the buildroot one... maybe we need a generic 
> infrastructure for kconfig enabled packages...)
>
>

I find it a strange strategy to put .config in git: this is why we have the configs/ directory, and a target 'make savedefconfig' to
store the .config configuration in a defconfig file. If you don't like defconfigs, you can still copy .config directly in the
configs/ subdirectory.

Best regards,
Thomas
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Project layout : where to put the .config files

[Mike Zick]

On Thu, 30 Jan 2014 10:53:58 +0100 (CET)
Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:

> Buildroot is currently in the process of cleanly separating files
> that are generated by buildroot and are throwaway files (in the 
> output directory) from files that are configuration files (the
> BR2_EXTERNAL directory) 
>

Are you sure of that?

The discussion that I have seen here is that BR2_EXTERNAL tree
is for the "proprietary" files, not just configuration files.

Note: 
Here "proprietary" is used as a generalization for the
portion of the overall build tree that is not required to be
made public.

And:
This list has already had the discussion over how the .config
files should be considered.
I.E: Are they some of the "required public" files to meet the
requirements of Section 3 of the GPLv2 license.

Mike

[Buildroot] Project layout : where to put the .config files

[Jeremy Rosen]

----- Mail original -----
> On Thu, 30 Jan 2014 10:53:58 +0100 (CET)
> Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:
> 
> > Buildroot is currently in the process of cleanly separating files
> > that are generated by buildroot and are throwaway files (in the
> > output directory) from files that are configuration files (the
> > BR2_EXTERNAL directory)
> >
> 
> Are you sure of that?
> 
> The discussion that I have seen here is that BR2_EXTERNAL tree
> is for the "proprietary" files, not just configuration files.
> 

That's was my understanding,yes...

> Note:
> Here "proprietary" is used as a generalization for the
> portion of the overall build tree that is not required to be
> made public.
> 

well, let's use another term. .config is not about having 
close source stuff in buildroot, it's about the configuration
of a project. The project I am building is here :
https://github.com/Openwide-Ingenierie/raspaudio
it's entirely free, but it's a livingroom audio station 
for my home, and buildroot only upstreams board 
defconfigs which are defined as the minimum setup to 
get the hardware to work correctly.

I am in a different use-case here, I am building a complete
appliance using buildroot. the defconfig approch doesn't 
really suit me and this as nothing to do with license.


> And:
> This list has already had the discussion over how the .config
> files should be considered.
> I.E: Are they some of the "required public" files to meet the
> requirements of Section 3 of the GPLv2 license.
> 

That's a legal issue, i'm discussing the organisational aspect.

how to organize a project that is based on build root but is not
a new board (in my case it's a raspberry-pi based firmware)

I am pretty certain that I'm a different use-case, one that is 
not properly documented. I'm trying it out and once I have found
a way to deal with it i'll submit any patch/doc changes needed.

I would gladly submit my projet to upstream too, but once again 
buildroot (so far) only takes minimal configuration to set up
boards, not complete projects.

> Mike
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] Project layout : where to put the .config files

[Mike Zick]

On Thu, 30 Jan 2014 13:34:31 +0100 (CET)
Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:

> I would gladly submit my projet to upstream too, but once again 
> buildroot (so far) only takes minimal configuration to set up
> boards, not complete projects.
>

Agreed on that point -

Buildroot is only a project component, one that builds a 
populated, root file system.

So far in the evolution of Buildroot, the other project
components required to fully define a project have been
user supplied.

It has never been a complete project build system.

Which is the use case that you want to consider.
One that certainly deserves conversation.

If my summary above is even close to an understanding
of the use case you are proposing, then - - -

Prior to your question of "where to put the .config files?",
I think there is a higher level question:
"How to support project definition files?"

I.E: To make Buildroot the central control point for the
other components that make up a complete project.

Perhaps a: BR_EXTERNAL/sub-tree ?
Perhaps a: BR_PROJECT tree ?

Keeping in mind that we do not want to upset the world
of users that use Buildroot commercially.

For instance:
The subject of "are the .config files 'required public' files?"
The current set-up leaves that answer to the end user with
commands that will include them in the 'public' buildroot tree.

Mike

[Buildroot] Project layout : where to put the .config files

[Jeremy Rosen]

> 
> > I would gladly submit my projet to upstream too, but once again
> > buildroot (so far) only takes minimal configuration to set up
> > boards, not complete projects.
> >
> 
> Agreed on that point -
> 
> Buildroot is only a project component, one that builds a
> populated, root file system.
> 
> So far in the evolution of Buildroot, the other project
> components required to fully define a project have been
> user supplied.
> 
> It has never been a complete project build system.
> 
> Which is the use case that you want to consider.
> One that certainly deserves conversation.

yup, I like the way you present it.

you can have a look at the project I mentionned above,
with post-install scripts and overlays I am able to 
make a complete project managed with buildroot, that's
why I focused on the .config aspect which is the part
I am not entirely happy with


> 
> If my summary above is even close to an understanding
> of the use case you are proposing, then - - -
> 
> Prior to your question of "where to put the .config files?",
> I think there is a higher level question:
> "How to support project definition files?"
> 

the Makefile in my project fills that role (iiuc 
what you mean) it is where I define where I want my buildfiles
to go (by defining O=) and where I want my config files to go
(by defining BR2_EXTERNAL and BR2_DEFCONFIG)

> I.E: To make Buildroot the central control point for the
> other components that make up a complete project.
> 
> Perhaps a: BR_EXTERNAL/sub-tree ?
> Perhaps a: BR_PROJECT tree ?
> 
> Keeping in mind that we do not want to upset the world
> of users that use Buildroot commercially.
> 

yes, but i'm pretty sure most of us would be glad to have a 
"clean way" to do a complete project under buildroot


> For instance:
> The subject of "are the .config files 'required public' files?"
> The current set-up leaves that answer to the end user with
> commands that will include them in the 'public' buildroot tree.
> 

I'm not sure what you mean here... I'm mainly thinking in term
of project organisation. if the .config is considered derived 
from buildroot then the exact position doesn't matter. I have
to make it public, and usually in another way than by upstreaming
since this is a complete project and not a board defconfig...


> Mike
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] Project layout : where to put the .config files

[Mike Zick]

On Thu, 30 Jan 2014 15:10:03 +0100 (CET)
Jeremy Rosen <jeremy.rosen@openwide.fr> wrote:

> > I.E: To make Buildroot the central control point for the
> > other components that make up a complete project.
> > 
> > Perhaps a: BR_EXTERNAL/sub-tree ?
> > Perhaps a: BR_PROJECT tree ?
> > 
> > Keeping in mind that we do not want to upset the world
> > of users that use Buildroot commercially.
> >   
> 
> yes, but i'm pretty sure most of us would be glad to have a 
> "clean way" to do a complete project under buildroot
> 

Ah, another point of agreement then.

The current "put them in 'public' tree and move if required"
and the current "put them in the 'private' tree and move if
required" is becoming a bit messy.

So without trying to answer my own questions above -

Start with adding some infra-structure for 'project related'
files; and
Then using that to support the collection of .config files.

That would be of general use to the Buildroot system and
consistent with the "keep it clean, keep it simple" objective.

The end-user could still decide on the 'ship/don't ship' question
by including the 'project related' stuff or not.

It could also help answer the question of: "BR provides hooks
to external scripting - but where do I put them in an organized
manner?"

> 
> > For instance:
> > The subject of "are the .config files 'required public' files?"
> > The current set-up leaves that answer to the end user with
> > commands that will include them in the 'public' buildroot tree.
> >   
> 
> I'm not sure what you mean here... I'm mainly thinking in term
> of project organisation. if the .config is considered derived 
> from buildroot then the exact position doesn't matter. I have
> to make it public, and usually in another way than by upstreaming
> since this is a complete project and not a board defconfig...
> 

There has already been (years ago) a Buildroot fork from those
that wanted it to become a complete project system.

So just providing the basic organizational infra-structure is
probably a good idea now.

- - - -

Let us pause here for others to comment.

Mike