[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

Hello everybody

For some time I have been looking into how to integrate buildroot in a project
where buildroot would be the engine to provide the filesystem building 
infrastructure but where the main repository would not be a clone of the 
buildroot project. The idea is to try to separate cleanly what is meant to
go upstream from what is not and still be able to use git to properly save
the data that is closely linked to the project

I have now reached a point where things "simply works" and I can show my work
to the bigger buildroot community.


I will first introduce the project itself, then i'll discuss how to create
your own project based on that infrastructure and then I'll discuss what
could be upstreamed from the project and how


So, with no further delay, let's introduce Raspaudio.

Raspaudio is a small project that builds a filesystem for the raspberry pi
which allows you to easily stream audio content from the pulseaudio server
on your desktop to the audio output of the raspberry pi. Compile the 
filesystem, plug the raspberry, configure the PC, you're done.

Details on these steps can be found at 

http://openwide-ingenierie.github.io/raspaudio/

I won't go further than that in the project, the idea is that this is a
"finished" buildroot project. you clone it, you compile it, you install
it.

   $ git clone --recursive git at github.com:Openwide-Ingenierie/raspaudio.git
   $ cd raspaudio
   $ make


HOW TO REUSE THE RASPAUDIO INFRASTRUCTURE

The whole point of raspaudio is to test the buildroot infrastructure and 
provide an example that is reusable for other projects. If you want to 
make a similar project, here is how to do it

  $ mkdir new_project
  $ cd new_project
  $ git init
  $ git submodule add git://git.buildroot.net/buildroot buildroot
  $ wget https://raw.github.com/Openwide-Ingenierie/raspaudio/master/Makefile
  $ touch Config.in
  $ touch external.mk
  $ echo "output/" > .gitignore
  $ echo "dl/" >> .gitignore

Change the field PROJECT_NAME in the Makefile to the name of your project

  $ git add Makefile external.mk Config.in .gitignore
  $ git commit -m"Project infrastructure"

You can now configure your project in the usual buildroot way. For example :

  $ make raspberrypi_defconfig
  $ make menuconfig

A file named $(PROJECT_NAME)_defconfig will appear. Add this project to git
You can now work with your project in the usual way and all the important
information will be saved in git.

I tried to make the whole thing as close to the buildroot way as possible, 
all Makefile targets should work properly and most infrasturcture variables
(DL_DIR, BR2_EXTERNAL...) are set for you. This is here to help you quickly 
have the infrastructure in place to deploy your own projects based on buildroot.


WHERE TO GO FROM HERE

The main thing I have noticed is that I have been developing post-build scripts
that are completely generic and could easily be reused. It would be nice to 
upstream these scripts (in particular, the script that automatically adds the 
public key of the user doing the build to the authorized keys on the target)

I am also developing an external Config.in that allows generating 
/etc/network/interfaces via menuconfig. This would also be nice to upstream,
maybe in a "contrib" subdirectory.

Next, the Makefile of raspaudio (which is the core of the 
reusable infrastructure) could also be kept by buildroot. This would bring
all the good sides of upstreaming (up to date with buildroot, not lost
in a third party website etc...) I'd like that to happen but I can understand 
if this is not the direction the project wants to go.

Last but not least, I am not a Makefile expert. I did things the way I was 
able to, but I'm sure there are better ways to do that. So feel free to
suggest a better way to integrate stuff


    Regards

    J?r?my Rosen

fight key loggers : write some perl using vim


Open Wide Ingenierie

23, rue Daviel
75013 Paris - France
www.openwide.fr

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Alvaro Gamez]

Hi, Jeremy

Personally, I see potential in this.

Have you thought of a way to include custom packages outside the buildroot
hierarchy?

Just as an example to illustrate the question,  in the case of NOOBS
(https://github.com/raspberrypi/noobs), maybe the NOOBS application
(
https://github.com/raspberrypi/noobs/blob/master/buildroot/package/recovery/recovery.mk
)
itself isn't well suited for buildroot upstream for it's very architecure
dependent.

So, have you thought of some way to place this directory outside buildroots
hierarchy but
let it be managed by buildroot? That, alongside your scripts and makefiles
would allow
to create root file systems with custom packages without "polluting" the
buildroot
directory/repository.

Kind regards



2014-03-10 10:00 GMT+01:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:

> Hello everybody
>
> For some time I have been looking into how to integrate buildroot in a
> project
> where buildroot would be the engine to provide the filesystem building
> infrastructure but where the main repository would not be a clone of the
> buildroot project. The idea is to try to separate cleanly what is meant to
> go upstream from what is not and still be able to use git to properly save
> the data that is closely linked to the project
>
> I have now reached a point where things "simply works" and I can show my
> work
> to the bigger buildroot community.
>
>
> I will first introduce the project itself, then i'll discuss how to create
> your own project based on that infrastructure and then I'll discuss what
> could be upstreamed from the project and how
>
>
> So, with no further delay, let's introduce Raspaudio.
>
> Raspaudio is a small project that builds a filesystem for the raspberry pi
> which allows you to easily stream audio content from the pulseaudio server
> on your desktop to the audio output of the raspberry pi. Compile the
> filesystem, plug the raspberry, configure the PC, you're done.
>
> Details on these steps can be found at
>
> http://openwide-ingenierie.github.io/raspaudio/
>
> I won't go further than that in the project, the idea is that this is a
> "finished" buildroot project. you clone it, you compile it, you install
> it.
>
>    $ git clone --recursive git at github.com:
> Openwide-Ingenierie/raspaudio.git
>    $ cd raspaudio
>    $ make
>
>
> HOW TO REUSE THE RASPAUDIO INFRASTRUCTURE
>
> The whole point of raspaudio is to test the buildroot infrastructure and
> provide an example that is reusable for other projects. If you want to
> make a similar project, here is how to do it
>
>   $ mkdir new_project
>   $ cd new_project
>   $ git init
>   $ git submodule add git://git.buildroot.net/buildroot buildroot
>   $ wget
> https://raw.github.com/Openwide-Ingenierie/raspaudio/master/Makefile
>   $ touch Config.in
>   $ touch external.mk
>   $ echo "output/" > .gitignore
>   $ echo "dl/" >> .gitignore
>
> Change the field PROJECT_NAME in the Makefile to the name of your project
>
>   $ git add Makefile external.mk Config.in .gitignore
>   $ git commit -m"Project infrastructure"
>
> You can now configure your project in the usual buildroot way. For example
> :
>
>   $ make raspberrypi_defconfig
>   $ make menuconfig
>
> A file named $(PROJECT_NAME)_defconfig will appear. Add this project to git
> You can now work with your project in the usual way and all the important
> information will be saved in git.
>
> I tried to make the whole thing as close to the buildroot way as possible,
> all Makefile targets should work properly and most infrasturcture variables
> (DL_DIR, BR2_EXTERNAL...) are set for you. This is here to help you quickly
> have the infrastructure in place to deploy your own projects based on
> buildroot.
>
>
> WHERE TO GO FROM HERE
>
> The main thing I have noticed is that I have been developing post-build
> scripts
> that are completely generic and could easily be reused. It would be nice to
> upstream these scripts (in particular, the script that automatically adds
> the
> public key of the user doing the build to the authorized keys on the
> target)
>
> I am also developing an external Config.in that allows generating
> /etc/network/interfaces via menuconfig. This would also be nice to
> upstream,
> maybe in a "contrib" subdirectory.
>
> Next, the Makefile of raspaudio (which is the core of the
> reusable infrastructure) could also be kept by buildroot. This would bring
> all the good sides of upstreaming (up to date with buildroot, not lost
> in a third party website etc...) I'd like that to happen but I can
> understand
> if this is not the direction the project wants to go.
>
> Last but not least, I am not a Makefile expert. I did things the way I was
> able to, but I'm sure there are better ways to do that. So feel free to
> suggest a better way to integrate stuff
>
>
>     Regards
>
>     J?r?my Rosen
>
> fight key loggers : write some perl using vim
>
>
> Open Wide Ingenierie
>
> 23, rue Daviel
> 75013 Paris - France
> www.openwide.fr
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot




-- 
?lvaro G?mez Machado
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140310/10050c74/attachment.html>

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

> 
> 
> Hi, Jeremy
> 
> Personally, I see potential in this.
> 
> Have you thought of a way to include custom packages outside the
> buildroot hierarchy?
> 
> 
> Just as an example to illustrate the question, in the case of NOOBS
> ( https://github.com/raspberrypi/noobs ), maybe the NOOBS application
> (
> https://github.com/raspberrypi/noobs/blob/master/buildroot/package/recovery/recovery.mk
> )
> itself isn't well suited for buildroot upstream for it's very
> architecure dependent.
> 


yes, this is exactly what I wanted to avoid, a project that has to 
include all of buildroot in a subdirectory and thus has problems
following upstream. 

You can include external software in the raspaudio infrastructre because
this infrastucture relies on the BR2_EXTERNAL feature of buildroot
that allows external software to be built.

You just need to include your own software via the external.mk and
Config.in files that are at the root of raspaudio-based project.

I need to test and document that properly, but it works


> 
> So, have you thought of some way to place this directory outside
> buildroots hierarchy but
> let it be managed by buildroot? That, alongside your scripts and
> makefiles would allow
> to create root file systems with custom packages without "polluting"
> the buildroot
> directory/repository.
> 


That's exactly the point of raspaudio. Feel free to test and report any
problem you have. I have been the only one testing it so far and I
might have missed some important stuff..


Boucman

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Alvaro Gamez]

Hi, Jeremy, all

I've been testing this for a few days and I've come to love it :)

For now, I've just found a problem.

I've tried to replace buildroot's default package definition for
libconfuse with my own (it's just a version bump),
but since I don't want to touch buildroot's master branch, I've added
a copy of package/libconfuse to the $BR2_EXTERNAL directory, alongisde
custom Config.in and external.mk

This, however, results in duplicated definitions, and although make
tries to solve it by overriding previously declared definitions with
the new ones, there are some conflicts:

On make libconfuse-source, the following is reported:
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:19:
warning: overriding recipe for target 'libconfuse-show-depends'
package/libconfuse/libconfuse.mk:19: warning: ignoring old recipe for
target 'libconfuse-show-depends'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:19:
warning: overriding recipe for target 'libconfuse-graph-depends'
package/libconfuse/libconfuse.mk:19: warning: ignoring old recipe for
target 'libconfuse-graph-depends'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:19:
warning: overriding recipe for target 'libconfuse-clean-for-rebuild'
package/libconfuse/libconfuse.mk:19: warning: ignoring old recipe for
target 'libconfuse-clean-for-rebuild'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:19:
warning: overriding recipe for target
'libconfuse-clean-for-reconfigure'
package/libconfuse/libconfuse.mk:19: warning: ignoring old recipe for
target 'libconfuse-clean-for-reconfigure'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:19:
warning: overriding recipe for target 'libconfuse-legal-info'
package/libconfuse/libconfuse.mk:19: warning: ignoring old recipe for
target 'libconfuse-legal-info'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:20:
warning: overriding recipe for target 'host-libconfuse-show-depends'
package/libconfuse/libconfuse.mk:20: warning: ignoring old recipe for
target 'host-libconfuse-show-depends'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:20:
warning: overriding recipe for target 'host-libconfuse-graph-depends'
package/libconfuse/libconfuse.mk:20: warning: ignoring old recipe for
target 'host-libconfuse-graph-depends'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:20:
warning: overriding recipe for target
'host-libconfuse-clean-for-rebuild'
package/libconfuse/libconfuse.mk:20: warning: ignoring old recipe for
target 'host-libconfuse-clean-for-rebuild'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:20:
warning: overriding recipe for target
'host-libconfuse-clean-for-reconfigure'
package/libconfuse/libconfuse.mk:20: warning: ignoring old recipe for
target 'host-libconfuse-clean-for-reconfigure'
/home/sslayer/Proyectos/ART/Firmware-SBC/external_packages/libconfuse/libconfuse.mk:20:
warning: overriding recipe for target 'host-libconfuse-legal-info'
package/libconfuse/libconfuse.mk:20: warning: ignoring old recipe for
target 'host-libconfuse-legal-info'



That is fine, and
https://github.com/martinh/libconfuse/archive/master/libconfuse-master.tar.gz
is downloaded.

Now, make libconfuse-extract reports the same log as above, but adds this:
>>> libconfuse master Extracting
gzip -d -c /home/sslayer/Proyectos/ART/Firmware-SBC/dl/libconfuse-master.tar.gz
| tar --strip-components=1 -C
/home/sslayer/Proyectos/ART/Firmware-SBC/ARTMidrange/output/build/libconfuse-master
 -xf -
>>> libconfuse master Extracting
gzip -d -c /home/sslayer/Proyectos/ART/Firmware-SBC/dl/libconfuse-master.tar.gz
| tar --strip-components=1 -C
/home/sslayer/Proyectos/ART/Firmware-SBC/ARTMidrange/output/build/libconfuse-master
 -xf -

It seems that it is trying to extract both versions, the one declared
on buildroot and the one declared by me outside buildroot's directory.
This is not very important since the end result is the same. However,
next step fails:



make libconfuse-patch reports this at the end of execution:

No file to patch.  Skipping patch.
1 out of 1 hunk ignored
Patch failed!  Please fix libconfuse-0000-fix-configure-ac.patch!
package/pkg-generic.mk:146: recipe for target
'/home/sslayer/Proyectos/ART/Firmware-SBC/ARTMidrange/output/build/libconfuse-V2_7/.stamp_patched'
failed
make[1]: *** [/home/sslayer/Proyectos/ART/Firmware-SBC/ARTMidrange/output/build/libconfuse-V2_7/.stamp_patched]
Error 1
common.mk:54: recipe for target '_all' failed
make: *** [_all] Error 2


As can be seen, the patch is being applied to libconfuse-V2_7, which
is buildroot's declared version of libconfuse, while it should have
been applied to master version as I declared on the outside
buildroot's directory external makefile.

For now, I'll have to maintain a different buildroot branch in which
this package is modified, since I've not being able to override
buildroot's definition.

All in all, however, I'm very happy with the environment your Makefile
provides. I've done a couple changes that I hope will publish in the
following week for comments.

Kind regards,


2014-03-10 16:38 GMT+01:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
>
>
>>
>>
>> Hi, Jeremy
>>
>> Personally, I see potential in this.
>>
>> Have you thought of a way to include custom packages outside the
>> buildroot hierarchy?
>>
>>
>> Just as an example to illustrate the question, in the case of NOOBS
>> ( https://github.com/raspberrypi/noobs ), maybe the NOOBS application
>> (
>> https://github.com/raspberrypi/noobs/blob/master/buildroot/package/recovery/recovery.mk
>> )
>> itself isn't well suited for buildroot upstream for it's very
>> architecure dependent.
>>
>
>
> yes, this is exactly what I wanted to avoid, a project that has to
> include all of buildroot in a subdirectory and thus has problems
> following upstream.
>
> You can include external software in the raspaudio infrastructre because
> this infrastucture relies on the BR2_EXTERNAL feature of buildroot
> that allows external software to be built.
>
> You just need to include your own software via the external.mk and
> Config.in files that are at the root of raspaudio-based project.
>
> I need to test and document that properly, but it works
>
>
>>
>> So, have you thought of some way to place this directory outside
>> buildroots hierarchy but
>> let it be managed by buildroot? That, alongside your scripts and
>> makefiles would allow
>> to create root file systems with custom packages without "polluting"
>> the buildroot
>> directory/repository.
>>
>
>
> That's exactly the point of raspaudio. Feel free to test and report any
> problem you have. I have been the only one testing it so far and I
> might have missed some important stuff..
>
>
> Boucman



-- 
?lvaro G?mez Machado

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

> Hi, Jeremy, all
> 
> I've been testing this for a few days and I've come to love it :)
> 
> For now, I've just found a problem.
> 
> I've tried to replace buildroot's default package definition for
> libconfuse with my own (it's just a version bump),
> but since I don't want to touch buildroot's master branch, I've added
> a copy of package/libconfuse to the $BR2_EXTERNAL directory,
> alongisde
> custom Config.in and external.mk
> 
> This, however, results in duplicated definitions, and although make
> tries to solve it by overriding previously declared definitions with
> the new ones, there are some conflicts:
> 
> 

...

> 
> As can be seen, the patch is being applied to libconfuse-V2_7, which
> is buildroot's declared version of libconfuse, while it should have
> been applied to master version as I declared on the outside
> buildroot's directory external makefile.
> 
> For now, I'll have to maintain a different buildroot branch in which
> this package is modified, since I've not being able to override
> buildroot's definition.
> 

That's an incencitive to upstream, no ? :P

more seriously, if you want to do that, I would recommand 
* either renaming the package entirely. 
* or use the OVERRRIDE_SRC_DIR macro (see usermanual)

> All in all, however, I'm very happy with the environment your
> Makefile
> provides. I've done a couple changes that I hope will publish in the
> following week for comments.
> 

please do, I'll happily integrate... At this stage the framework satisfies all my needs so I need external ideas to make it live :)

> Kind regards,
> 
> 
> 2014-03-10 16:38 GMT+01:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> >
> >
> >>
> >>
> >> Hi, Jeremy
> >>
> >> Personally, I see potential in this.
> >>
> >> Have you thought of a way to include custom packages outside the
> >> buildroot hierarchy?
> >>
> >>
> >> Just as an example to illustrate the question, in the case of
> >> NOOBS
> >> ( https://github.com/raspberrypi/noobs ), maybe the NOOBS
> >> application
> >> (
> >> https://github.com/raspberrypi/noobs/blob/master/buildroot/package/recovery/recovery.mk
> >> )
> >> itself isn't well suited for buildroot upstream for it's very
> >> architecure dependent.
> >>
> >
> >
> > yes, this is exactly what I wanted to avoid, a project that has to
> > include all of buildroot in a subdirectory and thus has problems
> > following upstream.
> >
> > You can include external software in the raspaudio infrastructre
> > because
> > this infrastucture relies on the BR2_EXTERNAL feature of buildroot
> > that allows external software to be built.
> >
> > You just need to include your own software via the external.mk and
> > Config.in files that are at the root of raspaudio-based project.
> >
> > I need to test and document that properly, but it works
> >
> >
> >>
> >> So, have you thought of some way to place this directory outside
> >> buildroots hierarchy but
> >> let it be managed by buildroot? That, alongside your scripts and
> >> makefiles would allow
> >> to create root file systems with custom packages without
> >> "polluting"
> >> the buildroot
> >> directory/repository.
> >>
> >
> >
> > That's exactly the point of raspaudio. Feel free to test and report
> > any
> > problem you have. I have been the only one testing it so far and I
> > might have missed some important stuff..
> >
> >
> > Boucman
> 
> 
> 
> --
> ?lvaro G?mez Machado
> 

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Alvaro Gamez]

Hi!

2014-09-26 13:56 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> That's an incencitive to upstream, no ? :P
>
> more seriously, if you want to do that, I would recommand
> * either renaming the package entirely.
> * or use the OVERRRIDE_SRC_DIR macro (see usermanual)

On this case I would. It's a simple bump version from V2_7 to github master,
but since it's not a release I'm not sure it is desirable. Maybe it's
better to wait for V2_8,
although libconfuse's development is pretty much stagnated.

I don't want to rename the package, because that would mean having to
modify all packages that depend on this one,
so at the end it's less cumbersome to have a custom branch of
buildroot in which I have made these little changes.

>> All in all, however, I'm very happy with the environment your
>> Makefile
>> provides. I've done a couple changes that I hope will publish in the
>> following week for comments.
>>
>
> please do, I'll happily integrate... At this stage the framework satisfies all my needs so I need external ideas to make it live :)

I've just uploaded a few files to this github repository.

https://github.com/agamez/buildroot-submodule

There are only a couple differences between that version and yours.
First, I solved a typo on your Makefile (now called common.mk),
and then I've made it so that several projects can coexist under the
same directory.
Thus, two Makefile exist (Makefile.Project1 and Makefile.Project-X)

Configuration and buildroot generated files for each one are now saved
under Project1 and Project-X directory.

This now satisfies my need of two almost similar projects, sharing up
to 90% source code but with a couple differences.
I need to be able to compile both and have them on different
directories, thus the two output directories.

Now you are able to make -f Makefile.Project1 menuconfig, or make -f
Makefile.Project-X or whatever,
without interfering one project with the other.

I hope you like it!

Regards

-- 
?lvaro G?mez Machado

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

----- Mail original -----
> Hi!
> 
> 2014-09-26 13:56 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> > That's an incencitive to upstream, no ? :P
> >
> > more seriously, if you want to do that, I would recommand
> > * either renaming the package entirely.
> > * or use the OVERRRIDE_SRC_DIR macro (see usermanual)
> 
> On this case I would. It's a simple bump version from V2_7 to github
> master,
> but since it's not a release I'm not sure it is desirable. Maybe it's
> better to wait for V2_8,
> although libconfuse's development is pretty much stagnated.
> 
> I don't want to rename the package, because that would mean having to
> modify all packages that depend on this one,
> so at the end it's less cumbersome to have a custom branch of
> buildroot in which I have made these little changes.
> 

Ok, that's probably a case of cloning libconfuse (as a submodule )in 
your main directory and using OVERRIDE_SRC_DIR to use it..

> >> All in all, however, I'm very happy with the environment your
> >> Makefile
> >> provides. I've done a couple changes that I hope will publish in
> >> the
> >> following week for comments.
> >>
> >
> > please do, I'll happily integrate... At this stage the framework
> > satisfies all my needs so I need external ideas to make it live :)
> 
> I've just uploaded a few files to this github repository.
> 
> https://github.com/agamez/buildroot-submodule
> 

Ok, since you have a created a github project, I have cloned it myself
if you want my clone to be the "root" clone, just destroy yours and then
re-clone mine... 

> There are only a couple differences between that version and yours.
> First, I solved a typo on your Makefile (now called common.mk),
> and then I've made it so that several projects can coexist under the
> same directory.
> Thus, two Makefile exist (Makefile.Project1 and Makefile.Project-X)
> 
> Configuration and buildroot generated files for each one are now
> saved
> under Project1 and Project-X directory.
> 

And you can have a Makefile with an empty project name for the case
where you only want one project, good

> This now satisfies my need of two almost similar projects, sharing up
> to 90% source code but with a couple differences.
> I need to be able to compile both and have them on different
> directories, thus the two output directories.
> 

yes typicall use-case for me is a project that needs to run on two
different hardware

* all overlays/custom packages are common
* buildroot source obviously need to be the same
* dl/ can be shared
* config files are different
* build/ directory is completely different

> Now you are able to make -f Makefile.Project1 menuconfig, or make -f
> Makefile.Project-X or whatever,
> without interfering one project with the other.
> 
> I hope you like it!
> 

A lot, keep me posted if you change more stuff...

I'll add a README for cleanliness...


> Regards
> 
> --
> ?lvaro G?mez Machado
> 

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

Looking into it a bit deeper, I have a couple of questions for you..

in your current layout 

* BR2_EXTERNAL is shared, so Config.in is shared and external.mk 
is shared
* each project has its own overlay
* each project has its own local.mk
* each project has its own patch subdirectory...
* dl is shared

so I need to understand your use-case a little more. 
As I said my use case is "same project, but different hardware" so
my favorite layout would be

* shared BR2_EXTERNAL
* shared overlay
* shared patches
* different config files
* don't care about local.mk, might as well make it different

This layout makes more sense to me (shared overlay and shared patches)
but I want to make sure it fits your use-case before I do the change

It makes little sense to me to have completely different projects
which are just sharing dl/ since dl is not saved in git and it's
more handy to have completely separate git repo in this case...


If you can tell me a little more, I'll see how I adapt this...

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Alvaro Gamez]

Hi, Jeremy

Well, maybe I went all crazy splitting everything for each project,
but there's some reason behind it.

My main project is destined to run in a very specific hardware, with
external devices that pass information to the system. A real example
is a GPS device.

On the other hand, I have a x86 machine which I use as a demo-machine
with no external hardware attached (no GPS, for example). This
requires that the information that is typically received from these
external devices must be emulated by some piece of software.

So, in my case, of course dl/ directoy is common, Config.in and
external.mk are shared. I believe local.mk can also be shared,
although in my current layout it is not. It seems possible that some
patches could be different, but I guess they could also be shared.

I need, however, to have different overlays, such as, for example, a
'gpsd' emulator can run instead of the real gpsd. I manage this with a
different inittab file and different configuration files that I place
on each project's overlay. Alongside this trivial differences, I must
load into the demo-machine some big files (a couple of GB) that
emulate the data that, in the real machine, is received from external
sources. These files would not even fit on the storage of the real
machine.

Well, all in all, I think only the overlays may need to be different,
so there's that.

Regards


2014-10-01 16:08 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
>
>
> Looking into it a bit deeper, I have a couple of questions for you..
>
> in your current layout
>
> * BR2_EXTERNAL is shared, so Config.in is shared and external.mk
> is shared
> * each project has its own overlay
> * each project has its own local.mk
> * each project has its own patch subdirectory...
> * dl is shared
>
> so I need to understand your use-case a little more.
> As I said my use case is "same project, but different hardware" so
> my favorite layout would be
>
> * shared BR2_EXTERNAL
> * shared overlay
> * shared patches
> * different config files
> * don't care about local.mk, might as well make it different
>
> This layout makes more sense to me (shared overlay and shared patches)
> but I want to make sure it fits your use-case before I do the change
>
> It makes little sense to me to have completely different projects
> which are just sharing dl/ since dl is not saved in git and it's
> more handy to have completely separate git repo in this case...
>
>
> If you can tell me a little more, I'll see how I adapt this...



-- 
?lvaro G?mez Machado

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

----- Mail original -----
> Hi, Jeremy
> 
> Well, maybe I went all crazy splitting everything for each project,
> but there's some reason behind it.

that's what I expected and why I asked :)

> 
> My main project is destined to run in a very specific hardware, with
> external devices that pass information to the system. A real example
> is a GPS device.
> 
> On the other hand, I have a x86 machine which I use as a demo-machine
> with no external hardware attached (no GPS, for example). This
> requires that the information that is typically received from these
> external devices must be emulated by some piece of software.
> 
> So, in my case, of course dl/ directoy is common, Config.in and
> external.mk are shared. I believe local.mk can also be shared,
> although in my current layout it is not. 

there is no point in not sharing it, since external.mk fullfills the 
same purpose and is already shared

>It seems possible that some
> patches could be different, but I guess they could also be shared.
> 
> I need, however, to have different overlays, such as, for example, a
> 'gpsd' emulator can run instead of the real gpsd. I manage this with
> a
> different inittab file and different configuration files that I place
> on each project's overlay. Alongside this trivial differences, I must
> load into the demo-machine some big files (a couple of GB) that
> emulate the data that, in the real machine, is received from external
> sources. These files would not even fit on the storage of the real
> machine.
> 
> Well, all in all, I think only the overlays may need to be different,
> so there's that.
> 


hmm.. ok, that complicates things a bit... I'll look into the BR internal
to see if it's possible to have two overlays (one automatically set by
BR2_EXTERNAL and one set via the configuration menu) and if the project is
ready to accept that...

the only think we can't easily double would be Config.in beacause that file 
needs to exist prior to menuconfig. The only way to do it would be to specify
it in the make command line (the internal one we generate, not the toplevel
one) and again see if buildroot is interested... I'll see what I can do

I also added an "anonymous" project for the common use-case where you
only need one subproject, and i'm working on a README...


> Regards
> 
> 
> 2014-10-01 16:08 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> >
> >
> > Looking into it a bit deeper, I have a couple of questions for
> > you..
> >
> > in your current layout
> >
> > * BR2_EXTERNAL is shared, so Config.in is shared and external.mk
> > is shared
> > * each project has its own overlay
> > * each project has its own local.mk
> > * each project has its own patch subdirectory...
> > * dl is shared
> >
> > so I need to understand your use-case a little more.
> > As I said my use case is "same project, but different hardware" so
> > my favorite layout would be
> >
> > * shared BR2_EXTERNAL
> > * shared overlay
> > * shared patches
> > * different config files
> > * don't care about local.mk, might as well make it different
> >
> > This layout makes more sense to me (shared overlay and shared
> > patches)
> > but I want to make sure it fits your use-case before I do the
> > change
> >
> > It makes little sense to me to have completely different projects
> > which are just sharing dl/ since dl is not saved in git and it's
> > more handy to have completely separate git repo in this case...
> >
> >
> > If you can tell me a little more, I'll see how I adapt this...
> 
> 
> 
> --
> ?lvaro G?mez Machado
> 

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Alvaro Gamez]

Hi!

I've been thinking about the need for different overlays.
Even though that's how I firstly organized my setup, I am now thinking
it may not be the best course of action.

I've realized that in my setup, the two not-equal overlays are, in
fact, equal except for a couple directories and /etc/inittab.
I believe that it may be a better solution to just create a package
that installs these big demo files I need on the demo machine,
and simply select that package on the demo defconfig, instead of
having two almost equal overlays.

I would still have to solve the differences in /etc/inittab, but maybe
this file can be selected via a post build script.

What do you think?


2014-10-01 16:37 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
>
> ----- Mail original -----
>> Hi, Jeremy
>>
>> Well, maybe I went all crazy splitting everything for each project,
>> but there's some reason behind it.
>
> that's what I expected and why I asked :)
>
>>
>> My main project is destined to run in a very specific hardware, with
>> external devices that pass information to the system. A real example
>> is a GPS device.
>>
>> On the other hand, I have a x86 machine which I use as a demo-machine
>> with no external hardware attached (no GPS, for example). This
>> requires that the information that is typically received from these
>> external devices must be emulated by some piece of software.
>>
>> So, in my case, of course dl/ directoy is common, Config.in and
>> external.mk are shared. I believe local.mk can also be shared,
>> although in my current layout it is not.
>
> there is no point in not sharing it, since external.mk fullfills the
> same purpose and is already shared
>
>>It seems possible that some
>> patches could be different, but I guess they could also be shared.
>>
>> I need, however, to have different overlays, such as, for example, a
>> 'gpsd' emulator can run instead of the real gpsd. I manage this with
>> a
>> different inittab file and different configuration files that I place
>> on each project's overlay. Alongside this trivial differences, I must
>> load into the demo-machine some big files (a couple of GB) that
>> emulate the data that, in the real machine, is received from external
>> sources. These files would not even fit on the storage of the real
>> machine.
>>
>> Well, all in all, I think only the overlays may need to be different,
>> so there's that.
>>
>
>
> hmm.. ok, that complicates things a bit... I'll look into the BR internal
> to see if it's possible to have two overlays (one automatically set by
> BR2_EXTERNAL and one set via the configuration menu) and if the project is
> ready to accept that...
>
> the only think we can't easily double would be Config.in beacause that file
> needs to exist prior to menuconfig. The only way to do it would be to specify
> it in the make command line (the internal one we generate, not the toplevel
> one) and again see if buildroot is interested... I'll see what I can do
>
> I also added an "anonymous" project for the common use-case where you
> only need one subproject, and i'm working on a README...
>
>
>> Regards
>>
>>
>> 2014-10-01 16:08 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
>> >
>> >
>> > Looking into it a bit deeper, I have a couple of questions for
>> > you..
>> >
>> > in your current layout
>> >
>> > * BR2_EXTERNAL is shared, so Config.in is shared and external.mk
>> > is shared
>> > * each project has its own overlay
>> > * each project has its own local.mk
>> > * each project has its own patch subdirectory...
>> > * dl is shared
>> >
>> > so I need to understand your use-case a little more.
>> > As I said my use case is "same project, but different hardware" so
>> > my favorite layout would be
>> >
>> > * shared BR2_EXTERNAL
>> > * shared overlay
>> > * shared patches
>> > * different config files
>> > * don't care about local.mk, might as well make it different
>> >
>> > This layout makes more sense to me (shared overlay and shared
>> > patches)
>> > but I want to make sure it fits your use-case before I do the
>> > change
>> >
>> > It makes little sense to me to have completely different projects
>> > which are just sharing dl/ since dl is not saved in git and it's
>> > more handy to have completely separate git repo in this case...
>> >
>> >
>> > If you can tell me a little more, I'll see how I adapt this...
>>
>>
>>
>> --
>> ?lvaro G?mez Machado
>>



-- 
?lvaro G?mez Machado

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

> Hi!
> 
> I've been thinking about the need for different overlays.
> Even though that's how I firstly organized my setup, I am now
> thinking
> it may not be the best course of action.
> 
> I've realized that in my setup, the two not-equal overlays are, in
> fact, equal except for a couple directories and /etc/inittab.
> I believe that it may be a better solution to just create a package
> that installs these big demo files I need on the demo machine,
> and simply select that package on the demo defconfig, instead of
> having two almost equal overlays.
> 
> I would still have to solve the differences in /etc/inittab, but
> maybe
> this file can be selected via a post build script.
> 
> What do you think?
> 

That would work and would indeed solve your problem, but I still need
to think the big picture through...

I have been reorganising/writing down how to use buildroot-submodule
for various use-cases, and I sould push something soon. I will 
probably change the project to focus on the "single variant" case 
while documenting when and how to have multiple variants.

If we stay with the use-case we have outlined so far (one logical 
project on multiple architectures) then we need multiple overlays.
Having different configuration files for different machines seems
a really needed feature. I can autogenerate some Config.in or .mk
files to automatically have things being configurable per-variant
or globally (wip) but there is only one overlay directory and no
easy way to "merge" multiple overlay directories from within BR

I need to think about that part some more, i'm open to suggestion


> 
> 2014-10-01 16:37 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> >
> > ----- Mail original -----
> >> Hi, Jeremy
> >>
> >> Well, maybe I went all crazy splitting everything for each
> >> project,
> >> but there's some reason behind it.
> >
> > that's what I expected and why I asked :)
> >
> >>
> >> My main project is destined to run in a very specific hardware,
> >> with
> >> external devices that pass information to the system. A real
> >> example
> >> is a GPS device.
> >>
> >> On the other hand, I have a x86 machine which I use as a
> >> demo-machine
> >> with no external hardware attached (no GPS, for example). This
> >> requires that the information that is typically received from
> >> these
> >> external devices must be emulated by some piece of software.
> >>
> >> So, in my case, of course dl/ directoy is common, Config.in and
> >> external.mk are shared. I believe local.mk can also be shared,
> >> although in my current layout it is not.
> >
> > there is no point in not sharing it, since external.mk fullfills
> > the
> > same purpose and is already shared
> >
> >>It seems possible that some
> >> patches could be different, but I guess they could also be shared.
> >>
> >> I need, however, to have different overlays, such as, for example,
> >> a
> >> 'gpsd' emulator can run instead of the real gpsd. I manage this
> >> with
> >> a
> >> different inittab file and different configuration files that I
> >> place
> >> on each project's overlay. Alongside this trivial differences, I
> >> must
> >> load into the demo-machine some big files (a couple of GB) that
> >> emulate the data that, in the real machine, is received from
> >> external
> >> sources. These files would not even fit on the storage of the real
> >> machine.
> >>
> >> Well, all in all, I think only the overlays may need to be
> >> different,
> >> so there's that.
> >>
> >
> >
> > hmm.. ok, that complicates things a bit... I'll look into the BR
> > internal
> > to see if it's possible to have two overlays (one automatically set
> > by
> > BR2_EXTERNAL and one set via the configuration menu) and if the
> > project is
> > ready to accept that...
> >
> > the only think we can't easily double would be Config.in beacause
> > that file
> > needs to exist prior to menuconfig. The only way to do it would be
> > to specify
> > it in the make command line (the internal one we generate, not the
> > toplevel
> > one) and again see if buildroot is interested... I'll see what I
> > can do
> >
> > I also added an "anonymous" project for the common use-case where
> > you
> > only need one subproject, and i'm working on a README...
> >
> >
> >> Regards
> >>
> >>
> >> 2014-10-01 16:08 GMT+02:00 Jeremy Rosen
> >> <jeremy.rosen@openwide.fr>:
> >> >
> >> >
> >> > Looking into it a bit deeper, I have a couple of questions for
> >> > you..
> >> >
> >> > in your current layout
> >> >
> >> > * BR2_EXTERNAL is shared, so Config.in is shared and external.mk
> >> > is shared
> >> > * each project has its own overlay
> >> > * each project has its own local.mk
> >> > * each project has its own patch subdirectory...
> >> > * dl is shared
> >> >
> >> > so I need to understand your use-case a little more.
> >> > As I said my use case is "same project, but different hardware"
> >> > so
> >> > my favorite layout would be
> >> >
> >> > * shared BR2_EXTERNAL
> >> > * shared overlay
> >> > * shared patches
> >> > * different config files
> >> > * don't care about local.mk, might as well make it different
> >> >
> >> > This layout makes more sense to me (shared overlay and shared
> >> > patches)
> >> > but I want to make sure it fits your use-case before I do the
> >> > change
> >> >
> >> > It makes little sense to me to have completely different
> >> > projects
> >> > which are just sharing dl/ since dl is not saved in git and it's
> >> > more handy to have completely separate git repo in this case...
> >> >
> >> >
> >> > If you can tell me a little more, I'll see how I adapt this...
> >>
> >>
> >>
> >> --
> >> ?lvaro G?mez Machado
> >>
> 
> 
> 
> --
> ?lvaro G?mez Machado
> 

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

Ok, I have made quite a few changes, some of them are really interesting

* I have added a big README.md (which needs proofrreading) that covers
  all sorts of interesting things, both on the project and how to 
  customize buildroot in general
* I have removed your Makefile.Project* files, I standardize on a 
  single project being "standard" and multiple projects being
  "advanced" (but it's properly documented)
* I have changed defaults to "change as much as possible with 
  variants. This should not break your use-case since
  * all is set via BR settings, so it can be overridden
  * if it is already set, it won't be overwritten by the
     infrastructure
* I have documented how to have per-project configuration options via
  variables
* I have documented how to use variants to have the toolchain be a 
  separated project. This has been discussed here recently and is 
  rather easy to do with the submodule infrastructure


Feel free to test/break/send me feedback. This is starting to loo really
great.

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Alvaro Gamez]

Hi, Jeremy

That's a pretty long README and it's very clear. I've just sent you a
pull request fixing a few typos, I couldn't think of any else to
change.

I'm now using this latest version with my project and everything seems fine.
I'm using a post build script to modify inittab accordingly and
packages to selectively install the binary blobs I need on the
simulation machine, so now I live happy with a fully common overlay.

I didn't know about building the toolchain as a separate project. It
looks cool, although I'm currently using prebuilt toolchains and have
not tested it yet.


Regards,

2014-10-03 10:03 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> Ok, I have made quite a few changes, some of them are really interesting
>
> * I have added a big README.md (which needs proofrreading) that covers
>   all sorts of interesting things, both on the project and how to
>   customize buildroot in general
> * I have removed your Makefile.Project* files, I standardize on a
>   single project being "standard" and multiple projects being
>   "advanced" (but it's properly documented)
> * I have changed defaults to "change as much as possible with
>   variants. This should not break your use-case since
>   * all is set via BR settings, so it can be overridden
>   * if it is already set, it won't be overwritten by the
>      infrastructure
> * I have documented how to have per-project configuration options via
>   variables
> * I have documented how to use variants to have the toolchain be a
>   separated project. This has been discussed here recently and is
>   rather easy to do with the submodule infrastructure
>
>
> Feel free to test/break/send me feedback. This is starting to loo really
> great.



-- 
?lvaro G?mez Machado

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

----- Mail original -----
> Hi, Jeremy
> 
> That's a pretty long README and it's very clear. I've just sent you a
> pull request fixing a few typos, I couldn't think of any else to
> change.

thx, merged.

yeah, I have to admit that this README is more or less an internal 
documentation on how to personalize buildroot


> 
> I'm now using this latest version with my project and everything
> seems fine.
> I'm using a post build script to modify inittab accordingly and
> packages to selectively install the binary blobs I need on the
> simulation machine, so now I live happy with a fully common overlay.
> 

it would be handy to have multiple overlays, but it's not possible with
the current buildroot infrastructure...

This is the last point where you can't have a common+per-variant config
in buildroot submodule. I'm considering a BR patch, but nothing has 
been written yet...

> I didn't know about building the toolchain as a separate project. It
> looks cool, although I'm currently using prebuilt toolchains and have
> not tested it yet.
> 

Yeah, that was a "hey, that would be cool" moment while writing the 
documentation. I quickly tested it and it works, so I added the 
section to the documentation


I'm glad this helps

Regards 
J?r?my

> 
> Regards,
> 
> 2014-10-03 10:03 GMT+02:00 Jeremy Rosen <jeremy.rosen@openwide.fr>:
> > Ok, I have made quite a few changes, some of them are really
> > interesting
> >
> > * I have added a big README.md (which needs proofrreading) that
> > covers
> >   all sorts of interesting things, both on the project and how to
> >   customize buildroot in general
> > * I have removed your Makefile.Project* files, I standardize on a
> >   single project being "standard" and multiple projects being
> >   "advanced" (but it's properly documented)
> > * I have changed defaults to "change as much as possible with
> >   variants. This should not break your use-case since
> >   * all is set via BR settings, so it can be overridden
> >   * if it is already set, it won't be overwritten by the
> >      infrastructure
> > * I have documented how to have per-project configuration options
> > via
> >   variables
> > * I have documented how to use variants to have the toolchain be a
> >   separated project. This has been discussed here recently and is
> >   rather easy to do with the submodule infrastructure
> >
> >
> > Feel free to test/break/send me feedback. This is starting to loo
> > really
> > great.
> 
> 
> 
> --
> ?lvaro G?mez Machado
> 

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Arnout Vandecappelle]

On 08/10/14 12:11, Jeremy Rosen wrote:
>> > 
> it would be handy to have multiple overlays, but it's not possible with
> the current buildroot infrastructure...

 You're talking about the rootfs overlay, right? BR2_ROOTFS_OVERLAY can be a
space-separated list of overlay directories.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Raspaudio : how to use buildroot as a git submodule

[Jeremy Rosen]

Cordi
----- Mail original -----
> On 08/10/14 12:11, Jeremy Rosen wrote:
> >> > 
> > it would be handy to have multiple overlays, but it's not possible
> > with
> > the current buildroot infrastructure...
> 
>  You're talking about the rootfs overlay, right? BR2_ROOTFS_OVERLAY
>  can be a
> space-separated list of overlay directories.
> 

yeah, I realized that recently while re-reading the docs... thx for
pointing it out anyway.

>  Regards,
>  Arnout
> 
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR
> Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
>