[Buildroot] [RFC] New feature to handle a custom board folder

[Buildroot] [RFC] New feature to handle a custom board folder

[Ryan Barnett]

All,

I would like to introduce a new concept I've got for how the board
folder is brought into buildroot. I propose adding a feature to
buildroot to treat the board folder as a package. In treating the board
folder as a package, it would allow a board folder to be downloaded
using currently supported methods of packages (HG/Git/SVN/archives from
websites). It would be placed as an archive in the DL_DIR and extracted
to BUILD_DIR/board. I will go into more detail further down about the
structure.

The motivation for this feature is that most of the customization done
to buildroot in order to support  a project (board) is usually
contained to a board folder. However, in many work environments in
which buildroot is used, GIT isn't a preferred version control method
(mine included) so being able to track mainline of buildroot is very
difficult.

NOTE: I haven't attempted to implement this yet as this an idea that 
is still in its infancies. 

Theory
---------
In theory the custom board folder structure would work as follows:

1. User specifies the custom board folder in menuconfig
    a. Initial though would to place it as follows in menuconfig

Target Options 
 > Target Board Folder Type 
     > No Custom Board Folder (default)
     > External Custom Board Folder
 > Target Board Folder Site Method
     > Git
     > HG
     > SVN
     > List goes on...
 > Board folder contains custom applications
     COMMENTARY: This feature allows the use for downloading and building
     a company's custom applications that need to place in the release.
 > (Future Option) Check folder out using method above
     COMMENTARY: This feature would be useful during development of 
     board folder to allow the actual checkout board folder to the 
BUILD_DIR
     to allow for modifications tha easily pushed back.

Target Packages
 > Text editor and viewers
 > Custom board applications
    > App A
    > App B
 
    COMMENTARY: the custom board applications would only appear when 
    "Board folder contains custom applications" is enabled. The 
application 
    list from custom board applications would be sourced an applications 
folder
    within the custom board folder. These apps wouldn't display until the 
board
    folder is actually extract. 
 
    One issue I see with this is how the makefile structure would handle 
    building without the board folder checked out and the make rules for 
these 
    packages being defined.

2. Board folder structure would be as follows:

> BUILD_DIR
   > board (new)
      > kernel-patches
      > local-skeleton
      > packages (this would be the board specific packages)
      > uboot-patches
      > config files
   > build
   > host
   > target

3. User says "make board-folder" and the custom board folder is extracted

4. The user can say "make menuconfig" and their board customization will 
   appear and would be available for selection.

5. User say "make" and buildroot builds them their system

Summary
--------
With adding this feature to buildroot, it will help companies/users easily 

bring in their customization into buildroot with just using a defconfig.

Please provide me with feedback in if this idea is feasible/be useful. I'm 

also very open suggestions to help make this a reality.




Ryan J Barnett / Software Engineer / Platform SW 
MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
Phone: 319-263-3880 / VPN: 263-3880 
rjbarnet at rockwellcollins.com
www.rockwellcollins.com 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130906/73d37c78/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 2004 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130906/73d37c78/attachment.gif>

[Buildroot] [RFC] New feature to handle a custom board folder

[Arnout Vandecappelle]

  Hi Ryan,

On 06/09/13 20:45, Ryan Barnett wrote:
> All,
>
> I would like to introduce a new concept I've got for how the board
> folder is brought into buildroot.  I propose adding a feature to
> buildroot to treat the board folder as a package. In treating the board
> folder as a package, it would allow a board folder to be downloaded
> using currently supported methods of packages (HG/Git/SVN/archives from
> websites). It would be placed as an archive in the DL_DIR and extracted
> to BUILD_DIR/board. I will go into more detail further down about the
> structure.

  Something like this has been proposed in buildroot several times 
already (including by me), and it has been rejected every time. The main 
reasons for rejection is that it adds a lot of complexity, while the 
current setup is already very simple and feature-complete for the user. 
See below for an explanation why your proposal is very complicated.

  The simple way to do it is as follows:

- import buildroot in your VCS;

- create directory 'package/mycompany';

- add 'source package/mycompany/Config.in' to package/Config.in;

- create directory 'board/mycompany';

- create file 'configs/mycompanyboard_defconfig'.


  Since the diff to mainline is limited to a bunch of new files and a 
single line change in package/Config.in, merging is normally not an issue.


> The motivation for this feature is that most of the customization done
> to buildroot in order to support  a project (board) is usually
> contained to a board folder. However, in many work environments in
> which buildroot is used, GIT isn't a preferred version control method
> (mine included) so being able to track mainline of buildroot is very
> difficult.

  That's a situation that we didn't expect, I think. In most company 
environments, buildroot is only occasionally updated. Also, you would 
typically require buildroot to be maintained under the "preferred version 
control method", because you make some changes - cfr. all the changes 
that you're upstreaming now.

  Also, keeping the buildroot integration files for proprietary packages 
outside of the buildroot tree makes your repository a bit messy. Putting 
this stuff inside the buildroot tree is cleaner and more understandable.


> NOTE: I haven't attempted to implement this yet as this an idea that
> is still in its infancies.
>
> Theory
> ---------
> In theory the custom board folder structure would work as follows:
>
> 1. User specifies the custom board folder in menuconfig
>      a. Initial though would to place it as follows in menuconfig
>
> Target Options
>   > Target Board Folder Type
>       > No Custom Board Folder (default)
>       > External Custom Board Folder
>   > Target Board Folder Site Method
>       > Git
>       > HG
>       > SVN
>       > List goes on...
>   > Board folder contains custom applications
>       COMMENTARY: This feature allows the use for downloading and building
>       a company's custom applications that need to place in the release.
>   > (Future Option) Check folder out using method above
>       COMMENTARY: This feature would be useful during development of
>       board folder to allow the actual checkout board folder to the
> BUILD_DIR
>       to allow for modifications tha easily pushed back.
>
> Target Packages
>   > Text editor and viewers
>   > Custom board applications
>      > App A
>      > App B
>
>      COMMENTARY: the custom board applications would only appear when
>      "Board folder contains custom applications" is enabled. The application
>      list from custom board applications would be sourced an applications
> folder
>      within the custom board folder. These apps wouldn't display until
> the board
>      folder is actually extract.

  Because of limitations in the Kconfig tool, this is very difficult to 
do in practice. You need some serious patching of Kconfig, and even then 
it probably doesn't work very well. A patch was submitted that attempted 
to implement this about a year ago IIRC.


>      One issue I see with this is how the makefile structure would handle
>      building without the board folder checked out and the make rules for
> these
>      packages being defined.

  Yeah, that was not even supported by the patch I mentioned above.


> 2. Board folder structure would be as follows:
>
>  > BUILD_DIR
>     > board (new)
>        > kernel-patches
>        > local-skeleton
>        > packages (this would be the board specific packages)
>        > uboot-patches
>        > config files
>     > build
>     > host
>     > target
>
> 3. User says "make board-folder" and the custom board folder is extracted
>
> 4. The user can say "make menuconfig" and their board customization will
>     appear and would be available for selection.
>
> 5. User say "make" and buildroot builds them their system

  This is making things a lot more difficult for the user, compared to 
when everything would just be in the buildroot tree, no?


  Regards,
  Arnout

>
> Summary
> --------
> With adding this feature to buildroot, it will help companies/users easily
> bring in their customization into buildroot with just using a defconfig.
>
> Please provide me with feedback in if this idea is feasible/be useful. I'm
> also very open suggestions to help make this a reality.
>
> -------------------------------------------------------------------------
> 	*Ryan J Barnett* / Software Engineer / Platform SW
> MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
> Phone: 319-263-3880 / VPN: 263-3880 _
> __rjbarnet at rockwellcollins.com_ <mailto:rjbarnet@rockwellcollins.com>_
> __www.rockwellcollins.com_ <http://www.rockwellcollins.com/>
>
>
>
>
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC] New feature to handle a custom board folder

[rjbarnet at rockwellcollins.com]

Hi Arnout -

Thanks for the feedback. Please see my responses to your comments below.

Arnout Vandecappelle <arnout@mind.be> wrote on 09/09/2013 11:53:53 AM:

>   Hi Ryan,
>
> On 06/09/13 20:45, Ryan Barnett wrote:
> > All,
> >
> > I would like to introduce a new concept I've got for how the board
> > folder is brought into buildroot.  I propose adding a feature to
> > buildroot to treat the board folder as a package. In treating the 
board
> > folder as a package, it would allow a board folder to be downloaded
> > using currently supported methods of packages (HG/Git/SVN/archives 
from
> > websites). It would be placed as an archive in the DL_DIR and 
extracted
> > to BUILD_DIR/board. I will go into more detail further down about the
> > structure.
>
>   Something like this has been proposed in buildroot several times
> already (including by me), and it has been rejected every time. The main
> reasons for rejection is that it adds a lot of complexity, while the
> current setup is already very simple and feature-complete for the user.
> See below for an explanation why your proposal is very complicated.

During my search of the mailing list archives I did not come across any
of this proposals. I probably didn't come up with the correct search
terms to find this however.

>
>   The simple way to do it is as follows:
>
> - import buildroot in your VCS;
>
> - create directory 'package/mycompany';
>
> - add 'source package/mycompany/Config.in' to package/Config.in;
>
> - create directory 'board/mycompany';
>
> - create file 'configs/mycompanyboard_defconfig'.

Yes - currently this is the way that we are doing this. However, the 
problem
my company faces is that we have to spread buildroot configurations for 
boards over
multiple different repositories. This means that we have around 10-15 
different
check-in's of buildroot. For various reasons, there are things in the 
board
folder that have to be kept in the different repositories. Allowing the 
board
folder to be checked out from a configuration file would help us eliminate 
this
need to have so many different version of buildroot in our environment.

In proposing this idea - I was hoping that I could remove the reasons for 
keeping
so many versions of buildroot in different repositories.

>   Since the diff to mainline is limited to a bunch of new files and a
> single line change in package/Config.in, merging is normally not an 
issue.

In my ideal world, managing the diff to mainline would be easy with GIT 
but
using SVN it not quite as easy though possible.

> > The motivation for this feature is that most of the customization done
> > to buildroot in order to support  a project (board) is usually
> > contained to a board folder. However, in many work environments in
> > which buildroot is used, GIT isn't a preferred version control method
> > (mine included) so being able to track mainline of buildroot is very
> > difficult.
>
>   That's a situation that we didn't expect, I think. In most company
> environments, buildroot is only occasionally updated. Also, you would
> typically require buildroot to be maintained under the "preferred 
version
> control method", because you make some changes - cfr. all the changes
> that you're upstreaming now.

I agree that buildroot be maintained under the "preferred VCS" as things
stand now. However, if the only modifications made to buildroot are
limited to a "board folder", the requirement to maintain the whole
of buildroot in the "preferred VCS" could be removed.

Another motivation for this is that during a development cycle for a board
we would like to easily track mainline. So we could constantly follow
buildroot mainline - using git- until we have to lock it down which is 
when
we go to verification of the board/software where we lock the board down 
to
a specific stable release of buildroot.

As for changes that would make to mainline (minus board folder and custom
packages), it would become easy for us to submit patches to mainline.

We have gone through the development process a couple times with the board
folder method you mentioned above, but our biggest challenge we always run
into is trying to distinguish "our" changes from buildroot's because we 
have
usually merged in some sort of patches from mainline buildroot and have 
trouble distinguishing what changes were ours vs. ones that came from you 
guys.
Since SVN doesn't have a good way to import GIT patches and make sure you 
know that those changes were from an external source (other than to go 
through the SVN Log).

>   Also, keeping the buildroot integration files for proprietary packages
> outside of the buildroot tree makes your repository a bit messy. Putting
> this stuff inside the buildroot tree is cleaner and more understandable.

Agreed with more understandable but reason above could be modification of 
why it might now always be the cleanest.

> >      COMMENTARY: the custom board applications would only appear when
> >      "Board folder contains custom applications" is enabled. The 
application
> >      list from custom board applications would be sourced an 
applications
> > folder
> >      within the custom board folder. These apps wouldn't display until
> > the board
> >      folder is actually extract.
>
>   Because of limitations in the Kconfig tool, this is very difficult to
> do in practice. You need some serious patching of Kconfig, and even then
> it probably doesn't work very well. A patch was submitted that attempted
> to implement this about a year ago IIRC.

This was my fear but I was hoping for the best.

Do you guys have any other ideas for this problem? I would really like 
hear other
ways that could possibly lead to compromise. One idea that is coming to 
mind
now as I type response is to treat the various things that are usually 
board
specific as packages.

I was thinking that packages could be added for the following:
 * Extend Rootfs Overlay to allow downloading of an overlay
 * Allow downloading of configs
 * Allow downloading of kernel patches
 * Allow downloading of u-boot patches
     COMMENTARY: I don't know if there could be a more generic 
implementation
     of this where any application that allows custom patches, could 
utilize
     the downloading of patches from some site.
 * Package that allows building of "custom app"
     COMMENTARY: this custom app would probably build more than just one
     application but could install by copying to a "install" directory 
which
     would then be overlaid to the target.

I understand that this may add more complications to buildroot and deviate 
a
bit from the current "simple" way buildroot works now. I hope though that 
I've
illustrated that this "simple" method isn't always so simple for companies 
that
don't have the ability to use GIT as their preferred VCS.

> > 5. User say "make" and buildroot builds them their system
>
>   This is making things a lot more difficult for the user, compared to
> when everything would just be in the buildroot tree, no?
>
>
>   Regards,
>   Arnout

Bottom line: the motivation  for these changes is to 1) reduce the number 
of
modified version of builroot we have and 2) allow me to take a snapshot of
buildroot and use it without custom modification to it.

At the end of the day I'm probably digging myself into a hole here but I 
hope
I've illustrated my reasoning behind this and hopefuly can make case for
something like this being incorporated into buildroot. Maybe someday I'll 
be
able to achieve my nirvana of having a buildroot where the only changes 
that
need to be made are to a "defconfig".

Thanks,
-Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130909/09db20ef/attachment-0001.html>