From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yann E. MORIN Date: Sun, 25 Sep 2016 16:52:54 +0200 Subject: [Buildroot] [PATCH 00/10 v5] br2-external: support multiple trees at once Message-ID: List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net Hello All! This rather complex series introduces support for using multiple br2-external trees at once. Currently, Buildroot can use what we call a br2-external tree, that is a location where one can store customisations outside of the Buildroot tree: - defconfig files; - packages; - custom make rules and logic; - company-, project- and board-related configurations files, like a kernel config file, a rootfs overlay; - any other content that does not belong to the Buildroot tree... This is very nice and very handy! However, it is not possible to use more than one such br2-external tree at a time; Buildroot only supports using at most one br2-external tree. Having support for using more than one br2-external tree at the same time would allow more flexibility organising those out-of-tree customisations. One could use a br2-external tree to store FLOSS packages not yet in Buildroot and a second br2-external tree for proprietary packages; such a separation would make compliance with FLOSS licenses easier (by allowing distributing only the br2-external tree with FLOSS packages, keeping the proprietary bits private). It could also be used to commonalise br2-external trees from different internal projects, where each would provide a set of extensions (new packages, new boards defconfigs...) that a project could cherry-pick. And countless other possibilities. This series is an attempt at making it possible to use such setups. The series is organised as such: - patch 1 moves the hard-coded sourcing of BR2_EXTERNA/Config.in from the main Config.in into the generated Kconfig snippet. - patch 2 gets rid of our dummy br2-external tree, since we now can include the br2-external only if specified by the user. - patch 3 moves the handling of BR2_EXTERNAL out of the main Makefile into the generated $(BASE_DIR)/.br-external.mk Makefile fragment. So far, this fragment was already generated, but only memorised the value of BR2_EXTERNAL so that it need not be specified at each call; it now also defines a new internal variable, BR2_EXTERNAL_MK, that contains the path to the external.mk file. This allows us to offload all the logic about handling a br2-external tree to a helper script, whether it be about the Kconfig snippet or the Makefile fragment. - patch 4 adds more validation of a br2-external tree than we previously had. - patch 5 introduces a layout change in the br2-external tree: a br2-external tree must now declare its name. This name is used to construct a set of variables unique to that br2-external tree. This change is the foundation for using multiple br2-external trees at once, as their paths will now be in different variables; see below for more explanations. As we now generate the Kconfig snippet and Makefile fragment, we can easily (pre-)compute the variables from the helper script. This change renders existing br2-external trees incompatible with the new layout, as the BR2_EXTERNAL variable is now no longer to be used by br2-external trees, and is no longer available in Kconfig (although it still is available in Makefile, but we can't remove it as we need it for the infra). See path 9 for the associated documentation. - patch 6 is the documentation for the br2-external name. - patch 7 adds a blurb in the manual about how to easily convert an old br2-external tree. This is referenced from the helper script and displayed in case of error. - patch 8 is the core of the change, that adds suport for using multiple br2-external trees at once. Again, because we generate the Kconfig snippet and Makefile fragment, it is very easy to (pre-)compute the per-br2-external tree variables. Each tree is now identified with its name, which is displayed both in the Kconfig menu for that tree, as well as in the comment listing the defconfig files from that tree. - patch 9 is the documentation for mutliple br2-external trees. - patch 10 amkes it opssible for a br2-external tree to override defcongis present in Buildroot, as well as override defconfigs in other br2-external trees (by the order they are listed in BR2_EXTERNAL). The pivotal change in this series is implemented in patch 5, and aims at providing br2-external trees with a way to find their own files, given that their locations at runtime is completely unkown, and can not even be deduced relative to Buildroot's own TOPDIR; this is explained below. Unlike Makefiles, Kconfig does not allow us to redefine a variable at various moments of the parsing, because the values are only valid and acted upon when the complete Kconfig code has been parsed. So, we need a way to provide Config.in files from each br2-external with their actual location. For that, we need different variables, one for each br2-external tree. So, we need a way to uniquely identify each br2-external tree, and we need a stable scheme. We can't use indexes (because they would not be stable across various runs or various ordering of the br2-external trees); we can't use path-based hashing (because paths are not stable accross users/machines/time); we can't use content-based hashing (external.mk and/or Config.in may all be empty, giving the same hash); we can't use hashes altogether because they are not human-friendly. The solution is to require br2-external trees to provide a unique name that is used as a kind of 'name-space' for that tree: we eventually provide, for each br2-external tree, a variable, BR2_EXTERNAL_$(NAME)_PATH, that points to the location of that tree; because $(NAME) is known to the br2-external tree, it also knows BR2_EXTERNAL_$(NAME)_PATH exists, so it can use it, and we export it both in Config.in and external.mk . Therefore, br2-external trees must now use $(BR2_EXTERNAL_$(MY_NAME)_PATH) to refer to their own files. It also prevent them from (accidentally) including files from other br2-external trees. Note: unlike previous iterations, it is now mandatory that a br2-external tree defines its name, the rationale being that it is realtively trivial to upgrade an existing br2-external tree to the new layout. I'd like to thank Arnout for looking almost in realtime at this iteration as I was hacking on it and providing very useful review, suggestions, hints and very constructive criticism. Thanks! :-) And that's about all for this time. ;-) Changes v4 -> v5: - introduce br2-external conversion earlier in the series (Julien) - fix intermediate-state br2-external helper script (Julien) - misc typoes Regards, Yann E. MORIN. The following changes since commit 78af81de59d4f2d739263a3f7c2c53cf6b859cd6 samba4: bump to version 4.4.6 (2016-09-24 16:40:10 +0200) are available in the git repository at: git://git.buildroot.org/~ymorin/git/buildroot.git for you to fetch changes up to 9e5c5a93eadc131329cb9295cb1d518d8064aeef core: allow a br2-external tree to override a defconfig (2016-09-25 00:39:12 +0200) ---------------------------------------------------------------- Yann E. MORIN (10): core: do not hard-code inclusion of br2-external in Kconfig core: get rid of our dummy br2-external tree core: offload handling of BR2_EXTERNAL into the script core/br2-external: validate even more core: introduce per br2-external NAME docs/manual: document the br2-external NAME docs/manual: add appendix to convert old br2-external trees core: add support for multiple br2-external trees docs/manual: document multi br2-external core: allow a br2-external tree to override a defconfig Config.in | 11 -- Makefile | 86 ++++++---- boot/barebox/barebox-aux/Config.in | 3 +- boot/barebox/barebox/Config.in | 3 +- docs/manual/adding-packages-asciidoc.txt | 6 +- docs/manual/adding-packages-perl.txt | 2 +- docs/manual/adding-packages-python.txt | 2 +- docs/manual/appendix.txt | 1 + docs/manual/br2-external-converting.txt | 39 +++++ docs/manual/customize-directory-structure.txt | 13 +- docs/manual/customize-outside-br.txt | 171 +++++++++++++------- docs/manual/customize-packages.txt | 25 +-- package/Makefile.in | 3 +- package/pkg-generic.mk | 4 +- support/dummy-external/Config.in | 0 support/dummy-external/external.mk | 0 support/misc/utils.mk | 4 + support/scripts/br2-external | 219 ++++++++++++++++++++++++++ 18 files changed, 454 insertions(+), 138 deletions(-) create mode 100644 docs/manual/br2-external-converting.txt delete mode 100644 support/dummy-external/Config.in delete mode 100644 support/dummy-external/external.mk create mode 100755 support/scripts/br2-external -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'