[Buildroot] [PATCH v2 0/4] legal-info: save the external-toolchain source archive

[Buildroot] [PATCH v2 0/4] legal-info: save the external-toolchain source archive

[Luca Ceresoli]

Hi,

during the Buildroot Developer Days in October last year, prompted by
Thomas De Schampheleire, we discussed a way to improve 'make
legal-info' to collect the external toolchain source code.

The external toolchain is a special package because
TOOLCHAIN_EXTERNAL_SITE/SOURCE points to a file that is indeed a binary
archive. The "actual" source code is usually provided by toolchain vendors
as a separate file.

During the BDD we stated [1]:

  We'll need to add a FOO_ACTUAL_SOURCES to the infrastructure: if set (to a
  tarball URL) it forces 'make legal-info' to download that tarball and
  store in the tarball directory instead of the FOO_SOURCE tarball.

I submitted a first patchset in January [2], which got some comments
and change requests by Arnout. One of them was to cleanup to trailing
slash in all FOO_SITE variables, for which I submitted a separate
patchset, now in v4 [3].

The two patchset are pretty much indepent anyway, and this one rebases
seamlessly on top of the slash-removal one.

Now it's time to revive this patchset. Changes are quite minor, however.

The core change is in patch 1. I added two variables to the pkg-generic infra:
FOO_ACTUAL_SOURCE_TARBALL and FOO_ACTUAL_SOURCE_SITE, which default to the
value of the good old FOO_SOURCE and FOO_SITE. Then I use the new variables
for the legal-info processing.

If these variables are overridden, which will be the case for external
toolchains, they trigger an extra $(call DOWNLOAD,...) to fetch the actual
package source tarball.

Patches 2 and 3 set the TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL to
appropriate values in external-toolchain.mk for most toolchains.
TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_SITE is never set because it happens to be
equal to TOOLCHAIN_EXTERNAL_SITE in all the cases I implemented.

Patch 4 documents it all in the manual.

The current solution works fine according to all my tests.

There are a few improvements that might be done.

The first limitation it that there's no "actual" version of
FOO_EXTRA_DOWNLOADS. Thus Blackfin toolchains, which use that feature,
cannot be handled in a complete way. Arnout proposed FOO_ACTUAL_SOURCE
to become a list of files rather than a single file. It should work,
but I prefer to add it later and the the bulk of the changes up and
running first.

The other is there's no provision for extracting the license files
from the downloaded tarball, which would be a nice addition. I'd leave
that for the future as well. It's a separate feature, after all.

That's all folks, here's the code. Be aware that some toolchains are
not yet implemented: Blackfin for the reason stated above, musl
because there seem to be no complete source code tarball.

Regards,
Luca

[1] http://www.elinux.org/Buildroot:DeveloperDaysELCE2014#State_of_legal-info_infrastructure.2C_improvements_to_be_made.3F
[2] http://lists.busybox.net/pipermail/buildroot/2015-January/116303.html
[3] http://lists.busybox.net/pipermail/buildroot/2015-October/140813.html

Luca Ceresoli (4):
  legal-info: allow to declare the actual sources for binary packages
  toolchain-external: mass-define actual source tarball for known
    patterns
  toolchain-external: define actual sources for arago toolchains
  docs/manual: document new actual source package variables

 Makefile                                           |  1 -
 docs/manual/adding-packages-generic.txt            | 19 ++++++++++++++++++
 package/pkg-generic.mk                             | 23 +++++++++++++++++-----
 toolchain/toolchain-external/toolchain-external.mk | 12 +++++++++++
 4 files changed, 49 insertions(+), 6 deletions(-)

-- 
1.9.1

[Buildroot] [PATCH v2 1/4] legal-info: allow to declare the actual sources for binary packages

[Luca Ceresoli]

The FOO_SITE/FOO_SOURCE variables usually point to a tarball containing
source code.

For the downloaded external toolchains this is not true, the "source"
tarball actually contains binaries. This is fine for making Buildroot
work, but for legal-info we really want to ship real source code, not
binaries.

Luckily, some (hopefully all) toolchain vendors publish a downloadable
tarball containing the source code counterpart for their binary
packages.

Here we allow the user to declare the URL of this other tarball in the
pair of variables FOO_ACTUAL_SOURCE_TARBALL (by default equal to
FOO_SOURCE) and FOO_ACTUAL_SOURCE_SITE (by default equal to FOO_SITE).
If the "actual source" package can be downloaded from the same
directory as the binary package, then only FOO_ACTUAL_SOURCE_TARBALL
needs to be set.

Note this change is not strictly toolchain-specific: it might be useful
for other packages that happen to ship binaries in the same way.

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---

Changes v1 -> v2:
 - Fix warning text ($(1) instead of "the toolchain source code"),
   reported by Arnout
 - Get rid of the trailing-slash-removal hack [as in
   $$($(2)_ACTUAL_SOURCE_SITE:/=)], there's another patchset to fix that.
 - Fix a typo.
---
 Makefile               |  1 -
 package/pkg-generic.mk | 23 ++++++++++++++++++-----
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/Makefile b/Makefile
index fdbca02..304789f 100644
--- a/Makefile
+++ b/Makefile
@@ -647,7 +647,6 @@ legal-info-prepare: $(LEGAL_INFO_DIR)
 	@$(call legal-manifest,PACKAGE,VERSION,LICENSE,LICENSE FILES,SOURCE ARCHIVE,SOURCE SITE,HOST)
 	@$(call legal-manifest,buildroot,$(BR2_VERSION_FULL),GPLv2+,COPYING,not saved,not saved,HOST)
 	@$(call legal-warning,the Buildroot source code has not been saved)
-	@$(call legal-warning,the toolchain has not been saved)
 	@cp $(BR2_CONFIG) $(LEGAL_INFO_DIR)/buildroot.config
 
 legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p)-all-legal-info) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5201fca..13654dd 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -708,12 +708,18 @@ ifneq ($$($(2)_SITE_METHOD),local)
 ifneq ($$($(2)_SITE_METHOD),override)
 # Packages that have a tarball need it downloaded beforehand
 $(1)-legal-info: $(1)-source $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
-$(2)_MANIFEST_TARBALL = $$($(2)_SOURCE)
-$(2)_MANIFEST_SITE = $$(call qstrip,$$($(2)_SITE))
 endif
 endif
 endif
 
+# If FOO_ACTUAL_SOURCE_TARBALL is explicitly defined, it means FOO_SOURCE is
+# indeed a binary (e.g. external toolchain) and FOO_ACTUAL_SOURCE_TARBALL/_SITE
+# point to the actual sources tarball. Use the actual sources for legal-info.
+# For most packages the FOO_SITE/FOO_SOURCE pair points to real source code,
+# so these are the defaults for FOO_ACTUAL_*.
+$(2)_ACTUAL_SOURCE_TARBALL ?= $$($(2)_SOURCE)
+$(2)_ACTUAL_SOURCE_SITE    ?= $$(call qstrip,$$($(2)_SITE))
+
 # legal-info: produce legally relevant info.
 $(1)-legal-info:
 # Packages without a source are assumed to be part of Buildroot, skip them.
@@ -744,13 +750,20 @@ else
 # Other packages
 
 ifeq ($$($(2)_REDISTRIBUTE),YES)
+ifeq ($$($(2)_ACTUAL_SOURCE_TARBALL),)
+	@$(call legal-warning,$(1): the source code has not been saved)
+else
+ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
+	$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
+endif
 # Copy the source tarball (just hardlink if possible)
-	@cp -l $$(DL_DIR)/$$($(2)_SOURCE) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \
-	   cp $$(DL_DIR)/$$($(2)_SOURCE) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
+	@cp -l $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \
+	    cp $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
+endif # ($$($(2)_ACTUAL_SOURCE_TARBALL),)
 endif # redistribute
 
 endif # other packages
-	@$$(call legal-manifest,$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_LICENSE),$$($(2)_MANIFEST_LICENSE_FILES),$$($(2)_MANIFEST_TARBALL),$$($(2)_MANIFEST_SITE),$$(call UPPERCASE,$(4)))
+	@$$(call legal-manifest,$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_LICENSE),$$($(2)_MANIFEST_LICENSE_FILES),$$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_ACTUAL_SOURCE_SITE),$$(call UPPERCASE,$(4)))
 endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 	$$(foreach hook,$$($(2)_POST_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
 
-- 
1.9.1

[Buildroot] [PATCH v2 2/4] toolchain-external: mass-define actual source tarball for known patterns

[Luca Ceresoli]

For some external toolchain vendors the actual source code URL can be simply
derived from the binary file URL.

Here we obtain TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL for all Mentor and
Linaro toolchains with a few $(subst) calls.

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---

Changes v1 -> v2:
 - Wrap a long line as much as possible (Arnout).
---
 toolchain/toolchain-external/toolchain-external.mk | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index 79afdaa..fecf3de 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -422,6 +422,16 @@ TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)
 BR_NO_CHECK_HASH_FOR += $(TOOLCHAIN_EXTERNAL_SOURCE)
 endif
 
+# Some toolchain vendors have a regular file naming pattern.
+# For them, mass-define _ACTUAL_SOURCE_TARBALL based _SITE.
+ifneq ($(findstring sourcery.mentor.com/public/gnu_toolchain,$(TOOLCHAIN_EXTERNAL_SITE)),)
+TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL ?= \
+	$(subst -i686-pc-linux-gnu.tar.bz2,.src.tar.bz2,$(subst -i686-pc-linux-gnu-i386-linux.tar.bz2,-i686-pc-linux-gnu.src.tar.bz2,$(TOOLCHAIN_EXTERNAL_SOURCE)))
+else ifneq ($(findstring http://releases.linaro.org,$(TOOLCHAIN_EXTERNAL_SITE)),)
+TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL ?= \
+	$(subst _linux.tar.xz,_src.tar.bz2,$(TOOLCHAIN_EXTERNAL_SOURCE))
+endif
+
 # In fact, we don't need to download the toolchain, since it is already
 # available on the system, so force the site and source to be empty so
 # that nothing will be downloaded/extracted.
-- 
1.9.1

[Buildroot] [PATCH v2 3/4] toolchain-external: define actual sources for arago toolchains

[Luca Ceresoli]

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index fecf3de..950f75c 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -296,6 +296,7 @@ TOOLCHAIN_EXTERNAL_SOURCE = arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-
 else ifeq ($(BR2_TOOLCHAIN_EXTERNAL_ARAGO_ARMV7A_201109),y)
 TOOLCHAIN_EXTERNAL_SITE = http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/exports
 TOOLCHAIN_EXTERNAL_SOURCE = arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2
+TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL = arago-toolchain-2011.09-sources.tar.bz2
 define TOOLCHAIN_EXTERNAL_FIXUP_CMDS
 	mv $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/armv7a/* $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/
 	rm -rf $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/
@@ -303,6 +304,7 @@ endef
 else ifeq ($(BR2_TOOLCHAIN_EXTERNAL_ARAGO_ARMV5TE_201109),y)
 TOOLCHAIN_EXTERNAL_SITE = http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/exports
 TOOLCHAIN_EXTERNAL_SOURCE = arago-2011.09-armv5te-linux-gnueabi-sdk.tar.bz2
+TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL = arago-toolchain-2011.09-sources.tar.bz2
 define TOOLCHAIN_EXTERNAL_FIXUP_CMDS
 	mv $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/armv5te/* $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/
 	rm -rf $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/
-- 
1.9.1

[Buildroot] [PATCH v2 4/4] docs/manual: document new actual source package variables

[Luca Ceresoli]

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 docs/manual/adding-packages-generic.txt | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 40798c0..9d6401f 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -388,6 +388,25 @@ information is (assuming the package name is +libfoo+) :
   to let you know, and +not saved+ will appear in the +license files+ field
   of the manifest file for this package.
 
+* +LIBFOO_ACTUAL_SOURCE_TARBALL+ only applies to packages whose
+  +LIBFOO_SITE+ / +LIBTOO_SOURCE+ pair points to an archive that does
+  not actually contain source code, but binary code. This a very
+  uncommon case, only known to apply to external toolchains which come
+  already compiled, although theoretically it might apply to other
+  packages. In such cases a separate tarball is usually available with
+  the actual source code. Set +LIBFOO_ACTUAL_SOURCE_TARBALL+ to the
+  name of the actual source code archive and Buildroot will download
+  it and use it when you run +make legal-info+ to collect
+  legally-relevant material.  Note this file will not be downloaded
+  during regular builds nor by +make source+.
+
+* +LIBFOO_ACTUAL_SOURCE_SITE+ provides the location of the actual
+  source tarball. The default value is +LIBFOO_SITE+, so you don't
+  need to set this variable if the binary and source archives are
+  hosted on the same directory.  If +LIBFOO_ACTUAL_SOURCE_TARBALL+ is
+  not set, it doesn't make sense to define
+  +LIBFOO_ACTUAL_SOURCE_SITE+.
+
 * +LIBFOO_REDISTRIBUTE+ can be set to +YES+ (default) or +NO+ to indicate if
   the package source code is allowed to be redistributed. Set it to +NO+ for
   non-opensource packages: Buildroot will not save the source code for this
-- 
1.9.1

[Buildroot] [PATCH v2 0/4] legal-info: save the external-toolchain source archive

[Thomas Petazzoni]

Dear Luca Ceresoli,

On Sat,  3 Oct 2015 23:18:17 +0200, Luca Ceresoli wrote:

> Luca Ceresoli (4):
>   legal-info: allow to declare the actual sources for binary packages
>   toolchain-external: mass-define actual source tarball for known
>     patterns
>   toolchain-external: define actual sources for arago toolchains
>   docs/manual: document new actual source package variables

All four patches applied, with only the removal of unneeded warning in
the first patch.

We noted several things during the review:

 * The ifneq ($$($(2)_SITE_METHOD),override) test will always be false
   because SITE_METHOD is never set to "override". This part of the
   code should be reworked to simply test $(2)_OVERRIDE_SRCDIR instead:
   when empty, it indicates that the packages is neither a "local"
   site-method package nor a package that has its source directory
   overridden.

 * Maybe the fetching of the "actual source" could be deferred to a
   separate $(1)-legal-info-source target, to refactor with the
   existing dependency of $(1)-legal-info on $(1)-source.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com