[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Yann E. MORIN]

Currently, the wording in the manual instructs the user to generate
tarball from "the contents of the +output/host+ directory".

This is pretty confusing, because taken literally, this would amount to
runing a command like:

    tar cf my-sdk.tar -C output/host/ .

This creates a tarbomb [0], which is very bad practice, because when
extracted, it creates mutiple files in the current directory.

One one really want to do, is create a tarball of the host/ directory,
with something like:

    tar cf my-sdk.tar -C output host/

However, this is not much better, because the top-most dirdctory would
have a very common name, host/, which is pretty easy to get conflict
with.

So, we fix that mess by creating the archive ourselves, giving it and
the top-most directory a recogniseable name, based on the target tuple
and the Buildroot version.

Since this is an output file, we located it in the images/ directory.

Update the manual accordignly.

Speaking of the manual.. It was referring to "output/host/", but that
is only valid for in-tree builds. For out-of-tree builds, this is just
"host/". To avoid confusion, use the name of the vairiable $(HOST_DIR),
which, fortunately, happens to be valid in both cases.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Wolfgang Grandegger <wg@grandegger.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Stefan Becker <chemobejk@gmail.com>
---
 Makefile                                  |  7 ++++++-
 docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++---------
 2 files changed, 23 insertions(+), 10 deletions(-)

diff --git a/Makefile b/Makefile
index 4b998bdb65..3f460f8d92 100644
--- a/Makefile
+++ b/Makefile
@@ -574,13 +574,18 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: target-post-image
 
 .PHONY: sdk
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
+		.
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
diff --git a/docs/manual/using-buildroot-toolchain.txt b/docs/manual/using-buildroot-toolchain.txt
index 3246dc2411..d1490b4c38 100644
--- a/docs/manual/using-buildroot-toolchain.txt
+++ b/docs/manual/using-buildroot-toolchain.txt
@@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in
 +output/host/bin/+ to your PATH environment variable and then to
 use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc.
 
-It is possible to relocate the toolchain, this allows to distribute
-the toolchain to other developers to build applications for your
-target. To achieve this:
+Alternatively, Buildroot can also export the toolchain and all the
+development files of all selected packages, as an SDK, by running the
+command +make sdk+. This generates a tarball of the content of the host
+directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
+and located in the output directory +$(BINARIES_DIR)+.
 
-* run +make sdk+, which prepares the toolchain to be relocatable;
-* tarball the contents of the +output/host+ directory;
-* distribute the resulting tarball.
+This tarball can then be distributed to application developpers, when
+they want to develop their applications that are not (yet) packaged as
+a Buildroot package.
 
-Once the toolchain is installed to the new location, the user must run
-the +relocate-sdk.sh+ script to make sure all paths are updated with
-the new location.
+Upon extracting the SDK tarball, the user must run the script
++relocate-sdk.sh+, to make sure all paths are updated with the new
+location.
 
+.Note
+This SDK can not be re-used as an external toolchain, because it
+contains pre-built libraries that could be conflicting with the ones
+packaged in Buildroot (e.g. when an old SDK would be re-used with a
+newer Buildroot version), unless it was built from a configuration
+with no package enabled.
-- 
2.14.1

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Yann E. MORIN]

All,

On 2018-06-09 23:06 +0200, Yann E. MORIN spake thusly:
> Currently, the wording in the manual instructs the user to generate
> tarball from "the contents of the +output/host+ directory".
> 
> This is pretty confusing, because taken literally, this would amount to
> runing a command like:
> 
>     tar cf my-sdk.tar -C output/host/ .
> 
> This creates a tarbomb [0], which is very bad practice, because when

I forgot to add the reference to the tarbomb:
    https://en.wikipedia.org/wiki/Tar_(computing)#Tarbomb

Regards,
Yann E. MORIN.

> extracted, it creates mutiple files in the current directory.
> 
> One one really want to do, is create a tarball of the host/ directory,
> with something like:
> 
>     tar cf my-sdk.tar -C output host/
> 
> However, this is not much better, because the top-most dirdctory would
> have a very common name, host/, which is pretty easy to get conflict
> with.
> 
> So, we fix that mess by creating the archive ourselves, giving it and
> the top-most directory a recogniseable name, based on the target tuple
> and the Buildroot version.
> 
> Since this is an output file, we located it in the images/ directory.
> 
> Update the manual accordignly.
> 
> Speaking of the manual.. It was referring to "output/host/", but that
> is only valid for in-tree builds. For out-of-tree builds, this is just
> "host/". To avoid confusion, use the name of the vairiable $(HOST_DIR),
> which, fortunately, happens to be valid in both cases.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Wolfgang Grandegger <wg@grandegger.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Stefan Becker <chemobejk@gmail.com>
> ---
>  Makefile                                  |  7 ++++++-
>  docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++---------
>  2 files changed, 23 insertions(+), 10 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 4b998bdb65..3f460f8d92 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -574,13 +574,18 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
>  world: target-post-image
>  
>  .PHONY: sdk
> -sdk: world
> +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
>  	@$(call MESSAGE,"Rendering the SDK relocatable")
>  	$(TOPDIR)/support/scripts/fix-rpath host
>  	$(TOPDIR)/support/scripts/fix-rpath staging
>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>  	mkdir -p $(HOST_DIR)/share/buildroot
>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> +	$(Q)mkdir -p $(BINARIES_DIR)
> +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
> +		-C $(HOST_DIR) \
> +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
> +		.
>  
>  # Populating the staging with the base directories is handled by the skeleton package
>  $(STAGING_DIR):
> diff --git a/docs/manual/using-buildroot-toolchain.txt b/docs/manual/using-buildroot-toolchain.txt
> index 3246dc2411..d1490b4c38 100644
> --- a/docs/manual/using-buildroot-toolchain.txt
> +++ b/docs/manual/using-buildroot-toolchain.txt
> @@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in
>  +output/host/bin/+ to your PATH environment variable and then to
>  use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc.
>  
> -It is possible to relocate the toolchain, this allows to distribute
> -the toolchain to other developers to build applications for your
> -target. To achieve this:
> +Alternatively, Buildroot can also export the toolchain and all the
> +development files of all selected packages, as an SDK, by running the
> +command +make sdk+. This generates a tarball of the content of the host
> +directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
> +and located in the output directory +$(BINARIES_DIR)+.
>  
> -* run +make sdk+, which prepares the toolchain to be relocatable;
> -* tarball the contents of the +output/host+ directory;
> -* distribute the resulting tarball.
> +This tarball can then be distributed to application developpers, when
> +they want to develop their applications that are not (yet) packaged as
> +a Buildroot package.
>  
> -Once the toolchain is installed to the new location, the user must run
> -the +relocate-sdk.sh+ script to make sure all paths are updated with
> -the new location.
> +Upon extracting the SDK tarball, the user must run the script
> ++relocate-sdk.sh+, to make sure all paths are updated with the new
> +location.
>  
> +.Note
> +This SDK can not be re-used as an external toolchain, because it
> +contains pre-built libraries that could be conflicting with the ones
> +packaged in Buildroot (e.g. when an old SDK would be re-used with a
> +newer Buildroot version), unless it was built from a configuration
> +with no package enabled.
> -- 
> 2.14.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Thomas Petazzoni]

Hello,

On Sat,  9 Jun 2018 23:06:07 +0200, Yann E. MORIN wrote:
> Currently, the wording in the manual instructs the user to generate
> tarball from "the contents of the +output/host+ directory".
> 
> This is pretty confusing, because taken literally, this would amount to
> runing a command like:
> 
>     tar cf my-sdk.tar -C output/host/ .
> 
> This creates a tarbomb [0], which is very bad practice, because when
> extracted, it creates mutiple files in the current directory.
> 
> One one really want to do, is create a tarball of the host/ directory,

"One one"

> with something like:
> 
>     tar cf my-sdk.tar -C output host/
> 
> However, this is not much better, because the top-most dirdctory would

directory

> have a very common name, host/, which is pretty easy to get conflict
> with.
> 
> So, we fix that mess by creating the archive ourselves, giving it and

remove the final "and"

> the top-most directory a recogniseable name, based on the target tuple
> and the Buildroot version.
> 
> Since this is an output file, we located it in the images/ directory.

located -> locate

or maybe "place", "store" ?

> 
> Update the manual accordignly.

accordingly

> 
> Speaking of the manual.. It was referring to "output/host/", but that
> is only valid for in-tree builds. For out-of-tree builds, this is just
> "host/". To avoid confusion, use the name of the vairiable $(HOST_DIR),

variable

> which, fortunately, happens to be valid in both cases.

>  .PHONY: sdk
> -sdk: world
> +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
>  	@$(call MESSAGE,"Rendering the SDK relocatable")
>  	$(TOPDIR)/support/scripts/fix-rpath host
>  	$(TOPDIR)/support/scripts/fix-rpath staging
>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>  	mkdir -p $(HOST_DIR)/share/buildroot
>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> +	$(Q)mkdir -p $(BINARIES_DIR)
> +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
> +		-C $(HOST_DIR) \
> +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
> +		.

Generally, I am fine with the principle, I believe it indeed makes
sense to provide a tarball that is ready to use.

I was a bit concerned about backward compatibility behavior for people
already using "make sdk". But in fact your change is fine from this
point of view: if people have scripts today that run "make sdk" and
create a tarball from output/host, they will still work fine.

> -It is possible to relocate the toolchain, this allows to distribute
> -the toolchain to other developers to build applications for your
> -target. To achieve this:
> +Alternatively, Buildroot can also export the toolchain and all the
> +development files of all selected packages,

"all the development files" -> "the development files", otherwise the
repetition of "all" is a bit annoying.

> as an SDK, by running the
> +command +make sdk+. This generates a tarball of the content of the host
> +directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
> +and located in the output directory +$(BINARIES_DIR)+.
>  
> -* run +make sdk+, which prepares the toolchain to be relocatable;
> -* tarball the contents of the +output/host+ directory;
> -* distribute the resulting tarball.
> +This tarball can then be distributed to application developpers, when
> +they want to develop their applications that are not (yet) packaged as
> +a Buildroot package.
>  
> -Once the toolchain is installed to the new location, the user must run
> -the +relocate-sdk.sh+ script to make sure all paths are updated with
> -the new location.
> +Upon extracting the SDK tarball, the user must run the script
> ++relocate-sdk.sh+, to make sure all paths are updated with the new
> +location.
>  
> +.Note
> +This SDK can not be re-used as an external toolchain, because it
> +contains pre-built libraries that could be conflicting with the ones
> +packaged in Buildroot (e.g. when an old SDK would be re-used with a
> +newer Buildroot version), unless it was built from a configuration
> +with no package enabled.

Looks good otherwise. Thanks!

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Yann E. MORIN]

Thomas, All,

On 2018-06-10 08:03 +0200, Thomas Petazzoni spake thusly:
> On Sat,  9 Jun 2018 23:06:07 +0200, Yann E. MORIN wrote:
[--SNIP--]
> > So, we fix that mess by creating the archive ourselves, giving it and
> remove the final "and"

Here, 'it' refers to 'the archive' listed just before. We want to give
'the archive' and 'the top-most directory'. So, this is really valid
english grammar. ;-)

I'll rephrase to avoid confusion:

    So, we fix that mess by giving the top-most directory a
    recogniseable name, based on the target tuple and the Buildroot
    version, which we also use as the name of the archive (suffixed
    with the usual +.tar.gz+.)

> > the top-most directory a recogniseable name, based on the target tuple
> > and the Buildroot version.
> > 
> > Since this is an output file, we located it in the images/ directory.
> located -> locate
> or maybe "place", "store" ?

OK.

> >  .PHONY: sdk
> > -sdk: world
> > +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
> >  	@$(call MESSAGE,"Rendering the SDK relocatable")
> >  	$(TOPDIR)/support/scripts/fix-rpath host
> >  	$(TOPDIR)/support/scripts/fix-rpath staging
> >  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> >  	mkdir -p $(HOST_DIR)/share/buildroot
> >  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> > +	$(Q)mkdir -p $(BINARIES_DIR)
> > +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
> > +		-C $(HOST_DIR) \
> > +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
> > +		.
> 
> Generally, I am fine with the principle, I believe it indeed makes
> sense to provide a tarball that is ready to use.
> 
> I was a bit concerned about backward compatibility behavior for people
> already using "make sdk". But in fact your change is fine from this
> point of view: if people have scripts today that run "make sdk" and
> create a tarball from output/host, they will still work fine.
> 
> > -It is possible to relocate the toolchain, this allows to distribute
> > -the toolchain to other developers to build applications for your
> > -target. To achieve this:
> > +Alternatively, Buildroot can also export the toolchain and all the
> > +development files of all selected packages,
> 
> "all the development files" -> "the development files", otherwise the
> repetition of "all" is a bit annoying.

ACK.

> Looks good otherwise. Thanks!

OK, thanks for the review! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Arnout Vandecappelle]

On 09-06-18 23:06, Yann E. MORIN wrote:
[snip]
> So, we fix that mess by creating the archive ourselves, giving it and
> the top-most directory a recogniseable name, based on the target tuple
                           recognisable
> and the Buildroot version.

 Great idea!

> Since this is an output file, we located it in the images/ directory.
> 
> Update the manual accordignly.
                    accordingly
> 
> Speaking of the manual.. It was referring to "output/host/", but that
> is only valid for in-tree builds. For out-of-tree builds, this is just
> "host/". To avoid confusion, use the name of the vairiable $(HOST_DIR),
                                                   variable

> which, fortunately, happens to be valid in both cases.

 Small nit about this part: this is the "user" section of the manual, and
HOST_DIR hasn't been introduced there. So I'm not so sure that HOST_DIR is such
an improvement. If you want to be fully accurate, you could use "output/host or
$(O)/host".

 Alternatively, we could introduce HOST_DIR (and STAGING and TARGET probably) in
the O= section in common-usage.txt.


> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Wolfgang Grandegger <wg@grandegger.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Stefan Becker <chemobejk@gmail.com>
> ---
>  Makefile                                  |  7 ++++++-
>  docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++---------
>  2 files changed, 23 insertions(+), 10 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 4b998bdb65..3f460f8d92 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -574,13 +574,18 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
>  world: target-post-image
>  
>  .PHONY: sdk
> -sdk: world
> +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
>  	@$(call MESSAGE,"Rendering the SDK relocatable")
>  	$(TOPDIR)/support/scripts/fix-rpath host
>  	$(TOPDIR)/support/scripts/fix-rpath staging
>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>  	mkdir -p $(HOST_DIR)/share/buildroot
>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> +	$(Q)mkdir -p $(BINARIES_DIR)
> +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \

 Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
probably gives a significant hit to the build time. But since it's explicit in
'make sdk' I don't mind that much.

> +		-C $(HOST_DIR) \
> +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \

 Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
variable?

 And let's do some bikeshedding about the name... I don't like the . between sdk
and the tuple. Also, I think it's better if sdk is at the beginning instead of
buildroot. Or maybe even the tuple, that comes closer to how you invoke the
compiler itself. Finally, Debian uses _ to distinguish the different parts in
their .deb filenames, because both the package name and the version usually
contain dashes and it may be hard to distinguish afterwards. So, I propose:

$(GNU_TARGET_NAME)_sdk-buildroot-$(BR2_VERSION_FULL).

> +		.

 It's a really stupid nit, but I don't like this single . isolated on a line,
because it's so easy to miss. So perhaps you could make it:

	--transform=... \
	-C $(HOST_DIR) .

(I also like the -C to be just before the files argument).

>  
>  # Populating the staging with the base directories is handled by the skeleton package
>  $(STAGING_DIR):
> diff --git a/docs/manual/using-buildroot-toolchain.txt b/docs/manual/using-buildroot-toolchain.txt
> index 3246dc2411..d1490b4c38 100644
> --- a/docs/manual/using-buildroot-toolchain.txt
> +++ b/docs/manual/using-buildroot-toolchain.txt
> @@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in
>  +output/host/bin/+ to your PATH environment variable and then to
>  use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc.
>  
> -It is possible to relocate the toolchain, this allows to distribute
> -the toolchain to other developers to build applications for your
> -target. To achieve this:
> +Alternatively, Buildroot can also export the toolchain and all the
> +development files of all selected packages, as an SDK, by running the
> +command +make sdk+. This generates a tarball of the content of the host
> +directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
> +and located in the output directory +$(BINARIES_DIR)+.
>  
> -* run +make sdk+, which prepares the toolchain to be relocatable;
> -* tarball the contents of the +output/host+ directory;
> -* distribute the resulting tarball.
> +This tarball can then be distributed to application developpers, when
                                                       developers

> +they want to develop their applications that are not (yet) packaged as
> +a Buildroot package.
>  
> -Once the toolchain is installed to the new location, the user must run
> -the +relocate-sdk.sh+ script to make sure all paths are updated with
> -the new location.
> +Upon extracting the SDK tarball, the user must run the script
> ++relocate-sdk.sh+, to make sure all paths are updated with the new

 Maybe make it explicit that this script is included at the top directory of the
SDK.


> +location.
>  
> +.Note
> +This SDK can not be re-used as an external toolchain, because it
> +contains pre-built libraries that could be conflicting with the ones
> +packaged in Buildroot (e.g. when an old SDK would be re-used with a
> +newer Buildroot version), unless it was built from a configuration
> +with no package enabled.

 This part really should be a separate patch, it has nothing to do with the SDK
tarball. Also, I disagree with the statement: it is perfectly usable as an
external toolchain. The problem is: any library packages included in the SDK are
not going to be used by the new Buildroot build. There is no chance of conflict,
though, since the external toolchain logic only copies over the required libraries.

 Regards,
 Arnout


> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Trent Piepho]

On Sun, 2018-06-10 at 23:21 +0200, Arnout Vandecappelle wrote:
>  
> >  .PHONY: sdk
> > -sdk: world
> > +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
> >  	@$(call MESSAGE,"Rendering the SDK relocatable")
> >  	$(TOPDIR)/support/scripts/fix-rpath host
> >  	$(TOPDIR)/support/scripts/fix-rpath staging
> >  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> >  	mkdir -p $(HOST_DIR)/share/buildroot
> >  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> > +	$(Q)mkdir -p $(BINARIES_DIR)
> > +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
> 
>  Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
> probably gives a significant hit to the build time. But since it's explicit in
> 'make sdk' I don't mind that much.
> 

I'm using the SDK with Docker.  It can add tar.gz files into the
container, but now tar.xz.  Having to recompress the archive would be a
significant increase in build time.


> > +		-C $(HOST_DIR) \
> > +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
> 
>  Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
> variable?

So having extracted an SDK, in an automated CI script, how does one use
it?  I'll add it to the Docker container:

ARG sdk_file
ENV SDK=/mnt/sdk
ADD --chown=user:user ${sdk_file} ${SDK}

Now it needs to be relocated.  Something like:

RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh

Is the problem with using this tarball apparent?  The dockerfile needs
to be hand edited on every single build to use the correct path to the
SDK.

I've already added a "sdk-tar" to my external.mk to do this, but used
the tarbomb approach.  While annoying for those who extract it in their
home dir, it's much easier to use in an automated processes since there
is no unknown path name component.

It is good if archives that are going to be used in an automated
process do not contain unpredictable path names!

Now one might say that I can just make a tarball with a different
process.  But that would mean creating two SDK tarballs, since there is
no way to prep the host dir for making a SDK tar without also making
the tarball.  That's a significant amount of wasted time.

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Thomas Petazzoni]

Hello,

On Mon, 11 Jun 2018 17:20:57 +0000, Trent Piepho wrote:

> Is the problem with using this tarball apparent?  The dockerfile needs
> to be hand edited on every single build to use the correct path to the
> SDK.
> 
> I've already added a "sdk-tar" to my external.mk to do this, but used
> the tarbomb approach.  While annoying for those who extract it in their
> home dir, it's much easier to use in an automated processes since there
> is no unknown path name component.

You want to use the --strip-components=1 option of tar, which Buildroot
is already using when extracting the tarballs of upstream packages into
the output/build/<package>-<version>/ directories.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Trent Piepho]

On Mon, 2018-06-11 at 21:01 +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Mon, 11 Jun 2018 17:20:57 +0000, Trent Piepho wrote:
> 
> > Is the problem with using this tarball apparent?  The dockerfile needs
> > to be hand edited on every single build to use the correct path to the
> > SDK.
> > 
> > I've already added a "sdk-tar" to my external.mk to do this, but used
> > the tarbomb approach.  While annoying for those who extract it in their
> > home dir, it's much easier to use in an automated processes since there
> > is no unknown path name component.
> 
> You want to use the --strip-components=1 option of tar, which Buildroot
> is already using when extracting the tarballs of upstream packages into
> the output/build/<package>-<version>/ directories.

Which is great when using GNU tar that has that option.  But Docker
does not.

I can make a Docker friendly tarball myself, as I do now.  But after
this patch there is no way to do that without also making an unused
non-Docker friendly tarball.  Which is pretty costly on VMs with poor
io performance.

Some ideas to avoid this:
1. Make tarball generation a new target instead of "sdk".
2. Make tarball generation the "sdk" target but add another target that
does what "sdk" used to do.
3. Allow the tarball path prefix to be specified in some way, so that
it is easier to inject into an automated process that uses the SDK. 
E.g., it can be defined as a constant in the defconfig, or passed from
a higher level to both buildroot when the SDK is made and to whatever
uses the SDK.

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Thomas Petazzoni]

Hello,

On Mon, 11 Jun 2018 22:10:55 +0000, Trent Piepho wrote:

> Some ideas to avoid this:
> 1. Make tarball generation a new target instead of "sdk".
> 2. Make tarball generation the "sdk" target but add another target that
> does what "sdk" used to do.

For both (1) and (2), the proposal of Yann is to have "make sdk" do
what it does today + some new thing.

So there is nothing that prevents to continue using "make sdk" as
you're using it today, and ignore the new tarball that is generated.
I.e, the change Yann is proposing is backward compatible: "make sdk" is
still doing what it used to do.

> 3. Allow the tarball path prefix to be specified in some way, so that
> it is easier to inject into an automated process that uses the SDK. 
> E.g., it can be defined as a constant in the defconfig, or passed from
> a higher level to both buildroot when the SDK is made and to whatever
> uses the SDK.

That is another option indeed. Perhaps that makes sense, yes.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Trent Piepho]

On Tue, 2018-06-12 at 11:57 +0200, Thomas Petazzoni wrote:
> Hello,
> 
> On Mon, 11 Jun 2018 22:10:55 +0000, Trent Piepho wrote:
> 
> > Some ideas to avoid this:
> > 1. Make tarball generation a new target instead of "sdk".
> > 2. Make tarball generation the "sdk" target but add another target that
> > does what "sdk" used to do.
> 
> For both (1) and (2), the proposal of Yann is to have "make sdk" do
> what it does today + some new thing.
> 
> So there is nothing that prevents to continue using "make sdk" as
> you're using it today, and ignore the new tarball that is generated.
> I.e, the change Yann is proposing is backward compatible: "make sdk" is
> still doing what it used to do

But it will be much slower now, as creating and compressing gigabyte
tarballs takes a while.  If there were a prep-sdk target, then I could
use that.

> > 3. Allow the tarball path prefix to be specified in some way, so that
> > it is easier to inject into an automated process that uses the SDK. 
> > E.g., it can be defined as a constant in the defconfig, or passed from
> > a higher level to both buildroot when the SDK is made and to whatever
> > uses the SDK.
> 
> That is another option indeed. Perhaps that makes sense, yes.

I'd find it better, as I have jobs that make different types of SDK.  I
could give them different prefixes, but not have the prefix change from
commit to commit.

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Arnout Vandecappelle]

On 11-06-18 19:20, Trent Piepho wrote:
> On Sun, 2018-06-10 at 23:21 +0200, Arnout Vandecappelle wrote:
>>  
>>>  .PHONY: sdk
>>> -sdk: world
>>> +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
>>>  	@$(call MESSAGE,"Rendering the SDK relocatable")
>>>  	$(TOPDIR)/support/scripts/fix-rpath host
>>>  	$(TOPDIR)/support/scripts/fix-rpath staging
>>>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>>>  	mkdir -p $(HOST_DIR)/share/buildroot
>>>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
>>> +	$(Q)mkdir -p $(BINARIES_DIR)
>>> +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
>>
>>  Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
>> probably gives a significant hit to the build time. But since it's explicit in
>> 'make sdk' I don't mind that much.
>>
> 
> I'm using the SDK with Docker.  It can add tar.gz files into the
> container, but now tar.xz.  Having to recompress the archive would be a
> significant increase in build time.

 Duh. Calling tar inside the container explicitly, rather than letting docker to
the extract directly, isn't too much of a problem either, no?

> 
> 
>>> +		-C $(HOST_DIR) \
>>> +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
>>
>>  Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
>> variable?
> 
> So having extracted an SDK, in an automated CI script, how does one use
> it?  I'll add it to the Docker container:
> 
> ARG sdk_file
> ENV SDK=/mnt/sdk
> ADD --chown=user:user ${sdk_file} ${SDK}

 My docker-fu is not too great, but you can do something similar to expose the
tarball directly, and then do

RUN tar -xf ${sdk_file} --strip-components=1

no?


> Now it needs to be relocated.  Something like:
> 
> RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh

 Here you could to

RUN cd ${SDK}/buildroot-sdk.*/ && ./relocate-sdk.sh

> 
> Is the problem with using this tarball apparent?  The dockerfile needs
> to be hand edited on every single build to use the correct path to the
> SDK.
> 
> I've already added a "sdk-tar" to my external.mk to do this, but used
> the tarbomb approach.  While annoying for those who extract it in their
> home dir, it's much easier to use in an automated processes since there
> is no unknown path name component.
> 
> It is good if archives that are going to be used in an automated
> process do not contain unpredictable path names!
> 
> Now one might say that I can just make a tarball with a different
> process.  But that would mean creating two SDK tarballs, since there is
> no way to prep the host dir for making a SDK tar without also making
> the tarball.  That's a significant amount of wasted time.

 That's what Thomas proposed in his last mail, and I agree it's not great.

 That said, if you're building an SDK, it's probably because you're not changing
it too often. As long as you're developing the Buildroot OS itself, there's
generally no point generating the SDK. If you really are closely co-developing
your application and the Buildroot OS, you're probably better of including the
application as a package rather than building it externally.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Trent Piepho]

On Tue, 2018-06-12 at 15:30 +0200, Arnout Vandecappelle wrote:
> 
> On 11-06-18 19:20, Trent Piepho wrote:
> > On Sun, 2018-06-10 at 23:21 +0200, Arnout Vandecappelle wrote:
> > >  
> > > >  .PHONY: sdk
> > > > -sdk: world
> > > > +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
> > > >  	@$(call MESSAGE,"Rendering the SDK relocatable")
> > > >  	$(TOPDIR)/support/scripts/fix-rpath host
> > > >  	$(TOPDIR)/support/scripts/fix-rpath staging
> > > >  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> > > >  	mkdir -p $(HOST_DIR)/share/buildroot
> > > >  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> > > > +	$(Q)mkdir -p $(BINARIES_DIR)
> > > > +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
> > > 
> > >  Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
> > > probably gives a significant hit to the build time. But since it's explicit in
> > > 'make sdk' I don't mind that much.
> > > 
> > 
> > I'm using the SDK with Docker.  It can add tar.gz files into the
> > container, but now tar.xz.  Having to recompress the archive would be a
> > significant increase in build time.
> 
>  Duh. Calling tar inside the container explicitly, rather than letting docker to
> the extract directly, isn't too much of a problem either, no?

I believe COPY + RUN ends up being more layers than just ADD.  Though
current docker supports xz and bz2 so it would work to use a better
compression.

But, in the tests I've done on our vsphere system, using gz was still
fastest.  I'm more interested in making the CI process faster than in
saving disk space.

> 
> > > > +		-C $(HOST_DIR) \
> > > > +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
> > > 
> > >  Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
> > > variable?
> > 
> > So having extracted an SDK, in an automated CI script, how does one use
> > it?  I'll add it to the Docker container:
> > 
> > ARG sdk_file
> > ENV SDK=/mnt/sdk
> > ADD --chown=user:user ${sdk_file} ${SDK}
> 
>  My docker-fu is not too great, but you can do something similar to expose the
> tarball directly, and then do
> 
> RUN tar -xf ${sdk_file} --strip-components=1
> 
> no?

It would be more like:

RUN dnf install -y tar
ARG sdk_file
ENV SDK=/mnt/sdk
COPY ${sdk_file} ${SDK}
RUN tar -xf ${SDK}/${sdk_file} -C ${SDK} --strip-components=1 --owner user:user
RUN rm ${SDK}/${sdk_file}

decompress option needs to be specified to tar, based on the compressor used.

This uses more layers.


> > Now it needs to be relocated.  Something like:
> > 
> > RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh
> 
>  Here you could to
> 
> RUN cd ${SDK}/buildroot-sdk.*/ && ./relocate-sdk.sh

Assumes only one directory will match the pattern.  Would also require
the user of the SDK to do the same thing, assume the only directory in
${SDK} is the one they want and use a shell wildcard expansion to find
it.  Which I admit could work, though I think it's a kludge.  I needed
to do the same thing to find the sysroot location and compiler prefix
inside the SDK, and I think that's a kludge too, and wish buildroot put
a pointer in a known location in the sdk to where things in unknown
locations (sysroot) are.  But I haven't gotten around to making a patch
to do that yet.

>  That said, if you're building an SDK, it's probably because you're not changing
> it too often. As long as you're developing the Buildroot OS itself, there's
> generally no point generating the SDK. If you really are closely co-developing
> your application and the Buildroot OS, you're probably better of including the
> application as a package rather than building it externally.

The latter, both the SDK and the applications using the SDK change very
 frequently.  Which is why I'm so concerned about speed.

While the applications are buildroot packages as well, there are
downsides to doing development that way.  So they exist as stand alone
packages that can be compiled outside of buildroot.  No different than
any other package in buildroot in that regard.

But how to build a package stand alone in a CI process, in an
environment than can build it correctly, and also closely tracks the
image buildroot will make for the final product?  Use the buildroot
SDK!  So that's what I'm using it for.

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Stefan Becker]

Hi Trent, All,

On Tue, Jun 12, 2018 at 8:47 PM Trent Piepho <tpiepho@impinj.com> wrote:
>

I didn't want to write anything to this thread at first, but then I
saw that what Trent wrote matches 100% what I was thinking when
looking at the change.

I'm very disappointed with the attitude displayed by the buildroot
developers in this thread. Trent has valid points and IMHO has made
sensible suggestions how to address those. But all I see from the
project developers is the typical knee-jerk reaction to SW build or SW
integration issues, i.e. "buildroot is always correct, your system is
doing it wrong".

Can we please all take a step backward and agree that there is a whole
universe outside buildroot with different (integration) requirements?
For the change under review I would suggest:

(A) there are existing users of the "sdk" target and this change would
modify its behaviour in an undesired/surprising fashion.

(B) instead this change should introduce e.g. the target
"sdk-tarball", which depends on "sdk" and generates the new tarball
format. It is then up to the users of "sdk" to decide if they can use
the new build artifact or not.

(C) having a build artefact that changes the name in every build is
bad for automatic build setups. So (B) must offer a way for automatic
build scripts to set the generated tarball name. If the user can use
the new tarball format in his integration, then he can switch from
"sdk" to "sdk-tarball" with the existing name (and possibly using
--strip-components in downstream build scripts).

Just my ?0.02...

       Stefan

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Thomas Petazzoni]

Hello,

On Tue, 12 Jun 2018 21:07:35 +0300, Stefan Becker wrote:

> I didn't want to write anything to this thread at first, but then I
> saw that what Trent wrote matches 100% what I was thinking when
> looking at the change.
> 
> I'm very disappointed with the attitude displayed by the buildroot
> developers in this thread.

I believe you are also very abrupt with this statement. For example, I
myself asked as a reply to Yann's proposal if there wasn't a backward
compatibility issue.

> Trent has valid points and IMHO has made
> sensible suggestions how to address those. But all I see from the
> project developers is the typical knee-jerk reaction to SW build or SW
> integration issues, i.e. "buildroot is always correct, your system is
> doing it wrong".
> 
> Can we please all take a step backward and agree that there is a whole
> universe outside buildroot with different (integration) requirements?

For the record, I am using Buildroot for toolchains.bootlin.com, which
is running Buildroot in a CI system to produce toolchains, and it uses
"make sdk". So I am also affected by behavior changes in "make sdk".

> For the change under review I would suggest:
> 
> (A) there are existing users of the "sdk" target and this change would
> modify its behaviour in an undesired/surprising fashion.
> 
> (B) instead this change should introduce e.g. the target
> "sdk-tarball", which depends on "sdk" and generates the new tarball
> format. It is then up to the users of "sdk" to decide if they can use
> the new build artifact or not.

As said above, when I first looked at Yann's proposal, I thought there
could be a backward compatibility problem. But in fact, what the new
"make sdk" does is a strict super-set of what the old "make sdk" was
doing. So if you're not happy with the new "make sdk" behavior, this
shouldn't really matter: it still does the exact same thing as before,
it just *additionally* provides a convenience tarball.

However, one could argue that despite the behavior being preserved, it
still changes a few things:

 - More time required due to generating the tarball.

 - More disk space being consumed, especially with the tarball name
   changing pretty frequently due to containing the Git commit id.

> (C) having a build artefact that changes the name in every build is
> bad for automatic build setups.

True.

> So (B) must offer a way for automatic
> build scripts to set the generated tarball name. If the user can use
> the new tarball format in his integration, then he can switch from
> "sdk" to "sdk-tarball" with the existing name (and possibly using
> --strip-components in downstream build scripts).

But where did we say that we would not accept something that allows to
customize the output tarball name ? Where did we say that using a
separate target was not acceptable ?

I think you're really rushing way too fast to a conclusion that we're
not listening. Yes, we are asking questions about the use cases, yes we
are trying to understand such use cases, and see how they fit in the
big picture.

I believe one of the reason we all like and use Buildroot is because of
its simplicity. And simplicity is a very difficult thing to preserve:
the tendency of all projects is to become more and more complicated, to
address more and more special use cases. So it is quite normal that we
question the use cases, try to understand them, and see which part of
those use cases should be solved by Buildroot itself, and which use
cases should be solved by external tooling. There is always going to be
a tension between people wanted to solve more use cases inside
Buildroot and therefore increasing its complexity, and the wish to keep
Buildroot simple.

Let's try to keep the discussion open-minded, please :-)

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Stefan Becker]

Hi Thomas, All,

On Tue, Jun 12, 2018 at 10:01 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> I believe one of the reason we all like and use Buildroot is because of
> its simplicity. And simplicity is a very difficult thing to preserve:

IMHO this change is the very opposite of "preserving simplicity". It
adds unwanted things to the buildroot build. The unwanted things might
even be unusable for existing users of "sdk".

Please don't get me wrong. I'm not against the SDK tarball itself.

But I'm against building something for naught. If the tarball turns
out to be unusable I get slower buildroot builds. Which in turns will
force me to extract what "original sdk without tarball" was doing out
of buildroot makefile and hard-code it into my build script. And then
I can remove "sdk" from the buildroot make command. I fail to see how
that is simpler.

If you don't offer a way to specify the generated tarball name, fine.
Then I won't use sdk-tarball at all, even if the tarball itself would
be usable in our downstream integration. Or I could come up with a way
to "guess" the build artefact name in my build script and symlink it
to the wanted build artefact name (again, I fail to see how this is
simpler).

I don't understand why "sdk-tarball: sdk" is not an acceptable
solution. It would be much simpler.

Regards, Stefan

PS: just to be complete, this is what my current build script does
after "make all sdk" has been completed:

    (cd output/host && tar -cvJf ../host.tar.xz *)

The collected build artefact is "host.tar.xz". AFAIR that name can't
be changed dynamically without breaking the automatic downstream build
flow in Jenkins.

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Andreas Naumann]

Hi,

on a general note, I do find that buildroot does an outstanding job in 
providing a lot of simple to use functionality while keeping it within a 
well defined scope!
To help achieve that for SDK generation, I'll describe my use 
cases/preferences.

Am 12.06.2018 um 21:25 schrieb Stefan Becker:
> Hi Thomas, All,
> 
<snip>
> 
> But I'm against building something for naught. If the tarball turns
> out to be unusable I get slower buildroot builds. Which in turns will
> force me to extract what "original sdk without tarball" was doing out
> of buildroot makefile and hard-code it into my build script. And then
> I can remove "sdk" from the buildroot make command. I fail to see how
> that is simpler.

I also prefer speed over storage size (gz vs. xz) and definitely want to 
spend both only for building what I need.

> 
> If you don't offer a way to specify the generated tarball name, fine.
> Then I won't use sdk-tarball at all, even if the tarball itself would
> be usable in our downstream integration. Or I could come up with a way
> to "guess" the build artefact name in my build script and symlink it
> to the wanted build artefact name (again, I fail to see how this is
> simpler).

Even though the sdk target has been around for a while, I still use 
custom steps to a) build the SDK on a absolute path to avoid relocation 
needs and b) tar it with a project-specific name.

So if b) is provided by buildroot I'd be very interested to finally give 
up my addon mess (modifiying HOST_DIR before the build) and switch to 
buildroots way.
However, for maintenance reasons I want to support existing project 
setups, which would only be "simple" if there was an easy way to
- run the relocation previous to packing
- adjust the location to be packed accordingly
- and define the name of the tarball with a set of "beyond buildroot" 
Project-Variables

I guess one way to achieve that would be baking this sort of 
customization into the defconfig. However, that makes usage of 
savedefconfig more or less useless to me because it leads to constantly 
changing defconfigs. (Which is undesirable as my messing with HOST_DIR 
has showed me).
Allowing environment variables to control the process would work too, 
but I dont know if thats considered "good practise" or "simple" either.


best regards,
Andreas

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Arnout Vandecappelle]

 Hi Andreas,

On 13-06-18 10:32, Andreas Naumann wrote:
> Hi,
> 
> on a general note, I do find that buildroot does an outstanding job in providing
> a lot of simple to use functionality while keeping it within a well defined scope!
> To help achieve that for SDK generation, I'll describe my use cases/preferences.

 Thanks for your input!


> Am 12.06.2018 um 21:25 schrieb Stefan Becker:
>> Hi Thomas, All,
>>
> <snip>
>>
>> But I'm against building something for naught. If the tarball turns
>> out to be unusable I get slower buildroot builds. Which in turns will
>> force me to extract what "original sdk without tarball" was doing out
>> of buildroot makefile and hard-code it into my build script. And then
>> I can remove "sdk" from the buildroot make command. I fail to see how
>> that is simpler.
> 
> I also prefer speed over storage size (gz vs. xz) and definitely want to spend
> both only for building what I need.

 Yep, it's pretty clear that my .xz idea doesn't get much traction :-)


>> If you don't offer a way to specify the generated tarball name, fine.
>> Then I won't use sdk-tarball at all, even if the tarball itself would
>> be usable in our downstream integration. Or I could come up with a way
>> to "guess" the build artefact name in my build script and symlink it
>> to the wanted build artefact name (again, I fail to see how this is
>> simpler).
> 
> Even though the sdk target has been around for a while, I still use custom steps
> to a) build the SDK on a absolute path to avoid relocation needs and b) tar it
> with a project-specific name.
> 
> So if b) is provided by buildroot I'd be very interested to finally give up my
> addon mess (modifiying HOST_DIR before the build and switch to buildroots way.
 Why is specifying a BR2_HOST_DIR  on the make command line messy? It is in fact
Buildroot's way :-)

> However, for maintenance reasons I want to support existing project setups,
> which would only be "simple" if there was an easy way to
> - run the relocation previous to packing
> - adjust the location to be packed accordingly
> - and define the name of the tarball with a set of "beyond buildroot"
> Project-Variables

 This starts to sound complicated... I would say, if you're content with a
non-relocatable SDK, setting BR2_HOST_DIR is the way to go.

 Perhaps we could consider a post-sdk script. That would get called with the
relevant variables so you could set up all the symlinks you want to find the
cross-compile tuple etc. Although, really, calling 'make printvars
VARS=GNU_TARGET_NAME' from your CI script doesn't sound like the end of the
world to me.


> I guess one way to achieve that would be baking this sort of customization into
> the defconfig. However, that makes usage of savedefconfig more or less useless
> to me because it leads to constantly changing defconfigs. (Which is undesirable
> as my messing with HOST_DIR has showed me).
> Allowing environment variables to control the process would work too, but I dont
> know if thats considered "good practise" or "simple" either.

 I don't really like putting stuff in environment variables. I'd rather have it
in local.mk variables then (which automatically means that they can also be set
on the command line). But even so, it smells like unneeded complexity to me.

 Summarizing where we are in this thread now, I think:

1. The tarball generation should be separated from the sdk generation itself,
with e.g.

sdk-tar: sdk
	@$(TAR) ...

2. The tarball should really be created as a .gz, not as an .xz or anything else.

3. In addition to the unique name, there should be a symlink/hardlink with a
constant name.

 Note that this doesn't solve the potential issue that the images/ directory is
constantly growing. So perhaps we should just give up on the unique name, and
instead only generate a constant name, like we currently do for all our build
artefacts.


 With all this in mind, I've marked the patch as Changes Requested.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Andreas Naumann]

Hi Arnout,

Am 13.06.2018 um 12:12 schrieb Arnout Vandecappelle:
>   Hi Andreas,
> 

> 
>>> If you don't offer a way to specify the generated tarball name, fine.
>>> Then I won't use sdk-tarball at all, even if the tarball itself would
>>> be usable in our downstream integration. Or I could come up with a way
>>> to "guess" the build artefact name in my build script and symlink it
>>> to the wanted build artefact name (again, I fail to see how this is
>>> simpler).
>>
>> Even though the sdk target has been around for a while, I still use custom steps
>> to a) build the SDK on a absolute path to avoid relocation needs and b) tar it
>> with a project-specific name.
>>
>> So if b) is provided by buildroot I'd be very interested to finally give up my
>> addon mess (modifiying HOST_DIR before the build and switch to buildroots way.
>   Why is specifying a BR2_HOST_DIR  on the make command line messy? It is in fact
> Buildroot's way :-)

Specifying on command line would have been nice but remember this being 
ignored in favour of the .config setting. So I always modified .config 
before the build, which I do regard as somewhat messy.

> 
>> However, for maintenance reasons I want to support existing project setups,
>> which would only be "simple" if there was an easy way to
>> - run the relocation previous to packing
>> - adjust the location to be packed accordingly
>> - and define the name of the tarball with a set of "beyond buildroot"
>> Project-Variables
> 
>   This starts to sound complicated... I would say, if you're content with a
> non-relocatable SDK, setting BR2_HOST_DIR is the way to go.

I just tried again to set BR_HOST_DIR via environment but it doesnt 
propagate. It would work with make -e ... but I fear hard to discover 
side effects doing that. So I'd rather go with the relocation in the future.

> 
>   Perhaps we could consider a post-sdk script. That would get called with the
> relevant variables so you could set up all the symlinks you want to find the
> cross-compile tuple etc. Although, really, calling 'make printvars
> VARS=GNU_TARGET_NAME' from your CI script doesn't sound like the end of the
> world to me.

Actually, I'm using the printvars solution right now, no problem.

> 
> 
>> I guess one way to achieve that would be baking this sort of customization into
>> the defconfig. However, that makes usage of savedefconfig more or less useless
>> to me because it leads to constantly changing defconfigs. (Which is undesirable
>> as my messing with HOST_DIR has showed me).
>> Allowing environment variables to control the process would work too, but I dont
>> know if thats considered "good practise" or "simple" either.
> 
>   I don't really like putting stuff in environment variables. I'd rather have it
> in local.mk variables then (which automatically means that they can also be set
> on the command line). But even so, it smells like unneeded complexity to me.

Just as I fail in overwriting BR2_HOST_DIR from commandline I fail in 
setting BR2_PACKAGE_OVERRIDE_FILE to something else than defined in 
.config. I guess it would be really helpful if you could give me an 
example on how to do that.

> 
>   Summarizing where we are in this thread now, I think:
> 
> 1. The tarball generation should be separated from the sdk generation itself,
> with e.g.
> 
> sdk-tar: sdk
> 	@$(TAR) ...
> 
> 2. The tarball should really be created as a .gz, not as an .xz or anything else.
> 
> 3. In addition to the unique name, there should be a symlink/hardlink with a
> constant name.
> 
>   Note that this doesn't solve the potential issue that the images/ directory is
> constantly growing. So perhaps we should just give up on the unique name, and
> instead only generate a constant name, like we currently do for all our build
> artefacts.

Since the SDK tarball is something that is "thrown around" between 
developers on it's own, it might make sense to somehow being able to 
distinguish it?
For my case, I include a project version in the name, but no buildnumber 
or SHA. That way it just changes between project releases. Also, I have 
a separate sdk/ folder for it, because I dont like mixing target with 
host stuff. (and it can be cleaned independently).

So while it's easy enough to rename the tarball to an appropriate name 
after creating it with make sdk-tar, the contained top-level folder 
should somehow match this name right?


regards,
Andreas

> 
> 
>   With all this in mind, I've marked the patch as Changes Requested.
> 
>   Regards,
>   Arnout
> 

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Arnout Vandecappelle]

On 12-06-18 21:01, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 12 Jun 2018 21:07:35 +0300, Stefan Becker wrote:
> 
>> I didn't want to write anything to this thread at first, but then I
>> saw that what Trent wrote matches 100% what I was thinking when
>> looking at the change.
>>
>> I'm very disappointed with the attitude displayed by the buildroot
>> developers in this thread.
> 
> I believe you are also very abrupt with this statement. For example, I
> myself asked as a reply to Yann's proposal if there wasn't a backward
> compatibility issue.

 Thomas, you can't deny that, whatever your (and my) intention was in this
thread, it clearly has the effect on (some) people that they think "the
buildroot developers" are not paying attention to their needs.

 I don't know why that is or what we can do about it (without investing still
more time in composing our mails), but the least we can do is acknowledge this
situation.


 Regards,
 Arnout

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Thomas Petazzoni]

Arnout,

On Wed, 13 Jun 2018 11:47:02 +0200, Arnout Vandecappelle wrote:

> >> I didn't want to write anything to this thread at first, but then I
> >> saw that what Trent wrote matches 100% what I was thinking when
> >> looking at the change.
> >>
> >> I'm very disappointed with the attitude displayed by the buildroot
> >> developers in this thread.  
> > 
> > I believe you are also very abrupt with this statement. For example, I
> > myself asked as a reply to Yann's proposal if there wasn't a backward
> > compatibility issue.  
> 
>  Thomas, you can't deny that, whatever your (and my) intention was in this
> thread, it clearly has the effect on (some) people that they think "the
> buildroot developers" are not paying attention to their needs.
> 
>  I don't know why that is or what we can do about it (without investing still
> more time in composing our mails), but the least we can do is acknowledge this
> situation.

I will have to read this specific thread again to see what could
communicate this feeling.

However, generally speaking, I do agree that we tend to push back on
people proposing to add more complexity to Buildroot, for the reasons
I've highlighted before. And I definitely understand that such regular
push back, or sometimes slowness to take a decision on specific topics
can be frustrating. I do completely acknowledge that.

However, this push back and/or slowness on topics that suggest changing
some core aspect of Buildroot or its complexity level is actually more
a feature than a bug. It is what allows to preserve Buildroot core
principles on the long run. The contributors pushing for a little bit
more complexity here only see a little bit more complexity in one
specific place. People who have been around for a much longer period of
time (I'm "celebrating" this year my 10th year contributing to
Buildroot, and Peter has been contributing for longer than that, etc.)
have a different perspective, as they see the overall complexity
increase, not just the specific complexity increase caused by one patch.

But, yes I acknowledge that we are reluctant to change, and sometimes
need a lot of convincing arguments to change our minds.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Stefan Becker]

Hi Thomas,

On Wed, Jun 13, 2018 at 12:57 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> However, generally speaking, I do agree that we tend to push back on
> people proposing to add more complexity to Buildroot, for the reasons
> I've highlighted before.

Wouldn't it then be much simpler to reject this change, i.e. *not*
adding tarball generation to buildroot SW build at all?

Regards, Stefan

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Thomas Petazzoni]

Hello,

On Wed, 13 Jun 2018 13:03:45 +0300, Stefan Becker wrote:

> On Wed, Jun 13, 2018 at 12:57 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > However, generally speaking, I do agree that we tend to push back on
> > people proposing to add more complexity to Buildroot, for the reasons
> > I've highlighted before.  
> 
> Wouldn't it then be much simpler to reject this change, i.e. *not*
> adding tarball generation to buildroot SW build at all?

That's one option. Another option is to:

 (1) Use another make target, such as sdk-tarball, as you suggested

 (2) Use a stable name for the output, like
     output/images/buildroot-sdk.tar.gz. Indeed for no other Buildroot
     output we include the Git commit ID or the target tuple in the
     name of the file being generated.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 > I think you're really rushing way too fast to a conclusion that we're
 > not listening. Yes, we are asking questions about the use cases, yes we
 > are trying to understand such use cases, and see how they fit in the
 > big picture.

 > I believe one of the reason we all like and use Buildroot is because of
 > its simplicity. And simplicity is a very difficult thing to preserve:
 > the tendency of all projects is to become more and more complicated, to
 > address more and more special use cases. So it is quite normal that we
 > question the use cases, try to understand them, and see which part of
 > those use cases should be solved by Buildroot itself, and which use
 > cases should be solved by external tooling. There is always going to be
 > a tension between people wanted to solve more use cases inside
 > Buildroot and therefore increasing its complexity, and the wish to keep
 > Buildroot simple.

 > Let's try to keep the discussion open-minded, please :-)

Completely agreed. Thank you for describing it so clearly!

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Arnout Vandecappelle]

On 12-06-18 19:47, Trent Piepho wrote:
> On Tue, 2018-06-12 at 15:30 +0200, Arnout Vandecappelle wrote:
>>
>> On 11-06-18 19:20, Trent Piepho wrote:
>>> On Sun, 2018-06-10 at 23:21 +0200, Arnout Vandecappelle wrote:
>>>>  
>>>>>  .PHONY: sdk
>>>>> -sdk: world
>>>>> +sdk: world $(BR2_TAR_HOST_DEPENDENCY)
>>>>>  	@$(call MESSAGE,"Rendering the SDK relocatable")
>>>>>  	$(TOPDIR)/support/scripts/fix-rpath host
>>>>>  	$(TOPDIR)/support/scripts/fix-rpath staging
>>>>>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>>>>>  	mkdir -p $(HOST_DIR)/share/buildroot
>>>>>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
>>>>> +	$(Q)mkdir -p $(BINARIES_DIR)
>>>>> +	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
>>>>
>>>>  Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
>>>> probably gives a significant hit to the build time. But since it's explicit in
>>>> 'make sdk' I don't mind that much.
>>>>
>>>
>>> I'm using the SDK with Docker.  It can add tar.gz files into the
>>> container, but now tar.xz.  Having to recompress the archive would be a
>>> significant increase in build time.
>>
>>  Duh. Calling tar inside the container explicitly, rather than letting docker to
>> the extract directly, isn't too much of a problem either, no?
> 
> I believe COPY + RUN ends up being more layers than just ADD. 

 You can collapse the RUN commands, see below.

> Though
> current docker supports xz and bz2 so it would work to use a better
> compression.
> 
> But, in the tests I've done on our vsphere system, using gz was still
> fastest.  I'm more interested in making the CI process faster than in
> saving disk space.

 Good point, gz is much faster than xz, both for compressing and decompressing.
So it indeed makes sense to just keep it as a .gz. IIRC lz4 would be even
better, but that requires installing lz4...

>>>>> +		-C $(HOST_DIR) \
>>>>> +		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
>>>>
>>>>  Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
>>>> variable?
>>>
>>> So having extracted an SDK, in an automated CI script, how does one use
>>> it?  I'll add it to the Docker container:
>>>
>>> ARG sdk_file
>>> ENV SDK=/mnt/sdk
>>> ADD --chown=user:user ${sdk_file} ${SDK}
>>
>>  My docker-fu is not too great, but you can do something similar to expose the
>> tarball directly, and then do
>>
>> RUN tar -xf ${sdk_file} --strip-components=1
>>
>> no?
> 
> It would be more like:
> 
> RUN dnf install -y tar

 You could just have tar already in your base image, of course...

> ARG sdk_file
> ENV SDK=/mnt/sdk
> COPY ${sdk_file} ${SDK}

 Isn't there something that "links" the file rather than copying it into the image?

> RUN tar -xf ${SDK}/${sdk_file} -C ${SDK} --strip-components=1 --owner user:user
> RUN rm ${SDK}/${sdk_file}

 I think this will create 3 layers: one for the COPY and one for each RUN. You
can reduce it to two with:

RUN tar -xf ${SDK}/${sdk_file} -C ${SDK} --strip-components=1 &&
    rm ${SDK}/${sdk_file}

> 
> decompress option needs to be specified to tar, based on the compressor used.
> 
> This uses more layers.
> 
> 
>>> Now it needs to be relocated.  Something like:
>>>
>>> RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh
>>
>>  Here you could to
>>
>> RUN cd ${SDK}/buildroot-sdk.*/ && ./relocate-sdk.sh
> 
> Assumes only one directory will match the pattern.

 If you re-create the docker image every time (like is done above), only one
will match.

>  Would also require
> the user of the SDK to do the same thing, assume the only directory in
> ${SDK} is the one they want and use a shell wildcard expansion to find
> it.

 Indeed, not so nice. So --strip-components is better :-)

>  Which I admit could work, though I think it's a kludge.  I needed
> to do the same thing to find the sysroot location and compiler prefix
> inside the SDK, and I think that's a kludge too, and wish buildroot put
> a pointer in a known location in the sdk to where things in unknown
> locations (sysroot) are.

 Good point. In the output/ directory, we have ${STAGING_DIR} which does
something like that (points to the sysroot, still doesn't explicitly say what
the CROSS_COMPILE prefix is but you could parse it out of the symlink). So we
could add the staging symlink to the tarball as well, and while we're at it also
add a host symlink. Although actually, no, I think that just looks ugly. There
must be a better way.

>  But I haven't gotten around to making a patch
> to do that yet.
> 
>>  That said, if you're building an SDK, it's probably because you're not changing
>> it too often. As long as you're developing the Buildroot OS itself, there's
>> generally no point generating the SDK. If you really are closely co-developing
>> your application and the Buildroot OS, you're probably better of including the
>> application as a package rather than building it externally.
> 
> The latter, both the SDK and the applications using the SDK change very
>  frequently.  Which is why I'm so concerned about speed.
> 
> While the applications are buildroot packages as well, there are
> downsides to doing development that way.  So they exist as stand alone
> packages that can be compiled outside of buildroot.  No different than
> any other package in buildroot in that regard.
> 
> But how to build a package stand alone in a CI process, in an
> environment than can build it correctly, and also closely tracks the
> image buildroot will make for the final product?  Use the buildroot
> SDK!  So that's what I'm using it for.

 Very good point indeed. I'm glad I asked :-)

 Is the idea that you have one buildroot build, it creates a docker image, and
that docker image is then used for several application builds? Sounds like a
pretty neat setup.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Stefan Becker]

Hi Arnout, All,

On Wed, Jun 13, 2018 at 10:46 AM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>  Is the idea that you have one buildroot build, it creates a docker image, and
> that docker image is then used for several application builds? Sounds like a
> pretty neat setup.

Docker isn't really a requirement for such an approach, but it can
make life much simpler in a fractured IT environment.

We use the sdk build artefact from a common buildroot automated build
in our CI system without using docker. Having fast CI builds (< 5mins)
for our own components/device images and that no developer ever has to
run slow buildroot builds are a big plus :-) As there is only one
buildroot build makes tracking buildroot upstream much easier.

Regards, Stefan

[Buildroot] [PATCH] core/sdk: generate the SDK tarball ourselves

[Peter Korsgaard]

>>>>> "Trent" == Trent Piepho <tpiepho@impinj.com> writes:

Hi,

 > So having extracted an SDK, in an automated CI script, how does one use
 > it?  I'll add it to the Docker container:

 > ARG sdk_file
 > ENV SDK=/mnt/sdk
 > ADD --chown=user:user ${sdk_file} ${SDK}

 > Now it needs to be relocated.  Something like:

 > RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh

 > Is the problem with using this tarball apparent?  The dockerfile needs
 > to be hand edited on every single build to use the correct path to the
 > SDK.

Why not use a wildcard? E.G. cd ${SDK}/buildroot-sdk* && ./relocate-sdk.sh?

Nothing else should match the wildcard inside the container.

-- 
Bye, Peter Korsgaard