[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Stefan Becker]

If the variable content has newlines in it then the currently dumped
content can't be fed again to GNU make. Add the option DEFINE_VARS which
causes the variables to be dumped using

   define VAR
   ... line 1 ...
   ... line 2 ...
   ...
   endef

Updated the manual accordingly.

Signed-off-by: Stefan Becker <chemobejk@gmail.com>
---
 Makefile                  | 10 +++++++---
 docs/manual/make-tips.txt | 20 ++++++++++++++++++++
 2 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 0724f28f45..3e27195de0 100644
--- a/Makefile
+++ b/Makefile
@@ -974,9 +974,13 @@ printvars:
 		$(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \
 		$(if $(filter-out environment% default automatic, \
 				$(origin $V)), \
-		$(if $(QUOTED_VARS),\
-			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
-			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
+		$(if $(DEFINE_VARS), \
+			$(info define $V) \
+			$(info $(if $(RAW_VARS),$(value $V),$($V))) \
+			$(info endef), \
+			$(if $(QUOTED_VARS),\
+				$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
+				$(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))))
 # ' Syntax colouring...
 
 .PHONY: clean
diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt
index ea1d825bef..ba87e5d873 100644
--- a/docs/manual/make-tips.txt
+++ b/docs/manual/make-tips.txt
@@ -92,6 +92,8 @@ It is possible to tweak the output using some variables:
 
 - +VARS+ will limit the listing to variables which names match the
   specified make-pattern
+- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve
+  newlines in the value
 - +QUOTED_VARS+, if set to +YES+, will single-quote the value
 - +RAW_VARS+, if set to +YES+, will print the unexpanded value
 
@@ -106,6 +108,24 @@ For example:
  BUSYBOX_RDEPENDENCIES=ncurses util-linux
 ----
 
+----
+ $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES
+ define BUSYBOX_DEPENDENCIES
+ skeleton toolchain
+ endef
+ define BUSYBOX_FINAL_ALL_DEPENDENCIES
+ skeleton toolchain
+ endef
+ define BUSYBOX_FINAL_DEPENDENCIES
+ skeleton toolchain
+ endef
+ define BUSYBOX_FINAL_PATCH_DEPENDENCIES
+ endef
+ define BUSYBOX_RDEPENDENCIES
+ ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache
+ endef
+----
+
 ----
  $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES
  BUSYBOX_DEPENDENCIES='skeleton toolchain'
-- 
2.14.3

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Yann E. MORIN]

Stefan, All,

On 2018-04-16 14:58 +0300, Stefan Becker spake thusly:
> If the variable content has newlines in it then the currently dumped
> content can't be fed again to GNU make. Add the option DEFINE_VARS which
> causes the variables to be dumped using
> 
>    define VAR
>    ... line 1 ...
>    ... line 2 ...
>    ...
>    endef
> 
> Updated the manual accordingly.
> 
> Signed-off-by: Stefan Becker <chemobejk@gmail.com>

What did change from the previous iteration, which is visible, and still
pending, there:
    https://patchwork.ozlabs.org/project/buildroot/list/

As you can see, there are a *lot* of pending patches, so resending yours
will not help much, unless there was a change, in which case you should
say so as a post-commit log (i.e. below a '---' line).

I'm marking the old iteration as supserseded, now...

Regards,
Yann E. MORIN.

> ---
>  Makefile                  | 10 +++++++---
>  docs/manual/make-tips.txt | 20 ++++++++++++++++++++
>  2 files changed, 27 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 0724f28f45..3e27195de0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -974,9 +974,13 @@ printvars:
>  		$(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \
>  		$(if $(filter-out environment% default automatic, \
>  				$(origin $V)), \
> -		$(if $(QUOTED_VARS),\
> -			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> -			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
> +		$(if $(DEFINE_VARS), \
> +			$(info define $V) \
> +			$(info $(if $(RAW_VARS),$(value $V),$($V))) \
> +			$(info endef), \
> +			$(if $(QUOTED_VARS),\
> +				$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> +				$(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))))
>  # ' Syntax colouring...
>  
>  .PHONY: clean
> diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt
> index ea1d825bef..ba87e5d873 100644
> --- a/docs/manual/make-tips.txt
> +++ b/docs/manual/make-tips.txt
> @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables:
>  
>  - +VARS+ will limit the listing to variables which names match the
>    specified make-pattern
> +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve
> +  newlines in the value
>  - +QUOTED_VARS+, if set to +YES+, will single-quote the value
>  - +RAW_VARS+, if set to +YES+, will print the unexpanded value
>  
> @@ -106,6 +108,24 @@ For example:
>   BUSYBOX_RDEPENDENCIES=ncurses util-linux
>  ----
>  
> +----
> + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES
> + define BUSYBOX_DEPENDENCIES
> + skeleton toolchain
> + endef
> + define BUSYBOX_FINAL_ALL_DEPENDENCIES
> + skeleton toolchain
> + endef
> + define BUSYBOX_FINAL_DEPENDENCIES
> + skeleton toolchain
> + endef
> + define BUSYBOX_FINAL_PATCH_DEPENDENCIES
> + endef
> + define BUSYBOX_RDEPENDENCIES
> + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache
> + endef
> +----
> +
>  ----
>   $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES
>   BUSYBOX_DEPENDENCIES='skeleton toolchain'
> -- 
> 2.14.3
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Arnout Vandecappelle]

On 16-04-18 13:58, Stefan Becker wrote:
> If the variable content has newlines in it then the currently dumped
> content can't be fed again to GNU make. Add the option DEFINE_VARS which
> causes the variables to be dumped using
> 
>    define VAR
>    ... line 1 ...
>    ... line 2 ...
>    ...
>    endef

 I'm still not convinced that this is such a great idea.

1. As explained in reply to your first iteration, there shouldn't be a need for
including Buildroot variables in a surrounding Makefile.

2. At least equally relevant would be output that is appropriate for Python,
JSON, ...

3. The implementation is not complete. In case of RAW_VARS, it's pretty hard to
use in practice because also all the referenced variables would need to be
retrieved. In the expanded case, any remaining $ will be interpreted by your
surrounding make instead of being passed down to the shell like they should. Try
e.g. CANFESTIVAL_INSTALL_TARGET_CMDS.

Note that QUOTED_VARS is not perfect either, mainly because there are some make
variables that are not legal shell variables (e.g. all the 4th stuff).

 So, if anything changes in here, I'd rather have some output that is in some
intermediate format that can easily be converted into or parsed by shell,
python, etc.

 Regards,
 Arnout

> 
> Updated the manual accordingly.
> 
> Signed-off-by: Stefan Becker <chemobejk@gmail.com>
> ---
>  Makefile                  | 10 +++++++---
>  docs/manual/make-tips.txt | 20 ++++++++++++++++++++
>  2 files changed, 27 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 0724f28f45..3e27195de0 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -974,9 +974,13 @@ printvars:
>  		$(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \
>  		$(if $(filter-out environment% default automatic, \
>  				$(origin $V)), \
> -		$(if $(QUOTED_VARS),\
> -			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> -			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
> +		$(if $(DEFINE_VARS), \
> +			$(info define $V) \
> +			$(info $(if $(RAW_VARS),$(value $V),$($V))) \
> +			$(info endef), \
> +			$(if $(QUOTED_VARS),\
> +				$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> +				$(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))))
>  # ' Syntax colouring...
>  
>  .PHONY: clean
> diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt
> index ea1d825bef..ba87e5d873 100644
> --- a/docs/manual/make-tips.txt
> +++ b/docs/manual/make-tips.txt
> @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables:
>  
>  - +VARS+ will limit the listing to variables which names match the
>    specified make-pattern
> +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve
> +  newlines in the value
>  - +QUOTED_VARS+, if set to +YES+, will single-quote the value
>  - +RAW_VARS+, if set to +YES+, will print the unexpanded value
>  
> @@ -106,6 +108,24 @@ For example:
>   BUSYBOX_RDEPENDENCIES=ncurses util-linux
>  ----
>  
> +----
> + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES
> + define BUSYBOX_DEPENDENCIES
> + skeleton toolchain
> + endef
> + define BUSYBOX_FINAL_ALL_DEPENDENCIES
> + skeleton toolchain
> + endef
> + define BUSYBOX_FINAL_DEPENDENCIES
> + skeleton toolchain
> + endef
> + define BUSYBOX_FINAL_PATCH_DEPENDENCIES
> + endef
> + define BUSYBOX_RDEPENDENCIES
> + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache
> + endef
> +----
> +
>  ----
>   $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES
>   BUSYBOX_DEPENDENCIES='skeleton toolchain'
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Stefan Becker]

Hi,

On Thu, Apr 19, 2018 at 1:27 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>  I'm still not convinced that this is such a great idea.
>
> 1. As explained in reply to your first iteration, there shouldn't be a need for
> including Buildroot variables in a surrounding Makefile.

There is nothing you can say that will change the fact that a
monolithic meta build system is too slow for the CI of our own
components. So let's just agree to disagree on this point.

> 2. At least equally relevant would be output that is appropriate for Python,
> JSON, ...

I see your point. How about this instead?

$ external/scripts/buildstep.sh printvars MULTILINE_VARS=1
VARS="CANFESTIVAL_INSTALL_TARGET_CMDS"
MULTILINE_VAR_START CANFESTIVAL_INSTALL_TARGET_CMDS
        for d in src drivers ; do
PATH="/workarea/stefanb/repos/buildroot-new/output/host/bin:/workarea/stefanb/repos/buildroot-new/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/stefanb/.local/bin:/home/stefanb/bin"
/usr/bin/make -j1 -C ./$d install
PYTHON=/workarea/stefanb/repos/buildroot-new/output/host/bin/python2
DESTDIR=/workarea/stefanb/repos/buildroot-new/output/target || exit 1
; done
MULTILINE_VAR_END CANFESTIVAL_INSTALL_TARGET_CMDS

i.e. the consumer of the MULTILINE_VARS=1 output is then responsible
to post process it into the format he requires, e.g. for my intended
purpose I would use

... | sed  \
    -e 's/^MULTILINE_VAR_START \(.\+\)$/define \1/' \
    -e 's/^MULTILINE_VAR_END .\+$/endef/'

> 3. The implementation is not complete. In case of RAW_VARS, it's pretty hard to
> use in practice because also all the referenced variables would need to be
> retrieved. In the expanded case, any remaining $ will be interpreted by your
> surrounding make instead of being passed down to the shell like they should. Try
> e.g. CANFESTIVAL_INSTALL_TARGET_CMDS.
>
> Note that QUOTED_VARS is not perfect either, mainly because there are some make
> variables that are not legal shell variables (e.g. all the 4th stuff).

IMHO RAW_VARS, as-is today, is only meant to produce human readable
format for debug purposes. I.e. RAW_VARS=1 with any of the formats
none/QUOTE_VARS/MULTILINE_VARS will not be usable for post-processing,
unless the post-processing knows how to parse for variables and
request them.

Fixing RAW_VARS in such a way that any variable detected would be
added to VARS list and then recurse to get all referenced variables,
is outside the scope of my change.

Or alternatively: drop RAW_VARS support from
QUOTED_VARS/MULTILINE_VARS branches.

Regards, Stefan

[Buildroot] [PATCH v2] core: enhance printvars for variables with newlines

[Stefan Becker]

The current output formats of printvars do not preserve newlines. Add
the option MULTILINE_VARS which causes the variable content to be dumped
in a format that preserves newlines:

   MULTILINE_VAR_START VAR
   ... line 1 ...
   ... line 2 ...
   ...
   MULTILINE_VAR_END VAR

Updated the manual accordingly.

Signed-off-by: Stefan Becker <chemobejk@gmail.com>
---
 Makefile                  | 10 +++++++---
 docs/manual/make-tips.txt | 20 ++++++++++++++++++++
 2 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 54f940659c..bd78da8362 100644
--- a/Makefile
+++ b/Makefile
@@ -979,9 +979,13 @@ printvars:
 		$(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \
 		$(if $(filter-out environment% default automatic, \
 				$(origin $V)), \
-		$(if $(QUOTED_VARS),\
-			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
-			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
+		$(if $(MULTILINE_VARS), \
+			$(info MULTILINE_VAR_START $V) \
+			$(info $(if $(RAW_VARS),$(value $V),$($V))) \
+			$(info MULTILINE_VAR_END $V), \
+			$(if $(QUOTED_VARS),\
+				$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
+				$(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))))
 # ' Syntax colouring...
 
 .PHONY: clean
diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt
index ea1d825bef..c796bd73e2 100644
--- a/docs/manual/make-tips.txt
+++ b/docs/manual/make-tips.txt
@@ -92,6 +92,8 @@ It is possible to tweak the output using some variables:
 
 - +VARS+ will limit the listing to variables which names match the
   specified make-pattern
+- +MULTILINE_VARS+, if set to +YES+, will use a start and end marker around
+  the variable content to preserve newlines in the value
 - +QUOTED_VARS+, if set to +YES+, will single-quote the value
 - +RAW_VARS+, if set to +YES+, will print the unexpanded value
 
@@ -106,6 +108,24 @@ For example:
  BUSYBOX_RDEPENDENCIES=ncurses util-linux
 ----
 
+----
+ $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES MULTILINE_VARS=YES
+ MULTILINE_VAR_START BUSYBOX_DEPENDENCIES
+ skeleton toolchain
+ MULTILINE_VAR_END BUSYBOX_DEPENDENCIES
+ MULTILINE_VAR_START BUSYBOX_FINAL_ALL_DEPENDENCIES
+ skeleton toolchain
+ MULTILINE_VAR_END BUSYBOX_FINAL_ALL_DEPENDENCIES
+ MULTILINE_VAR_START BUSYBOX_FINAL_DEPENDENCIES
+ skeleton toolchain
+ MULTILINE_VAR_END BUSYBOX_FINAL_DEPENDENCIES
+ MULTILINE_VAR_START BUSYBOX_FINAL_PATCH_DEPENDENCIES
+ MULTILINE_VAR_END BUSYBOX_FINAL_PATCH_DEPENDENCIES
+ MULTILINE_VAR_START BUSYBOX_RDEPENDENCIES
+ ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache
+ MULTILINE_VAR_END BUSYBOX_RDEPENDENCIES
+----
+
 ----
  $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES
  BUSYBOX_DEPENDENCIES='skeleton toolchain'
-- 
2.14.3

[Buildroot] [PATCH v2] core: enhance printvars for variables with newlines

[Yann E. MORIN]

Stefan, All,

On 2018-04-19 10:40 +0300, Stefan Becker spake thusly:
> The current output formats of printvars do not preserve newlines.

This is wrong, as I explained in my reply to your previous iteration:

    http://lists.busybox.net/pipermail/buildroot/2018-April/219369.html

I am still not convinced by this new iteration... Basically, it changes
the start and stop makers from 'define' and 'endef' to MULTILINE_VAR_START
and MULTILINE_VAR_END; this is no real improvement. Note: I've seen your
other reply that sugest changing the marker so something else, and I
don;t think this is a better improvement either...

But I'm leaving this new iteration open for more reviews by others.

Regards,
Yann E. MORIN.

> Add
> the option MULTILINE_VARS which causes the variable content to be dumped
> in a format that preserves newlines:
> 
>    MULTILINE_VAR_START VAR
>    ... line 1 ...
>    ... line 2 ...
>    ...
>    MULTILINE_VAR_END VAR
> 
> Updated the manual accordingly.
> 
> Signed-off-by: Stefan Becker <chemobejk@gmail.com>
> ---
>  Makefile                  | 10 +++++++---
>  docs/manual/make-tips.txt | 20 ++++++++++++++++++++
>  2 files changed, 27 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 54f940659c..bd78da8362 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -979,9 +979,13 @@ printvars:
>  		$(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \
>  		$(if $(filter-out environment% default automatic, \
>  				$(origin $V)), \
> -		$(if $(QUOTED_VARS),\
> -			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> -			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
> +		$(if $(MULTILINE_VARS), \
> +			$(info MULTILINE_VAR_START $V) \
> +			$(info $(if $(RAW_VARS),$(value $V),$($V))) \
> +			$(info MULTILINE_VAR_END $V), \
> +			$(if $(QUOTED_VARS),\
> +				$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> +				$(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))))
>  # ' Syntax colouring...
>  
>  .PHONY: clean
> diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt
> index ea1d825bef..c796bd73e2 100644
> --- a/docs/manual/make-tips.txt
> +++ b/docs/manual/make-tips.txt
> @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables:
>  
>  - +VARS+ will limit the listing to variables which names match the
>    specified make-pattern
> +- +MULTILINE_VARS+, if set to +YES+, will use a start and end marker around
> +  the variable content to preserve newlines in the value
>  - +QUOTED_VARS+, if set to +YES+, will single-quote the value
>  - +RAW_VARS+, if set to +YES+, will print the unexpanded value
>  
> @@ -106,6 +108,24 @@ For example:
>   BUSYBOX_RDEPENDENCIES=ncurses util-linux
>  ----
>  
> +----
> + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES MULTILINE_VARS=YES
> + MULTILINE_VAR_START BUSYBOX_DEPENDENCIES
> + skeleton toolchain
> + MULTILINE_VAR_END BUSYBOX_DEPENDENCIES
> + MULTILINE_VAR_START BUSYBOX_FINAL_ALL_DEPENDENCIES
> + skeleton toolchain
> + MULTILINE_VAR_END BUSYBOX_FINAL_ALL_DEPENDENCIES
> + MULTILINE_VAR_START BUSYBOX_FINAL_DEPENDENCIES
> + skeleton toolchain
> + MULTILINE_VAR_END BUSYBOX_FINAL_DEPENDENCIES
> + MULTILINE_VAR_START BUSYBOX_FINAL_PATCH_DEPENDENCIES
> + MULTILINE_VAR_END BUSYBOX_FINAL_PATCH_DEPENDENCIES
> + MULTILINE_VAR_START BUSYBOX_RDEPENDENCIES
> + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache
> + MULTILINE_VAR_END BUSYBOX_RDEPENDENCIES
> +----
> +
>  ----
>   $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES
>   BUSYBOX_DEPENDENCIES='skeleton toolchain'
> -- 
> 2.14.3
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Thomas Petazzoni]

Hello Stefan,

On Thu, 19 Apr 2018 10:14:40 +0300, Stefan Becker wrote:

> On Thu, Apr 19, 2018 at 1:27 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> >
> >  I'm still not convinced that this is such a great idea.
> >
> > 1. As explained in reply to your first iteration, there shouldn't be a need for
> > including Buildroot variables in a surrounding Makefile.  
> 
> There is nothing you can say that will change the fact that a
> monolithic meta build system is too slow for the CI of our own
> components. So let's just agree to disagree on this point.

I'm jumping back to this very old thread. Could you explain (perhaps
re-explain) what you're trying to do ?

If you need to just re-run a specific step of a specific package, why
don't you use "make foo-build", "make foo-rebuild", etc. ?

It seems very hackish to extract the commands that Buildroot executes
to execute them outside the context of Buildroot. This doesn't really
help to make a case for accepting a patch like you're proposing. Could
you first convince us that there is a valid and reasonable use-case,
and then we can discuss how to support this use-case with Buildroot.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Stefan Becker]

Hi,

On Sun, Dec 16, 2018 at 3:54 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> I'm jumping back to this very old thread. Could you explain (perhaps
> re-explain) what you're trying to do ?

Modular builds to enable usable CI builds, i.e. any change a developer
pushes to one of the internal components will be available as a
flash-able image in less than 5 minutes. CI builds are not possible
with a monolithic mega meta build system approach like buildroot.

> If you need to just re-run a specific step of a specific package, why
> don't you use "make foo-build", "make foo-rebuild", etc. ?

buildroot meta build system assumes that the *whole* source code &
build tree is available. In a modular build approach that is not the
case for the "combining" build, e.g. the flash image build, as it only
receives the build artefacts from the module builds.

> It seems very hackish to extract the commands that Buildroot executes
> to execute them outside the context of Buildroot. This doesn't really
> help to make a case for accepting a patch like you're proposing. Could
> you first convince us that there is a valid and reasonable use-case,
> and then we can discuss how to support this use-case with Buildroot.

Understandable from the PoV of monolithic meta build system advocates
like buildroot, yocto, Android, ...

But I live in the real world and simply can''t ignore Amdahl's law.
Furthermore I can't, or management doesn't allow me, to spend huge
amounts of money on build machines that can run monolithic mega meta
build systems within time spans acceptable for CI (which in the end is
futile anyway due to Amdahl's law as SW only ever grows in size. Trust
me, I've been there...). Hence I've chosen a modular build approach to
address the issue.

As it became clear that this patch would not be accepted any time
soon, I reverted it and implemented a different approach based on
quoted var printout and sed post-processing. While not 100%
fool-proof, it does work for the few build variables that are required
for the modular build approach.

Anyway, I no longer work for the company that is using buildroot, so I
don't really care if you accept this patch or not.

Regards, Stefan

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Thomas Petazzoni]

Hello Stefan,

Thanks for the feedback.

On Sun, 16 Dec 2018 16:57:02 +0200, Stefan Becker wrote:

> > I'm jumping back to this very old thread. Could you explain (perhaps
> > re-explain) what you're trying to do ?  
> 
> Modular builds to enable usable CI builds, i.e. any change a developer
> pushes to one of the internal components will be available as a
> flash-able image in less than 5 minutes. CI builds are not possible
> with a monolithic mega meta build system approach like buildroot.

Yes, this I understand.

> > If you need to just re-run a specific step of a specific package, why
> > don't you use "make foo-build", "make foo-rebuild", etc. ?  
> 
> buildroot meta build system assumes that the *whole* source code &
> build tree is available. In a modular build approach that is not the
> case for the "combining" build, e.g. the flash image build, as it only
> receives the build artefacts from the module builds.

I'm not sure to follow you on this last sentence.

But yes, Buildroot assumes that the whole source code and build tree is
available.

> > It seems very hackish to extract the commands that Buildroot executes
> > to execute them outside the context of Buildroot. This doesn't really
> > help to make a case for accepting a patch like you're proposing. Could
> > you first convince us that there is a valid and reasonable use-case,
> > and then we can discuss how to support this use-case with Buildroot.  
> 
> Understandable from the PoV of monolithic meta build system advocates
> like buildroot, yocto, Android, ...
> 
> But I live in the real world and simply can''t ignore Amdahl's law.
> Furthermore I can't, or management doesn't allow me, to spend huge
> amounts of money on build machines that can run monolithic mega meta
> build systems within time spans acceptable for CI (which in the end is
> futile anyway due to Amdahl's law as SW only ever grows in size. Trust
> me, I've been there...). Hence I've chosen a modular build approach to
> address the issue.
> 
> As it became clear that this patch would not be accepted any time
> soon, I reverted it and implemented a different approach based on
> quoted var printout and sed post-processing. While not 100%
> fool-proof, it does work for the few build variables that are required
> for the modular build approach.

You did not answer my question: why didn't you use features of
Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ?

How in practice you were using those commands extracted from Buildroot
variables to achieve "modular builds" ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Stefan Becker]

Hi,

On Sun, Dec 16, 2018 at 5:14 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> You did not answer my question: why didn't you use features of
> Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ?

That would require the buildroot source tree to be available. Plus
adding all internal components, even the ones that have nothing to
with buildroot and do not use any of its host build tools, to the
buildroot meta build system for buildroot build to recognize the
generated build artefacts.

> How in practice you were using those commands extracted from Buildroot
> variables to achieve "modular builds" ?

That depends on the module, e.g.

* buildroot: output/target/, host/ plus selected buildroot meta build
system source files required for flash image build + module install
script
* kernel: output from modules_install & install + module install script
* ... (you get the picture)

But definitely no source code is included in the module build
artefact. All internal components have been removed from buildroot
meta build system, i.e. buildroot module build only builds components
included in its component list, i.e. buildroot module build only needs
to be executed after pulling changes from upstream.

Regards, Stefan

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Stefan Becker]

Hi,

On Sun, Dec 16, 2018 at 5:33 PM Stefan Becker <chemobejk@gmail.com> wrote:
>
> * buildroot: output/target/, host/ plus selected buildroot meta build
> system source files required for flash image build + module install
> script

To be more exact:

* build artefact 1: SDK
* build artefact 2: output/target + selected buildroot meta build
system source files required for flash image build + module install
script

Build artefact is a dependency for module builds that require target
toolchain, e.g. kernel build.

Regards, Stefan

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Trent Piepho]

On Sun, 2018-12-16 at 16:14 +0100, Thomas Petazzoni wrote:
> 
> > Understandable from the PoV of monolithic meta build system advocates
> > like buildroot, yocto, Android, ...
> > 
> > As it became clear that this patch would not be accepted any time
> > soon, I reverted it and implemented a different approach based on
> > quoted var printout and sed post-processing. While not 100%
> > fool-proof, it does work for the few build variables that are required
> > for the modular build approach.
> 
> You did not answer my question: why didn't you use features of
> Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ?

I am also using buildroot with modern style CI.  While I find <pkg>-
rebuild works great for manual developer workflow, there are issues
with CI integration.

In the CI system, we have many many git branches active at any time. 
They come and go frequently and don't start at the same branch point. 
There are also many build agents (themselves VMs) which perform the
actual build.

So say we have a new branch we wish to test.  The agent it is
dispatched to happens to have the output tree from the last successful
build on that agent (this is in itself very problematic!).  That last
build might be from anything in the git tree.  The head of a different
branch.  An ancestor commit but from months ago.  A commit from the
future w.r.t. the branch we are building now.

The chance of <pkg>-rebuild working correctly is quite small.  There
will undoubtedly be many other packages which have changed between the
last build on this agent and the once being attempted.

In addition to the above, preserving the build tree between jobs on a
agent ends up being a major obstacle in Bamboo.  Trying to archive and
restore the multi-gigabyte output is very slow.

And given two git hashes to a tree with a buildroot + br-extern, what
packages exactly must be rebuilt with <pkg>-rebuild to go from one hash
to the other?  This is also not so easy to find automatically.

So, my latest attempt to solve this problem - getting back quicker CI
feedback - was done another way.

What I do is build an SDK with buildroot.  This is part of a nightly
build process that also produces a firmware image and runs system tests
against real hardware, in addition to making an SDK.  Two SDKs in fact,
one for the real ARM target and another for an x86_64 target.

Each internal package has a CI job which fetches the latest SDK and
builds the package, pointing cmake or autoconf or whatever build system
the package uses at the SDK.  Since only the package is built, it can
produce feedback quickly.  The toolchain and libraries and so on is 
all from the SDK and so the same as in the actual buildroot build of
the entire firmware image.

The x86_64 SDK has the same package versions as the ARM firmware, made
via the same buildroot.  Because the target is x86_64, the binaries it
produces can be run locally on the x86_64 build agents.  I do this to
allow running the unit tests of a package quickly in CI, with needing
to use real hardware or qemu to run an ARM unit test suite.

I make a new package infra for our packages to try to handle them all
in a consistent way.  A complete .mk file for one might look like this:

TRANSPORT_PROXY_DEPENDENCIES = zeromq lttng-libust
$(eval $(impinj-cmake-package))

Given the SDK, there's really nothing else I need from buildroot, and
the above .mk file, to build this package in CI.

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Carlos Santos]

> From: "Trent Piepho" <tpiepho@impinj.com>
> To: "Thomas Petazzoni" <thomas.petazzoni@bootlin.com>, chemobejk at gmail.com
> Cc: "Yann Morin" <yann.morin.1998@free.fr>, "buildroot" <buildroot@buildroot.org>
> Sent: Segunda-feira, 17 de dezembro de 2018 18:08:58
> Subject: Re: [Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

> On Sun, 2018-12-16 at 16:14 +0100, Thomas Petazzoni wrote:
>> 
>> > Understandable from the PoV of monolithic meta build system advocates
>> > like buildroot, yocto, Android, ...
>> > 
>> > As it became clear that this patch would not be accepted any time
>> > soon, I reverted it and implemented a different approach based on
>> > quoted var printout and sed post-processing. While not 100%
>> > fool-proof, it does work for the few build variables that are required
>> > for the modular build approach.
>> 
>> You did not answer my question: why didn't you use features of
>> Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ?
> 
> I am also using buildroot with modern style CI.  While I find <pkg>-
> rebuild works great for manual developer workflow, there are issues
> with CI integration.
> 
> In the CI system, we have many many git branches active at any time.
> They come and go frequently and don't start at the same branch point.
> There are also many build agents (themselves VMs) which perform the
> actual build.
> 
> So say we have a new branch we wish to test.  The agent it is
> dispatched to happens to have the output tree from the last successful
> build on that agent (this is in itself very problematic!).  That last
> build might be from anything in the git tree.  The head of a different
> branch.  An ancestor commit but from months ago.  A commit from the
> future w.r.t. the branch we are building now.
> 
> The chance of <pkg>-rebuild working correctly is quite small.  There
> will undoubtedly be many other packages which have changed between the
> last build on this agent and the once being attempted.
> 
> In addition to the above, preserving the build tree between jobs on a
> agent ends up being a major obstacle in Bamboo.  Trying to archive and
> restore the multi-gigabyte output is very slow.
> 
> And given two git hashes to a tree with a buildroot + br-extern, what
> packages exactly must be rebuilt with <pkg>-rebuild to go from one hash
> to the other?  This is also not so easy to find automatically.

You may be interested on these threads:

    http://lists.busybox.net/pipermail/buildroot/2015-October/143107.html
    http://lists.busybox.net/pipermail/buildroot/2017-October/204165.html

> So, my latest attempt to solve this problem - getting back quicker CI
> feedback - was done another way.
> 
> What I do is build an SDK with buildroot.  This is part of a nightly
> build process that also produces a firmware image and runs system tests
> against real hardware, in addition to making an SDK.  Two SDKs in fact,
> one for the real ARM target and another for an x86_64 target.
> 
> Each internal package has a CI job which fetches the latest SDK and
> builds the package, pointing cmake or autoconf or whatever build system
> the package uses at the SDK.  Since only the package is built, it can
> produce feedback quickly.  The toolchain and libraries and so on is
> all from the SDK and so the same as in the actual buildroot build of
> the entire firmware image.
> 
> The x86_64 SDK has the same package versions as the ARM firmware, made
> via the same buildroot.  Because the target is x86_64, the binaries it
> produces can be run locally on the x86_64 build agents.  I do this to
> allow running the unit tests of a package quickly in CI, with needing
> to use real hardware or qemu to run an ARM unit test suite.
> 
> I make a new package infra for our packages to try to handle them all
> in a consistent way.  A complete .mk file for one might look like this:
> 
> TRANSPORT_PROXY_DEPENDENCIES = zeromq lttng-libust
> $(eval $(impinj-cmake-package))
> 
> Given the SDK, there's really nothing else I need from buildroot, and
> the above .mk file, to build this package in CI.

If you need an SDK perhaps you should try Yocto, which in its turn also
has some drawbacks. We (DATACOM) attempted to use it years ago and gave
up because the development process was too complicated and slow.

The main drawback of an SDK is that as you integrate new packages and
change the base system the SDK becomes outdated. It may work for a few
packages but as we started adding modules (currently there are 295) it
became impossible to build and publish new SDK versions at the same pace
without going crazy updating our development environments every day.

-- 
Carlos Santos (Casantos) - DATACOM, P&D

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Trent Piepho]

On Mon, 2018-12-17 at 22:24 -0200, Carlos Santos wrote:
> > 
> > What I do is build an SDK with buildroot.  This is part of a nightly
> > build process that also produces a firmware image and runs system tests
> > against real hardware, in addition to making an SDK.  Two SDKs in fact,
> > one for the real ARM target and another for an x86_64 target.
> > 
> > Each internal package has a CI job which fetches the latest SDK and
> > builds the package, pointing cmake or autoconf or whatever build system
> > the package uses at the SDK.  Since only the package is built, it can
> > produce feedback quickly.  The toolchain and libraries and so on is
> > all from the SDK and so the same as in the actual buildroot build of
> > the entire firmware image.
> > 
> > The x86_64 SDK has the same package versions as the ARM firmware, made
> > via the same buildroot.  Because the target is x86_64, the binaries it
> > produces can be run locally on the x86_64 build agents.  I do this to
> > allow running the unit tests of a package quickly in CI, with needing
> > to use real hardware or qemu to run an ARM unit test suite.
> > 
> > I make a new package infra for our packages to try to handle them all
> > in a consistent way.  A complete .mk file for one might look like this:
> > 
> > TRANSPORT_PROXY_DEPENDENCIES = zeromq lttng-libust
> > $(eval $(impinj-cmake-package))
> > 
> > Given the SDK, there's really nothing else I need from buildroot, and
> > the above .mk file, to build this package in CI.
> 
> If you need an SDK perhaps you should try Yocto, which in its turn also
> has some drawbacks. We (DATACOM) attempted to use it years ago and gave
> up because the development process was too complicated and slow.

I'm familiar with yocto, but chose buildroot for similar reasons. 
Buildroot has an SDK feature and it works pretty well for what I want. 
It's not an alternative to using buildroot, where you make a binary
BSP, check it into a repository, and the forever use and patch the
binary BSP and never use buildroot again.  I am not a fan of that
design.

> The main drawback of an SDK is that as you integrate new packages and
> change the base system the SDK becomes outdated. It may work for a few
> packages but as we started adding modules (currently there are 295) it
> became impossible to build and publish new SDK versions at the same pace
> without going crazy updating our development environments every day.

The key here is that the SDK is created by buildroot from the very same
repository that creates the firmware.  If your buildroot build is up to
date, then your SDK is up to date too.

What the SDK really is, is a way to make binary artifact of an entire
buildroot build, then use that artifact to rebuild one single package
without rebuilding the rest of buildroot.

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Yann E. MORIN]

Arnout, All,

On 2018-04-19 00:27 +0200, Arnout Vandecappelle spake thusly:
> On 16-04-18 13:58, Stefan Becker wrote:
> > If the variable content has newlines in it then the currently dumped
> > content can't be fed again to GNU make. Add the option DEFINE_VARS which
> > causes the variables to be dumped using
> > 
> >    define VAR
> >    ... line 1 ...
> >    ... line 2 ...
> >    ...
> >    endef
> 
>  I'm still not convinced that this is such a great idea.

Neither am I... And if I look around, I think the problem explained in
the commit log is already covered:

    $ make defconfig
    $ make -s printvars VARS=HOST_CARGO_INSTALL_CMDS QUOTED_VARS=YES
    HOST_CARGO_INSTALL_CMDS='       /usr/bin/install -D -m 0755 ./target/release/cargo /home/ymorin/dev/buildroot/O/host/bin/cargo
            /usr/bin/install -D package/cargo/config.in /home/ymorin/dev/buildroot/O/host/share/cargo/config
            /bin/sed -i -e '\''s/@RUSTC_TARGET_NAME@/i686-unknown-linux-gnu/'\'' /home/ymorin/dev/buildroot/O/host/share/cargo/config
            /bin/sed -i -e '\''s/@CROSS_PREFIX@/i686-buildroot-linux-uclibc-/'\'' /home/ymorin/dev/buildroot/O/host/share/cargo/config'

I.e. the newlines are embedded in the quoted value, so at the very
worst, one can re-inject this value in a new variable (with a bit of
shell trickery, yes, but rather easily).

Note that, when it encounters a \-continued line, make joins it with the
next line, squeezing leading and trailing spaces/atbs into a single
space. But otherwise, newlines are kept as-is. This can be seen above,
but also in this one, where it is even more obvious that newlines are
kept:

    $ eval make -s printvars VARS=SYSTEMD_USERS QUOTED_VARS=YES
    SYSTEMD_USERS=' - - input -1 * - - - Input device group
            - - systemd-journal -1 * - - - Journal
            - - render -1 * - - - DRI rendering nodes
            - - kvm -1 * - - - kvm nodes
            systemd-bus-proxy -1 systemd-bus-proxy -1 * - - - Proxy D-Bus messages to/from a bus
            systemd-journal-gateway -1 systemd-journal-gateway -1 * /var/log/journal - - Journal Gateway
            systemd-journal-remote -1 systemd-journal-remote -1 * /var/log/journal/remote - - Journal Remote
            systemd-journal-upload -1 systemd-journal-upload -1 * - - - Journal Upload



            '

Compare with the non-expanded, raw value:

    $ eval make -s printvars VARS=SYSTEMD_USERS QUOTED_VARS=YES RAW_VARS=YES
    SYSTEMD_USERS=' - - input -1 * - - - Input device group
            - - systemd-journal -1 * - - - Journal
            - - render -1 * - - - DRI rendering nodes
            - - kvm -1 * - - - kvm nodes
            systemd-bus-proxy -1 systemd-bus-proxy -1 * - - - Proxy D-Bus messages to/from a bus
            systemd-journal-gateway -1 systemd-journal-gateway -1 * /var/log/journal - - Journal Gateway
            systemd-journal-remote -1 systemd-journal-remote -1 * /var/log/journal/remote - - Journal Remote
            systemd-journal-upload -1 systemd-journal-upload -1 * - - - Journal Upload
            $(SYSTEMD_COREDUMP_USER)
            $(SYSTEMD_NETWORKD_USER)
            $(SYSTEMD_RESOLVED_USER)
            $(SYSTEMD_TIMESYNCD_USER)'

So, newlines are still present, and we can especially see this is the
case around the variables that are expanded to empty (the coredump,
networkd, resolved, and timesyncd users).

So, the whole excuse for the change, as explained in the commit log,
does not stand: newlines are preserved.

> 1. As explained in reply to your first iteration, there shouldn't be a need for
> including Buildroot variables in a surrounding Makefile.

That's because, as I remember and understand it, Stefan finds that
Buildroot is not fast enough for his use-case, and so extracts such
commands in an attempt to feed them back into his own build mechanism
around Buildroot. Or something like that; I did not understand the
details fully...

> 2. At least equally relevant would be output that is appropriate for Python,
> JSON, ...

I'm not sure I agree with you here: json is about representtaion, python
is a completely different language, while what Stefan is initially
asking for is a way to retrieve part of the makefile rules for feeding
them back into make.

But there is a solution for that: run make in debug mode, and scrape its
output.

> 3. The implementation is not complete. In case of RAW_VARS, it's pretty hard to
> use in practice because also all the referenced variables would need to be
> retrieved. In the expanded case, any remaining $ will be interpreted by your
> surrounding make instead of being passed down to the shell like they should. Try
> e.g. CANFESTIVAL_INSTALL_TARGET_CMDS.
> 
> Note that QUOTED_VARS is not perfect either, mainly because there are some make
> variables that are not legal shell variables (e.g. all the 4th stuff).

Right.

>  So, if anything changes in here, I'd rather have some output that is in some
> intermediate format that can easily be converted into or parsed by shell,
> python, etc.

And I'd rather not: quoted values already contain the expanded value
that is passed to the shell (and because raw shell code), and we can't
do better than that. The raw values contain the code as it was written
by the packager, and we can't do better than that either.

Anyway, as explained by Arnout, this new output format can't produce
working make code, so I've marked this patch as rejected in patchwork.

If you can come with a better solution that works in all cases and is
generic enough, then we can look back at it.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> > 
> > Updated the manual accordingly.
> > 
> > Signed-off-by: Stefan Becker <chemobejk@gmail.com>
> > ---
> >  Makefile                  | 10 +++++++---
> >  docs/manual/make-tips.txt | 20 ++++++++++++++++++++
> >  2 files changed, 27 insertions(+), 3 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index 0724f28f45..3e27195de0 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -974,9 +974,13 @@ printvars:
> >  		$(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \
> >  		$(if $(filter-out environment% default automatic, \
> >  				$(origin $V)), \
> > -		$(if $(QUOTED_VARS),\
> > -			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> > -			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
> > +		$(if $(DEFINE_VARS), \
> > +			$(info define $V) \
> > +			$(info $(if $(RAW_VARS),$(value $V),$($V))) \
> > +			$(info endef), \
> > +			$(if $(QUOTED_VARS),\
> > +				$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
> > +				$(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))))
> >  # ' Syntax colouring...
> >  
> >  .PHONY: clean
> > diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt
> > index ea1d825bef..ba87e5d873 100644
> > --- a/docs/manual/make-tips.txt
> > +++ b/docs/manual/make-tips.txt
> > @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables:
> >  
> >  - +VARS+ will limit the listing to variables which names match the
> >    specified make-pattern
> > +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve
> > +  newlines in the value
> >  - +QUOTED_VARS+, if set to +YES+, will single-quote the value
> >  - +RAW_VARS+, if set to +YES+, will print the unexpanded value
> >  
> > @@ -106,6 +108,24 @@ For example:
> >   BUSYBOX_RDEPENDENCIES=ncurses util-linux
> >  ----
> >  
> > +----
> > + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES
> > + define BUSYBOX_DEPENDENCIES
> > + skeleton toolchain
> > + endef
> > + define BUSYBOX_FINAL_ALL_DEPENDENCIES
> > + skeleton toolchain
> > + endef
> > + define BUSYBOX_FINAL_DEPENDENCIES
> > + skeleton toolchain
> > + endef
> > + define BUSYBOX_FINAL_PATCH_DEPENDENCIES
> > + endef
> > + define BUSYBOX_RDEPENDENCIES
> > + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache
> > + endef
> > +----
> > +
> >  ----
> >   $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES
> >   BUSYBOX_DEPENDENCIES='skeleton toolchain'
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines

[Stefan Becker]

Hi,

On Thu, Apr 19, 2018 at 10:47 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Compare with the non-expanded, raw value:
>
>     $ eval make -s printvars VARS=SYSTEMD_USERS QUOTED_VARS=YES RAW_VARS=YES
>     SYSTEMD_USERS=' - - input -1 * - - - Input device group
>             - - systemd-journal -1 * - - - Journal
>             - - render -1 * - - - DRI rendering nodes
>             - - kvm -1 * - - - kvm nodes
>             systemd-bus-proxy -1 systemd-bus-proxy -1 * - - - Proxy D-Bus messages to/from a bus
>             systemd-journal-gateway -1 systemd-journal-gateway -1 * /var/log/journal - - Journal Gateway
>             systemd-journal-remote -1 systemd-journal-remote -1 * /var/log/journal/remote - - Journal Remote
>             systemd-journal-upload -1 systemd-journal-upload -1 * - - - Journal Upload
>             $(SYSTEMD_COREDUMP_USER)
>             $(SYSTEMD_NETWORKD_USER)
>             $(SYSTEMD_RESOLVED_USER)
>             $(SYSTEMD_TIMESYNCD_USER)'
>
> So, newlines are still present, and we can especially see this is the
> case around the variables that are expanded to empty (the coredump,
> networkd, resolved, and timesyncd users).
>
> So, the whole excuse for the change, as explained in the commit log,
> does not stand: newlines are preserved.

Can you please explain to me how to detect reliably where the variable
content ends with QUOTED_VARS=YES? Example of variables with and
without newlines in the same output:

$ external/scripts/buildstep.sh printvars QUOTED_VARS=1
VARS="CANFESTIVAL_INSTALL_TARGET_CMDS SYSTEMD_USERS"
CANFESTIVAL_INSTALL_TARGET_CMDS='       for d in src drivers ; do
PATH="/workarea/stefanb/repos/buildroot-new/output/host/bin:/workarea/stefanb/repos/buildroot-new/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/stefanb/.local/bin:/home/stefanb/bin"
/usr/bin/make -j1 -C ./$d install
PYTHON=/workarea/stefanb/repos/buildroot-new/output/host/bin/python2
DESTDIR=/workarea/stefanb/repos/buildroot-new/output/target || exit 1
; done'
SYSTEMD_USERS=' - - input -1 * - - - Input device group
        - - systemd-journal -1 * - - - Journal
...
        systemd-resolve -1 systemd-resolve -1 * - - - Network Name
Resolution Manager
        systemd-timesync -1 systemd-timesync -1 * - - - Network Time
Synchronization'

$ external/scripts/buildstep.sh printvars QUOTED_VARS=1
VARS="CANFESTIVAL_INSTALL_TARGET_CMDS SYSTEMD_USERS" | wc -l
13

MULTILINE_VARS=YES adds an additional seperator that makes it possible
to reliably detect the end of the variable content.


One improvement idea would be to use the content of MULTILINE_VARS as
the separator, e.g.

.... MULTILINE_VARS=some-random-seperator-XYZ ...
START-some-random-seperator-XYZ VAR
line 1
line 2
...
END-some-random-seperator-XYZ VAR

That way you could cases where the variable content matches a fixed separator.

Regards, Stefan