* [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time
@ 2022-08-01 20:42 Yann E. MORIN
2022-08-02 9:23 ` Quentin Schulz
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Yann E. MORIN @ 2022-08-01 20:42 UTC (permalink / raw)
To: buildroot; +Cc: Quentin Schulz, Roosen Henri, Yann E. MORIN, Thomas Petazzoni
Commit 5c54c3ef3db2 (Makefile: workaround make 4.3 issue for 'printvars
and 'show-vars') did not fully fix the show-vars case, which still
segfaults.
Overall, show-vars generates a JSON blurb. That is supposed to be
machine-readable, so we do not care that the variables are sorted, so
we get rid of it to (slightly) simplify the code.
Then, we currently iterate twice on the list of variables: the first one
to filter-out the 'internal' variables, and the second one to filter
only the variables matching the pattern. We can do away by iterating
only once, and applying both filters at once.
Since we now have an 'and' condition, we can take advantage of it: when
none of the items in $(and) are empty, $(and) evaluates to the last
item, while it evaluates to empty if any of the items is empty. So we
can coalesce the $(if) and $(and) together: $(if $(and a,b),c) is
equivalent to: $(and a,b,c) ; this gains us one parentheses depth.
Finally, the cause for the segfault is an overly-long call to $(info).
Reducing that is not easy: we want to call clean-json on the whole of
the JSON blurb, so we can't emit the individual variables one by one, or
the trailing comma would not be trimmed away.
So, we go crazy: we just output each word from clean-json with $(info).
We can do that, because mk-json-str transforms all spaces in a string
to an escaped UTF-8 sequence, so we will never have spaces in values;
the keys are the variables, so they won't have spaces either; spaces in
the rest of the JSON blurb are totally optional, so we don't care how
many there are. We know there are spaces, because we explicitly
introduce some (after "expanded" or "raw", for example), so we should
never hit a too-big word for $(info) to print.
Thanks to Henri for the suggestion to push $(info) further inside the
macro.
Reported-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Roosen Henri <Henri.Roosen@ginzinger.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Changes v1 -> v2:
- coalesce $(if) with $(and)
- don't output a trailing $(space) when calling $(info)
---
Makefile | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile
index a743e42f91..07b594ea8a 100644
--- a/Makefile
+++ b/Makefile
@@ -1073,17 +1073,24 @@ printvars:
show-vars: VARS?=%
show-vars:
@:
- $(info $(call clean-json, { \
+ $(foreach i, \
+ $(call clean-json, { \
$(foreach V, \
- $(sort $(foreach X, $(.VARIABLES), $(filter $(VARS),$(X)))), \
- $(if $(filter-out environment% default automatic, $(origin $V)), \
+ $(.VARIABLES), \
+ $(and $(filter $(VARS),$(V)) \
+ , \
+ $(filter-out environment% default automatic, $(origin $V)) \
+ , \
"$V": { \
"expanded": $(call mk-json-str,$($V))$(comma) \
"raw": $(call mk-json-str,$(value $V)) \
}$(comma) \
) \
) \
- } ))
+ } ) \
+ , \
+ $(info $(i)) \
+ )
.PHONY: clean
clean:
--
2.25.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time
2022-08-01 20:42 [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time Yann E. MORIN
@ 2022-08-02 9:23 ` Quentin Schulz
2022-08-03 16:37 ` Thomas Petazzoni via buildroot
2022-09-14 9:29 ` Peter Korsgaard
2 siblings, 0 replies; 4+ messages in thread
From: Quentin Schulz @ 2022-08-02 9:23 UTC (permalink / raw)
To: Yann E. MORIN, buildroot; +Cc: Roosen Henri, Thomas Petazzoni
Hi Yann,
On 8/1/22 22:42, Yann E. MORIN wrote:
> Commit 5c54c3ef3db2 (Makefile: workaround make 4.3 issue for 'printvars
> and 'show-vars') did not fully fix the show-vars case, which still
> segfaults.
>
> Overall, show-vars generates a JSON blurb. That is supposed to be
> machine-readable, so we do not care that the variables are sorted, so
> we get rid of it to (slightly) simplify the code.
>
> Then, we currently iterate twice on the list of variables: the first one
> to filter-out the 'internal' variables, and the second one to filter
> only the variables matching the pattern. We can do away by iterating
> only once, and applying both filters at once.
>
> Since we now have an 'and' condition, we can take advantage of it: when
> none of the items in $(and) are empty, $(and) evaluates to the last
> item, while it evaluates to empty if any of the items is empty. So we
> can coalesce the $(if) and $(and) together: $(if $(and a,b),c) is
> equivalent to: $(and a,b,c) ; this gains us one parentheses depth.
>
> Finally, the cause for the segfault is an overly-long call to $(info).
> Reducing that is not easy: we want to call clean-json on the whole of
> the JSON blurb, so we can't emit the individual variables one by one, or
> the trailing comma would not be trimmed away.
>
> So, we go crazy: we just output each word from clean-json with $(info).
>
> We can do that, because mk-json-str transforms all spaces in a string
> to an escaped UTF-8 sequence, so we will never have spaces in values;
> the keys are the variables, so they won't have spaces either; spaces in
> the rest of the JSON blurb are totally optional, so we don't care how
> many there are. We know there are spaces, because we explicitly
> introduce some (after "expanded" or "raw", for example), so we should
> never hit a too-big word for $(info) to print.
>
> Thanks to Henri for the suggestion to push $(info) further inside the
> macro.
>
> Reported-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Roosen Henri <Henri.Roosen@ginzinger.com>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>
Tested-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Upon request, "time efficiency" regression test on old make:
From within debian:buster container (make 4.2.1), from master
2a592f0b2a400825a95e19113ce996bbc5c431d7:
# make raspberrypi4_defconfig
# time make show-vars > foo.json
real 0m19.001s
user 0m15.828s
sys 0m3.305s
# time make printvars VARS='%' > foo.txt
real 0m16.271s
user 0m13.056s
sys 0m3.360s
# ls -l foo.json foo.txt
-rw-r--r--. 1 qschulz qschulz 62448149 Aug 2 09:06 foo.json
-rw-r--r--. 1 qschulz qschulz 32813206 Aug 2 09:07 foo.txt
From within debian:buster container (make 4.2.1), with this patch NOT
applied and 5c54c3ef3d commit reverted:
# make raspberrypi4_defconfig
# time make show-vars > foo.json
real 0m24.325s
user 0m19.617s
sys 0m4.905s
# time make printvars VARS='%' > foo.txt
real 0m16.223s
user 0m12.895s
sys 0m3.469s
# ls -l foo.json foo.txt
-rw-r--r--. 1 qschulz qschulz 62448149 Aug 2 09:04 foo.json
-rw-r--r--. 1 qschulz qschulz 32813206 Aug 2 09:09 foo.txt
From within debian:buster container (make 4.2.1), with this patch applied:
# make raspberrypi4_defconfig
# time make show-vars > foo.json
real 0m22.634s
user 0m17.542s
sys 0m5.202s
# time make printvars VARS='%' > foo.txt
real 0m16.829s
user 0m13.381s
sys 0m3.579s
# ls -l foo.json foo.txt
-rw-r--r--. 1 qschulz qschulz 62448149 Aug 2 09:02 foo.json
-rw-r--r--. 1 qschulz qschulz 32813206 Aug 2 09:14 foo.txt
I wouldn't trust too much the reports from the time tool because I had
some runs where it was twice longer for some reason.
In short, it does not seem to impact too much time-wise.
Thanks,
Quentin
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time
2022-08-01 20:42 [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time Yann E. MORIN
2022-08-02 9:23 ` Quentin Schulz
@ 2022-08-03 16:37 ` Thomas Petazzoni via buildroot
2022-09-14 9:29 ` Peter Korsgaard
2 siblings, 0 replies; 4+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-08-03 16:37 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Quentin Schulz, Roosen Henri, buildroot
On Mon, 1 Aug 2022 22:42:27 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> Commit 5c54c3ef3db2 (Makefile: workaround make 4.3 issue for 'printvars
> and 'show-vars') did not fully fix the show-vars case, which still
> segfaults.
>
> Overall, show-vars generates a JSON blurb. That is supposed to be
> machine-readable, so we do not care that the variables are sorted, so
> we get rid of it to (slightly) simplify the code.
>
> Then, we currently iterate twice on the list of variables: the first one
> to filter-out the 'internal' variables, and the second one to filter
> only the variables matching the pattern. We can do away by iterating
> only once, and applying both filters at once.
>
> Since we now have an 'and' condition, we can take advantage of it: when
> none of the items in $(and) are empty, $(and) evaluates to the last
> item, while it evaluates to empty if any of the items is empty. So we
> can coalesce the $(if) and $(and) together: $(if $(and a,b),c) is
> equivalent to: $(and a,b,c) ; this gains us one parentheses depth.
>
> Finally, the cause for the segfault is an overly-long call to $(info).
> Reducing that is not easy: we want to call clean-json on the whole of
> the JSON blurb, so we can't emit the individual variables one by one, or
> the trailing comma would not be trimmed away.
>
> So, we go crazy: we just output each word from clean-json with $(info).
>
> We can do that, because mk-json-str transforms all spaces in a string
> to an escaped UTF-8 sequence, so we will never have spaces in values;
> the keys are the variables, so they won't have spaces either; spaces in
> the rest of the JSON blurb are totally optional, so we don't care how
> many there are. We know there are spaces, because we explicitly
> introduce some (after "expanded" or "raw", for example), so we should
> never hit a too-big word for $(info) to print.
>
> Thanks to Henri for the suggestion to push $(info) further inside the
> macro.
>
> Reported-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Roosen Henri <Henri.Roosen@ginzinger.com>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>
> ---
> Changes v1 -> v2:
> - coalesce $(if) with $(and)
> - don't output a trailing $(space) when calling $(info)
> ---
> Makefile | 15 +++++++++++----
> 1 file changed, 11 insertions(+), 4 deletions(-)
Applied to master, thanks.
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time
2022-08-01 20:42 [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time Yann E. MORIN
2022-08-02 9:23 ` Quentin Schulz
2022-08-03 16:37 ` Thomas Petazzoni via buildroot
@ 2022-09-14 9:29 ` Peter Korsgaard
2 siblings, 0 replies; 4+ messages in thread
From: Peter Korsgaard @ 2022-09-14 9:29 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Quentin Schulz, Roosen Henri, Thomas Petazzoni, buildroot
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> Commit 5c54c3ef3db2 (Makefile: workaround make 4.3 issue for 'printvars
> and 'show-vars') did not fully fix the show-vars case, which still
> segfaults.
> Overall, show-vars generates a JSON blurb. That is supposed to be
> machine-readable, so we do not care that the variables are sorted, so
> we get rid of it to (slightly) simplify the code.
> Then, we currently iterate twice on the list of variables: the first one
> to filter-out the 'internal' variables, and the second one to filter
> only the variables matching the pattern. We can do away by iterating
> only once, and applying both filters at once.
> Since we now have an 'and' condition, we can take advantage of it: when
> none of the items in $(and) are empty, $(and) evaluates to the last
> item, while it evaluates to empty if any of the items is empty. So we
> can coalesce the $(if) and $(and) together: $(if $(and a,b),c) is
> equivalent to: $(and a,b,c) ; this gains us one parentheses depth.
> Finally, the cause for the segfault is an overly-long call to $(info).
> Reducing that is not easy: we want to call clean-json on the whole of
> the JSON blurb, so we can't emit the individual variables one by one, or
> the trailing comma would not be trimmed away.
> So, we go crazy: we just output each word from clean-json with $(info).
> We can do that, because mk-json-str transforms all spaces in a string
> to an escaped UTF-8 sequence, so we will never have spaces in values;
> the keys are the variables, so they won't have spaces either; spaces in
> the rest of the JSON blurb are totally optional, so we don't care how
> many there are. We know there are spaces, because we explicitly
> introduce some (after "expanded" or "raw", for example), so we should
> never hit a too-big word for $(info) to print.
> Thanks to Henri for the suggestion to push $(info) further inside the
> macro.
> Reported-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Roosen Henri <Henri.Roosen@ginzinger.com>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> Changes v1 -> v2:
> - coalesce $(if) with $(and)
> - don't output a trailing $(space) when calling $(info)
Committed to 2022.05.x and 2022.02.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-09-14 9:29 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-08-01 20:42 [Buildroot] [PATCHv2] Makefile: fix show-vars for good this time Yann E. MORIN
2022-08-02 9:23 ` Quentin Schulz
2022-08-03 16:37 ` Thomas Petazzoni via buildroot
2022-09-14 9:29 ` Peter Korsgaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox