* [Buildroot] [RFC 0/3] Speeding up the Makefile parsing
@ 2014-03-23 17:42 Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function Thomas Petazzoni
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Thomas Petazzoni @ 2014-03-23 17:42 UTC (permalink / raw)
To: buildroot
Hello,
The current strategy of Buildroot with regard to .mk files is to
always include all of them. However, with an ever-increasing number of
packages, the number of .mk files to parse is growing significantly,
and this makes the execution of any trivial make target take at least
12 seconds on a relatively fast SSD equipped laptop. For example, a
simple thing like "make help" or "make external-deps" takes 12 seconds
solely because of the parsing time of more than a thousand of package
makefiles.
Of course, in a given configuration, the vast majority of these
package makefiles are not useful. The idea of this patch series is to
start the discussion on whether we can find a solution to only include
the makefiles that are actually useful for the current configuration,
at least for the packages.
The proposed RFC solution is to look at the BR2_PACKAGE_<foo>
variables, and from them derive the name of the package .mk files to
be included. It is a simple change in the main Makefile, but it has
two major consequences that we need to discuss on whether they are
acceptable or not:
* We need to have BR2_PACKAGE_HOST_<foo> options for *all* host
packages. Of course, the vast majority of them can be blind
options, but it means that we have to ensure all the host packages
are properly selected at the Config.in level. That's a fairly
significant change.
* Since the logic assume that BR2_PACKAGE_FOO is always
package/foo/foo.mk, it means we have to 1/ guarantee that the
package option name matches the package file name, and 2/ guarantee
that we don't use any sub-directories to store packages.
Again, this is only a RFC set of patches. There are not meant for
inclusion, and it is only sent to get the discussion started about
this topic. I would have no real problem if the conclusion of the
discussion would be that any solution to the problem has too many
constraints compared to the benefits. Even though I believe the
parsing time of the makefiles is going sooner or later to be a problem
as the number of packages we support continues to grow.
Also, the patches are just the result of an hour or two of hacking
while flying, it hasn't been extensively tested, and there might be
corner cases I have overlooked.
Best regards,
Thomas
Thomas Petazzoni (3):
pkg-utils: introduce a make-based LOWERCASE function
Makefile: include only the package .mk files that are needed
package: add a few needed host options for a few packages
Makefile | 7 ++++++-
package/expat/Config.in | 2 ++
package/expat/Config.in.host | 4 ++++
package/pkg-utils.mk | 15 +++++++--------
package/pkgconf/Config.in | 2 ++
package/pkgconf/Config.in.host | 2 ++
package/python/Config.in | 3 +++
package/python/Config.in.host | 3 +++
8 files changed, 29 insertions(+), 9 deletions(-)
create mode 100644 package/expat/Config.in.host
create mode 100644 package/pkgconf/Config.in.host
create mode 100644 package/python/Config.in.host
--
1.8.3.2
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function
2014-03-23 17:42 [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Thomas Petazzoni
@ 2014-03-23 17:43 ` Thomas Petazzoni
2014-03-25 21:47 ` Arnout Vandecappelle
2014-03-23 17:43 ` [Buildroot] [RFC 2/3] Makefile: include only the package .mk files that are needed Thomas Petazzoni
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Thomas Petazzoni @ 2014-03-23 17:43 UTC (permalink / raw)
To: buildroot
Until now, our UPPERCASE function was implemented purely in make for
performance reasons, but our LOWERCASE function was implemented by
calling "tr", which was reasonable due to the fact that LOWERCASE was
rarely used.
An upcoming change is going to make LOWERCASE used quite a bit more,
so we turn it into a pure make function.
Moreover, we want this LOWERCASE function to turn a "_" into a "-" and
not a ".", so we slightly adjust the existing FROM and TO lists to
make this possible. This doesn't change the behavior of the UPPERCASE
macro because both "-" and "." are converted into "_" by this
function.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
package/pkg-utils.mk | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 91a1981..8c3e534 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -13,7 +13,7 @@
# as this macro is used a lot it matters
# This works by creating translation character pairs (E.G. a:A b:B)
# and then looping though all of them running $(subst from,to,text)
-[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z . -
+[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
[TO] := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
UPPERCASE = $(strip $(eval __tmp := $1) \
@@ -23,13 +23,12 @@ UPPERCASE = $(strip $(eval __tmp := $1) \
$(__tmp)))) \
$(__tmp))
-# LOWERCASE macro -- transforms its arguments to lowercase
-# The above non-tr implementation is not needed, because LOWERCASE is not
-# called very often
-
-define LOWERCASE
-$(shell echo $1 | tr '[:upper:]' '[:lower:]')
-endef
+LOWERCASE = $(strip $(eval __tmp := $1) \
+ $(foreach c, $(join $(addsuffix :,$([TO])),$([FROM])), \
+ $(eval __tmp := \
+ $(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),\
+ $(__tmp)))) \
+ $(__tmp))
#
# Manipulation of .config files based on the Kconfig
--
1.8.3.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 2/3] Makefile: include only the package .mk files that are needed
2014-03-23 17:42 [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function Thomas Petazzoni
@ 2014-03-23 17:43 ` Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 3/3] package: add a few needed host options for a few packages Thomas Petazzoni
2014-03-23 20:38 ` [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Peter Korsgaard
3 siblings, 0 replies; 9+ messages in thread
From: Thomas Petazzoni @ 2014-03-23 17:43 UTC (permalink / raw)
To: buildroot
Instead of including all the package .mk files, that take a long time
to parse, this commit changes the Makefile logic to only include the
package .mk files for the selected packages. To do so, it looks at all
the BR2_PACKAGE_ variables, convert their name into a package file
name, and includes this file. This way, if BR2_PACKAGE_FOO=y or
BR2_PACKAGE_HOST_FOO=y are set, the package/foo/foo.mk is included.
The main purpose of this change is to significantly speed up the
execution of make when there is actually nothing or little to do: an
empty Buildroot invocation currently takes 12 seconds on a relatively
fast SSD equipped laptop, and this is only going to grow up with the
number of packages added in Buildroot.
This change of course has a number of consequences:
* We must define blind BR2_PACKAGE_HOST_<foo> options for *all* host
packages, and ensure they are selected as needed at the Config.in
level. Therefore for host packages, describing the dependency in
the .mk file would no longer be useful.
* The name of a package option must exactly match the location of the
package, which also means we can no longer have sub-directories for
packages.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Makefile | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index 2b13917..7441882 100644
--- a/Makefile
+++ b/Makefile
@@ -386,7 +386,12 @@ ifneq ($(PACKAGE_OVERRIDE_FILE),)
-include $(PACKAGE_OVERRIDE_FILE)
endif
-include $(sort $(wildcard package/*/*.mk))
+define pkgopt
+$(call LOWERCASE,$(patsubst BR2_PACKAGE_%,%,$(patsubst BR2_PACKAGE_HOST_%,%,$(1))))
+endef
+
+-include $(sort $(foreach V,$(filter BR2_PACKAGE_%,$(.VARIABLES)),\
+ $(wildcard $(TOPDIR)/package/$(call pkgopt,$V)/$(call pkgopt,$V).mk)))
include boot/common.mk
include linux/linux.mk
--
1.8.3.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 3/3] package: add a few needed host options for a few packages
2014-03-23 17:42 [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 2/3] Makefile: include only the package .mk files that are needed Thomas Petazzoni
@ 2014-03-23 17:43 ` Thomas Petazzoni
2014-03-25 22:06 ` Arnout Vandecappelle
2014-03-23 20:38 ` [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Peter Korsgaard
3 siblings, 1 reply; 9+ messages in thread
From: Thomas Petazzoni @ 2014-03-23 17:43 UTC (permalink / raw)
To: buildroot
This shows a few examples of what is needed to get the host package
dependencies correct.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
package/expat/Config.in | 2 ++
package/expat/Config.in.host | 4 ++++
package/pkgconf/Config.in | 2 ++
package/pkgconf/Config.in.host | 2 ++
package/python/Config.in | 3 +++
package/python/Config.in.host | 3 +++
6 files changed, 16 insertions(+)
create mode 100644 package/expat/Config.in.host
create mode 100644 package/pkgconf/Config.in.host
create mode 100644 package/python/Config.in.host
diff --git a/package/expat/Config.in b/package/expat/Config.in
index 758fb7d..5b4bd39 100644
--- a/package/expat/Config.in
+++ b/package/expat/Config.in
@@ -1,3 +1,5 @@
+source "package/expat/Config.in.host"
+
config BR2_PACKAGE_EXPAT
bool "expat"
help
diff --git a/package/expat/Config.in.host b/package/expat/Config.in.host
new file mode 100644
index 0000000..9d0b0af
--- /dev/null
+++ b/package/expat/Config.in.host
@@ -0,0 +1,4 @@
+config BR2_PACKAGE_HOST_EXPAT
+ bool
+ select BR2_PACKAGE_HOST_PKGCONF
+
diff --git a/package/pkgconf/Config.in b/package/pkgconf/Config.in
index f95847f..56c1c48 100644
--- a/package/pkgconf/Config.in
+++ b/package/pkgconf/Config.in
@@ -1,3 +1,5 @@
+source "package/pkgconf/Config.in.host"
+
config BR2_PACKAGE_PKGCONF
bool "pkgconf"
help
diff --git a/package/pkgconf/Config.in.host b/package/pkgconf/Config.in.host
new file mode 100644
index 0000000..e324360
--- /dev/null
+++ b/package/pkgconf/Config.in.host
@@ -0,0 +1,2 @@
+config BR2_PACKAGE_HOST_PKGCONF
+ bool
diff --git a/package/python/Config.in b/package/python/Config.in
index a0c78c6..253f36b 100644
--- a/package/python/Config.in
+++ b/package/python/Config.in
@@ -1,9 +1,12 @@
+source "package/python/Config.in.host"
+
config BR2_PACKAGE_PYTHON
bool "python"
depends on BR2_USE_WCHAR
# uses fork()
depends on BR2_USE_MMU
select BR2_PACKAGE_LIBFFI
+ select BR2_PACKAGE_HOST_PYTHON
help
The python language interpreter.
diff --git a/package/python/Config.in.host b/package/python/Config.in.host
new file mode 100644
index 0000000..2eca521
--- /dev/null
+++ b/package/python/Config.in.host
@@ -0,0 +1,3 @@
+config BR2_PACKAGE_HOST_PYTHON
+ bool
+ select BR2_PACKAGE_HOST_EXPAT
--
1.8.3.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 0/3] Speeding up the Makefile parsing
2014-03-23 17:42 [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Thomas Petazzoni
` (2 preceding siblings ...)
2014-03-23 17:43 ` [Buildroot] [RFC 3/3] package: add a few needed host options for a few packages Thomas Petazzoni
@ 2014-03-23 20:38 ` Peter Korsgaard
2014-03-23 20:57 ` Thomas Petazzoni
3 siblings, 1 reply; 9+ messages in thread
From: Peter Korsgaard @ 2014-03-23 20:38 UTC (permalink / raw)
To: buildroot
>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> Hello,
> The current strategy of Buildroot with regard to .mk files is to
> always include all of them. However, with an ever-increasing number of
> packages, the number of .mk files to parse is growing significantly,
> and this makes the execution of any trivial make target take at least
> 12 seconds on a relatively fast SSD equipped laptop. For example, a
> simple thing like "make help" or "make external-deps" takes 12 seconds
> solely because of the parsing time of more than a thousand of package
> makefiles.
.. and how fast is it after this series?
> The proposed RFC solution is to look at the BR2_PACKAGE_<foo>
> variables, and from them derive the name of the package .mk files to
> be included. It is a simple change in the main Makefile, but it has
> two major consequences that we need to discuss on whether they are
> acceptable or not:
> * We need to have BR2_PACKAGE_HOST_<foo> options for *all* host
> packages. Of course, the vast majority of them can be blind
> options, but it means that we have to ensure all the host packages
> are properly selected at the Config.in level. That's a fairly
> significant change.
I don't think this is too bad. Maybe we could look into generating the
_DEPENDENCIES line from depends/select statements in the Config.in
automatically, then we only need to explicitly handle the optional
dependencies in the .mk file (or decide to also handle them in Kconfig
if possible).
> * Since the logic assume that BR2_PACKAGE_FOO is always
> package/foo/foo.mk, it means we have to 1/ guarantee that the
> package option name matches the package file name, and 2/ guarantee
> that we don't use any sub-directories to store packages.
The 2nd part is more serious as we use subdirs today to group related
packages and have supported them "forever". We also used to tell people
(before BR2_EXTERNAL) to put their own packages under package/<company>.
I guess playing tricks with $(wildcard) to find the .mk file is going to
add quite some performance overhead.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 0/3] Speeding up the Makefile parsing
2014-03-23 20:38 ` [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Peter Korsgaard
@ 2014-03-23 20:57 ` Thomas Petazzoni
2014-03-25 21:03 ` Arnout Vandecappelle
0 siblings, 1 reply; 9+ messages in thread
From: Thomas Petazzoni @ 2014-03-23 20:57 UTC (permalink / raw)
To: buildroot
Dear Peter Korsgaard,
On Sun, 23 Mar 2014 21:38:50 +0100, Peter Korsgaard wrote:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
>
> > Hello,
> > The current strategy of Buildroot with regard to .mk files is to
> > always include all of them. However, with an ever-increasing number of
> > packages, the number of .mk files to parse is growing significantly,
> > and this makes the execution of any trivial make target take at least
> > 12 seconds on a relatively fast SSD equipped laptop. For example, a
> > simple thing like "make help" or "make external-deps" takes 12 seconds
> > solely because of the parsing time of more than a thousand of package
> > makefiles.
>
> .. and how fast is it after this series?
Before the series:
$ time make help
[...]
real 0m15.760s
user 0m14.818s
sys 0m0.969s
After the series:
$ time make help
[...]
real 0m1.209s
user 0m0.807s
sys 0m0.416s
> > The proposed RFC solution is to look at the BR2_PACKAGE_<foo>
> > variables, and from them derive the name of the package .mk files to
> > be included. It is a simple change in the main Makefile, but it has
> > two major consequences that we need to discuss on whether they are
> > acceptable or not:
>
> > * We need to have BR2_PACKAGE_HOST_<foo> options for *all* host
> > packages. Of course, the vast majority of them can be blind
> > options, but it means that we have to ensure all the host packages
> > are properly selected at the Config.in level. That's a fairly
> > significant change.
>
> I don't think this is too bad. Maybe we could look into generating the
> _DEPENDENCIES line from depends/select statements in the Config.in
> automatically, then we only need to explicitly handle the optional
> dependencies in the .mk file (or decide to also handle them in Kconfig
> if possible).
Why not, but doing this would require a pre-processing step to process
the Config.in file and generates the makefile dependencies from it.
Also, having the mandatory dependencies handled automatically but not
the optional dependencies seems a bit weird, no?
That being said, doing for host packages what we do today for target
packages doesn't that horrible, especially since we don't have all the
crazy dependencies on toolchain options that are complicated to handle
for target packages.
> > * Since the logic assume that BR2_PACKAGE_FOO is always
> > package/foo/foo.mk, it means we have to 1/ guarantee that the
> > package option name matches the package file name, and 2/ guarantee
> > that we don't use any sub-directories to store packages.
>
> The 2nd part is more serious as we use subdirs today to group related
> packages and have supported them "forever". We also used to tell people
> (before BR2_EXTERNAL) to put their own packages under package/<company>.
>
> I guess playing tricks with $(wildcard) to find the .mk file is going to
> add quite some performance overhead.
Yes. This part is more problematic, and I don't have a good solution
for it.
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 0/3] Speeding up the Makefile parsing
2014-03-23 20:57 ` Thomas Petazzoni
@ 2014-03-25 21:03 ` Arnout Vandecappelle
0 siblings, 0 replies; 9+ messages in thread
From: Arnout Vandecappelle @ 2014-03-25 21:03 UTC (permalink / raw)
To: buildroot
Hi Thomas,
On 23/03/14 21:57, Thomas Petazzoni wrote:
> Dear Peter Korsgaard,
>
> On Sun, 23 Mar 2014 21:38:50 +0100, Peter Korsgaard wrote:
>>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
>>
>> > Hello,
>> > The current strategy of Buildroot with regard to .mk files is to
>> > always include all of them. However, with an ever-increasing number of
>> > packages, the number of .mk files to parse is growing significantly,
>> > and this makes the execution of any trivial make target take at least
>> > 12 seconds on a relatively fast SSD equipped laptop. For example, a
>> > simple thing like "make help" or "make external-deps" takes 12 seconds
>> > solely because of the parsing time of more than a thousand of package
>> > makefiles.
>>
>> .. and how fast is it after this series?
>
> Before the series:
>
> $ time make help
> [...]
> real 0m15.760s
> user 0m14.818s
> sys 0m0.969s
>
> After the series:
>
> $ time make help
> [...]
> real 0m1.209s
> user 0m0.807s
> sys 0m0.416s
I was at first a bit wary of your proposed approach, as I would have
preferred to still include all .mk files and instead optimise the
implementation of generic-package. However, I did some further
experiments with
make -qp > /dev/null
on a reasonable-sized .config (make -qp is what is used by bash
completion so is very relevant IMHO).
Before the series:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m3.452s
user 0m2.980s
sys 0m0.092s
After the series:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m0.389s
user 0m0.280s
sys 0m0.004s
Upper bound of generic-package optimisation (just define *-package macros
as empty):
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m1.234s
user 0m0.872s
sys 0m0.064s
And on my allpackageyesconfig, there is even a tiny speedup (due to the
twenty-ish packages that are not included):
Before:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m3.325s
user 0m3.044s
sys 0m0.116s
After:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m3.040s
user 0m2.792s
sys 0m0.096s
With those kind of results, the approach you propose really seems the
way to go.
>> > The proposed RFC solution is to look at the BR2_PACKAGE_<foo>
>> > variables, and from them derive the name of the package .mk files to
>> > be included. It is a simple change in the main Makefile, but it has
>> > two major consequences that we need to discuss on whether they are
>> > acceptable or not:
>>
>> > * We need to have BR2_PACKAGE_HOST_<foo> options for *all* host
>> > packages. Of course, the vast majority of them can be blind
>> > options, but it means that we have to ensure all the host packages
>> > are properly selected at the Config.in level. That's a fairly
>> > significant change.
>>
>> I don't think this is too bad. Maybe we could look into generating the
>> _DEPENDENCIES line from depends/select statements in the Config.in
>> automatically, then we only need to explicitly handle the optional
>> dependencies in the .mk file (or decide to also handle them in Kconfig
>> if possible).
Actually, we could generate the Config.in from the .mk file (like
OpenWRT does). Probably easier than the reverse.
>
> Why not, but doing this would require a pre-processing step to process
> the Config.in file and generates the makefile dependencies from it.
> Also, having the mandatory dependencies handled automatically but not
> the optional dependencies seems a bit weird, no?
I think that Peter meant that the _DEPENDENCIES variable would be
defined by some make function magic, not that a .mk file is generated.
Something like:
_DEPENDENCIES = $(call LOWERCASE,$(shell \
sed -n '/.*select BR2_PACKAGE_\([A-Z0-9_]*\).*/s//\1/p' $(pkgdir)))
>
> That being said, doing for host packages what we do today for target
> packages doesn't that horrible, especially since we don't have all the
> crazy dependencies on toolchain options that are complicated to handle
> for target packages.
Right.
There are anyway a number of other reasons why it would be nice to have
blind options for the host packages.
* Facilitate package check script.
* Remove the hacks to generate *-source target.
* Probably others.
>
>> > * Since the logic assume that BR2_PACKAGE_FOO is always
>> > package/foo/foo.mk, it means we have to 1/ guarantee that the
>> > package option name matches the package file name, and 2/ guarantee
>> > that we don't use any sub-directories to store packages.
>>
>> The 2nd part is more serious as we use subdirs today to group related
>> packages and have supported them "forever". We also used to tell people
>> (before BR2_EXTERNAL) to put their own packages under package/<company>.
>>
>> I guess playing tricks with $(wildcard) to find the .mk file is going to
>> add quite some performance overhead.
>
> Yes. This part is more problematic, and I don't have a good solution
> for it.
We can require that packages in subdirs have a corresponding option for
the directory, cfr. BR2_PACKAGE_XORG7 (which would mean that the
directory has to be renamed). In some cases this could be a blind option
that is enabled any time that one of its packages is enabled (e.g. for
the freescale-imx stuff), in some cases there already happens to be a
package with that name (e.g. gstreamer). gstreamer doesn't even have to
be modified for it to work. It does mean that a few more .mk files will
be included than strictly necessary, but I don't think that's a worry.
For xorg7, we could still apply the same pattern in package/xorg7/xorg7.mk
Bottom line: I'd like to see this series!
Regards,
Arnout
>
> Thomas
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function
2014-03-23 17:43 ` [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function Thomas Petazzoni
@ 2014-03-25 21:47 ` Arnout Vandecappelle
0 siblings, 0 replies; 9+ messages in thread
From: Arnout Vandecappelle @ 2014-03-25 21:47 UTC (permalink / raw)
To: buildroot
On 23/03/14 18:43, Thomas Petazzoni wrote:
> Until now, our UPPERCASE function was implemented purely in make for
> performance reasons, but our LOWERCASE function was implemented by
> calling "tr", which was reasonable due to the fact that LOWERCASE was
> rarely used.
>
> An upcoming change is going to make LOWERCASE used quite a bit more,
> so we turn it into a pure make function.
>
> Moreover, we want this LOWERCASE function to turn a "_" into a "-" and
> not a ".", so we slightly adjust the existing FROM and TO lists to
> make this possible. This doesn't change the behavior of the UPPERCASE
> macro because both "-" and "." are converted into "_" by this
> function.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> package/pkg-utils.mk | 15 +++++++--------
> 1 file changed, 7 insertions(+), 8 deletions(-)
>
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 91a1981..8c3e534 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -13,7 +13,7 @@
> # as this macro is used a lot it matters
> # This works by creating translation character pairs (E.G. a:A b:B)
> # and then looping though all of them running $(subst from,to,text)
> -[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z . -
> +[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
> [TO] := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
>
> UPPERCASE = $(strip $(eval __tmp := $1) \
> @@ -23,13 +23,12 @@ UPPERCASE = $(strip $(eval __tmp := $1) \
> $(__tmp)))) \
> $(__tmp))
>
> -# LOWERCASE macro -- transforms its arguments to lowercase
> -# The above non-tr implementation is not needed, because LOWERCASE is not
> -# called very often
> -
> -define LOWERCASE
> -$(shell echo $1 | tr '[:upper:]' '[:lower:]')
> -endef
> +LOWERCASE = $(strip $(eval __tmp := $1) \
> + $(foreach c, $(join $(addsuffix :,$([TO])),$([FROM])), \
> + $(eval __tmp := \
> + $(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),\
> + $(__tmp)))) \
> + $(__tmp))
I get a further 30-40% speedup with the following definition:
define caseconvert-helper
$(1) = $$(strip \
$$(eval __tmp := $$(1))\
$(foreach c, $(2),\
$$(eval __tmp := $$(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),$$(__tmp))))\
$$(__tmp))
endef
$(eval $(call caseconvert-helper,UPPERCASE,$(join $(addsuffix :,$([FROM])),$([TO]))))
$(eval $(call caseconvert-helper,LOWERCASE,$(join $(addsuffix :,$([TO])),$([FROM]))))
(this avoids re-evaluating the foreach in every call).
Regards,
Arnout
>
> #
> # Manipulation of .config files based on the Kconfig
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Buildroot] [RFC 3/3] package: add a few needed host options for a few packages
2014-03-23 17:43 ` [Buildroot] [RFC 3/3] package: add a few needed host options for a few packages Thomas Petazzoni
@ 2014-03-25 22:06 ` Arnout Vandecappelle
0 siblings, 0 replies; 9+ messages in thread
From: Arnout Vandecappelle @ 2014-03-25 22:06 UTC (permalink / raw)
To: buildroot
On 23/03/14 18:43, Thomas Petazzoni wrote:
> This shows a few examples of what is needed to get the host package
> dependencies correct.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> package/expat/Config.in | 2 ++
> package/expat/Config.in.host | 4 ++++
> package/pkgconf/Config.in | 2 ++
> package/pkgconf/Config.in.host | 2 ++
> package/python/Config.in | 3 +++
> package/python/Config.in.host | 3 +++
> 6 files changed, 16 insertions(+)
> create mode 100644 package/expat/Config.in.host
> create mode 100644 package/pkgconf/Config.in.host
> create mode 100644 package/python/Config.in.host
>
> diff --git a/package/expat/Config.in b/package/expat/Config.in
> index 758fb7d..5b4bd39 100644
> --- a/package/expat/Config.in
> +++ b/package/expat/Config.in
> @@ -1,3 +1,5 @@
> +source "package/expat/Config.in.host"
I think this should go into package/Config.in.host like the rest.
Regards,
Arnout
> +
> config BR2_PACKAGE_EXPAT
> bool "expat"
> help
> diff --git a/package/expat/Config.in.host b/package/expat/Config.in.host
> new file mode 100644
> index 0000000..9d0b0af
> --- /dev/null
> +++ b/package/expat/Config.in.host
> @@ -0,0 +1,4 @@
> +config BR2_PACKAGE_HOST_EXPAT
> + bool
> + select BR2_PACKAGE_HOST_PKGCONF
> +
> diff --git a/package/pkgconf/Config.in b/package/pkgconf/Config.in
> index f95847f..56c1c48 100644
> --- a/package/pkgconf/Config.in
> +++ b/package/pkgconf/Config.in
> @@ -1,3 +1,5 @@
> +source "package/pkgconf/Config.in.host"
> +
> config BR2_PACKAGE_PKGCONF
> bool "pkgconf"
> help
> diff --git a/package/pkgconf/Config.in.host b/package/pkgconf/Config.in.host
> new file mode 100644
> index 0000000..e324360
> --- /dev/null
> +++ b/package/pkgconf/Config.in.host
> @@ -0,0 +1,2 @@
> +config BR2_PACKAGE_HOST_PKGCONF
> + bool
> diff --git a/package/python/Config.in b/package/python/Config.in
> index a0c78c6..253f36b 100644
> --- a/package/python/Config.in
> +++ b/package/python/Config.in
> @@ -1,9 +1,12 @@
> +source "package/python/Config.in.host"
> +
> config BR2_PACKAGE_PYTHON
> bool "python"
> depends on BR2_USE_WCHAR
> # uses fork()
> depends on BR2_USE_MMU
> select BR2_PACKAGE_LIBFFI
> + select BR2_PACKAGE_HOST_PYTHON
> help
> The python language interpreter.
>
> diff --git a/package/python/Config.in.host b/package/python/Config.in.host
> new file mode 100644
> index 0000000..2eca521
> --- /dev/null
> +++ b/package/python/Config.in.host
> @@ -0,0 +1,3 @@
> +config BR2_PACKAGE_HOST_PYTHON
> + bool
> + select BR2_PACKAGE_HOST_EXPAT
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-03-25 22:06 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-03-23 17:42 [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 1/3] pkg-utils: introduce a make-based LOWERCASE function Thomas Petazzoni
2014-03-25 21:47 ` Arnout Vandecappelle
2014-03-23 17:43 ` [Buildroot] [RFC 2/3] Makefile: include only the package .mk files that are needed Thomas Petazzoni
2014-03-23 17:43 ` [Buildroot] [RFC 3/3] package: add a few needed host options for a few packages Thomas Petazzoni
2014-03-25 22:06 ` Arnout Vandecappelle
2014-03-23 20:38 ` [Buildroot] [RFC 0/3] Speeding up the Makefile parsing Peter Korsgaard
2014-03-23 20:57 ` Thomas Petazzoni
2014-03-25 21:03 ` Arnout Vandecappelle
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox