[Buildroot] [Question] Including only needed makefiles?

[Buildroot] [Question] Including only needed makefiles?

[Masahiro Yamada]

Hi expert.

I am not an active developer of Buildroot.
Apology if I am asking a silly question.


It takes a few seconds for Buildroot to start building things
after executing "make".

I think it is because Buildroot parses all the makefiles
included by the following wildcard pattern:

include $(sort $(wildcard package/*/*.mk))


Some of them are really needed to build desired packages,
but most of them are not necessary at all.



I found commit 863036378b257d9a9eb9908322edaf29f2121ae7
("package/c-libraries: need linux-headers").

This commit added the dependency to Kconfig
rather than leaving it to Makefile.


I have two questions.


[1]

Are we going towards inter-package dependency in Kconfig
rather than in Makefiles?



[2]
If we achieve the perfect dependency in Kconfig,
perhaps can we include makefiles that are really
enabled by Kconfig?   For example, like this?


pkg-$(BR2_PACKAGE_BUSYBOX) += busybox
pkg-$(BR2_PACKAGE_LAME)    += lame
pkg-$(BR2_PACKAGE_GZIP)    += gzip
   ...


include $(addprefix packages, $(addsuffix Makefike, $(pkg-y)))




I understand this is too big change,
and I am not sure if it will work well or not.

But, including only needed makefiles would improve
the response of "make" anyway.


-- 
Best Regards
Masahiro Yamada

[Buildroot] [Question] Including only needed makefiles?

[Thomas Petazzoni]

Hello Yamada-san,

On Wed, 17 Feb 2016 13:00:26 +0900, Masahiro Yamada wrote:

> I am not an active developer of Buildroot.
> Apology if I am asking a silly question.

Your question is not silly at all!

> It takes a few seconds for Buildroot to start building things
> after executing "make".
> 
> I think it is because Buildroot parses all the makefiles
> included by the following wildcard pattern:
> 
> include $(sort $(wildcard package/*/*.mk))
> 
> Some of them are really needed to build desired packages,
> but most of them are not necessary at all.

Correct.

> I found commit 863036378b257d9a9eb9908322edaf29f2121ae7
> ("package/c-libraries: need linux-headers").
> 
> This commit added the dependency to Kconfig
> rather than leaving it to Makefile.
> 
> 
> I have two questions.
> 
> 
> [1]
> 
> Are we going towards inter-package dependency in Kconfig
> rather than in Makefiles?

In Buildroot, we have a duplication of the expression of the
dependencies:

 * Dependencies are expressed at the kconfig level in Config.in files,
   by means of "select" and "depends on" statements, to ensure the
   consistency of the configuration at the kconfig level. For example,
   if one package needs bzip2, we want this package to "select"
   BR2_PACKAGE_BZIP2, so that all other packages can see that
   BR2_PACKAGE_BZIP2 is enabled and potentially use this support.
   However, dependencies at the kconfig level do *not* enforce the
   build ordering.

 * Dependencies are also expressed at the makefile level in .mk file,
   by means of the <pkg>_DEPENDENCIES variable. These dependencies are
   here to enforce build ordering (i.e be sure that the dependencies
   have been built before the package that needs them).

Due to this duplication of dependencies, we added in commit
28f67899e54f15c6e48484ea2976c3e431756bcb (and improved in
3e1b33a5349b21197da88405f32d3fc57a385421) makes sure that a package
cannot be built if its Config.in option is not enabled.

Indeed, we had lots of packages that had a given dependency in their
<pkg>_DEPENDENCIES variable, but this dependency was not accounted for
in the package Config.in file. Which is why we added this check.

Due to this check, some errors popped up, and especially the fact that
the linux-headers package did not have any Config.in option, so when
our check was verifying that its Config.in option was enabled, it was
detecting a failure. So we added a Config.in option to the
linux-headers package, to make it a regular "target" package in
Buildroot.

> [2]
> If we achieve the perfect dependency in Kconfig,
> perhaps can we include makefiles that are really
> enabled by Kconfig?   For example, like this?
> 
> 
> pkg-$(BR2_PACKAGE_BUSYBOX) += busybox
> pkg-$(BR2_PACKAGE_LAME)    += lame
> pkg-$(BR2_PACKAGE_GZIP)    += gzip
>    ...
> 
> 
> include $(addprefix packages, $(addsuffix Makefike, $(pkg-y)))

I propose such a change two years ago, see my RFC patch series at:

   http://lists.busybox.net/pipermail/buildroot/2014-March/092548.html

The main complexity is that while target packages all have a Config.in
options, it is not the case for host packages. And therefore, changing
to a per package makefile inclusion requires adding lots and lots of
Config.in options for host packages.

Another issue (explained in the cover letter) was that we could no
longer use sub-directories in packages, such as
package/qt5/qt5base/qt5base.mk.

If you look at the feedback of the patch series, it was generally quite
positive, but this needs more work, and is only saving a few seconds of
make parsing, which are not such a big deal. So the effort is fairly
big for limited savings.

I would personally rank implementing top-level parallel build at a much
higher priority level than this micro-optimization.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [Question] Including only needed makefiles?

[Yann E. MORIN]

Yamada-san, Thomas, All,

On 2016-02-17 09:11 +0100, Thomas Petazzoni spake thusly:
> On Wed, 17 Feb 2016 13:00:26 +0900, Masahiro Yamada wrote:
> > I am not an active developer of Buildroot.
> > Apology if I am asking a silly question.
> Your question is not silly at all!

Indeed. No question is stupid. You're welcome to ask.

[--SNIP--]
> > [2]
> > If we achieve the perfect dependency in Kconfig,
> > perhaps can we include makefiles that are really
> > enabled by Kconfig?   For example, like this?
> > 
> > pkg-$(BR2_PACKAGE_BUSYBOX) += busybox
> > pkg-$(BR2_PACKAGE_LAME)    += lame
> > pkg-$(BR2_PACKAGE_GZIP)    += gzip
> >    ...
> > 
> > include $(addprefix packages, $(addsuffix Makefike, $(pkg-y)))
> 
> I propose such a change two years ago, see my RFC patch series at:
> 
>    http://lists.busybox.net/pipermail/buildroot/2014-March/092548.html
> 
> The main complexity is that while target packages all have a Config.in
> options, it is not the case for host packages. And therefore, changing
> to a per package makefile inclusion requires adding lots and lots of
> Config.in options for host packages.
> 
> Another issue (explained in the cover letter) was that we could no
> longer use sub-directories in packages, such as
> package/qt5/qt5base/qt5base.mk.

Well, except with Yamada's proposal (or something similar), we could
explicitly manage the sub-directories, like so:

    pkg-$(BR2_PACKAGE_FOO) += foo
    pkg-$(BR2_PACKAGE_BAR) += sub-dir/bar

    -include $(addprefix packages/,$(foreach p,$(pkg),$(p)/$(notdir $(p)).mk))

Of course, that means explicitly listing all packages, instead of the
current glob-include. But sinc ewe're already explicitly source-ing
Config.in files, this would make it symetric for the .mk files as well.

> If you look at the feedback of the patch series, it was generally quite
> positive, but this needs more work, and is only saving a few seconds of
> make parsing, which are not such a big deal. So the effort is fairly
> big for limited savings.

Indeed, I don't believe the few startup seconds are that problematic.
Yes, they are annoying.

It takes roughly 3s on my machine (dual core-i5 with SSD), but I can see
that it could take quite a bit more on leww powerful machines,
especially those with rotating rust...

> I would personally rank implementing top-level parallel build at a much
> higher priority level than this micro-optimization.

Agreed.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [Question] Including only needed makefiles?

[Arnout Vandecappelle]

On 17-02-16 10:59, Yann E. MORIN wrote:
> Yamada-san, Thomas, All,
> 
> On 2016-02-17 09:11 +0100, Thomas Petazzoni spake thusly:
>> On Wed, 17 Feb 2016 13:00:26 +0900, Masahiro Yamada wrote:
>>> I am not an active developer of Buildroot.
>>> Apology if I am asking a silly question.
>> Your question is not silly at all!
> 
> Indeed. No question is stupid. You're welcome to ask.
> 
> [--SNIP--]
>>> [2]
>>> If we achieve the perfect dependency in Kconfig,
>>> perhaps can we include makefiles that are really
>>> enabled by Kconfig?   For example, like this?
>>>
>>> pkg-$(BR2_PACKAGE_BUSYBOX) += busybox
>>> pkg-$(BR2_PACKAGE_LAME)    += lame
>>> pkg-$(BR2_PACKAGE_GZIP)    += gzip
>>>    ...
>>>
>>> include $(addprefix packages, $(addsuffix Makefike, $(pkg-y)))
>>
>> I propose such a change two years ago, see my RFC patch series at:
>>
>>    http://lists.busybox.net/pipermail/buildroot/2014-March/092548.html
>>
>> The main complexity is that while target packages all have a Config.in
>> options, it is not the case for host packages. And therefore, changing
>> to a per package makefile inclusion requires adding lots and lots of
>> Config.in options for host packages.
>>
>> Another issue (explained in the cover letter) was that we could no
>> longer use sub-directories in packages, such as
>> package/qt5/qt5base/qt5base.mk.
> 
> Well, except with Yamada's proposal (or something similar), we could
> explicitly manage the sub-directories, like so:
> 
>     pkg-$(BR2_PACKAGE_FOO) += foo
>     pkg-$(BR2_PACKAGE_BAR) += sub-dir/bar

 One big problem with this: what to do with host packages?

> 
>     -include $(addprefix packages/,$(foreach p,$(pkg),$(p)/$(notdir $(p)).mk))
> 
> Of course, that means explicitly listing all packages, instead of the
> current glob-include. But sinc ewe're already explicitly source-ing
> Config.in files, this would make it symetric for the .mk files as well.

 I have the feeling this is going to be an added maintainance burden. Or at
least, a burden for adding packages. But it's not really that big a deal.

 We'd have to think about what to do with the subdirectories though.

> 
>> If you look at the feedback of the patch series, it was generally quite
>> positive, but this needs more work, and is only saving a few seconds of
>> make parsing, which are not such a big deal. So the effort is fairly
>> big for limited savings.
> 
> Indeed, I don't believe the few startup seconds are that problematic.
> Yes, they are annoying.

 They're horribly annoying. I'm an avid user of bash-completion and my tab key
remains pressed virtually continuously. So I'm all the time going 'make
busy<TAB>' Argh!


 Regards,
 Arnout

> 
> It takes roughly 3s on my machine (dual core-i5 with SSD), but I can see
> that it could take quite a bit more on leww powerful machines,
> especially those with rotating rust...
> 
>> I would personally rank implementing top-level parallel build at a much
>> higher priority level than this micro-optimization.
> 
> Agreed.
> 
> Regards,
> Yann E. MORIN.
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF