[Buildroot] [Bug 4922] New: Please consider introducing host-make for building software packages within the buildroot

[Buildroot] [Bug 4922] New: Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

           Summary: Please consider introducing host-make for building
                    software packages within the buildroot
           Product: buildroot
           Version: 2012.02
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: Other
        AssignedTo: unassigned at buildroot.uclibc.org
        ReportedBy: develop at kristov.de
                CC: buildroot at uclibc.org
   Estimated Hours: 0.0


Created attachment 4136
  --> https://bugs.busybox.net/attachment.cgi?id=4136
changes the core build system to correctly include host-make into the build
process

Currently, the host's 'make' utility is used to build the software packages in
the buildroot. However, it would be more flexible if -- similar to
host-autoconf, host-automake etc. -- at the very beginning a buildroot-specific
host-make would be built (using the host's 'make') which is later used for
building all the rest.

Why would this be useful? For one, it would allow to be independent of the
host's 'make' and, consequently, guarantee specific 'make' behaviour.
Additionally, one would be able to patch 'make'. In my case, I wanted to patch
make 3.82 such that $(wildcard ...) results are sorted as in previous versions,
as this is the only way to guarantee that the same binaries are produced each
time the software packages are built. (Otherwise, the *.o files may be passed
in different orders to the linker as, due to -j2, the object files may be
created differently on each build, causing slightly different binaries due to
symbols being reordered.)

I have attached a simple patch that allows this. I'm afraid the patch is not
quite complete as it does not change the 'make' version check in
support/dependencies/dependencies.sh which can probably be weakened as the
host's 'make' is basically only used for building host-make and some
configuration programs. But it should definitely provide a good start. With
this patch, I'm able to successfully using a patched 'make' in the current
buildroot (2012.02).

Kind regards,

Christoph Schulz

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

--- Comment #1 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> 2012-03-15 07:07:31 UTC ---
I am not sure I would like the fact that Buildroot uses a modified make,
because this means that we would have to maintain and update this make
modification in the future.

If this problem in make is annoying for your use case (and I do understand it),
wouldn't it be better to report it in the upstream project, provide a patch
(maybe adding a new make option that restores the previous behaviour, or
something), so that future versions of make will work properly, or at least
offer the option of working properly.

The problem is really two-fold here :

 * In general, I don't like building more and more host tools unnecessarily. I
want to retain the fact that when Buildroot is told to build a Busybox-based
system, it builds Busybox and *nothing* else.

 * In particular, I don't think it's a good idea for us to carry patches on the
different components that actually change their behaviour or add features. We
don't want to be maintaining those patches. These changes need to be upstream.
the only patches that we accept to carry are build fixes for cross-compilation,
or bug fixes when those are already on their way to be integrated upstream.

Of course, this is just my opinion, the opinion of others in the Buildroot
community may be different, and I am not the Buildroot maintainer, so my voice
is really one amongst many others.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

--- Comment #2 from Christoph Schulz <develop@kristov.de> 2012-03-15 09:33:19 UTC ---
I do not want to encourage using a patched make. I only want to let the user
this possibility. Following your first argument, this implies for me that we
should also ban host-autoconf, host-automake etc. from the buildroot because
they need not be specially configured for cross-compiling and, consequently,
one could easily require the host environment to provide them (as it is
currently done for 'make'). So I really fail to see how 'make' does not fit
into the rest of the 'host-' packages.

By the way, autoconf e.g. does get patched in the buildroot.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

Christoph Schulz <develop@kristov.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #4136|0                           |1
        is obsolete|                            |

--- Comment #3 from Christoph Schulz <develop@kristov.de> 2012-03-16 10:04:24 UTC ---
Created attachment 4154
  --> https://bugs.busybox.net/attachment.cgi?id=4154
changes the core build system to correctly include host-make into the build
process

This patch corrects a bug where making a noconfig_target failed due to missing
MAKEORIG: use always original $(MAKE) in main Makefile which is overridden only
for buildroot packages to be built.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

Christoph Schulz <develop@kristov.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #4154|0                           |1
        is obsolete|                            |

--- Comment #4 from Christoph Schulz <develop@kristov.de> 2012-03-23 20:59:25 UTC ---
Created attachment 4190
  --> https://bugs.busybox.net/attachment.cgi?id=4190
changes the core build system to correctly include host-make into the build
process

The updated version of the patch now also works on a freshly unpacked archive;
it uses $(MAKEORIG) instead of $(MAKE) when building the targets
'silentoldconfig' and 'external-deps'.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

--- Comment #5 from Peter Korsgaard <jacmet@uclibc.org> 2012-03-23 22:26:58 UTC ---
(In reply to comment #2)
> I do not want to encourage using a patched make. I only want to let the user
> this possibility. Following your first argument, this implies for me that we
> should also ban host-autoconf, host-automake etc. from the buildroot because
> they need not be specially configured for cross-compiling and, consequently,
> one could easily require the host environment to provide them (as it is
> currently done for 'make'). So I really fail to see how 'make' does not fit
> into the rest of the 'host-' packages.

The choice between building our own or using whatever is present on the host
depends on several factors. Among those are:

- How common is the package? How likely is the user to already have it present?
- Do we have strict version requirements?
- How many dependencies does it have? How long does it take to build (and how
much space does it need)?

I agree with Thomas that I don't really think the scale for make is tipping
towards a host package.


> By the way, autoconf e.g. does get patched in the buildroot.

For bugfixes, yes - Not for features.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

--- Comment #6 from Christoph Schulz <develop@kristov.de> 2012-03-24 09:47:18 UTC ---
Hello!

(In reply to comment #5)
> The choice between building our own or using whatever is present on the host
> depends on several factors. Among those are:
> 
> - How common is the package? How likely is the user to already have it
> present?
> - Do we have strict version requirements?

I think so:

# Check for minimal make version (note: this check will break at make 10.x)
MIN_MAKE_VERSION=3.81
ifneq ($(firstword $(sort $(MAKE_VERSION)
$(MIN_MAKE_VERSION))),$(MIN_MAKE_VERSION))
$(error You have make '$(MAKE_VERSION)' installed. GNU make >=
$(MIN_MAKE_VERSION) is required)
endif

So at least make 3.81 is required. Remember that, although e.g. make 3.80 has
been released 2002, it took years for the various Linux distributions to pick
it up (e.g. Gentoo put it into the stable tree in 2005).

> - How many dependencies does it have? How long does it take to build (and how
> much space does it need)?

It does not have any dependencies (with the exception of autotools). It does
not take much time to build. It does not need much space (the archive is
smaller as that of e.g. autoconf-2.65).

>  [...]
> > By the way, autoconf e.g. does get patched in the buildroot.
> 
> For bugfixes, yes - Not for features.

Fair enough. But in my eyes, not being able to build reproducable binaries _is_
a bug. Otherwise I would not have started the whole discussion ;-) I don't
think you will be able to stick to make 3.81 forever...

Please consider my last argument: According to
support/dependencies/dependencies.sh, GNU make is apparently the _only_ package
needed for building for which there exist strict version requirements (the
other one is gcc >= 2.95, but gcc 2.95 has been released in 1999, much earlier
than make 3.81). So why not build a host-make which definitely meets our
requirements?


Regards,

Christoph Schulz

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

Christoph Schulz <develop@kristov.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |5006

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 4922] Please consider introducing host-make for building software packages within the buildroot

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=4922

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX

--- Comment #7 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> 2013-05-26 15:16:03 UTC ---
Making Buildroot build make is not going to solve the problem entirely:
Buildroot itself would still be using the 'make' installed by the distribution,
which may be problematic if the 'make' installed by the distribution doesn't
work properly (for one reason or another).

Conclusion: since make is such a fundamental tool for Buildroot, if something
is broken/inappropriate in make, we need a correct make to be installed prior
to running Buildroot. We are not going to build a new 'make', from source, at
every Buildroot invocation.

Please get in touch with make upstream, and provide a patch that fixes the
behavior of $(wildcard ...) according to your needs.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.