[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Yann E. MORIN]

If building out-of-tree, add a Makefile wrapper that calls-out to the real
Makefile with proper args.

Avoids having to pass -C and O= every time we call make.

This is highly inspired from how the Linux kernel does it, and portions of
it have been used. We can't use exactly the same implementation as the
kernel does, because:

 - the kernel always overwrites the wrapper at each call: doing so in
   buildroot makes the kconfig stuff be rebuilt every time;

 - the script writing the wrapper has been expunged of the few lines
   that were too kernel-related: in buildroot we do not need the version
   string in the wrapper, and we do not have a patchlevel version;

 - "in-tree build" does not have the same meaning for the kernel and for
   buildroot: for the kernel, $(O) point to the $(TOPDIR), while for
   buildroot $(O) points to $(TOPDIR)/output.

For more complete explnanations, see:
  http://lists.busybox.net/pipermail/buildroot/2010-September/037815.html

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
---
 Makefile           |   18 ++++++++++++++-
 scripts/mkmakefile |   53 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index faa802b..ec425c6 100644
--- a/Makefile
+++ b/Makefile
@@ -45,6 +45,7 @@ space:=$(empty) $(empty)
 ifneq ("$(origin O)", "command line")
 O:=output
 CONFIG_DIR:=$(TOPDIR)
+NEED_WRAPPER=
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. Busybox does). As make automatically
@@ -60,6 +61,7 @@ override O:=$(O)
 CONFIG_DIR:=$(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+NEED_WRAPPER=y
 endif
 
 # $(shell find . -name *_defconfig |sed 's/.*\///')
@@ -324,7 +326,7 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: prepare dependencies dirs $(BASE_TARGETS) $(TARGETS_ALL)
 
 
-.PHONY: all world dirs clean distclean source \
+.PHONY: all world dirs clean distclean source outputmakefile \
 	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) \
 	$(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
@@ -453,7 +455,7 @@ all: menuconfig
 HOSTCFLAGS=$(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
-$(BUILD_DIR)/buildroot-config/%onf:
+$(BUILD_DIR)/buildroot-config/%onf: |outputmakefile
 	mkdir -p $(@D)/lxdialog
 	$(MAKE) CC="$(HOSTCC)" obj=$(@D) -C $(CONFIG) -f Makefile.br $(@F)
 
@@ -559,6 +561,18 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 # Cleanup and misc junk
 #
 #############################################################
+
+ifeq ($(NEED_WRAPPER),y)
+# outputmakefile generates a Makefile in the output directory, if using a
+# separate output directory. This allows convenient use of make in the
+# output directory.
+outputmakefile:
+	$(Q)$(SHELL) $(TOPDIR)/scripts/mkmakefile $(CURDIR) $(O)
+else
+outputmakefile:
+	@true
+endif
+
 clean:
 	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(STAMP_DIR) $(BUILD_DIR) $(TOOLCHAIN_DIR)
diff --git a/scripts/mkmakefile b/scripts/mkmakefile
new file mode 100644
index 0000000..39dd292
--- /dev/null
+++ b/scripts/mkmakefile
@@ -0,0 +1,53 @@
+#!/bin/sh
+# Generates a small Makefile used in the root of the output
+# directory, to allow make to be started from there.
+# The Makefile also allow for more convinient build of external modules
+
+# Usage
+# $1 - Kernel src directory
+# $2 - Output directory
+
+
+test ! -r $2/Makefile -o -O $2/Makefile || exit 0
+# Only overwrite automatically generated Makefiles
+# (so we do not overwrite buildroot Makefile)
+if test -e $2/Makefile && ! grep -E "^# Buildroot Makefile wrapper$" $2/Makefile >/dev/null 2>&1
+then
+        exit 0
+fi
+
+cat << EOF > $2/.Makefile
+# Buildroot Makefile wrapper
+# Automatically generated by $0: don't edit
+
+lastword = \$(word \$(words \$(1)),\$(1))
+makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
+
+MAKEARGS := -C $1
+MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$(patsubst %/,%,\$(makedir))
+
+MAKEFLAGS += --no-print-directory
+
+.PHONY: all \$(MAKECMDGOALS)
+
+all     := \$(filter-out all Makefile,\$(MAKECMDGOALS))
+
+all:
+	\$(MAKE) \$(MAKEARGS) \$(all)
+
+Makefile:;
+
+\$(all): all
+	@:
+
+%/: all
+	@:
+EOF
+
+if ! cmp $2/.Makefile $2/Makefile >/dev/null 2>&1; then
+    echo "  GEN    Makefile"
+    rm -f $2/Makefile
+    mv $2/.Makefile $2/Makefile
+else
+    rm -f $2/.Makefile
+fi
-- 
1.7.1

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:

 Yann> If building out-of-tree, add a Makefile wrapper that calls-out to the real
 Yann> Makefile with proper args.

 Yann> Avoids having to pass -C and O= every time we call make.

 Yann> This is highly inspired from how the Linux kernel does it, and portions of
 Yann> it have been used. We can't use exactly the same implementation as the
 Yann> kernel does, because:

 Yann>  - the kernel always overwrites the wrapper at each call: doing so in
 Yann>    buildroot makes the kconfig stuff be rebuilt every time;

Huh? We do call mkmakefile every time, but you do the extra trick about
.Makefile (why?). I guess the reason why you have the kconfig stuff
rebuilding is that you added the phony outputmake dependency to the
kconfig binaries rather than the phony menuconfig/oldconfig/.. targets.

 Yann>  - the script writing the wrapper has been expunged of the few lines
 Yann>    that were too kernel-related: in buildroot we do not need the version
 Yann>    string in the wrapper, and we do not have a patchlevel version;

 Yann> +
 Yann> +ifeq ($(NEED_WRAPPER),y)
 Yann> +# outputmakefile generates a Makefile in the output directory, if using a
 Yann> +# separate output directory. This allows convenient use of make in the
 Yann> +# output directory.
 Yann> +outputmakefile:
 Yann> +	$(Q)$(SHELL) $(TOPDIR)/scripts/mkmakefile $(CURDIR) $(O)
 Yann> +else
 Yann> +outputmakefile:
 Yann> +	@true
 Yann> +endif

A makefile target without any rules is OK, so you could get rid of the
true. We use TOPDIR everywhere else instead of CURDIR, so I would prefer
to use it here. It also makes more sense to make mkmakefile executable
and get rid of the SHELL.

 Yann> +
 Yann> +if ! cmp $2/.Makefile $2/Makefile >/dev/null 2>&1; then
 Yann> +    echo "  GEN    Makefile"
 Yann> +    rm -f $2/Makefile
 Yann> +    mv $2/.Makefile $2/Makefile
 Yann> +else
 Yann> +    rm -f $2/.Makefile
 Yann> +fi

I would prefer to stick as close as possible to the version in the
kernel sources, so I'll remove this. What is the reason for it? You
already created the Makefile, so it cannot be because of performance.

I've committed a slightly tweaked version of it, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Yann E. MORIN]

On Sunday 26 September 2010 23:48:28 Peter Korsgaard wrote:
>  Yann>  - the kernel always overwrites the wrapper at each call: doing so in
>  Yann>    buildroot makes the kconfig stuff be rebuilt every time;
> Huh? We do call mkmakefile every time, but you do the extra trick about
> .Makefile (why?). I guess the reason why you have the kconfig stuff
> rebuilding is that you added the phony outputmake dependency to the
> kconfig binaries rather than the phony menuconfig/oldconfig/.. targets.

Ah, right, most probably.

>  Yann>  - the script writing the wrapper has been expunged of the few lines
>  Yann>    that were too kernel-related: in buildroot we do not need the version
>  Yann>    string in the wrapper, and we do not have a patchlevel version;
> 
>  Yann> +
>  Yann> +ifeq ($(NEED_WRAPPER),y)
>  Yann> +# outputmakefile generates a Makefile in the output directory, if using a
>  Yann> +# separate output directory. This allows convenient use of make in the
>  Yann> +# output directory.
>  Yann> +outputmakefile:
>  Yann> +	$(Q)$(SHELL) $(TOPDIR)/scripts/mkmakefile $(CURDIR) $(O)
>  Yann> +else
>  Yann> +outputmakefile:
>  Yann> +	@true
>  Yann> +endif
> 
> A makefile target without any rules is OK, so you could get rid of the
> true.

The issue with an no-rule target is that it is considered to be always re-done.
But that's no issue now the kconfig is not re-built every time.

> We use TOPDIR everywhere else instead of CURDIR, so I would prefer 
> to use it here.

I see. I used CURDIR as this is set by make, so it is sure set. But TOPDIR
is better indeed, as it makes it homogenous with the rest of the Makefiles.

> It also makes more sense to make mkmakefile executable 
> and get rid of the SHELL.

I used $(SHELL) because I was too often bitten by scripts that lacked the
execution bit set. Using $(SHELL) is a way to not care about it.

>  Yann> +
>  Yann> +if ! cmp $2/.Makefile $2/Makefile >/dev/null 2>&1; then
>  Yann> +    echo "  GEN    Makefile"
>  Yann> +    rm -f $2/Makefile
>  Yann> +    mv $2/.Makefile $2/Makefile
>  Yann> +else
>  Yann> +    rm -f $2/.Makefile
>  Yann> +fi
> I would prefer to stick as close as possible to the version in the
> kernel sources, so I'll remove this. What is the reason for it? You
> already created the Makefile, so it cannot be because of performance.

It was in order not too touch the wrapper in case it did not change, so
the kconfig was not rebuilt. With the new dependencies you fixed, it is
indeed no longer needed, and we can always overwrite it.

> I've committed a slightly tweaked version of it, thanks.

Thank you for the fixes! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Lionel Landwerlin]

Yann,

I'm having a problem with the last buildroot on git. I'm not sure this
is related to your patches.
In a clean buildroot repository (no output directory already present),
if you do :
	$ make menuconfig 

The .config files are created in the root of the buildroot tree instead
of in the output/ directory.

I might be wrong, but I was used to the second behavior.
Could you check this isn't a problem introduced by your patches ?
Otherwise I will try figure this out later.

Thanks,

--
Lionel Landwerlin

Le dimanche 26 septembre 2010 ? 10:56 +0200, Yann E. MORIN a ?crit :
> If building out-of-tree, add a Makefile wrapper that calls-out to the real
> Makefile with proper args.
> 
> Avoids having to pass -C and O= every time we call make.
> 
> This is highly inspired from how the Linux kernel does it, and portions of
> it have been used. We can't use exactly the same implementation as the
> kernel does, because:
> 
>  - the kernel always overwrites the wrapper at each call: doing so in
>    buildroot makes the kconfig stuff be rebuilt every time;
> 
>  - the script writing the wrapper has been expunged of the few lines
>    that were too kernel-related: in buildroot we do not need the version
>    string in the wrapper, and we do not have a patchlevel version;
> 
>  - "in-tree build" does not have the same meaning for the kernel and for
>    buildroot: for the kernel, $(O) point to the $(TOPDIR), while for
>    buildroot $(O) points to $(TOPDIR)/output.
> 
> For more complete explnanations, see:
>   http://lists.busybox.net/pipermail/buildroot/2010-September/037815.html

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Peter Korsgaard]

>>>>> "Lionel" == Lionel Landwerlin <llandwerlin@gmail.com> writes:

 Lionel> Yann,
 Lionel> I'm having a problem with the last buildroot on git. I'm not sure this
 Lionel> is related to your patches.
 Lionel> In a clean buildroot repository (no output directory already present),
 Lionel> if you do :
 Lionel> 	$ make menuconfig 

 Lionel> The .config files are created in the root of the buildroot tree instead
 Lionel> of in the output/ directory.

That's how it has always been. It would be conceptually cleaner to put
it in output/, but we're keeping it in the toplevel dir for legacy
reasons (and because other projects using kconfig do that as well).

 Lionel> I might be wrong, but I was used to the second behavior.
 Lionel> Could you check this isn't a problem introduced by your patches ?
 Lionel> Otherwise I will try figure this out later.

No, we have never put it in output by default. If you want that, simply
run make O=output menuconfig.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Yann E. MORIN]

If building out-of-tree, add a Makefile wrapper that calls-out
to the real Makefile with proper args.

Avoids having to pass -C and O= every time we call make.

This is highly inspired from how the Linux kernel does it, and
portions of it have been used.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
---
 Makefile           |   18 +++++++++++++++-
 scripts/mkmakefile |   50 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index faa802b..ec425c6 100644
--- a/Makefile
+++ b/Makefile
@@ -45,6 +45,7 @@ space:=$(empty) $(empty)
 ifneq ("$(origin O)", "command line")
 O:=output
 CONFIG_DIR:=$(TOPDIR)
+NEED_WRAPPER=
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. Busybox does). As make automatically
@@ -60,6 +61,7 @@ override O:=$(O)
 CONFIG_DIR:=$(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+NEED_WRAPPER=y
 endif
 
 # $(shell find . -name *_defconfig |sed 's/.*\///')
@@ -324,7 +326,7 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: prepare dependencies dirs $(BASE_TARGETS) $(TARGETS_ALL)
 
 
-.PHONY: all world dirs clean distclean source \
+.PHONY: all world dirs clean distclean source outputmakefile \
 	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) \
 	$(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
@@ -453,7 +455,7 @@ all: menuconfig
 HOSTCFLAGS=$(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
-$(BUILD_DIR)/buildroot-config/%onf:
+$(BUILD_DIR)/buildroot-config/%onf: |outputmakefile
 	mkdir -p $(@D)/lxdialog
 	$(MAKE) CC="$(HOSTCC)" obj=$(@D) -C $(CONFIG) -f Makefile.br $(@F)
 
@@ -559,6 +561,18 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 # Cleanup and misc junk
 #
 #############################################################
+
+ifeq ($(NEED_WRAPPER),y)
+# outputmakefile generates a Makefile in the output directory, if using a
+# separate output directory. This allows convenient use of make in the
+# output directory.
+outputmakefile:
+	$(Q)$(SHELL) $(TOPDIR)/scripts/mkmakefile $(CURDIR) $(O)
+else
+outputmakefile:
+	@true
+endif
+
 clean:
 	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(STAMP_DIR) $(BUILD_DIR) $(TOOLCHAIN_DIR)
diff --git a/scripts/mkmakefile b/scripts/mkmakefile
new file mode 100644
index 0000000..b2b3555
--- /dev/null
+++ b/scripts/mkmakefile
@@ -0,0 +1,50 @@
+#!/bin/sh
+# Generates a small Makefile used in the root of the output
+# directory, to allow make to be started from there.
+# The Makefile also allow for more convinient build of external modules
+
+# Usage
+# $1 - Kernel src directory
+# $2 - Output directory
+
+
+test ! -r $2/Makefile -o -O $2/Makefile || exit 0
+# Only overwrite automatically generated Makefiles
+# (so we do not overwrite buildroot Makefile)
+if test -e $2/Makefile && ! grep -E "^# Buildroot Makefile wrapper$" $2/Makefile >/dev/null 2>&1
+then
+        exit 0
+fi
+
+cat << EOF > $2/.Makefile
+# Buildroot Makefile wrapper
+# Automatically generated by $0: don't edit
+
+lastword = \$(word \$(words \$(1)),\$(1))
+makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
+
+MAKEARGS := -C $1
+MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$(patsubst %/,%,\$(makedir))
+
+MAKEFLAGS += --no-print-directory
+
+.PHONY: all \$(MAKECMDGOALS)
+
+all     := \$(filter-out all Makefile,\$(MAKECMDGOALS))
+
+all:
+	\$(MAKE) \$(MAKEARGS) \$(all)
+
+Makefile:;
+
+\$(all) %/: all
+	@:
+EOF
+
+if ! cmp $2/.Makefile $2/Makefile >/dev/null 2>&1; then
+    echo "  GEN    Makefile"
+    rm -f $2/Makefile
+    mv $2/.Makefile $2/Makefile
+else
+    rm -f $2/.Makefile
+fi
-- 
1.7.1

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@anciens.enib.fr> writes:

 Yann> If building out-of-tree, add a Makefile wrapper that calls-out
 Yann> to the real Makefile with proper args.

 Yann> Avoids having to pass -C and O= every time we call make.

Nice, looks good.

 Yann> This is highly inspired from how the Linux kernel does it, and
 Yann> portions of it have been used.

Why only portions? Can't we use it all? That would make updating easier
in the future. Related to this, mkmakefile got updated in the kernel
back in August to support make 3.82, care to resync?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] Makefile: generate a Makefile wrapper in $(O)

[Yann E. MORIN]

Peter, All,

On Thursday 23 September 2010 23:04:49 Peter Korsgaard wrote:
>  Yann> If building out-of-tree, add a Makefile wrapper that calls-out
>  Yann> to the real Makefile with proper args.
>  Yann> Avoids having to pass -C and O= every time we call make.
>  Yann> This is highly inspired from how the Linux kernel does it, and
>  Yann> portions of it have been used.
> 
> Why only portions? Can't we use it all? That would make updating easier
> in the future.

Unfortunately, our 'out-of-tree' build in not similar to the one of the
kernel. In fact, we always do out-of-tree in buildroot. Let me explain...

Let's assume that $(TOPDIR) is the directory that holds the top-level
Makefile (of buildroot or the kernel).

In the Linux kernel, everything is output in $(O). The value of $(O) is
by default $(TOPDIR), and they call that "in-tree build". When the user
sets "O=blah" on the command line, then everything is put into that, and
they call that "out-of-tree build".

Now, in buildroot, we always output everything in $(O). The value of $(O)
is by default set to $(TOPDIR)/output, and we call that "in-tree build".
When the user sets "O=blah" on the command line, then everything is put
into that, and we call that "out-of-tree build".

See the slight difference in the "in-tree" case? That explains why I
couldn't use the same construct in the top-level Makefile.

Then, I have updated the mkmakefile script to remove any kernel-related
stuff. For example, we do not have a need for version and patchlevel,
and we do not have a 'quiet' build (where commands are replaced with
their beautified outputs, like CC foo/bar.o).

Next, in the Linux kernel, they always overwrite the wrapper Makefile
at each run. Doing so in buildroot caused the kconfig stuff to be always
rebuilt. So I added a check to only install the wrapper if its content
changed since last time it was isntalled.

I hope this clarifies the reasons why I said "portions of it". Those
"portions" would amount to 80-90% of this patch.

> Related to this, mkmakefile got updated in the kernel 
> back in August to support make 3.82, care to resync?

Will do, yes. Thanks for the review.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'