[Buildroot] Makefile.autotools.in stamp files breaks multiple projects

[Buildroot] Makefile.autotools.in stamp files breaks multiple projects

[Hamish Moffatt]

As previously discussed, Makefile.autotools.in creates stamp files in
$(BUILD_DIR)/$(PKG) such as .stamp_installed. This will prevent the
package being properly installed into multiple projects built out of the
same source tree. (The first project will build ok but subsequent
projects won't install the files.)

I see a few solutions;

1. Each autotools-using package could (optionally?) specify the name of
a test target file to use, instead of using an explicit stamp file.
Hans-Christian sent a patch for this already.

Obseleting (some or all of) the stamp files wouldn't be bad anyway. Many
of them could be replaced with actual targets, eg Makefile depends on
running configure (rather than using .stamp_configured).

2. The .stamp_install files could be put somewhere under $(TARGET_DIR)
instead of $(BUILD_DIR); they'd need to be stripped out by the various
generators in target/.. though.

3. Sharing $(BUILD_DIR) between multiple projects should be scrapped. In
practice, redefining BUILD_DIR = $(PROJECT_BUILD_DIR) would do it.


I think sharing $(BUILD_DIR) between multiple projects is quite broken.
The only benefit is that the packages only need to be compiled once. As
a negative though, we've got
- packages not being installed properly due to stamp files (the problem
  above).
- different projects might be using different compilers or different
  compiler flags, and the right ones won't be used for each project.
- same for different kernel headers potentially.
- We've already got exceptions for packages which can be customised,
  like busybox and the kernel.


Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

[Buildroot] Makefile.autotools.in stamp files breaks multipleprojects

[Ulf Samuelsson]

> As previously discussed, Makefile.autotools.in creates stamp files in
> $(BUILD_DIR)/$(PKG) such as .stamp_installed. This will prevent the
> package being properly installed into multiple projects built out of the
> same source tree. (The first project will build ok but subsequent
> projects won't install the files.)
>
> I see a few solutions;
>
> 1. Each autotools-using package could (optionally?) specify the name of
> a test target file to use, instead of using an explicit stamp file.
> Hans-Christian sent a patch for this already.
>

That seems to be a good solution, except that if the build breaks after
that file has been installed, other neccessary files might not be copied.
You have to ensure that this is the last file that gets installed by the 
package build.
Even if this happens, then it could break if the package version gets 
updated
and the install order changes.

> Obseleting (some or all of) the stamp files wouldn't be bad anyway. Many
> of them could be replaced with actual targets, eg Makefile depends on
> running configure (rather than using .stamp_configured).
>
> 2. The .stamp_install files could be put somewhere under $(TARGET_DIR)
> instead of $(BUILD_DIR); they'd need to be stripped out by the various
> generators in target/.. though.

You can install the stamps in "project_build_<arch>/<project>/.stamps"
as long as the make clean stuff cleans that up as well.

Populating the target file system with stamps is not a good idea.

> 3. Sharing $(BUILD_DIR) between multiple projects should be scrapped. In
> practice, redefining BUILD_DIR = $(PROJECT_BUILD_DIR) would do it.

Packages which can be configured by buildroot should remain in $(BUILD_DIR)
while packages that can be configured should be in $(PROJECT_BUILD_DIR).
Until this happens you should not have projects with different options or
use a separate buildroot tree.

You can have projects which have totally different set of packages.

>
> I think sharing $(BUILD_DIR) between multiple projects is quite broken.
> The only benefit is that the packages only need to be compiled once.

Which is a significant time saving, if you have to build multiple projects.

> As a negative though, we've got packages not being installed
> properly due to stamp files (the problem above).

That is a bug which was introduced by the Makefile.autotools.in, so this 
file needs to be fixed.

> - different projects might be using different compilers or different
>  compiler flags, and the right ones won't be used for each project.
> - same for different kernel headers potentially.

If you want different options for the toolchain, you need to use a separate 
buildroot tree.

> - We've already got exceptions for packages which can be customised,
>  like busybox and the kernel.
>
>
> Hamish
> -- 


Best Regards
Ulf Samuelsson                ulf at atmel.com
Atmel Nordic AB
Mail:  Box 2033, 174 02 Sundbyberg, Sweden
Visit:  Kavalleriv?gen 24, 174 58 Sundbyberg, Sweden
Phone +46 (8) 441 54 22     Fax +46 (8) 441 54 29
GSM    +46 (706) 22 44 57


Technical support when I am not available:
AT90 AVR Applications Group: mailto:avr at atmel.com
AT91 ARM Applications Group: mailto:at91support at atmel.com
AVR32 Applications Group        mailto:avr32 at atmel.com
http://www.avrfreaks.net/;            http://avr32linux.org/
http://www.at91.com/ ;                 http://www.linux4sam.org/ 

[Buildroot] Makefile.autotools.in stamp files breaks multipleprojects

[Hamish Moffatt]

On Thu, Sep 04, 2008 at 06:02:27AM +0200, Ulf Samuelsson wrote:
>> As previously discussed, Makefile.autotools.in creates stamp files in
>> $(BUILD_DIR)/$(PKG) such as .stamp_installed. This will prevent the
>> package being properly installed into multiple projects built out of the
>> same source tree. (The first project will build ok but subsequent
>> projects won't install the files.)
>>
>> I see a few solutions;
>>
>> 1. Each autotools-using package could (optionally?) specify the name of
>> a test target file to use, instead of using an explicit stamp file.
>> Hans-Christian sent a patch for this already.
>>
>
> That seems to be a good solution, except that if the build breaks after
> that file has been installed, other neccessary files might not be copied.
> You have to ensure that this is the last file that gets installed by the  
> package build.
> Even if this happens, then it could break if the package version gets  
> updated
> and the install order changes.

Yes it's a bit fragile. Most packages that don't use stamp files
probably just choose a random file, or the main binary - doing the same
for the autotools packages won't be any worse.

>> Obseleting (some or all of) the stamp files wouldn't be bad anyway. Many
>> of them could be replaced with actual targets, eg Makefile depends on
>> running configure (rather than using .stamp_configured).
>>
>> 2. The .stamp_install files could be put somewhere under $(TARGET_DIR)
>> instead of $(BUILD_DIR); they'd need to be stripped out by the various
>> generators in target/.. though.
>
> You can install the stamps in "project_build_<arch>/<project>/.stamps"
> as long as the make clean stuff cleans that up as well.
>
> Populating the target file system with stamps is not a good idea.

I'd prefer to use real files, but that will require us to update every
package that uses autotools in order to get the benefits. Using a stamp
file in project_build_$arch/$project/ somewhere is a good partial
solution though.

>> 3. Sharing $(BUILD_DIR) between multiple projects should be scrapped. In
>> practice, redefining BUILD_DIR = $(PROJECT_BUILD_DIR) would do it.
>
> Packages which can be configured by buildroot should remain in $(BUILD_DIR)
> while packages that can be configured should be in $(PROJECT_BUILD_DIR).
> Until this happens you should not have projects with different options or
> use a separate buildroot tree.
>
> You can have projects which have totally different set of packages.
>
>>
>> I think sharing $(BUILD_DIR) between multiple projects is quite broken.
>> The only benefit is that the packages only need to be compiled once.
>
> Which is a significant time saving, if you have to build multiple projects.

Negligible with modern hardware, IMHO. I've got about 6 projects built
in my working tree, all sharing the toolchain.. from scratch each of them 
would take under 10 minutes to build, and not long to do an incremental
build.

Longer if you're building X I suppose.

We'd be better off having separate build directories but trying to get
ccache to do its thing.

>> - different projects might be using different compilers or different
>>  compiler flags, and the right ones won't be used for each project.
>> - same for different kernel headers potentially.
>
> If you want different options for the toolchain, you need to use a 
> separate buildroot tree.

It'd be better if that was enforced in some way - right now you would
just get silent breakage.



Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

[Buildroot] Makefile.autotools.in stamp files breaks multiple projects

[Hamish Moffatt]

On Thu, Sep 04, 2008 at 12:50:35PM +1000, Hamish Moffatt wrote:
> 2. The .stamp_install files could be put somewhere under $(TARGET_DIR)
> instead of $(BUILD_DIR); they'd need to be stripped out by the various
> generators in target/.. though.

Here's my patch to move the target-installed and post-install-hook
stamps to $(PROJECT_BUILD_DIR)/.stamp/$(PKG)_xyz instead of
$(BUILD_DIR)/$(PKG)/.stamp_xyz, so that these actions will be repeated
for each project.

Any comments before I commit it?

The patch also fixes up the bridge-utils package, which is the only
package that overrides the uninstall target. I'm not sure how well
uninstall works with multiple projects, but this probably doesn't make
it any worse.


thanks
Hamish


Index: package/Makefile.autotools.in
===================================================================
--- package/Makefile.autotools.in	(revision 6342)
+++ package/Makefile.autotools.in	(working copy)
@@ -234,15 +234,16 @@
 	touch $@
 
 # Install to target dir
-$(BUILD_DIR)/%/.stamp_target_installed:
+$(PROJECT_BUILD_DIR)/.stamp/%_target_installed:
 	$(call MESSAGE,"Installing to target")
-	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_INSTALL_TARGET_OPT) -C $(@D)/$($(PKG)_SUBDIR)
+	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_INSTALL_TARGET_OPT) -C $($(PKG)_DIR)/$($(PKG)_SUBDIR)
 	$(if $(BR2_HAVE_MANPAGES),,for d in man share/man; do \
 		rm -rf $(TARGET_DIR)/$$d $(TARGET_DIR)/usr/$$d; \
 	done)
 	$(if $(BR2_HAVE_INFOPAGES),,for d in info share/info; do \
 		rm -rf $(TARGET_DIR)/$$d $(TARGET_DIR)/usr/$$d; \
 	done)
+	$(Q)mkdir -p $(@D)
 	touch $@
 
 $(BUILD_DIR)/%/.stamp_cleaned:
@@ -255,7 +256,7 @@
 	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_UNINSTALL_STAGING_OPT) -C $(@D)/$($(PKG)_SUBDIR)
 	rm -f $(@D)/.stamp_staging_installed
 	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_UNINSTALL_TARGET_OPT) -C $(@D)/$($(PKG)_SUBDIR)
-	rm -f $(@D)/.stamp_target_installed
+	rm -f $(PROJECT_BUILD_DIR)/.stamp/$(PKG)_target_installed
 
 $(BUILD_DIR)/%/.stamp_dircleaned:
 	rm -Rf $(@D)
@@ -302,7 +303,8 @@
 
 
 # define sub-target stamps
-$(2)_TARGET_INSTALL_TARGET =	$$($(2)_DIR)/.stamp_target_installed
+# targets which affect $(TARGET_DIR) must use a unique stamp for each $(PROJECT)
+$(2)_TARGET_INSTALL_TARGET =	$(PROJECT_BUILD_DIR)/.stamp/$(1)_target_installed
 $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
 $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
 $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
@@ -317,7 +319,7 @@
 $(2)_HOOK_POST_EXTRACT =	$$($(2)_DIR)/.stamp_hook_post_extract
 $(2)_HOOK_POST_CONFIGURE =	$$($(2)_DIR)/.stamp_hook_post_configure
 $(2)_HOOK_POST_BUILD =		$$($(2)_DIR)/.stamp_hook_post_build
-$(2)_HOOK_POST_INSTALL =	$$($(2)_DIR)/.stamp_hook_post_install
+$(2)_HOOK_POST_INSTALL =	$(PROJECT_BUILD_DIR)/.stamp/$(1)_hook_post_install
 
 # human-friendly targets and target sequencing
 $(1):			$(1)-install
Index: package/bridge-utils/bridge.mk
===================================================================
--- package/bridge-utils/bridge.mk	(revision 6342)
+++ package/bridge-utils/bridge.mk	(working copy)
@@ -22,9 +22,9 @@
 	touch $@
 
 # bridge has no uninstall target
-$(BUILD_DIR)/bridge-$(BRIDGE_VERSION)/.stamp_uninstalled:
+$(BRIDGE_TARGET_UNINSTALL):
 	$(call MESSAGE,"Uninstalling")
 	rm -f $(addprefix $(TARGET_DIR)/usr/,lib/libbridge.a \
 		include/libbridge.h man/man8/brctl.8 sbin/brctl)
-	rm -f $(@D)/.stamp_target_installed
+	rm -f $(BRIDGE_TARGET_INSTALL_TARGET) $(BRIDGE_HOOK_POST_INSTALL)
 

-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

[Buildroot] Makefile.autotools.in stamp files breaks multiple projects

[Hans-Christian Egtvedt]

On Fri, 5 Sep 2008 14:30:47 +1000
Hamish Moffatt <hamish@cloud.net.au> wrote:

> On Thu, Sep 04, 2008 at 12:50:35PM +1000, Hamish Moffatt wrote:
> > 2. The .stamp_install files could be put somewhere under
> > $(TARGET_DIR) instead of $(BUILD_DIR); they'd need to be stripped
> > out by the various generators in target/.. though.
> 
> Here's my patch to move the target-installed and post-install-hook
> stamps to $(PROJECT_BUILD_DIR)/.stamp/$(PKG)_xyz instead of
> $(BUILD_DIR)/$(PKG)/.stamp_xyz, so that these actions will be repeated
> for each project.
> 
> Any comments before I commit it?
> 

I think it sounds nice, I did not look into the patch in detail but the
solution seems good. The stamp directory could even be visible IMHO.

<snip bridge-utils fix and patch for target install fix>

-- 
Best regards,
Hans-Christian Egtvedt

[Buildroot] Makefile.autotools.in stamp files breaks multiple projects

[Hamish Moffatt]

On Mon, Sep 08, 2008 at 03:39:54PM +0200, Hans-Christian Egtvedt wrote:
> On Fri, 5 Sep 2008 14:30:47 +1000
> Hamish Moffatt <hamish@cloud.net.au> wrote:
> 
> > On Thu, Sep 04, 2008 at 12:50:35PM +1000, Hamish Moffatt wrote:
> > > 2. The .stamp_install files could be put somewhere under
> > > $(TARGET_DIR) instead of $(BUILD_DIR); they'd need to be stripped
> > > out by the various generators in target/.. though.
> > 
> > Here's my patch to move the target-installed and post-install-hook
> > stamps to $(PROJECT_BUILD_DIR)/.stamp/$(PKG)_xyz instead of
> > $(BUILD_DIR)/$(PKG)/.stamp_xyz, so that these actions will be repeated
> > for each project.
> > 
> > Any comments before I commit it?
> > 
> 
> I think it sounds nice, I did not look into the patch in detail but the
> solution seems good. The stamp directory could even be visible IMHO.

That's true, I might rename it to $(PROJECT_BUILD_DIR)/autotools-stamps.

Thanks for your feedback. I'll commit tomorrow.

Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>