[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Bjørn Forsman]

This patch fixes the following use case:

 1. Build, hack, ...
 2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON)
    and/or the packages selected in menuconfig. If only there was a way
    to *easily* rebuild $(TARGET_DIR) so it is in sync...

With this patch, just remove $(TARGET_DIR) and Buildroot will recreate
it.

Main changes:
 * Always copy $(TARGET_SKELETON) to $(TARGET_DIR).
 * If $(TARGET_DIR) needs to be created (i.e. if the directory is
   missing), remove stamp files so all currently selected packages will
   be reinstalled.
 * If $(TARGET_DIR)/lib/modules/ is missing, try to install kernel
   modules.

Also:
 * remove code touching .fakeroot.00000 file (not needed)
 * remove code trying to remove CVS and .svn stuff from target skeleton
   (not needed)

Signed-off-by: Bj?rn Forsman <bjorn.forsman@gmail.com>
---
Changes in v2:
 * remove gcc stamp file so libgcc_s.so.1 will be copied to TARGET_DIR
 * make sure kernel modules get reinstalled

 Makefile       |   26 +++++++++++---------------
 linux/linux.mk |    6 ++++--
 2 files changed, 15 insertions(+), 17 deletions(-)

diff --git a/Makefile b/Makefile
index d954178..4d41e98 100644
--- a/Makefile
+++ b/Makefile
@@ -364,7 +364,7 @@ $(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
 dirs: $(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BR2_DEPENDS_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
-$(BASE_TARGETS): dirs $(O)/toolchainfile.cmake
+$(BASE_TARGETS): dirs $(O)/toolchainfile.cmake copy-target-skeleton
 
 $(BUILD_DIR)/buildroot-config/auto.conf: $(CONFIG_DIR)/.config
 	$(MAKE) $(EXTRAMAKEARGS) silentoldconfig
@@ -390,9 +390,10 @@ $(O)/toolchainfile.cmake:
 	" > $@
 
 .PHONY: all world dirs clean distclean source outputmakefile \
+	copy-target-skeleton \
 	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) \
-	$(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
+	$(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) \
 	$(HOST_DIR) $(BR2_DEPENDS_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
 #############################################################
@@ -416,19 +417,14 @@ ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 TARGET_SKELETON=$(BR2_ROOTFS_SKELETON_CUSTOM_PATH)
 endif
 
-$(BUILD_DIR)/.root:
-	mkdir -p $(TARGET_DIR)
-	if ! [ -d "$(TARGET_DIR)/bin" ]; then \
-		if [ -d "$(TARGET_SKELETON)" ]; then \
-			cp -fa $(TARGET_SKELETON)/* $(TARGET_DIR)/; \
-		fi; \
-		touch $(STAGING_DIR)/.fakeroot.00000; \
-	fi
-	-find $(TARGET_DIR) -type d -name CVS -print0 -o -name .svn -print0 | xargs -0 rm -rf
-	-find $(TARGET_DIR) -type f \( -name .empty -o -name '*~' \) -print0 | xargs -0 rm -rf
-	touch $@
+copy-target-skeleton: $(TARGET_DIR)
+	cp -ua $(TARGET_SKELETON)/* $(TARGET_DIR)/ && \
+	find $(TARGET_DIR) -type f \( -name .empty -o -name '*~' \) -print0 | xargs -0 rm -rf
 
-$(TARGET_DIR): $(BUILD_DIR)/.root
+$(TARGET_DIR):
+	mkdir $(TARGET_DIR)
+	rm -f $(BUILD_DIR)/*/.stamp_target_installed
+	rm -f $(STAMP_DIR)/gcc_libs_target_installed
 
 erase-fakeroots:
 	rm -f $(BUILD_DIR)/.fakeroot*
diff --git a/linux/linux.mk b/linux/linux.mk
index 9076fb9..0ad8c0f 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -147,6 +147,9 @@ $(LINUX26_DIR)/.stamp_compiled: $(LINUX26_DIR)/.stamp_configured $(LINUX26_DIR)/
 $(LINUX26_DIR)/.stamp_installed: $(LINUX26_DIR)/.stamp_compiled
 	@$(call MESSAGE,"Installing kernel")
 	cp $(LINUX26_IMAGE_PATH) $(BINARIES_DIR)
+	$(Q)touch $@
+
+$(TARGET_DIR)/lib/modules:
 	# Install modules and remove symbolic links pointing to build
 	# directories, not relevant on the target
 	@if [ $(shell grep -c "CONFIG_MODULES=y" $(LINUX26_DIR)/.config) != 0 ] ; then 	\
@@ -155,9 +158,8 @@ $(LINUX26_DIR)/.stamp_installed: $(LINUX26_DIR)/.stamp_compiled
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/build ;	\
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/source ;	\
 	fi
-	$(Q)touch $@
 
-linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(LINUX26_DIR)/.stamp_installed
+linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(LINUX26_DIR)/.stamp_installed $(TARGET_DIR)/lib/modules
 
 linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig linux26-xconfig linux26-gconfig linux26-nconfig: dirs $(LINUX26_DIR)/.stamp_configured
 	$(MAKE) $(LINUX26_MAKE_FLAGS) -C $(LINUX26_DIR) \
-- 
1.7.1

[Buildroot] [PATCH v2 2/2] linux.mk: depend on actual kernel image instead of stamp file

[Bjørn Forsman]

Use the real output file (kernel image) as target instead of a stamp
file. This way Buildroot cannot be tricked into thinking that the kernel
image is installed when it's not.

Signed-off-by: Bj?rn Forsman <bjorn.forsman@gmail.com>
---
 linux/linux.mk |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index 0ad8c0f..5855658 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -144,7 +144,7 @@ $(LINUX26_DIR)/.stamp_compiled: $(LINUX26_DIR)/.stamp_configured $(LINUX26_DIR)/
 	$(Q)touch $@
 
 # Installation
-$(LINUX26_DIR)/.stamp_installed: $(LINUX26_DIR)/.stamp_compiled
+$(BINARIES_DIR)/$(LINUX26_IMAGE_NAME): $(LINUX26_DIR)/.stamp_compiled
 	@$(call MESSAGE,"Installing kernel")
 	cp $(LINUX26_IMAGE_PATH) $(BINARIES_DIR)
 	$(Q)touch $@
@@ -153,13 +153,13 @@ $(TARGET_DIR)/lib/modules:
 	# Install modules and remove symbolic links pointing to build
 	# directories, not relevant on the target
 	@if [ $(shell grep -c "CONFIG_MODULES=y" $(LINUX26_DIR)/.config) != 0 ] ; then 	\
-		$(TARGET_MAKE_ENV) $(MAKE1) $(LINUX26_MAKE_FLAGS) -C $(@D) 		\
+		$(TARGET_MAKE_ENV) $(MAKE1) $(LINUX26_MAKE_FLAGS) -C $(LINUX26_DIR)	\
 			DEPMOD="$(HOST_DIR)/usr/sbin/depmod" modules_install ;		\
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/build ;	\
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/source ;	\
 	fi
 
-linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(LINUX26_DIR)/.stamp_installed $(TARGET_DIR)/lib/modules
+linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(BINARIES_DIR)/$(LINUX26_IMAGE_NAME) $(TARGET_DIR)/lib/modules
 
 linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig linux26-xconfig linux26-gconfig linux26-nconfig: dirs $(LINUX26_DIR)/.stamp_configured
 	$(MAKE) $(LINUX26_MAKE_FLAGS) -C $(LINUX26_DIR) \
@@ -167,7 +167,7 @@ linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig li
 
 # Support for rebuilding the kernel after the initramfs file list has
 # been generated in $(BINARIES_DIR)/rootfs.initramfs.
-$(LINUX26_DIR)/.stamp_initramfs_rebuilt: $(LINUX26_DIR)/.stamp_installed $(BINARIES_DIR)/rootfs.initramfs
+$(LINUX26_DIR)/.stamp_initramfs_rebuilt: $(BINARIES_DIR)/$(LINUX26_IMAGE_NAME) $(BINARIES_DIR)/rootfs.initramfs
 	@$(call MESSAGE,"Rebuilding kernel with initramfs")
 	# Remove the previously generated initramfs which was empty,
 	# to make sure the kernel will actually regenerate it.
-- 
1.7.1

[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Bjørn Forsman]

2011/2/18 Bj?rn Forsman <bjorn.forsman@gmail.com>:
> This patch fixes the following use case:
>
> ?1. Build, hack, ...
> ?2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON)
> ? ?and/or the packages selected in menuconfig. If only there was a way
> ? ?to *easily* rebuild $(TARGET_DIR) so it is in sync...
>
> With this patch, just remove $(TARGET_DIR) and Buildroot will recreate
> it.
>
> Main changes:
> ?* Always copy $(TARGET_SKELETON) to $(TARGET_DIR).
> ?* If $(TARGET_DIR) needs to be created (i.e. if the directory is
> ? missing), remove stamp files so all currently selected packages will
> ? be reinstalled.
> ?* If $(TARGET_DIR)/lib/modules/ is missing, try to install kernel
> ? modules.
>
> Also:
> ?* remove code touching .fakeroot.00000 file (not needed)
> ?* remove code trying to remove CVS and .svn stuff from target skeleton
> ? (not needed)
>
> Signed-off-by: Bj?rn Forsman <bjorn.forsman@gmail.com>
> ---
> Changes in v2:
> ?* remove gcc stamp file so libgcc_s.so.1 will be copied to TARGET_DIR
> ?* make sure kernel modules get reinstalled
>
> ?Makefile ? ? ? | ? 26 +++++++++++---------------
> ?linux/linux.mk | ? ?6 ++++--
> ?2 files changed, 15 insertions(+), 17 deletions(-)

Ping?

[Buildroot] [PATCH v2 2/2] linux.mk: depend on actual kernel image instead of stamp file

[Bjørn Forsman]

2011/2/18 Bj?rn Forsman <bjorn.forsman@gmail.com>:
> Use the real output file (kernel image) as target instead of a stamp
> file. This way Buildroot cannot be tricked into thinking that the kernel
> image is installed when it's not.
>
> Signed-off-by: Bj?rn Forsman <bjorn.forsman@gmail.com>
> ---
> ?linux/linux.mk | ? ?8 ++++----
> ?1 files changed, 4 insertions(+), 4 deletions(-)

Ping?

[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Bjørn Forsman]

2011/2/21 Bj?rn Forsman <bjorn.forsman@gmail.com>:
> 2011/2/18 Bj?rn Forsman <bjorn.forsman@gmail.com>:
>> This patch fixes the following use case:
>>
>> ?1. Build, hack, ...
>> ?2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON)
>> ? ?and/or the packages selected in menuconfig. If only there was a way
>> ? ?to *easily* rebuild $(TARGET_DIR) so it is in sync...
>>
>> With this patch, just remove $(TARGET_DIR) and Buildroot will recreate
>> it.
>>
>> Main changes:
>> ?* Always copy $(TARGET_SKELETON) to $(TARGET_DIR).
>> ?* If $(TARGET_DIR) needs to be created (i.e. if the directory is
>> ? missing), remove stamp files so all currently selected packages will
>> ? be reinstalled.
>> ?* If $(TARGET_DIR)/lib/modules/ is missing, try to install kernel
>> ? modules.
>>
>> Also:
>> ?* remove code touching .fakeroot.00000 file (not needed)
>> ?* remove code trying to remove CVS and .svn stuff from target skeleton
>> ? (not needed)
>>
>> Signed-off-by: Bj?rn Forsman <bjorn.forsman@gmail.com>
>> ---
>> Changes in v2:
>> ?* remove gcc stamp file so libgcc_s.so.1 will be copied to TARGET_DIR
>> ?* make sure kernel modules get reinstalled
>>
>> ?Makefile ? ? ? | ? 26 +++++++++++---------------
>> ?linux/linux.mk | ? ?6 ++++--
>> ?2 files changed, 15 insertions(+), 17 deletions(-)
>
> Ping?

Ping ping :-)

Now that the new cycle has opened, any comments on this?

I'd really like to get this functionality in mainline :-)

Best regards,
Bj?rn Forsman

[Buildroot] [PATCH v2 2/2] linux.mk: depend on actual kernel image instead of stamp file

[Bjørn Forsman]

2011/2/21 Bj?rn Forsman <bjorn.forsman@gmail.com>:
> 2011/2/18 Bj?rn Forsman <bjorn.forsman@gmail.com>:
>> Use the real output file (kernel image) as target instead of a stamp
>> file. This way Buildroot cannot be tricked into thinking that the kernel
>> image is installed when it's not.
>>
>> Signed-off-by: Bj?rn Forsman <bjorn.forsman@gmail.com>
>> ---
>> ?linux/linux.mk | ? ?8 ++++----
>> ?1 files changed, 4 insertions(+), 4 deletions(-)
>
> Ping?

Ping ping :-)

Any comments?

Best regards,
Bj?rn Forsman

[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Thomas Petazzoni]

On Tue, 1 Mar 2011 17:03:22 +0100
Bj?rn Forsman <bjorn.forsman@gmail.com> wrote:

> Ping ping :-)
> 
> Now that the new cycle has opened, any comments on this?
> 
> I'd really like to get this functionality in mainline :-)

I think the functionality is great, but I'd prefer to see it integrated
on top of the packaging work started by Lionel rather than coded as
proposed in the current patch. I am also not sure that rebuilding the
$(TARGET_DIR) without rebuilding the $(STAGING_DIR) is a good thing.

However, it's true that the packaging work started by Lionel is not
ready for inclusion yet, so maybe we should merge this as is, and then
update it once we have the proper packaging mechanism working.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v2 2/2] linux.mk: depend on actual kernel image instead of stamp file

[Thomas Petazzoni]

Hello,

On Fri, 18 Feb 2011 00:44:54 +0100
Bj?rn Forsman <bjorn.forsman@gmail.com> wrote:

> Use the real output file (kernel image) as target instead of a stamp
> file. This way Buildroot cannot be tricked into thinking that the kernel
> image is installed when it's not.

Well, we switched *all* packages from dependencies on installed files
to dependencies on stamp files when migrating them from hand-coded
makefiles to the package infrastructures, so I don't think we should do
the opposite for the Linux kernel package.

But maybe we need to provide for each package (kernel included) a clean
and nice mechanism to delete their stamp files so that they get
reinstalled at the next make invocation.

Regardless of the solution chosen, I think any change in this direction
should come with a clear update in the documentation. I don't want to
see dozens of different hacks^Wmechanisms to do partial rebuilds being
added all over the place without any coherency nor documentation.

Regards!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Bjørn Forsman]

On 1 March 2011 17:20, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> On Tue, 1 Mar 2011 17:03:22 +0100
> Bj?rn Forsman <bjorn.forsman@gmail.com> wrote:
>
>> Ping ping :-)
>>
>> Now that the new cycle has opened, any comments on this?
>>
>> I'd really like to get this functionality in mainline :-)
>
> I think the functionality is great, but I'd prefer to see it integrated
> on top of the packaging work started by Lionel rather than coded as
> proposed in the current patch.

I see.

> I am also not sure that rebuilding the
> $(TARGET_DIR) without rebuilding the $(STAGING_DIR) is a good thing.

TARGET_DIR depends on STAGING_DIR, and not the other way around, so I
don't see why STAGING_DIR should be rebuilt. Am I missing something?

> However, it's true that the packaging work started by Lionel is not
> ready for inclusion yet, so maybe we should merge this as is, and then
> update it once we have the proper packaging mechanism working.

Yes, please :-)

> Regards,
>
> Thomas
> --
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com

[Buildroot] [PATCH v2 2/2] linux.mk: depend on actual kernel image instead of stamp file

[Bjørn Forsman]

On 1 March 2011 17:23, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Fri, 18 Feb 2011 00:44:54 +0100
> Bj?rn Forsman <bjorn.forsman@gmail.com> wrote:
>
>> Use the real output file (kernel image) as target instead of a stamp
>> file. This way Buildroot cannot be tricked into thinking that the kernel
>> image is installed when it's not.
>
> Well, we switched *all* packages from dependencies on installed files
> to dependencies on stamp files when migrating them from hand-coded
> makefiles to the package infrastructures, so I don't think we should do
> the opposite for the Linux kernel package.

Yes, I know. But I think that the Linux kernel image is special. It is
one file. It is easy to track. Using stamp files for *generic
packages* makes sense because it is difficult to keep track of all the
files they may install.

Here is how Buildroot currently behave:

1. build and see that output/images/ contain rootfs.tar and zImage.
2. remove output/images/ and rebuild
3. output/images/ now only has rootfs.tar, no zImage!

That is just confusing behaviour and I want to fix it.

> But maybe we need to provide for each package (kernel included) a clean
> and nice mechanism to delete their stamp files so that they get
> reinstalled at the next make invocation.

That is possible. But wouldn't you then have to track dependencies
yourself? I'd rather "rm -rf output/target/ output/images/" and have
Buildroot rebuild these dirs automatically than issuing some "make
pkg1-clean-stamp ... pkgN-clean-stamp".

Or did I misunderstand you idea?

> Regardless of the solution chosen, I think any change in this direction
> should come with a clear update in the documentation. I don't want to
> see dozens of different hacks^Wmechanisms to do partial rebuilds being
> added all over the place without any coherency nor documentation.

I wasn't thinking that big :-)  This patch is just a small usability
fix IMO, so I don't think is warrants a doc update.

> Regards!
>
> Thomas
> --
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com

[Buildroot] [PATCH v2 2/2] linux.mk: depend on actual kernel image instead of stamp file

[Bjørn Forsman]

Hi Thomas,

I forgot to say thank you for your feedback in my previous mail. So,
thank you for your feedback :-)

Also, I have an idea for a patch that provide almost the same
functionality as this one -- without removing stamp files (in the
patch): If output/images/ needs to be created, remove all stamp files
from packages that install into output/images/.

It is basically the same method as patch 1 in this series. What do you think?

Right now I'm not sure which method I like best, but I figured I
should post it anyway.

Best regards,
Bj?rn Forsman

[Buildroot] [PATCH v2 1/2] Makefile: improve $(TARGET_DIR) (re)creation

[Bjørn Forsman]

2011/2/18 Bj?rn Forsman <bjorn.forsman@gmail.com>:
> This patch fixes the following use case:
>
> ?1. Build, hack, ...
> ?2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON)
> ? ?and/or the packages selected in menuconfig. If only there was a way
> ? ?to *easily* rebuild $(TARGET_DIR) so it is in sync...

There is a bug in this patch: Qt isn't completely reinstalled (libs
are, but not demos/examples). I will send an updated patch shortly.

Best regards,
Bj?rn Forsman