[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Quotient Remainder]

Sometimes it's necessary to fix something up in the source code of a package.
Usually this requires deleting the stamp file and runinng "make" again.
This patch set allows this sequence to be done by just issuing either a top-level
"make rebuild" to rebuild all packages or a package-specific command, e.g.
"make busybox-rebuild".  The package-specific command will normally be followed by
a "make" (*not* "make rebuild") command to sweep up and generate the RFS.

There may be some subtle harmful effects from rebuilding packages unconditionally
but I've been using this approach for some time and noticed nothing.  The original
patch was based on v2010.08 and I've made efforts to update but may have missed some
things.  It's mainly testing the water anyway as an RFC.

This feature may be useful if Buildroot is extended to being used as a day-to-day
development environment as opposed to a build-once tool.

Quotient Remainder (2):
  Allow rebuilding of generic targets.
  Add top-level rebuild target.

 Makefile                    |   10 ++++++++++
 fs/common.mk                |    2 ++
 package/Makefile.package.in |   22 ++++++++++++++++++++++
 target/generic/Makefile.in  |    6 ++++++
 4 files changed, 40 insertions(+), 0 deletions(-)

-- 
1.7.6

[Buildroot] [PATCH 1/2] Allow rebuilding of generic targets.

[Quotient Remainder]

Individual packages using the GENTARGETS and AUTOTARGETS infrastructure
can now be rebuilt by issuing "make ${pkg}-rebuild".
Changes can be made in output/build/${pkg}/* and make ${pkg}-rebuild will
run the normal build command for that directory again.

Signed-off-by: Quotient Remainder <quotientvremainder@gmail.com>
---
 package/Makefile.package.in |   22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index 74087cd..27e5721 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -290,6 +290,11 @@ $(BUILD_DIR)/%/.stamp_built::
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
+# Rebuild
+$(BUILD_DIR)/%/.stamp_rebuilt:
+	@$(call MESSAGE,"Rebuilding")
+	$($(PKG)_REBUILD_CMDS)
+
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call MESSAGE,'Installing to host directory')
@@ -425,6 +430,7 @@ $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
 $(2)_TARGET_INSTALL_IMAGES =	$$($(2)_DIR)/.stamp_images_installed
 $(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
 $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
+$(2)_TARGET_REBUILD =		$$($(2)_DIR)/.stamp_rebuilt
 $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
 $(2)_TARGET_PATCH =		$$($(2)_DIR)/.stamp_patched
 $(2)_TARGET_EXTRACT =		$$($(2)_DIR)/.stamp_extracted
@@ -485,6 +491,8 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+$(1)-rebuild:	$$($(2)_TARGET_REBUILD) $(1)
+
 $(1)-configure:		$(1)-patch \
 			$$($(2)_TARGET_CONFIGURE)
 
@@ -514,6 +522,7 @@ $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
+$$($(2)_TARGET_REBUILD):		PKG=$(2)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
 $$($(2)_TARGET_PATCH):			PKG=$(2)
 $$($(2)_TARGET_PATCH):			RAWNAME=$(patsubst host-%,%,$(1))
@@ -534,6 +543,19 @@ else
 $(2)_KCONFIG_VAR = BR2_PACKAGE_$(2)
 endif
 
+ifndef $(2)_REBUILD_CMDS
+ ifeq ($(5),target)
+  define $(2)_REBUILD_CMDS
+	$(Q)rm -f $(2)_TARGET_BUILD $(2)_TARGET_INSTALL_TARGET \
+		$(2)_TARGET_INSTALL_STAGING $(2)_TARGET_INSTALL_IMAGES
+  endef
+ else
+  define $(2)_REBUILD_CMDS
+	$(Q)rm -f $(2)_TARGET_BUILD $(2)_TARGET_INSTALL_HOST
+  endef
+ endif
+endif
+
 # add package to the general list of targets if requested by the buildroot
 # configuration
 ifeq ($$($$($(2)_KCONFIG_VAR)),y)
-- 
1.7.6

[Buildroot] [PATCH 2/2] Add top-level rebuild target.

[Quotient Remainder]

This runs the rebuild command for all enabled targets and runs the RFS
packaging commands.

Signed-off-by: Quotient Remainder <quotientvremainder@gmail.com>
---
 Makefile                   |   10 ++++++++++
 fs/common.mk               |    2 ++
 target/generic/Makefile.in |    6 ++++++
 3 files changed, 18 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 2f3802c..1c846ff 100644
--- a/Makefile
+++ b/Makefile
@@ -316,17 +316,21 @@ include target/Makefile.in
 include linux/linux.mk
 
 TARGETS+=target-finalize
+TARGETS_NOREBUILD+=target-finalize
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 TARGETS+=target-purgelocales
+TARGETS_NOREBUILD += target-purgelocales
 endif
 
 include fs/common.mk
 
 TARGETS+=erase-fakeroots
+TARGETS_NOREBUILD+=erase-fakeroots
 
 TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
+TARGETS_REBUILD:=$(patsubst %,%-rebuild,$(filter-out $(TARGETS_NOREBUILD),$(TARGETS)))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
 TARGETS_ALL:=$(patsubst %,__real_tgt_%,$(TARGETS))
 
@@ -497,6 +501,9 @@ external-deps:
 show-targets:
 	@echo $(TARGETS)
 
+show-rebuild-targets:
+	@echo $(TARGETS_REBUILD)
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 all: menuconfig
@@ -628,6 +635,8 @@ endif
 
 cross: $(BASE_TARGETS)
 
+rebuild: $(TARGETS_REBUILD) $(TARGETS_NOREBUILD)
+
 help:
 	@echo 'Cleaning:'
 	@echo '  clean                  - delete all files created by build'
@@ -635,6 +644,7 @@ help:
 	@echo
 	@echo 'Build:'
 	@echo '  all                    - make world'
+	@echo '  rebuild                - rerun build commands for all enabled targets'
 	@echo
 	@echo 'Configuration:'
 	@echo '  menuconfig             - interactive curses-based configurator'
diff --git a/fs/common.mk b/fs/common.mk
index 4d50514..594ca5e 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -71,6 +71,8 @@ rootfs-$(1)-show-depends:
 
 rootfs-$(1): $(BINARIES_DIR)/rootfs.$(1) $(ROOTFS_$(2)_POST_TARGETS)
 
+$(1)-root-rebuild: $(1)-root
+
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS += rootfs-$(1)
 endif
diff --git a/target/generic/Makefile.in b/target/generic/Makefile.in
index 4185202..8a29174 100644
--- a/target/generic/Makefile.in
+++ b/target/generic/Makefile.in
@@ -33,21 +33,27 @@ target-generic-dont-remount-rw:
 
 ifneq ($(TARGET_GENERIC_HOSTNAME),)
 TARGETS += target-generic-hostname
+TARGETS_NOREBUILD += target-generic-hostname
 endif
 
 ifneq ($(TARGET_GENERIC_ISSUE),)
 TARGETS += target-generic-issue
+TARGETS_NOREBUILD += target-generic-issue
 endif
 
 ifeq ($(BR2_ROOTFS_SKELETON_DEFAULT),y)
 ifeq ($(BR2_PACKAGE_SYSVINIT),y)
 TARGETS += target-generic-getty-sysvinit
+TARGETS_NOREBUILD += target-generic-getty-sysvinit
 else
 TARGETS += target-generic-getty-busybox
+TARGETS_NOREBUILD += target-generic-getty-busybox
 endif
 ifeq ($(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),y)
 TARGETS += target-generic-do-remount-rw
+TARGETS_NOREBUILD += target-generic-do-remount-rw
 else
 TARGETS += target-generic-dont-remount-rw
+TARGETS_NOREBUILD += target-generic-dont-remount-rw
 endif
 endif
-- 
1.7.6

[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Thomas De Schampheleire]

On Mon, Jul 18, 2011 at 11:15 AM, Quotient Remainder
<quotientvremainder@gmail.com> wrote:
> Sometimes it's necessary to fix something up in the source code of a package.
> Usually this requires deleting the stamp file and runinng "make" again.
> This patch set allows this sequence to be done by just issuing either a top-level
> "make rebuild" to rebuild all packages or a package-specific command, e.g.
> "make busybox-rebuild". ?The package-specific command will normally be followed by
> a "make" (*not* "make rebuild") command to sweep up and generate the RFS.
>
> There may be some subtle harmful effects from rebuilding packages unconditionally
> but I've been using this approach for some time and noticed nothing. ?The original
> patch was based on v2010.08 and I've made efforts to update but may have missed some
> things. ?It's mainly testing the water anyway as an RFC.
>
> This feature may be useful if Buildroot is extended to being used as a day-to-day
> development environment as opposed to a build-once tool.

We are currently using Buildroot in such a development mode, and
indeed face the problems you are solving with this patch. So: thumbs
up!

But it is not only 'rebuild' that is useful. The conditions under
which the configure command of a certain package originally ran, may
have changed. A similar thing is true for patches: you may have
updated the patchfile. In such cases, generic reconfigure and repatch
targets are very welcome, together with the rebuild target you already
made...

Thomas

[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Quotient Remainder]

Hello Thomas,

On Mon, 2011-07-18 at 11:30 +0200, Thomas De Schampheleire wrote:
> We are currently using Buildroot in such a development mode, and
> indeed face the problems you are solving with this patch. So: thumbs
> up!
> 
> But it is not only 'rebuild' that is useful. The conditions under
> which the configure command of a certain package originally ran, may
> have changed. A similar thing is true for patches: you may have
> updated the patchfile. In such cases, generic reconfigure and repatch
> targets are very welcome, together with the rebuild target you already
> made...

xxx-reconfigure should be fairly straightforward since no
TARGET_NORECONFIGURE variable would be needed, just remove the stamp and
make xxx-configure.

Would xxx-repatch be effectively a re-extraction?  That is, the patch
has been applied, then changed and so can't be applied in reverse so
you'll have to erase the extracted package and re-extract, patch, ...
So in effect it's a "make xxx-dirclean xxx" operation, right?

[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Thomas De Schampheleire]

Hi,

On Mon, Jul 18, 2011 at 4:02 PM, Quotient Remainder
<quotientvremainder@gmail.com> wrote:
> Hello Thomas,
>
> On Mon, 2011-07-18 at 11:30 +0200, Thomas De Schampheleire wrote:
>> We are currently using Buildroot in such a development mode, and
>> indeed face the problems you are solving with this patch. So: thumbs
>> up!
>>
>> But it is not only 'rebuild' that is useful. The conditions under
>> which the configure command of a certain package originally ran, may
>> have changed. A similar thing is true for patches: you may have
>> updated the patchfile. In such cases, generic reconfigure and repatch
>> targets are very welcome, together with the rebuild target you already
>> made...
>
> xxx-reconfigure should be fairly straightforward since no
> TARGET_NORECONFIGURE variable would be needed, just remove the stamp and
> make xxx-configure.
>
> Would xxx-repatch be effectively a re-extraction? ?That is, the patch
> has been applied, then changed and so can't be applied in reverse so
> you'll have to erase the extracted package and re-extract, patch, ...
> So in effect it's a "make xxx-dirclean xxx" operation, right?

Yes, you're right. Really repatching would require re-extracting.
Let's skip this target.

Thomas

[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Quotient Remainder]

On Mon, 2011-07-18 at 16:14 +0200, Thomas De Schampheleire wrote:
> > xxx-reconfigure should be fairly straightforward since no
> > TARGET_NORECONFIGURE variable would be needed, just remove the stamp and
> > make xxx-configure.
> >
> > Would xxx-repatch be effectively a re-extraction?  That is, the patch
> > has been applied, then changed and so can't be applied in reverse so
> > you'll have to erase the extracted package and re-extract, patch, ...
> > So in effect it's a "make xxx-dirclean xxx" operation, right?
> 
> Yes, you're right. Really repatching would require re-extracting.
> Let's skip this target.

What about xxx-reconfigure; should it (1) reconfigure and then build the
package to completion (including installation) or (2) just configure and
stop, requiring a separate step to complete?
There's an argument in favour of both approaches.
(1) means that the package is installed to target/staging, etc. with one
command.
(2) clearly (?!) demonstrates that a top-level "make" is required to
press the changes into an RFS/tarball image.

[Buildroot] [PATCH 1/2] Allow rebuilding of generic targets.

[Thomas Petazzoni]

Hello,

Le Mon, 18 Jul 2011 10:15:52 +0100,
Quotient Remainder <quotientvremainder@gmail.com> a ?crit :

> Individual packages using the GENTARGETS and AUTOTARGETS infrastructure
> can now be rebuilt by issuing "make ${pkg}-rebuild".
> Changes can be made in output/build/${pkg}/* and make ${pkg}-rebuild will
> run the normal build command for that directory again.
> 
> Signed-off-by: Quotient Remainder <quotientvremainder@gmail.com>

I also have a patch (below) that implements a similar feature
(<pkg>-rebuild and <pkg>-reconfigure), but it also does the "make"
again automatically as well to regenerate the root filesystem. The
implementation is a bit different, as it doesn't use a new fake stamp
file, but simply uses phony targets.

commit 6242b6c713a435c1afa4fb04655024539d0d6106
Author: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Date:   Wed May 18 23:23:44 2011 +0200

    package: add <pkg>-rebuild and <pkg>-reconfigure
    
    We are often asked "how can I restart the build of a package ?" or
    "how can I restart the build of package from the configure part
    ?". Obviously, tweaking with stamp files is possible, but not very
    user friendly.
    
    Therefore this patch adds two new per-package targets: <pkg>-rebuild
    and <pkg>-reconfigure. They will remove the right stamp files and
    restart the complete build process (by using the 'all' target, so that
    not only the package is reconfigured, recompiled and reinstalled, but
    the root filesystem images are also regenerated).
    
    Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index 1cfffab..b876d9d 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -504,6 +504,19 @@ $(1)-clean:		$(1)-uninstall \
 
 $(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
 
+$(1)-clean-for-rebuild:
+			rm -f $$($(2)_TARGET_BUILD)
+			rm -f $$($(2)_TARGET_INSTALL_STAGING)
+			rm -f $$($(2)_TARGET_INSTALL_TARGET)
+			rm -f $$($(2)_TARGET_INSTALL_HOST)
+
+$(1)-rebuild:		$(1)-clean-for-rebuild all
+
+$(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
+			rm -f $$($(2)_TARGET_CONFIGURE)
+
+$(1)-reconfigure:	$(1)-clean-for-reconfigure all
+
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)


-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC PATCH 0/2] Add the ability to rebuild packages.

[Thomas De Schampheleire]

On Mon, Jul 18, 2011 at 5:08 PM, Quotient Remainder
<quotientvremainder@gmail.com> wrote:
> On Mon, 2011-07-18 at 16:14 +0200, Thomas De Schampheleire wrote:
>> > xxx-reconfigure should be fairly straightforward since no
>> > TARGET_NORECONFIGURE variable would be needed, just remove the stamp and
>> > make xxx-configure.
>> >
>> > Would xxx-repatch be effectively a re-extraction? ?That is, the patch
>> > has been applied, then changed and so can't be applied in reverse so
>> > you'll have to erase the extracted package and re-extract, patch, ...
>> > So in effect it's a "make xxx-dirclean xxx" operation, right?
>>
>> Yes, you're right. Really repatching would require re-extracting.
>> Let's skip this target.
>
> What about xxx-reconfigure; should it (1) reconfigure and then build the
> package to completion (including installation) or (2) just configure and
> stop, requiring a separate step to complete?
> There's an argument in favour of both approaches.
> (1) means that the package is installed to target/staging, etc. with one
> command.
> (2) clearly (?!) demonstrates that a top-level "make" is required to
> press the changes into an RFS/tarball image.

I think an all-in-one approach where the reconfigure step also
continues to rebuild and reinstall, is most interesting. In most
cases, a reconfigure step will be followed by a rebuild step, so why
not do it directly?
If there would be a use case to have finer-grained control over which
steps are executed, we'd need to implement a way to specify exactly
which steps to perform, for example with targets xxx-only-reconfigure,
xxx-only-rebuild. But I'm not sure whether anyone would use these.

Thomas

[Buildroot] [PATCH 1/2] Allow rebuilding of generic targets.

[Thomas De Schampheleire]

Hi Thomas,

On Tue, Jul 19, 2011 at 7:05 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> Le Mon, 18 Jul 2011 10:15:52 +0100,
> Quotient Remainder <quotientvremainder@gmail.com> a ?crit :
>
>> Individual packages using the GENTARGETS and AUTOTARGETS infrastructure
>> can now be rebuilt by issuing "make ${pkg}-rebuild".
>> Changes can be made in output/build/${pkg}/* and make ${pkg}-rebuild will
>> run the normal build command for that directory again.
>>
>> Signed-off-by: Quotient Remainder <quotientvremainder@gmail.com>
>
> I also have a patch (below) that implements a similar feature
> (<pkg>-rebuild and <pkg>-reconfigure), but it also does the "make"
> again automatically as well to regenerate the root filesystem. The
> implementation is a bit different, as it doesn't use a new fake stamp
> file, but simply uses phony targets.

Has this patch already been applied?

I like the fact that rebuild and reconfigure are waterfall steps that
continue to the regeneration of the root filesystem. This was also my
last comment for Quotient's reconfigure (in the discussion of PATCH
0/2).

I'll leave it up to you and Quotient to agree upon which patch
implementation makes it through...

Best regards,
Thomas

[Buildroot] [PATCH 1/2] Allow rebuilding of generic targets.

[Quotient Remainder]

If Thomas' patch works (and I can see no reason why it wouldn't) then I'd
prefer it because of its simpler, cleaner approach.
Is there a case where we might want to override the clean-for commands per
package?
On 19 I?il 2011 07:52, "Thomas De Schampheleire" <
patrickdepinguin+buildroot@gmail.com> wrote:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20110719/3e34da9d/attachment.html>

[Buildroot] [PATCH 1/2] Allow rebuilding of generic targets.

[Thomas Petazzoni]

Le Tue, 19 Jul 2011 08:52:07 +0200,
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :

> Has this patch already been applied?

No, not yet. I intend to send a pull request for it (and other
features) quite soon. Maybe today, or tomorrow morning.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com