[Buildroot] [PATCH 1/1] Makefile: add target to clean targetfs

[Buildroot] [PATCH 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

If you had some files added to the targetfs (fs-overlay, new packages,
...) and you no longer need them, buildroot does not offer a simple way
to remove these items from the output/target directory. The rule added
by this commit allows you to clean the targetfs. Issuing 'make'
afterward will generate a new and clean targetfs.

A section in the documentation is also added. It describes few examples
when developers could need it.

Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>
---
 Makefile                                |  5 +++++
 docs/manual/advanced.txt                |  2 ++
 docs/manual/clean-target-filesystem.txt | 37 +++++++++++++++++++++++++++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/clean-target-filesystem.txt

diff --git a/Makefile b/Makefile
index af043a3..eee2dbb 100644
--- a/Makefile
+++ b/Makefile
@@ -679,6 +679,11 @@ graph-depends: graph-depends-requirements
 	|tee $(BASE_DIR)/graphs/$(@).dot \
 	|dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) -o $(BASE_DIR)/graphs/$(@).$(BR_GRAPH_OUT)
 
+clean-target:
+	rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
+		$(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed \
+		$(BUILD_DIR)/linux-*/.stamp_installed $(BUILD_DIR)/*/.built
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 all: menuconfig
diff --git a/docs/manual/advanced.txt b/docs/manual/advanced.txt
index b7bfc49..1cf0427 100644
--- a/docs/manual/advanced.txt
+++ b/docs/manual/advanced.txt
@@ -14,3 +14,5 @@ include::download-location.txt[]
 include::package-make-target.txt[]
 
 include::using-buildroot-development.txt[]
+
+include::clean-target-filesystem.txt[]
diff --git a/docs/manual/clean-target-filesystem.txt b/docs/manual/clean-target-filesystem.txt
new file mode 100644
index 0000000..0274474
--- /dev/null
+++ b/docs/manual/clean-target-filesystem.txt
@@ -0,0 +1,37 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+==== Clean target filesystem
+
+There are various cases where you would like to clean the target
+filesystem. Especially once you understood that buildroot build system
+is kept simple, and does not allow to uninstall files. Let's take some
+examples:
+
+1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
+   board. But you figure out that +sshd+ does not run well on your
+   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
+   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
+   gets built and installed. Problem: +openssh+ files are still present
+   in +output/target+!
+
+2. You enabled +PACKAGE_NTPD+ and set the local timezone to
+   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
+   out that +Montreal+ local has been replaced by +Toronto+ (French
+   looks like a rogue language). You enter +make menuconfig+, make the
+   change, save, and run +make+ again. Problem:
+   +output/target/etc/timezone+ still has +America/Montreal+ content!
+
+Solution to this is either to identify which package to reconfigure, or
+remove files manually, or destroy the target filesystem and recreate it.
+This latter option is convenient: you don't have to care about which
+packages to rebuild, and it is fast. Here is how to recreate your
+target filesystem, and get rid of filesystem _stains_:
+
+------------------
+make clean-target
+make
+------------------
+
++clean-target+ removes +output/target+, tells buildroot to recreate the
+target skeleton, and install each package again.
-- 
1.9.1

[Buildroot] [PATCH 1/1] Makefile: add target to clean targetfs

[Baruch Siach]

Hi Emeric,

On Mon, Mar 09, 2015 at 08:25:00PM -0400, Emeric Vigier wrote:
> +clean-target:
> +	rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
> +		$(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed \
> +		$(BUILD_DIR)/linux-*/.stamp_installed $(BUILD_DIR)/*/.built

.stamp_built ?

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

On 15-03-10 12:45 AM, Baruch Siach wrote:
> Hi Emeric,
>
> On Mon, Mar 09, 2015 at 08:25:00PM -0400, Emeric Vigier wrote:
>> +clean-target:
>> +	rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
>> +		$(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed \
>> +		$(BUILD_DIR)/linux-*/.stamp_installed $(BUILD_DIR)/*/.built
>
> .stamp_built ?

Yes, you are right.

Also it seems that linux-*/.stamp_installed is deprecated. I only see 
.stamp_target_installed files.

>
> baruch
>

Emeric

[Buildroot] [PATCH 1/1] Makefile: add target to clean targetfs

[Baruch Siach]

Hi Emeric,

On Tue, Mar 10, 2015 at 10:56:45AM -0400, Emeric Vigier wrote:
> On 15-03-10 12:45 AM, Baruch Siach wrote:
> >On Mon, Mar 09, 2015 at 08:25:00PM -0400, Emeric Vigier wrote:
> >>+clean-target:
> >>+	rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
> >>+		$(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed \
> >>+		$(BUILD_DIR)/linux-*/.stamp_installed $(BUILD_DIR)/*/.built
> >
> >.stamp_built ?
> 
> Yes, you are right.

But why? Isn't .stamp_*_installed removal enough for your needs?

> Also it seems that linux-*/.stamp_installed is deprecated. I only see
> .stamp_target_installed files.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

If you had some files added to the targetfs (fs-overlay, new packages,
...) and you no longer need them, buildroot does not offer a simple way
to remove these items from the output/target directory. The rule added
by this commit allows you to clean the targetfs. Issuing 'make'
afterward will generate a new and clean targetfs.

A section in the documentation is also added. It describes few examples
when developers could need it. 'make help' is also updated accordingly.

Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>
---
Changes v1 -> v2
  - remove non-existent .built and deprecated .stamp_installed
    references (suggested by Baruch Siach)
  - mention the new target in 'make help'
---
 Makefile                                |  5 +++++
 docs/manual/advanced.txt                |  2 ++
 docs/manual/clean-target-filesystem.txt | 37 +++++++++++++++++++++++++++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/clean-target-filesystem.txt

diff --git a/Makefile b/Makefile
index af043a3..45ad2aa 100644
--- a/Makefile
+++ b/Makefile
@@ -679,6 +679,10 @@ graph-depends: graph-depends-requirements
 	|tee $(BASE_DIR)/graphs/$(@).dot \
 	|dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) -o $(BASE_DIR)/graphs/$(@).$(BR_GRAPH_OUT)
 
+clean-target:
+	rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
+		$(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 all: menuconfig
@@ -849,6 +853,7 @@ help:
 	@echo 'Cleaning:'
 	@echo '  clean                  - delete all files created by build'
 	@echo '  distclean              - delete all non-source files (including .config)'
+	@echo '  clean-target           - delete targetfs, issuing make again will recreate it'
 	@echo
 	@echo 'Build:'
 	@echo '  all                    - make world'
diff --git a/docs/manual/advanced.txt b/docs/manual/advanced.txt
index b7bfc49..1cf0427 100644
--- a/docs/manual/advanced.txt
+++ b/docs/manual/advanced.txt
@@ -14,3 +14,5 @@ include::download-location.txt[]
 include::package-make-target.txt[]
 
 include::using-buildroot-development.txt[]
+
+include::clean-target-filesystem.txt[]
diff --git a/docs/manual/clean-target-filesystem.txt b/docs/manual/clean-target-filesystem.txt
new file mode 100644
index 0000000..0274474
--- /dev/null
+++ b/docs/manual/clean-target-filesystem.txt
@@ -0,0 +1,37 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+==== Clean target filesystem
+
+There are various cases where you would like to clean the target
+filesystem. Especially once you understood that buildroot build system
+is kept simple, and does not allow to uninstall files. Let's take some
+examples:
+
+1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
+   board. But you figure out that +sshd+ does not run well on your
+   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
+   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
+   gets built and installed. Problem: +openssh+ files are still present
+   in +output/target+!
+
+2. You enabled +PACKAGE_NTPD+ and set the local timezone to
+   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
+   out that +Montreal+ local has been replaced by +Toronto+ (French
+   looks like a rogue language). You enter +make menuconfig+, make the
+   change, save, and run +make+ again. Problem:
+   +output/target/etc/timezone+ still has +America/Montreal+ content!
+
+Solution to this is either to identify which package to reconfigure, or
+remove files manually, or destroy the target filesystem and recreate it.
+This latter option is convenient: you don't have to care about which
+packages to rebuild, and it is fast. Here is how to recreate your
+target filesystem, and get rid of filesystem _stains_:
+
+------------------
+make clean-target
+make
+------------------
+
++clean-target+ removes +output/target+, tells buildroot to recreate the
+target skeleton, and install each package again.
-- 
1.9.1

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Ryan Barnett]

Emeric,

In the future - could you put a version - such as 'v2' - into the
title utilizing the -'-subject-prefix' parameter of git-send-email?

For details and an example see the buildroot manual and section 21.5.2:

http://nightly.buildroot.org/manual.html#submitting-patches

On Tue, Mar 10, 2015 at 2:28 PM, Emeric Vigier
<emeric.vigier@savoirfairelinux.com> wrote:
> If you had some files added to the targetfs (fs-overlay, new packages,
> ...) and you no longer need them, buildroot does not offer a simple way
> to remove these items from the output/target directory. The rule added
> by this commit allows you to clean the targetfs. Issuing 'make'
> afterward will generate a new and clean targetfs.
>
> A section in the documentation is also added. It describes few examples
> when developers could need it. 'make help' is also updated accordingly.
>
> Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>
> ---
> Changes v1 -> v2
>   - remove non-existent .built and deprecated .stamp_installed
>     references (suggested by Baruch Siach)
>   - mention the new target in 'make help'
> ---
>  Makefile                                |  5 +++++
>  docs/manual/advanced.txt                |  2 ++
>  docs/manual/clean-target-filesystem.txt | 37 +++++++++++++++++++++++++++++++++
>  3 files changed, 44 insertions(+)
>  create mode 100644 docs/manual/clean-target-filesystem.txt

[...]

> diff --git a/docs/manual/advanced.txt b/docs/manual/advanced.txt
> index b7bfc49..1cf0427 100644
> --- a/docs/manual/advanced.txt
> +++ b/docs/manual/advanced.txt
> @@ -14,3 +14,5 @@ include::download-location.txt[]
>  include::package-make-target.txt[]
>
>  include::using-buildroot-development.txt[]
> +
> +include::clean-target-filesystem.txt[]
> diff --git a/docs/manual/clean-target-filesystem.txt b/docs/manual/clean-target-filesystem.txt
> new file mode 100644
> index 0000000..0274474
> --- /dev/null
> +++ b/docs/manual/clean-target-filesystem.txt

Would it make sense to also update the section of the manual that
talks about understand when a full-rebuild is necessary? You could
also update that section to reference this advance usage section.

http://nightly.buildroot.org/manual.html#full-rebuild

> @@ -0,0 +1,37 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +==== Clean target filesystem
> +
> +There are various cases where you would like to clean the target
> +filesystem. Especially once you understood that buildroot build system
> +is kept simple, and does not allow to uninstall files. Let's take some
> +examples:
> +
> +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
> +   board. But you figure out that +sshd+ does not run well on your
> +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
> +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
> +   gets built and installed. Problem: +openssh+ files are still present
> +   in +output/target+!

One case that comes to mind here is that will this ensure that
packages are installed in the correct order with dependencies? I am
not exactly sure how the dependencies for packages works, but if you
remove all the '.stamp_target_install' will that ensure that all the
packages get installed in the correct order?

Busybox is the package that comes to my mind first that we need to
ensure the installation order. From testing your scenario outlined
above it appears that busybox gets installed before even dropbear
would get built which would ensure that any full version of busybox
would get install correctly over the symlinks to busybox.

> +2. You enabled +PACKAGE_NTPD+ and set the local timezone to
> +   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
> +   out that +Montreal+ local has been replaced by +Toronto+ (French
> +   looks like a rogue language). You enter +make menuconfig+, make the
> +   change, save, and run +make+ again. Problem:
> +   +output/target/etc/timezone+ still has +America/Montreal+ content!
> +
> +Solution to this is either to identify which package to reconfigure, or

I believe this first 'or' should be removed so that the sentence would read:

The solution to this is to either identify which package to
reconfigure, remove files manually, or destroy the target filesystem
and recreate it.

> +remove files manually, or destroy the target filesystem and recreate it.
> +This latter option is convenient: you don't have to care about which
> +packages to rebuild, and it is fast. Here is how to recreate your
> +target filesystem, and get rid of filesystem _stains_:
> +
> +------------------
> +make clean-target
> +make
> +------------------
> +
> ++clean-target+ removes +output/target+, tells buildroot to recreate the
> +target skeleton, and install each package again.

This is a rather simple solution to the issue of recreating a 'clean'
target filesystem so it will be interesting to hear what the
maintainers (Peter/Thomas P) opinions are on this patch.

Thanks,
-Ryan

-- 
Ryan Barnett / Sr Software Engineer
Airborne Information Systems / Security Systems and Software
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
ryan.barnett at rockwellcollins.com
www.rockwellcollins.com

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Baruch Siach]

Hi Ryan,

On Tue, Mar 10, 2015 at 06:41:39PM -0500, Ryan Barnett wrote:
> > +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
> > +   board. But you figure out that +sshd+ does not run well on your
> > +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
> > +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
> > +   gets built and installed. Problem: +openssh+ files are still present
> > +   in +output/target+!
> 
> One case that comes to mind here is that will this ensure that
> packages are installed in the correct order with dependencies? I am
> not exactly sure how the dependencies for packages works, but if you
> remove all the '.stamp_target_install' will that ensure that all the
> packages get installed in the correct order?
> 
> Busybox is the package that comes to my mind first that we need to
> ensure the installation order. From testing your scenario outlined
> above it appears that busybox gets installed before even dropbear
> would get built which would ensure that any full version of busybox
> would get install correctly over the symlinks to busybox.

The package dependencies system should ensure the right installation order. 
The following excerpts from package/pkg-generic.mk demonstrate that:

$(2)_TARGET_INSTALL_TARGET =    $$($(2)_DIR)/.stamp_target_installed

...

$(1):                   $(1)-install

...

$(1)-install:           $(1)-install-staging $(1)-install-target $(1)-install-images

...

$(1)-install-target:            $$($(2)_TARGET_INSTALL_TARGET)

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Angelo Compagnucci]

Hi Baruch,

I submitted a couple of patches some times ago, but they was rejected
because potentially dangerous for users[1][2].
So probably you will not have that much luck with this patch.

Sincerely, Angelo.

[1] http://lists.busybox.net/pipermail/buildroot/2014-July/101963.html
[2] http://lists.busybox.net/pipermail/buildroot/2014-July/103143.html

2015-03-11 6:06 GMT+01:00 Baruch Siach <baruch@tkos.co.il>:
> Hi Ryan,
>
> On Tue, Mar 10, 2015 at 06:41:39PM -0500, Ryan Barnett wrote:
>> > +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
>> > +   board. But you figure out that +sshd+ does not run well on your
>> > +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
>> > +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
>> > +   gets built and installed. Problem: +openssh+ files are still present
>> > +   in +output/target+!
>>
>> One case that comes to mind here is that will this ensure that
>> packages are installed in the correct order with dependencies? I am
>> not exactly sure how the dependencies for packages works, but if you
>> remove all the '.stamp_target_install' will that ensure that all the
>> packages get installed in the correct order?
>>
>> Busybox is the package that comes to my mind first that we need to
>> ensure the installation order. From testing your scenario outlined
>> above it appears that busybox gets installed before even dropbear
>> would get built which would ensure that any full version of busybox
>> would get install correctly over the symlinks to busybox.
>
> The package dependencies system should ensure the right installation order.
> The following excerpts from package/pkg-generic.mk demonstrate that:
>
> $(2)_TARGET_INSTALL_TARGET =    $$($(2)_DIR)/.stamp_target_installed
>
> ...
>
> $(1):                   $(1)-install
>
> ...
>
> $(1)-install:           $(1)-install-staging $(1)-install-target $(1)-install-images
>
> ...
>
> $(1)-install-target:            $$($(2)_TARGET_INSTALL_TARGET)
>
> baruch
>
> --
>      http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
> =}------------------------------------------------ooO--U--Ooo------------{=
>    - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot



-- 
Profile: http://it.linkedin.com/in/compagnucciangelo

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Baruch Siach]

Hi Angelo,

On Wed, Mar 11, 2015 at 09:22:49AM +0100, Angelo Compagnucci wrote:
> I submitted a couple of patches some times ago, but they was rejected
> because potentially dangerous for users[1][2].
> So probably you will not have that much luck with this patch.
>
> [1] http://lists.busybox.net/pipermail/buildroot/2014-July/101963.html
> [2] http://lists.busybox.net/pipermail/buildroot/2014-July/103143.html

Well, it is not me suggesting this patch, it is Emeric Vigier. But yes, one of 
our clients asked for this functionality, so I really want this patch applied.

As for previous rejections, the response of ThomasP[1] and Arnout[2] seems to 
indicate that they would be willing to accept this functional addition 
provided that it is (1) properly documented[1], and (2) prints a "big fat 
warning"[2]. Something like Thomas' scenario description[3] should be added to 
the documentation part of this patch.

Thanks for reminding us this previous discussion.

[1] http://lists.busybox.net/pipermail/buildroot/2014-July/101966.html

[2] http://lists.busybox.net/pipermail/buildroot/2014-July/102064.html

[3] http://lists.busybox.net/pipermail/buildroot/2014-July/101941.html

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Angelo Compagnucci]

Hi Baruch,

2015-03-11 9:58 GMT+01:00 Baruch Siach <baruch@tkos.co.il>:
> Hi Angelo,
>
> On Wed, Mar 11, 2015 at 09:22:49AM +0100, Angelo Compagnucci wrote:
>> I submitted a couple of patches some times ago, but they was rejected
>> because potentially dangerous for users[1][2].
>> So probably you will not have that much luck with this patch.
>>
>> [1] http://lists.busybox.net/pipermail/buildroot/2014-July/101963.html
>> [2] http://lists.busybox.net/pipermail/buildroot/2014-July/103143.html
>
> Well, it is not me suggesting this patch, it is Emeric Vigier. But yes, one of
> our clients asked for this functionality, so I really want this patch applied.
>
> As for previous rejections, the response of ThomasP[1] and Arnout[2] seems to
> indicate that they would be willing to accept this functional addition
> provided that it is (1) properly documented[1], and (2) prints a "big fat
> warning"[2]. Something like Thomas' scenario description[3] should be added to
> the documentation part of this patch.
>
> Thanks for reminding us this previous discussion.
>
> [1] http://lists.busybox.net/pipermail/buildroot/2014-July/101966.html
>
> [2] http://lists.busybox.net/pipermail/buildroot/2014-July/102064.html
>
> [3] http://lists.busybox.net/pipermail/buildroot/2014-July/101941.html

I have my patch on top of my git rebase everytime I pull from master,
so I will be veeeeery glad if you can have it upstreamed!
I love so much the option to clean the target that I use it everyday!

Sincerely, Angelo

>
> baruch
>
> --
>      http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
> =}------------------------------------------------ooO--U--Ooo------------{=
>    - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -



-- 
Profile: http://it.linkedin.com/in/compagnucciangelo

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

On 15-03-12 04:32 AM, Angelo Compagnucci wrote:
> Hi Baruch,
>
> 2015-03-11 9:58 GMT+01:00 Baruch Siach <baruch@tkos.co.il>:
>> Hi Angelo,
>>
>> On Wed, Mar 11, 2015 at 09:22:49AM +0100, Angelo Compagnucci wrote:
>>> I submitted a couple of patches some times ago, but they was rejected
>>> because potentially dangerous for users[1][2].
>>> So probably you will not have that much luck with this patch.
>>>
>>> [1] http://lists.busybox.net/pipermail/buildroot/2014-July/101963.html
>>> [2] http://lists.busybox.net/pipermail/buildroot/2014-July/103143.html
>>
>> Well, it is not me suggesting this patch, it is Emeric Vigier. But yes, one of
>> our clients asked for this functionality, so I really want this patch applied.
>>
>> As for previous rejections, the response of ThomasP[1] and Arnout[2] seems to
>> indicate that they would be willing to accept this functional addition
>> provided that it is (1) properly documented[1], and (2) prints a "big fat
>> warning"[2]. Something like Thomas' scenario description[3] should be added to
>> the documentation part of this patch.
>>
>> Thanks for reminding us this previous discussion.
>>
>> [1] http://lists.busybox.net/pipermail/buildroot/2014-July/101966.html
>>
>> [2] http://lists.busybox.net/pipermail/buildroot/2014-July/102064.html
>>
>> [3] http://lists.busybox.net/pipermail/buildroot/2014-July/101941.html
>
> I have my patch on top of my git rebase everytime I pull from master,
> so I will be veeeeery glad if you can have it upstreamed!
> I love so much the option to clean the target that I use it everyday!

Indeed, it seems I had the same scenario in mind as you when you 
submitted your patch. Let me add your signed-off to my patch, and we'll 
both push strong to have it upstreamed!

Signed-off-by: Angelo Compagnucci <angelo.compagnucci@gmail.com>

I'll send a v3 soon.

>
> Sincerely, Angelo
>
>>
>> baruch
>>
>> --
>>       http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
>> =}------------------------------------------------ooO--U--Ooo------------{=
>>     - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -
>

Emeric

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

On 15-03-10 07:41 PM, Ryan Barnett wrote:
> Emeric,
>
> In the future - could you put a version - such as 'v2' - into the
> title utilizing the -'-subject-prefix' parameter of git-send-email?

I did, using git format-patch:

   $ git format-patch -M -n -s --subject-prefix="PATCH v2" -o outgoing 
buildroot/master

And I see [PATCH v2 1/1] in the title. Did you miss something?

>
> For details and an example see the buildroot manual and section 21.5.2:
>
> http://nightly.buildroot.org/manual.html#submitting-patches
>

[...]

>
> Would it make sense to also update the section of the manual that
> talks about understand when a full-rebuild is necessary? You could
> also update that section to reference this advance usage section.
>
> http://nightly.buildroot.org/manual.html#full-rebuild

I believe so, yes. I'll add a brief scenario in this chapter and see how 
I can warn users about package dependencies.

>
>> @@ -0,0 +1,37 @@
>> +// -*- mode:doc; -*-
>> +// vim: set syntax=asciidoc:
>> +
>> +==== Clean target filesystem
>> +
>> +There are various cases where you would like to clean the target
>> +filesystem. Especially once you understood that buildroot build system
>> +is kept simple, and does not allow to uninstall files. Let's take some
>> +examples:
>> +
>> +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
>> +   board. But you figure out that +sshd+ does not run well on your
>> +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
>> +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+

Oops, small typo here:
-and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
+and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +dropbear+

>> +   gets built and installed. Problem: +openssh+ files are still present
>> +   in +output/target+!
>
> One case that comes to mind here is that will this ensure that
> packages are installed in the correct order with dependencies? I am
> not exactly sure how the dependencies for packages works, but if you
> remove all the '.stamp_target_install' will that ensure that all the
> packages get installed in the correct order?
>
> Busybox is the package that comes to my mind first that we need to
> ensure the installation order. From testing your scenario outlined
> above it appears that busybox gets installed before even dropbear
> would get built which would ensure that any full version of busybox
> would get install correctly over the symlinks to busybox.
>
>> +2. You enabled +PACKAGE_NTPD+ and set the local timezone to
>> +   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
>> +   out that +Montreal+ local has been replaced by +Toronto+ (French
>> +   looks like a rogue language). You enter +make menuconfig+, make the
>> +   change, save, and run +make+ again. Problem:
>> +   +output/target/etc/timezone+ still has +America/Montreal+ content!
>> +
>> +Solution to this is either to identify which package to reconfigure, or
>
> I believe this first 'or' should be removed so that the sentence would read:
>
> The solution to this is to either identify which package to
> reconfigure, remove files manually, or destroy the target filesystem
> and recreate it.

Ok, I'll fix that.

>
>> +remove files manually, or destroy the target filesystem and recreate it.
>> +This latter option is convenient: you don't have to care about which
>> +packages to rebuild, and it is fast. Here is how to recreate your
>> +target filesystem, and get rid of filesystem _stains_:
>> +
>> +------------------
>> +make clean-target
>> +make
>> +------------------
>> +
>> ++clean-target+ removes +output/target+, tells buildroot to recreate the
>> +target skeleton, and install each package again.
>
> This is a rather simple solution to the issue of recreating a 'clean'
> target filesystem so it will be interesting to hear what the
> maintainers (Peter/Thomas P) opinions are on this patch.
>
> Thanks,
> -Ryan
>

Emeric

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Ryan Barnett]

Emeric,

On Thu, Mar 12, 2015 at 9:44 AM, Emeric Vigier
<emeric.vigier@savoirfairelinux.com> wrote:
> On 15-03-10 07:41 PM, Ryan Barnett wrote:
>>
>> Emeric,
>>
>> In the future - could you put a version - such as 'v2' - into the
>> title utilizing the -'-subject-prefix' parameter of git-send-email?
>
>
> I did, using git format-patch:
>
>   $ git format-patch -M -n -s --subject-prefix="PATCH v2" -o outgoing
> buildroot/master
>
> And I see [PATCH v2 1/1] in the title. Did you miss something?

Sorry for the confusion. I looked at the your previous email when
reviewing but wrote the response in this one.

Thanks,
-Ryan

-- 
Ryan Barnett / Sr Software Engineer
Airborne Information Systems / Security Systems and Software
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
ryan.barnett at rockwellcollins.com
www.rockwellcollins.com

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Angelo Compagnucci]

2015-03-10 20:28 GMT+01:00 Emeric Vigier <emeric.vigier@savoirfairelinux.com>:
> If you had some files added to the targetfs (fs-overlay, new packages,
> ...) and you no longer need them, buildroot does not offer a simple way
> to remove these items from the output/target directory. The rule added
> by this commit allows you to clean the targetfs. Issuing 'make'
> afterward will generate a new and clean targetfs.
>
> A section in the documentation is also added. It describes few examples
> when developers could need it. 'make help' is also updated accordingly.
>
> Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>
Reviewed-by: Angelo Compagnucci <angelo.compagnucci@gmail.com>
> ---
> Changes v1 -> v2
>   - remove non-existent .built and deprecated .stamp_installed
>     references (suggested by Baruch Siach)
>   - mention the new target in 'make help'
> ---
>  Makefile                                |  5 +++++
>  docs/manual/advanced.txt                |  2 ++
>  docs/manual/clean-target-filesystem.txt | 37 +++++++++++++++++++++++++++++++++
>  3 files changed, 44 insertions(+)
>  create mode 100644 docs/manual/clean-target-filesystem.txt
>
> diff --git a/Makefile b/Makefile
> index af043a3..45ad2aa 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -679,6 +679,10 @@ graph-depends: graph-depends-requirements
>         |tee $(BASE_DIR)/graphs/$(@).dot \
>         |dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) -o $(BASE_DIR)/graphs/$(@).$(BR_GRAPH_OUT)
>
> +clean-target:
> +       rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
> +               $(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed
> +
>  else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>
>  all: menuconfig
> @@ -849,6 +853,7 @@ help:
>         @echo 'Cleaning:'
>         @echo '  clean                  - delete all files created by build'
>         @echo '  distclean              - delete all non-source files (including .config)'
> +       @echo '  clean-target           - delete targetfs, issuing make again will recreate it'
>         @echo
>         @echo 'Build:'
>         @echo '  all                    - make world'
> diff --git a/docs/manual/advanced.txt b/docs/manual/advanced.txt
> index b7bfc49..1cf0427 100644
> --- a/docs/manual/advanced.txt
> +++ b/docs/manual/advanced.txt
> @@ -14,3 +14,5 @@ include::download-location.txt[]
>  include::package-make-target.txt[]
>
>  include::using-buildroot-development.txt[]
> +
> +include::clean-target-filesystem.txt[]
> diff --git a/docs/manual/clean-target-filesystem.txt b/docs/manual/clean-target-filesystem.txt
> new file mode 100644
> index 0000000..0274474
> --- /dev/null
> +++ b/docs/manual/clean-target-filesystem.txt
> @@ -0,0 +1,37 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +==== Clean target filesystem
> +
> +There are various cases where you would like to clean the target
> +filesystem. Especially once you understood that buildroot build system
> +is kept simple, and does not allow to uninstall files. Let's take some
> +examples:
> +
> +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
> +   board. But you figure out that +sshd+ does not run well on your
> +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
> +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+, +openssh+
> +   gets built and installed. Problem: +openssh+ files are still present
> +   in +output/target+!
> +
> +2. You enabled +PACKAGE_NTPD+ and set the local timezone to
> +   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
> +   out that +Montreal+ local has been replaced by +Toronto+ (French
> +   looks like a rogue language). You enter +make menuconfig+, make the
> +   change, save, and run +make+ again. Problem:
> +   +output/target/etc/timezone+ still has +America/Montreal+ content!
> +
> +Solution to this is either to identify which package to reconfigure, or
> +remove files manually, or destroy the target filesystem and recreate it.
> +This latter option is convenient: you don't have to care about which
> +packages to rebuild, and it is fast. Here is how to recreate your
> +target filesystem, and get rid of filesystem _stains_:
> +
> +------------------
> +make clean-target
> +make
> +------------------
> +
> ++clean-target+ removes +output/target+, tells buildroot to recreate the
> +target skeleton, and install each package again.
> --
> 1.9.1

This patch works for me, I think the documentation is well written and usefull.

Sincerely, Angelo.

>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot



-- 
Profile: http://it.linkedin.com/in/compagnucciangelo

[Buildroot] [PATCH v2 1/1] Makefile: add target to clean targetfs

[Jérôme Oufella]

Emeric Vigier <emeric.vigier@savoirfairelinux.com> writes:

> +make clean-target

Wouldn't it make more sense to name it target-dirclean or
target-clean-something, to comply with Buildroot's usual semantics ?

-- 
J?r?me Oufella

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

If you had some files added to the targetfs (fs-overlay, new packages,
...) and you no longer need them, buildroot does not offer a simple way
to remove these items from the output/target directory. The rule added
by this commit allows you to clean the targetfs. Issuing 'make'
afterward will generate a new and clean targetfs.

A section in the documentation is also added. It describes few examples
when developers could need it. 'make help' is also updated accordingly.
And full-rebuild section of the documentation is updated with content
and links.

Change-Id: I42b640e845bb4195fb7b160281b16ee40883f01f
Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>
---
Changes v2 -> v3
  - rename clean-target into target-clean (sugg. Jerome Oufella)
  - print a big fat warning at the end of target-clean (sugg. Arnout
    Vandecappelle)
  - mention the new rule in full-rebuild manual section (sugg. Ryan
    Barnett)

Changes v1 -> v2
  - remove non-existent .built and deprecated .stamp_installed
    references (suggested by Baruch Siach)
  - mention the new target in 'make help'
---
 Makefile                                | 13 +++++++++++
 docs/manual/advanced.txt                |  2 ++
 docs/manual/clean-target-filesystem.txt | 39 +++++++++++++++++++++++++++++++++
 docs/manual/rebuilding-packages.txt     | 29 ++++++++++++++----------
 4 files changed, 71 insertions(+), 12 deletions(-)
 create mode 100644 docs/manual/clean-target-filesystem.txt

diff --git a/Makefile b/Makefile
index 3983fce..b9db264 100644
--- a/Makefile
+++ b/Makefile
@@ -666,6 +666,18 @@ graph-depends: graph-depends-requirements
 	|tee $(GRAPHS_DIR)/$(@).dot \
 	|dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) -o $(GRAPHS_DIR)/$(@).$(BR_GRAPH_OUT)
 
+target-clean:
+	@echo "----------------------------------------------------------------"
+	@echo "WARNING! target-clean does not fix package dependencies for you."
+	@echo "Think twice about what you are attempting! For instance, if you "
+	@echo "remove openssl needed by git, and just run make target-clean "
+	@echo "without rebuilding git, your rootfs will embed a broken git. It "
+	@echo "will fail to execute because it is linked again +openssl+, and  "
+	@echo "this library is not present anymore on your target rootfs."
+	@echo "----------------------------------------------------------------"
+	rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
+		$(BUILD_DIR)/*/.stamp_target_installed $(BUILD_DIR)/*/.stamp_staging_installed
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 all: menuconfig
@@ -821,6 +833,7 @@ help:
 	@echo 'Cleaning:'
 	@echo '  clean                  - delete all files created by build'
 	@echo '  distclean              - delete all non-source files (including .config)'
+	@echo '  target-clean           - delete targetfs and staging directories'
 	@echo
 	@echo 'Build:'
 	@echo '  all                    - make world'
diff --git a/docs/manual/advanced.txt b/docs/manual/advanced.txt
index b7bfc49..1cf0427 100644
--- a/docs/manual/advanced.txt
+++ b/docs/manual/advanced.txt
@@ -14,3 +14,5 @@ include::download-location.txt[]
 include::package-make-target.txt[]
 
 include::using-buildroot-development.txt[]
+
+include::clean-target-filesystem.txt[]
diff --git a/docs/manual/clean-target-filesystem.txt b/docs/manual/clean-target-filesystem.txt
new file mode 100644
index 0000000..3e8181e
--- /dev/null
+++ b/docs/manual/clean-target-filesystem.txt
@@ -0,0 +1,39 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[clean-target-filesystem]]
+==== Clean target filesystem
+
+There are various cases where you would like to clean the target
+filesystem. Especially once you understood that buildroot does not
+have rules to uninstall files. Let's take some examples:
+
+1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
+   board. But you figure out that +sshd+ does not run well on your
+   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
+   and enable +PACKAGE_DROPBEAR+ instead. You run +make+: +dropbear+
+   gets built and installed, fine. But +openssh+ files are still present
+   in +output/target+!
+
+2. You enabled +PACKAGE_NTPD+ and set the local timezone to
+   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
+   out that +Montreal+ local has been replaced by +Toronto+ (French
+   looks like a rogue language). You enter +make menuconfig+, make the
+   change, save, and run +make+ again. Problem:
+   +output/target/etc/timezone+ still has +America/Montreal+ content!
+
+Solution to this is either to identify which package to reconfigure, or
+remove files manually, or destroy the target filesystem and recreate it.
+This latter option is convenient: you don't have to care about which
+packages to rebuild, and it is quite fast. Here is how to recreate your
+target filesystem, and get rid of filesystem _stains_:
+
+------------------
+make target-clean
+make
+------------------
+
++target-clean+ removes +output/target+. That tells buildroot to recreate
+the target skeleton, and install each package again. This is a
+convenient way to rebuild your target filesystem cleanly without
+rebuilding everything.
diff --git a/docs/manual/rebuilding-packages.txt b/docs/manual/rebuilding-packages.txt
index 6faa67a..82385ae 100644
--- a/docs/manual/rebuilding-packages.txt
+++ b/docs/manual/rebuilding-packages.txt
@@ -39,17 +39,22 @@ can help you understand how to work with Buildroot:
    you would like to have SSL support in +ctorrent+, so you enable the
    +openssl+ package in Buildroot configuration and restart the
    build. Buildroot will detect that +openssl+ should be built and
-   will be build it, but it will not detect that +ctorrent+ should be
-   rebuilt to benefit from +openssl+ to add OpenSSL support. You will
-   either have to do a full rebuild, or rebuild +ctorrent+ itself.
+   will build it, but it will not detect that +ctorrent+ should be
+   rebuilt to benefit from +openssl+ support. You will either have to
+   do a full rebuild, or rebuild +ctorrent+ itself.
 
  * When a package is removed from the configuration, Buildroot does
    not do anything special. It does not remove the files installed by
    this package from the target root filesystem or from the toolchain
-   _sysroot_. A full rebuild is needed to get rid of this
-   package. However, generally you don't necessarily need this package
-   to be removed right now: you can wait for the next lunch break to
-   restart the build from scratch.
+   _sysroot_. To make sure that the target filesystem and _sysroot_
+   reflects your changes, you can use +make target-clean+ as described
+   in xref:clean-target-filesystem[]. Warning! +target-clean+
+   does not fix package dependencies for you. You still have to think
+   twice about what you are attempting. For instance, if you remove
+   +openssl+ needed by +git+, and just +target-clean+ without
+   rebuilding +git+, your rootfs will embed a broken +git+. It will
+   fail to work because it is linked again +openssl+, but this library
+   is not present anymore on your target rootfs.
 
  * When the sub-options of a package are changed, the package is not
    automatically rebuilt. After making such changes, rebuilding only
@@ -59,11 +64,11 @@ can help you understand how to work with Buildroot:
    not track when a package should be rebuilt: once a package has been
    built, it is never rebuilt unless explicitly told to do so.
 
- * When a change to the root filesystem skeleton is made, a full
-   rebuild is needed. However, when changes to the root filesystem
-   overlay, a post-build script or a post-image script are made,
-   there is no need for a full rebuild: a simple +make+ invocation
-   will take the changes into account.
+ * When a change to the root filesystem skeleton is made, clean the
+   target filesystem (xref:clean-target-filesystem[]). However, when
+   changes to the root filesystem overlay, a post-build script or a
+   post-image script are made, this is not needed: a simple +make+
+   invocation will take the changes into account.
 
 Generally speaking, when you're facing a build error and you're unsure
 of the potential consequences of the configuration changes you've
-- 
1.9.1

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Baruch Siach]

Hi Emeric,

Thanks for following up on this. A few comments below.

On Sun, Apr 26, 2015 at 08:45:55PM -0400, Emeric Vigier wrote:
> If you had some files added to the targetfs (fs-overlay, new packages,
> ...) and you no longer need them, buildroot does not offer a simple way
> to remove these items from the output/target directory. The rule added
> by this commit allows you to clean the targetfs. Issuing 'make'
> afterward will generate a new and clean targetfs.
> 
> A section in the documentation is also added. It describes few examples
> when developers could need it. 'make help' is also updated accordingly.
> And full-rebuild section of the documentation is updated with content
> and links.
> 
> Change-Id: I42b640e845bb4195fb7b160281b16ee40883f01f

This Gerrit specific line is not useful for upstream Buildroot. Please remove.

> Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>

[snip]

> diff --git a/docs/manual/clean-target-filesystem.txt 
> b/docs/manual/clean-target-filesystem.txt
> new file mode 100644
> index 0000000..3e8181e
> --- /dev/null
> +++ b/docs/manual/clean-target-filesystem.txt
> @@ -0,0 +1,39 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[clean-target-filesystem]]
> +==== Clean target filesystem
> +
> +There are various cases where you would like to clean the target
> +filesystem. Especially once you understood that buildroot does not
> +have rules to uninstall files. Let's take some examples:
> +
> +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
> +   board. But you figure out that +sshd+ does not run well on your
> +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
> +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+: +dropbear+
> +   gets built and installed, fine. But +openssh+ files are still present
> +   in +output/target+!
> +
> +2. You enabled +PACKAGE_NTPD+ and set the local timezone to
> +   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
> +   out that +Montreal+ local has been replaced by +Toronto+ (French
> +   looks like a rogue language). You enter +make menuconfig+, make the
> +   change, save, and run +make+ again. Problem:
> +   +output/target/etc/timezone+ still has +America/Montreal+ content!
> +
> +Solution to this is either to identify which package to reconfigure, or
> +remove files manually, or destroy the target filesystem and recreate it.
> +This latter option is convenient: you don't have to care about which
> +packages to rebuild, and it is quite fast. Here is how to recreate your
> +target filesystem, and get rid of filesystem _stains_:
> +
> +------------------
> +make target-clean
> +make
> +------------------
> +
> ++target-clean+ removes +output/target+. That tells buildroot to recreate
> +the target skeleton, and install each package again. This is a
> +convenient way to rebuild your target filesystem cleanly without
> +rebuilding everything.

The warning is worth repeating here, I think.

> diff --git a/docs/manual/rebuilding-packages.txt b/docs/manual/rebuilding-packages.txt
> index 6faa67a..82385ae 100644
> --- a/docs/manual/rebuilding-packages.txt
> +++ b/docs/manual/rebuilding-packages.txt
> @@ -39,17 +39,22 @@ can help you understand how to work with Buildroot:
>     you would like to have SSL support in +ctorrent+, so you enable the
>     +openssl+ package in Buildroot configuration and restart the
>     build. Buildroot will detect that +openssl+ should be built and
> -   will be build it, but it will not detect that +ctorrent+ should be
> -   rebuilt to benefit from +openssl+ to add OpenSSL support. You will
> -   either have to do a full rebuild, or rebuild +ctorrent+ itself.
> +   will build it, but it will not detect that +ctorrent+ should be
> +   rebuilt to benefit from +openssl+ support. You will either have to
> +   do a full rebuild, or rebuild +ctorrent+ itself.

This is an unrelated fix. Please submit in a separate patch.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

On 15-04-27 12:11 AM, Baruch Siach wrote:
> Hi Emeric,

Hi Baruch,

>>
>> Change-Id: I42b640e845bb4195fb7b160281b16ee40883f01f
>
> This Gerrit specific line is not useful for upstream Buildroot. Please remove.

Sure.

>
>> Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>
>
> [snip]
>
>> diff --git a/docs/manual/clean-target-filesystem.txt
>> b/docs/manual/clean-target-filesystem.txt
>> new file mode 100644
>> index 0000000..3e8181e
>> --- /dev/null
>> +++ b/docs/manual/clean-target-filesystem.txt
>> @@ -0,0 +1,39 @@
>> +// -*- mode:doc; -*-
>> +// vim: set syntax=asciidoc:
>> +
>> +[[clean-target-filesystem]]
>> +==== Clean target filesystem
>> +
>> +There are various cases where you would like to clean the target
>> +filesystem. Especially once you understood that buildroot does not
>> +have rules to uninstall files. Let's take some examples:
>> +
>> +1. You enabled +PACKAGE_OPENSSH+ to have a SSH server running on your
>> +   board. But you figure out that +sshd+ does not run well on your
>> +   target's architecture. Thus you decide to disable +PACKAGE_OPENSSH+
>> +   and enable +PACKAGE_DROPBEAR+ instead. You run +make+: +dropbear+
>> +   gets built and installed, fine. But +openssh+ files are still present
>> +   in +output/target+!
>> +
>> +2. You enabled +PACKAGE_NTPD+ and set the local timezone to
>> +   +America/Montreal+ with +BR2_TARGET_LOCALTIME+ config. Then you find
>> +   out that +Montreal+ local has been replaced by +Toronto+ (French
>> +   looks like a rogue language). You enter +make menuconfig+, make the
>> +   change, save, and run +make+ again. Problem:
>> +   +output/target/etc/timezone+ still has +America/Montreal+ content!
>> +
>> +Solution to this is either to identify which package to reconfigure, or
>> +remove files manually, or destroy the target filesystem and recreate it.
>> +This latter option is convenient: you don't have to care about which
>> +packages to rebuild, and it is quite fast. Here is how to recreate your
>> +target filesystem, and get rid of filesystem _stains_:
>> +
>> +------------------
>> +make target-clean
>> +make
>> +------------------
>> +
>> ++target-clean+ removes +output/target+. That tells buildroot to recreate
>> +the target skeleton, and install each package again. This is a
>> +convenient way to rebuild your target filesystem cleanly without
>> +rebuilding everything.
>
> The warning is worth repeating here, I think.

I don't like to repeat things. I suggest to move the warning here and 
put emphasis on the link from full-rebuild to here.

>
>> diff --git a/docs/manual/rebuilding-packages.txt b/docs/manual/rebuilding-packages.txt
>> index 6faa67a..82385ae 100644
>> --- a/docs/manual/rebuilding-packages.txt
>> +++ b/docs/manual/rebuilding-packages.txt
>> @@ -39,17 +39,22 @@ can help you understand how to work with Buildroot:
>>      you would like to have SSL support in +ctorrent+, so you enable the
>>      +openssl+ package in Buildroot configuration and restart the
>>      build. Buildroot will detect that +openssl+ should be built and
>> -   will be build it, but it will not detect that +ctorrent+ should be
>> -   rebuilt to benefit from +openssl+ to add OpenSSL support. You will
>> -   either have to do a full rebuild, or rebuild +ctorrent+ itself.
>> +   will build it, but it will not detect that +ctorrent+ should be
>> +   rebuilt to benefit from +openssl+ support. You will either have to
>> +   do a full rebuild, or rebuild +ctorrent+ itself.
>
> This is an unrelated fix. Please submit in a separate patch.

Sure.

>
> baruch

thanks for your review,
-- 
Emeric

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Andreas Naumann]

Am 27.04.2015 um 02:45 schrieb Emeric Vigier:
> If you had some files added to the targetfs (fs-overlay, new packages,
> ...) and you no longer need them, buildroot does not offer a simple way
> to remove these items from the output/target directory. The rule added
> by this commit allows you to clean the targetfs. Issuing 'make'
> afterward will generate a new and clean targetfs.
>
> A section in the documentation is also added. It describes few examples
> when developers could need it. 'make help' is also updated accordingly.
> And full-rebuild section of the documentation is updated with content
> and links.
>

Hi Emeric,

this is very helpful, in fact i have a similar patch running for quite 
some time where I remove target/ and images/ and the corresponding 
stamps. There are some problems though and I want to share my findings 
with you.
To begin with i was not 100% sure if the recreated target is the same as 
after a clean build. So I wrote a small check script which compares the 
original with the re-installed target trees. Here's the comparison i do:

   rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude 
'tmp/ldconfig/aux-cache' output/target/ target.orig/

As you see it already has some files excluded that are always recreated 
differently.
In addition I need to delete .stamp_host_installed from host-gcc-final* 
to force reinstall of libstdc++ into target (using external linaro 
toolchain).

Another problem that showed up was that some of the package install 
steps dont seem separated very well. E.g. qt5 examples copies everything 
from a certain staging-dir/* to target/..
A later qt5 module also creates files in that staging-dir, so next round 
there's more files in target.

I realize this is due to my way of not deleteing the staging dir any 
longer - I used to in the beginning. If I recall correctly this is 
because I noticed some packages copy files into staging/ during compile 
and these files are then missing after a reinstall. I'm not entirely 
sure though.

So my question is, in your approach, are you certain the re-installed 
staging/ is the same as the original for all packages? Maybe you can run 
some comparison like my rsync line for staging as well.


regards,
Andreas

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

On 15-04-27 04:28 AM, Andreas Naumann wrote:
>
>
> Am 27.04.2015 um 02:45 schrieb Emeric Vigier:
>> If you had some files added to the targetfs (fs-overlay, new packages,
>> ...) and you no longer need them, buildroot does not offer a simple way
>> to remove these items from the output/target directory. The rule added
>> by this commit allows you to clean the targetfs. Issuing 'make'
>> afterward will generate a new and clean targetfs.
>>
>> A section in the documentation is also added. It describes few examples
>> when developers could need it. 'make help' is also updated accordingly.
>> And full-rebuild section of the documentation is updated with content
>> and links.
>>
>
> Hi Emeric,
>
> this is very helpful, in fact i have a similar patch running for quite
> some time where I remove target/ and images/ and the corresponding
> stamps. There are some problems though and I want to share my findings
> with you.
> To begin with i was not 100% sure if the recreated target is the same as
> after a clean build. So I wrote a small check script which compares the
> original with the re-installed target trees. Here's the comparison i do:
>
>    rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude
> 'tmp/ldconfig/aux-cache' output/target/ target.orig/
>
> As you see it already has some files excluded that are always recreated
> differently.
> In addition I need to delete .stamp_host_installed from host-gcc-final*
> to force reinstall of libstdc++ into target (using external linaro
> toolchain).

In fact, the target-clean rule deletes output/staging, but that's just a 
symlink, thus the content is not erased.  My goal was to recreate the 
targetfs, not the toolchain's sysroot. But that's true that I force the 
re-installation of packages containing INSTALL_STAGING directives.

The manual says that if you change something in the toolchain, you have 
to do a full rebuild. I don't change this rule of thumb. So I don't 
believe I need to delete .stamp_host_installed here. But let me test 
with a Linaro toolchain and libstdc++.

>
> Another problem that showed up was that some of the package install
> steps dont seem separated very well. E.g. qt5 examples copies everything
> from a certain staging-dir/* to target/..

OMG, that's nasty, isn't it? Why so? Why doesn't qt copies from 
output/build to output/target?

> A later qt5 module also creates files in that staging-dir, so next round
> there's more files in target.

OMG, isn't that nastier? Isn't it an oppportunity to patch qt, rather 
than changing my current patch?

>
> I realize this is due to my way of not deleteing the staging dir any
> longer - I used to in the beginning. If I recall correctly this is
> because I noticed some packages copy files into staging/ during compile
> and these files are then missing after a reinstall. I'm not entirely
> sure though.
>
> So my question is, in your approach, are you certain the re-installed
> staging/ is the same as the original for all packages? Maybe you can run
> some comparison like my rsync line for staging as well.

That's worth a try. I'd also be interested by some test results in your 
environment if you find the time.

>
> regards,
> Andreas
>

thanks for sharing this,
-- 
Emeric

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

On 15-04-27 04:28 AM, Andreas Naumann wrote:
>
>
> Am 27.04.2015 um 02:45 schrieb Emeric Vigier:
>> If you had some files added to the targetfs (fs-overlay, new packages,
>> ...) and you no longer need them, buildroot does not offer a simple way
>> to remove these items from the output/target directory. The rule added
>> by this commit allows you to clean the targetfs. Issuing 'make'
>> afterward will generate a new and clean targetfs.
>>
>> A section in the documentation is also added. It describes few examples
>> when developers could need it. 'make help' is also updated accordingly.
>> And full-rebuild section of the documentation is updated with content
>> and links.
>>
>
> Hi Emeric,
>
> this is very helpful, in fact i have a similar patch running for quite
> some time where I remove target/ and images/ and the corresponding
> stamps. There are some problems though and I want to share my findings
> with you.
> To begin with i was not 100% sure if the recreated target is the same as
> after a clean build. So I wrote a small check script which compares the
> original with the re-installed target trees. Here's the comparison i do:
>
>    rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude
> 'tmp/ldconfig/aux-cache' output/target/ target.orig/

Hi Andreas,

Sorry for the delay (BOFH excuse #273: The cord jumped over and hit the 
power switch). I finally have some time to work on this.

I ran your rsync command on a beaglebone_defconfig and ended up with 
some differences indeed:

$ rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude 
'tmp/ldconfig/aux-cache' output/target/ output/target.orig/
sending incremental file list
deleting lib/libgcc_s.so.1
deleting lib/libgcc_s.so
deleting lib/libatomic.so.1.1.0
deleting lib/libatomic.so.1
deleting lib/libatomic.so

sent 12,704 bytes  received 165 bytes  8,579.33 bytes/sec
total size is 1,671,066  speedup is 129.85 (DRY RUN)

> As you see it already has some files excluded that are always recreated
> differently.
> In addition I need to delete .stamp_host_installed from host-gcc-final*
> to force reinstall of libstdc++ into target (using external linaro
> toolchain).

removing $(BUILD_DIR)/host-gcc-final-*/.stamp_host_installed in the 
'target-clean' recipe fixed that issue. But is this solution generic enough?

> Another problem that showed up was that some of the package install
> steps dont seem separated very well. E.g. qt5 examples copies everything
> from a certain staging-dir/* to target/..
> A later qt5 module also creates files in that staging-dir, so next round
> there's more files in target.
>
> I realize this is due to my way of not deleteing the staging dir any
> longer - I used to in the beginning. If I recall correctly this is
> because I noticed some packages copy files into staging/ during compile
> and these files are then missing after a reinstall. I'm not entirely
> sure though.

I enabled qt4 and compare the targetfs after a target-clean, no differences:

$ rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude 
'tmp/ldconfig/aux-cache' output/target/ output/target.orig.qt4/
sending incremental file list

sent 15,095 bytes  received 62 bytes  30,314.00 bytes/sec
total size is 16,028,992  speedup is 1,057.53 (DRY RUN)

With what QT configs did you meet problems specifically?

> So my question is, in your approach, are you certain the re-installed
> staging/ is the same as the original for all packages? Maybe you can run
> some comparison like my rsync line for staging as well.

Yes, I have to check that too.

>
>
> regards,
> Andreas

thanks,
--
Emeric

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Emeric Vigier]

>> Another problem that showed up was that some of the package install
>> steps dont seem separated very well. E.g. qt5 examples copies everything
>> from a certain staging-dir/* to target/..
>> A later qt5 module also creates files in that staging-dir, so next round
>> there's more files in target.
>>
>> I realize this is due to my way of not deleteing the staging dir any
>> longer - I used to in the beginning. If I recall correctly this is
>> because I noticed some packages copy files into staging/ during compile
>> and these files are then missing after a reinstall. I'm not entirely
>> sure though.
>
> I enabled qt4 and compare the targetfs after a target-clean, no
> differences:
>
> $ rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude
> 'tmp/ldconfig/aux-cache' output/target/ output/target.orig.qt4/
> sending incremental file list
>
> sent 15,095 bytes  received 62 bytes  30,314.00 bytes/sec
> total size is 16,028,992  speedup is 1,057.53 (DRY RUN)
>
> With what QT configs did you meet problems specifically?

Qt4 showed no difference in the targetfs.

I disabled Qt4 and enabled Qt5 with several modules and saw no 
differences in targetfs either. Here is my Qt5 config:

   http://pastie.org/10296840

I had to switch from a buildroot toolchain to a Linaro-gcc-4.9 toolchain 
to build Qt5. I haven't enabled Qt5-webkit because I fell into this 
GCC-4.9 bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61207.

>> So my question is, in your approach, are you certain the re-installed
>> staging/ is the same as the original for all packages? Maybe you can run
>> some comparison like my rsync line for staging as well.

I only delete staging symlink, not the sysroot content. This patch does 
not aim at fixing the black magic that some packages may be doing 
between the staging dir and the target dir, just recreating the targetfs.

So the only improvement I have in mind for this patch is this:

         rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
-         $(BUILD_DIR)/*/.stamp_target_installed 
$(BUILD_DIR)/*/.stamp_staging_installed
+         $(BUILD_DIR)/*/.stamp_target_installed 
$(BUILD_DIR)/*/.stamp_staging_installed \
+         $(BUILD_DIR)/host-gcc-final-*/.stamp_host_installed

Do you have other things in mind?

>> regards,
>> Andreas
>
> thanks,
> --
> Emeric

Emeric

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Andreas Naumann]

Hi Emeric,

nice that you pick this up again. comments below...

Am 16.07.2015 um 17:15 schrieb Emeric Vigier:
>>> Another problem that showed up was that some of the package install
>>> steps dont seem separated very well. E.g. qt5 examples copies everything
>>> from a certain staging-dir/* to target/..
>>> A later qt5 module also creates files in that staging-dir, so next round
>>> there's more files in target.
>>>
>>> I realize this is due to my way of not deleteing the staging dir any
>>> longer - I used to in the beginning. If I recall correctly this is
>>> because I noticed some packages copy files into staging/ during compile
>>> and these files are then missing after a reinstall. I'm not entirely
>>> sure though.
>>
>> I enabled qt4 and compare the targetfs after a target-clean, no
>> differences:
>>
>> $ rsync -rvncl --delete --exclude '*.pyc' --exclude '*.pyo' --exclude
>> 'tmp/ldconfig/aux-cache' output/target/ output/target.orig.qt4/
>> sending incremental file list
>>
>> sent 15,095 bytes  received 62 bytes  30,314.00 bytes/sec
>> total size is 16,028,992  speedup is 1,057.53 (DRY RUN)
>>
>> With what QT configs did you meet problems specifically?
>
> Qt4 showed no difference in the targetfs.
>
> I disabled Qt4 and enabled Qt5 with several modules and saw no
> differences in targetfs either. Here is my Qt5 config:
>
>    http://pastie.org/10296840
>
> I had to switch from a buildroot toolchain to a Linaro-gcc-4.9 toolchain
> to build Qt5. I haven't enabled Qt5-webkit because I fell into this
> GCC-4.9 bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61207.

When compiling qt5-webkit I've probably hit the same bug as you, so my 
reaction was to switch from linaro gcc 4.9 to buildroots internal 
toolchain 4.8 (a few month ago).  That way webkit compiled.

The "reinstall" problems then started when selecting 
qt5-webkit-examples. I've also had problems with qt4, but I dont recall 
in which exact configuration.

>
>>> So my question is, in your approach, are you certain the re-installed
>>> staging/ is the same as the original for all packages? Maybe you can run
>>> some comparison like my rsync line for staging as well.
>
> I only delete staging symlink, not the sysroot content. This patch does

Ok, so to me it seems there's little difference to my not touching 
staging at all. What's your intention of deleting the symlink only?

> not aim at fixing the black magic that some packages may be doing
> between the staging dir and the target dir, just recreating the targetfs.

> So the only improvement I have in mind for this patch is this:
>
>          rm -rf $(TARGET_DIR) $(STAGING_DIR) $(BUILD_DIR)/.root \
> -         $(BUILD_DIR)/*/.stamp_target_installed
> $(BUILD_DIR)/*/.stamp_staging_installed
> +         $(BUILD_DIR)/*/.stamp_target_installed
> $(BUILD_DIR)/*/.stamp_staging_installed \
> +         $(BUILD_DIR)/host-gcc-final-*/.stamp_host_installed
>
> Do you have other things in mind?

I dont know how hard it is to compile an allyesdefconfig, but that, in 
combination with the rsync check, would be a nice testcase to prove 
correctness.
However, I'm on vacation for another week, so right now cant really work 
on this.


cheers,
Andreas


>
>>> regards,
>>> Andreas
>>
>> thanks,
>> --
>> Emeric
>
> Emeric
>

[Buildroot] [PATCH v3 1/1] Makefile: add target to clean targetfs

[Arnout Vandecappelle]

On 27-04-15 01:45, Emeric Vigier wrote:
> If you had some files added to the targetfs (fs-overlay, new packages,
> ...) and you no longer need them, buildroot does not offer a simple way
> to remove these items from the output/target directory. The rule added
> by this commit allows you to clean the targetfs. Issuing 'make'
> afterward will generate a new and clean targetfs.
> 
> A section in the documentation is also added. It describes few examples
> when developers could need it. 'make help' is also updated accordingly.
> And full-rebuild section of the documentation is updated with content
> and links.
> 
> Change-Id: I42b640e845bb4195fb7b160281b16ee40883f01f
> Signed-off-by: Emeric Vigier <emeric.vigier@savoirfairelinux.com>

 Hi Emeric,

 Since there were some comments on this patch and there also seem to be cases
where it doesn't really work well, I've marked it as Changes Requested in our
patch tracking system. So unless you resubmit a new version that fixes the
comments, we will forget about your contribution.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF