[Buildroot] [PATCH 0 of 5 v2] Add find utility functions / exclude files from strip / reduce python

[Buildroot] [PATCH 0 of 5 v2] Add find utility functions / exclude files from strip / reduce python

[Thomas De Schampheleire]

This patch series adds a few utility functions to Makefile.package.in that
help in composing find commands, and uses these functions to:
* exclude user-configured files and directories from being stripped in
 target-finalize, and
* reduce a python installation to the bare minimum.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 Config.in                   |  19 +++++++++++++++++++
 Makefile                    |   9 +++++++--
 package/Makefile.package.in |  11 +++++++++++
 package/python/Config.in    |  26 ++++++++++++++++++++++----
 package/python/python.mk    |  28 +++++++++++++++++++++++++++-
 5 files changed, 86 insertions(+), 7 deletions(-)

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas De Schampheleire]

# HG changeset patch
# User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
# Date 1318518890 -7200
# Node ID b6579582d02dd802adca94e2b38972cc19cd8b76
# Parent  baa870774b887fb01a9de0cc69161f250553f896
Makefile.package.in: add utility functions find*clauses and notfirstword

This patch adds a few utility functions to Makefile.package.in.

Functions finddirclauses and findfileclauses help in building a find command
that skips a set of directories and performs operations on a set of files.
This pattern can for example be used to keep certain files or directories from
being stripped, or to remove certain files from a package installation.

The notfirstword function is the inverse of the 'firstword' function in make:
it returns all but the first word.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
v2: integrate _cont variants in main find*clauses functions. Thanks, Arnout.

 package/Makefile.package.in |  11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)
 TERM_BOLD := $(shell tput smso)
 TERM_RESET := $(shell tput rmso)
 
+# Utility functions for 'find'
+# findfileclauses: creates "-name 'X' -o -name 'Y'"
+# [1:namelist]
+findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1)))
+# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'"
+# [1:basedir, 2:namelist]
+finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))
+
+# Miscellaneous utility functions
+notfirstword=$(wordlist 2,$(words $(1)),$(1))
+
 # Download method commands
 WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET)
 SVN:=$(call qstrip,$(BR2_SVN))

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas Petazzoni]

Le Thu, 17 Nov 2011 20:41:48 +0100,
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :

> Functions finddirclauses and findfileclauses help in building a find
> command that skips a set of directories and performs operations on a
> set of files. This pattern can for example be used to keep certain
> files or directories from being stripped, or to remove certain files
> from a package installation.

I must say I am not a big fan of this. Those macros are horribly
complicated to understand and they make the code really cryptic. I
think I would really prefer to see the implementation of these "do not
strip this file/directory" option to be done without those cryptic
macros.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas De Schampheleire]

Hi Thomas,

On Thu, Nov 17, 2011 at 8:50 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Thu, 17 Nov 2011 20:41:48 +0100,
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :
>
>> Functions finddirclauses and findfileclauses help in building a find
>> command that skips a set of directories and performs operations on a
>> set of files. This pattern can for example be used to keep certain
>> files or directories from being stripped, or to remove certain files
>> from a package installation.
>
> I must say I am not a big fan of this. Those macros are horribly
> complicated to understand and they make the code really cryptic. I
> think I would really prefer to see the implementation of these "do not
> strip this file/directory" option to be done without those cryptic
> macros.

Do you mean to add the contents of the macro directly in the
target-finalize target? I fear this will not make target-finalize much
clearer.

My reasoning of putting this in a macro, is that the complexity is
just in one place. If a user can trust that the macro is right, he
doesn't have to care about it.

Also, I don't immediately see a simpler way of adding such -name
clauses to the find command, based on user input.

Thanks for your feedback,
Thomas

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Arnout Vandecappelle]

On Thursday 17 November 2011 19:41:48 Thomas De Schampheleire wrote:
> # HG changeset patch
> # User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
> # Date 1318518890 -7200
> # Node ID b6579582d02dd802adca94e2b38972cc19cd8b76
> # Parent  baa870774b887fb01a9de0cc69161f250553f896
> Makefile.package.in: add utility functions find*clauses and notfirstword

 Is it possible to tell hg to remove these lines?  git-am will include them
in the commit message...

> 
> This patch adds a few utility functions to Makefile.package.in.
> 
> Functions finddirclauses and findfileclauses help in building a find command
> that skips a set of directories and performs operations on a set of files.
> This pattern can for example be used to keep certain files or directories from
> being stripped, or to remove certain files from a package installation.
> 
> The notfirstword function is the inverse of the 'firstword' function in make:
> it returns all but the first word.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

 I think you could safely have added a
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 And to ThomasP: this helper does simplify patch 3/5 a lot.  Even though
the macros look a bit ugly indeed.

 Regards,
 Arnout

> 
> ---
> v2: integrate _cont variants in main find*clauses functions. Thanks, Arnout.
> 
>  package/Makefile.package.in |  11 +++++++++++
>  1 files changed, 11 insertions(+), 0 deletions(-)
> 
> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
> --- a/package/Makefile.package.in
> +++ b/package/Makefile.package.in
> @@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)
>  TERM_BOLD := $(shell tput smso)
>  TERM_RESET := $(shell tput rmso)
>  
> +# Utility functions for 'find'
> +# findfileclauses: creates "-name 'X' -o -name 'Y'"
> +# [1:namelist]
> +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1)))
> +# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'"
> +# [1:basedir, 2:namelist]
> +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))
> +
> +# Miscellaneous utility functions
> +notfirstword=$(wordlist 2,$(words $(1)),$(1))
> +
>  # Download method commands
>  WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET)
>  SVN:=$(call qstrip,$(BR2_SVN))
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas De Schampheleire]

On Fri, Nov 18, 2011 at 1:10 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On Thursday 17 November 2011 19:41:48 Thomas De Schampheleire wrote:
>> # HG changeset patch
>> # User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
>> # Date 1318518890 -7200
>> # Node ID b6579582d02dd802adca94e2b38972cc19cd8b76
>> # Parent ?baa870774b887fb01a9de0cc69161f250553f896
>> Makefile.package.in: add utility functions find*clauses and notfirstword
>
> ?Is it possible to tell hg to remove these lines? ?git-am will include them
> in the commit message...

Yes, absolutely, the --plain option to 'hg email' does just that. I
normally always give that option but seems I didn't this time, sorry!

>
>>
>> This patch adds a few utility functions to Makefile.package.in.
>>
>> Functions finddirclauses and findfileclauses help in building a find command
>> that skips a set of directories and performs operations on a set of files.
>> This pattern can for example be used to keep certain files or directories from
>> being stripped, or to remove certain files from a package installation.
>>
>> The notfirstword function is the inverse of the 'firstword' function in make:
>> it returns all but the first word.
>>
>> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
>
> ?I think you could safely have added a
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>
> ?And to ThomasP: this helper does simplify patch 3/5 a lot. ?Even though
> the macros look a bit ugly indeed.
>
> ?Regards,
> ?Arnout
>
>>
>> ---
>> v2: integrate _cont variants in main find*clauses functions. Thanks, Arnout.
>>
>> ?package/Makefile.package.in | ?11 +++++++++++
>> ?1 files changed, 11 insertions(+), 0 deletions(-)
>>
>> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
>> --- a/package/Makefile.package.in
>> +++ b/package/Makefile.package.in
>> @@ -87,6 +87,17 @@ MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)
>> ?TERM_BOLD := $(shell tput smso)
>> ?TERM_RESET := $(shell tput rmso)
>>
>> +# Utility functions for 'find'
>> +# findfileclauses: creates "-name 'X' -o -name 'Y'"
>> +# [1:namelist]
>> +findfileclauses=-name '$(firstword $(1))' $(patsubst %,-o -name '%',$(call notfirstword,$(1)))
>> +# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'"
>> +# [1:basedir, 2:namelist]
>> +finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))
>> +
>> +# Miscellaneous utility functions
>> +notfirstword=$(wordlist 2,$(words $(1)),$(1))
>> +
>> ?# Download method commands
>> ?WGET:=$(call qstrip,$(BR2_WGET)) $(QUIET)
>> ?SVN:=$(call qstrip,$(BR2_SVN))
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>>
>
> --
> Arnout Vandecappelle ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? arnout at mind be
> Senior Embedded Software Architect ? ? ? ? ? ? ? ? +32-16-286540
> Essensium/Mind ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium ? ? ? ? ? ? ? ?BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: ?31BB CF53 8660 6F88 345D ?54CC A836 5879 20D7 CF43
>

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Cam Hutchison]

Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:

Apologies if you have seen this, but I haven't seen my response to this
come through (probably a gmane stuffup).

>+# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'"
>+# [1:basedir, 2:namelist]
>+finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))

I think the first -wholename argument is meant to be
  '$(1)/$(firstword $(2))'

according to the comment above it.

Also, my man page for find(1) says this about -wholename:
  See -path.    This alternative is less portable than -path.

Do you find -wholename to be more readable? It's not an option I've used
much, so I need to look up the man page anyway, but -path is easier to
look up because you do not get redirected to another option like you do
with -wholename.

[Buildroot] [PATCH 1 of 5 v2] Makefile.package.in: add utility functions find*clauses and notfirstword

[Thomas De Schampheleire]

Hello Cam,

On Sat, Nov 19, 2011 at 8:13 AM, Cam Hutchison <camh@xdna.net> wrote:
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:
>
> Apologies if you have seen this, but I haven't seen my response to this
> come through (probably a gmane stuffup).
>
>>+# finddirclauses: creates "-wholename 'basedir/dirX' -o -wholename 'basedir/dirY'"
>>+# [1:basedir, 2:namelist]
>>+finddirclauses=-wholename '$(firstword $(2))' $(patsubst %,-o -wholename '$(1)/%',$(call notfirstword,$(2)))
>
> I think the first -wholename argument is meant to be
> ?'$(1)/$(firstword $(2))'
>
> according to the comment above it.

You are absolutely right. I can't believe I missed that! Thanks for noticing.

>
> Also, my man page for find(1) says this about -wholename:
> ?See -path. ? ?This alternative is less portable than -path.

That is funny. My man page redirects in the opposite way:

       -path pattern
              See -wholename.   The predicate -path is also supported
by HP-UX find.

       -wholename pattern
              File name matches shell pattern pattern.  The
metacharacters do not treat ?/? or ?.? specially; so, for example,
                        find . -wholename ?./sr*sc?
              will print an entry for a directory called ?./src/misc?
(if one exists).  To ignore a whole directory tree, use -prune
              rather than checking every file in the tree.  For
example, to skip the directory ?src/emacs? and all files and
              directories under it, and print the names of the other
files found, do something like this:
                        find . -wholename ?./src/emacs? -prune -o -print

But beware, this is CentOS 5, and probably more ancient than your system.

>
> Do you find -wholename to be more readable? It's not an option I've used
> much, so I need to look up the man page anyway, but -path is easier to
> look up because you do not get redirected to another option like you do
> with -wholename.

The main reason was because my man page uses -wholename when giving
-prune information, and I was not aware of -path. But I'll change to
-path if that's more portable.

Thanks,
Thomas

[Buildroot] [PATCH 2 of 5 v2] stripping: use findfileclauses utility function

[Thomas De Schampheleire]

# HG changeset patch
# User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
# Date 1318576710 -7200
# Node ID 26239e4561037c19ccfc7ac7ea216d36a6a8e87e
# Parent  b6579582d02dd802adca94e2b38972cc19cd8b76
stripping: use findfileclauses utility function

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 Makefile |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -454,7 +454,7 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
 	find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
 endif
-	find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \
+	find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \
 		xargs $(STRIPCMD) 2>/dev/null || true
 	find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
 		xargs -r $(KSTRIPCMD) || true

[Buildroot] [PATCH 2 of 5 v2] stripping: use findfileclauses utility function

[Arnout Vandecappelle]

On Thursday 17 November 2011 19:41:56 Thomas De Schampheleire wrote:
> # HG changeset patch
> # User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
> # Date 1318576710 -7200
> # Node ID 26239e4561037c19ccfc7ac7ea216d36a6a8e87e
> # Parent  b6579582d02dd802adca94e2b38972cc19cd8b76
> stripping: use findfileclauses utility function
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

 I think this patch is redundant: it doesn't simplify anything, nobody would
want to apply or revert it individually.

 Regards,
 Arnout


> 
> ---
>  Makefile |  2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> --- a/Makefile
> +++ b/Makefile
> @@ -454,7 +454,7 @@ endif
>  ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
>  	find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
>  endif
> -	find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \
> +	find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \
>  		xargs $(STRIPCMD) 2>/dev/null || true
>  	find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
>  		xargs -r $(KSTRIPCMD) || true
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 2 of 5 v2] stripping: use findfileclauses utility function

[Cam Hutchison]

Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:

>diff --git a/Makefile b/Makefile
>--- a/Makefile
>+++ b/Makefile
>@@ -454,7 +454,7 @@ endif
> ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
> 	find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
> endif
>-	find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \
>+	find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \

This is less readable than what was there before. The findfileclauses
macro only makes sense when it's expanding input from a variable.
Literals should stay as they are...

> 		xargs $(STRIPCMD) 2>/dev/null || true
> 	find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \

... just like this one.

[Buildroot] [PATCH 2 of 5 v2] stripping: use findfileclauses utility function

[Thomas De Schampheleire]

On Fri, Nov 18, 2011 at 2:46 AM, Cam Hutchison <camh@xdna.net> wrote:
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:
>
>>diff --git a/Makefile b/Makefile
>>--- a/Makefile
>>+++ b/Makefile
>>@@ -454,7 +454,7 @@ endif
>> ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
>> ? ? ? find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
>> endif
>>- ? ? ?find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' | \
>>+ ? ? ?find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \
>
> This is less readable than what was there before. The findfileclauses
> macro only makes sense when it's expanding input from a variable.
> Literals should stay as they are...
>
>> ? ? ? ? ? ? ? xargs $(STRIPCMD) 2>/dev/null || true
>> ? ? ? find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
>
> ... just like this one.

The problem arises when you combine such a literal with files from a
variable: the variable may be empty or not, and then you don't know
whether or not to add '-o'.
If you do the following:
find $(TARGET_DIR) -type f -perm +111 '!' -name 'libthread_db*.so*' -o
$(call findfileclauses,$(BR2_EXCLUDE_FILES))
while BR2_EXCLUDE_FILES is empty, the syntax is incorrect. Changing
the order doesn't help:
find $(TARGET_DIR) -type f -perm +111 '!' $(call
findfileclauses,$(BR2_EXCLUDE_FILES)) -o -name 'libthread_db*.so*'

I agree that this patch on its own is not good. However, I think it'll
be necessary in combination with the BR2_EXCLUDE_FILES patch.

Best regards,
Thomas

[Buildroot] [PATCH 3 of 5 v2] build: add option to exclude executables/dirs from being stripped

[Thomas De Schampheleire]

# HG changeset patch
# User Sven Neumann <s.neumann@raumfeld.com>
# Date 1314025139 -7200
# Node ID 76f1a5eeba5e79cb841338d5317e3c618bf7c93c
# Parent  26239e4561037c19ccfc7ac7ea216d36a6a8e87e
build: add option to exclude executables/dirs from being stripped

Sometimes it may be desirable to keep debug symbols for some binaries and
libraries on the target. This commit introduces the config option
BR2_STRIP_EXCLUDE_FILES, which is interpreted as a list of such binaries
and libraries, and the option BR2_STRIP_EXCLUDE_DIRS, which indicates
directories excluded from stripping entirely.
These exclusions are passed to the find command in the target-finalize step.

Signed-off-by: Sven Neumann <s.neumann@raumfeld.com>
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
v2: add the possibility to exclude entire directories

 Config.in |  19 +++++++++++++++++++
 Makefile  |   9 +++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/Config.in b/Config.in
--- a/Config.in
+++ b/Config.in
@@ -247,6 +247,25 @@ config BR2_STRIP_none
 	  none    do not strip (only for debugging!)
 endchoice
 
+config BR2_STRIP_EXCLUDE_FILES
+	string "executables that should not be stripped"
+	depends on !BR2_STRIP_none
+	default ""
+	help
+	  You may specify a space-separated list of binaries and libraries
+	  here that should not be stripped on the target.
+
+config BR2_STRIP_EXCLUDE_DIRS
+	string "directories that should be skipped when stripping"
+	depends on !BR2_STRIP_none
+	default ""
+	help
+	  You may specify a space-separated list of directories that should
+	  be skipped when stripping. Binaries and libraries in these
+	  directories will not be touched.
+	  The directories should be specified relative to the target directory,
+	  without leading slash.
+
 choice
 	prompt "gcc optimization level"
 	default BR2_OPTIMIZE_S
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -430,6 +430,12 @@ endif
 erase-fakeroots:
 	rm -f $(BUILD_DIR)/.fakeroot*
 
+STRIP_FIND_CMD=find $(TARGET_DIR)
+ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS)))
+STRIP_FIND_CMD+=\( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o
+endif
+STRIP_FIND_CMD+=-type f -perm +111 -not \( $(call findfileclauses,libthread_db*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
+
 target-finalize:
 ifeq ($(BR2_HAVE_DEVFILES),y)
 	( support/scripts/copy.sh $(STAGING_DIR) $(TARGET_DIR) )
@@ -454,8 +460,7 @@ endif
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
 	find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
 endif
-	find $(TARGET_DIR) -type f -perm +111 '!' $(call findfileclauses,libthread_db*.so*) | \
-		xargs $(STRIPCMD) 2>/dev/null || true
+	$(STRIP_FIND_CMD) | xargs $(STRIPCMD) 2>/dev/null || true
 	find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
 		xargs -r $(KSTRIPCMD) || true
 

[Buildroot] [PATCH 4 of 5 v2] python config: move configuration into menu

[Thomas De Schampheleire]

# HG changeset patch
# User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
# Date 1318517686 -7200
# Node ID 97f99d539f586caa866009c9702596c63f3637fe
# Parent  76f1a5eeba5e79cb841338d5317e3c618bf7c93c
python config: move configuration into menu

Group all Python-related configuration in a separate menuconfig.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 package/python/Config.in |  8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/package/python/Config.in b/package/python/Config.in
--- a/package/python/Config.in
+++ b/package/python/Config.in
@@ -1,4 +1,7 @@
-config BR2_PACKAGE_PYTHON
+comment "python requires a toolchain with WCHAR support"
+	depends on !BR2_USE_WCHAR
+
+menuconfig BR2_PACKAGE_PYTHON
 	bool "python"
 	depends on BR2_USE_WCHAR
 	select BR2_PACKAGE_LIBFFI
@@ -7,9 +10,6 @@ config BR2_PACKAGE_PYTHON
 
 	  http://www.python.org/
 
-comment "python requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON
 
 choice

[Buildroot] [PATCH 5 of 5 v2] python: allow reduced installation

[Thomas De Schampheleire]

# HG changeset patch
# User Thomas De Schampheleire <thomas.de_schampheleire@alcatel-lucent.com>
# Date 1318517171 -7200
# Node ID 3fb04c960d0e58ea5bb10ecbed382fc1c41c39eb
# Parent  97f99d539f586caa866009c9702596c63f3637fe
python: allow reduced installation

A default python installation takes more than 13M of disk space on target.
The python library takes about 4,5M, and the other 9M is occupied by various
modules in /usr/lib/python2.7. A usable installation doesn't need such a large
module directory. A minimal configuration with only those modules needed to
start an interactive session requires only about 450K, plus the python
library itself.

This patch adds an option to reduce the number of installed python modules to
the bare minimum. Additional modules to install can be specified in the config
file as a string.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
As an example, to be able to run the 'dstat' tool, I had to add the following
extra modules:
BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES="__future__.pyc sched.pyc heapq.pyc
bisect.pyc collections.pyc keyword.pyc getopt.pyc fnmatch.pyc getpass.pyc
glob.pyc time.so itertools.so operator.so resource.so _collections.so"

 package/python/Config.in |  18 ++++++++++++++++++
 package/python/python.mk |  28 +++++++++++++++++++++++++++-
 2 files changed, 45 insertions(+), 1 deletions(-)

diff --git a/package/python/Config.in b/package/python/Config.in
--- a/package/python/Config.in
+++ b/package/python/Config.in
@@ -95,4 +95,22 @@ config BR2_PACKAGE_PYTHON_ZLIB
 
 endmenu
 
+
+config BR2_PACKAGE_PYTHON_REDUCED
+	bool "reduced python installation"
+	help
+	  Select this option to greatly reduce the number of python modules that are
+	  installed on target (from approx. 9M to 400K). Only the modules needed to
+	  start an interactive python session are installed by default.
+	  You can install additional modules by specifying them in
+	  BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES.
+
+config BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES
+	string "extra modules to include in reduced python installation"
+	depends on BR2_PACKAGE_PYTHON_REDUCED
+	help
+	  To install additional modules in a reduced Python installation,
+	  specify them here, space-separated. This list should only include the
+	  file name, not the directory part.
+
 endif
diff --git a/package/python/python.mk b/package/python/python.mk
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -144,12 +144,14 @@ ifneq ($(BR2_HAVE_DEVFILES),y)
 PYTHON_POST_INSTALL_TARGET_HOOKS += PYTHON_REMOVE_DEVFILES
 endif
 
+PYTHON_TARGET_DIR=$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+
 #
 # Remove useless files. In the config/ directory, only the Makefile
 # and the pyconfig.h files are needed at runtime.
 #
 define PYTHON_REMOVE_USELESS_FILES
-	for i in `find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/config/ \
+	for i in `find $(PYTHON_TARGET_DIR)/config/ \
 		-type f -not -name pyconfig.h -a -not -name Makefile` ; do \
 		rm -f $$i ; \
 	done
@@ -157,6 +159,30 @@ endef
 
 PYTHON_POST_INSTALL_TARGET_HOOKS += PYTHON_REMOVE_USELESS_FILES
 
+#
+# Reduced Python installation: remove unneeded modules
+#
+ifeq ($(BR2_PACKAGE_PYTHON_REDUCED),y)
+PYTHON_REDUCED_MINIMAL_MODULES=$(patsubst %,%.pyc,\
+	site os stat posixpath genericpath warnings linecache types UserDict \
+	_abcoll abc _weakrefset copy_reg sysconfig re sre_compile sre_parse \
+	sre_constants) \
+	$(call qstrip,$(BR2_PACKAGE_PYTHON_REDUCED_EXTRA_MODULES))
+PYTHON_REDUCED_MINIMAL_DIRS=config
+
+PYTHON_REDUCED_FIND_CMD=find $(PYTHON_TARGET_DIR)
+PYTHON_REDUCED_FIND_CMD+=\( $(call finddirclauses,$(PYTHON_TARGET_DIR),$(PYTHON_REDUCED_MINIMAL_DIRS)) \) -prune -o
+PYTHON_REDUCED_FIND_CMD += -type f -not \( $(call findfileclauses,$(PYTHON_REDUCED_MINIMAL_MODULES)) \) -print
+
+define PYTHON_REDUCE_INSTALLATION
+	for i in `$(PYTHON_REDUCED_FIND_CMD)`; do \
+		rm -f $$i ; \
+	done
+endef
+
+PYTHON_POST_INSTALL_TARGET_HOOKS += PYTHON_REDUCE_INSTALLATION
+endif
+
 PYTHON_AUTORECONF = YES
 
 $(eval $(call AUTOTARGETS))

[Buildroot] [PATCH 5 of 5 v2] python: allow reduced installation

[Thomas Petazzoni]

Le Thu, 17 Nov 2011 20:42:04 +0100,
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :

> A default python installation takes more than 13M of disk space on
> target. The python library takes about 4,5M, and the other 9M is
> occupied by various modules in /usr/lib/python2.7. A usable
> installation doesn't need such a large module directory. A minimal
> configuration with only those modules needed to start an interactive
> session requires only about 450K, plus the python library itself.
> 
> This patch adds an option to reduce the number of installed python
> modules to the bare minimum. Additional modules to install can be
> specified in the config file as a string.

While I agree that the feature is nice, I am not a fan of the
implementation. Currently, we already have a way of enabling/disabling
certain Python modules with one Kconfig option for each module that can
be enabled/disabled. The list of modules that can be enabled/disabled
this way has been built taking into account two factors :

 * We want to be able to configure out modules that have external
   dependencies (on other libraries)

 * We want to be able to configure certain modules that take a huge
   amount of space (I'm thinking of all the japanese/korean handling)
   and may not be useful in most situations

So we already have a way of configuring which modules are
built/installed with one Kconfig option per-module.

Now, you're adding a new option "I want a minimal version of Python"
with an additional option "but I also want those additional modules". I
think that taken together with the existing per-module Kconfig options,
it makes the Python configuration very strange. I am not sure what to
propose exactly (having one Kconfig option for each and every module
would be horrible), but I don't like really much this proposal.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 5 of 5 v2] python: allow reduced installation

[Arnout Vandecappelle]

On Thursday 17 November 2011 19:55:01 Thomas Petazzoni wrote:
> Le Thu, 17 Nov 2011 20:42:04 +0100,
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :
> 
> > A default python installation takes more than 13M of disk space on
> > target. The python library takes about 4,5M, and the other 9M is
> > occupied by various modules in /usr/lib/python2.7. A usable
> > installation doesn't need such a large module directory. A minimal
> > configuration with only those modules needed to start an interactive
> > session requires only about 450K, plus the python library itself.
> > 
> > This patch adds an option to reduce the number of installed python
> > modules to the bare minimum. Additional modules to install can be
> > specified in the config file as a string.
> 
> While I agree that the feature is nice, I am not a fan of the
> implementation. Currently, we already have a way of enabling/disabling
> certain Python modules with one Kconfig option for each module that can
> be enabled/disabled. The list of modules that can be enabled/disabled
> this way has been built taking into account two factors :
> 
>  * We want to be able to configure out modules that have external
>    dependencies (on other libraries)
> 
>  * We want to be able to configure certain modules that take a huge
>    amount of space (I'm thinking of all the japanese/korean handling)
>    and may not be useful in most situations
> 
> So we already have a way of configuring which modules are
> built/installed with one Kconfig option per-module.
> 
> Now, you're adding a new option "I want a minimal version of Python"
> with an additional option "but I also want those additional modules". I
> think that taken together with the existing per-module Kconfig options,
> it makes the Python configuration very strange. I am not sure what to
> propose exactly (having one Kconfig option for each and every module
> would be horrible), but I don't like really much this proposal.

 Maybe a middle way is to have the reduced setup by default, with a
BR2_PACKAGE_PYTHON_EXTRA_MODULES option and another option to enable
all 'usual' modules (which would disable the EXTRA_MODULES option).
At a later stage, more options for 'usual' modules can be added.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 5 of 5 v2] python: allow reduced installation

[Thomas De Schampheleire]

Hi Thomas,

On Thu, Nov 17, 2011 at 8:55 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Thu, 17 Nov 2011 20:42:04 +0100,
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :
>
>> A default python installation takes more than 13M of disk space on
>> target. The python library takes about 4,5M, and the other 9M is
>> occupied by various modules in /usr/lib/python2.7. A usable
>> installation doesn't need such a large module directory. A minimal
>> configuration with only those modules needed to start an interactive
>> session requires only about 450K, plus the python library itself.
>>
>> This patch adds an option to reduce the number of installed python
>> modules to the bare minimum. Additional modules to install can be
>> specified in the config file as a string.
>
> While I agree that the feature is nice, I am not a fan of the
> implementation. Currently, we already have a way of enabling/disabling
> certain Python modules with one Kconfig option for each module that can
> be enabled/disabled. The list of modules that can be enabled/disabled
> this way has been built taking into account two factors :
>
> ?* We want to be able to configure out modules that have external
> ? dependencies (on other libraries)
>
> ?* We want to be able to configure certain modules that take a huge
> ? amount of space (I'm thinking of all the japanese/korean handling)
> ? and may not be useful in most situations
>
> So we already have a way of configuring which modules are
> built/installed with one Kconfig option per-module.
>
> Now, you're adding a new option "I want a minimal version of Python"
> with an additional option "but I also want those additional modules". I
> think that taken together with the existing per-module Kconfig options,
> it makes the Python configuration very strange. I am not sure what to
> propose exactly (having one Kconfig option for each and every module
> would be horrible), but I don't like really much this proposal.

Ok, it's true that this makes the Python configuration pretty strange.
We need to improve this.

Having a separate option for each module is indeed way overkill / undesirable.
Having a single string option to specify what you want or what you
don't want is also not very user-friendly, certainly not in case of
dependencies.

The 'reduced python' option I proposed is really an expert option, as
you have to determine which additional modules you need for a given
script. But, in cases where space is limited, I think it is of value.

An alternative would be to identify a limited number of module
'groups', each having a Kconfig option, but I'm not sure whether this
is at all possible.

What if we split the configuration in two ways?
* Either you start from the almost-full installation, with the option
of deselecting certain packages (this corresponds to the current
situation)
* Or you tick the 'expert' option, and then you start from the reduced
installation, with the option of enabling certain packages with a
string option.
The main difference between this and what is proposed in the patch, is
that the two ways of configuring become mutually exclusive which makes
it clearer.

Best regards,
Thomas