[Buildroot] [pull request v2] Pull request for branch yem/instrument-build

[Buildroot] [pull request v2] Pull request for branch yem/instrument-build

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

This series is an RFC to add support for instrumenting the build process,
reworked after discussion both on the ML and during the developpers' day.

It works as thus;
  - we register a set of hooks
  - hooks are called before and after each steps

There are currently two hooks defined:
  - log steps with timings  (as an example of hos to do a hook)
  - call a user-supplied script

This is just a RFC for now, and if it is accepted, new hooks can be
defined later on.

I'm planning on adding at least one other hook, that runs all scripts
present in support/step-hooks/ (or whatever its name).

Open for comments! ;-)

Changes v1 -> v2:
  - removed the intra-steps instrumentation  (Thomas, Peter)
  - accept a list of user-supplied hooks, not only one hook  (Thomas)

Regards,
Yann E. MORIN.


The following changes since commit 200bf74e8e4686c3f132a6600ae6803eda8b588a:

  Update config.guess and config.sub scripts to latest versions. (2013-10-27 15:29:38 +0100)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/instrument-build

for you to fetch changes up to a4b9a0c71f89f16b00a75ba0b37f19c71c0300c3:

  pkg-infra: add user-supplied step-hooks (2013-10-27 16:51:34 +0000)

----------------------------------------------------------------
Yann E. MORIN (4):
      post-{build,images} hooks: export BUILD_DIR too
      pkg-infra: introduce pre/post-step hooks
      pkg-infra: add hook to log timing of steps
      pkg-infra: add user-supplied step-hooks

 Makefile                            |  4 +--
 docs/manual/customize-rootfs.txt    |  5 ++--
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/Makefile.in                 |  5 ++++
 package/pkg-generic.mk              | 59 +++++++++++++++++++++++++++++++++++++
 6 files changed, 104 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/debugging-buildroot.txt

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Also export BUILD_DIR for post-{build,images} hooks, so they do have
a place to store generated files.

Note: this will be more einteresting for the instrumentation of steps,
to come in a later patch.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 Makefile                         | 4 ++--
 docs/manual/customize-rootfs.txt | 5 +++--
 package/Makefile.in              | 5 +++++
 3 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index f266e2d..0113f75 100644
--- a/Makefile
+++ b/Makefile
@@ -511,7 +511,7 @@ endif
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
 		$(call MESSAGE,"Executing post-build script $(s)"); \
-		$(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge
@@ -557,7 +557,7 @@ endif
 target-post-image:
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
-		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 toolchain-eclipse-register:
 	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt
index 49a6b4b..450b3d5 100644
--- a/docs/manual/customize-rootfs.txt
+++ b/docs/manual/customize-rootfs.txt
@@ -41,6 +41,7 @@ there are a few ways to customize the resulting target filesystem.
     - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
     - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
       xref:generic-package-reference[]
+    - +BUILD_DIR+: the directory where packages are extracted and built
     - +BINARIES_DIR+: the place where all binary files (aka images) are
       stored
     - +BASE_DIR+: the base output directory
@@ -78,8 +79,8 @@ in one of these _post-image scripts_ will require special handling
 
 Just like for the _post-build scripts_ mentioned above, you also have
 access to the following environment variables from your _post-image
-scripts_: +BUILDROOT_CONFIG+, +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+,
-+BINARIES_DIR+ and +BASE_DIR+.
+scripts_: +BUILDROOT_CONFIG+, +BUILD_DIR+, +HOST_DIR+, +STAGING_DIR+,
++TARGET_DIR+, +BINARIES_DIR+ and +BASE_DIR+.
 
 Additionally, each of the +BR2_ROOTFS_POST_BUILD_SCRIPT+ and
 +BR2_ROOTFS_POST_IMAGE_SCRIPT+ scripts will be passed the arguments
diff --git a/package/Makefile.in b/package/Makefile.in
index 612f3c7..2224f27 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -275,6 +275,11 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
 		PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig" \
 		PERLLIB="$(HOST_DIR)/usr/lib/perl"
 
+# This extra environment we can not export ourselves, so we have
+# to expicitly pass it to user-supplied external hooks (eg.
+# post-build, post-images)
+USER_HOOKS_EXTRA_ENV=\
+	BUILD_DIR=$(BUILD_DIR)
 
 ################################################################################
 # settings we need to pass to configure
-- 
1.8.1.2

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This hooks will let us instrument the build process in many ways:
  - log current step to see what broke
  - time each step to see what is worth optimising
  - sanity-check installed files (rpath, overwritten files...)
  - call user-provided script
  - ...

The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
Here is the list of available steps (8 total):
  - extract
  - patch
  - configure
  - build
  - install-host
  - install-staging
  - install-image
  - install-target

The download, clean and uninstall steps are not instrumented on purpose.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-generic.mk | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 4bba4b5..799555c 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -21,6 +21,29 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each steps
+################################################################################
+
+# Those two functions are called by each steps below.
+# They are responsible for calling all hooks defined in $(STEP_HOOKS)
+# and pass each of them two arguments:
+#   $1: either 'start' or 'end'
+#   $2: the name of the step
+#   $3: the name of the package
+
+# Start step
+# $1: step name
+define step_start
+	$(foreach hook,$(STEP_HOOKS),$(call $(hook),start,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+# End step
+# $1: step name
+define step_end
+	$(foreach hook,$(STEP_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
 
@@ -55,6 +78,7 @@ endif
 
 # Unpack the archive
 $(BUILD_DIR)/%/.stamp_extracted:
+	@$(call step_start,extract)
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -62,6 +86,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(Q)chmod -R +rw $(@D)
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,extract)
 
 # Rsync the source directory if the <pkg>_OVERRIDE_SRCDIR feature is
 # used.
@@ -90,6 +115,7 @@ endif
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
 $(BUILD_DIR)/%/.stamp_patched:
+	@$(call step_start,patch)
 	@$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -106,31 +132,39 @@ $(BUILD_DIR)/%/.stamp_patched:
 	)
 	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,patch)
 
 # Configure
 $(BUILD_DIR)/%/.stamp_configured:
+	@$(call step_start,configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Configuring")
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,configure)
 
 # Build
 $(BUILD_DIR)/%/.stamp_built::
+	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,build)
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
+	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-host)
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
+	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -142,16 +176,20 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-staging)
 
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
+	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-image)
 
 # Install to target dir
 $(BUILD_DIR)/%/.stamp_target_installed:
+	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -163,6 +201,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-target)
 
 # Clean package
 $(BUILD_DIR)/%/.stamp_cleaned:
-- 
1.8.1.2

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Thomas De Schampheleire]

"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
>From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
>This hooks will let us instrument the build process in many ways:
>  - log current step to see what broke
>  - time each step to see what is worth optimising
>  - sanity-check installed files (rpath, overwritten files...)
>  - call user-provided script
>  - ...
>
>The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
>Here is the list of available steps (8 total):
>  - extract
>  - patch
>  - configure
>  - build
>  - install-host
>  - install-staging
>  - install-image
>  - install-target
>
>The download, clean and uninstall steps are not instrumented on purpose.
>
>Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>---
> package/pkg-generic.mk | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
>diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>index 4bba4b5..799555c 100644
>--- a/package/pkg-generic.mk
>+++ b/package/pkg-generic.mk
>@@ -21,6 +21,29 @@
> ################################################################################
> 
> ################################################################################
>+# Helper functions to catch start/end of each steps

each step

>+################################################################################
>+
>+# Those two functions are called by each steps below.

each step

>+# They are responsible for calling all hooks defined in $(STEP_HOOKS)

I personally find the name STEP_HOOKS pretty vague, if you don't know the context you have no clue what it could be. The other hooks POST_EXTRACT etc are clear from their name.
Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

Best regards,
Thomas

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Ryan Barnett]

Thomas D, Yann,

Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote on 10/27/2013 
04:40:09 PM:

> > 
################################################################################
> > 
> > 
################################################################################
> >+# Helper functions to catch start/end of each steps
> 
> each step
> 
> 
>+################################################################################
> >+
> >+# Those two functions are called by each steps below.
> 
> each step
> 
> >+# They are responsible for calling all hooks defined in $(STEP_HOOKS)
> 
> I personally find the name STEP_HOOKS pretty vague, if you don't know 
the
> context you have no clue what it could be. The other hooks POST_EXTRACT 
etc
> are clear from their name.
>
> Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

But that is so long to type :)
 
Thanks,
-Ryan

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Yann E. MORIN]

On 2013-10-27 22:40 +0100, Thomas De Schampheleire spake thusly:
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> >From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >This hooks will let us instrument the build process in many ways:
[--SNIP--]
> >diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> >index 4bba4b5..799555c 100644
> >--- a/package/pkg-generic.mk
> >+++ b/package/pkg-generic.mk
> >@@ -21,6 +21,29 @@
> > ################################################################################
> > 
> > ################################################################################
> >+# Helper functions to catch start/end of each steps
> 
> each step
> 
> >+################################################################################
> >+
> >+# Those two functions are called by each steps below.
> 
> each step

Both: OK.

> >+# They are responsible for calling all hooks defined in $(STEP_HOOKS)
> 
> I personally find the name STEP_HOOKS pretty vague, if you don't know
> the context you have no clue what it could be. The other hooks
> POST_EXTRACT etc are clear from their name.
> Maybe, GLOBAL_INSTRUMENTATION_HOOKS?

I'm on the side of Ryan here: it is way too long. :-p
I'll try to see if I can come with a better name, though.

Thanks for the review!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Thomas De Schampheleire]

Hi Yann, Ryan,

On Tue, Oct 29, 2013 at 7:43 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> On 2013-10-27 22:40 +0100, Thomas De Schampheleire spake thusly:

>> I personally find the name STEP_HOOKS pretty vague, if you don't know
>> the context you have no clue what it could be. The other hooks
>> POST_EXTRACT etc are clear from their name.
>> Maybe, GLOBAL_INSTRUMENTATION_HOOKS?
>
> I'm on the side of Ryan here: it is way too long. :-p
> I'll try to see if I can come with a better name, though.
>

I'm fine with looking at other names, of course, but I don't see how
GLOBAL_INSTRUMENTATION_HOOKS
is longer than
FOO_POST_INSTALL_STAGING_HOOKS
FOO_POST_INSTALL_TARGET_HOOKS
or
CONNTRACK_TOOLS_PRE_CONFIGURE_HOOKS

Besides, it's not that you're writing this so regularly that the
length is really so important, IMHO.

Best regards,
Thomas

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Yann E. MORIN]

Thomas DS, All,

On 2013-10-29 21:03 +0100, Thomas De Schampheleire spake thusly:
> Hi Yann, Ryan,
> 
> On Tue, Oct 29, 2013 at 7:43 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > On 2013-10-27 22:40 +0100, Thomas De Schampheleire spake thusly:
> 
> >> I personally find the name STEP_HOOKS pretty vague, if you don't know
> >> the context you have no clue what it could be. The other hooks
> >> POST_EXTRACT etc are clear from their name.
> >> Maybe, GLOBAL_INSTRUMENTATION_HOOKS?
> >
> > I'm on the side of Ryan here: it is way too long. :-p
> > I'll try to see if I can come with a better name, though.
> >
> 
> I'm fine with looking at other names, of course, but I don't see how
> GLOBAL_INSTRUMENTATION_HOOKS
> is longer than
> FOO_POST_INSTALL_STAGING_HOOKS
> FOO_POST_INSTALL_TARGET_HOOKS
> or
> CONNTRACK_TOOLS_PRE_CONFIGURE_HOOKS
> 
> Besides, it's not that you're writing this so regularly that the
> length is really so important, IMHO.

Yeah, yeah, right. :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 3/4] pkg-infra: add hook to log timing of steps

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

The timing information is stored in the file $(O)/build-time.log

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-generic.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 799555c..72e0e82 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -43,6 +43,17 @@ define step_end
 	$(foreach hook,$(STEP_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
 endef
 
+#######################################
+# Actual steps hooks
+
+# Time steps
+define step_time
+	printf "%s:%-5.5s:%-20.20s: %s\n"           \
+	       "$$(date +%s)" "$(1)" "$(2)" "$(3)"  \
+	       >>"$(BUILD_DIR)/build-time.log"
+endef
+STEP_HOOKS += step_time
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2

[Buildroot] [PATCH 4/4] pkg-infra: add user-supplied step-hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Allow user to supply their own step-hooks by passing a variable
on the make command-line:
    make BR2_STEP_USER_HOOK=/path/to/my/script

This can be usefull to run site-specific actions at each step of the
build process, such as logging installed, removed or modified files,
do sanity checks on installed files...

It is possible to call more than one script, by passing a space-separated
lists of scripts to call.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/pkg-generic.mk              |  9 +++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 docs/manual/debugging-buildroot.txt

diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
new file mode 100644
index 0000000..7e66208
--- /dev/null
+++ b/docs/manual/debugging-buildroot.txt
@@ -0,0 +1,33 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[debugging-buildroot]]
+
+Debugging Buildroot
+-------------------
+
+It is possible to instrument the steps +Buildroot+ does when building
+packages. Define the variable +BR2_STEPS_USER_HOOKS+ to contain the path
+of one or more scripts (or other executables), in a space-separated list,
+you want called before and after each step. The script is called with
+three parameters:
+
+  - +start+ or +end+ to denote the start (resp. the end) of a step;
+  - the name of the step about to be started, or which just ended.
+  - the name of the package
+
+For example :
+
+----
+make BR2_STEPS_USER_HOOKS="/path/to/my/script1 /path/to/my/script2"
+----
+
+That script has access to the following variables:
+
+  - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
+  - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
+    xref:generic-package-reference[]
+  - +BUILD_DIR+: the directory where packages are extracted and built
+  - +BINARIES_DIR+: the place where all binary files (aka images) are
+    stored
+  - +BASE_DIR+: the base output directory
diff --git a/docs/manual/developer-guide.txt b/docs/manual/developer-guide.txt
index 43272f5..2e6b7aa 100644
--- a/docs/manual/developer-guide.txt
+++ b/docs/manual/developer-guide.txt
@@ -13,3 +13,5 @@ include::patch-policy.txt[]
 include::download-infra.txt[]
 
 include::board-support.txt[]
+
+include::debugging-buildroot.txt[]
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 72e0e82..625dabc 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -54,6 +54,15 @@ define step_time
 endef
 STEP_HOOKS += step_time
 
+# User-supplied script
+define step_user
+	@$(foreach user_hook, $(BR2_STEPS_USER_HOOKS), \
+		$(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
+endef
+ifneq ($(BR2_STEPS_USER_HOOKS),)
+STEP_HOOKS += step_user
+endif
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
-- 
1.8.1.2

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Thomas De Schampheleire]

"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
>From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
>Also export BUILD_DIR for post-{build,images} hooks, so they do have
>a place to store generated files.
>
>Note: this will be more einteresting for the instrumentation of steps,
>to come in a later patch.
>
>Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>---
> Makefile                         | 4 ++--
> docs/manual/customize-rootfs.txt | 5 +++--
> package/Makefile.in              | 5 +++++
> 3 files changed, 10 insertions(+), 4 deletions(-)
>
>diff --git a/Makefile b/Makefile
>index f266e2d..0113f75 100644
>--- a/Makefile
>+++ b/Makefile
>@@ -511,7 +511,7 @@ endif
> 
> 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
> 		$(call MESSAGE,"Executing post-build script $(s)"); \
>-		$(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>+		$(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
> 
> ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
> LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge
>@@ -557,7 +557,7 @@ endif
> target-post-image:
> 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
> 		$(call MESSAGE,"Executing post-image script $(s)"); \
>-		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>+		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
> 
> toolchain-eclipse-register:
> 	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
>diff --git a/docs/manual/customize-rootfs.txt b/docs/manual/customize-rootfs.txt
>index 49a6b4b..450b3d5 100644
>--- a/docs/manual/customize-rootfs.txt
>+++ b/docs/manual/customize-rootfs.txt
>@@ -41,6 +41,7 @@ there are a few ways to customize the resulting target filesystem.
>     - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
>     - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see
>       xref:generic-package-reference[]
>+    - +BUILD_DIR+: the directory where packages are extracted and built
>     - +BINARIES_DIR+: the place where all binary files (aka images) are
>       stored
>     - +BASE_DIR+: the base output directory
>@@ -78,8 +79,8 @@ in one of these _post-image scripts_ will require special handling
> 
> Just like for the _post-build scripts_ mentioned above, you also have
> access to the following environment variables from your _post-image
>-scripts_: +BUILDROOT_CONFIG+, +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+,
>-+BINARIES_DIR+ and +BASE_DIR+.
>+scripts_: +BUILDROOT_CONFIG+, +BUILD_DIR+, +HOST_DIR+, +STAGING_DIR+,
>++TARGET_DIR+, +BINARIES_DIR+ and +BASE_DIR+.
> 
> Additionally, each of the +BR2_ROOTFS_POST_BUILD_SCRIPT+ and
> +BR2_ROOTFS_POST_IMAGE_SCRIPT+ scripts will be passed the arguments
>diff --git a/package/Makefile.in b/package/Makefile.in
>index 612f3c7..2224f27 100644
>--- a/package/Makefile.in
>+++ b/package/Makefile.in
>@@ -275,6 +275,11 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
> 		PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig" \
> 		PERLLIB="$(HOST_DIR)/usr/lib/perl"
> 
>+# This extra environment we can not export ourselves, so we have
>+# to expicitly pass it to user-supplied external hooks (eg.
>+# post-build, post-images)
>+USER_HOOKS_EXTRA_ENV=\
>+	BUILD_DIR=$(BUILD_DIR)

I'd add a bit more info here: we cannot export BUILD_DIR globally  because some packages, like uboot, also use this same variable for another purpose.

Best regards,
Thomas

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Sun, 27 Oct 2013 22:12:34 +0100, Thomas De Schampheleire wrote:

> >+# This extra environment we can not export ourselves, so we have
> >+# to expicitly pass it to user-supplied external hooks (eg.
> >+# post-build, post-images)
> >+USER_HOOKS_EXTRA_ENV=\
> >+	BUILD_DIR=$(BUILD_DIR)
> 
> I'd add a bit more info here: we cannot export BUILD_DIR globally  because some packages, like uboot, also use this same variable for another purpose.

(Unfortunately, we've asked you to not wrapped patches you reply to,
but now, it has the consequence that you no longer wrap what you write.
Well, I guess with K9 it's hard to get both).

Regarding BUILD_DIR and other variables being exported, is it really
safe to export other variables with a relatively generic name such as
TARGET_DIR, BASE_DIR, HOST_DIR, BINARIES_DIR, etc. ? Should we instead
expose them to post-build/post-image scripts as BR2_<something> ? But
well, of course, it means breaking the existing API, which isn't nice.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Yann E. MORIN]

Thomas P, All,

On 2013-10-28 06:17 +0100, Thomas Petazzoni spake thusly:
> Dear Thomas De Schampheleire,
> 
> On Sun, 27 Oct 2013 22:12:34 +0100, Thomas De Schampheleire wrote:
> 
> > >+# This extra environment we can not export ourselves, so we have
> > >+# to expicitly pass it to user-supplied external hooks (eg.
> > >+# post-build, post-images)
> > >+USER_HOOKS_EXTRA_ENV=\
> > >+	BUILD_DIR=$(BUILD_DIR)
[--SNIP--]
> Regarding BUILD_DIR and other variables being exported, is it really
> safe to export other variables with a relatively generic name such as
> TARGET_DIR, BASE_DIR, HOST_DIR, BINARIES_DIR, etc. ? Should we instead
> expose them to post-build/post-image scripts as BR2_<something> ? But
> well, of course, it means breaking the existing API, which isn't nice.

If we export all of those variables with this mechanism rathe than with
the make's 'export', then we again dwell on the sagfe side, and we need
not rename them.

We should have named them BR2_something from the beginning, even for our
internal use, to avoid any name-clashing with the calling environment.
For example:

    $ export BUILD_DIR=/path/to/my/project/build.dir
    $ mkdir -p "${BUILD_DIR}/buildroot.output"
    $ make -C "${BUILD_DIR}//buildroot.output"

Suppose the user has post-{build,image} or other rsource files (eg.
kernel .config) in its ${BUILD_DIR}, then his/her configuration is
broken.

But that's the way story goes, we fscked up at the beginning, we have to
live with them, now... :-(

Anyway, I'll keep this change as-is, and make a new cset that export all
these variables to using this new mechanism, so they only get exported
for post-{build,image} scripts (and the new steps-user-hook).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Thomas De Schampheleire]

Hi Thomas,

On Mon, Oct 28, 2013 at 6:17 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
[..]
>
> (Unfortunately, we've asked you to not wrapped patches you reply to,
> but now, it has the consequence that you no longer wrap what you write.
> Well, I guess with K9 it's hard to get both).
>

Hmm :-o
I'm not sure if I can easily fix this with K9. My earlier bug report
and even partial patch on the k9 mailing list didn't get any response.
I'll check the source but I'll try not to write long sentences in the
mean time ;-)

Best regards,
Thomas

[Buildroot] [PATCH 1/4] post-{build, images} hooks: export BUILD_DIR too

[Yann E. MORIN]

Thomas DS, All,

On 2013-10-27 22:12 +0100, Thomas De Schampheleire spake thusly:
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> >From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> >Also export BUILD_DIR for post-{build,images} hooks, so they do have
> >a place to store generated files.
[--SNIP--]
> >+# This extra environment we can not export ourselves, so we have
> >+# to expicitly pass it to user-supplied external hooks (eg.
> >+# post-build, post-images)
> >+USER_HOOKS_EXTRA_ENV=\
> >+	BUILD_DIR=$(BUILD_DIR)
> 
> I'd add a bit more info here: we cannot export BUILD_DIR globally
> because some packages, like uboot, also use this same variable for
> another purpose.

Will do, thanks.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [pull request v3] Pull request for branch yem/instrument-build

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

This series adds support for instrumenting the build process, reworked
after discussion both on the ML and during the developpers' day.

It works as thus:
  - we register a set of hooks
  - hooks are called before and after each steps

There are currently two hooks defined:
  - log steps with timings (as an example of how to do a hook)
  - call a user-supplied script

More hooks can be added later on. Specifically, we may want to add a
script (in support/scripts or wherever) that runs some verifications
after packages are installed (in target/ and staging/), for example
to check for RPATH, or to check scripts do not have build-dir paths
in them (eg python scripts)...

I'm planning on adding at least one other hook, that runs all scripts
present in support/step-hooks/ (or whatever its name).

Changes v2 -> v3:
  - rename the internal variable that holds the hooks  (Thomas DS)
  - misc typo fixes  (Thomas DS)

Changes v1 -> v2:
  - removed the intra-steps instrumentation  (Thomas, Peter)
  - accept a list of user-supplied hooks, not only one hook  (Thomas)

Regards,
Yann E. MORIN.


The following changes since commit 13ae0075a94895c1b021d20905867ee5f354bd49:

  uboot-tools: factor out common mkimage infrastructure (2013-11-11 00:37:03 +0100)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/instrument-build

for you to fetch changes up to 1be7266b22b1c6104d2050be1b1f5e75cbf0e362:

  pkg-infra: add user-supplied step-hooks (2013-11-11 00:45:59 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
      post-{build,images} hooks: export BUILD_DIR too
      pkg-infra: introduce pre/post-step hooks
      pkg-infra: add hook to log timing of steps
      pkg-infra: add user-supplied step-hooks

 Makefile                            |  4 +--
 docs/manual/customize-rootfs.txt    |  5 ++--
 docs/manual/debugging-buildroot.txt | 33 +++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/Makefile.in                 |  6 ++++
 package/pkg-generic.mk              | 59 +++++++++++++++++++++++++++++++++++++
 6 files changed, 105 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/debugging-buildroot.txt

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This hooks will let us instrument the build process in many ways:
  - log current step to see what broke
  - time each step to see what is worth optimising
  - sanity-check installed files (rpath, overwritten files...)
  - call user-provided script
  - ...

The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
Here is the list of available steps (8 total):
  - extract
  - patch
  - configure
  - build
  - install-host
  - install-staging
  - install-image
  - install-target

The download, clean and uninstall steps are not instrumented on purpose.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 package/pkg-generic.mk | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 75fda02..02ab8a3 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -21,6 +21,29 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each step
+################################################################################
+
+# Those two functions are called by each step below.
+# They are responsible for calling all hooks defined in $(STEP_HOOKS)
+# and pass each of them two arguments:
+#   $1: either 'start' or 'end'
+#   $2: the name of the step
+#   $3: the name of the package
+
+# Start step
+# $1: step name
+define step_start
+	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),start,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+# End step
+# $1: step name
+define step_end
+	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
 
@@ -55,6 +78,7 @@ endif
 
 # Unpack the archive
 $(BUILD_DIR)/%/.stamp_extracted:
+	@$(call step_start,extract)
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -62,6 +86,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(Q)chmod -R +rw $(@D)
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,extract)
 
 # Rsync the source directory if the <pkg>_OVERRIDE_SRCDIR feature is
 # used.
@@ -91,6 +116,7 @@ endif
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
 $(BUILD_DIR)/%/.stamp_patched:
+	@$(call step_start,patch)
 	@$(call MESSAGE,"Patching")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -107,31 +133,39 @@ $(BUILD_DIR)/%/.stamp_patched:
 	)
 	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,patch)
 
 # Configure
 $(BUILD_DIR)/%/.stamp_configured:
+	@$(call step_start,configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Configuring")
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,configure)
 
 # Build
 $(BUILD_DIR)/%/.stamp_built::
+	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,build)
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
+	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-host)
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
+	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -143,16 +177,20 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-staging)
 
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
+	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-image)
 
 # Install to target dir
 $(BUILD_DIR)/%/.stamp_target_installed:
+	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -164,6 +202,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-target)
 
 # Clean package
 $(BUILD_DIR)/%/.stamp_cleaned:
-- 
1.8.1.2

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Thomas De Schampheleire]

Hi Yann,

On Mon, Nov 11, 2013 at 12:51 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> This hooks will let us instrument the build process in many ways:
>   - log current step to see what broke
>   - time each step to see what is worth optimising
>   - sanity-check installed files (rpath, overwritten files...)
>   - call user-provided script
>   - ...
>
> The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
> Here is the list of available steps (8 total):
>   - extract
>   - patch
>   - configure
>   - build
>   - install-host
>   - install-staging
>   - install-image
>   - install-target
>
> The download, clean and uninstall steps are not instrumented on purpose.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  package/pkg-generic.mk | 39 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 39 insertions(+)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 75fda02..02ab8a3 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -21,6 +21,29 @@
>  ################################################################################
>
>  ################################################################################
> +# Helper functions to catch start/end of each step
> +################################################################################
> +
> +# Those two functions are called by each step below.
> +# They are responsible for calling all hooks defined in $(STEP_HOOKS)

This comment should match the new name.

> +# and pass each of them two arguments:
> +#   $1: either 'start' or 'end'
> +#   $2: the name of the step
> +#   $3: the name of the package
> +
> +# Start step
> +# $1: step name
> +define step_start
> +       $(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),start,$(1),$($(PKG)_NAME))$(sep))

What a great name you picked ;-)

Best regards,
Thomas

[Buildroot] [pull request v4] Pull request for branch yem/instrument-build

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

This series adds support for instrumenting the build process, reworked
after discussion both on the ML and during the developpers' day.

It works as thus:
  - we register a set of hooks
  - hooks are called before and after each steps

There are currently two hooks defined:
  - log steps with timings (as an example of how to do a hook)
  - call a user-supplied scripts

More hooks can be added later on. Specifically, we may want to add a
script (in support/scripts or wherever) that runs some verifications
after packages are installed (in target/ and staging/), for example
to check for RPATH, or to check scripts do not have build-dir paths
in them (eg python scripts)...

I'm planning on adding at least one other hook, that runs all scripts
present in support/step-hooks/ (or whatever its name).

Changes v3 -> v4:
  - rename user-supplied hook  (Thomas DS)
  - typoes  (Thomas DS)

Changes v2 -> v3:
  - rename the internal variable that holds the hooks  (Thomas DS)
  - misc typo fixes  (Thomas DS)

Changes v1 -> v2:
  - removed the intra-steps instrumentation  (Thomas, Peter)
  - accept a list of user-supplied hooks, not only one hook  (Thomas)

Regards,
Yann E. MORIN.


The following changes since commit 6d93f061eccad3b81a27e8cf02cb8b29e9dba44e:

  ethtool: bump version to 3.12.1 (2013-11-11 15:06:57 +0100)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/instrument-build

for you to fetch changes up to 8e74486ebbb0c14cd011d2143b988d261f592f7a:

  pkg-infra: add user-supplied step-hooks (2013-11-11 15:59:16 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
      post-{build,images} hooks: export BUILD_DIR too
      pkg-infra: introduce pre/post-step hooks
      pkg-infra: add hook to log timing of steps
      pkg-infra: add user-supplied step-hooks

 Makefile                            |  4 +--
 docs/manual/customize-rootfs.txt    |  5 ++--
 docs/manual/debugging-buildroot.txt | 33 ++++++++++++++++++++
 docs/manual/developer-guide.txt     |  2 ++
 package/Makefile.in                 |  6 ++++
 package/pkg-generic.mk              | 60 +++++++++++++++++++++++++++++++++++++
 6 files changed, 106 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/debugging-buildroot.txt

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This hooks will let us instrument the build process in many ways:
  - log current step to see what broke
  - time each step to see what is worth optimising
  - sanity-check installed files (rpath, overwritten files...)
  - call user-provided script
  - ...

The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
Here is the list of available steps (8 total):
  - extract
  - patch
  - configure
  - build
  - install-host
  - install-staging
  - install-image
  - install-target

The download, clean and uninstall steps are not instrumented on purpose.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 75fda02..4519223 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -21,6 +21,30 @@
 ################################################################################
 
 ################################################################################
+# Helper functions to catch start/end of each step
+################################################################################
+
+# Those two functions are called by each step below.
+# They are responsible for calling all hooks defined in
+# $(GLOBAL_INSTRUMENTATION_HOOKS) and pass each of them
+# three arguments:
+#   $1: either 'start' or 'end'
+#   $2: the name of the step
+#   $3: the name of the package
+
+# Start step
+# $1: step name
+define step_start
+	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),start,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+# End step
+# $1: step name
+define step_end
+	$(foreach hook,$(GLOBAL_INSTRUMENTATION_HOOKS),$(call $(hook),end,$(1),$($(PKG)_NAME))$(sep))
+endef
+
+################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
 
@@ -55,6 +79,7 @@ endif
 
 # Unpack the archive
 $(BUILD_DIR)/%/.stamp_extracted:
+	@$(call step_start,extract)
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -62,6 +87,7 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(Q)chmod -R +rw $(@D)
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,extract)
 
 # Rsync the source directory if the <pkg>_OVERRIDE_SRCDIR feature is
 # used.
@@ -91,6 +117,7 @@ endif
 $(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
 $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS = $($(PKG)_DIR_PREFIX)/$(RAWNAME) $(call qstrip,$(BR2_GLOBAL_PATCH_DIR))/$(RAWNAME)
 $(BUILD_DIR)/%/.stamp_patched:
+	@$(call step_start,patch)
 	@$(call MESSAGE,"Patching")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(notdir $(p))$(sep))
@@ -107,31 +134,39 @@ $(BUILD_DIR)/%/.stamp_patched:
 	)
 	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,patch)
 
 # Configure
 $(BUILD_DIR)/%/.stamp_configured:
+	@$(call step_start,configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Configuring")
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,configure)
 
 # Build
 $(BUILD_DIR)/%/.stamp_built::
+	@$(call step_start,build)
 	@$(call MESSAGE,"Building")
 	$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,build)
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
+	@$(call step_start,install-host)
 	@$(call MESSAGE,"Installing to host directory")
 	$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-host)
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
+	@$(call step_start,install-staging)
 	@$(call MESSAGE,"Installing to staging directory")
 	$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
@@ -143,16 +178,20 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-staging)
 
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
+	@$(call step_start,install-image)
 	@$(call MESSAGE,"Installing to images directory")
 	$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	@$(call step_end,install-image)
 
 # Install to target dir
 $(BUILD_DIR)/%/.stamp_target_installed:
+	@$(call step_start,install-target)
 	@$(call MESSAGE,"Installing to target")
 	$(if $(BR2_INIT_SYSTEMD),\
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
@@ -164,6 +203,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
 	fi
 	$(Q)touch $@
+	@$(call step_end,install-target)
 
 # Clean package
 $(BUILD_DIR)/%/.stamp_cleaned:
-- 
1.8.1.2

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Thomas De Schampheleire]

On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> This hooks will let us instrument the build process in many ways:
>   - log current step to see what broke
>   - time each step to see what is worth optimising
>   - sanity-check installed files (rpath, overwritten files...)
>   - call user-provided script
>   - ...
>
> The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
> Here is the list of available steps (8 total):
>   - extract
>   - patch
>   - configure
>   - build
>   - install-host
>   - install-staging
>   - install-image
>   - install-target
>
> The download, clean and uninstall steps are not instrumented on purpose.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 40 insertions(+)

Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 2/4] pkg-infra: introduce pre/post-step hooks

[Samuel Martin]

2013/11/11 Thomas De Schampheleire <patrickdepinguin@gmail.com>

> On Mon, Nov 11, 2013 at 4:03 PM, Yann E. MORIN <yann.morin.1998@free.fr>
> wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > This hooks will let us instrument the build process in many ways:
> >   - log current step to see what broke
> >   - time each step to see what is worth optimising
> >   - sanity-check installed files (rpath, overwritten files...)
> >   - call user-provided script
> >   - ...
> >
> > The steps are coarse-grain, and all have a 'start' and a 'end' hooks.
> > Here is the list of available steps (8 total):
> >   - extract
> >   - patch
> >   - configure
> >   - build
> >   - install-host
> >   - install-staging
> >   - install-image
> >   - install-target
> >
> > The download, clean and uninstall steps are not instrumented on purpose.
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > ---
> >  package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 40 insertions(+)
>
> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131111/172daead/attachment.html>