All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC][PATCH] sstate: run recipe-provided hooks outside of ${B}
@ 2015-09-22 14:11 Ross Burton
  2015-09-22 17:30 ` Martin Jansa
  0 siblings, 1 reply; 3+ messages in thread
From: Ross Burton @ 2015-09-22 14:11 UTC (permalink / raw)
  To: openembedded-core

To avoid races between the sstate tasks/hooks using ${B} as the cwd, and other
tasks such as cmake_do_configure which deletes and re-creates ${B}, ensure that
all sstate hooks are run in the right directory, and run the prefunc/postfunc in WORKDIR.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/classes/sstate.bbclass | 26 ++++++++++----------------
 1 file changed, 10 insertions(+), 16 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index b9ad6da..695929c 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -61,16 +61,6 @@ SSTATE_SIG_PASSPHRASE ?= ""
 # Whether to verify the GnUPG signatures when extracting sstate archives
 SSTATE_VERIFY_SIG ?= "0"
 
-# Specify dirs in which the shell function is executed and don't use ${B}
-# as default dirs to avoid possible race about ${B} with other task.
-sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
-sstate_unpack_package[dirs] = "${SSTATE_INSTDIR}"
-
-# Do not run sstate_hardcode_path() in ${B}:
-# the ${B} maybe removed by cmake_do_configure() while
-# sstate_hardcode_path() running.
-sstate_hardcode_path[dirs] = "${SSTATE_BUILDDIR}"
-
 python () {
     if bb.data.inherits_class('native', d):
         d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False))
@@ -267,7 +257,7 @@ def sstate_install(ss, d):
             oe.path.copyhardlinktree(state[1], state[2])
 
     for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
-        bb.build.exec_func(postinst, d)
+        bb.build.exec_func(postinst, d, (os.getcwd(),))
 
     for lock in locks:
         bb.utils.unlockfile(lock)
@@ -307,7 +297,8 @@ def sstate_installpkg(ss, d):
             bb.warn("Cannot verify signature on sstate package %s" % sstatepkg)
 
     for f in (d.getVar('SSTATEPREINSTFUNCS', True) or '').split() + ['sstate_unpack_package'] + (d.getVar('SSTATEPOSTUNPACKFUNCS', True) or '').split():
-        bb.build.exec_func(f, d)
+        # All hooks should run in the SSTATE_INSTDIR
+        bb.build.exec_func(f, d, (sstateinst,))
 
     for state in ss['dirs']:
         prepdir(state[1])
@@ -579,8 +570,9 @@ def sstate_package(ss, d):
 
     for f in (d.getVar('SSTATECREATEFUNCS', True) or '').split() + ['sstate_create_package'] + \
              (d.getVar('SSTATEPOSTCREATEFUNCS', True) or '').split():
-        bb.build.exec_func(f, d)
-  
+        # All hooks should run in SSTATE_BUILDDIR.
+        bb.build.exec_func(f, d, (sstatebuild,))
+
     bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
 
     return
@@ -642,19 +634,21 @@ python sstate_task_prefunc () {
     shared_state = sstate_state_fromvars(d)
     sstate_clean(shared_state, d)
 }
+sstate_task_prefunc[dirs] = "${WORKDIR}"
 
 python sstate_task_postfunc () {
     shared_state = sstate_state_fromvars(d)
     sstate_install(shared_state, d)
     for intercept in shared_state['interceptfuncs']:
-        bb.build.exec_func(intercept, d)
+        bb.build.exec_func(intercept, d, (os.getcwd(),))
     omask = os.umask(002)
     if omask != 002:
        bb.note("Using umask 002 (not %0o) for sstate packaging" % omask)
     sstate_package(shared_state, d)
     os.umask(omask)
 }
-  
+sstate_task_postfunc[dirs] = "${WORKDIR}"
+
 
 #
 # Shell function to generate a sstate package from a directory
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [RFC][PATCH] sstate: run recipe-provided hooks outside of ${B}
  2015-09-22 14:11 [RFC][PATCH] sstate: run recipe-provided hooks outside of ${B} Ross Burton
@ 2015-09-22 17:30 ` Martin Jansa
  2015-09-22 20:52   ` Burton, Ross
  0 siblings, 1 reply; 3+ messages in thread
From: Martin Jansa @ 2015-09-22 17:30 UTC (permalink / raw)
  To: Ross Burton; +Cc: openembedded-core

[-- Attachment #1: Type: text/plain, Size: 4155 bytes --]

On Tue, Sep 22, 2015 at 03:11:53PM +0100, Ross Burton wrote:
> To avoid races between the sstate tasks/hooks using ${B} as the cwd, and other
> tasks such as cmake_do_configure which deletes and re-creates ${B}, ensure that
> all sstate hooks are run in the right directory, and run the prefunc/postfunc in WORKDIR.

No error in first 126 iterations of:
for i in `seq -w 1 1000`; do bitbake -c cleansstate perf; bitbake -c build -f perf 2>&1 | tee log.perf.$i; done

so it's good improvement, thanks!

Please get this merged in fido and dizzy as well.

> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>  meta/classes/sstate.bbclass | 26 ++++++++++----------------
>  1 file changed, 10 insertions(+), 16 deletions(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index b9ad6da..695929c 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -61,16 +61,6 @@ SSTATE_SIG_PASSPHRASE ?= ""
>  # Whether to verify the GnUPG signatures when extracting sstate archives
>  SSTATE_VERIFY_SIG ?= "0"
>  
> -# Specify dirs in which the shell function is executed and don't use ${B}
> -# as default dirs to avoid possible race about ${B} with other task.
> -sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
> -sstate_unpack_package[dirs] = "${SSTATE_INSTDIR}"
> -
> -# Do not run sstate_hardcode_path() in ${B}:
> -# the ${B} maybe removed by cmake_do_configure() while
> -# sstate_hardcode_path() running.
> -sstate_hardcode_path[dirs] = "${SSTATE_BUILDDIR}"
> -
>  python () {
>      if bb.data.inherits_class('native', d):
>          d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False))
> @@ -267,7 +257,7 @@ def sstate_install(ss, d):
>              oe.path.copyhardlinktree(state[1], state[2])
>  
>      for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
> -        bb.build.exec_func(postinst, d)
> +        bb.build.exec_func(postinst, d, (os.getcwd(),))
>  
>      for lock in locks:
>          bb.utils.unlockfile(lock)
> @@ -307,7 +297,8 @@ def sstate_installpkg(ss, d):
>              bb.warn("Cannot verify signature on sstate package %s" % sstatepkg)
>  
>      for f in (d.getVar('SSTATEPREINSTFUNCS', True) or '').split() + ['sstate_unpack_package'] + (d.getVar('SSTATEPOSTUNPACKFUNCS', True) or '').split():
> -        bb.build.exec_func(f, d)
> +        # All hooks should run in the SSTATE_INSTDIR
> +        bb.build.exec_func(f, d, (sstateinst,))
>  
>      for state in ss['dirs']:
>          prepdir(state[1])
> @@ -579,8 +570,9 @@ def sstate_package(ss, d):
>  
>      for f in (d.getVar('SSTATECREATEFUNCS', True) or '').split() + ['sstate_create_package'] + \
>               (d.getVar('SSTATEPOSTCREATEFUNCS', True) or '').split():
> -        bb.build.exec_func(f, d)
> -  
> +        # All hooks should run in SSTATE_BUILDDIR.
> +        bb.build.exec_func(f, d, (sstatebuild,))
> +
>      bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
>  
>      return
> @@ -642,19 +634,21 @@ python sstate_task_prefunc () {
>      shared_state = sstate_state_fromvars(d)
>      sstate_clean(shared_state, d)
>  }
> +sstate_task_prefunc[dirs] = "${WORKDIR}"
>  
>  python sstate_task_postfunc () {
>      shared_state = sstate_state_fromvars(d)
>      sstate_install(shared_state, d)
>      for intercept in shared_state['interceptfuncs']:
> -        bb.build.exec_func(intercept, d)
> +        bb.build.exec_func(intercept, d, (os.getcwd(),))
>      omask = os.umask(002)
>      if omask != 002:
>         bb.note("Using umask 002 (not %0o) for sstate packaging" % omask)
>      sstate_package(shared_state, d)
>      os.umask(omask)
>  }
> -  
> +sstate_task_postfunc[dirs] = "${WORKDIR}"
> +
>  
>  #
>  # Shell function to generate a sstate package from a directory
> -- 
> 2.1.4
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [RFC][PATCH] sstate: run recipe-provided hooks outside of ${B}
  2015-09-22 17:30 ` Martin Jansa
@ 2015-09-22 20:52   ` Burton, Ross
  0 siblings, 0 replies; 3+ messages in thread
From: Burton, Ross @ 2015-09-22 20:52 UTC (permalink / raw)
  To: Martin Jansa; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On 22 September 2015 at 18:30, Martin Jansa <martin.jansa@gmail.com> wrote:

> No error in first 126 iterations of:
> for i in `seq -w 1 1000`; do bitbake -c cleansstate perf; bitbake -c build
> -f perf 2>&1 | tee log.perf.$i; done
>
> so it's good improvement, thanks!
>
> Please get this merged in fido and dizzy as well.
>

Just found a problem in it so I'll be sending a v2, but thanks for the
testing.

Ross

[-- Attachment #2: Type: text/html, Size: 856 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2015-09-22 20:52 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-09-22 14:11 [RFC][PATCH] sstate: run recipe-provided hooks outside of ${B} Ross Burton
2015-09-22 17:30 ` Martin Jansa
2015-09-22 20:52   ` Burton, Ross

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.