[PATCH] devtool: execute associated functions while preparing the source tree

[PATCH] devtool: execute associated functions while preparing the source tree

[Jiajie Hu]

Execute prefuncs and postfuncs associated with the task while preparing
the source tree. If any changes are made to the source tree by these
prefuncs or postfuncs, a separate commit will be created so that the
changes won't get incorporated with other patches.

Signed-off-by: Jiajie Hu <jiajie.hu@intel.com>
---
 scripts/lib/devtool/standard.py | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index e4d2a57..8302112 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -424,8 +424,7 @@ class BbTaskExecutor(object):
         self.rdata = rdata
         self.executed = []
 
-    def exec_func(self, func, report):
-        """Run bitbake task function"""
+    def _exec_func(self, func, report):
         if not func in self.executed:
             deps = self.rdata.getVarFlag(func, 'deps', False)
             if deps:
@@ -435,12 +434,25 @@ class BbTaskExecutor(object):
                 logger.info('Executing %s...' % func)
             fn = self.rdata.getVar('FILE', True)
             localdata = bb.build._task_data(fn, func, self.rdata)
+            prefuncs = localdata.getVarFlag(func, 'prefuncs', True)
+            postfuncs = localdata.getVarFlag(func, 'postfuncs', True)
             try:
+                for prefunc in (prefuncs or '').split():
+                    bb.build.exec_func(prefunc, localdata)
+                    yield prefunc
                 bb.build.exec_func(func, localdata)
+                for postfunc in (postfuncs or '').split():
+                    bb.build.exec_func(postfunc, localdata)
+                    yield postfunc
             except bb.build.FuncFailed as e:
                 raise DevtoolError(str(e))
             self.executed.append(func)
 
+    def exec_func(self, func, report):
+        """Run bitbake task function"""
+        for step in self._exec_func(func, report):
+            pass
+
 
 class PatchTaskExecutor(BbTaskExecutor):
     def __init__(self, rdata):
@@ -462,7 +474,12 @@ class PatchTaskExecutor(BbTaskExecutor):
                 else:
                     os.rmdir(patchdir)
 
-        super(PatchTaskExecutor, self).exec_func(func, report)
+        for step in super(PatchTaskExecutor, self)._exec_func(func, report):
+            if self.check_git and os.path.exists(srcsubdir):
+                stdout, _ = bb.process.run('git status --porcelain', cwd=srcsubdir)
+                if stdout:
+                    bb.process.run('git add .; git %s commit -a -m "Committing changes from %s\n\n%s"' % (' '.join(self.useroptions), step, GitApplyTree.ignore_commit_prefix + ' - from %s' % step), cwd=srcsubdir)
+
         if self.check_git and os.path.exists(srcsubdir):
             if func == 'do_patch':
                 if os.path.exists(patchdir):
-- 
1.9.1

Re: [PATCH] devtool: execute associated functions while preparing the source tree

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3315 bytes --]

On Mon, Nov 28, 2016 at 7:59 PM, Jiajie Hu <jiajie.hu@intel.com> wrote:

> Execute prefuncs and postfuncs associated with the task while preparing
> the source tree. If any changes are made to the source tree by these
> prefuncs or postfuncs, a separate commit will be created so that the
> changes won't get incorporated with other patches.
>
> Signed-off-by: Jiajie Hu <jiajie.hu@intel.com>
> ---
>  scripts/lib/devtool/standard.py | 23 ++++++++++++++++++++---
>  1 file changed, 20 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/
> standard.py
> index e4d2a57..8302112 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -424,8 +424,7 @@ class BbTaskExecutor(object):
>          self.rdata = rdata
>          self.executed = []
>
> -    def exec_func(self, func, report):
> -        """Run bitbake task function"""
> +    def _exec_func(self, func, report):
>          if not func in self.executed:
>              deps = self.rdata.getVarFlag(func, 'deps', False)
>              if deps:
> @@ -435,12 +434,25 @@ class BbTaskExecutor(object):
>                  logger.info('Executing %s...' % func)
>              fn = self.rdata.getVar('FILE', True)
>              localdata = bb.build._task_data(fn, func, self.rdata)
> +            prefuncs = localdata.getVarFlag(func, 'prefuncs', True)
> +            postfuncs = localdata.getVarFlag(func, 'postfuncs', True)
>              try:
> +                for prefunc in (prefuncs or '').split():
> +                    bb.build.exec_func(prefunc, localdata)
> +                    yield prefunc
>                  bb.build.exec_func(func, localdata)
> +                for postfunc in (postfuncs or '').split():
> +                    bb.build.exec_func(postfunc, localdata)
> +                    yield postfunc
>              except bb.build.FuncFailed as e:
>                  raise DevtoolError(str(e))
>              self.executed.append(func)
>
> +    def exec_func(self, func, report):
> +        """Run bitbake task function"""
> +        for step in self._exec_func(func, report):
> +            pass
> +
>
>  class PatchTaskExecutor(BbTaskExecutor):
>      def __init__(self, rdata):
> @@ -462,7 +474,12 @@ class PatchTaskExecutor(BbTaskExecutor):
>                  else:
>                      os.rmdir(patchdir)
>
> -        super(PatchTaskExecutor, self).exec_func(func, report)
> +        for step in super(PatchTaskExecutor, self)._exec_func(func,
> report):
> +            if self.check_git and os.path.exists(srcsubdir):
> +                stdout, _ = bb.process.run('git status --porcelain',
> cwd=srcsubdir)
> +                if stdout:
> +                    bb.process.run('git add .; git %s commit -a -m
> "Committing changes from %s\n\n%s"' % (' '.join(self.useroptions), step,
> GitApplyTree.ignore_commit_prefix + ' - from %s' % step), cwd=srcsubdir)
>

This will do the commit after the first postfunc is done, so won’t the
changes from the main func and the first postfunc both be included in that
commit?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4719 bytes --]

Re: [PATCH] devtool: execute associated functions while preparing the source tree

[Jiajie Hu]

On Monday, November 28, 2016 11:04:40 PM Christopher Larson wrote:
> This will do the commit after the first postfunc is done, so won’t the
> changes from the main func and the first postfunc both be included in that
> commit?

I suppose all changes from the main func have already been committed in 
exec_func. As an example, `devtool modify x11vnc`, where x11vnc comes from 
meta-oe, seems to work as expected.

Re: [PATCH] devtool: execute associated functions while preparing the source tree

[Paul Eggleton]

Hi Jiajie,

On Tue, 29 Nov 2016 10:59:09 Jiajie Hu wrote:
> Execute prefuncs and postfuncs associated with the task while preparing
> the source tree. If any changes are made to the source tree by these
> prefuncs or postfuncs, a separate commit will be created so that the
> changes won't get incorporated with other patches.

So this in itself looks reasonable in theory, but I have to warn you that when 
we switch to tinfoil2 [1] these commits are going to be done via the metadata 
instead, and beyond a possible hack that I'd really rather not think about 
it's going to be a bit difficult to intercept the prefuncs/postfuncs there. At 
least they will be executed though since with tinfoil2 we'll be going through 
a much more standard path to run the tasks.

Cheers,
Paul

[1] https://wiki.yoctoproject.org/wiki/Tinfoil2

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] devtool: execute associated functions while preparing the source tree

[Jiajie Hu]

Hi, Paul

On Wednesday, November 30, 2016 07:01:30 AM Paul Eggleton wrote:
> So this in itself looks reasonable in theory, but I have to warn you that
> when we switch to tinfoil2 [1] these commits are going to be done via the
> metadata instead, and beyond a possible hack that I'd really rather not
> think about it's going to be a bit difficult to intercept the
> prefuncs/postfuncs there. At least they will be executed though since with
> tinfoil2 we'll be going through a much more standard path to run the tasks.
> 
> Cheers,
> Paul
> 
> [1] https://wiki.yoctoproject.org/wiki/Tinfoil2

Glad to know that my fix can be superseded by tinfoil2 APIs. Do you have any 
estimation when the switch will happen?

Re: [PATCH] devtool: execute associated functions while preparing the source tree

[Paul Eggleton]

On Wed, 30 Nov 2016 09:33:22 Jiajie Hu wrote:
> On Wednesday, November 30, 2016 07:01:30 AM Paul Eggleton wrote:
> > So this in itself looks reasonable in theory, but I have to warn you that
> > when we switch to tinfoil2 [1] these commits are going to be done via the
> > metadata instead, and beyond a possible hack that I'd really rather not
> > think about it's going to be a bit difficult to intercept the
> > prefuncs/postfuncs there. At least they will be executed though since with
> > tinfoil2 we'll be going through a much more standard path to run the
> > tasks.
> >
> > [1] https://wiki.yoctoproject.org/wiki/Tinfoil2
> 
> Glad to know that my fix can be superseded by tinfoil2 APIs. Do you have any
> estimation when the switch will happen?

The target is 2.3 milestone 1, which is in December. However there are still a 
lot of things to be fixed as I've noted on the wiki page I linked.

Thinking about it though it's possible I could split out this particular 
change and get it sent earlier - in fact that may be a good idea anyway. I'll 
look into that.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] devtool: execute associated functions while preparing the source tree

[Paul Eggleton]

On Wed, 30 Nov 2016 14:39:12 Paul Eggleton wrote:
> On Wed, 30 Nov 2016 09:33:22 Jiajie Hu wrote:
> > On Wednesday, November 30, 2016 07:01:30 AM Paul Eggleton wrote:
> > > So this in itself looks reasonable in theory, but I have to warn you
> > > that when we switch to tinfoil2 [1] these commits are going to be done
> > > via the metadata instead, and beyond a possible hack that I'd really
> > > rather not think about it's going to be a bit difficult to intercept the
> > > prefuncs/postfuncs there. At least they will be executed though since
> > > with tinfoil2 we'll be going through a much more standard path to run
> > > the tasks.
> > > 
> > > [1] https://wiki.yoctoproject.org/wiki/Tinfoil2
> > 
> > Glad to know that my fix can be superseded by tinfoil2 APIs. Do you have
> > any estimation when the switch will happen?
> 
> The target is 2.3 milestone 1, which is in December. However there are still
> a lot of things to be fixed as I've noted on the wiki page I linked.
> 
> Thinking about it though it's possible I could split out this particular
> change and get it sent earlier - in fact that may be a good idea anyway.
> I'll look into that.

Actually, I spoke too soon. Whilst moving the creation of commits out to the 
metadata can be done separately (and is already implemented in a separate 
patch), it won't actually help on its own because it doesn't change how the 
tasks are executed. The bit we need is executing the tasks as they would be 
executed normally, and unfortunately that's not possible without the full 
tinfoil2 changeset.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre