[PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Richard Purdie]

This is a first attempt at logic to determine when a sstate dependency needs
to be installed and when it does not. Its a start at the logic and errs on the
side of caution, as it gets wider testing, we can refine the logic as needed.

This code should allow a significant performance speedup to certain workflows, for
example "bitbake xxx-image -c rootfs" will not populate the target sysroot.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index c085c62..470e0ee 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -620,3 +620,55 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     return ret
 
+BB_SETSCENE_DEPVALID = "setscene_depvalid"
+
+def setscene_depvalid(task, taskdependees, notneeded, d):
+    # taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME]
+    # task is included in taskdependees too
+
+    bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task])))
+
+    def isNative(x):
+        return x.endswith("-native")
+    def isNativeCross(x):
+        return x.endswith("-native") or x.endswith("-cross") or x.endswith("-cross-initial")
+    def isSafeDep(x):
+        if x in ["quilt-native", "autoconf-native", "automake-native", "gnu-config-native", "libtool-native", "pkgconfig-native", "gcc-cross", "binutils-cross"]:
+            return True
+        return False
+
+    # We can skip these "safe" dependencies since the aren't runtime dependencies, just build time
+    if isSafeDep(taskdependees[task][0]) and taskdependees[task][1] == "do_populate_sysroot":
+        return True
+
+    # We only need to trigger populate_lic through direct dependencies
+    if taskdependees[task][1] == "do_populate_lic":
+        return True
+
+    for dep in taskdependees:
+        bb.debug(2, "  considering dependency: %s" % (str(taskdependees[dep])))
+        if task == dep:
+            continue
+        if dep in notneeded:
+            continue
+        # do_package_write_* and do_package doesn't need do_package
+        if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+            continue
+        # do_package_write_* and do_package doesn't need do_populate_sysroot
+        if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+            continue
+        # Native/Cross packages don't exist and are noexec anyway
+        if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
+            continue
+        # Native/Cross populate_sysroot need their dependencies
+        if isNativeCross(taskdependees[task][0]) and isNativeCross(taskdependees[dep][0]) and taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
+            return False
+        # Target populate_sysroot do not need their dependencies
+        if taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
+            continue
+
+        # Safe fallthrough default
+        bb.debug(2, " Default setscene dependency fall through due to dependency: %s" % (str(taskdependees[dep])))
+        return False
+    return True
+

Re: [PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 4316 bytes --]

On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> This is a first attempt at logic to determine when a sstate dependency
> needs
> to be installed and when it does not. Its a start at the logic and errs on
> the
>

s/errs/errors/


> side of caution, as it gets wider testing, we can refine the logic as
> needed.
>
> This code should allow a significant performance speedup to certain
> workflows, for
> example "bitbake xxx-image -c rootfs" will not populate the target sysroot.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index c085c62..470e0ee 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -620,3 +620,55 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash,
> sq_hashfn, d):
>
>      return ret
>
> +BB_SETSCENE_DEPVALID = "setscene_depvalid"
> +
> +def setscene_depvalid(task, taskdependees, notneeded, d):
> +    # taskdependees is a dict of tasks which depend on task, each being a
> 3 item list of [PN, TASKNAME, FILENAME]
> +    # task is included in taskdependees too
> +
> +    bb.debug(2, "Considering setscene task: %s" %
> (str(taskdependees[task])))
> +
> +    def isNative(x):
> +        return x.endswith("-native")
> +    def isNativeCross(x):
> +        return x.endswith("-native") or x.endswith("-cross") or
> x.endswith("-cross-initial")
> +    def isSafeDep(x):
> +        if x in ["quilt-native", "autoconf-native", "automake-native",
> "gnu-config-native", "libtool-native", "pkgconfig-native", "gcc-cross",
> "binutils-cross"]:
> +            return True
> +        return False
> +
> +    # We can skip these "safe" dependencies since the aren't runtime
> dependencies, just build time
> +    if isSafeDep(taskdependees[task][0]) and taskdependees[task][1] ==
> "do_populate_sysroot":
> +        return True
> +
> +    # We only need to trigger populate_lic through direct dependencies
> +    if taskdependees[task][1] == "do_populate_lic":
> +        return True
> +
> +    for dep in taskdependees:
> +        bb.debug(2, "  considering dependency: %s" %
> (str(taskdependees[dep])))
> +        if task == dep:
> +            continue
> +        if dep in notneeded:
> +            continue
> +        # do_package_write_* and do_package doesn't need do_package
> +        if taskdependees[task][1] == "do_package" and
> taskdependees[dep][1] in ['do_package', 'do_package_write_deb',
> 'do_package_write_ipk', 'do_package_write_rpm']:
> +            continue
>

I understand why you does this but it would be better if we had a central
place to have the do_package_write_$pkg recorded or when adding new type
this could be forgotten.


> +        # do_package_write_* and do_package doesn't need
> do_populate_sysroot
> +        if taskdependees[task][1] == "do_populate_sysroot" and
> taskdependees[dep][1] in ['do_package', 'do_package_write_deb',
> 'do_package_write_ipk', 'do_package_write_rpm']:
> +            continue
> +        # Native/Cross packages don't exist and are noexec anyway
> +        if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1]
> in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
> +            continue
> +        # Native/Cross populate_sysroot need their dependencies
> +        if isNativeCross(taskdependees[task][0]) and
> isNativeCross(taskdependees[dep][0]) and taskdependees[task][1] ==
> 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
> +            return False
> +        # Target populate_sysroot do not need their dependencies
> +        if taskdependees[task][1] == 'do_populate_sysroot' and
> taskdependees[dep][1] == 'do_populate_sysroot':
> +            continue
> +
> +        # Safe fallthrough default
> +        bb.debug(2, " Default setscene dependency fall through due to
> dependency: %s" % (str(taskdependees[dep])))
> +        return False
> +    return True
> +
>

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

[-- Attachment #2: Type: text/html, Size: 5558 bytes --]

Re: [PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Burton, Ross]

On 16 November 2012 15:58, Otavio Salvador <otavio@ossystems.com.br> wrote:
>> Its a start at the logic and errs on the
>
> s/errs/errors/

http://www.thefreedictionary.com/errs+on+the+side

Ross

Re: [PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Richard Purdie]

On Fri, 2012-11-16 at 13:58 -0200, Otavio Salvador wrote:
> On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>         This is a first attempt at logic to determine when a sstate
>         dependency needs
>         to be installed and when it does not. Its a start at the logic
>         and errs on the
>
> s/errs/errors/

It is actually meant to be errs :)

>         +        # do_package_write_* and do_package doesn't need
>         do_package
>         +        if taskdependees[task][1] == "do_package" and
>         taskdependees[dep][1] in ['do_package',
>         'do_package_write_deb', 'do_package_write_ipk',
>         'do_package_write_rpm']:
>         +            continue
>
> I understand why you does this but it would be better if we had a
> central place to have the do_package_write_$pkg recorded or when
> adding new type this could be forgotten.

Its a nice idea but I can't see any practical way to do this at the
moment.

Cheers,

Richard

Re: [PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Fri, Nov 16, 2012 at 2:09 PM, Burton, Ross <ross.burton@intel.com> wrote:

> On 16 November 2012 15:58, Otavio Salvador <otavio@ossystems.com.br>
> wrote:
> >> Its a start at the logic and errs on the
> >
> > s/errs/errors/
>
> http://www.thefreedictionary.com/errs+on+the+side


Cool!

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

[-- Attachment #2: Type: text/html, Size: 1143 bytes --]

Re: [PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

On Fri, Nov 16, 2012 at 2:12 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2012-11-16 at 13:58 -0200, Otavio Salvador wrote:
> > On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> >         This is a first attempt at logic to determine when a sstate
> >         dependency needs
> >         to be installed and when it does not. Its a start at the logic
> >         and errs on the
> >
> > s/errs/errors/
>
> It is actually meant to be errs :)
>
> >         +        # do_package_write_* and do_package doesn't need
> >         do_package
> >         +        if taskdependees[task][1] == "do_package" and
> >         taskdependees[dep][1] in ['do_package',
> >         'do_package_write_deb', 'do_package_write_ipk',
> >         'do_package_write_rpm']:
> >         +            continue
> >
> > I understand why you does this but it would be better if we had a
> > central place to have the do_package_write_$pkg recorded or when
> > adding new type this could be forgotten.
>
> Its a nice idea but I can't see any practical way to do this at the
> moment.
>

We might hava a variable that provide all package backends names? Like:

PACKAGE_BACKENDS = "ipk deb rpm"

?

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

[-- Attachment #2: Type: text/html, Size: 2278 bytes --]

Re: [PATCH] sstate: Implement a setscene dependency validation routine to allow skipping of some sstate installation

[Richard Purdie]

On Fri, 2012-11-16 at 14:22 -0200, Otavio Salvador wrote:
> On Fri, Nov 16, 2012 at 2:12 PM, Richard Purdie 
>         >         +        # do_package_write_* and do_package doesn't
>         need
>         >         do_package
>         >         +        if taskdependees[task][1] == "do_package"
>         and
>         >         taskdependees[dep][1] in ['do_package',
>         >         'do_package_write_deb', 'do_package_write_ipk',
>         >         'do_package_write_rpm']:
>         >         +            continue
>         >
>         > I understand why you does this but it would be better if we
>         had a
>         > central place to have the do_package_write_$pkg recorded or
>         when
>         > adding new type this could be forgotten.
>
>         Its a nice idea but I can't see any practical way to do this
>         at the
>         moment.
>
> We might hava a variable that provide all package backends names?
> Like:
>
> PACKAGE_BACKENDS = "ipk deb rpm"

That works for some usages but not where the expressions are in calls to
things like addtask.

I think there are other problems worth spending time on rather that this
which at best will probably cover 50% of the usages.

Cheers,

Richard