[PATCH RFC] autotools: Improve configure dependency code for finding m4 files

[PATCH RFC] autotools: Improve configure dependency code for finding m4 files

[Richard Purdie]

We have an open bug about the warnings issues in builds from an sstate
cache when something like glib-2.0 gets rebuilt. The issue is that
sstate is "clever" and prunes unneeded dependencies out the tree.

For example is X depends on pkgconfig-native but we've already build X
and installed it from sstate, it will not get installed when you build Y
which depends on X.

This patch changes the logic to match the sstate behaviour and prune out
unnecessary dependencies from the scope of aclocal. This in turn removes
the warning about missing manifest files.

The issue is that this patch exposes holes in our DEPENDS in recipes,
specifically that some native tools are not listed, specifically, and
problematically, pkgconfig, gtk-doc and intltool-native in particular.

I've sent out patches against OE-Core that address the bulk of the
issues there however I'm conscious this is probably going to a bug issue
in other layers and may be too annoying to consider at this point. The
other alternative is simply to turn the warning into a debug statement.

I appreciate the code below has commented blocks, this is simply debug
I've left around for now. It will be cleaned from any final version.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass
index 0dc1e6b..34d432e 100644
--- a/meta/classes/autotools.bbclass
+++ b/meta/classes/autotools.bbclass
@@ -142,15 +142,51 @@ python autotools_copy_aclocals () {
             return
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+    #bb.warn(str(taskdepdata))
     pn = d.getVar("PN", True)
     aclocaldir = d.getVar("ACLOCALDIR", True)
     oe.path.remove(aclocaldir)
     bb.utils.mkdirhier(aclocaldir)
+    start = None
     configuredeps = []
+
     for dep in taskdepdata:
         data = taskdepdata[dep]
-        if data[1] == "do_configure" and data[0] != pn:
-            configuredeps.append(data[0])
+        if data[1] == "do_configure" and data[0] == pn:
+            start = dep
+            break
+    if not start:
+        bb.fatal("Couldn't find ourself in BB_TASKDEPDATA?")
+
+    # We need to find configure tasks which are either from <target> -> <target>
+    # or <native> -> <native> but not <target> -> <native> unless they're direct
+    # dependencies. This mirrors what would get restored from sstate.
+    done = [dep]
+    next = [dep]
+    while next:
+        new = []
+        for dep in next:
+            data = taskdepdata[dep]
+            for datadep in data[3]:
+                if datadep in done:
+                    continue
+                done.append(datadep)
+                if (not data[0].endswith("-native")) and taskdepdata[datadep][0].endswith("-native") and dep != start:
+                    continue
+                new.append(datadep)
+                if taskdepdata[datadep][1] == "do_configure":
+                    configuredeps.append(taskdepdata[datadep][0])
+        next = new
+
+    #configuredeps2 = []
+    #for dep in taskdepdata:
+    #    data = taskdepdata[dep]
+    #    if data[1] == "do_configure" and data[0] != pn:
+    #        configuredeps2.append(data[0])
+    #configuredeps.sort()
+    #configuredeps2.sort()
+    #bb.warn(str(configuredeps))
+    #bb.warn(str(configuredeps2))
 
     cp = []
     for c in configuredeps:

Re: [PATCH RFC] autotools: Improve configure dependency code for finding m4 files

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4010 bytes --]

On Mon, Jun 02, 2014 at 09:15:01PM +0100, Richard Purdie wrote:
> We have an open bug about the warnings issues in builds from an sstate
> cache when something like glib-2.0 gets rebuilt. The issue is that
> sstate is "clever" and prunes unneeded dependencies out the tree.
> 
> For example is X depends on pkgconfig-native but we've already build X
> and installed it from sstate, it will not get installed when you build Y
> which depends on X.
> 
> This patch changes the logic to match the sstate behaviour and prune out
> unnecessary dependencies from the scope of aclocal. This in turn removes
> the warning about missing manifest files.
> 
> The issue is that this patch exposes holes in our DEPENDS in recipes,
> specifically that some native tools are not listed, specifically, and
> problematically, pkgconfig, gtk-doc and intltool-native in particular.
> 
> I've sent out patches against OE-Core that address the bulk of the
> issues there however I'm conscious this is probably going to a bug issue
> in other layers and may be too annoying to consider at this point. The
> other alternative is simply to turn the warning into a debug statement.

I'm fine with new warning like this, because I would say that fixing
this has higher priority than e.g. QA warnings we have, so it's
annoying, but easy to fix.

If you wait +- a week I can do test run on my world builds and report
how many warnings it shows.

> I appreciate the code below has commented blocks, this is simply debug
> I've left around for now. It will be cleaned from any final version.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass
> index 0dc1e6b..34d432e 100644
> --- a/meta/classes/autotools.bbclass
> +++ b/meta/classes/autotools.bbclass
> @@ -142,15 +142,51 @@ python autotools_copy_aclocals () {
>              return
>  
>      taskdepdata = d.getVar("BB_TASKDEPDATA", False)
> +    #bb.warn(str(taskdepdata))
>      pn = d.getVar("PN", True)
>      aclocaldir = d.getVar("ACLOCALDIR", True)
>      oe.path.remove(aclocaldir)
>      bb.utils.mkdirhier(aclocaldir)
> +    start = None
>      configuredeps = []
> +
>      for dep in taskdepdata:
>          data = taskdepdata[dep]
> -        if data[1] == "do_configure" and data[0] != pn:
> -            configuredeps.append(data[0])
> +        if data[1] == "do_configure" and data[0] == pn:
> +            start = dep
> +            break
> +    if not start:
> +        bb.fatal("Couldn't find ourself in BB_TASKDEPDATA?")
> +
> +    # We need to find configure tasks which are either from <target> -> <target>
> +    # or <native> -> <native> but not <target> -> <native> unless they're direct
> +    # dependencies. This mirrors what would get restored from sstate.
> +    done = [dep]
> +    next = [dep]
> +    while next:
> +        new = []
> +        for dep in next:
> +            data = taskdepdata[dep]
> +            for datadep in data[3]:
> +                if datadep in done:
> +                    continue
> +                done.append(datadep)
> +                if (not data[0].endswith("-native")) and taskdepdata[datadep][0].endswith("-native") and dep != start:
> +                    continue
> +                new.append(datadep)
> +                if taskdepdata[datadep][1] == "do_configure":
> +                    configuredeps.append(taskdepdata[datadep][0])
> +        next = new
> +
> +    #configuredeps2 = []
> +    #for dep in taskdepdata:
> +    #    data = taskdepdata[dep]
> +    #    if data[1] == "do_configure" and data[0] != pn:
> +    #        configuredeps2.append(data[0])
> +    #configuredeps.sort()
> +    #configuredeps2.sort()
> +    #bb.warn(str(configuredeps))
> +    #bb.warn(str(configuredeps2))
>  
>      cp = []
>      for c in configuredeps:
> 
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH RFC] autotools: Improve configure dependency code for finding m4 files

[Richard Purdie]

On Mon, 2014-06-02 at 23:47 +0200, Martin Jansa wrote:
> On Mon, Jun 02, 2014 at 09:15:01PM +0100, Richard Purdie wrote:
> > We have an open bug about the warnings issues in builds from an sstate
> > cache when something like glib-2.0 gets rebuilt. The issue is that
> > sstate is "clever" and prunes unneeded dependencies out the tree.
> > 
> > For example is X depends on pkgconfig-native but we've already build X
> > and installed it from sstate, it will not get installed when you build Y
> > which depends on X.
> > 
> > This patch changes the logic to match the sstate behaviour and prune out
> > unnecessary dependencies from the scope of aclocal. This in turn removes
> > the warning about missing manifest files.
> > 
> > The issue is that this patch exposes holes in our DEPENDS in recipes,
> > specifically that some native tools are not listed, specifically, and
> > problematically, pkgconfig, gtk-doc and intltool-native in particular.
> > 
> > I've sent out patches against OE-Core that address the bulk of the
> > issues there however I'm conscious this is probably going to a bug issue
> > in other layers and may be too annoying to consider at this point. The
> > other alternative is simply to turn the warning into a debug statement.
> 
> I'm fine with new warning like this, because I would say that fixing
> this has higher priority than e.g. QA warnings we have, so it's
> annoying, but easy to fix.
> 
> If you wait +- a week I can do test run on my world builds and report
> how many warnings it shows.

I think the trouble is going to be that the warning only indicates a
potential problem, some of them will be valid issues and some won't be.

If you include this patch, it will remove those warnings but the cases
of missing dependencies will become hard do_configure fails when it
can't match macros like those from pkg-config.

I can't think of a manageable way of just giving warnings about true
missing dependencies :/.

There isn't any rush with this change, its just something to further
think about I guess. We could do with getting a feel for the scale of
the problem though.

On IRC you asked why your minimal from sstate builds don't show this
issue. That is a good question and one I'll look into as I'd have
expected them to.

Cheers,

Richard