Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Richard Purdie
> Sent: den 23 januari 2017 23:38
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH] cooker: Drop package-depends.dot and
> pn-depends.dot generation
> 
> A long time ago when we switched to task basked execution we added
> task-depends.dot and generated package-depends.dot and pn-depends.dot
> for compatibility as best we could.	
> 
> The problem is they contain partial data about the taskgraph, its
> incomplete and tends to confuse users.
> 
> I propose we remove the two compatibilty outputs and just generate
> the one which contains definitive data.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Is there any alternative to removing package-depends.dot and 
especially pn-depends.dot? The reason I ask is because we use 
pn-dependes.dot when visualizing the dependencies between packages. 
Even though task-depends.dot is more complete, trying to render it 
is near impossible given the sheer number of nodes and edges it 
contains. Already visualizing pn-depends.dot is hard but possible 
with some gvpr and tred filtering. Here are some statistics from 
one of my typical builds:

                        Nodes  Edges
                        -----  ------
pn-depends.dot           1183   12087
package-depends.dot      5939   55615
task-depends.dot        13756  125870

As can be seen, the number of nodes and edges in task-depends.dot 
is a magnitude greater than in pn-depends.dot.

//Peter

Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Richard Purdie]

On Wed, 2017-01-25 at 11:05 +0000, Peter Kjellerstedt wrote:
> > 
> > -----Original Message-----
> > From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> > devel-bounces@lists.openembedded.org] On Behalf Of Richard Purdie
> > Sent: den 23 januari 2017 23:38
> > To: bitbake-devel@lists.openembedded.org
> > Subject: [bitbake-devel] [PATCH] cooker: Drop package-depends.dot
> > and
> > pn-depends.dot generation
> > 
> > A long time ago when we switched to task basked execution we added
> > task-depends.dot and generated package-depends.dot and pn-
> > depends.dot
> > for compatibility as best we could.	
> > 
> > The problem is they contain partial data about the taskgraph, its
> > incomplete and tends to confuse users.
> > 
> > I propose we remove the two compatibilty outputs and just generate
> > the one which contains definitive data.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> Is there any alternative to removing package-depends.dot and 
> especially pn-depends.dot?

We don't have to remove them but what worries me is that the data in
them is incomplete. Its some subset of what the real task graph looks
like and at some point someone is going to complain the data was
inaccurate/misleading (which it is).

>  The reason I ask is because we use 
> pn-dependes.dot when visualizing the dependencies between packages. 
> Even though task-depends.dot is more complete, trying to render it 
> is near impossible given the sheer number of nodes and edges it 
> contains. Already visualizing pn-depends.dot is hard but possible 
> with some gvpr and tred filtering. Here are some statistics from 
> one of my typical builds:
> 
>                         Nodes  Edges
>                         -----  ------
> pn-depends.dot           1183   12087
> package-depends.dot      5939   55615
> task-depends.dot        13756  125870
> 
> As can be seen, the number of nodes and edges in task-depends.dot 
> is a magnitude greater than in pn-depends.dot.

Its certainly simpler, but the data is just plain buggy. I'm not sure
I'd trust anything those files told me, even if they are easier to
view.

I appreciate task-depends is hard to load graphically, it is helpful
even as a text file though, I do use it a lot since its a near direct
dump of bitbake's internal parsed task dependencies.

Cheers,

Richard

Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Paul Eggleton]

On Wednesday, 25 January 2017 11:12:57 AM NZDT Richard Purdie wrote:
> On Wed, 2017-01-25 at 11:05 +0000, Peter Kjellerstedt wrote:
> > pn-dependes.dot when visualizing the dependencies between packages. 
> > Even though task-depends.dot is more complete, trying to render it 
> > is near impossible given the sheer number of nodes and edges it 
> > contains. Already visualizing pn-depends.dot is hard but possible 
> > with some gvpr and tred filtering. Here are some statistics from 
> > one of my typical builds:
> > 
> >                         Nodes  Edges
> >                         -----  ------
> > pn-depends.dot           1183   12087
> > package-depends.dot      5939   55615
> > task-depends.dot        13756  125870
> > 
> > As can be seen, the number of nodes and edges in task-depends.dot 
> > is a magnitude greater than in pn-depends.dot.
> 
> Its certainly simpler, but the data is just plain buggy. I'm not sure
> I'd trust anything those files told me, even if they are easier to
> view.
> 
> I appreciate task-depends is hard to load graphically, it is helpful
> even as a text file though, I do use it a lot since its a near direct
> dump of bitbake's internal parsed task dependencies.

What if we collapsed down task-depends.dot to PNs only in order to produce 
pn-depends.dot? Then at least it would be reasonably accurate.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Peter Kjellerstedt]

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: den 25 januari 2017 20:17
> To: Richard Purdie
> Cc: bitbake-devel@lists.openembedded.org; Peter Kjellerstedt
> Subject: Re: [bitbake-devel] [PATCH] cooker: Drop package-depends.dot
> and pn-depends.dot generation
> 
> On Wednesday, 25 January 2017 11:12:57 AM NZDT Richard Purdie wrote:
> > On Wed, 2017-01-25 at 11:05 +0000, Peter Kjellerstedt wrote:
> > > pn-dependes.dot when visualizing the dependencies between packages.
> > > Even though task-depends.dot is more complete, trying to render it
> > > is near impossible given the sheer number of nodes and edges it
> > > contains. Already visualizing pn-depends.dot is hard but possible
> > > with some gvpr and tred filtering. Here are some statistics from
> > > one of my typical builds:
> > >
> > >                         Nodes  Edges
> > >                         -----  ------
> > > pn-depends.dot           1183   12087
> > > package-depends.dot      5939   55615
> > > task-depends.dot        13756  125870
> > >
> > > As can be seen, the number of nodes and edges in task-depends.dot
> > > is a magnitude greater than in pn-depends.dot.
> >
> > Its certainly simpler, but the data is just plain buggy. I'm not sure
> > I'd trust anything those files told me, even if they are easier to
> > view.
> >
> > I appreciate task-depends is hard to load graphically, it is helpful
> > even as a text file though, I do use it a lot since its a near direct
> > dump of bitbake's internal parsed task dependencies.
> 
> What if we collapsed down task-depends.dot to PNs only in order to
> produce pn-depends.dot? Then at least it would be reasonably accurate.
> 
> Cheers,
> Paul

That would work for me at least.

//Peter

Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Peter Kjellerstedt
> Sent: den 30 januari 2017 11:19
> To: Paul Eggleton; Richard Purdie
> Cc: bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH] cooker: Drop package-depends.dot
> and pn-depends.dot generation
> 
> > -----Original Message-----
> > From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> > Sent: den 25 januari 2017 20:17
> > To: Richard Purdie
> > Cc: bitbake-devel@lists.openembedded.org; Peter Kjellerstedt
> > Subject: Re: [bitbake-devel] [PATCH] cooker: Drop package-depends.dot
> > and pn-depends.dot generation
> >
> > On Wednesday, 25 January 2017 11:12:57 AM NZDT Richard Purdie wrote:
> > > On Wed, 2017-01-25 at 11:05 +0000, Peter Kjellerstedt wrote:
> > > > pn-dependes.dot when visualizing the dependencies between
> > > > packages. Even though task-depends.dot is more complete, trying 
> > > > to render it is near impossible given the sheer number of nodes 
> > > > and edges it contains. Already visualizing pn-depends.dot is hard 
> > > > but possible with some gvpr and tred filtering. Here are some 
> > > > statistics from one of my typical builds:
> > > >
> > > >                         Nodes  Edges
> > > >                         -----  ------
> > > > pn-depends.dot           1183   12087
> > > > package-depends.dot      5939   55615
> > > > task-depends.dot        13756  125870
> > > >
> > > > As can be seen, the number of nodes and edges in task-depends.dot
> > > > is a magnitude greater than in pn-depends.dot.
> > >
> > > Its certainly simpler, but the data is just plain buggy. I'm not
> > > sure I'd trust anything those files told me, even if they are 
> > > easier to view.
> > >
> > > I appreciate task-depends is hard to load graphically, it is
> > > helpful even as a text file though, I do use it a lot since 
> > > its a near direct dump of bitbake's internal parsed task 
> > > dependencies.
> >
> > What if we collapsed down task-depends.dot to PNs only in order to
> > produce pn-depends.dot? Then at least it would be reasonably
> > accurate.
> >
> > Cheers,
> > Paul
> 
> That would work for me at least.
> 
> //Peter

Another concern here is the depexp UI which I believe currently uses 
the same data that ends up in pn-depends.dot. (I noticed this the 
other day when I could not find a dependency in the depexp UI that I 
expected should be there, and based on the discussion in this mail 
thread then searched in task-depends.dot and found the dependency 
there as expected).

//Peter

Re: [PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Tobias Hagelborn]

On 01/25/2017 08:17 PM, Paul Eggleton wrote:
> On Wednesday, 25 January 2017 11:12:57 AM NZDT Richard Purdie wrote:
>> On Wed, 2017-01-25 at 11:05 +0000, Peter Kjellerstedt wrote:
>>> pn-dependes.dot when visualizing the dependencies between packages.
>>> Even though task-depends.dot is more complete, trying to render it
>>> is near impossible given the sheer number of nodes and edges it
>>> contains. Already visualizing pn-depends.dot is hard but possible
>>> with some gvpr and tred filtering. Here are some statistics from
>>> one of my typical builds:
>>>
>>>                         Nodes  Edges
>>>                         -----  ------
>>> pn-depends.dot           1183   12087
>>> package-depends.dot      5939   55615
>>> task-depends.dot        13756  125870
>>>
>>> As can be seen, the number of nodes and edges in task-depends.dot
>>> is a magnitude greater than in pn-depends.dot.
>>
>> Its certainly simpler, but the data is just plain buggy. I'm not sure
>> I'd trust anything those files told me, even if they are easier to
>> view.
>>
>> I appreciate task-depends is hard to load graphically, it is helpful
>> even as a text file though, I do use it a lot since its a near direct
>> dump of bitbake's internal parsed task dependencies.
>
> What if we collapsed down task-depends.dot to PNs only in order to produce
> pn-depends.dot? Then at least it would be reasonably accurate.
>
> Cheers,
> Paul
>

Hi,

It is a great initiative to make the dependency graphs more correct.
I just wanted to note that I have found very good use of the 
pn-buildlist as a measure to force a certain task on, for instance, all 
recipe dependencies in an image or other recipe. It is sort of a good 
input for a "for each" cmdline scripts.

ex: xargs -a pn-buildlist bitbake -c mytask

This can of cause be done by filtering stuff out from the tasks-depends 
or pn-depends but there is a risk of parse error trying to separate task 
name from packet name if not following conventions.

So, perhaps you are aware of another preferred way to achieve this or if 
you could consider still providing the pn-buildlist (although correct 
and derived from task-depends)?

Cheers
Tobias

[PATCH] cooker: Drop package-depends.dot and pn-depends.dot generation

[Richard Purdie]

A long time ago when we switched to task basked execution we added
task-depends.dot and generated package-depends.dot and pn-depends.dot
for compatibility as best we could.

The problem is they contain partial data about the taskgraph, its
incomplete and tends to confuse users.

I propose we remove the two compatibilty outputs and just generate
the one which contains definitive data.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 43 -------------------------------------------
 1 file changed, 43 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 80007c5..e654a60 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -949,49 +949,6 @@ class BBCooker:
 
         depgraph = self.generateTaskDepTreeData(pkgs_to_build, task)
 
-        # Prints a flattened form of package-depends below where subpackages of a package are merged into the main pn
-        depends_file = open('pn-depends.dot', 'w' )
-        buildlist_file = open('pn-buildlist', 'w' )
-        print("digraph depends {", file=depends_file)
-        for pn in depgraph["pn"]:
-            fn = depgraph["pn"][pn]["filename"]
-            version = depgraph["pn"][pn]["version"]
-            print('"%s" [label="%s %s\\n%s"]' % (pn, pn, version, fn), file=depends_file)
-            print("%s" % pn, file=buildlist_file)
-        buildlist_file.close()
-        logger.info("PN build list saved to 'pn-buildlist'")
-        for pn in depgraph["depends"]:
-            for depend in depgraph["depends"][pn]:
-                print('"%s" -> "%s" [style=solid]' % (pn, depend), file=depends_file)
-        for pn in depgraph["rdepends-pn"]:
-            for rdepend in depgraph["rdepends-pn"][pn]:
-                print('"%s" -> "%s" [style=dashed]' % (pn, rdepend), file=depends_file)
-        print("}", file=depends_file)
-        depends_file.close()
-        logger.info("PN dependencies saved to 'pn-depends.dot'")
-
-        depends_file = open('package-depends.dot', 'w' )
-        print("digraph depends {", file=depends_file)
-        for package in depgraph["packages"]:
-            pn = depgraph["packages"][package]["pn"]
-            fn = depgraph["packages"][package]["filename"]
-            version = depgraph["packages"][package]["version"]
-            if package == pn:
-                print('"%s" [label="%s %s\\n%s"]' % (pn, pn, version, fn), file=depends_file)
-            else:
-                print('"%s" [label="%s(%s) %s\\n%s"]' % (package, package, pn, version, fn), file=depends_file)
-            for depend in depgraph["depends"][pn]:
-                print('"%s" -> "%s" [style=solid]' % (package, depend), file=depends_file)
-        for package in depgraph["rdepends-pkg"]:
-            for rdepend in depgraph["rdepends-pkg"][package]:
-                print('"%s" -> "%s" [style=dashed]' % (package, rdepend), file=depends_file)
-        for package in depgraph["rrecs-pkg"]:
-            for rdepend in depgraph["rrecs-pkg"][package]:
-                print('"%s" -> "%s" [style=dotted]' % (package, rdepend), file=depends_file)
-        print("}", file=depends_file)
-        depends_file.close()
-        logger.info("Package dependencies saved to 'package-depends.dot'")
-
         tdepends_file = open('task-depends.dot', 'w' )
         print("digraph depends {", file=tdepends_file)
         for task in depgraph["tdepends"]:
-- 
2.7.4