[Buildroot] [PATCHv2] pkg-infra: allow dumping reverse dependencies of a package

[Buildroot] [PATCHv2] pkg-infra: allow dumping reverse dependencies of a package

[Yann E. MORIN]

Finding the packages that select another one in a specific configuration
is not very trivial:

  - when optional, the dependency is not expressed in Kconfig

  - looking at the .mk files is not very nice.

Introduce a way to dump reverse dependencies of packages, i.e. the list
of packages that directly depend on that package. Like for direct
dependencies, we limit the list to the first-order reverse dependencies.

Document it in the main help; use the opportunity to also document
foo-show-depends.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

---
Changes v1 -> v2:
  - only add dependencies for enabled packages  (Thomas)
  - typoes  (Arnout)
---
 Makefile               | 2 ++
 package/pkg-generic.mk | 7 +++++++
 2 files changed, 9 insertions(+)

diff --git a/Makefile b/Makefile
index 3ba1490..5c65f90 100644
--- a/Makefile
+++ b/Makefile
@@ -974,6 +974,8 @@ help:
 	@echo '  <pkg>-depends          - Build <pkg>'\''s dependencies'
 	@echo '  <pkg>-configure        - Build <pkg> up to the configure step'
 	@echo '  <pkg>-build            - Build <pkg> up to the build step'
+	@echo '  <pkg>-show-depends     - List packages on which <pkg> depends'
+	@echo '  <pkg>-show-rdepends    - List packages which have <pkg> as a dependency'
 	@echo '  <pkg>-graph-depends    - Generate a graph of <pkg>'\''s dependencies'
 	@echo '  <pkg>-dirclean         - Remove <pkg> build directory'
 	@echo '  <pkg>-reconfigure      - Restart the build from the configure step'
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index e726ac5..81bb82c 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -698,6 +698,9 @@ $(1)-show-version:
 $(1)-show-depends:
 			@echo $$($(2)_FINAL_ALL_DEPENDENCIES)
 
+$(1)-show-rdepends:
+			@echo $$($(2)_RDEPENDENCIES)
+
 $(1)-graph-depends: graph-depends-requirements
 			@$$(INSTALL) -d $$(GRAPHS_DIR)
 			@cd "$$(CONFIG_DIR)"; \
@@ -854,6 +857,10 @@ $$(foreach pkg,$$($(2)_PROVIDES),\
 	$$(eval $$(call virt-provides-single,$$(pkg),$$(call UPPERCASE,$$(pkg)),$(1))$$(sep)))
 endif
 
+# Register package as a reverse-dependencies of all its dependencies
+$$(eval $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),\
+	$$(call UPPERCASE,$$(p))_RDEPENDENCIES += $(1)$$(sep)))
+
 # Ensure unified variable name conventions between all packages Some
 # of the variables are used by more than one infrastructure; so,
 # rather than duplicating the checks in each infrastructure, we check
-- 
2.7.4

[Buildroot] [PATCH] core/graph-depends: add option to graph reverse dependencies

[Yann E. MORIN]

Now that we can dump the reverse dependencies of a package, add the
ability to graph those.

It does not make sense to do a full reverse graph, as it would be
semantically equivalent to the direct graph. So we only provide a
per-package reverse graph.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile                      |  3 ++-
 package/pkg-generic.mk        | 21 ++++++++++++++-------
 support/scripts/graph-depends | 18 ++++++++++++++++--
 3 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/Makefile b/Makefile
index 5c65f90..c00e200 100644
--- a/Makefile
+++ b/Makefile
@@ -761,7 +761,7 @@ graph-depends: graph-depends-requirements
 	@$(INSTALL) -d $(GRAPHS_DIR)
 	@cd "$(CONFIG_DIR)"; \
 	$(TOPDIR)/support/scripts/graph-depends $(BR2_GRAPH_DEPS_OPTS) \
-		-o $(GRAPHS_DIR)/$(@).dot
+		--direct -o $(GRAPHS_DIR)/$(@).dot
 	dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) \
 		-o $(GRAPHS_DIR)/$(@).$(BR_GRAPH_OUT) \
 		$(GRAPHS_DIR)/$(@).dot
@@ -977,6 +977,7 @@ help:
 	@echo '  <pkg>-show-depends     - List packages on which <pkg> depends'
 	@echo '  <pkg>-show-rdepends    - List packages which have <pkg> as a dependency'
 	@echo '  <pkg>-graph-depends    - Generate a graph of <pkg>'\''s dependencies'
+	@echo '  <pkg>-graph-rdepends   - Generate a graph of <pkg>'\''s reverse dependencies'
 	@echo '  <pkg>-dirclean         - Remove <pkg> build directory'
 	@echo '  <pkg>-reconfigure      - Restart the build from the configure step'
 	@echo '  <pkg>-rebuild          - Restart the build from the build step'
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 81bb82c..3349092 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -317,6 +317,16 @@ be selected at a time. Please fix your configuration)
 endif
 endef
 
+define pkg-graph-depends
+	@$$(INSTALL) -d $$(GRAPHS_DIR)
+	@cd "$$(CONFIG_DIR)"; \
+	$$(TOPDIR)/support/scripts/graph-depends $$(BR2_GRAPH_DEPS_OPTS) \
+		-p $(1) $(2) -o $$(GRAPHS_DIR)/$$(@).dot
+	dot $$(BR2_GRAPH_DOT_OPTS) -T$$(BR_GRAPH_OUT) \
+		-o $$(GRAPHS_DIR)/$$(@).$$(BR_GRAPH_OUT) \
+		$$(GRAPHS_DIR)/$$(@).dot
+endef
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -702,13 +712,10 @@ $(1)-show-rdepends:
 			@echo $$($(2)_RDEPENDENCIES)
 
 $(1)-graph-depends: graph-depends-requirements
-			@$$(INSTALL) -d $$(GRAPHS_DIR)
-			@cd "$$(CONFIG_DIR)"; \
-			$$(TOPDIR)/support/scripts/graph-depends $$(BR2_GRAPH_DEPS_OPTS) \
-				-p $(1) -o $$(GRAPHS_DIR)/$$(@).dot
-			dot $$(BR2_GRAPH_DOT_OPTS) -T$$(BR_GRAPH_OUT) \
-				-o $$(GRAPHS_DIR)/$$(@).$$(BR_GRAPH_OUT) \
-				$$(GRAPHS_DIR)/$$(@).dot
+	$(call pkg-graph-depends,$(1),--direct)
+
+$(1)-graph-rdepends: graph-depends-requirements
+	$(call pkg-graph-depends,$(1),--reverse)
 
 $(1)-all-source:	$(1)-source
 $(1)-all-source:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source)
diff --git a/support/scripts/graph-depends b/support/scripts/graph-depends
index cb00383..c3c97cb 100755
--- a/support/scripts/graph-depends
+++ b/support/scripts/graph-depends
@@ -63,6 +63,10 @@ parser.add_argument("--transitive", dest="transitive", action='store_true',
                     default=False)
 parser.add_argument("--no-transitive", dest="transitive", action='store_false',
                     help="Draw (do not draw) transitive dependencies")
+parser.add_argument("--direct", dest="direct", action='store_true', default=True,
+                    help="Draw direct dependencies (the default)")
+parser.add_argument("--reverse", dest="direct", action='store_false',
+                    help="Draw reverse dependencies")
 args = parser.parse_args()
 
 check_only = args.check_only
@@ -95,6 +99,16 @@ else:
 
 transitive = args.transitive
 
+if args.direct:
+    rule = "show-depends"
+    arrow_dir = "forward"
+else:
+    if mode == MODE_FULL:
+        sys.stderr.write("--reverse needs a package\n")
+        sys.exit(1)
+    rule = "show-rdepends"
+    arrow_dir = "back"
+
 # Get the colours: we need exactly three colours,
 # so no need not split more than 4
 # We'll let 'dot' validate the colours...
@@ -151,7 +165,7 @@ def get_depends(pkgs):
     sys.stderr.write("Getting dependencies for %s\n" % pkgs)
     cmd = ["make", "-s", "--no-print-directory" ]
     for pkg in pkgs:
-        cmd.append("%s-show-depends" % pkg)
+        cmd.append("%s-%s" % (pkg, rule))
     p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
     output = p.communicate()[0]
     if p.returncode != 0:
@@ -418,7 +432,7 @@ def print_pkg_deps(depth, pkg):
                     add = False
                     break
             if add:
-                outfile.write("%s -> %s\n" % (pkg_node_name(pkg), pkg_node_name(d)))
+                outfile.write("%s -> %s [dir=%s]\n" % (pkg_node_name(pkg), pkg_node_name(d), arrow_dir))
                 print_pkg_deps(depth+1, d)
 
 # Start printing the graph data
-- 
2.7.4

[Buildroot] [PATCH] core/graph-depends: add option to graph reverse dependencies

[Thomas Petazzoni]

Hello,

On Sun, 23 Oct 2016 19:19:44 +0200, Yann E. MORIN wrote:
> Now that we can dump the reverse dependencies of a package, add the
> ability to graph those.
> 
> It does not make sense to do a full reverse graph, as it would be
> semantically equivalent to the direct graph. So we only provide a
> per-package reverse graph.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Makefile                      |  3 ++-
>  package/pkg-generic.mk        | 21 ++++++++++++++-------
>  support/scripts/graph-depends | 18 ++++++++++++++++--
>  3 files changed, 32 insertions(+), 10 deletions(-)

I've applied, thanks. However, our logic that gets rid of "redundant"
dependencies also has the consequence of creating some graphs that are
quite weird.

I tested a configuration where I enabled python and libglib2, both of
which depend on libffi. When you do:

$ make libffi-show-rdepends

You get as expected "libglib2 python"

However, when you do

$ make libffi-graph-rdepends

The resulting graph is a bit weird. You would expect something like
this:

           libffi
       ------/ \------
       |             |
    libglib2       python

But instead, what you get is:

  libffi
    /\
    ||
   python
    /\
    ||
 util-linux
    /\
    ||
  libglib2

Because indeed libglib2 depends on util-linux, and util-linux can
optionally build python bindings and therefore depend on python.

Not sure what we can do about this: keeping all dependencies was really
creating huge and unreadable graphs.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH] core/graph-depends: add option to graph reverse dependencies

[Yann E. MORIN]

Thomas, All,

On 2016-10-25 23:04 +0200, Thomas Petazzoni spake thusly:
> On Sun, 23 Oct 2016 19:19:44 +0200, Yann E. MORIN wrote:
> > Now that we can dump the reverse dependencies of a package, add the
> > ability to graph those.
[--SNIP--]
> I've applied, thanks. However, our logic that gets rid of "redundant"
> dependencies also has the consequence of creating some graphs that are
> quite weird.
> 
> I tested a configuration where I enabled python and libglib2, both of
> which depend on libffi. When you do:
> 
> $ make libffi-show-rdepends
> 
> You get as expected "libglib2 python"
> 
> However, when you do
> 
> $ make libffi-graph-rdepends
> 
> The resulting graph is a bit weird. You would expect something like
> this:
> 
>            libffi
>        ------/ \------
>        |             |
>     libglib2       python

make BR2_GRAPH_DEPS_OPTS='--transitive -d1' libffi-graph-rdepends
would give you the expected graph.

But using options is not what you want. ;-)

> But instead, what you get is:
> 
>   libffi
>     /\
>     ||
>    python
>     /\
>     ||
>  util-linux
>     /\
>     ||
>   libglib2
> 
> Because indeed libglib2 depends on util-linux, and util-linux can
> optionally build python bindings and therefore depend on python.
> 
> Not sure what we can do about this: keeping all dependencies was really
> creating huge and unreadable graphs.

Well, that's the crux of the issue: keeping the whole graph is quickly
unworkable with even non-complex configurations.

Maybe what we should stress is that the graph-depends and graph-rdepends
display the build-order dependencies.

But yes, the apparent discrepancies between foo-show-rdepends and
foo-graph-rdepends can look weird on first sight, but both are correct.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv2] pkg-infra: allow dumping reverse dependencies of a package

[Thomas Petazzoni]

Hello,

On Sun, 23 Oct 2016 17:28:51 +0200, Yann E. MORIN wrote:
> Finding the packages that select another one in a specific configuration
> is not very trivial:
> 
>   - when optional, the dependency is not expressed in Kconfig
> 
>   - looking at the .mk files is not very nice.
> 
> Introduce a way to dump reverse dependencies of packages, i.e. the list
> of packages that directly depend on that package. Like for direct
> dependencies, we limit the list to the first-order reverse dependencies.
> 
> Document it in the main help; use the opportunity to also document
> foo-show-depends.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> 
> ---
> Changes v1 -> v2:
>   - only add dependencies for enabled packages  (Thomas)
>   - typoes  (Arnout)
> ---
>  Makefile               | 2 ++
>  package/pkg-generic.mk | 7 +++++++
>  2 files changed, 9 insertions(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com