[Buildroot] [PATCH 01/12 v3] infra/pkg-download: return just a list of URIs

[Buildroot] [PATCH 01/12 v3] infra/pkg-download: return just a list of URIs

[Yann E. MORIN]

Currently, the internal DOWNLOAD_URIS variable is set to be a list of
options to pass to the download wrapper, with each URI passed as
'-u URI'.

This precludes using that variable to get just the list of URIs, in
case we need to do something else with it.

Fix the variable to really only contain the list of URIs.

Adapt the caller accordingly.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v1 -> v2:
  - use foreach instead of patsubst  (Thomas)
---
 package/pkg-download.mk | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 7cd87c38ff..4163333998 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -75,17 +75,17 @@ export BR_NO_CHECK_HASH_FOR =
 
 ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
 DOWNLOAD_URIS += \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
+	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
 endif
 
 ifeq ($(BR2_PRIMARY_SITE_ONLY),)
 DOWNLOAD_URIS += \
-	-u $(patsubst %/,%,$(dir $(call qstrip,$(1))))
+	$(patsubst %/,%,$(dir $(call qstrip,$(1))))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode)
+	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode)
 endif
 endif
 
@@ -101,7 +101,7 @@ define DOWNLOAD
 		-N '$($(PKG)_RAWNAME)' \
 		-o '$($(PKG)_DL_DIR)/$(notdir $(1))' \
 		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		$(DOWNLOAD_URIS) \
+		$(foreach uri,$(DOWNLOAD_URIS),-u $(uri)) \
 		$(QUIET) \
 		-- \
 		$($(PKG)_DL_OPTS)
-- 
2.14.1

[Buildroot] [PATCH 02/12 v3] infra/pkg-download: make the URI list a callable macro

[Yann E. MORIN]

Currently, that variable is context-dependent, as it expects the PKG
variable to exist and be defined to the current package.

This is not so clean, so change the variable to a callable macro.

Adapt the caller accordingly.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v1 -> v2:
  - split DOWNLOAD/DOWNLOAD_URIS comment  (Thomas DS)
---
 package/pkg-download.mk | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 4163333998..623caf9325 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -63,19 +63,19 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
 export BR_NO_CHECK_HASH_FOR =
 
 ################################################################################
-# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
-# source from:
+# DOWNLOAD_URIS - List the candidates URIs where to get the package from:
 # 1) BR2_PRIMARY_SITE if enabled
 # 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
 # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
 #
 # Argument 1 is the source location
+# Argument 2 is the upper-case package name
 #
 ################################################################################
 
 ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
 DOWNLOAD_URIS += \
-	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(2)_DL_SUBDIR)),urlencode) \
 	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
 endif
 
@@ -84,11 +84,19 @@ DOWNLOAD_URIS += \
 	$(patsubst %/,%,$(dir $(call qstrip,$(1))))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
-	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(2)_DL_SUBDIR)),urlencode) \
 	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode)
 endif
 endif
 
+################################################################################
+# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
+# source from the list returned by DOWNLOAD_URIS.
+#
+# Argument 1 is the source location
+#
+################################################################################
+
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
 	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
@@ -101,7 +109,7 @@ define DOWNLOAD
 		-N '$($(PKG)_RAWNAME)' \
 		-o '$($(PKG)_DL_DIR)/$(notdir $(1))' \
 		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		$(foreach uri,$(DOWNLOAD_URIS),-u $(uri)) \
+		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(PKG)),-u $(uri)) \
 		$(QUIET) \
 		-- \
 		$($(PKG)_DL_OPTS)
-- 
2.14.1

[Buildroot] [PATCH 03/12 v3] infra/pkg-download: get rid of the FLOCK variable

[Yann E. MORIN]

The FLOCK variable is context-dependent, and expects the PKG
variable to be set to the current package.

This is not so nice. Besides, it is used in a single location.

Get rid of this intermediate variable, and directly use flock
where we need it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
 package/pkg-download.mk | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 623caf9325..96a12b1f2e 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -18,7 +18,6 @@ export SCP := $(call qstrip,$(BR2_SCP))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
-FLOCK = flock $($(PKG)_DL_DIR)/
 
 # DL_DIR may have been set already from the environment
 ifeq ($(origin DL_DIR),undefined)
@@ -99,7 +98,7 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
-	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) flock $($(PKG)_DL_DIR)/ $(DL_WRAPPER) \
 		-c '$($(PKG)_DL_VERSION)' \
 		-d '$($(PKG)_DL_DIR)' \
 		-D '$(DL_DIR)' \
-- 
2.14.1

[Buildroot] [PATCH 04/12 v3] infra/pkg-download: make the DOWNLOAD macro fully parameterised

[Yann E. MORIN]

Currently, the DOWNLOAD macro is context-dependent and expects
the PKG variable to be set to the current package.

This is not so nice.

Change the macro to expect the upper-case package name as a
parameter, rather than expect it from a variable.

Adapt the caller accordingly.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
changes v1 -> v2:
  - DOWNLOAD_URIS was called twice in v1, once with $(PKG) and once with
    $(1) (due to an incorrect rebase)
---
 package/pkg-download.mk | 23 ++++++++++++-----------
 package/pkg-generic.mk  |  4 ++--
 2 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 96a12b1f2e..de619ba90a 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -93,23 +93,24 @@ endif
 # source from the list returned by DOWNLOAD_URIS.
 #
 # Argument 1 is the source location
+# Argument 2 is the upper-case package name
 #
 ################################################################################
 
 define DOWNLOAD
-	$(Q)mkdir -p $($(PKG)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(PKG)_DL_DIR)/ $(DL_WRAPPER) \
-		-c '$($(PKG)_DL_VERSION)' \
-		-d '$($(PKG)_DL_DIR)' \
+	$(Q)mkdir -p $($(2)_DL_DIR)
+	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/ $(DL_WRAPPER) \
+		-c '$($(2)_DL_VERSION)' \
+		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
 		-f '$(notdir $(1))' \
-		-H '$($(PKG)_HASH_FILE)' \
-		-n '$($(PKG)_BASENAME_RAW)' \
-		-N '$($(PKG)_RAWNAME)' \
-		-o '$($(PKG)_DL_DIR)/$(notdir $(1))' \
-		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(PKG)),-u $(uri)) \
+		-H '$($(2)_HASH_FILE)' \
+		-n '$($(2)_BASENAME_RAW)' \
+		-N '$($(2)_RAWNAME)' \
+		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_GIT_SUBMODULES),-r) \
+		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
 		-- \
-		$($(PKG)_DL_OPTS)
+		$($(2)_DL_OPTS)
 endef
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index a83813e28d..8ff8cff12d 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -149,7 +149,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
 			break ; \
 		fi ; \
 	done
-	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p))$(sep))
+	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
 	$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
 	$(Q)mkdir -p $(@D)
 	@$(call step_end,download)
@@ -158,7 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
 # Retrieve actual source archive, e.g. for prebuilt external toolchains
 $(BUILD_DIR)/%/.stamp_actual_downloaded:
 	@$(call step_start,actual-download)
-	$(call DOWNLOAD,$($(PKG)_ACTUAL_SOURCE_SITE)/$($(PKG)_ACTUAL_SOURCE_TARBALL))
+	$(call DOWNLOAD,$($(PKG)_ACTUAL_SOURCE_SITE)/$($(PKG)_ACTUAL_SOURCE_TARBALL),$(PKG))
 	$(Q)mkdir -p $(@D)
 	@$(call step_end,actual-download)
 	$(Q)touch $@
-- 
2.14.1

[Buildroot] [PATCH 05/12 v3] infra/utils: add helper to generate comma-separated lists

[Yann E. MORIN]

Add a helper macro that, from a space-separated list of items, returns a
comma-separated list of the quoted items.

This will be useful when we need to generate lists in JSON, later...

Code suggested by Thomas P.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v1 -> v2:
  - don't rely on implicit space in replacement text, instead use
    explicit $(space)  (Thmas DS)
---
 support/misc/utils.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/support/misc/utils.mk b/support/misc/utils.mk
index c44319338e..b422c67d76 100644
--- a/support/misc/utils.mk
+++ b/support/misc/utils.mk
@@ -70,6 +70,10 @@ finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
 # notfirstword(wordlist): returns all but the first word in wordlist
 notfirstword = $(wordlist 2,$(words $(1)),$(1))
 
+# build a comma-separated list of quoted items, from a space-separated
+# list of unquoted items:   a b c d  -->  "a", "b", "c", "d"
+make-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,"%",$(strip $(1))))
+
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
 define sep
-- 
2.14.1

[Buildroot] [PATCH 06/12 v3] fs: introduce variables with name and type

[Yann E. MORIN]

This makes the filesystems resemble packages yet a bit more, and will
allow sorting "items" on their type and names, when indexed from the
upper-case names.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 fs/common.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 4ad51fdd0a..286d671d06 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -42,6 +42,8 @@ define ROOTFS_REPRODUCIBLE
 endef
 endif
 
+ROOTFS_COMMON_NAME = rootfs-common
+ROOTFS_COMMON_TYPE = rootfs
 ROOTFS_COMMON_DEPENDENCIES = \
 	host-fakeroot host-makedevs \
 	$(BR2_TAR_HOST_DEPENDENCY) \
@@ -77,6 +79,8 @@ rootfs-common-show-depends:
 # all variable references except the arguments must be $$-quoted.
 define inner-rootfs
 
+ROOTFS_$(2)_NAME = rootfs-$(1)
+ROOTFS_$(2)_TYPE = rootfs
 ROOTFS_$(2)_IMAGE_NAME ?= rootfs.$(1)
 ROOTFS_$(2)_FINAL_IMAGE_NAME = $$(strip $$(ROOTFS_$(2)_IMAGE_NAME))
 ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1)
-- 
2.14.1

[Buildroot] [PATCH 07/12 v3] fs: introduce variable with all recursive dependencies

[Yann E. MORIN]

This allows getting all the recursive dependencies of filesystems,
ike we have for packages, and allows us to treat both in a similar
fashion.

Reported-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 fs/common.mk | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 286d671d06..5ec28ca183 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -49,6 +49,18 @@ ROOTFS_COMMON_DEPENDENCIES = \
 	$(BR2_TAR_HOST_DEPENDENCY) \
 	$(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd)
 
+ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES = $(sort \
+	$(if $(filter undefined,$(origin ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X)), \
+		$(eval ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X := \
+			$(foreach p, \
+				$(ROOTFS_COMMON_DEPENDENCIES), \
+				$(p) \
+				$($(call UPPERCASE,$(p))_FINAL_RECURSIVE_DEPENDENCIES) \
+			) \
+		) \
+	) \
+	$(ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X))
+
 rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES))
 	$(info rootfs-common: host)
 	$(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d)))
@@ -88,6 +100,18 @@ ROOTFS_$(2)_TARGET_DIR = $$(ROOTFS_$(2)_DIR)/target
 
 ROOTFS_$(2)_DEPENDENCIES += rootfs-common
 
+ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \
+	$$(if $$(filter undefined,$$(origin ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X)), \
+		$$(eval ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X := \
+			$$(foreach p, \
+				$$(ROOTFS_$(2)_DEPENDENCIES), \
+				$$(p) \
+				$$($$(call UPPERCASE,$$(p))_FINAL_RECURSIVE_DEPENDENCIES) \
+			) \
+		) \
+	) \
+	$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X))
+
 rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES))
 	$$(info rootfs-$(1): host)
 	$$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d)))
-- 
2.14.1

[Buildroot] [PATCH 00/12 v3] infra: add solution to dump metadata from packages (branch yem/show-info-2)

[Yann E. MORIN]

Hello All!

This series is a proposal to allow extracting metadata about packages,
in a way that makes it reusable for tooling outside of Buildroot, that
is both reliable and extendable, without causing too much burden on
Buildroot own infrastructure.

This series introduces a global 'show-info' rule, and a per package and
per filesystem 'show-info' rule, that dump the package metadata as a
JSON blob. JSON is a key-value based serialisation format. As such, it
is easy to parse (there are implementations in virtually all languages,
and there are even tools like jq, that can be used from shell scripts).
But most importantly, it is easy to extend without breakign existing
tools (as long as they use proper JSON parsers, that is).

This series first introduces 4 cleanup preparatory patches in the
download infra, so that it gets easy to get the list of URIs to report
them. Strictly speaking, only patches 1 and 2 are required for this
series, but the cleanup was extended to the infra with patches 3 and 4,
for consistency.

Then, the series introduces a per package and per filesystem show-info
rule. The per filesystem rule is needed to be able to get the info about
packages that are a dependency of filesystems.

Finally, a global show-info rule is introduced that generates a JSON
array with all the information about a build.

In the end, the output for a packages would look like:

    $ make cracklib-show-info |jq . -
    {
      "cracklib": {
        "type": "target",
        "virtual": false,
        "version": "2.9.7",
        "licenses": "LGPL-2.1",
        "downloads": [
          {
            "source": "cracklib-2.9.7.tar.gz",
            "URIs": [
              "https+https://github.com/cracklib/cracklib/releases/download/v2.9.7",
              "http|urlencode+http://sources.buildroot.net/cracklib",
              "http|urlencode+http://sources.buildroot.net"
            ]
          },
          {
            "source": "cracklib-words-2.9.7.gz",
            "URIs": [
              "https://github.com/cracklib/cracklib/releases/download/v2.9.7",
              "http|urlencode+http://sources.buildroot.net/cracklib",
              "http|urlencode+http://sources.buildroot.net"
            ]
          }
        ],
        "dependencies": [
          "host-cracklib",
          "host-skeleton",
          "skeleton",
          "toolchain"
        ],
        "reverse_dependencies": [
          "libpwquality",
          "linux-pam"
        ]
      }
    }

While the whole show-info would look like (elipsed and manually indented
for readbility):

    $ make show-info
    {
      "host-gcc-final": { "type": "host", "virtual": false, ... },
      "host-binutils": { "type": "host", "virtual": false, ... },
      "toolchain-buildroot": { "type": "target", "virtual": true, ... },
      "gettext-tiny": { "type": "target", "virtual": false, ... },
      "gettext": { "type": "target", "virtual": true, ... },
      "ifupdown-scripts": { "type": "target", "virtual": false, ... },
      ...
      "rootfs-common": { "type": "rootfs", ... },
      "rootfs-tar": { "type": "rootfs", ... }
    }

Changes v2 -> v3:
  - set the virutal field in a single place  (Arnout)
  - rename internal macro  (Arnout)
  - don't add opening and closing {} in clean-json  (Arnout)
  - add to 'make help'

Changes v1 -> v2:
  - DOWNLOAD_URIS uses foreach instead of patsubst  (Thomas P.)
  - rename dependencies keys  (Thomas DS)
  - split DOWNLOAD/DOWNLOAD_URIS comments  (Thomas DS)
  - make-comma-list uses an explicit $(space)  (Thomas DS)
  - top-level show-info is a dictionary, not an array  (Arnout)
  - use a macro to generate a JSON element for an item  (Arnout)
  - drop the 'null' object in download list  (Arnout, Thomas DS)
  - change graph-depends to use that instead of show-dependency-graph
  - drop show-dependency-graph
  - fix double list of URIs
  - fix typoes


Regards,
Yann E. MORIN.


The following changes since commit a45a3997d8ee0aa592c6a64ce300ee727ad7dc54

  package/cjson: security bump to version 1.7.11 (2019-04-15 21:40:14 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to add67b53daa3c056dad8577f57d483c2bbda3fc6

  core: remove show-depednency-tree (2019-04-15 21:46:41 +0200)


----------------------------------------------------------------
Yann E. MORIN (12):
      infra/pkg-download: return just a list of URIs
      infra/pkg-download: make the URI list a callable macro
      infra/pkg-download: get rid of the FLOCK variable
      infra/pkg-download: make the DOWNLOAD macro fully parameterised
      infra/utils: add helper to generate comma-separated lists
      fs: introduce variables with name and type
      fs: introduce variable with all recursive dependencies
      fs: add all recursive dependencies to packages list
      core: introduce new global show-info
      core: add per-package and per-filesystem show-info
      support/scripts: use show-info to extract dependency graph
      core: remove show-depednency-tree

 Makefile                     | 21 ++++++++++++---
 fs/common.mk                 | 45 +++++++++++++++++++++++++------
 package/pkg-download.mk      | 46 ++++++++++++++++++-------------
 package/pkg-generic.mk       | 11 ++++----
 package/pkg-utils.mk         | 64 ++++++++++++++++++++++++++++++++++++++++++++
 support/misc/utils.mk        |  4 +++
 support/scripts/brpkgutil.py | 30 ++++++++++-----------
 7 files changed, 168 insertions(+), 53 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 08/12 v3] fs: add all recursive dependencies to packages list

[Yann E. MORIN]

Currently, only first-level dependencies of a filesystem are added to
the global list of packages, thus missing all recursive dependencies.

Use the newly introduced recursive variable instead, which already
contains the rootfs-common dependencies too.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/common.mk b/fs/common.mk
index 5ec28ca183..f7989eac57 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -190,7 +190,7 @@ rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS_ROOTFS += rootfs-$(1)
-PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES) $$(ROOTFS_COMMON_DEPENDENCIES))
+PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES))
 endif
 
 # Check for legacy POST_TARGETS rules
-- 
2.14.1

[Buildroot] [PATCH 09/12 v3] core: introduce new global show-info

[Yann E. MORIN]

Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.

Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.

Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...

We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).

The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.

It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma:  [ { ... }, { ... } ]

So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".

The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).

Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v2 -> v3:
  - set the virtual field in a single place  (Arnout)
  - rename the macros  (Arnout)
  - don't add opening and closing {} in clean-json  (Arnout)
  - add to 'make help'

Changes v1 -> v2:
  - make it a macro to be called  (Arnout)
  - make it a top-level rule  (Arnout)
---
 Makefile             | 16 +++++++++++++
 package/pkg-utils.mk | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 80 insertions(+)

diff --git a/Makefile b/Makefile
index 60bf7d7d08..0f5331ec7e 100644
--- a/Makefile
+++ b/Makefile
@@ -903,6 +903,21 @@ check-dependencies:
 	@cd "$(CONFIG_DIR)"; \
 	$(TOPDIR)/support/scripts/graph-depends -C
 
+.PHONY: show-info
+show-info:
+	@:
+	$(info $(call clean-json, \
+			{ $(foreach p, \
+				$(sort $(foreach i,$(PACKAGES) $(TARGETS_ROOTFS), \
+						$(i) \
+						$($(call UPPERCASE,$(i))_FINAL_RECURSIVE_DEPENDENCIES) \
+					) \
+				), \
+				$(call json-info,$(call UPPERCASE,$(p)))$(comma) \
+			) } \
+		) \
+	)
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
@@ -1128,6 +1143,7 @@ help:
 	@echo '  source                 - download all sources needed for offline-build'
 	@echo '  external-deps          - list external packages used'
 	@echo '  legal-info             - generate info about license compliance'
+	@echo '  show-info              - generate info about packages, as a JSON blurb'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index bffd79dfb0..b7280e930f 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -62,6 +62,70 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
 endif
 endef
 
+# json-info -- return package or filesystem metadata formatted as an entry
+#              of a JSON dictionnary
+# $(1): upper-case package or filesystem name
+define json-info
+	"$($(1)_NAME)": {
+		"type": "$($(1)_TYPE)",
+		$(if $(filter rootfs,$($(1)_TYPE)), \
+			$(call _json-info-fs,$(1)), \
+			$(call _json-info-pkg,$(1)), \
+		)
+	}
+endef
+
+# _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers
+# for json-info, above
+define _json-info-pkg
+	$(if $($(1)_IS_VIRTUAL), \
+		"virtual": true$(comma),
+		"virtual": false$(comma)
+		$(call _json-info-pkg-details,$(1)) \
+	)
+	"dependencies": [
+		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
+	],
+	"reverse_dependencies": [
+		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+	]
+endef
+
+define _json-info-pkg-details
+	"version": "$($(1)_DL_VERSION)",
+	"licenses": "$($(1)_LICENSE)",
+	"downloads": [
+	$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
+		{
+			"source": "$(notdir $(dl))",
+			"uris": [
+				$(call make-comma-list,
+					$(subst \|,|,
+						$(call DOWNLOAD_URIS,$(dl),$(1))
+					)
+				)
+			]
+		},
+	)
+	],
+endef
+
+define _json-info-fs
+	"dependencies": [
+		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
+	]
+endef
+
+# clean-json -- cleanup pseudo-json into clean json:
+#  - remove commas before closing ] and }
+#  - minify with $(strip)
+clean-json = $(strip \
+	$(subst $(comma)},}, $(subst $(comma)$(space)},$(space)}, \
+	$(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
+		$(strip $(1)) \
+	)))) \
+)
+
 #
 # legal-info helper functions
 #
-- 
2.14.1

[Buildroot] [PATCH 10/12 v3] core: add per-package and per-filesystem show-info

[Yann E. MORIN]

Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.

Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.

Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:

    $ make {foo,bar}-show-info
    { "foo": { foo stuff } }
    { "bar": { bar stuff } }

However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):

    $ make {foo,bar}-show-info |jq -s . -
    [
      { "foo": { foo stuff } },
      { "bar": { bar stuff } }
    ]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v2 -> v3:
  - add to 'make help'

Changes v1 -> v2:
  - make it based on the new show-info macro  (Arnout)
---
 Makefile               |  1 +
 fs/common.mk           | 11 ++++++++++-
 package/pkg-generic.mk |  5 +++++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0f5331ec7e..522c0b0606 100644
--- a/Makefile
+++ b/Makefile
@@ -1111,6 +1111,7 @@ help:
 	@echo '  <pkg>-depends          - Build <pkg>'\''s dependencies'
 	@echo '  <pkg>-configure        - Build <pkg> up to the configure step'
 	@echo '  <pkg>-build            - Build <pkg> up to the build step'
+	@echo '  <pkg>-show-info        - generate info about <pkg>, as a JSON blurb'
 	@echo '  <pkg>-show-depends     - List packages on which <pkg> depends'
 	@echo '  <pkg>-show-rdepends    - List packages which have <pkg> as a dependency'
 	@echo '  <pkg>-show-recursive-depends'
diff --git a/fs/common.mk b/fs/common.mk
index f7989eac57..6dd1774726 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -87,6 +87,11 @@ endif
 rootfs-common-show-depends:
 	@echo $(ROOTFS_COMMON_DEPENDENCIES)
 
+.PHONY: rootfs-common-show-info
+rootfs-common-show-info:
+	@:
+	$(info $(call clean-json,{ $(call json-info,ROOTFS_COMMON) }))
+
 # Since this function will be called from within an $(eval ...)
 # all variable references except the arguments must be $$-quoted.
 define inner-rootfs
@@ -184,9 +189,13 @@ endif
 rootfs-$(1)-show-depends:
 	@echo $$(ROOTFS_$(2)_DEPENDENCIES)
 
+rootfs-$(1)-show-info:
+	@:
+	$$(info $$(call clean-json,{ $$(call json-info,ROOTFS_$(2)) }))
+
 rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME)
 
-.PHONY: rootfs-$(1) rootfs-$(1)-show-depends
+.PHONY: rootfs-$(1) rootfs-$(1)-show-depends rootfs-$(1)-show-info
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS_ROOTFS += rootfs-$(1)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 8ff8cff12d..67782138b4 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -867,6 +867,10 @@ $(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_AL
 	$$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION)))
 	$$(info $(1) -> $$($(2)_FINAL_ALL_DEPENDENCIES))
 
+$(1)-show-info:
+	@:
+	$$(info $$(call clean-json,{ $$(call json-info,$(2)) }))
+
 $(1)-graph-depends: graph-depends-requirements
 	$(call pkg-graph-depends,$(1),--direct)
 
@@ -1099,6 +1103,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-rsync \
 	$(1)-show-dependency-tree \
 	$(1)-show-depends \
+	$(1)-show-info \
 	$(1)-show-version \
 	$(1)-source
 
-- 
2.14.1

[Buildroot] [PATCH 11/12 v3] support/scripts: use show-info to extract dependency graph

[Yann E. MORIN]

Currently, we extract the dependency graph from the aptly named but
ad-hoc show-dependency-graph rule.

We now have a better solution to report package information, with
show-info.

Since show-dependency-graph never went into a release so far, and
show-info does provide the same (and more), swith to using show-info.

Thanks to Adam for suggesting the coding style to have a readable code
that is not ugly but still pleases flake8. Thanks to Arnout for
suggesting the use of dict.get() to further simplify the code.

Note: we do not use th reverse_dependencies field because it only
contains those packages that have a kconfig option, so we'd miss most
host packages.

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Adam Duskett <aduskett@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/scripts/brpkgutil.py | 30 ++++++++++++++----------------
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/support/scripts/brpkgutil.py b/support/scripts/brpkgutil.py
index f65068d348..780cdef49c 100644
--- a/support/scripts/brpkgutil.py
+++ b/support/scripts/brpkgutil.py
@@ -1,12 +1,12 @@
 # Copyright (C) 2010-2013 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
 # Copyright (C) 2019 Yann E. MORIN <yann.morin.1998@free.fr>
 
+import json
 import logging
 import os
 import subprocess
 from collections import defaultdict
 
-
 # This function returns a tuple of four dictionaries, all using package
 # names as keys:
 # - a dictionary which values are the lists of packages that are the
@@ -19,7 +19,7 @@ from collections import defaultdict
 def get_dependency_tree():
     logging.info("Getting dependency tree...")
 
-    deps = defaultdict(list)
+    deps = {}
     rdeps = defaultdict(list)
     types = {}
     versions = {}
@@ -29,23 +29,21 @@ def get_dependency_tree():
     types['all'] = 'target'
     versions['all'] = ''
 
-    cmd = ["make", "-s", "--no-print-directory", "show-dependency-tree"]
+    cmd = ["make", "-s", "--no-print-directory", "show-info"]
     with open(os.devnull, 'wb') as devnull:
         p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull,
                              universal_newlines=True)
-        output = p.communicate()[0]
+        pkg_list = json.loads(p.communicate()[0])
 
-    for l in output.splitlines():
-        if " -> " in l:
-            pkg = l.split(" -> ")[0]
-            deps[pkg] += l.split(" -> ")[1].split()
-            for p in l.split(" -> ")[1].split():
-                rdeps[p].append(pkg)
-        else:
-            pkg, type_version = l.split(": ", 1)
-            t, v = "{} -".format(type_version).split(None, 2)[:2]
-            deps['all'].append(pkg)
-            types[pkg] = t
-            versions[pkg] = v
+    for pkg in pkg_list:
+        deps['all'].append(pkg)
+        types[pkg] = pkg_list[pkg]["type"]
+        deps[pkg] = pkg_list[pkg].get("dependencies", [])
+        for p in deps[pkg]:
+            rdeps[p].append(pkg)
+        versions[pkg] = \
+            None if pkg_list[pkg]["type"] == "rootfs" \
+            else "virtual" if pkg_list[pkg]["virtual"] \
+            else pkg_list[pkg]["version"]
 
     return (deps, rdeps, types, versions)
-- 
2.14.1

[Buildroot] [PATCH 12/12 v3] core: remove show-depednency-tree

[Yann E. MORIN]

show-depednency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to geaph-depends.

show-dependency-tree is no longer used, no that graph-depends has been
switched over to using the more versatile show-info.

Beside, show-dependency-tree has never been part of a release.

Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 Makefile               | 4 ----
 fs/common.mk           | 8 --------
 package/pkg-generic.mk | 6 ------
 3 files changed, 18 deletions(-)

diff --git a/Makefile b/Makefile
index 522c0b0606..f1e70551d8 100644
--- a/Makefile
+++ b/Makefile
@@ -876,10 +876,6 @@ graph-depends-requirements:
 	@dot -? >/dev/null 2>&1 || \
 		{ echo "ERROR: The 'dot' program from Graphviz is needed for graph-depends" >&2; exit 1; }
 
-.PHONY: show-dependency-tree
-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(PACKAGES) $(TARGETS_ROOTFS))
-	@:
-
 .PHONY: graph-depends
 graph-depends: graph-depends-requirements
 	@$(INSTALL) -d $(GRAPHS_DIR)
diff --git a/fs/common.mk b/fs/common.mk
index 6dd1774726..fceb38c5b9 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -61,10 +61,6 @@ ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES = $(sort \
 	) \
 	$(ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X))
 
-rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES))
-	$(info rootfs-common: host)
-	$(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d)))
-
 .PHONY: rootfs-common
 rootfs-common: $(ROOTFS_COMMON_DEPENDENCIES) target-finalize
 	@$(call MESSAGE,"Generating root filesystems common tables")
@@ -117,10 +113,6 @@ ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \
 	) \
 	$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X))
 
-rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES))
-	$$(info rootfs-$(1): host)
-	$$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d)))
-
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
 ROOTFS_$(2)_COMPRESS_EXT = .gz
 ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 67782138b4..e44f5d913c 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -862,11 +862,6 @@ $(1)-show-build-order: $$(patsubst %,%-show-build-order,$$($(2)_FINAL_ALL_DEPEND
 	@:
 	$$(info $(1))
 
-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_ALL_DEPENDENCIES))
-	@:
-	$$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION)))
-	$$(info $(1) -> $$($(2)_FINAL_ALL_DEPENDENCIES))
-
 $(1)-show-info:
 	@:
 	$$(info $$(call clean-json,{ $$(call json-info,$(2)) }))
@@ -1101,7 +1096,6 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-reconfigure \
 	$(1)-reinstall \
 	$(1)-rsync \
-	$(1)-show-dependency-tree \
 	$(1)-show-depends \
 	$(1)-show-info \
 	$(1)-show-version \
-- 
2.14.1

[Buildroot] [PATCH 11/12 v3] support/scripts: use show-info to extract dependency graph

[Yann E. MORIN]

Arnout, All,

On 2019-04-15 21:47 +0200, Yann E. MORIN spake thusly:
> Currently, we extract the dependency graph from the aptly named but
> ad-hoc show-dependency-graph rule.
> 
> We now have a better solution to report package information, with
> show-info.
> 
> Since show-dependency-graph never went into a release so far, and
> show-info does provide the same (and more), swith to using show-info.
> 
> Thanks to Adam for suggesting the coding style to have a readable code
> that is not ugly but still pleases flake8. Thanks to Arnout for
> suggesting the use of dict.get() to further simplify the code.
> 
> Note: we do not use th reverse_dependencies field because it only
> contains those packages that have a kconfig option, so we'd miss most
> host packages.
> 
> Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Adam Duskett <aduskett@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
> ---
>  support/scripts/brpkgutil.py | 30 ++++++++++++++----------------
>  1 file changed, 14 insertions(+), 16 deletions(-)
> 
> diff --git a/support/scripts/brpkgutil.py b/support/scripts/brpkgutil.py
> index f65068d348..780cdef49c 100644
> --- a/support/scripts/brpkgutil.py
> +++ b/support/scripts/brpkgutil.py
> @@ -1,12 +1,12 @@
>  # Copyright (C) 2010-2013 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
>  # Copyright (C) 2019 Yann E. MORIN <yann.morin.1998@free.fr>
>  
> +import json
>  import logging
>  import os
>  import subprocess
>  from collections import defaultdict
>  
> -

Spurious line removal... Can you fix when applying, or should I respin?

Regards,
Yann E. MORIN.

>  # This function returns a tuple of four dictionaries, all using package
>  # names as keys:
>  # - a dictionary which values are the lists of packages that are the
> @@ -19,7 +19,7 @@ from collections import defaultdict
>  def get_dependency_tree():
>      logging.info("Getting dependency tree...")
>  
> -    deps = defaultdict(list)
> +    deps = {}
>      rdeps = defaultdict(list)
>      types = {}
>      versions = {}
> @@ -29,23 +29,21 @@ def get_dependency_tree():
>      types['all'] = 'target'
>      versions['all'] = ''
>  
> -    cmd = ["make", "-s", "--no-print-directory", "show-dependency-tree"]
> +    cmd = ["make", "-s", "--no-print-directory", "show-info"]
>      with open(os.devnull, 'wb') as devnull:
>          p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull,
>                               universal_newlines=True)
> -        output = p.communicate()[0]
> +        pkg_list = json.loads(p.communicate()[0])
>  
> -    for l in output.splitlines():
> -        if " -> " in l:
> -            pkg = l.split(" -> ")[0]
> -            deps[pkg] += l.split(" -> ")[1].split()
> -            for p in l.split(" -> ")[1].split():
> -                rdeps[p].append(pkg)
> -        else:
> -            pkg, type_version = l.split(": ", 1)
> -            t, v = "{} -".format(type_version).split(None, 2)[:2]
> -            deps['all'].append(pkg)
> -            types[pkg] = t
> -            versions[pkg] = v
> +    for pkg in pkg_list:
> +        deps['all'].append(pkg)
> +        types[pkg] = pkg_list[pkg]["type"]
> +        deps[pkg] = pkg_list[pkg].get("dependencies", [])
> +        for p in deps[pkg]:
> +            rdeps[p].append(pkg)
> +        versions[pkg] = \
> +            None if pkg_list[pkg]["type"] == "rootfs" \
> +            else "virtual" if pkg_list[pkg]["virtual"] \
> +            else pkg_list[pkg]["version"]
>  
>      return (deps, rdeps, types, versions)
> -- 
> 2.14.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 00/12 v3] infra: add solution to dump metadata from packages (branch yem/show-info-2)

[Arnout Vandecappelle]

On 15/04/2019 21:47, Yann E. MORIN wrote:
> Yann E. MORIN (12):
>       infra/pkg-download: return just a list of URIs
>       infra/pkg-download: make the URI list a callable macro
>       infra/pkg-download: get rid of the FLOCK variable
>       infra/pkg-download: make the DOWNLOAD macro fully parameterised
>       infra/utils: add helper to generate comma-separated lists
>       fs: introduce variables with name and type
>       fs: introduce variable with all recursive dependencies
>       fs: add all recursive dependencies to packages list
>       core: introduce new global show-info
>       core: add per-package and per-filesystem show-info

 1-10 applied to master, thanks.

>       support/scripts: use show-info to extract dependency graph
>       core: remove show-depednency-tree

 I never looked at these before so that might take some time still.

 Regards,
 Arnout

[Buildroot] [PATCH 11/12 v3] support/scripts: use show-info to extract dependency graph

[Ricardo Martincoski]

Hello,

One question related to an output that changes with this patch and also some
nits.

On Mon, Apr 15, 2019 at 04:47 PM, Yann E. MORIN wrote:

> Currently, we extract the dependency graph from the aptly named but
> ad-hoc show-dependency-graph rule.
> 
> We now have a better solution to report package information, with
> show-info.
> 
> Since show-dependency-graph never went into a release so far, and
> show-info does provide the same (and more), swith to using show-info.

s/swith/switch/

> 
> Thanks to Adam for suggesting the coding style to have a readable code
> that is not ugly but still pleases flake8. Thanks to Arnout for
> suggesting the use of dict.get() to further simplify the code.
> 
> Note: we do not use th reverse_dependencies field because it only

s/th/the/

> contains those packages that have a kconfig option, so we'd miss most
> host packages.

When I do this before and after this patch:
$ make qemu_arm_versatile_defconfig
$ make graph-depends
The color for "rootfs-common" and "rootfs-ext2" changes.
Can you reproduce it?
Is this intended?
If not intended, do we care enough?

[snip]
> +        versions[pkg] = \
> +            None if pkg_list[pkg]["type"] == "rootfs" \
> +            else "virtual" if pkg_list[pkg]["virtual"] \
> +            else pkg_list[pkg]["version"]

Why not the straightforward version?
        if pkg_list[pkg]["type"] == "rootfs":
            versions[pkg] = None
        elif pkg_list[pkg]["virtual"]:
            versions[pkg] = "virtual"
        else:
            versions[pkg] = pkg_list[pkg]["version"]


Regards,
Ricardo

[Buildroot] [PATCH 11/12 v3] support/scripts: use show-info to extract dependency graph

[Yann E. MORIN]

Ricardo, All,

[Typoes fixed, thanks.]

On 2019-04-21 21:14 -0300, Ricardo Martincoski spake thusly:
> On Mon, Apr 15, 2019 at 04:47 PM, Yann E. MORIN wrote:
> > Since show-dependency-graph never went into a release so far, and
> > show-info does provide the same (and more), swith to using show-info.
[--SNIP--]
> When I do this before and after this patch:
> $ make qemu_arm_versatile_defconfig
> $ make graph-depends
> The color for "rootfs-common" and "rootfs-ext2" changes.
> Can you reproduce it?

Yes.

> Is this intended?

More or less, yes.

> If not intended, do we care enough?

IMHO, not really.

However, I still have further improvements to do about the graphs, and
one of them was to add a new 'colour' to rootfs items.

> 
> [snip]
> > +        versions[pkg] = \
> > +            None if pkg_list[pkg]["type"] == "rootfs" \
> > +            else "virtual" if pkg_list[pkg]["virtual"] \
> > +            else pkg_list[pkg]["version"]
> 
> Why not the straightforward version?
>         if pkg_list[pkg]["type"] == "rootfs":
>             versions[pkg] = None
>         elif pkg_list[pkg]["virtual"]:
>             versions[pkg] = "virtual"
>         else:
>             versions[pkg] = pkg_list[pkg]["version"]

Because this is not pythonic!

Seriously, yes, the main reason is to write python scripts in the most
pythonic way I can, for two reasons:

 1- I try to use the idioms and best practices of the language I write
    in, so that those that come later and are pofficient in the language
    can maintain it without having to cleanup before;

 2- I learn python along the way...

So yes, this looks very more pythonic than the if-else-blocs which look
much more like C or shell...

And it takes 6 lines instead of 4! ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'