[PATCH v4 0/8] Record and rework component dependencies

Anatoly Burakov
2024-11-26T14:40:04Z

As part of the meson build, we can record the dependencies for each
component as we process it, logging them to a file. This file can be
used as input to a number of other scripts and tools, for example, to
graph the dependencies, or to allow higher-level build-config tools to
automatically enable component requirements, etc.

The first patch of this set separates dependencies inside meson into
optional or mandatory. The second patch of this set generates the basic
dependency tree. The third patch does some processing of that dependency
tree to identify cases where dependencies are being unnecessarily
specified. Reducing these makes it easier to have readable dependency
graphs in future, without affecting the build.

The following 4 patches are based on the output of the second patch, and
greatly cut down the number of direct dependency links between
components. Even with the cut-down dependencies, the full dependency
graph is nigh-unreadable, so the final patch adds a new script to
generate dependency tree subgraphs, creating dot files for e.g. the
dependencies of a particular component, or a component class such as
mempool drivers.

v3 -> v4:
- Update to latest main

v2 -> v3:
- Split dependencies into optional and mandatory
- Fixup graph scripts to read and generate graphs that encode optional
dependencies into the graph
- Python version fixes to avoid using features not available in minimum
supported Python version
- Formatting with Ruff, and PEP-484 compliance

Anatoly Burakov (1):
build: split dependencies into mandatory and optional

Bruce Richardson (7):
build: output a dependency log in build directory
devtools: add script to flag unneeded dependencies
build: remove kvargs from driver class dependencies
build: reduce library dependencies
build: reduce driver dependencies
build: reduce app dependencies
devtools: add script to generate DPDK dependency graphs

app/dumpcap/meson.build | 2 +-
app/graph/meson.build | 2 +-
app/meson.build | 11 +-
app/pdump/meson.build | 2 +-
app/proc-info/meson.build | 4 +-
app/test-bbdev/meson.build | 8 +-
app/test-crypto-perf/meson.build | 4 +-
app/test-fib/meson.build | 2 +-
app/test-pmd/meson.build | 26 +--
app/test-sad/meson.build | 2 +-
app/test/meson.build | 14 +-
buildtools/log-deps.py | 81 ++++++++
buildtools/meson.build | 2 +
devtools/draw-dependency-graphs.py | 223 +++++++++++++++++++++
devtools/find-duplicate-deps.py | 53 +++++
drivers/baseband/fpga_5gnr_fec/meson.build | 2 +-
drivers/baseband/fpga_lte_fec/meson.build | 2 +-
drivers/baseband/la12xx/meson.build | 2 +-
drivers/baseband/null/meson.build | 2 +-
drivers/baseband/turbo_sw/meson.build | 2 +-
drivers/bus/auxiliary/meson.build | 2 -
drivers/bus/dpaa/meson.build | 2 +-
drivers/bus/fslmc/meson.build | 2 +-
drivers/bus/ifpga/meson.build | 2 +-
drivers/bus/pci/meson.build | 4 +-
drivers/bus/platform/meson.build | 1 -
drivers/bus/uacce/meson.build | 2 -
drivers/bus/vdev/meson.build | 2 -
drivers/common/cnxk/meson.build | 4 +-
drivers/common/cpt/meson.build | 2 +-
drivers/common/idpf/meson.build | 2 +-
drivers/common/mlx5/meson.build | 2 +-
drivers/compress/mlx5/meson.build | 2 +-
drivers/compress/nitrox/meson.build | 2 +-
drivers/compress/octeontx/meson.build | 2 +-
drivers/crypto/bcmfs/meson.build | 2 +-
drivers/crypto/cnxk/meson.build | 2 +-
drivers/crypto/dpaa_sec/meson.build | 2 +-
drivers/crypto/ipsec_mb/meson.build | 2 +-
drivers/crypto/mlx5/meson.build | 2 +-
drivers/crypto/nitrox/meson.build | 2 +-
drivers/dma/cnxk/meson.build | 2 +-
drivers/dma/dpaa/meson.build | 2 +-
drivers/dma/dpaa2/meson.build | 2 +-
drivers/dma/odm/meson.build | 2 +-
drivers/dma/skeleton/meson.build | 2 +-
drivers/event/cnxk/meson.build | 2 +-
drivers/event/dlb2/meson.build | 2 +-
drivers/event/dpaa2/meson.build | 2 +-
drivers/event/meson.build | 2 +-
drivers/event/octeontx/meson.build | 3 +-
drivers/event/sw/meson.build | 2 +-
drivers/mempool/cnxk/meson.build | 2 +-
drivers/mempool/dpaa/meson.build | 2 +-
drivers/mempool/dpaa2/meson.build | 2 +-
drivers/mempool/octeontx/meson.build | 2 +-
drivers/meson.build | 9 +-
drivers/net/cnxk/meson.build | 4 +-
drivers/net/iavf/meson.build | 2 +-
drivers/net/ice/meson.build | 2 +-
drivers/net/mana/meson.build | 2 +-
drivers/net/meson.build | 2 +-
drivers/net/mlx5/meson.build | 2 +-
drivers/net/sfc/meson.build | 2 +-
drivers/net/softnic/meson.build | 2 +-
drivers/raw/cnxk_bphy/meson.build | 2 +-
drivers/raw/cnxk_gpio/meson.build | 2 +-
drivers/raw/ntb/meson.build | 2 +-
drivers/raw/skeleton/meson.build | 2 +-
drivers/regex/meson.build | 2 +-
drivers/regex/mlx5/meson.build | 2 +-
drivers/vdpa/ifc/meson.build | 2 +-
drivers/vdpa/meson.build | 3 +-
drivers/vdpa/mlx5/meson.build | 2 +-
drivers/vdpa/sfc/meson.build | 2 +-
examples/ethtool/meson.build | 2 +-
examples/l2fwd-crypto/meson.build | 2 +-
examples/l3fwd/meson.build | 2 +-
examples/meson.build | 11 +-
examples/vm_power_manager/meson.build | 6 +-
lib/argparse/meson.build | 2 +-
lib/bbdev/meson.build | 2 +-
lib/bitratestats/meson.build | 2 +-
lib/bpf/meson.build | 2 +-
lib/cmdline/meson.build | 2 +-
lib/compressdev/meson.build | 2 +-
lib/cryptodev/meson.build | 2 +-
lib/dispatcher/meson.build | 2 +-
lib/distributor/meson.build | 2 +-
lib/dmadev/meson.build | 2 -
lib/eal/meson.build | 5 +-
lib/efd/meson.build | 2 +-
lib/ethdev/meson.build | 2 +-
lib/eventdev/meson.build | 3 +-
lib/fib/meson.build | 4 +-
lib/gpudev/meson.build | 2 +-
lib/graph/meson.build | 2 +-
lib/gro/meson.build | 2 +-
lib/gso/meson.build | 2 +-
lib/hash/meson.build | 4 +-
lib/ip_frag/meson.build | 2 +-
lib/ipsec/meson.build | 2 +-
lib/kvargs/meson.build | 2 +-
lib/latencystats/meson.build | 2 +-
lib/lpm/meson.build | 3 +-
lib/mbuf/meson.build | 2 +-
lib/member/meson.build | 2 +-
lib/mempool/meson.build | 2 +-
lib/meson.build | 8 +-
lib/metrics/meson.build | 2 +-
lib/mldev/meson.build | 2 +-
lib/net/meson.build | 2 +-
lib/node/meson.build | 2 +-
lib/pcapng/meson.build | 2 +-
lib/pdcp/meson.build | 2 +-
lib/pdump/meson.build | 2 +-
lib/pipeline/meson.build | 2 +-
lib/port/meson.build | 2 +-
lib/power/meson.build | 3 +-
lib/rawdev/meson.build | 2 -
lib/rcu/meson.build | 2 +-
lib/regexdev/meson.build | 2 +-
lib/reorder/meson.build | 2 +-
lib/rib/meson.build | 2 +-
lib/ring/meson.build | 1 -
lib/sched/meson.build | 2 +-
lib/security/meson.build | 2 +-
lib/table/meson.build | 2 +-
lib/telemetry/meson.build | 2 +-
lib/vhost/meson.build | 2 +-
130 files changed, 533 insertions(+), 175 deletions(-)
create mode 100644 buildtools/log-deps.py
create mode 100755 devtools/draw-dependency-graphs.py
create mode 100755 devtools/find-duplicate-deps.py

--
2.43.5