[PATCH v2 00/38] binman: Add support for bintools and missing tools

[Simon Glass <sjg@chromium.org>]

At present binman uses binary tools (like cbfstool, futiltiy, lz4) in an
ad-hoc manner. Various parts of binman use tools.Run() to run tools as
needed. If a tool is missing, an error is produced and binman stops.

However this is not ideal. CI systems want to be able to complete the
build, even if tools are missing. Ideally binman would deal with missing
binary tools the same way it deals with missing binary blobs: make a note
of it and move on.

This series introduces this feature to binman.

`Bintool` is the name binman gives to a binary tool which it uses to
create and manipulate binaries that binman cannot handle itself.

Binman provides various features to manage bintools:

- Determining whether the tool is currently installed
- Downloading or building the tool
- Determining the version of the tool that is installed
- Deciding which tools are needed to build an image

As with external blobs, bintools (which are like 'external' tools) can be
missing. When building an image requires a bintool and it is not
installed, binman detects this and reports the problem, but continues to
build an image.

Of course the image will not work, but binman reports which bintools are needed
and also provide a way to fetch them.

The final patch shows how this works in practice with a chosen board. The
Odroid-C2 is quite a complicated image with many steps. It is an ideal
example for how Binman can be used.

The series is available at u-boot-dm/bin-working

Changes in v2:
- Substantial rewrite, introducing the concept of bintools

Simon Glass (38):
  Makefile: Fake external blobs by default with binman
  binman: Tweak elf tests for a toolchain change
  mkimage: Show the external-offset error
  binman: Expand the external FIT test a little
  patman: Allow running a tool and returning the full result
  buildman: Move the download function to tools
  patman: Tidy up the download function a little
  patman: Add a function to find a tool on the path
  binman: Write fake blobs to the output directory
  binman: Drop the image name from the fake-blob message
  binman: Allow faked blobs in blob-ext-list
  binman: Correct path for fip_util
  binman: Add installation instructions
  binman: Add support for bintools
  binman: Plumb in support for bintools
  binman: Add tests for bintool
  binman: Add a bintool implementation for cbfstool
  binman: Add a bintool implementation for fiptool
  binman: Add a bintool implementation for futility
  binman: Add a bintool implementation for ifwitool
  binman: Add a bintool implementation for mkimage
  binman: Enable bintool tests including cmdline processing
  binman: Convert to using the CBFS bintool
  binman: Convert to using the FIP bintool
  binman: Convert to using the futility bintool
  binman: Convert to using the ifwitool bintool
  binman: Convert to using the mkimage bintool
  binman: Move compression into binman
  binman: Tidy up pylint warnings in comp_util
  binman: Add a bintool implementation for lz4
  binman: Convert to using the lz4 bintool
  binman: Add a bintool implementation for lzma_alone
  binman: Convert to using the lzma_alone bintool
  binman: Plumb in support for missing bintools
  binman: Complete test coverage of comp_util
  binman: Add a command to generate bintool docs
  binman: Add documentation for bintools
  RFC: Move Odroid-C2 to use binman to produce the image

 Makefile                                     |   2 +-
 arch/arm/dts/meson-sm1-odroid-c4-u-boot.dtsi | 107 +++++
 arch/arm/mach-meson/Kconfig                  |   1 +
 doc/board/amlogic/odroid-c4.rst              | 127 ++---
 doc/develop/package/bintools.rst             |   1 +
 tools/binman/binman.rst                      |  98 +++-
 tools/binman/bintool.py                      | 466 +++++++++++++++++++
 tools/binman/bintool_test.py                 | 353 ++++++++++++++
 tools/binman/bintools.rst                    | 139 ++++++
 tools/binman/btool/_aml_common.py            |  47 ++
 tools/binman/btool/_testing.py               |  36 ++
 tools/binman/btool/aml_encrypt_g12a.py       |  82 ++++
 tools/binman/btool/aml_encrypt_g12b.py       |  83 ++++
 tools/binman/btool/cbfstool.py               | 219 +++++++++
 tools/binman/btool/fiptool.py                | 123 +++++
 tools/binman/btool/futility.py               | 178 +++++++
 tools/binman/btool/ifwitool.py               | 166 +++++++
 tools/binman/btool/lz4.py                    | 140 ++++++
 tools/binman/btool/lzma_alone.py             | 126 +++++
 tools/binman/btool/mkimage.py                |  80 ++++
 tools/binman/cbfs_util.py                    |  33 +-
 tools/binman/cbfs_util_test.py               |  59 +--
 tools/binman/cmdline.py                      |  12 +
 tools/binman/comp_util.py                    |  76 +++
 tools/binman/control.py                      |  50 +-
 tools/binman/elf_test.py                     |   8 +-
 tools/binman/entries.rst                     | 138 ++++++
 tools/binman/entry.py                        |  67 ++-
 tools/binman/etype/aml_encrypt.py            | 262 +++++++++++
 tools/binman/etype/blob.py                   |   8 +-
 tools/binman/etype/blob_ext_list.py          |   1 +
 tools/binman/etype/fit.py                    |  20 +-
 tools/binman/etype/gbb.py                    |  37 +-
 tools/binman/etype/intel_ifwi.py             |  25 +-
 tools/binman/etype/mkimage.py                |  13 +-
 tools/binman/etype/section.py                |  19 +-
 tools/binman/etype/vblock.py                 |  32 +-
 tools/binman/fip_util.py                     |  26 --
 tools/binman/fip_util_test.py                |  25 +-
 tools/binman/ftest.py                        | 173 ++++++-
 tools/binman/image.py                        |  14 +
 tools/binman/main.py                         |  11 +-
 tools/binman/missing-blob-help               |   6 +
 tools/binman/test/162_fit_external.dts       |   2 +-
 tools/binman/test/213_aml_encrypt.dts        |  51 ++
 tools/binman/test/214_list_no_dtb.dts        |  23 +
 tools/binman/test/218_blob_ext_list_fake.dts |  14 +
 tools/buildman/toolchain.py                  |  46 +-
 tools/fit_image.c                            |   5 +-
 tools/patman/tools.py                        | 238 +++++-----
 50 files changed, 3619 insertions(+), 449 deletions(-)
 create mode 120000 doc/develop/package/bintools.rst
 create mode 100644 tools/binman/bintool.py
 create mode 100644 tools/binman/bintool_test.py
 create mode 100644 tools/binman/bintools.rst
 create mode 100644 tools/binman/btool/_aml_common.py
 create mode 100644 tools/binman/btool/_testing.py
 create mode 100644 tools/binman/btool/aml_encrypt_g12a.py
 create mode 100644 tools/binman/btool/aml_encrypt_g12b.py
 create mode 100644 tools/binman/btool/cbfstool.py
 create mode 100644 tools/binman/btool/fiptool.py
 create mode 100644 tools/binman/btool/futility.py
 create mode 100644 tools/binman/btool/ifwitool.py
 create mode 100644 tools/binman/btool/lz4.py
 create mode 100644 tools/binman/btool/lzma_alone.py
 create mode 100644 tools/binman/btool/mkimage.py
 create mode 100644 tools/binman/comp_util.py
 create mode 100644 tools/binman/etype/aml_encrypt.py
 create mode 100644 tools/binman/test/213_aml_encrypt.dts
 create mode 100644 tools/binman/test/214_list_no_dtb.dts
 create mode 100644 tools/binman/test/218_blob_ext_list_fake.dts

-- 
2.34.1.575.g55b058a8bb-goog