From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3288C433EF for ; Mon, 10 Jan 2022 03:14:48 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2994581428; Mon, 10 Jan 2022 04:14:45 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="NNeooEeO"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 095C8811DF; Mon, 10 Jan 2022 04:14:43 +0100 (CET) Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 5A109811E3 for ; Mon, 10 Jan 2022 04:14:35 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd31.google.com with SMTP id y70so15794100iof.2 for ; Sun, 09 Jan 2022 19:14:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=1TMwm7E5cr3XiHlWVKROw6CUm7jBLq1wk0e9chaSu6M=; b=NNeooEeOXcglGCcdSY65X2TdeXo1qJOTcVxGHuVxws1Jjr6RpJYH8MjXC2wtRzSqxB C2LLIZZh4bhpFofuvsxsMRmI8Udx+KEF5hAzDKqLFQ/slIK7oTGcwYhDQMwcXlpwKCum ICr0svOK48SqEuZ4hxTi0glIFdfdzjVwNXVHo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=1TMwm7E5cr3XiHlWVKROw6CUm7jBLq1wk0e9chaSu6M=; b=hiN6Z9wRnYNwS3Xort6kPAdqnXi27pFPG8qQSXfBPRDlarMOFDnVfpiPYYBW5r6nfc ZGF3Q686RuXibwY/yeDfsj7R4+zJRIemvMlNZAL2wy5msj4ysXFn9MJ9zsPUGppTgnlO alAn7A1jYz6Bhb1ZMYzLRkQRwkluJ1J+azo+PrApz2iLUvJjyiBvFWsbO3/+QSkeXWMC oqyH9lEFoCcLK22nXkb83udMQB8FNefPp5VneOftkLSwGfRq+DVGkD5ZLQHR9hjiAa9G Tw1x9DUQ2KpHz6xYQbt4ZrW9UmfPn0YLk4qUieYmee4MJ2scUN4APPHCqUeNOrO0quL0 Qf2w== X-Gm-Message-State: AOAM531Y/mrd1VLgJDtr8kd1A9h1DYWp3ihvZTYdkbxcNMGzxyEytn/x bjX5i/V8q85mHwSipjp7l4NDRMcFNo0IHA== X-Google-Smtp-Source: ABdhPJzH7tfIVIZHBvI9kJXq+nQn2JiBdR8TOwrj4GXJidKhvTku9QMP5OtMTbspA92K6jjfncBTnQ== X-Received: by 2002:a05:6638:240b:: with SMTP id z11mr34621490jat.121.1641784473748; Sun, 09 Jan 2022 19:14:33 -0800 (PST) Received: from sjg1.roam.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id d16sm3542182ioy.29.2022.01.09.19.14.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Jan 2022 19:14:33 -0800 (PST) From: Simon Glass To: U-Boot Mailing List Cc: Simon Glass , Joe Hershberger , Marek Vasut , Masahiro Yamada , Stefano Babic , u-boot-custodians@lists.denx.de Subject: [PATCH v2 00/38] binman: Add support for bintools and missing tools Date: Sun, 9 Jan 2022 20:13:35 -0700 Message-Id: <20220110031413.1970836-1-sjg@chromium.org> X-Mailer: git-send-email 2.34.1.575.g55b058a8bb-goog MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean At present binman uses binary tools (like cbfstool, futiltiy, lz4) in an ad-hoc manner. Various parts of binman use tools.Run() to run tools as needed. If a tool is missing, an error is produced and binman stops. However this is not ideal. CI systems want to be able to complete the build, even if tools are missing. Ideally binman would deal with missing binary tools the same way it deals with missing binary blobs: make a note of it and move on. This series introduces this feature to binman. `Bintool` is the name binman gives to a binary tool which it uses to create and manipulate binaries that binman cannot handle itself. Binman provides various features to manage bintools: - Determining whether the tool is currently installed - Downloading or building the tool - Determining the version of the tool that is installed - Deciding which tools are needed to build an image As with external blobs, bintools (which are like 'external' tools) can be missing. When building an image requires a bintool and it is not installed, binman detects this and reports the problem, but continues to build an image. Of course the image will not work, but binman reports which bintools are needed and also provide a way to fetch them. The final patch shows how this works in practice with a chosen board. The Odroid-C2 is quite a complicated image with many steps. It is an ideal example for how Binman can be used. The series is available at u-boot-dm/bin-working Changes in v2: - Substantial rewrite, introducing the concept of bintools Simon Glass (38): Makefile: Fake external blobs by default with binman binman: Tweak elf tests for a toolchain change mkimage: Show the external-offset error binman: Expand the external FIT test a little patman: Allow running a tool and returning the full result buildman: Move the download function to tools patman: Tidy up the download function a little patman: Add a function to find a tool on the path binman: Write fake blobs to the output directory binman: Drop the image name from the fake-blob message binman: Allow faked blobs in blob-ext-list binman: Correct path for fip_util binman: Add installation instructions binman: Add support for bintools binman: Plumb in support for bintools binman: Add tests for bintool binman: Add a bintool implementation for cbfstool binman: Add a bintool implementation for fiptool binman: Add a bintool implementation for futility binman: Add a bintool implementation for ifwitool binman: Add a bintool implementation for mkimage binman: Enable bintool tests including cmdline processing binman: Convert to using the CBFS bintool binman: Convert to using the FIP bintool binman: Convert to using the futility bintool binman: Convert to using the ifwitool bintool binman: Convert to using the mkimage bintool binman: Move compression into binman binman: Tidy up pylint warnings in comp_util binman: Add a bintool implementation for lz4 binman: Convert to using the lz4 bintool binman: Add a bintool implementation for lzma_alone binman: Convert to using the lzma_alone bintool binman: Plumb in support for missing bintools binman: Complete test coverage of comp_util binman: Add a command to generate bintool docs binman: Add documentation for bintools RFC: Move Odroid-C2 to use binman to produce the image Makefile | 2 +- arch/arm/dts/meson-sm1-odroid-c4-u-boot.dtsi | 107 +++++ arch/arm/mach-meson/Kconfig | 1 + doc/board/amlogic/odroid-c4.rst | 127 ++--- doc/develop/package/bintools.rst | 1 + tools/binman/binman.rst | 98 +++- tools/binman/bintool.py | 466 +++++++++++++++++++ tools/binman/bintool_test.py | 353 ++++++++++++++ tools/binman/bintools.rst | 139 ++++++ tools/binman/btool/_aml_common.py | 47 ++ tools/binman/btool/_testing.py | 36 ++ tools/binman/btool/aml_encrypt_g12a.py | 82 ++++ tools/binman/btool/aml_encrypt_g12b.py | 83 ++++ tools/binman/btool/cbfstool.py | 219 +++++++++ tools/binman/btool/fiptool.py | 123 +++++ tools/binman/btool/futility.py | 178 +++++++ tools/binman/btool/ifwitool.py | 166 +++++++ tools/binman/btool/lz4.py | 140 ++++++ tools/binman/btool/lzma_alone.py | 126 +++++ tools/binman/btool/mkimage.py | 80 ++++ tools/binman/cbfs_util.py | 33 +- tools/binman/cbfs_util_test.py | 59 +-- tools/binman/cmdline.py | 12 + tools/binman/comp_util.py | 76 +++ tools/binman/control.py | 50 +- tools/binman/elf_test.py | 8 +- tools/binman/entries.rst | 138 ++++++ tools/binman/entry.py | 67 ++- tools/binman/etype/aml_encrypt.py | 262 +++++++++++ tools/binman/etype/blob.py | 8 +- tools/binman/etype/blob_ext_list.py | 1 + tools/binman/etype/fit.py | 20 +- tools/binman/etype/gbb.py | 37 +- tools/binman/etype/intel_ifwi.py | 25 +- tools/binman/etype/mkimage.py | 13 +- tools/binman/etype/section.py | 19 +- tools/binman/etype/vblock.py | 32 +- tools/binman/fip_util.py | 26 -- tools/binman/fip_util_test.py | 25 +- tools/binman/ftest.py | 173 ++++++- tools/binman/image.py | 14 + tools/binman/main.py | 11 +- tools/binman/missing-blob-help | 6 + tools/binman/test/162_fit_external.dts | 2 +- tools/binman/test/213_aml_encrypt.dts | 51 ++ tools/binman/test/214_list_no_dtb.dts | 23 + tools/binman/test/218_blob_ext_list_fake.dts | 14 + tools/buildman/toolchain.py | 46 +- tools/fit_image.c | 5 +- tools/patman/tools.py | 238 +++++----- 50 files changed, 3619 insertions(+), 449 deletions(-) create mode 120000 doc/develop/package/bintools.rst create mode 100644 tools/binman/bintool.py create mode 100644 tools/binman/bintool_test.py create mode 100644 tools/binman/bintools.rst create mode 100644 tools/binman/btool/_aml_common.py create mode 100644 tools/binman/btool/_testing.py create mode 100644 tools/binman/btool/aml_encrypt_g12a.py create mode 100644 tools/binman/btool/aml_encrypt_g12b.py create mode 100644 tools/binman/btool/cbfstool.py create mode 100644 tools/binman/btool/fiptool.py create mode 100644 tools/binman/btool/futility.py create mode 100644 tools/binman/btool/ifwitool.py create mode 100644 tools/binman/btool/lz4.py create mode 100644 tools/binman/btool/lzma_alone.py create mode 100644 tools/binman/btool/mkimage.py create mode 100644 tools/binman/comp_util.py create mode 100644 tools/binman/etype/aml_encrypt.py create mode 100644 tools/binman/test/213_aml_encrypt.dts create mode 100644 tools/binman/test/214_list_no_dtb.dts create mode 100644 tools/binman/test/218_blob_ext_list_fake.dts -- 2.34.1.575.g55b058a8bb-goog