[PATCH v3 0/4] fit: allow signing with an OpenSSL engine

[Quentin Schulz <foss+uboot@0leil.net>]

I have a couple of products whose U-Boot FIT is signed via a proprietary
OpenSSL engine which only expects the name of a "slot" to select the key
to sign data with.

Currently mkimage fit support expects either a key-dir (-k) or a
key-file (-G) as a toggle for signing, however this doesn't apply to our
usecase because we use an OpenSSL engine (so no key-file to provide)
which doesn't mimic a directory layout like key-dir implies. Moreover,
binman really expects private keys (.key extension) to be available in
this key-dir directory, which we of course cannot provide.

This series allows to sign a FIT image with mkimage (and binman) with
an OpenSSL engine, including PKCS11 and custom engines. If a key-dir
needs to be passed (which is typical for PKCS11), one can do so by using
fit,engine-keydir.

Note that the public key (.crt extension) still needs to be available if
one wants to embed it for signature verification (which is probably what
one wants to do :) ). It is probably possible to use the engine for
getting the public key instead of storing it on disk, but this needs to
be added to fdt_add_pubkey and then binman, through a mechanism
different from fit,engine*.

One issue though is that since binman resolves key paths absolutely and
that I don't believe an OpenSSL engine would happen to have the exact
same key_id value than a local absolute path, fit,encrypt and
fit,engine cannot cohabit. An issue for the next person who wants
an OpenSSL engine AND encrypt the same FIT image, I don't.

Note that LibreSSL supports neither engines nor providers as far as I
could tell (engine support has been explicitly removed).

Note that OpenSSL engines have been deprecated since 3.0 (Q3-2021),
however note that OpenSSL 3.5 still seems to support engines (git grep)
and is EOL end of Q1 2030.

+Cc Eddie who's working on OpenSSL provider support, maybe we can work
together on supporting both engines and providers via the same API we
expose to the user?
+Cc Wolfgang who seems interested in signing with PKCS11 engine which
should now be supported.
+Cc Peter who seems interested according to v1

If anyone has an idea on how to test PKCS11 with SOftHSMv2 with id=
passed in fit,engine-keydir, I'm all ears.

I'm also wondering if the explanation around fit,engine-keydir aren't
too much. After all, they are passed verbatim to mkimage as -k argument
and the special cases are all specific to mkimage and not binman.

This depends on https://lore.kernel.org/all/20251121-pkcs11-ci-v1-1-8ba6da56c2f5@cherry.de/T/#u

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
Changes in v3:
- did NOT add T-b and R-B on patch 3 and 4 since there's been some
  rework,
- added R-b on patch 2,
- reworded commit log of patch 3,
- reworded documentation of fit,engine and the special case of pkcs11 to
  hopefully be easier to digest,
- put fit,engine and fit,engine-keydir extraction into a separate
  function,
- fixed issues due to modification of the environment in tests failing
  other tests, by using unittest.mock.patch.dict() on os.environ as
  suggested by the unittest.mock doc,
- added test for fit,encrypt + fit,engine raising an issue to keep 100%
  coverage for fit etype,
- fixed issue when running binman with only one process (-T or -P 1) by
  renaming uniquely files and directories created in the input dir in
  individual tests as it is shared among all tests run by a process,
- removed apt-get install of the lib packages in the test as
  Bintool.apt_install() is racey (apt itself is...) and would fail when
  running binman with multiple processes. The package is now part of the
  CI container (well, in a different series). If it's not detected, the
  test is skipped.
- Link to v2: https://patch.msgid.link/20251117-binman-engine-v2-0-e053f683c0b8@cherry.de

Changes in v2:
- added R-b on patch 1,
- did NOT add T-b and R-B on patch 3 since there's been some rework,
- added binman test for signing with dummy RSA engine,
- added binman test for signing with SoftHSMv2 PKCS11 engine,
- added binman test for signing with SoftHSMv2 PKCS11 engine and
  specific keydir (which contains an object= to tackle that special
  case)
- added dummy RSA engine for binman test,
- renamed property to fit,engine as my hunch is that it could be used
  for encrypting as well (from the same engine with the same keydir but
  likely a different key-name-hint so that a different keypair is used)
- added fit,engine-keydir to easily support PKCS11 with arguments other
  than object= taken from key-name-hint,
- Link to v1: https://patch.msgid.link/20251031-binman-engine-v1-0-c13c1b5dac43@cherry.de

---
Quentin Schulz (4):
      fit: support signing with only an engine_id
      tools: binman: mkimage: add support for passing the engine
      tools: binman: fit: add support for OpenSSL engines
      tools: binman: fit: add tests for signing with an OpenSSL engine

 tools/binman/btool/mkimage.py                      |   5 +-
 tools/binman/btool/softhsm2_util.py                |  21 ++
 tools/binman/entries.rst                           |  54 ++++-
 tools/binman/etype/fit.py                          |  93 ++++++++-
 tools/binman/ftest.py                              | 223 +++++++++++++++++++++
 tools/binman/test/340_dummy-rsa4096.crt            |  31 +++
 tools/binman/test/340_fit_signature_engine.dts     |  99 +++++++++
 .../test/340_fit_signature_engine_encrypt.dts      | 100 +++++++++
 .../test/340_fit_signature_engine_pkcs11.dts       |  99 +++++++++
 .../340_fit_signature_engine_pkcs11_object.dts     | 100 +++++++++
 tools/binman/test/340_openssl.conf                 |  10 +
 tools/binman/test/340_softhsm2.conf                |  16 ++
 tools/binman/test/Makefile                         |   6 +-
 tools/binman/test/dummy-rsa-engine.c               | 149 ++++++++++++++
 tools/fit_image.c                                  |   3 +-
 tools/image-host.c                                 |   4 +-
 16 files changed, 1001 insertions(+), 12 deletions(-)
---
base-commit: a642f401d700034c82128defc2d6f9c18065de1f
change-id: 20251030-binman-engine-e349b02696d0

Best regards,
-- 
Quentin Schulz <quentin.schulz@cherry.de>