[PATCH v5 0/8] scripts/make_fit: Support ramdisks and faster operations

[PATCH v5 0/8] scripts/make_fit: Support ramdisks and faster operations

[Simon Glass]

This series updates 'make image.fit' to support adding a ramdisk to the
FIT, either one provided as a parameter or one created from all the
kernel modules.

It also includes a few performance improvement, so that building a FIT
from ~450MB of kernel/module/devicetree files only takes a few seconds
on a modern machine.

Changes in v5:
- Fix 'use' typo
- Add a new patch to split out module targets into a variable
- Build modules automatically if needed (fix from Nicolas Schier)

Changes in v4:
- Update the commit message
- Provide the list of modules from the Makefile
- Reduce verbosity (don't print every module filename)
- Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
- Use an empty FIT_MODULES to disable the feature, instead of '0'
- Make use of the 'modules' dependency to ensure modules are built
- Pass the list of modules to the script

Changes in v3:
- Move the ramdisk chunk into the correct patch
- Add a comment at the top of the file about the -r option
- Count the ramdisk in the total files
- Update the commit message
- Add a way to add built modules into the FIT

Changes in v2:
- Don't compress the ramdisk as it is already compressed

Simon Glass (8):
  scripts/make_fit: Speed up operation
  scripts/make_fit: Support an initial ramdisk
  scripts/make_fit: Move dtb processing into a function
  scripts/make_fit: Provide a way to add built modules
  kbuild: Split out module targets into a variable
  kbuild: Allow adding modules into the FIT ramdisk
  scripts/make_fit: Support a few more parallel compressors
  scripts/make_fit: Compress dtbs in parallel

 Makefile             |   8 +-
 arch/arm64/Makefile  |   1 +
 scripts/Makefile.lib |  10 +-
 scripts/make_fit.py  | 264 +++++++++++++++++++++++++++++++++++++------
 4 files changed, 243 insertions(+), 40 deletions(-)

-- 
2.43.0

base-commit: 4a71531471926e3c391665ee9c42f4e0295a4585
branch: fita5

[PATCH v5 1/8] scripts/make_fit: Speed up operation

[Simon Glass]

The kernel is likely at least 16MB so we may as well use that as a step
size when reallocating space for the FIT in memory. Pack the FIT at the
end, so there is no wasted space.

This reduces the time to pack by an order of magnitude, or so.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
---

(no changes since v3)

Changes in v3:
- Move the ramdisk chunk into the correct patch

 scripts/make_fit.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 1683e5ec6e67..0f5e7c4b8aed 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -98,7 +98,7 @@ def setup_fit(fsw, name):
         fsw (libfdt.FdtSw): Object to use for writing
         name (str): Name of kernel image
     """
-    fsw.INC_SIZE = 65536
+    fsw.INC_SIZE = 16 << 20
     fsw.finish_reservemap()
     fsw.begin_node('')
     fsw.property_string('description', f'{name} with devicetree set')
@@ -299,7 +299,9 @@ def build_fit(args):
     finish_fit(fsw, entries)
 
     # Include the kernel itself in the returned file count
-    return fsw.as_fdt().as_bytearray(), seq + 1, size
+    fdt = fsw.as_fdt()
+    fdt.pack()
+    return fdt.as_bytearray(), seq + 1, size
 
 
 def run_make_fit():
-- 
2.43.0

[PATCH v5 2/8] scripts/make_fit: Support an initial ramdisk

[Simon Glass]

FIT (Flat Image Tree) allows a ramdisk to be included in each
configuration. Add support for this to the script.

This feature is not available via 'make image.fit' since the ramdisk
likely needs to be built separately anyway, e.g. using modules from
the kernel build. A later patch in this series provides support for
doing that.

Note that the uncompressed size is not correct when a ramdisk is used,
since it is too expensive to decompress the ramdisk.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
---

Changes in v5:
- Fix 'use' typo

Changes in v4:
- Update the commit message

Changes in v3:
- Add a comment at the top of the file about the -r option
- Count the ramdisk in the total files
- Update the commit message

Changes in v2:
- Don't compress the ramdisk as it is already compressed

 scripts/make_fit.py | 52 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 44 insertions(+), 8 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 0f5e7c4b8aed..984371f505bc 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -10,10 +10,14 @@
 Usage:
     make_fit.py -A arm64 -n 'Linux-6.6' -O linux
         -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
-        @arch/arm64/boot/dts/dtbs-list -E -c gzip
+        -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
+        -E -c gzip
 
-Creates a FIT containing the supplied kernel and a set of devicetree files,
-either specified individually or listed in a file (with an '@' prefix).
+Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
+devicetree files, either specified individually or listed in a file (with an
+'@' prefix).
+
+Use -r to specify an existing ramdisk/initrd file.
 
 Use -E to generate an external FIT (where the data is placed after the
 FIT data structure). This allows parsing of the data without loading
@@ -29,8 +33,6 @@ looks at the .cmd files produced by the kernel build.
 
 The resulting FIT can be booted by bootloaders which support FIT, such
 as U-Boot, Linuxboot, Tianocore, etc.
-
-Note that this tool does not yet support adding a ramdisk / initrd.
 """
 
 import argparse
@@ -81,6 +83,8 @@ def parse_args():
           help='Specifies the operating system')
     parser.add_argument('-k', '--kernel', type=str, required=True,
           help='Specifies the (uncompressed) kernel input file (.itk)')
+    parser.add_argument('-r', '--ramdisk', type=str,
+          help='Specifies the ramdisk/initrd input file')
     parser.add_argument('-v', '--verbose', action='store_true',
                         help='Enable verbose output')
     parser.add_argument('dtbs', type=str, nargs='*',
@@ -133,7 +137,28 @@ def write_kernel(fsw, data, args):
         fsw.property_u32('entry', 0)
 
 
-def finish_fit(fsw, entries):
+def write_ramdisk(fsw, data, args):
+    """Write out the ramdisk image
+
+    Writes a ramdisk node along with the required properties
+
+    Args:
+        fsw (libfdt.FdtSw): Object to use for writing
+        data (bytes): Data to write (possibly compressed)
+        args (Namespace): Contains necessary strings:
+            arch: FIT architecture, e.g. 'arm64'
+            fit_os: Operating Systems, e.g. 'linux'
+    """
+    with fsw.add_node('ramdisk'):
+        fsw.property_string('description', 'Ramdisk')
+        fsw.property_string('type', 'ramdisk')
+        fsw.property_string('arch', args.arch)
+        fsw.property_string('os', args.os)
+        fsw.property('data', data)
+        fsw.property_u32('load', 0)
+
+
+def finish_fit(fsw, entries, has_ramdisk=False):
     """Finish the FIT ready for use
 
     Writes the /configurations node and subnodes
@@ -143,6 +168,7 @@ def finish_fit(fsw, entries):
         entries (list of tuple): List of configurations:
             str: Description of model
             str: Compatible stringlist
+        has_ramdisk (bool): True if a ramdisk is included in the FIT
     """
     fsw.end_node()
     seq = 0
@@ -154,6 +180,8 @@ def finish_fit(fsw, entries):
                 fsw.property_string('description', model)
                 fsw.property('fdt', bytes(''.join(f'fdt-{x}\x00' for x in files), "ascii"))
                 fsw.property_string('kernel', 'kernel')
+                if has_ramdisk:
+                    fsw.property_string('ramdisk', 'ramdisk')
     fsw.end_node()
 
 
@@ -274,6 +302,14 @@ def build_fit(args):
     size += os.path.getsize(args.kernel)
     write_kernel(fsw, comp_data, args)
 
+    # Handle the ramdisk if provided. Compression is not supported as it is
+    # already compressed.
+    if args.ramdisk:
+        with open(args.ramdisk, 'rb') as inf:
+            data = inf.read()
+        size += len(data)
+        write_ramdisk(fsw, data, args)
+
     for fname in args.dtbs:
         # Ignore non-DTB (*.dtb) files
         if os.path.splitext(fname)[1] != '.dtb':
@@ -296,12 +332,12 @@ def build_fit(args):
 
         entries.append([model, compat, files_seq])
 
-    finish_fit(fsw, entries)
+    finish_fit(fsw, entries, bool(args.ramdisk))
 
     # Include the kernel itself in the returned file count
     fdt = fsw.as_fdt()
     fdt.pack()
-    return fdt.as_bytearray(), seq + 1, size
+    return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size
 
 
 def run_make_fit():
-- 
2.43.0

[PATCH v5 3/8] scripts/make_fit: Move dtb processing into a function

[Simon Glass]

Since build_fit() is getting quite long, move the dtb processing into a
separate function.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
---

(no changes since v1)

 scripts/make_fit.py | 67 +++++++++++++++++++++++++++++----------------
 1 file changed, 44 insertions(+), 23 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 984371f505bc..1a74a9dcd85e 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -277,6 +277,47 @@ def process_dtb(fname, args):
 
     return (model, compat, files)
 
+
+def _process_dtbs(args, fsw, entries, fdts):
+    """Process all DTB files and add them to the FIT
+
+    Args:
+        args: Program arguments
+        fsw: FIT writer object
+        entries: List to append entries to
+        fdts: Dictionary of processed DTBs
+
+    Returns:
+        tuple:
+            Number of files processed
+            Total size of files processed
+    """
+    seq = 0
+    size = 0
+    for fname in args.dtbs:
+        # Ignore non-DTB (*.dtb) files
+        if os.path.splitext(fname)[1] != '.dtb':
+            continue
+
+        try:
+            (model, compat, files) = process_dtb(fname, args)
+        except Exception as e:
+            sys.stderr.write(f'Error processing {fname}:\n')
+            raise e
+
+        for fn in files:
+            if fn not in fdts:
+                seq += 1
+                size += os.path.getsize(fn)
+                output_dtb(fsw, seq, fn, args.arch, args.compress)
+                fdts[fn] = seq
+
+        files_seq = [fdts[fn] for fn in files]
+        entries.append([model, compat, files_seq])
+
+    return seq, size
+
+
 def build_fit(args):
     """Build the FIT from the provided files and arguments
 
@@ -289,7 +330,6 @@ def build_fit(args):
             int: Number of configurations generated
             size: Total uncompressed size of data
     """
-    seq = 0
     size = 0
     fsw = libfdt.FdtSw()
     setup_fit(fsw, args.name)
@@ -310,34 +350,15 @@ def build_fit(args):
         size += len(data)
         write_ramdisk(fsw, data, args)
 
-    for fname in args.dtbs:
-        # Ignore non-DTB (*.dtb) files
-        if os.path.splitext(fname)[1] != '.dtb':
-            continue
-
-        try:
-            (model, compat, files) = process_dtb(fname, args)
-        except Exception as e:
-            sys.stderr.write(f"Error processing {fname}:\n")
-            raise e
-
-        for fn in files:
-            if fn not in fdts:
-                seq += 1
-                size += os.path.getsize(fn)
-                output_dtb(fsw, seq, fn, args.arch, args.compress)
-                fdts[fn] = seq
-
-        files_seq = [fdts[fn] for fn in files]
-
-        entries.append([model, compat, files_seq])
+    count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
+    size += fdt_size
 
     finish_fit(fsw, entries, bool(args.ramdisk))
 
     # Include the kernel itself in the returned file count
     fdt = fsw.as_fdt()
     fdt.pack()
-    return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size
+    return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
 
 
 def run_make_fit():
-- 
2.43.0

[PATCH v5 4/8] scripts/make_fit: Provide a way to add built modules

[Simon Glass]

Provide arguments to support building a ramdisk from a directory tree of
modules. This is a convenient way to try out a kernel with its modules.

This makes use of the cpio tool rather than attempting to use a python
module or our own code. The list of modules is provided in a file.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---

(no changes since v4)

Changes in v4:
- Provide the list of modules from the Makefile
- Reduce verbosity (don't print every module filename)

Changes in v3:
- Add a way to add built modules into the FIT

 scripts/make_fit.py | 98 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 90 insertions(+), 8 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 1a74a9dcd85e..3db129f40b20 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -13,11 +13,17 @@ Usage:
         -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
         -E -c gzip
 
+    # Build with modules ramdisk instead of external ramdisk:
+    make_fit.py -A arm64 -n 'Linux-6.17' -O linux
+        -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
+        -m module1.ko module2.ko module3.ko @arch/arm64/boot/dts/dtbs-list
+
 Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
 devicetree files, either specified individually or listed in a file (with an
 '@' prefix).
 
 Use -r to specify an existing ramdisk/initrd file.
+Use -m to build a ramdisk from specified kernel module files.
 
 Use -E to generate an external FIT (where the data is placed after the
 FIT data structure). This allows parsing of the data without loading
@@ -38,6 +44,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
 import argparse
 import collections
 import os
+import shutil
 import subprocess
 import sys
 import tempfile
@@ -83,8 +90,14 @@ def parse_args():
           help='Specifies the operating system')
     parser.add_argument('-k', '--kernel', type=str, required=True,
           help='Specifies the (uncompressed) kernel input file (.itk)')
-    parser.add_argument('-r', '--ramdisk', type=str,
+
+    # Create mutually exclusive group for ramdisk options
+    rd_group = parser.add_mutually_exclusive_group()
+    rd_group.add_argument('-r', '--ramdisk', type=str,
           help='Specifies the ramdisk/initrd input file')
+    rd_group.add_argument('-m', '--modules', type=str, nargs='+',
+          help='List of module filenames to include in ramdisk')
+
     parser.add_argument('-v', '--verbose', action='store_true',
                         help='Enable verbose output')
     parser.add_argument('dtbs', type=str, nargs='*',
@@ -240,6 +253,60 @@ def output_dtb(fsw, seq, fname, arch, compress):
         fsw.property('data', compressed)
 
 
+def build_ramdisk(args, tmpdir):
+    """Build a cpio ramdisk containing kernel modules
+
+    Similar to mkinitramfs, this creates a compressed cpio-archive containing
+    the kernel modules for the current kernel version.
+
+    Args:
+        args (Namespace): Program arguments
+        tmpdir (str): Temporary directory to use for modules installation
+
+    Returns:
+        tuple:
+            bytes: Compressed cpio data containing modules
+            int: total uncompressed size
+    """
+    suppress = None if args.verbose else subprocess.DEVNULL
+
+    if args.verbose:
+        print(f'Copying {len(args.modules)} modules to ramdisk')
+
+    # Create output-directory structure
+    outdir = os.path.join(tmpdir, 'initramfs')
+    modules_dir = os.path.join(outdir, 'usr', 'lib', 'modules')
+    os.makedirs(modules_dir, exist_ok=True)
+
+    # Copy in the specified modules
+    for module in args.modules:
+        dest_path = os.path.join(modules_dir, os.path.basename(module))
+        shutil.copy2(module, dest_path)
+
+    if args.verbose:
+        print(f'Creating cpio archive from {outdir}')
+
+    with tempfile.NamedTemporaryFile() as cpio_file:
+        # Change to initramfs directory and create cpio archive
+        with subprocess.Popen(['find', '.', '-print0'], cwd=outdir,
+                              stdout=subprocess.PIPE) as find:
+            with subprocess.Popen(['cpio', '-o', '-0', '-H', 'newc'],
+                                  stdin=find.stdout, stdout=cpio_file,
+                                  stderr=suppress, cwd=outdir) as cpio:
+                find.stdout.close()
+                cpio.wait()
+                find.wait()
+
+                if cpio.returncode != 0:
+                    raise RuntimeError('Failed to create cpio archive')
+
+        cpio_file.seek(0)  # Reset to beginning for reading
+        if args.verbose:
+            print('Reading ramdisk...' if args.compress == 'none' else
+                  f'Compressing ramdisk with {args.compress}...')
+        return compress_data(cpio_file, args.compress), cpio_file.tell()
+
+
 def process_dtb(fname, args):
     """Process an input DTB, decomposing it if requested and is possible
 
@@ -318,11 +385,12 @@ def _process_dtbs(args, fsw, entries, fdts):
     return seq, size
 
 
-def build_fit(args):
+def build_fit(args, tmpdir):
     """Build the FIT from the provided files and arguments
 
     Args:
         args (Namespace): Program arguments
+        tmpdir (str): Temporary directory for any temporary files
 
     Returns:
         tuple:
@@ -344,20 +412,29 @@ def build_fit(args):
 
     # Handle the ramdisk if provided. Compression is not supported as it is
     # already compressed.
+    ramdisk_data = None
     if args.ramdisk:
         with open(args.ramdisk, 'rb') as inf:
-            data = inf.read()
-        size += len(data)
-        write_ramdisk(fsw, data, args)
+            ramdisk_data = inf.read()
+        size += len(ramdisk_data)
+    elif args.modules:
+        if args.verbose:
+            print('Building modules ramdisk...')
+        ramdisk_data, uncomp_size = build_ramdisk(args, tmpdir)
+        size += uncomp_size
+
+    if ramdisk_data:
+        write_ramdisk(fsw, ramdisk_data, args)
 
     count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
     size += fdt_size
 
-    finish_fit(fsw, entries, bool(args.ramdisk))
+    finish_fit(fsw, entries, has_ramdisk=bool(ramdisk_data))
 
-    # Include the kernel itself in the returned file count
     fdt = fsw.as_fdt()
     fdt.pack()
+
+    # Count FDT files, kernel, plus ramdisk if present
     return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
 
 
@@ -365,7 +442,12 @@ def run_make_fit():
     """Run the tool's main logic"""
     args = parse_args()
 
-    out_data, count, size = build_fit(args)
+    tmpdir = tempfile.mkdtemp(prefix='make_fit_')
+    try:
+        out_data, count, size = build_fit(args, tmpdir)
+    finally:
+        shutil.rmtree(tmpdir)
+
     with open(args.output, 'wb') as outf:
         outf.write(out_data)
 
-- 
2.43.0

[PATCH v5 5/8] kbuild: Split out module targets into a variable

[Simon Glass]

Add a modules-targets variable to list the targets which cause modules
to be built, since we want to add a conditional target.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v5:
- Add a new patch to split out module targets into a variable

 Makefile | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 638bc09a546a..8cd46222fc48 100644
--- a/Makefile
+++ b/Makefile
@@ -772,7 +772,12 @@ endif
 # in addition to whatever we do anyway.
 # Just "make" or "make all" shall build modules as well
 
-ifneq ($(filter all modules nsdeps compile_commands.json clang-%,$(MAKECMDGOALS)),)
+modules-targets := all
+modules-targets += modules
+modules-targets += nsdeps
+modules-targets += compile_commands.json
+modules-targets += clang-%
+ifneq ($(filter $(modules-targets),$(MAKECMDGOALS)),)
   KBUILD_MODULES := y
 endif
 
-- 
2.43.0

[PATCH v5 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Simon Glass]

Support 'make image.fit FIT_MODULES=1' to put all the modules into a
ramdisk image within the FIT.

Add image.fit as a target which requires modules, so that modules will
built automatically when using FIT_MODULES=1

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
Acked-by: Nathan Chancellor <nathan@kernel.org>
---

Changes in v5:
- Build modules automatically if needed (fix from Nicolas Schier)

Changes in v4:
- Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
- Use an empty FIT_MODULES to disable the feature, instead of '0'
- Make use of the 'modules' dependency to ensure modules are built
- Pass the list of modules to the script

 Makefile             |  1 +
 arch/arm64/Makefile  |  1 +
 scripts/Makefile.lib | 10 ++++++++--
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 8cd46222fc48..4eccaef95826 100644
--- a/Makefile
+++ b/Makefile
@@ -773,6 +773,7 @@ endif
 # Just "make" or "make all" shall build modules as well
 
 modules-targets := all
+modules-targets += $(if $(FIT_MODULES),image.fit)
 modules-targets += modules
 modules-targets += nsdeps
 modules-targets += compile_commands.json
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 73a10f65ce8b..7036f251ab40 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -174,6 +174,7 @@ endif
 all:	$(notdir $(KBUILD_IMAGE))
 
 image.fit: dtbs
+image.fit: $(if $(FIT_MODULES),modules)
 
 vmlinuz.efi image.fit: Image
 $(BOOT_TARGETS): vmlinux
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 1d581ba5df66..c6a3aa653035 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -398,11 +398,17 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
 # Use this to override the compression algorithm
 FIT_COMPRESSION ?= gzip
 
+# Set this to 1 to include an initrd with all the kernel modules
+FIT_MODULES ?=
+
 quiet_cmd_fit = FIT     $@
-      cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
-		--name '$(UIMAGE_NAME)' \
+      cmd_fit = $(if $(FIT_MODULES), \
+		find $(objtree) -name '*.ko' > $(objtree)/.modules-list 2>/dev/null &&) \
+		$(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
+		--name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
 		$(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
 		$(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
+		$(if $(FIT_MODULES),--modules @$(objtree)/.modules-list) \
 		--compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
 
 # XZ
-- 
2.43.0

[PATCH v5 7/8] scripts/make_fit: Support a few more parallel compressors

[Simon Glass]

Add support for pbzip2 and plzip which can compress in parallel. This
speeds up the ramdisk compression.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 scripts/make_fit.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 3db129f40b20..10a040f4eb83 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -57,10 +57,10 @@ import libfdt
 CompTool = collections.namedtuple('CompTool', 'ext,tools')
 
 COMP_TOOLS = {
-    'bzip2': CompTool('.bz2', 'bzip2'),
+    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
     'gzip': CompTool('.gz', 'pigz,gzip'),
     'lz4': CompTool('.lz4', 'lz4'),
-    'lzma': CompTool('.lzma', 'lzma'),
+    'lzma': CompTool('.lzma', 'plzip,lzma'),
     'lzo': CompTool('.lzo', 'lzop'),
     'zstd': CompTool('.zstd', 'zstd'),
 }
@@ -220,7 +220,12 @@ def compress_data(inf, compress):
             done = False
             for tool in comp.tools.split(','):
                 try:
-                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
+                    # Add parallel flags for tools that support them
+                    cmd = [tool]
+                    if tool in ('zstd', 'xz'):
+                        cmd.extend(['-T0'])  # Use all available cores
+                    cmd.append('-c')
+                    subprocess.call(cmd, stdin=inf, stdout=outf)
                     done = True
                     break
                 except FileNotFoundError:
-- 
2.43.0

[PATCH v5 8/8] scripts/make_fit: Compress dtbs in parallel

[Simon Glass]

When there are 1500 device tree files it takes quite a while to compress
them. Do it in parallel.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 scripts/make_fit.py | 56 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 50 insertions(+), 6 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 10a040f4eb83..d1af496f3a57 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -43,6 +43,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
 
 import argparse
 import collections
+import multiprocessing
 import os
 import shutil
 import subprocess
@@ -237,15 +238,31 @@ def compress_data(inf, compress):
     return comp_data
 
 
-def output_dtb(fsw, seq, fname, arch, compress):
+def compress_dtb(fname, compress):
+    """Compress a single DTB file
+
+    Args:
+        fname (str): Filename containing the DTB
+        compress (str): Compression algorithm, e.g. 'gzip'
+
+    Returns:
+        tuple: (str: fname, bytes: compressed_data)
+    """
+    with open(fname, 'rb') as inf:
+        compressed = compress_data(inf, compress)
+    return fname, compressed
+
+
+def output_dtb(fsw, seq, fname, arch, compress, data=None):
     """Write out a single devicetree to the FIT
 
     Args:
         fsw (libfdt.FdtSw): Object to use for writing
         seq (int): Sequence number (1 for first)
         fname (str): Filename containing the DTB
-        arch: FIT architecture, e.g. 'arm64'
+        arch (str): FIT architecture, e.g. 'arm64'
         compress (str): Compressed algorithm, e.g. 'gzip'
+        data (bytes): Pre-compressed data (optional)
     """
     with fsw.add_node(f'fdt-{seq}'):
         fsw.property_string('description', os.path.basename(fname))
@@ -253,9 +270,10 @@ def output_dtb(fsw, seq, fname, arch, compress):
         fsw.property_string('arch', arch)
         fsw.property_string('compression', compress)
 
-        with open(fname, 'rb') as inf:
-            compressed = compress_data(inf, compress)
-        fsw.property('data', compressed)
+        if data is None:
+            with open(fname, 'rb') as inf:
+                data = compress_data(inf, compress)
+        fsw.property('data', data)
 
 
 def build_ramdisk(args, tmpdir):
@@ -366,6 +384,11 @@ def _process_dtbs(args, fsw, entries, fdts):
     """
     seq = 0
     size = 0
+
+    # First figure out the unique DTB files that need compression
+    todo = []
+    file_info = []  # List of (fname, model, compat, files) tuples
+
     for fname in args.dtbs:
         # Ignore non-DTB (*.dtb) files
         if os.path.splitext(fname)[1] != '.dtb':
@@ -377,11 +400,32 @@ def _process_dtbs(args, fsw, entries, fdts):
             sys.stderr.write(f'Error processing {fname}:\n')
             raise e
 
+        file_info.append((fname, model, compat, files))
+        for fn in files:
+            if fn not in fdts and fn not in todo:
+                todo.append(fn)
+
+    # Compress all DTBs in parallel
+    cache = {}
+    if todo and args.compress != 'none':
+        if args.verbose:
+            print(f'Compressing {len(todo)} DTBs...')
+
+        with multiprocessing.Pool() as pool:
+            compress_args = [(fn, args.compress) for fn in todo]
+            # unpacks each tuple, calls compress_dtb(fn, compress) in parallel
+            results = pool.starmap(compress_dtb, compress_args)
+
+        cache = dict(results)
+
+    # Now write all DTBs to the FIT using pre-compressed data
+    for fname, model, compat, files in file_info:
         for fn in files:
             if fn not in fdts:
                 seq += 1
                 size += os.path.getsize(fn)
-                output_dtb(fsw, seq, fn, args.arch, args.compress)
+                output_dtb(fsw, seq, fn, args.arch, args.compress,
+                           cache.get(fn))
                 fdts[fn] = seq
 
         files_seq = [fdts[fn] for fn in files]
-- 
2.43.0

Re: [PATCH v5 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Thomas Weißschuh]

On Fri, Nov 14, 2025 at 07:27:32AM -0700, Simon Glass wrote:
> Support 'make image.fit FIT_MODULES=1' to put all the modules into a
> ramdisk image within the FIT.
> 
> Add image.fit as a target which requires modules, so that modules will
> built automatically when using FIT_MODULES=1
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
> Acked-by: Nathan Chancellor <nathan@kernel.org>
> ---
> 
> Changes in v5:
> - Build modules automatically if needed (fix from Nicolas Schier)
> 
> Changes in v4:
> - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
> - Use an empty FIT_MODULES to disable the feature, instead of '0'
> - Make use of the 'modules' dependency to ensure modules are built
> - Pass the list of modules to the script
> 
>  Makefile             |  1 +
>  arch/arm64/Makefile  |  1 +
>  scripts/Makefile.lib | 10 ++++++++--
>  3 files changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 8cd46222fc48..4eccaef95826 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -773,6 +773,7 @@ endif
>  # Just "make" or "make all" shall build modules as well
>  
>  modules-targets := all
> +modules-targets += $(if $(FIT_MODULES),image.fit)
>  modules-targets += modules
>  modules-targets += nsdeps
>  modules-targets += compile_commands.json
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 73a10f65ce8b..7036f251ab40 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -174,6 +174,7 @@ endif
>  all:	$(notdir $(KBUILD_IMAGE))
>  
>  image.fit: dtbs
> +image.fit: $(if $(FIT_MODULES),modules)
>  
>  vmlinuz.efi image.fit: Image
>  $(BOOT_TARGETS): vmlinux
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 1d581ba5df66..c6a3aa653035 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -398,11 +398,17 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
>  # Use this to override the compression algorithm
>  FIT_COMPRESSION ?= gzip
>  
> +# Set this to 1 to include an initrd with all the kernel modules
> +FIT_MODULES ?=

'0' will also trigger that behavior.

> +
>  quiet_cmd_fit = FIT     $@
> -      cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> -		--name '$(UIMAGE_NAME)' \
> +      cmd_fit = $(if $(FIT_MODULES), \
> +		find $(objtree) -name '*.ko' > $(objtree)/.modules-list 2>/dev/null &&) \

This will include stale module files. You can get an up-to-date list from
$(objtree)/modules.order with a bit post-processing.
Maybe kbuild can be extended to also create a list of the .ko files.
(I would be interested in that for my own usecases, too)

> +		$(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> +		--name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
>  		$(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
>  		$(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> +		$(if $(FIT_MODULES),--modules @$(objtree)/.modules-list) \
>  		--compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
>  
>  # XZ
> -- 
> 2.43.0
> 

Re: [PATCH v5 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Nicolas Schier]

On Fri, Nov 14, 2025 at 04:29:33PM +0100, Thomas Weißschuh wrote:
> On Fri, Nov 14, 2025 at 07:27:32AM -0700, Simon Glass wrote:
> > Support 'make image.fit FIT_MODULES=1' to put all the modules into a
> > ramdisk image within the FIT.
> > 
> > Add image.fit as a target which requires modules, so that modules will
> > built automatically when using FIT_MODULES=1
> > 
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
> > Acked-by: Nathan Chancellor <nathan@kernel.org>
> > ---
> > 
> > Changes in v5:
> > - Build modules automatically if needed (fix from Nicolas Schier)
> > 
> > Changes in v4:
> > - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
> > - Use an empty FIT_MODULES to disable the feature, instead of '0'
> > - Make use of the 'modules' dependency to ensure modules are built
> > - Pass the list of modules to the script
> > 
> >  Makefile             |  1 +
> >  arch/arm64/Makefile  |  1 +
> >  scripts/Makefile.lib | 10 ++++++++--
> >  3 files changed, 10 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index 8cd46222fc48..4eccaef95826 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -773,6 +773,7 @@ endif
> >  # Just "make" or "make all" shall build modules as well
> >  
> >  modules-targets := all
> > +modules-targets += $(if $(FIT_MODULES),image.fit)
> >  modules-targets += modules
> >  modules-targets += nsdeps
> >  modules-targets += compile_commands.json
> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> > index 73a10f65ce8b..7036f251ab40 100644
> > --- a/arch/arm64/Makefile
> > +++ b/arch/arm64/Makefile
> > @@ -174,6 +174,7 @@ endif
> >  all:	$(notdir $(KBUILD_IMAGE))
> >  
> >  image.fit: dtbs
> > +image.fit: $(if $(FIT_MODULES),modules)
> >  
> >  vmlinuz.efi image.fit: Image
> >  $(BOOT_TARGETS): vmlinux
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index 1d581ba5df66..c6a3aa653035 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -398,11 +398,17 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
> >  # Use this to override the compression algorithm
> >  FIT_COMPRESSION ?= gzip
> >  
> > +# Set this to 1 to include an initrd with all the kernel modules
> > +FIT_MODULES ?=
> 
> '0' will also trigger that behavior.
> 
> > +
> >  quiet_cmd_fit = FIT     $@
> > -      cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> > -		--name '$(UIMAGE_NAME)' \
> > +      cmd_fit = $(if $(FIT_MODULES), \
> > +		find $(objtree) -name '*.ko' > $(objtree)/.modules-list 2>/dev/null &&) \
> 
> This will include stale module files. You can get an up-to-date list from
> $(objtree)/modules.order with a bit post-processing.
> Maybe kbuild can be extended to also create a list of the .ko files.
> (I would be interested in that for my own usecases, too)

oh yes, thanks for the pointer.  This is indeed quite simple and much
better than calling find.  For in-tree kmods:

    compiled-modules = $(patsubst %.o,%.ko,$(call read-file, $(objtree)/modules.order))

But as we need the list of modules in a file, we can also add a
$(call write-file,FILE,TEXT) macro (cp. read-file in
scripts/Kbuild.include).

Thomas, is this sufficient for your use case?  Or do you also need a
make target outputting the list of kmods?


Kind regards,
Nicolas

Re: [PATCH v5 4/8] scripts/make_fit: Provide a way to add built modules

[Ahmad Fatoum]

Hello Simon,

On 14.11.25 15:27, Simon Glass wrote:
> Provide arguments to support building a ramdisk from a directory tree of
> modules. This is a convenient way to try out a kernel with its modules.
> 
> This makes use of the cpio tool rather than attempting to use a python
> module or our own code. The list of modules is provided in a file.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>
> Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

Thanks for taking up the suggestion! I just Cc'd you on a patch that
generates a CPIO with the modules directly as a make target.

I'd prefer going down that route instead as to reuse as much existing code
as possible.

Maybe this even covers Thomas' use case from the other thread?

Thanks,
Ahmad

> ---
> 
> (no changes since v4)
> 
> Changes in v4:
> - Provide the list of modules from the Makefile
> - Reduce verbosity (don't print every module filename)
> 
> Changes in v3:
> - Add a way to add built modules into the FIT
> 
>  scripts/make_fit.py | 98 +++++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 90 insertions(+), 8 deletions(-)
> 
> diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> index 1a74a9dcd85e..3db129f40b20 100755
> --- a/scripts/make_fit.py
> +++ b/scripts/make_fit.py
> @@ -13,11 +13,17 @@ Usage:
>          -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
>          -E -c gzip
>  
> +    # Build with modules ramdisk instead of external ramdisk:
> +    make_fit.py -A arm64 -n 'Linux-6.17' -O linux
> +        -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
> +        -m module1.ko module2.ko module3.ko @arch/arm64/boot/dts/dtbs-list
> +
>  Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
>  devicetree files, either specified individually or listed in a file (with an
>  '@' prefix).
>  
>  Use -r to specify an existing ramdisk/initrd file.
> +Use -m to build a ramdisk from specified kernel module files.
>  
>  Use -E to generate an external FIT (where the data is placed after the
>  FIT data structure). This allows parsing of the data without loading
> @@ -38,6 +44,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
>  import argparse
>  import collections
>  import os
> +import shutil
>  import subprocess
>  import sys
>  import tempfile
> @@ -83,8 +90,14 @@ def parse_args():
>            help='Specifies the operating system')
>      parser.add_argument('-k', '--kernel', type=str, required=True,
>            help='Specifies the (uncompressed) kernel input file (.itk)')
> -    parser.add_argument('-r', '--ramdisk', type=str,
> +
> +    # Create mutually exclusive group for ramdisk options
> +    rd_group = parser.add_mutually_exclusive_group()
> +    rd_group.add_argument('-r', '--ramdisk', type=str,
>            help='Specifies the ramdisk/initrd input file')
> +    rd_group.add_argument('-m', '--modules', type=str, nargs='+',
> +          help='List of module filenames to include in ramdisk')
> +
>      parser.add_argument('-v', '--verbose', action='store_true',
>                          help='Enable verbose output')
>      parser.add_argument('dtbs', type=str, nargs='*',
> @@ -240,6 +253,60 @@ def output_dtb(fsw, seq, fname, arch, compress):
>          fsw.property('data', compressed)
>  
>  
> +def build_ramdisk(args, tmpdir):
> +    """Build a cpio ramdisk containing kernel modules
> +
> +    Similar to mkinitramfs, this creates a compressed cpio-archive containing
> +    the kernel modules for the current kernel version.
> +
> +    Args:
> +        args (Namespace): Program arguments
> +        tmpdir (str): Temporary directory to use for modules installation
> +
> +    Returns:
> +        tuple:
> +            bytes: Compressed cpio data containing modules
> +            int: total uncompressed size
> +    """
> +    suppress = None if args.verbose else subprocess.DEVNULL
> +
> +    if args.verbose:
> +        print(f'Copying {len(args.modules)} modules to ramdisk')
> +
> +    # Create output-directory structure
> +    outdir = os.path.join(tmpdir, 'initramfs')
> +    modules_dir = os.path.join(outdir, 'usr', 'lib', 'modules')
> +    os.makedirs(modules_dir, exist_ok=True)
> +
> +    # Copy in the specified modules
> +    for module in args.modules:
> +        dest_path = os.path.join(modules_dir, os.path.basename(module))
> +        shutil.copy2(module, dest_path)
> +
> +    if args.verbose:
> +        print(f'Creating cpio archive from {outdir}')
> +
> +    with tempfile.NamedTemporaryFile() as cpio_file:
> +        # Change to initramfs directory and create cpio archive
> +        with subprocess.Popen(['find', '.', '-print0'], cwd=outdir,
> +                              stdout=subprocess.PIPE) as find:
> +            with subprocess.Popen(['cpio', '-o', '-0', '-H', 'newc'],
> +                                  stdin=find.stdout, stdout=cpio_file,
> +                                  stderr=suppress, cwd=outdir) as cpio:
> +                find.stdout.close()
> +                cpio.wait()
> +                find.wait()
> +
> +                if cpio.returncode != 0:
> +                    raise RuntimeError('Failed to create cpio archive')
> +
> +        cpio_file.seek(0)  # Reset to beginning for reading
> +        if args.verbose:
> +            print('Reading ramdisk...' if args.compress == 'none' else
> +                  f'Compressing ramdisk with {args.compress}...')
> +        return compress_data(cpio_file, args.compress), cpio_file.tell()
> +
> +
>  def process_dtb(fname, args):
>      """Process an input DTB, decomposing it if requested and is possible
>  
> @@ -318,11 +385,12 @@ def _process_dtbs(args, fsw, entries, fdts):
>      return seq, size
>  
>  
> -def build_fit(args):
> +def build_fit(args, tmpdir):
>      """Build the FIT from the provided files and arguments
>  
>      Args:
>          args (Namespace): Program arguments
> +        tmpdir (str): Temporary directory for any temporary files
>  
>      Returns:
>          tuple:
> @@ -344,20 +412,29 @@ def build_fit(args):
>  
>      # Handle the ramdisk if provided. Compression is not supported as it is
>      # already compressed.
> +    ramdisk_data = None
>      if args.ramdisk:
>          with open(args.ramdisk, 'rb') as inf:
> -            data = inf.read()
> -        size += len(data)
> -        write_ramdisk(fsw, data, args)
> +            ramdisk_data = inf.read()
> +        size += len(ramdisk_data)
> +    elif args.modules:
> +        if args.verbose:
> +            print('Building modules ramdisk...')
> +        ramdisk_data, uncomp_size = build_ramdisk(args, tmpdir)
> +        size += uncomp_size
> +
> +    if ramdisk_data:
> +        write_ramdisk(fsw, ramdisk_data, args)
>  
>      count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
>      size += fdt_size
>  
> -    finish_fit(fsw, entries, bool(args.ramdisk))
> +    finish_fit(fsw, entries, has_ramdisk=bool(ramdisk_data))
>  
> -    # Include the kernel itself in the returned file count
>      fdt = fsw.as_fdt()
>      fdt.pack()
> +
> +    # Count FDT files, kernel, plus ramdisk if present
>      return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
>  
>  
> @@ -365,7 +442,12 @@ def run_make_fit():
>      """Run the tool's main logic"""
>      args = parse_args()
>  
> -    out_data, count, size = build_fit(args)
> +    tmpdir = tempfile.mkdtemp(prefix='make_fit_')
> +    try:
> +        out_data, count, size = build_fit(args, tmpdir)
> +    finally:
> +        shutil.rmtree(tmpdir)
> +
>      with open(args.output, 'wb') as outf:
>          outf.write(out_data)
>  


-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [PATCH v5 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Thomas Weißschuh]

On Fri, Nov 14, 2025 at 08:50:27PM +0100, Nicolas Schier wrote:
> On Fri, Nov 14, 2025 at 04:29:33PM +0100, Thomas Weißschuh wrote:
> > On Fri, Nov 14, 2025 at 07:27:32AM -0700, Simon Glass wrote:

(...)

> > >  quiet_cmd_fit = FIT     $@
> > > -      cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> > > -		--name '$(UIMAGE_NAME)' \
> > > +      cmd_fit = $(if $(FIT_MODULES), \
> > > +		find $(objtree) -name '*.ko' > $(objtree)/.modules-list 2>/dev/null &&) \
> > 
> > This will include stale module files. You can get an up-to-date list from
> > $(objtree)/modules.order with a bit post-processing.
> > Maybe kbuild can be extended to also create a list of the .ko files.
> > (I would be interested in that for my own usecases, too)
> 
> oh yes, thanks for the pointer.  This is indeed quite simple and much
> better than calling find.  For in-tree kmods:
> 
>     compiled-modules = $(patsubst %.o,%.ko,$(call read-file, $(objtree)/modules.order))

Tiny nitpick: IMO 'built' modules would be more accurate than 'compiled' modules.

> But as we need the list of modules in a file, we can also add a
> $(call write-file,FILE,TEXT) macro (cp. read-file in
> scripts/Kbuild.include).

> Thomas, is this sufficient for your use case?  Or do you also need a
> make target outputting the list of kmods?

For my usecase I need it in a file [0], passing them on the command line will
likely run into argument list length limitations.

A new file 'modules.built' generated from modules.order would be useful both
for this patch, mine and other tools like scripts/package/builddeb.
(Or 'modules.loadable', matching the scheme from 'modules.builtin')
On the other hand the modules.order seems to have quite some special logic in
the Makefile, so it may not be straigtforward to add such a file.

[0] https://lore.kernel.org/lkml/20250429-module-hashes-v3-9-00e9258def9e@weissschuh.net/#Z31scripts:module-hashes.sh


Thomas

Re: [PATCH v5 4/8] scripts/make_fit: Provide a way to add built modules

[Simon Glass]

Hi Ahmad,

On Sat, 15 Nov 2025 at 07:39, Ahmad Fatoum <a.fatoum@pengutronix.de> wrote:
>
> Hello Simon,
>
> On 14.11.25 15:27, Simon Glass wrote:
> > Provide arguments to support building a ramdisk from a directory tree of
> > modules. This is a convenient way to try out a kernel with its modules.
> >
> > This makes use of the cpio tool rather than attempting to use a python
> > module or our own code. The list of modules is provided in a file.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > Reviewed-by: Nicolas Schier <nsc@kernel.org>
> > Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
>
> Thanks for taking up the suggestion! I just Cc'd you on a patch that
> generates a CPIO with the modules directly as a make target.
>
> I'd prefer going down that route instead as to reuse as much existing code
> as possible.
>
> Maybe this even covers Thomas' use case from the other thread?

From my testing, FIT_MODULES=1 this seems to require that
modules_install be done first. But my existing approach builds a FIT
without needing to do the install step.

Just adding the dependency, like this:

image.fit: $(if $(FIT_MODULES),cpio-modules-pkg)

results in:

time ARCH=arm64
CROSS_COMPILE=/home/sglass/.buildman-toolchains/gcc-13.1.0-nolibc/aarch64-linux/bin/aarch64-linux-
make O=/tmp/kern -j30 image.fit FIT_MODULES=1 -s
fixdep: error opening file: init/.main.o.d: No such file or directory
make[4]: *** [/scratch/sglass/linux/scripts/Makefile.build:287:
init/main.o] Error 2
make[4]: *** Deleting file 'init/main.o'
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [/scratch/sglass/linux/scripts/Makefile.build:556: init] Error 2
make[3]: *** Waiting for unfinished jobs....
/home/sglass/.buildman-toolchains/gcc-13.1.0-nolibc/aarch64-linux/bin/aarch64-linux-ar:
init/main.o: No such file or directory
make[6]: *** [/scratch/sglass/linux/scripts/Makefile.build:473:
init/built-in.a] Error 123
make[5]: *** [/scratch/sglass/linux/scripts/Makefile.build:556: init] Error 2
make[4]: *** [/scratch/sglass/linux/Makefile:2017: .] Error 2
make[3]: *** [/scratch/sglass/linux/scripts/Makefile.package:193:
modules-install] Error 2
make[2]: *** [/scratch/sglass/linux/Makefile:1647: cpio-modules-pkg] Error 2
make[2]: *** Waiting for unfinished jobs....

Makefile has the ability to sequence things, but in this case we are
effectively adding a target within the Makefile.

What do you suggest?

[..]

Regards,
Simon