[PATCH v6 0/8] scripts/make_fit: Support ramdisks and faster operations

[PATCH v6 0/8] scripts/make_fit: Support ramdisks and faster operations

[Simon Glass]

This series updates 'make image.fit' to support adding a ramdisk to the
FIT, either one provided as a parameter or one created from all the
kernel modules.

It also includes a few performance improvement, so that building a FIT
from ~450MB of kernel/module/devicetree files only takes a few seconds
on a modern machine.

Note: I included support for using the existing cpio script as suggested
by Ahmad, but a fallback remains in case that is not available.

https://lore.kernel.org/lkml/CAFLszThpTg-FBoNG_tQ0xve0LkYWER85EJeHuem-_JUD8J1Ocw@mail.gmail.com/

Changes in v6:
- Using the modules.order file instead of 'find'
- Use gen_initramfs.sh where available
- Mention that FIT_MODULES just needs to be non-empty
- Make use of modules.order instead of using 'find'

Changes in v5:
- Fix 'use' typo
- Add a new patch to split out module targets into a variable
- Build modules automatically if needed (fix from Nicolas Schier)

Changes in v4:
- Update the commit message
- Provide the list of modules from the Makefile
- Reduce verbosity (don't print every module filename)
- Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
- Use an empty FIT_MODULES to disable the feature, instead of '0'
- Make use of the 'modules' dependency to ensure modules are built
- Pass the list of modules to the script

Changes in v3:
- Move the ramdisk chunk into the correct patch
- Add a comment at the top of the file about the -r option
- Count the ramdisk in the total files
- Update the commit message
- Add a way to add built modules into the FIT

Changes in v2:
- Don't compress the ramdisk as it is already compressed

Simon Glass (8):
  scripts/make_fit: Speed up operation
  scripts/make_fit: Support an initial ramdisk
  scripts/make_fit: Move dtb processing into a function
  scripts/make_fit: Provide a way to add built modules
  kbuild: Split out module targets into a variable
  kbuild: Allow adding modules into the FIT ramdisk
  scripts/make_fit: Support a few more parallel compressors
  scripts/make_fit: Compress dtbs in parallel

 Makefile             |   8 +-
 arch/arm64/Makefile  |   1 +
 scripts/Makefile.lib |   6 +-
 scripts/make_fit.py  | 293 +++++++++++++++++++++++++++++++++++++------
 4 files changed, 269 insertions(+), 39 deletions(-)

-- 
2.43.0

base-commit: 4a71531471926e3c391665ee9c42f4e0295a4585
branch: fita6

[PATCH v6 1/8] scripts/make_fit: Speed up operation

[Simon Glass]

The kernel is likely at least 16MB so we may as well use that as a step
size when reallocating space for the FIT in memory. Pack the FIT at the
end, so there is no wasted space.

This reduces the time to pack by an order of magnitude, or so.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
---

(no changes since v3)

Changes in v3:
- Move the ramdisk chunk into the correct patch

 scripts/make_fit.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 1683e5ec6e67..0f5e7c4b8aed 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -98,7 +98,7 @@ def setup_fit(fsw, name):
         fsw (libfdt.FdtSw): Object to use for writing
         name (str): Name of kernel image
     """
-    fsw.INC_SIZE = 65536
+    fsw.INC_SIZE = 16 << 20
     fsw.finish_reservemap()
     fsw.begin_node('')
     fsw.property_string('description', f'{name} with devicetree set')
@@ -299,7 +299,9 @@ def build_fit(args):
     finish_fit(fsw, entries)
 
     # Include the kernel itself in the returned file count
-    return fsw.as_fdt().as_bytearray(), seq + 1, size
+    fdt = fsw.as_fdt()
+    fdt.pack()
+    return fdt.as_bytearray(), seq + 1, size
 
 
 def run_make_fit():
-- 
2.43.0

Re: [PATCH v6 1/8] scripts/make_fit: Speed up operation

[Ahmad Fatoum]

On 11/19/25 7:13 PM, Simon Glass wrote:
> The kernel is likely at least 16MB so we may as well use that as a step
> size when reallocating space for the FIT in memory. Pack the FIT at the
> end, so there is no wasted space.
> 
> This reduces the time to pack by an order of magnitude, or so.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>

Reviewed-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

> ---
> 
> (no changes since v3)
> 
> Changes in v3:
> - Move the ramdisk chunk into the correct patch
> 
>  scripts/make_fit.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> index 1683e5ec6e67..0f5e7c4b8aed 100755
> --- a/scripts/make_fit.py
> +++ b/scripts/make_fit.py
> @@ -98,7 +98,7 @@ def setup_fit(fsw, name):
>          fsw (libfdt.FdtSw): Object to use for writing
>          name (str): Name of kernel image
>      """
> -    fsw.INC_SIZE = 65536
> +    fsw.INC_SIZE = 16 << 20
>      fsw.finish_reservemap()
>      fsw.begin_node('')
>      fsw.property_string('description', f'{name} with devicetree set')
> @@ -299,7 +299,9 @@ def build_fit(args):
>      finish_fit(fsw, entries)
>  
>      # Include the kernel itself in the returned file count
> -    return fsw.as_fdt().as_bytearray(), seq + 1, size
> +    fdt = fsw.as_fdt()
> +    fdt.pack()
> +    return fdt.as_bytearray(), seq + 1, size
>  
>  
>  def run_make_fit():

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

Re: [PATCH v6 1/8] scripts/make_fit: Speed up operation

[Chen-Yu Tsai]

On Thu, Nov 20, 2025 at 2:14 AM Simon Glass <sjg@chromium.org> wrote:
>
> The kernel is likely at least 16MB so we may as well use that as a step
> size when reallocating space for the FIT in memory. Pack the FIT at the
> end, so there is no wasted space.
>
> This reduces the time to pack by an order of magnitude, or so.

In my case the time spent compressing the blobs seems to be much
larger than the packing portion, so I couldn't really tell if
this made a difference or not.

Despite that,

> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>

Tested-by: Chen-Yu Tsai <wenst@chromium.org>

[PATCH v6 2/8] scripts/make_fit: Support an initial ramdisk

[Simon Glass]

FIT (Flat Image Tree) allows a ramdisk to be included in each
configuration. Add support for this to the script.

This feature is not available via 'make image.fit' since the ramdisk
likely needs to be built separately anyway, e.g. using modules from
the kernel build. A later patch in this series provides support for
doing that.

Note that the uncompressed size is not correct when a ramdisk is used,
since it is too expensive to decompress the ramdisk.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
---

(no changes since v5)

Changes in v5:
- Fix 'use' typo

Changes in v4:
- Update the commit message

Changes in v3:
- Add a comment at the top of the file about the -r option
- Count the ramdisk in the total files
- Update the commit message

Changes in v2:
- Don't compress the ramdisk as it is already compressed

 scripts/make_fit.py | 52 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 44 insertions(+), 8 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 0f5e7c4b8aed..984371f505bc 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -10,10 +10,14 @@
 Usage:
     make_fit.py -A arm64 -n 'Linux-6.6' -O linux
         -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
-        @arch/arm64/boot/dts/dtbs-list -E -c gzip
+        -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
+        -E -c gzip
 
-Creates a FIT containing the supplied kernel and a set of devicetree files,
-either specified individually or listed in a file (with an '@' prefix).
+Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
+devicetree files, either specified individually or listed in a file (with an
+'@' prefix).
+
+Use -r to specify an existing ramdisk/initrd file.
 
 Use -E to generate an external FIT (where the data is placed after the
 FIT data structure). This allows parsing of the data without loading
@@ -29,8 +33,6 @@ looks at the .cmd files produced by the kernel build.
 
 The resulting FIT can be booted by bootloaders which support FIT, such
 as U-Boot, Linuxboot, Tianocore, etc.
-
-Note that this tool does not yet support adding a ramdisk / initrd.
 """
 
 import argparse
@@ -81,6 +83,8 @@ def parse_args():
           help='Specifies the operating system')
     parser.add_argument('-k', '--kernel', type=str, required=True,
           help='Specifies the (uncompressed) kernel input file (.itk)')
+    parser.add_argument('-r', '--ramdisk', type=str,
+          help='Specifies the ramdisk/initrd input file')
     parser.add_argument('-v', '--verbose', action='store_true',
                         help='Enable verbose output')
     parser.add_argument('dtbs', type=str, nargs='*',
@@ -133,7 +137,28 @@ def write_kernel(fsw, data, args):
         fsw.property_u32('entry', 0)
 
 
-def finish_fit(fsw, entries):
+def write_ramdisk(fsw, data, args):
+    """Write out the ramdisk image
+
+    Writes a ramdisk node along with the required properties
+
+    Args:
+        fsw (libfdt.FdtSw): Object to use for writing
+        data (bytes): Data to write (possibly compressed)
+        args (Namespace): Contains necessary strings:
+            arch: FIT architecture, e.g. 'arm64'
+            fit_os: Operating Systems, e.g. 'linux'
+    """
+    with fsw.add_node('ramdisk'):
+        fsw.property_string('description', 'Ramdisk')
+        fsw.property_string('type', 'ramdisk')
+        fsw.property_string('arch', args.arch)
+        fsw.property_string('os', args.os)
+        fsw.property('data', data)
+        fsw.property_u32('load', 0)
+
+
+def finish_fit(fsw, entries, has_ramdisk=False):
     """Finish the FIT ready for use
 
     Writes the /configurations node and subnodes
@@ -143,6 +168,7 @@ def finish_fit(fsw, entries):
         entries (list of tuple): List of configurations:
             str: Description of model
             str: Compatible stringlist
+        has_ramdisk (bool): True if a ramdisk is included in the FIT
     """
     fsw.end_node()
     seq = 0
@@ -154,6 +180,8 @@ def finish_fit(fsw, entries):
                 fsw.property_string('description', model)
                 fsw.property('fdt', bytes(''.join(f'fdt-{x}\x00' for x in files), "ascii"))
                 fsw.property_string('kernel', 'kernel')
+                if has_ramdisk:
+                    fsw.property_string('ramdisk', 'ramdisk')
     fsw.end_node()
 
 
@@ -274,6 +302,14 @@ def build_fit(args):
     size += os.path.getsize(args.kernel)
     write_kernel(fsw, comp_data, args)
 
+    # Handle the ramdisk if provided. Compression is not supported as it is
+    # already compressed.
+    if args.ramdisk:
+        with open(args.ramdisk, 'rb') as inf:
+            data = inf.read()
+        size += len(data)
+        write_ramdisk(fsw, data, args)
+
     for fname in args.dtbs:
         # Ignore non-DTB (*.dtb) files
         if os.path.splitext(fname)[1] != '.dtb':
@@ -296,12 +332,12 @@ def build_fit(args):
 
         entries.append([model, compat, files_seq])
 
-    finish_fit(fsw, entries)
+    finish_fit(fsw, entries, bool(args.ramdisk))
 
     # Include the kernel itself in the returned file count
     fdt = fsw.as_fdt()
     fdt.pack()
-    return fdt.as_bytearray(), seq + 1, size
+    return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size
 
 
 def run_make_fit():
-- 
2.43.0

Re: [PATCH v6 2/8] scripts/make_fit: Support an initial ramdisk

[Ahmad Fatoum]

On 11/19/25 7:13 PM, Simon Glass wrote:
> FIT (Flat Image Tree) allows a ramdisk to be included in each
> configuration. Add support for this to the script.
> 
> This feature is not available via 'make image.fit' since the ramdisk
> likely needs to be built separately anyway, e.g. using modules from
> the kernel build. A later patch in this series provides support for
> doing that.
> 
> Note that the uncompressed size is not correct when a ramdisk is used,
> since it is too expensive to decompress the ramdisk.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>

With load address removed:

Reviewed-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

> +    with fsw.add_node('ramdisk'):> +
fsw.property_string('description', 'Ramdisk')
> +        fsw.property_string('type', 'ramdisk')
> +        fsw.property_string('arch', args.arch)
> +        fsw.property_string('os', args.os)
> +        fsw.property('data', data)

The spec reads as if a compression = "none" is mandatory for ramdisks:

If the data is compressed but it should not be uncompressed by the
loader (e.g. compressed ramdisk <pair: ramdisk; compressed), this should
also be set to “none”.

Thoughts?

> +        fsw.property_u32('load', 0)

0 is a valid SDRAM address on many SoCs. I think this should be dropped.

> +
> +
> +def finish_fit(fsw, entries, has_ramdisk=False):
>      """Finish the FIT ready for use
>  
>      Writes the /configurations node and subnodes
> @@ -143,6 +168,7 @@ def finish_fit(fsw, entries):
>          entries (list of tuple): List of configurations:
>              str: Description of model
>              str: Compatible stringlist
> +        has_ramdisk (bool): True if a ramdisk is included in the FIT
>      """
>      fsw.end_node()
>      seq = 0
> @@ -154,6 +180,8 @@ def finish_fit(fsw, entries):
>                  fsw.property_string('description', model)
>                  fsw.property('fdt', bytes(''.join(f'fdt-{x}\x00' for x in files), "ascii"))
>                  fsw.property_string('kernel', 'kernel')
> +                if has_ramdisk:
> +                    fsw.property_string('ramdisk', 'ramdisk')
>      fsw.end_node()
>  
>  
> @@ -274,6 +302,14 @@ def build_fit(args):
>      size += os.path.getsize(args.kernel)
>      write_kernel(fsw, comp_data, args)
>  
> +    # Handle the ramdisk if provided. Compression is not supported as it is
> +    # already compressed.
> +    if args.ramdisk:
> +        with open(args.ramdisk, 'rb') as inf:
> +            data = inf.read()
> +        size += len(data)
> +        write_ramdisk(fsw, data, args)
> +
>      for fname in args.dtbs:
>          # Ignore non-DTB (*.dtb) files
>          if os.path.splitext(fname)[1] != '.dtb':
> @@ -296,12 +332,12 @@ def build_fit(args):
>  
>          entries.append([model, compat, files_seq])
>  
> -    finish_fit(fsw, entries)
> +    finish_fit(fsw, entries, bool(args.ramdisk))
>  
>      # Include the kernel itself in the returned file count
>      fdt = fsw.as_fdt()
>      fdt.pack()
> -    return fdt.as_bytearray(), seq + 1, size
> +    return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size
>  
>  
>  def run_make_fit():

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

[PATCH v6 3/8] scripts/make_fit: Move dtb processing into a function

[Simon Glass]

Since build_fit() is getting quite long, move the dtb processing into a
separate function.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Nicolas Schier <nsc@kernel.org>
---

(no changes since v1)

 scripts/make_fit.py | 67 +++++++++++++++++++++++++++++----------------
 1 file changed, 44 insertions(+), 23 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 984371f505bc..1a74a9dcd85e 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -277,6 +277,47 @@ def process_dtb(fname, args):
 
     return (model, compat, files)
 
+
+def _process_dtbs(args, fsw, entries, fdts):
+    """Process all DTB files and add them to the FIT
+
+    Args:
+        args: Program arguments
+        fsw: FIT writer object
+        entries: List to append entries to
+        fdts: Dictionary of processed DTBs
+
+    Returns:
+        tuple:
+            Number of files processed
+            Total size of files processed
+    """
+    seq = 0
+    size = 0
+    for fname in args.dtbs:
+        # Ignore non-DTB (*.dtb) files
+        if os.path.splitext(fname)[1] != '.dtb':
+            continue
+
+        try:
+            (model, compat, files) = process_dtb(fname, args)
+        except Exception as e:
+            sys.stderr.write(f'Error processing {fname}:\n')
+            raise e
+
+        for fn in files:
+            if fn not in fdts:
+                seq += 1
+                size += os.path.getsize(fn)
+                output_dtb(fsw, seq, fn, args.arch, args.compress)
+                fdts[fn] = seq
+
+        files_seq = [fdts[fn] for fn in files]
+        entries.append([model, compat, files_seq])
+
+    return seq, size
+
+
 def build_fit(args):
     """Build the FIT from the provided files and arguments
 
@@ -289,7 +330,6 @@ def build_fit(args):
             int: Number of configurations generated
             size: Total uncompressed size of data
     """
-    seq = 0
     size = 0
     fsw = libfdt.FdtSw()
     setup_fit(fsw, args.name)
@@ -310,34 +350,15 @@ def build_fit(args):
         size += len(data)
         write_ramdisk(fsw, data, args)
 
-    for fname in args.dtbs:
-        # Ignore non-DTB (*.dtb) files
-        if os.path.splitext(fname)[1] != '.dtb':
-            continue
-
-        try:
-            (model, compat, files) = process_dtb(fname, args)
-        except Exception as e:
-            sys.stderr.write(f"Error processing {fname}:\n")
-            raise e
-
-        for fn in files:
-            if fn not in fdts:
-                seq += 1
-                size += os.path.getsize(fn)
-                output_dtb(fsw, seq, fn, args.arch, args.compress)
-                fdts[fn] = seq
-
-        files_seq = [fdts[fn] for fn in files]
-
-        entries.append([model, compat, files_seq])
+    count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
+    size += fdt_size
 
     finish_fit(fsw, entries, bool(args.ramdisk))
 
     # Include the kernel itself in the returned file count
     fdt = fsw.as_fdt()
     fdt.pack()
-    return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size
+    return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
 
 
 def run_make_fit():
-- 
2.43.0

Re: [PATCH v6 3/8] scripts/make_fit: Move dtb processing into a function

[Ahmad Fatoum]

On 11/19/25 7:13 PM, Simon Glass wrote:
> Since build_fit() is getting quite long, move the dtb processing into a
> separate function.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>

Reviewed-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

Re: [PATCH v6 3/8] scripts/make_fit: Move dtb processing into a function

[Chen-Yu Tsai]

On Thu, Nov 20, 2025 at 2:14 AM Simon Glass <sjg@chromium.org> wrote:
>
> Since build_fit() is getting quite long, move the dtb processing into a
> separate function.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Reviewed-by: Nicolas Schier <nsc@kernel.org>
> ---
>
> (no changes since v1)
>
>  scripts/make_fit.py | 67 +++++++++++++++++++++++++++++----------------
>  1 file changed, 44 insertions(+), 23 deletions(-)
>
> diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> index 984371f505bc..1a74a9dcd85e 100755
> --- a/scripts/make_fit.py
> +++ b/scripts/make_fit.py
> @@ -277,6 +277,47 @@ def process_dtb(fname, args):
>
>      return (model, compat, files)
>
> +
> +def _process_dtbs(args, fsw, entries, fdts):
> +    """Process all DTB files and add them to the FIT
> +
> +    Args:
> +        args: Program arguments
> +        fsw: FIT writer object
> +        entries: List to append entries to
> +        fdts: Dictionary of processed DTBs
> +
> +    Returns:
> +        tuple:
> +            Number of files processed
> +            Total size of files processed
> +    """
> +    seq = 0
> +    size = 0
> +    for fname in args.dtbs:
> +        # Ignore non-DTB (*.dtb) files
> +        if os.path.splitext(fname)[1] != '.dtb':
> +            continue
> +
> +        try:
> +            (model, compat, files) = process_dtb(fname, args)
> +        except Exception as e:
> +            sys.stderr.write(f'Error processing {fname}:\n')

`git diff --color-moved` revealed that this bit was changed from
f"" to f'', which I think is a bit unexpected for pure code movement.

> +            raise e
> +
> +        for fn in files:
> +            if fn not in fdts:
> +                seq += 1
> +                size += os.path.getsize(fn)
> +                output_dtb(fsw, seq, fn, args.arch, args.compress)
> +                fdts[fn] = seq
> +
> +        files_seq = [fdts[fn] for fn in files]
> +        entries.append([model, compat, files_seq])
> +
> +    return seq, size
> +
> +
>  def build_fit(args):
>      """Build the FIT from the provided files and arguments
>
> @@ -289,7 +330,6 @@ def build_fit(args):
>              int: Number of configurations generated
>              size: Total uncompressed size of data
>      """
> -    seq = 0
>      size = 0
>      fsw = libfdt.FdtSw()
>      setup_fit(fsw, args.name)
> @@ -310,34 +350,15 @@ def build_fit(args):
>          size += len(data)
>          write_ramdisk(fsw, data, args)
>
> -    for fname in args.dtbs:
> -        # Ignore non-DTB (*.dtb) files
> -        if os.path.splitext(fname)[1] != '.dtb':
> -            continue
> -
> -        try:
> -            (model, compat, files) = process_dtb(fname, args)
> -        except Exception as e:
> -            sys.stderr.write(f"Error processing {fname}:\n")
> -            raise e
> -
> -        for fn in files:
> -            if fn not in fdts:
> -                seq += 1
> -                size += os.path.getsize(fn)
> -                output_dtb(fsw, seq, fn, args.arch, args.compress)
> -                fdts[fn] = seq
> -
> -        files_seq = [fdts[fn] for fn in files]
> -
> -        entries.append([model, compat, files_seq])
> +    count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
> +    size += fdt_size
>
>      finish_fit(fsw, entries, bool(args.ramdisk))
>
>      # Include the kernel itself in the returned file count
>      fdt = fsw.as_fdt()
>      fdt.pack()
> -    return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size
> +    return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size

Reviewed-by: Chen-Yu Tsai <wenst@chromium.org>

>
>  def run_make_fit():
> --
> 2.43.0
>

[PATCH v6 4/8] scripts/make_fit: Provide a way to add built modules

[Simon Glass]

Provide arguments to support building a ramdisk from a directory tree of
modules. This is a convenient way to try out a kernel with its modules.

This makes use usr/gen_initramfs.sh where it can be found. Failing that
it uses the cpio tool rather than attempting to use a python module or
our own code. The list of modules is provided in a file.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---
Note: The approach of depending on modules_install caused build problems
as mentioned in the cover letter.

I dropped the review tag from Nicolas Schier since there are quite a few
changes in this version.

Changes in v6:
- Using the modules.order file instead of 'find'
- Use gen_initramfs.sh where available

Changes in v4:
- Provide the list of modules from the Makefile
- Reduce verbosity (don't print every module filename)

Changes in v3:
- Add a way to add built modules into the FIT

 scripts/make_fit.py | 127 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 119 insertions(+), 8 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 1a74a9dcd85e..9dfef11fc4b3 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -13,11 +13,17 @@ Usage:
         -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
         -E -c gzip
 
+    # Build with modules ramdisk instead of external ramdisk:
+    make_fit.py -A arm64 -n 'Linux-6.17' -O linux
+        -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
+        -m module1.ko module2.ko module3.ko @arch/arm64/boot/dts/dtbs-list
+
 Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
 devicetree files, either specified individually or listed in a file (with an
 '@' prefix).
 
 Use -r to specify an existing ramdisk/initrd file.
+Use -m to build a ramdisk from specified kernel module files.
 
 Use -E to generate an external FIT (where the data is placed after the
 FIT data structure). This allows parsing of the data without loading
@@ -38,6 +44,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
 import argparse
 import collections
 import os
+import shutil
 import subprocess
 import sys
 import tempfile
@@ -83,8 +90,14 @@ def parse_args():
           help='Specifies the operating system')
     parser.add_argument('-k', '--kernel', type=str, required=True,
           help='Specifies the (uncompressed) kernel input file (.itk)')
-    parser.add_argument('-r', '--ramdisk', type=str,
+
+    # Create mutually exclusive group for ramdisk options
+    rd_group = parser.add_mutually_exclusive_group()
+    rd_group.add_argument('-r', '--ramdisk', type=str,
           help='Specifies the ramdisk/initrd input file')
+    rd_group.add_argument('-m', '--modules', type=str, nargs='+',
+          help='List of module filenames to include in ramdisk')
+
     parser.add_argument('-v', '--verbose', action='store_true',
                         help='Enable verbose output')
     parser.add_argument('dtbs', type=str, nargs='*',
@@ -240,6 +253,89 @@ def output_dtb(fsw, seq, fname, arch, compress):
         fsw.property('data', compressed)
 
 
+def find_gen_initramfs():
+    """Find the gen_initramfs.sh script
+
+    Looks for usr/gen_initramfs.sh relative to the current directory,
+    or in the source directory if running from a build directory.
+
+    Returns:
+        str: Path to gen_initramfs.sh if found, None otherwise
+    """
+    path = 'usr/gen_initramfs.sh'
+    if not os.path.exists(path):
+        srctree = os.environ.get('srctree')
+        if not srctree:
+            return None
+        path = os.path.join(srctree, path)
+        if not os.path.exists(path):
+            return None
+
+    return path
+
+
+def build_ramdisk(args, tmpdir):
+    """Build a cpio ramdisk containing kernel modules
+
+    Similar to mkinitramfs, this creates a compressed cpio-archive containing
+    the kernel modules for the current kernel version.
+
+    Args:
+        args (Namespace): Program arguments
+        tmpdir (str): Temporary directory to use for modules installation
+
+    Returns:
+        tuple:
+            bytes: Compressed cpio data containing modules
+            int: total uncompressed size
+    """
+    suppress = None if args.verbose else subprocess.DEVNULL
+
+    if args.verbose:
+        print(f'Copying {len(args.modules)} modules to ramdisk')
+
+    # Create output-directory structure
+    outdir = os.path.join(tmpdir, 'initramfs')
+    modules_dir = os.path.join(outdir, 'usr', 'lib', 'modules')
+    os.makedirs(modules_dir, exist_ok=True)
+
+    # Copy in the specified modules, renaming .o to .ko
+    for module in args.modules:
+        name, ext = os.path.splitext(module)
+        dest = name + '.ko' if ext == '.o' else module
+
+        # Preserve directory structure under modules_dir
+        dest_path = os.path.join(modules_dir, dest)
+        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
+        shutil.copy2(module, dest_path)
+
+    if args.verbose:
+        print(f'Creating cpio archive from {outdir}')
+
+    with tempfile.NamedTemporaryFile() as cpio_file:
+        # Use gen_initramfs.sh to create the cpio archive if possible
+        gen_initramfs = find_gen_initramfs()
+        if gen_initramfs:
+            subprocess.run([gen_initramfs, '-o', cpio_file.name, outdir],
+                           check=True, stdout=suppress, stderr=suppress)
+        else:
+            # Change to initramfs directory and create cpio archive
+            with subprocess.Popen(['find', '.', '-print0'], cwd=outdir,
+                                  stdout=subprocess.PIPE) as find:
+                with subprocess.Popen(['cpio', '-o', '-0', '-H', 'newc'],
+                                      stdin=find.stdout, stdout=cpio_file,
+                                      stderr=suppress, cwd=outdir) as cpio:
+                    find.stdout.close()
+                    cpio.wait()
+                    find.wait()
+
+        cpio_file.seek(0)  # Reset to beginning for reading
+        if args.verbose:
+            print('Reading ramdisk...' if args.compress == 'none' else
+                  f'Compressing ramdisk with {args.compress}...')
+        return compress_data(cpio_file, args.compress), cpio_file.tell()
+
+
 def process_dtb(fname, args):
     """Process an input DTB, decomposing it if requested and is possible
 
@@ -318,11 +414,12 @@ def _process_dtbs(args, fsw, entries, fdts):
     return seq, size
 
 
-def build_fit(args):
+def build_fit(args, tmpdir):
     """Build the FIT from the provided files and arguments
 
     Args:
         args (Namespace): Program arguments
+        tmpdir (str): Temporary directory for any temporary files
 
     Returns:
         tuple:
@@ -344,20 +441,29 @@ def build_fit(args):
 
     # Handle the ramdisk if provided. Compression is not supported as it is
     # already compressed.
+    ramdisk_data = None
     if args.ramdisk:
         with open(args.ramdisk, 'rb') as inf:
-            data = inf.read()
-        size += len(data)
-        write_ramdisk(fsw, data, args)
+            ramdisk_data = inf.read()
+        size += len(ramdisk_data)
+    elif args.modules:
+        if args.verbose:
+            print('Building modules ramdisk...')
+        ramdisk_data, uncomp_size = build_ramdisk(args, tmpdir)
+        size += uncomp_size
+
+    if ramdisk_data:
+        write_ramdisk(fsw, ramdisk_data, args)
 
     count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
     size += fdt_size
 
-    finish_fit(fsw, entries, bool(args.ramdisk))
+    finish_fit(fsw, entries, has_ramdisk=bool(ramdisk_data))
 
-    # Include the kernel itself in the returned file count
     fdt = fsw.as_fdt()
     fdt.pack()
+
+    # Count FDT files, kernel, plus ramdisk if present
     return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
 
 
@@ -365,7 +471,12 @@ def run_make_fit():
     """Run the tool's main logic"""
     args = parse_args()
 
-    out_data, count, size = build_fit(args)
+    tmpdir = tempfile.mkdtemp(prefix='make_fit_')
+    try:
+        out_data, count, size = build_fit(args, tmpdir)
+    finally:
+        shutil.rmtree(tmpdir)
+
     with open(args.output, 'wb') as outf:
         outf.write(out_data)
 
-- 
2.43.0

Re: [PATCH v6 4/8] scripts/make_fit: Provide a way to add built modules

[Ahmad Fatoum]

Hello Simon,

On 11/19/25 7:13 PM, Simon Glass wrote:
> Provide arguments to support building a ramdisk from a directory tree of
> modules. This is a convenient way to try out a kernel with its modules.
> 
> This makes use usr/gen_initramfs.sh where it can be found. Failing that
> it uses the cpio tool rather than attempting to use a python module or
> our own code. The list of modules is provided in a file.

I don't think the fallback is a good idea. This script is part of the
kernel tree and usr/gen_initramfs.sh is as well, so we shouldn't
duplicate the logic. With a dependency on gen_init_cpio (See my
modules-cpio-pkg patch), we should always have the script and the
executable available and it's fine to throw an error if it's not.

Thanks for putting in the work!

Cheers,
Ahmad


> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> ---
> Note: The approach of depending on modules_install caused build problems
> as mentioned in the cover letter.
> 
> I dropped the review tag from Nicolas Schier since there are quite a few
> changes in this version.
> 
> Changes in v6:
> - Using the modules.order file instead of 'find'
> - Use gen_initramfs.sh where available
> 
> Changes in v4:
> - Provide the list of modules from the Makefile
> - Reduce verbosity (don't print every module filename)
> 
> Changes in v3:
> - Add a way to add built modules into the FIT
> 
>  scripts/make_fit.py | 127 +++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 119 insertions(+), 8 deletions(-)
> 
> diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> index 1a74a9dcd85e..9dfef11fc4b3 100755
> --- a/scripts/make_fit.py
> +++ b/scripts/make_fit.py
> @@ -13,11 +13,17 @@ Usage:
>          -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
>          -E -c gzip
>  
> +    # Build with modules ramdisk instead of external ramdisk:
> +    make_fit.py -A arm64 -n 'Linux-6.17' -O linux
> +        -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
> +        -m module1.ko module2.ko module3.ko @arch/arm64/boot/dts/dtbs-list
> +
>  Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
>  devicetree files, either specified individually or listed in a file (with an
>  '@' prefix).
>  
>  Use -r to specify an existing ramdisk/initrd file.
> +Use -m to build a ramdisk from specified kernel module files.
>  
>  Use -E to generate an external FIT (where the data is placed after the
>  FIT data structure). This allows parsing of the data without loading
> @@ -38,6 +44,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
>  import argparse
>  import collections
>  import os
> +import shutil
>  import subprocess
>  import sys
>  import tempfile
> @@ -83,8 +90,14 @@ def parse_args():
>            help='Specifies the operating system')
>      parser.add_argument('-k', '--kernel', type=str, required=True,
>            help='Specifies the (uncompressed) kernel input file (.itk)')
> -    parser.add_argument('-r', '--ramdisk', type=str,
> +
> +    # Create mutually exclusive group for ramdisk options
> +    rd_group = parser.add_mutually_exclusive_group()
> +    rd_group.add_argument('-r', '--ramdisk', type=str,
>            help='Specifies the ramdisk/initrd input file')
> +    rd_group.add_argument('-m', '--modules', type=str, nargs='+',
> +          help='List of module filenames to include in ramdisk')
> +
>      parser.add_argument('-v', '--verbose', action='store_true',
>                          help='Enable verbose output')
>      parser.add_argument('dtbs', type=str, nargs='*',
> @@ -240,6 +253,89 @@ def output_dtb(fsw, seq, fname, arch, compress):
>          fsw.property('data', compressed)
>  
>  
> +def find_gen_initramfs():
> +    """Find the gen_initramfs.sh script
> +
> +    Looks for usr/gen_initramfs.sh relative to the current directory,
> +    or in the source directory if running from a build directory.
> +
> +    Returns:
> +        str: Path to gen_initramfs.sh if found, None otherwise
> +    """
> +    path = 'usr/gen_initramfs.sh'
> +    if not os.path.exists(path):
> +        srctree = os.environ.get('srctree')
> +        if not srctree:
> +            return None
> +        path = os.path.join(srctree, path)
> +        if not os.path.exists(path):
> +            return None
> +
> +    return path
> +
> +
> +def build_ramdisk(args, tmpdir):
> +    """Build a cpio ramdisk containing kernel modules
> +
> +    Similar to mkinitramfs, this creates a compressed cpio-archive containing
> +    the kernel modules for the current kernel version.
> +
> +    Args:
> +        args (Namespace): Program arguments
> +        tmpdir (str): Temporary directory to use for modules installation
> +
> +    Returns:
> +        tuple:
> +            bytes: Compressed cpio data containing modules
> +            int: total uncompressed size
> +    """
> +    suppress = None if args.verbose else subprocess.DEVNULL
> +
> +    if args.verbose:
> +        print(f'Copying {len(args.modules)} modules to ramdisk')
> +
> +    # Create output-directory structure
> +    outdir = os.path.join(tmpdir, 'initramfs')
> +    modules_dir = os.path.join(outdir, 'usr', 'lib', 'modules')
> +    os.makedirs(modules_dir, exist_ok=True)
> +
> +    # Copy in the specified modules, renaming .o to .ko
> +    for module in args.modules:
> +        name, ext = os.path.splitext(module)
> +        dest = name + '.ko' if ext == '.o' else module
> +
> +        # Preserve directory structure under modules_dir
> +        dest_path = os.path.join(modules_dir, dest)
> +        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
> +        shutil.copy2(module, dest_path)
> +
> +    if args.verbose:
> +        print(f'Creating cpio archive from {outdir}')
> +
> +    with tempfile.NamedTemporaryFile() as cpio_file:
> +        # Use gen_initramfs.sh to create the cpio archive if possible
> +        gen_initramfs = find_gen_initramfs()
> +        if gen_initramfs:
> +            subprocess.run([gen_initramfs, '-o', cpio_file.name, outdir],
> +                           check=True, stdout=suppress, stderr=suppress)
> +        else:
> +            # Change to initramfs directory and create cpio archive
> +            with subprocess.Popen(['find', '.', '-print0'], cwd=outdir,
> +                                  stdout=subprocess.PIPE) as find:
> +                with subprocess.Popen(['cpio', '-o', '-0', '-H', 'newc'],
> +                                      stdin=find.stdout, stdout=cpio_file,
> +                                      stderr=suppress, cwd=outdir) as cpio:
> +                    find.stdout.close()
> +                    cpio.wait()
> +                    find.wait()
> +
> +        cpio_file.seek(0)  # Reset to beginning for reading
> +        if args.verbose:
> +            print('Reading ramdisk...' if args.compress == 'none' else
> +                  f'Compressing ramdisk with {args.compress}...')
> +        return compress_data(cpio_file, args.compress), cpio_file.tell()

I don't think the fallback is a good idea as it duplicates logic we have
elsewhere. Just t

> +
> +
>  def process_dtb(fname, args):
>      """Process an input DTB, decomposing it if requested and is possible
>  
> @@ -318,11 +414,12 @@ def _process_dtbs(args, fsw, entries, fdts):
>      return seq, size
>  
>  
> -def build_fit(args):
> +def build_fit(args, tmpdir):
>      """Build the FIT from the provided files and arguments
>  
>      Args:
>          args (Namespace): Program arguments
> +        tmpdir (str): Temporary directory for any temporary files
>  
>      Returns:
>          tuple:
> @@ -344,20 +441,29 @@ def build_fit(args):
>  
>      # Handle the ramdisk if provided. Compression is not supported as it is
>      # already compressed.
> +    ramdisk_data = None
>      if args.ramdisk:
>          with open(args.ramdisk, 'rb') as inf:
> -            data = inf.read()
> -        size += len(data)
> -        write_ramdisk(fsw, data, args)
> +            ramdisk_data = inf.read()
> +        size += len(ramdisk_data)
> +    elif args.modules:
> +        if args.verbose:
> +            print('Building modules ramdisk...')
> +        ramdisk_data, uncomp_size = build_ramdisk(args, tmpdir)
> +        size += uncomp_size
> +
> +    if ramdisk_data:
> +        write_ramdisk(fsw, ramdisk_data, args)
>  
>      count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
>      size += fdt_size
>  
> -    finish_fit(fsw, entries, bool(args.ramdisk))
> +    finish_fit(fsw, entries, has_ramdisk=bool(ramdisk_data))
>  
> -    # Include the kernel itself in the returned file count
>      fdt = fsw.as_fdt()
>      fdt.pack()
> +
> +    # Count FDT files, kernel, plus ramdisk if present
>      return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
>  
>  
> @@ -365,7 +471,12 @@ def run_make_fit():
>      """Run the tool's main logic"""
>      args = parse_args()
>  
> -    out_data, count, size = build_fit(args)
> +    tmpdir = tempfile.mkdtemp(prefix='make_fit_')
> +    try:
> +        out_data, count, size = build_fit(args, tmpdir)
> +    finally:
> +        shutil.rmtree(tmpdir)
> +
>      with open(args.output, 'wb') as outf:
>          outf.write(out_data)
>  

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

Re: [PATCH v6 4/8] scripts/make_fit: Provide a way to add built modules

[Simon Glass]

Hi Ahmad,

On Wed, 26 Nov 2025 at 04:09, Ahmad Fatoum <a.fatoum@pengutronix.de> wrote:
>
> Hello Simon,
>
> On 11/19/25 7:13 PM, Simon Glass wrote:
> > Provide arguments to support building a ramdisk from a directory tree of
> > modules. This is a convenient way to try out a kernel with its modules.
> >
> > This makes use usr/gen_initramfs.sh where it can be found. Failing that
> > it uses the cpio tool rather than attempting to use a python module or
> > our own code. The list of modules is provided in a file.
>
> I don't think the fallback is a good idea. This script is part of the
> kernel tree and usr/gen_initramfs.sh is as well, so we shouldn't
> duplicate the logic. With a dependency on gen_init_cpio (See my
> modules-cpio-pkg patch), we should always have the script and the
> executable available and it's fine to throw an error if it's not.

For now I would like to leave it in, since it works. I mentioned the
trouble I had trying to depend on a target which depends on an install
target. But once your patch goes in I can try again with the targets
and see if I can make it work.

>
> Thanks for putting in the work!

Cheers! It has certainly been an effort :-)

Regards,
Simon

>
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > ---
> > Note: The approach of depending on modules_install caused build problems
> > as mentioned in the cover letter.
> >
> > I dropped the review tag from Nicolas Schier since there are quite a few
> > changes in this version.
> >
> > Changes in v6:
> > - Using the modules.order file instead of 'find'
> > - Use gen_initramfs.sh where available
> >
> > Changes in v4:
> > - Provide the list of modules from the Makefile
> > - Reduce verbosity (don't print every module filename)
> >
> > Changes in v3:
> > - Add a way to add built modules into the FIT
> >
> >  scripts/make_fit.py | 127 +++++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 119 insertions(+), 8 deletions(-)
> >
> > diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> > index 1a74a9dcd85e..9dfef11fc4b3 100755
> > --- a/scripts/make_fit.py
> > +++ b/scripts/make_fit.py
> > @@ -13,11 +13,17 @@ Usage:
> >          -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-list
> >          -E -c gzip
> >
> > +    # Build with modules ramdisk instead of external ramdisk:
> > +    make_fit.py -A arm64 -n 'Linux-6.17' -O linux
> > +        -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk
> > +        -m module1.ko module2.ko module3.ko @arch/arm64/boot/dts/dtbs-list
> > +
> >  Creates a FIT containing the supplied kernel, an optional ramdisk, and a set of
> >  devicetree files, either specified individually or listed in a file (with an
> >  '@' prefix).
> >
> >  Use -r to specify an existing ramdisk/initrd file.
> > +Use -m to build a ramdisk from specified kernel module files.
> >
> >  Use -E to generate an external FIT (where the data is placed after the
> >  FIT data structure). This allows parsing of the data without loading
> > @@ -38,6 +44,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
> >  import argparse
> >  import collections
> >  import os
> > +import shutil
> >  import subprocess
> >  import sys
> >  import tempfile
> > @@ -83,8 +90,14 @@ def parse_args():
> >            help='Specifies the operating system')
> >      parser.add_argument('-k', '--kernel', type=str, required=True,
> >            help='Specifies the (uncompressed) kernel input file (.itk)')
> > -    parser.add_argument('-r', '--ramdisk', type=str,
> > +
> > +    # Create mutually exclusive group for ramdisk options
> > +    rd_group = parser.add_mutually_exclusive_group()
> > +    rd_group.add_argument('-r', '--ramdisk', type=str,
> >            help='Specifies the ramdisk/initrd input file')
> > +    rd_group.add_argument('-m', '--modules', type=str, nargs='+',
> > +          help='List of module filenames to include in ramdisk')
> > +
> >      parser.add_argument('-v', '--verbose', action='store_true',
> >                          help='Enable verbose output')
> >      parser.add_argument('dtbs', type=str, nargs='*',
> > @@ -240,6 +253,89 @@ def output_dtb(fsw, seq, fname, arch, compress):
> >          fsw.property('data', compressed)
> >
> >
> > +def find_gen_initramfs():
> > +    """Find the gen_initramfs.sh script
> > +
> > +    Looks for usr/gen_initramfs.sh relative to the current directory,
> > +    or in the source directory if running from a build directory.
> > +
> > +    Returns:
> > +        str: Path to gen_initramfs.sh if found, None otherwise
> > +    """
> > +    path = 'usr/gen_initramfs.sh'
> > +    if not os.path.exists(path):
> > +        srctree = os.environ.get('srctree')
> > +        if not srctree:
> > +            return None
> > +        path = os.path.join(srctree, path)
> > +        if not os.path.exists(path):
> > +            return None
> > +
> > +    return path
> > +
> > +
> > +def build_ramdisk(args, tmpdir):
> > +    """Build a cpio ramdisk containing kernel modules
> > +
> > +    Similar to mkinitramfs, this creates a compressed cpio-archive containing
> > +    the kernel modules for the current kernel version.
> > +
> > +    Args:
> > +        args (Namespace): Program arguments
> > +        tmpdir (str): Temporary directory to use for modules installation
> > +
> > +    Returns:
> > +        tuple:
> > +            bytes: Compressed cpio data containing modules
> > +            int: total uncompressed size
> > +    """
> > +    suppress = None if args.verbose else subprocess.DEVNULL
> > +
> > +    if args.verbose:
> > +        print(f'Copying {len(args.modules)} modules to ramdisk')
> > +
> > +    # Create output-directory structure
> > +    outdir = os.path.join(tmpdir, 'initramfs')
> > +    modules_dir = os.path.join(outdir, 'usr', 'lib', 'modules')
> > +    os.makedirs(modules_dir, exist_ok=True)
> > +
> > +    # Copy in the specified modules, renaming .o to .ko
> > +    for module in args.modules:
> > +        name, ext = os.path.splitext(module)
> > +        dest = name + '.ko' if ext == '.o' else module
> > +
> > +        # Preserve directory structure under modules_dir
> > +        dest_path = os.path.join(modules_dir, dest)
> > +        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
> > +        shutil.copy2(module, dest_path)
> > +
> > +    if args.verbose:
> > +        print(f'Creating cpio archive from {outdir}')
> > +
> > +    with tempfile.NamedTemporaryFile() as cpio_file:
> > +        # Use gen_initramfs.sh to create the cpio archive if possible
> > +        gen_initramfs = find_gen_initramfs()
> > +        if gen_initramfs:
> > +            subprocess.run([gen_initramfs, '-o', cpio_file.name, outdir],
> > +                           check=True, stdout=suppress, stderr=suppress)
> > +        else:
> > +            # Change to initramfs directory and create cpio archive
> > +            with subprocess.Popen(['find', '.', '-print0'], cwd=outdir,
> > +                                  stdout=subprocess.PIPE) as find:
> > +                with subprocess.Popen(['cpio', '-o', '-0', '-H', 'newc'],
> > +                                      stdin=find.stdout, stdout=cpio_file,
> > +                                      stderr=suppress, cwd=outdir) as cpio:
> > +                    find.stdout.close()
> > +                    cpio.wait()
> > +                    find.wait()
> > +
> > +        cpio_file.seek(0)  # Reset to beginning for reading
> > +        if args.verbose:
> > +            print('Reading ramdisk...' if args.compress == 'none' else
> > +                  f'Compressing ramdisk with {args.compress}...')
> > +        return compress_data(cpio_file, args.compress), cpio_file.tell()
>
> I don't think the fallback is a good idea as it duplicates logic we have
> elsewhere. Just t
>
> > +
> > +
> >  def process_dtb(fname, args):
> >      """Process an input DTB, decomposing it if requested and is possible
> >
> > @@ -318,11 +414,12 @@ def _process_dtbs(args, fsw, entries, fdts):
> >      return seq, size
> >
> >
> > -def build_fit(args):
> > +def build_fit(args, tmpdir):
> >      """Build the FIT from the provided files and arguments
> >
> >      Args:
> >          args (Namespace): Program arguments
> > +        tmpdir (str): Temporary directory for any temporary files
> >
> >      Returns:
> >          tuple:
> > @@ -344,20 +441,29 @@ def build_fit(args):
> >
> >      # Handle the ramdisk if provided. Compression is not supported as it is
> >      # already compressed.
> > +    ramdisk_data = None
> >      if args.ramdisk:
> >          with open(args.ramdisk, 'rb') as inf:
> > -            data = inf.read()
> > -        size += len(data)
> > -        write_ramdisk(fsw, data, args)
> > +            ramdisk_data = inf.read()
> > +        size += len(ramdisk_data)
> > +    elif args.modules:
> > +        if args.verbose:
> > +            print('Building modules ramdisk...')
> > +        ramdisk_data, uncomp_size = build_ramdisk(args, tmpdir)
> > +        size += uncomp_size
> > +
> > +    if ramdisk_data:
> > +        write_ramdisk(fsw, ramdisk_data, args)
> >
> >      count, fdt_size = _process_dtbs(args, fsw, entries, fdts)
> >      size += fdt_size
> >
> > -    finish_fit(fsw, entries, bool(args.ramdisk))
> > +    finish_fit(fsw, entries, has_ramdisk=bool(ramdisk_data))
> >
> > -    # Include the kernel itself in the returned file count
> >      fdt = fsw.as_fdt()
> >      fdt.pack()
> > +
> > +    # Count FDT files, kernel, plus ramdisk if present
> >      return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size
> >
> >
> > @@ -365,7 +471,12 @@ def run_make_fit():
> >      """Run the tool's main logic"""
> >      args = parse_args()
> >
> > -    out_data, count, size = build_fit(args)
> > +    tmpdir = tempfile.mkdtemp(prefix='make_fit_')
> > +    try:
> > +        out_data, count, size = build_fit(args, tmpdir)
> > +    finally:
> > +        shutil.rmtree(tmpdir)
> > +
> >      with open(args.output, 'wb') as outf:
> >          outf.write(out_data)
> >
>
> --
> Pengutronix e.K.                  |                             |
> Steuerwalder Str. 21              | http://www.pengutronix.de/  |
> 31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |
>

[PATCH v6 5/8] kbuild: Split out module targets into a variable

[Simon Glass]

Add a modules-targets variable to list the targets which cause modules
to be built, since we want to add a conditional target.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v5)

Changes in v5:
- Add a new patch to split out module targets into a variable

 Makefile | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 638bc09a546a..8cd46222fc48 100644
--- a/Makefile
+++ b/Makefile
@@ -772,7 +772,12 @@ endif
 # in addition to whatever we do anyway.
 # Just "make" or "make all" shall build modules as well
 
-ifneq ($(filter all modules nsdeps compile_commands.json clang-%,$(MAKECMDGOALS)),)
+modules-targets := all
+modules-targets += modules
+modules-targets += nsdeps
+modules-targets += compile_commands.json
+modules-targets += clang-%
+ifneq ($(filter $(modules-targets),$(MAKECMDGOALS)),)
   KBUILD_MODULES := y
 endif
 
-- 
2.43.0

Re: [PATCH v6 5/8] kbuild: Split out module targets into a variable

[Nicolas Schier]

On Wed, Nov 19, 2025 at 11:13:26AM -0700, Simon Glass wrote:
> Add a modules-targets variable to list the targets which cause modules
> to be built, since we want to add a conditional target.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
> (no changes since v5)
> 
> Changes in v5:
> - Add a new patch to split out module targets into a variable
> 
>  Makefile | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 638bc09a546a..8cd46222fc48 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -772,7 +772,12 @@ endif
>  # in addition to whatever we do anyway.
>  # Just "make" or "make all" shall build modules as well
>  
> -ifneq ($(filter all modules nsdeps compile_commands.json clang-%,$(MAKECMDGOALS)),)
> +modules-targets := all
> +modules-targets += modules
> +modules-targets += nsdeps
> +modules-targets += compile_commands.json
> +modules-targets += clang-%
> +ifneq ($(filter $(modules-targets),$(MAKECMDGOALS)),)
>    KBUILD_MODULES := y
>  endif
>  
> -- 
> 2.43.0
> 

Reviewed-by: Nicolas Schier <nsc@kernel.org>

Re: [PATCH v6 5/8] kbuild: Split out module targets into a variable

[Ahmad Fatoum]

On 11/19/25 7:13 PM, Simon Glass wrote:
> Add a modules-targets variable to list the targets which cause modules
> to be built, since we want to add a conditional target.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Reviewed-by: Ahmad Fatoum <a.fatoum@pengutronix.de>

> ---
> 
> (no changes since v5)
> 
> Changes in v5:
> - Add a new patch to split out module targets into a variable
> 
>  Makefile | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 638bc09a546a..8cd46222fc48 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -772,7 +772,12 @@ endif
>  # in addition to whatever we do anyway.
>  # Just "make" or "make all" shall build modules as well
>  
> -ifneq ($(filter all modules nsdeps compile_commands.json clang-%,$(MAKECMDGOALS)),)
> +modules-targets := all
> +modules-targets += modules
> +modules-targets += nsdeps
> +modules-targets += compile_commands.json
> +modules-targets += clang-%
> +ifneq ($(filter $(modules-targets),$(MAKECMDGOALS)),)
>    KBUILD_MODULES := y
>  endif
>  

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

[PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Simon Glass]

Support 'make image.fit FIT_MODULES=1' to put all the modules into a
ramdisk image within the FIT.

Add image.fit as a target which requires modules, so that modules will
built automatically when using FIT_MODULES is not empty.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
Acked-by: Nathan Chancellor <nathan@kernel.org>
---

Changes in v6:
- Mention that FIT_MODULES just needs to be non-empty
- Make use of modules.order instead of using 'find'

Changes in v5:
- Build modules automatically if needed (fix from Nicolas Schier)

Changes in v4:
- Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
- Use an empty FIT_MODULES to disable the feature, instead of '0'
- Make use of the 'modules' dependency to ensure modules are built
- Pass the list of modules to the script

 Makefile             | 1 +
 arch/arm64/Makefile  | 1 +
 scripts/Makefile.lib | 6 +++++-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 8cd46222fc48..4eccaef95826 100644
--- a/Makefile
+++ b/Makefile
@@ -773,6 +773,7 @@ endif
 # Just "make" or "make all" shall build modules as well
 
 modules-targets := all
+modules-targets += $(if $(FIT_MODULES),image.fit)
 modules-targets += modules
 modules-targets += nsdeps
 modules-targets += compile_commands.json
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 73a10f65ce8b..7036f251ab40 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -174,6 +174,7 @@ endif
 all:	$(notdir $(KBUILD_IMAGE))
 
 image.fit: dtbs
+image.fit: $(if $(FIT_MODULES),modules)
 
 vmlinuz.efi image.fit: Image
 $(BOOT_TARGETS): vmlinux
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 1d581ba5df66..28e0cc0865b1 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -398,11 +398,15 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
 # Use this to override the compression algorithm
 FIT_COMPRESSION ?= gzip
 
+# Set this to non-empty to include an initrd with all the kernel modules
+FIT_MODULES ?=
+
 quiet_cmd_fit = FIT     $@
       cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
-		--name '$(UIMAGE_NAME)' \
+		--name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
 		$(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
 		$(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
+		$(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
 		--compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
 
 # XZ
-- 
2.43.0

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Nicolas Schier]

On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> Support 'make image.fit FIT_MODULES=1' to put all the modules into a
> ramdisk image within the FIT.
> 
> Add image.fit as a target which requires modules, so that modules will
> built automatically when using FIT_MODULES is not empty.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>

I think the

Reviewed-by: Nicolas Schier <nsc@kernel.org>

is enough :)

Thanks.


> Acked-by: Nathan Chancellor <nathan@kernel.org>
> ---
> 
> Changes in v6:
> - Mention that FIT_MODULES just needs to be non-empty
> - Make use of modules.order instead of using 'find'
> 
> Changes in v5:
> - Build modules automatically if needed (fix from Nicolas Schier)
> 
> Changes in v4:
> - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
> - Use an empty FIT_MODULES to disable the feature, instead of '0'
> - Make use of the 'modules' dependency to ensure modules are built
> - Pass the list of modules to the script
> 
>  Makefile             | 1 +
>  arch/arm64/Makefile  | 1 +
>  scripts/Makefile.lib | 6 +++++-
>  3 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 8cd46222fc48..4eccaef95826 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -773,6 +773,7 @@ endif
>  # Just "make" or "make all" shall build modules as well
>  
>  modules-targets := all
> +modules-targets += $(if $(FIT_MODULES),image.fit)
>  modules-targets += modules
>  modules-targets += nsdeps
>  modules-targets += compile_commands.json
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 73a10f65ce8b..7036f251ab40 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -174,6 +174,7 @@ endif
>  all:	$(notdir $(KBUILD_IMAGE))
>  
>  image.fit: dtbs
> +image.fit: $(if $(FIT_MODULES),modules)
>  
>  vmlinuz.efi image.fit: Image
>  $(BOOT_TARGETS): vmlinux
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 1d581ba5df66..28e0cc0865b1 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -398,11 +398,15 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
>  # Use this to override the compression algorithm
>  FIT_COMPRESSION ?= gzip
>  
> +# Set this to non-empty to include an initrd with all the kernel modules
> +FIT_MODULES ?=
> +
>  quiet_cmd_fit = FIT     $@
>        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> -		--name '$(UIMAGE_NAME)' \
> +		--name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
>  		$(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
>  		$(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> +		$(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
>  		--compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
>  
>  # XZ
> -- 
> 2.43.0
> 

-- 
Nicolas

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Thomas Weißschuh]

On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> Support 'make image.fit FIT_MODULES=1' to put all the modules into a
> ramdisk image within the FIT.
> 
> Add image.fit as a target which requires modules, so that modules will
> built automatically when using FIT_MODULES is not empty.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
> Acked-by: Nathan Chancellor <nathan@kernel.org>
> ---
> 
> Changes in v6:
> - Mention that FIT_MODULES just needs to be non-empty
> - Make use of modules.order instead of using 'find'
> 
> Changes in v5:
> - Build modules automatically if needed (fix from Nicolas Schier)
> 
> Changes in v4:
> - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
> - Use an empty FIT_MODULES to disable the feature, instead of '0'
> - Make use of the 'modules' dependency to ensure modules are built
> - Pass the list of modules to the script
> 
>  Makefile             | 1 +
>  arch/arm64/Makefile  | 1 +
>  scripts/Makefile.lib | 6 +++++-
>  3 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 8cd46222fc48..4eccaef95826 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -773,6 +773,7 @@ endif
>  # Just "make" or "make all" shall build modules as well
>  
>  modules-targets := all
> +modules-targets += $(if $(FIT_MODULES),image.fit)
>  modules-targets += modules
>  modules-targets += nsdeps
>  modules-targets += compile_commands.json
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index 73a10f65ce8b..7036f251ab40 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -174,6 +174,7 @@ endif
>  all:	$(notdir $(KBUILD_IMAGE))
>  
>  image.fit: dtbs
> +image.fit: $(if $(FIT_MODULES),modules)
>  
>  vmlinuz.efi image.fit: Image
>  $(BOOT_TARGETS): vmlinux
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 1d581ba5df66..28e0cc0865b1 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -398,11 +398,15 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
>  # Use this to override the compression algorithm
>  FIT_COMPRESSION ?= gzip
>  
> +# Set this to non-empty to include an initrd with all the kernel modules
> +FIT_MODULES ?=
> +
>  quiet_cmd_fit = FIT     $@
>        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> -		--name '$(UIMAGE_NAME)' \
> +		--name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \

Remnant of a previous revision?

>  		$(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
>  		$(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> +		$(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \

I am wondering how module dependencies work without the depmod invocation
and modules.dep file.

>  		--compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
>  
>  # XZ
> -- 
> 2.43.0
> 

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Nicolas Schier]

On Thu, Nov 20, 2025 at 08:49:14AM +0100, Thomas Weißschuh wrote:
> On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> > Support 'make image.fit FIT_MODULES=1' to put all the modules into a
> > ramdisk image within the FIT.
> > 
> > Add image.fit as a target which requires modules, so that modules will
> > built automatically when using FIT_MODULES is not empty.
> > 
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
> > Acked-by: Nathan Chancellor <nathan@kernel.org>
> > ---
> > 
> > Changes in v6:
> > - Mention that FIT_MODULES just needs to be non-empty
> > - Make use of modules.order instead of using 'find'
> > 
> > Changes in v5:
> > - Build modules automatically if needed (fix from Nicolas Schier)
> > 
> > Changes in v4:
> > - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
> > - Use an empty FIT_MODULES to disable the feature, instead of '0'
> > - Make use of the 'modules' dependency to ensure modules are built
> > - Pass the list of modules to the script
> > 
> >  Makefile             | 1 +
> >  arch/arm64/Makefile  | 1 +
> >  scripts/Makefile.lib | 6 +++++-
> >  3 files changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index 8cd46222fc48..4eccaef95826 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -773,6 +773,7 @@ endif
> >  # Just "make" or "make all" shall build modules as well
> >  
> >  modules-targets := all
> > +modules-targets += $(if $(FIT_MODULES),image.fit)
> >  modules-targets += modules
> >  modules-targets += nsdeps
> >  modules-targets += compile_commands.json
> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> > index 73a10f65ce8b..7036f251ab40 100644
> > --- a/arch/arm64/Makefile
> > +++ b/arch/arm64/Makefile
> > @@ -174,6 +174,7 @@ endif
> >  all:	$(notdir $(KBUILD_IMAGE))
> >  
> >  image.fit: dtbs
> > +image.fit: $(if $(FIT_MODULES),modules)
> >  
> >  vmlinuz.efi image.fit: Image
> >  $(BOOT_TARGETS): vmlinux
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index 1d581ba5df66..28e0cc0865b1 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -398,11 +398,15 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
> >  # Use this to override the compression algorithm
> >  FIT_COMPRESSION ?= gzip
> >  
> > +# Set this to non-empty to include an initrd with all the kernel modules
> > +FIT_MODULES ?=
> > +
> >  quiet_cmd_fit = FIT     $@
> >        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> > -		--name '$(UIMAGE_NAME)' \
> > +		--name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
> 
> Remnant of a previous revision?
> 
> >  		$(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
> >  		$(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> > +		$(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
> 
> I am wondering how module dependencies work without the depmod invocation
> and modules.dep file.

oh, good point.  Stripping and signing and module compression is also
done during 'modules_install'.


-- 
Nicolas

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Simon Glass]

Hi Thomas,

On Thu, 20 Nov 2025 at 00:49, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> > Support 'make image.fit FIT_MODULES=1' to put all the modules into a
> > ramdisk image within the FIT.
> >
> > Add image.fit as a target which requires modules, so that modules will
> > built automatically when using FIT_MODULES is not empty.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > Suggested-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > Suggested-by: Reviewed-by: Nicolas Schier <nsc@kernel.org>
> > Acked-by: Nathan Chancellor <nathan@kernel.org>
> > ---
> >
> > Changes in v6:
> > - Mention that FIT_MODULES just needs to be non-empty
> > - Make use of modules.order instead of using 'find'
> >
> > Changes in v5:
> > - Build modules automatically if needed (fix from Nicolas Schier)
> >
> > Changes in v4:
> > - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS'
> > - Use an empty FIT_MODULES to disable the feature, instead of '0'
> > - Make use of the 'modules' dependency to ensure modules are built
> > - Pass the list of modules to the script
> >
> >  Makefile             | 1 +
> >  arch/arm64/Makefile  | 1 +
> >  scripts/Makefile.lib | 6 +++++-
> >  3 files changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/Makefile b/Makefile
> > index 8cd46222fc48..4eccaef95826 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -773,6 +773,7 @@ endif
> >  # Just "make" or "make all" shall build modules as well
> >
> >  modules-targets := all
> > +modules-targets += $(if $(FIT_MODULES),image.fit)
> >  modules-targets += modules
> >  modules-targets += nsdeps
> >  modules-targets += compile_commands.json
> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> > index 73a10f65ce8b..7036f251ab40 100644
> > --- a/arch/arm64/Makefile
> > +++ b/arch/arm64/Makefile
> > @@ -174,6 +174,7 @@ endif
> >  all: $(notdir $(KBUILD_IMAGE))
> >
> >  image.fit: dtbs
> > +image.fit: $(if $(FIT_MODULES),modules)
> >
> >  vmlinuz.efi image.fit: Image
> >  $(BOOT_TARGETS): vmlinux
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index 1d581ba5df66..28e0cc0865b1 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -398,11 +398,15 @@ MAKE_FIT := $(srctree)/scripts/make_fit.py
> >  # Use this to override the compression algorithm
> >  FIT_COMPRESSION ?= gzip
> >
> > +# Set this to non-empty to include an initrd with all the kernel modules
> > +FIT_MODULES ?=
> > +
> >  quiet_cmd_fit = FIT     $@
> >        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> > -             --name '$(UIMAGE_NAME)' \
> > +             --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
>
> Remnant of a previous revision?

The flags are there to allow extra options to be passed if needed.

>
> >               $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
> >               $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> > +             $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
>
> I am wondering how module dependencies work without the depmod invocation
> and modules.dep file.

We have a mechanism to place a pre-build initrd with the filesystem,
etc. into the FIT. But for this particular feature (suggested by Ahmad
Fatoum) we are just providing the raw modules. Presumably another
initrd would be needed to provide the startup files?

>
> >               --compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
> >
> >  # XZ
> > --
> > 2.43.0
> >

Regards,
Simon

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Thomas Weißschuh]

On Tue, Nov 25, 2025 at 02:58:12PM -0700, Simon Glass wrote:
> On Thu, 20 Nov 2025 at 00:49, Thomas Weißschuh
> <thomas.weissschuh@linutronix.de> wrote:
> >
> > On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:

(...)

> > >  quiet_cmd_fit = FIT     $@
> > >        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> > > -             --name '$(UIMAGE_NAME)' \
> > > +             --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
> >
> > Remnant of a previous revision?
> 
> The flags are there to allow extra options to be passed if needed.

Are they necessary for the module functionality added here?
If not I'd put them into a dedicated commit.

> >
> > >               $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
> > >               $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> > > +             $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
> >
> > I am wondering how module dependencies work without the depmod invocation
> > and modules.dep file.
> 
> We have a mechanism to place a pre-build initrd with the filesystem,
> etc. into the FIT. But for this particular feature (suggested by Ahmad
> Fatoum) we are just providing the raw modules. Presumably another
> initrd would be needed to provide the startup files?

modules.dep is more than optional and generic startup files but an integral
part of a module tree. Without it, any module depending on another module's
symbols will fail to load. Also the modules will be unsigned, potentially
making them unloadable. Ahmad's patch does produce a complete and fully
functional module tree by means of 'make headers_install'.

> > >               --compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
> > >
> > >  # XZ
> > > --
> > > 2.43.0

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Ahmad Fatoum]

Hi,

On 11/26/25 8:16 AM, Thomas Weißschuh wrote:
> On Tue, Nov 25, 2025 at 02:58:12PM -0700, Simon Glass wrote:
>> On Thu, 20 Nov 2025 at 00:49, Thomas Weißschuh
>> <thomas.weissschuh@linutronix.de> wrote:
>>>
>>> On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> 
> (...)
> 
>>>>  quiet_cmd_fit = FIT     $@
>>>>        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
>>>> -             --name '$(UIMAGE_NAME)' \
>>>> +             --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
>>>
>>> Remnant of a previous revision?
>>
>> The flags are there to allow extra options to be passed if needed.
> 
> Are they necessary for the module functionality added here?
> If not I'd put them into a dedicated commit.
> 
>>>
>>>>               $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
>>>>               $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
>>>> +             $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
>>>
>>> I am wondering how module dependencies work without the depmod invocation
>>> and modules.dep file.
>>
>> We have a mechanism to place a pre-build initrd with the filesystem,
>> etc. into the FIT. But for this particular feature (suggested by Ahmad
>> Fatoum) we are just providing the raw modules. Presumably another
>> initrd would be needed to provide the startup files?
> 
> modules.dep is more than optional and generic startup files but an integral
> part of a module tree. Without it, any module depending on another module's
> symbols will fail to load. Also the modules will be unsigned, potentially
> making them unloadable.

I'll use the occasion to elaborate a bit on why I thought adding modules
is a good idea.

- You have a system boot from FIT and maybe even a r/o rootfs
- You want to boot a different kernel without any userspace changes,
e.g. to bisect
- Fortunately, you have a build target that generates you a FIT with
kernel, enabled device trees and all modules (including deps and such)
- In the bootloader[1], you specify that a CPIO with a minimal init[2]
that bindmounts /lib/modules in the initramfs over the rootfs modules
before pivot_root

and that's it, you are running your new kernel with the old rootfs
unchanged. I believe this would be really handy, which is why I
suggested it.

> Ahmad's patch does produce a complete and fully
> functional module tree by means of 'make headers_install'.

I originally thought that we could generate the CPIO normally as part of
the kernel build and then we can readily depend on it in the rule that
invokes make_fit.py.

If this proves to be too cumbersome, I think it's already an improvement
if the user can manually run make modules-cpio-pkg and then make
image.fit with the initrd specified. A single target would be neater of
course, but I didn't intend for this to stall the series.

It can always follow later.

[1]: For my particular usage, I intend to extend the barebox boot
override mechanism. Currently, it's possible to do:
barebox$ boot -o bootm.initrd=/mnt/tftp/my-ramdisk.cpio /mnt/tftp/my-fit
and I want to extend it to allow appending any number of CPIOs:
barebox$ boot -o bootm.initrd=":/mnt/tftp/my-init.cpio" /mnt/tftp/my-fit

[2]: The bind mount logic for the initramfs init will probably go into
https://github.com/pengutronix/rsinit, which I can compile once per
architecture, put into my TFTP srv path and then not have to worry about it.

Cheers,
Ahmad

> 
>>>>               --compress $(FIT_COMPRESSION) -k $< @$(word 2,$^)
>>>>
>>>>  # XZ
>>>> --
>>>> 2.43.0
> 

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Thomas Weißschuh]

On Wed, Nov 26, 2025 at 12:26:49PM +0100, Ahmad Fatoum wrote:
> On 11/26/25 8:16 AM, Thomas Weißschuh wrote:
> > On Tue, Nov 25, 2025 at 02:58:12PM -0700, Simon Glass wrote:
> >> On Thu, 20 Nov 2025 at 00:49, Thomas Weißschuh
> >> <thomas.weissschuh@linutronix.de> wrote:
> >>>
> >>> On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> > 
> > (...)
> > 
> >>>>  quiet_cmd_fit = FIT     $@
> >>>>        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> >>>> -             --name '$(UIMAGE_NAME)' \
> >>>> +             --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
> >>>
> >>> Remnant of a previous revision?
> >>
> >> The flags are there to allow extra options to be passed if needed.
> > 
> > Are they necessary for the module functionality added here?
> > If not I'd put them into a dedicated commit.
> > 
> >>>
> >>>>               $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
> >>>>               $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> >>>> +             $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
> >>>
> >>> I am wondering how module dependencies work without the depmod invocation
> >>> and modules.dep file.
> >>
> >> We have a mechanism to place a pre-build initrd with the filesystem,
> >> etc. into the FIT. But for this particular feature (suggested by Ahmad
> >> Fatoum) we are just providing the raw modules. Presumably another
> >> initrd would be needed to provide the startup files?
> > 
> > modules.dep is more than optional and generic startup files but an integral
> > part of a module tree. Without it, any module depending on another module's
> > symbols will fail to load. Also the modules will be unsigned, potentially
> > making them unloadable.
> 
> I'll use the occasion to elaborate a bit on why I thought adding modules
> is a good idea.
> 
> - You have a system boot from FIT and maybe even a r/o rootfs
> - You want to boot a different kernel without any userspace changes,
> e.g. to bisect
> - Fortunately, you have a build target that generates you a FIT with
> kernel, enabled device trees and all modules (including deps and such)
> - In the bootloader[1], you specify that a CPIO with a minimal init[2]
> that bindmounts /lib/modules in the initramfs over the rootfs modules
> before pivot_root
> 
> and that's it, you are running your new kernel with the old rootfs
> unchanged. I believe this would be really handy, which is why I
> suggested it.

The idea sounds good.

> > Ahmad's patch does produce a complete and fully
> > functional module tree by means of 'make headers_install'.
> 
> I originally thought that we could generate the CPIO normally as part of
> the kernel build and then we can readily depend on it in the rule that
> invokes make_fit.py.

That works, but it is not what the patch under discussion does, or did.

> If this proves to be too cumbersome, I think it's already an improvement
> if the user can manually run make modules-cpio-pkg and then make
> image.fit with the initrd specified. A single target would be neater of
> course, but I didn't intend for this to stall the series.

The single target idea would require 'modules-cpio-pkg' to not be a PHONY
target anymore but to properly track dependencies. Otherwise the CPIO and FIT
image will be rebuilt even if no sources change. Proper dependencies are always
better than PHONY targets, but it will be a bit of additional work.

> It can always follow later.

Yep. But for the patch as it is proposed I am still wondering how it will work
without modules.dep and friends.

(...)

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Simon Glass]

Hi Thomas,

On Tue, 2 Dec 2025 at 03:31, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> On Wed, Nov 26, 2025 at 12:26:49PM +0100, Ahmad Fatoum wrote:
> > On 11/26/25 8:16 AM, Thomas Weißschuh wrote:
> > > On Tue, Nov 25, 2025 at 02:58:12PM -0700, Simon Glass wrote:
> > >> On Thu, 20 Nov 2025 at 00:49, Thomas Weißschuh
> > >> <thomas.weissschuh@linutronix.de> wrote:
> > >>>
> > >>> On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
> > >
> > > (...)
> > >
> > >>>>  quiet_cmd_fit = FIT     $@
> > >>>>        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
> > >>>> -             --name '$(UIMAGE_NAME)' \
> > >>>> +             --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
> > >>>
> > >>> Remnant of a previous revision?
> > >>
> > >> The flags are there to allow extra options to be passed if needed.
> > >
> > > Are they necessary for the module functionality added here?
> > > If not I'd put them into a dedicated commit.
> > >
> > >>>
> > >>>>               $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
> > >>>>               $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
> > >>>> +             $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
> > >>>
> > >>> I am wondering how module dependencies work without the depmod invocation
> > >>> and modules.dep file.
> > >>
> > >> We have a mechanism to place a pre-build initrd with the filesystem,
> > >> etc. into the FIT. But for this particular feature (suggested by Ahmad
> > >> Fatoum) we are just providing the raw modules. Presumably another
> > >> initrd would be needed to provide the startup files?
> > >
> > > modules.dep is more than optional and generic startup files but an integral
> > > part of a module tree. Without it, any module depending on another module's
> > > symbols will fail to load. Also the modules will be unsigned, potentially
> > > making them unloadable.
> >
> > I'll use the occasion to elaborate a bit on why I thought adding modules
> > is a good idea.
> >
> > - You have a system boot from FIT and maybe even a r/o rootfs
> > - You want to boot a different kernel without any userspace changes,
> > e.g. to bisect
> > - Fortunately, you have a build target that generates you a FIT with
> > kernel, enabled device trees and all modules (including deps and such)
> > - In the bootloader[1], you specify that a CPIO with a minimal init[2]
> > that bindmounts /lib/modules in the initramfs over the rootfs modules
> > before pivot_root
> >
> > and that's it, you are running your new kernel with the old rootfs
> > unchanged. I believe this would be really handy, which is why I
> > suggested it.
>
> The idea sounds good.
>
> > > Ahmad's patch does produce a complete and fully
> > > functional module tree by means of 'make headers_install'.
> >
> > I originally thought that we could generate the CPIO normally as part of
> > the kernel build and then we can readily depend on it in the rule that
> > invokes make_fit.py.
>
> That works, but it is not what the patch under discussion does, or did.
>
> > If this proves to be too cumbersome, I think it's already an improvement
> > if the user can manually run make modules-cpio-pkg and then make
> > image.fit with the initrd specified. A single target would be neater of
> > course, but I didn't intend for this to stall the series.
>
> The single target idea would require 'modules-cpio-pkg' to not be a PHONY
> target anymore but to properly track dependencies. Otherwise the CPIO and FIT
> image will be rebuilt even if no sources change. Proper dependencies are always
> better than PHONY targets, but it will be a bit of additional work.
>
> > It can always follow later.
>
> Yep. But for the patch as it is proposed I am still wondering how it will work
> without modules.dep and friends.
>
> (...)

I'm going to send a v7 and perhaps Ahmad can help to refine this.
Unfortunately the modules generation has turned into a significant
detour. We can either drop it, or continue to try to resolve this.

Regards,
SImon

Re: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk

[Ahmad Fatoum]

Hi Simon,

On 12/11/25 2:31 PM, Simon Glass wrote:
> Hi Thomas,
> 
> On Tue, 2 Dec 2025 at 03:31, Thomas Weißschuh
> <thomas.weissschuh@linutronix.de> wrote:
>>
>> On Wed, Nov 26, 2025 at 12:26:49PM +0100, Ahmad Fatoum wrote:
>>> On 11/26/25 8:16 AM, Thomas Weißschuh wrote:
>>>> On Tue, Nov 25, 2025 at 02:58:12PM -0700, Simon Glass wrote:
>>>>> On Thu, 20 Nov 2025 at 00:49, Thomas Weißschuh
>>>>> <thomas.weissschuh@linutronix.de> wrote:
>>>>>>
>>>>>> On Wed, Nov 19, 2025 at 11:13:27AM -0700, Simon Glass wrote:
>>>>
>>>> (...)
>>>>
>>>>>>>  quiet_cmd_fit = FIT     $@
>>>>>>>        cmd_fit = $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \
>>>>>>> -             --name '$(UIMAGE_NAME)' \
>>>>>>> +             --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \
>>>>>>
>>>>>> Remnant of a previous revision?
>>>>>
>>>>> The flags are there to allow extra options to be passed if needed.
>>>>
>>>> Are they necessary for the module functionality added here?
>>>> If not I'd put them into a dedicated commit.
>>>>
>>>>>>
>>>>>>>               $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \
>>>>>>>               $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \
>>>>>>> +             $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \
>>>>>>
>>>>>> I am wondering how module dependencies work without the depmod invocation
>>>>>> and modules.dep file.
>>>>>
>>>>> We have a mechanism to place a pre-build initrd with the filesystem,
>>>>> etc. into the FIT. But for this particular feature (suggested by Ahmad
>>>>> Fatoum) we are just providing the raw modules. Presumably another
>>>>> initrd would be needed to provide the startup files?
>>>>
>>>> modules.dep is more than optional and generic startup files but an integral
>>>> part of a module tree. Without it, any module depending on another module's
>>>> symbols will fail to load. Also the modules will be unsigned, potentially
>>>> making them unloadable.
>>>
>>> I'll use the occasion to elaborate a bit on why I thought adding modules
>>> is a good idea.
>>>
>>> - You have a system boot from FIT and maybe even a r/o rootfs
>>> - You want to boot a different kernel without any userspace changes,
>>> e.g. to bisect
>>> - Fortunately, you have a build target that generates you a FIT with
>>> kernel, enabled device trees and all modules (including deps and such)
>>> - In the bootloader[1], you specify that a CPIO with a minimal init[2]
>>> that bindmounts /lib/modules in the initramfs over the rootfs modules
>>> before pivot_root
>>>
>>> and that's it, you are running your new kernel with the old rootfs
>>> unchanged. I believe this would be really handy, which is why I
>>> suggested it.
>>
>> The idea sounds good.
>>
>>>> Ahmad's patch does produce a complete and fully
>>>> functional module tree by means of 'make headers_install'.
>>>
>>> I originally thought that we could generate the CPIO normally as part of
>>> the kernel build and then we can readily depend on it in the rule that
>>> invokes make_fit.py.
>>
>> That works, but it is not what the patch under discussion does, or did.
>>
>>> If this proves to be too cumbersome, I think it's already an improvement
>>> if the user can manually run make modules-cpio-pkg and then make
>>> image.fit with the initrd specified. A single target would be neater of
>>> course, but I didn't intend for this to stall the series.
>>
>> The single target idea would require 'modules-cpio-pkg' to not be a PHONY
>> target anymore but to properly track dependencies. Otherwise the CPIO and FIT
>> image will be rebuilt even if no sources change. Proper dependencies are always
>> better than PHONY targets, but it will be a bit of additional work.
>>
>>> It can always follow later.
>>
>> Yep. But for the patch as it is proposed I am still wondering how it will work
>> without modules.dep and friends.
>>
>> (...)
> 
> I'm going to send a v7 and perhaps Ahmad can help to refine this.
> Unfortunately the modules generation has turned into a significant
> detour. We can either drop it, or continue to try to resolve this.

I'd suggest to drop it and tackle that separately.

Sorry for the inconvenience,
Ahmad

> 
> Regards,
> SImon
> 

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |

[PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors

[Simon Glass]

Add support for pbzip2 and plzip which can compress in parallel. This
speeds up the ramdisk compression.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 scripts/make_fit.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 9dfef11fc4b3..64038f17a51e 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -57,10 +57,10 @@ import libfdt
 CompTool = collections.namedtuple('CompTool', 'ext,tools')
 
 COMP_TOOLS = {
-    'bzip2': CompTool('.bz2', 'bzip2'),
+    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
     'gzip': CompTool('.gz', 'pigz,gzip'),
     'lz4': CompTool('.lz4', 'lz4'),
-    'lzma': CompTool('.lzma', 'lzma'),
+    'lzma': CompTool('.lzma', 'plzip,lzma'),
     'lzo': CompTool('.lzo', 'lzop'),
     'zstd': CompTool('.zstd', 'zstd'),
 }
@@ -220,7 +220,12 @@ def compress_data(inf, compress):
             done = False
             for tool in comp.tools.split(','):
                 try:
-                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
+                    # Add parallel flags for tools that support them
+                    cmd = [tool]
+                    if tool in ('zstd', 'xz'):
+                        cmd.extend(['-T0'])  # Use all available cores
+                    cmd.append('-c')
+                    subprocess.call(cmd, stdin=inf, stdout=outf)
                     done = True
                     break
                 except FileNotFoundError:
-- 
2.43.0

Re: [PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors

[Chen-Yu Tsai]

On Thu, Nov 20, 2025 at 2:14 AM Simon Glass <sjg@chromium.org> wrote:
>
> Add support for pbzip2 and plzip which can compress in parallel. This
> speeds up the ramdisk compression.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> (no changes since v1)
>
>  scripts/make_fit.py | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> index 9dfef11fc4b3..64038f17a51e 100755
> --- a/scripts/make_fit.py
> +++ b/scripts/make_fit.py
> @@ -57,10 +57,10 @@ import libfdt
>  CompTool = collections.namedtuple('CompTool', 'ext,tools')
>
>  COMP_TOOLS = {
> -    'bzip2': CompTool('.bz2', 'bzip2'),
> +    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
>      'gzip': CompTool('.gz', 'pigz,gzip'),
>      'lz4': CompTool('.lz4', 'lz4'),
> -    'lzma': CompTool('.lzma', 'lzma'),
> +    'lzma': CompTool('.lzma', 'plzip,lzma'),
>      'lzo': CompTool('.lzo', 'lzop'),
>      'zstd': CompTool('.zstd', 'zstd'),
>  }
> @@ -220,7 +220,12 @@ def compress_data(inf, compress):
>              done = False
>              for tool in comp.tools.split(','):
>                  try:
> -                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
> +                    # Add parallel flags for tools that support them
> +                    cmd = [tool]
> +                    if tool in ('zstd', 'xz'):

There's no 'xz' tool in the list though.

And this hunk isn't mentioned or explained in the commit message.

> +                        cmd.extend(['-T0'])  # Use all available cores

Zero is already the default for 'xz' in 5.8.1.


ChenYu

> +                    cmd.append('-c')
> +                    subprocess.call(cmd, stdin=inf, stdout=outf)
>                      done = True
>                      break
>                  except FileNotFoundError:
> --
> 2.43.0
>

Re: [PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors

[Simon Glass]

Hi Chen-Yu,

On Tue, 2 Dec 2025 at 03:18, Chen-Yu Tsai <wenst@chromium.org> wrote:
>
> On Thu, Nov 20, 2025 at 2:14 AM Simon Glass <sjg@chromium.org> wrote:
> >
> > Add support for pbzip2 and plzip which can compress in parallel. This
> > speeds up the ramdisk compression.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> > (no changes since v1)
> >
> >  scripts/make_fit.py | 11 ++++++++---
> >  1 file changed, 8 insertions(+), 3 deletions(-)
> >
> > diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> > index 9dfef11fc4b3..64038f17a51e 100755
> > --- a/scripts/make_fit.py
> > +++ b/scripts/make_fit.py
> > @@ -57,10 +57,10 @@ import libfdt
> >  CompTool = collections.namedtuple('CompTool', 'ext,tools')
> >
> >  COMP_TOOLS = {
> > -    'bzip2': CompTool('.bz2', 'bzip2'),
> > +    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
> >      'gzip': CompTool('.gz', 'pigz,gzip'),
> >      'lz4': CompTool('.lz4', 'lz4'),
> > -    'lzma': CompTool('.lzma', 'lzma'),
> > +    'lzma': CompTool('.lzma', 'plzip,lzma'),
> >      'lzo': CompTool('.lzo', 'lzop'),
> >      'zstd': CompTool('.zstd', 'zstd'),
> >  }
> > @@ -220,7 +220,12 @@ def compress_data(inf, compress):
> >              done = False
> >              for tool in comp.tools.split(','):
> >                  try:
> > -                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
> > +                    # Add parallel flags for tools that support them
> > +                    cmd = [tool]
> > +                    if tool in ('zstd', 'xz'):
>
> There's no 'xz' tool in the list though.
>
> And this hunk isn't mentioned or explained in the commit message.

OK I will update it.

>
> > +                        cmd.extend(['-T0'])  # Use all available cores
>
> Zero is already the default for 'xz' in 5.8.1.

My system is fairly up-to-date (Noble) and I have:

xz -V
xz (XZ Utils) 5.4.5
liblzma 5.4.5

Does the new version reject -T0 ? If not, then I think it is best to
provide it in this script.

>
>
> ChenYu
>
> > +                    cmd.append('-c')
> > +                    subprocess.call(cmd, stdin=inf, stdout=outf)
> >                      done = True
> >                      break
> >                  except FileNotFoundError:
> > --
> > 2.43.0
> >

Regards,
Simon

[PATCH v6 8/8] scripts/make_fit: Compress dtbs in parallel

[Simon Glass]

When there are 1500 device tree files it takes quite a while to compress
them. Do it in parallel.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 scripts/make_fit.py | 56 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 50 insertions(+), 6 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 64038f17a51e..0a58b5b378b0 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -43,6 +43,7 @@ as U-Boot, Linuxboot, Tianocore, etc.
 
 import argparse
 import collections
+import multiprocessing
 import os
 import shutil
 import subprocess
@@ -237,15 +238,31 @@ def compress_data(inf, compress):
     return comp_data
 
 
-def output_dtb(fsw, seq, fname, arch, compress):
+def compress_dtb(fname, compress):
+    """Compress a single DTB file
+
+    Args:
+        fname (str): Filename containing the DTB
+        compress (str): Compression algorithm, e.g. 'gzip'
+
+    Returns:
+        tuple: (str: fname, bytes: compressed_data)
+    """
+    with open(fname, 'rb') as inf:
+        compressed = compress_data(inf, compress)
+    return fname, compressed
+
+
+def output_dtb(fsw, seq, fname, arch, compress, data=None):
     """Write out a single devicetree to the FIT
 
     Args:
         fsw (libfdt.FdtSw): Object to use for writing
         seq (int): Sequence number (1 for first)
         fname (str): Filename containing the DTB
-        arch: FIT architecture, e.g. 'arm64'
+        arch (str): FIT architecture, e.g. 'arm64'
         compress (str): Compressed algorithm, e.g. 'gzip'
+        data (bytes): Pre-compressed data (optional)
     """
     with fsw.add_node(f'fdt-{seq}'):
         fsw.property_string('description', os.path.basename(fname))
@@ -253,9 +270,10 @@ def output_dtb(fsw, seq, fname, arch, compress):
         fsw.property_string('arch', arch)
         fsw.property_string('compression', compress)
 
-        with open(fname, 'rb') as inf:
-            compressed = compress_data(inf, compress)
-        fsw.property('data', compressed)
+        if data is None:
+            with open(fname, 'rb') as inf:
+                data = compress_data(inf, compress)
+        fsw.property('data', data)
 
 
 def find_gen_initramfs():
@@ -395,6 +413,11 @@ def _process_dtbs(args, fsw, entries, fdts):
     """
     seq = 0
     size = 0
+
+    # First figure out the unique DTB files that need compression
+    todo = []
+    file_info = []  # List of (fname, model, compat, files) tuples
+
     for fname in args.dtbs:
         # Ignore non-DTB (*.dtb) files
         if os.path.splitext(fname)[1] != '.dtb':
@@ -406,11 +429,32 @@ def _process_dtbs(args, fsw, entries, fdts):
             sys.stderr.write(f'Error processing {fname}:\n')
             raise e
 
+        file_info.append((fname, model, compat, files))
+        for fn in files:
+            if fn not in fdts and fn not in todo:
+                todo.append(fn)
+
+    # Compress all DTBs in parallel
+    cache = {}
+    if todo and args.compress != 'none':
+        if args.verbose:
+            print(f'Compressing {len(todo)} DTBs...')
+
+        with multiprocessing.Pool() as pool:
+            compress_args = [(fn, args.compress) for fn in todo]
+            # unpacks each tuple, calls compress_dtb(fn, compress) in parallel
+            results = pool.starmap(compress_dtb, compress_args)
+
+        cache = dict(results)
+
+    # Now write all DTBs to the FIT using pre-compressed data
+    for fname, model, compat, files in file_info:
         for fn in files:
             if fn not in fdts:
                 seq += 1
                 size += os.path.getsize(fn)
-                output_dtb(fsw, seq, fn, args.arch, args.compress)
+                output_dtb(fsw, seq, fn, args.arch, args.compress,
+                           cache.get(fn))
                 fdts[fn] = seq
 
         files_seq = [fdts[fn] for fn in files]
-- 
2.43.0