[RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem

[RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem

[AshishKumar Mishra]

[-- Attachment #1.1: Type: text/plain, Size: 4170 bytes --]

Hello Community Members  ,

I would like to propose a change to how we handle sparse image generation for ext* file-systems in OpenEmbedded,
specifically moving away from the generic img2simg tool toward the specialised ext2simg in the e2fsprogs [contrib/android]

*1) Problem*
Currently, image_types_sparse.bbclass uses img2simg for all file-system types.
img2simg is a  tool that reads the raw image to find zero-filled blocks.
For ext* file-systems, this is inefficient compared to  ext2simg from e2fsprogs [contrib/android]
which understands the file-system structure and uses the block bitmap to determine sparse areas.

*2) History*
Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.

*3) Proposed Changes*
To implement this, I am looking at a cross-layer approach:

a. meta-oe (android-tools):
Export libsparse, libbase, and liblog headers and libraries to the sysroot. Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected include paths for e2fsprogs contrib tools.

b. oe-core (e2fsprogs):
Add a conditional compilation step in e2fsprogs-native to build the ext4sparse utility located in contrib/android/.
This would link against the exported libsparse from android-tools-native.

c. meta-oe (image_types_sparse.bbclass):
Update CONVERSION_CMD:sparse to detect ext* types.
Use  ext2simg from the e2fsprogs [contrib/android] for these types while falling back to img2simg for others (like f2fs).

*4) Points for Discussion / Feasibility*
I am seeking feedback on the following:

*a. Layer Dependency:*
This introduces a tighter coupling between oe-core and meta-oe.
Is it acceptable for e2fsprogs-native (core) to optionally depend on android-tools-native (meta-oe),
or i can create an bbapend file in meta-oe called e2fsprogs_%.bbappend.

*b. Build Integration:*
In my PoC, I've used a manual ${CC} call in do_compile:append.
Would it be preferred to patch the e2fsprogs Makefile to handle this "contrib" tool more natively?

*c. Library Handling:*
To ensure the native binary finds libsparse at image creation time, I am currently using LD_LIBRARY_PATH in the class.
I suspect a better approach would be ensuring proper RPATH during the e2fsprogs build.
If members can help for any better way of doing this

*d. I have a working proof of concept using RPI4 build*
Attached alon are the patches
0001-e2fsprogs-creates-ext4sparse-using-android-tools-lib.patch
0001-meta-oe-image_types_sparse-use-ext4sparse-for-ext-im.patch

*5) Logical layout of current POC*
This commit bridges the gap between these two recipes:
1. android-tools:
- Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
- Modified do_install to export libsparse, libbase, and liblog
headers and libraries to the sysroot, enabling external linking.

2. image_types_sparse.bbclass:
- Updated CONVERSION_CMD to branch by filesystem type, calling
ext4sparse specifically for ext* images.
- Exported LD_LIBRARY_PATH to the native sysroot to ensure the
ext4sparse binary can locate its shared dependencies at runtime.

3. Layer Configuration:
- Updated layer.conf to support dynamic-layers for SELinux and
synchronized BBMASK entries to prioritize the updated tools.

*6) Benefit :*
This reduces the time and cost involved while flashing Android RFS / Fastboot etc
Even in CI/CD , which all adds to ease of usage and have monetary benefit involved with continous CI/CD

Looking forward to thoughts / feedback from community members

Please do let me know if any specific platform or group needs to be addressed to have such discussion
If there is any specific meeting involved where i can join and explain , please do let me know
I can connect to team members and works as per suggestion from community to fine tune / rework if any specific changes community suggest

Thanks ,
Ashish Kumar Mishra

[-- Attachment #1.2: Type: text/html, Size: 9585 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-meta-oe-image_types_sparse-use-ext4sparse-for-ext-im.patch --]
[-- Type: text/x-patch; name="0001-meta-oe-image_types_sparse-use-ext4sparse-for-ext-im.patch", Size: 7255 bytes --]

From 533648b1208131d4adab15c48e5c7eb5c0772d73 Mon Sep 17 00:00:00 2001
From: AshishKumar Mishra <ashishkumar.mishra@bmwtechworks.in>
Date: Mon, 9 Feb 2026 11:11:08 +0530
Subject: [PATCH] meta-oe: image_types_sparse: use ext4sparse for ext* image
 conversion

Update the sparse image conversion process for ext* filesystems to use
ext4sparse (ext2simg) instead of the generic img2simg tool.

Unlike img2simg, which treats all zero-blocks identically, ext4sparse leverages
e2fsprogs knowledge of the filesystem.
This allows it to mark zero-blocks outside of the ext-filesystem as
"don't care" (skip chunks) rather than "fill-with-zeros" (raw chunks)
resulting in more optimized and smaller sparse images.

Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.

This commit bridges the gap between these two recipes:
1. android-tools:
   - Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
   - Modified do_install to export libsparse, libbase, and liblog
     headers and libraries to the sysroot, enabling external linking.

2. image_types_sparse.bbclass:
   - Updated CONVERSION_CMD to branch by filesystem type, calling
     ext4sparse specifically for ext* images.
   - Exported LD_LIBRARY_PATH to the native sysroot to ensure the
     ext4sparse binary can locate its shared dependencies at runtime.

3. Layer Configuration:
   - Updated layer.conf to support dynamic-layers for SELinux and
     synchronized BBMASK entries to prioritize the updated tools.

Upstream-Status: Pending

Signed-off-by: AshishKumar Mishra <ashishkumar.mishra@bmwtechworks.in>
---
 meta-oe/classes/image_types_sparse.bbclass    | 26 +++++++---
 meta-oe/conf/layer.conf                       | 13 ++++-
 .../android-tools/android-tools_29.0.6.r14.bb | 51 +++++++++++++++++++
 3 files changed, 81 insertions(+), 9 deletions(-)

diff --git a/meta-oe/classes/image_types_sparse.bbclass b/meta-oe/classes/image_types_sparse.bbclass
index 5416c2a019..03e0ecd6c3 100644
--- a/meta-oe/classes/image_types_sparse.bbclass
+++ b/meta-oe/classes/image_types_sparse.bbclass
@@ -12,12 +12,22 @@ CONVERSIONTYPES += "sparse"
 DELETE_RAWIMAGE_AFTER_SPARSE_CMD ??= "0"
 
 CONVERSION_CMD:sparse = " \
-    truncate --no-create --size=%${SPARSE_BLOCK_SIZE} "${IMAGE_NAME}.${type}"; \
-    img2simg -s "${IMAGE_NAME}.${type}" "${IMAGE_NAME}.${type}.sparse" ${SPARSE_BLOCK_SIZE}; \
-    if [ "${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}" = "1" ]; then \
-        rm -f ${IMAGE_NAME}.${type};\
-        bbwarn "Raw file ${IMAGE_NAME}.${type} removed" ;\
-    fi;\
- "
+    truncate --no-create --size=%${SPARSE_BLOCK_SIZE} '${IMAGE_NAME}.${type}'; \
+    case '${type}' in \
+        ext*) \
+            export LD_LIBRARY_PATH='${STAGING_LIBDIR_NATIVE}:${LD_LIBRARY_PATH}'; \
+            \
+            bbwarn 'Running e2fsprogs-derived ext4sparse...' ; \
+            ext4sparse '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.simg' || bberror 'ext4sparse failed' \
+            ;; \
+        *) \
+            bbwarn 'Generating sparse image for non-ext filesystem...'; \
+            img2simg -s '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.sparse' ${SPARSE_BLOCK_SIZE}; \
+            ;; \
+    esac; \
+    if [ '${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}' = '1' ]; then \
+        rm -f '${IMAGE_NAME}.${type}'; \
+    fi; \
+"
 
-CONVERSION_DEPENDS_sparse = "android-tools-native"
+CONVERSION_DEPENDS:sparse = "android-tools-native e2fsprogs-native"
diff --git a/meta-oe/conf/layer.conf b/meta-oe/conf/layer.conf
index 186ff9a488..9ce2cdb66c 100644
--- a/meta-oe/conf/layer.conf
+++ b/meta-oe/conf/layer.conf
@@ -12,7 +12,16 @@
 BBPATH .= ":${LAYERDIR}"
 
 # We have a recipes directory, add to BBFILES
-BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
+# Adding dynamic layers recipes
+BBFILES += "\
+    ${LAYERDIR}/recipes-*/*/*.bb \
+    ${LAYERDIR}/recipes-*/*/*.bbappend \
+    ${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
+    ${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
+"
+
+# Manually removing default android-tools recipes
+BBMASK += "meta-openembedded/meta-oe/recipes-devtools/android-tools/android-tools_5.1.1.r37.bb"
 
 BBFILE_COLLECTIONS += "openembedded-layer"
 BBFILE_PATTERN_openembedded-layer := "^${LAYERDIR}/"
@@ -40,6 +49,8 @@ BBFILES_DYNAMIC += " \
     perl-layer:${LAYERDIR}/dynamic-layers/perl-layer/recipes-*/*/*.bbappend \
     selinux:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
     selinux:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
+    selinux-layer:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
+    selinux-layer:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
 "
 
 # This should only be incremented on significant changes that will
diff --git a/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb b/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
index 74928ed171..0a8518a1dd 100644
--- a/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
+++ b/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
@@ -190,3 +190,54 @@ FILES:${PN}-fstools = "\
 FILES:${PN} += "${libdir}/android ${libdir}/android/*"
 
 BBCLASSEXTEND = "native"
+
+android_tools_enable_devmode() {
+    touch ${IMAGE_ROOTFS}/etc/usb-debugging-enabled
+}
+
+ROOTFS_POSTPROCESS_COMMAND_${PN}-adbd += "${@bb.utils.contains("USB_DEBUGGING_ENABLED", "1", "android_tools_enable_devmode;", "", d)}"
+
+# e2fsprogs expecting headers in sparse/ subdirectory
+do_install:append() {
+    # to resolve the "Multiple shlib providers" conflict.
+    rm -f ${D}${libdir}/android/libsparse.so*
+    rm -f ${D}${libdir}/android/libbase.so*
+    rm -f ${D}${libdir}/android/liblog.so*
+
+    if [ -d "${S}/system/core/libsparse/include/sparse" ]; then
+        install -d ${D}${includedir}/sparse
+        cp -r ${S}/system/core/libsparse/include/sparse/* ${D}${includedir}/sparse/
+    else
+        bberror "Sparse headers not found in ${S}/system/core/libsparse/include/sparse"
+    fi
+
+    install -d ${D}${libdir}
+    for lib in libsparse libbase liblog; do
+        if [ -f "${S}/debian/out/system/core/${lib}.so" ]; then
+            bbwarn "Installing ${lib} to sysroot"
+            install -m 0755 ${S}/debian/out/system/core/${lib}.so ${D}${libdir}/${lib}.so.0
+            ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+        fi
+    done
+
+    install -d ${D}${libdir}/android
+    for lib in libsparse libbase liblog; do
+        if [ -f "${D}${libdir}/${lib}.so.0" ]; then
+            ln -sf ../${lib}.so.0 ${D}${libdir}/android/${lib}.so.0
+            ln -sf ../${lib}.so ${D}${libdir}/android/${lib}.so
+        fi
+    done
+}
+
+FILES:${PN}-dev += " \
+    ${includedir}/sparse \
+    ${libdir}/lib*.so \
+    ${libdir}/android/lib*.so \
+"
+
+FILES:${PN} += " \
+    ${libdir}/lib*.so.* \
+    ${libdir}/android/lib*.so.* \
+"
+
+SYSROOT_DIRS:append:class-native = " ${includedir} ${libdir}"
-- 
2.34.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-e2fsprogs-creates-ext4sparse-using-android-tools-lib.patch --]
[-- Type: text/x-patch; name="0001-e2fsprogs-creates-ext4sparse-using-android-tools-lib.patch", Size: 1965 bytes --]

From d43f0747a57a976578f0bfa259fae11063b6b12d Mon Sep 17 00:00:00 2001
From: AshishKumar Mishra <ashishkumar.mishra@bmwtechworks.in>
Date: Mon, 9 Feb 2026 11:14:33 +0530
Subject: [PATCH] e2fsprogs: creates ext4sparse using android-tools library

Add a custom do_compile step to build the ext4sparse utility (ext2simg)
when building for the native class using android-tools-native.
Includes debug logging for build verification

Upstream-Status: Pending

Signed-off-by: AshishKumar Mishra <ashishkumar.mishra@bmwtechworks.in>
---
 .../e2fsprogs/e2fsprogs_1.47.3.bb             | 23 +++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb
index 40658399d9..090dbd1282 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.47.3.bb
@@ -151,3 +151,26 @@ do_install_ptest() {
         install -d ${D}${PTEST_PATH}/data
         install -m 0644 ${B}/tests/test_data.tmp ${D}${PTEST_PATH}/data/
 }
+
+DEPENDS:append:class-native = " android-tools-native "
+
+do_compile:append:class-native() {
+    bbwarn "=============== DEBUG ======================="
+    bbwarn "Compiling ext2simg.c with native toolchain"
+    # Source directory for ext2simg.c in the e2fsprogs tree
+    SRC_EXT2SIMG="${S}/contrib/android"
+
+    INCLUDES="-I${S}/lib -I${B}/lib -I${SRC_EXT2SIMG}/lib"
+
+    # STAGING_LIBDIR_NATIVE: where android-tools-native installed libsparse
+    LIBS="-L${B}/lib -L${STAGING_LIBDIR_NATIVE} -lsparse -lext2fs -lz -lcom_err"
+
+    ${CC} ${CFLAGS} ${LDFLAGS} -o "${B}/ext4sparse" "${SRC_EXT2SIMG}/ext2simg.c" ${INCLUDES} ${LIBS}
+    bbwarn "ext4sparse compilation finished"
+    bbwarn "=============== DEBUG ======================="
+}
+
+do_install:append:class-native() {
+    install -d ${D}${bindir}
+    install -m 0755 ${B}/ext4sparse ${D}${bindir}/ext4sparse
+}
-- 
2.34.1

Re: [RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem

[AshishKumar Mishra]

[-- Attachment #1: Type: text/plain, Size: 120 bytes --]

Hi Team,
Any feedback on this RFC or any other emailing list where i should route this discussion

Thanks ,
Ashish

[-- Attachment #2: Type: text/html, Size: 206 bytes --]

Re: [OE-core] [RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 3015 bytes --]

Hi,

On Sun, 2026-02-08 at 22:12 -0800, AshishKumar Mishra via
lists.openembedded.org wrote:
> Hello Community Members  , 
>  
> I would like to propose a change to how we handle sparse image
> generation for ext* file-systems in OpenEmbedded, 
> specifically moving away from the generic img2simg tool toward the
> specialised ext2simg in the e2fsprogs [contrib/android]

Firstly, you should probably mention that image_types_sparse is in
meta-oe, not oe-core. You should therefore probably mention this on the
list where meta-oe is maintained too.

> 1) Problem
> Currently, image_types_sparse.bbclass uses img2simg for all file-
> system types. 
> img2simg is a  tool that reads the raw image to find zero-filled
> blocks. 
> For ext* file-systems, this is inefficient compared to  ext2simg from
> e2fsprogs [contrib/android]
> which understands the file-system structure and uses the block bitmap
> to determine sparse areas.

That seems reasonable.

> 2) History
> Previously, 'ext2simg' was bundled within android-tools (v5.1 and
> older).
> The utility has since migrated to the e2fsprogs project
> (contrib/android).
> However, upstream e2fsprogs does not provide a build system for this
> tool,
> and it requires libsparse from android-tools to compile.
>  
>  
> 3) Proposed Changes
> To implement this, I am looking at a cross-layer approach:
>  
> a. meta-oe (android-tools):
> Export libsparse, libbase, and liblog headers and libraries to the
> sysroot. Currently, these are often internal to the build.
> Reorganise header installation to ${includedir}/sparse to match the
> expected include paths for e2fsprogs contrib tools.
>  
> b. oe-core (e2fsprogs):
> Add a conditional compilation step in e2fsprogs-native to build the
> ext4sparse utility located in contrib/android/.
> This would link against the exported libsparse from android-tools-
> native.
>  
> c. meta-oe (image_types_sparse.bbclass):
> Update CONVERSION_CMD:sparse to detect ext* types.
> Use  ext2simg from the e2fsprogs [contrib/android] for these types
> while falling back to img2simg for others (like f2fs).
>  
>  
> 4) Points for Discussion / Feasibility
>      I am seeking feedback on the following:
>  
> a. Layer Dependency: 
> This introduces a tighter coupling between oe-core and meta-oe. 
> Is it acceptable for e2fsprogs-native (core) to optionally depend on
> android-tools-native (meta-oe), 
> or i can create an bbapend file in meta-oe called
> e2fsprogs_%.bbappend. 

oe-core cannot depend on meta-oe.

The bbappend is one approach but it means e2fsprogs would rebuild
whenever a layer is added/removed and it would likely fail the yocto-
check-layer tests. It would also mean anything depending on e2fsprogs
would also rebuild and sstate would not be reused. None of this is
particularly good.

Instead, how about a separate recipe, e2fsprogs-ext4sparse which just
builds this utility?

Cheers,

Richard

[-- Attachment #2: Type: text/html, Size: 5996 bytes --]

Re: [OE-core] [RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem

[AshishKumar Mishra]

[-- Attachment #1: Type: text/plain, Size: 3691 bytes --]

Thanks Richard
Will rework on this approach
W.r.t e2fsprogs-ext4sparse , would approach to be singularly in meta-oe

Please feel free to share comments / feedback .
I will try to incorporate them

- Ashish

________________________________
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: 10 February 2026 14:09
To: AshishKumar Mishra <AshishKumar.Mishra@bmwtechworks.in>; openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem

You don't often get email from richard.purdie@linuxfoundation.org. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Caution: "External email, be cautious especially with link(s), attachment(s) or QR code(s)".
Hi,

On Sun, 2026-02-08 at 22:12 -0800, AshishKumar Mishra via lists.openembedded.org wrote:
Hello Community Members  ,

I would like to propose a change to how we handle sparse image generation for ext* file-systems in OpenEmbedded,
specifically moving away from the generic img2simg tool toward the specialised ext2simg in the e2fsprogs [contrib/android]

Firstly, you should probably mention that image_types_sparse is in meta-oe, not oe-core. You should therefore probably mention this on the list where meta-oe is maintained too.

1) Problem
Currently, image_types_sparse.bbclass uses img2simg for all file-system types.
img2simg is a  tool that reads the raw image to find zero-filled blocks.
For ext* file-systems, this is inefficient compared to  ext2simg from e2fsprogs [contrib/android]
which understands the file-system structure and uses the block bitmap to determine sparse areas.

That seems reasonable.

2) History
Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.


3) Proposed Changes
To implement this, I am looking at a cross-layer approach:

a. meta-oe (android-tools):
Export libsparse, libbase, and liblog headers and libraries to the sysroot. Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected include paths for e2fsprogs contrib tools.

b. oe-core (e2fsprogs):
Add a conditional compilation step in e2fsprogs-native to build the ext4sparse utility located in contrib/android/.
This would link against the exported libsparse from android-tools-native.

c. meta-oe (image_types_sparse.bbclass):
Update CONVERSION_CMD:sparse to detect ext* types.
Use  ext2simg from the e2fsprogs [contrib/android] for these types while falling back to img2simg for others (like f2fs).


4) Points for Discussion / Feasibility
     I am seeking feedback on the following:

a. Layer Dependency:
This introduces a tighter coupling between oe-core and meta-oe.
Is it acceptable for e2fsprogs-native (core) to optionally depend on android-tools-native (meta-oe),
or i can create an bbapend file in meta-oe called e2fsprogs_%.bbappend.

oe-core cannot depend on meta-oe.

The bbappend is one approach but it means e2fsprogs would rebuild whenever a layer is added/removed and it would likely fail the yocto-check-layer tests. It would also mean anything depending on e2fsprogs would also rebuild and sstate would not be reused. None of this is particularly good.

Instead, how about a separate recipe, e2fsprogs-ext4sparse which just builds this utility?

Cheers,

Richard


[-- Attachment #2: Type: text/html, Size: 17126 bytes --]