Openembedded Devel Discussions
 help / color / mirror / Atom feed
* [RFC_V2] [meta-oe]: Improving ext* sparse image generation: migrating from img2simg to ext2simg_android for ext* filesystem
@ 2026-02-19  8:06 AshishKumar Mishra
  2026-02-22 22:03 ` [oe] " Gyorgy Sarvari
  0 siblings, 1 reply; 7+ messages in thread
From: AshishKumar Mishra @ 2026-02-19  8:06 UTC (permalink / raw)
  To: openembedded-devel


[-- Attachment #1.1: Type: text/plain, Size: 4122 bytes --]

Hello Community Members,

I would like to propose a change to how we handle sparse image generation for ext* file-systems in OpenEmbedded,
specifically moving away from the generic img2simg tool toward the specialised ext2simg_android in the e2fsprogs [contrib/android]

*1) Problem*
Currently, image_types_sparse.bbclass uses img2simg for all file-system types.
img2simg is a  tool that reads the raw image to find zero-filled blocks.
For ext* file-systems, this is inefficient compared to  ext2simg from e2fsprogs [contrib/android]
which understands the file-system structure and uses the block bitmap to determine sparse areas.

*2) History*
Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.

*3) Proposed Changes*
To implement this , below approach is proposed for feedback:

a. meta-oe (android-tools):
Export libsparse, libbase, and liblog headers and libraries to the sysroot. Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected include paths for e2fsprogs contrib tools.

b.  meta-oe (ext2simg_android )
Have created an fresh recipe which focus only on getting ext2simg.c compiled with andorid sparse.h
(as per suggestion on first RFC
openembedded-core@lists.openembedded.org | [RFC] Improving ext* sparse image generation: migrating from img2simg to ext4sparse for ext* filesystem ( https://lists.openembedded.org/g/openembedded-core/topic/rfc_improving_ext_sparse/117715350 )
)
This would link against the exported libsparse from android-tools-native.

c. meta-oe (image_types_sparse.bbclass):
Update CONVERSION_CMD:sparse to detect ext* types.
Use  ext2simg from the e2fsprogs [contrib/android] for these types while falling back to img2simg for others (like f2fs).

*4) Points for Discussion / Feasibility*
I am seeking feedback on the following:

*a. Build Integration:*
In my PoC, I've used a manual ${CC} call in do_compile:append.
Would it be preferred to patch the e2fsprogs Makefile to handle this "contrib" tool more natively?

*b. Library Handling:*
To ensure the native binary finds libsparse at image creation time, I am currently using LD_LIBRARY_PATH in the class.
I suspect a better approach would be ensuring proper RPATH during the e2fsprogs build.
If members can help for any better way of doing this

*c. I have a working proof of concept using RPI4 build*
Attached along are the patch 001-meta-oe-image_types_sparse-use-ext2simg_android-for-.patch

*5) Logical layout of current POC*
This commit bridges the gap between these two recipes:
1. android-tools:
- Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
- Modified do_install to export libsparse, libbase, and liblog headers and libraries to the sysroot, enabling external linking.

2. image_types_sparse.bbclass:
- Updated CONVERSION_CMD to branch by filesystem type, calling ext2simg_android specifically for ext* images.
- Exported LD_LIBRARY_PATH to the native sysroot to ensure the ext2simg_android binary can locate its shared dependencies at runtime.

3. Layer Configuration:
- Updated layer.conf to support dynamic-layers for SELinux and synchronized BBMASK entries to prioritize the updated tools.

*6) Benefit :*
This reduces the time and cost involved while flashing Android RFS / Fastboot etc
Even in CI/CD , which all adds to ease of usage and have monetary benefit involved with continous CI/CD

Looking forward to thoughts / feedback from community members

Please do let me know if any specific platform or group needs to be addressed to have such discussion
If there is any specific meeting involved where i can join and explain, please do let me know
I can connect to team members and works as per suggestion from community to fine tune / rework if any specific changes community suggest

Thanks ,
Ashish Kumar Mishra

[-- Attachment #1.2: Type: text/html, Size: 66338 bytes --]

[-- Attachment #2: 0001-meta-oe-image_types_sparse-use-ext2simg_android-for-.patch --]
[-- Type: application/octet-stream, Size: 10573 bytes --]

From 28bb0409b21b64d6dd439a38d41436ea22a694d5 Mon Sep 17 00:00:00 2001
From: AshishKumar Mishra <ashishkumar.mishra@bmwtechworks.in>
Date: Thu, 19 Feb 2026 13:14:22 +0530
Subject: [PATCH] meta-oe: image_types_sparse: use ext2simg_android for ext*
 image conversion

Update the sparse image conversion process for ext* filesystems to use
ext2simg_android (ext2simg) instead of the generic img2simg tool.

Unlike img2simg, which treats all zero-blocks identically, ext2simg_android
leverages e2fsprogs knowledge of the filesystem.
This allows it to mark zero-blocks outside of the ext-filesystem as
"don't care" (skip chunks) rather than "fill-with-zeros" (raw chunks)
resulting in more optimized and smaller sparse images.

Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.

This commit bridges the gap between these two recipes:
1. android-tools:
   - Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
   - Modified do_install to export libsparse, libbase, and liblog
     headers and libraries to the sysroot, enabling external linking.

2. image_types_sparse.bbclass:
   - Updated CONVERSION_CMD to branch by filesystem type, calling
     ext2simg_android specifically for ext* images.
   - Exported LD_LIBRARY_PATH to the native sysroot to ensure the
     ext2simg_android binary can locate its shared dependencies at runtime.

3. Layer Configuration:
   - Updated layer.conf to support dynamic-layers for SELinux and
     synchronized BBMASK entries to prioritize the updated tools.

This is ver2 implementaion based on suugestion on RFC
https://lists.openembedded.org/g/openembedded-core/topic/rfc_improving_ext_sparse/117715350

Upstream-Status: Pending

Signed-off-by: AshishKumar Mishra <ashishkumar.mishra@bmwtechworks.in>
---
 meta-oe/classes/image_types_sparse.bbclass    | 28 ++++---
 meta-oe/conf/layer.conf                       | 13 ++-
 .../android-tools/android-tools_29.0.6.r14.bb | 51 ++++++++++++
 .../e2fsprogs-ext4sparse_1.0.bb               | 80 +++++++++++++++++++
 4 files changed, 162 insertions(+), 10 deletions(-)
 create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse_1.0.bb

diff --git a/meta-oe/classes/image_types_sparse.bbclass b/meta-oe/classes/image_types_sparse.bbclass
index 5416c2a019..30c802b5d1 100644
--- a/meta-oe/classes/image_types_sparse.bbclass
+++ b/meta-oe/classes/image_types_sparse.bbclass
@@ -12,12 +12,22 @@ CONVERSIONTYPES += "sparse"
 DELETE_RAWIMAGE_AFTER_SPARSE_CMD ??= "0"
 
 CONVERSION_CMD:sparse = " \
-    truncate --no-create --size=%${SPARSE_BLOCK_SIZE} "${IMAGE_NAME}.${type}"; \
-    img2simg -s "${IMAGE_NAME}.${type}" "${IMAGE_NAME}.${type}.sparse" ${SPARSE_BLOCK_SIZE}; \
-    if [ "${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}" = "1" ]; then \
-        rm -f ${IMAGE_NAME}.${type};\
-        bbwarn "Raw file ${IMAGE_NAME}.${type} removed" ;\
-    fi;\
- "
-
-CONVERSION_DEPENDS_sparse = "android-tools-native"
+    truncate --no-create --size=%${SPARSE_BLOCK_SIZE} '${IMAGE_NAME}.${type}'; \
+    case '${type}' in \
+        ext*) \
+            export LD_LIBRARY_PATH='${STAGING_LIBDIR_NATIVE}:${LD_LIBRARY_PATH}'; \
+            \
+            bbwarn 'Running e2fsprogs-derived ext2simg_android..' ; \
+            ext2simg_android '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.simg' || bberror 'ext2simg_android failed' \
+            ;; \
+        *) \
+            bbwarn 'Generating sparse image for non-ext filesystem...'; \
+            img2simg -s '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.sparse' ${SPARSE_BLOCK_SIZE}; \
+            ;; \
+    esac; \
+    if [ '${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}' = '1' ]; then \
+        rm -f '${IMAGE_NAME}.${type}'; \
+    fi; \
+"
+CONVERSION_DEPENDS:sparse = "android-tools-native e2fsprogs-ext4sparse-native"
+do_image_ext4[depends] += "e2fsprogs-ext4sparse-native:do_populate_sysroot"
diff --git a/meta-oe/conf/layer.conf b/meta-oe/conf/layer.conf
index 186ff9a488..9ce2cdb66c 100644
--- a/meta-oe/conf/layer.conf
+++ b/meta-oe/conf/layer.conf
@@ -12,7 +12,16 @@
 BBPATH .= ":${LAYERDIR}"
 
 # We have a recipes directory, add to BBFILES
-BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
+# Adding dynamic layers recipes
+BBFILES += "\
+    ${LAYERDIR}/recipes-*/*/*.bb \
+    ${LAYERDIR}/recipes-*/*/*.bbappend \
+    ${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
+    ${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
+"
+
+# Manually removing default android-tools recipes
+BBMASK += "meta-openembedded/meta-oe/recipes-devtools/android-tools/android-tools_5.1.1.r37.bb"
 
 BBFILE_COLLECTIONS += "openembedded-layer"
 BBFILE_PATTERN_openembedded-layer := "^${LAYERDIR}/"
@@ -40,6 +49,8 @@ BBFILES_DYNAMIC += " \
     perl-layer:${LAYERDIR}/dynamic-layers/perl-layer/recipes-*/*/*.bbappend \
     selinux:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
     selinux:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
+    selinux-layer:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bb \
+    selinux-layer:${LAYERDIR}/dynamic-layers/selinux/recipes-*/*/*.bbappend \
 "
 
 # This should only be incremented on significant changes that will
diff --git a/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb b/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
index 74928ed171..0a8518a1dd 100644
--- a/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
+++ b/meta-oe/dynamic-layers/selinux/recipes-devtool/android-tools/android-tools_29.0.6.r14.bb
@@ -190,3 +190,54 @@ FILES:${PN}-fstools = "\
 FILES:${PN} += "${libdir}/android ${libdir}/android/*"
 
 BBCLASSEXTEND = "native"
+
+android_tools_enable_devmode() {
+    touch ${IMAGE_ROOTFS}/etc/usb-debugging-enabled
+}
+
+ROOTFS_POSTPROCESS_COMMAND_${PN}-adbd += "${@bb.utils.contains("USB_DEBUGGING_ENABLED", "1", "android_tools_enable_devmode;", "", d)}"
+
+# e2fsprogs expecting headers in sparse/ subdirectory
+do_install:append() {
+    # to resolve the "Multiple shlib providers" conflict.
+    rm -f ${D}${libdir}/android/libsparse.so*
+    rm -f ${D}${libdir}/android/libbase.so*
+    rm -f ${D}${libdir}/android/liblog.so*
+
+    if [ -d "${S}/system/core/libsparse/include/sparse" ]; then
+        install -d ${D}${includedir}/sparse
+        cp -r ${S}/system/core/libsparse/include/sparse/* ${D}${includedir}/sparse/
+    else
+        bberror "Sparse headers not found in ${S}/system/core/libsparse/include/sparse"
+    fi
+
+    install -d ${D}${libdir}
+    for lib in libsparse libbase liblog; do
+        if [ -f "${S}/debian/out/system/core/${lib}.so" ]; then
+            bbwarn "Installing ${lib} to sysroot"
+            install -m 0755 ${S}/debian/out/system/core/${lib}.so ${D}${libdir}/${lib}.so.0
+            ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+        fi
+    done
+
+    install -d ${D}${libdir}/android
+    for lib in libsparse libbase liblog; do
+        if [ -f "${D}${libdir}/${lib}.so.0" ]; then
+            ln -sf ../${lib}.so.0 ${D}${libdir}/android/${lib}.so.0
+            ln -sf ../${lib}.so ${D}${libdir}/android/${lib}.so
+        fi
+    done
+}
+
+FILES:${PN}-dev += " \
+    ${includedir}/sparse \
+    ${libdir}/lib*.so \
+    ${libdir}/android/lib*.so \
+"
+
+FILES:${PN} += " \
+    ${libdir}/lib*.so.* \
+    ${libdir}/android/lib*.so.* \
+"
+
+SYSROOT_DIRS:append:class-native = " ${includedir} ${libdir}"
diff --git a/meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse_1.0.bb b/meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse_1.0.bb
new file mode 100644
index 0000000000..afe74ea279
--- /dev/null
+++ b/meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse_1.0.bb
@@ -0,0 +1,80 @@
+require e2fsprogs-ext4sparse.inc
+
+inherit native
+
+SRC_URI += "file://remove.ldconfig.call.patch \
+           file://mkdir_p.patch \
+           file://0001-misc-create_inode.c-Fix-for-file-larger-than-2GB.patch \
+           "
+SRC_URI:append:class-native = " \
+           file://e2fsprogs-fix-missing-check-for-permission-denied.patch \
+           file://quiet-debugfs.patch \
+           "
+
+SRCREV = "da631e117dcf8797bfda0f48bdaa05ac0fbcf7af"
+
+EXTRA_OECONF = " \
+    --libdir=${base_libdir} \
+    --sbindir=${base_sbindir} \
+    --enable-elf-shlibs \
+    --enable-largefile \
+    --disable-libuuid \
+    --disable-libblkid \
+    --disable-uuidd \
+    --disable-fuse2fs \
+    --with-crond-dir=no \
+"
+
+DEPENDS += "util-linux-native"
+
+# Fix e2fsprogs-native do_compile error with old kernel such as 5.10
+CACHED_CONFIGUREVARS:class-native = "ac_cv_header_linux_fsverity_h=no"
+
+do_install () {
+    bbwarn "=============== DEBUG ======================="
+}
+
+
+RDEPENDS:${PN} = ""
+RRECOMMENDS:${PN} = ""
+ALTERNATIVE:${PN} = ""
+
+PACKAGES = "${PN}"
+FILES:${PN} = "${bindir}/ext2simg_android"
+FILES:${PN}:class-native += "${bindir}/ext2simg_android"
+
+DEPENDS:append:class-native = " android-tools-native "
+
+do_compile:append:class-native() {
+    bbwarn "=============== COMPILE_APPEND: DEBUG START ======================="
+    bbwarn "Compiling ext2simg.c with native toolchain"
+
+    #include <ext2fs/ext2fs.h>
+    #include <et/com_err.h>
+    oe_runmake -C ${B}/lib/et
+    oe_runmake -C ${B}/lib/ext2fs
+
+    # Source directory for ext2simg.c in the e2fsprogs tree
+    SRC_EXT2SIMG="${S}/contrib/android"
+
+    INCLUDES="-I${S}/lib -I${B}/lib -I${SRC_EXT2SIMG}/lib"
+
+    # STAGING_LIBDIR_NATIVE: where android-tools-native installed libsparse
+    LIBS="-L${B}/lib -L${STAGING_LIBDIR_NATIVE} -lsparse -lext2fs -lz -lcom_err"
+
+    ${CC} ${CFLAGS} ${LDFLAGS} -o "${B}/ext2simg_android" "${SRC_EXT2SIMG}/ext2simg.c" ${INCLUDES} ${LIBS}
+    bbwarn "ext2simg_android compilation finished"
+    bbwarn "=============== COMPILE_APPEND: DEBUG STOP ======================="
+}
+
+do_install:append:class-native() {
+    bbwarn "=============== INSTALL_APPEND: DEBUG START ======================="
+
+    bbwarn "We have D as ${D}"
+    bbwarn "We have bindir as ${bindir}"
+    bbwarn "We have base_bindir as ${base_bindir}"
+    install -d ${D}${bindir}
+    install -m 0755 ${B}/ext2simg_android ${D}${bindir}/ext2simg_android
+
+    bbwarn "=============== INSTALL_APPEND: DEBUG STOP ======================="
+}
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-07-21 16:29 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-02-19  8:06 [RFC_V2] [meta-oe]: Improving ext* sparse image generation: migrating from img2simg to ext2simg_android for ext* filesystem AshishKumar Mishra
2026-02-22 22:03 ` [oe] " Gyorgy Sarvari
2026-02-23  5:43   ` AshishKumar Mishra
2026-03-02  4:45     ` AshishKumar Mishra
2026-03-03  6:20       ` [oe] " Gyorgy Sarvari
2026-03-03  6:24         ` AshishKumar Mishra
2026-07-21 16:29         ` AshishKumar Mishra

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox