[meta-openembedded][PATCH v3 0/3] image_types_sparse: use ext2simg_android for ext* sparse images

[meta-openembedded][PATCH v3 0/3] image_types_sparse: use ext2simg_android for ext* sparse images

[AshishKumar Mishra]

This series switches ext* sparse image generation from the generic
img2simg to ext2simg_android from e2fsprogs contrib/android. 
Unlike img2simg,which scans the raw image for zero-filled blocks, 
ext2simg_android understands the ext* filesystem structure and uses the block 
bitmap to determine sparse areas, which is faster and produces smaller 
sparse images. 
This reduces flashing time for Android RFS/fastboot images and cuts cost in
CI/CD pipelines. img2simg is retained for non-ext types such as f2fs.

  1. e2fsprogs-ext4sparse-native: a standalone native recipe that
     compiles ext2simg.c against android-tools' libsparse.
  2. android-tools: export libsparse/libbase/liblog headers and libs
     to the sysroot (under ${includedir}/sparse) so the recipe above
     can link against them.
  3. image_types_sparse.bbclass: branch CONVERSION_CMD:sparse by
     filesystem type, calling ext2simg_android for ext* and falling
     back to img2simg otherwise.
	
Changes in v3:
  - Android tools has been moved away from selinux dynamic-layers
  - Reworked into a standalone e2fsprogs-ext4sparse recipe in meta-oe
    as per feedback on the RFC
  - Included the previously-missing e2fsprogs-ext4sparse recipe file
    that was absent from the v2 (RFC_V2) attachment.

Changes in v2 (RFC_V2):
  - Split the single cross-layer PoC into a dedicated ext2simg_android
    recipe rather than patching e2fsprogs-native directly.

RFC (v1):
  - Initial cross-layer proof of concept using a manual ${CC} call in
    do_compile:append and LD_LIBRARY_PATH for runtime linking.


AshishKumar Mishra (3):
  e2fsprogs-ext4sparse-native: compiles ext2simg.c using  android-tools-native
  android-tools: export libsparse/libbase/liblog headers and libs to sysroot
  image_types_sparse: switch ext* conversion to ext2simg_android

 meta-oe/classes/image_types_sparse.bbclass    | 16 ++++--
 .../android-tools/android-tools_%.bbappend    | 54 +++++++++++++++++++
 .../e2fsprogs-ext4sparse-native_1.47.4.bb     | 34 ++++++++++++
 3 files changed, 101 insertions(+), 3 deletions(-)
 create mode 100644 meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend
 create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb

-- 
2.43.0

[meta-openembedded][PATCH v3 1/3] e2fsprogs-ext4sparse-native: compiles ext2simg.c using android-tools-native

[AshishKumar Mishra]

Add a custom do_compile step to build the  ext2simg_android utility (ext2simg)
when building for the native class.
This can be used to generate ext* based sparse image

Signed-off-by: AshishKumar Mishra <emailaddress.ashish@gmail.com>
---
 .../e2fsprogs-ext4sparse-native_1.47.4.bb     | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb

diff --git a/meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb b/meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb
new file mode 100644
index 0000000000..04f9aaaf2b
--- /dev/null
+++ b/meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb
@@ -0,0 +1,32 @@
+SUMMARY = "Native ext sparse conversion utility from e2fsprogs"
+DESCRIPTION = "Builds only ext2simg for host-side ext* sparse image conversion"
+HOMEPAGE = "http://e2fsprogs.sourceforge.net/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://contrib/android/ext2simg.c;beginline=1;endline=14;md5=fc8a826484cc82548adeaad2579adfca"
+
+DEPENDS = "e2fsprogs-native android-tools-native"
+
+SRC_URI = "git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git;branch=master;protocol=https"
+SRCREV = "ece89fac4603e400155b7bbf6326284f8511bca9"
+
+inherit native
+
+do_compile() {
+    bbnote "Compiling ext2simg.c with native toolchain"
+    # Source directory for ext2simg.c in the e2fsprogs tree
+    SRC_EXT2SIMG="${S}/contrib/android"
+
+    INCLUDES="-I${S}/lib -I${B}/lib -I${SRC_EXT2SIMG}/lib"
+
+    # STAGING_LIBDIR_NATIVE: where android-tools-native installed libsparse
+    LIBS="-L${B}/lib -L${RECIPE_SYSROOT_NATIVE}${libdir} -L${RECIPE_SYSROOT_NATIVE}${libdir}/android -lsparse -lext2fs -lz -lcom_err"
+
+    ${CC} ${CFLAGS} ${LDFLAGS} -o "${B}/ext2simg_android" "${SRC_EXT2SIMG}/ext2simg.c" ${INCLUDES} ${LIBS}
+    bbnote "ext2simg_android compilation finished"
+}
+
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${B}/ext2simg_android ${D}${bindir}/ext2simg_android
+}
+
-- 
2.43.0

[meta-openembedded][PATCH v3 2/3] android-tools: export libsparse/libbase/liblog headers and libs to sysroot

[AshishKumar Mishra]

Export libsparse, libbase, and liblog headers and libraries to the sysroot.
Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected
include paths for e2fsprogs contrib tools.

Signed-off-by: AshishKumar Mishra <emailaddress.ashish@gmail.com>
---
 .../android-tools/android-tools_%.bbappend    | 54 +++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend

diff --git a/meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend b/meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend
new file mode 100644
index 0000000000..50c54145b8
--- /dev/null
+++ b/meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend
@@ -0,0 +1,54 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
+
+# e2fsprogs expecting headers in sparse/ subdirectory
+do_install:append:class-native() {
+    # to resolve the "Multiple shlib providers" conflict.
+    rm -f ${D}${libdir}/android/libsparse.so*
+    rm -f ${D}${libdir}/android/libbase.so*
+    rm -f ${D}${libdir}/android/liblog.so*
+
+    if [ -d "${S}/system/core/libsparse/include/sparse" ]; then
+        install -d ${D}${includedir}/sparse
+        cp -r ${S}/system/core/libsparse/include/sparse/* ${D}${includedir}/sparse/
+    else
+        bbfatal "Sparse headers not found in ${S}/system/core/libsparse/include/sparse"
+    fi
+
+    install -d ${D}${libdir}
+    for lib in libsparse libbase liblog; do
+        if [ -f "${S}/debian/out/system/${lib}.so" ]; then
+            bbnote "Installing ${lib} from ${S}/debian/out/system/${lib}.so"
+            install -m 0755 ${S}/debian/out/system/${lib}.so ${D}${libdir}/${lib}.so.0
+            ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+        elif [ -f "${S}/debian/out/system/${lib}.so.0" ]; then
+            bbnote "Installing ${lib} from ${S}/debian/out/system/${lib}.so.0"
+            install -m 0755 ${S}/debian/out/system/${lib}.so.0 ${D}${libdir}/${lib}.so.0
+            ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so
+        else
+            bberror "${lib} not found in ${S}/debian/out/system"
+        fi
+    done
+
+    install -d ${D}${libdir}/android
+    for lib in libsparse libbase liblog; do
+        if [ -f "${D}${libdir}/${lib}.so.0" ]; then
+            ln -sf ../${lib}.so.0 ${D}${libdir}/android/${lib}.so.0
+            ln -sf ../${lib}.so ${D}${libdir}/android/${lib}.so
+        fi
+    done
+}
+
+FILES:${PN}-dev += " \
+    ${includedir}/sparse \
+    ${libdir}/lib*.so \
+    ${libdir}/android/lib*.so \
+"
+
+FILES:${PN} += " \
+    ${libdir}/lib*.so.* \
+    ${libdir}/android/lib*.so.* \
+"
+
+SYSROOT_DIRS:append:class-native = " ${includedir} ${libdir}"
+
+DEPENDS:append:class-native = " googletest-native"
-- 
2.43.0

[meta-openembedded][PATCH v3 3/3] image_types_sparse: switch ext* conversion to ext2simg_android

[AshishKumar Mishra]

We have RFC discussion at
https://lists.openembedded.org/g/openembedded-devel/message/124499
https://lists.openembedded.org/g/openembedded-devel/message/125497

Move ext* sparse image generation from 'img2simg' to ' ext2simg_android'

Signed-off-by: AshishKumar Mishra <emailaddress.ashish@gmail.com>
---
 meta-oe/classes/image_types_sparse.bbclass | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/meta-oe/classes/image_types_sparse.bbclass b/meta-oe/classes/image_types_sparse.bbclass
index 5416c2a019..5c1d92bd18 100644
--- a/meta-oe/classes/image_types_sparse.bbclass
+++ b/meta-oe/classes/image_types_sparse.bbclass
@@ -13,11 +13,21 @@ DELETE_RAWIMAGE_AFTER_SPARSE_CMD ??= "0"
 
 CONVERSION_CMD:sparse = " \
     truncate --no-create --size=%${SPARSE_BLOCK_SIZE} "${IMAGE_NAME}.${type}"; \
-    img2simg -s "${IMAGE_NAME}.${type}" "${IMAGE_NAME}.${type}.sparse" ${SPARSE_BLOCK_SIZE}; \
+    case '${type}' in \
+        ext*) \
+            bbnote 'Running e2fsprogs-derived ext2simg_android..' ; \
+            ext2simg_android '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.simg' || bberror 'ext2simg_android failed' \
+            ;; \
+        *) \
+            bbnote 'Generating sparse image for non-ext filesystem...'; \
+            img2simg -s '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.sparse' ${SPARSE_BLOCK_SIZE}; \
+            ;; \
+    esac; \
     if [ "${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}" = "1" ]; then \
         rm -f ${IMAGE_NAME}.${type};\
-        bbwarn "Raw file ${IMAGE_NAME}.${type} removed" ;\
+        bbnote "Raw file ${IMAGE_NAME}.${type} removed" ;\
     fi;\
  "
 
-CONVERSION_DEPENDS_sparse = "android-tools-native"
+CONVERSION_DEPENDS_sparse = "android-tools-native e2fsprogs-ext4sparse-native"
+do_image_ext4[depends] += "e2fsprogs-ext4sparse-native:do_populate_sysroot"
-- 
2.43.0

Re: [meta-openembedded][PATCH v3 0/3] image_types_sparse: use ext2simg_android for ext* sparse images

[AshishKumar Mishra]

[-- Attachment #1: Type: text/plain, Size: 112 bytes --]

Have update in the RFC-2 thread in parallel https://lists.openembedded.org/g/openembedded-devel/message/124499

[-- Attachment #2: Type: text/html, Size: 241 bytes --]