Hello Community Members ,
I would like to propose a change to how we handle sparse image generation for ext* file-systems in OpenEmbedded,
specifically moving away from the generic img2simg tool toward the specialised ext2simg in the e2fsprogs [contrib/android]
1) Problem
Currently, image_types_sparse.bbclass uses img2simg for all file-system types.
img2simg is a tool that reads the raw image to find zero-filled blocks.
For ext* file-systems, this is inefficient compared to ext2simg from e2fsprogs [contrib/android]
which understands the file-system structure and uses the block bitmap to determine sparse areas.
2) History
Previously, 'ext2simg' was bundled within android-tools (v5.1 and older).
The utility has since migrated to the e2fsprogs project (contrib/android).
However, upstream e2fsprogs does not provide a build system for this tool,
and it requires libsparse from android-tools to compile.
3) Proposed Changes
To implement this, I am looking at a cross-layer approach:
a. meta-oe (android-tools):
Export libsparse, libbase, and liblog headers and libraries to the sysroot. Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected include paths for e2fsprogs contrib tools.
b. oe-core (e2fsprogs):
Add a conditional compilation step in e2fsprogs-native to build the ext4sparse utility located in contrib/android/.
This would link against the exported libsparse from android-tools-native.
c. meta-oe (image_types_sparse.bbclass):
Update CONVERSION_CMD:sparse to detect ext* types.
Use ext2simg from the e2fsprogs [contrib/android] for these types while falling back to img2simg for others (like f2fs).
4) Points for Discussion / Feasibility
I am seeking feedback on the following:
a. Layer Dependency:
This introduces a tighter coupling between oe-core and meta-oe.
Is it acceptable for e2fsprogs-native (core) to optionally depend on android-tools-native (meta-oe),
or i can create an bbapend file in meta-oe called e2fsprogs_%.bbappend.
b. Build Integration:
In my PoC, I've used a manual ${CC} call in do_compile:append.
Would it be preferred to patch the e2fsprogs Makefile to handle this "contrib" tool more natively?
c. Library Handling:
To ensure the native binary finds libsparse at image creation time, I am currently using LD_LIBRARY_PATH in the class.
I suspect a better approach would be ensuring proper RPATH during the e2fsprogs build.
If members can help for any better way of doing this
d. I have a working proof of concept using RPI4 build
Attached alon are the patches
0001-e2fsprogs-creates-ext4sparse-using-android-tools-lib.patch
0001-meta-oe-image_types_sparse-use-ext4sparse-for-ext-im.patch
5) Logical layout of current POC
This commit bridges the gap between these two recipes:
1. android-tools:
- Updated to v29.0.6 and masked v5.1.1 to provide modern libraries.
- Modified do_install to export libsparse, libbase, and liblog
headers and libraries to the sysroot, enabling external linking.
2. image_types_sparse.bbclass:
- Updated CONVERSION_CMD to branch by filesystem type, calling
ext4sparse specifically for ext* images.
- Exported LD_LIBRARY_PATH to the native sysroot to ensure the
ext4sparse binary can locate its shared dependencies at runtime.
3. Layer Configuration:
- Updated layer.conf to support dynamic-layers for SELinux and
synchronized BBMASK entries to prioritize the updated tools.
6) Benefit :
This reduces the time and cost involved while flashing Android RFS / Fastboot etc
Even in CI/CD , which all adds to ease of usage and have monetary benefit involved with continous CI/CD
Looking forward to thoughts / feedback from community members
Please do let me know if any specific platform or group needs to be addressed to have such discussion
If there is any specific meeting involved where i can join and explain , please do let me know
I can connect to team members and works as per suggestion from community to fine tune / rework if any specific changes community suggest
Thanks ,
Ashish Kumar Mishra