[PATCH experimental-tests] erofs-utils: tests: add test for rebuild fulldata

[PATCH experimental-tests] erofs-utils: tests: add test for rebuild fulldata

[Lucas Karpinski]

Add a regression test for building a self-contained image out of two other
erofs images.

1. Create img1 & img2 with various file types and sizes.
2. Rebuild a new self-contained image of img1 & img2.
3. Delete img1 & img2 to prove that this isn't use the blob based rebuild
   path.
4. Mount the new self-contained image and compare the checksum to the
   original folders.

Signed-off-by: Lucas Karpinski <lkarpinski@nvidia.com>
---
 tests/Makefile.am   |  3 ++
 tests/erofs/030     | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/erofs/030.out |  2 ++
 3 files changed, 93 insertions(+)

diff --git a/tests/Makefile.am b/tests/Makefile.am
index d8ac0678..125aa8f6 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -126,6 +126,9 @@ TESTS += erofs/028
 # 029 - test FUSE daemon and kernel error handling on corrupted inodes
 TESTS += erofs/029
 
+# 030 - verify multi-source fulldata rebuild
+TESTS += erofs/030
+
 # NEW TEST CASE HERE
 # TESTS += erofs/999
 
diff --git a/tests/erofs/030 b/tests/erofs/030
new file mode 100755
index 00000000..8111be10
--- /dev/null
+++ b/tests/erofs/030
@@ -0,0 +1,88 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0+
+#
+# verify multi-source fulldata rebuild merges multiple EROFS images into
+# a single self-contained image
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$(echo $0 | awk '{print $((NF-1))"/"$NF}' FS="/")
+
+# get standard environment, filters and checks
+. "${srcdir}/common/rc"
+
+cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+_require_erofs
+_require_fssum
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+echo "QA output created by $seq"
+
+if [ -z "$SCRATCH_DEV" ]; then
+	SCRATCH_DEV=$tmp/erofs_$seq.img
+	rm -f $SCRATCH_DEV
+fi
+
+src1dir="$tmp/$seq.src1"
+src2dir="$tmp/$seq.src2"
+combdir="$tmp/$seq.combined"
+img1="$tmp/$seq.img1"
+img2="$tmp/$seq.img2"
+
+rm -rf $src1dir $src2dir $combdir
+rm -f $img1 $img2
+
+# Two source trees that share a top-level `common/` directory so the
+# rebuild must merge directories, with otherwise disjoint contents.
+mkdir -p $src1dir/tree1 $src1dir/common/only1
+printf "image 1 small file\n" > $src1dir/tree1/small
+printf "image 1 common\n" > $src1dir/common/only1/a.txt
+yes "image 1 payload" | head -c 65536 > $src1dir/tree1/big1
+ln $src1dir/tree1/small $src1dir/tree1/hardlink
+ln -s small $src1dir/tree1/symlink
+
+mkdir -p $src2dir/tree2 $src2dir/common/only2
+printf "image 2 file\n"   > $src2dir/tree2/file2.txt
+printf "image 2 common\n" > $src2dir/common/only2/b.txt
+yes "image 2 payload" | head -c 32768 > $src2dir/tree2/big2
+
+# combined reference tree
+mkdir -p $combdir
+cp -a $src1dir/. $combdir/
+cp -a $src2dir/. $combdir/
+
+# build two source EROFS images
+$MKFS_EROFS_PROG $img1 $src1dir >> $seqres.full 2>&1 || _fail "failed to create $img1"
+$MKFS_EROFS_PROG $img2 $src2dir >> $seqres.full 2>&1 || _fail "failed to create $img2"
+
+# multi-source fulldata rebuild
+$MKFS_EROFS_PROG --clean=data $SCRATCH_DEV $img1 $img2 >> $seqres.full 2>&1 || \
+	_fail "failed to rebuild merged image with --clean=data"
+$FSCK_EROFS_PROG --extract $SCRATCH_DEV >> $seqres.full 2>&1 || \
+	_fail "merged image failed integrity check"
+
+# remove both source images: the merged image must be self-contained
+rm -f $img1 $img2
+
+FSSUM_OPTS="-MAC"
+[ $FSTYP = "erofsfuse" ] && FSSUM_OPTS="${FSSUM_OPTS}T"
+
+_scratch_mount 2>>$seqres.full
+sum1=`$FSSUM_PROG $FSSUM_OPTS $combdir`
+sum2=`$FSSUM_PROG $FSSUM_OPTS $SCRATCH_MNT`
+echo "combined source checksum: $sum1" >>$seqres.full
+echo "merged image checksum: $sum2" >>$seqres.full
+_scratch_unmount
+
+[ "x$sum1" = "x$sum2" ] || _fail "multi-source fulldata rebuild mismatch"
+
+echo Silence is golden
+status=0
+exit 0
diff --git a/tests/erofs/030.out b/tests/erofs/030.out
new file mode 100644
index 00000000..06a1c8fe
--- /dev/null
+++ b/tests/erofs/030.out
@@ -0,0 +1,2 @@
+QA output created by 030
+Silence is golden

---
base-commit: feadb09fcfa1b05ca930cfce871d0e56f6bd3a95
change-id: 20260417-b4-fulldata-test-b8e15272bc59

Best regards,
-- 
Lucas Karpinski <lkarpinski@nvidia.com>

Re: [PATCH experimental-tests] erofs-utils: tests: add test for rebuild fulldata

[Gao Xiang]

Hi Lucas,

On 2026/4/18 04:06, Lucas Karpinski wrote:
> Add a regression test for building a self-contained image out of two other
> erofs images.
> 
> 1. Create img1 & img2 with various file types and sizes.
> 2. Rebuild a new self-contained image of img1 & img2.
> 3. Delete img1 & img2 to prove that this isn't use the blob based rebuild
>     path.
> 4. Mount the new self-contained image and compare the checksum to the
>     original folders.

Thanks for the patch. I wonder if it's possible to:
  1) Use a common dataset as the base layer (see other existing tests);
  2) Mount two overlayfs to get the incremental layers;
     - one to generate some artifacial dirs, files;
     - the other to change/delete some files, or delete dirs, new another
       dirs;
  3) Test typical metadata (whiteout, opaque), and data with:
     a) fsmerged erofs;
     b) overlayfs mount;
     fssum those two mounts.

Thanks,
Gao Xiang

[PATCH v2 experimental-tests] erofs-utils: tests: add test for rebuild fulldata

[Lucas Karpinski]

Add a regression test for building a self-contained image out of two other
erofs images. Validate that image against a live overlayfs mount.

1. Build a base erofs image with deterministic fixtures.
2. Produce two overlayfs upperdirs; generating whitouts and opaque markers.
3. Build an erofs image per layer, then merge all three.
4. Delete the source layers to validate the merged image is self-contained.
5. Mount the merged image and a read-only stacked overlayfs and assert both
have identical checksums.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Lucas Karpinski <lkarpinski@nvidia.com>
---
 tests/Makefile.am   |   3 +
 tests/erofs/030     | 179 ++++++++++++++++++++++++++++++++++++++++++++
 tests/erofs/030.out |   2 +
 3 files changed, 184 insertions(+)
 create mode 100755 tests/erofs/030
 create mode 100644 tests/erofs/030.out

diff --git a/tests/Makefile.am b/tests/Makefile.am
index d8ac0678..125aa8f6 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -126,6 +126,9 @@ TESTS += erofs/028
 # 029 - test FUSE daemon and kernel error handling on corrupted inodes
 TESTS += erofs/029

+# 030 - verify multi-source fulldata rebuild
+TESTS += erofs/030
+
 # NEW TEST CASE HERE
 # TESTS += erofs/999

diff --git a/tests/erofs/030 b/tests/erofs/030
new file mode 100755
index 00000000..afbb69dd
--- /dev/null
+++ b/tests/erofs/030
@@ -0,0 +1,179 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0+
+#
+# verify `mkfs.erofs --clean=data` merges a base image and two
+# overlayfs-derived layers into an image matching a live overlayfs view
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$(echo $0 | awk '{print $((NF-1))"/"$NF}' FS="/")
+
+. "${srcdir}/common/rc"
+
+cleanup()
+{
+	cd /
+	for m in "$ref_mnt" "$ovl2_mnt" "$ovl1_mnt"; do
+		[ -n "$m" ] || continue
+		$UMOUNT_PROG "$m" 2>/dev/null || $UMOUNT_PROG -l "$m" 2>/dev/null
+	done
+	rm -rf $tmp.*
+}
+
+# _mount_overlay_rw <lowers> <upper> <work> <mnt>
+_mount_overlay_rw()
+{
+	$MOUNT_PROG -t overlay overlay \
+		-o lowerdir="$1",upperdir="$2",workdir="$3" "$4" \
+		>>$seqres.full 2>&1 || _fail "failed to mount overlay at $4"
+}
+
+# overlay teardown can race with kernel dentry release; fall back to lazy
+_overlay_unmount()
+{
+	sync
+	$UMOUNT_PROG "$1" >>$seqres.full 2>&1 && return
+	$UMOUNT_PROG -l "$1" >>$seqres.full 2>&1 ||
+		_fail "failed to unmount overlay at $1"
+}
+
+# kernel overlay can uses chrdev(0,0) or trusted.overlay.whiteout=y, but
+# mkfs.erofs only understands the chrdev form, so promote
+_normalize_overlay_whiteouts()
+{
+	local p
+
+	for p in `find "$1" -type f`; do
+		getfattr -n trusted.overlay.whiteout "$p" >/dev/null 2>&1 || continue
+		rm -f "$p" && mknod "$p" c 0 0 >>$seqres.full 2>&1 ||
+			_fail "failed to convert whiteout $p"
+	done
+}
+
+# drop overlay bookkeeping xattrs (origin/impure/uuid/...) that are
+# invisible through a live overlay; keep opaque, --ovlfs-strip=1 handles it
+_strip_overlay_bookkeeping_xattrs()
+{
+	local name p
+
+	for p in `find "$1" -mindepth 1`; do
+		getfattr -d -m 'trusted\.overlay\.' --absolute-names "$p" \
+			2>/dev/null |
+		awk -F= '/^trusted\.overlay\./ {print $1}' |
+		while read name; do
+			[ "$name" = "trusted.overlay.opaque" ] && continue
+			setfattr -x "$name" "$p" >>$seqres.full 2>&1 ||
+				_fail "failed to strip $name from $p"
+		done
+	done
+}
+
+# normalize a kernel-produced upperdir and build an EROFS image from it
+_build_layer_image()
+{
+	local upper="$1" img="$2"
+
+	_normalize_overlay_whiteouts "$upper"
+	_strip_overlay_bookkeeping_xattrs "$upper"
+	$MKFS_EROFS_PROG "$img" "$upper" >>$seqres.full 2>&1 ||
+		_fail "failed to build layer image $img"
+}
+
+_require_erofs
+_require_overlayfs
+_require_fssum
+_require_xattr
+
+rm -f $seqres.full
+echo "QA output created by $seq"
+
+if [ -z "$SCRATCH_DEV" ]; then
+	SCRATCH_DEV=$tmp/erofs_$seq.img
+	rm -f $SCRATCH_DEV
+fi
+
+basedir="$tmp/$seq.base"
+baseimg="$tmp/$seq.base.erofs"
+img1="$tmp/$seq.layer1.erofs"
+img2="$tmp/$seq.layer2.erofs"
+upper1="$tmp/$seq.upper1"; work1="$tmp/$seq.work1"
+upper2="$tmp/$seq.upper2"; work2="$tmp/$seq.work2"
+ovl1_mnt="$tmp/$seq.ovl1.mnt"
+ovl2_mnt="$tmp/$seq.ovl2.mnt"
+ref_mnt="$tmp/$seq.ref.mnt"
+
+mkdir -p "$basedir" "$upper1" "$work1" "$upper2" "$work2" \
+	"$ovl1_mnt" "$ovl2_mnt" "$ref_mnt"
+
+# collect files pending for verification
+dirs=`find ../ -maxdepth 1 -type d -exec printf {}: \;`
+IFS=':'
+for d in $dirs; do
+	[ "$d" = '../' ] && continue
+	[ -z "${d##\.\./tests*}" ] && continue
+	[ -z "${d##\.\./\.*}" ] && continue
+	cp -R "$d" "$basedir"
+done
+unset IFS
+
+# deterministic fixtures for the overlay layers to manipulate
+mkdir -p "$basedir/app/opaque_dir/subdir" "$basedir/app/delete_dir" \
+	"$basedir/app/keep_dir"
+echo "base content" > "$basedir/app/keep_dir/change.txt"
+echo "base whiteout target" > "$basedir/app/keep_dir/remove_l1.txt"
+echo "base layer2 target" > "$basedir/app/keep_dir/remove_l2.txt"
+echo "from base" > "$basedir/app/opaque_dir/subdir/original.txt"
+echo "to be removed" > "$basedir/app/delete_dir/original.txt"
+
+$MKFS_EROFS_PROG "$baseimg" "$basedir" >>$seqres.full 2>&1 ||
+	_fail "failed to create base image"
+
+# layer 1: add dir+file, modify a file, delete a file (whiteout)
+_mount_overlay_rw "$basedir" "$upper1" "$work1" "$ovl1_mnt"
+mkdir -p "$ovl1_mnt/app/l1_new"
+echo "layer1 add" > "$ovl1_mnt/app/l1_new/new.txt"
+echo "layer1 edit" > "$ovl1_mnt/app/keep_dir/change.txt"
+rm -f "$ovl1_mnt/app/keep_dir/remove_l1.txt"
+_overlay_unmount "$ovl1_mnt"
+_build_layer_image "$upper1" "$img1"
+
+# layer 2 stacked on (upper1, basedir): whiteout file+dir, add file,
+# opaque-replace a directory so the lower subtree is fully hidden
+_mount_overlay_rw "$upper1:$basedir" "$upper2" "$work2" "$ovl2_mnt"
+rm -rf "$ovl2_mnt/app/delete_dir" "$ovl2_mnt/app/opaque_dir"
+rm -f "$ovl2_mnt/app/keep_dir/remove_l2.txt"
+mkdir -p "$ovl2_mnt/app/l2_new" "$ovl2_mnt/app/opaque_dir"
+echo "layer2 add" > "$ovl2_mnt/app/l2_new/new2.txt"
+echo "opaque replaces lower" > "$ovl2_mnt/app/opaque_dir/replacement.txt"
+_overlay_unmount "$ovl2_mnt"
+_build_layer_image "$upper2" "$img2"
+
+# merge both source images into a single self-contained image
+$MKFS_EROFS_PROG --clean=data --ovlfs-strip=1 "$SCRATCH_DEV" \
+	"$baseimg" "$img1" "$img2" >>$seqres.full 2>&1 ||
+	_fail "failed to rebuild merged image with --clean=data"
+
+# remove both source images: the merged image must be self-contained
+rm -f "$img1" "$img2"
+
+# live read-only stacked overlay of the same layers
+$MOUNT_PROG -t overlay overlay \
+	-o ro,lowerdir="$upper2:$upper1:$basedir" "$ref_mnt" \
+	>>$seqres.full 2>&1 || _fail "failed to mount reference overlay"
+_scratch_mount 2>>$seqres.full
+
+FSSUM_OPTS="-MAC"
+[ $FSTYP = "erofsfuse" ] && FSSUM_OPTS="${FSSUM_OPTS}T"
+sum_overlay=`$FSSUM_PROG $FSSUM_OPTS "$ref_mnt"`
+sum_fsmerged=`$FSSUM_PROG $FSSUM_OPTS "$SCRATCH_MNT"`
+echo "overlayfs checksum:  $sum_overlay"  >>$seqres.full
+echo "fsmerged checksum:   $sum_fsmerged" >>$seqres.full
+
+_scratch_unmount
+_overlay_unmount "$ref_mnt"
+
+[ "x$sum_overlay" = "x$sum_fsmerged" ] ||
+	_fail "overlayfs and fsmerged checksums mismatch"
+
+echo Silence is golden
+status=0
+exit 0
diff --git a/tests/erofs/030.out b/tests/erofs/030.out
new file mode 100644
index 00000000..06a1c8fe
--- /dev/null
+++ b/tests/erofs/030.out
@@ -0,0 +1,2 @@
+QA output created by 030
+Silence is golden
--