[PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta_only format options

[PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta_only format options

[Hongbo Li]

In this patchset, we have added two formatting options --meta_fix and
--meta_only to extend the ability of EROFS. In the case of using OBS,
we can convert the directory tree structure from OBS into the erofs
image and implement on-demand loading logic based on this. Since OBS
objects are often large, we need to separate the metadata area from
the data area, which is the reason we introduce the --meta_fix option.
To accelerate the formatting process, we can skip the formatting of
the raw data by adding --meta_only option.

A simple usage example is as follows:
  1. Build one xattr with OBS key in s3fs.
  2. mkfs.erofs --meta_fix --meta_only data.img /mnt/s3fs to format
  3. Implement the loading logic in kernel or userspace.

Based on the above logic, we can easily expose the directory tree
from OBS to users in the form of the EROFS file system and implement
on-demand data loading for large OBS objects.

Hongbo Li (4):
  erofs-utils: lib: introduce --meta_fix format option
  erofs-utils: lib: Implement the main logic for --meta_fix option
  erofs-utils: lib: add --meta_only format option
  erofs-utils: lib: remove the compile warning

 include/erofs/blobraw.h |  29 +++++++++
 include/erofs/config.h  |   2 +
 lib/Makefile.am         |   3 +-
 lib/blobraw.c           | 129 ++++++++++++++++++++++++++++++++++++++++
 lib/decompress.c        |   3 +
 lib/inode.c             |   6 ++
 mkfs/main.c             |  49 +++++++++++++++
 7 files changed, 220 insertions(+), 1 deletion(-)
 create mode 100755 include/erofs/blobraw.h
 create mode 100755 lib/blobraw.c

-- 
2.22.0

[PATCH RFC 1/4] erofs-utils: lib: introduce --meta_fix format option

[Hongbo Li]

The option --meta_fix format option allow the mkfs.erofs
can fix the metadata area at the front of the image file.
It makes the disk format easier.

Now we only allow to fix the metadata area at the front
in EROFS_INODE_FLAG_PLAIN layout.

Signed-off-by: Hongbo Li <lihongbo22@huawei.com>
---
 include/erofs/config.h |  1 +
 mkfs/main.c            | 22 ++++++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/include/erofs/config.h b/include/erofs/config.h
index 92c1467..ac6dd9b 100644
--- a/include/erofs/config.h
+++ b/include/erofs/config.h
@@ -66,6 +66,7 @@ struct erofs_configure {
 	bool c_xattr_name_filter;
 	bool c_ovlfs_strip;
 	bool c_hard_dereference;
+	bool c_meta_fix;
 
 #ifdef HAVE_LIBSELINUX
 	struct selabel_handle *sehnd;
diff --git a/mkfs/main.c b/mkfs/main.c
index 6d1a2de..42635c6 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -87,6 +87,7 @@ static struct option long_options[] = {
 	{"sort", required_argument, NULL, 527},
 	{"hard-dereference", no_argument, NULL, 528},
 	{"dsunit", required_argument, NULL, 529},
+	{"meta_fix", no_argument, NULL, 530},
 	{0, 0, 0, 0},
 };
 
@@ -191,6 +192,7 @@ static void usage(int argc, char **argv)
 		"                                            headerball=file data is omited in the source stream)\n"
 		" --ovlfs-strip=<0,1>   strip overlayfs metadata in the target image (e.g. whiteouts)\n"
 		" --quiet               quiet execution (do not write anything to standard output.)\n"
+		" --meta_fix            make metadata area fixed at the front of the image file\n"
 #ifndef NDEBUG
 		" --random-pclusterblks randomize pclusterblks for big pcluster (debugging only)\n"
 		" --random-algorithms   randomize per-file algorithms (debugging only)\n"
@@ -886,6 +888,9 @@ static int mkfs_parse_options_cfg(int argc, char *argv[])
 				return -EINVAL;
 			}
 			break;
+		case 530:
+			cfg.c_meta_fix = true;
+			break;
 		case 'V':
 			version();
 			exit(0);
@@ -903,6 +908,23 @@ static int mkfs_parse_options_cfg(int argc, char *argv[])
 		return -EINVAL;
 	}
 
+	if (cfg.c_meta_fix) {
+		if (cfg.c_compr_opts[0].alg) {
+			erofs_err("--meta_fix cannot be used with compress case");
+			return -EINVAL;
+		}
+
+		if (cfg.c_chunkbits) {
+			erofs_err("--meta_fix cannot be used with --chunksize");
+			return -EINVAL;
+		}
+
+		if (cfg.c_inline_data) {
+			erofs_info("--meta_fix force to turn off the inline_data");
+			cfg.c_inline_data = false;
+		}
+	}
+
 	/* TODO: can be implemented with (deviceslot) mapped_blkaddr */
 	if (cfg.c_blobdev_path &&
 	    cfg.c_force_chunkformat == FORCE_INODE_BLOCK_MAP) {
-- 
2.22.0

[PATCH RFC 2/4] erofs-utils: lib: Implement the main logic for --meta_fix option

[Hongbo Li]

The option --meta_fix will fix the metadata area, so for the data
area, we first dump them into a temporary file to avoid alloc buffer
from @g_sbi.bmgr. Then we remmap the blkaddr based on the offset in
@g_sbi.bmgr after metadata finishing, and dump it into @g_sbi.bmgr
directly.

Signed-off-by: Hongbo Li <lihongbo22@huawei.com>
---
 include/erofs/blobraw.h |  28 ++++++++++
 lib/Makefile.am         |   3 +-
 lib/blobraw.c           | 121 ++++++++++++++++++++++++++++++++++++++++
 lib/inode.c             |   6 ++
 mkfs/main.c             |  15 +++++
 5 files changed, 172 insertions(+), 1 deletion(-)
 create mode 100755 include/erofs/blobraw.h
 create mode 100755 lib/blobraw.c

diff --git a/include/erofs/blobraw.h b/include/erofs/blobraw.h
new file mode 100755
index 0000000..d56bb46
--- /dev/null
+++ b/include/erofs/blobraw.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR Apache-2.0 */
+/*
+ * erofs-utils/lib/blobraw.h
+ *
+ * Copyright (C) 2025, HUAWEI, Inc.
+ */
+#ifndef __EROFS_BLOBRAW_H
+#define __EROFS_BLOBRAW_H
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+
+#include "erofs/internal.h"
+
+void erofs_blobraw_remap_blkaddr(struct erofs_inode *inode);
+int erofs_blobraw_write_file(struct erofs_inode *inode, int fd);
+int erofs_blobraw_init();
+void erofs_blobraw_exit();
+int erofs_mkfs_dump_rawblob(struct erofs_sb_info *sbi, bool drop_blob);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
+
diff --git a/lib/Makefile.am b/lib/Makefile.am
index bdc74ad..c0804b3 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -3,6 +3,7 @@
 noinst_LTLIBRARIES = liberofs.la
 noinst_HEADERS = $(top_srcdir)/include/erofs_fs.h \
       $(top_srcdir)/include/erofs/blobchunk.h \
+      $(top_srcdir)/include/erofs/blobraw.h \
       $(top_srcdir)/include/erofs/block_list.h \
       $(top_srcdir)/include/erofs/cache.h \
       $(top_srcdir)/include/erofs/compress.h \
@@ -33,7 +34,7 @@ noinst_HEADERS = $(top_srcdir)/include/erofs_fs.h \
 noinst_HEADERS += compressor.h
 liberofs_la_SOURCES = config.c io.c cache.c super.c inode.c xattr.c exclude.c \
 		      namei.c data.c compress.c compressor.c zmap.c decompress.c \
-		      compress_hints.c hashmap.c sha256.c blobchunk.c dir.c \
+		      compress_hints.c hashmap.c sha256.c blobchunk.c blobraw.c dir.c \
 		      fragments.c dedupe.c uuid_unparse.c uuid.c tar.c \
 		      block_list.c rebuild.c diskbuf.c bitops.c dedupe_ext.c
 
diff --git a/lib/blobraw.c b/lib/blobraw.c
new file mode 100755
index 0000000..248fc30
--- /dev/null
+++ b/lib/blobraw.c
@@ -0,0 +1,121 @@
+// SPDX-License-Identifier: GPL-2.0+ OR Apache-2.0
+/*
+ * erofs-utils/lib/blobraw.c
+ *
+ * Copyright (C) 2025, HUAWEI, Inc.
+ */
+#define _GNU_SOURCE
+#include <stdlib.h>
+#include <unistd.h>
+
+#include "erofs/blobraw.h"
+#include "erofs/print.h"
+#include "erofs/cache.h"
+#include "liberofs_private.h"
+
+static int blobfile = -1;		/* tmp fd for blob buffer */
+static erofs_blk_t remapped_base;	/* metadata block end address */
+static erofs_off_t bloboff;		/* current blob file offset */
+
+void erofs_blobraw_remap_blkaddr(struct erofs_inode *inode)
+{
+	if (inode->i_size > 0)
+		inode->u.i_blkaddr += remapped_base;
+}
+
+int erofs_blobraw_write_file(struct erofs_inode *inode, int fd)
+{
+	static u8 zeroed[EROFS_MAX_BLOCK_SIZE];
+	struct erofs_sb_info *sbi = inode->sbi;
+	unsigned int padding;
+	erofs_off_t foff;
+	ssize_t length, ret, isize = inode->i_size;
+	u64 pos_in, pos_out = bloboff;
+
+	pos_in = 0;
+	do {
+		length = min_t(erofs_off_t, isize,  SSIZE_MAX);
+		ret = erofs_copy_file_range(fd, &pos_in,
+				blobfile, &pos_out, length);
+	} while (ret > 0 && (isize -= ret));
+
+	foff = lseek(blobfile, inode->i_size, SEEK_CUR);
+	DBG_BUGON(foff != bloboff + inode->i_size);
+
+	padding = erofs_blkoff(sbi, inode->i_size);
+	if (padding) {
+		padding = erofs_blksiz(sbi) - padding;
+		ret = __erofs_io_write(blobfile, zeroed, padding);
+		if (ret > 0 && ret != padding)
+			return -EIO;
+	}
+
+	inode->datalayout = EROFS_INODE_FLAT_PLAIN;
+	inode->u.i_blkaddr = erofs_blknr(sbi, bloboff);
+	bloboff += round_up(inode->i_size, erofs_blksiz(sbi));
+
+	return 0;
+}
+
+int erofs_mkfs_dump_rawblob(struct erofs_sb_info *sbi, bool drop_blob)
+{
+	struct erofs_buffer_head *bh;
+	erofs_off_t blob_size;
+	ssize_t length, ret = 0;
+	u64 pos_in, pos_out;
+
+	if (blobfile < 0) {
+		erofs_err("dump rawblob failed");
+		return -1;
+	}
+
+	blob_size = bloboff;
+
+	bh = erofs_balloc(sbi->bmgr, DATA, blob_size, 0);
+	if (IS_ERR(bh))
+		return PTR_ERR(bh);
+
+	erofs_mapbh(NULL, bh->block);
+
+	pos_out = erofs_btell(bh, false);
+	remapped_base = erofs_blknr(sbi, pos_out);
+	pos_out += sbi->bdev.offset;
+
+	/* copy the raw blob data */
+	if (!drop_blob) {
+		pos_in = 0;
+		do {
+			length = min_t(erofs_off_t, blob_size,  SSIZE_MAX);
+			ret = erofs_copy_file_range(blobfile, &pos_in,
+					sbi->bdev.fd, &pos_out, length);
+		} while (ret > 0 && (blob_size -= ret));
+
+		if (ret >= 0) {
+			if (blob_size) {
+				erofs_err("failed to append the remaining %llu-byte blob data",
+					  blob_size);
+				ret = -EIO;
+			} else {
+				ret = 0;
+			}
+		}
+	}
+
+	erofs_bdrop(bh, false);
+	return ret;
+}
+
+int erofs_blobraw_init()
+{
+	blobfile = erofs_tmpfile();
+	if (blobfile < 0)
+		return -errno;
+
+	return 0;
+}
+
+void erofs_blobraw_exit()
+{
+	if (blobfile >= 0)
+		close(blobfile);
+}
diff --git a/lib/inode.c b/lib/inode.c
index 108aa9e..39257c7 100644
--- a/lib/inode.c
+++ b/lib/inode.c
@@ -28,6 +28,7 @@
 #include "erofs/block_list.h"
 #include "erofs/compress_hints.h"
 #include "erofs/blobchunk.h"
+#include "erofs/blobraw.h"
 #include "erofs/fragments.h"
 #include "liberofs_private.h"
 
@@ -579,6 +580,8 @@ int erofs_write_unencoded_file(struct erofs_inode *inode, int fd, u64 fpos)
 		return erofs_blob_write_chunked_file(inode, fd, fpos);
 	}
 
+	if (cfg.c_meta_fix)
+		return erofs_blobraw_write_file(inode, fd);
 	/* fallback to all data uncompressed */
 	return write_uncompressed_file_from_fd(inode, fd);
 }
@@ -600,6 +603,9 @@ int erofs_iflush(struct erofs_inode *inode)
 	else
 		off = erofs_iloc(inode);
 
+	if (S_ISREG(inode->i_mode) && cfg.c_meta_fix)
+		erofs_blobraw_remap_blkaddr(inode);
+
 	if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode) ||
 	    S_ISFIFO(inode->i_mode) || S_ISSOCK(inode->i_mode))
 		u1.rdev = cpu_to_le32(inode->u.i_rdev);
diff --git a/mkfs/main.c b/mkfs/main.c
index 42635c6..82db452 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -26,6 +26,7 @@
 #include "erofs/block_list.h"
 #include "erofs/compress_hints.h"
 #include "erofs/blobchunk.h"
+#include "erofs/blobraw.h"
 #include "erofs/fragments.h"
 #include "erofs/rebuild.h"
 #include "../lib/liberofs_private.h"
@@ -1422,6 +1423,12 @@ int main(int argc, char **argv)
 			return 1;
 	}
 
+	if (cfg.c_meta_fix) {
+		err = erofs_blobraw_init();
+		if (err)
+			return -1;
+	}
+
 	if (((erofstar.index_mode && !erofstar.headeronly_mode) &&
 	    !erofstar.mapfile) || cfg.c_blobdev_path) {
 		err = erofs_mkfs_init_devices(&g_sbi, 1);
@@ -1498,6 +1505,12 @@ int main(int argc, char **argv)
 			goto exit;
 	}
 
+	if (cfg.c_meta_fix) {
+		err = erofs_mkfs_dump_rawblob(&g_sbi, cfg.c_meta_fix);
+		if (err)
+			goto exit;
+	}
+
 	/* flush all buffers except for the superblock */
 	err = erofs_bflush(g_sbi.bmgr, NULL);
 	if (err)
@@ -1537,6 +1550,8 @@ exit:
 	erofs_cleanup_exclude_rules();
 	if (cfg.c_chunkbits)
 		erofs_blob_exit();
+	if (cfg.c_meta_fix)
+		erofs_blobraw_exit();
 	erofs_packedfile_exit(&g_sbi);
 	erofs_xattr_cleanup_name_prefixes();
 	erofs_rebuild_cleanup();
-- 
2.22.0

[PATCH RFC 3/4] erofs-utils: lib: add --meta_only format option

[Hongbo Li]

This option is used to drop the data part based on --meta_fix.
It allows the user can only get the metadata part of the erofs
images.

Signed-off-by: Hongbo Li <lihongbo22@huawei.com>
---
 include/erofs/blobraw.h |  3 ++-
 include/erofs/config.h  |  1 +
 lib/blobraw.c           | 42 ++++++++++++++++++++++++-----------------
 lib/inode.c             |  2 +-
 mkfs/main.c             | 12 ++++++++++++
 5 files changed, 41 insertions(+), 19 deletions(-)

diff --git a/include/erofs/blobraw.h b/include/erofs/blobraw.h
index d56bb46..68df6b8 100755
--- a/include/erofs/blobraw.h
+++ b/include/erofs/blobraw.h
@@ -14,8 +14,9 @@ extern "C"
 
 #include "erofs/internal.h"
 
+erofs_blk_t erofs_blobraw_total(struct erofs_sb_info *sbi);
 void erofs_blobraw_remap_blkaddr(struct erofs_inode *inode);
-int erofs_blobraw_write_file(struct erofs_inode *inode, int fd);
+int erofs_blobraw_write_file(struct erofs_inode *inode, int fd, bool drop_blob);
 int erofs_blobraw_init();
 void erofs_blobraw_exit();
 int erofs_mkfs_dump_rawblob(struct erofs_sb_info *sbi, bool drop_blob);
diff --git a/include/erofs/config.h b/include/erofs/config.h
index ac6dd9b..491b3aa 100644
--- a/include/erofs/config.h
+++ b/include/erofs/config.h
@@ -67,6 +67,7 @@ struct erofs_configure {
 	bool c_ovlfs_strip;
 	bool c_hard_dereference;
 	bool c_meta_fix;
+	bool c_meta_only;
 
 #ifdef HAVE_LIBSELINUX
 	struct selabel_handle *sehnd;
diff --git a/lib/blobraw.c b/lib/blobraw.c
index 248fc30..9113a49 100755
--- a/lib/blobraw.c
+++ b/lib/blobraw.c
@@ -17,13 +17,19 @@ static int blobfile = -1;		/* tmp fd for blob buffer */
 static erofs_blk_t remapped_base;	/* metadata block end address */
 static erofs_off_t bloboff;		/* current blob file offset */
 
+erofs_blk_t erofs_blobraw_total(struct erofs_sb_info *sbi)
+{
+	return erofs_blknr(sbi, bloboff);
+}
+
 void erofs_blobraw_remap_blkaddr(struct erofs_inode *inode)
 {
 	if (inode->i_size > 0)
 		inode->u.i_blkaddr += remapped_base;
 }
 
-int erofs_blobraw_write_file(struct erofs_inode *inode, int fd)
+int erofs_blobraw_write_file(struct erofs_inode *inode,
+				     int fd, bool drop_blob)
 {
 	static u8 zeroed[EROFS_MAX_BLOCK_SIZE];
 	struct erofs_sb_info *sbi = inode->sbi;
@@ -32,22 +38,24 @@ int erofs_blobraw_write_file(struct erofs_inode *inode, int fd)
 	ssize_t length, ret, isize = inode->i_size;
 	u64 pos_in, pos_out = bloboff;
 
-	pos_in = 0;
-	do {
-		length = min_t(erofs_off_t, isize,  SSIZE_MAX);
-		ret = erofs_copy_file_range(fd, &pos_in,
-				blobfile, &pos_out, length);
-	} while (ret > 0 && (isize -= ret));
-
-	foff = lseek(blobfile, inode->i_size, SEEK_CUR);
-	DBG_BUGON(foff != bloboff + inode->i_size);
-
-	padding = erofs_blkoff(sbi, inode->i_size);
-	if (padding) {
-		padding = erofs_blksiz(sbi) - padding;
-		ret = __erofs_io_write(blobfile, zeroed, padding);
-		if (ret > 0 && ret != padding)
-			return -EIO;
+	if (!drop_blob) {
+		pos_in = 0;
+		do {
+			length = min_t(erofs_off_t, isize,  SSIZE_MAX);
+			ret = erofs_copy_file_range(fd, &pos_in,
+					blobfile, &pos_out, length);
+		} while (ret > 0 && (isize -= ret));
+
+		foff = lseek(blobfile, inode->i_size, SEEK_CUR);
+		DBG_BUGON(foff != bloboff + inode->i_size);
+
+		padding = erofs_blkoff(sbi, inode->i_size);
+		if (padding) {
+			padding = erofs_blksiz(sbi) - padding;
+			ret = __erofs_io_write(blobfile, zeroed, padding);
+			if (ret > 0 && ret != padding)
+				return -EIO;
+		}
 	}
 
 	inode->datalayout = EROFS_INODE_FLAT_PLAIN;
diff --git a/lib/inode.c b/lib/inode.c
index 39257c7..6eb275b 100644
--- a/lib/inode.c
+++ b/lib/inode.c
@@ -581,7 +581,7 @@ int erofs_write_unencoded_file(struct erofs_inode *inode, int fd, u64 fpos)
 	}
 
 	if (cfg.c_meta_fix)
-		return erofs_blobraw_write_file(inode, fd);
+		return erofs_blobraw_write_file(inode, fd, cfg.c_meta_only);
 	/* fallback to all data uncompressed */
 	return write_uncompressed_file_from_fd(inode, fd);
 }
diff --git a/mkfs/main.c b/mkfs/main.c
index 82db452..be3ee7f 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -89,6 +89,7 @@ static struct option long_options[] = {
 	{"hard-dereference", no_argument, NULL, 528},
 	{"dsunit", required_argument, NULL, 529},
 	{"meta_fix", no_argument, NULL, 530},
+	{"meta_only", no_argument, NULL, 531},
 	{0, 0, 0, 0},
 };
 
@@ -194,6 +195,7 @@ static void usage(int argc, char **argv)
 		" --ovlfs-strip=<0,1>   strip overlayfs metadata in the target image (e.g. whiteouts)\n"
 		" --quiet               quiet execution (do not write anything to standard output.)\n"
 		" --meta_fix            make metadata area fixed at the front of the image file\n"
+		" --meta_only           only keep the metadata area in image file\n"
 #ifndef NDEBUG
 		" --random-pclusterblks randomize pclusterblks for big pcluster (debugging only)\n"
 		" --random-algorithms   randomize per-file algorithms (debugging only)\n"
@@ -892,6 +894,9 @@ static int mkfs_parse_options_cfg(int argc, char *argv[])
 		case 530:
 			cfg.c_meta_fix = true;
 			break;
+		case 531:
+			cfg.c_meta_only = true;
+			break;
 		case 'V':
 			version();
 			exit(0);
@@ -909,6 +914,11 @@ static int mkfs_parse_options_cfg(int argc, char *argv[])
 		return -EINVAL;
 	}
 
+	if (cfg.c_meta_only && !cfg.c_meta_fix) {
+		erofs_err("--meta_only must be used together with --meta_fix");
+		return -EINVAL;
+	}
+
 	if (cfg.c_meta_fix) {
 		if (cfg.c_compr_opts[0].alg) {
 			erofs_err("--meta_fix cannot be used with compress case");
@@ -1529,6 +1539,8 @@ int main(int argc, char **argv)
 	if (err)
 		goto exit;
 
+	if (cfg.c_meta_only)
+		nblocks -= erofs_blobraw_total(&g_sbi);
 	err = erofs_dev_resize(&g_sbi, nblocks);
 
 	if (!err && erofs_sb_has_sb_chksum(&g_sbi)) {
-- 
2.22.0

[PATCH RFC 4/4] erofs-utils: lib: remove the compile warning

[Hongbo Li]

Add the condition macro to avoid the compile warning.

Signed-off-by: Hongbo Li <lihongbo22@huawei.com>
---
 lib/decompress.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/decompress.c b/lib/decompress.c
index 3f553a8..1e9fad7 100644
--- a/lib/decompress.c
+++ b/lib/decompress.c
@@ -9,6 +9,8 @@
 #include "erofs/err.h"
 #include "erofs/print.h"
 
+#if defined(HAVE_LIBZSTD) || defined(HAVE_QPL) || defined(HAVE_LIBDEFLATE) || \
+    defined(HAVE_ZLIB) || defined(HAVE_LIBLZMA) || defined(LZ4_ENABLED)
 static unsigned int z_erofs_fixup_insize(const u8 *padbuf, unsigned int padbufsize)
 {
 	unsigned int inputmargin;
@@ -17,6 +19,7 @@ static unsigned int z_erofs_fixup_insize(const u8 *padbuf, unsigned int padbufsi
 	     !padbuf[inputmargin]; ++inputmargin);
 	return inputmargin;
 }
+#endif
 
 #ifdef HAVE_LIBZSTD
 #include <zstd.h>
-- 
2.22.0

Re: [PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta_only format options

[Gao Xiang]

Hi Hongbo,

On 2025/4/22 20:36, Hongbo Li wrote:
> In this patchset, we have added two formatting options --meta_fix and
> --meta_only to extend the ability of EROFS. In the case of using OBS,
> we can convert the directory tree structure from OBS into the erofs
> image and implement on-demand loading logic based on this. Since OBS
> objects are often large, we need to separate the metadata area from
> the data area, which is the reason we introduce the --meta_fix option.
> To accelerate the formatting process, we can skip the formatting of
> the raw data by adding --meta_only option.

Thanks for the patches!

I wonder if it's possible to reuse blobchunk.c codebase for
such usage in the short term.

Also I hope there could be some better option knobs for these
new features.

> 
> A simple usage example is as follows:
>    1. Build one xattr with OBS key in s3fs.
>    2. mkfs.erofs --meta_fix --meta_only data.img /mnt/s3fs to format
>    3. Implement the loading logic in kernel or userspace.
> 
> Based on the above logic, we can easily expose the directory tree
> from OBS to users in the form of the EROFS file system and implement
> on-demand data loading for large OBS objects.

BTW, It's possible to upstream OBS implementation to erofs-utils
if any chance?

Thanks,
Gao Xiang

Re: [PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta_only format options

[Hongbo Li]

On 2025/4/22 21:50, Gao Xiang wrote:
> Hi Hongbo,
> 
> On 2025/4/22 20:36, Hongbo Li wrote:
>> In this patchset, we have added two formatting options --meta_fix and
>> --meta_only to extend the ability of EROFS. In the case of using OBS,
>> we can convert the directory tree structure from OBS into the erofs
>> image and implement on-demand loading logic based on this. Since OBS
>> objects are often large, we need to separate the metadata area from
>> the data area, which is the reason we introduce the --meta_fix option.
>> To accelerate the formatting process, we can skip the formatting of
>> the raw data by adding --meta_only option.
> 
> Thanks for the patches!
> 
> I wonder if it's possible to reuse blobchunk.c codebase for
> such usage in the short term.
> 
Our initial plan was to reuse the blobchunk.c logic, but we found that 
the chunk-based layout has some minor issues when handling contiguous 
blocks—it would result in multiple elements in the chunk index array 
(whereas blobraw expects only oversized chunks).

Thanks,
Hongbo

> Also I hope there could be some better option knobs for these
> new features.
> 
>>
>> A simple usage example is as follows:
>>    1. Build one xattr with OBS key in s3fs.
>>    2. mkfs.erofs --meta_fix --meta_only data.img /mnt/s3fs to format
>>    3. Implement the loading logic in kernel or userspace.
>>
>> Based on the above logic, we can easily expose the directory tree
>> from OBS to users in the form of the EROFS file system and implement
>> on-demand data loading for large OBS objects.
> 
> BTW, It's possible to upstream OBS implementation to erofs-utils
> if any chance?
> 
> Thanks,
> Gao Xiang

Re: [PATCH RFC 0/4] erofs-utils: Add --meta_fix and --meta_only format options

[Gao Xiang]

On 2025/4/22 22:37, Hongbo Li wrote:
> 
> 
> On 2025/4/22 21:50, Gao Xiang wrote:
>> Hi Hongbo,
>>
>> On 2025/4/22 20:36, Hongbo Li wrote:
>>> In this patchset, we have added two formatting options --meta_fix and
>>> --meta_only to extend the ability of EROFS. In the case of using OBS,
>>> we can convert the directory tree structure from OBS into the erofs
>>> image and implement on-demand loading logic based on this. Since OBS
>>> objects are often large, we need to separate the metadata area from
>>> the data area, which is the reason we introduce the --meta_fix option.
>>> To accelerate the formatting process, we can skip the formatting of
>>> the raw data by adding --meta_only option.
>>
>> Thanks for the patches!
>>
>> I wonder if it's possible to reuse blobchunk.c codebase for
>> such usage in the short term.
>>
> Our initial plan was to reuse the blobchunk.c logic, but we found that the chunk-based layout has some minor issues when handling contiguous blocks—it would result in multiple elements in the chunk index array (whereas blobraw expects only oversized chunks).

Sorry for late reply.

It seems blobchunk bugs? Could you submit fixes for this?

Thanks,
Gao Xiang