Rebuild mode for tail-pack layouts

Rebuild mode for tail-pack layouts

[xtex]

Hi!

In erofs_rebuild_write_blob_index (rebuild.c), only EROFS_INODE_CHUNK_BASED 
and FLAT_PLAIN are implemented, so when generating a metadata index with 
rebuild mode, the sources cannot use tail-pack nor inline data layout.
However, disabling tail-packing can lead to great disk-space waste in many 
cases, especially when the file-system consists of a lot of small files.

Thus I attempted to implement FLAT_INLINE for it, only to realize that the 
current chunk entry formats can only represent physical addresses that are 
block-aligned while tail-pack extent is not.

I wonder what do you think about adding a new chunk entry format? And how 
should it be named?

I would suggest the following structure:
struct erofs_inode_chunk_index_tp {
	__le16 startblk_hi;	/* starting block number MSB */
	__le16 device_id;	/* back-end storage id (with bits masked) 
*/
	__le32 startblk_lo;	/* starting block number of this chunk */
	/* new fields below */
	__le16 startblk_off;	/* starting block offset */
	__le16 reserved;
} __packed;
The 16b offset should be enough unless we are to support block size > 64K.
The reserved field is added for alignment.

Best wishes.

-- 
xtex (Bingwu Zhang, a.k.a Zerogawa Chiaki) @ Thu, 30 Apr 2026 11:45:17 +0000

Re: Rebuild mode for tail-pack layouts

[Gao Xiang]

Hi xtex,

On 2026/4/30 20:09, xtex wrote:
> Hi!
> 
> In erofs_rebuild_write_blob_index (rebuild.c), only EROFS_INODE_CHUNK_BASED
> and FLAT_PLAIN are implemented, so when generating a metadata index with
> rebuild mode, the sources cannot use tail-pack nor inline data layout.
> However, disabling tail-packing can lead to great disk-space waste in many
> cases, especially when the file-system consists of a lot of small files.
> 
> Thus I attempted to implement FLAT_INLINE for it, only to realize that the
> current chunk entry formats can only represent physical addresses that are
> block-aligned while tail-pack extent is not.
> 
> I wonder what do you think about adding a new chunk entry format? And how
> should it be named?
> 
> I would suggest the following structure:
> struct erofs_inode_chunk_index_tp {
> 	__le16 startblk_hi;	/* starting block number MSB */
> 	__le16 device_id;	/* back-end storage id (with bits masked)
> */
> 	__le32 startblk_lo;	/* starting block number of this chunk */
> 	/* new fields below */
> 	__le16 startblk_off;	/* starting block offset */
> 	__le16 reserved;
> } __packed;
> The 16b offset should be enough unless we are to support block size > 64K.
> The reserved field is added for alignment.

Sorry about the late response.

Thanks for the question.

FLAT_INLINE can be used for index rebuilding, which can work with
uniaddr (or mapped_blkaddr) since the blkaddr will be mapped
into the relative address based on the blob starting with
mapped_blkaddr:

https://erofs.docs.kernel.org/en/latest/ondisk/chunked_format.html#device-table

But I agree the expression in the page above is a bit
ambigious through.

Thanks,
Gao Xiang


> 
> Best wishes.
> 

Re: Rebuild mode for tail-pack layouts

[Bingwu Zhang]

Hi,

On Monday, May 4, 2026 11:13:23 PM China Standard Time Gao Xiang wrote:
> Hi xtex,
> 
> On 2026/4/30 20:09, xtex wrote:
> > Hi!
> > 
> > In erofs_rebuild_write_blob_index (rebuild.c), only
> > EROFS_INODE_CHUNK_BASED
> > and FLAT_PLAIN are implemented, so when generating a metadata index with
> > rebuild mode, the sources cannot use tail-pack nor inline data layout.
> > However, disabling tail-packing can lead to great disk-space waste in many
> > cases, especially when the file-system consists of a lot of small files.
> > 
> > Thus I attempted to implement FLAT_INLINE for it, only to realize that the
> > current chunk entry formats can only represent physical addresses that are
> > block-aligned while tail-pack extent is not.
> > 
> > I wonder what do you think about adding a new chunk entry format? And how
> > should it be named?
> > 
> > I would suggest the following structure:
> > struct erofs_inode_chunk_index_tp {
> > 
> > 	__le16 startblk_hi;	/* starting block number MSB */
> > 	__le16 device_id;	/* back-end storage id (with bits masked)
> > 
> > */
> > 
> > 	__le32 startblk_lo;	/* starting block number of this chunk */
> > 	/* new fields below */
> > 	__le16 startblk_off;	/* starting block offset */
> > 	__le16 reserved;
> > 
> > } __packed;
> > The 16b offset should be enough unless we are to support block size > 64K.
> > The reserved field is added for alignment.
> 
> Sorry about the late response.
> 
> Thanks for the question.
> 
> FLAT_INLINE can be used for index rebuilding, which can work with
> uniaddr (or mapped_blkaddr) since the blkaddr will be mapped
> into the relative address based on the blob starting with
> mapped_blkaddr:
> 
> https://erofs.docs.kernel.org/en/latest/ondisk/chunked_format.html#device-ta
> ble
> 
> But I agree the expression in the page above is a bit
> ambigious through.
> 
> Thanks,
> Gao Xiang
> 
> > Best wishes.

Sorry for the late response and thanks for your answer.

I am sorry about that I didn't get it.
In __erofs_map_blocks:
> map->m_pa = erofs_pos(sbi, startblk);
and
> #define erofs_pos(sbi, nr)      ((erofs_off_t)(nr) << (sbi)->blkszbits)
It seems like that the mapped PA is always block-aligned? However, the last 
chunk of inline data is not?

Best wishes.

-- 
xtex (a.k.a. Bingwu Zhang) @ Sat, 09 May 2026 08:24:53 +0000

Re: Rebuild mode for tail-pack layouts

[Gao Xiang]

On Sat, May 09, 2026 at 04:37:43PM +0800, Bingwu Zhang wrote:
> Hi,
> 
> On Monday, May 4, 2026 11:13:23 PM China Standard Time Gao Xiang wrote:
> > Hi xtex,
> > 
> > On 2026/4/30 20:09, xtex wrote:
> > > Hi!
> > > 
> > > In erofs_rebuild_write_blob_index (rebuild.c), only
> > > EROFS_INODE_CHUNK_BASED
> > > and FLAT_PLAIN are implemented, so when generating a metadata index with
> > > rebuild mode, the sources cannot use tail-pack nor inline data layout.
> > > However, disabling tail-packing can lead to great disk-space waste in many
> > > cases, especially when the file-system consists of a lot of small files.
> > > 
> > > Thus I attempted to implement FLAT_INLINE for it, only to realize that the
> > > current chunk entry formats can only represent physical addresses that are
> > > block-aligned while tail-pack extent is not.
> > > 
> > > I wonder what do you think about adding a new chunk entry format? And how
> > > should it be named?
> > > 
> > > I would suggest the following structure:
> > > struct erofs_inode_chunk_index_tp {
> > > 
> > > 	__le16 startblk_hi;	/* starting block number MSB */
> > > 	__le16 device_id;	/* back-end storage id (with bits masked)
> > > 
> > > */
> > > 
> > > 	__le32 startblk_lo;	/* starting block number of this chunk */
> > > 	/* new fields below */
> > > 	__le16 startblk_off;	/* starting block offset */
> > > 	__le16 reserved;
> > > 
> > > } __packed;
> > > The 16b offset should be enough unless we are to support block size > 64K.
> > > The reserved field is added for alignment.
> > 
> > Sorry about the late response.
> > 
> > Thanks for the question.
> > 
> > FLAT_INLINE can be used for index rebuilding, which can work with
> > uniaddr (or mapped_blkaddr) since the blkaddr will be mapped
> > into the relative address based on the blob starting with
> > mapped_blkaddr:
> > 
> > https://erofs.docs.kernel.org/en/latest/ondisk/chunked_format.html#device-ta
> > ble
> > 
> > But I agree the expression in the page above is a bit
> > ambigious through.
> > 
> > Thanks,
> > Gao Xiang
> > 
> > > Best wishes.
> 
> Sorry for the late response and thanks for your answer.
> 
> I am sorry about that I didn't get it.
> In __erofs_map_blocks:
> > map->m_pa = erofs_pos(sbi, startblk);
> and
> > #define erofs_pos(sbi, nr)      ((erofs_off_t)(nr) << (sbi)->blkszbits)
> It seems like that the mapped PA is always block-aligned? However, the last 
> chunk of inline data is not?

Yes, sorry I didn't express explicitly.

I mean the main data except the trailing inline data part
can be remapped into another external blob.

But inline data should be kept with the metadata; otherwise it
won't be called _inline data_ anymore.

Or do you have a case we have to redirect the inline data?

Thanks,
Gao Xiang

> 
> Best wishes.
> 
> -- 
> xtex (a.k.a. Bingwu Zhang) @ Sat, 09 May 2026 08:24:53 +0000
> 
> 
> 
> 

Re: Rebuild mode for tail-pack layouts

[Bingwu Zhang]

On Saturday, May 9, 2026 6:03:15 PM China Standard Time Gao Xiang wrote:
> On Sat, May 09, 2026 at 04:37:43PM +0800, Bingwu Zhang wrote:
> > Hi,
> > 
> > On Monday, May 4, 2026 11:13:23 PM China Standard Time Gao Xiang wrote:
> > > Hi xtex,
> > > 
> > > On 2026/4/30 20:09, xtex wrote:
> > > > Hi!
> > > > 
> > > > In erofs_rebuild_write_blob_index (rebuild.c), only
> > > > EROFS_INODE_CHUNK_BASED
> > > > and FLAT_PLAIN are implemented, so when generating a metadata index
> > > > with
> > > > rebuild mode, the sources cannot use tail-pack nor inline data layout.
> > > > However, disabling tail-packing can lead to great disk-space waste in
> > > > many
> > > > cases, especially when the file-system consists of a lot of small
> > > > files.
> > > > 
> > > > Thus I attempted to implement FLAT_INLINE for it, only to realize that
> > > > the
> > > > current chunk entry formats can only represent physical addresses that
> > > > are
> > > > block-aligned while tail-pack extent is not.
> > > > 
> > > > I wonder what do you think about adding a new chunk entry format? And
> > > > how
> > > > should it be named?
> > > > 
> > > > I would suggest the following structure:
> > > > struct erofs_inode_chunk_index_tp {
> > > > 
> > > > 	__le16 startblk_hi;	/* starting block number MSB */
> > > > 	__le16 device_id;	/* back-end storage id (with bits masked)
> > > > 
> > > > */
> > > > 
> > > > 	__le32 startblk_lo;	/* starting block number of this chunk */
> > > > 	/* new fields below */
> > > > 	__le16 startblk_off;	/* starting block offset */
> > > > 	__le16 reserved;
> > > > 
> > > > } __packed;
> > > > The 16b offset should be enough unless we are to support block size >
> > > > 64K.
> > > > The reserved field is added for alignment.
> > > 
> > > Sorry about the late response.
> > > 
> > > Thanks for the question.
> > > 
> > > FLAT_INLINE can be used for index rebuilding, which can work with
> > > uniaddr (or mapped_blkaddr) since the blkaddr will be mapped
> > > into the relative address based on the blob starting with
> > > mapped_blkaddr:
> > > 
> > > https://erofs.docs.kernel.org/en/latest/ondisk/chunked_format.html#devic
> > > e-ta ble
> > > 
> > > But I agree the expression in the page above is a bit
> > > ambigious through.
> > > 
> > > Thanks,
> > > Gao Xiang
> > > 
> > > > Best wishes.
> > 
> > Sorry for the late response and thanks for your answer.
> > 
> > I am sorry about that I didn't get it.
> > 
> > In __erofs_map_blocks:
> > > map->m_pa = erofs_pos(sbi, startblk);
> > 
> > and
> > 
> > > #define erofs_pos(sbi, nr)      ((erofs_off_t)(nr) << (sbi)->blkszbits)
> > 
> > It seems like that the mapped PA is always block-aligned? However, the
> > last
> > chunk of inline data is not?
> 
> Yes, sorry I didn't express explicitly.
> 
> I mean the main data except the trailing inline data part
> can be remapped into another external blob.
> 
> But inline data should be kept with the metadata; otherwise it
> won't be called _inline data_ anymore.
> 
> Or do you have a case we have to redirect the inline data?
> 
> Thanks,
> Gao Xiang
> 
> > Best wishes.

Ah! Thanks for your explanation. I initially thought that copying inline data 
would increase the size of the metadata. Thanks!

Best wishes.

-- 
xtex (a.k.a. Bingwu Zhang) @ Sat, 09 May 2026 14:27:41 +0000