Re: [PATCH] erofs-utils: get rid of inode->i_srcpath

[胡玮文 <sehuww@mail.scut.edu.cn>]

On Mon, Apr 05, 2021 at 07:35:25PM +0800, Gao Xiang wrote:
> Hi Weiwen,
> 
> On Mon, Apr 05, 2021 at 05:49:50PM +0800, Hu Weiwen wrote:
> > This cut the memory usage by half.
> > 
> > Signed-off-by: Hu Weiwen <sehuww@mail.scut.edu.cn>
> > ---
> 
> I have to hold on this patch for now, 3 main reasons here:
>  1) I'd like to apply big pcluster first, which is a main new feature
>     for erofs-utils 1.3;

OK

>  2) even though it saves memory usage (and not sure how much memory
>     you need to save), I think srcpath is more useful for debugging
>     and file tracing; Also, I think we could flush inode in advance to
>     save memory usage (e.g. by using bflush()) as well.

I think flush in advance is a good idea. But it needs more investigation.
Ideally, I want to make the space complexity O(1), so that we can pack an
arbitrarily large number of files. It's better not to save every seen inode
structure in memory.

If I remember it correctly, when packing a dataset consisting of ~5m files,
mkfs.erofs takes 30-40GiB memory. I think this can hinder some use cases.

As for debugging, maybe we can create a routine that travels up the directory
tree to rebuild the path when needed? (Just like `pwd'.) Or, for mkfs, we can
just pass around a pointer to the current working path.

Thanks,
Hu Weiwen

>  3) if src_path is too large, how about malloc() it as the first step,
>     even that, I'd like to apply it after erofs-utils 1.3 is out...
> 
> Thanks,
> Gao Xiang