Re: [Buildroot] [PATCH 2/2] package/pkg-utils: teach per-package-rsync to copy or hardlink dest

[Herve Codina via buildroot <buildroot@buildroot.org>]

Yann, All,

On Wed, 18 Oct 2023 18:18:59 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Hervé, All,
> 
> On 2023-10-18 17:38 +0200, Herve Codina via buildroot spake thusly:
> > On Wed, 18 Oct 2023 17:20:47 +0200
> > "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:  
> > > Hervé, All  
> > > > > --- a/package/pkg-utils.mk
> > > > > +++ b/package/pkg-utils.mk
> > > > > @@ -214,10 +214,19 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
> > > > >  # $1: space-separated list of packages to rsync from
> > > > >  # $2: 'host' or 'target'
> > > > >  # $3: destination directory
> > > > > +# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest
> > > > >  define per-package-rsync
> > > > >  	mkdir -p $(3)
> > > > >  	$(foreach pkg,$(1),\
> > > > > -		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
> > > > > +		rsync -a \
> > > > > +			--hard-links \    
> > > > You preserve hard links (--hard-links) in all cases (copy and hardlink).
> > > > In case of copy, is it correct to preserve hard links ?    
> > > Yes, --hard-links is:
> > > 
> > >     This tells rsync to look for hard-linked files in the source and
> > >     link together the corresponding files on the destination.  
> > In case of 'copy', you keep hard-links already present in source and so the
> > destination (final host/target dirs) is not a full copy.  
> 
> Ah, it does not hard-link something in dest into src. The hardlinks are
> only in the destination. See below...
> 
> > Some hard-links can be present and post-build scripts can still corrupt some
> > per-package sources.
> > 
> > I don't understand why keeping hard-links on this last copy is needed and I
> > probably miss something...  
> 
> That is because we can have hard-links in host or target. For example,
> git will install a lot of hard-links in /usr/libexec/git-core/
> (excerpt):
> 
>     $ ls -li per-package/git/target/usr/libexec/git-core/
>     9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-var
>     9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-commit
>     9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-pack
>     9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-tag
>     9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-version
> 
>     $ ls -li target/usr/libexec/git-core/
>     9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-var
>     9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-commit
>     9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-pack
>     9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-tag
>     9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-version
> 
> As you can see, in git's PPD, git tools are hard-links, and in target/
> they also are hard-links. But the inode is different, so the hard-links
> in target/ are different from the hard-links in the PPD.
> 
> However, in all the PPDs of packages that have git as a dependency, the
> hard-links would all be to the same inode, 9588770.
> 
> (The size delta is because the files in target/ are stripped.)
> 
> So, in case of "copy", this is actually a copy, that keeps existing
> hard-links in the source, as new hard-links in the destination. Without
> --hard-links, the files in target/usr/libexec/git-core/ would all have a
> different inode, as they would be different files, and that would
> tremendously increase the filesystem size (squashfs would notice, and
> would store the content only once, but would still create as many inodes
> as needed, though).
> 

Thanks a lot for these explanations!
It is clearer in my mind now.

Maybe, in the commit log:
--- 8< ---
rsync --hard-links option allows to keep existing hard-links in the source
as new hard-links in the destination ("copy" of hard-links).
Having this hard-links "copy" contributes to the directories size
reduction.
--- 8< --- 

With all that said:
Reviewed-by: Herve Codina <herve.codina@bootlin.com>

Best regards,
Hervé
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot