Re: [Buildroot] [PATCH 1/1] ppd-merge: speed up per-package-rsync

[Herve Codina via buildroot <buildroot@buildroot.org>]

Hi Yann,

On Tue, 28 Nov 2023 22:07:17 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Brandon, All,
> 
> +Thomas for the intial work on this topic
> 
> On 2023-11-27 22:41 +0000, Brandon Maier spake thusly:
> > The per-package-rsync stage can add a significant amount of time to
> > builds. They can also be annoying as the slowest rsyncs, the final
> > target-finalize and host-finalize stages, run on `make all` which we use
> > frequently during development.
> > 
> > The per-package-rsync is slow because it launches a new rsync for each
> > source tree, and each rsync must rescan the destination directory and
> > potentially overwrite files multiple times. So instead we manually walk
> > the source trees to find only the files that should be written to the
> > destination, then feed that to a single instance of rsync.  
> 
> Sorry, I haven't looked into the script in details, but why can't we
> just do something like:
> 
>     rsync -a \
>         --hard-links \
>         $(if $(filter hardlink,$(4)), \
>             --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/, \
>             $(if $(filter copy,$(4)), \
>                 $(empty), \
>                 $(error per-package-rsync can only "copy" or "hardlink", not "$(4)") \
>             ) \
>         ) \
>         $(patsubst %, $(PER_PACKAGE_DIR)/%/$(2)/, $(1)) \
>         $(3)
> 
> rsync can take multiple sources, and is usually pretty fast, so I would
> very much prefer that we do not invent our own tooling if a single rsync
> can do the job.
> 
> Also, the copy situation is now only ever needed when we aggregate the
> final target and host directories, while the hard-linking case is used
> when assembling each per-package; so there is no point in benchmarking
> the copy mode for the per-package preparation, or the hard-linking for
> the final target and host assembling.
> 
> Hervé, Thomas, do you remember if we ever addressed using a single rsync
> rather than multiple ones in your previous work on the topic, and if so,
> why we did not use that?

I think we did only multiple rsync calls.
We had in mind the file overwrite detection and so we did the operation
per package keeping the $(foreach pkg,$(1), ...) loop.

And why we did not use that ?
First because I didn't think about it...
and second because we were in the strategy to check overwrites per package.

Best regards,
Hervé
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot