From: Yann E. MORIN <yann.morin.1998@free.fr>
To: buildroot@busybox.net
Subject: [Buildroot] [PATCH 08/11] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR
Date: Sat, 25 Jul 2020 09:13:40 +0200 [thread overview]
Message-ID: <20200725071340.GL2296@scaer> (raw)
In-Reply-To: <87k0ys16gw.fsf@dell.be.48ers.dk>
Peter, All,
On 2020-07-25 07:58 +0200, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> > Thomas, All,
> > On 2020-05-01 23:23 +0200, Yann E. MORIN spake thusly:
> >> On 2020-04-30 11:52 +0200, Thomas Petazzoni spake thusly:
> > [--SNIP--]
> >> > +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
> >> > +# $(1): base directory to search in
> >> > +# $(2): suffix of file (optional)
> >> > +define pkg_detect_overwrite_before
> >> > + cd $(1); \
> >> > + LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5
>
> > Note that this is very slow: it spawns a new md5sum process for each
> > file it encounters.
>
> > There is a better solution, though:
>
> > find $(1) -type f -print0 |xargs -0 md5sum > $($(PKG)_DIR)/.files$(2).md5
>
> And perhaps add -r / --no-run-if-empty to xargs for good measure, even
Yes.
> if it is unlikely to be completely empty.
It would be for the skeleton and the host skeleton.
> > Your original code took 11s (the second time, with a hot VFS cache),
> > while my proposal got it down to 2s (again, hot VFS cache).
>
> > We could also try to parallelise the job:
>
> > find $(1) -type f -print0 |xargs -P $(PARALLEL_JOBS) -0 md5sum > $($(PKG)_DIR)/.files$(2).md5
>
> Will that not scramble the order of the hashes in the list?
But do we even care ? The order returned by find is not even guaranteed,
as this depends on the order of the entries in the directory anyway,
so...
Also, note that xargs will not run an md5sum process for each file, but
will coalesce multiple calls into a single one.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
next prev parent reply other threads:[~2020-07-25 7:13 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-30 9:52 [Buildroot] [PATCH 00/11] Overwritten file detection, improvements to file listing logic Thomas Petazzoni
2020-04-30 9:52 ` [Buildroot] [PATCH 01/11] package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_after Thomas Petazzoni
2020-04-30 9:52 ` [Buildroot] [PATCH 02/11] package/pkg-generic.mk: drop useless $(1) argument in step_pkg_size_{before, after} Thomas Petazzoni
2020-04-30 9:52 ` [Buildroot] [PATCH 03/11] package/pkg-generic.mk: introduce final 'install' step Thomas Petazzoni
2020-04-30 9:52 ` [Buildroot] [PATCH 04/11] package/pkg-generic.mk: create directories upfront in the configure step Thomas Petazzoni
2020-04-30 9:52 ` [Buildroot] [PATCH 05/11] package/pkg-generic.mk: rework pkg_size logic with the "installed" step Thomas Petazzoni
2020-05-01 20:44 ` Yann E. MORIN
2020-05-02 9:52 ` Thomas Petazzoni
2020-04-30 9:52 ` [Buildroot] [PATCH 06/11] package/pkg-generic.mk: exclude the staging sub-directory Thomas Petazzoni
2020-05-01 20:57 ` Yann E. MORIN
2020-04-30 9:52 ` [Buildroot] [PATCH 07/11] package/pkg-generic.mk: move pkg_size_{before, after} and check_bin_arch functions Thomas Petazzoni
2020-05-01 21:01 ` Yann E. MORIN
2020-04-30 9:52 ` [Buildroot] [PATCH 08/11] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR Thomas Petazzoni
2020-05-01 21:23 ` Yann E. MORIN
2020-07-24 19:53 ` Yann E. MORIN
2020-07-25 5:58 ` Peter Korsgaard
2020-07-25 7:13 ` Yann E. MORIN [this message]
2020-07-25 8:12 ` Peter Korsgaard
2020-04-30 9:52 ` [Buildroot] [PATCH 09/11] support/testing/infra: add log_file_path() function Thomas Petazzoni
2020-07-24 20:11 ` Yann E. MORIN
2020-04-30 9:52 ` [Buildroot] [PATCH 10/11] support/testing/tests: add test for check_bin_arch Thomas Petazzoni
2020-07-25 11:59 ` Yann E. MORIN
2020-04-30 9:52 ` [Buildroot] [PATCH 11/11] support/testing/tests: add test for file overwrite detection Thomas Petazzoni
2020-07-25 12:05 ` Yann E. MORIN
2020-07-23 20:54 ` [Buildroot] [PATCH 00/11] Overwritten file detection, improvements to file listing logic Yann E. MORIN
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200725071340.GL2296@scaer \
--to=yann.morin.1998@free.fr \
--cc=buildroot@busybox.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox