From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BCAEC432BE for ; Tue, 17 Aug 2021 08:41:42 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5377E60FBF for ; Tue, 17 Aug 2021 08:41:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5377E60FBF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=bootlin.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=busybox.net Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 27CC680F05; Tue, 17 Aug 2021 08:41:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aiW5IJDZVLfk; Tue, 17 Aug 2021 08:41:37 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp1.osuosl.org (Postfix) with ESMTP id CCA9D81066; Tue, 17 Aug 2021 08:41:36 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 22AC71BF2AE for ; Tue, 17 Aug 2021 08:40:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 104E0400FC for ; Tue, 17 Aug 2021 08:40:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pMjQxkxheoan for ; Tue, 17 Aug 2021 08:39:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp2.osuosl.org (Postfix) with ESMTPS id 38BA240256 for ; Tue, 17 Aug 2021 08:39:56 +0000 (UTC) Received: (Authenticated sender: herve.codina@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPA id D710F1BF203; Tue, 17 Aug 2021 08:39:53 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Tue, 17 Aug 2021 10:39:22 +0200 Message-Id: <20210817083930.3718711-9-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210817083930.3718711-1-herve.codina@bootlin.com> References: <20210817083930.3718711-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 08/16] package/pkg-generic.mk: generate final rsync exclude file list X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Herve Codina , Naumann Andreas , Peter Seiderer , Julien Corjon , Thomas Petazzoni , "Yann E . MORIN" , Ricardo Martincoski Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR. In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync. Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie /sysroot) is filtered out. The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites. Using an empty directory for per-package installation directory would be the simplest way to find what a package installs. However, as it has been discussed in the past, this is fraught with unworkable issues. For example, some paths may be hard-coded at configure time and/or build time, and thus the package would still install in the original stagin we presented it with (for target/, this is not an isue, because target/ is never looked at during configure or build, only at install time). Or a package installation process just assumes that the directory structure exists (bad, but eh...). Signed-off-by: Herve Codina --- Changes v1 to v2: - Added a note in the commit message related to why an empty directory is not used to find what a package installs. Changes v2 to v3: None package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 414862e267..4226cfe0dc 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -168,6 +168,30 @@ define pkg_size_after rm -f $($(PKG)_DIR)/.files-list$(2).after endef +# Functions to collect final rsync exclusion files + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_before + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after + LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after +endef + define check_bin_arch support/scripts/check-bin-arch -p $($(PKG)_NAME) \ -l $($(PKG)_DIR)/.files-list.txt \ @@ -290,6 +314,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call MESSAGE,"Configuring") $(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call pkg_final_rsync_before,$(TARGET_DIR)) + @$(call pkg_final_rsync_before,$(HOST_DIR),-host) @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) @@ -422,6 +448,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_final_rsync_after,$(TARGET_DIR)) + @$(call pkg_final_rsync_after,$(HOST_DIR),-host) @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ -- 2.31.1 _______________________________________________ buildroot mailing list buildroot@busybox.net http://lists.busybox.net/mailman/listinfo/buildroot