[Buildroot] [PATCH v2] package/pkg-utils: Prevent per-package syncing stale deps

[Buildroot] [PATCH v2] package/pkg-utils: Prevent per-package syncing stale deps

[John Ernberg]

From: Jonas Blixt <jonas.blixt@actia.se>

From: Jonas Blixt <jonas.blixt@actia.se>

This filters out the dependency inputs when syncing the per-package
target directory to only include 'skeleton' and its dependencies, if
they exist. Since other dependencies are not needed in the target dir
for successful building they can be ignored and depend on the ppd-merge
to populate the files from the other packages, unlike the staging dir
which is untouched here.

This solves the problem where stale files from dependencies will
linger in the per-package target directories and may up in the final
target directory after the ppd-merge, overwriting the fresh ones,
causing unecessarly many rebuild steps to ensure that the latest
version of a dependency is included in the final target directory.

Example:

Package A, containing liba
Package B, depends on liba

If liba receives a bug fix that does not require a rebuild of Package B
Pacckage B target directory will still contain the previous version
of liba from Package A. During the final rsync the Package B target
dir will be rsync'ed after Package A's target dir  to the final target
directory which now will contain the stale version of liba.

For our particular configuration the build time was not notably affected:
Before: 12m4
After: 11m43

Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Jacob Chen <chenhao37@lixiang.com>
Signed-off-by: Jonas Blixt <jonas.blixt@actia.se>
Signed-off-by: John Ernberg <john.ernberg@actia.se>

---

Changes since v1: https://lore.kernel.org/buildroot/20240801115712.1458767-1-jonas.blixt@actia.se/
 - Rework diff to be a bit more digestiable
 - Rework commit message to attempt to contain a bit more information and clarity

We sent this patch back in August and it didn't get any feedback.

Since this is a bit of a tricky matter I reworked the commit message a
bit to try to clarify the problem further to make review easier.

The issue has been brought up a few times on the list before:
https://lore.kernel.org/buildroot/tencent_7A5A407C574E45921E432385B01446CC4506@qq.com/
https://lore.kernel.org/buildroot/20231110141242.3796161-1-john.ernberg@actia.se/
https://lore.kernel.org/buildroot/20191212130253.2cb543ba@windsurf/
---
 package/pkg-utils.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index a0dbeac18e..65234ea7f6 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -237,10 +237,18 @@ endef
 # argument, so that this function can be used to prepare with
 # different set of dependencies (download, extract, configure, etc.)
 #
+# The per-package sync of target is filtered to only sync skeleton
+# and the skeletons dependencies when they exist. This prevents stale
+# copies of dependencies in the per-package target directory and thus
+# prevents stale files from propagating to the final target directory
+# during rebuilds of packages.
+#
 # $1: space-separated list of packages to rsync from
 define prepare-per-package-directory
 	$(call per-package-rsync,$(1),host,$(HOST_DIR),hardlink)
-	$(call per-package-rsync,$(1),target,$(TARGET_DIR),hardlink)
+	$(call per-package-rsync,\
+		$(filter $(SKELETON_FINAL_RECURSIVE_DEPENDENCIES) skeleton, $(1)),\
+		target,$(TARGET_DIR),hardlink)
 endef
 
 # Ensure files like .la, .pc, .pri, .cmake, and so on, point to the
-- 
2.47.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2] package/pkg-utils: Prevent per-package syncing stale deps

[Arnout Vandecappelle via buildroot]

  Hi John, Jonas,

On 27/12/2024 17:06, John Ernberg wrote:
> From: Jonas Blixt <jonas.blixt@actia.se>
> 
> From: Jonas Blixt <jonas.blixt@actia.se>
> 
> This filters out the dependency inputs when syncing the per-package
> target directory to only include 'skeleton' and its dependencies, if
> they exist. Since other dependencies are not needed in the target dir
> for successful building they can be ignored and depend on the ppd-merge
> to populate the files from the other packages, unlike the staging dir
> which is untouched here.

  I may be missing something, but doesn't this break the combination of busybox 
with e.g. attr? The busybox install script checks if the file already exists and 
does not create the symlink when it does. busybox depends on attr (and dozens of 
other packages) for exactly that reason.

  There are other packages as well that rely on the presence of something in the 
target directory.

> This solves the problem where stale files from dependencies will
> linger in the per-package target directories and may up in the final
> target directory after the ppd-merge, overwriting the fresh ones,
> causing unecessarly many rebuild steps to ensure that the latest
> version of a dependency is included in the final target directory.

  Yep, we're aware of this problem, and haven't found a solution :-)

  Regards,
  Arnout

> 
> Example:
> 
> Package A, containing liba
> Package B, depends on liba
> 
> If liba receives a bug fix that does not require a rebuild of Package B
> Pacckage B target directory will still contain the previous version
> of liba from Package A. During the final rsync the Package B target
> dir will be rsync'ed after Package A's target dir  to the final target
> directory which now will contain the stale version of liba.
> 
> For our particular configuration the build time was not notably affected:
> Before: 12m4
> After: 11m43
> 
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Jacob Chen <chenhao37@lixiang.com>
> Signed-off-by: Jonas Blixt <jonas.blixt@actia.se>
> Signed-off-by: John Ernberg <john.ernberg@actia.se>
> 
> ---
> 
> Changes since v1: https://lore.kernel.org/buildroot/20240801115712.1458767-1-jonas.blixt@actia.se/
>   - Rework diff to be a bit more digestiable
>   - Rework commit message to attempt to contain a bit more information and clarity
> 
> We sent this patch back in August and it didn't get any feedback.
> 
> Since this is a bit of a tricky matter I reworked the commit message a
> bit to try to clarify the problem further to make review easier.
> 
> The issue has been brought up a few times on the list before:
> https://lore.kernel.org/buildroot/tencent_7A5A407C574E45921E432385B01446CC4506@qq.com/
> https://lore.kernel.org/buildroot/20231110141242.3796161-1-john.ernberg@actia.se/
> https://lore.kernel.org/buildroot/20191212130253.2cb543ba@windsurf/
> ---
>   package/pkg-utils.mk | 10 +++++++++-
>   1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index a0dbeac18e..65234ea7f6 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -237,10 +237,18 @@ endef
>   # argument, so that this function can be used to prepare with
>   # different set of dependencies (download, extract, configure, etc.)
>   #
> +# The per-package sync of target is filtered to only sync skeleton
> +# and the skeletons dependencies when they exist. This prevents stale
> +# copies of dependencies in the per-package target directory and thus
> +# prevents stale files from propagating to the final target directory
> +# during rebuilds of packages.
> +#
>   # $1: space-separated list of packages to rsync from
>   define prepare-per-package-directory
>   	$(call per-package-rsync,$(1),host,$(HOST_DIR),hardlink)
> -	$(call per-package-rsync,$(1),target,$(TARGET_DIR),hardlink)
> +	$(call per-package-rsync,\
> +		$(filter $(SKELETON_FINAL_RECURSIVE_DEPENDENCIES) skeleton, $(1)),\
> +		target,$(TARGET_DIR),hardlink)
>   endef
>   
>   # Ensure files like .la, .pc, .pri, .cmake, and so on, point to the

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2] package/pkg-utils: Prevent per-package syncing stale deps

[John Ernberg via buildroot]

Hi Arnout,

On 2/4/26 5:34 PM, Arnout Vandecappelle wrote:
>   Hi John, Jonas,
> 
> On 27/12/2024 17:06, John Ernberg wrote:
>> From: Jonas Blixt <jonas.blixt@actia.se>
>>
>> From: Jonas Blixt <jonas.blixt@actia.se>
>>
>> This filters out the dependency inputs when syncing the per-package
>> target directory to only include 'skeleton' and its dependencies, if
>> they exist. Since other dependencies are not needed in the target dir
>> for successful building they can be ignored and depend on the ppd-merge
>> to populate the files from the other packages, unlike the staging dir
>> which is untouched here.
> 
>   I may be missing something, but doesn't this break the combination of 
> busybox
> with e.g. attr? The busybox install script checks if the file already 
> exists and
> does not create the symlink when it does. busybox depends on attr (and 
> dozens of
> other packages) for exactly that reason.
> 
>   There are other packages as well that rely on the presence of 
> something in the
> target directory.

We didn't notice any breakages of busybox, but we also only use about
10 utils out of busybox (none of them are part of the conflict
packages on our config).

It did not even cross my mind packages might do that. Thank you for
bringing it up!

(I haven't inventoried which other packages yet, will do if it becomes
necessary after below points considered)

I currently see 2 ways to approach this problem, please let me know
what you think about them.

1) We move such steps (such as generating the busybox applet links)
    to the TARGET_FINALIZE_HOOKS stage. This will need us to install
    more files into target/ that will need to be ignored later like
    the TARGET_DIR_WARNING_FILE is ignored.

    Packages may need to be able to define additional files themselves
    that need to be skipped in rootfs generation.

    Looking at busybox it might not be exactly trivial but it should
    be doable.

2) Introduce <pkg>-rebuild-rdepends and <pkg>-reconfigure-rdepends,
    which trigger the appropriate rebuild of all the rdepends.

    This would be the same thing we do today to work around the
    problem (make <pkg>-rebuild && make <rdep1>-reconfigure ...) but
    automatically from inside buildroot.

> 
>> This solves the problem where stale files from dependencies will
>> linger in the per-package target directories and may up in the final
>> target directory after the ppd-merge, overwriting the fresh ones,
>> causing unecessarly many rebuild steps to ensure that the latest
>> version of a dependency is included in the final target directory.
> 
>   Yep, we're aware of this problem, and haven't found a solution :-)

It is indeed a tricky problem, it is bothering me enough though that
I'm willing to work on it as much as I can.

Best regards // John Ernberg

> 
>   Regards,
>   Arnout
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2] package/pkg-utils: Prevent per-package syncing stale deps

[Arnout Vandecappelle via buildroot]

  Hi John,

  [Adding Yann for hopefully more insights.]

On 27/02/2026 17:37, John Ernberg wrote:
> Hi Arnout,
> 
> On 2/4/26 5:34 PM, Arnout Vandecappelle wrote:
>>    Hi John, Jonas,
>>
>> On 27/12/2024 17:06, John Ernberg wrote:
>>> From: Jonas Blixt <jonas.blixt@actia.se>
>>>
>>> From: Jonas Blixt <jonas.blixt@actia.se>
>>>
>>> This filters out the dependency inputs when syncing the per-package
>>> target directory to only include 'skeleton' and its dependencies, if
>>> they exist. Since other dependencies are not needed in the target dir
>>> for successful building they can be ignored and depend on the ppd-merge
>>> to populate the files from the other packages, unlike the staging dir
>>> which is untouched here.
>>
>>    I may be missing something, but doesn't this break the combination of
>> busybox
>> with e.g. attr? The busybox install script checks if the file already
>> exists and
>> does not create the symlink when it does. busybox depends on attr (and
>> dozens of
>> other packages) for exactly that reason.
>>
>>    There are other packages as well that rely on the presence of
>> something in the
>> target directory.
> 
> We didn't notice any breakages of busybox, but we also only use about
> 10 utils out of busybox (none of them are part of the conflict
> packages on our config).
> 
> It did not even cross my mind packages might do that. Thank you for
> bringing it up!

  The annoying thing is that we don't even _know_ which packages do that.

  Solving this issue actually has the nice side effect that we can have a check 
that no two packages install the same file. We had a patch adding such a check 
in the beginning of PPD, but we had to drop it because it triggered too many 
failures that were not easy to solve.

> (I haven't inventoried which other packages yet, will do if it becomes
> necessary after below points considered)
> 
> I currently see 2 ways to approach this problem, please let me know
> what you think about them.
> 
> 1) We move such steps (such as generating the busybox applet links)
>      to the TARGET_FINALIZE_HOOKS stage. This will need us to install
>      more files into target/ that will need to be ignored later like
>      the TARGET_DIR_WARNING_FILE is ignored.

  I'm not sure what exactly you mean here, but I guess the idea is "do something 
explicit for each package that causes a conflict". I definitely like that idea - 
in general in Buildroot we like to do things explicitly rather than magically.

  Specifically for busybox, I actually think the approach should be to _remove_ 
the symlinks that would lead to conflict (and also remove the dependencies in 
busybox.mk). So something like:

ifeq ($(BR2_PACKAGE_ATTR),y)
BUSYBOX_RM_FILES += /usr/bin/attr /usr/bin/getfattr /usr/bin/setfattr
endif

define BUSYBOX_INSTALL_TARGET
	...
	$(RM) $(addprefix $(TARGET_DIR)/,$(BUSYBOX_RM_FILES))
endef

This is still a little bit un-nice because it could require changes in the 
busybox package when you update attr, but in practice that will happen extremely 
rarely.


>      Packages may need to be able to define additional files themselves
>      that need to be skipped in rootfs generation.

  It's not just skipping though. It happens less and less now, but there are 
also packages that _modify_ an existing file at installation time. The 
traditional example of this is updating the "directory" when installing info 
files (which we don't actually care about because documentation gets removed 
anyway, but it's an easy to understand example).

  Anyway, those as well can be handled on a case-by-case basis in 
_TARGET_FINALIZE_HOOKS.


>      Looking at busybox it might not be exactly trivial but it should
>      be doable.
> 
> 2) Introduce <pkg>-rebuild-rdepends and <pkg>-reconfigure-rdepends,
>      which trigger the appropriate rebuild of all the rdepends.
> 
>      This would be the same thing we do today to work around the
>      problem (make <pkg>-rebuild && make <rdep1>-reconfigure ...) but
>      automatically from inside buildroot.
> 
>>
>>> This solves the problem where stale files from dependencies will
>>> linger in the per-package target directories and may up in the final
>>> target directory after the ppd-merge, overwriting the fresh ones,
>>> causing unecessarly many rebuild steps to ensure that the latest
>>> version of a dependency is included in the final target directory.
>>
>>    Yep, we're aware of this problem, and haven't found a solution :-)
> 
> It is indeed a tricky problem, it is bothering me enough though that
> I'm willing to work on it as much as I can.

  I'm pretty sure there were other issues as well. I've tried to dig into 
reports from previous developer meetings [1][2] but couldn't find anything 
concrete. Perhaps Yann still remembers something? I do remember we looked at it 
in depth in the hackaton at Yann's place, but of course I don't remember what 
the problem was that we couldn't resolve at the time...

  Regards,
  Arnout

> 
> Best regards // John Ernberg
> 
>>
>>    Regards,
>>    Arnout


[1] https://elinux.org/Buildroot:DeveloperDaysELCE2017#Top-level_parallel_build
[2] 
https://elinux.org/Buildroot:DeveloperDaysFOSDEM2020#Status_of_top-level_parallel_build_support.2C_where_do_we_go_from_there


_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2] package/pkg-utils: Prevent per-package syncing stale deps

[John Ernberg via buildroot]

Hi Arnout,

On 3/3/26 10:29 PM, Arnout Vandecappelle wrote:
>   Hi John,
> 
>   [Adding Yann for hopefully more insights.]
> 
> On 27/02/2026 17:37, John Ernberg wrote:
>> Hi Arnout,
>>
>> On 2/4/26 5:34 PM, Arnout Vandecappelle wrote:
>>>    Hi John, Jonas,
>>>
>>> On 27/12/2024 17:06, John Ernberg wrote:
>>>> From: Jonas Blixt <jonas.blixt@actia.se>
>>>>
>>>> From: Jonas Blixt <jonas.blixt@actia.se>
>>>>
>>>> This filters out the dependency inputs when syncing the per-package
>>>> target directory to only include 'skeleton' and its dependencies, if
>>>> they exist. Since other dependencies are not needed in the target dir
>>>> for successful building they can be ignored and depend on the ppd-merge
>>>> to populate the files from the other packages, unlike the staging dir
>>>> which is untouched here.
>>>
>>>    I may be missing something, but doesn't this break the combination of
>>> busybox
>>> with e.g. attr? The busybox install script checks if the file already
>>> exists and
>>> does not create the symlink when it does. busybox depends on attr (and
>>> dozens of
>>> other packages) for exactly that reason.
>>>
>>>    There are other packages as well that rely on the presence of
>>> something in the
>>> target directory.
>>
>> We didn't notice any breakages of busybox, but we also only use about
>> 10 utils out of busybox (none of them are part of the conflict
>> packages on our config).
>>
>> It did not even cross my mind packages might do that. Thank you for
>> bringing it up!
> 
>   The annoying thing is that we don't even _know_ which packages do that.
> 
>   Solving this issue actually has the nice side effect that we can have 
> a check
> that no two packages install the same file. We had a patch adding such a 
> check
> in the beginning of PPD, but we had to drop it because it triggered too 
> many
> failures that were not easy to solve.

Thanks for the hint about this script, I'll find it and see if it helps 
me locally during testing.

> 
>> (I haven't inventoried which other packages yet, will do if it becomes
>> necessary after below points considered)
>>
>> I currently see 2 ways to approach this problem, please let me know
>> what you think about them.
>>
>> 1) We move such steps (such as generating the busybox applet links)
>>      to the TARGET_FINALIZE_HOOKS stage. This will need us to install
>>      more files into target/ that will need to be ignored later like
>>      the TARGET_DIR_WARNING_FILE is ignored.
> 
>   I'm not sure what exactly you mean here, but I guess the idea is "do 
> something
> explicit for each package that causes a conflict". I definitely like 
> that idea -
> in general in Buildroot we like to do things explicitly rather than 
> magically.
> 
>   Specifically for busybox, I actually think the approach should be to 
> _remove_
> the symlinks that would lead to conflict (and also remove the 
> dependencies in
> busybox.mk). So something like:
> 
> ifeq ($(BR2_PACKAGE_ATTR),y)
> BUSYBOX_RM_FILES += /usr/bin/attr /usr/bin/getfattr /usr/bin/setfattr
> endif
> 
> define BUSYBOX_INSTALL_TARGET
>         ...
>         $(RM) $(addprefix $(TARGET_DIR)/,$(BUSYBOX_RM_FILES))
> endef

That's definitely a nicer approach than I came up with. I'll test this 
out first.

> 
> This is still a little bit un-nice because it could require changes in the
> busybox package when you update attr, but in practice that will happen 
> extremely
> rarely.
> 
> 
>>      Packages may need to be able to define additional files themselves
>>      that need to be skipped in rootfs generation.
> 
>   It's not just skipping though. It happens less and less now, but there 
> are
> also packages that _modify_ an existing file at installation time. The
> traditional example of this is updating the "directory" when installing 
> info
> files (which we don't actually care about because documentation gets 
> removed
> anyway, but it's an easy to understand example).
> 
>   Anyway, those as well can be handled on a case-by-case basis in
> _TARGET_FINALIZE_HOOKS.
> 
> 
>>      Looking at busybox it might not be exactly trivial but it should
>>      be doable.
>>
>> 2) Introduce <pkg>-rebuild-rdepends and <pkg>-reconfigure-rdepends,
>>      which trigger the appropriate rebuild of all the rdepends.
>>
>>      This would be the same thing we do today to work around the
>>      problem (make <pkg>-rebuild && make <rdep1>-reconfigure ...) but
>>      automatically from inside buildroot.
>>
>>>
>>>> This solves the problem where stale files from dependencies will
>>>> linger in the per-package target directories and may up in the final
>>>> target directory after the ppd-merge, overwriting the fresh ones,
>>>> causing unecessarly many rebuild steps to ensure that the latest
>>>> version of a dependency is included in the final target directory.
>>>
>>>    Yep, we're aware of this problem, and haven't found a solution :-)
>>
>> It is indeed a tricky problem, it is bothering me enough though that
>> I'm willing to work on it as much as I can.
> 
>   I'm pretty sure there were other issues as well. I've tried to dig into
> reports from previous developer meetings [1][2] but couldn't find anything
> concrete. Perhaps Yann still remembers something? I do remember we 
> looked at it
> in depth in the hackaton at Yann's place, but of course I don't remember 
> what
> the problem was that we couldn't resolve at the time...

Thanks for those links. They looks like a really good starting point.
I've already run into the dependency issue with overwriting files, and 
have some ideas for solving it once this syncing issue is resolved.

Best regards // John Ernberg

> 
>   Regards,
>   Arnout
> 
>>
>> Best regards // John Ernberg
>>
>>>
>>>    Regards,
>>>    Arnout
> 
> 
> [1] https://elinux.org/Buildroot:DeveloperDaysELCE2017#Top- 
> level_parallel_build
> [2]
> https://elinux.org/Buildroot:DeveloperDaysFOSDEM2020#Status_of_top- 
> level_parallel_build_support.2C_where_do_we_go_from_there
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot