[Buildroot] [PATCH] package/linux-tools: Fix parallel compilation

[Buildroot] [PATCH] package/linux-tools: Fix parallel compilation

[Charlie Jenkins]

linux-tools opportunistically set linux as only a patch dependency. This
unfortunately introduces a race condition in the shared linux scripts
when using PER_PACKAGE_DIRECTORIES and using top level parallelism.  The
race manifests as the error "/bin/sh: 1: scripts/basic/fixdep:
Permission denied". This happens when the linux package and the
linux-tools package are being compiled in parallel. During the setup,
each of these tools ends up compiling scripts/basic/fixdep. When the
timing is correct, one of these packages will try to use fixdep while
the other is compiling it, resulting in fixdep being briefly not
available.

To fix this, set linux as a hard dependency instead of a patch
dependency.

Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
---
While the timing of this seems like it should very rarely be triggered,
on my build server I hit this issue nearly 100% of the time.
---
 package/linux-tools/linux-tools.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/linux-tools/linux-tools.mk b/package/linux-tools/linux-tools.mk
index 81b1fbc17605d6f003558d113143b64b5af6714d..a5d4a66f761fc44b156a92edaaf21774529f52d5 100644
--- a/package/linux-tools/linux-tools.mk
+++ b/package/linux-tools/linux-tools.mk
@@ -20,7 +20,7 @@
 include $(sort $(wildcard package/linux-tools/*.mk.in))
 
 # We only need the kernel to be extracted, not actually built
-LINUX_TOOLS_PATCH_DEPENDENCIES = linux
+LINUX_TOOLS_DEPENDENCIES = linux
 
 # Install Linux kernel tools in the staging directory since some tools
 # may install shared libraries and headers (e.g. cpupower).

---
base-commit: 7336c5fa93a896130511d661f0d42359a046ec9a
change-id: 20250409-fix_linux_tools_parallel-2323829caf82
-- 
- Charlie

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/linux-tools: Fix parallel compilation

[Thomas Petazzoni via buildroot]

Hello,

I'm adding Yann in Cc, as he wrote a good chunk of the linux-tools
stuff.

On Wed, 09 Apr 2025 01:51:00 -0700
Charlie Jenkins <charlie@rivosinc.com> wrote:

> linux-tools opportunistically set linux as only a patch dependency. This
> unfortunately introduces a race condition in the shared linux scripts
> when using PER_PACKAGE_DIRECTORIES and using top level parallelism.  The
> race manifests as the error "/bin/sh: 1: scripts/basic/fixdep:
> Permission denied". This happens when the linux package and the
> linux-tools package are being compiled in parallel. During the setup,
> each of these tools ends up compiling scripts/basic/fixdep. When the
> timing is correct, one of these packages will try to use fixdep while
> the other is compiling it, resulting in fixdep being briefly not
> available.
> 
> To fix this, set linux as a hard dependency instead of a patch
> dependency.
> 
> Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
> ---
> While the timing of this seems like it should very rarely be triggered,
> on my build server I hit this issue nearly 100% of the time.
> ---
>  package/linux-tools/linux-tools.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/linux-tools/linux-tools.mk b/package/linux-tools/linux-tools.mk
> index 81b1fbc17605d6f003558d113143b64b5af6714d..a5d4a66f761fc44b156a92edaaf21774529f52d5 100644
> --- a/package/linux-tools/linux-tools.mk
> +++ b/package/linux-tools/linux-tools.mk
> @@ -20,7 +20,7 @@
>  include $(sort $(wildcard package/linux-tools/*.mk.in))
>  
>  # We only need the kernel to be extracted, not actually built

This comment then needs to be updated. Maybe it can be kept, but
extended with why we're using a normal dependency, despite the fact
that we only need the kernel to be extracted.

> -LINUX_TOOLS_PATCH_DEPENDENCIES = linux
> +LINUX_TOOLS_DEPENDENCIES = linux

But otherwise, unless I'm missing something, it looks like a reasonably
solution to the problem.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/linux-tools: Fix parallel compilation

[Yann E. MORIN]

Charlie, All,

On 2025-04-19 23:33 +0200, Thomas Petazzoni spake thusly:
> On Wed, 09 Apr 2025 01:51:00 -0700
> Charlie Jenkins <charlie@rivosinc.com> wrote:
> > linux-tools opportunistically set linux as only a patch dependency. This
> > unfortunately introduces a race condition in the shared linux scripts
> > when using PER_PACKAGE_DIRECTORIES and using top level parallelism.  The
> > race manifests as the error "/bin/sh: 1: scripts/basic/fixdep:
> > Permission denied". This happens when the linux package and the
> > linux-tools package are being compiled in parallel. During the setup,
> > each of these tools ends up compiling scripts/basic/fixdep. When the
> > timing is correct, one of these packages will try to use fixdep while
> > the other is compiling it, resulting in fixdep being briefly not
> > available.

I am surprised that the linux-tools would use fixdep. As I understand
it, fixdep is there to '"Optimize" a list of dependencies as spit out by
gcc -MD' because (paraphrasing the description from fixdep.c) basically
everything in the kernel ends up icluding autoconf.h. However,
autoconf.h contains the configuration of the kernel; I don't see why
that would be used by the userland code...

And what puzzles me then, is that running 'make clean; make linux-tools'
would currently end up with a kernel tree that is not yet configured, so
fixdep would not have the necessary input to work with, but that does
not prevent the build from succeeding today...

Can you provide a bit more details as to what tool actually triggers
that call, or if any does, and why that tool would need to do that?

> > To fix this, set linux as a hard dependency instead of a patch

s/hard/build/

But otherwise, I agree that this is the best fix. As you said, the patch
dependency was but an optimisation.

> > dependency.
> 
> > Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>

When you respin with a bit more details, and an update to the comment as
pointed out by Thomas, you can add:

    Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> > ---
> > While the timing of this seems like it should very rarely be triggered,
> > on my build server I hit this issue nearly 100% of the time.
> > ---
> >  package/linux-tools/linux-tools.mk | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/package/linux-tools/linux-tools.mk b/package/linux-tools/linux-tools.mk
> > index 81b1fbc17605d6f003558d113143b64b5af6714d..a5d4a66f761fc44b156a92edaaf21774529f52d5 100644
> > --- a/package/linux-tools/linux-tools.mk
> > +++ b/package/linux-tools/linux-tools.mk
> > @@ -20,7 +20,7 @@
> >  include $(sort $(wildcard package/linux-tools/*.mk.in))
> >  
> >  # We only need the kernel to be extracted, not actually built
> 
> This comment then needs to be updated. Maybe it can be kept, but
> extended with why we're using a normal dependency, despite the fact
> that we only need the kernel to be extracted.
> 
> > -LINUX_TOOLS_PATCH_DEPENDENCIES = linux
> > +LINUX_TOOLS_DEPENDENCIES = linux
> 
> But otherwise, unless I'm missing something, it looks like a reasonably
> solution to the problem.
> 
> Thomas
> -- 
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] package/linux-tools: Fix parallel compilation

[Charlie Jenkins]

On Sun, Apr 20, 2025 at 09:11:53AM +0200, Yann E. MORIN wrote:
> Charlie, All,
> 
> On 2025-04-19 23:33 +0200, Thomas Petazzoni spake thusly:
> > On Wed, 09 Apr 2025 01:51:00 -0700
> > Charlie Jenkins <charlie@rivosinc.com> wrote:
> > > linux-tools opportunistically set linux as only a patch dependency. This
> > > unfortunately introduces a race condition in the shared linux scripts
> > > when using PER_PACKAGE_DIRECTORIES and using top level parallelism.  The
> > > race manifests as the error "/bin/sh: 1: scripts/basic/fixdep:
> > > Permission denied". This happens when the linux package and the
> > > linux-tools package are being compiled in parallel. During the setup,
> > > each of these tools ends up compiling scripts/basic/fixdep. When the
> > > timing is correct, one of these packages will try to use fixdep while
> > > the other is compiling it, resulting in fixdep being briefly not
> > > available.
> 
> I am surprised that the linux-tools would use fixdep. As I understand
> it, fixdep is there to '"Optimize" a list of dependencies as spit out by
> gcc -MD' because (paraphrasing the description from fixdep.c) basically
> everything in the kernel ends up icluding autoconf.h. However,
> autoconf.h contains the configuration of the kernel; I don't see why
> that would be used by the userland code...
> 
> And what puzzles me then, is that running 'make clean; make linux-tools'
> would currently end up with a kernel tree that is not yet configured, so
> fixdep would not have the necessary input to work with, but that does
> not prevent the build from succeeding today...
> 
> Can you provide a bit more details as to what tool actually triggers
> that call, or if any does, and why that tool would need to do that?

It's perf that is using it. perf uses a constrained version of kconfig
-- all the configs are detected at build time by figuring out what the
system supports and looking at environment variables instead of being
selected by menuconfig. Since it's using kconfig, it also needs to solve
the autoconf.h problem and so it uses fixdep.

rtla and bpf do the same thing.

> 
> > > To fix this, set linux as a hard dependency instead of a patch
> 
> s/hard/build/
> 
> But otherwise, I agree that this is the best fix. As you said, the patch
> dependency was but an optimisation.
> 
> > > dependency.
> > 
> > > Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
> 
> When you respin with a bit more details, and an update to the comment as
> pointed out by Thomas, you can add:
> 
>     Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>

Thanks! Yes, I will add the perf details.

> 
> Regards,
> Yann E. MORIN.
> 
> > > ---
> > > While the timing of this seems like it should very rarely be triggered,
> > > on my build server I hit this issue nearly 100% of the time.
> > > ---
> > >  package/linux-tools/linux-tools.mk | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/package/linux-tools/linux-tools.mk b/package/linux-tools/linux-tools.mk
> > > index 81b1fbc17605d6f003558d113143b64b5af6714d..a5d4a66f761fc44b156a92edaaf21774529f52d5 100644
> > > --- a/package/linux-tools/linux-tools.mk
> > > +++ b/package/linux-tools/linux-tools.mk
> > > @@ -20,7 +20,7 @@
> > >  include $(sort $(wildcard package/linux-tools/*.mk.in))
> > >  
> > >  # We only need the kernel to be extracted, not actually built
> > 
> > This comment then needs to be updated. Maybe it can be kept, but
> > extended with why we're using a normal dependency, despite the fact
> > that we only need the kernel to be extracted.
> > 
> > > -LINUX_TOOLS_PATCH_DEPENDENCIES = linux
> > > +LINUX_TOOLS_DEPENDENCIES = linux
> > 
> > But otherwise, unless I'm missing something, it looks like a reasonably
> > solution to the problem.
> > 
> > Thomas
> > -- 
> > Thomas Petazzoni, co-owner and CEO, Bootlin
> > Embedded Linux and Kernel engineering and training
> > https://bootlin.com
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot