[PATCH 0/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[PATCH 0/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Thomas Weißschuh]

Make sure the byte order and ABI of the userprogs matches the one of the
kernel, similar to how the bit size is handled.
Otherwise the userprogs may not be executable.
This happens for example on powerpc little endian, or riscv32.

These patches where originally part of my series "kunit: Introduce UAPI
testing framework" [0], but that isn't going anywhere right now and the
patches are useful on their own.

[0] kunit: Introduce UAPI testing framework

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Thomas Weißschuh (2):
      kbuild: userprogs: avoid duplication of flags inherited from kernel
      kbuild: userprogs: also inherit byte order and ABI from kernel

 Makefile | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
---
base-commit: 8f5ae30d69d7543eee0d70083daf4de8fe15d585
change-id: 20250813-kbuild-userprogs-bits-03c117da4d50

Best regards,
-- 
Thomas Weißschuh <thomas.weissschuh@linutronix.de>

[PATCH 1/2] kbuild: userprogs: avoid duplication of flags inherited from kernel

[Thomas Weißschuh]

The duplication makes maintenance harder. Changes need to be done in two
places and the lines will grow overly long.

Use an intermediary variable instead.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Reviewed-by: Nicolas Schier <n.schier@avm.de>
Acked-by: Masahiro Yamada <masahiroy@kernel.org>
---
 Makefile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 6bfe776bf3c5ff0cf187dc6719dd5817cd4af2ca..d0f5262a9c0f3b4aa79a91c20cc149d034ffa0b7 100644
--- a/Makefile
+++ b/Makefile
@@ -1138,8 +1138,9 @@ LDFLAGS_vmlinux	+= --emit-relocs --discard-none
 endif
 
 # Align the bit size of userspace programs with the kernel
-KBUILD_USERCFLAGS  += $(filter -m32 -m64 --target=%, $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
-KBUILD_USERLDFLAGS += $(filter -m32 -m64 --target=%, $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
+USERFLAGS_FROM_KERNEL := -m32 -m64 --target=%
+KBUILD_USERCFLAGS  += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
+KBUILD_USERLDFLAGS += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
 
 # userspace programs are linked via the compiler, use the correct linker
 ifdef CONFIG_CC_IS_CLANG

-- 
2.50.1

[PATCH 2/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Thomas Weißschuh]

Make sure the byte order and ABI of the userprogs matches the one of the
kernel, similar to how the bit size is handled.
Otherwise the userprogs may not be executable.
This happens for example on powerpc little endian, or riscv32.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Reviewed-by: Nicolas Schier <n.schier@avm.de>
Acked-by: Masahiro Yamada <masahiroy@kernel.org>

---
Difference to original series:
* Also handle -EL/-EB for MIPS
---
 Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index d0f5262a9c0f3b4aa79a91c20cc149d034ffa0b7..7d40f84d5efde18ed3a2f4d8cf7a9b1ec3610ed4 100644
--- a/Makefile
+++ b/Makefile
@@ -1137,8 +1137,8 @@ ifneq ($(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS),)
 LDFLAGS_vmlinux	+= --emit-relocs --discard-none
 endif
 
-# Align the bit size of userspace programs with the kernel
-USERFLAGS_FROM_KERNEL := -m32 -m64 --target=%
+# Align the bit size, byte order and architecture of userspace programs with the kernel
+USERFLAGS_FROM_KERNEL := -m32 -m64 -mlittle-endian -mbig-endian -EL -EB --target=% -march=% -mabi=%
 KBUILD_USERCFLAGS  += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
 KBUILD_USERLDFLAGS += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
 

-- 
2.50.1

Re: [PATCH 0/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Nathan Chancellor]

On Wed, Aug 13, 2025 at 07:43:39AM +0200, Thomas Weißschuh wrote:
> Make sure the byte order and ABI of the userprogs matches the one of the
> kernel, similar to how the bit size is handled.
> Otherwise the userprogs may not be executable.
> This happens for example on powerpc little endian, or riscv32.
> 
> These patches where originally part of my series "kunit: Introduce UAPI
> testing framework" [0], but that isn't going anywhere right now and the
> patches are useful on their own.
> 
> [0] kunit: Introduce UAPI testing framework
> ---
> Thomas Weißschuh (2):
>       kbuild: userprogs: avoid duplication of flags inherited from kernel
>       kbuild: userprogs: also inherit byte order and ABI from kernel

Seems reasonable to me. Should I fast track this via kbuild-fixes or
should I just apply it to kbuild-next? I am guessing you only noticed
this in the context of developing [0] so it might not be a big issue in
the wild?

Cheers,
Nathan

Re: [PATCH 0/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Thomas Weißschuh]

On Thu, Aug 14, 2025 at 11:46:46AM -0700, Nathan Chancellor wrote:
> On Wed, Aug 13, 2025 at 07:43:39AM +0200, Thomas Weißschuh wrote:
> > Make sure the byte order and ABI of the userprogs matches the one of the
> > kernel, similar to how the bit size is handled.
> > Otherwise the userprogs may not be executable.
> > This happens for example on powerpc little endian, or riscv32.
> > 
> > These patches where originally part of my series "kunit: Introduce UAPI
> > testing framework" [0], but that isn't going anywhere right now and the
> > patches are useful on their own.
> > 
> > [0] kunit: Introduce UAPI testing framework

[0] https://lore.kernel.org/lkml/20250717-kunit-kselftests-v5-0-442b711cde2e@linutronix.de/

> > ---
> > Thomas Weißschuh (2):
> >       kbuild: userprogs: avoid duplication of flags inherited from kernel
> >       kbuild: userprogs: also inherit byte order and ABI from kernel
> 
> Seems reasonable to me. Should I fast track this via kbuild-fixes or
> should I just apply it to kbuild-next? I am guessing you only noticed
> this in the context of developing [0] so it might not be a big issue in
> the wild?

kbuild-next should be fine.


Thomas

Re: [PATCH 0/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Nathan Chancellor]

On Wed, 13 Aug 2025 07:43:39 +0200, Thomas Weißschuh wrote:
> Make sure the byte order and ABI of the userprogs matches the one of the
> kernel, similar to how the bit size is handled.
> Otherwise the userprogs may not be executable.
> This happens for example on powerpc little endian, or riscv32.
> 
> These patches where originally part of my series "kunit: Introduce UAPI
> testing framework" [0], but that isn't going anywhere right now and the
> patches are useful on their own.
> 
> [...]

Applied, thanks!

[1/2] kbuild: userprogs: avoid duplication of flags inherited from kernel
      https://git.kernel.org/kbuild/c/c5afee88548e4
[2/2] kbuild: userprogs: also inherit byte order and ABI from kernel
      https://git.kernel.org/kbuild/c/478494044bb42

Best regards,
-- 
Nathan Chancellor <nathan@kernel.org>

Re: [PATCH 2/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Thomas Weißschuh]

Hi!

On Wed, Aug 13, 2025 at 07:43:41AM +0200, Thomas Weißschuh wrote:
> Make sure the byte order and ABI of the userprogs matches the one of the
> kernel, similar to how the bit size is handled.
> Otherwise the userprogs may not be executable.
> This happens for example on powerpc little endian, or riscv32.
> 
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> Reviewed-by: Nicolas Schier <n.schier@avm.de>
> Acked-by: Masahiro Yamada <masahiroy@kernel.org>
> 
> ---
> Difference to original series:
> * Also handle -EL/-EB for MIPS
> ---
>  Makefile | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index d0f5262a9c0f3b4aa79a91c20cc149d034ffa0b7..7d40f84d5efde18ed3a2f4d8cf7a9b1ec3610ed4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1137,8 +1137,8 @@ ifneq ($(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS),)
>  LDFLAGS_vmlinux	+= --emit-relocs --discard-none
>  endif
>  
> -# Align the bit size of userspace programs with the kernel
> -USERFLAGS_FROM_KERNEL := -m32 -m64 --target=%
> +# Align the bit size, byte order and architecture of userspace programs with the kernel
> +USERFLAGS_FROM_KERNEL := -m32 -m64 -mlittle-endian -mbig-endian -EL -EB --target=% -march=% -mabi=%
>  KBUILD_USERCFLAGS  += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
>  KBUILD_USERLDFLAGS += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))

I looked some more at the breakage reported from -next [0].
For architectures with multiple possible byte orders or ABIs the toolchain
might be able to build the kernel for all combinations but might not have a
matching libc for them. Currently userprogs uses the default byteorder and
ABI from the compiler, which will match the included libcs if there is any.
However the resulting binary might not run on the built kernel.
CC_CAN_LINK can be extended to generically handle different byte orders, as
for those we have standard kconfig symbols. But handling ABIs would need to
be architecture specific and a bit more complex.

We can't use KBUILD_*FLAGS for CC_CAN_LINK, as they are not yet set during
the configuration stage.

I see the following options:

* Add byte order and architecture-specific ABI handling to CC_CAN_LINK
* Accept that userprogs might not be runnable on the built kernel
* Let the user manually set CC_CAN_LINK to override the autodetection
* Add separate handling for runnable userprogs
* Use tools/include/nolibc/ for userprogs instead of the toolchain libc
  (unlikely, but I wanted do mention the option)


Thomas

[0] https://lore.kernel.org/lkml/20250818140143.61b8c466@canb.auug.org.au/

Re: [PATCH 2/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Nathan Chancellor]

Hi Thomas,

On Wed, Aug 27, 2025 at 08:31:00AM +0200, Thomas Weißschuh wrote:
> On Wed, Aug 13, 2025 at 07:43:41AM +0200, Thomas Weißschuh wrote:
...
> > diff --git a/Makefile b/Makefile
> > index d0f5262a9c0f3b4aa79a91c20cc149d034ffa0b7..7d40f84d5efde18ed3a2f4d8cf7a9b1ec3610ed4 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -1137,8 +1137,8 @@ ifneq ($(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS),)
> >  LDFLAGS_vmlinux	+= --emit-relocs --discard-none
> >  endif
> >  
> > -# Align the bit size of userspace programs with the kernel
> > -USERFLAGS_FROM_KERNEL := -m32 -m64 --target=%
> > +# Align the bit size, byte order and architecture of userspace programs with the kernel
> > +USERFLAGS_FROM_KERNEL := -m32 -m64 -mlittle-endian -mbig-endian -EL -EB --target=% -march=% -mabi=%
> >  KBUILD_USERCFLAGS  += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
> >  KBUILD_USERLDFLAGS += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
> 
> I looked some more at the breakage reported from -next [0].

Thanks for looking into this.

> For architectures with multiple possible byte orders or ABIs the toolchain
> might be able to build the kernel for all combinations but might not have a
> matching libc for them. Currently userprogs uses the default byteorder and
> ABI from the compiler, which will match the included libcs if there is any.
> However the resulting binary might not run on the built kernel.
> CC_CAN_LINK can be extended to generically handle different byte orders, as
> for those we have standard kconfig symbols. But handling ABIs would need to
> be architecture specific and a bit more complex.
> 
> We can't use KBUILD_*FLAGS for CC_CAN_LINK, as they are not yet set during
> the configuration stage.

Right, this was the biggest thing that I noticed, which would really
help us out...

> I see the following options:
> 
> * Add byte order and architecture-specific ABI handling to CC_CAN_LINK

How do you envision this? Different default lines for each combination?
Feels like that could get complicated quickly but this would probably be
the objectively most robust and "hands off" option.

> * Accept that userprogs might not be runnable on the built kernel

I do wonder how common running the userprogs are. Obviously you hit this
in testing but I am curious if others have reported this.

> * Let the user manually set CC_CAN_LINK to override the autodetection

I am not sure I like this one but maybe we could have a hidden
CC_MAY_LINK with something like the current CC_CAN_LINK then CC_CAN_LINK
could be turned into something like

    config CC_CAN_LINK
        bool "Toolchain can link userspace executables" if !COMPILE_TEST
        default CC_MAY_LINK

I am not surethat would actually help us here plus I still do not like
the idea of throwing this back to the user.

> * Add separate handling for runnable userprogs

What do you mean by this?

> * Use tools/include/nolibc/ for userprogs instead of the toolchain libc
>   (unlikely, but I wanted do mention the option)

This could be interesting to explore but I assume using a cross libc
could be desirable for some people depending on the testing?

Cheers,
Nathan

Re: [PATCH 2/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Thomas Weißschuh]

On Wed, Aug 27, 2025 at 03:49:35PM -0700, Nathan Chancellor wrote:
> On Wed, Aug 27, 2025 at 08:31:00AM +0200, Thomas Weißschuh wrote:
> > On Wed, Aug 13, 2025 at 07:43:41AM +0200, Thomas Weißschuh wrote:
> ...
> > > diff --git a/Makefile b/Makefile
> > > index d0f5262a9c0f3b4aa79a91c20cc149d034ffa0b7..7d40f84d5efde18ed3a2f4d8cf7a9b1ec3610ed4 100644
> > > --- a/Makefile
> > > +++ b/Makefile
> > > @@ -1137,8 +1137,8 @@ ifneq ($(CONFIG_ARCH_VMLINUX_NEEDS_RELOCS),)
> > >  LDFLAGS_vmlinux	+= --emit-relocs --discard-none
> > >  endif
> > >  
> > > -# Align the bit size of userspace programs with the kernel
> > > -USERFLAGS_FROM_KERNEL := -m32 -m64 --target=%
> > > +# Align the bit size, byte order and architecture of userspace programs with the kernel
> > > +USERFLAGS_FROM_KERNEL := -m32 -m64 -mlittle-endian -mbig-endian -EL -EB --target=% -march=% -mabi=%
> > >  KBUILD_USERCFLAGS  += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))
> > >  KBUILD_USERLDFLAGS += $(filter $(USERFLAGS_FROM_KERNEL), $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS))

(...)

> > For architectures with multiple possible byte orders or ABIs the toolchain
> > might be able to build the kernel for all combinations but might not have a
> > matching libc for them. Currently userprogs uses the default byteorder and
> > ABI from the compiler, which will match the included libcs if there is any.
> > However the resulting binary might not run on the built kernel.
> > CC_CAN_LINK can be extended to generically handle different byte orders, as
> > for those we have standard kconfig symbols. But handling ABIs would need to
> > be architecture specific and a bit more complex.
> > 
> > We can't use KBUILD_*FLAGS for CC_CAN_LINK, as they are not yet set during
> > the configuration stage.
> 
> Right, this was the biggest thing that I noticed, which would really
> help us out...
> 
> > I see the following options:
> > 
> > * Add byte order and architecture-specific ABI handling to CC_CAN_LINK
> 
> How do you envision this? Different default lines for each combination?

Exactly. The normal cases can be handled generically. For example the kconfig
below works for architectures which only differ in byte order and 32bit/64bit,
which are most of them. MIPS should require more logic.
Also I'm ignoring x32, as it is never the kernel's native ABI.

 config CC_CAN_LINK
        bool
+       default $(cc_can_link_user,$(m64-flag) -mlittle-endian) if 64BIT && CPU_LITTLE_ENDIAN
+       default $(cc_can_link_user,$(m64-flag) -mbig-endian) if 64BIT && CPU_BIG_ENDIAN
        default $(cc_can_link_user,$(m64-flag)) if 64BIT
+       default $(cc_can_link_user,$(m32-flag) -mlittle-endian) if CPU_LITTLE_ENDIAN
+       default $(cc_can_link_user,$(m32-flag) -mbig-endian) if CPU_BIG_ENDIAN
        default $(cc_can_link_user,$(m32-flag))


> Feels like that could get complicated quickly but this would probably be
> the objectively most robust and "hands off" option.

Agreed.

> > * Accept that userprogs might not be runnable on the built kernel
> 
> I do wonder how common running the userprogs are. Obviously you hit this
> in testing but I am curious if others have reported this.

I'm not aware of any reports. Usage of userprogs seems to be fairly low in
general, judging by the issues I ran into which nobody reported or fixed
before. One of the original reasons for the introduction of userprogs were
usermode drivers. These indeed required execution. But the whole framework
was removed recently, although it might come back at some point.

> > * Let the user manually set CC_CAN_LINK to override the autodetection

(...)

> I am not surethat would actually help us here plus I still do not like
> the idea of throwing this back to the user.

Agreed. My personal preference would be my proposal from above.

> > * Add separate handling for runnable userprogs
> 
> What do you mean by this?

I see two possibilities.
* New kconfig and FLAGS mirroring the existing ones to build runnable
  userprogs.
* Introspection of the userprog executable ELF to see if it can run on the
  current kernel configuration.

> > * Use tools/include/nolibc/ for userprogs instead of the toolchain libc
> >   (unlikely, but I wanted do mention the option)
> 
> This could be interesting to explore but I assume using a cross libc
> could be desirable for some people depending on the testing?

Yes, probably. Although the test code in samples/ should interact directly with
the kernel, without much involvement from libc.


Thomas

Re: [PATCH 2/2] kbuild: userprogs: also inherit byte order and ABI from kernel

[Nathan Chancellor]

On Mon, Sep 01, 2025 at 11:51:03AM +0200, Thomas Weißschuh wrote:
> Exactly. The normal cases can be handled generically. For example the kconfig
> below works for architectures which only differ in byte order and 32bit/64bit,
> which are most of them. MIPS should require more logic.
> Also I'm ignoring x32, as it is never the kernel's native ABI.
> 
>  config CC_CAN_LINK
>         bool
> +       default $(cc_can_link_user,$(m64-flag) -mlittle-endian) if 64BIT && CPU_LITTLE_ENDIAN
> +       default $(cc_can_link_user,$(m64-flag) -mbig-endian) if 64BIT && CPU_BIG_ENDIAN
>         default $(cc_can_link_user,$(m64-flag)) if 64BIT
> +       default $(cc_can_link_user,$(m32-flag) -mlittle-endian) if CPU_LITTLE_ENDIAN
> +       default $(cc_can_link_user,$(m32-flag) -mbig-endian) if CPU_BIG_ENDIAN
>         default $(cc_can_link_user,$(m32-flag))
> 
> 
> > Feels like that could get complicated quickly but this would probably be
> > the objectively most robust and "hands off" option.
> 
> Agreed.

Nicolas might feel differently but this does not seem terrible to me,
especially with a macro to wrap the common logic, which is where I felt
like things could get unwieldy. Feel free to send an RFC if it is not
too much work.

Cheers,
Nathan