PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

[Stuart Little]

I am trying to compile on an x86_64 host for a 32-bit system; my config is at

https://termbin.com/v8jl

I am getting numerous errors of the form

./include/linux/kasan-checks.h:17:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

and

./include/linux/kcsan-checks.h:143:6: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

and

./arch/x86/include/asm/arch_hweight.h:16:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

(those include files indicated whom I should add to this list; apologies if this reaches you in error).

The full log of the build is at

https://termbin.com/wbgs

---

5.11.0-rc6 built fine last week on this same setup. 

Re: PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

[Stuart Little]

The result of the bisect on the issue reported in the previous message:

--- cut ---

20bf2b378729c4a0366a53e2018a0b70ace94bcd is the first bad commit
commit 20bf2b378729c4a0366a53e2018a0b70ace94bcd
Author: Josh Poimboeuf <jpoimboe@redhat.com>
Date:   Thu Jan 28 15:52:19 2021 -0600

    x86/build: Disable CET instrumentation in the kernel
    
    With retpolines disabled, some configurations of GCC, and specifically
    the GCC versions 9 and 10 in Ubuntu will add Intel CET instrumentation
    to the kernel by default. That breaks certain tracing scenarios by
    adding a superfluous ENDBR64 instruction before the fentry call, for
    functions which can be called indirectly.
    
    CET instrumentation isn't currently necessary in the kernel, as CET is
    only supported in user space. Disable it unconditionally and move it
    into the x86's Makefile as CET/CFI... enablement should be a per-arch
    decision anyway.
    
     [ bp: Massage and extend commit message. ]
    
    Fixes: 29be86d7f9cb ("kbuild: add -fcf-protection=none when using retpoline flags")
    Reported-by: Nikolay Borisov <nborisov@suse.com>
    Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
    Signed-off-by: Borislav Petkov <bp@suse.de>
    Reviewed-by: Nikolay Borisov <nborisov@suse.com>
    Tested-by: Nikolay Borisov <nborisov@suse.com>
    Cc: <stable@vger.kernel.org>
    Cc: Seth Forshee <seth.forshee@canonical.com>
    Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
    Link: https://lkml.kernel.org/r/20210128215219.6kct3h2eiustncws@treble

 Makefile          | 6 ------
 arch/x86/Makefile | 3 +++
 2 files changed, 3 insertions(+), 6 deletions(-)

--- end ---

On Sun, Feb 07, 2021 at 06:31:22PM -0500, Stuart Little wrote:
> I am trying to compile on an x86_64 host for a 32-bit system; my config is at
> 
> https://termbin.com/v8jl
> 
> I am getting numerous errors of the form
> 
> ./include/linux/kasan-checks.h:17:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> 
> and
> 
> ./include/linux/kcsan-checks.h:143:6: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> 
> and
> 
> ./arch/x86/include/asm/arch_hweight.h:16:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> 
> (those include files indicated whom I should add to this list; apologies if this reaches you in error).
> 
> The full log of the build is at
> 
> https://termbin.com/wbgs
> 
> ---
> 
> 5.11.0-rc6 built fine last week on this same setup. 

Re: PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

[Stuart Little]

And for good measure: reverting that commit 

20bf2b378729c4a0366a53e2018a0b70ace94bcd

flagged by the bisect right on top of the current tree compiles fine. 

On Sun, Feb 07, 2021 at 07:26:01PM -0500, Stuart Little wrote:
> The result of the bisect on the issue reported in the previous message:
> 
> --- cut ---
> 
> 20bf2b378729c4a0366a53e2018a0b70ace94bcd is the first bad commit
> commit 20bf2b378729c4a0366a53e2018a0b70ace94bcd
> Author: Josh Poimboeuf <jpoimboe@redhat.com>
> Date:   Thu Jan 28 15:52:19 2021 -0600
> 
>     x86/build: Disable CET instrumentation in the kernel
>     
>     With retpolines disabled, some configurations of GCC, and specifically
>     the GCC versions 9 and 10 in Ubuntu will add Intel CET instrumentation
>     to the kernel by default. That breaks certain tracing scenarios by
>     adding a superfluous ENDBR64 instruction before the fentry call, for
>     functions which can be called indirectly.
>     
>     CET instrumentation isn't currently necessary in the kernel, as CET is
>     only supported in user space. Disable it unconditionally and move it
>     into the x86's Makefile as CET/CFI... enablement should be a per-arch
>     decision anyway.
>     
>      [ bp: Massage and extend commit message. ]
>     
>     Fixes: 29be86d7f9cb ("kbuild: add -fcf-protection=none when using retpoline flags")
>     Reported-by: Nikolay Borisov <nborisov@suse.com>
>     Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
>     Signed-off-by: Borislav Petkov <bp@suse.de>
>     Reviewed-by: Nikolay Borisov <nborisov@suse.com>
>     Tested-by: Nikolay Borisov <nborisov@suse.com>
>     Cc: <stable@vger.kernel.org>
>     Cc: Seth Forshee <seth.forshee@canonical.com>
>     Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
>     Link: https://lkml.kernel.org/r/20210128215219.6kct3h2eiustncws@treble
> 
>  Makefile          | 6 ------
>  arch/x86/Makefile | 3 +++
>  2 files changed, 3 insertions(+), 6 deletions(-)
> 
> --- end ---
> 
> On Sun, Feb 07, 2021 at 06:31:22PM -0500, Stuart Little wrote:
> > I am trying to compile on an x86_64 host for a 32-bit system; my config is at
> > 
> > https://termbin.com/v8jl
> > 
> > I am getting numerous errors of the form
> > 
> > ./include/linux/kasan-checks.h:17:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> > 
> > and
> > 
> > ./include/linux/kcsan-checks.h:143:6: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> > 
> > and
> > 
> > ./arch/x86/include/asm/arch_hweight.h:16:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> > 
> > (those include files indicated whom I should add to this list; apologies if this reaches you in error).
> > 
> > The full log of the build is at
> > 
> > https://termbin.com/wbgs
> > 
> > ---
> > 
> > 5.11.0-rc6 built fine last week on this same setup. 

Re: PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

[Marco Elver]

On Mon, 8 Feb 2021 at 01:40, Stuart Little <achirvasub@gmail.com> wrote:
>
> And for good measure: reverting that commit
>
> 20bf2b378729c4a0366a53e2018a0b70ace94bcd
>
> flagged by the bisect right on top of the current tree compiles fine.
>
> On Sun, Feb 07, 2021 at 07:26:01PM -0500, Stuart Little wrote:
> > The result of the bisect on the issue reported in the previous message:
> >
> > --- cut ---
> >
> > 20bf2b378729c4a0366a53e2018a0b70ace94bcd is the first bad commit
> > commit 20bf2b378729c4a0366a53e2018a0b70ace94bcd
> > Author: Josh Poimboeuf <jpoimboe@redhat.com>
> > Date:   Thu Jan 28 15:52:19 2021 -0600
> >
> >     x86/build: Disable CET instrumentation in the kernel
> >
> >     With retpolines disabled, some configurations of GCC, and specifically
> >     the GCC versions 9 and 10 in Ubuntu will add Intel CET instrumentation
> >     to the kernel by default. That breaks certain tracing scenarios by
> >     adding a superfluous ENDBR64 instruction before the fentry call, for
> >     functions which can be called indirectly.
> >
> >     CET instrumentation isn't currently necessary in the kernel, as CET is
> >     only supported in user space. Disable it unconditionally and move it
> >     into the x86's Makefile as CET/CFI... enablement should be a per-arch
> >     decision anyway.
> >
> >      [ bp: Massage and extend commit message. ]
> >
> >     Fixes: 29be86d7f9cb ("kbuild: add -fcf-protection=none when using retpoline flags")
> >     Reported-by: Nikolay Borisov <nborisov@suse.com>
> >     Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> >     Signed-off-by: Borislav Petkov <bp@suse.de>
> >     Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> >     Tested-by: Nikolay Borisov <nborisov@suse.com>
> >     Cc: <stable@vger.kernel.org>
> >     Cc: Seth Forshee <seth.forshee@canonical.com>
> >     Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
> >     Link: https://lkml.kernel.org/r/20210128215219.6kct3h2eiustncws@treble
> >
> >  Makefile          | 6 ------
> >  arch/x86/Makefile | 3 +++
> >  2 files changed, 3 insertions(+), 6 deletions(-)
> >
> > --- end ---
> >
> > On Sun, Feb 07, 2021 at 06:31:22PM -0500, Stuart Little wrote:
> > > I am trying to compile on an x86_64 host for a 32-bit system; my config is at
> > >
> > > https://termbin.com/v8jl
> > >
> > > I am getting numerous errors of the form
> > >
> > > ./include/linux/kasan-checks.h:17:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

This is an empty static inline function...

> > > and
> > >
> > > ./include/linux/kcsan-checks.h:143:6: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

... and so is this. I think these have very little to do with the
problem that you reported. My guess is they show up because these are
included very early.

> > > and
> > >
> > > ./arch/x86/include/asm/arch_hweight.h:16:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible
> > >
> > > (those include files indicated whom I should add to this list; apologies if this reaches you in error).
> > >
> > > The full log of the build is at
> > >
> > > https://termbin.com/wbgs

The commonality between all these errors is that they originate from
compiling arch/x86/entry/vdso/vdso32/vclock_gettime.c.

Is the build system adding special flags for vdso? In which case, it's
probably just GCC complaining about every function definition (static
inline or otherwise) for that TU if (for whatever reason) it's
delaying the flag compatibility check until it inspects function
attributes.

And indeed, I can see:

  RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline
-mindirect-branch-register

And taking any test source with even an empty function definition:

  > gcc -mindirect-branch=thunk-inline -fcf-protection test.c
  > test.c: In function ‘main’:
  > test.c:6:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are
not compatible

> > > 5.11.0-rc6 built fine last week on this same setup.

Thanks,
-- Marco

Re: PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

[Borislav Petkov]

On Sun, Feb 07, 2021 at 07:40:32PM -0500, Stuart Little wrote:
> > On Sun, Feb 07, 2021 at 06:31:22PM -0500, Stuart Little wrote:
> > > I am trying to compile on an x86_64 host for a 32-bit system; my config is at
> > > 
> > > https://termbin.com/v8jl
> > > 
> > > I am getting numerous errors of the form
> > > 
> > > ./include/linux/kasan-checks.h:17:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

Does this fix it?

---

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 5857917f83ee..30920d70b48b 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -50,6 +50,9 @@ export BITS
 KBUILD_CFLAGS += -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
 KBUILD_CFLAGS += $(call cc-option,-mno-avx,)
 
+# Intel CET isn't enabled in the kernel
+KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
+
 ifeq ($(CONFIG_X86_32),y)
         BITS := 32
         UTS_MACHINE := i386
@@ -120,9 +123,6 @@ else
 
         KBUILD_CFLAGS += -mno-red-zone
         KBUILD_CFLAGS += -mcmodel=kernel
-
-	# Intel CET isn't enabled in the kernel
-	KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
 endif
 
 ifdef CONFIG_X86_X32


-- 
Regards/Gruss,
    Boris.

SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG Nürnberg

Re: PROBLEM: 5.11.0-rc7 fails to compile with error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

[AC]

That did fix it, thank you!

On February 8, 2021 7:12:27 AM EST, Borislav Petkov <bp@suse.de> wrote:
>On Sun, Feb 07, 2021 at 07:40:32PM -0500, Stuart Little wrote:
>> > On Sun, Feb 07, 2021 at 06:31:22PM -0500, Stuart Little wrote:
>> > > I am trying to compile on an x86_64 host for a 32-bit system; my
>config is at
>> > > 
>> > > https://termbin.com/v8jl
>> > > 
>> > > I am getting numerous errors of the form
>> > > 
>> > > ./include/linux/kasan-checks.h:17:1: error: ‘-mindirect-branch’
>and ‘-fcf-protection’ are not compatible
>
>Does this fix it?
>
>---
>
>diff --git a/arch/x86/Makefile b/arch/x86/Makefile
>index 5857917f83ee..30920d70b48b 100644
>--- a/arch/x86/Makefile
>+++ b/arch/x86/Makefile
>@@ -50,6 +50,9 @@ export BITS
> KBUILD_CFLAGS += -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
> KBUILD_CFLAGS += $(call cc-option,-mno-avx,)
> 
>+# Intel CET isn't enabled in the kernel
>+KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
>+
> ifeq ($(CONFIG_X86_32),y)
>         BITS := 32
>         UTS_MACHINE := i386
>@@ -120,9 +123,6 @@ else
> 
>         KBUILD_CFLAGS += -mno-red-zone
>         KBUILD_CFLAGS += -mcmodel=kernel
>-
>-	# Intel CET isn't enabled in the kernel
>-	KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
> endif
> 
> ifdef CONFIG_X86_X32

[PATCH] x86/build: Disable CET instrumentation in the kernel for 32-bit too

[Borislav Petkov]

On Mon, Feb 08, 2021 at 10:19:33AM -0500, AC wrote:
> That did fix it, thank you!

Thanks!

---
From: Borislav Petkov <bp@suse.de>
Date: Mon, 8 Feb 2021 16:43:30 +0100
Subject: [PATCH] x86/build: Disable CET instrumentation in the kernel for 32-bit too

Commit

  20bf2b378729 ("x86/build: Disable CET instrumentation in the kernel")

disabled CET instrumentation which gets added by default by the Ubuntu
gcc9 and 10 by default, but did that only for 64-bit builds. It would
still fail when building a 32-bit target. So disable CET for all x86
builds.

Fixes: 20bf2b378729 ("x86/build: Disable CET instrumentation in the kernel")
Reported-by: AC <achirvasub@gmail.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Tested-by: AC <achirvasub@gmail.com>
Link: https://lkml.kernel.org/r/YCCIgMHkzh/xT4ex@arch-chirva.localdomain
---
 arch/x86/Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 109c7f86483c..b9f58b8993b3 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -50,6 +50,9 @@ export BITS
 KBUILD_CFLAGS += -mno-sse -mno-mmx -mno-sse2 -mno-3dnow
 KBUILD_CFLAGS += $(call cc-option,-mno-avx,)
 
+# Intel CET isn't enabled in the kernel
+KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
+
 ifeq ($(CONFIG_X86_32),y)
         BITS := 32
         UTS_MACHINE := i386
@@ -120,9 +123,6 @@ else
 
         KBUILD_CFLAGS += -mno-red-zone
         KBUILD_CFLAGS += -mcmodel=kernel
-
-	# Intel CET isn't enabled in the kernel
-	KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none)
 endif
 
 ifdef CONFIG_X86_X32
-- 
2.29.2

-- 
Regards/Gruss,
    Boris.

SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG Nürnberg

Re: [PATCH] x86/build: Disable CET instrumentation in the kernel for 32-bit too

[Josh Poimboeuf]

On Mon, Feb 08, 2021 at 05:25:43PM +0100, Borislav Petkov wrote:
> On Mon, Feb 08, 2021 at 10:19:33AM -0500, AC wrote:
> > That did fix it, thank you!
> 
> Thanks!
> 
> ---
> From: Borislav Petkov <bp@suse.de>
> Date: Mon, 8 Feb 2021 16:43:30 +0100
> Subject: [PATCH] x86/build: Disable CET instrumentation in the kernel for 32-bit too
> 
> Commit
> 
>   20bf2b378729 ("x86/build: Disable CET instrumentation in the kernel")
> 
> disabled CET instrumentation which gets added by default by the Ubuntu
> gcc9 and 10 by default, but did that only for 64-bit builds. It would
> still fail when building a 32-bit target. So disable CET for all x86
> builds.
> 
> Fixes: 20bf2b378729 ("x86/build: Disable CET instrumentation in the kernel")
> Reported-by: AC <achirvasub@gmail.com>
> Signed-off-by: Borislav Petkov <bp@suse.de>
> Tested-by: AC <achirvasub@gmail.com>
> Link: https://lkml.kernel.org/r/YCCIgMHkzh/xT4ex@arch-chirva.localdomain

Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>

-- 
Josh