[PATCH v2] x86: add hintable NOPs emulation

[PATCH v2] x86: add hintable NOPs emulation

[Marcos Del Sol Vives]

Hintable NOPs are a series of instructions introduced by Intel with the
Pentium Pro (i686), and described in US patent US5701442A.

These instructions were reserved to allow backwards-compatible changes
in the instruction set possible, by having old processors treat them as
variable-length NOPs, while having other semantics in modern processors.

Some modern uses are:
 - Multi-byte/long NOPs
 - Indirect Branch Tracking (ENDBR32)
 - Shadow Stack (part of CET)

Some processors advertising i686 compatibility lack full support for
them, which may cause #UD to be incorrectly triggered, crashing software
that uses then with an unexpected SIGILL.

One such software is sudo in Debian bookworm, which is compiled with
GCC -fcf-protection=branch and contains ENDBR32 instructions. It crashes
on my Vortex86DX3 processor and VIA C3 Nehalem processors [1].

This patch is a much simplified version of my previous patch for x86
instruction emulation [2], that only emulates hintable NOPs.

When #UD is raised, it checks if the opcode corresponds to a hintable NOP
in user space. If true, it warns the user via the dmesg and advances the
instruction pointer, thus emulating its expected NOP behaviour.

[1]: https://lists.debian.org/debian-devel/2023/10/msg00118.html
[2]: https://lore.kernel.org/all/20210626130313.1283485-1-marcos@orca.pet/

Signed-off-by: Marcos Del Sol Vives <marcos@orca.pet>
---
 arch/x86/Kconfig        | 29 +++++++++++++++++++++++++++++
 arch/x86/kernel/traps.c | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 58d890fe2100..a6daebdc2573 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1286,6 +1286,35 @@ config X86_IOPL_IOPERM
 	  ability to disable interrupts from user space which would be
 	  granted if the hardware IOPL mechanism would be used.
 
+config X86_HNOP_EMU
+	bool "Hintable NOPs emulation"
+	depends on X86_32
+	default y
+	help
+	  Hintable NOPs are a series of instructions introduced by Intel with
+	  the Pentium Pro (i686), and described in US patent US5701442A.
+
+	  These instructions were reserved to allow backwards-compatible
+	  changes in the instruction set possible, by having old processors
+	  treat them as variable-length NOPs, while having other semantics in
+	  modern processors.
+
+	  Some modern uses are:
+	   - Multi-byte/long NOPs
+	   - Indirect Branch Tracking (ENDBR32)
+	   - Shadow Stack (part of CET)
+
+	  Some processors advertising i686 compatibility (such as Cyrix MII,
+	  VIA C3 Nehalem or DM&P Vortex86DX3) lack full support for them,
+	  which may cause SIGILL to be incorrectly raised in user space when
+	  a hintable NOP is encountered.
+
+	  Say Y here if you want the kernel to emulate them, allowing programs
+	  that make use of them to run transparently on such processors.
+
+	  This emulation has no performance penalty for processors that
+	  properly support them, so if unsure, enable it.
+
 config TOSHIBA
 	tristate "Toshiba Laptop support"
 	depends on X86_32
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 36354b470590..22b51c4186e7 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -295,12 +295,45 @@ DEFINE_IDTENTRY(exc_overflow)
 	do_error_trap(regs, 0, "overflow", X86_TRAP_OF, SIGSEGV, 0, NULL);
 }
 
+static bool handle_hnop(struct pt_regs *regs)
+{
+	unsigned char buf[MAX_INSN_SIZE];
+	unsigned long nr_copied;
+	struct insn insn;
+
+	if (!IS_ENABLED(CONFIG_X86_HNOP_EMU))
+		return false;
+
+	nr_copied = insn_fetch_from_user(regs, buf);
+	if (nr_copied <= 0)
+		return false;
+
+	if (!insn_decode_from_regs(&insn, regs, buf, nr_copied))
+		return false;
+
+	/* Hintable NOPs cover 0F 18 to 0F 1F */
+	if (insn.opcode.bytes[0] != 0x0F ||
+	    insn.opcode.bytes[1] < 0x18 || insn.opcode.bytes[1] > 0x1F)
+		return false;
+
+	pr_warn_once("%s[%d] uses hintable NOPs that your processor does not support.\n"
+		     "The kernel is emulating them; the performance of this "
+		     "and other executables using them will be impacted.\n",
+		     current->comm, task_pid_nr(current));
+
+	regs->ip += insn.length;
+	return true;
+}
+
 #ifdef CONFIG_X86_F00F_BUG
 void handle_invalid_op(struct pt_regs *regs)
 #else
 static inline void handle_invalid_op(struct pt_regs *regs)
 #endif
 {
+	if (user_mode(regs) && handle_hnop(regs))
+		return;
+
 	do_error_trap(regs, 0, "invalid opcode", X86_TRAP_UD, SIGILL,
 		      ILL_ILLOPN, error_get_trap_addr(regs));
 }
-- 
2.34.1

Re: [PATCH v2] x86: add hintable NOPs emulation

[H. Peter Anvin]

On August 20, 2025 4:04:35 AM PDT, Marcos Del Sol Vives <marcos@orca.pet> wrote:
>Hintable NOPs are a series of instructions introduced by Intel with the
>Pentium Pro (i686), and described in US patent US5701442A.
>
>These instructions were reserved to allow backwards-compatible changes
>in the instruction set possible, by having old processors treat them as
>variable-length NOPs, while having other semantics in modern processors.
>
>Some modern uses are:
> - Multi-byte/long NOPs
> - Indirect Branch Tracking (ENDBR32)
> - Shadow Stack (part of CET)
>
>Some processors advertising i686 compatibility lack full support for
>them, which may cause #UD to be incorrectly triggered, crashing software
>that uses then with an unexpected SIGILL.
>
>One such software is sudo in Debian bookworm, which is compiled with
>GCC -fcf-protection=branch and contains ENDBR32 instructions. It crashes
>on my Vortex86DX3 processor and VIA C3 Nehalem processors [1].
>
>This patch is a much simplified version of my previous patch for x86
>instruction emulation [2], that only emulates hintable NOPs.
>
>When #UD is raised, it checks if the opcode corresponds to a hintable NOP
>in user space. If true, it warns the user via the dmesg and advances the
>instruction pointer, thus emulating its expected NOP behaviour.
>
>[1]: https://lists.debian.org/debian-devel/2023/10/msg00118.html
>[2]: https://lore.kernel.org/all/20210626130313.1283485-1-marcos@orca.pet/
>
>Signed-off-by: Marcos Del Sol Vives <marcos@orca.pet>
>---
> arch/x86/Kconfig        | 29 +++++++++++++++++++++++++++++
> arch/x86/kernel/traps.c | 33 +++++++++++++++++++++++++++++++++
> 2 files changed, 62 insertions(+)
>
>diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>index 58d890fe2100..a6daebdc2573 100644
>--- a/arch/x86/Kconfig
>+++ b/arch/x86/Kconfig
>@@ -1286,6 +1286,35 @@ config X86_IOPL_IOPERM
> 	  ability to disable interrupts from user space which would be
> 	  granted if the hardware IOPL mechanism would be used.
> 
>+config X86_HNOP_EMU
>+	bool "Hintable NOPs emulation"
>+	depends on X86_32
>+	default y
>+	help
>+	  Hintable NOPs are a series of instructions introduced by Intel with
>+	  the Pentium Pro (i686), and described in US patent US5701442A.
>+
>+	  These instructions were reserved to allow backwards-compatible
>+	  changes in the instruction set possible, by having old processors
>+	  treat them as variable-length NOPs, while having other semantics in
>+	  modern processors.
>+
>+	  Some modern uses are:
>+	   - Multi-byte/long NOPs
>+	   - Indirect Branch Tracking (ENDBR32)
>+	   - Shadow Stack (part of CET)
>+
>+	  Some processors advertising i686 compatibility (such as Cyrix MII,
>+	  VIA C3 Nehalem or DM&P Vortex86DX3) lack full support for them,
>+	  which may cause SIGILL to be incorrectly raised in user space when
>+	  a hintable NOP is encountered.
>+
>+	  Say Y here if you want the kernel to emulate them, allowing programs
>+	  that make use of them to run transparently on such processors.
>+
>+	  This emulation has no performance penalty for processors that
>+	  properly support them, so if unsure, enable it.
>+
> config TOSHIBA
> 	tristate "Toshiba Laptop support"
> 	depends on X86_32
>diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
>index 36354b470590..22b51c4186e7 100644
>--- a/arch/x86/kernel/traps.c
>+++ b/arch/x86/kernel/traps.c
>@@ -295,12 +295,45 @@ DEFINE_IDTENTRY(exc_overflow)
> 	do_error_trap(regs, 0, "overflow", X86_TRAP_OF, SIGSEGV, 0, NULL);
> }
> 
>+static bool handle_hnop(struct pt_regs *regs)
>+{
>+	unsigned char buf[MAX_INSN_SIZE];
>+	unsigned long nr_copied;
>+	struct insn insn;
>+
>+	if (!IS_ENABLED(CONFIG_X86_HNOP_EMU))
>+		return false;
>+
>+	nr_copied = insn_fetch_from_user(regs, buf);
>+	if (nr_copied <= 0)
>+		return false;
>+
>+	if (!insn_decode_from_regs(&insn, regs, buf, nr_copied))
>+		return false;
>+
>+	/* Hintable NOPs cover 0F 18 to 0F 1F */
>+	if (insn.opcode.bytes[0] != 0x0F ||
>+	    insn.opcode.bytes[1] < 0x18 || insn.opcode.bytes[1] > 0x1F)
>+		return false;
>+
>+	pr_warn_once("%s[%d] uses hintable NOPs that your processor does not support.\n"
>+		     "The kernel is emulating them; the performance of this "
>+		     "and other executables using them will be impacted.\n",
>+		     current->comm, task_pid_nr(current));
>+
>+	regs->ip += insn.length;
>+	return true;
>+}
>+
> #ifdef CONFIG_X86_F00F_BUG
> void handle_invalid_op(struct pt_regs *regs)
> #else
> static inline void handle_invalid_op(struct pt_regs *regs)
> #endif
> {
>+	if (user_mode(regs) && handle_hnop(regs))
>+		return;
>+
> 	do_error_trap(regs, 0, "invalid opcode", X86_TRAP_UD, SIGILL,
> 		      ILL_ILLOPN, error_get_trap_addr(regs));
> }

Interesting that they build with endbr32 support, since the kernel vdso isn't endbr32 compliant, as I found...

Re: [PATCH v2] x86: add hintable NOPs emulation

[kernel test robot]

Hello,

kernel test robot noticed "BUG:sleeping_function_called_from_invalid_context_at_include/linux/uaccess.h" on:

commit: 09c737e0df5a3dbf40e8da1d6e168bd6d7fd19f0 ("[PATCH v2] x86: add hintable NOPs emulation")
url: https://github.com/intel-lab-lkp/linux/commits/Marcos-Del-Sol-Vives/x86-add-hintable-NOPs-emulation/20250820-190618
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 894af4a1cde61c3401f237184fb770f72ff12df8
patch link: https://lore.kernel.org/all/20250820110437.560107-1-marcos@orca.pet/
patch subject: [PATCH v2] x86: add hintable NOPs emulation

in testcase: trinity
version: trinity-i386-abe9de86-1_20230429
with following parameters:

	runtime: 600s



config: i386-randconfig-004-20250827
compiler: clang-20
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202508291620.bcfb3924-lkp@intel.com


[   24.176151][ T2696] BUG: sleeping function called from invalid context at include/linux/uaccess.h:162
[   24.176703][ T2696] in_atomic(): 0, irqs_disabled(): 1, non_block: 0, pid: 2696, name: trinity-c4
[   24.177213][ T2696] preempt_count: 0, expected: 0
[   24.177492][ T2696] no locks held by trinity-c4/2696.
[   24.177788][ T2696] irq event stamp: 335112
[ 24.178030][ T2696] hardirqs last enabled at (335111): irqentry_exit (kernel/entry/common.c:210) 
[ 24.178521][ T2696] hardirqs last disabled at (335112): irqentry_enter (kernel/entry/common.c:?) 
[ 24.179004][ T2696] softirqs last enabled at (332212): __do_softirq (kernel/softirq.c:614) 
[ 24.179473][ T2696] softirqs last disabled at (332207): __do_softirq (kernel/softirq.c:614) 
[   24.179948][ T2696] CPU: 1 UID: 65534 PID: 2696 Comm: trinity-c4 Tainted: G                T   6.17.0-rc2-00017-g09c737e0df5a #1 VOLUNTARY
[   24.179952][ T2696] Tainted: [T]=RANDSTRUCT
[   24.179954][ T2696] Call Trace:
[ 24.179956][ T2696] __dump_stack (lib/dump_stack.c:95) 
[ 24.179961][ T2696] dump_stack_lvl (lib/dump_stack.c:123) 
[ 24.179963][ T2696] ? nbcon_get_cpu_emergency_nesting (kernel/printk/nbcon.c:1375) 
[ 24.179967][ T2696] dump_stack (lib/dump_stack.c:129) 
[ 24.179969][ T2696] __might_resched (kernel/sched/core.c:8958) 
[ 24.179976][ T2696] __might_sleep (kernel/sched/core.c:8887) 
[ 24.179979][ T2696] __might_fault (mm/memory.c:6957) 
[ 24.179983][ T2696] _copy_from_user (include/linux/uaccess.h:?) 
[ 24.179991][ T2696] insn_fetch_from_user (include/linux/uaccess.h:212 arch/x86/lib/insn-eval.c:1516) 
[ 24.179995][ T2696] handle_invalid_op (arch/x86/kernel/traps.c:308) 
[ 24.180010][ T2696] ? exc_overflow (arch/x86/kernel/traps.c:417) 
[ 24.180012][ T2696] exc_invalid_op (arch/x86/kernel/traps.c:432) 
[ 24.180014][ T2696] handle_exception (arch/x86/entry/entry_32.S:1055) 
[   24.180017][ T2696] EIP: 0x434433
[ 24.180025][ T2696] Code: d8 ba 3b ab 45 74 a5 5b 8a ef a2 59 f6 b5 0c b9 82 33 c3 f1 23 2e da 6f 60 f0 65 13 13 44 c5 b7 3a 06 f6 a2 3c 53 db 3b 28 30 <f0> ce 36 c2 77 c0 2b 9b fd 83 f2 04 89 08 45 5d 8b 0b c3 d0 8a 38
All code
========
   0:	d8 ba 3b ab 45 74    	fdivrs 0x7445ab3b(%rdx)
   6:	a5                   	movsl  %ds:(%rsi),%es:(%rdi)
   7:	5b                   	pop    %rbx
   8:	8a ef                	mov    %bh,%ch
   a:	a2 59 f6 b5 0c b9 82 	movabs %al,0xc33382b90cb5f659
  11:	33 c3 
  13:	f1                   	int1
  14:	23 2e                	and    (%rsi),%ebp
  16:	da 6f 60             	fisubrl 0x60(%rdi)
  19:	f0 65 13 13          	lock adc %gs:(%rbx),%edx
  1d:	44 c5 b7 3a          	(bad)
  21:	06                   	(bad)
  22:	f6 a2 3c 53 db 3b    	mulb   0x3bdb533c(%rdx)
  28:	28 30                	sub    %dh,(%rax)
  2a:*	f0 ce                	lock (bad)		<-- trapping instruction
  2c:	36 c2 77 c0          	ss ret $0xc077
  30:	2b 9b fd 83 f2 04    	sub    0x4f283fd(%rbx),%ebx
  36:	89 08                	mov    %ecx,(%rax)
  38:	45 5d                	rex.RB pop %r13
  3a:	8b 0b                	mov    (%rbx),%ecx
  3c:	c3                   	ret
  3d:	d0                   	.byte 0xd0
  3e:	8a 38                	mov    (%rax),%bh

Code starting with the faulting instruction
===========================================
   0:	f0 ce                	lock (bad)
   2:	36 c2 77 c0          	ss ret $0xc077
   6:	2b 9b fd 83 f2 04    	sub    0x4f283fd(%rbx),%ebx
   c:	89 08                	mov    %ecx,(%rax)
   e:	45 5d                	rex.RB pop %r13
  10:	8b 0b                	mov    (%rbx),%ecx
  12:	c3                   	ret
  13:	d0                   	.byte 0xd0
  14:	8a 38                	mov    (%rax),%bh
[   24.180028][ T2696] EAX: 00400000 EBX: 00465000 ECX: 00058000 EDX: 00100000
[   24.180030][ T2696] ESI: b72df000 EDI: b7f53234 EBP: b72df030 ESP: bfd7948c
[   24.180031][ T2696] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010246
[ 24.180036][ T2696] ? exc_overflow (arch/x86/kernel/traps.c:417) 


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250829/202508291620.bcfb3924-lkp@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v2] x86: add hintable NOPs emulation

[Marcos Del Sol Vives]

El 30/08/2025 a las 8:56, kernel test robot escribió:
> [   24.176151][ T2696] BUG: sleeping function called from invalid context at include/linux/uaccess.h:162
> [   24.176703][ T2696] in_atomic(): 0, irqs_disabled(): 1, non_block: 0, pid: 2696, name: trinity-c4
> [   24.177213][ T2696] preempt_count: 0, expected: 0
> [   24.177492][ T2696] no locks held by trinity-c4/2696.
> [   24.177788][ T2696] irq event stamp: 335112
> [ 24.178030][ T2696] hardirqs last enabled at (335111): irqentry_exit (kernel/entry/common.c:210) 
> [ 24.178521][ T2696] hardirqs last disabled at (335112): irqentry_enter (kernel/entry/common.c:?) 
> [ 24.179004][ T2696] softirqs last enabled at (332212): __do_softirq (kernel/softirq.c:614) 
> [ 24.179473][ T2696] softirqs last disabled at (332207): __do_softirq (kernel/softirq.c:614) 
> [   24.179948][ T2696] CPU: 1 UID: 65534 PID: 2696 Comm: trinity-c4 Tainted: G                T   6.17.0-rc2-00017-g09c737e0df5a #1 VOLUNTARY
> [   24.179952][ T2696] Tainted: [T]=RANDSTRUCT
> [   24.179954][ T2696] Call Trace:
> [ 24.179956][ T2696] __dump_stack (lib/dump_stack.c:95) 
> [ 24.179961][ T2696] dump_stack_lvl (lib/dump_stack.c:123) 
> [ 24.179963][ T2696] ? nbcon_get_cpu_emergency_nesting (kernel/printk/nbcon.c:1375) 
> [ 24.179967][ T2696] dump_stack (lib/dump_stack.c:129) 
> [ 24.179969][ T2696] __might_resched (kernel/sched/core.c:8958) 
> [ 24.179976][ T2696] __might_sleep (kernel/sched/core.c:8887) 
> [ 24.179979][ T2696] __might_fault (mm/memory.c:6957) 
> [ 24.179983][ T2696] _copy_from_user (include/linux/uaccess.h:?) 
> [ 24.179991][ T2696] insn_fetch_from_user (include/linux/uaccess.h:212 arch/x86/lib/insn-eval.c:1516) 
> [ 24.179995][ T2696] handle_invalid_op (arch/x86/kernel/traps.c:308) 
> [ 24.180010][ T2696] ? exc_overflow (arch/x86/kernel/traps.c:417) 
> [ 24.180012][ T2696] exc_invalid_op (arch/x86/kernel/traps.c:432) 
> [ 24.180014][ T2696] handle_exception (arch/x86/entry/entry_32.S:1055)

I am familiar with interrupts on microcontrollers and embedded systems,
but not with Linux's, so unsure how to proceed.

I've seen UMIP and IOPL emulation and they both run with interrupts enabled,
by means of cond_local_irq_enable, and then fetch from memory using regular
insn_fetch_from_user/get_user which may sleep.

VC, on the other hand, uses the insn_fetch_from_user_inatomic.

Can someone chime in on what should I do for this? Enable IRQs temporarily
using cond_local_irq_enable or local_irq_enable, or use the inatomic
version?

Re: [PATCH v2] x86: add hintable NOPs emulation

[H. Peter Anvin]

On August 31, 2025 7:34:05 AM PDT, Marcos Del Sol Vives <marcos@orca.pet> wrote:
>El 30/08/2025 a las 8:56, kernel test robot escribió:
>> [   24.176151][ T2696] BUG: sleeping function called from invalid context at include/linux/uaccess.h:162
>> [   24.176703][ T2696] in_atomic(): 0, irqs_disabled(): 1, non_block: 0, pid: 2696, name: trinity-c4
>> [   24.177213][ T2696] preempt_count: 0, expected: 0
>> [   24.177492][ T2696] no locks held by trinity-c4/2696.
>> [   24.177788][ T2696] irq event stamp: 335112
>> [ 24.178030][ T2696] hardirqs last enabled at (335111): irqentry_exit (kernel/entry/common.c:210) 
>> [ 24.178521][ T2696] hardirqs last disabled at (335112): irqentry_enter (kernel/entry/common.c:?) 
>> [ 24.179004][ T2696] softirqs last enabled at (332212): __do_softirq (kernel/softirq.c:614) 
>> [ 24.179473][ T2696] softirqs last disabled at (332207): __do_softirq (kernel/softirq.c:614) 
>> [   24.179948][ T2696] CPU: 1 UID: 65534 PID: 2696 Comm: trinity-c4 Tainted: G                T   6.17.0-rc2-00017-g09c737e0df5a #1 VOLUNTARY
>> [   24.179952][ T2696] Tainted: [T]=RANDSTRUCT
>> [   24.179954][ T2696] Call Trace:
>> [ 24.179956][ T2696] __dump_stack (lib/dump_stack.c:95) 
>> [ 24.179961][ T2696] dump_stack_lvl (lib/dump_stack.c:123) 
>> [ 24.179963][ T2696] ? nbcon_get_cpu_emergency_nesting (kernel/printk/nbcon.c:1375) 
>> [ 24.179967][ T2696] dump_stack (lib/dump_stack.c:129) 
>> [ 24.179969][ T2696] __might_resched (kernel/sched/core.c:8958) 
>> [ 24.179976][ T2696] __might_sleep (kernel/sched/core.c:8887) 
>> [ 24.179979][ T2696] __might_fault (mm/memory.c:6957) 
>> [ 24.179983][ T2696] _copy_from_user (include/linux/uaccess.h:?) 
>> [ 24.179991][ T2696] insn_fetch_from_user (include/linux/uaccess.h:212 arch/x86/lib/insn-eval.c:1516) 
>> [ 24.179995][ T2696] handle_invalid_op (arch/x86/kernel/traps.c:308) 
>> [ 24.180010][ T2696] ? exc_overflow (arch/x86/kernel/traps.c:417) 
>> [ 24.180012][ T2696] exc_invalid_op (arch/x86/kernel/traps.c:432) 
>> [ 24.180014][ T2696] handle_exception (arch/x86/entry/entry_32.S:1055)
>
>I am familiar with interrupts on microcontrollers and embedded systems,
>but not with Linux's, so unsure how to proceed.
>
>I've seen UMIP and IOPL emulation and they both run with interrupts enabled,
>by means of cond_local_irq_enable, and then fetch from memory using regular
>insn_fetch_from_user/get_user which may sleep.
>
>VC, on the other hand, uses the insn_fetch_from_user_inatomic.
>
>Can someone chime in on what should I do for this? Enable IRQs temporarily
>using cond_local_irq_enable or local_irq_enable, or use the inatomic
>version?

I'm really, *really* starting to question this approach. 

The right thing would be to not compile in endbr instructions on 32 bits, since they aren't supported by the kernel (if you try to enable CET/IBT you will crash instantly, because there is no endbr32 at the vdso system can entry point right now.)

We are talking about a very small number of CPUs stuck between generation 5 and generation 6. Most likely we are talking about a much smaller number than the i486 support we have been debating lately...

Re: [PATCH v2] x86: add hintable NOPs emulation

[David Laight]

On Sun, 31 Aug 2025 16:34:05 +0200
Marcos Del Sol Vives <marcos@orca.pet> wrote:

> El 30/08/2025 a las 8:56, kernel test robot escribió:
> > [   24.176151][ T2696] BUG: sleeping function called from invalid context at include/linux/uaccess.h:162
> > [   24.176703][ T2696] in_atomic(): 0, irqs_disabled(): 1, non_block: 0, pid: 2696, name: trinity-c4
> > [   24.177213][ T2696] preempt_count: 0, expected: 0
> > [   24.177492][ T2696] no locks held by trinity-c4/2696.
> > [   24.177788][ T2696] irq event stamp: 335112
> > [ 24.178030][ T2696] hardirqs last enabled at (335111): irqentry_exit (kernel/entry/common.c:210) 
> > [ 24.178521][ T2696] hardirqs last disabled at (335112): irqentry_enter (kernel/entry/common.c:?) 
> > [ 24.179004][ T2696] softirqs last enabled at (332212): __do_softirq (kernel/softirq.c:614) 
> > [ 24.179473][ T2696] softirqs last disabled at (332207): __do_softirq (kernel/softirq.c:614) 
> > [   24.179948][ T2696] CPU: 1 UID: 65534 PID: 2696 Comm: trinity-c4 Tainted: G                T   6.17.0-rc2-00017-g09c737e0df5a #1 VOLUNTARY
> > [   24.179952][ T2696] Tainted: [T]=RANDSTRUCT
> > [   24.179954][ T2696] Call Trace:
> > [ 24.179956][ T2696] __dump_stack (lib/dump_stack.c:95) 
> > [ 24.179961][ T2696] dump_stack_lvl (lib/dump_stack.c:123) 
> > [ 24.179963][ T2696] ? nbcon_get_cpu_emergency_nesting (kernel/printk/nbcon.c:1375) 
> > [ 24.179967][ T2696] dump_stack (lib/dump_stack.c:129) 
> > [ 24.179969][ T2696] __might_resched (kernel/sched/core.c:8958) 
> > [ 24.179976][ T2696] __might_sleep (kernel/sched/core.c:8887) 
> > [ 24.179979][ T2696] __might_fault (mm/memory.c:6957) 
> > [ 24.179983][ T2696] _copy_from_user (include/linux/uaccess.h:?) 
> > [ 24.179991][ T2696] insn_fetch_from_user (include/linux/uaccess.h:212 arch/x86/lib/insn-eval.c:1516) 
> > [ 24.179995][ T2696] handle_invalid_op (arch/x86/kernel/traps.c:308) 
> > [ 24.180010][ T2696] ? exc_overflow (arch/x86/kernel/traps.c:417) 
> > [ 24.180012][ T2696] exc_invalid_op (arch/x86/kernel/traps.c:432) 
> > [ 24.180014][ T2696] handle_exception (arch/x86/entry/entry_32.S:1055)  
> 
> I am familiar with interrupts on microcontrollers and embedded systems,
> but not with Linux's, so unsure how to proceed.
> 
> I've seen UMIP and IOPL emulation and they both run with interrupts enabled,
> by means of cond_local_irq_enable, and then fetch from memory using regular
> insn_fetch_from_user/get_user which may sleep.
> 
> VC, on the other hand, uses the insn_fetch_from_user_inatomic.
> 
> Can someone chime in on what should I do for this? Enable IRQs temporarily
> using cond_local_irq_enable or local_irq_enable, or use the inatomic
> version?
> 

My 2c:
Enabling interrupts might have all sorts of side effects.
In this case it should be ok to use the 'inatomic' read of userspace
that will fail in the very unlikely case where the page got unmapped
between userpace executing the instruction and the trap handler
trying to read it.
If that happens just return back to userspace and re-execute the 'bad'
instruction - it will fault again and hopefully you'll manage to
read it the second time (or eventually).

I guess there might be a problem if the cpu is faulting on the first
few bytes of the instruction and a malicious program has put those
bytes right at the end of a page - and not mapped the following page.
You'd need to differentiate those from the valid:
	for (i = 0; i < 256; i++)
		hintable_nop();

	David

Re: [PATCH v2] x86: add hintable NOPs emulation

[Marcos Del Sol Vives]

El 31/08/2025 a las 21:32, H. Peter Anvin escribió:
> I'm really, *really* starting to question this approach. 
> 
> The right thing would be to not compile in endbr instructions on 32 bits,
> since they aren't supported by the kernel (if you try to enable CET/IBT
> you will crash instantly, because there is no endbr32 at the vdso system
> can entry point right now.)
> 
> We are talking about a very small number of CPUs stuck between generation
> 5 and generation 6. Most likely we are talking about a much smaller number
> than the i486 support we have been debating lately...

I assume the kernel is currently simply ignoring the CET flag on ELF for
all 32-bit x86 binaries and libraries?

Anyhow, seeing this is indeed becoming more tricky and error-prone, I'm
contating the sudo developer which is, so far, the only software that
has this issue.

Re: [PATCH v2] x86: add hintable NOPs emulation

[H. Peter Anvin]

On September 1, 2025 4:43:15 AM PDT, Marcos Del Sol Vives <marcos@orca.pet> wrote:
>El 31/08/2025 a las 21:32, H. Peter Anvin escribió:
>> I'm really, *really* starting to question this approach. 
>> 
>> The right thing would be to not compile in endbr instructions on 32 bits,
>> since they aren't supported by the kernel (if you try to enable CET/IBT
>> you will crash instantly, because there is no endbr32 at the vdso system
>> can entry point right now.)
>> 
>> We are talking about a very small number of CPUs stuck between generation
>> 5 and generation 6. Most likely we are talking about a much smaller number
>> than the i486 support we have been debating lately...
>
>I assume the kernel is currently simply ignoring the CET flag on ELF for
>all 32-bit x86 binaries and libraries?
>
>Anyhow, seeing this is indeed becoming more tricky and error-prone, I'm
>contating the sudo developer which is, so far, the only software that
>has this issue.

Yep.

Re: [PATCH v2] x86: add hintable NOPs emulation

[Marc Haber]

Hello,

I am one of the maintainers of sudo in Debian. The submitter of this 
kernel patch has been lobbying for changes to sudo for a longer time, 
asking me to disable -fcf-protection in the Debian packages for i386.

I am reluctant do doing so because I don't want to disable a security 
feature for the complete distribution just because it doesn't work for a 
rather exotic CPU family, that - to my understanding but please correct 
me if I'm wrong - claims to be i686 but actually isn't.

On Mon, Sep 01, 2025 at 02:28:04PM -0700, H. Peter Anvin wrote:
>On September 1, 2025 4:43:15 AM PDT, Marcos Del Sol Vives <marcos@orca.pet> wrote:
>>I assume the kernel is currently simply ignoring the CET flag on ELF for
>>all 32-bit x86 binaries and libraries?
>>
>>Anyhow, seeing this is indeed becoming more tricky and error-prone, I'm
>>contating the sudo developer which is, so far, the only software that
>>has this issue.
>
>Yep.

So you're saying that -fcf-protection is basically a no-op on i386 and 
will never have an effect?

For me, as the maintainer of the Debian package, this is an academic 
question. The change that the submitter wants in Debian will not happen 
in Debian unless the technical committee overrides my decision. And 
Debian oldstable is the last release that had full support for the i386 
architecture anyway. We reduced i386 support to be a "partial 
architecture" in current stable, targeting that architecture for 
containers and multi-arch systems with an amd64 kernel.

Will -fcf-protection have an effect when an i386 sudo is being executed 
with a amd64 kernel?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: [PATCH v2] x86: add hintable NOPs emulation

[Marcos Del Sol Vives]

El 02/09/2025 a las 10:16, Marc Haber escribió:
> So you're saying that -fcf-protection is basically a no-op on i386 and
> will never have an effect?

Feel free to wait for the reply of someone that is not "lobbying" for
this, but the documentation seems fairly clear on this [1]:

"Today in the 64-bit kernel, only userspace shadow stack and kernel IBT are
supported."

Checking the kernel trap handler itself [2]:

DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
{
	if (user_mode(regs)) {
		if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
			do_user_cp_fault(regs, error_code);
		else
			do_unexpected_cp(regs, error_code);
	} else {
		if (cpu_feature_enabled(X86_FEATURE_IBT))
			do_kernel_cp_fault(regs, error_code);
		else
			do_unexpected_cp(regs, error_code);
	}
}

And shadow stacks, the user mode protection, is explicitely "not supported
for 32 bit" [3]:

static int shstk_setup(void)
{
	struct thread_shstk *shstk = &current->thread.shstk;
	unsigned long addr, size;

	/* Already enabled */
	if (features_enabled(ARCH_SHSTK_SHSTK))
		return 0;

	/* Also not supported for 32 bit */
	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) || in_ia32_syscall())
		return -EOPNOTSUPP;
	...
}

"in_ia32_syscall" expands to constant "true" on 32-bit kernels, and
"not in 32-bit compatibility mode" for 64-bit kernels.

So TL;DR: unless I'm mistaken, the problematic ENDBRs do at the moment,
not even in 64-bit, offer any security improvements for user-mode
applications. Only shadow stacks are supported, and only in 64-bit.

1: https://docs.kernel.org/arch/x86/shstk.html
2: https://github.com/torvalds/linux/blob/v6.17-rc4/arch/x86/kernel/cet.c#L149-L162
3: https://github.com/torvalds/linux/blob/v6.17-rc4/arch/x86/kernel/shstk.c#L166-L168