Re: [patch 16/18] x86/entry: Get rid of the sys_ni_syscall() indirection

[Thomas Gleixner <tglx@kernel.org>]

On Fri, Oct 23 2026 at 08:38, H. Peter Anvin wrote:
> On October 23, 2026 7:24:22 AM PDT, Thomas Gleixner <tglx@kernel.org> wrote:
>>The real question is how we ended up there. By design or as an
>>implementation detail which got later set in stone and the selftest was
>>written to test the implementation instead of testing against a
>>specification.
>>
>>I went back in history and in 2.6.12 the behavior was for both i386 and
>>x86_64:
>>
>>  regs->ax = -ENOSYS;
>>  syscall_trace_enter();
>>  if (syscall_nr < MAX)
>>  	do_syscall();
>>  ....
>>
>>IOW, a tracer modification of regs->ax is not overwritten when the
>>syscall number is out of range.
>>
>>Between 2.6.19-rc2 and -rc3 x86_64 was changed to _always_ overwrite
>>regs->ax when the syscall number is out of range.
>>
>>In the 3.8 merge window seccomp introduced the -1 == skip bogosity which
>>got fixed up and made "consistent" in several changes afterwards. Just
>>that it never reached a state of consistency due to the overloading.
>>
>>So are we stuck with this behavior or do we sit down and define what the
>>correct behavior should be?
>>
>
> Going back on something from 3.8 that is actively used seems unwise. I
> fear that that ship sailed a long time ago, even though I hate it just
> as much as you do.

Well, overwriting the tracer written value for invalid syscall numbers
is something we can't undo. Though the interesting question is whether
the -1 overloading needs to be preserved. The new code uses returns
false when the syscall has to be aborted and that leaves the return code
untouched. When syscall_enter_trace() returns true, the syscall is
invoked, so it would be consistent to overwrite the return code with
-ENOSYS even for -1. But yeah, some esoteric nonsense might rely on that
behaviour for the wrong reasons.

Fix below.

Thanks,

        tglx
---
--- a/arch/x86/entry/syscall_32.c
+++ b/arch/x86/entry/syscall_32.c
@@ -77,6 +77,8 @@ static __always_inline void do_syscall_3
 	if (likely(nr < IA32_NR_syscalls)) {
 		nr = array_index_nospec(nr, IA32_NR_syscalls);
 		regs->ax = ia32_sys_call(regs, (unsigned int)nr);
+	} else {
+		regs->ax = -ENOSYS;
 	}
 }
 
--- a/arch/x86/entry/syscall_64.c
+++ b/arch/x86/entry/syscall_64.c
@@ -64,15 +64,20 @@ static __always_inline bool do_syscall_x
 	return false;
 }
 
-static __always_inline void do_syscall_x32(struct pt_regs *regs, unsigned long nr)
+static __always_inline bool do_syscall_x32(struct pt_regs *regs, unsigned long nr)
 {
+	if (!IS_ENABLED(CONFIG_X86_X32_ABI))
+		return false;
+
 	/* Adjust the starting offset of the table */
 	nr -= __X32_SYSCALL_BIT;
 
-	if (IS_ENABLED(CONFIG_X86_X32_ABI) && likely(nr < X32_NR_syscalls)) {
+	if (likely(nr < X32_NR_syscalls)) {
 		nr = array_index_nospec(nr, X32_NR_syscalls);
 		regs->ax = x32_sys_call(regs, (unsigned int)nr);
+		return true;
 	}
+	return false;
 }
 
 /* Returns true to return using SYSRET, or false to use IRET */
@@ -81,8 +86,21 @@ static __always_inline void do_syscall_x
 	if (likely(syscall_enter_from_user_mode_randomize_stack(regs, &nr))) {
 		instrumentation_begin();
 
-		if (!do_syscall_x64(regs, nr))
-			do_syscall_x32(regs, nr);
+		if (unlikely(!do_syscall_x64(regs, nr))) {
+			/*
+			 * The special treatment of syscall number -1 is a
+			 * historical bogosity as the original code used -1 to
+			 * indicate syscall abort, which is not distinguishable
+			 * from a syscall which is invoked with syscall number
+			 * -1. The historic behavior is to keep the return value
+			 * in regs->ax unmodified in that case, which means it's
+			 * either the preset value -ENOSYS or a value which was
+			 * written by a tracer. If the syscall number is invalid
+			 * but not -1 then overwrite any tracer modification.
+			 */
+			if (!do_syscall_x32(regs, nr) && nr != -1L)
+				regs->ax = -ENOSYS;
+		}
 
 		instrumentation_end();
 	}