[PATCH v2] x86: entry_32.S: change ESPFIX test to not touch PT_OLDSS(%esp)

[PATCH v2] x86: entry_32.S: change ESPFIX test to not touch PT_OLDSS(%esp)

[Denys Vlasenko]

Old code was trying to avoid having three branch insns,
but instead it has a chain of six insns where each insn
depends on previos one.

And it was touching PT_OLDSS(%esp) unconditionally, even when it may
contain bogus data. Elsewhere we have to jump thru hoops
just to make sure here PT_OLDSS(%esp) is at least in a valid page.

All this just to have one branch instead of three?

The new code simply checks each condition.
All three checks can run in parallel on an out-of-order CPU.
Most of the time, none of branches will be taken.

Comparison of object code:
    Old:
     1e6:   8b 44 24 38             mov    0x38(%esp),%eax
     1ea:   8a 64 24 40             mov    0x40(%esp),%ah
     1ee:   8a 44 24 34             mov    0x34(%esp),%al
     1f2:   25 03 04 02 00          and    $0x20403,%eax
     1f7:   3d 03 04 00 00          cmp    $0x403,%eax
     1fc:   74 0f                   je     20d <ldt_ss>
    New:
     1e6:   f6 44 24 3a 02          testb  $0x2,0x3a(%esp)
     1eb:   75 0e                   jne    1fb <restore_nocheck>
     1ed:   f6 44 24 34 03          testb  $0x3,0x34(%esp)
     1f2:   74 07                   je     1fb <restore_nocheck>
     1f4:   f6 44 24 40 04          testb  $0x4,0x40(%esp)
     1f9:   75 0f                   jne    20a <ldt_ss>

Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
CC: Linus Torvalds <torvalds@linux-foundation.org>
CC: Steven Rostedt <rostedt@goodmis.org>
CC: Ingo Molnar <mingo@kernel.org>
CC: Borislav Petkov <bp@alien8.de>
CC: "H. Peter Anvin" <hpa@zytor.com>
CC: Andy Lutomirski <luto@amacapital.net>
CC: Oleg Nesterov <oleg@redhat.com>
CC: Frederic Weisbecker <fweisbec@gmail.com>
CC: Alexei Starovoitov <ast@plumgrid.com>
CC: Will Drewry <wad@chromium.org>
CC: Kees Cook <keescook@chromium.org>
CC: x86@kernel.org
CC: linux-kernel@vger.kernel.org
---
Changes since v1:
  shorter insn to test EFLAGS.VM (suggested by Linus)
  check EFLAGS.VM only if CONFIG_VM86

 arch/x86/kernel/entry_32.S | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
index e33ba51..d185b5e 100644
--- a/arch/x86/kernel/entry_32.S
+++ b/arch/x86/kernel/entry_32.S
@@ -516,16 +516,17 @@ restore_all:
 	TRACE_IRQS_IRET
 restore_all_notrace:
 #ifdef CONFIG_X86_ESPFIX32
-	movl PT_EFLAGS(%esp), %eax	# mix EFLAGS, SS and CS
-	# Warning: PT_OLDSS(%esp) contains the wrong/random values if we
-	# are returning to the kernel.
-	# See comments in process.c:copy_thread() for details.
-	movb PT_OLDSS(%esp), %ah
-	movb PT_CS(%esp), %al
-	andl $(X86_EFLAGS_VM | (SEGMENT_TI_MASK << 8) | SEGMENT_RPL_MASK), %eax
-	cmpl $((SEGMENT_LDT << 8) | USER_RPL), %eax
 	CFI_REMEMBER_STATE
-	je ldt_ss			# returning to user-space with LDT SS
+#ifdef CONFIG_VM86
+	testb	$2, (PT_EFLAGS+2)(%esp)
+	jnz	restore_nocheck		# EFLAGS.VM set, not it
+#endif
+	testb	$3, PT_CS(%esp)
+	jz	restore_nocheck		# CPL0, not it
+	# Note: we access PT_OLDSS only when we know it exists.
+	# If PT_CS is from CPL0, it does not exist.
+	testb	$SEGMENT_LDT, PT_OLDSS(%esp)
+	jnz	ldt_ss			# returning to user-space with LDT SS
 #endif
 restore_nocheck:
 	RESTORE_REGS 4			# skip orig_eax/error_code
-- 
1.8.1.4

Re: [PATCH v2] x86: entry_32.S: change ESPFIX test to not touch PT_OLDSS(%esp)

[Ingo Molnar]

* Denys Vlasenko <dvlasenk@redhat.com> wrote:

> Old code was trying to avoid having three branch insns,
> but instead it has a chain of six insns where each insn
> depends on previos one.
> 
> And it was touching PT_OLDSS(%esp) unconditionally, even when it may
> contain bogus data. Elsewhere we have to jump thru hoops
> just to make sure here PT_OLDSS(%esp) is at least in a valid page.
> 
> All this just to have one branch instead of three?
> 
> The new code simply checks each condition.
> All three checks can run in parallel on an out-of-order CPU.
> Most of the time, none of branches will be taken.
> 
> Comparison of object code:
>     Old:
>      1e6:   8b 44 24 38             mov    0x38(%esp),%eax
>      1ea:   8a 64 24 40             mov    0x40(%esp),%ah
>      1ee:   8a 44 24 34             mov    0x34(%esp),%al
>      1f2:   25 03 04 02 00          and    $0x20403,%eax
>      1f7:   3d 03 04 00 00          cmp    $0x403,%eax
>      1fc:   74 0f                   je     20d <ldt_ss>
>     New:
>      1e6:   f6 44 24 3a 02          testb  $0x2,0x3a(%esp)
>      1eb:   75 0e                   jne    1fb <restore_nocheck>
>      1ed:   f6 44 24 34 03          testb  $0x3,0x34(%esp)
>      1f2:   74 07                   je     1fb <restore_nocheck>
>      1f4:   f6 44 24 40 04          testb  $0x4,0x40(%esp)
>      1f9:   75 0f                   jne    20a <ldt_ss>

Please do some benchmarking of this: a tight loop of getpid or getppid 
syscalls ought to be enough to be able to time this accurately.

Thanks,

	Ingo

Re: [PATCH v2] x86: entry_32.S: change ESPFIX test to not touch PT_OLDSS(%esp)

[Andy Lutomirski]

On Tue, Mar 10, 2015 at 12:57 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Denys Vlasenko <dvlasenk@redhat.com> wrote:
>
>> Old code was trying to avoid having three branch insns,
>> but instead it has a chain of six insns where each insn
>> depends on previos one.
>>
>> And it was touching PT_OLDSS(%esp) unconditionally, even when it may
>> contain bogus data. Elsewhere we have to jump thru hoops
>> just to make sure here PT_OLDSS(%esp) is at least in a valid page.
>>
>> All this just to have one branch instead of three?
>>
>> The new code simply checks each condition.
>> All three checks can run in parallel on an out-of-order CPU.
>> Most of the time, none of branches will be taken.
>>
>> Comparison of object code:
>>     Old:
>>      1e6:   8b 44 24 38             mov    0x38(%esp),%eax
>>      1ea:   8a 64 24 40             mov    0x40(%esp),%ah
>>      1ee:   8a 44 24 34             mov    0x34(%esp),%al
>>      1f2:   25 03 04 02 00          and    $0x20403,%eax
>>      1f7:   3d 03 04 00 00          cmp    $0x403,%eax
>>      1fc:   74 0f                   je     20d <ldt_ss>
>>     New:
>>      1e6:   f6 44 24 3a 02          testb  $0x2,0x3a(%esp)
>>      1eb:   75 0e                   jne    1fb <restore_nocheck>
>>      1ed:   f6 44 24 34 03          testb  $0x3,0x34(%esp)
>>      1f2:   74 07                   je     1fb <restore_nocheck>
>>      1f4:   f6 44 24 40 04          testb  $0x4,0x40(%esp)
>>      1f9:   75 0f                   jne    20a <ldt_ss>
>
> Please do some benchmarking of this: a tight loop of getpid or getppid
> syscalls ought to be enough to be able to time this accurately.

Before you benchmark, I think you should reorder it: check CS, then
OLDSS, then EFLAGS.  The case where CS & 3 == 0, OLDSS & 4 == 4, and
EFLAGS & VM == VM should be *extremely* rare.

I'm going to hold off on resending my sp0/sp1/ss cleanups until we
resolve this -- if it turns out that your code is the same or faster
and therefore gets merged, then I think that all the -8 crap can just
be deleted instead of being fixed.

--Andy

>
> Thanks,
>
>         Ingo



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: [PATCH v2] x86: entry_32.S: change ESPFIX test to not touch PT_OLDSS(%esp)

[Denys Vlasenko]

On Wed, Mar 11, 2015 at 4:35 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>>> Comparison of object code:
>>>     Old:
>>>      1e6:   8b 44 24 38             mov    0x38(%esp),%eax
>>>      1ea:   8a 64 24 40             mov    0x40(%esp),%ah
>>>      1ee:   8a 44 24 34             mov    0x34(%esp),%al
>>>      1f2:   25 03 04 02 00          and    $0x20403,%eax
>>>      1f7:   3d 03 04 00 00          cmp    $0x403,%eax
>>>      1fc:   74 0f                   je     20d <ldt_ss>
>>>     New:
>>>      1e6:   f6 44 24 3a 02          testb  $0x2,0x3a(%esp)
>>>      1eb:   75 0e                   jne    1fb <restore_nocheck>
>>>      1ed:   f6 44 24 34 03          testb  $0x3,0x34(%esp)
>>>      1f2:   74 07                   je     1fb <restore_nocheck>
>>>      1f4:   f6 44 24 40 04          testb  $0x4,0x40(%esp)
>>>      1f9:   75 0f                   jne    20a <ldt_ss>
>>
>> Please do some benchmarking of this: a tight loop of getpid or getppid
>> syscalls ought to be enough to be able to time this accurately.
>
> Before you benchmark, I think you should reorder it: check CS, then
> OLDSS, then EFLAGS.  The case where CS & 3 == 0, OLDSS & 4 == 4, and
> EFLAGS & VM == VM should be *extremely* rare.
>
> I'm going to hold off on resending my sp0/sp1/ss cleanups until we
> resolve this -- if it turns out that your code is the same or faster
> and therefore gets merged, then I think that all the -8 crap can just
> be deleted instead of being fixed.

I thought that I got NAKed by Linus on this change?

Basically, he wants to retain that padding because it guards against
_future_ bugs_ where someone would touch pt_regs->ss
and it will fail, very rarely. I think it's a sound reason to retain
the padding.


Anyway, since you ask. I benchmarked current code against this patch:

 restore_all_notrace:
 #ifdef CONFIG_X86_ESPFIX32
-       movl PT_EFLAGS(%esp), %eax      # mix EFLAGS, SS and CS
-       # Warning: PT_OLDSS(%esp) contains the wrong/random values if we
-       # are returning to the kernel.
-       # See comments in process.c:copy_thread() for details.
-       movb PT_OLDSS(%esp), %ah
-       movb PT_CS(%esp), %al
-       andl $(X86_EFLAGS_VM | (SEGMENT_TI_MASK << 8) | SEGMENT_RPL_MASK), %eax
-       cmpl $((SEGMENT_LDT << 8) | USER_RPL), %eax
        CFI_REMEMBER_STATE
-       je ldt_ss                       # returning to user-space with LDT SS
+       testb   $3, PT_CS(%esp)
+       jz      restore_nocheck         # CPL0, not it
+#ifdef CONFIG_VM86
+       testb   $2, (PT_EFLAGS+2)(%esp)
+       jnz     restore_nocheck         # EFLAGS.VM set, not it
+#endif
+       # Note: we access PT_OLDSS only when we know it exists.
+       # If PT_CS is from CPL0, it does not exist.
+       testb   $SEGMENT_LDT, PT_OLDSS(%esp)
+       jnz     ldt_ss                  # returning to user-space with LDT SS
 #endif

This code only gets executed on int80 path, not on sysenter.
The test was to run 10 million getpids.

The numbers I've got are *the same* before and after the patch:
227.99 ns per getpid().

I double-checked my test setup by adding a few PAUSE insns
in this code - which, as expected, was seen easily in the test run.
I did not forget to enable ESPFIX32 and VM86 in the .config.