From: Peter Zijlstra <peterz@infradead.org>
To: x86@kernel.org
Cc: linux-kernel@vger.kernel.org, peterz@infradead.org
Subject: [PATCH 2/2] x86/entry: Harden return-to-user
Date: Mon, 20 Nov 2023 15:33:46 +0100 [thread overview]
Message-ID: <20231120143626.753200755@infradead.org> (raw)
In-Reply-To: 20231120143344.584345121@infradead.org
Make the DEBUG_ENTRY check that validates CS is a user segment
unconditional and move it nearer to IRET.
PRE:
140,026,608 cycles:k ( +- 0.01% )
236,696,176 instructions:k # 1.69 insn per cycle ( +- 0.00% )
POST:
139,957,681 cycles:k ( +- 0.01% )
236,681,819 instructions:k # 1.69 insn per cycle ( +- 0.00% )
(this is with --repeat 100 and the run-to-run variance is bigger than
the difference shown)
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/entry/entry_64.S | 18 ++++++++----------
1 file changed, 8 insertions(+), 10 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -559,13 +559,6 @@ SYM_CODE_END(\asmsym)
SYM_CODE_START_LOCAL(common_interrupt_return)
SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL)
IBRS_EXIT
-#ifdef CONFIG_DEBUG_ENTRY
- /* Assert that pt_regs indicates user mode. */
- testb $3, CS(%rsp)
- jnz 1f
- ud2
-1:
-#endif
#ifdef CONFIG_XEN_PV
ALTERNATIVE "", "jmp xenpv_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV
#endif
@@ -576,8 +569,14 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_
STACKLEAK_ERASE
POP_REGS
add $8, %rsp /* orig_ax */
+ UNWIND_HINT_IRET_REGS
+
+.Lswapgs_and_iret:
swapgs
- jmp .Lnative_iret
+ /* Assert that the IRET frame indicates user mode. */
+ testb $3, 8(%rsp)
+ jnz .Lnative_iret
+ ud2
#ifdef CONFIG_PAGE_TABLE_ISOLATION
.Lpti_restore_regs_and_return_to_usermode:
@@ -613,8 +612,7 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_
/* Restore RDI. */
popq %rdi
- swapgs
- jmp .Lnative_iret
+ jmp .Lswapgs_and_iret
#endif
SYM_INNER_LABEL(restore_regs_and_return_to_kernel, SYM_L_GLOBAL)
next prev parent reply other threads:[~2023-11-20 14:38 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-20 14:33 [PATCH 0/2] x86/entry: Trade cycles Peter Zijlstra
2023-11-20 14:33 ` [PATCH 1/2] x86/entry: Optimize common_interrupt_return() Peter Zijlstra
2023-11-21 13:04 ` [tip: x86/entry] " tip-bot2 for Peter Zijlstra
2023-11-20 14:33 ` Peter Zijlstra [this message]
2023-11-21 13:04 ` [tip: x86/entry] x86/entry: Harden return-to-user tip-bot2 for Peter Zijlstra
2023-11-21 15:26 ` [PATCH 0/2] x86/entry: Trade cycles Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231120143626.753200755@infradead.org \
--to=peterz@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox