From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S968591AbeCSSJF (ORCPT ); Mon, 19 Mar 2018 14:09:05 -0400 Received: from mga14.intel.com ([192.55.52.115]:62510 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S968536AbeCSSIz (ORCPT ); Mon, 19 Mar 2018 14:08:55 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,331,1517904000"; d="scan'208";a="39330595" From: "Chang S. Bae" To: x86@kernel.org Cc: luto@kernel.org, ak@linux.intel.com, hpa@zytor.com, markus.t.metzger@intel.com, tony.luck@intel.com, ravi.v.shankar@intel.com, linux-kernel@vger.kernel.org, chang.seok.bae@intel.com Subject: [PATCH 11/15] x86/fsgsbase/64: Preserve FS/GS state in __switch_to if FSGSBASE is on Date: Mon, 19 Mar 2018 10:49:23 -0700 Message-Id: <1521481767-22113-12-git-send-email-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1521481767-22113-1-git-send-email-chang.seok.bae@intel.com> References: <1521481767-22113-1-git-send-email-chang.seok.bae@intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Andy Lutomirski With the new FSGSBASE instructions, we can efficiently read and write the FS and GS bases in __switch_to. Use that capability to preserve the full state. This will enable user code to do whatever it wants with the new instructions without any kernel-induced gotchas. (There can still be architectural gotchas: movl %gs,%eax; movl %eax,%gs may change GSBASE if WRGSBASE was used, but users are expected to read the CPU manual before doing things like that.) This is a considerable speedup. It seems to save about 100 cycles per context switch compared to the baseline 4.6-rc1 behavior on my Skylake laptop. Signed-off-by: Andy Lutomirski [chang: 5~10% performance improvement on context switch micro- benchmark, when FS/GS base is actively switched.] Signed-off-by: Chang S. Bae Reviewed-by: Andi Kleen Cc: H. Peter Anvin --- arch/x86/kernel/process_64.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 877636a..7249a54 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -234,8 +234,18 @@ static __always_inline void save_fsgs(struct task_struct *task) { savesegment(fs, task->thread.fsindex); savesegment(gs, task->thread.gsindex); - save_base_legacy(task, task->thread.fsindex, FS); - save_base_legacy(task, task->thread.gsindex, GS); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + /* + * If FSGSBASE is enabled, we can't make any useful guesses + * about the base, and user code expects us to save the current + * value. Fortunately, reading the base directly is efficient. + */ + task->thread.fsbase = rdfsbase(); + task->thread.gsbase = rd_shadow_gsbase(); + } else { + save_base_legacy(task, task->thread.fsindex, FS); + save_base_legacy(task, task->thread.gsindex, GS); + } } static __always_inline void loadseg(enum which_selector which, @@ -300,10 +310,22 @@ static __always_inline void load_seg_legacy(unsigned short prev_index, static __always_inline void load_fsgs(struct thread_struct *prev, struct thread_struct *next) { - load_seg_legacy(prev->fsindex, prev->fsbase, - next->fsindex, next->fsbase, FS); - load_seg_legacy(prev->gsindex, prev->gsbase, - next->gsindex, next->gsbase, GS); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + /* Update the FS and GS selectors if they could have changed. */ + if (unlikely(prev->fsindex || next->fsindex)) + loadseg(FS, next->fsindex); + if (unlikely(prev->gsindex || next->gsindex)) + loadseg(GS, next->gsindex); + + /* Update the bases. */ + wrfsbase(next->fsbase); + wr_shadow_gsbase(next->gsbase); + } else { + load_seg_legacy(prev->fsindex, prev->fsbase, + next->fsindex, next->fsbase, FS); + load_seg_legacy(prev->gsindex, prev->gsbase, + next->gsindex, next->gsbase, GS); + } } static unsigned long task_seg_base(struct task_struct *task, -- 2.7.4