[PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

With ever growing extended state registers (xstate) on x86 processors, kernel
needs to cope with issue of growing memory space occupied by xstate. The xsave
area is holding more and more xstate registers, growing from legacy FP and
SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.

The recently introduced compacted format of xsave area saves xstates only
for enabled states. This patch set saves the xsave area space per process
in compacted format by xsaves/xrstors instructions.

xsaves/xrstors also supports modified optimization which tracks modified
xstates and only saves modified xstates to xsave area to improve saving
performance.

For more information about xsaves/xrstors instructions, please refer
to latest Intel X86 SDM.


Fenghua Yu (15):
  x86/xsaves: Detect xsaves/xrstors feature
  x86/xsaves: Add kernel parameter to disable xsaves/xrstors
  x86/alternative: Add alternative_input_2
  x86/xsaves: Change compacted format xsave area header
  x86/xsaves: Define macros for xsave instructions
  x86/xsaves: Define macro for handle xsave/xrstor fault
  x86/xsaves: Use xsaves/xrstors for saving and restoring xsave area
  x86/xsaves: Use xsaves/xrstors for context switch
  x86/xsaves: Use xsave/xrstor for saving and restoring user space
    context
  x86/xsaves: Clear reserved bits in xsave header
  x86/xsaves: Add xsaves and xrstors support for booting time
  x86/xsaves: Save xstate to task's xsave area in __save_fpu during
    booting time
  x86/xsaves: Call booting time xsaves and xrstors in setup_init_fpu_buf
  x86/xsaves: Enable xsaves/xrstors
  Define kernel API to get address of each state in xsave area

 Documentation/kernel-parameters.txt   |   9 ++
 arch/x86/include/asm/alternative.h    |  13 ++
 arch/x86/include/asm/cpufeature.h     |   2 +
 arch/x86/include/asm/fpu-internal.h   |   9 +-
 arch/x86/include/asm/processor.h      |   4 +-
 arch/x86/include/asm/xsave.h          | 220 +++++++++++++++++++++++++---------
 arch/x86/include/uapi/asm/msr-index.h |   2 +
 arch/x86/kernel/cpu/common.c          |   8 ++
 arch/x86/kernel/cpu/scattered.c       |   1 +
 arch/x86/kernel/i387.c                |   2 +-
 arch/x86/kernel/process.c             |   1 +
 arch/x86/kernel/xsave.c               | 112 +++++++++++++++--
 12 files changed, 313 insertions(+), 70 deletions(-)

-- 
1.8.1.2

[PATCH 01/15] x86/xsaves: Detect xsaves/xrstors feature

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Detect the xsaves feature in processor extended state enumberation sub-leaf
(eax=0x0d, ecx=1):
Bit 03: Supports xsaves/xrstors and IA32_XSS if set

The IA32_XSS MSR (index DA0H) contains a state-component bitmap that specifies
the state components that software has enabled xsaves and xrstors to manage.
If the bit corresponding to a state component is clear in XCR0 | IA32_XSS,
xsaves and xrstors will not operate on that state component, regardless of
the value of the instruction mask.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/cpufeature.h     | 2 ++
 arch/x86/include/uapi/asm/msr-index.h | 2 ++
 arch/x86/kernel/cpu/scattered.c       | 1 +
 3 files changed, 5 insertions(+)

diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h
index e265ff9..efede2e 100644
--- a/arch/x86/include/asm/cpufeature.h
+++ b/arch/x86/include/asm/cpufeature.h
@@ -186,6 +186,7 @@
 #define X86_FEATURE_DTHERM	(7*32+ 7) /* Digital Thermal Sensor */
 #define X86_FEATURE_HW_PSTATE	(7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK (7*32+ 9) /* AMD ProcFeedbackInterface */
+#define X86_FEATURE_XSAVES	(7*32+13) /* xsaves/xrstors/xss */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW  (8*32+ 0) /* Intel TPR Shadow */
@@ -328,6 +329,7 @@ extern const char * const x86_power_flags[32];
 #define cpu_has_x2apic		boot_cpu_has(X86_FEATURE_X2APIC)
 #define cpu_has_xsave		boot_cpu_has(X86_FEATURE_XSAVE)
 #define cpu_has_xsaveopt	boot_cpu_has(X86_FEATURE_XSAVEOPT)
+#define cpu_has_xsaves		boot_cpu_has(X86_FEATURE_XSAVES)
 #define cpu_has_osxsave		boot_cpu_has(X86_FEATURE_OSXSAVE)
 #define cpu_has_hypervisor	boot_cpu_has(X86_FEATURE_HYPERVISOR)
 #define cpu_has_pclmulqdq	boot_cpu_has(X86_FEATURE_PCLMULQDQ)
diff --git a/arch/x86/include/uapi/asm/msr-index.h b/arch/x86/include/uapi/asm/msr-index.h
index fcf2b3a..ce04632 100644
--- a/arch/x86/include/uapi/asm/msr-index.h
+++ b/arch/x86/include/uapi/asm/msr-index.h
@@ -311,6 +311,8 @@
 #define MSR_IA32_UCODE_WRITE		0x00000079
 #define MSR_IA32_UCODE_REV		0x0000008b
 
+#define MSR_IA32_XSS			0x00000da0
+
 #define MSR_IA32_PERF_STATUS		0x00000198
 #define MSR_IA32_PERF_CTL		0x00000199
 #define MSR_AMD_PSTATE_DEF_BASE		0xc0010064
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index b6f794a..8b29ea2 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -39,6 +39,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c)
 		{ X86_FEATURE_APERFMPERF,	CR_ECX, 0, 0x00000006, 0 },
 		{ X86_FEATURE_EPB,		CR_ECX, 3, 0x00000006, 0 },
 		{ X86_FEATURE_XSAVEOPT,		CR_EAX,	0, 0x0000000d, 1 },
+		{ X86_FEATURE_XSAVES,		CR_EAX,	3, 0x0000000d, 1 },
 		{ X86_FEATURE_HW_PSTATE,	CR_EDX, 7, 0x80000007, 0 },
 		{ X86_FEATURE_CPB,		CR_EDX, 9, 0x80000007, 0 },
 		{ X86_FEATURE_PROC_FEEDBACK,	CR_EDX,11, 0x80000007, 0 },
-- 
1.8.1.2

[PATCH 02/15] x86/xsaves: Add kernel parameter to disable xsaves/xrstors

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Add kernel parameter noxsaves to disable xsaves/xrstors feature.

Add kernel parameter noxsaveopt in doc.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 Documentation/kernel-parameters.txt | 9 +++++++++
 arch/x86/kernel/cpu/common.c        | 8 ++++++++
 2 files changed, 17 insertions(+)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 4384217..8541752 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2124,6 +2124,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			and restore using xsave. The kernel will fallback to
 			enabling legacy floating-point and sse state.
 
+	noxsaveopt	[X86] Disables x86 extended register state context
+			switch using xsaveopt. The kernel wil fallback to use
+			not optimized xsave to do context switch.
+
+	noxsaves	[X86] Disables x86 extended register state context
+			switch and save/restore compacted formated xsave
+			area using xsaves. Then the kernel uses standard
+			xsave area for xstate.
+
 	eagerfpu=	[X86]
 			on	enable eager fpu restore
 			off	disable eager fpu restore
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index a135239..409f47c 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -146,6 +146,7 @@ static int __init x86_xsave_setup(char *s)
 {
 	setup_clear_cpu_cap(X86_FEATURE_XSAVE);
 	setup_clear_cpu_cap(X86_FEATURE_XSAVEOPT);
+	setup_clear_cpu_cap(X86_FEATURE_XSAVES);
 	setup_clear_cpu_cap(X86_FEATURE_AVX);
 	setup_clear_cpu_cap(X86_FEATURE_AVX2);
 	return 1;
@@ -159,6 +160,13 @@ static int __init x86_xsaveopt_setup(char *s)
 }
 __setup("noxsaveopt", x86_xsaveopt_setup);
 
+static int __init x86_xsaves_setup(char *s)
+{
+	setup_clear_cpu_cap(X86_FEATURE_XSAVES);
+	return 1;
+}
+__setup("noxsaves", x86_xsaves_setup);
+
 #ifdef CONFIG_X86_32
 static int cachesize_override = -1;
 static int disable_x86_serial_nr = 1;
-- 
1.8.1.2

[PATCH 03/15] x86/alternative: Add alternative_input_2

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Add alternative_input_2 to take two features and respective instructions.

If CPU has feature2, newinstr2 is used.

Otherwise, if CPU has feature1, newinstr1 is used.

Otherwise, oldinstr is used.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/alternative.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 0a3f9c9..eb8d80b 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -161,6 +161,19 @@ static inline int alternatives_text_reserved(void *start, void *end)
 	asm volatile (ALTERNATIVE(oldinstr, newinstr, feature)		\
 		: : "i" (0), ## input)
 
+/*
+ * Like alternative_input, but there are two features and respective
+ * instructions.
+ * If CPU has feature2, newinstr2 is used.
+ * Otherwise, if CPU has feature1, newinstr1 is used.
+ * Otherwise, oldinstr is used.
+ */
+#define alternative_input_2(oldinstr, newinstr1, feature1, newinstr2,	     \
+			   feature2, input...)				     \
+	asm volatile(ALTERNATIVE_2(oldinstr, newinstr1, feature1,	     \
+		newinstr2, feature2)					     \
+		: : "i" (0), ## input)
+
 /* Like alternative_input, but with a single output argument */
 #define alternative_io(oldinstr, newinstr, feature, output, input...)	\
 	asm volatile (ALTERNATIVE(oldinstr, newinstr, feature)		\
-- 
1.8.1.2

[PATCH 04/15] x86/xsaves: Change compacted format xsave area header

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

With compacted format xsave area supported, the xsave area header is changed
to support both compacted format and standard format xsave area.

The XSAVE header of an XSAVE area comprises the 64 bytes starting at offset
512 from the area\u2019s base address:
        - Bytes 7:0 of the XSAVE header is a state-component bitmap called
XSTATE_BV. It identifies the state components in the XSAVE area.
        - Bytes 15:8 of the XSAVE header is a state-component bitmap called
XCOMP_BV. It is used as follows:
                - XCOMP_BV[63] indicates the format of the extended region of
the XSAVE area. If it is clear, the standard format is used. If it is set,
the compacted format is used; XCOMP_BV[62:0] provide format specifics.
                - XCOMP_BV[63] determines which form of the XRSTOR instruction
is used. If the bit is set, the compacted form is used; otherwise, the standard
form is used. See Section 13.7.
        - All bits in XCOMP_BV should be 0 if the processor does not support
the compaction extensions to the XSAVE feature set.
        - Bytes 63:16 of the XSAVE header are reserved.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/processor.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index a4ea023..2c8d3b8 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -386,8 +386,8 @@ struct bndcsr_struct {
 
 struct xsave_hdr_struct {
 	u64 xstate_bv;
-	u64 reserved1[2];
-	u64 reserved2[5];
+	u64 xcomp_bv;
+	u64 reserved[6];
 } __attribute__((packed));
 
 struct xsave_struct {
-- 
1.8.1.2

[PATCH 05/15] x86/xsaves: Define macros for xsave instructions

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Define macros for xsave, xsaveopt, xsaves, xrstor, xrstors inline
instructions. The instructions will be used for saving and rstoring xstate.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index d949ef2..71bdde4 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -52,6 +52,12 @@ extern void xsave_init(void);
 extern void update_regset_xstate_info(unsigned int size, u64 xstate_mask);
 extern int init_fpu(struct task_struct *child);
 
+#define XSAVE		".byte " REX_PREFIX "0x0f,0xae,0x27"
+#define XSAVEOPT	".byte " REX_PREFIX "0x0f,0xae,0x37"
+#define XSAVES		".byte " REX_PREFIX "0x0f,0xc7,0x2f"
+#define XRSTOR		".byte " REX_PREFIX "0x0f,0xae,0x2f"
+#define XRSTORS		".byte " REX_PREFIX "0x0f,0xc7,0x1f"
+
 static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 {
 	int err;
-- 
1.8.1.2

[PATCH 06/15] x86/xsaves: Define macro for handle xsave/xrstor fault

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Define macro to handle fault generated by xsave, xsaveopt, xsaves, xrstor,
xrstors instructions.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index 71bdde4..76c2459 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -58,6 +58,13 @@ extern int init_fpu(struct task_struct *child);
 #define XRSTOR		".byte " REX_PREFIX "0x0f,0xae,0x2f"
 #define XRSTORS		".byte " REX_PREFIX "0x0f,0xc7,0x1f"
 
+#define xstate_fault	".section .fixup,\"ax\"\n"	\
+			"3:  movl $-1,%[err]\n"		\
+			"    jmp  2b\n"			\
+			".previous\n"			\
+			_ASM_EXTABLE(1b, 3b)		\
+			: [err] "=r" (err)
+
 static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 {
 	int err;
-- 
1.8.1.2

[PATCH 07/15] x86/xsaves: Use xsaves/xrstors for saving and restoring xsave area

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

If xsaves is eanbled, use xsaves/xrstors for saving and restoring xstate.
xsaves supports compacted format, init optimization, modified optimization,
and supervisor states.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h | 82 +++++++++++++++++++++++++++++++++-----------
 1 file changed, 62 insertions(+), 20 deletions(-)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index 76c2459..e538a0e 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -65,6 +65,68 @@ extern int init_fpu(struct task_struct *child);
 			_ASM_EXTABLE(1b, 3b)		\
 			: [err] "=r" (err)
 
+/*
+ * Save processor xstate to xsave area.
+ */
+static inline void xsave_state(struct xsave_struct *fx, u64 mask)
+{
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+	int err = 0;
+
+	/*
+	 * If xsaves is enabled, xsaves replaces xsaveopt because
+	 * it supports compact format and supervisor states in addition to
+	 * modified optimization in xsaveopt.
+	 *
+	 * Otherwise, if xsaveopt is enabled, xsaveopt replaces xsave
+	 * because xsaveopt supports modified optimization which is not
+	 * supported by xsave.
+	 *
+	 * If none of xsaves and xsaveopt is enabled, use xsave.
+	 */
+	alternative_input_2(
+		"1:"XSAVE,
+		"1:"XSAVEOPT,
+		X86_FEATURE_XSAVEOPT,
+		"1:"XSAVES,
+		X86_FEATURE_XSAVES,
+		[fx] "D" (fx), "a" (lmask), "d" (hmask) :
+		"memory");
+	asm volatile("2:\n\t"
+		     xstate_fault
+		     : "0" (0)
+		     : "memory");
+}
+
+/*
+ * Restore processor xstate from xsave area.
+ */
+static inline int xrstor_state(struct xsave_struct *fx, u64 mask)
+{
+	int err = 0;
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+
+	/*
+	 * Use xrstors to restore context if it is enabled. xrstors supports
+	 * compacted format of xsave area which is not supported by xrstor.
+	 */
+	alternative_input(
+		"1: " XRSTOR,
+		"1: " XRSTORS,
+		X86_FEATURE_XSAVES,
+		"D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+		: "memory");
+
+	asm volatile("2:\n"
+		     xstate_fault
+		     : "0" (0)
+		     : "memory");
+
+	return err;
+}
+
 static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 {
 	int err;
@@ -130,26 +192,6 @@ static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 	return err;
 }
 
-static inline void xrstor_state(struct xsave_struct *fx, u64 mask)
-{
-	u32 lmask = mask;
-	u32 hmask = mask >> 32;
-
-	asm volatile(".byte " REX_PREFIX "0x0f,0xae,0x2f\n\t"
-		     : : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
-		     :   "memory");
-}
-
-static inline void xsave_state(struct xsave_struct *fx, u64 mask)
-{
-	u32 lmask = mask;
-	u32 hmask = mask >> 32;
-
-	asm volatile(".byte " REX_PREFIX "0x0f,0xae,0x27\n\t"
-		     : : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
-		     :   "memory");
-}
-
 static inline void fpu_xsave(struct fpu *fpu)
 {
 	/* This, however, we can work around by forcing the compiler to select
-- 
1.8.1.2

[PATCH 08/15] x86/xsaves: Use xsaves/xrstors for context switch

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

If xsaves is eanbled, use xsaves/xrstors for context switch to utilize
compacted format xsave area and modified optimization.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h | 37 ++++++++++++-------------------------
 1 file changed, 12 insertions(+), 25 deletions(-)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index e538a0e..cfee67e 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -127,22 +127,20 @@ static inline int xrstor_state(struct xsave_struct *fx, u64 mask)
 	return err;
 }
 
-static inline int fpu_xrstor_checking(struct xsave_struct *fx)
+/*
+ * Save xstate context for old process during context switch.
+ */
+static inline void fpu_xsave(struct fpu *fpu)
 {
-	int err;
-
-	asm volatile("1: .byte " REX_PREFIX "0x0f,0xae,0x2f\n\t"
-		     "2:\n"
-		     ".section .fixup,\"ax\"\n"
-		     "3:  movl $-1,%[err]\n"
-		     "    jmp  2b\n"
-		     ".previous\n"
-		     _ASM_EXTABLE(1b, 3b)
-		     : [err] "=r" (err)
-		     : "D" (fx), "m" (*fx), "a" (-1), "d" (-1), "0" (0)
-		     : "memory");
+	xsave_state(&fpu->state->xsave, -1);
+}
 
-	return err;
+/*
+ * Restore xstate context for new process during context switch.
+ */
+static inline int fpu_xrstor_checking(struct xsave_struct *fx)
+{
+	return xrstor_state(fx, -1);
 }
 
 static inline int xsave_user(struct xsave_struct __user *buf)
@@ -192,15 +190,4 @@ static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 	return err;
 }
 
-static inline void fpu_xsave(struct fpu *fpu)
-{
-	/* This, however, we can work around by forcing the compiler to select
-	   an addressing mode that doesn't require extended registers. */
-	alternative_input(
-		".byte " REX_PREFIX "0x0f,0xae,0x27",
-		".byte " REX_PREFIX "0x0f,0xae,0x37",
-		X86_FEATURE_XSAVEOPT,
-		[fx] "D" (&fpu->state->xsave), "a" (-1), "d" (-1) :
-		"memory");
-}
 #endif
-- 
1.8.1.2

[PATCH 09/15] x86/xsaves: Use xsave/xrstor for saving and restoring user space context

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

We use legacy xsave/xrstor for user space context. No xsaveopt or xsaves is
used here.

We don't use modified optimization which is implemented in xsaveopt and xsaves
because xrstor/xrstors might track a different application.

We don't use compacted format xsave area in xsaves for backward compatibility
because old applications which don't understand compacted format of xsave area.
This may cosume more user context memory than compacted format. But in this
way legacy applications don't need any change to work with kernel.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index cfee67e..a8f31d0 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -143,6 +143,16 @@ static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 	return xrstor_state(fx, -1);
 }
 
+/*
+ * Save xstate to user space xsave area.
+ *
+ * We don't use modified optimization because xrstor/xrstors might track
+ * a different application.
+ *
+ * We don't use compacted format xsave area for
+ * backward compatibility for old applications which don't understand
+ * compacted format of xsave area.
+ */
 static inline int xsave_user(struct xsave_struct __user *buf)
 {
 	int err;
@@ -156,35 +166,28 @@ static inline int xsave_user(struct xsave_struct __user *buf)
 		return -EFAULT;
 
 	__asm__ __volatile__(ASM_STAC "\n"
-			     "1: .byte " REX_PREFIX "0x0f,0xae,0x27\n"
+			     "1:"XSAVE"\n"
 			     "2: " ASM_CLAC "\n"
-			     ".section .fixup,\"ax\"\n"
-			     "3:  movl $-1,%[err]\n"
-			     "    jmp  2b\n"
-			     ".previous\n"
-			     _ASM_EXTABLE(1b,3b)
-			     : [err] "=r" (err)
+			     xstate_fault
 			     : "D" (buf), "a" (-1), "d" (-1), "0" (0)
 			     : "memory");
 	return err;
 }
 
+/*
+ * Restore xstate from user space xsave area.
+ */
 static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 {
-	int err;
+	int err = 0;
 	struct xsave_struct *xstate = ((__force struct xsave_struct *)buf);
 	u32 lmask = mask;
 	u32 hmask = mask >> 32;
 
 	__asm__ __volatile__(ASM_STAC "\n"
-			     "1: .byte " REX_PREFIX "0x0f,0xae,0x2f\n"
+			     "1:"XRSTOR"\n"
 			     "2: " ASM_CLAC "\n"
-			     ".section .fixup,\"ax\"\n"
-			     "3:  movl $-1,%[err]\n"
-			     "    jmp  2b\n"
-			     ".previous\n"
-			     _ASM_EXTABLE(1b,3b)
-			     : [err] "=r" (err)
+			     xstate_fault
 			     : "D" (xstate), "a" (lmask), "d" (hmask), "0" (0)
 			     : "memory");	/* memory required? */
 	return err;
-- 
1.8.1.2

[PATCH 10/15] x86/xsaves: Clear reserved bits in xsave header

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

The reserved bits (128~511) in the xsave header must be zero. Clear the bits
here.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/i387.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/i387.c b/arch/x86/kernel/i387.c
index d5dd808..a9a4229 100644
--- a/arch/x86/kernel/i387.c
+++ b/arch/x86/kernel/i387.c
@@ -375,7 +375,7 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset,
 	/*
 	 * These bits must be zero.
 	 */
-	xsave_hdr->reserved1[0] = xsave_hdr->reserved1[1] = 0;
+	memset(xsave_hdr->reserved, 0, 48);
 
 	return ret;
 }
-- 
1.8.1.2

[PATCH 11/15] x86/xsaves: Add xsaves and xrstors support for booting time

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

Since cpu caps are not enabled yet during early booting time, alternative
can not be used in functions to access xsave area. Therefore, we define
two new functions xrstor_state_booting and xsave_state_booting to access
xsave area during early booting time.

Because xsave_state_booting() can be called by run-time function __save_fpu(),
it's not defined as __init and will stay in memory during running time. But
it's a small function and the memory cost is small.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h | 60 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index a8f31d0..0de695d 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -66,6 +66,66 @@ extern int init_fpu(struct task_struct *child);
 			: [err] "=r" (err)
 
 /*
+ * This function is called only during boot time when x86 caps are not set
+ * up and alternative can not be used yet.
+ */
+static void xsave_state_booting(struct xsave_struct *fx, u64 mask)
+{
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+	int err = 0;
+
+	WARN_ON(system_state != SYSTEM_BOOTING);
+
+	if (boot_cpu_has(X86_FEATURE_XSAVES))
+		asm volatile("1:"XSAVES"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+	else
+		asm volatile("1:"XSAVE"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+
+	asm volatile(xstate_fault
+		     : "0" (0)
+		     : "memory");
+
+	return err;
+}
+
+/*
+ * This function is called only during boot time when x86 caps are not set
+ * up and alternative can not be used yet.
+ */
+static inline int xrstor_state_booting(struct xsave_struct *fx, u64 mask)
+{
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+	int err = 0;
+
+	WARN_ON(system_state != SYSTEM_BOOTING);
+
+	if (boot_cpu_has(X86_FEATURE_XSAVES))
+		asm volatile("1:"XRSTORS"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+	else
+		asm volatile("1:"XRSTOR"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+
+	asm volatile(xstate_fault
+		     : "0" (0)
+		     : "memory");
+
+	return err;
+}
+
+/*
  * Save processor xstate to xsave area.
  */
 static inline void xsave_state(struct xsave_struct *fx, u64 mask)
-- 
1.8.1.2

[PATCH 12/15] x86/xsaves: Save xstate to task's xsave area in __save_fpu during booting time

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

__save_fpu() can be called during early booting time when cpu caps are not
enabled and alternative can not be used yet. Therefore, it calls
xsave_state_booting() to save xstate to task's xsave area.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/fpu-internal.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/fpu-internal.h b/arch/x86/include/asm/fpu-internal.h
index cea1c76..6099c0e 100644
--- a/arch/x86/include/asm/fpu-internal.h
+++ b/arch/x86/include/asm/fpu-internal.h
@@ -508,9 +508,12 @@ static inline void user_fpu_begin(void)
 
 static inline void __save_fpu(struct task_struct *tsk)
 {
-	if (use_xsave())
-		xsave_state(&tsk->thread.fpu.state->xsave, -1);
-	else
+	if (use_xsave()) {
+		if (unlikely(system_state == SYSTEM_BOOTING))
+			xsave_state_booting(&tsk->thread.fpu.state->xsave, -1);
+		else
+			xsave_state(&tsk->thread.fpu.state->xsave, -1);
+	} else
 		fpu_fxsave(&tsk->thread.fpu);
 }
 
-- 
1.8.1.2

[PATCH 13/15] x86/xsaves: Call booting time xsaves and xrstors in setup_init_fpu_buf

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

setup_init_fpu_buf() calls booting time xsaves and xrstors to save/restore
xstate in xsave area.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/xsave.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/xsave.c b/arch/x86/kernel/xsave.c
index a4b451c..8fa7c7d 100644
--- a/arch/x86/kernel/xsave.c
+++ b/arch/x86/kernel/xsave.c
@@ -496,15 +496,21 @@ static void __init setup_init_fpu_buf(void)
 
 	setup_xstate_features();
 
+	if (cpu_has_xsaves) {
+		init_xstate_buf->xsave_hdr.xcomp_bv =
+						(u64)1 << 63 | pcntxt_mask;
+		init_xstate_buf->xsave_hdr.xstate_bv = pcntxt_mask;
+	}
+
 	/*
 	 * Init all the features state with header_bv being 0x0
 	 */
-	xrstor_state(init_xstate_buf, -1);
+	xrstor_state_booting(init_xstate_buf, -1);
 	/*
 	 * Dump the init state again. This is to identify the init state
 	 * of any feature which is not represented by all zero's.
 	 */
-	xsave_state(init_xstate_buf, -1);
+	xsave_state_booting(init_xstate_buf, -1);
 }
 
 static enum { AUTO, ENABLE, DISABLE } eagerfpu = AUTO;
-- 
1.8.1.2

[PATCH 14/15] x86/xsaves: Enable xsaves/xrstors

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

If xsaves/xrstors is enabled, compacted format of xsave area will be used
and less memory may be used for context switching per process. Modified
optimization implemented in xsaves/xrstors improves performance of saving
xstate as well.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/kernel/xsave.c | 38 ++++++++++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/xsave.c b/arch/x86/kernel/xsave.c
index 8fa7c7d..2e9a77d 100644
--- a/arch/x86/kernel/xsave.c
+++ b/arch/x86/kernel/xsave.c
@@ -8,6 +8,7 @@
 
 #include <linux/bootmem.h>
 #include <linux/compat.h>
+#include <linux/cpu.h>
 #include <asm/i387.h>
 #include <asm/fpu-internal.h>
 #include <asm/sigframe.h>
@@ -24,7 +25,9 @@ u64 pcntxt_mask;
 struct xsave_struct *init_xstate_buf;
 
 static struct _fpx_sw_bytes fx_sw_reserved, fx_sw_reserved_ia32;
-static unsigned int *xstate_offsets, *xstate_sizes, xstate_features;
+static unsigned int *xstate_offsets, *xstate_sizes;
+static unsigned int *xstate_comp_offsets, *xstate_comp_sizes;
+static unsigned int xstate_features;
 
 /*
  * If a processor implementation discern that a processor state component is
@@ -283,7 +286,7 @@ sanitize_restored_xstate(struct task_struct *tsk,
 
 	if (use_xsave()) {
 		/* These bits must be zero. */
-		xsave_hdr->reserved1[0] = xsave_hdr->reserved1[1] = 0;
+		memset(xsave_hdr->reserved, 0, 48);
 
 		/*
 		 * Init the state that is not present in the memory
@@ -526,6 +529,30 @@ static int __init eager_fpu_setup(char *s)
 }
 __setup("eagerfpu=", eager_fpu_setup);
 
+
+/*
+ * Calculate total size of enabled xstates in XCR0/pcntxt_mask.
+ */
+static void __init init_xstate_size(void)
+{
+	unsigned int eax, ebx, ecx, edx;
+	int i;
+
+	if (!cpu_has_xsaves) {
+		cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
+		xstate_size = ebx;
+		return;
+	}
+
+	xstate_size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
+	for (i = 2; i < 64; i++) {
+		if (test_bit(i, (unsigned long *)&pcntxt_mask)) {
+			cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx);
+			xstate_size += eax;
+		}
+	}
+}
+
 /*
  * Enable and initialize the xsave feature.
  */
@@ -557,8 +584,7 @@ static void __init xstate_enable_boot_cpu(void)
 	/*
 	 * Recompute the context size for enabled features
 	 */
-	cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
-	xstate_size = ebx;
+	init_xstate_size();
 
 	update_regset_xstate_info(xstate_size, pcntxt_mask);
 	prepare_fx_sw_frame();
@@ -578,8 +604,8 @@ static void __init xstate_enable_boot_cpu(void)
 		}
 	}
 
-	pr_info("enabled xstate_bv 0x%llx, cntxt size 0x%x\n",
-		pcntxt_mask, xstate_size);
+	pr_info("enabled xstate_bv 0x%llx, cntxt size 0x%x using %s\n",
+		pcntxt_mask, xstate_size, cpu_has_xsaves ? "xsaves" : "xsave");
 }
 
 /*
-- 
1.8.1.2

[PATCH 15/15] Define kernel API to get address of each state in xsave area

[Fenghua Yu]

From: Fenghua Yu <fenghua.yu@intel.com>

In standard format, each state is saved in the xsave area in fixed offset.
In compacted format, offset of each saved state only can be calculated during
run time because some xstates may not be saved.

get_xsave_addr() returns address of a given state saved in a xsave area.

It can be called in kernel to get address of each xstate in xsave area in
either standard format or compacted format.

Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
---
 arch/x86/include/asm/xsave.h |  3 +++
 arch/x86/kernel/process.c    |  1 +
 arch/x86/kernel/xsave.c      | 64 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+)

diff --git a/arch/x86/include/asm/xsave.h b/arch/x86/include/asm/xsave.h
index 0de695d..575e156 100644
--- a/arch/x86/include/asm/xsave.h
+++ b/arch/x86/include/asm/xsave.h
@@ -253,4 +253,7 @@ static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 	return err;
 }
 
+void *get_xsave_addr(struct xsave_struct *xsave, int xstate);
+void setup_xstate_comp(void);
+
 #endif
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index 4505e2a..f804dc9 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -93,6 +93,7 @@ void arch_task_cache_init(void)
         	kmem_cache_create("task_xstate", xstate_size,
 				  __alignof__(union thread_xstate),
 				  SLAB_PANIC | SLAB_NOTRACK, NULL);
+	setup_xstate_comp();
 }
 
 /*
diff --git a/arch/x86/kernel/xsave.c b/arch/x86/kernel/xsave.c
index 2e9a77d..87e77f4 100644
--- a/arch/x86/kernel/xsave.c
+++ b/arch/x86/kernel/xsave.c
@@ -482,6 +482,47 @@ static void __init setup_xstate_features(void)
 }
 
 /*
+ * This function sets up offsets and sizes of all extended states in
+ * xsave area. This supports both standard format and compacted format
+ * of the xsave aread.
+ *
+ * Input: void
+ * Output: void
+ */
+void setup_xstate_comp(void)
+{
+	int i;
+
+	xstate_comp_offsets = kmalloc(xstate_features * sizeof(int),
+				      GFP_KERNEL);
+	xstate_comp_sizes = kmalloc(xstate_features * sizeof(int), GFP_KERNEL);
+
+	if (!cpu_has_xsaves) {
+		for (i = 2; i < xstate_features; i++) {
+			if (test_bit(i, (unsigned long *)&pcntxt_mask)) {
+				xstate_comp_offsets[i] = xstate_offsets[i];
+				xstate_comp_sizes[i] = xstate_sizes[i];
+			}
+		}
+		return;
+	}
+
+	xstate_comp_offsets[2] = FXSAVE_SIZE + XSAVE_HDR_SIZE;
+
+	for (i = 2; i < xstate_features; i++) {
+		if (test_bit(i, (unsigned long *)&pcntxt_mask))
+			xstate_comp_sizes[i] = xstate_sizes[i];
+		else
+			xstate_comp_sizes[i] = 0;
+
+		if (i > 2)
+			xstate_comp_offsets[i] = xstate_comp_offsets[i-1]
+					+ xstate_comp_sizes[i-1];
+
+	}
+}
+
+/*
  * setup the xstate image representing the init state
  */
 static void __init setup_init_fpu_buf(void)
@@ -667,3 +708,26 @@ void eager_fpu_init(void)
 	else
 		fxrstor_checking(&init_xstate_buf->i387);
 }
+
+/*
+ * Given the xsave area and a state inside, this function returns the
+ * address of the state.
+ *
+ * This is the API that is called to get xstate address in either
+ * standard format or compacted format of xsave area.
+ *
+ * Inputs:
+ *	xsave: base address of the xsave area;
+ *	xstate: state which is defined in xsave.h (e.g. XSTATE_FP, XSTATE_SSE,
+ *	etc.)
+ * Output:
+ *	address of the state in the xsave area.
+ */
+void *get_xsave_addr(struct xsave_struct *xsave, int xstate)
+{
+	int feature = fls64(xstate) - 1;
+	if (!test_bit(feature, (unsigned long *)&pcntxt_mask))
+		return NULL;
+
+	return (void *)xsave + xstate_comp_offsets[feature];
+}
-- 
1.8.1.2

Re: [PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[Andy Lutomirski]

On 05/26/2014 10:01 AM, Fenghua Yu wrote:
> From: Fenghua Yu <fenghua.yu@intel.com>
> 
> With ever growing extended state registers (xstate) on x86 processors, kernel
> needs to cope with issue of growing memory space occupied by xstate. The xsave
> area is holding more and more xstate registers, growing from legacy FP and
> SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.
> 
> The recently introduced compacted format of xsave area saves xstates only
> for enabled states. This patch set saves the xsave area space per process
> in compacted format by xsaves/xrstors instructions.

Are we going to want to encourage userspace to do something like
sticking vzeroupper right before each syscall to make any
xsaves/xrestores faster?

--Andy

RE: [PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[Yu, Fenghua]

> From: Andy Lutomirski [mailto:luto@amacapital.net]
> On 05/26/2014 10:01 AM, Fenghua Yu wrote:
> > From: Fenghua Yu <fenghua.yu@intel.com>
> >
> > With ever growing extended state registers (xstate) on x86 processors,
> kernel
> > needs to cope with issue of growing memory space occupied by xstate.
> The xsave
> > area is holding more and more xstate registers, growing from legacy
> FP and
> > SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.
> >
> > The recently introduced compacted format of xsave area saves xstates
> only
> > for enabled states. This patch set saves the xsave area space per
> process
> > in compacted format by xsaves/xrstors instructions.
> 
> Are we going to want to encourage userspace to do something like
> sticking vzeroupper right before each syscall to make any
> xsaves/xrestores faster?

This patch set allow compacted format in kernel and standard format
in user space. This works fine for both kernel and user application.

As said in the patch 9/15, there would be two issues for user applications
to use xsaves/xrstors: one is modified optimization doesn't work for
user applications and the other one is legacy applications don't
know compacted format of xsave area.

So I don't see we need xsaves/xrstors in user space. But I could be
wrong.

Thanks.

-Fenghua

Re: [PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[H. Peter Anvin]

On 05/26/2014 01:13 PM, Yu, Fenghua wrote:
> 
> So I don't see we need xsaves/xrstors in user space. But I could be
> wrong.
> 

XSAVES/XRSTORS are privileged.  The user-space equivalents are
XSAVEC/XRSTOR.

	-hpa

Re: [PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[Andy Lutomirski]

On Mon, May 26, 2014 at 1:13 PM, Yu, Fenghua <fenghua.yu@intel.com> wrote:
>> From: Andy Lutomirski [mailto:luto@amacapital.net]
>> On 05/26/2014 10:01 AM, Fenghua Yu wrote:
>> > From: Fenghua Yu <fenghua.yu@intel.com>
>> >
>> > With ever growing extended state registers (xstate) on x86 processors,
>> kernel
>> > needs to cope with issue of growing memory space occupied by xstate.
>> The xsave
>> > area is holding more and more xstate registers, growing from legacy
>> FP and
>> > SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.
>> >
>> > The recently introduced compacted format of xsave area saves xstates
>> only
>> > for enabled states. This patch set saves the xsave area space per
>> process
>> > in compacted format by xsaves/xrstors instructions.
>>
>> Are we going to want to encourage userspace to do something like
>> sticking vzeroupper right before each syscall to make any
>> xsaves/xrestores faster?
>
> This patch set allow compacted format in kernel and standard format
> in user space. This works fine for both kernel and user application.

My question is purely about optimization: if userspace does a blocking
system call, will it be significantly faster if userspace zeros out as
much of the extended state as possible before doing the system call?

I think I tried this once with xsaveopt and decided that it didn't
make much of a difference.

--Andy

Re: [PATCH 0/15] x86/xsaves: Optimize xstate context switch by xsaves/xrstors

[H. Peter Anvin]

On 05/26/2014 02:40 PM, Andy Lutomirski wrote:
>>>
>>> Are we going to want to encourage userspace to do something like
>>> sticking vzeroupper right before each syscall to make any
>>> xsaves/xrestores faster?
>>
>> This patch set allow compacted format in kernel and standard format
>> in user space. This works fine for both kernel and user application.
> 
> My question is purely about optimization: if userspace does a blocking
> system call, will it be significantly faster if userspace zeros out as
> much of the extended state as possible before doing the system call?
> 
> I think I tried this once with xsaveopt and decided that it didn't
> make much of a difference.
> 

XSAVEOPT and XSAVES would be the same in this case.  Keep in mind, too,
that relatively few system calls result in context switches, which is
the only time this ends up mattering.

	-hpa