From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) by kanga.kvack.org (Postfix) with ESMTP id 26BFD6B025B for ; Fri, 29 Jan 2016 14:43:04 -0500 (EST) Received: by mail-pa0-f41.google.com with SMTP id uo6so46883364pac.1 for ; Fri, 29 Jan 2016 11:43:04 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id ks7si3927262pab.129.2016.01.29.11.43.03 for ; Fri, 29 Jan 2016 11:43:03 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v3 0/3] x86/mm: INVPCID support Date: Fri, 29 Jan 2016 11:42:56 -0800 Message-Id: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Boris, I think you already have these prerequisites queued up: http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.) I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own. Changes from v2: - Add macros for the INVPCID mode numbers. - Add a changelog message for the chicken bit. v1 was exactly identical to patches 2-4 of the PCID RFC series. Andy Lutomirski (3): x86/mm: Add INVPCID helpers x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings Documentation/kernel-parameters.txt | 2 ++ arch/x86/include/asm/tlbflush.h | 57 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 16 +++++++++++ 3 files changed, 75 insertions(+) -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f172.google.com (mail-pf0-f172.google.com [209.85.192.172]) by kanga.kvack.org (Postfix) with ESMTP id 4B9446B025C for ; Fri, 29 Jan 2016 14:43:05 -0500 (EST) Received: by mail-pf0-f172.google.com with SMTP id 65so46914237pfd.2 for ; Fri, 29 Jan 2016 11:43:05 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id r19si26023708pfi.140.2016.01.29.11.43.04 for ; Fri, 29 Jan 2016 11:43:04 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v3 1/3] x86/mm: Add INVPCID helpers Date: Fri, 29 Jan 2016 11:42:57 -0800 Message-Id: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> In-Reply-To: References: In-Reply-To: References: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski This adds helpers for each of the four currently-specified INVPCID modes. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6df2029405a3..8b576832777e 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -7,6 +7,54 @@ #include #include +static inline void __invpcid(unsigned long pcid, unsigned long addr, + unsigned long type) +{ + u64 desc[2] = { pcid, addr }; + + /* + * The memory clobber is because the whole point is to invalidate + * stale TLB entries and, especially if we're flushing global + * mappings, we don't want the compiler to reorder any subsequent + * memory accesses before the TLB flush. + * + * The hex opcode is invpcid (%ecx), %eax in 32-bit mode and + * invpcid (%rcx), %rax in long mode. + */ + asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" + : : "m" (desc), "a" (type), "c" (desc) : "memory"); +} + +#define INVPCID_TYPE_INDIV_ADDR 0 +#define INVPCID_TYPE_SINGLE_CTXT 1 +#define INVPCID_TYPE_ALL_INCL_GLOBAL 2 +#define INVPCID_TYPE_ALL_NON_GLOBAL 3 + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invpcid_flush_one(unsigned long pcid, + unsigned long addr) +{ + __invpcid(pcid, addr, INVPCID_TYPE_INDIV_ADDR); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invpcid_flush_single_context(unsigned long pcid) +{ + __invpcid(pcid, 0, INVPCID_TYPE_SINGLE_CTXT); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invpcid_flush_all(void) +{ + __invpcid(0, 0, INVPCID_TYPE_ALL_INCL_GLOBAL); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invpcid_flush_all_nonglobals(void) +{ + __invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL); +} + #ifdef CONFIG_PARAVIRT #include #else -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f42.google.com (mail-pa0-f42.google.com [209.85.220.42]) by kanga.kvack.org (Postfix) with ESMTP id CB5036B025E for ; Fri, 29 Jan 2016 14:43:06 -0500 (EST) Received: by mail-pa0-f42.google.com with SMTP id ho8so46072695pac.2 for ; Fri, 29 Jan 2016 11:43:06 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id x9si5089817pas.214.2016.01.29.11.43.05 for ; Fri, 29 Jan 2016 11:43:06 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v3 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Date: Fri, 29 Jan 2016 11:42:58 -0800 Message-Id: In-Reply-To: References: In-Reply-To: References: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski This adds a chicken bit to turn off INVPCID in case something goes wrong. It's an early_param because we do TLB flushes before we parse __setup parameters. Signed-off-by: Andy Lutomirski --- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 742f69d18fc8..b34e55e00bae 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2508,6 +2508,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nointroute [IA-64] + noinvpcid [X86] Disable the INVPCID cpu feature. + nojitter [IA-64] Disables jitter checking for ITC timers. no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c2b7522cbf35..48196980c1c7 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s) } __setup("nompx", x86_mpx_setup); +static int __init x86_noinvpcid_setup(char *s) +{ + /* noinvpcid doesn't accept parameters */ + if (s) + return -EINVAL; + + /* do not emit a message if the feature is not present */ + if (!boot_cpu_has(X86_FEATURE_INVPCID)) + return 0; + + setup_clear_cpu_cap(X86_FEATURE_INVPCID); + pr_info("noinvpcid: INVPCID feature disabled\n"); + return 0; +} +early_param("noinvpcid", x86_noinvpcid_setup); + #ifdef CONFIG_X86_32 static int cachesize_override = -1; static int disable_x86_serial_nr = 1; -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) by kanga.kvack.org (Postfix) with ESMTP id 0D1F66B025F for ; Fri, 29 Jan 2016 14:43:08 -0500 (EST) Received: by mail-pa0-f43.google.com with SMTP id yy13so45999423pab.3 for ; Fri, 29 Jan 2016 11:43:08 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id wu1si3928300pab.71.2016.01.29.11.43.07 for ; Fri, 29 Jan 2016 11:43:07 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v3 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Date: Fri, 29 Jan 2016 11:42:59 -0800 Message-Id: In-Reply-To: References: In-Reply-To: References: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 8b576832777e..985f1162c505 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -152,6 +152,15 @@ static inline void __native_flush_tlb_global(void) { unsigned long flags; + if (static_cpu_has_safe(X86_FEATURE_INVPCID)) { + /* + * Using INVPCID is considerably faster than a pair of writes + * to CR4 sandwiched inside an IRQ flag save/restore. + */ + invpcid_flush_all(); + return; + } + /* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: [PATCH v3 0/3] x86/mm: INVPCID support Date: Mon, 1 Feb 2016 11:51:32 +0100 Message-ID: <20160201105132.GB6438@pd.tnic> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Andy Lutomirski , Ingo Molnar Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin List-Id: linux-mm.kvack.org On Fri, Jan 29, 2016 at 11:42:56AM -0800, Andy Lutomirski wrote: > Boris, I think you already have these prerequisites queued up: > > http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com > http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com > > This is a straightforward speedup on Ivy Bridge and newer, IIRC. > (I tested on Skylake. INVPCID is not available on Sandy Bridge. > I don't have Ivy Bridge, Haswell or Broadwell to test on, so I > could be wrong as to when the feature was introduced.) > > I think we should consider these patches separately from the rest > of the PCID stuff -- they barely interact, and this part is much > simpler and is useful on its own. > > Changes from v2: > - Add macros for the INVPCID mode numbers. > - Add a changelog message for the chicken bit. > > v1 was exactly identical to patches 2-4 of the PCID RFC series. > Andy Lutomirski (3): > x86/mm: Add INVPCID helpers > x86/mm: Add a noinvpcid option to turn off INVPCID > x86/mm: If INVPCID is available, use it to flush global mappings > > Documentation/kernel-parameters.txt | 2 ++ > arch/x86/include/asm/tlbflush.h | 57 +++++++++++++++++++++++++++++++++++++ > arch/x86/kernel/cpu/common.c | 16 +++++++++++ > 3 files changed, 75 insertions(+) All 5 (3 INVPCID + 2 KASAN ones at the URLs above): Reviewed-by: Borislav Petkov -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756923AbcA2TnH (ORCPT ); Fri, 29 Jan 2016 14:43:07 -0500 Received: from mail.kernel.org ([198.145.29.136]:58719 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756771AbcA2TnE (ORCPT ); Fri, 29 Jan 2016 14:43:04 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v3 1/3] x86/mm: Add INVPCID helpers Date: Fri, 29 Jan 2016 11:42:57 -0800 Message-Id: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> X-Mailer: git-send-email 2.5.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This adds helpers for each of the four currently-specified INVPCID modes. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6df2029405a3..8b576832777e 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -7,6 +7,54 @@ #include #include +static inline void __invpcid(unsigned long pcid, unsigned long addr, + unsigned long type) +{ + u64 desc[2] = { pcid, addr }; + + /* + * The memory clobber is because the whole point is to invalidate + * stale TLB entries and, especially if we're flushing global + * mappings, we don't want the compiler to reorder any subsequent + * memory accesses before the TLB flush. + * + * The hex opcode is invpcid (%ecx), %eax in 32-bit mode and + * invpcid (%rcx), %rax in long mode. + */ + asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" + : : "m" (desc), "a" (type), "c" (desc) : "memory"); +} + +#define INVPCID_TYPE_INDIV_ADDR 0 +#define INVPCID_TYPE_SINGLE_CTXT 1 +#define INVPCID_TYPE_ALL_INCL_GLOBAL 2 +#define INVPCID_TYPE_ALL_NON_GLOBAL 3 + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invpcid_flush_one(unsigned long pcid, + unsigned long addr) +{ + __invpcid(pcid, addr, INVPCID_TYPE_INDIV_ADDR); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invpcid_flush_single_context(unsigned long pcid) +{ + __invpcid(pcid, 0, INVPCID_TYPE_SINGLE_CTXT); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invpcid_flush_all(void) +{ + __invpcid(0, 0, INVPCID_TYPE_ALL_INCL_GLOBAL); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invpcid_flush_all_nonglobals(void) +{ + __invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL); +} + #ifdef CONFIG_PARAVIRT #include #else -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933090AbcA2TnM (ORCPT ); Fri, 29 Jan 2016 14:43:12 -0500 Received: from mail.kernel.org ([198.145.29.136]:58768 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756908AbcA2TnH (ORCPT ); Fri, 29 Jan 2016 14:43:07 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v3 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Date: Fri, 29 Jan 2016 11:42:59 -0800 Message-Id: X-Mailer: git-send-email 2.5.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 8b576832777e..985f1162c505 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -152,6 +152,15 @@ static inline void __native_flush_tlb_global(void) { unsigned long flags; + if (static_cpu_has_safe(X86_FEATURE_INVPCID)) { + /* + * Using INVPCID is considerably faster than a pair of writes + * to CR4 sandwiched inside an IRQ flag save/restore. + */ + invpcid_flush_all(); + return; + } + /* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756896AbcA2TnG (ORCPT ); Fri, 29 Jan 2016 14:43:06 -0500 Received: from mail.kernel.org ([198.145.29.136]:58693 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756753AbcA2TnE (ORCPT ); Fri, 29 Jan 2016 14:43:04 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v3 0/3] x86/mm: INVPCID support Date: Fri, 29 Jan 2016 11:42:56 -0800 Message-Id: X-Mailer: git-send-email 2.5.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Boris, I think you already have these prerequisites queued up: http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.) I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own. Changes from v2: - Add macros for the INVPCID mode numbers. - Add a changelog message for the chicken bit. v1 was exactly identical to patches 2-4 of the PCID RFC series. Andy Lutomirski (3): x86/mm: Add INVPCID helpers x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings Documentation/kernel-parameters.txt | 2 ++ arch/x86/include/asm/tlbflush.h | 57 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 16 +++++++++++ 3 files changed, 75 insertions(+) -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756937AbcA2Tnk (ORCPT ); Fri, 29 Jan 2016 14:43:40 -0500 Received: from mail.kernel.org ([198.145.29.136]:58746 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756890AbcA2TnG (ORCPT ); Fri, 29 Jan 2016 14:43:06 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v3 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Date: Fri, 29 Jan 2016 11:42:58 -0800 Message-Id: X-Mailer: git-send-email 2.5.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This adds a chicken bit to turn off INVPCID in case something goes wrong. It's an early_param because we do TLB flushes before we parse __setup parameters. Signed-off-by: Andy Lutomirski --- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 742f69d18fc8..b34e55e00bae 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2508,6 +2508,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nointroute [IA-64] + noinvpcid [X86] Disable the INVPCID cpu feature. + nojitter [IA-64] Disables jitter checking for ITC timers. no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c2b7522cbf35..48196980c1c7 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s) } __setup("nompx", x86_mpx_setup); +static int __init x86_noinvpcid_setup(char *s) +{ + /* noinvpcid doesn't accept parameters */ + if (s) + return -EINVAL; + + /* do not emit a message if the feature is not present */ + if (!boot_cpu_has(X86_FEATURE_INVPCID)) + return 0; + + setup_clear_cpu_cap(X86_FEATURE_INVPCID); + pr_info("noinvpcid: INVPCID feature disabled\n"); + return 0; +} +early_param("noinvpcid", x86_noinvpcid_setup); + #ifdef CONFIG_X86_32 static int cachesize_override = -1; static int disable_x86_serial_nr = 1; -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757744AbcBIQIz (ORCPT ); Tue, 9 Feb 2016 11:08:55 -0500 Received: from terminus.zytor.com ([198.137.202.10]:38381 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757723AbcBIQIw (ORCPT ); Tue, 9 Feb 2016 11:08:52 -0500 Date: Tue, 9 Feb 2016 08:07:31 -0800 From: tip-bot for Andy Lutomirski Message-ID: Cc: toshi.kani@hp.com, luto@kernel.org, aryabinin@virtuozzo.com, hpa@zytor.com, linux-kernel@vger.kernel.org, torvalds@linux-foundation.org, luto@amacapital.net, bp@suse.de, peterz@infradead.org, dvlasenk@redhat.com, dave.hansen@linux.intel.com, akpm@linux-foundation.org, brgerst@gmail.com, mcgrof@suse.com, oleg@redhat.com, mingo@kernel.org, tglx@linutronix.de, bp@alien8.de Reply-To: mingo@kernel.org, tglx@linutronix.de, bp@alien8.de, oleg@redhat.com, brgerst@gmail.com, mcgrof@suse.com, dave.hansen@linux.intel.com, akpm@linux-foundation.org, dvlasenk@redhat.com, peterz@infradead.org, bp@suse.de, torvalds@linux-foundation.org, luto@amacapital.net, linux-kernel@vger.kernel.org, toshi.kani@hp.com, luto@kernel.org, aryabinin@virtuozzo.com, hpa@zytor.com In-Reply-To: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> References: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/mm] x86/mm: Add INVPCID helpers Git-Commit-ID: 060a402a1ddb551455ee410de2eadd3349f2801b X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 060a402a1ddb551455ee410de2eadd3349f2801b Gitweb: http://git.kernel.org/tip/060a402a1ddb551455ee410de2eadd3349f2801b Author: Andy Lutomirski AuthorDate: Fri, 29 Jan 2016 11:42:57 -0800 Committer: Ingo Molnar CommitDate: Tue, 9 Feb 2016 13:36:10 +0100 x86/mm: Add INVPCID helpers This adds helpers for each of the four currently-specified INVPCID modes. Signed-off-by: Andy Lutomirski Reviewed-by: Borislav Petkov Cc: Andrew Morton Cc: Andrey Ryabinin Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Luis R. Rodriguez Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Toshi Kani Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org Signed-off-by: Ingo Molnar --- arch/x86/include/asm/tlbflush.h | 48 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6df2029..8b57683 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -7,6 +7,54 @@ #include #include +static inline void __invpcid(unsigned long pcid, unsigned long addr, + unsigned long type) +{ + u64 desc[2] = { pcid, addr }; + + /* + * The memory clobber is because the whole point is to invalidate + * stale TLB entries and, especially if we're flushing global + * mappings, we don't want the compiler to reorder any subsequent + * memory accesses before the TLB flush. + * + * The hex opcode is invpcid (%ecx), %eax in 32-bit mode and + * invpcid (%rcx), %rax in long mode. + */ + asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" + : : "m" (desc), "a" (type), "c" (desc) : "memory"); +} + +#define INVPCID_TYPE_INDIV_ADDR 0 +#define INVPCID_TYPE_SINGLE_CTXT 1 +#define INVPCID_TYPE_ALL_INCL_GLOBAL 2 +#define INVPCID_TYPE_ALL_NON_GLOBAL 3 + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invpcid_flush_one(unsigned long pcid, + unsigned long addr) +{ + __invpcid(pcid, addr, INVPCID_TYPE_INDIV_ADDR); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invpcid_flush_single_context(unsigned long pcid) +{ + __invpcid(pcid, 0, INVPCID_TYPE_SINGLE_CTXT); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invpcid_flush_all(void) +{ + __invpcid(0, 0, INVPCID_TYPE_ALL_INCL_GLOBAL); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invpcid_flush_all_nonglobals(void) +{ + __invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL); +} + #ifdef CONFIG_PARAVIRT #include #else From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757820AbcBIQJ1 (ORCPT ); Tue, 9 Feb 2016 11:09:27 -0500 Received: from terminus.zytor.com ([198.137.202.10]:38416 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757798AbcBIQJY (ORCPT ); Tue, 9 Feb 2016 11:09:24 -0500 Date: Tue, 9 Feb 2016 08:07:52 -0800 From: tip-bot for Andy Lutomirski Message-ID: Cc: dave.hansen@linux.intel.com, bp@alien8.de, luto@kernel.org, mingo@kernel.org, hpa@zytor.com, tglx@linutronix.de, luto@amacapital.net, aryabinin@virtuozzo.com, dvlasenk@redhat.com, bp@suse.de, peterz@infradead.org, torvalds@linux-foundation.org, toshi.kani@hp.com, mcgrof@suse.com, oleg@redhat.com, brgerst@gmail.com, akpm@linux-foundation.org, linux-kernel@vger.kernel.org Reply-To: mingo@kernel.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, bp@suse.de, aryabinin@virtuozzo.com, dvlasenk@redhat.com, luto@amacapital.net, tglx@linutronix.de, hpa@zytor.com, oleg@redhat.com, brgerst@gmail.com, toshi.kani@hp.com, torvalds@linux-foundation.org, mcgrof@suse.com, peterz@infradead.org, akpm@linux-foundation.org, linux-kernel@vger.kernel.org In-Reply-To: References: To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/mm] x86/mm: Add a 'noinvpcid' boot option to turn off INVPCID Git-Commit-ID: d12a72b844a49d4162f24cefdab30bed3f86730e X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: d12a72b844a49d4162f24cefdab30bed3f86730e Gitweb: http://git.kernel.org/tip/d12a72b844a49d4162f24cefdab30bed3f86730e Author: Andy Lutomirski AuthorDate: Fri, 29 Jan 2016 11:42:58 -0800 Committer: Ingo Molnar CommitDate: Tue, 9 Feb 2016 13:36:10 +0100 x86/mm: Add a 'noinvpcid' boot option to turn off INVPCID This adds a chicken bit to turn off INVPCID in case something goes wrong. It's an early_param() because we do TLB flushes before we parse __setup() parameters. Signed-off-by: Andy Lutomirski Reviewed-by: Borislav Petkov Cc: Andrew Morton Cc: Andrey Ryabinin Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Luis R. Rodriguez Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Toshi Kani Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/f586317ed1bc2b87aee652267e515b90051af385.1454096309.git.luto@kernel.org Signed-off-by: Ingo Molnar --- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 551ecf0..e4c4d2a 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2566,6 +2566,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nointroute [IA-64] + noinvpcid [X86] Disable the INVPCID cpu feature. + nojitter [IA-64] Disables jitter checking for ITC timers. no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 37830de..f4d0aa6 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s) } __setup("nompx", x86_mpx_setup); +static int __init x86_noinvpcid_setup(char *s) +{ + /* noinvpcid doesn't accept parameters */ + if (s) + return -EINVAL; + + /* do not emit a message if the feature is not present */ + if (!boot_cpu_has(X86_FEATURE_INVPCID)) + return 0; + + setup_clear_cpu_cap(X86_FEATURE_INVPCID); + pr_info("noinvpcid: INVPCID feature disabled\n"); + return 0; +} +early_param("noinvpcid", x86_noinvpcid_setup); + #ifdef CONFIG_X86_32 static int cachesize_override = -1; static int disable_x86_serial_nr = 1; From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932887AbcBIQOK (ORCPT ); Tue, 9 Feb 2016 11:14:10 -0500 Received: from terminus.zytor.com ([198.137.202.10]:38419 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757803AbcBIQJ0 (ORCPT ); Tue, 9 Feb 2016 11:09:26 -0500 Date: Tue, 9 Feb 2016 08:08:12 -0800 From: tip-bot for Andy Lutomirski Message-ID: Cc: dvlasenk@redhat.com, mcgrof@suse.com, oleg@redhat.com, aryabinin@virtuozzo.com, luto@amacapital.net, linux-kernel@vger.kernel.org, hpa@zytor.com, tglx@linutronix.de, dave.hansen@linux.intel.com, mingo@kernel.org, bp@alien8.de, peterz@infradead.org, toshi.kani@hp.com, akpm@linux-foundation.org, bp@suse.de, torvalds@linux-foundation.org, luto@kernel.org, brgerst@gmail.com Reply-To: bp@alien8.de, mingo@kernel.org, luto@amacapital.net, linux-kernel@vger.kernel.org, oleg@redhat.com, aryabinin@virtuozzo.com, mcgrof@suse.com, dvlasenk@redhat.com, tglx@linutronix.de, dave.hansen@linux.intel.com, hpa@zytor.com, torvalds@linux-foundation.org, brgerst@gmail.com, luto@kernel.org, toshi.kani@hp.com, peterz@infradead.org, bp@suse.de, akpm@linux-foundation.org In-Reply-To: References: To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/mm] x86/mm: If INVPCID is available, use it to flush global mappings Git-Commit-ID: d8bced79af1db6734f66b42064cc773cada2ce99 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: d8bced79af1db6734f66b42064cc773cada2ce99 Gitweb: http://git.kernel.org/tip/d8bced79af1db6734f66b42064cc773cada2ce99 Author: Andy Lutomirski AuthorDate: Fri, 29 Jan 2016 11:42:59 -0800 Committer: Ingo Molnar CommitDate: Tue, 9 Feb 2016 13:36:11 +0100 x86/mm: If INVPCID is available, use it to flush global mappings On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns. Signed-off-by: Andy Lutomirski Reviewed-by: Borislav Petkov Cc: Andrew Morton Cc: Andrey Ryabinin Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Luis R. Rodriguez Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Toshi Kani Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/ed0ef62581c0ea9c99b9bf6df726015e96d44743.1454096309.git.luto@kernel.org Signed-off-by: Ingo Molnar --- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 8b57683..fc9a2fd 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -152,6 +152,15 @@ static inline void __native_flush_tlb_global(void) { unsigned long flags; + if (static_cpu_has(X86_FEATURE_INVPCID)) { + /* + * Using INVPCID is considerably faster than a pair of writes + * to CR4 sandwiched inside an IRQ flag save/restore. + */ + invpcid_flush_all(); + return; + } + /* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757745AbcBJMIe (ORCPT ); Wed, 10 Feb 2016 07:08:34 -0500 Received: from mail.skyhub.de ([78.46.96.112]:60871 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752813AbcBJMIc (ORCPT ); Wed, 10 Feb 2016 07:08:32 -0500 Date: Wed, 10 Feb 2016 13:08:27 +0100 From: Borislav Petkov To: mingo@kernel.org, luto@amacapital.net Cc: tglx@linutronix.de, oleg@redhat.com, brgerst@gmail.com, mcgrof@suse.com, dave.hansen@linux.intel.com, akpm@linux-foundation.org, dvlasenk@redhat.com, peterz@infradead.org, torvalds@linux-foundation.org, linux-kernel@vger.kernel.org, toshi.kani@hp.com, luto@kernel.org, aryabinin@virtuozzo.com, hpa@zytor.com, Michael Matz , linux-tip-commits@vger.kernel.org Subject: Re: [tip:x86/mm] x86/mm: Add INVPCID helpers Message-ID: <20160210120827.GA11832@pd.tnic> References: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Feb 09, 2016 at 08:07:31AM -0800, tip-bot for Andy Lutomirski wrote: > Commit-ID: 060a402a1ddb551455ee410de2eadd3349f2801b > Gitweb: http://git.kernel.org/tip/060a402a1ddb551455ee410de2eadd3349f2801b > Author: Andy Lutomirski > AuthorDate: Fri, 29 Jan 2016 11:42:57 -0800 > Committer: Ingo Molnar > CommitDate: Tue, 9 Feb 2016 13:36:10 +0100 > > x86/mm: Add INVPCID helpers > > This adds helpers for each of the four currently-specified INVPCID > modes. ... > +static inline void __invpcid(unsigned long pcid, unsigned long addr, > + unsigned long type) > +{ > + u64 desc[2] = { pcid, addr }; > + > + /* > + * The memory clobber is because the whole point is to invalidate > + * stale TLB entries and, especially if we're flushing global > + * mappings, we don't want the compiler to reorder any subsequent > + * memory accesses before the TLB flush. > + * > + * The hex opcode is invpcid (%ecx), %eax in 32-bit mode and > + * invpcid (%rcx), %rax in long mode. > + */ > + asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" > + : : "m" (desc), "a" (type), "c" (desc) : "memory"); --- From: Borislav Petkov Date: Wed, 10 Feb 2016 12:53:48 +0100 Subject: [PATCH] x86/mm: Fix INVPCID asm constraint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit So we want to specify the dependency on both @pcid and @addr so that the compiler doesn't reorder accesses to them *before* the TLB flush. But for that to work, we need to express this properly in the inline asm and deref the whole desc array, not the pointer to it. See clwb() for an example. This fixes the build error on 32-bit: arch/x86/include/asm/tlbflush.h: In function ‘__invpcid’: arch/x86/include/asm/tlbflush.h:26:18: error: memory input 0 is not directly addressable which gcc4.7 caught but 5.x didn't. Which is strange. :-\ Signed-off-by: Borislav Petkov Cc: Andrew Morton Cc: Andrey Ryabinin Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Ingo Molnar Cc: Linus Torvalds Cc: Luis R. Rodriguez Cc: Michael Matz Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Toshi Kani Cc: linux-mm@kvack.org --- arch/x86/include/asm/tlbflush.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6f9e27aa2aaf..c820b2a9e026 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, * invpcid (%rcx), %rax in long mode. */ asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" - : : "m" (desc), "a" (type), "c" (desc) : "memory"); + : : "m" (*desc), "a" (type), "c" (desc) : "memory"); } #define INVPCID_TYPE_INDIV_ADDR 0 -- 2.3.5 -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751527AbcBJNsI (ORCPT ); Wed, 10 Feb 2016 08:48:08 -0500 Received: from mx2.suse.de ([195.135.220.15]:51037 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751029AbcBJNsF (ORCPT ); Wed, 10 Feb 2016 08:48:05 -0500 Date: Wed, 10 Feb 2016 14:48:02 +0100 (CET) From: Michael Matz To: Borislav Petkov cc: mingo@kernel.org, luto@amacapital.net, tglx@linutronix.de, oleg@redhat.com, brgerst@gmail.com, mcgrof@suse.com, dave.hansen@linux.intel.com, akpm@linux-foundation.org, dvlasenk@redhat.com, peterz@infradead.org, torvalds@linux-foundation.org, linux-kernel@vger.kernel.org, toshi.kani@hp.com, luto@kernel.org, aryabinin@virtuozzo.com, hpa@zytor.com, linux-tip-commits@vger.kernel.org Subject: Re: [tip:x86/mm] x86/mm: Add INVPCID helpers In-Reply-To: <20160210120827.GA11832@pd.tnic> Message-ID: References: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> <20160210120827.GA11832@pd.tnic> User-Agent: Alpine 2.20 (LSU 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On Wed, 10 Feb 2016, Borislav Petkov wrote: > --- a/arch/x86/include/asm/tlbflush.h > +++ b/arch/x86/include/asm/tlbflush.h > @@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, > * invpcid (%rcx), %rax in long mode. > */ > asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" > - : : "m" (desc), "a" (type), "c" (desc) : "memory"); > + : : "m" (*desc), "a" (type), "c" (desc) : "memory"); That still doesn't do what you want. Arrays in C are funny. *desc is exactly equivalent to desc[0], _not_ to the whole array, indeed there's no C syntax to name an lvalue of array type in normal expressions. You need to jump through hoops for this: "m" (*(struct {unsigned long x[2];} *)desc) It'd probably be easier to simply declare the descriptor as a struct, rather than an array, then the original syntax would have been mostly correct: struct {u64 d[2];} desc = { pcid, addr }; asm ... "m" (desc), "c" (&desc) Ciao, Michael. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751493AbcBJOvZ (ORCPT ); Wed, 10 Feb 2016 09:51:25 -0500 Received: from mail.skyhub.de ([78.46.96.112]:52558 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750761AbcBJOvX (ORCPT ); Wed, 10 Feb 2016 09:51:23 -0500 Date: Wed, 10 Feb 2016 15:51:16 +0100 From: Borislav Petkov To: Michael Matz , luto@amacapital.net Cc: mingo@kernel.org, tglx@linutronix.de, oleg@redhat.com, brgerst@gmail.com, mcgrof@suse.com, dave.hansen@linux.intel.com, akpm@linux-foundation.org, dvlasenk@redhat.com, peterz@infradead.org, torvalds@linux-foundation.org, linux-kernel@vger.kernel.org, toshi.kani@hp.com, luto@kernel.org, aryabinin@virtuozzo.com, hpa@zytor.com, linux-tip-commits@vger.kernel.org Subject: [PATCH -v1.1] x86/mm: Fix INVPCID asm constraint Message-ID: <20160210145116.GF23914@pd.tnic> References: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> <20160210120827.GA11832@pd.tnic> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Feb 10, 2016 at 02:48:02PM +0100, Michael Matz wrote: > Hi, > > On Wed, 10 Feb 2016, Borislav Petkov wrote: > > > --- a/arch/x86/include/asm/tlbflush.h > > +++ b/arch/x86/include/asm/tlbflush.h > > @@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, > > * invpcid (%rcx), %rax in long mode. > > */ > > asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" > > - : : "m" (desc), "a" (type), "c" (desc) : "memory"); > > + : : "m" (*desc), "a" (type), "c" (desc) : "memory"); > > That still doesn't do what you want. Arrays in C are funny. *desc is > exactly equivalent to desc[0], _not_ to the whole array, Doh! > indeed there's no C syntax to name an lvalue of array type in normal > expressions. You need to jump through hoops for this: > > "m" (*(struct {unsigned long x[2];} *)desc) Aha! That's why we wrapped the array in clwb() in a struct too, btw: static inline void clwb(volatile void *__p) { volatile struct { char x[64]; } *p = __p; ... > It'd probably be easier to simply declare the descriptor as a struct, > rather than an array, then the original syntax would have been mostly > correct: > > struct {u64 d[2];} desc = { pcid, addr }; > asm ... "m" (desc), "c" (&desc) Sounds better. Done. How does that below look like? Thanks Micha! --- From: Borislav Petkov Date: Wed, 10 Feb 2016 12:53:48 +0100 Subject: [PATCH -v1.1] x86/mm: Fix INVPCID asm constraint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit So we want to specify the dependency on both @pcid and @addr so that the compiler doesn't reorder accesses to them *before* the TLB flush. But for that to work, we need to express this properly in the inline asm and deref the whole desc array, not the pointer to it. See clwb() for an example. This fixes the build error on 32-bit: arch/x86/include/asm/tlbflush.h: In function ‘__invpcid’: arch/x86/include/asm/tlbflush.h:26:18: error: memory input 0 is not directly addressable which gcc4.7 caught but 5.x didn't. Which is strange. :-\ Signed-off-by: Borislav Petkov Cc: Andrew Morton Cc: Andrey Ryabinin Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Ingo Molnar Cc: Linus Torvalds Cc: Luis R. Rodriguez Cc: Michael Matz Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Toshi Kani Cc: linux-mm@kvack.org --- arch/x86/include/asm/tlbflush.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6f9e27aa2aaf..c24b4224d439 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -11,7 +11,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, unsigned long type) { - u64 desc[2] = { pcid, addr }; + struct { u64 d[2]; } desc = { { pcid, addr } }; /* * The memory clobber is because the whole point is to invalidate @@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, * invpcid (%rcx), %rax in long mode. */ asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" - : : "m" (desc), "a" (type), "c" (desc) : "memory"); + : : "m" (desc), "a" (type), "c" (&desc) : "memory"); } #define INVPCID_TYPE_INDIV_ADDR 0 -- 2.3.5 -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752689AbcBKCtN (ORCPT ); Wed, 10 Feb 2016 21:49:13 -0500 Received: from mail-ob0-f176.google.com ([209.85.214.176]:36022 "EHLO mail-ob0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750748AbcBKCtK convert rfc822-to-8bit (ORCPT ); Wed, 10 Feb 2016 21:49:10 -0500 MIME-Version: 1.0 In-Reply-To: <20160210145116.GF23914@pd.tnic> References: <8a62b23ad686888cee01da134c91409e22064db9.1454096309.git.luto@kernel.org> <20160210120827.GA11832@pd.tnic> <20160210145116.GF23914@pd.tnic> From: Andy Lutomirski Date: Wed, 10 Feb 2016 18:48:50 -0800 Message-ID: Subject: Re: [PATCH -v1.1] x86/mm: Fix INVPCID asm constraint To: Borislav Petkov Cc: Michael Matz , Ingo Molnar , Thomas Gleixner , Oleg Nesterov , Brian Gerst , Luis Rodriguez , Dave Hansen , Andrew Morton , Denys Vlasenko , Peter Zijlstra , Linus Torvalds , "linux-kernel@vger.kernel.org" , Toshi Kani , Andrew Lutomirski , Andrey Ryabinin , "H. Peter Anvin" , "linux-tip-commits@vger.kernel.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Feb 10, 2016 at 6:51 AM, Borislav Petkov wrote: > On Wed, Feb 10, 2016 at 02:48:02PM +0100, Michael Matz wrote: >> Hi, >> >> On Wed, 10 Feb 2016, Borislav Petkov wrote: >> >> > --- a/arch/x86/include/asm/tlbflush.h >> > +++ b/arch/x86/include/asm/tlbflush.h >> > @@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, >> > * invpcid (%rcx), %rax in long mode. >> > */ >> > asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" >> > - : : "m" (desc), "a" (type), "c" (desc) : "memory"); >> > + : : "m" (*desc), "a" (type), "c" (desc) : "memory"); >> >> That still doesn't do what you want. Arrays in C are funny. *desc is >> exactly equivalent to desc[0], _not_ to the whole array, > > Doh! > >> indeed there's no C syntax to name an lvalue of array type in normal >> expressions. You need to jump through hoops for this: >> >> "m" (*(struct {unsigned long x[2];} *)desc) > > Aha! That's why we wrapped the array in clwb() in a struct too, btw: > > static inline void clwb(volatile void *__p) > { > volatile struct { char x[64]; } *p = __p; > ... > >> It'd probably be easier to simply declare the descriptor as a struct, >> rather than an array, then the original syntax would have been mostly >> correct: >> >> struct {u64 d[2];} desc = { pcid, addr }; >> asm ... "m" (desc), "c" (&desc) > > Sounds better. Done. How does that below look like? > > Thanks Micha! > > --- > From: Borislav Petkov > Date: Wed, 10 Feb 2016 12:53:48 +0100 > Subject: [PATCH -v1.1] x86/mm: Fix INVPCID asm constraint > MIME-Version: 1.0 > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: 8bit > > So we want to specify the dependency on both @pcid and @addr so that the > compiler doesn't reorder accesses to them *before* the TLB flush. But > for that to work, we need to express this properly in the inline asm and > deref the whole desc array, not the pointer to it. See clwb() for an > example. > > This fixes the build error on 32-bit: > > arch/x86/include/asm/tlbflush.h: In function ‘__invpcid’: > arch/x86/include/asm/tlbflush.h:26:18: error: memory input 0 is not directly addressable > Acked-by: Andy Lutomirski