From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f170.google.com (mail-pf0-f170.google.com [209.85.192.170]) by kanga.kvack.org (Postfix) with ESMTP id CA7756B0253 for ; Mon, 25 Jan 2016 13:37:47 -0500 (EST) Received: by mail-pf0-f170.google.com with SMTP id q63so88248074pfb.1 for ; Mon, 25 Jan 2016 10:37:47 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id xu3si22168401pab.94.2016.01.25.10.37.47 for ; Mon, 25 Jan 2016 10:37:47 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v2 0/3] x86/mm: INVPCID support Date: Mon, 25 Jan 2016 10:37:41 -0800 Message-Id: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Ingo, before applying this, please apply these two KASAN fixes: http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com Without those fixes, this series will trigger a KASAN bug. This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.) I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own. This is exactly identical to patches 2-4 of the PCID RFC series. Andy Lutomirski (3): x86/mm: Add INVPCID helpers x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings Documentation/kernel-parameters.txt | 2 ++ arch/x86/include/asm/tlbflush.h | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++ 3 files changed, 68 insertions(+) -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f46.google.com (mail-pa0-f46.google.com [209.85.220.46]) by kanga.kvack.org (Postfix) with ESMTP id 3CFD06B0254 for ; Mon, 25 Jan 2016 13:37:49 -0500 (EST) Received: by mail-pa0-f46.google.com with SMTP id uo6so86371207pac.1 for ; Mon, 25 Jan 2016 10:37:49 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id r72si35293889pfb.1.2016.01.25.10.37.48 for ; Mon, 25 Jan 2016 10:37:48 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v2 1/3] x86/mm: Add INVPCID helpers Date: Mon, 25 Jan 2016 10:37:42 -0800 Message-Id: In-Reply-To: References: In-Reply-To: References: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski This adds helpers for each of the four currently-specified INVPCID modes. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6df2029405a3..20fc38d8478a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -7,6 +7,47 @@ #include #include +static inline void __invpcid(unsigned long pcid, unsigned long addr, + unsigned long type) +{ + u64 desc[2] = { pcid, addr }; + + /* + * The memory clobber is because the whole point is to invalidate + * stale TLB entries and, especially if we're flushing global + * mappings, we don't want the compiler to reorder any subsequent + * memory accesses before the TLB flush. + */ + asm volatile ( + ".byte 0x66, 0x0f, 0x38, 0x82, 0x01" /* invpcid (%cx), %ax */ + : : "m" (desc), "a" (type), "c" (desc) : "memory"); +} + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invpcid_flush_one(unsigned long pcid, + unsigned long addr) +{ + __invpcid(pcid, addr, 0); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invpcid_flush_single_context(unsigned long pcid) +{ + __invpcid(pcid, 0, 1); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invpcid_flush_everything(void) +{ + __invpcid(0, 0, 2); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invpcid_flush_all_nonglobals(void) +{ + __invpcid(0, 0, 3); +} + #ifdef CONFIG_PARAVIRT #include #else -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) by kanga.kvack.org (Postfix) with ESMTP id D61C16B0255 for ; Mon, 25 Jan 2016 13:37:51 -0500 (EST) Received: by mail-pa0-f43.google.com with SMTP id cy9so84742173pac.0 for ; Mon, 25 Jan 2016 10:37:51 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id b17si35216481pfj.86.2016.01.25.10.37.49 for ; Mon, 25 Jan 2016 10:37:49 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v2 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Date: Mon, 25 Jan 2016 10:37:43 -0800 Message-Id: <321b1dde2b4341df44a7c408381c83905aa1762c.1453746505.git.luto@kernel.org> In-Reply-To: References: In-Reply-To: References: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Signed-off-by: Andy Lutomirski --- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 742f69d18fc8..b34e55e00bae 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2508,6 +2508,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nointroute [IA-64] + noinvpcid [X86] Disable the INVPCID cpu feature. + nojitter [IA-64] Disables jitter checking for ITC timers. no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c2b7522cbf35..48196980c1c7 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s) } __setup("nompx", x86_mpx_setup); +static int __init x86_noinvpcid_setup(char *s) +{ + /* noinvpcid doesn't accept parameters */ + if (s) + return -EINVAL; + + /* do not emit a message if the feature is not present */ + if (!boot_cpu_has(X86_FEATURE_INVPCID)) + return 0; + + setup_clear_cpu_cap(X86_FEATURE_INVPCID); + pr_info("noinvpcid: INVPCID feature disabled\n"); + return 0; +} +early_param("noinvpcid", x86_noinvpcid_setup); + #ifdef CONFIG_X86_32 static int cachesize_override = -1; static int disable_x86_serial_nr = 1; -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) by kanga.kvack.org (Postfix) with ESMTP id C19CA6B0256 for ; Mon, 25 Jan 2016 13:37:53 -0500 (EST) Received: by mail-pa0-f41.google.com with SMTP id yy13so84459796pab.3 for ; Mon, 25 Jan 2016 10:37:53 -0800 (PST) Received: from mail.kernel.org (mail.kernel.org. [198.145.29.136]) by mx.google.com with ESMTP id rp7si35138147pab.99.2016.01.25.10.37.50 for ; Mon, 25 Jan 2016 10:37:50 -0800 (PST) From: Andy Lutomirski Subject: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Date: Mon, 25 Jan 2016 10:37:44 -0800 Message-Id: In-Reply-To: References: In-Reply-To: References: Sender: owner-linux-mm@kvack.org List-ID: To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 20fc38d8478a..4eba5164430d 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -145,6 +145,15 @@ static inline void __native_flush_tlb_global(void) { unsigned long flags; + if (static_cpu_has_safe(X86_FEATURE_INVPCID)) { + /* + * Using INVPCID is considerably faster than a pair of writes + * to CR4 sandwiched inside an IRQ flag save/restore. + */ + invpcid_flush_everything(); + return; + } + /* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can -- 2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by kanga.kvack.org (Postfix) with ESMTP id 0D5606B0255 for ; Mon, 25 Jan 2016 13:57:11 -0500 (EST) Received: by mail-wm0-f42.google.com with SMTP id n5so95465733wmn.0 for ; Mon, 25 Jan 2016 10:57:11 -0800 (PST) Received: from mail-wm0-x244.google.com (mail-wm0-x244.google.com. [2a00:1450:400c:c09::244]) by mx.google.com with ESMTPS id t187si106206wmg.54.2016.01.25.10.57.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Jan 2016 10:57:09 -0800 (PST) Received: by mail-wm0-x244.google.com with SMTP id u188so13045173wmu.0 for ; Mon, 25 Jan 2016 10:57:09 -0800 (PST) Date: Mon, 25 Jan 2016 19:57:06 +0100 From: Ingo Molnar Subject: Re: [PATCH v2 0/3] x86/mm: INVPCID support Message-ID: <20160125185706.GA28416@gmail.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Andy Lutomirski Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin * Andy Lutomirski wrote: > Ingo, before applying this, please apply these two KASAN fixes: > > http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com > http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com > > Without those fixes, this series will trigger a KASAN bug. > > This is a straightforward speedup on Ivy Bridge and newer, IIRC. > (I tested on Skylake. INVPCID is not available on Sandy Bridge. > I don't have Ivy Bridge, Haswell or Broadwell to test on, so I > could be wrong as to when the feature was introduced.) > > I think we should consider these patches separately from the rest > of the PCID stuff -- they barely interact, and this part is much > simpler and is useful on its own. > > This is exactly identical to patches 2-4 of the PCID RFC series. > > Andy Lutomirski (3): > x86/mm: Add INVPCID helpers > x86/mm: Add a noinvpcid option to turn off INVPCID > x86/mm: If INVPCID is available, use it to flush global mappings > > Documentation/kernel-parameters.txt | 2 ++ > arch/x86/include/asm/tlbflush.h | 50 +++++++++++++++++++++++++++++++++++++ > arch/x86/kernel/cpu/common.c | 16 ++++++++++++ > 3 files changed, 68 insertions(+) Ok, I'll pick these up tomorrow unless there are objections. Thanks, Ingo -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by kanga.kvack.org (Postfix) with ESMTP id 634436B0253 for ; Wed, 27 Jan 2016 05:10:17 -0500 (EST) Received: by mail-wm0-f42.google.com with SMTP id r129so138521146wmr.0 for ; Wed, 27 Jan 2016 02:10:17 -0800 (PST) Received: from Galois.linutronix.de (linutronix.de. [2001:470:1f0b:db:abcd:42:0:1]) by mx.google.com with ESMTPS id hn6si7433918wjc.229.2016.01.27.02.10.16 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 27 Jan 2016 02:10:16 -0800 (PST) Date: Wed, 27 Jan 2016 11:09:04 +0100 (CET) From: Thomas Gleixner Subject: Re: several messages In-Reply-To: <20160125185706.GA28416@gmail.com> Message-ID: References: <20160125185706.GA28416@gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Andy Lutomirski , Ingo Molnar Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin On Mon, 25 Jan 2016, Andy Lutomirski wrote: > This is a straightforward speedup on Ivy Bridge and newer, IIRC. > (I tested on Skylake. INVPCID is not available on Sandy Bridge. > I don't have Ivy Bridge, Haswell or Broadwell to test on, so I > could be wrong as to when the feature was introduced.) Haswell and Broadwell have it. No idea about ivy bridge. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-f176.google.com (mail-ob0-f176.google.com [209.85.214.176]) by kanga.kvack.org (Postfix) with ESMTP id D61136B0009 for ; Fri, 29 Jan 2016 12:35:42 -0500 (EST) Received: by mail-ob0-f176.google.com with SMTP id ba1so68947014obb.3 for ; Fri, 29 Jan 2016 09:35:42 -0800 (PST) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com. [2607:f8b0:4003:c01::22a]) by mx.google.com with ESMTPS id i64si15736445oif.58.2016.01.29.09.35.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jan 2016 09:35:41 -0800 (PST) Received: by mail-ob0-x22a.google.com with SMTP id ny8so47196363obc.2 for ; Fri, 29 Jan 2016 09:35:41 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20160129142625.GH10187@pd.tnic> References: <20160129142625.GH10187@pd.tnic> From: Andy Lutomirski Date: Fri, 29 Jan 2016 09:35:22 -0800 Message-ID: Subject: Re: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Borislav Petkov Cc: Andy Lutomirski , X86 ML , "linux-kernel@vger.kernel.org" , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin On Fri, Jan 29, 2016 at 6:26 AM, Borislav Petkov wrote: > On Mon, Jan 25, 2016 at 10:37:44AM -0800, Andy Lutomirski wrote: >> On my Skylake laptop, INVPCID function 2 (flush absolutely >> everything) takes about 376ns, whereas saving flags, twiddling >> CR4.PGE to flush global mappings, and restoring flags takes about >> 539ns. > > FWIW, I ran your microbenchmark on the IVB laptop I have here 3 times > and some of the numbers from each run are pretty unstable. Not that it > means a whole lot - the thing doesn't have INVPCID support. > > I'm just questioning the microbenchmark and whether we should be rather > doing those measurements with a real benchmark, whatever that means. My > limited experience says that measuring TLB performance is hard. > > ./context_switch_latency 0 thread same > use_xstate = 0 > Using threads > 1: 100000 iters at 2676.2 ns/switch > 2: 100000 iters at 2700.2 ns/switch > 3: 100000 iters at 2656.1 ns/switch > > ./context_switch_latency 0 thread different > use_xstate = 0 > Using threads > 1: 100000 iters at 5174.8 ns/switch > 2: 100000 iters at 5140.5 ns/switch > 3: 100000 iters at 5292.9 ns/switch > > ./context_switch_latency 0 process same > use_xstate = 0 > Using a subprocess > 1: 100000 iters at 2361.2 ns/switch > 2: 100000 iters at 2332.2 ns/switch > 3: 100000 iters at 3436.9 ns/switch > > ./context_switch_latency 0 process different > use_xstate = 0 > Using a subprocess > 1: 100000 iters at 4713.6 ns/switch > 2: 100000 iters at 4957.5 ns/switch > 3: 100000 iters at 5012.2 ns/switch > > ./context_switch_latency 1 thread same > use_xstate = 1 > Using threads > 1: 100000 iters at 2505.6 ns/switch > 2: 100000 iters at 2483.1 ns/switch > 3: 100000 iters at 2479.7 ns/switch > > ./context_switch_latency 1 thread different > use_xstate = 1 > Using threads > 1: 100000 iters at 5245.9 ns/switch > 2: 100000 iters at 5241.1 ns/switch > 3: 100000 iters at 5220.3 ns/switch > > ./context_switch_latency 1 process same > use_xstate = 1 > Using a subprocess > 1: 100000 iters at 2329.8 ns/switch > 2: 100000 iters at 2350.2 ns/switch > 3: 100000 iters at 2500.9 ns/switch > > ./context_switch_latency 1 process different > use_xstate = 1 > Using a subprocess > 1: 100000 iters at 4970.7 ns/switch > 2: 100000 iters at 5034.0 ns/switch > 3: 100000 iters at 4991.6 ns/switch > I'll fiddle with that benchmark a little bit. Maybe I can make it suck less. If anyone knows a good non-micro benchmark for this, let me know. I refuse to use dbus as my benchmark :) FWIW, I benchmarked cr4 vs invpcid by adding a prctl and calling it in a loop. If Ingo's fpu benchmark thing ever lands, I'll gladly send a patch to add TLB flushes to it. --Andy -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: [PATCH v2 1/3] x86/mm: Add INVPCID helpers Date: Fri, 29 Jan 2016 12:19:37 +0100 Message-ID: <20160129111937.GB10187@pd.tnic> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Andy Lutomirski Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin List-Id: linux-mm.kvack.org On Mon, Jan 25, 2016 at 10:37:42AM -0800, Andy Lutomirski wrote: > This adds helpers for each of the four currently-specified INVPCID > modes. > > Signed-off-by: Andy Lutomirski > --- > arch/x86/include/asm/tlbflush.h | 41 +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 41 insertions(+) > > diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h > index 6df2029405a3..20fc38d8478a 100644 > --- a/arch/x86/include/asm/tlbflush.h > +++ b/arch/x86/include/asm/tlbflush.h > @@ -7,6 +7,47 @@ > #include > #include > > +static inline void __invpcid(unsigned long pcid, unsigned long addr, > + unsigned long type) > +{ > + u64 desc[2] = { pcid, addr }; > + > + /* > + * The memory clobber is because the whole point is to invalidate > + * stale TLB entries and, especially if we're flushing global > + * mappings, we don't want the compiler to reorder any subsequent > + * memory accesses before the TLB flush. > + */ > + asm volatile ( Yeah, no need for that linebreak here: asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01" reads fine too. > + ".byte 0x66, 0x0f, 0x38, 0x82, 0x01" /* invpcid (%cx), %ax */ > + : : "m" (desc), "a" (type), "c" (desc) : "memory"); > +} > + Please add defines for the invalidation types: #define INVPCID_TYPE_INDIVIDUAL 0 #define INVPCID_TYPE_SINGLE_CTXT 1 #define INVPCID_TYPE_ALL 2 #define INVPCID_TYPE_ALL_NON_GLOBAL 3 and add macros: #define invpcid_flush_one(pcid, addr) __invpcid(pcid, addr, INVPCID_TYPE_INDIVIDUAL) ... and so on. Oh, and the "flush everything" macro I'd call invpcid_flush_all() like tlb_flush_all(). Thanks. -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: [PATCH v2 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Date: Fri, 29 Jan 2016 12:21:07 +0100 Message-ID: <20160129112106.GC10187@pd.tnic> References: <321b1dde2b4341df44a7c408381c83905aa1762c.1453746505.git.luto@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Return-path: Content-Disposition: inline In-Reply-To: <321b1dde2b4341df44a7c408381c83905aa1762c.1453746505.git.luto@kernel.org> Sender: linux-kernel-owner@vger.kernel.org To: Andy Lutomirski Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin List-Id: linux-mm.kvack.org On Mon, Jan 25, 2016 at 10:37:43AM -0800, Andy Lutomirski wrote: <--- Commit message please, albeit a trivial one like "Add a chicken bit ..." > Signed-off-by: Andy Lutomirski > --- > Documentation/kernel-parameters.txt | 2 ++ > arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ > 2 files changed, 18 insertions(+) -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: several messages Date: Fri, 29 Jan 2016 14:21:14 +0100 Message-ID: <20160129132114.GF10187@pd.tnic> References: <20160125185706.GA28416@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Thomas Gleixner Cc: Andy Lutomirski , Ingo Molnar , x86@kernel.org, linux-kernel@vger.kernel.org, Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin List-Id: linux-mm.kvack.org On Wed, Jan 27, 2016 at 11:09:04AM +0100, Thomas Gleixner wrote: > On Mon, 25 Jan 2016, Andy Lutomirski wrote: > > This is a straightforward speedup on Ivy Bridge and newer, IIRC. > > (I tested on Skylake. INVPCID is not available on Sandy Bridge. > > I don't have Ivy Bridge, Haswell or Broadwell to test on, so I > > could be wrong as to when the feature was introduced.) > > Haswell and Broadwell have it. No idea about ivy bridge. I have an IVB model 58. It doesn't have it: CPUID_0x00000007: EAX=0x00000000, EBX=0x00000281, ECX=0x00000000, EDX=0x00000000 INVPCID should be EBX[10]. -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Date: Fri, 29 Jan 2016 15:26:25 +0100 Message-ID: <20160129142625.GH10187@pd.tnic> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Andy Lutomirski Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin List-Id: linux-mm.kvack.org On Mon, Jan 25, 2016 at 10:37:44AM -0800, Andy Lutomirski wrote: > On my Skylake laptop, INVPCID function 2 (flush absolutely > everything) takes about 376ns, whereas saving flags, twiddling > CR4.PGE to flush global mappings, and restoring flags takes about > 539ns. FWIW, I ran your microbenchmark on the IVB laptop I have here 3 times and some of the numbers from each run are pretty unstable. Not that it means a whole lot - the thing doesn't have INVPCID support. I'm just questioning the microbenchmark and whether we should be rather doing those measurements with a real benchmark, whatever that means. My limited experience says that measuring TLB performance is hard. ./context_switch_latency 0 thread same use_xstate = 0 Using threads 1: 100000 iters at 2676.2 ns/switch 2: 100000 iters at 2700.2 ns/switch 3: 100000 iters at 2656.1 ns/switch ./context_switch_latency 0 thread different use_xstate = 0 Using threads 1: 100000 iters at 5174.8 ns/switch 2: 100000 iters at 5140.5 ns/switch 3: 100000 iters at 5292.9 ns/switch ./context_switch_latency 0 process same use_xstate = 0 Using a subprocess 1: 100000 iters at 2361.2 ns/switch 2: 100000 iters at 2332.2 ns/switch 3: 100000 iters at 3436.9 ns/switch ./context_switch_latency 0 process different use_xstate = 0 Using a subprocess 1: 100000 iters at 4713.6 ns/switch 2: 100000 iters at 4957.5 ns/switch 3: 100000 iters at 5012.2 ns/switch ./context_switch_latency 1 thread same use_xstate = 1 Using threads 1: 100000 iters at 2505.6 ns/switch 2: 100000 iters at 2483.1 ns/switch 3: 100000 iters at 2479.7 ns/switch ./context_switch_latency 1 thread different use_xstate = 1 Using threads 1: 100000 iters at 5245.9 ns/switch 2: 100000 iters at 5241.1 ns/switch 3: 100000 iters at 5220.3 ns/switch ./context_switch_latency 1 process same use_xstate = 1 Using a subprocess 1: 100000 iters at 2329.8 ns/switch 2: 100000 iters at 2350.2 ns/switch 3: 100000 iters at 2500.9 ns/switch ./context_switch_latency 1 process different use_xstate = 1 Using a subprocess 1: 100000 iters at 4970.7 ns/switch 2: 100000 iters at 5034.0 ns/switch 3: 100000 iters at 4991.6 ns/switch -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Date: Fri, 29 Jan 2016 19:27:17 +0100 Message-ID: <20160129182717.GA17459@pd.tnic> References: <20160129142625.GH10187@pd.tnic> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Andy Lutomirski Cc: Andy Lutomirski , X86 ML , "linux-kernel@vger.kernel.org" , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin List-Id: linux-mm.kvack.org On Fri, Jan 29, 2016 at 09:35:22AM -0800, Andy Lutomirski wrote: > I'll fiddle with that benchmark a little bit. Maybe I can make it > suck less. If anyone knows a good non-micro benchmark for this, let > me know. Yeah, I don't know of a good one. The TLB and all those intermediary walker caches modern x86 CPUs have are really good. So it is hard to measure any improvements there. I guess in this particular case, if one can't measure slowdowns and the code is simple enough, then we're good enough. In theory, we are carefully killing less TLB entries and the related cached page walker data so that should be a good thing... > I refuse to use dbus as my benchmark :) Ha! > FWIW, I benchmarked cr4 vs invpcid by adding a prctl and calling it in > a loop. Apparently INVPCID is faster than the two CR4 writes. -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S964932AbcAYShx (ORCPT ); Mon, 25 Jan 2016 13:37:53 -0500 Received: from mail.kernel.org ([198.145.29.136]:33186 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933660AbcAYShr (ORCPT ); Mon, 25 Jan 2016 13:37:47 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v2 0/3] x86/mm: INVPCID support Date: Mon, 25 Jan 2016 10:37:41 -0800 Message-Id: X-Mailer: git-send-email 2.5.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Ingo, before applying this, please apply these two KASAN fixes: http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com Without those fixes, this series will trigger a KASAN bug. This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.) I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own. This is exactly identical to patches 2-4 of the PCID RFC series. Andy Lutomirski (3): x86/mm: Add INVPCID helpers x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings Documentation/kernel-parameters.txt | 2 ++ arch/x86/include/asm/tlbflush.h | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++ 3 files changed, 68 insertions(+) -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S964949AbcAYSh4 (ORCPT ); Mon, 25 Jan 2016 13:37:56 -0500 Received: from mail.kernel.org ([198.145.29.136]:33219 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S964915AbcAYShs (ORCPT ); Mon, 25 Jan 2016 13:37:48 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v2 1/3] x86/mm: Add INVPCID helpers Date: Mon, 25 Jan 2016 10:37:42 -0800 Message-Id: X-Mailer: git-send-email 2.5.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This adds helpers for each of the four currently-specified INVPCID modes. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 6df2029405a3..20fc38d8478a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -7,6 +7,47 @@ #include #include +static inline void __invpcid(unsigned long pcid, unsigned long addr, + unsigned long type) +{ + u64 desc[2] = { pcid, addr }; + + /* + * The memory clobber is because the whole point is to invalidate + * stale TLB entries and, especially if we're flushing global + * mappings, we don't want the compiler to reorder any subsequent + * memory accesses before the TLB flush. + */ + asm volatile ( + ".byte 0x66, 0x0f, 0x38, 0x82, 0x01" /* invpcid (%cx), %ax */ + : : "m" (desc), "a" (type), "c" (desc) : "memory"); +} + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invpcid_flush_one(unsigned long pcid, + unsigned long addr) +{ + __invpcid(pcid, addr, 0); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invpcid_flush_single_context(unsigned long pcid) +{ + __invpcid(pcid, 0, 1); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invpcid_flush_everything(void) +{ + __invpcid(0, 0, 2); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invpcid_flush_all_nonglobals(void) +{ + __invpcid(0, 0, 3); +} + #ifdef CONFIG_PARAVIRT #include #else -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934154AbcAYSiz (ORCPT ); Mon, 25 Jan 2016 13:38:55 -0500 Received: from mail.kernel.org ([198.145.29.136]:33280 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S964921AbcAYShv (ORCPT ); Mon, 25 Jan 2016 13:37:51 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Date: Mon, 25 Jan 2016 10:37:44 -0800 Message-Id: X-Mailer: git-send-email 2.5.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns. Signed-off-by: Andy Lutomirski --- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 20fc38d8478a..4eba5164430d 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -145,6 +145,15 @@ static inline void __native_flush_tlb_global(void) { unsigned long flags; + if (static_cpu_has_safe(X86_FEATURE_INVPCID)) { + /* + * Using INVPCID is considerably faster than a pair of writes + * to CR4 sandwiched inside an IRQ flag save/restore. + */ + invpcid_flush_everything(); + return; + } + /* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934169AbcAYSjJ (ORCPT ); Mon, 25 Jan 2016 13:39:09 -0500 Received: from mail.kernel.org ([198.145.29.136]:33249 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S964917AbcAYSht (ORCPT ); Mon, 25 Jan 2016 13:37:49 -0500 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin , Andy Lutomirski Subject: [PATCH v2 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Date: Mon, 25 Jan 2016 10:37:43 -0800 Message-Id: <321b1dde2b4341df44a7c408381c83905aa1762c.1453746505.git.luto@kernel.org> X-Mailer: git-send-email 2.5.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Signed-off-by: Andy Lutomirski --- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 742f69d18fc8..b34e55e00bae 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2508,6 +2508,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nointroute [IA-64] + noinvpcid [X86] Disable the INVPCID cpu feature. + nojitter [IA-64] Disables jitter checking for ITC timers. no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c2b7522cbf35..48196980c1c7 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s) } __setup("nompx", x86_mpx_setup); +static int __init x86_noinvpcid_setup(char *s) +{ + /* noinvpcid doesn't accept parameters */ + if (s) + return -EINVAL; + + /* do not emit a message if the feature is not present */ + if (!boot_cpu_has(X86_FEATURE_INVPCID)) + return 0; + + setup_clear_cpu_cap(X86_FEATURE_INVPCID); + pr_info("noinvpcid: INVPCID feature disabled\n"); + return 0; +} +early_param("noinvpcid", x86_noinvpcid_setup); + #ifdef CONFIG_X86_32 static int cachesize_override = -1; static int disable_x86_serial_nr = 1; -- 2.5.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933981AbcAYS5R (ORCPT ); Mon, 25 Jan 2016 13:57:17 -0500 Received: from mail-wm0-f66.google.com ([74.125.82.66]:34539 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932960AbcAYS5M (ORCPT ); Mon, 25 Jan 2016 13:57:12 -0500 Date: Mon, 25 Jan 2016 19:57:06 +0100 From: Ingo Molnar To: Andy Lutomirski Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin Subject: Re: [PATCH v2 0/3] x86/mm: INVPCID support Message-ID: <20160125185706.GA28416@gmail.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Andy Lutomirski wrote: > Ingo, before applying this, please apply these two KASAN fixes: > > http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com > http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com > > Without those fixes, this series will trigger a KASAN bug. > > This is a straightforward speedup on Ivy Bridge and newer, IIRC. > (I tested on Skylake. INVPCID is not available on Sandy Bridge. > I don't have Ivy Bridge, Haswell or Broadwell to test on, so I > could be wrong as to when the feature was introduced.) > > I think we should consider these patches separately from the rest > of the PCID stuff -- they barely interact, and this part is much > simpler and is useful on its own. > > This is exactly identical to patches 2-4 of the PCID RFC series. > > Andy Lutomirski (3): > x86/mm: Add INVPCID helpers > x86/mm: Add a noinvpcid option to turn off INVPCID > x86/mm: If INVPCID is available, use it to flush global mappings > > Documentation/kernel-parameters.txt | 2 ++ > arch/x86/include/asm/tlbflush.h | 50 +++++++++++++++++++++++++++++++++++++ > arch/x86/kernel/cpu/common.c | 16 ++++++++++++ > 3 files changed, 68 insertions(+) Ok, I'll pick these up tomorrow unless there are objections. Thanks, Ingo From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932268AbcA0KKU (ORCPT ); Wed, 27 Jan 2016 05:10:20 -0500 Received: from www.linutronix.de ([62.245.132.108]:42209 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932139AbcA0KKI (ORCPT ); Wed, 27 Jan 2016 05:10:08 -0500 Date: Wed, 27 Jan 2016 11:09:04 +0100 (CET) From: Thomas Gleixner To: Andy Lutomirski , Ingo Molnar cc: x86@kernel.org, linux-kernel@vger.kernel.org, Borislav Petkov , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin Subject: Re: several messages In-Reply-To: <20160125185706.GA28416@gmail.com> Message-ID: References: <20160125185706.GA28416@gmail.com> User-Agent: Alpine 2.11 (DEB 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Linutronix-Spam-Score: -1.0 X-Linutronix-Spam-Level: - X-Linutronix-Spam-Status: No , -1.0 points, 5.0 required, ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 25 Jan 2016, Andy Lutomirski wrote: > This is a straightforward speedup on Ivy Bridge and newer, IIRC. > (I tested on Skylake. INVPCID is not available on Sandy Bridge. > I don't have Ivy Bridge, Haswell or Broadwell to test on, so I > could be wrong as to when the feature was introduced.) Haswell and Broadwell have it. No idea about ivy bridge. Thanks, tglx From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756656AbcA2Rfn (ORCPT ); Fri, 29 Jan 2016 12:35:43 -0500 Received: from mail-ob0-f171.google.com ([209.85.214.171]:34411 "EHLO mail-ob0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751753AbcA2Rfm (ORCPT ); Fri, 29 Jan 2016 12:35:42 -0500 MIME-Version: 1.0 In-Reply-To: <20160129142625.GH10187@pd.tnic> References: <20160129142625.GH10187@pd.tnic> From: Andy Lutomirski Date: Fri, 29 Jan 2016 09:35:22 -0800 Message-ID: Subject: Re: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings To: Borislav Petkov Cc: Andy Lutomirski , X86 ML , "linux-kernel@vger.kernel.org" , Brian Gerst , Dave Hansen , Linus Torvalds , Oleg Nesterov , "linux-mm@kvack.org" , Andrey Ryabinin Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jan 29, 2016 at 6:26 AM, Borislav Petkov wrote: > On Mon, Jan 25, 2016 at 10:37:44AM -0800, Andy Lutomirski wrote: >> On my Skylake laptop, INVPCID function 2 (flush absolutely >> everything) takes about 376ns, whereas saving flags, twiddling >> CR4.PGE to flush global mappings, and restoring flags takes about >> 539ns. > > FWIW, I ran your microbenchmark on the IVB laptop I have here 3 times > and some of the numbers from each run are pretty unstable. Not that it > means a whole lot - the thing doesn't have INVPCID support. > > I'm just questioning the microbenchmark and whether we should be rather > doing those measurements with a real benchmark, whatever that means. My > limited experience says that measuring TLB performance is hard. > > ./context_switch_latency 0 thread same > use_xstate = 0 > Using threads > 1: 100000 iters at 2676.2 ns/switch > 2: 100000 iters at 2700.2 ns/switch > 3: 100000 iters at 2656.1 ns/switch > > ./context_switch_latency 0 thread different > use_xstate = 0 > Using threads > 1: 100000 iters at 5174.8 ns/switch > 2: 100000 iters at 5140.5 ns/switch > 3: 100000 iters at 5292.9 ns/switch > > ./context_switch_latency 0 process same > use_xstate = 0 > Using a subprocess > 1: 100000 iters at 2361.2 ns/switch > 2: 100000 iters at 2332.2 ns/switch > 3: 100000 iters at 3436.9 ns/switch > > ./context_switch_latency 0 process different > use_xstate = 0 > Using a subprocess > 1: 100000 iters at 4713.6 ns/switch > 2: 100000 iters at 4957.5 ns/switch > 3: 100000 iters at 5012.2 ns/switch > > ./context_switch_latency 1 thread same > use_xstate = 1 > Using threads > 1: 100000 iters at 2505.6 ns/switch > 2: 100000 iters at 2483.1 ns/switch > 3: 100000 iters at 2479.7 ns/switch > > ./context_switch_latency 1 thread different > use_xstate = 1 > Using threads > 1: 100000 iters at 5245.9 ns/switch > 2: 100000 iters at 5241.1 ns/switch > 3: 100000 iters at 5220.3 ns/switch > > ./context_switch_latency 1 process same > use_xstate = 1 > Using a subprocess > 1: 100000 iters at 2329.8 ns/switch > 2: 100000 iters at 2350.2 ns/switch > 3: 100000 iters at 2500.9 ns/switch > > ./context_switch_latency 1 process different > use_xstate = 1 > Using a subprocess > 1: 100000 iters at 4970.7 ns/switch > 2: 100000 iters at 5034.0 ns/switch > 3: 100000 iters at 4991.6 ns/switch > I'll fiddle with that benchmark a little bit. Maybe I can make it suck less. If anyone knows a good non-micro benchmark for this, let me know. I refuse to use dbus as my benchmark :) FWIW, I benchmarked cr4 vs invpcid by adding a prctl and calling it in a loop. If Ingo's fpu benchmark thing ever lands, I'll gladly send a patch to add TLB flushes to it. --Andy