From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754894AbbGXSDh (ORCPT ); Fri, 24 Jul 2015 14:03:37 -0400 Received: from casper.infradead.org ([85.118.1.10]:46820 "EHLO casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753770AbbGXSBj (ORCPT ); Fri, 24 Jul 2015 14:01:39 -0400 Message-Id: <20150724175823.226513471@infradead.org> User-Agent: quilt/0.61-1 Date: Fri, 24 Jul 2015 19:52:16 +0200 From: Peter Zijlstra To: linux-kernel@vger.kernel.org, mingo@kernel.org Cc: jasonbaron0@gmail.com, bp@alien8.de, luto@amacapital.net, tglx@linutronix.de, rostedt@goodmis.org, will.deacon@arm.com, liuj97@gmail.com, rabin@rab.in, ralf@linux-mips.org, ddaney@caviumnetworks.com, benh@kernel.crashing.org, michael@ellerman.id.au, heiko.carstens@de.ibm.com, davem@davemloft.net, peterz@infradead.org Subject: [RFC][PATCH 7/7] x86, tsc: Employ static_branch_likely() References: <20150724175209.814173117@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline; filename=peterz-static-key-use-new.patch Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Because of the static_key restrictions we had to take an unconditional jump for the most likely case, causing $I bloat. Rewrite to use the new primitives. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/tsc.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -38,7 +38,7 @@ static int __read_mostly tsc_unstable; erroneous rdtsc usage on !cpu_has_tsc processors */ static int __read_mostly tsc_disabled = -1; -static struct static_key __use_tsc = STATIC_KEY_INIT; +static DEFINE_STATIC_KEY_FALSE(__use_tsc); int tsc_clocksource_reliable; @@ -274,7 +274,12 @@ static void set_cyc2ns_scale(unsigned lo */ u64 native_sched_clock(void) { - u64 tsc_now; + if (static_branch_likely(&__use_tsc)) { + u64 tsc_now = rdtsc(); + + /* return the value in ns */ + return cycles_2_ns(tsc_now); + } /* * Fall back to jiffies if there's no TSC available: @@ -284,16 +289,9 @@ u64 native_sched_clock(void) * very important for it to be as fast as the platform * can achieve it. ) */ - if (!static_key_false(&__use_tsc)) { - /* No locking but a rare wrong value is not a big deal: */ - return (jiffies_64 - INITIAL_JIFFIES) * (1000000000 / HZ); - } - - /* read the Time Stamp Counter: */ - tsc_now = rdtsc(); - /* return the value in ns */ - return cycles_2_ns(tsc_now); + /* No locking but a rare wrong value is not a big deal: */ + return (jiffies_64 - INITIAL_JIFFIES) * (1000000000 / HZ); } /* We need to define a real function for sched_clock, to override the @@ -1204,7 +1202,7 @@ void __init tsc_init(void) /* now allow native_sched_clock() to use rdtsc */ tsc_disabled = 0; - static_key_slow_inc(&__use_tsc); + static_branch_enable(&__use_tsc); if (!no_sched_irq_time) enable_sched_clock_irqtime();