From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932912AbbJINPR (ORCPT ); Fri, 9 Oct 2015 09:15:17 -0400 Received: from terminus.zytor.com ([198.137.202.10]:46223 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751563AbbJINPP (ORCPT ); Fri, 9 Oct 2015 09:15:15 -0400 Date: Fri, 9 Oct 2015 06:14:24 -0700 From: tip-bot for Andy Lutomirski Message-ID: Cc: peterz@infradead.org, bp@alien8.de, linux-kernel@vger.kernel.org, hpa@zytor.com, tglx@linutronix.de, dvlasenk@redhat.com, brgerst@gmail.com, torvalds@linux-foundation.org, mingo@kernel.org, luto@amacapital.net, luto@kernel.org Reply-To: luto@kernel.org, luto@amacapital.net, mingo@kernel.org, torvalds@linux-foundation.org, brgerst@gmail.com, dvlasenk@redhat.com, tglx@linutronix.de, hpa@zytor.com, linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org In-Reply-To: <8472036ff1f4b426b4c4c3e3d0b3bf5264407c0c.1444091585.git.luto@kernel.org> References: <8472036ff1f4b426b4c4c3e3d0b3bf5264407c0c.1444091585.git.luto@kernel.org> To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/asm] x86/entry: Force inlining of 32-bit syscall code Git-Commit-ID: 33c52129f45e06d9ce23e1a3d50bf9fd6770748b X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 33c52129f45e06d9ce23e1a3d50bf9fd6770748b Gitweb: http://git.kernel.org/tip/33c52129f45e06d9ce23e1a3d50bf9fd6770748b Author: Andy Lutomirski AuthorDate: Mon, 5 Oct 2015 17:48:19 -0700 Committer: Ingo Molnar CommitDate: Fri, 9 Oct 2015 09:41:12 +0200 x86/entry: Force inlining of 32-bit syscall code On systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation. This saves 4-6 cycles. Signed-off-by: Andy Lutomirski Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: linux-kernel@vger.kernel.org Link: http://lkml.kernel.org/r/8472036ff1f4b426b4c4c3e3d0b3bf5264407c0c.1444091585.git.luto@kernel.org Signed-off-by: Ingo Molnar --- arch/x86/entry/common.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 03aacd1..d5eee85 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -324,9 +324,11 @@ __visible void syscall_return_slowpath(struct pt_regs *regs) #if defined(CONFIG_X86_32) || defined(CONFIG_IA32_EMULATION) /* * Does a 32-bit syscall. Called with IRQs on and does all entry and - * exit work and returns with IRQs off. + * exit work and returns with IRQs off. This function is extremely hot + * in workloads that use it, and it's usually called from + * do_fast_syscall_32, so forcibly inline it to improve performance. */ -static void do_syscall_32_irqs_on(struct pt_regs *regs) +static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) { struct thread_info *ti = pt_regs_to_thread_info(regs); unsigned int nr = (unsigned int)regs->orig_ax; @@ -345,7 +347,7 @@ static void do_syscall_32_irqs_on(struct pt_regs *regs) nr = syscall_trace_enter(regs); } - if (nr < IA32_NR_syscalls) { + if (likely(nr < IA32_NR_syscalls)) { /* * It's possible that a 32-bit syscall implementation * takes a 64-bit parameter but nonetheless assumes that