[PATCH, resend] x86-64: improve insn scheduling in SAVE_ARGS_IRQ

[PATCH, resend] x86-64: improve insn scheduling in SAVE_ARGS_IRQ

[Jan Beulich]

In one case, use an address register that was computed earlier (and
with a simpler instruction), thus reducing the risk of a stall.

In the second case, eliminate a branch by using a conditional move (as
is already done in call_softirq and xen_do_hypervisor_callback).

Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andi Kleen <ak@linux.intel.com>

---
 arch/x86/kernel/entry_64.S |    7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

--- 3.3-rc4/arch/x86/kernel/entry_64.S
+++ 3.3-rc4-x86_64-entry-sched/arch/x86/kernel/entry_64.S
@@ -320,7 +320,7 @@ ENDPROC(native_usergs_sysret64)
 	movq %rsp, %rsi
 
 	leaq -RBP(%rsp),%rdi	/* arg1 for handler */
-	testl $3, CS(%rdi)
+	testl $3, CS-RBP(%rsi)
 	je 1f
 	SWAPGS
 	/*
@@ -330,11 +330,10 @@ ENDPROC(native_usergs_sysret64)
 	 * moving irq_enter into assembly, which would be too much work)
 	 */
 1:	incl PER_CPU_VAR(irq_count)
-	jne 2f
-	mov PER_CPU_VAR(irq_stack_ptr),%rsp
+	cmovzq PER_CPU_VAR(irq_stack_ptr),%rsp
 	CFI_DEF_CFA_REGISTER	rsi
 
-2:	/* Store previous stack value */
+	/* Store previous stack value */
 	pushq %rsi
 	CFI_ESCAPE	0x0f /* DW_CFA_def_cfa_expression */, 6, \
 			0x77 /* DW_OP_breg7 */, 0, \

[tip:x86/asm] x86-64: Improve insn scheduling in SAVE_ARGS_IRQ

[tip-bot for Jan Beulich]

Commit-ID:  69466466ce889cd2cbc8cda9ff1c6083f48cc7f9
Gitweb:     http://git.kernel.org/tip/69466466ce889cd2cbc8cda9ff1c6083f48cc7f9
Author:     Jan Beulich <JBeulich@suse.com>
AuthorDate: Fri, 24 Feb 2012 11:55:01 +0000
Committer:  H. Peter Anvin <hpa@linux.intel.com>
CommitDate: Fri, 24 Feb 2012 11:46:28 -0800

x86-64: Improve insn scheduling in SAVE_ARGS_IRQ

In one case, use an address register that was computed earlier (and
with a simpler instruction), thus reducing the risk of a stall.

In the second case, eliminate a branch by using a conditional move (as
is already done in call_softirq and xen_do_hypervisor_callback).

Signed-off-by: Jan Beulich <jbeulich@suse.com>
Link: http://lkml.kernel.org/r/4F4788A50200007800074A26@nat28.tlf.novell.com
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
---
 arch/x86/kernel/entry_64.S |    7 +++----
 1 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index a20e1cb..211b2e1 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -319,7 +319,7 @@ ENDPROC(native_usergs_sysret64)
 	movq %rsp, %rsi
 
 	leaq -RBP(%rsp),%rdi	/* arg1 for handler */
-	testl $3, CS(%rdi)
+	testl $3, CS-RBP(%rsi)
 	je 1f
 	SWAPGS
 	/*
@@ -329,11 +329,10 @@ ENDPROC(native_usergs_sysret64)
 	 * moving irq_enter into assembly, which would be too much work)
 	 */
 1:	incl PER_CPU_VAR(irq_count)
-	jne 2f
-	mov PER_CPU_VAR(irq_stack_ptr),%rsp
+	cmovzq PER_CPU_VAR(irq_stack_ptr),%rsp
 	CFI_DEF_CFA_REGISTER	rsi
 
-2:	/* Store previous stack value */
+	/* Store previous stack value */
 	pushq %rsi
 	CFI_ESCAPE	0x0f /* DW_CFA_def_cfa_expression */, 6, \
 			0x77 /* DW_OP_breg7 */, 0, \

Re: [PATCH, resend] x86-64: improve insn scheduling in SAVE_ARGS_IRQ

[H. Peter Anvin]

On 02/24/2012 03:55 AM, Jan Beulich wrote:
>  	leaq -RBP(%rsp),%rdi	/* arg1 for handler */
> -	testl $3, CS(%rdi)
> +	testl $3, CS-RBP(%rsi)
>  	je 1f

IIRC there is no imm8 version of testl; could this be a testb instead or
is that going to cause stalls on some platforms you think?

	-hpa

Re: [PATCH, resend] x86-64: improve insn scheduling in SAVE_ARGS_IRQ

[Jan Beulich]

>>> On 24.02.12 at 21:21, "H. Peter Anvin" <hpa@zytor.com> wrote:
> On 02/24/2012 03:55 AM, Jan Beulich wrote:
>>  	leaq -RBP(%rsp),%rdi	/* arg1 for handler */
>> -	testl $3, CS(%rdi)
>> +	testl $3, CS-RBP(%rsi)
>>  	je 1f
> 
> IIRC there is no imm8 version of testl; could this be a testb instead or
> is that going to cause stalls on some platforms you think?

As far as I'm aware, memory operations don't suffer "partial register
stalls" or alike, so yes, using a testb here ought to be appropriate.

Jan