[PATCH 1/2] futex: Use atomic64_inc_return() in get_inode_sequence_number()

[PATCH 1/2] futex: Use atomic64_inc_return() in get_inode_sequence_number()

[Uros Bizjak]

Use atomic64_inc_return(&ref) instead of atomic64_add_return(1, &ref)
to use optimized implementation and ease register pressure around
the primitive for targets that implement optimized variant.

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Darren Hart <dvhart@infradead.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: "André Almeida" <andrealmeid@igalia.com>
---
 kernel/futex/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 136768ae2637..3146730e55f7 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -181,7 +181,7 @@ static u64 get_inode_sequence_number(struct inode *inode)
 		return old;
 
 	for (;;) {
-		u64 new = atomic64_add_return(1, &i_seq);
+		u64 new = atomic64_inc_return(&i_seq);
 		if (WARN_ON_ONCE(!new))
 			continue;
 
-- 
2.46.2

[PATCH 2/2] futex: Use atomic64_try_cmpxchg_relaxed() in get_inode_sequence_number()

[Uros Bizjak]

Optimize get_inode_sequence_number() to use simpler and faster:

  !atomic64_try_cmpxchg_relaxed(*ptr, &old, new)

instead of:

  atomic64_cmpxchg relaxed(*ptr, old, new) != old

The x86 CMPXCHG instruction returns success in ZF flag, so
this change saves a compare after cmpxchg. The generated
code improves from:

 3da:	31 c0                	xor    %eax,%eax
 3dc:	f0 48 0f b1 8a 38 01 	lock cmpxchg %rcx,0x138(%rdx)
 3e3:	00 00
 3e5:	48 85 c0             	test   %rax,%rax
 3e8:	48 0f 44 c1          	cmove  %rcx,%rax

to:

 3da:	31 c0                	xor    %eax,%eax
 3dc:	f0 48 0f b1 8a 38 01 	lock cmpxchg %rcx,0x138(%rdx)
 3e3:	00 00
 3e5:	48 0f 44 c1          	cmove  %rcx,%rax

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Darren Hart <dvhart@infradead.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: "André Almeida" <andrealmeid@igalia.com>
---
 kernel/futex/core.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 3146730e55f7..11795439efb7 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -185,8 +185,9 @@ static u64 get_inode_sequence_number(struct inode *inode)
 		if (WARN_ON_ONCE(!new))
 			continue;
 
-		old = atomic64_cmpxchg_relaxed(&inode->i_sequence, 0, new);
-		if (old)
+		old = 0;
+		if (!atomic64_try_cmpxchg_relaxed(&inode->i_sequence,
+						  &old, new))
 			return old;
 		return new;
 	}
-- 
2.46.2

Re: [PATCH 2/2] futex: Use atomic64_try_cmpxchg_relaxed() in get_inode_sequence_number()

[André Almeida]

Em 10/10/2024 04:10, Uros Bizjak escreveu:
> Optimize get_inode_sequence_number() to use simpler and faster:
> 
>    !atomic64_try_cmpxchg_relaxed(*ptr, &old, new)
> 
> instead of:
> 
>    atomic64_cmpxchg relaxed(*ptr, old, new) != old
> 
> The x86 CMPXCHG instruction returns success in ZF flag, so
> this change saves a compare after cmpxchg. The generated
> code improves from:
> 
>   3da:	31 c0                	xor    %eax,%eax
>   3dc:	f0 48 0f b1 8a 38 01 	lock cmpxchg %rcx,0x138(%rdx)
>   3e3:	00 00
>   3e5:	48 85 c0             	test   %rax,%rax
>   3e8:	48 0f 44 c1          	cmove  %rcx,%rax
> 
> to:
> 
>   3da:	31 c0                	xor    %eax,%eax
>   3dc:	f0 48 0f b1 8a 38 01 	lock cmpxchg %rcx,0x138(%rdx)
>   3e3:	00 00
>   3e5:	48 0f 44 c1          	cmove  %rcx,%rax
> 
> Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Darren Hart <dvhart@infradead.org>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: "André Almeida" <andrealmeid@igalia.com>

Reviewed-by: André Almeida <andrealmeid@igalia.com>

[tip: locking/core] futex: Use atomic64_try_cmpxchg_relaxed() in get_inode_sequence_number()

[tip-bot2 for Uros Bizjak]

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     87347f148061b48c3495fb61dcbad384760da9cf
Gitweb:        https://git.kernel.org/tip/87347f148061b48c3495fb61dcbad384760da9cf
Author:        Uros Bizjak <ubizjak@gmail.com>
AuthorDate:    Thu, 10 Oct 2024 09:10:05 +02:00
Committer:     Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Thu, 17 Oct 2024 22:02:27 +02:00

futex: Use atomic64_try_cmpxchg_relaxed() in get_inode_sequence_number()

Optimize get_inode_sequence_number() to use simpler and faster:

  !atomic64_try_cmpxchg_relaxed(*ptr, &old, new)

instead of:

  atomic64_cmpxchg relaxed(*ptr, old, new) != old

The x86 CMPXCHG instruction returns success in ZF flag, so
this change saves a compare after cmpxchg. The generated
code improves from:

 3da:	31 c0                	xor    %eax,%eax
 3dc:	f0 48 0f b1 8a 38 01 	lock cmpxchg %rcx,0x138(%rdx)
 3e3:	00 00
 3e5:	48 85 c0             	test   %rax,%rax
 3e8:	48 0f 44 c1          	cmove  %rcx,%rax

to:

 3da:	31 c0                	xor    %eax,%eax
 3dc:	f0 48 0f b1 8a 38 01 	lock cmpxchg %rcx,0x138(%rdx)
 3e3:	00 00
 3e5:	48 0f 44 c1          	cmove  %rcx,%rax

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Link: https://lore.kernel.org/all/20241010071023.21913-2-ubizjak@gmail.com

---
 kernel/futex/core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 3146730..692912b 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -185,8 +185,8 @@ static u64 get_inode_sequence_number(struct inode *inode)
 		if (WARN_ON_ONCE(!new))
 			continue;
 
-		old = atomic64_cmpxchg_relaxed(&inode->i_sequence, 0, new);
-		if (old)
+		old = 0;
+		if (!atomic64_try_cmpxchg_relaxed(&inode->i_sequence, &old, new))
 			return old;
 		return new;
 	}

Re: [PATCH 1/2] futex: Use atomic64_inc_return() in get_inode_sequence_number()

[André Almeida]

Em 10/10/2024 04:10, Uros Bizjak escreveu:
> Use atomic64_inc_return(&ref) instead of atomic64_add_return(1, &ref)
> to use optimized implementation and ease register pressure around
> the primitive for targets that implement optimized variant.
> 
> Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Darren Hart <dvhart@infradead.org>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: "André Almeida" <andrealmeid@igalia.com>
> ---

Reviewed-by: André Almeida <andrealmeid@igalia.com>

[tip: locking/core] futex: Use atomic64_inc_return() in get_inode_sequence_number()

[tip-bot2 for Uros Bizjak]

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     19298f48694987fac843261c84e24834c255b451
Gitweb:        https://git.kernel.org/tip/19298f48694987fac843261c84e24834c255b451
Author:        Uros Bizjak <ubizjak@gmail.com>
AuthorDate:    Thu, 10 Oct 2024 09:10:04 +02:00
Committer:     Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Thu, 17 Oct 2024 22:02:27 +02:00

futex: Use atomic64_inc_return() in get_inode_sequence_number()

Use atomic64_inc_return(&ref) instead of atomic64_add_return(1, &ref)
to use optimized implementation and ease register pressure around
the primitive for targets that implement optimized variant.

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: André Almeida <andrealmeid@igalia.com>
Link: https://lore.kernel.org/all/20241010071023.21913-1-ubizjak@gmail.com

---
 kernel/futex/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 136768a..3146730 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -181,7 +181,7 @@ static u64 get_inode_sequence_number(struct inode *inode)
 		return old;
 
 	for (;;) {
-		u64 new = atomic64_add_return(1, &i_seq);
+		u64 new = atomic64_inc_return(&i_seq);
 		if (WARN_ON_ONCE(!new))
 			continue;