[PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[Mathieu Desnoyers]

Compiler CSE and SSA GVN optimizations can cause the address dependency
of addresses returned by rcu_dereference to be lost when comparing those
pointers with either constants or previously loaded pointers.

Introduce ptr_eq() to compare two addresses while preserving the address
dependencies for later use of the address. It should be used when
comparing an address returned by rcu_dereference().

This is needed to prevent the compiler CSE and SSA GVN optimizations
from using @a (or @b) in places where the source refers to @b (or @a)
based on the fact that after the comparison, the two are known to be
equal, which does not preserve address dependencies and allows the
following misordering speculations:

- If @b is a constant, the compiler can issue the loads which depend
  on @a before loading @a.
- If @b is a register populated by a prior load, weakly-ordered
  CPUs can speculate loads which depend on @a before loading @a.

The same logic applies with @a and @b swapped.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Suggested-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Tested-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Acked-by: "Paul E. McKenney" <paulmck@kernel.org>
Acked-by: Alan Stern <stern@rowland.harvard.edu>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Alan Stern <stern@rowland.harvard.edu>
Cc: John Stultz <jstultz@google.com>
Cc: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Uladzislau Rezki <urezki@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Zqiang <qiang.zhang1211@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: maged.michael@gmail.com
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Gary Guo <gary@garyguo.net>
Cc: Jonas Oberhauser <jonas.oberhauser@huaweicloud.com>
Cc: rcu@vger.kernel.org
Cc: linux-mm@kvack.org
Cc: lkmm@lists.linux.dev
Cc: Nikita Popov <github@npopov.com>
Cc: llvm@lists.linux.dev
---
Changes since v0:
- Include feedback from Alan Stern.
---
 include/linux/compiler.h | 63 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index 70cb31d60538..0a8c8e53cdad 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -166,6 +166,69 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 	__PASTE(name,						\
 	__PASTE(_, __COUNTER__)))
 
+/*
+ * Compare two addresses while preserving the address dependencies for
+ * later use of the address. It should be used when comparing an address
+ * returned by rcu_dereference().
+ *
+ * This is needed to prevent the compiler CSE and SSA GVN optimizations
+ * from using @a (or @b) in places where the source refers to @b (or @a)
+ * based on the fact that after the comparison, the two are known to be
+ * equal, which does not preserve address dependencies and allows the
+ * following misordering speculations:
+ *
+ * - If @b is a constant, the compiler can issue the loads which depend
+ *   on @a before loading @a.
+ * - If @b is a register populated by a prior load, weakly-ordered
+ *   CPUs can speculate loads which depend on @a before loading @a.
+ *
+ * The same logic applies with @a and @b swapped.
+ *
+ * Return value: true if pointers are equal, false otherwise.
+ *
+ * The compiler barrier() is ineffective at fixing this issue. It does
+ * not prevent the compiler CSE from losing the address dependency:
+ *
+ * int fct_2_volatile_barriers(void)
+ * {
+ *     int *a, *b;
+ *
+ *     do {
+ *         a = READ_ONCE(p);
+ *         asm volatile ("" : : : "memory");
+ *         b = READ_ONCE(p);
+ *     } while (a != b);
+ *     asm volatile ("" : : : "memory");  <-- barrier()
+ *     return *b;
+ * }
+ *
+ * With gcc 14.2 (arm64):
+ *
+ * fct_2_volatile_barriers:
+ *         adrp    x0, .LANCHOR0
+ *         add     x0, x0, :lo12:.LANCHOR0
+ * .L2:
+ *         ldr     x1, [x0]  <-- x1 populated by first load.
+ *         ldr     x2, [x0]
+ *         cmp     x1, x2
+ *         bne     .L2
+ *         ldr     w0, [x1]  <-- x1 is used for access which should depend on b.
+ *         ret
+ *
+ * On weakly-ordered architectures, this lets CPU speculation use the
+ * result from the first load to speculate "ldr w0, [x1]" before
+ * "ldr x2, [x0]".
+ * Based on the RCU documentation, the control dependency does not
+ * prevent the CPU from speculating loads.
+ */
+static __always_inline
+int ptr_eq(const volatile void *a, const volatile void *b)
+{
+	OPTIMIZER_HIDE_VAR(a);
+	OPTIMIZER_HIDE_VAR(b);
+	return a == b;
+}
+
 /**
  * data_race - mark an expression as containing intentional data races
  *
-- 
2.43.0

Re: [PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[Linus Torvalds]

On Sun, 27 Sept 2026 at 08:51, Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> +static __always_inline
> +int ptr_eq(const volatile void *a, const volatile void *b)
> +{
> +       OPTIMIZER_HIDE_VAR(a);
> +       OPTIMIZER_HIDE_VAR(b);
> +       return a == b;
> +}

I had to think about why I hate this so much.

It's probably entirely senseless, but this just rubs me the wrong way.
Those OPTIMIZER_HIDE_VAR() statements are hiding the wrong thing. They
are telling the compiler to treat valid information (the pointers) as
if they had changed, so then the compiler will have to just reload
those pointers when they are used later (and "used later" is part of
the whole *point* of this, after all).

But it was never the pointer values that needed hiding from the
compiler. It's always just the comparison. And while the "optimal"
thing would be to just do something like

  static __always_inline bool ptr_eq(const volatile void *a, const
volatile void *b)
  {
        bool ret;
        asm("cmpq %0,%1":"=@ccz" (ret):"r" (a),"r"(b))
        return ret;
  }

that is an architecture-specific thing - and then have that
OPTIMIZER_HIDE_VAR() available as a fallback.

I don't know if it's really worth it, but I really do despise that
"hide the wrong values" thing that generates two new values just to
compare them and throw them away, and adds register pressure for no
good reason.

               Linus

Re: [PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[Mathieu Desnoyers]

On 2026-09-29 10:47, Linus Torvalds wrote:
> On Sun, 27 Sept 2026 at 08:51, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>>
>> +static __always_inline
>> +int ptr_eq(const volatile void *a, const volatile void *b)
>> +{
>> +       OPTIMIZER_HIDE_VAR(a);
>> +       OPTIMIZER_HIDE_VAR(b);
>> +       return a == b;
>> +}
> 
> I had to think about why I hate this so much.
> 
> It's probably entirely senseless, but this just rubs me the wrong way.
> Those OPTIMIZER_HIDE_VAR() statements are hiding the wrong thing. They
> are telling the compiler to treat valid information (the pointers) as
> if they had changed, so then the compiler will have to just reload
> those pointers when they are used later (and "used later" is part of
> the whole *point* of this, after all).
> 
> But it was never the pointer values that needed hiding from the
> compiler. It's always just the comparison. And while the "optimal"
> thing would be to just do something like
> 
>    static __always_inline bool ptr_eq(const volatile void *a, const
> volatile void *b)
>    {
>          bool ret;
>          asm("cmpq %0,%1":"=@ccz" (ret):"r" (a),"r"(b))
>          return ret;
>    }
> 
> that is an architecture-specific thing - and then have that
> OPTIMIZER_HIDE_VAR() available as a fallback.
> 
> I don't know if it's really worth it, but I really do despise that
> "hide the wrong values" thing that generates two new values just to
> compare them and throw them away, and adds register pressure for no
> good reason.

I agree that doing it in asm would generate better code. Also, I've
been told that clang now preserves the address dependency in this
scenario, so perhaps we'd want to use preprocessor conditionals to
select how to implement ptr_eq based on:

- architecture (allowing asm implementation overrides),
- compiler (e.g. if compiler is clang >= version X, just do a plain
   comparison). Likewise for gcc if it ever decides to fix this
   behavior.

I think this "hide var" hack is a fallback which can be used as a
starting point, and then we can specialize based on architecture and
compiler.

Or do you prefer this in a different order ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[Linus Torvalds]

On Tue, 29 Sept 2026 at 08:04, Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> I agree that doing it in asm would generate better code. Also, I've
> been told that clang now preserves the address dependency in this
> scenario, so perhaps we'd want to use preprocessor conditionals to
> select how to implement ptr_eq based on:
>
> - architecture (allowing asm implementation overrides),
> - compiler (e.g. if compiler is clang >= version X, just do a plain
>    comparison). Likewise for gcc if it ever decides to fix this
>    behavior.
>
> I think this "hide var" hack is a fallback which can be used as a
> starting point, and then we can specialize based on architecture and
> compiler.
>
> Or do you prefer this in a different order ?

Oh, if there's some sane way to tell that the compiler already honors
address dependencies, then that should be done first and the whole
thing should just become a simple

 (a) == (b)

for that situation - allowing the compiler then the freedom to do
whatever (which can involve not using a register at all, but a compare
to memory, or whatever - the compiler might have reasons to avoid the
'cmp' and use another sequence entirely)

Then a "if we have an architecture fallback".

And then that OPTIMIZER_HIDE_VAR() thing as a last fallback.

               Linus

Re: [PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[Mathieu Desnoyers]

On 2026-09-29 11:16, Linus Torvalds wrote:
> On Tue, 29 Sept 2026 at 08:04, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>>
>> I agree that doing it in asm would generate better code. Also, I've
>> been told that clang now preserves the address dependency in this
>> scenario, so perhaps we'd want to use preprocessor conditionals to
>> select how to implement ptr_eq based on:
>>
>> - architecture (allowing asm implementation overrides),
>> - compiler (e.g. if compiler is clang >= version X, just do a plain
>>     comparison). Likewise for gcc if it ever decides to fix this
>>     behavior.
>>
>> I think this "hide var" hack is a fallback which can be used as a
>> starting point, and then we can specialize based on architecture and
>> compiler.
>>
>> Or do you prefer this in a different order ?
> 
> Oh, if there's some sane way to tell that the compiler already honors
> address dependencies, then that should be done first and the whole
> thing should just become a simple
> 
>   (a) == (b)
> 
> for that situation - allowing the compiler then the freedom to do
> whatever (which can involve not using a register at all, but a compare
> to memory, or whatever - the compiler might have reasons to avoid the
> 'cmp' and use another sequence entirely)

For the records, here is the relevant compiler discussion:

https://github.com/llvm/llvm-project/issues/34577

and it seems like they still have unfixed scenarios,
which may or may not affect the hazptr use-case:

https://github.com/llvm/llvm-project/issues/220206

Considering this, I would not be inclined to jump to the conclusion
that using the compiler compare for hazptr acquire load/test/reload
is safe today without a careful analysis.

> 
> Then a "if we have an architecture fallback".
> 
> And then that OPTIMIZER_HIDE_VAR() thing as a last fallback.

OK,

Thanks!

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [PATCH hazptr 2/4] compiler.h: Introduce ptr_eq() to preserve address dependency

[Gary Guo]

On Tue Sep 29, 2026 at 4:42 PM BST, Mathieu Desnoyers wrote:
> On 2026-09-29 11:16, Linus Torvalds wrote:
>> On Tue, 29 Sept 2026 at 08:04, Mathieu Desnoyers
>> <mathieu.desnoyers@efficios.com> wrote:
>>>
>>> I agree that doing it in asm would generate better code. Also, I've
>>> been told that clang now preserves the address dependency in this
>>> scenario, so perhaps we'd want to use preprocessor conditionals to
>>> select how to implement ptr_eq based on:
>>>
>>> - architecture (allowing asm implementation overrides),
>>> - compiler (e.g. if compiler is clang >= version X, just do a plain
>>>     comparison). Likewise for gcc if it ever decides to fix this
>>>     behavior.
>>>
>>> I think this "hide var" hack is a fallback which can be used as a
>>> starting point, and then we can specialize based on architecture and
>>> compiler.
>>>
>>> Or do you prefer this in a different order ?
>> 
>> Oh, if there's some sane way to tell that the compiler already honors
>> address dependencies, then that should be done first and the whole
>> thing should just become a simple
>> 
>>   (a) == (b)
>> 
>> for that situation - allowing the compiler then the freedom to do
>> whatever (which can involve not using a register at all, but a compare
>> to memory, or whatever - the compiler might have reasons to avoid the
>> 'cmp' and use another sequence entirely)
>
> For the records, here is the relevant compiler discussion:
>
> https://github.com/llvm/llvm-project/issues/34577
>
> and it seems like they still have unfixed scenarios,
> which may or may not affect the hazptr use-case:
>
> https://github.com/llvm/llvm-project/issues/220206
>
> Considering this, I would not be inclined to jump to the conclusion
> that using the compiler compare for hazptr acquire load/test/reload
> is safe today without a careful analysis.

Out of the existing cases I think the only problematic one for hazptr is the
comparison against costant global. Which isn't going to be a problem for users
that don't depend on a fixed adress.

But it might be a problem for the lockdep case.

If we use inline asm for ptr_eq I think there's very little optimization
potentials left for compilers, so I think unconditionally using asm would be
fine.

Best,
Gary

[PATCH hazptr 3/4] Documentation: RCU: Refer to ptr_eq()

[Mathieu Desnoyers]

Refer to ptr_eq() in the rcu_dereference() documentation.

ptr_eq() is a mechanism that preserves address dependencies when
comparing pointers, and should be favored when comparing a pointer
obtained from rcu_dereference() against another pointer.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Alan Stern <stern@rowland.harvard.edu>
Acked-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Alan Stern <stern@rowland.harvard.edu>
Cc: John Stultz <jstultz@google.com>
Cc: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Uladzislau Rezki <urezki@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Zqiang <qiang.zhang1211@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Waiman Long <longman@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: maged.michael@gmail.com
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Gary Guo <gary@garyguo.net>
Cc: Jonas Oberhauser <jonas.oberhauser@huaweicloud.com>
Cc: rcu@vger.kernel.org
Cc: linux-mm@kvack.org
Cc: lkmm@lists.linux.dev
Cc: Nikita Popov <github@npopov.com>
Cc: llvm@lists.linux.dev
---
Changes since v1:
- Include feedback from Paul E. McKenney.

Changes since v0:
- Include feedback from Alan Stern.
---
 Documentation/RCU/rcu_dereference.rst | 38 +++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 5 deletions(-)

diff --git a/Documentation/RCU/rcu_dereference.rst b/Documentation/RCU/rcu_dereference.rst
index 5bc3785ebfc2..85b5f79ebf14 100644
--- a/Documentation/RCU/rcu_dereference.rst
+++ b/Documentation/RCU/rcu_dereference.rst
@@ -104,11 +104,12 @@ readers working properly:
 	after such branches, but can speculate loads, which can again
 	result in misordering bugs.
 
--	Be very careful about comparing pointers obtained from
-	rcu_dereference() against non-NULL values.  As Linus Torvalds
-	explained, if the two pointers are equal, the compiler could
-	substitute the pointer you are comparing against for the pointer
-	obtained from rcu_dereference().  For example::
+-	Use operations that preserve address dependencies (such as
+	"ptr_eq()") to compare pointers obtained from rcu_dereference()
+	against non-NULL pointers. As Linus Torvalds explained, if the
+	two pointers are equal, the compiler could substitute the
+	pointer you are comparing against for the pointer obtained from
+	rcu_dereference().  For example::
 
 		p = rcu_dereference(gp);
 		if (p == &default_struct)
@@ -125,6 +126,29 @@ readers working properly:
 	On ARM and Power hardware, the load from "default_struct.a"
 	can now be speculated, such that it might happen before the
 	rcu_dereference().  This could result in bugs due to misordering.
+	Performing the comparison with "ptr_eq()" ensures the compiler
+	does not perform such transformation.
+
+	If the comparison is against another pointer, the compiler is
+	allowed to use either pointer for the following accesses, which
+	loses the address dependency and allows weakly-ordered
+	architectures such as ARM and PowerPC to speculate the
+	address-dependent load before rcu_dereference().  For example::
+
+		p1 = READ_ONCE(gp);
+		p2 = rcu_dereference(gp);
+		if (p1 == p2)  /* BUGGY!!! */
+			do_default(p2->a);
+
+	The compiler can use p1->a rather than p2->a, destroying the
+	address dependency.  Performing the comparison with "ptr_eq()"
+	ensures the compiler preserves the address dependencies.
+	Corrected code::
+
+		p1 = READ_ONCE(gp);
+		p2 = rcu_dereference(gp);
+		if (ptr_eq(p1, p2))
+			do_default(p2->a);
 
 	However, comparisons are OK in the following cases:
 
@@ -204,6 +228,10 @@ readers working properly:
 		comparison will provide exactly the information that the
 		compiler needs to deduce the value of the pointer.
 
+	When in doubt, use operations that preserve address dependencies
+	(such as "ptr_eq()") to compare pointers obtained from
+	rcu_dereference() against non-NULL pointers.
+
 -	Disable any value-speculation optimizations that your compiler
 	might provide, especially if you are making use of feedback-based
 	optimizations that take data collected from prior runs.  Such
-- 
2.43.0