[PATCH 1/3] rcu: Return early if callback is not specified

[PATCH 1/3] rcu: Return early if callback is not specified

[Uladzislau Rezki (Sony)]

Currently the call_rcu() API does not check whether a callback
pointer is NULL. If NULL is passed, rcu_core() will try to invoke
it, resulting in NULL pointer dereference and a kernel crash.

To prevent this and improve debuggability, this patch adds a check
for NULL and emits a kernel stack trace to help identify a faulty
caller.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
 kernel/rcu/tree.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index e8a4b720d7d2..14d4499c6fc3 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
 	/* Misaligned rcu_head! */
 	WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
 
+	/* Avoid NULL dereference if callback is NULL. */
+	if (WARN_ON_ONCE(!func))
+		return;
+
 	if (debug_rcu_head_queue(head)) {
 		/*
 		 * Probable double call_rcu(), so leak the callback.
-- 
2.39.5

[PATCH 2/3] rcu: Enable rcu_normal_wake_from_gp on small systems

[Uladzislau Rezki (Sony)]

Automatically enable the rcu_normal_wake_from_gp parameter on
systems with a small number of CPUs. The activation threshold
is set to 16 CPUs.

This helps to reduce a latency of normal synchronize_rcu() API
by waking up GP-waiters earlier and decoupling synchronize_rcu()
callers from regular callback handling.

A benchmark running 64 parallel jobs invoking synchronize_rcu()
demonstrates a notable latency reduction with the setting enabled.

Latency distribution (microseconds):

<default>
 0      - 9999   : 1
 10000  - 19999  : 4
 20000  - 29999  : 399
 30000  - 39999  : 3197
 40000  - 49999  : 10428
 50000  - 59999  : 17363
 60000  - 69999  : 15529
 70000  - 79999  : 9287
 80000  - 89999  : 4249
 90000  - 99999  : 1915
 100000 - 109999 : 922
 110000 - 119999 : 390
 120000 - 129999 : 187
 ...
<default>

<rcu_normal_wake_from_gp>
 0      - 9999  : 1
 10000  - 19999 : 234
 20000  - 29999 : 6678
 30000  - 39999 : 33463
 40000  - 49999 : 20669
 50000  - 59999 : 2766
 60000  - 69999 : 183
 ...
<rcu_normal_wake_from_gp>

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
 kernel/rcu/tree.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 14d4499c6fc3..c0e0b38a08dc 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1625,7 +1625,9 @@ static void rcu_sr_put_wait_head(struct llist_node *node)
 	atomic_set_release(&sr_wn->inuse, 0);
 }
 
-/* Disabled by default. */
+/* Enable rcu_normal_wake_from_gp automatically on small systems. */
+#define WAKE_FROM_GP_CPU_THRESHOLD 16
+
 static int rcu_normal_wake_from_gp;
 module_param(rcu_normal_wake_from_gp, int, 0644);
 static struct workqueue_struct *sync_wq;
@@ -4847,6 +4849,9 @@ void __init rcu_init(void)
 	sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0);
 	WARN_ON(!sync_wq);
 
+	if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
+		WRITE_ONCE(rcu_normal_wake_from_gp, 1);
+
 	/* Fill in default value for rcutree.qovld boot parameter. */
 	/* -After- the rcu_node ->lock fields are initialized! */
 	if (qovld < 0)
-- 
2.39.5

Re: [PATCH 2/3] rcu: Enable rcu_normal_wake_from_gp on small systems

[Joel Fernandes]

On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
> Automatically enable the rcu_normal_wake_from_gp parameter on
> systems with a small number of CPUs. The activation threshold
> is set to 16 CPUs.
> 
> This helps to reduce a latency of normal synchronize_rcu() API
> by waking up GP-waiters earlier and decoupling synchronize_rcu()
> callers from regular callback handling.
> 
> A benchmark running 64 parallel jobs invoking synchronize_rcu()
> demonstrates a notable latency reduction with the setting enabled.
> 
> Latency distribution (microseconds):
> 
> <default>
>  0      - 9999   : 1
>  10000  - 19999  : 4
>  20000  - 29999  : 399
>  30000  - 39999  : 3197
>  40000  - 49999  : 10428
>  50000  - 59999  : 17363
>  60000  - 69999  : 15529
>  70000  - 79999  : 9287
>  80000  - 89999  : 4249
>  90000  - 99999  : 1915
>  100000 - 109999 : 922
>  110000 - 119999 : 390
>  120000 - 129999 : 187
>  ...
> <default>
> 
> <rcu_normal_wake_from_gp>
>  0      - 9999  : 1
>  10000  - 19999 : 234
>  20000  - 29999 : 6678
>  30000  - 39999 : 33463
>  40000  - 49999 : 20669
>  50000  - 59999 : 2766
>  60000  - 69999 : 183
>  ...
> <rcu_normal_wake_from_gp>
> 
> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
> ---
>  kernel/rcu/tree.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 14d4499c6fc3..c0e0b38a08dc 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1625,7 +1625,9 @@ static void rcu_sr_put_wait_head(struct llist_node *node)
>  	atomic_set_release(&sr_wn->inuse, 0);
>  }
>  
> -/* Disabled by default. */
> +/* Enable rcu_normal_wake_from_gp automatically on small systems. */
> +#define WAKE_FROM_GP_CPU_THRESHOLD 16
> +
>  static int rcu_normal_wake_from_gp;
>  module_param(rcu_normal_wake_from_gp, int, 0644);
>  static struct workqueue_struct *sync_wq;
> @@ -4847,6 +4849,9 @@ void __init rcu_init(void)
>  	sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0);
>  	WARN_ON(!sync_wq);
>  
> +	if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
> +		WRITE_ONCE(rcu_normal_wake_from_gp, 1);
> +
I think this will get weird if user explictly specifies
rcutree.rcu_normal_wake_from_gp=0 ? Then we're silently overriding the param.
Maybe, initialize it to -1, and then if it was set 0 by user, don't override it.
But otherwise, set it to 1.  Per your third patch, '1' is a default, not a
forced value.

thanks,

 - Joel

Re: [PATCH 2/3] rcu: Enable rcu_normal_wake_from_gp on small systems

[Uladzislau Rezki]

On Tue, Jun 10, 2025 at 02:34:10PM -0400, Joel Fernandes wrote:
> 
> 
> On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
> > Automatically enable the rcu_normal_wake_from_gp parameter on
> > systems with a small number of CPUs. The activation threshold
> > is set to 16 CPUs.
> > 
> > This helps to reduce a latency of normal synchronize_rcu() API
> > by waking up GP-waiters earlier and decoupling synchronize_rcu()
> > callers from regular callback handling.
> > 
> > A benchmark running 64 parallel jobs invoking synchronize_rcu()
> > demonstrates a notable latency reduction with the setting enabled.
> > 
> > Latency distribution (microseconds):
> > 
> > <default>
> >  0      - 9999   : 1
> >  10000  - 19999  : 4
> >  20000  - 29999  : 399
> >  30000  - 39999  : 3197
> >  40000  - 49999  : 10428
> >  50000  - 59999  : 17363
> >  60000  - 69999  : 15529
> >  70000  - 79999  : 9287
> >  80000  - 89999  : 4249
> >  90000  - 99999  : 1915
> >  100000 - 109999 : 922
> >  110000 - 119999 : 390
> >  120000 - 129999 : 187
> >  ...
> > <default>
> > 
> > <rcu_normal_wake_from_gp>
> >  0      - 9999  : 1
> >  10000  - 19999 : 234
> >  20000  - 29999 : 6678
> >  30000  - 39999 : 33463
> >  40000  - 49999 : 20669
> >  50000  - 59999 : 2766
> >  60000  - 69999 : 183
> >  ...
> > <rcu_normal_wake_from_gp>
> > 
> > Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
> > ---
> >  kernel/rcu/tree.c | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 14d4499c6fc3..c0e0b38a08dc 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -1625,7 +1625,9 @@ static void rcu_sr_put_wait_head(struct llist_node *node)
> >  	atomic_set_release(&sr_wn->inuse, 0);
> >  }
> >  
> > -/* Disabled by default. */
> > +/* Enable rcu_normal_wake_from_gp automatically on small systems. */
> > +#define WAKE_FROM_GP_CPU_THRESHOLD 16
> > +
> >  static int rcu_normal_wake_from_gp;
> >  module_param(rcu_normal_wake_from_gp, int, 0644);
> >  static struct workqueue_struct *sync_wq;
> > @@ -4847,6 +4849,9 @@ void __init rcu_init(void)
> >  	sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0);
> >  	WARN_ON(!sync_wq);
> >  
> > +	if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
> > +		WRITE_ONCE(rcu_normal_wake_from_gp, 1);
> > +
> I think this will get weird if user explictly specifies
> rcutree.rcu_normal_wake_from_gp=0 ? Then we're silently overriding the param.
> Maybe, initialize it to -1, and then if it was set 0 by user, don't override it.
> But otherwise, set it to 1.  Per your third patch, '1' is a default, not a
> forced value.
> 
Right. That case should be fixed.

--
Uladzislau Rezki

[PATCH 3/3] Documentation/kernel-parameters: Update rcu_normal_wake_from_gp doc

[Uladzislau Rezki (Sony)]

Update the documentation about rcu_normal_wake_from_gp parameter.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index f1f2c0874da9..a2c47bdf75cf 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5485,7 +5485,7 @@
 			echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp
 			or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1"
 
-			Default is 0.
+			Default is 1 if num_possible_cpus() <= 16 otherwise 0.
 
 	rcuscale.gp_async= [KNL]
 			Measure performance of asynchronous
-- 
2.39.5

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Joel Fernandes]

On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
> Currently the call_rcu() API does not check whether a callback
> pointer is NULL. If NULL is passed, rcu_core() will try to invoke
> it, resulting in NULL pointer dereference and a kernel crash.
> 
> To prevent this and improve debuggability, this patch adds a check
> for NULL and emits a kernel stack trace to help identify a faulty
> caller.
> 
> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>

Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>

I will add this first one (only this one since we're discussing the others) to a
new rcu/fixes-for-6.16 branch, but let me know if any objections.

Will push that branch out during -rc2 or -rc3 after sufficient testing.

thanks,

 - Joel

> ---
>  kernel/rcu/tree.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index e8a4b720d7d2..14d4499c6fc3 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
>  	/* Misaligned rcu_head! */
>  	WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
>  
> +	/* Avoid NULL dereference if callback is NULL. */
> +	if (WARN_ON_ONCE(!func))
> +		return;
> +
>  	if (debug_rcu_head_queue(head)) {
>  		/*
>  		 * Probable double call_rcu(), so leak the callback.

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Uladzislau Rezki]

On Tue, Jun 10, 2025 at 03:33:32PM -0400, Joel Fernandes wrote:
> 
> 
> On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
> > Currently the call_rcu() API does not check whether a callback
> > pointer is NULL. If NULL is passed, rcu_core() will try to invoke
> > it, resulting in NULL pointer dereference and a kernel crash.
> > 
> > To prevent this and improve debuggability, this patch adds a check
> > for NULL and emits a kernel stack trace to help identify a faulty
> > caller.
> > 
> > Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
> 
> Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
> 
> I will add this first one (only this one since we're discussing the others) to a
> new rcu/fixes-for-6.16 branch, but let me know if any objections.
> 
> Will push that branch out during -rc2 or -rc3 after sufficient testing.
> 
Yep, that sounds good to me about rc-2/3 releases.

--
Uladzislau Rezki

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Boqun Feng]

On Tue, Jun 10, 2025, at 12:33 PM, Joel Fernandes wrote:
> On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
>> Currently the call_rcu() API does not check whether a callback
>> pointer is NULL. If NULL is passed, rcu_core() will try to invoke
>> it, resulting in NULL pointer dereference and a kernel crash.
>> 
>> To prevent this and improve debuggability, this patch adds a check
>> for NULL and emits a kernel stack trace to help identify a faulty
>> caller.
>> 
>> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
>
> Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
>

Reviewed-by: Boqun Feng <boqun.feng@gmail.com>

> I will add this first one (only this one since we're discussing the others) to a
> new rcu/fixes-for-6.16 branch, but let me know if any objections.
>

Not sure it’s urgent enough given the current evidence.

Regards,
Boqun

> Will push that branch out during -rc2 or -rc3 after sufficient testing.
>
> thanks,
>
>  - Joel
>
>> ---
>>  kernel/rcu/tree.c | 4 ++++
>>  1 file changed, 4 insertions(+)
>> 
>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>> index e8a4b720d7d2..14d4499c6fc3 100644
>> --- a/kernel/rcu/tree.c
>> +++ b/kernel/rcu/tree.c
>> @@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
>>  	/* Misaligned rcu_head! */
>>  	WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
>>  
>> +	/* Avoid NULL dereference if callback is NULL. */
>> +	if (WARN_ON_ONCE(!func))
>> +		return;
>> +
>>  	if (debug_rcu_head_queue(head)) {
>>  		/*
>>  		 * Probable double call_rcu(), so leak the callback.

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Boqun Feng]

On Thu, Jun 12, 2025, at 10:30 AM, Boqun Feng wrote:
> On Tue, Jun 10, 2025, at 12:33 PM, Joel Fernandes wrote:
>> On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
>>> Currently the call_rcu() API does not check whether a callback
>>> pointer is NULL. If NULL is passed, rcu_core() will try to invoke
>>> it, resulting in NULL pointer dereference and a kernel crash.
>>> 
>>> To prevent this and improve debuggability, this patch adds a check
>>> for NULL and emits a kernel stack trace to help identify a faulty
>>> caller.
>>> 
>>> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
>>
>> Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
>>
>
> Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
>

(Accidentally sent from another email account, FWIW,
this is Boqun Feng and I approve this message, sorry
couldn’t resist :))

Regards,
Boqun

>> I will add this first one (only this one since we're discussing the others) to a
>> new rcu/fixes-for-6.16 branch, but let me know if any objections.
>>
>
> Not sure it’s urgent enough given the current evidence.
>
> Regards,
> Boqun
>
>> Will push that branch out during -rc2 or -rc3 after sufficient testing.
>>
>> thanks,
>>
>>  - Joel
>>
>>> ---
>>>  kernel/rcu/tree.c | 4 ++++
>>>  1 file changed, 4 insertions(+)
>>> 
>>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>>> index e8a4b720d7d2..14d4499c6fc3 100644
>>> --- a/kernel/rcu/tree.c
>>> +++ b/kernel/rcu/tree.c
>>> @@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
>>>  	/* Misaligned rcu_head! */
>>>  	WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
>>>  
>>> +	/* Avoid NULL dereference if callback is NULL. */
>>> +	if (WARN_ON_ONCE(!func))
>>> +		return;
>>> +
>>>  	if (debug_rcu_head_queue(head)) {
>>>  		/*
>>>  		 * Probable double call_rcu(), so leak the callback.

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Uladzislau Rezki]

On Thu, Jun 12, 2025 at 10:30:38AM -0700, Boqun Feng wrote:
> 
> 
> On Tue, Jun 10, 2025, at 12:33 PM, Joel Fernandes wrote:
> > On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
> >> Currently the call_rcu() API does not check whether a callback
> >> pointer is NULL. If NULL is passed, rcu_core() will try to invoke
> >> it, resulting in NULL pointer dereference and a kernel crash.
> >> 
> >> To prevent this and improve debuggability, this patch adds a check
> >> for NULL and emits a kernel stack trace to help identify a faulty
> >> caller.
> >> 
> >> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
> >
> > Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
> >
> 
> Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
> 
Thank you for review, Boqun!

> > I will add this first one (only this one since we're discussing the others) to a
> > new rcu/fixes-for-6.16 branch, but let me know if any objections.
> >
> 
> Not sure it’s urgent enough given the current evidence.
> 
Let me clarify it a bit. My point is that, we get a kernel crash in a
subsystem we are responsible for, i.e. no matter if there are faulty
users of it(third party applications), the point is users can crash it.

The kernel robot reports it and it is already a strong indication that
the subsystem is not hardened against invalid inputs:

"BUG: unable to handle kernel NULL pointer dereference in rcu_core (3)"

so this in the rcu_core() which is part of RCU.

But, anyway Joel should decide. I shared my opinion :)

--
Uladzislau Rezki

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Boqun Feng]

On Thu, Jun 12, 2025 at 07:46:12PM +0200, Uladzislau Rezki wrote:
> On Thu, Jun 12, 2025 at 10:30:38AM -0700, Boqun Feng wrote:
> > 
> > 
> > On Tue, Jun 10, 2025, at 12:33 PM, Joel Fernandes wrote:
> > > On 6/10/2025 1:34 PM, Uladzislau Rezki (Sony) wrote:
> > >> Currently the call_rcu() API does not check whether a callback
> > >> pointer is NULL. If NULL is passed, rcu_core() will try to invoke
> > >> it, resulting in NULL pointer dereference and a kernel crash.
> > >> 
> > >> To prevent this and improve debuggability, this patch adds a check
> > >> for NULL and emits a kernel stack trace to help identify a faulty
> > >> caller.
> > >> 
> > >> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
> > >
> > > Reviewed-by: Joel Fernandes <joelagnelf@nvidia.com>
> > >
> > 
> > Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
> > 
> Thank you for review, Boqun!
> 
> > > I will add this first one (only this one since we're discussing the others) to a
> > > new rcu/fixes-for-6.16 branch, but let me know if any objections.
> > >
> > 
> > Not sure it´s urgent enough given the current evidence.
> > 
> Let me clarify it a bit. My point is that, we get a kernel crash in a
> subsystem we are responsible for, i.e. no matter if there are faulty
> users of it(third party applications), the point is users can crash it.
> 

Fair enough.

> The kernel robot reports it and it is already a strong indication that
> the subsystem is not hardened against invalid inputs:
> 
> "BUG: unable to handle kernel NULL pointer dereference in rcu_core (3)"
> 
> so this in the rcu_core() which is part of RCU.
> 
> But, anyway Joel should decide. I shared my opinion :)
> 

Of course, my point is that the urgency is not high enough so we have to
put it in rcu/fixes, but it's a fix, and if Joel had the time to do
it, feel free. Joel's decision.

Regards,
Boqun

> --
> Uladzislau Rezki

Re: [PATCH 1/3] rcu: Return early if callback is not specified

[Joel Fernandes]

On 6/12/2025 5:03 PM, Boqun Feng wrote:
[..]
> 
>> The kernel robot reports it and it is already a strong indication that
>> the subsystem is not hardened against invalid inputs:
>>
>> "BUG: unable to handle kernel NULL pointer dereference in rcu_core (3)"
>>
>> so this in the rcu_core() which is part of RCU.
>>
>> But, anyway Joel should decide. I shared my opinion :)
>>
> 
> Of course, my point is that the urgency is not high enough so we have to
> put it in rcu/fixes, but it's a fix, and if Joel had the time to do
> it, feel free. Joel's decision.
> 
Yeah I feel Vlad's fix for a crash is important so I'll send this up to Linus
for 6.16 after some testing this week. For the other 2 patches, since that is
triggered by a trace point, I'll just let Neeraj take them for 6.17.

thanks,

 - Joel