[RFC PATCH 1/1] rcu: Allow user to ovveride RCU_NOCB_CPU_ALL at boot time

[RFC PATCH 1/1] rcu: Allow user to ovveride RCU_NOCB_CPU_ALL at boot time

[Pranith Kumar]

A kernel built with RCU_NOCB_CPU_ALL build time option will offload callbacks
from all CPUs. The user cannot override this behavior without recompiling the
kernel with the RCU_NOCB_CPU_ALL option turned off. 

This commit allows the user to override the build-time option by using the
rcu_nocbs= boot time option without needing to recompile the kernel.

Please note that this is how NO_HZ_FULL_ALL build time option works and this
commit makes it work similar to that.

Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>
---
 init/Kconfig             | 14 +++++++-------
 kernel/rcu/tree_plugin.h |  8 ++++----
 2 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index 41066e4..7d363a4 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -768,13 +768,13 @@ config RCU_NOCB_CPU_ALL
 	bool "All CPUs are build_forced no-CBs CPUs"
 	depends on RCU_NOCB_CPU
 	help
-	  This option forces all CPUs to be no-CBs CPUs.  The rcu_nocbs=
-	  boot parameter will be ignored.  All CPUs' RCU callbacks will
-	  be executed in the context of per-CPU rcuo kthreads created for
-	  this purpose.  Assuming that the kthreads whose names start with
-	  "rcuo" are bound to "housekeeping" CPUs, this reduces OS jitter
-	  on the remaining CPUs, but might decrease memory locality during
-	  RCU-callback invocation, thus potentially degrading throughput.
+	  If the user doesn't pass the rcu_nocbs= boot option, force all CPUs
+	  to be no-CBs CPUs.  All CPUs' RCU callbacks will be executed in the
+	  context of per-CPU rcuo kthreads created for this purpose. Assuming
+	  that the kthreads whose names start with "rcuo" are bound to
+	  "housekeeping" CPUs, this reduces OS jitter on the remaining CPUs, but
+	  might decrease memory locality during RCU-callback invocation, thus
+	  potentially degrading throughput.
 
 	  Select this if all CPUs need to be no-CBs CPUs for real-time
 	  or energy-efficiency reasons.
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index a2113f6..b97a939 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -92,15 +92,15 @@ static void __init rcu_bootup_announce_oddness(void)
 	if (!have_rcu_nocb_mask) {
 		zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL);
 		have_rcu_nocb_mask = true;
+#ifdef CONFIG_RCU_NOCB_CPU_ALL
+		pr_info("\tOffload RCU callbacks from all CPUs\n");
+		cpumask_copy(rcu_nocb_mask, cpu_possible_mask);
+#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */
 	}
 #ifdef CONFIG_RCU_NOCB_CPU_ZERO
 	pr_info("\tOffload RCU callbacks from CPU 0\n");
 	cpumask_set_cpu(0, rcu_nocb_mask);
 #endif /* #ifdef CONFIG_RCU_NOCB_CPU_ZERO */
-#ifdef CONFIG_RCU_NOCB_CPU_ALL
-	pr_info("\tOffload RCU callbacks from all CPUs\n");
-	cpumask_copy(rcu_nocb_mask, cpu_possible_mask);
-#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */
 #endif /* #ifndef CONFIG_RCU_NOCB_CPU_NONE */
 	if (have_rcu_nocb_mask) {
 		if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) {
-- 
1.9.1

Re: [RFC PATCH 1/1] rcu: Allow user to ovveride RCU_NOCB_CPU_ALL at boot time

[Paul E. McKenney]

On Wed, Jul 16, 2014 at 06:38:08PM -0400, Pranith Kumar wrote:
> A kernel built with RCU_NOCB_CPU_ALL build time option will offload callbacks
> from all CPUs. The user cannot override this behavior without recompiling the
> kernel with the RCU_NOCB_CPU_ALL option turned off. 
> 
> This commit allows the user to override the build-time option by using the
> rcu_nocbs= boot time option without needing to recompile the kernel.
> 
> Please note that this is how NO_HZ_FULL_ALL build time option works and this
> commit makes it work similar to that.
> 
> Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>

I cannot accept this patch.  For one thing, tick_nohz_init_all() looks
a bit on the unconditional side when CONFIG_NO_HZ_FULL_ALL=y.  For
another thing, we really do not want to be handing the user a tool that
allows CPUs that are nohz_full to not be no-CBs CPUs.  For another thing,
if we add this and it turns out to be a bad idea, it will be difficult
to take it back -- someone somewhere will no doubt have scripted the
boot parameter.

							Thanx, Paul

> ---
>  init/Kconfig             | 14 +++++++-------
>  kernel/rcu/tree_plugin.h |  8 ++++----
>  2 files changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/init/Kconfig b/init/Kconfig
> index 41066e4..7d363a4 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -768,13 +768,13 @@ config RCU_NOCB_CPU_ALL
>  	bool "All CPUs are build_forced no-CBs CPUs"
>  	depends on RCU_NOCB_CPU
>  	help
> -	  This option forces all CPUs to be no-CBs CPUs.  The rcu_nocbs=
> -	  boot parameter will be ignored.  All CPUs' RCU callbacks will
> -	  be executed in the context of per-CPU rcuo kthreads created for
> -	  this purpose.  Assuming that the kthreads whose names start with
> -	  "rcuo" are bound to "housekeeping" CPUs, this reduces OS jitter
> -	  on the remaining CPUs, but might decrease memory locality during
> -	  RCU-callback invocation, thus potentially degrading throughput.
> +	  If the user doesn't pass the rcu_nocbs= boot option, force all CPUs
> +	  to be no-CBs CPUs.  All CPUs' RCU callbacks will be executed in the
> +	  context of per-CPU rcuo kthreads created for this purpose. Assuming
> +	  that the kthreads whose names start with "rcuo" are bound to
> +	  "housekeeping" CPUs, this reduces OS jitter on the remaining CPUs, but
> +	  might decrease memory locality during RCU-callback invocation, thus
> +	  potentially degrading throughput.
> 
>  	  Select this if all CPUs need to be no-CBs CPUs for real-time
>  	  or energy-efficiency reasons.
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index a2113f6..b97a939 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -92,15 +92,15 @@ static void __init rcu_bootup_announce_oddness(void)
>  	if (!have_rcu_nocb_mask) {
>  		zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL);
>  		have_rcu_nocb_mask = true;
> +#ifdef CONFIG_RCU_NOCB_CPU_ALL
> +		pr_info("\tOffload RCU callbacks from all CPUs\n");
> +		cpumask_copy(rcu_nocb_mask, cpu_possible_mask);
> +#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */
>  	}
>  #ifdef CONFIG_RCU_NOCB_CPU_ZERO
>  	pr_info("\tOffload RCU callbacks from CPU 0\n");
>  	cpumask_set_cpu(0, rcu_nocb_mask);
>  #endif /* #ifdef CONFIG_RCU_NOCB_CPU_ZERO */
> -#ifdef CONFIG_RCU_NOCB_CPU_ALL
> -	pr_info("\tOffload RCU callbacks from all CPUs\n");
> -	cpumask_copy(rcu_nocb_mask, cpu_possible_mask);
> -#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */
>  #endif /* #ifndef CONFIG_RCU_NOCB_CPU_NONE */
>  	if (have_rcu_nocb_mask) {
>  		if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) {
> -- 
> 1.9.1
> 

Re: [RFC PATCH 1/1] rcu: Allow user to ovveride RCU_NOCB_CPU_ALL at boot time

[Pranith Kumar]

On 07/16/2014 08:26 PM, Paul E. McKenney wrote:
> On Wed, Jul 16, 2014 at 06:38:08PM -0400, Pranith Kumar wrote:
>> A kernel built with RCU_NOCB_CPU_ALL build time option will offload callbacks
>> from all CPUs. The user cannot override this behavior without recompiling the
>> kernel with the RCU_NOCB_CPU_ALL option turned off. 
>>
>> This commit allows the user to override the build-time option by using the
>> rcu_nocbs= boot time option without needing to recompile the kernel.
>>
>> Please note that this is how NO_HZ_FULL_ALL build time option works and this
>> commit makes it work similar to that.
>>
>> Signed-off-by: Pranith Kumar <bobby.prani@gmail.com>

Hi Paul,

> I cannot accept this patch.  For one thing, tick_nohz_init_all() looks
> a bit on the unconditional side when CONFIG_NO_HZ_FULL_ALL=y.  For
> another thing, we really do not want to be handing the user a tool that
> allows CPUs that are nohz_full to not be no-CBs CPUs.  For another thing,

I thought the latest patch does not allow that by ORing the nohz_full and
rcu_nocbs mask. Doesn't it? All nohz_full CPUs will be nocb CPUS. We can mention
this explicitly in the kernel-parameters.txt file.

> if we add this and it turns out to be a bad idea, it will be difficult
> to take it back -- someone somewhere will no doubt have scripted the
> boot parameter.

This option already exists in the kernel when RCU_NOCB_CPU_ALL is not set. The
user can pass in rcu_nocbs= at boot time. I am not sure we are adding anything
new with this patch. 

Most of the distro kernels set RCU_NOCB_CPU_ALL and rightly so, as it is the
suggested and most appropriate option. This patch will make it easier for users
who want to specify nocb CPUs for their needs, without having to recompile the
kernel.

> 
> 							Thanx, Paul
> 
> 

--
Pranith