Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Paul E. McKenney]

On Sat, Apr 18, 2015 at 04:32:38PM +0200, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Sat, Apr 18, 2015 at 03:03:41PM +0200, Ingo Molnar wrote:
> > > 
> > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > > > Hello, Ingo,
> > > > 
> > > > This series contains a single change that fixes Kconfig asking pointless
> > > > questions (https://lkml.org/lkml/2015/4/14/616).  This is an RFC pull
> > > > because there has not yet been a -next build for April 16th.  If you
> > > > would prefer to wait until after -next has pulled this, please let me
> > > > know and I will redo this pull request after that has happened.
> > > > 
> > > > In the meantime, this change is available in the git repository at:
> > > > 
> > > >   git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git for-mingo
> > > > 
> > > > for you to fetch changes up to 8d7dc9283f399e1fda4e48a1c453f689326d9396:
> > > > 
> > > >   rcu: Control grace-period delays directly from value (2015-04-14 19:33:59 -0700)
> > > > 
> > > > ----------------------------------------------------------------
> > > > Paul E. McKenney (1):
> > > >       rcu: Control grace-period delays directly from value
> > > > 
> > > >  kernel/rcu/tree.c | 16 +++++++++-------
> > > >  lib/Kconfig.debug |  1 +
> > > >  2 files changed, 10 insertions(+), 7 deletions(-)
> > > 
> > > Pulled, thanks a lot Paul!
> > > 
> > > Note, while this fixes Linus's immediate complaint that arose from the 
> > > new option, I still think we need to do more fixes in this area.
> > 
> > Good point!
> > 
> > > To demonstrate the current situation I tried the following experiment, 
> > > I did a 'make defconfig' on an x86 box and then took the .config and 
> > > deleted all 'RCU Subsystem' options not marked as debugging.
> > > 
> > > Then I did a 'make oldconfig' to see what kinds of questions a user is 
> > > facing when trying to configure RCU:
> > > 
> > > 	*
> > > 	* Restart config...
> > > 	*
> > > 	*
> > > 	* RCU Subsystem
> > > 	*
> > > 	RCU Implementation
> > > 	> 1. Tree-based hierarchical RCU (TREE_RCU) (NEW)
> > > 	choice[1]: 1
> > 
> > Hmmm...  Given that there is no choice, I agree that it is a bit silly
> > to ask...
> 
> To clarify: this doesn't actually ask - it gets skipped by the kconfig 
> tool. All the rest is an interactive prompt.

Ah, good point!

> > > 	Task_based RCU implementation using voluntary context switch (TASKS_RCU) [N/y/?] (NEW) 
> > 
> > Agreed, this one should be driven directly off of CONFIG_RCU_TORTURE_TEST
> > and the tracing use case.
> 
> Yeah.

OK, will do.

> > > 	Consider userspace as in RCU extended quiescent state (RCU_USER_QS) [N/y/?] (NEW) 
> > 
> > This should be driven directly off of CONFIG_NO_HZ_FULL, unless 
> > Frederic knows something I don't.
> 
> Yes.

Then unless Frederic objects...  ;-)

> > > 	Tree-based hierarchical RCU fanout value (RCU_FANOUT) [64] (NEW) 
> > 
> > Hmmm...  I could drop/obscure this one in favor of a boot parameter.
> 
> Well, what I think might be even bette to make it scale based on 
> CONFIG_NR_CPUS. Distros already actively manage the 'maximum number of 
> CPUs we support', so relying on that value makes sense.
> 
> So if someone sets CONFIG_NR_CPUS to 1024, it gets scaled accordingly. 
> If CONFIG_NR_CPUS is set to 2, it gets scaled to a minimal config. 
> Note that this would excercise and test the affected codepaths better 
> as well, as we'd get different size setups.
> 
> As for the boot option to override it: what would be the usecase for 
> that?

Well, in normal circumstances, it should be 64 for 64-bit systems and
32 for 32-bit systems, regardless of number of CPUs.  But if you had
an odd-sized multisocket system with extremely high socket-to-socket
memory latencies, you might want to select a different value.  For
a silly example, suppose your system had 27 hardware threads per socket.
Then you might want to set both RCU_FANOUT_LEAF and RCU_FANOUT to 27.

Or use a boot parameter to do so, as can be done today for RCU_FANOUT_LEAF.

@@@

> > > 	Tree-based hierarchical RCU leaf-level fanout value (RCU_FANOUT_LEAF) [16] (NEW) 
> > 
> > Ditto -- though large configurations really do set this to 64 in 
> > combination with the skew_tick boot parameter.  Maybe we need to 
> > drive these off of some large-system parameter, like CONFIG_MAX_SMP.
> 
> Or rather CONFIG_NR_CPUS. CONFIG_MAX_SMP is really a debugging thing, 
> to configure the system to the silliest high settings that doesn't 
> outright crash - but it doesn't make much sense otherwise.

Except that setting RCU_FANOUT_LEAF to 64 without also booting with
skew_tick=1 is a really bad idea, as the synchronized scheduling-clock
interrupts will cause ugly levels of lock contention on the rcu_node
->lock.  :-(

But perhaps making the default value of sched_skew_tick be 1 if
RCU_FANOUT_LEAF is greater than 16 is the right solution.

> > > 	Disable tree-based hierarchical RCU auto-balancing (RCU_FANOUT_EXACT) [N/y/?] (NEW) 
> > 
> > I should just make this a boot parameter.  Absolutely no reason for 
> > it to be a Kconfig parameter.
> 
> Again I'd size this to NR_CPUS - and for the boot parameter, I'd think 
> about actual usecases.

The intended use case is related to the odd-sized systems mentioned
for RCU_FANOUT.  By default, we spread CPUs across the leaf-level
rcu_node structures to reduce lock contention, via RCU_FANOUT_EXACT=n.
Systems with high remote memory latencies might want RCU_FANOUT_EXACT=y
to have full control of the geometry.

Maybe I should just eliminate this choice, forcing the current default.

> > > 	Accelerate last non-dyntick-idle CPU's grace periods (RCU_FAST_NO_HZ) [N/y/?] (NEW) 
> > 
> > On this one, I have no idea.  Its purpose is energy efficiency, but 
> > it does have some downsides, for example, increasing idle entry/exit 
> > latency. I am a bit nervous about having it be a boot parameter 
> > because that would leave an extra compare-branch in the path.  This 
> > one will require some thought.
> 
> Keeping this one configurable, with a good default and a good 
> explanation makes sense. There's a lot of 
> 
> > > 	Real-time priority to use for RCU worker threads (RCU_KTHREAD_PRIO) [0] (NEW) 
> > 
> > Indeed, Linus complained about this one.  ;-)
> 
> :-) Yes, it's an essentially unanswerable question.
> 
> > This Kconfig parameter is a stopgap, and needs a real solution.  
> > People with crazy-heavy workloads involving realtime cannot live 
> > without it, but that means that most people don't have to care.  I 
> > have had solving this on my list, and this clearly increases its 
> > priority.
> 
> So what value do they use, prio 99? 98? It might be better to offer 
> this option as a binary choice, and set a given priority. If -rt 
> people complain then they might help us in solving it properly.

I honestly do not remember what priority they were using, it is
not in email, and I don't keep IRC logs that far back.  Adding
linux-rt-users@vger.kernel.org on CC.

> > > 	Offload RCU callback processing from boot-selected CPUs (RCU_NOCB_CPU) [N/y/?] (NEW) 
> > 
> > Hmmm...  Maybe a boot parameter, but I thought that there was some 
> > reason that this was problematic.  I will have to take another look.
> > 
> > Anyway, this one is important to non-NO_HZ_FULL real-time workloads. 
> > In a -rt kernel, making CONFIG_PREEMPT_RT (or whatever it is these 
> > days) drive this one makes a lot of sense.
> 
> Ok.

But in the meantime, it looks like making non-default settings depend on
RCU_EXPERT it the right thing to do.

> > > 	#
> > > 	# configuration written to .config
> > > 	#
> > > 
> > > Only TREE_RCU is available on defconfig, so all the other options 
> > > marked with '(NEW)' were offered as an interactive prompt.
> > > 
> > > I don't think that any of the 8 interactive options (!) here are 
> > > particularly useful to even advanced users who configure kernels, and 
> > > I don't think they should be offered under non-expert settings.
> > 
> > Would it make sense to have a CONFIG_RCU_EXPERT setting to hide the 
> > remaining settings?  That would reduce the common-case number of 
> > questions to one, which would be a quick and safe improvement. 
> > Especially when combined with the changes I called out above.
> 
> Yes, that's absolutely sensible - although I'd also do the 
> CONFIG_NR_CPUS based auto-scaling if it's not set, to make sure 
> distros don't end up tuning this (inevitably imperfectly) which won't 
> flow back upstream:
> 
> That's the other main problem with widely tunable, numeric settings, 
> beyond their user hostility: if they are wrong and are corrected in a 
> distro they don't flow back to upstream, so they are dead end 
> mechanisms as far as code quality and good defaults are concerned.

OK, I will put the surviving options under CONFIG_RCU_EXPERT, and I will
check around to see if I can find any cases of distros setting them to
non-default values.

							Thanx, Paul

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

On Sat, 18 Apr 2015 19:05:42 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > 	Real-time priority to use for RCU worker threads (RCU_KTHREAD_PRIO) [0] (NEW) 
> > > 
> > > Indeed, Linus complained about this one.  ;-)
> > 
> > :-) Yes, it's an essentially unanswerable question.
> > 
> > > This Kconfig parameter is a stopgap, and needs a real solution.  
> > > People with crazy-heavy workloads involving realtime cannot live 
> > > without it, but that means that most people don't have to care.  I 
> > > have had solving this on my list, and this clearly increases its 
> > > priority.
> > 
> > So what value do they use, prio 99? 98? It might be better to offer 
> > this option as a binary choice, and set a given priority. If -rt 
> > people complain then they might help us in solving it properly.
> 
> I honestly do not remember what priority they were using, it is
> not in email, and I don't keep IRC logs that far back.  Adding
> linux-rt-users@vger.kernel.org on CC.

As I recall, we started out using fifo:1, but when you get heavy
workloads running at higher fifo priorities, we wanted to boost the rcu
worker threads over those workloads. 

Currently the irq threads default to fifo:50, so maybe a good
default choice for the rcu threads on RT is fifo:49. That of course
presumes rational behavior on the part of application developers. 

I seem to recall that you and I had a discussion about making this
value a runtime knob in /sys but that didn't go anywhere. Do we need to
crank that up again and just use the config as a default/starting
value? If so then we could just default to fifo:1 and let sysadmins
tweak the value to match up with the workload. 

Clark

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Paul E. McKenney]

On Mon, Apr 20, 2015 at 11:35:54AM -0500, Clark Williams wrote:
> On Sat, 18 Apr 2015 19:05:42 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > > 	Real-time priority to use for RCU worker threads (RCU_KTHREAD_PRIO) [0] (NEW) 
> > > > 
> > > > Indeed, Linus complained about this one.  ;-)
> > > 
> > > :-) Yes, it's an essentially unanswerable question.
> > > 
> > > > This Kconfig parameter is a stopgap, and needs a real solution.  
> > > > People with crazy-heavy workloads involving realtime cannot live 
> > > > without it, but that means that most people don't have to care.  I 
> > > > have had solving this on my list, and this clearly increases its 
> > > > priority.
> > > 
> > > So what value do they use, prio 99? 98? It might be better to offer 
> > > this option as a binary choice, and set a given priority. If -rt 
> > > people complain then they might help us in solving it properly.
> > 
> > I honestly do not remember what priority they were using, it is
> > not in email, and I don't keep IRC logs that far back.  Adding
> > linux-rt-users@vger.kernel.org on CC.
> 
> As I recall, we started out using fifo:1, but when you get heavy
> workloads running at higher fifo priorities, we wanted to boost the rcu
> worker threads over those workloads. 
> 
> Currently the irq threads default to fifo:50, so maybe a good
> default choice for the rcu threads on RT is fifo:49. That of course
> presumes rational behavior on the part of application developers. 
> 
> I seem to recall that you and I had a discussion about making this
> value a runtime knob in /sys but that didn't go anywhere. Do we need to
> crank that up again and just use the config as a default/starting
> value? If so then we could just default to fifo:1 and let sysadmins
> tweak the value to match up with the workload. 

The sysfs knob might be nice, but as far as I know nobody has been
complaining about it.

Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
So how about if the Kconfig parameter selects either SCHED_OTHER
(the default) or SCHED_FIFO:1, and then the boot parameter can be used
to select other values.

That said, if the lack of a sysfs knob has been causing real problems,
let's make that happen.

							Thanx, Paul

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

On Mon, 20 Apr 2015 10:09:03 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Mon, Apr 20, 2015 at 11:35:54AM -0500, Clark Williams wrote:
> > On Sat, 18 Apr 2015 19:05:42 -0700
> > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > > > 	Real-time priority to use for RCU worker threads (RCU_KTHREAD_PRIO) [0] (NEW) 
> > > > > 
> > > > > Indeed, Linus complained about this one.  ;-)
> > > > 
> > > > :-) Yes, it's an essentially unanswerable question.
> > > > 
> > > > > This Kconfig parameter is a stopgap, and needs a real solution.  
> > > > > People with crazy-heavy workloads involving realtime cannot live 
> > > > > without it, but that means that most people don't have to care.  I 
> > > > > have had solving this on my list, and this clearly increases its 
> > > > > priority.
> > > > 
> > > > So what value do they use, prio 99? 98? It might be better to offer 
> > > > this option as a binary choice, and set a given priority. If -rt 
> > > > people complain then they might help us in solving it properly.
> > > 
> > > I honestly do not remember what priority they were using, it is
> > > not in email, and I don't keep IRC logs that far back.  Adding
> > > linux-rt-users@vger.kernel.org on CC.
> > 
> > As I recall, we started out using fifo:1, but when you get heavy
> > workloads running at higher fifo priorities, we wanted to boost the rcu
> > worker threads over those workloads. 
> > 
> > Currently the irq threads default to fifo:50, so maybe a good
> > default choice for the rcu threads on RT is fifo:49. That of course
> > presumes rational behavior on the part of application developers. 
> > 
> > I seem to recall that you and I had a discussion about making this
> > value a runtime knob in /sys but that didn't go anywhere. Do we need to
> > crank that up again and just use the config as a default/starting
> > value? If so then we could just default to fifo:1 and let sysadmins
> > tweak the value to match up with the workload. 
> 
> The sysfs knob might be nice, but as far as I know nobody has been
> complaining about it.
> 
> Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> So how about if the Kconfig parameter selects either SCHED_OTHER
> (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> to select other values.

Yeah, that will work. 

> 
> That said, if the lack of a sysfs knob has been causing real problems,
> let's make that happen.

I'll talk to the other RT-ers and get back to you on that. I suspect
most folks would like it just to not have to reboot while tuning, but
not sure it's worth the extra code. 

Clark

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Daniel Bristot de Oliveira]

On 04/20/2015 02:59 PM, Clark Williams wrote:
>> > That said, if the lack of a sysfs knob has been causing real problems,
>> > let's make that happen.
> I'll talk to the other RT-ers and get back to you on that. I suspect
> most folks would like it just to not have to reboot while tuning, but
> not sure it's worth the extra code. 

I agree that users do not like to reboot the system while tuning
policy:priority. Thus, be able to adjust it without reboot is a good option.

Daniel

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> 
> The sysfs knob might be nice, but as far as I know nobody has been
> complaining about it.
> 
> Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> So how about if the Kconfig parameter selects either SCHED_OTHER
> (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> to select other values.
> 
> That said, if the lack of a sysfs knob has been causing real problems,
> let's make that happen.

But then it's too late, because the time of something getting into the kernel
to the time people can use it can be months if not years.

I see no harm in adding one. Pretty much every kernel parameter I added for
ftrace, has a sysctrl knob for it. (Not a sysfs knob, but a /proc/sys/kernel
knob which is different).

-- Steve

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> > 
> > The sysfs knob might be nice, but as far as I know nobody has been
> > complaining about it.
> > 
> > Besides, we already have the rcutree.kthread_prio= kernel-boot 
> > parameter. So how about if the Kconfig parameter selects either 
> > SCHED_OTHER (the default) or SCHED_FIFO:1, and then the boot 
> > parameter can be used to select other values.
> > 
> > That said, if the lack of a sysfs knob has been causing real 
> > problems, let's make that happen.
> 
> But then it's too late, because the time of something getting into 
> the kernel to the time people can use it can be months if not years.
> 
> I see no harm in adding one. Pretty much every kernel parameter I 
> added for ftrace, has a sysctrl knob for it. (Not a sysfs knob, but 
> a /proc/sys/kernel knob which is different).

So the disadvantage is that if a boot default is wrong, we'll hear 
about it eventually and can fix/improve it.

If a sysctl knob is wrong, people will just 'tune' it and forget to 
propagate it to the kernel proper (why should they).

Which is fine for something like ftrace and other ad-hoc 
instrumentation that is generally very fine tuned to a given bug or 
given piece of hardware, but for something like the RCU implementation 
of the kernel - even if it's just a RT side thought of it - I'm not so 
sure about it.

Thanks,

	Ingo

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Mon, 20 Apr 2015 20:09:04 +0200
Ingo Molnar <mingo@kernel.org> wrote:


> So the disadvantage is that if a boot default is wrong, we'll hear 
> about it eventually and can fix/improve it.
> 
> If a sysctl knob is wrong, people will just 'tune' it and forget to 
> propagate it to the kernel proper (why should they).

My fear is that there is no one true value. One person complains about
it, we change it, then someone else complains about the new value. That
would be even worse.

> 
> Which is fine for something like ftrace and other ad-hoc 
> instrumentation that is generally very fine tuned to a given bug or 
> given piece of hardware, but for something like the RCU implementation 
> of the kernel - even if it's just a RT side thought of it - I'm not so 
> sure about it.

I would argue than every case is different, and only the sysadmin would
know the right value. Thus, just set it to one, and if that's not good
enough, then the sysadmins can change it to their needs.

-- Steve

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Mon, 20 Apr 2015 20:09:04 +0200
> Ingo Molnar <mingo@kernel.org> wrote:
> 
> 
> > So the disadvantage is that if a boot default is wrong, we'll hear 
> > about it eventually and can fix/improve it.
> > 
> > If a sysctl knob is wrong, people will just 'tune' it and forget 
> > to propagate it to the kernel proper (why should they).
> 
> My fear is that there is no one true value. [...]

Do we know that?

> [...] One person complains about it, we change it, then someone else 
> complains about the new value. That would be even worse.

At that point we can still add a sysctl, if valid arguments are 
offered.

> > Which is fine for something like ftrace and other ad-hoc 
> > instrumentation that is generally very fine tuned to a given bug 
> > or given piece of hardware, but for something like the RCU 
> > implementation of the kernel - even if it's just a RT side thought 
> > of it - I'm not so sure about it.
> 
> I would argue than every case is different, and only the sysadmin 
> would know the right value. Thus, just set it to one, and if that's 
> not good enough, then the sysadmins can change it to their needs.

Well, we had really bad experience with sysctls in the past, in 
particular in the VM: with various settings exposed and distros 
'tuning' them - sometimes radically changing the way the system 
worked, confusing everyone involved.

So I'm in general opposed to sysctls for core kernel behavior - except 
for cases where we don't know better.

Instrumentation - especially instrumentation that should have been 
implemented mostly in user-space, like ftrace ;-) - is another special 
case that should stay as flexible as possible via sysctls, obviously.

Thanks,

	Ingo

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Mon, 20 Apr 2015 20:28:32 +0200
Ingo Molnar <mingo@kernel.org> wrote:

> Instrumentation - especially instrumentation that should have been 
> implemented mostly in user-space, like ftrace ;-) - is another special 
> case that should stay as flexible as possible via sysctls, obviously.

I know I used ftrace as an example, but a more appropriate example
would be the sched knobs, as this is more about rcu scheduling than
anything else.

See:

sched_autogroup_enabled      sched_rr_timeslice_ms
sched_child_runs_first       sched_rt_period_us
sched_domain/                sched_rt_runtime_us
sched_latency_ns             sched_shares_window_ns
sched_migration_cost_ns      sched_time_avg_ms
sched_min_granularity_ns     sched_tunable_scaling
sched_nr_migrate             sched_wakeup_granularity_ns

In particular, the sched_rt_* ones.


-- Steve

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Mon, 20 Apr 2015 20:28:32 +0200
> Ingo Molnar <mingo@kernel.org> wrote:
> 
> > Instrumentation - especially instrumentation that should have been 
> > implemented mostly in user-space, like ftrace ;-) - is another special 
> > case that should stay as flexible as possible via sysctls, obviously.
> 
> I know I used ftrace as an example, but a more appropriate example
> would be the sched knobs, as this is more about rcu scheduling than
> anything else.
> 
> See:
> 
> sched_autogroup_enabled      sched_rr_timeslice_ms
> sched_child_runs_first       sched_rt_period_us
> sched_domain/                sched_rt_runtime_us
> sched_latency_ns             sched_shares_window_ns
> sched_migration_cost_ns      sched_time_avg_ms
> sched_min_granularity_ns     sched_tunable_scaling
> sched_nr_migrate             sched_wakeup_granularity_ns

You are comparing apples to oranges.

1)

Many of these are only sysctls if CONFIG_SCHED_DEBUG is enabled, see:

 triton:~/tip> git grep const_debug kernel/sched/*.c
 kernel/sched/core.c:const_debug unsigned int sysctl_sched_features =
 kernel/sched/core.c:const_debug unsigned int sysctl_sched_nr_migrate = 32;
 kernel/sched/core.c:const_debug unsigned int sysctl_sched_time_avg = MSEC_PER_SEC;
 kernel/sched/core.c:const_debug unsigned int sysctl_timer_migration = 1;
 kernel/sched/fair.c:const_debug unsigned int sysctl_sched_migration_cost = 500000UL;

and they turn into 'const' otherwise:

 /*
  * Tunables that become constants when CONFIG_SCHED_DEBUG is off:
  */
 #ifdef CONFIG_SCHED_DEBUG
 # include <linux/static_key.h>
 # define const_debug __read_mostly
 #else
 # define const_debug const
 #endif

2)

A handful of them are simple on/off knobs, such as the 
sched_child_runs_first quirk, or the sched_autogroup_enabled.

3)

There's basically just the three sched_rt_* ones that are 'true' 
tunables (not on/off knobs), mostly because of ABI weakness: 
setscheduler() has no interface for them.

Note that modern scheduler policies, like SCHED_DEADLINE, get all 
their policy parameters from the sched_setparam() user-space ABI, they 
are not driven by sysctls.

So my point stands.

Thanks,

	Ingo

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Tue, 21 Apr 2015 08:42:23 +0200
Ingo Molnar <mingo@kernel.org> wrote:


> Note that modern scheduler policies, like SCHED_DEADLINE, get all 
> their policy parameters from the sched_setparam() user-space ABI, they 
> are not driven by sysctls.

Right, and when I realized that we can do the same with chrt, I
suggested using that. Clark thought that the prio of the rcu threads
always went back to the hard coded value set at compile time. I looked
at the code and didn't see that and then asked Paul about it. Seems
chrt should work. Before my argument was based on not having any way at
run time to change the rcu priorities, which I feel is bad. But if
there is a way, then that should be what we tell users to use.

-- Steve

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Mike Galbraith]

On Mon, 2015-04-20 at 14:21 -0400, Steven Rostedt wrote:
> 
> I would argue than every case is different, and only the sysadmin 
> would
> know the right value. Thus, just set it to one, and if that's not 
> good
> enough, then the sysadmins can change it to their needs.

Agreed.  I don't have it turned on in my -rt kernels, because I don't 
want to force a knight in shining (priority x) armor on users.

        -Mike

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> 
> The sysfs knob might be nice, but as far as I know nobody has been
> complaining about it.
> 
> Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> So how about if the Kconfig parameter selects either SCHED_OTHER
> (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> to select other values.

Hmm, what priority is this for anyway. To change the priority of the boost
value at run time, do we only need to change the priority of the rcub threads?

And the priority of the other rcu threads can change as well with a simple
chrt?

If that's the case, then we don't need a sysctl knob at all.

-- Steve


> 
> That said, if the lack of a sysfs knob has been causing real problems,
> let's make that happen.

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Paul E. McKenney]

On Mon, Apr 20, 2015 at 04:40:49PM -0400, Steven Rostedt wrote:
> On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> > 
> > The sysfs knob might be nice, but as far as I know nobody has been
> > complaining about it.
> > 
> > Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> > So how about if the Kconfig parameter selects either SCHED_OTHER
> > (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> > to select other values.
> 
> Hmm, what priority is this for anyway. To change the priority of the boost
> value at run time, do we only need to change the priority of the rcub threads?
> 
> And the priority of the other rcu threads can change as well with a simple
> chrt?
> 
> If that's the case, then we don't need a sysctl knob at all.

For the grace-period kthreads and the boost kthread, that is the case.
It is also the case for the per-CPU kthreads that invoke RCU callbacks
for the non-offloaded RCU_BOOST configuration (and that replace all
softirq RCU work in -rt).

So, should I just ditch all of the priority-setting within RCU and tell
users to just use chrt?

							Thanx, Paul

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

On Mon, 20 Apr 2015 14:15:04 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Mon, Apr 20, 2015 at 04:40:49PM -0400, Steven Rostedt wrote:
> > On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> > > 
> > > The sysfs knob might be nice, but as far as I know nobody has been
> > > complaining about it.
> > > 
> > > Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> > > So how about if the Kconfig parameter selects either SCHED_OTHER
> > > (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> > > to select other values.
> > 
> > Hmm, what priority is this for anyway. To change the priority of the boost
> > value at run time, do we only need to change the priority of the rcub threads?
> > 
> > And the priority of the other rcu threads can change as well with a simple
> > chrt?
> > 
> > If that's the case, then we don't need a sysctl knob at all.
> 
> For the grace-period kthreads and the boost kthread, that is the case.
> It is also the case for the per-CPU kthreads that invoke RCU callbacks
> for the non-offloaded RCU_BOOST configuration (and that replace all
> softirq RCU work in -rt).
> 
> So, should I just ditch all of the priority-setting within RCU and tell
> users to just use chrt?

Looks to me like all we need to do is tell people if they need a boost
higher than the compiled in default (RCU_KTHREAD_PRIO), then chrt the
priority of the rcub thread to the desired priority. 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Paul E. McKenney]

On Mon, Apr 20, 2015 at 04:50:07PM -0500, Clark Williams wrote:
> On Mon, 20 Apr 2015 14:15:04 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Mon, Apr 20, 2015 at 04:40:49PM -0400, Steven Rostedt wrote:
> > > On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> > > > 
> > > > The sysfs knob might be nice, but as far as I know nobody has been
> > > > complaining about it.
> > > > 
> > > > Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> > > > So how about if the Kconfig parameter selects either SCHED_OTHER
> > > > (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> > > > to select other values.
> > > 
> > > Hmm, what priority is this for anyway. To change the priority of the boost
> > > value at run time, do we only need to change the priority of the rcub threads?
> > > 
> > > And the priority of the other rcu threads can change as well with a simple
> > > chrt?
> > > 
> > > If that's the case, then we don't need a sysctl knob at all.
> > 
> > For the grace-period kthreads and the boost kthread, that is the case.
> > It is also the case for the per-CPU kthreads that invoke RCU callbacks
> > for the non-offloaded RCU_BOOST configuration (and that replace all
> > softirq RCU work in -rt).
> > 
> > So, should I just ditch all of the priority-setting within RCU and tell
> > users to just use chrt?
> 
> Looks to me like all we need to do is tell people if they need a boost
> higher than the compiled in default (RCU_KTHREAD_PRIO), then chrt the
> priority of the rcub thread to the desired priority. 

There's the rub.  They also need to chrt the RCU grace-period kthreads
as well as the per-CPU kthreads (rcuc).  Which is a pain and easy to
get wrong.

So at this point, I am leaning towards keeping RCU_KTHREAD_PRIO, but
hiding it behind RCU_EXPERT.  Someone in an emergency situation can use
chrt to get RCU going, at least assuming that they had the foresight to
leave a prio-99 shell running somewhere and assuming that they do the
chrt before the system hits OOM.  But they have to do all that anyway
if they were to use a sysfs or similar interface.  And it is easy to
tell when you have boosted all the necessary kthreads because RCU
grace periods start advancing once again.  You don't get that feedback
when you set things up at boot time.  ;-)

So again, at least for the moment, I believe that RCU need not provide
a run-time interface for changing RCU kthread priorities, that the
RCU_KTHREAD_PRIO Kconfig parameter should remain, except that it needs
to be hidden behind RCU_EXPERT, and that the rcutree.kthread_prio=
kernel-boot parameter should also remain.

Seem reasonable?

							Thanx, Paul

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Mon, 20 Apr 2015 18:22:58 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Mon, Apr 20, 2015 at 04:50:07PM -0500, Clark Williams wrote:
> > On Mon, 20 Apr 2015 14:15:04 -0700
> > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > 
> > > On Mon, Apr 20, 2015 at 04:40:49PM -0400, Steven Rostedt wrote:
> > > > On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> > > > > 
> > > > > The sysfs knob might be nice, but as far as I know nobody has been
> > > > > complaining about it.
> > > > > 
> > > > > Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> > > > > So how about if the Kconfig parameter selects either SCHED_OTHER
> > > > > (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> > > > > to select other values.
> > > > 
> > > > Hmm, what priority is this for anyway. To change the priority of the boost
> > > > value at run time, do we only need to change the priority of the rcub threads?
> > > > 
> > > > And the priority of the other rcu threads can change as well with a simple
> > > > chrt?
> > > > 
> > > > If that's the case, then we don't need a sysctl knob at all.
> > > 
> > > For the grace-period kthreads and the boost kthread, that is the case.
> > > It is also the case for the per-CPU kthreads that invoke RCU callbacks
> > > for the non-offloaded RCU_BOOST configuration (and that replace all
> > > softirq RCU work in -rt).
> > > 
> > > So, should I just ditch all of the priority-setting within RCU and tell
> > > users to just use chrt?
> > 
> > Looks to me like all we need to do is tell people if they need a boost
> > higher than the compiled in default (RCU_KTHREAD_PRIO), then chrt the
> > priority of the rcub thread to the desired priority. 
> 
> There's the rub.  They also need to chrt the RCU grace-period kthreads
> as well as the per-CPU kthreads (rcuc).  Which is a pain and easy to
> get wrong.
> 
> So at this point, I am leaning towards keeping RCU_KTHREAD_PRIO, but
> hiding it behind RCU_EXPERT.  Someone in an emergency situation can use
> chrt to get RCU going, at least assuming that they had the foresight to
> leave a prio-99 shell running somewhere and assuming that they do the
> chrt before the system hits OOM.  But they have to do all that anyway
> if they were to use a sysfs or similar interface.  And it is easy to
> tell when you have boosted all the necessary kthreads because RCU
> grace periods start advancing once again.  You don't get that feedback
> when you set things up at boot time.  ;-)
> 
> So again, at least for the moment, I believe that RCU need not provide
> a run-time interface for changing RCU kthread priorities, that the
> RCU_KTHREAD_PRIO Kconfig parameter should remain, except that it needs
> to be hidden behind RCU_EXPERT, and that the rcutree.kthread_prio=
> kernel-boot parameter should also remain.
> 
> Seem reasonable?
> 

Does chrt override the kthread_prio at run time? If so, then great.
Otherwise, the sysadmin should still have a way to control their
priorities of kernel threads (with few exceptions like the migration
thread).

-- Steve

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Paul E. McKenney]

On Tue, Apr 21, 2015 at 09:12:32AM -0400, Steven Rostedt wrote:
> On Mon, 20 Apr 2015 18:22:58 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Mon, Apr 20, 2015 at 04:50:07PM -0500, Clark Williams wrote:
> > > On Mon, 20 Apr 2015 14:15:04 -0700
> > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > > > On Mon, Apr 20, 2015 at 04:40:49PM -0400, Steven Rostedt wrote:
> > > > > On Mon, Apr 20, 2015 at 10:09:03AM -0700, Paul E. McKenney wrote:
> > > > > > 
> > > > > > The sysfs knob might be nice, but as far as I know nobody has been
> > > > > > complaining about it.
> > > > > > 
> > > > > > Besides, we already have the rcutree.kthread_prio= kernel-boot parameter.
> > > > > > So how about if the Kconfig parameter selects either SCHED_OTHER
> > > > > > (the default) or SCHED_FIFO:1, and then the boot parameter can be used
> > > > > > to select other values.
> > > > > 
> > > > > Hmm, what priority is this for anyway. To change the priority of the boost
> > > > > value at run time, do we only need to change the priority of the rcub threads?
> > > > > 
> > > > > And the priority of the other rcu threads can change as well with a simple
> > > > > chrt?
> > > > > 
> > > > > If that's the case, then we don't need a sysctl knob at all.
> > > > 
> > > > For the grace-period kthreads and the boost kthread, that is the case.
> > > > It is also the case for the per-CPU kthreads that invoke RCU callbacks
> > > > for the non-offloaded RCU_BOOST configuration (and that replace all
> > > > softirq RCU work in -rt).
> > > > 
> > > > So, should I just ditch all of the priority-setting within RCU and tell
> > > > users to just use chrt?
> > > 
> > > Looks to me like all we need to do is tell people if they need a boost
> > > higher than the compiled in default (RCU_KTHREAD_PRIO), then chrt the
> > > priority of the rcub thread to the desired priority. 
> > 
> > There's the rub.  They also need to chrt the RCU grace-period kthreads
> > as well as the per-CPU kthreads (rcuc).  Which is a pain and easy to
> > get wrong.
> > 
> > So at this point, I am leaning towards keeping RCU_KTHREAD_PRIO, but
> > hiding it behind RCU_EXPERT.  Someone in an emergency situation can use
> > chrt to get RCU going, at least assuming that they had the foresight to
> > leave a prio-99 shell running somewhere and assuming that they do the
> > chrt before the system hits OOM.  But they have to do all that anyway
> > if they were to use a sysfs or similar interface.  And it is easy to
> > tell when you have boosted all the necessary kthreads because RCU
> > grace periods start advancing once again.  You don't get that feedback
> > when you set things up at boot time.  ;-)
> > 
> > So again, at least for the moment, I believe that RCU need not provide
> > a run-time interface for changing RCU kthread priorities, that the
> > RCU_KTHREAD_PRIO Kconfig parameter should remain, except that it needs
> > to be hidden behind RCU_EXPERT, and that the rcutree.kthread_prio=
> > kernel-boot parameter should also remain.
> > 
> > Seem reasonable?
> 
> Does chrt override the kthread_prio at run time? If so, then great.
> Otherwise, the sysadmin should still have a way to control their
> priorities of kernel threads (with few exceptions like the migration
> thread).

Yep, RCU sets the prios only at boot time, so if they are set differently
at runtime, they should stay set differently.  Unless chrt refuses to
work on kthreads or something. ;-)

							Thanx, Paul

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Steven Rostedt]

On Tue, 21 Apr 2015 08:01:22 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> > > Seem reasonable?
> > 
> > Does chrt override the kthread_prio at run time? If so, then great.
> > Otherwise, the sysadmin should still have a way to control their
> > priorities of kernel threads (with few exceptions like the migration
> > thread).
> 
> Yep, RCU sets the prios only at boot time, so if they are set differently
> at runtime, they should stay set differently.  Unless chrt refuses to
> work on kthreads or something. ;-)

Great! This all sounds reasonable to me :-)

-- Steve

Re: [GIT RFC PULL rcu/urgent] Prevent Kconfig from asking pointless questions

[Paul E. McKenney]

On Tue, Apr 21, 2015 at 11:50:27AM -0400, Steven Rostedt wrote:
> On Tue, 21 Apr 2015 08:01:22 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > > > Seem reasonable?
> > > 
> > > Does chrt override the kthread_prio at run time? If so, then great.
> > > Otherwise, the sysadmin should still have a way to control their
> > > priorities of kernel threads (with few exceptions like the migration
> > > thread).
> > 
> > Yep, RCU sets the prios only at boot time, so if they are set differently
> > at runtime, they should stay set differently.  Unless chrt refuses to
> > work on kthreads or something. ;-)
> 
> Great! This all sounds reasonable to me :-)

OK, will proceed as planned, then!

							Thanx, Paul