rcutorture: Can not Disable RT throttling

rcutorture: Can not Disable RT throttling

[Z qiang]

When running build-in rcutorture tests in 6.5.0-rc4-rt, and found that,
although the value of /proc/sys/kernel/sched_rt_runtime_us is -1,

cat /sys/kernel/debug/sched/debug
....
rt_rq[6]:
  .rt_nr_running                 : 4
  .rt_nr_migratory               : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

but the rt_runtime still is 950.000000.
set sysctl_sched_rt_runtime in rcu_torture_disable_rt_throttle()
does not disable rt-throttling.

Currently I'm testing this way to disable rt-throttling:

diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 338cd150973f..bb9fee51a476 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -22,7 +22,7 @@ unsigned int sysctl_sched_rt_period = 1000000;
  * part of the period that we allow rt tasks to run in us.
  * default: 0.95s
  */
-int sysctl_sched_rt_runtime = 950000;
+int sysctl_sched_rt_runtime = IS_BUILTIN(CONFIG_RCU_TORTURE_TEST) ?
-1 : 950000;

 #ifdef CONFIG_SYSCTL
 static int sysctl_sched_rr_timeslice = (MSEC_PER_SEC / HZ) * RR_TIMESLICE;


Is there a better way to solve it?

Thanks
Zqiang

Re: rcutorture: Can not Disable RT throttling

[Joel Fernandes]

On Tue, Aug 22, 2023 at 3:37 AM Z qiang <qiang.zhang1211@gmail.com> wrote:
>
> When running build-in rcutorture tests in 6.5.0-rc4-rt, and found that,
> although the value of /proc/sys/kernel/sched_rt_runtime_us is -1,
>
> cat /sys/kernel/debug/sched/debug
> ....
> rt_rq[6]:
>   .rt_nr_running                 : 4
>   .rt_nr_migratory               : 0
>   .rt_throttled                  : 0
>   .rt_time                       : 0.000000
>   .rt_runtime                    : 950.000000
>
> but the rt_runtime still is 950.000000.
> set sysctl_sched_rt_runtime in rcu_torture_disable_rt_throttle()
> does not disable rt-throttling.

I think you have hit a bug. I think the problem is
rcu_torture_disable_rt_throttle() modifies the sysctl knobs, but does
not change def_rt_bandwidth and reinitialize the rt_rq. I think we
need to call sched_rt_do_global() like the sysfs handler does, or
change the sysctl knobs to be earlier in the boot process before the
rt_rq are initialized.

Or better yet (not sure if it is possible) trigger the sysctl change
via the sysctl layer and let it do the same logic.

Thoughts?

Thanks.

Re: rcutorture: Can not Disable RT throttling

[Paul E. McKenney]

On Wed, Aug 23, 2023 at 09:50:37AM -0400, Joel Fernandes wrote:
> On Tue, Aug 22, 2023 at 3:37 AM Z qiang <qiang.zhang1211@gmail.com> wrote:
> >
> > When running build-in rcutorture tests in 6.5.0-rc4-rt, and found that,
> > although the value of /proc/sys/kernel/sched_rt_runtime_us is -1,
> >
> > cat /sys/kernel/debug/sched/debug
> > ....
> > rt_rq[6]:
> >   .rt_nr_running                 : 4
> >   .rt_nr_migratory               : 0
> >   .rt_throttled                  : 0
> >   .rt_time                       : 0.000000
> >   .rt_runtime                    : 950.000000
> >
> > but the rt_runtime still is 950.000000.
> > set sysctl_sched_rt_runtime in rcu_torture_disable_rt_throttle()
> > does not disable rt-throttling.
> 
> I think you have hit a bug. I think the problem is
> rcu_torture_disable_rt_throttle() modifies the sysctl knobs, but does
> not change def_rt_bandwidth and reinitialize the rt_rq. I think we
> need to call sched_rt_do_global() like the sysfs handler does, or
> change the sysctl knobs to be earlier in the boot process before the
> rt_rq are initialized.
> 
> Or better yet (not sure if it is possible) trigger the sysctl change
> via the sysctl layer and let it do the same logic.

That would be difficult in built-in rcutorture testing due to the fact
that there is not much in the way of userspace.  I suppose we could
invoke the sysfs handler so as to mock up a userspace access, but that
might not be the most robust approach.

Another way is to disable preemption in the real-time kthreads.  Which
might need careful implementation to avoid "scheduling while atomic"
and friends.

							Thanx, Paul

Re: rcutorture: Can not Disable RT throttling

[Z qiang]

>
> On Wed, Aug 23, 2023 at 09:50:37AM -0400, Joel Fernandes wrote:
> > On Tue, Aug 22, 2023 at 3:37 AM Z qiang <qiang.zhang1211@gmail.com> wrote:
> > >
> > > When running build-in rcutorture tests in 6.5.0-rc4-rt, and found that,
> > > although the value of /proc/sys/kernel/sched_rt_runtime_us is -1,
> > >
> > > cat /sys/kernel/debug/sched/debug
> > > ....
> > > rt_rq[6]:
> > >   .rt_nr_running                 : 4
> > >   .rt_nr_migratory               : 0
> > >   .rt_throttled                  : 0
> > >   .rt_time                       : 0.000000
> > >   .rt_runtime                    : 950.000000
> > >
> > > but the rt_runtime still is 950.000000.
> > > set sysctl_sched_rt_runtime in rcu_torture_disable_rt_throttle()
> > > does not disable rt-throttling.
> >
> > I think you have hit a bug. I think the problem is
> > rcu_torture_disable_rt_throttle() modifies the sysctl knobs, but does
> > not change def_rt_bandwidth and reinitialize the rt_rq. I think we
> > need to call sched_rt_do_global() like the sysfs handler does, or
> > change the sysctl knobs to be earlier in the boot process before the
> > rt_rq are initialized.

I've thought about something like this before, call the sched_rt*() in
the sched_rt_handler(),
this requires exporting these functions.

> >
> > Or better yet (not sure if it is possible) trigger the sysctl change
> > via the sysctl layer and let it do the same logic.

This is good, but I haven't found it yet

>
> That would be difficult in built-in rcutorture testing due to the fact
> that there is not much in the way of userspace.  I suppose we could
> invoke the sysfs handler so as to mock up a userspace access, but that
> might not be the most robust approach.

Does it mean to use filp_open() to open sched_rt_runtime_us file and
write -1 through kernel_write() ?


>
> Another way is to disable preemption in the real-time kthreads.  Which
> might need careful implementation to avoid "scheduling while atomic"
> and friends.
>
>                                                         Thanx, Paul

Re: rcutorture: Can not Disable RT throttling

[Joel Fernandes]

On Thu, Aug 24, 2023 at 10:30:37AM +0800, Z qiang wrote:
[..]
> > That would be difficult in built-in rcutorture testing due to the fact
> > that there is not much in the way of userspace.  I suppose we could
> > invoke the sysfs handler so as to mock up a userspace access, but that
> > might not be the most robust approach.

Yes.

> Does it mean to use filp_open() to open sched_rt_runtime_us file and
> write -1 through kernel_write() ?

I am afraid so but that might become hard to maintain in the long run. I
prefer exporting the necessary functions from RT to the kernel and just
calling those.

thanks,

 - Joel