RCU simplification and RT needs

RCU simplification and RT needs

[Paul E. McKenney]

Hello!

At Linus's request, I am simplifying the Linux-kernel RCU implementation,
which includes removing code that implements features and options that
are no longer needed.  This is not a half-hearted effort.  In fact,
I expect that my submission to the next merge window will be a net
removal of more than 2500 lines of code.

But wait, there is more!  ;-)

Although the following two features are not being axed in v4.13, they
will be in v4.14 unless someone makes a convincing case for them:

1.	The ability to build a CONFIG_RCU_NOCB_CPUS=y kernel without
	also specifying CONFIG_NO_HZ_FULL.

	Unless someone speaks for this configuration option,
	CONFIG_RCU_NOCB_CPUS will be slaved off of CONFIG_NO_HZ_FULL,
	and the rcu_nocbs= boot parameter will be dropped.  (RCU would
	instead use the nohz_full= boot parameter to determine which
	CPUs get their callbacks offloaded.)

2.	The ability to specify polling for callback-offloaded CPUs.  This
	means that the rcu_nocb_poll= boot parameter will be dropped,
	and the CPU doing call_rcu() would do explicit wakeups, when
	needed, to get the corresponding rcuo kthread on the job.

	I have no evidence that anyone has ever used this option, other
	than me running the occasional rcutorture test.

So, anyone need either of these?  If not, out they go!  ;-)

							Thanx, Paul

AW: RCU simplification and RT needs

[eg Engleder Gerhard]

> Hello!
Hello Paul,
> 
> At Linus's request, I am simplifying the Linux-kernel RCU implementation, which
> includes removing code that implements features and options that are no longer
> needed.  This is not a half-hearted effort.  In fact, I expect that my submission to
> the next merge window will be a net removal of more than 2500 lines of code.
> 
> But wait, there is more!  ;-)
> 
> Although the following two features are not being axed in v4.13, they will be in
> v4.14 unless someone makes a convincing case for them:
> 
> 1.	The ability to build a CONFIG_RCU_NOCB_CPUS=y kernel without
> 	also specifying CONFIG_NO_HZ_FULL.
> 
> 	Unless someone speaks for this configuration option,
> 	CONFIG_RCU_NOCB_CPUS will be slaved off of CONFIG_NO_HZ_FULL,
> 	and the rcu_nocbs= boot parameter will be dropped.  (RCU would
> 	instead use the nohz_full= boot parameter to determine which
> 	CPUs get their callbacks offloaded.)

We are using CONFIG_RCU_NOCB_CPUS=y without CONFIG_NO_HZ_FULL
with 4.9-rt in our products.
We are not using CONFIG_NO_HZ_FULL, because its use was not recommended for
real time. We set CONFIG_RCU_NOCB_CPUS=y and CONFIG_RCU_NOCB_CPU_ALL=y
to reduce the jitter. This configuration showed the minimal jitter with our application.

We are not relying on the configuration itself. Our goal is just a minimum jitter. Thus, if
CONFIG_NO_HZ_FULL can be used for real time with no negative impact on the
jitter, then we can enable this option.
@tglx: Is CONFIG_NO_HZ_FULL the preferred choice for real-time kernels?

Gerhard

Re: RCU simplification and RT needs

[Daniel Bristot de Oliveira]

On 06/05/2017 10:35 PM, Paul E. McKenney wrote:
> Hello!
> 
> At Linus's request, I am simplifying the Linux-kernel RCU implementation,
> which includes removing code that implements features and options that
> are no longer needed.  This is not a half-hearted effort.  In fact,
> I expect that my submission to the next merge window will be a net
> removal of more than 2500 lines of code.

Nice :-)

> But wait, there is more!  ;-)
> 
> Although the following two features are not being axed in v4.13, they
> will be in v4.14 unless someone makes a convincing case for them:
> 
> 1.	The ability to build a CONFIG_RCU_NOCB_CPUS=y kernel without
> 	also specifying CONFIG_NO_HZ_FULL.
> 
> 	Unless someone speaks for this configuration option,
> 	CONFIG_RCU_NOCB_CPUS will be slaved off of CONFIG_NO_HZ_FULL,
> 	and the rcu_nocbs= boot parameter will be dropped.  (RCU would
> 	instead use the nohz_full= boot parameter to determine which
> 	CPUs get their callbacks offloaded.)

In our product (RHEL-RT), we do not have rcu offload enabled on all CPUs
by default, that's because there are some cases in which customers want
to avoid context switches - they let rcuc/ as a low priority thread and
let it do all the job, when there is nothing else to do. So, we let the
user decide in which CPUs they want to offload RCU.

The side effect of enforcing NOHZ_FULL on those cases are two:

1) NOHZ_FULL enables NOHZ, which causes an overhead in the
exit-from-idle path, increasing the average latency;

2) Not all RT users want to pay the syscall overheads involved on
NOHZ_FULL. NOHZ_FULL is very nice for HPC users (user-space busy-loop
tools) but not for RT users doing fine grained events. These users like
to offload RCU callbacks on some CPUs, but do not want to pay the
NOHZ_FULL price.

So, for event/response real-time users, NOHZ_FULL + NOHZ causes
undesired overhead, while they want to have rcu_nocbs= enabled. That is
why I believe that having both rcu_nocbs= and  nohz_full= separated is
very useful.

> 2.	The ability to specify polling for callback-offloaded CPUs.  This
> 	means that the rcu_nocb_poll= boot parameter will be dropped,
> 	and the CPU doing call_rcu() would do explicit wakeups, when
> 	needed, to get the corresponding rcuo kthread on the job.
> 
> 	I have no evidence that anyone has ever used this option, other
> 	than me running the occasional rcutorture test.

We are using it in some cases. There are cases in which users do not
want to see any interference in a CPU, let's call them heavy CPU
isolation users. They do not want to see any job there other than their
user-space busy-loop application. In those cases, they do not want to
see the rcuc/ threads being awakened - not even to juts wake up another
thread. Although these cases look more HPC than RT, those users want to
use the RT kernel to avoid latency in other real-time threads running in
the same system. I think that the best example of those users is NFV people.

> So, anyone need either of these?  If not, out they go!  ;-)
As we deal with many different sort of real-time/HPC workload in the
enterprise RT world... we end up facing many different user-cases, and
those options are very useful for us.

-- Daniel

Re: RCU simplification and RT needs

[Paul E. McKenney]

On Tue, Jun 06, 2017 at 09:57:47AM +0200, Daniel Bristot de Oliveira wrote:
> 
> On 06/05/2017 10:35 PM, Paul E. McKenney wrote:
> > Hello!
> > 
> > At Linus's request, I am simplifying the Linux-kernel RCU implementation,
> > which includes removing code that implements features and options that
> > are no longer needed.  This is not a half-hearted effort.  In fact,
> > I expect that my submission to the next merge window will be a net
> > removal of more than 2500 lines of code.
> 
> Nice :-)
> 
> > But wait, there is more!  ;-)
> > 
> > Although the following two features are not being axed in v4.13, they
> > will be in v4.14 unless someone makes a convincing case for them:
> > 
> > 1.	The ability to build a CONFIG_RCU_NOCB_CPUS=y kernel without
> > 	also specifying CONFIG_NO_HZ_FULL.
> > 
> > 	Unless someone speaks for this configuration option,
> > 	CONFIG_RCU_NOCB_CPUS will be slaved off of CONFIG_NO_HZ_FULL,
> > 	and the rcu_nocbs= boot parameter will be dropped.  (RCU would
> > 	instead use the nohz_full= boot parameter to determine which
> > 	CPUs get their callbacks offloaded.)
> 
> In our product (RHEL-RT), we do not have rcu offload enabled on all CPUs
> by default, that's because there are some cases in which customers want
> to avoid context switches - they let rcuc/ as a low priority thread and
> let it do all the job, when there is nothing else to do. So, we let the
> user decide in which CPUs they want to offload RCU.
> 
> The side effect of enforcing NOHZ_FULL on those cases are two:
> 
> 1) NOHZ_FULL enables NOHZ, which causes an overhead in the
> exit-from-idle path, increasing the average latency;
> 
> 2) Not all RT users want to pay the syscall overheads involved on
> NOHZ_FULL. NOHZ_FULL is very nice for HPC users (user-space busy-loop
> tools) but not for RT users doing fine grained events. These users like
> to offload RCU callbacks on some CPUs, but do not want to pay the
> NOHZ_FULL price.
> 
> So, for event/response real-time users, NOHZ_FULL + NOHZ causes
> undesired overhead, while they want to have rcu_nocbs= enabled. That is
> why I believe that having both rcu_nocbs= and  nohz_full= separated is
> very useful.
> 
> > 2.	The ability to specify polling for callback-offloaded CPUs.  This
> > 	means that the rcu_nocb_poll= boot parameter will be dropped,
> > 	and the CPU doing call_rcu() would do explicit wakeups, when
> > 	needed, to get the corresponding rcuo kthread on the job.
> > 
> > 	I have no evidence that anyone has ever used this option, other
> > 	than me running the occasional rcutorture test.
> 
> We are using it in some cases. There are cases in which users do not
> want to see any interference in a CPU, let's call them heavy CPU
> isolation users. They do not want to see any job there other than their
> user-space busy-loop application. In those cases, they do not want to
> see the rcuc/ threads being awakened - not even to juts wake up another
> thread. Although these cases look more HPC than RT, those users want to
> use the RT kernel to avoid latency in other real-time threads running in
> the same system. I think that the best example of those users is NFV people.
> 
> > So, anyone need either of these?  If not, out they go!  ;-)
> As we deal with many different sort of real-time/HPC workload in the
> enterprise RT world... we end up facing many different user-cases, and
> those options are very useful for us.

Thanks to both you and Gerhard!

Looks like I should figure on keeping these, at least for the time being.

							Thanx, Paul