Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[Andrei Hurynovich]

Hi.

We are trying to build realtime(-ish) system based on rhel6(kernel 
2.6.32-642.1.1.el6.x86_64).

We used isolcpus to remove some cpus from process 
scheduling(isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19).

We spin off a program thread that set's its cpu affinity to one of those 
isolated cpus, sets its scheduling chass to SCHED_FIFO, spins in a loop 
and never sched_yield()-s to the kernel.

We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are 
NEVER interrupted.

We are observing that the program thread is indeed realtime and is never 
interrupted.

After some time working like this, the system becomes irresponsive - ssh 
connections start failing with timeout, existing connections hang when 
trying to read/write to physical disks(reading procfs or writing to 
tmpfs is unaffected).

***
According to /proc/sched_debug, it seems that kernel still schedules 
some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for 
example cpu 18 get tasks events/18 and kblockd/18 that are stuck in 
runnable(but not running state), so those kernel processes never got a 
single time slice because our realtime process hogs 100% of cpu.

And kernel/18 and kblockd/18 never migrate to other cpus because these 
tasks are pinned to cpu 18.
***

Check please these /proc/sched_debug snapshots, you can see that 
events/18 and kblockd/18 sum-exec counters are not increasing:
https://gist.github.com/altmind/5cf4aad87a4a082441c1ca9378a06154




The question is: Is it possible to never schedule kernel tasks on 
selected cpus?

-- 
Thanks,
Andrei Hurynovich

Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[valdis.kletnieks at vt.edu]

On Wed, 28 Jun 2017 08:39:07 -0500, Andrei Hurynovich said:
> We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are
> NEVER interrupted.

> According to /proc/sched_debug, it seems that kernel still schedules
> some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for
> example cpu 18 get tasks events/18 and kblockd/18 that are stuck in
> runnable(but not running state), so those kernel processes never got a
> single time slice because our realtime process hogs 100% of cpu.

This is what happens when you have a priority inversion - when you tell the system
to give 100% to a process, you shouldn't be surprised when other tasks don't
get any service.

> The question is: Is it possible to never schedule kernel tasks on
> selected cpus?

Only if the userspace process on that CPU never makes system calls - which is
very unlikely if the process has actual real-time requirements.

Also, if your "real-time" process is taking 100% of the CPU, you have a disaster
waiting to happen.  You have zero headroom for dealing with unexpected events.
Thought experiment:  What happens if your real-world part of the system has
an unexpected error, that requires 1% of a CPU for error recovery?  You are
forced to either ignore the error or miss a real-time deadline.

You might want to think about dividing up your process into 2 parts - one that
handles the *actual* real-time work and only uses (for example) 20-30% of a
CPU, and the parts that don't have actual real-time constraints that can then
run with the rest of the available CPU, but allow other threads (such as kernel)
to execute as well.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170628/6c639e8d/attachment.bin 

Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[Andrei Hurynovich]

Thank you Valdis.

Yes, I'm basically getting what I want - the RT proc never ever gives up 
to the system. There are a plenty of cores left to run non-rt tasks on 
the machine.


The question is why this old 2.6 kernel decide that it needs per-cpu 
events and kblockd tasks.

Maybe someone can give a hint in what subsystem's documentation can I 
find anything about workqueue tasks.

/Documentation/kernel-per-CPU-kthreads.txt is great, but described 
controls appeared only in 3.10.x :)


On 06/28/2017 01:04 PM, valdis.kletnieks at vt.edu wrote:
> On Wed, 28 Jun 2017 08:39:07 -0500, Andrei Hurynovich said:
>> We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are
>> NEVER interrupted.
>> According to /proc/sched_debug, it seems that kernel still schedules
>> some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for
>> example cpu 18 get tasks events/18 and kblockd/18 that are stuck in
>> runnable(but not running state), so those kernel processes never got a
>> single time slice because our realtime process hogs 100% of cpu.
> This is what happens when you have a priority inversion - when you tell the system
> to give 100% to a process, you shouldn't be surprised when other tasks don't
> get any service.
>
>> The question is: Is it possible to never schedule kernel tasks on
>> selected cpus?
> Only if the userspace process on that CPU never makes system calls - which is
> very unlikely if the process has actual real-time requirements.
>
> Also, if your "real-time" process is taking 100% of the CPU, you have a disaster
> waiting to happen.  You have zero headroom for dealing with unexpected events.
> Thought experiment:  What happens if your real-world part of the system has
> an unexpected error, that requires 1% of a CPU for error recovery?  You are
> forced to either ignore the error or miss a real-time deadline.
>
> You might want to think about dividing up your process into 2 parts - one that
> handles the *actual* real-time work and only uses (for example) 20-30% of a
> CPU, and the parts that don't have actual real-time constraints that can then
> run with the rest of the available CPU, but allow other threads (such as kernel)
> to execute as well.
>

-- 
Thanks,
Andrei Hurynovich

Charlesworth Research LLC.
http://www.charlesworthresearch.com/

Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[valdis.kletnieks at vt.edu]

On Wed, 28 Jun 2017 14:02:37 -0500, Andrei Hurynovich said:
> The question is why this old 2.6 kernel decide that it needs per-cpu
> events and kblockd tasks.

You have per-cpu events ecause your real-time process issues syscalls, and
syscalls do things inside the kernel that require per-CPU infrastructure.

You have kblockd and other per-cpu threads because you're using an old ancient
kernel that doesn't have Frederic Weisbecker's CONFIG_NO_HZ_FULL support, or
the support for a single system-wide kblockd, or a mess of other stuff.  So it
isn't that it "decides" to do it per-cpu, it's because 2.6.32 had it done that
way for simplicity, and in the *8 years* since 2.6.32 was released, people
cleaned much of that stuff up.

[/usr/src/linux-next] git diff --shortstat v2.6.32 next-20170627
 64638 files changed, 17302278 insertions(+), 5102365 deletions(-)

Yes, you're 17 *million* lines of code behind the times.  Much of what you are
complaining about was solved half a decade or more ago.

Or as Greg suggested, use a modern kernel. :)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170628/3f77ea30/attachment.bin 

Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[Greg KH]

On Wed, Jun 28, 2017 at 08:39:07AM -0500, Andrei Hurynovich wrote:
> Hi.
> 
> We are trying to build realtime(-ish) system based on rhel6(kernel 
> 2.6.32-642.1.1.el6.x86_64).

Wow, you do realize that is a _very_ old and obsolete kernel, supported
by no one except Red Hat.  If you stick with it, you are going to have
to get your support from them, not the community, as you are already
paying for it :)

Good luck!

greg k-h

Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

[Rik van Riel]

On Wed, 2017-06-28 at 08:39 -0500, Andrei Hurynovich wrote:
> Hi.
> 
> We are trying to build realtime(-ish) system based on rhel6(kernel?
> 2.6.32-642.1.1.el6.x86_64).
> 
> We used isolcpus to remove some cpus from process?
> scheduling(isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19).
> 
> We spin off a program thread that set's its cpu affinity to one of
> those?
> isolated cpus, sets its scheduling chass to SCHED_FIFO, spins in a
> loop?
> and never sched_yield()-s to the kernel.
> 
> We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads
> are?
> NEVER interrupted.

You want an actual realtime kernel for that to work
right. The real time kernel currently supported by
Red Hat is 3.10 based, not 2.6.32 based.

-- 
All rights reversed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170628/f1a5fb2a/attachment.bin