Can I disable RT throttling?

Can I disable RT throttling?

[David Mulder]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

Hi.

I'm running a 10us control loop by (under vxWorks) setting one thread's priority to max and not yielding ever (letting other tasks run on other cores), but Linux seems to thwart that capability: Ubuntu swaps out my thread occasionally for hundreds of microseconds; Yocto prints a "[sched_delayed] sched: RT throttling activated" message as soon as I start my thread, so it seems likely that it will swap out my thread periodically (I'm not quite able to confirm that yet), and even the fastest thread swap that I've heard of is too slow.

I tried changing the kernel's Preemption Model to "No Forced Preemption (Server)", but that didn't remove the RT throttling message.  I browsed the rest of menuconfig but nothing looked related.  I read an article from 2008 that talked about the kernel reserving 5% of the CPU for non-SCHED_FIFO tasks, so that's the direction and terminology I looked for in menuconfig.

So maybe disabling RT throttling is all I need to do, maybe some other approach is needed.  Hopefully someone will know more than I do.

Thanks!
Dave Mulder


[-- Attachment #2: Type: text/html, Size: 3565 bytes --]

Re: Can I disable RT throttling?

[Bruce Ashfield]

On 13-03-08 5:36 PM, David Mulder wrote:
> Hi.
>
> I’m running a 10us control loop by (under vxWorks) setting one thread’s
> priority to max and not yielding ever (letting other tasks run on other
> cores), but Linux seems to thwart that capability: Ubuntu swaps out my
> thread occasionally for hundreds of microseconds; Yocto prints a
> “[sched_delayed] sched: RT throttling activated” message as soon as I
> start my thread, so it seems likely that it will swap out my thread
> periodically (I’m not quite able to confirm that yet), and even the
> fastest thread swap that I’ve heard of is too slow.

You are looking for kernel functionality that doesn't exist yet. In
any SMP system there are sources of cross cpu interference that can't
be removed, if you do, the global state machines of the kernel will
break and the system will eventually come to a halt.

The system is trying to save you from yourself, by throttling the
RT task from taking the entire system down.

There is work in the mainline kernel and -rt communities around cpu
and cpu isolation (some of which we'll try and make available via
the yocto meta-virtualization or meta-realtime layer, when they are
ready), it goes by names such as task_nohz or adaptive_nohz. It's
a complex problem to solve (interrupts, rcu, lapic), and there's not
likely to be anything available in the short term.

That's the mainline/scalable point of view, there are plenty of "custom"
and one off solutions to the problem, such as hotplugging the cpu out
of the system and running in an AMP configuration where a bare metal,
or RTOS can monopolize a CPU since the global state machines don't
interfere. These have their advantages and disadvantages, but if you
are coming from an existing RTOS application, they might be the
closest options to get you the performance you are looking for.

Alternatively, there's the preempt-rt kernel (that we make available
in linux-yocto, if that suits) that if your control loop is
interrupt driven, you'll be closer to your needs since the kernel
is far more preemptible and deterministic and might be able to meet
your deadlines.

There are experimental patches like sched_deadline (EDF) that might
also be applicable, depending on your application architecture.

Searching for "making Linux hard realtime", you'll find plenty of
talks and research over the years for yet more custom ways to get
things done.

So that's my summary, as you can see, it's a long standing, evolving
and complex story .. and not something that likely has an 'out of the
box' solution at the moment.

Cheers,

Bruce

>
> I 0tried changing the kernel’s Preemption Model to “No Forced Preemption
> (Server)”, but that didn’t remove the RT throttling message. I browsed
> the rest of menuconfig but nothing looked related. I read an article
> from 2008 that talked about the kernel reserving 5% of the CPU for
> non-SCHED_FIFO tasks, so that’s the direction and terminology I looked
> for in menuconfig.
>
> So maybe disabling RT throttling is all I need to do, maybe some other
> approach is needed. Hopefully someone will know more than I do.
>
> Thanks!
>
> Dave Mulder
>
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Can I disable RT throttling?

[Vin Shelton]

On Fri, Mar 8, 2013 at 10:28 PM, Bruce Ashfield
<bruce.ashfield@windriver.com> wrote:
> On 13-03-08 5:36 PM, David Mulder wrote:
>> I’m running a 10us control loop by (under vxWorks) setting one thread’s
>> priority to max and not yielding ever (letting other tasks run on other
>> cores), but Linux seems to thwart that capability: Ubuntu swaps out my
>> thread occasionally for hundreds of microseconds; Yocto prints a
>> “[sched_delayed] sched: RT throttling activated” message as soon as I
>> start my thread, so it seems likely that it will swap out my thread
>> periodically (I’m not quite able to confirm that yet), and even the
>> fastest thread swap that I’ve heard of is too slow.
>
>
> You are looking for kernel functionality that doesn't exist yet. In
> any SMP system there are sources of cross cpu interference that can't
> be removed, if you do, the global state machines of the kernel will
> break and the system will eventually come to a halt.
>
> The system is trying to save you from yourself, by throttling the
> RT task from taking the entire system down.
>
> There is work in the mainline kernel and -rt communities around cpu
> and cpu isolation (some of which we'll try and make available via
> the yocto meta-virtualization or meta-realtime layer, when they are
> ready), it goes by names such as task_nohz or adaptive_nohz. It's
> a complex problem to solve (interrupts, rcu, lapic), and there's not
> likely to be anything available in the short term.

At ELC, Steve Rostedt said that this feature is currently targeted at
the June mainline kernel release, but there's a lot of work still to
be done, so that's not a firm date.

Regards,
  Vin Shelton

Re: Can I disable RT throttling?

[Bruce Ashfield]

On Sat, Mar 9, 2013 at 12:12 PM, Vin Shelton <acs@alumni.princeton.edu> wrote:
> On Fri, Mar 8, 2013 at 10:28 PM, Bruce Ashfield
> <bruce.ashfield@windriver.com> wrote:
>> On 13-03-08 5:36 PM, David Mulder wrote:
>>> I’m running a 10us control loop by (under vxWorks) setting one thread’s
>>> priority to max and not yielding ever (letting other tasks run on other
>>> cores), but Linux seems to thwart that capability: Ubuntu swaps out my
>>> thread occasionally for hundreds of microseconds; Yocto prints a
>>> “[sched_delayed] sched: RT throttling activated” message as soon as I
>>> start my thread, so it seems likely that it will swap out my thread
>>> periodically (I’m not quite able to confirm that yet), and even the
>>> fastest thread swap that I’ve heard of is too slow.
>>
>>
>> You are looking for kernel functionality that doesn't exist yet. In
>> any SMP system there are sources of cross cpu interference that can't
>> be removed, if you do, the global state machines of the kernel will
>> break and the system will eventually come to a halt.
>>
>> The system is trying to save you from yourself, by throttling the
>> RT task from taking the entire system down.
>>
>> There is work in the mainline kernel and -rt communities around cpu
>> and cpu isolation (some of which we'll try and make available via
>> the yocto meta-virtualization or meta-realtime layer, when they are
>> ready), it goes by names such as task_nohz or adaptive_nohz. It's
>> a complex problem to solve (interrupts, rcu, lapic), and there's not
>> likely to be anything available in the short term.
>
> At ELC, Steve Rostedt said that this feature is currently targeted at
> the June mainline kernel release, but there's a lot of work still to
> be done, so that's not a firm date.

Absolutely. There's no such thing as a firm date for anything, I (among
others) are following the development closely, testing, and helping wherever
possible.

Cheers,

Bruce

>
> Regards,
>   Vin Shelton
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Can I disable RT throttling?

[David Mulder]

Wow! Thanks, Bruce, Fredrik, and everyone, for all this useful info!

> > I'm running a 10us control loop by (under vxWorks) setting one thread's
> > priority to max and not yielding ever
> 
> <bruce.ashfield@windriver.com> wrote:
> You are looking for kernel functionality that doesn't exist yet. In
> any SMP system there are sources of cross cpu interference that can't
> be removed, if you do, the global state machines of the kernel will
> break and the system will eventually come to a halt.
> 
> That's the mainline/scalable point of view, there are plenty of "custom"
> and one off solutions to the problem, such as hotplugging the cpu out
> of the system and running in an AMP configuration where a bare metal,
> or RTOS can monopolize a CPU since the global state machines don't
> interfere. These have their advantages and disadvantages, but if you
> are coming from an existing RTOS application, they might be the
> closest options to get you the performance you are looking for.

An AMP configuration does seem like the closest thing to what the RTOS gave me.  Do you have more info about that?  Specifically, I see that I can remove a core from the scheduler ($ echo 0 > /sys/devices/system/cpu/<core number>/online), but how can I make my run my code on that core?  Do I get the code running on that core first and then remove the core, or vice-versa, or some other concept?  I do PCIe reads/writes - are those affected?  I use atomic read-write-modify locks for critical sections - will those still function correctly (I can't imagine they would break, but I thought I'd ask)?

> > So maybe disabling RT throttling is all I need to do, maybe some other
> > approach is needed. Hopefully someone will know more than I do.
> >

> <fredrik.markstrom@gmail.com> wrote:
> The RT-throttling can be disabled with:
>
> echo -1 >/proc/sys/kernel/sched_rt_runtime_us
> echo -1 >/proc/sys/kernel/sched_rt_period_us
>
> But as mentioned in the other answers things like the system tick, ipi:s
> etc. will interfere. But significantly less then the rt-throttling :)

Thanks, Fredrick!  I will try this.

Dave Mulder

Re: Can I disable RT throttling?

[Trevor Woerner]

On Mon, Mar 11, 2013 at 12:15 PM, David Mulder <dmulder@lasx.com> wrote:
> but how can I make my run my code on that core?

You can call sched_setaffinity() before fork()ing your task, or use
taskset on the cmdline.

(sorry if this has already been answered but I'm not seeing any
responses due to mailing list delays)

Re: Can I disable RT throttling?

[David Mulder]

> From: Trevor Woerner [mailto:twoerner@gmail.com]
> You can call sched_setaffinity() before fork()ing your task, or use
> taskset on the cmdline.

Will that work on a core that's offline?

Re: Can I disable RT throttling?

[Bruce Ashfield]

On 13-03-11 01:38 PM, David Mulder wrote:
>> From: Trevor Woerner [mailto:twoerner@gmail.com]
>> You can call sched_setaffinity() before fork()ing your task, or use
>> taskset on the cmdline.
>
> Will that work on a core that's offline?

Nope. Only with an online core controlled by the Linux scheduler.
If you do end up trying to get AMP working, you need to plumbing
to load the other OS/kernel in a reserved memory location, set the
program counter and start the OS.

But that secondary OS has to know how to behave in a system that
it doesn't control, and you'll need ways to communicate with it
from Linux.

remoteproc/rpmsg can solve some of the issues that I mention, but
it is far from out of the box.

That's why there's more interest in running a single task with
exclusive CPU in userspace. The work and scaffolding required to
get an AMP system up and running is non trivial.

Cheers,

Bruce

> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Can I disable RT throttling?

[David Mulder]

> > Will that work on a core that's offline?
> 
> Nope. Only with an online core controlled by the Linux scheduler.
> If you do end up trying to get AMP working, you need to plumbing
> to load the other OS/kernel in a reserved memory location, set the
> program counter and start the OS.
> 
> But that secondary OS has to know how to behave in a system that
> it doesn't control, and you'll need ways to communicate with it
> from Linux.
> 
> remoteproc/rpmsg can solve some of the issues that I mention, but
> it is far from out of the box.
> 
> That's why there's more interest in running a single task with
> exclusive CPU in userspace. The work and scaffolding required to
> get an AMP system up and running is non trivial.

I kinda thought so, but I was hopeful.

After speaking with some co-workers, I have a new perspective on these delays:  yes, we are trying to do a 10us control loop, but if we miss a step or two occasionally we can accept that.  And looking online I see people indicating context switch times well below 10us (Core-i cpus), which is better than I had anticipated, and should be workable.  So I'm going to approach this problem by just trying to squeeze the kernel as much as I can.  Some things that I see to squeeze are /dev/cpu_dma_latency (should be 0) or max_cstate (should be as low as possible (0, maybe 1)), possibly idle=poll.  Are there other kernel parameters that can minimize kernel interference/time?  And perhaps hints about how to set them in Yocto or menuconfig?

Thanks!
Dave

Re: Can I disable RT throttling?

[Bruce Ashfield]

On 13-03-11 2:13 PM, David Mulder wrote:
>>> Will that work on a core that's offline?
>>
>> Nope. Only with an online core controlled by the Linux scheduler.
>> If you do end up trying to get AMP working, you need to plumbing
>> to load the other OS/kernel in a reserved memory location, set the
>> program counter and start the OS.
>>
>> But that secondary OS has to know how to behave in a system that
>> it doesn't control, and you'll need ways to communicate with it
>> from Linux.
>>
>> remoteproc/rpmsg can solve some of the issues that I mention, but
>> it is far from out of the box.
>>
>> That's why there's more interest in running a single task with
>> exclusive CPU in userspace. The work and scaffolding required to
>> get an AMP system up and running is non trivial.
>
> I kinda thought so, but I was hopeful.
>
> After speaking with some co-workers, I have a new perspective on these delays:  yes, we are trying to do a 10us control loop, but if we miss a step or two occasionally we can accept that.  And looking online I see people indicating context switch times well below 10us (Core-i cpus), which is better than I had anticipated, and should be workable.  So I'm going to approach this problem by just trying to squeeze the kernel as much as I can.  Some things that I see to squeeze are /dev/cpu_dma_latency (should be 0) or max_cstate (should be as low as possible (0, maybe 1)), possibly idle=poll.  Are there other kernel parameters that can minimize kernel interference/time?  And perhaps hints about how to set them in Yocto or menuconfig?

There's nothing "out of the box" that I can recommend, outside of saying
"it depends on your platform". It's a matter of knowing your devices,
their interrupts, and the configuration of your kernel. Using things
like CONFIG_NOHZ will remove the timer tick, and hence ticks that you
may not need, you want to move device interrupts off your core, except
for the one that you want. Use cgroups/cpusets to control resources and
the scheduler off your core with "other" tasks. pin/lock memory to
avoid page faults, etc.

If you check out the preempt-rt wiki page on kernel.org, a lot of the
information there applies to making sure that your prioritized thread
gets the most run time that it can.

As we progress with the meta-realtime layer, scripts for the above,
system configuration, services and will be part of the layer and might
be of use.

Cheers,

Bruce


>
> Thanks!
> Dave

Re: Can I disable RT throttling?

[Fredrik Markström]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

The RT-throttling can be disabled with:

echo -1 >/proc/sys/kernel/sched_rt_runtime_us
echo -1 >/proc/sys/kernel/sched_rt_period_us

But as mentioned in the other answers things like the system tick, ipi:s
etc. will interfere. But significantly less then the rt-throttling :)

/Fredrik

[-- Attachment #2: Type: text/html, Size: 429 bytes --]

Re: Can I disable RT throttling?

[Bruce Ashfield]

On Sun, Mar 10, 2013 at 7:42 AM, Fredrik Markström
<fredrik.markstrom@gmail.com> wrote:
> The RT-throttling can be disabled with:
>
> echo -1 >/proc/sys/kernel/sched_rt_runtime_us
> echo -1 >/proc/sys/kernel/sched_rt_period_us
>
> But as mentioned in the other answers things like the system tick, ipi:s
> etc. will interfere. But significantly less then the rt-throttling :)

Absolutely! And as long as you really know what you are doing, you
won't kill the
system .. turning off the throttling definitely gives a bit more rope
to that end as
well :P

Cheers,

Bruce

>
> /Fredrik
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"