RE: Real-time kernel thread performance and optimization

[Simon Falsig <simon@newtec.dk>]

>
> Simon,
>
> >>>> [..]
> >>>> Bonus-question:
> >>>>    - Additionally, I've tried running cyclictest alongside with all
> >>>> the above, and it actually performs rather well, without any
> >>>> substantial spikes. A strange thing is though, that the results are
> >>>> actually better with load than without? (running with -t1 -p 80 -n
-i
> 10000 -l 10000)
> >>>>    - Loaded: Min: 16, Avg: 41, Max: 177
> >>>>    - No load: Min: 16, Avg: 97, Max: 263
> >>>
> >>> If the system is less loaded, then the idle thread might be able to
> >>> enter deeper levels of sleep.  Deeper levels of sleep have larger
> >>> latencies to exit.  You would have to look at your processor
> >>> specific values for exiting sleep states to see if this is
> >>> sufficient to explain the difference.
> >> If running a half-decent version of cyclictest, sleep states are
> >> generally disabled while cyclictest is running. Please watch the line
> >>     # /dev/cpu_dma_latency set to 0us which essentially documents
> >> this mechanism. Yes, the name of the variable "cpu_dma_latency" is
> >> not obvious and cyclictest could do a better job by writing
> >>     Wrote 0 to /dev/cpu_dma_latency and keeping the path open to
> prevent
> >>     all cores from entering any sleep state but this is another
story.
> >>
> >> A patch that was merged to 3.7 allows to individually enable or
> >> disable sleep states of the ladder governor
> >>
>
(http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=6
2d
> 6ae880e3e76098d5e345decd2dce443975889).
> >> It smoothly applies to 3.6-RT as well. This allows to fine-tune the
> >> sleep states by state and core, while the /dev/cpu_dma_latency
> >> mechanism acts on all states and cores, e.g. to disable sleep state 2
> >> and all deeper states of the ladder governor on core #0, use:
> >>     echo 1>/sys/devices/system/cpu/cpu0/cpuidle/state2/disable
> >>
> >> BTW: To analyze how much time a core spent in a specific sleep state,
> >> simply look repeatedly at the "time" variable of a core's sleep
state, e.g.
> for core #0:
> >> # for i in /sys/devices/system/cpu/cpu0/cpuidle/state[0-4]
> >>   >  do
> >>   >  echo -e "`cat $i/name`:\t`cat $i/time`"
> >>   >  done
> >> POLL:	1342984105
> >> C1-IVB:	737109
> >> C3-IVB:	3852451
> >> C6-IVB:	1702683112
> >> C7-IVB:	4366946606
> >> While cyclictest is running with /dev/cpu_dma_latency set to 0, only
> >> the POLL state times are increasing.
> > Thanks for the reply! As I wrote in my reply to Frank, I'm not
> > completely sure if P states are correctly implemented in our system.
> > We're using a custom BIOS for our custom board, and while P states do
> > show up and are modifiable (I've currently installed the
> > userspace-governor, and am manually setting the clock-frequency to the
> > lowest possible at startup), our board guy is not sure that changing
> > it actually has any effect on the processor. Yay...:/
> Sorry, but this is a complete misunderstanding. C states and P states
are very
> different
(http://software.intel.com/en-us/blogs/2008/03/12/c-states-and-
> p-states-are-very-different).
> The point made by Frank and my answer related to C states (aka sleep
> states) a processor may enter when idle. The Linux C state interface is
called
> cpuidle. The P states you are referring to are related to the
processor's clock
> frequency that may be lowered at any time irrespective of idle state.
The
> Linux P state interface is called cpufreq. P states generally affect the
real-
> time capabilities in a linear and proportional way, e.g. a CPU board
with a
> worst-case latency of 100 microseconds at 1 GHz will have a latency of
> approximately 200 microseconds at 500 MHz.
> When idle and in deep C state, however, the processor may take several
> milliseconds to wake up and answer an asynchronous external event. This
is
> why deep C states should be disabled in a real-time system that may
become
> idle. And this is why I mentioned the new interface that allows to
individually
> disable a particular sleep state of a particular processor core to
ensure its
> deterministic behavior while the other cores still may run in
energy-saving
> mode.
>
> Hope this helps,
> Carsten.

Ah, I actually came to suspect as much after I posted the above. But I
presume C-states also need to be supported in the BIOS? We have a new
revision of our board (including an updated BIOS) coming along soon(ish),
so I'll try having a further look at things once I get my hands on it.

In any case, thank you for the nice explanation - much appreciated!

Best regards,
Simon Falsig