Re: CONFIG_CRITICAL_IRQSOFF_TIMING

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Gerrit Binnenmars]

> Datum: 29/10/07 02:09 PM
> Van: "Steven Rostedt" 
> Aan: "Gerrit Binnenmars" 
> CC: linux-rt-users@vger.kernel.org
> Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> 
> 
> --
> On Sun, 28 Oct 2007, Gerrit Binnenmars wrote:
> 
> > Hello,
> >
> > Can someone send me a .config file that shows the options needed to get the interrupt off
> > timing values in preempt_max_latency, please. I only see a value of 0 in this file
> > I am using kernel 2.6.22.1 with the corresponding rt patch.
> >
> > Thanks in advance,
> 
> Make sure you have event trace and latency tracing as well as
> "Interrupts-off critical section latency timing".
> 
> Then you need to echo 0 > /proc/sys/kernel/preempt_max_latency.
> 
Hello Steve,

Thanks for your quick response, but the preempt_max_latency remains 0. Also the latency_hist shows no samples taken.
Can you or someone else have a look at the corresponding CONFIG options?

CONFIG_LTT_PROBE_LOCKING=m
# CONFIG_LTT_PROBE_STACK is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
# CONFIG_PROVE_LOCKING is not set
CONFIG_LOCKDEP=y
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_LOCKDEP is not set
CONFIG_TRACE_IRQFLAGS=y
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
CONFIG_PREEMPT_TRACE=y
CONFIG_EVENT_TRACE=y
CONFIG_FUNCTION_TRACE=y
# CONFIG_WAKEUP_TIMING is not set
CONFIG_LATENCY_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
CONFIG_CRITICAL_IRQSOFF_TIMING=y
CONFIG_INTERRUPT_OFF_HIST=y
CONFIG_CRITICAL_TIMING=y
CONFIG_DEBUG_TRACE_IRQFLAGS=y
CONFIG_LATENCY_TIMING=y
CONFIG_CRITICAL_LATENCY_HIST=y
CONFIG_LATENCY_HIST=y
CONFIG_MCOUNT=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_LIST is not set
CONFIG_FRAME_POINTER=y
CONFIG_FORCED_INLINING=y
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
CONFIG_DEBUG_RODATA=y
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y

Greetings Gerrit

> -- Steve
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> 

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Daniel Walker]

On Tue, 2007-10-30 at 21:52 +0100, Gerrit Binnenmars wrote:

> CONFIG_LTT_PROBE_LOCKING=m
> # CONFIG_LTT_PROBE_STACK is not set

I wonder which kernel your using? I don't recall LTT being part of real
time in the past .. What the architecture too ?

> CONFIG_CRITICAL_IRQSOFF_TIMING=y
> CONFIG_INTERRUPT_OFF_HIST=y
> CONFIG_CRITICAL_TIMING=y
> CONFIG_DEBUG_TRACE_IRQFLAGS=y
> CONFIG_LATENCY_TIMING=y

You might try disabling the histogram option temporarily, it may have be
effecting things (not sure how tho) ..

Daniel

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Steven Rostedt]

I somehow was removed from the CC list, so I didn't see it.

<rant>I hate lists that think it's best to only post to the list and tell
you to take all people off the CC list. Those are lists that are run by
people that have way too much time on their hands to be reading mailing
lists all day. Between LKML, linux-rt-users, virt, lguest, and a ton of
internal lists,  I can't keep up! When in a coding frenzy, I may
not read lists for days.</rant>


--
On Tue, 30 Oct 2007, Gerrit Binnenmars wrote:

> > Datum: 29/10/07 02:09 PM
> > Van: "Steven Rostedt"
> > Aan: "Gerrit Binnenmars"
> > CC: linux-rt-users@vger.kernel.org
> > Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> >
> >
> > --
> > On Sun, 28 Oct 2007, Gerrit Binnenmars wrote:
> >
> > > Hello,
> > >
> > > Can someone send me a .config file that shows the options needed to get the interrupt off
> > > timing values in preempt_max_latency, please. I only see a value of 0 in this file
> > > I am using kernel 2.6.22.1 with the corresponding rt patch.
> > >
> > > Thanks in advance,
> >
> > Make sure you have event trace and latency tracing as well as
> > "Interrupts-off critical section latency timing".
> >
> > Then you need to echo 0 > /proc/sys/kernel/preempt_max_latency.
> >
> Hello Steve,
>
> Thanks for your quick response, but the preempt_max_latency remains 0. Also the latency_hist shows no samples taken.
> Can you or someone else have a look at the corresponding CONFIG options?
>
> CONFIG_LTT_PROBE_LOCKING=m
> # CONFIG_LTT_PROBE_STACK is not set
>
> #
> # Kernel hacking
> #

[...]

> CONFIG_INTERRUPT_OFF_HIST=y
> CONFIG_CRITICAL_LATENCY_HIST=y
> CONFIG_LATENCY_HIST=y

Turn off all histograms. They will kill any measurements you are taking.
And yes, you will not get the output from latency_trace that you are
after.

-- Steve

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Gerrit Binnenmars]

> Datum: 31/10/07 01:17 AM
> Van: "Steven Rostedt" 
> Aan: "Gerrit Binnenmars" 
> CC: linux-rt-users@vger.kernel.org
> Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> 
> 
> --
> On Tue, 30 Oct 2007, Gerrit Binnenmars wrote:
> 
> > > Datum: 29/10/07 02:09 PM
> > > Van: "Steven Rostedt"
> > > Aan: "Gerrit Binnenmars"
> > > CC: linux-rt-users@vger.kernel.org
> > > Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> > >
> > >
> > > --
> > > On Sun, 28 Oct 2007, Gerrit Binnenmars wrote:
> > >
> > > > Hello,
> > > >
> > > > Can someone send me a .config file that shows the options needed to get the interrupt off
> > > > timing values in preempt_max_latency, please. I only see a value of 0 in this file
> > > > I am using kernel 2.6.22.1 with the corresponding rt patch.
> > > >
> > > > Thanks in advance,
> > >
> > > Make sure you have event trace and latency tracing as well as
> > > "Interrupts-off critical section latency timing".
> > >
> > > Then you need to echo 0 > /proc/sys/kernel/preempt_max_latency.
> > >
> > Hello Steve,
> >
> > Thanks for your quick response, but the preempt_max_latency remains 0. Also the latency_hist shows no samples taken.
> > Can you or someone else have a look at the corresponding CONFIG options?
> >
> > CONFIG_LTT_PROBE_LOCKING=m
> > # CONFIG_LTT_PROBE_STACK is not set
> >
> > #
> > # Kernel hacking
> > #
> 
> [...]
> 
> > CONFIG_INTERRUPT_OFF_HIST=y
> > CONFIG_CRITICAL_LATENCY_HIST=y
> > CONFIG_LATENCY_HIST=y
> 
> Turn off all histograms. They will kill any measurements you are taking.
> And yes, you will not get the output from latency_trace that you are
> after.

Steven and Daniel, I tried your suggestion with the following resulsts:
CRITICAL_IRQSOFF_TIMING: preempt_max_latency remains 0 even under load, 
                         cyclictest -b is able to store a trace in /proc/latency_trace
CRITICAL_PREEMPT_TIMING: kernel won´t start at all (vmlinuz is read from tftp server but nothing afterwards)
WAKEUP_TIMING:           is working as expected.

So switching off the histograms is an improvement. For the moment, I can live with this situation 
(especially the cyclictest -b option is very useful),
but I wonder if anyone else has the same problems?

Regards, Gerrit

> 
> -- Steve
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
>

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Steven Rostedt]

--
> > [...]
> >
> > > CONFIG_INTERRUPT_OFF_HIST=y
> > > CONFIG_CRITICAL_LATENCY_HIST=y
> > > CONFIG_LATENCY_HIST=y
> >
> > Turn off all histograms. They will kill any measurements you are taking.
> > And yes, you will not get the output from latency_trace that you are
> > after.
>
> Steven and Daniel, I tried your suggestion with the following resulsts:
> CRITICAL_IRQSOFF_TIMING: preempt_max_latency remains 0 even under load,
>                          cyclictest -b is able to store a trace in /proc/latency_trace
> CRITICAL_PREEMPT_TIMING: kernel won´t start at all (vmlinuz is read from tftp server but nothing afterwards)
> WAKEUP_TIMING:           is working as expected.
>
> So switching off the histograms is an improvement. For the moment, I can live with this situation
> (especially the cyclictest -b option is very useful),
> but I wonder if anyone else has the same problems?

critcal irqs off timings wont start with wakeup timing on. You need to
disable that. But that you can do from the command line.

echo 0 > /proc/sys/kernel/wakeup_timing

-- Steve

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Gerrit Binnenmars]

> Datum: 30/10/07 10:38 PM
> Van: "Daniel Walker" 
> Aan: gerritbinnenmars@wanadoo.nl
> CC: linux-rt-users@vger.kernel.org
> Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> 
> On Tue, 2007-10-30 at 22:26 +0100, Gerrit Binnenmars wrote:
> 
> > Daniel, I first applied the rt patchset and then the LTTng patch 
> > (http://ltt.polymtl.ca/lttng/patch-2.6.22.1-rt4-lttng-0.9.10.tar.gz).
> > Then LTT can be selected in the Instrumentation part of the config file. 
> > further ltt-control is needed (http://ltt.polymtl.ca/lttng/ltt-control-0.43-10082007.tar.gz)
> > 
> > The viewer is only working on i386 but that is not really a problem.
> > 
> > So, I applied the available patches without doing anything special.
> 
> Ok .
> 
> > Is LTT causing troubles with the preempt_max_latency options?
> 
> I doubt it .. I haven't used the histogram options in a month or so ..
> But if I remember correctly the value of preempt_max_latency is suppose
> to remain at zero .. Usually preempt_max_latency is used to prune off
> uninteresting latencies (ones under the max) but if you have the
> histogram option turn on then all latencies points need to be recorded.
> The data can be viewed in in /proc/latency_hist/ .. In 2.6.22 there is
> no way to reset the histograms for preempt off, or interrupts off
> timing.

Daniel,
Thanks for your suggestion. The latency_hist shows a max long value for the minimum latency. 
All other values remain 0. I think the tracer is not running at all, but I am not sure. 
I will remove the latency_hist option tomorrow and recheck preempt_max_latency.

Gerrit
> 
> Daniel
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> 

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Gerrit Binnenmars]

> Datum: 30/10/07 10:11 PM
> Van: "Daniel Walker" 
> Aan: gerritbinnenmars@wanadoo.nl
> CC: linux-rt-users@vger.kernel.org
> Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> 
> On Tue, 2007-10-30 at 22:09 +0100, Gerrit Binnenmars wrote:
> > > Datum: 30/10/07 10:02 PM
> > > Van: "Daniel Walker" 
> > > Aan: gerritbinnenmars@wanadoo.nl
> > > CC: linux-rt-users@vger.kernel.org
> > > Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> > > 
> > > On Tue, 2007-10-30 at 21:52 +0100, Gerrit Binnenmars wrote:
> > > 
> > > > CONFIG_LTT_PROBE_LOCKING=m
> > > > # CONFIG_LTT_PROBE_STACK is not set
> > > 
> > > I wonder which kernel your using? I don't recall LTT being part of real
> > > time in the past .. What the architecture too ?
> > 
> > Hello Daniel,
> > I use version 2.6.22.1-rt4, see the following link for LTT compatibility
> > http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-lttv-compatibility.html
> > 
> > I have used LTT on both i386 an on a PowerPC (MPC8555) architecture.
> > It is a great help in understanding real-time behaviour.
> 
> Did you add LTT into your kernel on your own?
> 
> Daniel
> 
> 
>
Daniel, I first applied the rt patchset and then the LTTng patch 
(http://ltt.polymtl.ca/lttng/patch-2.6.22.1-rt4-lttng-0.9.10.tar.gz).
Then LTT can be selected in the Instrumentation part of the config file. 
further ltt-control is needed (http://ltt.polymtl.ca/lttng/ltt-control-0.43-10082007.tar.gz)

The viewer is only working on i386 but that is not really a problem.

So, I applied the available patches without doing anything special.
Is LTT causing troubles with the preempt_max_latency options?

Gerrit 

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Daniel Walker]

On Tue, 2007-10-30 at 22:26 +0100, Gerrit Binnenmars wrote:

> Daniel, I first applied the rt patchset and then the LTTng patch 
> (http://ltt.polymtl.ca/lttng/patch-2.6.22.1-rt4-lttng-0.9.10.tar.gz).
> Then LTT can be selected in the Instrumentation part of the config file. 
> further ltt-control is needed (http://ltt.polymtl.ca/lttng/ltt-control-0.43-10082007.tar.gz)
> 
> The viewer is only working on i386 but that is not really a problem.
> 
> So, I applied the available patches without doing anything special.

Ok .

> Is LTT causing troubles with the preempt_max_latency options?

I doubt it .. I haven't used the histogram options in a month or so ..
But if I remember correctly the value of preempt_max_latency is suppose
to remain at zero .. Usually preempt_max_latency is used to prune off
uninteresting latencies (ones under the max) but if you have the
histogram option turn on then all latencies points need to be recorded.
The data can be viewed in in /proc/latency_hist/ .. In 2.6.22 there is
no way to reset the histograms for preempt off, or interrupts off
timing.

Daniel

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Gerrit Binnenmars]

> Datum: 30/10/07 10:02 PM
> Van: "Daniel Walker" 
> Aan: gerritbinnenmars@wanadoo.nl
> CC: linux-rt-users@vger.kernel.org
> Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> 
> On Tue, 2007-10-30 at 21:52 +0100, Gerrit Binnenmars wrote:
> 
> > CONFIG_LTT_PROBE_LOCKING=m
> > # CONFIG_LTT_PROBE_STACK is not set
> 
> I wonder which kernel your using? I don't recall LTT being part of real
> time in the past .. What the architecture too ?

Hello Daniel,
I use version 2.6.22.1-rt4, see the following link for LTT compatibility
http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-lttv-compatibility.html

I have used LTT on both i386 an on a PowerPC (MPC8555) architecture.
It is a great help in understanding real-time behaviour.

Gerrit

> 
> > CONFIG_CRITICAL_IRQSOFF_TIMING=y
> > CONFIG_INTERRUPT_OFF_HIST=y
> > CONFIG_CRITICAL_TIMING=y
> > CONFIG_DEBUG_TRACE_IRQFLAGS=y
> > CONFIG_LATENCY_TIMING=y
> 
> You might try disabling the histogram option temporarily, it may have be
> effecting things (not sure how tho) ..
> 
> Daniel
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> 
> 

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Daniel Walker]

On Tue, 2007-10-30 at 22:09 +0100, Gerrit Binnenmars wrote:
> > Datum: 30/10/07 10:02 PM
> > Van: "Daniel Walker" 
> > Aan: gerritbinnenmars@wanadoo.nl
> > CC: linux-rt-users@vger.kernel.org
> > Onderwerp : Re: CONFIG_CRITICAL_IRQSOFF_TIMING
> > 
> > On Tue, 2007-10-30 at 21:52 +0100, Gerrit Binnenmars wrote:
> > 
> > > CONFIG_LTT_PROBE_LOCKING=m
> > > # CONFIG_LTT_PROBE_STACK is not set
> > 
> > I wonder which kernel your using? I don't recall LTT being part of real
> > time in the past .. What the architecture too ?
> 
> Hello Daniel,
> I use version 2.6.22.1-rt4, see the following link for LTT compatibility
> http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-lttv-compatibility.html
> 
> I have used LTT on both i386 an on a PowerPC (MPC8555) architecture.
> It is a great help in understanding real-time behaviour.

Did you add LTT into your kernel on your own?

Daniel

CONFIG_CRITICAL_IRQSOFF_TIMING

[Gerrit Binnenmars]

Hello,

Can someone send me a .config file that shows the options needed to get the interrupt off 
timing values in preempt_max_latency, please. I only see a value of 0 in this file
I am using kernel 2.6.22.1 with the corresponding rt patch.

Thanks in advance,

Gerrit Binnenmars 

Re: CONFIG_CRITICAL_IRQSOFF_TIMING

[Steven Rostedt]

--
On Sun, 28 Oct 2007, Gerrit Binnenmars wrote:

> Hello,
>
> Can someone send me a .config file that shows the options needed to get the interrupt off
> timing values in preempt_max_latency, please. I only see a value of 0 in this file
> I am using kernel 2.6.22.1 with the corresponding rt patch.
>
> Thanks in advance,

Make sure you have event trace and latency tracing as well as
"Interrupts-off critical section latency timing".

Then you need to echo 0 > /proc/sys/kernel/preempt_max_latency.

-- Steve