Re: Help regarding Cyclictest.

Re: Help regarding Cyclictest.

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 5439 bytes --]


On Fri, 2 Jan 2015 15:02:11 +0530
Ipsit Kumar <kumaripsit1@gmail.com> wrote:

> Hi sir,
> Good Afternoon.
> Thank you for your reply.
> 
> I am facing some issues with cyclictest.
> Basically i am using this tool on ARM platform, specifically on TCI6614 evm.
> I am seeing strange spike in the cyclictest result.
> 
> 
> 
> 
> 
> *root # ./cyclictest -t1 -p80 -i 10000 -l 10000# /dev/cpu_dma_latency set
> to 0uspolicy: fifo: loadavg: 0.70 0.48 0.61 1/434 15283           T: 0
> (15226) P:80 I:10000 C:  10000 Min:      5 Act:    9 Avg:   17 Max:    5797*
> 
> When I am plotting the Histogram, I see the latency to go beyond 3ms once.
> I am unable to figure out what is the reason behind this.
> Checked with the kernel configuration and found that the CPU Frequency
> scaling is Disabled.
> No power management features are enabled.
> 
> It would be a great help, if you could point out something I am doing
> wrong.
> 
> Thank you.

Sorry for the slow response time, but I've been swamped with things
here at work following the Christmas break. 

First, I don't see anything offhand that you're doing wrong. What
version of the realtime kernel are you running? 

I think your best bet may be to take this to the Linux Realtime users
mailing list (in CC). 

To get a sense of where things are going wrong, you can run cyclictest
with the -b option, like this:

$ cyclictest -t1 -p80 -i 10000 -l 10000 -b 500

This will enable the ftrace system and will cause the event tracepoints
to be enabled. When a latency is measured greater than the specified
argument (in this case 500 microseconds), cyclictest will first stop
ftrace and will then exit. You can at that point use the trace-cmd
command to save off the trace buffers:

$ trace-cmd extract

This will generate the file trace.bin which is a binary
representation of the kernel trace buffers. You can then generate a
latency trace report using:

$ trace-cmd report -l

This will give you an idea of what code paths were taken prior to the
latency spike. From here it's a matter of figuring out what happened
and how to prevent it from happening.

> 
> On Mon, Dec 29, 2014 at 4:06 AM, Clark Williams <williams@redhat.com> wrote:
> 
> > You won't be able to use cyclictest to determine those. Cyclictest
> > measures scheduler latency (the amount of time from an event occurrence
> > to when a thread is scheduled because of that event). Scheduler latency
> > has interrupt latency and context switch time as components.
> >
> > Also, realize that interrupt latency and context switch time are going
> > to be close to constants, as well as very small components of the
> > overall scheduler latency. If you're going to measure those, you should
> > look at ftrace tracepoints or the 'perf' utility.
> >
> > On Wed, 24 Dec 2014 13:17:03 +0530
> > Ipsit Kumar <kumaripsit1@gmail.com> wrote:
> >
> > > Hello sir,
> > > Its really great to use this tool.
> > > I was trying to know the interrupt Latency and the context switch
> > latency.
> > > Would you like to give me pointer to trace those using cyclic test and
> > > the way to measure the performance using the the ARM performance counter
> > ?
> > >
> > > Thanks a lot for your help sir.
> > >
> > > On Wed, Dec 24, 2014 at 9:14 AM, Ipsit Kumar <kumaripsit1@gmail.com>
> > wrote:
> > >
> > > > Thanks a lot for your reply.
> > > >
> > > >
> > > >
> > > > On Tue, Dec 23, 2014 at 11:56 PM, Clark Williams <williams@redhat.com>
> > > > wrote:
> > > >
> > > >> On Tue, 23 Dec 2014 15:52:18 +0530
> > > >> Ipsit Kumar <kumaripsit1@gmail.com> wrote:
> > > >>
> > > >> > hello sir,
> > > >> > I am using the cyclictest tool to do some latency test on my linux
> > rt
> > > >> > kernel.
> > > >> > I am unable to understand the statistics that are being printed.
> > > >> > I want to know what is min max and avg that are being printed ?
> > > >> > Would be a great help to hear from you sir.
> > > >> >
> > > >> > Thank you.
> > > >> >
> > > >> > --
> > > >> > Regards:
> > > >> > Ipsit Kumar Senapati
> > > >> > *.*
> > > >>
> > > >> Cyclictest measures what I call "scheduler latency". It measures the
> > > >> time between when the program should wakeup and when it actually does.
> > > >> Some pseudo code:
> > > >>
> > > >>         t1 = clock_gettime()
> > > >>         while running
> > > >>                 clock_nanosleep(interval)
> > > >>                 t2 = clock_gettime()
> > > >>                 delta = t2 - (t1 + interval)
> > > >>                 t1 = t2
> > > >>
> > > >> So in a perfect world we would wakeup at time t1+interval, but of
> > > >> course we're not perfect and there's system overhead to take into
> > > >> account. The delta value is the scheduler latency.
> > > >>
> > > >> The min value printed out is the lowest scheduler latency recorded,
> > > >> while the max value is the largest scheduler latency recorded for a
> > > >> particular core, both reported in microseconds. The avg value is a
> > > >> running average of the delta values for a particular core.
> > > >>
> > > >> Clark
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Regards:
> > > > Ipsit Kumar Senapati
> > > > *.*
> > > >
> > >
> > >
> > >
> > > --
> > > Regards:
> > > Ipsit Kumar Senapati
> > > *.*
> >
> 
> 
> 
> -- 
> Regards:
> Ipsit Kumar Senapati
> *.*

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Help regarding Cyclictest.

[Steven Rostedt]

On Tue, 6 Jan 2015 15:26:49 -0600
Clark Williams <williams@redhat.com> wrote:

> 
> On Fri, 2 Jan 2015 15:02:11 +0530
> Ipsit Kumar <kumaripsit1@gmail.com> wrote:
> 
> > Hi sir,
> > Good Afternoon.
> > Thank you for your reply.
> > 
> > I am facing some issues with cyclictest.
> > Basically i am using this tool on ARM platform, specifically on TCI6614 evm.
> > I am seeing strange spike in the cyclictest result.

What kernel are you running?

> > 
> > 
> > 
> > 
> > 
> > *root # ./cyclictest -t1 -p80 -i 10000 -l 10000# /dev/cpu_dma_latency set
> > to 0uspolicy: fifo: loadavg: 0.70 0.48 0.61 1/434 15283           T: 0
> > (15226) P:80 I:10000 C:  10000 Min:      5 Act:    9 Avg:   17 Max:    5797*
> > 
> > When I am plotting the Histogram, I see the latency to go beyond 3ms once.
> > I am unable to figure out what is the reason behind this.
> > Checked with the kernel configuration and found that the CPU Frequency
> > scaling is Disabled.
> > No power management features are enabled.
> > 
> > It would be a great help, if you could point out something I am doing
> > wrong.
> > 
> > Thank you.
> 
> Sorry for the slow response time, but I've been swamped with things
> here at work following the Christmas break. 
> 
> First, I don't see anything offhand that you're doing wrong. What
> version of the realtime kernel are you running? 
> 
> I think your best bet may be to take this to the Linux Realtime users
> mailing list (in CC). 
> 
> To get a sense of where things are going wrong, you can run cyclictest
> with the -b option, like this:
> 
> $ cyclictest -t1 -p80 -i 10000 -l 10000 -b 500

You can also download the latest trace-cmd with the profile feature and
try this:

 # trace-cmd profile --stderr -F -c cyclictest -t1 -p80 -i 10000 -l
 10000 2> profile.out

And when you see the spike, hit ctrl-C and see the output.


Get the latest at:

 git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

-- Steve

> 
> This will enable the ftrace system and will cause the event tracepoints
> to be enabled. When a latency is measured greater than the specified
> argument (in this case 500 microseconds), cyclictest will first stop
> ftrace and will then exit. You can at that point use the trace-cmd
> command to save off the trace buffers:
> 
> $ trace-cmd extract
> 
> This will generate the file trace.bin which is a binary
> representation of the kernel trace buffers. You can then generate a
> latency trace report using:
> 
> $ trace-cmd report -l
> 
> This will give you an idea of what code paths were taken prior to the
> latency spike. From here it's a matter of figuring out what happened
> and how to prevent it from happening.
> 

Re: Help regarding Cyclictest.

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Fri, 9 Jan 2015 09:37:07 +0530
Ipsit Kumar <kumaripsit1@gmail.com> wrote:

> Thanks for the reply.
> I think I forgot to mention about the kernel I am running. It is Linux
> Kernel 3.3 with preemption enabled.
> Could this be a problem ? And do you think this kind of behavior is
> expected if I use a RT Patch.
> 

Large spikes are not unusual running on a stock kernel. If you are
trying to prevent these spikes, you will need to run an RT kernel

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]