Thank you Luca for that wonderful explanation of the work that you are
doing. Meanwhile, i am not an expert in RT linux Kernel hard/soft. So, i
would prefer experts to comment on your proposal (Darren/others, will
you !!). I will be more interested in the test cases, automating them,
and make sure that affected parties can make the best use of it through
LTP. And the test cases helps in improving the kernel features that you
are developing. Slowly i would be working with you to integrate the test
suites to LTP, as we move ahead with comments/reviews from the experts.

On Tue, 2008-07-01 at 20:14 +0200, Luca Recchia wrote:
> Dear All,
>
> just some details about our test.
>
>     We're actually dealing with Hard Real-Time problems (on x86) and
> we need a way
> to measure latencies in the IRQ handling kernel flow-paths.
> We do not only need to measure the global (from the rise of an
> interrupt to its
> complete management) IRQ handling latency, but also the intermediate
> (inter-function) latency values.
>
> Our main objective is to evaluate Hard Real-Time feautures like
> predictability
> and time latency of IRQ response and IRQ management flow.
>
>
>     The kernel patch and the modules we have developed are in this
> direction. The
> patch allows us to measure (with little overhead) the execution times
> between the main functions of the IRQ management flow. The modules
> read the
> values we measure and place them into proc fs (in a more readable
> manner).
>
>     All time-values that are taken in the test are measured by reading
> the TSC
> register. Both the user space task and the kernel IRQ management
> flow-path should
> be bound on one processor, so to avoid discrepancies between different
> TSC reads
> on different CPUs.
>
>
>     The test is composed by a user space application which is waiting
> (with a
> blocking read on /dev/rtc) for RTC interrupts. We re-program the RTC
> device to
> launch interrupts at fixed frequency.
>
>     First fo all the user space test application set the real time
> priority and
> the real time policy SCHED_FIFO of the user task, then re-program the
> RTC device
> to launch interrupts at fixed frequency, and go to sleep waiting for
> RTC interrupt.
>
> Immediately after being woke up, the user space test application
> performs a TSC read,
> which constitutes the final measure of the IRQ management flow.
> Furthermore this read
> helps in telling kernel test latencies from spurious kernel
> inter-function latencies
> that may occur before the beginning of the test.
> When the test is over the user apllication print the TSC values on the
> console.
>
>     The patch we introduce in the kernel is able to record up to
> SAMPLES samples
> of inter-function latencies throughout the kernel path activated by
> RTC
> interrupt management flow-path.
>
> When the test is over, it is possible to read the inter-function
> latencies
> (sampled in the kernel) through the proc fs, using "delta_read_irq"
> and "irq_latencies_tsc" modules.
>
>     The "delta_read_irq" module is used to read the inter-function
> latencies values
> as the max, min, and total TSC cycles elapsed during the measured
> intervals.
>
>     With the "irq_latencies_tsc" module it is possible to read the TSC
> values taken
> inside IRQ flow-path functions.
> By comparing these values with those obtained in the user space
> application, one
> can evaluate the whole latencies in the IRQ management flow (from the
> arrival
> of an hardware interrupt to the system, to the beginning of the test
> application).
>
>     From the analisys of these results we can estimate the worst-case
> latency and
> the periodic jitter of an interrupt management flow-path.
> We are also able to evaluate the required (re)scheduling effort and
> inter-function
> latencies.
>
> The execution-jitter of the kernel paths we analyze is our measure to
> evaluate the
> predictability of "end-to-end" IRQ management (from the arrival of an
> hardware interrupt
> to the system, to the beginning of the test application).
>
> The measured jitter is the standard deviation of the sampled
> latencies,
> relatively to their mean.
>
>     At the moment, we make use of a spreadsheet to evaluate
> std.deviation, worst,
> best and average inter-function and end-to-end latencies. We build
> these results
> using 1000 sampled latency values (these values are obtained through
> the module
> "irq_latencies_tsc" and through the user space application).
>
> It is important to note that the samples we measure in the kernel are
> in clock cycle
> unit and we have to transform them into time unit. The resolution of
> the measure
> heavily depends on the CPU frequency and on the time spent in reading
> the TSC.
> We have estimate for our measure a base definition of 10 ns.
>
>     The test is not yet automated, and doesn't respect LTP's
> requirement. At the moment it is just a quite usable tool that is

Even if it does not use LTP specific libraries, it would be OK for it to
be initially integrated to LTP. But the major(and Primary) requirement
is to automate them. And we should be able to get the output/results
automatically and should be able to analyze them in human-readable
format.

As one of my goals is to bring all Linux testing efforts under unified
umbrella, i would rather like this test suite to me maintained from LTP.
I would like to have an option where it would run this test suite along
with the default LTP together, or, each one separately depending on user
choice.
Seems that our engagement will be long.

Regards--
Subrata

>  being mainly use
> as a base comparison between Vanilla kernel and RT kernel project
> (http://www.eu.kernel.org/pub/linux/kernel/projects/rt/).
>
>
>
> Regards,
>
> Luca Recchia
>

>