hwlat latency readings [WAS: IRC Alert]

hwlat latency readings [WAS: IRC Alert]

[Jon Masters]

On Fri, 2009-07-03 at 11:54 +0000, jbl wrote:

> [11:52:41] <jbl:#linux-rt> +jonmasters: ping
> [11:53:27] <jbl> +how long should be max latencies seen with hwlat detector?

It's a US holiday, so I'll reply by email as I'm not really around for
IRC, but I don't want to forget to add something to this - especially as
people seem to be trying to also track down new latencies in RT itself.
I don't think these are SMI related - though now you can check.

The default for the latency detector is to regard any unexplained time
interval over (greater than, not greater than or equal to) 10us as a
latency worthy of some regard. But you can also configure it down to 1us
since anything over 0us represents some kind of interruption - it's all
down to what threshold you actually care about (few people notice 1us).

Anyway. How high should it go? Theoretically, that is an open ended
question. But we have seen in practice latencies up to many milliseconds
or even higher - I don't expect you'll go over 100ms and in reality, I
don't expect it to be anything like that, maybe hundreds of us on a
modern enough system. But the range is really down to whatever crap the
BIOS is doing when it's taking your CPU cycles away, and that could
literally be anything (reformulating the coke recipe isn't unlikely).

If you're seeing something irregular, for widely long periods of time it
is less likely to be the kind of SMIs we see, which do tend to be
semi-predictable as they rely upon counters for most of the
non-asynchronous (e.g. hardware emulation) activities. But if you see
something short many times per second, or something longer on a
predictable schedule then it is quite likely to be an SMI hitting you.

Jon.

P.S. I know there's a slight bug in the wake_up handling in hwlat that
is causing some issues - I will have a fix available next week.

Re: hwlat latency readings [WAS: IRC Alert]

[Jan Blunck]

On Fri, Jul 3, 2009 at 7:50 PM, Jon Masters<jonathan@jonmasters.org> wrote:
> On Fri, 2009-07-03 at 11:54 +0000, jbl wrote:
>
>> [11:52:41] <jbl:#linux-rt> +jonmasters: ping
>> [11:53:27] <jbl> +how long should be max latencies seen with hwlat detector?
>
> It's a US holiday, so I'll reply by email as I'm not really around for
> IRC, but I don't want to forget to add something to this - especially as
> people seem to be trying to also track down new latencies in RT itself.
> I don't think these are SMI related - though now you can check.

First I wondered if I should reply via twitter or blog ... just to be
more 2.0 ;)

> The default for the latency detector is to regard any unexplained time
> interval over (greater than, not greater than or equal to) 10us as a
> latency worthy of some regard. But you can also configure it down to 1us
> since anything over 0us represents some kind of interruption - it's all
> down to what threshold you actually care about (few people notice 1us).
>
> Anyway. How high should it go? Theoretically, that is an open ended
> question. But we have seen in practice latencies up to many milliseconds
> or even higher - I don't expect you'll go over 100ms and in reality, I
> don't expect it to be anything like that, maybe hundreds of us on a
> modern enough system. But the range is really down to whatever crap the
> BIOS is doing when it's taking your CPU cycles away, and that could
> literally be anything (reformulating the coke recipe isn't unlikely).
>
> If you're seeing something irregular, for widely long periods of time it
> is less likely to be the kind of SMIs we see, which do tend to be
> semi-predictable as they rely upon counters for most of the
> non-asynchronous (e.g. hardware emulation) activities. But if you see
> something short many times per second, or something longer on a
> predictable schedule then it is quite likely to be an SMI hitting you.

I was wondering about the "samples recorded" value. First I thought
there is something wrong since that value was always zero on my test
machine. I expected that this this value would reflect somehow
duration / window length. I guess the documentation could be extended
to explain a bit how to work with various settings of sample window
and width to find out the exact interval of the disturbance.

Thanks,
Jan

Re: hwlat latency readings [WAS: IRC Alert]

[Jon Masters]

On Tue, 2009-07-07 at 16:03 +0200, Jan Blunck wrote:

> I was wondering about the "samples recorded" value. First I thought
> there is something wrong since that value was always zero on my test
> machine. I expected that this this value would reflect somehow
> duration / window length. I guess the documentation could be extended
> to explain a bit how to work with various settings of sample window
> and width to find out the exact interval of the disturbance.

Good point :) I'll take a look when I'm back from Japan.

Jon.