Why is real time pinging not possible with RT kernels?

Why is real time pinging not possible with RT kernels?

[rolf.freitag]

Hi,

I tried ping as a minimal latency test, e. g.

ionice -c3 -p $$
renice +19 -p $$
ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost

but when cyclictest shows a worst-case latency of 40 mikroseconds,
ping shows more than 10,000 (after 1 day run).
I tried different kernels, e. g.
SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
but with the same result, low cyclictest values, high ping values.
What is the reason?

Regards,

Rolf

Re: Why is real time pinging not possible with RT kernels?

[Chris Friesen]

On 06/10/2017 12:13 PM, rolf.freitag@email.de wrote:
> Hi,
>
> I tried ping as a minimal latency test, e. g.
>
> ionice -c3 -p $$
> renice +19 -p $$
> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>
> but when cyclictest shows a worst-case latency of 40 mikroseconds,
> ping shows more than 10,000 (after 1 day run).
> I tried different kernels, e. g.
> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
> but with the same result, low cyclictest values, high ping values.
> What is the reason?

That's a great question, I'm curious what responses you get.

It might be useful to use a custom ping client (or remove the -q option and 
parse the output) so you could track the latencies and generate a histogram to 
see if that 10ms latency was a single outlier or whether it happened fairly 
frequently.

Chris

Re: Why is real time pinging not possible with RT kernels?

[Corey Minyard]

On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
> Hi,
>
> I tried ping as a minimal latency test, e. g.
>
> ionice -c3 -p $$
> renice +19 -p $$
> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>
> but when cyclictest shows a worst-case latency of 40 mikroseconds,
> ping shows more than 10,000 (after 1 day run).
> I tried different kernels, e. g.
> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
> but with the same result, low cyclictest values, high ping values.
> What is the reason?

Since no one else has answered, I guess I will.

Creating a real-time *system* is hard work.  Just running a program on a
real-time OS does not magically make it a real-time system.  You really
have to know what you are doing.  I suggest you read "POSIX.4
Programmers Guide: Programming for the Real World" for an intro to
the subject.  It's fairly close to what Linux does, though there are things
that Linux does beyond what is in there.

A few notes on what you have done:

  * ionice doesn't affect network I/O.
  * renice does not set real-time priorities.
  * You don't do anything with memory locking.  What happens
    if memory in your program gets paged out then is required?
  * Something on the other end of the ping is running, too.  It
    also needs to be real-time, too, and it's not by default.

I'd suggest you need to spend some time learning before you attempt
something like this.

-corey

Re: Why is real time pinging not possible with RT kernels?

[Peter C. Wallace]

On Mon, 12 Jun 2017, Corey Minyard wrote:

> Date: Mon, 12 Jun 2017 07:57:18 -0500
> From: Corey Minyard <cminyard@mvista.com>
> To: rolf.freitag@email.de, linux-rt-users@vger.kernel.org
> Subject: Re: Why is real time pinging not possible with RT kernels?
> 
> On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
>> Hi,
>> 
>> I tried ping as a minimal latency test, e. g.
>> 
>> ionice -c3 -p $$
>> renice +19 -p $$
>> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>> 
>> but when cyclictest shows a worst-case latency of 40 mikroseconds,
>> ping shows more than 10,000 (after 1 day run).
>> I tried different kernels, e. g.
>> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
>> but with the same result, low cyclictest values, high ping values.
>> What is the reason?
>
> Since no one else has answered, I guess I will.
>
> Creating a real-time *system* is hard work.  Just running a program on a
> real-time OS does not magically make it a real-time system.  You really
> have to know what you are doing.  I suggest you read "POSIX.4
> Programmers Guide: Programming for the Real World" for an intro to
> the subject.  It's fairly close to what Linux does, though there are things
> that Linux does beyond what is in there.
>
> A few notes on what you have done:
>
> * ionice doesn't affect network I/O.
> * renice does not set real-time priorities.
> * You don't do anything with memory locking.  What happens
>   if memory in your program gets paged out then is required?
> * Something on the other end of the ping is running, too.  It
>   also needs to be real-time, too, and it's not by default.
>
> I'd suggest you need to spend some time learning before you attempt
> something like this.
>
> -corey
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Lowly user perspective:

You can get ping to be approximately realtime by setting its priority but it 
will do awful things to your system load since ping was not written to be RT 
capable.

Heres 1/2 a day of 100 Hz pinging:


pcw@pcw-G41M-Combo:~$ sudo chrt 99 ping -i .010 -q localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
^C
--- localhost ping statistics ---
4311151 packets transmitted, 4311151 received, 0% packet loss, time 43111566ms
rtt min/avg/max/mdev = 0.005/0.007/0.133/0.005 ms

pcw@pcw-G41M-Combo:~$ uname -a
Linux pcw-G41M-Combo 4.1.38-rt46 #1 SMP PREEMPT RT Sat Mar 11 10:51:18 PST 
2017 i686 i686 i686 GNU/Linux


Peter Wallace

Re: Why is real time pinging not possible with RT kernels?

[Chris Friesen]

On 06/12/2017 06:57 AM, Corey Minyard wrote:
> On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
>> Hi,
>>
>> I tried ping as a minimal latency test, e. g.
>>
>> ionice -c3 -p $$
>> renice +19 -p $$
>> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>>
>> but when cyclictest shows a worst-case latency of 40 mikroseconds,
>> ping shows more than 10,000 (after 1 day run).
>> I tried different kernels, e. g.
>> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
>> but with the same result, low cyclictest values, high ping values.
>> What is the reason?
>
> Since no one else has answered, I guess I will.
>
> Creating a real-time *system* is hard work.  Just running a program on a
> real-time OS does not magically make it a real-time system.  You really
> have to know what you are doing.  I suggest you read "POSIX.4
> Programmers Guide: Programming for the Real World" for an intro to
> the subject.  It's fairly close to what Linux does, though there are things
> that Linux does beyond what is in there.
>
> A few notes on what you have done:
>
>   * ionice doesn't affect network I/O.
>   * renice does not set real-time priorities.

Good point...the OP probably wants something like "sudo chrt <prio> <cmd> ...."

>   * You don't do anything with memory locking.  What happens
>     if memory in your program gets paged out then is required?

I suppose this would require a custom "ping" client to ensure all necessary 
memory is allocated and locked before starting the actual ping.

>   * Something on the other end of the ping is running, too.  It
>     also needs to be real-time, too, and it's not by default.

The standard ICMP ping would be handled by the kernel.  With linux-rt I guess 
you'd have the ethernet interrupt thread involved as well as ksoftirqd?

Chris

Re: Why is real time pinging not possible with RT kernels?

[Corey Minyard]

On 06/12/2017 11:45 AM, Chris Friesen wrote:
> On 06/12/2017 06:57 AM, Corey Minyard wrote:
>> On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
>>> Hi,
>>>
>>> I tried ping as a minimal latency test, e. g.
>>>
>>> ionice -c3 -p $$
>>> renice +19 -p $$
>>> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>>>
>>> but when cyclictest shows a worst-case latency of 40 mikroseconds,
>>> ping shows more than 10,000 (after 1 day run).
>>> I tried different kernels, e. g.
>>> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
>>> but with the same result, low cyclictest values, high ping values.
>>> What is the reason?
>>
>> Since no one else has answered, I guess I will.
>>
>> Creating a real-time *system* is hard work.  Just running a program on a
>> real-time OS does not magically make it a real-time system.  You really
>> have to know what you are doing.  I suggest you read "POSIX.4
>> Programmers Guide: Programming for the Real World" for an intro to
>> the subject.  It's fairly close to what Linux does, though there are 
>> things
>> that Linux does beyond what is in there.
>>
>> A few notes on what you have done:
>>
>>   * ionice doesn't affect network I/O.
>>   * renice does not set real-time priorities.
>
> Good point...the OP probably wants something like "sudo chrt <prio> 
> <cmd> ...."
>
>>   * You don't do anything with memory locking.  What happens
>>     if memory in your program gets paged out then is required?
>
> I suppose this would require a custom "ping" client to ensure all 
> necessary memory is allocated and locked before starting the actual ping.
>

Yes, I don't know if there is a way to do this without doing it in the 
application.

>>   * Something on the other end of the ping is running, too.  It
>>     also needs to be real-time, too, and it's not by default.
>
> The standard ICMP ping would be handled by the kernel.  With linux-rt 
> I guess you'd have the ethernet interrupt thread involved as well as 
> ksoftirqd?
>

I didn't look at what would be required, but those are likely candidates.

I didn't mean the above as a comprehensive list or anything like that.  
I just
wanted to drive home the point that this requires end-to-end thinking
and careful design.  It's hard.

-corey

Re: Why is real time pinging not possible with RT kernels?

[Sebastian Andrzej Siewior]

On 2017-06-12 07:57:18 [-0500], Corey Minyard wrote:
> A few notes on what you have done:
> 
>  * ionice doesn't affect network I/O.
>  * renice does not set real-time priorities.
>  * You don't do anything with memory locking.  What happens
>    if memory in your program gets paged out then is required?
>  * Something on the other end of the ping is running, too.  It
>    also needs to be real-time, too, and it's not by default.

You forgot to mention that the real-time kernel is not automatically
overclocking the software in terms of performance (but providing
determinism).

> I'd suggest you need to spend some time learning before you attempt
> something like this.
> 
> -corey

Sebastian

Re: Why is real time pinging not possible with RT kernels?

[Chris Friesen]

On 06/13/2017 05:58 AM, Sebastian Andrzej Siewior wrote:
> On 2017-06-12 07:57:18 [-0500], Corey Minyard wrote:
>> A few notes on what you have done:
>>
>>   * ionice doesn't affect network I/O.
>>   * renice does not set real-time priorities.
>>   * You don't do anything with memory locking.  What happens
>>     if memory in your program gets paged out then is required?
>>   * Something on the other end of the ping is running, too.  It
>>     also needs to be real-time, too, and it's not by default.
>
> You forgot to mention that the real-time kernel is not automatically
> overclocking the software in terms of performance (but providing
> determinism).

Pinging localhost *should* be reasonably quick.  It's basically a test of the 
scheduler and context switching (once all the memory is locked in place).

Chris

Re: Why is real time pinging not possible with RT kernels?

[Corey Minyard]

On 06/13/2017 11:06 AM, Chris Friesen wrote:
> On 06/13/2017 05:58 AM, Sebastian Andrzej Siewior wrote:
>> On 2017-06-12 07:57:18 [-0500], Corey Minyard wrote:
>>> A few notes on what you have done:
>>>
>>>   * ionice doesn't affect network I/O.
>>>   * renice does not set real-time priorities.
>>>   * You don't do anything with memory locking.  What happens
>>>     if memory in your program gets paged out then is required?
>>>   * Something on the other end of the ping is running, too. It
>>>     also needs to be real-time, too, and it's not by default.
>>
>> You forgot to mention that the real-time kernel is not automatically
>> overclocking the software in terms of performance (but providing
>> determinism).
>
> Pinging localhost *should* be reasonably quick.  It's basically a test 
> of the scheduler and context switching (once all the memory is locked 
> in place).
>
> Chris

That's not what Sebastian is talking about.  Some people believe that
running a task on an RT kernel will make it run faster.  RT is not about
making your program run faster.  It's about, if you have a properly
designed application, having your program meet certain latency
requirements.  RT generally reduces throughput.

Pinging localhost should, of course, be reasonably quick, but we
are talking about latency guarantees.   Generally being reasonably
quick is not the same thing as guaranteeing latency every single
time.

To the original poster's credit, they were talking about latency, not
throughput, so I didn't address the performance question.  They
seemed to be asking the right question, they just didn't know how
it all works on Linux.

-corey

Re: Why is real time pinging not possible with RT kernels?

[rolf.freitag]

Hi,
  
Gesendet: Montag, 12. Juni 2017 um 18:45 Uhr
Von: "Chris Friesen" <chris.friesen@windriver.com>
...
>> A few notes on what you have done:
>>
>> * ionice doesn't affect network I/O.
>> * renice does not set real-time priorities.
>
> Good point...the OP probably wants something like "sudo chrt <prio> <cmd> ...."

Ok, i added chrt in the script:

ionice -c3 -p $$
renice +19 -p $$
chrt -f -p 99 $$
ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost

and now the worst case latency is about twice the value measured with cyclictest:

27893223 packets transmitted, 27893223 received, 0% packet loss, time 27921273ms
rtt min/avg/max/mdev = 0.022/0.023/0.280/0.006 ms


>> * You don't do anything with memory locking. What happens
>> if memory in your program gets paged out then is required?

I use much RAM and no swap, so i have implicit memory locking.

Regards,

Rolf

Re: Why is real time pinging not possible with RT kernels?

[Corey Minyard]

On 06/17/2017 03:37 AM, rolf.freitag@email.de wrote:
> Hi,
>    
> Gesendet: Montag, 12. Juni 2017 um 18:45 Uhr
> Von: "Chris Friesen" <chris.friesen@windriver.com>
> ...
>>> A few notes on what you have done:
>>>
>>> * ionice doesn't affect network I/O.
>>> * renice does not set real-time priorities.
>> Good point...the OP probably wants something like "sudo chrt <prio> <cmd> ...."
> Ok, i added chrt in the script:
>
> ionice -c3 -p $$
> renice +19 -p $$
> chrt -f -p 99 $$
> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>
> and now the worst case latency is about twice the value measured with cyclictest:
>
> 27893223 packets transmitted, 27893223 received, 0% packet loss, time 27921273ms
> rtt min/avg/max/mdev = 0.022/0.023/0.280/0.006 ms

That is pretty good, really.  However, I'm guessing you aren't doing 
anything
else on the system.

When we test RT, we do so while running a heavy I/O, network, and CPU load.
If you do that, you may see very different results, though I doubt 
cyclictest
will move much.

That's what RT is about.  If someone decides to log on to your system 
running
an RT application, and compiles a kernel or something, your RT application
still meets its deadlines.

>
>>> * You don't do anything with memory locking. What happens
>>> if memory in your program gets paged out then is required?
> I use much RAM and no swap, so i have implicit memory locking.

That is not implicit memory locking.  The kernel is free to swap out 
anything
that is backed by disk, like your executable code.  Lots of memory helps,
but does not offer a guarantee.  Lack of swap doesn't matter (much).

-corey

> Regards,
>
> Rolf
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html