* Re: Why is real time pinging not possible with RT kernels?
2017-06-12 12:57 ` Corey Minyard
@ 2017-06-12 13:17 ` Peter C. Wallace
2017-06-12 16:45 ` Chris Friesen
2017-06-13 11:58 ` Sebastian Andrzej Siewior
2 siblings, 0 replies; 11+ messages in thread
From: Peter C. Wallace @ 2017-06-12 13:17 UTC (permalink / raw)
To: Corey Minyard; +Cc: rolf.freitag, linux-rt-users
On Mon, 12 Jun 2017, Corey Minyard wrote:
> Date: Mon, 12 Jun 2017 07:57:18 -0500
> From: Corey Minyard <cminyard@mvista.com>
> To: rolf.freitag@email.de, linux-rt-users@vger.kernel.org
> Subject: Re: Why is real time pinging not possible with RT kernels?
>
> On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
>> Hi,
>>
>> I tried ping as a minimal latency test, e. g.
>>
>> ionice -c3 -p $$
>> renice +19 -p $$
>> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>>
>> but when cyclictest shows a worst-case latency of 40 mikroseconds,
>> ping shows more than 10,000 (after 1 day run).
>> I tried different kernels, e. g.
>> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
>> but with the same result, low cyclictest values, high ping values.
>> What is the reason?
>
> Since no one else has answered, I guess I will.
>
> Creating a real-time *system* is hard work. Just running a program on a
> real-time OS does not magically make it a real-time system. You really
> have to know what you are doing. I suggest you read "POSIX.4
> Programmers Guide: Programming for the Real World" for an intro to
> the subject. It's fairly close to what Linux does, though there are things
> that Linux does beyond what is in there.
>
> A few notes on what you have done:
>
> * ionice doesn't affect network I/O.
> * renice does not set real-time priorities.
> * You don't do anything with memory locking. What happens
> if memory in your program gets paged out then is required?
> * Something on the other end of the ping is running, too. It
> also needs to be real-time, too, and it's not by default.
>
> I'd suggest you need to spend some time learning before you attempt
> something like this.
>
> -corey
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Lowly user perspective:
You can get ping to be approximately realtime by setting its priority but it
will do awful things to your system load since ping was not written to be RT
capable.
Heres 1/2 a day of 100 Hz pinging:
pcw@pcw-G41M-Combo:~$ sudo chrt 99 ping -i .010 -q localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
^C
--- localhost ping statistics ---
4311151 packets transmitted, 4311151 received, 0% packet loss, time 43111566ms
rtt min/avg/max/mdev = 0.005/0.007/0.133/0.005 ms
pcw@pcw-G41M-Combo:~$ uname -a
Linux pcw-G41M-Combo 4.1.38-rt46 #1 SMP PREEMPT RT Sat Mar 11 10:51:18 PST
2017 i686 i686 i686 GNU/Linux
Peter Wallace
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-12 12:57 ` Corey Minyard
2017-06-12 13:17 ` Peter C. Wallace
@ 2017-06-12 16:45 ` Chris Friesen
2017-06-12 17:16 ` Corey Minyard
2017-06-17 8:37 ` rolf.freitag
2017-06-13 11:58 ` Sebastian Andrzej Siewior
2 siblings, 2 replies; 11+ messages in thread
From: Chris Friesen @ 2017-06-12 16:45 UTC (permalink / raw)
To: Corey Minyard, rolf.freitag, linux-rt-users
On 06/12/2017 06:57 AM, Corey Minyard wrote:
> On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
>> Hi,
>>
>> I tried ping as a minimal latency test, e. g.
>>
>> ionice -c3 -p $$
>> renice +19 -p $$
>> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>>
>> but when cyclictest shows a worst-case latency of 40 mikroseconds,
>> ping shows more than 10,000 (after 1 day run).
>> I tried different kernels, e. g.
>> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
>> but with the same result, low cyclictest values, high ping values.
>> What is the reason?
>
> Since no one else has answered, I guess I will.
>
> Creating a real-time *system* is hard work. Just running a program on a
> real-time OS does not magically make it a real-time system. You really
> have to know what you are doing. I suggest you read "POSIX.4
> Programmers Guide: Programming for the Real World" for an intro to
> the subject. It's fairly close to what Linux does, though there are things
> that Linux does beyond what is in there.
>
> A few notes on what you have done:
>
> * ionice doesn't affect network I/O.
> * renice does not set real-time priorities.
Good point...the OP probably wants something like "sudo chrt <prio> <cmd> ...."
> * You don't do anything with memory locking. What happens
> if memory in your program gets paged out then is required?
I suppose this would require a custom "ping" client to ensure all necessary
memory is allocated and locked before starting the actual ping.
> * Something on the other end of the ping is running, too. It
> also needs to be real-time, too, and it's not by default.
The standard ICMP ping would be handled by the kernel. With linux-rt I guess
you'd have the ethernet interrupt thread involved as well as ksoftirqd?
Chris
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-12 16:45 ` Chris Friesen
@ 2017-06-12 17:16 ` Corey Minyard
2017-06-17 8:37 ` rolf.freitag
1 sibling, 0 replies; 11+ messages in thread
From: Corey Minyard @ 2017-06-12 17:16 UTC (permalink / raw)
To: Chris Friesen, rolf.freitag, linux-rt-users
On 06/12/2017 11:45 AM, Chris Friesen wrote:
> On 06/12/2017 06:57 AM, Corey Minyard wrote:
>> On 06/10/2017 01:13 PM, rolf.freitag@email.de wrote:
>>> Hi,
>>>
>>> I tried ping as a minimal latency test, e. g.
>>>
>>> ionice -c3 -p $$
>>> renice +19 -p $$
>>> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>>>
>>> but when cyclictest shows a worst-case latency of 40 mikroseconds,
>>> ping shows more than 10,000 (after 1 day run).
>>> I tried different kernels, e. g.
>>> SMP PREEMPT RT Debian 4.6.4-1~bpo8+1 (2016-08-11) x86_64
>>> but with the same result, low cyclictest values, high ping values.
>>> What is the reason?
>>
>> Since no one else has answered, I guess I will.
>>
>> Creating a real-time *system* is hard work. Just running a program on a
>> real-time OS does not magically make it a real-time system. You really
>> have to know what you are doing. I suggest you read "POSIX.4
>> Programmers Guide: Programming for the Real World" for an intro to
>> the subject. It's fairly close to what Linux does, though there are
>> things
>> that Linux does beyond what is in there.
>>
>> A few notes on what you have done:
>>
>> * ionice doesn't affect network I/O.
>> * renice does not set real-time priorities.
>
> Good point...the OP probably wants something like "sudo chrt <prio>
> <cmd> ...."
>
>> * You don't do anything with memory locking. What happens
>> if memory in your program gets paged out then is required?
>
> I suppose this would require a custom "ping" client to ensure all
> necessary memory is allocated and locked before starting the actual ping.
>
Yes, I don't know if there is a way to do this without doing it in the
application.
>> * Something on the other end of the ping is running, too. It
>> also needs to be real-time, too, and it's not by default.
>
> The standard ICMP ping would be handled by the kernel. With linux-rt
> I guess you'd have the ethernet interrupt thread involved as well as
> ksoftirqd?
>
I didn't look at what would be required, but those are likely candidates.
I didn't mean the above as a comprehensive list or anything like that.
I just
wanted to drive home the point that this requires end-to-end thinking
and careful design. It's hard.
-corey
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-12 16:45 ` Chris Friesen
2017-06-12 17:16 ` Corey Minyard
@ 2017-06-17 8:37 ` rolf.freitag
2017-06-17 12:10 ` Corey Minyard
1 sibling, 1 reply; 11+ messages in thread
From: rolf.freitag @ 2017-06-17 8:37 UTC (permalink / raw)
To: linux-rt-users
Hi,
Gesendet: Montag, 12. Juni 2017 um 18:45 Uhr
Von: "Chris Friesen" <chris.friesen@windriver.com>
...
>> A few notes on what you have done:
>>
>> * ionice doesn't affect network I/O.
>> * renice does not set real-time priorities.
>
> Good point...the OP probably wants something like "sudo chrt <prio> <cmd> ...."
Ok, i added chrt in the script:
ionice -c3 -p $$
renice +19 -p $$
chrt -f -p 99 $$
ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
and now the worst case latency is about twice the value measured with cyclictest:
27893223 packets transmitted, 27893223 received, 0% packet loss, time 27921273ms
rtt min/avg/max/mdev = 0.022/0.023/0.280/0.006 ms
>> * You don't do anything with memory locking. What happens
>> if memory in your program gets paged out then is required?
I use much RAM and no swap, so i have implicit memory locking.
Regards,
Rolf
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-17 8:37 ` rolf.freitag
@ 2017-06-17 12:10 ` Corey Minyard
0 siblings, 0 replies; 11+ messages in thread
From: Corey Minyard @ 2017-06-17 12:10 UTC (permalink / raw)
To: rolf.freitag, linux-rt-users
On 06/17/2017 03:37 AM, rolf.freitag@email.de wrote:
> Hi,
>
> Gesendet: Montag, 12. Juni 2017 um 18:45 Uhr
> Von: "Chris Friesen" <chris.friesen@windriver.com>
> ...
>>> A few notes on what you have done:
>>>
>>> * ionice doesn't affect network I/O.
>>> * renice does not set real-time priorities.
>> Good point...the OP probably wants something like "sudo chrt <prio> <cmd> ...."
> Ok, i added chrt in the script:
>
> ionice -c3 -p $$
> renice +19 -p $$
> chrt -f -p 99 $$
> ping -q -s 28 -l 1 -p 0f1e2d3c4b5a6978 -i 0.001 localhost
>
> and now the worst case latency is about twice the value measured with cyclictest:
>
> 27893223 packets transmitted, 27893223 received, 0% packet loss, time 27921273ms
> rtt min/avg/max/mdev = 0.022/0.023/0.280/0.006 ms
That is pretty good, really. However, I'm guessing you aren't doing
anything
else on the system.
When we test RT, we do so while running a heavy I/O, network, and CPU load.
If you do that, you may see very different results, though I doubt
cyclictest
will move much.
That's what RT is about. If someone decides to log on to your system
running
an RT application, and compiles a kernel or something, your RT application
still meets its deadlines.
>
>>> * You don't do anything with memory locking. What happens
>>> if memory in your program gets paged out then is required?
> I use much RAM and no swap, so i have implicit memory locking.
That is not implicit memory locking. The kernel is free to swap out
anything
that is backed by disk, like your executable code. Lots of memory helps,
but does not offer a guarantee. Lack of swap doesn't matter (much).
-corey
> Regards,
>
> Rolf
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-12 12:57 ` Corey Minyard
2017-06-12 13:17 ` Peter C. Wallace
2017-06-12 16:45 ` Chris Friesen
@ 2017-06-13 11:58 ` Sebastian Andrzej Siewior
2017-06-13 16:06 ` Chris Friesen
2 siblings, 1 reply; 11+ messages in thread
From: Sebastian Andrzej Siewior @ 2017-06-13 11:58 UTC (permalink / raw)
To: Corey Minyard; +Cc: rolf.freitag, linux-rt-users
On 2017-06-12 07:57:18 [-0500], Corey Minyard wrote:
> A few notes on what you have done:
>
> * ionice doesn't affect network I/O.
> * renice does not set real-time priorities.
> * You don't do anything with memory locking. What happens
> if memory in your program gets paged out then is required?
> * Something on the other end of the ping is running, too. It
> also needs to be real-time, too, and it's not by default.
You forgot to mention that the real-time kernel is not automatically
overclocking the software in terms of performance (but providing
determinism).
> I'd suggest you need to spend some time learning before you attempt
> something like this.
>
> -corey
Sebastian
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-13 11:58 ` Sebastian Andrzej Siewior
@ 2017-06-13 16:06 ` Chris Friesen
2017-06-13 16:53 ` Corey Minyard
0 siblings, 1 reply; 11+ messages in thread
From: Chris Friesen @ 2017-06-13 16:06 UTC (permalink / raw)
To: Sebastian Andrzej Siewior, Corey Minyard; +Cc: rolf.freitag, linux-rt-users
On 06/13/2017 05:58 AM, Sebastian Andrzej Siewior wrote:
> On 2017-06-12 07:57:18 [-0500], Corey Minyard wrote:
>> A few notes on what you have done:
>>
>> * ionice doesn't affect network I/O.
>> * renice does not set real-time priorities.
>> * You don't do anything with memory locking. What happens
>> if memory in your program gets paged out then is required?
>> * Something on the other end of the ping is running, too. It
>> also needs to be real-time, too, and it's not by default.
>
> You forgot to mention that the real-time kernel is not automatically
> overclocking the software in terms of performance (but providing
> determinism).
Pinging localhost *should* be reasonably quick. It's basically a test of the
scheduler and context switching (once all the memory is locked in place).
Chris
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Why is real time pinging not possible with RT kernels?
2017-06-13 16:06 ` Chris Friesen
@ 2017-06-13 16:53 ` Corey Minyard
0 siblings, 0 replies; 11+ messages in thread
From: Corey Minyard @ 2017-06-13 16:53 UTC (permalink / raw)
To: Chris Friesen, Sebastian Andrzej Siewior; +Cc: rolf.freitag, linux-rt-users
On 06/13/2017 11:06 AM, Chris Friesen wrote:
> On 06/13/2017 05:58 AM, Sebastian Andrzej Siewior wrote:
>> On 2017-06-12 07:57:18 [-0500], Corey Minyard wrote:
>>> A few notes on what you have done:
>>>
>>> * ionice doesn't affect network I/O.
>>> * renice does not set real-time priorities.
>>> * You don't do anything with memory locking. What happens
>>> if memory in your program gets paged out then is required?
>>> * Something on the other end of the ping is running, too. It
>>> also needs to be real-time, too, and it's not by default.
>>
>> You forgot to mention that the real-time kernel is not automatically
>> overclocking the software in terms of performance (but providing
>> determinism).
>
> Pinging localhost *should* be reasonably quick. It's basically a test
> of the scheduler and context switching (once all the memory is locked
> in place).
>
> Chris
That's not what Sebastian is talking about. Some people believe that
running a task on an RT kernel will make it run faster. RT is not about
making your program run faster. It's about, if you have a properly
designed application, having your program meet certain latency
requirements. RT generally reduces throughput.
Pinging localhost should, of course, be reasonably quick, but we
are talking about latency guarantees. Generally being reasonably
quick is not the same thing as guaranteeing latency every single
time.
To the original poster's credit, they were talking about latency, not
throughput, so I didn't address the performance question. They
seemed to be asking the right question, they just didn't know how
it all works on Linux.
-corey
^ permalink raw reply [flat|nested] 11+ messages in thread