Re: [PATCH v3 0/2] allow creating thread with real-time priority

[Thomas Monjalon <thomas@monjalon.net>]

26/10/2023 18:50, Morten Brørup:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > Sent: Thursday, 26 October 2023 18.07
> > 
> > 26/10/2023 17:54, Bruce Richardson:
> > > On Thu, Oct 26, 2023 at 04:59:51PM +0200, Morten Brørup wrote:
> > > > > From: Morten Brørup [mailto:mb@smartsharesystems.com]
> > > > > Sent: Thursday, 26 October 2023 16.50
> > > > >
> > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > Sent: Thursday, 26 October 2023 16.31
> > > > > >
> > > > > > 26/10/2023 16:08, Morten Brørup:
> > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > Sent: Thursday, 26 October 2023 16.05
> > > > > > > >
> > > > > > > > 26/10/2023 15:57, Morten Brørup:
> > > > > > > > > > From: Morten Brørup [mailto:mb@smartsharesystems.com]
> > > > > > > > > > Sent: Thursday, 26 October 2023 15.45
> > > > > > > > > >
> > > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > > Sent: Thursday, 26 October 2023 15.37
> > > > > > > > > > >
> > > > > > > > > > > 25/10/2023 18:31, Thomas Monjalon:
> > > > > > > > > > > > Real-time thread priority was been forbidden on Unix
> > > > > > > > > > > > because of problems they can cause.
> > > > > > > > > > > > Warnings and helpers are added to avoid deadlocks,
> > > > > > > > > > > > so real-time can be allowed on all systems.
> > > > > > > > > > >
> > > > > > > > > > > Unit test is failing:
> > > > > > > > > > > DPDK:fast-tests / threads_autotest      TIMEOUT 600.01
> > s
> > > > > > > > > > >
> > > > > > > > > > > It is seen in only 1 target (maybe the failure
> > occurence is
> > > > > random):
> > > > > > > > > > >   Debian 11 (Buster) (ARM) | PASS
> > > > > > > > > > >   Fedora 37 (ARM)          | PASS
> > > > > > > > > > >   CentOS Stream 9 (ARM)    | FAIL
> > > > > > > > > > >   Fedora 38 (ARM)          | PASS
> > > > > > > > > > >   Fedora 38 (ARM Clang)    | PASS
> > > > > > > > > > >   Ubuntu 20.04 (ARM)       | PASS
> > > > > > > > > > >
> > > > > > > > > > > I need to send a v4 with new implementation and better
> > comments.
> > > > > > > > > > > The Unix sleep will be upgraded from 1 ns to 1 us in
> > case it makes
> > > > > a
> > > > > > > > > > > difference.
> > > > > > > > > >
> > > > > > > > > > It will not make a difference. The kernel will go
> > through the
> > > > > sleeping
> > > > > > > > steps,
> > > > > > > > > > then wake up again and see the real-time thread is ready
> > to run, and
> > > > > > then
> > > > > > > > > > immediately schedule it.
> > > > > > > > > >
> > > > > > > > > > For testing purposes, consider sleeping 10 milliseconds
> > or something
> > > > > > > > > > significant like that.
> > > > > > > > >
> > > > > > > > > A bit more details...
> > > > > > > > >
> > > > > > > > > In our recent tests, nanosleep() itself took around 50 us.
> > So you need
> > > > > > to
> > > > > > > > sleep longer than that for your thread not to be runnable
> > when the
> > > > > > nanosleep()
> > > > > > > > wakes up again, because 50 us has already passed in
> > "nanosleep
> > > > > overhead".
> > > > > > > > > 10 milliseconds provides plenty of margin, and corresponds
> > to 10
> > > > > jiffies
> > > > > > on
> > > > > > > > a 1000 Hz kernel. (I don't know if it makes any difference
> > for the
> > > > > kernel
> > > > > > > > scheduler if the timer crosses a jiffy border or not.)
> > > > > > > >
> > > > > > > > 10 ms looks like an eternity.
> > > > > > >
> > > > > > > Agree. It is only for functional testing, not for production!
> > > > > >
> > > > > > Realtime thread won't make any sense if we have to insert a long
> > sleep.
> > > > >
> > > > > It seems David came to our rescue here!
> > > > >
> > > > > I have just tried running our test again with
> > prctl(PR_SET_TIMERSLACK) of 1
> > > > > ns, and the nanosleep(1 ns) delay dropped from ca. 50 us to ca.
> > 2.5 us.
> > > > >
> > > > > The timeout parameter to epoll_wait() is in milliseconds, which is
> > useless for
> > > > > low-latency.
> > > > > Perhaps real-time threads can be used with epoll() combined with
> > timerfd for
> > > > > nanosecond resolution timeout.
> > > >
> > > > Or epoll_pwait2(), which has nanosecond resolution timeout.
> > > >
> > > > Unfortunately, rte_epoll_wait() is not an experimental API anymore,
> > so we cannot change its timeout parameter from milliseconds to micro- or
> > nanoseconds. We would have to introduce a new API for this.
> > > >
> > >
> > > Just an idea - can we change the timeout parameter to float rather
> > than int,
> > > and then use function versioning for backward compatibility for any
> > > binaries passing int?
> > > That way the actual meaning of the parameter doesn't change, but it
> > still
> > > allows sub-millisecond values (all-be-it with some loss of accuracy
> > due to
> > > float).
> 
> Too exotic for my taste. I would rather introduce rte_epoll_wait_ns() with timeout in nanoseconds than pass a float.
> 
> > 
> > Sorry I'm not following why you want to use rte_epoll_wait()?
> 
> I don't have experience with it yet, but it seems to be the official DPDK API for blocking I/O system call.
> 
> > 
> > If the realtime thread has some blocking system calls,
> > no sleep is needed I think.
> 
> Correct.
> 
> > For average realtime thread, I suggest the API
> > rte_thread_yield_realtime()
> > which could wait for 1 ms or less by default.
> 
> If we introduce a yield() function, it should yield according to the O/S scheduling policy, e.g. the rest of the time slice allocated to the thread by the O/S scheduler (although that might not be available for real-time prioritized threads in Linux). I don't think we can make it O/S agnostic.
> 
> I don't think it should wait a fixed amount of time - we already have rte_delay_us_sleep() for that.
> 
> In my experiments with power saving, I ended up with a varying sleep duration, depending on traffic load. The l3fwd-power example also uses a varying sleep duration depending on traffic load.

I feel you lost the goal of this: schedule other threads (especially kernel threads).
So we don't care about the sleep duration at all,
except we want it minimal while allowing scheduling.

> > For smaller sleep, we can use PR_SET_TIMERSLACK and
> > rte_delay_us_sleep().
> 
> Agree.
> 
> > If we provide an API for PR_SET_TIMERSLACK, we could adapt the duration
> > of rte_thread_yield_realtime() dynamically after calling prctl().
> > 
> 
> I'm not sure exposing an API for PR_SET_TIMERSLACK is the right solution.
> 
> I would rather have the EAL set the timer slack to minimum (1 ns) at EAL initialization. An EAL command line parameter could be added to change the default from 1 ns.
> 
> Also, something similar needs to be done for Windows.

Windows should be fine with Sleep(0).