[PATCH] rtla/timerlat: Lock the memory of the user-space threads

[PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Daniel Wagner]

With -u, the latency timerlat reports runs until the measuring thread
is back in user space. Nothing keeps that thread's memory resident,
so under memory pressure the kernel reclaims its code, stack and libc
pages, and the page faults on the way back to user space count as
latency.

On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
timerlat reported up to 40 ms for the user-space threads, which took
283 major faults, while the kernel threads' latency stayed below
1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
0.9 ms with it.

With this change timerlat reports 0.9 ms for the user-space threads
under the same load, and they take no major faults.

Lock the user-space threads' memory before they start measuring, as
cyclictest -m does.

Signed-off-by: Daniel Wagner <wagi@monom.org>
---
 tools/tracing/rtla/src/timerlat_u.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
index c80edaf07b00..42aaf1d7f216 100644
--- a/tools/tracing/rtla/src/timerlat_u.c
+++ b/tools/tracing/rtla/src/timerlat_u.c
@@ -15,6 +15,7 @@
 #include <pthread.h>
 #include <sys/wait.h>
 #include <sys/prctl.h>
+#include <sys/mman.h>
 
 #include "common.h"
 #include "timerlat_u.h"
@@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
 		}
 	}
 
+	/*
+	 * Keep the measuring process in memory: the latency it reports
+	 * runs until it is back in user space, and a page fault on the way
+	 * (its code, stack or libc pages reclaimed under memory pressure)
+	 * would count as latency. cyclictest does the same with -m.
+	 */
+	if (mlockall(MCL_CURRENT | MCL_FUTURE))
+		err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
+
 	/*
 	 * This is the tool's loop. If you want to use as base for your own tool...
 	 * go ahead.

---
base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
change-id: 20261112-rtla-mlock-2a572f10533e

Best regards,
--  
Daniel Wagner <wagi@monom.org>

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Tomas Glozar]

On Thu, Nov 12, 2026 at 10:26 PM Daniel Wagner <wagi@monom.org> wrote:
>
> With -u, the latency timerlat reports runs until the measuring thread
> is back in user space. Nothing keeps that thread's memory resident,
> so under memory pressure the kernel reclaims its code, stack and libc
> pages, and the page faults on the way back to user space count as
> latency.
>
> On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
> timerlat reported up to 40 ms for the user-space threads, which took
> 283 major faults, while the kernel threads' latency stayed below
> 1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
> 0.9 ms with it.
>

Yes, this is a known problem. I have just not seen the timerlat user
thread faulting in practice, maybe because I use rtla mostly on
servers with an "absurd" amount of RAM. Or maybe I just have not
noticed it yet.

> With this change timerlat reports 0.9 ms for the user-space threads
> under the same load, and they take no major faults.
>
> Lock the user-space threads' memory before they start measuring, as
> cyclictest -m does.
>

Thank you for measuring this.

> Signed-off-by: Daniel Wagner <wagi@monom.org>
> ---
>  tools/tracing/rtla/src/timerlat_u.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
> index c80edaf07b00..42aaf1d7f216 100644
> --- a/tools/tracing/rtla/src/timerlat_u.c
> +++ b/tools/tracing/rtla/src/timerlat_u.c
> @@ -15,6 +15,7 @@
>  #include <pthread.h>
>  #include <sys/wait.h>
>  #include <sys/prctl.h>
> +#include <sys/mman.h>
>
>  #include "common.h"
>  #include "timerlat_u.h"
> @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
>                 }
>         }
>
> +       /*
> +        * Keep the measuring process in memory: the latency it reports
> +        * runs until it is back in user space, and a page fault on the way
> +        * (its code, stack or libc pages reclaimed under memory pressure)
> +        * would count as latency. cyclictest does the same with -m.
> +        */
> +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> +

Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?

We can add it as a boolean field into struct timerlat_params and pass
down to struct timerlat_u_params. See e.g. 2ff4464e3283
("rtla/osnoise: Add IPI tracking cmdline option") for reference.

>         /*
>          * This is the tool's loop. If you want to use as base for your own tool...
>          * go ahead.
>
> ---
> base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
> change-id: 20261112-rtla-mlock-2a572f10533e
>
> Best regards,
> --
> Daniel Wagner <wagi@monom.org>
>
>

Thanks,

Tomas

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Sebastian Andrzej Siewior]

On 2026-11-13 10:30:33 [+0100], Tomas Glozar wrote:
> > --- a/tools/tracing/rtla/src/timerlat_u.c
> > +++ b/tools/tracing/rtla/src/timerlat_u.c
> > @@ -15,6 +15,7 @@
> >  #include <pthread.h>
> >  #include <sys/wait.h>
> >  #include <sys/prctl.h>
> > +#include <sys/mman.h>
> >
> >  #include "common.h"
> >  #include "timerlat_u.h"
> > @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
> >                 }
> >         }
> >
> > +       /*
> > +        * Keep the measuring process in memory: the latency it reports
> > +        * runs until it is back in user space, and a page fault on the way
> > +        * (its code, stack or libc pages reclaimed under memory pressure)
> > +        * would count as latency. cyclictest does the same with -m.
> > +        */
> > +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> > +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> > +
> 
> Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?
> 
> We can add it as a boolean field into struct timerlat_params and pass
> down to struct timerlat_u_params. See e.g. 2ff4464e3283
> ("rtla/osnoise: Add IPI tracking cmdline option") for reference.

Why make this an option? I would prefer a RT task doing the right thing
right away. cyclictest has a history of several options to do the right
thing. Is the reason why lock should be disabled?

Sebastian