[PATCH] rtla/timerlat: Lock the memory of the user-space threads

[PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Daniel Wagner]

With -u, the latency timerlat reports runs until the measuring thread
is back in user space. Nothing keeps that thread's memory resident,
so under memory pressure the kernel reclaims its code, stack and libc
pages, and the page faults on the way back to user space count as
latency.

On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
timerlat reported up to 40 ms for the user-space threads, which took
283 major faults, while the kernel threads' latency stayed below
1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
0.9 ms with it.

With this change timerlat reports 0.9 ms for the user-space threads
under the same load, and they take no major faults.

Lock the user-space threads' memory before they start measuring, as
cyclictest -m does.

Signed-off-by: Daniel Wagner <wagi@monom.org>
---
 tools/tracing/rtla/src/timerlat_u.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
index c80edaf07b00..42aaf1d7f216 100644
--- a/tools/tracing/rtla/src/timerlat_u.c
+++ b/tools/tracing/rtla/src/timerlat_u.c
@@ -15,6 +15,7 @@
 #include <pthread.h>
 #include <sys/wait.h>
 #include <sys/prctl.h>
+#include <sys/mman.h>
 
 #include "common.h"
 #include "timerlat_u.h"
@@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
 		}
 	}
 
+	/*
+	 * Keep the measuring process in memory: the latency it reports
+	 * runs until it is back in user space, and a page fault on the way
+	 * (its code, stack or libc pages reclaimed under memory pressure)
+	 * would count as latency. cyclictest does the same with -m.
+	 */
+	if (mlockall(MCL_CURRENT | MCL_FUTURE))
+		err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
+
 	/*
 	 * This is the tool's loop. If you want to use as base for your own tool...
 	 * go ahead.

---
base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
change-id: 20261112-rtla-mlock-2a572f10533e

Best regards,
--  
Daniel Wagner <wagi@monom.org>

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Tomas Glozar]

On Thu, Nov 12, 2026 at 10:26 PM Daniel Wagner <wagi@monom.org> wrote:
>
> With -u, the latency timerlat reports runs until the measuring thread
> is back in user space. Nothing keeps that thread's memory resident,
> so under memory pressure the kernel reclaims its code, stack and libc
> pages, and the page faults on the way back to user space count as
> latency.
>
> On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
> timerlat reported up to 40 ms for the user-space threads, which took
> 283 major faults, while the kernel threads' latency stayed below
> 1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
> 0.9 ms with it.
>

Yes, this is a known problem. I have just not seen the timerlat user
thread faulting in practice, maybe because I use rtla mostly on
servers with an "absurd" amount of RAM. Or maybe I just have not
noticed it yet.

> With this change timerlat reports 0.9 ms for the user-space threads
> under the same load, and they take no major faults.
>
> Lock the user-space threads' memory before they start measuring, as
> cyclictest -m does.
>

Thank you for measuring this.

> Signed-off-by: Daniel Wagner <wagi@monom.org>
> ---
>  tools/tracing/rtla/src/timerlat_u.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
> index c80edaf07b00..42aaf1d7f216 100644
> --- a/tools/tracing/rtla/src/timerlat_u.c
> +++ b/tools/tracing/rtla/src/timerlat_u.c
> @@ -15,6 +15,7 @@
>  #include <pthread.h>
>  #include <sys/wait.h>
>  #include <sys/prctl.h>
> +#include <sys/mman.h>
>
>  #include "common.h"
>  #include "timerlat_u.h"
> @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
>                 }
>         }
>
> +       /*
> +        * Keep the measuring process in memory: the latency it reports
> +        * runs until it is back in user space, and a page fault on the way
> +        * (its code, stack or libc pages reclaimed under memory pressure)
> +        * would count as latency. cyclictest does the same with -m.
> +        */
> +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> +

Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?

We can add it as a boolean field into struct timerlat_params and pass
down to struct timerlat_u_params. See e.g. 2ff4464e3283
("rtla/osnoise: Add IPI tracking cmdline option") for reference.

>         /*
>          * This is the tool's loop. If you want to use as base for your own tool...
>          * go ahead.
>
> ---
> base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
> change-id: 20261112-rtla-mlock-2a572f10533e
>
> Best regards,
> --
> Daniel Wagner <wagi@monom.org>
>
>

Thanks,

Tomas

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Sebastian Andrzej Siewior]

On 2026-11-13 10:30:33 [+0100], Tomas Glozar wrote:
> > --- a/tools/tracing/rtla/src/timerlat_u.c
> > +++ b/tools/tracing/rtla/src/timerlat_u.c
> > @@ -15,6 +15,7 @@
> >  #include <pthread.h>
> >  #include <sys/wait.h>
> >  #include <sys/prctl.h>
> > +#include <sys/mman.h>
> >
> >  #include "common.h"
> >  #include "timerlat_u.h"
> > @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
> >                 }
> >         }
> >
> > +       /*
> > +        * Keep the measuring process in memory: the latency it reports
> > +        * runs until it is back in user space, and a page fault on the way
> > +        * (its code, stack or libc pages reclaimed under memory pressure)
> > +        * would count as latency. cyclictest does the same with -m.
> > +        */
> > +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> > +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> > +
> 
> Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?
> 
> We can add it as a boolean field into struct timerlat_params and pass
> down to struct timerlat_u_params. See e.g. 2ff4464e3283
> ("rtla/osnoise: Add IPI tracking cmdline option") for reference.

Why make this an option? I would prefer a RT task doing the right thing
right away. cyclictest has a history of several options to do the right
thing. Is the reason why lock should be disabled?

Sebastian

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Tomas Glozar]

On Fri, Nov 13, 2026 at 10:44 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> > > +       /*
> > > +        * Keep the measuring process in memory: the latency it reports
> > > +        * runs until it is back in user space, and a page fault on the way
> > > +        * (its code, stack or libc pages reclaimed under memory pressure)
> > > +        * would count as latency. cyclictest does the same with -m.
> > > +        */
> > > +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> > > +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> > > +
> >
> > Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?
> >
> > We can add it as a boolean field into struct timerlat_params and pass
> > down to struct timerlat_u_params. See e.g. 2ff4464e3283
> > ("rtla/osnoise: Add IPI tracking cmdline option") for reference.
>
> Why make this an option? I would prefer a RT task doing the right thing
> right away. cyclictest has a history of several options to do the right
> thing. Is the reason why lock should be disabled?
>
> Sebastian
>

I agree, you are right. I was thinking about rtla setting
/dev/cpu_dma_latency to 0 as an option rather than by default (unlike
cyclictest, in that case). If rtla's idea is to measure the system in
its current configuration, it makes sense not to tune it by default.
But the rtla user thread not having mlockall is a misconfiguration of
the rtla thread (i,e. the measurement tool), not of the system.

Plus, cyclictest can run as a non-root user that might not have
permissions to mlockall. That is not an issue with rtla, which always
runs as root.

So I take that back. mlockall by default is fine.

Tomas

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Sebastian Andrzej Siewior]

On 2026-11-13 11:07:15 [+0100], Tomas Glozar wrote:
> I agree, you are right. I was thinking about rtla setting
> /dev/cpu_dma_latency to 0 as an option rather than by default (unlike
> cyclictest, in that case). If rtla's idea is to measure the system in

Oh, yes. cpu_dma_latency is tricky because of all that happens to a
system setup and might not be expected and then people wonder why the
system behaves differently if a certain application runs.
mlock on the other hand is something clearly defined and explained why
it is important.

> its current configuration, it makes sense not to tune it by default.
> But the rtla user thread not having mlockall is a misconfiguration of
> the rtla thread (i,e. the measurement tool), not of the system.
> 
> Plus, cyclictest can run as a non-root user that might not have
> permissions to mlockall. That is not an issue with rtla, which always
> runs as root.

I don't think mlock is restricted to a root user. It might be that "huge"
application might exceed their memory limit during mlock and "usually"
their memory is just virtual and never faulted in.

> So I take that back. mlockall by default is fine.
Oki.

> Tomas

Sebastian