[PATCH v2] rtla/timerlat: Lock the memory of the user-space threads

[PATCH v2] rtla/timerlat: Lock the memory of the user-space threads

[Daniel Wagner]

With -u, the latency timerlat reports runs until the measuring thread
is back in user space. Nothing keeps that thread's memory resident,
so under memory pressure the kernel reclaims its code, stack and libc
pages, and the page faults on the way back to user space count as
latency.

On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
timerlat reported up to 40 ms for the user-space threads, which took
283 major faults, while the kernel threads' latency stayed below
1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
0.9 ms with it.

With this change timerlat reports 0.9 ms for the user-space threads
under the same load, and they take no major faults.

Lock the user-space threads' memory before they start measuring, as
cyclictest -m does.

Signed-off-by: Daniel Wagner <wagi@monom.org>
---
Changes in v2:
- Shorted comment.
- Moved the mlock up before the sched switch as suggested by John
- Link to v1: https://patch.msgid.link/20261112-rtla-mlock-v1-1-418986711b20@monom.org
---
 tools/tracing/rtla/src/timerlat_u.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
index c80edaf07b00..5e7a23fb9fc3 100644
--- a/tools/tracing/rtla/src/timerlat_u.c
+++ b/tools/tracing/rtla/src/timerlat_u.c
@@ -15,6 +15,7 @@
 #include <pthread.h>
 #include <sys/wait.h>
 #include <sys/prctl.h>
+#include <sys/mman.h>
 
 #include "common.h"
 #include "timerlat_u.h"
@@ -37,6 +38,13 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
 	cpu_set_t set;
 	int retval;
 
+	/*
+	 * Fault in and lock all memory before switching to a real-time
+	 * priority, so that page faults don't show up as latency.
+	 */
+	if (mlockall(MCL_CURRENT | MCL_FUTURE))
+		err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
+
 	/*
 	 * This all is only setting up the tool.
 	 */

---
base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
change-id: 20261112-rtla-mlock-2a572f10533e

Best regards,
--  
Daniel Wagner <wagi@monom.org>

Re: [PATCH v2] rtla/timerlat: Lock the memory of the user-space threads

[John Ogness]

Hi Daniel,

On 2026-11-13, Daniel Wagner <wagi@monom.org> wrote:
> diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
> index c80edaf07b00..5e7a23fb9fc3 100644
> --- a/tools/tracing/rtla/src/timerlat_u.c
> +++ b/tools/tracing/rtla/src/timerlat_u.c
> @@ -37,6 +38,13 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
>  	cpu_set_t set;
>  	int retval;
>  
> +	/*
> +	 * Fault in and lock all memory before switching to a real-time
> +	 * priority, so that page faults don't show up as latency.
> +	 */

It isn't because the page faults show up as latencies. It is because
doing non-real-time work with a real-time priority is stealing the CPU
away from other real-time tasks that presumably have actual real-time
work to do.

This program peforms severeal non-real-time activities with the
real-time priority, which is not setting a good example. But I am only
reviewing your patch, not the full source.

John

Re: [PATCH v2] rtla/timerlat: Lock the memory of the user-space threads

[Tomas Glozar]

On Fri, Nov 13, 2026 at 3:56 PM John Ogness <john.ogness@linutronix.de> wrote:
>
> It isn't because the page faults show up as latencies. It is because
> doing non-real-time work with a real-time priority is stealing the CPU
> away from other real-time tasks that presumably have actual real-time
> work to do.
>
> This program peforms severeal non-real-time activities with the
> real-time priority, which is not setting a good example. But I am only
> reviewing your patch, not the full source.
>
> John
>

The only other activities the user thread performs with RT priority on
before entering the timerlat interface read loop are setting the
cgroup and opening the timerlat interface file descriptor. I agree
that ideally, setting the priority should be done just before the
loop.

However, I don't think either activity takes too long, so the
difference is probably negligible in practice. Unlike mlockall, which
(if I understand your point correctly) might pre-fault from disk

Tomas

Re: [PATCH v2] rtla/timerlat: Lock the memory of the user-space threads

[John Ogness]

On 2026-11-13, Tomas Glozar <tglozar@redhat.com> wrote:
> The only other activities the user thread performs with RT priority on
> before entering the timerlat interface read loop are setting the
> cgroup and opening the timerlat interface file descriptor. I agree
> that ideally, setting the priority should be done just before the
> loop.

If you say it is ideal to do X, then why not do X? Real-time tasks
(especially running at rtprio 95!) need to respect other real-time tasks
on the system. Developers new to RT might see this code and copy it. The
code even has a comment:

"If you want to use as base for your own tool... go ahead."

For me there are two issues here:

1. RT latency measurement tooling needs to be a little intrusive to a
system as practical. Otherwise it may not be possible for developers to
use the tooling on their real environment.

2. Official RT tooling should set the example of how to program for RT
Linux correctly.

John

[PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Daniel Wagner]

With -u, the latency timerlat reports runs until the measuring thread
is back in user space. Nothing keeps that thread's memory resident,
so under memory pressure the kernel reclaims its code, stack and libc
pages, and the page faults on the way back to user space count as
latency.

On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
timerlat reported up to 40 ms for the user-space threads, which took
283 major faults, while the kernel threads' latency stayed below
1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
0.9 ms with it.

With this change timerlat reports 0.9 ms for the user-space threads
under the same load, and they take no major faults.

Lock the user-space threads' memory before they start measuring, as
cyclictest -m does.

Signed-off-by: Daniel Wagner <wagi@monom.org>
---
 tools/tracing/rtla/src/timerlat_u.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
index c80edaf07b00..42aaf1d7f216 100644
--- a/tools/tracing/rtla/src/timerlat_u.c
+++ b/tools/tracing/rtla/src/timerlat_u.c
@@ -15,6 +15,7 @@
 #include <pthread.h>
 #include <sys/wait.h>
 #include <sys/prctl.h>
+#include <sys/mman.h>
 
 #include "common.h"
 #include "timerlat_u.h"
@@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
 		}
 	}
 
+	/*
+	 * Keep the measuring process in memory: the latency it reports
+	 * runs until it is back in user space, and a page fault on the way
+	 * (its code, stack or libc pages reclaimed under memory pressure)
+	 * would count as latency. cyclictest does the same with -m.
+	 */
+	if (mlockall(MCL_CURRENT | MCL_FUTURE))
+		err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
+
 	/*
 	 * This is the tool's loop. If you want to use as base for your own tool...
 	 * go ahead.

---
base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
change-id: 20261112-rtla-mlock-2a572f10533e

Best regards,
--  
Daniel Wagner <wagi@monom.org>

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Tomas Glozar]

On Thu, Nov 12, 2026 at 10:26 PM Daniel Wagner <wagi@monom.org> wrote:
>
> With -u, the latency timerlat reports runs until the measuring thread
> is back in user space. Nothing keeps that thread's memory resident,
> so under memory pressure the kernel reclaims its code, stack and libc
> pages, and the page faults on the way back to user space count as
> latency.
>
> On a Raspberry Pi 3 (PREEMPT_RT 6.12) under stress-ng --class memory,
> timerlat reported up to 40 ms for the user-space threads, which took
> 283 major faults, while the kernel threads' latency stayed below
> 1 ms. cyclictest under the same load: 133 ms without -m (mlockall),
> 0.9 ms with it.
>

Yes, this is a known problem. I have just not seen the timerlat user
thread faulting in practice, maybe because I use rtla mostly on
servers with an "absurd" amount of RAM. Or maybe I just have not
noticed it yet.

> With this change timerlat reports 0.9 ms for the user-space threads
> under the same load, and they take no major faults.
>
> Lock the user-space threads' memory before they start measuring, as
> cyclictest -m does.
>

Thank you for measuring this.

> Signed-off-by: Daniel Wagner <wagi@monom.org>
> ---
>  tools/tracing/rtla/src/timerlat_u.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
> index c80edaf07b00..42aaf1d7f216 100644
> --- a/tools/tracing/rtla/src/timerlat_u.c
> +++ b/tools/tracing/rtla/src/timerlat_u.c
> @@ -15,6 +15,7 @@
>  #include <pthread.h>
>  #include <sys/wait.h>
>  #include <sys/prctl.h>
> +#include <sys/mman.h>
>
>  #include "common.h"
>  #include "timerlat_u.h"
> @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
>                 }
>         }
>
> +       /*
> +        * Keep the measuring process in memory: the latency it reports
> +        * runs until it is back in user space, and a page fault on the way
> +        * (its code, stack or libc pages reclaimed under memory pressure)
> +        * would count as latency. cyclictest does the same with -m.
> +        */
> +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> +

Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?

We can add it as a boolean field into struct timerlat_params and pass
down to struct timerlat_u_params. See e.g. 2ff4464e3283
("rtla/osnoise: Add IPI tracking cmdline option") for reference.

>         /*
>          * This is the tool's loop. If you want to use as base for your own tool...
>          * go ahead.
>
> ---
> base-commit: ee83899302a0457a3ba47464a13e8682bd7ba27d
> change-id: 20261112-rtla-mlock-2a572f10533e
>
> Best regards,
> --
> Daniel Wagner <wagi@monom.org>
>
>

Thanks,

Tomas

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Sebastian Andrzej Siewior]

On 2026-11-13 10:30:33 [+0100], Tomas Glozar wrote:
> > --- a/tools/tracing/rtla/src/timerlat_u.c
> > +++ b/tools/tracing/rtla/src/timerlat_u.c
> > @@ -15,6 +15,7 @@
> >  #include <pthread.h>
> >  #include <sys/wait.h>
> >  #include <sys/prctl.h>
> > +#include <sys/mman.h>
> >
> >  #include "common.h"
> >  #include "timerlat_u.h"
> > @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
> >                 }
> >         }
> >
> > +       /*
> > +        * Keep the measuring process in memory: the latency it reports
> > +        * runs until it is back in user space, and a page fault on the way
> > +        * (its code, stack or libc pages reclaimed under memory pressure)
> > +        * would count as latency. cyclictest does the same with -m.
> > +        */
> > +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> > +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> > +
> 
> Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?
> 
> We can add it as a boolean field into struct timerlat_params and pass
> down to struct timerlat_u_params. See e.g. 2ff4464e3283
> ("rtla/osnoise: Add IPI tracking cmdline option") for reference.

Why make this an option? I would prefer a RT task doing the right thing
right away. cyclictest has a history of several options to do the right
thing. Is the reason why lock should be disabled?

Sebastian

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Tomas Glozar]

On Fri, Nov 13, 2026 at 10:44 AM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> > > +       /*
> > > +        * Keep the measuring process in memory: the latency it reports
> > > +        * runs until it is back in user space, and a page fault on the way
> > > +        * (its code, stack or libc pages reclaimed under memory pressure)
> > > +        * would count as latency. cyclictest does the same with -m.
> > > +        */
> > > +       if (mlockall(MCL_CURRENT | MCL_FUTURE))
> > > +               err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> > > +
> >
> > Can we make this opt-in, gated by a -m/--mlockall option, like cyclictest?
> >
> > We can add it as a boolean field into struct timerlat_params and pass
> > down to struct timerlat_u_params. See e.g. 2ff4464e3283
> > ("rtla/osnoise: Add IPI tracking cmdline option") for reference.
>
> Why make this an option? I would prefer a RT task doing the right thing
> right away. cyclictest has a history of several options to do the right
> thing. Is the reason why lock should be disabled?
>
> Sebastian
>

I agree, you are right. I was thinking about rtla setting
/dev/cpu_dma_latency to 0 as an option rather than by default (unlike
cyclictest, in that case). If rtla's idea is to measure the system in
its current configuration, it makes sense not to tune it by default.
But the rtla user thread not having mlockall is a misconfiguration of
the rtla thread (i,e. the measurement tool), not of the system.

Plus, cyclictest can run as a non-root user that might not have
permissions to mlockall. That is not an issue with rtla, which always
runs as root.

So I take that back. mlockall by default is fine.

Tomas

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Sebastian Andrzej Siewior]

On 2026-11-13 11:07:15 [+0100], Tomas Glozar wrote:
> I agree, you are right. I was thinking about rtla setting
> /dev/cpu_dma_latency to 0 as an option rather than by default (unlike
> cyclictest, in that case). If rtla's idea is to measure the system in

Oh, yes. cpu_dma_latency is tricky because of all that happens to a
system setup and might not be expected and then people wonder why the
system behaves differently if a certain application runs.
mlock on the other hand is something clearly defined and explained why
it is important.

> its current configuration, it makes sense not to tune it by default.
> But the rtla user thread not having mlockall is a misconfiguration of
> the rtla thread (i,e. the measurement tool), not of the system.
> 
> Plus, cyclictest can run as a non-root user that might not have
> permissions to mlockall. That is not an issue with rtla, which always
> runs as root.

I don't think mlock is restricted to a root user. It might be that "huge"
application might exceed their memory limit during mlock and "usually"
their memory is just virtual and never faulted in.

> So I take that back. mlockall by default is fine.
Oki.

> Tomas

Sebastian

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[Daniel Wagner]

On Fri, Nov 13, 2026 at 10:30:33AM +0100, Tomas Glozar wrote:
> > Lock the user-space threads' memory before they start measuring, as
> > cyclictest -m does.
> >
> 
> Thank you for measuring this.

FWIW, I am updating/improving my homelab which involves my RT test
setup. I only have older and smaller HW hooked up in my lab. The switch
to rtla and using stress-ng was on the TODO list. stress-ng has a bunch
of stressors which exhaust memory or CPU time. You might be able to
reproduce this as well and bigger machines. Let me know if you want some
tips.

Cheers,
Daniel

Re: [PATCH v2] rtla/timerlat: Lock the memory of the user-space threads

[Daniel Wagner]

Hi John,

On Fri, Nov 13, 2026 at 04:02:26PM +0106, John Ogness wrote:
> On 2026-11-13, Daniel Wagner <wagi@monom.org> wrote:
> > diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
> > index c80edaf07b00..5e7a23fb9fc3 100644
> > --- a/tools/tracing/rtla/src/timerlat_u.c
> > +++ b/tools/tracing/rtla/src/timerlat_u.c
> > @@ -37,6 +38,13 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
> >  	cpu_set_t set;
> >  	int retval;
> >  
> > +	/*
> > +	 * Fault in and lock all memory before switching to a real-time
> > +	 * priority, so that page faults don't show up as latency.
> > +	 */
> 
> It isn't because the page faults show up as latencies.

Page faults happened during the measurement.
 
> It is because
> doing non-real-time work with a real-time priority is stealing the CPU
> away from other real-time tasks that presumably have actual real-time
> work to do.

Could be but I think it's more likely the faulting measurement thread is
seeing the latency.

> This program peforms severeal non-real-time activities with the
> real-time priority, which is not setting a good example. But I am only
> reviewing your patch, not the full source.

The only thing I see which could be moved is the
tracefs_instance_file_open part. Not really sure if this is worth it,
though I am fine with restructuring. I don't have a strong opinion.

Thanks,
Daniel

Re: [PATCH] rtla/timerlat: Lock the memory of the user-space threads

[John Ogness]

Hi Daniel,

On 2026-11-12, Daniel Wagner <wagi@monom.org> wrote:
> diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
> index c80edaf07b00..42aaf1d7f216 100644
> --- a/tools/tracing/rtla/src/timerlat_u.c
> +++ b/tools/tracing/rtla/src/timerlat_u.c
> @@ -69,6 +70,15 @@ static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
>  		}
>  	}
>  
> +	/*
> +	 * Keep the measuring process in memory: the latency it reports
> +	 * runs until it is back in user space, and a page fault on the way
> +	 * (its code, stack or libc pages reclaimed under memory pressure)
> +	 * would count as latency. cyclictest does the same with -m.
> +	 */
> +	if (mlockall(MCL_CURRENT | MCL_FUTURE))
> +		err_msg("Could not lock timerlat u memory: %s\n", strerror(errno));
> +

Page faulting should happen _before_ the task has a real-time
priority. Real-time priorities are for real-time workloads.

Just move this up 30 lines.

John Ogness