Minor page faults from memory compaction causing in-band transitions

Minor page faults from memory compaction causing in-band transitions

[Brandon Ho]

Hi Xenomai team,

I'm working on a real-time control application using Xenomai and we've been
experiencing unexpected in-band transitions caused by minor page faults during
kernel memory compaction. Even though our RT threads use mlockall(MCL_CURRENT |
MCL_FUTURE) and we've pre-faulted memory, the kernel's compaction process seems
to temporarily invalidate PTEs on our locked pages, causing faults when the RT
thread accesses them.

Has this issue come up before? I'm wondering if there's a mechanism to reserve
a section of RAM that's excluded from compaction/migration entirely, or if
there are kernel/Xenomai configurations we should be using to prevent this.

Any guidance would be appreciated!

Thanks,
Brandon

Re: Minor page faults from memory compaction causing in-band transitions

[Jan Kiszka]

On 01.04.26 01:14, Brandon Ho wrote:
> Hi Xenomai team,
> 
> I'm working on a real-time control application using Xenomai and we've been
> experiencing unexpected in-band transitions caused by minor page faults during
> kernel memory compaction. Even though our RT threads use mlockall(MCL_CURRENT |
> MCL_FUTURE) and we've pre-faulted memory, the kernel's compaction process seems
> to temporarily invalidate PTEs on our locked pages, causing faults when the RT
> thread accesses them.
> 
> Has this issue come up before? I'm wondering if there's a mechanism to reserve
> a section of RAM that's excluded from compaction/migration entirely, or if
> there are kernel/Xenomai configurations we should be using to prevent this.
> 
> Any guidance would be appreciated!

Memory migration as well as compaction are not compatible with
deterministic system behavior. Both Xenomai 3 and 4 should warn about
them, and you should disabling these features in your kernel.
Unfortunately, current kernels do not allow a more fine-grained control.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Brandon Ho <brandonho667@gmail.com> writes:

> Hi Xenomai team,
>
> I'm working on a real-time control application using Xenomai and we've been
> experiencing unexpected in-band transitions caused by minor page faults during
> kernel memory compaction. Even though our RT threads use mlockall(MCL_CURRENT |
> MCL_FUTURE) and we've pre-faulted memory, the kernel's compaction process seems
> to temporarily invalidate PTEs on our locked pages, causing faults when the RT
> thread accesses them.
>
> Has this issue come up before?

Yes, this is a known issue.

> I'm wondering if there's a mechanism to reserve
> a section of RAM that's excluded from compaction/migration entirely,
> or if

Not to my knowledge.

> there are kernel/Xenomai configurations we should be using to prevent this.
>
> Any guidance would be appreciated!
>

As Jan mentioned, the only way is to disable all options which select
CONFIG_COMPACTION ATM.

-- 
Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Brandon Ho]

Thank you for the responses! I recently came across the
`compact_unevictable_allowed` sysctl option and it seems like it could be a
good fit for our use case. Setting it to 0 would prevent the kernel from
compacting/migrating locked pages, which should eliminate the minor faults
we're seeing during compaction.

Could we possibly set `compact_unevictable_allowed=0` by default when
CONFIG_EVL is enabled?

https://www.infradead.org/~mchehab/kernel_docs/admin-guide/sysctl/vm.html#compact-unevictable-allowed

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Brandon Ho <brandonho667@gmail.com> writes:

> Thank you for the responses! I recently came across the
> `compact_unevictable_allowed` sysctl option and it seems like it could be a
> good fit for our use case. Setting it to 0 would prevent the kernel from
> compacting/migrating locked pages, which should eliminate the minor faults
> we're seeing during compaction.
>
> Could we possibly set `compact_unevictable_allowed=0` by default when
> CONFIG_EVL is enabled?

Definitely, yes. Thanks for chasing this issue down. This patch is going
to be merged upstream asap, you may want to apply it for a preview.

diff --git a/mm/Kconfig b/mm/Kconfig
index 7bf469354faf3..759af896a5c7b 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -648,7 +648,7 @@ config COMPACTION
 config COMPACT_UNEVICTABLE_DEFAULT
 	int
 	depends on COMPACTION
-	default 0 if PREEMPT_RT
+	default 0 if PREEMPT_RT || DOVETAIL
 	default 1
 
 #

>
> https://www.infradead.org/~mchehab/kernel_docs/admin-guide/sysctl/vm.html#compact-unevictable-allowed

-- 
Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Jay Sridharan <jsridharan@relativityspace.com> writes:

> I think it would also be very useful to add these considerations to the "Caveat" page in the EVL documentation. 
>

Agreed.

> Perhaps some discussion on the impacts of CONFIG_COMPACTION, CONFIG_MIGRATION and CONFIG_TRANSPARENT_HUGEPAGE, as
> well as the impact of COMPACT_UNEVICTABLE_DEFAULT.
>
> I know the Caveat page already has an in-depth discussion of CPU isolation and SMIs, so it feels like a good place to mention it. 
>
> Is there a place where we can submit a patch to the docs?
>

Yes, please send a patch to this list, the markdown files for the
website can be cloned from [1].

[1] https://gitlab.com/Xenomai/xenomai4/website

-- 
Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Florian Bezdeka]

On Mon, 2026-04-13 at 20:31 +0200, Philippe Gerum wrote:
> Brandon Ho <brandonho667@gmail.com> writes:
> 
> > Thank you for the responses! I recently came across the
> > `compact_unevictable_allowed` sysctl option and it seems like it could be a
> > good fit for our use case. Setting it to 0 would prevent the kernel from
> > compacting/migrating locked pages, which should eliminate the minor faults
> > we're seeing during compaction.
> > 
> > Could we possibly set `compact_unevictable_allowed=0` by default when
> > CONFIG_EVL is enabled?
> 
> Definitely, yes. Thanks for chasing this issue down. This patch is going
> to be merged upstream asap, you may want to apply it for a preview.

Targeting linux-evl or linux-dovetail?

This is of interest for Xenomai 3 as well, so merging on Dovetail-level
would be nice.

The same applies for the documentation update. Once we have it for x4 we
should consider that for x4 as well.

> 
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 7bf469354faf3..759af896a5c7b 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -648,7 +648,7 @@ config COMPACTION
>  config COMPACT_UNEVICTABLE_DEFAULT
>  	int
>  	depends on COMPACTION
> -	default 0 if PREEMPT_RT
> +	default 0 if PREEMPT_RT || DOVETAIL
>  	default 1
>  
>  #
> 
> > 
> > https://www.infradead.org/~mchehab/kernel_docs/admin-guide/sysctl/vm.html#compact-unevictable-allowed
> 
> --
> Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Florian Bezdeka <florian.bezdeka@siemens.com> writes:

> On Mon, 2026-04-13 at 20:31 +0200, Philippe Gerum wrote:
>> Brandon Ho <brandonho667@gmail.com> writes:
>> 
>> > Thank you for the responses! I recently came across the
>> > `compact_unevictable_allowed` sysctl option and it seems like it could be a
>> > good fit for our use case. Setting it to 0 would prevent the kernel from
>> > compacting/migrating locked pages, which should eliminate the minor faults
>> > we're seeing during compaction.
>> > 
>> > Could we possibly set `compact_unevictable_allowed=0` by default when
>> > CONFIG_EVL is enabled?
>> 
>> Definitely, yes. Thanks for chasing this issue down. This patch is going
>> to be merged upstream asap, you may want to apply it for a preview.
>
> Targeting linux-evl or linux-dovetail?
>

linux-dovetail, this is a general issue for any latency-sensitive code.


-- 
Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Florian Bezdeka <florian.bezdeka@siemens.com> writes:
>
>> On Mon, 2026-04-13 at 20:31 +0200, Philippe Gerum wrote:
>>> Brandon Ho <brandonho667@gmail.com> writes:
>>> 
>>> > Thank you for the responses! I recently came across the
>>> > `compact_unevictable_allowed` sysctl option and it seems like it could be a
>>> > good fit for our use case. Setting it to 0 would prevent the kernel from
>>> > compacting/migrating locked pages, which should eliminate the minor faults
>>> > we're seeing during compaction.
>>> > 
>>> > Could we possibly set `compact_unevictable_allowed=0` by default when
>>> > CONFIG_EVL is enabled?
>>> 
>>> Definitely, yes. Thanks for chasing this issue down. This patch is going
>>> to be merged upstream asap, you may want to apply it for a preview.
>>
>> Targeting linux-evl or linux-dovetail?
>>
>
> linux-dovetail, this is a general issue for any latency-sensitive code.

While reviewing all PREEMPT_RT exclusion cases in Kconfigs, some of
which Dovetail may want to share, I found this potential one:

--- a/init/Kconfig
+++ b/init/Kconfig
@@ -947,7 +947,7 @@ config NUMA_BALANCING
 	bool "Memory placement aware NUMA scheduler"
 	depends on ARCH_SUPPORTS_NUMA_BALANCING
 	depends on !ARCH_WANT_NUMA_VARIABLE_LOCALITY
-	depends on SMP && NUMA && MIGRATION && !PREEMPT_RT
+	depends on SMP && NUMA && MIGRATION && !PREEMPT_RT && !DOVETAIL

Any objection to disabling NUMA_BALANCING when operating in dual kernel
mode? (CONFIG_DOVETAIL=y).

-- 
Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Jay Sridharan]

Hi Phillipe,

Let me know what you think of this patch for the docs:

diff --git a/content/core/caveat.md b/content/core/caveat.md
index b24e270..e11d605 100644
--- a/content/core/caveat.md
+++ b/content/core/caveat.md
@@ -70,6 +70,68 @@ which depend on instrumenting the spinlock constructs (e.g.
 `CONFIG_DEBUG_PREEMPT`), you may want to disable all the related kernel
 options, starting with `CONFIG_SMP`.

+### Memory compaction and page migration impact real-time behavior
{#caveat-memory-compaction}
+
+Several kernel configuration options related to memory management can
+introduce unpredictable latency through page migration and page fault
+handling, which is problematic for real-time workloads:
+
+- `CONFIG_COMPACTION`: Enables memory compaction to reduce fragmentation
+  by migrating pages to create larger contiguous memory regions. The
+  compaction process can trigger page migrations that introduce latency.
+
+- `CONFIG_MIGRATION`: Allows the migration of the physical location of
+  memory pages of processes while the virtual addresses are not changed.
+  Useful for allowing the kernel to move pages between NUMA
+  nodes or during compaction. Page migration involves copying page
+  contents and updating page table entries, which can take time, or
+  cause page faults when page table entries become temporarily unavailable.
+  The use of `mlock` or `mlockall` does **NOT** automatically guarantee
+  that a page will not be migrated. See below for more.
+
+- `CONFIG_TRANSPARENT_HUGEPAGE`: Transparent Hugepages (THP) reduces page
+  faults by mapping 2MB instead of 4KB pages, but causes higher latency
+  during faults due to intensive memory allocation and zeroing. While
+  reducing the number of faults, THP can incur higher latency during
+  initial memory access or when khugepaged compacts memory.
+
+The best approach depends on your workload characteristics. In an ideal
+situation, you would disable `CONFIG_COMPACTION`,
+`CONFIG_MIGRATION`, and `CONFIG_TRANSPARENT_HUGEPAGE` entirely. In other
+situations, you may need to keep these options enabled
+
+If you are running applications that alloc/free memory often, and/or need
+a steady source of consecutive pages, you may need to keep `CONFIG_MIGRATION`
+and `CONFIG_COMPACTION` enabled. Without it, your in-band applications may
+experience out-of-memory issues.
+
+Luckily, procfs provides tunables to control compaction behavior:
+
+- `vm.compaction_proactiveness`: determines how aggressively compaction is
+  done in the background. Write of a non zero value to this tunable will
+  immediately trigger the proactive compaction. Setting it to 0 disables
+  proactive compaction.
+- `vm.compact_unevictable_allowed`: When set to 1, compaction is allowed
+  to examine the unevictable lru (mlocked pages) for pages to compact. This
+  should be used on systems where stalls for minor page faults are an
+  acceptable trade for large contiguous free memory. Set to 0 to prevent
+  compaction from moving pages that are unevictable. On EVL, the default
+  value is 0 in order to avoid a page fault due to compaction.
+  (`CONFIG_COMPACT_UNEVICTABLE_DEFAULT`)
+
+procfs also provides an interface to manually trigger a memory compaction
+operation using `vm.compact_memory`.
+
+Using these options, you can perform a sequence such as the following to
+prime the system for reliability:
+
+- Launch EVL threads. At the end of initialization, but before
calling mlockall..
+- Trigger a memory compaction manually using `vm.compact_memory`
+- Once complete, call `mlockall` to lock pages.
+- Then, disable `vm.compact_unevictable_allowed` to prevent those pages
+  from getting migrated.
+- Finally, launch other in-band applications.
+
 ## Architecture-specific issues

 ### x86 {#x86-caveat}

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Hi Jay,

Looks good to me, very useful for understanding some of the intricacies
of virtual memory management vs real-time workloads, and how to address
them. I would happily merge a commit with this information as-is.

[I noticed a couple of hiccups breaking the patch format likely due to
the MUA].

Jay Sridharan <jayasurya.sridharan@gmail.com> writes:

> Hi Phillipe,
>
> Let me know what you think of this patch for the docs:
>
> diff --git a/content/core/caveat.md b/content/core/caveat.md
> index b24e270..e11d605 100644
> --- a/content/core/caveat.md
> +++ b/content/core/caveat.md
> @@ -70,6 +70,68 @@ which depend on instrumenting the spinlock constructs (e.g.
>  `CONFIG_DEBUG_PREEMPT`), you may want to disable all the related kernel
>  options, starting with `CONFIG_SMP`.
>
> +### Memory compaction and page migration impact real-time behavior
> {#caveat-memory-compaction}

<line wrapped>

> +
> +- Launch EVL threads. At the end of initialization, but before
> calling mlockall..

<line wrapped>

-- 
Philippe.

Re: Minor page faults from memory compaction causing in-band transitions

[Jay Sridharan]

I think it would also be very useful to add these considerations to
the "Caveat" page in the EVL documentation.

Perhaps some discussion on the impacts of CONFIG_COMPACTION,
CONFIG_MIGRATION and CONFIG_TRANSPARENT_HUGEPAGE, as well as the
impact of COMPACT_UNEVICTABLE_DEFAULT.

I know the Caveat page already has an in-depth discussion of CPU
isolation and SMIs, so it feels like a good place to mention it.

Is there a place where we can submit a patch to the docs?

Jay

Minor page faults from memory compaction causing in-band transitions

[Brandon Ho]

Hi Xenomai team,

I'm working on a real-time control application using Xenomai and we've been experiencing unexpected in-band transitions caused by minor page faults during kernel memory compaction. Even though our RT threads use mlockall(MCL_CURRENT | MCL_FUTURE) and we've pre-faulted memory, the kernel's compaction process seems to temporarily invalidate PTEs on our locked pages, causing faults when the RT thread accesses them.

We've tried the usual mitigations (disabling THP, MAP_LOCKED, etc.) but compaction can still trigger these faults on already-locked pages.

Has this issue come up before? I'm wondering if there's a mechanism to reserve a section of RAM that's excluded from compaction/migration entirely, or if there are kernel/Xenomai configurations we should be using to prevent this.

Any guidance would be appreciated!

Thanks,
Brandon