Re: Minor page faults from memory compaction causing in-band transitions

Re: Minor page faults from memory compaction causing in-band transitions

[Jay Sridharan]

Hi Phillipe,

Let me know what you think of this patch for the docs:

diff --git a/content/core/caveat.md b/content/core/caveat.md
index b24e270..e11d605 100644
--- a/content/core/caveat.md
+++ b/content/core/caveat.md
@@ -70,6 +70,68 @@ which depend on instrumenting the spinlock constructs (e.g.
 `CONFIG_DEBUG_PREEMPT`), you may want to disable all the related kernel
 options, starting with `CONFIG_SMP`.

+### Memory compaction and page migration impact real-time behavior
{#caveat-memory-compaction}
+
+Several kernel configuration options related to memory management can
+introduce unpredictable latency through page migration and page fault
+handling, which is problematic for real-time workloads:
+
+- `CONFIG_COMPACTION`: Enables memory compaction to reduce fragmentation
+  by migrating pages to create larger contiguous memory regions. The
+  compaction process can trigger page migrations that introduce latency.
+
+- `CONFIG_MIGRATION`: Allows the migration of the physical location of
+  memory pages of processes while the virtual addresses are not changed.
+  Useful for allowing the kernel to move pages between NUMA
+  nodes or during compaction. Page migration involves copying page
+  contents and updating page table entries, which can take time, or
+  cause page faults when page table entries become temporarily unavailable.
+  The use of `mlock` or `mlockall` does **NOT** automatically guarantee
+  that a page will not be migrated. See below for more.
+
+- `CONFIG_TRANSPARENT_HUGEPAGE`: Transparent Hugepages (THP) reduces page
+  faults by mapping 2MB instead of 4KB pages, but causes higher latency
+  during faults due to intensive memory allocation and zeroing. While
+  reducing the number of faults, THP can incur higher latency during
+  initial memory access or when khugepaged compacts memory.
+
+The best approach depends on your workload characteristics. In an ideal
+situation, you would disable `CONFIG_COMPACTION`,
+`CONFIG_MIGRATION`, and `CONFIG_TRANSPARENT_HUGEPAGE` entirely. In other
+situations, you may need to keep these options enabled
+
+If you are running applications that alloc/free memory often, and/or need
+a steady source of consecutive pages, you may need to keep `CONFIG_MIGRATION`
+and `CONFIG_COMPACTION` enabled. Without it, your in-band applications may
+experience out-of-memory issues.
+
+Luckily, procfs provides tunables to control compaction behavior:
+
+- `vm.compaction_proactiveness`: determines how aggressively compaction is
+  done in the background. Write of a non zero value to this tunable will
+  immediately trigger the proactive compaction. Setting it to 0 disables
+  proactive compaction.
+- `vm.compact_unevictable_allowed`: When set to 1, compaction is allowed
+  to examine the unevictable lru (mlocked pages) for pages to compact. This
+  should be used on systems where stalls for minor page faults are an
+  acceptable trade for large contiguous free memory. Set to 0 to prevent
+  compaction from moving pages that are unevictable. On EVL, the default
+  value is 0 in order to avoid a page fault due to compaction.
+  (`CONFIG_COMPACT_UNEVICTABLE_DEFAULT`)
+
+procfs also provides an interface to manually trigger a memory compaction
+operation using `vm.compact_memory`.
+
+Using these options, you can perform a sequence such as the following to
+prime the system for reliability:
+
+- Launch EVL threads. At the end of initialization, but before
calling mlockall..
+- Trigger a memory compaction manually using `vm.compact_memory`
+- Once complete, call `mlockall` to lock pages.
+- Then, disable `vm.compact_unevictable_allowed` to prevent those pages
+  from getting migrated.
+- Finally, launch other in-band applications.
+
 ## Architecture-specific issues

 ### x86 {#x86-caveat}

Re: Minor page faults from memory compaction causing in-band transitions

[Philippe Gerum]

Hi Jay,

Looks good to me, very useful for understanding some of the intricacies
of virtual memory management vs real-time workloads, and how to address
them. I would happily merge a commit with this information as-is.

[I noticed a couple of hiccups breaking the patch format likely due to
the MUA].

Jay Sridharan <jayasurya.sridharan@gmail.com> writes:

> Hi Phillipe,
>
> Let me know what you think of this patch for the docs:
>
> diff --git a/content/core/caveat.md b/content/core/caveat.md
> index b24e270..e11d605 100644
> --- a/content/core/caveat.md
> +++ b/content/core/caveat.md
> @@ -70,6 +70,68 @@ which depend on instrumenting the spinlock constructs (e.g.
>  `CONFIG_DEBUG_PREEMPT`), you may want to disable all the related kernel
>  options, starting with `CONFIG_SMP`.
>
> +### Memory compaction and page migration impact real-time behavior
> {#caveat-memory-compaction}

<line wrapped>

> +
> +- Launch EVL threads. At the end of initialization, but before
> calling mlockall..

<line wrapped>

-- 
Philippe.

[PATCH] update caveat.md about memory compaction

[Jay Sridharan]

From: Jay Sridharan <jsridharan@relativityspace.com>

Thanks Phillipe - maybe try it now? First time sending in patches via email...

---
 content/core/caveat.md | 62 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/content/core/caveat.md b/content/core/caveat.md
index b24e270..912b5ba 100644
--- a/content/core/caveat.md
+++ b/content/core/caveat.md
@@ -70,6 +70,68 @@ which depend on instrumenting the spinlock constructs (e.g.
 `CONFIG_DEBUG_PREEMPT`), you may want to disable all the related kernel
 options, starting with `CONFIG_SMP`.
 
+### Memory compaction and page migration impact real-time behavior {#caveat-memory-compaction}
+
+Several kernel configuration options related to memory management can
+introduce unpredictable latency through page migration and page fault
+handling, which is problematic for real-time workloads:
+
+- `CONFIG_COMPACTION`: Enables memory compaction to reduce fragmentation
+  by migrating pages to create larger contiguous memory regions. The
+  compaction process can trigger page migrations that introduce latency.
+
+- `CONFIG_MIGRATION`: Allows the migration of the physical location of
+  memory pages of processes while the virtual addresses are not changed.
+  Useful for allowing the kernel to move pages between NUMA
+  nodes or during compaction. Page migration involves copying page
+  contents and updating page table entries, which can take time, or
+  cause page faults when page table entries become temporarily unavailable.
+  The use of `mlock` or `mlockall` does **NOT** automatically guarantee
+  that a page will not be migrated. See below for more.
+
+- `CONFIG_TRANSPARENT_HUGEPAGE`: Transparent Hugepages (THP) reduces page
+  faults by mapping 2MB instead of 4KB pages, but causes higher latency
+  during faults due to intensive memory allocation and zeroing. While
+  reducing the number of faults, THP can incur higher latency during
+  initial memory access or when khugepaged compacts memory.
+
+The best approach depends on your workload characteristics. In an ideal
+situation, you would disable `CONFIG_COMPACTION`,
+`CONFIG_MIGRATION`, and `CONFIG_TRANSPARENT_HUGEPAGE` entirely. In other
+situations, you may need to keep these options enabled
+
+If you are running applications that alloc/free memory often, and/or need
+a steady source of consecutive pages, you may need to keep `CONFIG_MIGRATION`
+and `CONFIG_COMPACTION` enabled. Without it, your in-band applications may
+experience out-of-memory issues.
+
+Luckily, procfs provides tunables to control compaction behavior:
+
+- `vm.compaction_proactiveness`: determines how aggressively compaction is
+  done in the background. Write of a non zero value to this tunable will
+  immediately trigger the proactive compaction. Setting it to 0 disables
+  proactive compaction.
+- `vm.compact_unevictable_allowed`: When set to 1, compaction is allowed
+  to examine the unevictable lru (mlocked pages) for pages to compact. This
+  should be used on systems where stalls for minor page faults are an
+  acceptable trade for large contiguous free memory. Set to 0 to prevent
+  compaction from moving pages that are unevictable. On EVL, the default
+  value is 0 in order to avoid a page fault due to compaction.
+  (`CONFIG_COMPACT_UNEVICTABLE_DEFAULT`)
+
+procfs also provides an interface to manually trigger a memory compaction
+operation using `vm.compact_memory`.
+
+Using these options, you can perform a sequence such as the following to
+prime the system for reliability:
+
+- Launch EVL threads. At the end of initialization, but before calling mlockall..
+- Trigger a memory compaction manually using `vm.compact_memory`
+- Once complete, call `mlockall` to lock pages.
+- Then, disable `vm.compact_unevictable_allowed` to prevent those pages
+  from getting migrated.
+- Finally, launch other in-band applications.
+
 ## Architecture-specific issues
 
 ### x86 {#x86-caveat}
-- 
2.43.0

Re: [PATCH] update caveat.md about memory compaction

[Philippe Gerum]

Jay Sridharan <jayasurya.sridharan@gmail.com> writes:

> From: Jay Sridharan <jsridharan@relativityspace.com>
>
> Thanks Phillipe - maybe try it now? First time sending in patches via email...
>

Merged thanks. The changes should appear within minutes (i.e. after the CI run).

-- 
Philippe.