Re: [RFC PATCH 0/4] mm, bpf: BPF based THP adjustment

["Zi Yan" <ziy@nvidia.com>]

Hi Yafang,

We recently added a new THP entry in MAINTAINERS file[1], do you mind ccing
people there in your next version? (I added them here)

[1] https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/tree/MAINTAINERS?h=mm-everything#n15589

On Mon Apr 28, 2025 at 10:41 PM EDT, Yafang Shao wrote:
> In our container environment, we aim to enable THP selectively—allowing
> specific services to use it while restricting others. This approach is
> driven by the following considerations:
>
> 1. Memory Fragmentation
>    THP can lead to increased memory fragmentation, so we want to limit its
>    use across services.
> 2. Performance Impact
>    Some services see no benefit from THP, making its usage unnecessary.
> 3. Performance Gains
>    Certain workloads, such as machine learning services, experience
>    significant performance improvements with THP, so we enable it for them
>    specifically. 
>
> Since multiple services run on a single host in a containerized environment,
> enabling THP globally is not ideal. Previously, we set THP to madvise,
> allowing selected services to opt in via MADV_HUGEPAGE. However, this
> approach had limitation:
>
> - Some services inadvertently used madvise(MADV_HUGEPAGE) through
>   third-party libraries, bypassing our restrictions.

Basically, you want more precise control of THP enablement and the
ability of overriding madvise() from userspace.

In terms of overriding madvise(), do you have any concrete example of
these third-party libraries? madvise() users are supposed to know what
they are doing, so I wonder why they are causing trouble in your
environment.

>
> To address this issue, we initially hooked the __x64_sys_madvise() syscall,
> which is error-injectable, to blacklist unwanted services. While this
> worked, it was error-prone and ineffective for services needing always mode,
> as modifying their code to use madvise was impractical.
>
> To achieve finer-grained control, we introduced an fmod_ret-based solution.
> Now, we dynamically adjust THP settings per service by hooking
> hugepage_global_{enabled,always}() via BPF. This allows us to set THP to
> enable or disable on a per-service basis without global impact.

hugepage_global_*() are whole system knobs. How did you use it to
achieve per-service control? In terms of per-service, does it mean
you need per-memcg group (I assume each service has its own memcg) THP
configuration?

>
> The hugepage_global_{enabled,always}() functions currently share the same
> BPF hook, which limits THP configuration to either always or never. While
> this suffices for our specific use cases, full support for all three modes
> (always, madvise, and never) would require splitting them into separate
> hooks.
>
> This is the initial RFC patch—feedback is welcome!
>
> Yafang Shao (4):
>   mm: move hugepage_global_{enabled,always}() to internal.h
>   mm: pass VMA parameter to hugepage_global_{enabled,always}()
>   mm: add BPF hook for THP adjustment
>   selftests/bpf: Add selftest for THP adjustment
>
>  include/linux/huge_mm.h                       |  54 +-----
>  mm/Makefile                                   |   3 +
>  mm/bpf.c                                      |  36 ++++
>  mm/bpf.h                                      |  21 +++
>  mm/huge_memory.c                              |  50 ++++-
>  mm/internal.h                                 |  21 +++
>  mm/khugepaged.c                               |  18 +-
>  tools/testing/selftests/bpf/config            |   1 +
>  .../selftests/bpf/prog_tests/thp_adjust.c     | 176 ++++++++++++++++++
>  .../selftests/bpf/progs/test_thp_adjust.c     |  32 ++++
>  10 files changed, 344 insertions(+), 68 deletions(-)
>  create mode 100644 mm/bpf.c
>  create mode 100644 mm/bpf.h
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/thp_adjust.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_thp_adjust.c




-- 
Best Regards,
Yan, Zi