Re: [RFC PATCH v2 0/5] mm, bpf: BPF based THP adjustment

[David Hildenbrand <david@redhat.com>]

On 27.05.25 07:46, Yafang Shao wrote:
> On Mon, May 26, 2025 at 6:49 PM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 26.05.25 11:37, Yafang Shao wrote:
>>> On Mon, May 26, 2025 at 4:14 PM David Hildenbrand <david@redhat.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Let’s summarize the current state of the discussion and identify how
>>>>> to move forward.
>>>>>
>>>>> - Global-Only Control is Not Viable
>>>>> We all seem to agree that a global-only control for THP is unwise. In
>>>>> practice, some workloads benefit from THP while others do not, so a
>>>>> one-size-fits-all approach doesn’t work.
>>>>>
>>>>> - Should We Use "Always" or "Madvise"?
>>>>> I suspect no one would choose 'always' in its current state. ;)
>>>>
>>>> IIRC, RHEL9 has the default set to "always" for a long time.
>>>
>>> good to know.
>>>
>>>>
>>>> I guess it really depends on how different the workloads are that you
>>>> are running on the same machine.
>>>
>>> Correct. If we want to enable THP for specific workloads without
>>> modifying the kernel, we must isolate them on dedicated servers.
>>> However, this approach wastes resources and is not an acceptable
>>> solution.
>>>
>>>>
>>>>    > Both Lorenzo and David propose relying on the madvise mode. However,>
>>>> since madvise is an unprivileged userspace mechanism, any user can
>>>>> freely adjust their THP policy. This makes fine-grained control
>>>>> impossible without breaking userspace compatibility—an undesirable
>>>>> tradeoff.
>>>>
>>>> If required, we could look into a "sealing" mechanism, that would
>>>> essentially lock modification attempts performed by the process (i.e.,
>>>> MADV_HUGEPAGE).
>>>
>>> If we don’t introduce a new THP mode and instead rely solely on
>>> madvise, the "sealing" mechanism could either violate the intended
>>> semantics of madvise(), or simply break madvise() entirely, right?
>>
>> We would have to be a bit careful, yes.
>>
>> Errors from MADV_HUGEPAGE/MADV_NOHUGEPAGE are often ignored, because
>> these options also fail with -EINVAL on kernels without THP support.
>>
>> Ignoring MADV_NOHUGEPAGE can be problematic with userfaultfd.
>>
>> What you likely really want to do is seal when you configured
>> MADV_NOHUGEPAGE to be the default, and fail MADV_HUGEPAGE later.
>>
>>>>
>>>> The could be added on top of the current proposals that are flying
>>>> around, and could be done e.g., per-process.
>>>
>>> How about introducing a dedicated "process" mode? This would allow
>>> each process to use different THP modes—some in "always," others in
>>> "madvise," and the rest in "never." Future THP modes could also be
>>> added to this framework.
>>
>> We have to be really careful about not creating even more mess with more
>> modes.
>>
>> How would that design look like in detail (how would we set it per
>> process etc?)?
> 
> I have a preliminary idea to implement this using BPF. 

I don't think we want to add such a mechanism (new mode) where the 
primary configuration mechanism is through bpf.

Maybe bpf could be used as an alternative, but we should look into a 
reasonable alternative first, like the discussed mctrl()/.../ raised in 
the process_madvise() series.

No "bpf" mode in disguise, please :)

> We could define
> the API as follows:
> 
> struct bpf_thp_ops {
>         /**
>          * @task_thp_mode: Get the THP mode for a specific task
>          *
>          * Return:
>          * - TASK_THP_ALWAYS: "always" mode
>          * - TASK_THP_MADVISE: "madvise" mode
>          * - TASK_THP_NEVER: "never" mode
>          * Future modes can also be added.
>          */
>         int (*task_thp_mode)(struct task_struct *p);
> };
> 
> For observability, we could add a "THP mode" field to
> /proc/[pid]/status. For example:
> 
> $ grep "THP mode" /proc/123/status
> always
> $ grep "THP mode" /proc/456/status
> madvise
> $ grep "THP mode" /proc/789/status
> never
> 
> The THP mode for each task would be determined by the attached BPF
> program based on the task's attributes. We would place the BPF hook in
> appropriate kernel functions. Note that this setting wouldn't be
> inherited during fork/exec - the BPF program would make the decision
> dynamically for each task.

What would be the mode (default) when the bpf program would not be active?

> This approach also enables runtime adjustments to THP modes based on
> system-wide conditions, such as memory fragmentation or other
> performance overheads. The BPF program could adapt policies
> dynamically, optimizing THP behavior in response to changing
> workloads.

I am not sure that is the proper way to handle these scenarios: I never 
heard that people would be adjusting the system-wide policy dynamically 
in that way either.

Whatever we do, we have to make sure that what we add won't 
over-complicate things in the future. Having tooling dynamically adjust 
the THP policy of processes that coarsely sounds ... very wrong long-term.

 > > As Liam pointed out in another thread, naming is challenging here -
> "process" might not be the most accurate term for this context.

No, it's not even a per-process thing. It is per MM, and a MM might be 
used by multiple processes ...

-- 
Cheers,

David / dhildenb