Re: [RFC PATCH] mm: show mthp_fault_alloc and mthp_fault_fallback of multi-size THPs

[Ryan Roberts <ryan.roberts@arm.com>]

On 27/03/2024 11:35, David Hildenbrand wrote:
> On 26.03.24 23:19, Barry Song wrote:
>> On Tue, Mar 26, 2024 at 4:40 PM Barry Song <21cnbao@gmail.com> wrote:
>>>
>>> On Tue, Mar 26, 2024 at 4:25 PM Matthew Wilcox <willy@infradead.org> wrote:
>>>>
>>>> On Tue, Mar 26, 2024 at 04:01:03PM +1300, Barry Song wrote:
>>>>> Profiling a system blindly with mTHP has become challenging due
>>>>> to the lack of visibility into its operations. While displaying
>>>>> additional statistics such as partial map/unmap actions may
>>>>> spark debate, presenting the success rate of mTHP allocations
>>>>> appears to be a straightforward and pressing need.
>>>>
>>>> Ummm ... no?  Not like this anyway.  It has the bad assumption that
>>>> "mTHP" only comes in one size.
>>>
>>>
>>> I had initially considered per-size allocation and fallback before sending
>>> the RFC. However, in order to prompt discussion and exploration
>>> into profiling possibilities, I opted to send the simplest code instead.
>>>
>>> We could consider two options for displaying per-size statistics.
>>>
>>> 1. A single file could be used to display data for all sizes.
>>> 1024KiB fault allocation:
>>> 1024KiB fault fallback:
>>> 512KiB fault allocation:
>>> 512KiB fault fallback:
>>> ....
>>> 64KiB fault allocation:
>>> 64KiB fault fallback:
>>>
>>> 2. A separate file for each size
>>> For example,
>>>
>>> /sys/kernel/debug/transparent_hugepage/hugepages-1024kB/vmstat
>>> /sys/kernel/debug/transparent_hugepage/hugepages-512kB/vmstat
>>> ...
>>> /sys/kernel/debug/transparent_hugepage/hugepages-64kB/vmstat
>>>
>>
>> Hi Ryan, David, Willy, Yu,
> 
> Hi!
> 
>>
>> I'm collecting feedback on whether you'd prefer access to something similar
>> to /sys/kernel/debug/transparent_hugepage/hugepages-<size>/stat to help
>> determine the direction to take for this patch.
> 
> I discussed in the past that we might want to place statistics into sysfs. The
> idea was to place them into our new hierarchy:
> 
> /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/...
> 
> following the "one value per file" sysfs design principle.
> 
> We could have a new folder "stats" in there that contains files with statistics
> we care about.
> 
> Of course, we could also place that initially into debugfs in a similar fashion,
> and move it over once the interface is considered good and stable.
> 
> My 2 cents would be to avoid a "single file".

Yes I agree with this. We discussed in the past and I summarised the outcome
here: https://lore.kernel.org/linux-mm/6cc7d781-884f-4d8f-a175-8609732b87eb@arm.com/

There are more counters on my list than what you are proposing here. But
conclusion was that they should be per-size and exposed through sysfs as David
suggests.

Personally I think those counters are obviously needed so prefer to go straight
to sysfs if we can :)

I had a low priority todo item to look at this - very pleased that you're taking
it on!

Copy/pasting my original summary here, for the lazy:

--8<----

I just want to try to summarise the counters we have discussed in this thread to
check my understanding:

1. global mTHP successful allocation counter, per mTHP size (inc only)
2. global mTHP failed allocation counter, per mTHP size (inc only)
3. global mTHP currently allocated counter, per mTHP size (inc and dec)
4. global "mTHP became partially mapped 1 or more processes" counter (inc only)

I geuss the above should apply to both page cache and anon? Do we want separate
counters for each?

I'm not sure if we would want 4. to be per mTHP or a single counter for all?
Probably the former if it provides a bit more info for neglegable cost.

Where should these be exposed? I guess /proc/vmstats is the obvious place, but I
don't think there is any precident for per-size counters (especially where the
sizes will change depending on system). Perhaps it would be better to expose
them in their per-size directories in
/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB ?


Additional to above global counters, there is a case for adding a per-process
version of 4. to smaps to consider.

--8<----

Thanks,
Ryan


> 
>>
>> This is important to us because we're keen on understanding how often
>> folios allocations fail on a system with limited memory, such as a phone.
>>
>> Presently, I've observed a success rate of under 8% for 64KiB allocations.
>> Yet, integrating Yu's TAO optimization [1] and establishing an 800MiB
>> nomerge zone on a phone with 8GiB memory, there's a substantial
>> enhancement in the success rate, reaching approximately 40%. I'm still
>> fine-tuning the optimal size for the zone.
> 
> Just as a side note:
> 
> I didn't have the capacity to comment in detail on the "new zones" proposal
> in-depth so far (I'm hoping / assume there will be discussions at LSF/MM), but
> I'm hoping we can avoid that for now and instead improve our pageblock
> infrastructure, like Johannes is trying to, to achieve similar gains.
> 
> I suspect "some things we can do with new zones we can also do with pageblocks
> inside a zone". For example, there were discussions in the past to have "sticky
> movable" pageblocks: pageblocks that may only contain movable data. One could do
> the same with "pageblocks may not contain allocations < order X" etc. So one
> could similarly optimize the memmap to some degree for these pageblocks.
> 
> IMHO we should first try making THP <= pageblock allocations more reliable, not
> using new zones, and I'm happy that Johannes et al. are doing work in that
> direction. But it's a longer discussion to be had at LSF/MM.
>