Re: [LSF/MM/BPF TOPIC] Using hardware counters to determine hot/cold pages

["Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>]

SeongJae Park <sj@kernel.org> writes:

> Hi Aneesh,
>
> On Fri, 17 Feb 2023 17:28:09 +0530 Aneesh Kumar K V <aneesh.kumar@linux.ibm.com> wrote:
>
>> PowerPC architecture (POWER10) supports a Hot/Cold page tracking
>> facility that provides access counter and access affinity details at
>> configurable page size granularity [1]. I have been looking at using
>> this counter in different areas of the kernel such as
>> 
>> 1) Page reclaim/demotion
>> 2) THP utilization
>> 3) Page promotion.
>> 
>> I have done some MGLRU integration and would like to discuss the
>> observation with the rest of the community. It is still not clear what
>> are the best ways to integrate these hardware counters in the Linux
>> kernel.
>
> Sounds very interesting.  I think DAMON might be one another option, because it
> is designed to be easy to extended with various source of access
> information[1], and provides an abstraction layer for access temparature based
> memory management[2], namely Data Access Monitoring-based Operation Schemes
> (DAMOS).
>
>> Attached is the performance graph showing how the mongodb/ycsb
>> benchmark performs when using hardware counters with MGLRU aging. An
>> early RFC version of the code can be found at
>> https://github.com/kvaneesh/linux/commit/b472e2c8080823bb4114c286270aea3e18ffe221
>> . I also expect we can get some numbers w.r.t THP usage before the
>> conference.
>
> I also have experimented a DAMON-based THP optimization[3], which shown
> interesting results.
>
> Hope to discuss about this with you at LSF/MM.  FYI, I also proposed an LSF/MM
> topic for DAMON[4].
>
> [1] https://docs.kernel.org/mm/damon/design.html#configurable-layers
> [2] https://docs.kernel.org/mm/damon/api.html#c.damos
> [3] https://www.amazon.science/publications/daos-data-access-aware-operating-system
> [4] https://lore.kernel.org/damon/20230214003328.55285-1-sj@kernel.org/
>
>

The hardware counters that are supported in the case of POWER10 are
based on physical addresses. The hardware facility will count the access
across a physical address range and there is a counter for each page
that gives the access count and also information about which node did
access the page.

I haven't spent much time studying DAMON so I might be wrong here. The
reason I avoided using DAMON for the POC was because my goal was to
evaluate how the hardware counters measured against the pte reference
bit and I was not sure I could evaluate that using the DAMON action
facility.

I do agree that we could add a layer similar to DAMON_PADDR and expose
the details to userspace. But I was not sure we can take action based on
that. In most cases what I wanted was to move the coldest page in the
Numa node to swap. So that is relative hotness rather than moving a page
that got a hotness value less than 10 to swap even though we can figure
out a way to make the latter similar to the former.


I will look at DAMON and see if that is the best framework for things
like this.

-aneesh