Re: [LSF/MM/BPF TOPIC] Locally attached memory tiering

["Huang, Ying" <ying.huang@intel.com>]

Davidlohr Bueso <dave@stgolabs.net> writes:

> On Thu, 09 May 2024, Huang, Ying wrote:
>
>>With the default configuration, current NUMA balancing based promotion
>>solution will almost try to promote any faulting pages.  To select hot
>>pages to promote and control thrashing between NUMA nodes, the promote
>>rate limit needs to be configured.  For example, via,
>>
>>echo 200 > /proc/sys/kernel/numa_balancing_promote_rate_limit_MBps
>>
>>200MB hot pages will be selected and promoted every second.  Can you try it?
>
> Yes, I've played with this tunnable and, just like the LRU approach, it
> shows nice micro wins (less amount of promotions/demotions) but little for
> actual benchmark improvements at a higher level, merely noise level or
> very sublte wins. In fact, the actual data from that series for this
> parameter was a ~2% pmbench win with the rate limiting, but a 69% promotion
> rate descrease.

Thanks a lot for update!

IIUC, page promotion/demotion only helps performance if there are hot
pages in the slow memory and cold pages in the hot memory.  This may be
not true for quite some workloads configurations.

For example, the default allocation mechanism is local first.  In the
context of memory tiering, it's the fast memory first.  In various
workloads, it's quite normal that hot pages will be allocated firstly.
This makes it unnecessary to optimize the page placement until there's
some configuration changes in the system.

So, to evaluate the optimization, we need to

1) check the overhead of the optimization when page placement is almost
optimal already.

2) find configurations where the page placement isn't good enough, and
check whether memory tiering optimization works.

> And this is really my point, how much effort do we want to put in optimizing
> software mechanisms for hot page detection? Are there other benchmarks we
> should be using? And perhaps doing the async promotion and not incurring in
> the numa balancing overhead and comparing the cost of migration before
> promoting would yield some better numbers, but that also might be easy to
> get wrong when compared to the relative hotness of the page.

I believe that there are still quite some spaces to optimize the
software mechanisms.  The current implementation is as simple as
possible in fact.

--
Best Regards,
Huang, Ying