DAMON testing and benchmarking

DAMON testing and benchmarking

[Piyush Sachdeva]

Dear Mr. SeongJae Park,
I hope this email finds you well.

For the last few months, I have been looking at DAMON from an end-users
perspective and a developer's PoV. Most recently, I was focusing on
`lru_sort.c` module that uses the `lru_prio` and `lru_deprio` operations
which result in a more precise reclaim. In my understanding, enabling
the "lru_sort.c" module would make intelligent decisions based on the
access frequency of the pages and end up preventing hot page
swaps. Hence, when integrated with an LRU algorithm, it should improve
it.

If you could share any test/benchmark that you might have run to verify
the above assumption? I did find the result numbers you posted (link
below), but that doesn't mention the "plrus-*" scheme numbers. It also
doesn't have numbers for running the `pageout` operation on the entire
physical address space (paddr) i.e. the `pprcl` scheme. So, if you can
link those too, it would be amazing.

Can you also share any real-world (memory-management specific) workload
results that you would have used with DAMON in your experiments? Like
either MongoDB or memcached over Parsec3.0 (including splash2x) - which,
in my understanding, is less memory intensive and more architecture
inclined.

I also had a question regarding schemes - A scheme is highly tweakable,
and it's what the efficiency of DAMON rests upon. The more precise the
scheme, the more efficient DAMON will be. Hence, I'd be thankful if you
can help me derive a config that would provide the best results.
Hope to hear from you soon.

Test results: https://damonitor.github.io/test/result/perf/latest/html/

--
Regards,
Piyush Sachdeva

Re: DAMON testing and benchmarking

[SeongJae Park]

Hi Piyush,

On Tue, 13 Jun 2023 12:18:48 +0530 Piyush Sachdeva <piyushs@linux.ibm.com> wrote:

> Dear Mr. SeongJae Park,
> I hope this email finds you well.

It did, thank you for this email :)

> 
> For the last few months, I have been looking at DAMON from an end-users
> perspective and a developer's PoV. Most recently, I was focusing on
> `lru_sort.c` module that uses the `lru_prio` and `lru_deprio` operations
> which result in a more precise reclaim. In my understanding, enabling
> the "lru_sort.c" module would make intelligent decisions based on the
> access frequency of the pages and end up preventing hot page
> swaps. Hence, when integrated with an LRU algorithm, it should improve
> it.
> 
> If you could share any test/benchmark that you might have run to verify
> the above assumption?

Yes, of course.  I will share those soon.

> I did find the result numbers you posted (link below), but that doesn't
> mention the "plrus-*" scheme numbers. It also doesn't have numbers for
> running the `pageout` operation on the entire physical address space (paddr)
> i.e. the `pprcl` scheme. So, if you can link those too, it would be amazing.

We run an automated test[1] every day, against the latest damon/next tree.  And
the page you linked is the output of the test.  The latet version contains the
results from `pprcl`[2] and `plrus`[3], but I was too lazy at updating the
document, sorry.  I will try to clean up the mess as soon as possible.

> 
> Can you also share any real-world (memory-management specific) workload
> results that you would have used with DAMON in your experiments? Like
> either MongoDB or memcached over Parsec3.0 (including splash2x) - which,
> in my understanding, is less memory intensive and more architecture
> inclined.

On my personal testing setup, I'm using only parsec3 and splash2x at the
moment.  We heard some production DB system is using DAMON_RECLAIM and achieved
about 20% memory footprint reduction, though.

> 
> I also had a question regarding schemes - A scheme is highly tweakable,
> and it's what the efficiency of DAMON rests upon. The more precise the
> scheme, the more efficient DAMON will be. Hence, I'd be thankful if you
> can help me derive a config that would provide the best results.

Very good point.  Unfortunately, repeats of experience and adjustment is the
only way as of now, like other tuning practices.  Nevertheless, DAMOS supports
some safeguards such as quotas[4], watermarks[5], and filters[6].  Because
quotas feature provides prioritization, setting the access pattern a little
wide, and more focusing on tuning of the quotas might be a good practice.

I'm trying to add some more easy-to-use intuitive tuning knobs, including
feedback-based quota auto-tuning, which I shared the rough idea at LSFMM[7].

> Hope to hear from you soon.

Thank you again for this great questions.  Please feel free to ask any question
or helps :)

[1] https://github.com/awslabs/damon-tests/blob/next/perf/full_run.sh
[2] https://github.com/awslabs/damon-tests/blob/next/perf/schemes/pdarc_v4_2_2.json
[3] https://github.com/awslabs/damon-tests/blob/next/perf/schemes/plrus-2.json
[4] https://www.kernel.org/doc/html/next/mm/damon/design.html#quotas
[5] https://www.kernel.org/doc/html/next/mm/damon/design.html#watermarks
[6] https://www.kernel.org/doc/html/next/mm/damon/design.html#filters
[7] https://lwn.net/Articles/931769/


Thanks,
SJ

> 
> Test results: https://damonitor.github.io/test/result/perf/latest/html/
> 
> --
> Regards,
> Piyush Sachdeva
> 

Re: DAMON testing and benchmarking

[Piyush Sachdeva]

Dear SeongJae Park,

SeongJae Park <sj@kernel.org> writes:

> Hi Piyush,
>
> On Tue, 13 Jun 2023 12:18:48 +0530 Piyush Sachdeva <piyushs@linux.ibm.com> wrote:
>
>> Dear Mr. SeongJae Park,
>> I hope this email finds you well.
>
> It did, thank you for this email :)
>
>> 
>> For the last few months, I have been looking at DAMON from an end-users
>> perspective and a developer's PoV. Most recently, I was focusing on
>> `lru_sort.c` module that uses the `lru_prio` and `lru_deprio` operations
>> which result in a more precise reclaim. In my understanding, enabling
>> the "lru_sort.c" module would make intelligent decisions based on the
>> access frequency of the pages and end up preventing hot page
>> swaps. Hence, when integrated with an LRU algorithm, it should improve
>> it.
>> 
>> If you could share any test/benchmark that you might have run to verify
>> the above assumption?
>
> Yes, of course.  I will share those soon.
>
>> I did find the result numbers you posted (link below), but that doesn't
>> mention the "plrus-*" scheme numbers. It also doesn't have numbers for
>> running the `pageout` operation on the entire physical address space (paddr)
>> i.e. the `pprcl` scheme. So, if you can link those too, it would be amazing.
>
> We run an automated test[1] every day, against the latest damon/next tree.  And
> the page you linked is the output of the test.  The latet version contains the
> results from `pprcl`[2] and `plrus`[3], but I was too lazy at updating the
> document, sorry.  I will try to clean up the mess as soon as possible.
>
>> 
>> Can you also share any real-world (memory-management specific) workload
>> results that you would have used with DAMON in your experiments? Like
>> either MongoDB or memcached over Parsec3.0 (including splash2x) - which,
>> in my understanding, is less memory intensive and more architecture
>> inclined.
>
> On my personal testing setup, I'm using only parsec3 and splash2x at the
> moment.  We heard some production DB system is using DAMON_RECLAIM and achieved
> about 20% memory footprint reduction, though.
>

Are you aware of the specifications of the DB system that you are
mentioning? 20% is an amazing number and if possible I would like to
know more about the details of the workload.

>> 
>> I also had a question regarding schemes - A scheme is highly tweakable,
>> and it's what the efficiency of DAMON rests upon. The more precise the
>> scheme, the more efficient DAMON will be. Hence, I'd be thankful if you
>> can help me derive a config that would provide the best results.
>
> Very good point.  Unfortunately, repeats of experience and adjustment is the
> only way as of now, like other tuning practices.  Nevertheless, DAMOS supports
> some safeguards such as quotas[4], watermarks[5], and filters[6].  Because
> quotas feature provides prioritization, setting the access pattern a little
> wide, and more focusing on tuning of the quotas might be a good practice.
>
> I'm trying to add some more easy-to-use intuitive tuning knobs, including
> feedback-based quota auto-tuning, which I shared the rough idea at LSFMM[7].
>

I did go through your presentation and the summary article on lwn.net .
The feedback-mechanism based self-tuning does sound ingriguing and
promising to me.

>> Hope to hear from you soon.
>
> Thank you again for this great questions.  Please feel free to ask any question
> or helps :)
>
> [1] https://github.com/awslabs/damon-tests/blob/next/perf/full_run.sh
> [2] https://github.com/awslabs/damon-tests/blob/next/perf/schemes/pdarc_v4_2_2.json
> [3] https://github.com/awslabs/damon-tests/blob/next/perf/schemes/plrus-2.json
> [4] https://www.kernel.org/doc/html/next/mm/damon/design.html#quotas
> [5] https://www.kernel.org/doc/html/next/mm/damon/design.html#watermarks
> [6] https://www.kernel.org/doc/html/next/mm/damon/design.html#filters
> [7] https://lwn.net/Articles/931769/
>
>
> Thanks,
> SJ
>
>> 
>> Test results: https://damonitor.github.io/test/result/perf/latest/html/
>> 
>> --
>> Regards,
>> Piyush Sachdeva
>>

Thank you for the information and all the links. 

--
Regards,
Piyush Sachdeva

Re: DAMON testing and benchmarking

[SeongJae Park]

Hi Piyush,


Sorry for late response.  Somehow I forgot replying.

On Fri, 16 Jun 2023 12:22:50 +0530 Piyush Sachdeva <piyushs@linux.vnet.ibm.com> wrote:

> Dear SeongJae Park,
> 
> SeongJae Park <sj@kernel.org> writes:
> 
> > Hi Piyush,
> >
> > On Tue, 13 Jun 2023 12:18:48 +0530 Piyush Sachdeva <piyushs@linux.ibm.com> wrote:
> >
[...]
> >> For the last few months, I have been looking at DAMON from an end-users
> >> perspective and a developer's PoV. Most recently, I was focusing on
> >> `lru_sort.c` module that uses the `lru_prio` and `lru_deprio` operations
> >> which result in a more precise reclaim. In my understanding, enabling
> >> the "lru_sort.c" module would make intelligent decisions based on the
> >> access frequency of the pages and end up preventing hot page
> >> swaps. Hence, when integrated with an LRU algorithm, it should improve
> >> it.
> >> 
> >> If you could share any test/benchmark that you might have run to verify
> >> the above assumption?
> >
> > Yes, of course.  I will share those soon.
> >
> >> I did find the result numbers you posted (link below), but that doesn't
> >> mention the "plrus-*" scheme numbers. It also doesn't have numbers for
> >> running the `pageout` operation on the entire physical address space (paddr)
> >> i.e. the `pprcl` scheme. So, if you can link those too, it would be amazing.
> >
> > We run an automated test[1] every day, against the latest damon/next tree.  And
> > the page you linked is the output of the test.  The latet version contains the
> > results from `pprcl`[2] and `plrus`[3], but I was too lazy at updating the
> > document, sorry.  I will try to clean up the mess as soon as possible.

I just uploaded my daily DAMON performance test results that I collected since
2022-06-10 on the DAMON project web site[0].  It may better to add some more
visualization such as weekly rolling average of the report in future.  I
actually wanted to do that, but I was too lazy.  So sharing the data first.  I
will try to further make it easier to read.

> >
> >> 
> >> Can you also share any real-world (memory-management specific) workload
> >> results that you would have used with DAMON in your experiments? Like
> >> either MongoDB or memcached over Parsec3.0 (including splash2x) - which,
> >> in my understanding, is less memory intensive and more architecture
> >> inclined.
> >
> > On my personal testing setup, I'm using only parsec3 and splash2x at the
> > moment.  We heard some production DB system is using DAMON_RECLAIM and achieved
> > about 20% memory footprint reduction, though.
> >
> 
> Are you aware of the specifications of the DB system that you are
> mentioning? 20% is an amazing number and if possible I would like to
> know more about the details of the workload.

It is a relational database system.  Unfortunately I also not having very
detailed information about the product.  ByteDance also recently shared[1]
their DAMON-based 30 % memory reduction for MySQL.

> 
> >> 
> >> I also had a question regarding schemes - A scheme is highly tweakable,
> >> and it's what the efficiency of DAMON rests upon. The more precise the
> >> scheme, the more efficient DAMON will be. Hence, I'd be thankful if you
> >> can help me derive a config that would provide the best results.
> >
> > Very good point.  Unfortunately, repeats of experience and adjustment is the
> > only way as of now, like other tuning practices.  Nevertheless, DAMOS supports
> > some safeguards such as quotas[4], watermarks[5], and filters[6].  Because
> > quotas feature provides prioritization, setting the access pattern a little
> > wide, and more focusing on tuning of the quotas might be a good practice.
> >
> > I'm trying to add some more easy-to-use intuitive tuning knobs, including
> > feedback-based quota auto-tuning, which I shared the rough idea at LSFMM[7].
> >
> 
> I did go through your presentation and the summary article on lwn.net .
> The feedback-mechanism based self-tuning does sound ingriguing and
> promising to me.

The patches for core part of the feature has recently posted, and currently
merged into the mm tree[2].  Hopefully it will be merged into the mainline in
the next merge window.  The patches require users' intervention though.  A
followup patches for making it almost self-tuned by DAMON will hopefully posted
soon.

Apology again for this too late response.  Please feel free to ask any question
or help.

[0] https://damonitor.github.io/test/result/perf/index.html
[1] https://lpc.events/event/17/contributions/1520/
[2] https://lore.kernel.org/damon/20231130023652.50284-1-sj@kernel.org/


Thanks,
SJ

[...]