Follow-Up on Alexandre's Transparent Huge Pages Testing

Follow-Up on Alexandre's Transparent Huge Pages Testing

[Mark Nelson]

Hi Everyone,

A while back Alexandre Derumier posted some test results looking at how 
transparent huge pages can reduce memory usage with jemalloc.  I went 
back and ran a number of new tests on the community performance cluster 
to verify his findings and also look at how performance and cpu usage 
were affected, both during various fio benchmark tests and also during a 
4k random write recovery scenario.  I tested tcmalloc 2.4 with 32MB 
thread cache, 128MB thread cache, and jemalloc 4.0.

The gist of it is that I also see a reduction in memory usage, most 
pronounced with jemalloc.  Unfortuantely the best reduction in memory 
usage is when memory usage is already fairly low.  The most important 
case is the memory spike when OSDs are marked back up/in during a 
recovery test.  In this case there is still a benefit, though memory 
usage is still a little higher than TCMalloc with 128MB thread cache. 
There's a little bit of a concerning trend where memory usage appears to 
increase fairly quickly after the recovery test is complete and the 
post-recovery phase of the benchmark is running.  That will likely need 
to be investigate in more depth.

I have been doing some other tests with the async messenger and 
newstore, but those will have to wait for another paper.

Here's are the results:

https://drive.google.com/file/d/0B2gTBZrkrnpZY3U3TUU3RkJVeVk/view

Mark

RE: Follow-Up on Alexandre's Transparent Huge Pages Testing

[Dałek, Piotr]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Mark Nelson
> Sent: Tuesday, September 29, 2015 1:34 AM
> 
> Hi Everyone,
> 
> A while back Alexandre Derumier posted some test results looking at how
> transparent huge pages can reduce memory usage with jemalloc.  I went
> back and ran a number of new tests on the community performance cluster
> to verify his findings and also look at how performance and cpu usage were
> affected, both during various fio benchmark tests and also during a 4k
> random write recovery scenario.  I tested tcmalloc 2.4 with 32MB thread
> cache, 128MB thread cache, and jemalloc 4.0. 
> [..]
> Here's are the results:
> 
> https://drive.google.com/file/d/0B2gTBZrkrnpZY3U3TUU3RkJVeVk/view

From my point of view, this looks excellent. I generally didn't like the idea of moving to Jemalloc because of memory usage increase (which is absurdly high as you can see from the graphs), but with THP disabled things look way better.
How about trying Jemalloc 4 without THP and with Ceph patched with https://github.com/ceph/ceph/pull/5855? I wonder how much difference it'll give on Incerta.


With best regards / Pozdrawiam
Piotr Dałek

Re: Follow-Up on Alexandre's Transparent Huge Pages Testing

[Mark Nelson]

On 09/29/2015 12:59 AM, Dałek, Piotr wrote:
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Mark Nelson
>> Sent: Tuesday, September 29, 2015 1:34 AM
>>
>> Hi Everyone,
>>
>> A while back Alexandre Derumier posted some test results looking at how
>> transparent huge pages can reduce memory usage with jemalloc.  I went
>> back and ran a number of new tests on the community performance cluster
>> to verify his findings and also look at how performance and cpu usage were
>> affected, both during various fio benchmark tests and also during a 4k
>> random write recovery scenario.  I tested tcmalloc 2.4 with 32MB thread
>> cache, 128MB thread cache, and jemalloc 4.0.
>> [..]
>> Here's are the results:
>>
>> https://drive.google.com/file/d/0B2gTBZrkrnpZY3U3TUU3RkJVeVk/view
>
>  From my point of view, this looks excellent. I generally didn't like the idea of moving to Jemalloc because of memory usage increase (which is absurdly high as you can see from the graphs), but with THP disabled things look way better.
> How about trying Jemalloc 4 without THP and with Ceph patched with https://github.com/ceph/ceph/pull/5855? I wonder how much difference it'll give on Incerta.

I actually did try 5855, but there was a nasty monitor memory leak in 
master that prevented the tests from running.  It didn't get fixed until 
early last week and 5855 no longer merges cleanly with latest master.  I 
haven't had time to go back and see what's conflicting, but if you want 
to take a look and update it I would be happy to give it a whirl. :)

Mark

>
>
> With best regards / Pozdrawiam
> Piotr Dałek
>
>
> N�����r��y���b�X��ǧv�^�)޺{.n�+���z�]z���{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!tml=
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Follow-Up on Alexandre's Transparent Huge Pages Testing

[Dałek, Piotr]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Mark Nelson
> Sent: Tuesday, September 29, 2015 3:22 PM
>
> >> https://drive.google.com/file/d/0B2gTBZrkrnpZY3U3TUU3RkJVeVk/view
> >
> >  From my point of view, this looks excellent. I generally didn't like the idea
> > of moving to Jemalloc because of memory usage increase (which is absurdly
> > high as you can see from the graphs), but with THP disabled things look way
> > better.
> > How about trying Jemalloc 4 without THP and with Ceph patched with
> > https://github.com/ceph/ceph/pull/5855? I wonder how much difference it'll
> > give on Incerta.
> 
> I actually did try 5855, but there was a nasty monitor memory leak in master
> that prevented the tests from running.  It didn't get fixed until early last
> week and 5855 no longer merges cleanly with latest master.  I haven't had
> time to go back and see what's conflicting, but if you want to take a look and
> update it I would be happy to give it a whirl. :)

I will, if I won't run into any troubles. I read about mon leak being fixed so I tried to rebuild my cluster from current master (I was badly hit with that leak, as I told on #ceph-devel some time ago) and after a restart of Ceph components, only 29 (out of 98) OSDs were reporting to mons (all of them were running, though), after reverting to the code from around beginning of September, the cluster booted up fine, so either it was a one-time build hiccup, or something nasty went into master recently - I'll check again tomorrow.


With best regards / Pozdrawiam
Piotr Dałek