Initial performance cluster SimpleMessenger vs AsyncMessenger results

Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Mark Nelson]

Hi Guy,

Given all of the recent data on how different memory allocator 
configurations improve SimpleMessenger performance (and the effect of 
memory allocators and transparent hugepages on RSS memory usage), I 
thought I'd run some tests looking how AsyncMessenger does in 
comparison.  We spoke about these a bit at the last performance meeting 
but here's the full write up.  The rough conclusion as of right now 
appears to be:

1) AsyncMessenger performance is not dependent on the memory allocator 
like with SimpleMessenger.

2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB 
(ie default) thread cache.

3) AsyncMessenger is consistently faster than SimpleMessenger for 128K 
random reads.

4) AsyncMessenger is sometimes slower than SimpleMessenger when memory 
allocator optimizations are used.

5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.

Here's a link to the paper:

https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view

Mark

Re: Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Haomai Wang]

[-- Attachment #1.1: Type: text/plain, Size: 1523 bytes --]

COOL

Interesting that async messenger will consume more memory than simple, in
my mind I always think async should use less memory. I will give a look at
this

On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Hi Guy,
>
> Given all of the recent data on how different memory allocator
> configurations improve SimpleMessenger performance (and the effect of
> memory allocators and transparent hugepages on RSS memory usage), I thought
> I'd run some tests looking how AsyncMessenger does in comparison.  We spoke
> about these a bit at the last performance meeting but here's the full write
> up.  The rough conclusion as of right now appears to be:
>
> 1) AsyncMessenger performance is not dependent on the memory allocator
> like with SimpleMessenger.
>
> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
> default) thread cache.
>
> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
> random reads.
>
> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
> allocator optimizations are used.
>
> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>
> Here's a link to the paper:
>
> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>
> Mark
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Best Regards,

Wheat

[-- Attachment #1.2: Type: text/html, Size: 2313 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Haomai Wang]

resend

On Tue, Oct 13, 2015 at 10:56 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
> COOL
>
> Interesting that async messenger will consume more memory than simple, in my
> mind I always think async should use less memory. I will give a look at this
>
> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com> wrote:
>>
>> Hi Guy,
>>
>> Given all of the recent data on how different memory allocator
>> configurations improve SimpleMessenger performance (and the effect of memory
>> allocators and transparent hugepages on RSS memory usage), I thought I'd run
>> some tests looking how AsyncMessenger does in comparison.  We spoke about
>> these a bit at the last performance meeting but here's the full write up.
>> The rough conclusion as of right now appears to be:
>>
>> 1) AsyncMessenger performance is not dependent on the memory allocator
>> like with SimpleMessenger.
>>
>> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
>> default) thread cache.
>>
>> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
>> random reads.
>>
>> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
>> allocator optimizations are used.
>>
>> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>>
>> Here's a link to the paper:
>>
>> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>>
>> Mark
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> --
>
> Best Regards,
>
> Wheat



-- 
Best Regards,

Wheat

Re: Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Sage Weil]

On Tue, 13 Oct 2015, Haomai Wang wrote:
> resend
> 
> On Tue, Oct 13, 2015 at 10:56 AM, Haomai Wang <haomaiwang-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > COOL
> >
> > Interesting that async messenger will consume more memory than simple, in my
> > mind I always think async should use less memory. I will give a look at this

Yeah.. I was surprised to see this as well.  The other conclusions 
are quite promising, though!

Note that we still have a few outstanding issues with async msgr that 
cropped up during the infernalis testing, so for now we're doing 
just simplemessenger to get infernalis out the door.  That gives us the 
balance of this next cycle until Jewel to shake out the remaining issues 
so that hopefully we can make this the default for Jewel!

sage


> >
> > On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >>
> >> Hi Guy,
> >>
> >> Given all of the recent data on how different memory allocator
> >> configurations improve SimpleMessenger performance (and the effect of memory
> >> allocators and transparent hugepages on RSS memory usage), I thought I'd run
> >> some tests looking how AsyncMessenger does in comparison.  We spoke about
> >> these a bit at the last performance meeting but here's the full write up.
> >> The rough conclusion as of right now appears to be:
> >>
> >> 1) AsyncMessenger performance is not dependent on the memory allocator
> >> like with SimpleMessenger.
> >>
> >> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
> >> default) thread cache.
> >>
> >> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
> >> random reads.
> >>
> >> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
> >> allocator optimizations are used.
> >>
> >> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
> >>
> >> Here's a link to the paper:
> >>
> >> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
> >>
> >> Mark
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> >
> > --
> >
> > Best Regards,
> >
> > Wheat
> 
> 
> 
> -- 
> Best Regards,
> 
> Wheat
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Mark Nelson]

Hi Haomai,

Great!  I haven't had a chance to dig in and look at it with valgrind 
yet, but if I get a chance after I'm done with newstore fragment testing 
and somnath's writepath work I'll try to go back and dig in if you 
haven't had a chance yet.

Mark

On 10/12/2015 09:56 PM, Haomai Wang wrote:
> resend
>
> On Tue, Oct 13, 2015 at 10:56 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> COOL
>>
>> Interesting that async messenger will consume more memory than simple, in my
>> mind I always think async should use less memory. I will give a look at this
>>
>> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com> wrote:
>>>
>>> Hi Guy,
>>>
>>> Given all of the recent data on how different memory allocator
>>> configurations improve SimpleMessenger performance (and the effect of memory
>>> allocators and transparent hugepages on RSS memory usage), I thought I'd run
>>> some tests looking how AsyncMessenger does in comparison.  We spoke about
>>> these a bit at the last performance meeting but here's the full write up.
>>> The rough conclusion as of right now appears to be:
>>>
>>> 1) AsyncMessenger performance is not dependent on the memory allocator
>>> like with SimpleMessenger.
>>>
>>> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
>>> default) thread cache.
>>>
>>> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
>>> random reads.
>>>
>>> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
>>> allocator optimizations are used.
>>>
>>> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>>>
>>> Here's a link to the paper:
>>>
>>> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>>>
>>> Mark
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>>
>> --
>>
>> Best Regards,
>>
>> Wheat
>
>
>

Re: Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Chen, Xiaoxi]

Hi Mark,
     The Async result  in 128K drops quickly after some point, is that because of the testing methodology?
      
     Other conclusion looks to me like simple messenger + Jemalloc is the best practice till now as it has the same performance as async but using much less memory?

-Xiaoxi

> -----Original Message-----
> From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On Behalf Of
> Mark Nelson
> Sent: Tuesday, October 13, 2015 9:03 PM
> To: Haomai Wang
> Cc: ceph-devel; ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger vs
> AsyncMessenger results
> 
> Hi Haomai,
> 
> Great!  I haven't had a chance to dig in and look at it with valgrind yet, but if I
> get a chance after I'm done with newstore fragment testing and somnath's
> writepath work I'll try to go back and dig in if you haven't had a chance yet.
> 
> Mark
> 
> On 10/12/2015 09:56 PM, Haomai Wang wrote:
> > resend
> >
> > On Tue, Oct 13, 2015 at 10:56 AM, Haomai Wang <haomaiwang-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> wrote:
> >> COOL
> >>
> >> Interesting that async messenger will consume more memory than
> >> simple, in my mind I always think async should use less memory. I
> >> will give a look at this
> >>
> >> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> wrote:
> >>>
> >>> Hi Guy,
> >>>
> >>> Given all of the recent data on how different memory allocator
> >>> configurations improve SimpleMessenger performance (and the effect
> >>> of memory allocators and transparent hugepages on RSS memory usage),
> >>> I thought I'd run some tests looking how AsyncMessenger does in
> >>> comparison.  We spoke about these a bit at the last performance
> meeting but here's the full write up.
> >>> The rough conclusion as of right now appears to be:
> >>>
> >>> 1) AsyncMessenger performance is not dependent on the memory
> >>> allocator like with SimpleMessenger.
> >>>
> >>> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc +
> >>> 32MB (ie
> >>> default) thread cache.
> >>>
> >>> 3) AsyncMessenger is consistently faster than SimpleMessenger for
> >>> 128K random reads.
> >>>
> >>> 4) AsyncMessenger is sometimes slower than SimpleMessenger when
> >>> memory allocator optimizations are used.
> >>>
> >>> 5) AsyncMessenger currently uses far more RSS memory than
> SimpleMessenger.
> >>>
> >>> Here's a link to the paper:
> >>>
> >>> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
> >>>
> >>> Mark
> >>> _______________________________________________
> >>> ceph-users mailing list
> >>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Best Regards,
> >>
> >> Wheat
> >
> >
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Mark Nelson]

Hi Xiaoxi,

I would ignore the tails on those tests.  I suspect it's just some fio 
processes finishing earlier than others and the associated aggregate 
performance dropping off.  These reads tests are so fast that my 
original guess at reasonable volume sizes for 300 second tests appear to 
be off.

Mark

On 10/14/2015 10:57 AM, Chen, Xiaoxi wrote:
> Hi Mark,
>       The Async result  in 128K drops quickly after some point, is that because of the testing methodology?
>
>       Other conclusion looks to me like simple messenger + Jemalloc is the best practice till now as it has the same performance as async but using much less memory?
>
> -Xiaoxi
>
>> -----Original Message-----
>> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf Of
>> Mark Nelson
>> Sent: Tuesday, October 13, 2015 9:03 PM
>> To: Haomai Wang
>> Cc: ceph-devel; ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger vs
>> AsyncMessenger results
>>
>> Hi Haomai,
>>
>> Great!  I haven't had a chance to dig in and look at it with valgrind yet, but if I
>> get a chance after I'm done with newstore fragment testing and somnath's
>> writepath work I'll try to go back and dig in if you haven't had a chance yet.
>>
>> Mark
>>
>> On 10/12/2015 09:56 PM, Haomai Wang wrote:
>>> resend
>>>
>>> On Tue, Oct 13, 2015 at 10:56 AM, Haomai Wang <haomaiwang@gmail.com>
>> wrote:
>>>> COOL
>>>>
>>>> Interesting that async messenger will consume more memory than
>>>> simple, in my mind I always think async should use less memory. I
>>>> will give a look at this
>>>>
>>>> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com>
>> wrote:
>>>>>
>>>>> Hi Guy,
>>>>>
>>>>> Given all of the recent data on how different memory allocator
>>>>> configurations improve SimpleMessenger performance (and the effect
>>>>> of memory allocators and transparent hugepages on RSS memory usage),
>>>>> I thought I'd run some tests looking how AsyncMessenger does in
>>>>> comparison.  We spoke about these a bit at the last performance
>> meeting but here's the full write up.
>>>>> The rough conclusion as of right now appears to be:
>>>>>
>>>>> 1) AsyncMessenger performance is not dependent on the memory
>>>>> allocator like with SimpleMessenger.
>>>>>
>>>>> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc +
>>>>> 32MB (ie
>>>>> default) thread cache.
>>>>>
>>>>> 3) AsyncMessenger is consistently faster than SimpleMessenger for
>>>>> 128K random reads.
>>>>>
>>>>> 4) AsyncMessenger is sometimes slower than SimpleMessenger when
>>>>> memory allocator optimizations are used.
>>>>>
>>>>> 5) AsyncMessenger currently uses far more RSS memory than
>> SimpleMessenger.
>>>>>
>>>>> Here's a link to the paper:
>>>>>
>>>>> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>>>>>
>>>>> Mark
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Best Regards,
>>>>
>>>> Wheat
>>>
>>>
>>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Somnath Roy]

[-- Attachment #1.1: Type: text/plain, Size: 3148 bytes --]

Mark,
Thanks for this data. This means probably simple messenger (not OSD core) is not doing optimal job of handling memory.

Haomai,
I am not that familiar with Async messenger code base, do you have an explanation of the behavior (like good performance with default tcmalloc) Mark reported ? Is it using lot less thread overall than Simple ?
Also, it seems Async messenger has some inefficiencies in the io path and that’s why it is not performing as well as simple if the memory allocation stuff is optimally handled.
Could you please send out any documentation around Async messenger ? I tried to google it , but, not even blueprint is popping up.


Thanks & Regards
Somnath
From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf Of Haomai Wang
Sent: Monday, October 12, 2015 7:57 PM
To: Mark Nelson
Cc: ceph-devel; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

COOL

Interesting that async messenger will consume more memory than simple, in my mind I always think async should use less memory. I will give a look at this

On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com<mailto:mnelson@redhat.com>> wrote:
Hi Guy,

Given all of the recent data on how different memory allocator configurations improve SimpleMessenger performance (and the effect of memory allocators and transparent hugepages on RSS memory usage), I thought I'd run some tests looking how AsyncMessenger does in comparison.  We spoke about these a bit at the last performance meeting but here's the full write up.  The rough conclusion as of right now appears to be:

1) AsyncMessenger performance is not dependent on the memory allocator like with SimpleMessenger.

2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie default) thread cache.

3) AsyncMessenger is consistently faster than SimpleMessenger for 128K random reads.

4) AsyncMessenger is sometimes slower than SimpleMessenger when memory allocator optimizations are used.

5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.

Here's a link to the paper:

https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view

Mark
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--

Best Regards,

Wheat

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).


[-- Attachment #1.2: Type: text/html, Size: 7748 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Haomai Wang]

On Tue, Oct 13, 2015 at 12:18 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
> Mark,
>
> Thanks for this data. This means probably simple messenger (not OSD core) is
> not doing optimal job of handling memory.
>
>
>
> Haomai,
>
> I am not that familiar with Async messenger code base, do you have an
> explanation of the behavior (like good performance with default tcmalloc)
> Mark reported ? Is it using lot less thread overall than Simple ?

Originally async messenger mainly want to solve with high thread
number problem which limited the ceph cluster size. High context
switch and cpu usage caused by simple messenger under large cluster.

Recently we have memory problem discussed on ML and I also spend times
to think about the root cause. Currently I would like to consider the
simple messenger's memory usage is deviating from the design of
tcmalloc. Tcmalloc is aimed to provide memory with local cache, and it
also has memory control among all threads, if we have too much
threads, it may let tcmalloc busy with memory lock contention.

Async messenger uses thread pool to serve connections, it make all
blocking calls in simple messenger async.

>
> Also, it seems Async messenger has some inefficiencies in the io path and
> that’s why it is not performing as well as simple if the memory allocation
> stuff is optimally handled.

Yep, simple messenger use two threads(one for read, one for write) to
serve one connection, async messenger at most have one thread to serve
one connection and multi connection  will share the same thread.

Next, I would like to have several plans to improve performance:
1. add poll mode support, I hope it can help enhance high performance
storage need
2. add load balance ability among worker threads
3. move more works out of messenger thread.

>
> Could you please send out any documentation around Async messenger ? I tried
> to google it , but, not even blueprint is popping up.

>
>
>
>
>
> Thanks & Regards
>
> Somnath
>
> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf Of
> Haomai Wang
> Sent: Monday, October 12, 2015 7:57 PM
> To: Mark Nelson
> Cc: ceph-devel; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger vs
> AsyncMessenger results
>
>
>
> COOL
>
>
>
> Interesting that async messenger will consume more memory than simple, in my
> mind I always think async should use less memory. I will give a look at this
>
>
>
> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com> wrote:
>
> Hi Guy,
>
> Given all of the recent data on how different memory allocator
> configurations improve SimpleMessenger performance (and the effect of memory
> allocators and transparent hugepages on RSS memory usage), I thought I'd run
> some tests looking how AsyncMessenger does in comparison.  We spoke about
> these a bit at the last performance meeting but here's the full write up.
> The rough conclusion as of right now appears to be:
>
> 1) AsyncMessenger performance is not dependent on the memory allocator like
> with SimpleMessenger.
>
> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
> default) thread cache.
>
> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
> random reads.
>
> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
> allocator optimizations are used.
>
> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>
> Here's a link to the paper:
>
> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>
> Mark
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
> --
>
> Best Regards,
>
> Wheat
>
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is
> intended only for the use of the designated recipient(s) named above. If the
> reader of this message is not the intended recipient, you are hereby
> notified that you have received this message in error and that any review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by telephone or e-mail (as shown above) immediately and destroy
> any and all copies of this message in your possession (whether hard copies
> or electronically stored copies).
>



-- 
Best Regards,

Wheat
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Somnath Roy]

Thanks Haomai..
Since Async messenger is always using a constant number of threads , there could be a potential performance problem of scaling up the client connections keeping the constant number of OSDs ?
May be it's a good tradeoff..

Regards
Somnath


-----Original Message-----
From: Haomai Wang [mailto:haomaiwang@gmail.com]
Sent: Monday, October 12, 2015 11:35 PM
To: Somnath Roy
Cc: Mark Nelson; ceph-devel; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

On Tue, Oct 13, 2015 at 12:18 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
> Mark,
>
> Thanks for this data. This means probably simple messenger (not OSD
> core) is not doing optimal job of handling memory.
>
>
>
> Haomai,
>
> I am not that familiar with Async messenger code base, do you have an
> explanation of the behavior (like good performance with default
> tcmalloc) Mark reported ? Is it using lot less thread overall than Simple ?

Originally async messenger mainly want to solve with high thread number problem which limited the ceph cluster size. High context switch and cpu usage caused by simple messenger under large cluster.

Recently we have memory problem discussed on ML and I also spend times to think about the root cause. Currently I would like to consider the simple messenger's memory usage is deviating from the design of tcmalloc. Tcmalloc is aimed to provide memory with local cache, and it also has memory control among all threads, if we have too much threads, it may let tcmalloc busy with memory lock contention.

Async messenger uses thread pool to serve connections, it make all blocking calls in simple messenger async.

>
> Also, it seems Async messenger has some inefficiencies in the io path
> and that’s why it is not performing as well as simple if the memory
> allocation stuff is optimally handled.

Yep, simple messenger use two threads(one for read, one for write) to serve one connection, async messenger at most have one thread to serve one connection and multi connection  will share the same thread.

Next, I would like to have several plans to improve performance:
1. add poll mode support, I hope it can help enhance high performance storage need 2. add load balance ability among worker threads 3. move more works out of messenger thread.

>
> Could you please send out any documentation around Async messenger ? I
> tried to google it , but, not even blueprint is popping up.

>
>
>
>
>
> Thanks & Regards
>
> Somnath
>
> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf
> Of Haomai Wang
> Sent: Monday, October 12, 2015 7:57 PM
> To: Mark Nelson
> Cc: ceph-devel; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger
> vs AsyncMessenger results
>
>
>
> COOL
>
>
>
> Interesting that async messenger will consume more memory than simple,
> in my mind I always think async should use less memory. I will give a
> look at this
>
>
>
> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com> wrote:
>
> Hi Guy,
>
> Given all of the recent data on how different memory allocator
> configurations improve SimpleMessenger performance (and the effect of
> memory allocators and transparent hugepages on RSS memory usage), I
> thought I'd run some tests looking how AsyncMessenger does in
> comparison.  We spoke about these a bit at the last performance meeting but here's the full write up.
> The rough conclusion as of right now appears to be:
>
> 1) AsyncMessenger performance is not dependent on the memory allocator
> like with SimpleMessenger.
>
> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB
> (ie
> default) thread cache.
>
> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
> random reads.
>
> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
> allocator optimizations are used.
>
> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>
> Here's a link to the paper:
>
> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>
> Mark
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
> --
>
> Best Regards,
>
> Wheat
>
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message
> is intended only for the use of the designated recipient(s) named
> above. If the reader of this message is not the intended recipient,
> you are hereby notified that you have received this message in error
> and that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically stored copies).
>



--
Best Regards,

Wheat

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Haomai Wang]

Yep, as I said below, I consider to add auto scale up/down for worker
threads with connection load balance ability. It may let users not
entangled with how much thread number I need. :-(

Actually thread number for config value is a pain in ceph osd io stack.....

On Tue, Oct 13, 2015 at 2:45 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
> Thanks Haomai..
> Since Async messenger is always using a constant number of threads , there could be a potential performance problem of scaling up the client connections keeping the constant number of OSDs ?
> May be it's a good tradeoff..
>
> Regards
> Somnath
>
>
> -----Original Message-----
> From: Haomai Wang [mailto:haomaiwang@gmail.com]
> Sent: Monday, October 12, 2015 11:35 PM
> To: Somnath Roy
> Cc: Mark Nelson; ceph-devel; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results
>
> On Tue, Oct 13, 2015 at 12:18 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
>> Mark,
>>
>> Thanks for this data. This means probably simple messenger (not OSD
>> core) is not doing optimal job of handling memory.
>>
>>
>>
>> Haomai,
>>
>> I am not that familiar with Async messenger code base, do you have an
>> explanation of the behavior (like good performance with default
>> tcmalloc) Mark reported ? Is it using lot less thread overall than Simple ?
>
> Originally async messenger mainly want to solve with high thread number problem which limited the ceph cluster size. High context switch and cpu usage caused by simple messenger under large cluster.
>
> Recently we have memory problem discussed on ML and I also spend times to think about the root cause. Currently I would like to consider the simple messenger's memory usage is deviating from the design of tcmalloc. Tcmalloc is aimed to provide memory with local cache, and it also has memory control among all threads, if we have too much threads, it may let tcmalloc busy with memory lock contention.
>
> Async messenger uses thread pool to serve connections, it make all blocking calls in simple messenger async.
>
>>
>> Also, it seems Async messenger has some inefficiencies in the io path
>> and that’s why it is not performing as well as simple if the memory
>> allocation stuff is optimally handled.
>
> Yep, simple messenger use two threads(one for read, one for write) to serve one connection, async messenger at most have one thread to serve one connection and multi connection  will share the same thread.
>
> Next, I would like to have several plans to improve performance:
> 1. add poll mode support, I hope it can help enhance high performance storage need 2. add load balance ability among worker threads 3. move more works out of messenger thread.
>
>>
>> Could you please send out any documentation around Async messenger ? I
>> tried to google it , but, not even blueprint is popping up.
>
>>
>>
>>
>>
>>
>> Thanks & Regards
>>
>> Somnath
>>
>> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf
>> Of Haomai Wang
>> Sent: Monday, October 12, 2015 7:57 PM
>> To: Mark Nelson
>> Cc: ceph-devel; ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Initial performance cluster SimpleMessenger
>> vs AsyncMessenger results
>>
>>
>>
>> COOL
>>
>>
>>
>> Interesting that async messenger will consume more memory than simple,
>> in my mind I always think async should use less memory. I will give a
>> look at this
>>
>>
>>
>> On Tue, Oct 13, 2015 at 12:50 AM, Mark Nelson <mnelson@redhat.com> wrote:
>>
>> Hi Guy,
>>
>> Given all of the recent data on how different memory allocator
>> configurations improve SimpleMessenger performance (and the effect of
>> memory allocators and transparent hugepages on RSS memory usage), I
>> thought I'd run some tests looking how AsyncMessenger does in
>> comparison.  We spoke about these a bit at the last performance meeting but here's the full write up.
>> The rough conclusion as of right now appears to be:
>>
>> 1) AsyncMessenger performance is not dependent on the memory allocator
>> like with SimpleMessenger.
>>
>> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB
>> (ie
>> default) thread cache.
>>
>> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
>> random reads.
>>
>> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
>> allocator optimizations are used.
>>
>> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>>
>> Here's a link to the paper:
>>
>> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>>
>> Mark
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>>
>>
>> --
>>
>> Best Regards,
>>
>> Wheat
>>
>>
>> ________________________________
>>
>> PLEASE NOTE: The information contained in this electronic mail message
>> is intended only for the use of the designated recipient(s) named
>> above. If the reader of this message is not the intended recipient,
>> you are hereby notified that you have received this message in error
>> and that any review, dissemination, distribution, or copying of this
>> message is strictly prohibited. If you have received this
>> communication in error, please notify the sender by telephone or
>> e-mail (as shown above) immediately and destroy any and all copies of
>> this message in your possession (whether hard copies or electronically stored copies).
>>
>
>
>
> --
> Best Regards,
>
> Wheat
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>



-- 
Best Regards,

Wheat
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ceph-users] Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Dałek, Piotr]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Tuesday, October 13, 2015 8:46 AM
> 
> Thanks Haomai..
> Since Async messenger is always using a constant number of threads , there
> could be a potential performance problem of scaling up the client
> connections keeping the constant number of OSDs ?
> May be it's a good tradeoff..

It's not that big issue when you look realistically at it. In fact, having more threads than around 2 * available_logical_cpus is going to drag performance down, so it's better to have thread wait than make it forcing context switches. The point of using more threads per process is to have it spend less time waiting for I/O and better utilize current multi-core CPUs. Having threads fighting for CPU and/or I/O time is worse than having them underutilized, which is particularly true with spinning drives (which aren't going anywhere any soon; not every customer is going to accept $1700 price tag per drive that has only 800GB of capacity) and slower CPUs (again, not every customer is going to accept $1200 price tag per CPU).

With best regards / Pozdrawiam
Piotr Dałek

Re: Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Gregory Farnum]

On Mon, Oct 12, 2015 at 9:50 AM, Mark Nelson <mnelson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> Hi Guy,
>
> Given all of the recent data on how different memory allocator
> configurations improve SimpleMessenger performance (and the effect of memory
> allocators and transparent hugepages on RSS memory usage), I thought I'd run
> some tests looking how AsyncMessenger does in comparison.  We spoke about
> these a bit at the last performance meeting but here's the full write up.
> The rough conclusion as of right now appears to be:
>
> 1) AsyncMessenger performance is not dependent on the memory allocator like
> with SimpleMessenger.
>
> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
> default) thread cache.
>
> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
> random reads.
>
> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
> allocator optimizations are used.
>
> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>
> Here's a link to the paper:
>
> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view

Can you clarify these tests a bit more? I can't make the number of
nodes, OSDs, and SSDs work out properly. Were the FIO jobs 256
concurrent ops per job, or in aggregate? Is there any more info that
might suggest why the 128KB rand-read (but not read nor write, and not
4k rand-read) was so asymmetrical?

Re: Initial performance cluster SimpleMessenger vs AsyncMessenger results

[Mark Nelson]

On 10/12/2015 11:12 PM, Gregory Farnum wrote:
> On Mon, Oct 12, 2015 at 9:50 AM, Mark Nelson <mnelson@redhat.com> wrote:
>> Hi Guy,
>>
>> Given all of the recent data on how different memory allocator
>> configurations improve SimpleMessenger performance (and the effect of memory
>> allocators and transparent hugepages on RSS memory usage), I thought I'd run
>> some tests looking how AsyncMessenger does in comparison.  We spoke about
>> these a bit at the last performance meeting but here's the full write up.
>> The rough conclusion as of right now appears to be:
>>
>> 1) AsyncMessenger performance is not dependent on the memory allocator like
>> with SimpleMessenger.
>>
>> 2) AsyncMessenger is faster than SimpleMessenger with TCMalloc + 32MB (ie
>> default) thread cache.
>>
>> 3) AsyncMessenger is consistently faster than SimpleMessenger for 128K
>> random reads.
>>
>> 4) AsyncMessenger is sometimes slower than SimpleMessenger when memory
>> allocator optimizations are used.
>>
>> 5) AsyncMessenger currently uses far more RSS memory than SimpleMessenger.
>>
>> Here's a link to the paper:
>>
>> https://drive.google.com/file/d/0B2gTBZrkrnpZS1Q4VktjZkhrNHc/view
>
> Can you clarify these tests a bit more? I can't make the number of
> nodes, OSDs, and SSDs work out properly. Were the FIO jobs 256
> concurrent ops per job, or in aggregate? Is there any more info that
> might suggest why the 128KB rand-read (but not read nor write, and not
> 4k rand-read) was so asymmetrical?
>

Hi Greg,

Resending this to the list for posterity as I realized I only sent it 
you earlier:

- 4 Nodes
- 4 P3700s per node
- 4 OSDs per P3700 (Similar to Intel's setup in Jiangang and Jian's paper)

Each node also acted as an fio client using the librbd engine:

- 4 Nodes
- 2 volumes per node
- 1 fio process per volume
- 32 concurrent IOs per fio process

The 128KB random read results are interesting.  In memory allocator 
tests I saw performance decrease with more threadcache or when TCMalloc 
was used, and in the past I've seen odd performance characteristics 
around this IO size.  I think it must be a difficult case for the memory 
allocator to handle consistently well and AsyncMesseneger maybe just 
sidesteps the problem.

Mark