tcmalloc issue

tcmalloc issue

[Somnath Roy]

Sage,
Here is the tcmalloc issue I was talking in the performance meeting.

http://code.google.com/p/gperftools/issues/detail?id=585

The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top  trace.

25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans()
12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)
11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
1.56% [kernel] [k] __copy_user_nocache
1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int)
1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long)


The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release.
See the following link.

http://code.google.com/p/gperftools/

We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4.

It will be really helpful if you can push them to incorporate this in 14.04 updates.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: tcmalloc issue

[Sage Weil]

Hi James,

Is it possible to get tcmalloc in trusty updated?  See below.

Thanks!
sage

On Thu, 26 Mar 2015, Somnath Roy wrote:

> Sage,
> Here is the tcmalloc issue I was talking in the performance meeting.
> 
> http://code.google.com/p/gperftools/issues/detail?id=585
> 
> The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top  trace.
> 
> 25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans()
> 12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)
> 11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
> 1.56% [kernel] [k] __copy_user_nocache
> 1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int)
> 1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
> 0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long)
> 
> 
> The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release.
> See the following link.
> 
> http://code.google.com/p/gperftools/
> 
> We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4.
> 
> It will be really helpful if you can push them to incorporate this in 14.04 updates.
> 
> Thanks & Regards
> Somnath
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: tcmalloc issue

[Somnath Roy]

Hi James,
Could you please let us know if this is possible ?
If so, any timeline would be very helpful to plan our product release.

Thanks & Regards
Somnath


-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net] 
Sent: Thursday, March 26, 2015 4:47 PM
To: james.page@ubuntu.com; ceph-maintainers@ceph.com
Cc: Somnath Roy; ceph-devel@vger.kernel.org
Subject: Re: tcmalloc issue

Hi James,

Is it possible to get tcmalloc in trusty updated?  See below.

Thanks!
sage

On Thu, 26 Mar 2015, Somnath Roy wrote:

> Sage,
> Here is the tcmalloc issue I was talking in the performance meeting.
> 
> http://code.google.com/p/gperftools/issues/detail?id=585
> 
> The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top  trace.
> 
> 25.73% libtcmalloc.so.4.1.2 [.] 
> tcmalloc::CentralFreeList::FetchFromSpans()
> 12.52% libtcmalloc.so.4.1.2 [.] 
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr
> eeList*, unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.] 
> tcmalloc::CentralFreeList::ReleaseToSpans(void*)
> 1.56% [kernel] [k] __copy_user_nocache 1.44% libtcmalloc.so.4.1.2 [.] 
> tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39% 
> libtcmalloc.so.4.1.2 [.] 
> tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
> 0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long)
> 
> 
> The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release.
> See the following link.
> 
> http://code.google.com/p/gperftools/
> 
> We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4.
> 
> It will be really helpful if you can push them to incorporate this in 14.04 updates.
> 
> Thanks & Regards
> Somnath
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: tcmalloc issue

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've logged:

https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/1439277

The upstream fix applies cleanly to the version of gperftools in 14.04
- - I just need a good reproducer for the SRU test case.

If anyone has one, please detail in the bug report above - I'll also
ping on the upstream bug (but that is from 2013 so might not get much
luck).

On 26/03/15 23:47, Sage Weil wrote:
> Hi James,
> 
> Is it possible to get tcmalloc in trusty updated?  See below.
> 
> Thanks! sage
> 
> On Thu, 26 Mar 2015, Somnath Roy wrote:
> 
>> Sage, Here is the tcmalloc issue I was talking in the performance
>> meeting.
>> 
>> http://code.google.com/p/gperftools/issues/detail?id=585
>> 
>> The tcmalloc is consuming lot of cpus (and eventually slowing
>> down the performance) with the following perf top  trace.
>> 
>> 25.73% libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::FetchFromSpans() 12.52%
>> libtcmalloc.so.4.1.2 [.]
>> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::F
reeList*,
>> unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::ReleaseToSpans(void*) 1.56% [kernel]
>> [k] __copy_user_nocache 1.44% libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39%
>> libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 0.89%
>> libtcmalloc.so.4.1.2 [.] operator new(unsigned long)
>> 
>> 
>> The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but
>> the env variable setting for this is a noop in the binary we are
>> having in Ubuntu 14.04. It is fixed in gperftools-2.1.90
>> release. See the following link.
>> 
>> http://code.google.com/p/gperftools/
>> 
>> We can request Ubuntu guys to incorporate this version or the
>> latest tcmalloc version which is gperftools-2.4.
>> 
>> It will be really helpful if you can push them to incorporate
>> this in 14.04 updates.
>> 
>> Thanks & Regards Somnath
>> 
>> ________________________________
>> 
>> PLEASE NOTE: The information contained in this electronic mail
>> message is intended only for the use of the designated
>> recipient(s) named above. If the reader of this message is not
>> the intended recipient, you are hereby notified that you have
>> received this message in error and that any review,
>> dissemination, distribution, or copying of this message is
>> strictly prohibited. If you have received this communication in
>> error, please notify the sender by telephone or e-mail (as shown
>> above) immediately and destroy any and all copies of this message
>> in your possession (whether hard copies or electronically stored
>> copies).
>> 
>> -- To unsubscribe from this list: send the line "unsubscribe
>> ceph-devel" in the body of a message to
>> majordomo@vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>> 
>> 
> 


- -- 
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVHBUUAAoJEL/srsug59jDfPYP/2zE62fCmvp9o+x7CxtDsALv
JvlEAdtjR7F4epx/IWlrJcRW7Z8EMmXYu7Enlsc4koZhSoIgw92SzBilaSwt+d7V
3M0xhFImlvednXJn8ppSBxIf5qrc8r57jQhnLv1rnw4+JrP+U6C994jmZCERKiN+
Ph9+j+A7yBmIqinjdx9kbes29/wZuUgRCyF5c7Wm2xm4W3aN8ZthNKa7mbXvzO9Y
ZNtXGcKZdKMWHbZYAmU64Sj82YQOWAdG/3FXOf+FmFOXDQG380UKCmd2aDRcqyvd
ySBFrJVoZG2s9wr9lJBhnIebAs48WoRPXhsDXtknncw0L/ZkErF8c8HKUm4WIztM
0fO/Zhue09vLTgOuBOmxcW6g0TW0CFhFPWXdTrb/gRfns6mOcuu57CdsvTePTEvH
iYpWQQaccaqgO5Yt2Y7lNR7JwoEYBEX4CahaxO2XmN9wl9N/RJxMd4AhyxKCuimL
La9YLZszHf+8qGMUjzY3NE6bgXlqUzjuNCE7ROu5rlyobrd4iRi8r4jeuZ37bR+m
0h5SY9UyAISegSwCmtwr457FqeC+V52vlrkYZy06+yxaOaeXd3pP9GZ2o0CLhqcP
cOva6Nt8OIemdKa6H1xm/V7uLqkJXl1xx3zEcONpzijmUS+2L6v4L3yhZQewif3d
Ms0rECbXfF8tmuDxn8MB
=P322
-----END PGP SIGNATURE-----

RE: tcmalloc issue

[Somnath Roy]

Thanks James !

the tcmalloc bug is reported in the following link.

http://code.google.com/p/gperftools/issues/detail?id=585

So, do you want us to give the steps on how we are hitting the following trace or to reproduce the bug reported in the above link ?

If you are asking about hitting the stack trace I mentioned below, it is just running ceph with lot of IOs for longer period of time.

Let us know.

Regards
Somnath

-----Original Message-----
From: James Page [mailto:james.page@ubuntu.com]
Sent: Wednesday, April 01, 2015 8:56 AM
To: Sage Weil; ceph-maintainers@ceph.com
Cc: Somnath Roy; ceph-devel@vger.kernel.org
Subject: Re: tcmalloc issue

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've logged:

https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/1439277

The upstream fix applies cleanly to the version of gperftools in 14.04
- - I just need a good reproducer for the SRU test case.

If anyone has one, please detail in the bug report above - I'll also ping on the upstream bug (but that is from 2013 so might not get much luck).

On 26/03/15 23:47, Sage Weil wrote:
> Hi James,
>
> Is it possible to get tcmalloc in trusty updated?  See below.
>
> Thanks! sage
>
> On Thu, 26 Mar 2015, Somnath Roy wrote:
>
>> Sage, Here is the tcmalloc issue I was talking in the performance
>> meeting.
>>
>> http://code.google.com/p/gperftools/issues/detail?id=585
>>
>> The tcmalloc is consuming lot of cpus (and eventually slowing down
>> the performance) with the following perf top  trace.
>>
>> 25.73% libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::FetchFromSpans() 12.52%
>> libtcmalloc.so.4.1.2 [.]
>> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::F
reeList*,
>> unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::ReleaseToSpans(void*) 1.56% [kernel] [k]
>> __copy_user_nocache 1.44% libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39%
>> libtcmalloc.so.4.1.2 [.]
>> tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 0.89%
>> libtcmalloc.so.4.1.2 [.] operator new(unsigned long)
>>
>>
>> The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the
>> env variable setting for this is a noop in the binary we are having
>> in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. See the
>> following link.
>>
>> http://code.google.com/p/gperftools/
>>
>> We can request Ubuntu guys to incorporate this version or the latest
>> tcmalloc version which is gperftools-2.4.
>>
>> It will be really helpful if you can push them to incorporate this in
>> 14.04 updates.
>>
>> Thanks & Regards Somnath
>>
>> ________________________________
>>
>> PLEASE NOTE: The information contained in this electronic mail
>> message is intended only for the use of the designated
>> recipient(s) named above. If the reader of this message is not the
>> intended recipient, you are hereby notified that you have received
>> this message in error and that any review, dissemination,
>> distribution, or copying of this message is strictly prohibited. If
>> you have received this communication in error, please notify the
>> sender by telephone or e-mail (as shown
>> above) immediately and destroy any and all copies of this message in
>> your possession (whether hard copies or electronically stored
>> copies).
>>
>> -- To unsubscribe from this list: send the line "unsubscribe
>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>


- --
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVHBUUAAoJEL/srsug59jDfPYP/2zE62fCmvp9o+x7CxtDsALv
JvlEAdtjR7F4epx/IWlrJcRW7Z8EMmXYu7Enlsc4koZhSoIgw92SzBilaSwt+d7V
3M0xhFImlvednXJn8ppSBxIf5qrc8r57jQhnLv1rnw4+JrP+U6C994jmZCERKiN+
Ph9+j+A7yBmIqinjdx9kbes29/wZuUgRCyF5c7Wm2xm4W3aN8ZthNKa7mbXvzO9Y
ZNtXGcKZdKMWHbZYAmU64Sj82YQOWAdG/3FXOf+FmFOXDQG380UKCmd2aDRcqyvd
ySBFrJVoZG2s9wr9lJBhnIebAs48WoRPXhsDXtknncw0L/ZkErF8c8HKUm4WIztM
0fO/Zhue09vLTgOuBOmxcW6g0TW0CFhFPWXdTrb/gRfns6mOcuu57CdsvTePTEvH
iYpWQQaccaqgO5Yt2Y7lNR7JwoEYBEX4CahaxO2XmN9wl9N/RJxMd4AhyxKCuimL
La9YLZszHf+8qGMUjzY3NE6bgXlqUzjuNCE7ROu5rlyobrd4iRi8r4jeuZ37bR+m
0h5SY9UyAISegSwCmtwr457FqeC+V52vlrkYZy06+yxaOaeXd3pP9GZ2o0CLhqcP
cOva6Nt8OIemdKa6H1xm/V7uLqkJXl1xx3zEcONpzijmUS+2L6v4L3yhZQewif3d
Ms0rECbXfF8tmuDxn8MB
=P322
-----END PGP SIGNATURE-----

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: tcmalloc issue

[Alexandre DERUMIER]

Hi,

does this bug also apply on rhel7(.1)/centos7(.1) ?


In Intank repository, I see

in rhel7 repository:

http://ceph.com/rpm-giant/rhel7/x86_64/gperftools-libs-2.1-1.el7.x86_64.rpm  


But I can't find it in centos repository

http://ceph.com/rpm-giant/el7/x86_64/

?



Does Intank will backport the patch to gperftools-libs-2.1-1.el7.x86_64.rpm  ?




----- Mail original -----
De: "Somnath Roy" <Somnath.Roy@sandisk.com>
À: "James Page" <james.page@ubuntu.com>, "Sage Weil" <sage@newdream.net>, ceph-maintainers@ceph.com
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Mercredi 1 Avril 2015 19:54:32
Objet: RE: tcmalloc issue

Thanks James ! 

the tcmalloc bug is reported in the following link. 

http://code.google.com/p/gperftools/issues/detail?id=585 

So, do you want us to give the steps on how we are hitting the following trace or to reproduce the bug reported in the above link ? 

If you are asking about hitting the stack trace I mentioned below, it is just running ceph with lot of IOs for longer period of time. 

Let us know. 

Regards 
Somnath 

-----Original Message----- 
From: James Page [mailto:james.page@ubuntu.com] 
Sent: Wednesday, April 01, 2015 8:56 AM 
To: Sage Weil; ceph-maintainers@ceph.com 
Cc: Somnath Roy; ceph-devel@vger.kernel.org 
Subject: Re: tcmalloc issue 

-----BEGIN PGP SIGNED MESSAGE----- 
Hash: SHA256 

I've logged: 

https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/1439277 

The upstream fix applies cleanly to the version of gperftools in 14.04 
- - I just need a good reproducer for the SRU test case. 

If anyone has one, please detail in the bug report above - I'll also ping on the upstream bug (but that is from 2013 so might not get much luck). 

On 26/03/15 23:47, Sage Weil wrote: 
> Hi James, 
> 
> Is it possible to get tcmalloc in trusty updated? See below. 
> 
> Thanks! sage 
> 
> On Thu, 26 Mar 2015, Somnath Roy wrote: 
> 
>> Sage, Here is the tcmalloc issue I was talking in the performance 
>> meeting. 
>> 
>> http://code.google.com/p/gperftools/issues/detail?id=585 
>> 
>> The tcmalloc is consuming lot of cpus (and eventually slowing down 
>> the performance) with the following perf top trace. 
>> 
>> 25.73% libtcmalloc.so.4.1.2 [.] 
>> tcmalloc::CentralFreeList::FetchFromSpans() 12.52% 
>> libtcmalloc.so.4.1.2 [.] 
>> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::F 
reeList*, 
>> unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.] 
>> tcmalloc::CentralFreeList::ReleaseToSpans(void*) 1.56% [kernel] [k] 
>> __copy_user_nocache 1.44% libtcmalloc.so.4.1.2 [.] 
>> tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39% 
>> libtcmalloc.so.4.1.2 [.] 
>> tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 0.89% 
>> libtcmalloc.so.4.1.2 [.] operator new(unsigned long) 
>> 
>> 
>> The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the 
>> env variable setting for this is a noop in the binary we are having 
>> in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. See the 
>> following link. 
>> 
>> http://code.google.com/p/gperftools/ 
>> 
>> We can request Ubuntu guys to incorporate this version or the latest 
>> tcmalloc version which is gperftools-2.4. 
>> 
>> It will be really helpful if you can push them to incorporate this in 
>> 14.04 updates. 
>> 
>> Thanks & Regards Somnath 
>> 
>> ________________________________ 
>> 
>> PLEASE NOTE: The information contained in this electronic mail 
>> message is intended only for the use of the designated 
>> recipient(s) named above. If the reader of this message is not the 
>> intended recipient, you are hereby notified that you have received 
>> this message in error and that any review, dissemination, 
>> distribution, or copying of this message is strictly prohibited. If 
>> you have received this communication in error, please notify the 
>> sender by telephone or e-mail (as shown 
>> above) immediately and destroy any and all copies of this message in 
>> your possession (whether hard copies or electronically stored 
>> copies). 
>> 
>> -- To unsubscribe from this list: send the line "unsubscribe 
>> ceph-devel" in the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at http://vger.kernel.org/majordomo-info.html 
>> 
>> 
> 


- -- 
James Page 
Ubuntu and Debian Developer 
james.page@ubuntu.com 
jamespage@debian.org 
-----BEGIN PGP SIGNATURE----- 
Version: GnuPG v2 

iQIcBAEBCAAGBQJVHBUUAAoJEL/srsug59jDfPYP/2zE62fCmvp9o+x7CxtDsALv 
JvlEAdtjR7F4epx/IWlrJcRW7Z8EMmXYu7Enlsc4koZhSoIgw92SzBilaSwt+d7V 
3M0xhFImlvednXJn8ppSBxIf5qrc8r57jQhnLv1rnw4+JrP+U6C994jmZCERKiN+ 
Ph9+j+A7yBmIqinjdx9kbes29/wZuUgRCyF5c7Wm2xm4W3aN8ZthNKa7mbXvzO9Y 
ZNtXGcKZdKMWHbZYAmU64Sj82YQOWAdG/3FXOf+FmFOXDQG380UKCmd2aDRcqyvd 
ySBFrJVoZG2s9wr9lJBhnIebAs48WoRPXhsDXtknncw0L/ZkErF8c8HKUm4WIztM 
0fO/Zhue09vLTgOuBOmxcW6g0TW0CFhFPWXdTrb/gRfns6mOcuu57CdsvTePTEvH 
iYpWQQaccaqgO5Yt2Y7lNR7JwoEYBEX4CahaxO2XmN9wl9N/RJxMd4AhyxKCuimL 
La9YLZszHf+8qGMUjzY3NE6bgXlqUzjuNCE7ROu5rlyobrd4iRi8r4jeuZ37bR+m 
0h5SY9UyAISegSwCmtwr457FqeC+V52vlrkYZy06+yxaOaeXd3pP9GZ2o0CLhqcP 
cOva6Nt8OIemdKa6H1xm/V7uLqkJXl1xx3zEcONpzijmUS+2L6v4L3yhZQewif3d 
Ms0rECbXfF8tmuDxn8MB 
=P322 
-----END PGP SIGNATURE----- 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tcmalloc issue

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/04/15 18:54, Somnath Roy wrote:
> the tcmalloc bug is reported in the following link.
> 
> http://code.google.com/p/gperftools/issues/detail?id=585
> 
> So, do you want us to give the steps on how we are hitting the
> following trace or to reproduce the bug reported in the above link
> ?
> 
> If you are asking about hitting the stack trace I mentioned below,
> it is just running ceph with lot of IOs for longer period of time.

A minimal test case on how to reproduce the bug is sufficient; we can
of course complete the second activity as well as soon as we have the
update available.

Cheers

James


- -- 
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVJhUoAAoJEL/srsug59jD/WoQAJqZYJ82xcDDxuD5C9RnUXha
2UtYNXZTK9O1NdiLqsNw14SPKtOQ6IDyfJjvB2v+ddfvRxhlfSM6LmC4ERa0t0SW
a3tfwx9fQ0WEz2r35nkupHwo+Z4yfTAuHgGom3rH6tq4gbAInn7J+4unsxghV1kf
op/c9qNJZV9jqfqMZFFgwwptK7rJ4mKJDvGIRU4ddc9HEKnqukJHQAHh3zUlNXC9
6y11RAHFh0FcgEEHys3o/ui0JVzNjmP8HJXISzyWSqO6J2NxqRNYoKGTk81zBvrf
IkWDDujCcTQoAU2aW6ERRvG1Dkzc9oTLdcmb0gGCKKpqa1m1GjnqS1B39rjheXio
HK5L9HO11Fs+9uONBXWvI6P0GXi3UQoe1NaVQB85EL+NPFbTdKr9zw8nOyyqZ9Y5
X55uVRAscLV/jb1vYIU5JxSNWof4BtPSjQzq29cithPqfWyF2N+eyQbCrAaIu1oP
QGSnryCsoWDARyvjMIFxcsd2X6nFnOd9DMgumi6O1ksKAe1P38k1iUWFwXXutby6
6eTdZtOj9Wc4VNRPb8SYBkBSlU+dCgRVMlNUImcCKWR6w8+Y9BbQU7OwQGEzj9Xw
f9sdi+4JA04l/BXdAtlVwSj4uSm2Q6R/YK4szpRGs24EOV6dX+aCMK5ezBIliPo0
D/7ZD03rXkiBzHuBqiay
=066U
-----END PGP SIGNATURE-----

RE: tcmalloc issue

[Chaitanya Huilgol]

Hi James,

Please find source of the test program to verify this bug at http://pastebin.com/KAxQZARm
compile it with  -ltcmalloc
To test:
Export a non-default value of the cache size, for eg
# export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=67108864
Run the test prog, For the buggy version you will see

Tcmalloc BUG! TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES: 67108864 Internal Size: 33554432 different

With the bug fixed, you should see

"Tcmalloc Internal and Env cache size are same:67108864"

Hope this helps.

Regards,
Chaitanya

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of James Page
Sent: Thursday, April 09, 2015 11:29 AM
To: Somnath Roy; Sage Weil; ceph-maintainers@ceph.com
Cc: ceph-devel@vger.kernel.org
Subject: Re: tcmalloc issue

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/04/15 18:54, Somnath Roy wrote:
> the tcmalloc bug is reported in the following link.
>
> http://code.google.com/p/gperftools/issues/detail?id=585
>
> So, do you want us to give the steps on how we are hitting the
> following trace or to reproduce the bug reported in the above link ?
>
> If you are asking about hitting the stack trace I mentioned below, it
> is just running ceph with lot of IOs for longer period of time.

A minimal test case on how to reproduce the bug is sufficient; we can of course complete the second activity as well as soon as we have the update available.

Cheers

James


- --
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVJhUoAAoJEL/srsug59jD/WoQAJqZYJ82xcDDxuD5C9RnUXha
2UtYNXZTK9O1NdiLqsNw14SPKtOQ6IDyfJjvB2v+ddfvRxhlfSM6LmC4ERa0t0SW
a3tfwx9fQ0WEz2r35nkupHwo+Z4yfTAuHgGom3rH6tq4gbAInn7J+4unsxghV1kf
op/c9qNJZV9jqfqMZFFgwwptK7rJ4mKJDvGIRU4ddc9HEKnqukJHQAHh3zUlNXC9
6y11RAHFh0FcgEEHys3o/ui0JVzNjmP8HJXISzyWSqO6J2NxqRNYoKGTk81zBvrf
IkWDDujCcTQoAU2aW6ERRvG1Dkzc9oTLdcmb0gGCKKpqa1m1GjnqS1B39rjheXio
HK5L9HO11Fs+9uONBXWvI6P0GXi3UQoe1NaVQB85EL+NPFbTdKr9zw8nOyyqZ9Y5
X55uVRAscLV/jb1vYIU5JxSNWof4BtPSjQzq29cithPqfWyF2N+eyQbCrAaIu1oP
QGSnryCsoWDARyvjMIFxcsd2X6nFnOd9DMgumi6O1ksKAe1P38k1iUWFwXXutby6
6eTdZtOj9Wc4VNRPb8SYBkBSlU+dCgRVMlNUImcCKWR6w8+Y9BbQU7OwQGEzj9Xw
f9sdi+4JA04l/BXdAtlVwSj4uSm2Q6R/YK4szpRGs24EOV6dX+aCMK5ezBIliPo0
D/7ZD03rXkiBzHuBqiay
=066U
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: tcmalloc issue

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Thanks Chaitanya

I've tested and uploaded the patched version to trusty-proposed - the
SRU team should accept that into testing in the next day or so - they
will update the bug when that happens.

On 09/04/15 10:00, Chaitanya Huilgol wrote:
> Hi James,
> 
> Please find source of the test program to verify this bug at
> http://pastebin.com/KAxQZARm compile it with  -ltcmalloc To test: 
> Export a non-default value of the cache size, for eg # export
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=67108864 Run the test prog,
> For the buggy version you will see
> 
> Tcmalloc BUG! TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES: 67108864
> Internal Size: 33554432 different
> 
> With the bug fixed, you should see
> 
> "Tcmalloc Internal and Env cache size are same:67108864"
> 
> Hope this helps.
> 
> Regards, Chaitanya
> 
> -----Original Message----- From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of James Page 
> Sent: Thursday, April 09, 2015 11:29 AM To: Somnath Roy; Sage Weil;
> ceph-maintainers@ceph.com Cc: ceph-devel@vger.kernel.org Subject:
> Re: tcmalloc issue
> 
> On 01/04/15 18:54, Somnath Roy wrote:
>> the tcmalloc bug is reported in the following link.
> 
>> http://code.google.com/p/gperftools/issues/detail?id=585
> 
>> So, do you want us to give the steps on how we are hitting the 
>> following trace or to reproduce the bug reported in the above
>> link ?
> 
>> If you are asking about hitting the stack trace I mentioned
>> below, it is just running ceph with lot of IOs for longer period
>> of time.
> 
> A minimal test case on how to reproduce the bug is sufficient; we
> can of course complete the second activity as well as soon as we
> have the update available.
> 
> Cheers
> 
> James
> 
> 
> -- To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail
> message is intended only for the use of the designated recipient(s)
> named above. If the reader of this message is not the intended
> recipient, you are hereby notified that you have received this
> message in error and that any review, dissemination, distribution,
> or copying of this message is strictly prohibited. If you have
> received this communication in error, please notify the sender by
> telephone or e-mail (as shown above) immediately and destroy any
> and all copies of this message in your possession (whether hard
> copies or electronically stored copies).
> 

- -- 
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVJokAAAoJEL/srsug59jDnNsQAM1h8T+XbzbyxIk31MSGuh6h
TfFs4ioBmv6IJFn7E+Le3WES3/Yqmhy1Sf1AHr1QtQNARJ9qqmNuU+M7IA/84A6o
6VvxKBb3fHWGJieG/ymaVVREuJUHJLfEEaqyZOrh++nsyuzf/nAymKSzJMl5Z8o1
ugOSqS/GryXORW2nzuWIHeAh3feETyBe2rFBeYTYkR65fsFyjMK5DTHNLh0SFTR3
Elc+qdyamau+XwlRZOz9ClyELk0vP2iWsaNj/KJrXGm104aCaf2NhnsX4IW4yz1s
GkcPhaFtR/V9d/txS8syMmh9b01rQL6Zxje3n0C63E0/AqLqBHF7NX60xbeiqAvw
5xPTK/6e4dARPaIzzCgY4vEjX/CQ38kFQYikU6lSIww873LgtdPAzJp15axH40E4
1YdtMAoSva/hVwkkjPLK7a8SVg4+sfVOho4FH7m8TZmLxpMXSH+ea4SFN9s9uuEa
jnA3mo5T50EeXQuYvmO87A4ABKGrOwBOvr3rmF4q/GnHGdONI+fctQW7JSmjJsMH
WrOwAWyte+A5hRRySiPw7EXqrj583l/iTaP+pd0eyWTcpoKKxp3Ver2nIsWGEe7+
Mq1lFii8Z3Z4rqthjWEg4iVw3HwQS+ArjYISuy4TnQUcnbG/Avp0liMM4VaBndF1
gefiZ4SMb5yxPriWndzf
=H69u
-----END PGP SIGNATURE-----

Re: tcmalloc issue

[James Page]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Folks

The proposed fix is now in the 'proposed' pocket for 14.04 - see:

https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/1439277

on how to enable that pocket for testing if you'd like to confirm that
you are able to use the required feature now.

Cheers

James

- -- 
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVL6F1AAoJEL/srsug59jDRGoP/AuQ8jGGAKgsRGu/aSnAzG0N
Aepa9AEnE2CH310HiYXqaKsmthKLwFy6vEkL0AkL4c4oDNVh9sdTaqBWV8qZXkxo
iSzWMeoId54du5F/wgk5e+Itk78tdxmdBhiALlCyudm2erlIYzc1iMLwBY1AIO3u
Ii7KngmPZj1d8DtUtfWAMoY8HbKSeZLpPpsu0mNOE6yzY/AycPlF/FPpWYoV2VBF
fUEOWu3FCst8YwGnegMwi3PIrjdvvVOfU1QxRJgeP8/oy5QB+LqAIHQEwpLsiASv
CJDZXj82SbFhMJxSj2vFF8WdfoNFz80DeMtdp268zjxaHVHVEjEszPAaZr862EHE
5qIXfAYgTYXtfI9p12OZm1PX7ogH45pfk5iMOZyZIWijkAnFHdMZ3ePHUMa+cBeR
DBE3Zbd7BJ+jFQ53rtkTI1L9TmUikad5BRVJXB9bVIENDOxj69vS7dn960FsWme8
CulQ/Scil/Da8vauo/itKx2ey3tVuxX96hUAHubCjBgQyPPmW/KTqICYLgMMUy2n
ZPMwVfKeM29KuLtyCu/VGdKtjaqjUl5QvTL0NJQaS18wFbsJOBGjwS2Mqfy2mVyU
3SDLAshD43m6Se2St8gEMIL3n5WYdGTksqb/iBVRKUfl3FhuBLTpb4ixtz3D2Tn1
HS2oqV/Bl/XFE0iyvPoM
=WHzY
-----END PGP SIGNATURE-----

RE: tcmalloc issue

[Somnath Roy]

Thanks James !
We will try this out.

Regards
Somnath

-----Original Message-----
From: James Page [mailto:james.page@ubuntu.com]
Sent: Thursday, April 16, 2015 4:48 AM
To: Chaitanya Huilgol; Somnath Roy; Sage Weil; ceph-maintainers@ceph.com
Cc: ceph-devel@vger.kernel.org
Subject: Re: tcmalloc issue

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Folks

The proposed fix is now in the 'proposed' pocket for 14.04 - see:

https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/1439277

on how to enable that pocket for testing if you'd like to confirm that you are able to use the required feature now.

Cheers

James

- --
James Page
Ubuntu and Debian Developer
james.page@ubuntu.com
jamespage@debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVL6F1AAoJEL/srsug59jDRGoP/AuQ8jGGAKgsRGu/aSnAzG0N
Aepa9AEnE2CH310HiYXqaKsmthKLwFy6vEkL0AkL4c4oDNVh9sdTaqBWV8qZXkxo
iSzWMeoId54du5F/wgk5e+Itk78tdxmdBhiALlCyudm2erlIYzc1iMLwBY1AIO3u
Ii7KngmPZj1d8DtUtfWAMoY8HbKSeZLpPpsu0mNOE6yzY/AycPlF/FPpWYoV2VBF
fUEOWu3FCst8YwGnegMwi3PIrjdvvVOfU1QxRJgeP8/oy5QB+LqAIHQEwpLsiASv
CJDZXj82SbFhMJxSj2vFF8WdfoNFz80DeMtdp268zjxaHVHVEjEszPAaZr862EHE
5qIXfAYgTYXtfI9p12OZm1PX7ogH45pfk5iMOZyZIWijkAnFHdMZ3ePHUMa+cBeR
DBE3Zbd7BJ+jFQ53rtkTI1L9TmUikad5BRVJXB9bVIENDOxj69vS7dn960FsWme8
CulQ/Scil/Da8vauo/itKx2ey3tVuxX96hUAHubCjBgQyPPmW/KTqICYLgMMUy2n
ZPMwVfKeM29KuLtyCu/VGdKtjaqjUl5QvTL0NJQaS18wFbsJOBGjwS2Mqfy2mVyU
3SDLAshD43m6Se2St8gEMIL3n5WYdGTksqb/iBVRKUfl3FhuBLTpb4ixtz3D2Tn1
HS2oqV/Bl/XFE0iyvPoM
=WHzY
-----END PGP SIGNATURE-----

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: tcmalloc issue

[Mark Nelson]

FWIW, I've seen perf top traces like this in the past.  It's making me 
wonder how many times I've seen this and not connected the dots.

Mark

On 03/26/2015 06:18 PM, Somnath Roy wrote:
> Sage,
> Here is the tcmalloc issue I was talking in the performance meeting.
>
> http://code.google.com/p/gperftools/issues/detail?id=585
>
> The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top  trace.
>
> 25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans()
> 12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)
> 11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
> 1.56% [kernel] [k] __copy_user_nocache
> 1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int)
> 1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
> 0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long)
>
>
> The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release.
> See the following link.
>
> http://code.google.com/p/gperftools/
>
> We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4.
>
> It will be really helpful if you can push them to incorporate this in 14.04 updates.
>
> Thanks & Regards
> Somnath
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: tcmalloc issue

[Alexandre DERUMIER]

Hi Sommath,

is it only on osds ?

or also clients ?

What is the tmalloc version on ubuntu ? (I would like to known if the problem exist also on debian)

Regards,

Alexandre
----- Mail original -----
De: "Somnath Roy" <Somnath.Roy@sandisk.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Vendredi 27 Mars 2015 00:18:57
Objet: tcmalloc issue

Sage, 
Here is the tcmalloc issue I was talking in the performance meeting. 

http://code.google.com/p/gperftools/issues/detail?id=585 

The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top trace. 

25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans() 
12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 
11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*) 
1.56% [kernel] [k] __copy_user_nocache 
1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 
1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 
0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long) 


The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. 
See the following link. 

http://code.google.com/p/gperftools/ 

We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4. 

It will be really helpful if you can push them to incorporate this in 14.04 updates. 

Thanks & Regards 
Somnath 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: tcmalloc issue

[Chaitanya Huilgol]

Hi,

Ubuntu Package version info: 
Package: libtcmalloc-minimal4
Versions: 
2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages)
 
On the OSD, increasing the thread caches from the default 32M has definitely helped, 
we have not tested this on client nodes though - probably will help there too. 

As a temporary non-intrusive workaround with the existing libtcmalloc-minimal4 library,
we are explicitly setting the thread cached values via rados-classes. (You can actually put this code into the OSD init path too) See patch below. 

Regards,
Chaitanya


From cafdbd41bb3492209716516910fa3a0e856ac7b8 Mon Sep 17 00:00:00 2001
From: Chaitanya Huilgol <chaitanya.huilgol@sandisk.com>
Date: Fri, 27 Mar 2015 15:16:35 +0530
Subject: [PATCH] Set tcmalloc thread cache via rados-class

Workaround for bug in tcmalloc (2.1-2ubuntu1)which does not use the exported
thread cache size value via environment variable.  Using library init path
to explicitly set this value in tcmalloc. No actual object handlers are
registered and hence does not affect the data path.

Signed-off-by: Chaitanya Huilgol <chaitanya.huilgol@sandisk.com>
---
 src/cls/Makefile.am                  |  5 ++++
 src/cls/tcmalloc_env/tcmalloc_env.cc | 52 ++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)
 create mode 100644 src/cls/tcmalloc_env/tcmalloc_env.cc

diff --git a/src/cls/Makefile.am b/src/cls/Makefile.am
index ea44fe7..abcf007 100644
--- a/src/cls/Makefile.am
+++ b/src/cls/Makefile.am
@@ -5,6 +5,11 @@ libcls_hello_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)
 libcls_hello_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
 radoslib_LTLIBRARIES += libcls_hello.la
 
+libcls_tcmalloc_env_la_SOURCES = cls/tcmalloc_env/tcmalloc_env.cc
+libcls_tcmalloc_env_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)
+libcls_tcmalloc_env_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
+radoslib_LTLIBRARIES += libcls_tcmalloc_env.la
+
 libcls_rbd_la_SOURCES = cls/rbd/cls_rbd.cc
 libcls_rbd_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)
 libcls_rbd_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
diff --git a/src/cls/tcmalloc_env/tcmalloc_env.cc b/src/cls/tcmalloc_env/tcmalloc_env.cc
new file mode 100644
index 0000000..0907183
--- /dev/null
+++ b/src/cls/tcmalloc_env/tcmalloc_env.cc
@@ -0,0 +1,52 @@
+/*
+ * rados-classes based plugin to set TCmalloc environment variable
+ * 'TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES' as the existing tcmalloc
+ * library in ubuntu 14.04 LTS and centos 7.0 has a bug which does
+ * not honor this env
+ */ 
+#include <cstdlib>
+#ifdef HAVE_GPERFTOOLS_HEAP_PROFILER_H
+#include <gperftools/heap-profiler.h>
+#else
+#include <google/heap-profiler.h>
+#endif
+
+#ifdef HAVE_GPERFTOOLS_MALLOC_EXTENSION_H
+#include <gperftools/malloc_extension.h>
+#else
+#include <google/malloc_extension.h>
+#endif
+#include "objclass/objclass.h"
+
+CLS_VER (1, 0) CLS_NAME (tcmalloc_env)
+#define DEFAULT_CACHE_SIZE (32 * 1024 * 1024)
+
+void
+__cls_init ()
+{
+  size_t result;
+  size_t cache_sz;
+  char *env_cache_sz_str;
+
+  CLS_LOG (0, "TCMALLOC-ENV: Search");
+  env_cache_sz_str = getenv ("TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES");
+  if (env_cache_sz_str) {
+    cache_sz = strtoul (env_cache_sz_str, NULL, 0);
+    CLS_LOG(0, "TCMALLOC-ENV: Found: %lu\n", cache_sz);
+    if (cache_sz > DEFAULT_CACHE_SIZE) {
+      MallocExtension::instance ()->
+        SetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", cache_sz);
+      result = 0;
+      MallocExtension::instance ()->
+        GetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", &result);
+      CLS_LOG (0, "TCMALLOC-ENV:Verify: max_total_thread_cache_bytes=%lu\n",
+        (unsigned long) result);
+    } else {
+      CLS_LOG (0, "TCMALLOC-ENV: Exported CacheSz=%lu <= Default=%lu - Ignored\n",
+        (unsigned long) cache_sz, (unsigned long) DEFAULT_CACHE_SIZE);
+    }
+  } else {
+      CLS_LOG(0, "TCMALLOC-ENV: Not Found\n");
+  }
+}
+/* EOF */
-- 
1.9.1

----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Alexandre DERUMIER
Sent: Friday, March 27, 2015 12:52 PM
To: Somnath Roy
Cc: ceph-devel
Subject: Re: tcmalloc issue

Hi Sommath,

is it only on osds ?

or also clients ?

What is the tmalloc version on ubuntu ? (I would like to known if the problem exist also on debian)

Regards,

Alexandre
----- Mail original -----
De: "Somnath Roy" <Somnath.Roy@sandisk.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Vendredi 27 Mars 2015 00:18:57
Objet: tcmalloc issue

Sage,
Here is the tcmalloc issue I was talking in the performance meeting. 

http://code.google.com/p/gperftools/issues/detail?id=585 

The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top trace. 

25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans()
12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
1.56% [kernel] [k] __copy_user_nocache
1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long) 


The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. 
See the following link. 

http://code.google.com/p/gperftools/ 

We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4. 

It will be really helpful if you can push them to incorporate this in 14.04 updates. 

Thanks & Regards
Somnath 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tcmalloc issue

[Alexandre DERUMIER]

>>Ubuntu Package version info: 
>>Package: libtcmalloc-minimal4 
>>Versions: 
>>2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages) 

Thanks !

looking at debian,
debian wheezy : libtcmalloc-minimal4 (2.0-2)

Seem to have the bug (I have looked inside debian src package)

debian jessie : libtcmalloc-minimal4 (2.2.1-0.2)
seem to be ok



>>we have not tested this on client nodes though - probably will help there too. 

I ask about client side, because from my debian client previous bench, 
I have 4x more cpu on client librbd vs krbd, so maybe it's coming from libtcmalloc



----- Mail original -----
De: "Chaitanya Huilgol" <Chaitanya.Huilgol@sandisk.com>
À: "aderumier" <aderumier@odiso.com>, "Somnath Roy" <Somnath.Roy@sandisk.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Vendredi 27 Mars 2015 10:53:30
Objet: RE: tcmalloc issue

Hi, 

Ubuntu Package version info: 
Package: libtcmalloc-minimal4 
Versions: 
2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages) 

On the OSD, increasing the thread caches from the default 32M has definitely helped, 
we have not tested this on client nodes though - probably will help there too. 

As a temporary non-intrusive workaround with the existing libtcmalloc-minimal4 library, 
we are explicitly setting the thread cached values via rados-classes. (You can actually put this code into the OSD init path too) See patch below. 

Regards, 
Chaitanya 


From cafdbd41bb3492209716516910fa3a0e856ac7b8 Mon Sep 17 00:00:00 2001 
From: Chaitanya Huilgol <chaitanya.huilgol@sandisk.com> 
Date: Fri, 27 Mar 2015 15:16:35 +0530 
Subject: [PATCH] Set tcmalloc thread cache via rados-class 

Workaround for bug in tcmalloc (2.1-2ubuntu1)which does not use the exported 
thread cache size value via environment variable. Using library init path 
to explicitly set this value in tcmalloc. No actual object handlers are 
registered and hence does not affect the data path. 

Signed-off-by: Chaitanya Huilgol <chaitanya.huilgol@sandisk.com> 
--- 
src/cls/Makefile.am | 5 ++++ 
src/cls/tcmalloc_env/tcmalloc_env.cc | 52 ++++++++++++++++++++++++++++++++++++ 
2 files changed, 57 insertions(+) 
create mode 100644 src/cls/tcmalloc_env/tcmalloc_env.cc 

diff --git a/src/cls/Makefile.am b/src/cls/Makefile.am 
index ea44fe7..abcf007 100644 
--- a/src/cls/Makefile.am 
+++ b/src/cls/Makefile.am 
@@ -5,6 +5,11 @@ libcls_hello_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
libcls_hello_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' 
radoslib_LTLIBRARIES += libcls_hello.la 

+libcls_tcmalloc_env_la_SOURCES = cls/tcmalloc_env/tcmalloc_env.cc 
+libcls_tcmalloc_env_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
+libcls_tcmalloc_env_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' 
+radoslib_LTLIBRARIES += libcls_tcmalloc_env.la 
+ 
libcls_rbd_la_SOURCES = cls/rbd/cls_rbd.cc 
libcls_rbd_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
libcls_rbd_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' 
diff --git a/src/cls/tcmalloc_env/tcmalloc_env.cc b/src/cls/tcmalloc_env/tcmalloc_env.cc 
new file mode 100644 
index 0000000..0907183 
--- /dev/null 
+++ b/src/cls/tcmalloc_env/tcmalloc_env.cc 
@@ -0,0 +1,52 @@ 
+/* 
+ * rados-classes based plugin to set TCmalloc environment variable 
+ * 'TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES' as the existing tcmalloc 
+ * library in ubuntu 14.04 LTS and centos 7.0 has a bug which does 
+ * not honor this env 
+ */ 
+#include <cstdlib> 
+#ifdef HAVE_GPERFTOOLS_HEAP_PROFILER_H 
+#include <gperftools/heap-profiler.h> 
+#else 
+#include <google/heap-profiler.h> 
+#endif 
+ 
+#ifdef HAVE_GPERFTOOLS_MALLOC_EXTENSION_H 
+#include <gperftools/malloc_extension.h> 
+#else 
+#include <google/malloc_extension.h> 
+#endif 
+#include "objclass/objclass.h" 
+ 
+CLS_VER (1, 0) CLS_NAME (tcmalloc_env) 
+#define DEFAULT_CACHE_SIZE (32 * 1024 * 1024) 
+ 
+void 
+__cls_init () 
+{ 
+ size_t result; 
+ size_t cache_sz; 
+ char *env_cache_sz_str; 
+ 
+ CLS_LOG (0, "TCMALLOC-ENV: Search"); 
+ env_cache_sz_str = getenv ("TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES"); 
+ if (env_cache_sz_str) { 
+ cache_sz = strtoul (env_cache_sz_str, NULL, 0); 
+ CLS_LOG(0, "TCMALLOC-ENV: Found: %lu\n", cache_sz); 
+ if (cache_sz > DEFAULT_CACHE_SIZE) { 
+ MallocExtension::instance ()-> 
+ SetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", cache_sz); 
+ result = 0; 
+ MallocExtension::instance ()-> 
+ GetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", &result); 
+ CLS_LOG (0, "TCMALLOC-ENV:Verify: max_total_thread_cache_bytes=%lu\n", 
+ (unsigned long) result); 
+ } else { 
+ CLS_LOG (0, "TCMALLOC-ENV: Exported CacheSz=%lu <= Default=%lu - Ignored\n", 
+ (unsigned long) cache_sz, (unsigned long) DEFAULT_CACHE_SIZE); 
+ } 
+ } else { 
+ CLS_LOG(0, "TCMALLOC-ENV: Not Found\n"); 
+ } 
+} 
+/* EOF */ 
-- 
1.9.1 

----Original Message----- 
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Alexandre DERUMIER 
Sent: Friday, March 27, 2015 12:52 PM 
To: Somnath Roy 
Cc: ceph-devel 
Subject: Re: tcmalloc issue 

Hi Sommath, 

is it only on osds ? 

or also clients ? 

What is the tmalloc version on ubuntu ? (I would like to known if the problem exist also on debian) 

Regards, 

Alexandre 
----- Mail original ----- 
De: "Somnath Roy" <Somnath.Roy@sandisk.com> 
À: "ceph-devel" <ceph-devel@vger.kernel.org> 
Envoyé: Vendredi 27 Mars 2015 00:18:57 
Objet: tcmalloc issue 

Sage, 
Here is the tcmalloc issue I was talking in the performance meeting. 

http://code.google.com/p/gperftools/issues/detail?id=585 

The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top trace. 

25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans() 
12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*) 
1.56% [kernel] [k] __copy_user_nocache 
1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 
0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long) 


The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. 
See the following link. 

http://code.google.com/p/gperftools/ 

We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4. 

It will be really helpful if you can push them to incorporate this in 14.04 updates. 

Thanks & Regards 
Somnath 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html 

-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: tcmalloc issue

[Chaitanya Huilgol]

Hi,

Ubuntu Package version info: 
Package: libtcmalloc-minimal4
Versions: 
2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages)
 
On the OSD, increasing the thread caches from the default 32M has definitely helped, we have not tested this on client nodes though - probably will help there too. 

As a temporary non-intrusive workaround with the existing libtcmalloc-minimal4 library, we are explicitly setting the thread cached values via rados-classes. (You can actually put this code into the OSD init path too) See patch below. 

Regards,
Chaitanya


From cafdbd41bb3492209716516910fa3a0e856ac7b8 Mon Sep 17 00:00:00 2001
From: Chaitanya Huilgol <chaitanya.huilgol@sandisk.com>
Date: Fri, 27 Mar 2015 15:16:35 +0530
Subject: [PATCH] Set tcmalloc thread cache via rados-class

Workaround for bug in tcmalloc (2.1-2ubuntu1)which does not use the exported thread cache size value via environment variable.  Using library init path to explicitly set this value in tcmalloc. No actual object handlers are registered and hence does not affect the data path.

Signed-off-by: Chaitanya Huilgol <chaitanya.huilgol@sandisk.com>
---
 src/cls/Makefile.am                  |  5 ++++
 src/cls/tcmalloc_env/tcmalloc_env.cc | 52 ++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)
 create mode 100644 src/cls/tcmalloc_env/tcmalloc_env.cc

diff --git a/src/cls/Makefile.am b/src/cls/Makefile.am index ea44fe7..abcf007 100644
--- a/src/cls/Makefile.am
+++ b/src/cls/Makefile.am
@@ -5,6 +5,11 @@ libcls_hello_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)  libcls_hello_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
 radoslib_LTLIBRARIES += libcls_hello.la
 
+libcls_tcmalloc_env_la_SOURCES = cls/tcmalloc_env/tcmalloc_env.cc 
+libcls_tcmalloc_env_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
+libcls_tcmalloc_env_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
+radoslib_LTLIBRARIES += libcls_tcmalloc_env.la
+
 libcls_rbd_la_SOURCES = cls/rbd/cls_rbd.cc  libcls_rbd_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)  libcls_rbd_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
diff --git a/src/cls/tcmalloc_env/tcmalloc_env.cc b/src/cls/tcmalloc_env/tcmalloc_env.cc
new file mode 100644
index 0000000..0907183
--- /dev/null
+++ b/src/cls/tcmalloc_env/tcmalloc_env.cc
@@ -0,0 +1,52 @@
+/*
+ * rados-classes based plugin to set TCmalloc environment variable
+ * 'TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES' as the existing tcmalloc
+ * library in ubuntu 14.04 LTS and centos 7.0 has a bug which does
+ * not honor this env
+ */
+#include <cstdlib>
+#ifdef HAVE_GPERFTOOLS_HEAP_PROFILER_H
+#include <gperftools/heap-profiler.h>
+#else
+#include <google/heap-profiler.h>
+#endif
+
+#ifdef HAVE_GPERFTOOLS_MALLOC_EXTENSION_H
+#include <gperftools/malloc_extension.h> #else #include 
+<google/malloc_extension.h> #endif #include "objclass/objclass.h"
+
+CLS_VER (1, 0) CLS_NAME (tcmalloc_env)
+#define DEFAULT_CACHE_SIZE (32 * 1024 * 1024)
+
+void
+__cls_init ()
+{
+  size_t result;
+  size_t cache_sz;
+  char *env_cache_sz_str;
+
+  CLS_LOG (0, "TCMALLOC-ENV: Search");
+  env_cache_sz_str = getenv ("TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES");
+  if (env_cache_sz_str) {
+    cache_sz = strtoul (env_cache_sz_str, NULL, 0);
+    CLS_LOG(0, "TCMALLOC-ENV: Found: %lu\n", cache_sz);
+    if (cache_sz > DEFAULT_CACHE_SIZE) {
+      MallocExtension::instance ()->
+        SetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", cache_sz);
+      result = 0;
+      MallocExtension::instance ()->
+        GetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", &result);
+      CLS_LOG (0, "TCMALLOC-ENV:Verify: max_total_thread_cache_bytes=%lu\n",
+        (unsigned long) result);
+    } else {
+      CLS_LOG (0, "TCMALLOC-ENV: Exported CacheSz=%lu <= Default=%lu - Ignored\n",
+        (unsigned long) cache_sz, (unsigned long) DEFAULT_CACHE_SIZE);
+    }
+  } else {
+      CLS_LOG(0, "TCMALLOC-ENV: Not Found\n");
+  }
+}
+/* EOF */
--
1.9.1

----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Alexandre DERUMIER
Sent: Friday, March 27, 2015 12:52 PM
To: Somnath Roy
Cc: ceph-devel
Subject: Re: tcmalloc issue

Hi Sommath,

is it only on osds ?

or also clients ?

What is the tmalloc version on ubuntu ? (I would like to known if the problem exist also on debian)

Regards,

Alexandre
----- Mail original -----
De: "Somnath Roy" <Somnath.Roy@sandisk.com>
À: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Vendredi 27 Mars 2015 00:18:57
Objet: tcmalloc issue

Sage,
Here is the tcmalloc issue I was talking in the performance meeting. 

http://code.google.com/p/gperftools/issues/detail?id=585 

The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top trace. 

25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans()
12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
1.56% [kernel] [k] __copy_user_nocache
1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long) 


The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. 
See the following link. 

http://code.google.com/p/gperftools/ 

We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4. 

It will be really helpful if you can push them to incorporate this in 14.04 updates. 

Thanks & Regards
Somnath 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html