Re: Hitting tcmalloc bug even with patch applied

[Mark Nelson <mnelson@redhat.com>]

On 04/27/2015 03:24 PM, Milosz Tanski wrote:
>
>
> On 4/27/15 8:06 AM, Alexandre DERUMIER wrote:
>> Hi,
>>
>> I'm hitting the tcmalloc even with patch apply.
>> It's mainly occur when I try to bench fio with a lot jobs (20 - 40 jobs)
>>
>> Does It need to tuned something in osd environnement variable ?
>>
>>
>> I double check it with
>>
>> #g++ -o gperftest gperftest.c -ltcmalloc
>> # export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=67108864
>> # ./gperftest
>> Tcmalloc OK! Internal and Env cache size are same:67108864
>>
>>
>> perf top
>> -------
>>    10.04%  libtcmalloc.so.4.1.2  [.] tcmalloc::ThreadCache::ReleaseToCentralCache
>>     8.19%  libtcmalloc.so.4.1.2  [.] tcmalloc::CentralFreeList::FetchFromSpans
>>     3.89%  libtcmalloc.so.4.1.2  [.] tcmalloc::CentralFreeList::ReleaseToSpans
>>     2.04%  libtcmalloc.so.4.1.2  [.] tcmalloc::CentralFreeList::ReleaseListToSpans
>>     1.79%  libtcmalloc.so.4.1.2  [.] operator new
>>     1.25%  ceph-osd              [.] ConfFile::load_from_buffer
>>     1.21%  libtcmalloc.so.4.1.2  [.] operator delete
>>     1.14%  [kernel]              [k] _raw_spin_lock
>>     1.08%  libstdc++.so.6.0.19   [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string
>>     1.04%  [kernel]              [k] __schedule
>>     1.00%  libpthread-2.17.so    [.] pthread_mutex_trylock
>>     0.90%  [kernel]              [k] native_write_msr_safe
>>     0.89%  [kernel]              [k] __switch_to
>>     0.79%  [kernel]              [k] _raw_spin_lock_irqsave
>>     0.73%  [kernel]              [k] copy_user_enhanced_fast_string
>>
>
> This is obviously going to be more painful but .... can you perform a capture for one OSD process using, pref record -p $OSD_PID. Ideally one with a callgraph and one without.
>
> That can be helpful to investigate further. Can see which parts of those tcmalloc functions are the biggest offer in terms of time. We can also see if there's a new/delete pastern in OSD code that is somehow trigger this degenerate case.

If on a newish (3.11+) kernel that has libunwind compiled into perf, 
I've found that dwarf callgraphs are much more detailed.  The frequency 
may need to be lowered to make it work well.  -F 100 or something perhaps.

>
>>
>>
>> Regards,
>>
>> Alexandre
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>