Re: client cpu usage : kbrd vs librbd perf report

[Mark Nelson <mark.nelson@inktank.com>]

On 11/13/2014 10:29 AM, Sage Weil wrote:
> On Thu, 13 Nov 2014, Alexandre DERUMIER wrote:
>>>> I think we need to figure out why so much time is being spent
>>>> mallocing/freeing memory. Got to get those symbols resolved!
>>
>> Ok, I don't known why, but if I remove all ceph -dbg packages, I'm seeing the rbd && rados symbols now...
>>
>> I have udpdate the files:
>>
>> http://odisoweb1.odiso.net/cephperf/perf-librbd/report.txt
>
> Ran it through c++filt:
>
> 	https://gist.github.com/88ba9409f5d201b957a1
>
> I'm a bit suprised by the some of the items near the top
> (bufferlist.clear() callers).  I'm sure several of those can be
> streamlined to avoid temporary bufferlists.  I don't see any super
> egregious users of the allocator, though.
>
> The memcpy callers might be a good place to start...
>
> sage

Wasn't josh looking into some of this a year ago?  Did anything ever 
come of that work?

>
>
>
>
>
>>
>>
>>
>>
>> ----- Mail original -----
>>
>> De: "Mark Nelson" <mark.nelson@inktank.com>
>> ?: "Alexandre DERUMIER" <aderumier@odiso.com>, "Ceph Devel" <ceph-devel@vger.kernel.org>
>> Cc: "Mark Nelson" <mark.nelson@inktank.com>, "Sage Weil" <sweil@redhat.com>, "Somnath Roy" <somnath.roy@sandisk.com>
>> Envoy?: Jeudi 13 Novembre 2014 15:20:40
>> Objet: Re: client cpu usage : kbrd vs librbd perf report
>>
>> On 11/13/2014 05:15 AM, Alexandre DERUMIER wrote:
>>> Hi,
>>>
>>> I have redone perf with dwarf
>>>
>>> perf record -g --call-graph dwarf -a -F 99 -- sleep 60
>>>
>>> I have put perf reports, ceph conf, fio config here:
>>>
>>> http://odisoweb1.odiso.net/cephperf/
>>>
>>> test setup
>>> -----------
>>> client cpu config : 8 x Intel(R) Xeon(R) CPU E5-2603 v2 @ 1.80GHz
>>> ceph cluster : 3 nodes (same cpu than client) with 2 osd each (intel ssd s3500), test pool with replication x1
>>> rbd volume size : 10G (almost all reads are done in osd buffer cache)
>>>
>>> benchmark with fio 4k randread, with 1 rbd volume. (also tested with 20 rbd volumes, results are equals).
>>> debian wheezy - kernel 3.17 - and ceph packages from master on gitbuilder
>>>
>>> (BTW, I have installed librbd/rados dbg packages but I have missing symbols ?)
>>
>> I think if you run perf report with verbose enabled it will tell you
>> which symbols are missing:
>>
>> perf report -v 2>&1 | less
>>
>> If you have them but it's not detecting them properly you can clean out
>> the cache or even manually reassign the symbols but it's annoying.
>>
>>>
>>>
>>>
>>> Global results:
>>> ---------------
>>> librbd : 60000iops : 98% cpu
>>> krbd : 90000iops : 32% cpu
>>>
>>>
>>> So, librbd usage is 4,5x more than krbd for same ios throughput
>>>
>>> The difference seem to be quite huge, is it expected ?
>>
>> This is kind of the wild west. With that many IOPS we are running into
>> new bottlenecks. :)
>>
>>>
>>>
>>>
>>>
>>> librbd perf report:
>>> -------------------------
>>> top cpu usage
>>> --------------
>>> 25.71% fio libc-2.13.so
>>> 17.69% fio librados.so.2.0.0
>>> 12.38% fio librbd.so.1.0.0
>>> 27.99% fio [kernel.kallsyms]
>>> 4.19% fio libpthread-2.13.so
>>>
>>>
>>> libc-2.13.so (seem that malloc/free use a lot of cpu here)
>>> ------------
>>> 21.05%-- _int_malloc
>>> 14.36%-- free
>>> 13.66%-- malloc
>>> 9.89%-- __lll_unlock_wake_private
>>> 5.35%-- __clone
>>> 4.38%-- __poll
>>> 3.77%-- __memcpy_ssse3
>>> 1.64%-- vfprintf
>>> 1.02%-- arena_get2
>>>
>>
>> I think we need to figure out why so much time is being spent
>> mallocing/freeing memory. Got to get those symbols resolved!
>>
>>> fio [kernel.kallsyms] : seem to have a lot of futex functions here
>>> -----------------------
>>> 5.27%-- _raw_spin_lock
>>> 3.88%-- futex_wake
>>> 2.88%-- __switch_to
>>> 2.74%-- system_call
>>> 2.70%-- __schedule
>>> 2.52%-- tcp_sendmsg
>>> 2.47%-- futex_wait_setup
>>> 2.28%-- _raw_spin_lock_irqsave
>>> 2.16%-- idle_cpu
>>> 1.66%-- enqueue_task_fair
>>> 1.57%-- native_write_msr_safe
>>> 1.49%-- hash_futex
>>> 1.46%-- futex_wait
>>> 1.40%-- reschedule_interrupt
>>> 1.37%-- try_to_wake_up
>>> 1.28%-- account_entity_enqueue
>>> 1.25%-- copy_user_enhanced_fast_string
>>> 1.25%-- futex_requeue
>>> 1.24%-- __fget
>>> 1.24%-- update_curr
>>> 1.20%-- tcp_write_xmit
>>> 1.14%-- wake_futex
>>> 1.08%-- scheduler_ipi
>>> 1.05%-- select_task_rq_fair
>>> 1.01%-- dequeue_task_fair
>>> 0.97%-- do_futex
>>> 0.97%-- futex_wait_queue_me
>>> 0.83%-- cpuacct_charge
>>> 0.82%-- tcp_transmit_skb
>>> ...
>>>
>>>
>>> Regards,
>>>
>>> Alexandre
>>>
>>>
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>