Re: extreme ceph-osd cpu load for rand. 4k write

[Stefan Priebe <s.priebe@profihost.ag>]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

Am 08.11.2012 17:06, schrieb Mark Nelson:
> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote:
>> Am 08.11.2012 16:01, schrieb Sage Weil:
>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote:
>>>> Is there any way to find out why a ceph-osd process takes around 10
>>>> times more
>>>> load on rand 4k writes than on 4k reads?
>>>
>>> Something like perf or oprofile is probably your best bet.  perf can be
>>> tedious to deploy, depending on where your kernel is coming from.
>>> oprofile seems to be deprecated, although I've had good results with
>>> it in
>>> the past.
>>
>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly
>> i've no idea what todo with it next.
>
> Pour yourself a stiff drink! (haha!)
>
> Try just doing a "perf report" in the directory where you've got the
> data file.  Here's a nice tutorial:
>
> https://perf.wiki.kernel.org/index.php/Tutorial
>
> Also, if you see missing symbols you might benefit by chowning the file
> to root and running perf report as root.  If you still see missing
> symbols, you may want to just give up and try sysprof.

I've now used google perftools / google CPU profiler. It was the only 
tool who worked out of the box ;-)

Attached is a PDF with a profiled ceph-osd process while 4k random write.

Stefan

[-- Attachment #2: out.pdf --]
[-- Type: application/pdf, Size: 20834 bytes --]