Re: extreme ceph-osd cpu load for rand. 4k write

[Josh Durgin <josh.durgin@inktank.com>]

On 11/08/2012 01:27 PM, Stefan Priebe wrote:
> Am 08.11.2012 17:06, schrieb Mark Nelson:
>> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote:
>>> Am 08.11.2012 16:01, schrieb Sage Weil:
>>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote:
>>>>> Is there any way to find out why a ceph-osd process takes around 10
>>>>> times more
>>>>> load on rand 4k writes than on 4k reads?
>>>>
>>>> Something like perf or oprofile is probably your best bet.  perf can be
>>>> tedious to deploy, depending on where your kernel is coming from.
>>>> oprofile seems to be deprecated, although I've had good results with
>>>> it in
>>>> the past.
>>>
>>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly
>>> i've no idea what todo with it next.
>>
>> Pour yourself a stiff drink! (haha!)
>>
>> Try just doing a "perf report" in the directory where you've got the
>> data file.  Here's a nice tutorial:
>>
>> https://perf.wiki.kernel.org/index.php/Tutorial
>>
>> Also, if you see missing symbols you might benefit by chowning the file
>> to root and running perf report as root.  If you still see missing
>> symbols, you may want to just give up and try sysprof.
>
> I've now used google perftools / google CPU profiler. It was the only
> tool who worked out of the box ;-)
>
> Attached is a PDF with a profiled ceph-osd process while 4k random write.

It looks like a not insignificant portion of time is spent in the
logging infrastructure. Could you add this to the osds' configuration
to prevent any debug log gathering (it's logged/gathered):

debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0

Josh