Re: extreme ceph-osd cpu load for rand. 4k write

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

[-- Attachment #1: Type: text/plain, Size: 2890 bytes --]

New graph from today. fsetxattr seems to take a lot of CPU too.

Am 09.11.2012 11:09, schrieb Stefan Priebe - Profihost AG:
>
> Disabling the logging with:
>   debug lockdep = 0/0
>   debug context = 0/0
>   debug crush = 0/0
>   debug buffer = 0/0
>   debug timer = 0/0
>   debug journaler = 0/0
>   debug osd = 0/0
>   debug optracker = 0/0
>   debug objclass = 0/0
>   debug filestore = 0/0
>   debug journal = 0/0
>   debug ms = 0/0
>   debug monc = 0/0
>   debug tp = 0/0
>   debug auth = 0/0
>   debug finisher = 0/0
>   debug heartbeatmap = 0/0
>   debug perfcounter = 0/0
>   debug asok = 0/0
>   debug throttle = 0/0
>
> reduced the CPU load about 50% ! So each OSD process now takes only one
> whole 3.6Ghz core instead of two.
>
> Have you looked at my latest profile graph with disabled debug options?
>
> Greets,
> Stefan
>
>
> Am 08.11.2012 17:06, schrieb Mark Nelson:
>> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote:
>>> Am 08.11.2012 16:01, schrieb Sage Weil:
>>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote:
>>>>> Is there any way to find out why a ceph-osd process takes around 10
>>>>> times more
>>>>> load on rand 4k writes than on 4k reads?
>>>>
>>>> Something like perf or oprofile is probably your best bet.  perf can be
>>>> tedious to deploy, depending on where your kernel is coming from.
>>>> oprofile seems to be deprecated, although I've had good results with
>>>> it in
>>>> the past.
>>>
>>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly
>>> i've no idea what todo with it next.
>>
>> Pour yourself a stiff drink! (haha!)
>>
>> Try just doing a "perf report" in the directory where you've got the
>> data file.  Here's a nice tutorial:
>>
>> https://perf.wiki.kernel.org/index.php/Tutorial
>>
>> Also, if you see missing symbols you might benefit by chowning the file
>> to root and running perf report as root.  If you still see missing
>> symbols, you may want to just give up and try sysprof.
>>
>>>
>>>>   would love to see where the CPU is spending most of it's time.
>>>> This is
>>>> on current master?
>>> Yes
>>>
>>>>  I expect there are still some low-hanging fruit that
>>>> can bring CPU utilization down (or even boost iops).
>>> Would be great to find them.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: out.pdf --]
[-- Type: application/pdf, Size: 20401 bytes --]