Re: extreme ceph-osd cpu load for rand. 4k write

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Disabling the logging with:
  debug lockdep = 0/0
  debug context = 0/0
  debug crush = 0/0
  debug buffer = 0/0
  debug timer = 0/0
  debug journaler = 0/0
  debug osd = 0/0
  debug optracker = 0/0
  debug objclass = 0/0
  debug filestore = 0/0
  debug journal = 0/0
  debug ms = 0/0
  debug monc = 0/0
  debug tp = 0/0
  debug auth = 0/0
  debug finisher = 0/0
  debug heartbeatmap = 0/0
  debug perfcounter = 0/0
  debug asok = 0/0
  debug throttle = 0/0

reduced the CPU load about 50% ! So each OSD process now takes only one 
whole 3.6Ghz core instead of two.

Have you looked at my latest profile graph with disabled debug options?

Greets,
Stefan


Am 08.11.2012 17:06, schrieb Mark Nelson:
> On 11/08/2012 09:45 AM, Stefan Priebe - Profihost AG wrote:
>> Am 08.11.2012 16:01, schrieb Sage Weil:
>>> On Thu, 8 Nov 2012, Stefan Priebe - Profihost AG wrote:
>>>> Is there any way to find out why a ceph-osd process takes around 10
>>>> times more
>>>> load on rand 4k writes than on 4k reads?
>>>
>>> Something like perf or oprofile is probably your best bet.  perf can be
>>> tedious to deploy, depending on where your kernel is coming from.
>>> oprofile seems to be deprecated, although I've had good results with
>>> it in
>>> the past.
>>
>> I've recorded 10s with perf - it is now a 300MB perf.data file. Sadly
>> i've no idea what todo with it next.
>
> Pour yourself a stiff drink! (haha!)
>
> Try just doing a "perf report" in the directory where you've got the
> data file.  Here's a nice tutorial:
>
> https://perf.wiki.kernel.org/index.php/Tutorial
>
> Also, if you see missing symbols you might benefit by chowning the file
> to root and running perf report as root.  If you still see missing
> symbols, you may want to just give up and try sysprof.
>
>>
>>>   would love to see where the CPU is spending most of it's time.
>>> This is
>>> on current master?
>> Yes
>>
>>>  I expect there are still some low-hanging fruit that
>>> can bring CPU utilization down (or even boost iops).
>> Would be great to find them.
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html