Re: [pve-devel] less cores more iops / speed

[Josh Durgin <josh.durgin@inktank.com>]

On 11/12/2012 07:33 AM, Stefan Priebe - Profihost AG wrote:
> Adding this to ceph.conf on kvm host adds another 2000 iops (20.000
> iop/s with one VM). I'm sure most of them are useless on a client kvm /
> rbd host but i don't know which one makes sense ;-)
>
> [global]
>          debug ms = 0/0
>          debug rbd = 0/0
>          debug lockdep = 0/0
>          debug context = 0/0
>          debug crush = 0/0
>          debug buffer = 0/0
>          debug timer = 0/0
>          debug journaler = 0/0
>          debug osd = 0/0
>          debug optracker = 0/0
>          debug objclass = 0/0
>          debug filestore = 0/0
>          debug journal = 0/0
>          debug ms = 0/0
>          debug monc = 0/0
>          debug tp = 0/0
>          debug auth = 0/0
>          debug finisher = 0/0
>          debug heartbeatmap = 0/0
>          debug perfcounter = 0/0
>          debug asok = 0/0
>          debug throttle = 0/0
>
> [client]

For the client side you'd these settings to disable all debug logging:

[client]
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug filer = 0/0
debug objecter = 0/0
debug rados = 0/0
debug rbd = 0/0
debug objectcacher = 0/0
debug client = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0

Josh

>          debug ms = 0/0
>          debug rbd = 0/0
>          debug lockdep = 0/0
>          debug context = 0/0
>          debug crush = 0/0
>          debug buffer = 0/0
>          debug timer = 0/0
>          debug journaler = 0/0
>          debug osd = 0/0
>          debug optracker = 0/0
>          debug objclass = 0/0
>          debug filestore = 0/0
>          debug journal = 0/0
>          debug ms = 0/0
>          debug monc = 0/0
>          debug tp = 0/0
>          debug auth = 0/0
>          debug finisher = 0/0
>          debug heartbeatmap = 0/0
>          debug perfcounter = 0/0
>          debug asok = 0/0
>          debug throttle = 0/0
>
> Stefan
>
> Am 12.11.2012 15:35, schrieb Alexandre DERUMIER:
>> Another idea,
>>
>> do you have tried to put
>>>>>>   debug lockdep = 0/0
>>>>>>   debug context = 0/0
>>>>>>   debug crush = 0/0
>>>>>>   debug buffer = 0/0
>>>>>>   debug timer = 0/0
>>>>>>   debug journaler = 0/0
>>>>>>   debug osd = 0/0
>>>>>>   debug optracker = 0/0
>>>>>>   debug objclass = 0/0
>>>>>>   debug filestore = 0/0
>>>>>>   debug journal = 0/0
>>>>>>   debug ms = 0/0
>>>>>>   debug monc = 0/0
>>>>>>   debug tp = 0/0
>>>>>>   debug auth = 0/0
>>>>>>   debug finisher = 0/0
>>>>>>   debug heartbeatmap = 0/0
>>>>>>   debug perfcounter = 0/0
>>>>>>   debug asok = 0/0
>>>>>>   debug throttle = 0/0
>>
>> in a ceph.conf on your kvm host ?
>>
>>
>> ----- Mail original -----
>>
>> De: "Alexandre DERUMIER" <aderumier@odiso.com>
>> À: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
>> Cc: pve-devel@pve.proxmox.com
>> Envoyé: Lundi 12 Novembre 2012 15:26:36
>> Objet: Re: [pve-devel] less cores more iops / speed
>>
>> Maybe some tracing on kvm process could give us clues to find where
>> the cpu is used ?
>>
>> Also another idea, can you try with "auth supported=none" ? maybe they
>> are some overhead with ceph authenfication ?
>>
>>
>>
>>
>> ----- Mail original -----
>>
>> De: "Alexandre DERUMIER" <aderumier@odiso.com>
>> À: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
>> Cc: pve-devel@pve.proxmox.com
>> Envoyé: Lundi 12 Novembre 2012 15:20:07
>> Objet: Re: [pve-devel] less cores more iops / speed
>>
>> Ok thanks.
>>
>> Seem to use a lot of cpu vs nfs,iscsi ...
>>
>> I hope that ceph dev will work on this soon !
>>
>>
>> ----- Mail original -----
>>
>> De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>> Cc: "eric" <eric@netwalk.com>, pve-devel@pve.proxmox.com
>> Envoyé: Lundi 12 Novembre 2012 15:05:08
>> Objet: Re: [pve-devel] less cores more iops / speed
>>
>> Am 12.11.2012 13:49, schrieb Alexandre DERUMIER:
>>>>> One VM on one Host: 18.000 IOP/s
>>>>> Two VM on one Host: 2x11.000 IOP/s
>>>>> Three VM on one Host: 3x7.000 IOP/s
>>>
>>> And host cpu is 100% ?
>>
>> No. For three VMs yes. For one and two no. I think librbd / rbd
>> implementation in kvm is the bottleneck here.
>>
>> Stefan
>>
>>> ----- Mail original -----
>>>
>>> De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
>>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>>> Cc: "eric" <eric@netwalk.com>, pve-devel@pve.proxmox.com
>>> Envoyé: Lundi 12 Novembre 2012 12:58:35
>>> Objet: Re: [pve-devel] less cores more iops / speed
>>>
>>> Am 12.11.2012 08:51, schrieb Alexandre DERUMIER:
>>>>>> Right now RBD in KVM is limited by CPU speed.
>>>>
>>>> Good to known, so it's seem lack of threading, or maybe somes locks.
>>>> (so faster cpu give more iops).
>>>>
>>>> If you lauch parallel fio on same host on different guest, do you
>>>> get more total iops ? (for me it's scale)
>>>
>>> One VM on one Host: 18.000 IOP/s
>>> Two VM on one Host: 2x11.000 IOP/s
>>> Three VM on one Host: 3x7.000 IOP/s
>>>
>>>> if you launch 2 parallel fio, on same guest (on differents disk), do
>>>> you get more iops ? (for me, it doesn't scale, so raid0 in guest
>>>> doesn't help).
>>> No it doesn't scale.
>>>
>>> Stefan
>>>
>>>> ----- Mail original -----
>>>>
>>>> De: "Stefan Priebe" <s.priebe@profihost.ag>
>>>> À: "Alexandre DERUMIER" <aderumier@odiso.com>
>>>> Cc: "eric" <eric@netwalk.com>, pve-devel@pve.proxmox.com
>>>> Envoyé: Dimanche 11 Novembre 2012 13:07:36
>>>> Objet: Re: [pve-devel] less cores more iops / speed
>>>>
>>>> Am 11.11.2012 12:12, schrieb Alexandre DERUMIER:
>>>>> If I remember good, stefan can achieve 100.000 iops with iscsi with
>>>>> same kvm host.
>>>>
>>>> Correct but this was always with scsi-generic and I/O multipathing on
>>>> host. rbd does not support scsi-generic ;-(
>>>>
>>>>> I have checked ceph mailing, stefan seem to have resolved his
>>>>> problem with dual core with bios update !
>>>> Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on
>>>> Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz
>>>> Single Xeon vs. 2.5Ghz Dual Xeon.
>>>>
>>>> Right now RBD in KVM is limited by CPU speed.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel@pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel@pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html