Re: [Performance] Improvement on DB Performance

[Mike Dawson <mike.dawson@cloudapt.com>]

Haomai,

Thanks for finding this!


Sage,

We have a client that runs an io intensive, closed-source software 
package that seems to issue overzealous flushes which may benefit from 
this patch (or the other methods you mention). If you were to spin a wip 
build based on Dumpling, I'll be a willing tester.

Thanks,
Mike Dawson

On 5/21/2014 11:23 AM, Sage Weil wrote:
> On Wed, 21 May 2014, Haomai Wang wrote:
>> I pushed the commit to fix this problem(https://github.com/ceph/ceph/pull/1848).
>>
>> With test program(Each sync request is issued with ten write request),
>> a significant improvement is noticed.
>>
>> aio_flush                          sum: 914750     avg: 1239   count:
>> 738      max: 4714   min: 1011
>> flush_set                          sum: 904200     avg: 1225   count:
>> 738      max: 4698   min: 999
>> flush                              sum: 641648     avg: 173    count:
>> 3690     max: 1340   min: 128
>>
>> Compared to last mail, it reduce each aio_flush request to 1239 ns
>> instead of 24145 ns.
>
> Good catch!  That's a great improvement.
>
> The patch looks clearly correct.  We can probably do even better by
> putting the Objects on a list when they get the first dirty buffer so that
> we only cycle through the dirty ones.  Or, have a global list of dirty
> buffers (instead of dirty objects -> dirty buffers).
>
> sage
>
>>
>> I hope it's the root cause for db on rbd performance.
>>
>> On Wed, May 21, 2014 at 6:15 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>> Hi all,
>>>
>>> I remember there exists discuss about DB(mysql) performance on rbd.
>>> Recently I test mysql-bench with rbd and found awful performance. So I
>>> dive into it and find that main cause is "flush" request from guest.
>>> As we know, applications such as mysql, ceph has own journal for
>>> durable and journal usually send sync&direct io. If fs barrier is on,
>>> each sync io operation make kernel issue "sync"(barrier) request to
>>> block device. Here, qemu will call "rbd_aio_flush" to apply.
>>>
>>> Via systemtap, I found a amazing thing:
>>> aio_flush                          sum: 4177085    avg: 24145  count:
>>> 173      max: 28172  min: 22747
>>> flush_set                          sum: 4172116    avg: 24116  count:
>>> 173      max: 28034  min: 22733
>>> flush                              sum: 3029910    avg: 4      count:
>>> 670477   max: 1893   min: 3
>>>
>>> This statistic info is gathered in 5s. Most of consuming time is on
>>> "ObjectCacher::flush". What's more, with time increasing, the flush
>>> count will be increasing.
>>>
>>> After view source, I find the root cause is "ObjectCacher::flush_set",
>>> it will iterator the "object_set" and look for dirty buffer. And
>>> "object_set"  contains all objects ever opened.  For example:
>>>
>>> 2014-05-21 18:01:37.959013 7f785c7c6700  0 objectcacher flush_set
>>> total: 5919 flushed: 5
>>> 2014-05-21 18:01:37.999698 7f785c7c6700  0 objectcacher flush_set
>>> total: 5919 flushed: 5
>>> 2014-05-21 18:01:38.038405 7f785c7c6700  0 objectcacher flush_set
>>> total: 5920 flushed: 5
>>> 2014-05-21 18:01:38.080118 7f785c7c6700  0 objectcacher flush_set
>>> total: 5920 flushed: 5
>>> 2014-05-21 18:01:38.119792 7f785c7c6700  0 objectcacher flush_set
>>> total: 5921 flushed: 5
>>> 2014-05-21 18:01:38.162004 7f785c7c6700  0 objectcacher flush_set
>>> total: 5922 flushed: 5
>>> 2014-05-21 18:01:38.202755 7f785c7c6700  0 objectcacher flush_set
>>> total: 5923 flushed: 5
>>> 2014-05-21 18:01:38.243880 7f785c7c6700  0 objectcacher flush_set
>>> total: 5923 flushed: 5
>>> 2014-05-21 18:01:38.284399 7f785c7c6700  0 objectcacher flush_set
>>> total: 5923 flushed: 5
>>>
>>> These logs record the iteration info, the loop will check 5920 objects
>>> but only 5 objects are dirty.
>>>
>>> So I think the solution is make "ObjectCacher::flush_set" only
>>> iterator the objects which is dirty.
>>>
>>> --
>>> Best Regards,
>>>
>>> Wheat
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>