Re: Ceph performance on Ubuntu Oneiric vs Ubuntu Precise

[Mark Nelson <mark.nelson@inktank.com>]

Hi Greg,

Yep, 3 monitors each on their own node.

Mark

On 06/18/2012 01:04 PM, Gregory Farnum wrote:
> Do I correctly assume that these nodes hosted only the OSDs, and the
> monitors were on a separate node?
>
> On Mon, Jun 18, 2012 at 10:56 AM, Mark Nelson<mark.nelson@inktank.com>  wrote:
>> Hi Guys,
>>
>> I've been tracking down some performance issues over the past month with our
>> internal test nodes and believe I have narrowed it down to something related
>> to Ubuntu Oneiric.  Tests done on nodes running Ubuntu Precise are
>> significantly faster.
>>
>> One of the major differences between the releases is the support for syncfs
>> in libc.  Theoretically this shouldn't have a big effect on btrfs so I'm not
>> totally sure that this is the culprit.  Having said that, previous tests
>> showed good SSD performance on Oneiric leading me to believe the lower
>> latency mitigates the effect.  Some of spinning disk seekwatcher results for
>> Oneiric are quite strange with long periods of inactivity on the OSD data
>> disks.
>>
>> I wanted to post these results for those of you who have had performance
>> problems in the past.  If you are continuing to have issues, you may want to
>> try testing on precise and see if you notice any changes.  It is possible
>> that all of this could be specific to our internal testing nodes, so I
>> wouldn't mind hearing if other people have seen similar behavior.
>>
>> These tests were done using rados bench with 16 concurrent requests. There
>> are two nodes that each have a single 7200rpm OSD data disk and journal on a
>> second 7200rpm disk.  Replication is set at the default level (2).  Kernel
>> is 3.4 in all cases.
>>
>> Here's a run down (Numbers are MB/s)
>>
>> 4KB Requests
>>
>>                         BTRFS   EXT4    XFS
>> Ceph 0.46/Oneiric:      0.073   0.694   0.723
>> Ceph 0.46/Precise:      2.15    2.031   1.546
>> Ceph 0.47.2/Oneiric:    1.072   0.836   0.749
>> Ceph 0.47.2/Precise:    2.566   2.579   1.498
>>
>> 128KB Requests:
>>
>>                         BTRFS   EXT4    XFS
>> Ceph 0.46/Oneiric:      11.874  20.066  12.641
>> Ceph 0.46/Precise:      49.304  39.736  38.982
>> Ceph 0.47.2/Oneiric:    13.81   19.05   12.739
>> Ceph 0.47.2/Precise:    47.943  49.655  36.764
>>
>>
>> 4MB Requests:
>>
>>                         BTRFS   EXT4    XFS
>> Ceph 0.46/Oneiric:      110.202 26.58   15.445
>> Ceph 0.46/Precise:      135.975 128.759 106.426
>> Ceph 0.47.2/Oneiric:    91.337  46.277  23.897
>> Ceph 0.47.2/Precise:    136.906 134.955 106.545
>>
>> I've posted seekwatcher results for all of the tests:
>>
>> Ceph 0.46/Oneiric:      http://nhm.ceph.com/movies/sprint/test2
>> Ceph 0.46/Precise:      http://nhm.ceph.com/movies/sprint/test3
>> Ceph 0.47.2/Oneiric:    http://nhm.ceph.com/movies/sprint/test4
>> Ceph 0.47.2/Precise:    http://nhm.ceph.com/movies/sprint/test5
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html