From: Mark Nelson <mark.nelson@inktank.com>
To: Gregory Farnum <greg@inktank.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: Ceph performance on Ubuntu Oneiric vs Ubuntu Precise
Date: Mon, 18 Jun 2012 13:07:53 -0500 [thread overview]
Message-ID: <4FDF6E79.6080203@inktank.com> (raw)
In-Reply-To: <CAPYLRzgM=_1CpXB3Kr+=RFdaC3KPiVHGQyRurmTZJ=Zopsx=QA@mail.gmail.com>
Hi Greg,
Yep, 3 monitors each on their own node.
Mark
On 06/18/2012 01:04 PM, Gregory Farnum wrote:
> Do I correctly assume that these nodes hosted only the OSDs, and the
> monitors were on a separate node?
>
> On Mon, Jun 18, 2012 at 10:56 AM, Mark Nelson<mark.nelson@inktank.com> wrote:
>> Hi Guys,
>>
>> I've been tracking down some performance issues over the past month with our
>> internal test nodes and believe I have narrowed it down to something related
>> to Ubuntu Oneiric. Tests done on nodes running Ubuntu Precise are
>> significantly faster.
>>
>> One of the major differences between the releases is the support for syncfs
>> in libc. Theoretically this shouldn't have a big effect on btrfs so I'm not
>> totally sure that this is the culprit. Having said that, previous tests
>> showed good SSD performance on Oneiric leading me to believe the lower
>> latency mitigates the effect. Some of spinning disk seekwatcher results for
>> Oneiric are quite strange with long periods of inactivity on the OSD data
>> disks.
>>
>> I wanted to post these results for those of you who have had performance
>> problems in the past. If you are continuing to have issues, you may want to
>> try testing on precise and see if you notice any changes. It is possible
>> that all of this could be specific to our internal testing nodes, so I
>> wouldn't mind hearing if other people have seen similar behavior.
>>
>> These tests were done using rados bench with 16 concurrent requests. There
>> are two nodes that each have a single 7200rpm OSD data disk and journal on a
>> second 7200rpm disk. Replication is set at the default level (2). Kernel
>> is 3.4 in all cases.
>>
>> Here's a run down (Numbers are MB/s)
>>
>> 4KB Requests
>>
>> BTRFS EXT4 XFS
>> Ceph 0.46/Oneiric: 0.073 0.694 0.723
>> Ceph 0.46/Precise: 2.15 2.031 1.546
>> Ceph 0.47.2/Oneiric: 1.072 0.836 0.749
>> Ceph 0.47.2/Precise: 2.566 2.579 1.498
>>
>> 128KB Requests:
>>
>> BTRFS EXT4 XFS
>> Ceph 0.46/Oneiric: 11.874 20.066 12.641
>> Ceph 0.46/Precise: 49.304 39.736 38.982
>> Ceph 0.47.2/Oneiric: 13.81 19.05 12.739
>> Ceph 0.47.2/Precise: 47.943 49.655 36.764
>>
>>
>> 4MB Requests:
>>
>> BTRFS EXT4 XFS
>> Ceph 0.46/Oneiric: 110.202 26.58 15.445
>> Ceph 0.46/Precise: 135.975 128.759 106.426
>> Ceph 0.47.2/Oneiric: 91.337 46.277 23.897
>> Ceph 0.47.2/Precise: 136.906 134.955 106.545
>>
>> I've posted seekwatcher results for all of the tests:
>>
>> Ceph 0.46/Oneiric: http://nhm.ceph.com/movies/sprint/test2
>> Ceph 0.46/Precise: http://nhm.ceph.com/movies/sprint/test3
>> Ceph 0.47.2/Oneiric: http://nhm.ceph.com/movies/sprint/test4
>> Ceph 0.47.2/Precise: http://nhm.ceph.com/movies/sprint/test5
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-06-18 18:08 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-06-18 17:56 Ceph performance on Ubuntu Oneiric vs Ubuntu Precise Mark Nelson
2012-06-18 18:04 ` Gregory Farnum
2012-06-18 18:07 ` Mark Nelson [this message]
2012-06-18 18:12 ` Tommi Virtanen
2012-06-18 18:17 ` Mark Nelson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FDF6E79.6080203@inktank.com \
--to=mark.nelson@inktank.com \
--cc=ceph-devel@vger.kernel.org \
--cc=greg@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox