From: Mark Nelson <mark.nelson@inktank.com>
To: Andreas Joachim Peters <Andreas.Joachim.Peters@cern.ch>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: Object Write Latency
Date: Fri, 20 Sep 2013 08:11:29 -0500 [thread overview]
Message-ID: <523C4981.9080406@inktank.com> (raw)
In-Reply-To: <3472A07E6605974CBC9BC573F1BC02E4A52724A9@PLOXCHG03.cern.ch>
On 09/20/2013 07:27 AM, Andreas Joachim Peters wrote:
> Hi,
Hi Andreas!
>
> we made some benchmarks about object read/write latencies on the CERN ceph installation.
>
> The cluster has 44 nodes and ~1k disks, all on 10GE and the pool configuration has 3 copies.
> Client & Server is 0.67.
>
> The latencies we observe (using tiny objects ... 5 bytes) on the idle pool:
>
> write full object(sync) ~65-80ms
> append to object ~60-75ms
> set xattr object ~65-80ms
> lock object ~65-80ms
> stat object ~1ms
>
> We seem to saturate the pools writing ~ 20k objects/s (= internally 60k/s).
Out of curiosity, how much difference do you see with write latencies if
you do the same thing to a pool with 1 copy?
>
> Is there an easy explanation for 80 ms (quasi without payload) and a possible tuning to reduce that?
> I measured (append few bytes +fsync) on such a disk around 33ms which explains probably part of the latency.
I've been wanting to really dig into object write latency in RADOS but
just haven't had the time to devote to it yet. I've been doing some
simple rados bench tests to a 8-SSD test node and am topping out at
about 8-9K write IOPS and 26K read IOPS (no replication) though with
little tuning. I suspect there are many areas in the code where we
could improve things.
>
> Then I tried with the async API to see if there is a difference in the measurement between wait_for_complete or wait_for_safe ... shouldn't wait_for_complete be much shorter, but I get always comparable results ...
Hrm, I'm going to let Sage or someone else comment on this.
>
> Thanks, Andreas.--
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2013-09-20 13:11 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-20 12:27 Object Write Latency Andreas Joachim Peters
2013-09-20 13:11 ` Mark Nelson [this message]
2013-09-23 7:34 ` Dan van der Ster
2013-09-20 14:47 ` Gregory Farnum
2013-09-23 7:37 ` Dan van der Ster
2013-09-23 18:18 ` Sage Weil
2013-09-24 9:13 ` Dan van der Ster
2013-09-24 10:04 ` Dan van der Ster
[not found] ` <etPan.52417ad3.3804823e.f19a@leseb-mba.local>
2013-09-24 12:30 ` Sébastien Han
2013-09-24 12:33 ` Dan van der Ster
2013-09-24 12:39 ` Sébastien Han
2013-09-20 15:34 ` Sage Weil
2013-09-23 7:39 ` Dan van der Ster
2013-09-23 15:38 ` Andreas Joachim Peters
2013-09-23 16:03 ` Mark Nelson
2013-09-23 21:40 ` Andreas Joachim Peters
2013-09-24 8:55 ` Dan van der Ster
2013-09-24 15:20 ` Sage Weil
2013-09-23 16:19 ` Sage Weil
2013-09-23 15:40 ` Sage Weil
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=523C4981.9080406@inktank.com \
--to=mark.nelson@inktank.com \
--cc=Andreas.Joachim.Peters@cern.ch \
--cc=ceph-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.