From: "hzwulibin" <hzwulibin@gmail.com>
To: ceph-devel <ceph-devel@vger.kernel.org>,
ceph-users <ceph-users@ceph.com>
Cc: Sage Weil <sage@newdream.net>, Haomai Wang <haomaiwang@gmail.com>
Subject: use object size of 32k rather than 4M
Date: Wed, 23 Dec 2015 20:00:29 +0800 [thread overview]
Message-ID: <567A8CD6.10707@gmail.com> (raw)
Hi, cephers, Sage and Haomai
Recently we stuck of the performance down problem when recoverying. The scene is simple:
1. run fio with rand write(bs=4k)
2. stop one osd; sleep 10; start the osd
3. the IOPS drop from 6K to about 200
We now know the SSD which that osd on is the bottleneck when recovery. After read the code, we find the IO of that
SSD come from two ways:
1. normal recovery IO
2. user IO but in the missing list, need to recovery the 4M object first.
So our first step is limit the recovery IO to slow down the stress of that SSD. That helps in some scene, but not this one.
We have 36 OSD with 3 replicas, so when one osd down, about 1/12 objects will be in degraded state.
When we run fio with 4k randwrite, about 1/12 io will stuck and need to recovery the 4M object first.
That really enlarge the stress the that SSD.
In order to reduce the enlarge impact, we want to change the default size of the object from 4M to 32k.
We know that will increase the number of the objects of one OSD and make remove process become longer.
Hmm, here i want to ask your guys is there any other potential problems will 32k size have? If no obvious problem, will could dive into
it and do more test on it.
Many thanks!
--------------
hzwulibin
2015-12-23
next reply other threads:[~2015-12-23 12:00 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-23 12:00 hzwulibin [this message]
[not found] ` <567A8CD6.10707-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2015-12-23 12:57 ` use object size of 32k rather than 4M Van Leeuwen, Robert
2015-12-23 13:14 ` [ceph-users] " hzwulibin
2015-12-23 13:57 ` Van Leeuwen, Robert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=567A8CD6.10707@gmail.com \
--to=hzwulibin@gmail.com \
--cc=ceph-devel@vger.kernel.org \
--cc=ceph-users@ceph.com \
--cc=haomaiwang@gmail.com \
--cc=sage@newdream.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.