* low performance of ceph, why ?
@ 2012-07-26 14:53 Hosfore
2012-07-26 15:40 ` Mark Nelson
0 siblings, 1 reply; 3+ messages in thread
From: Hosfore @ 2012-07-26 14:53 UTC (permalink / raw)
To: ceph-devel
Now I configure a small cluster of ceph, with mon and mds on the same machine,
two osds on another two machine, and I have configure the data and journal dir
on two disks separately. But the test result of mdtest is so poor as below:
------------------------------------------------------------------------------
fs90:/mnt/ceph # mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
-- started at 07/26/2012 22:37:08 --
mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes
Command line used: mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
Path: /mnt/ceph
FS: 1.8 TiB Used FS: 5.8% Inodes: 0.0 Mi Used Inodes: 100.0%
1 tasks, 200 files/directories
SUMMARY: (of 2 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation: 2218.451 1138.848 1678.650 539.802
Directory stat : 870187.552 840541.884 855364.718 14822.834
Directory removal : 2830.938 2828.647 2829.793 1.146
File creation : 1987.224 1972.523 1979.873 7.350
File stat : 854237.067 850771.602 852504.335 1732.732
File removal : 2651.082 2164.132 2407.607 243.475
Tree creation : 1680.410 1559.801 1620.105 60.305
Tree removal : 1.011 0.578 0.794 0.216
-- finished at 07/26/2012 22:37:11 --
-----------------------------------------------------------------------------
The file system of osd disk is ext4. So is the config of mds or osd leads to
this poor result ? I have try this test on the local ext3 file system and the
dir creation speed is usually more than 10000 times per second. My ceph.conf
is as below:
-------------------------------------------------------------------------------
[global]
[mon]
mon data = /data/mon$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 1
[mon.0]
host = fs98
mon addr = 10.0.2.98:6789
[mds]
; where the mds keeps it's secret encryption keys
keyring = /data/keyring.$name
; where the mds keeps it's secret encryption keys
keyring = /data/keyring.$name
mds cache size = 3000000
[mds.alpha]
host = fs98
[osd]
; This is where the btrfs volume will be mounted.
osd data = /data
filestore xattr use omap = true
osd journal = /ceph/journal
osd journal size = 512
[osd.0]
host = fs97
[osd.1]
host= fs91
-------------------------------------------------------------------------------
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: low performance of ceph, why ?
2012-07-26 14:53 low performance of ceph, why ? Hosfore
@ 2012-07-26 15:40 ` Mark Nelson
[not found] ` <1ffadfdf.26.138c448f479.Coremail.18010151050@126.com>
0 siblings, 1 reply; 3+ messages in thread
From: Mark Nelson @ 2012-07-26 15:40 UTC (permalink / raw)
To: Hosfore; +Cc: ceph-devel
On 07/26/2012 09:53 AM, Hosfore wrote:
> Now I configure a small cluster of ceph, with mon and mds on the same machine,
> two osds on another two machine, and I have configure the data and journal dir
> on two disks separately. But the test result of mdtest is so poor as below:
> ------------------------------------------------------------------------------
> fs90:/mnt/ceph # mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
> -- started at 07/26/2012 22:37:08 --
>
> mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes
> Command line used: mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
> Path: /mnt/ceph
> FS: 1.8 TiB Used FS: 5.8% Inodes: 0.0 Mi Used Inodes: 100.0%
>
> 1 tasks, 200 files/directories
>
> SUMMARY: (of 2 iterations)
> Operation Max Min Mean Std Dev
> --------- --- --- ---- -------
> Directory creation: 2218.451 1138.848 1678.650 539.802
> Directory stat : 870187.552 840541.884 855364.718 14822.834
> Directory removal : 2830.938 2828.647 2829.793 1.146
> File creation : 1987.224 1972.523 1979.873 7.350
> File stat : 854237.067 850771.602 852504.335 1732.732
> File removal : 2651.082 2164.132 2407.607 243.475
> Tree creation : 1680.410 1559.801 1620.105 60.305
> Tree removal : 1.011 0.578 0.794 0.216
>
> -- finished at 07/26/2012 22:37:11 --
> -----------------------------------------------------------------------------
Hi,
Thanks for taking the time to test this. I haven't been able to really
dig into metadata tests like mdtest yet, though it's on my list of
things to do! For now, my guess is that all of the overhead by having
more layers of code, network, and general lack of optimizations with
CephFS are probably holding things back. This is something we will
eventually be working on, but right now our focus is more on RadosGW and
RBD. You may want to look at:
http://ceph.newdream.net/papers/weil-ceph-osdi06.pdf
On page 10 there are some MDS performance numbers. With a single MDS it
looks like your numbers are roughly in-line with the makedirs and
makefiles numbers Sage reported at the time.
Mark
--
Mark Nelson
Performance Engineer
Inktank
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-07-27 2:05 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-07-26 14:53 low performance of ceph, why ? Hosfore
2012-07-26 15:40 ` Mark Nelson
[not found] ` <1ffadfdf.26.138c448f479.Coremail.18010151050@126.com>
2012-07-27 2:05 ` Mark Nelson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.