low performance of ceph, why ?

low performance of ceph, why ?

[Hosfore]

Now I configure a small cluster of ceph, with mon and mds on the same machine, 
two osds on another two machine, and I have configure the data and journal dir 
on two disks separately. But the test result of mdtest is so poor as below:
------------------------------------------------------------------------------
fs90:/mnt/ceph # mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
-- started at 07/26/2012 22:37:08 --

mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes
Command line used: mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
Path: /mnt/ceph
FS: 1.8 TiB   Used FS: 5.8%   Inodes: 0.0 Mi   Used Inodes: 100.0%

1 tasks, 200 files/directories

SUMMARY: (of 2 iterations)
   Operation                  Max        Min       Mean    Std Dev
   ---------                  ---        ---       ----    -------
   Directory creation:   2218.451   1138.848   1678.650    539.802
   Directory stat    : 870187.552 840541.884 855364.718  14822.834
   Directory removal :   2830.938   2828.647   2829.793      1.146
   File creation     :   1987.224   1972.523   1979.873      7.350
   File stat         : 854237.067 850771.602 852504.335   1732.732
   File removal      :   2651.082   2164.132   2407.607    243.475
   Tree creation     :   1680.410   1559.801   1620.105     60.305
   Tree removal      :      1.011      0.578      0.794      0.216

-- finished at 07/26/2012 22:37:11 --
-----------------------------------------------------------------------------

The file system of osd disk is ext4. So is the config of mds or osd leads to 
this poor result ? I have try this test on the local ext3 file system and the 
dir creation speed is usually more than 10000 times per second. My ceph.conf 
is as below:
-------------------------------------------------------------------------------
[global]

[mon]
        mon data = /data/mon$id

        ; some minimal logging (just message traffic) to aid debugging
        debug ms = 1

[mon.0]
        host = fs98
        mon addr = 10.0.2.98:6789

[mds]
        ; where the mds keeps it's secret encryption keys
        keyring = /data/keyring.$name
       ; where the mds keeps it's secret encryption keys
        keyring = /data/keyring.$name
        mds cache size = 3000000

[mds.alpha]
        host = fs98

[osd]
        ; This is where the btrfs volume will be mounted.
        osd data = /data
        filestore xattr use omap = true

        osd journal = /ceph/journal
        osd journal size = 512

[osd.0]
        host = fs97
[osd.1]
        host= fs91

-------------------------------------------------------------------------------

Re: low performance of ceph, why ?

[Mark Nelson]

On 07/26/2012 09:53 AM, Hosfore wrote:
> Now I configure a small cluster of ceph, with mon and mds on the same machine,
> two osds on another two machine, and I have configure the data and journal dir
> on two disks separately. But the test result of mdtest is so poor as below:
> ------------------------------------------------------------------------------
> fs90:/mnt/ceph # mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
> -- started at 07/26/2012 22:37:08 --
>
> mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes
> Command line used: mdtest -d /mnt/ceph/tt3 -n 200 -i 2 -w 0
> Path: /mnt/ceph
> FS: 1.8 TiB   Used FS: 5.8%   Inodes: 0.0 Mi   Used Inodes: 100.0%
>
> 1 tasks, 200 files/directories
>
> SUMMARY: (of 2 iterations)
>     Operation                  Max        Min       Mean    Std Dev
>     ---------                  ---        ---       ----    -------
>     Directory creation:   2218.451   1138.848   1678.650    539.802
>     Directory stat    : 870187.552 840541.884 855364.718  14822.834
>     Directory removal :   2830.938   2828.647   2829.793      1.146
>     File creation     :   1987.224   1972.523   1979.873      7.350
>     File stat         : 854237.067 850771.602 852504.335   1732.732
>     File removal      :   2651.082   2164.132   2407.607    243.475
>     Tree creation     :   1680.410   1559.801   1620.105     60.305
>     Tree removal      :      1.011      0.578      0.794      0.216
>
> -- finished at 07/26/2012 22:37:11 --
> -----------------------------------------------------------------------------

Hi,

Thanks for taking the time to test this.  I haven't been able to really 
dig into metadata tests like mdtest yet, though it's on my list of 
things to do!  For now, my guess is that all of the overhead by having 
more layers of code, network, and general lack of optimizations with 
CephFS are probably holding things back.  This is something we will 
eventually be working on, but right now our focus is more on RadosGW and 
RBD.  You may want to look at:

http://ceph.newdream.net/papers/weil-ceph-osdi06.pdf

On page 10 there are some MDS performance numbers.  With a single MDS it 
looks like your numbers are roughly in-line with the makedirs and 
makefiles numbers Sage reported at the time.

Mark

-- 
Mark Nelson
Performance Engineer
Inktank

Re: low performance of ceph, why ?

[Mark Nelson]

On 07/26/2012 12:15 PM, 马四 wrote:
> Hi, Mark
> 　　Thanks for your help. May I know some details about your cluster that
> could support 250 000 metadata operations per second as described in the
> paper abstract:　
> ＂Performance measurements　under a variety of workloads show that Ceph
> has excellent
> 　I/O performance and calable metadata management,　supportingmore than
> 250,000
> metadata operations　per second．＂
> How many mdses and osds are there in the cluster?

It looks like the 250,000 number likely came from the number of 
openssh+include and openssh+lib opeartions/MDS/second that are shown on 
page 10 when there are 128 MDSes active.  I believe Multi-MDS setups are 
considered experimental at this time, so it isn't something you probably 
want to do in production.

> Is the OSD supported by btrfs ?

I believe these tests were done with a filesystem that was originally 
developed for ceph but replaced later with btrfs/others.

> Now, I have known that by separating journal and data of OSD and
> replacing ext3 with btrfs will
> speed up the system, could you prompt me more tips on optimizing the
> perfomance and
> may I have your config file ceph.conf ?

Putting the journal and data disks on seperate drives will certainly 
help.  You will also want to mount the underlying filesystem with 
noatime.  Turning off the filestore flusher may help or hurt performance 
depending on what is going on.

A simple example conf file can be found here:
http://ceph.com/wiki/Cluster_configuration

> BTW, in P10, ie. Part 6.2, the paper talks about diskless MDS and MDS
> without a local disk, but
> look up through the configuration items of MDS, I just could not find
> the item for configure
> the local disk for MDS, could you tell me how to configure this ? Thanks.

Honestly I'm not sure.  Sage may have more input here.

> Sincerely,
> Hosfore

Thanks,
Mark
-- 
Mark Nelson
Performance Engineer
Inktank
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html