Re: Ceph performance

[Roman Alekseev <rs.alekseev@gmail.com>]

On 29.10.2012 22:57, Sam Lang wrote:
>
> Hi Roman,
>
> Is this with the ceph fuse client or the ceph kernel module?
>
> Its not surprising that the local file system (/home) is so much 
> faster than a mounted ceph volume, especially the first time the 
> directory tree is traversed (metadata results are cached at the client 
> to improve performance).  Try running the same find command on the 
> ceph volume and see if the cached results at the client improve 
> performance at all.
>
> In order to understand what the performance of ceph should be capable 
> of doing with your deployment for this specific workload, you should 
> run iperf between two nodes to get an idea of your latency limits.
>
> Also, I noticed that the real timings you listed for ceph and /home 
> are offset by exactly 17 minutes (user and sys are identical).  Was 
> that a copy/paste error, by chance?
>
> -sam
>
> On 10/29/2012 09:01 AM, Roman Alekseev wrote:
>> Hi,
>>
>> Kindly guide me how to improve performance on the cluster which consist
>> of 5 dedicated servers:
>>
>> - ceph.conf: http://pastebin.com/hT3qEhUF
>> - file system on all drives is ext4
>> - mount options "user_xattr"
>> - each server has :
>> CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
>> MEM: 4Gb DDR2
>> - 1Gb network
>>
>> Simple test:
>>
>> mounted as ceph
>> root@client1:/mnt/mycephfs# time find . | wc -l
>> 83932
>>
>> real    17m55.399s
>> user    0m0.152s
>> sys    0m1.528s
>>
>> on 1 HDD:
>>
>> root@client1:/home# time find . | wc -l
>> 83932
>>
>> real    0m55.399s
>> user    0m0.152s
>> sys    0m1.528s
>>
>> Please help me to find out the issue. Thanks.
>>
>
Hi Sam,

     I use the Ceph fs only as kernel module, because we need to get its 
powerful performance but as I can see it is slower then distributed file 
system based on fuse, for example, MooseFS performed the same test for 3 
min.
Here is the result iperf test beetwen client and osd server:
root@asrv151:~# iperf -c client -i 1
------------------------------------------------------------
Client connecting to clientIP, TCP port 5001
TCP window size: 96.1 KByte (default)
------------------------------------------------------------
[  3] local osd_server port 50106 connected with clientIP port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    112 MBytes    941 Mbits/sec
[  3]  1.0- 2.0 sec    110 MBytes    924 Mbits/sec
[  3]  2.0- 3.0 sec    108 MBytes    905 Mbits/sec
[  3]  3.0- 4.0 sec    109 MBytes    917 Mbits/sec
[  3]  4.0- 5.0 sec    110 MBytes    926 Mbits/sec
[  3]  5.0- 6.0 sec    109 MBytes    915 Mbits/sec
[  3]  6.0- 7.0 sec    110 MBytes    926 Mbits/sec
[  3]  7.0- 8.0 sec    108 MBytes    908 Mbits/sec
[  3]  8.0- 9.0 sec    107 MBytes    897 Mbits/sec
[  3]  9.0-10.0 sec    106 MBytes    886 Mbits/sec
[  3]  0.0-10.0 sec  1.06 GBytes    914 Mbits/sec

ceph -w results:

  health HEALTH_OK
    monmap e3: 3 mons at {a=mon.a:6789/0,b=mon.b:6789/0,c=mon.c:6789/0}, 
election epoch 10, quorum 0,1,2 a,b,c
    osdmap e132: 5 osds: 5 up, 5 in
     pgmap v11720: 384 pgs: 384 active+clean; 1880 MB data, 10679 MB 
used, 5185 GB / 5473 GB avail
    mdsmap e4: 1/1/1 up {0=a=up:active}

2012-10-30 12:23:09.830677 osd.2 [WRN] slow request 30.135787 seconds 
old, received at 2012-10-30 12:22:39.694780: osd_op(mds.0.1:309216 
10000017163.00000000 [setxattr path (69),setxattr parent (196),tmapput 
0~596] 1.724c80f7) v4 currently waiting for sub ops
2012-10-30 12:23:10.109637 mon.0 [INF] pgmap v11720: 384 pgs: 384 
active+clean; 1880 MB data, 10679 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:12.918038 mon.0 [INF] pgmap v11721: 384 pgs: 384 
active+clean; 1880 MB data, 10680 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:13.977044 mon.0 [INF] pgmap v11722: 384 pgs: 384 
active+clean; 1880 MB data, 10681 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:10.587391 osd.3 [WRN] 6 slow requests, 6 included 
below; oldest blocked for > 30.808352 secs
2012-10-30 12:23:10.587398 osd.3 [WRN] slow request 30.808352 seconds 
old, received at 2012-10-30 12:22:39.778971: osd_op(mds.0.1:308701 
200.000002e5 [write 976010~5402] 1.adbeb1a) v4 currently waiting for sub ops
2012-10-30 12:23:10.587403 osd.3 [WRN] slow request 30.796417 seconds 
old, received at 2012-10-30 12:22:39.790906: osd_op(mds.0.1:308702 
200.000002e5 [write 981412~6019] 1.adbeb1a) v4 currently waiting for sub ops
2012-10-30 12:23:10.587408 osd.3 [WRN] slow request 30.796347 seconds 
old, received at 2012-10-30 12:22:39.790976: osd_op(mds.0.1:308703 
200.000002e5 [write 987431~61892] 1.adbeb1a) v4 currently waiting for 
sub ops
2012-10-30 12:23:10.587413 osd.3 [WRN] slow request 30.530228 seconds 
old, received at 2012-10-30 12:22:40.057095: osd_op(mds.0.1:308704 
200.000002e5 [write 1049323~6630] 1.adbeb1a) v4 currently waiting for 
sub ops
2012-10-30 12:23:10.587417 osd.3 [WRN] slow request 30.530027 seconds 
old, received at 2012-10-30 12:22:40.057296: osd_op(mds.0.1:308705 
200.000002e5 [write 1055953~20679] 1.adbeb1a) v4 currently waiting for 
sub ops


At the same time I'm copy data to ceph mounted storage.

I dunno what can I do to resolve this problem :(
Any advices will be greatly appreciated.

-- 
Kind regards,

R. Alekseev

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html