Ceph performance

Ceph performance

[Roman Alekseev]

Hi,

Kindly guide me how to improve performance on the cluster which consist 
of 5 dedicated servers:

- ceph.conf: http://pastebin.com/hT3qEhUF
- file system on all drives is ext4
- mount options "user_xattr"
- each server has :
CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
MEM: 4Gb DDR2
- 1Gb network

Simple test:

mounted as ceph
root@client1:/mnt/mycephfs# time find . | wc -l
83932

real    17m55.399s
user    0m0.152s
sys    0m1.528s

on 1 HDD:

root@client1:/home# time find . | wc -l
83932

real    0m55.399s
user    0m0.152s
sys    0m1.528s

Please help me to find out the issue. Thanks.

-- 
Kind regards,

R. Alekseev

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Sam Lang]

Hi Roman,

Is this with the ceph fuse client or the ceph kernel module?

Its not surprising that the local file system (/home) is so much faster 
than a mounted ceph volume, especially the first time the directory tree 
is traversed (metadata results are cached at the client to improve 
performance).  Try running the same find command on the ceph volume and 
see if the cached results at the client improve performance at all.

In order to understand what the performance of ceph should be capable of 
doing with your deployment for this specific workload, you should run 
iperf between two nodes to get an idea of your latency limits.

Also, I noticed that the real timings you listed for ceph and /home are 
offset by exactly 17 minutes (user and sys are identical).  Was that a 
copy/paste error, by chance?

-sam

On 10/29/2012 09:01 AM, Roman Alekseev wrote:
> Hi,
>
> Kindly guide me how to improve performance on the cluster which consist
> of 5 dedicated servers:
>
> - ceph.conf: http://pastebin.com/hT3qEhUF
> - file system on all drives is ext4
> - mount options "user_xattr"
> - each server has :
> CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
> MEM: 4Gb DDR2
> - 1Gb network
>
> Simple test:
>
> mounted as ceph
> root@client1:/mnt/mycephfs# time find . | wc -l
> 83932
>
> real    17m55.399s
> user    0m0.152s
> sys    0m1.528s
>
> on 1 HDD:
>
> root@client1:/home# time find . | wc -l
> 83932
>
> real    0m55.399s
> user    0m0.152s
> sys    0m1.528s
>
> Please help me to find out the issue. Thanks.
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Roman Alekseev]

On 29.10.2012 22:57, Sam Lang wrote:
>
> Hi Roman,
>
> Is this with the ceph fuse client or the ceph kernel module?
>
> Its not surprising that the local file system (/home) is so much 
> faster than a mounted ceph volume, especially the first time the 
> directory tree is traversed (metadata results are cached at the client 
> to improve performance).  Try running the same find command on the 
> ceph volume and see if the cached results at the client improve 
> performance at all.
>
> In order to understand what the performance of ceph should be capable 
> of doing with your deployment for this specific workload, you should 
> run iperf between two nodes to get an idea of your latency limits.
>
> Also, I noticed that the real timings you listed for ceph and /home 
> are offset by exactly 17 minutes (user and sys are identical).  Was 
> that a copy/paste error, by chance?
>
> -sam
>
> On 10/29/2012 09:01 AM, Roman Alekseev wrote:
>> Hi,
>>
>> Kindly guide me how to improve performance on the cluster which consist
>> of 5 dedicated servers:
>>
>> - ceph.conf: http://pastebin.com/hT3qEhUF
>> - file system on all drives is ext4
>> - mount options "user_xattr"
>> - each server has :
>> CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
>> MEM: 4Gb DDR2
>> - 1Gb network
>>
>> Simple test:
>>
>> mounted as ceph
>> root@client1:/mnt/mycephfs# time find . | wc -l
>> 83932
>>
>> real    17m55.399s
>> user    0m0.152s
>> sys    0m1.528s
>>
>> on 1 HDD:
>>
>> root@client1:/home# time find . | wc -l
>> 83932
>>
>> real    0m55.399s
>> user    0m0.152s
>> sys    0m1.528s
>>
>> Please help me to find out the issue. Thanks.
>>
>
Hi Sam,

     I use the Ceph fs only as kernel module, because we need to get its 
powerful performance but as I can see it is slower then distributed file 
system based on fuse, for example, MooseFS performed the same test for 3 
min.
Here is the result iperf test beetwen client and osd server:
root@asrv151:~# iperf -c client -i 1
------------------------------------------------------------
Client connecting to clientIP, TCP port 5001
TCP window size: 96.1 KByte (default)
------------------------------------------------------------
[  3] local osd_server port 50106 connected with clientIP port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    112 MBytes    941 Mbits/sec
[  3]  1.0- 2.0 sec    110 MBytes    924 Mbits/sec
[  3]  2.0- 3.0 sec    108 MBytes    905 Mbits/sec
[  3]  3.0- 4.0 sec    109 MBytes    917 Mbits/sec
[  3]  4.0- 5.0 sec    110 MBytes    926 Mbits/sec
[  3]  5.0- 6.0 sec    109 MBytes    915 Mbits/sec
[  3]  6.0- 7.0 sec    110 MBytes    926 Mbits/sec
[  3]  7.0- 8.0 sec    108 MBytes    908 Mbits/sec
[  3]  8.0- 9.0 sec    107 MBytes    897 Mbits/sec
[  3]  9.0-10.0 sec    106 MBytes    886 Mbits/sec
[  3]  0.0-10.0 sec  1.06 GBytes    914 Mbits/sec

ceph -w results:

  health HEALTH_OK
    monmap e3: 3 mons at {a=mon.a:6789/0,b=mon.b:6789/0,c=mon.c:6789/0}, 
election epoch 10, quorum 0,1,2 a,b,c
    osdmap e132: 5 osds: 5 up, 5 in
     pgmap v11720: 384 pgs: 384 active+clean; 1880 MB data, 10679 MB 
used, 5185 GB / 5473 GB avail
    mdsmap e4: 1/1/1 up {0=a=up:active}

2012-10-30 12:23:09.830677 osd.2 [WRN] slow request 30.135787 seconds 
old, received at 2012-10-30 12:22:39.694780: osd_op(mds.0.1:309216 
10000017163.00000000 [setxattr path (69),setxattr parent (196),tmapput 
0~596] 1.724c80f7) v4 currently waiting for sub ops
2012-10-30 12:23:10.109637 mon.0 [INF] pgmap v11720: 384 pgs: 384 
active+clean; 1880 MB data, 10679 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:12.918038 mon.0 [INF] pgmap v11721: 384 pgs: 384 
active+clean; 1880 MB data, 10680 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:13.977044 mon.0 [INF] pgmap v11722: 384 pgs: 384 
active+clean; 1880 MB data, 10681 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:10.587391 osd.3 [WRN] 6 slow requests, 6 included 
below; oldest blocked for > 30.808352 secs
2012-10-30 12:23:10.587398 osd.3 [WRN] slow request 30.808352 seconds 
old, received at 2012-10-30 12:22:39.778971: osd_op(mds.0.1:308701 
200.000002e5 [write 976010~5402] 1.adbeb1a) v4 currently waiting for sub ops
2012-10-30 12:23:10.587403 osd.3 [WRN] slow request 30.796417 seconds 
old, received at 2012-10-30 12:22:39.790906: osd_op(mds.0.1:308702 
200.000002e5 [write 981412~6019] 1.adbeb1a) v4 currently waiting for sub ops
2012-10-30 12:23:10.587408 osd.3 [WRN] slow request 30.796347 seconds 
old, received at 2012-10-30 12:22:39.790976: osd_op(mds.0.1:308703 
200.000002e5 [write 987431~61892] 1.adbeb1a) v4 currently waiting for 
sub ops
2012-10-30 12:23:10.587413 osd.3 [WRN] slow request 30.530228 seconds 
old, received at 2012-10-30 12:22:40.057095: osd_op(mds.0.1:308704 
200.000002e5 [write 1049323~6630] 1.adbeb1a) v4 currently waiting for 
sub ops
2012-10-30 12:23:10.587417 osd.3 [WRN] slow request 30.530027 seconds 
old, received at 2012-10-30 12:22:40.057296: osd_op(mds.0.1:308705 
200.000002e5 [write 1055953~20679] 1.adbeb1a) v4 currently waiting for 
sub ops


At the same time I'm copy data to ceph mounted storage.

I dunno what can I do to resolve this problem :(
Any advices will be greatly appreciated.

-- 
Kind regards,

R. Alekseev

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Gregory Farnum]

On Tue, Oct 30, 2012 at 9:27 AM, Roman Alekseev <rs.alekseev@gmail.com> wrote:
> On 29.10.2012 22:57, Sam Lang wrote:
>>
>>
>> Hi Roman,
>>
>> Is this with the ceph fuse client or the ceph kernel module?
>>
>> Its not surprising that the local file system (/home) is so much faster
>> than a mounted ceph volume, especially the first time the directory tree is
>> traversed (metadata results are cached at the client to improve
>> performance).  Try running the same find command on the ceph volume and see
>> if the cached results at the client improve performance at all.
>>
>> In order to understand what the performance of ceph should be capable of
>> doing with your deployment for this specific workload, you should run iperf
>> between two nodes to get an idea of your latency limits.
>>
>> Also, I noticed that the real timings you listed for ceph and /home are
>> offset by exactly 17 minutes (user and sys are identical).  Was that a
>> copy/paste error, by chance?
>>
>> -sam
>>
>> On 10/29/2012 09:01 AM, Roman Alekseev wrote:
>>>
>>> Hi,
>>>
>>> Kindly guide me how to improve performance on the cluster which consist
>>> of 5 dedicated servers:
>>>
>>> - ceph.conf: http://pastebin.com/hT3qEhUF
>>> - file system on all drives is ext4
>>> - mount options "user_xattr"
>>> - each server has :
>>> CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
>>> MEM: 4Gb DDR2
>>> - 1Gb network
>>>
>>> Simple test:
>>>
>>> mounted as ceph
>>> root@client1:/mnt/mycephfs# time find . | wc -l
>>> 83932
>>>
>>> real    17m55.399s
>>> user    0m0.152s
>>> sys    0m1.528s
>>>
>>> on 1 HDD:
>>>
>>> root@client1:/home# time find . | wc -l
>>> 83932
>>>
>>> real    0m55.399s
>>> user    0m0.152s
>>> sys    0m1.528s
>>>
>>> Please help me to find out the issue. Thanks.
>>>
>>
> Hi Sam,
>
>     I use the Ceph fs only as kernel module, because we need to get its
> powerful performance but as I can see it is slower then distributed file
> system based on fuse, for example, MooseFS performed the same test for 3
> min.
> Here is the result iperf test beetwen client and osd server:
> root@asrv151:~# iperf -c client -i 1
> ------------------------------------------------------------
> Client connecting to clientIP, TCP port 5001
> TCP window size: 96.1 KByte (default)
> ------------------------------------------------------------
> [  3] local osd_server port 50106 connected with clientIP port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 1.0 sec    112 MBytes    941 Mbits/sec
> [  3]  1.0- 2.0 sec    110 MBytes    924 Mbits/sec
> [  3]  2.0- 3.0 sec    108 MBytes    905 Mbits/sec
> [  3]  3.0- 4.0 sec    109 MBytes    917 Mbits/sec
> [  3]  4.0- 5.0 sec    110 MBytes    926 Mbits/sec
> [  3]  5.0- 6.0 sec    109 MBytes    915 Mbits/sec
> [  3]  6.0- 7.0 sec    110 MBytes    926 Mbits/sec
> [  3]  7.0- 8.0 sec    108 MBytes    908 Mbits/sec
> [  3]  8.0- 9.0 sec    107 MBytes    897 Mbits/sec
> [  3]  9.0-10.0 sec    106 MBytes    886 Mbits/sec
> [  3]  0.0-10.0 sec  1.06 GBytes    914 Mbits/sec
>
> ceph -w results:
>
>  health HEALTH_OK
>    monmap e3: 3 mons at {a=mon.a:6789/0,b=mon.b:6789/0,c=mon.c:6789/0},
> election epoch 10, quorum 0,1,2 a,b,c
>    osdmap e132: 5 osds: 5 up, 5 in
>     pgmap v11720: 384 pgs: 384 active+clean; 1880 MB data, 10679 MB used,
> 5185 GB / 5473 GB avail
>    mdsmap e4: 1/1/1 up {0=a=up:active}
>
> 2012-10-30 12:23:09.830677 osd.2 [WRN] slow request 30.135787 seconds old,
> received at 2012-10-30 12:22:39.694780: osd_op(mds.0.1:309216
> 10000017163.00000000 [setxattr path (69),setxattr parent (196),tmapput
> 0~596] 1.724c80f7) v4 currently waiting for sub ops
> 2012-10-30 12:23:10.109637 mon.0 [INF] pgmap v11720: 384 pgs: 384
> active+clean; 1880 MB data, 10679 MB used, 5185 GB / 5473 GB avail
> 2012-10-30 12:23:12.918038 mon.0 [INF] pgmap v11721: 384 pgs: 384
> active+clean; 1880 MB data, 10680 MB used, 5185 GB / 5473 GB avail
> 2012-10-30 12:23:13.977044 mon.0 [INF] pgmap v11722: 384 pgs: 384
> active+clean; 1880 MB data, 10681 MB used, 5185 GB / 5473 GB avail
> 2012-10-30 12:23:10.587391 osd.3 [WRN] 6 slow requests, 6 included below;
> oldest blocked for > 30.808352 secs
> 2012-10-30 12:23:10.587398 osd.3 [WRN] slow request 30.808352 seconds old,
> received at 2012-10-30 12:22:39.778971: osd_op(mds.0.1:308701 200.000002e5
> [write 976010~5402] 1.adbeb1a) v4 currently waiting for sub ops
> 2012-10-30 12:23:10.587403 osd.3 [WRN] slow request 30.796417 seconds old,
> received at 2012-10-30 12:22:39.790906: osd_op(mds.0.1:308702 200.000002e5
> [write 981412~6019] 1.adbeb1a) v4 currently waiting for sub ops
> 2012-10-30 12:23:10.587408 osd.3 [WRN] slow request 30.796347 seconds old,
> received at 2012-10-30 12:22:39.790976: osd_op(mds.0.1:308703 200.000002e5
> [write 987431~61892] 1.adbeb1a) v4 currently waiting for sub ops
> 2012-10-30 12:23:10.587413 osd.3 [WRN] slow request 30.530228 seconds old,
> received at 2012-10-30 12:22:40.057095: osd_op(mds.0.1:308704 200.000002e5
> [write 1049323~6630] 1.adbeb1a) v4 currently waiting for sub ops
> 2012-10-30 12:23:10.587417 osd.3 [WRN] slow request 30.530027 seconds old,
> received at 2012-10-30 12:22:40.057296: osd_op(mds.0.1:308705 200.000002e5
> [write 1055953~20679] 1.adbeb1a) v4 currently waiting for sub ops
>
>
> At the same time I'm copy data to ceph mounted storage.
>
> I dunno what can I do to resolve this problem :(
> Any advices will be greatly appreciated.

Is it the same client copying data into cephfs or a different one?
I see here that you have several slow requests; it looks like maybe
you're overloading your disks. That could impact metadata lookups if
the MDS doesn't have everything cached; have you tried running this
test without data ingest? (Obviously we'd like it to be faster even
so, but if it's disk contention there's not a lot we can do.)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Maciej Gałkiewicz]

I have been experiencing the same problem. Tell us more about your
disks. Are they shared with OS and/or mds, journal? Paste ceph.conf.

regards
Maciej Galkiewicz

Re: Ceph performance

[Roman Alekseev]

On 30.10.2012 13:10, Gregory Farnum wrote:
> On Tue, Oct 30, 2012 at 9:27 AM, Roman Alekseev <rs.alekseev@gmail.com> wrote:
>> On 29.10.2012 22:57, Sam Lang wrote:
>>>
>>> Hi Roman,
>>>
>>> Is this with the ceph fuse client or the ceph kernel module?
>>>
>>> Its not surprising that the local file system (/home) is so much faster
>>> than a mounted ceph volume, especially the first time the directory tree is
>>> traversed (metadata results are cached at the client to improve
>>> performance).  Try running the same find command on the ceph volume and see
>>> if the cached results at the client improve performance at all.
>>>
>>> In order to understand what the performance of ceph should be capable of
>>> doing with your deployment for this specific workload, you should run iperf
>>> between two nodes to get an idea of your latency limits.
>>>
>>> Also, I noticed that the real timings you listed for ceph and /home are
>>> offset by exactly 17 minutes (user and sys are identical).  Was that a
>>> copy/paste error, by chance?
>>>
>>> -sam
>>>
>>> On 10/29/2012 09:01 AM, Roman Alekseev wrote:
>>>> Hi,
>>>>
>>>> Kindly guide me how to improve performance on the cluster which consist
>>>> of 5 dedicated servers:
>>>>
>>>> - ceph.conf: http://pastebin.com/hT3qEhUF
>>>> - file system on all drives is ext4
>>>> - mount options "user_xattr"
>>>> - each server has :
>>>> CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
>>>> MEM: 4Gb DDR2
>>>> - 1Gb network
>>>>
>>>> Simple test:
>>>>
>>>> mounted as ceph
>>>> root@client1:/mnt/mycephfs# time find . | wc -l
>>>> 83932
>>>>
>>>> real    17m55.399s
>>>> user    0m0.152s
>>>> sys    0m1.528s
>>>>
>>>> on 1 HDD:
>>>>
>>>> root@client1:/home# time find . | wc -l
>>>> 83932
>>>>
>>>> real    0m55.399s
>>>> user    0m0.152s
>>>> sys    0m1.528s
>>>>
>>>> Please help me to find out the issue. Thanks.
>>>>
>> Hi Sam,
>>
>>      I use the Ceph fs only as kernel module, because we need to get its
>> powerful performance but as I can see it is slower then distributed file
>> system based on fuse, for example, MooseFS performed the same test for 3
>> min.
>> Here is the result iperf test beetwen client and osd server:
>> root@asrv151:~# iperf -c client -i 1
>> ------------------------------------------------------------
>> Client connecting to clientIP, TCP port 5001
>> TCP window size: 96.1 KByte (default)
>> ------------------------------------------------------------
>> [  3] local osd_server port 50106 connected with clientIP port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 1.0 sec    112 MBytes    941 Mbits/sec
>> [  3]  1.0- 2.0 sec    110 MBytes    924 Mbits/sec
>> [  3]  2.0- 3.0 sec    108 MBytes    905 Mbits/sec
>> [  3]  3.0- 4.0 sec    109 MBytes    917 Mbits/sec
>> [  3]  4.0- 5.0 sec    110 MBytes    926 Mbits/sec
>> [  3]  5.0- 6.0 sec    109 MBytes    915 Mbits/sec
>> [  3]  6.0- 7.0 sec    110 MBytes    926 Mbits/sec
>> [  3]  7.0- 8.0 sec    108 MBytes    908 Mbits/sec
>> [  3]  8.0- 9.0 sec    107 MBytes    897 Mbits/sec
>> [  3]  9.0-10.0 sec    106 MBytes    886 Mbits/sec
>> [  3]  0.0-10.0 sec  1.06 GBytes    914 Mbits/sec
>>
>> ceph -w results:
>>
>>   health HEALTH_OK
>>     monmap e3: 3 mons at {a=mon.a:6789/0,b=mon.b:6789/0,c=mon.c:6789/0},
>> election epoch 10, quorum 0,1,2 a,b,c
>>     osdmap e132: 5 osds: 5 up, 5 in
>>      pgmap v11720: 384 pgs: 384 active+clean; 1880 MB data, 10679 MB used,
>> 5185 GB / 5473 GB avail
>>     mdsmap e4: 1/1/1 up {0=a=up:active}
>>
>> 2012-10-30 12:23:09.830677 osd.2 [WRN] slow request 30.135787 seconds old,
>> received at 2012-10-30 12:22:39.694780: osd_op(mds.0.1:309216
>> 10000017163.00000000 [setxattr path (69),setxattr parent (196),tmapput
>> 0~596] 1.724c80f7) v4 currently waiting for sub ops
>> 2012-10-30 12:23:10.109637 mon.0 [INF] pgmap v11720: 384 pgs: 384
>> active+clean; 1880 MB data, 10679 MB used, 5185 GB / 5473 GB avail
>> 2012-10-30 12:23:12.918038 mon.0 [INF] pgmap v11721: 384 pgs: 384
>> active+clean; 1880 MB data, 10680 MB used, 5185 GB / 5473 GB avail
>> 2012-10-30 12:23:13.977044 mon.0 [INF] pgmap v11722: 384 pgs: 384
>> active+clean; 1880 MB data, 10681 MB used, 5185 GB / 5473 GB avail
>> 2012-10-30 12:23:10.587391 osd.3 [WRN] 6 slow requests, 6 included below;
>> oldest blocked for > 30.808352 secs
>> 2012-10-30 12:23:10.587398 osd.3 [WRN] slow request 30.808352 seconds old,
>> received at 2012-10-30 12:22:39.778971: osd_op(mds.0.1:308701 200.000002e5
>> [write 976010~5402] 1.adbeb1a) v4 currently waiting for sub ops
>> 2012-10-30 12:23:10.587403 osd.3 [WRN] slow request 30.796417 seconds old,
>> received at 2012-10-30 12:22:39.790906: osd_op(mds.0.1:308702 200.000002e5
>> [write 981412~6019] 1.adbeb1a) v4 currently waiting for sub ops
>> 2012-10-30 12:23:10.587408 osd.3 [WRN] slow request 30.796347 seconds old,
>> received at 2012-10-30 12:22:39.790976: osd_op(mds.0.1:308703 200.000002e5
>> [write 987431~61892] 1.adbeb1a) v4 currently waiting for sub ops
>> 2012-10-30 12:23:10.587413 osd.3 [WRN] slow request 30.530228 seconds old,
>> received at 2012-10-30 12:22:40.057095: osd_op(mds.0.1:308704 200.000002e5
>> [write 1049323~6630] 1.adbeb1a) v4 currently waiting for sub ops
>> 2012-10-30 12:23:10.587417 osd.3 [WRN] slow request 30.530027 seconds old,
>> received at 2012-10-30 12:22:40.057296: osd_op(mds.0.1:308705 200.000002e5
>> [write 1055953~20679] 1.adbeb1a) v4 currently waiting for sub ops
>>
>>
>> At the same time I'm copy data to ceph mounted storage.
>>
>> I dunno what can I do to resolve this problem :(
>> Any advices will be greatly appreciated.
> Is it the same client copying data into cephfs or a different one?
> I see here that you have several slow requests; it looks like maybe
> you're overloading your disks. That could impact metadata lookups if
> the MDS doesn't have everything cached; have you tried running this
> test without data ingest? (Obviously we'd like it to be faster even
> so, but if it's disk contention there's not a lot we can do.)
> -Greg
Dear Greg,

Yes, this was the same client. Sorry, could you please explain me with 
more details how can I "test without data ingest"?
Also I can rebuild my cluster from scratch and make all tests again.

I have 5 dedicated servers and I think if I create ceph cluster from 
them it shouldn't be slower then the same cluster based on fuse 
technology. Am I right?

Thanks.

-- 
Kind regards,

R. Alekseev

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Gregory Farnum]

On Tue, Oct 30, 2012 at 11:04 AM, Roman Alekseev <rs.alekseev@gmail.com> wrote:
> On 30.10.2012 13:10, Gregory Farnum wrote:
>>
>> On Tue, Oct 30, 2012 at 9:27 AM, Roman Alekseev <rs.alekseev@gmail.com>
>> wrote:
>>>
>>> On 29.10.2012 22:57, Sam Lang wrote:
>>>>
>>>>
>>>> Hi Roman,
>>>>
>>>> Is this with the ceph fuse client or the ceph kernel module?
>>>>
>>>> Its not surprising that the local file system (/home) is so much faster
>>>> than a mounted ceph volume, especially the first time the directory tree
>>>> is
>>>> traversed (metadata results are cached at the client to improve
>>>> performance).  Try running the same find command on the ceph volume and
>>>> see
>>>> if the cached results at the client improve performance at all.
>>>>
>>>> In order to understand what the performance of ceph should be capable of
>>>> doing with your deployment for this specific workload, you should run
>>>> iperf
>>>> between two nodes to get an idea of your latency limits.
>>>>
>>>> Also, I noticed that the real timings you listed for ceph and /home are
>>>> offset by exactly 17 minutes (user and sys are identical).  Was that a
>>>> copy/paste error, by chance?
>>>>
>>>> -sam
>>>>
>>>> On 10/29/2012 09:01 AM, Roman Alekseev wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Kindly guide me how to improve performance on the cluster which consist
>>>>> of 5 dedicated servers:
>>>>>
>>>>> - ceph.conf: http://pastebin.com/hT3qEhUF
>>>>> - file system on all drives is ext4
>>>>> - mount options "user_xattr"
>>>>> - each server has :
>>>>> CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
>>>>> MEM: 4Gb DDR2
>>>>> - 1Gb network
>>>>>
>>>>> Simple test:
>>>>>
>>>>> mounted as ceph
>>>>> root@client1:/mnt/mycephfs# time find . | wc -l
>>>>> 83932
>>>>>
>>>>> real    17m55.399s
>>>>> user    0m0.152s
>>>>> sys    0m1.528s
>>>>>
>>>>> on 1 HDD:
>>>>>
>>>>> root@client1:/home# time find . | wc -l
>>>>> 83932
>>>>>
>>>>> real    0m55.399s
>>>>> user    0m0.152s
>>>>> sys    0m1.528s
>>>>>
>>>>> Please help me to find out the issue. Thanks.
>>>>>
>>> Hi Sam,
>>>
>>>      I use the Ceph fs only as kernel module, because we need to get its
>>> powerful performance but as I can see it is slower then distributed file
>>> system based on fuse, for example, MooseFS performed the same test for 3
>>> min.
>>> Here is the result iperf test beetwen client and osd server:
>>> root@asrv151:~# iperf -c client -i 1
>>> ------------------------------------------------------------
>>> Client connecting to clientIP, TCP port 5001
>>> TCP window size: 96.1 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local osd_server port 50106 connected with clientIP port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec    112 MBytes    941 Mbits/sec
>>> [  3]  1.0- 2.0 sec    110 MBytes    924 Mbits/sec
>>> [  3]  2.0- 3.0 sec    108 MBytes    905 Mbits/sec
>>> [  3]  3.0- 4.0 sec    109 MBytes    917 Mbits/sec
>>> [  3]  4.0- 5.0 sec    110 MBytes    926 Mbits/sec
>>> [  3]  5.0- 6.0 sec    109 MBytes    915 Mbits/sec
>>> [  3]  6.0- 7.0 sec    110 MBytes    926 Mbits/sec
>>> [  3]  7.0- 8.0 sec    108 MBytes    908 Mbits/sec
>>> [  3]  8.0- 9.0 sec    107 MBytes    897 Mbits/sec
>>> [  3]  9.0-10.0 sec    106 MBytes    886 Mbits/sec
>>> [  3]  0.0-10.0 sec  1.06 GBytes    914 Mbits/sec
>>>
>>> ceph -w results:
>>>
>>>   health HEALTH_OK
>>>     monmap e3: 3 mons at {a=mon.a:6789/0,b=mon.b:6789/0,c=mon.c:6789/0},
>>> election epoch 10, quorum 0,1,2 a,b,c
>>>     osdmap e132: 5 osds: 5 up, 5 in
>>>      pgmap v11720: 384 pgs: 384 active+clean; 1880 MB data, 10679 MB
>>> used,
>>> 5185 GB / 5473 GB avail
>>>     mdsmap e4: 1/1/1 up {0=a=up:active}
>>>
>>> 2012-10-30 12:23:09.830677 osd.2 [WRN] slow request 30.135787 seconds
>>> old,
>>> received at 2012-10-30 12:22:39.694780: osd_op(mds.0.1:309216
>>> 10000017163.00000000 [setxattr path (69),setxattr parent (196),tmapput
>>> 0~596] 1.724c80f7) v4 currently waiting for sub ops
>>> 2012-10-30 12:23:10.109637 mon.0 [INF] pgmap v11720: 384 pgs: 384
>>> active+clean; 1880 MB data, 10679 MB used, 5185 GB / 5473 GB avail
>>> 2012-10-30 12:23:12.918038 mon.0 [INF] pgmap v11721: 384 pgs: 384
>>> active+clean; 1880 MB data, 10680 MB used, 5185 GB / 5473 GB avail
>>> 2012-10-30 12:23:13.977044 mon.0 [INF] pgmap v11722: 384 pgs: 384
>>> active+clean; 1880 MB data, 10681 MB used, 5185 GB / 5473 GB avail
>>> 2012-10-30 12:23:10.587391 osd.3 [WRN] 6 slow requests, 6 included below;
>>> oldest blocked for > 30.808352 secs
>>> 2012-10-30 12:23:10.587398 osd.3 [WRN] slow request 30.808352 seconds
>>> old,
>>> received at 2012-10-30 12:22:39.778971: osd_op(mds.0.1:308701
>>> 200.000002e5
>>> [write 976010~5402] 1.adbeb1a) v4 currently waiting for sub ops
>>> 2012-10-30 12:23:10.587403 osd.3 [WRN] slow request 30.796417 seconds
>>> old,
>>> received at 2012-10-30 12:22:39.790906: osd_op(mds.0.1:308702
>>> 200.000002e5
>>> [write 981412~6019] 1.adbeb1a) v4 currently waiting for sub ops
>>> 2012-10-30 12:23:10.587408 osd.3 [WRN] slow request 30.796347 seconds
>>> old,
>>> received at 2012-10-30 12:22:39.790976: osd_op(mds.0.1:308703
>>> 200.000002e5
>>> [write 987431~61892] 1.adbeb1a) v4 currently waiting for sub ops
>>> 2012-10-30 12:23:10.587413 osd.3 [WRN] slow request 30.530228 seconds
>>> old,
>>> received at 2012-10-30 12:22:40.057095: osd_op(mds.0.1:308704
>>> 200.000002e5
>>> [write 1049323~6630] 1.adbeb1a) v4 currently waiting for sub ops
>>> 2012-10-30 12:23:10.587417 osd.3 [WRN] slow request 30.530027 seconds
>>> old,
>>> received at 2012-10-30 12:22:40.057296: osd_op(mds.0.1:308705
>>> 200.000002e5
>>> [write 1055953~20679] 1.adbeb1a) v4 currently waiting for sub ops
>>>
>>>
>>> At the same time I'm copy data to ceph mounted storage.
>>>
>>> I dunno what can I do to resolve this problem :(
>>> Any advices will be greatly appreciated.
>>
>> Is it the same client copying data into cephfs or a different one?
>> I see here that you have several slow requests; it looks like maybe
>> you're overloading your disks. That could impact metadata lookups if
>> the MDS doesn't have everything cached; have you tried running this
>> test without data ingest? (Obviously we'd like it to be faster even
>> so, but if it's disk contention there's not a lot we can do.)
>> -Greg
>
> Dear Greg,
>
> Yes, this was the same client. Sorry, could you please explain me with more
> details how can I "test without data ingest"?

You said you were copying data into the cluster while doing this test.
It really shouldn't matter, but if you had enough activity going
on...it might. :/

> Also I can rebuild my cluster from scratch and make all tests again.
>
> I have 5 dedicated servers and I think if I create ceph cluster from them it
> shouldn't be slower then the same cluster based on fuse technology. Am I
> right?
Well, the cluster isn't really based on FUSE technology, that's just
the client. You're certainly seeing slower performance than I expect,
but Ceph is doing two copies of every write and I think MooseFS
defaults to one, for instance.
I'm concerned in particular about those "slow write" warnings, which
tend to indicate your data disks are overloaded. Can you share your
ceph.conf and your hardware configuration?
When you're done copying data into the cluster, can you also run
"rados -p data bench 60 write" and report the results back?
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Maciej Gałkiewicz]

> ServerA(mon+osd):
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             9.2G  2.4G  6.4G  27% /
> tmpfs                 5.9G     0  5.9G   0% /lib/init/rw
> udev                  5.9G  148K  5.9G   1% /dev
> tmpfs                 5.9G     0  5.9G   0% /dev/shm
> /dev/sda7             674G  483M  629G   1% /var/lib/ceph/mon
> /dev/sda6             917G  2.4G  868G   1% /var/lib/ceph/osd/ceph-4

I see that you have separate partitions for osd and mon. Some of them
are on the same disk as OS. I would strongly recommend having separate
disk (not just partition) for each osd.

There is another problem probably more important. By default journal
is stored in /var/lib/ceph/osd/$cluster-$id/journal which means you
have data and journal on the same disk. It is what slows you down the
most. I was amazed how faster the cluster can be just by putting the
journal somewhere else (tmpfs - only for tests, another disk). From my
experiance, there is no need for separate disk for mon. It can go with
OS.

Run rados bench everytime you make an improvement in your cluster.

-- 
Regards
Maciej Galkiewicz

Re: Ceph performance

[Roman Alekseev]

On 30.10.2012 14:47, Maciej Gałkiewicz wrote:
>> ServerA(mon+osd):
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sda1             9.2G  2.4G  6.4G  27% /
>> tmpfs                 5.9G     0  5.9G   0% /lib/init/rw
>> udev                  5.9G  148K  5.9G   1% /dev
>> tmpfs                 5.9G     0  5.9G   0% /dev/shm
>> /dev/sda7             674G  483M  629G   1% /var/lib/ceph/mon
>> /dev/sda6             917G  2.4G  868G   1% /var/lib/ceph/osd/ceph-4
> I see that you have separate partitions for osd and mon. Some of them
> are on the same disk as OS. I would strongly recommend having separate
> disk (not just partition) for each osd.
>
> There is another problem probably more important. By default journal
> is stored in /var/lib/ceph/osd/$cluster-$id/journal which means you
> have data and journal on the same disk. It is what slows you down the
> most. I was amazed how faster the cluster can be just by putting the
> journal somewhere else (tmpfs - only for tests, another disk). From my
> experiance, there is no need for separate disk for mon. It can go with
> OS.
>
> Run rados bench everytime you make an improvement in your cluster.
>
Oops, it's actually true!!!

Give me a moment to check this out and thank you for opening my eyes.
Let me to notify you about the results of implementing your recommendations.

I'm going to completely rebuild my cluster. Thank so much.

-- 
Kind regards,

R. Alekseev

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph performance

[Maciej Gałkiewicz]

> Give me a moment to check this out and thank you for opening my eyes.
> Let me to notify you about the results of implementing your recommendations.

I am looking forward to hearing from you. One more thing. Run rados
bench right now to see from what performance level are you starting. I
suggest 2 tests for default block size and lets say 4KB.

Here is very useful paper which helped me a lot:
http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/

-- 
Regards
Maciej Galkiewicz