* Re: ceph on ubuntu and centos
@ 2013-10-07 22:16 hjwsm1989
2013-10-07 23:54 ` Samuel Just
0 siblings, 1 reply; 3+ messages in thread
From: hjwsm1989 @ 2013-10-07 22:16 UTC (permalink / raw)
To: Samuel Just; +Cc: huangyellowhuang, ceph-devel
thanks for your reply !
the ubuntu is 3.8,centos kernel version is 2.6.32.
which setting item should we change to get the smoth write speed ?
we tried tune some parameters:
osd op threads= 8
filestore op threads = 8
filestore max op queue = 30
which one will have the largest effect on performance?
thanks
Samuel Just <sam.just@inktank.com>编写：
>Interesting! What kernel versions were running on the 13.10 and
>centos 6.4 clusters?
>-Sam
>
>On Fri, Oct 4, 2013 at 6:33 PM, huangyellowhuang
><huangyellowhuang@163.com> wrote:
>> Hi,all
>> We test the ceph version 0.69 (6ca6f2f9f754031f4acdb971b71c92c9762e18c3) on
>> Ubuntu server 13.10 and centos 6.4 final
>> Our cluster configuration:
>> 3 host machine, each runs 3 OSDs(use XFS as backend fs),MON and MDS runs on
>> one of the three host,
>> We have one KClient on Ubuntu server 13.10
>>
>> The cluster runs on Ubuntu works fine and a few ‘slow requests’ msgs, about
>> 100MB/s write speed.
>> But the cluster runs on centos is very bad, avg 30MB/s write speed, many osd
>> requests slow:
>> 2013-10-05 08:35:09.931145 mon.0 [INF] pgmap v928: 192 pgs: 192 active+clean
>> ; 50873 MB data, 101716 MB used, 13857 GB / 13956 GB avail; 115 MB/s wr, 28
>> op/s
>> 2013-10-05 08:35:12.087614 mon.0 [INF] pgmap v929: 192 pgs: 192 active+clean
>> ; 50901 MB data, 101780 MB used, 13857 GB / 13956 GB avail; 32593 KB/s wr, 8
>> op/s
>> 2013-10-05 08:35:03.963979 osd.0 [WRN] 37 slow requests, 1 included below; o
>> ldest blocked for > 798.235962 secs
>> 2013-10-05 08:35:03.963984 osd.0 [WRN] slow request 240.831078 seconds old,
>> received at 2013-10-05 08:31:03.132836: osd_op(mds.0.1:375 200.00000000 [wri
>> tefull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>> 2013-10-05 08:35:08.965134 osd.0 [WRN] 37 slow requests, 1 included below; o
>> ldest blocked for > 803.237127 secs
>> 2013-10-05 08:35:08.965139 osd.0 [WRN] slow request 480.312618 seconds old,
>> received at 2013-10-05 08:27:08.652461: osd_op(mds.0.1:307 200.00000000 [wri
>> tefull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>> 2013-10-05 08:35:10.965619 osd.0 [WRN] 37 slow requests, 1 included below; o
>> ldest blocked for > 805.237600 secs
>> 2013-10-05 08:35:10.965624 osd.0 [WRN] slow request 120.946652 seconds old,
>> received at 2013-10-05 08:33:10.018900: osd_op(mds.0.1:404 200.00000000 [wri
>> tefull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>> 2013-10-05 08:35:11.965986 osd.0 [WRN] 37 slow requests, 1 included below; o
>> ldest blocked for > 806.237800 secs
>> 2013-10-05 08:35:11.965992 osd.0 [WRN] slow request 60.474314 seconds old, r
>> eceived at 2013-10-05 08:34:11.491438: osd_op(mds.0.1:430 200.00000000 [writ
>> efull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>>
>> And we need to build a cluster have 4 hosts, each has 18 OSDs and 1 kclient,
>> every kclient server as samba server that serves 4 samba clients.
>> 1) Which linux distrubtion should we used? Centos or Ubuntu?
>> 2) What results in ceph performances so large on different distribution?
>> 3) It seems the bottleneck is underlayer fs can not handle requests fast as
>> ceph expect, bc the ‘slow requests’ shows if a request does not handle after
>> 30s.
>>
>> Thanks!
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>--
>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: ceph on ubuntu and centos
2013-10-07 22:16 ceph on ubuntu and centos hjwsm1989
@ 2013-10-07 23:54 ` Samuel Just
2013-10-08 2:37 ` 答复: " hjwsm1989-gmail
0 siblings, 1 reply; 3+ messages in thread
From: Samuel Just @ 2013-10-07 23:54 UTC (permalink / raw)
To: hjwsm1989@gmail.com; +Cc: huangyellowhuang, ceph-devel
You might try
osd client message size cap = 26214400
osd client message cap = 25.
osd op threads = 8 and filestore op threads = 8 might also be good.
Let us know what you find!
Sounds like the the kernel is the most obvious candidate for slowness
on Centos 6.4, is there a 3.0+ kernel around for Centos 6.4 you could
try?
-Sam
On Mon, Oct 7, 2013 at 3:16 PM, hjwsm1989@gmail.com <hjwsm1989@gmail.com> wrote:
> thanks for your reply !
> the ubuntu is 3.8,centos kernel version is 2.6.32.
> which setting item should we change to get the smoth write speed ?
> we tried tune some parameters:
> osd op threads= 8
> filestore op threads = 8
> filestore max op queue = 30
> which one will have the largest effect on performance?
>
> thanks
>
> Samuel Just <sam.just@inktank.com>编写：
>
>>Interesting! What kernel versions were running on the 13.10 and
>>centos 6.4 clusters?
>>-Sam
>>
>>On Fri, Oct 4, 2013 at 6:33 PM, huangyellowhuang
>><huangyellowhuang@163.com> wrote:
>>> Hi,all
>>> We test the ceph version 0.69 (6ca6f2f9f754031f4acdb971b71c92c9762e18c3) on
>>> Ubuntu server 13.10 and centos 6.4 final
>>> Our cluster configuration:
>>> 3 host machine, each runs 3 OSDs(use XFS as backend fs),MON and MDS runs on
>>> one of the three host,
>>> We have one KClient on Ubuntu server 13.10
>>>
>>> The cluster runs on Ubuntu works fine and a few ‘slow requests’ msgs, about
>>> 100MB/s write speed.
>>> But the cluster runs on centos is very bad, avg 30MB/s write speed, many osd
>>> requests slow:
>>> 2013-10-05 08:35:09.931145 mon.0 [INF] pgmap v928: 192 pgs: 192 active+clean
>>> ; 50873 MB data, 101716 MB used, 13857 GB / 13956 GB avail; 115 MB/s wr, 28
>>> op/s
>>> 2013-10-05 08:35:12.087614 mon.0 [INF] pgmap v929: 192 pgs: 192 active+clean
>>> ; 50901 MB data, 101780 MB used, 13857 GB / 13956 GB avail; 32593 KB/s wr, 8
>>> op/s
>>> 2013-10-05 08:35:03.963979 osd.0 [WRN] 37 slow requests, 1 included below; o
>>> ldest blocked for > 798.235962 secs
>>> 2013-10-05 08:35:03.963984 osd.0 [WRN] slow request 240.831078 seconds old,
>>> received at 2013-10-05 08:31:03.132836: osd_op(mds.0.1:375 200.00000000 [wri
>>> tefull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>>> 2013-10-05 08:35:08.965134 osd.0 [WRN] 37 slow requests, 1 included below; o
>>> ldest blocked for > 803.237127 secs
>>> 2013-10-05 08:35:08.965139 osd.0 [WRN] slow request 480.312618 seconds old,
>>> received at 2013-10-05 08:27:08.652461: osd_op(mds.0.1:307 200.00000000 [wri
>>> tefull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>>> 2013-10-05 08:35:10.965619 osd.0 [WRN] 37 slow requests, 1 included below; o
>>> ldest blocked for > 805.237600 secs
>>> 2013-10-05 08:35:10.965624 osd.0 [WRN] slow request 120.946652 seconds old,
>>> received at 2013-10-05 08:33:10.018900: osd_op(mds.0.1:404 200.00000000 [wri
>>> tefull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>>> 2013-10-05 08:35:11.965986 osd.0 [WRN] 37 slow requests, 1 included below; o
>>> ldest blocked for > 806.237800 secs
>>> 2013-10-05 08:35:11.965992 osd.0 [WRN] slow request 60.474314 seconds old, r
>>> eceived at 2013-10-05 08:34:11.491438: osd_op(mds.0.1:430 200.00000000 [writ
>>> efull 0~84] 1.844f3494 e47) v4 currently no flag points reached
>>>
>>> And we need to build a cluster have 4 hosts, each has 18 OSDs and 1 kclient,
>>> every kclient server as samba server that serves 4 samba clients.
>>> 1) Which linux distrubtion should we used? Centos or Ubuntu?
>>> 2) What results in ceph performances so large on different distribution?
>>> 3) It seems the bottleneck is underlayer fs can not handle requests fast as
>>> ceph expect, bc the ‘slow requests’ shows if a request does not handle after
>>> 30s.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>--
>>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* 答复: ceph on ubuntu and centos
2013-10-07 23:54 ` Samuel Just
@ 2013-10-08 2:37 ` hjwsm1989-gmail
0 siblings, 0 replies; 3+ messages in thread
From: hjwsm1989-gmail @ 2013-10-08 2:37 UTC (permalink / raw)
To: 'Samuel Just'; +Cc: 'huangyellowhuang', 'ceph-devel'
Thanks
We tried the settings you recommened.
For my cluster on Ubuntu 13.10, I have 4 OSDs on one host, I set :
osd client message size cap = 52428800
osd client message cap = 50
From client side, the write speed between 50~100MB/s, avg is 79MB/s.
But on the centos 6.4 cluster, I upgrade the kernel to 3.8, it seems no performance improve:
I have 2 hosts and each runs 3 OSDs(total 6 OSDs)
The write speed is smooth, but speed is less than that run on Ubuntu.
I think the replica write may result to this difference.
-----邮件原件-----
发件人: Samuel Just [mailto:sam.just@inktank.com]
发送时间: 2013年10月8日 7:55
收件人: hjwsm1989@gmail.com
抄送: huangyellowhuang; ceph-devel
主题: Re: ceph on ubuntu and centos
You might try
osd client message size cap = 26214400
osd client message cap = 25.
osd op threads = 8 and filestore op threads = 8 might also be good.
Let us know what you find!
Sounds like the the kernel is the most obvious candidate for slowness on Centos 6.4, is there a 3.0+ kernel around for Centos 6.4 you could try?
-Sam
On Mon, Oct 7, 2013 at 3:16 PM, hjwsm1989@gmail.com <hjwsm1989@gmail.com> wrote:
> thanks for your reply !
> the ubuntu is 3.8,centos kernel version is 2.6.32.
> which setting item should we change to get the smoth write speed ?
> we tried tune some parameters:
> osd op threads= 8
> filestore op threads = 8
> filestore max op queue = 30
> which one will have the largest effect on performance?
>
> thanks
>
> Samuel Just <sam.just@inktank.com>编写：
>
>>Interesting! What kernel versions were running on the 13.10 and
>>centos 6.4 clusters?
>>-Sam
>>
>>On Fri, Oct 4, 2013 at 6:33 PM, huangyellowhuang
>><huangyellowhuang@163.com> wrote:
>>> Hi,all
>>> We test the ceph version 0.69
>>> (6ca6f2f9f754031f4acdb971b71c92c9762e18c3) on Ubuntu server 13.10
>>> and centos 6.4 final Our cluster configuration:
>>> 3 host machine, each runs 3 OSDs(use XFS as backend fs),MON and MDS
>>> runs on one of the three host, We have one KClient on Ubuntu server
>>> 13.10
>>>
>>> The cluster runs on Ubuntu works fine and a few ‘slow requests’
>>> msgs, about 100MB/s write speed.
>>> But the cluster runs on centos is very bad, avg 30MB/s write speed,
>>> many osd requests slow:
>>> 2013-10-05 08:35:09.931145 mon.0 [INF] pgmap v928: 192 pgs: 192
>>> active+clean ; 50873 MB data, 101716 MB used, 13857 GB / 13956 GB
>>> avail; 115 MB/s wr, 28 op/s
>>> 2013-10-05 08:35:12.087614 mon.0 [INF] pgmap v929: 192 pgs: 192
>>> active+clean ; 50901 MB data, 101780 MB used, 13857 GB / 13956 GB
>>> avail; 32593 KB/s wr, 8 op/s
>>> 2013-10-05 08:35:03.963979 osd.0 [WRN] 37 slow requests, 1 included
>>> below; o ldest blocked for > 798.235962 secs
>>> 2013-10-05 08:35:03.963984 osd.0 [WRN] slow request 240.831078
>>> seconds old, received at 2013-10-05 08:31:03.132836:
>>> osd_op(mds.0.1:375 200.00000000 [wri tefull 0~84] 1.844f3494 e47) v4
>>> currently no flag points reached
>>> 2013-10-05 08:35:08.965134 osd.0 [WRN] 37 slow requests, 1 included
>>> below; o ldest blocked for > 803.237127 secs
>>> 2013-10-05 08:35:08.965139 osd.0 [WRN] slow request 480.312618
>>> seconds old, received at 2013-10-05 08:27:08.652461:
>>> osd_op(mds.0.1:307 200.00000000 [wri tefull 0~84] 1.844f3494 e47) v4
>>> currently no flag points reached
>>> 2013-10-05 08:35:10.965619 osd.0 [WRN] 37 slow requests, 1 included
>>> below; o ldest blocked for > 805.237600 secs
>>> 2013-10-05 08:35:10.965624 osd.0 [WRN] slow request 120.946652
>>> seconds old, received at 2013-10-05 08:33:10.018900:
>>> osd_op(mds.0.1:404 200.00000000 [wri tefull 0~84] 1.844f3494 e47) v4
>>> currently no flag points reached
>>> 2013-10-05 08:35:11.965986 osd.0 [WRN] 37 slow requests, 1 included
>>> below; o ldest blocked for > 806.237800 secs
>>> 2013-10-05 08:35:11.965992 osd.0 [WRN] slow request 60.474314
>>> seconds old, r eceived at 2013-10-05 08:34:11.491438:
>>> osd_op(mds.0.1:430 200.00000000 [writ efull 0~84] 1.844f3494 e47) v4
>>> currently no flag points reached
>>>
>>> And we need to build a cluster have 4 hosts, each has 18 OSDs and 1
>>> kclient, every kclient server as samba server that serves 4 samba clients.
>>> 1) Which linux distrubtion should we used? Centos or Ubuntu?
>>> 2) What results in ceph performances so large on different distribution?
>>> 3) It seems the bottleneck is underlayer fs can not handle requests
>>> fast as ceph expect, bc the ‘slow requests’ shows if a request does
>>> not handle after 30s.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>--
>>To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>in the body of a message to majordomo@vger.kernel.org More majordomo
>>info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2013-10-08 2:37 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-07 22:16 ceph on ubuntu and centos hjwsm1989
2013-10-07 23:54 ` Samuel Just
2013-10-08 2:37 ` 答复: " hjwsm1989-gmail
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.