Any concern about Ceph on CentOS

Any concern about Ceph on CentOS

[Chen, Xiaoxi]

Hi list,
   I would like to ask if anyone really run Ceph on CentOS/RHEL? Since the kernel version for Cent/RHEL is much older than that of Ubuntu, I am thinking about whether we have some known performance/functionality issue?
   Thanks for everyone could share your insight for Ceph+CentOS.
																												Xiaoxi 

Re: Any concern about Ceph on CentOS

[Haomai Wang]

[-- Attachment #1.1: Type: text/plain, Size: 851 bytes --]

We have deployed Ceph on CentOS 6.3 and will have a full test on it. The old
kernel version and rbd.ko is problem but we will not touch rbd.ko.

Best regards,
Wheats



在 2013-7-17，上午9:18，"Chen, Xiaoxi" <xiaoxi.chen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> 写道：

> Hi list,
>   I would like to ask if anyone really run Ceph on CentOS/RHEL? Since the kernel version for Cent/RHEL is much older than that of Ubuntu, I am thinking about whether we have some known performance/functionality issue?
>   Thanks for everyone could share your insight for Ceph+CentOS.
> 																												Xiaoxi 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #1.2: Type: text/html, Size: 3731 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Any concern about Ceph on CentOS

[Kasper Dieter]

Hi Xiaoxi,

we are really running Ceph on CentOS-6.4
(6 server nodes, 3 client nodes, 160 OSDs).

We put a 3.8.13 Kernel on top and installed the ceph-0.61.4 cluster with mkcephfs, 
because ceph-deploy seems to be still very buggy and has big dependencies to the newest python.

ceph.ko
rbd.ko
and ceph-fuse are running fast, well and stable (so far ;-) ).

Cheers,
-Dieter

On Wed, Jul 17, 2013 at 03:18:22AM +0200, Chen, Xiaoxi wrote:
> Hi list,
>    I would like to ask if anyone really run Ceph on CentOS/RHEL? 
>    Since the kernel version for Cent/RHEL is much older than that of Ubuntu, 
>    I am thinking about whether we have some known performance/functionality issue?
>    Thanks for everyone could share your insight for Ceph+CentOS.
> 																												Xiaoxi 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Any concern about Ceph on CentOS

[Haomai Wang]

Hi Kasper,

Can you talk about how you make use of Ceph and detail information on CentOS?

I guess that you use the CephFS on Ceph Cluster?

Best regards,
Wheats



在 2013-7-17，下午2:16，Kasper Dieter <dieter.kasper@ts.fujitsu.com> 写道：

> Hi Xiaoxi,
> 
> we are really running Ceph on CentOS-6.4
> (6 server nodes, 3 client nodes, 160 OSDs).
> 
> We put a 3.8.13 Kernel on top and installed the ceph-0.61.4 cluster with mkcephfs, 
> because ceph-deploy seems to be still very buggy and has big dependencies to the newest python.
> 
> ceph.ko
> rbd.ko
> and ceph-fuse are running fast, well and stable (so far ;-) ).
> 
> Cheers,
> -Dieter
> 
> On Wed, Jul 17, 2013 at 03:18:22AM +0200, Chen, Xiaoxi wrote:
>> Hi list,
>>   I would like to ask if anyone really run Ceph on CentOS/RHEL? 
>>   Since the kernel version for Cent/RHEL is much older than that of Ubuntu, 
>>   I am thinking about whether we have some known performance/functionality issue?
>>   Thanks for everyone could share your insight for Ceph+CentOS.
>> 																												Xiaoxi 
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Any concern about Ceph on CentOS

[Chen, Xiaoxi]

Hi Dieter,
    Thanks a lot for the information. Could I learn more about your use case ? Would you care performance ?
    And may I know why you want CentOS+ Custom kernel ? I thought people use CentOS for stability concern but if using CentOS + Custom Kernel, why don't you just use Ubuntu or Fedora(although Fedora is usually too aggressive).
                                                                      Xiaoxi

-----Original Message-----
From: ceph-devel-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:ceph-devel-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Kasper Dieter
Sent: Wednesday, July 17, 2013 2:17 PM
To: Chen, Xiaoxi
Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; ceph-users-Qp0mS5GaXlQ@public.gmane.org
Subject: Re: Any concern about Ceph on CentOS

Hi Xiaoxi,

we are really running Ceph on CentOS-6.4
(6 server nodes, 3 client nodes, 160 OSDs).

We put a 3.8.13 Kernel on top and installed the ceph-0.61.4 cluster with mkcephfs, because ceph-deploy seems to be still very buggy and has big dependencies to the newest python.

ceph.ko
rbd.ko
and ceph-fuse are running fast, well and stable (so far ;-) ).

Cheers,
-Dieter

On Wed, Jul 17, 2013 at 03:18:22AM +0200, Chen, Xiaoxi wrote:
> Hi list,
>    I would like to ask if anyone really run Ceph on CentOS/RHEL? 
>    Since the kernel version for Cent/RHEL is much older than that of Ubuntu, 
>    I am thinking about whether we have some known performance/functionality issue?
>    Thanks for everyone could share your insight for Ceph+CentOS.
> 																												Xiaoxi
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Any concern about Ceph on CentOS

[Kasper Dieter]

Hi Xiaoxi, Wheats,

you hit the right point: We are looking for an Enterprise Linux distribution as base for Ceph.
RHEL and CentOS has a very broad distribution and a high acceptance in the Data Center due to our observation.

The pain of this distro (from Ceph point of view) is the old kernel.
As you know a 'new' kernel ( >= 3.6) is mandatory for stability and features of the Kernel CephFS and RBD (and btrfs).

So, we decided to use a very stable distro (CentOS-6.4) and combine it with a new, but mature/stable kernel (3.8.13).

I'm running fedora on my Laptop:
Linux oder 3.9.9-301.fc19.x86_64 #1 SMP Thu Jul 4 15:10:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
but as you said it is "too aggressive".
Another aspect might be our good knowledge on RHEL and small experience with Ubuntu.

It is our goal to run all three presentation layers Block/File/Object in one Ceph cluster.
We have good experience with block and file and also the performance looks quit well:

[root@rx37-4 ~]# ceph osd dump | grep SAS-group-2
pool 5 'SAS-group-2' rep size 2 min_size 1 crush_ruleset 4 object_hash rjenkins pg_num 3300 pgp_num 3300 last_change 306 owner 0

[root@rx37-2 cephfs]# dd if=/dev/zero of=/mnt/cephfs/sas-pool-5/file1 bs=4194304 count=10240
10240+0 records in
10240+0 records out
42949672960 bytes (43 GB) copied, 23.8877 s, 1.8 GB/s

[root@rx37-2 ~]# df -k | grep ceph
10.10.38.13:/        40253585408 2546787328 37706798080   7% /mnt/cephfs

Cheers,
-Dieter


On Wed, Jul 17, 2013 at 08:27:52AM +0200, Chen, Xiaoxi wrote:
> Hi Dieter,
>     Thanks a lot for the information. Could I learn more about your use case ? Would you care performance ?
>     And may I know why you want CentOS+ Custom kernel ? 
>     I thought people use CentOS for stability concern but if using CentOS + Custom Kernel, 
>     why don't you just use Ubuntu or Fedora(although Fedora is usually too aggressive).
>                                                                       Xiaoxi
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Kasper Dieter
> Sent: Wednesday, July 17, 2013 2:17 PM
> To: Chen, Xiaoxi
> Cc: ceph-devel@vger.kernel.org; ceph-users@ceph.com
> Subject: Re: Any concern about Ceph on CentOS
> 
> Hi Xiaoxi,
> 
> we are really running Ceph on CentOS-6.4
> (6 server nodes, 3 client nodes, 160 OSDs).
> 
> We put a 3.8.13 Kernel on top and installed the ceph-0.61.4 cluster with mkcephfs, because ceph-deploy seems to be still very buggy and has big dependencies to the newest python.
> 
> ceph.ko
> rbd.ko
> and ceph-fuse are running fast, well and stable (so far ;-) ).
> 
> Cheers,
> -Dieter
> 
> On Wed, Jul 17, 2013 at 03:18:22AM +0200, Chen, Xiaoxi wrote:
> > Hi list,
> >    I would like to ask if anyone really run Ceph on CentOS/RHEL? 
> >    Since the kernel version for Cent/RHEL is much older than that of Ubuntu, 
> >    I am thinking about whether we have some known performance/functionality issue?
> >    Thanks for everyone could share your insight for Ceph+CentOS.
> > 																												Xiaoxi
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Any concern about Ceph on CentOS

[Liu Yuan]

On Wed, Jul 17, 2013 at 08:59:43AM +0200, Kasper Dieter wrote:
> Hi Xiaoxi, Wheats,
> 
> you hit the right point: We are looking for an Enterprise Linux distribution as base for Ceph.
> RHEL and CentOS has a very broad distribution and a high acceptance in the Data Center due to our observation.
> 
> The pain of this distro (from Ceph point of view) is the old kernel.
> As you know a 'new' kernel ( >= 3.6) is mandatory for stability and features of the Kernel CephFS and RBD (and btrfs).
> 
> So, we decided to use a very stable distro (CentOS-6.4) and combine it with a new, but mature/stable kernel (3.8.13).
> 
> I'm running fedora on my Laptop:
> Linux oder 3.9.9-301.fc19.x86_64 #1 SMP Thu Jul 4 15:10:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> but as you said it is "too aggressive".
> Another aspect might be our good knowledge on RHEL and small experience with Ubuntu.
> 
> It is our goal to run all three presentation layers Block/File/Object in one Ceph cluster.
> We have good experience with block and file and also the performance looks quit well:
> 
> [root@rx37-4 ~]# ceph osd dump | grep SAS-group-2
> pool 5 'SAS-group-2' rep size 2 min_size 1 crush_ruleset 4 object_hash rjenkins pg_num 3300 pgp_num 3300 last_change 306 owner 0
> 
> [root@rx37-2 cephfs]# dd if=/dev/zero of=/mnt/cephfs/sas-pool-5/file1 bs=4194304 count=10240
> 10240+0 records in
> 10240+0 records out
> 42949672960 bytes (43 GB) copied, 23.8877 s, 1.8 GB/s
> 

You were just testing memory performance, no? You can try append
'oflag=direct,sync' or 'oflag=direct' to your dd. This will give you real speed.

Thanks
Yuan