Minimum Recommended Hardware

Minimum Recommended Hardware

[Dyweni - Ceph-Devel]

 Hi!

 I've been doing some casual testing of Ceph (GIT Master) and am seeing 
 some really slow speeds:
  - rados bench:  5-6MB/s
  - QEMU-KVM:  1-2MB/s

 As part of my performance troubleshooting plan, I'd like to find out 
 what the list considers the "minimum recommended" hardware configuration 
 to be in order to run Ceph and achieve a reasonably good transfer rate 
 (I'm thinking 50MB/s+?).

 My current setup is this:

 All machines PXE boot and run their OS from a customized initramfs 
 image.  Hard drives are used solely for data storage.  All machines run 
 the 2.6.39-rc7-git13 Linux Kernel.  OSD Journal lives in the OSD Data 
 partition and is 1000 MB.  I use the default (2) number of replicas for 
 all pools.  No iptables or Ceph authentication being used.  All machines 
 are connected by an 8 port Linksys Gigabit switch.

 I use a set of custom shell scripts to build and start the Ceph cluster 
 from bare metal (once the machines are booted from PXE).

 I've tried both EXT4 and BTRFS file systems on the OSDs, but both give 
 the same speeds.

 1 x MON
   Dual P3 930Mhz, 256K Cache
   512MB RAM
   No HD
   100Mb NIC

 1 x MDS
   Dual Athlon 64 X2 3800+, 512K Cache
   4GB RAM
   No HD
   100Mb NIC

 4 x OSD
   OSD1 =
     P3 1Ghz, 256K Cache
     2GB RAM
     80GB IDE (dd_rescue can read about 45MB/s sustained)
     1Gb NIC (e1000 driver)
   OSD2 =
     P3 550Mhz, 512K Cache
     768MB RAM
     40GB IDE (dd_rescue can read about 30MB/s sustained)
     1Gb NIC (e1000 driver)
   OSD3 =
     Celeron 1Ghz, 128K Cache
     512MB RAM
     18GB IDE (dd_rescue can read about 15MB/s sustained)
     1Gb NIC (e1000 driver)
   OSD4 =
     P3 1Ghz, 256K Cache
     512MB RAM
     20GB IDE (dd_rescue can read about 40MB/s sustained)
     1Gb NIC (e1000 driver)


 Thanks,
 Dyweni

Re: Minimum Recommended Hardware

[Jeff Wu]

Hi ,

My osd hosts configuration :

cpu : 4 processor, Intel(R) Core(TM)2 Quad CPU  Q8400  @ 2.66GHz
mem : 4GB
NIC : 1GB

disk:  
SATA disk WDC WD3200AAJS-0  320GB ---> save osd data
SATA SSD  OCZ-VERTEX2  60GB  ----->  save osd journal 

filesystem : btrfs 

rados writing performance  is about 50~60M/sec
I also tried to use ramdisk to save osd journal ,
rados writing performance reach to 75~85M/sec

Regards,

Jeff.

test logs:

1. " rados bench 60 write -p data" 

Total time run:        61.034223
Total writes made:     901
Write size:            4194304
Bandwidth (MB/sec):    59.049 

Average Latency:       1.0838
Max latency:           7.27347
Min latency:           0.177559


Total time run:        61.275064
Total writes made:     880
Write size:            4194304
Bandwidth (MB/sec):    57.446 

Average Latency:       1.11073
Max latency:           10.9278
Min latency:           0.170368


Total time run:        60.733064
Total writes made:     915
Write size:            4194304
Bandwidth (MB/sec):    60.264 

Average Latency:       1.0616
Max latency:           6.86648
Min latency:           0.186103


2." ceph osd tell osd-N bench" 

2011-05-18 10:11:18.979422   log 2011-05-18 19:06:23.149264 osd0
172.16.35.75:6800/3408 143 : [INF] bench: wrote 1024 MB in blocks of
4096 KB in 21.099019 sec at 49697 KB/sec
2011-05-18 10:12:18.468762   log 2011-05-18 19:07:23.482783 osd1
172.16.35.75:6803/3791 150 : [INF] bench: wrote 1024 MB in blocks of
4096 KB in 15.492175 sec at 67684 KB/sec
2011-05-18 10:11:21.629888   log 2011-05-18 19:06:59.173391 osd2
172.16.35.76:6800/3345 165 : [INF] bench: wrote 1024 MB in blocks of
4096 KB in 19.412487 sec at 54015 KB/sec
2011-05-18 10:11:23.696908   log 2011-05-18 19:09:52.778198 osd4
172.16.35.77:6800/1727 141 : [INF] bench: wrote 1024 MB in blocks of
4096 KB in 15.393317 sec at 68118 KB/sec
2011-05-18 10:11:24.797029   log 2011-05-18 19:07:02.344256 osd3
172.16.35.76:6803/3732 158 : [INF] bench: wrote 1024 MB in blocks of
4096 KB in 19.813493 sec at 52922 KB/sec
2011-05-18 10:11:32.572687   log 2011-05-18 19:10:02.520102 osd5
172.16.35.77:6803/1819 126 : [INF] bench: wrote 1024 MB in blocks of
4096 KB in 23.028046 sec at 45534 KB/sec






On Thu, 2011-05-19 at 13:27 +0800, Dyweni - Ceph-Devel wrote:
> Hi!
> 
>  I've been doing some casual testing of Ceph (GIT Master) and am seeing 
>  some really slow speeds:
>   - rados bench:  5-6MB/s
>   - QEMU-KVM:  1-2MB/s
> 
>  As part of my performance troubleshooting plan, I'd like to find out 
>  what the list considers the "minimum recommended" hardware configuration 
>  to be in order to run Ceph and achieve a reasonably good transfer rate 
>  (I'm thinking 50MB/s+?).
> 
>  My current setup is this:
> 
>  All machines PXE boot and run their OS from a customized initramfs 
>  image.  Hard drives are used solely for data storage.  All machines run 
>  the 2.6.39-rc7-git13 Linux Kernel.  OSD Journal lives in the OSD Data 
>  partition and is 1000 MB.  I use the default (2) number of replicas for 
>  all pools.  No iptables or Ceph authentication being used.  All machines 
>  are connected by an 8 port Linksys Gigabit switch.
> 
>  I use a set of custom shell scripts to build and start the Ceph cluster 
>  from bare metal (once the machines are booted from PXE).
> 
>  I've tried both EXT4 and BTRFS file systems on the OSDs, but both give 
>  the same speeds.
> 
>  1 x MON
>    Dual P3 930Mhz, 256K Cache
>    512MB RAM
>    No HD
>    100Mb NIC
> 
>  1 x MDS
>    Dual Athlon 64 X2 3800+, 512K Cache
>    4GB RAM
>    No HD
>    100Mb NIC
> 
>  4 x OSD
>    OSD1 =
>      P3 1Ghz, 256K Cache
>      2GB RAM
>      80GB IDE (dd_rescue can read about 45MB/s sustained)
>      1Gb NIC (e1000 driver)
>    OSD2 =
>      P3 550Mhz, 512K Cache
>      768MB RAM
>      40GB IDE (dd_rescue can read about 30MB/s sustained)
>      1Gb NIC (e1000 driver)
>    OSD3 =
>      Celeron 1Ghz, 128K Cache
>      512MB RAM
>      18GB IDE (dd_rescue can read about 15MB/s sustained)
>      1Gb NIC (e1000 driver)
>    OSD4 =
>      P3 1Ghz, 256K Cache
>      512MB RAM
>      20GB IDE (dd_rescue can read about 40MB/s sustained)
>      1Gb NIC (e1000 driver)
> 
> 
>  Thanks,
>  Dyweni
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Minimum Recommended Hardware

[Gregory Farnum]

On Thu, May 19, 2011 at 1:27 AM, Dyweni - Ceph-Devel
<YS3fpFE2ykfB@dyweni.com> wrote:
> 4 x OSD
>  OSD1 =
>    P3 1Ghz, 256K Cache
>    2GB RAM
>    80GB IDE (dd_rescue can read about 45MB/s sustained)
>    1Gb NIC (e1000 driver)
>  OSD2 =
>    P3 550Mhz, 512K Cache
>    768MB RAM
>    40GB IDE (dd_rescue can read about 30MB/s sustained)
>    1Gb NIC (e1000 driver)
>  OSD3 =
>    Celeron 1Ghz, 128K Cache
>    512MB RAM
>    18GB IDE (dd_rescue can read about 15MB/s sustained)
>    1Gb NIC (e1000 driver)
>  OSD4 =
>    P3 1Ghz, 256K Cache
>    512MB RAM
>    20GB IDE (dd_rescue can read about 40MB/s sustained)
>    1Gb NIC (e1000 driver)

I'm pretty sure these drives are your problem -- they're just not very
fast. Remember that in general, each write to Ceph is going to write
to both the journal and the data store -- which cuts your drives down
to 22.5, 15, 7.5, 20 MB/s in the best case (and likely a lot worse --
the journal will be a streaming write but the data store will be
random writes, and everything's slower at random writes). On top of
that, we've found that the slowest drives in a cluster can have a
pretty large impact on performance since the faster OSDs end up
consistently waiting on the slow drives.

We have seen other issues with slow CPUs, but those generally only
manifest themselves during recovery and are getting active improvement
as we come across them. :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html