* Re: Performence test on ceph v0.23 + EXT4 and Btrfs [not found] ` <1291002250.1872.116.camel@cephhost> @ 2010-11-29 3:53 ` Jeff Wu 0 siblings, 0 replies; 14+ messages in thread From: Jeff Wu @ 2010-11-29 3:53 UTC (permalink / raw) To: sage, yehuda, gregf, colinm; +Cc: ceph-devel, mllv HTML ---> TEXT,re-send ================================================= Hi , I've recently been using FFSB and iozone to do performance test with Ceph v0.23 on my platform. FFSB configuration file and ceph.conf attached . I am using one server(172.16.10.171) for the MDS and MON daemons and client host, one server(172.16.10.42) is for OSD0 ,one server(172.16.10.65) is for OSD1. The three machines all have Gigabit ethernet cards and connect with Gigabit Router. The disks are formatted using ext4 in no-journal mode and btrfs mode. The following is my patform infos and test results: ceph: 0.23 OS:ubuntu 10.10 x86_64 ,2.6.35-22-generic kernel. Ethernet: one Gigabit MON MDS CLIENT HOST: CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz Memory: 2GB cleint: mount.ceph 172.16.10.171:6789:/ /mnt/ceph OSD0 host: CPU:Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz Memory: 2GB OSD1 host: CPU:AMD Athlon(tm) 64 X2 Dual Core Processor 3600+ Memory: 4GB 1) here are FFSB test result on ceph+btrfs disk 8 thread 16 threads 32 threads large_file_create 14.7 MB/sec 16.4 MB/sec 17.8 MB/sec sequential_reads 15.5 MB/sec 16 MB/sec 17 MB/sec random_reads 490 KB/sec 594 KB/sec 664 KB/sec random_writes 57.2 MB/sec 68.4 MB/sec 72.1 MB/sec mailserver Read:85.8KB/sec Read : 236KB/sec Read:286KB/sec Write : 36KB/sec Write : 132KB/sec Write:129KB/sec 2) For comparison, here are the FFSB test result on ceph+ext4 disk with no journal 8 thread 16 threads 32 threads large_file_create 7.92 MB/sec 8.09 MB/sec 8.46 MB/sec sequential_reads 8.19 MB/sec 8.77 MB/sec 8.14 MB/sec random_reads 786 KB/sec 556 KB/sec 170 KB/sec random_writes 52.9 MB/sec 63 MB/sec 59.1 MB/sec mailserver Read:456KB/sec Read : 249KB/sec Read:485KB/sec Write : 228KB/sec Write : 120KB/sec Write:226KB/sec 3) here are iozone test result on ceph+btrfs disk,file size 6GB , Output is in Kbytes/sec Iozone: Performance Test of File I/O Version $Revision: 3.353 $ Compiled for 64 bit mode. Build: linux-ia64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li. Run began: Fri Nov 26 09:00:33 2010 Include close in write timing Include fsync in write timing Auto Mode Using minimum file size of 6291456 kilobytes. Using maximum file size of 6291456 kilobytes. Excel chart generation enabled Command line used: ./benchmark/iozone/iozone_x86_64 -c -e -a -n 6144M -g 6144M -i 0 -i 1 -i 2 -f /mnt/ceph/f1 -Rb ./benchmark/iozone/iozone.201011260900.xls Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 6291456 64 6627 6417 9898 10334 3629 5908 6291456 128 6803 7182 10200 10582 5106 6268 6291456 256 6734 7249 10821 11224 7348 7135 6291456 512 7109 7213 10538 10682 9392 7788 6291456 1024 6932 7616 11204 10873 8673 8467 6291456 2048 7896 7669 11025 9981 10258 7770 6291456 4096 6933 7084 10590 10703 10450 7758 6291456 8192 7215 7192 10490 10700 11110 7838 6291456 16384 6557 6646 10224 11179 10738 7062 4) For comparison, here are the iozone test result on ceph+ext4 disk with no journal,file size 6GB , Output is in Kbytes/sec Iozone: Performance Test of File I/O Version $Revision: 3.353 $ Compiled for 64 bit mode. Build: linux-ia64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li. Run began: Thu Nov 25 08:42:25 2010 Include close in write timing Include fsync in write timing Auto Mode Using minimum file size of 6291456 kilobytes. Using maximum file size of 6291456 kilobytes. Excel chart generation enabled Command line used: ./benchmark/iozone/iozone_x86_64 -c -e -a -n 6144M -g 6144M -i 0 -i 1 -i 2 -f /mnt/ceph/f1 -Rb ./benchmark/iozone/iozone.201011250841.xls Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 6291456 64 7214 7128 9847 9991 3112 4168 6291456 128 7514 7367 10601 10281 4667 6041 6291456 256 7420 7414 11041 11238 6933 7860 6291456 512 8190 8120 11449 11166 9001 7959 6291456 1024 7611 7702 10497 10391 7497 8887 6291456 2048 7516 7408 9908 10254 8639 8387 6291456 4096 7355 7453 10383 10598 9469 7554 6291456 8192 7415 7651 10244 10240 9868 8450 6291456 16384 7200 7166 9877 9778 9829 8228 Are these results reasonable ? which seem too slow , maybe,i do something wrong. Could you give me some ceph performance test results for the reference ? Any ideas ,please let me know ,thanks. Jeff.Wu =============================ceph.conf ======================================= ; ; Sample ceph ceph.conf file. ; ; This file defines cluster membership, the various locations ; that Ceph stores data, and any other runtime options. ; If a 'host' is defined for a daemon, the start/stop script will ; verify that it matches the hostname (or else ignore it). If it is ; not defined, it is assumed that the daemon is intended to start on ; the current host (e.g., in a setup with a startup.conf on each ; node). ; global [global] ; enable secure authentication ; auth supported = cephx keyring = /etc/ceph/keyring.bin ; monitors ; You need at least one. You need at least three if you want to ; tolerate any node failures. Always create an odd number. [mon] mon data = /opt/ceph/data/mon$id ; logging, for debugging monitor crashes, in order of ; their likelihood of being helpful :) ;debug ms = 20 ;debug mon = 20 ;debug paxos = 20 ;debug auth = 20 [mon0] host = ubuntu-mon0 mon addr = 172.16.10.171:6789 [mon1] host = ubuntu-mon0 mon addr = 172.16.10.171:6790 [mon2] host = ubuntu-mon0 mon addr = 172.16.10.171:6791 ; mds ; You need at least one. Define two to get a standby. [mds] ; where the mds keeps it's secret encryption keys keyring = /etc/ceph/keyring.$name ; mds logging to debug issues. ;debug ms = 20 ;debug mds = 20 [mds.0] host = ubuntu-mon0 [mds.1] host = ubuntu-mon0 ; osd ; You need at least one. Two if you want data to be replicated. ; Define as many as you like. [osd] ; This is where the btrfs volume will be mounted. osd data = /opt/ceph/data/osd$id ;osd journal = /opt/ceph/data/osd$id/journal osd class tmp = /var/lib/ceph/tmp ; Ideally, make this a separate disk or partition. A few ; hundred MB should be enough; more if you have fast or many ; disks. You can use a file under the osd data dir if need be ; (e.g. /data/osd$id/journal), but it will be slower than a ; separate disk or partition. ; This is an example of a file-based journal. ; osd journal size = 1000 ; journal size, in megabytes keyring = /etc/ceph/keyring.$name ; osd logging to debug osd issues, in order of likelihood of being ; helpful ;debug ms = 20 ;debug osd = 20 ;debug filestore = 20 ;debug journal = 20 [osd0] host = ubuntu-osd0 ;osd journal size = 1000 ; journal size, in megabytes ; if 'btrfs devs' is not specified, you're responsible for ; setting up the 'osd data' dir. if it is not btrfs, things ; will behave up until you try to recover from a crash (which ; usually fine for basic testing). ; btrfs devs = /dev/sdx [osd1] host = ubuntu-osd1 ;osd data = /opt/data/osd$id ;osd journal = /opt/data/osd$id/journal ;filestore journal writeahead = true ;osd journal size = 1000 ; journal size, in megabytes ;btrfs devs = /dev/sdy ;[osd2] ;host = zeta ;btrfs devs = /dev/sdx ;[osd3] ;host = eta ;btrfs devs = /dev/sdy ================================= large_files_create======================================== # Large file creates # Creating 1024 MB files. time=300 alignio=1 directio=0 [filesystem0] location=%TESTPATH% # All created files will be 1024 MB. min_filesize=1024M max_filesize=1024M [end0] [threadgroup0] num_threads=32 # 8,16 create_weight=1 write_blocksize=4K [stats] enable_stats=1 enable_range=1 msec_range 0.00 0.01 msec_range 0.01 0.02 msec_range 0.02 0.05 msec_range 0.05 0.10 msec_range 0.10 0.20 msec_range 0.20 0.50 msec_range 0.50 1.00 msec_range 1.00 2.00 msec_range 2.00 5.00 msec_range 5.00 10.00 msec_range 10.00 20.00 msec_range 20.00 50.00 msec_range 50.00 100.00 msec_range 100.00 200.00 msec_range 200.00 500.00 msec_range 500.00 1000.00 msec_range 1000.00 2000.00 msec_range 2000.00 5000.00 msec_range 5000.00 10000.00 [end] [end0] ================================= mail server ======================================== # Mail server simulation. # 1024 file time=300 alignio=1 directio=0 [filesystem0] location=%TESTPATH% num_files=1024 num_dirs=100 # File sizes range from 1kB to 1MB. size_weight 1KB 10 size_weight 2KB 15 size_weight 4KB 16 size_weight 8KB 16 size_weight 16KB 15 size_weight 32KB 10 size_weight 64KB 8 size_weight 128KB 4 size_weight 256KB 3 size_weight 512KB 2 size_weight 1MB 1 [end0] [threadgroup0] num_threads=32 # 8,16 readall_weight=4 create_fsync_weight=2 delete_weight=1 write_size=4KB write_blocksize=4KB read_size=4KB read_blocksize=4KB [stats] enable_stats=1 enable_range=1 msec_range 0.00 0.01 msec_range 0.01 0.02 msec_range 0.02 0.05 msec_range 0.05 0.10 msec_range 0.10 0.20 msec_range 0.20 0.50 msec_range 0.50 1.00 msec_range 1.00 2.00 msec_range 2.00 5.00 msec_range 5.00 10.00 msec_range 10.00 20.00 msec_range 20.00 50.00 msec_range 50.00 100.00 msec_range 100.00 200.00 msec_range 200.00 500.00 msec_range 500.00 1000.00 msec_range 1000.00 2000.00 msec_range 2000.00 5000.00 msec_range 5000.00 10000.00 [end] [end0] ================================= random reads======================================== # Large file random reads. # 256 files, 100MB per file. time=300 # 5 min alignio=1 [filesystem0] location=%TESTPATH% num_files=256 min_filesize=100M # 100 MB max_filesize=100M reuse=1 [end0] [threadgroup0] num_threads=32 # 8,16 read_random=1 read_weight=1 read_size=1M # 1 MB read_blocksize=4k [stats] enable_stats=1 enable_range=1 msec_range 0.00 0.01 msec_range 0.01 0.02 msec_range 0.02 0.05 msec_range 0.05 0.10 msec_range 0.10 0.20 msec_range 0.20 0.50 msec_range 0.50 1.00 msec_range 1.00 2.00 msec_range 2.00 5.00 msec_range 5.00 10.00 msec_range 10.00 20.00 msec_range 20.00 50.00 msec_range 50.00 100.00 msec_range 100.00 200.00 msec_range 200.00 500.00 msec_range 500.00 1000.00 msec_range 1000.00 2000.00 msec_range 2000.00 5000.00 msec_range 5000.00 10000.00 [end] [end0] ================================= random writes======================================== # Large file random writes. # 256 files, 100MB per file. time=300 # 5 min alignio=1 [filesystem0] location=%TESTPATH% num_files=256 min_filesize=100M # 100 MB max_filesize=100M reuse=1 [end0] [threadgroup0] num_threads=32 # 8,16 write_random=1 write_weight=1 write_size=1M # 1 MB write_blocksize=4k [stats] enable_stats=1 enable_range=1 msec_range 0.00 0.01 msec_range 0.01 0.02 msec_range 0.02 0.05 msec_range 0.05 0.10 msec_range 0.10 0.20 msec_range 0.20 0.50 msec_range 0.50 1.00 msec_range 1.00 2.00 msec_range 2.00 5.00 msec_range 5.00 10.00 msec_range 10.00 20.00 msec_range 20.00 50.00 msec_range 50.00 100.00 msec_range 100.00 200.00 msec_range 200.00 500.00 msec_range 500.00 1000.00 msec_range 1000.00 2000.00 msec_range 2000.00 5000.00 msec_range 5000.00 10000.00 [end] [end0] ================================= sequential reads======================================== # Large file sequential reads. # 256 files, 100MB per file. time=300 # 5 min alignio=1 [filesystem0] location=%TESTPATH% num_files=256 min_filesize=100M # 100 MB max_filesize=100M # 100 MB reuse=1 [end0] [threadgroup0] num_threads=32 # 8,16 read_weight=1 read_size=1M # 1 MB read_blocksize=4k [stats] enable_stats=1 enable_range=1 msec_range 0.00 0.01 msec_range 0.01 0.02 msec_range 0.02 0.05 msec_range 0.05 0.10 msec_range 0.10 0.20 msec_range 0.20 0.50 msec_range 0.50 1.00 msec_range 1.00 2.00 msec_range 2.00 5.00 msec_range 5.00 10.00 msec_range 10.00 20.00 msec_range 20.00 50.00 msec_range 50.00 100.00 msec_range 100.00 200.00 msec_range 200.00 500.00 msec_range 500.00 1000.00 msec_range 1000.00 2000.00 msec_range 2000.00 5000.00 msec_range 5000.00 10000.00 [end] [end0] ================================================================== ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs [not found] <1291001135.1872.106.camel@cephhost> [not found] ` <1291001398.1872.107.camel@cephhost> @ 2010-11-29 17:07 ` Gregory Farnum 2010-11-30 2:55 ` Jeff Wu 1 sibling, 1 reply; 14+ messages in thread From: Gregory Farnum @ 2010-11-29 17:07 UTC (permalink / raw) To: cpwu; +Cc: sage, yehuda, colinm, ceph-devel, mllv On Sun, Nov 28, 2010 at 7:25 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > Hi , > > I've recently been using FFSB and iozone to do performance test with Ceph > v0.23 on my platform. > the attachment file are FFSB configuration file and ceph.conf. So you're not using OSD journals in either test configuration? You're going to get pretty terrible write results without a journal. The reads are clearly slower than they should be and you could probably get better results by adjusting the caching behaviors. We haven't done too much work on optimizing read behavior. Could you run "ceph osd tell * bench", then run "ceph -w", and report the results? (That'll just run local benchmarking on the OSD to report the approximate write speed it's capable of.) You can also run "rados -p data bench 60 write", and then "rados -p data bench 0 seq" to get a simpler (better understood) performance test. With this data as a baseline we can start looking at what might be causing trouble. -Greg ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-11-29 17:07 ` Gregory Farnum @ 2010-11-30 2:55 ` Jeff Wu 2010-11-30 3:18 ` Gregory Farnum 0 siblings, 1 reply; 14+ messages in thread From: Jeff Wu @ 2010-11-30 2:55 UTC (permalink / raw) To: Gregory Farnum Cc: sage@newdream.net, yehuda@hq.newdream.net, colinm@hq.newdream.net, ceph-devel@vger.kernel.org, Andrew Lv 在 2010-11-30二的 01:07 +0800，Gregory Farnum写道： > On Sun, Nov 28, 2010 at 7:25 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > > > Hi , > > > > I've recently been using FFSB and iozone to do performance test with Ceph > > v0.23 on my platform. > > the attachment file are FFSB configuration file and ceph.conf. > So you're not using OSD journals in either test configuration? You're > going to get pretty terrible write results without a journal. > The reads are clearly slower than they should be and you could > probably get better results by adjusting the caching behaviors. We > haven't done too much work on optimizing read behavior. Hi ,Greg , thank you for your suggestions. > > Could you run "ceph osd tell * bench", then run "ceph -w", and report > the results? (That'll just run local benchmarking on the OSD to report > the approximate write speed it's capable of.) I run six times for the command: "$ sudo ceph osd tell 0/1 bench " use "$ sudo ceph -w" to get the following results: osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.906598 sec at 49775 KB/sec osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.023294 sec at 49384 KB/sec osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.834682 sec at 51535 KB/sec osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 20.792697 sec at 37547 KB/sec osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.058412 sec at 77191 KB/sec osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.113612 sec at 47369 KB/sec The detail test logs attached below. > You can also run "rados -p data bench 60 write", and then "rados -p > data bench 0 seq" to get a simpler (better understood) performance > test. I run twice for the command: "rados -p data bench 60 write" , Get the results: $ sudo rados -p data bench 60 write .......................... .......................... Total time run: 76.182225 Total writes made: 121 Write size: 4194304 Bandwidth (MB/sec): 6.219 Average Latency: 13.3068 Max latency: 23.9986 Min latency: 7.01847 $ sudo rados -p data bench 60 write .......................... .......................... Total time run: 74.830651 Total writes made: 97 Write size: 4194304 Bandwidth (MB/sec): 4.714 Average Latency: 15.5064 Max latency: 24.5641 Min latency: 3.50005 but run "$ sudo rados -p data bench 0 seq" fail to get the results, Maybe it's a bug,ceph version 0.23. logs: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 16 16 0 0 0 - 0 read got -2 error during benchmark: -2 ./common/Mutex.h: In function 'Mutex::~Mutex()': ./common/Mutex.h:97: FAILED assert(nlock == 0) ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) ............................... ............................... The detail logs addached below. > With this data as a baseline we can start looking at what might be > causing trouble. > -Greg ========================================================== 1. Process transoft@ubuntu-mon0:/usr/local/etc/ceph$ ps -ef root 12919 1 2 Nov26 ? 02:23:45 /usr/local/bin/cmon -i 0 -c ceph.conf root 12952 1 1 Nov26 ? 01:30:03 /usr/local/bin/cmon -i 1 -c ceph.conf root 12987 1 1 Nov26 ? 01:51:42 /usr/local/bin/cmon -i 2 -c ceph.conf root 13036 1 1 Nov26 ? 01:00:08 /usr/local/bin/cmds -i 0 -c ceph.conf root 13075 1 0 Nov26 ? 00:47:47 /usr/local/bin/cmds -i 1 -c ceph.conf ========================================================== 2. ceph -w transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 0 bench 2010-11-30 08:47:09.080125 mon <- [osd,tell,0,bench] 2010-11-30 08:47:09.080378 mon1 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 1 bench 2010-11-30 08:47:54.520159 mon <- [osd,tell,1,bench] 2010-11-30 08:47:54.520433 mon2 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 0 bench 2010-11-30 08:48:29.590115 mon <- [osd,tell,0,bench] 2010-11-30 08:48:29.590365 mon2 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 1 bench 2010-11-30 08:48:47.450092 mon <- [osd,tell,1,bench] 2010-11-30 08:48:47.450341 mon1 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 0 bench 2010-11-30 08:49:27.240742 mon <- [osd,tell,0,bench] 2010-11-30 08:49:27.241076 mon1 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 1 bench 2010-11-30 08:49:43.500749 mon <- [osd,tell,1,bench] 2010-11-30 08:49:43.501043 mon2 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph -w 2010-11-30 08:47:04.150457 pg v8701: 528 pgs: 528 active+clean; 10674 KB data, 442 MB used, 219 GB / 219 GB avail 2010-11-30 08:47:04.151228 mds e5: 1/1/1 up {0=up:active}, 1 up:standby 2010-11-30 08:47:04.151253 osd e6: 2 osds: 2 up, 2 in 2010-11-30 08:47:04.151319 log 2010-11-30 08:45:17.795260 osd0 172.16.10.42:6800/16864 3 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.828886 sec at 51682 KB/sec 2010-11-30 08:47:04.151375 mon e1: 3 mons at {0=172.16.10.171:6789/0,1=172.16.10.171:6790/0,2=172.16.10.171:6791/0} 2010-11-30 08:47:22.487639 log 2010-11-30 08:47:00.704960 osd0 172.16.10.42:6800/16864 4 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.906598 sec at 49775 KB/sec 2010-11-30 08:48:17.047842 log 2010-11-30 09:52:29.975820 osd1 172.16.10.65:6800/6678 3 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.023294 sec at 49384 KB/sec 2010-11-30 08:48:42.915344 log 2010-11-30 08:48:21.135651 osd0 172.16.10.42:6800/16864 5 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.834682 sec at 51535 KB/sec 2010-11-30 08:49:10.000303 log 2010-11-30 09:53:22.645957 osd1 172.16.10.65:6800/6678 4 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 20.792697 sec at 37547 KB/sec 2010-11-30 08:49:40.809023 log 2010-11-30 08:49:19.005413 osd0 172.16.10.42:6800/16864 6 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.058412 sec at 77191 KB/sec 2010-11-30 08:50:06.184830 log 2010-11-30 09:54:19.064900 osd1 172.16.10.65:6800/6678 5 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.113612 sec at 47369 KB/sec =========================================================== 3. rados -p data bench 60 write transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo rados -p data bench 60 write Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds. sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 16 16 0 0 0 - 0 1 16 16 0 0 0 - 0 2 16 16 0 0 0 - 0 3 16 16 0 0 0 - 0 4 16 16 0 0 0 - 0 5 16 17 1 0.793474 0.8 7.9252 7.9252 6 16 17 1 0.662028 0 - 7.9252 7 16 17 1 0.567947 0 - 7.9252 8 16 24 8 3.97823 9.33333 10.5391 10.1201 9 16 24 8 3.53801 0 - 10.1201 10 16 38 22 8.76019 28 7.13522 9.96534 11 16 38 22 7.96645 0 - 9.96534 12 16 38 22 7.30461 0 - 9.96534 13 16 38 22 6.74431 0 - 9.96534 14 16 38 22 6.26388 0 - 9.96534 15 16 38 22 5.84735 0 - 9.96534 16 16 41 25 6.23039 2 8.7632 10.3001 17 16 41 25 5.86468 0 - 10.3001 18 16 41 25 5.53953 0 - 10.3001 19 16 52 36 7.55792 14.6667 10.3215 11.3923 min lat: 7.01847 max lat: 15.8599 avg lat: 11.3923 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 52 36 7.18061 0 - 11.3923 21 16 56 40 7.59925 8 13.7597 11.8294 22 16 56 40 7.25439 0 - 11.8294 23 16 56 40 6.93949 0 - 11.8294 24 16 56 40 6.65079 0 - 11.8294 25 16 63 47 7.50258 7 7.33881 11.2982 26 16 63 47 7.21441 0 - 11.2982 27 16 68 52 7.68667 10 9.28991 11.1051 28 16 68 52 7.41251 0 - 11.1051 29 16 68 52 7.15725 0 - 11.1051 30 16 68 52 6.91898 0 - 11.1051 31 16 68 52 6.69606 0 - 11.1051 32 16 79 63 7.85927 8.8 15.0751 11.933 33 16 79 63 7.62121 0 - 11.933 34 16 79 63 7.3973 0 - 11.933 35 16 79 63 7.18618 0 - 11.933 36 16 79 63 6.98678 0 - 11.933 37 16 84 68 7.33767 4 17.4944 12.3426 38 16 84 68 7.14464 0 - 12.3426 39 16 85 69 7.06399 2 8.24284 12.2832 min lat: 7.01847 max lat: 19.7384 avg lat: 12.2832 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 40 16 85 69 6.88751 0 - 12.2832 41 16 85 69 6.71971 0 - 12.2832 42 16 85 69 6.55985 0 - 12.2832 43 16 87 71 6.59315 2 15.8845 12.3846 44 16 87 71 6.44343 0 - 12.3846 45 16 87 71 6.30037 0 - 12.3846 46 16 87 71 6.16352 0 - 12.3846 47 16 91 75 6.37235 4 21.8578 12.8896 48 16 91 75 6.23967 0 - 12.8896 49 16 101 85 6.92742 20 18.2741 13.1211 50 16 101 85 6.78897 0 - 13.1211 51 16 101 85 6.65595 0 - 13.1211 52 16 101 85 6.52804 0 - 13.1211 53 16 101 85 6.40496 0 - 13.1211 54 16 111 95 7.02602 8 13.1645 12.939 55 16 111 95 6.89835 0 - 12.939 56 16 111 95 6.77524 0 - 12.939 57 16 111 95 6.65646 0 - 12.939 58 16 111 95 6.54176 0 - 12.939 59 16 115 99 6.70174 3.2 13.1073 12.9458 min lat: 7.01847 max lat: 21.8578 avg lat: 12.9364 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 60 16 120 104 6.92291 20 11.3313 12.9364 61 15 121 106 6.94046 8 13.3389 12.9444 62 15 121 106 6.82859 0 - 12.9444 63 15 121 106 6.72026 0 - 12.9444 64 15 121 106 6.61533 0 - 12.9444 65 15 121 106 6.51362 0 - 12.9444 66 11 121 110 6.65706 3.2 14.3524 12.9974 67 11 121 110 6.55777 0 - 12.9974 68 11 121 110 6.46139 0 - 12.9974 69 11 121 110 6.3678 0 - 12.9974 70 11 121 110 6.27689 0 - 12.9974 71 2 121 119 6.69486 7.2 13.2646 13.1676 72 2 121 119 6.60193 0 - 13.1676 73 2 121 119 6.51154 0 - 13.1676 74 2 121 119 6.4236 0 - 13.1676 75 2 121 119 6.338 0 - 13.1676 76 2 121 119 6.25465 0 - 13.1676 Total time run: 76.182225 Total writes made: 121 Write size: 4194304 Bandwidth (MB/sec): 6.219 Average Latency: 13.3068 Max latency: 23.9986 Min latency: 7.01847 transoft@ubuntu-mon0:/usr/local/etc/ceph$ transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo rados -p data bench 60 write Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds. sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 16 16 0 0 0 - 0 1 16 16 0 0 0 - 0 2 16 16 0 0 0 - 0 3 16 17 1 1.31537 1.33333 3.50005 3.50005 4 16 17 1 0.989638 0 - 3.50005 5 16 17 1 0.793226 0 - 3.50005 6 16 17 1 0.661871 0 - 3.50005 7 16 17 1 0.567841 0 - 3.50005 8 16 17 1 0.497201 0 - 3.50005 9 16 17 1 0.442194 0 - 3.50005 10 16 17 1 0.398146 0 - 3.50005 11 16 17 1 0.362079 0 - 3.50005 12 16 25 9 2.98799 3.55556 12.0273 17.9704 13 16 27 11 3.37183 8 12.0974 16.8994 14 16 33 17 4.83996 24 12.8105 16.4478 15 16 33 17 4.5181 0 - 16.4478 16 16 33 17 4.2364 0 - 16.4478 17 16 33 17 3.98777 0 - 16.4478 18 16 33 17 3.76669 0 - 16.4478 19 16 38 22 4.6185 4 14.6053 16.0123 min lat: 3.50005 max lat: 20.9926 avg lat: 16.0123 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 38 22 4.38793 0 - 16.0123 21 16 38 22 4.17939 0 - 16.0123 22 16 38 22 3.98976 0 - 16.0123 23 16 38 22 3.81659 0 - 16.0123 24 16 38 22 3.65782 0 - 16.0123 25 16 47 31 4.94834 6 13.0266 15.7939 26 16 47 31 4.75829 0 - 15.7939 27 16 47 31 4.58231 0 - 15.7939 28 16 47 31 4.41888 0 - 15.7939 29 16 47 31 4.26672 0 - 15.7939 30 16 47 31 4.12467 0 - 15.7939 31 16 54 38 4.89315 4.66667 14.5712 16.1038 32 16 54 38 4.74042 0 - 16.1038 33 16 54 38 4.59694 0 - 16.1038 34 16 54 38 4.46188 0 - 16.1038 35 16 54 38 4.33454 0 - 16.1038 36 16 63 47 5.21238 7.2 12.2193 15.3573 37 16 66 50 5.39536 12 9.59416 15.0125 38 16 66 50 5.25352 0 - 15.0125 39 16 66 50 5.11895 0 - 15.0125 min lat: 3.50005 max lat: 24.5641 avg lat: 15.0125 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 40 16 66 50 4.99107 0 - 15.0125 41 16 66 50 4.86947 0 - 15.0125 42 16 73 57 5.41915 5.6 11.6831 14.5749 43 16 73 57 5.29323 0 - 14.5749 44 16 73 57 5.17304 0 - 14.5749 45 16 73 57 5.05817 0 - 14.5749 46 16 73 57 4.94831 0 - 14.5749 47 16 82 66 5.60781 7.2 14.1181 14.6534 48 16 82 66 5.49105 0 - 14.6534 49 16 82 66 5.37907 0 - 14.6534 50 16 82 66 5.27157 0 - 14.6534 51 16 82 66 5.16829 0 - 14.6534 52 16 82 66 5.06898 0 - 14.6534 53 16 88 72 5.42554 4 14.2779 14.6222 54 16 88 72 5.32512 0 - 14.6222 55 16 88 72 5.22837 0 - 14.6222 56 16 88 72 5.13507 0 - 14.6222 57 16 88 72 5.04505 0 - 14.6222 58 16 88 72 4.95812 0 - 14.6222 59 16 90 74 5.00954 1.33333 18.7183 14.808 min lat: 3.50005 max lat: 24.5641 avg lat: 14.8389 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 60 16 96 80 5.32549 24 16.2963 14.8389 61 16 96 80 5.23826 0 - 14.8389 62 16 96 80 5.15383 0 - 14.8389 63 16 96 80 5.07208 0 - 14.8389 64 13 97 84 5.24252 4 13.2043 15.0243 65 13 97 84 5.16191 0 - 15.0243 66 13 97 84 5.08375 0 - 15.0243 67 13 97 84 5.00792 0 - 15.0243 68 13 97 84 4.93432 0 - 15.0243 69 13 97 84 4.86285 0 - 15.0243 70 5 97 92 5.24994 5.33333 18.0332 15.3781 71 5 97 92 5.17604 0 - 15.3781 72 5 97 92 5.10419 0 - 15.3781 73 5 97 92 5.0343 0 - 15.3781 74 5 97 92 4.96631 0 - 15.3781 Total time run: 74.830651 Total writes made: 97 Write size: 4194304 Bandwidth (MB/sec): 4.714 Average Latency: 15.5064 Max latency: 24.5641 Min latency: 3.50005 transoft@ubuntu-mon0:/usr/local/etc/ceph$ =========================================================== 4. rados -p data bench 0 seq transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo rados -p data bench 0 seq sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 16 16 0 0 0 - 0 read got -2 error during benchmark: -2 ./common/Mutex.h: In function 'Mutex::~Mutex()': ./common/Mutex.h:97: FAILED assert(nlock == 0) ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) 1: rados() [0x40c941] 2: (exit()+0xe2) [0x7f518ea744f2] 3: (__libc_start_main()+0x105) [0x7f518ea59d95] 4: rados() [0x405b19] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ./common/Mutex.h: In function 'Mutex::~Mutex()': ./common/Mutex.h:97: FAILED assert(nlock == 0) ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) 1: rados() [0x40c941] 2: (exit()+0xe2) [0x7f518ea744f2] 3: (__libc_start_main()+0x105) [0x7f518ea59d95] 4: rados() [0x405b19] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (ABRT) *** ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) 1: (sigabrt_handler(int)+0x91) [0x7f518fe7dec1] 2: (()+0x33c20) [0x7f518ea6ec20] 3: (gsignal()+0x35) [0x7f518ea6eba5] 4: (abort()+0x180) [0x7f518ea726b0] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f518f3126bd] 6: (()+0xb9906) [0x7f518f310906] 7: (()+0xb9933) [0x7f518f310933] 8: (()+0xb9a3e) [0x7f518f310a3e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x456) [0x7f518fe5ab96] 10: rados() [0x40c941] 11: (exit()+0xe2) [0x7f518ea744f2] 12: (__libc_start_main()+0x105) [0x7f518ea59d95] 13: rados() [0x405b19] Aborted transoft@ubuntu-mon0:/usr/local/etc/ceph$ transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo rados -p data bench 60 seq sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 16 16 0 0 0 - 0 read got -2 error during benchmark: -2 ./common/Mutex.h: In function 'Mutex::~Mutex()': ./common/Mutex.h:97: FAILED assert(nlock == 0) ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) 1: rados() [0x40c941] 2: (exit()+0xe2) [0x7f795c31c4f2] 3: (__libc_start_main()+0x105) [0x7f795c301d95] 4: rados() [0x405b19] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ./common/Mutex.h: In function 'Mutex::~Mutex()': ./common/Mutex.h:97: FAILED assert(nlock == 0) ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) 1: rados() [0x40c941] 2: (exit()+0xe2) [0x7f795c31c4f2] 3: (__libc_start_main()+0x105) [0x7f795c301d95] 4: rados() [0x405b19] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (ABRT) *** ceph version 0.23 (commit:5d1d8d0c4602be9819cc9f7aea562fccbb005a56) 1: (sigabrt_handler(int)+0x91) [0x7f795d725ec1] 2: (()+0x33c20) [0x7f795c316c20] 3: (gsignal()+0x35) [0x7f795c316ba5] 4: (abort()+0x180) [0x7f795c31a6b0] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f795cbba6bd] 6: (()+0xb9906) [0x7f795cbb8906] 7: (()+0xb9933) [0x7f795cbb8933] 8: (()+0xb9a3e) [0x7f795cbb8a3e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x456) [0x7f795d702b96] 10: rados() [0x40c941] 11: (exit()+0xe2) [0x7f795c31c4f2] 12: (__libc_start_main()+0x105) [0x7f795c301d95] 13: rados() [0x405b19] Aborted transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo rados -p data bench 60 rand Random test not implemented yet! error during benchmark: -1 transoft@ubuntu-mon0:/usr/local/etc/ceph$ Best Regards Jeff.Wu -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-11-30 2:55 ` Jeff Wu @ 2010-11-30 3:18 ` Gregory Farnum 2010-11-30 6:19 ` Jeff Wu 0 siblings, 1 reply; 14+ messages in thread From: Gregory Farnum @ 2010-11-30 3:18 UTC (permalink / raw) To: cpwu; +Cc: ceph-devel@vger.kernel.org, Andrew Lv On Mon, Nov 29, 2010 at 6:55 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: >> Could you run "ceph osd tell * bench", then run "ceph -w", and report >> the results? (That'll just run local benchmarking on the OSD to report >> the approximate write speed it's capable of.) > > I run six times for the command: "$ sudo ceph osd tell 0/1 bench " > use "$ sudo ceph -w" to get the following results: > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.906598 sec > at 49775 KB/sec > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.023294 sec > at 49384 KB/sec > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.834682 sec > at 51535 KB/sec > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 20.792697 sec > at 37547 KB/sec > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.058412 sec > at 77191 KB/sec > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.113612 sec > at 47369 KB/sec Okay, those are a bit slow but reasonable. Based on these I'd expect you to generally manage about 40-50MB/s (since everything is replicated; in a 2-disk configuration it'll just be the speed of your slowest disk), assuming a properly configured system. >> You can also run "rados -p data bench 60 write", and then "rados -p >> data bench 0 seq" to get a simpler (better understood) performance >> test. > > I run twice for the command: "rados -p data bench 60 write" , > Get the results: > $ sudo rados -p data bench 60 write > .......................... > .......................... > > Total time run: 76.182225 > Total writes made: 121 > Write size: 4194304 > Bandwidth (MB/sec): 6.219 > > Average Latency: 13.3068 > Max latency: 23.9986 > Min latency: 7.01847 WOAH. That's a lot of latency. Rather more than I'd expect to get just from seek times in a non-journaled environment. What's the round-trip time to ping the OSDs from your client? Are your disks okay? > but run "$ sudo rados -p data bench 0 seq" > fail to get the results, Maybe it's a bug,ceph version 0.23. Oh right, sorry. We put in a quick fix to make the write benchmark scale across multiple client writers and forgot to adjust the read benchmark. *oops* -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-11-30 3:18 ` Gregory Farnum @ 2010-11-30 6:19 ` Jeff Wu 2010-11-30 17:07 ` Gregory Farnum 0 siblings, 1 reply; 14+ messages in thread From: Jeff Wu @ 2010-11-30 6:19 UTC (permalink / raw) To: Gregory Farnum; +Cc: ceph-devel@vger.kernel.org, Andrew Lv 在 2010-11-30二的 11:18 +0800，Gregory Farnum写道： > On Mon, Nov 29, 2010 at 6:55 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > >> Could you run "ceph osd tell * bench", then run "ceph -w", and report > >> the results? (That'll just run local benchmarking on the OSD to report > >> the approximate write speed it's capable of.) > > > > I run six times for the command: "$ sudo ceph osd tell 0/1 bench " > > use "$ sudo ceph -w" to get the following results: > > > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.906598 sec > > at 49775 KB/sec > > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.023294 sec > > at 49384 KB/sec > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.834682 sec > > at 51535 KB/sec > > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 20.792697 sec > > at 37547 KB/sec > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.058412 sec > > at 77191 KB/sec > > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.113612 sec > > at 47369 KB/sec > Okay, those are a bit slow but reasonable. Based on these I'd expect > you to generally manage about 40-50MB/s (since everything is > replicated; in a 2-disk configuration it'll just be the speed of your > slowest disk), assuming a properly configured system. Is "40-50MB/s" the speed that it run bench at local btrfs disk ? not the speed that run bench from client to osd server ? with this speed ,run bench from client to osd server ,will which get about 20~25MB/s( 40~50MB /2 )speed ? > > >> You can also run "rados -p data bench 60 write", and then "rados -p > >> data bench 0 seq" to get a simpler (better understood) performance > >> test. > > > > I run twice for the command: "rados -p data bench 60 write" , > > Get the results: > > $ sudo rados -p data bench 60 write > > .......................... > > .......................... > > > > Total time run: 76.182225 > > Total writes made: 121 > > Write size: 4194304 > > Bandwidth (MB/sec): 6.219 > > > > Average Latency: 13.3068 > > Max latency: 23.9986 > > Min latency: 7.01847 > WOAH. That's a lot of latency. Rather more than I'd expect to get just > from seek times in a non-journaled environment. What's the round-trip > time to ping the OSDs from your client? Are your disks okay? > Hi,i get the RTT and the disks are okay . 1. RTT: client ping OSD0: rtt min/avg/max/mdev = 0.103/0.124/0.143/0.016 ms client ping OSD1: rtt min/avg/max/mdev = 0.112/0.116/0.122/0.005 ms 2. disks 1) mon data save in sda4, which formatted with mkfs.btrfs $lsscsi [0:0:0:0] disk ATA WDC WD3200AAKS-7 02.0 /dev/sda $fdisk -l ............. /dev/sda4 17264 38913 173897568+ 83 Linux 2) OSD0 data save in sda5,which formatted with mkfs.btrfs ~$ lsscsi [0:0:0:0] disk ATA ST3320418AS CC45 /dev/sda $ fdisk -l ....................... /dev/sda5 18480 38913 164131649 83 Linux 3) OSD1 data save in sda5,which formatted with mkfs.btrfs $ lsscsi [0:0:0:0] disk ATA ST3160812AS 3.AD /dev/sda $ fdisk -l /dev/sda5 11185 19452 66405470+ 83 Linux more detail logs attached(see below) Thank you jeff.wu > > but run "$ sudo rados -p data bench 0 seq" > > fail to get the results, Maybe it's a bug,ceph version 0.23. > Oh right, sorry. We put in a quick fix to make the write benchmark > scale across multiple client writers and forgot to adjust the read > benchmark. *oops* thank you ! > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html logs: ================================================================= transoft@ubuntu-mon0:/usr/local/etc/ceph$ ping 172.16.10.65 PING 172.16.10.65 (172.16.10.65) 56(84) bytes of data. 64 bytes from 172.16.10.65: icmp_req=1 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=2 ttl=64 time=0.121 ms 64 bytes from 172.16.10.65: icmp_req=3 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=4 ttl=64 time=0.124 ms 64 bytes from 172.16.10.65: icmp_req=5 ttl=64 time=0.128 ms 64 bytes from 172.16.10.65: icmp_req=6 ttl=64 time=0.128 ms 64 bytes from 172.16.10.65: icmp_req=7 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=8 ttl=64 time=0.126 ms 64 bytes from 172.16.10.65: icmp_req=9 ttl=64 time=0.126 ms 64 bytes from 172.16.10.65: icmp_req=10 ttl=64 time=0.128 ms 64 bytes from 172.16.10.65: icmp_req=11 ttl=64 time=0.120 ms 64 bytes from 172.16.10.65: icmp_req=12 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=13 ttl=64 time=0.120 ms 64 bytes from 172.16.10.65: icmp_req=14 ttl=64 time=0.125 ms 64 bytes from 172.16.10.65: icmp_req=15 ttl=64 time=0.131 ms 64 bytes from 172.16.10.65: icmp_req=16 ttl=64 time=0.125 ms 64 bytes from 172.16.10.65: icmp_req=17 ttl=64 time=0.124 ms 64 bytes from 172.16.10.65: icmp_req=18 ttl=64 time=0.103 ms 64 bytes from 172.16.10.65: icmp_req=19 ttl=64 time=0.124 ms 64 bytes from 172.16.10.65: icmp_req=20 ttl=64 time=0.125 ms 64 bytes from 172.16.10.65: icmp_req=21 ttl=64 time=0.143 ms 64 bytes from 172.16.10.65: icmp_req=22 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=23 ttl=64 time=0.117 ms ^C --- 172.16.10.65 ping statistics --- 23 packets transmitted, 23 received, 0% packet loss, time 21999ms rtt min/avg/max/mdev = 0.103/0.124/0.143/0.016 ms transoft@ubuntu-mon0:/usr/local/etc/ceph$ ping 172.16.10.42 PING 172.16.10.42 (172.16.10.42) 56(84) bytes of data. 64 bytes from 172.16.10.42: icmp_req=1 ttl=64 time=0.121 ms 64 bytes from 172.16.10.42: icmp_req=2 ttl=64 time=0.116 ms 64 bytes from 172.16.10.42: icmp_req=3 ttl=64 time=0.122 ms 64 bytes from 172.16.10.42: icmp_req=5 ttl=64 time=0.117 ms 64 bytes from 172.16.10.42: icmp_req=6 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=7 ttl=64 time=0.117 ms 64 bytes from 172.16.10.42: icmp_req=8 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=9 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=10 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=11 ttl=64 time=0.117 ms 64 bytes from 172.16.10.42: icmp_req=12 ttl=64 time=0.114 ms 64 bytes from 172.16.10.42: icmp_req=13 ttl=64 time=0.121 ms ^C --- 172.16.10.42 ping statistics --- 13 packets transmitted, 12 received, 7% packet loss, time 11999ms rtt min/avg/max/mdev = 0.112/0.116/0.122/0.005 ms transoft@ubuntu-mon0:/usr/local/etc/ceph$ transoft@ubuntu-mon0:~$ lsscsi [0:0:0:0] disk ATA WDC WD3200AAKS-7 02.0 /dev/sda transoft@ubuntu-mon0:~$ sudo fdisk -l [sudo] password for transoft: Disk /dev/sda: 320.1 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00006279 Device Boot Start End Blocks Id System /dev/sda1 * 1 4864 39061504 83 Linux /dev/sda2 4864 17264 99607553 5 Extended /dev/sda3 38914 38914 0+ 83 Linux /dev/sda4 17264 38913 173897568+ 83 Linux /dev/sda5 4864 17021 97654784 83 Linux /dev/sda6 17021 17264 1951744 82 Linux swap / Solaris Partition table entries are not in disk order transoft@ubuntu-mon0:~$ transoft@ubuntu-osd0:~$ lsscsi [0:0:0:0] disk ATA ST3320418AS CC45 /dev/sda transoft@ubuntu-osd0:~$ sudo fdisk -l [sudo] password for transoft: Disk /dev/sda: 320.1 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x50000000 Device Boot Start End Blocks Id System /dev/sda1 * 1 6079 48827392 83 Linux /dev/sda2 6079 18237 97655808 83 Linux /dev/sda3 18237 18480 1952768 82 Linux swap / Solaris /dev/sda4 18480 38913 164131680+ 5 Extended /dev/sda5 18480 38913 164131649 83 Linux transoft@ubuntu-osd0:~$ transoft@ubuntu-osd1:~$ lsscsi [0:0:0:0] disk ATA ST3160812AS 3.AD /dev/sda transoft@ubuntu-osd1:~$ sudo fdisk -l [sudo] password for transoft: Disk /dev/sda: 160.0 GB, 160000000000 bytes 255 heads, 63 sectors/track, 19452 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xc45cc45c Device Boot Start End Blocks Id System /dev/sda1 * 1 3648 29295616 83 Linux /dev/sda2 3648 10942 58593280 83 Linux /dev/sda3 10942 11185 1952768 82 Linux swap / Solaris /dev/sda4 11185 19452 66405502 5 Extended /dev/sda5 11185 19452 66405470+ 83 Linux transoft@ubuntu-osd1:~$ -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-11-30 6:19 ` Jeff Wu @ 2010-11-30 17:07 ` Gregory Farnum 2010-12-01 1:35 ` Jeff Wu 0 siblings, 1 reply; 14+ messages in thread From: Gregory Farnum @ 2010-11-30 17:07 UTC (permalink / raw) To: cpwu; +Cc: ceph-devel@vger.kernel.org, Andrew Lv On Mon, Nov 29, 2010 at 10:19 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > Is "40-50MB/s" the speed that it run bench at local btrfs disk ? > not the speed that run bench from client to osd server ? > with this speed ,run bench from client to osd server ,will which get > about 20~25MB/s( 40~50MB /2 )speed ? Data on Ceph is replicated across 2 OSDs (by default; this is configurable). So while figuring out potential performance involves a lot of variables, in a simple case like this where you aren't bounded by network bandwidth you'll find that your read/write performance simply tracks the slower disk. I'd expect your Ceph tests (at least the streaming ones) to run at 40-50MB/s. Given that everything else is okay, I cannot stress enough that running without a journal is going to cause significant performance degradations. I have a hard time believing that it's responsible for 13-second latencies, but it's possible. So how about you set up a journal (it can just be a file or new partition on the drives you're already using) and report back your results after you do that. :) Adding a journal to the OSDs lets them turn all their random writes into streaming ones. -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-11-30 17:07 ` Gregory Farnum @ 2010-12-01 1:35 ` Jeff Wu 2010-12-01 6:59 ` Jeff Wu 0 siblings, 1 reply; 14+ messages in thread From: Jeff Wu @ 2010-12-01 1:35 UTC (permalink / raw) To: Gregory Farnum; +Cc: ceph-devel@vger.kernel.org, Andrew Lv 在 2010-12-01三的 01:07 +0800，Gregory Farnum写道： > On Mon, Nov 29, 2010 at 10:19 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > Is "40-50MB/s" the speed that it run bench at local btrfs disk ? > > not the speed that run bench from client to osd server ? > > with this speed ,run bench from client to osd server ,will which get > > about 20~25MB/s( 40~50MB /2 )speed ? > Data on Ceph is replicated across 2 OSDs (by default; this is > configurable). So while figuring out potential performance involves a > lot of variables, in a simple case like this where you aren't bounded > by network bandwidth you'll find that your read/write performance > simply tracks the slower disk. I'd expect your Ceph tests (at least > the streaming ones) to run at 40-50MB/s. Hi Greg,thank you very much for your quickly reply. > > Given that everything else is okay, I cannot stress enough that > running without a journal is going to cause significant performance > degradations. I have a hard time believing that it's responsible for > 13-second latencies, but it's possible. So how about you set up a > journal (it can just be a file or new partition on the drives you're > already using) and report back your results after you do that. :) I will add journal to ceph.conf to try it . > Adding a journal to the OSDs lets them turn all their random writes > into streaming ones. > -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-01 1:35 ` Jeff Wu @ 2010-12-01 6:59 ` Jeff Wu 2010-12-01 16:05 ` Gregory Farnum 0 siblings, 1 reply; 14+ messages in thread From: Jeff Wu @ 2010-12-01 6:59 UTC (permalink / raw) To: Gregory Farnum; +Cc: ceph-devel@vger.kernel.org, Andrew Lv 在 2010-12-01三的 09:35 +0800，Jeff Wu写道： > > 在 2010-12-01三的 01:07 +0800，Gregory Farnum写道： > > On Mon, Nov 29, 2010 at 10:19 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > > Is "40-50MB/s" the speed that it run bench at local btrfs disk ? > > > not the speed that run bench from client to osd server ? > > > with this speed ,run bench from client to osd server ,will which get > > > about 20~25MB/s( 40~50MB /2 )speed ? > > Data on Ceph is replicated across 2 OSDs (by default; this is > > configurable). So while figuring out potential performance involves a > > lot of variables, in a simple case like this where you aren't bounded > > by network bandwidth you'll find that your read/write performance > > simply tracks the slower disk. I'd expect your Ceph tests (at least > > the streaming ones) to run at 40-50MB/s. > > Hi Greg,thank you very much for your quickly reply. > > > > Given that everything else is okay, I cannot stress enough that > > running without a journal is going to cause significant performance > > degradations. I have a hard time believing that it's responsible for > > 13-second latencies, but it's possible. So how about you set up a > > journal (it can just be a file or new partition on the drives you're > > already using) and report back your results after you do that. :) > > I will add journal to ceph.conf to try it . > > Hi ,greg, With your suggestions, i add the journal config: " osd data = /opt/ceph/data/osd$id osd journal = /home/transoft/data/osd$id/journal filestore journal writeahead = true osd journal size = 10000 " to ceph.conf. the detail ceph.conf attached below. then , run six times for the commad: "$ sudo ceph osd tell 0/1 bench" ,get the results: $ sudo ceph -w osd0 172.16.10.42:6800/17347 1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 29.818194 sec at 28201 KB/sec osd0 172.16.10.42:6800/17347 2 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 30.013058 sec at 34801 KB/sec osd0 172.16.10.42:6800/17347 3 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 30.463511 sec at 30274 KB/sec osd1 172.16.10.65:6800/4845 1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 165.067603 sec at 6329 KB/sec osd1 172.16.10.65:6800/4845 2 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 181.034333 sec at 5782 KB/sec osd1 172.16.10.65:6800/4845 3 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 196.055812 sec at 5334 KB/sec and i also use "dd" to test raw drive, get the logs: 1. OSD0, mkfs.btrfs format /opt $ sudo dd if=/dev/zero of=/opt/dd.img bs=2M count=1024 1024+0 records in 1024+0 records out 2147483648 bytes transfered in 21.4497 secs(100 MB/sec) 2. OSD1 ,mkfs. btrfs format /opt ~$ sudo dd if=/dev/zero of=/opt/dd.img bs=2M count=1024 1024+0 records in 1024+0 records out 2147483648 bytes transfered in 48.2037 secs(44.6 MB/sec) with these logs, OSD1 disk speed might limit the test performance. and i also detect a issue ,take the following steps: $. mckephfs -c ceph.conf -v --mkbtrfs -a $ init-ceph - ceph.conf --btrfs -v -a start then execute: $ init-ceph - ceph.conf --btrfs -v -a stop this command can't stop OSD0 and OSD1 cosd process: OSD0: /usr/local/bin/cosd -i 0 -c ceph.conf OSD1: /usr/local/bin/cosd -i 1 -c ceph.conf then , i create the folder "/var/run/ceph" at OSD0 and OSD1 host manually. execute: $ init-ceph - ceph.conf --btrfs -v -a stop this command can stop OSD0 and OSD1 cosd process: /usr/local/bin/cosd -i 0 -c ceph.conf /usr/local/bin/cosd -i 1 -c ceph.conf Thanks, Jeff.Wu > > > Adding a journal to the OSDs lets them turn all their random writes > > into streaming ones. > > -Greg > ========================================================= transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 0 bench 2010-12-01 10:45:13.670910 mon <- [osd,tell,0,bench] 2010-12-01 10:45:13.671180 mon1 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 0 bench 2010-12-01 10:45:29.350198 mon <- [osd,tell,0,bench] 2010-12-01 10:45:29.350457 mon1 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 0 bench 2010-12-01 10:45:31.000281 mon <- [osd,tell,0,bench] 2010-12-01 10:45:31.000560 mon0 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 1 bench 2010-12-01 10:45:34.860782 mon <- [osd,tell,1,bench] 2010-12-01 10:45:34.861020 mon1 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 1 bench 2010-12-01 10:45:36.760811 mon <- [osd,tell,1,bench] 2010-12-01 10:45:36.761161 mon2 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph osd tell 1 bench 2010-12-01 10:45:37.530714 mon <- [osd,tell,1,bench] 2010-12-01 10:45:37.530968 mon2 -> 'ok' (0) transoft@ubuntu-mon0:/usr/local/etc/ceph$ sudo ceph -w 2010-12-01 10:44:59.450653 pg v13: 528 pgs: 528 active+clean; 12 KB data, 5304 KB used, 219 GB / 219 GB avail 2010-12-01 10:44:59.451365 mds e5: 1/1/1 up {0=up:active}, 1 up:standby 2010-12-01 10:44:59.451387 osd e6: 2 osds: 2 up, 2 in 2010-12-01 10:44:59.451412 log 2010-12-01 10:43:43.044865 mon0 172.16.10.171:6789/0 7 : [INF] mds0 172.16.10.171:6801/2482 up:active 2010-12-01 10:44:59.451440 mon e1: 3 mons at {0=172.16.10.171:6789/0,1=172.16.10.171:6790/0,2=172.16.10.171:6791/0} 2010-12-01 10:46:45.000262 log 2010-12-01 10:45:15.599526 osd0 172.16.10.42:6800/17347 1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 29.818194 sec at 28201 KB/sec 2010-12-01 10:46:45.000262 log 2010-12-01 10:45:46.062142 osd0 172.16.10.42:6800/17347 2 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 30.013058 sec at 34801 KB/sec 2010-12-01 10:46:45.000262 log 2010-12-01 10:46:16.836607 osd0 172.16.10.42:6800/17347 3 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 30.463511 sec at 30274 KB/sec 2010-12-01 10:48:20.042152 pg v14: 528 pgs: 528 active+clean; 32780 KB data, 888 MB used, 218 GB / 219 GB avail 2010-12-01 10:50:50.038298 pg v15: 528 pgs: 528 active+clean; 73740 KB data, 54928 KB used, 219 GB / 219 GB avail 2010-12-01 10:52:15.074470 pg v16: 528 pgs: 528 active+clean; 73740 KB data, 79440 KB used, 219 GB / 219 GB avail 2010-12-01 10:54:55.546098 log 2010-12-01 11:52:34.244851 osd1 172.16.10.65:6800/4845 1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 165.067603 sec at 6329 KB/sec 2010-12-01 10:54:55.546098 log 2010-12-01 11:55:52.010739 osd1 172.16.10.65:6800/4845 2 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 181.034333 sec at 5782 KB/sec 2010-12-01 10:54:55.546098 log 2010-12-01 11:59:09.560115 osd1 172.16.10.65:6800/4845 3 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 196.055812 sec at 5334 KB/sec 2010-12-01 10:55:01.001357 pg v17: 528 pgs: 528 active+clean; 73741 KB data, 1106 MB used, 218 GB / 219 GB avail ============ceph.conf==================== ; ; Sample ceph ceph.conf file. ; ; This file defines cluster membership, the various locations ; that Ceph stores data, and any other runtime options. ; If a 'host' is defined for a daemon, the start/stop script will ; verify that it matches the hostname (or else ignore it). If it is ; not defined, it is assumed that the daemon is intended to start on ; the current host (e.g., in a setup with a startup.conf on each ; node). ; global [global] ; enable secure authentication ; auth supported = cephx keyring = /etc/ceph/keyring.bin ; monitors ; You need at least one. You need at least three if you want to ; tolerate any node failures. Always create an odd number. [mon] mon data = /opt/ceph/data/mon$id ;mon data = /home/transoft/data/mon$id ; logging, for debugging monitor crashes, in order of ; their likelihood of being helpful :) ;debug ms = 20 ;debug mon = 20 ;debug paxos = 20 ;debug auth = 20 [mon0] host = ubuntu-mon0 mon addr = 172.16.10.171:6789 [mon1] host = ubuntu-mon0 mon addr = 172.16.10.171:6790 [mon2] host = ubuntu-mon0 mon addr = 172.16.10.171:6791 ; mds ; You need at least one. Define two to get a standby. [mds] ; where the mds keeps it's secret encryption keys keyring = /etc/ceph/keyring.$name ; mds logging to debug issues. ;debug ms = 20 ;debug mds = 20 [mds.0] host = ubuntu-mon0 [mds.1] host = ubuntu-mon0 ; osd ; You need at least one. Two if you want data to be replicated. ; Define as many as you like. [osd] ; This is where the btrfs volume will be mounted. ;osd data = /opt/ceph/data/osd$id osd class tmp = /var/lib/ceph/tmp ; Ideally, make this a separate disk or partition. A few ; hundred MB should be enough; more if you have fast or many ; disks. You can use a file under the osd data dir if need be ; (e.g. /data/osd$id/journal), but it will be slower than a ; separate disk or partition. ; This is an example of a file-based journal. ;osd journal = /home/transoft/data/osd$id/journal ;filestore journal writeahead = true ; journal size, in megabytes ;osd journal size = 1000 keyring = /etc/ceph/keyring.$name ; osd logging to debug osd issues, in order of likelihood of being ; helpful ;debug ms = 20 ;debug osd = 20 ;debug filestore = 20 ;debug journal = 20 [osd0] host = ubuntu-osd0 osd data = /opt/ceph/data/osd$id osd journal = /home/transoft/data/osd$id/journal filestore journal writeahead = true osd journal size = 10000 ; if 'btrfs devs' is not specified, you're responsible for ; setting up the 'osd data' dir. if it is not btrfs, things ; will behave up until you try to recover from a crash (which ; usually fine for basic testing). ; btrfs devs = /dev/sdx [osd1] host = ubuntu-osd1 osd data = /opt/ceph/data/osd$id osd journal = /home/transoft/data/osd$id/journal filestore journal writeahead = true osd journal size = 10000 ;btrfs devs = /dev/sdy ;[osd2] ;host = zeta ;btrfs devs = /dev/sdx ;[osd3] ;host = eta ;btrfs devs = /dev/sdy -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-01 6:59 ` Jeff Wu @ 2010-12-01 16:05 ` Gregory Farnum 2010-12-02 1:38 ` Jeff Wu 0 siblings, 1 reply; 14+ messages in thread From: Gregory Farnum @ 2010-12-01 16:05 UTC (permalink / raw) To: cpwu; +Cc: ceph-devel@vger.kernel.org, Andrew Lv On Tue, Nov 30, 2010 at 10:57 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > Hi ,greg, > > With your suggestions, i add the journal config: > " > osd data = /opt/ceph/data/osd$id > osd journal = /home/transoft/data/osd$id/journal > filestore journal writeahead = true > osd journal size = 10000 > " > to ceph.conf. the detail ceph.conf attached below. > > then , run six times for the commad: "$ sudo ceph osd tell 0/1 bench" ,get > the results: > > > $ sudo ceph -w > > osd0 172.16.10.42:6800/17347 1 : [INF] bench: wrote 1024 MB in blocks of > 4096 KB in 29.818194 sec at 28201 KB/sec > osd0 172.16.10.42:6800/17347 2 : [INF] bench: wrote 1024 MB in blocks of > 4096 KB in 30.013058 sec at 34801 KB/sec > osd0 172.16.10.42:6800/17347 3 : [INF] bench: wrote 1024 MB in blocks of > 4096 KB in 30.463511 sec at 30274 KB/sec > > osd1 172.16.10.65:6800/4845 1 : [INF] bench: wrote 1024 MB in blocks of 4096 > KB in 165.067603 sec at 6329 KB/sec > osd1 172.16.10.65:6800/4845 2 : [INF] bench: wrote 1024 MB in blocks of 4096 > KB in 181.034333 sec at 5782 KB/sec > osd1 172.16.10.65:6800/4845 3 : [INF] bench: wrote 1024 MB in blocks of 4096 > KB in 196.055812 sec at 5334 KB/sec > > and i also use "dd" to test raw drive, get the logs: > > 1. OSD0, mkfs.btrfs format /opt > > $ sudo dd if=/dev/zero of=/opt/dd.img bs=2M count=1024 > 1024+0 records in > 1024+0 records out > 2147483648 bytes transfered in 21.4497 secs(100 MB/sec) > > 2. OSD1 ,mkfs. btrfs format /opt > > ~$ sudo dd if=/dev/zero of=/opt/dd.img bs=2M count=1024 > 1024+0 records in > 1024+0 records out > 2147483648 bytes transfered in 48.2037 secs(44.6 MB/sec) > > with these logs, OSD1 disk speed might limit the test performance. Yes, it looks to me like your OSD1 disk isn't working properly. Switching it from one streaming write to two streaming writes shouldn't reduce it to 20% of its original performance (~45MB/s to 5MB/s*2). The same change in tasks doesn't impact OSD0's disk at all. > and i also detect a issue ,take the following steps: > > $. mckephfs -c ceph.conf -v --mkbtrfs -a > $ init-ceph - ceph.conf --btrfs -v -a start > then execute: > $ init-ceph - ceph.conf --btrfs -v -a stop > > this command can't stop OSD0 and OSD1 cosd process: > OSD0: > /usr/local/bin/cosd -i 0 -c ceph.conf > OSD1: > /usr/local/bin/cosd -i 1 -c ceph.conf Not sure I understand what you're doing here. Also, it looks like you've got a malformed command there -- you don't specify the "-c" option, just the nonexistent "-" option. ;) -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-01 16:05 ` Gregory Farnum @ 2010-12-02 1:38 ` Jeff Wu 2010-12-02 2:35 ` Gregory Farnum 0 siblings, 1 reply; 14+ messages in thread From: Jeff Wu @ 2010-12-02 1:38 UTC (permalink / raw) To: Gregory Farnum; +Cc: ceph-devel@vger.kernel.org, Andrew Lv 在 2010-12-02四的 00:05 +0800，Gregory Farnum写道： > On Tue, Nov 30, 2010 at 10:57 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > Hi ,greg, > > > > With your suggestions, i add the journal config: > > " > > osd data = /opt/ceph/data/osd$id > > osd journal = /home/transoft/data/osd$id/journal > > filestore journal writeahead = true > > osd journal size = 10000 > > " > > to ceph.conf. the detail ceph.conf attached below. > > > > then , run six times for the commad: "$ sudo ceph osd tell 0/1 bench" ,get > > the results: > > > > > > $ sudo ceph -w > > > > osd0 172.16.10.42:6800/17347 1 : [INF] bench: wrote 1024 MB in blocks of > > 4096 KB in 29.818194 sec at 28201 KB/sec > > osd0 172.16.10.42:6800/17347 2 : [INF] bench: wrote 1024 MB in blocks of > > 4096 KB in 30.013058 sec at 34801 KB/sec > > osd0 172.16.10.42:6800/17347 3 : [INF] bench: wrote 1024 MB in blocks of > > 4096 KB in 30.463511 sec at 30274 KB/sec > > > > osd1 172.16.10.65:6800/4845 1 : [INF] bench: wrote 1024 MB in blocks of 4096 > > KB in 165.067603 sec at 6329 KB/sec > > osd1 172.16.10.65:6800/4845 2 : [INF] bench: wrote 1024 MB in blocks of 4096 > > KB in 181.034333 sec at 5782 KB/sec > > osd1 172.16.10.65:6800/4845 3 : [INF] bench: wrote 1024 MB in blocks of 4096 > > KB in 196.055812 sec at 5334 KB/sec > > > > and i also use "dd" to test raw drive, get the logs: > > > > 1. OSD0, mkfs.btrfs format /opt > > > > $ sudo dd if=/dev/zero of=/opt/dd.img bs=2M count=1024 > > 1024+0 records in > > 1024+0 records out > > 2147483648 bytes transfered in 21.4497 secs(100 MB/sec) > > > > 2. OSD1 ,mkfs. btrfs format /opt > > > > ~$ sudo dd if=/dev/zero of=/opt/dd.img bs=2M count=1024 > > 1024+0 records in > > 1024+0 records out > > 2147483648 bytes transfered in 48.2037 secs(44.6 MB/sec) > > > > with these logs, OSD1 disk speed might limit the test performance. > Yes, it looks to me like your OSD1 disk isn't working properly. > Switching it from one streaming write to two streaming writes > shouldn't reduce it to 20% of its original performance (~45MB/s to > 5MB/s*2). The same change in tasks doesn't impact OSD0's disk at all. > Hi greg, thank you for your detail comments. > > and i also detect a issue ,take the following steps: > > > > $. mckephfs -c ceph.conf -v --mkbtrfs -a > > $ init-ceph - ceph.conf --btrfs -v -a start > > then execute: > > $ init-ceph - ceph.conf --btrfs -v -a stop > > > > this command can't stop OSD0 and OSD1 cosd process: > > OSD0: > > /usr/local/bin/cosd -i 0 -c ceph.conf > > OSD1: > > /usr/local/bin/cosd -i 1 -c ceph.conf > Not sure I understand what you're doing here. Also, it looks like > you've got a malformed command there -- you don't specify the "-c" > option, just the nonexistent "-" option. ;) Oh,Sorry, i mean that , if i don't create folder "/var/run/ceph" at OSD hosts manually. Execute the command : "$init-ceph -c ceph.conf --btrfs -v -a stop " ,which can't auto-kill OSD host cosd process, like this : OSD0 host: $ ps -ef | grep cosd root 13987 1 0 Dec01 ? 00:02:55 /usr/local/bin/cosd -i 0 -c ceph.conf OSD1 host: $ ps -ef | grep cosd root 14028 1 0 Dec01 ? 00:02:53 /usr/local/bin/cosd -i 1 -c ceph.conf I have to execute "kill -9 13987 " and "kill -9 14028" to kill cosd process manually, or ,next time , it will fail to execute "$ init-ceph -c ceph.conf --btrfs -v -a start " command. jeff,wu -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-02 1:38 ` Jeff Wu @ 2010-12-02 2:35 ` Gregory Farnum 2010-12-02 3:22 ` Jeff Wu 0 siblings, 1 reply; 14+ messages in thread From: Gregory Farnum @ 2010-12-02 2:35 UTC (permalink / raw) To: cpwu, Sage Weil; +Cc: ceph-devel@vger.kernel.org, Andrew Lv On Wed, Dec 1, 2010 at 5:38 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: >> > and i also detect a issue ,take the following steps: >> > >> > $. mckephfs -c ceph.conf -v --mkbtrfs -a >> > $ init-ceph - ceph.conf --btrfs -v -a start >> > then execute: >> > $ init-ceph - ceph.conf --btrfs -v -a stop >> > >> > this command can't stop OSD0 and OSD1 cosd process: >> > OSD0: >> > /usr/local/bin/cosd -i 0 -c ceph.conf >> > OSD1: >> > /usr/local/bin/cosd -i 1 -c ceph.conf >> Not sure I understand what you're doing here. Also, it looks like >> you've got a malformed command there -- you don't specify the "-c" >> option, just the nonexistent "-" option. ;) > > Oh,Sorry, i mean that , if i don't create folder "/var/run/ceph" at OSD > hosts manually. Execute the command : "$init-ceph -c ceph.conf --btrfs > -v -a stop " ,which can't auto-kill OSD host cosd process, like this : > OSD0 host: > $ ps -ef | grep cosd > root 13987 1 0 Dec01 ? 00:02:55 /usr/local/bin/cosd -i 0 > -c ceph.conf > OSD1 host: > $ ps -ef | grep cosd > root 14028 1 0 Dec01 ? 00:02:53 /usr/local/bin/cosd -i 1 > -c ceph.conf > > I have to execute "kill -9 13987 " and "kill -9 14028" to kill cosd > process manually, or ,next time , it will fail to execute "$ init-ceph > -c ceph.conf --btrfs -v -a start " command. Ah. I think that /var/run/ceph is where init-ceph stores the PIDs. It ought to be created automatically; if it's not we should fix that. What version of Ceph are you running, and where from? I'd imagine it's being packaged wrong or something. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-02 2:35 ` Gregory Farnum @ 2010-12-02 3:22 ` Jeff Wu 2010-12-02 6:10 ` Sage Weil 0 siblings, 1 reply; 14+ messages in thread From: Jeff Wu @ 2010-12-02 3:22 UTC (permalink / raw) To: Gregory Farnum; +Cc: Sage Weil, ceph-devel@vger.kernel.org, Andrew Lv 在 2010-12-02四的 10:35 +0800，Gregory Farnum写道： > On Wed, Dec 1, 2010 at 5:38 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > >> > and i also detect a issue ,take the following steps: > >> > > >> > $. mckephfs -c ceph.conf -v --mkbtrfs -a > >> > $ init-ceph - ceph.conf --btrfs -v -a start > >> > then execute: > >> > $ init-ceph - ceph.conf --btrfs -v -a stop > >> > > >> > this command can't stop OSD0 and OSD1 cosd process: > >> > OSD0: > >> > /usr/local/bin/cosd -i 0 -c ceph.conf > >> > OSD1: > >> > /usr/local/bin/cosd -i 1 -c ceph.conf > >> Not sure I understand what you're doing here. Also, it looks like > >> you've got a malformed command there -- you don't specify the "-c" > >> option, just the nonexistent "-" option. ;) > > > > Oh,Sorry, i mean that , if i don't create folder "/var/run/ceph" at OSD > > hosts manually. Execute the command : "$init-ceph -c ceph.conf --btrfs > > -v -a stop " ,which can't auto-kill OSD host cosd process, like this : > > OSD0 host: > > $ ps -ef | grep cosd > > root 13987 1 0 Dec01 ? 00:02:55 /usr/local/bin/cosd -i 0 > > -c ceph.conf > > OSD1 host: > > $ ps -ef | grep cosd > > root 14028 1 0 Dec01 ? 00:02:53 /usr/local/bin/cosd -i 1 > > -c ceph.conf > > > > I have to execute "kill -9 13987 " and "kill -9 14028" to kill cosd > > process manually, or ,next time , it will fail to execute "$ init-ceph > > -c ceph.conf --btrfs -v -a start " command. > Ah. I think that /var/run/ceph is where init-ceph stores the PIDs. It > ought to be created automatically; if it's not we should fix that. > What version of Ceph are you running, and where from? I'd imagine it's > being packaged wrong or something. Hi , i download ceph 0.23 from http://ceph.newdream.net/download/ceph-0.23.tar.gz So , maybe , should i add " [global] pid file = /var/run/ceph/$name.pid .............................. " at ceph.conf ? > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-02 3:22 ` Jeff Wu @ 2010-12-02 6:10 ` Sage Weil 2010-12-02 7:31 ` Jeff Wu 0 siblings, 1 reply; 14+ messages in thread From: Sage Weil @ 2010-12-02 6:10 UTC (permalink / raw) To: Jeff Wu; +Cc: Gregory Farnum, ceph-devel@vger.kernel.org, Andrew Lv [-- Attachment #1: Type: TEXT/PLAIN, Size: 2278 bytes --] On Thu, 2 Dec 2010, Jeff Wu wrote: > ÿÿ 2010-12-02ÿÿÿÿ 10:35 +0800ÿÿGregory Farnumÿÿÿÿÿÿ > > On Wed, Dec 1, 2010 at 5:38 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > >> > and i also detect a issue ,take the following steps: > > >> > > > >> > $. mckephfs -c ceph.conf -v --mkbtrfs -a > > >> > $ init-ceph - ceph.conf --btrfs -v -a start > > >> > then execute: > > >> > $ init-ceph - ceph.conf --btrfs -v -a stop > > >> > > > >> > this command can't stop OSD0 and OSD1 cosd process: > > >> > OSD0: > > >> > /usr/local/bin/cosd -i 0 -c ceph.conf > > >> > OSD1: > > >> > /usr/local/bin/cosd -i 1 -c ceph.conf > > >> Not sure I understand what you're doing here. Also, it looks like > > >> you've got a malformed command there -- you don't specify the "-c" > > >> option, just the nonexistent "-" option. ;) > > > > > > Oh,Sorry, i mean that , if i don't create folder "/var/run/ceph" at OSD > > > hosts manually. Execute the command : "$init-ceph -c ceph.conf --btrfs > > > -v -a stop " ,which can't auto-kill OSD host cosd process, like this : > > > OSD0 host: > > > $ ps -ef | grep cosd > > > root 13987 1 0 Dec01 ? 00:02:55 /usr/local/bin/cosd -i 0 > > > -c ceph.conf > > > OSD1 host: > > > $ ps -ef | grep cosd > > > root 14028 1 0 Dec01 ? 00:02:53 /usr/local/bin/cosd -i 1 > > > -c ceph.conf > > > > > > I have to execute "kill -9 13987 " and "kill -9 14028" to kill cosd > > > process manually, or ,next time , it will fail to execute "$ init-ceph > > > -c ceph.conf --btrfs -v -a start " command. > > Ah. I think that /var/run/ceph is where init-ceph stores the PIDs. It > > ought to be created automatically; if it's not we should fix that. > > What version of Ceph are you running, and where from? I'd imagine it's > > being packaged wrong or something. > > Hi , > i download ceph 0.23 from > http://ceph.newdream.net/download/ceph-0.23.tar.gz > > So , maybe , should i add > " > [global] > pid file = /var/run/ceph/$name.pid That's the default, so no... I think the problem is that 'make install' does 'mkdir -p /var/run/ceph'. The .deb and .rpm create the dir, but an install from source does not. There is probably a similar problem with the osd class tmp dir. sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Performence test on ceph v0.23 + EXT4 and Btrfs 2010-12-02 6:10 ` Sage Weil @ 2010-12-02 7:31 ` Jeff Wu 0 siblings, 0 replies; 14+ messages in thread From: Jeff Wu @ 2010-12-02 7:31 UTC (permalink / raw) To: Sage Weil; +Cc: Gregory Farnum, ceph-devel@vger.kernel.org, Andrew Lv 在 2010-12-02四的 14:10 +0800，Sage Weil写道： > On Thu, 2 Dec 2010, Jeff Wu wrote: > > ÿÿ 2010-12-02ÿÿÿÿ 10:35 +0800ÿÿGregory Farnumÿÿÿÿÿÿ > > > On Wed, Dec 1, 2010 at 5:38 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote: > > > >> > and i also detect a issue ,take the following steps: > > > >> > > > > >> > $. mckephfs -c ceph.conf -v --mkbtrfs -a > > > >> > $ init-ceph - ceph.conf --btrfs -v -a start > > > >> > then execute: > > > >> > $ init-ceph - ceph.conf --btrfs -v -a stop > > > >> > > > > >> > this command can't stop OSD0 and OSD1 cosd process: > > > >> > OSD0: > > > >> > /usr/local/bin/cosd -i 0 -c ceph.conf > > > >> > OSD1: > > > >> > /usr/local/bin/cosd -i 1 -c ceph.conf > > > >> Not sure I understand what you're doing here. Also, it looks like > > > >> you've got a malformed command there -- you don't specify the "-c" > > > >> option, just the nonexistent "-" option. ;) > > > > > > > > Oh,Sorry, i mean that , if i don't create folder "/var/run/ceph" at OSD > > > > hosts manually. Execute the command : "$init-ceph -c ceph.conf --btrfs > > > > -v -a stop " ,which can't auto-kill OSD host cosd process, like this : > > > > OSD0 host: > > > > $ ps -ef | grep cosd > > > > root 13987 1 0 Dec01 ? 00:02:55 /usr/local/bin/cosd -i 0 > > > > -c ceph.conf > > > > OSD1 host: > > > > $ ps -ef | grep cosd > > > > root 14028 1 0 Dec01 ? 00:02:53 /usr/local/bin/cosd -i 1 > > > > -c ceph.conf > > > > > > > > I have to execute "kill -9 13987 " and "kill -9 14028" to kill cosd > > > > process manually, or ,next time , it will fail to execute "$ init-ceph > > > > -c ceph.conf --btrfs -v -a start " command. > > > Ah. I think that /var/run/ceph is where init-ceph stores the PIDs. It > > > ought to be created automatically; if it's not we should fix that. > > > What version of Ceph are you running, and where from? I'd imagine it's > > > being packaged wrong or something. > > > > Hi , > > i download ceph 0.23 from > > http://ceph.newdream.net/download/ceph-0.23.tar.gz > > > > So , maybe , should i add > > " > > [global] > > pid file = /var/run/ceph/$name.pid > > That's the default, so no... I think the problem is that 'make install' > does 'mkdir -p /var/run/ceph'. The .deb and .rpm create the dir, but an > install from source does not. There is probably a similar problem with > the osd class tmp dir. > Yes, when i try to use RBD , need also create "/var/lib/ceph/tmp" mamually. I install ceph server with ceph-0.23.tar.gz. Jeff,wu > sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2010-12-02 7:30 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1291001135.1872.106.camel@cephhost>
[not found] ` <1291001398.1872.107.camel@cephhost>
[not found] ` <1291001962.1872.113.camel@cephhost>
[not found] ` <1291002250.1872.116.camel@cephhost>
2010-11-29 3:53 ` Performence test on ceph v0.23 + EXT4 and Btrfs Jeff Wu
2010-11-29 17:07 ` Gregory Farnum
2010-11-30 2:55 ` Jeff Wu
2010-11-30 3:18 ` Gregory Farnum
2010-11-30 6:19 ` Jeff Wu
2010-11-30 17:07 ` Gregory Farnum
2010-12-01 1:35 ` Jeff Wu
2010-12-01 6:59 ` Jeff Wu
2010-12-01 16:05 ` Gregory Farnum
2010-12-02 1:38 ` Jeff Wu
2010-12-02 2:35 ` Gregory Farnum
2010-12-02 3:22 ` Jeff Wu
2010-12-02 6:10 ` Sage Weil
2010-12-02 7:31 ` Jeff Wu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.