Re: Performence test on ceph v0.23 + EXT4 and Btrfs

[Jeff Wu <cpwu@tnsoft.com.cn>]

在 2010-11-30二的 11:18 +0800，Gregory Farnum写道：
> On Mon, Nov 29, 2010 at 6:55 PM, Jeff Wu <cpwu@tnsoft.com.cn> wrote:
> >> Could you run "ceph osd tell * bench", then run "ceph -w", and report
> >> the results? (That'll just run local benchmarking on the OSD to report
> >> the approximate write speed it's capable of.)
> >
> > I run six times for the command: "$ sudo ceph osd tell 0/1 bench "
> > use "$ sudo ceph -w" to get the following results:
> >
> > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.906598 sec
> > at 49775 KB/sec
> > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.023294 sec
> > at 49384 KB/sec
> > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.834682 sec
> > at 51535 KB/sec
> > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 20.792697 sec
> > at 37547 KB/sec
> > osd0  : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.058412 sec
> > at 77191 KB/sec
> > osd1  : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.113612 sec
> > at 47369 KB/sec
> Okay, those are a bit slow but reasonable. Based on these I'd expect
> you to generally manage about 40-50MB/s (since everything is
> replicated; in a 2-disk configuration it'll just be the speed of your
> slowest disk), assuming a properly configured system.

Is "40-50MB/s" the speed that it run bench at local btrfs disk ?
not the speed that run bench from client to osd server ? 
with this speed ,run bench from client to osd server ,will which  get
about 20~25MB/s( 40~50MB /2 )speed ?


> 
> >> You can also run "rados -p data bench 60 write", and then "rados -p
> >> data bench 0 seq" to get a simpler (better understood) performance
> >> test.
> >
> > I run twice for the command: "rados -p data bench 60 write" ,
> > Get the results:
> > $ sudo rados -p data bench 60 write
> > ..........................
> > ..........................
> >
> > Total time run:        76.182225
> > Total writes made:     121
> > Write size:            4194304
> > Bandwidth (MB/sec):    6.219
> >
> > Average Latency:       13.3068
> > Max latency:           23.9986
> > Min latency:           7.01847
> WOAH. That's a lot of latency. Rather more than I'd expect to get just
> from seek times in a non-journaled environment. What's the round-trip
> time to ping the OSDs from your client? Are your disks okay?
> 
Hi,i get the RTT and the disks are okay .

1. RTT:
client ping OSD0:
rtt min/avg/max/mdev = 0.103/0.124/0.143/0.016 ms
client ping OSD1:
rtt min/avg/max/mdev = 0.112/0.116/0.122/0.005 ms

2. disks

1) mon data save in sda4, which formatted with mkfs.btrfs
$lsscsi
[0:0:0:0]    disk    ATA      WDC WD3200AAKS-7 02.0  /dev/sda
$fdisk -l
.............
/dev/sda4           17264       38913   173897568+  83  Linux

2) OSD0 data save in  sda5,which formatted with mkfs.btrfs
~$ lsscsi
[0:0:0:0]    disk    ATA      ST3320418AS      CC45  /dev/sda
$ fdisk -l 
.......................
/dev/sda5           18480       38913   164131649   83  Linux


3) OSD1 data save in  sda5,which formatted with mkfs.btrfs

$ lsscsi
[0:0:0:0]    disk    ATA      ST3160812AS      3.AD  /dev/sda
$ fdisk -l
/dev/sda5           11185       19452    66405470+  83  Linux


more detail logs attached(see below)


Thank you 
jeff.wu

> > but run  "$ sudo rados -p data bench 0 seq"
> > fail to get the results, Maybe it's a bug,ceph version 0.23.
> Oh right, sorry. We put in a quick fix to make the write benchmark
> scale across multiple client writers and forgot to adjust the read
> benchmark. *oops*

thank you !

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

logs:
=================================================================

transoft@ubuntu-mon0:/usr/local/etc/ceph$ ping 172.16.10.65
PING 172.16.10.65 (172.16.10.65) 56(84) bytes of data.
64 bytes from 172.16.10.65: icmp_req=1 ttl=64 time=0.127 ms
64 bytes from 172.16.10.65: icmp_req=2 ttl=64 time=0.121 ms
64 bytes from 172.16.10.65: icmp_req=3 ttl=64 time=0.127 ms
64 bytes from 172.16.10.65: icmp_req=4 ttl=64 time=0.124 ms
64 bytes from 172.16.10.65: icmp_req=5 ttl=64 time=0.128 ms
64 bytes from 172.16.10.65: icmp_req=6 ttl=64 time=0.128 ms
64 bytes from 172.16.10.65: icmp_req=7 ttl=64 time=0.127 ms
64 bytes from 172.16.10.65: icmp_req=8 ttl=64 time=0.126 ms
64 bytes from 172.16.10.65: icmp_req=9 ttl=64 time=0.126 ms
64 bytes from 172.16.10.65: icmp_req=10 ttl=64 time=0.128 ms
64 bytes from 172.16.10.65: icmp_req=11 ttl=64 time=0.120 ms
64 bytes from 172.16.10.65: icmp_req=12 ttl=64 time=0.127 ms
64 bytes from 172.16.10.65: icmp_req=13 ttl=64 time=0.120 ms
64 bytes from 172.16.10.65: icmp_req=14 ttl=64 time=0.125 ms
64 bytes from 172.16.10.65: icmp_req=15 ttl=64 time=0.131 ms
64 bytes from 172.16.10.65: icmp_req=16 ttl=64 time=0.125 ms
64 bytes from 172.16.10.65: icmp_req=17 ttl=64 time=0.124 ms
64 bytes from 172.16.10.65: icmp_req=18 ttl=64 time=0.103 ms
64 bytes from 172.16.10.65: icmp_req=19 ttl=64 time=0.124 ms
64 bytes from 172.16.10.65: icmp_req=20 ttl=64 time=0.125 ms
64 bytes from 172.16.10.65: icmp_req=21 ttl=64 time=0.143 ms
64 bytes from 172.16.10.65: icmp_req=22 ttl=64 time=0.127 ms
64 bytes from 172.16.10.65: icmp_req=23 ttl=64 time=0.117 ms
^C
--- 172.16.10.65 ping statistics ---
23 packets transmitted, 23 received, 0% packet loss, time 21999ms
rtt min/avg/max/mdev = 0.103/0.124/0.143/0.016 ms


transoft@ubuntu-mon0:/usr/local/etc/ceph$ ping 172.16.10.42
PING 172.16.10.42 (172.16.10.42) 56(84) bytes of data.
64 bytes from 172.16.10.42: icmp_req=1 ttl=64 time=0.121 ms
64 bytes from 172.16.10.42: icmp_req=2 ttl=64 time=0.116 ms
64 bytes from 172.16.10.42: icmp_req=3 ttl=64 time=0.122 ms
64 bytes from 172.16.10.42: icmp_req=5 ttl=64 time=0.117 ms
64 bytes from 172.16.10.42: icmp_req=6 ttl=64 time=0.112 ms
64 bytes from 172.16.10.42: icmp_req=7 ttl=64 time=0.117 ms
64 bytes from 172.16.10.42: icmp_req=8 ttl=64 time=0.112 ms
64 bytes from 172.16.10.42: icmp_req=9 ttl=64 time=0.112 ms
64 bytes from 172.16.10.42: icmp_req=10 ttl=64 time=0.112 ms
64 bytes from 172.16.10.42: icmp_req=11 ttl=64 time=0.117 ms
64 bytes from 172.16.10.42: icmp_req=12 ttl=64 time=0.114 ms
64 bytes from 172.16.10.42: icmp_req=13 ttl=64 time=0.121 ms
^C
--- 172.16.10.42 ping statistics ---
13 packets transmitted, 12 received, 7% packet loss, time 11999ms
rtt min/avg/max/mdev = 0.112/0.116/0.122/0.005 ms
transoft@ubuntu-mon0:/usr/local/etc/ceph$ 


transoft@ubuntu-mon0:~$ lsscsi
[0:0:0:0]    disk    ATA      WDC WD3200AAKS-7 02.0  /dev/sda
transoft@ubuntu-mon0:~$ sudo fdisk -l
[sudo] password for transoft: 

Disk /dev/sda: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00006279

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        4864    39061504   83  Linux
/dev/sda2            4864       17264    99607553    5  Extended
/dev/sda3           38914       38914           0+  83  Linux
/dev/sda4           17264       38913   173897568+  83  Linux
/dev/sda5            4864       17021    97654784   83  Linux
/dev/sda6           17021       17264     1951744   82  Linux swap /
Solaris

Partition table entries are not in disk order
transoft@ubuntu-mon0:~$ 



transoft@ubuntu-osd0:~$ lsscsi
[0:0:0:0]    disk    ATA      ST3320418AS      CC45  /dev/sda
transoft@ubuntu-osd0:~$ sudo fdisk -l
[sudo] password for transoft: 

Disk /dev/sda: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x50000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        6079    48827392   83  Linux
/dev/sda2            6079       18237    97655808   83  Linux
/dev/sda3           18237       18480     1952768   82  Linux swap /
Solaris
/dev/sda4           18480       38913   164131680+   5  Extended
/dev/sda5           18480       38913   164131649   83  Linux
transoft@ubuntu-osd0:~$ 



transoft@ubuntu-osd1:~$ lsscsi
[0:0:0:0]    disk    ATA      ST3160812AS      3.AD  /dev/sda
transoft@ubuntu-osd1:~$ sudo fdisk -l
[sudo] password for transoft: 

Disk /dev/sda: 160.0 GB, 160000000000 bytes
255 heads, 63 sectors/track, 19452 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc45cc45c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3648    29295616   83  Linux
/dev/sda2            3648       10942    58593280   83  Linux
/dev/sda3           10942       11185     1952768   82  Linux swap /
Solaris
/dev/sda4           11185       19452    66405502    5  Extended
/dev/sda5           11185       19452    66405470+  83  Linux
transoft@ubuntu-osd1:~$ 

























--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html