simple performance tests

simple performance tests

[Thomas Mueller]

hi

i've done some performance tests on a small system.

HW:
* Supermicro X7SPA-HF (with IMPI and KVMoverIP integrated))
* Atom D510 1.6GHz DualCore and HT
* WD RE3 500GB 7200umin disks
* 4GB RAM

SW:
* Debian 5.0 "Lenny"
* Kernel 2.6.32 (backports.org)
* ceph/unstable
* ceph-client-standalone/unstable-backport

values are MB/s. all tests are local.

dbench 1 results:
* 1 disk, btrfs: 115
* 2 disks, btrfs data/metadata raid0: 115
* 1 osd/disk, 1 mds, 1 mon: 7
* 2 osd/disk, 1 mds, 1mon: 6.2117
* 1 disk, btrfs, samba: 14.02
* 2 disks, btrfs data/metadata raid0, samba: 13.781

dbench 10 results:
* 1 disk, btrfs: 271.724
* 2 disks, btrfs data/metadata raid0: 270.043
* 1 osd/disk, 1 mds, 1 mon: -
* 2 osd/disk, 1 mds, 1mon: 12.1862
* 1 disk, btrfs, samba: 20.22
* 2 disks, btfs data/metadata raid0, samba: 20.26

dd bs=1M count=8192 if=/dev/zero
* 1 disk, btrfs: 126
* 2 disks, btrfs data/metadata raid0:  271
* 1 osd/disk, 1 mds, 1 mon: -
* 2 osd/disk, 1 mds, 1mon: 51.2
* 1 disk, btrfs, samba: 95.2
* 2 disks, btrfs data/metadata raid0, samba: 85.9

will add a AOC-USAS-H8iR (LSI) RAID card to the mix if  i get the cables 
needed.

i'm a bit suprised about the low rates of ceph. i'll try to to the same 
tests on a more powerfull machine. 

- Thomas

Re: simple performance tests

[Wido den Hollander]

Hi,

You were running all the Ceph components on the same host? E.g, mon, mds
and osd on the same machine?

Imho that is not the way to test Ceph.

I've done some benchmarking myself with 6 physical machines for OSD's
(different hw in each machine) and i was seeing about 30 ~ 40MB/sec over
a Gigabit network.

Ofcourse, that could be a lot better, but i think they are focussing on
stability right now rather then performance.

Also, a local FS will be faster than a network filesystem, since the
local filesystem doesn't have to take multiple clients into the
equasion.

Try benchmarking Ceph against NFS and you will start seeing different
results.

-- 
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Wed, 2010-06-16 at 09:15 +0000, Thomas Mueller wrote:
> hi
> 
> i've done some performance tests on a small system.
> 
> HW:
> * Supermicro X7SPA-HF (with IMPI and KVMoverIP integrated))
> * Atom D510 1.6GHz DualCore and HT
> * WD RE3 500GB 7200umin disks
> * 4GB RAM
> 
> SW:
> * Debian 5.0 "Lenny"
> * Kernel 2.6.32 (backports.org)
> * ceph/unstable
> * ceph-client-standalone/unstable-backport
> 
> values are MB/s. all tests are local.
> 
> dbench 1 results:
> * 1 disk, btrfs: 115
> * 2 disks, btrfs data/metadata raid0: 115
> * 1 osd/disk, 1 mds, 1 mon: 7
> * 2 osd/disk, 1 mds, 1mon: 6.2117
> * 1 disk, btrfs, samba: 14.02
> * 2 disks, btrfs data/metadata raid0, samba: 13.781
> 
> dbench 10 results:
> * 1 disk, btrfs: 271.724
> * 2 disks, btrfs data/metadata raid0: 270.043
> * 1 osd/disk, 1 mds, 1 mon: -
> * 2 osd/disk, 1 mds, 1mon: 12.1862
> * 1 disk, btrfs, samba: 20.22
> * 2 disks, btfs data/metadata raid0, samba: 20.26
> 
> dd bs=1M count=8192 if=/dev/zero
> * 1 disk, btrfs: 126
> * 2 disks, btrfs data/metadata raid0:  271
> * 1 osd/disk, 1 mds, 1 mon: -
> * 2 osd/disk, 1 mds, 1mon: 51.2
> * 1 disk, btrfs, samba: 95.2
> * 2 disks, btrfs data/metadata raid0, samba: 85.9
> 
> will add a AOC-USAS-H8iR (LSI) RAID card to the mix if  i get the cables 
> needed.
> 
> i'm a bit suprised about the low rates of ceph. i'll try to to the same 
> tests on a more powerfull machine. 
> 
> - Thomas
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: simple performance tests

[Thomas Mueller]

On 16.06.2010 11:49, Wido den Hollander wrote:
> Hi,
>
> You were running all the Ceph components on the same host? E.g, mon, mds
> and osd on the same machine?

yes, all on one.



>
> Imho that is not the way to test Ceph.

of course this is not the inteded use uf ceph. i wanted to know the 
performance penalty between local access and access with ceph. didn't 
expect it to be that large (it's also huge with samba - so not a ceph 
specific "problem").

i'll do more testing in the next weeks in a more ceph-apropriate 
envirnoment. maybe also comparing to glusterfs.

>
> I've done some benchmarking myself with 6 physical machines for OSD's
> (different hw in each machine) and i was seeing about 30 ~ 40MB/sec over
> a Gigabit network.

what was the workload involved with your benchmarking? if these are 
large writes or reads then 30-40mb/s are IMHO not what i would expect of 
6xGigE (6x80MB/s). I would expect the GigE port of the client to be the 
bottleneck.


> Try benchmarking Ceph against NFS and you will start seeing different
> results.
>

thats why i've included samba in the mix. i wanted nfs but it crashed 
the machine (heard about nfs problems in 2.6.32). samba wins all the 
"competitions" in samba vs. ceph.

- Thomas

Re: simple performance tests

[Gregory Farnum]

How's your CPU utilization on that processor? It'd be good if all
those daemons ran fine on an Atom but I'm not sure?
What's your config file look like? Journaling enabled?

On Wed, Jun 16, 2010 at 2:15 AM, Thomas Mueller <thomas@chaschperli.ch> wrote:
> hi
>
> i've done some performance tests on a small system.
>
> HW:
> * Supermicro X7SPA-HF (with IMPI and KVMoverIP integrated))
> * Atom D510 1.6GHz DualCore and HT
> * WD RE3 500GB 7200umin disks
> * 4GB RAM
>
> SW:
> * Debian 5.0 "Lenny"
> * Kernel 2.6.32 (backports.org)
> * ceph/unstable
> * ceph-client-standalone/unstable-backport
>
> values are MB/s. all tests are local.
>
> dbench 1 results:
> * 1 disk, btrfs: 115
> * 2 disks, btrfs data/metadata raid0: 115
> * 1 osd/disk, 1 mds, 1 mon: 7
> * 2 osd/disk, 1 mds, 1mon: 6.2117
> * 1 disk, btrfs, samba: 14.02
> * 2 disks, btrfs data/metadata raid0, samba: 13.781
>
> dbench 10 results:
> * 1 disk, btrfs: 271.724
> * 2 disks, btrfs data/metadata raid0: 270.043
> * 1 osd/disk, 1 mds, 1 mon: -
> * 2 osd/disk, 1 mds, 1mon: 12.1862
> * 1 disk, btrfs, samba: 20.22
> * 2 disks, btfs data/metadata raid0, samba: 20.26
>
> dd bs=1M count=8192 if=/dev/zero
> * 1 disk, btrfs: 126
> * 2 disks, btrfs data/metadata raid0:  271
> * 1 osd/disk, 1 mds, 1 mon: -
> * 2 osd/disk, 1 mds, 1mon: 51.2
> * 1 disk, btrfs, samba: 95.2
> * 2 disks, btrfs data/metadata raid0, samba: 85.9
>
> will add a AOC-USAS-H8iR (LSI) RAID card to the mix if  i get the cables
> needed.
>
> i'm a bit suprised about the low rates of ceph. i'll try to to the same
> tests on a more powerfull machine.
>
> - Thomas
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: simple performance tests

[K. Richard Pixley]

Hi.  I've been running some similar tests recently, (on ubuntu, but still).

Thomas Mueller wrote:
> dbench 1 results:
> * 1 disk, btrfs: 115
> * 2 disks, btrfs data/metadata raid0: 115
>   
This is suspicious.

I'm also not sure whether you mean that you're running btrfs on an md or 
lvm stripe or using multiple device btrfs file systems.

My numbers are showing that btrfs multidevice support may not be very 
good right now but I get pretty good results from lvm stripes.  The 
hardware I'm testing now is the first hardware I've tried, (of prolly a 
couple dozen over the years), where the controller can actually keep all 
disks spinning.  Most cannot.

Try building an lvm stripe on your two disks and testing dd directly to 
the stripe.

--rich

Re: simple performance tests

[Thomas Mueller]

Am Wed, 16 Jun 2010 07:45:47 -0700 schrieb Gregory Farnum:

> How's your CPU utilization on that processor? It'd be good if all those
> daemons ran fine on an Atom but I'm not sure?

I'll do some load monitoring and post the results.

> What's your config file
> look like? Journaling enabled?

very simple one derived from sample-ceph.conf:


[global]


[mon]
        mon data = /data/mon$id

[mon0]
        host = ceph
        mon addr = 192.168.236.162:6789

[mds]
        keyring = /data/keyring.$name

[mds.ceph]
        host = ceph

[osd]
        osd data = /data/osd$id
        osd journal = /data/osd$id/journal
        osd journal size = 100

[osd0]
        host = ceph
        btrfs devs = /dev/sda2

[osd1]
        host = ceph
        btrfs devs = /dev/sdb2

Re: simple performance tests

[Thomas Mueller]

Am Thu, 17 Jun 2010 08:53:07 +0000 schrieb Thomas Mueller:

> Am Wed, 16 Jun 2010 07:45:47 -0700 schrieb Gregory Farnum:
> 
>> How's your CPU utilization on that processor? It'd be good if all those
>> daemons ran fine on an Atom but I'm not sure?
> 
> I'll do some load monitoring and post the results.

I getting used to atop - nice tool to have a summary of what's going on.

it showed that cmds ate 50% of the cpu power.

i've started the MDS on another computer. with 3 osds (1 disk per osd) 
and journal on a partition (same disk as osd) i get now 47 mb/s in dbench 
10 test (ceph data/metadata size 1) and with "size 2" 31.3649mb/s

dd bs=1M count=8192 gives me 107mb/s ("size 1")

so now ceph outperforms samba. :)

- Thomas