xio messenger prelim benchmark

xio messenger prelim benchmark

[Vu Pham]

[-- Attachment #1: Type: text/plain, Size: 3290 bytes --]

Hi,

I would like to share some benchmarking numbers on xio messenger and 
simple messenger

HW/SW configuration:
---------------------
. 1 32-core Xeon E5-2697V3 2.6G (Haswell) node, 64GB of memory.
. Hyperthreading is ON/enabled, 64 cores
. Mellanox ConnectX3-EN 40Gb/s HCAs, fw- 2.33.5000
. Mellanox SX1012 40Gb/s switch EN.
. Ubuntu 14.04 LTS stock kernel
. MLNX_OFED_LINUX-2.4-1.0.0 sw package
. Accelio master branch (tag v-1.3)
. Ceph master (Jan-29) + pr #3544 (xio spread portals).
. Use ramdisks and filestore backend
. Use fio_rbd on user rbd as client

1 OSD, 1 client node
-------------------------
a. 1 rbd image
xio messenger:
. ~9100 iops (4K random write, 6 cores used on osd node, numjobs=1, 
iodepth=64)
. ~21k iops (4K random read, 4 cores used, numjobs=1, iodepth=32)
. ~121k iops (4K random read, 15 cores used, numjobs=8, iodepth=32)
. ~520MB/s (256K random write, 3 cores used, numjobs=1, iodepth=64)
. ~3140MB/s (256K random read, 4 cores used, numjobs=1, iodepth=32)
. ~4330MB/s (256K random read, 6 cores used, numjobs=8)
simple messenger:
. ~8500 iops (4K random write, 7 cores used)
. ~20k iops (4K random read, 5 cores used)
. ~105k iops (4K random read, 20 cores used, numjobs=8, iopdepth=32)
. ~450MB/s (256K random write, 3 cores used)
. ~1140MB/s (256K random read, 3 cores used)
. ~4330MB/s (256K random read, 8 cores used, numjobs=8)


b. 2 rbd images on two separated pools, 2 fio_rbd instances
xio messenger:
. ~9100 iops (4K random write, 6 cores used on osd node, each fio_rbd 
instance has numjobs=1, iodepth=64)
. ~155k iops (4K random read, 19 cores used, each fio_rbd instance has 
numjobs=8, iodepth=32)
. ~4225MB/s (256K random read, 6 cores used, each fio_rbd instance has 
numjobs=1, iodepth=32)
. ~4330MB/s (256K random read, 8 cores used, each fio_rbd instance has 
numjobs=8, iodepth=32)

simple messenger:
. ~7800 iops (4K random write, 7 cores used on osd node, each fio_rbd 
instance has numjobs=1, iodepth=64)
. ~125k iops (4K random read, 25 cores used, each fio_rbd instance has 
numjobs=8, iodepth=32)
. ~2068MB/s (256K random read, 4 cores used, each fio_rbd instance has 
numjobs=1, iodepth=32)
. ~4330MB/s (256K random read, 11 cores used, each fio_rbd instance has 
numjobs=8, iodepth=32)

2 OSDs, 1 client node, 4 rbd images on 4 separated pools
--------------------------------------------------------------------
4K random read: xio messenger max at ~272k iops, simple messenger max at 
~170k iops


4 OSDs, 1 client node, 4 rbd images on 4 separated pools
---------------------------------------------------------------------
4K random read: xio messenger max at ~355k iops, simple messenger max at 
~204k iops


8 OSDs, 1 client node, 4 rbd images on 4 separated pools
---------------------------------------------------------------------
4K random read: xio messenger max at ~355K iops, simple messenger max at 
~225k iops


I attach here the ceph configuration files that I used.
Please note that I enable flow-control and turn off header_crc & 
data_crc for both xio & simple.

Are the simple messenger numbers looking reasonable and in the ballpark?
Please share your numbers and configuration if you have higher numbers.

thanks,
-vu

[-- Attachment #2: ceph_perf_simple_1osd.conf --]
[-- Type: application/octet-stream, Size: 2465 bytes --]

[global]

	admin_socket = /ceph-test/var/run/ceph/ceph-$name.$id.asok

	; no secure authentication
	auth_supported = none
	auth_service_required = none
	auth_client_required = none
	auth_cluster_required = none

	; allow to open a lot of files
	max_open_files = 131072

	; setup logging
	log_file = /ceph-test/var/log/ceph/$name.log
	pid_file = /ceph-test/var/run/ceph/$name.pid

	filestore_xattr_use_omap = 1

	osd_pool_default_size = 1
	osd_pool_default_min_size = 1

	; turn off debugs
	; a
	debug_auth = 0/0
	debug_asok = 0/0

	; b
	debug_buffer = 0/0

	; c
	debug_client = 0/0
	debug_context = 0/0
	debug_crush = 0/0
	debug_crypto = 0/0

	; f
	debug_filer = 0/0
	debug_filestore = 0/0
	debug_finisher = 0/0

	; h
	debug_heartbeatmap = 0/0

	; j
	debug_journal = 0/0
	debug_journaler = 0/0

	; l
	debug_lockdep = 0/0

	; m
	debug_monclient = 0/0
	debug_mon = 0/0
	debug_monc = 0/0
	debug_ms = 0/0

	; o
	debug_objclass = 0/0
	debug_objecter = 0/0
	debug_objectcacher = 0/0
	debug_optracker = 0/0
	debug_osd = 0/0

	; p
	debug_paxos = 0/0
	debug_perfcounter = 0/0

	; r
	debug_rados = 0/0
	debug_rbd = 0/0
	debug_rgw = 0/0

	; t
	debug_timer = 0/0
	debug_tp = 0/0
	debug_throttle = 0/0

	; messenger type
	ms_type = simple

	ms_tcp_nodelay = true
        ms_crc_header = false
        ms_crc_data = false

	osd_op_threads = 2
	filesore_op_threads = 3
	filestore_fd_cache_size = 64
	filestore_fd_cache_shards = 16
	osd_op_num_threads_per_shard = 1
	osd_op_num_shards = 25

	throttle_perf_counter = false
	ms_dispatch_throttle_bytes = 0

	rbd_cache = false

[mon]
	mon_data = /ceph-test/ceph-data/mon.$id

[mon.0]
	host = vlab-017
	mon_addr = 12.20.1.117:16789
	;user = root

[mds]
	; where the mds keeps its secret encryption keys
	keyring = /ceph-test/ceph-data/keyring.$name

	cluster_addr = 12.20.1.117:26789
	public_addr = 12.20.1.117:36789

	objecter_timeout = 10
	mds_reconnect_timeout = 5
	mds_beacon_interval = 2

[mds.0]
	host = vlab-017

[osd]
	osd_client_message_size_cap = 0
	osd_client_message_cap = 0
	osd_enable_op_tracker = false

	osd_data = /ceph-test/ceph-data/osd.$id
	osd_journal = /ceph-test/ceph-data/osd.$id/journal
	osd_journal_size = 256
	osd_scrub_load_threshold = 2.5
	;osd_objectstore = memstore
	osd_mkfs_type = xfs
	osd_mount_options_xfs = rw,noatime

	osd_class_dir = /opt/ceph/lib/rados-classes
	ms_bind_port_min = 3000

[osd.0]
	host = vlab-017
	;user = root

	devs = /dev/ram0
	ms_bind_port_min = 7100
	ms_bind_port_max = 7200


[-- Attachment #3: ceph_perf_xio_1osd.conf --]
[-- Type: application/octet-stream, Size: 2765 bytes --]

[global]

	admin_socket = /ceph-test/var/run/ceph/ceph-$name.$id.asok

	; no secure authentication
	auth_supported = none
	auth_service_required = none
	auth_client_required = none
	auth_cluster_required = none

	; allow to open a lot of files
	max_open_files = 131072

	; setup logging
	log_file = /ceph-test/var/log/ceph/$name.log
	pid_file = /ceph-test/var/run/ceph/$name.pid

        ; default datacrc & headercrc is true
        ms_crc_header = false
        ms_crc_data = false

	filestore_xattr_use_omap = 1

	osd_pool_default_size = 1
	osd_pool_default_min_size = 1

	; turn off debugs
	; a
	debug_auth = 0/0
	debug_asok = 0/0

	; b
	debug_buffer = 0/0

	; c
	debug_client = 0/0
	debug_context = 0/0
	debug_crush = 0/0
	debug_crypto = 0/0

	; f
	debug_filer = 0/0
	debug_filestore = 0/0
	debug_finisher = 0/0

	; h
	debug_heartbeatmap = 0/0

	; j
	debug_journal = 0/0
	debug_journaler = 0/0

	; l
	debug_lockdep = 0/0

	; m
	debug_monclient = 0/0
	debug_mon = 0/0
	debug_monc = 0/0
	debug_ms = 0/0

	; o
	debug_objclass = 0/0
	debug_objecter = 0/0
	debug_objectcacher = 0/0
	debug_optracker = 0/0
	debug_osd = 0/0

	; p
	debug_paxos = 0/0
	debug_perfcounter = 0/0

	; r
	debug_rados = 0/0
	debug_rbd = 0/0
	debug_rgw = 0/0

	; t
	debug_timer = 0/0
	debug_tp = 0/0
	debug_throttle = 0/0

	; x
	debug_xio = 2

	; rdma setting
	rdma_local = 12.20.1.117
	enable experimental unrecoverable data corrupting features = ms-type-xio
	ms_type = xio
	;xio_queue_depth = 128
	xio_mp_max_64 = 262144
	xio_mp_max_256 = 262144
	xio_mp_max_1k = 262144
	xio_mp_max_page = 131072
	xio_portal_threads = 8

	osd_op_threads = 2
	filesore_op_threads = 3
	filestore_fd_cache_size = 64
	filestore_fd_cache_shards = 16

	osd_op_num_threads_per_shard = 1
	osd_op_num_shards = 25

	throttle_perf_counter = false
	ms_dispatch_throttle_bytes = 0

	rbd_cache = false

[osd]
	osd_client_message_size_cap = 0
	osd_client_message_cap = 0
	osd_enable_op_tracker = false

	osd_data = /ceph-test/ceph-data/osd.$id
	osd_journal = /ceph-test/ceph-data/osd.$id/journal
	osd_journal_size = 256
	osd_scrub_load_threshold = 2.5
	;osd_objectstore = memstore
	osd_mkfs_type = xfs
	osd_mount_options_xfs = rw,noatime

	osd_class_dir = /opt/ceph/lib/rados-classes
	ms_bind_port_min = 3000

[osd.0]
	host = vlab-017
	;user = root

	devs = /dev/ram0
	ms_bind_port_min = 7100
	ms_bind_port_max = 7400

[mon]
	mon_data = /ceph-test/ceph-data/mon.$id

[mon.0]
	host = vlab-017
	mon_addr = 12.20.1.117:16789
	;user = root

[mds]
	; where the mds keeps its secret encryption keys
	keyring = /ceph-test/ceph-data/keyring.$name

	cluster_addr = 12.20.1.117:26789
	public_addr = 12.20.1.117:36789

	objecter_timeout = 10
	mds_reconnect_timeout = 5
	mds_beacon_interval = 2

[mds.0]
	host = vlab-017
	;user = root

Re: xio messenger prelim benchmark

[Mark Nelson]

On 02/04/2015 02:08 PM, Vu Pham wrote:
> Hi,
>
> I would like to share some benchmarking numbers on xio messenger and
> simple messenger
>
> HW/SW configuration:
> ---------------------
> . 1 32-core Xeon E5-2697V3 2.6G (Haswell) node, 64GB of memory.
> . Hyperthreading is ON/enabled, 64 cores
> . Mellanox ConnectX3-EN 40Gb/s HCAs, fw- 2.33.5000
> . Mellanox SX1012 40Gb/s switch EN.
> . Ubuntu 14.04 LTS stock kernel
> . MLNX_OFED_LINUX-2.4-1.0.0 sw package
> . Accelio master branch (tag v-1.3)
> . Ceph master (Jan-29) + pr #3544 (xio spread portals).
> . Use ramdisks and filestore backend
> . Use fio_rbd on user rbd as client
>
> 1 OSD, 1 client node
> -------------------------
> a. 1 rbd image
> xio messenger:
> . ~9100 iops (4K random write, 6 cores used on osd node, numjobs=1,
> iodepth=64)
> . ~21k iops (4K random read, 4 cores used, numjobs=1, iodepth=32)
> . ~121k iops (4K random read, 15 cores used, numjobs=8, iodepth=32)
> . ~520MB/s (256K random write, 3 cores used, numjobs=1, iodepth=64)
> . ~3140MB/s (256K random read, 4 cores used, numjobs=1, iodepth=32)
> . ~4330MB/s (256K random read, 6 cores used, numjobs=8)
> simple messenger:
> . ~8500 iops (4K random write, 7 cores used)
> . ~20k iops (4K random read, 5 cores used)
> . ~105k iops (4K random read, 20 cores used, numjobs=8, iopdepth=32)
> . ~450MB/s (256K random write, 3 cores used)
> . ~1140MB/s (256K random read, 3 cores used)
> . ~4330MB/s (256K random read, 8 cores used, numjobs=8)
>
>
> b. 2 rbd images on two separated pools, 2 fio_rbd instances
> xio messenger:
> . ~9100 iops (4K random write, 6 cores used on osd node, each fio_rbd
> instance has numjobs=1, iodepth=64)
> . ~155k iops (4K random read, 19 cores used, each fio_rbd instance has
> numjobs=8, iodepth=32)
> . ~4225MB/s (256K random read, 6 cores used, each fio_rbd instance has
> numjobs=1, iodepth=32)
> . ~4330MB/s (256K random read, 8 cores used, each fio_rbd instance has
> numjobs=8, iodepth=32)
>
> simple messenger:
> . ~7800 iops (4K random write, 7 cores used on osd node, each fio_rbd
> instance has numjobs=1, iodepth=64)
> . ~125k iops (4K random read, 25 cores used, each fio_rbd instance has
> numjobs=8, iodepth=32)
> . ~2068MB/s (256K random read, 4 cores used, each fio_rbd instance has
> numjobs=1, iodepth=32)
> . ~4330MB/s (256K random read, 11 cores used, each fio_rbd instance has
> numjobs=8, iodepth=32)
>
> 2 OSDs, 1 client node, 4 rbd images on 4 separated pools
> --------------------------------------------------------------------
> 4K random read: xio messenger max at ~272k iops, simple messenger max at
> ~170k iops
>
>
> 4 OSDs, 1 client node, 4 rbd images on 4 separated pools
> ---------------------------------------------------------------------
> 4K random read: xio messenger max at ~355k iops, simple messenger max at
> ~204k iops
>
>
> 8 OSDs, 1 client node, 4 rbd images on 4 separated pools
> ---------------------------------------------------------------------
> 4K random read: xio messenger max at ~355K iops, simple messenger max at
> ~225k iops
>
>
> I attach here the ceph configuration files that I used.
> Please note that I enable flow-control and turn off header_crc &
> data_crc for both xio & simple.
>
> Are the simple messenger numbers looking reasonable and in the ballpark?
> Please share your numbers and configuration if you have higher numbers.

These numbers look great Vu.  You may want to look at the auth numbers I 
just posted as you are getting sufficiently high enough IOPs that doing 
things like disabling in-memory dubgging, disabling auth, and testing on 
RHEL may make a difference for you.

>
> thanks,
> -vu
>

Re: xio messenger prelim benchmark

[Vu Pham]

>>>Yes, xio messenger which is implemented over Accelio can run over 
>>>rdma
>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>>>enabled xio messenger / Accelio-tcp yet.
>
>Oh, ok, Great !
>
>
>>>xio messenger is currently working with user mode clients. We only
>>>validated/tested user mode rbd client.
>>>Sandisk are working on krbd over kAccelio implementation. As of last
>>>week, SanDisk have basic I/Os working.
>>>Hopefully krbd/kAccelio will be available soon.
>
>Great, so we can expect even better results :)
>
>
>BTW, do you have some client side cpu usage benchmark ?
>
>
Sorry, I did not collect the cpu usage @client

BTW since you will use Mellanox sx1012 EN switch for your benchmark, you 
can take advantage of 56Gbe/s over 40Gbe/s with a license upgrade.

With my minimum setup of single node cluster (single OSD, 2 
pools/images) + 1 client node with 16 fio_rbd client streams, we already 
see the benefit of big pipe.

             ib_send_bw     64k read (ceph)      256k read(ceph)
---------------------------------------------------------------------------------------
40Gbe: 4350 MB/s       4300 MB/s             4350 MB/s
56Gbe: 5100 MB/s       4850 MB/s             5000 MB/s

-vu

Re: xio messenger prelim benchmark

[Vu Pham]

>
>>>>Yes, xio messenger which is implemented over Accelio can run over
>>>>rdma
>>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>>>>enabled xio messenger / Accelio-tcp yet.
>>
>>Oh, ok, Great !
>>
>>
>>>>xio messenger is currently working with user mode clients. We only
>>>>validated/tested user mode rbd client.
>>>>Sandisk are working on krbd over kAccelio implementation. As of last
>>>>week, SanDisk have basic I/Os working.
>>>>Hopefully krbd/kAccelio will be available soon.
>>
>>Great, so we can expect even better results :)
>>
>>
>>BTW, do you have some client side cpu usage benchmark ?
>>
>>
>Sorry, I did not collect the cpu usage @client
>

I just rerun the test with 2 OSDs, 4 fio_clients on 4 separated 
pools/images, each client uses numjobs=1 for random write test and 
numjobs=8 for random read test. I also measured the cpu usage@client 
this time

xio messenger:
--------------------
. 4K random write: ~26k iops (10 cores used at OSD, 2 cores used at 
client)
. 4K random read: ~332k iops (34 cores used at OSD, 4 cores used at 
client)
. 256K random write: ~1464 MB/s (6 cores used at OSD, 2 cores used at 
client)
. 256K random read: ~4300 MB/s (6 cores used at OSD, 5 cores used at 
client)

simple:
----------
. 4K random write: ~25k iops (10 cores used at OSD, 3 cores used at 
client)
. 4K random read: ~230k iops (38 cores used at OSD, 30 cores used at 
client)
. 256K random write: ~1475 MB/s (6 cores used at OSD, 2 cores used at 
client)
. 256K random read: ~4300 MB/s (7 cores used at OSD, 7 cores used at 
client)

-vu

Re: xio messenger prelim benchmark

[Vu Pham]

On 2/6/2015 10:41:15 AM, "Vu Pham" <vuhuong@mellanox.com> wrote:

>
>>>>Yes, xio messenger which is implemented over Accelio can run over
>>>>rdma
>>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>>>>enabled xio messenger / Accelio-tcp yet.
>>
>>Oh, ok, Great !
>>
>>
>>>>xio messenger is currently working with user mode clients. We only
>>>>validated/tested user mode rbd client.
>>>>Sandisk are working on krbd over kAccelio implementation. As of last
>>>>week, SanDisk have basic I/Os working.
>>>>Hopefully krbd/kAccelio will be available soon.
>>
>>Great, so we can expect even better results :)
>>
>>
>>BTW, do you have some client side cpu usage benchmark ?
>>
>>
>Sorry, I did not collect the cpu usage @client
>
>BTW since you will use Mellanox sx1012 EN switch for your benchmark, 
>you
>can take advantage of 56Gbe/s over 40Gbe/s with a license upgrade.
>
>With my minimum setup of single node cluster (single OSD, 2
>pools/images) + 1 client node with 16 fio_rbd client streams, we 
>already
>see the benefit of big pipe.
>
>              ib_send_bw 64k read (ceph) 256k read(ceph)
>---------------------------------------------------------------------------------------
>40Gbe: 4350 MB/s 4300 MB/s 4350 MB/s
>56Gbe: 5100 MB/s 4850 MB/s 5000 MB/s
>

On raw data (ib_send_bw), I get only 5100 MB/s on 56Gbe/s link because 
I'm using default mtu=1500.
There is RoCE header overhead per frame and which RoCE's mtu is 
selected/used (RoCE's mtu is 512-bytes in this case)

With jumbo frames (RoCE's mtu can be 2K or 4K), the raw bw can get to 
~50Gb/s

-vu

Re: xio messenger prelim benchmark

[Vu Pham]

Hello Mark,


>Hi Vu,
>
>Was that with auth enabled?


I disabled auth on both run for both simple & xio (xio currently is not 
supporting cephx)

This time I turned on max speed for all CPUs and get better results for 
both simple & xio




>
>On Feb 6, 2015 3:02 PM, "Vu Pham" <vuhuong@mellanox.com> wrote:
>>
>> >
>> >>>>Yes, xio messenger which is implemented over Accelio can run over
>> >>>>rdma
>> >>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>> >>>>enabled xio messenger / Accelio-tcp yet.
>> >>
>> >>Oh, ok, Great !
>> >>
>> >>
>> >>>>xio messenger is currently working with user mode clients. We only
>> >>>>validated/tested user mode rbd client.
>> >>>>Sandisk are working on krbd over kAccelio implementation. As of 
>>last
>> >>>>week, SanDisk have basic I/Os working.
>> >>>>Hopefully krbd/kAccelio will be available soon.
>> >>
>> >>Great, so we can expect even better results :)
>> >>
>> >>
>> >>BTW, do you have some client side cpu usage benchmark ?
>> >>
>> >>
>> >Sorry, I did not collect the cpu usage @client
>> >
>>
>>I just rerun the test with 2 OSDs, 4 fio_clients on 4 separated
>>pools/images, each client uses numjobs=1 for random write test and
>>numjobs=8 for random read test. I also measured the cpu usage@client
>>this time
>>
>>xio messenger:
>>--------------------
>>. 4K random write: ~26k iops (10 cores used at OSD, 2 cores used at
>>client)
>>. 4K random read: ~332k iops (34 cores used at OSD, 4 cores used at
>>client)
>>. 256K random write: ~1464 MB/s (6 cores used at OSD, 2 cores used at
>>client)
>>. 256K random read: ~4300 MB/s (6 cores used at OSD, 5 cores used at
>>client)
>>
>>simple:
>>----------
>>. 4K random write: ~25k iops (10 cores used at OSD, 3 cores used at
>>client)
>>. 4K random read: ~230k iops (38 cores used at OSD, 30 cores used at
>>client)
>>. 256K random write: ~1475 MB/s (6 cores used at OSD, 2 cores used at
>>client)
>>. 256K random read: ~4300 MB/s (7 cores used at OSD, 7 cores used at
>>client)
>>
>>-vu