Re: [PATCH] rdma ioengine improvement

[Jens Axboe <axboe@kernel.dk>]

On 2012-10-23 03:29, Yufei Ren wrote:
> From: Yufei Ren <renyufei83@gmail.com>
> 
> 1) Use fio shipped reentrant and thread-safe rand to replace buggy rand().
> 2) Add a pause time period before client start committing IOs.
> 	In SEND/RECV test, it's a good practice to setup the iodepth of
> 	of the RECV side deeper than that of the SEND side to
> 	avoid RNR (receiver not ready) error. The
> 	SEND side may send so many unsolicited message before
> 	RECV side commits sufficient recv buffers into recv queue.
> 	This may lead to RNR error. Here, SEND side pauses for a while
> 	during which RECV side commits sufficient recv buffers.
> 3) Fix server thread hanging bug.
> 	For RDMA WRITE/READ test, No IO bytes are accumulated in server side
> 	during test. Server thread indicates its task completion by changing
> 	its state into `done' as an alternitive. Add `td->done' checking in
> 	do_io().
> 4) Some comments revision.
> 	export EXTFLAGS and EXTLIBS with '+=".
> 	explanation on exchanging steps of RDMA ioengine control messages.

I applied the below on top. We can't reuse td->__random_state, as
that'll impact the generation of random numbers used for IO unit
offsets.

diff --git a/engines/rdma.c b/engines/rdma.c
index 2633432..87b061a 100644
--- a/engines/rdma.c
+++ b/engines/rdma.c
@@ -119,6 +119,8 @@ struct rdmaio_data {
 	int io_u_flight_nr;
 	struct io_u **io_us_completed;
 	int io_u_completed_nr;
+
+	struct frand_state rand_state;
 };
 
 static int client_recv(struct thread_data *td, struct ibv_wc *wc)
@@ -617,7 +619,7 @@ static int fio_rdmaio_send(struct thread_data *td, struct io_u **io_us,
 			if (td->o.use_os_rand)
 				index = os_random_long(&td->random_state) % rd->rmt_nr;
 			else
-				index = __rand(&td->__random_state) % rd->rmt_nr;
+				index = __rand(&rd->rand_state) % rd->rmt_nr;
 			r_io_u_d->sq_wr.opcode = IBV_WR_RDMA_WRITE;
 			r_io_u_d->sq_wr.wr.rdma.rkey = rd->rmt_us[index].rkey;
 			r_io_u_d->sq_wr.wr.rdma.remote_addr = \
@@ -630,7 +632,7 @@ static int fio_rdmaio_send(struct thread_data *td, struct io_u **io_us,
 			if (td->o.use_os_rand)
 				index = os_random_long(&td->random_state) % rd->rmt_nr;
 			else
-				index = __rand(&td->__random_state) % rd->rmt_nr;
+				index = __rand(&rd->rand_state) % rd->rmt_nr;
 			r_io_u_d->sq_wr.opcode = IBV_WR_RDMA_READ;
 			r_io_u_d->sq_wr.wr.rdma.rkey = rd->rmt_us[index].rkey;
 			r_io_u_d->sq_wr.wr.rdma.remote_addr = \
@@ -1210,6 +1212,7 @@ static int fio_rdmaio_setup(struct thread_data *td)
 		rd = malloc(sizeof(*rd));;
 
 		memset(rd, 0, sizeof(*rd));
+		init_rand_seed(&rd->rand_state, GOLDEN_RATIO_PRIME);
 		td->io_ops->data = rd;
 	}
 

-- 
Jens Axboe