RDMA FETCH_AND_ADD and CMP_AND_SWAP fail

[floaterions <floaterions-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

Hello All,

I can successfully send RDMA READ/WRITE,
but I can't get RDMA atomic operations to work.
I get an error when calling ibv_post_send function in the client,
and the errno will be set to "Invalid Arguments."
Below I pasted important parts of my code.
Could you please check my code and let me know if I'm missing anything?


*********** client side *****************:
-- Registering the memory regions --
mr = ibv_reg_mr(pd, buff, size, IBV_ACCESS_LOCAL_WRITE);
// and the size is 8

if (!mr){
   fprintf(stderr, "Error, memory registration failed\n");
   return -1;
}

-- Preparing RDMA ATOMIC FETCH AND 
struct ibv_send_wr wr, *bad_wr = NULL;
struct ibv_sge sge;
	
memset(&sge, 0, sizeof(sge));
sge.addr 		= buff;
sge.length 	= 8;
sge.lkey 		= mr->lkey;
	
memset(&wr, 0, sizeof(wr));
wr.wr_id     = 0;
wr.opcode  = IBV_WR_ATOMIC_FETCH_AND_ADD;
wr.sg_list 			= &sge;
wr.num_sge 		= 1;
wr.send_flags 		= IBV_SEND_SIGNALED;

wr.wr.atomic.remote_addr	= remote_buffer;
wr.wr.atomic.rkey        	= peer_mr->rkey;
wr.wr.atomic.compare_add	= 1ULL; 

if (ibv_post_send(qp, &wr, &bad_wr)) {
    fprintf(stderr, "Error, ibv_post_send() failed\n");
    return -1;
}
********* End of Client side *******

****** Server side ****************
-- Registering the memory regions --
mr = ibv_reg_mr(pd,  rdma_region_timestamp_oracle,
         sizeof(TimestampOracle), 
         IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE
         | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC ));

if (!mr){
   fprintf(stderr, "Error, memory registration() failed\n");
   return -1;
}


NOTE: TimestampOracle is a class with two int members,
so its size is 8 bytes (satisfies 64-bit condition for RDMA ATOMIC operations)

Thank you for your helps

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html