From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0887280963742228837==" MIME-Version: 1.0 From: Kumaraparameshwaran Rathnavel Subject: Re: [SPDK] RDMA Queue Pair Date: Sat, 01 Jul 2017 12:08:56 +0530 Message-ID: In-Reply-To: 1498838859.3097.17.camel@intel.com List-ID: To: spdk@lists.01.org --===============0887280963742228837== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Thanks Benjamin. = Even this was my understanding. Thanking You, Param. > On 30-Jun-2017, at 9:37 PM, Walker, Benjamin wrote: > = > On Fri, 2017-06-30 at 18:09 +0530, Kumaraparameshwaran Rathnavel wrote: >> Hi All, >> = >> I would like to get few pointers on how the RDMA is used in NVMf Target = . The >> below statement decides the value of Qdepth of NVMf Qpair. >> nvmf_min(max_rw_depth, addr->attr.max_qp_rd_atom and anything more than = this >> will be Queued irrespective of what the upper layer has the Queue depth = as. = >> = >> And the value for attr.max_qp_rd_atom is got by Querying the device. I s= ee >> that ideally in most of NICs the value is 16. So does this mean that in a >> Queue pair there cannot be more RDMA requests than this value. > = > = > The queue depth calculations for RDMA are actually quite involved. RDMA h= as four > base operations - send, recv, read, and write - and the available queue d= epth > for read/write is different than for send/recv. RDMA send/recv operations= are > used to send the NVMe commands and responses, while the RDMA read/write > operations are used to transfer data. It's just the read/write limit that= is > usually 16 per queue pair, limited by the NIC capabilities. The send/recv= limit > is much higher, with the NIC supporting queues of anywhere from 1024 to 6= 4k > depth. > = > Therefore, we report typically 128 queue depth available and the target w= ill > gather up to 128 total commands at a time. If the commands are all NVMe r= eads, > for example, the target will queue up the full 128 queue depth to the bac= king > SSD. However, it can only perform 16 simultaneous RDMA read or writes pe= r queue > pair, so often commands will get queued in the RDMA layer waiting on that= limit. > = >> = >> Please correct me if I am wrong. >> = >> Thanking You, >> Param. >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============0887280963742228837==--