Re: [PATCH v2 00/18] NFS/RDMA client patches for v4.7

[Steve Wise <swise@opengridcomputing.com>]

On 4/26/2016 9:57 AM, Chuck Lever wrote:
>
>> On Apr 26, 2016, at 10:13 AM, Steve Wise <swise@opengridcomputing.com> wrote:
>>
>> Hey Chuck, I'm testing this series on cxgb4.    I'm running 'iozone -a -+d -I' on a share and watching the server stats.  Are the starve numbers expected?
>
> Yes, unless you're seeing much higher numbers than
> you used to.
>
>
>> Every 5.0s: for s in  /proc/sys/sunrpc/svc_rdma/rdma_* ; do echo -n "$(basename $s): "; cat $s; done                              Tue Apr 26 07:10:17 2016
>>
>> rdma_stat_read: 379872
>> rdma_stat_recv: 498144
>> rdma_stat_rq_poll: 0
>> rdma_stat_rq_prod: 0
>> rdma_stat_rq_starve: 675564
>
> This means work was enqueued on the svc_xprt, but by the
> time the upper layer invoked svc_rdma_recvfrom, the work
> was already handled by an earlier wake-up.
>
> I'm not exactly sure why this happens, but it seems to be
> normal (if suboptimal).
>
>
>> rdma_stat_sq_poll: 0
>> rdma_stat_sq_prod: 0
>> rdma_stat_sq_starve: 1748000
>
> No SQ space to post a Send, so the caller is put to sleep.
>
> The server chronically underestimates the SQ depth, especially
> for FRWR. I haven't figured out a better way to estimate it.
>
> But it's generally harmless, as there is a mechanism to put
> callers to sleep until there is space on the SQ.
>


Thanks.

With this iw_cxgb4 drain fix applied:

[PATCH 3/3] iw_cxgb4: handing draining an idle qp

http://www.spinics.net/lists/linux-rdma/msg34927.html

The series tests good over cxgb4.

Tested-by: Steve Wise <swise@opengridcomputing.com>