Re: Adjusting minimum packet size or "wait to merge requests" in SRP

[David Dillow <dillowda-1Heg1YXhbW8@public.gmane.org>]

On Wed, 2009-10-28 at 13:38 -0600, Chris Worley wrote:
> There is no scheduler running on either target or initiator on the
> drives in question (sorry I worded that incorrectly initially), or so
> I've been told (this information is second-hand). 

So, noop scheduler, then?

Under noop, the block layer will send requests as soon as it can without
merging. If it has more requests outstanding than the queue length on
the SRP initiator, then it will merge the new request with the queued
ones if possible.

>  I did see iostat
> output from the initiator in his case, where there were long waits and
> service times that I'm guessing was due to some coalescing/merging.
> There was also a hint in the iostat output that a scheduler was
> enabled, as there were non-zero values (occasionally) under the
> [rw]qm/s columns, which, if I understand iostat correctly, means there
> is a scheduler merging results.
> 
> So you're saying there is no hold-off for merging on the initiator
> side of the IB/SRP stack?

The SRP initiator just hands off requests as quick as they are sent to
it by the block layer. You can control how big those requests are by
tuning /sys/block/$DEV/queue/max_sectors_kb up to .../max_hw_sectors_kb
which gets set by the max_sect parameter when adding the SRP target.

You can get some hold-off potentially by using a non-noop scheduler for
the block device, see /sys/block/$DEV/queue/scheduler. 'as' or
'deadline' may fit your bill, but they have a habit of breaking up
requests into smaller chunks.

Also, you want 'options ib_srp srp_sg_tablesize=255'
in /etc/modprobe.conf, as by default it only allows 12 scatter/gather
entries, which will only guarantee a 48KB request size. Using 255
guarantees you can send a 1020KB request. Of course, if the pages
coalesce in the request, you can send much larger requests before
running out of S/G entires. max_sectors_kb will limit what gets sent in
either case.

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html