Re: [RFC 0/8] Reliably generate large request from SRP

David Dillow
2011-01-24T17:53:34Z

On Mon, 2011-01-24 at 10:32 -0500, Or Gerlitz wrote:
> David Dillow wrote:
> >> if we look on the 50% for SAS/1M IOs that you're presenting, can you tell
> >> what made the difference, srp went from sg_tablesize of 255 to 256 so the
> >> upper layers where able to provide 1M as one IO
>
> > This win is from sg_tablesize going from 255 to 256 in this case; the HW
> > really likes that better than getting two requests -- one for 1020 KB
> > and one for 4 KB.
>
> Its always nice to find the simplest explanation to the greatest
> improvement... going to the 2nd largest gains
>
> > SAS 2M 520 MB/s 861 MB/s
> > SAS 4M 529 MB/s 921 MB/s
> > SAS 8M 600 MB/s 951 MB/s
>
> I wonder what made the difference here? it can't be only the 255 -->
> 256 sg_tablesize change, for the 2M case the change to use 512 pages
> FMRs could let you use one rkey/fmr for the whole IO but not for 4M/8M

Actually, it very much was the sg_tablesize going from 255 to 2048 (512
actually used for 2 MB requests). This was pushed into 2 FMRs, as I was
only using 256 entry FMRs for the test; I did not rerun the tests after
I added the 512 entry FMR change.

So, for above 1 MB, we always used indirect descriptors. The target used
has a high overhead per command, and cannot aggregate back-end disk
commands when the write cache is disabled, so it thrives on large
requests.

--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office