Re: Very long raid5 init/rebuild times

[Stan Hoeppner <stan@hardwarefreak.com>]

On 2/4/2014 7:19 PM, Phillip Susi wrote:
> On 02/04/2014 04:58 PM, Stan Hoeppner wrote:
>> Yes, I stated that in this post
>>
>> http://www.spinics.net/lists/raid/msg45726.html
>>
>> in the context of achieving greater throughput with an FIO job
>> file configured to use O_DIRECT, a job file I created, that the OP
>> was using for testing.  That job file is quoted further down in
>> this same post, and is included in my posts prior this one in the
>> thread.  Apparently you ignored them.  The context of my comment
>> above is clearly established multiple times earlier in the thread.
>>
>> In my paragraph directly preceding the statement you quote above,
>> I stated this:
>>
>> "Serial submission typically doesn't reach peak throughput...  You 
>> usually must submit asynchronously or in parallel to reach maximum 
>> throughput."
>>
>> And again this is in the context of the FIO job file using
>> O_DIRECT, and this statement is factual.  As I repeated earlier
>> today, O_DIRECT is used because measuring actual throughput at the
>> disks is straightforward.  To increase O_DIRECT write throughput in
>> FIO, you typically need parallel submission or AIO.  This is well
>> known.
> 
> Ahh, I did not gather that O_DIRECT was already assumed.  In that
> case, then I was simply restating the same thing: that you want aio
> with O_DIRECT, but otherwise, buffered IO works fine too ( which is
> what the OP was using with dd, which is why it sounded like you were
> saying not to do that, that you must use O_DIRECT + aio because
> buffered IO won't get you the performance you're looking for ).

I guess maybe I wasn't clear with my wording at that time.  Yes, IIRC he
was doing dd through buffer cache.  My point to him was that O_DIRECT dd
gives more accurate throughput numbers, but a single stream may not be
sufficient to peak the disks.  Which is why I recommended FIO and
provided a job file, as it can do multiple O_DIRECT streams.  AIO can
reduce dispatch latency thus increasing throughput, but not to the
extent that multiple streams will, as the latter can fully overlap the
single stream dispatch latency, keeping the pipeline full.

-- 
Stan