Re: best base / worst case RAID 5,6 write speeds

[Phil Turmel <philip@turmel.org>]

On 12/10/2015 04:14 PM, Dallas Clement wrote:

> Exactly.  I'm not expecting RMWs to be happening for large sequential
> writes.  But yet my RAID 5, 6 sequential write performance is still
> very poor.  As mentioned earlier, I'm getting around 95 MB/s on the
> inner side of these disks.  With 12 of them, my RAID 6 write speed
> should be (12 - 2) * 95 = 950 MB/s.  I'm getting about 300 MB/s less
> than that for this scenario.  I have the disks split up among three
> different controllers.  There should be plenty of bandwidth.  Several
> days ago I ran fio on each of the 12 disks concurrently.  I was able
> to see the disks at or near 100% utilization and wMB/s around 160-170
> MB/s.  That's why I started focusing on RAID as being the potential
> bottleneck.
> 
>> That's why I questioned O_SYNC when you were using a filesystem: it
>> prevents merging, and forces seeking to do small metadata writes.
>> Basically turning a sequential workload into a random one.
> 
> Yes, that certainly makes sense.  Not using O_SYNC anymore.  Just O_DIRECT.

Sounds like its time to break out blktrace to see what's really
happening between your array and its member devices.

With diffs from old kernels to new.

Phil