Re: *terrible* direct-write performance with raid5

[Michael Tokarev <mjt@tls.msk.ru>]

dean gaudet wrote:
> On Tue, 22 Feb 2005, Michael Tokarev wrote:
> 
> 
>>When debugging some other problem, I noticied that
>>direct-io (O_DIRECT) write speed on a software raid5
>>is terrible slow.  Here's a small table just to show
>>the idea (not numbers by itself as they vary from system
>>to system but how they relate to each other).  I measured
>>"plain" single-drive performance (sdX below), performance
>>of a raid5 array composed from 5 sdX drives, and ext3
>>filesystem (the file on the filesystem was pre-created
>>during tests).  Speed measurements performed with 8Kbyte
>>buffer aka write(fd, buf, 8192*1024), units a Mb/sec.
> 
> with O_DIRECT you told the kernel it couldn't cache anything... you're 
> managing the cache.  you should either be writing 64KiB or you should 
> change your chunksize to 8KiB (if it goes that low).

The picture does not change at all when changing raid chunk size.
With 8kb chunk speed is exactly the same as with 64kb or 256kb chunk.

Yes, increasing write buffer size helps alot.  Here's the write
performance in mb/sec for direct-write into an md device which
is a raid5 array built from 5 drives depending on the write
buffer size (in kb):

  buffer  md raid5    sdX
            speed    speed
      1      0.2       14
      2      0.4       26
      4      0.9       41
      8      1.7       44
     16      3.9       44
     32     72.6       44
     64     84.6       ..
    128     97.1
    256     53.7
    512     64.1
   1024     74.5

I've no idea why there's a drop in speed after 128kb blocksize,
but the more important is a huge drop with 32->16 kb blocksize.

The numbers are almost exactly the same with several chunksizes --
256kb, 64kb (default), 8kb and 4kb.

(note raid5 performs faster than a single drive, it's expectable
as it is possible to write to several drives in parallel).

The numbers also does not depend much on seeking -- obviously the
speed with seeking is worse than the above for sequential write,
but not much worse (about 10..20%, not >20 times as with 72 vs 4
mb/sec).

/mjt