Re: recommended way to add ssd cache to mdraid array

[Thomas Fjellstrom <thomas@fjellstrom.ca>]

On Thu Jan 10, 2013, Stan Hoeppner wrote:
> On 1/10/2013 3:36 PM, Chris Murphy wrote:
> > On Jan 10, 2013, at 3:49 AM, Thomas Fjellstrom <thomas@fjellstrom.ca> wrote:
> >> A lot of it will be streaming. Some may end up being random read/writes.
> >> The test is just to gauge over all performance of the setup. 600MBs
> >> read is far more than I need, but having writes at 1/3 that seems odd
> >> to me.
> > 
> > Tell us how many disks there are, and what the chunk size is. It could be
> > too small if you have too few disks which results in a small full stripe
> > size for a video context. If you're using the default, it could be too
> > big and you're getting a lot of RWM. Stan, and others, can better answer
> > this.
> 
> Thomas is using a benchmark, and a single one at that, to judge the
> performance.  He's not using his actual workloads.  Tuning/tweaking to
> increase the numbers in a benchmark could be detrimental to actual
> performance instead of providing a boost.  One must be careful.
> 
> Regarding RAID6, it will always have horrible performance compared to
> non-parity RAID levels and even RAID5, for anything but full stripe
> aligned writes, which means writing new large files or doing large
> appends to existing files.

Considering its a rather simple use case, mostly streaming video, and misc
file sharing for my home network, an iozone test should be rather telling.
Especially the full test, from 4k up to 16mb

                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        33554432       4  243295  221756   628767   624081    1028    4627   16822  7468777    17740   233295   231092  582036   579131
        33554432       8  241134  225728   628264   627015    2027    8879   25977 10030302    19578   228923   233928  591478   584892
        33554432      16  233758  228122   633406   618248    3952   13635   35676 10166457    19968   227599   229698  579267   576850
        33554432      32  232390  219484   625968   625627    7604   18800   44252 10728450    24976   216880   222545  556513   555371
        33554432      64  222936  206166   631659   627823   14112   22837   52259 11243595    30251   196243   192755  498602   494354
        33554432     128  214740  182619   628604   626407   25088   26719   64912 11232068    39867   198638   185078  463505   467853
        33554432     256  202543  185964   626614   624367   44363   34763   73939 10148251    62349   176724   191899  593517   595646
        33554432     512  208081  188584   632188   629547   72617   39145   84876  9660408    89877   182736   172912  610681   608870
        33554432    1024  196429  166125   630785   632413  116793   51904  133342  8687679   121956   168756   175225  620587   616722
        33554432    2048  185399  167484   622180   627606  188571   70789  218009  5357136   370189   171019   166128  637830   637120
        33554432    4096  198340  188695   632693   628225  289971   95211  278098  4836433   611529   161664   170469  665617   655268
        33554432    8192  177919  167524   632030   629077  371602  115228  384030  4934570   618061   161562   176033  708542   709788
        33554432   16384  196639  183744   631478   627518  485622  133467  462861  4890426   644615   175411   179795  725966   734364

> However, everything is relative.  This RAID6 may have plenty of random
> and streaming write/read throughput for Thomas.  But a single benchmark
> isn't going to inform him accurately.

200MB/s may be enough, but the difference between the read and write
throughput is a bit unexpected. It's not a weak machine (core i3-2120, dual
core 3.2Ghz with HT, 16GB ECC 1333Mhz ram), and this is basically all its
going to be doing.

> > You said these are unpartitioned disks, I think. In which case alignment
> > of 4096 byte sectors isn't a factor if these are AF disks.
> > 
> > Unlikely to make up the difference is the scheduler. Parallel fs's like
> > XFS don't perform nearly as well with CFQ, so you should have a kernel
> > parameter elevator=noop.
> 
> If the HBAs have [BB|FB]WC then one should probably use noop as the
> cache schedules the actual IO to the drives.  If the HBAs lack cache,
> then deadline often provides better performance.  Testing of each is
> required on a system and workload basis.  With two identical systems
> (hardware/RAID/OS) one may perform better with noop, the other with
> deadline.  The determining factor is the applications' IO patterns.

Mostly streaming reads, some long rsync's to copy stuff back and forth, file
share duties (downloads etc).

> > Another thing to look at is md/stripe_cache_size which probably needs to
> > be higher for your application.
> > 
> > Another thing to look at is if you're using XFS, what your mount options
> > are. Invariably with an array of this size you need to be mounting with
> > the inode64 option.
> 
> The desired allocator behavior is independent of array size but, once
> again, dependent on the workloads.  inode64 is only needed for large
> filesystems with lots of files, where 1TB may not be enough for the
> directory inodes.  Or, for mixed metadata/data heavy workloads.
> 
> For many workloads including databases, video ingestion, etc, the
> inode32 allocator is preferred, regardless of array size.  This is the
> linux-raid list so I'll not go into detail of the XFS allocators.

If you have the time and the desire, I'd like to hear about it off list.

> >> The reason I've selected RAID6 to begin with is I've read (on this
> >> mailing list, and on some hardware tech sites) that even with SAS
> >> drives, the rebuild/resync time on a large array using large disks
> >> (2TB+) is long enough that it gives more than enough time for another
> >> disk to hit a random read error,
> > 
> > This is true for high density consumer SATA drives. It's not nearly as
> > applicable for low to moderate density nearline SATA which has an order
> > of magnitude lower UER, or for enterprise SAS (and some enterprise SATA)
> > which has yet another order of magnitude lower UER.  So it depends on
> > the disks, and the RAID size, and the backup/restore strategy.
> 
> Yes, enterprise drives have a much larger spare sector pool.
> 
> WRT rebuild time, this is one more reason to use RAID10 or a concat of
> RAID1s.  The rebuild time is low, constant, predictable.  For 2TB drives
> about 5-6 hours at 100% rebuild rate.  And rebuild time, for any array
> type, with gargantuan drives, is yet one more reason not to use the
> largest drives you can get your hands on.  Using 1TB drives will cut
> that to 2.5-3 hours, and using 500GB drives will cut it down to 1.25-1.5
> hours, as all these drives tend to have similar streaming write rates.
> 
> To wit, as a general rule I always build my arrays with the smallest
> drives I can get away with for the workload at hand.  Yes, for a given
> TB total it increases acquisition cost of drives, HBAs, enclosures, and
> cables, and power consumption, but it also increases spindle count--thus
> performance-- while decreasing rebuild times substantially/dramatically.

I'd go raid10 or something if I had the space, but this little 10TB nas (which
is the goal, a small, quiet, not too slow, 10TB nas with some kind of
redundancy) only fits 7 3.5" HDDs.

Maybe sometime in the future I'll get a big 3 or 4 u case with a crap load of
3.5" HDD bays, but for now, this is what I have (as well as my old array,
7x1TB RAID5+XFS in 4in3 hot swap bays with room for 8 drives, but haven't
bothered to expand the old array, and I have the new one almost ready to go).

I don't know if it impacts anything at all, but when burning in these drives
after I bought them, I ran the same full iozone test a couple times, and each
drive shows 150MB/s read, and similar write times (100-120+?). It impressed me
somewhat, to see a mechanical hard drive go that fast. I remember back a few
years ago thinking 80MBs was fast for a HDD.

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca