Re: "Missing" RAID devices

[keld@keldix.com]

On Thu, May 23, 2013 at 12:59:39AM -0500, Stan Hoeppner wrote:
> On 5/22/2013 6:26 PM, Phil Turmel wrote:
> > On 05/22/2013 06:43 PM, Stan Hoeppner wrote:
> >> Sorry for the dup Phil, hit the wrong reply button.
> > 
> > No worries.
> > 
> >> On 5/21/2013 7:02 PM, Phil Turmel wrote:
> >> ...
> >>> ...First is /dev/md1, a small (~500m) n-way
> >>> ...as /boot.  The other, /dev/md2, uses
> >>> ...raid10,far3 or raid6.
> >>>
> >>> I put LVM on top of /dev/md2, with LVs for swap, ... /tmp
> >>
> >> Swap and tmp atop an LV atop RAID6?  The former will always RMW on page
> >> writes, the latter quite often will cause RMW.  As you stated your
> >> performance requirements are modest.  However, for the archives, putting
> >> swap on a parity array, let alone a double parity array, is not good
> >> practice.
> > 
> > Ah, good point.  Hasn't hurt me yet, but it would if I pushed anything
> > hard.  I'll have to revise my baseline to always have a small raid10,f3
> > to go with the raid6.
> 
> Yeah, the kicker here is that swap on a parity array seems to work fine,
> right up until the moment it doesn't.  And that's when the kernel goes
> into heavy swapping due to any number of causes.  When that happens,
> you're heavily into RMW, disk heads are bang'n, latency goes through the
> roof.  If any programs are trying to access files on the parity array,
> say a mildly busy IMAP, FTP, etc, server, everything grinds to a halt.
> 
> With your particular setup, instead you might use n additional
> partitions, one each across the physical disks that comprise your n-way
> RAID1.  You would configure the partition type of each as (82) Linux
> swap, and add them all to fstab with equal priority.  The kernel will
> interleave the 4KB swap page writes evenly across all of these
> partitions, yielding swap performance similar to an n-way RAID0 stripe.
> 
> The downside to this setup is the kernel probably crashes if you lose
> one of these disks and thus the swap partition on it.  So you could
> simply make another md/RAID1 of these n partitions if n is an odd number
> of spindles.  Or n/2 RAID1 arrays if n is even.  Then put one swap
> partition on each RAID1 device and do swap interleaving across the RAID1
> pairs as described above in the non RAID case.
> 
> The reason for this last configuration is simple-- more swap throughput
> for the same number of physical writes.  With a 4 drive RAID1 and a
> single swap partition atop, each 4KB page write to swap generates a 4KB
> write to each of the 4 disks, 16KB total.  If you create two RAID1s and
> put a swap partition on each and interleave them, each 4KB page write to
> swap generates only two 4KB writes, 8KB total.  Here for each 16KB
> written you're moving two pages to swap instead of one.  Thus your swap
> bandwidth is doubled.  But you still have redundancy and crash avoidance
> if one disk fails.  You may be tempted to use md/RAID10 of some layout
> to optimize for writes, but you'd gain nothing, and you'd lose some
> performance due to overhead.  The partitions you'll be using in this
> case are so small that they easily fit in a single physical disk track,
> thus no head movement is required to seek between sectors, only rotation
> of the platter.
> 
> Another advantage to this hybrid approach is less disk space consumed.
> If you need 8GB of swap, a 4-way RAID1 swap partition requires 32GB of
> disk space, 8GB per disk.  With the n/2 RAID1 approach and 4 disks it
> requires half that, 16GB.  With the no redundancy interleaved approach
> it requires 1/4th, only 2GB per disk, 8GB total.  With today's
> mechanical disk capacities this isn't a concern.  But if using SSDs it
> can be.
> 
> > Meanwhile, I'm applying some of the general ideas I've seen from you:
> > I've acquired a pair of Crucial M4 SSDs for my new home media server to
> > keep small files and databases away from the bulk storage.  Not in
> > service yet, but I'm very pleased so far.
> 
> If the two are competing for seeks thus slowing everything down, moving
> the random access stuff to SSD should help.
> 
> > I'm pretty sure the new kit is way overkill for a media server... :-)
> 
> Not so many years ago folks would have said the same about 4TB mech
> drives. ;)

I think a raid10,far3 is a good choice for swap, then you will enjoy
RAID0-like reading speed. and good write speed (compared to raid6),
and a chance of live surviving if just one drive keeps functioning.

best regards
keld