nonstandard raid(4-ish) scheme

nonstandard raid(4-ish) scheme

[Lennert Buytenhek]

(I'm not on the list, please CC on replies.)

Hi!

While recently setting up a media server for a HTPC, I was wondering
whether it'd be possible to set things up so that most disks can be
kept spun down most of the time.

Since the majority of the I/O on this sort of server is reads, the
first idea was to store (directories of) media files entirely on
individual disks, i.e. to not distribute them over all disks by
striping, as that would require spinning up each disk on reads.  So,
creating a large RAID(5/6) array out of all of the disks wouldn't be
option here.

RAID1ing pairs of disks would work, but this is somewhat wasteful,
as it eats up half of your space.

The ideal way of doing this would be to take N-1 disks out of your N
disk set, and create separate filesystems on each of the disks, and
store different sets of files on them, while using the last disk,
disk N, as a parity disk of the N-1 data disks.

This is equivalent to raid4 with the stripe size equal the size of
one disk, and where the data disks get individual /dev/mdX entries
instead of being concatenated into one large /dev/mdX.

Would this be easy to implement?


thanks,
Lennert

Re: nonstandard raid(4-ish) scheme

[John Robinson]

On 24/01/2012 03:00, Lennert Buytenhek wrote:
> (I'm not on the list, please CC on replies.)
>
> Hi!
>
> While recently setting up a media server for a HTPC, I was wondering
> whether it'd be possible to set things up so that most disks can be
> kept spun down most of the time.
>
> Since the majority of the I/O on this sort of server is reads, the
> first idea was to store (directories of) media files entirely on
> individual disks, i.e. to not distribute them over all disks by
> striping, as that would require spinning up each disk on reads.  So,
> creating a large RAID(5/6) array out of all of the disks wouldn't be
> option here.
>
> RAID1ing pairs of disks would work, but this is somewhat wasteful,
> as it eats up half of your space.
>
> The ideal way of doing this would be to take N-1 disks out of your N
> disk set, and create separate filesystems on each of the disks, and
> store different sets of files on them, while using the last disk,
> disk N, as a parity disk of the N-1 data disks.
>
> This is equivalent to raid4 with the stripe size equal the size of
> one disk, and where the data disks get individual /dev/mdX entries
> instead of being concatenated into one large /dev/mdX.
>
> Would this be easy to implement?

I think it already has been, as UnRAID: 
http://en.wikipedia.org/wiki/Non-standard_RAID_levels#UnRAID

Cheers,

John.

Re: nonstandard raid(4-ish) scheme

[Christoph Hellwig]

On Tue, Jan 24, 2012 at 04:00:15AM +0100, Lennert Buytenhek wrote:
> The ideal way of doing this would be to take N-1 disks out of your N
> disk set, and create separate filesystems on each of the disks, and
> store different sets of files on them, while using the last disk,
> disk N, as a parity disk of the N-1 data disks.
> 
> This is equivalent to raid4 with the stripe size equal the size of
> one disk, and where the data disks get individual /dev/mdX entries
> instead of being concatenated into one large /dev/mdX.
> 
> Would this be easy to implement?

I think it would be fairly easy.  Take the existing RAID4 code, and
instead of striping over the data disks use them in a linear, concatenated
fashion.  Then make sure the filesystem places whole directories on
one of them.  Using the "filestreams" mount option in XFS, combined with
aligning the allocation groups to the data disk boundaries should do
that for you.

Re: nonstandard raid(4-ish) scheme

[David Brown]

On 24/01/12 04:00, Lennert Buytenhek wrote:
> (I'm not on the list, please CC on replies.)
>
> Hi!
>
> While recently setting up a media server for a HTPC, I was wondering
> whether it'd be possible to set things up so that most disks can be
> kept spun down most of the time.
>
> Since the majority of the I/O on this sort of server is reads, the
> first idea was to store (directories of) media files entirely on
> individual disks, i.e. to not distribute them over all disks by
> striping, as that would require spinning up each disk on reads.  So,
> creating a large RAID(5/6) array out of all of the disks wouldn't be
> option here.
>
> RAID1ing pairs of disks would work, but this is somewhat wasteful,
> as it eats up half of your space.
>
> The ideal way of doing this would be to take N-1 disks out of your N
> disk set, and create separate filesystems on each of the disks, and
> store different sets of files on them, while using the last disk,
> disk N, as a parity disk of the N-1 data disks.
>
> This is equivalent to raid4 with the stripe size equal the size of
> one disk, and where the data disks get individual /dev/mdX entries
> instead of being concatenated into one large /dev/mdX.
>
> Would this be easy to implement?
>
>
> thanks,
> Lennert

What about a raid5 with a truly massive chunk size?  Brute force and 
inelegant, but simple to try.