Network-based RAID6

Network-based RAID6

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Hello,

Let's say I have 10 machines, each having 1TB of free disk space. They
communicate over a gigabit network.

I would like to create a fault-tolerant array from these machines and their
storage, that would match RAID6 in overhead and fault resilience.
That is, it should provide 8 TB of usable space and tolerate a 2-member
failure without data loss.

Does anyone know if any of the current distributed filesystems will meet these
requirements? Working/stable/non-experimental level code is preferred.

The simple and obvious solution is mdadm over AoE/iSCSI, but maybe there is
something else built with network awareness from ground up, that would be a
better choice?

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Network-based RAID6

[CoolCold]

On Wed, Mar 30, 2011 at 11:11 AM, Roman Mamedov <rm@romanrm.ru> wrote:
> Hello,
>
> Let's say I have 10 machines, each having 1TB of free disk space. They
> communicate over a gigabit network.
>
> I would like to create a fault-tolerant array from these machines and their
> storage, that would match RAID6 in overhead and fault resilience.
> That is, it should provide 8 TB of usable space and tolerate a 2-member
> failure without data loss.
>
> Does anyone know if any of the current distributed filesystems will meet these
> requirements? Working/stable/non-experimental level code is preferred.
>
> The simple and obvious solution is mdadm over AoE/iSCSI, but maybe there is
> something else built with network awareness from ground up, that would be a
> better choice?
Looking on your host I think you speak russian, so may be this will be
somehow helpful for you
http://community.livejournal.com/ru_root/2216389.html

>
> --
> With respect,
> Roman
>



-- 
Best regards,
[COOLCOLD-RIPN]

Re: Network-based RAID6

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Wed, 30 Mar 2011 11:20:41 +0400
CoolCold <coolthecold@gmail.com> wrote:

> Looking on your host I think you speak russian, so may be this will be
> somehow helpful for you
> http://community.livejournal.com/ru_root/2216389.html

Thanks -- I have looked through the websites of some distributed filesystems
(Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
could find, all filesystems I read about so far are at most capable of RAID0 or
RAID1-like modes, where fault-tolerance is either not provided, or achieved
only by "all data is replicated across N nodes", which of course divides the
total usable space by N. I haven't found any FS which would do block-level
replication relying not on dumb copies, but on RAID5/6-like parity algorithms
for fault-tolerance. Maybe I missed something?

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Network-based RAID6

[Stan Hoeppner]

Roman Mamedov put forth on 3/30/2011 3:49 AM:
> On Wed, 30 Mar 2011 11:20:41 +0400
> CoolCold <coolthecold@gmail.com> wrote:
> 
>> Looking on your host I think you speak russian, so may be this will be
>> somehow helpful for you
>> http://community.livejournal.com/ru_root/2216389.html
> 
> Thanks -- I have looked through the websites of some distributed filesystems
> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
> could find, all filesystems I read about so far are at most capable of RAID0 or
> RAID1-like modes, where fault-tolerance is either not provided, or achieved
> only by "all data is replicated across N nodes", which of course divides the
> total usable space by N. I haven't found any FS which would do block-level
> replication relying not on dumb copies, but on RAID5/6-like parity algorithms
> for fault-tolerance. Maybe I missed something?

You likely won't find any distributed filesystem that performs block
level replication over the network, at least not a FOSS one.  These are
filesystems, mind you, not distributed block device drivers.  If they
perform any replication to afford a level of fault tolerance, it will be
at the file level, not the block level.

If you want true block level replication over a network, look into DRBD.
 However, it is also limited to mirroring.

-- 
Stan

Re: Network-based RAID6

[Roberto Spadim]

OCFS with DRBD could work very nice in linux
i didn't tested mdadm with ndb in a production enviroment
DRBD have brainsplit solutions, since you will run a complex
filesystem, i consider using working solutions, DRBD and OCFS is nice
yes, mdadm can run under DRBD
DRBD = raid1 over network
OCFS = oracle filesystem (cluster filesystem)
it works

2011/3/30 Stan Hoeppner <stan@hardwarefreak.com>:
> Roman Mamedov put forth on 3/30/2011 3:49 AM:
>> On Wed, 30 Mar 2011 11:20:41 +0400
>> CoolCold <coolthecold@gmail.com> wrote:
>>
>>> Looking on your host I think you speak russian, so may be this will be
>>> somehow helpful for you
>>> http://community.livejournal.com/ru_root/2216389.html
>>
>> Thanks -- I have looked through the websites of some distributed filesystems
>> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
>> could find, all filesystems I read about so far are at most capable of RAID0 or
>> RAID1-like modes, where fault-tolerance is either not provided, or achieved
>> only by "all data is replicated across N nodes", which of course divides the
>> total usable space by N. I haven't found any FS which would do block-level
>> replication relying not on dumb copies, but on RAID5/6-like parity algorithms
>> for fault-tolerance. Maybe I missed something?
>
> You likely won't find any distributed filesystem that performs block
> level replication over the network, at least not a FOSS one.  These are
> filesystems, mind you, not distributed block device drivers.  If they
> perform any replication to afford a level of fault tolerance, it will be
> at the file level, not the block level.
>
> If you want true block level replication over a network, look into DRBD.
>  However, it is also limited to mirroring.
>
> --
> Stan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

[Miles Fidelman]

I wonder how OCFS (or some other distributed file system) would run over 
btfrs (raid10, no raid6), over AOE or iSCSI block devices.

Roberto Spadim wrote:
> OCFS with DRBD could work very nice in linux
> i didn't tested mdadm with ndb in a production enviroment
> DRBD have brainsplit solutions, since you will run a complex
> filesystem, i consider using working solutions, DRBD and OCFS is nice
> yes, mdadm can run under DRBD
> DRBD = raid1 over network
> OCFS = oracle filesystem (cluster filesystem)
> it works
>
> 2011/3/30 Stan Hoeppner<stan@hardwarefreak.com>:
>    
>> Roman Mamedov put forth on 3/30/2011 3:49 AM:
>>      
>>> On Wed, 30 Mar 2011 11:20:41 +0400
>>> CoolCold<coolthecold@gmail.com>  wrote:
>>>
>>>        
>>>> Looking on your host I think you speak russian, so may be this will be
>>>> somehow helpful for you
>>>> http://community.livejournal.com/ru_root/2216389.html
>>>>          
>>> Thanks -- I have looked through the websites of some distributed filesystems
>>> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from what I
>>> could find, all filesystems I read about so far are at most capable of RAID0 or
>>> RAID1-like modes, where fault-tolerance is either not provided, or achieved
>>> only by "all data is replicated across N nodes", which of course divides the
>>> total usable space by N. I haven't found any FS which would do block-level
>>> replication relying not on dumb copies, but on RAID5/6-like parity algorithms
>>> for fault-tolerance. Maybe I missed something?
>>>        
>> You likely won't find any distributed filesystem that performs block
>> level replication over the network, at least not a FOSS one.  These are
>> filesystems, mind you, not distributed block device drivers.  If they
>> perform any replication to afford a level of fault tolerance, it will be
>> at the file level, not the block level.
>>
>> If you want true block level replication over a network, look into DRBD.
>>   However, it is also limited to mirroring.
>>
>> --
>> Stan
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>      
>
>
>    


-- 
In theory, there is no difference between theory and practice.
In<fnord>  practice, there is.   .... Yogi Berra

Re: Network-based RAID6

[Roberto Spadim]

ocfs is a file system, btrfs too, why using two filesystems? only ocfs
would be used when cluster filesystem
you want more disks per computer using others computers as disks?
or you want more disks in cluster using others computers as nodes of a cluster?

first is a single writer filesystem (ext4,reiserfs,xfs)
second is a cluster filesystem (ocfs, and others, check wikipedia for
filesystem lists)

you can use raid1 as block device for filesystem, but i remember that
some cluster filesystems have replication (raid1)

2011/3/30 Miles Fidelman <mfidelman@meetinghouse.net>:
> I wonder how OCFS (or some other distributed file system) would run over
> btfrs (raid10, no raid6), over AOE or iSCSI block devices.
>
> Roberto Spadim wrote:
>>
>> OCFS with DRBD could work very nice in linux
>> i didn't tested mdadm with ndb in a production enviroment
>> DRBD have brainsplit solutions, since you will run a complex
>> filesystem, i consider using working solutions, DRBD and OCFS is nice
>> yes, mdadm can run under DRBD
>> DRBD = raid1 over network
>> OCFS = oracle filesystem (cluster filesystem)
>> it works
>>
>> 2011/3/30 Stan Hoeppner<stan@hardwarefreak.com>:
>>
>>>
>>> Roman Mamedov put forth on 3/30/2011 3:49 AM:
>>>
>>>>
>>>> On Wed, 30 Mar 2011 11:20:41 +0400
>>>> CoolCold<coolthecold@gmail.com>  wrote:
>>>>
>>>>
>>>>>
>>>>> Looking on your host I think you speak russian, so may be this will be
>>>>> somehow helpful for you
>>>>> http://community.livejournal.com/ru_root/2216389.html
>>>>>
>>>>
>>>> Thanks -- I have looked through the websites of some distributed
>>>> filesystems
>>>> (Ceph, GlusterFS, MooseFS etc) and checked this thread too, but from
>>>> what I
>>>> could find, all filesystems I read about so far are at most capable of
>>>> RAID0 or
>>>> RAID1-like modes, where fault-tolerance is either not provided, or
>>>> achieved
>>>> only by "all data is replicated across N nodes", which of course divides
>>>> the
>>>> total usable space by N. I haven't found any FS which would do
>>>> block-level
>>>> replication relying not on dumb copies, but on RAID5/6-like parity
>>>> algorithms
>>>> for fault-tolerance. Maybe I missed something?
>>>>
>>>
>>> You likely won't find any distributed filesystem that performs block
>>> level replication over the network, at least not a FOSS one.  These are
>>> filesystems, mind you, not distributed block device drivers.  If they
>>> perform any replication to afford a level of fault tolerance, it will be
>>> at the file level, not the block level.
>>>
>>> If you want true block level replication over a network, look into DRBD.
>>>  However, it is also limited to mirroring.
>>>
>>> --
>>> Stan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>>
>>
>
>
> --
> In theory, there is no difference between theory and practice.
> In<fnord>  practice, there is.   .... Yogi Berra
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

On Wed, 30 Mar 2011 08:35:04 -0500
Stan Hoeppner <stan@hardwarefreak.com> wrote:

> If you want true block level replication over a network, look into DRBD.
>  However, it is also limited to mirroring.

So I guess a viable and interesting solution could be an mdadm stripe over two
or more DRBD mirrors, effectively providing RAID10. Although that's not quite
what I am looking for (RAID5/6-levels of overhead and resilience).

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Network-based RAID6

[Roberto Spadim]

DRBD - computer replication
raid1 - disk 'replication'

what your needs?

2011/3/30 Roman Mamedov <rm@romanrm.ru>:
> On Wed, 30 Mar 2011 08:35:04 -0500
> Stan Hoeppner <stan@hardwarefreak.com> wrote:
>
>> If you want true block level replication over a network, look into DRBD.
>>  However, it is also limited to mirroring.
>
> So I guess a viable and interesting solution could be an mdadm stripe over two
> or more DRBD mirrors, effectively providing RAID10. Although that's not quite
> what I am looking for (RAID5/6-levels of overhead and resilience).
>
> --
> With respect,
> Roman
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network-based RAID6

[Stan Hoeppner]

Roman Mamedov put forth on 3/30/2011 12:50 PM:
> On Wed, 30 Mar 2011 08:35:04 -0500
> Stan Hoeppner <stan@hardwarefreak.com> wrote:
> 
>> If you want true block level replication over a network, look into DRBD.
>>  However, it is also limited to mirroring.
> 
> So I guess a viable and interesting solution could be an mdadm stripe over two
> or more DRBD mirrors, effectively providing RAID10. Although that's not quite
> what I am looking for (RAID5/6-levels of overhead and resilience).

The mdraid driver sits beneath DRBD.  What you suggest above is impossible.

-- 
Stan

Re: Network-based RAID6

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

On Wed, 30 Mar 2011 23:56:26 -0500
Stan Hoeppner <stan@hardwarefreak.com> wrote:

> Roman Mamedov put forth on 3/30/2011 12:50 PM:
> > So I guess a viable and interesting solution could be an mdadm stripe over
> > two or more DRBD mirrors, effectively providing RAID10. Although that's
> > not quite what I am looking for (RAID5/6-levels of overhead and
> > resilience).
> 
> The mdraid driver sits beneath DRBD.  What you suggest above is impossible.

I thought DRBD presents just a regular kernel-level block device in /dev, and
seeing how one can create mdraid out of just any kind of block device,
including those provided by AoE, iSCSI, LVM, dmcrypt or even 'loop', are you
really sure this matters here? 

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Network-based RAID6

[Stan Hoeppner]

Roman Mamedov put forth on 3/31/2011 12:16 AM:
> On Wed, 30 Mar 2011 23:56:26 -0500
> Stan Hoeppner <stan@hardwarefreak.com> wrote:
> 
>> Roman Mamedov put forth on 3/30/2011 12:50 PM:
>>> So I guess a viable and interesting solution could be an mdadm stripe over
>>> two or more DRBD mirrors, effectively providing RAID10. Although that's
>>> not quite what I am looking for (RAID5/6-levels of overhead and
>>> resilience).
>>
>> The mdraid driver sits beneath DRBD.  What you suggest above is impossible.
> 
> I thought DRBD presents just a regular kernel-level block device in /dev, and
> seeing how one can create mdraid out of just any kind of block device,
> including those provided by AoE, iSCSI, LVM, dmcrypt or even 'loop', are you
> really sure this matters here? 

Do it and report your results.

-- 
Stan

Re: Network-based RAID6

[hansbkk]

On Thu, Mar 31, 2011 at 12:16 PM, Roman Mamedov <rm@romanrm.ru> wrote:
> On Wed, 30 Mar 2011 23:56:26 -0500
> I thought DRBD presents just a regular kernel-level block device in /dev, and
> seeing how one can create mdraid out of just any kind of block device,
> including those provided by AoE, iSCSI, LVM, dmcrypt or even 'loop', are you
> really sure this matters here?

The only advantage of RAID5/6 over mirroring is cost savings,
certainly not greater fault tolerance taking rebuilding times into
account - if you're planning to set up separate whole servers for each
component of your redundancy strategy, it seems odd to me you're
trying to saving a few bucks on hard drives.

DRBD is a well-regarded solution for this application, but it is
oriented toward mirroring whole filesystems (usually those of
mission-critical servers) that may themselves already be protected
locally with RAID. I would advise you follow their standard
recommendations at first, don't get too "creative" if you're actually
looking for fault-tolerance rather than just experimenting around.

All that said, if you're looking for a fun experiment and go ahead,
please do document your results and report back here!