Combining OCFS2 with Linux software RAID-0?

Combining OCFS2 with Linux software RAID-0?

[Patrick J. LoPresti]

Is it possible to run an OCFS2 file system on top of Linux software RAID?

Here is my situation.  I have four identical disk chassis that perform
hardware RAID internally.  Each chassis has a pair of fiber channel
ports, and I can assign the same LUN to both ports.  I want to connect
all of these chassis to two Linux systems.  I want the two Linux
systems to share a file system that is striped across all four chassis
for performance.

I know I can use software RAID (mdadm) to do RAID-0 striping across
the four chassis on a single machine; I have tried this, it works
fine, and the performance is tremendous.  I also know I can use OCFS2
to create a single filesystem on a single chassis that is shared
between my two Linux systems.  What I want is to combine these two
things.

Suse's documentation
(http://www.novell.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/raidyast.html)
says:

"IMPORTANT:Software RAID is not supported underneath clustered file
systems such as OCFS2, because RAID does not support concurrent
activation. If you want RAID for OCFS2, you need the RAID to be
handled by the storage subsystem."

Because my disk chassis already perform hardware RAID-5, I only need
Linux to do the striping (RAID-0) in software.  So for me, there is no
issue about "which node should rebuild the RAID" etc.  I understand
that Linux md stores meta-data on the partitions and is not cluster
aware, but will this create problems for OCFS2 even if it is just RAID
0?

Has anybody tried something like this?  Are there alternative RAID-0
solutions for Linux that would be expected to work?

Thank you.

 - Pat

Re: Combining OCFS2 with Linux software RAID-0?

[Neil Brown]

On Wed, 9 Dec 2009 15:03:11 -0800
"Patrick J. LoPresti" <lopresti@gmail.com> wrote:

> Is it possible to run an OCFS2 file system on top of Linux software RAID?
> 
> Here is my situation.  I have four identical disk chassis that perform
> hardware RAID internally.  Each chassis has a pair of fiber channel
> ports, and I can assign the same LUN to both ports.  I want to connect
> all of these chassis to two Linux systems.  I want the two Linux
> systems to share a file system that is striped across all four chassis
> for performance.
> 
> I know I can use software RAID (mdadm) to do RAID-0 striping across
> the four chassis on a single machine; I have tried this, it works
> fine, and the performance is tremendous.  I also know I can use OCFS2
> to create a single filesystem on a single chassis that is shared
> between my two Linux systems.  What I want is to combine these two
> things.
> 
> Suse's documentation
> (http://www.novell.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/raidyast.html)
> says:
> 
> "IMPORTANT:Software RAID is not supported underneath clustered file
> systems such as OCFS2, because RAID does not support concurrent
> activation. If you want RAID for OCFS2, you need the RAID to be
> handled by the storage subsystem."
> 
> Because my disk chassis already perform hardware RAID-5, I only need
> Linux to do the striping (RAID-0) in software.  So for me, there is no
> issue about "which node should rebuild the RAID" etc.  I understand
> that Linux md stores meta-data on the partitions and is not cluster
> aware, but will this create problems for OCFS2 even if it is just RAID
> 0?
> 
> Has anybody tried something like this?  Are there alternative RAID-0
> solutions for Linux that would be expected to work?


For RAID0 or LINEAR, this should work - give it a try.

If it doesn't work in some way, let me know and I'll see about fixing it.

It might work for RAID1 one day to, but is unlikely to ever work for 
RAID5.

NeilBrown

Re: Combining OCFS2 with Linux software RAID-0?

[Luca Berra]

On Thu, Dec 10, 2009 at 12:37:03PM +1100, Neil Brown wrote:
>On Wed, 9 Dec 2009 15:03:11 -0800
>> Has anybody tried something like this?  Are there alternative RAID-0
>> solutions for Linux that would be expected to work?
>
>For RAID0 or LINEAR, this should work - give it a try.
>
_with_ a superblock? or without?
what happens if one node modifies the superblock while the other is
running.

>It might work for RAID1 one day to, but is unlikely to ever work for 
>RAID5.
With an external metadata handler i believe it would be possible to
support up to raid1, maybe 10.

While in similar setups, data protection should be handled by the
storage system, raid 1 has its uses in a failover setup with two sites.
I don't see a real use for raid5/6 in such scenarios.

L.

-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: Combining OCFS2 with Linux software RAID-0?

[Wolfgang Denk]

Dear Luca Berra,

In message <20091211071526.GA4062@maude.comedia.it> you wrote:
>
> While in similar setups, data protection should be handled by the
> storage system, raid 1 has its uses in a failover setup with two sites.

Would one not rather use something like DRBD for such a szenario? And
OCFS2 on top of that?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Time is an illusion perpetrated by the manufacturers of space.

Re: Combining OCFS2 with Linux software RAID-0?

[Luca Berra]

On Fri, Dec 11, 2009 at 08:30:02AM +0100, Wolfgang Denk wrote:
>Dear Luca Berra,
>
>In message <20091211071526.GA4062@maude.comedia.it> you wrote:
>>
>> While in similar setups, data protection should be handled by the
>> storage system, raid 1 has its uses in a failover setup with two sites.
>
>Would one not rather use something like DRBD for such a szenario? And
>OCFS2 on top of that?

it depends, if i have an high speed (i.e. FC) link between the two site
i might want to avoid the overhead, besides if we have more than one
node for site, would drbd work?

L.

-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: Combining OCFS2 with Linux software RAID-0?

[Neil Brown]

On Fri, 11 Dec 2009 08:15:26 +0100
Luca Berra <bluca@comedia.it> wrote:

> On Thu, Dec 10, 2009 at 12:37:03PM +1100, Neil Brown wrote:
> >On Wed, 9 Dec 2009 15:03:11 -0800
> >> Has anybody tried something like this?  Are there alternative RAID-0
> >> solutions for Linux that would be expected to work?
> >
> >For RAID0 or LINEAR, this should work - give it a try.
> >
> _with_ a superblock? or without?

Either.

> what happens if one node modifies the superblock while the other is
> running.

Once a RAID0 has been created, there is never any need to modify the
superblock, and I'm fairly sure we don't.

> 
> >It might work for RAID1 one day to, but is unlikely to ever work for 
> >RAID5.
> With an external metadata handler i believe it would be possible to
> support up to raid1, maybe 10.

That is the idea, yes.

> 
> While in similar setups, data protection should be handled by the
> storage system, raid 1 has its uses in a failover setup with two sites.
> I don't see a real use for raid5/6 in such scenarios.

If I had 16 nodes, each with a local device and a fast interconnect, I might
want to have a filesystem that spanned all of the devices but survived the
failure of any two nodes.  Then having a stripe/parity layout across the
devices would be useful.
However I think the only way to get good performance would be to require that
the filesystem does full-stripe writes every time.
So I think that would look more like a clusterised ZFS ... but who knows.

NeilBrown

Re: [Ocfs2-users] Combining OCFS2 with Linux software RAID-0?

[Sunil Mushran]

It should work. All ocfs2 needs is a stable shared storage that
is consistent across all nodes all the time.

Patrick J. LoPresti wrote:
> Is it possible to run an OCFS2 file system on top of Linux software RAID?
>
> Here is my situation.  I have four identical disk chassis that perform
> hardware RAID internally.  Each chassis has a pair of fiber channel
> ports, and I can assign the same LUN to both ports.  I want to connect
> all of these chassis to two Linux systems.  I want the two Linux
> systems to share a file system that is striped across all four chassis
> for performance.
>
> I know I can use software RAID (mdadm) to do RAID-0 striping across
> the four chassis on a single machine; I have tried this, it works
> fine, and the performance is tremendous.  I also know I can use OCFS2
> to create a single filesystem on a single chassis that is shared
> between my two Linux systems.  What I want is to combine these two
> things.
>
> Suse's documentation
> (http://www.novell.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/raidyast.html)
> says:
>
> "IMPORTANT:Software RAID is not supported underneath clustered file
> systems such as OCFS2, because RAID does not support concurrent
> activation. If you want RAID for OCFS2, you need the RAID to be
> handled by the storage subsystem."
>
> Because my disk chassis already perform hardware RAID-5, I only need
> Linux to do the striping (RAID-0) in software.  So for me, there is no
> issue about "which node should rebuild the RAID" etc.  I understand
> that Linux md stores meta-data on the partitions and is not cluster
> aware, but will this create problems for OCFS2 even if it is just RAID
> 0?
>
> Has anybody tried something like this?  Are there alternative RAID-0
> solutions for Linux that would be expected to work?
>
> Thank you.
>
>  - Pat
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>   

Re: [Ocfs2-users] Combining OCFS2 with Linux software RAID-0?

[Joel Becker]

On Thu, Dec 10, 2009 at 11:40:43AM -0800, Sunil Mushran wrote:
> It should work. All ocfs2 needs is a stable shared storage that
> is consistent across all nodes all the time.

Patrick,
	What Sunil says is correct; your solution should work.  The big
issue with software RAID is that many forms of RAID can change the
storage topology automatically.  Think of failing one disk and adding a
hot-spare in its place.  If you had such a configuration, one node would
be running on the hot spare while all the other nodes would still be
using the original LUN.  Corruption!  So RAID 5, RAID 6, even RAID 1 -
anything where the RAID software can make an automatic decision that is
seen by one node and not the others - is ripe for data loss.
	RAID 0 and linear concatenation don't have this automatic
problem.  But they still can cause data loss if a fat-fingered admin
accidentally changes the topology on one node while another node is
still using it.  This is an easier mistake to make than you think.  You
have to be absolutely sure your procedures never let this happen.  This
is why our documents say "No Software RAID".
	Hardware RAID doesn't have this problem because it is invisible
to the nodes.  If you change the hardware, all nodes see the change at
the same time.  It looks like the same LUN as always.
	If somehow the software RAID code became cluster aware and
coordinated topology changes across the cluster, then ocfs2 would be
able to support it.  Until that time, we generally have to say "No
Software RAID"; when folks like you come along we say "You can do RAID
0, but caveat emptor!"

Joel

-- 

Life's Little Instruction Book #139

	"Never deprive someone of hope; it might be all they have."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127