accessing mirrired lvm on shared storage

accessing mirrired lvm on shared storage

[Matthias Eble]

Hi all,

I've got an extended setup whith two Systems, each with two FC cards. 
Every card is connected to a seperate disk array (so one system accesses 
two arrays). The other node has access to the same two arrays (standby).

The active server mirrors the data (4 LUNs) between the two arrays via 
md. On top is a LVM physical volume. The other system is meant to be 
booted but not acessing the VGs.

My question is, if it is possible to let both systems set up the md 
mirror without corrupting the data? Is there any data written even when 
the VGs are not taken active? I think I remember that LVM refuses to 
activate volumegroups which are active on another system, right?
This would save me from caring about IO fencing.

thanks for your help in advance..
matthias

RE: accessing mirrired lvm on shared storage

[Stern, Rick (Serviceguard Linux)]

Matthias,

Your best bet is to not activate the md mirrors until you need to.
There is a theoretical case where one system (A) can be writing data
while the other (B) is doing a remirror.   B reads data in prep for a
write, A writes to that same block, then B writes the old data.  

LVM does not refuse to activate volume groups.  While there is a cluster
volume manager, it doesn't help in this case as it requires an interface
to the cluster sw.

Regards,
Rick

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Matthias Eble
Sent: Thursday, April 06, 2006 11:20 AM
To: linux-raid@vger.kernel.org
Subject: accessing mirrired lvm on shared storage


Hi all,

I've got an extended setup whith two Systems, each with two FC cards. 
Every card is connected to a seperate disk array (so one system accesses

two arrays). The other node has access to the same two arrays (standby).

The active server mirrors the data (4 LUNs) between the two arrays via 
md. On top is a LVM physical volume. The other system is meant to be 
booted but not acessing the VGs.

My question is, if it is possible to let both systems set up the md 
mirror without corrupting the data? Is there any data written even when 
the VGs are not taken active? I think I remember that LVM refuses to 
activate volumegroups which are active on another system, right?
This would save me from caring about IO fencing.

thanks for your help in advance..
matthias
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: accessing mirrired lvm on shared storage

[Chris Osicki]

Matthias

I have currently four clusters which mirror shared storage. I've
always pay great attention not to have an array active on both
cluster nodes. I can imagine data corruption would happen soon or
late.
Unfortunately md lacks the ability to mark an array as
used/busy/you_name_it. Sometime ago I asked on this list for such an
enhancement (see thread with subject "Question: array locking,
possible"). Although I managed (with great help from few people on 
this list) to attract Neil's attention, I couldn't fine enough
arguments to convince him to put this topic on hist TO-DO list.
Neil, you see the constantly growing number of potential users of this
feature? ;-)


Regards,
Chris

PS Matthias, just curious, you don't have FC failover, right?

On Thu, 06 Apr 2006 20:19:53 +0200
Matthias Eble 	<matthias.eble@mailing.kaufland-informationssysteme.com> wrote:

> 
> Hi all,
> 
> I've got an extended setup whith two Systems, each with two FC cards. 
> Every card is connected to a seperate disk array (so one system accesses 
> two arrays). The other node has access to the same two arrays (standby).
> 
> The active server mirrors the data (4 LUNs) between the two arrays via 
> md. On top is a LVM physical volume. The other system is meant to be 
> booted but not acessing the VGs.
> 
> My question is, if it is possible to let both systems set up the md 
> mirror without corrupting the data? Is there any data written even when 
> the VGs are not taken active? I think I remember that LVM refuses to 
> activate volumegroups which are active on another system, right?
> This would save me from caring about IO fencing.
> 
> thanks for your help in advance..
> matthias
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: accessing mirrired lvm on shared storage

[Matthias Eble]

> Neil, you see the constantly growing number of potential users of this
> feature? ;-)

locking would be great (to me), indeed.

> PS Matthias, just curious, you don't have FC failover, right?
well, I have only 3 pci slots and can't use FC switches. the two boxes 
(server and storage) run on different locations in the building (water 
damages might be possible). Please tell me if you know a better way to 
do the job.

thanks

Re: accessing mirrired lvm on shared storage

[Neil Brown]

On Friday April 7, osk@admin.swisscom-mobile.ch wrote:
> Unfortunately md lacks the ability to mark an array as
> used/busy/you_name_it. Sometime ago I asked on this list for such an
> enhancement (see thread with subject "Question: array locking,
> possible"). Although I managed (with great help from few people on 
> this list) to attract Neil's attention, I couldn't fine enough
> arguments to convince him to put this topic on hist TO-DO list.
> Neil, you see the constantly growing number of potential users of this
> feature? ;-)

I don't think that just marking an array "don't mount" is really a
useful solution.  And if it was, it would be something done in 'mdadm'
rather than in 'md'.

What you really want is cluster wide locking using DLM or similar.
That way when the node which has active use of the array fails,
another node can pick up automatically.
Then we could put a flag in the superblock which says 'shared', and md
would need a special request to assemble such an array.

One thing that is on my todo list is supporting shared raid1, so that
several nodes in the cluster can assemble the same raid1 and access it
- providing that the clients all do proper mutual exclusion as
e.g. OCFS does.

Your desire to have only-assembled-once would be trivial to include in
that.

NeilBrown

Re: accessing mirrired lvm on shared storage

[Chris Osicki]

On Wed, 12 Apr 2006 14:09:52 +1000
Neil Brown <neilb@suse.de> wrote:

> On Friday April 7, osk@admin.swisscom-mobile.ch wrote:
> > Unfortunately md lacks the ability to mark an array as
> > used/busy/you_name_it. Sometime ago I asked on this list for such an
> > enhancement (see thread with subject "Question: array locking,
> > possible"). Although I managed (with great help from few people on 
> > this list) to attract Neil's attention, I couldn't fine enough
> > arguments to convince him to put this topic on hist TO-DO list.
> > Neil, you see the constantly growing number of potential users of this
> > feature? ;-)
> 
> I don't think that just marking an array "don't mount" is really a
> useful solution.  And if it was, it would be something done in 'mdadm'
> rather than in 'md'.

I don't think I understand this bit, sorry, but see below.

> 
> What you really want is cluster wide locking using DLM or similar.
> That way when the node which has active use of the array fails,
> another node can pick up automatically.

But I also want to prevent accidentally activation of the array on the
stand-by node. That would mean make mdadm cluster or a lock manager
aware, which we certainly don't want. 
That's why I'm asking for a flag on the array which would be far less complex 
solution than a Lock Manager.

> Then we could put a flag in the superblock which says 'shared', and md
> would need a special request to assemble such an array.

That would mean, if I get you right, "Attention, it could be used on
another host, go and check or let me assemble it anyway if you know
what you're doing". 

I was thinking about a flag saying "locked" which would
mean "array is assembled/used". 
Look, if I have a cluster (active/stand-by) when starting it I assemble
my array in exclusive mode by setting the "locked" flag. If I do a
manual fail-over of my package/service using the array in question,
when stopping the array the flag gets cleared. The node which takes
over finds the array "unlocked", locks, assembles and uses it. Now the
active node crashes without clearing the flag. It is now
responsibility of the cluster software to force the assembly of the
array on the node taking over.
And nobody can accidentally/unwittingly assemble the array on stand-by
node (without giving the option -I_know_what_I_am_doing ;-), 
which currently is my main concern as I haven't experienced any
crashes or malfunction of my clusters yet. Touching wood ....

It is how ServiceGuard cluster software on HP-UX works except that
disk mirroring and locking is done in LVM. Moreover LVM does know
about SG Cluster and prevents you from doing certain operations
depending on the current state of the cluster.

> 
> One thing that is on my todo list is supporting shared raid1, so that
> several nodes in the cluster can assemble the same raid1 and access it
> - providing that the clients all do proper mutual exclusion as
> e.g. OCFS does.
> 
> Your desire to have only-assembled-once would be trivial to include in
> that.

If this is what I described above, I hold my breath ;-)

> 
> NeilBrown
> 

Thanks very much for your time.

Regards,
Chris

Re: accessing mirrired lvm on shared storage

[Mike Snitzer]

On 4/12/06, Neil Brown <neilb@suse.de> wrote:

> One thing that is on my todo list is supporting shared raid1, so that
> several nodes in the cluster can assemble the same raid1 and access it
> - providing that the clients all do proper mutual exclusion as
> e.g. OCFS does.

Very cool... that would be extremely nice to have.  Any estimate on
when you might get to this?

Mike

Re: accessing mirrired lvm on shared storage

[Neil Brown]

On Thursday April 13, snitzer@gmail.com wrote:
> On 4/12/06, Neil Brown <neilb@suse.de> wrote:
> 
> > One thing that is on my todo list is supporting shared raid1, so that
> > several nodes in the cluster can assemble the same raid1 and access it
> > - providing that the clients all do proper mutual exclusion as
> > e.g. OCFS does.
> 
> Very cool... that would be extremely nice to have.  Any estimate on
> when you might get to this?
> 

I'm working on it, but there are lots of distractions....

The first step is getting support into the kernel for various
operations like suspending and resuming IO and resync.
That is progressing nicely.

The next step is writing a user-space app which co-ordinates
everything.  e.g. makes sure device-failed and device-added events
happen everywhere before they are acknowledged anywhere and stuff like
that.  I don't know how that will progress.
I hope to have something to demonstrate early in the second half of
the year.

NeilBrown

Re: accessing mirrired lvm on shared storage

[Mike Snitzer]

On 4/16/06, Neil Brown <neilb@suse.de> wrote:
> On Thursday April 13, snitzer@gmail.com wrote:
> > On 4/12/06, Neil Brown <neilb@suse.de> wrote:
> >
> > > One thing that is on my todo list is supporting shared raid1, so that
> > > several nodes in the cluster can assemble the same raid1 and access it
> > > - providing that the clients all do proper mutual exclusion as
> > > e.g. OCFS does.
> >
> > Very cool... that would be extremely nice to have.  Any estimate on
> > when you might get to this?
> >
>
> I'm working on it, but there are lots of distractions....
>
> The first step is getting support into the kernel for various
> operations like suspending and resuming IO and resync.
> That is progressing nicely.

Sounds good... will it be possible to suspend/resume IO to only
specific members of the raid1 (aka partial IO/resync suspend/resume)? 
If not I have a tangential raid1 suspend/resume question: is there a
better/cleaner way to suspend and resume a raid1 mirror than removing
and re-adding a member?

That is you:
1) establish a 2 disk raid1
2) suspend the mirror but allow degraded changes to occur  (remove member?)
3) after a user specified interval resume the mirror to resync (re-add member?)
4) goto 2

Using the write-intent bitmap the resync should be relatively cheap. 
However, would it be better to just use mdadm to tag a member as
write-mostly and enable write-behind on the raid1?  BUT is there a way
to set the write-behind to 0 to force a resync at a certain time (it
would appear write-behind is a create-time feature)?

thanks,
mike

Re: accessing mirrired lvm on shared storage

[Michael Tokarev]

Neil Brown wrote:
[]
>> Very cool... that would be extremely nice to have.  Any estimate on
>> when you might get to this?
>>
> 
> I'm working on it, but there are lots of distractions....

Neil, is there anything you're NOT working on? ;)

Sorry just can't resist... ;)

/mjt