[linux-lvm] cluster LVM

[linux-lvm] cluster LVM

[Jordi Prats]

Hi all,
I've been googling for a while but I found nothing about the current 
limitations of CLVM. Someone told me that you must use just one logical 
volume for each volume group. Is that true?

Anyone could tell me witch are and if exists any risks of loosing data 
by using CLVM given certain configuration? There's anything that it's 
not safe to use, like snapshots, pvmove...?

Many thanks,
Jordi

Re: [linux-lvm] cluster LVM

[Patrick Caulfield]

Jordi Prats wrote:
> Hi all,
> I've been googling for a while but I found nothing about the current
> limitations of CLVM. Someone told me that you must use just one logical
> volume for each volume group. Is that true?

No, you can have as many LVs in a VG as you like.

> Anyone could tell me witch are and if exists any risks of loosing data
> by using CLVM given certain configuration? There's anything that it's
> not safe to use, like snapshots, pvmove...?

Snapshots & pvmove are not currently cluster aware. so if you need to snapshot
or move a logical volume it needs to be active on only one node.

If you are sharing data using a filesystem, then you must use a cluster-aware
filesystem if it is to be mounted on more than one node at a time. GFS or ocfs2
are just such filesystems.


-- 
Patrick

Re: [linux-lvm] cluster LVM

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Patrick Caulfield wrote:
> Jordi Prats wrote:
>> Hi all,
>> I've been googling for a while but I found nothing about the current
>> limitations of CLVM. Someone told me that you must use just one logical
>> volume for each volume group. Is that true?
> 
> No, you can have as many LVs in a VG as you like.

I think Jordi's referring to the HA-LVM agents in rgmanager, which do
(did?) require that each VG contain only a single LV in order to
guarantee metadata consistency when a HA-LVM resource is relocated
within the cluster.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFHFcZj6YSQoMYUY94RAq4vAJ9tXKh6xz7slxR+6rF8hR0HQPa32QCdED1m
rrWCvxKZVcSctaT2zPdhwXU=
=W87k
-----END PGP SIGNATURE-----

[linux-lvm] Cluster LVM

[Gitansh Chadha]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

hi....
well i am an undergraduate student from Pune ,India and we implementing a cluster LVM as our final year project.
we are manipulating   vg_status in vg_t structure to determine the status the of a volume group.basically playing with the bits in it.so we can have a shared and exclusive locked volume group.
we also have a administrator which communicates with all nodes in the cluster and basically to maintain the consistency of the LVMTAB.
any suggestions?

gitansh




[-- Attachment #2: Type: text/html, Size: 1093 bytes --]

Re: [linux-lvm] Cluster LVM

[Jeff Layton]

Gitansh,

   God bless you! I've been asking for this for a long time, but
the Sistina people who sponsor LVM are moving to a closed
commercial solution.
   I can't answer your questions because I don't really do too
much low level programming, but I would love to be kept in
the loop on your progress.

Thanks very much and Good Luck!

Jeff Layton

Lockheed-Martin


Gitansh Chadha wrote:

> hi....well i am an undergraduate student from Pune ,India and we implementing a cluster LVM as our final year
> project.we are manipulating   vg_status in vg_t structure to determine the status the of a volume group.basically
> playing with the bits in it.so we can have a shared and exclusive locked volume group.we also have a administrator
> which communicates with all nodes in the cluster and basically to maintain the consistency of the LVMTAB.any
> suggestions? gitansh

Re: [linux-lvm] Cluster LVM

[Remco Post]

Hi,

clusters all have one major problem to solve: split brain. Make sure that the 
node manipulating the LV/VG does indeed have the right to do so, and doesn't 
screw up everything for everybody. Maybe introduce a (LV/VG) master concept 
into the VG, so as long as that is set, no other node in the cluster will 
manipulate anything in the vg. (just some random rant, ignore if I'm talking 
nonsense ;)


> Gitansh,
> 
>    God bless you! I've been asking for this for a long time, but
> the Sistina people who sponsor LVM are moving to a closed
> commercial solution.
>    I can't answer your questions because I don't really do too
> much low level programming, but I would love to be kept in
> the loop on your progress.
> 
> Thanks very much and Good Luck!
> 
> Jeff Layton
> 
> Lockheed-Martin
> 
> 
> Gitansh Chadha wrote:
> 
> > hi....well i am an undergraduate student from Pune ,India and we implementing a cluster LVM as our final year
> > project.we are manipulating   vg_status in vg_t structure to determine the status the of a volume group.basically
> > playing with the bits in it.so we can have a shared and exclusive locked volume group.we also have a administrator
> > which communicates with all nodes in the cluster and basically to maintain the consistency of the LVMTAB.any
> > suggestions? gitansh
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
> 

-- 
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: [linux-lvm] Cluster LVM

[Andreas Dilger]

On Feb 27, 2002  12:45 +0100, Remco Post wrote:
> clusters all have one major problem to solve: split brain. Make sure that the 
> node manipulating the LV/VG does indeed have the right to do so, and doesn't 
> screw up everything for everybody. Maybe introduce a (LV/VG) master concept 
> into the VG, so as long as that is set, no other node in the cluster will 
> manipulate anything in the vg. (just some random rant, ignore if I'm talking 
> nonsense ;)

Having a single master is a _terrible_ idea, since this would mean that
if that node is down you are unable to change the VG until/if it is back
working again.

Rather use a DLM (as the other poster suggested) and a client needs to
get a lock on the VG in order to change anything.  You can start with
a lock per VG and one per shared PV not in a VG, and then go more fine
grained if you want (e.g. write lock on VG and specific LV being changed,
which does not block users of other LVs).  In the end, because you are
not changing a VG configuration very often, and it doesn't take very long
when you do (excluding pvmove), the big lock is probably enough.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] Cluster LVM

[Ragnar Kjørstad]

On Wed, Feb 27, 2002 at 10:19:49AM -0700, Andreas Dilger wrote:
> On Feb 27, 2002  12:45 +0100, Remco Post wrote:
> > clusters all have one major problem to solve: split brain. Make sure that the 
> > node manipulating the LV/VG does indeed have the right to do so, and doesn't 
> > screw up everything for everybody. Maybe introduce a (LV/VG) master concept 
> > into the VG, so as long as that is set, no other node in the cluster will 
> > manipulate anything in the vg. (just some random rant, ignore if I'm talking 
> > nonsense ;)
> 
> Having a single master is a _terrible_ idea, since this would mean that
> if that node is down you are unable to change the VG until/if it is back
> working again.

The VG being locked is not _that_ terrible - after all - you only need
this to change your storage configuration (I don't expet the LVM to
actually lock access to the device).

> Rather use a DLM (as the other poster suggested) and a client needs to
> get a lock on the VG in order to change anything.  You can start with
> a lock per VG and one per shared PV not in a VG, and then go more fine
> grained if you want (e.g. write lock on VG and specific LV being changed,
> which does not block users of other LVs).  In the end, because you are
> not changing a VG configuration very often, and it doesn't take very long
> when you do (excluding pvmove), the big lock is probably enough.

Alternatively one could implement the cluster LVM as a simple
"resource", and leave it to the resource manager to do locking and
STONITH. The ocf (Open Cluster Framework) or "linux-ha" lists would be
good places to query for more information.



-- 
Ragnar Kjørstad
Big Storage

Re: [linux-lvm] Cluster LVM

[Remco Post]

> The VG being locked is not _that_ terrible - after all - you only need
> this to change your storage configuration (I don't expet the LVM to
> actually lock access to the device).

What happens with a lvextend? I guess it's perfectly possible to extend an lv while the other node(s) in the cluster are accessing it. Or are ther any problems? Problems that arise during the fs-grow operation are up to the fs anyway, right?


-- 
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: [linux-lvm] Cluster LVM

[Prashant Kharche]

we have studied the DLM of IBM and basically we are
impleneting a similar concept.we have a lock manger
daemon at the administartor and a client daemon at the
nodes.The main aim is to maintain the consistency of
the metadata.When tools are run they run with a lock
and and at that time the filesystem cannot access the
volume groups.
also at the release of lock vgscan at all nodes leads
to the updation of the metadata.

Prashant




--- Remco Post <r.post@sara.nl> wrote:
> 
> > The VG being locked is not _that_ terrible - after
> all - you only need
> > this to change your storage configuration (I don't
> expet the LVM to
> > actually lock access to the device).
> 
> What happens with a lvextend? I guess it's perfectly
> possible to extend an lv while the other node(s) in
> the cluster are accessing it. Or are ther any
> problems? Problems that arise during the fs-grow
> operation are up to the fs anyway, right?
> 
> 
> -- 
> Met vriendelijke groeten,
> 
> Remco Post
> 
> SARA - Stichting Academisch Rekencentrum Amsterdam
> High Performance Computing  Tel. +31 20 592 8008   
> Fax. +31 20 668 3167
> 
> "I really didn't foresee the Internet. But then,
> neither did the computer
> industry. Not that that tells us very much of course
> - the computer industry
> didn't even foresee that the century was going to
> end." -- Douglas Adams
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

__________________________________________________
Do You Yahoo!?
Yahoo! Greetings - Send FREE e-cards for every occasion!
http://greetings.yahoo.com

Re: [linux-lvm] Cluster LVM

[Joe Thornber]

On Thu, Feb 28, 2002 at 02:59:51AM -0800, Prashant Kharche wrote:
> When tools are run they run with a lock
> and and at that time the filesystem cannot access the
> volume groups.

How are you preventing remote nodes from accessing the volume groups ?
Or are you working on an 'offline' solution, where there can be no
users of logical volumes while you resize them ?

- Joe

Re: [linux-lvm] Cluster LVM

[Prashant Kharche]

  Basically, we are putting a lock on the VG while it
is getting resized or for any tool which tries to
access VG .. So that no other user can access the
metadata of that VG.. 
WE ARE ONLY MATAINING THE CONSISTENCY OF THE VG
METADATA.. 
We thought of protecting VG DATA (NOT METADATA), but
it becomes very completed.. so as if now we are not
taking care of FS access..  
--- Joe Thornber <joe@fib011235813.fsnet.co.uk> wrote:
> On Thu, Feb 28, 2002 at 02:59:51AM -0800, Prashant
> Kharche wrote:
> > When tools are run they run with a lock
> > and and at that time the filesystem cannot access
> the
> > volume groups.
> 
> How are you preventing remote nodes from accessing
> the volume groups ?
> Or are you working on an 'offline' solution, where
> there can be no
> users of logical volumes while you resize them ?
> 
> - Joe
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

__________________________________________________
Do You Yahoo!?
Yahoo! Greetings - Send FREE e-cards for every occasion!
http://greetings.yahoo.com

Re: [linux-lvm] Cluster LVM

[Joe Thornber]

On Thu, Feb 28, 2002 at 05:03:37AM -0800, Prashant Kharche wrote:
> 
>   Basically, we are putting a lock on the VG while it
> is getting resized or for any tool which tries to
> access VG .. So that no other user can access the
> metadata of that VG.. 
> WE ARE ONLY MATAINING THE CONSISTENCY OF THE VG
> METADATA.. 
> We thought of protecting VG DATA (NOT METADATA), but
> it becomes very completed.. so as if now we are not
> taking care of FS access..  

You need to take care that the volume groups running on all nodes
*always* reflect the metadata on disk.

For instance, if have two nodes running vg0/lvol0, and I issue a
pvmove from node1, when does node2 find out that the mapping has
changed ?

- Joe

Re: [linux-lvm] Cluster LVM

[Prashant Kharche]

--- Joe Thornber <joe@fib011235813.fsnet.co.uk> wrote:
> On Thu, Feb 28, 2002 at 05:03:37AM -0800, Prashant
> Kharche wrote:
> > 
> >   Basically, we are putting a lock on the VG while
> it
> > is getting resized or for any tool which tries to
> > access VG .. So that no other user can access the
> > metadata of that VG.. 
> > WE ARE ONLY MATAINING THE CONSISTENCY OF THE VG
> > METADATA.. 
> > We thought of protecting VG DATA (NOT METADATA),
> but
> > it becomes very completed.. so as if now we are
> not
> > taking care of FS access..  
> 
> You need to take care that the volume groups running
> on all nodes
> *always* reflect the metadata on disk.
> 
> For instance, if have two nodes running vg0/lvol0,
> and I issue a
> pvmove from node1, when does node2 find out that the
> mapping has
> changed ?
> 
> - Joe
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at
http://www.sistina.com/lvm/Pages/howto.html

when node1 issues a command pvmove, LOCK MANAGER will
block all the nodes from accessing the METADATA for
that VG and as soon as it finishes updating the
METADATA, pvmove itself writes the updated kernel VGDA
on the DISK.. so for this period, METADATA consistency
is maintained. 

__________________________________________________
Do You Yahoo!?
Yahoo! Greetings - Send FREE e-cards for every occasion!
http://greetings.yahoo.com

Re: [linux-lvm] Cluster LVM

[Remco Post]

> 
> --- Joe Thornber <joe@fib011235813.fsnet.co.uk> wrote:
> > On Thu, Feb 28, 2002 at 05:03:37AM -0800, Prashant
> > Kharche wrote:
> > > 
> > >   Basically, we are putting a lock on the VG while
> > it
> > > is getting resized or for any tool which tries to
> > > access VG .. So that no other user can access the
> > > metadata of that VG.. 
> > > WE ARE ONLY MATAINING THE CONSISTENCY OF THE VG
> > > METADATA.. 
> > > We thought of protecting VG DATA (NOT METADATA),
> > but
> > > it becomes very completed.. so as if now we are
> > not
> > > taking care of FS access..  
> > 
> > You need to take care that the volume groups running
> > on all nodes
> > *always* reflect the metadata on disk.
> > 
> > For instance, if have two nodes running vg0/lvol0,
> > and I issue a
> > pvmove from node1, when does node2 find out that the
> > mapping has
> > changed ?
> > 
> > - Joe
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at
> http://www.sistina.com/lvm/Pages/howto.html
> 
> when node1 issues a command pvmove, LOCK MANAGER will
> block all the nodes from accessing the METADATA for
> that VG and as soon as it finishes updating the
> METADATA, pvmove itself writes the updated kernel VGDA
> on the DISK.. so for this period, METADATA consistency
> is maintained. 
> 

During the pv move, not only needs the metadata be protected, but also the fs 
data,eg. when one server is moving a data block from one disk tot the other, 
all other servers must be notified that that block is currently inaccessable, 
so no fs data will get lost during transaction, or am I missing something here?


-- 
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: [linux-lvm] Cluster LVM

[Joe Thornber]

On Thu, Feb 28, 2002 at 07:06:58AM -0800, Prashant Kharche wrote:
> when node1 issues a command pvmove, LOCK MANAGER will
> block all the nodes from accessing the METADATA for
> that VG and as soon as it finishes updating the
> METADATA, pvmove itself writes the updated kernel VGDA
> on the DISK.. so for this period, METADATA consistency
> is maintained. 

This is not enough, you will end up with a trashed system since node2
will still be writing data to the wrong place.  I think you should
consider only doing offline operations.  ie, acquiring the lock should
fail if any other nodes have activated the vg.

If you really insist on doing live operations I suggest you look at
the device-mapper driver (LVM2), which contains the suspend
functionality that you will need.

- Joe

Re: [linux-lvm] Cluster LVM

[Remco Post]

> Having a single master is a _terrible_ idea, since this would mean that
> if that node is down you are unable to change the VG until/if it is back
> working again.
> 
> Rather use a DLM (as the other poster suggested) and a client needs to
> get a lock on the VG in order to change anything.  You can start with
> a lock per VG and one per shared PV not in a VG, and then go more fine
> grained if you want (e.g. write lock on VG and specific LV being changed,
> which does not block users of other LVs).  In the end, because you are
> not changing a VG configuration very often, and it doesn't take very long
> when you do (excluding pvmove), the big lock is probably enough.
> 
> Cheers, Andreas

Agreed,

I was unclear about my ideas. Having the one big log is probably all you ever need, since all manipulations on a lv or a pv in a vg have impact on the vg.


-- 
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: [linux-lvm] Cluster LVM

[tim]

Quoth Gitansh Chadha:
> hi....
> well i am an undergraduate student from Pune ,India and we implementing a cluster LVM as our final year project.
> we are manipulating   vg_status in vg_t structure to determine the status the of a volume group.basically playing with the bits in it.so we can have a shared and exclusive locked volume group.
> we also have a administrator which communicates with all nodes in the cluster and basically to maintain the consistency of the LVMTAB.
> any suggestions?

Have you looked at IBM's old Distributed Lock Manager?  I believe it was
used in DFS/DCE for pretty much the same type of thing.  If not, you may
be able to get some good algorithms out of it. 

http://www-124.ibm.com/developerworks/projects/dlm

The reason I would consider implementing code that's already been
tested is that subtle bugs in your implementation would almost certainly
hose any VG that had two "owners" at a point in time.  Al Bazinet at IBM
always seemed to be finding new possibilities for this in DFS.  Likewise
I have become a bit gun-shy about kernelspace changes since hosing my
installation's "live" storage for 30 minutes (recovery & failover went
fine, but apocalyptic unscheduled downtime always sucks, doesn't it?).
I was using a much more primitive mirroring-the-LVMTAB method with
non-shared drives in a Heartbeat cluster, and the outage was due to VM
corruption within the kernel affecting a snapshot vgchange (see 1.0.3).
The old saw, "if you're going to re-invent the wheel, at least make it
round" applies here as ever.  Very clever idea you have come up with,
I'd just suggest borrowing tested code to shorten the "beta" period.

-- 
  "What I feel for her isn't simple hate. It is an all-encompassing 
   repulsion not unlike what you might feel if you woke up to discover
   a four-pound cockroach using your toothbrush."
      --Eric Umansky, in a Washington Post piece on Alanis Morrissette

[linux-lvm] Cluster LVM

[Jos Visser]

Hi,

I know you guys are working on Cluster LVM. Do you have something like
specs or a design document that I could peruse and maybe comment on?

++Jos

-- 
Each betrayal begins with trust.

Re: [linux-lvm] Cluster LVM

[Heinz J. Mauelshagen]

On Wed, Mar 14, 2001 at 04:22:43PM +0100, Jos Visser wrote:
> Hi,
> 
> I know you guys are working on Cluster LVM.

Yes, we need to have more capacity for it after
NON-Cluster-LVM stabilization and the related 1.0 release.

> Do you have something like
> specs or a design document that I could peruse and maybe comment on?

We'ld really appreciate your comments etc on our spec and 
will probably be able to provide it in April.

> 
> ++Jos
> 
> -- 
> Each betrayal begins with trust.
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Cluster LVM

[Jeffrey B Layton]

"Heinz J. Mauelshagen" wrote:

> On Wed, Mar 14, 2001 at 04:22:43PM +0100, Jos Visser wrote:
> > Hi,
> >
> > I know you guys are working on Cluster LVM.
>
> Yes, we need to have more capacity for it after
> NON-Cluster-LVM stabilization and the related 1.0 release.

I'm not familiar with Cluster LVM since I don't subscribe
to the developers list. Is this something along the lines of
PVFS?

Thanks,

Jeff


>
>
> > Do you have something like
> > specs or a design document that I could peruse and maybe comment on?
>
> We'ld really appreciate your comments etc on our spec and
> will probably be able to provide it in April.
>
> >
> > ++Jos
> >
> > --
> > Each betrayal begins with trust.
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
>
> --
>
> Regards,
> Heinz    -- The LVM Guy --
>
> *** Software bugs are stupid.
>     Nevertheless it needs not so stupid people to solve them ***
>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> Heinz Mauelshagen                                 Sistina Software Inc.
> Senior Consultant/Developer                       Am Sonnenhang 11
>                                                   56242 Marienrachdorf
>                                                   Germany
> Mauelshagen@Sistina.com                           +49 2626 141200
>                                                        FAX 924446
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm