[linux-lvm] Sharing LVM's via NFS on Redhat 8.0

[linux-lvm] Sharing LVM's via NFS on Redhat 8.0

[DJ TweeQ]

Phone rings and i forget to complete my previous post :)

Regarding Sharing LVM LV's via NFS on Redhat 8.0...ie. mounting an LV from 
another machine while already having a LV on the same machine. Is it ok or 
not ok? I was talking to a buddy and he doesn't think it's a problem or an 
issue but...

the current documentation on Sistina's site states that sharing LV's is 
dangerous unless it's a SCSI or Fibre Storage Device?...and not only that 
...adminisitration can only be done on one node? (is it really possible to 
create/resize lv's from one node/machine to another (that has an LV or was 
it meant to mean that it's only possible on cluster aware network storage 
devices)??? ....

http://tldp.org/HOWTO/LVM-HOWTO/x1232.html
under "Dangerous Operations - Sharing Logical LVM Volumes"

ie.

>If you need to do any changes to the LVM metadata (regardless of >whether 
>it affects volumes mounted on other nodes) you must go through >the 
>following sequence. In the steps below ``admin node'' is any >arbirarily 
>chosen node in the cluster. Admin node                   >Other nodes
>----------                   -----------
>                             Close all Logical volumes (umount)
>                             vgchange -an
><make changes, eg lvextend>
>                             vgscan
>                             vgchange -ay

I'm thinking that it might not be a good idea to try to do maintenance on a 
LV that is mounted on another system...but i think it makes a whole heck of 
alot of sense to create/resize/mount an LV on a system before trying to 
mount it via an NFS share. :)

Any comments? I personally don't think it would be an issue either (to mount 
an LV on another box/system via an NFS mount) but i'd rather not take the 
chance of messing up any of the data on those LV's cuz their stuffed :)

Oh one other thing...if it has been proven that a Redhat 8.0 2.4-24?.x 
kernel can be patched to LVM 1.0.6+ ....will the data on those LV's be 
affected in any way or will it all be ok? :)







_________________________________________________________________
MSN 8 with e-mail virus protection service: 2 months FREE*  
http://join.msn.com/?page=features/virus

Re: [linux-lvm] Sharing LVM's via NFS on Redhat 8.0

[Patrick Caulfield]

On Tue, Feb 18, 2003 at 12:27:49AM +0000, DJ TweeQ wrote:
> 
> Phone rings and i forget to complete my previous post :)
> 
> Regarding Sharing LVM LV's via NFS on Redhat 8.0...ie. mounting an LV from 
> another machine while already having a LV on the same machine. Is it ok or 
> not ok? I was talking to a buddy and he doesn't think it's a problem or an 
> issue but...
> 
> the current documentation on Sistina's site states that sharing LV's is 
> dangerous unless it's a SCSI or Fibre Storage Device?...and not only that 
> ...adminisitration can only be done on one node? (is it really possible to 
> create/resize lv's from one node/machine to another (that has an LV or was 
> it meant to mean that it's only possible on cluster aware network storage 
> devices)??? ....
> 

Sharing over NFS is fine. What you are doing with NFS is sharing the filesystem.
The dangerous behaviour mentioned is when you are sharing the actual block
device that is the LV. A subtle but very important distinction.

This is safe:

+----+   NFS   +---------+
|    |---------|  Box B  |
|boxA|         |         | 
+----+         +---------+ 
                    | SCSI/IDE/FC
                    |
		  ^^^^^^^
		  |disks|
		  |     |
		  |     |
		  |     |
		  ^^^^^^^
		  
		  
This is dangerous:

+----+         +----+
|    |         |    |
|boxA|         |boxB| 
+----+         +----+ 
      \	      / 
       ^^^^^^^
       |disks|
       |     |
       |     |
       |     |
       ^^^^^^^
  


patrick

Re: [linux-lvm] Sharing LVM's via NFS on Redhat 8.0

[Jesse Keating]

On Friday 21 February 2003 00:22, Patrick Caulfield uttered:
> Sharing over NFS is fine. What you are doing with NFS is sharing the
> filesystem. The dangerous behaviour mentioned is when you are sharing the
> actual block device that is the LV. A subtle but very important
> distinction.

If you want to share the actual block devices, look into HyperSCSI.  It allows 
you to "share" out your block devices, and clients can access them as local 
SCSI interfaces.  2 clients can access the same LVM group concurrently, just 
as if they were both on the same system.

http://nst.dsi.a-star.edu.sg/mcsa/hyperscsi/index.html <-- hyperscsi's home

http://geek.j2solutions.net/stuff/HyperSCSI <-- A test case I ran through 
yesterday.

-- 
Jesse Keating RHCE MCSE
Pogo Linux -- Support Tech
tel: 888.828.7646

Re: [linux-lvm] Sharing LVM's via NFS on Redhat 8.0

[grobe]

Hi!

> Regarding Sharing LVM LV's via NFS on Redhat 8.0...ie. mounting an LV from
> another machine while already having a LV on the same machine. Is it ok or
> not ok? I was talking to a buddy and he doesn't think it's a problem or an
> issue but...
> the current documentation on Sistina's site states that sharing LV's is 
> dangerous unless it's a SCSI or Fibre Storage Device?...and not only that 
> ...adminisitration can only be done on one node? (is it really possible to
> create/resize lv's from one node/machine to another (that has an LV or was
> it meant to mean that it's only possible on cluster aware network storage 
> devices)??? ....

One question: did you read the various messages following your first
question?

Please understand the differende between a LVM volume and a (exported or
local) filesystem.

How do you want to share a LV which is not on a scsi or fc device? Never
heard about shared ide...;-))) Do you want to share a device or the data on the
device???

CU Lars.

-- 
+++ GMX - Mail, Messaging & more  http://www.gmx.net +++
Bitte l�cheln! Fotogalerie online mit GMX ohne eigene Homepage!

Re: [linux-lvm] Sharing LVM's via NFS on Redhat 8.0

[Jesse Keating]

On Friday 21 February 2003 09:25, grobe@gmx.net wrote:
> How do you want to share a LV which is not on a scsi or fc device? Never
> heard about shared ide...;-))) Do you want to share a device or the data on
> the device???

HyperSCSI will allow for shared IDE devices (;

-- 
Jesse Keating RHCE MCSE
Pogo Linux -- Support Tech
tel: (888) 828-7646

[linux-lvm] High level architecture (Long)

[Stephen Perkins]

[-- Attachment #1: Type: text/plain, Size: 8340 bytes --]

Hi all,

This needs to be cross-posted between several mailing lists.... But alas
there seems to be no way to coordinate that.  

Companies that have huge pockets and huge needs, for example Visa, have
exceedingly strong disaster recovery and failover operation in place and
working. Their operations can quickly fail over to multiple
geographically separated locations with small impact on their service.
The precise details of how it is done are beyond me. However, we should
be able to draw a complete picture of what _can_ be done with Linux
using all the pieces that _are_ available (and at least show what is
lacking).

I am hoping to start a discussion on the architecture of an all Linux
distributed, replicated, highly available, virtualized platform.  It
seems that many of the pieces are in place (muliple pieces in many
instances).  However, I'm looking to spark discussion on the best
architecture of such a beast.

Specifically, what I want is to get as close to possible at providing:

1) A highly available, peer replicated (i.e. active-active),
geographically separated (n locations where n=2 and n>2), virtualized,
and Backed-up SAN solution.  That's a tall order.  Companies such as
FalconStor provide some of this, but at an exceptionally high price.

2) SCSI over IP or something similar is desirable so that application
servers can use the exported virtualized storage (either locally on the
same LAN or remotely over a WAN).  Security here is a big ?.

3) A highly available virtualized application server pool that supports
automatic failover to a remote location.  A combination of Clusters,
LVS, user-mode-linux and replicated SANs can provide a potential
solution.

4) An eye towards to the cost of servers, infrastructure, colocation,
and bandwidth.  I.e. few servers at expensive locations and more servers
at less expensive and less well equipped places.  User-mode linux for
server consolidation through virtualization, etc.

5) An eye towards managmenet of the solution?  Are all these pieces
manageable without a large staff?  Are there recommended commercial
Linux solutions for management?

-----  AS AN END RESULT ---------
I would like, as an end result, to publish some type of HOW-TO on
combining all the version of sofware and hardware that are available on
1 specific architecture that will provide for this.  Since we plan on
deploying at 2 and possibly 3 separate sites, the work should easily
scale back to 1 local site (although the reverse is obviously not true).
---------------------------------

This list would be the place to ask regarding #1 and possibly #2.  Here
is what I have envisioned. But I'm not sure it is the best way to handle
things:

1) At this time, I am using RedHat 8.0 and kernel 2.4.19 (full version
... Not RedHat version).

2) I opted to use LVM 1.0.6 versus LVM2 since I'm looking at a
production environment.

3) I have 2 identical Compaq TaskSmart n2400 boxes.  These boxes have
hardware RAID-5 and currently support internal disks that are configured
as a single RAID-5 volume.  During the RH install, I partitioned the
single RAID-5 volume to leave a large chunk available as a LVM
partition.  I created a single VG on this, added the large partition as
the PVs, and can now create LVs as needed.  This seems to work fine.
Under ideal conditions, this provides a scalable and virtualized storage
pool.

4) On each system, I created identical logical volumes.  I installed
DRBD version 0.6.1 on top of LVM to export a logical volume as a network
block device.  I was able to do a network mirror between the two LVs  to
provide for a replicated logical volume.  

5) I have not addressed the security of the replicated information.
(Opted to use DRBD versus NBD or ENBD).  Just recently heard about
HyperSCSI but don't know much about it. There seem to be other packages
as well.    It is blasphemy to mention EVMS on this list?

I never completed work on fail-over scripts to automatically bring up
and export the replicated information.  Part of this is due to my
questions on the best way to "export" the data (see below).

Here are the problems:

1) Each Tasksmart is a SPOF.  It would be better to have some type of
SAN solution where I can plug in multiple LVM manager machines that are
clustered into a 2-node active-active or active-passive mechanism.
These nodes could then "export" the logical volumes to the people who
need them. I have installed heartbeat on a couple of machines and have
it working, so a small 2 node cluster is possible here.  However, then
we have the issue of how to "access" the remote storage (either fibre
channel or dual access scsi).  What are the other alternatives?  ISCSI
initiators that sit between a Cisco storage router?  How would one
virtualize the storage pool if ISCSI were involved?

2) What is the best way to "export" the replication information?  I'm
guessing it is best to do this at the block level (although I believe
products such as double-take/rsync do it at the file system level).  Is
DRBD the best solution for wide area replication?   

3) What is the best way to "export" the blocks or file system for
servers that need to access the logical volume?  NFS at the file systme
level?  iSCSI (if I can figure out how to make the logical volume a
target) at the block level?  I envision that the application servers
will actually be user-mode-linux servers that are consolidated on an
active-active Linux cluster pair.  

4) BTW, I envision that each application server will have its own
logical volume so that we can minimize the potential of concurrent
access to a logical volume.  If the app servers are clustered, then
fencing the resources is a cluster problem and not a SAN problem. What
are the concurrent access problems for access and replication.

3) What is the best way to secure the information?  What about
encryption of the replication tunnel?  Hardware VPN or a software
solution?  How important and stable etc would an encrypted file system
be?

4) What is the best way to back up the SAN solution?  We have qualstar
tape libraries available.... Commercial bakup apps?  Amanda?  Done at
the file system level or block level?  I probably need both the security
of replication (for direct failover) and tape backup (for generational
capability).

Here are the questions:

1) Is the architecture already done somewhere else (complete with fixed
versions of OS and apps that are "known to work")?  I don't want to
re-invent the wheel.

2) What would be the best way to coordinate people's feedback between
multiple mailing lists (LVM, LVS, iSCSI, etc)?

3) What would be the best way to show the architecture and apply peoples
comments and feedback?

With so many Linux projects working, it is very difficult to get a grasp
on all the pieces that would be needed to provide what I want.  However,
it does appear that all or most of the pieces are available.  I have
looked at combining:

A) Heartbeat + Keepalived + LVS  to provide a highly available LVS
director.  This small cluster can be put into a very expensive and very
highly available colocation with lots of redundant bandwidth.  It
"exports" the highly available IP addresses that are used to provide
services.  If this cluster, or the network to which it is attached goes
down, all services go down (there are no automatic solutions to solve
this type of problem with any immediacy that I can find... Things like
RR DNS etc... All suffer problems).  Through LVS TUN mode, LVS
real-servers can exist at geographically remote sites that are not as
cost-prohibitive.

B) Heartbeat to provide a high-availabilty Linux cluster that then runs
mulitple instances of "User mode linux".  This provides highly available
virtualized servers that can act as the "LVS Real-servers"

C) LVM, DRBD to provide replicated virtualized storage.  However,  I
have not figured out the best way to export this storage to the
user-mode-linux virtual servers listed in step B.

I do not have ISCSI (seem to be missing target support), secure
replication, Good clustering failover of user-mode-linux virtual servers
to geographically separate locations (in case of network failure),
backup, scalable storage short of physically adding internal or direct
attached disks to the Compaq Tasksmart machines. 

Thoughts to the rather long posting?

TIA,

- Steve

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3786 bytes --]

re: [linux-lvm] High level architecture (Long)

[Greg Freemyer]

Steve,

You are right about that being a long e-mail.  I have a Compaq (HP) background, and they call this a Disaster Tolerant SAN (DT-SAN).

Your also right about DT-SANs requiring deep pockets, and even with Linux I think it will take some fairly deep pockets.

HP has a consulting business specifically targeting DT-SANs. I don't know if they support Linux or not.  (It used to be VMS and Tru64 only.) 

http://h18000.www1.hp.com/products/storageworks/solutions/dtsan/

I have no idea where you will get a overall interest in your DT-SAN project 

Maybe someone at OSDL (http://www.osdl.org/) or on the linux-ha (heartbeat) list can give you a recommendation, or maybe you will have to start your own DT-SAN project

(A little more below)

 >>  Hi all,

 >>  1) A highly available, peer replicated (i.e. active-active),
 >>  geographically separated (n locations where n=2 and n>2), virtualized,
 >>  and Backed-up SAN solution.  That's a tall order.  Companies such as
 >>  FalconStor provide some of this, but at an exceptionally high price.

You do know how expensive that is from a communications perspective don't you.  The commercial offerings I have seen require a T3 or larger.  

I have never priced one, but I assume your talking the 10's of thousands per month if you are going between major cities.

I have priced a 100mbit connection across the north end of Atlanta.  That actually wasn't so bad.  Around $2K/month IIRC, but I think that is a rare situation with Atlanta having excess fiber in the ground.

 >>  2) SCSI over IP or something similar is desirable so that application
 >>  servers can use the exported virtualized storage (either locally on the
 >>  same LAN or remotely over a WAN).  Security here is a big ?.

I have tried the Intel iSCSI demo initiator/target pair and they seem to work well.  (I was just curious, so I did not do much testing.)

http://sourceforge.net/projects/intel-iscsi

The target software allows any Linux box to export a drive (or LV) to a Linux computer running the initiator.  

Since iSCSI was just formally approved, there should be a flurry of real targets, but I'm not sure thats what you want anyway.

Another competitive product is HyperSCSI, but I don't think HyperSCSI supports routers, so it is purely LAN.  (ie. It is a pure ethernet protocol, not a TCP/IP protocol).

 >>  3) A highly available virtualized application server pool that supports
 >>  automatic failover to a remote location.  A combination of Clusters,
 >>  LVS, user-mode-linux and replicated SANs can provide a potential
 >>  solution.

I follow clustering, and I don't know any split clustering solutions for Linux.  

One big issue is when the VIP moves, you have to update the routers to send traffic for the VIP to the new location.  This can be done, but it takes coordination with your ISP, or a lot of your own infrastructure.