[linux-lvm] Lvm Strange Problem

[linux-lvm] Lvm Strange Problem

[Jörg Stephan]

Hi all,

i have some trouble at work caused by strange thing happend at our lvm
volumes. I hope that some of you have any idea, because i'am out of any
good one.

We use an EMC Storage system with iscsi volume on different xen hosts.
These gentoo hardened xen hosts have an lv on that iscsi target. Now
there are many machines running for more than 400 days. Some days ago
the datacenter moved over to a new building and all the machines were
turned off.
We powered them up, and about 50% of the more than 30 servers on the
iscsi were timestamped 23. feb 2010.

For now it seems that most of the freezed systems were snapshoted last year.

So, any hint would be useful

Thx

Joerg

Re: [linux-lvm] Lvm Strange Problem

[Stuart D. Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 754 bytes --]

On Mon, 11 Oct 2010, J�rg Stephan wrote:

> We powered them up, and about 50% of the more than 30 servers on the
> iscsi were timestamped 23. feb 2010.
> 
> For now it seems that most of the freezed systems were snapshoted last year.

It seems pretty clear that all your servers have both a snapshot and a
live LV.  At power up, they search for their disk and randomly find
either the live or the snapshot first.  You need to take the snapshots 
out of the search path.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Lvm Strange Problem

[Jörg Stephan]

Am 11.10.2010 14:40, schrieb Stuart D. Gathman:
> On Mon, 11 Oct 2010, J�rg Stephan wrote:
>
>   
>> We powered them up, and about 50% of the more than 30 servers on the
>> iscsi were timestamped 23. feb 2010.
>>
>> For now it seems that most of the freezed systems were snapshoted last year.
>>     
> It seems pretty clear that all your servers have both a snapshot and a
> live LV.  At power up, they search for their disk and randomly find
> either the live or the snapshot first.  You need to take the snapshots 
> out of the search path.
>
>   

Hi again,

well, mostly the snapshot was removed some days after they were made.
Also the snapshots had of course different names, how could it be that
they were used? And even if this, why are all machines with the failure
bound to the same date?

Greets

J�rg

Re: [linux-lvm] Lvm Strange Problem

[Stuart D. Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1301 bytes --]

On Mon, 11 Oct 2010, J�rg Stephan wrote:

> > It seems pretty clear that all your servers have both a snapshot and a
> > live LV.  At power up, they search for their disk and randomly find
> > either the live or the snapshot first.  You need to take the snapshots 
> > out of the search path.
> >
> well, mostly the snapshot was removed some days after they were made.
> Also the snapshots had of course different names, how could it be that
> they were used? And even if this, why are all machines with the failure
> bound to the same date?

I believe your problem is with your SAN, not LVM (you did mention using iSCSI).
I'm talking about SAN snapshots, not LVM snapshots (although the SAN server 
could very well use LVM underneath).  The SAN volumes have no names, just
LUNs.  At boot, they are searched for a matching UUID, just like directly
attached physical volumes.  When taking a SAN snapshot (clone), the UUID
of the clone is the same as the original, whether a filesystem or VG is
on the SAN LUN.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Lvm Strange Problem

[Jörg Stephan]

> I believe your problem is with your SAN, not LVM (you did mention using iSCSI).
> I'm talking about SAN snapshots, not LVM snapshots (although the SAN server 
> could very well use LVM underneath).  The SAN volumes have no names, just
> LUNs.  At boot, they are searched for a matching UUID, just like directly
> attached physical volumes.  When taking a SAN snapshot (clone), the UUID
> of the clone is the same as the original, whether a filesystem or VG is
> on the SAN LUN.
>
>   

Okay, but we dont use SAN snapshots, we use an lvcreate --snapshot on
the client. So the storage just dont know about them.

What happens is, we lv snapshot a volume, mount it copy the data to the
new volume we create after snapshooting the source, remove the old
snapshot and turning of the new machine. It rans about half an year,
gets stoped, and after it turning on again the lvm data on the iscsi
volume is last cahnged on the 23 februar 2010, even if the snapshot was
made much earlier.

Greets

Re: [linux-lvm] Lvm Strange Problem

[Stuart D. Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1475 bytes --]

On Mon, 11 Oct 2010, J�rg Stephan wrote:

> > I believe your problem is with your SAN, not LVM (you did mention using
> > iSCSI).  I'm talking about SAN snapshots, not LVM snapshots (although the
> > SAN server could very well use LVM underneath).  The SAN volumes have no
> > names, just LUNs.  At boot, they are searched for a matching UUID, just
> > like directly attached physical volumes.  When taking a SAN snapshot
> > (clone), the UUID of the clone is the same as the original, whether a
> > filesystem or VG is on the SAN LUN.
> 
> Okay, but we dont use SAN snapshots, we use an lvcreate --snapshot on
> the client. So the storage just dont know about them.
> 
> What happens is, we lv snapshot a volume, mount it copy the data to the
> new volume we create after snapshooting the source, remove the old
> snapshot and turning of the new machine. It rans about half an year,
> gets stoped, and after it turning on again the lvm data on the iscsi
> volume is last cahnged on the 23 februar 2010, even if the snapshot was
> made much earlier.

If the contents of (50% of) your SAN disks have reverted, then your issue
is with SAN (iSCSI is a SAN protocol), not with LVM.  

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Lvm Strange Problem

[Stuart D. Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 930 bytes --]

On Mon, 11 Oct 2010, Stuart D. Gathman wrote:

> On Mon, 11 Oct 2010, J�rg Stephan wrote:
> 
> > We powered them up, and about 50% of the more than 30 servers on the
> > iscsi were timestamped 23. feb 2010.
> > 
> > For now it seems that most of the freezed systems were snapshoted last year.
> 
> It seems pretty clear that all your servers have both a snapshot and a
> live LV.  At power up, they search for their disk and randomly find
> either the live or the snapshot first.  You need to take the snapshots 
> out of the search path.

Alternatively, you might change the UUID of the contained VG or filesystem
when taking a snapshot.  

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Lvm Strange Problem

[beswars]

Sent from my HTC Touch Pro2 on the Now Network from Sprint�.

-----Original Message-----
From: J�rg Stephan <ml@johestephan.de>
Sent: Monday, October 11, 2010 7:23
To: linux-lvm@redhat.com
Subject: Re: [linux-lvm] Lvm Strange Problem


> I believe your problem is with your SAN, not LVM (you did mention using iSCSI).
> I'm talking about SAN snapshots, not LVM snapshots (although the SAN server 
> could very well use LVM underneath).  The SAN volumes have no names, just
> LUNs.  At boot, they are searched for a matching UUID, just like directly
> attached physical volumes.  When taking a SAN snapshot (clone), the UUID
> of the clone is the same as the original, whether a filesystem or VG is
> on the SAN LUN.
>
>   

Okay, but we dont use SAN snapshots, we use an lvcreate --snapshot on
the client. So the storage just dont know about them.

What happens is, we lv snapshot a volume, mount it copy the data to the
new volume we create after snapshooting the source, remove the old
snapshot and turning of the new machine. It rans about half an year,
gets stoped, and after it turning on again the lvm data on the iscsi
volume is last cahnged on the 23 februar 2010, even if the snapshot was
made much earlier.

Greets

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/