[linux-lvm] Serious problem and I don't know where to turn.

[linux-lvm] Serious problem and I don't know where to turn.

[WarlockD]

[-- Attachment #1: Type: text/plain, Size: 3527 bytes --]

I screwed up, big time.  If it was just one or two mistakes than I think I
could of solved it, but now I am not sure how to fix this.

 

Some back ground:  I recently picked up a DSN-3200 cheap off ebay.  Its
basically an Iscsi SAN system that holds 15 drives.  VERY simplistic but
hey, does RAID5.  I added 5 drives and set up a simple LVM2 group off the
first partition of the array.  (/dev/sdb1)  Followed the directions and made
a logical drive, formatted to ext3 and everything was great.

 

A few months latter Fry's was having a 500gb drive sale, so I picked up 5
more drives and expanded the array (The DSN only tapes on extra drives to
the existing one, rather than expanding the array a drive at a time.  Think
JOBD with RAID5 arrays)  I now had an extra, or exact 1.7tb of space.  I
create another LVM2 partition (/dev/sdb2), expand both the VG and LV.

 

This would have been all fine, except I ran into an issue with e2fsck.  To
expand the ext3, it wanted me to run e2fsck -f on the volume.  The problem
was that eveytime I run the command, it would seem to lock up the iscsi
driver.  I tried looking for any kind of hints on why this was with no
avail.  After hours of working on it, I threw up my hands.  I figured if I
can never e2fsck the volume, it might be bad for me to extend it in the
first place.  At the time, I didn't have a way to back up the array, so I
did something stupid.

 

I deleted the /dev/sdb2 partition.  Mind you, this was after a few days and
the line of thinking was that I would delete it, create an ext3 partition,
and copy the data over.  Of course it gave a lot of errors in LVM2
afterwards.  I could still mount the volume so I thought I was ok.  

 

A day or two passes and I wake up to find the server has locked up.  I do a
hard reboot to find the little 160gig IDE drive in it has failed.  Doesn't
spin up on power.  There goes my backup metadata.

 

Sigh.  I would have been better off with an ext3 partition to begin with,
but I liked the flexibility that LVM2 offered.  To be honest I don't
remember a lot of the commands I have used either.  Any help would be
appreciated.  It even looks like I lost the logical drive.  But I know the
data is still there, is there any kind of utility I can just "get" the ext3
partition out of there?

 

 

[root@server archive]# vgscan

  Reading all physical volumes.  This may take a while...

  Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

  Found volume group "VolGroup00" using metadata type lvm2

  Found volume group "ANIME" using metadata type lvm2

[root@server archive]# ls

VolGroup00_00000.vg

[root@server archive]# vgchange -a y

  Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

  2 logical volume(s) in volume group "VolGroup00" now active

  0 logical volume(s) in volume group "ANIME" now active

[root@server archive]# pvscan

  Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

  PV /dev/hda2   VG VolGroup00   lvm2 [148.94 GB / 0    free]

  PV /dev/sdb1   VG ANIME        lvm2 [1.82 TB / 1.82 TB free]

  PV /dev/sda2                   lvm2 [1.82 TB]

  Total: 3 [3.78 TB] / in use: 2 [1.96 TB] / in no VG: 1 [1.82 TB]

[root@server archive]# lvscan

  Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

  ACTIVE            '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

  ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server archive]#

 

 


[-- Attachment #2: Type: text/html, Size: 7185 bytes --]

Re: [linux-lvm] Serious problem and I don't know where to turn.

[Richard van den Berg]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

WarlockD wrote:
> [root@server archive]# vgscan
>
>   Reading all physical volumes.  This may take a while...
>
>   Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1
> not /dev/sda1
>

How did this happen? Why is the PV magic number on /dev/sdb1 exactly the
same as on /dev/sda1?

> [root@server archive]# lvscan
>
>   Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1
> not /dev/sda1
>
>   ACTIVE            '/dev/VolGroup00/LogVol00' [147.00 GB] inherit
>
>   ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
>
> [root@server archive]#
>

I looks like your lvs are up. Can't you just mount them?

Sincerely,

Richard van den Berg


[-- Attachment #2: Type: text/html, Size: 2731 bytes --]

RE: [linux-lvm] Serious problem and I don't know where to turn.

[WarlockD]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

AH, well for the first issue it was because I had the iScsi portal's
incorrect.  So it was doing two separate connections under my two Ethernet
cards instead of a single bonded connection, that issue is fixed.

 

The two ACTIVE lvm's are the boot drive.  This is a default Centros install
so it sets the drive up like that.  Here is a better out

 

 

[root@server ~]# lvscan -a

  ACTIVE            '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

  ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server ~]# vgscan

  Reading all physical volumes.  This may take a while...

  Found volume group "VolGroup00" using metadata type lvm2

  Found volume group "ANIME" using metadata type lvm2

 

Basicly, I have a volume called "ANIME-logical" I am trying to restore, but
I don't have the metadata.  Is there a way I can get to it?

 

From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On
Behalf Of Richard van den Berg
Sent: Thursday, June 28, 2007 4:52 PM
To: LVM general discussion and development
Subject: Re: [linux-lvm] Serious problem and I don't know where to turn.

 

WarlockD wrote: 

[root@server archive]# vgscan 

  Reading all physical volumes.  This may take a while...

  Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1


How did this happen? Why is the PV magic number on /dev/sdb1 exactly the
same as on /dev/sda1?




[root@server archive]# lvscan

  Found duplicate PV 5Up3QCLOFKXTDA1OHoWUfMrYne4SdXmb: using /dev/sdb1 not
/dev/sda1

  ACTIVE            '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

  ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server archive]#


I looks like your lvs are up. Can't you just mount them?

Sincerely,



Richard van den Berg

[-- Attachment #2: Type: text/html, Size: 6579 bytes --]

Re: [linux-lvm] Serious problem and I don't know where to turn.

[Richard van den Berg]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

WarlockD wrote:
>
> [root@server ~]# lvscan -a
>
>   ACTIVE            '/dev/VolGroup00/LogVol00' [147.00 GB] inherit
>
>   ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
>
> [root@server ~]# vgscan
>
>   Reading all physical volumes.  This may take a while...
>
>   Found volume group "VolGroup00" using metadata type lvm2
>
>   Found volume group "ANIME" using metadata type lvm2

So the vg "ANIME" is found, but is not activated? That kind of makes
sense because it is missing the pv on the deleted /dev/sdb2 partition.
According to the vgreduce man pages you can:

vgreduce --removemissing ANIME

to remove all  missing physical volumes from the volume group and make
the volume group consistent again. It's a good idea to run this option
with --test  first to find out what it would remove before running it
for real. There is also "--partial" option you can use to activate vgs
and lvs even if they are missing data. Read the man pages and be very
careful before you proceed.

> Basicly, I have a volume called “ANIME-logical” I am trying to
> restore, but I don’t have the metadata.  Is there a way I can get to it?
>

Without the metadata that's going to be tough. If the filesystem is all
continuous on your RAID5 partition, technically you should be able to
mount it if you can create a device pointing at the right start, and
with the right size. Good luck figuring that out.

Another approach could be to recreate the /dev/sdb2 partition in exactly
the same way you did before. The metadata might still be there, and the
vg might come up.

Sincerely,

Richard van den Berg

[-- Attachment #2: Type: text/html, Size: 4436 bytes --]

RE: [linux-lvm] Serious problem and I don't know where to turn.

[WarlockD]

[-- Attachment #1: Type: text/plain, Size: 2545 bytes --]

The good news is the logical drive came up.  I just recreated the logical drive exactly as it was and its even detected, with a missing superblock, as an ext3 partition. The bad news is that e2fsck still froze on me.  I can’t tell if it’s the iSCSI driver that is doing it (The logical drive comes up as “busy” when I try to access it after killing e2fsck) or something with lvm2 as doubtful as that is.

 

I picked up some hard drives and put it in my old raid card for a RAID0, and doing a dd if=/dev/ANIME/testlv of=/dev/sdc1.  If this works, and e2fsck works, then I got my data back, yea!

 

Now what to do with it:P

 

From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Richard van den Berg
Sent: Friday, June 29, 2007 1:23 AM
To: LVM general discussion and development
Subject: Re: [linux-lvm] Serious problem and I don't know where to turn.

 

WarlockD wrote:



[root@server ~]# lvscan -a

  ACTIVE            '/dev/VolGroup00/LogVol00' [147.00 GB] inherit

  ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit

[root@server ~]# vgscan

  Reading all physical volumes.  This may take a while...

  Found volume group "VolGroup00" using metadata type lvm2

  Found volume group "ANIME" using metadata type lvm2


So the vg "ANIME" is found, but is not activated? That kind of makes sense because it is missing the pv on the deleted /dev/sdb2 partition. According to the vgreduce man pages you can:

vgreduce --removemissing ANIME

to remove all  missing physical volumes from the volume group and make the volume group consistent again. It's a good idea to run this option with --test  first to find out what it would remove before running it for real. There is also "--partial" option you can use to activate vgs and lvs even if they are missing data. Read the man pages and be very careful before you proceed.




Basicly, I have a volume called “ANIME-logical” I am trying to restore, but I don’t have the metadata.  Is there a way I can get to it?


Without the metadata that's going to be tough. If the filesystem is all continuous on your RAID5 partition, technically you should be able to mount it if you can create a device pointing at the right start, and with the right size. Good luck figuring that out.

Another approach could be to recreate the /dev/sdb2 partition in exactly the same way you did before. The metadata might still be there, and the vg might come up.

Sincerely,

Richard van den Berg


[-- Attachment #2: Type: text/html, Size: 7091 bytes --]