[linux-lvm] Restore LVM after drive error

[linux-lvm] Restore LVM after drive error

[LoMp0101]

Hello,

I recived an smart warning for one of my scsi disk that it might faile
some day, so i added
another disk to move the lvm data from the failing disk another. But the
"new" disk faild and
the system crashed during the pvmove process. Now all lvm commands i
try say:
 "Couldn't find volume 'pvmove0' for segment 'start_extend' " and
"Couldn't read all logical volumes for volume group system".
Where 'system' is my vg. I tried pvmove --abort and vgreduce
--removemissing but nothing worked.
(now i know that vgreduce --removemissing dosn't solve the pvmove
problem). My /-Directory is whithin the system vg, so i could not access
the backup configuration of the lvm ! Is thre any way to undo the pvmove
changes and rescue my data ?!

I have searched the inet for solutions but found nothing that worked.

Please Help me !

-- 
Psssst! Schon vom neuen GMX MultiMessenger geh�rt?
Der kanns mit allen: http://www.gmx.net/de/go/multimessenger

Re: [linux-lvm] Restore LVM after drive error

[Jim Schatzman]

Sir-

In my opinion, the docs for pvmove should include the warning "do not use pvmove to copy data from a bad disk". Once you start it, it seems that you are hosed. The original pvmove fails, any attempt to resume pvmove also fails, and pvmove --abort also fails. Vgreduce also fails in this situation.

At this point, I recommend that you use one of two possible methods of partial recovery:

1) By hook or by crook, get the filesystems on the vg to mount. You may have to use dmsetup and "-P" to enable VG to assemble the volume while erroring out on a bad drive.

Unfortunately, it does seem that pvmove scrambles things so that I have not been able recover from this situation. Using "-P" does not seem to work. Again, in my opinion, folks should be warning: DO NOT ATTEMPT TO USE PVMOVE TO RECOVER FROM A BAD DISK - YOU WILL NOT LIKE THE RESULT.

Once mounted, copy all the files you can off the old file system. Copy files in small groups. When you get errors, stop immediately and shift to another directory. You may have to power off while in operation if the drive seems to be looping and your system log is filling up with error messages.

Since you are getting the "couldn't read all logical volumes" message - this suggests that Linux is unable to enable one or more of your disk drives.  Check the system log and find out why. If it is a "software reset failed", you may want to try powering down the whole thing (computer and drives) many times - maybe, eventually, the drives will come up. Once the drives are all up, the volume group should work and you should be able to mount the FS. Also, try  turning the bad drive(s) over or standing them on end. Sometimes that works. "-P" might work on "vgchange -Pay" and/or "lvchange -Pay" to enable you to use the VGs and LVs with a bad drive.

If desperate, try replacing the PC board on the bad drives. See http://myharddrivedied.com/presentations.html for detailed instructions. If you have a spare, identical drive handy, this may help.


2) I haven't tried this - it is theoretical: Acquire a working Linux system. Put your bad LVM drives in this system one at a time along with an identical new drive. Use dd on the raw device to copy from the bad drive to the new drive. Tell dd to ignore errors. This should mirror the old bad drives onto new good ones, to the extent that the old data could be read. I am not sure how to get LVM to recognize the new drive(s), but I am told that you can put the complete collection of old/good and new drives on one system and get LVM to reassemble them. Preferably copy the good files somewhere else. If you are brave, run fsck.


Some general recommendations based on my experiences:

1)  Never put anything you care about in a multi-disk JBOD volume group. One bad drive destroys the entire VG. Unreliable. Especially don't do this for your system partition. This is unfortunately, and seriously limits the usefulness of LVM, in my opinion. If LVM had better error recovery, then I would have a better opinion of it.

2)  I have had a number of cheap SATA to very expensive SCSI drives go belly up within one week. Suggest that you burn in your new drives for a week or two prior to relying on them.

3) Use RAID 5 or 6 if you can't stand to lose everything. 

Again, it is unfortunate, but pvmove does bad things if either the old drive or the new drive fail. Don't use it unless you are confident that it will succeed.  You could run "e2fsck -c" for this purpose, but it would be better to use badblocks directly.

Jim





At 07:47 AM 7/7/2007, you wrote:
>Hello,
>
>I recived an smart warning for one of my scsi disk that it might faile
>some day, so i added
>another disk to move the lvm data from the failing disk another. But the
>"new" disk faild and
>the system crashed during the pvmove process. Now all lvm commands i
>try say:
> "Couldn't find volume 'pvmove0' for segment 'start_extend' " and
>"Couldn't read all logical volumes for volume group system".
>Where 'system' is my vg. I tried pvmove --abort and vgreduce
>--removemissing but nothing worked.
>(now i know that vgreduce --removemissing dosn't solve the pvmove
>problem). My /-Directory is whithin the system vg, so i could not access
>the backup configuration of the lvm ! Is thre any way to undo the pvmove
>changes and rescue my data ?!
>
>I have searched the inet for solutions but found nothing that worked.
>
>Please Help me !
>
>-- 
>Psssst! Schon vom neuen GMX MultiMessenger geh�rt?
>Der kanns mit allen: http://www.gmx.net/de/go/multimessenger
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@redhat.com
>https://www.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Restore LVM after drive error

[Jim Schatzman]

Another note:

Re: Using dd to image a failing hard drive. Ryan Nelson in an earlier posting explained how to get your VG back online:

"vgcfgbackup, edit the backup, edit the ids in the logical_volumes
sections, check your work carefully, and vgcfgrestore?"

This will allow you to substitute the new id for the old and LVM should then be good to go. Presumably, you will have lost some data, so I would recommend mounting the FS readonly and copying files elsewhere. If you don't care what may happen, you could mount RW, run fsck and repair the filesystem.

Jim

Re: [linux-lvm] Restore LVM after drive error

[Lomp Hemul]

Hello

thanks for your replay. I have noticed that in your case the "old" disk has faild, but in my case the new disk faild. That means that the disks of the old LVM are all woring ! No bad Blocks or so. But SMART told me that one device will fail in future. So i put that bad disk in (i testet it befor - it showed no error).
My first atempt to get everything back working was to remove the device, throw it out of my hw-array config and boot. Well,i think pvmode tries to access this device , but it is no longer availible !
I have another disk that is bigger than the drive that faild. Does pvmove worke again if i put that drive in its place and start pvmove again ? Perhaps i have to mark the partition on the disk as lvm bevore, but is there a chance that pvmove just continues ? By the way how do i tell pvmove to continue ?

F F�rster  

-------- Original-Nachricht --------
Datum: Sat, 07 Jul 2007 13:12:16 -0600
Von: Jim Schatzman <james.schatzman@futurelabusa.com>
An: LVM general discussion and development <linux-lvm@redhat.com>
Betreff: Re: [linux-lvm] Restore LVM after drive error

> Another note:
> 
> Re: Using dd to image a failing hard drive. Ryan Nelson in an earlier
> posting explained how to get your VG back online:
> 
> "vgcfgbackup, edit the backup, edit the ids in the logical_volumes
> sections, check your work carefully, and vgcfgrestore?"
> 
> This will allow you to substitute the new id for the old and LVM should
> then be good to go. Presumably, you will have lost some data, so I would
> recommend mounting the FS readonly and copying files elsewhere. If you don't
> care what may happen, you could mount RW, run fsck and repair the filesystem.
> 
> Jim
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 
GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS.
Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail

Re: [linux-lvm] Restore LVM after drive error

[Jim Schatzman]

Mr F:

By no means am I an expert. My info is strictly empirical. However, I think that putting in a new drive will not directly help. Pvmove wants to continue moving to the previous new drive. 

Again, you said in one of your posts that LVM reported "Couldn't read all logical volumes for volume group system".  It seems to me that I have seen this message when LVM cannot access one physical drive of the VG. I would advise you to figure out why this is happening. I believe that this means that one of your disks is not accessible. That is bad. Either you took the drive out or it has failed. 

Assuming that this is the new/bad drive, then do the vgcfgbackup/vgcfgrestore thing while replacing the bad drive with a good one and adding the new drive to the VG. Reboot. Then execute "pvmove". With luck, pvmove will resume the move. However, I have doubts that it will work. Pvmove seems to be pretty unforgiving.

If that doesn't work, then I think that you are reduced to using one of the recovery methods.

Good luck.

Jim







At 04:15 PM 7/7/2007, you wrote:
>Hello
>
>thanks for your replay. I have noticed that in your case the "old" disk has faild, but in my case the new disk faild. That means that the disks of the old LVM are all woring ! No bad Blocks or so. But SMART told me that one device will fail in future. So i put that bad disk in (i testet it befor - it showed no error).
>My first atempt to get everything back working was to remove the device, throw it out of my hw-array config and boot. Well,i think pvmode tries to access this device , but it is no longer availible !
>I have another disk that is bigger than the drive that faild. Does pvmove worke again if i put that drive in its place and start pvmove again ? Perhaps i have to mark the partition on the disk as lvm bevore, but is there a chance that pvmove just continues ? By the way how do i tell pvmove to continue ?
>
>F F�rster  
>
>-------- Original-Nachricht --------
>Datum: Sat, 07 Jul 2007 13:12:16 -0600
>Von: Jim Schatzman <james.schatzman@futurelabusa.com>
>An: LVM general discussion and development <linux-lvm@redhat.com>
>Betreff: Re: [linux-lvm] Restore LVM after drive error
>
>> Another note:
>> 
>> Re: Using dd to image a failing hard drive. Ryan Nelson in an earlier
>> posting explained how to get your VG back online:
>> 
>> "vgcfgbackup, edit the backup, edit the ids in the logical_volumes
>> sections, check your work carefully, and vgcfgrestore?"
>> 
>> This will allow you to substitute the new id for the old and LVM should
>> then be good to go. Presumably, you will have lost some data, so I would
>> recommend mounting the FS readonly and copying files elsewhere. If you don't
>> care what may happen, you could mount RW, run fsck and repair the filesystem.
>> 
>> Jim
>> 
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>-- 
>GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS.
>Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@redhat.com
>https://www.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/