[linux-lvm] waah waah - disaster recovery

[linux-lvm] waah waah - disaster recovery

[Paul Pianta]

hi people

I have a major problem involving lvm and quite possibly software raid.
Here's how it goes ...

I installed a machine with 2 identical 200Gb disks and I set them up
using software RAID (on Fedora Core 2). I made three partitions on each
disk. /boot and / were created on md0 and md1 respectively using ext3.

The third partition was setup as a large volume group on md2 which was
mounted at /data. Within this vg I set up 3 logical volumes lv-backup,
lv-isos, lv-repositories.

The installation and setup went great and all was good for a while until
the first hard disk /dev/hda died. No problem - I had installed grub on
the mirror disk (/dev/hdc) and so was able to boot into the system.

Yet again all was going along fine, running on one disk while waiting
for a replacement disk to come back from Maxtor. I knew this was not
wise to run so long on a single disk but when the boss says to do it - I
thought well he knows what he's doing ...

And then Mr Murphy and his famous law kicked in and the second drive
(the only drive) froze up on me last night!

Rebooting the machine, I was able to fsck my way through the /boot and /
partition errors but the boot process gets stuck when trying to read the
journal for /data/repositories. The /data/backup and /data/isos volumes
mounted ok after they recovered their respective journals, but I don't
ever get past the 'recovering journal' message for /data/repositories.

Needless to say - there is some important data on the /dev/md2 partition
inside of the logical volume group lv-repositories. I have tried several
live cds but none of them seem to understand that I have any LVM volumes
setup.

Does anyone have any ideas on how I might be able to restore/mount
the /dev/md2 partition - even if it is just enough to copy the data
elsewhere??

thanks

pantz

Re: [linux-lvm] waah waah - disaster recovery

[Robin Green]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

On Fri, Feb 11, 2005 at 02:05:23PM -0500, Paul Pianta wrote:
> Rebooting the machine, I was able to fsck my way through the /boot and /
> partition errors but the boot process gets stuck when trying to read the
> journal for /data/repositories. The /data/backup and /data/isos volumes
> mounted ok after they recovered their respective journals, but I don't
> ever get past the 'recovering journal' message for /data/repositories.

Yeah, the ext3 journal recovery code in the kernel seems to be buggy.
(I use reiserfs on lvm2 and I have never had any problems with that, by the
way.)

Try this:

1. Boot with a live CD and comment out the line in /etc/fstab on your root
partition that refers to /data/repositories, by putting a # at the start of
the line. Note down which device corresponds to /data/repositories.

2. Reboot from your hard drive

3. Run fsck /dev/[device noted in step 1]

That should sort it. (I'm assuming that one of your live CDs can at least
mount your / partition.)
-- 
Robin

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: [linux-lvm] waah waah - disaster recovery

[Rosenstrauch, David]

> Does anyone have any ideas on how I might be able to restore/mount
> the /dev/md2 partition - even if it is just enough to copy the data
> elsewhere??
> 
> thanks
> 
> pantz


I just last week had a HD die on me - although in a non-raid/non-lvm setup.
Still, some of these techniques might work for you.

First off, since you just had 2 drives die, I would think you now have carte
blanche to go buy 2 new disks, so do that.  How many drives can your machine
hold?

* If only 2 drives, then install one of the new disks, and the bad disk.
Create 2 partitions on the new disk: 1 for the OS, and 1 for the recovery data
(i.e., the recovery partition needs to be at least the size of the partition
you're trying to recover.)

* If it can hold more than 2 drives, then just install both new disks and the
bad disk.  Put the OS on one of the new disks and a partition for the recovery
data on the other new one.

Install the OS onto one of the new disks.  Now, since you have a bootable
system with an OS, you can install LVM onto it.  Plus you'll now have access
to dd, which you can use to do the recovery.  (Trying to recover from a dd'ed
copy of a bad disk can often give better results than trying to recover from
the actual disk.)

Basically just dd the data from the partition on the bad disk to the partition
on the new disk (e.g., "dd if=/dev/hda2 of=/dev/hdb3").  Theoretically, you've
now got a copy of the original partition, and theoretically LVM should be able
to use it just like the old one.  Do fsck's on all the relevant partitions
and/or LV's once you're done with this, as there could be glitches that
occurred.

Assuming all goes well, you then clean up:  take out the bad disk, set up the
new data disk properly, etc.

I can't guarantee this'll work of course.  The partition could be really hosed
and unable to be recovered by dd, it might not pass a fsck, partitions/lv's
might not be able to be mounted, etc.  But this did work for me last week.  If
there were glitches in any of the recovered data, I haven't found them yet,
and it passed the fsck without a problem.


One other option to consider:  perhaps you can use Knoppix as a rescue disk
here.  IIRC, the Knoppix Live CD only has LVM 1, but I think I heard about a
Knoppix DVD available somewhere that has LVM 2.  This might make the process
easier, as you wouldn't have to worry about getting the OS installed and
running on one of the HD's first.  You'd could just run the OS off of the
CD/DVD and do your recovery that way, and deal with installing the OS
afterwards.

HTH, and that my limited of knowledge on LVM and RAID hasn't led me to
misunderstand the situation or give you any bad advice.

Good luck!

DR

==============================================================================
This message is for the sole use of the intended recipient. If you received
this message in error please delete it and notify us. If this message was
misdirected, CSFB does not waive any confidentiality or privilege. CSFB
retains and monitors electronic communications sent through its network.
Instructions transmitted over this system are not binding on CSFB until they
are confirmed by us. Message transmission is not guaranteed to be secure.
==============================================================================

Re: [linux-lvm] waah waah - disaster recovery

[Rickard Olsson]

Rosenstrauch, David wrote:

> Basically just dd the data from the partition on the bad disk to the partition
> on the new disk (e.g., "dd if=/dev/hda2 of=/dev/hdb3").

dd_rescue.

http://freshmeat.net/projects/ddrescue/
http://www.gnu.org/software/ddrescue/ddrescue.html

Nice feature: If the OP still had had the first disk, he could have run 
dd_rescue on *both* disks (since they were copies of each other) and in 
effect recovered all data that was still readable on *either* disk 
(unless disk #1 was totally dead, of course).



The Gentoo LiveCD has LVM2 support.

http://www.gentoo.org/doc/en/lvm2.xml

     / Rickard Olsson,IT-Konsult/
    / Telefon: +46 70 635 01 42/
   / http://www.webhackande.se/

Re: [linux-lvm] waah waah - disaster recovery

[David Brown]

On Fri, Feb 11, 2005 at 07:43:39PM -0000, Rosenstrauch, David wrote:

> One other option to consider:  perhaps you can use Knoppix as a rescue disk
> here.  IIRC, the Knoppix Live CD only has LVM 1, but I think I heard about a
> Knoppix DVD available somewhere that has LVM 2.  This might make the process
> easier, as you wouldn't have to worry about getting the OS installed and
> running on one of the HD's first.  You'd could just run the OS off of the
> CD/DVD and do your recovery that way, and deal with installing the OS
> afterwards.

The latest gentoo liveCD has LVM 2 on it, but in the great gentoo spirit,
it is probably out of date.

Dave

RE: [linux-lvm] waah waah - disaster recovery

[Paul Pianta]

On Fri, 2005-02-11 at 19:43 +0000, Rosenstrauch, David wrote:
> 
> > Does anyone have any ideas on how I might be able to restore/mount
> > the /dev/md2 partition - even if it is just enough to copy the data
> > elsewhere??
> > 
> > thanks
> > 
> > pantz
> 
> Basically just dd the data from the partition on the bad disk to the partition
> on the new disk (e.g., "dd if=/dev/hda2 of=/dev/hdb3").  Theoretically, you've
> now got a copy of the original partition, and theoretically LVM should be able
> to use it just like the old one.  Do fsck's on all the relevant partitions
> and/or LV's once you're done with this, as there could be glitches that
> occurred.

Wow - thanks for the quick responses guys! Wouldn't have anything to do
with mooching about on a Friday afternoon, catching up on emails coz the
work you have to do can wait until Monday ;)

I am attempting a bit of a combination of the two suggestions.

Editing /etc/fstab to not mount the dodgy partition got me to boot into
the system ok - but running fsck gives LOTS of bad looking errors and it
just loops on the same bad sector and I have to reboot the machine to
get out of it.

I am now dd'ing the dodgy partition to a new drive that I installed in
the machine.

My question now is a bit off topic but what would be the recommended way
of dd'ing my corrupt /dev/hdc5 partition to the new /dev/hda drive. 

- Should I have created a partition on /dev/hda exactly the same size
as /dev/hdc5 and then done a 'dd if=/dev/hdc5 of=/dev/hda1'?

- Will 'dd if=/dev/hdc5 of=/dev/hda' give me something mountable later?

- Should I have dd'ed the partition to a file in a partition /dev/hda1
on the new disk and loop mounted it after?

Sorry for being OT but I am just interested ...

thanks

pantz

RE: [linux-lvm] waah waah - disaster recovery

[Rosenstrauch, David]

> Rosenstrauch, David wrote:
> 
> > Basically just dd the data from the partition on the bad 
> disk to the partition
> > on the new disk (e.g., "dd if=/dev/hda2 of=/dev/hdb3").
> 
> dd_rescue.
> 
> http://freshmeat.net/projects/ddrescue/
> http://www.gnu.org/software/ddrescue/ddrescue.html


Wow - cool util!  Wish I had known about this a week ago!  :-)


> The Gentoo LiveCD has LVM2 support.
> 
> http://www.gentoo.org/doc/en/lvm2.xml

>      / Rickard Olsson,IT-Konsult/


Also very good to know.  Thanks for the info, Rickard!

DR

==============================================================================
This message is for the sole use of the intended recipient. If you received
this message in error please delete it and notify us. If this message was
misdirected, CSFB does not waive any confidentiality or privilege. CSFB
retains and monitors electronic communications sent through its network.
Instructions transmitted over this system are not binding on CSFB until they
are confirmed by us. Message transmission is not guaranteed to be secure.
==============================================================================

RE: [linux-lvm] waah waah - disaster recovery

[Paul Pianta]

On Fri, 2005-02-11 at 20:45 +0000, Rosenstrauch, David wrote:
> 
> > Rosenstrauch, David wrote:
> > 
> > > Basically just dd the data from the partition on the bad 
> > disk to the partition
> > > on the new disk (e.g., "dd if=/dev/hda2 of=/dev/hdb3").
> > 
> > dd_rescue.
> > 
> > http://freshmeat.net/projects/ddrescue/
> > http://www.gnu.org/software/ddrescue/ddrescue.html
> 
> 
> Wow - cool util!  Wish I had known about this a week ago!  :-)

Oooooooh yeah - cool util! It just saved my ass - I have managed to
recover everything that I needed by doing a
'dd_rescue /dev/hdc /dev/hda' while running the systemRescueCd live CD.

Thanks again all for your excellent tips :)

pantz