[linux-lvm] Drive failure

[linux-lvm] Drive failure

[B. J. Zolp]

One of the drives in my volume group is developing bad sectors on it.  When I 
mount the logical volume spanning that drive my entire machine locks up.  
Should I be able to dd (without reading the bad sectors) the failing drive 
(assuming it spins up and reads all the good sectors) to the replacement 
drive, then run fsck on the unmounted volume and then mount the volume with 
minimal data loss?

The other option I can think of is remove that drive from the volume and then 
remount it, but that would mean a loss of everything that is one the failing 
drive, plus I am not sure how reiserfs works with large chunks of its file 
system removed like that.


thanks

bjz

Re: [linux-lvm] Drive failure

[Herbert Poetzl]

On Wed, May 14, 2003 at 10:41:37AM -0500, B. J. Zolp wrote:
> One of the drives in my volume group is developing bad sectors on it.  When I 
> mount the logical volume spanning that drive my entire machine locks up.  
> Should I be able to dd (without reading the bad sectors) the failing drive 
> (assuming it spins up and reads all the good sectors) to the replacement 
> drive, then run fsck on the unmounted volume and then mount the volume with 
> minimal data loss?

I guess, this should work ... but make sure, that
you do the copy operation on another system, otherwise
the LVM stuff could/will be irritated by two diffent
disks with the same signature (after copying over)

> The other option I can think of is remove that drive from the volume and then 
> remount it, but that would mean a loss of everything that is one the failing 
> drive, plus I am not sure how reiserfs works with large chunks of its file 
> system removed like that.

I wouldn't do that, neither with reiserfs nor with 
any other unix filesystem ... only as last escape
if you disk is completely unreadable ...

anyway, if possible make a copy of ALL disks involved
before you do any fscking so you can go back and
try any other approach ...

best,
Herbert


> thanks
> 
> bjz
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Drive failure

[Ewen McNeill]

In message <20030515055917.GB19045@www.13thfloor.at>, Herbert Poetzl writes:
>On Wed, May 14, 2003 at 10:41:37AM -0500, B. J. Zolp wrote:
>> Should I be able to dd (without reading the bad sectors) the failing drive 
>> (assuming it spins up and reads all the good sectors) to the replacement 
>> drive, then run fsck on the unmounted volume and then mount the volume with 
>> minimal data loss?
>
>I guess, this should work ... but make sure, that
>you do the copy operation on another system, otherwise
>the LVM stuff could/will be irritated by two diffent
>disks with the same signature (after copying over)

Other things to consider:

- make sure you do the dd with an option to replace unreadable sectors
  with blank sectors (otherwise nothing will be written out for the
  unreadable sectors, causing everything to be "out of alignment" after
  the first bad block)

- do the copy in single user mode, preferable booted in such a way that
  the LVM isn't active (or on another system as suggested above)

- it'll take longer but consider copying in sector-by-sector chunks, as
  it'll reduce the amount skipped  (eg, bs=512)

- you may want to map which logical volumes, and which files on those
  volumes, are affected by the bad sectors before you start, so you know
  which files you'll lose  (you could use something like "badblocks" --
  in _read_only_ mode!! -- to identify the affected blocks)

But aside from that, given an identical sized replacement disk and/or
replacement partition, I'd guess it should, in theory, work.  I've seen
similar things done with other systems with logical volume managers
(eg, HP/UX 10.20) reasonably successfully (we ended up restoring much
of the data from backup anyway to get a consistent database snapshot,
but it did save a bunch of volume reconstruction time, etc).

Ewen

Re: [linux-lvm] Drive failure

[B. J. Zolp]

On Thursday 15 May 2003 00:59, Herbert Poetzl wrote:
> On Wed, May 14, 2003 at 10:41:37AM -0500, B. J. Zolp wrote:
> > One of the drives in my volume group is developing bad sectors on it. 
> > When I mount the logical volume spanning that drive my entire machine
> > locks up. Should I be able to dd (without reading the bad sectors) the
> > failing drive (assuming it spins up and reads all the good sectors) to
> > the replacement drive, then run fsck on the unmounted volume and then
> > mount the volume with minimal data loss?
>
> I guess, this should work ... but make sure, that
> you do the copy operation on another system, otherwise
> the LVM stuff could/will be irritated by two diffent
> disks with the same signature (after copying over)
>

What if the LVM is down?  Then when the copy is complete I replace the bad 
drive with the replacement and bring the LVM back up.  That seems like it 
should work.

> > The other option I can think of is remove that drive from the volume and
> > then remount it, but that would mean a loss of everything that is one the
> > failing drive, plus I am not sure how reiserfs works with large chunks of
> > its file system removed like that.
>
> I wouldn't do that, neither with reiserfs nor with
> any other unix filesystem ... only as last escape
> if you disk is completely unreadable ...
>
> anyway, if possible make a copy of ALL disks involved
> before you do any fscking so you can go back and
> try any other approach ...
>
> best,
> Herbert
>
> > thanks
> >
> > bjz
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Drive failure

[B. J. Zolp]

On Thursday 15 May 2003 03:14, Ewen McNeill wrote:
> In message <20030515055917.GB19045@www.13thfloor.at>, Herbert Poetzl writes:
> >On Wed, May 14, 2003 at 10:41:37AM -0500, B. J. Zolp wrote:
> >> Should I be able to dd (without reading the bad sectors) the failing
> >> drive (assuming it spins up and reads all the good sectors) to the
> >> replacement drive, then run fsck on the unmounted volume and then mount
> >> the volume with minimal data loss?
> >
> >I guess, this should work ... but make sure, that
> >you do the copy operation on another system, otherwise
> >the LVM stuff could/will be irritated by two diffent
> >disks with the same signature (after copying over)
>
> Other things to consider:
>
> - make sure you do the dd with an option to replace unreadable sectors
>   with blank sectors (otherwise nothing will be written out for the
>   unreadable sectors, causing everything to be "out of alignment" after
>   the first bad block)
>

I plan on using the noerror switch for dd, what else would I need to use to 
make sure it replaces with blank sectors.  I could not find anything in info 
dd or man dd.

> - do the copy in single user mode, preferable booted in such a way that
>   the LVM isn't active (or on another system as suggested above)
>
> - it'll take longer but consider copying in sector-by-sector chunks, as
>   it'll reduce the amount skipped  (eg, bs=512)
>

great idea

> - you may want to map which logical volumes, and which files on those
>   volumes, are affected by the bad sectors before you start, so you know
>   which files you'll lose  (you could use something like "badblocks" --
>   in _read_only_ mode!! -- to identify the affected blocks)
>
> But aside from that, given an identical sized replacement disk and/or
> replacement partition, I'd guess it should, in theory, work.  I've seen
> similar things done with other systems with logical volume managers
> (eg, HP/UX 10.20) reasonably successfully (we ended up restoring much
> of the data from backup anyway to get a consistent database snapshot,
> but it did save a bunch of volume reconstruction time, etc).
>
> Ewen
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Drive failure

[Ewen McNeill]

In message <200305150917.21504.bjzolp@wisc.edu>, "B. J. Zolp" writes:
>On Thursday 15 May 2003 03:14, Ewen McNeill wrote:
>> - make sure you do the dd with an option to replace unreadable sectors
>>   with blank sectors (otherwise nothing will be written out for the
>>   unreadable sectors, causing everything to be "out of alignment" after
>>   the first bad block)
>
>I plan on using the noerror switch for dd, what else would I need to use to 
>make sure it replaces with blank sectors.  I could not find anything in info 
>dd or man dd.

The option I was thinking of is conv=noerror,sync

The "noerror" is "continue reading after read errors", and the "sync"
is "pad each input block with zero bytes to block size".  Possibly GNU
dd is better behaved, but my recollection is that at least some "dd"s
will happily skip over blocks they can't read given "conv=noerror" (by
assuming they just read 0 bytes that time), but result in effectively
fewer blocks written out than read in, effectively shuffling everything
after the blocks with errors down a bit.  This is, suffice to say,
a bad thing to have happen to your file system.

You might want to practice a little with copying from the disk into some
temporary location (eg outputting to /dev/null will do for this
purpose), and watch the "input blocks" and "output blocks" counts that
are reported -- if they're not the same, you're going to have a problem.

Ewen

Re: [linux-lvm] Drive failure

[Herbert Poetzl]

On Thu, May 15, 2003 at 09:09:17AM -0500, B. J. Zolp wrote:
> On Thursday 15 May 2003 00:59, Herbert Poetzl wrote:
> > On Wed, May 14, 2003 at 10:41:37AM -0500, B. J. Zolp wrote:
> > > One of the drives in my volume group is developing bad sectors on it. 
> > > When I mount the logical volume spanning that drive my entire machine
> > > locks up. Should I be able to dd (without reading the bad sectors) the
> > > failing drive (assuming it spins up and reads all the good sectors) to
> > > the replacement drive, then run fsck on the unmounted volume and then
> > > mount the volume with minimal data loss?
> >
> > I guess, this should work ... but make sure, that
> > you do the copy operation on another system, otherwise
> > the LVM stuff could/will be irritated by two diffent
> > disks with the same signature (after copying over)
> >
> 
> What if the LVM is down?  Then when the copy is complete I replace the bad 
> drive with the replacement and bring the LVM back up.  That seems like it 
> should work.

just make sure that the kernel or any start script
scans for vgs or recreates lvmtab or makes any updates
and/or changes to the pvs ...

given that, it should work ...

> > > The other option I can think of is remove that drive from the volume and
> > > then remount it, but that would mean a loss of everything that is one the
> > > failing drive, plus I am not sure how reiserfs works with large chunks of
> > > its file system removed like that.
> >
> > I wouldn't do that, neither with reiserfs nor with
> > any other unix filesystem ... only as last escape
> > if you disk is completely unreadable ...
> >
> > anyway, if possible make a copy of ALL disks involved
> > before you do any fscking so you can go back and
> > try any other approach ...
> >
> > best,
> > Herbert
> >
> > > thanks
> > >
> > > bjz
> > >
> > > _______________________________________________
> > > linux-lvm mailing list
> > > linux-lvm@sistina.com
> > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Drive failure

[B. J. Zolp]

On Thursday 15 May 2003 15:39, Ewen McNeill wrote:
> In message <200305150917.21504.bjzolp@wisc.edu>, "B. J. Zolp" writes:
> >On Thursday 15 May 2003 03:14, Ewen McNeill wrote:
> >> - make sure you do the dd with an option to replace unreadable sectors
> >>   with blank sectors (otherwise nothing will be written out for the
> >>   unreadable sectors, causing everything to be "out of alignment" after
> >>   the first bad block)
> >
> >I plan on using the noerror switch for dd, what else would I need to use
> > to make sure it replaces with blank sectors.  I could not find anything
> > in info dd or man dd.
>
> The option I was thinking of is conv=noerror,sync
>
> The "noerror" is "continue reading after read errors", and the "sync"
> is "pad each input block with zero bytes to block size".  Possibly GNU
> dd is better behaved, but my recollection is that at least some "dd"s
> will happily skip over blocks they can't read given "conv=noerror" (by
> assuming they just read 0 bytes that time), but result in effectively
> fewer blocks written out than read in, effectively shuffling everything
> after the blocks with errors down a bit.  This is, suffice to say,
> a bad thing to have happen to your file system.
>
> You might want to practice a little with copying from the disk into some
> temporary location (eg outputting to /dev/null will do for this
> purpose), and watch the "input blocks" and "output blocks" counts that
> are reported -- if they're not the same, you're going to have a problem.

I'm doing a
 dd bs=512 conv=noerror,sync if=/dev/hdf of=/dev/null 
right now.  It has been going for a few hours, but does not seem to be taking 
up any cpu cycles (it looks hung up).  I will let it go over night and see 
how it is tomorrow.  What would you suggest I do if the I can only get the 
data on the HD up until the bad sector (or where ever it is hung up).  This 
is the last drive in the logical volume, so there was not too much data on 
it, but how would I go about added the new replacement drive to the logical 
volume without messing up my entire filesystem?

Thanks again.

bjz



>
> Ewen
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Drive failure

[Ewen McNeill]

In message <200305151834.10480.bjzolp@wisc.edu>, "B. J. Zolp" writes:
>I'm doing a
> dd bs=512 conv=noerror,sync if=/dev/hdf of=/dev/null 
>right now.  It has been going for a few hours, but does not seem to be taking 
>up any cpu cycles (it looks hung up).  

I suspect that it will "hang" several times during the process,
especially if you have IDE disks, each time it encounters a bad block
(IDE is somewhat more prone to this as the IDE code seems to retry the
read more times, and to not get quite the same degree of error reporting
as the SCSI code does -- eg, to distinguish how fatal the problem is).

Watching the kernel log output (eg, dmesg) will probably give you some
insight into what is going on.

Also with bs=512 it will be a very slow copy at the best of times.

>What would you suggest I do if the I can only get the data on the HD up 
>until the bad sector (or where ever it is hung up).  

With some care you can skip over the bad sectors another way; look at
the "skip=" argument.  I've recovered data from bad floppies in the past
by DDing each of the readable chunks out, making up padding blocks of
the right size, and joining it all together.  

Again you've got to be careful that the blocks you read off go back onto
the new disk in the same position as they came off the old disk.

>This is the last drive in the logical volume, so there was not too much 
>data on it, but how would I go about added the new replacement drive to 
>the logical volume without messing up my entire filesystem?

Pass.  This is almost certainly file system specific, and even then I
don't know how well any common Linux file system would cope with loosing
whole chunks of the file system.  (I also don't know the right LVM magic
to substitute in another -- blank or mostly blank -- disk in place of
the broken one.)

Ewen

Re: [linux-lvm] Drive failure

[Urs Thuermann]

"B. J. Zolp" <bjzolp@wisc.edu> writes:

> I'm doing a
>  dd bs=512 conv=noerror,sync if=/dev/hdf of=/dev/null 
> right now.  It has been going for a few hours, but does not seem to be taking 
> up any cpu cycles (it looks hung up).  I will let it go over night and see 
> how it is tomorrow.  What would you suggest I do if the I can only get the 
> data on the HD up until the bad sector (or where ever it is hung up).  This 
> is the last drive in the logical volume, so there was not too much data on 
> it, but how would I go about added the new replacement drive to the logical 
> volume without messing up my entire filesystem?

You could also try dd_rescue(1), which was written, AFAIR, by Kurt
Garloff.  You should find it using Google.


urs