[linux-lvm] removing a bad HD

[linux-lvm] removing a bad HD

[Trond Michelsen]

Hi.

Recently a harddrive in my LVM started giving me read-errors, so
naturally I wanted to replace the drive with a new one. Now I've got the
replacement disk, and I've moved nearly all the PE's with pvmove. 

But, there is one PE that it can't move

--8<--
# pvmove -v /dev/hdk:646
[...]
pvmove -- /dev/hdk [PE 646 [lv0 [LE 15369]] -> /dev/hdg [PE 17557] [1/1]
/dev/vg0/group::/dev/vg0/lv0: 3900 10600832, 2200 287670784
pvmove -- ERROR "Input/output error" copying extent from "/dev/hdk"

pvmove -- ERROR "Input/output error" moving physical extents
--8<--

I've also tried with -i, but it had noe effect.
During the move-attempt, the syslog gets a few of these messages:

--8<--
Oct  4 21:13:03 2h0qoj kernel: hdk: read_intr: status=0x59 { DriveReady SeekComplete DataRequest Error }
Oct  4 21:13:03 2h0qoj kernel: hdk: read_intr: error=0x40 { UncorrectableError }, LBAsect=10609463, sector=10609456
Oct  4 21:13:03 2h0qoj kernel: end_request: I/O error, dev 39:00 (hdk), sector 10609456
--8<--


Anyway, because of this, vgreduce won't let me remove the drive from the
LVM

--8<--
# vgreduce -v vg0 /dev/hdk
vgreduce -- locking logical volume manager
vgreduce -- checking volume group name "vg0"
vgreduce -- checking volume group "vg0" existence
vgreduce -- checking volume group "vg0" activity
vgreduce -- reading data of volume group "vg0" from disk(s)
vgreduce -- reducing VGDA structures of volume group "vg0"
vgreduce -- ERROR: can't reduce volume group "vg0" by used physical volume "/dev/hdk"
--8<--

So, uhm. Now what? What will happen if I just turn off the coputer and
remove the drive? Will I be able to use the rest of the LVM without
losing anything other than the 8MB in the still active PE from the
faulty drive? 

Oh, and how do I determine which files will be affected? Is there any
way to find out which inodes that reside on the faulty LE?


I suspect that this is sort of a FAQ, and I apologise if I missed an
obvious entry in the HOWTO or the FAQ, but my searches have been
unsuccessful. The closest I've come is instructions on how to use the
device-mapper, which I understand is a LVM2 feature. I'm running LVM
1.0.3

more info on the drive and the volume group:

--8<--
# pvdisplay /dev/hdk
--- Physical volume ---
PV Name               /dev/hdk
VG Name               vg0
PV Size               74.53 GB [156301488 secs] / NOT usable 8.19 MB
[LVM: 165 KB]
PV#                   2
PV Status             available
Allocatable           yes
Cur LV                1
PE Size (KByte)       8192
Total PE              9538
Free PE               9537
Allocated PE          1
PV UUID               T3DIq4-sVkG-mM9j-gxJp-pjsa-U00w-X12d3w


# vgdisplay vg0
--- Volume group ---
VG Name               vg0
VG Access             read/write
VG Status             available/resizable
VG #                  0
MAX LV                256
Cur LV                2
Open LV               0
MAX LV Size           511.98 GB
Max PV                256
Cur PV                5
Act PV                5
VG Size               568.46 GB
PE Size               8 MB
Total PE              72763
Alloc PE / Size       61243 / 478.46 GB
Free  PE / Size       11520 / 90 GB
VG UUID               VIwKSd-wcyb-huBo-fUPR-bmp0-XRR5-DH2Y7q
--8<--


-- 
Trond Michelsen

Re: [linux-lvm] removing a bad HD

[Rickard Olsson]

Trond Michelsen wrote:

> Recently a harddrive in my LVM started giving me read-errors, so
> naturally I wanted to replace the drive with a new one.

I recently went through this (with the added complication of some bad 
metadata) and you have a few options.

The best one may be to use dd (with NOERROR) to raw copy the contents 
from the old disk to a new one of the same size and then use the new one 
when doing the pvmove.

Another path is to upgrade to LVM2 (you may have to do this anyway, I 
needed some functionality in the LVM2 pvcreate to pull my stunt off) and 
use the "partial" feature. I haven't tried this myself.

There are more details of my tribulations just a few weeks back in the 
list archives, but feel free to ask if there's something missing (I had 
some private conversations with Heinz on the topic as well).

    / Rickard Olsson,IT-Konsult/
   / Telefon: +46 70 635 01 42/
  / http://www.webhackande.se/

Re: [linux-lvm] removing a bad HD

[Trond Michelsen]

On Mon, Oct 06, 2003 at 06:04:05AM +0000, Rickard Olsson wrote:
> Trond Michelsen wrote:
> 
> > Recently a harddrive in my LVM started giving me read-errors, so
> > naturally I wanted to replace the drive with a new one.
> 
> I recently went through this (with the added complication of some bad 
> metadata) and you have a few options.
> 
> The best one may be to use dd (with NOERROR) to raw copy the contents 
> from the old disk to a new one of the same size and then use the new one 
> when doing the pvmove.
> 
> Another path is to upgrade to LVM2 (you may have to do this anyway, I 
> needed some functionality in the LVM2 pvcreate to pull my stunt off) and 
> use the "partial" feature. I haven't tried this myself.
> 
> There are more details of my tribulations just a few weeks back in the 
> list archives, but feel free to ask if there's something missing (I had 
> some private conversations with Heinz on the topic as well).

Thanks, but is it really necessary to copy the PE at all? It's broken,
and I don't care about that 8MB of data. Isn't it possible to just
reassign the LE that's mapped to the broken PE to a completely empty PE
at some other disk?

At the moment I'm a bit tempted to just set up a new VG, and migrate
data from the old VG file by file, disk by disk.

-- 
  // Trond Michelsen
\X/  mike@crusaders.no

Re: [linux-lvm] removing a bad HD

[Rickard Olsson]

Trond Michelsen wrote:

> Thanks, but is it really necessary to copy the PE at all? It's broken,
> and I don't care about that 8MB of data. Isn't it possible to just
> reassign the LE that's mapped to the broken PE to a completely empty PE
> at some other disk?

It's possible that Heinz or one of the other gurus know of a way, but I 
don't. If pvmove -i can't get past the read errors... Theoretically, you 
could be able to hexedit the LVM metadata but I really don't want to 
read about that in the news afterwards, if you know what I mean.

> At the moment I'm a bit tempted to just set up a new VG, and migrate
> data from the old VG file by file, disk by disk.

If you have a spare disk to start the new VG with, that's probably a 
good idea for several reasons (fresh filesystem, fresh metadata, the 
ability to check disks offline before adding them to the VG and so on).

I did this too since my old extents were rapidly becoming too small. New 
LVs default to a max size of 2TB, when I originally created mine I 
believe they were 250GB unless specified otherwise.

    / Rickard Olsson,IT-Konsult/
   / Telefon: +46 70 635 01 42/
  / http://www.webhackande.se/

Re: [linux-lvm] removing a bad HD

[Trond Michelsen]

On Wed, Oct 08, 2003 at 09:22:26AM +0200, Rickard Olsson wrote:
>> At the moment I'm a bit tempted to just set up a new VG, and migrate
>> data from the old VG file by file, disk by disk.
> If you have a spare disk to start the new VG with, that's probably a 
> good idea for several reasons (fresh filesystem, fresh metadata, the 
> ability to check disks offline before adding them to the VG and so on).

> I did this too since my old extents were rapidly becoming too small. New 
> LVs default to a max size of 2TB, when I originally created mine I 
> believe they were 250GB unless specified otherwise.

Yeah, it occured to me that this was probably the best opportunity I had
to increase the PE-size. So I set it to 32MB, giving me a maximum
LV-size of 2TB. 

BTW: What are the effects of choosing a large PE-size? Apart from
getting larger LV's, of course? Does it effect performance or memory
usage at all?

-- 
Trond Michelsen

Re: [linux-lvm] removing a bad HD

[Rickard Olsson]

Trond Michelsen wrote:

> Yeah, it occured to me that this was probably the best opportunity I had
> to increase the PE-size. So I set it to 32MB, giving me a maximum
> LV-size of 2TB. 

"That should be enough for anyone." ;-)

> BTW: What are the effects of choosing a large PE-size? Apart from
> getting larger LV's, of course? Does it effect performance or memory
> usage at all?

None that I have noticed. Then again, my main fileserver gets something 
like 1% load average and 10% mem usage (unless I'm shuffling large files 
locally, the gigabit adapter and Samba's buffers eat both up pretty 
quickly). Pulling notions out of my ass, I'd guess that larger PE-sizes 
are better now than they were before since most disks have larger 
buffers, our boxes have more RAM and Moore's law is seldom broken.

Heinz, is this the reasoning behind upping the default PE size or were 
there other considerations?

     / Rickard Olsson,IT-Konsult/
    / Telefon: +46 70 635 01 42/
   / http://www.webhackande.se/