[linux-lvm] Re: How to handle Bad Block relocation with LVM?

[linux-lvm] Re: How to handle Bad Block relocation with LVM?

[Eric Hopper]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

I manually relocated a few bad blocks on a bad IBM drive I had when I
replaced the drive.  It took a lot of time and effort.  I had to run the
dd command many times very carefully to make it work.

One big problem for me was that read-ahead obscured which actual sectors
were in error.  I needed a 'raw' LVM device, but I don't think such a
thing exists for LVM1 on Linux 2.4.x.

What I did was used pvmove to move the PE containing the bad block to a
different spot on the hard drive, then allocated a new LV that was one
LE long, and forced it to allocate the PE containing the bad block. 
Then I used dd to carefully copy over the LE in sections, narrowing down
the location of the bad sectors until I had copied everything that could
possibly be read.

After that, I ran fsck on the filesystem that had originally contained
the bad block, and I was fine.  I checked carefully, and it didn't even
seem that I had lost any data.

Long, time consuming process though.

Actually, it may have been even ickier than I first thought.

It could be that pvmove wouldn't work, and I had to shorten the LV
containing the bad block (the BLV) to contain all PEs prior to the bad
one, allocate a new LV (the NLV) containing all the bad PE, lengthen the
BLV by 1 PE, using a brand new PE, then lengthen it to its original
length so it would contain all the PEs after that bad PE, the do the
procedure I outlined above.

Now that I think of it, I'm nearly positive that pvmove didn't work.  I
had dearly wished for some kind of option to pvmove that would force it
to try as hard as it could to get good reads of all the sectors in a PE,
then move the LE to a new PE, even if there were errors.

Have fun (if at all possible),
-- 
Eric Hopper <hopper@omnifarious.org>
Omnifarious Software

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] Re: How to handle Bad Block relocation with LVM?

[Joe Thornber]

Eric,

We would like to automate the process that you have described in LVM2
at some point.  So if you get an error on an LV and new PE will be
allocated, as much data as possible copied from the bad PE to the new
PE, and then remap the LV so that it's using the new PE (very much
like a small pvmove).

The EVMS team are writing a bad block relocator target for device
mapper, but I don't feel it's neccessary to add yet another device
layer to the LVs.  If I have a bad block I don't mind loosing a whole
PE (people may not agree with me on this ?)

- Joe

Re: [linux-lvm] Re: How to handle Bad Block relocation with LVM?

[Eric M. Hopper]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

On Mon, 2003-02-14 at 17:26, Joe Thornber <joe@fib011235813.fsnet.co.uk> wrote:
> Eric,
> 
> We would like to automate the process that you have described in LVM2
> at some point.  So if you get an error on an LV and new PE will be
> allocated, as much data as possible copied from the bad PE to the new
> PE, and then remap the LV so that it's using the new PE (very much
> like a small pvmove).
> 
> The EVMS team are writing a bad block relocator target for device
> mapper, but I don't feel it's neccessary to add yet another device
> layer to the LVs.  If I have a bad block I don't mind loosing a whole
> PE (people may not agree with me on this ?)

I actually quite agree.  Here are a few points:

o A PE is typically less than 0.01% of my total disk space.

o Corruption on a drive tends to spread, especially if sectors near the
corruption are accessed frequently, so it's best to avoid a whole
section of a drive around a bad spot.

o In general, I would want to throw away a drive that had so many bad
sectors that it's firmware relocation software could no longer handle
them.  So, my main purpose in relocating the PE is to recover as much
data from it as I can before junking the drive.

Have fun (if at all possible),
-- 
The best we can hope for concerning the people at large is that they
be properly armed.  -- Alexander Hamilton
-- Eric Hopper (hopper@omnifarious.org  http://www.omnifarious.org/~hopper) --

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 185 bytes --]

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Brian Murrell]

Joe Thornber <joe <at> fib011235813.fsnet.co.uk> writes:
> 
> Eric,
> 
> We would like to automate the process that you have described in LVM2
> at some point.  So if you get an error on an LV and new PE will be
> allocated, as much data as possible copied from the bad PE to the new
> PE, and then remap the LV so that it's using the new PE (very much
> like a small pvmove).
> 
> The EVMS team are writing a bad block relocator target for device
> mapper, but I don't feel it's neccessary to add yet another device
> layer to the LVs.  If I have a bad block I don't mind loosing a whole
> PE (people may not agree with me on this ?)

To resurrect a really, really, old thread, did anything ever get done in LVM2
to either automatically or manually map out PEs with bad blocks in them?

Does anyone have a recipe for doing this -- to save me the time of figuring it 
all out for myself?

Cheers,
b.

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Zdenek Kabelac]

Dne 28.11.2012 14:27, Brian Murrell napsal(a):
> Joe Thornber <joe <at> fib011235813.fsnet.co.uk> writes:
>>
>> Eric,
>>
>> We would like to automate the process that you have described in LVM2
>> at some point.  So if you get an error on an LV and new PE will be
>> allocated, as much data as possible copied from the bad PE to the new
>> PE, and then remap the LV so that it's using the new PE (very much
>> like a small pvmove).
>>
>> The EVMS team are writing a bad block relocator target for device
>> mapper, but I don't feel it's neccessary to add yet another device
>> layer to the LVs.  If I have a bad block I don't mind loosing a whole
>> PE (people may not agree with me on this ?)
>
> To resurrect a really, really, old thread, did anything ever get done in LVM2
> to either automatically or manually map out PEs with bad blocks in them?
>
> Does anyone have a recipe for doing this -- to save me the time of figuring it
> all out for myself?


Sorry, no automated tool.

You could possibly pvmove separated PEs manually with set of pvmove commands.
But I'd strongly recommend to get rid of such broken driver quickly then you 
loose any more data - IMHO it's the most efficient solution cost & time.

Zdenek

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

On 12-11-28 08:57 AM, Zdenek Kabelac wrote:
> 
> Sorry, no automated tool.

Pity,

> You could possibly pvmove separated PEs manually with set of pvmove
> commands.

So, is the basic premise to just find the PE that is sitting on a bad
block and just pvmove it into an LV created just for the purpose of
holding PEs that are on bad blocks?

So what happens when I pvmove a PE out of an LV?  I take it LVM moves
the data (or at least tries in this case) on the PE being pvmoved onto
another PE before moving it?

Oh, but wait.  pvmove (typically) moves PEs between physical volumes.
Can it be used to remap PEs like this?

> But I'd strongly recommend to get rid of such broken driver quickly then
> you loose any more data - IMHO it's the most efficient solution cost &
> time.

I'm not overly worried.  There is no critical data on that disk/machine
and it's fully backed up twice a day.  I do appreciate your concern and
warning though.

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Lars Ellenberg]

On Thu, Nov 29, 2012 at 07:26:24AM -0500, Brian J. Murrell wrote:
> On 12-11-28 08:57 AM, Zdenek Kabelac wrote:
> > 
> > Sorry, no automated tool.
> 
> Pity,
> 
> > You could possibly pvmove separated PEs manually with set of pvmove
> > commands.
> 
> So, is the basic premise to just find the PE that is sitting on a bad
> block and just pvmove it into an LV created just for the purpose of
> holding PEs that are on bad blocks?
> 
> So what happens when I pvmove a PE out of an LV?  I take it LVM moves
> the data (or at least tries in this case) on the PE being pvmoved onto
> another PE before moving it?
> 
> Oh, but wait.  pvmove (typically) moves PEs between physical volumes.
> Can it be used to remap PEs like this?

So what do you know?
You either know that pysical sector P on some physical disk is broken.
Or you know that logical sector L in some logical volume is broken.

If you do
pvs --unit s --segment -o vg_name,lv_name,seg_start,seg_size,seg_start_pe,pe_start,seg_pe_ranges

That should give you all you need to transform them into each other,
and to transform the sector number to PE number.

Having the PE number, you can easily do
pvmove /dev/broken:PE /dev/somewhere-else

Or with alloc anywhere even elsewhere on the same broken disk.
# If you don't have an other PV available,
# but there are free "healthy" extents on the same PV:
# pvmove --alloc anywhere /dev/broken:PE /dev/broken
Which would likely not be the smartest idea ;-)

You should then create one LV named e.g. "BAD_BLOCKS",
which you would create/extend to cover that bad PE,
so that won't be re-allocated again later:
lvextend VG/BAD_BLOCKS -l +1 /dev/broken:PE

Better yet, pvchange -an /dev/broken,
so it won't be used for new LVs anymore,
and pvmove /dev/broken completely to somewhere else.

So much for the theory, how I would try to do this.
In case I would do this at all.
Which I probably won't, if I had an other PV available.

	;-)

I'm unsure how pvmove will handle IO errors, though.

> > But I'd strongly recommend to get rid of such broken driver quickly then
> > you loose any more data - IMHO it's the most efficient solution cost &
> > time.

Right.

	Lars

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Stuart D Gathman]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

Long ago, Nostradamus foresaw that on 11/28/2012 08:57 AM, Zdenek
Kabelac would write:
>
>> To resurrect a really, really, old thread, did anything ever get done
>> in LVM2
>> to either automatically or manually map out PEs with bad blocks in them?
>>
>> Does anyone have a recipe for doing this -- to save me the time of
>> figuring it
>> all out for myself?
>
>
> Sorry, no automated tool.
>
> You could possibly pvmove separated PEs manually with set of pvmove
> commands.
> But I'd strongly recommend to get rid of such broken driver quickly
> then you loose any more data - IMHO it's the most efficient solution
> cost & time.
There are many situations where you are out with a laptop in a region
where there is no Fry's to purchase a new drive.  Often, such region
additionally impose stiff duties, many times the price, if you try to
order said drive online.  That's assuming internet is usable today in
said region.

So "replace the drive" might the best policy ideally, but is often
impossible for travellers in tech-primitive regions.

Having helped people in such situations (where internet at least was
working), I've used the attached script to help find affected LVs and files.

[-- Attachment #2: lbatofile.py --]
[-- Type: text/x-python, Size: 5652 bytes --]

#!/usr/bin/python
# Identify partition, LV, file containing a sector 

# Copyright (C) 2010,2012 Stuart D. Gathman
# Shared under GNU Public License v2 or later
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.

#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.

#   You should have received a copy of the GNU General Public License along
#   with this program; if not, write to the Free Software Foundation, Inc.,
#   51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

import sys
from subprocess import Popen,PIPE

ID_LVM = 0x8e
ID_LINUX = 0x83
ID_EXT = 0x05
ID_RAID = 0xfd

def idtoname(id):
  if id == ID_LVM: return "Linux LVM"
  if id == ID_LINUX: return "Linux Filesystem"
  if id == ID_EXT: return "Extended Partition"
  if id == ID_RAID: return "Software RAID"
  return hex(id)

class Segment(object):
  __slots__ = ('pe1st','pelst','lvpath','le1st','lelst')
  def __init__(self,pe1st,pelst):
    self.pe1st = pe1st;
    self.pelst = pelst;
  def __str__(self):
    return "Seg:%d-%d:%s:%d-%d" % (
      self.pe1st,self.pelst,self.lvpath,self.le1st,self.lelst)

def cmdoutput(cmd):
  p = Popen(cmd, shell=True, stdout=PIPE)
  try:
    for ln in p.stdout:
      yield ln
  finally:
    p.stdout.close()
    p.wait()

def icheck(fs,blk):
  "Return inum from block number, or 0 if free space."
  for ln in cmdoutput("debugfs -R 'icheck %d' '%s' 2>/dev/null"%(blk,fs)):
    b,i = ln.strip().split(None,1)
    if not b[0].isdigit(): continue
    if int(b) == blk:
      if i.startswith('<'):
	return 0
      return int(i)
  raise ValueError('%s: invalid block: %d'%(fs,blk))

def ncheck(fs,inum):
  "Return filename from inode number, or None if not linked."
  for ln in cmdoutput("debugfs -R 'ncheck %d' '%s' 2>/dev/null"%(inum,fs)):
    i,n = ln.strip().split(None,1)
    if not i[0].isdigit(): continue
    if int(i) == inum:
      return n
  return None

def blkid(fs):
  "Return dictionary of block device attributes"
  d = {}
  for ln in cmdoutput("blkid -o export '%s'"%fs):
    k,v = ln.strip().split('=',1)
    d[k] = v
  return d

def getpvmap(pv):
  pe_start = 192 * 2
  pe_size = None
  seg = None
  segs = []
  for ln in cmdoutput("pvdisplay --units k -m %s"%pv):
    a = ln.strip().split()
    if not a: continue
    if a[0] == 'Physical' and a[4].endswith(':'):
      pe1st = int(a[2])
      pelst = int(a[4][:-1])
      seg = Segment(pe1st,pelst)
    elif seg and a[0] == 'Logical':
      if a[1] == 'volume':
	seg.lvpath = a[2]
      elif a[1] == 'extents':
	seg.le1st = int(a[2])
	seg.lelst = int(a[4])
	segs.append(seg)
    elif a[0] == 'PE' and a[1] == 'Size':
      if a[2] == "(KByte)":
	pe_size = int(a[3]) * 2
      elif a[3] == 'KiB':
	pe_size = int(float(a[2])) * 2
  if segs:
    for ln in cmdoutput("pvs --units k -o+pe_start %s"%pv):
      a = ln.split()
      if a[0] == pv:
        lst = a[-1]
	if lst.lower().endswith('k'):
	  pe_start = int(float(lst[:-1]))*2
	  return pe_start,pe_size,segs
  return None

def findlv(pv,sect):
  res = getpvmap(pv)
  if not res: return None
  pe_start,pe_size,m = res
  if sect < pe_start:
    raise Exception("Bad sector in PV metadata area")
  pe = int((sect - pe_start)/pe_size)
  pebeg = pe * pe_size + pe_start
  peoff = sect - pebeg
  for s in m:
    if s.pe1st <= pe <= s.pelst:
      le = s.le1st + pe - s.pe1st
      return s.lvpath,le * pe_size + peoff

def getmdmap():
  with open('/proc/mdstat','rt') as fp:
    m = []
    for ln in fp:
      if ln.startswith('md'):
	a = ln.split(':')
	raid = a[0].strip()
	devs = []
	a = a[1].split()
	for d in a[2:]:
	  devs.append(d.split('[')[0])
	m.append((raid,devs))
    return m

def parse_sfdisk(s):
  for ln in s:
    try:
      part,desc = ln.split(':')
      if part.startswith('/dev/'):
        d = {}
        for p in desc.split(','):
	  name,val = p.split('=')
	  name = name.strip()
	  if name.lower() == 'id':
	    d[name] = int(val,16)
	  else:
	    d[name] = int(val)
	yield part.strip(),d
    except ValueError:
      continue

def findpart(wd,lba):
  s = cmdoutput("sfdisk -d %s"%wd)
  parts = [ (part,d['start'],d['size'],d['Id']) for part,d in parse_sfdisk(s) ]
  for part,start,sz,Id in parts:
    if Id == ID_EXT: continue
    if start <= lba < start + sz:
      return part,lba - start,Id
  return None

if __name__ == '__main__':
  wd = sys.argv[1]
  lba = int(sys.argv[2])
  print wd,lba,"Whole Disk"
  res = findpart(wd,lba)
  if not res:
    print "LBA is outside any partition"
    sys.exit(1)
  part,sect,Id = res
  print part,sect,idtoname(Id)
  if Id == ID_LVM:
    bd,sect = findlv(part,sect)
    # FIXME: problems if LV is snapshot
  elif Id == ID_LINUX:
    bd = part
  else:
    if Id == ID_RAID:
      for md,devs in getmdmap():
	for dev in devs:
	  if part == "/dev/"+dev:
	    part = "/dev/"+md
	    break
        else: continue
	break
    res = findlv(part,sect)
    if res:
      print "PV =",part
      bd,sect = res
    else:
      bd = part
  blksiz = 4096
  blk = int(sect * 512 / blksiz)
  p = blkid(bd)
  try:
    t = p['TYPE']
  except:
    print bd,p
    raise
  print "fs=%s block=%d %s"%(bd,blk,t)
  if t.startswith('ext'):
    inum = icheck(bd,blk)
    if inum:
      fn = ncheck(bd,inum)
      print "file=%s inum=%d"%(fn,inum)
    else:
      print "<free space>"

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

On 12-11-29 09:04 AM, Lars Ellenberg wrote:
> 
> If you do
> pvs --unit s --segment -o vg_name,lv_name,seg_start,seg_size,seg_start_pe,pe_start,seg_pe_ranges

Right.  Let's assume I can find the PE.

> Having the PE number, you can easily do
> pvmove /dev/broken:PE /dev/somewhere-else

Right but that...


> Or with alloc anywhere even elsewhere on the same broken disk.

As well as that just puts a new PE in where the one with the damaged
block is but returns the PE with the damaged block back to the free list
to be allocated again at some point in the future, yes?

> # If you don't have an other PV available,
> # but there are free "healthy" extents on the same PV:
> # pvmove --alloc anywhere /dev/broken:PE /dev/broken
> Which would likely not be the smartest idea ;-)

Right because of the above, yes?  Or is there something else nasty about it?

> You should then create one LV named e.g. "BAD_BLOCKS",
> which you would create/extend to cover that bad PE,
> so that won't be re-allocated again later:
> lvextend VG/BAD_BLOCKS -l +1 /dev/broken:PE

Ahhh.  But in this case I want lvcreate -l 1 /dev/broken:PE since I
don't yet have my "badblocks" LV.  I would of course use lvextend next
time.  :-)

> Better yet, pvchange -an /dev/broken,
> so it won't be used for new LVs anymore,
> and pvmove /dev/broken completely to somewhere else.

Yeah, of course ideally.  But as I mentioned, I'm not terribly worried
about loss in this case.

> I'm unsure how pvmove will handle IO errors, though.

I thought I read somewhere about pvmove being persistent through IO
errors but I can't seem to find it any more.  I guess we'll see.  :-)

It seems the pvmove just powered through.  Sweet.

I confirmed, using Stuart Gathman's (very nifty!) lbatofile.py program
that the file that was producing a read error from before the pvmove
read with no error after it and now I have my bad PE in a "badblocks" LV.

Super sweet!

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [linux-lvm] How to handle Bad Block relocation with LVM?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 250 bytes --]

On 12-11-29 10:55 AM, Stuart D Gathman wrote:
> 
> Having helped people in such situations (where internet at least was
> working), I've used the attached script to help find affected LVs and files.

Awesome script!

Much thanks!

b.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]