badblocks, sector/bit copies and other musings

badblocks, sector/bit copies and other musings

[Noah White]

Greetings,

I had some general questions regarding how the kernel/fs handle bad blocks
and such and how that relates to various copying techniques such as Norton
Ghost, hardware disk duplicators (the old octopus's) and such.

Specifically, I'm curious as to how the following situation is handled. If I
have:

drive A which is say a 30 GB IDE drive which I've built in a standard
fashion with the 2.2.12-20 kernel and created two ext-2 partitions and one
swap partition

drive b which is also the same model 30 GB IDE drive which is empty and has
no file system on it OR has a windows file system (FAT or NTFS) 

Now in the case of copying drive A onto drive B using a sector or bit copy
technique and I:

1) Use Ghost with sector copy mode OR
2) Use a hardware harness which does a straight bit copy of the drive (and
is non-file system aware)

How will the bad blocks be mapped? From the file system perspective will
drive B think its bad blocks are the same as drive A thus setting drive B up
for possible errors because its bad block mappings are incorrect?

This leads to my ultimate question which is what is the safest and fastest
way to dup a linux/ext-2 drive? 

Thanks in advance,

-Noah

Re: badblocks, sector/bit copies and other musings

[Helge Hafting]

Noah White wrote:
> 
> Greetings,
> 
> I had some general questions regarding how the kernel/fs handle bad blocks
> and such and how that relates to various copying techniques such as Norton
> Ghost, hardware disk duplicators (the old octopus's) and such.
> 
> Specifically, I'm curious as to how the following situation is handled. If I
> have:
> 
> drive A which is say a 30 GB IDE drive which I've built in a standard
> fashion with the 2.2.12-20 kernel and created two ext-2 partitions and one
> swap partition
> 
> drive b which is also the same model 30 GB IDE drive which is empty and has
> no file system on it OR has a windows file system (FAT or NTFS)
> 
> Now in the case of copying drive A onto drive B using a sector or bit copy
> technique and I:
> 
> 1) Use Ghost with sector copy mode OR
> 2) Use a hardware harness which does a straight bit copy of the drive (and
> is non-file system aware)

3) dd if=/dev/hda of=/dev/hdb
 
> How will the bad blocks be mapped? From the file system perspective will
> drive B think its bad blocks are the same as drive A thus setting drive B up
> for possible errors because its bad block mappings are incorrect?

Disks handle bad blocks themselves - to some extent.  You never see
that,
the drive simply remaps a sector to some other location with spare
sectors.
You can use non-fs aware copy techniques as long as you haven't run
out of spare sectors.  You start getting io errors when you do run
out - and then the fs have to mark blocks as bad.  (You may even need
a fsck or badblocks run to get this right.)

You _need_ a fs-aware way of copying once this happens.  such as "cp -a"
or similiar.  A non-fs aware copy of a fs with bad blocks will give
you a fs on hdb that have some sectors marked bad who aren't bad -
because
you copied the bad block list from the other drive.  This is no good.
Even worse if hdb actually had bad blocks of its own - then you get
bad blocks marked as good.
 
If you really intent to copy stuff onto a blank drive that has bad
blocks,
do this:

1. Use mkfs to create an empty fs on the B fs.  Use the option that
checks
   for bad blocks.  Possibly also use badblocks for a more thorough
   test, i.e. a write test.  Now you have a fs with all the bad blocks
   mapped.
2. Use fsck -i or badblocks on the A fs as well. This so you'll
   avoid unexpected bad blocks during the copy.  That might stop
   the copy somewhere in the middle and is no fun.
3. copy from one fs to another using a fs-aware copy like "cp".  A nice 
   side effect is that the copy gets defragmented.
   This way do not mess up bad block information, as you're simply
   reading files from one fs and writing them on the other.  And
   the fs will not try to write to known bad blocks.

> This leads to my ultimate question which is what is the safest and fastest
> way to dup a linux/ext-2 drive?

The non-fs aware copy may be faster if the fs is almost full.  The
fs aware way is the safe one in precence of bad blocks, and may
be faster too if there's lots of free space.  The fs-aware ways
don't copy "free space", the non-fs aware ways do!

Helge Hafting

Re: badblocks, sector/bit copies and other musings

[Hans-Peter Jansen]

[sorry for jumpimg in this late]

On Tuesday, 9. April 2002 10:41, Helge Hafting wrote:
> Noah White wrote:
> > Greetings,
> >
> > I had some general questions regarding how the kernel/fs handle bad
> > blocks and such and how that relates to various copying techniques such
> > as Norton Ghost, hardware disk duplicators (the old octopus's) and such.

> 3) dd if=/dev/hda of=/dev/hdb
>
> > How will the bad blocks be mapped? From the file system perspective will
> > drive B think its bad blocks are the same as drive A thus setting drive B
> > up for possible errors because its bad block mappings are incorrect?
>
> Disks handle bad blocks themselves - to some extent.  You never see
> that,

hopefully

> the drive simply remaps a sector to some other location with spare
> sectors.

> You can use non-fs aware copy techniques as long as you haven't run
> out of spare sectors.  You start getting io errors when you do run
> out - and then the fs have to mark blocks as bad.  (You may even need
> a fsck or badblocks run to get this right.)

Not always true. Remember the ibm ide hd shame last autumn... They throw
hard sector errors, triggered by dirty spin down from power loss :-((( 
I had to suffer from that not too long ago...

> You _need_ a fs-aware way of copying once this happens.  such as "cp -a"
> or similiar.  A non-fs aware copy of a fs with bad blocks will give
> you a fs on hdb that have some sectors marked bad who aren't bad -
> because
> you copied the bad block list from the other drive.  This is no good.

Sure about this one? IMHO, a sector copy wouldn't transfer the bad sector 
mapping. It's a drive internal mapping.

> Even worse if hdb actually had bad blocks of its own - then you get
> bad blocks marked as good.
>
> If you really intent to copy stuff onto a blank drive that has bad
> blocks,
> do this:
>
> 1. Use mkfs to create an empty fs on the B fs.  Use the option that
> checks
>    for bad blocks.  Possibly also use badblocks for a more thorough
>    test, i.e. a write test.  Now you have a fs with all the bad blocks
>    mapped.
> 2. Use fsck -i or badblocks on the A fs as well. This so you'll
>    avoid unexpected bad blocks during the copy.  That might stop
>    the copy somewhere in the middle and is no fun.
> 3. copy from one fs to another using a fs-aware copy like "cp".  A nice
>    side effect is that the copy gets defragmented.
>    This way do not mess up bad block information, as you're simply
>    reading files from one fs and writing them on the other.  And
>    the fs will not try to write to known bad blocks.

The problem with this in the above mentioned scenario is, that cp will
fall over, when running into failures. At least it will truncate
the file, which is not always desired... I had most success with Kurt
Garloff's dd_rescue. In my case, I was able to rescue a reiserfs 
partition with 8 hard sector errors with minimum loss, although one
of them was in a meta data sector (with _current_ reiserfsck).
I wrote a little script to locate the zeroed sectors, which were mainly
in some ml mbox files. Rest was a matter of vi ;).

> > This leads to my ultimate question which is what is the safest and
> > fastest way to dup a linux/ext-2 drive?
>
> The non-fs aware copy may be faster if the fs is almost full.  The
> fs aware way is the safe one in precence of bad blocks, and may
> be faster too if there's lots of free space.  The fs-aware ways
> don't copy "free space", the non-fs aware ways do!

and sometimes, it's not that easy...

> Helge Hafting

Cheers,
  Hans-Peter