Dying disk and filesystem choice.

Dying disk and filesystem choice.

[monkeyiq]

Hi,
  Could I please be CC'd replies.

  To keep it short and sweet, I have a 45Gb IBM drive that
is slowly dying by getting more bad sectors. I have already
returned my first one to get the current disk, so would like
to use the current one for a while before returning it for 
another disk that will prolly just start dying again.

I am using reiserfs at the moment, which doesn't really like 
to work on a dying drive. for example, doing a make fails to
work even though it is *creating* files on the disk, it fails
to do so because it hits new bad sectors and doesn't seem to
remap them. 

I am wondering what advise on filesystem choice the list as
and any other options I can use to get the kernel to remap
bad blocks.

Thanks.

-- 
---------------------------------------------------
It's the question, http://witme.sourceforge.net
If you think education is expensive, try ignorance.
		-- Derek Bok, president of Harvard

Re: Dying disk and filesystem choice.

[Hans Reiser]

monkeyiq wrote:
> 
> Hi,
>   Could I please be CC'd replies.
> 
>   To keep it short and sweet, I have a 45Gb IBM drive that
> is slowly dying by getting more bad sectors. I have already
> returned my first one to get the current disk, so would like
> to use the current one for a while before returning it for
> another disk that will prolly just start dying again.
> 
> I am using reiserfs at the moment, which doesn't really like
> to work on a dying drive. for example, doing a make fails to
> work even though it is *creating* files on the disk, it fails
> to do so because it hits new bad sectors and doesn't seem to
> remap them.
> 
> I am wondering what advise on filesystem choice the list as
> and any other options I can use to get the kernel to remap
> bad blocks.
> 
> Thanks.
> 
> --
> ---------------------------------------------------
> It's the question, http://witme.sourceforge.net
> If you think education is expensive, try ignorance.
>                 -- Derek Bok, president of Harvard
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

You can get the badblocks patch from Nikita and continue using reiserfs if you
want.  ext2 will also work.

We haven't sent the badblocks patch to Linus solely because it is a feature not
a bugfix, and code-freeze is on.

Hans

Re: Dying disk and filesystem choice.

[Andreas Dilger]

It was written:
>   To keep it short and sweet, I have a 45Gb IBM drive that
> is slowly dying by getting more bad sectors. I have already
> returned my first one to get the current disk, so would like
> to use the current one for a while before returning it for 
> another disk that will prolly just start dying again.
> 
> I am using reiserfs at the moment, which doesn't really like 
> to work on a dying drive. for example, doing a make fails to
> work even though it is *creating* files on the disk, it fails
> to do so because it hits new bad sectors and doesn't seem to
> remap them. 
> 
> I am wondering what advise on filesystem choice the list as
> and any other options I can use to get the kernel to remap
> bad blocks.

Well reiserfs is probably a very bad choice at this point.  It
does not have any bad blocks support (yet), so as soon as you have
a bad block you are stuck.

You should probably use ext2, along with the badblocks program to
check which blocks in the disk are bad.  It will do a read-write
test, and either your disk will remap the bad blocks, or they will
be detected.

Even so, keep your backups current...

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: Dying disk and filesystem choice.

[Andi Kleen]

On Thu, May 24, 2001 at 12:58:14AM -0600, Andreas Dilger wrote:
> Well reiserfs is probably a very bad choice at this point.  It
> does not have any bad blocks support (yet), so as soon as you have
> a bad block you are stuck.

reiserfs doesn't, but the HD usually has transparently in its firmware.
So it hits a bad block; you see an IO error and the next time you hit
the block the firmware has mapped in a fresh one from its internal
reserves.

-Andi

Re: Dying disk and filesystem choice.

[Jens Axboe]

On Thu, May 24 2001, Andi Kleen wrote:
> On Thu, May 24, 2001 at 12:58:14AM -0600, Andreas Dilger wrote:
> > Well reiserfs is probably a very bad choice at this point.  It
> > does not have any bad blocks support (yet), so as soon as you have
> > a bad block you are stuck.
> 
> reiserfs doesn't, but the HD usually has transparently in its firmware.
> So it hits a bad block; you see an IO error and the next time you hit
> the block the firmware has mapped in a fresh one from its internal
> reserves.

In fact you will typically only see an I/O error if the drive _can't_
remap the sector anymore, because it has run out. No point in reporting
a condition that was recovered.

I'd still say, that if you get bad block errors reported from your disk
it's long overdue for replacement.

-- 
Jens Axboe

Re: Dying disk and filesystem choice.

[monkeyiq]

Jens Axboe <axboe@suse.de> writes:

> On Thu, May 24 2001, Andi Kleen wrote:
> > On Thu, May 24, 2001 at 12:58:14AM -0600, Andreas Dilger wrote:
> > > Well reiserfs is probably a very bad choice at this point.  It
> > > does not have any bad blocks support (yet), so as soon as you have
> > > a bad block you are stuck.
> > 
> > reiserfs doesn't, but the HD usually has transparently in its firmware.
> > So it hits a bad block; you see an IO error and the next time you hit
> > the block the firmware has mapped in a fresh one from its internal
> > reserves.
> 
> In fact you will typically only see an I/O error if the drive _can't_
> remap the sector anymore, because it has run out. No point in reporting
> a condition that was recovered.
> 
> I'd still say, that if you get bad block errors reported from your disk
> it's long overdue for replacement.
> 
> -- 
> Jens Axboe

Well, I have been fighting this for a while now, its just that today it
finally hit /home in a big way :( basically its giving a DoS trying 
to compile stuff (which needs to write 25Mb+ library files)

The drive gives the patented IBM "replace me" audio and the kernel graces
me with these little chums:

May 24 14:50:22 kloof kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
May 24 14:50:22 kloof kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=78294495, sector=957496
May 24 14:50:22 kloof kernel: end_request: I/O error, dev 03:08 (hda), sector 957496
May 24 14:50:22 kloof kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
May 24 14:50:23 kloof kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=78294495, sector=957504
May 24 14:50:23 kloof kernel: end_request: I/O error, dev 03:08 (hda), sector 957504
May 24 14:50:23 kloof kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
May 24 14:50:23 kloof kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=78294495, sector=957512

I already knew that the drive was rooted by using the IBM DFT tool.
I am backing up as I type. Then, roll the dice with yet another 
IBM drive :-/
 

-- 
---------------------------------------------------
It's the question, http://witme.sourceforge.net
If you think education is expensive, try ignorance.
		-- Derek Bok, president of Harvard

Re: Dying disk and filesystem choice.

[Jonathan Lundell]

At 12:19 PM +0200 2001-05-24, Jens Axboe wrote:
>In fact you will typically only see an I/O error if the drive _can't_
>remap the sector anymore, because it has run out. No point in reporting
>a condition that was recovered.
>
>I'd still say, that if you get bad block errors reported from your disk
>it's long overdue for replacement.

This can't be right. It implies that the drive is returning bogus 
data with no error indication. Remapping a bad sector is not the same 
as recovering it.
-- 
/Jonathan Lundell.

Re: Dying disk and filesystem choice.

[Jonathan Lundell]

At 10:31 AM +0200 2001-05-24, Andi Kleen wrote:
>reiserfs doesn't, but the HD usually has transparently in its firmware.
>So it hits a bad block; you see an IO error and the next time you hit
>the block the firmware has mapped in a fresh one from its internal
>reserves.

Drives have remapping capability, but it's the first I've heard of HD 
firmware doing it automatically. I'd be very interested in reading 
the relevant documentation, if you could provide a pointer. Seems to 
me if a drive *could* do this, you'd certainly want to turn it 
(automatic remapping) off. There's way too much chance that a system 
will read the remapped sector and assume that it contains the 
original data. That would be hopelessly corrupting.
-- 
/Jonathan Lundell.

Re: Dying disk and filesystem choice.

[Andi Kleen]

On Thu, May 24, 2001 at 08:50:04AM -0700, Jonathan Lundell wrote:
> At 10:31 AM +0200 2001-05-24, Andi Kleen wrote:
> >reiserfs doesn't, but the HD usually has transparently in its firmware.
> >So it hits a bad block; you see an IO error and the next time you hit
> >the block the firmware has mapped in a fresh one from its internal
> >reserves.
> 
> Drives have remapping capability, but it's the first I've heard of HD 
> firmware doing it automatically. I'd be very interested in reading 
> the relevant documentation, if you could provide a pointer. Seems to 
> me if a drive *could* do this, you'd certainly want to turn it 
> (automatic remapping) off. There's way too much chance that a system 
> will read the remapped sector and assume that it contains the 
> original data. That would be hopelessly corrupting.

There are two scenarios: read and write. For write doing remapping transparent
is all fine, as the data is destroyed anyways.
For read it returns an IO error once and the next time you read from that 
block it contains fresh (or partly recovered) data.

-Andi

Re: Dying disk and filesystem choice.

[Jonathan Lundell]

At 5:56 PM +0200 2001-05-24, Andi Kleen wrote:
>On Thu, May 24, 2001 at 08:50:04AM -0700, Jonathan Lundell wrote:
>  > At 10:31 AM +0200 2001-05-24, Andi Kleen wrote:
>>  >reiserfs doesn't, but the HD usually has transparently in its firmware.
>>  >So it hits a bad block; you see an IO error and the next time you hit
>>  >the block the firmware has mapped in a fresh one from its internal
>>  >reserves.
>>
>>  Drives have remapping capability, but it's the first I've heard of HD
>>  firmware doing it automatically. I'd be very interested in reading
>>  the relevant documentation, if you could provide a pointer. Seems to
>>  me if a drive *could* do this, you'd certainly want to turn it
>>  (automatic remapping) off. There's way too much chance that a system
>>  will read the remapped sector and assume that it contains the
>>  original data. That would be hopelessly corrupting.
>
>There are two scenarios: read and write. For write doing remapping transparent
>is all fine, as the data is destroyed anyways.
>For read it returns an IO error once and the next time you read from that
>block it contains fresh (or partly recovered) data.

What HDs are we talking about, specifically?

WRT writes, how does the drive detect the error?

WRT reads, there are too many filesystems that would accept the 
second (no-IO-error) read as being the original good data.

IBM's UltraStar drives have an option (a bit in a vendor-unique mode 
page) that enables automatic reassignment, but it's done safely. If 
an unrecoverable read error is reported, the block is entered in a 
list of reassignment candidates. If that block is subsequently 
written, it's written back to the original location, and then 
verified. If the verify fails, the block is reassigned and rewritten; 
if it succeeds, it's left in the original location, and the block is 
removed from the reassignment candidate list.

Notice that invalid data is never returned without an error 
indication. That's critical.
-- 
/Jonathan Lundell.

Re: Dying disk and filesystem choice.

[Hans Reiser]

Andi Kleen wrote:
> 
> On Thu, May 24, 2001 at 12:58:14AM -0600, Andreas Dilger wrote:
> > Well reiserfs is probably a very bad choice at this point.  It
> > does not have any bad blocks support (yet), so as soon as you have
> > a bad block you are stuck.
> 
> reiserfs doesn't, but the HD usually has transparently in its firmware.
> So it hits a bad block; you see an IO error and the next time you hit
> the block the firmware has mapped in a fresh one from its internal
> reserves.
> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


No, reiserfs does have badblock support!!!!

You just have to get it as a separate patch from us because it was written after
code freeze.

Hans

Re: Dying disk and filesystem choice.

[David Rees]

On Thu, May 24, 2001 at 09:53:45AM -0700, Hans Reiser wrote:
> 
> No, reiserfs does have badblock support!!!!
> 
> You just have to get it as a separate patch from us because it was written after
> code freeze.

Any chance that you'll be putting them on www.namesys.com for easy download?

-Dave

RE: Dying disk and filesystem choice.

[Cress, Andrew R]

>At 12:19 PM +0200 2001-05-24, Jens Axboe wrote:
>>In fact you will typically only see an I/O error if the drive _can't_
>>remap the sector anymore, because it has run out. No point in reporting
>>a condition that was recovered.
>>
>>I'd still say, that if you get bad block errors reported from your disk
>>it's long overdue for replacement.

On Thursday, May 24, 2001 11:46 AM, Jonathan Lundell wrote:
>This can't be right. It implies that the drive is returning bogus 
>data with no error indication. Remapping a bad sector is not the same 
>as recovering it.

The automatic reallocation of bad spots on a drive is safe unless paired
with the write cache enabled bit in the disk mode pages (configuration).
WCE, in some cases, can cause the write/read to both take place from cache
with good status, and if the bad spot is reallocated later, when the cache
is flushed, the reporting gets very confusing.

In general:
It is a common misconception that if a disk begins to show some bad spots,
that it should be replaced.  In fact, a disk can have a reliable, useful
life for years after bad spots begin to show up.  The number of bad spots in
the Grown Defect List over a specified time as a % of capacity can be an
predictor of impending failure, however.

Andy Cress

Re: Dying disk and filesystem choice.

[J Sloan]

Excellent!

Will this be in resierfs 4.0 then?

cu

jjs

Hans Reiser schrieb:

> No, reiserfs does have badblock support!!!!
>
> You just have to get it as a separate patch from us because it was written after
> code freeze.

Re: Dying disk and filesystem choice.

[Erik Mouw]

On Thu, May 24, 2001 at 09:53:45AM -0700, Hans Reiser wrote:
> No, reiserfs does have badblock support!!!!
> 
> You just have to get it as a separate patch from us because it was
> written after code freeze.

IMHO we are not that deep into code freeze anymore. Freevxfs got added
in linux-2.4.5-pre*, so I think that a patch that adds a useful feature
like badblock support would be OK.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Department
of Electrical Engineering, Faculty of Information Technology and Systems,
Delft University of Technology, PO BOX 5031,  2600 GA Delft, The Netherlands
Phone: +31-15-2783635  Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/

Re: Dying disk and filesystem choice.

[Hans Reiser]

J Sloan wrote:
> 
> Excellent!
> 
> Will this be in resierfs 4.0 then?
> 
> cu
> 
> jjs
> 
> Hans Reiser schrieb:
> 
> > No, reiserfs does have badblock support!!!!
> >
> > You just have to get it as a separate patch from us because it was written after
> > code freeze.


No, version 4 won't ship until september 2002, these features are all v3.7,
which will be sent to Linus as soon as 2.5.1 opens up.

Hans

Re: Dying disk and filesystem choice.

[Hans Reiser]

Erik Mouw wrote:
> 
> On Thu, May 24, 2001 at 09:53:45AM -0700, Hans Reiser wrote:
> > No, reiserfs does have badblock support!!!!
> >
> > You just have to get it as a separate patch from us because it was
> > written after code freeze.
> 
> IMHO we are not that deep into code freeze anymore. Freevxfs got added
> in linux-2.4.5-pre*, so I think that a patch that adds a useful feature
> like badblock support would be OK.
> 
> Erik
> 
> --
> J.A.K. (Erik) Mouw, Information and Communication Theory Group, Department
> of Electrical Engineering, Faculty of Information Technology and Systems,
> Delft University of Technology, PO BOX 5031,  2600 GA Delft, The Netherlands
> Phone: +31-15-2783635  Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
> WWW: http://www-ict.its.tudelft.nl/~erik/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

vxfs is probably a completely separate fs that won't destabilize other
filesystems, or at least I hope so.  ReiserFS is stable code now, we aren't
going to change that by adding features unless they go into the ac series for
six weeks or so first.  ReiserFS is the SuSE recommended FS, and we can't go
destabilizing it.  I am told by sistina.com (maintainers of LVM) that in their
surveys of the users of LVM, 90% use ReiserFS, and the users of LVM tend very
much to be persons with mission critical servers.  We sent Linus a patch to mark
us as stable not experimental.  When we say stable, it means something.  Right
now it means zero (yes, zero) new bug reports that are not user error or old
versions or fsck, since 2.4.4 came out.  

fsck is improving dramatically in stability every week.  I used it myself last
week, and got my data back minus the root directory after trashing the front of
my partition accidentally.  (Which gave me a chance to review its end user
usability, which is also improving.)  We aren't yet ready to pass the zero a
random block and see it recover always excepting what was zero'd test, but we
will be before long.  One of the things we realized recently is that if the user
knows what got trashed, and he can tell this to the FS, it can be very useful
for bitmap blocks.  Sorry, I wander here.

Hans

Re: Dying disk and filesystem choice.

[monkeyiq]

Hans Reiser <reiser@namesys.com> writes:

> 
> 
> No, reiserfs does have badblock support!!!!
> 
> You just have to get it as a separate patch from us because it was written after
> code freeze.
> 
> Hans
> 

It might be nice to have a link to that patch from the "download" page.
I didn't see that patch the first time from
ftp://ftp.namesys.com/pub/misc-patches/

because I assumed that the interesting patches were linked from the 
downloads page. Atleast a hint that there is a badblocks patch available
on the ftp site on the downloads page, so that the reader can go grab 
this patch. 

Hopefully this will extend my mileage until drive replacement :)
Though I think I'll back a few more things up before attempting 
a kernel compile.


-- 
---------------------------------------------------
It's the question, http://witme.sourceforge.net
If you think education is expensive, try ignorance.
		-- Derek Bok, president of Harvard

Re: Dying disk and filesystem choice.

[Alan Cox]

> IMHO we are not that deep into code freeze anymore. Freevxfs got added
> in linux-2.4.5-pre*, so I think that a patch that adds a useful feature
> like badblock support would be OK.

FreeVxFS changes precisely nothing in the behaviour of any other fs - its like
adding a new driver.

Updating Reiserfs requires a lot more care because it has the potential to
harm existing stable setups

Re: Dying disk and filesystem choice.

[Juan Quintela]

>>>>> "erik" == Erik Mouw <J.A.K.Mouw@ITS.TUDelft.NL> writes:

erik> On Thu, May 24, 2001 at 09:53:45AM -0700, Hans Reiser wrote:
>> No, reiserfs does have badblock support!!!!
>> 
>> You just have to get it as a separate patch from us because it was
>> written after code freeze.

erik> IMHO we are not that deep into code freeze anymore. Freevxfs got added
erik> in linux-2.4.5-pre*, so I think that a patch that adds a useful feature
erik> like badblock support would be OK.

A new filesystem don't touch anything if you don't use it.  Some
reasoning for inclusion of new drivers.  Big changes in a working
driver could make it not to work (very bad in the stable series).

Later, Juan.


-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

Re: Dying disk and filesystem choice.

[Chris Mason]

On Thursday, May 24, 2001 11:16:58 PM +0100 Alan Cox
<alan@lxorguk.ukuu.org.uk> wrote:

>> IMHO we are not that deep into code freeze anymore. Freevxfs got added
>> in linux-2.4.5-pre*, so I think that a patch that adds a useful feature
>> like badblock support would be OK.
> 
> FreeVxFS changes precisely nothing in the behaviour of any other fs - its
> like adding a new driver.
> 
> Updating Reiserfs requires a lot more care because it has the potential to
> harm existing stable setups

This has been mostly covered, but just in case.  There are two different
freezes, the kernel, and in reiserfs.  The reiserfs part isn't something
Alan or Linus have imposed on us, we just wanted to limit the reiserfs
changes as much as possible during the early kernel releases.

The end result is that some larger scale issues are unfixed (memory
pressure from VM, lost files after a crash), but we have been able to focus
on the critical hoses-my-files/crashes-my-box kinds of bugs.

-chris

Re: Dying disk and filesystem choice.

[Hans Reiser]

Chris Mason wrote:
> 
> On Thursday, May 24, 2001 11:16:58 PM +0100 Alan Cox
> <alan@lxorguk.ukuu.org.uk> wrote:
> 
> >> IMHO we are not that deep into code freeze anymore. Freevxfs got added
> >> in linux-2.4.5-pre*, so I think that a patch that adds a useful feature
> >> like badblock support would be OK.
> >
> > FreeVxFS changes precisely nothing in the behaviour of any other fs - its
> > like adding a new driver.
> >
> > Updating Reiserfs requires a lot more care because it has the potential to
> > harm existing stable setups
> 
> This has been mostly covered, but just in case.  There are two different
> freezes, the kernel, and in reiserfs.  The reiserfs part isn't something
> Alan or Linus have imposed on us, we just wanted to limit the reiserfs
> changes as much as possible during the early kernel releases.
> 
> The end result is that some larger scale issues are unfixed (memory
> pressure from VM, lost files after a crash), but we have been able to focus
> on the critical hoses-my-files/crashes-my-box kinds of bugs.
> 
> -chris
No, our policy is strictly in sync with and reflective of that of the rest of
the linux-kernel.  Since the ac series has a different policy, we can be
different in regards to the ac series.  

And I don't begin to comprehend your not sending in the lost disk space after
crash bug fix (I assume it is what you mean when you refer to lost files after a
crash, because I know of no lost files after a crash bug, please phrase things
more carefully), and it really annoys me and the users, frankly.  Why you
consider that a feature is beyond me.

monstr, could you fix it please and send the fix in?  We can't wait for Chris to
send it in any longer.

Thx,

Hans

Re: Dying disk and filesystem choice.

[Chris Mason]

On Friday, May 25, 2001 09:21:42 AM -0700 Hans Reiser <reiser@namesys.com>
wrote:
> No, our policy is strictly in sync with and reflective of that of the
> rest of the linux-kernel.  Since the ac series has a different policy, we
> can be different in regards to the ac series.  

Not really, our policy has been much more restrictive than the rest of the
kernel.  Look at the patches we didn't send in.

> 
> And I don't begin to comprehend your not sending in the lost disk space
> after crash bug fix (I assume it is what you mean when you refer to lost
> files after a crash, because I know of no lost files after a crash bug,
> please phrase things more carefully), and it really annoys me and the
> users, frankly.  Why you consider that a feature is beyond me.

The patch is a _huge_ change to the way files are deleted and truncated, to
what happens during mount, and to the way transactions work.  It is
effectively a format extension, and must be verified against both 2.2.x
kernels and 2.4.x kernels, in both disk formats.

Before I even consider introducing a change of this size, I want to be as
sure as I can the rest of the code is stable.  It is the only way we can
debug it and stay sane.  Even after I release the code, I won't want it in
an ac series for a while.  It does much more harm than good if it somehow
ruins compatibility with an older kernel, especially in 2.4.x.  

Yes, it is a bug fix.  But, it is a very different kind of bug fix than
something that corrupts files at random, or something that doesn't get
buffers to disk at the right time.  

I won't pretend the fix isn't important, but I won't allow larger changes
to ruin the progress we've made so far.

-chris

Re: Dying disk and filesystem choice.

[Hans Reiser]

monstr will code it, leave it to him.

Hans