FEATURE Req: integrate badblocks check into fsck.reiser*

FEATURE Req: integrate badblocks check into fsck.reiser*

[Peter]

Perhaps this has been mentioned before. If so, sorry. IMHO, it would be
useful to integrate a call to badblocks in the fsck/mkfs.reiser* programs
so that more thorough disk checking can be done at format time. Sort of
like the option e2fsck -c. If this is added, the output could be fed
immediately to the reiser format program and badblocks spared prior to
filesystem use.

JM$0.02

-- 
Peter
+++++
Do not reply to this email, it is a spam trap and not monitored.
I can be reached via this list, or via 
jabber: pete4abw at jabber.org
ICQ: 73676357

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Vladimir V. Saveliev]

Hello

On Friday 01 September 2006 22:23, Peter wrote:
> Perhaps this has been mentioned before. If so, sorry. IMHO, it would be
> useful to integrate a call to badblocks in the fsck/mkfs.reiser* programs
> so that more thorough disk checking can be done at format time. Sort of
> like the option e2fsck -c. If this is added, the output could be fed
> immediately to the reiser format program and badblocks spared prior to
> filesystem use.
>
> JM$0.02

both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of bad 
blocks. We thought that should be enough.

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Hans Reiser]

Vladimir V. Saveliev wrote:
> Hello
>
> On Friday 01 September 2006 22:23, Peter wrote:
>   
>> Perhaps this has been mentioned before. If so, sorry. IMHO, it would be
>> useful to integrate a call to badblocks in the fsck/mkfs.reiser* programs
>> so that more thorough disk checking can be done at format time. Sort of
>> like the option e2fsck -c. If this is added, the output could be fed
>> immediately to the reiser format program and badblocks spared prior to
>> filesystem use.
>>
>> JM$0.02
>>     
>
> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of bad 
> blocks. We thought that should be enough.
>
>
>   
I'll take a patch....;-)

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[David Masover]

Vladimir V. Saveliev wrote:
> Hello
> 
> On Friday 01 September 2006 22:23, Peter wrote:
>> Perhaps this has been mentioned before. If so, sorry. IMHO, it would be
>> useful to integrate a call to badblocks in the fsck/mkfs.reiser* programs
>> so that more thorough disk checking can be done at format time. Sort of
>> like the option e2fsck -c. If this is added, the output could be fed
>> immediately to the reiser format program and badblocks spared prior to
>> filesystem use.
>>
>> JM$0.02
> 
> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of bad 
> blocks. We thought that should be enough.

It really should.  Why bother with a patch?  Just write a wrapper script 
that runs badblocks and passes in the list to mkfs.

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Hans Reiser]

David Masover wrote:
> Vladimir V. Saveliev wrote:
>> Hello
>>
>> On Friday 01 September 2006 22:23, Peter wrote:
>>> Perhaps this has been mentioned before. If so, sorry. IMHO, it would be
>>> useful to integrate a call to badblocks in the fsck/mkfs.reiser*
>>> programs
>>> so that more thorough disk checking can be done at format time. Sort of
>>> like the option e2fsck -c. If this is added, the output could be fed
>>> immediately to the reiser format program and badblocks spared prior to
>>> filesystem use.
>>>
>>> JM$0.02
>>
>> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of
>> bad blocks. We thought that should be enough.
>
> It really should.  Why bother with a patch?  Just write a wrapper
> script that runs badblocks and passes in the list to mkfs.
>
>
For average sysadmins, it would be nice to just add a flag and it
happens.  I will take a patch, but I won't write it.;-)

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Peter]

On Fri, 01 Sep 2006 17:27:20 -0500, David Masover wrote:
snip...
>> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of
>> bad blocks. We thought that should be enough.
> 
> It really should.  Why bother with a patch?  Just write a wrapper script
> that runs badblocks and passes in the list to mkfs.

It was just a thought from userland. My perspective was that a user, not a
hard-boiled geek, might get lulled into a false sense of security but may
not have the wherewithal to write a wrapper. If nothing else, when the
final doc is written (did I say final?:)), it should include a notice
about not running badblocks.

-- 
Peter
+++++
Do not reply to this email, it is a spam trap and not monitored.
I can be reached via this list, or via 
jabber: pete4abw at jabber.org
ICQ: 73676357

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[David Masover]

Peter wrote:
> On Fri, 01 Sep 2006 17:27:20 -0500, David Masover wrote:
> snip...
>>> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of
>>> bad blocks. We thought that should be enough.
>> It really should.  Why bother with a patch?  Just write a wrapper script
>> that runs badblocks and passes in the list to mkfs.
> 
> It was just a thought from userland. My perspective was that a user, not a
> hard-boiled geek, might get lulled into a false sense of security but may
> not have the wherewithal to write a wrapper. If nothing else, when the
> final doc is written (did I say final?:)), it should include a notice
> about not running badblocks.

Well, let's see...  Most hard drives come more thoroughly tested at the 
factory than anything badblocks would do.  Also, it seems redundant to 
have every single mkfs have to implement a badblocks flag..

I'd suggest a universal wrapper, then, or a modification to the "mkfs" 
frontend, so that this works the same way across all filesystems. 
Something like "mkfs -B -t reiser4"

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Hans Reiser]

David Masover wrote:
> Peter wrote:
>> On Fri, 01 Sep 2006 17:27:20 -0500, David Masover wrote:
>> snip...
>>>> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of
>>>> bad blocks. We thought that should be enough.
>>> It really should.  Why bother with a patch?  Just write a wrapper
>>> script
>>> that runs badblocks and passes in the list to mkfs.
>>
>> It was just a thought from userland. My perspective was that a user,
>> not a
>> hard-boiled geek, might get lulled into a false sense of security but
>> may
>> not have the wherewithal to write a wrapper. If nothing else, when the
>> final doc is written (did I say final?:)), it should include a notice
>> about not running badblocks.
>
> Well, let's see...  Most hard drives come more thoroughly tested at
> the factory than anything badblocks would do.
They leave more tested, they arrive.... ;-)  and you assume it is a new
drive.....
> Also, it seems redundant to have every single mkfs have to implement a
> badblocks flag..
>
> I'd suggest a universal wrapper, then, or a modification to the "mkfs"
> frontend, so that this works the same way across all filesystems.
> Something like "mkfs -B -t reiser4"
>
>

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Ric Wheeler]

David Masover wrote:

> Peter wrote:
>
>> On Fri, 01 Sep 2006 17:27:20 -0500, David Masover wrote:
>> snip...
>>
>>>> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of
>>>> bad blocks. We thought that should be enough.
>>>
>>> It really should.  Why bother with a patch?  Just write a wrapper 
>>> script
>>> that runs badblocks and passes in the list to mkfs.
>>
>>
>> It was just a thought from userland. My perspective was that a user, 
>> not a
>> hard-boiled geek, might get lulled into a false sense of security but 
>> may
>> not have the wherewithal to write a wrapper. If nothing else, when the
>> final doc is written (did I say final?:)), it should include a notice
>> about not running badblocks.
>
>
> Well, let's see...  Most hard drives come more thoroughly tested at 
> the factory than anything badblocks would do.  Also, it seems 
> redundant to have every single mkfs have to implement a badblocks flag..
>
> I'd suggest a universal wrapper, then, or a modification to the "mkfs" 
> frontend, so that this works the same way across all filesystems. 
> Something like "mkfs -B -t reiser4"


I don't think that modern drives that fail writes are worth using for a 
brand new file system.

While failing reads is quite common and can be caused by temporal issues 
(dirt on the platter, a bad write, etc), failed writes are almost always 
a sign that you have a serious issue.  Almost all modern drives remap 
each failed write to a bad sector automatically.  This action only fails 
if you have exhausted this remapping area (or have some really nasty 
issue like a bad cable, bad write head, etc).

Having mkfs ignore bad writes would seem to encourage users to create a 
new file system on a disk that is known to be bad & most likely not 
going to function well.  If a user ever has a golden opportunity to toss 
a drive in the trash, it is when they notice mkfs fails ;-)  This option 
to mkfs sounds like an invitation to disaster.

The other tools (debugreiserfs, reiserfsck, etc) do need to be able to 
handle bad blocks as well as possible since they are often needed during 
a salvage operation. in order to recover data (which might need to be 
migrated to a new disk). It is not clear to me that passing a list of 
bad blocks helps them as much as a robust general purpose error recovery 
support.

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Hans Reiser]

Ric Wheeler wrote:
>
>
> David Masover wrote:
>
>> Peter wrote:
>>
>>> On Fri, 01 Sep 2006 17:27:20 -0500, David Masover wrote:
>>> snip...
>>>
>>>>> both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of
>>>>> bad blocks. We thought that should be enough.
>>>>
>>>> It really should.  Why bother with a patch?  Just write a wrapper
>>>> script
>>>> that runs badblocks and passes in the list to mkfs.
>>>
>>>
>>> It was just a thought from userland. My perspective was that a user,
>>> not a
>>> hard-boiled geek, might get lulled into a false sense of security
>>> but may
>>> not have the wherewithal to write a wrapper. If nothing else, when the
>>> final doc is written (did I say final?:)), it should include a notice
>>> about not running badblocks.
>>
>>
>> Well, let's see...  Most hard drives come more thoroughly tested at
>> the factory than anything badblocks would do.  Also, it seems
>> redundant to have every single mkfs have to implement a badblocks flag..
>>
>> I'd suggest a universal wrapper, then, or a modification to the
>> "mkfs" frontend, so that this works the same way across all
>> filesystems. Something like "mkfs -B -t reiser4"
>
>
> I don't think that modern drives that fail writes are worth using for
> a brand new file system.
>
> While failing reads is quite common and can be caused by temporal
> issues (dirt on the platter, a bad write, etc), failed writes are
> almost always a sign that you have a serious issue.  Almost all modern
> drives remap each failed write to a bad sector automatically.  This
> action only fails if you have exhausted this remapping area (or have
> some really nasty issue like a bad cable, bad write head, etc).
>
> Having mkfs ignore bad writes would seem to encourage users to create
> a new file system on a disk that is known to be bad & most likely not
> going to function well.  If a user ever has a golden opportunity to
> toss a drive in the trash, it is when they notice mkfs fails ;-)  This
> option to mkfs sounds like an invitation to disaster.
Yes, you are right, the option should be to run badblocks and then fail
if it finds any.
>
> The other tools (debugreiserfs, reiserfsck, etc) do need to be able to
> handle bad blocks as well as possible since they are often needed
> during a salvage operation. in order to recover data (which might need
> to be migrated to a new disk). It is not clear to me that passing a
> list of bad blocks helps them as much as a robust general purpose
> error recovery support.
>
>
>
>
>
>
>
>

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[David Masover]

Hans Reiser wrote:
> Ric Wheeler wrote:

>> Having mkfs ignore bad writes would seem to encourage users to create
>> a new file system on a disk that is known to be bad & most likely not
>> going to function well.  If a user ever has a golden opportunity to
>> toss a drive in the trash, it is when they notice mkfs fails ;-)  This
>> option to mkfs sounds like an invitation to disaster.
> Yes, you are right, the option should be to run badblocks and then fail
> if it finds any.

Unless it creates significantly more work for us, there should be an 
option to run badblocks, and if it finds any, it should prompt the user 
(with BIG FAT CAPSLOCK WARNINGS) whether they want to format anyway. 
Formatting anyway should work, and we should be able to have blocks 
marked bad.

It would also be nice to be able to change this later -- to pass in a 
list of badblocks to, say, fsck (which I think is the original request). 
  This is especially nice for recovery, if you don't have the luxury of 
copying a whole disk image to another drive before running fsck.

That's not to say that we should automatically detect and relocate bad 
blocks during normal operation (while the FS is mounted), but 
deliberately removing functionality to protect you from yourself isn't 
the Linux Way.  Linux has a long history of kernel config options that 
say things like "YOU WILL LOSE DATA. You have been warned."

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Ric Wheeler]

David Masover wrote:

> Hans Reiser wrote:
>
>> Ric Wheeler wrote:
>
>
>>> Having mkfs ignore bad writes would seem to encourage users to create
>>> a new file system on a disk that is known to be bad & most likely not
>>> going to function well.  If a user ever has a golden opportunity to
>>> toss a drive in the trash, it is when they notice mkfs fails ;-)  This
>>> option to mkfs sounds like an invitation to disaster.
>>
>> Yes, you are right, the option should be to run badblocks and then fail
>> if it finds any.
>
>
> Unless it creates significantly more work for us, there should be an 
> option to run badblocks, and if it finds any, it should prompt the 
> user (with BIG FAT CAPSLOCK WARNINGS) whether they want to format 
> anyway. Formatting anyway should work, and we should be able to have 
> blocks marked bad.


I think that you are missing the way modern drives behave.  To give a 
typical example, on a 300 GB drive, we typically have 2000 or more extra 
sectors that are used for automatic remapping.  Theses sectors are 
consumed only when the drive retries a failed write multiple times.

If you fail a write, that means (barring even worse failure modes like a 
whole head going south) that all of these sectors have been consumed.

If they have not been consumed, the user will never see the remapping 
(it happens as part of a normal write, just takes longer than usual).

We really, really do not need a list of bad blocks to avoid during 
writing a new file system image. 

I think that the more interesting case is handling bad blocks during 
recovery.  It is not clear to me that fsck needs a list, but we have 
worked with Hans and Vladamir to get support for doing a reverse mapping 
(given a list of bad blocks, show the user what files, etc got hit).

>
> It would also be nice to be able to change this later -- to pass in a 
> list of badblocks to, say, fsck (which I think is the original 
> request).  This is especially nice for recovery, if you don't have the 
> luxury of copying a whole disk image to another drive before running 
> fsck.
>
> That's not to say that we should automatically detect and relocate bad 
> blocks during normal operation (while the FS is mounted), but 
> deliberately removing functionality to protect you from yourself isn't 
> the Linux Way.  Linux has a long history of kernel config options that 
> say things like "YOU WILL LOSE DATA. You have been warned."


The linux way is not to review ideas and see if they merit us coding 
them up.  LKML is nothing if not a long list of good/bad/weird ideas 
that get proposed, reviewed and often as not dumped in the dust bin of 
history ;-)  Ideas are good, discussion is great, but we should not 
invest in features that are known to produce a definite failure.

I spend a lot of time monitoring and helping debug file system/disk 
failures on a huge installed base of reiser3 file systems (running on 
sata and pata drives). As part of this, I spend a lot of time talking to 
disk vendors about how and when to pull the plug on bad drives.

ric

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[David Masover]

Ric Wheeler wrote:
> 
> 
> David Masover wrote:
> 
>> Hans Reiser wrote:
>>
>>> Ric Wheeler wrote:
>>
>>
>>>> Having mkfs ignore bad writes would seem to encourage users to create
>>>> a new file system on a disk that is known to be bad & most likely not
>>>> going to function well.  If a user ever has a golden opportunity to
>>>> toss a drive in the trash, it is when they notice mkfs fails ;-)  This
>>>> option to mkfs sounds like an invitation to disaster.
>>>
>>> Yes, you are right, the option should be to run badblocks and then fail
>>> if it finds any.
>>
>>
>> Unless it creates significantly more work for us, there should be an 
>> option to run badblocks, and if it finds any, it should prompt the 
>> user (with BIG FAT CAPSLOCK WARNINGS) whether they want to format 
>> anyway. Formatting anyway should work, and we should be able to have 
>> blocks marked bad.
> 
> 
> I think that you are missing the way modern drives behave.  To give a 
> typical example, on a 300 GB drive, we typically have 2000 or more extra 
> sectors that are used for automatic remapping.  Theses sectors are 
> consumed only when the drive retries a failed write multiple times.

Oh, I'm not disputing that mkfs should discourage users from using 
broken drives.  Presumably, smart admins wouldn't see this often, 
because they'd be monitoring SMART.

> We really, really do not need a list of bad blocks to avoid during 
> writing a new file system image.

Why do you presume to make this decision for users?

I don't think we need CONFIG_LEGACY_PTYS -- they're insecure, and almost 
never needed.  But we should still leave them in.  The burden is on us 
to show that it's taking real work to implement and maintain.

> I think that the more interesting case is handling bad blocks during 
> recovery.  It is not clear to me that fsck needs a list, but we have 
> worked with Hans and Vladamir to get support for doing a reverse mapping 
> (given a list of bad blocks, show the user what files, etc got hit).

Yes, it seems like fsck would be much better off that way.  In this 
case, of course, I'd prefer to avoid hitting that problem -- use RAID, 
make regular backups, toss out the disk and restore.  Being able to 
"repair bad blocks" would tend to encourage a user to keep using a bad 
disk, but I don't want to force my opinion on everyone when there's a 
reasonable way for all of us to be happy.

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Ric Wheeler]

David Masover wrote:
> Ric Wheeler wrote:
>>
>> I think that you are missing the way modern drives behave.  To give a 
>> typical example, on a 300 GB drive, we typically have 2000 or more 
>> extra sectors that are used for automatic remapping.  Theses sectors 
>> are consumed only when the drive retries a failed write multiple times.
> 
> 
> Oh, I'm not disputing that mkfs should discourage users from using 
> broken drives.  Presumably, smart admins wouldn't see this often, 
> because they'd be monitoring SMART.
> 
>> We really, really do not need a list of bad blocks to avoid during 
>> writing a new file system image.
> 
> 
> Why do you presume to make this decision for users?

It's not a decision that I want to make for users, it is a decision that 
Hans and his team need to make about how best to spend their limited 
resources.

Allowing users to put down reiser3/4 file systems on crap drives takes 
effort on their part and will result in an increased work load.

It will also give more users a bad experience with the file system, 
since users rarely have the in depth knowledge required to make this 
kind of choice.

> 
> I don't think we need CONFIG_LEGACY_PTYS -- they're insecure, and almost 
> never needed.  But we should still leave them in.  The burden is on us 
> to show that it's taking real work to implement and maintain.

This is a request for a new feature to allow users to do something, by 
design, that is extremely likely to lose all of their data. Not to 
extend support for an existing (braindead) legacy.

> 
>> I think that the more interesting case is handling bad blocks during 
>> recovery.  It is not clear to me that fsck needs a list, but we have 
>> worked with Hans and Vladamir to get support for doing a reverse 
>> mapping (given a list of bad blocks, show the user what files, etc got 
>> hit).
> 
> 
> Yes, it seems like fsck would be much better off that way.  In this 
> case, of course, I'd prefer to avoid hitting that problem -- use RAID, 
> make regular backups, toss out the disk and restore.  Being able to 
> "repair bad blocks" would tend to encourage a user to keep using a bad 
> disk, but I don't want to force my opinion on everyone when there's a 
> reasonable way for all of us to be happy.

Here we mostly agree.  The need for enhanced tools is not to encourage 
people to keep using bad drives, rather to allow them to fsck & remount 
a drive for data recovery.  If you cannot mount & fsck fails to repair 
enough to give you at least a readable file system, then you are in real 
trouble ;-)

Also, unlike failing writes, disk read errors are quite often ephemeral 
and will be self correcting on the next write (you might get an error 
from dust, etc that gets "swept" clean on the next write).

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[David Masover]

Ric Wheeler wrote:
> David Masover wrote:

>> Why do you presume to make this decision for users?
> 
> It's not a decision that I want to make for users, it is a decision that 
> Hans and his team need to make about how best to spend their limited 
> resources.

Agreed.  It's not important if it takes more than, say, an hour.

> It will also give more users a bad experience with the file system, 
> since users rarely have the in depth knowledge required to make this 
> kind of choice.

While it's true that most users just click through dialog boxes, I 
imagine this would be sufficient:

===WARNING===WARNING===WARNING===
---------------------------------
THIS DISK IS BAD!
If you continue with the format,
we will not help you when you lose data.
When, not if.  You are strongly encouraged to
THROW THIS DISK OUT!
---------------------------------------------
ARE YOU ABSOLUTELY SURE YOU WANT TO CONTINUE?
(yes/no):

And require an actual "yes" or "no" answer.  No "y/n".

Now, compare that to a filesystem which doesn't allow badblocks in mkfs 
at all.  While it's rare, I suspect that would be a worse experience if 
you actually had a real need for it.  If you've got a huge 300 gig drive 
with some bad blocks, you can always throw some stuff on it anyway, for 
backup, or stuff you don't care about, even knowing it'll fail soon.

Again, probably not a high priority item at all, but it certainly won't 
make the user experience worse.  Any user who says "yes" to the above 
warning does not get to complain about their experience.

> Here we mostly agree.  The need for enhanced tools is not to encourage 
> people to keep using bad drives, rather to allow them to fsck & remount 
> a drive for data recovery.  If you cannot mount & fsck fails to repair 
> enough to give you at least a readable file system, then you are in real 
> trouble ;-)
> 
> Also, unlike failing writes, disk read errors are quite often ephemeral 
> and will be self correcting on the next write (you might get an error 
> from dust, etc that gets "swept" clean on the next write).

Either one, I would personally feel quite a lot safer grabbing a disk 
image and doing the fsck once the image was on known good media.

One thing that would be even better here, though, if you don't want to 
spend the time for a huge backup:  A way to tell badblocks to only scan 
space which is actually being used, and then enough free space to make 
sure relocations work.  If you're mounting readonly, you shouldn't care 
about marking every single bad sector in free space.  I guess this would 
require a lot more intelligence from fsck, though -- it would have to be 
able to constantly check for bad blocks, as opposed to just running 
badblocks once and grabbing a list to avoid.

Re: FEATURE Req: integrate badblocks check into fsck.reiser*

[Peter]

On Thu, 07 Sep 2006 12:35:06 -0500, David Masover wrote:

all snip.....

All I was getting at originally was to make fsck.reiser4 behave similarly
to other fsck/mkfs programs so that it _could_, when requested, check a
disk/partition for bad blocks. Despite the improved capabilities of modern
drives, sh*t happens. And I, for one, would like the option of knowing
before formatting a new partition if it's OK to do so. Sure, I can run
badblocks myself, but this is counter to what I do with other fsck
programs.

A wrapper script would work fine, and I suppose, some sort of pipe output.
However, I'm just looking at this from a userland perspective, not that of
a sysadmin. Obviously, a sysadmin would not put any disk into production
that has not been carefully tested.

If this is superfluous, that's OK. I can easily work around it, but it
should be noted somewhere in a README that reiserfs/4 does not do bad
block checking during format and only a quick format is done.

-- 
Peter
+++++
Do not reply to this email, it is a spam trap and not monitored.
I can be reached via this list, or via 
jabber: pete4abw at jabber.org
ICQ: 73676357