Re: Corrupted/unreadable journal: reiser vs. ext3

Re: Corrupted/unreadable journal: reiser vs. ext3

[Dirk Schenkewitz]

Looks like i'm under a curse...

I can post, but I don't get any messages from the mailing list, 
only those that are explicitely addressed to me. I got to know
about all these other messages only because I requested the last
100 messages again :-/ Hmpf.

Back to the point:

Hans Reiser wrote (in response to Anders Widman):

> For some users it would be better to boot to a corrupted filesystem 
> because running fsck is more of a problem than putting their data at 
> higher risk.  For datalogging, it is probably conceivable to just toss 
> the journal and lose the more recent updates to it.  For the default 
> metadata journaling, this just does not seem prudent.

But I think that not everybody will know about if it's better to toss
the journal or to keep it. I wouldn't, and I know some people who are
much less interested in filesystems and the stuff around them than me.
Even SysOps.

> I really prefer making users understand that they have a problem they 
> need to do something about.  This is just my style.  I want them to fail 
> to boot, and after some effort learn that there is this thing called 
> fsck, and dd_rescue, and that it is time to buy another hard drive and 
> chuck their current one.

For me, it was alarming enough seeing ext3 drop the journal. In fact,
THAT was the point where I went to investigate in other directions
instead of blaming the filesystem. 

> It would be best though if they were given detailed instructions about
> how they need to do this when the code hits that bad block.

Agreed! The only problem is, that putting a bad drive to eternal rest
might not solve the problem, as long as the REASON for the drive gone
bad stays uncovered. (I had that said drive in use for less than 4
months (if my memory servers, er, serves my well) - it was like new.

> If we handle the journal block error without downtime, the user will 
> never chuck the hard drive, and that is bad in the longterm.

Not agreed, unless you continue without a warning.

happy coding
	dirk
-- 
Dirk Schenkewitz 

InterFace AG                 fon: +49 (0)89 / 610 49 - 126
Leipziger Str. 16            fax: +49 (0)89 / 610 49 - 83
D-82008 Unterhaching         
http://www.interface-ag.de   mailto:dirk.schenkewitz@interface-ag.de

Re: Corrupted/unreadable journal: reiser vs. ext3

[Zygo Blaxell]

In article <3E4AA902.86F15815@interface-ag.com>,
Dirk Schenkewitz  <Dirk.Schenkewitz@interface-ag.com> wrote:
>For me, it was alarming enough seeing ext3 drop the journal. In fact,
>THAT was the point where I went to investigate in other directions
>instead of blaming the filesystem. 

The kernel block device messages complaining about I/O errors from the
device aren't sufficient to tell you that there is a serious problem?
Or was this device silently corrupting data without reporting errors?

>The only problem is, that putting a bad drive to eternal rest
>might not solve the problem, as long as the REASON for the drive gone
>bad stays uncovered. (I had that said drive in use for less than 4
>months (if my memory servers, er, serves my well) - it was like new.

I've had disks that were DOA (literally--Medium Errors during
partitioning and mke2fs, followed by mechanical noises and total failure
in a matter of a few minutes).  I've had several disks that failed a
week or two after first installation.  

The M in MTBF is Mean, not Maximum or Minimum.  For every disk that
lasts 10 years or more, there's an equal and opposite disk that dies
within a few minutes.

>Hans Reiser wrote (in response to Anders Widman):
>> If we handle the journal block error without downtime, the user will 
>> never chuck the hard drive, and that is bad in the longterm.
>
>Not agreed, unless you continue without a warning.

I'd prefer to continue in read-only mode, and refuse further read-write
mounts with an error until the filesystem is fscked.  I really like
systems that can still boot and let me (attempt to) run diagnostic
tools even when they're otherwise really unhealthy.  I don't care if
recently written data is corrupt or missing--I probably didn't write to
the diagnostic tools within the last journal interval, and if the
filesystem is read-only I can't make any metadata corruption worse.

I would think that most people notice that something's wrong if they
can't write to their filesystems any more.  I certainly wouldn't want
the filesystem to be modified if there's something known to be wrong
with the metadata.  But if I can't read any of the data at all because
some tiny part of it is suspicious, I just get annoyed.  :-P

-- 
Zygo Blaxell (Laptop) <zblaxell@feedme.hungrycats.org>
GPG = D13D 6651 F446 9787 600B AD1E CCF3 6F93 2823 44AD

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Zygo Blaxell wrote:

>>Hans Reiser wrote (in response to Anders Widman):
>>    
>>
>>>If we handle the journal block error without downtime, the user will 
>>>never chuck the hard drive, and that is bad in the longterm.
>>>      
>>>
>>Not agreed, unless you continue without a warning.
>>    
>>
>
>I'd prefer to continue in read-only mode, and refuse further read-write
>mounts with an error until the filesystem is fscked.
>
Yes, I agree with that.

>  I really like
>systems that can still boot and let me (attempt to) run diagnostic
>tools even when they're otherwise really unhealthy.  I don't care if
>recently written data is corrupt or missing--I probably didn't write to
>the diagnostic tools within the last journal interval, and if the
>filesystem is read-only I can't make any metadata corruption worse.
>
>I would think that most people notice that something's wrong if they
>can't write to their filesystems any more.  I certainly wouldn't want
>the filesystem to be modified if there's something known to be wrong
>with the metadata.  But if I can't read any of the data at all because
>some tiny part of it is suspicious, I just get annoyed.  :-P
>
>  
>

Agreed.  I think that this is actually what we do currently.  Oleg, can 
you check that?

-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Oleg Drokin]

Hello!

On Fri, Feb 14, 2003 at 03:32:42AM +0300, Hans Reiser wrote:

> > I really like
> >systems that can still boot and let me (attempt to) run diagnostic
> >tools even when they're otherwise really unhealthy.  I don't care if
> >recently written data is corrupt or missing--I probably didn't write to
> >the diagnostic tools within the last journal interval, and if the
> >filesystem is read-only I can't make any metadata corruption worse.
> >I would think that most people notice that something's wrong if they
> >can't write to their filesystems any more.  I certainly wouldn't want
> >the filesystem to be modified if there's something known to be wrong
> >with the metadata.  But if I can't read any of the data at all because
> >some tiny part of it is suspicious, I just get annoyed.  :-P
> Agreed.  I think that this is actually what we do currently.  Oleg, can 
> you check that?

Currently we panic if write to journal area fails. We report IO error to
userspace if non-journaled write fails it seems (I will check it again).

Bye,
    Oleg 

Re: Corrupted/unreadable journal: reiser vs. ext3

[Andreas Dilger]

On Feb 14, 2003  11:18 +0300, Oleg Drokin wrote:
> Currently we panic if write to journal area fails. We report IO error to
> userspace if non-journaled write fails it seems (I will check it again).

I'm thinking "panic" isn't going to help the user's data any more than
not commiting the change...  How about remount-ro, or have a mount option
like ext3 "errors={panic,remount-ro,warning}"?  If you marked the filesystem
and/or journal in error and mount read-only, and force a full fsck at
reboot time at least the user has a chance - otherwise the node might
just panic in a loop.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Corrupted/unreadable journal: reiser vs. ext3

[Oleg Drokin]

Hello!

On Fri, Feb 14, 2003 at 03:13:16AM -0700, Andreas Dilger wrote:
> > Currently we panic if write to journal area fails. We report IO error to
> > userspace if non-journaled write fails it seems (I will check it again).
> I'm thinking "panic" isn't going to help the user's data any more than
> not commiting the change...  How about remount-ro, or have a mount option

SuSE people work on this.

> like ext3 "errors={panic,remount-ro,warning}"?  If you marked the filesystem
> and/or journal in error and mount read-only, and force a full fsck at
> reboot time at least the user has a chance - otherwise the node might
> just panic in a loop.

It hangs on panic ;) (because it does BUG() ), so no cyclical reboot.
There is even big chance that everything not touching problematic fs will
survive and continue to work.
Given that nobody runfs reiserfsck at boot, "full fsck" aproach won't work.
Ah, and reiserfsck ignores -a command line switch because
"we do not trust our fsck yet" (c) Hans.

Bye,
    Oleg

Re: Corrupted/unreadable journal: reiser vs. ext3

[Andreas Dilger]

On Feb 14, 2003  13:17 +0300, Oleg Drokin wrote:
> On Fri, Feb 14, 2003 at 03:13:16AM -0700, Andreas Dilger wrote:
> > > Currently we panic if write to journal area fails. We report IO error to
> > > userspace if non-journaled write fails it seems (I will check it again).
> > I'm thinking "panic" isn't going to help the user's data any more than
> > not commiting the change...  How about remount-ro, or have a mount option
> 
> SuSE people work on this.
> 
> > like ext3 "errors={panic,remount-ro,warning}"?  If you marked the filesystem
> > and/or journal in error and mount read-only, and force a full fsck at
> > reboot time at least the user has a chance - otherwise the node might
> > just panic in a loop.
> 
> It hangs on panic ;) (because it does BUG() ), so no cyclical reboot.

Ah, you said panic, but panic != BUG...  There is a "reboot-on-panic" flag
that is often set on servers so they don't sit stupidly when they could
reboot and start working again.

> There is even big chance that everything not touching problematic fs will
> survive and continue to work.
> Given that nobody runfs reiserfsck at boot, "full fsck" aproach won't work.
> Ah, and reiserfsck ignores -a command line switch because
> "we do not trust our fsck yet" (c) Hans.

Yeah, I keep giving him good reasons to change his mind, even a little,
like "have 'reiserfsck -a' just check the superblock and return with a
code > 1 if there is an error" so that an admin can at least do something
about it if the filesystem is broken, before it gets mounted/written to
again and the brokenness multiplies unknown to the user...

Next, add journal replay to reiserfsck if it isn't already there, and
_then_ do the same check as above, keeping a field in the journal header
to synchronously write an error to in fatal cases, instead of into the
superblock and where it is overwritten by journal replay.

That is all e2fsck does for ext3 filesystems, and it only takes a fraction
of a second to complete (no longer than it takes in-kernel journal replay
to complete at mount time, really) but the user wins by being able to fix
the filesystem before the whole system has booted and possibly corrupted
more data.

Regardless of whether reiserfsck is trusted or not to check/fix the
whole filesystem automatically, the above is not a risky change.  If you
wanted to go for more reliability, you could start adding quick "read
only" checks at periodic intervals like ext2 even if you never fix the
filesystem without user intervention.  The most common error we see on
the ext3 these days is due to memory/disk corruption that is caught by
the kernel or with a periodic check, which no amount of journaling can
fix or prevent.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Corrupted/unreadable journal: reiser vs. ext3

[Oleg Drokin]

Hello!

On Fri, Feb 14, 2003 at 03:50:34AM -0700, Andreas Dilger wrote:
> > > like ext3 "errors={panic,remount-ro,warning}"?  If you marked the filesystem
> > > and/or journal in error and mount read-only, and force a full fsck at
> > > reboot time at least the user has a chance - otherwise the node might
> > > just panic in a loop.
> > It hangs on panic ;) (because it does BUG() ), so no cyclical reboot.
> Ah, you said panic, but panic != BUG...  There is a "reboot-on-panic" flag

Yes, I know. I just used the wrong word. Tere is reiserfs_panic, but it does
BUG(), hence the confusion. (btw panic() in 2.5 seems not to halt the machine.
I was able to continue work on 2.5 after panic() was called. Though that was
during 2.5.20-something I think, so may be it is back to normal behaviour).

> that is often set on servers so they don't sit stupidly when they could
> reboot and start working again.

Yeah, I use that on my servers too.

Bye,
    Oleg 

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Andreas Dilger wrote:

>On Feb 14, 2003  13:17 +0300, Oleg Drokin wrote:
>  
>
>>On Fri, Feb 14, 2003 at 03:13:16AM -0700, Andreas Dilger wrote:
>>    
>>
>>>>Currently we panic if write to journal area fails. We report IO error to
>>>>userspace if non-journaled write fails it seems (I will check it again).
>>>>        
>>>>
>>>I'm thinking "panic" isn't going to help the user's data any more than
>>>not commiting the change...  How about remount-ro, or have a mount option
>>>      
>>>
>>SuSE people work on this.
>>
>>    
>>
>>>like ext3 "errors={panic,remount-ro,warning}"?  If you marked the filesystem
>>>and/or journal in error and mount read-only, and force a full fsck at
>>>reboot time at least the user has a chance - otherwise the node might
>>>just panic in a loop.
>>>      
>>>
>>It hangs on panic ;) (because it does BUG() ), so no cyclical reboot.
>>    
>>
>
>Ah, you said panic, but panic != BUG...  There is a "reboot-on-panic" flag
>that is often set on servers so they don't sit stupidly when they could
>reboot and start working again.
>
>  
>
>>There is even big chance that everything not touching problematic fs will
>>survive and continue to work.
>>Given that nobody runfs reiserfsck at boot, "full fsck" aproach won't work.
>>Ah, and reiserfsck ignores -a command line switch because
>>"we do not trust our fsck yet" (c) Hans.
>>    
>>
>
>Yeah, I keep giving him good reasons to change his mind, even a little,
>like "have 'reiserfsck -a' just check the superblock and return with a
>code > 1 if there is an error" so that an admin can at least do something
>about it if the filesystem is broken, before it gets mounted/written to
>again and the brokenness multiplies unknown to the user...
>
I don't understand you.

>
>Next, add journal replay to reiserfsck if it isn't already there,
>
Why, when it is in the kernel?

> and
>_then_ do the same check as above, keeping a field in the journal header
>to synchronously write an error to in fatal cases, instead of into the
>superblock and where it is overwritten by journal replay.
>
>That is all e2fsck does for ext3 filesystems, and it only takes a fraction
>of a second to complete (no longer than it takes in-kernel journal replay
>to complete at mount time, really) but the user wins by being able to fix
>the filesystem before the whole system has booted and possibly corrupted
>more data.
>
>Regardless of whether reiserfsck is trusted or not to check/fix the
>whole filesystem automatically, the above is not a risky change.  If you
>wanted to go for more reliability, you could start adding quick "read
>only" checks at periodic intervals like ext2 even if you never fix the
>filesystem without user intervention.  The most common error we see on
>the ext3 these days is due to memory/disk corruption that is caught by
>the kernel or with a periodic check, which no amount of journaling can
>fix or prevent.
>
I hate it when booting causes me to get stuck waiting for an fsck.

Probably fsck is stable enough now that we should encourage people to 
run it readonly regularly, but it should not be forced on them.

Maybe having some code to check whether fsck was run in the last 3 
months, and if not then if the user types y in the next 30 seconds 
during boot it will be run, would make sense.

The ext2 tradition of checking the number of mounts since the last fsck 
is simply counting the wrong thing.

>
>Cheers, Andreas
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
>
>  
>


-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Rudy Zijlstra]

On Fri, 14 Feb 2003, Hans Reiser wrote:

> Andreas Dilger wrote:
>
> >On Feb 14, 2003  13:17 +0300, Oleg Drokin wrote:
> >
> >
> >>On Fri, Feb 14, 2003 at 03:13:16AM -0700, Andreas Dilger wrote:
> >>
> >>
> >>>>Currently we panic if write to journal area fails. We report IO error to
> >>>>userspace if non-journaled write fails it seems (I will check it again).
> >>>>
> >>>>
> >>>I'm thinking "panic" isn't going to help the user's data any more than
> >>>not commiting the change...  How about remount-ro, or have a mount option
> >>>
> >>>
> >>SuSE people work on this.
> >>
> >>
> >>
> >>>like ext3 "errors={panic,remount-ro,warning}"?  If you marked the filesystem
> >>>and/or journal in error and mount read-only, and force a full fsck at
> >>>reboot time at least the user has a chance - otherwise the node might
> >>>just panic in a loop.
> >>>
> >>>
> >>It hangs on panic ;) (because it does BUG() ), so no cyclical reboot.
> >>
> >>
> >
> >Ah, you said panic, but panic != BUG...  There is a "reboot-on-panic" flag
> >that is often set on servers so they don't sit stupidly when they could
> >reboot and start working again.
> >
> >
> >
> >>There is even big chance that everything not touching problematic fs will
> >>survive and continue to work.
> >>Given that nobody runfs reiserfsck at boot, "full fsck" aproach won't work.
> >>Ah, and reiserfsck ignores -a command line switch because
> >>"we do not trust our fsck yet" (c) Hans.
> >>
> >>
> >
> >Yeah, I keep giving him good reasons to change his mind, even a little,
> >like "have 'reiserfsck -a' just check the superblock and return with a
> >code > 1 if there is an error" so that an admin can at least do something
> >about it if the filesystem is broken, before it gets mounted/written to
> >again and the brokenness multiplies unknown to the user...
> >
> I don't understand you.
>
> >
> >Next, add journal replay to reiserfsck if it isn't already there,
> >
> Why, when it is in the kernel?
Just to be certain, on read-only mount the journal replay is also done?
This would then solve the above question.

This would need support in the kernel to force read-only mount in case an
error flag as proposed above is set. For redundancy this would need to be
present in 2 places. (one place might be on the bad block..)



>
> > and
> >_then_ do the same check as above, keeping a field in the journal header
> >to synchronously write an error to in fatal cases, instead of into the
> >superblock and where it is overwritten by journal replay.
> >
> >That is all e2fsck does for ext3 filesystems, and it only takes a fraction
> >of a second to complete (no longer than it takes in-kernel journal replay
> >to complete at mount time, really) but the user wins by being able to fix
> >the filesystem before the whole system has booted and possibly corrupted
> >more data.
> >
> >Regardless of whether reiserfsck is trusted or not to check/fix the
> >whole filesystem automatically, the above is not a risky change.  If you
> >wanted to go for more reliability, you could start adding quick "read
> >only" checks at periodic intervals like ext2 even if you never fix the
> >filesystem without user intervention.  The most common error we see on
> >the ext3 these days is due to memory/disk corruption that is caught by
> >the kernel or with a periodic check, which no amount of journaling can
> >fix or prevent.
> >
> I hate it when booting causes me to get stuck waiting for an fsck.
>

So do I, its why i use Reiserfs...

> Probably fsck is stable enough now that we should encourage people to
> run it readonly regularly, but it should not be forced on them.
>

Is it then possible to give the -a switch a sensible meaning?

> Maybe having some code to check whether fsck was run in the last 3
> months, and if not then if the user types y in the next 30 seconds
> during boot it will be run, would make sense.
>
Regards,

Rudy

Re: Corrupted/unreadable journal: reiser vs. ext3

[Andreas Dilger]

On Feb 14, 2003  16:34 +0300, Hans Reiser wrote:
> Andreas Dilger wrote:
> >Yeah, I keep giving him good reasons to change his mind, even a little,
> >like "have 'reiserfsck -a' just check the superblock and return with a
> >code > 1 if there is an error" so that an admin can at least do something
> >about it if the filesystem is broken, before it gets mounted/written to
> >again and the brokenness multiplies unknown to the user...
>
> I don't understand you.

Ok, so the reiserfs kernel code detects an error on disk, what does it
do?  Print out an error message, maybe BUG?  There is an "error" field
in the reiserfs superblock, I hope it is set when the kernel detects
something bad.

So, now what happens?  Maybe the user doesn't read their syslog and
doesn't see the error, or the error is just a prelude to memory corruption
which causes the system to crash.  When the system boots again, it goes
on its merry way, mounting the reiserfs filesystem with _known_ errors
on it, using bad allocation bitmaps, directories btrees, etc and maybe
double allocating blocks or overwriting blocks from other files causing
them to become corrupt, etc, etc, etc.  Until finally the filesystem is
totally corrupt, the system crashes miserably, the user emails this list
and reiserfsck has an impossible job trying to fix the filesystem.

Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
simply check for a valid reiserfs superblock and the absence of the
"error" flag before declaring the filesystem clean and allowing the
system to boot.

What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
code OVERWRITES the superblock error status at mount time, making it
worse than useless, since each mount hides any errors that were detected
before the crash:

	s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
	s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;

> >Next, add journal replay to reiserfsck if it isn't already there,
> >
> Why, when it is in the kernel?

Because that is the next stage to allowing reiserfsck do checks on the
filesystem after a crash.  Do you tell me you would rather (and you
must, because it obviously currently does) have reiserfsck just throw
away everything in the journal, leaving possibly inconsistent data in
the filesystem for it to check?  Or maybe make the user mount the
filesystem (which obviously has problems or they wouldn't be running
reiserfsck to do a full check) just to clear out the journal and maybe
risk crashing or corruption if the filesystem is strangely corrupted?

> > and
> >_then_ do the same check as above, keeping a field in the journal header
> >to synchronously write an error to in fatal cases, instead of into the
> >superblock and where it is overwritten by journal replay.
> >
> >That is all e2fsck does for ext3 filesystems, and it only takes a fraction
> >of a second to complete (no longer than it takes in-kernel journal replay
> >to complete at mount time, really) but the user wins by being able to fix
> >the filesystem before the whole system has booted and possibly corrupted
> >more data.
>
> >Regardless of whether reiserfsck is trusted or not to check/fix the
> >whole filesystem automatically, the above is not a risky change.  If you
> >wanted to go for more reliability, you could start adding quick "read
> >only" checks at periodic intervals like ext2 even if you never fix the
> >filesystem without user intervention.  The most common error we see on
> >the ext3 these days is due to memory/disk corruption that is caught by
> >the kernel or with a periodic check, which no amount of journaling can
> >fix or prevent.
>
> I hate it when booting causes me to get stuck waiting for an fsck.

You don't hate it more when you lose data?  You are a strange man then.

> Probably fsck is stable enough now that we should encourage people to 
> run it readonly regularly, but it should not be forced on them.

I'm nowhere suggesting that reiserfsck _has_ to implement the additional
periodic read-only checks at boot time, mostly just that it do _very_simple_
checks that take .01 seconds at boot time before the filesystem is mounted,
so that the admin/user at least _notices_ that there is an error instead of
not finding out until their filesystem is totally corrupted.

> Maybe having some code to check whether fsck was run in the last 3 
> months, and if not then if the user types y in the next 30 seconds 
> during boot it will be run, would make sense.

Sure, that would be great, given the prevelance of memory errors and
IDE DMA errors that show up these days, which the filesystem and the
journal can do nothing about.

> The ext2 tradition of checking the number of mounts since the last fsck 
> is simply counting the wrong thing.

It's only a matter of defaults safe vs. fast...  e2fsck defaults to safe,
checking occasionally for possible corruption, vs. reiserfs waiting for
fatal corruption before forcing the user to run reiserfsck (which is so
heavily discouraged (on the list, documentation, when run), that nobody
runs it for fear of damaging their filesystem further.  You are well aware
that the e2fsck check intervals can be tuned per-filesystem and even
disabled if desired (it prints options for how to do this at mke2fs time
and is clearly documented for the experienced user).  For a boot-once-a-day
machine, the default is to check about once a month (at most 6 months for
the time check), and if machines are crashing more often, then they should
probably be checked more often because _something_ has to be causing crashes.

Having reiserfsck just do read-only checks shouldn't force you to type
"yes" (and we mean "yes" because this is so scary, mere mortals shouldn't
be doing this).  Hans, you've always talked about making things easy for
the average user (error messages and such), don't you think that making
a data consistency check for the user a little less intimidating too?

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Andreas Dilger wrote:

>On Feb 14, 2003  16:34 +0300, Hans Reiser wrote:
>  
>
>>Andreas Dilger wrote:
>>    
>>
>>>Yeah, I keep giving him good reasons to change his mind, even a little,
>>>like "have 'reiserfsck -a' just check the superblock and return with a
>>>code > 1 if there is an error" so that an admin can at least do something
>>>about it if the filesystem is broken, before it gets mounted/written to
>>>again and the brokenness multiplies unknown to the user...
>>>      
>>>
>>I don't understand you.
>>    
>>
>
>Ok, so the reiserfs kernel code detects an error on disk, what does it
>do?  Print out an error message, maybe BUG?  There is an "error" field
>in the reiserfs superblock, I hope it is set when the kernel detects
>something bad.
>
>So, now what happens?  Maybe the user doesn't read their syslog and
>doesn't see the error, or the error is just a prelude to memory corruption
>which causes the system to crash.  When the system boots again, it goes
>on its merry way, mounting the reiserfs filesystem with _known_ errors
>on it, using bad allocation bitmaps, directories btrees, etc and maybe
>double allocating blocks or overwriting blocks from other files causing
>them to become corrupt, etc, etc, etc.  Until finally the filesystem is
>totally corrupt, the system crashes miserably, the user emails this list
>and reiserfsck has an impossible job trying to fix the filesystem.
>
>Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
>simply check for a valid reiserfs superblock and the absence of the
>"error" flag before declaring the filesystem clean and allowing the
>system to boot.
>
>What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
>code OVERWRITES the superblock error status at mount time, making it
>worse than useless, since each mount hides any errors that were detected
>before the crash:
>
>	s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
>	s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;
>
Andreas seems reasonable, Vitaly, what are your thoughts?

>
>  
>
>>>Next, add journal replay to reiserfsck if it isn't already there,
>>>
>>>      
>>>
>>Why, when it is in the kernel?
>>    
>>
>
>Because that is the next stage to allowing reiserfsck do checks on the
>filesystem after a crash.  Do you tell me you would rather (and you
>must, because it obviously currently does) have reiserfsck just throw
>away everything in the journal, leaving possibly inconsistent data in
>the filesystem for it to check?  Or maybe make the user mount the
>filesystem (which obviously has problems or they wouldn't be running
>reiserfsck to do a full check) just to clear out the journal and maybe
>risk crashing or corruption if the filesystem is strangely corrupted?
>
Vitaly, answer this.

>
>  
>
>>Maybe having some code to check whether fsck was run in the last 3 
>>months, and if not then if the user types y in the next 30 seconds 
>>during boot it will be run, would make sense.
>>    
>>
>
>Sure, that would be great, given the prevelance of memory errors and
>IDE DMA errors that show up these days, which the filesystem and the
>journal can do nothing about.
>
>  
>
>>The ext2 tradition of checking the number of mounts since the last fsck 
>>is simply counting the wrong thing.
>>    
>>
>
>It's only a matter of defaults safe vs. fast...  e2fsck defaults to safe,
>checking occasionally for possible corruption, vs. reiserfs waiting for
>fatal corruption before forcing the user to run reiserfsck (which is so
>heavily discouraged (on the list, documentation, when run), that nobody
>runs it for fear of damaging their filesystem further. 
>
It is probably not so dangerous anymore.

> You are well aware
>that the e2fsck check intervals can be tuned per-filesystem and even
>disabled if desired (it prints options for how to do this at mke2fs time
>and is clearly documented for the experienced user).  For a boot-once-a-day
>machine, the default is to check about once a month (at most 6 months for
>the time check), and if machines are crashing more often, then they should
>probably be checked more often because _something_ has to be causing crashes.
>
The idea that how often you boot determines how often it checks is just 
silly, sorry.

>
>Having reiserfsck just do read-only checks shouldn't force you to type
>"yes" (and we mean "yes" because this is so scary, mere mortals shouldn't
>be doing this).  Hans, you've always talked about making things easy for
>the average user (error messages and such), don't you think that making
>a data consistency check for the user a little less intimidating too?
>
>  
>
I think that you should have to agree that you have time to wait for 
fsck before you get stuck with a 1 day large server fsck.

-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Vitaly Fertman]

> >Ok, so the reiserfs kernel code detects an error on disk, what does it
> >do?  Print out an error message, maybe BUG?  There is an "error" field
> >in the reiserfs superblock, I hope it is set when the kernel detects
> >something bad.
> >
> >So, now what happens?  Maybe the user doesn't read their syslog and
> >doesn't see the error, or the error is just a prelude to memory corruption
> >which causes the system to crash.  When the system boots again, it goes
> >on its merry way, mounting the reiserfs filesystem with _known_ errors
> >on it, using bad allocation bitmaps, directories btrees, etc and maybe
> >double allocating blocks or overwriting blocks from other files causing
> >them to become corrupt, etc, etc, etc.  Until finally the filesystem is
> >totally corrupt, the system crashes miserably, the user emails this list
> >and reiserfsck has an impossible job trying to fix the filesystem.
> >
> >Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
> >simply check for a valid reiserfs superblock and the absence of the
> >"error" flag before declaring the filesystem clean and allowing the
> >system to boot.
> >
> >What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
> >code OVERWRITES the superblock error status at mount time, making it
> >worse than useless, since each mount hides any errors that were detected
> >before the crash:
> >
> >	s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
> >	s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;
>
> Andreas seems reasonable, Vitaly, what are your thoughts?
>
> >>>Next, add journal replay to reiserfsck if it isn't already there,
> >>
> >>Why, when it is in the kernel?
> >
> >Because that is the next stage to allowing reiserfsck do checks on the
> >filesystem after a crash.  Do you tell me you would rather (and you
> >must, because it obviously currently does) have reiserfsck just throw
> >away everything in the journal, leaving possibly inconsistent data in
> >the filesystem for it to check?  Or maybe make the user mount the
> >filesystem (which obviously has problems or they wouldn't be running
> >reiserfsck to do a full check) just to clear out the journal and maybe
> >risk crashing or corruption if the filesystem is strangely corrupted?
>
> Vitaly, answer this.

Ok, so probably we should make the following changes. The kernel set IO_ERROR
and FS_ERROR flags. 
In the case of IO_ERROR reiserfsck prints the message about hardware problems 
and returns error, so the fs does not get mounted at boot. On attempt mounting 
the fs with IO_ERROR flag set it is mounted ro with some message about hardware 
problems. When you are sure that problems disappeared you can mount it with a 
spetial option cleaning this flag and probably reiserfstune will have some 
option cleaning these flags also.
In the case of FS_ERROR - search_by_key failed or beyond end of device access 
or similar - reiserfsck gets -a option at boot, replays the journal if needed 
and checks for the flag. No flag - returns OK. Else - run fix-fixable. Errors
left - returns 'errors left uncorrected' and the fs does not get mounted at 
boot. On attempt mounting the fs with the flag just print the message about 
mounting the fs with errors and mount it. Not ro here as kernel will not do 
deep analysis of errors and it could be just a small insignificant error.

-- 

Thanks,
Vitaly Fertman

Re: Corrupted/unreadable journal: reiser vs. ext3

[Vitaly Fertman]

On Saturday 15 February 2003 15:51, Vitaly Fertman wrote:
> > >Ok, so the reiserfs kernel code detects an error on disk, what does it
> > >do?  Print out an error message, maybe BUG?  There is an "error" field
> > >in the reiserfs superblock, I hope it is set when the kernel detects
> > >something bad.
> > >
> > >So, now what happens?  Maybe the user doesn't read their syslog and
> > >doesn't see the error, or the error is just a prelude to memory
> > > corruption which causes the system to crash.  When the system boots
> > > again, it goes on its merry way, mounting the reiserfs filesystem with
> > > _known_ errors on it, using bad allocation bitmaps, directories btrees,
> > > etc and maybe double allocating blocks or overwriting blocks from other
> > > files causing them to become corrupt, etc, etc, etc.  Until finally the
> > > filesystem is totally corrupt, the system crashes miserably, the user
> > > emails this list and reiserfsck has an impossible job trying to fix the
> > > filesystem.
> > >
> > >Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
> > >simply check for a valid reiserfs superblock and the absence of the
> > >"error" flag before declaring the filesystem clean and allowing the
> > >system to boot.
> > >
> > >What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
> > >code OVERWRITES the superblock error status at mount time, making it
> > >worse than useless, since each mount hides any errors that were detected
> > >before the crash:
> > >
> > >	s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
> > >	s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;
> >
> > Andreas seems reasonable, Vitaly, what are your thoughts?
> >
> > >>>Next, add journal replay to reiserfsck if it isn't already there,
> > >>
> > >>Why, when it is in the kernel?
> > >
> > >Because that is the next stage to allowing reiserfsck do checks on the
> > >filesystem after a crash.  Do you tell me you would rather (and you
> > >must, because it obviously currently does) have reiserfsck just throw
> > >away everything in the journal, leaving possibly inconsistent data in
> > >the filesystem for it to check?  Or maybe make the user mount the
> > >filesystem (which obviously has problems or they wouldn't be running
> > >reiserfsck to do a full check) just to clear out the journal and maybe
> > >risk crashing or corruption if the filesystem is strangely corrupted?
> >
> > Vitaly, answer this.
>
> Ok, so probably we should make the following changes. The kernel set
> IO_ERROR and FS_ERROR flags.
> In the case of IO_ERROR reiserfsck prints the message about hardware
> problems and returns error, so the fs does not get mounted at boot. On
> attempt mounting the fs with IO_ERROR flag set it is mounted ro with some
> message about hardware problems. When you are sure that problems
> disappeared you can mount it with a spetial option cleaning this flag and
> probably reiserfstune will have some option cleaning these flags also.
> In the case of FS_ERROR - search_by_key failed or beyond end of device
> access or similar - reiserfsck gets -a option at boot, replays the journal
> if needed and checks for the flag. No flag - returns OK. Else - run
> fix-fixable. Errors left - returns 'errors left uncorrected' and the fs
> does not get mounted at boot. On attempt mounting the fs with the flag just
> print the message about mounting the fs with errors and mount it. Not ro
> here as kernel will not do deep analysis of errors and it could be just a
> small insignificant error.

If fix-fixable finds fatal corruptions it sets FS_FATAL what means that the
second run of fix-fixable will not help to avoid another fsck run at the next 
boot.

-- 

Thanks,
Vitaly Fertman

Re: Corrupted/unreadable journal: reiser vs. ext3

[Anders Widman]

> Ok, so probably we should make the following changes. The kernel set IO_ERROR
> and FS_ERROR flags. 
> In the case of IO_ERROR reiserfsck prints the message about hardware problems
> and returns error, so the fs does not get mounted at boot. On attempt mounting
> the fs with IO_ERROR flag set it is mounted ro with some message about hardware
> problems. When you are sure that problems disappeared you can mount it with a
> spetial option cleaning this flag and probably reiserfstune will have some
> option cleaning these flags also.
> In the case of FS_ERROR - search_by_key failed or beyond end of device access
> or similar - reiserfsck gets -a option at boot, replays the journal if needed
> and checks for the flag. No flag - returns OK. Else - run fix-fixable. Errors
> left - returns 'errors left uncorrected' and the fs does not get mounted at
> boot. On attempt mounting the fs with the flag just print the message about
> mounting the fs with errors and mount it. Not ro here as kernel will not do
> deep analysis of errors and it could be just a small insignificant error.


  What  happens if there are FS errors on boot device? Not mounting ro
  will  make  system  inaccessible unless users has some other boot or
  rescue media.

  Of  course, this  is  where  admins  should  have  another  disk  or
  partition to boot and make FS repairs. :)

   - Anders

  

Re: Corrupted/unreadable journal: reiser vs. ext3

[Oleg Drokin]

Hello!

On Sat, Feb 15, 2003 at 02:04:54PM +0100, Anders Widman wrote:
> > Ok, so probably we should make the following changes. The kernel set IO_ERROR
> > and FS_ERROR flags. 
> > In the case of IO_ERROR reiserfsck prints the message about hardware problems
> > and returns error, so the fs does not get mounted at boot. On attempt mounting
> > the fs with IO_ERROR flag set it is mounted ro with some message about hardware
> > problems. When you are sure that problems disappeared you can mount it with a
> > spetial option cleaning this flag and probably reiserfstune will have some
> > option cleaning these flags also.
> > In the case of FS_ERROR - search_by_key failed or beyond end of device access
> > or similar - reiserfsck gets -a option at boot, replays the journal if needed
> > and checks for the flag. No flag - returns OK. Else - run fix-fixable. Errors
> > left - returns 'errors left uncorrected' and the fs does not get mounted at
> > boot. On attempt mounting the fs with the flag just print the message about
> > mounting the fs with errors and mount it. Not ro here as kernel will not do
> > deep analysis of errors and it could be just a small insignificant error.
>   What  happens if there are FS errors on boot device? Not mounting ro

It get's mounted readonly first, then fsck will exit with appropriate error code,
then system will refuse to continue booting process, I presume ;)

>   will  make  system  inaccessible unless users has some other boot or
>   rescue media.

Sure, just like with any other filesystem.
If you want to avoid that, don't run fsck on your root device during boot,
or ignore what it have said to you ;)

>   Of  course, this  is  where  admins  should  have  another  disk  or
>   partition to boot and make FS repairs. :)

Actually it is possible to reiserfsck (readonly) mounted filesystem. The only
prerequisite I found is reiserfsck binary (built statically it seems, or
with all the libs) should be located on another fs (tmpfs works just fine for me).
And of course little bit of reiserfsck patching is needed ;)
That's how I repair broken root filesystems on our test-boxes when rootfs is getting
corrupt, because some moron\bclear guy have not created rescue partition in there
and I hate booting from CDs (And CDs usually have outdated reiserfsck anyway). 

Bye,
    Oleg

Re: Corrupted/unreadable journal: reiser vs. ext3

[Andreas Dilger]

On Feb 14, 2003  22:19 +0300, Hans Reiser wrote:
> Andreas Dilger wrote:
> > You are well aware
> >that the e2fsck check intervals can be tuned per-filesystem and even
> >disabled if desired (it prints options for how to do this at mke2fs time
> >and is clearly documented for the experienced user).  For a boot-once-a-day
> >machine, the default is to check about once a month (at most 6 months for
> >the time check), and if machines are crashing more often, then they should
> >probably be checked more often because _something_ has to be causing crashes.
> >
> The idea that how often you boot determines how often it checks is just 
> silly, sorry.

I guess the shortcoming in the ext2 case is that it counts mounts and
not crashes.  If it were counting the number of times the filesystem
was uncleanly shut down instead of normal shutdowns, would that be more
acceptable?  The reason I'm still interested in crashes, even if they
are not filesystem-related crashes, is because there had to be _something_
which caused a crash (bad code, bad hardware, whatever), and once you have
any driver corrupting memory the chance that it is also corrupting filesystem
memory exists.

> >Having reiserfsck just do read-only checks shouldn't force you to type
> >"yes" (and we mean "yes" because this is so scary, mere mortals shouldn't
> >be doing this).  Hans, you've always talked about making things easy for
> >the average user (error messages and such), don't you think that making
> >a data consistency check for the user a little less intimidating too?
>
> I think that you should have to agree that you have time to wait for 
> fsck before you get stuck with a 1 day large server fsck.

That is definitely true.  However, my assumption would be that if someone
is running a system with terabytes of data they will read the man page
after waiting a day for fsck to complete, or lose their job.  It is entirely
possible for administrators to disable the per-mount e2fsck checking, and
the time-based (6 months by default) checking too, and do fsck themselves.
My experience would be that, like backups, people don't do that, so leaving
the 6 month check in protects users from themselves.

The other thing to keep in mind is that you can have different "levels" of
automated fsck at boot time, depending on how long they take.  You never
necessarily have to try and fix anything with "fsck -a", just detect errors
and leave it up to the user to decide what to do if you find a problem:
- always recover journal, validate superblock, error flag (< 1s)

Don't know how long it takes these things to run, so it is up to you to
trade off checks vs. speed, and you could even round-robin them (storing
the last checked item in the superblock or something):
- check block allocation bitmaps match superblock counts
- walk directory structure from root, checking for directory corruption
- check btree validity on inodes for up to 10 seconds (or whatever, storing
  last checked inode in superblock for restarting this test at next one)

By all means, don't do checks for an hour, or allow users to set the maximum
boot check duration in the superblock.  I'm sure users don't mind waiting
5s at boot time if it means they don't lose data.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Vitaly Fertman wrote:

>>>Ok, so the reiserfs kernel code detects an error on disk, what does it
>>>do?  Print out an error message, maybe BUG?  There is an "error" field
>>>in the reiserfs superblock, I hope it is set when the kernel detects
>>>something bad.
>>>
>>>So, now what happens?  Maybe the user doesn't read their syslog and
>>>doesn't see the error, or the error is just a prelude to memory corruption
>>>which causes the system to crash.  When the system boots again, it goes
>>>on its merry way, mounting the reiserfs filesystem with _known_ errors
>>>on it, using bad allocation bitmaps, directories btrees, etc and maybe
>>>double allocating blocks or overwriting blocks from other files causing
>>>them to become corrupt, etc, etc, etc.  Until finally the filesystem is
>>>totally corrupt, the system crashes miserably, the user emails this list
>>>and reiserfsck has an impossible job trying to fix the filesystem.
>>>
>>>Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
>>>simply check for a valid reiserfs superblock and the absence of the
>>>"error" flag before declaring the filesystem clean and allowing the
>>>system to boot.
>>>
>>>What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
>>>code OVERWRITES the superblock error status at mount time, making it
>>>worse than useless, since each mount hides any errors that were detected
>>>before the crash:
>>>
>>>	s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
>>>	s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;
>>>      
>>>
>>Andreas seems reasonable, Vitaly, what are your thoughts?
>>
>>    
>>
>>>>>Next, add journal replay to reiserfsck if it isn't already there,
>>>>>          
>>>>>
>>>>Why, when it is in the kernel?
>>>>        
>>>>
>>>Because that is the next stage to allowing reiserfsck do checks on the
>>>filesystem after a crash.  Do you tell me you would rather (and you
>>>must, because it obviously currently does) have reiserfsck just throw
>>>away everything in the journal, leaving possibly inconsistent data in
>>>the filesystem for it to check?  Or maybe make the user mount the
>>>filesystem (which obviously has problems or they wouldn't be running
>>>reiserfsck to do a full check) just to clear out the journal and maybe
>>>risk crashing or corruption if the filesystem is strangely corrupted?
>>>      
>>>
>>Vitaly, answer this.
>>    
>>
>
>Ok, so probably we should make the following changes. The kernel set IO_ERROR
>and FS_ERROR flags. 
>In the case of IO_ERROR reiserfsck prints the message about hardware problems 
>and returns error, so the fs does not get mounted at boot. On attempt mounting 
>the fs with IO_ERROR flag set it is mounted ro with some message about hardware 
>problems. When you are sure that problems disappeared you can mount it with a 
>spetial option cleaning this flag and probably reiserfstune will have some 
>option cleaning these flags also.
>In the case of FS_ERROR - search_by_key failed or beyond end of device access 
>or similar - reiserfsck gets -a option at boot, replays the journal if needed 
>and checks for the flag. No flag - returns OK. Else - run fix-fixable. Errors
>left - returns 'errors left uncorrected' and the fs does not get mounted at 
>boot. On attempt mounting the fs with the flag just print the message about 
>mounting the fs with errors and mount it. Not ro here as kernel will not do 
>deep analysis of errors and it could be just a small insignificant error.
>
>  
>
Sounds good to me.  Do it.  Reiser4 also.

-- 
Hans

Error <-> Partition Correspondance [was Re: Corrupted/unreadable journal: reiser vs. ext3]

[Manuel Krause]

On 02/17/2003 08:43 PM, Hans Reiser wrote:
> Vitaly Fertman wrote:
> 
>>>> Ok, so the reiserfs kernel code detects an error on disk, what does it
>>>> do?  Print out an error message, maybe BUG?  There is an "error" field
>>>> in the reiserfs superblock, I hope it is set when the kernel detects
>>>> something bad.
>>>>
>>>> So, now what happens?  Maybe the user doesn't read their syslog and
>>>> doesn't see the error, or the error is just a prelude to memory 
>>>> corruption
>>>> which causes the system to crash.  When the system boots again, it goes
>>>> on its merry way, mounting the reiserfs filesystem with _known_ errors
>>>> on it, using bad allocation bitmaps, directories btrees, etc and maybe
>>>> double allocating blocks or overwriting blocks from other files causing
>>>> them to become corrupt, etc, etc, etc.  Until finally the filesystem is
>>>> totally corrupt, the system crashes miserably, the user emails this 
>>>> list
>>>> and reiserfsck has an impossible job trying to fix the filesystem.
>>>>
>>>> Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
>>>> simply check for a valid reiserfs superblock and the absence of the
>>>> "error" flag before declaring the filesystem clean and allowing the
>>>> system to boot.
>>>>
>>>> What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
>>>> code OVERWRITES the superblock error status at mount time, making it
>>>> worse than useless, since each mount hides any errors that were 
>>>> detected
>>>> before the crash:
>>>>
>>>>     s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
>>>>     s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;
>>>>     
>>>
>>> Andreas seems reasonable, Vitaly, what are your thoughts?
>>>
>>>   
>>>
>>>>>> Next, add journal replay to reiserfsck if it isn't already there,
>>>>>>         
>>>>>
>>>>> Why, when it is in the kernel?
>>>>>       
>>>>
>>>> Because that is the next stage to allowing reiserfsck do checks on the
>>>> filesystem after a crash.  Do you tell me you would rather (and you
>>>> must, because it obviously currently does) have reiserfsck just throw
>>>> away everything in the journal, leaving possibly inconsistent data in
>>>> the filesystem for it to check?  Or maybe make the user mount the
>>>> filesystem (which obviously has problems or they wouldn't be running
>>>> reiserfsck to do a full check) just to clear out the journal and maybe
>>>> risk crashing or corruption if the filesystem is strangely corrupted?
>>>>     
>>>
>>> Vitaly, answer this.
>>>   
>>
>>
>> Ok, so probably we should make the following changes. The kernel set 
>> IO_ERROR
>> and FS_ERROR flags. In the case of IO_ERROR reiserfsck prints the 
>> message about hardware problems and returns error, so the fs does not 
>> get mounted at boot. On attempt mounting the fs with IO_ERROR flag set 
>> it is mounted ro with some message about hardware problems. When you 
>> are sure that problems disappeared you can mount it with a spetial 
>> option cleaning this flag and probably reiserfstune will have some 
>> option cleaning these flags also.
>> In the case of FS_ERROR - search_by_key failed or beyond end of device 
>> access or similar - reiserfsck gets -a option at boot, replays the 
>> journal if needed and checks for the flag. No flag - returns OK. Else 
>> - run fix-fixable. Errors
>> left - returns 'errors left uncorrected' and the fs does not get 
>> mounted at boot. On attempt mounting the fs with the flag just print 
>> the message about mounting the fs with errors and mount it. Not ro 
>> here as kernel will not do deep analysis of errors and it could be 
>> just a small insignificant error.
>>
>>  
>>
> Sounds good to me.  Do it.  Reiser4 also.

Hi!

BTW, do the ReiserFS errors nowadays print out a usable partition 
identification (like Chris actual data-logging patches perform at mount, 
e.g.)?

I mostly always have 2 partitions with ReiserFS mounted, so -- is it 
still meaningless to get an error message related to one of them in my logs?

[For long times now (more than 6 months) I did not get any ReiserFS 
errors any more even with data-logging and preempt-kernel applied -- I 
only read them over the list. So I don't know the real meaning of error 
messages' variables content any more... :-( or really :-)))   ]

I posted this circumstance some 3.6-ReiserFS levels ago and someone of 
your team wanted to implement this after his task-list was done, IIRC.

So, if it's not implemented explicitly in words so far, this would seem 
to me to be valuable for users, too, IMO.


Best regards,

Manuel

Re: Error <-> Partition Correspondance [was Re: Corrupted/unreadable journal: reiser vs. ext3]

[Oleg Drokin]

Hello!

On Tue, Feb 18, 2003 at 12:35:23AM +0100, Manuel Krause wrote:

> BTW, do the ReiserFS errors nowadays print out a usable partition 
> identification (like Chris actual data-logging patches perform at mount, 
> e.g.)?

Sometimes it does.

> I mostly always have 2 partitions with ReiserFS mounted, so -- is it 
> still meaningless to get an error message related to one of them in my logs?

It depends on what are the messages.

> I posted this circumstance some 3.6-ReiserFS levels ago and someone of 
> your team wanted to implement this after his task-list was done, IIRC.

Yes. I have a patch dated back to May 7th, 2002. But it was never
accepted for reason I don't remember already.
I will dig through my email, though. Probably I will give it another try.

Bye,
    Oleg

fsck on boot (was: Re: Corrupted/unreadable journal: reiser vs. ext3)

[Ookhoi]

Andreas Dilger wrote (ao):
> The other thing to keep in mind is that you can have different
> "levels" of automated fsck at boot time, depending on how long they
> take.  You never necessarily have to try and fix anything with "fsck
> -a", just detect errors and leave it up to the user to decide what to
> do if you find a problem: - always recover journal, validate
> superblock, error flag (< 1s)
> 
> Don't know how long it takes these things to run, so it is up to you
> to trade off checks vs. speed, and you could even round-robin them
> (storing the last checked item in the superblock or something):
> - check block allocation bitmaps match superblock counts
> - walk directory structure from root, checking for directory
>   corruption
> - check btree validity on inodes for up to 10 seconds (or whatever,
>   storing last checked inode in superblock for restarting this test at
>   next one)
> 
> By all means, don't do checks for an hour, or allow users to set the
> maximum boot check duration in the superblock.  I'm sure users don't
> mind waiting 5s at boot time if it means they don't lose data.

Yes! Yes! I agree so much on this .. Let fsck always run at boot, and
perform checks which take at most a few seconds all together.

Then dmesg will tell if something is wrong. Maybe it can also show the
error code in /proc/mounts ?

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Andreas Dilger wrote:

>On Feb 14, 2003  22:19 +0300, Hans Reiser wrote:
>  
>
>>Andreas Dilger wrote:
>>    
>>
>>>You are well aware
>>>that the e2fsck check intervals can be tuned per-filesystem and even
>>>disabled if desired (it prints options for how to do this at mke2fs time
>>>and is clearly documented for the experienced user).  For a boot-once-a-day
>>>machine, the default is to check about once a month (at most 6 months for
>>>the time check), and if machines are crashing more often, then they should
>>>probably be checked more often because _something_ has to be causing crashes.
>>>
>>>      
>>>
>>The idea that how often you boot determines how often it checks is just 
>>silly, sorry.
>>    
>>
>
>I guess the shortcoming in the ext2 case is that it counts mounts and
>not crashes.  If it were counting the number of times the filesystem
>was uncleanly shut down instead of normal shutdowns, would that be more
>acceptable?  The reason I'm still interested in crashes, even if they
>are not filesystem-related crashes, is because there had to be _something_
>which caused a crash (bad code, bad hardware, whatever), and once you have
>any driver corrupting memory the chance that it is also corrupting filesystem
>memory exists.
>
This is at least arguably legitimate;-)....

>
>  
>
>>>Having reiserfsck just do read-only checks shouldn't force you to type
>>>"yes" (and we mean "yes" because this is so scary, mere mortals shouldn't
>>>be doing this).  Hans, you've always talked about making things easy for
>>>the average user (error messages and such), don't you think that making
>>>a data consistency check for the user a little less intimidating too?
>>>      
>>>
>>I think that you should have to agree that you have time to wait for 
>>fsck before you get stuck with a 1 day large server fsck.
>>    
>>
>
>That is definitely true.  However, my assumption would be that if someone
>is running a system with terabytes of data they will read the man page
>after waiting a day for fsck to complete, or lose their job.
>
How much does a terabyte of disk cost?  A thousand dollars?  How much 
does a qualified sysadmin cost?  $100-200k in Silicon Valley (but 
rapidly reducing).

Yet this is still the wrong attitude....  our job is to make the 
software so that it works without hassle.   They don't need more items 
on their checklists, they need software that manages the checklists for 
them.

Also,  whether a sysadmin is willing to wait a day for fsck might depend 
on the day you ask him.

So I completely reject the argument you make.

>  It is entirely
>possible for administrators to disable the per-mount e2fsck checking, and
>the time-based (6 months by default) checking too, and do fsck themselves.
>My experience would be that, like backups, people don't do that, so leaving
>the 6 month check in protects users from themselves.
>
Most users don't know that they can do it, and those that do don't need 
us giving them more things they need to set when installing the OS.

>
>The other thing to keep in mind is that you can have different "levels" of
>automated fsck at boot time, depending on how long they take.  You never
>necessarily have to try and fix anything with "fsck -a", just detect errors
>and leave it up to the user to decide what to do if you find a problem:
>- always recover journal, validate superblock, error flag (< 1s)
>
>Don't know how long it takes these things to run, so it is up to you to
>trade off checks vs. speed, and you could even round-robin them (storing
>the last checked item in the superblock or something):
>- check block allocation bitmaps match superblock counts
>- walk directory structure from root, checking for directory corruption
>- check btree validity on inodes for up to 10 seconds (or whatever, storing
>  last checked inode in superblock for restarting this test at next one)
>
>By all means, don't do checks for an hour, or allow users to set the maximum
>boot check duration in the superblock.  I'm sure users don't mind waiting
>5s at boot time if it means they don't lose data.
>
I doubt that there is a lot we can check in 5 seconds on a filesystem 
with lots of small files, but I could be wrong.

>
>Cheers, Andreas
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
>
>  
>


-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Oleg Drokin]

Hello!

On Tue, Feb 18, 2003 at 09:21:54PM +0300, Hans Reiser wrote:

> >By all means, don't do checks for an hour, or allow users to set the 
> >maximum
> >boot check duration in the superblock.  I'm sure users don't mind waiting
> >5s at boot time if it means they don't lose data.
> I doubt that there is a lot we can check in 5 seconds on a filesystem 
> with lots of small files, but I could be wrong.

We discussed this with Vitaly today.
We can do some simply things and if some of these show wrong signs,
we can do a warning and sysadmin can schedule a downtime for
fsck.
Simple things are: read the bitmaps and compare used block counts with
what stored in superblock.
Check consistency of tree root (we have a pointer there in superblock,
also it's supposed level).
May be something else that is that simple.

Bye,
    Oleg

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Oleg Drokin wrote:

>Hello!
>
>On Tue, Feb 18, 2003 at 09:21:54PM +0300, Hans Reiser wrote:
>
>  
>
>>>By all means, don't do checks for an hour, or allow users to set the 
>>>maximum
>>>boot check duration in the superblock.  I'm sure users don't mind waiting
>>>5s at boot time if it means they don't lose data.
>>>      
>>>
>>I doubt that there is a lot we can check in 5 seconds on a filesystem 
>>with lots of small files, but I could be wrong.
>>    
>>
>
>We discussed this with Vitaly today.
>We can do some simply things and if some of these show wrong signs,
>we can do a warning and sysadmin can schedule a downtime for
>fsck.
>Simple things are: read the bitmaps and compare used block counts with
>what stored in superblock.
>Check consistency of tree root (we have a pointer there in superblock,
>also it's supposed level).
>May be something else that is that simple.
>
>Bye,
>    Oleg
>
>
>  
>
Ok.

-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Vitaly Fertman wrote:

>On Saturday 15 February 2003 15:51, Vitaly Fertman wrote:
>  
>
>>>>Ok, so the reiserfs kernel code detects an error on disk, what does it
>>>>do?  Print out an error message, maybe BUG?  There is an "error" field
>>>>in the reiserfs superblock, I hope it is set when the kernel detects
>>>>something bad.
>>>>
>>>>So, now what happens?  Maybe the user doesn't read their syslog and
>>>>doesn't see the error, or the error is just a prelude to memory
>>>>corruption which causes the system to crash.  When the system boots
>>>>again, it goes on its merry way, mounting the reiserfs filesystem with
>>>>_known_ errors on it, using bad allocation bitmaps, directories btrees,
>>>>etc and maybe double allocating blocks or overwriting blocks from other
>>>>files causing them to become corrupt, etc, etc, etc.  Until finally the
>>>>filesystem is totally corrupt, the system crashes miserably, the user
>>>>emails this list and reiserfsck has an impossible job trying to fix the
>>>>filesystem.
>>>>
>>>>Instead, what I propose is to have "reiserfsck -a" AS A STARTING POINT
>>>>simply check for a valid reiserfs superblock and the absence of the
>>>>"error" flag before declaring the filesystem clean and allowing the
>>>>system to boot.
>>>>
>>>>What's even worse, the reiserfs_read_super (at least 2.4.18 RH kernel)
>>>>code OVERWRITES the superblock error status at mount time, making it
>>>>worse than useless, since each mount hides any errors that were detected
>>>>before the crash:
>>>>
>>>>	s->u.reiserfs_sb.s_mount_state = SB_REISERFS_STATE(s);
>>>>	s->u.reiserfs_sb.s_mount_state = REISERFS_VALID_FS ;
>>>>        
>>>>
>>>Andreas seems reasonable, Vitaly, what are your thoughts?
>>>
>>>      
>>>
>>>>>>Next, add journal replay to reiserfsck if it isn't already there,
>>>>>>            
>>>>>>
>>>>>Why, when it is in the kernel?
>>>>>          
>>>>>
>>>>Because that is the next stage to allowing reiserfsck do checks on the
>>>>filesystem after a crash.  Do you tell me you would rather (and you
>>>>must, because it obviously currently does) have reiserfsck just throw
>>>>away everything in the journal, leaving possibly inconsistent data in
>>>>the filesystem for it to check?  Or maybe make the user mount the
>>>>filesystem (which obviously has problems or they wouldn't be running
>>>>reiserfsck to do a full check) just to clear out the journal and maybe
>>>>risk crashing or corruption if the filesystem is strangely corrupted?
>>>>        
>>>>
>>>Vitaly, answer this.
>>>      
>>>
>>Ok, so probably we should make the following changes. The kernel set
>>IO_ERROR and FS_ERROR flags.
>>In the case of IO_ERROR reiserfsck prints the message about hardware
>>problems and returns error, so the fs does not get mounted at boot. On
>>attempt mounting the fs with IO_ERROR flag set it is mounted ro with some
>>message about hardware problems. When you are sure that problems
>>disappeared you can mount it with a spetial option cleaning this flag and
>>probably reiserfstune will have some option cleaning these flags also.
>>In the case of FS_ERROR - search_by_key failed or beyond end of device
>>access or similar - reiserfsck gets -a option at boot, replays the journal
>>if needed and checks for the flag. No flag - returns OK. Else - run
>>fix-fixable. Errors left - returns 'errors left uncorrected' and the fs
>>does not get mounted at boot. On attempt mounting the fs with the flag just
>>print the message about mounting the fs with errors and mount it. Not ro
>>here as kernel will not do deep analysis of errors and it could be just a
>>small insignificant error.
>>    
>>
>
>If fix-fixable finds fatal corruptions it sets FS_FATAL what means that the
>second run of fix-fixable will not help to avoid another fsck run at the next 
>boot.
>
>  
>
Can you repeat what you said (more clearly).

-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Vitaly Fertman]

> >If fix-fixable finds fatal corruptions it sets FS_FATAL what means that
> > the second run of fix-fixable will not help to avoid another fsck run at
> > the next boot.
>
> Can you repeat what you said (more clearly).

If fs was not fixed, you do not want to have another run of fsck at 
the next boot. you already know that there are fatal corruptions and
you need to run rebuild-tree manually. So a special flag saying that 
would be helpful.

-- 

Thanks,
Vitaly Fertman

Re: Corrupted/unreadable journal: reiser vs. ext3

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

On Sat, 15 Feb 2003 15:37:10 MST, Andreas Dilger said:

> I guess the shortcoming in the ext2 case is that it counts mounts and
> not crashes.  If it were counting the number of times the filesystem
> was uncleanly shut down instead of normal shutdowns, would that be more
> acceptable?  The reason I'm still interested in crashes, even if they
> are not filesystem-related crashes, is because there had to be _something_
> which caused a crash (bad code, bad hardware, whatever), and once you have
> any driver corrupting memory the chance that it is also corrupting filesystem
> memory exists.

ext2/3 *intentionally* counts mounts rather than crashes.

It's possible for a filesystem to get non-noticably corrupted without
a crash (remember the Linux 2.4.11 mangle-on-shutdown bug?)  - it's stuff
like that (and slowly failing media) that it tries to catch by counting
mounts.
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Corrupted/unreadable journal: reiser vs. ext3

[Matthias Andree]

Valdis.Kletnieks@vt.edu writes:

> ext2/3 *intentionally* counts mounts rather than crashes.
>
> It's possible for a filesystem to get non-noticably corrupted without
> a crash (remember the Linux 2.4.11 mangle-on-shutdown bug?)  - it's stuff
> like that (and slowly failing media) that it tries to catch by counting
> mounts.

Leaving that aside, reordered writes (write cache, enabled by default on
most drives) is one of the other promiment reasons for creeping
doom... the write barrier code hasn't yet been synched into 2.4 main
stream and I doubt it ever will. It might make 2.6 though.

-- 
Matthias Andree

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Vitaly Fertman wrote:

>>>If fix-fixable finds fatal corruptions it sets FS_FATAL what means that
>>>the second run of fix-fixable will not help to avoid another fsck run at
>>>the next boot.
>>>      
>>>
>>Can you repeat what you said (more clearly).
>>    
>>
>
>If fs was not fixed, you do not want to have another run of fsck at 
>the next boot. you already know that there are fatal corruptions and
>you need to run rebuild-tree manually. So a special flag saying that 
>would be helpful.
>
>  
>
ok, along with special messages clearly explaining it to the user at 
each boot, yes?

-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Hans Reiser]

Valdis.Kletnieks@vt.edu wrote:

>On Sat, 15 Feb 2003 15:37:10 MST, Andreas Dilger said:
>
>  
>
>>I guess the shortcoming in the ext2 case is that it counts mounts and
>>not crashes.  If it were counting the number of times the filesystem
>>was uncleanly shut down instead of normal shutdowns, would that be more
>>acceptable?  The reason I'm still interested in crashes, even if they
>>are not filesystem-related crashes, is because there had to be _something_
>>which caused a crash (bad code, bad hardware, whatever), and once you have
>>any driver corrupting memory the chance that it is also corrupting filesystem
>>memory exists.
>>    
>>
>
>ext2/3 *intentionally* counts mounts rather than crashes.
>
>It's possible for a filesystem to get non-noticably corrupted without
>a crash (remember the Linux 2.4.11 mangle-on-shutdown bug?)  - it's stuff
>like that (and slowly failing media) that it tries to catch by counting
>mounts.
>  
>
Making users who often reboot to windows run fsck more often because of 
it is not very clever, in fact, it is just annoying, and I remember it 
well from my old ext2 days.

-- 
Hans

Re: Corrupted/unreadable journal: reiser vs. ext3

[Oleg Drokin]

Hello!

On Tue, Feb 18, 2003 at 11:02:42PM +0100, Matthias Andree wrote:

> > ext2/3 *intentionally* counts mounts rather than crashes.
> > It's possible for a filesystem to get non-noticably corrupted without
> > a crash (remember the Linux 2.4.11 mangle-on-shutdown bug?)  - it's stuff
> > like that (and slowly failing media) that it tries to catch by counting
> > mounts.
> Leaving that aside, reordered writes (write cache, enabled by default on
> most drives) is one of the other promiment reasons for creeping
> doom... the write barrier code hasn't yet been synched into 2.4 main
> stream and I doubt it ever will. It might make 2.6 though.

I saw Jens have submitted his code for review on lkml once again. So
I hope there is still a hope ;)

Bye,
    Oleg

Re: Error <-> Partition Correspondance [was Re: Corrupted/unreadable journal: reiser vs. ext3]

[Manuel Krause]

On 02/18/2003 07:54 AM, Oleg Drokin wrote:
> Hello!
> 
> On Tue, Feb 18, 2003 at 12:35:23AM +0100, Manuel Krause wrote:
> 
> 
>>BTW, do the ReiserFS errors nowadays print out a usable partition 
>>identification (like Chris actual data-logging patches perform at mount, 
>>e.g.)?
> 
> 
> Sometimes it does.
> 
> 
>>I mostly always have 2 partitions with ReiserFS mounted, so -- is it 
>>still meaningless to get an error message related to one of them in my logs?
> 
> 
> It depends on what are the messages.
> 
> 
>>I posted this circumstance some 3.6-ReiserFS levels ago and someone of 
>>your team wanted to implement this after his task-list was done, IIRC.
> 
> 
> Yes. I have a patch dated back to May 7th, 2002. But it was never
> accepted for reason I don't remember already.
> I will dig through my email, though. Probably I will give it another try.

Yes. I have the origin dated back to exactly that time, too. :-))

If you're ready with the patch (again) I would be glad to receive the 
preliminary version to use it anyways.


Thanks for your great work
(applies for all the teams, of course!),

Manuel

Re: reiserfs messages cleanup patch.

[Manuel Krause]

On 02/21/2003 08:37 AM, Oleg Drokin wrote:
> Hello!
> 
> On Fri, Feb 21, 2003 at 08:27:32AM +0100, Manuel Krause wrote:
> 
> 
>>If you're ready with the patch (again) I would be glad to receive the 
>>preliminary version to use it anyways.
> 
> 
> Our tester is just busy with other stuff, so I probably release
> the patch just for outside testing. If anyone would like to, of course.
> (NOTE, THIS WAS ONLY LIGHTLY TESTED BY ME).
> Patch is against latest 2.4 bitkeeper snapshot.
> Probably should apply fine to 2.4.21-pre4
> 
> Bye,
>     Oleg
> 

Thank you for the quick thing!

It doesn't apply to my kernel setup [-pre4 + data-logging + preempt]
-- too many hunks failing in my eyes.

BTW, the directio-fix fails on the same setup, too [alternatively].

Maybe I've time over the weekend to have a closer look.


Many thanks to you anyways and best regards,

Manuel


 > [PATCH: massage-cleanup.diff]

Re: reiserfs messages cleanup patch.

[Oleg Drokin]

Hello!

On Fri, Feb 21, 2003 at 09:22:16AM +0100, Manuel Krause wrote:

> It doesn't apply to my kernel setup [-pre4 + data-logging + preempt]
> -- too many hunks failing in my eyes.

datalogging is just too big of a change.

Bye,
    Oleg