New concept of ext3 disk checks

New concept of ext3 disk checks

[Otto Wyss]

I've just read about "Increasing ext3 Disk Checks In The Event Of
Improper Shutdown" in Kernel-Traffic. Since in theory journalling file
systems may prevent any disk corruption the disk checks from the ext2
area doesn't make much sense anymore. Also more and more system don't
boot regularly anymore so the time frame between check isn't
predictable. To take into account that the practice not always behaves
as the theory says, I suggest another concept for ext3 disk checks:

- Instead of checks forced during startup checks are done during runtime
(at low priority). It has to be determined if these checks are _only_
checks or if they also include possible fixes. Possible solution might
distinct between the severity of any discovered problem.

The advantage of such a concept are rather obvious, desktop systems
don't have to use ridiculous high check interval values or disable
checks altogether and server systems may run forever. Also checks may be
done first on the written disk sectors. On an average loaded system,
this way malfunctioning software are detected within minutes and
hardware possibly within days, a rather high improvement to the current
detection time of sometimes months.

O. Wyss

-- 
See a huge pile of work at "http://wyodesktop.sourceforge.net/"

Re: New concept of ext3 disk checks

[Bernd Eckenfels]

In article <411BAFCA.92217D16@orpatec.ch> you wrote:
> - Instead of checks forced during startup checks are done during runtime
> (at low priority). It has to be determined if these checks are _only_
> checks or if they also include possible fixes. Possible solution might
> distinct between the severity of any discovered problem.

BTW: FreeBSD5 supports the fsck of a unclean filesystem in the background.
(Actually it supports to mount unclean filesystems because they are "always"
consistent and do reclaiming of unreferenced objects in a fsck in background
based on a snapshot)

I am not sure why the softupdates are so reliable, that no fsck is needed,
but I know it is pretty cool to do it in background only, especially for
notebooks. And I agree that background checks of long running systems may
catch problems - however I am not sure how a consitent FS view is best
created for that.

Greetings
Bernd
-- 
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Re: New concept of ext3 disk checks

[Lee Revell]

On Thu, 2004-08-12 at 14:58, Bernd Eckenfels wrote:

> I am not sure why the softupdates are so reliable, that no fsck is needed

Softupdates were introduced to enhance performance by batching and
deferring disk operations.  I belive that initially, performance was the
only consideration in determining what actually got written to disk
when.  All the BSD/OS manuals I read at my last job warn against
enabling softupdates because data loss could occur.

Someone later realized that if you batch the disk operations in a way
that considers the atomicity of the filesystem operations being
implemented, softupdates can help *ensure* data integrity.  This is a
pretty recent development I think.

Lee

Re: New concept of ext3 disk checks

[Alan Cox]

On Iau, 2004-08-12 at 18:58, Otto Wyss wrote:
> The advantage of such a concept are rather obvious, desktop systems
> don't have to use ridiculous high check interval values or disable
> checks altogether and server systems may run forever. Also checks may be
> done first on the written disk sectors. On an average loaded system,
> this way malfunctioning software are detected within minutes and
> hardware possibly within days, a rather high improvement to the current
> detection time of sometimes months.

I think it would indeed be a good project. If anyone has patches please
send them along to the linux-fsdevel list

Re: New concept of ext3 disk checks

[Otto Wyss]

Alan Cox wrote:
> 
> I think it would indeed be a good project. If anyone has patches please
> send them along to the linux-fsdevel list

Unfortunately it won't be me, while I'm able to formulate a concept my
knowledge of Linux is too limited to do much work. Also I've lot of
other work as you might have seen in my signature. But I encurrage
anyone with more insight of Linux file systems to have a look if such a
concept can be implemented.

O. Wyss

-- 
See a huge pile of work at "http://wyodesktop.sourceforge.net/"

Re: New concept of ext3 disk checks

[Theodore Ts'o]

On Thu, Aug 12, 2004 at 07:58:33PM +0200, Otto Wyss wrote:
> - Instead of checks forced during startup checks are done during runtime
> (at low priority). It has to be determined if these checks are _only_
> checks or if they also include possible fixes. Possible solution might
> distinct between the severity of any discovered problem.

This is something that doesn't require any kernel patches, or any
other C coding for that matter, so it would be a great first project
for someone who wanted to learn how to use the device-mapper snapshot
feature.  Basically, what you do is the following in a shell script
which is fired off by cron once a week at 3am (or some other
appropriate time):

1) Create a clean, read-only snapshot of an ext3 filesystem using
device mapper.

2) Run e2fsck -f on the snapshot, and check to see if there are any
error on the filesystem.  Assuming a non-buggy kernel and properly
functioning hardware, there should be none.  Afterwards, release the
read-only snapshot.

3) If there are any errors, e-mail the output of e2fsck to the system
administrator.

4) If there were no errors detecting by the fsck run, run the command
"tune2fs -C 0 -T now /dev/XXX" on the live filesystem.  This sets the
mount count and last filesystem checked time to the appropriate values
in the superblock.

Tell you what --- if someone is willing to put the time into
developing such a script, I'll include it in the contrib section of
e2fsprogs.  I've put all the hooks to do this in e2fsprogs, and I've
wanted this for quite some time, but the last time I looked at it, the
command-line EVMS tools were truly gruesome to behold/use.  I believe
things have gotten much better since then, so this shouldn't be too
hard to do now.

					- Ted

Re: New concept of ext3 disk checks

[Bernd Eckenfels]

In article <20040812223907.GA7720@thunk.org> you wrote:
> 1) Create a clean, read-only snapshot of an ext3 filesystem using
> device mapper.

Speaking of clean, is there something like the XFS freeze for ext? I know
freezing XFS is pretty dangerous (swap, temp, root) so I think it is not
useable on all devices anyway.

> Tell you what --- if someone is willing to put the time into
> developing such a script, I'll include it in the contrib section of
> e2fsprogs.

I did that for XFS some time ago, but the parameters where all hardcoded. I
used the freeze command. Do you think one can skip that for ext3?

Greetings
Bernd
-- 
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Re: New concept of ext3 disk checks

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

On Aug 12, 2004  18:39 -0400, Theodore Ts'o wrote:
> 2) Run e2fsck -f on the snapshot, and check to see if there are any
> error on the filesystem.  Assuming a non-buggy kernel and properly
> functioning hardware, there should be none.  Afterwards, release the
> read-only snapshot.
> 
> 3) If there are any errors, e-mail the output of e2fsck to the system
> administrator.
> 
> 4) If there were no errors detecting by the fsck run, run the command
> "tune2fs -C 0 -T now /dev/XXX" on the live filesystem.  This sets the
> mount count and last filesystem checked time to the appropriate values
> in the superblock.
> 
> Tell you what --- if someone is willing to put the time into
> developing such a script, I'll include it in the contrib section of
> e2fsprogs.  I've put all the hooks to do this in e2fsprogs, and I've
> wanted this for quite some time, but the last time I looked at it, the
> command-line EVMS tools were truly gruesome to behold/use.  I believe
> things have gotten much better since then, so this shouldn't be too
> hard to do now.

I did this a few years ago, but it fell through the cracks.  The
script is written for the LVM1 code, but AFAIK the LVM2 code also
has compatible user tools.

http://www.mail-archive.com/reiserfs-list@namesys.com/msg08474.html

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: New concept of ext3 disk checks

[Otto Wyss]

On Aug 12, 2004  18:39 -0400, Theodore Ts'o wrote:
> > - Instead of checks forced during startup checks are done during runtime
> > (at low priority). It has to be determined if these checks are _only_
> > checks or if they also include possible fixes. Possible solution might
> > distinct between the severity of any discovered problem.
>
> This is something that doesn't require any kernel patches, or any
> other C coding for that matter, so it would be a great first project
> for someone who wanted to learn how to use the device-mapper snapshot
> feature. Basically, what you do is the following in a shell script
> which is fired off by cron once a week at 3am (or some other
> appropriate time):

Instead of a daily cron job I envision a solution where writes to the
disk are checked for correctness within a short time lag after they have
been done. Assume this time lag is set to a few minutes, on a high
performance system not each write of a certain node gets checked while
on a desktop system most probably each single write is. Choosing the
right time lag gives a balance of discovering problems fast against
additional disk access.

Okay, such tests could be done by a constantly running background task
in user space. But since journalling just should guarantee that any disk
access is done correct, even in case of problems, it should be
considered if such test can be integrated there. This has the advantage
that if journalling is able to guarantee correctness by other means
these test aren't needed at all and may be completely remove.

What I want to achieve with this new concept is that the file system
itself not only tries to prevent any data corruption but also tries to
detect and report it if corruption still has happened anyway. 

O. Wyss

-- 
See a huge pile of work at "http://wyodesktop.sourceforge.net/"

Re: New concept of ext3 disk checks

[Andreas Dilger]

On Aug 14, 2004  09:15 +0200, Otto Wyss wrote:
> Instead of a daily cron job I envision a solution where writes to the
> disk are checked for correctness within a short time lag after they have
> been done. Assume this time lag is set to a few minutes, on a high
> performance system not each write of a certain node gets checked while
> on a desktop system most probably each single write is. Choosing the
> right time lag gives a balance of discovering problems fast against
> additional disk access.
> 
> Okay, such tests could be done by a constantly running background task
> in user space. But since journalling just should guarantee that any disk
> access is done correct, even in case of problems, it should be
> considered if such test can be integrated there. This has the advantage
> that if journalling is able to guarantee correctness by other means
> these test aren't needed at all and may be completely remove.
> 
> What I want to achieve with this new concept is that the file system
> itself not only tries to prevent any data corruption but also tries to
> detect and report it if corruption still has happened anyway. 

The ext3 (and ext2) kernel code already does consistency checking of a
lot of data structures in the kernel, and if any errors are found the
superblock is marked with an "error" flag and the next time the system
is booted a full fsck is done.  The administrator has the option (via
the "errors=" mount option) to either panic the kernel, remount read-only,
or continue using the filesystem if an error is found.

The problem with re-reading blocks and checking for validity after a write
is that there is a good chance the block is still in cache (either kernel
buffer/page cache or disk cache) so this doesn't really add much robustness.
The other problem with this is that checking individual block validity
doesn't take the "big picture" into account since if we just wrote a block
to disk we assume that what we wrote is the correct thing so re-checking
this same data doesn't help much.

The periodic fsck of a filesystem snapshot, on the other hand, is as good 
as you can get for validity checking.  The only additional feature needed
is online repair, but that is a .00001% requirement vs. actually detecting
the error in the first place.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/

Re: New concept of ext3 disk checks

[Andi Kleen]

Theodore Ts'o <tytso@mit.edu> writes:

> 4) If there were no errors detecting by the fsck run, run the command
> "tune2fs -C 0 -T now /dev/XXX" on the live filesystem.  This sets the
> mount count and last filesystem checked time to the appropriate values
> in the superblock.

Is it safe now to run tune2fs on a mounted busy fs? afaik it would
need at least support to quiescence the fs temporarily. Otherwise you 
have a race window where changes to the superblock could get lost.

-Andi

Re: New concept of ext3 disk checks

[Theodore Ts'o]

On Fri, Aug 13, 2004 at 01:24:33AM +0200, Andi Kleen wrote:
> Theodore Ts'o <tytso@mit.edu> writes:
> 
> > 4) If there were no errors detecting by the fsck run, run the command
> > "tune2fs -C 0 -T now /dev/XXX" on the live filesystem.  This sets the
> > mount count and last filesystem checked time to the appropriate values
> > in the superblock.
> 
> Is it safe now to run tune2fs on a mounted busy fs? afaik it would
> need at least support to quiescence the fs temporarily. Otherwise you 
> have a race window where changes to the superblock could get lost.

Modern versions of e2fsprogs do byte-level writes only to the fields
that are being changed by the userspace program; given that these are
fields that are not touched by the kernel once the filesystem is
mounted, this is safe.

						- Ted