Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Roy Butler]

On Tue, Jul 13, 2004 at 02:31:43PM +0200, Waldo Bastian wrote:
 >
 > The sentiment among filesystem developers seem to be that they don't 
care if
 > they trash files as long as the filesystem itself remains in a consistent
 > state. This kind of dataloss is the result of that attitude, either go
 > complain with them if it bothers you, or use a filesystem that does 
it right.
 >

Exactly.  Don't blame KDE.  Using XFS is equivalent to using 
non-battery-backed NVRAM on an external disk array.  Great if 
performance is _the_ metric and lost results can easily be regenerated 
(like in frame rendering).  By example, if you create a file, write to 
it, and then delete it fast enough, it will never hit the disk under XFS.


Roy Butler

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Chris Wedgwood]

On Tue, Jul 13, 2004 at 11:27:50PM -0700, Roy Butler wrote:

> By example, if you create a file, write to it, and then delete it
> fast enough, it will never hit the disk under XFS.

Nor will it under some other filesystems...  and in the above scenario
I'm not sure that matters, why must a temporary file hit the disk at
all?


   --cw

Re: kconfig's file handling

[Roy Butler]

Chris,

Chris Wedgwood wrote:
> On Tue, Jul 13, 2004 at 11:27:50PM -0700, Roy Butler wrote:
> 
> 
>>By example, if you create a file, write to it, and then delete it
>>fast enough, it will never hit the disk under XFS.
> 
> 
> Nor will it under some other filesystems...  and in the above scenario
> I'm not sure that matters, why must a temporary file hit the disk at
> all?
> 
> 
>    --cw
> 

Suppose you want to do something with the file's contents before you 
delete it, but the system goes down before you have the chance.  Wasn't 
that the impetus for this thread?  With XFS, the potential for data loss 
is greater; that was my point.  I believe that everyone who pays extra 
for batter-backed I/O caches would agree. :)  Sure, different 
filesystems have their options, but this liability is ingrained in XFS.


Roy

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Waldo Bastian]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue July 13 2004 13:05, Oswald Buddenhagen wrote:
> heya,
>
> read the attachement first (put on your asbestos underwear first ;).
> not exactly news to me, but somehow i never got to fixing it ...

There is nothing to fix, we already use a tempfile + rename, it's in KSaveFile 
since 1999. Or just look with strace if you don't believe me. This Tim 
Connors guy shouldn't talk about things he obviously knows nothing about.

As far as I can see the problem is that the filesystem writes out the meta 
data before the actual file data hits the disk which creates a period of time 
in which the on-disk state of the filesystem contains trashed files. I 
believe ReiserFS actually has an option to do things in a sane order so that 
it doesn't trash recently used files on an unclean shutdown.

The sentiment among filesystem developers seem to be that they don't care if 
they trash files as long as the filesystem itself remains in a consistent 
state. This kind of dataloss is the result of that attitude, either go 
complain with them if it bothers you, or use a filesystem that does it right.

Cheers,
Waldo
- -- 
bastian@kde.org  |   KDE Community World Summit 2004  |  bastian@suse.com
bastian@kde.org  | 21-29 August, Ludwigsburg, Germany |  bastian@suse.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQFA89YvN4pvrENfboIRAptaAJ9YBv63UUL7Elcl3QsjYJPbGche7wCdEzKr
Y+I4Kyi+p+r/gPixNdXQphE=
=c3HD
-----END PGP SIGNATURE-----

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Tim Connors]

On Tue, 13 Jul 2004, Waldo Bastian wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Tue July 13 2004 13:05, Oswald Buddenhagen wrote:
> > heya,
> >
> > read the attachement first (put on your asbestos underwear first ;).
> > not exactly news to me, but somehow i never got to fixing it ...
>
> There is nothing to fix, we already use a tempfile + rename, it's in KSaveFile
> since 1999. Or just look with strace if you don't believe me. This Tim
> Connors guy shouldn't talk about things he obviously knows nothing about.

Sorry, I'm just basing the above on my *experience* with trying to unfsck
3 different peoples kde settings when the nfs server went down.

And since it was a controlled shutdown, there would have been no cache
worries, and the fact the kde corrupted itself points to having problems
within kde, not problems within a fs or nfs.

> As far as I can see the problem is that the filesystem writes out the meta
> data before the actual file data hits the disk which creates a period of time
> in which the on-disk state of the filesystem contains trashed files. I
> believe ReiserFS actually has an option to do things in a sane order so that
> it doesn't trash recently used files on an unclean shutdown.

Well, no other program I have ever used, which does the tempfile+rename
thing, has ever failed on me.

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
                  -o)
                  /\\    The penguins are coming...
                 _\_v       the penguins are coming...

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Oswald Buddenhagen]

On Tue, Jul 13, 2004 at 02:31:43PM +0200, Waldo Bastian wrote:
> There is nothing to fix, we already use a tempfile + rename, it's in
> KSaveFile since 1999.
>
oh, indeed. i didn't dig deep enough. sorry for the noise.
the symlink case is handled differently, though (and it falls back to
in-place rewriting too early - the symlink target could be potentially
handled without in-place rewriting as well). consequently the paragraph
about in-place rewriting is valid, even if it is only a border case.

another thing to consider is a complete audit of kde regarding the
creation of files; i'm sure not everybody uses KSaveFile or equivalent
code. no, i don't volunteer. :}

> As far as I can see the problem is that the filesystem writes out the
> meta data before the actual file data hits the disk which creates a
> period of time in which the on-disk state of the filesystem contains
> trashed files.
>
yup

> I believe ReiserFS actually has an option to do things in a sane order
> so that it doesn't trash recently used files on an unclean shutdown.
> 
not only reiserfs.

> The sentiment among filesystem developers seem to be that they don't
> care if they trash files as long as the filesystem itself remains in a
> consistent state. This kind of dataloss is the result of that
> attitude, either go complain with them if it bothers you, or use a
> filesystem that does it right.
> 
the code is there, the additional integrity just doesn't come for free,
so it's disabled by default.

greetings

-- 
Hi! I'm a .signature virus! Copy me into your ~/.signature, please!
--
Chaos, panic, and disorder - my work here is done.

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Theodore Ts'o]

On Tue, Jul 13, 2004 at 02:31:43PM +0200, Waldo Bastian wrote:
> 
> The sentiment among filesystem developers seem to be that they don't care if 
> they trash files as long as the filesystem itself remains in a consistent 
> state. This kind of dataloss is the result of that attitude, either go 
> complain with them if it bothers you, or use a filesystem that does it right.
> 

Ext3 with ordered writes (the default) gets this right. 

Unfortunately, cheating can give you better benchmark resuls, and some
people seem to care more about better benchmark results than silly
things like user's files not getting wiped.  For many workloads,
especially for user desktops, the disk bandwidth isn't saturated, and
given that most writes are asynchronous in nature, a faster write
benchmark for a particular filesystem or filesystme mode may not
translate into a user-visible difference.  So focusing on improved
write speeds at the cost of data robustness can very often be false
economy.

							- Ted

Re: kconfig's file handling (was: XFS: how to NOT null files on fsck?)

[Chris Wedgwood]

On Tue, Jul 13, 2004 at 02:31:43PM +0200, Waldo Bastian wrote:

> There is nothing to fix, we already use a tempfile + rename, it's in
> KSaveFile since 1999. Or just look with strace if you don't believe
> me. This Tim Connors guy shouldn't talk about things he obviously
> knows nothing about.

How about fsync on the tempfile before the rename?  Without getting
into a religious discussion about whether or not this should be
necessary, it will certainly help in many cases.

> This kind of dataloss is the result of that attitude, either go
> complain with them if it bothers you, or use a filesystem that does
> it right.

I'm not sure people can just change their fs, some people have no
options and for some platforms where KDE is used there might not be
any alternatives.  I really think fsync here would help in most if not
all of those cases and it shouldn't adversely affect the performance
(to the best of my knowledge KDE doesn't update dotfiles all that
often and they are pretty small).  Other applications do this already
and it seems to be very reliable for them.



  --cw

P.S. I'm a bit confused as to why files get smashed on ENOSPC and when
     NFS servers croak though.