LWN article: ext4 and data loss

LWN article: ext4 and data loss

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 504 bytes --]

http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

Very good, maybe similar patches for XFS would help?
IANA Coder, but could be a hint.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Eric Sandeen]

Michael Monnerie wrote:
> http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/
> 
> Very good, maybe similar patches for XFS would help?
> IANA Coder, but could be a hint.
> 
> mfg zmi
> 

ext4 is taking its hints from XFS in this regard, not the other way
around.  XFS dealt with this long ago.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Martin Steigerwald]

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
> Michael Monnerie wrote:
> > http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/
> >
> > Very good, maybe similar patches for XFS would help?
> > IANA Coder, but could be a hint.
> >
> > mfg zmi
>
> ext4 is taking its hints from XFS in this regard, not the other way
> around.  XFS dealt with this long ago.

Hmmm, I remember having had similar issues with XFS not to long ago, where 
at least some KDE configuration were lost or truncated. It seems 
applications will have to get rid of behavioral assumptions regation 
filesystem and use safe writing via fsync and whatever else for 
configuration and other important files.

I am thinking about a bug report for KDE at least.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Eric Sandeen]

Martin Steigerwald wrote:
> Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
>> Michael Monnerie wrote:
>>> http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/
>>>
>>> Very good, maybe similar patches for XFS would help?
>>> IANA Coder, but could be a hint.
>>>
>>> mfg zmi
>> ext4 is taking its hints from XFS in this regard, not the other way
>> around.  XFS dealt with this long ago.
> 
> Hmmm, I remember having had similar issues with XFS not to long ago, where 

depends on what you mean by not too long ago, I think.  Yes, kde had
this issue on xfs too, and xfs gave up on teaching apps to fsync, and
implemented the same sorts of things ext4 has done (or will do) to
mitigate this quite some time ago.

> at least some KDE configuration were lost or truncated. It seems 
> applications will have to get rid of behavioral assumptions regation 
> filesystem and use safe writing via fsync and whatever else for 
> configuration and other important files.

It's simple.  Want your data safe on disk?  fsync.  There's not a lot
more to it than that.  (and if fsync hurts perf too much, re-think how
you are storing your data)

Filesystems can hack around some heuristics to try to make unsafe apps
safer, but in the end, it's the app's job to make sure a buffered write
hits permanent storage when it matters.

-Eric

> I am thinking about a bug report for KDE at least.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Greg Banks]

Eric Sandeen wrote:
>
> It's simple.  Want your data safe on disk?  fsync.  There's not a lot
> more to it than that.  (and if fsync hurts perf too much, re-think how
> you are storing your data)
>
> Filesystems can hack around some heuristics to try to make unsafe apps
> safer, but in the end, it's the app's job to make sure a buffered write
> hits permanent storage when it matters.
>   

Stewart Smith has a highly entertaining presentation on this very topic.

http://www.linux.org.au/conf/2007/talk/278.html

-- 
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Eric Sandeen]

Greg Banks wrote:
> Eric Sandeen wrote:
>> It's simple.  Want your data safe on disk?  fsync.  There's not a lot
>> more to it than that.  (and if fsync hurts perf too much, re-think how
>> you are storing your data)
>>
>> Filesystems can hack around some heuristics to try to make unsafe apps
>> safer, but in the end, it's the app's job to make sure a buffered write
>> hits permanent storage when it matters.
>>   
> 
> Stewart Smith has a highly entertaining presentation on this very topic.
> 
> http://www.linux.org.au/conf/2007/talk/278.html
> 

and

http://www.flamingspork.com/talks/2007/06/eat_my_data.odp

In hindsight, http://linuxmafia.com/faq/Filesystems/reiserfs.html is
entertaining, too ;)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Greg Banks]

Eric Sandeen wrote:
>
> In hindsight, http://linuxmafia.com/faq/Filesystems/reiserfs.html is
> entertaining, too ;)
>
>   
There was a fascinating internal thread about that one when somebody
noticed it a few years back :-)

-- 
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Martin Steigerwald]

Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
> Martin Steigerwald wrote:
> > Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
> >> Michael Monnerie wrote:
> >>> http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/
> >>>
> >>> Very good, maybe similar patches for XFS would help?
> >>> IANA Coder, but could be a hint.
> >>>
> >>> mfg zmi
> >>
> >> ext4 is taking its hints from XFS in this regard, not the other way
> >> around.  XFS dealt with this long ago.
> >
> > Hmmm, I remember having had similar issues with XFS not to long ago,
> > where
>
> depends on what you mean by not too long ago, I think.  Yes, kde had
> this issue on xfs too, and xfs gave up on teaching apps to fsync, and
> implemented the same sorts of things ext4 has done (or will do) to
> mitigate this quite some time ago.

Well 2.6.28 and 2.6.27.7. See

http://oss.sgi.com/archives/xfs/2008-12/msg00540.html

> > at least some KDE configuration were lost or truncated. It seems
> > applications will have to get rid of behavioral assumptions regation
> > filesystem and use safe writing via fsync and whatever else for
> > configuration and other important files.
>
> It's simple.  Want your data safe on disk?  fsync.  There's not a lot
> more to it than that.  (and if fsync hurts perf too much, re-think how
> you are storing your data)
>
> Filesystems can hack around some heuristics to try to make unsafe apps
> safer, but in the end, it's the app's job to make sure a buffered write
> hits permanent storage when it matters.

Hmmm, okay. So here is:

http://bugs.kde.org/187172

Feel free to add there. You'd need a bugzilla login tough.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Eric Sandeen]

Martin Steigerwald wrote:
> Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
...

>> Filesystems can hack around some heuristics to try to make unsafe apps
>> safer, but in the end, it's the app's job to make sure a buffered write
>> hits permanent storage when it matters.
> 
> Hmmm, okay. So here is:
> 
> http://bugs.kde.org/187172
> 
> Feel free to add there. You'd need a bugzilla login tough.
> 
> Ciao,

Perhaps you can try this and add info if you like.

There is an environment variable KDE_EXTRA_FSYNC, put in with this
changelog:

Do not paranoidly sync every time, it causes I/O performance problems
for some users. People who still want it for whatever reason like using
XFS can set $KDE_EXTRA_FSYNC to 1."

This is not "extra" - it is necessary if you actually want the data to
be on-disk.

See also: http://lists.kde.org/?l=kde-devel&m=120880682813170&w=2

(note however that xfs is _not_ "zeroing just to be sure...")

http://api.kde.org/4.x-api/kdelibs-apidocs/kde3support/html/k3tempfile_8cpp-source.html

 bool
 K3TempFile::sync()
    int result = 0;

    if (mStream)
    {
       do {
          result = fflush(mStream); // We need to flush first otherwise
fsync may not have our data
       }
       while ((result == -1) && (errno == EINTR));

       if (result)
       {
          kWarning() << "K3TempFile: Error trying to flush " << mTmpName
 " << strerror(errno);
          mError = errno;
       }
    }

    if (mFd >= 0)
    {
       if( qgetenv( "KDE_EXTRA_FSYNC" ) == "1" )
       {
          result = FDATASYNC(mFd);
          if (result)
          {
             kWarning() << "K3TempFile: Error trying to sync " <<
mTmpName << ": " << strerror(errno);
             mError = errno;
          }
       }
    }

    return (mError == 0);
 }

So somebody knew it was the right thing to do, but then turned it off,
probably because of ext3's painful behavior on fsync (see also: firefox
"bug" from a while ago)

what's interesting is that the sync() method isn't automatically called
from the other methods, near as I can tell, so it's up to the api user
to invoke it; and even then it only does fflush() not fsync() by
default, which doesn't actually push things to disk.

I'd suggest turning on this KDE_EXTRA_FSYNC and see how things go from
there on.

Although, the API refs say that this is deprecated in favor of
KTemporaryFile, and I find no reference whatsoever to "sync" of any kind
in ktemporaryfile.cpp.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Eric Sandeen]

Eric Sandeen wrote:
> Martin Steigerwald wrote:
>> Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
> ...
> 
>>> Filesystems can hack around some heuristics to try to make unsafe apps
>>> safer, but in the end, it's the app's job to make sure a buffered write
>>> hits permanent storage when it matters.
>> Hmmm, okay. So here is:
>>
>> http://bugs.kde.org/187172
>>
>> Feel free to add there. You'd need a bugzilla login tough.

... getting OT sorry but in case anyone else is interested,
http://api.kde.org/4.x-api/kdelibs-apidocs/kdecore/html/classKSaveFile.html
also seems relevant.

I'd like to know just what KDE is doing here, so digging a little.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Martin Steigerwald]

Am Samstag 14 März 2009 schrieben Sie:
> Martin Steigerwald wrote:
> > Am Donnerstag 12 März 2009 schrieb Eric Sandeen:
>
> ...
>
> >> Filesystems can hack around some heuristics to try to make unsafe
> >> apps safer, but in the end, it's the app's job to make sure a
> >> buffered write hits permanent storage when it matters.
> >
> > Hmmm, okay. So here is:
> >
> > http://bugs.kde.org/187172
> >
> > Feel free to add there. You'd need a bugzilla login tough.
> >
> > Ciao,
>
> Perhaps you can try this and add info if you like.
>
> There is an environment variable KDE_EXTRA_FSYNC, put in with this
> changelog:

[...]

Added this and your other post and added

export KDE_EXTRA_FSYNC=1

to my .zshrc. Just logged out and on and do not notice a difference in 
speed. At least no abysmal performance degradation.

Would need to switch of the machine hardly to see if it helps. Not today 
anymore tough.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Peter Grandi]

[ ... usual misunderstanding about caching and transactions ... ]

>>>> ext4 is taking its hints from XFS in this regard, not the
>>>> other way around.  XFS dealt with this long ago.

>>> Hmmm, I remember having had similar issues with XFS not to
>>> long ago,

>> depends on what you mean by not too long ago, I think.  Yes,
>> kde had this issue on xfs too, and xfs gave up on teaching
>> apps to fsync, and implemented the same sorts of things ext4
>> has done (or will do) to mitigate this quite some time ago.

> Well 2.6.28 and 2.6.27.7. See
> http://oss.sgi.com/archives/xfs/2008-12/msg00540.html

>>> [ ... ] applications will have to get rid of behavioral
>>> assumptions regation filesystem and use safe writing via
>>> fsync and whatever else for configuration and other
>>> important files.

>> It's simple.  Want your data safe on disk?  fsync. There's
>> not a lot more to it than that. (and if fsync hurts perf too
>> much, re-think how you are storing your data)

>> Filesystems can hack around some heuristics to try to make
>> unsafe apps safer, but in the end, it's the app's job to make
>> sure a buffered write hits permanent storage when it matters.

This discussion is partially misguided, but then how many people
study storage system semantics...

The goal is to do atomic transactions: within a transaction
there are no guarantees, but at the end of transaction things
get stored permanently.

Unfortunately as described 'ext3' has historically done
''rolling'' auto-saving, so many people and application
developers have not appreciated the need for transaction
semantics (common attitude, for example how many programmers for
example check the return code of 'close'?).

Now under Linux and POSIX it is essentially impossible to do
atomic, persistent transactions, because:

* 'fsync' does NOT guarantee persistency. Only that *RAM*
  buffers are flushed; therefore host adapter and disk buffers
  are not required to be flushed.

* Linux write barriers also only guaranteeq ordering and not
  persistence, and there is a number of misguided people who
  think that this is how things should be.

> Hmmm, okay. So here is:
> http://bugs.kde.org/187172

In practice, for systems without caching host adapters, and with
'ext3', most of the time informal ''rolling'' transactions every
5s fool most people/work as if they were right, and as asserted
this has lulled developers into thinking that transactions don't
matter. Too bad this kills performance and/or reliability on
anything else.

This is just another example of how much userspace sucks

  http://lwn.net/Articles/192214/
  http://kernelslacker.livejournal.com/81262.html
  http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/38.pdf

Note that in a proper design where 'fsync' would guarantee
persistence, like in every transactional systems, lots of small
transactions have very sharp performance implications. People
who earn a living doing transactional systems therefore spend a
great deal of money and effort designing them to perform well
despite lots of small transactions, with 15k drives, vast
parallel RAID, bettery backed logs, etc.

You cannot have all of these:

* Reliable transactions.
* Fast with lots of small transactions.
* With cheap hardware.

In the end one must decided whether to follow the Microsoft
strategy (f*ck doing the right thing, cultivate bugs that users
are relying on) or the UNIX one (try to do the right thing).

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Martin Steigerwald]

Am Donnerstag 12 März 2009 schrieb Michael Monnerie:
> http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/
>
> Very good, maybe similar patches for XFS would help?
> IANA Coder, but could be a hint.

BTW I think you gave away a subcriber only link. The article is not 
officially available yet:

http://lwn.net/Articles/322823/

LWN might not be too happy with this.

Information about this is available elsewhere like on Heise:

http://www.heise.de/english/newsticker/news/134483

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Michael Monnerie]

On Samstag 14 März 2009 Martin Steigerwald wrote:
> Am Donnerstag 12 März 2009 schrieb Michael Monnerie:
> > http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/

> BTW I think you gave away a subcriber only link. The article is not
> officially available yet:
> http://lwn.net/Articles/322823/
> LWN might not be too happy with this.

Nope. I am a subscriber, and can generate such Links in order to 
provider an article to others. If you read that, they offer a "Free 
trial subscription" for 3 months, and encourage you to pay then. I can 
recommend lwn.net a lot, so maybe some people subscribe there because of 
my posting :-)

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: LWN article: ext4 and data loss

[Martin Steigerwald]

Am Montag 16 März 2009 schrieb Michael Monnerie:
> On Samstag 14 März 2009 Martin Steigerwald wrote:
> > Am Donnerstag 12 März 2009 schrieb Michael Monnerie:
> > > http://lwn.net/SubscriberLink/322823/e6979f02e5a73feb/
> >
> > BTW I think you gave away a subcriber only link. The article is not
> > officially available yet:
> > http://lwn.net/Articles/322823/
> > LWN might not be too happy with this.
>
> Nope. I am a subscriber, and can generate such Links in order to
> provider an article to others. If you read that, they offer a "Free
> trial subscription" for 3 months, and encourage you to pay then. I can
> recommend lwn.net a lot, so maybe some people subscribe there because
> of my posting :-)

Okay, nice to hear.

I like LWN a lot and consider subscribing to it.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs