Ext2, fsync() and MTA's?

Ext2, fsync() and MTA's?

[Andrew McNamara]

Is the following still true in 2.4 (fsync() doesn't ensure directory
and inode data is up to date)? I had a feeling this had been fixed.

Under Linux, the Postfix MTA sets "chattr +S" on it's spool directories
- obviously this hurts it's performance badly (compared to the BSD's).
It would be really nice to be able to say it's no longer necessary. It
wants to know that a file (file data, inode and directory entry) are
commited to stable storage when an fsync returns.

------- Forwarded Message

Date:    Fri, 11 May 2001 19:17:03 -0400
From:    wietse@porcupine.org (Wietse Venema)
To:      Postfix users <postfix-users@postfix.org>
Subject: Re: maildir or not

David W. Chapman Jr.:
> I think that's the point.  Linux's fsync doesn't fsynch like most other
> unices, it still writes async.  There fore its not totally stable because
> you think you are writting sync when in fact you aren't.  I might be wrong,
> but this is what I have been hearing.

Linux fsync flushes the file data blocks.

The debate is about Linux's directory updates.

Postfix can open/write/fsync/close a file successfully, and Linux
can still lose the file because the directory entry was not updated.

This is why Postfix turns on synchronous writes on dircetories.
Unfortunately, that also makes file writes synchronous.  I will
not sacrifice reliability for the sake of performance.

	Wietse

- -
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users


------- End of Forwarded Message

Re: Ext2, fsync() and MTA's?

[Alan Cox]

> Under Linux, the Postfix MTA sets "chattr +S" on it's spool directories
> - obviously this hurts it's performance badly (compared to the BSD's).

Not really. BSD directory updates are always synchronous in the cases postfix
cares about. At least on the old BSD FFS/UFS file systems. Thats the only
reason the postfix stuff doesnt need it on BSD.

> It would be really nice to be able to say it's no longer necessary. It
> wants to know that a file (file data, inode and directory entry) are
> commited to stable storage when an fsync returns.

fsync guarantees the inode data is up to date, fdatasync just the data. The
directory name SuS and posix dont cover or provide any method for guaranteeing.
The convention Linux adopted is that its valid to fsync() a directory file
handle. It seemed the logical abstraction.

Alan

Re: Ext2, fsync() and MTA's?

[Andrew McNamara]

>> Under Linux, the Postfix MTA sets "chattr +S" on it's spool directories
>> - obviously this hurts it's performance badly (compared to the BSD's).
>
>Not really. BSD directory updates are always synchronous in the cases postfix
>cares about. At least on the old BSD FFS/UFS file systems. Thats the only
>reason the postfix stuff doesnt need it on BSD.

Indeed, but doesn't "chattr +S" on the parent directory causes the file
data to be writen syncronously also, which would really hurt.

>> It would be really nice to be able to say it's no longer necessary. It
>> wants to know that a file (file data, inode and directory entry) are
>> commited to stable storage when an fsync returns.
>
>fsync guarantees the inode data is up to date, fdatasync just the data. The
>directory name SuS and posix dont cover or provide any method for guaranteeing.
>The convention Linux adopted is that its valid to fsync() a directory file
>handle. It seemed the logical abstraction.

I seem to recall that in 2.2, fsync behaved like fdatasync, and that
it's only in 2.4 that it also syncs metadata - is this correct?

Do the BSD's sync the directory data on an fsync of a file? I guess
this is the bone of contention - if linux is now doing exactly what BSD
does, it will make it a lot easier to answer accusations of this
nature.

Wietse isn't prepared to add code to Postfix to fsync the parent
directory's handle just for Linux - it would have to be done in a lot
of places (even though he wraps most system calls already), and he is
concerned one location may be missed: it's not something that can
tested for.

 ---
Andrew McNamara (System Architect)

connect.com.au Pty Ltd
Lvl 3, 213 Miller St, North Sydney, NSW 2060, Australia
Phone: +61 2 9409 2117, Fax: +61 2 9409 2111

Re: Ext2, fsync() and MTA's?

[Theodore Tso]

On Sun, May 13, 2001 at 12:53:37AM +1000, Andrew McNamara wrote:
> I seem to recall that in 2.2, fsync behaved like fdatasync, and that
> it's only in 2.4 that it also syncs metadata - is this correct?

No, fsync in 2.2 also sync'ed the metadata.  The question was always
the containing directory.

> 
> Do the BSD's sync the directory data on an fsync of a file? I guess
> this is the bone of contention - if linux is now doing exactly what BSD
> does, it will make it a lot easier to answer accusations of this
> nature.

I haven't looked at the BSD kernel code in a while, but it's actually
quite difficult to sync the directory data based on the file
descriptor, because you in general don't have a handle on the
directory entry used to open the file.  It turns out with Linux that
it's easier to do this, because of the dcache abstraction, but with
BSD internals, it'd be rather difficult, since you can't get back from
the inode to the directory which contains the inode.  

Sync'ing the directory entry also doesn't make much sense --- which
directory entry?  Suppose an application opens a file with O_CREAT,
and while the file descriptor is opened, it then creates 300 hard
links to that same file to different directories scattered all over
the filesystem.  It now does a fsync(); is it supposed to find all of
the directories which have hard links to that inode, and sync them
all?  That's pretty unreasonable.  

You could make the case that a file which is opened with the O_SYNC |
O_CREAT should sync the directory entry on an open, but to say that
fsync() should scan all directories on the filesystem to sync any hard
links that might have been created (possibly by some other process
that doesn't even have the file open) is quite ridiculous.  And if
that's the case, why should the directory which was used to open the
file be treated specially?

							- Ted

Re: Ext2, fsync() and MTA's?

[Stephen C. Tweedie]

Hi,

On Sun, May 13, 2001 at 12:53:37AM +1000, Andrew McNamara wrote:

> I seem to recall that in 2.2, fsync behaved like fdatasync, and that
> it's only in 2.4 that it also syncs metadata - is this correct?

No, fsync should be safe on 2.2.  There was a problem with O_SYNC not
syncing all metadata on 2.2 if you were extending a file, but that
never applied to fsync.

> Do the BSD's sync the directory data on an fsync of a file? I guess
> this is the bone of contention

No --- the old BSDs were safe because their directory operations were
fully synchronous so they *never* needed to be sync'ed manually.
According to SuS, an application relying on sync directory updates is
buggy, because SuS simply makes no such guarantees.

Just set chattr +S on the spool dir.  That's what the flag is for.
The biggest problem with that is that it propagates to subdirectories
and files --- would a version of the flag which applied only to
directories be a help here?

Cheers,
 Stephen

Re: Ext2, fsync() and MTA's?

[Theodore Tso]

On Mon, May 21, 2001 at 06:47:58PM +0100, Stephen C. Tweedie wrote:
>
> No --- the old BSDs were safe because their directory operations were
> fully synchronous so they *never* needed to be sync'ed manually.
> According to SuS, an application relying on sync directory updates is
> buggy, because SuS simply makes no such guarantees.
> 
> Just set chattr +S on the spool dir.  That's what the flag is for.
> The biggest problem with that is that it propagates to subdirectories
> and files --- would a version of the flag which applied only to
> directories be a help here?
> 

That's probably the right thing to add.

						- Ted

Re: Ext2, fsync() and MTA's?

[Oliver Xymoron]

On Mon, 21 May 2001, Theodore Tso wrote:

> On Mon, May 21, 2001 at 06:47:58PM +0100, Stephen C. Tweedie wrote:
>
> > Just set chattr +S on the spool dir.  That's what the flag is for.
> > The biggest problem with that is that it propagates to subdirectories
> > and files --- would a version of the flag which applied only to
> > directories be a help here?
>
> That's probably the right thing to add.

I'd vote for an async flag instead.

--
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Ext2, fsync() and MTA's?

[Stephen C. Tweedie]

Hi,

On Tue, May 22, 2001 at 10:50:51AM -0500, Oliver Xymoron wrote:
> On Mon, 21 May 2001, Theodore Tso wrote:
> 
> > On Mon, May 21, 2001 at 06:47:58PM +0100, Stephen C. Tweedie wrote:
> >
> > > Just set chattr +S on the spool dir.  That's what the flag is for.
> > > The biggest problem with that is that it propagates to subdirectories
> > > and files --- would a version of the flag which applied only to
> > > directories be a help here?
> >
> > That's probably the right thing to add.
> 
> I'd vote for an async flag instead.

Why???  Why change the default behaviour to be something much slower?

Cheers,
 Stephen

Re: Ext2, fsync() and MTA's?

[Oliver Xymoron]

On Tue, 22 May 2001, Stephen C. Tweedie wrote:

> On Tue, May 22, 2001 at 10:50:51AM -0500, Oliver Xymoron wrote:
> > On Mon, 21 May 2001, Theodore Tso wrote:
> >
> > > On Mon, May 21, 2001 at 06:47:58PM +0100, Stephen C. Tweedie wrote:
> > >
> > > > Just set chattr +S on the spool dir.  That's what the flag is for.
> > > > The biggest problem with that is that it propagates to subdirectories
> > > > and files --- would a version of the flag which applied only to
> > > > directories be a help here?
> > >
> > > That's probably the right thing to add.
> >
> > I'd vote for an async flag instead.
>
> Why???  Why change the default behaviour to be something much slower?

I was suggesting an async flag _in addition_ to the sync flag, both
propagating to subdirs. Nice and orthogonal.

--
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Ext2, fsync() and MTA's?

[Stephen C. Tweedie]

Hi,

On Tue, May 22, 2001 at 11:54:55AM -0500, Oliver Xymoron wrote:

> > > > That's probably the right thing to add.
> > >
> > > I'd vote for an async flag instead.
> >
> > Why???  Why change the default behaviour to be something much slower?
> 
> I was suggesting an async flag _in addition_ to the sync flag, both
> propagating to subdirs. Nice and orthogonal.

The whole problem is that the flag applies to both files and
directories, but we often only want it enforced on directories
(because we already have fsync for files).  Adding another orthogonal
file+dir async flag won't help that at all.

Cheers,
 Stephen

Re: Ext2, fsync() and MTA's?

[Marco d'Itri]

On May 21, "Stephen C. Tweedie" <sct@redhat.com> wrote:

 >Just set chattr +S on the spool dir.  That's what the flag is for.
 >The biggest problem with that is that it propagates to subdirectories
 >and files --- would a version of the flag which applied only to
 >directories be a help here?
Yes, please. It's what is really needed by MTA (not only for spool, but
for maildir delivery too).

-- 
ciao,
Marco

Re: Ext2, fsync() and MTA's?

[Stephen C. Tweedie]

Hi,

On Sat, May 12, 2001 at 03:13:55PM +0100, Alan Cox wrote:

> fsync guarantees the inode data is up to date, fdatasync just the data.

fdatasync guarantees "important" inode data too.  The only thing that
fdatasync is allowed to skip is the timestamps.

--Stephen