Re: aio_fsync() a directory ?

Re: aio_fsync() a directory ?

[Christoph Hellwig]

This change is completely contrary to real world behaviour.  No modern
filesystem I know of implements this behaviour as the default, and performance
with the coresponding mount options (usually -o dirsync on Linux) is terrible
as it forces a write out of the log (or corresponding action on non-log based
filesystems) and in the common case of volatile write caches a cache flush.

Retrospectively claiming this as standards behaviour in a "clarification"
is utterly wrong.

On Mon, Apr 22, 2013 at 11:15:38AM +0200, Xavier Roche wrote:
> Just to close this thread, Geoff Clare has summarized the changes regarding 
> aio_fsync() and relationship wrt. directory entry data.
>
> <http://austingroupbugs.net/view.php?id=672>
>
> "In the April 18 teleconference it was agreed that the standard
> should mandate that directory operations are always synchronized
> on conforming file systems, and should include warnings about
> non-conforming configurations. The proposed changes are as follows.
>
> Changes to XBD...
>
> At page 94 line 2581-2588 section 3.376 change:
>
>     For read, when the operation has been completed or diagnosed if
>     unsuccessful. The read is complete only when an image of the data
>     has been successfully transferred to the requesting process. If
>     there were any pending write requests affecting the data to be
>     read at the time that the synchronized read operation was
>     requested, these write requests are successfully transferred prior
>     to reading the data.
>
>     For write, when the operation has been completed or diagnosed if
>     unsuccessful. The write is complete only when the data specified
>     in the write request is successfully transferred and all file
>     system information required to retrieve the data is successfully
>     transferred.
>
> to:
>
>     For read operations, when the operation has been completed or
>     diagnosed if unsuccessful. The operation is complete only when an
>     image of the data has been successfully transferred to the
>     requesting process. If there were any pending write requests or
>     (if the file is a directory) directory modifications affecting
>     the data to be read at the time that the synchronized read
>     operation was requested, these requests are successfully
>     transferred prior to reading the data.
>
>     For write operations and directory modification operations, when
>     the operation has been completed or diagnosed if unsuccessful. The
>     operation is complete only when the written data or (if the file
>     is a directory) modified directory entries have been successfully
>     transferred to storage and all file system information required
>     to retrieve them is successfully transferred.
>
> At page 107 line 2859 add a new XBD 4.2 section (and renumber the
> current 4.2 and all later 4.x sections):
>
>     4.2 Directory Operations
>
>     All file system operations that read a directory or that modify
>     the contents of a directory (for example creating, unlinking, or
>     renaming a file) shall be completed as defined for synchronized
>     I/O data integrity completion (see section 3.376).
>
>     <small>Note: Although conforming file systems are required to
>        perform all directory modifications as synchronized I/O
>        operations, some file systems may support non-conforming
>        configurations (for example via mount options) where
>        directory modifications are not synchronized. Applications
>        that rely on directory modifications being synchronized should
>        only be used with such file systems in their conforming
>        configuration(s).</small>
>
> Changes to XSH...
>
> At page 574 line 19833 section aio_fsync() change the APPLICATION
> USAGE section from:
>
>     None.
>
> to:
>
>     Refer to fdatasync() and fsync().
>
> At page 815 line 27215 section fdatasync() append to the first
> paragraph:
>
>     If the file is a directory, an implicit fdatasync() is already
>     performed on every I/O operation (see XBD 4.2) and consequently
>     if fdatasync() is called explicitly it shall take no action and
>     shall return the value 0.
>
> At page 815 line 27232 section fdatasync() change the APPLICATION
> USAGE section from:
>
>     None.
>
> to:
>
>     Although conforming file systems are required to complete all
>     directory modifications as defined for synchronized I/O data
>     integrity completion, some file systems may support non-conforming
>     configurations (for example via mount options) where directory
>     modifications are not synchronized. When the file system is
>     configured in this way, calls to fdatasync() on directories may
>     cause I/O operations to be synchronized, rather than being a no-op.
>
> At page 954 line 31987 section fsync() add a new paragraph to the
> APPLICATION USAGE section:
>
>     Since conforming file systems are required to complete all
>     directory modifications as defined for synchronized I/O data
>     integrity completion (see XBD 4.2), calling fsync() on a directory
>     only synchronizes the file attributes such as timestamps. However,
>     some file systems may support non-conforming configurations (for
>     example via mount options) where modifications to directory
>     contents are not synchronized. When the file system is configured
>     in this way, calls to fsync() on directories may cause directory
>     contents to be synchronized in addition to file attributes.
>
> Changes to XRAT...
>
> At page 3444 line 115531 add a new XRAT A.4.2 section (and renumber
> the current A.4.2 and all later A.4.x sections):
>
>     A.4.2 Directory Operations
>
>     Earlier versions of this standard did not make clear that all directory
>     modifications are performed as synchronized I/O operations, although
>     that is the historical behavior and was always intended. Applications
>     have no need to call fdatasync() or fsync() on a directory unless
>     they want to synchronize the file attributes (using fsync()), provided
>     the directory is on a conforming file system. However, since
>     applications may wish to use fdatasync() or fsync() to synchronize
>     directory modifications on non-conforming file systems, implementations
>     are required to support fdatasync() on directories as a no-op on
>     conforming file systems."
---end quoted text---

Re: aio_fsync() a directory ?

[Xavier Roche]

On 04/24/2013 04:48 AM, Christoph Hellwig wrote:
> This change is completely contrary to real world behaviour.  No modern
> filesystem I know of implements this behaviour as the default, and performance
> with the coresponding mount options (usually -o dirsync on Linux) is terrible
> as it forces a write out of the log (or corresponding action on non-log based
> filesystems) and in the common case of volatile write caches a cache flush.
>
> Retrospectively claiming this as standards behaviour in a "clarification"
> is utterly wrong.

As I understand the wording, the original intent was to have all file 
operations synchronized (ie. operation committed permanently) at the 
"beginning" of the specification. The updated one suggests that an 
implementation may be non-conformant, and allow fsync() on a directory 
entry.

I'm also a bit puzzled by the default synchronization guarantee (which 
would lead an I/O per open/rename/etc. operation ?) ; even if many 
developers just do not care about filename synchronization. [ they are 
probably assuming that the fsync() on the file itself will also flush 
the related directory entrie(s) ]

Re: aio_fsync() a directory ?

[Al Viro]

On Wed, Apr 24, 2013 at 07:31:14AM +0200, Xavier Roche wrote:
> On 04/24/2013 04:48 AM, Christoph Hellwig wrote:
> >This change is completely contrary to real world behaviour.  No modern
> >filesystem I know of implements this behaviour as the default, and performance
> >with the coresponding mount options (usually -o dirsync on Linux) is terrible
> >as it forces a write out of the log (or corresponding action on non-log based
> >filesystems) and in the common case of volatile write caches a cache flush.
> >
> >Retrospectively claiming this as standards behaviour in a "clarification"
> >is utterly wrong.
> 
> As I understand the wording, the original intent was to have all
> file operations synchronized (ie. operation committed permanently)
> at the "beginning" of the specification. The updated one suggests
> that an implementation may be non-conformant, and allow fsync() on a
> directory entry.

Original intent of _what_?  With all due respect, original intent of
Austin Group is none of our concern - "we had always intended to make
that promise on your behalf, now we'd simply got around to saying so
clearly" inspires many things, but "oh, sure, it's binding for us now"
is *not* one of those.  Suggestions to do anatomically impossible things,
OTOH...

Re: aio_fsync() a directory ?

[Roland McGrath]

Take this up with the POSIX committee.  Complaining on implementors'
mailing lists won't have any effect on the future text of standards.

Re: aio_fsync() a directory ?

[Xavier Roche]

On 04/24/2013 06:34 PM, Roland McGrath wrote:
> Take this up with the POSIX committee.  Complaining on implementors'
> mailing lists won't have any effect on the future text of standards.

Absolutely. I'd like to mention that the austin-group-l mailing-list is 
opened to anyone wishing to contribute (with proper technical arguments, 
of course), and AFAICS opened to remarks (this bug was opened following 
one of my remarks - so you may consider this as my fault :p)

Re: aio_fsync() a directory ?

[Rich Felker]

On Wed, Apr 24, 2013 at 06:54:39PM +0200, Xavier Roche wrote:
> On 04/24/2013 06:34 PM, Roland McGrath wrote:
> >Take this up with the POSIX committee.  Complaining on implementors'
> >mailing lists won't have any effect on the future text of standards.
> 
> Absolutely. I'd like to mention that the austin-group-l mailing-list
> is opened to anyone wishing to contribute (with proper technical
> arguments, of course), and AFAICS opened to remarks (this bug was
> opened following one of my remarks - so you may consider this as my
> fault :p)

I'm guilty of a similar mistake, with issue #663, and would appreciate
any supporting arguments against the ridiculous change in the standard
they want to make and pretend is just a clarification of historic
practice. The proposed change also renders glibc non-conforming so I
think it's on-topic.

Rich

Re: aio_fsync() a directory ?

[Paul Eggert]

I have followed up with the Austin Group, here:

http://austingroupbugs.net/view.php?id=672#c1545

and would appreciate further review and followup
as needed.  To add further comments to that page,
one must be an Austin Group member; if you're not one
already, you can easily become one by using the
"Signup for a new account" link on that page.