ext3 and SPEC SFS Run rules.

ext3 and SPEC SFS Run rules.

[Tigran Aivazian]

Hi,

I have a simple question --- does ext3 conform to the SPEC SFS Run rules 
with default mount options or does one need to force the correct 
behaviour by mounting in some "special" way?. Here is the URL for the 
rules:

  http://www.spec.org/sfs97r1/docs/runrules.html

In particular, the bit that is worrying me is:

  For NFS Version 3, the server adheres to the protocol specification. In 
  particular the requirement that for STABLE write requests and COMMIT 
  operations the NFS server must not reply to the NFS client before any 
  modified file system data or metadata, with the exception of access times, 
  are written to stable storage for that specific or related operation. 
  See RFC 1813, NFSv3 protocol specification for a definition of STABLE 
  and COMMIT for NFS write requests.

As far as I can see from nfsd source this means:

a) write with 'stable' flag set does a f_op->write() with O_SYNC.

b) COMMIT3 just means f_op->fsync().

So, the question is really --- do ext3 O_SYNC write and fsync require some 
special mount options to work _properly_ or are they fine by default?

Looking at ext3_sync_file() it seems OK by default.

However, looking at ext3 ->write() operation it seems to forcibly commit 
only if it is mounted with EXT3_MOUNT_JOURNAL_DATA option (or if it is not 
a regular file or if EXT3_JOURNAL_DATA_FL inode flag is set).

Therefore, it would seem that for ext3 to be valid for SPEC SFS 3.0 (or 
2.0) run one has to mount it with "data=journal". Please correct me if I 
am wrong in coming to this conclusion.

Kind regards
Tigran

Re: ext3 and SPEC SFS Run rules.

[Andrew Morton]

Tigran Aivazian <tigran@aivazian.fsnet.co.uk> wrote:
>
> I have a simple question --- does ext3 conform to the SPEC SFS Run rules 
>  with default mount options or does one need to force the correct 
>  behaviour by mounting in some "special" way?. Here is the URL for the 
>  rules:
> 
>    http://www.spec.org/sfs97r1/docs/runrules.html
> 
>  In particular, the bit that is worrying me is:
> 
>    For NFS Version 3, the server adheres to the protocol specification. In 
>    particular the requirement that for STABLE write requests and COMMIT 
>    operations the NFS server must not reply to the NFS client before any 
>    modified file system data or metadata, with the exception of access times, 
>    are written to stable storage for that specific or related operation. 
>    See RFC 1813, NFSv3 protocol specification for a definition of STABLE 
>    and COMMIT for NFS write requests.
> 
>  As far as I can see from nfsd source this means:
> 
>  a) write with 'stable' flag set does a f_op->write() with O_SYNC.
> 
>  b) COMMIT3 just means f_op->fsync().
> 
>  So, the question is really --- do ext3 O_SYNC write and fsync require some 
>  special mount options to work _properly_ or are they fine by default?
> 
>  Looking at ext3_sync_file() it seems OK by default.
> 
>  However, looking at ext3 ->write() operation it seems to forcibly commit 
>  only if it is mounted with EXT3_MOUNT_JOURNAL_DATA option (or if it is not 
>  a regular file or if EXT3_JOURNAL_DATA_FL inode flag is set).

ext3 should be fully syncing data and metadata for both fsync() and O_SYNC
writes in all three journalling modes.  If not, that's a big bug.

Re: ext3 and SPEC SFS Run rules.

[Tigran Aivazian]

On Mon, 26 Jul 2004, Andrew Morton wrote:
> ext3 should be fully syncing data and metadata for both fsync() and O_SYNC
> writes in all three journalling modes.  If not, that's a big bug.

Ok, so, can I conclude that you are therefore saying that ext3 (with 
default mount options) is compliant with SPEC SFS Run rules wrt NFS 
protocol requirements:

   1.   For NFS Version 2, the server adheres to the protocol 
specification and in particular the requirement that for NFS write 
operations the NFS server must not reply to the NFS client before any 
modified file system data or metadata, with the exception of access times, 
are written to stable storage.
   2. For NFS Version 3, the server adheres to the protocol specification. 
In particular the requirement that for STABLE write requests and COMMIT 
operations the NFS server must not reply to the NFS client before any 
modified file system data or metadata, with the exception of access times, 
are written to stable storage for that specific or related operation. See 
RFC 1813, NFSv3 protocol specification for a definition of STABLE and 
COMMIT for NFS write requests.
   3. For NFS Version 3, operations which are specified to return wcc data 
must, in all cases, return TRUE and the correct attribute data. Those 
operations are:

      NFS Version 3
      SETATTR
      READLINK
      CREATE
      MKDIR
      SYMLINK
      MKNOD
      REMOVE
      RMDIR
      RENAME
      LINK

   4. The server must pass the benchmark validation for the NFS protocol 
being tested.
   5. When UDP is the network transport, UDP checksums must be calculated 
and verified for all NFS request and reply messages. In other words, 
checksums must be enabled on both the client and server.


Kind regards
Tigran

Re: ext3 and SPEC SFS Run rules.

[Tigran Aivazian]

I think the important part of rule 1. saying "server adheres to the 
protocol" is this bit from rfc1813:

   The following data modifying procedures are
   synchronous: WRITE (with stable flag set to FILE_SYNC), CREATE,
   MKDIR, SYMLINK, MKNOD, REMOVE, RMDIR, RENAME, LINK, and COMMIT.

E.g. if a client had a successful SYMLINK and the server crashed, the 
client is not going to discover on the next reboot (of the server) that 
the symlink is somehow disappeared...

Kind regards
Tigran

On Mon, 26 Jul 2004, Tigran Aivazian wrote:

> On Mon, 26 Jul 2004, Andrew Morton wrote:
> > ext3 should be fully syncing data and metadata for both fsync() and O_SYNC
> > writes in all three journalling modes.  If not, that's a big bug.
> 
> Ok, so, can I conclude that you are therefore saying that ext3 (with 
> default mount options) is compliant with SPEC SFS Run rules wrt NFS 
> protocol requirements:
> 
>    1.   For NFS Version 2, the server adheres to the protocol 
> specification and in particular the requirement that for NFS write 
> operations the NFS server must not reply to the NFS client before any 
> modified file system data or metadata, with the exception of access times, 
> are written to stable storage.
>    2. For NFS Version 3, the server adheres to the protocol specification. 
> In particular the requirement that for STABLE write requests and COMMIT 
> operations the NFS server must not reply to the NFS client before any 
> modified file system data or metadata, with the exception of access times, 
> are written to stable storage for that specific or related operation. See 
> RFC 1813, NFSv3 protocol specification for a definition of STABLE and 
> COMMIT for NFS write requests.
>    3. For NFS Version 3, operations which are specified to return wcc data 
> must, in all cases, return TRUE and the correct attribute data. Those 
> operations are:
> 
>       NFS Version 3
>       SETATTR
>       READLINK
>       CREATE
>       MKDIR
>       SYMLINK
>       MKNOD
>       REMOVE
>       RMDIR
>       RENAME
>       LINK
> 
>    4. The server must pass the benchmark validation for the NFS protocol 
> being tested.
>    5. When UDP is the network transport, UDP checksums must be calculated 
> and verified for all NFS request and reply messages. In other words, 
> checksums must be enabled on both the client and server.
> 
> 
> Kind regards
> Tigran
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: ext3 and SPEC SFS Run rules.

[Theodore Ts'o]

On Mon, Jul 26, 2004 at 10:12:01AM +0100, Tigran Aivazian wrote:
> On Mon, 26 Jul 2004, Andrew Morton wrote:
> > ext3 should be fully syncing data and metadata for both fsync() and O_SYNC
> > writes in all three journalling modes.  If not, that's a big bug.
> 
> Ok, so, can I conclude that you are therefore saying that ext3 (with 
> default mount options) is compliant with SPEC SFS Run rules wrt NFS 
> protocol requirements:

This is up to the NFS server, not to the underlying filesystem.  It
would be insane for every single local write() to have mandatory
O_SYNC semantics, even if it is required by the NFS protocol.  

In modern Linux implementations of the NFS server, you can specify
whether or not the synchronization semantics as demanded by the NFS
protocol, or whether the more relaxed async writebacks are done.
Obviously, this has some pretty serious performance vs robustness
implications.

						- Ted

Re: ext3 and SPEC SFS Run rules.

[Andrew Morton]

Tigran Aivazian <tigran@aivazian.fsnet.co.uk> wrote:
>
> On Mon, 26 Jul 2004, Andrew Morton wrote:
> > ext3 should be fully syncing data and metadata for both fsync() and O_SYNC
> > writes in all three journalling modes.  If not, that's a big bug.
> 
> Ok, so, can I conclude that you are therefore saying that ext3 (with 
> default mount options) is compliant with SPEC SFS Run rules wrt NFS 
> protocol requirements:

That is the design objective within ext3, the VFS and the NFS server, yes. 
Of course, there may be bugs in the implementation.

>    1.   For NFS Version 2, the server adheres to the protocol 
> specification and in particular the requirement that for NFS write 
> operations the NFS server must not reply to the NFS client before any 
> modified file system data or metadata, with the exception of access times, 
> are written to stable storage.
>    2. For NFS Version 3, the server adheres to the protocol specification. 
> In particular the requirement that for STABLE write requests and COMMIT 
> operations the NFS server must not reply to the NFS client before any 
> modified file system data or metadata, with the exception of access times, 
> are written to stable storage for that specific or related operation. See 
> RFC 1813, NFSv3 protocol specification for a definition of STABLE and 
> COMMIT for NFS write requests.
>    3. For NFS Version 3, operations which are specified to return wcc data 
> must, in all cases, return TRUE and the correct attribute data. Those 
> operations are:
> 
>       NFS Version 3
>       SETATTR
>       READLINK
>       CREATE
>       MKDIR
>       SYMLINK
>       MKNOD
>       REMOVE
>       RMDIR
>       RENAME
>       LINK

To confirm this we'd need to undertake an audit of the server
implementation, and we'll need to ask Neil about it.

Re: ext3 and SPEC SFS Run rules.

[Neil Brown]

On Monday July 26, akpm@osdl.org wrote:
> >    2. For NFS Version 3, the server adheres to the protocol specification. 
> > In particular the requirement that for STABLE write requests and COMMIT 
> > operations the NFS server must not reply to the NFS client before any 
> > modified file system data or metadata, with the exception of access times, 
> > are written to stable storage for that specific or related operation. See 
> > RFC 1813, NFSv3 protocol specification for a definition of STABLE and 
> > COMMIT for NFS write requests.

Providing you use the "sync" export option, the Linux kNFSd should
meet this requirement.

> >    3. For NFS Version 3, operations which are specified to return wcc data 
> > must, in all cases, return TRUE and the correct attribute data. Those 
> > operations are:
> > 
> >       NFS Version 3
> >       SETATTR
> >       READLINK
> >       CREATE
> >       MKDIR
> >       SYMLINK
> >       MKNOD
> >       REMOVE
> >       RMDIR
> >       RENAME
> >       LINK
> 
> To confirm this we'd need to undertake an audit of the server
> implementation, and we'll need to ask Neil about it.

To help with the audit: the wcc data is set by "fh_unlock".  i.e. when
a filehandle is unlocked (up(i_sem)) a copy of the state information
is taken and included where appropriate in the reply.

The only place that I am aware of where we *do*not* return wcc data is
for the WRITE request (which you have not listed).  As the underlying
filesystem is left to do whatever locking it thinks is appropriate,
and vfs_write does none, nfsd is not in a position to lock it itself
against sys_write and so cannot record before and after stat
information that is atomic w.r.t the update.

NeilBrown

Re: ext3 and SPEC SFS Run rules.

[Brian Pawlowski]

> The only place that I am aware of where we *do*not* return wcc data is
> for the WRITE request (which you have not listed).  As the underlying
> filesystem is left to do whatever locking it thinks is appropriate,
> and vfs_write does none, nfsd is not in a position to lock it itself
> against sys_write and so cannot record before and after stat
> information that is atomic w.r.t the update.

Without replaying conversations on V3 and wcc_ stuff I do believe that
long about 1993 or 1994 most decided that making the pre- and post-stat
info atomic w.r.t. the update was not "required" - because that would
be hard.

The WCC stuff is relegated to a hint to help detect conflicting
changes in that case I believe.

beepy

Re: ext3 and SPEC SFS Run rules.

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On Jul 26, 2004  10:18 +0100, Tigran Aivazian wrote:
> I think the important part of rule 1. saying "server adheres to the 
> protocol" is this bit from rfc1813:
> 
>    The following data modifying procedures are
>    synchronous: WRITE (with stable flag set to FILE_SYNC), CREATE,
>    MKDIR, SYMLINK, MKNOD, REMOVE, RMDIR, RENAME, LINK, and COMMIT.
> 
> E.g. if a client had a successful SYMLINK and the server crashed, the 
> client is not going to discover on the next reboot (of the server) that 
> the symlink is somehow disappeared...

Just an FYI - if you are running a heavy sync load like a sync NFS
server you may want to consider running a large external journal for
performance.  This means your journal data writes are always sequential
and to a different disk than the main filesystem.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]