f2fs data write ordering

f2fs data write ordering

[Jialin Li]

Hi,

When testing the crash consistency of f2fs, we saw in the documentation 
that one of the mount option "nobarrier" states that "if this option is 
set, no cache_flush commands are issued but f2fs still guarantees the 
write ordering of all the data writes." Does it imply that both with and 
without the "nobarrier" option set, the ordering of all data writes is 
guaranteed? And this ordering applies to data writes to different files 
too?

We wrote the following python test:

---------
import os

with open("a", "wb") as f:
   f.write(b"hello\n")
with open("b", "wb") as f:
   f.write(b"world\n")
os.sync()

fd0 = os.open("a", os.O_RDWR)
fd1 = os.open("b", os.O_RDWR)
os.pwrite(fd0, b'x', 1)
os.pwrite(fd1, b'y', 1)
os.fdatasync(fd1)
/* Crash here */
----------

However, after the crash, we were able to observe that file "b" on disk 
contains "wyrld" while the content of file "a" is still "hello": the two 
pwrites reached the disk out of order. This happened both when we set 
and unset the "nobarrier" option. Maybe we misunderstood the 
documentation regarding the ordering guarantee of f2fs, and it would be 
much appreciated if you could provide some clarifications.

Thanks,
Jialin

------------------------------------------------------------------------------

Re: f2fs data write ordering

[Jaegeuk Kim]

Hi,

On Thu, Jul 23, 2015 at 10:02:48AM -0700, Jialin Li wrote:
> Hi,
> 
> When testing the crash consistency of f2fs, we saw in the documentation 
> that one of the mount option "nobarrier" states that "if this option is 
> set, no cache_flush commands are issued but f2fs still guarantees the 
> write ordering of all the data writes." Does it imply that both with and 
> without the "nobarrier" option set, the ordering of all data writes is 
> guaranteed? And this ordering applies to data writes to different files 
> too?
> 
> We wrote the following python test:
> 
> ---------
> import os
> 
> with open("a", "wb") as f:
>    f.write(b"hello\n")
> with open("b", "wb") as f:
>    f.write(b"world\n")
> os.sync()
> 
> fd0 = os.open("a", os.O_RDWR)
> fd1 = os.open("b", os.O_RDWR)
> os.pwrite(fd0, b'x', 1)
> os.pwrite(fd1, b'y', 1)
> os.fdatasync(fd1)

This guarantees file "b"'s contents, not "a".

> /* Crash here */
> ----------
> 
> However, after the crash, we were able to observe that file "b" on disk 
> contains "wyrld" while the content of file "a" is still "hello": the two 
> pwrites reached the disk out of order. This happened both when we set 
> and unset the "nobarrier" option. Maybe we misunderstood the 
> documentation regarding the ordering guarantee of f2fs, and it would be 
> much appreciated if you could provide some clarifications.

The write ordering means the order of block writes submitted to the storage
by filesystem internally, which is not related to the user-made write ordering.
(e.g., sequential writes or random writes)

And, you need to understand that the recovery is different from write ordering.
If you want to guarantee any data updates, fsync/fdatasync/sync should be
called likewise the above scenario.
Otherwise, no matter what it was flushed to disk already, f2fs would not recover
its data. That's the POSIX rule.

OTOH, we can think about an internal cache inside a storage.
If a storage has a volatile cache, we should issue cache_flush to the storage
in order to guarantee that its data is recoverable from the storage side.
Otherwise, even if filesystem received end_io for fsync/fdatasync, the data can
be lost by the storage.

If there is no cache inside the storage or storage is safe for written data, we
can use "nobarrier" to enhance the performance.

Thanks,

> 
> Thanks,
> Jialin
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

------------------------------------------------------------------------------