ext3 default journal mode

ext3 default journal mode

[Ric Wheeler]

Now that we have put some some distance between us and the mega-thread, I think 
that it is time to put some sanity back into the ext3 journal mode options.

My preference is to revert the default back to ordered mode or (if Chris is 
ready) back to guarded mode.

As I mentioned in an earlier email, I think *every* distribution undid this 
default back to ordered mode.

Do any of us who deal in file systems really think that writeback is a 
reasonable default for unsophisticated users or non-FS experts (like those in 
the embedded space) who rely on KCONFIG to get a sense of a debate that they missed?

Failing that, I would like to at very least put sane text in the KCONFIG option.

Something like the following:

diff --git a/fs/ext3/Kconfig b/fs/ext3/Kconfig
index fb3c1a2..4057730 100644
--- a/fs/ext3/Kconfig
+++ b/fs/ext3/Kconfig
@@ -36,16 +36,19 @@ config EXT3_DEFAULTS_TO_ORDERED
           mode, and the journal capability allowed it, ext3 used to
           historically default to 'data=ordered'.

-         That was a rather unfortunate choice, because it leads to all
-         kinds of latency problems, and the 'data=writeback' mode is more
-         appropriate these days.
+         Data ordered mode is the mode used by most distributions, but can
+         introduce latency problems in some workloads, especially if you have
+         a very low latency device.

-         You should probably always answer 'n' here, and if you really
-         want to use 'data=ordered' mode, set it in the filesystem itself
-         with 'tune2fs -o journal_data_ordered'.
+         The current default mode is data writeback mode which favors latency but
+         is more prone to corruption after crashes and power failures and will
+         expose deleted data after a crash to newly allocated files.

-         But if you really want to enable the legacy default, you can do
-         so by answering 'y' to this question.
+         If you have been historically happy with ext3's performance, data ordered
+         mode will be a safe choice and you should answer "y" here.
+
+         If you understand the reliability and data privacy issues of data 
writeback
+         mode and are willing to make that trade off, answer "n".

Re: ext3 default journal mode

[Chris Mason]

On Sun, Jul 19, 2009 at 10:50:49PM -0400, Ric Wheeler wrote:
>
> Now that we have put some some distance between us and the mega-thread, I 
> think that it is time to put some sanity back into the ext3 journal mode 
> options.
>
> My preference is to revert the default back to ordered mode or (if Chris 
> is ready) back to guarded mode.

My plan was to get guarded into .30, but that clearly didn't happen.
So, I'll resend this week and try to have it ready for .31, unless
people would rather avoid the changes to the tried and true ext3.

-chris

Re: ext3 default journal mode

[Ric Wheeler]

On 07/20/2009 10:18 AM, Chris Mason wrote:
> On Sun, Jul 19, 2009 at 10:50:49PM -0400, Ric Wheeler wrote:
>    
>> Now that we have put some some distance between us and the mega-thread, I
>> think that it is time to put some sanity back into the ext3 journal mode
>> options.
>>
>> My preference is to revert the default back to ordered mode or (if Chris
>> is ready) back to guarded mode.
>>      
>
> My plan was to get guarded into .30, but that clearly didn't happen.
> So, I'll resend this week and try to have it ready for .31, unless
> people would rather avoid the changes to the tried and true ext3.
>
> -chris
>    

I have been thinking that guarded mode might make for a full replacement 
for writeback mode over time - the more I think about writeback mode, 
the less I like letting casual users hurt themselves with it.

I would be very happy to see guarded mode get resent :-)

Any idea how many distros actually ship today with writeback mode as 
their default?

ric

Re: ext3 default journal mode

[Theodore Tso]

Here's a revised proposal for the KCONFIG text.  

Hopefully this is balanced about the two sides of the issue, without
explicitly advocating for one choice versus another.

What do people think?

						- Ted

P.S.  Note that date=writeback does not make the filesystem more
"prone to corruption after crashes".


config EXT3_DEFAULTS_TO_ORDERED
	bool "Default to 'data=ordered' in ext3"
	depends on EXT3_FS
	help
	  If a filesystem does not explicitly specify a data ordering
	  mode, and the journal capability allowed it, ext3 used to
	  historically default to 'data=ordered'.

          Data=ordered mode is the mode used by most distributions, but can
          introduce latency problems in some workloads, especially if there 
	  is a combination of high bandwidth background writes and foreground
	  processes calling fsync() and waiting for the result.   In worst
	  case scenarios, the fsync() call can 500ms to multiple seconds 
	  to return.

	  The problem with using a default of data=writeback, however,
	  is that is that after a system crash or a power failure,
	  files that were written right before the system went down
	  could contain previously written data or other garbage.
	  With data=ordered mode, any blocks in the file will have
	  been data written by the application, avoiding a possibility
	  of a security breach, which is especially problematic on a
	  multi-user system.  Note, however, that data=ordered does
	  not guarantee that the file will be consistent at an
	  application level; the application must use fsync() at
	  appropriate commit points in order to guarantee
	  application-level consistency.

          If you have been historically happy with ext3's performance,
  	  data=ordered mode will be a safe choice and you should
  	  answer "y" here.  If you understand the reliability and data
  	  privacy issues of data=writeback and are willing to make
  	  that trade off, answer "n".

Re: ext3 default journal mode

[Ric Wheeler]

On 07/20/2009 05:29 PM, Theodore Tso wrote:
> Here's a revised proposal for the KCONFIG text.
>
> Hopefully this is balanced about the two sides of the issue, without
> explicitly advocating for one choice versus another.
>
> What do people think?
>
> 						- Ted


Hi Ted,

I think that this is a huge improvement - thanks!

Ric

>
> P.S.  Note that date=writeback does not make the filesystem more
> "prone to corruption after crashes".
>
>
> config EXT3_DEFAULTS_TO_ORDERED
> 	bool "Default to 'data=ordered' in ext3"
> 	depends on EXT3_FS
> 	help
> 	  If a filesystem does not explicitly specify a data ordering
> 	  mode, and the journal capability allowed it, ext3 used to
> 	  historically default to 'data=ordered'.
>
>            Data=ordered mode is the mode used by most distributions, but can
>            introduce latency problems in some workloads, especially if there
> 	  is a combination of high bandwidth background writes and foreground
> 	  processes calling fsync() and waiting for the result.   In worst
> 	  case scenarios, the fsync() call can 500ms to multiple seconds
> 	  to return.
>
> 	  The problem with using a default of data=writeback, however,
> 	  is that is that after a system crash or a power failure,
> 	  files that were written right before the system went down
> 	  could contain previously written data or other garbage.
> 	  With data=ordered mode, any blocks in the file will have
> 	  been data written by the application, avoiding a possibility
> 	  of a security breach, which is especially problematic on a
> 	  multi-user system.  Note, however, that data=ordered does
> 	  not guarantee that the file will be consistent at an
> 	  application level; the application must use fsync() at
> 	  appropriate commit points in order to guarantee
> 	  application-level consistency.
>
>            If you have been historically happy with ext3's performance,
>    	  data=ordered mode will be a safe choice and you should
>    	  answer "y" here.  If you understand the reliability and data
>    	  privacy issues of data=writeback and are willing to make
>    	  that trade off, answer "n".
>

Re: ext3 default journal mode

[Valerie Aurora]

On Mon, Jul 20, 2009 at 05:29:04PM -0400, Theodore Tso wrote:
> Here's a revised proposal for the KCONFIG text.  
> 
> Hopefully this is balanced about the two sides of the issue, without
> explicitly advocating for one choice versus another.
> 
> What do people think?

I think it's extremely accurate and detailed, but too long - people's
brains turn off after about the 15th line or so.  Here's an attempt to
distill your description down and refer out to another document (which
one?) for people who want to learn more.

(Sorry for the whitespace damage.)

-VAL

 config EXT3_DEFAULTS_TO_ORDERED
 	bool "Default to 'data=ordered' in ext3"
 	depends on EXT3_FS
 	help

          If the mount options for an ext3 filesystem do not
          include a journal mode, mount it in "data=ordered" mode.

          The journal mode options for ext3 have different tradeoffs
          between when data is guaranteed to be on disk and
          performance.  Many applications assume "data=ordered"
          semantics and may lose, destroy, or reveal other user's data
          in other journal modes.  However, "data=ordered" mode can
          also result in major performance problems, including long
          delays before an fsync() call returns.  For details, see:

	  XXX some document

	  Use "data=ordered" mode unless you know it is causing a
	  performance problem for your workload.

          If you are unsure, say 'Y'.

Re: ext3 default journal mode

[Andreas Dilger]

On Jul 20, 2009  19:04 -0400, Valerie Aurora wrote:
> I think it's extremely accurate and detailed, but too long - people's
> brains turn off after about the 15th line or so.  Here's an attempt to
> distill your description down and refer out to another document (which
> one?) for people who want to learn more.
> 
> (Sorry for the whitespace damage.)
> 
> -VAL
> 
>  config EXT3_DEFAULTS_TO_ORDERED
>  	bool "Default to 'data=ordered' in ext3"
>  	depends on EXT3_FS
>  	help
> 
>           If the mount options for an ext3 filesystem do not
>           include a journal mode, mount it in "data=ordered" mode.

I would make this a bit more clear:

            This option sets the default journal mode for ext3 filesystems
	    which do not explicitly specify it in /etc/fstab or at mount
	    time.  It is always possible to set the journal mode for each
	    filesystem independently with "data=writeback", "data=ordered",
	    or "data=journal" mount options.

>           The journal mode options for ext3 have different tradeoffs
>           between when data is guaranteed to be on disk and
>           performance.  Many applications assume "data=ordered"
>           semantics and may lose, destroy, or reveal other user's data
>           in other journal modes.  However, "data=ordered" mode can
>           also result in major performance problems, including long
>           delays before an fsync() call returns.  For details, see:

I think the "... lose, destroy, ..." part is confusing, as it mentions
"data=ordered" first and it isn't until the end of the sentence that
it is clear that "lose, destroy, ..." does not apply to data=ordered.
Also "data=journal" also does not apply in this case, only "data=writeback"
so we may as well call that out explicitly.

            ...  Many applications do not explicitly sync data and assume
	    "data=ordered" mode.  Saying 'N' here will use "data=writeback"
	    as the default for all ext3 filesystems, and may result in
	    files with no data, or garbage data from deleted files,
	    which is a security risk on a multi-user system.  However, ...

> 	  XXX some document
> 
> 	  Use "data=ordered" mode unless you know it is causing a
> 	  performance problem for your workload.
> 
>           If you are unsure, say 'Y'.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: ext3 default journal mode

[Valerie Aurora]

On Mon, Jul 20, 2009 at 05:36:20PM -0600, Andreas Dilger wrote:
> On Jul 20, 2009  19:04 -0400, Valerie Aurora wrote:
> > I think it's extremely accurate and detailed, but too long - people's
> > brains turn off after about the 15th line or so.  Here's an attempt to
> > distill your description down and refer out to another document (which
> > one?) for people who want to learn more.
> > 
> > (Sorry for the whitespace damage.)
> > 
> > -VAL
> > 
> >  config EXT3_DEFAULTS_TO_ORDERED
> >  	bool "Default to 'data=ordered' in ext3"
> >  	depends on EXT3_FS
> >  	help
> > 
> >           If the mount options for an ext3 filesystem do not
> >           include a journal mode, mount it in "data=ordered" mode.
> 
> I would make this a bit more clear:
> 
>             This option sets the default journal mode for ext3 filesystems
> 	    which do not explicitly specify it in /etc/fstab or at mount
> 	    time.  It is always possible to set the journal mode for each
> 	    filesystem independently with "data=writeback", "data=ordered",
> 	    or "data=journal" mount options.
> 
> >           The journal mode options for ext3 have different tradeoffs
> >           between when data is guaranteed to be on disk and
> >           performance.  Many applications assume "data=ordered"
> >           semantics and may lose, destroy, or reveal other user's data
> >           in other journal modes.  However, "data=ordered" mode can
> >           also result in major performance problems, including long
> >           delays before an fsync() call returns.  For details, see:
> 
> I think the "... lose, destroy, ..." part is confusing, as it mentions
> "data=ordered" first and it isn't until the end of the sentence that
> it is clear that "lose, destroy, ..." does not apply to data=ordered.
> Also "data=journal" also does not apply in this case, only "data=writeback"
> so we may as well call that out explicitly.
> 
>             ...  Many applications do not explicitly sync data and assume
> 	    "data=ordered" mode.  Saying 'N' here will use "data=writeback"
> 	    as the default for all ext3 filesystems, and may result in
> 	    files with no data, or garbage data from deleted files,
> 	    which is a security risk on a multi-user system.  However, ...

Good points.  I think Ted's rewrite addresses most of them, what do you think?

-VAL

Re: ext3 default journal mode

[Theodore Tso]

On Mon, Jul 20, 2009 at 07:04:03PM -0400, Valerie Aurora wrote:
> On Mon, Jul 20, 2009 at 05:29:04PM -0400, Theodore Tso wrote:
> > Here's a revised proposal for the KCONFIG text.  
> > 
> > Hopefully this is balanced about the two sides of the issue, without
> > explicitly advocating for one choice versus another.
> > 
> > What do people think?
> 
> I think it's extremely accurate and detailed, but too long - people's
> brains turn off after about the 15th line or so.  Here's an attempt to
> distill your description down and refer out to another document (which
> one?) for people who want to learn more.

Fair enough;  I've created an external document here:

http://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode

Comments?

>  config EXT3_DEFAULTS_TO_ORDERED
>  	bool "Default to 'data=ordered' in ext3"
>  	depends on EXT3_FS
>  	help
> 
>           If the mount options for an ext3 filesystem do not
>           include a journal mode, mount it in "data=ordered" mode.

Let me give a try at this:

           The journal mode options for ext3 have different tradeoffs
           between when data is guaranteed to be on disk and
           performance.  The use of "data=writeback" can cause
           unwritten data to appear in files after an system crash or
           power failure, which can be a security issue.  However,
           "data=ordered" mode can also result in major performance
           problems, including seconds-long delays before an fsync()
           call returns.  For details, see:

	   http://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode

          If you have been historically happy with ext3's performance,
          data=ordered mode will be a safe choice and you should
          answer "y" here.  If you understand the reliability and data
          privacy issues of data=writeback and are willing to make
          that trade off, answer "n".
	   
						- Ted

Re: ext3 default journal mode

[Valerie Aurora]

On Mon, Jul 20, 2009 at 10:00:50PM -0400, Theodore Tso wrote:
> On Mon, Jul 20, 2009 at 07:04:03PM -0400, Valerie Aurora wrote:
> > 
> > I think it's extremely accurate and detailed, but too long - people's
> > brains turn off after about the 15th line or so.  Here's an attempt to
> > distill your description down and refer out to another document (which
> > one?) for people who want to learn more.
> 
> Fair enough;  I've created an external document here:
> 
> http://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode
> 
> Comments?

Nice!

> >  config EXT3_DEFAULTS_TO_ORDERED
> >  	bool "Default to 'data=ordered' in ext3"
> >  	depends on EXT3_FS
> >  	help
> > 
> >           If the mount options for an ext3 filesystem do not
> >           include a journal mode, mount it in "data=ordered" mode.
> 
> Let me give a try at this:
> 
>            The journal mode options for ext3 have different tradeoffs
>            between when data is guaranteed to be on disk and
>            performance.  The use of "data=writeback" can cause
>            unwritten data to appear in files after an system crash or
>            power failure, which can be a security issue.  However,
>            "data=ordered" mode can also result in major performance
>            problems, including seconds-long delays before an fsync()
>            call returns.  For details, see:
> 
> 	   http://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode
> 
>           If you have been historically happy with ext3's performance,
>           data=ordered mode will be a safe choice and you should
>           answer "y" here.  If you understand the reliability and data
>           privacy issues of data=writeback and are willing to make
>           that trade off, answer "n".

This is great.  Ship it!

-VAL

Re: ext3 default journal mode

[Ric Wheeler]

On 07/21/2009 01:44 PM, Valerie Aurora wrote:
> On Mon, Jul 20, 2009 at 10:00:50PM -0400, Theodore Tso wrote:
>    
>> On Mon, Jul 20, 2009 at 07:04:03PM -0400, Valerie Aurora wrote:
>>      
>>> I think it's extremely accurate and detailed, but too long - people's
>>> brains turn off after about the 15th line or so.  Here's an attempt to
>>> distill your description down and refer out to another document (which
>>> one?) for people who want to learn more.
>>>        
>> Fair enough;  I've created an external document here:
>>
>> http://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode
>>
>> Comments?
>>      
>
> Nice!
>
>    
>>>   config EXT3_DEFAULTS_TO_ORDERED
>>>   	bool "Default to 'data=ordered' in ext3"
>>>   	depends on EXT3_FS
>>>   	help
>>>
>>>            If the mount options for an ext3 filesystem do not
>>>            include a journal mode, mount it in "data=ordered" mode.
>>>        
>> Let me give a try at this:
>>
>>             The journal mode options for ext3 have different tradeoffs
>>             between when data is guaranteed to be on disk and
>>             performance.  The use of "data=writeback" can cause
>>             unwritten data to appear in files after an system crash or
>>             power failure, which can be a security issue.  However,
>>             "data=ordered" mode can also result in major performance
>>             problems, including seconds-long delays before an fsync()
>>             call returns.  For details, see:
>>
>> 	   http://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode
>>
>>            If you have been historically happy with ext3's performance,
>>            data=ordered mode will be a safe choice and you should
>>            answer "y" here.  If you understand the reliability and data
>>            privacy issues of data=writeback and are willing to make
>>            that trade off, answer "n".
>>      
>
> This is great.  Ship it!
>
> -VAL
>    

I certainly agree - this is a vast improvement over what we have today, 
thanks!

Ric

Re: ext3 default journal mode

[Andi Kleen]

Ric Wheeler <rwheeler@redhat.com> writes:
>
> As I mentioned in an earlier email, I think *every* distribution undid
> this default back to ordered mode.

I would agree too that the default should be changed again. 

Shortly after the default was changed to writeback
I had a (unrelated crash) and afterwards had to fix up several
files in the file system manually that contained junk. Never
has happened before with ordered.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.