is the flush-on-close-after-truncate still needed?

is the flush-on-close-after-truncate still needed?

[Eric Sandeen]

After Lachlan's fix to separate on-disk and in-memory sizes, and only
update on-disk when data is on-disk
(http://www.linux.sgi.com/archives/xfs/2007-05/msg00020.html) is the
XFS_ITRUNCATED flag / flush-on-close-after-truncate still needed?

Thanks,
-Eric

Re: is the flush-on-close-after-truncate still needed?

[Dave Chinner]

On Wednesday 18 June 2008 10:09 am, Eric Sandeen wrote:
> After Lachlan's fix to separate on-disk and in-memory sizes, and only
> update on-disk when data is on-disk
> (http://www.linux.sgi.com/archives/xfs/2007-05/msg00020.html) is the
> XFS_ITRUNCATED flag / flush-on-close-after-truncate still needed?

Yes, because waiting 30s before writing back /etc/fstab after it
has been modified will result in lots of bug reports of /etc/fstab
being zero length after a crash instead of being full of NULLs.
We have had very few reports of zero length files or files with
NULLs since this change was made (regardless of the file size 
update ordering changes). i.e. if we remove this code then the
common case where NULL files occurred will return - only this
time as zero length files.

Cheers,

Dave.

Re: is the flush-on-close-after-truncate still needed?

[Eric Sandeen]

Dave Chinner wrote:
> On Wednesday 18 June 2008 10:09 am, Eric Sandeen wrote:
>> After Lachlan's fix to separate on-disk and in-memory sizes, and only
>> update on-disk when data is on-disk
>> (http://www.linux.sgi.com/archives/xfs/2007-05/msg00020.html) is the
>> XFS_ITRUNCATED flag / flush-on-close-after-truncate still needed?
> 
> Yes, because waiting 30s before writing back /etc/fstab after it
> has been modified will result in lots of bug reports of /etc/fstab
> being zero length after a crash instead of being full of NULLs.
> We have had very few reports of zero length files or files with
> NULLs since this change was made (regardless of the file size 
> update ordering changes). i.e. if we remove this code then the
> common case where NULL files occurred will return - only this
> time as zero length files.

Ah, right.  Ok, thanks!

-Eric

> Cheers,
> 
> Dave.
> 

Re: is the flush-on-close-after-truncate still needed?

[DS]

Hmm, but file overwrite in perl/php is slow, very slow.

Which FS is best for me?
XFS - perl/php overwrite problem
EXT3 - 32000 subdirs limit
REISER - no future
JFS - ?

DS

On Wed, Jun 18, 2008 at 10:49:07AM -0700, Dave Chinner wrote:
> On Wednesday 18 June 2008 10:09 am, Eric Sandeen wrote:
> > After Lachlan's fix to separate on-disk and in-memory sizes, and only
> > update on-disk when data is on-disk
> > (http://www.linux.sgi.com/archives/xfs/2007-05/msg00020.html) is the
> > XFS_ITRUNCATED flag / flush-on-close-after-truncate still needed?
> 
> Yes, because waiting 30s before writing back /etc/fstab after it
> has been modified will result in lots of bug reports of /etc/fstab
> being zero length after a crash instead of being full of NULLs.
> We have had very few reports of zero length files or files with
> NULLs since this change was made (regardless of the file size 
> update ordering changes). i.e. if we remove this code then the
> common case where NULL files occurred will return - only this
> time as zero length files.
> 
> Cheers,
> 
> Dave.
> 
> 

Re: is the flush-on-close-after-truncate still needed?

[Eric Sandeen]

DS wrote:
> Hmm, but file overwrite in perl/php is slow, very slow.

If you have control over your perl/php, perhaps you can change it to do
unlink/create/write instead of truncate/write?

-Eric

> Which FS is best for me?
> XFS - perl/php overwrite problem
> EXT3 - 32000 subdirs limit
> REISER - no future
> JFS - ?
> 
> DS
> 
> On Wed, Jun 18, 2008 at 10:49:07AM -0700, Dave Chinner wrote:
>> On Wednesday 18 June 2008 10:09 am, Eric Sandeen wrote:
>>> After Lachlan's fix to separate on-disk and in-memory sizes, and only
>>> update on-disk when data is on-disk
>>> (http://www.linux.sgi.com/archives/xfs/2007-05/msg00020.html) is the
>>> XFS_ITRUNCATED flag / flush-on-close-after-truncate still needed?
>> Yes, because waiting 30s before writing back /etc/fstab after it
>> has been modified will result in lots of bug reports of /etc/fstab
>> being zero length after a crash instead of being full of NULLs.
>> We have had very few reports of zero length files or files with
>> NULLs since this change was made (regardless of the file size 
>> update ordering changes). i.e. if we remove this code then the
>> common case where NULL files occurred will return - only this
>> time as zero length files.
>>
>> Cheers,
>>
>> Dave.
>>
>>
> 
> 

Re: is the flush-on-close-after-truncate still needed?

[Eric Sandeen]

Eric Sandeen wrote:
> DS wrote:
>> Hmm, but file overwrite in perl/php is slow, very slow.
> 
> If you have control over your perl/php, perhaps you can change it to do
> unlink/create/write instead of truncate/write?

Or even:

--- test.perl.orig        2008-06-26 16:22:48.163869293 -0500
+++ test.perl   2008-06-26 16:23:25.426869060 -0500
@@ -3,7 +3,7 @@
 $time=time();
 for ($i=1;$i<100;$i++)
 {
-open (SUBOR,">$i.txt");
+open (SUBOR,"+<$i.txt");
 print SUBOR "aaaaaaaaaaaaaaaaaaa\n";
 close (SUBOR);
 print "WRITE $i. FILE\n";

which gives you RW access, but does not do the truncate.

Or use sysopen.  Or ...

-Eric

Re: is the flush-on-close-after-truncate still needed?

[DS]

Thanx for interest.

There is no chance to change all scripts (too many customers and
thousands and thousands perl/php skripts).

I think it isn't right way compiling own perl/php libs with needed changes on
open/fopen function.

DS

On Thu, Jun 26, 2008 at 04:19:39PM -0500, Eric Sandeen wrote:
> DS wrote:
> > Hmm, but file overwrite in perl/php is slow, very slow.
> 
> If you have control over your perl/php, perhaps you can change it to do
> unlink/create/write instead of truncate/write?
> 
> -Eric
> 
> > Which FS is best for me?
> > XFS - perl/php overwrite problem
> > EXT3 - 32000 subdirs limit
> > REISER - no future
> > JFS - ?
> > 
> > DS
> > 
> > On Wed, Jun 18, 2008 at 10:49:07AM -0700, Dave Chinner wrote:
> >> On Wednesday 18 June 2008 10:09 am, Eric Sandeen wrote:
> >>> After Lachlan's fix to separate on-disk and in-memory sizes, and only
> >>> update on-disk when data is on-disk
> >>> (http://www.linux.sgi.com/archives/xfs/2007-05/msg00020.html) is the
> >>> XFS_ITRUNCATED flag / flush-on-close-after-truncate still needed?
> >> Yes, because waiting 30s before writing back /etc/fstab after it
> >> has been modified will result in lots of bug reports of /etc/fstab
> >> being zero length after a crash instead of being full of NULLs.
> >> We have had very few reports of zero length files or files with
> >> NULLs since this change was made (regardless of the file size 
> >> update ordering changes). i.e. if we remove this code then the
> >> common case where NULL files occurred will return - only this
> >> time as zero length files.
> >>
> >> Cheers,
> >>
> >> Dave.
> >>
> >>
> > 
> > 
> 

Re: is the flush-on-close-after-truncate still needed?

[Dave Chinner]

On Fri, Jun 27, 2008 at 09:13:12AM +0200, DS wrote:
> Thanx for interest.
> 
> There is no chance to change all scripts (too many customers and
> thousands and thousands perl/php skripts).
> 
> I think it isn't right way compiling own perl/php libs with needed changes on
> open/fopen function.

Overwriting files by truncating then first and then not fsync'ing
the file is a sure way to lose data if the system crashes.  That's
an application bug, not a filesystem bug, because the filesystem is
only doing what it is told to do. XFS is ensuring that lazy
application writers are unlikely to lose data when they carelessly
overwriting data.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com