sync() in 2.6.38.5

sync() in 2.6.38.5

[Paul Anderson]

Hi all,

2.6.38.5 (x64 intel, in todays case a 40TiByte SAN volume) appears to
have a bug whereby not all active metadata will be flushed even on a
quiescent machine (one that has nonetheless in the past been under
very high load).

We have tried several variations of clean shutdowns, combined with for
example the "echo 3 >/proc/sys/vm/drop_caches" trick to no avail - we
still get lost files (well, 0 length files).

We have several big servers scheduled to go down shortly, and I was
wondering if there are other ideas besides just coping all recent data
to another server.

Thanks,

Paul

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: sync() in 2.6.38.5

[Joe Landman]

On 11/29/2011 02:17 PM, Paul Anderson wrote:
> Hi all,
>
> 2.6.38.5 (x64 intel, in todays case a 40TiByte SAN volume) appears to
> have a bug whereby not all active metadata will be flushed even on a
> quiescent machine (one that has nonetheless in the past been under
> very high load).
>
> We have tried several variations of clean shutdowns, combined with for
> example the "echo 3>/proc/sys/vm/drop_caches" trick to no avail - we
> still get lost files (well, 0 length files).
>
> We have several big servers scheduled to go down shortly, and I was
> wondering if there are other ideas besides just coping all recent data
> to another server.

Set your vm dirty time to small values.  1 second (100 centiseconds) or 
so, among other things.  You can also force the mount to be synchronous 
(kills performance though).

Try

	mount -o remount,sync /mountpoint
	# not sure if this works with xfs though ...

	sysctl -w vm.dirty_writeback_centisecs=100
	sysctl -w vm.dirty_expire_centisecs=100
	sysctl -w vm.dirty_ratio=1



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: sync() in 2.6.38.5

[Christoph Hellwig]

On Tue, Nov 29, 2011 at 02:17:26PM -0500, Paul Anderson wrote:
> Hi all,
> 
> 2.6.38.5 (x64 intel, in todays case a 40TiByte SAN volume) appears to
> have a bug whereby not all active metadata will be flushed even on a
> quiescent machine (one that has nonetheless in the past been under
> very high load).
> 
> We have tried several variations of clean shutdowns, combined with for
> example the "echo 3 >/proc/sys/vm/drop_caches" trick to no avail - we
> still get lost files (well, 0 length files).
> 
> We have several big servers scheduled to go down shortly, and I was
> wondering if there are other ideas besides just coping all recent data
> to another server.

I'd really love to debug this.  We had a few reports of this issue
before, but I've neber been able to pinpoint it.  Do you remember
anything specific to the workload touching these files?

To be save I'd rsync data off the first one going down.  Can you try
to do an explicit fsync for every file, like

	find | xargs /usr/sbin/xfs_io -c 'fsync'

and see if that helps?  Answering that question would help us greatly
to pinpoint down the issue.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs