Re: Files not touched in weeks got truncated after a crash

From: Roger Willcocks <roger@filmlight.ltd.uk>
To: Guido Winkelmann <guido@ambient-entertainment.de>
Cc: xfs@oss.sgi.com
Subject: Re: Files not touched in weeks got truncated after a crash
Date: Wed, 13 Nov 2013 16:51:44 +0000	[thread overview]
Message-ID: <1384361504.4299.132.camel@localhost.localdomain> (raw)
In-Reply-To: <2662179.4mj0dgORXu@r008>

On Wed, 2013-11-13 at 17:36 +0100, Guido Winkelmann wrote:
> Hi,
> 
> We are having some trouble with one of our fileservers using XFS (on linux). 
> Yesterday, one of the external RAIDs on the server failed. Of course, it is 
> unavoidable that some data would get lost from the fileserver in such an 
> event, however, we lost a lot more files than would seem reasonable. In 
> particular, we lost a number of files that had not been written to (but had 
> been been read from, in some cases) in several weeks.
> 
> The data loss manifested itself through files being truncated to length 0 or 
> to some other size short of what they should be. (We happen to have an 
> external database that keeps track of that.)
> 
> The fileserver is based on CentOS 6.3 with kernel version 
> 2.6.32-279.9.1.el6.x86_64. It has got several external RAIDs in the 100 TB 
> range, connected via FibreChannel.
> 
> In case it matters: The server's primary role is as a samba server servicing a 
> large number of Windows XP and Windows 7 machines.
> 
> We had already been trying to reduce the possible impact of a hardware failure 
> by setting a few tunables in /etc/sysctl.conf to try and make the kernel not 
> keep dirty buffers around too long:
> 
> vm.dirty_background_bytes = 536870912
> vm.dirty_bytes = 134217728
> vm.dirty_writeback_centisecs = 500
> vm.dirty_expire_centisecs = 3000
> 
> and by issuing a sync from cron every 15 minutes:
> 
> 0,15,30,45 * * * * /bin/sync
> 
> Unfortunately, I seem to be unable so far to reproduce the issue on a smaller 
> system - and I cannot exactly just walk up to the in-production fileserver and 
> rip out yet another array just to see what happens...
> 
> This leaves me with a few questions:
> 
> Why did we lose so much data through the crash?
> 
> Why did not even a sync every 15 minutes prevent further damage?
> 
> What can we do to prevent this from happening again in the future?
> 
> Regards,
> 
> 	Guido

Syncing won't protect you from hardware failures.

Without more details about what happened to the external RAID and what
you did to recover it's impossible to answer your question.

--
Roger

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs