Re: Problem with disk

[Ric Wheeler <ric@emc.com>]

Tejun Heo wrote:
> 
> 
> Unfortunately, this can result in *massive* destruction of the 
> filesystem.  I lost my RAID-1 array earlier this year this way.  The FS 
> code systematically destroyed metadata of the filesystem and, on the 
> following reboot, fsck did the final blow, I think.  I ended up with 
> 100+Gbytes of unorganized data and I had to recover data by grep + bvi.

Were you running with Neil's fixes that make MD devices properly handle write 
barrier requests?  Until fairly recently (not sure when this was fixed), MD 
devices more or less dropped the barrier requests.

With properly working barriers, any journal file system should get you back to a 
consistent state after a power drop (although there are many less common ways 
that drives can potentially drop data).

> 
> This is an extreme case but it shows turning off writeback has its 
> advantages.  After the initial stress & panic attack subsided, I tried 
> to think about how to prevent such catastrophes, but there doesn't seem 
> to be a good way.  There's no way to tell 1. if the harddrive actually 
> lost the writeback cache content 2. if so, how much it has lost.  So, 
> unless the OS halts the system everytime something seems weird with the 
> disk, turning off writeback cache seems to be the only solution.
> 

Turning off the writeback cache is definitely the safe and conservative way to 
go for mission critical data unless you can be very certain that your barriers 
are properly working on the drive & IO stack.  We validate the cache flush 
commands with a s-ata analyzer (making sure that we see them on sync/transaction 
commits) and that they take a reasonable amount of time at the drive...

ric