power failure causes files loss

power failure causes files loss

[adamantzj]

Hello,

     I’m writing data on a server in my company, but power cut suddenly. I
had to restart the machine, then I found that many files in the working
directory had been lost. I made some tries with xfs_repair and it didn't
repair my corrupted file.

What are the reasons caused this problem? I would like to know is what can I
do to resolve this problem? Is XFS able to recover my file with its right
content, at least a consistent content? 

     I'm running a 2.6.24 kernel with xfsprogs-3.1.1, and the operating
system is RedHat Enterprise Linux 5 Server (64-bit).

     Do you know if this problem has often been encountered, there is a way
to proceed? 

Thank you !

Adamant

-- 
View this message in context: http://old.nabble.com/power-failure-causes-files-loss-tp28340118p28340118.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: power failure causes files loss

[Eric Sandeen]

adamantzj wrote:
> Hello,
> 
>      I’m writing data on a server in my company, but power cut suddenly. I
> had to restart the machine, then I found that many files in the working
> directory had been lost. I made some tries with xfs_repair and it didn't
> repair my corrupted file.
> 
> What are the reasons caused this problem? I would like to know is what can I
> do to resolve this problem? Is XFS able to recover my file with its right
> content, at least a consistent content? 

Well, anything written via buffered IO but not synced could be lost, in general.
This is true for any filesystem.

Also, depending on your storage, if barriers were not enabled on something with
a volatile write cache, this can cause problems on a power loss.  This would be
the case on lvm on rhel5 for example (if you had write caches on your drives).

You didn't say if repair found any problems, or if the filesystem had trouble
mounting post-loss, etc... this would be useful info.

But in general, unsynced buffered data -will- be lost on a power loss, for any
fs.

-Eric

>      I'm running a 2.6.24 kernel with xfsprogs-3.1.1, and the operating
> system is RedHat Enterprise Linux 5 Server (64-bit).
> 
>      Do you know if this problem has often been encountered, there is a way
> to proceed? 
> 
> Thank you !
> 
> Adamant
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: power failure causes files loss

[Stan Hoeppner]

adamantzj put forth on 4/23/2010 6:28 AM:
> 
> Hello,
> 
>      I’m writing data on a server in my company, but power cut suddenly. I
> had to restart the machine, then I found that many files in the working
> directory had been lost. I made some tries with xfs_repair and it didn't
> repair my corrupted file.
> 
> What are the reasons caused this problem? I would like to know is what can I
> do to resolve this problem? Is XFS able to recover my file with its right
> content, at least a consistent content? 
> 
>      I'm running a 2.6.24 kernel with xfsprogs-3.1.1, and the operating
> system is RedHat Enterprise Linux 5 Server (64-bit).
> 
>      Do you know if this problem has often been encountered, there is a way
> to proceed? 
> 
> Thank you !
> 
> Adamant

XFS isn't the cause of your problem.  Running without a proper UPS is the
cause of your problem.  Fix the cause of your problem by purchasing a
properly sized and instrumented/monitored UPS.  YOU CANNOT rely on your OS
or filesystem driver to prevent data loss due to power loss, or to properly
recover from such unexpected power loss if/when it occurs.  This is why you
absolutely need to eliminate the possibility of unexpected power loss.  This
is the sole purpose for the existence of companies such as APC, Liebert,
TrippLite and others.

I'd have thought by 2010 that the entire IT cadre worldwide would already be
educated on proper power backup planning and procedures.  I guess not.

Acquire a proper UPS and configure it properly, specifically meaning
monitoring and alerts which will allow for performing a proper shutdown
during an extended outage, and your problems will disappear.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: power failure causes files loss

[Hans-Peter Jansen]

On Saturday 24 April 2010, 00:25:41 Stan Hoeppner wrote:
> adamantzj put forth on 4/23/2010 6:28 AM:
> > Hello,
> >
> >      I’m writing data on a server in my company, but power cut
> > suddenly. I had to restart the machine, then I found that many files in
> > the working directory had been lost. I made some tries with xfs_repair
> > and it didn't repair my corrupted file.
> >
> > What are the reasons caused this problem? I would like to know is what
> > can I do to resolve this problem? Is XFS able to recover my file with
> > its right content, at least a consistent content?
> >
> >      I'm running a 2.6.24 kernel with xfsprogs-3.1.1, and the operating
> > system is RedHat Enterprise Linux 5 Server (64-bit).
> >
> >      Do you know if this problem has often been encountered, there is a
> > way to proceed?
> >
> > Thank you !
> >
> > Adamant
>
> XFS isn't the cause of your problem.  Running without a proper UPS is the
> cause of your problem.  Fix the cause of your problem by purchasing a
> properly sized and instrumented/monitored UPS.  YOU CANNOT rely on your
> OS or filesystem driver to prevent data loss due to power loss, or to
> properly recover from such unexpected power loss if/when it occurs.  This
> is why you absolutely need to eliminate the possibility of unexpected
> power loss.  This is the sole purpose for the existence of companies such
> as APC, Liebert, TrippLite and others.
>
> I'd have thought by 2010 that the entire IT cadre worldwide would already
> be educated on proper power backup planning and procedures.  I guess not.
>
> Acquire a proper UPS and configure it properly, specifically meaning
> monitoring and alerts which will allow for performing a proper shutdown
> during an extended outage, and your problems will disappear.

While you're right for the general case of course, there are many other 
failure conditions, that an UPS will not prevent, eg. kernel crashes. 
Therefore there is an vivid interest, that any storage based filesystems  
should stay consistent _any_ time (and usually filesystem developers spend 
an considerable amount of their time on fixing such problems, if they 
arise). 

Adamant, the people reading here are rarely trained in telepathic   
capabilities. If you need more specific answers, you need to provide more 
specific details, eg. mount options, error logs, etc.

Pete

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: power failure causes files loss

[Eric Sandeen]

Hans-Peter Jansen wrote:
> On Saturday 24 April 2010, 00:25:41 Stan Hoeppner wrote:
>> adamantzj put forth on 4/23/2010 6:28 AM:
>>> Hello,
>>>
>>>      I’m writing data on a server in my company, but power cut
>>> suddenly. I had to restart the machine, then I found that many files in
>>> the working directory had been lost. I made some tries with xfs_repair
>>> and it didn't repair my corrupted file.
>>>
>>> What are the reasons caused this problem? I would like to know is what
>>> can I do to resolve this problem? Is XFS able to recover my file with
>>> its right content, at least a consistent content?
>>>
>>>      I'm running a 2.6.24 kernel with xfsprogs-3.1.1, and the operating
>>> system is RedHat Enterprise Linux 5 Server (64-bit).
>>>
>>>      Do you know if this problem has often been encountered, there is a
>>> way to proceed?
>>>
>>> Thank you !
>>>
>>> Adamant
>> XFS isn't the cause of your problem.  Running without a proper UPS is the
>> cause of your problem.  Fix the cause of your problem by purchasing a
>> properly sized and instrumented/monitored UPS.  YOU CANNOT rely on your
>> OS or filesystem driver to prevent data loss due to power loss, or to
>> properly recover from such unexpected power loss if/when it occurs.  This
>> is why you absolutely need to eliminate the possibility of unexpected
>> power loss.  This is the sole purpose for the existence of companies such
>> as APC, Liebert, TrippLite and others.
>>
>> I'd have thought by 2010 that the entire IT cadre worldwide would already
>> be educated on proper power backup planning and procedures.  I guess not.
>>
>> Acquire a proper UPS and configure it properly, specifically meaning
>> monitoring and alerts which will allow for performing a proper shutdown
>> during an extended outage, and your problems will disappear.
> 
> While you're right for the general case of course, there are many other 
> failure conditions, that an UPS will not prevent, eg. kernel crashes. 
> Therefore there is an vivid interest, that any storage based filesystems  
> should stay consistent _any_ time (and usually filesystem developers spend 
> an considerable amount of their time on fixing such problems, if they 
> arise). 
> 
> Adamant, the people reading here are rarely trained in telepathic   
> capabilities. If you need more specific answers, you need to provide more 
> specific details, eg. mount options, error logs, etc.
> 
> Pete

See also:

"Eat My Data: How Everybody Gets File IO Wrong"
http://www.flamingspork.com/talks/

:)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: power failure causes files loss

[Emmanuel Florac]

Le Sat, 24 Apr 2010 19:05:43 +0200 vous écriviez:

> that any storage based filesystems  
> should stay consistent _any_ time

AFAIK only FreeBSD UFS-with soft-updates actually comes close to that,
at the huge cost of being impossibly complex :)

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: power failure causes files loss

[Peter Grandi]

[ ... usual call for 'O_PONIES' ... ]

> While you're right for the general case of course, there are
> many other failure conditions, that an UPS will not prevent,
> eg. kernel crashes.

> Therefore there is an vivid interest, that any storage based
> filesystems should stay consistent _any_ time (and usually
> filesystem developers spend an considerable amount of their
> time on fixing such problems, if they arise).

That's strictly speaking impossible, almost always undesirable,
and at the same time easy to achieve: if you want something like
that just mount with '-o sync' :-).

If you want that *and* excellent performance, just mount with
'-o ponies' instead :-).

If you want that *and* excellent performance *and* scalable
performance with small IO just mount with '-o unicorns' :-).

This topic has been already discussed a few times over in this
mailing list, and in the kernel mailing etc.; those who can get
it have gotten it, those who don't continue to whine.

The real problem is not so much the '-o ponies' (or '-o unicorns')
attitude, it is that even those who get it still have a hard time
with doing the right things because of limited or broken behaviour,
at all levels (from the POSIX semantics of 'fsync' to the Linux
page cache to the firmware of many drives).

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs