Linux RAID subsystem development
 help / color / mirror / Atom feed
* Suspected corruption on ACID databases due to no barrier support in ext3 on software raid-5 and hard resets
@ 2008-06-28  8:53 Leon Woestenberg
  2008-06-28 12:40 ` Jon Nelson
  0 siblings, 1 reply; 3+ messages in thread
From: Leon Woestenberg @ 2008-06-28  8:53 UTC (permalink / raw)
  To: LinuxRaid, sandeen

Hello all,

we are quite sure we are hitting data corruption on a few % of cases
on ACID* databases due to write caching enabled on drives in a
software RAID-5 configuration with ext3 in default data=ordered mode.

The machines are hard reset by a hardware watchdog when some esoteric
PCI device misbehaves.

We understand Linux software raid 5 does not pass-down barriers, is
that correct, and is this being implemented?

Also, our near-term direction of solution would be
0) disable write caches altogether, probably not feasible due to the
performance regression involved.
1) solve the misbehave (cause of reset).
2) use a shorter timed software watchdog to trigger the drives into
disabling their write caches, so that an imminent reboot has its
commits ordered.

Any other ideas?

Regards,
-- 
Leon

*http://en.wikipedia.org/wiki/ACID

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Suspected corruption on ACID databases due to no barrier support in ext3 on software raid-5 and hard resets
  2008-06-28  8:53 Suspected corruption on ACID databases due to no barrier support in ext3 on software raid-5 and hard resets Leon Woestenberg
@ 2008-06-28 12:40 ` Jon Nelson
  2008-06-29 19:44   ` Leon Woestenberg
  0 siblings, 1 reply; 3+ messages in thread
From: Jon Nelson @ 2008-06-28 12:40 UTC (permalink / raw)
  To: Leon Woestenberg; +Cc: LinuxRaid, sandeen

Because you are dealing with a database, data=ordered is not the right
mode to be using if you want the least chance of corruption.  Use
data=journal.

The problem as I understand it is that many databases use a small
number of very large files. When you are updating data in the
database, the data is not being written to new files but existing
files are being modified. If it were new files, then data=ordered
would be fine but since it is existing files being modified (in-place)
data=journal is more appropriate.

Of course, I might be wrong and would appreciate (polite) correction
but this is what I've been led to believe.


Now, due to the hardware reset, write caching on drives is a problem.
That you may have to disable - I've not found substantial performance
differences on when write caching is enabled anyway, but surely that
depends on a great many factors.


On Sat, Jun 28, 2008 at 3:53 AM, Leon Woestenberg
<leon.woestenberg@gmail.com> wrote:
> Hello all,
>
> we are quite sure we are hitting data corruption on a few % of cases
> on ACID* databases due to write caching enabled on drives in a
> software RAID-5 configuration with ext3 in default data=ordered mode.
>
> The machines are hard reset by a hardware watchdog when some esoteric
> PCI device misbehaves.
>
> We understand Linux software raid 5 does not pass-down barriers, is
> that correct, and is this being implemented?
>
> Also, our near-term direction of solution would be
> 0) disable write caches altogether, probably not feasible due to the
> performance regression involved.
> 1) solve the misbehave (cause of reset).
> 2) use a shorter timed software watchdog to trigger the drives into
> disabling their write caches, so that an imminent reboot has its
> commits ordered.

-- 
Jon

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Suspected corruption on ACID databases due to no barrier support in ext3 on software raid-5 and hard resets
  2008-06-28 12:40 ` Jon Nelson
@ 2008-06-29 19:44   ` Leon Woestenberg
  0 siblings, 0 replies; 3+ messages in thread
From: Leon Woestenberg @ 2008-06-29 19:44 UTC (permalink / raw)
  To: Jon Nelson; +Cc: LinuxRaid, sandeen

Hello Jon,

On Sat, Jun 28, 2008 at 2:40 PM, Jon Nelson
<jnelson-linux-raid@jamponi.net> wrote:
> database, the data is not being written to new files but existing
> files are being modified. If it were new files, then data=ordered
> would be fine but since it is existing files being modified (in-place)
> data=journal is more appropriate.
>
data=ordered should be fine in any case. Data is written directly to
disk (either modified data or newly allocated data, this does not
matter) and meta data goes through the journal log, written after the
data.

The problem lies in the fact that the driver can re-order the writes,
if write caching is enabled and barriers are not used (as in my case,
as md raid 5 does not support barriers).

> Of course, I might be wrong and would appreciate (polite) correction
> but this is what I've been led to believe.
>
I hope I was polite enough.

> Now, due to the hardware reset, write caching on drives is a problem.
>
Only because they re-order, not because of anything else.

> That you may have to disable - I've not found substantial performance
> differences on when write caching is enabled anyway, but surely that
> depends on a great many factors.
>
Indeed, we will be doing tests with that.

Regards,
-- 
Leon

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2008-06-29 19:44 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-06-28  8:53 Suspected corruption on ACID databases due to no barrier support in ext3 on software raid-5 and hard resets Leon Woestenberg
2008-06-28 12:40 ` Jon Nelson
2008-06-29 19:44   ` Leon Woestenberg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox