Need Warm Fuzzies, ReiserFS (3)

Need Warm Fuzzies, ReiserFS (3)

[Jim Miller]

Hi everyone,

I have recently been burned by the XFS file system -- After a recent 
system crash/hang we experienced a lot of file corruption and needed to 
restore from a backup that was ~12hrs old.  It seems that XFS keeps a 
lot of journal info in memory and a sudden system crash (hang) prevents 
it from writing out the journal. 

We run a few game servers that have a large number of large and small 
files that are actively accessed (and restoring from a backup 12hrs old 
made a lot of our gamers very unhappy).  We were running EXT3 (which in 
the past recovered nicely from hangs/crashes) but performance was so bad 
we needed a new FS so 6mos ago a decision was made to go with XFS. 
I would like to switch to ReiserFS (v3) (I understand Reiser4 isn't 
quite ready for production use) and was hoping to get the warm fuzzies 
about making this decision.  I know it's much faster than ext3 but at 
this point I need to feel good about it's ability to recover from a 
sudden system hang/crash/reset.


Thanks,
Jim

Re: Need Warm Fuzzies, ReiserFS (3)

[Christian Mayrhuber]

On Monday 20 December 2004 06:41, Jim Miller wrote:
> Hi everyone,
> 
> I have recently been burned by the XFS file system -- After a recent 
> system crash/hang we experienced a lot of file corruption and needed to 
> restore from a backup that was ~12hrs old.  It seems that XFS keeps a 
> lot of journal info in memory and a sudden system crash (hang) prevents 
> it from writing out the journal. 
> 
> We run a few game servers that have a large number of large and small 
> files that are actively accessed (and restoring from a backup 12hrs old 
> made a lot of our gamers very unhappy).  We were running EXT3 (which in 
> the past recovered nicely from hangs/crashes) but performance was so bad 
> we needed a new FS so 6mos ago a decision was made to go with XFS. 
> I would like to switch to ReiserFS (v3) (I understand Reiser4 isn't 
> quite ready for production use) and was hoping to get the warm fuzzies 
> about making this decision.  I know it's much faster than ext3 but at 
> this point I need to feel good about it's ability to recover from a 
> sudden system hang/crash/reset.
> 
> 
> Thanks,
> Jim

My experience of Reiserfs recovery is good. I had once a HW RAID5 array
with a defective controller unable to detect harddisk errors. Though two
of the five disks failed I was able to recover around 80% of the mail
folders.

However, there were cases on this list where reiserfsck had problems and
namesys needed a metadata dump to fix reiserfsck. I guess most problems
it had are fixed now. Note: always use the newest reiserfsck version from 
namesys :-)

If you need good performance and data integrity I'd go with the following
mount options for drives at controllers
1) with battery backup: rw,noatime,nodiratime,notail
2) without battery backup: rw,noatime,nodiratime,notail,barrier=flush
and a 2.6.9 kernel.

The barrier mount option should provide protection against a corrupted
journal during power failure for drives with write caching enabled.
(Mostly IDE)

-- 
lg, Chris

Re: Need Warm Fuzzies, ReiserFS (3)

[Tom Vier]

On Mon, Dec 20, 2004 at 01:31:25PM +0100, Christian Mayrhuber wrote:
> The barrier mount option should provide protection against a corrupted
> journal during power failure for drives with write caching enabled.
> (Mostly IDE)

There's a mount option for write barriers? It should be enabled by default.
Otherwise, that's asking for tons of users with corruption. 

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Need Warm Fuzzies, ReiserFS (3)

[Christian Mayrhuber]

On Monday 20 December 2004 16:38, Tom Vier wrote:
> On Mon, Dec 20, 2004 at 01:31:25PM +0100, Christian Mayrhuber wrote:
> > The barrier mount option should provide protection against a corrupted
> > journal during power failure for drives with write caching enabled.
> > (Mostly IDE)
> 
> There's a mount option for write barriers? It should be enabled by default.
> Otherwise, that's asking for tons of users with corruption. 
http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.9

Before disk barriers you had to disable harddisk write caching to prevent
journal corruption. This is what nearly no one did and blamed the
filesystems instead.

Disk barriers are only in since 2.6.9, so maybe these get turned on per 
default after some more testing. That did happen to the data=ordered journal 
mode.

-- 
lg, Chris

Re: Need Warm Fuzzies, ReiserFS (3)

[Spam]

  

> On Monday 20 December 2004 16:38, Tom Vier wrote:
>> On Mon, Dec 20, 2004 at 01:31:25PM +0100, Christian Mayrhuber wrote:
>> > The barrier mount option should provide protection against a corrupted
>> > journal during power failure for drives with write caching enabled.
>> > (Mostly IDE)
>> 
>> There's a mount option for write barriers? It should be enabled by default.
>> Otherwise, that's asking for tons of users with corruption. 
> http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.9

> Before disk barriers you had to disable harddisk write caching to prevent
> journal corruption. This is what nearly no one did and blamed the
> filesystems instead.

> Disk barriers are only in since 2.6.9, so maybe these get turned on per
> default after some more testing. That did happen to the data=ordered journal
> mode.

  What happen with the performance when these barriers are active?

  Is it only during power failure the data in the write cache is lost?
  
  Also, does anyone know if the data in the disk write cache is
  written out if the system crashes/freezes. Power failure is not a
  very common occurrence, but crashes are much more so.
  
  Is it possible to detect if filesystem was unmounted due to power
  failure or due to a kernel crash?

  ~S


-- 

Re: Need Warm Fuzzies, ReiserFS (3)

[Christian Mayrhuber]

On Monday 20 December 2004 17:32, Spam wrote:

>   What happen with the performance when these barriers are active?
It's faster than with disk writecache off.
I didn't benchmark barriers=flush and writecache on.

> 
>   Is it only during power failure the data in the write cache is lost?
No. See http://support.microsoft.com/kb/153296/EN-US/

>   
>   Also, does anyone know if the data in the disk write cache is
>   written out if the system crashes/freezes. Power failure is not a
>   very common occurrence, but crashes are much more so.
The cache should stay intact as long the harddisk is supplied with power.

>   
>   Is it possible to detect if filesystem was unmounted due to power
>   failure or due to a kernel crash?
Kernel panics don't tend to reset the machine but output a dump to the 
console /syslog via network and halt the machine.
If you have a UPS supplier supporting linux you will receive a power failure
notification event to take proper action.
For example: http://www2.apcupsd.com/

-- 
lg, Chris

Re: Need Warm Fuzzies, ReiserFS (3)

[Spam]

  

> On Monday 20 December 2004 17:32, Spam wrote:

>>   What happen with the performance when these barriers are active?
> It's faster than with disk writecache off.
> I didn't benchmark barriers=flush and writecache on.

>> 
>>   Is it only during power failure the data in the write cache is lost?
> No. See http://support.microsoft.com/kb/153296/EN-US/

  I think the answer to my question was yes. That link is about disk
  cache not flushed properly before Windows shuts the computer off.

>>   
>>   Also, does anyone know if the data in the disk write cache is
>>   written out if the system crashes/freezes. Power failure is not a
>>   very common occurrence, but crashes are much more so.
> The cache should stay intact as long the harddisk is supplied with power.

  So, the question remains. Will the drive flush its cache if it still
  has power or does it need to receive some kind of flush command from
  the OS? If not, then a OS crash will not harm the integrity of the
  data in the cache and the disk will flush it out.

>>   
>>   Is it possible to detect if filesystem was unmounted due to power
>>   failure or due to a kernel crash?
> Kernel panics don't tend to reset the machine but output a dump to the
> console /syslog via network and halt the machine.
> If you have a UPS supplier supporting linux you will receive a power failure
> notification event to take proper action.
> For example: http://www2.apcupsd.com/

  No, I meant through software. So that fsck could detect if system
  lost power and therefore make a more thorough test.

  If you have UPS then write-cache should never be dangerous?

  ~S


-- 

Re: Need Warm Fuzzies, ReiserFS (3)

[Christian Mayrhuber]

On Monday 20 December 2004 21:31, Spam wrote:
> 
> > On Monday 20 December 2004 17:32, Spam wrote:
> 
> >>   What happen with the performance when these barriers are active?
> > It's faster than with disk writecache off.
> > I didn't benchmark barriers=flush and writecache on.
> 
> >> 
> >>   Is it only during power failure the data in the write cache is lost?
> > No. See http://support.microsoft.com/kb/153296/EN-US/
> 
>   I think the answer to my question was yes. That link is about disk
>   cache not flushed properly before Windows shuts the computer off.
If the cache is not flushed prior to power off IDE drives may eat the cached 
data.

> 
> >>   
> >>   Also, does anyone know if the data in the disk write cache is
> >>   written out if the system crashes/freezes. Power failure is not a
> >>   very common occurrence, but crashes are much more so.
> > The cache should stay intact as long the harddisk is supplied with power.
> 
>   So, the question remains. Will the drive flush its cache if it still
>   has power or does it need to receive some kind of flush command from
>   the OS? If not, then a OS crash will not harm the integrity of the
>   data in the cache and the disk will flush it out.
The time of flush depends on the implementation of the drives cache flush 
algorithm. If you want to be sure that everything gets written then the OS 
has to send a flush command.
An OS crash does not eat the data in the drive cache.

> >>   Is it possible to detect if filesystem was unmounted due to power
> >>   failure or due to a kernel crash?
> > Kernel panics don't tend to reset the machine but output a dump to the
> > console /syslog via network and halt the machine.
> > If you have a UPS supplier supporting linux you will receive a power 
failure
> > notification event to take proper action.
> > For example: http://www2.apcupsd.com/
> 
>   No, I meant through software. So that fsck could detect if system
>   lost power and therefore make a more thorough test.
I don't think there is a possibility to detect if the platform doesn't
send a power failure event. I think the Intel Platform Monitoring Interface 
(IPMI) can do that. Maybe there is enough time for a flush till the power
wents off.

> 
>   If you have UPS then write-cache should never be dangerous?
If you perform a clean shutdown a power failure shouldn't.

-- 
lg, Chris

Re: Need Warm Fuzzies, ReiserFS (3)

[Tom Vier]

On Mon, Dec 20, 2004 at 09:31:10PM +0100, Spam wrote:
>   If you have UPS then write-cache should never be dangerous?

It's not AS dangerous. You could still lose a psu or someone could trip over
the power cord from the computer to the ups.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Need Warm Fuzzies, ReiserFS (3)

[Spam]

  

> On Mon, Dec 20, 2004 at 09:31:10PM +0100, Spam wrote:
>>   If you have UPS then write-cache should never be dangerous?

> It's not AS dangerous. You could still lose a psu or someone could trip over
> the power cord from the computer to the ups.

  Indeed. For a server then I would not run with write-cache enabled
  anyway. But for a home desktop etc I would run with it enabled.

  Most people with Win2000/XP have write-cache enabled. Yet there are
  rarely problems due to power failures?

´

-- 

Re: Need Warm Fuzzies, ReiserFS (3)

[Tom Vier]

On Mon, Dec 20, 2004 at 10:34:29PM +0100, Spam wrote:
>   Most people with Win2000/XP have write-cache enabled. Yet there are
>   rarely problems due to power failures?

fwih (from netbsd tech-kern) windows issues cache flushes. I don't know how
people know that, though.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Need Warm Fuzzies, ReiserFS (3)

[Tom Vier]

On Mon, Dec 20, 2004 at 05:25:03PM +0100, Christian Mayrhuber wrote:
> Disk barriers are only in since 2.6.9, so maybe these get turned on per 
> default after some more testing. That did happen to the data=ordered journal 
> mode.

I was under the impression that they were on by default. I wish this was
documented somewhere; it's very important and i bet a LOT of sysadmins don't
even consider the problems of having write-back caching enabled. At least
most scsi drives come with it disabled by default. There's only wb support
for ata and sata, fwih, too.

A quick grep didn't help me find if there's any other barrier mode other
than flush. It looked like reiserfs had support specifically for flush mode.
I would have thought it'd be done just as a call to something like blkwb().



What about software raid? Are the barriers pushed through the raid layer?



There's been some interesting discussion about this subject on the netbsd
tech-kern list, btw.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Need Warm Fuzzies, ReiserFS (3)

[Christian Mayrhuber]

On Monday 20 December 2004 20:23, Tom Vier wrote:
> I was under the impression that they were on by default. I wish this was
> documented somewhere; it's very important and i bet a LOT of sysadmins don't
> even consider the problems of having write-back caching enabled. At least
> most scsi drives come with it disabled by default. There's only wb support
> for ata and sata, fwih, too.
> 
> A quick grep didn't help me find if there's any other barrier mode other
> than flush. It looked like reiserfs had support specifically for flush mode.
> I would have thought it'd be done just as a call to something like blkwb().
> 
> 
> 
> What about software raid? Are the barriers pushed through the raid layer?

It seems so. Have a look at the kernel 2.6.9 changelog.

-- 
lg, Chris

Re: Need Warm Fuzzies, ReiserFS (3)

[Hans Reiser]

Jim Miller wrote:

> Hi everyone,
>
> I have recently been burned by the XFS file system -- After a recent 
> system crash/hang we experienced a lot of file corruption and needed 
> to restore from a backup that was ~12hrs old.  It seems that XFS keeps 
> a lot of journal info in memory and a sudden system crash (hang) 
> prevents it from writing out the journal.
> We run a few game servers that have a large number of large and small 
> files that are actively accessed (and restoring from a backup 12hrs 
> old made a lot of our gamers very unhappy).  We were running EXT3 
> (which in the past recovered nicely from hangs/crashes) but 
> performance was so bad we needed a new FS so 6mos ago a decision was 
> made to go with XFS. I would like to switch to ReiserFS (v3) (I 
> understand Reiser4 isn't quite ready for production use) and was 
> hoping to get the warm fuzzies about making this decision.  I know 
> it's much faster than ext3 but at this point I need to feel good about 
> it's ability to recover from a sudden system hang/crash/reset.
>
>
> Thanks,
> Jim
>
>
>
>
>
>
>
>
>
>
It is probably tunable in XFS how long it keeps it in ram.