Write barriers and hardware RAID

Write barriers and hardware RAID

[J Pälve]

Hi,

I'm setting up a server to provide storage for a couple of VMware ESXi 
servers. I'm using the latest stable Debian and I'm leaning towards using 
NFS with XFS. However, I'm concerned about data integrity in the event of 
power-out (we have UPS but they only last so long). Here are the specific 
questions I have:

- The XFS FAQ states that with battery backup'd RAID hardware, both write 
barriers and individual disk cache should be turned off. However, I'm 
getting better benchmark results with both turned on. What I'm wondering 
is, will write barriers work as intended when used with hardware RAID 
controller (PERC 6/E)? Googling only turned up results relating to 
software RAID.

- The XFS FAQ also states that virtualization products prevent write 
barriers from working correctly. Is this still the case (specifically with 
ESXi 3.5 and later) and is there anything that can be done about it? Does 
VMFS somehow work around this, or is the problem then just "out of sight, 
out of mind"?

Bregs,
   JPa

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Write barriers and hardware RAID

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 2704 bytes --]

I wrote that sections of the FAQ, so I should answer:

On Freitag 17 Juli 2009 J Pälve wrote:
> - The XFS FAQ states that with battery backup'd RAID hardware, both
> write barriers and individual disk cache should be turned off.
> However, I'm getting better benchmark results with both turned on.

I guess it's only the "hard disk cache" turned on leading to better 
performance. But that is a very, very dangerous setup: If you use a RAID 
with 16 hard disks, each having 32MB cache, on a power fail you can 
loose up to 16*32 = 512MB of data, as on a power outage hard disks 
simply drop their caches. And chances are *very* big that a significant 
amount of filesystem metadata is in there, trashing your filesystem 
badly. Never turn this on if you care about your data.

For write barriers, the performance should be a little bit lower if ON 
instead OFF.

> What I'm wondering is, will write barriers work as intended when used
> with hardware RAID controller (PERC 6/E)? Googling only turned up
> results relating to software RAID.

No. RAID controllers simulate written data by telling the host that a 
disk block has been written while it's only in the controller's cache. 
The controller will write it later, when he has time. So basically 
barriers only generate extra I/O there. This is valid if the controller 
has writes "write back". If set to "write through", the RAID controller 
simply does not cache writes, and directly writes them to disk, and only 
afterwards tell the host that data has been written. This will drop your 
write performance very significantly, on a server with much I/O you 
don't want to use write through (aka write cache off).

> - The XFS FAQ also states that virtualization products prevent write
> barriers from working correctly. Is this still the case (specifically
> with ESXi 3.5 and later) and is there anything that can be done about
> it? Does VMFS somehow work around this, or is the problem then just
> "out of sight, out of mind"?

I found an entry for the ".vmx" config file:
scsi0:0.writeThrough = "TRUE"

That should do the desired "do not cache this disk", but I didn't test 
it so far.
I wonder if someone knows of such a setting for XenServer?

If someone has a solution to "VM disk writes cached", I'd be happy to 
hear how to do that.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Write barriers and hardware RAID

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 2158 bytes --]

Please keep the discussion on the list also for others who search the 
same info later.

On Dienstag 21 Juli 2009 J Pälve wrote:
> Thank you for your clarification. I'm still wondering about this
> mention in the FAQ:
>
> "With a RAID controller with battery backed controller cache and
> cache in write back mode, you should turn off barriers - they are
> unnecessary in this case, and if the controller honors the cache
> flushes, it will be harmful to performance."
>
> If you issue a cache flush request to one of these controllers that
> honor them, will it only flush its own cache or also issue a cache
> flush request to the disks? In the latter case, wouldn't write
> barriers work correctly even with both controller and disk cache
> enabled?

If you use a RAID controller but then it allows cache flushes from the 
host to really flush it's cache, it's basically the same as setting the 
whole controller to "write through" mode (where he doesn't buffer 
anything). Your expensive RAID controller then acts like a dump switch 
and your only advantage is that you can connect several disks with 
parity. But your performance will be a mess. I can't believe anybody 
wanting performance will do that.
( We have some "up to 4 people" companies with HP servers where the 3-
disk RAID-5 is setup this way. As long as they only store word documents 
and such it's enough. We setup the whole controller write through then, 
battery backup is of course not needed. But disk write cache is always 
set to off, to prevent data loss on power fail.)

Back to your question: If there exists a hardware RAID controller 
honoring flushes, it should also flush the disks (I'd expect that). But 
I don't know any controller to really do that. If you do, please tell 
me.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs