XFS and XEN

XFS and XEN

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 906 bytes --]

Now I'm sure that the following configuration can destroy a PostgreSQL 
8.3.5 database:

* Linux host with XEN, XFS filesystem with "nobarrier", RAID controller 
with battery backed cache.
* XEN vm with XFS filesystem with "nobarrier" with postgresql
* your daughter with 3.5 years switching off the power supply of the 
server

I guess one shouldn't use "nobarrier" on a XEN XFS domU machine.
Would that help? I'd like to enter this in the FAQ, as people (including 
me) will be very upset to loose data, or destroy databases.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4






[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Christoph Hellwig]

On Tue, Feb 17, 2009 at 09:59:54AM +0100, Michael Monnerie wrote:
> Now I'm sure that the following configuration can destroy a PostgreSQL 
> 8.3.5 database:
> 
> * Linux host with XEN, XFS filesystem with "nobarrier", RAID controller 
> with battery backed cache.
> * XEN vm with XFS filesystem with "nobarrier" with postgresql
> * your daughter with 3.5 years switching off the power supply of the 
> server
> 
> I guess one shouldn't use "nobarrier" on a XEN XFS domU machine.
> Would that help? I'd like to enter this in the FAQ, as people (including 
> me) will be very upset to loose data, or destroy databases.

I'm not even sure the Xen driver implements barriers.  I haven't done
much with Xen, but with the default kvm/qemu config I can completely
wreck ext3 and XFS filesystems.  If using these virtualization solutions
make sure you do not cache the virtual block device in the host
pagecache, and that in doubt the write cache of the disk is turned off
as barriers most likely won't work through the virtualization stack.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

"Christoph Hellwig" <hch@infradead.org> schrieb:
> I'm not even sure the Xen driver implements barriers.  I haven't done
> much with Xen, but with the default kvm/qemu config I can completely
> wreck ext3 and XFS filesystems.  If using these virtualization solutions
> make sure you do not cache the virtual block device in the host
> pagecache, and that in doubt the write cache of the disk is turned off
> as barriers most likely won't work through the virtualization stack.

How can I do that?

mfg zmi


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Christoph Hellwig]

On Tue, Feb 17, 2009 at 03:00:39PM +0100, Michael Monnerie wrote:
> "Christoph Hellwig" <hch@infradead.org> schrieb:
> > I'm not even sure the Xen driver implements barriers.  I haven't done
> > much with Xen, but with the default kvm/qemu config I can completely
> > wreck ext3 and XFS filesystems.  If using these virtualization solutions
> > make sure you do not cache the virtual block device in the host
> > pagecache, and that in doubt the write cache of the disk is turned off
> > as barriers most likely won't work through the virtualization stack.
> 
> How can I do that?

Barriers are disabled using hdparm.  To make sure qemu/kvm doesn't
cache your image file specify cache=off on the command line specifying
that virtual disk.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 1120 bytes --]

On Donnerstag 19 Februar 2009 Christoph Hellwig wrote:
> Barriers are disabled using hdparm.  To make sure qemu/kvm doesn't
> cache your image file specify cache=off on the command line
> specifying that virtual disk.

I have "nobarriers" for XFS, and a RAID controller with writeback cache. 
The "cache=off" seems to be for kvm only, so doesn't help me with XEN.

> make sure you do not cache the virtual block device in the host
> pagecache, and that in doubt the write cache of the disk is turned off
> as barriers most likely won't work through the virtualization stack.

Isn't that what "tap:aio" in XEN does?

I really want and need to find a secure way to run XEN clients. Is that 
data loss I had the fault of XFS, XEN or something else?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 2165 bytes --]

On Dienstag 17 Februar 2009 Michael Monnerie wrote:
> Now I'm sure that the following configuration can destroy a
> PostgreSQL 8.3.5 database:
>
> * Linux host with XEN, XFS filesystem with "nobarrier", RAID
> controller with battery backed cache.
> * XEN vm with XFS filesystem with "nobarrier" with postgresql
> * your daughter with 3.5 years switching off the power supply of the
> server

I got this answer on the XEN list. It looks like XFS in not very secure. 
Do you share that opinion? I am still searching for a way to make sure 
the above never happens again. Should I use ext3/reiserfs/ocfs2 within a 
XEN domU, or is it not the fault of XFS my database got destroyed? Don't 
want to blame anyone, but I need a secure environment for the machines 
that should be in production soon, don't have time to recover databases 
or whatever just because of a crash.

from XEN list:
********************************************
if you don't have highly reliable hardware, with redundant channels and 
battery backed caches, etc. stay away from XFS.

it's a pity, but XFS is only as reliable as the hardware you give it.  
in other words: you can be really sure that a software issue won't trash 
your XFS filesystem; but also that a hardware failure definitely will.

ext3, OTOH, might not have such an impressive record of correctness; but 
it's designed with redundancy in mind.  almost all vital structures are 
either replicated or reconstructable from other hints.  that makes fsck 
slow but effective, and in most cases, the journal analysis done when 
remounting after an unclean shutdown is enough to put it back on track.

i sure wish i could use XFS in more circumstances, but the needed 
hardware is prohibitive.
********************************************

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Christoph Hellwig]

On Tue, Feb 24, 2009 at 04:04:24PM +0100, Michael Monnerie wrote:
> if you don't have highly reliable hardware, with redundant channels and 
> battery backed caches, etc. stay away from XFS.
> 
> it's a pity, but XFS is only as reliable as the hardware you give it.  
> in other words: you can be really sure that a software issue won't trash 
> your XFS filesystem; but also that a hardware failure definitely will.
> 
> ext3, OTOH, might not have such an impressive record of correctness; but 
> it's designed with redundancy in mind.  almost all vital structures are 
> either replicated or reconstructable from other hints.  that makes fsck 
> slow but effective, and in most cases, the journal analysis done when 
> remounting after an unclean shutdown is enough to put it back on track.
> 
> i sure wish i could use XFS in more circumstances, but the needed 
> hardware is prohibitive.

It's the usual BS.  The difference is just that you actually see the
corruption on XFS while it's pretty silent on extN.  If your Hardware
(or Hypervisor) is not reliable you _will_ lose data.  Either silently
or with a spectacular blowup if the filesystem actually has consistency
checking (which XFS has a lot).

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 1523 bytes --]

On Dienstag 24 Februar 2009 Christoph Hellwig wrote:
> It's the usual BS.  The difference is just that you actually see the
> corruption on XFS while it's pretty silent on extN.  If your Hardware
> (or Hypervisor) is not reliable you _will_ lose data.  Either
> silently or with a spectacular blowup if the filesystem actually has
> consistency checking (which XFS has a lot).

Thank you for the explanation. So to clear up: It was not XFS's fault, 
but came from XEN? Can I write it like that on the FAQ?:

Q: Which settings are best with virtualization like VMware, XEN, qemu?

The biggest problem is that those products seem to also virtualize disk 
writes in a way that even barriers don't work anymore, which means even 
a fsync is not reliable. Tests confirm that unplugging the power from 
such a system even with RAID controller with battery backed cache and 
hard disk cache turned off (which is save on a normal host) you can 
destroy a database within the virtual machine (client, domU whatever you 
call it).

In qemu you can specify cache=off on the line specifying the virtual 
disk. For others I have no information what to do.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

On Dienstag 24 Februar 2009 Christoph Hellwig wrote:
> The difference is just that you actually see the
> corruption on XFS while it's pretty silent on extN.  If your Hardware
> (or Hypervisor) is not reliable you _will_ lose data.  Either
> silently or with a spectacular blowup if the filesystem actually has
> consistency checking (which XFS has a lot).

One more question: My hardware should be reliable: RAID controller, 
battery backed cache, disk write cache=off. So even in the event of a 
power fail, nothing should happen. The controller should write open 
blocks after reboot.

But it doesn't. I've retested: power off. On bootup, the XEN domU 
PostgreSQL reported errors again. This time of course I had a good 
backup from a minute before. But still: Shouldn't it be impossible for 
such an error to happen, as my hardware shouldn't eat any data? I'm 
trying to find out where the DB gets destroyed. Is it that "just" 
breaking within a transaction where XEN doesn't correctly fsync is 
enough, despite all hardware else configured well?

Damn, that's a complicated world.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

I just got this info at XEN:
> LVM does not honor write-barriers.

Then I searched and found a thread were Eric also talked:
http://lkml.org/lkml/2008/5/16/390
http://hightechsorcery.com/2008/06/linux-write-barriers-write-caching-
lvm-and-filesystems

If I understand correctly, turning off disk cache write cache should be 
enough to be save, even when using LVM. Is it really XEN that messed the 
disk, or something else? Could the Areca controller driver do something 
wrong? I'd really love to get a stable and data-secure system, as you 
might understand.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS and XEN

[Michael Monnerie]

On Mittwoch 25 Februar 2009 Michael Monnerie wrote:
> Q: Which settings are best with virtualization like VMware, XEN,
> qemu?
>
> The biggest problem is that those products seem to also virtualize
> disk writes in a way that even barriers don't work anymore, which
> means even a fsync is not reliable. Tests confirm that unplugging the
> power from such a system even with RAID controller with battery
> backed cache and hard disk cache turned off (which is save on a
> normal host) you can destroy a database within the virtual machine
> (client, domU whatever you call it).
>
> In qemu you can specify cache=off on the line specifying the virtual
> disk. For others I have no information what to do.

In as question 26 now
http://xfs.org/index.php/XFS_FAQ

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs