KVM and kernel 2.6.30 file system madness

[Robert Wimmer <r.wimmer@tomorrow-focus.de>]

Hi there,

back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to
crash from time to time. That time we used XFS as filesystem (/ and /boot
where ext3/ext2). Since XFS worked so very well for us on physical
hosts the natural choise for our OSs in KVM of course was also XFS.
This was a bad idea because it caused some filesystem corruptions
on some KVMs when KVM crashed (without any message).
Somewhere I read that XFS in KVM should only be used with the
KVM parameter "cache=none". Since then this is now our default
for all KVMs (even with ext3). I thought by myself that KVM and an FS which
does heavy write caching like XFS is a bad choise so I decided that I can't
trust XFS inside a KVM anymore and so I switched all filesystems
in our KVMs to ext3. This was a good choice. No FS corruptions
anymore - well and no unplaned crashes of of KVM too ;-)
Since yesterday (no crash but FS corruptions)...

I installed kernel 2.6.30-r2 in one of our guests. This was a not so
good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5
and KVM is 84 (KVM 85 has issues with VNC display and 86 and
87 not in portage currently). Using qow2 as KVM image format.

I installed all the stuff we needed in the new KVM and a Postgres
database. But something was different. The database import was
suddenly fast as hell. I've never seen such good I/O throughput
in a KVM. Well after almost finished with the whole installation
process I noticed some strange ext3 messages in the "dmesg"
output. "Oh no... Not again problems with FS corruptions" I thought...
Well after a reboot of the KVM it was sure that the rootfs was
corrupted. /etc/hostname and some other files suddenly were
binary files :-( Lukely I was able to correct the problems with
fsck and get the files back from the backup.

So what happend in 2.6.30? Ah... I remembered immediately that
the kernel developers decided to switch the default value of the
journaling mode (data=...) from "ordered" to "writeback". Well...
Now I know why the database import was so fast... But at what
price? I'm really curious what happens when the major distributions
roll out their distributions with this default option.

So my question is: I'm the only one in the universe with this
FS problems? Am I completely wrong here? Is "data=ordered"
the recommended mode for ext3 in KVMs and even necessary
when KVM ist not crashing?  This kind of stuff sometimes makes
live to so easy... ;-)

Thanks for any hints and comments!
Robert