Re: KVM and kernel 2.6.30 file system madness

[Amit Shah <amit.shah@redhat.com>]

On (Thu) Jul 09 2009 [08:27:48], Robert Wimmer wrote:
> Hi there,
> 
> back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to
> crash from time to time. That time we used XFS as filesystem (/ and /boot
> where ext3/ext2). Since XFS worked so very well for us on physical
> hosts the natural choise for our OSs in KVM of course was also XFS.
> This was a bad idea because it caused some filesystem corruptions
> on some KVMs when KVM crashed (without any message).
> Somewhere I read that XFS in KVM should only be used with the
> KVM parameter "cache=none". Since then this is now our default
> for all KVMs (even with ext3). I thought by myself that KVM and an FS which
> does heavy write caching like XFS is a bad choise so I decided that I can't
> trust XFS inside a KVM anymore and so I switched all filesystems
> in our KVMs to ext3. This was a good choice. No FS corruptions
> anymore - well and no unplaned crashes of of KVM too ;-)
> Since yesterday (no crash but FS corruptions)...
> 
> I installed kernel 2.6.30-r2 in one of our guests. This was a not so
> good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5
> and KVM is 84 (KVM 85 has issues with VNC display and 86 and
> 87 not in portage currently). Using qow2 as KVM image format.
> 
> I installed all the stuff we needed in the new KVM and a Postgres
> database. But something was different. The database import was
> suddenly fast as hell. I've never seen such good I/O throughput
> in a KVM. Well after almost finished with the whole installation
> process I noticed some strange ext3 messages in the "dmesg"
> output. "Oh no... Not again problems with FS corruptions" I thought...
> Well after a reboot of the KVM it was sure that the rootfs was
> corrupted. /etc/hostname and some other files suddenly were
> binary files :-( Lukely I was able to correct the problems with
> fsck and get the files back from the backup.
> 
> So what happend in 2.6.30? Ah... I remembered immediately that
> the kernel developers decided to switch the default value of the
> journaling mode (data=...) from "ordered" to "writeback". Well...
> Now I know why the database import was so fast... But at what
> price? I'm really curious what happens when the major distributions
> roll out their distributions with this default option.

Distributions will likely change the default.

> So my question is: I'm the only one in the universe with this
> FS problems? Am I completely wrong here? Is "data=ordered"
> the recommended mode for ext3 in KVMs and even necessary
> when KVM ist not crashing?  This kind of stuff sometimes makes
> live to so easy... ;-)

Are you using virtio-block?

In any case, not using a released version always has risks.

		Amit