Re: KVM and kernel 2.6.30 file system madness

[Robert Wimmer <r.wimmer@tomorrow-focus.de>]

Hi!

> Are you using virtio-block?

Yes.

> In any case, not using a released version always has risks.

Well, what do you mean by "not using a released version"?
The package "gentoo-sources" always uses released
kernels. 2.6.30-r2 in Gentoo means that this is the third
update of the stable kernel version 2.6.30. The "-r*" releases
just contains small patches from the Gentoo kernel package
maintainers.

Thanks!
Robert


Amit Shah wrote:
> On (Thu) Jul 09 2009 [08:27:48], Robert Wimmer wrote:
>   
>> Hi there,
>>
>> back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to
>> crash from time to time. That time we used XFS as filesystem (/ and /boot
>> where ext3/ext2). Since XFS worked so very well for us on physical
>> hosts the natural choise for our OSs in KVM of course was also XFS.
>> This was a bad idea because it caused some filesystem corruptions
>> on some KVMs when KVM crashed (without any message).
>> Somewhere I read that XFS in KVM should only be used with the
>> KVM parameter "cache=none". Since then this is now our default
>> for all KVMs (even with ext3). I thought by myself that KVM and an FS which
>> does heavy write caching like XFS is a bad choise so I decided that I can't
>> trust XFS inside a KVM anymore and so I switched all filesystems
>> in our KVMs to ext3. This was a good choice. No FS corruptions
>> anymore - well and no unplaned crashes of of KVM too ;-)
>> Since yesterday (no crash but FS corruptions)...
>>
>> I installed kernel 2.6.30-r2 in one of our guests. This was a not so
>> good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5
>> and KVM is 84 (KVM 85 has issues with VNC display and 86 and
>> 87 not in portage currently). Using qow2 as KVM image format.
>>
>> I installed all the stuff we needed in the new KVM and a Postgres
>> database. But something was different. The database import was
>> suddenly fast as hell. I've never seen such good I/O throughput
>> in a KVM. Well after almost finished with the whole installation
>> process I noticed some strange ext3 messages in the "dmesg"
>> output. "Oh no... Not again problems with FS corruptions" I thought...
>> Well after a reboot of the KVM it was sure that the rootfs was
>> corrupted. /etc/hostname and some other files suddenly were
>> binary files :-( Lukely I was able to correct the problems with
>> fsck and get the files back from the backup.
>>
>> So what happend in 2.6.30? Ah... I remembered immediately that
>> the kernel developers decided to switch the default value of the
>> journaling mode (data=...) from "ordered" to "writeback". Well...
>> Now I know why the database import was so fast... But at what
>> price? I'm really curious what happens when the major distributions
>> roll out their distributions with this default option.
>>     
>
> Distributions will likely change the default.
>
>   
>> So my question is: I'm the only one in the universe with this
>> FS problems? Am I completely wrong here? Is "data=ordered"
>> the recommended mode for ext3 in KVMs and even necessary
>> when KVM ist not crashing?  This kind of stuff sometimes makes
>> live to so easy... ;-)
>>     
>
> Are you using virtio-block?
>
> In any case, not using a released version always has risks.
>
> 		Amit
>