KVM and kernel 2.6.30 file system madness

KVM and kernel 2.6.30 file system madness

[Robert Wimmer]

Hi there,

back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to
crash from time to time. That time we used XFS as filesystem (/ and /boot
where ext3/ext2). Since XFS worked so very well for us on physical
hosts the natural choise for our OSs in KVM of course was also XFS.
This was a bad idea because it caused some filesystem corruptions
on some KVMs when KVM crashed (without any message).
Somewhere I read that XFS in KVM should only be used with the
KVM parameter "cache=none". Since then this is now our default
for all KVMs (even with ext3). I thought by myself that KVM and an FS which
does heavy write caching like XFS is a bad choise so I decided that I can't
trust XFS inside a KVM anymore and so I switched all filesystems
in our KVMs to ext3. This was a good choice. No FS corruptions
anymore - well and no unplaned crashes of of KVM too ;-)
Since yesterday (no crash but FS corruptions)...

I installed kernel 2.6.30-r2 in one of our guests. This was a not so
good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5
and KVM is 84 (KVM 85 has issues with VNC display and 86 and
87 not in portage currently). Using qow2 as KVM image format.

I installed all the stuff we needed in the new KVM and a Postgres
database. But something was different. The database import was
suddenly fast as hell. I've never seen such good I/O throughput
in a KVM. Well after almost finished with the whole installation
process I noticed some strange ext3 messages in the "dmesg"
output. "Oh no... Not again problems with FS corruptions" I thought...
Well after a reboot of the KVM it was sure that the rootfs was
corrupted. /etc/hostname and some other files suddenly were
binary files :-( Lukely I was able to correct the problems with
fsck and get the files back from the backup.

So what happend in 2.6.30? Ah... I remembered immediately that
the kernel developers decided to switch the default value of the
journaling mode (data=...) from "ordered" to "writeback". Well...
Now I know why the database import was so fast... But at what
price? I'm really curious what happens when the major distributions
roll out their distributions with this default option.

So my question is: I'm the only one in the universe with this
FS problems? Am I completely wrong here? Is "data=ordered"
the recommended mode for ext3 in KVMs and even necessary
when KVM ist not crashing?  This kind of stuff sometimes makes
live to so easy... ;-)

Thanks for any hints and comments!
Robert

Re: KVM and kernel 2.6.30 file system madness

[Amit Shah]

On (Thu) Jul 09 2009 [08:27:48], Robert Wimmer wrote:
> Hi there,
> 
> back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to
> crash from time to time. That time we used XFS as filesystem (/ and /boot
> where ext3/ext2). Since XFS worked so very well for us on physical
> hosts the natural choise for our OSs in KVM of course was also XFS.
> This was a bad idea because it caused some filesystem corruptions
> on some KVMs when KVM crashed (without any message).
> Somewhere I read that XFS in KVM should only be used with the
> KVM parameter "cache=none". Since then this is now our default
> for all KVMs (even with ext3). I thought by myself that KVM and an FS which
> does heavy write caching like XFS is a bad choise so I decided that I can't
> trust XFS inside a KVM anymore and so I switched all filesystems
> in our KVMs to ext3. This was a good choice. No FS corruptions
> anymore - well and no unplaned crashes of of KVM too ;-)
> Since yesterday (no crash but FS corruptions)...
> 
> I installed kernel 2.6.30-r2 in one of our guests. This was a not so
> good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5
> and KVM is 84 (KVM 85 has issues with VNC display and 86 and
> 87 not in portage currently). Using qow2 as KVM image format.
> 
> I installed all the stuff we needed in the new KVM and a Postgres
> database. But something was different. The database import was
> suddenly fast as hell. I've never seen such good I/O throughput
> in a KVM. Well after almost finished with the whole installation
> process I noticed some strange ext3 messages in the "dmesg"
> output. "Oh no... Not again problems with FS corruptions" I thought...
> Well after a reboot of the KVM it was sure that the rootfs was
> corrupted. /etc/hostname and some other files suddenly were
> binary files :-( Lukely I was able to correct the problems with
> fsck and get the files back from the backup.
> 
> So what happend in 2.6.30? Ah... I remembered immediately that
> the kernel developers decided to switch the default value of the
> journaling mode (data=...) from "ordered" to "writeback". Well...
> Now I know why the database import was so fast... But at what
> price? I'm really curious what happens when the major distributions
> roll out their distributions with this default option.

Distributions will likely change the default.

> So my question is: I'm the only one in the universe with this
> FS problems? Am I completely wrong here? Is "data=ordered"
> the recommended mode for ext3 in KVMs and even necessary
> when KVM ist not crashing?  This kind of stuff sometimes makes
> live to so easy... ;-)

Are you using virtio-block?

In any case, not using a released version always has risks.

		Amit

Re: KVM and kernel 2.6.30 file system madness

[Robert Wimmer]

Hi!

> Are you using virtio-block?

Yes.

> In any case, not using a released version always has risks.

Well, what do you mean by "not using a released version"?
The package "gentoo-sources" always uses released
kernels. 2.6.30-r2 in Gentoo means that this is the third
update of the stable kernel version 2.6.30. The "-r*" releases
just contains small patches from the Gentoo kernel package
maintainers.

Thanks!
Robert


Amit Shah wrote:
> On (Thu) Jul 09 2009 [08:27:48], Robert Wimmer wrote:
>   
>> Hi there,
>>
>> back in days before kernel 2.6.25/2.6.26 and KVM 70-77 KVM decided to
>> crash from time to time. That time we used XFS as filesystem (/ and /boot
>> where ext3/ext2). Since XFS worked so very well for us on physical
>> hosts the natural choise for our OSs in KVM of course was also XFS.
>> This was a bad idea because it caused some filesystem corruptions
>> on some KVMs when KVM crashed (without any message).
>> Somewhere I read that XFS in KVM should only be used with the
>> KVM parameter "cache=none". Since then this is now our default
>> for all KVMs (even with ext3). I thought by myself that KVM and an FS which
>> does heavy write caching like XFS is a bad choise so I decided that I can't
>> trust XFS inside a KVM anymore and so I switched all filesystems
>> in our KVMs to ext3. This was a good choice. No FS corruptions
>> anymore - well and no unplaned crashes of of KVM too ;-)
>> Since yesterday (no crash but FS corruptions)...
>>
>> I installed kernel 2.6.30-r2 in one of our guests. This was a not so
>> good idea. All hosts and guest running Gentoo. Host kernel is 2.6.29-r5
>> and KVM is 84 (KVM 85 has issues with VNC display and 86 and
>> 87 not in portage currently). Using qow2 as KVM image format.
>>
>> I installed all the stuff we needed in the new KVM and a Postgres
>> database. But something was different. The database import was
>> suddenly fast as hell. I've never seen such good I/O throughput
>> in a KVM. Well after almost finished with the whole installation
>> process I noticed some strange ext3 messages in the "dmesg"
>> output. "Oh no... Not again problems with FS corruptions" I thought...
>> Well after a reboot of the KVM it was sure that the rootfs was
>> corrupted. /etc/hostname and some other files suddenly were
>> binary files :-( Lukely I was able to correct the problems with
>> fsck and get the files back from the backup.
>>
>> So what happend in 2.6.30? Ah... I remembered immediately that
>> the kernel developers decided to switch the default value of the
>> journaling mode (data=...) from "ordered" to "writeback". Well...
>> Now I know why the database import was so fast... But at what
>> price? I'm really curious what happens when the major distributions
>> roll out their distributions with this default option.
>>     
>
> Distributions will likely change the default.
>
>   
>> So my question is: I'm the only one in the universe with this
>> FS problems? Am I completely wrong here? Is "data=ordered"
>> the recommended mode for ext3 in KVMs and even necessary
>> when KVM ist not crashing?  This kind of stuff sometimes makes
>> live to so easy... ;-)
>>     
>
> Are you using virtio-block?
>
> In any case, not using a released version always has risks.
>
> 		Amit
>   

Re: KVM and kernel 2.6.30 file system madness

[Amit Shah]

On (Wed) Jul 15 2009 [09:52:36], Robert Wimmer wrote:
> Hi!
> 
> > Are you using virtio-block?
> 
> Yes.

OK, then there is a known problem. I think the fix is waiting to be
applied.

> > In any case, not using a released version always has risks.
> 
> Well, what do you mean by "not using a released version"?
> The package "gentoo-sources" always uses released
> kernels. 2.6.30-r2 in Gentoo means that this is the third
> update of the stable kernel version 2.6.30. The "-r*" releases
> just contains small patches from the Gentoo kernel package
> maintainers.

Nevermind; I thought it was a -rc2 release you're working with.

		Amit

Re: KVM and kernel 2.6.30 file system madness

[Mark van Walraven]

On Wed, Jul 15, 2009 at 02:33:03PM +0530, Amit Shah wrote:
> On (Wed) Jul 15 2009 [09:52:36], Robert Wimmer wrote:
> > Hi!
> > 
> > > Are you using virtio-block?
> > 
> > Yes.
> 
> OK, then there is a known problem. I think the fix is waiting to be
> applied.

Amit, would you kindly state the problem with virtio-block?

Thanks,

Mark.

Re: KVM and kernel 2.6.30 file system madness

[Amit Shah]

On (Tue) Jul 21 2009 [16:42:50], Mark van Walraven wrote:
> On Wed, Jul 15, 2009 at 02:33:03PM +0530, Amit Shah wrote:
> > On (Wed) Jul 15 2009 [09:52:36], Robert Wimmer wrote:
> > > Hi!
> > > 
> > > > Are you using virtio-block?
> > > 
> > > Yes.
> > 
> > OK, then there is a known problem. I think the fix is waiting to be
> > applied.
> 
> Amit, would you kindly state the problem with virtio-block?

I think it was this one: upstream commit

4eff3cae9c9809720c636e64bc72f212258e0bd5
virtio_blk: don't bounce highmem requests

		Amit