Re: writeback cache + h700 controller w/1gb nvcache, corruption on power loss

[Ron Edison <ron@idthq.com>]

Thank you very much, Stefan,

I would be very interested in how to ensure the guests are sending flushes. I'm unfamiliar with the example you gave, where is that configured? Primarily the guests are CentOS 4, 5 or 6. I am also curious if it would be advisable to switch to writethrough cache on each guest virtual disk and leave writeback enabled on the controller and if that would adversely affect performance of the guests.

The H700 controller had a charged battery. The server itself is virtually brand new in fact, no issues with it.

The disk corruption experienced was indeed lost data -- an fsck was necessary for 4 of the guests to boot at all in RW mode, they first came up read only. In the case of one of the guests there was actually files data / data lost after fsck was manually run upon reboot/single user mode. In some cases these were config files, in other database indexes, etc. This one of the 4 guests with the most severe corruption was not usable and we had to revert to a backup and pull current data out of it as much as possible.

In contrast we had an R410 server in the same rack that experienced the same power loss also running an H700 controller with NVcache, KVM, Centos6 but only 1 guest and considerably less load. That server and the one guest on it experienced zero corruption and came back up without manual intervention.


Ron

----- Original Message -----
From: "Stefan Hajnoczi" <stefanha@gmail.com>
To: "Ron Edison" <ron@idthq.com>
Cc: kvm@vger.kernel.org
Sent: Monday, April 16, 2012 1:34:41 AM
Subject: Re: writeback cache + h700 controller w/1gb nvcache, corruption on power loss

On Sun, Apr 15, 2012 at 5:16 AM, Ron Edison <ron@idthq.com> wrote:
> The server is a Dell R710 with an H700 controller with 1gb of nvcache. Writeback cache is enabled on the controller. There is a mix of linux and windows guests, some with qcow2 format vdisks and others with raw format vdisks. Some of these guests have wb cache enabled on the vdisks and some do not.

-drive cache=writeback is safe when the guest flushes appropriately.
If the guest is not sending flushes (e.g. ext3/4 barrier=0) then there
are no guarantees.

> About a third of the guests experienced disk corruption after coming back up after the host lost power. Based on what I have read, this should not have happened using the above configuration. The operating system is Centos 6.2, this is all direct attached storage configured as raid 1 mirrors.

Is the H700 battery charged?  My understanding is that writethrough
caching on the H700 will only be safe when the battery is present and
charged.

What do you mean by "guests experienced disk corruption"?  I would
expect the guests do an fsck when they are started again, just like a
physical machine that underwent power failure.  What exactly is
corrupted?  Does QEMU refuse to open the .qcow2 file?  Is there data
missing inside the guest?  Has application data (e.g. database) gone
into a bad state so you get errors?

Stefan