From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ron Edison Subject: Re: writeback cache + h700 controller w/1gb nvcache, corruption on power loss Date: Mon, 16 Apr 2012 01:51:14 -0700 (PDT) Message-ID: <28769.660.1334566274126.JavaMail.root@sys1.internetdefensetechnologies.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Stefan Hajnoczi Return-path: Received: from sys1.internetdefensetechnologies.com ([66.212.19.80]:45228 "EHLO sys1.internetdefensetechnologies.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751838Ab2DPIvc (ORCPT ); Mon, 16 Apr 2012 04:51:32 -0400 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: Thank you very much, Stefan, I would be very interested in how to ensure the guests are sending flushes. I'm unfamiliar with the example you gave, where is that configured? Primarily the guests are CentOS 4, 5 or 6. I am also curious if it would be advisable to switch to writethrough cache on each guest virtual disk and leave writeback enabled on the controller and if that would adversely affect performance of the guests. The H700 controller had a charged battery. The server itself is virtually brand new in fact, no issues with it. The disk corruption experienced was indeed lost data -- an fsck was necessary for 4 of the guests to boot at all in RW mode, they first came up read only. In the case of one of the guests there was actually files data / data lost after fsck was manually run upon reboot/single user mode. In some cases these were config files, in other database indexes, etc. This one of the 4 guests with the most severe corruption was not usable and we had to revert to a backup and pull current data out of it as much as possible. In contrast we had an R410 server in the same rack that experienced the same power loss also running an H700 controller with NVcache, KVM, Centos6 but only 1 guest and considerably less load. That server and the one guest on it experienced zero corruption and came back up without manual intervention. Ron ----- Original Message ----- From: "Stefan Hajnoczi" To: "Ron Edison" Cc: kvm@vger.kernel.org Sent: Monday, April 16, 2012 1:34:41 AM Subject: Re: writeback cache + h700 controller w/1gb nvcache, corruption on power loss On Sun, Apr 15, 2012 at 5:16 AM, Ron Edison wrote: > The server is a Dell R710 with an H700 controller with 1gb of nvcache. Writeback cache is enabled on the controller. There is a mix of linux and windows guests, some with qcow2 format vdisks and others with raw format vdisks. Some of these guests have wb cache enabled on the vdisks and some do not. -drive cache=writeback is safe when the guest flushes appropriately. If the guest is not sending flushes (e.g. ext3/4 barrier=0) then there are no guarantees. > About a third of the guests experienced disk corruption after coming back up after the host lost power. Based on what I have read, this should not have happened using the above configuration. The operating system is Centos 6.2, this is all direct attached storage configured as raid 1 mirrors. Is the H700 battery charged? My understanding is that writethrough caching on the H700 will only be safe when the battery is present and charged. What do you mean by "guests experienced disk corruption"? I would expect the guests do an fsck when they are started again, just like a physical machine that underwent power failure. What exactly is corrupted? Does QEMU refuse to open the .qcow2 file? Is there data missing inside the guest? Has application data (e.g. database) gone into a bad state so you get errors? Stefan