[Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Tiziano Müller]

Hi everyone

We have a couple of VMs, each with QCow2-files as disk images and we are
seeing random filesystem/qcow2 corruptions of the VM filesystems.

We have two types of corruptions:
* if the guest OS uses XFS, we get corruption errors originating most of
the time in xfs_da_do_buf inside the VM.
* if the guest OS uses ext4 (on rare occasions also XFS), there are most
of the time no noticeable faults in the guest until qemu shuts down with
the following messages (maybe that's a different case and unrelated to
the first one):
handle_dev_stop: stop
handle_dev_stop: stop
handle_dev_stop: stop
qcow2_free_clusters failed: Invalid argument
[...]
qcow2_free_clusters failed: Invalid argument
handle_dev_stop: stop
handle_dev_stop: stop
2012-10-09 00:52:49.294+0000: shutting down

(one time there was also a message before the end about a failed assert
in a qcow-related function)

Checking the image using `qemu-img check` then gives something like
this:

ERROR OFLAG_COPIED: offset=3bc30000 refcount=1
ERROR offset=c7e331: Cluster is not properly aligned; L2 entry
corrupted.
ERROR: invalid cluster offset=0x3aa4810400fa0000
ERROR offset=fa000000001000: Cluster is not properly aligned; L2 entry
corrupted.
ERROR: invalid cluster offset=0x80966181ff0000
ERROR: invalid cluster offset=0x80966182000000
ERROR offset=80966181ffffff: Cluster is not properly aligned; L2 entry
corrupted.
ERROR: cluster 280376143330560: copied flag must never be set for
compressed clusters
Warning: cluster offset=0x286d3d000000 is after the end of the image
file, can't properly check refcounts.
Warning: cluster offset=0x286d3d010000 is after the end of the image
file, can't properly check refcounts.

We can observe such corruptions on two servers:
* one with qemu-kvm-1.1 and kernel 3.5.2 (for the kernel configuration see [1])
* the second one with qemu-kvm-1.0 and kernel 3.2.6 (for the kernel configuration see [2])

The corruptions happen up a lot faster and more often on server 1 with
qemu-kvm-1.1 and do not depend on whether write operations happen or
not.

One test case was:
* xfs_repair /dev/vda2 -> corruptions found and fixed
* xfs_repair /dev/vda2 -> no corruptions found
* mount /dev/vda2 /mnt/something
* find /mnt/something > /dev/null
* umount /mnt/something
* xfs_repair /dev/vda2 -> corruptions found again

The interesting thing here is that xfs_repair shows bad magic numbers of
0x0 most of the time which would indicate that blocks get zeroed out and
not randomly overwritten.

The guest kernels range from 3.2 to 3.5, used distros: Gentoo, Fedora,
Ubuntu, all x86_64.

The host filesystem is XFS as well, on a LVM volume on a RAID1 (Megaraid
SAS). Management is done with libvirt and we are using virtio-blk with
cache=writeback (an example configuration of a VM is found in [3]).

We were unable to trigger this deliberately, some of the VMs were
running without problems for more than a month (server 2), some of them show
problems after a couple of hours (server 1).

We checked:
* in the host: memory and LVM using memtester, verify-data and
badblocks. We also tried with a brand new XFS volume and brand new qcow2
files.
* in the VMs: memory using memtest86+ and the virtual disk using
verify-data (verify-data -e /dev/vda3 1M 10000). Both test are currently
running without problems so far. The only suspicious thing is that
verify-data sometimes hangs for up to a minute (as do all block
operations in that VM), but `dmesg` does not show anything.

Does anyone has any ideas on how to narrow this problem down further or
how to debug it?

We are currently setting up another server with qemu-1.1.2 or qemu-1.2.0
and would like to be able to reproduce this problem before and make sure
we don't face the same problems there again.

We can provide logs of almost everything if required. We could also
provide SSH and/or Spice access to one or more VMs running on the server
if that helps and someone would be willing to help track down this
problem.

Similar problems were already mentioned here, but without a solution:
* https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1040033
* http://lists.gnu.org/archive/html/qemu-devel/2012-04/msg01756.html

Thanks in advance,
Tiziano

[1] server1 kernel config: http://bpaste.net/show/rajkzUNGLdmo95sl35YE/
[2] server2 kernel config: http://bpaste.net/show/cAFQ7FhJzfjAorIirUj6/
[3] libvirt vm xml: http://bpaste.net/show/D7nRCbr26f9nmapq5gTo/
[4] qemu-kvm invocation: http://bpaste.net/show/Mgrq9AeD2ehBfI43SH5u/
-- 
stepping stone GmbH
Neufeldstrasse 9
CH-3012 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
tiziano.mueller@stepping-stone.ch

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Stefan Hajnoczi]

On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
> One test case was:
> * xfs_repair /dev/vda2 -> corruptions found and fixed
> * xfs_repair /dev/vda2 -> no corruptions found

>From here...

> * mount /dev/vda2 /mnt/something
> * find /mnt/something > /dev/null
> * umount /mnt/something

...to here is mostly read-only.

> * xfs_repair /dev/vda2 -> corruptions found again

And yet corruptions are reported inside the guest!

Can you retry with "mount -o ro" just for sanity?  If you still see
corruptions when the device was mounted read-only, then perhaps the
mount/find/umount sequence itself is innocent.

Have you tried raw image files?  In other words, do you know that the
corruption only happens with qcow2?

Stefan

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Tiziano Müller]

Am Freitag, den 12.10.2012, 10:28 +0200 schrieb Stefan Hajnoczi:
> On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
> > One test case was:
> > * xfs_repair /dev/vda2 -> corruptions found and fixed
> > * xfs_repair /dev/vda2 -> no corruptions found
> 
> From here...
> 
> > * mount /dev/vda2 /mnt/something
> > * find /mnt/something > /dev/null
> > * umount /mnt/something
> 
> ...to here is mostly read-only.
> 
> > * xfs_repair /dev/vda2 -> corruptions found again
> 
> And yet corruptions are reported inside the guest!
> 
> Can you retry with "mount -o ro" just for sanity?  If you still see
> corruptions when the device was mounted read-only, then perhaps the
> mount/find/umount sequence itself is innocent.

Will try as soon as possible. The problem is that we had that case twice
so far and no way to trigger it.

> 
> Have you tried raw image files?  In other words, do you know that the
> corruption only happens with qcow2?

Will do right now.

thanks a lot!

-- 
stepping stone GmbH
Neufeldstrasse 9
CH-3012 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
tiziano.mueller@stepping-stone.ch

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Stefan Hajnoczi]

On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
> Checking the image using `qemu-img check` then gives something like
> this:
> 
> ERROR OFLAG_COPIED: offset=3bc30000 refcount=1
> ERROR offset=c7e331: Cluster is not properly aligned; L2 entry
> corrupted.

Is any other program accessing the qcow2 image on the host while the VM
is running?

For example, are you running qemu-img on the image while the VM is
running?

Stefan

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Tiziano Müller]

Am Freitag, den 12.10.2012, 10:33 +0200 schrieb Stefan Hajnoczi:
> On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
> > Checking the image using `qemu-img check` then gives something like
> > this:
> > 
> > ERROR OFLAG_COPIED: offset=3bc30000 refcount=1
> > ERROR offset=c7e331: Cluster is not properly aligned; L2 entry
> > corrupted.
> 
> Is any other program accessing the qcow2 image on the host while the VM
> is running?

> For example, are you running qemu-img on the image while the VM is
> running?

On some VMs we tried to extract filesystem snapshots at runtime:

  qemu-img convert -s snapshot-id original.qcow2 snapshot.qcow2

(yes, that's not consistent, we're switching to external snapshots).
But that should open the image read-only, right?

Other operations where the qemu-monitor-commands "savevm" and "delvm". 

Although: we created a new qcow2 and even in that the filesystem got
corrupted without any of the above actions. So we're pretty confident
that those operations are not the sole cause.


-- 
stepping stone GmbH
Neufeldstrasse 9
CH-3012 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
tiziano.mueller@stepping-stone.ch

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 10:53:29AM +0200, Tiziano Müller wrote:
> Am Freitag, den 12.10.2012, 10:33 +0200 schrieb Stefan Hajnoczi:
> > On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
> > > Checking the image using `qemu-img check` then gives something like
> > > this:
> > > 
> > > ERROR OFLAG_COPIED: offset=3bc30000 refcount=1
> > > ERROR offset=c7e331: Cluster is not properly aligned; L2 entry
> > > corrupted.
> > 
> > Is any other program accessing the qcow2 image on the host while the VM
> > is running?
> 
> > For example, are you running qemu-img on the image while the VM is
> > running?
> 
> On some VMs we tried to extract filesystem snapshots at runtime:
> 
>   qemu-img convert -s snapshot-id original.qcow2 snapshot.qcow2
> 
> (yes, that's not consistent, we're switching to external snapshots).
> But that should open the image read-only, right?
> 
> Other operations where the qemu-monitor-commands "savevm" and "delvm". 
> 
> Although: we created a new qcow2 and even in that the filesystem got
> corrupted without any of the above actions. So we're pretty confident
> that those operations are not the sole cause.

Okay, that's consistent with the other symptoms you've reported.

It's not clear whether the corruption arises inside qcow2 or if
something else is causing corruption and qcow2/xfs get upset.  That is
the next step to debugging this - hopefully it will become possible to
reproduce it reliably, at which point it is much easier to debug :).

Stefan

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Tiziano Müller]

Am Montag, den 15.10.2012, 09:48 +0200 schrieb Stefan Hajnoczi:
> Okay, that's consistent with the other symptoms you've reported.
> 
> It's not clear whether the corruption arises inside qcow2 or if
> something else is causing corruption and qcow2/xfs get upset.  That is
> the next step to debugging this - hopefully it will become possible to
> reproduce it reliably, at which point it is much easier to debug :).

Yes, we have now 3 different VM-configurations deployed on the server
where we saw the most corruptions so far:

* config 1: as-is
* config 2: raw-image format instead of qcow2 as you suggested
* config 3: vhost=off for the network

The third test-case is because we figured out that one major difference
between the two servers is that on the server with the most corruptions
we have vhost=on (automatically turned on by qemu/libvirt because the
vhost-net module got loaded). I know this does not make a lot of sense.

In fact, on the server without vhost-net (and with qemu-1.0) we only saw
one corruption after all (where the qcow2 got really messed up), so one
could assume that this was a single event and may have been a user
error. To summarize it:

* server 1 (qemu-kvm-1.1, host-kernel 3.5.2, vhost-net loaded and used)
has had +5 machines with corrupt filesystems (ext4 and xfs) and 1
machine with a corrupt qcow2.
* server 2 (qemu-kvm-1.0, host-kernel 3.2.6, vhost-net not loaded) has
had only 1 machine with a corrupt qcow2.

The mentioned Test-VMs are running since Friday and unfortunately none
of them had the decency to go corrupt again.

We will keep you posted,
thanks a lot for your help.

Best regards,
Tiziano

-- 
stepping stone GmbH
Neufeldstrasse 9
CH-3012 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
tiziano.mueller@stepping-stone.ch

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Kevin Wolf]

Am 12.10.2012 10:53, schrieb Tiziano Müller:
> Am Freitag, den 12.10.2012, 10:33 +0200 schrieb Stefan Hajnoczi:
>> On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
>>> Checking the image using `qemu-img check` then gives something like
>>> this:
>>>
>>> ERROR OFLAG_COPIED: offset=3bc30000 refcount=1
>>> ERROR offset=c7e331: Cluster is not properly aligned; L2 entry
>>> corrupted.
>>
>> Is any other program accessing the qcow2 image on the host while the VM
>> is running?
> 
>> For example, are you running qemu-img on the image while the VM is
>> running?
> 
> On some VMs we tried to extract filesystem snapshots at runtime:
> 
>   qemu-img convert -s snapshot-id original.qcow2 snapshot.qcow2
> 
> (yes, that's not consistent, we're switching to external snapshots).
> But that should open the image read-only, right?

Yes, in theory that should be harmless and even produce a correct copy
of the snapshot.

> Other operations where the qemu-monitor-commands "savevm" and "delvm". 
> 
> Although: we created a new qcow2 and even in that the filesystem got
> corrupted without any of the above actions. So we're pretty confident
> that those operations are not the sole cause.

So no internal snapshots are involved at all with this new image? I'm
asking because in the past non-reproducible failures were reported with
snapshots, but I'm not aware of any case that didn't use snapshots.

Any other non-default feature that you used, like compression?

Kevin

Re: [Qemu-devel] Silent filesystem/qcow2 corruptions with qemu-kvm-1.0 and 1.1.1

[Tiziano Müller]

Am Montag, den 15.10.2012, 13:11 +0200 schrieb Kevin Wolf:
> Am 12.10.2012 10:53, schrieb Tiziano Müller:
> > Am Freitag, den 12.10.2012, 10:33 +0200 schrieb Stefan Hajnoczi:
> >> On Thu, Oct 11, 2012 at 03:33:23PM +0200, Tiziano Müller wrote:
> >>> Checking the image using `qemu-img check` then gives something like
> >>> this:
> >>>
> >>> ERROR OFLAG_COPIED: offset=3bc30000 refcount=1
> >>> ERROR offset=c7e331: Cluster is not properly aligned; L2 entry
> >>> corrupted.
> >>
> >> Is any other program accessing the qcow2 image on the host while the VM
> >> is running?
> > 
> >> For example, are you running qemu-img on the image while the VM is
> >> running?
> > 
> > On some VMs we tried to extract filesystem snapshots at runtime:
> > 
> >   qemu-img convert -s snapshot-id original.qcow2 snapshot.qcow2
> > 
> > (yes, that's not consistent, we're switching to external snapshots).
> > But that should open the image read-only, right?
> 
> Yes, in theory that should be harmless and even produce a correct copy
> of the snapshot.

Good to know, thanks.

> 
> > Other operations where the qemu-monitor-commands "savevm" and "delvm". 
> > 
> > Although: we created a new qcow2 and even in that the filesystem got
> > corrupted without any of the above actions. So we're pretty confident
> > that those operations are not the sole cause.
> 
> So no internal snapshots are involved at all with this new image?

No. I just checked again: no internal snapshot are currently present nor
have been added/removed in the past.

>  I'm
> asking because in the past non-reproducible failures were reported with
> snapshots, but I'm not aware of any case that didn't use snapshots.
> 
> Any other non-default feature that you used, like compression?

No, we either create the qcows by hand using:
  qemu-img create -f qcow2 kvm-0XY_01.qcow2 30G

or (usually) using libvirt:
  virsh vol-create $pool $diskfile

where $diskfile points to a xml like this:

<volume>
  <name>${diskfilename}</name>
  <allocation>0</allocation>
  <capacity unit="G">${kvmsize}.qcow2</capacity>
  <target>
    <format type='qcow2'/>
    <permissions>
      <owner>0</owner>
      <group>3000</group>
      <mode>0660</mode>
    </permissions>
  </target>
</volume>

and $pool references a directory-based storage pool.

-- 
stepping stone GmbH
Neufeldstrasse 9
CH-3012 Bern
Telefon: +41 31 332 53 63
www.stepping-stone.ch
tiziano.mueller@stepping-stone.ch