DMA errors in guest caused by corrupted(?) disk image

DMA errors in guest caused by corrupted(?) disk image

[Matthew Palmer]

Hi,

I've just come across a somewhat strange problem that was suggested I
report to the list.

The problem manifested itself as DMA errors and the like popping up in the
guest, like I'd expect to see if a disk in a physical machine was dying,
like this:

  hda: dma_timer_expiry: dma status == 0x21

The VM has previously been quite stable until this problem started part of
the way through today.  Another guest on the same host machine is fine.

After talking to iggy on IRC, I tried running qemu-convert over the disk
image to copy it to another image, and that solved the problem.  So it looks
like disk image corruption somehow manages to manifest itself as a DMA error
in the guest...

I'm starting KVM like so:

  kvm -m 512 -net nic,macaddr=$macaddr -net tap,iface=$iface -hda hda.qc2

As the filename suggests, it's a qcow2 image, 30GB in size.

The guest is a 32 bit RHEL3 installation.  The host is a 64 bit Debian Lenny
machine, running kvm 84.

I've got a copy of the dodgy disk image, although it's 2.7GB so not so easy
to ship around.  I can do any diagnostics on it that people need to try and
track down the cause of the problem.  I tried doing another qemu-convert
(with a view to seeing the differences between the two images) but the copy
is 500MB smaller (zero blocks removed, presumably) so a diff probably isn't
going to help much.

- Matt

Re: DMA errors in guest caused by corrupted(?) disk image

[René Pfeiffer]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

On Mar 31, 2009 at 1355 +1100, Matthew Palmer appeared and said:
> Hi,
> 
> I've just come across a somewhat strange problem that was suggested I
> report to the list.
> 
> The problem manifested itself as DMA errors and the like popping up in the
> guest, like I'd expect to see if a disk in a physical machine was dying,
> like this:
> 
>   hda: dma_timer_expiry: dma status == 0x21
> [...]

I've seen a similar effect on my KVM host server. There were I/O errors
on one partition. When I tried to reformat the partition and restore its
content from backups, mkfs was reporting I/O errors, always at the same
block number. I added a second qcow2 file and moved the partition there.

> The VM has previously been quite stable until this problem started part of
> the way through today.  Another guest on the same host machine is fine.

The KVM host runs since January very stable. The system is a Debian
Lenny with custom kernel 2.6.28.8 and kvm-84. The filesystem for images
is ext4 and the guests use ext4 in part, too.

I cannot provide the images since they are quite large (about 20 to 44
GB) and they contain sensitive data.

I plan to move to 2.6.29.x or 2.6.30.x for reasons of the ext4 patches
regarding delayed allocation and to upgrade kvm, but I can to tests with
the current system if you wish.

Best,
René.

-- 
  )\._.,--....,'``.  fL  Let GNU/Linux work for you while you take a nap.
 /,   _.. \   _\  (`._ ,. R. Pfeiffer <lynx at luchs.at> + http://web.luchs.at/
`._.-(,_..'--(,_..'`-.;.'  - System administration + Consulting + Teaching -
Got mail delivery problems?  http://web.luchs.at/information/blockedmail.php

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: DMA errors in guest caused by corrupted(?) disk image

[Avi Kivity]

René Pfeiffer wrote:
> On Mar 31, 2009 at 1355 +1100, Matthew Palmer appeared and said:
>   
>> Hi,
>>
>> I've just come across a somewhat strange problem that was suggested I
>> report to the list.
>>
>> The problem manifested itself as DMA errors and the like popping up in the
>> guest, like I'd expect to see if a disk in a physical machine was dying,
>> like this:
>>
>>   hda: dma_timer_expiry: dma status == 0x21
>> [...]
>>     
>
> I've seen a similar effect on my KVM host server. There were I/O errors
> on one partition. When I tried to reformat the partition and restore its
> content from backups, mkfs was reporting I/O errors, always at the same
> block number. I added a second qcow2 file and moved the partition there.
>
>   

It's worth trying a qemu-img convert  to move the image to an LVM volume.


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.