[Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Lutz Vieweg <lvml@5t9.de>]

Hi,

I have been investigating strange stalls of virtual machines,
and realized that the VMs were (silently) paused because qemu
thinks there were I/O errors when writing to the host.

After using "werror=report,rerror=report" with "-drive" we now see
actual reporting of I/O errors to the guest, where they look like this:

> end_request: I/O error, dev vda, sector 7243680
> EXT4-fs warning (device vda1): ext4_end_bio:259: I/O error writing to inode 951097 (offset 3096576 size 4096 starting block 905461)
> end_request: I/O error, dev vda, sector 22018120
> Buffer I/O error on device vda1, logical block 2752009
> lost page write due to I/O error on vda1
> end_request: I/O error, dev vda, sector 12857032
> JBD2: Detected IO errors while flushing file data on vda1-8
> Aborting journal on device vda1-8.

The qemu instance in question is using an executable compiled
from current sources, running on vanilla linux-4.4.2 - and qemu
is started directly, not via any library or VM management framework.

The guest drive parameters are:
>  -drive "file=image.raw,if=virtio,format=raw,media=disk,cache=unsafe,werror=report,rerror=report"


I've searched the Web and found some people reporting similar symptoms,
but related to either time-outs with NFS, direct use of LVM / DRBD
partitions on the host and such - these circumstances do not apply here.
(We do use DRBD and LVM, but qemu is not accessing raw partitions,
just an ordinary file on an XFS filesystem, and the host does not report
any I/O errors on the device or filesystem layers.)


There seems to be a relationship between the occurence of the
I/O-errors reported to the guest and the load on the I/O system of
the host - the errors become more frequent (like "once per day")
when there is high load.


Is there any kind of timeout or something that might make qemu
assume a write operation on the host has failed?

Can you provide any hint on how to pursue the cause of these errors?
(I thought about using "strace -f -p ..." on qemu, but I don't
know what exactly to look for in the output - some failed "pwrite()"
to the image file?)

Regards,

Lutz Vieweg