[Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Lutz Vieweg]

Hi,

I have been investigating strange stalls of virtual machines,
and realized that the VMs were (silently) paused because qemu
thinks there were I/O errors when writing to the host.

After using "werror=report,rerror=report" with "-drive" we now see
actual reporting of I/O errors to the guest, where they look like this:

> end_request: I/O error, dev vda, sector 7243680
> EXT4-fs warning (device vda1): ext4_end_bio:259: I/O error writing to inode 951097 (offset 3096576 size 4096 starting block 905461)
> end_request: I/O error, dev vda, sector 22018120
> Buffer I/O error on device vda1, logical block 2752009
> lost page write due to I/O error on vda1
> end_request: I/O error, dev vda, sector 12857032
> JBD2: Detected IO errors while flushing file data on vda1-8
> Aborting journal on device vda1-8.

The qemu instance in question is using an executable compiled
from current sources, running on vanilla linux-4.4.2 - and qemu
is started directly, not via any library or VM management framework.

The guest drive parameters are:
>  -drive "file=image.raw,if=virtio,format=raw,media=disk,cache=unsafe,werror=report,rerror=report"


I've searched the Web and found some people reporting similar symptoms,
but related to either time-outs with NFS, direct use of LVM / DRBD
partitions on the host and such - these circumstances do not apply here.
(We do use DRBD and LVM, but qemu is not accessing raw partitions,
just an ordinary file on an XFS filesystem, and the host does not report
any I/O errors on the device or filesystem layers.)


There seems to be a relationship between the occurence of the
I/O-errors reported to the guest and the load on the I/O system of
the host - the errors become more frequent (like "once per day")
when there is high load.


Is there any kind of timeout or something that might make qemu
assume a write operation on the host has failed?

Can you provide any hint on how to pursue the cause of these errors?
(I thought about using "strace -f -p ..." on qemu, but I don't
know what exactly to look for in the output - some failed "pwrite()"
to the image file?)

Regards,

Lutz Vieweg

Re: [Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Fam Zheng]

On Tue, 04/19 19:47, Lutz Vieweg wrote:
> The guest drive parameters are:
> > -drive "file=image.raw,if=virtio,format=raw,media=disk,cache=unsafe,werror=report,rerror=report"

Given this implies aio=threads...

> Can you provide any hint on how to pursue the cause of these errors?
> (I thought about using "strace -f -p ..." on qemu, but I don't
> know what exactly to look for in the output - some failed "pwrite()"
> to the image file?)

... yes, this should work.

Fam

Re: [Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Kevin Wolf]

Am 20.04.2016 um 04:11 hat Fam Zheng geschrieben:
> On Tue, 04/19 19:47, Lutz Vieweg wrote:
> > The guest drive parameters are:
> > > -drive "file=image.raw,if=virtio,format=raw,media=disk,cache=unsafe,werror=report,rerror=report"
> 
> Given this implies aio=threads...
> 
> > Can you provide any hint on how to pursue the cause of these errors?
> > (I thought about using "strace -f -p ..." on qemu, but I don't
> > know what exactly to look for in the output - some failed "pwrite()"
> > to the image file?)
> 
> ... yes, this should work.

To catch all possible write failures, I think pwrite, pwritev and
possibly fdatasync need to be considered.

Kevin

Re: [Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Lutz Vieweg]

On 04/20/2016 01:50 PM, Kevin Wolf wrote:
> To catch all possible write failures, I think pwrite, pwritev and
> possibly fdatasync need to be considered.

I've now a
> strace -f -p 10727 -e trace=pwrite,pwritev,fdatasync,file -t 2>&1 | gzip -1 -c >trace.gz
attached to the qemu-process.

If the incident rate stays the same, by tomorrow I should be able
to correlate newly emitted I/O-errors in the guest with that log.

Regards,

Lutz Vieweg

Re: [Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Lutz Vieweg]

On 04/20/2016 04:38 PM, Lutz Vieweg wrote:
> I've now a
>> strace -f -p 10727 -e trace=pwrite,pwritev,fdatasync,file -t 2>&1 | gzip -1 -c >trace.gz
> attached to the qemu-process.
>
> If the incident rate stays the same, by tomorrow I should be able
> to correlate newly emitted I/O-errors in the guest with that log.

Ok, mystery solved:

> [pid 18241] 00:17:15 pwritev(16, [{..., 4096}, {..., 4096}], 2, 6585417728) = -1 ENOSPC (No space left on device)
> [pid 18241] 00:17:15 pwrite(16, ..., 4096, 6581915648) = -1 ENOSPC (No space left on device)
> [pid 18241] 00:17:15 pwrite(16, ..., 4096, 1048576) = -1 ENOSPC (No space left on device)
> [pid 18241] 00:17:15 pwrite(16, ..., 4096, 1048576) = -1 ENOSPC (No space left on device)

File descriptor fd=16 was associated with a raw image file that actually
resides on a btrfs filesystem, a constant-sized 16GB file with attributes
set to not use CopyOnWrite semantics.

Nevertheless, writes to such files can still yield ENOSPC due to a bug in btrfs:
> http://www.spinics.net/lists/linux-btrfs/msg52691.html

And indeed, the errors occured exactly at the time a backup procedure
was preparing a read-only snapshot with "btrfs subvolume snapshot -r" -
so until I can upgrade to a mainline kernel including the fix, I'll
pause the qemu process while the "btrfs subvolume snapshot -r" runs.

Thanks for the hints.
Sorry this turned out to be a btrfs rather than a qemu bug - I was
first misled to believe the image was on XFS.

Nevertheless, I think qemu could be somewhat more verbose, reporting
when and why it stops emulation. Something like a message to the monitor
or to standard out would be helpful to start with...

Regards,

Lutz Vieweg

Re: [Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Fam Zheng]

On Thu, 04/21 17:54, Lutz Vieweg wrote:
> Nevertheless, I think qemu could be somewhat more verbose, reporting
> when and why it stops emulation. Something like a message to the monitor
> or to standard out would be helpful to start with...

QEMU does report an error message to connected monitor if configured to. See
the werror= and rerror= options.

Fam

Re: [Qemu-devel] I/O errors reported to guest for raw-image-file backed /dev/vda - but host sees no I/O errors

[Lutz Vieweg]

On 04/21/2016 05:54 PM, Lutz Vieweg wrote:
> And indeed, the errors occured exactly at the time a backup procedure
> was preparing a read-only snapshot with "btrfs subvolume snapshot -r" -
> so until I can upgrade to a mainline kernel including the fix, I'll
> pause the qemu process while the "btrfs subvolume snapshot -r" runs.

Alas, this seems not to be sufficient to get rid of the symptom easily -
I guess that some of the "pwrite"s are executed asynchronously, at
a time when the qemu emulation is already in "pause" mode.

Will retry with some safeguard sleep time between the "stop" to
the monitor and the creation of the snapshot.

Regards,

Lutz Vieweg