[Qemu-devel] core dump with drive-mirror

[Qemu-devel] core dump with drive-mirror

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2958 bytes --]

I'm trying to track down a core dump with the QMP drive-mirror command.

# in one terminal:
cd /tmp
rm -f base.img snap1.img snap2.img copy.img

# base.img <- snap1.img <- snap2.img; intentionally populating base.img
# with a qcow2 header, but treating it as raw data
qemu-img create -f qcow2 base.img 10M
qemu-img create -f qcow2 -b base.img -o backing_fmt=raw snap1.img
qemu-img create -f qcow2 -b snap1.img -o backing_fmt=qcow2 snap2.img
cp base.img copy.img
# Yes, this command line is derived from libvirt...
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
 QEMU_AUDIO_DRV=none gdb --args  /usr/bin/qemu-system-x86_64 \
 -machine accel=kvm -name testvm1 -S \
 -machine pc-i440fx-2.0,accel=kvm,usb=off -m 256 \
 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 \
 -uuid 5a74eeb4-09c5-4fc2-869d-0e04c13f9db0 -no-user-config \
 -nodefaults -chardev socket,id=charmonitor,\
path=/var/lib/libvirt/qemu/testvm1.monitor,server,nowait \
 -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc \
 -no-shutdown -no-acpi -boot strict=on \
 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
 -drive file=/tmp/snap2.img,if=none,id=drive-virtio-disk0,format=qcow2 \
 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,\
drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 \
 -msg timestamp=on

# in next terminal:
mkdir -p /var/lib/libvirt/qemu
nc -u /var/lib/libvirt/qemu/testvm1.monitor
{"execute":"qmp_capabilities"}
{"execute":"cont"}
{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0",
"target":"/tmp/copy.img","format":"raw", "mode":"existing","sync":"full"}}

at which point, I get a core dump with the following trace:
#0  __memcpy_sse2_unaligned ()
    at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:41
#1  0x000055555563c9d9 in memcpy (__len=<optimized out>,
    __src=<optimized out>, __dest=<optimized out>)
    at /usr/include/bits/string3.h:51
#2  handle_aiocb_rw (aiocb=0x5555562064c0) at block/raw-posix.c:756
#3  0x000055555563d645 in aio_worker (arg=0x5555562064c0)
    at block/raw-posix.c:922
#4  0x000055555578774b in worker_thread (opaque=0x555556206800)
    at thread-pool.c:110
#5  0x00007ffff3f1ef33 in start_thread (arg=0x7fffe0cd3700)
    at pthread_create.c:309
#6  0x00007fffed636ded in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

and note that the file size of copy.img is no longer the same as what I
was expecting:
-rw-r--r--. 1 qemu qemu 197120 Jun 30 16:58 /tmp/base.img
-rw-r--r--. 1 qemu qemu 262144 Jun 30 17:14 /tmp/copy.img
-rw-r--r--. 1 qemu qemu 197120 Jun 30 16:58 /tmp/snap1.img
-rw-r--r--. 1 root root 197120 Jun 30 16:58 /tmp/snap2.img


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] core dump with drive-mirror

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2888 bytes --]

On 06/30/2014 05:16 PM, Eric Blake wrote:
> I'm trying to track down a core dump with the QMP drive-mirror command.

Oh, and a followup question:

I discovered that if I try to use drive-mirror to a file that is not
already large enough, the job starts successfully but fails at the point
where it exceeds the block size.  It would be much nicer if the job
auto-resized the destination to be large enough to hold the copy, but if
that is not possible, then if the drive-mirror would fail up front
rather than succeeding at starting a job that is doomed to failure,
where the failure is only reported by an event and lacks useful
information at why it failed.  To trigger this scenario, alter my setup...

> 
> # in one terminal:
> cd /tmp
> rm -f base.img snap1.img snap2.img copy.img
> 
> # base.img <- snap1.img <- snap2.img; intentionally populating base.img
> # with a qcow2 header, but treating it as raw data
> qemu-img create -f qcow2 base.img 10M

...by creating the base.img file with '-f raw'...

> qemu-img create -f qcow2 -b base.img -o backing_fmt=raw snap1.img
> qemu-img create -f qcow2 -b snap1.img -o backing_fmt=qcow2 snap2.img
> cp base.img copy.img

...and using 'touch copy.img' instead of 'cp base.img copy.img'.

> # Yes, this command line is derived from libvirt...
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
>  QEMU_AUDIO_DRV=none gdb --args  /usr/bin/qemu-system-x86_64 \
>  -machine accel=kvm -name testvm1 -S \
>  -machine pc-i440fx-2.0,accel=kvm,usb=off -m 256 \
>  -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 \
>  -uuid 5a74eeb4-09c5-4fc2-869d-0e04c13f9db0 -no-user-config \
>  -nodefaults -chardev socket,id=charmonitor,\
> path=/var/lib/libvirt/qemu/testvm1.monitor,server,nowait \
>  -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc \
>  -no-shutdown -no-acpi -boot strict=on \
>  -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
>  -drive file=/tmp/snap2.img,if=none,id=drive-virtio-disk0,format=qcow2 \
>  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,\
> drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
>  -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
>  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 \
>  -msg timestamp=on
> 
> # in next terminal:
> mkdir -p /var/lib/libvirt/qemu
> nc -u /var/lib/libvirt/qemu/testvm1.monitor
> {"execute":"qmp_capabilities"}
> {"execute":"cont"}
> {"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0",
> "target":"/tmp/copy.img","format":"raw", "mode":"existing","sync":"full"}}

everything else being the same, the drive-mirror starts but I get an
early event that the job fails, and query-blockjobs no longer knows
about it.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] core dump with drive-mirror

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On 06/30/2014 05:16 PM, Eric Blake wrote:
> I'm trying to track down a core dump with the QMP drive-mirror command.

Looks like the bug is related to a base image that is not a multiple of
a cluster size.

> 
> # in one terminal:
> cd /tmp
> rm -f base.img snap1.img snap2.img copy.img
> 
> # base.img <- snap1.img <- snap2.img; intentionally populating base.img
> # with a qcow2 header, but treating it as raw data
> qemu-img create -f qcow2 base.img 10M

If, right here, I inject:

truncate --size 262144 base.img

> qemu-img create -f qcow2 -b base.img -o backing_fmt=raw snap1.img
> qemu-img create -f qcow2 -b snap1.img -o backing_fmt=qcow2 snap2.img
> cp base.img copy.img
> # Yes, this command line is derived from libvirt...
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
>  QEMU_AUDIO_DRV=none gdb --args  /usr/bin/qemu-system-x86_64 \

...then everything else succeeds.  So it seems the problem is that qemu
is doing a lousy job of handling a backing file and/or destination file
that is not fully rounded out to a proper size.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] core dump with drive-mirror

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

On Mon, Jun 30, 2014 at 05:40:16PM -0600, Eric Blake wrote:
> On 06/30/2014 05:16 PM, Eric Blake wrote:
> > I'm trying to track down a core dump with the QMP drive-mirror command.
> 
> Looks like the bug is related to a base image that is not a multiple of
> a cluster size.
> 
> > 
> > # in one terminal:
> > cd /tmp
> > rm -f base.img snap1.img snap2.img copy.img
> > 
> > # base.img <- snap1.img <- snap2.img; intentionally populating base.img
> > # with a qcow2 header, but treating it as raw data
> > qemu-img create -f qcow2 base.img 10M
> 
> If, right here, I inject:
> 
> truncate --size 262144 base.img
> 
> > qemu-img create -f qcow2 -b base.img -o backing_fmt=raw snap1.img
> > qemu-img create -f qcow2 -b snap1.img -o backing_fmt=qcow2 snap2.img
> > cp base.img copy.img
> > # Yes, this command line is derived from libvirt...
> > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
> >  QEMU_AUDIO_DRV=none gdb --args  /usr/bin/qemu-system-x86_64 \
> 
> ...then everything else succeeds.  So it seems the problem is that qemu
> is doing a lousy job of handling a backing file and/or destination file
> that is not fully rounded out to a proper size.

Thanks for reporting this.  It's something we need to fix during the
QEMU 2.1 hard freeze that is starting today.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]