* [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
@ 2025-02-28 17:39 Andrey Drobyshev
2025-02-28 18:13 ` Steven Sistare
0 siblings, 1 reply; 13+ messages in thread
From: Andrey Drobyshev @ 2025-02-28 17:39 UTC (permalink / raw)
To: qemu-devel
Cc: Steve Sistare, William Roche, Gerd Hoffmann,
Daniel P. Berrangé, Denis V. Lunev, andrey.drobyshev
Hi all,
We've been experimenting with cpr-transfer migration mode recently and
have discovered the following issue with the guest QXL driver:
Run migration source:
> EMULATOR=/path/to/emulator
> ROOTFS=/path/to/image
> QMPSOCK=/var/run/alma8qmp-src.sock
>
> $EMULATOR -enable-kvm \
> -machine q35 \
> -cpu host -smp 2 -m 2G \
> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/ram0,share=on\
> -machine memory-backend=ram0 \
> -machine aux-ram-share=on \
> -drive file=$ROOTFS,media=disk,if=virtio \
> -qmp unix:$QMPSOCK,server=on,wait=off \
> -nographic \
> -device qxl-vga
Run migration target:
> EMULATOR=/path/to/emulator
> ROOTFS=/path/to/image
> QMPSOCK=/var/run/alma8qmp-dst.sock
>
> $EMULATOR -enable-kvm \
> -machine q35 \
> -cpu host -smp 2 -m 2G \
> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/ram0,share=on\
> -machine memory-backend=ram0 \
> -machine aux-ram-share=on \
> -drive file=$ROOTFS,media=disk,if=virtio \
> -qmp unix:$QMPSOCK,server=on,wait=off \
> -nographic \
> -device qxl-vga \
> -incoming tcp:0:44444 \
> -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
Launch the migration:
> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
> QMPSOCK=/var/run/alma8qmp-src.sock
>
> $QMPSHELL -p $QMPSOCK <<EOF
> migrate-set-parameters mode=cpr-transfer
> migrate channels=[{"channel-type":"main","addr":{"transport":"socket","type":"inet","host":"0","port":"44444"}},{"channel-type":"cpr","addr":{"transport":"socket","type":"unix","path":"/var/run/alma8cpr-dst.sock"}}]
> EOF
Then, after a while, QXL guest driver on target crashes spewing the
following messages:
> [ 73.962002] [TTM] Buffer eviction failed
> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001)
> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO
That seems to be a known kernel QXL driver bug:
https://lore.kernel.org/all/20220907094423.93581-1-min_halo@163.com/T/
https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
(the latter discussion contains that reproduce script which speeds up
the crash in the guest):
> #!/bin/bash
>
> chvt 3
>
> for j in $(seq 80); do
> echo "$(date) starting round $j"
> if [ "$(journalctl --boot | grep "failed to allocate VRAM BO")" != "" ]; then
> echo "bug was reproduced after $j tries"
> exit 1
> fi
> for i in $(seq 100); do
> dmesg > /dev/tty3
> done
> done
>
> echo "bug could not be reproduced"
> exit 0
The bug itself seems to remain unfixed, as I was able to reproduce that
with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
cpr-transfer code also seems to be buggy as it triggers the crash -
without the cpr-transfer migration the above reproduce doesn't lead to
crash on the source VM.
I suspect that, as cpr-transfer doesn't migrate the guest memory, but
rather passes it through the memory backend object, our code might
somehow corrupt the VRAM. However, I wasn't able to trace the
corruption so far.
Could somebody help the investigation and take a look into this? Any
suggestions would be appreciated. Thanks!
Andrey
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-02-28 17:39 [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration Andrey Drobyshev
@ 2025-02-28 18:13 ` Steven Sistare
2025-02-28 18:20 ` Steven Sistare
0 siblings, 1 reply; 13+ messages in thread
From: Steven Sistare @ 2025-02-28 18:13 UTC (permalink / raw)
To: Andrey Drobyshev, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
> Hi all,
>
> We've been experimenting with cpr-transfer migration mode recently and
> have discovered the following issue with the guest QXL driver:
>
> Run migration source:
>> EMULATOR=/path/to/emulator
>> ROOTFS=/path/to/image
>> QMPSOCK=/var/run/alma8qmp-src.sock
>>
>> $EMULATOR -enable-kvm \
>> -machine q35 \
>> -cpu host -smp 2 -m 2G \
>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/ram0,share=on\
>> -machine memory-backend=ram0 \
>> -machine aux-ram-share=on \
>> -drive file=$ROOTFS,media=disk,if=virtio \
>> -qmp unix:$QMPSOCK,server=on,wait=off \
>> -nographic \
>> -device qxl-vga
>
> Run migration target:
>> EMULATOR=/path/to/emulator
>> ROOTFS=/path/to/image
>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>
>> $EMULATOR -enable-kvm \
>> -machine q35 \
>> -cpu host -smp 2 -m 2G \
>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/ram0,share=on\
>> -machine memory-backend=ram0 \
>> -machine aux-ram-share=on \
>> -drive file=$ROOTFS,media=disk,if=virtio \
>> -qmp unix:$QMPSOCK,server=on,wait=off \
>> -nographic \
>> -device qxl-vga \
>> -incoming tcp:0:44444 \
>> -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>
>
> Launch the migration:
>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>> QMPSOCK=/var/run/alma8qmp-src.sock
>>
>> $QMPSHELL -p $QMPSOCK <<EOF
>> migrate-set-parameters mode=cpr-transfer
>> migrate channels=[{"channel-type":"main","addr":{"transport":"socket","type":"inet","host":"0","port":"44444"}},{"channel-type":"cpr","addr":{"transport":"socket","type":"unix","path":"/var/run/alma8cpr-dst.sock"}}]
>> EOF
>
> Then, after a while, QXL guest driver on target crashes spewing the
> following messages:
>> [ 73.962002] [TTM] Buffer eviction failed
>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001)
>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO
>
> That seems to be a known kernel QXL driver bug:
>
> https://lore.kernel.org/all/20220907094423.93581-1-min_halo@163.com/T/
> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>
> (the latter discussion contains that reproduce script which speeds up
> the crash in the guest):
>> #!/bin/bash
>>
>> chvt 3
>>
>> for j in $(seq 80); do
>> echo "$(date) starting round $j"
>> if [ "$(journalctl --boot | grep "failed to allocate VRAM BO")" != "" ]; then
>> echo "bug was reproduced after $j tries"
>> exit 1
>> fi
>> for i in $(seq 100); do
>> dmesg > /dev/tty3
>> done
>> done
>>
>> echo "bug could not be reproduced"
>> exit 0
>
> The bug itself seems to remain unfixed, as I was able to reproduce that
> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
> cpr-transfer code also seems to be buggy as it triggers the crash -
> without the cpr-transfer migration the above reproduce doesn't lead to
> crash on the source VM.
>
> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
> rather passes it through the memory backend object, our code might
> somehow corrupt the VRAM. However, I wasn't able to trace the
> corruption so far.
>
> Could somebody help the investigation and take a look into this? Any
> suggestions would be appreciated. Thanks!
Possibly some memory region created by qxl is not being preserved.
Try adding these traces to see what is preserved:
-trace enable='*cpr*'
-trace enable='*ram_alloc*'
- Steve
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-02-28 18:13 ` Steven Sistare
@ 2025-02-28 18:20 ` Steven Sistare
2025-02-28 18:35 ` Andrey Drobyshev
0 siblings, 1 reply; 13+ messages in thread
From: Steven Sistare @ 2025-02-28 18:20 UTC (permalink / raw)
To: Andrey Drobyshev, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
On 2/28/2025 1:13 PM, Steven Sistare wrote:
> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>> Hi all,
>>
>> We've been experimenting with cpr-transfer migration mode recently and
>> have discovered the following issue with the guest QXL driver:
>>
>> Run migration source:
>>> EMULATOR=/path/to/emulator
>>> ROOTFS=/path/to/image
>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>
>>> $EMULATOR -enable-kvm \
>>> -machine q35 \
>>> -cpu host -smp 2 -m 2G \
>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/ram0,share=on\
>>> -machine memory-backend=ram0 \
>>> -machine aux-ram-share=on \
>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>> -nographic \
>>> -device qxl-vga
>>
>> Run migration target:
>>> EMULATOR=/path/to/emulator
>>> ROOTFS=/path/to/image
>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>> $EMULATOR -enable-kvm \
>>> -machine q35 \
>>> -cpu host -smp 2 -m 2G \
>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/ram0,share=on\
>>> -machine memory-backend=ram0 \
>>> -machine aux-ram-share=on \
>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>> -nographic \
>>> -device qxl-vga \
>>> -incoming tcp:0:44444 \
>>> -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>
>>
>> Launch the migration:
>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>
>>> $QMPSHELL -p $QMPSOCK <<EOF
>>> migrate-set-parameters mode=cpr-transfer
>>> migrate channels=[{"channel-type":"main","addr":{"transport":"socket","type":"inet","host":"0","port":"44444"}},{"channel-type":"cpr","addr":{"transport":"socket","type":"unix","path":"/var/run/alma8cpr-dst.sock"}}]
>>> EOF
>>
>> Then, after a while, QXL guest driver on target crashes spewing the
>> following messages:
>>> [ 73.962002] [TTM] Buffer eviction failed
>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001)
>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO
>>
>> That seems to be a known kernel QXL driver bug:
>>
>> https://lore.kernel.org/all/20220907094423.93581-1-min_halo@163.com/T/
>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>
>> (the latter discussion contains that reproduce script which speeds up
>> the crash in the guest):
>>> #!/bin/bash
>>>
>>> chvt 3
>>>
>>> for j in $(seq 80); do
>>> echo "$(date) starting round $j"
>>> if [ "$(journalctl --boot | grep "failed to allocate VRAM BO")" != "" ]; then
>>> echo "bug was reproduced after $j tries"
>>> exit 1
>>> fi
>>> for i in $(seq 100); do
>>> dmesg > /dev/tty3
>>> done
>>> done
>>>
>>> echo "bug could not be reproduced"
>>> exit 0
>>
>> The bug itself seems to remain unfixed, as I was able to reproduce that
>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>> cpr-transfer code also seems to be buggy as it triggers the crash -
>> without the cpr-transfer migration the above reproduce doesn't lead to
>> crash on the source VM.
>>
>> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
>> rather passes it through the memory backend object, our code might
>> somehow corrupt the VRAM. However, I wasn't able to trace the
>> corruption so far.
>>
>> Could somebody help the investigation and take a look into this? Any
>> suggestions would be appreciated. Thanks!
>
> Possibly some memory region created by qxl is not being preserved.
> Try adding these traces to see what is preserved:
>
> -trace enable='*cpr*'
> -trace enable='*ram_alloc*'
Also try adding this patch to see if it flags any ram blocks as not
compatible with cpr. A message is printed at migration start time.
https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-email-steven.sistare@oracle.com/
- Steve
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-02-28 18:20 ` Steven Sistare
@ 2025-02-28 18:35 ` Andrey Drobyshev
2025-02-28 18:37 ` Andrey Drobyshev
0 siblings, 1 reply; 13+ messages in thread
From: Andrey Drobyshev @ 2025-02-28 18:35 UTC (permalink / raw)
To: Steven Sistare, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
On 2/28/25 8:20 PM, Steven Sistare wrote:
> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>> Hi all,
>>>
>>> We've been experimenting with cpr-transfer migration mode recently and
>>> have discovered the following issue with the guest QXL driver:
>>>
>>> Run migration source:
>>>> EMULATOR=/path/to/emulator
>>>> ROOTFS=/path/to/image
>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>
>>>> $EMULATOR -enable-kvm \
>>>> -machine q35 \
>>>> -cpu host -smp 2 -m 2G \
>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>> ram0,share=on\
>>>> -machine memory-backend=ram0 \
>>>> -machine aux-ram-share=on \
>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>> -nographic \
>>>> -device qxl-vga
>>>
>>> Run migration target:
>>>> EMULATOR=/path/to/emulator
>>>> ROOTFS=/path/to/image
>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>> $EMULATOR -enable-kvm \
>>>> -machine q35 \
>>>> -cpu host -smp 2 -m 2G \
>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>> ram0,share=on\
>>>> -machine memory-backend=ram0 \
>>>> -machine aux-ram-share=on \
>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>> -nographic \
>>>> -device qxl-vga \
>>>> -incoming tcp:0:44444 \
>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>
>>>
>>> Launch the migration:
>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>
>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>> migrate-set-parameters mode=cpr-transfer
>>>> migrate channels=[{"channel-type":"main","addr":
>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>> {"channel-type":"cpr","addr":
>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>> dst.sock"}}]
>>>> EOF
>>>
>>> Then, after a while, QXL guest driver on target crashes spewing the
>>> following messages:
>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>> 0x00000001)
>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>> allocate VRAM BO
>>>
>>> That seems to be a known kernel QXL driver bug:
>>>
>>> https://lore.kernel.org/all/20220907094423.93581-1-min_halo@163.com/T/
>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>
>>> (the latter discussion contains that reproduce script which speeds up
>>> the crash in the guest):
>>>> #!/bin/bash
>>>>
>>>> chvt 3
>>>>
>>>> for j in $(seq 80); do
>>>> echo "$(date) starting round $j"
>>>> if [ "$(journalctl --boot | grep "failed to allocate VRAM
>>>> BO")" != "" ]; then
>>>> echo "bug was reproduced after $j tries"
>>>> exit 1
>>>> fi
>>>> for i in $(seq 100); do
>>>> dmesg > /dev/tty3
>>>> done
>>>> done
>>>>
>>>> echo "bug could not be reproduced"
>>>> exit 0
>>>
>>> The bug itself seems to remain unfixed, as I was able to reproduce that
>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>> without the cpr-transfer migration the above reproduce doesn't lead to
>>> crash on the source VM.
>>>
>>> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
>>> rather passes it through the memory backend object, our code might
>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>> corruption so far.
>>>
>>> Could somebody help the investigation and take a look into this? Any
>>> suggestions would be appreciated. Thanks!
>>
>> Possibly some memory region created by qxl is not being preserved.
>> Try adding these traces to see what is preserved:
>>
>> -trace enable='*cpr*'
>> -trace enable='*ram_alloc*'
>
> Also try adding this patch to see if it flags any ram blocks as not
> compatible with cpr. A message is printed at migration start time.
> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-email-
> steven.sistare@oracle.com/
>
> - Steve
>
With the traces enabled + the "migration: ram block cpr blockers" patch
applied:
Source:
> cpr_find_fd pc.bios, id 0 returns -1
> cpr_save_fd pc.bios, id 0, fd 22
> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host 0x7fec18e00000
> cpr_find_fd pc.rom, id 0 returns -1
> cpr_save_fd pc.rom, id 0, fd 23
> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host 0x7fec18c00000
> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size 262144 fd 24 host 0x7fec18a00000
> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size 67108864 fd 25 host 0x7feb77e00000
> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192 fd 27 host 0x7fec18800000
> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size 67108864 fd 28 host 0x7feb73c00000
> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536 fd 34 host 0x7fec18600000
> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size 2097152 fd 35 host 0x7fec18200000
> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536 fd 36 host 0x7feb8b600000
> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd 37 host 0x7feb8b400000
>
> cpr_state_save cpr-transfer mode
> cpr_transfer_output /var/run/alma8cpr-dst.sock
Target:
> cpr_transfer_input /var/run/alma8cpr-dst.sock
> cpr_state_load cpr-transfer mode
> cpr_find_fd pc.bios, id 0 returns 20
> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host 0x7fcdc9800000
> cpr_find_fd pc.rom, id 0 returns 19
> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host 0x7fcdc9600000
> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size 262144 fd 18 host 0x7fcdc9400000
> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size 67108864 fd 17 host 0x7fcd27e00000
> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192 fd 16 host 0x7fcdc9200000
> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size 67108864 fd 15 host 0x7fcd23c00000
> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536 fd 14 host 0x7fcdc8800000
> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size 2097152 fd 13 host 0x7fcdc8400000
> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536 fd 11 host 0x7fcdc8200000
> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd 10 host 0x7fcd3be00000
Looks like both vga.vram and qxl.vram are being preserved (with the same
addresses), and no incompatible ram blocks are found during migration.
Andrey
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-02-28 18:35 ` Andrey Drobyshev
@ 2025-02-28 18:37 ` Andrey Drobyshev
2025-03-04 19:05 ` Steven Sistare
0 siblings, 1 reply; 13+ messages in thread
From: Andrey Drobyshev @ 2025-02-28 18:37 UTC (permalink / raw)
To: Steven Sistare, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
> On 2/28/25 8:20 PM, Steven Sistare wrote:
>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>> Hi all,
>>>>
>>>> We've been experimenting with cpr-transfer migration mode recently and
>>>> have discovered the following issue with the guest QXL driver:
>>>>
>>>> Run migration source:
>>>>> EMULATOR=/path/to/emulator
>>>>> ROOTFS=/path/to/image
>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>
>>>>> $EMULATOR -enable-kvm \
>>>>> -machine q35 \
>>>>> -cpu host -smp 2 -m 2G \
>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>> ram0,share=on\
>>>>> -machine memory-backend=ram0 \
>>>>> -machine aux-ram-share=on \
>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>> -nographic \
>>>>> -device qxl-vga
>>>>
>>>> Run migration target:
>>>>> EMULATOR=/path/to/emulator
>>>>> ROOTFS=/path/to/image
>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>> $EMULATOR -enable-kvm \
>>>>> -machine q35 \
>>>>> -cpu host -smp 2 -m 2G \
>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>> ram0,share=on\
>>>>> -machine memory-backend=ram0 \
>>>>> -machine aux-ram-share=on \
>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>> -nographic \
>>>>> -device qxl-vga \
>>>>> -incoming tcp:0:44444 \
>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>
>>>>
>>>> Launch the migration:
>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>
>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>> migrate-set-parameters mode=cpr-transfer
>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>> {"channel-type":"cpr","addr":
>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>> dst.sock"}}]
>>>>> EOF
>>>>
>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>> following messages:
>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>> 0x00000001)
>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>> allocate VRAM BO
>>>>
>>>> That seems to be a known kernel QXL driver bug:
>>>>
>>>> https://lore.kernel.org/all/20220907094423.93581-1-min_halo@163.com/T/
>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>
>>>> (the latter discussion contains that reproduce script which speeds up
>>>> the crash in the guest):
>>>>> #!/bin/bash
>>>>>
>>>>> chvt 3
>>>>>
>>>>> for j in $(seq 80); do
>>>>> echo "$(date) starting round $j"
>>>>> if [ "$(journalctl --boot | grep "failed to allocate VRAM
>>>>> BO")" != "" ]; then
>>>>> echo "bug was reproduced after $j tries"
>>>>> exit 1
>>>>> fi
>>>>> for i in $(seq 100); do
>>>>> dmesg > /dev/tty3
>>>>> done
>>>>> done
>>>>>
>>>>> echo "bug could not be reproduced"
>>>>> exit 0
>>>>
>>>> The bug itself seems to remain unfixed, as I was able to reproduce that
>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>> without the cpr-transfer migration the above reproduce doesn't lead to
>>>> crash on the source VM.
>>>>
>>>> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
>>>> rather passes it through the memory backend object, our code might
>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>> corruption so far.
>>>>
>>>> Could somebody help the investigation and take a look into this? Any
>>>> suggestions would be appreciated. Thanks!
>>>
>>> Possibly some memory region created by qxl is not being preserved.
>>> Try adding these traces to see what is preserved:
>>>
>>> -trace enable='*cpr*'
>>> -trace enable='*ram_alloc*'
>>
>> Also try adding this patch to see if it flags any ram blocks as not
>> compatible with cpr. A message is printed at migration start time.
>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-email-
>> steven.sistare@oracle.com/
>>
>> - Steve
>>
>
> With the traces enabled + the "migration: ram block cpr blockers" patch
> applied:
>
> Source:
>> cpr_find_fd pc.bios, id 0 returns -1
>> cpr_save_fd pc.bios, id 0, fd 22
>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host 0x7fec18e00000
>> cpr_find_fd pc.rom, id 0 returns -1
>> cpr_save_fd pc.rom, id 0, fd 23
>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host 0x7fec18c00000
>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size 262144 fd 24 host 0x7fec18a00000
>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size 67108864 fd 25 host 0x7feb77e00000
>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192 fd 27 host 0x7fec18800000
>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size 67108864 fd 28 host 0x7feb73c00000
>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536 fd 34 host 0x7fec18600000
>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size 2097152 fd 35 host 0x7fec18200000
>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536 fd 36 host 0x7feb8b600000
>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd 37 host 0x7feb8b400000
>>
>> cpr_state_save cpr-transfer mode
>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>
> Target:
>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>> cpr_state_load cpr-transfer mode
>> cpr_find_fd pc.bios, id 0 returns 20
>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host 0x7fcdc9800000
>> cpr_find_fd pc.rom, id 0 returns 19
>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host 0x7fcdc9600000
>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size 262144 fd 18 host 0x7fcdc9400000
>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size 67108864 fd 17 host 0x7fcd27e00000
>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192 fd 16 host 0x7fcdc9200000
>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size 67108864 fd 15 host 0x7fcd23c00000
>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536 fd 14 host 0x7fcdc8800000
>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size 2097152 fd 13 host 0x7fcdc8400000
>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536 fd 11 host 0x7fcdc8200000
>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd 10 host 0x7fcd3be00000
>
> Looks like both vga.vram and qxl.vram are being preserved (with the same
> addresses), and no incompatible ram blocks are found during migration.
>
Sorry, addressed are not the same, of course. However corresponding ram
blocks do seem to be preserved and initialized.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-02-28 18:37 ` Andrey Drobyshev
@ 2025-03-04 19:05 ` Steven Sistare
2025-03-05 16:50 ` Andrey Drobyshev
0 siblings, 1 reply; 13+ messages in thread
From: Steven Sistare @ 2025-03-04 19:05 UTC (permalink / raw)
To: Andrey Drobyshev, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
[-- Attachment #1: Type: text/plain, Size: 9438 bytes --]
On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>> Hi all,
>>>>>
>>>>> We've been experimenting with cpr-transfer migration mode recently and
>>>>> have discovered the following issue with the guest QXL driver:
>>>>>
>>>>> Run migration source:
>>>>>> EMULATOR=/path/to/emulator
>>>>>> ROOTFS=/path/to/image
>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>
>>>>>> $EMULATOR -enable-kvm \
>>>>>> -machine q35 \
>>>>>> -cpu host -smp 2 -m 2G \
>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>> ram0,share=on\
>>>>>> -machine memory-backend=ram0 \
>>>>>> -machine aux-ram-share=on \
>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>> -nographic \
>>>>>> -device qxl-vga
>>>>>
>>>>> Run migration target:
>>>>>> EMULATOR=/path/to/emulator
>>>>>> ROOTFS=/path/to/image
>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>> $EMULATOR -enable-kvm \
>>>>>> -machine q35 \
>>>>>> -cpu host -smp 2 -m 2G \
>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>> ram0,share=on\
>>>>>> -machine memory-backend=ram0 \
>>>>>> -machine aux-ram-share=on \
>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>> -nographic \
>>>>>> -device qxl-vga \
>>>>>> -incoming tcp:0:44444 \
>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>
>>>>>
>>>>> Launch the migration:
>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>
>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>> {"channel-type":"cpr","addr":
>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>> dst.sock"}}]
>>>>>> EOF
>>>>>
>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>> following messages:
>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>> 0x00000001)
>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>> allocate VRAM BO
>>>>>
>>>>> That seems to be a known kernel QXL driver bug:
>>>>>
>>>>> https://lore.kernel.org/all/20220907094423.93581-1-min_halo@163.com/T/
>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>
>>>>> (the latter discussion contains that reproduce script which speeds up
>>>>> the crash in the guest):
>>>>>> #!/bin/bash
>>>>>>
>>>>>> chvt 3
>>>>>>
>>>>>> for j in $(seq 80); do
>>>>>> echo "$(date) starting round $j"
>>>>>> if [ "$(journalctl --boot | grep "failed to allocate VRAM
>>>>>> BO")" != "" ]; then
>>>>>> echo "bug was reproduced after $j tries"
>>>>>> exit 1
>>>>>> fi
>>>>>> for i in $(seq 100); do
>>>>>> dmesg > /dev/tty3
>>>>>> done
>>>>>> done
>>>>>>
>>>>>> echo "bug could not be reproduced"
>>>>>> exit 0
>>>>>
>>>>> The bug itself seems to remain unfixed, as I was able to reproduce that
>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>> without the cpr-transfer migration the above reproduce doesn't lead to
>>>>> crash on the source VM.
>>>>>
>>>>> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
>>>>> rather passes it through the memory backend object, our code might
>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>> corruption so far.
>>>>>
>>>>> Could somebody help the investigation and take a look into this? Any
>>>>> suggestions would be appreciated. Thanks!
>>>>
>>>> Possibly some memory region created by qxl is not being preserved.
>>>> Try adding these traces to see what is preserved:
>>>>
>>>> -trace enable='*cpr*'
>>>> -trace enable='*ram_alloc*'
>>>
>>> Also try adding this patch to see if it flags any ram blocks as not
>>> compatible with cpr. A message is printed at migration start time.
>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-email-
>>> steven.sistare@oracle.com/
>>>
>>> - Steve
>>>
>>
>> With the traces enabled + the "migration: ram block cpr blockers" patch
>> applied:
>>
>> Source:
>>> cpr_find_fd pc.bios, id 0 returns -1
>>> cpr_save_fd pc.bios, id 0, fd 22
>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host 0x7fec18e00000
>>> cpr_find_fd pc.rom, id 0 returns -1
>>> cpr_save_fd pc.rom, id 0, fd 23
>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host 0x7fec18c00000
>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size 262144 fd 24 host 0x7fec18a00000
>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size 67108864 fd 25 host 0x7feb77e00000
>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192 fd 27 host 0x7fec18800000
>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size 67108864 fd 28 host 0x7feb73c00000
>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536 fd 34 host 0x7fec18600000
>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size 2097152 fd 35 host 0x7fec18200000
>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536 fd 36 host 0x7feb8b600000
>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd 37 host 0x7feb8b400000
>>>
>>> cpr_state_save cpr-transfer mode
>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>
>> Target:
>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>> cpr_state_load cpr-transfer mode
>>> cpr_find_fd pc.bios, id 0 returns 20
>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host 0x7fcdc9800000
>>> cpr_find_fd pc.rom, id 0 returns 19
>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host 0x7fcdc9600000
>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size 262144 fd 18 host 0x7fcdc9400000
>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size 67108864 fd 17 host 0x7fcd27e00000
>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192 fd 16 host 0x7fcdc9200000
>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size 67108864 fd 15 host 0x7fcd23c00000
>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536 fd 14 host 0x7fcdc8800000
>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size 2097152 fd 13 host 0x7fcdc8400000
>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536 fd 11 host 0x7fcdc8200000
>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd 10 host 0x7fcd3be00000
>>
>> Looks like both vga.vram and qxl.vram are being preserved (with the same
>> addresses), and no incompatible ram blocks are found during migration.
>
> Sorry, addressed are not the same, of course. However corresponding ram
> blocks do seem to be preserved and initialized.
So far, I have not reproduced the guest driver failure.
However, I have isolated places where new QEMU improperly writes to
the qxl memory regions prior to starting the guest, by mmap'ing them
readonly after cpr:
qemu_ram_alloc_internal()
if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
ram_flags |= RAM_READONLY;
new_block = qemu_ram_alloc_from_fd(...)
I have attached a draft fix; try it and let me know.
My console window looks fine before and after cpr, using
-vnc $hostip:0 -vga qxl
- Steve
[-- Attachment #2: 0001-hw-qxl-cpr-support-preliminary.patch --]
[-- Type: text/plain, Size: 2879 bytes --]
From ea448d37b845be1e3d02def788fa91e35ddff49b Mon Sep 17 00:00:00 2001
From: Steve Sistare <steven.sistare@oracle.com>
Date: Tue, 4 Mar 2025 10:47:40 -0800
Subject: [PATCH] hw/qxl: cpr support (preliminary)
During normal migration, new QEMU creates and initializes qxl ram and rom
memory regions, then loads the preserved contents of those regions from
vmstate.
During CPR, the memory regions are preserved in place, then new QEMU uses
those regions, but re-initializes them, which is wrong. Suppress writes
to the qxl memory regions during CPR load.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
hw/display/qxl.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/hw/display/qxl.c b/hw/display/qxl.c
index 2efdc77..70f7f15 100644
--- a/hw/display/qxl.c
+++ b/hw/display/qxl.c
@@ -30,6 +30,7 @@
#include "qemu/module.h"
#include "hw/qdev-properties.h"
#include "system/runstate.h"
+#include "migration/cpr.h"
#include "migration/vmstate.h"
#include "trace.h"
@@ -333,6 +334,10 @@ static void init_qxl_rom(PCIQXLDevice *d)
uint32_t fb;
int i, n;
+ if (cpr_get_incoming_mode() != MIG_MODE_NONE) {
+ goto skip_init;
+ }
+
memset(rom, 0, d->rom_size);
rom->magic = cpu_to_le32(QXL_ROM_MAGIC);
@@ -390,6 +395,7 @@ static void init_qxl_rom(PCIQXLDevice *d)
sizeof(rom->client_monitors_config));
}
+skip_init:
d->shadow_rom = *rom;
d->rom = rom;
d->modes = modes;
@@ -539,7 +545,9 @@ static void interface_set_compression_level(QXLInstance *sin, int level)
trace_qxl_interface_set_compression_level(qxl->id, level);
qxl->shadow_rom.compression_level = cpu_to_le32(level);
- qxl->rom->compression_level = cpu_to_le32(level);
+ if (cpr_get_incoming_mode() == MIG_MODE_NONE) {
+ qxl->rom->compression_level = cpu_to_le32(level);
+ }
qxl_rom_set_dirty(qxl);
}
@@ -1245,7 +1253,7 @@ static void qxl_hard_reset(PCIQXLDevice *d, int loadvm)
/* pre loadvm reset must not touch QXLRam. This lives in
* device memory, is migrated together with RAM and thus
* already loaded at this point */
- if (!loadvm) {
+ if (!loadvm && cpr_get_incoming_mode() == MIG_MODE_NONE) {
qxl_reset_state(d);
}
qemu_spice_create_host_memslot(&d->ssd);
@@ -2206,7 +2214,9 @@ static void qxl_realize_common(PCIQXLDevice *qxl, Error **errp)
qxl->update_irq = qemu_bh_new_guarded(qxl_update_irq_bh, qxl,
&DEVICE(qxl)->mem_reentrancy_guard);
- qxl_reset_state(qxl);
+ if (cpr_get_incoming_mode() == MIG_MODE_NONE) {
+ qxl_reset_state(qxl);
+ }
qxl->update_area_bh = qemu_bh_new_guarded(qxl_render_update_area_bh, qxl,
&DEVICE(qxl)->mem_reentrancy_guard);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-04 19:05 ` Steven Sistare
@ 2025-03-05 16:50 ` Andrey Drobyshev
2025-03-05 21:19 ` Steven Sistare
0 siblings, 1 reply; 13+ messages in thread
From: Andrey Drobyshev @ 2025-03-05 16:50 UTC (permalink / raw)
To: Steven Sistare, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
[-- Attachment #1: Type: text/plain, Size: 13553 bytes --]
On 3/4/25 9:05 PM, Steven Sistare wrote:
> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>> and
>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>
>>>>>> Run migration source:
>>>>>>> EMULATOR=/path/to/emulator
>>>>>>> ROOTFS=/path/to/image
>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>
>>>>>>> $EMULATOR -enable-kvm \
>>>>>>> -machine q35 \
>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>>> ram0,share=on\
>>>>>>> -machine memory-backend=ram0 \
>>>>>>> -machine aux-ram-share=on \
>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>> -nographic \
>>>>>>> -device qxl-vga
>>>>>>
>>>>>> Run migration target:
>>>>>>> EMULATOR=/path/to/emulator
>>>>>>> ROOTFS=/path/to/image
>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>> $EMULATOR -enable-kvm \
>>>>>>> -machine q35 \
>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>>> ram0,share=on\
>>>>>>> -machine memory-backend=ram0 \
>>>>>>> -machine aux-ram-share=on \
>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>> -nographic \
>>>>>>> -device qxl-vga \
>>>>>>> -incoming tcp:0:44444 \
>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>
>>>>>>
>>>>>> Launch the migration:
>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>
>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>> {"channel-type":"cpr","addr":
>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>> dst.sock"}}]
>>>>>>> EOF
>>>>>>
>>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>>> following messages:
>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>> 0x00000001)
>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>>> allocate VRAM BO
>>>>>>
>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>
>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>> min_halo@163.com/T/
>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>
>>>>>> (the latter discussion contains that reproduce script which speeds up
>>>>>> the crash in the guest):
>>>>>>> #!/bin/bash
>>>>>>>
>>>>>>> chvt 3
>>>>>>>
>>>>>>> for j in $(seq 80); do
>>>>>>> echo "$(date) starting round $j"
>>>>>>> if [ "$(journalctl --boot | grep "failed to allocate VRAM
>>>>>>> BO")" != "" ]; then
>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>> exit 1
>>>>>>> fi
>>>>>>> for i in $(seq 100); do
>>>>>>> dmesg > /dev/tty3
>>>>>>> done
>>>>>>> done
>>>>>>>
>>>>>>> echo "bug could not be reproduced"
>>>>>>> exit 0
>>>>>>
>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>> that
>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>> lead to
>>>>>> crash on the source VM.
>>>>>>
>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
>>>>>> rather passes it through the memory backend object, our code might
>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>> corruption so far.
>>>>>>
>>>>>> Could somebody help the investigation and take a look into this? Any
>>>>>> suggestions would be appreciated. Thanks!
>>>>>
>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>> Try adding these traces to see what is preserved:
>>>>>
>>>>> -trace enable='*cpr*'
>>>>> -trace enable='*ram_alloc*'
>>>>
>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>> compatible with cpr. A message is printed at migration start time.
>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>> email-
>>>> steven.sistare@oracle.com/
>>>>
>>>> - Steve
>>>>
>>>
>>> With the traces enabled + the "migration: ram block cpr blockers" patch
>>> applied:
>>>
>>> Source:
>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>> 0x7fec18e00000
>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>> 0x7fec18c00000
>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>> 262144 fd 24 host 0x7fec18a00000
>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>> 67108864 fd 25 host 0x7feb77e00000
>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>> fd 27 host 0x7fec18800000
>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>> 67108864 fd 28 host 0x7feb73c00000
>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>> fd 34 host 0x7fec18600000
>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>> 2097152 fd 35 host 0x7fec18200000
>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>> fd 36 host 0x7feb8b600000
>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>> 37 host 0x7feb8b400000
>>>>
>>>> cpr_state_save cpr-transfer mode
>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>
>>> Target:
>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>> cpr_state_load cpr-transfer mode
>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>> 0x7fcdc9800000
>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>> 0x7fcdc9600000
>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>> 262144 fd 18 host 0x7fcdc9400000
>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>> fd 16 host 0x7fcdc9200000
>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>> fd 14 host 0x7fcdc8800000
>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>> fd 11 host 0x7fcdc8200000
>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>> 10 host 0x7fcd3be00000
>>>
>>> Looks like both vga.vram and qxl.vram are being preserved (with the same
>>> addresses), and no incompatible ram blocks are found during migration.
>>
>> Sorry, addressed are not the same, of course. However corresponding ram
>> blocks do seem to be preserved and initialized.
>
> So far, I have not reproduced the guest driver failure.
>
> However, I have isolated places where new QEMU improperly writes to
> the qxl memory regions prior to starting the guest, by mmap'ing them
> readonly after cpr:
>
> qemu_ram_alloc_internal()
> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
> ram_flags |= RAM_READONLY;
> new_block = qemu_ram_alloc_from_fd(...)
>
> I have attached a draft fix; try it and let me know.
> My console window looks fine before and after cpr, using
> -vnc $hostip:0 -vga qxl
>
> - Steve
Regarding the reproduce: when I launch the buggy version with the same
options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
my VNC client silently hangs on the target after a while. Could it
happen on your stand as well? Could you try launching VM with
"-nographic -device qxl-vga"? That way VM's serial console is given you
directly in the shell, so when qxl driver crashes you're still able to
inspect the kernel messages.
As for your patch, I can report that it doesn't resolve the issue as it
is. But I was able to track down another possible memory corruption
using your approach with readonly mmap'ing:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
> (gdb) bt
> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70, errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70, errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70, errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70, value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70, v=0x5638996f3770, name=0x56389759b141 "realized", opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
> at ../qom/object.c:2374
> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70, name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
> at ../qom/object.c:1449
> #7 0x00005638970f8586 in object_property_set_qobject (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=0x5638996df900, errp=0x7ffd3c2b84e0)
> at ../qom/qom-qobject.c:28
> #8 0x00005638970f3d8d in object_property_set_bool (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true, errp=0x7ffd3c2b84e0)
> at ../qom/object.c:1519
> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70, bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
> #10 0x0000563896dba675 in qdev_device_add_from_qdict (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at ../system/qdev-monitor.c:714
> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0, opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/vl.c:1207
> #13 0x000056389737a6cc in qemu_opts_foreach
> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
> at ../util/qemu-option.c:1135
> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at ../system/vl.c:2745
> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40 <error_fatal>) at ../system/vl.c:2806
> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948) at ../system/vl.c:3838
> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at ../system/main.c:72
So the attached adjusted version of your patch does seem to help. At
least I can't reproduce the crash on my stand.
I'm wondering, could it be useful to explicitly mark all the reused
memory regions readonly upon cpr-transfer, and then make them writable
back again after the migration is done? That way we will be segfaulting
early on instead of debugging tricky memory corruptions.
Andrey
[-- Attachment #2: 0001-hw-qxl-cpr-support-preliminary.patch --]
[-- Type: text/x-patch, Size: 4317 bytes --]
From eed0321c6938204cc639ce79cdd93e311cd63a3d Mon Sep 17 00:00:00 2001
From: Steve Sistare <steven.sistare@oracle.com>
Date: Tue, 4 Mar 2025 10:47:40 -0800
Subject: [PATCH] hw/qxl: cpr support (preliminary)
During normal migration, new QEMU creates and initializes qxl ram and rom
memory regions, then loads the preserved contents of those regions from
vmstate.
During CPR, the memory regions are preserved in place, then new QEMU uses
those regions, but re-initializes them, which is wrong. Suppress writes
to the qxl memory regions during CPR load.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
hw/display/qxl.c | 42 +++++++++++++++++++++++++++++-------------
1 file changed, 29 insertions(+), 13 deletions(-)
diff --git a/hw/display/qxl.c b/hw/display/qxl.c
index 2efdc77e61..c9a2ac67e3 100644
--- a/hw/display/qxl.c
+++ b/hw/display/qxl.c
@@ -30,6 +30,7 @@
#include "qemu/module.h"
#include "hw/qdev-properties.h"
#include "system/runstate.h"
+#include "migration/cpr.h"
#include "migration/vmstate.h"
#include "trace.h"
@@ -333,6 +334,10 @@ static void init_qxl_rom(PCIQXLDevice *d)
uint32_t fb;
int i, n;
+ if (cpr_get_incoming_mode() != MIG_MODE_NONE) {
+ goto skip_init;
+ }
+
memset(rom, 0, d->rom_size);
rom->magic = cpu_to_le32(QXL_ROM_MAGIC);
@@ -390,6 +395,7 @@ static void init_qxl_rom(PCIQXLDevice *d)
sizeof(rom->client_monitors_config));
}
+skip_init:
d->shadow_rom = *rom;
d->rom = rom;
d->modes = modes;
@@ -400,24 +406,30 @@ static void init_qxl_ram(PCIQXLDevice *d)
uint8_t *buf;
uint32_t prod;
QXLReleaseRing *ring;
+ bool cpr_transfer = (cpr_get_incoming_mode() != MIG_MODE_NONE);
buf = d->vga.vram_ptr;
d->ram = (QXLRam *)(buf + le32_to_cpu(d->shadow_rom.ram_header_offset));
- d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
- d->ram->int_pending = cpu_to_le32(0);
- d->ram->int_mask = cpu_to_le32(0);
- d->ram->update_surface = 0;
- d->ram->monitors_config = 0;
- SPICE_RING_INIT(&d->ram->cmd_ring);
- SPICE_RING_INIT(&d->ram->cursor_ring);
- SPICE_RING_INIT(&d->ram->release_ring);
+
+ if (!cpr_transfer) {
+ d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
+ d->ram->int_pending = cpu_to_le32(0);
+ d->ram->int_mask = cpu_to_le32(0);
+ d->ram->update_surface = 0;
+ d->ram->monitors_config = 0;
+ SPICE_RING_INIT(&d->ram->cmd_ring);
+ SPICE_RING_INIT(&d->ram->cursor_ring);
+ SPICE_RING_INIT(&d->ram->release_ring);
+ }
ring = &d->ram->release_ring;
prod = ring->prod & SPICE_RING_INDEX_MASK(ring);
assert(prod < ARRAY_SIZE(ring->items));
- ring->items[prod].el = 0;
- qxl_ring_set_dirty(d);
+ if (!cpr_transfer) {
+ ring->items[prod].el = 0;
+ qxl_ring_set_dirty(d);
+ }
}
/* can be called from spice server thread context */
@@ -539,7 +551,9 @@ static void interface_set_compression_level(QXLInstance *sin, int level)
trace_qxl_interface_set_compression_level(qxl->id, level);
qxl->shadow_rom.compression_level = cpu_to_le32(level);
- qxl->rom->compression_level = cpu_to_le32(level);
+ if (cpr_get_incoming_mode() == MIG_MODE_NONE) {
+ qxl->rom->compression_level = cpu_to_le32(level);
+ }
qxl_rom_set_dirty(qxl);
}
@@ -1245,7 +1259,7 @@ static void qxl_hard_reset(PCIQXLDevice *d, int loadvm)
/* pre loadvm reset must not touch QXLRam. This lives in
* device memory, is migrated together with RAM and thus
* already loaded at this point */
- if (!loadvm) {
+ if (!loadvm && cpr_get_incoming_mode() == MIG_MODE_NONE) {
qxl_reset_state(d);
}
qemu_spice_create_host_memslot(&d->ssd);
@@ -2206,7 +2220,9 @@ static void qxl_realize_common(PCIQXLDevice *qxl, Error **errp)
qxl->update_irq = qemu_bh_new_guarded(qxl_update_irq_bh, qxl,
&DEVICE(qxl)->mem_reentrancy_guard);
- qxl_reset_state(qxl);
+ if (cpr_get_incoming_mode() == MIG_MODE_NONE) {
+ qxl_reset_state(qxl);
+ }
qxl->update_area_bh = qemu_bh_new_guarded(qxl_render_update_area_bh, qxl,
&DEVICE(qxl)->mem_reentrancy_guard);
--
2.43.5
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-05 16:50 ` Andrey Drobyshev
@ 2025-03-05 21:19 ` Steven Sistare
2025-03-06 9:59 ` Denis V. Lunev
2025-03-06 15:16 ` Andrey Drobyshev
0 siblings, 2 replies; 13+ messages in thread
From: Steven Sistare @ 2025-03-05 21:19 UTC (permalink / raw)
To: Andrey Drobyshev, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
[-- Attachment #1: Type: text/plain, Size: 14972 bytes --]
On 3/5/2025 11:50 AM, Andrey Drobyshev wrote:
> On 3/4/25 9:05 PM, Steven Sistare wrote:
>> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>>> and
>>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>>
>>>>>>> Run migration source:
>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>> ROOTFS=/path/to/image
>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>
>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>> -machine q35 \
>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>>>> ram0,share=on\
>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>> -machine aux-ram-share=on \
>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>> -nographic \
>>>>>>>> -device qxl-vga
>>>>>>>
>>>>>>> Run migration target:
>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>> ROOTFS=/path/to/image
>>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>> -machine q35 \
>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>>>> ram0,share=on\
>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>> -machine aux-ram-share=on \
>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>> -nographic \
>>>>>>>> -device qxl-vga \
>>>>>>>> -incoming tcp:0:44444 \
>>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>>
>>>>>>>
>>>>>>> Launch the migration:
>>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>
>>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>>> {"channel-type":"cpr","addr":
>>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>>> dst.sock"}}]
>>>>>>>> EOF
>>>>>>>
>>>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>>>> following messages:
>>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>>> 0x00000001)
>>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>>>> allocate VRAM BO
>>>>>>>
>>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>>
>>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>>> min_halo@163.com/T/
>>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>>
>>>>>>> (the latter discussion contains that reproduce script which speeds up
>>>>>>> the crash in the guest):
>>>>>>>> #!/bin/bash
>>>>>>>>
>>>>>>>> chvt 3
>>>>>>>>
>>>>>>>> for j in $(seq 80); do
>>>>>>>> echo "$(date) starting round $j"
>>>>>>>> if [ "$(journalctl --boot | grep "failed to allocate VRAM
>>>>>>>> BO")" != "" ]; then
>>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>>> exit 1
>>>>>>>> fi
>>>>>>>> for i in $(seq 100); do
>>>>>>>> dmesg > /dev/tty3
>>>>>>>> done
>>>>>>>> done
>>>>>>>>
>>>>>>>> echo "bug could not be reproduced"
>>>>>>>> exit 0
>>>>>>>
>>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>>> that
>>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>>> lead to
>>>>>>> crash on the source VM.
>>>>>>>
>>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest memory, but
>>>>>>> rather passes it through the memory backend object, our code might
>>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>>> corruption so far.
>>>>>>>
>>>>>>> Could somebody help the investigation and take a look into this? Any
>>>>>>> suggestions would be appreciated. Thanks!
>>>>>>
>>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>>> Try adding these traces to see what is preserved:
>>>>>>
>>>>>> -trace enable='*cpr*'
>>>>>> -trace enable='*ram_alloc*'
>>>>>
>>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>>> compatible with cpr. A message is printed at migration start time.
>>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>>> email-
>>>>> steven.sistare@oracle.com/
>>>>>
>>>>> - Steve
>>>>>
>>>>
>>>> With the traces enabled + the "migration: ram block cpr blockers" patch
>>>> applied:
>>>>
>>>> Source:
>>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>>> 0x7fec18e00000
>>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>>> 0x7fec18c00000
>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>> 262144 fd 24 host 0x7fec18a00000
>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>> 67108864 fd 25 host 0x7feb77e00000
>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>> fd 27 host 0x7fec18800000
>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>> 67108864 fd 28 host 0x7feb73c00000
>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>> fd 34 host 0x7fec18600000
>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>> 2097152 fd 35 host 0x7fec18200000
>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>> fd 36 host 0x7feb8b600000
>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>> 37 host 0x7feb8b400000
>>>>>
>>>>> cpr_state_save cpr-transfer mode
>>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>>
>>>> Target:
>>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>>> cpr_state_load cpr-transfer mode
>>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>>> 0x7fcdc9800000
>>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>>> 0x7fcdc9600000
>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>> 262144 fd 18 host 0x7fcdc9400000
>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>> fd 16 host 0x7fcdc9200000
>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>> fd 14 host 0x7fcdc8800000
>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>> fd 11 host 0x7fcdc8200000
>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>> 10 host 0x7fcd3be00000
>>>>
>>>> Looks like both vga.vram and qxl.vram are being preserved (with the same
>>>> addresses), and no incompatible ram blocks are found during migration.
>>>
>>> Sorry, addressed are not the same, of course. However corresponding ram
>>> blocks do seem to be preserved and initialized.
>>
>> So far, I have not reproduced the guest driver failure.
>>
>> However, I have isolated places where new QEMU improperly writes to
>> the qxl memory regions prior to starting the guest, by mmap'ing them
>> readonly after cpr:
>>
>> qemu_ram_alloc_internal()
>> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
>> ram_flags |= RAM_READONLY;
>> new_block = qemu_ram_alloc_from_fd(...)
>>
>> I have attached a draft fix; try it and let me know.
>> My console window looks fine before and after cpr, using
>> -vnc $hostip:0 -vga qxl
>>
>> - Steve
>
> Regarding the reproduce: when I launch the buggy version with the same
> options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
> my VNC client silently hangs on the target after a while. Could it
> happen on your stand as well?
cpr does not preserve the vnc connection and session. To test, I specify
port 0 for the source VM and port 1 for the dest. When the src vnc goes
dormant the dest vnc becomes active.
> Could you try launching VM with
> "-nographic -device qxl-vga"? That way VM's serial console is given you
> directly in the shell, so when qxl driver crashes you're still able to
> inspect the kernel messages.
I have been running like that, but have not reproduced the qxl driver crash,
and I suspect my guest image+kernel is too old. However, once I realized the
issue was post-cpr modification of qxl memory, I switched my attention to the
fix.
> As for your patch, I can report that it doesn't resolve the issue as it
> is. But I was able to track down another possible memory corruption
> using your approach with readonly mmap'ing:
>
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
>> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
>> (gdb) bt
>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70, errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
>> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70, errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
>> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70, errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
>> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70, value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
>> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70, v=0x5638996f3770, name=0x56389759b141 "realized", opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
>> at ../qom/object.c:2374
>> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70, name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
>> at ../qom/object.c:1449
>> #7 0x00005638970f8586 in object_property_set_qobject (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=0x5638996df900, errp=0x7ffd3c2b84e0)
>> at ../qom/qom-qobject.c:28
>> #8 0x00005638970f3d8d in object_property_set_bool (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true, errp=0x7ffd3c2b84e0)
>> at ../qom/object.c:1519
>> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70, bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
>> #10 0x0000563896dba675 in qdev_device_add_from_qdict (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at ../system/qdev-monitor.c:714
>> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
>> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0, opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/vl.c:1207
>> #13 0x000056389737a6cc in qemu_opts_foreach
>> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
>> at ../util/qemu-option.c:1135
>> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at ../system/vl.c:2745
>> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40 <error_fatal>) at ../system/vl.c:2806
>> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948) at ../system/vl.c:3838
>> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at ../system/main.c:72
>
> So the attached adjusted version of your patch does seem to help. At
> least I can't reproduce the crash on my stand.
Thanks for the stack trace; the calls to SPICE_RING_INIT in init_qxl_ram are
definitely harmful. Try V2 of the patch, attached, which skips the lines
of init_qxl_ram that modify guest memory.
> I'm wondering, could it be useful to explicitly mark all the reused
> memory regions readonly upon cpr-transfer, and then make them writable
> back again after the migration is done? That way we will be segfaulting
> early on instead of debugging tricky memory corruptions.
It's a useful debugging technique, but changing protection on a large memory region
can be too expensive for production due to TLB shootdowns.
Also, there are cases where writes are performed but the value is guaranteed to
be the same:
qxl_post_load()
qxl_set_mode()
d->rom->mode = cpu_to_le32(modenr);
The value is the same because mode and shadow_rom.mode were passed in vmstate
from old qemu.
- Steve
[-- Attachment #2: 0001-hw-qxl-cpr-support-preliminary-V2.patch --]
[-- Type: text/plain, Size: 3773 bytes --]
From a3848b48b65aa92751378c5bbc74d7ae4bb64932 Mon Sep 17 00:00:00 2001
From: Steve Sistare <steven.sistare@oracle.com>
Date: Tue, 4 Mar 2025 10:47:40 -0800
Subject: [PATCH] hw/qxl: cpr support (preliminary V2)
During normal migration, new QEMU creates and initializes qxl ram and rom
memory regions, then loads the preserved contents of those regions from
vmstate.
During CPR, the memory regions are preserved in place, then new QEMU uses
those regions, but re-initializes them, which is wrong. Suppress writes
to the qxl memory regions during CPR load.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
---
hw/display/qxl.c | 29 ++++++++++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
diff --git a/hw/display/qxl.c b/hw/display/qxl.c
index 2efdc77..b1c9fc1 100644
--- a/hw/display/qxl.c
+++ b/hw/display/qxl.c
@@ -30,6 +30,7 @@
#include "qemu/module.h"
#include "hw/qdev-properties.h"
#include "system/runstate.h"
+#include "migration/cpr.h"
#include "migration/vmstate.h"
#include "trace.h"
@@ -115,6 +116,8 @@ static QXLMode qxl_modes[] = {
QXL_MODE_EX(8192, 4320), /* 8k */
};
+#define CPR_IN (cpr_get_incoming_mode() != MIG_MODE_NONE)
+
static void qxl_send_events(PCIQXLDevice *d, uint32_t events);
static int qxl_destroy_primary(PCIQXLDevice *d, qxl_async_io async);
static void qxl_reset_memslots(PCIQXLDevice *d);
@@ -333,6 +336,10 @@ static void init_qxl_rom(PCIQXLDevice *d)
uint32_t fb;
int i, n;
+ if (CPR_IN) {
+ goto skip_init;
+ }
+
memset(rom, 0, d->rom_size);
rom->magic = cpu_to_le32(QXL_ROM_MAGIC);
@@ -390,6 +397,7 @@ static void init_qxl_rom(PCIQXLDevice *d)
sizeof(rom->client_monitors_config));
}
+skip_init:
d->shadow_rom = *rom;
d->rom = rom;
d->modes = modes;
@@ -403,6 +411,9 @@ static void init_qxl_ram(PCIQXLDevice *d)
buf = d->vga.vram_ptr;
d->ram = (QXLRam *)(buf + le32_to_cpu(d->shadow_rom.ram_header_offset));
+ if (CPR_IN) {
+ return;
+ }
d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
d->ram->int_pending = cpu_to_le32(0);
d->ram->int_mask = cpu_to_le32(0);
@@ -539,6 +550,10 @@ static void interface_set_compression_level(QXLInstance *sin, int level)
trace_qxl_interface_set_compression_level(qxl->id, level);
qxl->shadow_rom.compression_level = cpu_to_le32(level);
+ if (CPR_IN) {
+ assert(qxl->rom->compression_level == cpu_to_le32(level));
+ return;
+ }
qxl->rom->compression_level = cpu_to_le32(level);
qxl_rom_set_dirty(qxl);
}
@@ -997,7 +1012,8 @@ static void interface_set_client_capabilities(QXLInstance *sin,
}
if (runstate_check(RUN_STATE_INMIGRATE) ||
- runstate_check(RUN_STATE_POSTMIGRATE)) {
+ runstate_check(RUN_STATE_POSTMIGRATE) ||
+ CPR_IN) {
return;
}
@@ -1200,6 +1216,10 @@ static void qxl_reset_state(PCIQXLDevice *d)
{
QXLRom *rom = d->rom;
+ if (CPR_IN) {
+ return;
+ }
+
qxl_check_state(d);
d->shadow_rom.update_id = cpu_to_le32(0);
*rom = d->shadow_rom;
@@ -1370,8 +1390,11 @@ static int qxl_add_memslot(PCIQXLDevice *d, uint32_t slot_id, uint64_t delta,
memslot.virt_start = virt_start + (guest_start - pci_start);
memslot.virt_end = virt_start + (guest_end - pci_start);
memslot.addr_delta = memslot.virt_start - delta;
- memslot.generation = d->rom->slot_generation = 0;
- qxl_rom_set_dirty(d);
+ if (!CPR_IN) {
+ d->rom->slot_generation = 0;
+ qxl_rom_set_dirty(d);
+ }
+ memslot.generation = d->rom->slot_generation;
qemu_spice_add_memslot(&d->ssd, &memslot, async);
d->guest_slots[slot_id].mr = mr;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-05 21:19 ` Steven Sistare
@ 2025-03-06 9:59 ` Denis V. Lunev
2025-03-06 15:16 ` Andrey Drobyshev
1 sibling, 0 replies; 13+ messages in thread
From: Denis V. Lunev @ 2025-03-06 9:59 UTC (permalink / raw)
To: Steven Sistare, Andrey Drobyshev, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé
On 3/5/25 22:19, Steven Sistare wrote:
> On 3/5/2025 11:50 AM, Andrey Drobyshev wrote:
>> On 3/4/25 9:05 PM, Steven Sistare wrote:
>>> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>>>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>>>> and
>>>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>>>
>>>>>>>> Run migration source:
>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>
>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>> -machine q35 \
>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>> -object
>>>>>>>>> memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>>>>> ram0,share=on\
>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>> -nographic \
>>>>>>>>> -device qxl-vga
>>>>>>>>
>>>>>>>> Run migration target:
>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>> -machine q35 \
>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>> -object
>>>>>>>>> memory-backend-file,id=ram0,size=2G,mem-path=/dev/shm/
>>>>>>>>> ram0,share=on\
>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>> -nographic \
>>>>>>>>> -device qxl-vga \
>>>>>>>>> -incoming tcp:0:44444 \
>>>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>>>
>>>>>>>>
>>>>>>>> Launch the migration:
>>>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>
>>>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>>>> {"channel-type":"cpr","addr":
>>>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>>>> dst.sock"}}]
>>>>>>>>> EOF
>>>>>>>>
>>>>>>>> Then, after a while, QXL guest driver on target crashes spewing
>>>>>>>> the
>>>>>>>> following messages:
>>>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>>>> 0x00000001)
>>>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR*
>>>>>>>>> failed to
>>>>>>>>> allocate VRAM BO
>>>>>>>>
>>>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>>>
>>>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>>>> min_halo@163.com/T/
>>>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>>>
>>>>>>>> (the latter discussion contains that reproduce script which
>>>>>>>> speeds up
>>>>>>>> the crash in the guest):
>>>>>>>>> #!/bin/bash
>>>>>>>>>
>>>>>>>>> chvt 3
>>>>>>>>>
>>>>>>>>> for j in $(seq 80); do
>>>>>>>>> echo "$(date) starting round $j"
>>>>>>>>> if [ "$(journalctl --boot | grep "failed to
>>>>>>>>> allocate VRAM
>>>>>>>>> BO")" != "" ]; then
>>>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>>>> exit 1
>>>>>>>>> fi
>>>>>>>>> for i in $(seq 100); do
>>>>>>>>> dmesg > /dev/tty3
>>>>>>>>> done
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> echo "bug could not be reproduced"
>>>>>>>>> exit 0
>>>>>>>>
>>>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>>>> that
>>>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>>>> cpr-transfer code also seems to be buggy as it triggers the
>>>>>>>> crash -
>>>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>>>> lead to
>>>>>>>> crash on the source VM.
>>>>>>>>
>>>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest
>>>>>>>> memory, but
>>>>>>>> rather passes it through the memory backend object, our code might
>>>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>>>> corruption so far.
>>>>>>>>
>>>>>>>> Could somebody help the investigation and take a look into
>>>>>>>> this? Any
>>>>>>>> suggestions would be appreciated. Thanks!
>>>>>>>
>>>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>>>> Try adding these traces to see what is preserved:
>>>>>>>
>>>>>>> -trace enable='*cpr*'
>>>>>>> -trace enable='*ram_alloc*'
>>>>>>
>>>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>>>> compatible with cpr. A message is printed at migration start time.
>>>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>>>> email-
>>>>>> steven.sistare@oracle.com/
>>>>>>
>>>>>> - Steve
>>>>>>
>>>>>
>>>>> With the traces enabled + the "migration: ram block cpr blockers"
>>>>> patch
>>>>> applied:
>>>>>
>>>>> Source:
>>>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>>>> 0x7fec18e00000
>>>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>>>> 0x7fec18c00000
>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>> 262144 fd 24 host 0x7fec18a00000
>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>> 67108864 fd 25 host 0x7feb77e00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>> fd 27 host 0x7fec18800000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>> 67108864 fd 28 host 0x7feb73c00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>> fd 34 host 0x7fec18600000
>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>> 2097152 fd 35 host 0x7fec18200000
>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>> fd 36 host 0x7feb8b600000
>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>> 37 host 0x7feb8b400000
>>>>>>
>>>>>> cpr_state_save cpr-transfer mode
>>>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>>>
>>>>> Target:
>>>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>>>> cpr_state_load cpr-transfer mode
>>>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>>>> 0x7fcdc9800000
>>>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>>>> 0x7fcdc9600000
>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>> 262144 fd 18 host 0x7fcdc9400000
>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>> fd 16 host 0x7fcdc9200000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>> fd 14 host 0x7fcdc8800000
>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>> fd 11 host 0x7fcdc8200000
>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>> 10 host 0x7fcd3be00000
>>>>>
>>>>> Looks like both vga.vram and qxl.vram are being preserved (with
>>>>> the same
>>>>> addresses), and no incompatible ram blocks are found during
>>>>> migration.
>>>>
>>>> Sorry, addressed are not the same, of course. However
>>>> corresponding ram
>>>> blocks do seem to be preserved and initialized.
>>>
>>> So far, I have not reproduced the guest driver failure.
>>>
>>> However, I have isolated places where new QEMU improperly writes to
>>> the qxl memory regions prior to starting the guest, by mmap'ing them
>>> readonly after cpr:
>>>
>>> qemu_ram_alloc_internal()
>>> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
>>> ram_flags |= RAM_READONLY;
>>> new_block = qemu_ram_alloc_from_fd(...)
>>>
>>> I have attached a draft fix; try it and let me know.
>>> My console window looks fine before and after cpr, using
>>> -vnc $hostip:0 -vga qxl
>>>
>>> - Steve
>>
>> Regarding the reproduce: when I launch the buggy version with the same
>> options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
>> my VNC client silently hangs on the target after a while. Could it
>> happen on your stand as well?
>
> cpr does not preserve the vnc connection and session. To test, I specify
> port 0 for the source VM and port 1 for the dest. When the src vnc goes
> dormant the dest vnc becomes active.
>
>> Could you try launching VM with
>> "-nographic -device qxl-vga"? That way VM's serial console is given you
>> directly in the shell, so when qxl driver crashes you're still able to
>> inspect the kernel messages.
>
> I have been running like that, but have not reproduced the qxl driver
> crash,
> and I suspect my guest image+kernel is too old. However, once I
> realized the
> issue was post-cpr modification of qxl memory, I switched my attention
> to the
> fix.
>
>> As for your patch, I can report that it doesn't resolve the issue as it
>> is. But I was able to track down another possible memory corruption
>> using your approach with readonly mmap'ing:
>>
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
>>> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
>>> (gdb) bt
>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70,
>>> errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
>>> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70,
>>> errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
>>> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70,
>>> errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
>>> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70,
>>> value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
>>> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70,
>>> v=0x5638996f3770, name=0x56389759b141 "realized",
>>> opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
>>> at ../qom/object.c:2374
>>> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70,
>>> name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
>>> at ../qom/object.c:1449
>>> #7 0x00005638970f8586 in object_property_set_qobject
>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized",
>>> value=0x5638996df900, errp=0x7ffd3c2b84e0)
>>> at ../qom/qom-qobject.c:28
>>> #8 0x00005638970f3d8d in object_property_set_bool
>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true,
>>> errp=0x7ffd3c2b84e0)
>>> at ../qom/object.c:1519
>>> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70,
>>> bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
>>> #10 0x0000563896dba675 in qdev_device_add_from_qdict
>>> (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at
>>> ../system/qdev-monitor.c:714
>>> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150,
>>> errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
>>> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0,
>>> opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at
>>> ../system/vl.c:1207
>>> #13 0x000056389737a6cc in qemu_opts_foreach
>>> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca
>>> <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
>>> at ../util/qemu-option.c:1135
>>> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at
>>> ../system/vl.c:2745
>>> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40
>>> <error_fatal>) at ../system/vl.c:2806
>>> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948)
>>> at ../system/vl.c:3838
>>> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at
>>> ../system/main.c:72
>>
>> So the attached adjusted version of your patch does seem to help. At
>> least I can't reproduce the crash on my stand.
>
> Thanks for the stack trace; the calls to SPICE_RING_INIT in
> init_qxl_ram are
> definitely harmful. Try V2 of the patch, attached, which skips the lines
> of init_qxl_ram that modify guest memory.
>
>> I'm wondering, could it be useful to explicitly mark all the reused
>> memory regions readonly upon cpr-transfer, and then make them writable
>> back again after the migration is done? That way we will be segfaulting
>> early on instead of debugging tricky memory corruptions.
>
> It's a useful debugging technique, but changing protection on a large
> memory region
> can be too expensive for production due to TLB shootdowns.
>
Good point. Though we could move this code under non-default option to
avoid re-writing.
Den
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-05 21:19 ` Steven Sistare
2025-03-06 9:59 ` Denis V. Lunev
@ 2025-03-06 15:16 ` Andrey Drobyshev
2025-03-06 15:52 ` Denis V. Lunev
1 sibling, 1 reply; 13+ messages in thread
From: Andrey Drobyshev @ 2025-03-06 15:16 UTC (permalink / raw)
To: Steven Sistare, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé,
Denis V. Lunev
On 3/5/25 11:19 PM, Steven Sistare wrote:
> On 3/5/2025 11:50 AM, Andrey Drobyshev wrote:
>> On 3/4/25 9:05 PM, Steven Sistare wrote:
>>> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>>>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>>>> and
>>>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>>>
>>>>>>>> Run migration source:
>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>
>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>> -machine q35 \
>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>> dev/shm/
>>>>>>>>> ram0,share=on\
>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>> -nographic \
>>>>>>>>> -device qxl-vga
>>>>>>>>
>>>>>>>> Run migration target:
>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>> -machine q35 \
>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>> dev/shm/
>>>>>>>>> ram0,share=on\
>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>> -nographic \
>>>>>>>>> -device qxl-vga \
>>>>>>>>> -incoming tcp:0:44444 \
>>>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>>>
>>>>>>>>
>>>>>>>> Launch the migration:
>>>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>
>>>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>>>> {"channel-type":"cpr","addr":
>>>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>>>> dst.sock"}}]
>>>>>>>>> EOF
>>>>>>>>
>>>>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>>>>> following messages:
>>>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>>>> 0x00000001)
>>>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>>>>> allocate VRAM BO
>>>>>>>>
>>>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>>>
>>>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>>>> min_halo@163.com/T/
>>>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>>>
>>>>>>>> (the latter discussion contains that reproduce script which
>>>>>>>> speeds up
>>>>>>>> the crash in the guest):
>>>>>>>>> #!/bin/bash
>>>>>>>>>
>>>>>>>>> chvt 3
>>>>>>>>>
>>>>>>>>> for j in $(seq 80); do
>>>>>>>>> echo "$(date) starting round $j"
>>>>>>>>> if [ "$(journalctl --boot | grep "failed to allocate
>>>>>>>>> VRAM
>>>>>>>>> BO")" != "" ]; then
>>>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>>>> exit 1
>>>>>>>>> fi
>>>>>>>>> for i in $(seq 100); do
>>>>>>>>> dmesg > /dev/tty3
>>>>>>>>> done
>>>>>>>>> done
>>>>>>>>>
>>>>>>>>> echo "bug could not be reproduced"
>>>>>>>>> exit 0
>>>>>>>>
>>>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>>>> that
>>>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>>>> lead to
>>>>>>>> crash on the source VM.
>>>>>>>>
>>>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest
>>>>>>>> memory, but
>>>>>>>> rather passes it through the memory backend object, our code might
>>>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>>>> corruption so far.
>>>>>>>>
>>>>>>>> Could somebody help the investigation and take a look into
>>>>>>>> this? Any
>>>>>>>> suggestions would be appreciated. Thanks!
>>>>>>>
>>>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>>>> Try adding these traces to see what is preserved:
>>>>>>>
>>>>>>> -trace enable='*cpr*'
>>>>>>> -trace enable='*ram_alloc*'
>>>>>>
>>>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>>>> compatible with cpr. A message is printed at migration start time.
>>>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>>>> email-
>>>>>> steven.sistare@oracle.com/
>>>>>>
>>>>>> - Steve
>>>>>>
>>>>>
>>>>> With the traces enabled + the "migration: ram block cpr blockers"
>>>>> patch
>>>>> applied:
>>>>>
>>>>> Source:
>>>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>>>> 0x7fec18e00000
>>>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>>>> 0x7fec18c00000
>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>> 262144 fd 24 host 0x7fec18a00000
>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>> 67108864 fd 25 host 0x7feb77e00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>> fd 27 host 0x7fec18800000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>> 67108864 fd 28 host 0x7feb73c00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>> fd 34 host 0x7fec18600000
>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>> 2097152 fd 35 host 0x7fec18200000
>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>> fd 36 host 0x7feb8b600000
>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>> 37 host 0x7feb8b400000
>>>>>>
>>>>>> cpr_state_save cpr-transfer mode
>>>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>>>
>>>>> Target:
>>>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>>>> cpr_state_load cpr-transfer mode
>>>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>>>> 0x7fcdc9800000
>>>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>>>> 0x7fcdc9600000
>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>> 262144 fd 18 host 0x7fcdc9400000
>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>> fd 16 host 0x7fcdc9200000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>> fd 14 host 0x7fcdc8800000
>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>> fd 11 host 0x7fcdc8200000
>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>> 10 host 0x7fcd3be00000
>>>>>
>>>>> Looks like both vga.vram and qxl.vram are being preserved (with the
>>>>> same
>>>>> addresses), and no incompatible ram blocks are found during migration.
>>>>
>>>> Sorry, addressed are not the same, of course. However corresponding
>>>> ram
>>>> blocks do seem to be preserved and initialized.
>>>
>>> So far, I have not reproduced the guest driver failure.
>>>
>>> However, I have isolated places where new QEMU improperly writes to
>>> the qxl memory regions prior to starting the guest, by mmap'ing them
>>> readonly after cpr:
>>>
>>> qemu_ram_alloc_internal()
>>> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
>>> ram_flags |= RAM_READONLY;
>>> new_block = qemu_ram_alloc_from_fd(...)
>>>
>>> I have attached a draft fix; try it and let me know.
>>> My console window looks fine before and after cpr, using
>>> -vnc $hostip:0 -vga qxl
>>>
>>> - Steve
>>
>> Regarding the reproduce: when I launch the buggy version with the same
>> options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
>> my VNC client silently hangs on the target after a while. Could it
>> happen on your stand as well?
>
> cpr does not preserve the vnc connection and session. To test, I specify
> port 0 for the source VM and port 1 for the dest. When the src vnc goes
> dormant the dest vnc becomes active.
>
Sure, I meant that VNC on the dest (on the port 1) works for a while
after the migration and then hangs, apparently after the guest QXL crash.
>> Could you try launching VM with
>> "-nographic -device qxl-vga"? That way VM's serial console is given you
>> directly in the shell, so when qxl driver crashes you're still able to
>> inspect the kernel messages.
>
> I have been running like that, but have not reproduced the qxl driver
> crash,
> and I suspect my guest image+kernel is too old.
Yes, that's probably the case. But the crash occurs on my Fedora 41
guest with the 6.11.5-300.fc41.x86_64 kernel, so newer kernels seem to
be buggy.
> However, once I realized the
> issue was post-cpr modification of qxl memory, I switched my attention
> to the
> fix.
>
>> As for your patch, I can report that it doesn't resolve the issue as it
>> is. But I was able to track down another possible memory corruption
>> using your approach with readonly mmap'ing:
>>
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
>>> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
>>> (gdb) bt
>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70,
>>> errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
>>> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70,
>>> errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
>>> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70,
>>> errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
>>> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70,
>>> value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
>>> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70,
>>> v=0x5638996f3770, name=0x56389759b141 "realized",
>>> opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
>>> at ../qom/object.c:2374
>>> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70,
>>> name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
>>> at ../qom/object.c:1449
>>> #7 0x00005638970f8586 in object_property_set_qobject
>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized",
>>> value=0x5638996df900, errp=0x7ffd3c2b84e0)
>>> at ../qom/qom-qobject.c:28
>>> #8 0x00005638970f3d8d in object_property_set_bool
>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true,
>>> errp=0x7ffd3c2b84e0)
>>> at ../qom/object.c:1519
>>> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70,
>>> bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
>>> #10 0x0000563896dba675 in qdev_device_add_from_qdict
>>> (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at ../
>>> system/qdev-monitor.c:714
>>> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150,
>>> errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
>>> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0,
>>> opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/
>>> vl.c:1207
>>> #13 0x000056389737a6cc in qemu_opts_foreach
>>> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca
>>> <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
>>> at ../util/qemu-option.c:1135
>>> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at ../system/
>>> vl.c:2745
>>> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40
>>> <error_fatal>) at ../system/vl.c:2806
>>> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948)
>>> at ../system/vl.c:3838
>>> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at ../
>>> system/main.c:72
>>
>> So the attached adjusted version of your patch does seem to help. At
>> least I can't reproduce the crash on my stand.
>
> Thanks for the stack trace; the calls to SPICE_RING_INIT in init_qxl_ram
> are
> definitely harmful. Try V2 of the patch, attached, which skips the lines
> of init_qxl_ram that modify guest memory.
>
Thanks, your v2 patch does seem to prevent the crash. Would you re-send
it to the list as a proper fix?
>> I'm wondering, could it be useful to explicitly mark all the reused
>> memory regions readonly upon cpr-transfer, and then make them writable
>> back again after the migration is done? That way we will be segfaulting
>> early on instead of debugging tricky memory corruptions.
>
> It's a useful debugging technique, but changing protection on a large
> memory region
> can be too expensive for production due to TLB shootdowns.
>
> Also, there are cases where writes are performed but the value is
> guaranteed to
> be the same:
> qxl_post_load()
> qxl_set_mode()
> d->rom->mode = cpu_to_le32(modenr);
> The value is the same because mode and shadow_rom.mode were passed in
> vmstate
> from old qemu.
>
There're also cases where devices' ROM might be re-initialized. E.g.
this segfault occures upon further exploration of RO mapped RAM blocks:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
> 664 rep movsb
> [Current thread is 1 (Thread 0x7f6e7d08b480 (LWP 310379))]
> (gdb) bt
> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
> #1 0x000055aa1d030ecd in rom_set_mr (rom=0x55aa200ba380, owner=0x55aa2019ac10, name=0x7fffb8272bc0 "/rom@etc/acpi/tables", ro=true)
> at ../hw/core/loader.c:1032
> #2 0x000055aa1d031577 in rom_add_blob
> (name=0x55aa1da51f13 "etc/acpi/tables", blob=0x55aa208a1070, len=131072, max_len=2097152, addr=18446744073709551615, fw_file_name=0x55aa1da51f13 "etc/acpi/tables", fw_callback=0x55aa1d441f59 <acpi_build_update>, callback_opaque=0x55aa20ff0010, as=0x0, read_only=true) at ../hw/core/loader.c:1147
> #3 0x000055aa1cfd788d in acpi_add_rom_blob
> (update=0x55aa1d441f59 <acpi_build_update>, opaque=0x55aa20ff0010, blob=0x55aa1fc9aa00, name=0x55aa1da51f13 "etc/acpi/tables") at ../hw/acpi/utils.c:46
> #4 0x000055aa1d44213f in acpi_setup () at ../hw/i386/acpi-build.c:2720
> #5 0x000055aa1d434199 in pc_machine_done (notifier=0x55aa1ff15050, data=0x0) at ../hw/i386/pc.c:638
> #6 0x000055aa1d876845 in notifier_list_notify (list=0x55aa1ea25c10 <machine_init_done_notifiers>, data=0x0) at ../util/notify.c:39
> #7 0x000055aa1d039ee5 in qdev_machine_creation_done () at ../hw/core/machine.c:1749
> #8 0x000055aa1d2c7b3e in qemu_machine_creation_done (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2779
> #9 0x000055aa1d2c7c7d in qmp_x_exit_preconfig (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2807
> #10 0x000055aa1d2ca64f in qemu_init (argc=35, argv=0x7fffb82730e8) at ../system/vl.c:3838
> #11 0x000055aa1d79638c in main (argc=35, argv=0x7fffb82730e8) at ../system/main.c:72
I'm not sure whether ACPI tables ROM in particular is rewritten with the
same content, but there might be cases where ROM can be read from file
system upon initialization. That is undesirable as guest kernel
certainly won't be too happy about sudden change of the device's ROM
content.
So the issue we're dealing with here is any unwanted memory related
device initialization upon cpr.
For now the only thing that comes to my mind is to make a test where we
put as many devices as we can into a VM, make ram blocks RO upon cpr
(and remap them as RW later after migration is done, if needed), and
catch any unwanted memory violations. As Den suggested, we might
consider adding that behaviour as a separate non-default option (or
"migrate" command flag specific to cpr-transfer), which would only be
used in the testing.
Andrey
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-06 15:16 ` Andrey Drobyshev
@ 2025-03-06 15:52 ` Denis V. Lunev
2025-03-06 16:13 ` Steven Sistare
0 siblings, 1 reply; 13+ messages in thread
From: Denis V. Lunev @ 2025-03-06 15:52 UTC (permalink / raw)
To: Andrey Drobyshev, Steven Sistare, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé
On 3/6/25 16:16, Andrey Drobyshev wrote:
> On 3/5/25 11:19 PM, Steven Sistare wrote:
>> On 3/5/2025 11:50 AM, Andrey Drobyshev wrote:
>>> On 3/4/25 9:05 PM, Steven Sistare wrote:
>>>> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>>>>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>>>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>>>>> and
>>>>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>>>>
>>>>>>>>> Run migration source:
>>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>>
>>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>>> -machine q35 \
>>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>>> dev/shm/
>>>>>>>>>> ram0,share=on\
>>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>>> -nographic \
>>>>>>>>>> -device qxl-vga
>>>>>>>>> Run migration target:
>>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>>> -machine q35 \
>>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>>> dev/shm/
>>>>>>>>>> ram0,share=on\
>>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>>> -nographic \
>>>>>>>>>> -device qxl-vga \
>>>>>>>>>> -incoming tcp:0:44444 \
>>>>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>>>>
>>>>>>>>> Launch the migration:
>>>>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>>
>>>>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>>>>> {"channel-type":"cpr","addr":
>>>>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>>>>> dst.sock"}}]
>>>>>>>>>> EOF
>>>>>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>>>>>> following messages:
>>>>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>>>>> 0x00000001)
>>>>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>>>>>> allocate VRAM BO
>>>>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>>>>
>>>>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>>>>> min_halo@163.com/T/
>>>>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>>>>
>>>>>>>>> (the latter discussion contains that reproduce script which
>>>>>>>>> speeds up
>>>>>>>>> the crash in the guest):
>>>>>>>>>> #!/bin/bash
>>>>>>>>>>
>>>>>>>>>> chvt 3
>>>>>>>>>>
>>>>>>>>>> for j in $(seq 80); do
>>>>>>>>>> echo "$(date) starting round $j"
>>>>>>>>>> if [ "$(journalctl --boot | grep "failed to allocate
>>>>>>>>>> VRAM
>>>>>>>>>> BO")" != "" ]; then
>>>>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>>>>> exit 1
>>>>>>>>>> fi
>>>>>>>>>> for i in $(seq 100); do
>>>>>>>>>> dmesg > /dev/tty3
>>>>>>>>>> done
>>>>>>>>>> done
>>>>>>>>>>
>>>>>>>>>> echo "bug could not be reproduced"
>>>>>>>>>> exit 0
>>>>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>>>>> that
>>>>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>>>>> lead to
>>>>>>>>> crash on the source VM.
>>>>>>>>>
>>>>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest
>>>>>>>>> memory, but
>>>>>>>>> rather passes it through the memory backend object, our code might
>>>>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>>>>> corruption so far.
>>>>>>>>>
>>>>>>>>> Could somebody help the investigation and take a look into
>>>>>>>>> this? Any
>>>>>>>>> suggestions would be appreciated. Thanks!
>>>>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>>>>> Try adding these traces to see what is preserved:
>>>>>>>>
>>>>>>>> -trace enable='*cpr*'
>>>>>>>> -trace enable='*ram_alloc*'
>>>>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>>>>> compatible with cpr. A message is printed at migration start time.
>>>>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>>>>> email-
>>>>>>> steven.sistare@oracle.com/
>>>>>>>
>>>>>>> - Steve
>>>>>>>
>>>>>> With the traces enabled + the "migration: ram block cpr blockers"
>>>>>> patch
>>>>>> applied:
>>>>>>
>>>>>> Source:
>>>>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>>>>> 0x7fec18e00000
>>>>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>>>>> 0x7fec18c00000
>>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>>> 262144 fd 24 host 0x7fec18a00000
>>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>>> 67108864 fd 25 host 0x7feb77e00000
>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>>> fd 27 host 0x7fec18800000
>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>>> 67108864 fd 28 host 0x7feb73c00000
>>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>>> fd 34 host 0x7fec18600000
>>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>>> 2097152 fd 35 host 0x7fec18200000
>>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>>> fd 36 host 0x7feb8b600000
>>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>>> 37 host 0x7feb8b400000
>>>>>>>
>>>>>>> cpr_state_save cpr-transfer mode
>>>>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>>>> Target:
>>>>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>>>>> cpr_state_load cpr-transfer mode
>>>>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>>>>> 0x7fcdc9800000
>>>>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>>>>> 0x7fcdc9600000
>>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>>> 262144 fd 18 host 0x7fcdc9400000
>>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>>> fd 16 host 0x7fcdc9200000
>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>>> fd 14 host 0x7fcdc8800000
>>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>>> fd 11 host 0x7fcdc8200000
>>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>>> 10 host 0x7fcd3be00000
>>>>>> Looks like both vga.vram and qxl.vram are being preserved (with the
>>>>>> same
>>>>>> addresses), and no incompatible ram blocks are found during migration.
>>>>> Sorry, addressed are not the same, of course. However corresponding
>>>>> ram
>>>>> blocks do seem to be preserved and initialized.
>>>> So far, I have not reproduced the guest driver failure.
>>>>
>>>> However, I have isolated places where new QEMU improperly writes to
>>>> the qxl memory regions prior to starting the guest, by mmap'ing them
>>>> readonly after cpr:
>>>>
>>>> qemu_ram_alloc_internal()
>>>> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
>>>> ram_flags |= RAM_READONLY;
>>>> new_block = qemu_ram_alloc_from_fd(...)
>>>>
>>>> I have attached a draft fix; try it and let me know.
>>>> My console window looks fine before and after cpr, using
>>>> -vnc $hostip:0 -vga qxl
>>>>
>>>> - Steve
>>> Regarding the reproduce: when I launch the buggy version with the same
>>> options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
>>> my VNC client silently hangs on the target after a while. Could it
>>> happen on your stand as well?
>> cpr does not preserve the vnc connection and session. To test, I specify
>> port 0 for the source VM and port 1 for the dest. When the src vnc goes
>> dormant the dest vnc becomes active.
>>
> Sure, I meant that VNC on the dest (on the port 1) works for a while
> after the migration and then hangs, apparently after the guest QXL crash.
>
>>> Could you try launching VM with
>>> "-nographic -device qxl-vga"? That way VM's serial console is given you
>>> directly in the shell, so when qxl driver crashes you're still able to
>>> inspect the kernel messages.
>> I have been running like that, but have not reproduced the qxl driver
>> crash,
>> and I suspect my guest image+kernel is too old.
> Yes, that's probably the case. But the crash occurs on my Fedora 41
> guest with the 6.11.5-300.fc41.x86_64 kernel, so newer kernels seem to
> be buggy.
>
>
>> However, once I realized the
>> issue was post-cpr modification of qxl memory, I switched my attention
>> to the
>> fix.
>>
>>> As for your patch, I can report that it doesn't resolve the issue as it
>>> is. But I was able to track down another possible memory corruption
>>> using your approach with readonly mmap'ing:
>>>
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>>> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
>>>> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
>>>> (gdb) bt
>>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>>> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70,
>>>> errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
>>>> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70,
>>>> errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
>>>> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70,
>>>> errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
>>>> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70,
>>>> value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
>>>> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70,
>>>> v=0x5638996f3770, name=0x56389759b141 "realized",
>>>> opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
>>>> at ../qom/object.c:2374
>>>> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70,
>>>> name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
>>>> at ../qom/object.c:1449
>>>> #7 0x00005638970f8586 in object_property_set_qobject
>>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized",
>>>> value=0x5638996df900, errp=0x7ffd3c2b84e0)
>>>> at ../qom/qom-qobject.c:28
>>>> #8 0x00005638970f3d8d in object_property_set_bool
>>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true,
>>>> errp=0x7ffd3c2b84e0)
>>>> at ../qom/object.c:1519
>>>> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70,
>>>> bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
>>>> #10 0x0000563896dba675 in qdev_device_add_from_qdict
>>>> (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at ../
>>>> system/qdev-monitor.c:714
>>>> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150,
>>>> errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
>>>> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0,
>>>> opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/
>>>> vl.c:1207
>>>> #13 0x000056389737a6cc in qemu_opts_foreach
>>>> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca
>>>> <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
>>>> at ../util/qemu-option.c:1135
>>>> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at ../system/
>>>> vl.c:2745
>>>> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40
>>>> <error_fatal>) at ../system/vl.c:2806
>>>> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948)
>>>> at ../system/vl.c:3838
>>>> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at ../
>>>> system/main.c:72
>>> So the attached adjusted version of your patch does seem to help. At
>>> least I can't reproduce the crash on my stand.
>> Thanks for the stack trace; the calls to SPICE_RING_INIT in init_qxl_ram
>> are
>> definitely harmful. Try V2 of the patch, attached, which skips the lines
>> of init_qxl_ram that modify guest memory.
>>
> Thanks, your v2 patch does seem to prevent the crash. Would you re-send
> it to the list as a proper fix?
>
>>> I'm wondering, could it be useful to explicitly mark all the reused
>>> memory regions readonly upon cpr-transfer, and then make them writable
>>> back again after the migration is done? That way we will be segfaulting
>>> early on instead of debugging tricky memory corruptions.
>> It's a useful debugging technique, but changing protection on a large
>> memory region
>> can be too expensive for production due to TLB shootdowns.
>>
>> Also, there are cases where writes are performed but the value is
>> guaranteed to
>> be the same:
>> qxl_post_load()
>> qxl_set_mode()
>> d->rom->mode = cpu_to_le32(modenr);
>> The value is the same because mode and shadow_rom.mode were passed in
>> vmstate
>> from old qemu.
>>
> There're also cases where devices' ROM might be re-initialized. E.g.
> this segfault occures upon further exploration of RO mapped RAM blocks:
>
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
>> 664 rep movsb
>> [Current thread is 1 (Thread 0x7f6e7d08b480 (LWP 310379))]
>> (gdb) bt
>> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
>> #1 0x000055aa1d030ecd in rom_set_mr (rom=0x55aa200ba380, owner=0x55aa2019ac10, name=0x7fffb8272bc0 "/rom@etc/acpi/tables", ro=true)
>> at ../hw/core/loader.c:1032
>> #2 0x000055aa1d031577 in rom_add_blob
>> (name=0x55aa1da51f13 "etc/acpi/tables", blob=0x55aa208a1070, len=131072, max_len=2097152, addr=18446744073709551615, fw_file_name=0x55aa1da51f13 "etc/acpi/tables", fw_callback=0x55aa1d441f59 <acpi_build_update>, callback_opaque=0x55aa20ff0010, as=0x0, read_only=true) at ../hw/core/loader.c:1147
>> #3 0x000055aa1cfd788d in acpi_add_rom_blob
>> (update=0x55aa1d441f59 <acpi_build_update>, opaque=0x55aa20ff0010, blob=0x55aa1fc9aa00, name=0x55aa1da51f13 "etc/acpi/tables") at ../hw/acpi/utils.c:46
>> #4 0x000055aa1d44213f in acpi_setup () at ../hw/i386/acpi-build.c:2720
>> #5 0x000055aa1d434199 in pc_machine_done (notifier=0x55aa1ff15050, data=0x0) at ../hw/i386/pc.c:638
>> #6 0x000055aa1d876845 in notifier_list_notify (list=0x55aa1ea25c10 <machine_init_done_notifiers>, data=0x0) at ../util/notify.c:39
>> #7 0x000055aa1d039ee5 in qdev_machine_creation_done () at ../hw/core/machine.c:1749
>> #8 0x000055aa1d2c7b3e in qemu_machine_creation_done (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2779
>> #9 0x000055aa1d2c7c7d in qmp_x_exit_preconfig (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2807
>> #10 0x000055aa1d2ca64f in qemu_init (argc=35, argv=0x7fffb82730e8) at ../system/vl.c:3838
>> #11 0x000055aa1d79638c in main (argc=35, argv=0x7fffb82730e8) at ../system/main.c:72
> I'm not sure whether ACPI tables ROM in particular is rewritten with the
> same content, but there might be cases where ROM can be read from file
> system upon initialization. That is undesirable as guest kernel
> certainly won't be too happy about sudden change of the device's ROM
> content.
>
> So the issue we're dealing with here is any unwanted memory related
> device initialization upon cpr.
>
> For now the only thing that comes to my mind is to make a test where we
> put as many devices as we can into a VM, make ram blocks RO upon cpr
> (and remap them as RW later after migration is done, if needed), and
> catch any unwanted memory violations. As Den suggested, we might
> consider adding that behaviour as a separate non-default option (or
> "migrate" command flag specific to cpr-transfer), which would only be
> used in the testing.
>
> Andrey
No way. ACPI with the source must be used in the same way as BIOSes
and optional ROMs.
Den
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-06 15:52 ` Denis V. Lunev
@ 2025-03-06 16:13 ` Steven Sistare
2025-03-07 21:00 ` Steven Sistare
0 siblings, 1 reply; 13+ messages in thread
From: Steven Sistare @ 2025-03-06 16:13 UTC (permalink / raw)
To: Denis V. Lunev, Andrey Drobyshev, qemu-devel
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé
On 3/6/2025 10:52 AM, Denis V. Lunev wrote:
> On 3/6/25 16:16, Andrey Drobyshev wrote:
>> On 3/5/25 11:19 PM, Steven Sistare wrote:
>>> On 3/5/2025 11:50 AM, Andrey Drobyshev wrote:
>>>> On 3/4/25 9:05 PM, Steven Sistare wrote:
>>>>> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>>>>>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>>>>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>>>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>>>>>> and
>>>>>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>>>>>
>>>>>>>>>> Run migration source:
>>>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>>>
>>>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>>>> -machine q35 \
>>>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>>>> dev/shm/
>>>>>>>>>>> ram0,share=on\
>>>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>>>> -nographic \
>>>>>>>>>>> -device qxl-vga
>>>>>>>>>> Run migration target:
>>>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>>>> -machine q35 \
>>>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>>>> dev/shm/
>>>>>>>>>>> ram0,share=on\
>>>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>>>> -nographic \
>>>>>>>>>>> -device qxl-vga \
>>>>>>>>>>> -incoming tcp:0:44444 \
>>>>>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>>>>>
>>>>>>>>>> Launch the migration:
>>>>>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>>>
>>>>>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>>>>>> {"channel-type":"cpr","addr":
>>>>>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>>>>>> dst.sock"}}]
>>>>>>>>>>> EOF
>>>>>>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>>>>>>> following messages:
>>>>>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>>>>>> 0x00000001)
>>>>>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>>>>>>> allocate VRAM BO
>>>>>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>>>>>
>>>>>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>>>>>> min_halo@163.com/T/
>>>>>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>>>>>
>>>>>>>>>> (the latter discussion contains that reproduce script which
>>>>>>>>>> speeds up
>>>>>>>>>> the crash in the guest):
>>>>>>>>>>> #!/bin/bash
>>>>>>>>>>>
>>>>>>>>>>> chvt 3
>>>>>>>>>>>
>>>>>>>>>>> for j in $(seq 80); do
>>>>>>>>>>> echo "$(date) starting round $j"
>>>>>>>>>>> if [ "$(journalctl --boot | grep "failed to allocate
>>>>>>>>>>> VRAM
>>>>>>>>>>> BO")" != "" ]; then
>>>>>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>>>>>> exit 1
>>>>>>>>>>> fi
>>>>>>>>>>> for i in $(seq 100); do
>>>>>>>>>>> dmesg > /dev/tty3
>>>>>>>>>>> done
>>>>>>>>>>> done
>>>>>>>>>>>
>>>>>>>>>>> echo "bug could not be reproduced"
>>>>>>>>>>> exit 0
>>>>>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>>>>>> that
>>>>>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>>>>>> lead to
>>>>>>>>>> crash on the source VM.
>>>>>>>>>>
>>>>>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest
>>>>>>>>>> memory, but
>>>>>>>>>> rather passes it through the memory backend object, our code might
>>>>>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>>>>>> corruption so far.
>>>>>>>>>>
>>>>>>>>>> Could somebody help the investigation and take a look into
>>>>>>>>>> this? Any
>>>>>>>>>> suggestions would be appreciated. Thanks!
>>>>>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>>>>>> Try adding these traces to see what is preserved:
>>>>>>>>>
>>>>>>>>> -trace enable='*cpr*'
>>>>>>>>> -trace enable='*ram_alloc*'
>>>>>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>>>>>> compatible with cpr. A message is printed at migration start time.
>>>>>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>>>>>> email-
>>>>>>>> steven.sistare@oracle.com/
>>>>>>>>
>>>>>>>> - Steve
>>>>>>>>
>>>>>>> With the traces enabled + the "migration: ram block cpr blockers"
>>>>>>> patch
>>>>>>> applied:
>>>>>>>
>>>>>>> Source:
>>>>>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>>>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>>>>>> 0x7fec18e00000
>>>>>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>>>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>>>>>> 0x7fec18c00000
>>>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>>>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>>>> 262144 fd 24 host 0x7fec18a00000
>>>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>>>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>>>> 67108864 fd 25 host 0x7feb77e00000
>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>>>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>>>> fd 27 host 0x7fec18800000
>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>>>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>>>> 67108864 fd 28 host 0x7feb73c00000
>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>>>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>>>> fd 34 host 0x7fec18600000
>>>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>>>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>>>> 2097152 fd 35 host 0x7fec18200000
>>>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>>>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>>>> fd 36 host 0x7feb8b600000
>>>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>>>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>>>> 37 host 0x7feb8b400000
>>>>>>>>
>>>>>>>> cpr_state_save cpr-transfer mode
>>>>>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>>>>> Target:
>>>>>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>>>>>> cpr_state_load cpr-transfer mode
>>>>>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>>>>>> 0x7fcdc9800000
>>>>>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>>>>>> 0x7fcdc9600000
>>>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>>>> 262144 fd 18 host 0x7fcdc9400000
>>>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>>>> fd 16 host 0x7fcdc9200000
>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>>>> fd 14 host 0x7fcdc8800000
>>>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>>>> fd 11 host 0x7fcdc8200000
>>>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>>>> 10 host 0x7fcd3be00000
>>>>>>> Looks like both vga.vram and qxl.vram are being preserved (with the
>>>>>>> same
>>>>>>> addresses), and no incompatible ram blocks are found during migration.
>>>>>> Sorry, addressed are not the same, of course. However corresponding
>>>>>> ram
>>>>>> blocks do seem to be preserved and initialized.
>>>>> So far, I have not reproduced the guest driver failure.
>>>>>
>>>>> However, I have isolated places where new QEMU improperly writes to
>>>>> the qxl memory regions prior to starting the guest, by mmap'ing them
>>>>> readonly after cpr:
>>>>>
>>>>> qemu_ram_alloc_internal()
>>>>> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
>>>>> ram_flags |= RAM_READONLY;
>>>>> new_block = qemu_ram_alloc_from_fd(...)
>>>>>
>>>>> I have attached a draft fix; try it and let me know.
>>>>> My console window looks fine before and after cpr, using
>>>>> -vnc $hostip:0 -vga qxl
>>>>>
>>>>> - Steve
>>>> Regarding the reproduce: when I launch the buggy version with the same
>>>> options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
>>>> my VNC client silently hangs on the target after a while. Could it
>>>> happen on your stand as well?
>>> cpr does not preserve the vnc connection and session. To test, I specify
>>> port 0 for the source VM and port 1 for the dest. When the src vnc goes
>>> dormant the dest vnc becomes active.
>>>
>> Sure, I meant that VNC on the dest (on the port 1) works for a while
>> after the migration and then hangs, apparently after the guest QXL crash.
>>
>>>> Could you try launching VM with
>>>> "-nographic -device qxl-vga"? That way VM's serial console is given you
>>>> directly in the shell, so when qxl driver crashes you're still able to
>>>> inspect the kernel messages.
>>> I have been running like that, but have not reproduced the qxl driver
>>> crash,
>>> and I suspect my guest image+kernel is too old.
>> Yes, that's probably the case. But the crash occurs on my Fedora 41
>> guest with the 6.11.5-300.fc41.x86_64 kernel, so newer kernels seem to
>> be buggy.
>>
>>
>>> However, once I realized the
>>> issue was post-cpr modification of qxl memory, I switched my attention
>>> to the
>>> fix.
>>>
>>>> As for your patch, I can report that it doesn't resolve the issue as it
>>>> is. But I was able to track down another possible memory corruption
>>>> using your approach with readonly mmap'ing:
>>>>
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>>>> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
>>>>> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
>>>>> (gdb) bt
>>>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>>>> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70,
>>>>> errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
>>>>> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70,
>>>>> errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
>>>>> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70,
>>>>> errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
>>>>> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70,
>>>>> value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
>>>>> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70,
>>>>> v=0x5638996f3770, name=0x56389759b141 "realized",
>>>>> opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
>>>>> at ../qom/object.c:2374
>>>>> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70,
>>>>> name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
>>>>> at ../qom/object.c:1449
>>>>> #7 0x00005638970f8586 in object_property_set_qobject
>>>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized",
>>>>> value=0x5638996df900, errp=0x7ffd3c2b84e0)
>>>>> at ../qom/qom-qobject.c:28
>>>>> #8 0x00005638970f3d8d in object_property_set_bool
>>>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true,
>>>>> errp=0x7ffd3c2b84e0)
>>>>> at ../qom/object.c:1519
>>>>> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70,
>>>>> bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
>>>>> #10 0x0000563896dba675 in qdev_device_add_from_qdict
>>>>> (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at ../
>>>>> system/qdev-monitor.c:714
>>>>> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150,
>>>>> errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
>>>>> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0,
>>>>> opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/
>>>>> vl.c:1207
>>>>> #13 0x000056389737a6cc in qemu_opts_foreach
>>>>> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca
>>>>> <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
>>>>> at ../util/qemu-option.c:1135
>>>>> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at ../system/
>>>>> vl.c:2745
>>>>> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40
>>>>> <error_fatal>) at ../system/vl.c:2806
>>>>> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948)
>>>>> at ../system/vl.c:3838
>>>>> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at ../
>>>>> system/main.c:72
>>>> So the attached adjusted version of your patch does seem to help. At
>>>> least I can't reproduce the crash on my stand.
>>> Thanks for the stack trace; the calls to SPICE_RING_INIT in init_qxl_ram
>>> are
>>> definitely harmful. Try V2 of the patch, attached, which skips the lines
>>> of init_qxl_ram that modify guest memory.
>>>
>> Thanks, your v2 patch does seem to prevent the crash. Would you re-send
>> it to the list as a proper fix?
Yes. Was waiting for your confirmation.
>>>> I'm wondering, could it be useful to explicitly mark all the reused
>>>> memory regions readonly upon cpr-transfer, and then make them writable
>>>> back again after the migration is done? That way we will be segfaulting
>>>> early on instead of debugging tricky memory corruptions.
>>> It's a useful debugging technique, but changing protection on a large
>>> memory region
>>> can be too expensive for production due to TLB shootdowns.
>>>
>>> Also, there are cases where writes are performed but the value is
>>> guaranteed to
>>> be the same:
>>> qxl_post_load()
>>> qxl_set_mode()
>>> d->rom->mode = cpu_to_le32(modenr);
>>> The value is the same because mode and shadow_rom.mode were passed in
>>> vmstate
>>> from old qemu.
>>>
>> There're also cases where devices' ROM might be re-initialized. E.g.
>> this segfault occures upon further exploration of RO mapped RAM blocks:
>>
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
>>> 664 rep movsb
>>> [Current thread is 1 (Thread 0x7f6e7d08b480 (LWP 310379))]
>>> (gdb) bt
>>> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
>>> #1 0x000055aa1d030ecd in rom_set_mr (rom=0x55aa200ba380, owner=0x55aa2019ac10, name=0x7fffb8272bc0 "/rom@etc/acpi/tables", ro=true)
>>> at ../hw/core/loader.c:1032
>>> #2 0x000055aa1d031577 in rom_add_blob
>>> (name=0x55aa1da51f13 "etc/acpi/tables", blob=0x55aa208a1070, len=131072, max_len=2097152, addr=18446744073709551615, fw_file_name=0x55aa1da51f13 "etc/acpi/tables", fw_callback=0x55aa1d441f59 <acpi_build_update>, callback_opaque=0x55aa20ff0010, as=0x0, read_only=true) at ../hw/core/loader.c:1147
>>> #3 0x000055aa1cfd788d in acpi_add_rom_blob
>>> (update=0x55aa1d441f59 <acpi_build_update>, opaque=0x55aa20ff0010, blob=0x55aa1fc9aa00, name=0x55aa1da51f13 "etc/acpi/tables") at ../hw/acpi/utils.c:46
>>> #4 0x000055aa1d44213f in acpi_setup () at ../hw/i386/acpi-build.c:2720
>>> #5 0x000055aa1d434199 in pc_machine_done (notifier=0x55aa1ff15050, data=0x0) at ../hw/i386/pc.c:638
>>> #6 0x000055aa1d876845 in notifier_list_notify (list=0x55aa1ea25c10 <machine_init_done_notifiers>, data=0x0) at ../util/notify.c:39
>>> #7 0x000055aa1d039ee5 in qdev_machine_creation_done () at ../hw/core/machine.c:1749
>>> #8 0x000055aa1d2c7b3e in qemu_machine_creation_done (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2779
>>> #9 0x000055aa1d2c7c7d in qmp_x_exit_preconfig (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2807
>>> #10 0x000055aa1d2ca64f in qemu_init (argc=35, argv=0x7fffb82730e8) at ../system/vl.c:3838
>>> #11 0x000055aa1d79638c in main (argc=35, argv=0x7fffb82730e8) at ../system/main.c:72
>> I'm not sure whether ACPI tables ROM in particular is rewritten with the
>> same content, but there might be cases where ROM can be read from file
>> system upon initialization. That is undesirable as guest kernel
>> certainly won't be too happy about sudden change of the device's ROM
>> content.
>>
>> So the issue we're dealing with here is any unwanted memory related
>> device initialization upon cpr.
>>
>> For now the only thing that comes to my mind is to make a test where we
>> put as many devices as we can into a VM, make ram blocks RO upon cpr
>> (and remap them as RW later after migration is done, if needed), and
>> catch any unwanted memory violations. As Den suggested, we might
>> consider adding that behaviour as a separate non-default option (or
>> "migrate" command flag specific to cpr-transfer), which would only be
>> used in the testing.
I'll look into adding an option, but there may be too many false positives,
such as the qxl_set_mode case above. And the maintainers may object to me
eliminating the false positives by adding more CPR_IN tests, due to gratuitous
(from their POV) ugliness.
But I will use the technique to look for more write violations.
>> Andrey
> No way. ACPI with the source must be used in the same way as BIOSes
> and optional ROMs.
Yup, its a bug. Will fix.
- Steve
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration
2025-03-06 16:13 ` Steven Sistare
@ 2025-03-07 21:00 ` Steven Sistare
0 siblings, 0 replies; 13+ messages in thread
From: Steven Sistare @ 2025-03-07 21:00 UTC (permalink / raw)
To: Andrey Drobyshev
Cc: William Roche, Gerd Hoffmann, Daniel P. Berrangé, qemu-devel,
Denis V. Lunev
see
https://lore.kernel.org/qemu-devel/1741380954-341079-1-git-send-email-steven.sistare@oracle.com/
- Steve
On 3/6/2025 11:13 AM, Steven Sistare wrote:
> On 3/6/2025 10:52 AM, Denis V. Lunev wrote:
>> On 3/6/25 16:16, Andrey Drobyshev wrote:
>>> On 3/5/25 11:19 PM, Steven Sistare wrote:
>>>> On 3/5/2025 11:50 AM, Andrey Drobyshev wrote:
>>>>> On 3/4/25 9:05 PM, Steven Sistare wrote:
>>>>>> On 2/28/2025 1:37 PM, Andrey Drobyshev wrote:
>>>>>>> On 2/28/25 8:35 PM, Andrey Drobyshev wrote:
>>>>>>>> On 2/28/25 8:20 PM, Steven Sistare wrote:
>>>>>>>>> On 2/28/2025 1:13 PM, Steven Sistare wrote:
>>>>>>>>>> On 2/28/2025 12:39 PM, Andrey Drobyshev wrote:
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> We've been experimenting with cpr-transfer migration mode recently
>>>>>>>>>>> and
>>>>>>>>>>> have discovered the following issue with the guest QXL driver:
>>>>>>>>>>>
>>>>>>>>>>> Run migration source:
>>>>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>>>>
>>>>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>>>>> -machine q35 \
>>>>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>>>>> dev/shm/
>>>>>>>>>>>> ram0,share=on\
>>>>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>>>>> -nographic \
>>>>>>>>>>>> -device qxl-vga
>>>>>>>>>>> Run migration target:
>>>>>>>>>>>> EMULATOR=/path/to/emulator
>>>>>>>>>>>> ROOTFS=/path/to/image
>>>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-dst.sock
>>>>>>>>>>>> $EMULATOR -enable-kvm \
>>>>>>>>>>>> -machine q35 \
>>>>>>>>>>>> -cpu host -smp 2 -m 2G \
>>>>>>>>>>>> -object memory-backend-file,id=ram0,size=2G,mem-path=/
>>>>>>>>>>>> dev/shm/
>>>>>>>>>>>> ram0,share=on\
>>>>>>>>>>>> -machine memory-backend=ram0 \
>>>>>>>>>>>> -machine aux-ram-share=on \
>>>>>>>>>>>> -drive file=$ROOTFS,media=disk,if=virtio \
>>>>>>>>>>>> -qmp unix:$QMPSOCK,server=on,wait=off \
>>>>>>>>>>>> -nographic \
>>>>>>>>>>>> -device qxl-vga \
>>>>>>>>>>>> -incoming tcp:0:44444 \
>>>>>>>>>>>> -incoming '{"channel-type": "cpr", "addr": { "transport":
>>>>>>>>>>>> "socket", "type": "unix", "path": "/var/run/alma8cpr-dst.sock"}}'
>>>>>>>>>>>
>>>>>>>>>>> Launch the migration:
>>>>>>>>>>>> QMPSHELL=/root/src/qemu/master/scripts/qmp/qmp-shell
>>>>>>>>>>>> QMPSOCK=/var/run/alma8qmp-src.sock
>>>>>>>>>>>>
>>>>>>>>>>>> $QMPSHELL -p $QMPSOCK <<EOF
>>>>>>>>>>>> migrate-set-parameters mode=cpr-transfer
>>>>>>>>>>>> migrate channels=[{"channel-type":"main","addr":
>>>>>>>>>>>> {"transport":"socket","type":"inet","host":"0","port":"44444"}},
>>>>>>>>>>>> {"channel-type":"cpr","addr":
>>>>>>>>>>>> {"transport":"socket","type":"unix","path":"/var/run/alma8cpr-
>>>>>>>>>>>> dst.sock"}}]
>>>>>>>>>>>> EOF
>>>>>>>>>>> Then, after a while, QXL guest driver on target crashes spewing the
>>>>>>>>>>> following messages:
>>>>>>>>>>>> [ 73.962002] [TTM] Buffer eviction failed
>>>>>>>>>>>> [ 73.962072] qxl 0000:00:02.0: object_init failed for (3149824,
>>>>>>>>>>>> 0x00000001)
>>>>>>>>>>>> [ 73.962081] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to
>>>>>>>>>>>> allocate VRAM BO
>>>>>>>>>>> That seems to be a known kernel QXL driver bug:
>>>>>>>>>>>
>>>>>>>>>>> https://lore.kernel.org/all/20220907094423.93581-1-
>>>>>>>>>>> min_halo@163.com/T/
>>>>>>>>>>> https://lore.kernel.org/lkml/ZTgydqRlK6WX_b29@eldamar.lan/
>>>>>>>>>>>
>>>>>>>>>>> (the latter discussion contains that reproduce script which
>>>>>>>>>>> speeds up
>>>>>>>>>>> the crash in the guest):
>>>>>>>>>>>> #!/bin/bash
>>>>>>>>>>>>
>>>>>>>>>>>> chvt 3
>>>>>>>>>>>>
>>>>>>>>>>>> for j in $(seq 80); do
>>>>>>>>>>>> echo "$(date) starting round $j"
>>>>>>>>>>>> if [ "$(journalctl --boot | grep "failed to allocate
>>>>>>>>>>>> VRAM
>>>>>>>>>>>> BO")" != "" ]; then
>>>>>>>>>>>> echo "bug was reproduced after $j tries"
>>>>>>>>>>>> exit 1
>>>>>>>>>>>> fi
>>>>>>>>>>>> for i in $(seq 100); do
>>>>>>>>>>>> dmesg > /dev/tty3
>>>>>>>>>>>> done
>>>>>>>>>>>> done
>>>>>>>>>>>>
>>>>>>>>>>>> echo "bug could not be reproduced"
>>>>>>>>>>>> exit 0
>>>>>>>>>>> The bug itself seems to remain unfixed, as I was able to reproduce
>>>>>>>>>>> that
>>>>>>>>>>> with Fedora 41 guest, as well as AlmaLinux 8 guest. However our
>>>>>>>>>>> cpr-transfer code also seems to be buggy as it triggers the crash -
>>>>>>>>>>> without the cpr-transfer migration the above reproduce doesn't
>>>>>>>>>>> lead to
>>>>>>>>>>> crash on the source VM.
>>>>>>>>>>>
>>>>>>>>>>> I suspect that, as cpr-transfer doesn't migrate the guest
>>>>>>>>>>> memory, but
>>>>>>>>>>> rather passes it through the memory backend object, our code might
>>>>>>>>>>> somehow corrupt the VRAM. However, I wasn't able to trace the
>>>>>>>>>>> corruption so far.
>>>>>>>>>>>
>>>>>>>>>>> Could somebody help the investigation and take a look into
>>>>>>>>>>> this? Any
>>>>>>>>>>> suggestions would be appreciated. Thanks!
>>>>>>>>>> Possibly some memory region created by qxl is not being preserved.
>>>>>>>>>> Try adding these traces to see what is preserved:
>>>>>>>>>>
>>>>>>>>>> -trace enable='*cpr*'
>>>>>>>>>> -trace enable='*ram_alloc*'
>>>>>>>>> Also try adding this patch to see if it flags any ram blocks as not
>>>>>>>>> compatible with cpr. A message is printed at migration start time.
>>>>>>>>> https://lore.kernel.org/qemu-devel/1740667681-257312-1-git-send-
>>>>>>>>> email-
>>>>>>>>> steven.sistare@oracle.com/
>>>>>>>>>
>>>>>>>>> - Steve
>>>>>>>>>
>>>>>>>> With the traces enabled + the "migration: ram block cpr blockers"
>>>>>>>> patch
>>>>>>>> applied:
>>>>>>>>
>>>>>>>> Source:
>>>>>>>>> cpr_find_fd pc.bios, id 0 returns -1
>>>>>>>>> cpr_save_fd pc.bios, id 0, fd 22
>>>>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 22 host
>>>>>>>>> 0x7fec18e00000
>>>>>>>>> cpr_find_fd pc.rom, id 0 returns -1
>>>>>>>>> cpr_save_fd pc.rom, id 0, fd 23
>>>>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 23 host
>>>>>>>>> 0x7fec18c00000
>>>>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns -1
>>>>>>>>> cpr_save_fd 0000:00:01.0/e1000e.rom, id 0, fd 24
>>>>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>>>>> 262144 fd 24 host 0x7fec18a00000
>>>>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
>>>>>>>>> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 25
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>>>>> 67108864 fd 25 host 0x7feb77e00000
>>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns -1
>>>>>>>>> cpr_save_fd 0000:00:02.0/qxl.vrom, id 0, fd 27
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>>>>> fd 27 host 0x7fec18800000
>>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns -1
>>>>>>>>> cpr_save_fd 0000:00:02.0/qxl.vram, id 0, fd 28
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>>>>> 67108864 fd 28 host 0x7feb73c00000
>>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns -1
>>>>>>>>> cpr_save_fd 0000:00:02.0/qxl.rom, id 0, fd 34
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>>>>> fd 34 host 0x7fec18600000
>>>>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
>>>>>>>>> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 35
>>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>>>>> 2097152 fd 35 host 0x7fec18200000
>>>>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns -1
>>>>>>>>> cpr_save_fd /rom@etc/table-loader, id 0, fd 36
>>>>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>>>>> fd 36 host 0x7feb8b600000
>>>>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
>>>>>>>>> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 37
>>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>>>>> 37 host 0x7feb8b400000
>>>>>>>>>
>>>>>>>>> cpr_state_save cpr-transfer mode
>>>>>>>>> cpr_transfer_output /var/run/alma8cpr-dst.sock
>>>>>>>> Target:
>>>>>>>>> cpr_transfer_input /var/run/alma8cpr-dst.sock
>>>>>>>>> cpr_state_load cpr-transfer mode
>>>>>>>>> cpr_find_fd pc.bios, id 0 returns 20
>>>>>>>>> qemu_ram_alloc_shared pc.bios size 262144 max_size 262144 fd 20 host
>>>>>>>>> 0x7fcdc9800000
>>>>>>>>> cpr_find_fd pc.rom, id 0 returns 19
>>>>>>>>> qemu_ram_alloc_shared pc.rom size 131072 max_size 131072 fd 19 host
>>>>>>>>> 0x7fcdc9600000
>>>>>>>>> cpr_find_fd 0000:00:01.0/e1000e.rom, id 0 returns 18
>>>>>>>>> qemu_ram_alloc_shared 0000:00:01.0/e1000e.rom size 262144 max_size
>>>>>>>>> 262144 fd 18 host 0x7fcdc9400000
>>>>>>>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 17
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/vga.vram size 67108864 max_size
>>>>>>>>> 67108864 fd 17 host 0x7fcd27e00000
>>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vrom, id 0 returns 16
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vrom size 8192 max_size 8192
>>>>>>>>> fd 16 host 0x7fcdc9200000
>>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.vram, id 0 returns 15
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.vram size 67108864 max_size
>>>>>>>>> 67108864 fd 15 host 0x7fcd23c00000
>>>>>>>>> cpr_find_fd 0000:00:02.0/qxl.rom, id 0 returns 14
>>>>>>>>> qemu_ram_alloc_shared 0000:00:02.0/qxl.rom size 65536 max_size 65536
>>>>>>>>> fd 14 host 0x7fcdc8800000
>>>>>>>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 13
>>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/tables size 131072 max_size
>>>>>>>>> 2097152 fd 13 host 0x7fcdc8400000
>>>>>>>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 11
>>>>>>>>> qemu_ram_alloc_shared /rom@etc/table-loader size 4096 max_size 65536
>>>>>>>>> fd 11 host 0x7fcdc8200000
>>>>>>>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 10
>>>>>>>>> qemu_ram_alloc_shared /rom@etc/acpi/rsdp size 4096 max_size 4096 fd
>>>>>>>>> 10 host 0x7fcd3be00000
>>>>>>>> Looks like both vga.vram and qxl.vram are being preserved (with the
>>>>>>>> same
>>>>>>>> addresses), and no incompatible ram blocks are found during migration.
>>>>>>> Sorry, addressed are not the same, of course. However corresponding
>>>>>>> ram
>>>>>>> blocks do seem to be preserved and initialized.
>>>>>> So far, I have not reproduced the guest driver failure.
>>>>>>
>>>>>> However, I have isolated places where new QEMU improperly writes to
>>>>>> the qxl memory regions prior to starting the guest, by mmap'ing them
>>>>>> readonly after cpr:
>>>>>>
>>>>>> qemu_ram_alloc_internal()
>>>>>> if (reused && (strstr(name, "qxl") || strstr("name", "vga")))
>>>>>> ram_flags |= RAM_READONLY;
>>>>>> new_block = qemu_ram_alloc_from_fd(...)
>>>>>>
>>>>>> I have attached a draft fix; try it and let me know.
>>>>>> My console window looks fine before and after cpr, using
>>>>>> -vnc $hostip:0 -vga qxl
>>>>>>
>>>>>> - Steve
>>>>> Regarding the reproduce: when I launch the buggy version with the same
>>>>> options as you, i.e. "-vnc 0.0.0.0:$port -vga qxl", and do cpr-transfer,
>>>>> my VNC client silently hangs on the target after a while. Could it
>>>>> happen on your stand as well?
>>>> cpr does not preserve the vnc connection and session. To test, I specify
>>>> port 0 for the source VM and port 1 for the dest. When the src vnc goes
>>>> dormant the dest vnc becomes active.
>>>>
>>> Sure, I meant that VNC on the dest (on the port 1) works for a while
>>> after the migration and then hangs, apparently after the guest QXL crash.
>>>
>>>>> Could you try launching VM with
>>>>> "-nographic -device qxl-vga"? That way VM's serial console is given you
>>>>> directly in the shell, so when qxl driver crashes you're still able to
>>>>> inspect the kernel messages.
>>>> I have been running like that, but have not reproduced the qxl driver
>>>> crash,
>>>> and I suspect my guest image+kernel is too old.
>>> Yes, that's probably the case. But the crash occurs on my Fedora 41
>>> guest with the 6.11.5-300.fc41.x86_64 kernel, so newer kernels seem to
>>> be buggy.
>>>
>>>
>>>> However, once I realized the
>>>> issue was post-cpr modification of qxl memory, I switched my attention
>>>> to the
>>>> fix.
>>>>
>>>>> As for your patch, I can report that it doesn't resolve the issue as it
>>>>> is. But I was able to track down another possible memory corruption
>>>>> using your approach with readonly mmap'ing:
>>>>>
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>>>>> 412 d->ram->magic = cpu_to_le32(QXL_RAM_MAGIC);
>>>>>> [Current thread is 1 (Thread 0x7f1a4f83b480 (LWP 229798))]
>>>>>> (gdb) bt
>>>>>> #0 init_qxl_ram (d=0x5638996e0e70) at ../hw/display/qxl.c:412
>>>>>> #1 0x0000563896e7f467 in qxl_realize_common (qxl=0x5638996e0e70,
>>>>>> errp=0x7ffd3c2b8170) at ../hw/display/qxl.c:2142
>>>>>> #2 0x0000563896e7fda1 in qxl_realize_primary (dev=0x5638996e0e70,
>>>>>> errp=0x7ffd3c2b81d0) at ../hw/display/qxl.c:2257
>>>>>> #3 0x0000563896c7e8f2 in pci_qdev_realize (qdev=0x5638996e0e70,
>>>>>> errp=0x7ffd3c2b8250) at ../hw/pci/pci.c:2174
>>>>>> #4 0x00005638970eb54b in device_set_realized (obj=0x5638996e0e70,
>>>>>> value=true, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:494
>>>>>> #5 0x00005638970f5e14 in property_set_bool (obj=0x5638996e0e70,
>>>>>> v=0x5638996f3770, name=0x56389759b141 "realized",
>>>>>> opaque=0x5638987893d0, errp=0x7ffd3c2b84e0)
>>>>>> at ../qom/object.c:2374
>>>>>> #6 0x00005638970f39f8 in object_property_set (obj=0x5638996e0e70,
>>>>>> name=0x56389759b141 "realized", v=0x5638996f3770, errp=0x7ffd3c2b84e0)
>>>>>> at ../qom/object.c:1449
>>>>>> #7 0x00005638970f8586 in object_property_set_qobject
>>>>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized",
>>>>>> value=0x5638996df900, errp=0x7ffd3c2b84e0)
>>>>>> at ../qom/qom-qobject.c:28
>>>>>> #8 0x00005638970f3d8d in object_property_set_bool
>>>>>> (obj=0x5638996e0e70, name=0x56389759b141 "realized", value=true,
>>>>>> errp=0x7ffd3c2b84e0)
>>>>>> at ../qom/object.c:1519
>>>>>> #9 0x00005638970eacb0 in qdev_realize (dev=0x5638996e0e70,
>>>>>> bus=0x563898cf3c20, errp=0x7ffd3c2b84e0) at ../hw/core/qdev.c:276
>>>>>> #10 0x0000563896dba675 in qdev_device_add_from_qdict
>>>>>> (opts=0x5638996dfe50, from_json=false, errp=0x7ffd3c2b84e0) at ../
>>>>>> system/qdev-monitor.c:714
>>>>>> #11 0x0000563896dba721 in qdev_device_add (opts=0x563898786150,
>>>>>> errp=0x56389855dc40 <error_fatal>) at ../system/qdev-monitor.c:733
>>>>>> #12 0x0000563896dc48f1 in device_init_func (opaque=0x0,
>>>>>> opts=0x563898786150, errp=0x56389855dc40 <error_fatal>) at ../system/
>>>>>> vl.c:1207
>>>>>> #13 0x000056389737a6cc in qemu_opts_foreach
>>>>>> (list=0x563898427b60 <qemu_device_opts>, func=0x563896dc48ca
>>>>>> <device_init_func>, opaque=0x0, errp=0x56389855dc40 <error_fatal>)
>>>>>> at ../util/qemu-option.c:1135
>>>>>> #14 0x0000563896dc89b5 in qemu_create_cli_devices () at ../system/
>>>>>> vl.c:2745
>>>>>> #15 0x0000563896dc8c00 in qmp_x_exit_preconfig (errp=0x56389855dc40
>>>>>> <error_fatal>) at ../system/vl.c:2806
>>>>>> #16 0x0000563896dcb5de in qemu_init (argc=33, argv=0x7ffd3c2b8948)
>>>>>> at ../system/vl.c:3838
>>>>>> #17 0x0000563897297323 in main (argc=33, argv=0x7ffd3c2b8948) at ../
>>>>>> system/main.c:72
>>>>> So the attached adjusted version of your patch does seem to help. At
>>>>> least I can't reproduce the crash on my stand.
>>>> Thanks for the stack trace; the calls to SPICE_RING_INIT in init_qxl_ram
>>>> are
>>>> definitely harmful. Try V2 of the patch, attached, which skips the lines
>>>> of init_qxl_ram that modify guest memory.
>>>>
>>> Thanks, your v2 patch does seem to prevent the crash. Would you re-send
>>> it to the list as a proper fix?
>
> Yes. Was waiting for your confirmation.
>
>>>>> I'm wondering, could it be useful to explicitly mark all the reused
>>>>> memory regions readonly upon cpr-transfer, and then make them writable
>>>>> back again after the migration is done? That way we will be segfaulting
>>>>> early on instead of debugging tricky memory corruptions.
>>>> It's a useful debugging technique, but changing protection on a large
>>>> memory region
>>>> can be too expensive for production due to TLB shootdowns.
>>>>
>>>> Also, there are cases where writes are performed but the value is
>>>> guaranteed to
>>>> be the same:
>>>> qxl_post_load()
>>>> qxl_set_mode()
>>>> d->rom->mode = cpu_to_le32(modenr);
>>>> The value is the same because mode and shadow_rom.mode were passed in
>>>> vmstate
>>>> from old qemu.
>>>>
>>> There're also cases where devices' ROM might be re-initialized. E.g.
>>> this segfault occures upon further exploration of RO mapped RAM blocks:
>>>
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
>>>> 664 rep movsb
>>>> [Current thread is 1 (Thread 0x7f6e7d08b480 (LWP 310379))]
>>>> (gdb) bt
>>>> #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:664
>>>> #1 0x000055aa1d030ecd in rom_set_mr (rom=0x55aa200ba380, owner=0x55aa2019ac10, name=0x7fffb8272bc0 "/rom@etc/acpi/tables", ro=true)
>>>> at ../hw/core/loader.c:1032
>>>> #2 0x000055aa1d031577 in rom_add_blob
>>>> (name=0x55aa1da51f13 "etc/acpi/tables", blob=0x55aa208a1070, len=131072, max_len=2097152, addr=18446744073709551615, fw_file_name=0x55aa1da51f13 "etc/acpi/tables", fw_callback=0x55aa1d441f59 <acpi_build_update>, callback_opaque=0x55aa20ff0010, as=0x0, read_only=true) at ../hw/core/loader.c:1147
>>>> #3 0x000055aa1cfd788d in acpi_add_rom_blob
>>>> (update=0x55aa1d441f59 <acpi_build_update>, opaque=0x55aa20ff0010, blob=0x55aa1fc9aa00, name=0x55aa1da51f13 "etc/acpi/tables") at ../hw/acpi/utils.c:46
>>>> #4 0x000055aa1d44213f in acpi_setup () at ../hw/i386/acpi-build.c:2720
>>>> #5 0x000055aa1d434199 in pc_machine_done (notifier=0x55aa1ff15050, data=0x0) at ../hw/i386/pc.c:638
>>>> #6 0x000055aa1d876845 in notifier_list_notify (list=0x55aa1ea25c10 <machine_init_done_notifiers>, data=0x0) at ../util/notify.c:39
>>>> #7 0x000055aa1d039ee5 in qdev_machine_creation_done () at ../hw/core/machine.c:1749
>>>> #8 0x000055aa1d2c7b3e in qemu_machine_creation_done (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2779
>>>> #9 0x000055aa1d2c7c7d in qmp_x_exit_preconfig (errp=0x55aa1ea5cc40 <error_fatal>) at ../system/vl.c:2807
>>>> #10 0x000055aa1d2ca64f in qemu_init (argc=35, argv=0x7fffb82730e8) at ../system/vl.c:3838
>>>> #11 0x000055aa1d79638c in main (argc=35, argv=0x7fffb82730e8) at ../system/main.c:72
>>> I'm not sure whether ACPI tables ROM in particular is rewritten with the
>>> same content, but there might be cases where ROM can be read from file
>>> system upon initialization. That is undesirable as guest kernel
>>> certainly won't be too happy about sudden change of the device's ROM
>>> content.
>>>
>>> So the issue we're dealing with here is any unwanted memory related
>>> device initialization upon cpr.
>>>
>>> For now the only thing that comes to my mind is to make a test where we
>>> put as many devices as we can into a VM, make ram blocks RO upon cpr
>>> (and remap them as RW later after migration is done, if needed), and
>>> catch any unwanted memory violations. As Den suggested, we might
>>> consider adding that behaviour as a separate non-default option (or
>>> "migrate" command flag specific to cpr-transfer), which would only be
>>> used in the testing.
>
> I'll look into adding an option, but there may be too many false positives,
> such as the qxl_set_mode case above. And the maintainers may object to me
> eliminating the false positives by adding more CPR_IN tests, due to gratuitous
> (from their POV) ugliness.
>
> But I will use the technique to look for more write violations.
>
>>> Andrey
>> No way. ACPI with the source must be used in the same way as BIOSes
>> and optional ROMs.
>
> Yup, its a bug. Will fix.
>
> - Steve
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-03-07 21:01 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-02-28 17:39 [BUG, RFC] cpr-transfer: qxl guest driver crashes after migration Andrey Drobyshev
2025-02-28 18:13 ` Steven Sistare
2025-02-28 18:20 ` Steven Sistare
2025-02-28 18:35 ` Andrey Drobyshev
2025-02-28 18:37 ` Andrey Drobyshev
2025-03-04 19:05 ` Steven Sistare
2025-03-05 16:50 ` Andrey Drobyshev
2025-03-05 21:19 ` Steven Sistare
2025-03-06 9:59 ` Denis V. Lunev
2025-03-06 15:16 ` Andrey Drobyshev
2025-03-06 15:52 ` Denis V. Lunev
2025-03-06 16:13 ` Steven Sistare
2025-03-07 21:00 ` Steven Sistare
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).