Live update: q35 and pc-i440fx support (vapic)

Live update: q35 and pc-i440fx support (vapic)

[Chaney, Ben]

Hello Steve,

Thank you for posting the qemu cpr-transfer patches on qemu-devel. I am experimenting with them, and I noticed that cpr-transfer is failing on some qemu machine types. cpr-transfer is working for me on the microvm machine type but failing on q35 and pc-i440fx. When running in those configurations I get the following error on the new qemu process:


    qemu-system-x86_64: error while loading state for instance 0x0 of device 'kvm-tpr-opt'
    2025-02-18T14:46:52.550319Z qemu-system-x86_64: load of migration failed: Operation not permitted

The issue occurs when initializing the vapic device in the new qemu process. In vapic_map_rom_writable, rom_size is set to zero. This causes the following condition to be triggered:

    ram = memory_region_get_ram_ptr(section.mr);
    rom_size = ram[rom_paddr + 2] * ROM_BLOCK_SIZE;
    if (rom_size == 0) {
    return -1;
    }

This occurs on the qemu master branch (tested on commit 9736ee382e95ead06a838fe0b0498e0cb3845270) with the following qemu command line options:

Terminal 1:

    /opt/qemu-build/bin/qemu-system-x86_64 -nographic -cpu host,migratable=yes,-vmx,-svm,invpcid=off -display vnc=unix:/opt/bchaney-tmp/vnc.socket -enable-kvm -name bchaney_test1,debug-threads=on -smp 4 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on -rtc clock=vm -no-user-config -nodefaults -msg timestamp=on -bios /opt/linode-seabios/roms/bios.bin -machine q35,accel=kvm -cdrom /opt/bchaney-tmp/ubuntu-24.04.1-live-server-amd64.iso -qmp stdio

Terminal 2:

    /opt/qemu-build/bin/qemu-system-x86_64 -nographic -cpu host,migratable=yes,-vmx,-svm,invpcid=off -display vnc=unix:/opt/bchaney-tmp/vnc.socket -enable-kvm -name bchaney_test1,debug-threads=on -smp 4 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on -rtc clock=vm -no-user-config -nodefaults -msg timestamp=on -bios /opt/linode-seabios/roms/bios.bin -machine q35,accel=kvm -cdrom /opt/bchaney-tmp/ubuntu-24.04.1-live-server-amd64.iso -incoming '{"channel-type": "main", "addr": { "transport": "socket", "type": "unix", "path": "/opt/bchaney-tmp/main.sock"}}' -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "/opt/bchaney-tmp/cpr.sock"}}' -monitor stdio

Qmp commands executed (in Terminal 1):

    {"execute":"qmp_capabilities"}
    {"execute": "query-status"}
    {"execute":"migrate-set-parameters",
       "arguments":{"mode":"cpr-transfer"}}
    {"execute": "migrate", "arguments": { "channels": [
        {"channel-type": "main", "addr": { "transport": "socket", "type": "unix",
                   "path": "/opt/bchaney-tmp/main.sock"}},          
    {"channel-type": "cpr",
         "addr": { "transport": "socket", "type": "unix",
                    "path": "/opt/bchaney-tmp/cpr.sock"}}]}}
    {"execute": "query-status"}

Is this a hardware configuration that is currently intended to be supported? If not, will it be supported in the future?

Thank you,
Ben Chaney

Re: Live update: q35 and pc-i440fx support (vapic)

[Fabiano Rosas]

"Chaney, Ben" <bchaney@akamai.com> writes:

> Hello Steve,
>
> Thank you for posting the qemu cpr-transfer patches on qemu-devel. I am experimenting with them, and I noticed that cpr-transfer is failing on some qemu machine types. cpr-transfer is working for me on the microvm machine type but failing on q35 and pc-i440fx. When running in those configurations I get the following error on the new qemu process:
>
>
>     qemu-system-x86_64: error while loading state for instance 0x0 of device 'kvm-tpr-opt'
>     2025-02-18T14:46:52.550319Z qemu-system-x86_64: load of migration failed: Operation not permitted
>
> The issue occurs when initializing the vapic device in the new qemu process. In vapic_map_rom_writable, rom_size is set to zero. This causes the following condition to be triggered:
>
>     ram = memory_region_get_ram_ptr(section.mr);
>     rom_size = ram[rom_paddr + 2] * ROM_BLOCK_SIZE;
>     if (rom_size == 0) {
>     return -1;
>     }

Good job debugging the most opaque error message the migration code can emit.

>
> This occurs on the qemu master branch (tested on commit 9736ee382e95ead06a838fe0b0498e0cb3845270) with the following qemu command line options:
>
> Terminal 1:
>
>     /opt/qemu-build/bin/qemu-system-x86_64 -nographic -cpu host,migratable=yes,-vmx,-svm,invpcid=off -display vnc=unix:/opt/bchaney-tmp/vnc.socket -enable-kvm -name bchaney_test1,debug-threads=on -smp 4 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on -rtc clock=vm -no-user-config -nodefaults -msg timestamp=on -bios /opt/linode-seabios/roms/bios.bin -machine q35,accel=kvm -cdrom /opt/bchaney-tmp/ubuntu-24.04.1-live-server-amd64.iso -qmp stdio

You're missing an option that tells QEMU to actually make use of the
shared memory backend:

-machine memory-backend=ram0

You might be looking at an older version of the documentation. We've
fixed that and Steve is working on a better error message for it.

>
> Terminal 2:
>
>     /opt/qemu-build/bin/qemu-system-x86_64 -nographic -cpu host,migratable=yes,-vmx,-svm,invpcid=off -display vnc=unix:/opt/bchaney-tmp/vnc.socket -enable-kvm -name bchaney_test1,debug-threads=on -smp 4 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on -rtc clock=vm -no-user-config -nodefaults -msg timestamp=on -bios /opt/linode-seabios/roms/bios.bin -machine q35,accel=kvm -cdrom /opt/bchaney-tmp/ubuntu-24.04.1-live-server-amd64.iso -incoming '{"channel-type": "main", "addr": { "transport": "socket", "type": "unix", "path": "/opt/bchaney-tmp/main.sock"}}' -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "/opt/bchaney-tmp/cpr.sock"}}' -monitor stdio
>
> Qmp commands executed (in Terminal 1):
>
>     {"execute":"qmp_capabilities"}
>     {"execute": "query-status"}
>     {"execute":"migrate-set-parameters",
>        "arguments":{"mode":"cpr-transfer"}}
>     {"execute": "migrate", "arguments": { "channels": [
>         {"channel-type": "main", "addr": { "transport": "socket", "type": "unix",
>                    "path": "/opt/bchaney-tmp/main.sock"}},          
>     {"channel-type": "cpr",
>          "addr": { "transport": "socket", "type": "unix",
>                     "path": "/opt/bchaney-tmp/cpr.sock"}}]}}
>     {"execute": "query-status"}
>
> Is this a hardware configuration that is currently intended to be supported? If not, will it be supported in the future?
>
> Thank you,
> Ben Chaney

Re: Live update: q35 and pc-i440fx support (vapic)

[Steven Sistare]

On 2/19/2025 4:12 PM, Fabiano Rosas wrote:
> "Chaney, Ben" <bchaney@akamai.com> writes:
> 
>> Hello Steve,
>>
>> Thank you for posting the qemu cpr-transfer patches on qemu-devel. I am experimenting with them, and I noticed that cpr-transfer is failing on some qemu machine types. cpr-transfer is working for me on the microvm machine type but failing on q35 and pc-i440fx. When running in those configurations I get the following error on the new qemu process:
>>
>>
>>      qemu-system-x86_64: error while loading state for instance 0x0 of device 'kvm-tpr-opt'
>>      2025-02-18T14:46:52.550319Z qemu-system-x86_64: load of migration failed: Operation not permitted
>>
>> The issue occurs when initializing the vapic device in the new qemu process. In vapic_map_rom_writable, rom_size is set to zero. This causes the following condition to be triggered:
>>
>>      ram = memory_region_get_ram_ptr(section.mr);
>>      rom_size = ram[rom_paddr + 2] * ROM_BLOCK_SIZE;
>>      if (rom_size == 0) {
>>      return -1;
>>      }
> 
> Good job debugging the most opaque error message the migration code can emit.
> 
>>
>> This occurs on the qemu master branch (tested on commit 9736ee382e95ead06a838fe0b0498e0cb3845270) with the following qemu command line options:
>>
>> Terminal 1:
>>
>>      /opt/qemu-build/bin/qemu-system-x86_64 -nographic -cpu host,migratable=yes,-vmx,-svm,invpcid=off -display vnc=unix:/opt/bchaney-tmp/vnc.socket -enable-kvm -name bchaney_test1,debug-threads=on -smp 4 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on -rtc clock=vm -no-user-config -nodefaults -msg timestamp=on -bios /opt/linode-seabios/roms/bios.bin -machine q35,accel=kvm -cdrom /opt/bchaney-tmp/ubuntu-24.04.1-live-server-amd64.iso -qmp stdio
> 
> You're missing an option that tells QEMU to actually make use of the
> shared memory backend:
> 
> -machine memory-backend=ram0
> 
> You might be looking at an older version of the documentation. We've
> fixed that and Steve is working on a better error message for it.

Thank-you Fabiano for replying.  I was OOTO last week.

Ben, let me know if -machine memory-backend=ram0 does not fix the problem.

- Steve

>> Terminal 2:
>>
>>      /opt/qemu-build/bin/qemu-system-x86_64 -nographic -cpu host,migratable=yes,-vmx,-svm,invpcid=off -display vnc=unix:/opt/bchaney-tmp/vnc.socket -enable-kvm -name bchaney_test1,debug-threads=on -smp 4 -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m 4G -machine aux-ram-share=on -rtc clock=vm -no-user-config -nodefaults -msg timestamp=on -bios /opt/linode-seabios/roms/bios.bin -machine q35,accel=kvm -cdrom /opt/bchaney-tmp/ubuntu-24.04.1-live-server-amd64.iso -incoming '{"channel-type": "main", "addr": { "transport": "socket", "type": "unix", "path": "/opt/bchaney-tmp/main.sock"}}' -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "/opt/bchaney-tmp/cpr.sock"}}' -monitor stdio
>>
>> Qmp commands executed (in Terminal 1):
>>
>>      {"execute":"qmp_capabilities"}
>>      {"execute": "query-status"}
>>      {"execute":"migrate-set-parameters",
>>         "arguments":{"mode":"cpr-transfer"}}
>>      {"execute": "migrate", "arguments": { "channels": [
>>          {"channel-type": "main", "addr": { "transport": "socket", "type": "unix",
>>                     "path": "/opt/bchaney-tmp/main.sock"}},
>>      {"channel-type": "cpr",
>>           "addr": { "transport": "socket", "type": "unix",
>>                      "path": "/opt/bchaney-tmp/cpr.sock"}}]}}
>>      {"execute": "query-status"}
>>
>> Is this a hardware configuration that is currently intended to be supported? If not, will it be supported in the future?
>>
>> Thank you,
>> Ben Chaney