From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91015C02183 for ; Fri, 17 Jan 2025 19:07:15 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tYrfs-0003uN-1X; Fri, 17 Jan 2025 14:06:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tYrfZ-0003tH-Ou for qemu-devel@nongnu.org; Fri, 17 Jan 2025 14:06:16 -0500 Received: from smtp-out1.suse.de ([2a07:de40:b251:101:10:150:64:1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tYrfW-000699-Vp for qemu-devel@nongnu.org; Fri, 17 Jan 2025 14:06:13 -0500 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9893921185; Fri, 17 Jan 2025 19:06:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737140767; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z18urq6cAEi94oRS3jSYXG7CGeQYatN8WWXVtaB/Ggo=; b=h8mgvjQbe4xYDJz1YfDOFoBXGrTHC6cJoOo1ZF0YGZSQCejq2mVzN5OLqmdYDIw2GlgOHc arc5pxUONO/yBfLj2kKGZB/Ix8guNIQ1XpHGlBOoa3/sc+tckryiPC4tgcBQ/s9ENZZaun nus0vWyG75pHHIpkUtrpp92LwsfR9Ro= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737140767; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z18urq6cAEi94oRS3jSYXG7CGeQYatN8WWXVtaB/Ggo=; b=gcWzxpVTuH8QM6bFF4qHhB4zXQUSOdkZj6iaTsUevFfMTG+RwOLcl7lg+eXQGIZhFSrxKo xI+5yToy5V5XCUAw== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=h8mgvjQb; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=gcWzxpVT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737140767; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z18urq6cAEi94oRS3jSYXG7CGeQYatN8WWXVtaB/Ggo=; b=h8mgvjQbe4xYDJz1YfDOFoBXGrTHC6cJoOo1ZF0YGZSQCejq2mVzN5OLqmdYDIw2GlgOHc arc5pxUONO/yBfLj2kKGZB/Ix8guNIQ1XpHGlBOoa3/sc+tckryiPC4tgcBQ/s9ENZZaun nus0vWyG75pHHIpkUtrpp92LwsfR9Ro= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737140767; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z18urq6cAEi94oRS3jSYXG7CGeQYatN8WWXVtaB/Ggo=; b=gcWzxpVTuH8QM6bFF4qHhB4zXQUSOdkZj6iaTsUevFfMTG+RwOLcl7lg+eXQGIZhFSrxKo xI+5yToy5V5XCUAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 0EB4F13332; Fri, 17 Jan 2025 19:06:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id DeSULx6qimdHQgAAD6G6ig (envelope-from ); Fri, 17 Jan 2025 19:06:06 +0000 From: Fabiano Rosas To: Steven Sistare , qemu-devel@nongnu.org Cc: Peter Xu , David Hildenbrand , Marcel Apfelbaum , Eduardo Habkost , Philippe Mathieu-Daude , Paolo Bonzini , "Daniel P. Berrange" , Markus Armbruster Subject: Re: [PATCH V7 24/24] migration: cpr-transfer documentation In-Reply-To: <19a1f016-981e-49ea-9ead-bc64f69fa0c1@oracle.com> References: <1736967650-129648-1-git-send-email-steven.sistare@oracle.com> <1736967650-129648-25-git-send-email-steven.sistare@oracle.com> <87tt9xec2y.fsf@suse.de> <68bee183-ba2b-420c-b885-a6ba630f74fe@oracle.com> <87r051e9xb.fsf@suse.de> <19a1f016-981e-49ea-9ead-bc64f69fa0c1@oracle.com> Date: Fri, 17 Jan 2025 16:06:04 -0300 Message-ID: <87o705dzvn.fsf@suse.de> MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Queue-Id: 9893921185 X-Rspamd-Action: no action X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; FUZZY_BLOCKED(0.00)[rspamd.com]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_CC(0.00)[redhat.com,gmail.com,habkost.net,linaro.org]; DKIM_TRACE(0.00)[suse.de:+]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_SEVEN(0.00)[10]; TAGGED_RCPT(0.00)[]; MISSING_XM_UA(0.00)[]; RECEIVED_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:106:10:150:64:167:received]; ASN(0.00)[asn:25478, ipnet:::/0, country:RU]; RCVD_VIA_SMTP_AUTH(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid, suse.de:dkim, oracle.com:email, imap1.dmz-prg2.suse.org:rdns, imap1.dmz-prg2.suse.org:helo] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org Received-SPF: pass client-ip=2a07:de40:b251:101:10:150:64:1; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Steven Sistare writes: > On 1/17/2025 10:29 AM, Fabiano Rosas wrote: >> Steven Sistare writes: >> >>> On 1/17/2025 9:42 AM, Fabiano Rosas wrote: >>>> Steve Sistare writes: >>>> >>>>> Add documentation for the cpr-transfer migration mode. >>>>> >>>>> Signed-off-by: Steve Sistare >>>>> Reviewed-by: Peter Xu >>>>> --- >>>>> docs/devel/migration/CPR.rst | 182 ++++++++++++++++++++++++++++++++++++++++++- >>>>> 1 file changed, 180 insertions(+), 2 deletions(-) >>>>> >>>>> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst >>>>> index 63c3647..d6021d5 100644 >>>>> --- a/docs/devel/migration/CPR.rst >>>>> +++ b/docs/devel/migration/CPR.rst >>>>> @@ -5,7 +5,7 @@ CPR is the umbrella name for a set of migration modes in which the >>>>> VM is migrated to a new QEMU instance on the same host. It is >>>>> intended for use when the goal is to update host software components >>>>> that run the VM, such as QEMU or even the host kernel. At this time, >>>>> -cpr-reboot is the only available mode. >>>>> +the cpr-reboot and cpr-transfer modes are available. >>>>> >>>>> Because QEMU is restarted on the same host, with access to the same >>>>> local devices, CPR is allowed in certain cases where normal migration >>>>> @@ -53,7 +53,7 @@ RAM is copied to the migration URI. >>>>> Outgoing: >>>>> * Set the migration mode parameter to ``cpr-reboot``. >>>>> * Set the ``x-ignore-shared`` capability if desired. >>>>> - * Issue the ``migrate`` command. It is recommended the the URI be a >>>>> + * Issue the ``migrate`` command. It is recommended the URI be a >>>>> ``file`` type, but one can use other types such as ``exec``, >>>>> provided the command captures all the data from the outgoing side, >>>>> and provides all the data to the incoming side. >>>>> @@ -145,3 +145,181 @@ Caveats >>>>> >>>>> cpr-reboot mode may not be used with postcopy, background-snapshot, >>>>> or COLO. >>>>> + >>>>> +cpr-transfer mode >>>>> +----------------- >>>>> + >>>>> +This mode allows the user to transfer a guest to a new QEMU instance >>>>> +on the same host with minimal guest pause time, by preserving guest >>>>> +RAM in place, albeit with new virtual addresses in new QEMU. Devices >>>>> +and their pinned memory pages will also be preserved in a future QEMU >>>>> +release. >>>>> + >>>>> +The user starts new QEMU on the same host as old QEMU, with command- >>>>> +line arguments to create the same machine, plus the ``-incoming`` >>>>> +option for the main migration channel, like normal live migration. >>>>> +In addition, the user adds a second -incoming option with channel >>>>> +type ``cpr``. This CPR channel must support file descriptor transfer >>>>> +with SCM_RIGHTS, i.e. it must be a UNIX domain socket. >>>>> + >>>>> +To initiate CPR, the user issues a migrate command to old QEMU, >>>>> +adding a second migration channel of type ``cpr`` in the channels >>>>> +argument. Old QEMU stops the VM, saves state to the migration >>>>> +channels, and enters the postmigrate state. Execution resumes in >>>>> +new QEMU. >>>>> + >>>>> +New QEMU reads the CPR channel before opening a monitor, hence >>>>> +the CPR channel cannot be specified in the list of channels for a >>>>> +migrate-incoming command. It may only be specified on the command >>>>> +line. >>>>> + >>>>> +Usage >>>>> +^^^^^ >>>>> + >>>>> +Memory backend objects must have the ``share=on`` attribute. >>>>> + >>>>> +The VM must be started with the ``-machine aux-ram-share=on`` >>>>> +option. This causes implicit RAM blocks (those not described by >>>>> +a memory-backend object) to be allocated by mmap'ing a memfd. >>>>> +Examples include VGA and ROM. >>>>> + >>>>> +Outgoing: >>>>> + * Set the migration mode parameter to ``cpr-transfer``. >>>>> + * Issue the ``migrate`` command, containing a main channel and >>>>> + a cpr channel. >>>>> + >>>>> +Incoming: >>>>> + * Start new QEMU with two ``-incoming`` options. >>>>> + * If the VM was running when the outgoing ``migrate`` command was >>>>> + issued, then QEMU automatically resumes VM execution. >>>>> + >>>>> +Caveats >>>>> +^^^^^^^ >>>>> + >>>>> +cpr-transfer mode may not be used with postcopy, background-snapshot, >>>>> +or COLO. >>>>> + >>>>> +memory-backend-epc is not supported. >>>>> + >>>>> +The main incoming migration channel address cannot be a file type. >>>>> + >>>>> +If the main incoming channel address is an inet socket, then the port >>>>> +cannot be 0 (meaning dynamically choose a port). >>>>> + >>>>> +When using ``-incoming defer``, you must issue the migrate command to >>>>> +old QEMU before issuing any monitor commands to new QEMU, because new >>>>> +QEMU blocks waiting to read from the cpr channel before starting its >>>>> +monitor, and old QEMU does not write to the channel until the migrate >>>>> +command is issued. However, new QEMU does not open and read the >>>>> +main migration channel until you issue the migrate incoming command. >>>>> + >>>>> +Example 1: incoming channel >>>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> + >>>>> +In these examples, we simply restart the same version of QEMU, but >>>>> +in a real scenario one would start new QEMU on the incoming side. >>>>> +Note that new QEMU does not print the monitor prompt until old QEMU >>>>> +has issued the migrate command. The outgoing side uses QMP because >>>>> +HMP cannot specify a CPR channel. Some QMP responses are omitted for >>>>> +brevity. >>>>> + >>>>> +:: >>>>> + >>>>> + Outgoing: Incoming: >>>>> + >>>>> + # qemu-kvm -qmp stdio >>>>> + -object memory-backend-file,id=ram0,size=4G, >>>>> + mem-path=/dev/shm/ram0,share=on -m 4G >>>>> + -machine aux-ram-share=on >>>>> + ... >>>>> + # qemu-kvm -monitor stdio >>>>> + -incoming tcp:0:44444 >>>>> + -incoming '{"channel-type": "cpr", >>>>> + "addr": { "transport": "socket", >>>>> + "type": "unix", "path": "cpr.sock"}}' >>>>> + ... >>>> >>>> I'm attempting this and not having much success. Surely I'm missing >>>> something: >>>> >>>> >>>> $ qemu-system-x86_64 -cpu host -smp 16 -machine pc,accel=kvm \ >>>> -drive id=drive0,if=none,format=qcow2,file=img.qcow2 \ >>>> -device virtio-blk-pci,id=image1,drive=drive0,bootindex=0 \ >>>> -qmp unix:./dst-qmp.sock,server,wait=off \ >>>> -nographic -serial mon:stdio \ >>>> -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on \ >>>> -m 4G -machine aux-ram-share=on \ >>>> >>>> -incoming tcp:0:44444 \ >>>> -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "cpr.sock"}}' \ >>>> >>>> -trace loadvm_* -trace cpr_* -trace migration_* -trace migrate_* -trace qemu_loadvm_* >>>> >>>> cpr_transfer_input cpr.sock >>>> cpr_state_load cpr-transfer mode >>>> cpr_find_fd pc.bios, id 0 returns 15 >>>> cpr_find_fd pc.rom, id 0 returns 14 >>>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13 >>>> cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12 >>>> cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11 >>>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10 >>>> cpr_find_fd /rom@etc/table-loader, id 0 returns 8 >>>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3 >>>> migrate_set_state new state setup >>>> migration_socket_incoming_accepted >>>> migration_set_incoming_channel ioc=0x564dc31e7000 ioctype=qio-channel-socket >>>> migrate_set_state new state active >>>> loadvm_state_setup >>>> qemu_loadvm_state_section 1 >>>> qemu_loadvm_state_section_startfull 2(ram) 0 4 >>>> qemu_loadvm_state_section 3 >>>> qemu_loadvm_state_section_partend 2 >>>> qemu_loadvm_state_section 4 >>>> qemu_loadvm_state_section_startfull 0(timer) 0 2 >>>> qemu_loadvm_state_section 4 >>>> qemu_loadvm_state_section_startfull 1(slirp) 0 4 >>>> qemu_loadvm_state_section 4 >>>> qemu_loadvm_state_section_startfull 4(cpu_common) 0 1 >>>> qemu_loadvm_state_section 4 >>>> qemu_loadvm_state_section_startfull 5(cpu) 0 12 >>>> qemu_loadvm_state_section 4 >>>> qemu_loadvm_state_section_startfull 6(kvm-tpr-opt) 0 1 >>>> qemu-system-x86_64: error while loading state for instance 0x0 of device 'kvm-tpr-opt' >>>> qemu_loadvm_state_post_main -1 >>>> migrate_set_state new state failed >>>> migrate_error error=load of migration failed: Operation not permitted >>>> loadvm_state_cleanup >>>> qemu-system-x86_64: load of migration failed: Operation not permitted >>> >>> Check for a mismatch between the qemu args on the source vs dest. >>> Maybe -cpu. >> >> No.. they're the same: >> >> qemu-system-x86_64 -display none -cpu host -smp 4 -machine pc,accel=kvm >> -object >> memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m >> 4G -machine aux-ram-share=on -qmp stdio >> >> qemu-system-x86_64 -display none -cpu host -smp 4 -machine pc,accel=kvm >> -object >> memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m >> 4G -machine aux-ram-share=on -incoming tcp:0:44444 -incoming >> '{"channel-type": "cpr", "addr": { "transport": "socket", "type": >> "unix", "path": "cpr.sock"}}' -monitor stdio >> >> >> Here's the whole log, see if you spot something: >> >> $ (sleep 5; echo "{ 'execute': 'qmp_capabilities' } >> { 'execute': 'migrate-set-parameters','arguments':{ 'mode': 'cpr-transfer' } } >> { 'execute': 'migrate', 'arguments': \ >> { 'channels': [ \ >> {'channel-type': 'main', 'addr': { 'transport': 'socket', 'type': 'inet', \ >> 'host': '0', 'port': '44444' }}, \ >> {'channel-type': 'cpr', 'addr': { 'transport': 'socket', 'type': 'unix', \ >> 'path': 'cpr.sock' }} \ >> ]} \ >> }") | /home/fabiano/qemu-system-x86_64 -display none >> -cpu host -smp 4 -machine pc,accel=kvm -qmp stdio >> -object >> memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on >> -m 4G -machine aux-ram-share=on -trace cpr_* -trace >> migration_* -trace migrate_* -trace qemu_savevm_* >> -trace savevm_* >> >> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 9}, "package": "v9.2.0-987-gfd4129a8b9"}, "capabilities": ["oob"]}} >> cpr_find_fd pc.bios, id 0 returns -1 >> cpr_save_fd pc.bios, id 0, fd 22 >> cpr_find_fd pc.rom, id 0 returns -1 >> cpr_save_fd pc.rom, id 0, fd 23 >> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1 >> cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 24 >> cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns -1 >> cpr_save_fd 0000:00:02.0/vga.rom, id 0, fd 26 >> cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns -1 >> cpr_save_fd 0000:00:03.0/e1000.rom, id 0, fd 27 >> cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1 >> cpr_save_fd /rom@etc/acpi/tables, id 0, fd 28 >> cpr_find_fd /rom@etc/table-loader, id 0 returns -1 >> cpr_save_fd /rom@etc/table-loader, id 0, fd 29 >> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1 >> cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 30 >> migration_block_activation active-skipped >> {"return": {}} >> {"return": {}} >> migrate_set_state new state setup >> cpr_state_save cpr-transfer mode >> cpr_transfer_output cpr.sock >> {"return": {}} >> migration_socket_outgoing_connected hostname=0 >> migration_set_outgoing_channel ioc=0x55748ba10270 ioctype=qio-channel-socket hostname=0 err=(nil) >> {"timestamp": {"seconds": 1737127025, "microseconds": 606998}, "event": "STOP"} >> migration_completion_vm_stop ret 0 >> migration_transferred_bytes qemu_file 224 multifd 0 RDMA 0 >> savevm_state_header >> savevm_state_setup >> migration_bitmap_sync_start >> migration_bitmap_sync_end dirty_pages 0 >> migrate_set_state new state active >> migration_thread_setup_complete >> migration_transferred_bytes qemu_file 506 multifd 0 RDMA 0 >> migrate_pending_estimate estimate pending size 0 (pre = 0 post=0) >> migration_thread_low_pending 0 >> migrate_set_state new state device >> migration_block_activation inactive >> migration_precopy_complete >> savevm_section_start ram, section_id 2 >> migration_bitmap_sync_start >> migration_bitmap_sync_end dirty_pages 0 >> savevm_section_end ram, section_id 2 -> 0 >> savevm_section_start timer, section_id 0 >> savevm_section_end timer, section_id 0 -> 0 >> savevm_section_start slirp, section_id 1 >> savevm_section_end slirp, section_id 1 -> 0 >> savevm_section_start cpu_common, section_id 4 >> savevm_section_end cpu_common, section_id 4 -> 0 >> savevm_section_start cpu, section_id 5 >> savevm_section_end cpu, section_id 5 -> 0 >> savevm_section_start kvm-tpr-opt, section_id 6 >> savevm_section_end kvm-tpr-opt, section_id 6 -> 0 >> savevm_section_start apic, section_id 7 >> savevm_section_end apic, section_id 7 -> 0 >> savevm_section_start cpu_common, section_id 8 >> savevm_section_end cpu_common, section_id 8 -> 0 >> savevm_section_start cpu, section_id 9 >> savevm_section_end cpu, section_id 9 -> 0 >> savevm_section_start apic, section_id 10 >> savevm_section_end apic, section_id 10 -> 0 >> savevm_section_start cpu_common, section_id 11 >> savevm_section_end cpu_common, section_id 11 -> 0 >> savevm_section_start cpu, section_id 12 >> savevm_section_end cpu, section_id 12 -> 0 >> savevm_section_start apic, section_id 13 >> savevm_section_end apic, section_id 13 -> 0 >> savevm_section_start cpu_common, section_id 14 >> savevm_section_end cpu_common, section_id 14 -> 0 >> savevm_section_start cpu, section_id 15 >> savevm_section_end cpu, section_id 15 -> 0 >> savevm_section_start apic, section_id 16 >> savevm_section_end apic, section_id 16 -> 0 >> savevm_section_start kvmclock, section_id 17 >> savevm_section_end kvmclock, section_id 17 -> 0 >> savevm_section_start 0000:00:00.0/I440FX, section_id 18 >> savevm_section_end 0000:00:00.0/I440FX, section_id 18 -> 0 >> savevm_section_start PCIHost, section_id 19 >> savevm_section_end PCIHost, section_id 19 -> 0 >> savevm_section_start PCIBUS, section_id 20 >> savevm_section_end PCIBUS, section_id 20 -> 0 >> savevm_section_start fw_cfg, section_id 21 >> savevm_section_end fw_cfg, section_id 21 -> 0 >> savevm_section_start dma, section_id 22 >> savevm_section_end dma, section_id 22 -> 0 >> savevm_section_start dma, section_id 23 >> savevm_section_end dma, section_id 23 -> 0 >> savevm_section_start mc146818rtc, section_id 24 >> savevm_section_end mc146818rtc, section_id 24 -> 0 >> savevm_section_start 0000:00:01.1/ide, section_id 25 >> savevm_section_end 0000:00:01.1/ide, section_id 25 -> 0 >> savevm_section_start i2c_bus, section_id 26 >> savevm_section_end i2c_bus, section_id 26 -> 0 >> savevm_section_start 0000:00:01.3/piix4_pm, section_id 27 >> savevm_section_end 0000:00:01.3/piix4_pm, section_id 27 -> 0 >> savevm_section_start 0000:00:01.0/PIIX3, section_id 28 >> savevm_section_end 0000:00:01.0/PIIX3, section_id 28 -> 0 >> savevm_section_start i8259, section_id 29 >> savevm_section_end i8259, section_id 29 -> 0 >> savevm_section_start i8259, section_id 30 >> savevm_section_end i8259, section_id 30 -> 0 >> savevm_section_start ioapic, section_id 31 >> savevm_section_end ioapic, section_id 31 -> 0 >> savevm_section_start 0000:00:02.0/vga, section_id 32 >> savevm_section_end 0000:00:02.0/vga, section_id 32 -> 0 >> savevm_section_start hpet, section_id 33 >> savevm_section_end hpet, section_id 33 -> 0 >> savevm_section_start i8254, section_id 34 >> savevm_section_end i8254, section_id 34 -> 0 >> savevm_section_start pcspk, section_id 35 >> savevm_section_end pcspk, section_id 35 -> 0 >> savevm_section_start serial, section_id 36 >> savevm_section_end serial, section_id 36 -> 0 >> savevm_section_start parallel_isa, section_id 37 >> savevm_section_end parallel_isa, section_id 37 -> 0 >> savevm_section_start fdc, section_id 38 >> savevm_section_end fdc, section_id 38 -> 0 >> savevm_section_start ps2kbd, section_id 39 >> savevm_section_end ps2kbd, section_id 39 -> 0 >> savevm_section_start ps2mouse, section_id 40 >> savevm_section_end ps2mouse, section_id 40 -> 0 >> savevm_section_start pckbd, section_id 41 >> savevm_section_end pckbd, section_id 41 -> 0 >> savevm_section_start vmmouse, section_id 42 >> savevm_section_end vmmouse, section_id 42 -> 0 >> savevm_section_start port92, section_id 43 >> savevm_section_end port92, section_id 43 -> 0 >> savevm_section_start 0000:00:03.0/e1000, section_id 44 >> savevm_section_end 0000:00:03.0/e1000, section_id 44 -> 0 >> savevm_section_skip smbus-eeprom, section_id 45 >> savevm_section_skip smbus-eeprom, section_id 46 >> savevm_section_skip smbus-eeprom, section_id 47 >> savevm_section_skip smbus-eeprom, section_id 48 >> savevm_section_skip smbus-eeprom, section_id 49 >> savevm_section_skip smbus-eeprom, section_id 50 >> savevm_section_skip smbus-eeprom, section_id 51 >> savevm_section_skip smbus-eeprom, section_id 52 >> savevm_section_start acpi_build, section_id 53 >> savevm_section_end acpi_build, section_id 53 -> 0 >> savevm_section_start globalstate, section_id 54 >> migrate_global_state_pre_save saved state: running >> savevm_section_end globalstate, section_id 54 -> 0 >> migrate_error error=Unable to write to socket: Connection reset by peer >> migrate_set_state new state failed >> migration_thread_after_loop >> migration_block_activation active >> {"timestamp": {"seconds": 1737127025, "microseconds": 625404}, "event": "RESUME"} >> migrate_fd_cleanup >> savevm_state_cleanup >> qemu-system-x86_64: Unable to write to socket: Connection reset by peer >> >> >> $ /home/fabiano/qemu-system-x86_64 -display none -cpu host -smp 4 >> -machine pc,accel=kvm -object >> memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m >> 4G -machine aux-ram-share=on -incoming tcp:0:44444 -incoming >> '{"channel-type": "cpr", "addr": { "transport": "socket", "type": >> "unix", "path": "cpr.sock"}}' -trace loadvm_* -trace cpr_* -trace >> migration_* -trace migrate_* -monitor stdio >> >> cpr_transfer_input cpr.sock >> cpr_state_load cpr-transfer mode >> QEMU 9.2.50 monitor - type 'help' for more information >> cpr_find_fd pc.bios, id 0 returns 15 >> cpr_find_fd pc.rom, id 0 returns 14 >> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13 >> cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12 >> cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11 >> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10 >> cpr_find_fd /rom@etc/table-loader, id 0 returns 8 >> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3 >> migrate_set_state new state setup >> (qemu) migration_socket_incoming_accepted >> migration_set_incoming_channel ioc=0x5565cccb8e70 ioctype=qio-channel-socket >> migrate_set_state new state active >> loadvm_state_setup >> qemu-system-x86_64: error while loading state for instance 0x0 of device 'kvm-tpr-opt' >> migrate_set_state new state failed >> migrate_error error=load of migration failed: Operation not permitted >> loadvm_state_cleanup >> qemu-system-x86_64: load of migration failed: Operation not permitted > > Thank-you for the simple example. I reproduced the failure. > To fix, add "-machine aux-ram-share=on -machine memory-backend=ram0" > (The previous longer example had the former but lacked the latter). > Without that, the volatile pc.ram region is still in the mix. There you go, that kvm-tpr-opt message is almost always indicative of user error. I think because it's the first vmstate to be loaded. Nonetheless, we better update the documentation to: -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m -4G \ -machine memory-backend=ram0 \ -machine aux-ram-share=on > > I have a patch that adds a blocker if volatile ram is present, and would > have clearly diagnosed this problem. I will submit it now. Yes, I just tested and it's way better. Although the message asks for "share=on" and "aux-ram-share=on", which were already there. But there's not much we can do, basic proficiency with QEMU cmdline is a prerequisite. > > - Steve