Re: [RFC PATCH] migration: Synchronize CPUs sooner in postcopy switchover phase

[Juraj Marcin <jmarcin@redhat.com>]

Hi Peter,

On 2026-04-29 15:50, Peter Xu wrote:
> On Thu, Apr 23, 2026 at 05:45:20PM +0200, Juraj Marcin wrote:
> > From: Juraj Marcin <jmarcin@redhat.com>
> > 
> > Previously, the post init CPU synchronization with the accelerator, like
> > KVM, was performed in the bottom half of the POSTCOPY_RUN command
> > handler. However, this causes several problems.
> > 
> > First issue is that if CPU synchronization fails, the destination QEMU
> > crashes. However, it is too late to recover the source side as the
> > response to special PING has been already sent, and both sides are
> > already in the POSTCOPY_ACTIVE state.
> > 
> > By moving synchronization before responding, if the machine crashes, the
> > response is never sent and the source side can resume from the
> > POSTCOPY_DEVICE state.
> > 
> > Second issue is caused when migration is paused due to a network
> > failure or user command right after transitioning to the POSTCOPY_ACTIVE
> > state and the CPU synchronization causes a page fault. This page fault
> > blocks the CPU and the main QEMU threads and cannot be resolved until
> > postcopy migration is recovered. However, as libvirt also tries to
> > execute 'cont' QMP command at this time (destination side transitions
> > from POSTCOPY_DEVICE to POSTCOPY_ACTIVE), it will also halt waiting for
> > the response from the blocked main QEMU thread, unaware of the fact that
> > the migration is paused and needs to be recovered. Thus, it will wait
> > indefinitely and never report postcopy migration error.
> > 
> > When the CPU synchronization happens sooner, and the network fails
> > during it, the source side can transition from POSTCOPY_DEVICE to FAILED
> > state and resume safely. If migration is paused later, the main thread
> > won't be blocked by CPU synchronization and can respond to libvirt.
> > 
> > Signed-off-by: Juraj Marcin <jmarcin@redhat.com>
> > ---
> > I am posting this as RFC to discuss the point at which the CPU
> > synchronization should happen.
> > 
> > For the POSTCOPY_DEVICE state to be effective, this synchronization must
> > happen before the destination machine responds to the special PING
> > command. This leaves us with 2 options:
> > 
> > 1) In the PING command handler before responding to the specific
> >    request, as proposed in this patch.
> > 
> > 2) After loading CPU VMSD, for example in the post_load hook.
> > 
> > The first solution limits the number of places where synchronization
> > needs to happen, however, having it in the PING command handler feels
> > somewhat hacky.
> 
> Yes.
> 
> > 
> > I have also tested the second solution, and while it seems natural to
> > synchronize the CPU after its data is loaded in the post_load, there are
> > multiple CPU types and VMSDs and each one would need to call
> > synchronization in its post_load hook. This would basically revert Jan
> > Kiszka's commit [1] which refactored CPU synchronization and united it
> > to one place.
> 
> What I was thinking is not reverting all of it, but only removing the
> migration relevant paths for post_init().
> 
> For example, we have different reasons to sync CPU, and we have two
> directions to do that (from/to kernel, in KVM's context). For migration,
> what used to be confusing is why we need to have CPU specific post_init()
> hooks when each CPU has VMSD and post_load() on its own.
> 
> The other one is on savevm side (cpu_synchronize_all_states()) and we can
> at least keep it as-is for now, simplify the discussion.
> 
> So if we want to move this into any of such post_load(), it means removal
> of three call sites of post_init() only in migration/savevm.c, then do
> per-CPU's post_init() in its post_load() hook.
> 
> I don't know the real answer of this one; I recall you tried it out but hit
> some ARM specific issue.  Just to want to make sure we're on the same page
> on what you have experimented.

I thought I did, however, that ARM issue in the CI turned out to be
unrelated.

> 
> Said that, I do see some complexity over such a change already.  For
> example, cpu_vmstate_register() seems to be able to register more than one
> VMSDs for each vCPU.  I don't know if it means at least we can't simply put
> it into post_load() of vmstate_cpu_common, as when reaching there it is not
> guaranteed that all vCPUs' registers are uptodate..
> 
> Besides..
> 
> I do have another thought, though, to avoid the hacky part of this patch
> and looks pretty safe: dest QEMU does not reply directly to the
> QEMU_VM_PING_PACKAGED_LOADED ping, instead it sets a flag. Then we move the
> conditional PONG into loadvm_postcopy_handle_run().
> 
> With that, we can move post_init() into loadvm_postcopy_handle_run()
> altogether.
> 
>   loadvm_postcopy_handle_run():
>     ...
>     cpu_synchronize_all_post_init();
>     if (package_loaded_ping_received) {
>         send_pong();
>     }
>     migrate_set_state(POSTCOPY_DEVICE, POSTCOPY_ACTIVE);
>     migration_bh_schedule(mis);
> 
> Would this work?

Interesting idea, I think it should work, but I am a bit unsure about
delaying the PONG response in such way, if it could potentially break
anything.

Thank you!

Best regards,

Juraj Marcin

> 
> -- 
> Peter Xu
>