Re: [PATCH 4/4] migration: Introduce POSTCOPY_DEVICE state

["Dr. David Alan Gilbert" <dave@treblig.org>]

* Jiří Denemark (jdenemar@redhat.com) wrote:
> On Wed, Oct 01, 2025 at 16:10:44 +0100, Daniel P. Berrangé wrote:
> > On Wed, Oct 01, 2025 at 11:05:59AM +0000, Dr. David Alan Gilbert wrote:
> > > * Jiří Denemark (jdenemar@redhat.com) wrote:
> > > > On Tue, Sep 30, 2025 at 16:04:54 -0400, Peter Xu wrote:
> > > > > On Tue, Sep 30, 2025 at 09:53:31AM +0200, Jiří Denemark wrote:
> > > > > > On Thu, Sep 25, 2025 at 14:22:06 -0400, Peter Xu wrote:
> > > > > > > On Thu, Sep 25, 2025 at 01:54:40PM +0200, Jiří Denemark wrote:
> > > > > > > > On Mon, Sep 15, 2025 at 13:59:15 +0200, Juraj Marcin wrote:
> > > > > > > So far, dest QEMU will try to resume the VM after getting RUN command, that
> > > > > > > is what loadvm_postcopy_handle_run_bh() does, and it will (when autostart=1
> > > > > > > set): (1) firstly try to activate all block devices, iff it succeeded, (2)
> > > > > > > do vm_start(), at the end of which RESUME event will be generated.  So
> > > > > > > RESUME currently implies both disk activation success, and vm start worked.
> > > > > > > 
> > > > > > > > may still fail when locking disks fails (not sure if this is the only
> > > > > > > > way cont may fail). In this case we cannot cancel the migration on the
> > > > > > > 
> > > > > > > Is there any known issue with locking disks that dest would fail?  This
> > > > > > > really sound like we should have the admin taking a look.
> > > > > > 
> > > > > > Oh definitely, it would be some kind of an storage access issue on the
> > > > > > destination. But we'd like to give the admin an option to actually do
> > > > > > anything else than just killing the VM :-) Either by automatically
> > > > > > canceling the migration or allowing recovery once storage issues are
> > > > > > solved.
> > > > > 
> > > > > The problem is, if the storage locking stopped working properly, then how
> > > > > to guarantee the shared storage itself is working properly?
> > > > > 
> > > > > When I was replying previously, I was expecting the admin taking a look to
> > > > > fix the storage, I didn't expect the VM can still be recovered anymore if
> > > > > there's no confidence that the block devices will work all fine.  The
> > > > > locking errors to me may imply a block corruption already, or should I not
> > > > > see it like that?
> > > > 
> > > > If the storage itself is broken, there's clearly nothing we can do. But
> > > > the thing is we're accessing it from two distinct hosts. So while it may
> > > > work on the source, it can be broken on the destination. For example,
> > > > connection between the destination host and the storage may be broken.
> > > > Not sure how often this can happen in real life, but we have a bug
> > > > report that (artificially) breaking storage access on the destination
> > > > results in paused VM on the source which can only be killed.
> > > 
> > > I've got a vague memory that a tricky case is when some of your storage
> > > devices are broken on the destination, but not all.
> > > So you tell the block layer you want to take them on the destination
> > > some take their lock, one fails;  now what state are you in?
> > > I'm not sure if the block layer had a way of telling you what state
> > > you were in when I was last involved in that.
> > 
> > As long as the target QEMU CPUs have NOT started running, then
> > no I/O writes should have been sent to the storage, so the storage
> > should still be in a consistent state, and thus we can still try
> > to fail back to the source QEMU.
> > 
> > The "fun" problem here is that just because we /try/ to fail back
> > to the source QEMU, does not imply the source QEMU will now succeed
> > in re-acquiring the locks it just released a short time ago.
> 
> Right, but if we manage to get the source QEMU out of postcopy migration
> is such scenario, the VM may be manually resumed once storage issues are
> solved. So yes, we can't always rollback, but we should at least allow
> some kind of manual recovery.
> 
> > Consider the classic dead NFS server problem. The target may have
> > acquired 1 lock and failed on another lock because of a service
> > interruption to the NFS server. Well the target can't neccessarily
> > release the lock that it did successfully acquire now. So if we
> > fail back to the source, it'll be unable to reacquire the lock as
> > the target still holds it.
> > 
> > This doesn't mean we shouldn't try to fail back, but there will
> > always be some failures scenarios we'll struggle to recover from.
> > 
> > The "migration paused" state is our last chance, as that leaves
> > both QEMU's present while the admin tries to fix the underlying
> > problems.
> 
> IIUC from my conversation with Juraj switching to postcopy-paused can
> only happen when CPUs were already started.

But if you:
  a) Kill the destination
  b) Try to restart the source
  c) The source IO fails to restart

you could take the source to a normal 'paused' like you can do if storage
fails on the source. (as in -drive werror=stop  ?)

Dave

> Jirka
> 
-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/