From: "Michael S. Tsirkin" <mst@redhat.com>
To: Prasad Pandit <ppandit@redhat.com>
Cc: qemu-devel@nongnu.org, farosas@suse.de, jasowang@redhat.com,
mcoqueli@redhat.com, peterx@redhat.com,
Prasad Pandit <pjp@fedoraproject.org>
Subject: Re: [PATCH v2 0/2] Postcopy migration and vhost-user errors
Date: Tue, 10 Sep 2024 13:10:32 -0400 [thread overview]
Message-ID: <20240910131008-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20240828100914.105728-1-ppandit@redhat.com>
On Wed, Aug 28, 2024 at 03:39:12PM +0530, Prasad Pandit wrote:
> From: Prasad Pandit <pjp@fedoraproject.org>
>
> Hello,
>
> * virsh(1) offers multiple options to initiate Postcopy migration:
>
> 1) virsh migrate --postcopy --postcopy-after-precopy
> 2) virsh migrate --postcopy + virsh migrate-postcopy
> 3) virsh migrate --postcopy --timeout <N> --timeout-postcopy
>
> When Postcopy migration is invoked via method (2) or (3) above,
> the migrated guest on the destination host hangs sometimes.
>
> * During Postcopy migration, multiple threads are spawned on the destination
> host to start the guest and setup devices. One such thread starts vhost
> device via vhost_dev_start() function and another called fault_thread handles
> page faults in user space using kernel's userfaultfd(2) system.
>
> * When fault_thread exits upon completion of Postcopy migration, it sends a
> 'postcopy_end' message to the vhost-user device. But sometimes 'postcopy_end'
> message is sent while vhost device is being setup via vhost_dev_start().
>
> Thread-1 Thread-2
>
> vhost_dev_start postcopy_ram_incoming_cleanup
> vhost_device_iotlb_miss postcopy_notify
> vhost_backend_update_device_iotlb vhost_user_postcopy_notifier
> vhost_user_send_device_iotlb_msg vhost_user_postcopy_end
> process_message_reply process_message_reply
> vhost_user_read vhost_user_read
> vhost_user_read_header vhost_user_read_header
> "Fail to update device iotlb" "Failed to receive reply to postcopy_end"
>
> This creates confusion when vhost device receives 'postcopy_end' message while
> it is still trying to update IOTLB entries.
>
> This seems to leave the guest in a stranded/hung state because fault_thread
> has exited saying Postcopy migration has ended, but vhost-device is probably
> still expecting updates. QEMU logs following errors on the destination host
> ===
> ...
> qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
> qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
> qemu-kvm: vhost_user_postcopy_end: 700871,700900: Failed to receive reply to postcopy_end
> qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
> qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
> qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x8 instead of 0x5.
> qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
> qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x16 instead of 0x5.
> qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
> qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
> qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
> ===
So are we going to see a version with BQL?
> * Couple of patches here help to fix/handle these errors.
>
> Thank you.
> ---
> Prasad Pandit (2):
> vhost: fail device start if iotlb update fails
> vhost-user: add a request-reply lock
>
> hw/virtio/vhost-user.c | 74 ++++++++++++++++++++++++++++++++++
> hw/virtio/vhost.c | 6 ++-
> include/hw/virtio/vhost-user.h | 3 ++
> 3 files changed, 82 insertions(+), 1 deletion(-)
>
> --
> 2.46.0
next prev parent reply other threads:[~2024-09-10 17:11 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-28 10:09 [PATCH v2 0/2] Postcopy migration and vhost-user errors Prasad Pandit
2024-08-28 10:09 ` [PATCH v2 1/2] vhost: fail device start if iotlb update fails Prasad Pandit
2024-08-28 10:09 ` [PATCH v2 2/2] vhost-user: add a request-reply lock Prasad Pandit
2024-08-28 11:15 ` Michael S. Tsirkin
2024-08-29 5:39 ` Prasad Pandit
2024-08-29 6:23 ` Michael S. Tsirkin
2024-08-29 7:42 ` Michael S. Tsirkin
2024-08-29 9:15 ` Prasad Pandit
2024-08-29 9:21 ` Michael S. Tsirkin
2024-08-29 14:29 ` Peter Xu
2024-08-29 15:05 ` Michael S. Tsirkin
2024-08-29 20:29 ` Peter Xu
2024-09-10 17:10 ` Michael S. Tsirkin [this message]
2024-09-11 7:14 ` [PATCH v2 0/2] Postcopy migration and vhost-user errors Prasad Pandit
2024-09-11 9:46 ` Michael S. Tsirkin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240910131008-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=farosas@suse.de \
--cc=jasowang@redhat.com \
--cc=mcoqueli@redhat.com \
--cc=peterx@redhat.com \
--cc=pjp@fedoraproject.org \
--cc=ppandit@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).