From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6815CC4332F for ; Mon, 7 Nov 2022 09:46:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 663224B8BF; Mon, 7 Nov 2022 04:46:01 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PE1AggELhIct; Mon, 7 Nov 2022 04:46:00 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 0F0124B898; Mon, 7 Nov 2022 04:46:00 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id F3DCD4B836 for ; Mon, 7 Nov 2022 04:45:58 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xqmVl4w25Sne for ; Mon, 7 Nov 2022 04:45:57 -0500 (EST) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 8F9494B868 for ; Mon, 7 Nov 2022 04:45:57 -0500 (EST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2C80860F9D; Mon, 7 Nov 2022 09:45:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CBAC4C433B5; Mon, 7 Nov 2022 09:45:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667814356; bh=zQoTGd/6Ej84Wf6Su9oZKeEiDrTouIHTcjBSxVwXd8E=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=N3rlZjjTd8PqeF26i6aVJ96Gu9sJVTuDdKYTk5Yga8JSfXMtWLpyegvK6NyXmHc1J GckOWXxQLvTPyPgCyzSK7pbplRcqrPumOA1MJdsm857b0e/tCpgaywgHu+TPaN/hBR f3ghaiMzS74N5pYP4IXhkJqMp5Zhmic1tUdG3o3R7g6gKMxo7DSe/R0jKpml+4XzB/ gjDmN6wfTMttqv9ixE/Wbv3V6X7GBjR7CJ8dh9F3BuTzT7vTUN/feRNyKHyAYGnpNh JJeqlv6Oyr4E3FZEN6lu0PiwmiI2ZlAa+N/P2uukddaffKjvTgxy28QKkkM9vjnUIr dkO10w+doZtSA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oryhV-004M1E-Go; Mon, 07 Nov 2022 09:45:53 +0000 Date: Mon, 07 Nov 2022 09:45:53 +0000 Message-ID: <864jvbqer2.wl-maz@kernel.org> From: Marc Zyngier To: Gavin Shan Subject: Re: [PATCH v8 3/7] KVM: Support dirty ring in conjunction with bitmap In-Reply-To: <0f685682-ba39-53d4-766c-cc2b44ad48dc@redhat.com> References: <20221104234049.25103-1-gshan@redhat.com> <20221104234049.25103-4-gshan@redhat.com> <87o7tkf5re.wl-maz@kernel.org> <0f685682-ba39-53d4-766c-cc2b44ad48dc@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gshan@redhat.com, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, shuah@kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, ajones@ventanamicro.com, bgardon@google.com, dmatlack@google.com, will@kernel.org, suzuki.poulose@arm.com, alexandru.elisei@arm.com, pbonzini@redhat.com, peterx@redhat.com, seanjc@google.com, oliver.upton@linux.dev, zhenyzha@redhat.com, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, dmatlack@google.com, will@kernel.org, shan.gavin@gmail.com, bgardon@google.com, kvmarm@lists.linux.dev, pbonzini@redhat.com, zhenyzha@redhat.com, shuah@kernel.org, kvmarm@lists.cs.columbia.edu, ajones@ventanamicro.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Sun, 06 Nov 2022 21:40:49 +0000, Gavin Shan wrote: > > Hi Marc, > > On 11/6/22 11:43 PM, Marc Zyngier wrote: > > On Fri, 04 Nov 2022 23:40:45 +0000, > > Gavin Shan wrote: > >> > >> ARM64 needs to dirty memory outside of a VCPU context when VGIC/ITS is > >> enabled. It's conflicting with that ring-based dirty page tracking always > >> requires a running VCPU context. > >> > >> Introduce a new flavor of dirty ring that requires the use of both VCPU > >> dirty rings and a dirty bitmap. The expectation is that for non-VCPU > >> sources of dirty memory (such as the VGIC/ITS on arm64), KVM writes to > >> the dirty bitmap. Userspace should scan the dirty bitmap before migrating > >> the VM to the target. > >> > >> Use an additional capability to advertise this behavior. The newly added > >> capability (KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP) can't be enabled before > >> KVM_CAP_DIRTY_LOG_RING_ACQ_REL on ARM64. In this way, the newly added > >> capability is treated as an extension of KVM_CAP_DIRTY_LOG_RING_ACQ_REL. > >> > >> Suggested-by: Marc Zyngier > >> Suggested-by: Peter Xu > >> Co-developed-by: Oliver Upton > >> Signed-off-by: Oliver Upton > >> Signed-off-by: Gavin Shan > >> Acked-by: Peter Xu > >> --- > >> Documentation/virt/kvm/api.rst | 33 ++++++++++++++++++----- > >> include/linux/kvm_dirty_ring.h | 7 +++++ > >> include/linux/kvm_host.h | 1 + > >> include/uapi/linux/kvm.h | 1 + > >> virt/kvm/Kconfig | 8 ++++++ > >> virt/kvm/dirty_ring.c | 10 +++++++ > >> virt/kvm/kvm_main.c | 49 +++++++++++++++++++++++++++------- > >> 7 files changed, 93 insertions(+), 16 deletions(-) > >> > >> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst > >> index eee9f857a986..2ec32bd41792 100644 > >> --- a/Documentation/virt/kvm/api.rst > >> +++ b/Documentation/virt/kvm/api.rst > >> @@ -8003,13 +8003,6 @@ flushing is done by the KVM_GET_DIRTY_LOG ioctl). To achieve that, one > >> needs to kick the vcpu out of KVM_RUN using a signal. The resulting > >> vmexit ensures that all dirty GFNs are flushed to the dirty rings. > >> -NOTE: the capability KVM_CAP_DIRTY_LOG_RING and the > >> corresponding > >> -ioctl KVM_RESET_DIRTY_RINGS are mutual exclusive to the existing ioctls > >> -KVM_GET_DIRTY_LOG and KVM_CLEAR_DIRTY_LOG. After enabling > >> -KVM_CAP_DIRTY_LOG_RING with an acceptable dirty ring size, the virtual > >> -machine will switch to ring-buffer dirty page tracking and further > >> -KVM_GET_DIRTY_LOG or KVM_CLEAR_DIRTY_LOG ioctls will fail. > >> - > >> NOTE: KVM_CAP_DIRTY_LOG_RING_ACQ_REL is the only capability that > >> should be exposed by weakly ordered architecture, in order to indicate > >> the additional memory ordering requirements imposed on userspace when > >> @@ -8018,6 +8011,32 @@ Architecture with TSO-like ordering (such as x86) are allowed to > >> expose both KVM_CAP_DIRTY_LOG_RING and KVM_CAP_DIRTY_LOG_RING_ACQ_REL > >> to userspace. > >> +After using the dirty rings, the userspace needs to detect the > >> capability > > > > using? or enabling? What comes after suggest the latter. > > > > s/using/enabling in next revision :) > > >> +of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the ring structures > >> +need to be backed by per-slot bitmaps. With this capability advertised > >> +and supported, it means the architecture can dirty guest pages without > > > > If it is advertised, it is supported, right? > > > > Yes, s/advertised and supported/advertised in next revision. > > >> +vcpu/ring context, so that some of the dirty information will still be > >> +maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP > >> +can't be enabled until the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL > >> +has been enabled. > >> + > >> +Note that the bitmap here is only a backup of the ring structure, and > >> +normally should only contain a very small amount of dirty pages, which > > > > I don't think we can claim this. It is whatever amount of memory is > > dirtied outside of a vcpu context, and we shouldn't make any claim > > regarding the number of dirty pages. > > > > It's the pre-requisite to use the backup bitmap. Otherwise, the guest > will experience long down-time during migration, as mentioned by Peter > in another thread. So it's appropriate to mention the limit of dirty > pages here. See my alternative wording for this in the other sub-thread. > > >> +needs to be transferred during VM downtime. Collecting the dirty bitmap > >> +should be the very last thing that the VMM does before transmitting state > >> +to the target VM. VMM needs to ensure that the dirty state is final and > >> +avoid missing dirty pages from another ioctl ordered after the bitmap > >> +collection. > >> + > >> +To collect dirty bits in the backup bitmap, the userspace can use the > >> +same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG shouldn't be needed > >> +and its behavior is undefined since collecting the dirty bitmap always > >> +happens in the last phase of VM's migration. > > > > It isn't clear to me why KVM_CLEAR_DIRTY_LOG should be called out. If > > you have multiple devices that dirty the memory, such as multiple > > ITSs, why shouldn't userspace be allowed to snapshot the dirty state > > multiple time? This doesn't seem like a reasonable restriction, and I > > really dislike the idea of undefined behaviour here. > > > > It was actually documenting the expected QEMU's usage. With QEMU > excluded, KVM_CLEAR_DIRTY_LOG can be used as usual. Undefined behavior > seems not precise here. We can improve it like below, to avoid talking > about 'undefined behaviour'. > > To collect dirty bits in the backup bitmap, the userspace can use the > same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG shouldn't be needed > since collecting the dirty bitmap always happens in the last phase of > VM's migration. That's better, but the "shouldn't be needed" wording makes things ambiguous, and we shouldn't mention migration at all (this is not the only purpose of this API). I'd suggest this: To collect dirty bits in the backup bitmap, userspace can use the same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG isn't needed as long as all the generation of the dirty bits is done in a single pass. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E6DC7F for ; Mon, 7 Nov 2022 09:45:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CBAC4C433B5; Mon, 7 Nov 2022 09:45:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667814356; bh=zQoTGd/6Ej84Wf6Su9oZKeEiDrTouIHTcjBSxVwXd8E=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=N3rlZjjTd8PqeF26i6aVJ96Gu9sJVTuDdKYTk5Yga8JSfXMtWLpyegvK6NyXmHc1J GckOWXxQLvTPyPgCyzSK7pbplRcqrPumOA1MJdsm857b0e/tCpgaywgHu+TPaN/hBR f3ghaiMzS74N5pYP4IXhkJqMp5Zhmic1tUdG3o3R7g6gKMxo7DSe/R0jKpml+4XzB/ gjDmN6wfTMttqv9ixE/Wbv3V6X7GBjR7CJ8dh9F3BuTzT7vTUN/feRNyKHyAYGnpNh JJeqlv6Oyr4E3FZEN6lu0PiwmiI2ZlAa+N/P2uukddaffKjvTgxy28QKkkM9vjnUIr dkO10w+doZtSA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oryhV-004M1E-Go; Mon, 07 Nov 2022 09:45:53 +0000 Date: Mon, 07 Nov 2022 09:45:53 +0000 Message-ID: <864jvbqer2.wl-maz@kernel.org> From: Marc Zyngier To: Gavin Shan Cc: kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, shuah@kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, ajones@ventanamicro.com, bgardon@google.com, dmatlack@google.com, will@kernel.org, suzuki.poulose@arm.com, alexandru.elisei@arm.com, pbonzini@redhat.com, peterx@redhat.com, seanjc@google.com, oliver.upton@linux.dev, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: Re: [PATCH v8 3/7] KVM: Support dirty ring in conjunction with bitmap In-Reply-To: <0f685682-ba39-53d4-766c-cc2b44ad48dc@redhat.com> References: <20221104234049.25103-1-gshan@redhat.com> <20221104234049.25103-4-gshan@redhat.com> <87o7tkf5re.wl-maz@kernel.org> <0f685682-ba39-53d4-766c-cc2b44ad48dc@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gshan@redhat.com, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, shuah@kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, ajones@ventanamicro.com, bgardon@google.com, dmatlack@google.com, will@kernel.org, suzuki.poulose@arm.com, alexandru.elisei@arm.com, pbonzini@redhat.com, peterx@redhat.com, seanjc@google.com, oliver.upton@linux.dev, zhenyzha@redhat.com, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Message-ID: <20221107094553.neWz3xbnlwoT8crZliOlaZ_Ib1VPcGBTOds_qQSd4HM@z> On Sun, 06 Nov 2022 21:40:49 +0000, Gavin Shan wrote: > > Hi Marc, > > On 11/6/22 11:43 PM, Marc Zyngier wrote: > > On Fri, 04 Nov 2022 23:40:45 +0000, > > Gavin Shan wrote: > >> > >> ARM64 needs to dirty memory outside of a VCPU context when VGIC/ITS is > >> enabled. It's conflicting with that ring-based dirty page tracking always > >> requires a running VCPU context. > >> > >> Introduce a new flavor of dirty ring that requires the use of both VCPU > >> dirty rings and a dirty bitmap. The expectation is that for non-VCPU > >> sources of dirty memory (such as the VGIC/ITS on arm64), KVM writes to > >> the dirty bitmap. Userspace should scan the dirty bitmap before migrating > >> the VM to the target. > >> > >> Use an additional capability to advertise this behavior. The newly added > >> capability (KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP) can't be enabled before > >> KVM_CAP_DIRTY_LOG_RING_ACQ_REL on ARM64. In this way, the newly added > >> capability is treated as an extension of KVM_CAP_DIRTY_LOG_RING_ACQ_REL. > >> > >> Suggested-by: Marc Zyngier > >> Suggested-by: Peter Xu > >> Co-developed-by: Oliver Upton > >> Signed-off-by: Oliver Upton > >> Signed-off-by: Gavin Shan > >> Acked-by: Peter Xu > >> --- > >> Documentation/virt/kvm/api.rst | 33 ++++++++++++++++++----- > >> include/linux/kvm_dirty_ring.h | 7 +++++ > >> include/linux/kvm_host.h | 1 + > >> include/uapi/linux/kvm.h | 1 + > >> virt/kvm/Kconfig | 8 ++++++ > >> virt/kvm/dirty_ring.c | 10 +++++++ > >> virt/kvm/kvm_main.c | 49 +++++++++++++++++++++++++++------- > >> 7 files changed, 93 insertions(+), 16 deletions(-) > >> > >> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst > >> index eee9f857a986..2ec32bd41792 100644 > >> --- a/Documentation/virt/kvm/api.rst > >> +++ b/Documentation/virt/kvm/api.rst > >> @@ -8003,13 +8003,6 @@ flushing is done by the KVM_GET_DIRTY_LOG ioctl). To achieve that, one > >> needs to kick the vcpu out of KVM_RUN using a signal. The resulting > >> vmexit ensures that all dirty GFNs are flushed to the dirty rings. > >> -NOTE: the capability KVM_CAP_DIRTY_LOG_RING and the > >> corresponding > >> -ioctl KVM_RESET_DIRTY_RINGS are mutual exclusive to the existing ioctls > >> -KVM_GET_DIRTY_LOG and KVM_CLEAR_DIRTY_LOG. After enabling > >> -KVM_CAP_DIRTY_LOG_RING with an acceptable dirty ring size, the virtual > >> -machine will switch to ring-buffer dirty page tracking and further > >> -KVM_GET_DIRTY_LOG or KVM_CLEAR_DIRTY_LOG ioctls will fail. > >> - > >> NOTE: KVM_CAP_DIRTY_LOG_RING_ACQ_REL is the only capability that > >> should be exposed by weakly ordered architecture, in order to indicate > >> the additional memory ordering requirements imposed on userspace when > >> @@ -8018,6 +8011,32 @@ Architecture with TSO-like ordering (such as x86) are allowed to > >> expose both KVM_CAP_DIRTY_LOG_RING and KVM_CAP_DIRTY_LOG_RING_ACQ_REL > >> to userspace. > >> +After using the dirty rings, the userspace needs to detect the > >> capability > > > > using? or enabling? What comes after suggest the latter. > > > > s/using/enabling in next revision :) > > >> +of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the ring structures > >> +need to be backed by per-slot bitmaps. With this capability advertised > >> +and supported, it means the architecture can dirty guest pages without > > > > If it is advertised, it is supported, right? > > > > Yes, s/advertised and supported/advertised in next revision. > > >> +vcpu/ring context, so that some of the dirty information will still be > >> +maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP > >> +can't be enabled until the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL > >> +has been enabled. > >> + > >> +Note that the bitmap here is only a backup of the ring structure, and > >> +normally should only contain a very small amount of dirty pages, which > > > > I don't think we can claim this. It is whatever amount of memory is > > dirtied outside of a vcpu context, and we shouldn't make any claim > > regarding the number of dirty pages. > > > > It's the pre-requisite to use the backup bitmap. Otherwise, the guest > will experience long down-time during migration, as mentioned by Peter > in another thread. So it's appropriate to mention the limit of dirty > pages here. See my alternative wording for this in the other sub-thread. > > >> +needs to be transferred during VM downtime. Collecting the dirty bitmap > >> +should be the very last thing that the VMM does before transmitting state > >> +to the target VM. VMM needs to ensure that the dirty state is final and > >> +avoid missing dirty pages from another ioctl ordered after the bitmap > >> +collection. > >> + > >> +To collect dirty bits in the backup bitmap, the userspace can use the > >> +same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG shouldn't be needed > >> +and its behavior is undefined since collecting the dirty bitmap always > >> +happens in the last phase of VM's migration. > > > > It isn't clear to me why KVM_CLEAR_DIRTY_LOG should be called out. If > > you have multiple devices that dirty the memory, such as multiple > > ITSs, why shouldn't userspace be allowed to snapshot the dirty state > > multiple time? This doesn't seem like a reasonable restriction, and I > > really dislike the idea of undefined behaviour here. > > > > It was actually documenting the expected QEMU's usage. With QEMU > excluded, KVM_CLEAR_DIRTY_LOG can be used as usual. Undefined behavior > seems not precise here. We can improve it like below, to avoid talking > about 'undefined behaviour'. > > To collect dirty bits in the backup bitmap, the userspace can use the > same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG shouldn't be needed > since collecting the dirty bitmap always happens in the last phase of > VM's migration. That's better, but the "shouldn't be needed" wording makes things ambiguous, and we shouldn't mention migration at all (this is not the only purpose of this API). I'd suggest this: To collect dirty bits in the backup bitmap, userspace can use the same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG isn't needed as long as all the generation of the dirty bits is done in a single pass. Thanks, M. -- Without deviation from the norm, progress is not possible.