From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0ECBC433FE for ; Mon, 7 Nov 2022 11:33:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 17E9F4B8C9; Mon, 7 Nov 2022 06:33:55 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IL-JHD9e7UBM; Mon, 7 Nov 2022 06:33:44 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 13CB94B8B9; Mon, 7 Nov 2022 06:33:44 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4D9854B8AF for ; Mon, 7 Nov 2022 06:33:43 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Yvro+9ADAV6c for ; Mon, 7 Nov 2022 06:33:38 -0500 (EST) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 8872A4B875 for ; Mon, 7 Nov 2022 06:33:38 -0500 (EST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CDFD2B8101F; Mon, 7 Nov 2022 11:33:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53127C433C1; Mon, 7 Nov 2022 11:33:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667820800; bh=XwhO6MA3akkpMZda29WLKHtXktbNkiqedyDL/Uu/0hg=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=fMFATCoA7bcu62RyEYLe3h3gMegZyEK31YaDCV3SnIszSLr2lrDnhZrxTtmxfYnLs EvLRre58+9VTGhUsDJEP5s/fSmuZvrw3FvEi8NIn4KDNnk5FMLN0AMUn9N/aLgwDxK zxYeU3oD6ngGS/Nz8cUpRchfMnpC4zHGCoRUT3JRH4YqSLl9F6PnTBE7pKPJqssC2w eKyT7JJM7w2InebCWAyTfE1U+dLPHe6sszhTjoybsQiyVeWhhBm4Zvapb/PGkF4yHp zrMq1DrOvDV1PRzj2eJjWhdXcg4NT5ANgTe/m9Q24cHxrDxNqCmNcDnTmj+iMokbJz r++fGfP8EygqA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1os0NR-004NP0-TT; Mon, 07 Nov 2022 11:33:18 +0000 Date: Mon, 07 Nov 2022 11:33:17 +0000 Message-ID: <861qqfq9s2.wl-maz@kernel.org> From: Marc Zyngier To: Gavin Shan Subject: Re: [PATCH v8 3/7] KVM: Support dirty ring in conjunction with bitmap In-Reply-To: References: <20221104234049.25103-1-gshan@redhat.com> <20221104234049.25103-4-gshan@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gshan@redhat.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, will@kernel.org, shan.gavin@gmail.com, bgardon@google.com, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, shuah@kernel.org, kvmarm@lists.cs.columbia.edu, ajones@ventanamicro.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: shuah@kernel.org, kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, dmatlack@google.com, shan.gavin@gmail.com, bgardon@google.com, kvmarm@lists.linux.dev, pbonzini@redhat.com, zhenyzha@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu, ajones@ventanamicro.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Mon, 07 Nov 2022 10:45:34 +0000, Gavin Shan wrote: > > Hi Marc, Peter, Oliver and Sean, > > On 11/5/22 7:40 AM, Gavin Shan wrote: > > ARM64 needs to dirty memory outside of a VCPU context when VGIC/ITS is > > enabled. It's conflicting with that ring-based dirty page tracking always > > requires a running VCPU context. > > > > Introduce a new flavor of dirty ring that requires the use of both VCPU > > dirty rings and a dirty bitmap. The expectation is that for non-VCPU > > sources of dirty memory (such as the VGIC/ITS on arm64), KVM writes to > > the dirty bitmap. Userspace should scan the dirty bitmap before migrating > > the VM to the target. > > > > Use an additional capability to advertise this behavior. The newly added > > capability (KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP) can't be enabled before > > KVM_CAP_DIRTY_LOG_RING_ACQ_REL on ARM64. In this way, the newly added > > capability is treated as an extension of KVM_CAP_DIRTY_LOG_RING_ACQ_REL. > > > > Suggested-by: Marc Zyngier > > Suggested-by: Peter Xu > > Co-developed-by: Oliver Upton > > Signed-off-by: Oliver Upton > > Signed-off-by: Gavin Shan > > Acked-by: Peter Xu > > --- > > Documentation/virt/kvm/api.rst | 33 ++++++++++++++++++----- > > include/linux/kvm_dirty_ring.h | 7 +++++ > > include/linux/kvm_host.h | 1 + > > include/uapi/linux/kvm.h | 1 + > > virt/kvm/Kconfig | 8 ++++++ > > virt/kvm/dirty_ring.c | 10 +++++++ > > virt/kvm/kvm_main.c | 49 +++++++++++++++++++++++++++------- > > 7 files changed, 93 insertions(+), 16 deletions(-) > > > > diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst > > index eee9f857a986..2ec32bd41792 100644 > > --- a/Documentation/virt/kvm/api.rst > > +++ b/Documentation/virt/kvm/api.rst > > @@ -8003,13 +8003,6 @@ flushing is done by the KVM_GET_DIRTY_LOG ioctl). To achieve that, one > > needs to kick the vcpu out of KVM_RUN using a signal. The resulting > > vmexit ensures that all dirty GFNs are flushed to the dirty rings. > > -NOTE: the capability KVM_CAP_DIRTY_LOG_RING and the corresponding > > -ioctl KVM_RESET_DIRTY_RINGS are mutual exclusive to the existing ioctls > > -KVM_GET_DIRTY_LOG and KVM_CLEAR_DIRTY_LOG. After enabling > > -KVM_CAP_DIRTY_LOG_RING with an acceptable dirty ring size, the virtual > > -machine will switch to ring-buffer dirty page tracking and further > > -KVM_GET_DIRTY_LOG or KVM_CLEAR_DIRTY_LOG ioctls will fail. > > - > > NOTE: KVM_CAP_DIRTY_LOG_RING_ACQ_REL is the only capability that > > should be exposed by weakly ordered architecture, in order to indicate > > the additional memory ordering requirements imposed on userspace when > > @@ -8018,6 +8011,32 @@ Architecture with TSO-like ordering (such as x86) are allowed to > > expose both KVM_CAP_DIRTY_LOG_RING and KVM_CAP_DIRTY_LOG_RING_ACQ_REL > > to userspace. > > +After using the dirty rings, the userspace needs to detect the > > capability > > +of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the ring structures > > +need to be backed by per-slot bitmaps. With this capability advertised > > +and supported, it means the architecture can dirty guest pages without > > +vcpu/ring context, so that some of the dirty information will still be > > +maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP > > +can't be enabled until the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL > > +has been enabled. > > + > > +Note that the bitmap here is only a backup of the ring structure, and > > +normally should only contain a very small amount of dirty pages, which > > +needs to be transferred during VM downtime. Collecting the dirty bitmap > > +should be the very last thing that the VMM does before transmitting state > > +to the target VM. VMM needs to ensure that the dirty state is final and > > +avoid missing dirty pages from another ioctl ordered after the bitmap > > +collection. > > + > > +To collect dirty bits in the backup bitmap, the userspace can use the > > +same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG shouldn't be needed > > +and its behavior is undefined since collecting the dirty bitmap always > > +happens in the last phase of VM's migration. > > + > > +NOTE: One example of using the backup bitmap is saving arm64 vgic/its > > +tables through KVM_DEV_ARM_{VGIC_GRP_CTRL, ITS_SAVE_TABLES} command on > > +KVM device "kvm-arm-vgic-its" during VM's migration. > > + > > In order to speed up the review and reduce unnecessary respins. After > collecting comments on PATCH[v8 3/7] from Marc and Peter, I would change > above description as below. Could you please confirm it looks good to you? > > In the 4th paragraph, the words starting from "Collecting the dirty bitmap..." > to the end, was previously suggested by Oliver, even Marc suggested to avoid > mentioning "migration". > > After enabling the dirty rings, the userspace needs to detect the > capability of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the ring > structures need to be backed by per-slot bitmaps. With this capability s/need/can/. If there was a *need*, it should happen automatically without user intervention. > advertised, it means the architecture can dirty guest pages without > vcpu/ring context, so that some of the dirty information will still be > maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP > can't be enabled if the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL > hasn't been enabled, or any memslot has been existing. > > Note that the bitmap here is only a backup of the ring structure. The > use of the ring and bitmap combination is only beneficial if there is > only a very small amount of memory that is dirtied out of vcpu/ring > context. Otherwise, the stand-alone per-slot bitmap mechanism needs to > be considered. > > To collect dirty bits in the backup bitmap, userspace can use the same > KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG isn't needed as long as all > the generation of the dirty bits is done in a single pass. Collecting > the dirty bitmap should be the very last thing that the VMM does before > transmitting state to the target VM. VMM needs to ensure that the dirty > state is final and avoid missing dirty pages from another ioctl ordered > after the bitmap collection. I would replace "transmitting state to the target VM" with "considering the state as complete", as I still object to casting this API into the migration mold. People use this stuff more far more than migration (checkpointing, for example). > > NOTE: One example of using the backup bitmap is saving arm64 vgic/its > tables through KVM_DEV_ARM_{VGIC_GRP_CTRL, ITS_SAVE_TABLES} command on > KVM device "kvm-arm-vgic-its" during VM's migration. Same remark about migration. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD39728E2 for ; Mon, 7 Nov 2022 11:33:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53127C433C1; Mon, 7 Nov 2022 11:33:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667820800; bh=XwhO6MA3akkpMZda29WLKHtXktbNkiqedyDL/Uu/0hg=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=fMFATCoA7bcu62RyEYLe3h3gMegZyEK31YaDCV3SnIszSLr2lrDnhZrxTtmxfYnLs EvLRre58+9VTGhUsDJEP5s/fSmuZvrw3FvEi8NIn4KDNnk5FMLN0AMUn9N/aLgwDxK zxYeU3oD6ngGS/Nz8cUpRchfMnpC4zHGCoRUT3JRH4YqSLl9F6PnTBE7pKPJqssC2w eKyT7JJM7w2InebCWAyTfE1U+dLPHe6sszhTjoybsQiyVeWhhBm4Zvapb/PGkF4yHp zrMq1DrOvDV1PRzj2eJjWhdXcg4NT5ANgTe/m9Q24cHxrDxNqCmNcDnTmj+iMokbJz r++fGfP8EygqA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1os0NR-004NP0-TT; Mon, 07 Nov 2022 11:33:18 +0000 Date: Mon, 07 Nov 2022 11:33:17 +0000 Message-ID: <861qqfq9s2.wl-maz@kernel.org> From: Marc Zyngier To: Gavin Shan Cc: kvmarm@lists.linux.dev, kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, will@kernel.org, shan.gavin@gmail.com, bgardon@google.com, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, shuah@kernel.org, kvmarm@lists.cs.columbia.edu, ajones@ventanamicro.com Subject: Re: [PATCH v8 3/7] KVM: Support dirty ring in conjunction with bitmap In-Reply-To: References: <20221104234049.25103-1-gshan@redhat.com> <20221104234049.25103-4-gshan@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gshan@redhat.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, will@kernel.org, shan.gavin@gmail.com, bgardon@google.com, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, shuah@kernel.org, kvmarm@lists.cs.columbia.edu, ajones@ventanamicro.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Message-ID: <20221107113317.6QIbiDQzGzix19_avBdlvxgpkZuvFPz5_3fqnflz-9s@z> On Mon, 07 Nov 2022 10:45:34 +0000, Gavin Shan wrote: > > Hi Marc, Peter, Oliver and Sean, > > On 11/5/22 7:40 AM, Gavin Shan wrote: > > ARM64 needs to dirty memory outside of a VCPU context when VGIC/ITS is > > enabled. It's conflicting with that ring-based dirty page tracking always > > requires a running VCPU context. > > > > Introduce a new flavor of dirty ring that requires the use of both VCPU > > dirty rings and a dirty bitmap. The expectation is that for non-VCPU > > sources of dirty memory (such as the VGIC/ITS on arm64), KVM writes to > > the dirty bitmap. Userspace should scan the dirty bitmap before migrating > > the VM to the target. > > > > Use an additional capability to advertise this behavior. The newly added > > capability (KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP) can't be enabled before > > KVM_CAP_DIRTY_LOG_RING_ACQ_REL on ARM64. In this way, the newly added > > capability is treated as an extension of KVM_CAP_DIRTY_LOG_RING_ACQ_REL. > > > > Suggested-by: Marc Zyngier > > Suggested-by: Peter Xu > > Co-developed-by: Oliver Upton > > Signed-off-by: Oliver Upton > > Signed-off-by: Gavin Shan > > Acked-by: Peter Xu > > --- > > Documentation/virt/kvm/api.rst | 33 ++++++++++++++++++----- > > include/linux/kvm_dirty_ring.h | 7 +++++ > > include/linux/kvm_host.h | 1 + > > include/uapi/linux/kvm.h | 1 + > > virt/kvm/Kconfig | 8 ++++++ > > virt/kvm/dirty_ring.c | 10 +++++++ > > virt/kvm/kvm_main.c | 49 +++++++++++++++++++++++++++------- > > 7 files changed, 93 insertions(+), 16 deletions(-) > > > > diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst > > index eee9f857a986..2ec32bd41792 100644 > > --- a/Documentation/virt/kvm/api.rst > > +++ b/Documentation/virt/kvm/api.rst > > @@ -8003,13 +8003,6 @@ flushing is done by the KVM_GET_DIRTY_LOG ioctl). To achieve that, one > > needs to kick the vcpu out of KVM_RUN using a signal. The resulting > > vmexit ensures that all dirty GFNs are flushed to the dirty rings. > > -NOTE: the capability KVM_CAP_DIRTY_LOG_RING and the corresponding > > -ioctl KVM_RESET_DIRTY_RINGS are mutual exclusive to the existing ioctls > > -KVM_GET_DIRTY_LOG and KVM_CLEAR_DIRTY_LOG. After enabling > > -KVM_CAP_DIRTY_LOG_RING with an acceptable dirty ring size, the virtual > > -machine will switch to ring-buffer dirty page tracking and further > > -KVM_GET_DIRTY_LOG or KVM_CLEAR_DIRTY_LOG ioctls will fail. > > - > > NOTE: KVM_CAP_DIRTY_LOG_RING_ACQ_REL is the only capability that > > should be exposed by weakly ordered architecture, in order to indicate > > the additional memory ordering requirements imposed on userspace when > > @@ -8018,6 +8011,32 @@ Architecture with TSO-like ordering (such as x86) are allowed to > > expose both KVM_CAP_DIRTY_LOG_RING and KVM_CAP_DIRTY_LOG_RING_ACQ_REL > > to userspace. > > +After using the dirty rings, the userspace needs to detect the > > capability > > +of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the ring structures > > +need to be backed by per-slot bitmaps. With this capability advertised > > +and supported, it means the architecture can dirty guest pages without > > +vcpu/ring context, so that some of the dirty information will still be > > +maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP > > +can't be enabled until the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL > > +has been enabled. > > + > > +Note that the bitmap here is only a backup of the ring structure, and > > +normally should only contain a very small amount of dirty pages, which > > +needs to be transferred during VM downtime. Collecting the dirty bitmap > > +should be the very last thing that the VMM does before transmitting state > > +to the target VM. VMM needs to ensure that the dirty state is final and > > +avoid missing dirty pages from another ioctl ordered after the bitmap > > +collection. > > + > > +To collect dirty bits in the backup bitmap, the userspace can use the > > +same KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG shouldn't be needed > > +and its behavior is undefined since collecting the dirty bitmap always > > +happens in the last phase of VM's migration. > > + > > +NOTE: One example of using the backup bitmap is saving arm64 vgic/its > > +tables through KVM_DEV_ARM_{VGIC_GRP_CTRL, ITS_SAVE_TABLES} command on > > +KVM device "kvm-arm-vgic-its" during VM's migration. > > + > > In order to speed up the review and reduce unnecessary respins. After > collecting comments on PATCH[v8 3/7] from Marc and Peter, I would change > above description as below. Could you please confirm it looks good to you? > > In the 4th paragraph, the words starting from "Collecting the dirty bitmap..." > to the end, was previously suggested by Oliver, even Marc suggested to avoid > mentioning "migration". > > After enabling the dirty rings, the userspace needs to detect the > capability of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the ring > structures need to be backed by per-slot bitmaps. With this capability s/need/can/. If there was a *need*, it should happen automatically without user intervention. > advertised, it means the architecture can dirty guest pages without > vcpu/ring context, so that some of the dirty information will still be > maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP > can't be enabled if the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL > hasn't been enabled, or any memslot has been existing. > > Note that the bitmap here is only a backup of the ring structure. The > use of the ring and bitmap combination is only beneficial if there is > only a very small amount of memory that is dirtied out of vcpu/ring > context. Otherwise, the stand-alone per-slot bitmap mechanism needs to > be considered. > > To collect dirty bits in the backup bitmap, userspace can use the same > KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG isn't needed as long as all > the generation of the dirty bits is done in a single pass. Collecting > the dirty bitmap should be the very last thing that the VMM does before > transmitting state to the target VM. VMM needs to ensure that the dirty > state is final and avoid missing dirty pages from another ioctl ordered > after the bitmap collection. I would replace "transmitting state to the target VM" with "considering the state as complete", as I still object to casting this API into the migration mold. People use this stuff more far more than migration (checkpointing, for example). > > NOTE: One example of using the backup bitmap is saving arm64 vgic/its > tables through KVM_DEV_ARM_{VGIC_GRP_CTRL, ITS_SAVE_TABLES} command on > KVM device "kvm-arm-vgic-its" during VM's migration. Same remark about migration. Thanks, M. -- Without deviation from the norm, progress is not possible.