From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DFE6C433FE for ; Thu, 29 Sep 2022 14:34:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id A397A4B0C2; Thu, 29 Sep 2022 10:34:25 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3-qXqf3RLioC; Thu, 29 Sep 2022 10:34:24 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 5D7B643479; Thu, 29 Sep 2022 10:34:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 61226410FF for ; Thu, 29 Sep 2022 10:34:23 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AvJlB-KnQ24A for ; Thu, 29 Sep 2022 10:34:22 -0400 (EDT) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 03FE040B90 for ; Thu, 29 Sep 2022 10:34:21 -0400 (EDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 26F7C617BE; Thu, 29 Sep 2022 14:34:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84833C433D7; Thu, 29 Sep 2022 14:34:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1664462060; bh=vLsdE+UiVydlSCVGTZRM03z1/D003eiRv4TLqV2gTeM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=dTVvCj6z2TdNiD86AQBznQuneWdpO1PBz7bRk4RQ13ctBQvgVbWkE1JQe11uazyET C5SPzmqE6uMFWRLqMtANKj5AFNSjh9rOo/9B9xWL4lKerAQn5mfOVTCmeL/t19gcvN k04Q035uelAJgy1p/8E1HRP2K+l6ViiG9j1eAnhSjDJRBRA20/EXwbkSRkcHINFsfw BmkWY1rXRrmLd8xFmhv/2DNI7FRrJ+A4Wqli8SZ7q7m6OlSbG6Uph3eU3A97EprZ/y uFUTKJzFokN+jQeEEWhLm3JkiYgwV5QPJUzDaIL0l0+2JtdBnnhEKWDuJee1gt/Am1 MWMcqj0YzHD0A== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oducE-00DYdm-54; Thu, 29 Sep 2022 15:34:18 +0100 Date: Thu, 29 Sep 2022 15:34:17 +0100 Message-ID: <86h70q6yhi.wl-maz@kernel.org> From: Marc Zyngier To: Peter Xu Subject: Re: [PATCH v4 3/6] KVM: arm64: Enable ring-based dirty memory tracking In-Reply-To: References: <20220927005439.21130-1-gshan@redhat.com> <20220927005439.21130-4-gshan@redhat.com> <86sfkc7mg8.wl-maz@kernel.org> <320005d1-fe88-fd6a-be91-ddb56f1aa80f@redhat.com> <87y1u3hpmp.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: peterx@redhat.com, gshan@redhat.com, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, catalin.marinas@arm.com, bgardon@google.com, shuah@kernel.org, andrew.jones@linux.dev, will@kernel.org, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, shan.gavin@gmail.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, oliver.upton@linux.dev X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, will@kernel.org, shan.gavin@gmail.com, bgardon@google.com, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, shuah@kernel.org, kvmarm@lists.cs.columbia.edu X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Wed, 28 Sep 2022 15:52:00 +0100, Peter Xu wrote: > > On Wed, Sep 28, 2022 at 09:25:34AM +0100, Marc Zyngier wrote: > > Hi Gavin, > > > > On Wed, 28 Sep 2022 00:47:43 +0100, > > Gavin Shan wrote: > > > > > I have rough idea as below. It's appreciated if you can comment before I'm > > > going a head for the prototype. The overall idea is to introduce another > > > dirty ring for KVM (kvm-dirty-ring). It's updated and visited separately > > > to dirty ring for vcpu (vcpu-dirty-ring). > > > > > > - When the various VGIC/ITS table base addresses are specified, kvm-dirty-ring > > > entries are added to mark those pages as 'always-dirty'. In mark_page_dirty_in_slot(), > > > those 'always-dirty' pages will be skipped, no entries pushed to vcpu-dirty-ring. > > > > > > - Similar to vcpu-dirty-ring, kvm-dirty-ring is accessed from userspace through > > > mmap(kvm->fd). However, there won't have similar reset interface. It means > > > 'struct kvm_dirty_gfn::flags' won't track any information as we do for > > > vcpu-dirty-ring. In this regard, kvm-dirty-ring is purely shared buffer to > > > advertise 'always-dirty' pages from host to userspace. > > > - For QEMU, shutdown/suspend/resume cases won't be concerning > > > us any more. The > > > only concerned case is migration. When the migration is about to complete, > > > kvm-dirty-ring entries are fetched and the dirty bits are updated to global > > > dirty page bitmap and RAMBlock's dirty page bitmap. For this, I'm still reading > > > the code to find the best spot to do it. > > > > I think it makes a lot of sense to have a way to log writes that are > > not generated by a vpcu, such as the GIC and maybe other things in the > > future, such as DMA traffic (some SMMUs are able to track dirty pages > > as well). > > > > However, I don't really see the point in inventing a new mechanism for > > that. Why don't we simply allow non-vpcu dirty pages to be tracked in > > the dirty *bitmap*? > > > > From a kernel perspective, this is dead easy: > > > > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c > > index 5b064dbadaf4..ae9138f29d51 100644 > > --- a/virt/kvm/kvm_main.c > > +++ b/virt/kvm/kvm_main.c > > @@ -3305,7 +3305,7 @@ void mark_page_dirty_in_slot(struct kvm *kvm, > > struct kvm_vcpu *vcpu = kvm_get_running_vcpu(); > > > > #ifdef CONFIG_HAVE_KVM_DIRTY_RING > > - if (WARN_ON_ONCE(!vcpu) || WARN_ON_ONCE(vcpu->kvm != kvm)) > > + if (WARN_ON_ONCE(vcpu && vcpu->kvm != kvm)) > > return; > > #endif > > > > @@ -3313,10 +3313,11 @@ void mark_page_dirty_in_slot(struct kvm *kvm, > > unsigned long rel_gfn = gfn - memslot->base_gfn; > > u32 slot = (memslot->as_id << 16) | memslot->id; > > > > - if (kvm->dirty_ring_size) > > + if (vpcu && kvm->dirty_ring_size) > > kvm_dirty_ring_push(&vcpu->dirty_ring, > > slot, rel_gfn); > > - else > > + /* non-vpcu dirtying ends up in the global bitmap */ > > + if (!vcpu && memslot->dirty_bitmap) > > set_bit_le(rel_gfn, memslot->dirty_bitmap); > > } > > } > > > > though I'm sure there is a few more things to it. > > Yes, currently the bitmaps are not created when rings are enabled. > kvm_prepare_memory_region() has: > > else if (!kvm->dirty_ring_size) { > r = kvm_alloc_dirty_bitmap(new); > > But I think maybe that's a solution worth considering. Using the rings > have a major challenge on the limitation of ring size, so that for e.g. an > ioctl we need to make sure the pages to dirty within an ioctl procedure > will not be more than the ring can take. Using dirty bitmap for a last > phase sync of constant (but still very small amount of) dirty pages does > sound reasonable and can avoid that complexity. The payoff is we'll need > to allocate both the rings and the bitmaps. I think that's a reasonable price to pay. > > > > > To me, this is just a relaxation of an arbitrary limitation, as the > > current assumption that only vcpus can dirty memory doesn't hold at > > all. > > The initial dirty ring proposal has a per-vm ring, but after we > investigated x86 we found that all legal dirty paths are with a vcpu > context (except one outlier on kvmgt which fixed within itself), so we > dropped the per-vm ring. > > One thing to mention is that DMAs should not count in this case because > that's from device perspective, IOW either IOMMU or SMMU dirty tracking > should be reported to the device driver that interacts with the userspace > not from KVM interfaces (e.g. vfio with VFIO_IOMMU_DIRTY_PAGES). That even > includes emulated DMA like vhost (VHOST_SET_LOG_BASE). I beg to disagree. How do you make this work once we share the CPU page tables with the SMMU, meaning that there is only one true source of dirty bits? You can't separate the two. Which is another reason why the dirty-ring approach is pretty limited on ARM: it is an artificial construct that doesn't really work once we get to this point. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm