From mboxrd@z Thu Jan 1 00:00:00 1970 From: Radim =?utf-8?B?S3LEjW3DocWZ?= Subject: Re: [PATCH 3/6] KVM: Dirty memory tracking for performant checkpointing and improved live migration Date: Tue, 3 May 2016 16:11:19 +0200 Message-ID: <20160503141118.GA27975@potion> References: <201604261855.u3QItn85024244@dev1.sn.stratus.com> <33d8668e-2bba-af91-069e-6452609a6ff0@linux.intel.com> <20160429181911.GA2687@potion> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: "Cao, Lei" , Paolo Bonzini , "kvm@vger.kernel.org" To: "Huang, Kai" Return-path: Received: from mx1.redhat.com ([209.132.183.28]:34014 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755374AbcECOLX (ORCPT ); Tue, 3 May 2016 10:11:23 -0400 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: 2016-05-03 18:06+1200, Huang, Kai: > Actually my concern is, with your new mechanism to track guest dirty pages, > there will be two logdirty mechanisms (using bitmap and your per-vcpu list), > which I think is not good as it's a little bit redundant, given both > mechanisms are used for dirty page logging. > > I think your main concern of current bitmap mechanism is scanning bitmap > takes lots of time, especially when only few pages get dirty, you still have > to scan the entire bitmap, which results in bad performance if you runs > checkpoint very frequently. My suggestion is, instead of introducing two > logdirty data structures, maybe you can try to use another more efficient > data structure instead of bitmap for both current logdirty mechanism and > your new interfaces. Maybe Xen's log-dirty tree is a good reference. A sparse structure (buffer, tree, ...) also needs a mechanism to grow (store new entries), so concurrent accesses become a problem, because there has to be synchronization. I think that per-vcpu structure becomes mandatory when thousands VCPUs dirty memory at the same time. > Maybe Xen's log-dirty tree is a good reference. Is there some top-level overview? >>From a glance at the code, it looked like GPA bitmap sparsified with radix tree in a manner similar to the page table hierarchy. > Of course this is just my concern and I'll leave it to maintainers. I too would prefer if both userspace interfaces used a common backend. A possible backend for that is vcpu -> memslot -> sparse dirty log We should have dynamic sparse dirty log, to avoid wasting memory when there are many small memslots, but a linear structure is probably still fine. We don't care which vcpu dirtied the page, so it seems like a waste to have them in the hierarchy, but I can't think of designs where the sparse dirty log is rooted in memslot and its updates scale well. 'memslot -> sparse dirty log' usually evolve into buffering on the VCPU side before writing to the memslot or aren't efficient for sparse dataset. Where do you think is the balance between 'memslot -> bitmap' and 'vcpu -> memslot -> dirty buffer'? Thanks.