From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xiao Guangrong Subject: Re: [PATCH 1/2] KVM: MMU: Mark sp mmio cached when creating mmio spte Date: Thu, 14 Mar 2013 13:45:32 +0800 Message-ID: <514163FC.6000202@linux.vnet.ibm.com> References: <20130312174333.7f76148e.yoshikawa_takuya_b1@lab.ntt.co.jp> <20130312174440.5d5199ee.yoshikawa_takuya_b1@lab.ntt.co.jp> <5140094F.5080700@linux.vnet.ibm.com> <20130313162816.c62899dc.yoshikawa_takuya_b1@lab.ntt.co.jp> <51402DDA.607@linux.vnet.ibm.com> <20130313123358.GM11223@redhat.com> <51407441.4020200@linux.vnet.ibm.com> <20130313224056.8c9c87f4d95b332d2273a685@gmail.com> <514087A0.1000704@linux.vnet.ibm.com> <20130314015821.GA13261@amt.cnet> <51415C7A.402@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Marcelo Tosatti , Takuya Yoshikawa , Gleb Natapov , Takuya Yoshikawa , kvm@vger.kernel.org To: Xiao Guangrong Return-path: Received: from e23smtp08.au.ibm.com ([202.81.31.141]:59658 "EHLO e23smtp08.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752770Ab3CNFpo (ORCPT ); Thu, 14 Mar 2013 01:45:44 -0400 Received: from /spool/local by e23smtp08.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 14 Mar 2013 15:43:45 +1000 Received: from d23relay05.au.ibm.com (d23relay05.au.ibm.com [9.190.235.152]) by d23dlp01.au.ibm.com (Postfix) with ESMTP id 9A1B62CE8051 for ; Thu, 14 Mar 2013 16:45:37 +1100 (EST) Received: from d23av02.au.ibm.com (d23av02.au.ibm.com [9.190.235.138]) by d23relay05.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r2E5Wfw446596256 for ; Thu, 14 Mar 2013 16:32:42 +1100 Received: from d23av02.au.ibm.com (loopback [127.0.0.1]) by d23av02.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r2E5jZkE006962 for ; Thu, 14 Mar 2013 16:45:36 +1100 In-Reply-To: <51415C7A.402@linux.vnet.ibm.com> Sender: kvm-owner@vger.kernel.org List-ID: On 03/14/2013 01:13 PM, Xiao Guangrong wrote: > On 03/14/2013 09:58 AM, Marcelo Tosatti wrote: >> On Wed, Mar 13, 2013 at 10:05:20PM +0800, Xiao Guangrong wrote: >>> On 03/13/2013 09:40 PM, Takuya Yoshikawa wrote: >>>> On Wed, 13 Mar 2013 20:42:41 +0800 >>>> Xiao Guangrong wrote: >>>> >>>>>>>>> How about save all mmio spte into a mmio-rmap? >>>>>>>> >>>>>>>> The problem is that other mmu code would need to care about the pointers >>>>>>>> stored in the new rmap list: when mmu_shrink zaps shadow pages for example. >>>>>>> >>>>>>> It is not hard... all the codes have been wrapped by *zap_spte*. >>>>>>> >>>>>> So are you going to send a patch? What do you think about applying this >>>>>> as temporary solution? >>>>> >>>>> Hi Gleb, >>>>> >>>>> Since it only needs small change based on this patch, I think we can directly >>>>> apply the rmap-based way. >>>>> >>>>> Takuya, could you please do this? ;) >>>> >>>> Though I'm fine with my making the patch better, I'm still thinking >>>> about the bad side of it, though. >>>> >>>> In zap_spte, don't we need to search the pointer to be removed from the >>>> global mmio-rmap list? How long can that list be? >>> >>> It is not bad. On softmmu, the rmap list has already been long more than 300. >>> On hardmmu, normally the mmio spte is not frequently zapped (just set not clear). >>> >>> The worst case is zap-all-mmio-spte that removes all mmio-spte. This operation >>> can be speed up after applying my previous patch: >>> KVM: MMU: fast drop all spte on the pte_list >>> >>>> >>>> Implementing it will/may not be difficult but I'm not sure if we would >>>> get pure improvement. Unless it becomes 99% sure, I think we should >>>> first take a basic approach. >>> >>> I definitely sure zapping all mmio-sptes is fast than zapping mmio shadow >>> pages. ;) >> >> With a huge number of shadow pages (think 512GB guest, 262144 pte-level >> shadow pages to map), it might be a problem. > > That is one of the reasons why i think zap mmio shadow page is not good. ;) > > This patch needs to walk all shadow pages to find all mmio shadow page out > and zap them, it depends on how much memory is used on guest (huge memory > causes huge shadow page as you said). But the time of zapping mmio spte is > constant, no matter of memory used. > >> >>>> What do you think? >>> >>> I am considering if zap all shadow page is faster enough (after my patchset), do >>> we really need to care it? >> >> Still needed: your patch reduces kvm_mmu_zap_all() time, but as you can >> see with huge memory sized guests 100% improvement over the current >> situation will be a bottleneck (and as you noted the deletion case is >> still unsolved). > > The improvement can be greater if more memory is used. (I only used 2G memory in > guest since my test case is 32bit program which can not use huge memory, and > not lock contention in my testcase.) > > Actually, the time complexity of current kvm_mmu_zap_all is the same as zap ^^^^^ Sorry, not current way. It is the optimizing way in my patchset.