From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Date: Fri, 04 Sep 2015 14:22:31 +0000 Subject: Re: [Qemu-ppc] KVM memory slots limit on powerpc Message-Id: <1441376551.20355.427.camel@redhat.com> List-Id: References: <55E965FC.8080909@redhat.com> <55E96B87.8020900@de.ibm.com> <55E96CB9.4070100@suse.de> <55E96D6B.2070201@de.ibm.com> <55E97264.6090602@redhat.com> In-Reply-To: <55E97264.6090602@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Thomas Huth Cc: Christian Borntraeger , Alexander Graf , kvm-ppc@vger.kernel.org, qemu-ppc@nongnu.org, kvm@vger.kernel.org, David Gibson , Bharata B Rao On Fri, 2015-09-04 at 12:28 +0200, Thomas Huth wrote: > On 04/09/15 12:07, Christian Borntraeger wrote: > > Am 04.09.2015 um 12:04 schrieb Alexander Graf: > >> > >> > >> On 04.09.15 11:59, Christian Borntraeger wrote: > >>> Am 04.09.2015 um 11:35 schrieb Thomas Huth: > >>>> > >>>> Hi all, > >>>> > >>>> now that we get memory hotplugging for the spapr machine on qemu-ppc, > >>>> too, it seems like we easily can hit the amount of KVM-internal memory > >>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in > >>>> arch/powerpc/include/asm/kvm_host.h). For example, start > >>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m > >>>> 4G,slots2,maxmem@G" and then try to hot-plug all 32 DIMMs ... and > >>>> you'll see that it aborts way earlier already. > >>>> > >>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509 > >>>> already (+3 internal slots = 512) ... maybe we should now increase the > >>>> amount of slots on powerpc, too? Since we don't use internal slots on > >>>> POWER, would 512 be a good value? Or would less be sufficient, too? > >>> > >>> When you are at it, the s390 value should also be increased I guess. > >> > >> That constant defines the array size for the memslot array in struct kvm > >> which in turn again gets allocated by kzalloc, so it's pinned kernel > >> memory that is physically contiguous. Doing big allocations can turn > >> into problems during runtime. > >> > >> So maybe there is another way? Can we extend the memslot array size > >> dynamically somehow? Allocate it separately? How much memory does the > >> memslot array use up with 512 entries? > > > > Maybe some rcu protected scheme that doubles the amount of memslots for > > each overrun? Yes, that would be good and even reduce the footprint for > > systems with only a small number of memslots. > > Seems like Alex Williamson already posted a patchset for growable > memslots a couple of years ago: > > http://www.spinics.net/lists/kvm/msg50491.html > > But I didn't quite spot the result in that thread why it never has been > included upstream. Alex (W.), do you remember the outcome? IIRC it was simply thought to be more complicated than necessary. Once we started caching memory slot misses, the O(N) search of a larger fixed array didn't seem to bother anyone, so I abandoned the weight-balanced tree, which I think still had some re-balancing issues. Please run with the code and make it work if it's useful now. Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: [Qemu-ppc] KVM memory slots limit on powerpc Date: Fri, 04 Sep 2015 08:22:31 -0600 Message-ID: <1441376551.20355.427.camel@redhat.com> References: <55E965FC.8080909@redhat.com> <55E96B87.8020900@de.ibm.com> <55E96CB9.4070100@suse.de> <55E96D6B.2070201@de.ibm.com> <55E97264.6090602@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: Christian Borntraeger , Alexander Graf , kvm-ppc@vger.kernel.org, qemu-ppc@nongnu.org, kvm@vger.kernel.org, David Gibson , Bharata B Rao To: Thomas Huth Return-path: Received: from mx1.redhat.com ([209.132.183.28]:48158 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932559AbbIDOWd (ORCPT ); Fri, 4 Sep 2015 10:22:33 -0400 In-Reply-To: <55E97264.6090602@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Fri, 2015-09-04 at 12:28 +0200, Thomas Huth wrote: > On 04/09/15 12:07, Christian Borntraeger wrote: > > Am 04.09.2015 um 12:04 schrieb Alexander Graf: > >> > >> > >> On 04.09.15 11:59, Christian Borntraeger wrote: > >>> Am 04.09.2015 um 11:35 schrieb Thomas Huth: > >>>> > >>>> Hi all, > >>>> > >>>> now that we get memory hotplugging for the spapr machine on qemu-ppc, > >>>> too, it seems like we easily can hit the amount of KVM-internal memory > >>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in > >>>> arch/powerpc/include/asm/kvm_host.h). For example, start > >>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m > >>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and > >>>> you'll see that it aborts way earlier already. > >>>> > >>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509 > >>>> already (+3 internal slots = 512) ... maybe we should now increase the > >>>> amount of slots on powerpc, too? Since we don't use internal slots on > >>>> POWER, would 512 be a good value? Or would less be sufficient, too? > >>> > >>> When you are at it, the s390 value should also be increased I guess. > >> > >> That constant defines the array size for the memslot array in struct kvm > >> which in turn again gets allocated by kzalloc, so it's pinned kernel > >> memory that is physically contiguous. Doing big allocations can turn > >> into problems during runtime. > >> > >> So maybe there is another way? Can we extend the memslot array size > >> dynamically somehow? Allocate it separately? How much memory does the > >> memslot array use up with 512 entries? > > > > Maybe some rcu protected scheme that doubles the amount of memslots for > > each overrun? Yes, that would be good and even reduce the footprint for > > systems with only a small number of memslots. > > Seems like Alex Williamson already posted a patchset for growable > memslots a couple of years ago: > > http://www.spinics.net/lists/kvm/msg50491.html > > But I didn't quite spot the result in that thread why it never has been > included upstream. Alex (W.), do you remember the outcome? IIRC it was simply thought to be more complicated than necessary. Once we started caching memory slot misses, the O(N) search of a larger fixed array didn't seem to bother anyone, so I abandoned the weight-balanced tree, which I think still had some re-balancing issues. Please run with the code and make it work if it's useful now. Thanks, Alex