From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Huth Date: Fri, 04 Sep 2015 14:45:33 +0000 Subject: Re: [Qemu-ppc] KVM memory slots limit on powerpc Message-Id: <55E9AE8D.3030009@redhat.com> List-Id: References: <55E965FC.8080909@redhat.com> <55E96B87.8020900@de.ibm.com> <55E96CB9.4070100@suse.de> In-Reply-To: <55E96CB9.4070100@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Alexander Graf , Christian Borntraeger , kvm-ppc@vger.kernel.org, qemu-ppc@nongnu.org, kvm@vger.kernel.org Cc: David Gibson , Alex Williamson , Bharata B Rao On 04/09/15 12:04, Alexander Graf wrote: > > > On 04.09.15 11:59, Christian Borntraeger wrote: >> Am 04.09.2015 um 11:35 schrieb Thomas Huth: >>> >>> Hi all, >>> >>> now that we get memory hotplugging for the spapr machine on qemu-ppc, >>> too, it seems like we easily can hit the amount of KVM-internal memory >>> slots now ("#define KVM_USER_MEM_SLOTS 32" in >>> arch/powerpc/include/asm/kvm_host.h). For example, start >>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m >>> 4G,slots2,maxmem@G" and then try to hot-plug all 32 DIMMs ... and >>> you'll see that it aborts way earlier already. >>> >>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509 >>> already (+3 internal slots = 512) ... maybe we should now increase the >>> amount of slots on powerpc, too? Since we don't use internal slots on >>> POWER, would 512 be a good value? Or would less be sufficient, too? >> >> When you are at it, the s390 value should also be increased I guess. > > That constant defines the array size for the memslot array in struct kvm > which in turn again gets allocated by kzalloc, so it's pinned kernel > memory that is physically contiguous. Doing big allocations can turn > into problems during runtime. FWIW, I've just checked sizeof(struct kvm) with the current ppc64 kernel build from master branch, and it is 34144 bytes. So on a system that is using PAGE_SIZE = 64kB, there should be plenty of space left before we're getting into trouble. And even assuming the worst case, that we're on a system which still uses PAGE_SIZE = 4kB, the last page of the 34144 bytes is only filled with 1376 bytes, leaving 2720 bytes free right now. sizeof(struct kvm_memory_slot) is 48 bytes right now on powerpc, and you need two additional bytes per entry for the id_to_index array in struct kvm_memslots, i.e. we need 50 additional bytes per entry on ppc. That means we could increase KVM_USER_MEM_SLOTS by 2720 / 50 = 54 entries without getting into further trouble. I think we should leave some more additional bytes left in that last 4k page of the struct kvm region, ... so what about increasing KVM_USER_MEM_SLOTS to 32 + 48 = 80 now (instead of 32 + 54 = 86) to ease the memslot situation at least a little bit 'till we figured out a really final solution like growable memslots? Thomas From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Huth Subject: Re: [Qemu-ppc] KVM memory slots limit on powerpc Date: Fri, 4 Sep 2015 16:45:33 +0200 Message-ID: <55E9AE8D.3030009@redhat.com> References: <55E965FC.8080909@redhat.com> <55E96B87.8020900@de.ibm.com> <55E96CB9.4070100@suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: David Gibson , Alex Williamson , Bharata B Rao To: Alexander Graf , Christian Borntraeger , kvm-ppc@vger.kernel.org, qemu-ppc@nongnu.org, kvm@vger.kernel.org Return-path: Received: from mx1.redhat.com ([209.132.183.28]:33511 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750766AbbIDOpi (ORCPT ); Fri, 4 Sep 2015 10:45:38 -0400 In-Reply-To: <55E96CB9.4070100@suse.de> Sender: kvm-owner@vger.kernel.org List-ID: On 04/09/15 12:04, Alexander Graf wrote: > > > On 04.09.15 11:59, Christian Borntraeger wrote: >> Am 04.09.2015 um 11:35 schrieb Thomas Huth: >>> >>> Hi all, >>> >>> now that we get memory hotplugging for the spapr machine on qemu-ppc, >>> too, it seems like we easily can hit the amount of KVM-internal memory >>> slots now ("#define KVM_USER_MEM_SLOTS 32" in >>> arch/powerpc/include/asm/kvm_host.h). For example, start >>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m >>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and >>> you'll see that it aborts way earlier already. >>> >>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509 >>> already (+3 internal slots = 512) ... maybe we should now increase the >>> amount of slots on powerpc, too? Since we don't use internal slots on >>> POWER, would 512 be a good value? Or would less be sufficient, too? >> >> When you are at it, the s390 value should also be increased I guess. > > That constant defines the array size for the memslot array in struct kvm > which in turn again gets allocated by kzalloc, so it's pinned kernel > memory that is physically contiguous. Doing big allocations can turn > into problems during runtime. FWIW, I've just checked sizeof(struct kvm) with the current ppc64 kernel build from master branch, and it is 34144 bytes. So on a system that is using PAGE_SIZE = 64kB, there should be plenty of space left before we're getting into trouble. And even assuming the worst case, that we're on a system which still uses PAGE_SIZE = 4kB, the last page of the 34144 bytes is only filled with 1376 bytes, leaving 2720 bytes free right now. sizeof(struct kvm_memory_slot) is 48 bytes right now on powerpc, and you need two additional bytes per entry for the id_to_index array in struct kvm_memslots, i.e. we need 50 additional bytes per entry on ppc. That means we could increase KVM_USER_MEM_SLOTS by 2720 / 50 = 54 entries without getting into further trouble. I think we should leave some more additional bytes left in that last 4k page of the struct kvm region, ... so what about increasing KVM_USER_MEM_SLOTS to 32 + 48 = 80 now (instead of 32 + 54 = 86) to ease the memslot situation at least a little bit 'till we figured out a really final solution like growable memslots? Thomas