KVM memory slots limit on powerpc

KVM memory slots limit on powerpc

[Thomas Huth]

 Hi all,

now that we get memory hotplugging for the spapr machine on qemu-ppc,
too, it seems like we easily can hit the amount of KVM-internal memory
slots now ("#define KVM_USER_MEM_SLOTS 32" in
arch/powerpc/include/asm/kvm_host.h). For example, start
qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
you'll see that it aborts way earlier already.

The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
already (+3 internal slots = 512) ... maybe we should now increase the
amount of slots on powerpc, too? Since we don't use internal slots on
POWER, would 512 be a good value? Or would less be sufficient, too?

 Thomas

Re: KVM memory slots limit on powerpc

[Christian Borntraeger]

Am 04.09.2015 um 11:35 schrieb Thomas Huth:
> 
>  Hi all,
> 
> now that we get memory hotplugging for the spapr machine on qemu-ppc,
> too, it seems like we easily can hit the amount of KVM-internal memory
> slots now ("#define KVM_USER_MEM_SLOTS 32" in
> arch/powerpc/include/asm/kvm_host.h). For example, start
> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
> you'll see that it aborts way earlier already.
> 
> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
> already (+3 internal slots = 512) ... maybe we should now increase the
> amount of slots on powerpc, too? Since we don't use internal slots on
> POWER, would 512 be a good value? Or would less be sufficient, too?

When you are at it, the s390 value should also be increased I guess.

Christian

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Alexander Graf]

On 04.09.15 11:59, Christian Borntraeger wrote:
> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>
>>  Hi all,
>>
>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>> too, it seems like we easily can hit the amount of KVM-internal memory
>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>> arch/powerpc/include/asm/kvm_host.h). For example, start
>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>> you'll see that it aborts way earlier already.
>>
>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>> already (+3 internal slots = 512) ... maybe we should now increase the
>> amount of slots on powerpc, too? Since we don't use internal slots on
>> POWER, would 512 be a good value? Or would less be sufficient, too?
> 
> When you are at it, the s390 value should also be increased I guess.

That constant defines the array size for the memslot array in struct kvm
which in turn again gets allocated by kzalloc, so it's pinned kernel
memory that is physically contiguous. Doing big allocations can turn
into problems during runtime.

So maybe there is another way? Can we extend the memslot array size
dynamically somehow? Allocate it separately? How much memory does the
memslot array use up with 512 entries?


Alex

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Christian Borntraeger]

Am 04.09.2015 um 12:04 schrieb Alexander Graf:
> 
> 
> On 04.09.15 11:59, Christian Borntraeger wrote:
>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>
>>>  Hi all,
>>>
>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>> you'll see that it aborts way earlier already.
>>>
>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>
>> When you are at it, the s390 value should also be increased I guess.
> 
> That constant defines the array size for the memslot array in struct kvm
> which in turn again gets allocated by kzalloc, so it's pinned kernel
> memory that is physically contiguous. Doing big allocations can turn
> into problems during runtime.
> 
> So maybe there is another way? Can we extend the memslot array size
> dynamically somehow? Allocate it separately? How much memory does the
> memslot array use up with 512 entries?

Maybe some rcu protected scheme that doubles the amount of memslots for
each overrun? Yes, that would be good and even reduce the footprint for
systems with only a small number of memslots.

Christian

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Thomas Huth]

On 04/09/15 12:07, Christian Borntraeger wrote:
> Am 04.09.2015 um 12:04 schrieb Alexander Graf:
>>
>>
>> On 04.09.15 11:59, Christian Borntraeger wrote:
>>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>>
>>>>  Hi all,
>>>>
>>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>>> you'll see that it aborts way earlier already.
>>>>
>>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>>
>>> When you are at it, the s390 value should also be increased I guess.
>>
>> That constant defines the array size for the memslot array in struct kvm
>> which in turn again gets allocated by kzalloc, so it's pinned kernel
>> memory that is physically contiguous. Doing big allocations can turn
>> into problems during runtime.
>>
>> So maybe there is another way? Can we extend the memslot array size
>> dynamically somehow? Allocate it separately? How much memory does the
>> memslot array use up with 512 entries?
> 
> Maybe some rcu protected scheme that doubles the amount of memslots for
> each overrun? Yes, that would be good and even reduce the footprint for
> systems with only a small number of memslots.

Seems like Alex Williamson already posted a patchset for growable
memslots a couple of years ago:

 http://www.spinics.net/lists/kvm/msg50491.html

But I didn't quite spot the result in that thread why it never has been
included upstream. Alex (W.), do you remember the outcome?

 Thomas

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Benjamin Herrenschmidt]

On Fri, 2015-09-04 at 12:28 +0200, Thomas Huth wrote:

> > Maybe some rcu protected scheme that doubles the amount of memslots
> > for
> > each overrun? Yes, that would be good and even reduce the footprint
> > for
> > systems with only a small number of memslots.
> 
> Seems like Alex Williamson already posted a patchset for growable
> memslots a couple of years ago:
> 
>  http://www.spinics.net/lists/kvm/msg50491.html
> 
> But I didn't quite spot the result in that thread why it never has 
> been
> included upstream. Alex (W.), do you remember the outcome?

Isn't the memslot array *already* protected by RCU anyway ?

Cheers,
Ben.

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Alex Williamson]

On Fri, 2015-09-04 at 12:28 +0200, Thomas Huth wrote:
> On 04/09/15 12:07, Christian Borntraeger wrote:
> > Am 04.09.2015 um 12:04 schrieb Alexander Graf:
> >>
> >>
> >> On 04.09.15 11:59, Christian Borntraeger wrote:
> >>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
> >>>>
> >>>>  Hi all,
> >>>>
> >>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
> >>>> too, it seems like we easily can hit the amount of KVM-internal memory
> >>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
> >>>> arch/powerpc/include/asm/kvm_host.h). For example, start
> >>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
> >>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
> >>>> you'll see that it aborts way earlier already.
> >>>>
> >>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
> >>>> already (+3 internal slots = 512) ... maybe we should now increase the
> >>>> amount of slots on powerpc, too? Since we don't use internal slots on
> >>>> POWER, would 512 be a good value? Or would less be sufficient, too?
> >>>
> >>> When you are at it, the s390 value should also be increased I guess.
> >>
> >> That constant defines the array size for the memslot array in struct kvm
> >> which in turn again gets allocated by kzalloc, so it's pinned kernel
> >> memory that is physically contiguous. Doing big allocations can turn
> >> into problems during runtime.
> >>
> >> So maybe there is another way? Can we extend the memslot array size
> >> dynamically somehow? Allocate it separately? How much memory does the
> >> memslot array use up with 512 entries?
> > 
> > Maybe some rcu protected scheme that doubles the amount of memslots for
> > each overrun? Yes, that would be good and even reduce the footprint for
> > systems with only a small number of memslots.
> 
> Seems like Alex Williamson already posted a patchset for growable
> memslots a couple of years ago:
> 
>  http://www.spinics.net/lists/kvm/msg50491.html
> 
> But I didn't quite spot the result in that thread why it never has been
> included upstream. Alex (W.), do you remember the outcome?

IIRC it was simply thought to be more complicated than necessary.  Once
we started caching memory slot misses, the O(N) search of a larger fixed
array didn't seem to bother anyone, so I abandoned the weight-balanced
tree, which I think still had some re-balancing issues.  Please run with
the code and make it work if it's useful now.  Thanks,

Alex

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Thomas Huth]

On 04/09/15 12:04, Alexander Graf wrote:
> 
> 
> On 04.09.15 11:59, Christian Borntraeger wrote:
>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>
>>>  Hi all,
>>>
>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>> you'll see that it aborts way earlier already.
>>>
>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>
>> When you are at it, the s390 value should also be increased I guess.
> 
> That constant defines the array size for the memslot array in struct kvm
> which in turn again gets allocated by kzalloc, so it's pinned kernel
> memory that is physically contiguous. Doing big allocations can turn
> into problems during runtime.

FWIW, I've just checked sizeof(struct kvm) with the current ppc64 kernel
build from master branch, and it is 34144 bytes.
So on a system that is using PAGE_SIZE = 64kB, there should be plenty of
space left before we're getting into trouble.

And even assuming the worst case, that we're on a system which still
uses PAGE_SIZE = 4kB, the last page of the 34144 bytes is only filled
with 1376 bytes, leaving 2720 bytes free right now.
sizeof(struct kvm_memory_slot) is 48 bytes right now on powerpc, and you
need two additional bytes per entry for the id_to_index array in
struct kvm_memslots, i.e. we need 50 additional bytes per entry on ppc.
That means we could increase KVM_USER_MEM_SLOTS by 2720 / 50 = 54
entries without getting into further trouble.

I think we should leave some more additional bytes left in that last
4k page of the struct kvm region, ... so what about increasing
KVM_USER_MEM_SLOTS to 32 + 48 = 80 now (instead of 32 + 54 = 86) to
ease the memslot situation at least a little bit 'till we figured out
a really final solution like growable memslots?

 Thomas

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Igor Mammedov]

On Fri, 4 Sep 2015 12:04:41 +0200
Alexander Graf <agraf@suse.de> wrote:

> 
> 
> On 04.09.15 11:59, Christian Borntraeger wrote:
> > Am 04.09.2015 um 11:35 schrieb Thomas Huth:
> >>
> >>  Hi all,
> >>
> >> now that we get memory hotplugging for the spapr machine on qemu-ppc,
> >> too, it seems like we easily can hit the amount of KVM-internal memory
> >> slots now ("#define KVM_USER_MEM_SLOTS 32" in
> >> arch/powerpc/include/asm/kvm_host.h). For example, start
> >> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
> >> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
> >> you'll see that it aborts way earlier already.
> >>
> >> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
> >> already (+3 internal slots = 512) ... maybe we should now increase the
> >> amount of slots on powerpc, too? Since we don't use internal slots on
> >> POWER, would 512 be a good value? Or would less be sufficient, too?
> > 
> > When you are at it, the s390 value should also be increased I guess.
> 
> That constant defines the array size for the memslot array in struct kvm
> which in turn again gets allocated by kzalloc, so it's pinned kernel
> memory that is physically contiguous. Doing big allocations can turn
> into problems during runtime.
> 
> So maybe there is another way? Can we extend the memslot array size
> dynamically somehow? Allocate it separately? How much memory does the
> memslot array use up with 512 entries?

KVM switched memslots allocation to kvm_kvzalloc(), so it would fallback to vmalloc
 commit 744961341d472db6272ed9b42319a90f5a2aa7c4
 kvm: avoid page allocation failure in kvm_set_memory_region()

> 
> Alex
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Thomas Huth]

On 07/09/15 16:31, Igor Mammedov wrote:
> On Fri, 4 Sep 2015 12:04:41 +0200
> Alexander Graf <agraf@suse.de> wrote:
> 
>>
>>
>> On 04.09.15 11:59, Christian Borntraeger wrote:
>>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>>
>>>>  Hi all,
>>>>
>>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>>> you'll see that it aborts way earlier already.
>>>>
>>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>>
>>> When you are at it, the s390 value should also be increased I guess.
>>
>> That constant defines the array size for the memslot array in struct kvm
>> which in turn again gets allocated by kzalloc, so it's pinned kernel
>> memory that is physically contiguous. Doing big allocations can turn
>> into problems during runtime.
>>
>> So maybe there is another way? Can we extend the memslot array size
>> dynamically somehow? Allocate it separately? How much memory does the
>> memslot array use up with 512 entries?
> 
> KVM switched memslots allocation to kvm_kvzalloc(), so it would fallback to vmalloc
>  commit 744961341d472db6272ed9b42319a90f5a2aa7c4
>  kvm: avoid page allocation failure in kvm_set_memory_region()

Good hint, thanks for pointing that out! ... so increasing the array
size should not cause too much trouble :-)

 Thomas

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Christian Borntraeger]

Am 08.09.2015 um 08:05 schrieb Thomas Huth:
> On 07/09/15 16:31, Igor Mammedov wrote:
>> On Fri, 4 Sep 2015 12:04:41 +0200
>> Alexander Graf <agraf@suse.de> wrote:
>>
>>>
>>>
>>> On 04.09.15 11:59, Christian Borntraeger wrote:
>>>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>>>
>>>>>  Hi all,
>>>>>
>>>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>>>> you'll see that it aborts way earlier already.
>>>>>
>>>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>>>
>>>> When you are at it, the s390 value should also be increased I guess.
>>>
>>> That constant defines the array size for the memslot array in struct kvm
>>> which in turn again gets allocated by kzalloc, so it's pinned kernel
>>> memory that is physically contiguous. Doing big allocations can turn
>>> into problems during runtime.
>>>
>>> So maybe there is another way? Can we extend the memslot array size
>>> dynamically somehow? Allocate it separately? How much memory does the
>>> memslot array use up with 512 entries?
>>
>> KVM switched memslots allocation to kvm_kvzalloc(), so it would fallback to vmalloc
>>  commit 744961341d472db6272ed9b42319a90f5a2aa7c4
>>  kvm: avoid page allocation failure in kvm_set_memory_region()
> 
> Good hint, thanks for pointing that out! ... so increasing the array
> size should not cause too much trouble :-)

Changing the allocation of the memslots to a growing structure seems like a
good idea nevertheless. Any chance to do this as well when you are at it?

Christian

Re: [Qemu-ppc] KVM memory slots limit on powerpc

[Thomas Huth]

On 08/09/15 09:11, Christian Borntraeger wrote:
> Am 08.09.2015 um 08:05 schrieb Thomas Huth:
>> On 07/09/15 16:31, Igor Mammedov wrote:
>>> On Fri, 4 Sep 2015 12:04:41 +0200
>>> Alexander Graf <agraf@suse.de> wrote:
>>>> On 04.09.15 11:59, Christian Borntraeger wrote:
>>>>> Am 04.09.2015 um 11:35 schrieb Thomas Huth:
>>>>>>
>>>>>> now that we get memory hotplugging for the spapr machine on qemu-ppc,
>>>>>> too, it seems like we easily can hit the amount of KVM-internal memory
>>>>>> slots now ("#define KVM_USER_MEM_SLOTS 32" in
>>>>>> arch/powerpc/include/asm/kvm_host.h). For example, start
>>>>>> qemu-system-ppc64 with a couple of "-device secondary-vga" and "-m
>>>>>> 4G,slots=32,maxmem=40G" and then try to hot-plug all 32 DIMMs ... and
>>>>>> you'll see that it aborts way earlier already.
>>>>>>
>>>>>> The x86 code already increased the amount of KVM_USER_MEM_SLOTS to 509
>>>>>> already (+3 internal slots = 512) ... maybe we should now increase the
>>>>>> amount of slots on powerpc, too? Since we don't use internal slots on
>>>>>> POWER, would 512 be a good value? Or would less be sufficient, too?
>>>>>
>>>>> When you are at it, the s390 value should also be increased I guess.
>>>>
>>>> That constant defines the array size for the memslot array in struct kvm
>>>> which in turn again gets allocated by kzalloc, so it's pinned kernel
>>>> memory that is physically contiguous. Doing big allocations can turn
>>>> into problems during runtime.
>>>>
>>>> So maybe there is another way? Can we extend the memslot array size
>>>> dynamically somehow? Allocate it separately? How much memory does the
>>>> memslot array use up with 512 entries?
>>>
>>> KVM switched memslots allocation to kvm_kvzalloc(), so it would fallback to vmalloc
>>>  commit 744961341d472db6272ed9b42319a90f5a2aa7c4
>>>  kvm: avoid page allocation failure in kvm_set_memory_region()
>>
>> Good hint, thanks for pointing that out! ... so increasing the array
>> size should not cause too much trouble :-)
> 
> Changing the allocation of the memslots to a growing structure seems like a
> good idea nevertheless. Any chance to do this as well when you are at it?

I'd like to finish some other stuff first, so if you want to have a try,
feel free to do so ... if not, I'll have a closer look at this later.

 Thomas