Re: [PATCH v3 12/16] memory-device, vhost: Support automatic decision on the number of memslots

[David Hildenbrand <david@redhat.com>]

On 17.09.23 12:46, Maciej S. Szmigiero wrote:
> On 8.09.2023 16:21, David Hildenbrand wrote:
>> We want to support memory devices that can automatically decide how many
>> memslots they will use. In the worst case, they have to use a single
>> memslot.
>>
>> The target use cases are virtio-mem and the hyper-v balloon.
>>
>> Let's calculate a reasonable limit such a memory device may use, and
>> instruct the device to make a decision based on that limit. Use a simple
>> heuristic that considers:
>> * A memslot soft-limit for all memory devices of 256; also, to not
>>     consume too many memslots -- which could harm performance.
>> * Actually still free and unreserved memslots
>> * The percentage of the remaining device memory region that memory device
>>     will occupy.
>>
>> Further, while we properly check before plugging a memory device whether
>> there still is are free memslots, we have other memslot consumers (such as
>> boot memory, PCI BARs) that don't perform any checks and might dynamically
>> consume memslots without any prior reservation. So we might succeed in
>> plugging a memory device, but once we dynamically map a PCI BAR we would
>> be in trouble. Doing accounting / reservation / checks for all such
>> users is problematic (e.g., sometimes we might temporarily split boot
>> memory into two memslots, triggered by the BIOS).
>>
>> We use the historic magic memslot number of 509 as orientation to when
>> supporting 256 memory devices -> memslots (leaving 253 for boot memory and
>> other devices) has been proven to work reliable. We'll fallback to
>> suggesting a single memslot if we don't have at least 509 total memslots.
>>
>> Plugging vhost devices with less than 509 memslots available while we
>> have memory devices plugged that consume multiple memslots due to
>> automatic decisions can be problematic. Most configurations might just fail
>> due to "limit < used + reserved", however, it can also happen that these
>> memory devices would suddenly consume memslots that would actually be
>> required by other memslot consumers (boot, PCI BARs) later. Note that this
>> has always been sketchy with vhost devices that support only a small number
>> of memslots; but we don't want to make it any worse.So let's keep it simple
>> and simply reject plugging such vhost devices in such a configuration.
>>
>> Eventually, all vhost devices that want to be fully compatible with such
>> memory devices should support a decent number of memslots (>= 509).
>>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>> ---
> 
> Reviewed-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>

Thanks!

> 
> I would be nice to ultimately allow raising the 509 memslot limit,
> considering that KVM had supported 32k memslots for more than two years
> now and had a much more scalable implementation since early 2022.

It's all tricky due to vhost (and hotplug of such devices) and the QEMU 
internal address translation (which isn't that scalable).

I was thinking about having a parameter to configure the number of 
memslots for memory devices, such that one could manually raise the 
"256" limit for memory devices.

But for now I kept it simple, because it all turned out to become way to 
complicated.

-- 
Cheers,

David / dhildenb