Re: 9 TiB vm memory creation

David Hildenbrand
2022-02-14T16:32:13Z

>>>
>>> With KVM enabled it bails out with:
>>> qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=1, start=0x100000000, size=0x8ff40000000: Invalid argument
>>>
>>> all of that on a host with 32G of RAM/no swap.
>>>
>>>
>>
>> #define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
>>
>> ~8 TiB (7,999999)
>
> so essentially that's the our max for initial RAM
> (ignoring initial RAM slots before 4Gb)
>
> Are you aware of any attempts to make it larger?

Not really, I think for now only s390x had applicable machines where
you'd have that much memory on a single NUMA node.

>
> But can we use extra pc-dimm devices for additional memory (with 8TiB limit)
> as that will use another memslot?

I remember that was the workaround for now for some extremely large VMs
where you'd want a single NUMA node or a lot of memory for a single NUMA
node.

>
>
>>
>> In QEMU, we have
>>
>> static hwaddr kvm_max_slot_size = ~0;
>>
>> And only s390x sets
>>
>> kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
>>
>> with
>>
>> #define KVM_SLOT_MAX_BYTES (4UL * TiB)
> in QEMU default value is:
> static hwaddr kvm_max_slot_size = ~0
> it is kernel side that's failing

... and kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES) works around the
kernel limitation for s390x in user space.

I feel like the right thing would be to look into increasing the limit
in the kernel, and bail out if the kernel doesn't support it. Would
require a new kernel for starting gigantic VMs with a single large
memory backend, but then, it's a new use case.

--
Thanks,

David / dhildenb