Re: [PATCH][RESEND v3 1/3] hapvdimm: add a virtual DIMM device for memory hot-add protocols

[David Hildenbrand <david@redhat.com>]

> 
> The idea would seem reasonable, but: (there's always some "but")
> 1) Once we implement NUMA support we'd probably need multiple
> TYPE_MEMORY_DEVICEs anyway, since it seems one memdev can sit on only
> one NUMA node,
> 

Not necessarily. You could extend the hv-balloon device to have one 
memslot for each NUMA node. Of course, once again, you have to plan 
ahead how to distribute memory across NUMA nodes (same with virtio-mem).

Having that said, last time I checked, HV dynamic memory was 
force-disabled when enabling vNUMA under HV. Simply because balloon 
inflation is not NUMA aware.

> With virtio-mem one can simply have per-node virtio-mem devices.
> 
> 2) I'm not sure what's the overhead of having, let's say, 1 TiB backing
> memory device mostly marked madvise(MADV_DONTNEED).
> Like, how much memory + swap this setup would actually consume - that's
> something I would need to measure.

There are some WIP items to improve that (QEMU metadata (e.g., bitmaps), 
KVM metadata (e.g., per-memslot), Linux metadata (e.g., page tables).
Memory overcommit handling also has to be tackled.

So it would be a "shared" problem with virtio-mem and will be sorted out 
eventually :)

> 
> 3) In a public cloud environment malicious guests are a possibility.
> Currently (without things like resizable memslots) the best idea I tried
> was to place the whole QEMU process into a memory-limited cgroup
> (limited to the guest target size).

Yes. Protection of unplugged memory is on my TODO list for virtio-mem as 
well, to avoid having to rely on cgroups.

> 
> There are still some issues with it: one needs to reserve swap space up
> to the guest maximum size so the QEMU process doesn't get OOM-killed if
> guest touches that memory and the cgroup memory controller for some
> reason seems to start swapping even before reaching its limit (that's
> still under investigation why).

Yes, putting a memory cap on Linux was always tricky.

> 
>> Reboot? Logically unplug all memory and as the guest boots up, re-add the memory after the guest booted up.
>>
>> The only thing we can't do is the following: when going below 4G, we cannot resize boot memory.
>>
>>
>> But I recall that that's *exactly* how the HV version I played with ~2 years ago worked: always start up with some initial memory ("startup memory"). After the VM is up for some seconds, we either add more memory (requested > startup) or request the VM to inflate memory (requested < startup).
> 
> Hyper-V actually "cleans up" the guest memory map on reboot - if the
> guest was effectively resized up then on reboot the guest boot memory is
> resized up to match that last size.
> Similarly, if the guest was ballooned out - that amount of memory is
> removed from the boot memory on reboot.

Yes, it cleans up, but as I said last time I checked there was this 
concept of startup vs. minimum vs. maximum, at least for dynamic memory:

https://www.fastvue.co/tmgreporter/blog/understanding-hyper-v-dynamic-memory-dynamic-ram/

Startup RAM would be whatever you specify for "-m xG". If you go below 
min, you remove memory via deflation once the guest is up.

> 
> So it's not exactly doing a hot-add after the guest boots.

I recall BUG reports in Linux, that we got hv-balloon hot-add requests 
~1 minute after Linux booted up, because of the above reason of startup 
memory [in these BUG reports, memory onlining was disabled and the VM 
would run out of memory because we hotplugged too much memory]. That's 
why I remember that this approach once was done.

Maybe there are multiple implementations noways. At least in QEMU you 
could chose whatever makes most sense for QEMU.


> This approach (of resizing the boot memory) also avoids problems if the
> guest loses hot-add / ballooning capability after a reboot - for example,
> rebooting into a Linux guest from Windows with hv-balloon.

TBH, I wouldn't be too concerned about that scenario ("hotplugged memory 
to a guest, guest reboots into a weird OS, weird OS isn't able to use 
hotplugged memory). For virtio-mem, the important part was that you 
always "know" how much memory the VM is aware about. If you always start 
with "Startup memory" and hotadd later (only if you detected guest 
support after a bootup), you can handle that scenario.

> 
> But unfortunately such resizing the guest boot memory seems not trivial
> to implement in QEMU.

Yes, avoiding changing memory layout to keep memory migration feasible 
was another thing I considered when designing virtio-mem.


Anyhow, I'm just throwing out ideas here on how to eventually handle it 
differently.

-- 
Thanks,

David / dhildenb