RE: hv_balloon: Only works in ubuntu

[Vitaly Kuznetsov <vkuznets@redhat.com>]

"Michael Kelley (LINUX)" <mikelley@microsoft.com> writes:

> From: Florian Müller <max06.net@outlook.com> Sent: Saturday, June 11, 2022 12:45 PM
>> 
>> > > > >
>> > > > > Issues showed up when I set up a Kali Linux Guest. I missed the
>> > > > > memory configuration before booting up the instance, so it started
>> > > > > with 1GB of memory, and ballooning active between 512MB and
>> > > > > several TB of memory. Hyper-V started to allocate more and more
>> > > > > memory to this guest since the reported memory requirements also
>> > > > > increased. The guest kernel didn't see any of that allocated memory, as 
>> > > > > far as I can tell.
>> > > >
>> > > > Hot-adding new memory is done partially by the hv_balloon driver,
>> > > > but it also requires user space action.  The user space action is provided
>> > > > by udev rules.
>> > > > In your Ubuntu Server 20.04 guest, there's a file
>> > > > /lib/udev/rules.d/40-vm- hotadd.rules.
>> > > > This file contains udev rules for hot-adding memory and CPUs.  You
>> > > > should be able to copy this file with the udev rules onto your Kali
>> > > > system, and then the memory hot-add should work correctly.
>> > > >
>> > > > I'm not sure why Kali doesn't already have such udev rules.  You
>> > > > might grep through all the files in /lib/udev/rules.d and if there
>> > > > are any rules for SUBSYSTEM==memory.
>> > > > Sometimes there are rules present, but commented out.
>> > > >
>> > > Thanks, I'll check these ones out!
>> > >
>> > > In the meantime, I was able to get it working, by compiling a kernel
>> > > with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
>> > > Which was previously unset. It's enabled in ubuntu and it seems to
>> > > make hv_balloon work properly.
>> > > This seems to be the same case with Debian.
>> > >
>> >
>> > Yes, that looks like a good solution.  I didn't remember that there is a kernel
>> > config option to automatically do the onlining.  With this kernel option
>> > enabled, using a udev rule obviously isn't needed.  The kernel option was
>> > added in Linux kernel version 4.7, which might be after the last time I looked
>> > at Hyper-V memory hot-add in detail.
>> >
>> > Michael
>> >
>> 
>> Awesome!
>> 
>> Last question: Since not having the kernel option by default and also not having the
>> udev rule in some distributions causes the Hyper-V host to eat up all the memory up to
>> the defined limit (and to die eventually), should this be considered as a bug? And if the
>> answer is no, how can I (or anyone) forward the requirement to the publishers to be
>> solved at the source?
>> 
>> Thank you!
>> 
>
> It's unclear whether this should be treated as a bug.  We certainly want the
> "right" thing to happen as seamlessly as possible, but there are tradeoffs.  Back
> when Vitaly Kuznetsov added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE,
> I can see there was some debate about whether this option should be enabled
> by default. There was reluctance to do so because of potential backwards
> compatibility problems with other environments.  When hot-adding real
> physical memory to a bare-metal server, apparently you don't want to 
> automatically online the added memory.

On x86/ARM you most likely do (as why plugging in memory in the first
place?) but there are not many bare metal systems which allow that.

Note, there were some developments after I've added
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, namely

commit 5f47adf762b78cae97de58d9ff01d2d44db09467
Author: David Hildenbrand <david@redhat.com>
Date:   Mon Apr 6 20:07:44 2020 -0700

    mm/memory_hotplug: allow to specify a default online_type

>
> The Ubuntu image you were testing is specifically an image configured for use
> in an Azure VM, so it makes sense to have memory automatically onlined by
> the kernel.  So I looked at a generic Ubuntu 18.04 system, and it also has
> this kernel option set by default.  But as another data point, RHEL/CentOS 8.4
> does *not* have the kernel option set.  So each distro evidently makes their
> own decision about this.  

Note: Fedora kernels come with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y as
well.

> But both generic Ubuntu 18.04 and RHEL/CentOS 8.4
> have the udev rules.  The RHEL/CentOS 8.4 udev rule is more sophisticated
> in that it does not online the added memory when running on System/390
> and PowerPC.

I remember that with s390 we certainly don't want all memory to go
online automatically because but I don't remember the reason :-)

>
> I could envision changing the Linux kernel config rules to automatically set
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE whenever 
> CONFIG_HYPERV_BALLOON is selected. 

That could work for custom kernels but not for those shipped with
distributions as these will always have CONFIG_HYPERV_BALLOON if
Hyper-V is one of the supported targets (and it likely is).

> Alternatively, the Kali Linux folks could consider adding the
> appropriate udev rule.

Or just enable CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE. We've enabled
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE in Fedora in 2016 and haven't heard
many (any?) complaints (besides ppc64 where it was disabled) since.

Cc: David who mostly picked up the auto onlining work since.

-- 
Vitaly