[RFC 0/2] Drivers: hv: balloon: Temporary fixes for ARM64

[RFC 0/2] Drivers: hv: balloon: Temporary fixes for ARM64

[Boqun Feng]

Since Hyper-V always uses 4k pages, hv_balloon has some difficulties
working on ARM64 with larger pages[1]. Besides the memory hot add
messages of Hyper-V doesn't have the information of NUMA node id of the
added memory range, and ARM64 currently doesn't provide the conversion
from a physical address to a node id, as a result the hv_balloon driver
couldn't handle hot add properly when there are more than one NUMA node.

Among these issues, post_status() is easy to fix, while the unballoon
issue and the hot-add issue requires more discussion. To make the
hv_balloon driver work at the best effort, this patchset fixes the
post_status() and temporarily disable the balloon and hot-add
accordingly.

Looking forwards to comments and suggestions.

Regards,
Boqun

[1]: https://lore.kernel.org/lkml/20220105165028.1343706-1-vkuznets@redhat.com/

Boqun Feng (2):
  Drivers: hv: balloon: Support status report for larger page sizes
  Drivers: hv: balloon: Disable balloon and hot-add accordingly

 drivers/hv/hv_balloon.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

-- 
2.35.1

[RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes

[Boqun Feng]

DM_STATUS_REPORT expects the numbers of pages in the unit of 4k pages
(HV_HYP_PAGE) instead of guest pages, so to make it work when guest page
sizes are larger than 4k, convert the numbers of guest pages into the
numbers of HV_HYP_PAGEs.

Note that the numbers of guest pages are still used for tracing because
tracing is internal to the guest kernel.

Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
 drivers/hv/hv_balloon.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index f2d05bff4245..062156b88a87 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -17,6 +17,7 @@
 #include <linux/slab.h>
 #include <linux/kthread.h>
 #include <linux/completion.h>
+#include <linux/count_zeros.h>
 #include <linux/memory_hotplug.h>
 #include <linux/memory.h>
 #include <linux/notifier.h>
@@ -1130,6 +1131,7 @@ static void post_status(struct hv_dynmem_device *dm)
 	struct dm_status status;
 	unsigned long now = jiffies;
 	unsigned long last_post = last_post_time;
+	unsigned long num_pages_avail, num_pages_committed;
 
 	if (pressure_report_delay > 0) {
 		--pressure_report_delay;
@@ -1154,16 +1156,21 @@ static void post_status(struct hv_dynmem_device *dm)
 	 * num_pages_onlined) as committed to the host, otherwise it can try
 	 * asking us to balloon them out.
 	 */
-	status.num_avail = si_mem_available();
-	status.num_committed = vm_memory_committed() +
+	num_pages_avail = si_mem_available();
+	num_pages_committed = vm_memory_committed() +
 		dm->num_pages_ballooned +
 		(dm->num_pages_added > dm->num_pages_onlined ?
 		 dm->num_pages_added - dm->num_pages_onlined : 0) +
 		compute_balloon_floor();
 
-	trace_balloon_status(status.num_avail, status.num_committed,
+	trace_balloon_status(num_pages_avail, num_pages_committed,
 			     vm_memory_committed(), dm->num_pages_ballooned,
 			     dm->num_pages_added, dm->num_pages_onlined);
+
+	/* Convert numbers of pages into numbers of HV_HYP_PAGEs. */
+	status.num_avail = num_pages_avail * NR_HV_HYP_PAGES_IN_PAGE;
+	status.num_committed = num_pages_committed * NR_HV_HYP_PAGES_IN_PAGE;
+
 	/*
 	 * If our transaction ID is no longer current, just don't
 	 * send the status. This can happen if we were interrupted
-- 
2.35.1

RE: [RFC 1/2] Drivers: hv: balloon: Support status report for larger page sizes

[Michael Kelley (LINUX)]

From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 5:16 AM
> 
> DM_STATUS_REPORT expects the numbers of pages in the unit of 4k pages
> (HV_HYP_PAGE) instead of guest pages, so to make it work when guest page
> sizes are larger than 4k, convert the numbers of guest pages into the
> numbers of HV_HYP_PAGEs.
> 
> Note that the numbers of guest pages are still used for tracing because
> tracing is internal to the guest kernel.
> 
> Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
>  drivers/hv/hv_balloon.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index f2d05bff4245..062156b88a87 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -17,6 +17,7 @@
>  #include <linux/slab.h>
>  #include <linux/kthread.h>
>  #include <linux/completion.h>
> +#include <linux/count_zeros.h>
>  #include <linux/memory_hotplug.h>
>  #include <linux/memory.h>
>  #include <linux/notifier.h>
> @@ -1130,6 +1131,7 @@ static void post_status(struct hv_dynmem_device *dm)
>  	struct dm_status status;
>  	unsigned long now = jiffies;
>  	unsigned long last_post = last_post_time;
> +	unsigned long num_pages_avail, num_pages_committed;
> 
>  	if (pressure_report_delay > 0) {
>  		--pressure_report_delay;
> @@ -1154,16 +1156,21 @@ static void post_status(struct hv_dynmem_device *dm)
>  	 * num_pages_onlined) as committed to the host, otherwise it can try
>  	 * asking us to balloon them out.
>  	 */
> -	status.num_avail = si_mem_available();
> -	status.num_committed = vm_memory_committed() +
> +	num_pages_avail = si_mem_available();
> +	num_pages_committed = vm_memory_committed() +
>  		dm->num_pages_ballooned +
>  		(dm->num_pages_added > dm->num_pages_onlined ?
>  		 dm->num_pages_added - dm->num_pages_onlined : 0) +
>  		compute_balloon_floor();
> 
> -	trace_balloon_status(status.num_avail, status.num_committed,
> +	trace_balloon_status(num_pages_avail, num_pages_committed,
>  			     vm_memory_committed(), dm->num_pages_ballooned,
>  			     dm->num_pages_added, dm->num_pages_onlined);
> +
> +	/* Convert numbers of pages into numbers of HV_HYP_PAGEs. */
> +	status.num_avail = num_pages_avail * NR_HV_HYP_PAGES_IN_PAGE;
> +	status.num_committed = num_pages_committed * NR_HV_HYP_PAGES_IN_PAGE;
> +
>  	/*
>  	 * If our transaction ID is no longer current, just don't
>  	 * send the status. This can happen if we were interrupted
> --
> 2.35.1

Reviewed-by: Michael Kelley <mikelley@microsoft.com>

[RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Boqun Feng]

Currently there are known potential issues for balloon and hot-add on
ARM64:

*	Unballoon requests from Hyper-V should only unballoon ranges
	that are guest page size aligned, otherwise guests cannot handle
	because it's impossible to partially free a page.

*	Memory hot-add requests from Hyper-V should provide the NUMA
	node id of the added ranges or ARM64 should have a functional
	memory_add_physaddr_to_nid(), otherwise the node id is missing
	for add_memory().

These issues require discussions on design and implementation. In the
meanwhile, post_status() is working and essiential to guest monitoring.
Therefore instead of the entire hv_balloon driver, the balloon and
hot-add are disabled accordingly for now. Once the issues are fixed,
they can be re-enable in these cases.

Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
 drivers/hv/hv_balloon.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index 062156b88a87..35dcda20be85 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
 	 * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
 	 * currently still requires the bits to be set, so we have to add code
 	 * to fail the host's hot-add and balloon up/down requests, if any.
+	 *
+	 * We disable balloon if the page size is larger than 4k, since
+	 * currently it's unclear to us whether an unballoon request can make
+	 * sure all page ranges are guest page size aligned.
+	 *
+	 * We also disable hot add on ARM64, because we currently rely on
+	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
+	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
+	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
+	 * add_memory().
 	 */
-	cap_msg.caps.cap_bits.balloon = 1;
-	cap_msg.caps.cap_bits.hot_add = 1;
+	cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
+	cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
 
 	/*
 	 * Specify our alignment requirements as it relates
-- 
2.35.1

RE: [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Michael Kelley (LINUX)]

From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 5:16 AM
> 
> Currently there are known potential issues for balloon and hot-add on
> ARM64:
> 
> *	Unballoon requests from Hyper-V should only unballoon ranges
> 	that are guest page size aligned, otherwise guests cannot handle
> 	because it's impossible to partially free a page.
> 
> *	Memory hot-add requests from Hyper-V should provide the NUMA
> 	node id of the added ranges or ARM64 should have a functional
> 	memory_add_physaddr_to_nid(), otherwise the node id is missing
> 	for add_memory().
> 
> These issues require discussions on design and implementation. In the
> meanwhile, post_status() is working and essiential to guest monitoring.
> Therefore instead of the entire hv_balloon driver, the balloon and
> hot-add are disabled accordingly for now. Once the issues are fixed,
> they can be re-enable in these cases.
> 
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
>  drivers/hv/hv_balloon.c | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index 062156b88a87..35dcda20be85 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
>  	 * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
>  	 * currently still requires the bits to be set, so we have to add code
>  	 * to fail the host's hot-add and balloon up/down requests, if any.
> +	 *
> +	 * We disable balloon if the page size is larger than 4k, since
> +	 * currently it's unclear to us whether an unballoon request can make
> +	 * sure all page ranges are guest page size aligned.
> +	 *
> +	 * We also disable hot add on ARM64, because we currently rely on
> +	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> +	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> +	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> +	 * add_memory().
>  	 */
> -	cap_msg.caps.cap_bits.balloon = 1;
> -	cap_msg.caps.cap_bits.hot_add = 1;
> +	cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);

Any reasons not to use HV_HYP_PAGE_SIZE vs. open coding "4096"?  So

	cap_msg.caps.cap_bits.balloon = (PAGE_SIZE == HV_HYP_PAGE_SIZE);

> +	cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);

I think we should output a message so that there's no mystery as to 
whether ballooning and/or hot_add are disabled, and why.  Each setting
should have its own message.   Maybe something like:

	if (!cap_msg.caps.cap_bits.balloon)
		pr_info("Ballooning disabled because page size is not 4096 bytes\n");

	if (!cap_msg.cap_bits.hot_add)
		pr_info("Memory hot add disabled on ARM64\n");

> 
>  	/*
>  	 * Specify our alignment requirements as it relates
> --
> 2.35.1

Re: [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Boqun Feng]

On Wed, Feb 23, 2022 at 04:55:25PM +0000, Michael Kelley (LINUX) wrote:
> From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 5:16 AM
> > 
> > Currently there are known potential issues for balloon and hot-add on
> > ARM64:
> > 
> > *	Unballoon requests from Hyper-V should only unballoon ranges
> > 	that are guest page size aligned, otherwise guests cannot handle
> > 	because it's impossible to partially free a page.
> > 
> > *	Memory hot-add requests from Hyper-V should provide the NUMA
> > 	node id of the added ranges or ARM64 should have a functional
> > 	memory_add_physaddr_to_nid(), otherwise the node id is missing
> > 	for add_memory().
> > 
> > These issues require discussions on design and implementation. In the
> > meanwhile, post_status() is working and essiential to guest monitoring.
> > Therefore instead of the entire hv_balloon driver, the balloon and
> > hot-add are disabled accordingly for now. Once the issues are fixed,
> > they can be re-enable in these cases.
> > 
> > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > ---
> >  drivers/hv/hv_balloon.c | 14 ++++++++++++--
> >  1 file changed, 12 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> > index 062156b88a87..35dcda20be85 100644
> > --- a/drivers/hv/hv_balloon.c
> > +++ b/drivers/hv/hv_balloon.c
> > @@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
> >  	 * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
> >  	 * currently still requires the bits to be set, so we have to add code
> >  	 * to fail the host's hot-add and balloon up/down requests, if any.
> > +	 *
> > +	 * We disable balloon if the page size is larger than 4k, since
> > +	 * currently it's unclear to us whether an unballoon request can make
> > +	 * sure all page ranges are guest page size aligned.
> > +	 *
> > +	 * We also disable hot add on ARM64, because we currently rely on
> > +	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> > +	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> > +	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> > +	 * add_memory().
> >  	 */
> > -	cap_msg.caps.cap_bits.balloon = 1;
> > -	cap_msg.caps.cap_bits.hot_add = 1;
> > +	cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
> 
> Any reasons not to use HV_HYP_PAGE_SIZE vs. open coding "4096"?  So
> 
> 	cap_msg.caps.cap_bits.balloon = (PAGE_SIZE == HV_HYP_PAGE_SIZE);
> 

You're right. I will change that to it in the next version.

> > +	cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
> 
> I think we should output a message so that there's no mystery as to 
> whether ballooning and/or hot_add are disabled, and why.  Each setting
> should have its own message.   Maybe something like:
> 
> 	if (!cap_msg.caps.cap_bits.balloon)
> 		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> 
> 	if (!cap_msg.cap_bits.hot_add)
> 		pr_info("Memory hot add disabled on ARM64\n");
> 

I agree with your suggestion, however, while I'm at it, I think it's
better that we have functions that check and print, and .balloon and
.hot_add can rely on the return value, for example:

static int balloon_enabled(void)
{
	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
		return 0;
	}

	return 1;
}

// in balloon_vsp_connect()

	cap_msg.caps.cap_bits.balloon = balloon_enabled();

In this way, we keep the checking and reason printing in the same
function and it's easier to maintain the consistency.

Thoughts?

Regards,
Boqun

> > 
> >  	/*
> >  	 * Specify our alignment requirements as it relates
> > --
> > 2.35.1
> 

RE: [RFC 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Michael Kelley (LINUX)]

From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022 6:44 PM
> 
> On Wed, Feb 23, 2022 at 04:55:25PM +0000, Michael Kelley (LINUX) wrote:
> > From: Boqun Feng <boqun.feng@gmail.com> Sent: Wednesday, February 23, 2022
> 5:16 AM
> > >
> > > Currently there are known potential issues for balloon and hot-add on
> > > ARM64:
> > >
> > > *	Unballoon requests from Hyper-V should only unballoon ranges
> > > 	that are guest page size aligned, otherwise guests cannot handle
> > > 	because it's impossible to partially free a page.
> > >
> > > *	Memory hot-add requests from Hyper-V should provide the NUMA
> > > 	node id of the added ranges or ARM64 should have a functional
> > > 	memory_add_physaddr_to_nid(), otherwise the node id is missing
> > > 	for add_memory().
> > >
> > > These issues require discussions on design and implementation. In the
> > > meanwhile, post_status() is working and essiential to guest monitoring.
> > > Therefore instead of the entire hv_balloon driver, the balloon and
> > > hot-add are disabled accordingly for now. Once the issues are fixed,
> > > they can be re-enable in these cases.
> > >
> > > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > > ---
> > >  drivers/hv/hv_balloon.c | 14 ++++++++++++--
> > >  1 file changed, 12 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> > > index 062156b88a87..35dcda20be85 100644
> > > --- a/drivers/hv/hv_balloon.c
> > > +++ b/drivers/hv/hv_balloon.c
> > > @@ -1730,9 +1730,19 @@ static int balloon_connect_vsp(struct hv_device *dev)
> > >  	 * When hibernation (i.e. virtual ACPI S4 state) is enabled, the host
> > >  	 * currently still requires the bits to be set, so we have to add code
> > >  	 * to fail the host's hot-add and balloon up/down requests, if any.
> > > +	 *
> > > +	 * We disable balloon if the page size is larger than 4k, since
> > > +	 * currently it's unclear to us whether an unballoon request can make
> > > +	 * sure all page ranges are guest page size aligned.
> > > +	 *
> > > +	 * We also disable hot add on ARM64, because we currently rely on
> > > +	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> > > +	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> > > +	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information
> for
> > > +	 * add_memory().
> > >  	 */
> > > -	cap_msg.caps.cap_bits.balloon = 1;
> > > -	cap_msg.caps.cap_bits.hot_add = 1;
> > > +	cap_msg.caps.cap_bits.balloon = !(PAGE_SIZE > 4096UL);
> >
> > Any reasons not to use HV_HYP_PAGE_SIZE vs. open coding "4096"?  So
> >
> > 	cap_msg.caps.cap_bits.balloon = (PAGE_SIZE == HV_HYP_PAGE_SIZE);
> >
> 
> You're right. I will change that to it in the next version.
> 
> > > +	cap_msg.caps.cap_bits.hot_add = !IS_ENABLED(CONFIG_ARM64);
> >
> > I think we should output a message so that there's no mystery as to
> > whether ballooning and/or hot_add are disabled, and why.  Each setting
> > should have its own message.   Maybe something like:
> >
> > 	if (!cap_msg.caps.cap_bits.balloon)
> > 		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> >
> > 	if (!cap_msg.cap_bits.hot_add)
> > 		pr_info("Memory hot add disabled on ARM64\n");
> >
> 
> I agree with your suggestion, however, while I'm at it, I think it's
> better that we have functions that check and print, and .balloon and
> .hot_add can rely on the return value, for example:
> 
> static int balloon_enabled(void)
> {
> 	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> 		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> 		return 0;
> 	}
> 
> 	return 1;
> }
> 
> // in balloon_vsp_connect()
> 
> 	cap_msg.caps.cap_bits.balloon = balloon_enabled();
> 
> In this way, we keep the checking and reason printing in the same
> function and it's easier to maintain the consistency.
> 
> Thoughts?

Yes, that approach looks good to me.

Michael

[RFC v1.1] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Boqun Feng]

Currently there are known potential issues for balloon and hot-add on
ARM64:

*	Unballoon requests from Hyper-V should only unballoon ranges
	that are guest page size aligned, otherwise guests cannot handle
	because it's impossible to partially free a page.

*	Memory hot-add requests from Hyper-V should provide the NUMA
	node id of the added ranges or ARM64 should have a functional
	memory_add_physaddr_to_nid(), otherwise the node id is missing
	for add_memory().

These issues require discussions on design and implementation. In the
meanwhile, post_status() is working and essiential to guest monitoring.
Therefore instead of the entire hv_balloon driver, the balloon and
hot-add are disabled accordingly for now. Once the issues are fixed,
they can be re-enable in these cases.

Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
v1 --> v1.1:

*	Use HV_HYP_PAGE_SIZE instead of hard coding 4096 as suggested by
	Michael.

*	Explicitly print out the disable message if a function is
	disabled as suggested by Michael.

 drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index 062156b88a87..eee7402cfc02 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
 	}
 }
 
+static int ballooning_enabled(void)
+{
+	/*
+	 * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
+	 * since currently it's unclear to us whether an unballoon request can
+	 * make sure all page ranges are guest page size aligned.
+	 */
+	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
+		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
+		return 0;
+	}
+
+	return 1;
+}
+
+static int hot_add_enabled(void)
+{
+	/*
+	 * Disable hot add on ARM64, because we currently rely on
+	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
+	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
+	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
+	 * add_memory().
+	 */
+	if (IS_ENABLED(CONFIG_ARM64)) {
+		pr_info("Memory hot add disabled on ARM64\n");
+		return 0;
+	}
+
+	return 1;
+}
+
 static int balloon_connect_vsp(struct hv_device *dev)
 {
 	struct dm_version_request version_req;
@@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
 	 * currently still requires the bits to be set, so we have to add code
 	 * to fail the host's hot-add and balloon up/down requests, if any.
 	 */
-	cap_msg.caps.cap_bits.balloon = 1;
-	cap_msg.caps.cap_bits.hot_add = 1;
+	cap_msg.caps.cap_bits.balloon = ballooning_enabled();
+	cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
 
 	/*
 	 * Specify our alignment requirements as it relates
-- 
2.33.0

RE: [RFC v1.1] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Michael Kelley (LINUX)]

From: Boqun Feng <boqun.feng@gmail.com> Sent: Thursday, February 24, 2022 6:17 PM
> 
> Currently there are known potential issues for balloon and hot-add on
> ARM64:
> 
> *	Unballoon requests from Hyper-V should only unballoon ranges
> 	that are guest page size aligned, otherwise guests cannot handle
> 	because it's impossible to partially free a page.

The above problem occurs only when the guest page size is > 4 Kbytes.

> 
> *	Memory hot-add requests from Hyper-V should provide the NUMA
> 	node id of the added ranges or ARM64 should have a functional
> 	memory_add_physaddr_to_nid(), otherwise the node id is missing
> 	for add_memory().
> 
> These issues require discussions on design and implementation. In the
> meanwhile, post_status() is working and essiential to guest monitoring.

s/essiential/essential/

> Therefore instead of the entire hv_balloon driver, the balloon and
> hot-add are disabled accordingly for now. Once the issues are fixed,
> they can be re-enable in these cases.

Missing the word "disabling" in the first line?  Also the balloon
function is disabled only if the page size is > 4 Kbytes.

> 
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
> v1 --> v1.1:
> 
> *	Use HV_HYP_PAGE_SIZE instead of hard coding 4096 as suggested by
> 	Michael.
> 
> *	Explicitly print out the disable message if a function is
> 	disabled as suggested by Michael.
> 
>  drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
>  1 file changed, 34 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index 062156b88a87..eee7402cfc02 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
>  	}
>  }
> 
> +static int ballooning_enabled(void)
> +{
> +	/*
> +	 * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
> +	 * since currently it's unclear to us whether an unballoon request can
> +	 * make sure all page ranges are guest page size aligned.

My interpretation of the conversations with Hyper-V is that that they clearly
don't guarantee page ranges are guest page aligned.

> +	 */
> +	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> +		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> +		return 0;
> +	}
> +
> +	return 1;
> +}
> +
> +static int hot_add_enabled(void)
> +{
> +	/*
> +	 * Disable hot add on ARM64, because we currently rely on
> +	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> +	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> +	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> +	 * add_memory().
> +	 */
> +	if (IS_ENABLED(CONFIG_ARM64)) {
> +		pr_info("Memory hot add disabled on ARM64\n");
> +		return 0;
> +	}
> +
> +	return 1;
> +}
> +
>  static int balloon_connect_vsp(struct hv_device *dev)
>  {
>  	struct dm_version_request version_req;
> @@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
>  	 * currently still requires the bits to be set, so we have to add code
>  	 * to fail the host's hot-add and balloon up/down requests, if any.
>  	 */
> -	cap_msg.caps.cap_bits.balloon = 1;
> -	cap_msg.caps.cap_bits.hot_add = 1;
> +	cap_msg.caps.cap_bits.balloon = ballooning_enabled();
> +	cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
> 
>  	/*
>  	 * Specify our alignment requirements as it relates
> --
> 2.33.0

The code looks good to me.

Michael

Re: [RFC v1.1] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Boqun Feng]

On Fri, Feb 25, 2022 at 05:06:45PM +0000, Michael Kelley (LINUX) wrote:
> From: Boqun Feng <boqun.feng@gmail.com> Sent: Thursday, February 24, 2022 6:17 PM
> > 
> > Currently there are known potential issues for balloon and hot-add on
> > ARM64:
> > 
> > *	Unballoon requests from Hyper-V should only unballoon ranges
> > 	that are guest page size aligned, otherwise guests cannot handle
> > 	because it's impossible to partially free a page.
> 
> The above problem occurs only when the guest page size is > 4 Kbytes.
> 

Ok, I wil call it out in next version.

> > 
> > *	Memory hot-add requests from Hyper-V should provide the NUMA
> > 	node id of the added ranges or ARM64 should have a functional
> > 	memory_add_physaddr_to_nid(), otherwise the node id is missing
> > 	for add_memory().
> > 
> > These issues require discussions on design and implementation. In the
> > meanwhile, post_status() is working and essiential to guest monitoring.
> 
> s/essiential/essential/
> 
> > Therefore instead of the entire hv_balloon driver, the balloon and
> > hot-add are disabled accordingly for now. Once the issues are fixed,
> > they can be re-enable in these cases.
> 
> Missing the word "disabling" in the first line?  Also the balloon

The phrasing that I was trying to use here is "Instead of A, B and C are
disabled" or "B and C are disabled instead of A". Looks like I'm
inventing my own English? Any I will add the "disabling" in the next
version ;-)

Regards,
Boqun

> function is disabled only if the page size is > 4 Kbytes.
> 
> > 
> > Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> > ---
> > v1 --> v1.1:
> > 
> > *	Use HV_HYP_PAGE_SIZE instead of hard coding 4096 as suggested by
> > 	Michael.
> > 
> > *	Explicitly print out the disable message if a function is
> > 	disabled as suggested by Michael.
> > 
> >  drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
> >  1 file changed, 34 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> > index 062156b88a87..eee7402cfc02 100644
> > --- a/drivers/hv/hv_balloon.c
> > +++ b/drivers/hv/hv_balloon.c
> > @@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
> >  	}
> >  }
> > 
> > +static int ballooning_enabled(void)
> > +{
> > +	/*
> > +	 * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
> > +	 * since currently it's unclear to us whether an unballoon request can
> > +	 * make sure all page ranges are guest page size aligned.
> 
> My interpretation of the conversations with Hyper-V is that that they clearly
> don't guarantee page ranges are guest page aligned.
> 
> > +	 */
> > +	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> > +		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> > +		return 0;
> > +	}
> > +
> > +	return 1;
> > +}
> > +
> > +static int hot_add_enabled(void)
> > +{
> > +	/*
> > +	 * Disable hot add on ARM64, because we currently rely on
> > +	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> > +	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> > +	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> > +	 * add_memory().
> > +	 */
> > +	if (IS_ENABLED(CONFIG_ARM64)) {
> > +		pr_info("Memory hot add disabled on ARM64\n");
> > +		return 0;
> > +	}
> > +
> > +	return 1;
> > +}
> > +
> >  static int balloon_connect_vsp(struct hv_device *dev)
> >  {
> >  	struct dm_version_request version_req;
> > @@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
> >  	 * currently still requires the bits to be set, so we have to add code
> >  	 * to fail the host's hot-add and balloon up/down requests, if any.
> >  	 */
> > -	cap_msg.caps.cap_bits.balloon = 1;
> > -	cap_msg.caps.cap_bits.hot_add = 1;
> > +	cap_msg.caps.cap_bits.balloon = ballooning_enabled();
> > +	cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
> > 
> >  	/*
> >  	 * Specify our alignment requirements as it relates
> > --
> > 2.33.0
> 
> The code looks good to me.
> 
> Michael