[PATCH v2 0/2] Drivers: hv: balloon: Temporary fixes for ARM64

[PATCH v2 0/2] Drivers: hv: balloon: Temporary fixes for ARM64

[Boqun Feng]

v1: https://lore.kernel.org/lkml/20220223131548.2234326-1-boqun.feng@gmail.com/

Changes since v1:

*	Wording changes suggested by Michael Kelley.

Since Hyper-V always uses 4k pages, hv_balloon has some difficulties
working on ARM64 with larger pages[1]. Besides the memory hot add
messages of Hyper-V doesn't have the information of NUMA node id of the
added memory range, and ARM64 currently doesn't provide the conversion
from a physical address to a node id, as a result the hv_balloon driver
couldn't handle hot add properly when there are more than one NUMA node.

Among these issues, post_status() is easy to fix, while the unballoon
issue and the hot-add issue requires more discussion. To make the
hv_balloon driver work at the best effort, this patchset fixes the
post_status() and temporarily disable the balloon and hot-add
accordingly.

Looking forwards to comments and suggestions.

Regards,
Boqun

[1]: https://lore.kernel.org/lkml/20220105165028.1343706-1-vkuznets@redhat.com/

*** BLURB HERE ***

Boqun Feng (2):
  Drivers: hv: balloon: Support status report for larger page sizes
  Drivers: hv: balloon: Disable balloon and hot-add accordingly

 drivers/hv/hv_balloon.c | 49 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 44 insertions(+), 5 deletions(-)

-- 
2.35.1

[PATCH v2 1/2] Drivers: hv: balloon: Support status report for larger page sizes

[Boqun Feng]

DM_STATUS_REPORT expects the numbers of pages in the unit of 4k pages
(HV_HYP_PAGE) instead of guest pages, so to make it work when guest page
sizes are larger than 4k, convert the numbers of guest pages into the
numbers of HV_HYP_PAGEs.

Note that the numbers of guest pages are still used for tracing because
tracing is internal to the guest kernel.

Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
---
 drivers/hv/hv_balloon.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index f2d05bff4245..062156b88a87 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -17,6 +17,7 @@
 #include <linux/slab.h>
 #include <linux/kthread.h>
 #include <linux/completion.h>
+#include <linux/count_zeros.h>
 #include <linux/memory_hotplug.h>
 #include <linux/memory.h>
 #include <linux/notifier.h>
@@ -1130,6 +1131,7 @@ static void post_status(struct hv_dynmem_device *dm)
 	struct dm_status status;
 	unsigned long now = jiffies;
 	unsigned long last_post = last_post_time;
+	unsigned long num_pages_avail, num_pages_committed;
 
 	if (pressure_report_delay > 0) {
 		--pressure_report_delay;
@@ -1154,16 +1156,21 @@ static void post_status(struct hv_dynmem_device *dm)
 	 * num_pages_onlined) as committed to the host, otherwise it can try
 	 * asking us to balloon them out.
 	 */
-	status.num_avail = si_mem_available();
-	status.num_committed = vm_memory_committed() +
+	num_pages_avail = si_mem_available();
+	num_pages_committed = vm_memory_committed() +
 		dm->num_pages_ballooned +
 		(dm->num_pages_added > dm->num_pages_onlined ?
 		 dm->num_pages_added - dm->num_pages_onlined : 0) +
 		compute_balloon_floor();
 
-	trace_balloon_status(status.num_avail, status.num_committed,
+	trace_balloon_status(num_pages_avail, num_pages_committed,
 			     vm_memory_committed(), dm->num_pages_ballooned,
 			     dm->num_pages_added, dm->num_pages_onlined);
+
+	/* Convert numbers of pages into numbers of HV_HYP_PAGEs. */
+	status.num_avail = num_pages_avail * NR_HV_HYP_PAGES_IN_PAGE;
+	status.num_committed = num_pages_committed * NR_HV_HYP_PAGES_IN_PAGE;
+
 	/*
 	 * If our transaction ID is no longer current, just don't
 	 * send the status. This can happen if we were interrupted
-- 
2.35.1

Re: [PATCH v2 1/2] Drivers: hv: balloon: Support status report for larger page sizes

[Boqun Feng]

Hi,

I think we also want this patch in the 5.15 stable. Without this patch,
hv_balloon() will report incorrect memory usage to hypervisor when
running on ARM64 with PAGE_SIZE > 4k. Only 5.15 needs this because ARM64
support of HyperV guests arrived in 5.15.

Upstream id b3d6dd09ff00fdcf4f7c0cb54700ffd5dd343502

Cc: <stable@vger.kernel.org> # 5.15.x

Thanks!

Regards,
Boqun

On Fri, Mar 25, 2022 at 10:32:11AM +0800, Boqun Feng wrote:
> DM_STATUS_REPORT expects the numbers of pages in the unit of 4k pages
> (HV_HYP_PAGE) instead of guest pages, so to make it work when guest page
> sizes are larger than 4k, convert the numbers of guest pages into the
> numbers of HV_HYP_PAGEs.
> 
> Note that the numbers of guest pages are still used for tracing because
> tracing is internal to the guest kernel.
> 
> Reported-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> Reviewed-by: Michael Kelley <mikelley@microsoft.com>
> ---
>  drivers/hv/hv_balloon.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index f2d05bff4245..062156b88a87 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -17,6 +17,7 @@
>  #include <linux/slab.h>
>  #include <linux/kthread.h>
>  #include <linux/completion.h>
> +#include <linux/count_zeros.h>
>  #include <linux/memory_hotplug.h>
>  #include <linux/memory.h>
>  #include <linux/notifier.h>
> @@ -1130,6 +1131,7 @@ static void post_status(struct hv_dynmem_device *dm)
>  	struct dm_status status;
>  	unsigned long now = jiffies;
>  	unsigned long last_post = last_post_time;
> +	unsigned long num_pages_avail, num_pages_committed;
>  
>  	if (pressure_report_delay > 0) {
>  		--pressure_report_delay;
> @@ -1154,16 +1156,21 @@ static void post_status(struct hv_dynmem_device *dm)
>  	 * num_pages_onlined) as committed to the host, otherwise it can try
>  	 * asking us to balloon them out.
>  	 */
> -	status.num_avail = si_mem_available();
> -	status.num_committed = vm_memory_committed() +
> +	num_pages_avail = si_mem_available();
> +	num_pages_committed = vm_memory_committed() +
>  		dm->num_pages_ballooned +
>  		(dm->num_pages_added > dm->num_pages_onlined ?
>  		 dm->num_pages_added - dm->num_pages_onlined : 0) +
>  		compute_balloon_floor();
>  
> -	trace_balloon_status(status.num_avail, status.num_committed,
> +	trace_balloon_status(num_pages_avail, num_pages_committed,
>  			     vm_memory_committed(), dm->num_pages_ballooned,
>  			     dm->num_pages_added, dm->num_pages_onlined);
> +
> +	/* Convert numbers of pages into numbers of HV_HYP_PAGEs. */
> +	status.num_avail = num_pages_avail * NR_HV_HYP_PAGES_IN_PAGE;
> +	status.num_committed = num_pages_committed * NR_HV_HYP_PAGES_IN_PAGE;
> +
>  	/*
>  	 * If our transaction ID is no longer current, just don't
>  	 * send the status. This can happen if we were interrupted
> -- 
> 2.35.1
> 

Re: [PATCH v2 1/2] Drivers: hv: balloon: Support status report for larger page sizes

[Greg Kroah-Hartman]

On Wed, Aug 31, 2022 at 12:11:20PM -0700, Boqun Feng wrote:
> Hi,
> 
> I think we also want this patch in the 5.15 stable. Without this patch,
> hv_balloon() will report incorrect memory usage to hypervisor when
> running on ARM64 with PAGE_SIZE > 4k. Only 5.15 needs this because ARM64
> support of HyperV guests arrived in 5.15.
> 
> Upstream id b3d6dd09ff00fdcf4f7c0cb54700ffd5dd343502
> 
> Cc: <stable@vger.kernel.org> # 5.15.x

Now queued up, thanks.

greg k-h

[PATCH v2 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Boqun Feng]

Currently there are known potential issues for balloon and hot-add on
ARM64:

*	Unballoon requests from Hyper-V should only unballoon ranges
	that are guest page size aligned, otherwise guests cannot handle
	because it's impossible to partially free a page. This is a
	problem when guest page size > 4096 bytes.

*	Memory hot-add requests from Hyper-V should provide the NUMA
	node id of the added ranges or ARM64 should have a functional
	memory_add_physaddr_to_nid(), otherwise the node id is missing
	for add_memory().

These issues require discussions on design and implementation. In the
meanwhile, post_status() is working and essential to guest monitoring.
Therefore instead of disabling the entire hv_balloon driver, the
ballooning (when page size > 4096 bytes) and hot-add are disabled
accordingly for now. Once the issues are fixed, they can be re-enable in
these cases.

Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
---
 drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index 062156b88a87..eee7402cfc02 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
 	}
 }
 
+static int ballooning_enabled(void)
+{
+	/*
+	 * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
+	 * since currently it's unclear to us whether an unballoon request can
+	 * make sure all page ranges are guest page size aligned.
+	 */
+	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
+		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
+		return 0;
+	}
+
+	return 1;
+}
+
+static int hot_add_enabled(void)
+{
+	/*
+	 * Disable hot add on ARM64, because we currently rely on
+	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
+	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
+	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
+	 * add_memory().
+	 */
+	if (IS_ENABLED(CONFIG_ARM64)) {
+		pr_info("Memory hot add disabled on ARM64\n");
+		return 0;
+	}
+
+	return 1;
+}
+
 static int balloon_connect_vsp(struct hv_device *dev)
 {
 	struct dm_version_request version_req;
@@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
 	 * currently still requires the bits to be set, so we have to add code
 	 * to fail the host's hot-add and balloon up/down requests, if any.
 	 */
-	cap_msg.caps.cap_bits.balloon = 1;
-	cap_msg.caps.cap_bits.hot_add = 1;
+	cap_msg.caps.cap_bits.balloon = ballooning_enabled();
+	cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
 
 	/*
 	 * Specify our alignment requirements as it relates
-- 
2.35.1

RE: [PATCH v2 2/2] Drivers: hv: balloon: Disable balloon and hot-add accordingly

[Michael Kelley (LINUX)]

From: Boqun Feng <boqun.feng@gmail.com> Sent: Thursday, March 24, 2022 7:32 PM
> 
> Currently there are known potential issues for balloon and hot-add on
> ARM64:
> 
> *	Unballoon requests from Hyper-V should only unballoon ranges
> 	that are guest page size aligned, otherwise guests cannot handle
> 	because it's impossible to partially free a page. This is a
> 	problem when guest page size > 4096 bytes.
> 
> *	Memory hot-add requests from Hyper-V should provide the NUMA
> 	node id of the added ranges or ARM64 should have a functional
> 	memory_add_physaddr_to_nid(), otherwise the node id is missing
> 	for add_memory().
> 
> These issues require discussions on design and implementation. In the
> meanwhile, post_status() is working and essential to guest monitoring.
> Therefore instead of disabling the entire hv_balloon driver, the
> ballooning (when page size > 4096 bytes) and hot-add are disabled
> accordingly for now. Once the issues are fixed, they can be re-enable in
> these cases.
> 
> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
> ---
>  drivers/hv/hv_balloon.c | 36 ++++++++++++++++++++++++++++++++++--
>  1 file changed, 34 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
> index 062156b88a87..eee7402cfc02 100644
> --- a/drivers/hv/hv_balloon.c
> +++ b/drivers/hv/hv_balloon.c
> @@ -1660,6 +1660,38 @@ static void disable_page_reporting(void)
>  	}
>  }
> 
> +static int ballooning_enabled(void)
> +{
> +	/*
> +	 * Disable ballooning if the page size is not 4k (HV_HYP_PAGE_SIZE),
> +	 * since currently it's unclear to us whether an unballoon request can
> +	 * make sure all page ranges are guest page size aligned.
> +	 */
> +	if (PAGE_SIZE != HV_HYP_PAGE_SIZE) {
> +		pr_info("Ballooning disabled because page size is not 4096 bytes\n");
> +		return 0;
> +	}
> +
> +	return 1;
> +}
> +
> +static int hot_add_enabled(void)
> +{
> +	/*
> +	 * Disable hot add on ARM64, because we currently rely on
> +	 * memory_add_physaddr_to_nid() to get a node id of a hot add range,
> +	 * however ARM64's memory_add_physaddr_to_nid() always return 0 and
> +	 * DM_MEM_HOT_ADD_REQUEST doesn't have the NUMA node information for
> +	 * add_memory().
> +	 */
> +	if (IS_ENABLED(CONFIG_ARM64)) {
> +		pr_info("Memory hot add disabled on ARM64\n");
> +		return 0;
> +	}
> +
> +	return 1;
> +}
> +
>  static int balloon_connect_vsp(struct hv_device *dev)
>  {
>  	struct dm_version_request version_req;
> @@ -1731,8 +1763,8 @@ static int balloon_connect_vsp(struct hv_device *dev)
>  	 * currently still requires the bits to be set, so we have to add code
>  	 * to fail the host's hot-add and balloon up/down requests, if any.
>  	 */
> -	cap_msg.caps.cap_bits.balloon = 1;
> -	cap_msg.caps.cap_bits.hot_add = 1;
> +	cap_msg.caps.cap_bits.balloon = ballooning_enabled();
> +	cap_msg.caps.cap_bits.hot_add = hot_add_enabled();
> 
>  	/*
>  	 * Specify our alignment requirements as it relates
> --
> 2.35.1

Reviewed-by: Michael Kelley <mikelley@microsoft.com>

Re: [PATCH v2 0/2] Drivers: hv: balloon: Temporary fixes for ARM64

[Wei Liu]

On Fri, Mar 25, 2022 at 10:32:10AM +0800, Boqun Feng wrote:
> v1: https://lore.kernel.org/lkml/20220223131548.2234326-1-boqun.feng@gmail.com/
> Boqun Feng (2):
>   Drivers: hv: balloon: Support status report for larger page sizes
>   Drivers: hv: balloon: Disable balloon and hot-add accordingly
> 

Applied to hyperv-fixes. Thanks!