Re: [Xen-users] Xen memory allocation for dom0 and domU

[Ian Campbell <Ian.Campbell@citrix.com>]

On Wed, 2015-02-04 at 14:55 +0000, Ian Campbell wrote:
> On Wed, 2015-02-04 at 09:45 -0500, Jintack Lim wrote:
> > > Don't forget that Xen itself will consume some RAM, according to your
> > > numbers it's something between 256M and 350M on this system (that's more
> > > than my gut feeling expects, but not way out).
> > >
> > 
> > Yes, it seems Xen is consuming 289M.
> > Is it expected?
> 
> It's a bit more than my gut feeling would have said we used, but not so
> large I think it indicates something is very wrong.

It's the xenheap, which is 1/8 of the total RAM (at least 128M and
capped at 1GB), so in your case ~256M less whatever allocations made
from it during boot.

xenheap is Xen's "malloc heap" (always mapped), as opposed to the dom
heap which is where domain memory comes from and is demand mapped. The
domheap is what xl info calls "free memory" and what you need to
allocate in order to start a guest.

I originally used to think that domheap allocations would fall back to
the xenheap if the domheap was exhausted, but I think I was mistaken in
that.

This is only an issue on arm32 because on arm64 all of RAM is always
mapped and there is no separate domheap (on Seattle from your other mail
I think what you are seeing is the large frametable from the 16GB of
RAM)

I think that 1/8 RAM (min 128M) is probably too large. Something like
1/32 (min 32M) perhaps? On a 2GB system that would be 64M. 32-bit x86
used to manage with 12M FWIW.

I also think this should be controllable by the user.

Patch for all this below. Jan, I don't think there is any (possibly
historical on x86_32) x86 option we should be trying to be consistent
with.

Ian.


8<-------------------------

>From f41ab97bcefc74f0f7be76c91bb00e5bd32b7677 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Wed, 4 Feb 2015 16:36:36 +0000
Subject: [PATCH] xen: arm32: reduce default size of the xenheap

... and make it tunable via the command line.

1/8 of RAM is 128M on a 1GB system and 256M on a 2GB system etc,
which is a lot. 1/32 of RAM seems more reasonable. Also drop the
minimum to 32M.

Leave the maximum at 1GB.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 docs/misc/xen-command-line.markdown |    8 ++++++++
 xen/arch/arm/setup.c                |   25 ++++++++++++++++++-------
 2 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index bc316be..dac82ef 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -237,6 +237,14 @@ and not running softirqs. Reduce this if softirqs are not being run frequently
 enough. Setting this to a high value may cause boot failure, particularly if
 the NMI watchdog is also enabled.
 
+### xenheap\_size (arm32)
+> `= <size>`
+
+> Default: `1/32 RAM`
+
+Amount of RAM to set aside for the Xenheap. By default 1/32 of the RAM
+up to a maximum of 1GB and with a minimum of 32M.
+
 ### clocksource
 > `= pit | hpet | acpi`
 
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index a916ca6..5be1637 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -50,6 +50,11 @@ struct bootinfo __initdata bootinfo;
 
 struct cpuinfo_arm __read_mostly boot_cpu_data;
 
+#ifdef CONFIG_ARM_32
+static unsigned long opt_xenheap_size __initdata;
+size_param("xenheap_size", opt_xenheap_size);
+#endif
+
 static __used void init_done(void)
 {
     free_init_memory();
@@ -501,16 +506,21 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
      *
      *  - must be 32 MiB aligned
      *  - must not include Xen itself or the boot modules
-     *  - must be at most 1GB or 1/8 the total RAM in the system if less
-     *  - must be at least 128M
+     *  - must be at most 1GB or 1/32 the total RAM in the system if less
+     *  - must be at least 32M
      *
      * We try to allocate the largest xenheap possible within these
      * constraints.
      */
     heap_pages = ram_pages;
-    xenheap_pages = (heap_pages/8 + 0x1fffUL) & ~0x1fffUL;
-    xenheap_pages = max(xenheap_pages, 128UL<<(20-PAGE_SHIFT));
-    xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT));
+    if ( opt_xenheap_size )
+        xenheap_pages = opt_xenheap_size >> PAGE_SHIFT;
+    else
+    {
+        xenheap_pages = (heap_pages/32 + 0x1fffUL) & ~0x1fffUL;
+        xenheap_pages = max(xenheap_pages, 32UL<<(20-PAGE_SHIFT));
+        xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT));
+    }
 
     do
     {
@@ -528,8 +538,9 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
 
     domheap_pages = heap_pages - xenheap_pages;
 
-    printk("Xen heap: %"PRIpaddr"-%"PRIpaddr" (%lu pages)\n",
-            e - (pfn_to_paddr(xenheap_pages)), e, xenheap_pages);
+    printk("Xen heap: %"PRIpaddr"-%"PRIpaddr" (%lu pages%s)\n",
+           e - (pfn_to_paddr(xenheap_pages)), e, xenheap_pages,
+           opt_xenheap_size ? ", from command-line" : "");
     printk("Dom heap: %lu pages\n", domheap_pages);
 
     setup_xenheap_mappings((e >> PAGE_SHIFT) - xenheap_pages, xenheap_pages);
-- 
1.7.10.4