Re: [Xen-users] Xen memory allocation for dom0 and domU

Re: [Xen-users] Xen memory allocation for dom0 and domU

[Ian Campbell]

On Wed, 2015-02-04 at 14:55 +0000, Ian Campbell wrote:
> On Wed, 2015-02-04 at 09:45 -0500, Jintack Lim wrote:
> > > Don't forget that Xen itself will consume some RAM, according to your
> > > numbers it's something between 256M and 350M on this system (that's more
> > > than my gut feeling expects, but not way out).
> > >
> > 
> > Yes, it seems Xen is consuming 289M.
> > Is it expected?
> 
> It's a bit more than my gut feeling would have said we used, but not so
> large I think it indicates something is very wrong.

It's the xenheap, which is 1/8 of the total RAM (at least 128M and
capped at 1GB), so in your case ~256M less whatever allocations made
from it during boot.

xenheap is Xen's "malloc heap" (always mapped), as opposed to the dom
heap which is where domain memory comes from and is demand mapped. The
domheap is what xl info calls "free memory" and what you need to
allocate in order to start a guest.

I originally used to think that domheap allocations would fall back to
the xenheap if the domheap was exhausted, but I think I was mistaken in
that.

This is only an issue on arm32 because on arm64 all of RAM is always
mapped and there is no separate domheap (on Seattle from your other mail
I think what you are seeing is the large frametable from the 16GB of
RAM)

I think that 1/8 RAM (min 128M) is probably too large. Something like
1/32 (min 32M) perhaps? On a 2GB system that would be 64M. 32-bit x86
used to manage with 12M FWIW.

I also think this should be controllable by the user.

Patch for all this below. Jan, I don't think there is any (possibly
historical on x86_32) x86 option we should be trying to be consistent
with.

Ian.


8<-------------------------

>From f41ab97bcefc74f0f7be76c91bb00e5bd32b7677 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Wed, 4 Feb 2015 16:36:36 +0000
Subject: [PATCH] xen: arm32: reduce default size of the xenheap

... and make it tunable via the command line.

1/8 of RAM is 128M on a 1GB system and 256M on a 2GB system etc,
which is a lot. 1/32 of RAM seems more reasonable. Also drop the
minimum to 32M.

Leave the maximum at 1GB.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 docs/misc/xen-command-line.markdown |    8 ++++++++
 xen/arch/arm/setup.c                |   25 ++++++++++++++++++-------
 2 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index bc316be..dac82ef 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -237,6 +237,14 @@ and not running softirqs. Reduce this if softirqs are not being run frequently
 enough. Setting this to a high value may cause boot failure, particularly if
 the NMI watchdog is also enabled.
 
+### xenheap\_size (arm32)
+> `= <size>`
+
+> Default: `1/32 RAM`
+
+Amount of RAM to set aside for the Xenheap. By default 1/32 of the RAM
+up to a maximum of 1GB and with a minimum of 32M.
+
 ### clocksource
 > `= pit | hpet | acpi`
 
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index a916ca6..5be1637 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -50,6 +50,11 @@ struct bootinfo __initdata bootinfo;
 
 struct cpuinfo_arm __read_mostly boot_cpu_data;
 
+#ifdef CONFIG_ARM_32
+static unsigned long opt_xenheap_size __initdata;
+size_param("xenheap_size", opt_xenheap_size);
+#endif
+
 static __used void init_done(void)
 {
     free_init_memory();
@@ -501,16 +506,21 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
      *
      *  - must be 32 MiB aligned
      *  - must not include Xen itself or the boot modules
-     *  - must be at most 1GB or 1/8 the total RAM in the system if less
-     *  - must be at least 128M
+     *  - must be at most 1GB or 1/32 the total RAM in the system if less
+     *  - must be at least 32M
      *
      * We try to allocate the largest xenheap possible within these
      * constraints.
      */
     heap_pages = ram_pages;
-    xenheap_pages = (heap_pages/8 + 0x1fffUL) & ~0x1fffUL;
-    xenheap_pages = max(xenheap_pages, 128UL<<(20-PAGE_SHIFT));
-    xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT));
+    if ( opt_xenheap_size )
+        xenheap_pages = opt_xenheap_size >> PAGE_SHIFT;
+    else
+    {
+        xenheap_pages = (heap_pages/32 + 0x1fffUL) & ~0x1fffUL;
+        xenheap_pages = max(xenheap_pages, 32UL<<(20-PAGE_SHIFT));
+        xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT));
+    }
 
     do
     {
@@ -528,8 +538,9 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
 
     domheap_pages = heap_pages - xenheap_pages;
 
-    printk("Xen heap: %"PRIpaddr"-%"PRIpaddr" (%lu pages)\n",
-            e - (pfn_to_paddr(xenheap_pages)), e, xenheap_pages);
+    printk("Xen heap: %"PRIpaddr"-%"PRIpaddr" (%lu pages%s)\n",
+           e - (pfn_to_paddr(xenheap_pages)), e, xenheap_pages,
+           opt_xenheap_size ? ", from command-line" : "");
     printk("Dom heap: %lu pages\n", domheap_pages);
 
     setup_xenheap_mappings((e >> PAGE_SHIFT) - xenheap_pages, xenheap_pages);
-- 
1.7.10.4

Re: [Xen-users] Xen memory allocation for dom0 and domU

[Jan Beulich]

>>> On 04.02.15 at 17:41, <Ian.Campbell@citrix.com> wrote:
> I originally used to think that domheap allocations would fall back to
> the xenheap if the domheap was exhausted, but I think I was mistaken in
> that.

That's an arch choice actually - there are two variants of the Xen
heap allocation function.

> Patch for all this below. Jan, I don't think there is any (possibly
> historical on x86_32) x86 option we should be trying to be consistent
> with.

On x86-32 it was always fixed 16M. On x86-64 we had a
"xenheap_megabytes=" option before the sharing of the pools
got introduced.

Jan

Re: [Xen-users] Xen memory allocation for dom0 and domU

[Ian Campbell]

On Wed, 2015-02-04 at 16:51 +0000, Jan Beulich wrote:
> >>> On 04.02.15 at 17:41, <Ian.Campbell@citrix.com> wrote:
> > I originally used to think that domheap allocations would fall back to
> > the xenheap if the domheap was exhausted, but I think I was mistaken in
> > that.
> 
> That's an arch choice actually - there are two variants of the Xen
> heap allocation function.

Ah yes, I keep forgetting about the split which was added to the !
CONFIG_SEPARATE_XENHEAP case recently for x86.

arm32 currently uses CONFIG_SEPARATE_XENHEAP, I don't think it is worth
switching since arm32 will never have truly enormous amounts of RAM I
don't think, plus I'd quite like to be able to backport at least some
aspect of this patch (e.g. the cmdline option if not the change to the
defaults).

arm64 uses !CONFIG_SEPARATE_XENHEAP but doesn't currently register any
RAM above the xenheap_bits limit. We probably will at some point,
although due to the lack of PV guests we have more hypervisor address
space to use for 1:1 than x86 does.

> > Patch for all this below. Jan, I don't think there is any (possibly
> > historical on x86_32) x86 option we should be trying to be consistent
> > with.
> 
> On x86-32 it was always fixed 16M. On x86-64 we had a
> "xenheap_megabytes=" option before the sharing of the pools
> got introduced.

I suppose I should use the same thing for at least some sort of
consistency -- it's not like being able to set the xenheap at
sub-megabyte granularity is going to be very useful...

Ian.

Re: [Xen-users] Xen memory allocation for dom0 and domU

[Jintack Lim]

On Wed, Feb 4, 2015 at 11:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>
> Patch for all this below. Jan, I don't think there is any (possibly
> historical on x86_32) x86 option we should be trying to be consistent
> with.

Thanks, Ian.
Your patch works well!

Jintack

Re: [Xen-users] Xen memory allocation for dom0 and domU

[Jintack Lim]

On Wed, Feb 4, 2015 at 9:55 PM, Jintack Lim <jintack@cs.columbia.edu> wrote:
> On Wed, Feb 4, 2015 at 11:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>
>> Patch for all this below. Jan, I don't think there is any (possibly
>> historical on x86_32) x86 option we should be trying to be consistent
>> with.
>
> Thanks, Ian.
> Your patch works well!
>

Will this patch be pushed to the upstream?

I was not able to find this patch in here.
 http://xenbits.xen.org/gitweb/?p=xen.git;a=summary
Could you tell me where is the patch now?

> Jintack

Re: [Xen-users] Xen memory allocation for dom0 and domU

[Ian Campbell]

On Wed, 2015-02-04 at 22:39 -0500, Jintack Lim wrote:
> On Wed, Feb 4, 2015 at 9:55 PM, Jintack Lim <jintack@cs.columbia.edu> wrote:
> > On Wed, Feb 4, 2015 at 11:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >>
> >> Patch for all this below. Jan, I don't think there is any (possibly
> >> historical on x86_32) x86 option we should be trying to be consistent
> >> with.
> >
> > Thanks, Ian.
> > Your patch works well!
> >
> 
> Will this patch be pushed to the upstream?
> 
> I was not able to find this patch in here.
>  http://xenbits.xen.org/gitweb/?p=xen.git;a=summary
> Could you tell me where is the patch now?

It'll be reviewed on the xen-devel list and eventually (hopefully) acked
and committed to the staging branch, from where it will be automatically
tested and propagate to the master branch.

Based on Jan's comments I'm thinking there will be at least one more
revision.

Depending on how invasive the final patch is I may also consider it for
backport to the stable 4.5.x.

Ian.