* Re: [Xen-users] Xen memory allocation for dom0 and domU [not found] ` <1423061748.24924.10.camel@citrix.com> @ 2015-02-04 16:41 ` Ian Campbell 2015-02-04 16:51 ` Jan Beulich 2015-02-05 2:55 ` Jintack Lim 0 siblings, 2 replies; 6+ messages in thread From: Ian Campbell @ 2015-02-04 16:41 UTC (permalink / raw) To: Jintack Lim Cc: xen-users, Stefano Stabellini, Tim Deegan, Jan Beulich, xen-devel On Wed, 2015-02-04 at 14:55 +0000, Ian Campbell wrote: > On Wed, 2015-02-04 at 09:45 -0500, Jintack Lim wrote: > > > Don't forget that Xen itself will consume some RAM, according to your > > > numbers it's something between 256M and 350M on this system (that's more > > > than my gut feeling expects, but not way out). > > > > > > > Yes, it seems Xen is consuming 289M. > > Is it expected? > > It's a bit more than my gut feeling would have said we used, but not so > large I think it indicates something is very wrong. It's the xenheap, which is 1/8 of the total RAM (at least 128M and capped at 1GB), so in your case ~256M less whatever allocations made from it during boot. xenheap is Xen's "malloc heap" (always mapped), as opposed to the dom heap which is where domain memory comes from and is demand mapped. The domheap is what xl info calls "free memory" and what you need to allocate in order to start a guest. I originally used to think that domheap allocations would fall back to the xenheap if the domheap was exhausted, but I think I was mistaken in that. This is only an issue on arm32 because on arm64 all of RAM is always mapped and there is no separate domheap (on Seattle from your other mail I think what you are seeing is the large frametable from the 16GB of RAM) I think that 1/8 RAM (min 128M) is probably too large. Something like 1/32 (min 32M) perhaps? On a 2GB system that would be 64M. 32-bit x86 used to manage with 12M FWIW. I also think this should be controllable by the user. Patch for all this below. Jan, I don't think there is any (possibly historical on x86_32) x86 option we should be trying to be consistent with. Ian. 8<------------------------- >From f41ab97bcefc74f0f7be76c91bb00e5bd32b7677 Mon Sep 17 00:00:00 2001 From: Ian Campbell <ian.campbell@citrix.com> Date: Wed, 4 Feb 2015 16:36:36 +0000 Subject: [PATCH] xen: arm32: reduce default size of the xenheap ... and make it tunable via the command line. 1/8 of RAM is 128M on a 1GB system and 256M on a 2GB system etc, which is a lot. 1/32 of RAM seems more reasonable. Also drop the minimum to 32M. Leave the maximum at 1GB. Signed-off-by: Ian Campbell <ian.campbell@citrix.com> --- docs/misc/xen-command-line.markdown | 8 ++++++++ xen/arch/arm/setup.c | 25 ++++++++++++++++++------- 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown index bc316be..dac82ef 100644 --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -237,6 +237,14 @@ and not running softirqs. Reduce this if softirqs are not being run frequently enough. Setting this to a high value may cause boot failure, particularly if the NMI watchdog is also enabled. +### xenheap\_size (arm32) +> `= <size>` + +> Default: `1/32 RAM` + +Amount of RAM to set aside for the Xenheap. By default 1/32 of the RAM +up to a maximum of 1GB and with a minimum of 32M. + ### clocksource > `= pit | hpet | acpi` diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index a916ca6..5be1637 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -50,6 +50,11 @@ struct bootinfo __initdata bootinfo; struct cpuinfo_arm __read_mostly boot_cpu_data; +#ifdef CONFIG_ARM_32 +static unsigned long opt_xenheap_size __initdata; +size_param("xenheap_size", opt_xenheap_size); +#endif + static __used void init_done(void) { free_init_memory(); @@ -501,16 +506,21 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size) * * - must be 32 MiB aligned * - must not include Xen itself or the boot modules - * - must be at most 1GB or 1/8 the total RAM in the system if less - * - must be at least 128M + * - must be at most 1GB or 1/32 the total RAM in the system if less + * - must be at least 32M * * We try to allocate the largest xenheap possible within these * constraints. */ heap_pages = ram_pages; - xenheap_pages = (heap_pages/8 + 0x1fffUL) & ~0x1fffUL; - xenheap_pages = max(xenheap_pages, 128UL<<(20-PAGE_SHIFT)); - xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT)); + if ( opt_xenheap_size ) + xenheap_pages = opt_xenheap_size >> PAGE_SHIFT; + else + { + xenheap_pages = (heap_pages/32 + 0x1fffUL) & ~0x1fffUL; + xenheap_pages = max(xenheap_pages, 32UL<<(20-PAGE_SHIFT)); + xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT)); + } do { @@ -528,8 +538,9 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size) domheap_pages = heap_pages - xenheap_pages; - printk("Xen heap: %"PRIpaddr"-%"PRIpaddr" (%lu pages)\n", - e - (pfn_to_paddr(xenheap_pages)), e, xenheap_pages); + printk("Xen heap: %"PRIpaddr"-%"PRIpaddr" (%lu pages%s)\n", + e - (pfn_to_paddr(xenheap_pages)), e, xenheap_pages, + opt_xenheap_size ? ", from command-line" : ""); printk("Dom heap: %lu pages\n", domheap_pages); setup_xenheap_mappings((e >> PAGE_SHIFT) - xenheap_pages, xenheap_pages); -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Xen-users] Xen memory allocation for dom0 and domU 2015-02-04 16:41 ` [Xen-users] Xen memory allocation for dom0 and domU Ian Campbell @ 2015-02-04 16:51 ` Jan Beulich 2015-02-05 10:16 ` Ian Campbell 2015-02-05 2:55 ` Jintack Lim 1 sibling, 1 reply; 6+ messages in thread From: Jan Beulich @ 2015-02-04 16:51 UTC (permalink / raw) To: Ian Campbell Cc: xen-users, xen-devel, Tim Deegan, Jintack Lim, StefanoStabellini >>> On 04.02.15 at 17:41, <Ian.Campbell@citrix.com> wrote: > I originally used to think that domheap allocations would fall back to > the xenheap if the domheap was exhausted, but I think I was mistaken in > that. That's an arch choice actually - there are two variants of the Xen heap allocation function. > Patch for all this below. Jan, I don't think there is any (possibly > historical on x86_32) x86 option we should be trying to be consistent > with. On x86-32 it was always fixed 16M. On x86-64 we had a "xenheap_megabytes=" option before the sharing of the pools got introduced. Jan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xen-users] Xen memory allocation for dom0 and domU 2015-02-04 16:51 ` Jan Beulich @ 2015-02-05 10:16 ` Ian Campbell 0 siblings, 0 replies; 6+ messages in thread From: Ian Campbell @ 2015-02-05 10:16 UTC (permalink / raw) To: Jan Beulich Cc: xen-users, StefanoStabellini, Tim Deegan, Jintack Lim, xen-devel On Wed, 2015-02-04 at 16:51 +0000, Jan Beulich wrote: > >>> On 04.02.15 at 17:41, <Ian.Campbell@citrix.com> wrote: > > I originally used to think that domheap allocations would fall back to > > the xenheap if the domheap was exhausted, but I think I was mistaken in > > that. > > That's an arch choice actually - there are two variants of the Xen > heap allocation function. Ah yes, I keep forgetting about the split which was added to the ! CONFIG_SEPARATE_XENHEAP case recently for x86. arm32 currently uses CONFIG_SEPARATE_XENHEAP, I don't think it is worth switching since arm32 will never have truly enormous amounts of RAM I don't think, plus I'd quite like to be able to backport at least some aspect of this patch (e.g. the cmdline option if not the change to the defaults). arm64 uses !CONFIG_SEPARATE_XENHEAP but doesn't currently register any RAM above the xenheap_bits limit. We probably will at some point, although due to the lack of PV guests we have more hypervisor address space to use for 1:1 than x86 does. > > Patch for all this below. Jan, I don't think there is any (possibly > > historical on x86_32) x86 option we should be trying to be consistent > > with. > > On x86-32 it was always fixed 16M. On x86-64 we had a > "xenheap_megabytes=" option before the sharing of the pools > got introduced. I suppose I should use the same thing for at least some sort of consistency -- it's not like being able to set the xenheap at sub-megabyte granularity is going to be very useful... Ian. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xen-users] Xen memory allocation for dom0 and domU 2015-02-04 16:41 ` [Xen-users] Xen memory allocation for dom0 and domU Ian Campbell 2015-02-04 16:51 ` Jan Beulich @ 2015-02-05 2:55 ` Jintack Lim 2015-02-05 3:39 ` Jintack Lim 1 sibling, 1 reply; 6+ messages in thread From: Jintack Lim @ 2015-02-05 2:55 UTC (permalink / raw) To: Ian Campbell Cc: xen-users, Stefano Stabellini, Tim Deegan, Jan Beulich, xen-devel On Wed, Feb 4, 2015 at 11:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > Patch for all this below. Jan, I don't think there is any (possibly > historical on x86_32) x86 option we should be trying to be consistent > with. Thanks, Ian. Your patch works well! Jintack ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xen-users] Xen memory allocation for dom0 and domU 2015-02-05 2:55 ` Jintack Lim @ 2015-02-05 3:39 ` Jintack Lim 2015-02-05 10:07 ` Ian Campbell 0 siblings, 1 reply; 6+ messages in thread From: Jintack Lim @ 2015-02-05 3:39 UTC (permalink / raw) To: Ian Campbell Cc: xen-users, Stefano Stabellini, Tim Deegan, Jan Beulich, xen-devel On Wed, Feb 4, 2015 at 9:55 PM, Jintack Lim <jintack@cs.columbia.edu> wrote: > On Wed, Feb 4, 2015 at 11:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> >> Patch for all this below. Jan, I don't think there is any (possibly >> historical on x86_32) x86 option we should be trying to be consistent >> with. > > Thanks, Ian. > Your patch works well! > Will this patch be pushed to the upstream? I was not able to find this patch in here. http://xenbits.xen.org/gitweb/?p=xen.git;a=summary Could you tell me where is the patch now? > Jintack ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xen-users] Xen memory allocation for dom0 and domU 2015-02-05 3:39 ` Jintack Lim @ 2015-02-05 10:07 ` Ian Campbell 0 siblings, 0 replies; 6+ messages in thread From: Ian Campbell @ 2015-02-05 10:07 UTC (permalink / raw) To: Jintack Lim Cc: xen-users, Stefano Stabellini, Tim Deegan, Jan Beulich, xen-devel On Wed, 2015-02-04 at 22:39 -0500, Jintack Lim wrote: > On Wed, Feb 4, 2015 at 9:55 PM, Jintack Lim <jintack@cs.columbia.edu> wrote: > > On Wed, Feb 4, 2015 at 11:41 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > >> > >> Patch for all this below. Jan, I don't think there is any (possibly > >> historical on x86_32) x86 option we should be trying to be consistent > >> with. > > > > Thanks, Ian. > > Your patch works well! > > > > Will this patch be pushed to the upstream? > > I was not able to find this patch in here. > http://xenbits.xen.org/gitweb/?p=xen.git;a=summary > Could you tell me where is the patch now? It'll be reviewed on the xen-devel list and eventually (hopefully) acked and committed to the staging branch, from where it will be automatically tested and propagate to the master branch. Based on Jan's comments I'm thinking there will be at least one more revision. Depending on how invasive the final patch is I may also consider it for backport to the stable 4.5.x. Ian. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-02-05 10:16 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAHyh4xjOykrATAUXGqNd1h1ThiZ5FX680Pn=_mKxtePaAF-ubQ@mail.gmail.com>
[not found] ` <1423044595.17711.12.camel@citrix.com>
[not found] ` <CAHyh4xiPZsqWkuthvHHC-ina7Ua2YHqeAiiP6U-3oAC=M+pnHw@mail.gmail.com>
[not found] ` <1423061748.24924.10.camel@citrix.com>
2015-02-04 16:41 ` [Xen-users] Xen memory allocation for dom0 and domU Ian Campbell
2015-02-04 16:51 ` Jan Beulich
2015-02-05 10:16 ` Ian Campbell
2015-02-05 2:55 ` Jintack Lim
2015-02-05 3:39 ` Jintack Lim
2015-02-05 10:07 ` Ian Campbell
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.