can not use all available memory

can not use all available memory

[Alexander Bienzeisler]

Hello,

i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
16384) with free_memory : 28969 but got an error:

libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
dom0 is below the minimum threshold

I was able to create it with 8 gigs of ram, ending up with free_memory : 
20776

Is there something wrong with memory in 4.2? I can't seem to use all of it.

Cheers,
Alex

can not use all available memory

[Alexander Bienzeisler]

Hello,

i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
16384) with free_memory : 28969 but got an error:

libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
dom0 is below the minimum threshold

I was able to create it with 8 gigs of ram, ending up with free_memory : 
20776

Is there something wrong with memory in 4.2? I can't seem to use all of it.

Cheers,
Alex

Re: can not use all available memory

[Pasi Kärkkäinen]

On Fri, Nov 23, 2012 at 12:35:26PM +0100, Alexander Bienzeisler wrote:
> Hello,
> 
> i just tried to add a domU with 16 gigs of ram (maxmem = 16384,
> memory = 16384) with free_memory : 28969 but got an error:
> 
> libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for
> dom0 is below the minimum threshold
> 
> I was able to create it with 8 gigs of ram, ending up with
> free_memory : 20776
> 
> Is there something wrong with memory in 4.2? I can't seem to use all of it.
> 

What exact Xen version are you using? what dom0 kernel version? 

Also please pastebin "xl info" and "xl list" before and after starting the VM.

-- Pasi

Re: can not use all available memory

[Alexander Bienzeisler]

I am currently using Xen 4.2.1-pre built from git source. Kernel version 
3.1.1

xl info before:

host                   : susi-0
release                : 3.1.1-xen
version                : #1 SMP Fri Nov 18 10:16:22 CET 2011
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 31
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 3325
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98291
free_memory            : 28969
sharing_freed_memory   : 0
sharing_used_memory    : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 2
xen_extra              : .1-pre
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Wed Nov 14 11:46:07 2012 +0000 25922:8ca6372315f8
xen_commandline        : placeholder loglvl=all dom0_mem=2048M 
dom0_max_vcpus=2 dom0_vcpus_pin com2=115200 console=com2,vga
cc_compiler            : gcc (Debian 4.4.5-8) 4.4.5
cc_compile_by          : root
cc_compile_domain      : wsk.tu-chemnitz.de
cc_compile_date        : Thu Nov 22 13:50:09 CET 2012
xend_config_format     : 4

xl list before:

Name                                        ID   Mem VCPUs State   Time(s)
Domain-0                                     0  2048     2 r-----    3490.0
Squeeze                                     19  4091     2 -b----     370.3
win2k8_2_susi6                              31 16373    12 -b----    1619.3
win2k8_susi5                                33 32757    12 -b----    1264.0
win7_susi3                                  43  4085     4 -b----     416.0
scientific_susi7                            44  8192    16 -b----      39.9

domU config:

name = "centos_susi8"
uuid = "2e87b315-fd5a-dfbc-fe9d-6e121fdfa632"
maxmem = 16384
memory = 16384
vcpus = 16
builder = "generic"
bootloader = "pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ "type=vnc,vncunused=1,keymap=de" ]
disk = [ "phy:/dev/mapper/storage-SL5,xvda,w" ]
vif = [ "mac=00:50:56:37:61:17,bridge=xenbr0" ]

error message after:

xl create /root/configs/centos_susi8.cfg
Parsing config from /root/configs/centos_susi8.cfg
libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
dom0 is below the minimum threshold


with 8gigs:

xl info after:

host                   : susi-0
release                : 3.1.1-xen
version                : #1 SMP Fri Nov 18 10:16:22 CET 2011
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 31
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 3325
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98291
free_memory            : 20776
sharing_freed_memory   : 0
sharing_used_memory    : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 2
xen_extra              : .1-pre
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Wed Nov 14 11:46:07 2012 +0000 25922:8ca6372315f8
xen_commandline        : placeholder loglvl=all dom0_mem=2048M 
dom0_max_vcpus=2 dom0_vcpus_pin com2=115200 console=com2,vga
cc_compiler            : gcc (Debian 4.4.5-8) 4.4.5
cc_compile_by          : root
cc_compile_domain      : wsk.tu-chemnitz.de
cc_compile_date        : Thu Nov 22 13:50:09 CET 2012
xend_config_format     : 4

xl list after:

Name                                        ID   Mem VCPUs State   Time(s)
Domain-0                                     0  2048     2 r-----    3550.0
Squeeze                                     19  4091     2 -b----     381.5
win2k8_2_susi6                              31 16373    12 -b----    1627.1
win2k8_susi5                                33 32757    12 ------    1269.0
win7_susi3                                  43  4085     4 -b----     418.7
scientific_susi7                            44  8192    16 -b----      40.3
centos_susi8                                46  8192    16 -b----       4.4




Am 23.11.2012 12:52, schrieb Pasi Kärkkäinen:
> On Fri, Nov 23, 2012 at 12:35:26PM +0100, Alexander Bienzeisler wrote:
>> Hello,
>>
>> i just tried to add a domU with 16 gigs of ram (maxmem = 16384,
>> memory = 16384) with free_memory : 28969 but got an error:
>>
>> libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for
>> dom0 is below the minimum threshold
>>
>> I was able to create it with 8 gigs of ram, ending up with
>> free_memory : 20776
>>
>> Is there something wrong with memory in 4.2? I can't seem to use all of it.
>>
> What exact Xen version are you using? what dom0 kernel version?
>
> Also please pastebin "xl info" and "xl list" before and after starting the VM.
>
> -- Pasi
>

Re: can not use all available memory

[Ian Campbell]

This question belongs on xen-users.

On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
> Hello,
> 
> i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
> 16384) with free_memory : 28969 but got an error:
> 
> libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
> dom0 is below the minimum threshold

The clue is in this message -- you cannot safely balloon down a guest
(including dom0) to an arbitrarily small value relative to its initial
size. The kernel will have certain fixed datastructures etc which are
sized according to the maximum amount of RAM at start of day.

If you were to balloon down too low then you would run out of RAM and
the OOM killer would step in and start killing stuff, so xl includes a
safety catch which prevents this.

If you have a huge host and want a small dom0 use the dom0_mem option
and disable xl's autoballoon option or whatever the equivalent xend
option is.

Ian.

Re: can not use all available memory

[Pasi Kärkkäinen]

On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:
> This question belongs on xen-users.
> 
> On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
> > Hello,
> > 
> > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
> > 16384) with free_memory : 28969 but got an error:
> > 
> > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
> > dom0 is below the minimum threshold
> 
> The clue is in this message -- you cannot safely balloon down a guest
> (including dom0) to an arbitrarily small value relative to its initial
> size. The kernel will have certain fixed datastructures etc which are
> sized according to the maximum amount of RAM at start of day.
> 
> If you were to balloon down too low then you would run out of RAM and
> the OOM killer would step in and start killing stuff, so xl includes a
> safety catch which prevents this.
> 
> If you have a huge host and want a small dom0 use the dom0_mem option
> and disable xl's autoballoon option or whatever the equivalent xend
> option is.
> 

I think he was already using dom0_mem. He had 28969 free in Xen as reported by "xl info".

-- Pasi

Re: can not use all available memory

[Ian Campbell]

On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote:
> On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:
> > This question belongs on xen-users.
> > 
> > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
> > > Hello,
> > > 
> > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
> > > 16384) with free_memory : 28969 but got an error:
> > > 
> > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
> > > dom0 is below the minimum threshold
> > 
> > The clue is in this message -- you cannot safely balloon down a guest
> > (including dom0) to an arbitrarily small value relative to its initial
> > size. The kernel will have certain fixed datastructures etc which are
> > sized according to the maximum amount of RAM at start of day.
> > 
> > If you were to balloon down too low then you would run out of RAM and
> > the OOM killer would step in and start killing stuff, so xl includes a
> > safety catch which prevents this.
> > 
> > If you have a huge host and want a small dom0 use the dom0_mem option
> > and disable xl's autoballoon option or whatever the equivalent xend
> > option is.
> > 
> 
> I think he was already using dom0_mem. He had 28969 free in Xen as
> reported by "xl info".

Then he's missed disabling autoballoon.

> 
> -- Pasi
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Pasi Kärkkäinen]

On Fri, Nov 23, 2012 at 12:33:59PM +0000, Ian Campbell wrote:
> On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote:
> > On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:
> > > This question belongs on xen-users.
> > > 
> > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
> > > > Hello,
> > > > 
> > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
> > > > 16384) with free_memory : 28969 but got an error:
> > > > 
> > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
> > > > dom0 is below the minimum threshold
> > > 
> > > The clue is in this message -- you cannot safely balloon down a guest
> > > (including dom0) to an arbitrarily small value relative to its initial
> > > size. The kernel will have certain fixed datastructures etc which are
> > > sized according to the maximum amount of RAM at start of day.
> > > 
> > > If you were to balloon down too low then you would run out of RAM and
> > > the OOM killer would step in and start killing stuff, so xl includes a
> > > safety catch which prevents this.
> > > 
> > > If you have a huge host and want a small dom0 use the dom0_mem option
> > > and disable xl's autoballoon option or whatever the equivalent xend
> > > option is.
> > > 
> > 
> > I think he was already using dom0_mem. He had 28969 free in Xen as
> > reported by "xl info".
> 
> Then he's missed disabling autoballoon.
> 

As can be seen from the other email in this thread,
he's using dom0_mem=2048M, and he has 28GB of free memory in Xen.

Why is xl trying to balloon down dom0 (and failing) for no reason? 

-- Pasi

Re: can not use all available memory

[Ian Campbell]

On Fri, 2012-11-23 at 12:41 +0000, Pasi Kärkkäinen wrote:
> On Fri, Nov 23, 2012 at 12:33:59PM +0000, Ian Campbell wrote:
> > On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote:
> > > On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:
> > > > This question belongs on xen-users.
> > > > 
> > > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
> > > > > Hello,
> > > > > 
> > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
> > > > > 16384) with free_memory : 28969 but got an error:
> > > > > 
> > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
> > > > > dom0 is below the minimum threshold
> > > > 
> > > > The clue is in this message -- you cannot safely balloon down a guest
> > > > (including dom0) to an arbitrarily small value relative to its initial
> > > > size. The kernel will have certain fixed datastructures etc which are
> > > > sized according to the maximum amount of RAM at start of day.
> > > > 
> > > > If you were to balloon down too low then you would run out of RAM and
> > > > the OOM killer would step in and start killing stuff, so xl includes a
> > > > safety catch which prevents this.
> > > > 
> > > > If you have a huge host and want a small dom0 use the dom0_mem option
> > > > and disable xl's autoballoon option or whatever the equivalent xend
> > > > option is.
> > > > 
> > > 
> > > I think he was already using dom0_mem. He had 28969 free in Xen as
> > > reported by "xl info".
> > 
> > Then he's missed disabling autoballoon.
> > 
> 
> As can be seen from the other email in this thread,
> he's using dom0_mem=2048M, and he has 28GB of free memory in Xen.
> 
> Why is xl trying to balloon down dom0 (and failing) for no reason? 

Because autoballoon is enabled.

This has been discussed at length on the list before, please check the
archives.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Pasi Kärkkäinen]

On Fri, Nov 23, 2012 at 12:47:07PM +0000, Ian Campbell wrote:
> On Fri, 2012-11-23 at 12:41 +0000, Pasi Kärkkäinen wrote:
> > On Fri, Nov 23, 2012 at 12:33:59PM +0000, Ian Campbell wrote:
> > > On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote:
> > > > On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:
> > > > > This question belongs on xen-users.
> > > > > 
> > > > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 
> > > > > > 16384) with free_memory : 28969 but got an error:
> > > > > > 
> > > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for 
> > > > > > dom0 is below the minimum threshold
> > > > > 
> > > > > The clue is in this message -- you cannot safely balloon down a guest
> > > > > (including dom0) to an arbitrarily small value relative to its initial
> > > > > size. The kernel will have certain fixed datastructures etc which are
> > > > > sized according to the maximum amount of RAM at start of day.
> > > > > 
> > > > > If you were to balloon down too low then you would run out of RAM and
> > > > > the OOM killer would step in and start killing stuff, so xl includes a
> > > > > safety catch which prevents this.
> > > > > 
> > > > > If you have a huge host and want a small dom0 use the dom0_mem option
> > > > > and disable xl's autoballoon option or whatever the equivalent xend
> > > > > option is.
> > > > > 
> > > > 
> > > > I think he was already using dom0_mem. He had 28969 free in Xen as
> > > > reported by "xl info".
> > > 
> > > Then he's missed disabling autoballoon.
> > > 
> > 
> > As can be seen from the other email in this thread,
> > he's using dom0_mem=2048M, and he has 28GB of free memory in Xen.
> > 
> > Why is xl trying to balloon down dom0 (and failing) for no reason? 
> 
> Because autoballoon is enabled.
> 

Yep, autoballoon=0 fixed the problem.

> This has been discussed at length on the list before, please check the
> archives.
> 

I tried googling quickly but I didn't find anything relevant..

To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to 
balloon down dom0 if the hypervisor already has enough free memory for the VM ? 

In this case:
	- dom0_mem=2G
	- new VM to launch with size 16 GB.
	- Xen has 28 GB of free memory.

So clearly there's no need to try to balloon down dom0.. 
not-yet-implemented check/feature in xl, or a bug? 

Thanks,

-- Pasi

Re: can not use all available memory

[Ian Campbell]

On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:

> > This has been discussed at length on the list before, please check the
> > archives.
> > 
> 
> I tried googling quickly but I didn't find anything relevant..

I found
http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
in a matter of seconds, then:
http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html

> To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to 
> balloon down dom0 if the hypervisor already has enough free memory for the VM ? 
> 
> In this case:
> 	- dom0_mem=2G
> 	- new VM to launch with size 16 GB.
> 	- Xen has 28 GB of free memory.
> 
> So clearly there's no need to try to balloon down dom0.. 

Right, so don't set autoballoon then.

> not-yet-implemented check/feature in xl, or a bug? 

Neither, it is the intended behaviour of xl autoballoon, this option
means exactly "take the required memory from dom0".

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Pasi Kärkkäinen]

On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> 
> > > This has been discussed at length on the list before, please check the
> > > archives.
> > > 
> > 
> > I tried googling quickly but I didn't find anything relevant..
> 
> I found
> http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> in a matter of seconds, then:
> http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> 
> > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to 
> > balloon down dom0 if the hypervisor already has enough free memory for the VM ? 
> > 
> > In this case:
> > 	- dom0_mem=2G
> > 	- new VM to launch with size 16 GB.
> > 	- Xen has 28 GB of free memory.
> > 
> > So clearly there's no need to try to balloon down dom0.. 
> 
> Right, so don't set autoballoon then.
> 

It's enabled as a default.. so many people hit this problem.


> > not-yet-implemented check/feature in xl, or a bug? 
> 
> Neither, it is the intended behaviour of xl autoballoon, this option
> means exactly "take the required memory from dom0".
> 

http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html

"autoballoon=BOOLEAN

    If disabled then xl will not attempt to reduce the amount of memory assigned to domain 0 in order to create free memory when starting a new domain. You should set this if you use the dom0_mem hypervisor command line to reduce the amount of memory given to domain 0 by default.

    Default: 1"


I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem hypervisor command line .." 
At least I understood that text in the opposite way..

Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, 
so most of the memory is now free in Xen.. and then you try to start a big VM ? 
Aren't you going to hit the same problem as in this thread? 


-- Pasi

Re: can not use all available memory

[Ian Campbell]

On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
> On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> > 
> > > > This has been discussed at length on the list before, please check the
> > > > archives.
> > > > 
> > > 
> > > I tried googling quickly but I didn't find anything relevant..
> > 
> > I found
> > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> > in a matter of seconds, then:
> > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> > 
> > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to 
> > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? 
> > > 
> > > In this case:
> > > 	- dom0_mem=2G
> > > 	- new VM to launch with size 16 GB.
> > > 	- Xen has 28 GB of free memory.
> > > 
> > > So clearly there's no need to try to balloon down dom0.. 
> > 
> > Right, so don't set autoballoon then.
> > 
> 
> It's enabled as a default.. so many people hit this problem.
> 
> 
> > > not-yet-implemented check/feature in xl, or a bug? 
> > 
> > Neither, it is the intended behaviour of xl autoballoon, this option
> > means exactly "take the required memory from dom0".
> > 
> 
> http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
> 
> "autoballoon=BOOLEAN
> 
>     If disabled then xl will not attempt to reduce the amount of
> memory assigned to domain 0 in order to create free memory when
> starting a new domain. You should set this if you use the dom0_mem
> hypervisor command line to reduce the amount of memory given to domain
> 0 by default.
> 
>     Default: 1"
> 
> 
> I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem hypervisor command line .." 
> At least I understood that text in the opposite way..

Yes, there should be a s/set/unset/ in there I think.

> Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, 
> so most of the memory is now free in Xen.. 

xl balloons dom0 back up when it destroy domains with autoballoon=1.

> and then you try to start a big VM ? 
> Aren't you going to hit the same problem as in this thread? 
> 
> 
> -- Pasi
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Dan Magenheimer]

> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> Sent: Friday, November 23, 2012 6:34 AM
> To: Pasi Kärkkäinen
> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
> Subject: Re: [Xen-devel] can not use all available memory
> 
> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> > >
> > > > > This has been discussed at length on the list before, please check the
> > > > > archives.
> > > > >
> > > >
> > > > I tried googling quickly but I didn't find anything relevant..
> > >
> > > I found
> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> > > in a matter of seconds, then:
> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> > >
> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
> > > >
> > > > In this case:
> > > > 	- dom0_mem=2G
> > > > 	- new VM to launch with size 16 GB.
> > > > 	- Xen has 28 GB of free memory.
> > > >
> > > > So clearly there's no need to try to balloon down dom0..
> > >
> > > Right, so don't set autoballoon then.
> > >
> >
> > It's enabled as a default.. so many people hit this problem.
> >
> >
> > > > not-yet-implemented check/feature in xl, or a bug?
> > >
> > > Neither, it is the intended behaviour of xl autoballoon, this option
> > > means exactly "take the required memory from dom0".
> > >
> >
> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
> >
> > "autoballoon=BOOLEAN
> >
> >     If disabled then xl will not attempt to reduce the amount of
> > memory assigned to domain 0 in order to create free memory when
> > starting a new domain. You should set this if you use the dom0_mem
> > hypervisor command line to reduce the amount of memory given to domain
> > 0 by default.
> >
> >     Default: 1"
> >
> >
> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
> hypervisor command line .."
> > At least I understood that text in the opposite way..
> 
> Yes, there should be a s/set/unset/ in there I think.
> 
> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
> > so most of the memory is now free in Xen..
> 
> xl balloons dom0 back up when it destroy domains with autoballoon=1.
> 
> > and then you try to start a big VM ?
> > Aren't you going to hit the same problem as in this thread?

Hmmm... this behavior and default may make sense for the Citrix
memory model (single machine, dom0 is "the user" so you want it
to always hold most of physical RAM not used by guests).  But
probably not so for a more cloud-like memory model.

Is there any (easy) way to force autoballoon=0 if the hypervisor
dom0_mem boot option is specified?  Or is there some reasonably
sane case I am missing where a user would want both dom0_mem
and autoballoon=1?

Oracle VM always boots servers with dom0_mem= set so (if/when
OVM switches to use xl), OVM will always set autoballoon off.
So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
providers I am trying to help here.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Sander Eikelenboom]

Monday, November 26, 2012, 5:58:28 PM, you wrote:

>> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
>> Sent: Friday, November 23, 2012 6:34 AM
>> To: Pasi Kärkkäinen
>> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
>> Subject: Re: [Xen-devel] can not use all available memory
>> 
>> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
>> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
>> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
>> > >
>> > > > > This has been discussed at length on the list before, please check the
>> > > > > archives.
>> > > > >
>> > > >
>> > > > I tried googling quickly but I didn't find anything relevant..
>> > >
>> > > I found
>> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
>> > > in a matter of seconds, then:
>> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
>> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
>> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
>> > >
>> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
>> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
>> > > >
>> > > > In this case:
>> > > >         - dom0_mem=2G
>> > > >         - new VM to launch with size 16 GB.
>> > > >         - Xen has 28 GB of free memory.
>> > > >
>> > > > So clearly there's no need to try to balloon down dom0..
>> > >
>> > > Right, so don't set autoballoon then.
>> > >
>> >
>> > It's enabled as a default.. so many people hit this problem.
>> >
>> >
>> > > > not-yet-implemented check/feature in xl, or a bug?
>> > >
>> > > Neither, it is the intended behaviour of xl autoballoon, this option
>> > > means exactly "take the required memory from dom0".
>> > >
>> >
>> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
>> >
>> > "autoballoon=BOOLEAN
>> >
>> >     If disabled then xl will not attempt to reduce the amount of
>> > memory assigned to domain 0 in order to create free memory when
>> > starting a new domain. You should set this if you use the dom0_mem
>> > hypervisor command line to reduce the amount of memory given to domain
>> > 0 by default.
>> >
>> >     Default: 1"
>> >
>> >
>> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
>> hypervisor command line .."
>> > At least I understood that text in the opposite way..
>> 
>> Yes, there should be a s/set/unset/ in there I think.
>> 
>> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
>> > so most of the memory is now free in Xen..
>> 
>> xl balloons dom0 back up when it destroy domains with autoballoon=1.
>> 
>> > and then you try to start a big VM ?
>> > Aren't you going to hit the same problem as in this thread?

> Hmmm... this behavior and default may make sense for the Citrix
> memory model (single machine, dom0 is "the user" so you want it
> to always hold most of physical RAM not used by guests).  But
> probably not so for a more cloud-like memory model.

> Is there any (easy) way to force autoballoon=0 if the hypervisor
> dom0_mem boot option is specified?  Or is there some reasonably
> sane case I am missing where a user would want both dom0_mem
> and autoballoon=1?

> Oracle VM always boots servers with dom0_mem= set so (if/when
> OVM switches to use xl), OVM will always set autoballoon off.
> So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
> providers I am trying to help here.



Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that:
- While the guest i was trying to start required less memory than was freely available (according to xentop) outside of dom0
- It would fail, because xl started to try to balloon dom0 down which failed.

--
Sander


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Dan Magenheimer]

> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> Subject: Re: [Xen-devel] can not use all available memory
> 
> Monday, November 26, 2012, 5:58:28 PM, you wrote:
> 
> >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> >> Sent: Friday, November 23, 2012 6:34 AM
> >> To: Pasi Kärkkäinen
> >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
> >> Subject: Re: [Xen-devel] can not use all available memory
> >>
> >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
> >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> >> > >
> >> > > > > This has been discussed at length on the list before, please check the
> >> > > > > archives.
> >> > > > >
> >> > > >
> >> > > > I tried googling quickly but I didn't find anything relevant..
> >> > >
> >> > > I found
> >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> >> > > in a matter of seconds, then:
> >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> >> > >
> >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
> >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
> >> > > >
> >> > > > In this case:
> >> > > >         - dom0_mem=2G
> >> > > >         - new VM to launch with size 16 GB.
> >> > > >         - Xen has 28 GB of free memory.
> >> > > >
> >> > > > So clearly there's no need to try to balloon down dom0..
> >> > >
> >> > > Right, so don't set autoballoon then.
> >> > >
> >> >
> >> > It's enabled as a default.. so many people hit this problem.
> >> >
> >> >
> >> > > > not-yet-implemented check/feature in xl, or a bug?
> >> > >
> >> > > Neither, it is the intended behaviour of xl autoballoon, this option
> >> > > means exactly "take the required memory from dom0".
> >> > >
> >> >
> >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
> >> >
> >> > "autoballoon=BOOLEAN
> >> >
> >> >     If disabled then xl will not attempt to reduce the amount of
> >> > memory assigned to domain 0 in order to create free memory when
> >> > starting a new domain. You should set this if you use the dom0_mem
> >> > hypervisor command line to reduce the amount of memory given to domain
> >> > 0 by default.
> >> >
> >> >     Default: 1"
> >> >
> >> >
> >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
> >> hypervisor command line .."
> >> > At least I understood that text in the opposite way..
> >>
> >> Yes, there should be a s/set/unset/ in there I think.
> >>
> >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
> >> > so most of the memory is now free in Xen..
> >>
> >> xl balloons dom0 back up when it destroy domains with autoballoon=1.
> >>
> >> > and then you try to start a big VM ?
> >> > Aren't you going to hit the same problem as in this thread?
> 
> > Hmmm... this behavior and default may make sense for the Citrix
> > memory model (single machine, dom0 is "the user" so you want it
> > to always hold most of physical RAM not used by guests).  But
> > probably not so for a more cloud-like memory model.
> 
> > Is there any (easy) way to force autoballoon=0 if the hypervisor
> > dom0_mem boot option is specified?  Or is there some reasonably
> > sane case I am missing where a user would want both dom0_mem
> > and autoballoon=1?
> 
> > Oracle VM always boots servers with dom0_mem= set so (if/when
> > OVM switches to use xl), OVM will always set autoballoon off.
> > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
> > providers I am trying to help here.
> 
> 
> 
> Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that:
> - While the guest i was trying to start required less memory than was freely available (according to
> xentop) outside of dom0
> - It would fail, because xl started to try to balloon dom0 down which failed.

Hi Sander --

I could be wrong (and I am confident someone will correct me if I am) but
I think this is because the Citrix memory model assumes there is an
inference-driven policy engine for load-balancing memory across competing
virtual machines ("squeezed").  I suspect squeezed returns unallocated
xen "free" memory to dom0.

IMHO, such policy engines are good for demos and so salespeople can
say "yes, this product supports memory overcommit" but Transcendent
Memory goes quite a bit further (albeit not for guests with proprietary
OS's).

Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Pasi Kärkkäinen]

On Mon, Nov 26, 2012 at 12:20:40PM -0800, Dan Magenheimer wrote:
> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > Subject: Re: [Xen-devel] can not use all available memory
> > 
> > Monday, November 26, 2012, 5:58:28 PM, you wrote:
> > 
> > >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> > >> Sent: Friday, November 23, 2012 6:34 AM
> > >> To: Pasi Kärkkäinen
> > >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
> > >> Subject: Re: [Xen-devel] can not use all available memory
> > >>
> > >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
> > >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> > >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> > >> > >
> > >> > > > > This has been discussed at length on the list before, please check the
> > >> > > > > archives.
> > >> > > > >
> > >> > > >
> > >> > > > I tried googling quickly but I didn't find anything relevant..
> > >> > >
> > >> > > I found
> > >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> > >> > > in a matter of seconds, then:
> > >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> > >> > >
> > >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
> > >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
> > >> > > >
> > >> > > > In this case:
> > >> > > >         - dom0_mem=2G
> > >> > > >         - new VM to launch with size 16 GB.
> > >> > > >         - Xen has 28 GB of free memory.
> > >> > > >
> > >> > > > So clearly there's no need to try to balloon down dom0..
> > >> > >
> > >> > > Right, so don't set autoballoon then.
> > >> > >
> > >> >
> > >> > It's enabled as a default.. so many people hit this problem.
> > >> >
> > >> >
> > >> > > > not-yet-implemented check/feature in xl, or a bug?
> > >> > >
> > >> > > Neither, it is the intended behaviour of xl autoballoon, this option
> > >> > > means exactly "take the required memory from dom0".
> > >> > >
> > >> >
> > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
> > >> >
> > >> > "autoballoon=BOOLEAN
> > >> >
> > >> >     If disabled then xl will not attempt to reduce the amount of
> > >> > memory assigned to domain 0 in order to create free memory when
> > >> > starting a new domain. You should set this if you use the dom0_mem
> > >> > hypervisor command line to reduce the amount of memory given to domain
> > >> > 0 by default.
> > >> >
> > >> >     Default: 1"
> > >> >
> > >> >
> > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
> > >> hypervisor command line .."
> > >> > At least I understood that text in the opposite way..
> > >>
> > >> Yes, there should be a s/set/unset/ in there I think.
> > >>
> > >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
> > >> > so most of the memory is now free in Xen..
> > >>
> > >> xl balloons dom0 back up when it destroy domains with autoballoon=1.
> > >>
> > >> > and then you try to start a big VM ?
> > >> > Aren't you going to hit the same problem as in this thread?
> > 
> > > Hmmm... this behavior and default may make sense for the Citrix
> > > memory model (single machine, dom0 is "the user" so you want it
> > > to always hold most of physical RAM not used by guests).  But
> > > probably not so for a more cloud-like memory model.
> > 
> > > Is there any (easy) way to force autoballoon=0 if the hypervisor
> > > dom0_mem boot option is specified?  Or is there some reasonably
> > > sane case I am missing where a user would want both dom0_mem
> > > and autoballoon=1?
> > 
> > > Oracle VM always boots servers with dom0_mem= set so (if/when
> > > OVM switches to use xl), OVM will always set autoballoon off.
> > > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
> > > providers I am trying to help here.
> > 
> > 
> > 
> > Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that:
> > - While the guest i was trying to start required less memory than was freely available (according to
> > xentop) outside of dom0
> > - It would fail, because xl started to try to balloon dom0 down which failed.
> 
> Hi Sander --
> 
> I could be wrong (and I am confident someone will correct me if I am) but
> I think this is because the Citrix memory model assumes there is an
> inference-driven policy engine for load-balancing memory across competing
> virtual machines ("squeezed").  I suspect squeezed returns unallocated
> xen "free" memory to dom0.
> 
> IMHO, such policy engines are good for demos and so salespeople can
> say "yes, this product supports memory overcommit" but Transcendent
> Memory goes quite a bit further (albeit not for guests with proprietary
> OS's).
> 

Afaik XenServer/XCP and XenClient both use dom0_mem= option for Xen.

-- Pasi

Re: can not use all available memory

[Dan Magenheimer]

> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Subject: Re: [Xen-devel] can not use all available memory
> 
> On Mon, Nov 26, 2012 at 12:20:40PM -0800, Dan Magenheimer wrote:
> > > From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
> > > Subject: Re: [Xen-devel] can not use all available memory
> > >
> > > Monday, November 26, 2012, 5:58:28 PM, you wrote:
> > >
> > > >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> > > >> Sent: Friday, November 23, 2012 6:34 AM
> > > >> To: Pasi Kärkkäinen
> > > >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
> > > >> Subject: Re: [Xen-devel] can not use all available memory
> > > >>
> > > >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
> > > >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
> > > >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
> > > >> > >
> > > >> > > > > This has been discussed at length on the list before, please check the
> > > >> > > > > archives.
> > > >> > > > >
> > > >> > > >
> > > >> > > > I tried googling quickly but I didn't find anything relevant..
> > > >> > >
> > > >> > > I found
> > > >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
> > > >> > > in a matter of seconds, then:
> > > >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
> > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
> > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
> > > >> > >
> > > >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
> > > >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
> > > >> > > >
> > > >> > > > In this case:
> > > >> > > >         - dom0_mem=2G
> > > >> > > >         - new VM to launch with size 16 GB.
> > > >> > > >         - Xen has 28 GB of free memory.
> > > >> > > >
> > > >> > > > So clearly there's no need to try to balloon down dom0..
> > > >> > >
> > > >> > > Right, so don't set autoballoon then.
> > > >> > >
> > > >> >
> > > >> > It's enabled as a default.. so many people hit this problem.
> > > >> >
> > > >> >
> > > >> > > > not-yet-implemented check/feature in xl, or a bug?
> > > >> > >
> > > >> > > Neither, it is the intended behaviour of xl autoballoon, this option
> > > >> > > means exactly "take the required memory from dom0".
> > > >> > >
> > > >> >
> > > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
> > > >> >
> > > >> > "autoballoon=BOOLEAN
> > > >> >
> > > >> >     If disabled then xl will not attempt to reduce the amount of
> > > >> > memory assigned to domain 0 in order to create free memory when
> > > >> > starting a new domain. You should set this if you use the dom0_mem
> > > >> > hypervisor command line to reduce the amount of memory given to domain
> > > >> > 0 by default.
> > > >> >
> > > >> >     Default: 1"
> > > >> >
> > > >> >
> > > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
> > > >> hypervisor command line .."
> > > >> > At least I understood that text in the opposite way..
> > > >>
> > > >> Yes, there should be a s/set/unset/ in there I think.
> > > >>
> > > >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
> > > >> > so most of the memory is now free in Xen..
> > > >>
> > > >> xl balloons dom0 back up when it destroy domains with autoballoon=1.
> > > >>
> > > >> > and then you try to start a big VM ?
> > > >> > Aren't you going to hit the same problem as in this thread?
> > >
> > > > Hmmm... this behavior and default may make sense for the Citrix
> > > > memory model (single machine, dom0 is "the user" so you want it
> > > > to always hold most of physical RAM not used by guests).  But
> > > > probably not so for a more cloud-like memory model.
> > >
> > > > Is there any (easy) way to force autoballoon=0 if the hypervisor
> > > > dom0_mem boot option is specified?  Or is there some reasonably
> > > > sane case I am missing where a user would want both dom0_mem
> > > > and autoballoon=1?
> > >
> > > > Oracle VM always boots servers with dom0_mem= set so (if/when
> > > > OVM switches to use xl), OVM will always set autoballoon off.
> > > > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
> > > > providers I am trying to help here.
> > >
> > >
> > >
> > > Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that:
> > > - While the guest i was trying to start required less memory than was freely available (according
> to
> > > xentop) outside of dom0
> > > - It would fail, because xl started to try to balloon dom0 down which failed.
> >
> > Hi Sander --
> >
> > I could be wrong (and I am confident someone will correct me if I am) but
> > I think this is because the Citrix memory model assumes there is an
> > inference-driven policy engine for load-balancing memory across competing
> > virtual machines ("squeezed").  I suspect squeezed returns unallocated
> > xen "free" memory to dom0.

I forgot... it is called Dynamic Memory Control (DMC), not squeezed
in the XenServer product.

> > IMHO, such policy engines are good for demos and so salespeople can
> > say "yes, this product supports memory overcommit" but Transcendent
> > Memory goes quite a bit further (albeit not for guests with proprietary
> > OS's).
> 
> Afaik XenServer/XCP and XenClient both use dom0_mem= option for Xen.

Hmmm... looks like you are right.

Dan

Re: can not use all available memory

[Tim Deegan]

At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote:
> > > I could be wrong (and I am confident someone will correct me if I am) but
> > > I think this is because the Citrix memory model assumes there is an
> > > inference-driven policy engine for load-balancing memory across competing
> > > virtual machines ("squeezed").  I suspect squeezed returns unallocated
> > > xen "free" memory to dom0.
> 
> I forgot... it is called Dynamic Memory Control (DMC), not squeezed
> in the XenServer product.

AFAIK XenServer uses dom0_mem= and doesn't balloon com0 after boot time.
The idea of ballooning all free memory into dom0 is a xl-ism, inherited
from xend, and not really a "Citrix" one.  It's useful if you've
installed xen on a machine where dom0 is otherwise your main OS, but not
particularly for a dedicated platform.

Tim.

Re: can not use all available memory

[Dan Magenheimer]

> From: Tim Deegan [mailto:tim@xen.org]
> Subject: Re: [Xen-devel] can not use all available memory
> 
> At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote:
> > > > I could be wrong (and I am confident someone will correct me if I am) but
> > > > I think this is because the Citrix memory model assumes there is an
> > > > inference-driven policy engine for load-balancing memory across competing
> > > > virtual machines ("squeezed").  I suspect squeezed returns unallocated
> > > > xen "free" memory to dom0.
> >
> > I forgot... it is called Dynamic Memory Control (DMC), not squeezed
> > in the XenServer product.
> 
> AFAIK XenServer uses dom0_mem= and doesn't balloon com0 after boot time.
> The idea of ballooning all free memory into dom0 is a xl-ism, inherited
> from xend, and not really a "Citrix" one.  It's useful if you've
> installed xen on a machine where dom0 is otherwise your main OS, but not
> particularly for a dedicated platform.

"inherited from xend"... was the autoballoon default the same in xend?
I don't recall ever turning it off manually and, when testing tmem,
I'm sure I would have had to.  Or maybe xend did use hypervisor free
memory before trying to autoballoon dom0?

Re: can not use all available memory

[William Pitcock]

Hi!

On Mon, Nov 26, 2012 at 3:47 PM, Dan Magenheimer
<dan.magenheimer@oracle.com> wrote:
>> From: Tim Deegan [mailto:tim@xen.org]
>> Subject: Re: [Xen-devel] can not use all available memory
>>
>> At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote:
>> > > > I could be wrong (and I am confident someone will correct me if I am) but
>> > > > I think this is because the Citrix memory model assumes there is an
>> > > > inference-driven policy engine for load-balancing memory across competing
>> > > > virtual machines ("squeezed").  I suspect squeezed returns unallocated
>> > > > xen "free" memory to dom0.
>> >
>> > I forgot... it is called Dynamic Memory Control (DMC), not squeezed
>> > in the XenServer product.
>>
>> AFAIK XenServer uses dom0_mem= and doesn't balloon com0 after boot time.
>> The idea of ballooning all free memory into dom0 is a xl-ism, inherited
>> from xend, and not really a "Citrix" one.  It's useful if you've
>> installed xen on a machine where dom0 is otherwise your main OS, but not
>> particularly for a dedicated platform.
>
> "inherited from xend"... was the autoballoon default the same in xend?
> I don't recall ever turning it off manually and, when testing tmem,
> I'm sure I would have had to.  Or maybe xend did use hypervisor free
> memory before trying to autoballoon dom0?

My Xen 3.4 cluster at my old job used available hypervisor memory
before ballooning from dom0 on the hosts which had that enabled.

So, I believe the behaviour of the xl toolstack is different than that
of xend.  My newer Xen 4.2 cluster at my new job has autoballoon
disabled across all nodes, however.  But we are having some problems
with running our dom0 environment without udev, which I need to figure
out at some point...

William

Re: can not use all available memory

[Ian Campbell]

On Mon, 2012-11-26 at 21:47 +0000, Dan Magenheimer wrote:
> > From: Tim Deegan [mailto:tim@xen.org]
> > Subject: Re: [Xen-devel] can not use all available memory
> > 
> > At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote:
> > > > > I could be wrong (and I am confident someone will correct me if I am) but
> > > > > I think this is because the Citrix memory model assumes there is an
> > > > > inference-driven policy engine for load-balancing memory across competing
> > > > > virtual machines ("squeezed").  I suspect squeezed returns unallocated
> > > > > xen "free" memory to dom0.
> > >
> > > I forgot... it is called Dynamic Memory Control (DMC), not squeezed
> > > in the XenServer product.
> > 
> > AFAIK XenServer uses dom0_mem= and doesn't balloon com0 after boot time.
> > The idea of ballooning all free memory into dom0 is a xl-ism, inherited
> > from xend, and not really a "Citrix" one.  It's useful if you've
> > installed xen on a machine where dom0 is otherwise your main OS, but not
> > particularly for a dedicated platform.
> 
> "inherited from xend"... was the autoballoon default the same in xend?
> I don't recall ever turning it off manually and, when testing tmem,
> I'm sure I would have had to.  Or maybe xend did use hypervisor free
> memory before trying to autoballoon dom0?

http://wiki.xen.org/wiki/XenBestPractices#Xen_dom0_dedicated_memory_and_preventing_dom0_memory_ballooning
indicates that you should have been using "(enable-dom0-ballooning no)"
when using dom0_mem= with xend. Perhaps the failure case is not so bad
with xend as with xl though, perhaps because xend has a central daemon
which lets it make different choices.

Ian.

Re: can not use all available memory

[Sander Eikelenboom]

Monday, November 26, 2012, 9:20:40 PM, you wrote:

>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> Subject: Re: [Xen-devel] can not use all available memory
>> 
>> Monday, November 26, 2012, 5:58:28 PM, you wrote:
>> 
>> >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
>> >> Sent: Friday, November 23, 2012 6:34 AM
>> >> To: Pasi Kärkkäinen
>> >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler
>> >> Subject: Re: [Xen-devel] can not use all available memory
>> >>
>> >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:
>> >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:
>> >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:
>> >> > >
>> >> > > > > This has been discussed at length on the list before, please check the
>> >> > > > > archives.
>> >> > > > >
>> >> > > >
>> >> > > > I tried googling quickly but I didn't find anything relevant..
>> >> > >
>> >> > > I found
>> >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i
>> >> > > in a matter of seconds, then:
>> >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html
>> >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html
>> >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html
>> >> > >
>> >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to
>> >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ?
>> >> > > >
>> >> > > > In this case:
>> >> > > >         - dom0_mem=2G
>> >> > > >         - new VM to launch with size 16 GB.
>> >> > > >         - Xen has 28 GB of free memory.
>> >> > > >
>> >> > > > So clearly there's no need to try to balloon down dom0..
>> >> > >
>> >> > > Right, so don't set autoballoon then.
>> >> > >
>> >> >
>> >> > It's enabled as a default.. so many people hit this problem.
>> >> >
>> >> >
>> >> > > > not-yet-implemented check/feature in xl, or a bug?
>> >> > >
>> >> > > Neither, it is the intended behaviour of xl autoballoon, this option
>> >> > > means exactly "take the required memory from dom0".
>> >> > >
>> >> >
>> >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html
>> >> >
>> >> > "autoballoon=BOOLEAN
>> >> >
>> >> >     If disabled then xl will not attempt to reduce the amount of
>> >> > memory assigned to domain 0 in order to create free memory when
>> >> > starting a new domain. You should set this if you use the dom0_mem
>> >> > hypervisor command line to reduce the amount of memory given to domain
>> >> > 0 by default.
>> >> >
>> >> >     Default: 1"
>> >> >
>> >> >
>> >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem
>> >> hypervisor command line .."
>> >> > At least I understood that text in the opposite way..
>> >>
>> >> Yes, there should be a s/set/unset/ in there I think.
>> >>
>> >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs,
>> >> > so most of the memory is now free in Xen..
>> >>
>> >> xl balloons dom0 back up when it destroy domains with autoballoon=1.
>> >>
>> >> > and then you try to start a big VM ?
>> >> > Aren't you going to hit the same problem as in this thread?
>> 
>> > Hmmm... this behavior and default may make sense for the Citrix
>> > memory model (single machine, dom0 is "the user" so you want it
>> > to always hold most of physical RAM not used by guests).  But
>> > probably not so for a more cloud-like memory model.
>> 
>> > Is there any (easy) way to force autoballoon=0 if the hypervisor
>> > dom0_mem boot option is specified?  Or is there some reasonably
>> > sane case I am missing where a user would want both dom0_mem
>> > and autoballoon=1?
>> 
>> > Oracle VM always boots servers with dom0_mem= set so (if/when
>> > OVM switches to use xl), OVM will always set autoballoon off.
>> > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service
>> > providers I am trying to help here.
>> 
>> 
>> 
>> Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that:
>> - While the guest i was trying to start required less memory than was freely available (according to
>> xentop) outside of dom0
>> - It would fail, because xl started to try to balloon dom0 down which failed.

> Hi Sander --

> I could be wrong (and I am confident someone will correct me if I am) but
> I think this is because the Citrix memory model assumes there is an
> inference-driven policy engine for load-balancing memory across competing
> virtual machines ("squeezed").  I suspect squeezed returns unallocated
> xen "free" memory to dom0.

> IMHO, such policy engines are good for demos and so salespeople can
> say "yes, this product supports memory overcommit" but Transcendent
> Memory goes quite a bit further (albeit not for guests with proprietary
> OS's).

> Dan

Yes well my novice assumption was that if i have say 8GB of mem, and set dom0 max mem to say 1GB.
That auto ballooning would not interfere unless the total of guests i start go across the 7GB.

And it seems when i go under the, say about 1GB of free mem available outside of dom0, some ballooning code seems to interfere with creating guests.
Most of the time, the first attempt to start a guests then fails, a second attempt directly thereafter succeeds.

Switching off autoballoon in xl.conf makes it all work fine. Not a big problem in itself, but as Pasi said perhaps not quite clear and intuitive for everyone.

So your suggestion of disabling autoballooning when dom0_mem is set seems appealing. Although there perhaps is some use case in limiting dom0 to under 4GB for some drivers, but still leave the capability to autoballoon down ?

(and yes i don't like all the automatic voodoo in overcommiting memory and "hope for the best" all the algorithms involved can manage it.
I like things to predictably fail (and fail direct if possible), so good measures can be taken. A bit old-fashioned perhaps.

--
Sander





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can not use all available memory

[Alexander Bienzeisler]

As Ian said, i did not disable autoballooning. It works with 
autoballooning=0.

Thanks everyone and sorry for wasting your time.

Am 23.11.2012 13:33, schrieb Ian Campbell:
> On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote:
>> On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:
>>> This question belongs on xen-users.
>>>
>>> On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:
>>>> Hello,
>>>>
>>>> i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory =
>>>> 16384) with free_memory : 28969 but got an error:
>>>>
>>>> libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for
>>>> dom0 is below the minimum threshold
>>> The clue is in this message -- you cannot safely balloon down a guest
>>> (including dom0) to an arbitrarily small value relative to its initial
>>> size. The kernel will have certain fixed datastructures etc which are
>>> sized according to the maximum amount of RAM at start of day.
>>>
>>> If you were to balloon down too low then you would run out of RAM and
>>> the OOM killer would step in and start killing stuff, so xl includes a
>>> safety catch which prevents this.
>>>
>>> If you have a huge host and want a small dom0 use the dom0_mem option
>>> and disable xl's autoballoon option or whatever the equivalent xend
>>> option is.
>>>
>> I think he was already using dom0_mem. He had 28969 free in Xen as
>> reported by "xl info".
> Then he's missed disabling autoballoon.
>
>> -- Pasi
>>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel