Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[James Dingwall]

Hi,

I am having some memory ballooning problems which only seem to have 
appeared since upgrading to Xen 4.3.  In short if I set the domU 
configuration file as:

memory = 512
maxmem = 1024

Then even under memory pressure the guest domain does not balloon past 
the value of the 'memory' parameter.

xl info shows plenty of available memory in dom0:
total_memory           : 32767
free_memory            : 12445


xl list -l on the domain shows:
     "max_memkb": 1048576,
     "target_memkb": 524288,

xl top shows:
            NAME        guestdom
           STATE        --b---
        CPU(sec)        79
          CPU(%)        1.2
          MEM(k)        343116
          MEM(%)        1.0
       MAXMEM(k)        525312
       MAXMEM(%)        1.6
           VCPUS        2
            NETS        0
        NETTX(k)        0
        NETRX(k)        0
            VBDS        2
          VBD_OO        0
          VBD_RD        8639
          VBD_WR        129666
       VBD_RSECT        529846
       VBD_WSECT        1622593
            SSID        0


In the domU dmesg it seems as though the guest has some idea that it has 
a 'maxmem' size of 1024Mb:

[    0.000000] Memory: 408932K/1048188K available (4562K kernel code, 
541K rwdata, 1740K rodata, 924K init, 644K bss, 639256K reserved)


but /proc/iomap seems to have reduced it to 'memory':
00000000-00000fff : reserved
00001000-0009ffff : System RAM
000a0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
  01000000-01474c38 : Kernel code
  01474c39-016b167f : Kernel data
  017a0000-01840fff : Kernel bss
20000000-d7feffff : Unusable memory
d7ff0000-d7ffdfff : ACPI Tables
d7ffe000-d7ffffff : ACPI Non-volatile Storage
100000000-11fffffff : System RAM


After running 
xl mem-max guestdom 1024

xl list -l remains
   "max_memkb": 1048576,
   "target_memkb": 524288,

xl top has changed:
            NAME        guestdom
           STATE        --b---
        CPU(sec)        98
          CPU(%)        0.9
          MEM(k)        343116
          MEM(%)        1.0
       MAXMEM(k)        1049600
       MAXMEM(%)        3.1
           VCPUS        2
            NETS        0
        NETTX(k)        0
        NETRX(k)        0
            VBDS        2
          VBD_OO        0
          VBD_RD        9068
          VBD_WR        130257
       VBD_RSECT        549640
       VBD_WSECT        1628666
            SSID        0


Although /proc/iomem doesn't change the guest has no problem ballooning 
up to the 'maxmem' value.  It seems that this is a bug with xl not 
setting the appropriate paramter somewhere when the domain is created.

The domU and dom0 kernel version is 3.11.6, Xen is 4.3.  Please let me 
know if examinging any other settings before/after the xl mem-max 
command is run would be helpful.

Thanks,
James

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Wei Liu]

On Tue, Oct 29, 2013 at 10:26:00AM +0000, James Dingwall wrote:
> Hi,
> 
> I am having some memory ballooning problems which only seem to have 
> appeared since upgrading to Xen 4.3.  In short if I set the domU 
> configuration file as:
> 
> memory = 512
> maxmem = 1024
> 
> Then even under memory pressure the guest domain does not balloon past 
> the value of the 'memory' parameter.
> 
> xl info shows plenty of available memory in dom0:
> total_memory           : 32767
> free_memory            : 12445
> 
> 
> xl list -l on the domain shows:
>      "max_memkb": 1048576,
>      "target_memkb": 524288,
> 

This is parsed from your config file so they should always look OK to
you.

Does the following patch help?

Git blame tells me the change to set maxmem to target_memkb was
introduced 4 years ago so I suspect there's reason to do that. If we
cannot fix it here we need to insert the corresponding call later.

Wei.
----8<---
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 356f920..fb7965d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -235,7 +235,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
     libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
     libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus, &info->cpumap);
 
-    xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + LIBXL_MAXMEM_CONSTANT);
+    xc_domain_setmaxmem(ctx->xch, domid, info->max_memkb + LIBXL_MAXMEM_CONSTANT);
     xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
     state->store_domid = xs_domid ? atoi(xs_domid) : 0;
     free(xs_domid);

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Andrew Cooper]

On 29/10/13 10:59, Wei Liu wrote:
> On Tue, Oct 29, 2013 at 10:26:00AM +0000, James Dingwall wrote:
>> Hi,
>>
>> I am having some memory ballooning problems which only seem to have 
>> appeared since upgrading to Xen 4.3.  In short if I set the domU 
>> configuration file as:
>>
>> memory = 512
>> maxmem = 1024
>>
>> Then even under memory pressure the guest domain does not balloon past 
>> the value of the 'memory' parameter.
>>
>> xl info shows plenty of available memory in dom0:
>> total_memory           : 32767
>> free_memory            : 12445
>>
>>
>> xl list -l on the domain shows:
>>      "max_memkb": 1048576,
>>      "target_memkb": 524288,
>>
> This is parsed from your config file so they should always look OK to
> you.
>
> Does the following patch help?
>
> Git blame tells me the change to set maxmem to target_memkb was
> introduced 4 years ago so I suspect there's reason to do that. If we
> cannot fix it here we need to insert the corresponding call later.

When setmaxmem sets a limit lower than current, the domain can strictly
only balloon down until it is equal to or under the new limit.

Performing a setmaxmem hypercall to info->target_memkb will prevent from
domain from ballooning down then back up a bit, when it has been asked
to balloon down a lot; i.e. it must strictly balloon down to the limit
it has been given.

Therefore, I am not sure this change is valid.

~Andrew

>
> Wei.
> ----8<---
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index 356f920..fb7965d 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -235,7 +235,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
>      libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
>      libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus, &info->cpumap);
>  
> -    xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + LIBXL_MAXMEM_CONSTANT);
> +    xc_domain_setmaxmem(ctx->xch, domid, info->max_memkb + LIBXL_MAXMEM_CONSTANT);
>      xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
>      state->store_domid = xs_domid ? atoi(xs_domid) : 0;
>      free(xs_domid);
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Wei Liu]

On Tue, Oct 29, 2013 at 11:06:01AM +0000, Andrew Cooper wrote:
> On 29/10/13 10:59, Wei Liu wrote:
> > On Tue, Oct 29, 2013 at 10:26:00AM +0000, James Dingwall wrote:
> >> Hi,
> >>
> >> I am having some memory ballooning problems which only seem to have 
> >> appeared since upgrading to Xen 4.3.  In short if I set the domU 
> >> configuration file as:
> >>
> >> memory = 512
> >> maxmem = 1024
> >>
> >> Then even under memory pressure the guest domain does not balloon past 
> >> the value of the 'memory' parameter.
> >>
> >> xl info shows plenty of available memory in dom0:
> >> total_memory           : 32767
> >> free_memory            : 12445
> >>
> >>
> >> xl list -l on the domain shows:
> >>      "max_memkb": 1048576,
> >>      "target_memkb": 524288,
> >>
> > This is parsed from your config file so they should always look OK to
> > you.
> >
> > Does the following patch help?
> >
> > Git blame tells me the change to set maxmem to target_memkb was
> > introduced 4 years ago so I suspect there's reason to do that. If we
> > cannot fix it here we need to insert the corresponding call later.
> 
> When setmaxmem sets a limit lower than current, the domain can strictly
> only balloon down until it is equal to or under the new limit.
> 
> Performing a setmaxmem hypercall to info->target_memkb will prevent from
> domain from ballooning down then back up a bit, when it has been asked
> to balloon down a lot; i.e. it must strictly balloon down to the limit
> it has been given.
> 
> Therefore, I am not sure this change is valid.
> 
> ~Andrew
> 

Looking at the changeset 9905ac that introduced target_memkb,
target_memkb was always set to max_memkb, so my guess is that the
purpose was still to use max_memkb to be max memory limit.

Probably later changes to that portion of code altered that behavior.

The use of target_memkb in xc_domain_setmaxmem looks logically wrong to
me anyway...

Stefano, thoughts?

Wei.

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Konrad Rzeszutek Wilk]

On Tue, Oct 29, 2013 at 12:08:04PM +0000, Wei Liu wrote:
> On Tue, Oct 29, 2013 at 11:06:01AM +0000, Andrew Cooper wrote:
> > On 29/10/13 10:59, Wei Liu wrote:
> > > On Tue, Oct 29, 2013 at 10:26:00AM +0000, James Dingwall wrote:
> > >> Hi,
> > >>
> > >> I am having some memory ballooning problems which only seem to have 
> > >> appeared since upgrading to Xen 4.3.  In short if I set the domU 
> > >> configuration file as:
> > >>
> > >> memory = 512
> > >> maxmem = 1024
> > >>
> > >> Then even under memory pressure the guest domain does not balloon past 
> > >> the value of the 'memory' parameter.
> > >>
> > >> xl info shows plenty of available memory in dom0:
> > >> total_memory           : 32767
> > >> free_memory            : 12445
> > >>
> > >>
> > >> xl list -l on the domain shows:
> > >>      "max_memkb": 1048576,
> > >>      "target_memkb": 524288,
> > >>
> > > This is parsed from your config file so they should always look OK to
> > > you.
> > >
> > > Does the following patch help?
> > >
> > > Git blame tells me the change to set maxmem to target_memkb was
> > > introduced 4 years ago so I suspect there's reason to do that. If we
> > > cannot fix it here we need to insert the corresponding call later.
> > 
> > When setmaxmem sets a limit lower than current, the domain can strictly
> > only balloon down until it is equal to or under the new limit.
> > 
> > Performing a setmaxmem hypercall to info->target_memkb will prevent from
> > domain from ballooning down then back up a bit, when it has been asked
> > to balloon down a lot; i.e. it must strictly balloon down to the limit
> > it has been given.
> > 
> > Therefore, I am not sure this change is valid.
> > 
> > ~Andrew
> > 
> 
> Looking at the changeset 9905ac that introduced target_memkb,
> target_memkb was always set to max_memkb, so my guess is that the
> purpose was still to use max_memkb to be max memory limit.
> 
> Probably later changes to that portion of code altered that behavior.
> 
> The use of target_memkb in xc_domain_setmaxmem looks logically wrong to
> me anyway...
> 
> Stefano, thoughts?

Lets also include Daniel in this - as he was trying to fix this in the past.

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Daniel Kiper]

On Tue, Oct 29, 2013 at 10:49:35AM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 29, 2013 at 12:08:04PM +0000, Wei Liu wrote:
> > On Tue, Oct 29, 2013 at 11:06:01AM +0000, Andrew Cooper wrote:
> > > On 29/10/13 10:59, Wei Liu wrote:
> > > > On Tue, Oct 29, 2013 at 10:26:00AM +0000, James Dingwall wrote:
> > > >> Hi,
> > > >>
> > > >> I am having some memory ballooning problems which only seem to have
> > > >> appeared since upgrading to Xen 4.3.  In short if I set the domU
> > > >> configuration file as:
> > > >>
> > > >> memory = 512
> > > >> maxmem = 1024
> > > >>
> > > >> Then even under memory pressure the guest domain does not balloon past
> > > >> the value of the 'memory' parameter.
> > > >>
> > > >> xl info shows plenty of available memory in dom0:
> > > >> total_memory           : 32767
> > > >> free_memory            : 12445
> > > >>
> > > >>
> > > >> xl list -l on the domain shows:
> > > >>      "max_memkb": 1048576,
> > > >>      "target_memkb": 524288,
> > > >>
> > > > This is parsed from your config file so they should always look OK to
> > > > you.
> > > >
> > > > Does the following patch help?
> > > >
> > > > Git blame tells me the change to set maxmem to target_memkb was
> > > > introduced 4 years ago so I suspect there's reason to do that. If we
> > > > cannot fix it here we need to insert the corresponding call later.
> > >
> > > When setmaxmem sets a limit lower than current, the domain can strictly
> > > only balloon down until it is equal to or under the new limit.
> > >
> > > Performing a setmaxmem hypercall to info->target_memkb will prevent from
> > > domain from ballooning down then back up a bit, when it has been asked
> > > to balloon down a lot; i.e. it must strictly balloon down to the limit
> > > it has been given.
> > >
> > > Therefore, I am not sure this change is valid.
> > >
> > > ~Andrew
> > >
> >
> > Looking at the changeset 9905ac that introduced target_memkb,
> > target_memkb was always set to max_memkb, so my guess is that the
> > purpose was still to use max_memkb to be max memory limit.
> >
> > Probably later changes to that portion of code altered that behavior.
> >
> > The use of target_memkb in xc_domain_setmaxmem looks logically wrong to
> > me anyway...
> >
> > Stefano, thoughts?
>
> Lets also include Daniel in this - as he was trying to fix this in the past.

As I remember this behavior is by design. However, this is not in line with
xm behavior which sometimes makes a problem. I have discussed this with IanJ
and IanC (both CC-ed here) once. I have proposed some patch series to fix that
issue but later I was not able to continue work on it. If you wish I could
repost them once again with fixes requested by IanJ and IanC. You could also
find latest version of patches here:

http://lists.xenproject.org/archives/html/xen-devel/2013-04/msg03072.html

Daniel

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Wei Liu]

On Tue, Oct 29, 2013 at 07:02:06PM +0100, Daniel Kiper wrote:
[...]
> > >
> > > Stefano, thoughts?
> >
> > Lets also include Daniel in this - as he was trying to fix this in the past.
> 
> As I remember this behavior is by design. However, this is not in line with
> xm behavior which sometimes makes a problem. I have discussed this with IanJ
> and IanC (both CC-ed here) once. I have proposed some patch series to fix that
> issue but later I was not able to continue work on it. If you wish I could
> repost them once again with fixes requested by IanJ and IanC. You could also
> find latest version of patches here:
> 
> http://lists.xenproject.org/archives/html/xen-devel/2013-04/msg03072.html
> 

Oh, sure. If you have something new then please do. ;-)

AFAICT from the latest series, the real meat that fixes this problem is
in patch 2, which got no reply. Does that mean no objection was raised
on that approach?

Wei.

> Daniel

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[Daniel Kiper]

On Wed, Oct 30, 2013 at 11:39:51AM +0000, Wei Liu wrote:
> On Tue, Oct 29, 2013 at 07:02:06PM +0100, Daniel Kiper wrote:
> [...]
> > > >
> > > > Stefano, thoughts?
> > >
> > > Lets also include Daniel in this - as he was trying to fix this in the past.
> >
> > As I remember this behavior is by design. However, this is not in line with
> > xm behavior which sometimes makes a problem. I have discussed this with IanJ
> > and IanC (both CC-ed here) once. I have proposed some patch series to fix that
> > issue but later I was not able to continue work on it. If you wish I could
> > repost them once again with fixes requested by IanJ and IanC. You could also
> > find latest version of patches here:
> >
> > http://lists.xenproject.org/archives/html/xen-devel/2013-04/msg03072.html
> >
>
> Oh, sure. If you have something new then please do. ;-)
>
> AFAICT from the latest series, the real meat that fixes this problem is
> in patch 2, which got no reply. Does that mean no objection was raised

IIRC patch number 1 is much more important in your case. If you can test
it (and others if it is possible) then it will be nice. However, patches
may require some tweaks to be applied on current unstable tree.

> on that approach?

In general there were only small issues left. I will fix them and repost
when I will be sure that they solve your problems.

Daniel

Re: Bug - Xen 4.3 - xl ignores maxmem setting in domU config file

[James Dingwall]

On Tue, Oct 29, 2013 at 10:59:21AM +0000, Wei Liu wrote:
> On Tue, Oct 29, 2013 at 10:26:00AM +0000, James Dingwall wrote:
> > Hi,
> > 
> > I am having some memory ballooning problems which only seem to have 
> > appeared since upgrading to Xen 4.3.  In short if I set the domU 
> > configuration file as:
> > 
> > memory = 512
> > maxmem = 1024
> > 
> > Then even under memory pressure the guest domain does not balloon past 
> > the value of the 'memory' parameter.
> > 
> > xl info shows plenty of available memory in dom0:
> > total_memory           : 32767
> > free_memory            : 12445
> > 
> > 
> > xl list -l on the domain shows:
> >      "max_memkb": 1048576,
> >      "target_memkb": 524288,
> > 
> 
> This is parsed from your config file so they should always look OK to
> you.
> 
> Does the following patch help?
> 
> Git blame tells me the change to set maxmem to target_memkb was
> introduced 4 years ago so I suspect there's reason to do that. If we
> cannot fix it here we need to insert the corresponding call later.
> 
> Wei.
> ----8<---
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index 356f920..fb7965d 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -235,7 +235,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
>      libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap);
>      libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus, &info->cpumap);
>  
> -    xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + LIBXL_MAXMEM_CONSTANT);
> +    xc_domain_setmaxmem(ctx->xch, domid, info->max_memkb + LIBXL_MAXMEM_CONSTANT);
>      xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
>      state->store_domid = xs_domid ? atoi(xs_domid) : 0;
>      free(xs_domid);
> 

This patch solves the problem for me where maxmem in config file is not 
honoured during domain creation.  With it applied xl top immediately 
shows the correct value in the MAXMEM column.

I can see that xl_cmdimpl.c safely sets max_memkb for the case where the 
configuration file does not define maxmem.

Thanks,
James



-- 
------------------------------------------------------------------------
James Dingwall
e: james@dingwall.me.uk
w: http://www.dingwall.me.uk/
------------------------------------------------------------------------