[PATCH] Fix e820 mapping limit

[PATCH] Fix e820 mapping limit

[Daniel P. Berrange]

[-- Attachment #1: Type: text/plain, Size: 2906 bytes --]

The changeset '12803:df5fa63490f4da7b65c56087a68783dbcb7944f8' added a new
hypercall XENMEM_set_memory_map for specifying the e820 mapping. There was
a small bug in the userspace side of this change though - the e820 mapping
was specified based on the 'memory_static_min' domain info parameter which
means the memory map size is clamped to the value 'memory' config option.

What we actually want is to setup the e820 map limit based on 'maxmem'
config option, so that we can balloon up the guest to its max limit.

 eg
  If we boot with 'memory=400' and 'maxmem=800', the e820 map should be
  set to 800 MB, and the guest should allocate  only 400 MB at startup.
  It should then be possible to use 'xm mem-set' after boot to assgin
  upto this 800 MB.

Thus, the attached patch fixed the image.py class to setup the e820 mapping
based on 'memory_static_max' which is in turn based on 'maxmem'. Booting
with this patch I can verify that the guest kernel sees a 800 MB e820 map,
so this fixes the initial problem.

There appears to be a second issue somewhere in the stack though, because
even if do 'xm mem-set <domid> 600', while the ballon driver in the guest
will see the 600 MB target, it will still never try to allocate its 
increased target.

eg

  # grep mem /etc/xen/demo 
  memory = 410
  maxmem = 800
  # xm create demo

In the guest  'dmesg' shows:

  BIOS-provided physical RAM map:
   Xen: 0000000000000000 - 0000000032800000 (usable)

Back on the host:

  # xm list --long demo | grep mem
    (memory 410)
    (maxmem 800)
    (shadow_memory 0)
    (memory_dynamic_min 410)
    (memory_dynamic_max 410)

  # xm mem-set demo 600

  # xm list --long demo | grep mem
    (memory 600)
    (maxmem 800)
    (shadow_memory 0)
    (memory_dynamic_min 410)
    (memory_dynamic_max 410)

Looking at balloon driver in guest:

  $ cat /tmp/balloon 
  Current allocation:   419840 kB
  Requested target:     614400 kB
  Low-mem balloon:      407552 kB
  High-mem balloon:          0 kB
  Driver pages:              0 kB
  Xen hard limit:       419840 kB

And the memory target spceified in xenstore:

  # xenstore-ls | grep target
    target = "614400"


So, for some reason even though the guest e820 map is correct, and the memory
target got increased to 600 MB, the balloon driver is not actually ever trying
to increase its allocation.   Do we need to also fixup the memory_dynamic_min/max
parameters in XenD too ?  xm mem-set only affects memory_static_min

Anyway, attaching the patch to fix e820 map limits.

  Signed-off-by: Daniel Berrange <berrange@redhat.com

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

[-- Attachment #2: xen-mem-e820-size.patch --]
[-- Type: text/plain, Size: 2461 bytes --]

diff -r f5121d001d1a tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Sat Dec 09 16:29:52 2006 +0000
+++ b/tools/python/xen/xend/XendDomainInfo.py	Fri Dec 08 17:33:22 2006 -0500
@@ -895,6 +895,10 @@ class XendDomainInfo:
     def getMemoryTarget(self):
         """Get this domain's target memory size, in KB."""
         return self.info['memory_static_min'] * 1024
+
+    def getMemoryMaximum(self):
+        """Get this domain's maximum memory size, in KB."""
+        return self.info['memory_static_max'] * 1024
 
     def getResume(self):
         return str(self._resume)
diff -r f5121d001d1a tools/python/xen/xend/image.py
--- a/tools/python/xen/xend/image.py	Sat Dec 09 16:29:52 2006 +0000
+++ b/tools/python/xen/xend/image.py	Mon Dec 11 16:37:12 2006 -0500
@@ -144,6 +144,14 @@ class ImageHandler:
         architecture- or image-specific code may override this to
         add headroom where necessary."""
         return self.getRequiredAvailableMemory(self.vm.getMemoryTarget())
+
+    def getRequiredMaximumReservation(self):
+        """@param mem_kb The maximum possible memory, in KiB.
+        @return The corresponding required amount of memory to be free, also
+        in KiB. This is normally the same as getRequiredAvailableMemory, but
+        architecture- or image-specific code may override this to
+        add headroom where necessary."""
+        return self.getRequiredAvailableMemory(self.vm.getMemoryMaximum())
 
     def getRequiredShadowMemory(self, shadow_mem_kb, maxmem_kb):
         """@param shadow_mem_kb The configured shadow memory, in KiB.
@@ -539,6 +547,9 @@ class X86_HVM_ImageHandler(HVMImageHandl
     def getRequiredInitialReservation(self):
         return self.vm.getMemoryTarget()
 
+    def getRequiredMaximumReservation(self):
+        return self.vm.getMemoryMaximum()
+
     def getRequiredShadowMemory(self, shadow_mem_kb, maxmem_kb):
         # 256 pages (1MB) per vcpu,
         # plus 1 page per MiB of RAM for the P2M map,
@@ -553,7 +564,7 @@ class X86_Linux_ImageHandler(LinuxImageH
     def buildDomain(self):
         # set physical mapping limit
         # add an 8MB slack to balance backend allocations.
-        mem_kb = self.getRequiredInitialReservation() + (8 * 1024)
+        mem_kb = self.getRequiredMaximumReservation() + (8 * 1024)
         xc.domain_set_memmap_limit(self.vm.getDomid(), mem_kb)
         return LinuxImageHandler.buildDomain(self)
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Fix e820 mapping limit

[Daniel P. Berrange]

[-- Attachment #1: Type: text/plain, Size: 2700 bytes --]

On Tue, Dec 12, 2006 at 06:25:25PM +0000, Daniel P. Berrange wrote:
> The changeset '12803:df5fa63490f4da7b65c56087a68783dbcb7944f8' added a new
> hypercall XENMEM_set_memory_map for specifying the e820 mapping. There was
> a small bug in the userspace side of this change though - the e820 mapping
> was specified based on the 'memory_static_min' domain info parameter which
> means the memory map size is clamped to the value 'memory' config option.
> 
> What we actually want is to setup the e820 map limit based on 'maxmem'
> config option, so that we can balloon up the guest to its max limit.
> 
>  eg
>   If we boot with 'memory=400' and 'maxmem=800', the e820 map should be
>   set to 800 MB, and the guest should allocate  only 400 MB at startup.
>   It should then be possible to use 'xm mem-set' after boot to assgin
>   upto this 800 MB.
> 
> Thus, the attached patch fixed the image.py class to setup the e820 mapping
> based on 'memory_static_max' which is in turn based on 'maxmem'. Booting
> with this patch I can verify that the guest kernel sees a 800 MB e820 map,
> so this fixes the initial problem.
> 
> There appears to be a second issue somewhere in the stack though, because
> even if do 'xm mem-set <domid> 600', while the ballon driver in the guest
> will see the 600 MB target, it will still never try to allocate its 
> increased target.

Turns out this was a bug introduced in the refactoring of memory config
parameters from Xen API work. The memory & memmax parameters were getting
initialized the wrong way around. With this fixed & the e820 map fix, 
memory ballooning works perfectly:

  # xm create demo
  Using config file "/etc/xen/demo".
  Going to boot Fedora Core (2.6.18-1.2769.2.2.el5xen)
    kernel: /vmlinuz-2.6.18-1.2769.2.2.el5xen
    initrd: /initrd-2.6.18-1.2769.2.2.el5xen.img
  Started domain demo
  # xm list demo
  Name                                      ID   Mem VCPUs      State   Time(s)
  demo                                      16   410     2     -b----      0.4
  # xm mem-set demo 600
  # xm list demo
  Name                                      ID   Mem VCPUs      State   Time(s)
  demo                                      16   600     2     -b----      4.2


Attaching an updated patch which fixes both the memory ballooning issues
in one go

  Signed-off-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

[-- Attachment #2: xen-mem-e820-size-2.patch --]
[-- Type: text/plain, Size: 3153 bytes --]

diff -r f5121d001d1a tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Sat Dec 09 16:29:52 2006 +0000
+++ b/tools/python/xen/xend/XendDomainInfo.py	Tue Dec 12 14:06:25 2006 -0500
@@ -896,6 +896,10 @@ class XendDomainInfo:
         """Get this domain's target memory size, in KB."""
         return self.info['memory_static_min'] * 1024
 
+    def getMemoryMaximum(self):
+        """Get this domain's maximum memory size, in KB."""
+        return self.info['memory_static_max'] * 1024
+
     def getResume(self):
         return str(self._resume)
 
@@ -1363,9 +1367,9 @@ class XendDomainInfo:
             # Use architecture- and image-specific calculations to determine
             # the various headrooms necessary, given the raw configured
             # values. maxmem, memory, and shadow are all in KiB.
+            memory = self.image.getRequiredAvailableMemory(
+                self.info['memory_static_min'] * 1024)
             maxmem = self.image.getRequiredAvailableMemory(
-                self.info['memory_static_min'] * 1024)
-            memory = self.image.getRequiredAvailableMemory(
                 self.info['memory_static_max'] * 1024)
             shadow = self.image.getRequiredShadowMemory(
                 self.info['shadow_memory'] * 1024,
diff -r f5121d001d1a tools/python/xen/xend/image.py
--- a/tools/python/xen/xend/image.py	Sat Dec 09 16:29:52 2006 +0000
+++ b/tools/python/xen/xend/image.py	Mon Dec 11 16:37:12 2006 -0500
@@ -144,6 +144,14 @@ class ImageHandler:
         architecture- or image-specific code may override this to
         add headroom where necessary."""
         return self.getRequiredAvailableMemory(self.vm.getMemoryTarget())
+
+    def getRequiredMaximumReservation(self):
+        """@param mem_kb The maximum possible memory, in KiB.
+        @return The corresponding required amount of memory to be free, also
+        in KiB. This is normally the same as getRequiredAvailableMemory, but
+        architecture- or image-specific code may override this to
+        add headroom where necessary."""
+        return self.getRequiredAvailableMemory(self.vm.getMemoryMaximum())
 
     def getRequiredShadowMemory(self, shadow_mem_kb, maxmem_kb):
         """@param shadow_mem_kb The configured shadow memory, in KiB.
@@ -539,6 +547,9 @@ class X86_HVM_ImageHandler(HVMImageHandl
     def getRequiredInitialReservation(self):
         return self.vm.getMemoryTarget()
 
+    def getRequiredMaximumReservation(self):
+        return self.vm.getMemoryMaximum()
+
     def getRequiredShadowMemory(self, shadow_mem_kb, maxmem_kb):
         # 256 pages (1MB) per vcpu,
         # plus 1 page per MiB of RAM for the P2M map,
@@ -553,7 +564,7 @@ class X86_Linux_ImageHandler(LinuxImageH
     def buildDomain(self):
         # set physical mapping limit
         # add an 8MB slack to balance backend allocations.
-        mem_kb = self.getRequiredInitialReservation() + (8 * 1024)
+        mem_kb = self.getRequiredMaximumReservation() + (8 * 1024)
         xc.domain_set_memmap_limit(self.vm.getDomid(), mem_kb)
         return LinuxImageHandler.buildDomain(self)
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel