[PATCH] Make ballooning work with maxmem > mem (i386 version)

[PATCH] Make ballooning work with maxmem > mem (i386 version)

[Glauber de Oliveira Costa]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

Trying to start a guest with maxmem > mem and then balloon up to a value
greater than mem is currently failing. This have been already discovered
(patch sent some days ago) for x86_64. i386 suffers from the same
problem.

This patch fixes it.

-- 
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"

[-- Attachment #2: xen-balloon-i386.patch --]
[-- Type: text/plain, Size: 1456 bytes --]

# HG changeset patch
# User Glauber de Oliveira Costa <gcosta@redhat.com>
# Node ID 2e2e6a3f9b2d0ad21c666dfbdc075f918e0bc7be
# Parent  20204db0891b0b7c10959822e3283656c3600500
[LINUX] Extend physical mapping to maxmem instead of mem - i386 version

As currently physical mapping only reaches the initial reservation,
we're unable to balloon up to more than mem (even when maxmem > mem)
in any situation. This have already been found out to be broken and fixed
in x86_64

Signed-off-by: Glauber de Oliveira Costa <gcosta@redhat.com>

diff -r 20204db0891b -r 2e2e6a3f9b2d linux-2.6-xen-sparse/include/asm-i386/mach-xen/setup_arch_post.h
--- a/linux-2.6-xen-sparse/include/asm-i386/mach-xen/setup_arch_post.h	Thu Nov  2 18:52:04 2006
+++ b/linux-2.6-xen-sparse/include/asm-i386/mach-xen/setup_arch_post.h	Fri Nov 10 13:14:14 2006
@@ -13,6 +13,7 @@
 {
 	int rc;
 	struct xen_memory_map memmap;
+	unsigned long arg = DOMID_SELF;
 	/*
 	 * This is rather large for a stack variable but this early in
 	 * the boot process we know we have plenty slack space.
@@ -26,6 +27,11 @@
 	if ( rc == -ENOSYS ) {
 		memmap.nr_entries = 1;
 		map[0].addr = 0ULL;
+		rc = HYPERVISOR_memory_op(XENMEM_maximum_reservation, &arg);
+		if ( rc < 0 )
+			map[0].size = PFN_PHYS(xen_start_info->nr_pages);
+		else
+			map[0].size = PFN_PHYS(rc);
 		map[0].size = PFN_PHYS(xen_start_info->nr_pages);
 		/* 8MB slack (to balance backend allocations). */
 		map[0].size += 8ULL << 20;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Glauber de Oliveira Costa]

On Fri, Nov 10, 2006 at 03:23:15PM +0000, Keir Fraser wrote:
> On 10/11/06 13:24, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:
> 
> > Trying to start a guest with maxmem > mem and then balloon up to a value
> > greater than mem is currently failing. This have been already discovered
> > (patch sent some days ago) for x86_64. i386 suffers from the same
> > problem.
> > 
> > This patch fixes it.
> 
> I took both patches and then changed my mind and immediately reverted them.
> There is a better way: we should support the XENMEM_memory_map hypercall.
> We should provide a hypercall (domctl) to set a memory_map_limit parameter
> and then Xen can use that to fake a memory map when XENMEM_memory_map is
> called. The tools can set that parameter from config['maxmem'].

And what happens when the hypercall ever returns ENOSYS, like a kernel
running in a bit old Hypervisor? 

IMHO,If we have to ever fallback into default assumptions, it seems wiser 
to extend the physicall map to maximum_reservation, not current_reservation. 

-- 
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Keir Fraser]

On 10/11/06 15:33, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:

>> I took both patches and then changed my mind and immediately reverted them.
>> There is a better way: we should support the XENMEM_memory_map hypercall.
>> We should provide a hypercall (domctl) to set a memory_map_limit parameter
>> and then Xen can use that to fake a memory map when XENMEM_memory_map is
>> called. The tools can set that parameter from config['maxmem'].
> 
> And what happens when the hypercall ever returns ENOSYS, like a kernel
> running in a bit old Hypervisor?
> 
> IMHO,If we have to ever fallback into default assumptions, it seems wiser
> to extend the physicall map to maximum_reservation, not current_reservation.

Maxmem will in future be fixed to track tot_pages. That was its original
purpose: to cap what memory the guest is allowed *now*, not to tell it the
max that it will ever be allowed. Also on e.g., dom0 the max is implemented
as ~0UL (no limit). "Fortunately" this looks like a -ve return code from the
max_mem query hypercall, but if it didn't (for example we masked off the
high bit to make sure the return value was +ve; a fix I will probably
implement) then dom0 boot will be broken as it will try to allocate a
much-too-big p2m table.

We have a way to tell the guest its maximum-ever memory allowance, it's just
not implemented on the tools and hypervisor side yet. That should be fixed.

 -- Keir

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Daniel P. Berrange]

On Fri, Nov 10, 2006 at 03:43:10PM +0000, Keir Fraser wrote:
> On 10/11/06 15:33, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:
> 
> >> I took both patches and then changed my mind and immediately reverted them.
> >> There is a better way: we should support the XENMEM_memory_map hypercall.
> >> We should provide a hypercall (domctl) to set a memory_map_limit parameter
> >> and then Xen can use that to fake a memory map when XENMEM_memory_map is
> >> called. The tools can set that parameter from config['maxmem'].
> > 
> > And what happens when the hypercall ever returns ENOSYS, like a kernel
> > running in a bit old Hypervisor?
> > 
> > IMHO,If we have to ever fallback into default assumptions, it seems wiser
> > to extend the physicall map to maximum_reservation, not current_reservation.
> 
> Maxmem will in future be fixed to track tot_pages. That was its original
> purpose: to cap what memory the guest is allowed *now*, not to tell it the
> max that it will ever be allowed. Also on e.g., dom0 the max is implemented
> as ~0UL (no limit). "Fortunately" this looks like a -ve return code from the
> max_mem query hypercall, but if it didn't (for example we masked off the
> high bit to make sure the return value was +ve; a fix I will probably
> implement) then dom0 boot will be broken as it will try to allocate a
> much-too-big p2m table.
> 
> We have a way to tell the guest its maximum-ever memory allowance, it's just
> not implemented on the tools and hypervisor side yet. That should be fixed.

This is not going to be any help for guests running on existing 3.0.3 tree
though. Glauber's patch makes the maxmem stuff work correctly for guest
kernels without requiring an upgrade of the HV which is much nicer for
compatability allowing both current & future guests to work on both current
and future HVs. With the increasingly widespread deployment of Xen I think
this kind of compatability is very important.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Keir Fraser]

On 10/11/06 15:54, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> This is not going to be any help for guests running on existing 3.0.3 tree
> though. Glauber's patch makes the maxmem stuff work correctly for guest
> kernels without requiring an upgrade of the HV which is much nicer for
> compatability allowing both current & future guests to work on both current
> and future HVs. With the increasingly widespread deployment of Xen I think
> this kind of compatability is very important.

I'd rather have a scheme that worked with older guests than one which worked
with older hypervisors. It also happens to be the more tasteful solution
here.

 -- Keir

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Glauber de Oliveira Costa]

On Fri, Nov 10, 2006 at 03:43:10PM +0000, Keir Fraser wrote:
> On 10/11/06 15:33, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:
> 
> >> I took both patches and then changed my mind and immediately reverted them.
> >> There is a better way: we should support the XENMEM_memory_map hypercall.
> >> We should provide a hypercall (domctl) to set a memory_map_limit parameter
> >> and then Xen can use that to fake a memory map when XENMEM_memory_map is
> >> called. The tools can set that parameter from config['maxmem'].
> > 
> > And what happens when the hypercall ever returns ENOSYS, like a kernel
> > running in a bit old Hypervisor?
> > 
> > IMHO,If we have to ever fallback into default assumptions, it seems wiser
> > to extend the physicall map to maximum_reservation, not current_reservation.
> 
> Maxmem will in future be fixed to track tot_pages. That was its original
> purpose: to cap what memory the guest is allowed *now*, not to tell it the
> max that it will ever be allowed. 

In this scenario, what's the purpose of current_reservation, as the only
difference from it now, is that it returns tot_pages instead of
max_pages ?

-- 
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Keir Fraser]

On 10/11/06 16:10, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:

>> Maxmem will in future be fixed to track tot_pages. That was its original
>> purpose: to cap what memory the guest is allowed *now*, not to tell it the
>> max that it will ever be allowed.
> 
> In this scenario, what's the purpose of current_reservation, as the only
> difference from it now, is that it returns tot_pages instead of
> max_pages ?

The idea is that when you ask a guest to balloon down you will change its
balloon target, then you will give it 'reasonable time' to drop its memory
allocation (reflected by tot_pages). If it fails to comply you take action
(e.g., warn user, suspend or kill the guest, etc). If it complies you drop
max_pages.

So you see that max_pages tracks behind tot_pages when you balloon down, and
tracks ahead when you balloon up. It has a distinct purpose separate from
that of tot_pages. That's the theory anyway. It's never been properly
implemented in the python tools. But that may well change in future, so I'm
not going to bake assumptions into guest kernels.

 -- Keir

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Glauber de Oliveira Costa]

 
> We have a way to tell the guest its maximum-ever memory allowance, it's just
> not implemented on the tools and hypervisor side yet. That should be fixed.

Do we have any specification on how should this look like, and what kind
of behaviour it should provide?


-- 
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Keir Fraser]

On 10/11/06 8:27 pm, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:

>> We have a way to tell the guest its maximum-ever memory allowance, it's just
>> not implemented on the tools and hypervisor side yet. That should be fixed.
> 
> Do we have any specification on how should this look like, and what kind
> of behaviour it should provide?

XENMEM_set_memory_map taking a struct { domain, range_list }. This can
initialise a small array embedded in the domain structure.

Plumb through libxc and python xc.so. Probably make the call from
LinuxImageHandler (or whatever it's called) in image.py.

Done. 

Most of the pain is in the tools plumbing (in particular arg parsing in the
python wrapper).

 -- Keir

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Keir Fraser]

On 10/11/06 15:33, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:

>> I took both patches and then changed my mind and immediately reverted them.
>> There is a better way: we should support the XENMEM_memory_map hypercall.
>> We should provide a hypercall (domctl) to set a memory_map_limit parameter
>> and then Xen can use that to fake a memory map when XENMEM_memory_map is
>> called. The tools can set that parameter from config['maxmem'].
> 
> And what happens when the hypercall ever returns ENOSYS, like a kernel
> running in a bit old Hypervisor?

Oh, and bear in mind you can always use 'memory=' as a boot parameter to
force the p2m size and mem_map size.

 -- Keir

Re: [PATCH] Make ballooning work with maxmem > mem (i386 version)

[Daniel P. Berrange]

On Fri, Nov 10, 2006 at 03:44:07PM +0000, Keir Fraser wrote:
> 
> 
> 
> On 10/11/06 15:33, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:
> 
> >> I took both patches and then changed my mind and immediately reverted them.
> >> There is a better way: we should support the XENMEM_memory_map hypercall.
> >> We should provide a hypercall (domctl) to set a memory_map_limit parameter
> >> and then Xen can use that to fake a memory map when XENMEM_memory_map is
> >> called. The tools can set that parameter from config['maxmem'].
> > 
> > And what happens when the hypercall ever returns ENOSYS, like a kernel
> > running in a bit old Hypervisor?
> 
> Oh, and bear in mind you can always use 'memory=' as a boot parameter to
> force the p2m size and mem_map size.

That's not particuarly helpful when booting a domain with pygrub because
now everytime you change the memory settings in the config for the guest
you also need to update the grub config to actually make it take effect.
Or, if you're not using pygrub, then every time you change the 'mem'
setting in the guest config, you also need to update the 'extra' setting
for the command line.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|