32-bit PV guests and 168 GB boundary

32-bit PV guests and 168 GB boundary

[Nick Pegg]

Hello,

I've recently ran into a problem on a physical machine with a large
amount of memory where 32-bit PV guests would refuse to boot with this
error message:

xm create: Error: (4, 'Out of memory', "panic: xc_dom_boot.c:159:
xc_dom_boot_mem_init: can't allocate low memory for domain")

We discovered that there's a boundary where 32-bit PV guests can't be
allocated memory above 168GB, along with the work-around config option
to reserve memory for these guests (total_available_memory in
xend-config.sxp).

Could someone clarify why this is a restriction? Is this some sort of
hardware limitation?


Thanks,
Nick

Re: 32-bit PV guests and 168 GB boundary

[Jan Beulich]

>>> On 15.04.13 at 16:21, Nick Pegg <npegg@linode.com> wrote:
> I've recently ran into a problem on a physical machine with a large
> amount of memory where 32-bit PV guests would refuse to boot with this
> error message:
> 
> xm create: Error: (4, 'Out of memory', "panic: xc_dom_boot.c:159:
> xc_dom_boot_mem_init: can't allocate low memory for domain")
> 
> We discovered that there's a boundary where 32-bit PV guests can't be
> allocated memory above 168GB, along with the work-around config option
> to reserve memory for these guests (total_available_memory in
> xend-config.sxp).
> 
> Could someone clarify why this is a restriction? Is this some sort of
> hardware limitation?

No, this is an interface limitation: 32-bit PV guests can't be expected
to be able to deal with an address space hole of more than 168Mb
at the top of the address space, yet a bigger hole would be needed
to make the full (read-only) M2P table for the guest.

If you wanted to lift that limit, you'd have to make the guest capable
of falling back to using hypercalls for doing the M2P lookups for MFNs
outside the visible range - no one cared enough about this to
implement something like that.

Jan

Re: 32-bit PV guests and 168 GB boundary

[Andrew Cooper]

On 15/04/13 15:21, Nick Pegg wrote:
> Hello,
>
> I've recently ran into a problem on a physical machine with a large
> amount of memory where 32-bit PV guests would refuse to boot with this
> error message:
>
> xm create: Error: (4, 'Out of memory', "panic: xc_dom_boot.c:159:
> xc_dom_boot_mem_init: can't allocate low memory for domain")
>
> We discovered that there's a boundary where 32-bit PV guests can't be
> allocated memory above 168GB, along with the work-around config option
> to reserve memory for these guests (total_available_memory in
> xend-config.sxp).
>
> Could someone clarify why this is a restriction? Is this some sort of
> hardware limitation?
>
>
> Thanks,
> Nick

This is because of the amount of the Xen m2p table mapped into 32bit
kernel space, between 0xf5800000 and 0xffffffff.  PV guests require the
m2p table to perform paravirtualised memory management.  This is a
fundamental limit of 32bit PV guests.

~Andrew

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: 32-bit PV guests and 168 GB boundary

[Ian Campbell]

On Mon, 2013-04-15 at 15:21 +0100, Nick Pegg wrote:
> Hello,
> 
> I've recently ran into a problem on a physical machine with a large
> amount of memory where 32-bit PV guests would refuse to boot with this
> error message:
> 
> xm create: Error: (4, 'Out of memory', "panic: xc_dom_boot.c:159:
> xc_dom_boot_mem_init: can't allocate low memory for domain")
> 
> We discovered that there's a boundary where 32-bit PV guests can't be
> allocated memory above 168GB, along with the work-around config option
> to reserve memory for these guests (total_available_memory in
> xend-config.sxp).
> 
> Could someone clarify why this is a restriction? Is this some sort of
> hardware limitation?

It's a 32-bit PV guest limitation which arises due to the need to map
the m2p table into each PV guest, there is simply only enough room for
168GB worth of m2p in the 32-bit kernel address space before it starts
to impinge on other stuff. In theory the host admin could be given the
flexibility to make it a little bigger (not sure if any toolstack
implements this), but not by much.

There are various hacky things which could in theory be done, such as
giving 32-bit guests views onto different 168GB windows of physical
memory from which all their RAM must come, but none of those have been
implemented AFAIK.

Hopefully PVH will eventually support 32-bit and this restriction will
go away.

Ian.