xen does not see more than 173800500k of memory

xen does not see more than 173800500k of memory

[Krysan, Susan]

[-- Attachment #1.1: Type: text/plain, Size: 1320 bytes --]

Our test server (Unisys ES7000) has recently been upgraded with more
processors and more memory, so I am testing x86_64 xen with 64 cpus and
192gb.  I compiled xen with max_phys_cpus=64 and debug=y.  xen sees the
64 cpus, but only sees 173800500k of the 192gb.  I booted dom0 with
2048M and numa=on.  I get the same result when I configure the server to
have 176gb (xen still sees only 173800500k). Booting without numa=on
makes no difference.

 

xentop - 14:48:42   Xen 3.0-unstable

1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown

Mem: 173800500k total, 4324748k used, 169475752k free    CPUs: 64 @
3400MHz

      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
MAXMEM(%) VCPUS

 NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID

  Domain-0 -----r         95    0.0    2097152    1.2   no limit
n/a    32

    0        0        0    0        0        0        0    0

 

When I configure the server with 160gb or less, xen sees all of the
memory. 

 

Also, when I boot with the SLES kernel, SLES can see the all of the
memory.

 

I assume there is not a memory limit on xen.  I would greatly appreciate
any help to resolve this problem.

 

Thanks,

Sue Krysan

Linux Systems Group

Unisys Corporation

 

 


[-- Attachment #1.2: Type: text/html, Size: 6329 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen does not see more than 173800500k of memory

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 388 bytes --]

Œxm dmesg¹ output?

 -- Keir

On 22/8/07 23:33, "Krysan, Susan" <KRYSANS@unisys.com> wrote:

> 
> When I configure the server with 160gb or less, xen sees all of the memory.
>  
> Also, when I boot with the SLES kernel, SLES can see the all of the memory.
>  
> I assume there is not a memory limit on xen.  I would greatly appreciate any
> help to resolve this problem.



[-- Attachment #1.2: Type: text/html, Size: 972 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen does not see more than 173800500k of memory

[Jan Beulich]

This seems to correspond to the truncation at 166Gb for 32on64 support (see
xen/arch/x86/e820.c:machine_specific_memory_setup()). There used to be
a command line options to disallow creation of compat mode guests (thereby
allowing all memory to be used), but that option went away with fairly recent
changes.

Jan

>>> "Krysan, Susan" <KRYSANS@unisys.com> 23.08.07 00:33 >>>
Our test server (Unisys ES7000) has recently been upgraded with more
processors and more memory, so I am testing x86_64 xen with 64 cpus and
192gb.  I compiled xen with max_phys_cpus=64 and debug=y.  xen sees the
64 cpus, but only sees 173800500k of the 192gb.  I booted dom0 with
2048M and numa=on.  I get the same result when I configure the server to
have 176gb (xen still sees only 173800500k). Booting without numa=on
makes no difference.

 

xentop - 14:48:42   Xen 3.0-unstable

1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown

Mem: 173800500k total, 4324748k used, 169475752k free    CPUs: 64 @
3400MHz

      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
MAXMEM(%) VCPUS

 NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID

  Domain-0 -----r         95    0.0    2097152    1.2   no limit
n/a    32

    0        0        0    0        0        0        0    0

 

When I configure the server with 160gb or less, xen sees all of the
memory. 

 

Also, when I boot with the SLES kernel, SLES can see the all of the
memory.

 

I assume there is not a memory limit on xen.  I would greatly appreciate
any help to resolve this problem.

 

Thanks,

Sue Krysan

Linux Systems Group

Unisys Corporation

 

 

Re: xen does not see more than 173800500k of memory

[Keir Fraser]

This should be easily fixed by properly applying
domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it only
applied when the bitsize is explicitly specified by the caller?

I think that's the only thing to fix to allow the 166GB boot-time
restriction to be lifted, but am I missing something, Jan?

 -- Keir

On 23/8/07 08:34, "Jan Beulich" <jbeulich@novell.com> wrote:

> This seems to correspond to the truncation at 166Gb for 32on64 support (see
> xen/arch/x86/e820.c:machine_specific_memory_setup()). There used to be
> a command line options to disallow creation of compat mode guests (thereby
> allowing all memory to be used), but that option went away with fairly recent
> changes.
> 
> Jan
> 
>>>> "Krysan, Susan" <KRYSANS@unisys.com> 23.08.07 00:33 >>>
> Our test server (Unisys ES7000) has recently been upgraded with more
> processors and more memory, so I am testing x86_64 xen with 64 cpus and
> 192gb.  I compiled xen with max_phys_cpus=64 and debug=y.  xen sees the
> 64 cpus, but only sees 173800500k of the 192gb.  I booted dom0 with
> 2048M and numa=on.  I get the same result when I configure the server to
> have 176gb (xen still sees only 173800500k). Booting without numa=on
> makes no difference.
> 
>  
> 
> xentop - 14:48:42   Xen 3.0-unstable
> 
> 1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
> shutdown
> 
> Mem: 173800500k total, 4324748k used, 169475752k free    CPUs: 64 @
> 3400MHz
> 
>       NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k)
> MAXMEM(%) VCPUS
> 
>  NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
> 
>   Domain-0 -----r         95    0.0    2097152    1.2   no limit
> n/a    32
> 
>     0        0        0    0        0        0        0    0
> 
>  
> 
> When I configure the server with 160gb or less, xen sees all of the
> memory. 
> 
>  
> 
> Also, when I boot with the SLES kernel, SLES can see the all of the
> memory.
> 
>  
> 
> I assume there is not a memory limit on xen.  I would greatly appreciate
> any help to resolve this problem.
> 
>  
> 
> Thanks,
> 
> Sue Krysan
> 
> Linux Systems Group
> 
> Unisys Corporation
> 
>  
> 
>  
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xen does not see more than 173800500k of memory

[Jan Beulich]

>>> Keir Fraser <keir@xensource.com> 23.08.07 16:27 >>>
>This should be easily fixed by properly applying
>domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it only
>applied when the bitsize is explicitly specified by the caller?
>
>I think that's the only thing to fix to allow the 166GB boot-time
>restriction to be lifted, but am I missing something, Jan?

We had this discussion before - the problem is not restricting the allocations
a domain does, but pages getting passed to it from other domains, which (if
they happen to lie outside the 166Gb range) the domain then can't control.
And yes, you said page flipping is basically dead, but this isn't being
enforced (and probably can't as long as you want to support older guests
potentially using it).

Jan

Re: xen does not see more than 173800500k of memory

[Keir Fraser]

It's not a Xen security risk though. If you happen to use a compat guest
with page flipping then it just won't work. I think it's fair to say at this
point that that is just 'too bad'. If anyone really cares then they will
need to add a copy-to-low-memory path in Xen's page transfer code. The 166GB
restriction has to go.

 -- Keir

On 23/8/07 15:51, "Jan Beulich" <jbeulich@novell.com> wrote:

>>>> Keir Fraser <keir@xensource.com> 23.08.07 16:27 >>>
>> This should be easily fixed by properly applying
>> domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it only
>> applied when the bitsize is explicitly specified by the caller?
>> 
>> I think that's the only thing to fix to allow the 166GB boot-time
>> restriction to be lifted, but am I missing something, Jan?
> 
> We had this discussion before - the problem is not restricting the allocations
> a domain does, but pages getting passed to it from other domains, which (if
> they happen to lie outside the 166Gb range) the domain then can't control.
> And yes, you said page flipping is basically dead, but this isn't being
> enforced (and probably can't as long as you want to support older guests
> potentially using it).
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: xen does not see more than 173800500k of memory

[Subrahmanian, Raj]

>Re-sent. The mail I sent out Monday is apparently not on the list.
Thanks
Raj

All,
>From looking at the code, it seems that the proximate cause for this
limit is the truncation at 166Gb for 32on64 support in e820.c, and not
the domain_clamp_alloc_bitsize. 
If I understand the issue correctly, if this limit were removed from
e820.c, this would work, except for the side-effect on 32-on-64 guests?
Is this correct?
Also, where should I look to find Xen's page transfer code?
Thanks
Raj
>It's not a Xen security risk though. If you happen to use a compat 
>guest with page flipping then it just won't work. I think it's fair to 
>say at this point that that is just 'too bad'. If anyone really cares 
>then they will need to add a copy-to-low-memory path in Xen's page 
>transfer code. The 166GB restriction has to go.
>
> -- Keir
>
>On 23/8/07 15:51, "Jan Beulich" <jbeulich@novell.com> wrote:
>
>>>>> Keir Fraser <keir@xensource.com> 23.08.07 16:27 >>>
>>> This should be easily fixed by properly applying
>>> domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it 
>>> only applied when the bitsize is explicitly specified by the caller?
>>> 
>>> I think that's the only thing to fix to allow the 166GB boot-time 
>>> restriction to be lifted, but am I missing something, Jan?
>> 
>> We had this discussion before - the problem is not restricting the 
>> allocations a domain does, but pages getting passed to it from other 
>> domains, which (if they happen to lie outside the 166Gb
>range) the domain then can't control.
>> And yes, you said page flipping is basically dead, but this isn't 
>> being enforced (and probably can't as long as you want to support 
>> older guests potentially using it).
>> 
>> Jan