Problem with nr_nodes on large memory NUMA machine

Problem with nr_nodes on large memory NUMA machine

[beth kon]

We've run into an issue with an 8 node x3950 where xm info is showing 
only 6 nodes. I've traced the problem to the clip_to_limit function in 
arch/x86/e820.c.

#ifdef __x86_64__
   clip_to_limit((uint64_t)(MACH2PHYS_COMPAT_VIRT_END -
                            __HYPERVISOR_COMPAT_VIRT_START) << 10,
                 "Only the first %u GB of the physical memory map "
                 "can be accessed by 32-on-64 guests.");
#endif

Boot messages....
(XEN) WARNING: Only the first 166 GB of the physical memory map can be 
accessed by 32-on-64 guests. (XEN) Truncating memory map to 174063616kB

After the memory is clipped, acpi_scan_nodes runs cutoff_node, which 
limits the memory associated with each node according to the cutoff 
values. Then, acpi_scan_nodes calls unparse_node to "remove" nodes that 
don't have the minimum amount of memory, due to the clipping of the 
memory range.

Can someone explain what this is all about and why it might be necessary?

-- 
Elizabeth Kon (Beth)
IBM Linux Technology Center
Open Hypervisor Team
email: eak@us.ibm.com

Re: Problem with nr_nodes on large memory NUMA machine

[Keir Fraser]

32-bit guests on 64-bit hypervisor can only address the bottom 166GB of the
memory map, due to constraints on how much of the machine-to-phys array they
can reasonably map into their limited address space.

 -- Keir

On 19/10/07 16:02, "beth kon" <eak@us.ibm.com> wrote:

> We've run into an issue with an 8 node x3950 where xm info is showing
> only 6 nodes. I've traced the problem to the clip_to_limit function in
> arch/x86/e820.c.
> 
> #ifdef __x86_64__
>    clip_to_limit((uint64_t)(MACH2PHYS_COMPAT_VIRT_END -
>                             __HYPERVISOR_COMPAT_VIRT_START) << 10,
>                  "Only the first %u GB of the physical memory map "
>                  "can be accessed by 32-on-64 guests.");
> #endif
> 
> Boot messages....
> (XEN) WARNING: Only the first 166 GB of the physical memory map can be
> accessed by 32-on-64 guests. (XEN) Truncating memory map to 174063616kB
> 
> After the memory is clipped, acpi_scan_nodes runs cutoff_node, which
> limits the memory associated with each node according to the cutoff
> values. Then, acpi_scan_nodes calls unparse_node to "remove" nodes that
> don't have the minimum amount of memory, due to the clipping of the
> memory range.
> 
> Can someone explain what this is all about and why it might be necessary?

RE: Problem with nr_nodes on large memory NUMA machine

[Subrahmanian, Raj]

Beth,
We saw this issue on our boxes too. 
http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00479.html
I am trying to figure out how to write the copy-to-low-memory path. 
Keir, could you give me some suggestions?
Thanks
Raj
>32-bit guests on 64-bit hypervisor can only address the bottom 
>166GB of the memory map, due to constraints on how much of the 
>machine-to-phys array they can reasonably map into their 
>limited address space.
>
> -- Keir
>
>On 19/10/07 16:02, "beth kon" <eak@us.ibm.com> wrote:
>
>> We've run into an issue with an 8 node x3950 where xm info 
>is showing 
>> only 6 nodes. I've traced the problem to the clip_to_limit 
>function in 
>> arch/x86/e820.c.
>> 
>> #ifdef __x86_64__
>>    clip_to_limit((uint64_t)(MACH2PHYS_COMPAT_VIRT_END -
>>                             __HYPERVISOR_COMPAT_VIRT_START) << 10,
>>                  "Only the first %u GB of the physical memory map "
>>                  "can be accessed by 32-on-64 guests."); #endif
>> 
>> Boot messages....
>> (XEN) WARNING: Only the first 166 GB of the physical memory 
>map can be 
>> accessed by 32-on-64 guests. (XEN) Truncating memory map to 
>> 174063616kB
>> 
>> After the memory is clipped, acpi_scan_nodes runs cutoff_node, which 
>> limits the memory associated with each node according to the cutoff 
>> values. Then, acpi_scan_nodes calls unparse_node to "remove" nodes 
>> that don't have the minimum amount of memory, due to the clipping of 
>> the memory range.
>> 
>> Can someone explain what this is all about and why it might 
>be necessary?

Re: Problem with nr_nodes on large memory NUMA machine

[Keir Fraser]

On 19/10/07 16:21, "Subrahmanian, Raj" <raj.subrahmanian@unisys.com> wrote:

> Beth,
> We saw this issue on our boxes too.
> http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00479.html
> I am trying to figure out how to write the copy-to-low-memory path.
> Keir, could you give me some suggestions?

In gnttab_transfer(), if the foreign domain (e) is 32-on-64 and the page
being stolen from the local domain (d) is above 166GB then allocate anothr
domheap page for e, copy the stolen page contents to it. Then free the
stolen page and the new page takes its place.

 -- Keir