eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Jan Beulich]

>> We saw this issue on our boxes too.
>> http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00479.html 
>> I am trying to figure out how to write the copy-to-low-memory path.
>> Keir, could you give me some suggestions?
>
>In gnttab_transfer(), if the foreign domain (e) is 32-on-64 and the page
>being stolen from the local domain (d) is above 166GB then allocate anothr
>domheap page for e, copy the stolen page contents to it. Then free the
>stolen page and the new page takes its place.

I think page allocation in this path isn't nice, at least not without success
guarantee (not the least because because netback doesn't check return
values). I would therefore rather see a solution in placing the burden of
ensuring accessibility on the producer (netback) of the page, and fail the
transfer if the destination domain can't access the page (whether to be
nice and try an allocate-and-copy operation here is a secondary thing).

Netback would then need to determine the address size of netfront's domain
(just like blkback and blktap do, except that HVM domains should also be
treated as not requiring address restriction), and have two pools of pages
for use in transfers - one unrestricted and one limited to 37 address bits (the
two could be folded for resource efficiency if the machine has less than
128G). Besides that, netback would also start checking return values of the
multicall pieces.

Jan

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Keir Fraser]

On 27/11/07 08:43, "Jan Beulich" <jbeulich@novell.com> wrote:

> I think page allocation in this path isn't nice, at least not without success
> guarantee (not the least because because netback doesn't check return
> values). I would therefore rather see a solution in placing the burden of
> ensuring accessibility on the producer (netback) of the page, and fail the
> transfer if the destination domain can't access the page (whether to be
> nice and try an allocate-and-copy operation here is a secondary thing).
> 
> Netback would then need to determine the address size of netfront's domain
> (just like blkback and blktap do, except that HVM domains should also be
> treated as not requiring address restriction), and have two pools of pages
> for use in transfers - one unrestricted and one limited to 37 address bits
> (the
> two could be folded for resource efficiency if the machine has less than
> 128G). Besides that, netback would also start checking return values of the
> multicall pieces.

I don't get how your netback approach works. The pages we transfer do not
originate from netback, so it has little control over them. And, even if it
did, when we allocate pages for network receive we do not know which
domain's packet will end up in each buffer.

Personally I think doing it in Xen is perfectly good enough for supporting
this very out-of-date network receive mechanism.

 -- Keir

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Jan Beulich]

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 09:56 >>>
>On 27/11/07 08:43, "Jan Beulich" <jbeulich@novell.com> wrote:
>
>> I think page allocation in this path isn't nice, at least not without success
>> guarantee (not the least because because netback doesn't check return
>> values). I would therefore rather see a solution in placing the burden of
>> ensuring accessibility on the producer (netback) of the page, and fail the
>> transfer if the destination domain can't access the page (whether to be
>> nice and try an allocate-and-copy operation here is a secondary thing).
>> 
>> Netback would then need to determine the address size of netfront's domain
>> (just like blkback and blktap do, except that HVM domains should also be
>> treated as not requiring address restriction), and have two pools of pages
>> for use in transfers - one unrestricted and one limited to 37 address bits
>> (the
>> two could be folded for resource efficiency if the machine has less than
>> 128G). Besides that, netback would also start checking return values of the
>> multicall pieces.
>
>I don't get how your netback approach works. The pages we transfer do not
>originate from netback, so it has little control over them. And, even if it
>did, when we allocate pages for network receive we do not know which
>domain's packet will end up in each buffer.

Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
netback could take care of this by doing the copying there, as at that point i
already knows the destination domain.

>Personally I think doing it in Xen is perfectly good enough for supporting
>this very out-of-date network receive mechanism.

I'm not just concerned about netback here. The interface exists, and other
users might show up and/or exist already. Whether it would be acceptable
for them to do allocation and copying is unknown. You'd therefore either
need a way to prevent future users of the transfer mechanism, or set proper
requirements on its use. I think that placing extra requirements on the user
of the interface is better than introducing extra (possibly hard to reproduce/
recognize/debug) possibilities of failure.

Jan

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Keir Fraser]

On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:

>> I don't get how your netback approach works. The pages we transfer do not
>> originate from netback, so it has little control over them. And, even if it
>> did, when we allocate pages for network receive we do not know which
>> domain's packet will end up in each buffer.
> 
> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
> netback could take care of this by doing the copying there, as at that point i
> already knows the destination domain.

You may not know constraints on that domain's max_mfn though. We could add
an interface to Xen to interrogate that, but generally it's not something we
probably want to expose outside of Xen and the guest itself.

>> Personally I think doing it in Xen is perfectly good enough for supporting
>> this very out-of-date network receive mechanism.
> 
> I'm not just concerned about netback here. The interface exists, and other
> users might show up and/or exist already. Whether it would be acceptable
> for them to do allocation and copying is unknown. You'd therefore either
> need a way to prevent future users of the transfer mechanism, or set proper
> requirements on its use. I think that placing extra requirements on the user
> of the interface is better than introducing extra (possibly hard to reproduce/
> recognize/debug) possibilities of failure.

The interface is obsolete.

 -- Keir

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Jan Beulich]

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>
>>> I don't get how your netback approach works. The pages we transfer do not
>>> originate from netback, so it has little control over them. And, even if it
>>> did, when we allocate pages for network receive we do not know which
>>> domain's packet will end up in each buffer.
>> 
>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
>> netback could take care of this by doing the copying there, as at that point i
>> already knows the destination domain.
>
>You may not know constraints on that domain's max_mfn though. We could add
>an interface to Xen to interrogate that, but generally it's not something we
>probably want to expose outside of Xen and the guest itself.

What constraints other than the guest's address size influence its max_mfn?
Of course, if there's anything beyond the address size, then having a way to
obtain the constraint explicitly would be desirable. But otherwise (and as
fallback) using 37 bits (128G) seems quite reasonable.

>>> Personally I think doing it in Xen is perfectly good enough for supporting
>>> this very out-of-date network receive mechanism.
>> 
>> I'm not just concerned about netback here. The interface exists, and other
>> users might show up and/or exist already. Whether it would be acceptable
>> for them to do allocation and copying is unknown. You'd therefore either
>> need a way to prevent future users of the transfer mechanism, or set proper
>> requirements on its use. I think that placing extra requirements on the user
>> of the interface is better than introducing extra (possibly hard to reproduce/
>> recognize/debug) possibilities of failure.
>
>The interface is obsolete.

Then it should be clearly indicated as such, e.g. by a mechanism similar to
deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in netback should
then probably be conditional upon an extra config option, which could at once
be used to provide a note to Xen that the feature isn't being used so that the
function could return -ENOSYS and the clipping could be avoided/reverted.

Jan

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[beth kon]

Has there been any more thought on this subject? The discussion seems to 
have stalled, and we're hoping to find a way past this 166G limit...

Jan Beulich wrote:

>>>>Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>>>>        
>>>>
>>On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>>
>>    
>>
>>>>I don't get how your netback approach works. The pages we transfer do not
>>>>originate from netback, so it has little control over them. And, even if it
>>>>did, when we allocate pages for network receive we do not know which
>>>>domain's packet will end up in each buffer.
>>>>        
>>>>
>>>Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
>>>netback could take care of this by doing the copying there, as at that point i
>>>already knows the destination domain.
>>>      
>>>
>>You may not know constraints on that domain's max_mfn though. We could add
>>an interface to Xen to interrogate that, but generally it's not something we
>>probably want to expose outside of Xen and the guest itself.
>>    
>>
>
>What constraints other than the guest's address size influence its max_mfn?
>Of course, if there's anything beyond the address size, then having a way to
>obtain the constraint explicitly would be desirable. But otherwise (and as
>fallback) using 37 bits (128G) seems quite reasonable.
>
>  
>
>>>>Personally I think doing it in Xen is perfectly good enough for supporting
>>>>this very out-of-date network receive mechanism.
>>>>        
>>>>
>>>I'm not just concerned about netback here. The interface exists, and other
>>>users might show up and/or exist already. Whether it would be acceptable
>>>for them to do allocation and copying is unknown. You'd therefore either
>>>need a way to prevent future users of the transfer mechanism, or set proper
>>>requirements on its use. I think that placing extra requirements on the user
>>>of the interface is better than introducing extra (possibly hard to reproduce/
>>>recognize/debug) possibilities of failure.
>>>      
>>>
>>The interface is obsolete.
>>    
>>
>
>Then it should be clearly indicated as such, e.g. by a mechanism similar to
>deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in netback should
>then probably be conditional upon an extra config option, which could at once
>be used to provide a note to Xen that the feature isn't being used so that the
>function could return -ENOSYS and the clipping could be avoided/reverted.
>
>Jan
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>


-- 
Elizabeth Kon (Beth)
IBM Linux Technology Center
Open Hypervisor Team
email: eak@us.ibm.com

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Keir Fraser]

I'll get something in for 3.2.0.

 -- Keir


On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:

> Has there been any more thought on this subject? The discussion seems to
> have stalled, and we're hoping to find a way past this 166G limit...
> 
> Jan Beulich wrote:
> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>>>>>        
>>>>> 
>>> On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>>> 
>>>    
>>> 
>>>>> I don't get how your netback approach works. The pages we transfer do not
>>>>> originate from netback, so it has little control over them. And, even if
>>>>> it
>>>>> did, when we allocate pages for network receive we do not know which
>>>>> domain's packet will end up in each buffer.
>>>>>        
>>>>> 
>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
>>>> netback could take care of this by doing the copying there, as at that
>>>> point i
>>>> already knows the destination domain.
>>>>      
>>>> 
>>> You may not know constraints on that domain's max_mfn though. We could add
>>> an interface to Xen to interrogate that, but generally it's not something we
>>> probably want to expose outside of Xen and the guest itself.
>>>    
>>> 
>> 
>> What constraints other than the guest's address size influence its max_mfn?
>> Of course, if there's anything beyond the address size, then having a way to
>> obtain the constraint explicitly would be desirable. But otherwise (and as
>> fallback) using 37 bits (128G) seems quite reasonable.
>> 
>>  
>> 
>>>>> Personally I think doing it in Xen is perfectly good enough for supporting
>>>>> this very out-of-date network receive mechanism.
>>>>>        
>>>>> 
>>>> I'm not just concerned about netback here. The interface exists, and other
>>>> users might show up and/or exist already. Whether it would be acceptable
>>>> for them to do allocation and copying is unknown. You'd therefore either
>>>> need a way to prevent future users of the transfer mechanism, or set proper
>>>> requirements on its use. I think that placing extra requirements on the
>>>> user
>>>> of the interface is better than introducing extra (possibly hard to
>>>> reproduce/
>>>> recognize/debug) possibilities of failure.
>>>>      
>>>> 
>>> The interface is obsolete.
>>>    
>>> 
>> 
>> Then it should be clearly indicated as such, e.g. by a mechanism similar to
>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in netback
>> should
>> then probably be conditional upon an extra config option, which could at once
>> be used to provide a note to Xen that the feature isn't being used so that
>> the
>> function could return -ENOSYS and the clipping could be avoided/reverted.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>  
>> 
> 

RE: eliminating 166G limit (was Re: Problem with nr_nodeson large memory NUMA machine)

[Subrahmanian, Raj]

Beth,
I am working on it.
I spoke to Keir about it at the Xen summit.
I hope to have a patch that will fix the issue and have the high-to-low memory copy working soon.

Thanks
Raj

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of beth kon
>Sent: Monday, December 03, 2007 2:50 PM
>Cc: xen-devel@lists.xensource.com
>Subject: Re: eliminating 166G limit (was Re: [Xen-devel]
>Problem with nr_nodeson large memory NUMA machine)
>
>Has there been any more thought on this subject? The
>discussion seems to have stalled, and we're hoping to find a
>way past this 166G limit...
>
>Jan Beulich wrote:
>
>>>>>Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>>>>>
>>>>>
>>>On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>>>
>>>
>>>
>>>>>I don't get how your netback approach works. The pages we transfer
>>>>>do not originate from netback, so it has little control over them.
>>>>>And, even if it did, when we allocate pages for network receive we
>>>>>do not know which domain's packet will end up in each buffer.
>>>>>
>>>>>
>>>>Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
>>>>Nevertheless netback could take care of this by doing the copying
>>>>there, as at that point i already knows the destination domain.
>>>>
>>>>
>>>You may not know constraints on that domain's max_mfn
>though. We could
>>>add an interface to Xen to interrogate that, but generally it's not
>>>something we probably want to expose outside of Xen and the
>guest itself.
>>>
>>>
>>
>>What constraints other than the guest's address size
>influence its max_mfn?
>>Of course, if there's anything beyond the address size, then having a
>>way to obtain the constraint explicitly would be desirable. But
>>otherwise (and as
>>fallback) using 37 bits (128G) seems quite reasonable.
>>
>>
>>
>>>>>Personally I think doing it in Xen is perfectly good enough for
>>>>>supporting this very out-of-date network receive mechanism.
>>>>>
>>>>>
>>>>I'm not just concerned about netback here. The interface
>exists, and
>>>>other users might show up and/or exist already. Whether it would be
>>>>acceptable for them to do allocation and copying is unknown. You'd
>>>>therefore either need a way to prevent future users of the transfer
>>>>mechanism, or set proper requirements on its use. I think that
>>>>placing extra requirements on the user of the interface is better
>>>>than introducing extra (possibly hard to reproduce/
>>>>recognize/debug) possibilities of failure.
>>>>
>>>>
>>>The interface is obsolete.
>>>
>>>
>>
>>Then it should be clearly indicated as such, e.g. by a mechanism
>>similar to
>>deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
>>netback should then probably be conditional upon an extra config
>>option, which could at once be used to provide a note to Xen that the
>>feature isn't being used so that the function could return
>-ENOSYS and the clipping could be avoided/reverted.
>>
>>Jan
>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>
>
>--
>Elizabeth Kon (Beth)
>IBM Linux Technology Center
>Open Hypervisor Team
>email: eak@us.ibm.com
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

Re: eliminating 166G limit (was Re: Problem with nr_nodes on large memory NUMA machine)

[Keir Fraser]

Try xen-unstable changeset 16548.

 -- Keir

On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:

> Has there been any more thought on this subject? The discussion seems to
> have stalled, and we're hoping to find a way past this 166G limit...
> 
> Jan Beulich wrote:
> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>>>>>        
>>>>> 
>>> On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>>> 
>>>    
>>> 
>>>>> I don't get how your netback approach works. The pages we transfer do not
>>>>> originate from netback, so it has little control over them. And, even if
>>>>> it
>>>>> did, when we allocate pages for network receive we do not know which
>>>>> domain's packet will end up in each buffer.
>>>>>        
>>>>> 
>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag(). Nevertheless
>>>> netback could take care of this by doing the copying there, as at that
>>>> point i
>>>> already knows the destination domain.
>>>>      
>>>> 
>>> You may not know constraints on that domain's max_mfn though. We could add
>>> an interface to Xen to interrogate that, but generally it's not something we
>>> probably want to expose outside of Xen and the guest itself.
>>>    
>>> 
>> 
>> What constraints other than the guest's address size influence its max_mfn?
>> Of course, if there's anything beyond the address size, then having a way to
>> obtain the constraint explicitly would be desirable. But otherwise (and as
>> fallback) using 37 bits (128G) seems quite reasonable.
>> 
>>  
>> 
>>>>> Personally I think doing it in Xen is perfectly good enough for supporting
>>>>> this very out-of-date network receive mechanism.
>>>>>        
>>>>> 
>>>> I'm not just concerned about netback here. The interface exists, and other
>>>> users might show up and/or exist already. Whether it would be acceptable
>>>> for them to do allocation and copying is unknown. You'd therefore either
>>>> need a way to prevent future users of the transfer mechanism, or set proper
>>>> requirements on its use. I think that placing extra requirements on the
>>>> user
>>>> of the interface is better than introducing extra (possibly hard to
>>>> reproduce/
>>>> recognize/debug) possibilities of failure.
>>>>      
>>>> 
>>> The interface is obsolete.
>>>    
>>> 
>> 
>> Then it should be clearly indicated as such, e.g. by a mechanism similar to
>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in netback
>> should
>> then probably be conditional upon an extra config option, which could at once
>> be used to provide a note to Xen that the feature isn't being used so that
>> the
>> function could return -ENOSYS and the clipping could be avoided/reverted.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>  
>> 
> 

RE: eliminating 166G limit (was Re: Problem withnr_nodes on large memory NUMA machine)

[Krysan, Susan]

I tested this changeset on Unisys ES7000 with 256G ram and 64 processors
and it works:

xentop - 06:30:59   Xen 3.2-unstable
1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 268172340k total, 7669456k used, 260502884k free    CPUs: 64 @
3400MHz

I will be running our full test suite on this configuration today.

Thanks,
Sue Krysan
Linux Systems Group
Unisys Corporation
 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
Sent: Thursday, December 06, 2007 8:40 AM
To: eak@us.ibm.com
Cc: xen-devel@lists.xensource.com
Subject: Re: eliminating 166G limit (was Re: [Xen-devel] Problem
withnr_nodes on large memory NUMA machine)

Try xen-unstable changeset 16548.

 -- Keir

On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:

> Has there been any more thought on this subject? The discussion seems
to
> have stalled, and we're hoping to find a way past this 166G limit...
> 
> Jan Beulich wrote:
> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>>>>>        
>>>>> 
>>> On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>>> 
>>>    
>>> 
>>>>> I don't get how your netback approach works. The pages we transfer
do not
>>>>> originate from netback, so it has little control over them. And,
even if
>>>>> it
>>>>> did, when we allocate pages for network receive we do not know
which
>>>>> domain's packet will end up in each buffer.
>>>>>        
>>>>> 
>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
Nevertheless
>>>> netback could take care of this by doing the copying there, as at
that
>>>> point i
>>>> already knows the destination domain.
>>>>      
>>>> 
>>> You may not know constraints on that domain's max_mfn though. We
could add
>>> an interface to Xen to interrogate that, but generally it's not
something we
>>> probably want to expose outside of Xen and the guest itself.
>>>    
>>> 
>> 
>> What constraints other than the guest's address size influence its
max_mfn?
>> Of course, if there's anything beyond the address size, then having a
way to
>> obtain the constraint explicitly would be desirable. But otherwise
(and as
>> fallback) using 37 bits (128G) seems quite reasonable.
>> 
>>  
>> 
>>>>> Personally I think doing it in Xen is perfectly good enough for
supporting
>>>>> this very out-of-date network receive mechanism.
>>>>>        
>>>>> 
>>>> I'm not just concerned about netback here. The interface exists,
and other
>>>> users might show up and/or exist already. Whether it would be
acceptable
>>>> for them to do allocation and copying is unknown. You'd therefore
either
>>>> need a way to prevent future users of the transfer mechanism, or
set proper
>>>> requirements on its use. I think that placing extra requirements on
the
>>>> user
>>>> of the interface is better than introducing extra (possibly hard to
>>>> reproduce/
>>>> recognize/debug) possibilities of failure.
>>>>      
>>>> 
>>> The interface is obsolete.
>>>    
>>> 
>> 
>> Then it should be clearly indicated as such, e.g. by a mechanism
similar to
>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
netback
>> should
>> then probably be conditional upon an extra config option, which could
at once
>> be used to provide a note to Xen that the feature isn't being used so
that
>> the
>> function could return -ENOSYS and the clipping could be
avoided/reverted.
>> 
>> Jan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>  
>> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: eliminating 166G limit (was Re: Problem withnr_nodes on large memory NUMA machine)

[Subrahmanian, Raj]

Keir,
We were able to bring up a 250G machine with this changeset.
Raj

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>Sent: Thursday, December 06, 2007 8:40 AM
>To: eak@us.ibm.com
>Cc: xen-devel@lists.xensource.com
>Subject: Re: eliminating 166G limit (was Re: [Xen-devel]
>Problem withnr_nodes on large memory NUMA machine)
>
>Try xen-unstable changeset 16548.
>
> -- Keir
>
>On 3/12/07 19:49, "beth kon" <eak@us.ibm.com> wrote:
>
>> Has there been any more thought on this subject? The
>discussion seems
>> to have stalled, and we're hoping to find a way past this
>166G limit...
>>
>> Jan Beulich wrote:
>>
>>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 27.11.07 10:21 >>>
>>>>>>
>>>>>>
>>>> On 27/11/07 09:00, "Jan Beulich" <jbeulich@novell.com> wrote:
>>>>
>>>>
>>>>
>>>>>> I don't get how your netback approach works. The pages
>we transfer
>>>>>> do not originate from netback, so it has little control
>over them.
>>>>>> And, even if it did, when we allocate pages for network
>receive we
>>>>>> do not know which domain's packet will end up in each buffer.
>>>>>>
>>>>>>
>>>>> Oh, right, I mixed up old_mfn and new_mfn in netbk_gop_frag().
>>>>> Nevertheless netback could take care of this by doing the copying
>>>>> there, as at that point i already knows the destination domain.
>>>>>
>>>>>
>>>> You may not know constraints on that domain's max_mfn though. We
>>>> could add an interface to Xen to interrogate that, but generally
>>>> it's not something we probably want to expose outside of
>Xen and the guest itself.
>>>>
>>>>
>>>
>>> What constraints other than the guest's address size
>influence its max_mfn?
>>> Of course, if there's anything beyond the address size,
>then having a
>>> way to obtain the constraint explicitly would be desirable. But
>>> otherwise (and as
>>> fallback) using 37 bits (128G) seems quite reasonable.
>>>
>>>
>>>
>>>>>> Personally I think doing it in Xen is perfectly good enough for
>>>>>> supporting this very out-of-date network receive mechanism.
>>>>>>
>>>>>>
>>>>> I'm not just concerned about netback here. The interface exists,
>>>>> and other users might show up and/or exist already. Whether it
>>>>> would be acceptable for them to do allocation and copying is
>>>>> unknown. You'd therefore either need a way to prevent
>future users
>>>>> of the transfer mechanism, or set proper requirements on
>its use. I
>>>>> think that placing extra requirements on the user of the
>interface
>>>>> is better than introducing extra (possibly hard to reproduce/
>>>>> recognize/debug) possibilities of failure.
>>>>>
>>>>>
>>>> The interface is obsolete.
>>>>
>>>>
>>>
>>> Then it should be clearly indicated as such, e.g. by a mechanism
>>> similar to
>>> deprecated_irq_flag() in Linux 2.6.22. And as a result, its use in
>>> netback should then probably be conditional upon an extra config
>>> option, which could at once be used to provide a note to
>Xen that the
>>> feature isn't being used so that the function could return -ENOSYS
>>> and the clipping could be avoided/reverted.
>>>
>>> Jan
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>
>
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>

Re: eliminating 166G limit (was Re: Problem withnr_nodes on large memory NUMA machine)

[Yinghai Lu]

On Fri, Dec 7, 2007 at 6:20 AM, Krysan, Susan <KRYSANS@unisys.com> wrote:
> I tested this changeset on Unisys ES7000 with 256G ram and 64 processors
> and it works:
>
> xentop - 06:30:59   Xen 3.2-unstable
> 1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
> shutdown
> Mem: 268172340k total, 7669456k used, 260502884k free    CPUs: 64 @
> 3400MHz
>
> I will be running our full test suite on this configuration today.
>
> Thanks,
> Sue Krysan
> Linux Systems Group
> Unisys Corporation
>
>
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
> Sent: Thursday, December 06, 2007 8:40 AM
> To: eak@us.ibm.com
> Cc: xen-devel@lists.xensource.com
> Subject: Re: eliminating 166G limit (was Re: [Xen-devel] Problem
> withnr_nodes on large memory NUMA machine)
>
> Try xen-unstable changeset 16548.

Susan,

how about TOM? i mean MMIO size...

my system with 256g/32cores, and 0xe0000000 - 0x100000000 is mmio
mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
TOM: 00000000e0000000 aka 3584M
TOM2: 0000004020000000 aka 262656M

only can work with mem=200g

mem=224g, xen kernel will say alloc_bootmem_low can not get range..

xen kernel is from

http://people.redhat.com/dzickus/el5/92.el5/x86_64/

YH