RE: Re: [Xen-staging] [xen-3.1-testing] xend: Fix use ofPIFIsPhysical (takes no arguments).

RE: Re: [Xen-staging] [xen-3.1-testing] xend: Fix use ofPIFIsPhysical (takes no arguments).

[Krysan, Susan]

I am running on a Unisys ES7000 with changeset 15006, and I still see
this problem.  I believe it only happens when I stop the server,
reconfigure the amount of host processors and RAM, and then reboot. 

[2007-05-03 13:49:19 4204] INFO (SrvDaemon:331) Xend Daemon started
[2007-05-03 13:49:19 4204] INFO (SrvDaemon:335) Xend changeset: Wed May
02 15:27:10 2007 +0100 15006:b5cfbc8f7d2c.
[2007-05-03 13:49:19 4204] INFO (SrvDaemon:342) Xend version: Unknown.
[2007-05-03 13:49:21 4204] ERROR (SrvDaemon:353) Exception starting xend
(PIF is physical)
Traceback (most recent call last):
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
server/SrvDaemon.py", line 345, in run
    servers = SrvServer.create()
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
server/SrvServer.py", line 254, in create
    root.putChild('xend', SrvRoot())
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/web/S
rvDir.py", line 82, in get
    val = val.getobj()
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/web/S
rvDir.py", line 52, in getobj
    self.obj = klassobj()
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
XendNode.py", line 658, in instance
    inst = XendNode()
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
XendNode.py", line 168, in __init__
    XendPIF.recreate(pif, pif_uuid)
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
XendPIF.py", line 208, in recreate
    pif.destroy()
  File
"/home/unisys/xen-3.1-testing.hg/dist/install/usr/lib64/python/xen/xend/
XendPIF.py", line 309, in destroy
    raise PIFIsPhysical()
PIFIsPhysical: PIF is physical

Thanks,
Sue Krysan
Linux Systems Group
Unisys Corporation
 
-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tom Wilkie
Sent: Wednesday, May 02, 2007 12:16 PM
To: Alex Williamson
Cc: Pascal Bouchareine; Keir Fraser; xen-devel; Tom Wilkie
Subject: Re: [Xen-devel] Re: [Xen-staging] [xen-3.1-testing] xend: Fix
use ofPIFIsPhysical (takes no arguments).


On 2 May 2007, at 17:05, Alex Williamson wrote:
> On Wed, 2007-05-02 at 16:46 +0100, Tom Wilkie wrote:
>> Checked in a fix, and a fix for that fix.
>>
>> cset number 15001.
>
>    Confirmed, Thanks!
>
>> ps did you do something weird like turn the box off, remove a network

>> card, and turn it back on?  Thats the only way I can think of this 
>> bug getting triggered...  Or this could be related to the networks 
>> scripts not  being run when xend is started.
>
>    Removing a NIC from a powered off box doesn't seem so weird to me, 
> but no, the hardware config did not change at all.  The only 
> interesting config is that one port of a dual port e1000 NIC is hidden

> from dom0 using "pciback.hide=(0000:01:02.1)".  Otherwise it's a 
> standard Debian Etch system netbooting xen & xenlinux w/ matching 
> tools locally built and installed.  Thanks,

Okay, if you weren't changing the cards then it must have been the
previous bug mentioned by Pascal Bouchareine.

I've check in a fix for that too, cset 15002

Cheers

Tom

>
> 	Alex
>
> -- 
> Alex Williamson                             HP Open Source & Linux  
> Org.
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-staging] [xen-3.1-testing] xend: Fix use ofPIFIsPhysical (takes no arguments).

[Keir Fraser]

On 3/5/07 20:25, "Krysan, Susan" <KRYSANS@unisys.com> wrote:

> I am running on a Unisys ES7000 with changeset 15006, and I still see
> this problem.  I believe it only happens when I stop the server,
> reconfigure the amount of host processors and RAM, and then reboot.

Xen-unstable c/s 15006, I assume?

In that case your xend hasn't been properly re-installed: line 208 in your
Python backtrace changed from pif.destroy() to XendBase.destroy() in
changeset 15000. That was the fix for this bug. :-)

 -- Keir

Numa=on broken?

[Subrahmanian, Raj]

All,
It looks like numa=on is broken.
I have tried with the tip of the tree, with the changeset 14676 (which
fixes the numa=on problem) and the two patches that Ryan sent out. 
I am running with boot options acpi=on and numa=on.
I am running on an ES7000 with 2 nodes (2 cells of 8 processors and 32
GB memory each).

Thanks
Raj
PS. We(Unisys) are going to start running our weekly tests with default
numa=on. This weeks' tests brought out this problem.

 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                        
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Thu May  3 16:58:28 EDT 2007
 Latest ChangeSet: Thu May 03 11:22:58 2007 +0100 15007:c857bf38f015

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=512M acpi=on numa=on
(XEN)  0000000000000000 - 000000000009d000 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
(XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!



-----------------------------------------------------------------------

 __  __            _____  ___                     _        _     _      
 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                        
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Thu May  3 17:48:50 EDT 2007
 Latest ChangeSet: Sat Mar 31 12:20:31 2007 +0100 14676:c1578c694b39

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=512M acpi=on numa=on
(XEN)  0000000000000000 - 000000000009d000 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
(XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!


------------------------------------------------------------------------
-----
__  __            _____  ___                     _        _     _      
 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                        
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Thu May  3 18:07:41 EDT 2007
 Latest ChangeSet: Thu May 03 11:22:58 2007 +0100 15007:c857bf38f015

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=512M acpi=on numa=on
(XEN)  0000000000000000 - 000000000009d000 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
(XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!

Re: Numa=on broken?

[Ryan Harper]

* Subrahmanian, Raj <raj.subrahmanian@unisys.com> [2007-05-03 17:30]:
> All,
> It looks like numa=on is broken.
> I have tried with the tip of the tree, with the changeset 14676 (which
> fixes the numa=on problem) and the two patches that Ryan sent out. 
> I am running with boot options acpi=on and numa=on.
> I am running on an ES7000 with 2 nodes (2 cells of 8 processors and 32
> GB memory each).
> 
> Thanks
> Raj
> PS. We(Unisys) are going to start running our weekly tests with default
> numa=on. This weeks' tests brought out this problem.
> 
>  \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
>   \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
>   /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
>  /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
>                                                                         
>  http://www.cl.cam.ac.uk/netos/xen
>  University of Cambridge Computer Laboratory
> 
>  Xen version 3.0-unstable (root@site) (gcc version 4.1.2 20070115
> (prerelease) (SUSE Linux)) Thu May  3 16:58:28 EDT 2007
>  Latest ChangeSet: Thu May 03 11:22:58 2007 +0100 15007:c857bf38f015
> 
> (XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
> dom0_mem=512M acpi=on numa=on
> (XEN)  0000000000000000 - 000000000009d000 (usable)
> (XEN)  000000000009d800 - 00000000000a0000 (reserved)
> (XEN)  00000000000ce000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 0000000037e70000 (usable)
> (XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
> (XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
> (XEN)  0000000037f00000 - 00000000f0000000 (usable)
> (XEN)  00000000f8000000 - 00000000fec00000 (reserved)
> (XEN)  00000000fffc0000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000001008000000 (usable)
> (XEN) System RAM: 65407MB (66976820kB)
> (XEN) Cannot handle page request order 0!
> (XEN) Cannot handle page request order 2!
> (XEN) Cannot handle page request order 0!
> (XEN) Cannot handle page request order 2!
> (XEN) Cannot handle page request order 0!
> (XEN) Cannot handle page request order 2!
> (XEN) Cannot handle page request order 0!
> (XEN) Cannot handle page request order 2!
> (XEN) Cannot handle page request order 0!
> (XEN) Cannot handle page request order 2!

what repo and what changeset?  We fixed this issue a while ago, 

I just booted:

changeset:   15020:6f37c763bc0f from http://xenbits.xensource.com/xen-3.1-testing.hg

and

changeset:   15009:3a5722420de7 from http://xenbits.xensource.com/xen-unstable.hg

with no issues on my numa hardware.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: Numa=on broken?

[Subrahmanian, Raj]

>what repo and what changeset?  We fixed this issue a while ago, 
>
>I just booted:
>
>changeset:   15020:6f37c763bc0f from 
>http://xenbits.xensource.com/xen-3.1-testing.hg
>
>and
>
>changeset:   15009:3a5722420de7 from 
>http://xenbits.xensource.com/xen-unstable.hg
>
>with no issues on my numa hardware.
15007 from unstable.
I will try 15009 unstable and 15020 testing.
Raj

RE: Numa=on broken?

[Subrahmanian, Raj]

Ryan,
I did check out the mail discussion where you encountered and fixed
this.
I am seeing this again on 15009, unstable and in 15021 3.1. rc7.
I have included the debug messages in both runs.
Raj

 __  __            _____  ___                     _        _     _
 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Thu May  3 23:05:43 EDT 2007
 Latest ChangeSet: Thu May 03 19:25:47 2007 +0100 15009:3a5722420de7

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=512M acpi=on numa=on
(XEN)  0000000000000000 - 000000000009d000 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
(XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!
(XEN) Cannot handle page request order 0!

------------------------------------------------------------------------
----------------------------------

 __  __            _____  _   ___             _____
 \ \/ /___ _ __   |___ / / | / _ \    _ __ __|___  |
  \  // _ \ '_ \    |_ \ | || | | |__| '__/ __| / /
  /  \  __/ | | |  ___) || || |_| |__| | | (__ / /
 /_/\_\___|_| |_| |____(_)_(_)___/   |_|  \___/_/

 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.1.0-rc7 (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Fri May  4 01:52:44 EDT 2007
 Latest ChangeSet: Thu May 03 19:28:14 2007 +0100 15021:cc5800ecd71f

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=512M acpi=on numa=on
(XEN)  0000000000000000 - 000000000009d000 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
(XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) Cannot handle page request order 0!
(XEN) Cannot handle page request order 2!

Re: Numa=on broken?

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

* Subrahmanian, Raj <raj.subrahmanian@unisys.com> [2007-05-04 01:23]:
> Ryan,
> I did check out the mail discussion where you encountered and fixed
> this.
> I am seeing this again on 15009, unstable and in 15021 3.1. rc7.
> I have included the debug messages in both runs.

Hrm, well, we need to see which node the request is failing in so we can
figure out how to initialize the heap.  I'm attaching a patch that
should hopefully dump out which node the memory request is failing in.

Last time, it was the xmalloc() in xen/common/page_alloc.c in
init_heap_pages().

Give this patch a spin and email me the output.

I'd also be interested in your SRAT table output for this machine.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[-- Attachment #2: numa_debug.patch --]
[-- Type: text/plain, Size: 910 bytes --]

diff -r 3a5722420de7 xen/common/page_alloc.c
--- a/xen/common/page_alloc.c	Thu May 03 19:25:47 2007 +0100
+++ b/xen/common/page_alloc.c	Fri May 04 08:29:48 2007 -0500
@@ -353,6 +353,8 @@ static struct page_info *alloc_heap_page
     ASSERT(zone_lo <= zone_hi);
     ASSERT(zone_hi < NR_ZONES);
 
+    printk("%s: request for node %d\n", node);
+
     if ( unlikely(order > MAX_ORDER) )
         return NULL;
 
@@ -543,8 +545,10 @@ void init_heap_pages(
 
         if ( !avail[nid_curr] )
         {
+            printk("%s: attempting to initialize avail[%d]\n", nid_curr);
             avail[nid_curr] = xmalloc_array(unsigned long, NR_ZONES);
             memset(avail[nid_curr], 0, NR_ZONES * sizeof(long));
+            printk("%s: attempting to initialize _heap[%d]\n", nid_curr);
             _heap[nid_curr] = xmalloc(heap_by_zone_and_order_t);
             init_heap_block(_heap[nid_curr]);
         }

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Numa=on broken?

[Ryan Harper]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

* Ryan Harper <ryanh@us.ibm.com> [2007-05-04 08:34]:
> * Subrahmanian, Raj <raj.subrahmanian@unisys.com> [2007-05-04 01:23]:
> > Ryan,
> > I did check out the mail discussion where you encountered and fixed
> > this.
> > I am seeing this again on 15009, unstable and in 15021 3.1. rc7.
> > I have included the debug messages in both runs.
> 
> Hrm, well, we need to see which node the request is failing in so we can
> figure out how to initialize the heap.  I'm attaching a patch that
> should hopefully dump out which node the memory request is failing in.
> 
> Last time, it was the xmalloc() in xen/common/page_alloc.c in
> init_heap_pages().
> 
> Give this patch a spin and email me the output.

Here is one that actually compiles.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

[-- Attachment #2: numa_debug.patch --]
[-- Type: text/plain, Size: 940 bytes --]

diff -r 3a5722420de7 xen/common/page_alloc.c
--- a/xen/common/page_alloc.c	Thu May 03 19:25:47 2007 +0100
+++ b/xen/common/page_alloc.c	Fri May 04 08:36:42 2007 -0500
@@ -353,6 +353,8 @@ static struct page_info *alloc_heap_page
     ASSERT(zone_lo <= zone_hi);
     ASSERT(zone_hi < NR_ZONES);
 
+    printk("%s: request for node %d\n", __func__, node);
+
     if ( unlikely(order > MAX_ORDER) )
         return NULL;
 
@@ -543,8 +545,10 @@ void init_heap_pages(
 
         if ( !avail[nid_curr] )
         {
+            printk("%s: attempting to initialize avail[%d]\n", __func__, nid_curr);
             avail[nid_curr] = xmalloc_array(unsigned long, NR_ZONES);
             memset(avail[nid_curr], 0, NR_ZONES * sizeof(long));
+            printk("%s: attempting to initialize _heap[%d]\n", __func__, nid_curr);
             _heap[nid_curr] = xmalloc(heap_by_zone_and_order_t);
             init_heap_block(_heap[nid_curr]);
         }

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Numa=on broken?

[Subrahmanian, Raj]

Ryan
I will try this out once I get the machine.
Incidentally, it seems to work with numa=on on a single-node machine.
Raj 

>-----Original Message-----
>From: Ryan Harper [mailto:ryanh@us.ibm.com] 
>Sent: Friday, May 04, 2007 9:38 AM
>To: Ryan Harper
>Cc: Subrahmanian, Raj; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] Numa=on broken?
>
>* Ryan Harper <ryanh@us.ibm.com> [2007-05-04 08:34]:
>> * Subrahmanian, Raj <raj.subrahmanian@unisys.com> [2007-05-04 01:23]:
>> > Ryan,
>> > I did check out the mail discussion where you encountered 
>and fixed 
>> > this.
>> > I am seeing this again on 15009, unstable and in 15021 3.1. rc7.
>> > I have included the debug messages in both runs.
>> 
>> Hrm, well, we need to see which node the request is failing in so we 
>> can figure out how to initialize the heap.  I'm attaching a 
>patch that 
>> should hopefully dump out which node the memory request is 
>failing in.
>> 
>> Last time, it was the xmalloc() in xen/common/page_alloc.c in 
>> init_heap_pages().
>> 
>> Give this patch a spin and email me the output.
>
>Here is one that actually compiles.
>
>
>--
>Ryan Harper
>Software Engineer; Linux Technology Center IBM Corp., Austin, Tx
>(512) 838-9253   T/L: 678-9253
>ryanh@us.ibm.com
>

RE: Numa=on broken?

[Subrahmanian, Raj]

Ryan,
>> > Ryan,
>> > I did check out the mail discussion where you encountered 
>and fixed 
>> > this.
>> > I am seeing this again on 15009, unstable and in 15021 3.1. rc7.
>> > I have included the debug messages in both runs.
>> 
>> Hrm, well, we need to see which node the request is failing in so we 
>> can figure out how to initialize the heap.  I'm attaching a 
>patch that 
>> should hopefully dump out which node the memory request is 
>failing in.
>> 
>> Last time, it was the xmalloc() in xen/common/page_alloc.c in 
>> init_heap_pages().
>> 
>> Give this patch a spin and email me the output.
>
>Here is one that actually compiles.
Here's the output..
Unstable tree. Changeset 15009.
Thanks
Raj

 __  __            _____  ___                     _        _     _      

 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                        
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Sat May  5 00:04:27 EDT 2007
 Latest ChangeSet: Thu May 03 19:25:47 2007 +0100 15009:3a5722420de7

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=512M acpi=on numa=on
(XEN)  0000000000000000 - 000000000009d000 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed8000 (ACPI data)
(XEN)  0000000037ed8000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2!
(XEN) init_heap_pages: attempting to initialize avail[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 0!
(XEN) init_heap_pages: attempting to initialize _heap[1]
(XEN) alloc_heap_pages: request for node 0
(XEN) Cannot handle page request order 2! 

Re: Numa=on broken?

[Ryan Harper]

* Subrahmanian, Raj <raj.subrahmanian@unisys.com> [2007-05-07 15:25]:
> Ryan,
> >> > Ryan,
> >> > I did check out the mail discussion where you encountered 
> >and fixed 
> >> > this.
> >> > I am seeing this again on 15009, unstable and in 15021 3.1. rc7.
> >> > I have included the debug messages in both runs.
> >> 
> >> Hrm, well, we need to see which node the request is failing in so we 
> >> can figure out how to initialize the heap.  I'm attaching a 
> >patch that 
> >> should hopefully dump out which node the memory request is 
> >failing in.
> >> 
> >> Last time, it was the xmalloc() in xen/common/page_alloc.c in 
> >> init_heap_pages().
> >> 
> >> Give this patch a spin and email me the output.
> >
> >Here is one that actually compiles.
> Here's the output..
> Unstable tree. Changeset 15009.

Do you have the SRAT table information, we need to know if you have any 
memory in node0.  I believe this code relies on the xenheap residing in 
node0.
   
The call chain is likely:
   
xen/arch/x86/setup.c:calls init_xenheap_pages()
xen/common/page_alloc.c: init_heap_pages()

if the memory being added to the heap belongs to node1, the structure
is dynamically allocated using xmalloc() which will request a page
from the heap which is no memory is already in the heap, we fail.

That's what this looks like.  You can confirm this by adding some debug 
to the init_heap_pages() function in xen/common/page_alloc.c.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com