* Logical NUMA error during boot, and RFC patch
@ 2012-06-27 19:10 Andrew Cooper
2012-06-28 9:51 ` Jan Beulich
0 siblings, 1 reply; 4+ messages in thread
From: Andrew Cooper @ 2012-06-27 19:10 UTC (permalink / raw)
To: xen-devel@lists.xen.org, Jan Beulich, Keir Fraser, Dario Faggioli
[-- Attachment #1: Type: text/plain, Size: 953 bytes --]
Hello,
XenServer have recently acquired a quad-socket AMD Interlagos server and
I have been playing around with it, and discovered a logical error in
how Xen detects numa nodes.
The server has 8 NUMA nodes, 4 of which have memory attached (the even
nodes - see SRAT.dsl attached). This means that that
node_set_online(nodeid) gets called for each node with memory attached.
Later, in srat_detect_node(), node gets set to 0 if it was NUMA_NO_NODE,
or if not node_online(). This leads to all the processors on the odd
nodes being assigned to node 0, even though the odd nodes are present
(see interlagos-xl-info-n.log)
I present an RFC patch which changes srat_detect_node() to call
node_set_online() for each node, which appears to fix the logic.
Is this a sensible place to set the node online, or is there a better
way to fix this logic?
--
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com
[-- Attachment #2: SRAT.dsl --]
[-- Type: text/x-dsl, Size: 42713 bytes --]
/*
* Intel ACPI Component Architecture
* AML Disassembler version 20090521
*
* Disassembly of SRAT.dat, Wed Jun 27 18:03:58 2012
*
* ACPI Data Table [SRAT]
*
* Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue
*/
[000h 0000 4] Signature : "SRAT" /* System Resource Affinity Table */
[004h 0004 4] Table Length : 00000520
[008h 0008 1] Revision : 02
[009h 0009 1] Checksum : D4
[00Ah 0010 6] Oem ID : "AMD "
[010h 0016 8] Oem Table ID : "AGESA "
[018h 0024 4] Oem Revision : 00000001
[01Ch 0028 4] Asl Compiler ID : "AMD "
[020h 0032 4] Asl Compiler Revision : 00000001
[024h 0036 4] Table Revision : 00000001
[028h 0040 8] Reserved : 0000000000000000
[030h 0048 1] Subtable Type : 01 <Memory Affinity>
[031h 0049 1] Length : 28
[032h 0050 4] Proximity Domain : 00000000
[036h 0054 2] Reserved : 0000
[038h 0056 8] Base Address : 0000000000000000
[040h 0064 8] Address Length : 00000000000A0000
[048h 0072 4] Reserved : 00000000
[04Ch 0076 4] Flags (decoded below) : 00000001
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 0
[050h 0080 8] Reserved : 0000000000000000
[058h 0088 1] Subtable Type : 01 <Memory Affinity>
[059h 0089 1] Length : 28
[05Ah 0090 4] Proximity Domain : 00000000
[05Eh 0094 2] Reserved : 0000
[060h 0096 8] Base Address : 0000000000100000
[068h 0104 8] Address Length : 00000000BFF00000
[070h 0112 4] Reserved : 00000000
[074h 0116 4] Flags (decoded below) : 00000001
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 0
[078h 0120 8] Reserved : 0000000000000000
[080h 0128 1] Subtable Type : 01 <Memory Affinity>
[081h 0129 1] Length : 28
[082h 0130 4] Proximity Domain : 00000000
[086h 0134 2] Reserved : 0000
[088h 0136 8] Base Address : 0000000100000000
[090h 0144 8] Address Length : 0000000140000000
[098h 0152 4] Reserved : 00000000
[09Ch 0156 4] Flags (decoded below) : 00000001
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 0
[0A0h 0160 8] Reserved : 0000000000000000
[0A8h 0168 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[0A9h 0169 1] Length : 10
[0AAh 0170 1] Proximity Domain Low(8) : 00
[0ABh 0171 1] Apic ID : 00
[0ACh 0172 4] Flags (decoded below) : 00000001
Enabled : 1
[0B0h 0176 1] Local Sapic EID : 00
[0B1h 0177 3] Proximity Domain High(24) : 000000
[0B4h 0180 4] Reserved : 00000000
[0B8h 0184 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[0B9h 0185 1] Length : 10
[0BAh 0186 1] Proximity Domain Low(8) : 00
[0BBh 0187 1] Apic ID : 01
[0BCh 0188 4] Flags (decoded below) : 00000001
Enabled : 1
[0C0h 0192 1] Local Sapic EID : 00
[0C1h 0193 3] Proximity Domain High(24) : 000000
[0C4h 0196 4] Reserved : 00000000
[0C8h 0200 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[0C9h 0201 1] Length : 10
[0CAh 0202 1] Proximity Domain Low(8) : 00
[0CBh 0203 1] Apic ID : 02
[0CCh 0204 4] Flags (decoded below) : 00000001
Enabled : 1
[0D0h 0208 1] Local Sapic EID : 00
[0D1h 0209 3] Proximity Domain High(24) : 000000
[0D4h 0212 4] Reserved : 00000000
[0D8h 0216 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[0D9h 0217 1] Length : 10
[0DAh 0218 1] Proximity Domain Low(8) : 00
[0DBh 0219 1] Apic ID : 03
[0DCh 0220 4] Flags (decoded below) : 00000001
Enabled : 1
[0E0h 0224 1] Local Sapic EID : 00
[0E1h 0225 3] Proximity Domain High(24) : 000000
[0E4h 0228 4] Reserved : 00000000
[0E8h 0232 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[0E9h 0233 1] Length : 10
[0EAh 0234 1] Proximity Domain Low(8) : 00
[0EBh 0235 1] Apic ID : 04
[0ECh 0236 4] Flags (decoded below) : 00000001
Enabled : 1
[0F0h 0240 1] Local Sapic EID : 00
[0F1h 0241 3] Proximity Domain High(24) : 000000
[0F4h 0244 4] Reserved : 00000000
[0F8h 0248 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[0F9h 0249 1] Length : 10
[0FAh 0250 1] Proximity Domain Low(8) : 00
[0FBh 0251 1] Apic ID : 05
[0FCh 0252 4] Flags (decoded below) : 00000001
Enabled : 1
[100h 0256 1] Local Sapic EID : 00
[101h 0257 3] Proximity Domain High(24) : 000000
[104h 0260 4] Reserved : 00000000
[108h 0264 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[109h 0265 1] Length : 10
[10Ah 0266 1] Proximity Domain Low(8) : 00
[10Bh 0267 1] Apic ID : 06
[10Ch 0268 4] Flags (decoded below) : 00000001
Enabled : 1
[110h 0272 1] Local Sapic EID : 00
[111h 0273 3] Proximity Domain High(24) : 000000
[114h 0276 4] Reserved : 00000000
[118h 0280 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[119h 0281 1] Length : 10
[11Ah 0282 1] Proximity Domain Low(8) : 00
[11Bh 0283 1] Apic ID : 07
[11Ch 0284 4] Flags (decoded below) : 00000001
Enabled : 1
[120h 0288 1] Local Sapic EID : 00
[121h 0289 3] Proximity Domain High(24) : 000000
[124h 0292 4] Reserved : 00000000
[128h 0296 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[129h 0297 1] Length : 10
[12Ah 0298 1] Proximity Domain Low(8) : 01
[12Bh 0299 1] Apic ID : 08
[12Ch 0300 4] Flags (decoded below) : 00000001
Enabled : 1
[130h 0304 1] Local Sapic EID : 00
[131h 0305 3] Proximity Domain High(24) : 000000
[134h 0308 4] Reserved : 00000000
[138h 0312 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[139h 0313 1] Length : 10
[13Ah 0314 1] Proximity Domain Low(8) : 01
[13Bh 0315 1] Apic ID : 09
[13Ch 0316 4] Flags (decoded below) : 00000001
Enabled : 1
[140h 0320 1] Local Sapic EID : 00
[141h 0321 3] Proximity Domain High(24) : 000000
[144h 0324 4] Reserved : 00000000
[148h 0328 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[149h 0329 1] Length : 10
[14Ah 0330 1] Proximity Domain Low(8) : 01
[14Bh 0331 1] Apic ID : 0A
[14Ch 0332 4] Flags (decoded below) : 00000001
Enabled : 1
[150h 0336 1] Local Sapic EID : 00
[151h 0337 3] Proximity Domain High(24) : 000000
[154h 0340 4] Reserved : 00000000
[158h 0344 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[159h 0345 1] Length : 10
[15Ah 0346 1] Proximity Domain Low(8) : 01
[15Bh 0347 1] Apic ID : 0B
[15Ch 0348 4] Flags (decoded below) : 00000001
Enabled : 1
[160h 0352 1] Local Sapic EID : 00
[161h 0353 3] Proximity Domain High(24) : 000000
[164h 0356 4] Reserved : 00000000
[168h 0360 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[169h 0361 1] Length : 10
[16Ah 0362 1] Proximity Domain Low(8) : 01
[16Bh 0363 1] Apic ID : 0C
[16Ch 0364 4] Flags (decoded below) : 00000001
Enabled : 1
[170h 0368 1] Local Sapic EID : 00
[171h 0369 3] Proximity Domain High(24) : 000000
[174h 0372 4] Reserved : 00000000
[178h 0376 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[179h 0377 1] Length : 10
[17Ah 0378 1] Proximity Domain Low(8) : 01
[17Bh 0379 1] Apic ID : 0D
[17Ch 0380 4] Flags (decoded below) : 00000001
Enabled : 1
[180h 0384 1] Local Sapic EID : 00
[181h 0385 3] Proximity Domain High(24) : 000000
[184h 0388 4] Reserved : 00000000
[188h 0392 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[189h 0393 1] Length : 10
[18Ah 0394 1] Proximity Domain Low(8) : 01
[18Bh 0395 1] Apic ID : 0E
[18Ch 0396 4] Flags (decoded below) : 00000001
Enabled : 1
[190h 0400 1] Local Sapic EID : 00
[191h 0401 3] Proximity Domain High(24) : 000000
[194h 0404 4] Reserved : 00000000
[198h 0408 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[199h 0409 1] Length : 10
[19Ah 0410 1] Proximity Domain Low(8) : 01
[19Bh 0411 1] Apic ID : 0F
[19Ch 0412 4] Flags (decoded below) : 00000001
Enabled : 1
[1A0h 0416 1] Local Sapic EID : 00
[1A1h 0417 3] Proximity Domain High(24) : 000000
[1A4h 0420 4] Reserved : 00000000
[1A8h 0424 1] Subtable Type : 01 <Memory Affinity>
[1A9h 0425 1] Length : 28
[1AAh 0426 4] Proximity Domain : 00000002
[1AEh 0430 2] Reserved : 0000
[1B0h 0432 8] Base Address : 0000000240000000
[1B8h 0440 8] Address Length : 0000000200000000
[1C0h 0448 4] Reserved : 00000000
[1C4h 0452 4] Flags (decoded below) : 00000001
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 0
[1C8h 0456 8] Reserved : 0000000000000000
[1D0h 0464 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[1D1h 0465 1] Length : 10
[1D2h 0466 1] Proximity Domain Low(8) : 02
[1D3h 0467 1] Apic ID : 20
[1D4h 0468 4] Flags (decoded below) : 00000001
Enabled : 1
[1D8h 0472 1] Local Sapic EID : 00
[1D9h 0473 3] Proximity Domain High(24) : 000000
[1DCh 0476 4] Reserved : 00000000
[1E0h 0480 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[1E1h 0481 1] Length : 10
[1E2h 0482 1] Proximity Domain Low(8) : 02
[1E3h 0483 1] Apic ID : 21
[1E4h 0484 4] Flags (decoded below) : 00000001
Enabled : 1
[1E8h 0488 1] Local Sapic EID : 00
[1E9h 0489 3] Proximity Domain High(24) : 000000
[1ECh 0492 4] Reserved : 00000000
[1F0h 0496 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[1F1h 0497 1] Length : 10
[1F2h 0498 1] Proximity Domain Low(8) : 02
[1F3h 0499 1] Apic ID : 22
[1F4h 0500 4] Flags (decoded below) : 00000001
Enabled : 1
[1F8h 0504 1] Local Sapic EID : 00
[1F9h 0505 3] Proximity Domain High(24) : 000000
[1FCh 0508 4] Reserved : 00000000
[200h 0512 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[201h 0513 1] Length : 10
[202h 0514 1] Proximity Domain Low(8) : 02
[203h 0515 1] Apic ID : 23
[204h 0516 4] Flags (decoded below) : 00000001
Enabled : 1
[208h 0520 1] Local Sapic EID : 00
[209h 0521 3] Proximity Domain High(24) : 000000
[20Ch 0524 4] Reserved : 00000000
[210h 0528 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[211h 0529 1] Length : 10
[212h 0530 1] Proximity Domain Low(8) : 02
[213h 0531 1] Apic ID : 24
[214h 0532 4] Flags (decoded below) : 00000001
Enabled : 1
[218h 0536 1] Local Sapic EID : 00
[219h 0537 3] Proximity Domain High(24) : 000000
[21Ch 0540 4] Reserved : 00000000
[220h 0544 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[221h 0545 1] Length : 10
[222h 0546 1] Proximity Domain Low(8) : 02
[223h 0547 1] Apic ID : 25
[224h 0548 4] Flags (decoded below) : 00000001
Enabled : 1
[228h 0552 1] Local Sapic EID : 00
[229h 0553 3] Proximity Domain High(24) : 000000
[22Ch 0556 4] Reserved : 00000000
[230h 0560 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[231h 0561 1] Length : 10
[232h 0562 1] Proximity Domain Low(8) : 02
[233h 0563 1] Apic ID : 26
[234h 0564 4] Flags (decoded below) : 00000001
Enabled : 1
[238h 0568 1] Local Sapic EID : 00
[239h 0569 3] Proximity Domain High(24) : 000000
[23Ch 0572 4] Reserved : 00000000
[240h 0576 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[241h 0577 1] Length : 10
[242h 0578 1] Proximity Domain Low(8) : 02
[243h 0579 1] Apic ID : 27
[244h 0580 4] Flags (decoded below) : 00000001
Enabled : 1
[248h 0584 1] Local Sapic EID : 00
[249h 0585 3] Proximity Domain High(24) : 000000
[24Ch 0588 4] Reserved : 00000000
[250h 0592 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[251h 0593 1] Length : 10
[252h 0594 1] Proximity Domain Low(8) : 03
[253h 0595 1] Apic ID : 28
[254h 0596 4] Flags (decoded below) : 00000001
Enabled : 1
[258h 0600 1] Local Sapic EID : 00
[259h 0601 3] Proximity Domain High(24) : 000000
[25Ch 0604 4] Reserved : 00000000
[260h 0608 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[261h 0609 1] Length : 10
[262h 0610 1] Proximity Domain Low(8) : 03
[263h 0611 1] Apic ID : 29
[264h 0612 4] Flags (decoded below) : 00000001
Enabled : 1
[268h 0616 1] Local Sapic EID : 00
[269h 0617 3] Proximity Domain High(24) : 000000
[26Ch 0620 4] Reserved : 00000000
[270h 0624 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[271h 0625 1] Length : 10
[272h 0626 1] Proximity Domain Low(8) : 03
[273h 0627 1] Apic ID : 2A
[274h 0628 4] Flags (decoded below) : 00000001
Enabled : 1
[278h 0632 1] Local Sapic EID : 00
[279h 0633 3] Proximity Domain High(24) : 000000
[27Ch 0636 4] Reserved : 00000000
[280h 0640 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[281h 0641 1] Length : 10
[282h 0642 1] Proximity Domain Low(8) : 03
[283h 0643 1] Apic ID : 2B
[284h 0644 4] Flags (decoded below) : 00000001
Enabled : 1
[288h 0648 1] Local Sapic EID : 00
[289h 0649 3] Proximity Domain High(24) : 000000
[28Ch 0652 4] Reserved : 00000000
[290h 0656 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[291h 0657 1] Length : 10
[292h 0658 1] Proximity Domain Low(8) : 03
[293h 0659 1] Apic ID : 2C
[294h 0660 4] Flags (decoded below) : 00000001
Enabled : 1
[298h 0664 1] Local Sapic EID : 00
[299h 0665 3] Proximity Domain High(24) : 000000
[29Ch 0668 4] Reserved : 00000000
[2A0h 0672 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[2A1h 0673 1] Length : 10
[2A2h 0674 1] Proximity Domain Low(8) : 03
[2A3h 0675 1] Apic ID : 2D
[2A4h 0676 4] Flags (decoded below) : 00000001
Enabled : 1
[2A8h 0680 1] Local Sapic EID : 00
[2A9h 0681 3] Proximity Domain High(24) : 000000
[2ACh 0684 4] Reserved : 00000000
[2B0h 0688 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[2B1h 0689 1] Length : 10
[2B2h 0690 1] Proximity Domain Low(8) : 03
[2B3h 0691 1] Apic ID : 2E
[2B4h 0692 4] Flags (decoded below) : 00000001
Enabled : 1
[2B8h 0696 1] Local Sapic EID : 00
[2B9h 0697 3] Proximity Domain High(24) : 000000
[2BCh 0700 4] Reserved : 00000000
[2C0h 0704 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[2C1h 0705 1] Length : 10
[2C2h 0706 1] Proximity Domain Low(8) : 03
[2C3h 0707 1] Apic ID : 2F
[2C4h 0708 4] Flags (decoded below) : 00000001
Enabled : 1
[2C8h 0712 1] Local Sapic EID : 00
[2C9h 0713 3] Proximity Domain High(24) : 000000
[2CCh 0716 4] Reserved : 00000000
[2D0h 0720 1] Subtable Type : 01 <Memory Affinity>
[2D1h 0721 1] Length : 28
[2D2h 0722 4] Proximity Domain : 00000004
[2D6h 0726 2] Reserved : 0000
[2D8h 0728 8] Base Address : 0000000440000000
[2E0h 0736 8] Address Length : 0000000200000000
[2E8h 0744 4] Reserved : 00000000
[2ECh 0748 4] Flags (decoded below) : 00000001
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 0
[2F0h 0752 8] Reserved : 0000000000000000
[2F8h 0760 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[2F9h 0761 1] Length : 10
[2FAh 0762 1] Proximity Domain Low(8) : 04
[2FBh 0763 1] Apic ID : 40
[2FCh 0764 4] Flags (decoded below) : 00000001
Enabled : 1
[300h 0768 1] Local Sapic EID : 00
[301h 0769 3] Proximity Domain High(24) : 000000
[304h 0772 4] Reserved : 00000000
[308h 0776 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[309h 0777 1] Length : 10
[30Ah 0778 1] Proximity Domain Low(8) : 04
[30Bh 0779 1] Apic ID : 41
[30Ch 0780 4] Flags (decoded below) : 00000001
Enabled : 1
[310h 0784 1] Local Sapic EID : 00
[311h 0785 3] Proximity Domain High(24) : 000000
[314h 0788 4] Reserved : 00000000
[318h 0792 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[319h 0793 1] Length : 10
[31Ah 0794 1] Proximity Domain Low(8) : 04
[31Bh 0795 1] Apic ID : 42
[31Ch 0796 4] Flags (decoded below) : 00000001
Enabled : 1
[320h 0800 1] Local Sapic EID : 00
[321h 0801 3] Proximity Domain High(24) : 000000
[324h 0804 4] Reserved : 00000000
[328h 0808 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[329h 0809 1] Length : 10
[32Ah 0810 1] Proximity Domain Low(8) : 04
[32Bh 0811 1] Apic ID : 43
[32Ch 0812 4] Flags (decoded below) : 00000001
Enabled : 1
[330h 0816 1] Local Sapic EID : 00
[331h 0817 3] Proximity Domain High(24) : 000000
[334h 0820 4] Reserved : 00000000
[338h 0824 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[339h 0825 1] Length : 10
[33Ah 0826 1] Proximity Domain Low(8) : 04
[33Bh 0827 1] Apic ID : 44
[33Ch 0828 4] Flags (decoded below) : 00000001
Enabled : 1
[340h 0832 1] Local Sapic EID : 00
[341h 0833 3] Proximity Domain High(24) : 000000
[344h 0836 4] Reserved : 00000000
[348h 0840 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[349h 0841 1] Length : 10
[34Ah 0842 1] Proximity Domain Low(8) : 04
[34Bh 0843 1] Apic ID : 45
[34Ch 0844 4] Flags (decoded below) : 00000001
Enabled : 1
[350h 0848 1] Local Sapic EID : 00
[351h 0849 3] Proximity Domain High(24) : 000000
[354h 0852 4] Reserved : 00000000
[358h 0856 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[359h 0857 1] Length : 10
[35Ah 0858 1] Proximity Domain Low(8) : 04
[35Bh 0859 1] Apic ID : 46
[35Ch 0860 4] Flags (decoded below) : 00000001
Enabled : 1
[360h 0864 1] Local Sapic EID : 00
[361h 0865 3] Proximity Domain High(24) : 000000
[364h 0868 4] Reserved : 00000000
[368h 0872 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[369h 0873 1] Length : 10
[36Ah 0874 1] Proximity Domain Low(8) : 04
[36Bh 0875 1] Apic ID : 47
[36Ch 0876 4] Flags (decoded below) : 00000001
Enabled : 1
[370h 0880 1] Local Sapic EID : 00
[371h 0881 3] Proximity Domain High(24) : 000000
[374h 0884 4] Reserved : 00000000
[378h 0888 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[379h 0889 1] Length : 10
[37Ah 0890 1] Proximity Domain Low(8) : 05
[37Bh 0891 1] Apic ID : 48
[37Ch 0892 4] Flags (decoded below) : 00000001
Enabled : 1
[380h 0896 1] Local Sapic EID : 00
[381h 0897 3] Proximity Domain High(24) : 000000
[384h 0900 4] Reserved : 00000000
[388h 0904 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[389h 0905 1] Length : 10
[38Ah 0906 1] Proximity Domain Low(8) : 05
[38Bh 0907 1] Apic ID : 49
[38Ch 0908 4] Flags (decoded below) : 00000001
Enabled : 1
[390h 0912 1] Local Sapic EID : 00
[391h 0913 3] Proximity Domain High(24) : 000000
[394h 0916 4] Reserved : 00000000
[398h 0920 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[399h 0921 1] Length : 10
[39Ah 0922 1] Proximity Domain Low(8) : 05
[39Bh 0923 1] Apic ID : 4A
[39Ch 0924 4] Flags (decoded below) : 00000001
Enabled : 1
[3A0h 0928 1] Local Sapic EID : 00
[3A1h 0929 3] Proximity Domain High(24) : 000000
[3A4h 0932 4] Reserved : 00000000
[3A8h 0936 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[3A9h 0937 1] Length : 10
[3AAh 0938 1] Proximity Domain Low(8) : 05
[3ABh 0939 1] Apic ID : 4B
[3ACh 0940 4] Flags (decoded below) : 00000001
Enabled : 1
[3B0h 0944 1] Local Sapic EID : 00
[3B1h 0945 3] Proximity Domain High(24) : 000000
[3B4h 0948 4] Reserved : 00000000
[3B8h 0952 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[3B9h 0953 1] Length : 10
[3BAh 0954 1] Proximity Domain Low(8) : 05
[3BBh 0955 1] Apic ID : 4C
[3BCh 0956 4] Flags (decoded below) : 00000001
Enabled : 1
[3C0h 0960 1] Local Sapic EID : 00
[3C1h 0961 3] Proximity Domain High(24) : 000000
[3C4h 0964 4] Reserved : 00000000
[3C8h 0968 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[3C9h 0969 1] Length : 10
[3CAh 0970 1] Proximity Domain Low(8) : 05
[3CBh 0971 1] Apic ID : 4D
[3CCh 0972 4] Flags (decoded below) : 00000001
Enabled : 1
[3D0h 0976 1] Local Sapic EID : 00
[3D1h 0977 3] Proximity Domain High(24) : 000000
[3D4h 0980 4] Reserved : 00000000
[3D8h 0984 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[3D9h 0985 1] Length : 10
[3DAh 0986 1] Proximity Domain Low(8) : 05
[3DBh 0987 1] Apic ID : 4E
[3DCh 0988 4] Flags (decoded below) : 00000001
Enabled : 1
[3E0h 0992 1] Local Sapic EID : 00
[3E1h 0993 3] Proximity Domain High(24) : 000000
[3E4h 0996 4] Reserved : 00000000
[3E8h 1000 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[3E9h 1001 1] Length : 10
[3EAh 1002 1] Proximity Domain Low(8) : 05
[3EBh 1003 1] Apic ID : 4F
[3ECh 1004 4] Flags (decoded below) : 00000001
Enabled : 1
[3F0h 1008 1] Local Sapic EID : 00
[3F1h 1009 3] Proximity Domain High(24) : 000000
[3F4h 1012 4] Reserved : 00000000
[3F8h 1016 1] Subtable Type : 01 <Memory Affinity>
[3F9h 1017 1] Length : 28
[3FAh 1018 4] Proximity Domain : 00000006
[3FEh 1022 2] Reserved : 0000
[400h 1024 8] Base Address : 0000000640000000
[408h 1032 8] Address Length : 00000001FF000000
[410h 1040 4] Reserved : 00000000
[414h 1044 4] Flags (decoded below) : 00000001
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 0
[418h 1048 8] Reserved : 0000000000000000
[420h 1056 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[421h 1057 1] Length : 10
[422h 1058 1] Proximity Domain Low(8) : 06
[423h 1059 1] Apic ID : 60
[424h 1060 4] Flags (decoded below) : 00000001
Enabled : 1
[428h 1064 1] Local Sapic EID : 00
[429h 1065 3] Proximity Domain High(24) : 000000
[42Ch 1068 4] Reserved : 00000000
[430h 1072 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[431h 1073 1] Length : 10
[432h 1074 1] Proximity Domain Low(8) : 06
[433h 1075 1] Apic ID : 61
[434h 1076 4] Flags (decoded below) : 00000001
Enabled : 1
[438h 1080 1] Local Sapic EID : 00
[439h 1081 3] Proximity Domain High(24) : 000000
[43Ch 1084 4] Reserved : 00000000
[440h 1088 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[441h 1089 1] Length : 10
[442h 1090 1] Proximity Domain Low(8) : 06
[443h 1091 1] Apic ID : 62
[444h 1092 4] Flags (decoded below) : 00000001
Enabled : 1
[448h 1096 1] Local Sapic EID : 00
[449h 1097 3] Proximity Domain High(24) : 000000
[44Ch 1100 4] Reserved : 00000000
[450h 1104 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[451h 1105 1] Length : 10
[452h 1106 1] Proximity Domain Low(8) : 06
[453h 1107 1] Apic ID : 63
[454h 1108 4] Flags (decoded below) : 00000001
Enabled : 1
[458h 1112 1] Local Sapic EID : 00
[459h 1113 3] Proximity Domain High(24) : 000000
[45Ch 1116 4] Reserved : 00000000
[460h 1120 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[461h 1121 1] Length : 10
[462h 1122 1] Proximity Domain Low(8) : 06
[463h 1123 1] Apic ID : 64
[464h 1124 4] Flags (decoded below) : 00000001
Enabled : 1
[468h 1128 1] Local Sapic EID : 00
[469h 1129 3] Proximity Domain High(24) : 000000
[46Ch 1132 4] Reserved : 00000000
[470h 1136 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[471h 1137 1] Length : 10
[472h 1138 1] Proximity Domain Low(8) : 06
[473h 1139 1] Apic ID : 65
[474h 1140 4] Flags (decoded below) : 00000001
Enabled : 1
[478h 1144 1] Local Sapic EID : 00
[479h 1145 3] Proximity Domain High(24) : 000000
[47Ch 1148 4] Reserved : 00000000
[480h 1152 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[481h 1153 1] Length : 10
[482h 1154 1] Proximity Domain Low(8) : 06
[483h 1155 1] Apic ID : 66
[484h 1156 4] Flags (decoded below) : 00000001
Enabled : 1
[488h 1160 1] Local Sapic EID : 00
[489h 1161 3] Proximity Domain High(24) : 000000
[48Ch 1164 4] Reserved : 00000000
[490h 1168 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[491h 1169 1] Length : 10
[492h 1170 1] Proximity Domain Low(8) : 06
[493h 1171 1] Apic ID : 67
[494h 1172 4] Flags (decoded below) : 00000001
Enabled : 1
[498h 1176 1] Local Sapic EID : 00
[499h 1177 3] Proximity Domain High(24) : 000000
[49Ch 1180 4] Reserved : 00000000
[4A0h 1184 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[4A1h 1185 1] Length : 10
[4A2h 1186 1] Proximity Domain Low(8) : 07
[4A3h 1187 1] Apic ID : 68
[4A4h 1188 4] Flags (decoded below) : 00000001
Enabled : 1
[4A8h 1192 1] Local Sapic EID : 00
[4A9h 1193 3] Proximity Domain High(24) : 000000
[4ACh 1196 4] Reserved : 00000000
[4B0h 1200 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[4B1h 1201 1] Length : 10
[4B2h 1202 1] Proximity Domain Low(8) : 07
[4B3h 1203 1] Apic ID : 69
[4B4h 1204 4] Flags (decoded below) : 00000001
Enabled : 1
[4B8h 1208 1] Local Sapic EID : 00
[4B9h 1209 3] Proximity Domain High(24) : 000000
[4BCh 1212 4] Reserved : 00000000
[4C0h 1216 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[4C1h 1217 1] Length : 10
[4C2h 1218 1] Proximity Domain Low(8) : 07
[4C3h 1219 1] Apic ID : 6A
[4C4h 1220 4] Flags (decoded below) : 00000001
Enabled : 1
[4C8h 1224 1] Local Sapic EID : 00
[4C9h 1225 3] Proximity Domain High(24) : 000000
[4CCh 1228 4] Reserved : 00000000
[4D0h 1232 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[4D1h 1233 1] Length : 10
[4D2h 1234 1] Proximity Domain Low(8) : 07
[4D3h 1235 1] Apic ID : 6B
[4D4h 1236 4] Flags (decoded below) : 00000001
Enabled : 1
[4D8h 1240 1] Local Sapic EID : 00
[4D9h 1241 3] Proximity Domain High(24) : 000000
[4DCh 1244 4] Reserved : 00000000
[4E0h 1248 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[4E1h 1249 1] Length : 10
[4E2h 1250 1] Proximity Domain Low(8) : 07
[4E3h 1251 1] Apic ID : 6C
[4E4h 1252 4] Flags (decoded below) : 00000001
Enabled : 1
[4E8h 1256 1] Local Sapic EID : 00
[4E9h 1257 3] Proximity Domain High(24) : 000000
[4ECh 1260 4] Reserved : 00000000
[4F0h 1264 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[4F1h 1265 1] Length : 10
[4F2h 1266 1] Proximity Domain Low(8) : 07
[4F3h 1267 1] Apic ID : 6D
[4F4h 1268 4] Flags (decoded below) : 00000001
Enabled : 1
[4F8h 1272 1] Local Sapic EID : 00
[4F9h 1273 3] Proximity Domain High(24) : 000000
[4FCh 1276 4] Reserved : 00000000
[500h 1280 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[501h 1281 1] Length : 10
[502h 1282 1] Proximity Domain Low(8) : 07
[503h 1283 1] Apic ID : 6E
[504h 1284 4] Flags (decoded below) : 00000001
Enabled : 1
[508h 1288 1] Local Sapic EID : 00
[509h 1289 3] Proximity Domain High(24) : 000000
[50Ch 1292 4] Reserved : 00000000
[510h 1296 1] Subtable Type : 00 <Processor Local APIC/SAPIC Affinity>
[511h 1297 1] Length : 10
[512h 1298 1] Proximity Domain Low(8) : 07
[513h 1299 1] Apic ID : 6F
[514h 1300 4] Flags (decoded below) : 00000001
Enabled : 1
[518h 1304 1] Local Sapic EID : 00
[519h 1305 3] Proximity Domain High(24) : 000000
[51Ch 1308 4] Reserved : 00000000
Raw Table Data
0000: 53 52 41 54 20 05 00 00 02 D4 41 4D 44 20 20 20 SRAT .....AMD
0010: 41 47 45 53 41 20 20 20 01 00 00 00 41 4D 44 20 AGESA ....AMD
0020: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0030: 01 28 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .(..............
0040: 00 00 0A 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
0050: 00 00 00 00 00 00 00 00 01 28 00 00 00 00 00 00 .........(......
0060: 00 00 10 00 00 00 00 00 00 00 F0 BF 00 00 00 00 ................
0070: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0080: 01 28 00 00 00 00 00 00 00 00 00 00 01 00 00 00 .(..............
0090: 00 00 00 40 01 00 00 00 00 00 00 00 01 00 00 00 ...@............
00A0: 00 00 00 00 00 00 00 00 00 10 00 00 01 00 00 00 ................
00B0: 00 00 00 00 00 00 00 00 00 10 00 01 01 00 00 00 ................
00C0: 00 00 00 00 00 00 00 00 00 10 00 02 01 00 00 00 ................
00D0: 00 00 00 00 00 00 00 00 00 10 00 03 01 00 00 00 ................
00E0: 00 00 00 00 00 00 00 00 00 10 00 04 01 00 00 00 ................
00F0: 00 00 00 00 00 00 00 00 00 10 00 05 01 00 00 00 ................
0100: 00 00 00 00 00 00 00 00 00 10 00 06 01 00 00 00 ................
0110: 00 00 00 00 00 00 00 00 00 10 00 07 01 00 00 00 ................
0120: 00 00 00 00 00 00 00 00 00 10 01 08 01 00 00 00 ................
0130: 00 00 00 00 00 00 00 00 00 10 01 09 01 00 00 00 ................
0140: 00 00 00 00 00 00 00 00 00 10 01 0A 01 00 00 00 ................
0150: 00 00 00 00 00 00 00 00 00 10 01 0B 01 00 00 00 ................
0160: 00 00 00 00 00 00 00 00 00 10 01 0C 01 00 00 00 ................
0170: 00 00 00 00 00 00 00 00 00 10 01 0D 01 00 00 00 ................
0180: 00 00 00 00 00 00 00 00 00 10 01 0E 01 00 00 00 ................
0190: 00 00 00 00 00 00 00 00 00 10 01 0F 01 00 00 00 ................
01A0: 00 00 00 00 00 00 00 00 01 28 02 00 00 00 00 00 .........(......
01B0: 00 00 00 40 02 00 00 00 00 00 00 00 02 00 00 00 ...@............
01C0: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
01D0: 00 10 02 20 01 00 00 00 00 00 00 00 00 00 00 00 ... ............
01E0: 00 10 02 21 01 00 00 00 00 00 00 00 00 00 00 00 ...!............
01F0: 00 10 02 22 01 00 00 00 00 00 00 00 00 00 00 00 ..."............
0200: 00 10 02 23 01 00 00 00 00 00 00 00 00 00 00 00 ...#............
0210: 00 10 02 24 01 00 00 00 00 00 00 00 00 00 00 00 ...$............
0220: 00 10 02 25 01 00 00 00 00 00 00 00 00 00 00 00 ...%............
0230: 00 10 02 26 01 00 00 00 00 00 00 00 00 00 00 00 ...&............
0240: 00 10 02 27 01 00 00 00 00 00 00 00 00 00 00 00 ...'............
0250: 00 10 03 28 01 00 00 00 00 00 00 00 00 00 00 00 ...(............
0260: 00 10 03 29 01 00 00 00 00 00 00 00 00 00 00 00 ...)............
0270: 00 10 03 2A 01 00 00 00 00 00 00 00 00 00 00 00 ...*............
0280: 00 10 03 2B 01 00 00 00 00 00 00 00 00 00 00 00 ...+............
0290: 00 10 03 2C 01 00 00 00 00 00 00 00 00 00 00 00 ...,............
02A0: 00 10 03 2D 01 00 00 00 00 00 00 00 00 00 00 00 ...-............
02B0: 00 10 03 2E 01 00 00 00 00 00 00 00 00 00 00 00 ................
02C0: 00 10 03 2F 01 00 00 00 00 00 00 00 00 00 00 00 .../............
02D0: 01 28 04 00 00 00 00 00 00 00 00 40 04 00 00 00 .(.........@....
02E0: 00 00 00 00 02 00 00 00 00 00 00 00 01 00 00 00 ................
02F0: 00 00 00 00 00 00 00 00 00 10 04 40 01 00 00 00 ...........@....
0300: 00 00 00 00 00 00 00 00 00 10 04 41 01 00 00 00 ...........A....
0310: 00 00 00 00 00 00 00 00 00 10 04 42 01 00 00 00 ...........B....
0320: 00 00 00 00 00 00 00 00 00 10 04 43 01 00 00 00 ...........C....
0330: 00 00 00 00 00 00 00 00 00 10 04 44 01 00 00 00 ...........D....
0340: 00 00 00 00 00 00 00 00 00 10 04 45 01 00 00 00 ...........E....
0350: 00 00 00 00 00 00 00 00 00 10 04 46 01 00 00 00 ...........F....
0360: 00 00 00 00 00 00 00 00 00 10 04 47 01 00 00 00 ...........G....
0370: 00 00 00 00 00 00 00 00 00 10 05 48 01 00 00 00 ...........H....
0380: 00 00 00 00 00 00 00 00 00 10 05 49 01 00 00 00 ...........I....
0390: 00 00 00 00 00 00 00 00 00 10 05 4A 01 00 00 00 ...........J....
03A0: 00 00 00 00 00 00 00 00 00 10 05 4B 01 00 00 00 ...........K....
03B0: 00 00 00 00 00 00 00 00 00 10 05 4C 01 00 00 00 ...........L....
03C0: 00 00 00 00 00 00 00 00 00 10 05 4D 01 00 00 00 ...........M....
03D0: 00 00 00 00 00 00 00 00 00 10 05 4E 01 00 00 00 ...........N....
03E0: 00 00 00 00 00 00 00 00 00 10 05 4F 01 00 00 00 ...........O....
03F0: 00 00 00 00 00 00 00 00 01 28 06 00 00 00 00 00 .........(......
0400: 00 00 00 40 06 00 00 00 00 00 00 FF 01 00 00 00 ...@............
0410: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0420: 00 10 06 60 01 00 00 00 00 00 00 00 00 00 00 00 ...`............
0430: 00 10 06 61 01 00 00 00 00 00 00 00 00 00 00 00 ...a............
0440: 00 10 06 62 01 00 00 00 00 00 00 00 00 00 00 00 ...b............
0450: 00 10 06 63 01 00 00 00 00 00 00 00 00 00 00 00 ...c............
0460: 00 10 06 64 01 00 00 00 00 00 00 00 00 00 00 00 ...d............
0470: 00 10 06 65 01 00 00 00 00 00 00 00 00 00 00 00 ...e............
0480: 00 10 06 66 01 00 00 00 00 00 00 00 00 00 00 00 ...f............
0490: 00 10 06 67 01 00 00 00 00 00 00 00 00 00 00 00 ...g............
04A0: 00 10 07 68 01 00 00 00 00 00 00 00 00 00 00 00 ...h............
04B0: 00 10 07 69 01 00 00 00 00 00 00 00 00 00 00 00 ...i............
04C0: 00 10 07 6A 01 00 00 00 00 00 00 00 00 00 00 00 ...j............
04D0: 00 10 07 6B 01 00 00 00 00 00 00 00 00 00 00 00 ...k............
04E0: 00 10 07 6C 01 00 00 00 00 00 00 00 00 00 00 00 ...l............
04F0: 00 10 07 6D 01 00 00 00 00 00 00 00 00 00 00 00 ...m............
0500: 00 10 07 6E 01 00 00 00 00 00 00 00 00 00 00 00 ...n............
0510: 00 10 07 6F 01 00 00 00 00 00 00 00 00 00 00 00 ...o............
[-- Attachment #3: interlagos-xl-info-n.log --]
[-- Type: text/x-log, Size: 3472 bytes --]
host : bos-r815-1
release : 2.6.32.43-0.4.1.xs1.5.50.696.170738xen
version : #1 SMP Mon Jun 18 09:27:43 EDT 2012
machine : i686
nr_cpus : 64
max_cpu_id : 63
nr_nodes : 4
cores_per_socket : 8
threads_per_core : 2
cpu_mhz : 2600
hw_caps : 178bf3ff:2fd3fbff:00000000:00001710:12982203:00000000:01c9bfff:00000000
virt_caps : hvm hvm_directio
total_memory : 32742
free_memory : 31502
free_cpus : 0
cpu_topology :
cpu: core socket node
0: 0 0 0
1: 1 0 0
2: 2 0 0
3: 3 0 0
4: 4 0 0
5: 5 0 0
6: 6 0 0
7: 7 0 0
8: 8 0 0
9: 9 0 0
10: 10 0 0
11: 11 0 0
12: 12 0 0
13: 13 0 0
14: 14 0 0
15: 15 0 0
16: 0 1 2
17: 1 1 2
18: 2 1 2
19: 3 1 2
20: 4 1 2
21: 5 1 2
22: 6 1 2
23: 7 1 2
24: 8 1 0
25: 9 1 0
26: 10 1 0
27: 11 1 0
28: 12 1 0
29: 13 1 0
30: 14 1 0
31: 15 1 0
32: 0 2 4
33: 1 2 4
34: 2 2 4
35: 3 2 4
36: 4 2 4
37: 5 2 4
38: 6 2 4
39: 7 2 4
40: 8 2 0
41: 9 2 0
42: 10 2 0
43: 11 2 0
44: 12 2 0
45: 13 2 0
46: 14 2 0
47: 15 2 0
48: 0 3 6
49: 1 3 6
50: 2 3 6
51: 3 3 6
52: 4 3 6
53: 5 3 6
54: 6 3 6
55: 7 3 6
56: 8 3 0
57: 9 3 0
58: 10 3 0
59: 11 3 0
60: 12 3 0
61: 13 3 0
62: 14 3 0
63: 15 3 0
numa_info : none
xen_major : 4
xen_minor : 1
xen_extra : .2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xfdc00000
xen_changeset : Wed Jun 27 19:34:58 2012 +0100 23517:b004f4ef0cff
xen_commandline : com1=115200,8n1 console=com1,vga loglvl=all cpuinfo mem=1024G dom0_mem=752M,max:752M watchdog_timeout=300 cpuid_mask_xsave_eax=0 lowmem_emergency_pool=1M crashkernel=64M@32M dom0_max_vcpus=4
cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
cc_compile_by : andrewcoop
cc_compile_domain : uk.xensource.com
cc_compile_date : Wed Jun 27 14:34:59 EDT 2012
xend_config_format : 4
[-- Attachment #4: RFC-fix-numa-online.patch --]
[-- Type: text/x-patch, Size: 537 bytes --]
# HG changeset patch
# Parent c6c9d20963d7c5d8022a8cbe6bdaea359804f284
diff -r c6c9d20963d7 -r a0a7f546f22e xen/arch/x86/setup.c
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -195,8 +195,10 @@ void __devinit srat_detect_node(int cpu)
u32 apicid = x86_cpu_to_apicid[cpu];
node = apicid_to_node[apicid];
- if ( node == NUMA_NO_NODE || !node_online(node) )
+ if ( node == NUMA_NO_NODE )
node = 0;
+
+ node_set_online(node);
numa_set_node(cpu, node);
if ( opt_cpu_info && acpi_numa > 0 )
[-- Attachment #5: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Logical NUMA error during boot, and RFC patch
2012-06-27 19:10 Logical NUMA error during boot, and RFC patch Andrew Cooper
@ 2012-06-28 9:51 ` Jan Beulich
2012-06-28 10:29 ` Andrew Cooper
0 siblings, 1 reply; 4+ messages in thread
From: Jan Beulich @ 2012-06-28 9:51 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Dario Faggioli, Keir Fraser, xen-devel@lists.xen.org
>>> On 27.06.12 at 21:10, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> XenServer have recently acquired a quad-socket AMD Interlagos server and
> I have been playing around with it, and discovered a logical error in
> how Xen detects numa nodes.
>
> The server has 8 NUMA nodes, 4 of which have memory attached (the even
> nodes - see SRAT.dsl attached). This means that that
> node_set_online(nodeid) gets called for each node with memory attached.
> Later, in srat_detect_node(), node gets set to 0 if it was NUMA_NO_NODE,
> or if not node_online(). This leads to all the processors on the odd
> nodes being assigned to node 0, even though the odd nodes are present
> (see interlagos-xl-info-n.log)
>
> I present an RFC patch which changes srat_detect_node() to call
> node_set_online() for each node, which appears to fix the logic.
>
> Is this a sensible place to set the node online, or is there a better
> way to fix this logic?
While the place looks sensible, it has the possible problem of
potentially adding bits to the online map pretty late in the game.
As the memory-related invocations of node_set_online() come
out of numa_initmem_init()/acpi_scan_nodes(), perhaps the
(boot time) CPU-related ones should be done there too (I'd
still keep the adjustment you're already doing, to also cover
hotplug CPUs)?
Jan
Jan
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Logical NUMA error during boot, and RFC patch
2012-06-28 9:51 ` Jan Beulich
@ 2012-06-28 10:29 ` Andrew Cooper
2012-06-28 12:29 ` George Dunlap
0 siblings, 1 reply; 4+ messages in thread
From: Andrew Cooper @ 2012-06-28 10:29 UTC (permalink / raw)
To: Jan Beulich; +Cc: Dario Faggioli, Keir (Xen.org), xen-devel@lists.xen.org
On 28/06/12 10:51, Jan Beulich wrote:
>>>> On 27.06.12 at 21:10, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> XenServer have recently acquired a quad-socket AMD Interlagos server and
>> I have been playing around with it, and discovered a logical error in
>> how Xen detects numa nodes.
>>
>> The server has 8 NUMA nodes, 4 of which have memory attached (the even
>> nodes - see SRAT.dsl attached). This means that that
>> node_set_online(nodeid) gets called for each node with memory attached.
>> Later, in srat_detect_node(), node gets set to 0 if it was NUMA_NO_NODE,
>> or if not node_online(). This leads to all the processors on the odd
>> nodes being assigned to node 0, even though the odd nodes are present
>> (see interlagos-xl-info-n.log)
>>
>> I present an RFC patch which changes srat_detect_node() to call
>> node_set_online() for each node, which appears to fix the logic.
>>
>> Is this a sensible place to set the node online, or is there a better
>> way to fix this logic?
> While the place looks sensible, it has the possible problem of
> potentially adding bits to the online map pretty late in the game.
>
> As the memory-related invocations of node_set_online() come
> out of numa_initmem_init()/acpi_scan_nodes(), perhaps the
> (boot time) CPU-related ones should be done there too (I'd
> still keep the adjustment you're already doing, to also cover
> hotplug CPUs)?
>
> Jan
>
I have been doing quite a bit more testing this morning, and have come
to some sad conclusions.
This specific server is a Dell R815 loner, with 8x4GiB DIMMs, with 2
DIMMs hanging off each socket. As each socket is an interlagos
processor, there are 4 memory controllers (with 8 DIMM slots as they are
dual channel)
What this means is that per socket, one node has half of its available
DIMMs filled, and the other node has no memory. The performance
implications are severe, but as it appears that almost all combinations
of RAM you can select on the Dell website will leads to poor or worse
performance, I can foresee many systems like this in the future. (I
don't wish to single Dell out here, other than it happens to be the
provider of the server I am testing. Other server providers suffer the
same issues)
As to the problem at hand, I will investigate the numa code some more
and see about setting it up earlier.
--
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Logical NUMA error during boot, and RFC patch
2012-06-28 10:29 ` Andrew Cooper
@ 2012-06-28 12:29 ` George Dunlap
0 siblings, 0 replies; 4+ messages in thread
From: George Dunlap @ 2012-06-28 12:29 UTC (permalink / raw)
To: Andrew Cooper
Cc: Dario Faggioli, Keir (Xen.org), Jan Beulich,
xen-devel@lists.xen.org
On Thu, Jun 28, 2012 at 11:29 AM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> What this means is that per socket, one node has half of its available
> DIMMs filled, and the other node has no memory. The performance
> implications are severe, but as it appears that almost all combinations
> of RAM you can select on the Dell website will leads to poor or worse
> performance, I can foresee many systems like this in the future. (I
> don't wish to single Dell out here, other than it happens to be the
> provider of the server I am testing. Other server providers suffer the
> same issues)
Wow, that sounds almost more like AMP (asymmetric multiprocessing) than NUMA...
-George
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-06-28 12:29 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-06-27 19:10 Logical NUMA error during boot, and RFC patch Andrew Cooper
2012-06-28 9:51 ` Jan Beulich
2012-06-28 10:29 ` Andrew Cooper
2012-06-28 12:29 ` George Dunlap
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).