Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

[Carb, Brian A]

[-- Attachment #1.1: Type: text/plain, Size: 1709 bytes --]

Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on
3.2.0 Release Candidate
 
HOST:
Unisys ES7000/one, x86_64, 64 processors, 256gb memory
unstable changeset 16693 compiled with max_phys_cpus=64 and booted with
dom0_mem=4096M, numa=on, and xenheap_megabytes=64
from http://xenbits.xensource.com/staging/xen-unstable.hg pulled on
20080108:
 
OLD ISSUES:

-Bug #1037 - Shutdown of large domains takes a long time, during which
time xend is unresponsive (due to the synchronous tearing down of the
memory map); for example, shutdown of 180gb domu causes xend commands to
hang for 5 min 15 sec

 
TESTS:
 
1)  32-processor 64-bit SLES10 para-virtualized VM with 240gb memory -
run kernbench -ok
 
2)  32-processor 64-bit SLES10 hardware-virtualized VM with 240gb memory
- run kernbench - ok
 
3)  32-processor 64-bit Windows Datacenter hardware-virtualized VM with
240gb memory - run cpu intensive program - ok
 
XM-TEST:
 
Issue: Had to set MAX_DOMS to 5 in 11_create_concurrent_pos.py -
otherwise the test would hang forever after a varying number of domu
creations and would produce no trace output.
 
Xm-test timing summary:
  Run Started : Wed, 09 Jan 2008 10:45:24 -0500
  Run Stoped  : Wed, 09 Jan 2008 11:35:14 -0500

Xm-test execution summary:
  PASS:  109
  FAIL:  3
  XPASS: 1
  XFAIL: 2

Details:
 FAIL: 02_block_attach_file_device_pos
         xm block-attach returned invalid 256 != 0
 
XFAIL: 02_network_local_ping_pos
         ping loopback failed for size 65507. ping eth0 failed for size
65507.
 
XFAIL: 11_network_domU_ping_pos
         Ping failed for size 65507.
 

brian carb
unisys corporation - malvern, pa

 


[-- Attachment #1.2: Type: text/html, Size: 5085 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Carb, Brian A wrote:
> Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on
> 3.2.0 Release Candidate
>  
> HOST:
> Unisys ES7000/one, x86_64, 64 processors, 256gb memory
> unstable changeset 16693 compiled with max_phys_cpus=64 and booted with
> dom0_mem=4096M, numa=on, and xenheap_megabytes=64
> from http://xenbits.xensource.com/staging/xen-unstable.hg pulled on
> 20080108:
>  

<snip>

We have noticed problems with dom0 initialization when not
limiting the memory for dom0 when we reach 112GB. This is using
3.1.2 and also appears in 3.1.3 as it stood last week. Note that
3.1.0 had no such issue. The problem is definately a Hypervisor
change of some sort (as opposed to a dom0 kernel change).

Wondering if anyone else sees this? And while it makes a lot
of sense to limit dom0 memory we have not in the past. And it's
somewhat problematic to know what to limit it too. We are in
the process of trying to determine what change set introduced
this regression.

Thanks,
 Bill

Re: Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

[Keir Fraser]

On 15/1/08 13:50, "Bill Burns" <bburns@redhat.com> wrote:

> We have noticed problems with dom0 initialization when not
> limiting the memory for dom0 when we reach 112GB. This is using
> 3.1.2 and also appears in 3.1.3 as it stood last week. Note that
> 3.1.0 had no such issue. The problem is definately a Hypervisor
> change of some sort (as opposed to a dom0 kernel change).
> 
> Wondering if anyone else sees this? And while it makes a lot
> of sense to limit dom0 memory we have not in the past. And it's
> somewhat problematic to know what to limit it too. We are in
> the process of trying to determine what change set introduced
> this regression.

Have you tried 3.2.0 release candidates?

I doubt any other developers are going to run into this as we're not
overwhelmed with 128GB+ systems.

 -- Keir

Re: Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Keir Fraser wrote:
> On 15/1/08 13:50, "Bill Burns" <bburns@redhat.com> wrote:
> 
>> We have noticed problems with dom0 initialization when not
>> limiting the memory for dom0 when we reach 112GB. This is using
>> 3.1.2 and also appears in 3.1.3 as it stood last week. Note that
>> 3.1.0 had no such issue. The problem is definately a Hypervisor
>> change of some sort (as opposed to a dom0 kernel change).
>>
>> Wondering if anyone else sees this? And while it makes a lot
>> of sense to limit dom0 memory we have not in the past. And it's
>> somewhat problematic to know what to limit it too. We are in
>> the process of trying to determine what change set introduced
>> this regression.
> 
> Have you tried 3.2.0 release candidates?
> 
> I doubt any other developers are going to run into this as we're not
> overwhelmed with 128GB+ systems.
> 
>  -- Keir
> 
> 

No, I have not tried on 3.2.0. Will see if I can at some
point...

 Bill

Re: Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

[Keir Fraser]

On 15/1/08 16:15, "Bill Burns" <bburns@redhat.com> wrote:

>> Have you tried 3.2.0 release candidates?
>> 
>> I doubt any other developers are going to run into this as we're not
>> overwhelmed with 128GB+ systems.
>> 
>>  -- Keir
>> 
>> 
> 
> No, I have not tried on 3.2.0. Will see if I can at some
> point...

Also, do you have any more info to share on what actually goes wrong when
dom0 has 'too much' memory?

 Thanks,
 Keir

Re: Test results on Unisys ES7000 64x 256gb using unstable c/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Keir Fraser wrote:
> 
> 
> On 15/1/08 16:15, "Bill Burns" <bburns@redhat.com> wrote:
> 
>>> Have you tried 3.2.0 release candidates?
>>>
>>> I doubt any other developers are going to run into this as we're not
>>> overwhelmed with 128GB+ systems.
>>>
>>>  -- Keir
>>>
>>>
>> No, I have not tried on 3.2.0. Will see if I can at some
>> point...
> 
> Also, do you have any more info to share on what actually goes wrong when
> dom0 has 'too much' memory?
> 

The dom0 kernel spits out messages like the following starting around the
init of cpu1 time, and periodically thereafter.

Timer ISR/0: Time went backwards: delta=-50206266948 delta_cpu=13733052
shadow=8186343367 off=13649733458 processed=72042343367 cpu_processed=21822343367

Eventually just hanging (or making such slow progress to be
effectively hung).

 Bill



>  Thanks,
>  Keir
> 
> 

RE: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Ian Pratt]

> >> No, I have not tried on 3.2.0. Will see if I can at some
> >> point...
> >
> > Also, do you have any more info to share on what actually goes wrong
> when
> > dom0 has 'too much' memory?
> >
> 
> The dom0 kernel spits out messages like the following starting around
> the
> init of cpu1 time, and periodically thereafter.
> 
> Timer ISR/0: Time went backwards: delta=-50206266948
delta_cpu=13733052
> shadow=8186343367 off=13649733458 processed=72042343367
> cpu_processed=21822343367
> 
> Eventually just hanging (or making such slow progress to be
> effectively hung).

How many CPUs does the system have? Does the same large memory issue
occur if you have fewer physical CPUs?

Thanks,
Ian

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Ian Pratt wrote:
>>>> No, I have not tried on 3.2.0. Will see if I can at some
>>>> point...
>>> Also, do you have any more info to share on what actually goes wrong
>> when
>>> dom0 has 'too much' memory?
>>>
>> The dom0 kernel spits out messages like the following starting around
>> the
>> init of cpu1 time, and periodically thereafter.
>>
>> Timer ISR/0: Time went backwards: delta=-50206266948
> delta_cpu=13733052
>> shadow=8186343367 off=13649733458 processed=72042343367
>> cpu_processed=21822343367
>>
>> Eventually just hanging (or making such slow progress to be
>> effectively hung).
> 
> How many CPUs does the system have? Does the same large memory issue
> occur if you have fewer physical CPUs?
> 

The system has 64 but is only built for 32, so the others are
ignored. Don't know if the problem happens with less CPUs at
this point. Hope to get more data soon...

Thanks,
 Bill

> Thanks,
> Ian
> 
> 

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Bill Burns wrote:
> Ian Pratt wrote:
>>>>> No, I have not tried on 3.2.0. Will see if I can at some
>>>>> point...
>>>> Also, do you have any more info to share on what actually goes wrong
>>> when
>>>> dom0 has 'too much' memory?
>>>>
>>> The dom0 kernel spits out messages like the following starting around
>>> the
>>> init of cpu1 time, and periodically thereafter.
>>>
>>> Timer ISR/0: Time went backwards: delta=-50206266948
>> delta_cpu=13733052
>>> shadow=8186343367 off=13649733458 processed=72042343367
>>> cpu_processed=21822343367
>>>
>>> Eventually just hanging (or making such slow progress to be
>>> effectively hung).
>> How many CPUs does the system have? Does the same large memory issue
>> occur if you have fewer physical CPUs?
>>
> 
> The system has 64 but is only built for 32, so the others are
> ignored. Don't know if the problem happens with less CPUs at
> this point. Hope to get more data soon...
> 

Interestingly, the symptom seems to disappear with a
Hypervisor built to support all 64 CPUs. But I need to
get more time on the system to say that for sure.

 Bill



> Thanks,
>  Bill
> 
>> Thanks,
>> Ian
>>
>>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Bill Burns wrote:
> Bill Burns wrote:
>> Ian Pratt wrote:
>>>>>> No, I have not tried on 3.2.0. Will see if I can at some
>>>>>> point...
>>>>> Also, do you have any more info to share on what actually goes wrong
>>>> when
>>>>> dom0 has 'too much' memory?
>>>>>
>>>> The dom0 kernel spits out messages like the following starting around
>>>> the
>>>> init of cpu1 time, and periodically thereafter.
>>>>
>>>> Timer ISR/0: Time went backwards: delta=-50206266948
>>> delta_cpu=13733052
>>>> shadow=8186343367 off=13649733458 processed=72042343367
>>>> cpu_processed=21822343367
>>>>
>>>> Eventually just hanging (or making such slow progress to be
>>>> effectively hung).
>>> How many CPUs does the system have? Does the same large memory issue
>>> occur if you have fewer physical CPUs?
>>>
>> The system has 64 but is only built for 32, so the others are
>> ignored. Don't know if the problem happens with less CPUs at
>> this point. Hope to get more data soon...
>>
> 
> Interestingly, the symptom seems to disappear with a
> Hypervisor built to support all 64 CPUs. But I need to
> get more time on the system to say that for sure.
> 
Disregard the previous. It still happens. Continuing to debug..

 Bill



>  Bill
> 
> 
> 
>> Thanks,
>>  Bill
>>
>>> Thanks,
>>> Ian
>>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Bill Burns wrote:
> Bill Burns wrote:
>> Bill Burns wrote:
>>> Ian Pratt wrote:
>>>>>>> No, I have not tried on 3.2.0. Will see if I can at some
>>>>>>> point...
>>>>>> Also, do you have any more info to share on what actually goes wrong
>>>>> when
>>>>>> dom0 has 'too much' memory?
>>>>>>
>>>>> The dom0 kernel spits out messages like the following starting around
>>>>> the
>>>>> init of cpu1 time, and periodically thereafter.
>>>>>
>>>>> Timer ISR/0: Time went backwards: delta=-50206266948
>>>> delta_cpu=13733052
>>>>> shadow=8186343367 off=13649733458 processed=72042343367
>>>>> cpu_processed=21822343367
>>>>>
>>>>> Eventually just hanging (or making such slow progress to be
>>>>> effectively hung).
>>>> How many CPUs does the system have? Does the same large memory issue
>>>> occur if you have fewer physical CPUs?
>>>>
>>> The system has 64 but is only built for 32, so the others are
>>> ignored. Don't know if the problem happens with less CPUs at
>>> this point. Hope to get more data soon...
>>>
>> Interestingly, the symptom seems to disappear with a
>> Hypervisor built to support all 64 CPUs. But I need to
>> get more time on the system to say that for sure.
>>
> Disregard the previous. It still happens. Continuing to debug..
> 
>  

<snip>
Ok, some progress. Background is that 3.1.2 (and 3.1.3 at least
as it was a wek or two ago) fails to boot on a 64 CPU es7000 with
over 112GB of memory. This is with both HV & dom0 being x86_64.
The symptom is that the dom0 kernel gets time went backwards
error during init.

The patch at which this first fails is 15137, which is the patch
that introduces using the ACPI PM timer as the clock
source. If I include the next patch (that allows for clock
selection) and choose pit as clock source the system boots
fine. Without the arg the ACPI timer is used and I get the hang.

Don't know if this is unique to this platform or a
general issue with large memory. Seems that most folks
smartly limit the dom0 memory well below 112GB.

Note I have not yet tried 3.2. Also note that the
patch determination was not a pure binary search.
There is a later patch (15194) specific to es7000 that
I pulled in and the second patch (15045) breaks things
during the HV init without some later patches, so it
was left out.

 Bill

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Keir Fraser]

On 28/1/08 14:02, "Bill Burns" <bburns@redhat.com> wrote:

> Ok, some progress. Background is that 3.1.2 (and 3.1.3 at least
> as it was a wek or two ago) fails to boot on a 64 CPU es7000 with
> over 112GB of memory. This is with both HV & dom0 being x86_64.
> The symptom is that the dom0 kernel gets time went backwards
> error during init.
> 
> The patch at which this first fails is 15137, which is the patch
> that introduces using the ACPI PM timer as the clock
> source. If I include the next patch (that allows for clock
> selection) and choose pit as clock source the system boots
> fine. Without the arg the ACPI timer is used and I get the hang.

The obvious question then is what happens to the ACPI PM timer when dom0
gets more than 112GB of memory. Perhaps it's worth adding some tracing to
Xen and see whether e.g., the platform timer stops running?

 -- Keir

RE: Test results on Unisys ES7000 64x 256gb usingunstablec/s 16693 on 3.2.0 Release Candidate

[Carb, Brian A]

My environment also produces a panic (unstable cs 16852 on x64
SLES10sp1) when I boot with 64cpu and 256gb and dom0_mem not specified:
  traps.c:414:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0
[ec=0000]

with dom0_mem=4096, system boots fine.

 
brian carb
unisys corporation - malvern, pa

-----

complete entry from menu.lst:

title XEN
    root (hd0,0)
    kernel /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
dom0_mem=4096M console=com1,vga numa=on xenheap_megabytes=64
    module /boot/vmlinuz-xen
root=/dev/disk/by-id/scsi-3600508e000000000ff4df9da7f80cc05-part1
vga=0x314 resume=/dev/sda2 splash=silent showopts debug=y
    module /boot/initrd-xen

serial trace:

 \ \/ /___ _ __   |___ / |___ /    _   _ _ __  ___| |_ __ _| |__ | | ___

  \  // _ \ '_ \    |_ \   |_ \ __| | | | '_ \/ __| __/ _` | '_ \| |/ _
\
  /  \  __/ | | |  ___) | ___) |__| |_| | | | \__ \ || (_| | |_) | |
__/
 /_/\_\___|_| |_| |____(_)____/    \__,_|_|
|_|___/\__\__,_|_.__/|_|\___|
 

(XEN) Xen version 3.3-unstable (root@site) (gcc version 4.1.2 20070115
(prerelease) (SUSE Linux)) Wed Jan 23 10:52:42 EST 2008
(XEN) Latest ChangeSet: Tue Jan 22 19:26:52 2008 +0000
16852:087c2b76d360
(XEN) Command line: com1=115200,8n1 apic_verbosity=debug
console=com1,vga numa=on xenheap_megabytes=64
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000037e70000 (usable)
(XEN)  0000000037e70000 - 0000000037ed7000 (ACPI data)
(XEN)  0000000037ed7000 - 0000000037f00000 (ACPI NVS)
(XEN)  0000000037f00000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000004000000000 (usable)
(XEN) System RAM: 261887MB (268172340kB)
(XEN) ACPI: RSDP (v000 PTLTD                                 ) @
0x00000000000f8650
(XEN) ACPI: RSDT (v001 PTLTD    RSDT   0x06040000  LTP 0x06040000) @
0x0000000037e7d652
(XEN) ACPI: FADT (v004 UNISYS ZORRO    0x06040000 PTL  0x00000008) @
0x0000000037ed58d4
(XEN) ACPI: OEM1 (v001 UNISYS   OEM1   0x06040000  LTP 0x06040000) @
0x0000000037ed59c8
(XEN) ACPI: OEM2 (v001 UNISYS   OEM2   0x06040000  LTP 0x06040000) @
0x0000000037ed59f8
(XEN) ACPI: WSPT (v001 UNISYS   WSPT   0x06040000  LTP 0x06040000) @
0x0000000037ed65f4
(XEN) ACPI: SRAT (v001 UNISYS   SRAT   0x06040000  LTP 0x06040000) @
0x0000000037ed661c
(XEN) ACPI: MADT (v001 PTLTD  	 APIC   0x06040000  LTP 0x06040000) @
0x0000000037ed6bb4
(XEN) ACPI: SPCR (v001 PTLTD  $UCRTBL$ 0x06040000 PTL  0x00000001) @
0x0000000037ed6fb0
(XEN) ACPI: DSDT (v001  Intel  870 SMP 0x06040000 MSFT 0x03000000) @
0x0000000000000000
(XEN) SRAT: PXM 8 -> APIC 126 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 122 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 118 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 114 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 124 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 120 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 116 -> Node 0
(XEN) SRAT: PXM 8 -> APIC 112 -> Node 0
(XEN) SRAT: PXM 7 -> APIC 110 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 106 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 102 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 98 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 108 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 104 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 100 -> Node 1
(XEN) SRAT: PXM 7 -> APIC 96 -> Node 1
(XEN) SRAT: PXM 6 -> APIC 94 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 90 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 86 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 82 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 92 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 88 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 84 -> Node 2
(XEN) SRAT: PXM 6 -> APIC 80 -> Node 2
(XEN) SRAT: PXM 5 -> APIC 78 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 74 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 70 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 66 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 76 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 72 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 68 -> Node 3
(XEN) SRAT: PXM 5 -> APIC 64 -> Node 3
(XEN) SRAT: PXM 4 -> APIC 62 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 58 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 54 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 50 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 60 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 56 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 52 -> Node 4
(XEN) SRAT: PXM 4 -> APIC 48 -> Node 4
(XEN) SRAT: PXM 3 -> APIC 46 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 42 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 38 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 34 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 44 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 40 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 36 -> Node 5
(XEN) SRAT: PXM 3 -> APIC 32 -> Node 5
(XEN) SRAT: PXM 2 -> APIC 30 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 26 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 22 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 18 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 28 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 24 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 20 -> Node 6
(XEN) SRAT: PXM 2 -> APIC 16 -> Node 6
(XEN) SRAT: PXM 1 -> APIC 14 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 10 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 6 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 2 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 12 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 8 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 4 -> Node 7
(XEN) SRAT: PXM 1 -> APIC 0 -> Node 7
(XEN) SRAT: Node 7 PXM 1 0-100000
(XEN) SRAT: Node 7 PXM 1 0-800000000
(XEN) SRAT: Node 6 PXM 2 800000000-1000000000
(XEN) SRAT: Node 5 PXM 3 1000000000-1800000000
(XEN) SRAT: Node 4 PXM 4 1800000000-2000000000
(XEN) SRAT: Node 3 PXM 5 2000000000-2800000000
(XEN) SRAT: Node 2 PXM 6 2800000000-3000000000
(XEN) SRAT: Node 1 PXM 7 3000000000-3800000000
(XEN) SRAT: Node 0 PXM 8 3800000000-4000000000
(XEN) NUMA: Using 35 for the hash shift.
(XEN) Xen heap: 54MB (55756kB)
(XEN) Reserving non-aligned node boundary @ mfn 8388608
(XEN) Reserving non-aligned node boundary @ mfn 16777216
(XEN) Reserving non-aligned node boundary @ mfn 25165824
(XEN) Reserving non-aligned node boundary @ mfn 33554432
(XEN) Reserving non-aligned node boundary @ mfn 41943040
(XEN) Reserving non-aligned node boundary @ mfn 50331648
(XEN) Reserving non-aligned node boundary @ mfn 58720256
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) found SMP MP-table at 000f8680
(XEN) DMI present.
(XEN) UNISYS ES7000-ONE detected: force use of apic=bigsmp
(XEN) Using APIC driver bigsmp
(XEN) ACPI: PM-Timer IO Port: 0xd08
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x04] enabled)
(XEN) Processor #4 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x08] enabled)
(XEN) Processor #8 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x0c] enabled)
(XEN) Processor #12 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled)
(XEN) Processor #2 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x06] enabled)
(XEN) Processor #6 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled)
(XEN) Processor #10 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled)
(XEN) Processor #14 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)
(XEN) Processor #16 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x14] enabled)
(XEN) Processor #20 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x18] enabled)
(XEN) Processor #24 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x1c] enabled)
(XEN) Processor #28 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x12] enabled)
(XEN) Processor #18 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x16] enabled)
(XEN) Processor #22 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x1a] enabled)
(XEN) Processor #26 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x1e] enabled)
(XEN) Processor #30 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)
(XEN) Processor #32 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x24] enabled)
(XEN) Processor #36 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x28] enabled)
(XEN) Processor #40 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x2c] enabled)
(XEN) Processor #44 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x22] enabled)
(XEN) Processor #34 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x26] enabled)
(XEN) Processor #38 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x2a] enabled)
(XEN) Processor #42 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x2e] enabled)
(XEN) Processor #46 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x30] enabled)
(XEN) Processor #48 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x19] lapic_id[0x34] enabled)
(XEN) Processor #52 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x38] enabled)
(XEN) Processor #56 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3c] enabled)
(XEN) Processor #60 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x32] enabled)
(XEN) Processor #50 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x36] enabled)
(XEN) Processor #54 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3a] enabled)
(XEN) Processor #58 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] enabled)
(XEN) Processor #62 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x20] lapic_id[0x40] enabled)
(XEN) Processor #64 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x21] lapic_id[0x44] enabled)
(XEN) Processor #68 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x22] lapic_id[0x48] enabled)
(XEN) Processor #72 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x23] lapic_id[0x4c] enabled)
(XEN) Processor #76 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x24] lapic_id[0x42] enabled)
(XEN) Processor #66 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x25] lapic_id[0x46] enabled)
(XEN) Processor #70 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x26] lapic_id[0x4a] enabled)
(XEN) Processor #74 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x27] lapic_id[0x4e] enabled)
(XEN) Processor #78 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x28] lapic_id[0x50] enabled)
(XEN) Processor #80 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x29] lapic_id[0x54] enabled)
(XEN) Processor #84 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x58] enabled)
(XEN) Processor #88 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x5c] enabled)
(XEN) Processor #92 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x52] enabled)
(XEN) Processor #82 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x56] enabled)
(XEN) Processor #86 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x5a] enabled)
(XEN) Processor #90 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x5e] enabled)
(XEN) Processor #94 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x30] lapic_id[0x60] enabled)
(XEN) Processor #96 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x31] lapic_id[0x64] enabled)
(XEN) Processor #100 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x32] lapic_id[0x68] enabled)
(XEN) Processor #104 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x33] lapic_id[0x6c] enabled)
(XEN) Processor #108 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x34] lapic_id[0x62] enabled)
(XEN) Processor #98 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x35] lapic_id[0x66] enabled)
(XEN) Processor #102 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x36] lapic_id[0x6a] enabled)
(XEN) Processor #106 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x37] lapic_id[0x6e] enabled)
(XEN) Processor #110 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x38] lapic_id[0x70] enabled)
(XEN) Processor #112 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x39] lapic_id[0x74] enabled)
(XEN) Processor #116 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x78] enabled)
(XEN) Processor #120 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x7c] enabled)
(XEN) Processor #124 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x72] enabled)
(XEN) Processor #114 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x76] enabled)
(XEN) Processor #118 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x7a] enabled)
(XEN) Processor #122 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x7e] enabled)
(XEN) Processor #126 15:6 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x10] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x11] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x12] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x13] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x14] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x15] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x16] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x17] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x18] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x19] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x1a] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x1b] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x1c] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x1d] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x1e] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x1f] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x20] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x21] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x22] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x23] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x24] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x25] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x26] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x27] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x28] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x29] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x2a] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x2b] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x2c] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x2d] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x2e] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x2f] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x30] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x31] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x32] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x33] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x34] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x35] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x36] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x37] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x38] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x39] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x3a] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x3b] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x3c] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x3d] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x3e] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x3f] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec02000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 8, version 32, address 0xfec02000, GSI 24-47
(XEN) ACPI: IOAPIC (id[0x09] address[0xfec03000] gsi_base[48])
(XEN) IOAPIC[2]: apic_id 9, version 32, address 0xfec03000, GSI 48-71
(XEN) ACPI: IOAPIC (id[0x0a] address[0xfec04000] gsi_base[72])
(XEN) IOAPIC[3]: apic_id 10, version 32, address 0xfec04000, GSI 72-95
(XEN) ACPI: IOAPIC (id[0x0b] address[0xfec05000] gsi_base[96])
(XEN) IOAPIC[4]: apic_id 11, version 32, address 0xfec05000, GSI 96-119
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 5 I/O APICs
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) mapped APIC to ffff828bfffff000 (fee00000)
(XEN) mapped IOAPIC to ffff828bffffe000 (fec00000)
(XEN) mapped IOAPIC to ffff828bffffd000 (fec02000)
(XEN) mapped IOAPIC to ffff828bffffc000 (fec03000)
(XEN) mapped IOAPIC to ffff828bffffb000 (fec04000)
(XEN) mapped IOAPIC to ffff828bffffa000 (fec05000)
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 3400.125 MHz processor.
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) HVM: VMX enabled
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU0: Thermal monitoring enabled
(XEN) CPU0: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Getting VERSION: 50014
(XEN) Getting VERSION: 50014
(XEN) Getting ID: 0
(XEN) Getting LVT0: 10700
(XEN) Getting LVT1: 10400
(XEN) masked ExtINT on CPU#0
(XEN) Booting processor 1/4 eip 8c000
(XEN) Initializing CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU1: Thermal monitoring enabled
(XEN) CPU1: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 2/8 eip 8c000
(XEN) Initializing CPU#2
(XEN) masked ExtINT on CPU#2
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 2
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU2: Thermal monitoring enabled
(XEN) CPU2: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 3/12 eip 8c000
(XEN) Initializing CPU#3
(XEN) masked ExtINT on CPU#3
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 3
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU3: Thermal monitoring enabled
(XEN) CPU3: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 4/2 eip 8c000
(XEN) Initializing CPU#4
(XEN) masked ExtINT on CPU#4
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#4.
(XEN) CPU4: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU4: Thermal monitoring enabled
(XEN) CPU4: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 5/6 eip 8c000
(XEN) Initializing CPU#5
(XEN) masked ExtINT on CPU#5
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#5.
(XEN) CPU5: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU5: Thermal monitoring enabled
(XEN) CPU5: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 6/10 eip 8c000
(XEN) Initializing CPU#6
(XEN) masked ExtINT on CPU#6
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 2
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#6.
(XEN) CPU6: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU6: Thermal monitoring enabled
(XEN) CPU6: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 7/14 eip 8c000
(XEN) Initializing CPU#7
(XEN) masked ExtINT on CPU#7
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 3
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#7.
(XEN) CPU7: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU7: Thermal monitoring enabled
(XEN) CPU7: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 8/16 eip 8c000
(XEN) Initializing CPU#8
(XEN) masked ExtINT on CPU#8
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 4
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#8.
(XEN) CPU8: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU8: Thermal monitoring enabled
(XEN) CPU8: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 9/20 eip 8c000
(XEN) Initializing CPU#9
(XEN) masked ExtINT on CPU#9
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 5
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#9.
(XEN) CPU9: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU9: Thermal monitoring enabled
(XEN) CPU9: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 10/24 eip 8c000
(XEN) Initializing CPU#10
(XEN) masked ExtINT on CPU#10
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 6
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#10.
(XEN) CPU10: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU10: Thermal monitoring enabled
(XEN) CPU10: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 11/28 eip 8c000
(XEN) Initializing CPU#11
(XEN) masked ExtINT on CPU#11
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 7
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#11.
(XEN) CPU11: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU11: Thermal monitoring enabled
(XEN) CPU11: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 12/18 eip 8c000
(XEN) Initializing CPU#12
(XEN) masked ExtINT on CPU#12
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 4
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#12.
(XEN) CPU12: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU12: Thermal monitoring enabled
(XEN) CPU12: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 13/22 eip 8c000
(XEN) Initializing CPU#13
(XEN) masked ExtINT on CPU#13
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 5
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#13.
(XEN) CPU13: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU13: Thermal monitoring enabled
(XEN) CPU13: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 14/26 eip 8c000
(XEN) Initializing CPU#14
(XEN) masked ExtINT on CPU#14
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 6
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#14.
(XEN) CPU14: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU14: Thermal monitoring enabled
(XEN) CPU14: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 15/30 eip 8c000
(XEN) Initializing CPU#15
(XEN) masked ExtINT on CPU#15
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 7
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#15.
(XEN) CPU15: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU15: Thermal monitoring enabled
(XEN) CPU15: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 16/32 eip 8c000
(XEN) Initializing CPU#16
(XEN) masked ExtINT on CPU#16
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 8
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#16.
(XEN) CPU16: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU16: Thermal monitoring enabled
(XEN) CPU16: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 17/36 eip 8c000
(XEN) Initializing CPU#17
(XEN) masked ExtINT on CPU#17
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 9
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#17.
(XEN) CPU17: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU17: Thermal monitoring enabled
(XEN) CPU17: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 18/40 eip 8c000
(XEN) Initializing CPU#18
(XEN) masked ExtINT on CPU#18
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 10
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#18.
(XEN) CPU18: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU18: Thermal monitoring enabled
(XEN) CPU18: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 19/44 eip 8c000
(XEN) Initializing CPU#19
(XEN) masked ExtINT on CPU#19
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 11
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#19.
(XEN) CPU19: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU19: Thermal monitoring enabled
(XEN) CPU19: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 20/34 eip 8c000
(XEN) Initializing CPU#20
(XEN) masked ExtINT on CPU#20
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 8
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#20.
(XEN) CPU20: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU20: Thermal monitoring enabled
(XEN) CPU20: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 21/38 eip 8c000
(XEN) Initializing CPU#21
(XEN) masked ExtINT on CPU#21
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 9
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#21.
(XEN) CPU21: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU21: Thermal monitoring enabled
(XEN) CPU21: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 22/42 eip 8c000
(XEN) Initializing CPU#22
(XEN) masked ExtINT on CPU#22
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 10
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#22.
(XEN) CPU22: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU22: Thermal monitoring enabled
(XEN) CPU22: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 23/46 eip 8c000
(XEN) Initializing CPU#23
(XEN) masked ExtINT on CPU#23
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 11
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#23.
(XEN) CPU23: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU23: Thermal monitoring enabled
(XEN) CPU23: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 24/48 eip 8c000
(XEN) Initializing CPU#24
(XEN) masked ExtINT on CPU#24
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 12
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#24.
(XEN) CPU24: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU24: Thermal monitoring enabled
(XEN) CPU24: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 25/52 eip 8c000
(XEN) Initializing CPU#25
(XEN) masked ExtINT on CPU#25
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 13
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#25.
(XEN) CPU25: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU25: Thermal monitoring enabled
(XEN) CPU25: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 26/56 eip 8c000
(XEN) Initializing CPU#26
(XEN) masked ExtINT on CPU#26
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 14
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#26.
(XEN) CPU26: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU26: Thermal monitoring enabled
(XEN) CPU26: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 27/60 eip 8c000
(XEN) Initializing CPU#27
(XEN) masked ExtINT on CPU#27
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 15
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#27.
(XEN) CPU27: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU27: Thermal monitoring enabled
(XEN) CPU27: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 28/50 eip 8c000
(XEN) Initializing CPU#28
(XEN) masked ExtINT on CPU#28
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 12
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#28.
(XEN) CPU28: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU28: Thermal monitoring enabled
(XEN) CPU28: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 29/54 eip 8c000
(XEN) Initializing CPU#29
(XEN) masked ExtINT on CPU#29
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 13
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#29.
(XEN) CPU29: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU29: Thermal monitoring enabled
(XEN) CPU29: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 30/58 eip 8c000
(XEN) Initializing CPU#30
(XEN) masked ExtINT on CPU#30
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 14
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#30.
(XEN) CPU30: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU30: Thermal monitoring enabled
(XEN) CPU30: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 31/62 eip 8c000
(XEN) Initializing CPU#31
(XEN) masked ExtINT on CPU#31
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 15
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#31.
(XEN) CPU31: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU31: Thermal monitoring enabled
(XEN) CPU31: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 32/64 eip 8c000
(XEN) Initializing CPU#32
(XEN) masked ExtINT on CPU#32
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 16
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#32.
(XEN) CPU32: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU32: Thermal monitoring enabled
(XEN) CPU32: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 33/68 eip 8c000
(XEN) Initializing CPU#33
(XEN) masked ExtINT on CPU#33
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 17
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#33.
(XEN) CPU33: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU33: Thermal monitoring enabled
(XEN) CPU33: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 34/72 eip 8c000
(XEN) Initializing CPU#34
(XEN) masked ExtINT on CPU#34
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 18
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#34.
(XEN) CPU34: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU34: Thermal monitoring enabled
(XEN) CPU34: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 35/76 eip 8c000
(XEN) Initializing CPU#35
(XEN) masked ExtINT on CPU#35
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 19
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#35.
(XEN) CPU35: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU35: Thermal monitoring enabled
(XEN) CPU35: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 36/66 eip 8c000
(XEN) Initializing CPU#36
(XEN) masked ExtINT on CPU#36
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 16
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#36.
(XEN) CPU36: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU36: Thermal monitoring enabled
(XEN) CPU36: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 37/70 eip 8c000
(XEN) Initializing CPU#37
(XEN) masked ExtINT on CPU#37
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 17
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#37.
(XEN) CPU37: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU37: Thermal monitoring enabled
(XEN) CPU37: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 38/74 eip 8c000
(XEN) Initializing CPU#38
(XEN) masked ExtINT on CPU#38
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 18
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#38.
(XEN) CPU38: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU38: Thermal monitoring enabled
(XEN) CPU38: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 39/78 eip 8c000
(XEN) Initializing CPU#39
(XEN) masked ExtINT on CPU#39
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 19
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#39.
(XEN) CPU39: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU39: Thermal monitoring enabled
(XEN) CPU39: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 40/80 eip 8c000
(XEN) Initializing CPU#40
(XEN) masked ExtINT on CPU#40
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 20
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#40.
(XEN) CPU40: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU40: Thermal monitoring enabled
(XEN) CPU40: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 41/84 eip 8c000
(XEN) Initializing CPU#41
(XEN) masked ExtINT on CPU#41
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 21
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#41.
(XEN) CPU41: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU41: Thermal monitoring enabled
(XEN) CPU41: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 42/88 eip 8c000
(XEN) Initializing CPU#42
(XEN) masked ExtINT on CPU#42
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 22
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#42.
(XEN) CPU42: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU42: Thermal monitoring enabled
(XEN) CPU42: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 43/92 eip 8c000
(XEN) Initializing CPU#43
(XEN) masked ExtINT on CPU#43
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 23
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#43.
(XEN) CPU43: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU43: Thermal monitoring enabled
(XEN) CPU43: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 44/82 eip 8c000
(XEN) Initializing CPU#44
(XEN) masked ExtINT on CPU#44
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 20
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#44.
(XEN) CPU44: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU44: Thermal monitoring enabled
(XEN) CPU44: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 45/86 eip 8c000
(XEN) Initializing CPU#45
(XEN) masked ExtINT on CPU#45
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 21
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#45.
(XEN) CPU45: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU45: Thermal monitoring enabled
(XEN) CPU45: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 46/90 eip 8c000
(XEN) Initializing CPU#46
(XEN) masked ExtINT on CPU#46
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 22
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#46.
(XEN) CPU46: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU46: Thermal monitoring enabled
(XEN) CPU46: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 47/94 eip 8c000
(XEN) Initializing CPU#47
(XEN) masked ExtINT on CPU#47
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 23
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#47.
(XEN) CPU47: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU47: Thermal monitoring enabled
(XEN) CPU47: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 48/96 eip 8c000
(XEN) Initializing CPU#48
(XEN) masked ExtINT on CPU#48
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 24
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#48.
(XEN) CPU48: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU48: Thermal monitoring enabled
(XEN) CPU48: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 49/100 eip 8c000
(XEN) Initializing CPU#49
(XEN) masked ExtINT on CPU#49
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 25
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#49.
(XEN) CPU49: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU49: Thermal monitoring enabled
(XEN) CPU49: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 50/104 eip 8c000
(XEN) Initializing CPU#50
(XEN) masked ExtINT on CPU#50
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 26
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#50.
(XEN) CPU50: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU50: Thermal monitoring enabled
(XEN) CPU50: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 51/108 eip 8c000
(XEN) Initializing CPU#51
(XEN) masked ExtINT on CPU#51
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 27
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#51.
(XEN) CPU51: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU51: Thermal monitoring enabled
(XEN) CPU51: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 52/98 eip 8c000
(XEN) Initializing CPU#52
(XEN) masked ExtINT on CPU#52
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 24
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#52.
(XEN) CPU52: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU52: Thermal monitoring enabled
(XEN) CPU52: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 53/102 eip 8c000
(XEN) Initializing CPU#53
(XEN) masked ExtINT on CPU#53
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 25
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#53.
(XEN) CPU53: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU53: Thermal monitoring enabled
(XEN) CPU53: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 54/106 eip 8c000
(XEN) Initializing CPU#54
(XEN) masked ExtINT on CPU#54
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 26
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#54.
(XEN) CPU54: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU54: Thermal monitoring enabled
(XEN) CPU54: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 55/110 eip 8c000
(XEN) Initializing CPU#55
(XEN) masked ExtINT on CPU#55
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 27
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#55.
(XEN) CPU55: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU55: Thermal monitoring enabled
(XEN) CPU55: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 56/112 eip 8c000
(XEN) Initializing CPU#56
(XEN) masked ExtINT on CPU#56
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 28
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#56.
(XEN) CPU56: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU56: Thermal monitoring enabled
(XEN) CPU56: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 57/116 eip 8c000
(XEN) Initializing CPU#57
(XEN) masked ExtINT on CPU#57
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 29
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#57.
(XEN) CPU57: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU57: Thermal monitoring enabled
(XEN) CPU57: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 58/120 eip 8c000
(XEN) Initializing CPU#58
(XEN) masked ExtINT on CPU#58
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 30
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#58.
(XEN) CPU58: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU58: Thermal monitoring enabled
(XEN) CPU58: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 59/124 eip 8c000
(XEN) Initializing CPU#59
(XEN) masked ExtINT on CPU#59
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 31
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#59.
(XEN) CPU59: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU59: Thermal monitoring enabled
(XEN) CPU59: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 60/114 eip 8c000
(XEN) Initializing CPU#60
(XEN) masked ExtINT on CPU#60
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 28
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#60.
(XEN) CPU60: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU60: Thermal monitoring enabled
(XEN) CPU60: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 61/118 eip 8c000
(XEN) Initializing CPU#61
(XEN) masked ExtINT on CPU#61
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 29
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#61.
(XEN) CPU61: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU61: Thermal monitoring enabled
(XEN) CPU61: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 62/122 eip 8c000
(XEN) Initializing CPU#62
(XEN) masked ExtINT on CPU#62
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 30
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#62.
(XEN) CPU62: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU62: Thermal monitoring enabled
(XEN) CPU62: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Booting processor 63/126 eip 8c000
(XEN) Initializing CPU#63
(XEN) masked ExtINT on CPU#63
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 1024K
(XEN) CPU: L3 cache: 16384K
(XEN) CPU: Physical Processor ID: 31
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#63.
(XEN) CPU63: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU63: Thermal monitoring enabled
(XEN) CPU63: Intel Genuine Intel(R) CPU 3.40GHz stepping 08
(XEN) Total of 64 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) init IO_APIC IRQs
(XEN)  IO-APIC (apicid-pin) 0-0, 0-16, 0-17, 0-18, 0-19, 0-20, 0-21,
0-22, 0-23, 8-0, 8-1, 8-2, 8-3, 8-4, 8-5, 8-6, 8-7, 8-8, 8-9, 8-10,
8-11, 8-12, 8-13, 8-14, 8-15, 8-16, 8-17, 8-18, 8-19, 8-20, 8-21, 8-22,
8-23, 9-0, 9-1, 9-2, 9-3, 9-4, 9-5, 9-6, 9-7, 9-8, 9-9, 9-10, 9-11,
9-12, 9-13, 9-14, 9-15, 9-16, 9-17, 9-18, 9-19, 9-20, 9-21, 9-22, 9-23,
10-0, 10-1, 10-2, 10-3, 10-4, 10-5, 10-6, 10-7, 10-8, 10-9, 10-10,
10-11, 10-12, 10-13, 10-14, 10-15, 10-16, 10-17, 10-18, 10-19, 10-20,
10-21, 10-22, 10-23, 11-0, 11-1, 11-2, 11-3, 11-4, 11-5, 11-6, 11-7,
11-8, 11-9, 11-10, 11-11, 11-12, 11-13, 11-14, 11-15, 11-16, 11-17,
11-18, 11-19, 11-20, 11-21, 11-22, 11-23 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #0 registers: 24.
(XEN) number of IO-APIC #8 registers: 24.
(XEN) number of IO-APIC #9 registers: 24.
(XEN) number of IO-APIC #10 registers: 24.
(XEN) number of IO-APIC #11 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #0......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  0    0    0   0   0    0    0    28
(XEN)  02 000 00  0    0    0   0   0    0    0    F0
(XEN)  03 000 00  0    0    0   0   0    0    0    30
(XEN)  04 000 00  0    0    0   0   0    0    0    F1
(XEN)  05 000 00  0    0    0   0   0    0    0    38
(XEN)  06 000 00  0    0    0   0   0    0    0    40
(XEN)  07 000 00  0    0    0   0   0    0    0    48
(XEN)  08 000 00  0    0    0   0   0    0    0    50
(XEN)  09 000 00  1    1    0   0   0    0    0    58
(XEN)  0a 000 00  0    0    0   0   0    0    0    60
(XEN)  0b 000 00  0    0    0   0   0    0    0    68
(XEN)  0c 000 00  0    0    0   0   0    0    0    70
(XEN)  0d 000 00  0    0    0   0   0    0    0    78
(XEN)  0e 000 00  0    0    0   0   0    0    0    88
(XEN)  0f 000 00  0    0    0   0   0    0    0    90
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #8......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #9......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #10......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #11......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ40 -> 0:1
(XEN) IRQ48 -> 0:3
(XEN) IRQ241 -> 0:4
(XEN) IRQ56 -> 0:5
(XEN) IRQ64 -> 0:6
(XEN) IRQ72 -> 0:7
(XEN) IRQ80 -> 0:8
(XEN) IRQ88 -> 0:9
(XEN) IRQ96 -> 0:10
(XEN) IRQ104 -> 0:11
(XEN) IRQ112 -> 0:12
(XEN) IRQ120 -> 0:13
(XEN) IRQ136 -> 0:14
(XEN) IRQ144 -> 0:15
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) calibrating APIC timer ...
(XEN) ..... CPU clock speed is 3399.9136 MHz.
(XEN) ..... host bus clock speed is 199.9948 MHz.
(XEN) ..... bus_scale = 0x0000CCCC
(XEN) checking TSC synchronization across 64 CPUs: 
(XEN) CPU#0 had 7764161 usecs TSC skew, fixed it up.
(XEN) CPU#1 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#2 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#3 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#4 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#5 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#6 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#7 had 7764163 usecs TSC skew, fixed it up.
(XEN) CPU#8 had 3658558 usecs TSC skew, fixed it up.
(XEN) CPU#9 had 3658572 usecs TSC skew, fixed it up.
(XEN) CPU#10 had 3658564 usecs TSC skew, fixed it up.
(XEN) CPU#11 had 3658543 usecs TSC skew, fixed it up.
(XEN) CPU#12 had 3658558 usecs TSC skew, fixed it up.
(XEN) CPU#13 had 3658572 usecs TSC skew, fixed it up.
(XEN) CPU#14 had 3658564 usecs TSC skew, fixed it up.
(XEN) CPU#15 had 3658543 usecs TSC skew, fixed it up.
(XEN) CPU#16 had 2220416 usecs TSC skew, fixed it up.
(XEN) CPU#17 had 2220430 usecs TSC skew, fixed it up.
(XEN) CPU#18 had 2220431 usecs TSC skew, fixed it up.
(XEN) CPU#19 had 2220415 usecs TSC skew, fixed it up.
(XEN) CPU#20 had 2220416 usecs TSC skew, fixed it up.
(XEN) CPU#21 had 2220430 usecs TSC skew, fixed it up.
(XEN) CPU#22 had 2220431 usecs TSC skew, fixed it up.
(XEN) CPU#23 had 2220415 usecs TSC skew, fixed it up.
(XEN) CPU#24 had 748246 usecs TSC skew, fixed it up.
(XEN) CPU#25 had 748248 usecs TSC skew, fixed it up.
(XEN) CPU#26 had 748249 usecs TSC skew, fixed it up.
(XEN) CPU#27 had 748249 usecs TSC skew, fixed it up.
(XEN) CPU#28 had 748246 usecs TSC skew, fixed it up.
(XEN) CPU#29 had 748248 usecs TSC skew, fixed it up.
(XEN) CPU#30 had 748249 usecs TSC skew, fixed it up.
(XEN) CPU#31 had 748249 usecs TSC skew, fixed it up.
(XEN) CPU#32 had -823845 usecs TSC skew, fixed it up.
(XEN) CPU#33 had -823846 usecs TSC skew, fixed it up.
(XEN) CPU#34 had -823844 usecs TSC skew, fixed it up.
(XEN) CPU#35 had -823845 usecs TSC skew, fixed it up.
(XEN) CPU#36 had -823845 usecs TSC skew, fixed it up.
(XEN) CPU#37 had -823846 usecs TSC skew, fixed it up.
(XEN) CPU#38 had -823845 usecs TSC skew, fixed it up.
(XEN) CPU#39 had -823845 usecs TSC skew, fixed it up.
(XEN) CPU#40 had -2997081 usecs TSC skew, fixed it up.
(XEN) CPU#41 had -2997086 usecs TSC skew, fixed it up.
(XEN) CPU#42 had -2997087 usecs TSC skew, fixed it up.
(XEN) CPU#43 had -2997085 usecs TSC skew, fixed it up.
(XEN) CPU#44 had -2997081 usecs TSC skew, fixed it up.
(XEN) CPU#45 had -2997086 usecs TSC skew, fixed it up.
(XEN) CPU#46 had -2997087 usecs TSC skew, fixed it up.
(XEN) CPU#47 had -2997085 usecs TSC skew, fixed it up.
(XEN) CPU#48 had -4549163 usecs TSC skew, fixed it up.
(XEN) CPU#49 had -4549183 usecs TSC skew, fixed it up.
(XEN) CPU#50 had -4549167 usecs TSC skew, fixed it up.
(XEN) CPU#51 had -4549169 usecs TSC skew, fixed it up.
(XEN) CPU#52 had -4549163 usecs TSC skew, fixed it up.
(XEN) CPU#53 had -4549183 usecs TSC skew, fixed it up.
(XEN) CPU#54 had -4549167 usecs TSC skew, fixed it up.
(XEN) CPU#55 had -4549169 usecs TSC skew, fixed it up.
(XEN) CPU#56 had -6021290 usecs TSC skew, fixed it up.
(XEN) CPU#57 had -6021289 usecs TSC skew, fixed it up.
(XEN) CPU#58 had -6021305 usecs TSC skew, fixed it up.
(XEN) CPU#59 had -6021289 usecs TSC skew, fixed it up.
(XEN) CPU#60 had -6021290 usecs TSC skew, fixed it up.
(XEN) CPU#61 had -6021289 usecs TSC skew, fixed it up.
(XEN) CPU#62 had -6021305 usecs TSC skew, fixed it up.
(XEN) CPU#63 had -6021289 usecs TSC skew, fixed it up.
(XEN) Platform timer overflows in 234 jiffies.
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) CPU 0 APIC 0 -> Node 7
(XEN) CPU 1 APIC 4 -> Node 7
(XEN) CPU 2 APIC 8 -> Node 7
(XEN) CPU 3 APIC 12 -> Node 7
(XEN) CPU 4 APIC 2 -> Node 7
(XEN) CPU 5 APIC 6 -> Node 7
(XEN) CPU 6 APIC 10 -> Node 7
(XEN) CPU 7 APIC 14 -> Node 7
(XEN) CPU 8 APIC 16 -> Node 6
(XEN) CPU 9 APIC 20 -> Node 6
(XEN) CPU 10 APIC 24 -> Node 6
(XEN) CPU 11 APIC 28 -> Node 6
(XEN) CPU 12 APIC 18 -> Node 6
(XEN) CPU 13 APIC 22 -> Node 6
(XEN) CPU 14 APIC 26 -> Node 6
(XEN) CPU 15 APIC 30 -> Node 6
(XEN) CPU 16 APIC 32 -> Node 5
(XEN) CPU 17 APIC 36 -> Node 5
(XEN) CPU 18 APIC 40 -> Node 5
(XEN) CPU 19 APIC 44 -> Node 5
(XEN) CPU 20 APIC 34 -> Node 5
(XEN) CPU 21 APIC 38 -> Node 5
(XEN) CPU 22 APIC 42 -> Node 5
(XEN) CPU 23 APIC 46 -> Node 5
(XEN) CPU 24 APIC 48 -> Node 4
(XEN) CPU 25 APIC 52 -> Node 4
(XEN) CPU 26 APIC 56 -> Node 4
(XEN) CPU 27 APIC 60 -> Node 4
(XEN) CPU 28 APIC 50 -> Node 4
(XEN) CPU 29 APIC 54 -> Node 4
(XEN) CPU 30 APIC 58 -> Node 4
(XEN) CPU 31 APIC 62 -> Node 4
(XEN) CPU 32 APIC 64 -> Node 3
(XEN) CPU 33 APIC 68 -> Node 3
(XEN) CPU 34 APIC 72 -> Node 3
(XEN) CPU 35 APIC 76 -> Node 3
(XEN) CPU 36 APIC 66 -> Node 3
(XEN) CPU 37 APIC 70 -> Node 3
(XEN) CPU 38 APIC 74 -> Node 3
(XEN) CPU 39 APIC 78 -> Node 3
(XEN) CPU 40 APIC 80 -> Node 2
(XEN) CPU 41 APIC 84 -> Node 2
(XEN) CPU 42 APIC 88 -> Node 2
(XEN) CPU 43 APIC 92 -> Node 2
(XEN) CPU 44 APIC 82 -> Node 2
(XEN) CPU 45 APIC 86 -> Node 2
(XEN) CPU 46 APIC 90 -> Node 2
(XEN) CPU 47 APIC 94 -> Node 2
(XEN) CPU 48 APIC 96 -> Node 1
(XEN) CPU 49 APIC 100 -> Node 1
(XEN) CPU 50 APIC 104 -> Node 1
(XEN) CPU 51 APIC 108 -> Node 1
(XEN) CPU 52 APIC 98 -> Node 1
(XEN) CPU 53 APIC 102 -> Node 1
(XEN) CPU 54 APIC 106 -> Node 1
(XEN) CPU 55 APIC 110 -> Node 1
(XEN) CPU 56 APIC 112 -> Node 0
(XEN) CPU 57 APIC 116 -> Node 0
(XEN) CPU 58 APIC 120 -> Node 0
(XEN) CPU 59 APIC 124 -> Node 0
(XEN) CPU 60 APIC 114 -> Node 0
(XEN) CPU 61 APIC 118 -> Node 0
(XEN) CPU 62 APIC 122 -> Node 0
(XEN) CPU 63 APIC 126 -> Node 0
(XEN) Brought up 64 CPUs
(XEN) <PM> ACPI (supports S3)
(XEN) Machine check exception polling timer started.
(XEN) AMD IOMMU: Disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80200000 memsz=0x25d848
(XEN) elf_parse_binary: phdr: paddr=0xffffffff8045e000 memsz=0x8a928
(XEN) elf_parse_binary: phdr: paddr=0xffffffff804e9000 memsz=0xc08
(XEN) elf_parse_binary: phdr: paddr=0xffffffff804ea000 memsz=0xcb80c
(XEN) elf_parse_binary: memory: 0xffffffff80200000 -> 0xffffffff805b580c
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff80200000
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80206000
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: FEATURES =
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse: using notes from SHT_NOTE section
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0xffffffff80000000
(XEN)     virt_offset      = 0x0
(XEN)     virt_kstart      = 0xffffffff80200000
(XEN)     virt_kend        = 0xffffffff805b580c
(XEN)     virt_entry       = 0xffffffff80200000
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
0xffffffff805b580c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000480000000->00000004c0000000 (65879351 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff805b580c
(XEN)  Init. ramdisk: ffffffff805b6000->ffffffff80ce0400
(XEN)  Phys-Mach map: ffffffff80ce1000->ffffffffa057f9b8
(XEN)  Start info:    ffffffffa0580000->ffffffffa05804a4
(XEN)  Page tables:   ffffffffa0581000->ffffffffa0688000
(XEN)  Boot stack:    ffffffffa0688000->ffffffffa0689000
(XEN)  TOTAL:         ffffffff80000000->ffffffffa0800000
(XEN)  ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 32 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff80200000 ->
0xffffffff8045d848
(XEN) elf_load_binary: phdr 1 at 0xffffffff8045e000 ->
0xffffffff804e8928
(XEN) elf_load_binary: phdr 2 at 0xffffffff804e9000 ->
0xffffffff804e9c08
(XEN) elf_load_binary: phdr 3 at 0xffffffff804ea000 ->
0xffffffff80523108
(XEN) Initrd len 0x72a400, start at 0xffffffff805b6000
(XEN) Scrubbing Free RAM: ..done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 116kB init memory.
(XEN) traps.c:414:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0
[ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.3-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff804fe6f4>]
(XEN) RFLAGS: 0000000000000282   CONTEXT: guest
(XEN) rax: 00000000ffffffea   rbx: ffffffffc0125000   rcx:
00000004a0582920
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi:
00000000deadbeef
(XEN) rbp: ffffffffa0582000   rsp: ffffffff804ebe78   r8:
0000000003f14537
(XEN) r9:  0000000000000000   r10: 00000000deadbeef   r11:
0000000000001000
(XEN) r12: ffffffff80205000   r13: 0000003f14537000   r14:
000000001f9a1000
(XEN) r15: ffffffff80205000   cr0: 000000008005003b   cr4:
00000000000026f0
(XEN) cr3: 0000000480201000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff804ebe78:
(XEN)    00000004a0582920 0000000000001000 ffffffff804fe6f4
000000010000e030
(XEN)    0000000000010082 ffffffff804ebeb8 000000000000e02b
ffffffff804fe6f0
(XEN)    0000000000000000 ffffffff80230447 0000003000000008
ffffffff804ebfa8
(XEN)    ffffffff804ebee8 0000000020783000 0000003000000001
ffffffff805290bc
(XEN)    0000000000000000 ffffffff8052e29c ffffffff8052e29c
ffffffff804ebfb8
(XEN)    0000000000000000 ffffffff804f95b0 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000205023 0000000000000023 0000000000205023
ffffffff805290bd
(XEN)    0000000000040800 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 ffffffff804f35da 0000000000000000
0000000000000000
(XEN)    ffffffff8052e220 ffffffff804f3215 ffff800000000000
ffff804000000000
(XEN)    00000007ffffffff 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
00af9a000000ffff
(XEN)    00cf92000000ffff 00cffa000000ffff 00cff2000000ffff
00affa000000ffff
(XEN)    00cf9a000000ffff 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
[-- MARK -- Mon Jan 28 15:00:00 2008]

Re: Test results on Unisys ES7000 64x 256gb usingunstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Carb, Brian A wrote:
> My environment also produces a panic (unstable cs 16852 on x64
> SLES10sp1) when I boot with 64cpu and 256gb and dom0_mem not specified:
>   traps.c:414:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0
> [ec=0000]
> 
> with dom0_mem=4096, system boots fine.
> 
>  
> brian carb
> unisys corporation - malvern, pa
> 
> 

<snip>

This may be of use. When it fails, the dom0 reports:

	Xen reported: 6799.922 MHz processor.
	...
	Calibrating delay using timer specific routine.. 17069.04 BogoMIPS 	(lpj=34138093)

When it works, dom0 reports the correct numbers:
	Xen reported: 3399.952 MHz processor.
	...
	Calibrating delay using timer specific routine.. 8505.35 BogoMIPS (lpj=17010701)

Strange that the numbers are basically 2x the correct values.

Also have had it fail at a lower memory (96GB) with all
64 CPUs. When I was running a HV built for only 32 it worked
until I passed 112GB to dom0.

 Bill

RE: Test results on Unisys ES7000 64x 256gbusingunstablec/s 16693 on 3.2.0 Release Candidate

[Jan Beulich]

>...
>(XEN) traps.c:414:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0
>[ec=0000]
>(XEN) domain_crash_sync called from entry.S
>(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
>(XEN) ----[ Xen-3.3-unstable  x86_64  debug=y  Not tainted ]----
>(XEN) CPU:    0
>(XEN) RIP:    e033:[<ffffffff804fe6f4>]
>(XEN) RFLAGS: 0000000000000282   CONTEXT: guest
>(XEN) rax: 00000000ffffffea   rbx: ffffffffc0125000   rcx:
>00000004a0582920
>(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi:
>00000000deadbeef
>(XEN) rbp: ffffffffa0582000   rsp: ffffffff804ebe78   r8:
>0000000003f14537
>(XEN) r9:  0000000000000000   r10: 00000000deadbeef   r11:
>0000000000001000
>(XEN) r12: ffffffff80205000   r13: 0000003f14537000   r14:
>000000001f9a1000
>(XEN) r15: ffffffff80205000   cr0: 000000008005003b   cr4:
>00000000000026f0
>(XEN) cr3: 0000000480201000   cr2: 0000000000000000
>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
>(XEN) Guest stack trace from rsp=ffffffff804ebe78:
>...

This looks like a failed hypercall (and a BUG() following it), but without
having the kernel binary it's impossible to tell which one. You should
be able to find out, though. With that information, if it doesn't directly
provide a clue, it should then be possible to instrument the respective
hypercall handler to find out what's going wrong.

Jan

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Keir Fraser wrote:
> On 28/1/08 14:02, "Bill Burns" <bburns@redhat.com> wrote:
> 
>> Ok, some progress. Background is that 3.1.2 (and 3.1.3 at least
>> as it was a wek or two ago) fails to boot on a 64 CPU es7000 with
>> over 112GB of memory. This is with both HV & dom0 being x86_64.
>> The symptom is that the dom0 kernel gets time went backwards
>> error during init.
>>
>> The patch at which this first fails is 15137, which is the patch
>> that introduces using the ACPI PM timer as the clock
>> source. If I include the next patch (that allows for clock
>> selection) and choose pit as clock source the system boots
>> fine. Without the arg the ACPI timer is used and I get the hang.
> 
> The obvious question then is what happens to the ACPI PM timer when dom0
> gets more than 112GB of memory. Perhaps it's worth adding some tracing to
> Xen and see whether e.g., the platform timer stops running?
> 
>  -- Keir
> 
> 

I enabled the printk in local_time_calibration in Xen's time.c
and added a similar one to init_cpu_khz in time-xen.c in the
dom0 kernel.

The hypervisor outputs many line like:
	(XEN) ---10: 00000000 9697086f -1
where the key value is always 969xxxxx...

Until we get to scrubing free ram:

	(XEN) Initrd len 0x894600, start at 0xffffffff80702000
	(XEN) Scrubbing Free RAM: ---0: 80000000 498c0b61 -2
	(XEN) .done.

The bogus 498c0b61 value is seen by the dom0 kernel and is
used to improperly calculate the CPU speed. A further printk
in the dom0's get_time_values_from_xen shows that all the CPUs
except the first have good values, leading right into the
first time went backwards message...

ACPI: Core revision 20060707
get_time_values_from_xen tsc_to_nsec_mul 498c0b61 ver 2
Initializing CPU#1
get_time_values_from_xen tsc_to_nsec_mul 969703ce ver 2
Initializing CPU#2
get_time_values_from_xen tsc_to_nsec_mul 9697099e ver 2
Initializing CPU#3
get_time_values_from_xen tsc_to_nsec_mul 9697068c ver 2
Initializing CPU#4
get_time_values_from_xen tsc_to_nsec_mul 96970374 ver 2
Initializing CPU#5
get_time_values_from_xen tsc_to_nsec_mul 96970a55 ver 2
Initializing CPU#6
get_time_values_from_xen tsc_to_nsec_mul 96970a7c ver 2
Initializing CPU#7
get_time_values_from_xen tsc_to_nsec_mul 96970952 ver 2
Initializing CPU#8
get_time_values_from_xen tsc_to_nsec_mul 969708f3 ver 2
Initializing CPU#9
get_time_values_from_xen tsc_to_nsec_mul 969708f8 ver 2
Initializing CPU#10
get_time_values_from_xen tsc_to_nsec_mul 96970a55 ver 2
Initializing CPU#11
get_time_values_from_xen tsc_to_nsec_mul 969706f6 ver 2
Initializing CPU#12
get_time_values_from_xen tsc_to_nsec_mul 96970bdc ver 2
Initializing CPU#13
get_time_values_from_xen tsc_to_nsec_mul 9697069b ver 2
Initializing CPU#14
get_time_values_from_xen tsc_to_nsec_mul 96970997 ver 2
Initializing CPU#15
get_time_values_from_xen tsc_to_nsec_mul 969707c5 ver 2
Initializing CPU#16
get_time_values_from_xen tsc_to_nsec_mul 969707ff ver 2
Initializing CPU#17
get_time_values_from_xen tsc_to_nsec_mul 969707aa ver 2
Initializing CPU#18
get_time_values_from_xen tsc_to_nsec_mul 9697062a ver 2
Initializing CPU#19
get_time_values_from_xen tsc_to_nsec_mul 969707d7 ver 2
Initializing CPU#20
get_time_values_from_xen tsc_to_nsec_mul 969709be ver 2
Initializing CPU#21
get_time_values_from_xen tsc_to_nsec_mul 9697096f ver 2
Initializing CPU#22
get_time_values_from_xen tsc_to_nsec_mul 96970902 ver 2
Initializing CPU#23
get_time_values_from_xen tsc_to_nsec_mul 969709a8 ver 2
Initializing CPU#24
get_time_values_from_xen tsc_to_nsec_mul 96970778 ver 2
Initializing CPU#25
get_time_values_from_xen tsc_to_nsec_mul 969705ad ver 2
Initializing CPU#26
get_time_values_from_xen tsc_to_nsec_mul 96970b44 ver 2
Initializing CPU#27
get_time_values_from_xen tsc_to_nsec_mul 96970974 ver 2
Initializing CPU#28
get_time_values_from_xen tsc_to_nsec_mul 96970bb4 ver 2
Initializing CPU#29
get_time_values_from_xen tsc_to_nsec_mul 969708c1 ver 2
Initializing CPU#30
get_time_values_from_xen tsc_to_nsec_mul 96970c23 ver 2
Brought up 32 CPUs
Initializing CPU#31
get_time_values_from_xen tsc_to_nsec_mul 9697085b ver 2
get_time_values_from_xen tsc_to_nsec_mul 969703ce ver 2
get_time_values_from_xen tsc_to_nsec_mul 498c0b61 ver 2
Timer ISR/0: Time went backwards: delta=-35017583219 delta_cpu=10416781
shadow=9160708347 off=11318417225 processed=55496708347 cpu_processed=20468708347

Note that this Hypervisor was only built for 32 CPUs,
not all 64.

So the problem seems to occur in the HV itself when it
tries to scrub the free memory. Funny that when it has
lots to scrub, when dom0 is restricted to less memory,
there is no issue. But then there is little to scrub,
but a lot of memory for dom0, things go wrong.

 Bill

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Keir Fraser]

On 30/1/08 16:20, "Bill Burns" <bburns@redhat.com> wrote:

> Until we get to scrubing free ram:
> 
> (XEN) Initrd len 0x894600, start at 0xffffffff80702000
> (XEN) Scrubbing Free RAM: ---0: 80000000 498c0b61 -2
> (XEN) .done.

That error factor (0x80000000) is as bad as it possibly can be. This means
that local_stime is running at least one second ahead of master_stime.
Probably the platform timer (which is used to compute master_stime) has gone
mad. In this case of course the platform timer is the ACPI PM timer.

I suggest instrumenting read_pmtimer_count() to see what value is getting
returned from there, to ensure it is monotonically increasing at about the
right rate. And also that it *is* actually getting invoked!

 -- Keir

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

Keir Fraser wrote:
> On 30/1/08 16:20, "Bill Burns" <bburns@redhat.com> wrote:
> 
>> Until we get to scrubing free ram:
>>
>> (XEN) Initrd len 0x894600, start at 0xffffffff80702000
>> (XEN) Scrubbing Free RAM: ---0: 80000000 498c0b61 -2
>> (XEN) .done.
> 
> That error factor (0x80000000) is as bad as it possibly can be. This means
> that local_stime is running at least one second ahead of master_stime.
> Probably the platform timer (which is used to compute master_stime) has gone
> mad. In this case of course the platform timer is the ACPI PM timer.
> 
> I suggest instrumenting read_pmtimer_count() to see what value is getting
> returned from there, to ensure it is monotonically increasing at about the
> right rate. And also that it *is* actually getting invoked!
> 

I instrumented read_pmtimer_count and it does get called a couple
of times during startup and it returns an increasing value. I made
no attempt to determine if the rate was correct.

I also tried the 3.1.3 and 3.2.0 Hypervisors and they exhibit
the same problem on the es7000.

It's not clear if is this a platform specific problem
or not. So far the only reports of the problem are on the
es7000. So for now I am running the attached patch to
avoid the pmtimer on es7000 only.

 Bill


>  -- Keir
> 
> 


[-- Attachment #2: xen-es7000-nopmtimer.patch --]
[-- Type: text/x-patch, Size: 435 bytes --]

--- xen/arch/x86/genapic/es7000plat.c.orig	2007-12-06 12:48:38.000000000 -0500
+++ xen/arch/x86/genapic/es7000plat.c	2008-01-31 11:56:30.000000000 -0500
@@ -296,9 +296,14 @@ es7000_stop_cpu(int cpu)
 
 }
 
+extern u32 pmtmr_ioport;
+
 void __init
 es7000_sw_apic()
 {
+	/* don't use pmtimer on es7000 */
+	pmtmr_ioport = 0;
+  
 	if (es7000_plat && (es7000_plat != ES7000_ZORRO)) {
 		int mip_status;
 		struct mip_reg es7000_mip_reg;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Keir Fraser]

On 31/1/08 18:12, "Bill Burns" <bburns@redhat.com> wrote:

> I instrumented read_pmtimer_count and it does get called a couple
> of times during startup and it returns an increasing value. I made
> no attempt to determine if the rate was correct.
> 
> I also tried the 3.1.3 and 3.2.0 Hypervisors and they exhibit
> the same problem on the es7000.
> 
> It's not clear if is this a platform specific problem
> or not. So far the only reports of the problem are on the
> es7000. So for now I am running the attached patch to
> avoid the pmtimer on es7000 only.

Fair enough, but obviously this does need to be fixed: while not understood
it could point to any amount of other badness or corruption happening in the
background that we simply have not yet discovered.

 -- Keir

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Keir Fraser wrote:
> On 31/1/08 18:12, "Bill Burns" <bburns@redhat.com> wrote:
> 
>> I instrumented read_pmtimer_count and it does get called a couple
>> of times during startup and it returns an increasing value. I made
>> no attempt to determine if the rate was correct.
>>
>> I also tried the 3.1.3 and 3.2.0 Hypervisors and they exhibit
>> the same problem on the es7000.
>>
>> It's not clear if is this a platform specific problem
>> or not. So far the only reports of the problem are on the
>> es7000. So for now I am running the attached patch to
>> avoid the pmtimer on es7000 only.
> 
> Fair enough, but obviously this does need to be fixed: while not understood
> it could point to any amount of other badness or corruption happening in the
> background that we simply have not yet discovered.
> 
>  -- Keir
> 


I am hoping to get some feedback on a similar large
system. Limiting dom0 memory seems to be the standard
operating procedure on large memory systems so if it's
a common problem then it's being masked.

 Bill

Re: Test results on Unisys ES7000 64x 256gb using unstablec/s 16693 on 3.2.0 Release Candidate

[Bill Burns]

Bill Burns wrote:
> Keir Fraser wrote:
>> On 31/1/08 18:12, "Bill Burns" <bburns@redhat.com> wrote:
>>
>>> I instrumented read_pmtimer_count and it does get called a couple
>>> of times during startup and it returns an increasing value. I made
>>> no attempt to determine if the rate was correct.
>>>
>>> I also tried the 3.1.3 and 3.2.0 Hypervisors and they exhibit
>>> the same problem on the es7000.
>>>
>>> It's not clear if is this a platform specific problem
>>> or not. So far the only reports of the problem are on the
>>> es7000. So for now I am running the attached patch to
>>> avoid the pmtimer on es7000 only.
>> Fair enough, but obviously this does need to be fixed: while not understood
>> it could point to any amount of other badness or corruption happening in the
>> background that we simply have not yet discovered.
>>
>>  -- Keir
>>
> 
> 
> I am hoping to get some feedback on a similar large
> system. Limiting dom0 memory seems to be the standard
> operating procedure on large memory systems so if it's
> a common problem then it's being masked.
> 

Got a report on a non-es7000 failure as well. They did not
indicate the actual failure, but said it failed to boot.
They induced the failure by both increasing the value of the
dom0_mem arg and by removing the arg entirely. They did not try
clocksource, but I have aske them too. So we can't be sure
yet that it's the same thing. But I think odds are it is.

 Bill


>  Bill
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Large system boot problems

[Bill Burns]

Here is some debug of the large memory / pmtimer issue.

(for background see [Xen-devel] Test results on Unisys ES7000 64x 256gb using
unstablec/s 16693 on 3.2.0 Release Candidate from Jan 9, 2008)

Symptom: A system with lots of CPUs and memory can fail
to boot up properly. Dom0 gets time going backwards
errors and effectively hangs during initialization.
The cause of dom0's init failure is due to it
using bogus values for CPU0's speed, while the other
CPUs have proper speed info.

Workarounds: Increasing the memory retained by the
Hypervisor by either a dom0_mem or a xenheap arg will delay
the start of dom0 enough (while that memory is scrubbed)
that the HV cpu speed calculation will self-correct.
Changing the timer used can also work (pit works for me)
but basically it's a race and I expect that
with the right hardware situation it could fail too.

Details: With either pmtimer or pit the initial calculation
done for CPU0 speed is bad (at least on a large system). If the
dom0 starts quickly enough that it reads the bad CPU speed
data from the Hypervisor shared area before the Hypervisor
corrects it, dom0 is in trouble.

Debug details:
When the Xen boot has sized memory, detected and
booted all the CPUs, and gets to the point of

	(XEN) ENABLING IO-APIC IRQs

init_percpu_time gets called for the CPU0 and the
cpu_time values recorded are:

	(XEN) dump_cpu_time cpu0 addr ffff828c801ca520
	(XEN) local_tsc_stamp 1691332805
	(XEN) stime_master_stamp 0
	(XEN) stime_local_stamp 0
	(XEN) Platform timer overflows in 234 jiffies.
	(XEN) Platform timer is 3.579MHz ACPI PM Timer

Then domain 0 is loaded, and local_time_calibration for CPU0
gets called and actually does something. The "out count" below
indicates that it was called 315 times and due to

    if ( ((s64)stime_elapsed64 < (EPOCH / 2)) )

effectively did nothing on those calls.

The result of the calculations in local_time_calibration
with the huge difference in the tsc value screws up pretty
badly:

	(XEN) local_time_calibration error factor cpu0 is 0x80000000. out count 315
	(XEN) PRE0: tsc=1691332805 stime=0 master=0
	(XEN) CUR0: tsc=33466953185 stime=9345455787 master=4641208868 -> -4704246919
	(XEN) calibration_mul_frac 4ac8a18d tsc_shift -2

The bogus values here are then used by dom0 to incorrectly determine
the frequency of CPU0, while all other CPUs have correct values.

	Xen reported: 13692.820 MHz processor.

For the HV, this self corrects, as the next time local_time_calibration
gets called the data in cpu_time is properly set. But the damage
has been done and dom0 struggles to make progress and reports
time going backwards, etc.

The reason that limiting the memory given to dom0 fixes
the problem is that the loop that scrubs the memory
that the HV is keeping (scrub_heap_pages) periodically
calls process_pending_timers and if there is enough memory
there, then the correction happens before dom0 starts.

This recalls a comment from a vendor a few months ago
where they said you needed to add a xenheap arg to make
large memory work.

When doing clocksource=pit a similar thing happens where
the initial calc is bad, but it gets fixed before dom0
gets going (debug from PIT):

	(XEN) dump_cpu_time cpu0 addr ffff828c801ca520
	(XEN) local_tsc_stamp 226384274
	(XEN) stime_master_stamp 0
	(XEN) stime_local_stamp 0
	(XEN) Platform timer overflows in 2 jiffies.
	(XEN) Platform timer is 1.193MHz PIT

there are no "goto out's" taken, the next call to local_time_calibration
does the bad calculation

	(XEN) Scrubbing Free RAM: .local_time_calibration error factor cpu0 is
0x80000000. out count 0
	(XEN) PRE0: tsc=226384274 stime=0 master=0
	(XEN) CUR0: tsc=35424564759 stime=10351900878 master=1052517641 -> -9299383237
	(XEN) calibration_mul_frac 7a7b2a1a tsc_shift -5

next call to local_time_calibration fixes it..

	(XEN) calibration_mul_frac 969714d2 tsc_shift -1

and dom0 get the right stuff

	Xen reported: 3399.956 MHz processor.


Looking for ideas or suggestions on how to solve this issue.
Ideally we'd be able to prevent the bogus calculation in the
first place.

 Bill

Re: Large system boot problems

[Keir Fraser]

On 8/2/08 13:49, "Bill Burns" <bburns@redhat.com> wrote:

> Looking for ideas or suggestions on how to solve this issue.
> Ideally we'd be able to prevent the bogus calculation in the
> first place.

Could the bad value coming from calibration in init_pit_and_calibrate_tsc(),
or perhaps it is somehow caused perhaps by not-frequent-enough softirq
handling on CPU0 as it brings up all the secondary CPUs (and hence it is not
going through local_time_calibration() during that period)?

 -- Keir

Re: Large system boot problems

[Bill Burns]

Keir Fraser wrote:
> On 8/2/08 13:49, "Bill Burns" <bburns@redhat.com> wrote:
> 
>> Looking for ideas or suggestions on how to solve this issue.
>> Ideally we'd be able to prevent the bogus calculation in the
>> first place.
> 
> Could the bad value coming from calibration in init_pit_and_calibrate_tsc(),
> or perhaps it is somehow caused perhaps by not-frequent-enough softirq
> handling on CPU0 as it brings up all the secondary CPUs (and hence it is not
> going through local_time_calibration() during that period)?
> 

The message from early_time_init (caller of
iinit_pit_and_calibrate_tsc, indicates that the
initial detection is ok:

(pmtimer case)	(XEN) Detected 3400.114 MHz processor.
((pit case)  	(XEN) Detected 3400.165 MHz processor.

So I think it's the latter. The init of a large system
is staving off the soft irq so that the next calc fails.

 Bill


(XEN) Detected 3400.114 MHz processor.

>  -- Keir
> 
> 

Re: Large system boot problems

[Keir Fraser]

On 8/2/08 15:10, "Bill Burns" <bburns@redhat.com> wrote:

> The message from early_time_init (caller of
> iinit_pit_and_calibrate_tsc, indicates that the
> initial detection is ok:
> 
> (pmtimer case) (XEN) Detected 3400.114 MHz processor.
> ((pit case)   (XEN) Detected 3400.165 MHz processor.
> 
> So I think it's the latter. The init of a large system
> is staving off the soft irq so that the next calc fails.

Okay, well you could test this by inserting a process_pending_timers() in
the CPU-booting loop in smpboot.c. If you do timer work after booting each
CPU, perhaps that makes the problem go away?

But ultimately the calibration code should be robust to long delays before
it is executed. It shouldn't go haywire. So something is bad there. Do you
have a dump of the decision made by the calibration code on cpu0 the very
first time it actually gets invoked? We probably need to trace the hell out
of that first invocation to work out why it gets things so badly wrong.

 -- Keir

Re: Large system boot problems

[Bill Burns]

Keir Fraser wrote:
> On 8/2/08 15:10, "Bill Burns" <bburns@redhat.com> wrote:
> 
>> The message from early_time_init (caller of
>> iinit_pit_and_calibrate_tsc, indicates that the
>> initial detection is ok:
>>
>> (pmtimer case) (XEN) Detected 3400.114 MHz processor.
>> ((pit case)   (XEN) Detected 3400.165 MHz processor.
>>
>> So I think it's the latter. The init of a large system
>> is staving off the soft irq so that the next calc fails.
> 
> Okay, well you could test this by inserting a process_pending_timers() in
> the CPU-booting loop in smpboot.c. If you do timer work after booting each
> CPU, perhaps that makes the problem go away?

I woke up in the middle of the night with that idea
a few days ago and tried it without success. Seemed that
calls to process_pending_timers had no effect until
a certain point. But I need to go and look at that
some more and see why...


> But ultimately the calibration code should be robust to long delays before
> it is executed. It shouldn't go haywire. So something is bad there. Do you
> have a dump of the decision made by the calibration code on cpu0 the very
> first time it actually gets invoked? We probably need to trace the hell out
> of that first invocation to work out why it gets things so badly wrong.

I don't have more than in the earlier email where is shows the
large delta in tsc time, which seems to cause the bogus result.

 Bill


> 
>  -- Keir
> 
> 

Re: Large system boot problems

[Keir Fraser]

On 8/2/08 15:22, "Bill Burns" <bburns@redhat.com> wrote:

>> But ultimately the calibration code should be robust to long delays before
>> it is executed. It shouldn't go haywire. So something is bad there. Do you
>> have a dump of the decision made by the calibration code on cpu0 the very
>> first time it actually gets invoked? We probably need to trace the hell out
>> of that first invocation to work out why it gets things so badly wrong.
> 
> I don't have more than in the earlier email where is shows the
> large delta in tsc time, which seems to cause the bogus result.

Okay, well looking at the inputs on that first invocation -- master_stime
and local_stime -- they are totally out of sync. One says that 9.3s has
elapsed since init_xen_time() was invoked, the other says that 4.6s has
elapsed (curiously exactly half the time). The former is correct if the CPU
really is a 3.4GHz part and is running at full speed for the duration. But
you ought to be able to work out which is the correct ballpark by timing
with a stopwatch the time between init_xen_time() and that first invocation
on cpu0 of local_time_calibration() (you'll have to printk() when
init_xen_time() is executed).

 -- Keir

Re: Large system boot problems

[Bill Burns]

[-- Attachment #1: Type: text/plain, Size: 2752 bytes --]

Keir Fraser wrote:
> On 8/2/08 15:22, "Bill Burns" <bburns@redhat.com> wrote:
> 
>>> But ultimately the calibration code should be robust to long delays before
>>> it is executed. It shouldn't go haywire. So something is bad there. Do you
>>> have a dump of the decision made by the calibration code on cpu0 the very
>>> first time it actually gets invoked? We probably need to trace the hell out
>>> of that first invocation to work out why it gets things so badly wrong.
>> I don't have more than in the earlier email where is shows the
>> large delta in tsc time, which seems to cause the bogus result.
> 
> Okay, well looking at the inputs on that first invocation -- master_stime
> and local_stime -- they are totally out of sync. One says that 9.3s has
> elapsed since init_xen_time() was invoked, the other says that 4.6s has
> elapsed (curiously exactly half the time). The former is correct if the CPU
> really is a 3.4GHz part and is running at full speed for the duration. But
> you ought to be able to work out which is the correct ballpark by timing
> with a stopwatch the time between init_xen_time() and that first invocation
> on cpu0 of local_time_calibration() (you'll have to printk() when
> init_xen_time() is executed).
> 
>  -- Keir
> 
> 

Well, I have a proposed fix that fixes the major symptom
of dom0 reporting time going backwards and failing it initialize
properly. I must note that dom0 still reports the wrong speed for
CPU0 when only one iteration of local_time_calibration occurs
before dom0 gets going. I believe that second issue is probably
due to the large delta between the master and local stime.

The first call to local_time_calibration automatically fixes
local stime being behind.

But when a significant amount of time has elapsed before the
initial call to local_time_calibration the code that deals with
the local stime and tsc deltas is broken. When the 64 bit deltas
for local stime is manipulated down to a 32 bit value the
tsc delta is also adjusted, but the tsc_shift value is
not maintained.

There are two loops. The first shifts both the stime and
tsc vaules in sync but fails to record the tsc shift:

    while ( ((u32)stime_elapsed64 != stime_elapsed64) ||
            ((s32)stime_elapsed64 < 0) )
    {
        stime_elapsed64 >>= 1;
        tsc_elapsed64   >>= 1;
++      tsc_shift--;
    }


The second does the tsc shift alone, which is fine, but note
that it does record the tsc shift.

    /* tsc_elapsed <= 2*stime_elapsed */
    while ( tsc_elapsed64 > (stime_elapsed32 * 2) )
    {
        tsc_elapsed64 >>= 1;
        tsc_shift--;
    }

Making this one line change, as in the attached patch
yields a properly working dom0. Tested on both a small
memory and large memory system.

 Bill

[-- Attachment #2: fix-tsc-shift.patch --]
[-- Type: text/x-patch, Size: 336 bytes --]

--- arch/x86/time.c.orig	2008-02-12 07:16:48.000000000 -0500
+++ arch/x86/time.c	2008-02-12 11:19:47.000000000 -0500
@@ -857,6 +857,7 @@ static void local_time_calibration(void 
     {
         stime_elapsed64 >>= 1;
         tsc_elapsed64   >>= 1;
+        tsc_shift--;
     }
 
     /* stime_master_diff now fits in a 32-bit word. */

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Large system boot problems

[Keir Fraser]

On 12/2/08 16:34, "Bill Burns" <bburns@redhat.com> wrote:

> Making this one line change, as in the attached patch
> yields a properly working dom0. Tested on both a small
> memory and large memory system.

It's not really correct though. The idea of the calibration algorithm is to
find a multiplier d(system_time)/d(tsc) which we can use to calculate
system_time deltas from tsc deltas by a simple multiplication.

The tsc_shift is effectively the exponent part of a custom floating-point
format (that mantissa part being calibration_mul_frac). The idea is to shift
tsc_elapsed such that stime_elapsed/tsc_elapsed yields a fraction between
0.5 and 1, and therefore maximises significant bits in the mantissa
representation which has an implicit leading radix point.

*So*, the upshot of all this is that when we shift stime_elapsed and
tsc_elapsed together, we do not change the value of
d(delta_stime)/d(delta_tsc) and so of course the exponent of that value
(tsc_shift) does not change.

Put another way, you've perturbed the system and coincidentally made the bug
go into hiding. But it's not really fixed and, in fact, your patch is simply
broken from an algorithmic point of view.

 -- Keir