Kernel error with todays -unstable.

Kernel error with todays -unstable.

[Matt Ayres]

Got this changing from  2.6.16.13 to 2.6.16.29.. running .13 under the 
same xen binary works fine. 
 
 

 
                                                             __  __ 
        _____  ___                     _        _     _ 

  \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 

   \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ 
\
   /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | | 
__/
  /_/\_\___|_| |_| |____(_)___/    \__,_|_| 
|_|___/\__\__,_|_.__/|_|\___| 

 

  http://www.cl.cam.ac.uk/netos/xen 

  University of Cambridge Computer Laboratory 

 

  Xen version 3.0-unstable (root@unixshell.com) (gcc version 4.0.2 
20051125 (Red
Hat 4.0.2-8)) Thu Sep 21 14:00:38 EDT 2006 

  Latest ChangeSet: Thu Sep 21 15:04:29 2006 +0100 11560:5bc0e9a50687 

 

(XEN) Command line: /xen.gz dom0_mem=262144 com1=9600,8n1 sched=credit 

(XEN) Physical RAM map: 

(XEN)  0000000000000000 - 000000000009fc00 (usable) 

(XEN)  000000000009fc00 - 00000000000a0000 (reserved) 

(XEN)  00000000000e0000 - 0000000000100000 (reserved) 

(XEN)  0000000000100000 - 00000000faff0000 (usable) 

(XEN)  00000000faff0000 - 00000000fafff000 (ACPI data) 

(XEN)  00000000fafff000 - 00000000fb000000 (ACPI NVS) 

(XEN)  00000000ff780000 - 0000000100000000 (reserved) 

(XEN)  0000000100000000 - 0000000200000000 (usable) 

(XEN) System RAM: 8111MB (8306236kB) 

(XEN) Xen heap: 9MB (10152kB) 

(XEN) PAE enabled, limit: 16 GB 

(XEN) found SMP MP-table at 000ff780 

(XEN) DMI 2.3 present. 

(XEN) Using APIC driver default 

(XEN) ACPI: RSDP (v000 ACPIAM                                ) @ 
0x000f6890
(XEN) ACPI: RSDT (v001 A M I  OEMRSDT  0x06000429 MSFT 0x00000097) @ 
0xfaff0000
(XEN) ACPI: FADT (v001 A M I  OEMFACP  0x06000429 MSFT 0x00000097) @ 
0xfaff0200
(XEN) ACPI: MADT (v001 A M I  OEMAPIC  0x06000429 MSFT 0x00000097) @ 
0xfaff0380
(XEN) ACPI: OEMB (v001 A M I  OEMBIOS  0x06000429 MSFT 0x00000097) @ 
0xfafff040
(XEN) ACPI: SRAT (v001 A M I  OEMSRAT  0x06000429 MSFT 0x00000097) @ 
0xfaff3320
(XEN) ACPI: ASF! (v001 AMIASF AMDSTRET 0x00000001 INTL 0x02002026) @ 
0xfaff3410
(XEN) ACPI: DSDT (v001  0AAAA 0AAAA000 0x00000000 INTL 0x02002026) @ 
0x00000000
(XEN) ACPI: Local APIC address 0xfee00000 

(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) 

(XEN) Processor #0 15:5 APIC version 16 

(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) 

(XEN) Processor #1 15:5 APIC version 16 

(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) 

(XEN) IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23 

(XEN) ACPI: IOAPIC (id[0x03] address[0xfebff000] gsi_base[24]) 

(XEN) IOAPIC[1]: apic_id 3, version 17, address 0xfebff000, GSI 24-27 

(XEN) ACPI: IOAPIC (id[0x04] address[0xfebfe000] gsi_base[28]) 

(XEN) IOAPIC[2]: apic_id 4, version 17, address 0xfebfe000, GSI 28-31 

(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 

(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 

(XEN) ACPI: IRQ0 used by override. 

(XEN) ACPI: IRQ2 used by override. 

(XEN) Enabling APIC mode:  Flat.  Using 3 I/O APICs 

(XEN) Using ACPI (MADT) for SMP configuration information 

(XEN) Using scheduler: SMP Credit Scheduler (credit) 

(XEN) Initializing CPU#0 

(XEN) Detected 1793.652 MHz processor. 

(XEN) CPU0: AMD Flush Filter disabled 

(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) 

(XEN) CPU: L2 Cache: 1024K (64 bytes/line) 

(XEN) Intel machine check architecture supported. 

(XEN) Intel machine check reporting enabled on CPU#0. 

(XEN) CPU0: AMD Opteron(tm) Processor 244 stepping 08 

(XEN) Booting processor 1/1 eip 90000 

(XEN) Initializing CPU#1 

(XEN) CPU1: AMD Flush Filter disabled 

(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) 

(XEN) CPU: L2 Cache: 1024K (64 bytes/line) 

(XEN) AMD: Disabling C1 Clock Ramping Node #0 

(XEN) AMD: Disabling C1 Clock Ramping Node #1 

(XEN) Intel machine check architecture supported. 

(XEN) Intel machine check reporting enabled on CPU#1. 

(XEN) CPU1: AMD Opteron(tm) Processor 244 stepping 08 

(XEN) Total of 2 processors activated. 

(XEN) ENABLING IO-APIC IRQs 

(XEN)  -> Using new ACK method 

(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0 

(XEN) checking TSC synchronization across 2 CPUs: passed. 

(XEN) Platform timer is 1.193MHz PIT 

(XEN) Brought up 2 CPUs 

(XEN) Machine check exception polling timer started. 

(XEN) *** LOADING DOMAIN 0 *** 

(XEN) Domain 0 kernel supports features = { 0000001f }. 

(XEN) Domain 0 kernel requires features = { 00000000 }. 

(XEN) PHYSICAL MEMORY ARRANGEMENT: 

(XEN)  Dom0 alloc.:   0000000004800000->0000000005000000 (63488 pages to 
be allo
cated) 

(XEN) VIRTUAL MEMORY ARRANGEMENT: 

(XEN)  Loaded kernel: c0100000->c05bd628 

(XEN)  Init. ramdisk: c05be000->c05be000 

(XEN)  Phys-Mach map: c05be000->c05fe000 

(XEN)  Start info:    c05fe000->c05fe46c 

(XEN)  Page tables:   c05ff000->c0608000 

(XEN)  Boot stack:    c0608000->c0609000 

(XEN)  TOTAL:         c0000000->c0800000 

(XEN)  ENTRY ADDRESS: c0100000 

(XEN) Dom0 has maximum 2 VCPUs 

(XEN) Scrubbing Free RAM: 
...................................................... 

............................done. 

(XEN) Xen trace buffers: disabled 

(XEN) Xen is relinquishing VGA console. 

(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
input to Xen
). 

Linux version 2.6.16.29-xen (root@vm19.unixshell.com) (gcc version 4.0.2 
2005112
5 (Red Hat 4.0.2-8)) #4 SMP Thu Sep 21 14:01:34 EDT 2006 

BIOS-provided physical RAM map: 

  Xen: 0000000000000000 - 0000000010800000 (usable) 

0MB HIGHMEM available. 

264MB LOWMEM available. 

NX (Execute Disable) protection: active 

found SMP MP-table at 000ff780 

DMI 2.3 present. 

ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) 

ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) 

ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) 

IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23 

ACPI: IOAPIC (id[0x03] address[0xfebff000] gsi_base[24]) 

IOAPIC[1]: apic_id 3, version 17, address 0xfebff000, GSI 24-27 

ACPI: IOAPIC (id[0x04] address[0xfebfe000] gsi_base[28]) 

IOAPIC[2]: apic_id 4, version 17, address 0xfebfe000, GSI 28-31 

ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 

ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 

Enabling APIC mode:  Flat.  Using 3 I/O APICs 

Using ACPI (MADT) for SMP configuration information 

Allocating PCI resources starting at fb800000 (gap: fb000000:04780000) 

Built 1 zonelists 

Kernel command line: root=/dev/sda2 ro console=tty0 console=ttyS0,9600n8 
max_loo
p=256 elevator=cfq 

Enabling fast FPU save and restore... done. 

Enabling unmasked SIMD FPU exception support... done. 

Initializing CPU#0 

PID hash table entries: 2048 (order: 11, 32768 bytes) 

Xen reported: 1793.773 MHz processor. 

Console: colour VGA+ 80x25 

Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) 

Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) 

Software IO TLB enabled: 

  Aperture:     64 megabytes 

  Kernel range: 0x00000000c12bf000 - 0x00000000c52bf000 

vmalloc area: d1000000-f53fe000, maxmem 2d7fe000 

Memory: 187392k/270336k available (3314k kernel code, 74472k reserved, 
961k data
, 208k init, 0k highmem) 

Checking if this processor honours the WP bit even in supervisor mode... 
Ok.
Calibrating delay using timer specific routine.. 4486.13 BogoMIPS 
(lpj=8972260)
Mount-cache hash table entries: 512 

CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) 

CPU: L2 Cache: 1024K (64 bytes/line) 

Checking 'hlt' instruction... OK. 

ENABLING IO-APIC IRQs 

Brought up 2 CPUs 

Initializing CPU#1 

migration_cost=528 

Grant table initialized 

NET: Registered protocol family 16 

ACPI: bus type pci registered 

PCI: Using configuration type 1 

ACPI: Subsystem revision 20060127 

ACPI: Interpreter enabled 

ACPI: Using IOAPIC for interrupt routing 

ACPI: PCI Root Bridge [PCI0] (0000:00) 

ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 *5 6 7 9 10 11 12 14 15) 

ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *9 10 11 12 14 15) 

ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) 

ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 14 15) 

xen_mem: Initialising balloon driver. 

SCSI subsystem initialized 

usbcore: registered new driver usbfs 

usbcore: registered new driver hub 

PCI: Using ACPI for IRQ routing 

PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a 
report
TC classifier action (bugs to netdev@vger.kernel.org cc hadi@cyberus.ca) 

PCI: Bridge: 0000:00:06.0 

   IO window: b000-bfff 

   MEM window: fca00000-feafffff 

   PREFETCH window: disabled. 

PCI: Bridge: 0000:00:0a.0 

   IO window: a000-afff 

   MEM window: fb900000-fc9fffff 

   PREFETCH window: ff500000-ff5fffff 

PCI: Bridge: 0000:00:0b.0 

   IO window: disabled. 

   MEM window: disabled. 

   PREFETCH window: disabled. 

IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com> 

audit: initializing netlink socket (disabled) 

audit(1158849681.560:1): initialized 

Unable to handle kernel NULL pointer dereference at virtual address 
00000000
  printing eip: 

00000000 

005ff000 -> *pde = 00000000:04e00001 

00600000 -> *pme = 00000000:00000000 

Oops: 0010 [#1] 

SMP 

Modules linked in: 

CPU:    0 

EIP:    0061:[<00000000>]    Not tainted VLI 

EFLAGS: 00010246   (2.6.16.29-xen #4) 

EIP is at rest_init+0x3fefe000/0x23 

eax: c010206f   ebx: c0558f3c   ecx: 00000000   edx: c5320000 

esi: c01021c9   edi: 00000000   ebp: 0000ffbd   esp: c53220e7 

ds: 007b   es: 007b   ss: e021 

Process swapper (pid: 1, threadinfo=c5320000 task=c05bea90) 

Stack: <0>00000000 00000000 00000000 00000000 00000000 00000000 00000000 
0000000
0 

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000
Call Trace: 

  ======================= 

Unable to handle kernel NULL pointer dereference at virtual address 
00000000
  printing eip: 

c0111007 

005ff000 -> *pde = 00000000:04e00001 

00600000 -> *pme = 00000000:00000000 

Re: Kernel error with todays -unstable.

[Keir Fraser]

On 21/9/06 19:43, "Matt Ayres" <matta@tektonic.net> wrote:

> Got this changing from  2.6.16.13 to 2.6.16.29.. running .13 under the
> same xen binary works fine.

Is the Xen binary built from a recent repository (i.e., which includes
2.6.16.29), or an older Xen version? It shouldn't matter, but if you run
2.6.16.29 okay with latest Xen but not with older Xen then it probably
indicates a compatibility issue that we should look into.

 -- Keir

Re: Kernel error with todays -unstable.

[Keir Fraser]

On 21/9/06 19:52, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

>> Got this changing from  2.6.16.13 to 2.6.16.29.. running .13 under the
>> same xen binary works fine.
> 
> Is the Xen binary built from a recent repository (i.e., which includes
> 2.6.16.29), or an older Xen version? It shouldn't matter, but if you run
> 2.6.16.29 okay with latest Xen but not with older Xen then it probably
> indicates a compatibility issue that we should look into.

Ok, actually the logs you posted are sufficient to see that the Xen and
Linux are a 'matched pair'. So... Did you modify the Linux config at all?

 -- Keir

Re: Kernel error with todays -unstable.

[Matt Ayres]

Keir Fraser wrote:
> 
> 
> On 21/9/06 19:52, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>>> Got this changing from  2.6.16.13 to 2.6.16.29.. running .13 under the
>>> same xen binary works fine.
>> Is the Xen binary built from a recent repository (i.e., which includes
>> 2.6.16.29), or an older Xen version? It shouldn't matter, but if you run
>> 2.6.16.29 okay with latest Xen but not with older Xen then it probably
>> indicates a compatibility issue that we should look into.
> 
> Ok, actually the logs you posted are sufficient to see that the Xen and
> Linux are a 'matched pair'. So... Did you modify the Linux config at all?
> 

Yes, it is heavily modified. I can provide the config in you are 
interested (private or not). I'm not even really sure where rest_idle() 
would be, it sounds fairly generic.

Thanks,
Matt

Re: Kernel error with todays -unstable.

[Keir Fraser]

On 21/9/06 8:01 pm, "Matt Ayres" <matta@tektonic.net> wrote:

>> Ok, actually the logs you posted are sufficient to see that the Xen and
>> Linux are a 'matched pair'. So... Did you modify the Linux config at all?
>> 
> 
> Yes, it is heavily modified. I can provide the config in you are
> interested (private or not). I'm not even really sure where rest_idle()
> would be, it sounds fairly generic.

The rest_idle label is misleading. For some reason you have jumped to
address 0. Rest_idle just happens to be the last symbol in the symbol table,
so Linux has printed that plus a large offset.

Send me your config privately and I'll give it a quick spin.

 -- Keir

Re: Kernel error with todays -unstable.

[Matt Ayres]

Keir,

Have you any update on this bug I would label as _CRITICAL_ seeing as 
the same hardware, config file, and 2.6.16.13 worked just fine.  Even 
the newer hypervisor and 2.6.16.13 from previous -unstable tree boots fine.

Thanks,
Matt

Keir Fraser wrote:
> 
> 
> On 21/9/06 19:52, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>>> Got this changing from  2.6.16.13 to 2.6.16.29.. running .13 under the
>>> same xen binary works fine.
>> Is the Xen binary built from a recent repository (i.e., which includes
>> 2.6.16.29), or an older Xen version? It shouldn't matter, but if you run
>> 2.6.16.29 okay with latest Xen but not with older Xen then it probably
>> indicates a compatibility issue that we should look into.
> 
> Ok, actually the logs you posted are sufficient to see that the Xen and
> Linux are a 'matched pair'. So... Did you modify the Linux config at all?
> 
>  -- Keir
> 
> 

Re: Kernel error with todays -unstable.

[Keir Fraser]

Narrow it down to a particular config option and we'll have a chance.

 -- Keir

On 22/9/06 3:36 pm, "Matt Ayres" <matta@tektonic.net> wrote:

> Keir,
> 
> Have you any update on this bug I would label as _CRITICAL_ seeing as
> the same hardware, config file, and 2.6.16.13 worked just fine.  Even
> the newer hypervisor and 2.6.16.13 from previous -unstable tree boots fine.
> 
> Thanks,
> Matt
> 
> Keir Fraser wrote:
>> 
>> 
>> On 21/9/06 19:52, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
>> 
>>>> Got this changing from  2.6.16.13 to 2.6.16.29.. running .13 under the
>>>> same xen binary works fine.
>>> Is the Xen binary built from a recent repository (i.e., which includes
>>> 2.6.16.29), or an older Xen version? It shouldn't matter, but if you run
>>> 2.6.16.29 okay with latest Xen but not with older Xen then it probably
>>> indicates a compatibility issue that we should look into.
>> 
>> Ok, actually the logs you posted are sufficient to see that the Xen and
>> Linux are a 'matched pair'. So... Did you modify the Linux config at all?
>> 
>>  -- Keir
>> 
>> 

RE: Kernel error with todays -unstable.

[Ian Pratt]

 
> Have you any update on this bug I would label as _CRITICAL_ 
> seeing as the same hardware, config file, and 2.6.16.13 
> worked just fine.  Even the newer hypervisor and 2.6.16.13 
> from previous -unstable tree boots fine.

What cset was the most recent version of 2.6.16.13 that boted fine?

Ian