Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Martin J. Bligh]

Not sure if you've seen this already ... catching up on test results.

This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
problem.

Full logs:

mm2 - http://test.kernel.org/abat/50727/debug/console.log
mm3 - http://test.kernel.org/abat/51442/debug/console.log

config - http://test.kernel.org/abat/51442/build/dotconfig

I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
is failing because bus->sysdata is NULL. The disassembly and
structure offsets seem to line up for that.

#define pcibus_to_node(bus) (
	(struct pci_sysdata *)((bus)->sysdata))->node

struct pci_sysdata {
         int             domain;         /* PCI domain */
         int             node;           /* NUMA node */
};


BUG: unable to handle kernel NULL pointer dereference at virtual address 
00000004
  printing eip:
c02060d4
*pde = 0042c001
*pte = 00000000
Oops: 0000 [#1]
SMP
last sysfs file:
Modules linked in:
CPU:    2
EIP:    0060:[<c02060d4>]    Not tainted VLI
EFLAGS: 00010286   (2.6.18-mm2-autokern1 #1)
EIP is at pci_call_probe+0x19/0xb5
eax: 00000000   ebx: e778b400   ecx: e7400030   edx: c03b89a0
esi: e778b400   edi: 0000ffff   ebp: e69a2fa0   esp: e740dea4
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 1, ti=e740c000 task=e7400030 task.ti=e740c000)
Stack: ffffffed e778b400 c03b89a0 c02061a3 c03b89a0 e778b400 c03b873c 
c03b89a0
        e778b400 c03b89d4 c02061d6 c03b89a0 e778b400 e778b400 c03b89d4 
e778b448
        c0224800 e778b448 c03b89d4 e778b448 00000000 c03b89d4 c0224936 
e69a2fa0
Call Trace:
  [<c02061a3>] __pci_device_probe+0x33/0x47
  [<c02061d6>] pci_device_probe+0x1f/0x34
  [<c0224800>] really_probe+0x31/0xb9
  [<c0224936>] driver_probe_device+0x93/0x9c
  [<c02249b5>] __driver_attach+0x0/0x7c
  [<c02249fc>] __driver_attach+0x47/0x7c
  [<c0223e98>] bus_for_each_dev+0x47/0x6d
  [<c01fd05a>] kobject_add+0xa9/0xf2
  [<c0224a45>] driver_attach+0x14/0x18
  [<c02249b5>] __driver_attach+0x0/0x7c
  [<c022437b>] bus_add_driver+0x53/0xd0
  [<c0224d99>] driver_register+0x74/0x77
  [<c02063ea>] __pci_register_driver+0x6b/0x7a
  [<c04146c3>] qla1280_init+0xc/0xf
  [<c04007ff>] do_initcalls+0x55/0xe8
  [<c0184095>] proc_mkdir+0x12/0x16
  [<c0135136>] init_irq_proc+0x21/0x2f
  [<c01003b8>] init+0x0/0x148
  [<c010040d>] init+0x55/0x148
  [<c01033c7>] kernel_thread_helper+0x7/0x10

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Andrew Morton]

On Sun, 08 Oct 2006 00:02:07 -0700
"Martin J. Bligh" <mbligh@mbligh.org> wrote:

> Not sure if you've seen this already ... catching up on test results.

Nope.

> This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
> problem.
> 
> Full logs:
> 
> mm2 - http://test.kernel.org/abat/50727/debug/console.log
> mm3 - http://test.kernel.org/abat/51442/debug/console.log
> 
> config - http://test.kernel.org/abat/51442/build/dotconfig
> 
> I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
> is failing because bus->sysdata is NULL. The disassembly and
> structure offsets seem to line up for that.
> 
> #define pcibus_to_node(bus) (
> 	(struct pci_sysdata *)((bus)->sysdata))->node
> 
> struct pci_sysdata {
>          int             domain;         /* PCI domain */
>          int             node;           /* NUMA node */
> };
> 
> 
> BUG: unable to handle kernel NULL pointer dereference at virtual address 
> 00000004

I don't see anything in the mm1->mm2 additions which might have caused
this.  Don't know, sorry.  Bisection time?


>   printing eip:
> c02060d4
> *pde = 0042c001
> *pte = 00000000
> Oops: 0000 [#1]
> SMP
> last sysfs file:
> Modules linked in:
> CPU:    2
> EIP:    0060:[<c02060d4>]    Not tainted VLI
> EFLAGS: 00010286   (2.6.18-mm2-autokern1 #1)
> EIP is at pci_call_probe+0x19/0xb5
> eax: 00000000   ebx: e778b400   ecx: e7400030   edx: c03b89a0
> esi: e778b400   edi: 0000ffff   ebp: e69a2fa0   esp: e740dea4
> ds: 007b   es: 007b   ss: 0068
> Process swapper (pid: 1, ti=e740c000 task=e7400030 task.ti=e740c000)
> Stack: ffffffed e778b400 c03b89a0 c02061a3 c03b89a0 e778b400 c03b873c 
> c03b89a0
>         e778b400 c03b89d4 c02061d6 c03b89a0 e778b400 e778b400 c03b89d4 
> e778b448
>         c0224800 e778b448 c03b89d4 e778b448 00000000 c03b89d4 c0224936 
> e69a2fa0
> Call Trace:
>   [<c02061a3>] __pci_device_probe+0x33/0x47
>   [<c02061d6>] pci_device_probe+0x1f/0x34
>   [<c0224800>] really_probe+0x31/0xb9
>   [<c0224936>] driver_probe_device+0x93/0x9c
>   [<c02249b5>] __driver_attach+0x0/0x7c
>   [<c02249fc>] __driver_attach+0x47/0x7c
>   [<c0223e98>] bus_for_each_dev+0x47/0x6d
>   [<c01fd05a>] kobject_add+0xa9/0xf2
>   [<c0224a45>] driver_attach+0x14/0x18
>   [<c02249b5>] __driver_attach+0x0/0x7c
>   [<c022437b>] bus_add_driver+0x53/0xd0
>   [<c0224d99>] driver_register+0x74/0x77
>   [<c02063ea>] __pci_register_driver+0x6b/0x7a
>   [<c04146c3>] qla1280_init+0xc/0xf
>   [<c04007ff>] do_initcalls+0x55/0xe8
>   [<c0184095>] proc_mkdir+0x12/0x16
>   [<c0135136>] init_irq_proc+0x21/0x2f
>   [<c01003b8>] init+0x0/0x148
>   [<c010040d>] init+0x55/0x148
>   [<c01033c7>] kernel_thread_helper+0x7/0x10

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Badari Pulavarty]

On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote:
> Not sure if you've seen this already ... catching up on test results.
> 
> This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
> problem.
> 
> Full logs:
> 
> mm2 - http://test.kernel.org/abat/50727/debug/console.log
> mm3 - http://test.kernel.org/abat/51442/debug/console.log
> 
> config - http://test.kernel.org/abat/51442/build/dotconfig
> 
> I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
> is failing because bus->sysdata is NULL. The disassembly and
> structure offsets seem to line up for that.
> 
> #define pcibus_to_node(bus) (
> 	(struct pci_sysdata *)((bus)->sysdata))->node
> 
> struct pci_sysdata {
>          int             domain;         /* PCI domain */
>          int             node;           /* NUMA node */
> };
> 

Martin,

Jeff moved "node" to a proper field in sysdata, instead
of overloading sysdata itself. I think this is causing the
problem. I guess we could end up with sysdata = NULL in some
cases ? Since you are the NUMA-Q expert, where does sysdata 
gets set for NUMA-Q ? :)

-mm2 changed:

#define pcibus_to_node(bus) ((long) (bus)->sysdata)

to 

#define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
>node


Thanks,
Badari


> BUG: unable to handle kernel NULL pointer dereference at virtual address 
> 00000004
>   printing eip:
> c02060d4
> *pde = 0042c001
> *pte = 00000000
> Oops: 0000 [#1]
> SMP
> last sysfs file:
> Modules linked in:
> CPU:    2
> EIP:    0060:[<c02060d4>]    Not tainted VLI
> EFLAGS: 00010286   (2.6.18-mm2-autokern1 #1)
> EIP is at pci_call_probe+0x19/0xb5
> eax: 00000000   ebx: e778b400   ecx: e7400030   edx: c03b89a0
> esi: e778b400   edi: 0000ffff   ebp: e69a2fa0   esp: e740dea4
> ds: 007b   es: 007b   ss: 0068
> Process swapper (pid: 1, ti=e740c000 task=e7400030 task.ti=e740c000)
> Stack: ffffffed e778b400 c03b89a0 c02061a3 c03b89a0 e778b400 c03b873c 
> c03b89a0
>         e778b400 c03b89d4 c02061d6 c03b89a0 e778b400 e778b400 c03b89d4 
> e778b448
>         c0224800 e778b448 c03b89d4 e778b448 00000000 c03b89d4 c0224936 
> e69a2fa0
> Call Trace:
>   [<c02061a3>] __pci_device_probe+0x33/0x47
>   [<c02061d6>] pci_device_probe+0x1f/0x34
>   [<c0224800>] really_probe+0x31/0xb9
>   [<c0224936>] driver_probe_device+0x93/0x9c
>   [<c02249b5>] __driver_attach+0x0/0x7c
>   [<c02249fc>] __driver_attach+0x47/0x7c
>   [<c0223e98>] bus_for_each_dev+0x47/0x6d
>   [<c01fd05a>] kobject_add+0xa9/0xf2
>   [<c0224a45>] driver_attach+0x14/0x18
>   [<c02249b5>] __driver_attach+0x0/0x7c
>   [<c022437b>] bus_add_driver+0x53/0xd0
>   [<c0224d99>] driver_register+0x74/0x77
>   [<c02063ea>] __pci_register_driver+0x6b/0x7a
>   [<c04146c3>] qla1280_init+0xc/0xf
>   [<c04007ff>] do_initcalls+0x55/0xe8
>   [<c0184095>] proc_mkdir+0x12/0x16
>   [<c0135136>] init_irq_proc+0x21/0x2f
>   [<c01003b8>] init+0x0/0x148
>   [<c010040d>] init+0x55/0x148
>   [<c01033c7>] kernel_thread_helper+0x7/0x10
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Martin Bligh]

Badari Pulavarty wrote:
> On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote:
> 
>>Not sure if you've seen this already ... catching up on test results.
>>
>>This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
>>problem.
>>
>>Full logs:
>>
>>mm2 - http://test.kernel.org/abat/50727/debug/console.log
>>mm3 - http://test.kernel.org/abat/51442/debug/console.log
>>
>>config - http://test.kernel.org/abat/51442/build/dotconfig
>>
>>I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
>>is failing because bus->sysdata is NULL. The disassembly and
>>structure offsets seem to line up for that.
>>
>>#define pcibus_to_node(bus) (
>>	(struct pci_sysdata *)((bus)->sysdata))->node
>>
>>struct pci_sysdata {
>>         int             domain;         /* PCI domain */
>>         int             node;           /* NUMA node */
>>};
>>
> 
> 
> Martin,
> 
> Jeff moved "node" to a proper field in sysdata, instead
> of overloading sysdata itself. I think this is causing the
> problem. I guess we could end up with sysdata = NULL in some
> cases ? Since you are the NUMA-Q expert, where does sysdata 
> gets set for NUMA-Q ? :)
> 
> -mm2 changed:
> 
> #define pcibus_to_node(bus) ((long) (bus)->sysdata)
> 
> to 
> 
> #define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
> 
>>node

Buggered if I know, that's some strange pci thing ;-)

But can we revert whatever patch that was until it gets fixed, please?

Thanks,

M.

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Jeff Garzik]

Martin Bligh wrote:
> Badari Pulavarty wrote:
>> On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote:
>>
>>> Not sure if you've seen this already ... catching up on test results.
>>>
>>> This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
>>> problem.
>>>
>>> Full logs:
>>>
>>> mm2 - http://test.kernel.org/abat/50727/debug/console.log
>>> mm3 - http://test.kernel.org/abat/51442/debug/console.log
>>>
>>> config - http://test.kernel.org/abat/51442/build/dotconfig
>>>
>>> I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
>>> is failing because bus->sysdata is NULL. The disassembly and
>>> structure offsets seem to line up for that.
>>>
>>> #define pcibus_to_node(bus) (
>>>     (struct pci_sysdata *)((bus)->sysdata))->node
>>>
>>> struct pci_sysdata {
>>>         int             domain;         /* PCI domain */
>>>         int             node;           /* NUMA node */
>>> };
>>>
>>
>>
>> Martin,
>>
>> Jeff moved "node" to a proper field in sysdata, instead
>> of overloading sysdata itself. I think this is causing the
>> problem. I guess we could end up with sysdata = NULL in some
>> cases ? Since you are the NUMA-Q expert, where does sysdata gets set 
>> for NUMA-Q ? :)
>>
>> -mm2 changed:
>>
>> #define pcibus_to_node(bus) ((long) (bus)->sysdata)
>>
>> to
>> #define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
>>
>>> node
> 
> Buggered if I know, that's some strange pci thing ;-)
> 
> But can we revert whatever patch that was until it gets fixed, please?

It needs to get fixed, otherwise whose buses of PCI devices disappear on 
some machines.

Can you turn on PCI debugging?

	Jeff

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Sukadev Bhattiprolu]

[-- Attachment #1: Type: text/plain, Size: 5140 bytes --]

Jeff Garzik [jgarzik@pobox.com] wrote:
| Martin Bligh wrote:
| >Badari Pulavarty wrote:
| >>On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote:
| >>
| >>>Not sure if you've seen this already ... catching up on test results.
| >>>
| >>>This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
| >>>problem.
| >>>
| >>>Full logs:
| >>>
| >>>mm2 - http://test.kernel.org/abat/50727/debug/console.log
| >>>mm3 - http://test.kernel.org/abat/51442/debug/console.log
| >>>
| >>>config - http://test.kernel.org/abat/51442/build/dotconfig
| >>>
| >>>I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
| >>>is failing because bus->sysdata is NULL. The disassembly and
| >>>structure offsets seem to line up for that.
| >>>
| >>>#define pcibus_to_node(bus) (
| >>>    (struct pci_sysdata *)((bus)->sysdata))->node
| >>>
| >>>struct pci_sysdata {
| >>>        int             domain;         /* PCI domain */
| >>>        int             node;           /* NUMA node */
| >>>};
| >>>
| >>
| >>
| >>Martin,
| >>
| >>Jeff moved "node" to a proper field in sysdata, instead
| >>of overloading sysdata itself. I think this is causing the
| >>problem. I guess we could end up with sysdata = NULL in some
| >>cases ? Since you are the NUMA-Q expert, where does sysdata gets set 
| >>for NUMA-Q ? :)
| >>
| >>-mm2 changed:
| >>
| >>#define pcibus_to_node(bus) ((long) (bus)->sysdata)
| >>
| >>to
| >>#define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
| >>
| >>>node
| >
| >Buggered if I know, that's some strange pci thing ;-)
| >
| >But can we revert whatever patch that was until it gets fixed, please?
| 
| It needs to get fixed, otherwise whose buses of PCI devices disappear on 
| some machines.
| 
| Can you turn on PCI debugging?
| 
| 	Jeff

I turned on CONFIG_PCI_DEBUG. already had CONFIG_KERNEL_DEBUG and
CONFIG_DEBUG_INFO. Booted with "debug" parameter.

Here are the last few messages. Complete dmesg attached.

Let me know if there are other config tokens or boot options that may
provide more info.

Suka
---

PCI: Calling quirk c024da00 for 0000:00:0a.0
PCI: Calling quirk c031ce30 for 0000:00:0a.0
PCI: Calling quirk c024da00 for 0000:00:0b.0
PCI: Calling quirk c031ce30 for 0000:00:0b.0
PCI: Calling quirk c024da00 for 0000:00:0e.0
PCI: Calling quirk c031ce30 for 0000:00:0e.0
PCI: Calling quirk c024da00 for 0000:00:0e.1
PCI: Calling quirk c031ce30 for 0000:00:0e.1
PCI: Calling quirk c024da00 for 0000:00:0e.2
PCI: Calling quirk c031ce30 for 0000:00:0e.2
PCI: Calling quirk c024da00 for 0000:00:0e.3
PCI: Calling quirk c031ce30 for 0000:00:0e.3
PCI: Calling quirk c024da00 for 0000:00:10.0
PCI: Calling quirk c031ce30 for 0000:00:10.0
PCI: Calling quirk c024da00 for 0000:00:12.0
PCI: Calling quirk c05093b0 for 0000:00:12.0
PCI: Calling quirk c031ce30 for 0000:00:12.0
PCI: Calling quirk c024da00 for 0000:00:14.0
PCI: Calling quirk c05093b0 for 0000:00:14.0
PCI: Calling quirk c031ce30 for 0000:00:14.0
PCI: Calling quirk c024da00 for 0000:01:0c.0
PCI: Calling quirk c031ce30 for 0000:01:0c.0
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 7.2.9-k2
Copyright (c) 1999-2006 Intel Corporation.
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c024e92c
*pde = 00529001
*pte = 00000000
Oops: 0000 [#1]
SMP

last sysfs file:
Modules linked in:
CPU:    0
EIP:    0060:[<c024e92c>]    Not tainted VLI
EFLAGS: 00010286   (2.6.18-mm3 #3)
EIP is at pci_call_probe+0x1c/0xe0
eax: 00000000   ebx: dfe63c00   ecx: c10fca90   edx: c048ad60
esi: dfe63c00   edi: 0000000f   ebp: dfc3fd00   esp: c10ffe70
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 1, ti=c10fe000 task=c10fca90 task.ti=c10fe000)
Stack: dfe63c00 ffffffed ffffffed dfe63c00 c048b1c0 c024ea55 c048b1c0 dfe63c00
       c048ad60 c048b1c0 dfe63c00 c048b1f4 c024ea9f c048b1c0 dfe63c00 dfe63c48
       dfc3fd00 c02761f9 dfe63c48 00000286 dfff9060 000000d0 c048b1f4 dfc3fd00
Call Trace:
 [<c024ea55>] __pci_device_probe+0x65/0x80
 [<c024ea9f>] pci_device_probe+0x2f/0x50
 [<c02761f9>] really_probe+0xf9/0x100
 [<c02762e8>] driver_probe_device+0xc8/0xe0
 [<c03ae92d>] klist_next+0x5d/0xa0
 [<c02763a0>] __driver_attach+0x0/0xa0
 [<c0276430>] __driver_attach+0x90/0xa0
 [<c0275409>] bus_for_each_dev+0x69/0x80
 [<c0276465>] driver_attach+0x25/0x30
 [<c02763a0>] __driver_attach+0x0/0xa0
 [<c0275ad3>] bus_add_driver+0x73/0x140
 [<c024edc4>] __pci_register_driver+0x74/0x90
 [<c050c6f9>] tulip_init+0x29/0x30
 [<c04f2a62>] do_initcalls+0x42/0x140
 [<c01433eb>] register_irq_proc+0xab/0xd0
 [<c01003f0>] init+0x0/0x1a0
 [<c0143479>] init_irq_proc+0x39/0x50
 [<c01003f0>] init+0x0/0x1a0
 [<c0100451>] init+0x61/0x1a0
 [<c0103c0b>] kernel_thread_helper+0x7/0x1c
 =======================
Code: 74 92 eb 8e 8d 74 26 00 8d bc 27 00 00 00 00 57 56 53 83 ec 08 8b 5c 24 1c 89 e0 25 00 e0 ff ff 8b 08 8b 43 10 8b 79 5c 8b 40 44 <8b> 50 04 85 d2 78 11 0f a3 15 c0 9f 4e c0 19 c0 85 c0 0f 85 8c

[-- Attachment #2: dmesg-3.txt --]
[-- Type: text/plain, Size: 13799 bytes --]


BIOS-provided physical RAM map:
sanitize start
sanitize end
copy_e820_map() start: 0000000000000000 size: 000000000009fc00 end: 000000000009fc00 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 0000000000100000 size: 000000003ff00000 end: 0000000040000000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 00000000fec00000 size: 0000000000009000 end: 00000000fec09000 type: 2
copy_e820_map() start: 00000000ffe80000 size: 0000000000180000 end: 0000000100000000 type: 2
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 0000000000100000 - 0000000040000000 (usable)
 BIOS-e820: 00000000fec00000 - 00000000fec09000 (reserved)
 BIOS-e820: 00000000ffe80000 - 0000000100000000 (reserved)
Node: 0, start_pfn: 0, end_pfn: 159
  Setting physnode_map array to node 0 for pfns:
  0
Node: 0, start_pfn: 256, end_pfn: 262144
  Setting physnode_map array to node 0 for pfns:
  256 65792 131328 196864
Node: 0, start_pfn: 0, end_pfn: 262144
  Setting physnode_map array to node 0 for pfns:
  0 65536 131072 196608
Reserving 2560 pages of KVA for lmem_map of node 0
Shrinking node 0 from 262144 pages to 259584 pages
Reserving total of 2560 pages for numa KVA remap
kva_start_pfn ~ 226816 find_max_low_pfn() ~ 229376
max_pfn = 262144
128MB HIGHMEM available.
896MB LOWMEM available.
min_low_pfn = 1385, max_low_pfn = 229376, highstart_pfn = 229376
Low memory ends at vaddr f8000000
node 0 will remap to vaddr f7600000 - f8a00000
High memory starts at vaddr f8000000
found SMP MP-table at 000f6040
Entering add_active_range(0, 0, 259584) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   262144
early_node_map[1] active PFN ranges
    0:        0 ->   259584
On node 0 totalpages: 259584
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 236 pages used for memmap
  HighMem zone: 29972 pages, LIFO batch:7
DMI not present or invalid.
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: IBM NUMA Product ID: SBB          <6>Found an OEM MPC table at 1fff82a8- parsing it ...
Translation: record 0, type 1, quad 0, global 3, local 3
Translation: record 1, type 1, quad 0, global 1, local 1
Translation: record 2, type 1, quad 0, global 1, local 1
Translation: record 3, type 1, quad 0, global 1, local 1
Translation: record 4, type 3, quad 0, global 0, local 0
Translation: record 5, type 3, quad 0, global 1, local 1
Translation: record 6, type 4, quad 0, global 2, local 18
Translation: record 7, type 2, quad 0, global 13, local 14
Translation: record 8, type 2, quad 0, global 14, local 13
APIC at: 0xFEC08000
Processor #0 6:10 APIC version 17 (quad 0, apic 1)
Processor #4 6:10 APIC version 17 (quad 0, apic 8)
Processor #1 6:10 APIC version 17 (quad 0, apic 2)
Processor #2 6:10 APIC version 17 (quad 0, apic 4)
Bus #0 is PCI    (node 0)
Bus #1 is PCI    (node 0)
Bus #2 is EISA   (node 0)
I/O APIC #13 Version 17 at 0xFEC00000.
I/O APIC #14 Version 17 at 0xFEC01000.
Enabling APIC mode:  NUMA-Q.  Using 2 I/O APICs
Processors: 4
Allocating PCI resources starting at 50000000 (gap: 40000000:bec00000)
Detected 700.054 MHz processor.
Built 1 zonelists.  Total pages: 257556
Kernel command line: root=/dev/sda1 console=ttyS0,57600 reboot=bios debug
mapped APIC to ffffd000 (fec08000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec01000)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Initializing HighMem for node 0 (00038000:0003f600)
Memory: 1022216k/1048576k available (2757k kernel code, 15732k reserved, 1252k data, 216k init, 120832k highmem)
virtual kernel memory layout:
    fixmap  : 0xffe1c000 - 0xfffff000   (1932 kB)
    pkmap   : 0xffc00000 - 0xffe00000   (2048 kB)
    vmalloc : 0xf8800000 - 0xffbfe000   ( 115 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc04f2000 - 0xc0528000   ( 216 kB)
      .data : 0xc03b16ac - 0xc04eaa4c   (1252 kB)
      .text : 0xc0100000 - 0xc03b16ac   (2757 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 1401.31 BogoMIPS (lpj=2802636)
Mount-cache hash table entries: 512
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
Compat vDSO mapped to ffffe000.
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 16k freed
CPU0: Intel Pentium III (Cascades) stepping 00
Leaving ESR disabled.
Mapping cpu 0 to node 0
Booting processor 1/2 eip 2000
Storing NMI vector
Initializing CPU#1
Leaving ESR disabled.
Mapping cpu 1 to node 0
Calibrating delay using timer specific routine.. 1400.08 BogoMIPS (lpj=2800175)
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
CPU1: Intel Pentium III (Cascades) stepping 00
Booting processor 2/4 eip 2000
Storing NMI vector
Initializing CPU#2
Leaving ESR disabled.
Mapping cpu 2 to node 0
Calibrating delay using timer specific routine.. 1400.06 BogoMIPS (lpj=2800120)
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
CPU2: Intel Pentium III (Cascades) stepping 00
Booting processor 3/8 eip 2000
Storing NMI vector
Initializing CPU#3
Leaving ESR disabled.
Mapping cpu 3 to node 0
Calibrating delay using timer specific routine.. 1400.06 BogoMIPS (lpj=2800124)
CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 0000000000000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU serial number disabled.
CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
CPU3: Intel Pentium III (Cascades) stepping 00
Total of 4 processors activated (5601.52 BogoMIPS).
ExtINT not setup in hardware but reported by MP table
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=0 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ...
..... (found pin 0) ... failed.
...trying to set up timer as Virtual Wire IRQ... works.
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=12000
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfd231, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
SCSI subsystem initialized
PCI: Probing PCI hardware (bus 00)
PCI: Scanning bus 0000:00
PCI: Found 0000:00:0a.0 [1077/1020] 000100 00
PCI: Calling quirk c024db60 for 0000:00:0a.0
PCI: Calling quirk c024e240 for 0000:00:0a.0
PCI: Found 0000:00:0b.0 [1002/4750] 000300 00
PCI: Calling quirk c024db60 for 0000:00:0b.0
PCI: Calling quirk c024e240 for 0000:00:0b.0
Boot video device is 0000:00:0b.0
PCI: Found 0000:00:0e.0 [8086/7110] 000601 00
PCI: Calling quirk c024e050 for 0000:00:0e.0
PCI: Calling quirk c024db60 for 0000:00:0e.0
PCI: Calling quirk c024e240 for 0000:00:0e.0
PCI: Found 0000:00:0e.1 [8086/7111] 000101 00
PCI: Calling quirk c024e050 for 0000:00:0e.1
PCI: Calling quirk c024db60 for 0000:00:0e.1
PCI: Calling quirk c024e240 for 0000:00:0e.1
PCI: Found 0000:00:0e.2 [8086/7112] 000c03 00
PCI: Calling quirk c024e050 for 0000:00:0e.2
PCI: Calling quirk c024db60 for 0000:00:0e.2
PCI: Calling quirk c024e240 for 0000:00:0e.2
PCI: Found 0000:00:0e.3 [8086/7113] 000680 00
PCI: Calling quirk c024e050 for 0000:00:0e.3
PCI: Calling quirk c024d150 for 0000:00:0e.3
PCI quirk: region 0c00-0c3f claimed by PIIX4 ACPI
PIIX4 devres C PIO at 007c-007c
PCI: Calling quirk c024db60 for 0000:00:0e.3
PCI: Calling quirk c024e240 for 0000:00:0e.3
PCI: Found 0000:00:10.0 [8086/84ca] 000600 00
PCI: Calling quirk c024e050 for 0000:00:10.0
PCI: Calling quirk c024db60 for 0000:00:10.0
PCI: Calling quirk c024e240 for 0000:00:10.0
PCI: Calling quirk c032f670 for 0000:00:10.0
PCI: Searching for i450NX host bridges on 0000:00:10.0
PCI: Scanning bus 0000:01
PCI: Found 0000:01:0c.0 [1011/0009] 000200 00
PCI: Calling quirk c024db60 for 0000:01:0c.0
PCI: Calling quirk c024e240 for 0000:01:0c.0
PCI: Fixups for bus 0000:01
PCI: Bus scan for 0000:01 returning with max=01
PCI: Found 0000:00:12.0 [8086/84cb] 000600 00
PCI: Calling quirk c024e050 for 0000:00:12.0
PCI: Calling quirk c024db60 for 0000:00:12.0
PCI: Calling quirk c024e240 for 0000:00:12.0
PCI: Found 0000:00:14.0 [8086/84cb] 000600 00
PCI: Calling quirk c024e050 for 0000:00:14.0
PCI: Calling quirk c024db60 for 0000:00:14.0
PCI: Calling quirk c024e240 for 0000:00:14.0
PCI: Fixups for bus 0000:00
PCI: Bus scan for 0000:00 returning with max=00
PCI->APIC IRQ transform: 0000:01:0c.0[A] -> IRQ 40
PCI->APIC IRQ transform: 0000:00:0a.0[A] -> IRQ 23
PCI->APIC IRQ transform: 0000:00:0b.0[A] -> IRQ 19
  got res [50000000:5001ffff] bus [50000000:5001ffff] flags 7202 for BAR 6 of 0000:00:0b.0
  got res [50020000:5002ffff] bus [50020000:5002ffff] flags 7200 for BAR 6 of 0000:00:0a.0
  got res [1000:101f] bus [1000:101f] flags 101 for BAR 4 of 0000:00:0e.2
PCI: moved device 0000:00:0e.2 resource 4 (101) to 1000
  got res [50040000:5007ffff] bus [50040000:5007ffff] flags 7200 for BAR 6 of 0000:01:0c.0
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
highmem bounce pool size: 64 pages
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
JFS: nTxBlock = 7986, nTxLock = 63889
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
PCI: Calling quirk c024da00 for 0000:00:0a.0
PCI: Calling quirk c031ce30 for 0000:00:0a.0
PCI: Calling quirk c024da00 for 0000:00:0b.0
PCI: Calling quirk c031ce30 for 0000:00:0b.0
PCI: Calling quirk c024da00 for 0000:00:0e.0
PCI: Calling quirk c031ce30 for 0000:00:0e.0
PCI: Calling quirk c024da00 for 0000:00:0e.1
PCI: Calling quirk c031ce30 for 0000:00:0e.1
PCI: Calling quirk c024da00 for 0000:00:0e.2
PCI: Calling quirk c031ce30 for 0000:00:0e.2
PCI: Calling quirk c024da00 for 0000:00:0e.3
PCI: Calling quirk c031ce30 for 0000:00:0e.3
PCI: Calling quirk c024da00 for 0000:00:10.0
PCI: Calling quirk c031ce30 for 0000:00:10.0
PCI: Calling quirk c024da00 for 0000:00:12.0
PCI: Calling quirk c05093b0 for 0000:00:12.0
PCI: Calling quirk c031ce30 for 0000:00:12.0
PCI: Calling quirk c024da00 for 0000:00:14.0
PCI: Calling quirk c05093b0 for 0000:00:14.0
PCI: Calling quirk c031ce30 for 0000:00:14.0
PCI: Calling quirk c024da00 for 0000:01:0c.0
PCI: Calling quirk c031ce30 for 0000:01:0c.0
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 7.2.9-k2
Copyright (c) 1999-2006 Intel Corporation.
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c024e92c
*pde = 00529001
*pte = 00000000
Oops: 0000 [#1]
SMP
last sysfs file:
Modules linked in:
CPU:    0
EIP:    0060:[<c024e92c>]    Not tainted VLI
EFLAGS: 00010286   (2.6.18-mm3 #3)
EIP is at pci_call_probe+0x1c/0xe0
eax: 00000000   ebx: dfe63c00   ecx: c10fca90   edx: c048ad60
esi: dfe63c00   edi: 0000000f   ebp: dfc3fd00   esp: c10ffe70
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 1, ti=c10fe000 task=c10fca90 task.ti=c10fe000)
Stack: dfe63c00 ffffffed ffffffed dfe63c00 c048b1c0 c024ea55 c048b1c0 dfe63c00
       c048ad60 c048b1c0 dfe63c00 c048b1f4 c024ea9f c048b1c0 dfe63c00 dfe63c48
       dfc3fd00 c02761f9 dfe63c48 00000286 dfff9060 000000d0 c048b1f4 dfc3fd00
Call Trace:
 [<c024ea55>] __pci_device_probe+0x65/0x80
 [<c024ea9f>] pci_device_probe+0x2f/0x50
 [<c02761f9>] really_probe+0xf9/0x100
 [<c02762e8>] driver_probe_device+0xc8/0xe0
 [<c03ae92d>] klist_next+0x5d/0xa0
 [<c02763a0>] __driver_attach+0x0/0xa0
 [<c0276430>] __driver_attach+0x90/0xa0
 [<c0275409>] bus_for_each_dev+0x69/0x80
 [<c0276465>] driver_attach+0x25/0x30
 [<c02763a0>] __driver_attach+0x0/0xa0
 [<c0275ad3>] bus_add_driver+0x73/0x140
 [<c024edc4>] __pci_register_driver+0x74/0x90
 [<c050c6f9>] tulip_init+0x29/0x30
 [<c04f2a62>] do_initcalls+0x42/0x140
 [<c01433eb>] register_irq_proc+0xab/0xd0
 [<c01003f0>] init+0x0/0x1a0
 [<c0143479>] init_irq_proc+0x39/0x50
 [<c01003f0>] init+0x0/0x1a0
 [<c0100451>] init+0x61/0x1a0
 [<c0103c0b>] kernel_thread_helper+0x7/0x1c
 =======================
Code: 74 92 eb 8e 8d 74 26 00 8d bc 27 00 00 00 00 57 56 53 83 ec 08 8b 5c 24 1c 89 e0 25 00 e0 ff ff 8b 08 8b 43 10 8b 79 5c 8b 40 44 <8b> 50 04 85 d2 78 11 0f a3 15 c0 9f 4e c0 19 c0 85 c0 0f 85 8c
EIP: [<c024e92c>] pci_call_probe+0x1c/0xe0 SS:ESP 0068:c10ffe70
 <0>Kernel panic - not syncing: Attempted to kill init!

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Andy Whitcroft]

Martin Bligh wrote:
> Badari Pulavarty wrote:
>> On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote:
>>
>>> Not sure if you've seen this already ... catching up on test results.
>>>
>>> This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
>>> problem.
>>>
>>> Full logs:
>>>
>>> mm2 - http://test.kernel.org/abat/50727/debug/console.log
>>> mm3 - http://test.kernel.org/abat/51442/debug/console.log
>>>
>>> config - http://test.kernel.org/abat/51442/build/dotconfig
>>>
>>> I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
>>> is failing because bus->sysdata is NULL. The disassembly and
>>> structure offsets seem to line up for that.
>>>
>>> #define pcibus_to_node(bus) (
>>>     (struct pci_sysdata *)((bus)->sysdata))->node
>>>
>>> struct pci_sysdata {
>>>         int             domain;         /* PCI domain */
>>>         int             node;           /* NUMA node */
>>> };
>>>
>>
>>
>> Martin,
>>
>> Jeff moved "node" to a proper field in sysdata, instead
>> of overloading sysdata itself. I think this is causing the
>> problem. I guess we could end up with sysdata = NULL in some
>> cases ? Since you are the NUMA-Q expert, where does sysdata gets set
>> for NUMA-Q ? :)
>>
>> -mm2 changed:
>>
>> #define pcibus_to_node(bus) ((long) (bus)->sysdata)
>>
>> to
>> #define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
>>
>>> node
> 
> Buggered if I know, that's some strange pci thing ;-)
> 
> But can we revert whatever patch that was until it gets fixed, please?

Unless I am going very very mad, this has came up once before some
months ago.  We went through lots of pain finding the cause of this for
NUMA-Q and fixing it.  Something about not having a sysdata and needing
to initialise it.

Thought so, this was all discussed back in December 2005.

  http://lkml.org/lkml/2005/12/20/226

I'll go see if I can forward port the patch and address the remaining
issues with it.

-apw

Re: Panic in pci_call_probe from 2.6.18-mm2 and 2.6.18-mm3

[Greg KH]

On Fri, Oct 20, 2006 at 06:07:39PM +0100, Andy Whitcroft wrote:
> Martin Bligh wrote:
> > Badari Pulavarty wrote:
> >> On Sun, 2006-10-08 at 00:02 -0700, Martin J. Bligh wrote:
> >>
> >>> Not sure if you've seen this already ... catching up on test results.
> >>>
> >>> This was on NUMA-Q, on both -mm2 and -mm3. -mm1 didn't suffer from this
> >>> problem.
> >>>
> >>> Full logs:
> >>>
> >>> mm2 - http://test.kernel.org/abat/50727/debug/console.log
> >>> mm3 - http://test.kernel.org/abat/51442/debug/console.log
> >>>
> >>> config - http://test.kernel.org/abat/51442/build/dotconfig
> >>>
> >>> I'm guessing from the 00000004 that the pcibus_to_node(dev->bus)
> >>> is failing because bus->sysdata is NULL. The disassembly and
> >>> structure offsets seem to line up for that.
> >>>
> >>> #define pcibus_to_node(bus) (
> >>>     (struct pci_sysdata *)((bus)->sysdata))->node
> >>>
> >>> struct pci_sysdata {
> >>>         int             domain;         /* PCI domain */
> >>>         int             node;           /* NUMA node */
> >>> };
> >>>
> >>
> >>
> >> Martin,
> >>
> >> Jeff moved "node" to a proper field in sysdata, instead
> >> of overloading sysdata itself. I think this is causing the
> >> problem. I guess we could end up with sysdata = NULL in some
> >> cases ? Since you are the NUMA-Q expert, where does sysdata gets set
> >> for NUMA-Q ? :)
> >>
> >> -mm2 changed:
> >>
> >> #define pcibus_to_node(bus) ((long) (bus)->sysdata)
> >>
> >> to
> >> #define pcibus_to_node(bus) ((struct pci_sysdata *)((bus)->sysdata))-
> >>
> >>> node
> > 
> > Buggered if I know, that's some strange pci thing ;-)
> > 
> > But can we revert whatever patch that was until it gets fixed, please?
> 
> Unless I am going very very mad, this has came up once before some
> months ago.  We went through lots of pain finding the cause of this for
> NUMA-Q and fixing it.  Something about not having a sysdata and needing
> to initialise it.
> 
> Thought so, this was all discussed back in December 2005.
> 
>   http://lkml.org/lkml/2005/12/20/226
> 
> I'll go see if I can forward port the patch and address the remaining
> issues with it.

Yes, and I explicitly asked if this issue had been addressed again in
these patches.  That is why I rejected them oh so long ago...

bleah.

greg k-h

[PATCH] pci device ensure sysdata initialised v2

[Andy Whitcroft]

Ok, I've gone over the patches and retested them.  Other than having
to extend them to cover x86_64 they still seem to work as planned.  I
have updated the commentry to better explain the problem and the
fix as encapsulated here.

When this was proposed last time there was push back to get the
nodes right.  Whist this is clearly a good thing, I think we need
this as a first step if the underlying patches are going to stay
in.

-apw
=== 8< ===
pci device ensure sysdata initialised v2

We have been seeing panic's on NUMA systems in pci_call_probe() in
2.6.19-rc1-mm1 and later.  This is related to the changes introduced
in the commit below:

    [x86, PCI] Switch pci_bus::sysdata from NUMA node integer to a pointer
    0a247a58fc3e2ecfc17654301033e8b8d08df2a2

In this change the sysdata has changed from directly representing
a value (the node number in NUMA) to a pointer to a structure.
However, it seems that we do not always initialise this sysdata
before we probe the device.

Prior to the changes above the node was defaulted to 'NULL'
allocating the devices to node 0 unconditionally.  This patch adds
a default sysdata entry (pci_default_sysdata), this is then used
where 'NULL' was used previously.  pci_default_sysdata defaults
the node to unknown (-1).  This is a more accurate assignment,
mirroring the value returned where no topology support is provided
and no locality information is available.

There are only two uses of this value in the affected architectures
(x86, x86_64) and generic code:

1) in x86_64, dma_alloc_pages() looks up the node in order to
   allocate node local memory.  Here if the node is invalid we
   will default to the first online node.  Behaviour here should
   be unchanged.
2) in generic, pci_call_probe() looks up the node in order to
   restrict execution of the probe on the card local node, to
   favor node local allocation.  Where this is unknown previously
   we would force execution (and thereby allocation) to node 0,
   this is arguably wrong and using -1 releases this restriction.

In an ideal world we should be supplying a sysdata for the
appropriate node where it is known.  Where it is not known defaulting
to -1 seems a better course, and would help us where node 0 is
short of memory.

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---
diff --git a/arch/i386/pci/common.c b/arch/i386/pci/common.c
index aa0ac44..d3b8feb 100644
--- a/arch/i386/pci/common.c
+++ b/arch/i386/pci/common.c
@@ -27,6 +27,8 @@ unsigned long pirq_table_addr;
 struct pci_bus *pci_root_bus;
 struct pci_raw_ops *raw_pci_ops;
 
+struct pci_sysdata pci_default_sysdata = { .node = -1 };
+
 static int pci_read(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *value)
 {
 	return raw_pci_ops->read(pci_domain_nr(bus), bus->number,
diff --git a/arch/i386/pci/fixup.c b/arch/i386/pci/fixup.c
index 2bcd0a4..1ba35b3 100644
--- a/arch/i386/pci/fixup.c
+++ b/arch/i386/pci/fixup.c
@@ -25,9 +25,11 @@ static void __devinit pci_fixup_i450nx(s
 		pci_read_config_byte(d, reg++, &subb);
 		DBG("i450NX PXB %d: %02x/%02x/%02x\n", pxb, busno, suba, subb);
 		if (busno)
-			pci_scan_bus(busno, &pci_root_ops, NULL);	/* Bus A */
+			pci_scan_bus(busno, &pci_root_ops,
+					&pci_default_sysdata);	/* Bus A */
 		if (suba < subb)
-			pci_scan_bus(suba+1, &pci_root_ops, NULL);	/* Bus B */
+			pci_scan_bus(suba+1, &pci_root_ops,
+					&pci_default_sysdata);	/* Bus B */
 	}
 	pcibios_last_bus = -1;
 }
@@ -42,7 +44,7 @@ static void __devinit pci_fixup_i450gx(s
 	u8 busno;
 	pci_read_config_byte(d, 0x4a, &busno);
 	printk(KERN_INFO "PCI: i440KX/GX host bridge %s: secondary bus %02x\n", pci_name(d), busno);
-	pci_scan_bus(busno, &pci_root_ops, NULL);
+	pci_scan_bus(busno, &pci_root_ops, &pci_default_sysdata);
 	pcibios_last_bus = -1;
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82454GX, pci_fixup_i450gx);
diff --git a/arch/i386/pci/legacy.c b/arch/i386/pci/legacy.c
index 149a958..b076b25 100644
--- a/arch/i386/pci/legacy.c
+++ b/arch/i386/pci/legacy.c
@@ -26,7 +26,8 @@ static void __devinit pcibios_fixup_peer
 			    l != 0x0000 && l != 0xffff) {
 				DBG("Found device at %02x:%02x [%04x]\n", n, devfn, l);
 				printk(KERN_INFO "PCI: Discovered peer bus %02x\n", n);
-				pci_scan_bus(n, &pci_root_ops, NULL);
+				pci_scan_bus(n, &pci_root_ops, 
+						&pci_default_sysdata);
 				break;
 			}
 		}
diff --git a/arch/i386/pci/numa.c b/arch/i386/pci/numa.c
index adbe17a..8d2fd6c 100644
--- a/arch/i386/pci/numa.c
+++ b/arch/i386/pci/numa.c
@@ -97,9 +97,11 @@ static void __devinit pci_fixup_i450nx(s
 		pci_read_config_byte(d, reg++, &subb);
 		DBG("i450NX PXB %d: %02x/%02x/%02x\n", pxb, busno, suba, subb);
 		if (busno)
-			pci_scan_bus(QUADLOCAL2BUS(quad,busno), &pci_root_ops, NULL);	/* Bus A */
+			pci_scan_bus(QUADLOCAL2BUS(quad,busno), &pci_root_ops,
+					&pci_default_sysdata);	/* Bus A */
 		if (suba < subb)
-			pci_scan_bus(QUADLOCAL2BUS(quad,suba+1), &pci_root_ops, NULL);	/* Bus B */
+			pci_scan_bus(QUADLOCAL2BUS(quad,suba+1), &pci_root_ops,
+					&pci_default_sysdata);	/* Bus B */
 	}
 	pcibios_last_bus = -1;
 }
@@ -124,7 +126,7 @@ static int __init pci_numa_init(void)
 			printk("Scanning PCI bus %d for quad %d\n", 
 				QUADLOCAL2BUS(quad,0), quad);
 			pci_scan_bus(QUADLOCAL2BUS(quad,0), 
-				&pci_root_ops, NULL);
+				&pci_root_ops, &pci_default_sysdata);
 		}
 	return 0;
 }
diff --git a/arch/i386/pci/visws.c b/arch/i386/pci/visws.c
index f1b486d..2539371 100644
--- a/arch/i386/pci/visws.c
+++ b/arch/i386/pci/visws.c
@@ -101,8 +101,8 @@ static int __init pcibios_init(void)
 		"bridge B (PIIX4) bus: %u\n", pci_bus1, pci_bus0);
 
 	raw_pci_ops = &pci_direct_conf1;
-	pci_scan_bus(pci_bus0, &pci_root_ops, NULL);
-	pci_scan_bus(pci_bus1, &pci_root_ops, NULL);
+	pci_scan_bus(pci_bus0, &pci_root_ops, &pci_default_sysdata);
+	pci_scan_bus(pci_bus1, &pci_root_ops, &pci_default_sysdata);
 	pci_fixup_irqs(visws_swizzle, visws_map_irq);
 	pcibios_resource_survey();
 	return 0;
diff --git a/include/asm-i386/pci.h b/include/asm-i386/pci.h
index 2c8b5e9..d280b7f 100644
--- a/include/asm-i386/pci.h
+++ b/include/asm-i386/pci.h
@@ -8,6 +8,7 @@ struct pci_sysdata {
 	int		domain;		/* PCI domain */
 	int		node;		/* NUMA node */
 };
+extern struct pci_sysdata pci_default_sysdata;
 
 #ifdef CONFIG_PCI_DOMAINS
 static inline int pci_domain_nr(struct pci_bus *bus)
diff --git a/include/asm-x86_64/pci.h b/include/asm-x86_64/pci.h
index 550207f..5dc9aa5 100644
--- a/include/asm-x86_64/pci.h
+++ b/include/asm-x86_64/pci.h
@@ -10,6 +10,7 @@ struct pci_sysdata {
 	int		node;		/* NUMA node */
 	void*		iommu;		/* IOMMU private data */
 };
+extern struct pci_sysdata pci_default_sysdata;
 
 #ifdef CONFIG_PCI_DOMAINS
 static inline int pci_domain_nr(struct pci_bus *bus)

Re: [PATCH] pci device ensure sysdata initialised v2

[Jeff Garzik]

Andy Whitcroft wrote:
> Ok, I've gone over the patches and retested them.  Other than having
> to extend them to cover x86_64 they still seem to work as planned.  I
> have updated the commentry to better explain the problem and the
> fix as encapsulated here.
> 
> When this was proposed last time there was push back to get the
> nodes right.  Whist this is clearly a good thing, I think we need
> this as a first step if the underlying patches are going to stay
> in.

We definitely need something like this.  Having some sysdata static and 
some sysdata dynamic makes me nervous, and so I want to review the code 
paths in depth before applying this.  My gut feeling is that there is a 
bug remaining in this area, that this patch might exacerbate.

	Jeff