Re: [PATCH v2 00/29] Create PCI root buses with correct resources

Re: [PATCH v2 00/29] Create PCI root buses with correct resources

[Bjorn Helgaas]

On Fri, Oct 14, 2011 at 9:45 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Fri, 2011-10-14 at 09:33 -0600, Bjorn Helgaas wrote:
>> On Fri, Oct 14, 2011 at 9:14 AM, James Bottomley
>> <James.Bottomley@hansenpartnership.com> wrote:
>> > On Thu, 2011-10-13 at 22:27 -0600, Bjorn Helgaas wrote:
>> >> The most generic way to create a PCI root bus is with pci_scan_bus(), but
>> >> that doesn't allow you to specify what resources are available on the bus,
>> >> i.e., what the host bridge apertures are.
>> >
>> > Do you have an actual tree for this?  The PARISC pieces look a bit
>> > non-trivial and I'd like to test them out on the various sba/astro/dino
>> > systems.
>>
>> I don't yet, but it'd be great if you were willing to test them out,
>> so I'll set one up.
>>
>> While this is on your mind, I'm dubious about the pci_enable_bridges()
>> call in lba_driver_probe() and the pci_bus_assign_resources() call in
>> dino_probe().  These stand out as exceptions because no other arches
>> do that.
>>
>> If they weren't there, both places could use pci_scan_root_bus() like
>> most other arches.
>
> Right, the dino code actually has to rearrange some of the PCI resources
> to make sure we get our apertures correct.  That's the specific bit I
> was planning to test.  I think it's only relevant for dino SBA cards
> where the bridge doesn't set up the card correctly so we have to do it
> after the fact.

This series has been in linux-next for a while now (via Jesse's PCI
tree), so this would be a good time to try it out if you haven't
already.

I know ia64 has been exercised a bit (thanks, Tony), but I haven't
heard anything about arm, mips, parisc, powerpc, sparc, etc.

There is one outstanding bug fix for alpha that Jesse hasn't
integrated yet, but as far as I know, everything else is in good
shape.

Bjorn

Re: [PATCH v2 00/29] Create PCI root buses with correct resources

[David Miller]

From: Bjorn Helgaas <bhelgaas@google.com>
Date: Tue, 20 Dec 2011 08:55:53 -0700

> This series has been in linux-next for a while now (via Jesse's PCI
> tree), so this would be a good time to try it out if you haven't
> already.
> 
> I know ia64 has been exercised a bit (thanks, Tony), but I haven't
> heard anything about arm, mips, parisc, powerpc, sparc, etc.
> 
> There is one outstanding bug fix for alpha that Jesse hasn't
> integrated yet, but as far as I know, everything else is in good
> shape.

I'll give it a quick test on sparc64 today.

Re: [PATCH v2 00/29] Create PCI root buses with correct resources

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Thu, 22 Dec 2011 13:59:11 -0500 (EST)

> From: Bjorn Helgaas <bhelgaas@google.com>
> Date: Tue, 20 Dec 2011 08:55:53 -0700
> 
>> This series has been in linux-next for a while now (via Jesse's PCI
>> tree), so this would be a good time to try it out if you haven't
>> already.
>> 
>> I know ia64 has been exercised a bit (thanks, Tony), but I haven't
>> heard anything about arm, mips, parisc, powerpc, sparc, etc.
>> 
>> There is one outstanding bug fix for alpha that Jesse hasn't
>> integrated yet, but as far as I know, everything else is in good
>> shape.
> 
> I'll give it a quick test on sparc64 today.

Jesse's linux-next GIT branch boots up fine, but I get tons of
warnings that kobject_put() is being performed on an uninitialized
kobject.

It looks like populate_msi_sysfs() is where these kobjects are
supposed to be initialized.

Anyways, two examples:

[2766595.475642] mpt2sas1: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (7767632 kB)
[2766595.491604] ------------[ cut here ]------------
[2766595.500986] WARNING: at lib/kobject.c:595 kobject_put+0x48/0x80()
[2766595.513482] kobject: '(null)' (fffff801f9b7a620): is not initialized, yet kobject_put() is being called.
[2766595.532762] Modules linked in:
[2766595.539152] Call Trace:
[2766595.544391]  [000000000046944c] warn_slowpath_fmt+0x2c/0x40
[2766595.555837]  [00000000005ed188] kobject_put+0x48/0x80
[2766595.566261]  [0000000000622b68] free_msi_irqs+0xc8/0x120
[2766595.577196]  [0000000000623870] pci_enable_msix+0x270/0x420
[2766595.588665]  [00000000006a9fe8] mpt2sas_base_map_resources+0x2c8/0x480
[2766595.602023]  [00000000006acb8c] mpt2sas_base_attach+0xac/0x1280
[2766595.614166]  [00000000006b5c20] _scsih_probe+0x2a0/0x480
[2766595.625113]  [000000000061d048] pci_device_probe+0x108/0x120
[2766595.636746]  [0000000000671464] driver_probe_device+0x64/0x180
[2766595.648716]  [0000000000671600] __driver_attach+0x80/0xa0
[2766595.659838]  [0000000000670570] bus_for_each_dev+0x50/0x80
[2766595.671115]  [0000000000670cdc] bus_add_driver+0x9c/0x260
[2766595.682217]  [000000000067190c] driver_register+0x6c/0x160
[2766595.693508]  [000000000061d4d4] __pci_register_driver+0x34/0xc0
[2766595.705652]  [000000000096912c] _scsih_init+0x150/0x17c
[2766595.716413]  [0000000000426ad8] do_one_initcall+0x18/0x160
[2766595.727697] ---[ end trace 1cc904a40b8d2c88 ]---

--------------------

[2766607.786504] ------------[ cut here ]------------
[2766607.786519] WARNING: at lib/kobject.c:595 kobject_put+0x48/0x80()
[2766607.786529] kobject: '(null)' (fffff801fbec1280): is not initialized, yet kobject_put() is being called.
[2766607.786540] Modules linked in: igb(+) ehci_hcd(+) n2_rng rng_core
[2766607.786554] Call Trace:
[2766607.786568]  [000000000046944c] warn_slowpath_fmt+0x2c/0x40
[2766607.786581]  [00000000005ed188] kobject_put+0x48/0x80
[2766607.786596]  [0000000000622b68] free_msi_irqs+0xc8/0x120
[2766607.786610]  [0000000000622e1c] pci_enable_msi_block+0x25c/0x2c0
[2766607.786644]  [0000000010054b78] igb_probe+0x514/0xfc0 [igb]
[2766607.786658]  [000000000061d048] pci_device_probe+0x108/0x120
[2766607.786672]  [0000000000671464] driver_probe_device+0x64/0x180
[2766607.786683]  [0000000000671600] __driver_attach+0x80/0xa0
[2766607.786694]  [0000000000670570] bus_for_each_dev+0x50/0x80
[2766607.786707]  [0000000000670cdc] bus_add_driver+0x9c/0x260
[2766607.786718]  [000000000067190c] driver_register+0x6c/0x160
[2766607.786730]  [000000000061d4d4] __pci_register_driver+0x34/0xc0
[2766607.786743]  [0000000000426ad8] do_one_initcall+0x18/0x160
[2766607.786756]  [00000000004a33ec] SyS_init_module+0x6c/0x1c0
[2766607.786771]  [0000000000406194] linux_sparc_syscall32+0x34/0x40
[2766607.786779] ---[ end trace 1cc904a40b8d2c9d ]---

--------------------

Re: [PATCH v2 00/29] Create PCI root buses with correct resources

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Thu, 22 Dec 2011 14:15:37 -0500 (EST)

> It looks like populate_msi_sysfs() is where these kobjects are
> supposed to be initialized.

Ok, it looks like we violate the invariant the free_msi_irqs() should
only be invoked after populate_msi_sysfs() has successfully been run.

For example, in msix_capability_init(), if arch_setup_msi_irqs() fails
we will branch to "error" and do free_msi_irqs().

free_msi_irqs() unconditionally does puts on the MSI entry kobjects,
but we haven't initialized them in populate_msi_sysfs() and thus we
get the warnings.

Even if we did execute populate_msi_sysfs(), if that function
encountered an error part way through the MSI entry list, we'll still
hit this kobject_put() warning for the MSI entrys we did not process
and init.

In my case arch_setup_msi_irqs() is failing simply because MSI has
not been enabled on these PCI host bridges.  It should, and I'm
trying to figure out if this is some strange interaction with the
new resource code.

But this is independent of the above bugs that can result in erroneous
kobject puts.

Re: [PATCH v2 00/29] Create PCI root buses with correct resources

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Thu, 22 Dec 2011 14:56:29 -0500 (EST)

> In my case arch_setup_msi_irqs() is failing simply because MSI has
> not been enabled on these PCI host bridges.  It should, and I'm
> trying to figure out if this is some strange interaction with the
> new resource code.

It wasn't a problem with the resource changes, it's a sparc64 problem
that only triggers on more recent systems.

The MSI SYSFS error handling is, of course, still needing to be fixed.