The change "PCI: assign ioapic resource at hotplug" breaks my system

The change "PCI: assign ioapic resource at hotplug" breaks my system

[Roland Dreier]

The change "PCI: assign ioapic resource at hotplug" (commit
23186279658cea6d42a050400d3e79c56cb459b4 in Linus's tree) makes
networking stop working on my system (SuperMicro H8QC8 with four
dual-core Opteron 885 CPUs).  In particular, the on-board NIC stops
working, probably because it gets assigned the wrong IRQ (225 in the
non-working case, 217 in the working case)

With that patch applied, e1000 doesn't work.  Reverting just that
patch (shown below) from Linus's latest tree fixes things for me.

Please let me know what other debug information might be useful.

Thanks,
  Roland

Here's the patch I revert.  I'm not sure what it's trying to do, or
why it breaks my systems.  But anyway, reverting this fixes things for
me:

Author: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Date:   Tue Sep 12 10:21:44 2006 -0700

    PCI: assign ioapic resource at hotplug
    
    We need to assign resources to ioapics being hot-added. This patch
    changes pbus_assign_resources_sorted() to assign resources if the
    ioapic has no assigned resources.
    
    Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
    Signed-off-by: MUNEDA Takahiro <muneda.takahiro@jp.fujitsu.com>
    Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
    Signed-off-by: Kristen Carlson Accardi <kristen.c.accardi@intel.com>
    Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 47c1071..5440491 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -55,12 +55,19 @@ pbus_assign_resources_sorted(struct pci_
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		u16 class = dev->class >> 8;
 
-		/* Don't touch classless devices or host bridges or ioapics.  */
+		/* Don't touch classless devices or host bridges. */
 		if (class == PCI_CLASS_NOT_DEFINED ||
-		    class == PCI_CLASS_BRIDGE_HOST ||
-		    class == PCI_CLASS_SYSTEM_PIC)
+		    class == PCI_CLASS_BRIDGE_HOST)
 			continue;
 
+		/* Don't touch ioapics if it has the assigned resources. */
+		if (class == PCI_CLASS_SYSTEM_PIC) {
+			res = &dev->resource[0];
+			if (res[0].start || res[1].start || res[2].start ||
+			    res[3].start || res[4].start || res[5].start)
+				continue;
+		}
+
 		pdev_sort_resources(dev, &head);
 	}

Re: The change "PCI: assign ioapic resource at hotplug" breaks my system

[Roland Dreier]

One piece of information that might be useful is that lspci shows a
difference in the configuration of the PCI bridge IOAPIC.  In the good
(working) case, the IOAPIC memory region 0 is disabled, while in the
bad case it is enabled.

Here are full details: first, the good/working case:

	04:01.1 PIC: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC (rev 12) (prog-if 10 [IO-APIC])
		Subsystem: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
		Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
		Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
		Latency: 0
		Region 0: Memory at <ignored> (64-bit, non-prefetchable)
	00: 22 10 59 74 06 00 00 02 12 10 00 08 00 00 00 00
	10: 04 e0 af fe 00 00 00 00 00 00 00 00 00 00 00 00
	20: 00 00 00 00 00 00 00 00 00 00 00 00 22 10 59 74
	30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 00 00
	40: 00 00 00 00 03 00 00 00 04 e0 af fe 00 00 00 00

Then the bad (non-working e1000) case:

	04:01.1 PIC: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC (rev 12) (prog-if 10 [IO-APIC])
		Subsystem: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
		Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
		Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
		Latency: 0
		Region 0: Memory at e2100000 (64-bit, non-prefetchable) [size=4K]
	00: 22 10 59 74 06 00 00 02 12 10 00 08 00 00 00 00
	10: 04 00 10 e2 00 00 00 00 00 00 00 00 00 00 00 00
	20: 00 00 00 00 00 00 00 00 00 00 00 00 22 10 59 74
	30: 00 00 00 00 50 00 00 00 00 00 00 00 00 00 00 00
	40: 00 00 00 00 03 00 00 00 04 00 10 e2 00 00 00 00

I have no idea whether there's any significance to this.

 - R.

Re: The change "PCI: assign ioapic resource at hotplug" breaks my system

[Kenji Kaneshige]

Roland Dreier wrote:
> The change "PCI: assign ioapic resource at hotplug" (commit
> 23186279658cea6d42a050400d3e79c56cb459b4 in Linus's tree) makes
> networking stop working on my system (SuperMicro H8QC8 with four
> dual-core Opteron 885 CPUs).  In particular, the on-board NIC stops
> working, probably because it gets assigned the wrong IRQ (225 in the
> non-working case, 217 in the working case)
> 
> With that patch applied, e1000 doesn't work.  Reverting just that
> patch (shown below) from Linus's latest tree fixes things for me.
> 
> Please let me know what other debug information might be useful.
> 

The cause of this problem might be an wrong assumption that the 'start'
member of resource structure for ioapic device has non-zero value if the
resources are assigned by firmware. The 'start' member of ioapic device
seems not to be set even though the resources were actually assigned to
ioapic devices by firmware.

I made a patch to fix this problem against 2.6.18-git18. This patch
checks command register instead of checking 'start' member to see if
the ioapic is already enabled by firmware. Unfortunately, I don't have
any system to reproduce this problem. Could you please try it and let
me know whether the problem is fixed? If the patch below fixes the
problem, I'll resend it with description and Signed-off-by.

Thanks,
Kenji Kaneshige

---
 drivers/pci/setup-bus.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

Index: linux-2.6.18-git18/drivers/pci/setup-bus.c
===================================================================
--- linux-2.6.18-git18.orig/drivers/pci/setup-bus.c	2006-10-03 13:26:49.000000000 +0900
+++ linux-2.6.18-git18/drivers/pci/setup-bus.c	2006-10-03 13:35:00.000000000 +0900
@@ -55,16 +55,16 @@
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		u16 class = dev->class >> 8;
 
-		/* Don't touch classless devices or host bridges. */
+		/* Don't touch classless devices or host bridges or ioapics.  */
 		if (class == PCI_CLASS_NOT_DEFINED ||
 		    class == PCI_CLASS_BRIDGE_HOST)
 			continue;
 
-		/* Don't touch ioapics if it has the assigned resources. */
+		/* Don't touch ioapic devices already enabled by firmware */
 		if (class == PCI_CLASS_SYSTEM_PIC) {
-			res = &dev->resource[0];
-			if (res[0].start || res[1].start || res[2].start ||
-			    res[3].start || res[4].start || res[5].start)
+			u16 command;
+			pci_read_config_word(dev, PCI_COMMAND, &command);
+			if (command & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY))
 				continue;
 		}
 

Re: The change "PCI: assign ioapic resource at hotplug" breaks my system

[Roland Dreier]

    Kenji> The cause of this problem might be an wrong assumption that
    Kenji> the 'start' member of resource structure for ioapic device
    Kenji> has non-zero value if the resources are assigned by
    Kenji> firmware. The 'start' member of ioapic device seems not to
    Kenji> be set even though the resources were actually assigned to
    Kenji> ioapic devices by firmware.

    Kenji> I made a patch to fix this problem against
    Kenji> 2.6.18-git18. This patch checks command register instead of
    Kenji> checking 'start' member to see if the ioapic is already
    Kenji> enabled by firmware. Unfortunately, I don't have any system
    Kenji> to reproduce this problem. Could you please try it and let
    Kenji> me know whether the problem is fixed? If the patch below
    Kenji> fixes the problem, I'll resend it with description and
    Kenji> Signed-off-by.

Yes, applying this patch makes everything work on the same SuperMicro
motherboard that breaks with Linus's current tree.  Assuming this
doesn't break anything else, I think this should go upstream.

Thanks,
  Roland

Re: The change "PCI: assign ioapic resource at hotplug" breaks my system

[Stephen Hemminger]

On Tue, 03 Oct 2006 21:32:54 +0900
Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com> wrote:

> Roland Dreier wrote:
> > The change "PCI: assign ioapic resource at hotplug" (commit
> > 23186279658cea6d42a050400d3e79c56cb459b4 in Linus's tree) makes
> > networking stop working on my system (SuperMicro H8QC8 with four
> > dual-core Opteron 885 CPUs).  In particular, the on-board NIC stops
> > working, probably because it gets assigned the wrong IRQ (225 in the
> > non-working case, 217 in the working case)
> > 
> > With that patch applied, e1000 doesn't work.  Reverting just that
> > patch (shown below) from Linus's latest tree fixes things for me.
> > 
> > Please let me know what other debug information might be useful.
> > 
> 
> The cause of this problem might be an wrong assumption that the 'start'
> member of resource structure for ioapic device has non-zero value if the
> resources are assigned by firmware. The 'start' member of ioapic device
> seems not to be set even though the resources were actually assigned to
> ioapic devices by firmware.
> 
> I made a patch to fix this problem against 2.6.18-git18. This patch
> checks command register instead of checking 'start' member to see if
> the ioapic is already enabled by firmware. Unfortunately, I don't have
> any system to reproduce this problem. Could you please try it and let
> me know whether the problem is fixed? If the patch below fixes the
> problem, I'll resend it with description and Signed-off-by.
> 
> Thanks,
> Kenji Kaneshige
>

This also fixes my problems with the built in tg3 on the dual CPU Opteron
IBM workstation.

Re: The change "PCI: assign ioapic resource at hotplug" breaks my system

[Kenji Kaneshige]

Stephen, Roland,

Thank you very much for testing the patch.

Thanks,
Kenji Kaneshige


Stephen Hemminger wrote:
> On Tue, 03 Oct 2006 21:32:54 +0900
> Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com> wrote:
> 
>> Roland Dreier wrote:
>>> The change "PCI: assign ioapic resource at hotplug" (commit
>>> 23186279658cea6d42a050400d3e79c56cb459b4 in Linus's tree) makes
>>> networking stop working on my system (SuperMicro H8QC8 with four
>>> dual-core Opteron 885 CPUs).  In particular, the on-board NIC stops
>>> working, probably because it gets assigned the wrong IRQ (225 in the
>>> non-working case, 217 in the working case)
>>>
>>> With that patch applied, e1000 doesn't work.  Reverting just that
>>> patch (shown below) from Linus's latest tree fixes things for me.
>>>
>>> Please let me know what other debug information might be useful.
>>>
>> The cause of this problem might be an wrong assumption that the 'start'
>> member of resource structure for ioapic device has non-zero value if the
>> resources are assigned by firmware. The 'start' member of ioapic device
>> seems not to be set even though the resources were actually assigned to
>> ioapic devices by firmware.
>>
>> I made a patch to fix this problem against 2.6.18-git18. This patch
>> checks command register instead of checking 'start' member to see if
>> the ioapic is already enabled by firmware. Unfortunately, I don't have
>> any system to reproduce this problem. Could you please try it and let
>> me know whether the problem is fixed? If the patch below fixes the
>> problem, I'll resend it with description and Signed-off-by.
>>
>> Thanks,
>> Kenji Kaneshige
>>
> 
> This also fixes my problems with the built in tg3 on the dual CPU Opteron
> IBM workstation.
> 
>