[RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Several chipsets are known to not support MSI. Some support MSI but
disable it by default. Thus, several drivers implement their own way to
detect whether MSI works.

We introduce whitelisting of chipsets that are known to support MSI and
keep the existing backlisting to disable MSI for other chipsets. When it
is unknown whether the root chipset support MSI or not, we disable MSI
by default except if pci=forcemsi was passed.

Whitelisting is done by setting a new PCI_BUS_FLAGS_MSI in the chipset
subordinate bus. pci_enable_msi() thus starts by checking whether the
root chipset of the device has the MSI or NOMSI flag set.

Bus flags inheritance is dropped since it has been reported to be broken.

We currently whitelist all Intel chipsets, nVidia CK804 and ServerWorks
HT2000. Some others are probably missing such as nVidia chipsets for
Intel processors.

We introduces a generic code to check the MSI capability in the
HyperTransport configuration space, and use it in the generic HT MSI
quirk (only used for HT2000 so far) and in the CK804 specific quirk
(needs to check 2 HT MSI mapping).

Finally, we rename PCI_CAP_ID_HT_IRQCONF to PCI_CAP_ID_HT since its
value is the capability and not the irqconf subtype.

Signed-off-by: Brice Goglin <brice@myri.com>

---

I do not split this patch in multiple pieces since it is still small
and this is just a RFC. I will split it for the actual submission.

The patch adds PCI_DEVICE_ID_NVIDIA_CK804_PCIE to pciids.h and may thus
conflict with pci-nvidia-quirk-to-make-aer-pci-e-extended-capability-visible.patch
which adds it too (it has been pushed in Greg K-H's patchset today).


 Documentation/kernel-parameters.txt |    6 +
 arch/powerpc/sysdev/mpic.c          |    2
 drivers/pci/msi.c                   |   58 ++++++++++-----
 drivers/pci/pci.c                   |    2
 drivers/pci/pci.h                   |    1
 drivers/pci/probe.c                 |    2
 drivers/pci/quirks.c                |   95 +++++++++++++++++++++++++-
 include/linux/pci.h                 |    3
 include/linux/pci_ids.h             |    2
 include/linux/pci_regs.h            |    2
 10 files changed, 147 insertions(+), 26 deletions(-)

Index: linux-mm/drivers/pci/msi.c
===================================================================
--- linux-mm.orig/drivers/pci/msi.c	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/drivers/pci/msi.c	2006-06-13 14:55:36.000000000 -0400
@@ -28,6 +28,7 @@
 static kmem_cache_t* msi_cachep;
 
 static int pci_msi_enable = 1;
+static int pci_msi_force = 0;
 static int last_alloc_vector;
 static int nr_released_vectors;
 static int nr_reserved_vectors = NR_HP_RESERVED_VECTORS;
@@ -902,6 +903,34 @@
 }
 
 /**
+ * pci_msi_enabled - check whether MSI may be enabled on device
+ * @dev: pointer to the pci_dev data structure of MSI device function
+ *
+ * Check parent busses for MSI or NO_MSI flags, or disable except
+ * if forced.
+ **/
+int pci_msi_enabled(struct pci_dev * dev)
+{
+	struct pci_dev *pdev;
+
+	if (!pci_msi_enable || !dev || dev->no_msi)
+ 		return -1;
+
+	/* find root complex for our device */
+	pdev = dev;
+	while (pdev->bus && pdev->bus->self)
+		pdev = pdev->bus->self;
+
+	/* check its bus flags */
+	if (pdev->subordinate->bus_flags & PCI_BUS_FLAGS_MSI)
+		return 0;
+	if (pdev->subordinate->bus_flags & PCI_BUS_FLAGS_NO_MSI)
+		return -1;
+
+	return pci_msi_force ? 0 : -1;
+}
+
+/**
  * pci_enable_msi - configure device's MSI capability structure
  * @dev: pointer to the pci_dev data structure of MSI device function
  *
@@ -913,19 +942,11 @@
  **/
 int pci_enable_msi(struct pci_dev* dev)
 {
-	struct pci_bus *bus;
-	int pos, temp, status = -EINVAL;
+	int pos, temp, status;
 	u16 control;
 
-	if (!pci_msi_enable || !dev)
- 		return status;
-
-	if (dev->no_msi)
-		return status;
-
-	for (bus = dev->bus; bus; bus = bus->parent)
-		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
-			return -EINVAL;
+	if (pci_msi_enabled(dev) < 0)
+		return -EINVAL;
 
 	temp = dev->irq;
 
@@ -1135,22 +1156,14 @@
  **/
 int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
 {
-	struct pci_bus *bus;
 	int status, pos, nr_entries, free_vectors;
 	int i, j, temp;
 	u16 control;
 	unsigned long flags;
 
-	if (!pci_msi_enable || !dev || !entries)
- 		return -EINVAL;
-
-	if (dev->no_msi)
+	if (!entries || pci_msi_enabled(dev) < 0)
 		return -EINVAL;
 
-	for (bus = dev->bus; bus; bus = bus->parent)
-		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
-			return -EINVAL;
-
 	status = msi_init();
 	if (status < 0)
 		return status;
@@ -1350,6 +1363,11 @@
 	pci_msi_enable = 0;
 }
 
+void pci_force_msi(void)
+{
+	pci_msi_force = 1;
+}
+
 EXPORT_SYMBOL(pci_enable_msi);
 EXPORT_SYMBOL(pci_disable_msi);
 EXPORT_SYMBOL(pci_enable_msix);
Index: linux-mm/drivers/pci/pci.c
===================================================================
--- linux-mm.orig/drivers/pci/pci.c	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/drivers/pci/pci.c	2006-06-13 14:55:36.000000000 -0400
@@ -978,6 +978,8 @@
 		if (*str && (str = pcibios_setup(str)) && *str) {
 			if (!strcmp(str, "nomsi")) {
 				pci_no_msi();
+			} else if (!strcmp(str, "forcemsi")) {
+				pci_force_msi();
 			} else {
 				printk(KERN_ERR "PCI: Unknown option `%s'\n",
 						str);
Index: linux-mm/drivers/pci/pci.h
===================================================================
--- linux-mm.orig/drivers/pci/pci.h	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/drivers/pci/pci.h	2006-06-13 14:55:36.000000000 -0400
@@ -51,6 +51,7 @@
 #ifdef CONFIG_PCI_MSI
 void disable_msi_mode(struct pci_dev *dev, int pos, int type);
 void pci_no_msi(void);
+void pci_force_msi(void);
 #else
 static inline void disable_msi_mode(struct pci_dev *dev, int pos, int type) { }
 static inline void pci_no_msi(void) { }
Index: linux-mm/drivers/pci/quirks.c
===================================================================
--- linux-mm.orig/drivers/pci/quirks.c	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/drivers/pci/quirks.c	2006-06-16 22:26:42.000000000 -0400
@@ -588,7 +588,8 @@
         
 	if (dev->subordinate) {
 		printk(KERN_WARNING "PCI: MSI quirk detected. "
-		       "PCI_BUS_FLAGS_NO_MSI set for subordinate bus.\n");
+		       "PCI_BUS_FLAGS_NO_MSI set for %s subordinate bus.\n",
+		       pci_name(dev));
 		dev->subordinate->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
 	}
 
@@ -608,7 +609,9 @@
 static void __init quirk_svw_msi(struct pci_dev *dev)
 {
 	pci_msi_quirk = 1;
-	printk(KERN_WARNING "PCI: MSI quirk detected. pci_msi_quirk set.\n");
+	printk(KERN_WARNING "PCI: MSI quirk detected for %s. "
+	       "pci_msi_quirk set.\n",
+	       pci_name(dev));
 }
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SERVERWORKS, PCI_DEVICE_ID_SERVERWORKS_GCNB_LE, quirk_svw_msi );
 #endif /* CONFIG_X86_IO_APIC */
@@ -1556,6 +1559,94 @@
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,	0x1460,		quirk_p64h2_1k_io);
 
+/* Mark MSI bus flags on chipset that are known to support it */
+static void __devinit quirk_msi_supported(struct pci_dev *dev)
+{
+	if (dev->subordinate) {
+		printk(KERN_WARNING "PCI: MSI quirk detected. "
+		       "PCI_BUS_FLAGS_MSI set for subordinate bus.\n");
+		dev->subordinate->bus_flags |= PCI_BUS_FLAGS_MSI;
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_ANY_ID, quirk_msi_supported);
+
+/* Return MSI bus flags depending on MSI HT cap */
+static pci_bus_flags_t __devinit check_msi_ht_cap(struct pci_dev *dev)
+{
+	u8 cap_off;
+	int nbcap = 0;
+	cap_off = PCI_CAPABILITY_LIST - 1;
+
+	/* go through all caps looking for a hypertransport msi mapping */
+	while (pci_read_config_byte(dev, cap_off + 1, &cap_off) == 0 &&
+		nbcap++ <= 256 / 4) {
+		u32 cap_hdr;
+		if (cap_off == 0 || cap_off == 0xff)
+			break;
+		cap_off &= 0xfc;
+		/* msi mapping section according to hypertransport spec */
+		if (pci_read_config_dword(dev, cap_off, &cap_hdr) == 0
+		    && (cap_hdr & 0xff) == PCI_CAP_ID_HT /* hypertransport cap */
+		    && (cap_hdr & 0xf8000000) == 0xa8000000 /* msi mapping */) {
+			printk(KERN_INFO "PCI: Found MSI HT mapping on %s\n",
+			       pci_name(dev));
+			return cap_hdr & 0x10000 /* msi mapping cap enabled */
+			    ? PCI_BUS_FLAGS_MSI : PCI_BUS_FLAGS_NO_MSI;
+		}
+	}
+	return 0;
+}
+
+/* Check the hypertransport MSI mapping to know whether MSI is enabled or not */
+static void __devinit quirk_msi_ht_cap(struct pci_dev *dev)
+{
+	pci_bus_flags_t flags;
+
+	if (!dev->subordinate)
+		return;
+
+	flags = check_msi_ht_cap(dev);
+	if (flags) {
+		printk(KERN_WARNING "PCI: MSI quirk detected. "
+		       "%s set for %s subordinate bus.\n",
+		       flags == PCI_BUS_FLAGS_NO_MSI ?
+			   "PCI_BUS_FLAGS_NO_MSI" : "PCI_BUS_FLAGS_MSI",
+		       pci_name(dev));
+		dev->subordinate->bus_flags |= flags;
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SERVERWORKS, PCI_DEVICE_ID_SERVERWORKS_HT2000_PCIE, quirk_msi_ht_cap);
+
+/* The nVidia CK804 PCI-E might have 2 hypertransport MSI mapping.
+ * MSI are supported if the MSI cap is enabled on one of them.
+ */
+static void __devinit quirk_nvidia_ck804_msi_ht_cap(struct pci_dev *dev)
+{
+	struct pci_dev *pdev;
+	pci_bus_flags_t flags;
+
+	if (!dev->subordinate)
+		return;
+
+	/* check MSI HT cap on this chipset and the root one.
+	 * a single MSI flag is enough to be sure that MSI are supported.
+	 */
+	pdev = pci_find_slot(dev->bus->number, 0);
+	flags = check_msi_ht_cap(dev) | check_msi_ht_cap(pdev);
+	if (flags & PCI_BUS_FLAGS_MSI)
+		flags = PCI_BUS_FLAGS_MSI;
+
+	if (flags) {
+		printk(KERN_WARNING "PCI: MSI quirk detected. "
+		       "%s set for %s subordinate bus.\n",
+		       flags == PCI_BUS_FLAGS_NO_MSI ?
+			   "PCI_BUS_FLAGS_NO_MSI" : "PCI_BUS_FLAGS_MSI",
+		       pci_name(dev));
+		dev->subordinate->bus_flags |= flags;
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_DEVICE_ID_NVIDIA_CK804_PCIE, quirk_nvidia_ck804_msi_ht_cap);
+
 EXPORT_SYMBOL(pcie_mch_quirk);
 #ifdef CONFIG_HOTPLUG
 EXPORT_SYMBOL(pci_fixup_device);
Index: linux-mm/include/linux/pci.h
===================================================================
--- linux-mm.orig/include/linux/pci.h	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/include/linux/pci.h	2006-06-13 14:55:37.000000000 -0400
@@ -97,6 +97,7 @@
 typedef unsigned short __bitwise pci_bus_flags_t;
 enum pci_bus_flags {
 	PCI_BUS_FLAGS_NO_MSI = (__force pci_bus_flags_t) 1,
+	PCI_BUS_FLAGS_MSI = (__force pci_bus_flags_t) 2,
 };
 
 struct pci_cap_saved_state {
@@ -242,7 +243,7 @@
 	char		name[48];
 
 	unsigned short  bridge_ctl;	/* manage NO_ISA/FBB/et al behaviors */
-	pci_bus_flags_t bus_flags;	/* Inherited by child busses */
+	pci_bus_flags_t bus_flags;
 	struct device		*bridge;
 	struct class_device	class_dev;
 	struct bin_attribute	*legacy_io; /* legacy I/O for this bus */
Index: linux-mm/include/linux/pci_ids.h
===================================================================
--- linux-mm.orig/include/linux/pci_ids.h	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/include/linux/pci_ids.h	2006-06-13 14:59:58.000000000 -0400
@@ -1027,6 +1027,7 @@
 #define PCI_DEVICE_ID_NVIDIA_NVENET_8		0x0056
 #define PCI_DEVICE_ID_NVIDIA_NVENET_9		0x0057
 #define PCI_DEVICE_ID_NVIDIA_CK804_AUDIO	0x0059
+#define PCI_DEVICE_ID_NVIDIA_CK804_PCIE		0x005d
 #define PCI_DEVICE_ID_NVIDIA_NFORCE2_SMBUS	0x0064
 #define PCI_DEVICE_ID_NVIDIA_NFORCE2_IDE	0x0065
 #define PCI_DEVICE_ID_NVIDIA_NVENET_2		0x0066
@@ -1405,6 +1406,7 @@
 #define PCI_DEVICE_ID_SERVERWORKS_LE	  0x0009
 #define PCI_DEVICE_ID_SERVERWORKS_GCNB_LE 0x0017
 #define PCI_DEVICE_ID_SERVERWORKS_EPB	  0x0103
+#define PCI_DEVICE_ID_SERVERWORKS_HT2000_PCIE	0x0132
 #define PCI_DEVICE_ID_SERVERWORKS_OSB4	  0x0200
 #define PCI_DEVICE_ID_SERVERWORKS_CSB5	  0x0201
 #define PCI_DEVICE_ID_SERVERWORKS_CSB6    0x0203
Index: linux-mm/Documentation/kernel-parameters.txt
===================================================================
--- linux-mm.orig/Documentation/kernel-parameters.txt	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/Documentation/kernel-parameters.txt	2006-06-13 14:55:37.000000000 -0400
@@ -1190,6 +1190,12 @@
 		nomsi		[MSI] If the PCI_MSI kernel config parameter is
 				enabled, this kernel boot option can be used to
 				disable the use of MSI interrupts system-wide.
+		forcemsi	[MSI] If the PCI_MSI kernel config parameter is
+				enabled, force MSI to be enabled when its support
+				in parent busses is unknown. By default, MSI
+				are only enabled when parent busses are known
+				to support it. This option does not force MSI
+				when parent busses are known to not support it.
 		nosort		[IA-32] Don't sort PCI devices according to
 				order given by the PCI BIOS. This sorting is
 				done to get a device order compatible with
Index: linux-mm/drivers/pci/probe.c
===================================================================
--- linux-mm.orig/drivers/pci/probe.c	2006-06-12 10:08:25.000000000 -0400
+++ linux-mm/drivers/pci/probe.c	2006-06-13 14:55:37.000000000 -0400
@@ -351,7 +351,7 @@
 	child->parent = parent;
 	child->ops = parent->ops;
 	child->sysdata = parent->sysdata;
-	child->bus_flags = parent->bus_flags;
+	child->bus_flags = 0;
 	child->bridge = get_device(&bridge->dev);
 
 	child->class_dev.class = &pcibus_class;
Index: linux-mm/arch/powerpc/sysdev/mpic.c
===================================================================
--- linux-mm.orig/arch/powerpc/sysdev/mpic.c	2006-06-16 22:20:49.000000000 -0400
+++ linux-mm/arch/powerpc/sysdev/mpic.c	2006-06-16 22:20:57.000000000 -0400
@@ -250,7 +250,7 @@
 	for (pos = readb(devbase + PCI_CAPABILITY_LIST); pos != 0;
 	     pos = readb(devbase + pos + PCI_CAP_LIST_NEXT)) {
 		u8 id = readb(devbase + pos + PCI_CAP_LIST_ID);
-		if (id == PCI_CAP_ID_HT_IRQCONF) {
+		if (id == PCI_CAP_ID_HT) {
 			id = readb(devbase + pos + 3);
 			if (id == 0x80)
 				break;
Index: linux-mm/include/linux/pci_regs.h
===================================================================
--- linux-mm.orig/include/linux/pci_regs.h	2006-06-16 22:21:33.000000000 -0400
+++ linux-mm/include/linux/pci_regs.h	2006-06-16 22:22:00.000000000 -0400
@@ -196,7 +196,7 @@
 #define  PCI_CAP_ID_MSI		0x05	/* Message Signalled Interrupts */
 #define  PCI_CAP_ID_CHSWP	0x06	/* CompactPCI HotSwap */
 #define  PCI_CAP_ID_PCIX	0x07	/* PCI-X */
-#define  PCI_CAP_ID_HT_IRQCONF	0x08	/* HyperTransport IRQ Configuration */
+#define  PCI_CAP_ID_HT		0x08	/* HyperTransport capability */
 #define  PCI_CAP_ID_VNDR	0x09	/* Vendor specific capability */
 #define  PCI_CAP_ID_SHPC 	0x0C	/* PCI Standard Hot-Plug Controller */
 #define  PCI_CAP_ID_EXP 	0x10	/* PCI Express */

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Matthew Wilcox]

On Fri, Jun 16, 2006 at 11:01:46PM -0400, Brice Goglin wrote:
> Several chipsets are known to not support MSI. Some support MSI but
> disable it by default. Thus, several drivers implement their own way to
> detect whether MSI works.

Yes, and that needs to go away.  To be fair, we're in the early stages
of introducing generic MSI support, so it's understandable that people
have made expedient rather than architectural solutions to problems.

> We introduce whitelisting of chipsets that are known to support MSI and
> keep the existing backlisting to disable MSI for other chipsets. When it
> is unknown whether the root chipset support MSI or not, we disable MSI
> by default except if pci=forcemsi was passed.

I think that's a bad idea.  Blacklisting is the better idea in the long-term.

> Bus flags inheritance is dropped since it has been reported to be broken.

I must have missed that report.  Please elucidate.

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Matthew Wilcox wrote:
>> We introduce whitelisting of chipsets that are known to support MSI and
>> keep the existing backlisting to disable MSI for other chipsets. When it
>> is unknown whether the root chipset support MSI or not, we disable MSI
>> by default except if pci=forcemsi was passed.
>
> I think that's a bad idea. Blacklisting is the better idea in the
> long-term.

In this case, we absolutely need an exhaustive list of chipsets that do
not support MSI. And it looks like there are a lot of them in the
non-PCI-E world. Our concern is that trying to enable MSI when it does
not work breaks drivers. While disabling MSI generally keeps drivers
working (except for instance for the Infinipath hardware, but our
whitelisting of the nVidia chipset might help them :)).
For the long term, I hope we'll end up having an exhaustive list of
chipsets that do not support MSI. But in the meantime, our "whitelisting
+ disable when we don't know" might help getting MSI as much as possible
without breaking to many things.

>
>> Bus flags inheritance is dropped since it has been reported to be broken.
>
> I must have missed that report. Please elucidate.
>

It is based on the following thread:
    http://www.ussg.iu.edu/hypermail/linux/kernel/0605.2/0929.html

The amd8131 MSI quirk cannot be EARLY or HEADER because it would be
called before dev->subordinate has been set (then, PCI_BUS_FLAGS_NO_MSI
could not be set in the bus flags). But, flags are inherited while the
PCI hierarchy is scanned, which means the quirk has to be EARLY or HEADER.
To solve this issue, we would need a quirk that is called while the PCI
hierarchy is scanned (ie, before FINAL or ENABLE) and after the pci
child bus of the bridge is created (ie, after EARLY and HEADER). But I
am not sure whether adding a new quirk type for this is a good idea.

Since the MSI is always actually processed on the root chipset, we know
we have to check there. We thought that inheriting bus flags was not
really required. That's why my patch finds the root chipset and checks
bus flags there.

Brice

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Greg KH]

On Fri, Jun 16, 2006 at 11:01:46PM -0400, Brice Goglin wrote:
> Several chipsets are known to not support MSI. Some support MSI but
> disable it by default. Thus, several drivers implement their own way to
> detect whether MSI works.
> 
> We introduce whitelisting of chipsets that are known to support MSI and
> keep the existing backlisting to disable MSI for other chipsets. When it
> is unknown whether the root chipset support MSI or not, we disable MSI
> by default except if pci=forcemsi was passed.
> 
> Whitelisting is done by setting a new PCI_BUS_FLAGS_MSI in the chipset
> subordinate bus. pci_enable_msi() thus starts by checking whether the
> root chipset of the device has the MSI or NOMSI flag set.

Whitelisting looks all well and good today, and maybe for the rest of
the year.  But what about 3 years from now when everyone has shaken all
of the MSI bugs out of their chipsets finally?  Do you really want to
add a new quirk for _every_ new chipset that comes out?  I don't think
that it is managable over the long run.

I do like your checks to see if MSI is able to be enabled or not, and
maybe we can just invert them to mark those chips that do not support
MSI today?

> I do not split this patch in multiple pieces since it is still small
> and this is just a RFC. I will split it for the actual submission.

Please do, you are doing a lot of different things in here.

thanks,

greg k-h

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Greg KH wrote:
> Whitelisting looks all well and good today, and maybe for the rest of
> the year.  But what about 3 years from now when everyone has shaken all
> of the MSI bugs out of their chipsets finally?  Do you really want to
> add a new quirk for _every_ new chipset that comes out?  I don't think
> that it is managable over the long run.
>   

We could still reverse the default. Right now, unless pci=forcemsi is
passed, I disable MSI if we don't know whether the chipset supports it.
Once blacklisting has been improved/completed, we can enable MSI by
default (and keep "pci=nomsi" in case a non-blacklisted chipset appears).

Or we could enable MSI by default on PCI-E chipsets and disable by
default on non-PCI-E (ie we whitelist non-PCI-E only) ? PCI-E chipsets
seem to support MSI pretty well.

I am ok with any strategy as long as I don't end up passing "pci=nomsi"
on most new machines I will install in the next 10 years (as I did with
"noapic" in the past) :)

Brice

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Andi Kleen]

Brice Goglin <brice@myri.com> writes:
> 
> Or we could enable MSI by default on PCI-E chipsets and disable by
> default on non-PCI-E (ie we whitelist non-PCI-E only) ? PCI-E chipsets
> seem to support MSI pretty well.

It looks like at least Serverworks HT1000 has trouble with MSI
too, but it's PCI-Express. But I guess those can be black listed

Also I think Intel has supported it well for a long time so might
want to white list all from VENDOR == Intel.

Blacklisting all old non PCI-E bridges non Intel seems reasonable

It seems AMD 8132 can be made to work, but it needs a special
quirk too and then it can be white listed.

The rules will be relatively complicated I guess, but should
be doable.

-Andi

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Jeff Garzik]

Greg KH wrote:
> On Fri, Jun 16, 2006 at 11:01:46PM -0400, Brice Goglin wrote:
>> Several chipsets are known to not support MSI. Some support MSI but
>> disable it by default. Thus, several drivers implement their own way to
>> detect whether MSI works.
>>
>> We introduce whitelisting of chipsets that are known to support MSI and
>> keep the existing backlisting to disable MSI for other chipsets. When it
>> is unknown whether the root chipset support MSI or not, we disable MSI
>> by default except if pci=forcemsi was passed.
>>
>> Whitelisting is done by setting a new PCI_BUS_FLAGS_MSI in the chipset
>> subordinate bus. pci_enable_msi() thus starts by checking whether the
>> root chipset of the device has the MSI or NOMSI flag set.
> 
> Whitelisting looks all well and good today, and maybe for the rest of
> the year.  But what about 3 years from now when everyone has shaken all
> of the MSI bugs out of their chipsets finally?  Do you really want to
> add a new quirk for _every_ new chipset that comes out?  I don't think
> that it is managable over the long run.
> 
> I do like your checks to see if MSI is able to be enabled or not, and
> maybe we can just invert them to mark those chips that do not support
> MSI today?

My gut feeling is:

blacklist -> any Intel machines which fail (most work)
blacklist -> any PCI Express which fails (most should work)
whitelist -> any other situation which works

Regards,

	Jeff

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Andi Kleen wrote:
> Brice Goglin <brice@myri.com> writes:
>   
>> Or we could enable MSI by default on PCI-E chipsets and disable by
>> default on non-PCI-E (ie we whitelist non-PCI-E only) ? PCI-E chipsets
>> seem to support MSI pretty well.
>>     
>
> It looks like at least Serverworks HT1000 has trouble with MSI
> too, but it's PCI-Express. But I guess those can be black listed
>   

IIRC, HT1000 is the southbridge part of the HT2000 chipset. We have been
told that MSI works on this chipset. And from what we've seen/tested, it
is true. The problem is that MSI is often disabled by the BIOS. My
hypertransport MSI capability quirk should check it right.

Brice

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Alistair John Strachan]

On Saturday 17 June 2006 04:01, Brice Goglin wrote:
> Several chipsets are known to not support MSI. Some support MSI but
> disable it by default. Thus, several drivers implement their own way to
> detect whether MSI works.
>
> We introduce whitelisting of chipsets that are known to support MSI and
> keep the existing backlisting to disable MSI for other chipsets. When it
> is unknown whether the root chipset support MSI or not, we disable MSI
> by default except if pci=forcemsi was passed.

I tried your patch on 2.6.17-rc6 (fixed up a view rejects) in an attempt to 
get MSI working with my forcedeth NIC, which advertises the capability. The 
kernel now gives me the following:

PCI: Found MSI HT mapping on 0000:00:0b.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0b.0 subordinate 
bus.
PCI: Found MSI HT mapping on 0000:00:0c.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0c.0 subordinate 
bus.
PCI: Found MSI HT mapping on 0000:00:0d.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0d.0 subordinate 
bus.
PCI: Found MSI HT mapping on 0000:00:0e.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0e.0 subordinate 
bus.
PCI: Setting latency timer of device 0000:00:0b.0 to 64
pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0b.0:pcie00]
Allocate Port Service[0000:00:0b.0:pcie03]
PCI: Setting latency timer of device 0000:00:0c.0 to 64
pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0c.0:pcie00]
Allocate Port Service[0000:00:0c.0:pcie03]
PCI: Setting latency timer of device 0000:00:0d.0 to 64
pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0d.0:pcie00]
Allocate Port Service[0000:00:0d.0:pcie03]
PCI: Setting latency timer of device 0000:00:0e.0 to 64
pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0e.0:pcie00]
Allocate Port Service[0000:00:0e.0:pcie03]

Is the "vendor BIOS" causing the problem? Here's my /proc/interrupts after 
booting.

           CPU0       CPU1
  0:     574494          0    IO-APIC-edge  timer
  8:          0          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 15:       4891          0    IO-APIC-edge  ide1
 50:          0          0   IO-APIC-level  NVidia CK804
 58:          4          0   IO-APIC-level  ehci_hcd:usb1
 66:       6203          0   IO-APIC-level  ohci1394, wlan
 74:          3          0   IO-APIC-level  ohci1394
217:      21315          0   IO-APIC-level  libata, ohci_hcd:usb2
225:       1031          0   IO-APIC-level  libata
233:      28124          0   IO-APIC-level  EMU10K1, skge
NMI:        171         90
LOC:     574330     574263
ERR:          0
MIS:          0

It's an x86-64 kernel.

-- 
Cheers,
Alistair.

Third year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Alistair John Strachan wrote:
> On Saturday 17 June 2006 04:01, Brice Goglin wrote:
>   
> I tried your patch on 2.6.17-rc6 (fixed up a view rejects) in an attempt to 
> get MSI working with my forcedeth NIC, which advertises the capability. The 
> kernel now gives me the following:
>   

Well, the aim of my patch is to stop enabling MSI by default, and try to
find out when we can and when we cannot use MSI. 2.6.17 will only
disable MSI on a few chipsets. My patch will disable on all chipsets but
Intel and those having Hypertransport MSI capabilities (Serverworks
HT2000 and nVidia CK804). Thus, you cannot expect MSI to be enabled with
my patch if it was not before :/

> PCI: Found MSI HT mapping on 0000:00:0b.0
> PCI: Found MSI HT mapping on 0000:00:00.0
> PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0b.0 subordinate 
> bus.
> PCI: Found MSI HT mapping on 0000:00:0c.0
> PCI: Found MSI HT mapping on 0000:00:00.0
> PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0c.0 subordinate 
> bus.
> PCI: Found MSI HT mapping on 0000:00:0d.0
> PCI: Found MSI HT mapping on 0000:00:00.0
> PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0d.0 subordinate 
> bus.
> PCI: Found MSI HT mapping on 0000:00:0e.0
> PCI: Found MSI HT mapping on 0000:00:00.0
> PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0e.0 subordinate 
> bus.
>   

If your board is behind one of these bridges, MSI could work (looks like
you have 4 nVidia CK804 with MSI enabled in the Hypertransport mapping).


> pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS
> assign_interrupt_mode Found MSI capability
> Allocate Port Service[0000:00:0b.0:pcie00]
> Allocate Port Service[0000:00:0b.0:pcie03]
>   

It might be the reason why MSI does not work. But I am not familiar with
this code, so I can't help, sorry.

Brice

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Andi Kleen]

[you destroyed the cc list. Don't do that]

On Monday 19 June 2006 02:53, Greg Lindahl wrote:
> On Sat, Jun 17, 2006 at 10:48:18AM -0400, Brice Goglin wrote:
> > IIRC, HT1000 is the southbridge part of the HT2000 chipset. We have been
> > told that MSI works on this chipset. And from what we've seen/tested, it
> > is true.
>
> We can also say this is true -- our InfiniPath HCA requires MSI, so we
> really notice this.

Isn't your HCA directly connected to HTX? If yes it will
likely not run into PCI bridge bugs.

We got a Serverworks based Supermicro system where the driver for the 
integrated tg3 NIC complains about MSI not working. So either that particular
system has a specific BIOS issue or it is broken for on motherboard
devices.
 
I was extrapolating from that.

It doesn't complain on another Supermicro system, but that seems
to be because that particular BCM570x silicon revision is blacklisted
for MSI in the tg3 driver.

>From that experience I certainly cannot say that MSI works 
very well on Serverworks so far. It might be safer to blacklist
until someone can explain what's going on on these systems.
The problem is that not all drivers do the MSI probing tg3 do,
so if you got a system where MSI doesn't work. but it tells
the kernel it does, and a driver turns on MSI things just
break.

For me so far it looks like MSI is another HPET or mmconfig - something
that looks nice in the specification, but hardware/BIOS developers
don't seem to pay particular attention to and is ridden with 
platform bugs.

-Andi

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Xavier Bestel]

On Sat, 2006-06-17 at 07:05, Matthew Wilcox wrote:
> On Fri, Jun 16, 2006 at 11:01:46PM -0400, Brice Goglin wrote:
> > We introduce whitelisting of chipsets that are known to support MSI and
> > keep the existing backlisting to disable MSI for other chipsets. When it
> > is unknown whether the root chipset support MSI or not, we disable MSI
> > by default except if pci=forcemsi was passed.
> 
> I think that's a bad idea.  Blacklisting is the better idea in the long-term.

I think the option adopted elsewhere is: whitelist for old chipsets, and
blacklist for new chipsets. You just have to decide for a good date to
separate "old" and "new" to minimize the lists size.

	Xav

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Andi Kleen wrote:
> We got a Serverworks based Supermicro system where the driver for the 
> integrated tg3 NIC complains about MSI not working. So either that particular
> system has a specific BIOS issue or it is broken for on motherboard
> devices.
>  
> I was extrapolating from that.
>
> It doesn't complain on another Supermicro system, but that seems
> to be because that particular BCM570x silicon revision is blacklisted
> for MSI in the tg3 driver.
>
> From that experience I certainly cannot say that MSI works 
> very well on Serverworks so far. It might be safer to blacklist
> until someone can explain what's going on on these systems.
> The problem is that not all drivers do the MSI probing tg3 do,
> so if you got a system where MSI doesn't work. but it tells
> the kernel it does, and a driver turns on MSI things just
> break.
>   

Initially, we thought MSI did not work. But, we actually discovered that
it was caused by the MSI cap being disabled in the Hypertransport config
space. Enabling the MSI cap there seems to make it work.

My patches check the HT MSI cap in a quirk. So you will only get MSI for
your driver if the BIOS/chipset has been correctly initialized.

The next step would be to explicitely set the MSI cap in the
Hypertransport config space in my quirk. I have patches to do so, but
I'd rather wait until my current patches to are stabilized/merged.

Brice

Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Grant Grundler]

On Mon, Jun 19, 2006 at 01:19:55PM +0200, Xavier Bestel wrote:
> On Sat, 2006-06-17 at 07:05, Matthew Wilcox wrote:
> > On Fri, Jun 16, 2006 at 11:01:46PM -0400, Brice Goglin wrote:
> > > We introduce whitelisting of chipsets that are known to support MSI and
> > > keep the existing backlisting to disable MSI for other chipsets. When it
> > > is unknown whether the root chipset support MSI or not, we disable MSI
> > > by default except if pci=forcemsi was passed.
> > 
> > I think that's a bad idea.  Blacklisting is the better idea in the long-term.
> 
> I think the option adopted elsewhere is: whitelist for old chipsets, and
> blacklist for new chipsets. You just have to decide for a good date to
> separate "old" and "new" to minimize the lists size.

I agree with willy.

White lists work "well" only if one's goal is to reduce the number
of bug reports about _all_ HW. Most folks with _working_ MSI (but not
already on the whitelist) won't know they could report this as a bug.
Ie these chipsets likely won't ever get added to the whitelist.

thanks,
grant

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Greg Lindahl]

On Mon, Jun 19, 2006 at 10:28:50AM +0200, Andi Kleen wrote:

> Isn't your HCA directly connected to HTX? If yes it will
> likely not run into PCI bridge bugs.

I was referring to our new PCI Express HCA.

-- greg

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Dave Olson]

On Mon, 19 Jun 2006, Andi Kleen wrote:

| [you destroyed the cc list. Don't do that]
| 
| On Monday 19 June 2006 02:53, Greg Lindahl wrote:
| > On Sat, Jun 17, 2006 at 10:48:18AM -0400, Brice Goglin wrote:
| > > IIRC, HT1000 is the southbridge part of the HT2000 chipset. We have been
| > > told that MSI works on this chipset. And from what we've seen/tested, it
| > > is true.
| >
| > We can also say this is true -- our InfiniPath HCA requires MSI, so we
| > really notice this.
| 
| Isn't your HCA directly connected to HTX? If yes it will
| likely not run into PCI bridge bugs.

The HyperTransport is directly attached, but it still uses parts of
the MSI infrastructure.   Mostly we've direct-coded it, but at some
point the MSI code should be expanded to support HTX devices as well.

| We got a Serverworks based Supermicro system where the driver for the 
| integrated tg3 NIC complains about MSI not working. So either that particular
| system has a specific BIOS issue or it is broken for on motherboard
| devices.
|  
| I was extrapolating from that.

I've used our PCIe card on the serverworks HT2000/ST2000 chipset with MSI, and it seems
to work fine (once you enable MSI in the config space, which the current
BIOS doesn't seem to do for some reason).  It may be that it doesn't work
well with PCI, but does with PCIe (different parts of the chipset, after all,
so it seems possible).

| For me so far it looks like MSI is another HPET or mmconfig - something
| that looks nice in the specification, but hardware/BIOS developers
| don't seem to pay particular attention to and is ridden with 
| platform bugs.

Sure, that's true of almost everything new.   It remains broken until people
start using it, complain, and get the bugs fixed.   Some of us have a vested
interest in having MSI work, since it's the only way we can deliver interrupts.
We've already worked with a few BIOS writers to get problems fixed, and we'll
keep doing so as we find problems.   If enough hardware vendors and consumers
do so, the broken stuff will get fixed, and stay fixed, because it will get
tested.


Dave Olson
olson@unixfolk.com
http://www.unixfolk.com/dave

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Andi Kleen]

> Sure, that's true of almost everything new.   It remains broken until people
> start using it, complain, and get the bugs fixed.   Some of us have a vested
> interest in having MSI work, since it's the only way we can deliver interrupts.
> We've already worked with a few BIOS writers to get problems fixed, and we'll
> keep doing so as we find problems.   If enough hardware vendors and consumers
> do so, the broken stuff will get fixed, and stay fixed, because it will get
> tested.

Sometimes there are new things that work relatively well and only break occassionally
and then there are things where it seems to be the other way round.

My point was basically that every time we tried to turn on such a banana green feature
without white listing it was a sea of pain to get it to work everywhere
and tended to cause far too many non boots

(and any non boot is far worse than whatever performance advantage you get
from it) 

So if there are any more MSI problems comming up IMHO it should be white list/disabled 
by default and only turn on after a long time when Windows uses it by default 
or something. Greg, do you agree?

-Andi

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Jeff Garzik]

Andi Kleen wrote:
> So if there are any more MSI problems comming up IMHO it should be white list/disabled 
> by default and only turn on after a long time when Windows uses it by default 
> or something. Greg, do you agree?


We should be optimists, not pessimists.

MSI is useful enough that we should turn it on by default in newer systems.

	Jeff

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Andi Kleen]

On Tuesday 20 June 2006 10:02, Jeff Garzik wrote:
> Andi Kleen wrote:
> > So if there are any more MSI problems comming up IMHO it should be white list/disabled 
> > by default and only turn on after a long time when Windows uses it by default 
> > or something. Greg, do you agree?
> 
> 
> We should be optimists, not pessimists.

Yes, booting on all systems is overrated anyways, isn't it?

> 
> MSI is useful enough that we should turn it on by default in newer systems.

That is what we've tried so far and it seems to not work.

-Andi

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Jeff Garzik]

Andi Kleen wrote:
> On Tuesday 20 June 2006 10:02, Jeff Garzik wrote:
>> Andi Kleen wrote:
>>> So if there are any more MSI problems comming up IMHO it should be white list/disabled 
>>> by default and only turn on after a long time when Windows uses it by default 
>>> or something. Greg, do you agree?
>>
>> We should be optimists, not pessimists.
> 
> Yes, booting on all systems is overrated anyways, isn't it?

Don't be silly.  Whatever solution is arrived at will boot on all 
systems.  That's an obvious operational requirement.

This is how new technology always works in Linux.  We turn it on and see 
what works, and what doesn't.  And whether existing problems will 
disappear.  With MSI, I think we see them disappearing.

Newer systems seem to be doing better with MSI, in part because 
PCI-Express and other technologies trend towards MSI-style operation.

And the kernel's MSI code is finally getting cleaned up, and getting the 
attention it needs.


>> MSI is useful enough that we should turn it on by default in newer systems.
> 
> That is what we've tried so far and it seems to not work.

IMO that's an exaggeration.  On 50% of the x86-64 platforms (Intel), MSI 
has been working for quite some time.  On newer systems in the other 
half of the platforms, MSI seems be more usable than it has been in the 
past.

	Jeff

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Greg Lindahl]

On Tue, Jun 20, 2006 at 09:25:30AM +0200, Andi Kleen wrote:

> So if there are any more MSI problems comming up IMHO it should be white list/disabled 
> by default and only turn on after a long time when Windows uses it by default 
> or something. Greg, do you agree?

You probably meant the other Greg, but the situation doesn't seem to
be that bad. Brice's proposed whitelist covers almost all Opteron PCI
Express servers.

What is the list of things which are known to have problems? All
PCI-X?  We can ask some more people with PCI-X MSI cards what works
for them, i.e.  Mellanox.

-- greg

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Randy.Dunlap]

On Tue, 20 Jun 2006 13:03:52 -0700 Greg Lindahl wrote:

> On Tue, Jun 20, 2006 at 09:25:30AM +0200, Andi Kleen wrote:
> 
> > So if there are any more MSI problems comming up IMHO it should be white list/disabled 
> > by default and only turn on after a long time when Windows uses it by default 
> > or something. Greg, do you agree?
> 
> You probably meant the other Greg, but the situation doesn't seem to
> be that bad. Brice's proposed whitelist covers almost all Opteron PCI
> Express servers.

Why "almost"?  What does a user do if his/hers is not covered?
Does it cover the 10 new models that are available tomorrow?
(hypothetical question)

> What is the list of things which are known to have problems? All
> PCI-X?  We can ask some more people with PCI-X MSI cards what works
> for them, i.e.  Mellanox.

---
~Randy

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Roland Dreier]

    Greg> What is the list of things which are known to have problems?
    Greg> All PCI-X?  We can ask some more people with PCI-X MSI cards
    Greg> what works for them, i.e.  Mellanox.

Actually most PCI-X works as well.  AMD 8131 PCI-X bridges don't work,
and it seems (based on the quirk in the kernel) that there are broken
Serverworks chipsets, but I've never actually seen motherboards with
that on there.

But for example I have an old Xeon motherboard which lspci says has

    PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 03)

and MSI works fine there.

There seem to be two issues here though.  First, MSI interrupts don't
always work, because of chipset bugs, BIOS bugs, etc.  This is fairly
manageable because the worst case is usually a single device not
generating interrupts.

However, the other issue is that CONFIG_PCI_MSI forces some other
changes to x86 interrupt handling, even if no devices will ever use
MSI.  And the changes are such that some systems can't even boot with
CONFIG_PCI_MSI enabled.  This is the more severe problem, which needs
to be handled if you want distros to turn on MSI.

 - R.

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Randy.Dunlap wrote:
> On Tue, 20 Jun 2006 13:03:52 -0700 Greg Lindahl wrote:
>   
>> You probably meant the other Greg, but the situation doesn't seem to
>> be that bad. Brice's proposed whitelist covers almost all Opteron PCI
>> Express servers.
>>     
>
> Why "almost"?  What does a user do if his/hers is not covered?
> Does it cover the 10 new models that are available tomorrow?
> (hypothetical question)
>   

At least, it will not be worse than currently for new PCI-E chipsets (my
patch still enables MSI by default for these).

But, my quirks to check the MSI cap in the HT mapping might need to be
enabled for more chipsets (I only handle nVidia Ck804 and ServerWorks
HT2000).

Brice

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Greg Lindahl]

On Tue, Jun 20, 2006 at 01:20:49PM -0700, Randy.Dunlap wrote:

> Does it cover the 10 new models that are available tomorrow?
> (hypothetical question)

I see several people on this list who are aware of forthcoming PCI
Express server products, and they aren't objecting to broad
whitelisting of PCI Express.

Is there any known case of PCI Express and MSI not working on any
chipset? Andi, is the tg3 NIC that didn't work in a Supermicro system
on PCI-X or PCI Express?

-- greg

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Greg Lindahl wrote:
> Andi, is the tg3 NIC that didn't work in a Supermicro system
> on PCI-X or PCI Express?
>   

IIRC, Andi was talking about a Supermicro machine with a ServerWorks
HT2000 chipset. We have such a machine here. Its MSI is disabled in the
Hyper-transport mapping. But, MSI works once the HT capability is
enabled (and my quirk will detect it right).
For such machines, if people really want MSI, we'll need to enable the
HT cap in my quirk. But, as long as they just want IRQ to work,
detecting whether the HT cap is enabled or not should be enough.

Brice

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Jeff Garzik]

Brice Goglin wrote:
> Greg Lindahl wrote:
>> Andi, is the tg3 NIC that didn't work in a Supermicro system
>> on PCI-X or PCI Express?
>>   
> 
> IIRC, Andi was talking about a Supermicro machine with a ServerWorks
> HT2000 chipset. We have such a machine here. Its MSI is disabled in the
> Hyper-transport mapping. But, MSI works once the HT capability is
> enabled (and my quirk will detect it right).
> For such machines, if people really want MSI, we'll need to enable the
> HT cap in my quirk. But, as long as they just want IRQ to work,
> detecting whether the HT cap is enabled or not should be enough.

If it works reliably, we should definitely turn it on.

	Jeff, wishing his HT1000 did the same

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Jeff Garzik]

Roland Dreier wrote:
> However, the other issue is that CONFIG_PCI_MSI forces some other
> changes to x86 interrupt handling, even if no devices will ever use
> MSI.  And the changes are such that some systems can't even boot with
> CONFIG_PCI_MSI enabled.  This is the more severe problem, which needs
> to be handled if you want distros to turn on MSI.

Agreed.  We definitely want CONFIG_PCI_MSI-enabled kernels to work in 
all situations.

	Jeff

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Dave Olson]

On Tue, 20 Jun 2006, Brice Goglin wrote:

| Greg Lindahl wrote:
| > Andi, is the tg3 NIC that didn't work in a Supermicro system
| > on PCI-X or PCI Express?
| >   
| 
| IIRC, Andi was talking about a Supermicro machine with a ServerWorks
| HT2000 chipset. We have such a machine here. Its MSI is disabled in the
| Hyper-transport mapping. But, MSI works once the HT capability is
| enabled (and my quirk will detect it right).
| For such machines, if people really want MSI, we'll need to enable the
| HT cap in my quirk. But, as long as they just want IRQ to work,
| detecting whether the HT cap is enabled or not should be enough.

We definitely want that chipset to be on the whitelist.   It's used in
supermicro systems, the Mac systems with the 970 chip, and some large
vendor systems as well.

The detection of MSI not being enabled in the config space is 
a good thing to do (and warn about), of course, no problem with that.

I've seen no MSI-related issues with the ST2000 on 3 different
platforms, so far.

Dave Olson
olson@unixfolk.com

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Greg KH]

On Tue, Jun 20, 2006 at 09:25:30AM +0200, Andi Kleen wrote:
> 
> > Sure, that's true of almost everything new.   It remains broken until people
> > start using it, complain, and get the bugs fixed.   Some of us have a vested
> > interest in having MSI work, since it's the only way we can deliver interrupts.
> > We've already worked with a few BIOS writers to get problems fixed, and we'll
> > keep doing so as we find problems.   If enough hardware vendors and consumers
> > do so, the broken stuff will get fixed, and stay fixed, because it will get
> > tested.
> 
> Sometimes there are new things that work relatively well and only break occassionally
> and then there are things where it seems to be the other way round.
> 
> My point was basically that every time we tried to turn on such a banana green feature
> without white listing it was a sea of pain to get it to work everywhere
> and tended to cause far too many non boots
> 
> (and any non boot is far worse than whatever performance advantage you get
> from it) 
> 
> So if there are any more MSI problems comming up IMHO it should be
> white list/disabled by default and only turn on after a long time when
> Windows uses it by default or something. Greg, do you agree?

No, I don't want a whitelist, as it will be hard to always keep adding
stuff to it (unless we can somehow figure out how to put a "cut-off"
date check in there).  Yes, we do have a number of systems today that
have MSI issues, but the newer ones all work properly, and we should
continue on with the way we have today (blasklist problem boards, as the
rest of the PCI subsystem works with the quirks).

thanks,

greg "the optimist" k-h

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Greg KH wrote:
> No, I don't want a whitelist, as it will be hard to always keep adding
> stuff to it (unless we can somehow figure out how to put a "cut-off"
> date check in there).

My second patchset (Improve MSI detection v2) uses "PCI-E vs non-PCI-E"
as a cut-off "date". After reading all what people said in this thread,
I still think it is a good compromise (and very simple to implement) if
we blacklist PCI-E and whitelist non-PCI-E chipsets.

Brice

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Eric W. Biederman]

Jeff Garzik <jeff@garzik.org> writes:

> Andi Kleen wrote:
>> On Tuesday 20 June 2006 10:02, Jeff Garzik wrote:
>>> Andi Kleen wrote:
>>>> So if there are any more MSI problems comming up IMHO it should be white
>>>> list/disabled by default and only turn on after a long time when Windows
>>>> uses it by default or something. Greg, do you agree?
>>>
>>> We should be optimists, not pessimists.
>> Yes, booting on all systems is overrated anyways, isn't it?
>
> Don't be silly.  Whatever solution is arrived at will boot on all systems.
> That's an obvious operational requirement.
>
> This is how new technology always works in Linux.  We turn it on and see what
> works, and what doesn't.  And whether existing problems will disappear.  With
> MSI, I think we see them disappearing.
>
> Newer systems seem to be doing better with MSI, in part because PCI-Express and
> other technologies trend towards MSI-style operation.
>
> And the kernel's MSI code is finally getting cleaned up, and getting the
> attention it needs.
>
>
>>> MSI is useful enough that we should turn it on by default in newer systems.
>> That is what we've tried so far and it seems to not work.
>
> IMO that's an exaggeration.  On 50% of the x86-64 platforms (Intel), MSI has
> been working for quite some time.  On newer systems in the other half of the
> platforms, MSI seems be more usable than it has been in the past.

It would help if the msi code as more than a 3 year old proof of concept
that has intel assumptions all through it.  Things are getting better
but there are still issues there.

I suspect that on x86 hypertransport systems if we directed our writes
to 0xFDF8000000 instead of at 0xfee0000 we would have quite a bit more luck.
Hopefully I can test and confirm that soon.  It is still a trick to get
a card with a working MSI implementation.

My gut feel is what we want is not a while list or a black list but instead
a MSI window driver.  The location and the meaning of the window is on the
edge of being an architectural definition.  But it really isn't and we are
treating it like one.

So I think part of the reason we are having MSI is we are making unwarranted
assumptions.  Once we take a good look at our side and fix that then we
can tell if something was broken.

Eric

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Greg KH]

On Tue, Jun 20, 2006 at 06:27:45PM -0400, Brice Goglin wrote:
> Greg KH wrote:
> > No, I don't want a whitelist, as it will be hard to always keep adding
> > stuff to it (unless we can somehow figure out how to put a "cut-off"
> > date check in there).
> 
> My second patchset (Improve MSI detection v2) uses "PCI-E vs non-PCI-E"
> as a cut-off "date". After reading all what people said in this thread,
> I still think it is a good compromise (and very simple to implement) if
> we blacklist PCI-E and whitelist non-PCI-E chipsets.

No, that's not fair for those devices which do not have PCI-E and yet
have MSI (the original ones, that work just fine...)

Again, no "whitelist" please, just quirks to fix problems with ones that
we know we have problems with, just like all other PCI quirks...

thanks,

greg k-h

Re: [discuss] Re: [RFC] Whitelist chipsets supporting MSI and check Hyper-transport capabilities

[Brice Goglin]

Greg KH wrote:
> No, that's not fair for those devices which do not have PCI-E and yet
> have MSI (the original ones, that work just fine...)
>
> Again, no "whitelist" please, just quirks to fix problems with ones that
> we know we have problems with, just like all other PCI quirks...
>
> thanks,
>
> greg k-h
>   

Ok I will regenerate my patches to do so.

Brice