[PATCH v2 1/2] PCI: Add 'nodomains' boot option, and pci_domains_supported global

[PATCH v2 1/2] PCI: Add 'nodomains' boot option, and pci_domains_supported global

[Jeff Garzik]

* Introduce pci_domains_supported global, hardcoded to zero if
  !CONFIG_PCI_DOMAINS.

* Introduce 'nodomains' boot option, which clears pci_domains_supported
  on platforms that enable it by default (x86, x86-64, and others when
  they are converted to use this).

Signed-off-by: Jeff Garzik <jgarzik@redhat.com>
---
Revised per Andi's comments, and split into two patches.

As found in jgarzik/misc-2.6.git#ALL and jgarzik/misc-2.6.git#pciseg.

 Documentation/kernel-parameters.txt |    2 ++
 drivers/pci/pci.c                   |   13 +++++++++++++
 include/linux/pci.h                 |    7 +++++--
 3 files changed, 20 insertions(+), 2 deletions(-)

5d29ec863821020822c00384eca8b1006ab3c686
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 4d175c7..a4d43ae 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1269,6 +1269,8 @@ and is between 256 and 4096 characters. It is defined in the file
 				Mechanism 1.
 		conf2		[X86-32] Force use of PCI Configuration
 				Mechanism 2.
+		nodomains	[PCI] Disable support for multiple PCI
+				root domains (aka PCI segments, in ACPI-speak).
 		nommconf	[X86-32,X86_64] Disable use of MMCONFIG for PCI
 				Configuration
 		nomsi		[MSI] If the PCI_MSI kernel config parameter is
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 37c00f6..a82b4d1 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -22,6 +22,10 @@
 
 unsigned int pci_pm_d3_delay = 10;
 
+#ifdef CONFIG_PCI_DOMAINS
+int pci_domains_supported = 1;
+#endif
+
 #define DEFAULT_CARDBUS_IO_SIZE		(256)
 #define DEFAULT_CARDBUS_MEM_SIZE	(64*1024*1024)
 /* pci=cbmemsize=nnM,cbiosize=nn can override this */
@@ -1566,6 +1570,13 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)
 	return bars;
 }
 
+static void __devinit pci_no_domains(void)
+{
+#ifdef CONFIG_PCI_DOMAINS
+	pci_domains_supported = 0;
+#endif
+}
+
 static int __devinit pci_init(void)
 {
 	struct pci_dev *dev = NULL;
@@ -1585,6 +1596,8 @@ static int __devinit pci_setup(char *str)
 		if (*str && (str = pcibios_setup(str)) && *str) {
 			if (!strcmp(str, "nomsi")) {
 				pci_no_msi();
+			} else if (!strcmp(str, "nodomains")) {
+				pci_no_domains();
 			} else if (!strncmp(str, "cbiosize=", 9)) {
 				pci_cardbus_io_size = memparse(str + 9, &str);
 			} else if (!strncmp(str, "cbmemsize=", 10)) {
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 038a0dc..ee99959 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -685,13 +685,16 @@ extern void pci_unblock_user_cfg_access(struct pci_dev *dev);
  * a PCI domain is defined to be a set of PCI busses which share
  * configuration space.
  */
-#ifndef CONFIG_PCI_DOMAINS
+#ifdef CONFIG_PCI_DOMAINS
+extern int pci_domains_supported;
+#else
+enum { pci_domains_supported = 0; };
 static inline int pci_domain_nr(struct pci_bus *bus) { return 0; }
 static inline int pci_proc_domain(struct pci_bus *bus)
 {
 	return 0;
 }
-#endif
+#endif /* CONFIG_PCI_DOMAINS */
 
 #else /* CONFIG_PCI is not enabled */
 

[PATCH v2 2/2] X86: Introduce and enable PCI domain support

[Jeff Garzik]

* fix bug in pci_read() and pci_write() which prevented PCI domain
  support from working (hardcoded domain 0).

* unconditionally enable CONFIG_PCI_DOMAINS

* implement pci_domain_nr() and pci_proc_domain(), as required of
  all arches when CONFIG_PCI_DOMAINS is enabled.

* store domain in struct pci_sysdata, as assigned by ACPI

* support "pci=nodomains"

Signed-off-by: Jeff Garzik <jgarzik@redhat.com>
---
Revised per Andi's comments, and split into two patches.

As found in jgarzik/misc-2.6.git#ALL and jgarzik/misc-2.6.git#pciseg.

 arch/i386/Kconfig        |    5 +++++
 arch/i386/pci/acpi.c     |   13 +++++++------
 arch/i386/pci/common.c   |    6 ++++--
 arch/x86_64/Kconfig      |    5 +++++
 include/asm-i386/pci.h   |   12 ++++++++++++
 include/asm-x86_64/pci.h |   12 ++++++++++++
 6 files changed, 45 insertions(+), 8 deletions(-)

d2978874ffb1a980238ab68aa528b700a1ec2b30
diff --git a/arch/i386/Kconfig b/arch/i386/Kconfig
index 97b64d7..5237791 100644
--- a/arch/i386/Kconfig
+++ b/arch/i386/Kconfig
@@ -1137,6 +1137,11 @@ config PCI_MMCONFIG
 	depends on PCI && ACPI && (PCI_GOMMCONFIG || PCI_GOANY)
 	default y
 
+config PCI_DOMAINS
+	bool
+	depends on PCI
+	default y
+
 source "drivers/pci/pcie/Kconfig"
 
 source "drivers/pci/Kconfig"
diff --git a/arch/i386/pci/acpi.c b/arch/i386/pci/acpi.c
index bc8a44b..5c9827b 100644
--- a/arch/i386/pci/acpi.c
+++ b/arch/i386/pci/acpi.c
@@ -11,6 +11,12 @@ struct pci_bus * __devinit pci_acpi_scan_root(struct acpi_device *device, int do
 	struct pci_sysdata *sd;
 	int pxm;
 
+	if (domain && !pci_domains_supported) {
+		printk(KERN_WARNING "PCI: Multiple domains not supported "
+		       "(dom %d, bus %d)\n", domain, busnum);
+		return NULL;
+	}
+
 	/* Allocate per-root-bus (not per bus) arch-specific data.
 	 * TODO: leak; this memory is never freed.
 	 * It's arguable whether it's worth the trouble to care.
@@ -21,12 +27,7 @@ struct pci_bus * __devinit pci_acpi_scan_root(struct acpi_device *device, int do
 		return NULL;
 	}
 
-	if (domain != 0) {
-		printk(KERN_WARNING "PCI: Multiple domains not supported\n");
-		kfree(sd);
-		return NULL;
-	}
-
+	sd->domain = domain;
 	sd->node = -1;
 
 	pxm = acpi_get_pxm(device->handle);
diff --git a/arch/i386/pci/common.c b/arch/i386/pci/common.c
index ebc6f3c..6d7a274 100644
--- a/arch/i386/pci/common.c
+++ b/arch/i386/pci/common.c
@@ -29,12 +29,14 @@ struct pci_raw_ops *raw_pci_ops;
 
 static int pci_read(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *value)
 {
-	return raw_pci_ops->read(0, bus->number, devfn, where, size, value);
+	return raw_pci_ops->read(pci_domain_nr(bus), bus->number,
+				 devfn, where, size, value);
 }
 
 static int pci_write(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 value)
 {
-	return raw_pci_ops->write(0, bus->number, devfn, where, size, value);
+	return raw_pci_ops->write(pci_domain_nr(bus), bus->number,
+				  devfn, where, size, value);
 }
 
 struct pci_ops pci_root_ops = {
diff --git a/arch/x86_64/Kconfig b/arch/x86_64/Kconfig
index b4d9089..f3ea9e0 100644
--- a/arch/x86_64/Kconfig
+++ b/arch/x86_64/Kconfig
@@ -724,6 +724,11 @@ config PCI_MMCONFIG
 	bool "Support mmconfig PCI config space access"
 	depends on PCI && ACPI
 
+config PCI_DOMAINS
+	bool
+	depends on PCI
+	default y
+
 source "drivers/pci/pcie/Kconfig"
 
 source "drivers/pci/Kconfig"
diff --git a/include/asm-i386/pci.h b/include/asm-i386/pci.h
index 4fcacc7..0d91605 100644
--- a/include/asm-i386/pci.h
+++ b/include/asm-i386/pci.h
@@ -5,12 +5,24 @@
 #ifdef __KERNEL__
 
 struct pci_sysdata {
+	int		domain;		/* PCI domain */
 	int		node;		/* NUMA node */
 };
 
 /* scan a bus after allocating a pci_sysdata for it */
 extern struct pci_bus *pci_scan_bus_with_sysdata(int busno);
 
+static inline int pci_domain_nr(struct pci_bus *bus)
+{
+	struct pci_sysdata *sd = bus->sysdata;
+	return sd->domain;
+}
+
+static inline int pci_proc_domain(struct pci_bus *bus)
+{
+	return pci_domain_nr(bus);
+}
+
 #include <linux/mm.h>		/* for struct page */
 
 /* Can be used to override the logic in pci_scan_bus for skipping
diff --git a/include/asm-x86_64/pci.h b/include/asm-x86_64/pci.h
index 5da8cb0..0a123d6 100644
--- a/include/asm-x86_64/pci.h
+++ b/include/asm-x86_64/pci.h
@@ -6,12 +6,24 @@
 #ifdef __KERNEL__
 
 struct pci_sysdata {
+	int		domain;		/* PCI domain */
 	int		node;		/* NUMA node */
 	void*		iommu;		/* IOMMU private data */
 };
 
 extern struct pci_bus *pci_scan_bus_with_sysdata(int busno);
 
+static inline int pci_domain_nr(struct pci_bus *bus)
+{
+	struct pci_sysdata *sd = bus->sysdata;
+	return sd->domain;
+}
+
+static inline int pci_proc_domain(struct pci_bus *bus)
+{
+	return pci_domain_nr(bus);
+}
+
 #ifdef CONFIG_CALGARY_IOMMU
 static inline void* pci_iommu(struct pci_bus *bus)
 {

Re: [PATCH v2 2/2] X86: Introduce and enable PCI domain support

[Greg KH]

On Thu, Oct 11, 2007 at 04:58:30PM -0400, Jeff Garzik wrote:
> 
> * fix bug in pci_read() and pci_write() which prevented PCI domain
>   support from working (hardcoded domain 0).
> 
> * unconditionally enable CONFIG_PCI_DOMAINS
> 
> * implement pci_domain_nr() and pci_proc_domain(), as required of
>   all arches when CONFIG_PCI_DOMAINS is enabled.
> 
> * store domain in struct pci_sysdata, as assigned by ACPI
> 
> * support "pci=nodomains"
> 
> Signed-off-by: Jeff Garzik <jgarzik@redhat.com>
> ---
> Revised per Andi's comments, and split into two patches.

Ugh:
	In file included from include/linux/pci.h:785,
	                 from drivers/pci/access.c:2:
	include/asm/pci.h:16: error: redefinition of 'pci_domain_nr'
	include/linux/pci.h:692: error: previous definition of 'pci_domain_nr' was here
	include/asm/pci.h:22: error: redefinition of 'pci_proc_domain'
	include/linux/pci.h:694: error: previous definition of 'pci_proc_domain' was here
	make[1]: *** [drivers/pci/access.o] Error 1

I'll see if I can fix it up...

thanks,

greg k-h

Re: [PATCH v2 2/2] X86: Introduce and enable PCI domain support

[Greg KH]

On Thu, Oct 11, 2007 at 02:40:28PM -0700, Greg KH wrote:
> On Thu, Oct 11, 2007 at 04:58:30PM -0400, Jeff Garzik wrote:
> > 
> > * fix bug in pci_read() and pci_write() which prevented PCI domain
> >   support from working (hardcoded domain 0).
> > 
> > * unconditionally enable CONFIG_PCI_DOMAINS
> > 
> > * implement pci_domain_nr() and pci_proc_domain(), as required of
> >   all arches when CONFIG_PCI_DOMAINS is enabled.
> > 
> > * store domain in struct pci_sysdata, as assigned by ACPI
> > 
> > * support "pci=nodomains"
> > 
> > Signed-off-by: Jeff Garzik <jgarzik@redhat.com>
> > ---
> > Revised per Andi's comments, and split into two patches.
> 
> Ugh:
> 	In file included from include/linux/pci.h:785,
> 	                 from drivers/pci/access.c:2:
> 	include/asm/pci.h:16: error: redefinition of 'pci_domain_nr'
> 	include/linux/pci.h:692: error: previous definition of 'pci_domain_nr' was here
> 	include/asm/pci.h:22: error: redefinition of 'pci_proc_domain'
> 	include/linux/pci.h:694: error: previous definition of 'pci_proc_domain' was here
> 	make[1]: *** [drivers/pci/access.o] Error 1
> 
> I'll see if I can fix it up...

Hm, sorry, my fault, error with not upadating the config...

nevermind...

greg k-h

Re: [PATCH v2 1/2] PCI: Add 'nodomains' boot option, and pci_domains_supported global

[Jeff Garzik]

Jeff Garzik wrote:
> * Introduce pci_domains_supported global, hardcoded to zero if
>   !CONFIG_PCI_DOMAINS.
> 
> * Introduce 'nodomains' boot option, which clears pci_domains_supported
>   on platforms that enable it by default (x86, x86-64, and others when
>   they are converted to use this).
> 
> Signed-off-by: Jeff Garzik <jgarzik@redhat.com>
> ---
> Revised per Andi's comments, and split into two patches.
> 
> As found in jgarzik/misc-2.6.git#ALL and jgarzik/misc-2.6.git#pciseg.
> 
>  Documentation/kernel-parameters.txt |    2 ++
>  drivers/pci/pci.c                   |   13 +++++++++++++
>  include/linux/pci.h                 |    7 +++++--
>  3 files changed, 20 insertions(+), 2 deletions(-)

As a note, if this wasn't obvious, this was not implemented in 
x86-specific code because I think other arches can/should be able to 
hook into this.

The default !CONFIG_PCI_DOMAINS implementation continues to work for all 
such arches.

Re: [PATCH v2 1/2] PCI: Add 'nodomains' boot option, and pci_domains_supported global

[Greg KH]

On Thu, Oct 11, 2007 at 04:57:27PM -0400, Jeff Garzik wrote:
> +enum { pci_domains_supported = 0; };

I don't think you compiled this option :)

I've fixed the compiler error in my version.

thanks,

greg k-h

Re: [PATCH v2 1/2] PCI: Add 'nodomains' boot option, and pci_domains_supported global

[Jeff Garzik]

Greg KH wrote:
> On Thu, Oct 11, 2007 at 04:57:27PM -0400, Jeff Garzik wrote:
>> +enum { pci_domains_supported = 0; };
> 
> I don't think you compiled this option :)
> 
> I've fixed the compiler error in my version.

Thanks!