[PATCH] [3/3] PCI segment support

[PATCH] [3/3] PCI segment support

[Matthew Wilcox]

I'm putting this patch out for comments rather than merging, since I'm
aware that it won't apply to Linus' tree any more.

 - Add segment to pci_bus.
 - Change the sysfs name of each device to include a 16-bit segment ID.

diff -urpNX dontdiff linux-2.5.66/drivers/pci/probe.c linux-2.5.66-laptop/drivers/pci/probe.c
--- linux-2.5.66/drivers/pci/probe.c	2003-04-04 08:43:01.000000000 -0600
+++ linux-2.5.66-laptop/drivers/pci/probe.c	2003-04-04 14:36:37.000000000 -0600
@@ -510,7 +510,7 @@ pci_scan_device(struct pci_bus *bus, int
 	pci_name_device(dev);
 
 	/* now put in global tree */
-	strcpy(dev->dev.bus_id,dev->slot_name);
+	sprintf(dev->dev.bus_id, "%04x:%s", bus->segment, dev->slot_name);
 	dev->dev.dma_mask = &dev->dma_mask;
 
 	return dev;
diff -urpNX dontdiff linux-2.5.66/drivers/pcmcia/cardbus.c linux-2.5.66-laptop/drivers/pcmcia/cardbus.c
--- linux-2.5.66/drivers/pcmcia/cardbus.c	2003-04-04 08:43:59.000000000 -0600
+++ linux-2.5.66-laptop/drivers/pcmcia/cardbus.c	2003-04-04 14:36:36.000000000 -0600
@@ -282,7 +282,8 @@ int cb_alloc(socket_info_t * s)
 
 		pci_setup_device(dev);
 
-		strcpy(dev->dev.bus_id, dev->slot_name);
+		sprintf(dev->dev.bus_id, "%04x:%s", bus->segment,
+				dev->slot_name);
 
 		/* We need to assign resources for expansion ROM. */
 		for (r = 0; r < 7; r++) {
diff -urpNX dontdiff linux-2.5.66/include/linux/pci.h linux-2.5.66-laptop/include/linux/pci.h
--- linux-2.5.66/include/linux/pci.h	2003-04-04 08:43:21.000000000 -0600
+++ linux-2.5.66-laptop/include/linux/pci.h	2003-04-04 14:07:34.000000000 -0600
@@ -451,6 +451,7 @@ struct pci_bus {
 	void		*sysdata;	/* hook for sys-specific extension */
 	struct proc_dir_entry *procdir;	/* directory entry in /proc/bus/pci */
 
+	unsigned short	segment;	/* domain/segment number */
 	unsigned char	number;		/* bus number */
 	unsigned char	primary;	/* number of primary bridge */
 	unsigned char	secondary;	/* number of secondary bridge */

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

On Tue, Apr 08, 2003 at 12:44:11AM +0100, Matthew Wilcox wrote:
>  - Add segment to pci_bus.
>  - Change the sysfs name of each device to include a 16-bit segment ID.

First of all, the "segment" name is extremely misleading. PCI spec
assumes everywhere that "segment" is a group of devices sitting
on the same wires (ie primary and secondary buses of the PCI-to-PCI
bridge are *different* segments).
Let's call it "configuration space domain" instead.

Second, why not

-	strcpy(dev->dev.bus_id,dev->slot_name);
+	sprintf(dev->dev.bus_id, "%04x:%s", pci_controller_num(dev),
+		dev->slot_name);

?

Ivan.

Re: [PATCH] [3/3] PCI segment support

[Matthew Wilcox]

On Tue, Apr 08, 2003 at 08:38:24PM +0400, Ivan Kokshaysky wrote:
> On Tue, Apr 08, 2003 at 12:44:11AM +0100, Matthew Wilcox wrote:
> >  - Add segment to pci_bus.
> >  - Change the sysfs name of each device to include a 16-bit segment ID.
> 
> First of all, the "segment" name is extremely misleading. PCI spec
> assumes everywhere that "segment" is a group of devices sitting
> on the same wires (ie primary and secondary buses of the PCI-to-PCI
> bridge are *different* segments).

I don't mind changing it to `domain', I prefer the term myself.
ACPI calls it `_SEG' so I went with segment.

> Second, why not
> 
> -	strcpy(dev->dev.bus_id,dev->slot_name);
> +	sprintf(dev->dev.bus_id, "%04x:%s", pci_controller_num(dev),
> +		dev->slot_name);
> 
> ?

Because it's possible to have multiple pci root bridges in the same
pci domain.  This is true on at least HP's ia64 & parisc boxes.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

On Tue, Apr 08, 2003 at 05:50:26PM +0100, Matthew Wilcox wrote:
> Because it's possible to have multiple pci root bridges in the same
> pci domain.  This is true on at least HP's ia64 & parisc boxes.

I believe each PCI controller on these ia64/parisc boxes has its
own config space, and can support up to 256 bridged PCI buses, right?
Whether or not these PCI controllers share the same IO or MEM space is
irrelevant (because it's entirely implementation specific).

Ivan.

Re: [PATCH] [3/3] PCI segment support

[Matthew Wilcox]

On Tue, Apr 08, 2003 at 09:21:19PM +0400, Ivan Kokshaysky wrote:
> On Tue, Apr 08, 2003 at 05:50:26PM +0100, Matthew Wilcox wrote:
> > Because it's possible to have multiple pci root bridges in the same
> > pci domain.  This is true on at least HP's ia64 & parisc boxes.
> 
> I believe each PCI controller on these ia64/parisc boxes has its
> own config space, and can support up to 256 bridged PCI buses, right?
> Whether or not these PCI controllers share the same IO or MEM space is
> irrelevant (because it's entirely implementation specific).

I think hardware _could_ work like that, but it's never set up to work
like that in practice.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [PATCH] [3/3] PCI segment support

[Grant Grundler]

On Tue, Apr 08, 2003 at 06:31:09PM +0100, Matthew Wilcox wrote:
> On Tue, Apr 08, 2003 at 09:21:19PM +0400, Ivan Kokshaysky wrote:
...
> > I believe each PCI controller on these ia64/parisc boxes has its
> > own config space, and can support up to 256 bridged PCI buses, right?

yes - at least "HP Designed" chipsets do.

> > Whether or not these PCI controllers share the same IO or MEM space is
> > irrelevant (because it's entirely implementation specific).
> 
> I think hardware _could_ work like that, but it's never set up to work
> like that in practice.

I interpret "practice" as equivalent to "implementation" in this context.

I hedge by saying "HP designed" because HP tried to sell Intel itanic
boxes until current mckinley boxes became available. And SGI, IBM,
and other vendors are designing their own "high end" ia64 chipsets.
(eg NEC Azusa - 16 CPU itanic).

parisc firmware (both Legacy and PAT PDC) and current IA64 firmware number
the PCI busses.  I didn't want the parisc-linux implementation to
reprogram PCI Bus numbers because:
o lack of PCI Segment support (which I'm glad to see willy is fixing)
o the "view" of devices from both firmware and linux should match.

card-mode dino (PCI controller on GSC HBA) is the only exception where
firmware has no clue about the PCI controller.

hth,
grant

Re: [PATCH] [3/3] PCI segment support

[Matthew Wilcox]

Picking up this loose thread, here's a reimplementation of PCI domain
support.

 - Use pci_domain_nr() macro to determine which domain a bus or device
   is in.
 - Default implementation for architectures which don't support PCI
   domains.
 - Implementation for ia64.

I envisage ia64 will always turn on CONFIG_PCI_DOMAINS but x86 might
well have it as a user question.  I suspect most architectures would
never turn it on (yeah, I'm going to design an embedded ARM box with
multiple PCI domains.  sure.)

Index: arch/ia64/Kconfig
===================================================================
RCS file: /var/cvs/linux-2.5/arch/ia64/Kconfig,v
retrieving revision 1.12
diff -u -p -r1.12 Kconfig
--- arch/ia64/Kconfig	27 May 2003 17:21:18 -0000	1.12
+++ arch/ia64/Kconfig	8 Jun 2003 16:27:35 -0000
@@ -543,6 +543,10 @@ config PCI
 	  information about which PCI hardware does work under Linux and which
 	  doesn't.
 
+config PCI_DOMAINS
+	bool
+	default PCI
+
 source "drivers/pci/Kconfig"
 
 config HOTPLUG
Index: arch/ia64/hp/common/sba_iommu.c
===================================================================
RCS file: /var/cvs/linux-2.5/arch/ia64/hp/common/sba_iommu.c,v
retrieving revision 1.7
diff -u -p -r1.7 sba_iommu.c
--- arch/ia64/hp/common/sba_iommu.c	27 May 2003 17:21:18 -0000	1.7
+++ arch/ia64/hp/common/sba_iommu.c	8 Jun 2003 16:06:36 -0000
@@ -1889,7 +1889,7 @@ sba_connect_bus(struct pci_bus *bus)
 		handle = parent;
 	} while (ACPI_SUCCESS(status));
 
-	printk(KERN_WARNING "No IOC for PCI Bus %02x:%02x in ACPI\n", PCI_SEGMENT(bus), bus->number);
+	printk(KERN_WARNING "No IOC for PCI Bus %04x:%02x in ACPI\n", pci_domain_nr(bus), bus->number);
 }
 
 static int __init
Index: arch/ia64/pci/pci.c
===================================================================
RCS file: /var/cvs/linux-2.5/arch/ia64/pci/pci.c,v
retrieving revision 1.7
diff -u -p -r1.7 pci.c
--- arch/ia64/pci/pci.c	27 May 2003 17:21:22 -0000	1.7
+++ arch/ia64/pci/pci.c	8 Jun 2003 16:05:53 -0000
@@ -87,14 +87,14 @@ __pci_sal_write (int seg, int bus, int d
 static int
 pci_sal_read (struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *value)
 {
-	return __pci_sal_read(PCI_SEGMENT(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
+	return __pci_sal_read(pci_domain_nr(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
 			      where, size, value);
 }
 
 static int
 pci_sal_write (struct pci_bus *bus, unsigned int devfn, int where, int size, u32 value)
 {
-	return __pci_sal_write(PCI_SEGMENT(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
+	return __pci_sal_write(pci_domain_nr(bus), bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn),
 			       where, size, value);
 }
 
Index: drivers/pci/probe.c
===================================================================
RCS file: /var/cvs/linux-2.5/drivers/pci/probe.c,v
retrieving revision 1.14
diff -u -p -r1.14 probe.c
--- drivers/pci/probe.c	27 May 2003 17:25:03 -0000	1.14
+++ drivers/pci/probe.c	8 Jun 2003 16:36:59 -0000
@@ -528,7 +528,8 @@ pci_scan_device(struct pci_bus *bus, int
 	pci_name_device(dev);
 
 	/* now put in global tree */
-	strcpy(dev->dev.bus_id,dev->slot_name);
+	sprintf(dev->dev.bus_id, "%04x:%s", pci_domain_nr(bus),
+			dev->slot_name);
 	dev->dev.dma_mask = &dev->dma_mask;
 
 	return dev;
Index: include/asm-ia64/pci.h
===================================================================
RCS file: /var/cvs/linux-2.5/include/asm-ia64/pci.h,v
retrieving revision 1.7
diff -u -p -r1.7 pci.h
--- include/asm-ia64/pci.h	27 May 2003 17:28:04 -0000	1.7
+++ include/asm-ia64/pci.h	6 Jun 2003 19:57:29 -0000
@@ -95,7 +95,7 @@ struct pci_controller {
 };
 
 #define PCI_CONTROLLER(busdev) ((struct pci_controller *) busdev->sysdata)
-#define PCI_SEGMENT(busdev)    (PCI_CONTROLLER(busdev)->segment)
+#define pci_domain_nr(busdev)    (PCI_CONTROLLER(busdev)->segment)
 
 /* generic pci stuff */
 #include <asm-generic/pci.h>
Index: include/linux/pci.h
===================================================================
RCS file: /var/cvs/linux-2.5/include/linux/pci.h,v
retrieving revision 1.16
diff -u -p -r1.16 pci.h
--- include/linux/pci.h	27 May 2003 17:29:00 -0000	1.16
+++ include/linux/pci.h	6 Jun 2003 19:32:08 -0000
@@ -868,5 +868,15 @@ extern int pci_pci_problems;
 #define PCIPCI_VSFX		16
 #define PCIPCI_ALIMAGIK		32
 
+/*
+ * PCI domain support.  Sometimes called PCI segment (eg by ACPI),
+ * a PCI domain is defined to be a set of PCI busses which share
+ * configuration space.
+ */
+
+#ifndef CONFIG_PCI_DOMAINS
+#define pci_domain_nr(pdev)	0
+#endif
+
 #endif /* __KERNEL__ */
 #endif /* LINUX_PCI_H */

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [PATCH] [3/3] PCI segment support

[Russell King]

On Sun, Jun 08, 2003 at 05:43:51PM +0100, Matthew Wilcox wrote:
> I envisage ia64 will always turn on CONFIG_PCI_DOMAINS but x86 might
> well have it as a user question.  I suspect most architectures would
> never turn it on (yeah, I'm going to design an embedded ARM box with
> multiple PCI domains.  sure.)

Don't be so sure.  There's already ARM implementations where there are
multiple PCI buses hanging off the host bridge - mostly stuff from Intel
though.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: [PATCH] [3/3] PCI segment support

[Alan Cox]

On Sul, 2003-06-08 at 22:33, Russell King wrote:
> On Sun, Jun 08, 2003 at 05:43:51PM +0100, Matthew Wilcox wrote:
> > I envisage ia64 will always turn on CONFIG_PCI_DOMAINS but x86 might
> > well have it as a user question.  I suspect most architectures would
> > never turn it on (yeah, I'm going to design an embedded ARM box with
> > multiple PCI domains.  sure.)
> 
> Don't be so sure.  There's already ARM implementations where there are
> multiple PCI buses hanging off the host bridge - mostly stuff from Intel
> though.

And x86 also, although its normally multiple pci host bridges hanging
off an internal faster bus with a large collection of magic hardware
wizardry to make it look like one

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

On Sun, Jun 08, 2003 at 05:43:51PM +0100, Matthew Wilcox wrote:
>  - Use pci_domain_nr() macro to determine which domain a bus or device
>    is in.
>  - Default implementation for architectures which don't support PCI
>    domains.
>  - Implementation for ia64.

Looks good, but shouldn't we pass 'struct pci_bus *' instead
of pci_dev to pci_domain_nr()?
Because another place where the domain number must be checked on is
pci_bus_exists().

Ivan.

Re: [PATCH] [3/3] PCI segment support

[David S. Miller]

On Mon, 2003-06-09 at 03:07, Ivan Kokshaysky wrote:
> Looks good, but shouldn't we pass 'struct pci_bus *' instead
> of pci_dev to pci_domain_nr()?

I don't think it matters, but someone may find a useful
use of having the exact device available, who knows...

> Because another place where the domain number must be checked on is
> pci_bus_exists().

We could just pass the bus self device in this case.

-- 
David S. Miller <davem@redhat.com>

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

On Mon, Jun 09, 2003 at 03:20:56AM -0700, David S. Miller wrote:
> On Mon, 2003-06-09 at 03:07, Ivan Kokshaysky wrote:
> > Looks good, but shouldn't we pass 'struct pci_bus *' instead
> > of pci_dev to pci_domain_nr()?
> 
> I don't think it matters, but someone may find a useful
> use of having the exact device available, who knows...

Hmm. Actually the patch *does* use pci_bus. What got me confused is
definition in include/linux/pci.h:

+#ifndef CONFIG_PCI_DOMAINS
+#define pci_domain_nr(pdev)    0
		       ~~~~
+#endif

I think it should be changed to 'pbus' to avoid confusion.

> We could just pass the bus self device in this case.

Root buses often do not have the self device, e.g. on alpha.

Ivan.

Re: [PATCH] [3/3] PCI segment support

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Mon, 9 Jun 2003 14:42:42 +0400

   On Mon, Jun 09, 2003 at 03:20:56AM -0700, David S. Miller wrote:
   > We could just pass the bus self device in this case.
   
   Root buses often do not have the self device, e.g. on alpha.

How can people make PCI config space accesses to them?

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

On Mon, Jun 09, 2003 at 03:43:04AM -0700, David S. Miller wrote:
>    Root buses often do not have the self device, e.g. on alpha.
> 
> How can people make PCI config space accesses to them?

The root level controllers itself are not accessible from
PCI config space (unlike x86 host bridges). They have
dedicated control registers somewhere in the IO space.

Ivan.

Re: [PATCH] [3/3] PCI segment support

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Mon, 9 Jun 2003 15:00:18 +0400
   
   The root level controllers itself are not accessible from
   PCI config space (unlike x86 host bridges). They have
   dedicated control registers somewhere in the IO space.

This sounds more like a PCI host controller, not the
root of the actual PCI bus.

Re: [PATCH] [3/3] PCI segment support

[Matthew Wilcox]

On Mon, Jun 09, 2003 at 02:07:49PM +0400, Ivan Kokshaysky wrote:
> Looks good, but shouldn't we pass 'struct pci_bus *' instead
> of pci_dev to pci_domain_nr()?
> Because another place where the domain number must be checked on is
> pci_bus_exists().

hmm, yes, well.  There's a certain amount of sloppiness allowed with
it being a macro, in that bus->sysdata and dev->sysdata have the same
value so it works both ways.  Of course, this prohibits any architecture
from implementing it as a function, so we really should make up our minds
which it is to be.  It sounds like bus is more generally useful than dev,
so I'll make that explicit.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

On Mon, Jun 09, 2003 at 12:17:39PM +0100, Matthew Wilcox wrote:
> hmm, yes, well.  There's a certain amount of sloppiness allowed with
> it being a macro, in that bus->sysdata and dev->sysdata have the same
> value so it works both ways.

Well, it's true for many architectures, but not for all.
IIRC, bus->sysdata != dev->sysdata on sparc and parisc.

> Of course, this prohibits any architecture
> from implementing it as a function, so we really should make up our minds
> which it is to be.  It sounds like bus is more generally useful than dev,
> so I'll make that explicit.

Great.

Ivan.

Re: [PATCH] [3/3] PCI segment support

[Matthew Wilcox]

On Mon, Jun 09, 2003 at 03:26:16PM +0400, Ivan Kokshaysky wrote:
> On Mon, Jun 09, 2003 at 12:17:39PM +0100, Matthew Wilcox wrote:
> > hmm, yes, well.  There's a certain amount of sloppiness allowed with
> > it being a macro, in that bus->sysdata and dev->sysdata have the same
> > value so it works both ways.
> 
> Well, it's true for many architectures, but not for all.
> IIRC, bus->sysdata != dev->sysdata on sparc and parisc.

Certainly not true for parisc.  It might be true for sparc; I'm not quite
sure what arch/sparc/kernel/pcic.c is up to -- it seems a little bitrotten.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [PATCH] [3/3] PCI segment support

[Ivan Kokshaysky]

BTW, you can include this in the next version of the patch. :-)

Ivan.

--- 2.5/include/asm-alpha/pci.h	Tue May 27 05:00:20 2003
+++ linux/include/asm-alpha/pci.h	Mon Jun  9 15:29:51 2003
@@ -195,6 +195,9 @@ extern void
 pcibios_resource_to_bus(struct pci_dev *dev, struct pci_bus_region *region,
 			 struct resource *res);
 
+#define pci_domain_nr(pbus) ({ struct pci_controller *_hose_ = pbus->sysdata; \
+			       _hose_->index; })
+
 #endif /* __KERNEL__ */
 
 /* Values for the `which' argument to sys_pciconfig_iobase.  */
--- 2.5/arch/alpha/Kconfig	Mon Jun  9 12:39:39 2003
+++ linux/arch/alpha/Kconfig	Mon Jun  9 12:43:55 2003
@@ -295,6 +295,10 @@ config PCI
 	  information about which PCI hardware does work under Linux and which
 	  doesn't.
 
+config PCI_DOMAINS
+	bool
+	default PCI
+
 config ALPHA_CORE_AGP
 	bool
 	depends on ALPHA_GENERIC || ALPHA_TITAN || ALPHA_MARVEL