Re: PCI bridge range sizing bug

Re: PCI bridge range sizing bug

[Jesse Barnes]

On Thursday, April 5, 2007 3:37 pm Adam Jackson wrote:
> So I'm attempting to do something fairly heinous (X server across
> five video cards), and I hit a fun bug in bridge range setup.  See
> attached lspci and dmesg, but the short of it is I've got two VGA
> chips on one card behind a bridge, which is itself behind a second
> PCI bridge, and the bridge ranges get set up so that I can't map the
> ROMs, which means I can't post them, and therefore can't use them
> period.
>
> The alignment restriction on the ROMs seems a bit extreme:
>
> % sudo setpci -s 7:2 ROM_ADDRESS=ffffffff
> % sudo setpci -s 7:2 ROM_ADDRESS
> f0000001
>
> (same for 7:1) so that might be part of the problem.

...
Allocating PCI resources starting at 88000000 (gap: 80000000:7ff00000)
...

That's ~2G of space, which should be plenty for your PCI resources I 
hope?  If you have a bunch of cards with large BARS though you might be 
running out.

...
PCI: Bridge: 0000:00:01.0
  IO window: 4000-4fff
  MEM window: a3500000-a35fffff (1M)
  PREFETCH window: 90000000-97ffffff
PCI: Bridge: 0000:00:03.0
  IO window: disabled.
  MEM window: a3400000-a34fffff (1M)
  PREFETCH window: 98000000-9fffffff
PCI: Bridge: 0000:00:1c.0
  IO window: disabled.
  MEM window: a3300000-a33fffff (1M)
  PREFETCH window: 80000000-8fffffff
PCI: Bridge: 0000:00:1c.4
  IO window: 3000-3fff
  MEM window: a3200000-a32fffff (1M)
  PREFETCH window: a3700000-a37fffff
PCI: Bridge: 0000:00:1c.5
  IO window: 2000-2fff
  MEM window: a3100000-a31fffff (1M)
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:10000000@b0000000 for 
0000:07:01.0
PCI: Failed to allocate mem resource #6:10000000@b0000000 for 
0000:07:02.0
...

Yep, looks like those two devices had a problem.  Supposedly they want 
to sit at 256M?  Given that we're only giving each bridge 1M of memory 
space that would definitely be a problem.

The total so far is only 5M of PCI space... so we're not making good use 
of the 2G we were given.

...
PCI: Bridge: 0000:06:00.0
  IO window: disabled.
  MEM window: a1000000-a2ffffff (32M)
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: 1000-1fff
  MEM window: a1000000-a30fffff (~32M)
  PREFETCH window: a0000000-a0ffffff
...

And these bridges got more space somehow...  Greg who's in charge of our 
bridge resource allocation code?

Jesse

Re: PCI bridge range sizing bug

[Greg KH]

On Thu, Apr 19, 2007 at 04:11:50PM -0700, Jesse Barnes wrote:
> On Thursday, April 5, 2007 3:37 pm Adam Jackson wrote:
> > So I'm attempting to do something fairly heinous (X server across
> > five video cards), and I hit a fun bug in bridge range setup.  See
> > attached lspci and dmesg, but the short of it is I've got two VGA
> > chips on one card behind a bridge, which is itself behind a second
> > PCI bridge, and the bridge ranges get set up so that I can't map the
> > ROMs, which means I can't post them, and therefore can't use them
> > period.
> >
> > The alignment restriction on the ROMs seems a bit extreme:
> >
> > % sudo setpci -s 7:2 ROM_ADDRESS=ffffffff
> > % sudo setpci -s 7:2 ROM_ADDRESS
> > f0000001
> >
> > (same for 7:1) so that might be part of the problem.
> 
> ...
> Allocating PCI resources starting at 88000000 (gap: 80000000:7ff00000)
> ...
> 
> That's ~2G of space, which should be plenty for your PCI resources I 
> hope?  If you have a bunch of cards with large BARS though you might be 
> running out.
> 
> ...
> PCI: Bridge: 0000:00:01.0
>   IO window: 4000-4fff
>   MEM window: a3500000-a35fffff (1M)
>   PREFETCH window: 90000000-97ffffff
> PCI: Bridge: 0000:00:03.0
>   IO window: disabled.
>   MEM window: a3400000-a34fffff (1M)
>   PREFETCH window: 98000000-9fffffff
> PCI: Bridge: 0000:00:1c.0
>   IO window: disabled.
>   MEM window: a3300000-a33fffff (1M)
>   PREFETCH window: 80000000-8fffffff
> PCI: Bridge: 0000:00:1c.4
>   IO window: 3000-3fff
>   MEM window: a3200000-a32fffff (1M)
>   PREFETCH window: a3700000-a37fffff
> PCI: Bridge: 0000:00:1c.5
>   IO window: 2000-2fff
>   MEM window: a3100000-a31fffff (1M)
>   PREFETCH window: disabled.
> PCI: Failed to allocate mem resource #6:10000000@b0000000 for 
> 0000:07:01.0
> PCI: Failed to allocate mem resource #6:10000000@b0000000 for 
> 0000:07:02.0
> ...
> 
> Yep, looks like those two devices had a problem.  Supposedly they want 
> to sit at 256M?  Given that we're only giving each bridge 1M of memory 
> space that would definitely be a problem.
> 
> The total so far is only 5M of PCI space... so we're not making good use 
> of the 2G we were given.
> 
> ...
> PCI: Bridge: 0000:06:00.0
>   IO window: disabled.
>   MEM window: a1000000-a2ffffff (32M)
>   PREFETCH window: disabled.
> PCI: Bridge: 0000:00:1e.0
>   IO window: 1000-1fff
>   MEM window: a1000000-a30fffff (~32M)
>   PREFETCH window: a0000000-a0ffffff
> ...
> 
> And these bridges got more space somehow...  Greg who's in charge of our 
> bridge resource allocation code?

Ivan and Linus seem to be doing the most of the work in this area, I
gladly pass these issues on to them :)

thanks,

greg k-h

Re: PCI bridge range sizing bug

[Linus Torvalds]

On Thu, 19 Apr 2007, Greg KH wrote:

> On Thu, Apr 19, 2007 at 04:11:50PM -0700, Jesse Barnes wrote:
> > On Thursday, April 5, 2007 3:37 pm Adam Jackson wrote:
> > > So I'm attempting to do something fairly heinous (X server across
> > > five video cards), and I hit a fun bug in bridge range setup.  See
> > > attached lspci and dmesg, but the short of it is I've got two VGA
> > > chips on one card behind a bridge, which is itself behind a second
> > > PCI bridge, and the bridge ranges get set up so that I can't map the
> > > ROMs, which means I can't post them, and therefore can't use them
> > > period.

Ok, let me start out by saying that at this point in the development cycle 
(ie trying to get 2.6.21 out some day), I can't really find it in myself 
to care all that deeply about people doing something quote _that_ fairly 
heinous ;)

> > > The alignment restriction on the ROMs seems a bit extreme:
> > >
> > > % sudo setpci -s 7:2 ROM_ADDRESS=ffffffff
> > > % sudo setpci -s 7:2 ROM_ADDRESS
> > > f0000001

Yeah, they seem to want 256MB.

> > Yep, looks like those two devices had a problem.  Supposedly they want 
> > to sit at 256M?  Given that we're only giving each bridge 1M of memory 
> > space that would definitely be a problem.

We should be sizing the bridge regions by how much space the devices 
behind the bridges actually need, BUT I can guess at two problems:

 - On PC's, we generally trust any BIOS setup. If the bridge has been 
   initialized to some value that seems half-way valid we generally leave 
   it there. Moving things around tends to cause a lot  more problems than 
   it fixes.
 - we normally do *not* try to assign ROM resources at all, because a 
   number of video cards in particular do some really strange stuff with 
   the ROMS, like share the decoders for the ROM's and the other PCI 
   resources!

IOW, I think that what happens is that the BIOS has set it up to have a 
32MB window, and since the kernel doesn't think there is anything wrong 
with that, it leaves it well enough alone.

You could try adding "pci=rom" to the kernel command line, which should 
make the kernel try to assign space for the roms too.

I think we used to *never* assign PCI bus resources on x86, but that thing 
got fixed some time ago. Now I think we only re-assign them if they were 
unassigned *or* if the assignment wasn't working before. But I'm not 100% 
sure about that second part... It's been working so well that I don't 
think we've had a lot of problems with resource assignment lately, and 
I've paged it all out of my brain.

Ivan, can you remind my tired old brain?

		Linus

Re: PCI bridge range sizing bug

[Ivan Kokshaysky]

On Thu, Apr 19, 2007 at 05:19:20PM -0700, Linus Torvalds wrote:
> I think we used to *never* assign PCI bus resources on x86, but that thing 
> got fixed some time ago. Now I think we only re-assign them if they were 
> unassigned *or* if the assignment wasn't working before. But I'm not 100% 
> sure about that second part... It's been working so well that I don't 
> think we've had a lot of problems with resource assignment lately, and 
> I've paged it all out of my brain.
> 
> Ivan, can you remind my tired old brain?

No :-)  You are absolutely right - we re-assign only unassigned OR
conflicting resources. And yes, x86 kernel does accept any enabled
bridge ranges without looking at what is on the other side of the
bridge.

I think what we need is some very minimalistic validity check for
BIOS bridge setup: calculate sums of resource ranges of each type
(or just MEM and PREFETCH, should we care about IO these days?)
for devices that are behind the bridge, even without taking alignment
requirements into account. Then, if some window is too small, we just
let the pci_assign_unassigned_resources to take care of that.

Ivan.

Re: PCI bridge range sizing bug

[Jesse Barnes]

On Friday, April 20, 2007 2:23 am Ivan Kokshaysky wrote:
> On Thu, Apr 19, 2007 at 05:19:20PM -0700, Linus Torvalds wrote:
> > I think we used to *never* assign PCI bus resources on x86, but
> > that thing got fixed some time ago. Now I think we only re-assign
> > them if they were unassigned *or* if the assignment wasn't working
> > before. But I'm not 100% sure about that second part... It's been
> > working so well that I don't think we've had a lot of problems with
> > resource assignment lately, and I've paged it all out of my brain.
> >
> > Ivan, can you remind my tired old brain?
>
> No :-)  You are absolutely right - we re-assign only unassigned OR
> conflicting resources. And yes, x86 kernel does accept any enabled
> bridge ranges without looking at what is on the other side of the
> bridge.
>
> I think what we need is some very minimalistic validity check for
> BIOS bridge setup: calculate sums of resource ranges of each type
> (or just MEM and PREFETCH, should we care about IO these days?)
> for devices that are behind the bridge, even without taking alignment
> requirements into account. Then, if some window is too small, we just
> let the pci_assign_unassigned_resources to take care of that.

Sounds good, hopefully reassigning the bridge resources won't cause too 
much trouble.  Do you have time to hack this up?  If not, I could give 
it a try, as long as ajax is willing to test...

Thanks,
Jesse

Re: PCI bridge range sizing bug

[Linus Torvalds]

On Fri, 20 Apr 2007, Jesse Barnes wrote:
> 
> Sounds good, hopefully reassigning the bridge resources won't cause too 
> much trouble.  Do you have time to hack this up?  If not, I could give 
> it a try, as long as ajax is willing to test...

Actually, I would suggest we not do it automatically (because the need for 
it is just so low, and the downsides are potentially huge - there are just 
too many resources that are "hidden" from us through ACPI tricks and 
having hardware that doesn't actually expose their PCI resources fully 
through the normal PCI resource setup).

So I'd suggest that at least initially, we could/should just make it a 
decision based on a command line flag like the "pci=assign-busses" thing 
is now (which forces bus *numbers* to be re-assigned regardless of how 
they were assigned by the firmware, not the actual resources).

Because realistically, I don't think we'll see any common PC platform that 
actually needs it. Simply because I seriously doubt Windows reassignes bus 
resources either.

Ivan, want to add some way to force that allocation (something like 
"pci=assign-bus-resources")

		Linus

Re: PCI bridge range sizing bug

[Ivan Kokshaysky]

On Fri, Apr 20, 2007 at 11:28:42AM -0700, Linus Torvalds wrote:
> Actually, I would suggest we not do it automatically (because the need for 
> it is just so low, and the downsides are potentially huge - there are just 
> too many resources that are "hidden" from us through ACPI tricks and 
> having hardware that doesn't actually expose their PCI resources fully 
> through the normal PCI resource setup).

Definitely. I was intending to enable that *only* with some boot option.

> Ivan, want to add some way to force that allocation (something like 
> "pci=assign-bus-resources")

Yes, hopefully I'll get something in a next couple of days.

Ivan.

Re: PCI bridge range sizing bug

[Jesse Barnes]

On Friday, April 20, 2007 11:28 am Linus Torvalds wrote:
> On Fri, 20 Apr 2007, Jesse Barnes wrote:
> > Sounds good, hopefully reassigning the bridge resources won't cause
> > too much trouble.  Do you have time to hack this up?  If not, I
> > could give it a try, as long as ajax is willing to test...
>
> Actually, I would suggest we not do it automatically (because the
> need for it is just so low, and the downsides are potentially huge -
> there are just too many resources that are "hidden" from us through
> ACPI tricks and having hardware that doesn't actually expose their
> PCI resources fully through the normal PCI resource setup).

Yeah, that's probably prudent.  OTOH we should probably let the user 
know in no uncertain terms that some of the stuff behind one of their 
bridges will be inaccessible.

Thanks,
Jesse

Re: PCI bridge range sizing bug

[Rik van Riel]

Jesse Barnes wrote:
> On Friday, April 20, 2007 11:28 am Linus Torvalds wrote:
>> On Fri, 20 Apr 2007, Jesse Barnes wrote:
>>> Sounds good, hopefully reassigning the bridge resources won't cause
>>> too much trouble.  Do you have time to hack this up?  If not, I
>>> could give it a try, as long as ajax is willing to test...
>> Actually, I would suggest we not do it automatically (because the
>> need for it is just so low, and the downsides are potentially huge -
>> there are just too many resources that are "hidden" from us through
>> ACPI tricks and having hardware that doesn't actually expose their
>> PCI resources fully through the normal PCI resource setup).
> 
> Yeah, that's probably prudent.  OTOH we should probably let the user 
> know in no uncertain terms that some of the stuff behind one of their 
> bridges will be inaccessible.

Something like that would have made it a lot more obvious
why my Matrox PCIe x1 video card will not work in my Dell
9150, while a PCI video card does work.

The PCI video card directly sits on the bus, and gets its
resources assigned by the BIOS.

The PCIe video card turned out to be a PCIe to AGP bridge,
and the BIOS did not assign the needed PCI resources, making
the system crash when I started X.

X seemed to have some trouble reading the ROM, too...

-- 
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

Re: PCI bridge range sizing bug

[Jesse Barnes]

On Friday, April 20, 2007 1:30 pm Ivan Kokshaysky wrote:
> On Fri, Apr 20, 2007 at 11:28:42AM -0700, Linus Torvalds wrote:
> > Actually, I would suggest we not do it automatically (because the
> > need for it is just so low, and the downsides are potentially huge
> > - there are just too many resources that are "hidden" from us
> > through ACPI tricks and having hardware that doesn't actually
> > expose their PCI resources fully through the normal PCI resource
> > setup).
>
> Definitely. I was intending to enable that *only* with some boot
> option.
>
> > Ivan, want to add some way to force that allocation (something like
> > "pci=assign-bus-resources")
>
> Yes, hopefully I'll get something in a next couple of days.

Any update on this, Ivan?  Maybe I missed your post, but I haven't seen 
anything yet...

Thanks,
Jesse

Re: PCI bridge range sizing bug

[Ivan Kokshaysky]

On Mon, May 14, 2007 at 10:45:43AM -0700, Jesse Barnes wrote:
> Any update on this, Ivan?  Maybe I missed your post, but I haven't seen 
> anything yet...

No, I didn't post anything, sorry. This turned out to be not as trivial
as I thought - I've played with that code on amd64, and results were rather
discouraging. For example, I forced reassignment of video card resources
and initially this failed because of the way how pci/setup-* allocates ROM
addresses: ROMs are obviously prefetchable, so we're always trying to put
them in prefetchable windows of the bridge. Technically, this is correct,
but leads to the waste of PCI address space due to size/alignment
requirements, especially when ROM is assigned to the same window as
a prefetchable framebuffer resource (typically 64-256M).
On the other hand, we could allocate ROMs in non-prefetch ranges just
for free, as minimal memory window of the bridges is 1M and both MMIO
register blocks and ROMs are usually much smaller. 
Probably we need some global variable to control this (and some other things)
in pci/setup-*, just like pci_probe on i386...

Another funny thing found on x86_64 - if we ran out of PCI address space 
below 4G, resources get happily assigned above 4G, even if respective BARs
are 32-bit.

Anyway, here's what I have so far, though I doubt that it's a breakthrough
for Adam...

Ivan.

--- orig/arch/i386/pci/common.c	Sat Apr 21 21:55:30 2007
+++ linux/arch/i386/pci/common.c	Sat Apr 21 21:55:33 2007
@@ -290,6 +290,12 @@ static struct dmi_system_id __devinitdat
 	{}
 };
 
+static struct resource pci_mem32 = {
+	.name	= "PCI 32-bit memory space",
+	.end	= 0xffffffff,
+	.flags	= IORESOURCE_MEM,
+};
+
 struct pci_bus * __devinit pcibios_scan_root(int busnum)
 {
 	struct pci_bus *bus = NULL;
@@ -305,7 +311,13 @@ struct pci_bus * __devinit pcibios_scan_
 
 	printk(KERN_DEBUG "PCI: Probing PCI hardware (bus %02x)\n", busnum);
 
-	return pci_scan_bus_parented(NULL, busnum, &pci_root_ops, NULL);
+	bus = pci_scan_bus_parented(NULL, busnum, &pci_root_ops, NULL);
+	if (bus && bus->resource[1] == &iomem_resource) {
+		pci_mem32.start = pci_mem_start;
+		if (insert_resource(&iomem_resource, &pci_mem32) == 0)
+			bus->resource[1] = &pci_mem32;
+	}
+	return bus;
 }
 
 extern u8 pci_cache_line_size;
@@ -418,6 +418,9 @@ char * __devinit  pcibios_setup(char *st
 	} else if (!strcmp(str, "routeirq")) {
 		pci_routeirq = 1;
 		return NULL;
+	} else if (!strcmp(str, "assign-bus-resources")) {
+		pci_probe |= PCI_ASSIGN_BUS_RESOURCES;
+		return NULL;
 	}
 	return str;
 }
--- orig/arch/i386/pci/pci.h	Sat Apr 21 21:55:30 2007
+++ linux/arch/i386/pci/pci.h	Sat Apr 21 21:55:33 2007
@@ -26,6 +26,7 @@
 #define PCI_ASSIGN_ROMS		0x1000
 #define PCI_BIOS_IRQ_SCAN	0x2000
 #define PCI_ASSIGN_ALL_BUSSES	0x4000
+#define PCI_ASSIGN_BUS_RESOURCES	0x8000
 
 extern unsigned int pci_probe;
 extern unsigned long pirq_table_addr;
--- orig/arch/i386/pci/i386.c	Sat Apr 21 21:55:30 2007
+++ linux/arch/i386/pci/i386.c	Sat Apr 21 21:56:39 2007
@@ -60,6 +60,67 @@ pcibios_align_resource(void *data, struc
 	}
 }
 
+static int __init
+pcibios_estimate_bus_space(struct pci_dev *bridge, resource_size_t *mem,
+			   resource_size_t *pref)
+{
+	int i;
+	struct resource *r;
+	struct pci_dev *dev;
+
+	if (!bridge->subordinate ||
+	    (bridge->class >> 8) != PCI_CLASS_BRIDGE_PCI)
+		return 0;
+	list_for_each_entry(dev, &bridge->subordinate->devices, bus_list) {
+		for (i = 0; i <= PCI_ROM_RESOURCE; i++) {
+			r = &dev->resource[i];
+			if (r->flags & IORESOURCE_PREFETCH)
+				*pref += r->end + 1 - r->start;
+			else if (r->flags & IORESOURCE_MEM)
+				*mem += r->end + 1 - r->start;
+		}
+		pcibios_estimate_bus_space(dev, mem, pref);
+	}
+	return 1;
+}
+
+static void __init
+pcibios_validate_bus_ranges(struct pci_dev *bridge)
+{
+	resource_size_t memory = 0, prefetch = 0, bridge_mem, bridge_pref;
+	struct resource *bmem = &bridge->resource[PCI_BRIDGE_RESOURCES + 1];
+	struct resource *bpref = &bridge->resource[PCI_BRIDGE_RESOURCES + 2];
+
+	if (!(pci_probe & PCI_ASSIGN_BUS_RESOURCES) ||
+	    !pcibios_estimate_bus_space(bridge, &memory, &prefetch))
+		return;
+	bridge_mem = bmem->end + 1 - bmem->start;
+	bridge_pref = bpref->end + 1 - bpref->start;
+	/* First, check if bridge memory window can hold all the memory
+	   resources on the secondary buses. */
+	if (bridge_mem >= memory) {
+		/* Yes, it seems to be large enough and probably can hold
+		   [some of] prefetchable memory also. */
+		if (bridge_mem - memory < prefetch)
+			prefetch -= bridge_mem - memory;
+		else
+			prefetch = 0;
+	} else {
+		printk(KERN_INFO "PCI: MEM window of bridge %s too small "
+		       "(%llx, at least %llx required), will be reassigned\n",
+		       pci_name(bridge), (unsigned long long)bridge_mem,
+		       (unsigned long long)memory);
+		bmem->flags = 0;
+	}
+	/* Check the prefetchable memory window. */
+	if (bridge_pref < prefetch) {
+		printk(KERN_INFO "PCI: PREFETCH window of bridge %s too small, "
+		       "(%llx, at least %llx required), will be reassigned\n",
+		       pci_name(bridge), (unsigned long long)bridge_pref,
+		       (unsigned long long)prefetch);
+		bpref->flags = 0;
+	}
+}
 
 /*
  *  Handle resources of PCI devices.  If the world were perfect, we could
@@ -104,6 +165,7 @@ static void __init pcibios_allocate_bus_
 	/* Depth-First Search on bus tree */
 	list_for_each_entry(bus, bus_list, node) {
 		if ((dev = bus->self)) {
+			pcibios_validate_bus_ranges(dev);
 			for (idx = PCI_BRIDGE_RESOURCES;
 			    idx < PCI_NUM_RESOURCES; idx++) {
 				r = &dev->resource[idx];
--- orig/drivers/pci/setup-bus.c	Tue Apr 10 11:33:39 2007
+++ linux/drivers/pci/setup-bus.c	Sat Apr 21 19:56:28 2007
@@ -347,9 +347,12 @@ pbus_size_mem(struct pci_bus *bus, unsig
 		
 		for (i = 0; i < PCI_NUM_RESOURCES; i++) {
 			struct resource *r = &dev->resource[i];
-			unsigned long r_size;
+			unsigned long r_size, r_flags = r->flags;
 
-			if (r->parent || (r->flags & mask) != type)
+			if (i == PCI_ROM_RESOURCE)
+				r_flags &= ~IORESOURCE_PREFETCH;
+
+			if (r->parent || (r_flags & mask) != type)
 				continue;
 			r_size = r->end - r->start + 1;
 			/* For bridges size != alignment */