PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[System Design Works]

The kernel has a problem allocating resources for my PCI NIC.  Here is 
what the kernel is reporting:

# uname -a
Linux wopr 2.6.20-gentoo-r8 #7 SMP Sun May 20 20:56:56 PDT 2007 i686 AMD 
Athlon(tm) 64 X2 Dual Core Processor 3800+ AuthenticAMD GNU/Linux

# dmesg
...
PCI: BIOS Bug:  MCFG area at e0000000 is not E820-reserved
PCI: Not using MMCONFIG.
...
PCI: Cannot allocate resource region 0 of device 0000:02:02.0
...
PCI: Device 0000:02:02.0 not available because of resource collisions
skge: 0000:02:02.0 cannot enable PCI device
skge: probe of 0000:02:02.0 failed with error -22
...

I have seen other posts reporting similar error messages.  I would like 
to help resolve this problem, and I can do some testing if needed.  More 
info:

Kernel boot params:   pci=nomsi

PCI Device:
    D-Link DGE-530T  (10/100/1000 Gigabit Desktop Adapter)
    http://www.dlink.com/products/?pid=284

Motherboard:
    MSI K9AGM-FID  (AMD Socket AM2)
    Chipset:  SB600 / RS485
    http://www.msicomputer.com/product/p_spec.asp?model=K9AGM-FID&class=mb

# lspci -vvv
...
02:02.0 Non-VGA unclassified device: D-Link System Inc Unknown device 
4901 (rev 11)
        Subsystem: D-Link System Inc Unknown device 4901
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- 
ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR+
        Latency: 64 (5750ns min, 7250ns max), Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Region 1: I/O ports at b800 [size=256]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [48] Power Management version 2
                Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA 
PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
...

Thanks,

Wayne

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Jesse Barnes]

On Monday, May 21, 2007, System Design Works wrote:
> The kernel has a problem allocating resources for my PCI NIC.  Here is
> what the kernel is reporting:
>
> # uname -a
> Linux wopr 2.6.20-gentoo-r8 #7 SMP Sun May 20 20:56:56 PDT 2007 i686 AMD
> Athlon(tm) 64 X2 Dual Core Processor 3800+ AuthenticAMD GNU/Linux
>
> # dmesg
> ...
> PCI: BIOS Bug:  MCFG area at e0000000 is not E820-reserved
> PCI: Not using MMCONFIG.

This is actually an unrelated problem.  We're a little too conservative 
about using MCFG space (though this turns out to be a good thing on some 
of my machines), but shouldn't affect the rest of your PCI resource 
assignment.

> ...
> PCI: Cannot allocate resource region 0 of device 0000:02:02.0
> ...
> PCI: Device 0000:02:02.0 not available because of resource collisions
> skge: 0000:02:02.0 cannot enable PCI device
> skge: probe of 0000:02:02.0 failed with error -22
> ...
>
> I have seen other posts reporting similar error messages.  I would like
> to help resolve this problem, and I can do some testing if needed.  More
> info:
>
> Kernel boot params:   pci=nomsi

There's a recent thread about PCI resource assignment (sounds like your 
BIOS might be buggy btw, or you're somehow running out of space), search 
for the title "PCI bridge range sizing bug".  You may need the kernel to 
reassign the resource for your NIC before you can use it.  I think Ivan 
has some test patches along these lines.

If you can find out what resource it's colliding with, that might give you 
a clue.

Jesse

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Wayne Sherman]

Jesse Barnes wrote:
> There's a recent thread about PCI resource assignment (sounds like your 
> BIOS might be buggy btw, or you're somehow running out of space), search 
> for the title "PCI bridge range sizing bug".  You may need the kernel to 
> reassign the resource for your NIC before you can use it.  I think Ivan 
> has some test patches along these lines.
> 
> If you can find out what resource it's colliding with, that might give you 
> a clue.

I don't have anything else plugged in to the PC (except a USB drive). 
BIOS is set to PNP OS.  How do I find out what it is colliding with?

Here is a full lspci output:

# lspci -v

00:00.0 Host bridge: ATI Technologies Inc RS480 Host Bridge (rev 10)
	Subsystem: ATI Technologies Inc RS480 Host Bridge
	Flags: bus master, 66MHz, medium devsel, latency 0

00:01.0 PCI bridge: ATI Technologies Inc RS480 PCI Bridge (prog-if 00 
[Normal decode])
	Flags: bus master, 66MHz, medium devsel, latency 64
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: ff400000-ff4fffff
	Prefetchable memory behind bridge: 00000000fab00000-00000000fea00000
	Capabilities: [44] HyperTransport: MSI Mapping
	Capabilities: [b0] #0d [0000]

00:12.0 SATA controller: ATI Technologies Inc SB600 Non-Raid-5 SATA 
(prog-if 01 [AHCI 1.0])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7244
	Flags: bus master, 66MHz, medium devsel, latency 96, IRQ 18
	I/O ports at e800 [size=8]
	I/O ports at e400 [size=4]
	I/O ports at e000 [size=8]
	I/O ports at dc00 [size=4]
	I/O ports at d800 [size=16]
	Memory at ff6ffc00 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [60] Power Management version 2
	Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable-
	Capabilities: [70] #12 [0010]

00:13.0 USB Controller: ATI Technologies Inc SB600 USB (OHCI0) (prog-if 
10 [OHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
	Memory at ff6fe000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [d0] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-

00:13.1 USB Controller: ATI Technologies Inc SB600 USB (OHCI1) (prog-if 
10 [OHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 21
	Memory at ff6fd000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [d0] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-

00:13.2 USB Controller: ATI Technologies Inc SB600 USB (OHCI2) (prog-if 
10 [OHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
	Memory at ff6fc000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [d0] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-

00:13.3 USB Controller: ATI Technologies Inc SB600 USB (OHCI3) (prog-if 
10 [OHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 21
	Memory at ff6fb000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [d0] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-

00:13.4 USB Controller: ATI Technologies Inc SB600 USB (OHCI4) (prog-if 
10 [OHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
	Memory at ff6fa000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [d0] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-

00:13.5 USB Controller: ATI Technologies Inc SB600 USB Controller (EHCI) 
(prog-if 20 [EHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 20
	Memory at ff6ff800 (32-bit, non-prefetchable) [size=256]
	Capabilities: [c0] Power Management version 2
	Capabilities: [d0] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-
	Capabilities: [e4] Debug port

00:14.0 SMBus: ATI Technologies Inc SB600 SMBus (rev 13)
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: 66MHz, medium devsel
	I/O ports at 0b00 [size=16]
	Capabilities: [b0] HyperTransport: MSI Mapping

00:14.1 IDE interface: ATI Technologies Inc SB600 IDE (prog-if 8a 
[Master SecP PriP])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
	I/O ports at 01f0 [size=8]
	I/O ports at 03f4 [size=1]
	I/O ports at 0170 [size=8]
	I/O ports at 0374 [size=1]
	I/O ports at ff00 [size=16]
	Capabilities: [70] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-

00:14.3 ISA bridge: ATI Technologies Inc SB600 PCI to LPC Bridge
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 0

00:14.4 PCI bridge: ATI Technologies Inc SB600 PCI to PCI Bridge 
(prog-if 01 [Subtractive decode])
	Flags: bus master, 66MHz, medium devsel, latency 64
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=64
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: ff500000-ff5fffff
	Prefetchable memory behind bridge: 40000000-400fffff

00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
HyperTransport Technology Configuration
	Flags: fast devsel
	Capabilities: [80] HyperTransport: Host or Secondary Interface

00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
Address Map
	Flags: fast devsel

00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
DRAM Controller
	Flags: fast devsel

00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] 
Miscellaneous Control
	Flags: fast devsel
	Capabilities: [f0] #0f [0010]

01:05.0 VGA compatible controller: ATI Technologies Inc RS482 [Radeon 
Xpress 200] (prog-if 00 [VGA])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 7242
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 10
	Memory at fc000000 (32-bit, prefetchable) [size=32M]
	I/O ports at a800 [size=256]
	Memory at ff4f0000 (32-bit, non-prefetchable) [size=64K]
	Expansion ROM at ff4c0000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 2

02:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host 
Controller (rev c0) (prog-if 10 [OHCI])
	Subsystem: Micro-Star International Co., Ltd. Unknown device 242d
	Flags: bus master, medium devsel, latency 64, IRQ 16
	Memory at ff5ff800 (32-bit, non-prefetchable) [size=2K]
	I/O ports at bc00 [size=128]
	Capabilities: [50] Power Management version 2

02:02.0 Non-VGA unclassified device: D-Link System Inc Unknown device 
4901 (rev 11)
	Subsystem: D-Link System Inc Unknown device 4901
	Flags: bus master, 66MHz, fast devsel, latency 64, IRQ 11
	Memory at <ignored> (32-bit, non-prefetchable)
	I/O ports at b800 [size=256]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [48] Power Management version 2
	Capabilities: [50] Vital Product Data

02:06.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Unknown 
device 8167 (rev 10)
	Subsystem: Micro-Star International Co., Ltd. Unknown device 242c
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
	I/O ports at b400 [size=256]
	Memory at ff5ff400 (32-bit, non-prefetchable) [size=256]
	Expansion ROM at 40000000 [disabled] [size=128K]
	Capabilities: [dc] Power Management version 2

Thanks,

Wayne

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Jesse Barnes]

On Monday, May 21, 2007, Wayne Sherman wrote:
> Jesse Barnes wrote:
> > There's a recent thread about PCI resource assignment (sounds like
> > your BIOS might be buggy btw, or you're somehow running out of space),
> > search for the title "PCI bridge range sizing bug".  You may need the
> > kernel to reassign the resource for your NIC before you can use it.  I
> > think Ivan has some test patches along these lines.
> >
> > If you can find out what resource it's colliding with, that might give
> > you a clue.
>
> I don't have anything else plugged in to the PC (except a USB drive).
> BIOS is set to PNP OS.  How do I find out what it is colliding with?
>
> 02:02.0 Non-VGA unclassified device: D-Link System Inc Unknown device
> 4901 (rev 11)
> 	Subsystem: D-Link System Inc Unknown device 4901
> 	Flags: bus master, 66MHz, fast devsel, latency 64, IRQ 11
> 	Memory at <ignored> (32-bit, non-prefetchable)
> 	I/O ports at b800 [size=256]
> 	Expansion ROM at <ignored> [disabled]
> 	Capabilities: [48] Power Management version 2
> 	Capabilities: [50] Vital Product Data

Can you dump the memory BAR for this device?  I guess lspci hides it by 
default.  You may also be able to see it using 'od -t 
x4 /sys/devices/pci0000\:00/0000:02:02.0/config'.  I'm not sure how much 
that'll help though, you may be able to see which device its resources are 
overlapping with, but unless your BIOS gives you an option to disable it 
or change the way PCI space is allocated, you may be stuck...

Jesse

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Wayne Sherman]

Jesse Barnes wrote:
> Can you dump the memory BAR for this device?  I guess lspci hides it by 
> default.  You may also be able to see it using 'od -t 
> x4 /sys/devices/pci0000\:00/0000:02:02.0/config'.

The D-Link NIC is sitting behind this bridge:
   "00:14.4 PCI bridge: ATI Technologies Inc SB600 PCI to PCI Bridge"

These are the devices behind the bridge:
0000:02:00.0 - FireWire (IEEE 1394): VIA Technologies,
	Inc. IEEE 1394 Host Controller (rev c0)
0000:02:02.0 - D-Link NIC
0000:02:06.0 - Ethernet controller: Realtek Semiconductor Co., Ltd.
	Unknown device 8167 (rev 10)

Here is the D-LINK NIC:
# od -t x4 /sys/devices/pci0000:00/0000:00:14.4/0000:02:02.0/config

0000000 49011186 80b00117 00000011 00004010
0000020 fd5f8000 0000b801 00000000 00000000
0000040 00000000 00000000 00000000 49011186
0000060 fd5c0000 00000048 00000000 1d17010b
0000100 05f00000 01a08000 fc025001 11002000
0000120 80000003 01000000 01040000 00000000
0000140 00000000 00000000 00000000 00000000

# cat /sys/devices/pci0000:00/0000:00:14.4/0000:02:02.0/resource
0x0000000000000000 0x0000000000003fff 0x0000000000000200
0x000000000000b800 0x000000000000b8ff 0x0000000000000101
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x000000000001ffff 0x0000000000007200

In that same folder, "resource0" is 16384 bytes long and "resource1" is 
256 bytes.

I assume my error "Cannot allocate resource region 0" refers to the 
first base address register (config space offset 0x10).  That value 
indicates it is supposed to be memory mapped at address 0xfd5f8000.  I 
am guessing here, but does the length of "resource0" represent how long 
this region is supposed to be (16384 bytes)?

Now looking at the system memory map:
# cat /proc/iomem
00000000-00097fff : System RAM
00098000-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000cefff : Video ROM
000cf000-000cffff : Adapter ROM
000f0000-000fffff : System ROM
00100000-3dfbffff : System RAM
   00100000-003dd103 : Kernel code
   003dd104-004de0ff : Kernel data
3dfc0000-3dfcdfff : ACPI Tables
3dfce000-3dffffff : ACPI Non-volatile Storage
40000000-400fffff : PCI Bus #02
   40000000-4001ffff : 0000:02:06.0
fab00000-feafffff : PCI Bus #01		<<< offending memory region
   fc000000-fdffffff : 0000:01:05.0	<<< offending memory region
fed00000-fed003ff : HPET 2
ff400000-ff4fffff : PCI Bus #01
   ff4c0000-ff4dffff : 0000:01:05.0
   ff4f0000-ff4fffff : 0000:01:05.0
ff500000-ff5fffff : PCI Bus #02
   ff5ff400-ff5ff4ff : 0000:02:06.0
     ff5ff400-ff5ff4ff : r8169
   ff5ff800-ff5fffff : 0000:02:00.0
     ff5ff800-ff5fffff : ohci1394
ff6fa000-ff6fafff : 0000:00:13.4
   ff6fa000-ff6fafff : ohci_hcd
ff6fb000-ff6fbfff : 0000:00:13.3
   ff6fb000-ff6fbfff : ohci_hcd
ff6fc000-ff6fcfff : 0000:00:13.2
   ff6fc000-ff6fcfff : ohci_hcd
ff6fd000-ff6fdfff : 0000:00:13.1
   ff6fd000-ff6fdfff : ohci_hcd
ff6fe000-ff6fefff : 0000:00:13.0
   ff6fe000-ff6fefff : ohci_hcd
ff6ff800-ff6ff8ff : 0000:00:13.5
   ff6ff800-ff6ff8ff : ehci_hcd
ff6ffc00-ff6fffff : 0000:00:12.0
   ff6ffc00-ff6fffff : ahci
ff780000-ffffffff : reserved

So these devices are already occupying the space that the D-Link wants:

00:01.0 PCI bridge: ATI Technologies Inc RS480 PCI Bridge
01:05.0 VGA compatible controller: ATI Technologies Inc RS482 [Radeon 
Xpress 200]

But, the D-Link NIC is behind the bridge at
	ff500000-ff5fffff : PCI Bus #02

Is a PCI device behind a bridge supposed to be mapped into memory that 
its bridge encompasses?  If so, the D-Link is not being mapped into the 
right region, and the bridge it is behind does not have enough memory 
assigned to it (ff500000-ff5fffff : PCI Bus #02).

Thanks,

Wayne

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

On Tuesday, May 22, 2007, Wayne Sherman wrote:
> So these devices are already occupying the space that the D-Link wants:
>
> 00:01.0 PCI bridge: ATI Technologies Inc RS480 PCI Bridge
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS482 [Radeon
> Xpress 200]
>
> But, the D-Link NIC is behind the bridge at
> 	ff500000-ff5fffff : PCI Bus #02
>
> Is a PCI device behind a bridge supposed to be mapped into memory that
> its bridge encompasses?

Yes.

> If so, the D-Link is not being mapped into the 
> right region, and the bridge it is behind does not have enough memory
> assigned to it (ff500000-ff5fffff : PCI Bus #02).

Sounds familiar.  There are lots of cases where bridge windows aren't 
allocated properly so devices behind them are invisible or can't work.  
Check out the attached patch from Ivan, if you 
pass 'pci=assign-bus-resources' to your kernel at boot time, the code will 
try to reallocate bridge space for you if needed.

Jesse

[-- Attachment #2: pci-bridge-allocation.patch --]
[-- Type: text/x-diff, Size: 9034 bytes --]

From ink@jurassic.park.msu.ru Tue May 15 15:39:53 2007
Received: from virtuous by box128.bluehost.com with local-bsmtp (Exim 4.63)
	(envelope-from <linux-kernel-owner+jbarnes=40virtuousgeek.org-S1759714AbXEOWja@vger.kernel.org>)
	id 1Ho5gr-0001rh-22
	for jbarnes@virtuousgeek.org; Tue, 15 May 2007 16:40:21 -0600
X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on box128.bluehost.com
X-Spam-Level: 
X-Spam-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00 autolearn=no
	version=3.2.0
Received: from vger.kernel.org ([209.132.176.167])
	by box128.bluehost.com with esmtp (Exim 4.63)
	(envelope-from <linux-kernel-owner+jbarnes=40virtuousgeek.org-S1759714AbXEOWja@vger.kernel.org>)
	id 1Ho5gq-0001rT-HJ
	for jbarnes@virtuousgeek.org; Tue, 15 May 2007 16:40:20 -0600
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1759714AbXEOWja (ORCPT <rfc822;jbarnes@virtuousgeek.org>);
	Tue, 15 May 2007 18:39:30 -0400
Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1756955AbXEOWjU
	(ORCPT <rfc822;linux-kernel-outgoing>);
	Tue, 15 May 2007 18:39:20 -0400
Received: from jurassic.park.msu.ru ([195.208.223.243]:4396 "EHLO
	jurassic.park.msu.ru" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1756838AbXEOWjT (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 15 May 2007 18:39:19 -0400
Received: by jurassic.park.msu.ru (Postfix, from userid 500)
	id 424A411E9A1; Wed, 16 May 2007 02:39:53 +0400 (MSD)
Date:	Wed, 16 May 2007 02:39:53 +0400
From:	Ivan Kokshaysky <ink@jurassic.park.msu.ru>
To: Jesse Barnes <jesse.barnes@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
 Greg KH <gregkh@suse.de>,
 Adam Jackson <ajackson@redhat.com>,
 linux-kernel@vger.kernel.org
Subject: Re: PCI bridge range sizing bug
Message-ID: <20070516023953.A7394@jurassic.park.msu.ru>
References: <1175812632.17147.12.camel@localhost.localdomain> <alpine.LFD.0.98.0704201102000.9964@woody.linux-foundation.org> <20070421003020.A21503@jurassic.park.msu.ru> <200705141045.44059.jesse.barnes@intel.com>
Mime-Version: 1.0
Content-Type: text/plain;
  charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <200705141045.44059.jesse.barnes@intel.com>; from jesse.barnes@intel.com on Mon, May 14, 2007 at 10:45:43AM -0700
Precedence: bulk
X-Mailing-List:	linux-kernel@vger.kernel.org
X-Length: 8940
X-UID: 118711

On Mon, May 14, 2007 at 10:45:43AM -0700, Jesse Barnes wrote:
> Any update on this, Ivan?  Maybe I missed your post, but I haven't seen 
> anything yet...

No, I didn't post anything, sorry. This turned out to be not as trivial
as I thought - I've played with that code on amd64, and results were rather
discouraging. For example, I forced reassignment of video card resources
and initially this failed because of the way how pci/setup-* allocates ROM
addresses: ROMs are obviously prefetchable, so we're always trying to put
them in prefetchable windows of the bridge. Technically, this is correct,
but leads to the waste of PCI address space due to size/alignment
requirements, especially when ROM is assigned to the same window as
a prefetchable framebuffer resource (typically 64-256M).
On the other hand, we could allocate ROMs in non-prefetch ranges just
for free, as minimal memory window of the bridges is 1M and both MMIO
register blocks and ROMs are usually much smaller. 
Probably we need some global variable to control this (and some other things)
in pci/setup-*, just like pci_probe on i386...

Another funny thing found on x86_64 - if we ran out of PCI address space 
below 4G, resources get happily assigned above 4G, even if respective BARs
are 32-bit.

Anyway, here's what I have so far, though I doubt that it's a breakthrough
for Adam...

Ivan.

--- orig/arch/i386/pci/common.c	Sat Apr 21 21:55:30 2007
+++ linux/arch/i386/pci/common.c	Sat Apr 21 21:55:33 2007
@@ -290,6 +290,12 @@ static struct dmi_system_id __devinitdat
 	{}
 };
 
+static struct resource pci_mem32 = {
+	.name	= "PCI 32-bit memory space",
+	.end	= 0xffffffff,
+	.flags	= IORESOURCE_MEM,
+};
+
 struct pci_bus * __devinit pcibios_scan_root(int busnum)
 {
 	struct pci_bus *bus = NULL;
@@ -305,7 +311,13 @@ struct pci_bus * __devinit pcibios_scan_
 
 	printk(KERN_DEBUG "PCI: Probing PCI hardware (bus %02x)\n", busnum);
 
-	return pci_scan_bus_parented(NULL, busnum, &pci_root_ops, NULL);
+	bus = pci_scan_bus_parented(NULL, busnum, &pci_root_ops, NULL);
+	if (bus && bus->resource[1] == &iomem_resource) {
+		pci_mem32.start = pci_mem_start;
+		if (insert_resource(&iomem_resource, &pci_mem32) == 0)
+			bus->resource[1] = &pci_mem32;
+	}
+	return bus;
 }
 
 extern u8 pci_cache_line_size;
@@ -418,6 +418,9 @@ char * __devinit  pcibios_setup(char *st
 	} else if (!strcmp(str, "routeirq")) {
 		pci_routeirq = 1;
 		return NULL;
+	} else if (!strcmp(str, "assign-bus-resources")) {
+		pci_probe |= PCI_ASSIGN_BUS_RESOURCES;
+		return NULL;
 	}
 	return str;
 }
--- orig/arch/i386/pci/pci.h	Sat Apr 21 21:55:30 2007
+++ linux/arch/i386/pci/pci.h	Sat Apr 21 21:55:33 2007
@@ -26,6 +26,7 @@
 #define PCI_ASSIGN_ROMS		0x1000
 #define PCI_BIOS_IRQ_SCAN	0x2000
 #define PCI_ASSIGN_ALL_BUSSES	0x4000
+#define PCI_ASSIGN_BUS_RESOURCES	0x8000
 
 extern unsigned int pci_probe;
 extern unsigned long pirq_table_addr;
--- orig/arch/i386/pci/i386.c	Sat Apr 21 21:55:30 2007
+++ linux/arch/i386/pci/i386.c	Sat Apr 21 21:56:39 2007
@@ -60,6 +60,67 @@ pcibios_align_resource(void *data, struc
 	}
 }
 
+static int __init
+pcibios_estimate_bus_space(struct pci_dev *bridge, resource_size_t *mem,
+			   resource_size_t *pref)
+{
+	int i;
+	struct resource *r;
+	struct pci_dev *dev;
+
+	if (!bridge->subordinate ||
+	    (bridge->class >> 8) != PCI_CLASS_BRIDGE_PCI)
+		return 0;
+	list_for_each_entry(dev, &bridge->subordinate->devices, bus_list) {
+		for (i = 0; i <= PCI_ROM_RESOURCE; i++) {
+			r = &dev->resource[i];
+			if (r->flags & IORESOURCE_PREFETCH)
+				*pref += r->end + 1 - r->start;
+			else if (r->flags & IORESOURCE_MEM)
+				*mem += r->end + 1 - r->start;
+		}
+		pcibios_estimate_bus_space(dev, mem, pref);
+	}
+	return 1;
+}
+
+static void __init
+pcibios_validate_bus_ranges(struct pci_dev *bridge)
+{
+	resource_size_t memory = 0, prefetch = 0, bridge_mem, bridge_pref;
+	struct resource *bmem = &bridge->resource[PCI_BRIDGE_RESOURCES + 1];
+	struct resource *bpref = &bridge->resource[PCI_BRIDGE_RESOURCES + 2];
+
+	if (!(pci_probe & PCI_ASSIGN_BUS_RESOURCES) ||
+	    !pcibios_estimate_bus_space(bridge, &memory, &prefetch))
+		return;
+	bridge_mem = bmem->end + 1 - bmem->start;
+	bridge_pref = bpref->end + 1 - bpref->start;
+	/* First, check if bridge memory window can hold all the memory
+	   resources on the secondary buses. */
+	if (bridge_mem >= memory) {
+		/* Yes, it seems to be large enough and probably can hold
+		   [some of] prefetchable memory also. */
+		if (bridge_mem - memory < prefetch)
+			prefetch -= bridge_mem - memory;
+		else
+			prefetch = 0;
+	} else {
+		printk(KERN_INFO "PCI: MEM window of bridge %s too small "
+		       "(%llx, at least %llx required), will be reassigned\n",
+		       pci_name(bridge), (unsigned long long)bridge_mem,
+		       (unsigned long long)memory);
+		bmem->flags = 0;
+	}
+	/* Check the prefetchable memory window. */
+	if (bridge_pref < prefetch) {
+		printk(KERN_INFO "PCI: PREFETCH window of bridge %s too small, "
+		       "(%llx, at least %llx required), will be reassigned\n",
+		       pci_name(bridge), (unsigned long long)bridge_pref,
+		       (unsigned long long)prefetch);
+		bpref->flags = 0;
+	}
+}
 
 /*
  *  Handle resources of PCI devices.  If the world were perfect, we could
@@ -104,6 +165,7 @@ static void __init pcibios_allocate_bus_
 	/* Depth-First Search on bus tree */
 	list_for_each_entry(bus, bus_list, node) {
 		if ((dev = bus->self)) {
+			pcibios_validate_bus_ranges(dev);
 			for (idx = PCI_BRIDGE_RESOURCES;
 			    idx < PCI_NUM_RESOURCES; idx++) {
 				r = &dev->resource[idx];
--- orig/drivers/pci/setup-bus.c	Tue Apr 10 11:33:39 2007
+++ linux/drivers/pci/setup-bus.c	Sat Apr 21 19:56:28 2007
@@ -347,9 +347,12 @@ pbus_size_mem(struct pci_bus *bus, unsig
 		
 		for (i = 0; i < PCI_NUM_RESOURCES; i++) {
 			struct resource *r = &dev->resource[i];
-			unsigned long r_size;
+			unsigned long r_size, r_flags = r->flags;
 
-			if (r->parent || (r->flags & mask) != type)
+			if (i == PCI_ROM_RESOURCE)
+				r_flags &= ~IORESOURCE_PREFETCH;
+
+			if (r->parent || (r_flags & mask) != type)
 				continue;
 			r_size = r->end - r->start + 1;
 			/* For bridges size != alignment */
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Ivan Kokshaysky]

On Tue, May 22, 2007 at 04:31:22PM -0700, Jesse Barnes wrote:
> On Tuesday, May 22, 2007, Wayne Sherman wrote:
> > If so, the D-Link is not being mapped into the 
> > right region, and the bridge it is behind does not have enough memory
> > assigned to it (ff500000-ff5fffff : PCI Bus #02).
> 
> Sounds familiar.  There are lots of cases where bridge windows aren't 
> allocated properly so devices behind them are invisible or can't work.  
> Check out the attached patch from Ivan, if you 
> pass 'pci=assign-bus-resources' to your kernel at boot time, the code will 
> try to reallocate bridge space for you if needed.

No, it won't help. The 1M range (ff500000-ff5fffff) is more than enough.
The reason why the D-Link resource is not getting assigned is rather
interesting: as Wayne wrote

> Here is the D-LINK NIC:
> # od -t x4 /sys/devices/pci0000:00/0000:00:14.4/0000:02:02.0/config
> 
> 0000000 49011186 80b00117 00000011 00004010
			    ^^^^^^
which means that the device class is 0 (not defined).
And in drivers/pci/setup-bus.c we have

		/* Don't touch classless devices or host bridges or ioapics.  */
		if (class == PCI_CLASS_NOT_DEFINED ||
		    class == PCI_CLASS_BRIDGE_HOST)
			continue;

The short term fix would be to assign proper device class to D-Link NIC
using pci quirk, but I believe a proper solution is to remove all sorts of
"if (class == PCI_xxx)" limitations (alpha, for example needs none of them)
from generic code and mark critical stuff with IORESOURCE_PCI_FIXED
in arch-specific way...

Ivan.

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Wayne Sherman]

Ivan Kokshaysky wrote:
> No, it won't help. The 1M range (ff500000-ff5fffff) is more than enough.

Good catch, I didn't look close enough at the allocations of the devices 
under the bridge.

> The reason why the D-Link resource is not getting assigned is rather
> interesting: as Wayne wrote
> 
>> Here is the D-LINK NIC:
>> # od -t x4 /sys/devices/pci0000:00/0000:00:14.4/0000:02:02.0/config
>>
>> 0000000 49011186 80b00117 00000011 00004010
> 			    ^^^^^^
> which means that the device class is 0 (not defined).
> And in drivers/pci/setup-bus.c we have
> 
> 		/* Don't touch classless devices or host bridges or ioapics.  */
> 		if (class == PCI_CLASS_NOT_DEFINED ||
> 		    class == PCI_CLASS_BRIDGE_HOST)
> 			continue;
> 
> The short term fix would be to assign proper device class to D-Link NIC
> using pci quirk...

I would like to try this, where do I find "pci quirk"?

Thanks,

Wayne

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Jesse Barnes]

On Wednesday, May 23, 2007 5:20:46 Wayne Sherman wrote:
> Ivan Kokshaysky wrote:
> > No, it won't help. The 1M range (ff500000-ff5fffff) is more than enough.
>
> Good catch, I didn't look close enough at the allocations of the devices
> under the bridge.
>
> > The reason why the D-Link resource is not getting assigned is rather
> > interesting: as Wayne wrote
> >
> >> Here is the D-LINK NIC:
> >> # od -t x4 /sys/devices/pci0000:00/0000:00:14.4/0000:02:02.0/config
> >>
> >> 0000000 49011186 80b00117 00000011 00004010
> >
> > 			    ^^^^^^
> > which means that the device class is 0 (not defined).
> > And in drivers/pci/setup-bus.c we have
> >
> > 		/* Don't touch classless devices or host bridges or ioapics.  */
> > 		if (class == PCI_CLASS_NOT_DEFINED ||
> > 		    class == PCI_CLASS_BRIDGE_HOST)
> > 			continue;
> >
> > The short term fix would be to assign proper device class to D-Link NIC
> > using pci quirk...
>
> I would like to try this, where do I find "pci quirk"?

You'll need a patch roughly like this.  I'm not sure if it should be a header 
fixup or early fixup though...

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 65d6f23..801712f 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1690,6 +1690,14 @@ static void __devinit quirk_p64h2_1k_io(struct pci_dev 
*dev)
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,	0x1460,		quirk_p64h2_1k_io);
 
+/* Give unknown D-Link network adapters a proper class */
+static void __devinit quirk_dlink_unknown(struct pci_dev *dev)
+{
+	if (dev->class = PCI_CLASS_UNKNOWN)
+		dev->class = PCI_CLASS_NETWORK_ETHERNET;
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_DLINK, 0x4901, quirk_dlink_unknown);
+
 /* Fix the IOBL_ADR for 1k I/O space granularity on the Intel P64H2
  * The IOBL_ADR gets re-written to 4k boundaries in pci_setup_bridge()
  * in drivers/pci/setup-bus.c

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Jesse Barnes]

On Wednesday, May 23, 2007 8:08:23 Jesse Barnes wrote:
> On Wednesday, May 23, 2007 5:20:46 Wayne Sherman wrote:
> > Ivan Kokshaysky wrote:
> > > No, it won't help. The 1M range (ff500000-ff5fffff) is more than
> > > enough.
> >
> > Good catch, I didn't look close enough at the allocations of the devices
> > under the bridge.
> >
> > > The reason why the D-Link resource is not getting assigned is rather
> > > interesting: as Wayne wrote
> > >
> > >> Here is the D-LINK NIC:
> > >> # od -t x4 /sys/devices/pci0000:00/0000:00:14.4/0000:02:02.0/config
> > >>
> > >> 0000000 49011186 80b00117 00000011 00004010
> > >
> > > 			    ^^^^^^
> > > which means that the device class is 0 (not defined).
> > > And in drivers/pci/setup-bus.c we have
> > >
> > > 		/* Don't touch classless devices or host bridges or ioapics.  */
> > > 		if (class == PCI_CLASS_NOT_DEFINED ||
> > > 		    class == PCI_CLASS_BRIDGE_HOST)
> > > 			continue;
> > >
> > > The short term fix would be to assign proper device class to D-Link NIC
> > > using pci quirk...
> >
> > I would like to try this, where do I find "pci quirk"?
>
> You'll need a patch roughly like this.  I'm not sure if it should be a
> header fixup or early fixup though...
>
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 65d6f23..801712f 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -1690,6 +1690,14 @@ static void __devinit quirk_p64h2_1k_io(struct
> pci_dev *dev)
>  }
>  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,	0x1460,		quirk_p64h2_1k_io);
>
> +/* Give unknown D-Link network adapters a proper class */
> +static void __devinit quirk_dlink_unknown(struct pci_dev *dev)
> +{
> +	if (dev->class = PCI_CLASS_UNKNOWN)

Err, == of course.  Obviously I didn't test this. :)

Jesse

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Ivan Kokshaysky]

On Wed, May 23, 2007 at 08:10:54PM -0700, Jesse Barnes wrote:
> > +/* Give unknown D-Link network adapters a proper class */
> > +static void __devinit quirk_dlink_unknown(struct pci_dev *dev)
> > +{
> > +	if (dev->class = PCI_CLASS_UNKNOWN)
> 
> Err, == of course.  Obviously I didn't test this. :)

Actually, it should be something like this (also untested).

Ivan.

--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1690,6 +1690,14 @@ static void __devinit quirk_p64h2_1k_io(struct pci_dev 
*dev)
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,	0x1460,		quirk_p64h2_1k_io);
 
+/* Give unknown D-Link network adapters a proper class */
+static void __devinit quirk_dlink_unknown(struct pci_dev *dev)
+{
+	if ((dev->class >> 8) == PCI_CLASS_NOT_DEFINED)
+		dev->class = PCI_CLASS_NETWORK_ETHERNET << 8;
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_DLINK, 0x4901, quirk_dlink_unknown);
+
 /* Fix the IOBL_ADR for 1k I/O space granularity on the Intel P64H2
  * The IOBL_ADR gets re-written to 4k boundaries in pci_setup_bridge()
  * in drivers/pci/setup-bus.c

Re: PCI device problem - MMCONFIG, cannot allocate resource region, resource collisions

[Wayne Sherman]

Ivan Kokshaysky wrote:
> Actually, it should be something like this (also untested).
> 
> Ivan.
> 
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -1690,6 +1690,14 @@ static void __devinit quirk_p64h2_1k_io(struct pci_dev 
> *dev)
>  }
>  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,	0x1460,		quirk_p64h2_1k_io);
>  
> +/* Give unknown D-Link network adapters a proper class */
> +static void __devinit quirk_dlink_unknown(struct pci_dev *dev)
> +{
> +	if ((dev->class >> 8) == PCI_CLASS_NOT_DEFINED)
> +		dev->class = PCI_CLASS_NETWORK_ETHERNET << 8;
> +}
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_DLINK, 0x4901, quirk_dlink_unknown);

Ivan,

   I tried a couple variations of the patch above.  It got me further, 
but I still wasn't getting the base address assigned properly.  I 
suspected a bad card, so I tried another one I have here.  It is likely 
that I have been fighting with bad hardware all this time.  The other 
card has a known device ID, a proper class code, and does not give 
resource allocation errors.  I have an e-mail into D-Link to inquire 
about the buggy card.

   I appreciate both yours and Jesse's help to troubleshoot this problem.

Best Regards,

Wayne Sherman