I know this is really just a pci issue but have CC'd LKML just in case. Sorry LKML for the noise.

I have a situation where a card on a regular PCI bus (Compro gpiohsd) does dma's
directly into another PCI cards memory (VMIC-5565 reflective memory) living on
another PCI bus. These 2 cards are sometimes seperated by many bridges.
Expansion racks etc. We've been doing this forever. No problem (mostly).

I now have an AM3 based DFI DK 790FXB-M3H5 motherboard. This board has 3 regular
PCI slots and 3 PCI-E (16x) slots. I also have a PCI-E (x4) version of the VMIC-5565
reflective memory  card in one of the PCI-E slots and our gpiohsd card in one of the regular
PCI slots. All on the motherboard. No expansion slots being used. However I cannot get
data from our gpiohsd into the PCI-E VMIC-5565 cards memory. I can certainly get the data there
from a userland buffer, no problem. Just not from one card to the other directly. Oh and when
I put the regular PCI version of the VMIC into one of the regular PCI slots everything works
as expected. They are then both on the same PCI bus and no bridges are involved though.


Here is some information about my system configuration. I would very much appreciate any
help or pointers as to why this doesn't work. I'm no PCI expert but do have a basic understanding
of it and I don't see anything wrong myself.  I'm using a vanilla 2.6.26.8 kernel BTW and all
drivers involved are GPL.


Attached is a complete lspci -vvv 

Short version:

# lspci -tv
-[0000:00]-+-00.0  ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part
------>  +-02.0-[0000:01-02]----00.0-[0000:02]----0c.0  VMIC Device 5565
|          +-04.0-[0000:03-04]----00.0-[0000:04]--
|          +-09.0-[0000:05]----00.0  Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller
|          +-0b.0-[0000:06]----00.0  nVidia Corporation G70 [GeForce 7800 GT]
|          +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [IDE mode]
|          +-14.0  ATI Technologies Inc SBx00 SMBus Controller
|          +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
|          +-14.2  ATI Technologies Inc SBx00 Azalia
|          +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
|          +-14.4-[0000:07]--+-05.0  Compro Computer Services, Inc. Device 4610
--------<  |                 \-07.0  PLX Technology, Inc. Device 0480
           +-18.0  Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration
           +-18.1  Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map
           +-18.2  Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller
           +-18.3  Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control
           \-18.4  Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control


And also the lspci -vvv info on just the cards and bridges involved.

Motherboard bridge that the VMIC card with it's internal bridge is attached to:

00:02.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port A) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: b0000000-cfffffff
        Prefetchable memory behind bridge: 00000000fde00000-00000000fdefffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Suprise- LLActRep+ BwNot+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
                Address: fee0300c  Data: 41c9
        Capabilities: [b0] Subsystem: ATI Technologies Inc Device 5956
        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
        Capabilities: [100] Vendor Specific Information <?>
        Capabilities: [110] Virtual Channel <?>
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp


VMIC card with internal bridge:

01:00.0 PCI bridge: Intel Corporation Device 032d (rev 09) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: b0000000-cfffffff
        Prefetchable memory behind bridge: 00000000fde00000-00000000fdefffff
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 <256ns, L1 unlimited
                        ClockPM- Suprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [5c] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [6c] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [d8] PCI-X bridge device
                Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=conv
                Status: Dev=01:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
                Upstream: Capacity=65535 CommitmentLimit=65535
                Downstream: Capacity=65535 CommitmentLimit=65535
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [300] Power Budgeting <?>
        Kernel modules: shpchp

02:0c.0 Network controller: VMIC Device 5565
        Subsystem: PLD APPLICATIONS Device 0080
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at cffff000 (32-bit, non-prefetchable) [size=512]
        Region 1: I/O ports at ce00 [size=256]
        Region 2: Memory at cfffe000 (32-bit, non-prefetchable) [size=64]
        Region 3: Memory at b0000000 (32-bit, non-prefetchable) [size=256M]
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-



Here is the bridge on the MB and gpiohsd card info:

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Bus: primary=00, secondary=07, subordinate=07, sec-latency=64
        I/O behind bridge: 0000a000-0000afff
        Memory behind bridge: fc800000-fd7fffff
        Prefetchable memory behind bridge: fdb00000-fdbfffff
        Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

07:07.0 Intelligent controller [0e80]: PLX Technology, Inc. Device 0480 (rev 52)
        Subsystem: Compro Computer Services, Inc. Device 4620
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 22
        BIST result: 00
        Region 0: Memory at fd7fe000 (32-bit, non-prefetchable) [size=1K]
        Region 1: Memory at fc800000 (32-bit, non-prefetchable) [size=8M]
        Region 2: Memory at fd7fd000 (32-bit, non-prefetchable) [size=16]
        Capabilities: [40] #00 [0000]
        Capabilities: [54] #00 [0080]
        Capabilities: [58] Vital Product Data <?>


Again, any help as to why this doesn't work would be appreciated. 

Thanks in advance
Mark