Re: AMD_IOV: IO_PAGE_FALT trying to pass through Mellanox ConnectX HCA (debian testing)

[Ward Vandewege <ward@gnu.org>]

On Mon, Feb 07, 2011 at 11:41:33AM -0500, Konrad Rzeszutek Wilk wrote:
> Joerg,
> 
> Any idea what this error might signify?
> > (XEN) AMD_IOV: IO_PAGE_FALT: domain:3, device id:0x200, fault address:0x7e7ca000
> > (XEN) AMD_IOV: IO_PAGE_FALT: domain:3, device id:0x200, fault address:0x7e7ca040
> > (XEN) AMD_IOV: IO_PAGE_FALT: domain:3, device id:0x200, fault address:0x7e7ca080
> > (XEN) AMD_IOV: IO_PAGE_FALT: domain:3, device id:0x200, fault address:0x7e7ca0c0
> 
> We have been stabing in the dark enabling certain knobs, .. but I am
> just curious - the fault address - that is the real physical address right?
> >From the looks of it looks like a normal RAM region, not the PCI BAR space - the
> AMD VI chipset doesn't really distinguish between those, or does it?
> 
> Ward, can you post your lspci -vvv -s 02:00.0 output? I am curious to see
> what the PCI BAR space is.

Of course, here it is. Booted into 2.6.32-5-xen-amd64 #1 SMP Wed Jan 12
05:46:49 UTC 2011 x86_64 GNU/Linux, from the dom0:

# lspci -vvv -s 02:00.0 
02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
    Subsystem: Super Micro Computer Inc Device 0048
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 19
    Region 0: Memory at fea00000 (64-bit, non-prefetchable) [size=1M]
    Region 2: Memory at fc800000 (64-bit, prefetchable) [size=8M]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Connection timed out
        Not readable
    Capabilities: [9c] MSI-X: Enable- Count=256 Masked-
        Vector table: BAR=0 offset=0007c000
        PBA: BAR=0 offset=0007d000
    Capabilities: [60] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #8, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot-
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB
    Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap: MFVC- ACS-, Next Function: 1
        ARICtl: MFVC- ACS-, Function Group: 0
    Kernel driver in use: pciback

Thanks,
Ward.