Re: dmar pte read access not set error messages on hp dl388 gen8 systems

[Lu Baolu <baolu.lu@linux.intel.com>]

Hi,

On 12/5/19 10:53 AM, Jerry Snitselaar wrote:
> On Thu Dec 05 19, Lu Baolu wrote:
>> Hi,
>>
>> On 12/5/19 10:25 AM, Jerry Snitselaar wrote:
>>>>
>>>> It seems that iommu pci bus probe didn't enumerate device [01:00.2] and
>>>> [02:00.0], the corresponding context entries were not setup. Hence dma
>>>> fault generated when devices access the memory.
>>>>
>>>> Do these two devices show in "lspci" output? How do these devices get
>>>> enumerated by the system?
>>>>
>>>> Best regards,
>>>> baolu
>>>>
>>>
>>> They are there in the output, but it seems out of order:
>>
>>
>> [   23.446201] pci 0000:01:00.0: Adding to iommu group 25
>> [   23.448949] pci 0000:01:00.0: Using iommu dma mapping
>> [   23.450807] pci 0000:01:00.1: Adding to iommu group 25
>> [   23.452666] pci 0000:01:00.1: DMAR: Device uses a private identity 
>> domain.
>> [   23.455063] pci 0000:01:00.2: Adding to iommu group 25
>> [   23.456881] pci 0000:01:00.4: Adding to iommu group 25
>> [   23.458693] pci 0000:01:00.4: DMAR: Device uses a private identity 
>> domain.
>>
>> Oh, yes!
>>
>> So device 01:00.0 01:00.1 01:00.2 01:00.4 share a single group. The
>> default domain for this group has been set to DMA although iommu=pt has
>> been set. As the result, .0 .2 use DMA, but .1, .4 use IDENTITY. This is
>> not a valid configuration since all devices in a group should use a same
>> domain.
>>
>> Do you mind posting the "lspci -vvv" output of these devices? I want to
>> figure out why these devices request different domain type.
>>
>> Best regards,
>> baolu
>>
> 
> 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out 
> Standard Slave Instrumentation & System Support (rev 05)
>      Subsystem: Hewlett-Packard Company iLO4
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 0, Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 19
>      NUMA node: 0
>      Region 0: I/O ports at 3000 [size=256]
>      Region 1: Memory at f7bf0000 (32-bit, non-prefetchable) [size=512]
>      Region 2: I/O ports at 3400 [size=256]
>      Capabilities: [78] Power Management version 3
>          Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>          Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>      Capabilities: [b0] MSI: Enable- Count=1/1 Maskable- 64bit+
>          Address: 0000000000000000  Data: 0000
>      Capabilities: [c0] Express (v1) Legacy Endpoint, MSI 00
>          DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s 
> unlimited, L1 unlimited
>              ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>          DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ 
> Unsupported-
>              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>              MaxPayload 128 bytes, MaxReadReq 128 bytes
>          DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- 
> TransPend-
>          LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit 
> Latency L0s <4us
>              ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>          LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>          LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- 
> DLActive- BWMgmt- ABWMgmt-
>      Kernel driver in use: hpwdt
>      Kernel modules: hpwdt
> 
> 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA 
> G200EH (prog-if 00 [VGA controller])
>      Subsystem: Hewlett-Packard Company iLO4
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 0, Cache Line Size: 64 bytes
>      Interrupt: pin B routed to IRQ 16
>      NUMA node: 0
>      Region 0: Memory at f5000000 (32-bit, prefetchable) [size=16M]
>      Region 1: Memory at f7be0000 (32-bit, non-prefetchable) [size=16K]
>      Region 2: Memory at f7000000 (32-bit, non-prefetchable) [size=8M]
>      [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
>      Capabilities: [a8] Power Management version 3
>          Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>          Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>      Capabilities: [b0] MSI: Enable- Count=1/1 Maskable- 64bit+
>          Address: 0000000000000000  Data: 0000
>      Capabilities: [c0] Express (v1) Legacy Endpoint, MSI 00
>          DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s 
> unlimited, L1 unlimited
>              ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>          DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ 
> Unsupported-
>              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>              MaxPayload 128 bytes, MaxReadReq 128 bytes
>          DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- 
> TransPend-
>          LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit 
> Latency L0s <4us
>              ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>          LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>          LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- 
> DLActive- BWMgmt- ABWMgmt-
>      Kernel driver in use: mgag200
>      Kernel modules: mgag200
> 
> 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out 
> Standard Management Processor Support and Messaging (rev 05)
>      Subsystem: Hewlett-Packard Company iLO4
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 0, Cache Line Size: 64 bytes
>      Interrupt: pin B routed to IRQ 16
>      NUMA node: 0
>      Region 0: I/O ports at 3800 [size=256]
>      Region 1: Memory at f6ff0000 (32-bit, non-prefetchable) [size=256]
>      Region 2: Memory at f6e00000 (32-bit, non-prefetchable) [size=1M]
>      Region 3: Memory at f6d80000 (32-bit, non-prefetchable) [size=512K]
>      Region 4: Memory at f6d70000 (32-bit, non-prefetchable) [size=32K]
>      Region 5: Memory at f6d60000 (32-bit, non-prefetchable) [size=32K]
>      [virtual] Expansion ROM at f6d00000 [disabled] [size=64K]
>      Capabilities: [78] Power Management version 3
>          Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>          Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>      Capabilities: [b0] MSI: Enable- Count=1/1 Maskable- 64bit+
>          Address: 0000000000000000  Data: 0000
>      Capabilities: [c0] Express (v1) Legacy Endpoint, MSI 00
>          DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s 
> unlimited, L1 unlimited
>              ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>          DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ 
> Unsupported-
>              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>              MaxPayload 128 bytes, MaxReadReq 128 bytes
>          DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- 
> TransPend-
>          LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit 
> Latency L0s <4us
>              ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>          LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>          LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- 
> DLActive- BWMgmt- ABWMgmt-
>      Kernel driver in use: hpilo
>      Kernel modules: hpilo
> 
> 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out 
> Standard Virtual USB Controller (rev 02) (prog-if 00 [UHCI])
>      Subsystem: Hewlett-Packard Company iLO4
>      Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 0, Cache Line Size: 64 bytes
>      Interrupt: pin B routed to IRQ 16
>      NUMA node: 0
>      Region 4: I/O ports at 3c00 [size=32]
>      Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
>          Address: 0000000000000000  Data: 0000
>      Capabilities: [80] Express (v1) Legacy Endpoint, MSI 00
>          DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s 
> unlimited, L1 unlimited
>              ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>          DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ 
> Unsupported-
>              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>              MaxPayload 128 bytes, MaxReadReq 128 bytes
>          DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- 
> TransPend-
>          LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit 
> Latency L0s <4us
>              ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>          LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>          LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- 
> DLActive- BWMgmt- ABWMgmt-
>      Capabilities: [f0] Power Management version 3
>          Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>          Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>      Kernel driver in use: uhci_hcd
> 
> 02:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 
> Controllers (rev 01)
>      Subsystem: Hewlett-Packard Company P420i
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR- FastB2B- DisINTx+
>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 0, Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 52
>      NUMA node: 0
>      Region 0: Memory at f7d00000 (64-bit, non-prefetchable) [size=1M]
>      Region 2: Memory at f7cf0000 (64-bit, non-prefetchable) [size=1K]
>      Region 4: I/O ports at 5000 [size=256]
>      [virtual] Expansion ROM at f7c00000 [disabled] [size=512K]
>      Capabilities: [80] Power Management version 3
>          Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
> PME(D0+,D1+,D2-,D3hot+,D3cold-)
>          Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>      Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
>          Address: 0000000000000000  Data: 0000
>      Capabilities: [b0] MSI-X: Enable+ Count=64 Masked-
>          Vector table: BAR=0 offset=00002000
>          PBA: BAR=0 offset=00003000
>      Capabilities: [c0] Express (v2) Endpoint, MSI 00
>          DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, 
> L1 <1us
>              ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- 
> SlotPowerLimit 0.000W
>          DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ 
> Unsupported-
>              RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
>              MaxPayload 256 bytes, MaxReadReq 4096 bytes
>          DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- 
> TransPend-
>          LnkCap:    Port #0, Speed 8GT/s, Width x8, ASPM not supported
>              ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
>          LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>          LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk- 
> DLActive- BWMgmt- ABWMgmt-
>          DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF 
> Via message
>               AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
>          DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, 
> OBFF Disabled
>               AtomicOpsCtl: ReqEn-
>          LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
>               Transmit Margin: Normal Operating Range, 
> EnterModifiedCompliance- ComplianceSOS-
>               Compliance De-emphasis: -6dB
>          LnkSta2: Current De-emphasis Level: -3.5dB, 
> EqualizationComplete+, EqualizationPhase1+
>               EqualizationPhase2+, EqualizationPhase3+, 
> LinkEqualizationRequest-
>      Capabilities: [100 v2] Advanced Error Reporting
>          UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- 
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>          UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- 
> RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
>          UESvrt:    DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- 
> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>          CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>          CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>          AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- 
> ECRCChkCap+ ECRCChkEn-
>              MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>          HeaderLog: 00000000 00000000 00000000 00000000
>      Capabilities: [300 v1] #19
>      Kernel driver in use: hpsa
>      Kernel modules: hpsa
> 

Can you please try below change? Let's check whether the afending
address has been mapped for device 01.00.2.

$ git diff
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index db7bfd4f2d20..d9daf66be849 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -663,6 +663,8 @@ static int iommu_group_create_direct_mappings(struct 
iommu_group *group,
                         ret = iommu_map(domain, addr, addr, pg_size, 
entry->prot);
                         if (ret)
                                 goto out;
+
+                       dev_info(dev, "Setting identity map [0x%Lx - 
0x%Lx] for group %d\n", addr, addr + pg_size, group->id);
                 }

         }

I am doubting that device 01.00.2 is not in the device scope of

[    4.485108] DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff

By the way, does device 01.00.2 works well after binding the driver?

Best regards,
baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu