* Seeing some another issue with mixed domains in the same iommu_group
@ 2020-02-06 17:54 Jerry Snitselaar
2020-02-06 18:47 ` Jerry Snitselaar
0 siblings, 1 reply; 5+ messages in thread
From: Jerry Snitselaar @ 2020-02-06 17:54 UTC (permalink / raw)
To: iommu, Lu Baolu
Hi Baolu,
I'm seeing another issue with the devices in the HP ilo when the
system is booted with intel_iommu=on and iommu=pt (iommu=nopt does not
run into problems).
first system:
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
[ 21.208103] pci 0000:01:00.0: Adding to iommu group 24
[ 21.210911] pci 0000:01:00.0: Using iommu dma mapping
[ 21.212635] pci 0000:01:00.1: Adding to iommu group 24
[ 21.214326] pci 0000:01:00.1: Device uses a private identity domain.
[ 21.216507] pci 0000:01:00.2: Adding to iommu group 24
[ 21.618173] pci 0000:01:00.4: Adding to iommu group 24
[ 21.619839] pci 0000:01:00.4: Device uses a private identity domain.
[ 26.206832] uhci_hcd: USB Universal Host Controller Interface driver
[ 26.209044] uhci_hcd 0000:01:00.4: UHCI Host Controller
[ 26.210897] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3
[ 26.213247] uhci_hcd 0000:01:00.4: detected 8 ports
[ 26.214810] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports
[ 26.217153] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[ 26.219171] uhci_hcd 0000:01:00.4: 32bit DMA uses non-identity mapping
[ 26.221261] uhci_hcd 0000:01:00.4: unable to allocate consistent memory for frame list
[ 26.223787] uhci_hcd 0000:01:00.4: startup error -16
[ 26.225381] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[ 26.227378] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[ 26.229296] uhci_hcd: probe of 0000:01:00.4 failed with error -16
different system with similar issue:
01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 07)
01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200eH3 [102b:0538] (rev 02) (prog-if 00 [VGA controller])
01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 07)
01:00.4 USB controller [0c03]: Hewlett-Packard Company iLO5 Virtual USB Controller [103c:22f6] (prog-if 20 [EHCI])
[ 13.695663] pci 0000:01:00.0: Adding to iommu group 10
[ 13.703667] pci 0000:01:00.0: Using iommu dma mapping
[ 13.708871] pci 0000:01:00.1: Adding to iommu group 10
[ 13.714033] pci 0000:01:00.1: DMAR: Device uses a private identity domain.
[ 13.721033] pci 0000:01:00.2: Adding to iommu group 10
[ 13.726290] pci 0000:01:00.4: Adding to iommu group 10
[ 13.731453] pci 0000:01:00.4: DMAR: Device uses a private identity domain.
[ 17.157796] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 17.164348] ehci-pci: EHCI PCI platform driver
[ 17.170061] ehci-pci 0000:01:00.4: EHCI Host Controller
[ 17.175457] ehci-pci 0000:01:00.4: new USB bus registered, assigned bus number 1
[ 17.182912] ehci-pci 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping
[ 17.189988] ehci-pci 0000:01:00.4: can't setup: -12
[ 17.194884] ehci-pci 0000:01:00.4: USB bus 1 deregistered
[ 17.200567] ehci-pci 0000:01:00.4: init 0000:01:00.4 fail, -12
[ 17.206508] ehci-pci: probe of 0000:01:00.4 failed with error -12
I'm looking through the code and trying to debug it, but any thoughts on this?
Regards,
Jerry
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Seeing some another issue with mixed domains in the same iommu_group
2020-02-06 17:54 Seeing some another issue with mixed domains in the same iommu_group Jerry Snitselaar
@ 2020-02-06 18:47 ` Jerry Snitselaar
2020-02-06 21:36 ` Jerry Snitselaar
0 siblings, 1 reply; 5+ messages in thread
From: Jerry Snitselaar @ 2020-02-06 18:47 UTC (permalink / raw)
To: iommu, Lu Baolu
On Thu Feb 06 20, Jerry Snitselaar wrote:
>Hi Baolu,
>
>I'm seeing another issue with the devices in the HP ilo when the
>system is booted with intel_iommu=on and iommu=pt (iommu=nopt does not
>run into problems).
>
>first system:
>
>01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
>01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
>01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
>01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
>
>[ 21.208103] pci 0000:01:00.0: Adding to iommu group 24
>[ 21.210911] pci 0000:01:00.0: Using iommu dma mapping
>[ 21.212635] pci 0000:01:00.1: Adding to iommu group 24
>[ 21.214326] pci 0000:01:00.1: Device uses a private identity domain.
>[ 21.216507] pci 0000:01:00.2: Adding to iommu group 24
>[ 21.618173] pci 0000:01:00.4: Adding to iommu group 24
>[ 21.619839] pci 0000:01:00.4: Device uses a private identity domain.
>
>[ 26.206832] uhci_hcd: USB Universal Host Controller Interface driver
>[ 26.209044] uhci_hcd 0000:01:00.4: UHCI Host Controller
>[ 26.210897] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3
>[ 26.213247] uhci_hcd 0000:01:00.4: detected 8 ports
>[ 26.214810] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports
>[ 26.217153] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
>[ 26.219171] uhci_hcd 0000:01:00.4: 32bit DMA uses non-identity mapping
>[ 26.221261] uhci_hcd 0000:01:00.4: unable to allocate consistent memory for frame list
>[ 26.223787] uhci_hcd 0000:01:00.4: startup error -16
>[ 26.225381] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
>[ 26.227378] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
>[ 26.229296] uhci_hcd: probe of 0000:01:00.4 failed with error -16
>
>
>different system with similar issue:
>
>01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 07)
>01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200eH3 [102b:0538] (rev 02) (prog-if 00 [VGA controller])
>01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 07)
>01:00.4 USB controller [0c03]: Hewlett-Packard Company iLO5 Virtual USB Controller [103c:22f6] (prog-if 20 [EHCI])
>
>[ 13.695663] pci 0000:01:00.0: Adding to iommu group 10
>[ 13.703667] pci 0000:01:00.0: Using iommu dma mapping
>[ 13.708871] pci 0000:01:00.1: Adding to iommu group 10
>[ 13.714033] pci 0000:01:00.1: DMAR: Device uses a private identity domain.
>[ 13.721033] pci 0000:01:00.2: Adding to iommu group 10
>[ 13.726290] pci 0000:01:00.4: Adding to iommu group 10
>[ 13.731453] pci 0000:01:00.4: DMAR: Device uses a private identity domain.
>
>[ 17.157796] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
>[ 17.164348] ehci-pci: EHCI PCI platform driver
>[ 17.170061] ehci-pci 0000:01:00.4: EHCI Host Controller
>[ 17.175457] ehci-pci 0000:01:00.4: new USB bus registered, assigned bus number 1
>[ 17.182912] ehci-pci 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping
>[ 17.189988] ehci-pci 0000:01:00.4: can't setup: -12
>[ 17.194884] ehci-pci 0000:01:00.4: USB bus 1 deregistered
>[ 17.200567] ehci-pci 0000:01:00.4: init 0000:01:00.4 fail, -12
>[ 17.206508] ehci-pci: probe of 0000:01:00.4 failed with error -12
>
>
>I'm looking through the code and trying to debug it, but any thoughts on this?
>
>Regards,
>Jerry
In iommu_need_mapping, in a case like the above does something like dmar_insert_one_dev_info need to
happen to associate the device back with the group default domain? In intel_iommu_add_device it is
going to get removed and added to the identity domain, and then in iommu_need_mapping it gets removed
from the identity domain, and iommu_request_dma_domain_for_dev should return 0 because the group
default domain at this point is the correct type.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Seeing some another issue with mixed domains in the same iommu_group
2020-02-06 18:47 ` Jerry Snitselaar
@ 2020-02-06 21:36 ` Jerry Snitselaar
2020-02-06 23:16 ` Jerry Snitselaar
0 siblings, 1 reply; 5+ messages in thread
From: Jerry Snitselaar @ 2020-02-06 21:36 UTC (permalink / raw)
To: iommu, Lu Baolu
On Thu Feb 06 20, Jerry Snitselaar wrote:
>On Thu Feb 06 20, Jerry Snitselaar wrote:
>>Hi Baolu,
>>
>>I'm seeing another issue with the devices in the HP ilo when the
>>system is booted with intel_iommu=on and iommu=pt (iommu=nopt does not
>>run into problems).
>>
>>first system:
>>
>>01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
>>01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
>>01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
>>01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
>>
>>[ 21.208103] pci 0000:01:00.0: Adding to iommu group 24
>>[ 21.210911] pci 0000:01:00.0: Using iommu dma mapping
>>[ 21.212635] pci 0000:01:00.1: Adding to iommu group 24
>>[ 21.214326] pci 0000:01:00.1: Device uses a private identity domain.
>>[ 21.216507] pci 0000:01:00.2: Adding to iommu group 24
>>[ 21.618173] pci 0000:01:00.4: Adding to iommu group 24
>>[ 21.619839] pci 0000:01:00.4: Device uses a private identity domain.
>>
>>[ 26.206832] uhci_hcd: USB Universal Host Controller Interface driver
>>[ 26.209044] uhci_hcd 0000:01:00.4: UHCI Host Controller
>>[ 26.210897] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3
>>[ 26.213247] uhci_hcd 0000:01:00.4: detected 8 ports
>>[ 26.214810] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports
>>[ 26.217153] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
>>[ 26.219171] uhci_hcd 0000:01:00.4: 32bit DMA uses non-identity mapping
>>[ 26.221261] uhci_hcd 0000:01:00.4: unable to allocate consistent memory for frame list
>>[ 26.223787] uhci_hcd 0000:01:00.4: startup error -16
>>[ 26.225381] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
>>[ 26.227378] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
>>[ 26.229296] uhci_hcd: probe of 0000:01:00.4 failed with error -16
>>
>>
>>different system with similar issue:
>>
>>01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 07)
>>01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200eH3 [102b:0538] (rev 02) (prog-if 00 [VGA controller])
>>01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 07)
>>01:00.4 USB controller [0c03]: Hewlett-Packard Company iLO5 Virtual USB Controller [103c:22f6] (prog-if 20 [EHCI])
>>
>>[ 13.695663] pci 0000:01:00.0: Adding to iommu group 10
>>[ 13.703667] pci 0000:01:00.0: Using iommu dma mapping
>>[ 13.708871] pci 0000:01:00.1: Adding to iommu group 10
>>[ 13.714033] pci 0000:01:00.1: DMAR: Device uses a private identity domain.
>>[ 13.721033] pci 0000:01:00.2: Adding to iommu group 10
>>[ 13.726290] pci 0000:01:00.4: Adding to iommu group 10
>>[ 13.731453] pci 0000:01:00.4: DMAR: Device uses a private identity domain.
>>
>>[ 17.157796] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
>>[ 17.164348] ehci-pci: EHCI PCI platform driver
>>[ 17.170061] ehci-pci 0000:01:00.4: EHCI Host Controller
>>[ 17.175457] ehci-pci 0000:01:00.4: new USB bus registered, assigned bus number 1
>>[ 17.182912] ehci-pci 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping
>>[ 17.189988] ehci-pci 0000:01:00.4: can't setup: -12
>>[ 17.194884] ehci-pci 0000:01:00.4: USB bus 1 deregistered
>>[ 17.200567] ehci-pci 0000:01:00.4: init 0000:01:00.4 fail, -12
>>[ 17.206508] ehci-pci: probe of 0000:01:00.4 failed with error -12
>>
>>
>>I'm looking through the code and trying to debug it, but any thoughts on this?
>>
>>Regards,
>>Jerry
>
>In iommu_need_mapping, in a case like the above does something like dmar_insert_one_dev_info need to
>happen to associate the device back with the group default domain? In intel_iommu_add_device it is
>going to get removed and added to the identity domain, and then in iommu_need_mapping it gets removed
>from the identity domain, and iommu_request_dma_domain_for_dev should return 0 because the group
>default domain at this point is the correct type.
The above cases seem to be avoided by:
9235cb13d7d1 | 2020-01-24 | iommu/vt-d: Allow devices with RMRRs to use identity domain (Lu Baolu)
which results in the watchdog device no longer taking a dma domain and switching the group default.
Without that patch though when it gets into the iommu_need_mapping code for 0000:01:00.4 after
the following:
dmar_remove_one_dev_info(dev);
ret = iommu_request_dma_domain_for_dev(dev);
ret is 0 and dev->archdata.iommu is NULL. Even with 9235cb13d7d1 device_def_domain_type can return
return dma, but I'm not sure how likely it is for there to be an iommu group like that again where
the group default ends up dma, a device gets removed and added to the identity domain, and then
ends up in that code in iommu_need_mapping.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Seeing some another issue with mixed domains in the same iommu_group
2020-02-06 21:36 ` Jerry Snitselaar
@ 2020-02-06 23:16 ` Jerry Snitselaar
2020-02-07 7:19 ` Lu Baolu
0 siblings, 1 reply; 5+ messages in thread
From: Jerry Snitselaar @ 2020-02-06 23:16 UTC (permalink / raw)
To: iommu, Lu Baolu
On Thu Feb 06 20, Jerry Snitselaar wrote:
>
...
>The above cases seem to be avoided by:
>
>9235cb13d7d1 | 2020-01-24 | iommu/vt-d: Allow devices with RMRRs to use identity domain (Lu Baolu)
>
>which results in the watchdog device no longer taking a dma domain and switching the group default.
>
>
>Without that patch though when it gets into the iommu_need_mapping code for 0000:01:00.4 after
>the following:
>
> dmar_remove_one_dev_info(dev);
> ret = iommu_request_dma_domain_for_dev(dev);
>
>ret is 0 and dev->archdata.iommu is NULL. Even with 9235cb13d7d1 device_def_domain_type can return
>return dma, but I'm not sure how likely it is for there to be an iommu group like that again where
>the group default ends up dma, a device gets removed and added to the identity domain, and then
>ends up in that code in iommu_need_mapping.
>
>
Hi Baolu,
Would something along these lines makes sense?
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 9dc37672bf89..40cc8f5a3ebb 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3614,6 +3614,20 @@ static bool iommu_need_mapping(struct device *dev)
}
dmar_remove_one_dev_info(dev);
get_private_domain_for_dev(dev);
+ } else {
+ if (dev->archdata.iommu == NULL) {
+ struct iommu_domain *domain;
+ struct iommu_group *group;
+ struct dmar_domain *dmar_domain, *tmp;
+
+ group = iommu_group_get_for_dev(dev);
+ domain = iommu_group_default_domain(group);
+ dmar_domain = to_dmar_domain(domain);
+ tmp = set_domain_for_dev(dev, dmar_domain);
+ }
}
dev_info(dev, "32bit DMA uses non-identity mapping\n");
--
Obviously needs some checks added, but this was just an initial test I
was trying.
Regards,
Jerry
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: Seeing some another issue with mixed domains in the same iommu_group
2020-02-06 23:16 ` Jerry Snitselaar
@ 2020-02-07 7:19 ` Lu Baolu
0 siblings, 0 replies; 5+ messages in thread
From: Lu Baolu @ 2020-02-07 7:19 UTC (permalink / raw)
To: Jerry Snitselaar, iommu
Hi Jerry,
On 2020/2/7 7:16, Jerry Snitselaar wrote:
>
> Hi Baolu,
>
> Would something along these lines makes sense?
>
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 9dc37672bf89..40cc8f5a3ebb 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -3614,6 +3614,20 @@ static bool iommu_need_mapping(struct device *dev)
> }
> dmar_remove_one_dev_info(dev);
> get_private_domain_for_dev(dev);
> + } else {
> + if (dev->archdata.iommu == NULL) {
> + struct iommu_domain *domain;
> + struct iommu_group *group;
> + struct dmar_domain *dmar_domain, *tmp;
> +
> + group = iommu_group_get_for_dev(dev);
> + domain = iommu_group_default_domain(group);
> + dmar_domain = to_dmar_domain(domain);
> + tmp = set_domain_for_dev(dev, dmar_domain);
> + }
> }
>
> dev_info(dev, "32bit DMA uses non-identity mapping\n");
Thanks for reporting.
Actually, I prefer to removing this domain switch as long as users are
able to make a 32-bit device use DMA domain while system default is
identity or it breaks anything. 32-bit devices (or normally devices
with limited addressing capability over the whole system memory) using
DMA domain helps by removing the swiotlb performance overhead, which is
the original motivation of this code.
Best regards,
baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-02-07 7:19 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-02-06 17:54 Seeing some another issue with mixed domains in the same iommu_group Jerry Snitselaar
2020-02-06 18:47 ` Jerry Snitselaar
2020-02-06 21:36 ` Jerry Snitselaar
2020-02-06 23:16 ` Jerry Snitselaar
2020-02-07 7:19 ` Lu Baolu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox