* [PATCH] arm64: dts: juno: Enable smmu_pcie for Juno r1/r2
@ 2019-03-20 8:31 Leo Yan
2019-03-20 11:24 ` Robin Murphy
0 siblings, 1 reply; 3+ messages in thread
From: Leo Yan @ 2019-03-20 8:31 UTC (permalink / raw)
To: Liviu Dudau, Sudeep Holla, Lorenzo Pieralisi, Rob Herring,
Mark Rutland, Robin Murphy, linux-arm-kernel, devicetree,
linux-kernel
Cc: Leo Yan
Though PCIe controller has been enabled on Juno r1/r2, but it misses to
enable its connected SMMU. From the testing, even without set this SMMU
status property to 'okay', the PCIe NIC device still works well. Since
the SMMU is not enabled in DT binding and its iommu_groups is not
created properly, finally this prevents to enable vfio-pci for NIC
device with KVM.
After reviewing the code in dts, Juno r1/r2 both have enabled PCIe
controller but Juno r0 board (in juno.dts) doesn't contain PCIe
controller; hence this patch only change SMMU status property to 'okay'
for Juno r1/r2 dts files for the sake of only enabling it for Juno r1/r2
and avoiding touching Juno r0.
Committer testing on Juno r2:
Before:
root@debian:~# tree /sys/kernel/iommu_groups/
/sys/kernel/iommu_groups/
├── 0
│ ├── devices
│ │ ├── 7ffb0000.ohci -> ../../../../devices/platform/7ffb0000.ohci
│ │ └── 7ffc0000.ehci -> ../../../../devices/platform/7ffc0000.ehci
│ ├── reserved_regions
│ └── type
└── 1
├── devices
│ └── 20070000.etr -> ../../../../devices/platform/20070000.etr
├── reserved_regions
└── type
7 directories, 4 files
After:
root@debian:~# tree /sys/kernel/iommu_groups/
/sys/kernel/iommu_groups/
├── 0
│ ├── devices
│ │ ├── 0000:00:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0
│ │ ├── 0000:01:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0
│ │ ├── 0000:02:01.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:01.0
│ │ ├── 0000:02:02.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:02.0
│ │ ├── 0000:02:03.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:03.0
│ │ ├── 0000:02:0c.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:0c.0
│ │ ├── 0000:02:10.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:10.0
│ │ ├── 0000:02:1f.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:1f.0
│ │ ├── 0000:03:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:01.0/0000:03:00.0
│ │ └── 0000:08:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:1f.0/0000:08:00.0
│ ├── reserved_regions
│ └── type
├── 1
│ ├── devices
│ │ ├── 7ffb0000.ohci -> ../../../../devices/platform/7ffb0000.ohci
│ │ └── 7ffc0000.ehci -> ../../../../devices/platform/7ffc0000.ehci
│ ├── reserved_regions
│ └── type
└── 2
├── devices
│ └── 20070000.etr -> ../../../../devices/platform/20070000.etr
├── reserved_regions
└── type
19 directories, 6 files
Signed-off-by: Leo Yan <leo.yan@linaro.org>
---
arch/arm64/boot/dts/arm/juno-r1.dts | 4 ++++
arch/arm64/boot/dts/arm/juno-r2.dts | 4 ++++
2 files changed, 8 insertions(+)
diff --git a/arch/arm64/boot/dts/arm/juno-r1.dts b/arch/arm64/boot/dts/arm/juno-r1.dts
index b2b7ced633cf..67e161a6272a 100644
--- a/arch/arm64/boot/dts/arm/juno-r1.dts
+++ b/arch/arm64/boot/dts/arm/juno-r1.dts
@@ -226,6 +226,10 @@
status = "okay";
};
+&smmu_pcie {
+ status = "okay";
+};
+
&pcie_ctlr {
status = "okay";
};
diff --git a/arch/arm64/boot/dts/arm/juno-r2.dts b/arch/arm64/boot/dts/arm/juno-r2.dts
index ab77adb4f3c2..0e1c5c814b01 100644
--- a/arch/arm64/boot/dts/arm/juno-r2.dts
+++ b/arch/arm64/boot/dts/arm/juno-r2.dts
@@ -226,6 +226,10 @@
status = "okay";
};
+&smmu_pcie {
+ status = "okay";
+};
+
&pcie_ctlr {
status = "okay";
};
--
2.17.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH] arm64: dts: juno: Enable smmu_pcie for Juno r1/r2
2019-03-20 8:31 [PATCH] arm64: dts: juno: Enable smmu_pcie for Juno r1/r2 Leo Yan
@ 2019-03-20 11:24 ` Robin Murphy
2019-03-20 11:44 ` Leo Yan
0 siblings, 1 reply; 3+ messages in thread
From: Robin Murphy @ 2019-03-20 11:24 UTC (permalink / raw)
To: Leo Yan, Liviu Dudau, Sudeep Holla, Lorenzo Pieralisi,
Rob Herring, Mark Rutland, linux-arm-kernel, devicetree,
linux-kernel
Hi Leo,
On 20/03/2019 08:31, Leo Yan wrote:
> Though PCIe controller has been enabled on Juno r1/r2, but it misses to
> enable its connected SMMU. From the testing, even without set this SMMU
> status property to 'okay', the PCIe NIC device still works well. Since
> the SMMU is not enabled in DT binding and its iommu_groups is not
> created properly, finally this prevents to enable vfio-pci for NIC
> device with KVM.
FWIW, whilst it might appear to be fine, there are still potential
issues once the DMA API sees this SMMU and starts using it, which is why
this particular change remains as one of my oldest local hacks that I've
never yet considered upstreamable.
The IOMMU DMA ops happen to work for light usage now as a side-effect of
122fac030e912 combined with the current top-down allocation behaviour,
but as soon as IOVA usage/fragmentation leads to 32-bit DMA addresses
below 0x8000000, or full 40-bit addresses, being allocated then data
loss and other problems will happen (for the reasons explained on the
other thread). Similarly, it's also going to be fragile to any internal
changes in the IOVA allocator.
Until we have a decent solution for handling such 'windowed' DMA
restrictions (there do exist other systems with similar behaviour) I'd
be very wary of enabling the PCIe SMMU for all users who may not be
aware of the caveats. Given that the lack of stream IDs and SR-IOV
support prevents Juno from being a realistic virtualisation platform
anyway, I'm not convinced there's enough benefit to justify the risk.
Thanks,
Robin.
> After reviewing the code in dts, Juno r1/r2 both have enabled PCIe
> controller but Juno r0 board (in juno.dts) doesn't contain PCIe
> controller; hence this patch only change SMMU status property to 'okay'
> for Juno r1/r2 dts files for the sake of only enabling it for Juno r1/r2
> and avoiding touching Juno r0.
>
> Committer testing on Juno r2:
>
> Before:
>
> root@debian:~# tree /sys/kernel/iommu_groups/
> /sys/kernel/iommu_groups/
> ├── 0
> │ ├── devices
> │ │ ├── 7ffb0000.ohci -> ../../../../devices/platform/7ffb0000.ohci
> │ │ └── 7ffc0000.ehci -> ../../../../devices/platform/7ffc0000.ehci
> │ ├── reserved_regions
> │ └── type
> └── 1
> ├── devices
> │ └── 20070000.etr -> ../../../../devices/platform/20070000.etr
> ├── reserved_regions
> └── type
>
> 7 directories, 4 files
>
> After:
>
> root@debian:~# tree /sys/kernel/iommu_groups/
> /sys/kernel/iommu_groups/
> ├── 0
> │ ├── devices
> │ │ ├── 0000:00:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0
> │ │ ├── 0000:01:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0
> │ │ ├── 0000:02:01.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:01.0
> │ │ ├── 0000:02:02.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:02.0
> │ │ ├── 0000:02:03.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:03.0
> │ │ ├── 0000:02:0c.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:0c.0
> │ │ ├── 0000:02:10.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:10.0
> │ │ ├── 0000:02:1f.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:1f.0
> │ │ ├── 0000:03:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:01.0/0000:03:00.0
> │ │ └── 0000:08:00.0 -> ../../../../devices/platform/40000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/0000:02:1f.0/0000:08:00.0
> │ ├── reserved_regions
> │ └── type
> ├── 1
> │ ├── devices
> │ │ ├── 7ffb0000.ohci -> ../../../../devices/platform/7ffb0000.ohci
> │ │ └── 7ffc0000.ehci -> ../../../../devices/platform/7ffc0000.ehci
> │ ├── reserved_regions
> │ └── type
> └── 2
> ├── devices
> │ └── 20070000.etr -> ../../../../devices/platform/20070000.etr
> ├── reserved_regions
> └── type
>
> 19 directories, 6 files
>
> Signed-off-by: Leo Yan <leo.yan@linaro.org>
> ---
> arch/arm64/boot/dts/arm/juno-r1.dts | 4 ++++
> arch/arm64/boot/dts/arm/juno-r2.dts | 4 ++++
> 2 files changed, 8 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/arm/juno-r1.dts b/arch/arm64/boot/dts/arm/juno-r1.dts
> index b2b7ced633cf..67e161a6272a 100644
> --- a/arch/arm64/boot/dts/arm/juno-r1.dts
> +++ b/arch/arm64/boot/dts/arm/juno-r1.dts
> @@ -226,6 +226,10 @@
> status = "okay";
> };
>
> +&smmu_pcie {
> + status = "okay";
> +};
> +
> &pcie_ctlr {
> status = "okay";
> };
> diff --git a/arch/arm64/boot/dts/arm/juno-r2.dts b/arch/arm64/boot/dts/arm/juno-r2.dts
> index ab77adb4f3c2..0e1c5c814b01 100644
> --- a/arch/arm64/boot/dts/arm/juno-r2.dts
> +++ b/arch/arm64/boot/dts/arm/juno-r2.dts
> @@ -226,6 +226,10 @@
> status = "okay";
> };
>
> +&smmu_pcie {
> + status = "okay";
> +};
> +
> &pcie_ctlr {
> status = "okay";
> };
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] arm64: dts: juno: Enable smmu_pcie for Juno r1/r2
2019-03-20 11:24 ` Robin Murphy
@ 2019-03-20 11:44 ` Leo Yan
0 siblings, 0 replies; 3+ messages in thread
From: Leo Yan @ 2019-03-20 11:44 UTC (permalink / raw)
To: Robin Murphy
Cc: Mark Rutland, devicetree, Lorenzo Pieralisi, Liviu Dudau,
linux-kernel, Rob Herring, Sudeep Holla, linux-arm-kernel
Hi Robin,
On Wed, Mar 20, 2019 at 11:24:18AM +0000, Robin Murphy wrote:
> Hi Leo,
>
> On 20/03/2019 08:31, Leo Yan wrote:
> > Though PCIe controller has been enabled on Juno r1/r2, but it misses to
> > enable its connected SMMU. From the testing, even without set this SMMU
> > status property to 'okay', the PCIe NIC device still works well. Since
> > the SMMU is not enabled in DT binding and its iommu_groups is not
> > created properly, finally this prevents to enable vfio-pci for NIC
> > device with KVM.
>
> FWIW, whilst it might appear to be fine, there are still potential issues
> once the DMA API sees this SMMU and starts using it, which is why this
> particular change remains as one of my oldest local hacks that I've never
> yet considered upstreamable.
>
> The IOMMU DMA ops happen to work for light usage now as a side-effect of
> 122fac030e912 combined with the current top-down allocation behaviour, but
> as soon as IOVA usage/fragmentation leads to 32-bit DMA addresses below
> 0x8000000, or full 40-bit addresses, being allocated then data loss and
> other problems will happen (for the reasons explained on the other thread).
> Similarly, it's also going to be fragile to any internal changes in the IOVA
> allocator.
>
> Until we have a decent solution for handling such 'windowed' DMA
> restrictions (there do exist other systems with similar behaviour) I'd be
> very wary of enabling the PCIe SMMU for all users who may not be aware of
> the caveats. Given that the lack of stream IDs and SR-IOV support prevents
> Juno from being a realistic virtualisation platform anyway, I'm not
> convinced there's enough benefit to justify the risk.
Ah, I didn't connect the 'windowde' DMA restriction with the IOMMU
property setting in dt, which actually is deliberate.
Please drop this patch and will leave it to you. Thanks a lot for
related info.
[...]
Thanks,
Leo Yan
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-03-20 11:45 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-03-20 8:31 [PATCH] arm64: dts: juno: Enable smmu_pcie for Juno r1/r2 Leo Yan
2019-03-20 11:24 ` Robin Murphy
2019-03-20 11:44 ` Leo Yan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox