[PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Marcin Wojtas]

Armada 38x platforms marks all devices as coherent via
mvebu_hwcc_notifier(), whereas the standard way to determine
this is by of_dma_is_coherent(). Reflect the hardware
capabilities by adding 'dma-coherent' properties to the device tree.

Signed-off-by: Marcin Wojtas <mw@semihalf.com>
---
 arch/arm/boot/dts/armada-380.dtsi | 1 +
 arch/arm/boot/dts/armada-385.dtsi | 1 +
 arch/arm/boot/dts/armada-38x.dtsi | 1 +
 3 files changed, 3 insertions(+)

diff --git a/arch/arm/boot/dts/armada-380.dtsi b/arch/arm/boot/dts/armada-380.dtsi
index ce1dddb2269b..25d17550e2fc 100644
--- a/arch/arm/boot/dts/armada-380.dtsi
+++ b/arch/arm/boot/dts/armada-380.dtsi
@@ -38,6 +38,7 @@ pcie {
 			compatible = "marvell,armada-370-pcie";
 			status = "disabled";
 			device_type = "pci";
+			dma-coherent;
 
 			#address-cells = <3>;
 			#size-cells = <2>;
diff --git a/arch/arm/boot/dts/armada-385.dtsi b/arch/arm/boot/dts/armada-385.dtsi
index 83392b92dae2..6fb8c254cbdc 100644
--- a/arch/arm/boot/dts/armada-385.dtsi
+++ b/arch/arm/boot/dts/armada-385.dtsi
@@ -37,6 +37,7 @@ pciec: pcie {
 			compatible = "marvell,armada-370-pcie";
 			status = "disabled";
 			device_type = "pci";
+			dma-coherent;
 
 			#address-cells = <3>;
 			#size-cells = <2>;
diff --git a/arch/arm/boot/dts/armada-38x.dtsi b/arch/arm/boot/dts/armada-38x.dtsi
index 446861b6b17b..5801873dfcbe 100644
--- a/arch/arm/boot/dts/armada-38x.dtsi
+++ b/arch/arm/boot/dts/armada-38x.dtsi
@@ -102,6 +102,7 @@ internal-regs {
 			#address-cells = <1>;
 			#size-cells = <1>;
 			ranges = <0 MBUS_ID(0xf0, 0x01) 0 0x100000>;
+			dma-coherent;
 
 			sdramc: sdramc@1400 {
 				compatible = "marvell,armada-xp-sdram-controller";
-- 
2.29.0

Re: [PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Andrew Lunn]

On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote:
> Armada 38x platforms marks all devices as coherent via
> mvebu_hwcc_notifier(), whereas the standard way to determine
> this is by of_dma_is_coherent(). Reflect the hardware
> capabilities by adding 'dma-coherent' properties to the device tree.

Hi Marcin

Does this need to go to -rc for 6.0? The DMA issues being reported?
If so, please add a Fixed: tag.

   Andrew

Re: [PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Russell King (Oracle)]

On Sun, Oct 23, 2022 at 05:04:01PM +0200, Andrew Lunn wrote:
> On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote:
> > Armada 38x platforms marks all devices as coherent via
> > mvebu_hwcc_notifier(), whereas the standard way to determine
> > this is by of_dma_is_coherent(). Reflect the hardware
> > capabilities by adding 'dma-coherent' properties to the device tree.
> 
> Hi Marcin
> 
> Does this need to go to -rc for 6.0? The DMA issues being reported?
> If so, please add a Fixed: tag.

Are we absolutely sure this makes sense?

Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies
on stuff such as whether the kernel is in SMP mode or not (because
the page tables need to be appropriately marked as shared for
coherency with IO to work). We only enable the shared bit if we're
in SMP mode because (a) its difficult to do at runtime due to TLB
conflicts (requires switching the MMU off, rewriting the page tables
and switching the MMU back on), and (b) setting the shared bit for
CPUs that don't need it _can_ result in the CPUs basically bypassing
their caches and thus kill system performance.

So, if we have Armada 38x platforms that are operated in uniprocessor
mode, this patch can cause havoc on such a setup.

I would suggest utmost caution with this approach.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Marcin Wojtas]

niedz., 23 paź 2022 o 18:21 Russell King (Oracle)
<linux@armlinux.org.uk> napisał(a):
>
> On Sun, Oct 23, 2022 at 05:04:01PM +0200, Andrew Lunn wrote:
> > On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote:
> > > Armada 38x platforms marks all devices as coherent via
> > > mvebu_hwcc_notifier(), whereas the standard way to determine
> > > this is by of_dma_is_coherent(). Reflect the hardware
> > > capabilities by adding 'dma-coherent' properties to the device tree.
> >
> > Hi Marcin
> >
> > Does this need to go to -rc for 6.0? The DMA issues being reported?
> > If so, please add a Fixed: tag.
>
> Are we absolutely sure this makes sense?
>
> Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies
> on stuff such as whether the kernel is in SMP mode or not (because
> the page tables need to be appropriately marked as shared for
> coherency with IO to work). We only enable the shared bit if we're
> in SMP mode because (a) its difficult to do at runtime due to TLB
> conflicts (requires switching the MMU off, rewriting the page tables
> and switching the MMU back on), and (b) setting the shared bit for
> CPUs that don't need it _can_ result in the CPUs basically bypassing
> their caches and thus kill system performance.
>
> So, if we have Armada 38x platforms that are operated in uniprocessor
> mode, this patch can cause havoc on such a setup.
>
> I would suggest utmost caution with this approach.
>

Sure. In such a case the description of 380 variant (single core)
should remain untouched.

We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
- Don't change current behavior, i.e. perform a necessary kernel
configuration in "arm,pl310-cache" driver,
arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
- Satisfy of_dma_is_coherent() by adding `dma-coherent;` in
armada-385.dtsi only (IMO this would describe HW properly)
?

Best regards,
Marcin

Re: [PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Marek Behún]

On Sun, 23 Oct 2022 23:30:34 +0200
Marcin Wojtas <mw@semihalf.com> wrote:

> niedz., 23 paź 2022 o 18:21 Russell King (Oracle)
> <linux@armlinux.org.uk> napisał(a):
> >
> > On Sun, Oct 23, 2022 at 05:04:01PM +0200, Andrew Lunn wrote:  
> > > On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote:  
> > > > Armada 38x platforms marks all devices as coherent via
> > > > mvebu_hwcc_notifier(), whereas the standard way to determine
> > > > this is by of_dma_is_coherent(). Reflect the hardware
> > > > capabilities by adding 'dma-coherent' properties to the device tree.  
> > >
> > > Hi Marcin
> > >
> > > Does this need to go to -rc for 6.0? The DMA issues being reported?
> > > If so, please add a Fixed: tag.  
> >
> > Are we absolutely sure this makes sense?
> >
> > Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies
> > on stuff such as whether the kernel is in SMP mode or not (because
> > the page tables need to be appropriately marked as shared for
> > coherency with IO to work). We only enable the shared bit if we're
> > in SMP mode because (a) its difficult to do at runtime due to TLB
> > conflicts (requires switching the MMU off, rewriting the page tables
> > and switching the MMU back on), and (b) setting the shared bit for
> > CPUs that don't need it _can_ result in the CPUs basically bypassing
> > their caches and thus kill system performance.
> >
> > So, if we have Armada 38x platforms that are operated in uniprocessor
> > mode, this patch can cause havoc on such a setup.
> >
> > I would suggest utmost caution with this approach.
> >  
> 
> Sure. In such a case the description of 380 variant (single core)
> should remain untouched.
> 
> We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
> - Don't change current behavior, i.e. perform a necessary kernel
> configuration in "arm,pl310-cache" driver,
> arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
> - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in
> armada-385.dtsi only (IMO this would describe HW properly)
> ?

It will describe HW properly, but someone running older kernel compiled
with no SMP support will see a performance drop. I wonder how many
people do that.

Marek

Re: [PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Russell King (Oracle)]

On Mon, Oct 24, 2022 at 08:51:02AM +0200, Marek Behún wrote:
> > Sure. In such a case the description of 380 variant (single core)
> > should remain untouched.
> > 
> > We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
> > - Don't change current behavior, i.e. perform a necessary kernel
> > configuration in "arm,pl310-cache" driver,
> > arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
> > - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in
> > armada-385.dtsi only (IMO this would describe HW properly)
> > ?
> 
> It will describe HW properly, but someone running older kernel compiled
> with no SMP support will see a performance drop. I wonder how many
> people do that.

If the kernel is built without SMP support, the page table entries will
not have the shared bit set, and the system will _not_ be DMA-coherent.
Having DT mark devices as "dma-coherent" in this case will lead to data
corruption, because the DMA API will believe them to be DMA-coherent
when the page tables are not setup for that to work.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH] ARM: dts: armada-38x: Mark devices as dma-coherent

[Marcin Wojtas]

pon., 24 paź 2022 o 09:51 Russell King (Oracle)
<linux@armlinux.org.uk> napisał(a):
>
> On Mon, Oct 24, 2022 at 08:51:02AM +0200, Marek Behún wrote:
> > > Sure. In such a case the description of 380 variant (single core)
> > > should remain untouched.
> > >
> > > We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about:
> > > - Don't change current behavior, i.e. perform a necessary kernel
> > > configuration in "arm,pl310-cache" driver,
> > > arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT
> > > - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in
> > > armada-385.dtsi only (IMO this would describe HW properly)
> > > ?
> >
> > It will describe HW properly, but someone running older kernel compiled
> > with no SMP support will see a performance drop. I wonder how many
> > people do that.
>
> If the kernel is built without SMP support, the page table entries will
> not have the shared bit set, and the system will _not_ be DMA-coherent.
> Having DT mark devices as "dma-coherent" in this case will lead to data
> corruption, because the DMA API will believe them to be DMA-coherent
> when the page tables are not setup for that to work.
>

Thanks, for the explanation. Since we're heavily dependent on what
happens in the kernel we boot, it will be easier to just drop this
patch and keep using the DT as-is.

Best regards,
Marcin