[PATCH 0/5] arm: mvebu: Enable LPAE support for Armada XP SoCs

[PATCH 0/5] arm: mvebu: Enable LPAE support for Armada XP SoCs

[Gregory CLEMENT]

Hello,

The Armada XP SoCs have LPAE support. This patch set allow to run
kernel on this SoCs with LPAE support.

The biggest changes are the conversion of the device tree file to 64
bits in order to be able to use more than 4GB of memory (without this
the LPAE is pointless).

This patch set have been tested on a Armada XP GP board with 8GB of
DRAM with LPAE selected. It have been tested on Armada XP DB board
with 3GB of DRAM with and without LPAE. And also on the Armada 370 DB
board (without LPAE) to check that no regression appeared.

This patch set is based on 3.9-rc3 and is 3.10 material. The
git branch called lpae is available at:
https://github.com/MISL-EBU-System-SW/mainline-public.git.

Thanks,

Gregory CLEMENT (1):
  arm: dts: Convert mvebu device tree files to 64 bits

Lior Amsalem (4):
  arm: mvebu: Aligne the internal registers virtual base to support
    LPAE
  arm: mvebu: Select DMA_BOUNCE when LPAE is selected in Kconfig
  arm: mvebu: Enable pj4b on LPAE compilations
  arm: dts: Add a 64 bits version of the skeleton device tree

 arch/arm/boot/dts/armada-370-db.dts              |    2 +-
 arch/arm/boot/dts/armada-370-mirabox.dts         |    2 +-
 arch/arm/boot/dts/armada-370-rd.dts              |    2 +-
 arch/arm/boot/dts/armada-370-xp.dtsi             |   45 +++++++++++-----------
 arch/arm/boot/dts/armada-370.dtsi                |   28 +++++++-------
 arch/arm/boot/dts/armada-xp-db.dts               |    2 +-
 arch/arm/boot/dts/armada-xp-gp.dts               |   14 ++++---
 arch/arm/boot/dts/armada-xp-mv78230.dtsi         |    6 +--
 arch/arm/boot/dts/armada-xp-mv78260.dtsi         |   10 ++---
 arch/arm/boot/dts/armada-xp-mv78460.dtsi         |   10 ++---
 arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts |    2 +-
 arch/arm/boot/dts/armada-xp.dtsi                 |   34 ++++++++--------
 arch/arm/boot/dts/skeleton64.dtsi                |   13 +++++++
 arch/arm/include/debug/mvebu.S                   |    2 +-
 arch/arm/mach-mvebu/Kconfig                      |    1 +
 arch/arm/mach-mvebu/armada-370-xp.h              |    2 +-
 arch/arm/mm/proc-v7.S                            |    3 +-
 17 files changed, 98 insertions(+), 80 deletions(-)
 create mode 100644 arch/arm/boot/dts/skeleton64.dtsi

-- 
1.7.9.5

[PATCH 1/5] arm: mvebu: Aligne the internal registers virtual base to support LPAE

[Gregory CLEMENT]

From: Lior Amsalem <alior-eYqpPyKDWXRBDgjK7y7TUQ@public.gmane.org>

In order to be able to support he LPAE, the internal registers virtual
base must be aligned to 2MB.

Signed-off-by: Lior Amsalem <alior-eYqpPyKDWXRBDgjK7y7TUQ@public.gmane.org>
Signed-off-by: Gregory CLEMENT <gregory.clement-wi1+55ScJUtKEb57/3fJTNBPR1lH4CV8@public.gmane.org>
---
 arch/arm/include/debug/mvebu.S      |    2 +-
 arch/arm/mach-mvebu/armada-370-xp.h |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm/include/debug/mvebu.S b/arch/arm/include/debug/mvebu.S
index 865c6d0..df191af 100644
--- a/arch/arm/include/debug/mvebu.S
+++ b/arch/arm/include/debug/mvebu.S
@@ -12,7 +12,7 @@
 */
 
 #define ARMADA_370_XP_REGS_PHYS_BASE	0xd0000000
-#define ARMADA_370_XP_REGS_VIRT_BASE	0xfeb00000
+#define ARMADA_370_XP_REGS_VIRT_BASE	0xfec00000
 
 	.macro	addruart, rp, rv, tmp
 	ldr	\rp, =ARMADA_370_XP_REGS_PHYS_BASE
diff --git a/arch/arm/mach-mvebu/armada-370-xp.h b/arch/arm/mach-mvebu/armada-370-xp.h
index c6a7d74..c49c08e 100644
--- a/arch/arm/mach-mvebu/armada-370-xp.h
+++ b/arch/arm/mach-mvebu/armada-370-xp.h
@@ -16,7 +16,7 @@
 #define __MACH_ARMADA_370_XP_H
 
 #define ARMADA_370_XP_REGS_PHYS_BASE	0xd0000000
-#define ARMADA_370_XP_REGS_VIRT_BASE	IOMEM(0xfeb00000)
+#define ARMADA_370_XP_REGS_VIRT_BASE	IOMEM(0xfec00000)
 #define ARMADA_370_XP_REGS_SIZE		SZ_1M
 
 #ifdef CONFIG_SMP
-- 
1.7.9.5

[PATCH 2/5] arm: mvebu: Select DMA_BOUNCE when LPAE is selected in Kconfig

[Gregory CLEMENT]

From: Lior Amsalem <alior@marvell.com>

For mvebu IOs are 32 bits and we have 40 bits memory due to LPAE so
make sure we give 32 bits addresses to the IOs.

Signed-off-by: Lior Amsalem <alior@marvell.com>
Tested-by: Franklin <flin@marvell.com>
Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 arch/arm/mach-mvebu/Kconfig |    1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm/mach-mvebu/Kconfig b/arch/arm/mach-mvebu/Kconfig
index 440b13e..617da94 100644
--- a/arch/arm/mach-mvebu/Kconfig
+++ b/arch/arm/mach-mvebu/Kconfig
@@ -13,6 +13,7 @@ config ARCH_MVEBU
 	select MVEBU_CLK_CORE
 	select MVEBU_CLK_CPU
 	select MVEBU_CLK_GATING
+	select DMABOUNCE if ARM_LPAE
 
 if ARCH_MVEBU
 
-- 
1.7.9.5

Re: [PATCH 2/5] arm: mvebu: Select DMA_BOUNCE when LPAE is selected in Kconfig

[Rob Herring]

On 03/21/2013 11:26 AM, Gregory CLEMENT wrote:
> From: Lior Amsalem <alior@marvell.com>
> 
> For mvebu IOs are 32 bits and we have 40 bits memory due to LPAE so
> make sure we give 32 bits addresses to the IOs.
> 
> Signed-off-by: Lior Amsalem <alior@marvell.com>
> Tested-by: Franklin <flin@marvell.com>
> Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
> ---
>  arch/arm/mach-mvebu/Kconfig |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/arm/mach-mvebu/Kconfig b/arch/arm/mach-mvebu/Kconfig
> index 440b13e..617da94 100644
> --- a/arch/arm/mach-mvebu/Kconfig
> +++ b/arch/arm/mach-mvebu/Kconfig
> @@ -13,6 +13,7 @@ config ARCH_MVEBU
>  	select MVEBU_CLK_CORE
>  	select MVEBU_CLK_CPU
>  	select MVEBU_CLK_GATING
> +	select DMABOUNCE if ARM_LPAE

This alone is not enough to use bounce buffers. Am I missing something?

Rob

>  
>  if ARCH_MVEBU
>  
> 

Re: [PATCH 2/5] arm: mvebu: Select DMA_BOUNCE when LPAE is selected in Kconfig

[Andrew Lunn]

On Thu, Mar 21, 2013 at 05:26:15PM +0100, Gregory CLEMENT wrote:
> From: Lior Amsalem <alior@marvell.com>
> 
> For mvebu IOs are 32 bits and we have 40 bits memory due to LPAE so
> make sure we give 32 bits addresses to the IOs.

Hi Gregory, Lior

I don't really understand what this comment is supposed to mean. I
would of expect DMA and bounce to appear at least....

> Signed-off-by: Lior Amsalem <alior@marvell.com>
> Tested-by: Franklin <flin@marvell.com>
> Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
> ---
>  arch/arm/mach-mvebu/Kconfig |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/arm/mach-mvebu/Kconfig b/arch/arm/mach-mvebu/Kconfig
> index 440b13e..617da94 100644
> --- a/arch/arm/mach-mvebu/Kconfig
> +++ b/arch/arm/mach-mvebu/Kconfig
> @@ -13,6 +13,7 @@ config ARCH_MVEBU
>  	select MVEBU_CLK_CORE
>  	select MVEBU_CLK_CPU
>  	select MVEBU_CLK_GATING
> +	select DMABOUNCE if ARM_LPAE
>  
>  if ARCH_MVEBU
>  
> -- 
> 1.7.9.5
> 

Re: [PATCH 2/5] arm: mvebu: Select DMA_BOUNCE when LPAE is selected in Kconfig

[Russell King - ARM Linux]

On Thu, Mar 21, 2013 at 05:26:15PM +0100, Gregory CLEMENT wrote:
> From: Lior Amsalem <alior-eYqpPyKDWXRBDgjK7y7TUQ@public.gmane.org>
> 
> For mvebu IOs are 32 bits and we have 40 bits memory due to LPAE so
> make sure we give 32 bits addresses to the IOs.
> 
> Signed-off-by: Lior Amsalem <alior-eYqpPyKDWXRBDgjK7y7TUQ@public.gmane.org>
> Tested-by: Franklin <flin-eYqpPyKDWXRBDgjK7y7TUQ@public.gmane.org>
> Signed-off-by: Gregory CLEMENT <gregory.clement-wi1+55ScJUtKEb57/3fJTNBPR1lH4CV8@public.gmane.org>

Oh god no.  Please move away from the addition on DMABOUNCE - that code
creaks, doesn't have highmem support, and is known to give problems on
various platforms.

Instead, please rely on using the DMA mask and such like, just like on
x86.

Re: [PATCH 2/5] arm: mvebu: Select DMA_BOUNCE when LPAE is selected in Kconfig

[Krzysztof Halasa]

Russell King - ARM Linux <linux-lFZ/pmaqli7XmaaqVzeoHQ@public.gmane.org> writes:

> Oh god no.  Please move away from the addition on DMABOUNCE - that code
> creaks, doesn't have highmem support, and is known to give problems on
> various platforms.
>
> Instead, please rely on using the DMA mask and such like, just like on
> x86.

It still needs some sort of bouncing (for data passed to the device from
upper levels). I wonder if swiotlb doesn't do the same thing.
-- 
Krzysztof Halasa

[PATCH 3/5] arm: mvebu: Enable pj4b on LPAE compilations

[Gregory CLEMENT]

From: Lior Amsalem <alior@marvell.com>

pj4b cpus are LPAE capable so enable them on LPAE compilations

Signed-off-by: Lior Amsalem <alior@marvell.com>
Tested-by: Franklin <flin@marvell.com>
Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 arch/arm/mm/proc-v7.S |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index 3a3c015..159c747 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -402,6 +402,8 @@ __v7_ca9mp_proc_info:
 	__v7_proc __v7_ca9mp_setup
 	.size	__v7_ca9mp_proc_info, . - __v7_ca9mp_proc_info
 
+#endif	/* CONFIG_ARM_LPAE */
+
 	/*
 	 * Marvell PJ4B processor.
 	 */
@@ -411,7 +413,6 @@ __v7_pj4b_proc_info:
 	.long	0xfffffff0
 	__v7_proc __v7_pj4b_setup
 	.size	__v7_pj4b_proc_info, . - __v7_pj4b_proc_info
-#endif	/* CONFIG_ARM_LPAE */
 
 	/*
 	 * ARM Ltd. Cortex A7 processor.
-- 
1.7.9.5

[PATCH 4/5] arm: dts: Add a 64 bits version of the skeleton device tree

[Gregory CLEMENT]

From: Lior Amsalem <alior@marvell.com>

In order to be able to use more than 4GB address-cells and size-cells
have to be set to 2

Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
Signed-off-by: Lior Amsalem <alior@marvell.com>
---
 arch/arm/boot/dts/skeleton64.dtsi |   13 +++++++++++++
 1 file changed, 13 insertions(+)
 create mode 100644 arch/arm/boot/dts/skeleton64.dtsi

diff --git a/arch/arm/boot/dts/skeleton64.dtsi b/arch/arm/boot/dts/skeleton64.dtsi
new file mode 100644
index 0000000..1599415
--- /dev/null
+++ b/arch/arm/boot/dts/skeleton64.dtsi
@@ -0,0 +1,13 @@
+/*
+ * Skeleton device tree in the 64 bits version; the bare minimum
+ * needed to boot; just include and add a compatible value.  The
+ * bootloader will typically populate the memory node.
+ */
+
+/ {
+	#address-cells = <2>;
+	#size-cells = <2>;
+	chosen { };
+	aliases { };
+	memory { device_type = "memory"; reg = <0 0>; };
+};
-- 
1.7.9.5

[PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Gregory CLEMENT]

In order to be able to use more than 4GB of RAM when the LPAE is
activated, the dts must be converted in 64 bits.

Armada XP and Armada 370 share a dtsi file which have also be
converted to 64 bits. This lead to convert all the device tree files
to 64 bits even the one used for Armada 370 (which don't support
LPAE)

This was heavily based on the work of Lior Amsalem.

Signed-off-by: Lior Amsalem <alior@marvell.com>
Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
---
 arch/arm/boot/dts/armada-370-db.dts              |    2 +-
 arch/arm/boot/dts/armada-370-mirabox.dts         |    2 +-
 arch/arm/boot/dts/armada-370-rd.dts              |    2 +-
 arch/arm/boot/dts/armada-370-xp.dtsi             |   45 +++++++++++-----------
 arch/arm/boot/dts/armada-370.dtsi                |   28 +++++++-------
 arch/arm/boot/dts/armada-xp-db.dts               |    2 +-
 arch/arm/boot/dts/armada-xp-gp.dts               |   14 ++++---
 arch/arm/boot/dts/armada-xp-mv78230.dtsi         |    6 +--
 arch/arm/boot/dts/armada-xp-mv78260.dtsi         |   10 ++---
 arch/arm/boot/dts/armada-xp-mv78460.dtsi         |   10 ++---
 arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts |    2 +-
 arch/arm/boot/dts/armada-xp.dtsi                 |   34 ++++++++--------
 12 files changed, 80 insertions(+), 77 deletions(-)

diff --git a/arch/arm/boot/dts/armada-370-db.dts b/arch/arm/boot/dts/armada-370-db.dts
index e34b280..0bde82c 100644
--- a/arch/arm/boot/dts/armada-370-db.dts
+++ b/arch/arm/boot/dts/armada-370-db.dts
@@ -26,7 +26,7 @@
 
 	memory {
 		device_type = "memory";
-		reg = <0x00000000 0x40000000>; /* 1 GB */
+		reg = <0 0x00000000 0 0x40000000>; /* 1 GB */
 	};
 
 	soc {
diff --git a/arch/arm/boot/dts/armada-370-mirabox.dts b/arch/arm/boot/dts/armada-370-mirabox.dts
index dd0c57d..fd08cd6 100644
--- a/arch/arm/boot/dts/armada-370-mirabox.dts
+++ b/arch/arm/boot/dts/armada-370-mirabox.dts
@@ -21,7 +21,7 @@
 
 	memory {
 		device_type = "memory";
-		reg = <0x00000000 0x20000000>; /* 512 MB */
+		reg = <0 0x00000000 0 0x20000000>; /* 512 MB */
 	};
 
 	soc {
diff --git a/arch/arm/boot/dts/armada-370-rd.dts b/arch/arm/boot/dts/armada-370-rd.dts
index 070bba4..e51948d 100644
--- a/arch/arm/boot/dts/armada-370-rd.dts
+++ b/arch/arm/boot/dts/armada-370-rd.dts
@@ -24,7 +24,7 @@
 
 	memory {
 		device_type = "memory";
-		reg = <0x00000000 0x20000000>; /* 512 MB */
+		reg = <0 0x00000000 0 0x20000000>; /* 512 MB */
 	};
 
 	soc {
diff --git a/arch/arm/boot/dts/armada-370-xp.dtsi b/arch/arm/boot/dts/armada-370-xp.dtsi
index 5b70820..562f24c 100644
--- a/arch/arm/boot/dts/armada-370-xp.dtsi
+++ b/arch/arm/boot/dts/armada-370-xp.dtsi
@@ -15,8 +15,7 @@
  * This file contains the definitions that are common to the Armada
  * 370 and Armada XP SoC.
  */
-
-/include/ "skeleton.dtsi"
+/include/ "skeleton64.dtsi"
 
 / {
 	model = "Marvell Armada 370 and XP SoC";
@@ -37,20 +36,20 @@
 
 	coherency-fabric@d0020200 {
 		compatible = "marvell,coherency-fabric";
-		reg = <0xd0020200 0xb0>,
-		      <0xd0021810 0x1c>;
+		reg = <0 0xd0020200 0 0xb0>,
+		      <0 0xd0021810 0 0x1c>;
 	};
 
 	soc {
-		#address-cells = <1>;
-		#size-cells = <1>;
+		#address-cells = <2>;
+		#size-cells = <2>;
 		compatible = "simple-bus";
 		interrupt-parent = <&mpic>;
 		ranges;
 
 		serial@d0012000 {
 				compatible = "snps,dw-apb-uart";
-				reg = <0xd0012000 0x100>;
+				reg = <0 0xd0012000 0 0x100>;
 				reg-shift = <2>;
 				interrupts = <41>;
 				reg-io-width = <1>;
@@ -58,7 +57,7 @@
 		};
 		serial@d0012100 {
 				compatible = "snps,dw-apb-uart";
-				reg = <0xd0012100 0x100>;
+				reg = <0 0xd0012100 0 0x100>;
 				reg-shift = <2>;
 				interrupts = <42>;
 				reg-io-width = <1>;
@@ -67,20 +66,20 @@
 
 		timer@d0020300 {
 			       compatible = "marvell,armada-370-xp-timer";
-			       reg = <0xd0020300 0x30>,
-			       <0xd0021040 0x30>;
+			       reg = <0 0xd0020300 0 0x30>,
+			       <0 0xd0021040 0 0x30>;
 			       interrupts = <37>, <38>, <39>, <40>, <5>, <6>;
 			       clocks = <&coreclk 2>;
 		};
 
 		addr-decoding@d0020000 {
 			compatible = "marvell,armada-addr-decoding-controller";
-			reg = <0xd0020000 0x258>;
+			reg = <0 0xd0020000 0 0x258>;
 		};
 
 		sata@d00a0000 {
 			compatible = "marvell,orion-sata";
-			reg = <0xd00a0000 0x2400>;
+			reg = <0 0xd00a0000 0 0x2400>;
 			interrupts = <55>;
 			clocks = <&gateclk 15>, <&gateclk 30>;
 			clock-names = "0", "1";
@@ -91,12 +90,12 @@
 			#address-cells = <1>;
 			#size-cells = <0>;
 			compatible = "marvell,orion-mdio";
-			reg = <0xd0072004 0x4>;
+			reg = <0 0xd0072004 0 0x4>;
 		};
 
 		ethernet@d0070000 {
 				compatible = "marvell,armada-370-neta";
-				reg = <0xd0070000 0x2500>;
+				reg = <0 0xd0070000 0 0x2500>;
 				interrupts = <8>;
 				clocks = <&gateclk 4>;
 				status = "disabled";
@@ -104,7 +103,7 @@
 
 		ethernet@d0074000 {
 				compatible = "marvell,armada-370-neta";
-				reg = <0xd0074000 0x2500>;
+				reg = <0 0xd0074000 0 0x2500>;
 				interrupts = <10>;
 				clocks = <&gateclk 3>;
 				status = "disabled";
@@ -112,7 +111,7 @@
 
 		i2c0: i2c@d0011000 {
 			compatible = "marvell,mv64xxx-i2c";
-			reg = <0xd0011000 0x20>;
+			reg = <0 0xd0011000 0 0x20>;
 			#address-cells = <1>;
 			#size-cells = <0>;
 			interrupts = <31>;
@@ -123,7 +122,7 @@
 
 		i2c1: i2c@d0011100 {
 			compatible = "marvell,mv64xxx-i2c";
-			reg = <0xd0011100 0x20>;
+			reg = <0 0xd0011100 0 0x20>;
 			#address-cells = <1>;
 			#size-cells = <0>;
 			interrupts = <32>;
@@ -134,13 +133,13 @@
 
 		rtc@10300 {
 			compatible = "marvell,orion-rtc";
-			reg = <0xd0010300 0x20>;
+			reg = <0 0xd0010300 0 0x20>;
 			interrupts = <50>;
 		};
 
 		mvsdio@d00d4000 {
 			compatible = "marvell,orion-sdio";
-			reg = <0xd00d4000 0x200>;
+			reg = <0 0xd00d4000 0 0x200>;
 			interrupts = <54>;
 			clocks = <&gateclk 17>;
 			status = "disabled";
@@ -148,21 +147,21 @@
 
 		usb@d0050000 {
 			compatible = "marvell,orion-ehci";
-			reg = <0xd0050000 0x500>;
+			reg = <0 0xd0050000 0 0x500>;
 			interrupts = <45>;
 			status = "disabled";
 		};
 
 		usb@d0051000 {
 			compatible = "marvell,orion-ehci";
-			reg = <0xd0051000 0x500>;
+			reg = <0 0xd0051000 0 0x500>;
 			interrupts = <46>;
 			status = "disabled";
 		};
 
 		spi0: spi@d0010600 {
 			compatible = "marvell,orion-spi";
-			reg = <0xd0010600 0x28>;
+			reg = <0 0xd0010600 0 0x28>;
 			#address-cells = <1>;
 			#size-cells = <0>;
 			cell-index = <0>;
@@ -173,7 +172,7 @@
 
 		spi1: spi@d0010680 {
 			compatible = "marvell,orion-spi";
-			reg = <0xd0010680 0x28>;
+			reg = <0 0xd0010680 0 0x28>;
 			#address-cells = <1>;
 			#size-cells = <0>;
 			cell-index = <1>;
diff --git a/arch/arm/boot/dts/armada-370.dtsi b/arch/arm/boot/dts/armada-370.dtsi
index 8188d13..f757880 100644
--- a/arch/arm/boot/dts/armada-370.dtsi
+++ b/arch/arm/boot/dts/armada-370.dtsi
@@ -22,7 +22,7 @@
 	compatible = "marvell,armada370", "marvell,armada-370-xp";
 	L2: l2-cache {
 		compatible = "marvell,aurora-outer-cache";
-		reg = <0xd0008000 0x1000>;
+		reg = <0 0xd0008000 0 0x1000>;
 		cache-id-part = <0x100>;
 		wt-override;
 	};
@@ -34,19 +34,19 @@
 	};
 
 	mpic: interrupt-controller@d0020000 {
-	      reg = <0xd0020a00 0x1d0>,
-		    <0xd0021870 0x58>;
+	      reg = <0 0xd0020a00 0 0x1d0>,
+		    <0 0xd0021870 0 0x58>;
 	};
 
 	soc {
 		system-controller@d0018200 {
 				compatible = "marvell,armada-370-xp-system-controller";
-				reg = <0xd0018200 0x100>;
+				reg = <0 0xd0018200 0 0x100>;
 		};
 
 		pinctrl {
 			compatible = "marvell,mv88f6710-pinctrl";
-			reg = <0xd0018000 0x38>;
+			reg = <0 0xd0018000 0 0x38>;
 
 			sdio_pins1: sdio-pins1 {
 			      marvell,pins = "mpp9",  "mpp11", "mpp12",
@@ -63,7 +63,7 @@
 
 		gpio0: gpio@d0018100 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018100 0x40>;
+			reg = <0 0xd0018100 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -74,7 +74,7 @@
 
 		gpio1: gpio@d0018140 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018140 0x40>;
+			reg = <0 0xd0018140 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -85,7 +85,7 @@
 
 		gpio2: gpio@d0018180 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018180 0x40>;
+			reg = <0 0xd0018180 0 0x40>;
 			ngpios = <2>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -96,21 +96,21 @@
 
 		coreclk: mvebu-sar@d0018230 {
 			compatible = "marvell,armada-370-core-clock";
-			reg = <0xd0018230 0x08>;
+			reg = <0 0xd0018230 0 0x08>;
 			#clock-cells = <1>;
 		};
 
 		gateclk: clock-gating-control@d0018220 {
 			compatible = "marvell,armada-370-gating-clock";
-			reg = <0xd0018220 0x4>;
+			reg = <0 0xd0018220 0 0x4>;
 			clocks = <&coreclk 0>;
 			#clock-cells = <1>;
 		};
 
 		xor@d0060800 {
 			compatible = "marvell,orion-xor";
-			reg = <0xd0060800 0x100
-			       0xd0060A00 0x100>;
+			reg = <0 0xd0060800 0 0x100
+			       0 0xd0060A00 0 0x100>;
 			status = "okay";
 
 			xor00 {
@@ -128,8 +128,8 @@
 
 		xor@d0060900 {
 			compatible = "marvell,orion-xor";
-			reg = <0xd0060900 0x100
-			       0xd0060b00 0x100>;
+			reg = <0 0xd0060900 0 0x100
+			       0 0xd0060b00 0 0x100>;
 			status = "okay";
 
 			xor10 {
diff --git a/arch/arm/boot/dts/armada-xp-db.dts b/arch/arm/boot/dts/armada-xp-db.dts
index e83505e..4922f77 100644
--- a/arch/arm/boot/dts/armada-xp-db.dts
+++ b/arch/arm/boot/dts/armada-xp-db.dts
@@ -26,7 +26,7 @@
 
 	memory {
 		device_type = "memory";
-		reg = <0x00000000 0x80000000>; /* 2 GB */
+		reg = <0x00000000 0x00000000 0x00000000 0x80000000>; /* 2 GB */
 	};
 
 	soc {
diff --git a/arch/arm/boot/dts/armada-xp-gp.dts b/arch/arm/boot/dts/armada-xp-gp.dts
index 1c8afe2..78d9c2a 100644
--- a/arch/arm/boot/dts/armada-xp-gp.dts
+++ b/arch/arm/boot/dts/armada-xp-gp.dts
@@ -28,12 +28,16 @@
 		device_type = "memory";
 
 		/*
-		 * 4 GB of plug-in RAM modules by default but only 3GB
-		 * are visible, the amount of memory available can be
-		 * changed by the bootloader according the size of the
-		 * module actually plugged
+		 * 8 GB of plug-in RAM modules by default.The amount
+		 * of memory available can be changed by the
+		 * bootloader according the size of the module
+		 * actually plugged. Only 7GB are usable because
+		 * addresses from 0xC0000000 to 0xffffffff are used by
+		 * the internal registers of the SoC.
 		 */
-		reg = <0x00000000 0xC0000000>;
+		reg = <0x00000000 0x00000000 0x00000000 0xC0000000>,
+		      <0x00000001 0x00000000 0x00000001 0x00000000>;
+
 	};
 
 	soc {
diff --git a/arch/arm/boot/dts/armada-xp-mv78230.dtsi b/arch/arm/boot/dts/armada-xp-mv78230.dtsi
index f56c405..23eb8d61 100644
--- a/arch/arm/boot/dts/armada-xp-mv78230.dtsi
+++ b/arch/arm/boot/dts/armada-xp-mv78230.dtsi
@@ -46,7 +46,7 @@
 	soc {
 		pinctrl {
 			compatible = "marvell,mv78230-pinctrl";
-			reg = <0xd0018000 0x38>;
+			reg = <0 0xd0018000 0 0x38>;
 
 			sdio_pins: sdio-pins {
 				marvell,pins = "mpp30", "mpp31", "mpp32",
@@ -57,7 +57,7 @@
 
 		gpio0: gpio@d0018100 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018100 0x40>;
+			reg = <0 0xd0018100 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -68,7 +68,7 @@
 
 		gpio1: gpio@d0018140 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018140 0x40>;
+			reg = <0 0xd0018140 0 0x40>;
 			ngpios = <17>;
 			gpio-controller;
 			#gpio-cells = <2>;
diff --git a/arch/arm/boot/dts/armada-xp-mv78260.dtsi b/arch/arm/boot/dts/armada-xp-mv78260.dtsi
index f8f2b78..b2b0f45 100644
--- a/arch/arm/boot/dts/armada-xp-mv78260.dtsi
+++ b/arch/arm/boot/dts/armada-xp-mv78260.dtsi
@@ -47,7 +47,7 @@
 	soc {
 		pinctrl {
 			compatible = "marvell,mv78260-pinctrl";
-			reg = <0xd0018000 0x38>;
+			reg = <0 0xd0018000 0 0x38>;
 
 			sdio_pins: sdio-pins {
 				marvell,pins = "mpp30", "mpp31", "mpp32",
@@ -58,7 +58,7 @@
 
 		gpio0: gpio@d0018100 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018100 0x40>;
+			reg = <0 0xd0018100 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -69,7 +69,7 @@
 
 		gpio1: gpio@d0018140 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018140 0x40>;
+			reg = <0 0xd0018140 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -80,7 +80,7 @@
 
 		gpio2: gpio@d0018180 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018180 0x40>;
+			reg = <0 0xd0018180 0 0x40>;
 			ngpios = <3>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -91,7 +91,7 @@
 
 		ethernet@d0034000 {
 				compatible = "marvell,armada-370-neta";
-				reg = <0xd0034000 0x2500>;
+				reg = <0 0xd0034000 0 0x2500>;
 				interrupts = <14>;
 				clocks = <&gateclk 1>;
 				status = "disabled";
diff --git a/arch/arm/boot/dts/armada-xp-mv78460.dtsi b/arch/arm/boot/dts/armada-xp-mv78460.dtsi
index 936c25d..4de33c2 100644
--- a/arch/arm/boot/dts/armada-xp-mv78460.dtsi
+++ b/arch/arm/boot/dts/armada-xp-mv78460.dtsi
@@ -62,7 +62,7 @@
 	soc {
 		pinctrl {
 			compatible = "marvell,mv78460-pinctrl";
-			reg = <0xd0018000 0x38>;
+			reg = <0 0xd0018000 0 0x38>;
 
 			sdio_pins: sdio-pins {
 				marvell,pins = "mpp30", "mpp31", "mpp32",
@@ -73,7 +73,7 @@
 
 		gpio0: gpio@d0018100 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018100 0x40>;
+			reg = <0 0xd0018100 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -84,7 +84,7 @@
 
 		gpio1: gpio@d0018140 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018140 0x40>;
+			reg = <0 0xd0018140 0 0x40>;
 			ngpios = <32>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -95,7 +95,7 @@
 
 		gpio2: gpio@d0018180 {
 			compatible = "marvell,orion-gpio";
-			reg = <0xd0018180 0x40>;
+			reg = <0 0xd0018180 0 0x40>;
 			ngpios = <3>;
 			gpio-controller;
 			#gpio-cells = <2>;
@@ -106,7 +106,7 @@
 
 		ethernet@d0034000 {
 				compatible = "marvell,armada-370-neta";
-				reg = <0xd0034000 0x2500>;
+				reg = <0 0xd0034000 0 0x2500>;
 				interrupts = <14>;
 				clocks = <&gateclk 1>;
 				status = "disabled";
diff --git a/arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts b/arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts
index 3818a82..4b30f26 100644
--- a/arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts
+++ b/arch/arm/boot/dts/armada-xp-openblocks-ax3-4.dts
@@ -23,7 +23,7 @@
 
 	memory {
 		device_type = "memory";
-		reg = <0x00000000 0xC0000000>; /* 3 GB */
+		reg = <0x00000000 0x00000000 0x00000000 0xC0000000>; /* 3 GB */
 	};
 
 	soc {
diff --git a/arch/arm/boot/dts/armada-xp.dtsi b/arch/arm/boot/dts/armada-xp.dtsi
index ca00d83..5f8d6fc 100644
--- a/arch/arm/boot/dts/armada-xp.dtsi
+++ b/arch/arm/boot/dts/armada-xp.dtsi
@@ -24,26 +24,26 @@
 
 	L2: l2-cache {
 		compatible = "marvell,aurora-system-cache";
-		reg = <0xd0008000 0x1000>;
+		reg = <0 0xd0008000 0 0x1000>;
 		cache-id-part = <0x100>;
 		wt-override;
 	};
 
 	mpic: interrupt-controller@d0020000 {
-	      reg = <0xd0020a00 0x2d0>,
-		    <0xd0021070 0x58>;
+	      reg = <0 0xd0020a00 0 0x2d0>,
+		    <0 0xd0021070 0 0x58>;
 	};
 
 	armada-370-xp-pmsu@d0022000 {
 		compatible = "marvell,armada-370-xp-pmsu";
-		reg = <0xd0022100 0x430>,
-		      <0xd0020800 0x20>;
+		reg = <0 0xd0022100 0 0x430>,
+		      <0 0xd0020800 0 0x20>;
 	};
 
 	soc {
 		serial@d0012200 {
 				compatible = "snps,dw-apb-uart";
-				reg = <0xd0012200 0x100>;
+				reg = <0 0xd0012200 0 0x100>;
 				reg-shift = <2>;
 				interrupts = <43>;
 				reg-io-width = <1>;
@@ -51,7 +51,7 @@
 		};
 		serial@d0012300 {
 				compatible = "snps,dw-apb-uart";
-				reg = <0xd0012300 0x100>;
+				reg = <0 0xd0012300 0 0x100>;
 				reg-shift = <2>;
 				interrupts = <44>;
 				reg-io-width = <1>;
@@ -64,32 +64,32 @@
 
 		coreclk: mvebu-sar@d0018230 {
 			compatible = "marvell,armada-xp-core-clock";
-			reg = <0xd0018230 0x08>;
+			reg = <0 0xd0018230 0 0x08>;
 			#clock-cells = <1>;
 		};
 
 		cpuclk: clock-complex@d0018700 {
 			#clock-cells = <1>;
 			compatible = "marvell,armada-xp-cpu-clock";
-			reg = <0xd0018700 0xA0>;
+			reg = <0 0xd0018700 0 0xA0>;
 			clocks = <&coreclk 1>;
 		};
 
 		gateclk: clock-gating-control@d0018220 {
 			compatible = "marvell,armada-xp-gating-clock";
-			reg = <0xd0018220 0x4>;
+			reg = <0 0xd0018220 0 0x4>;
 			clocks = <&coreclk 0>;
 			#clock-cells = <1>;
 		};
 
 		system-controller@d0018200 {
 				compatible = "marvell,armada-370-xp-system-controller";
-				reg = <0xd0018200 0x500>;
+				reg = <0 0xd0018200 0 0x500>;
 		};
 
 		ethernet@d0030000 {
 				compatible = "marvell,armada-370-neta";
-				reg = <0xd0030000 0x2500>;
+				reg = <0 0xd0030000 0 0x2500>;
 				interrupts = <12>;
 				clocks = <&gateclk 2>;
 				status = "disabled";
@@ -97,8 +97,8 @@
 
 		xor@d0060900 {
 			compatible = "marvell,orion-xor";
-			reg = <0xd0060900 0x100
-			       0xd0060b00 0x100>;
+			reg = <0 0xd0060900 0 0x100
+			       0 0xd0060b00 0 0x100>;
 			clocks = <&gateclk 22>;
 			status = "okay";
 
@@ -117,8 +117,8 @@
 
 		xor@d00f0900 {
 			compatible = "marvell,orion-xor";
-			reg = <0xd00F0900 0x100
-			       0xd00F0B00 0x100>;
+			reg = <0 0xd00F0900 0 0x100
+			       0 0xd00F0B00 0 0x100>;
 			clocks = <&gateclk 28>;
 			status = "okay";
 
@@ -145,7 +145,7 @@
 
 		usb@d0052000 {
 			compatible = "marvell,orion-ehci";
-			reg = <0xd0052000 0x500>;
+			reg = <0 0xd0052000 0 0x500>;
 			interrupts = <47>;
 			clocks = <&gateclk 20>;
 			status = "disabled";
-- 
1.7.9.5

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Rob Herring]

On 03/21/2013 11:26 AM, Gregory CLEMENT wrote:
> In order to be able to use more than 4GB of RAM when the LPAE is
> activated, the dts must be converted in 64 bits.
> 
> Armada XP and Armada 370 share a dtsi file which have also be
> converted to 64 bits. This lead to convert all the device tree files
> to 64 bits even the one used for Armada 370 (which don't support
> LPAE)
> 
> This was heavily based on the work of Lior Amsalem.
> 
> Signed-off-by: Lior Amsalem <alior@marvell.com>
> Signed-off-by: Gregory CLEMENT <gregory.clement@free-electrons.com>

[snip]

> diff --git a/arch/arm/boot/dts/armada-370-xp.dtsi b/arch/arm/boot/dts/armada-370-xp.dtsi
> index 5b70820..562f24c 100644
> --- a/arch/arm/boot/dts/armada-370-xp.dtsi
> +++ b/arch/arm/boot/dts/armada-370-xp.dtsi
> @@ -15,8 +15,7 @@
>   * This file contains the definitions that are common to the Armada
>   * 370 and Armada XP SoC.
>   */
> -
> -/include/ "skeleton.dtsi"
> +/include/ "skeleton64.dtsi"
>  
>  / {
>  	model = "Marvell Armada 370 and XP SoC";
> @@ -37,20 +36,20 @@
>  
>  	coherency-fabric@d0020200 {
>  		compatible = "marvell,coherency-fabric";
> -		reg = <0xd0020200 0xb0>,
> -		      <0xd0021810 0x1c>;
> +		reg = <0 0xd0020200 0 0xb0>,
> +		      <0 0xd0021810 0 0x1c>;
>  	};
>  
>  	soc {
> -		#address-cells = <1>;
> -		#size-cells = <1>;
> +		#address-cells = <2>;
> +		#size-cells = <2>;

If all the addresses for the soc bus are below 4GB or even within a 4GB
range if using the ranges property, then changing all this and
everything below it is kind of pointless.

Rob

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Arnd Bergmann]

On Thursday 21 March 2013, Rob Herring wrote:
> >       soc {
> > -             #address-cells = <1>;
> > -             #size-cells = <1>;
> > +             #address-cells = <2>;
> > +             #size-cells = <2>;
> 
> If all the addresses for the soc bus are below 4GB or even within a 4GB
> range if using the ranges property, then changing all this and
> everything below it is kind of pointless.

Good point. We'll probably also have to change it all again when we add a new
binding for that bus in 3.10, so it makes sense to change it only once.

	Arnd

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Thomas Petazzoni]

Dear Arnd Bergmann,

On Thu, 21 Mar 2013 19:03:52 +0000, Arnd Bergmann wrote:
> On Thursday 21 March 2013, Rob Herring wrote:
> > >       soc {
> > > -             #address-cells = <1>;
> > > -             #size-cells = <1>;
> > > +             #address-cells = <2>;
> > > +             #size-cells = <2>;
> > 
> > If all the addresses for the soc bus are below 4GB or even within a 4GB
> > range if using the ranges property, then changing all this and
> > everything below it is kind of pointless.
> 
> Good point. We'll probably also have to change it all again when we add a new
> binding for that bus in 3.10, so it makes sense to change it only once.

In the mean time can we do something like:

	soc {
		compatible = "simple-bus";
		range = <...>;

		[... all the peripherals ...]
	};

with the range = <...> property converting the peripheral registers
base address (expressed as offsets in the reg = <...> properties of the
subnodes) into the absolute physical address?

I'm planning to work on the DT binding for the mvebu-mbus driver as
soon as the PCIe driver gets accepted, but it would be good to have an
intermediate solution to get the LPAE support in.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Arnd Bergmann]

On Thursday 21 March 2013, Thomas Petazzoni wrote:
> In the mean time can we do something like:
> 
>         soc {
>                 compatible = "simple-bus";
>                 range = <...>;
> 
>                 [... all the peripherals ...]
>         };
> 
> with the range = <...> property converting the peripheral registers
> base address (expressed as offsets in the reg = <...> properties of the
> subnodes) into the absolute physical address?

Yes, that is what Rob suggested you do.
 
> I'm planning to work on the DT binding for the mvebu-mbus driver as
> soon as the PCIe driver gets accepted, but it would be good to have an
> intermediate solution to get the LPAE support in.

Sounds good.

	Arnd

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Sebastian Hesselbarth]

On 03/21/2013 10:31 PM, Arnd Bergmann wrote:
> On Thursday 21 March 2013, Thomas Petazzoni wrote:
>> In the mean time can we do something like:
>>
>>          soc {
>>                  compatible = "simple-bus";
>>                  range =<...>;
>>
>>                  [... all the peripherals ...]
>>          };
>>
>> with the range =<...>  property converting the peripheral registers
>> base address (expressed as offsets in the reg =<...>  properties of the
>> subnodes) into the absolute physical address?
>
> Yes, that is what Rob suggested you do.

Thomas,

have a look at arch/arm/boot/dts/dove.dtsi, it uses ranges-property
and peripherals encoded as offsets.

Sebastian

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Andrew Lunn]

>  		/*
> -		 * 4 GB of plug-in RAM modules by default but only 3GB
> -		 * are visible, the amount of memory available can be
> -		 * changed by the bootloader according the size of the
> -		 * module actually plugged
> +		 * 8 GB of plug-in RAM modules by default.The amount
> +		 * of memory available can be changed by the
> +		 * bootloader according the size of the module
> +		 * actually plugged. Only 7GB are usable because
> +		 * addresses from 0xC0000000 to 0xffffffff are used by
> +		 * the internal registers of the SoC.
>  		 */
> -		reg = <0x00000000 0xC0000000>;
> +		reg = <0x00000000 0x00000000 0x00000000 0xC0000000>,
> +		      <0x00000001 0x00000000 0x00000001 0x00000000>;
> +

Hi Gregory

Could you recommend a document which introduces LPAE.

Only being able to address 7GB seems a bit odd to me. I kind of
expected you set up the translation tables to map a page in the 32 bit
address range to any arbitrary page in the 40 bit address range. So
leaving 0xC0000000 to 0xffffffff in the 32bit address range clear is
easy. But why do you loose space in the 40bit address range?

      Thanks
	Andrew

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Thomas Petazzoni]

Dear Andrew Lunn,

On Thu, 21 Mar 2013 21:15:33 +0100, Andrew Lunn wrote:

> Could you recommend a document which introduces LPAE.
> 
> Only being able to address 7GB seems a bit odd to me. I kind of
> expected you set up the translation tables to map a page in the 32 bit
> address range to any arbitrary page in the 40 bit address range. So
> leaving 0xC0000000 to 0xffffffff in the 32bit address range clear is
> easy. But why do you loose space in the 40bit address range?

translation tables convert virtual addresses to physical addresses.
Here, we are only talking about physical addresses. There is an overlap
between the physical addresses used by the RAM, and the physical
addresses at which I/O devices are visible.

And I'm not sure the SDRAM address decoding windows allows to split the
first 4 GB of RAM into two areas, one that would be mapped starting at
physical address 0x0, and another area that would be mapped at a
different address (above 4 GB).

However, I'm unsure why 0xC0000000 was chosen. Why not 0xD0000000,
where the internal registers currently start?

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Jason Cooper]

On Thu, Mar 21, 2013 at 09:22:36PM +0100, Thomas Petazzoni wrote:
> Dear Andrew Lunn,
> 
> On Thu, 21 Mar 2013 21:15:33 +0100, Andrew Lunn wrote:
> 
> > Could you recommend a document which introduces LPAE.
> > 
> > Only being able to address 7GB seems a bit odd to me. I kind of
> > expected you set up the translation tables to map a page in the 32 bit
> > address range to any arbitrary page in the 40 bit address range. So
> > leaving 0xC0000000 to 0xffffffff in the 32bit address range clear is
> > easy. But why do you loose space in the 40bit address range?
> 
> translation tables convert virtual addresses to physical addresses.
> Here, we are only talking about physical addresses. There is an overlap
> between the physical addresses used by the RAM, and the physical
> addresses at which I/O devices are visible.
> 
> And I'm not sure the SDRAM address decoding windows allows to split the
> first 4 GB of RAM into two areas, one that would be mapped starting at
> physical address 0x0, and another area that would be mapped at a
> different address (above 4 GB).
> 
> However, I'm unsure why 0xC0000000 was chosen. Why not 0xD0000000,
> where the internal registers currently start?

I had the same question earlier but got distracted by other things.
Thanks for bringing it up.  Gregory, shouldn't this be 0xD0000000?

thx,

Jason.

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Andrew Lunn]

On Thu, Mar 21, 2013 at 09:22:36PM +0100, Thomas Petazzoni wrote:
> Dear Andrew Lunn,
> 
> On Thu, 21 Mar 2013 21:15:33 +0100, Andrew Lunn wrote:
> 
> > Could you recommend a document which introduces LPAE.
> > 
> > Only being able to address 7GB seems a bit odd to me. I kind of
> > expected you set up the translation tables to map a page in the 32 bit
> > address range to any arbitrary page in the 40 bit address range. So
> > leaving 0xC0000000 to 0xffffffff in the 32bit address range clear is
> > easy. But why do you loose space in the 40bit address range?
> 
> translation tables convert virtual addresses to physical addresses.
> Here, we are only talking about physical addresses. There is an overlap
> between the physical addresses used by the RAM, and the physical
> addresses at which I/O devices are visible.
> 
> And I'm not sure the SDRAM address decoding windows allows to split the
> first 4 GB of RAM into two areas, one that would be mapped starting at
> physical address 0x0, and another area that would be mapped at a
> different address (above 4 GB).

So why not map the whole SDRAM above 4GB physical address?

   Thanks
	Andrew

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Thomas Petazzoni]

Dear Andrew Lunn,

On Thu, 21 Mar 2013 21:37:51 +0100, Andrew Lunn wrote:

> > And I'm not sure the SDRAM address decoding windows allows to split the
> > first 4 GB of RAM into two areas, one that would be mapped starting at
> > physical address 0x0, and another area that would be mapped at a
> > different address (above 4 GB).
> 
> So why not map the whole SDRAM above 4GB physical address?

That's a good question. The problem is most likely that this would
require to synchronize with U-Boot modifications, which is not easy to
achieve.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Thomas Petazzoni]

Dear Andrew Lunn,

On Thu, 21 Mar 2013 21:37:51 +0100, Andrew Lunn wrote:

> > And I'm not sure the SDRAM address decoding windows allows to split the
> > first 4 GB of RAM into two areas, one that would be mapped starting at
> > physical address 0x0, and another area that would be mapped at a
> > different address (above 4 GB).
> 
> So why not map the whole SDRAM above 4GB physical address?

As Lior rightly pointed out to me, this would prevent any device from
DMA-ing to or from the RAM. Devices can only access the first 32 bits
of the physical address space. So there must be some RAM below 4 GB.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Jason Gunthorpe]

On Thu, Mar 21, 2013 at 09:22:36PM +0100, Thomas Petazzoni wrote:

> And I'm not sure the SDRAM address decoding windows allows to split the
> first 4 GB of RAM into two areas, one that would be mapped starting at
> physical address 0x0, and another area that would be mapped at a
> different address (above 4 GB).

On prior Marvell chips the SDRAM is mapped on a rank by rank basis, so
each rank can be placed in DRAM properly.
 
> However, I'm unsure why 0xC0000000 was chosen. Why not 0xD0000000,
> where the internal registers currently start?

Probably because the 8GB is composed of 4 2GB ranks and what was done
is to map like this:

  Rank 0 ->           0 ->  0x80000000 (2G)
  Rank 1 -> 0x080000000 -> 0x0C0000000 (2G rank, 1G mapped)
  Rank 2 -> 0x100000000 -> 0x180000000
  Rank 3 -> 0x180000000 -> 0x300000000

The ranks must be power of two sizes, and aligned to their size,
so it isn't possible to get closer to 0xD0000000.

To recover the lost RAM the entire rank would need to be located
above 4G.

Or, better, locate all the internal registers above 8G and use
contiguous DRAM mapping from 0 -> 8GB

Linux is sensitive to the ratio of high/low memory, if that gets too
big it struggles, maximizing low memory is desirable - I assume ARM is
basically the same as x86 in this regard?

> So why not map the whole SDRAM above 4GB physical address?

I think the no-MMU boot mode would break if this was done?

Jason

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Thomas Petazzoni]

Dear Jason Gunthorpe,

On Thu, 21 Mar 2013 14:55:45 -0600, Jason Gunthorpe wrote:

> Or, better, locate all the internal registers above 8G and use
> contiguous DRAM mapping from 0 -> 8GB

I see two potential issues with this idea:

 *) It only works when LPAE is enabled, so we would have to have
    different internal register addresses depending on whether LPAE is
    enabled or not. Probably not impossible, but not very
    straightforward either.

 *) It would require Linux to change the internal registers address
    (for now the kernel relies on the bootloader). The problem is that
    we can't do it early enough to preserve the earlyprintk
    functionality. Maybe you have suggestions on how to achieve that?

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

RE: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Lior Amsalem]

Hi Thomas, Jason,

> From: Thomas Petazzoni [mailto:thomas.petazzoni@free-electrons.com]
> Sent: Thursday, March 21, 2013 11:15 PM
> 
> Dear Jason Gunthorpe,
> 
> On Thu, 21 Mar 2013 14:55:45 -0600, Jason Gunthorpe wrote:
> 
> > Or, better, locate all the internal registers above 8G and use
> > contiguous DRAM mapping from 0 -> 8GB
> 
> I see two potential issues with this idea:
> 
>  *) It only works when LPAE is enabled, so we would have to have
>     different internal register addresses depending on whether LPAE is
>     enabled or not. Probably not impossible, but not very
>     straightforward either.
> 
>  *) It would require Linux to change the internal registers address
>     (for now the kernel relies on the bootloader). The problem is that
>     we can't do it early enough to preserve the earlyprintk
>     functionality. Maybe you have suggestions on how to achieve that?
> 

Please note that all registers and IOs are still 32bit, the 40bit extension is on 
the CPU to DRAM path (windows) only.

Therefore, the register that sets the internal register base is 32bit :)

> Best regards,
> 
> Thomas
> --
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux development, consulting,
> training and support.
> http://free-electrons.com

Regards,
Lior Amsalem

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Jason Gunthorpe]

On Thu, Mar 21, 2013 at 11:35:21PM +0200, Lior Amsalem wrote:

> >  *) It would require Linux to change the internal registers address
> >     (for now the kernel relies on the bootloader). The problem is that
> >     we can't do it early enough to preserve the earlyprintk
> >     functionality. Maybe you have suggestions on how to achieve that?
> 
> Please note that all registers and IOs are still 32bit, the 40bit
> extension is on the CPU to DRAM path (windows) only.

Ah, I did wonder, since Ezequiel's MBUS driver was the same as on
prior families. I guess it is also the case that all on board DMA and
DMA from PCI-E devices are restricted to the low 32 bits as well?

I think lots of Linux workloads will really struggle under those
conditions..

Jason

RE: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Lior Amsalem]

> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> Sent: Thursday, March 21, 2013 11:46 PM
>
> On Thu, Mar 21, 2013 at 11:35:21PM +0200, Lior Amsalem wrote:
> 
> > >  *) It would require Linux to change the internal registers address
> > >     (for now the kernel relies on the bootloader). The problem is that
> > >     we can't do it early enough to preserve the earlyprintk
> > >     functionality. Maybe you have suggestions on how to achieve that?
> >
> > Please note that all registers and IOs are still 32bit, the 40bit
> > extension is on the CPU to DRAM path (windows) only.
> 
> Ah, I did wonder, since Ezequiel's MBUS driver was the same as on prior
> families. I guess it is also the case that all on board DMA and DMA from PCI-E
> devices are restricted to the low 32 bits as well?

Correct, that's why we sets DMABOUNCE config.

> 
> I think lots of Linux workloads will really struggle under those conditions..
> 
> Jason

Regards,
Lior Amsalem

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Arnd Bergmann]

On Thursday 21 March 2013, Lior Amsalem wrote:
> > From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> > Sent: Thursday, March 21, 2013 11:46 PM
> >
> > On Thu, Mar 21, 2013 at 11:35:21PM +0200, Lior Amsalem wrote:
> > 
> > > >  *) It would require Linux to change the internal registers address
> > > >     (for now the kernel relies on the bootloader). The problem is that
> > > >     we can't do it early enough to preserve the earlyprintk
> > > >     functionality. Maybe you have suggestions on how to achieve that?
> > >
> > > Please note that all registers and IOs are still 32bit, the 40bit
> > > extension is on the CPU to DRAM path (windows) only.
> > 
> > Ah, I did wonder, since Ezequiel's MBUS driver was the same as on prior
> > families. I guess it is also the case that all on board DMA and DMA from PCI-E
> > devices are restricted to the low 32 bits as well?
> 
> Correct, that's why we sets DMABOUNCE config.
> 

IIRC, DMABOUNCE does not work well in a multiplatform configuration. Have you tried
using swiotlb instead?

	Arnd

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Jason Gunthorpe]

On Thu, Mar 21, 2013 at 10:15:23PM +0100, Thomas Petazzoni wrote:
> Dear Jason Gunthorpe,
> 
> On Thu, 21 Mar 2013 14:55:45 -0600, Jason Gunthorpe wrote:
> 
> > Or, better, locate all the internal registers above 8G and use
> > contiguous DRAM mapping from 0 -> 8GB
> 
> I see two potential issues with this idea:
> 
>  *) It only works when LPAE is enabled, so we would have to have
>     different internal register addresses depending on whether LPAE is
>     enabled or not. Probably not impossible, but not very
>     straightforward either.

Ideally the internal register space address would come from the DT -
we are getting very close to that on Marvell, I think.. Things like
earlyprintk should ideally early parse the DT, harder I know :(

>  *) It would require Linux to change the internal registers address
>     (for now the kernel relies on the bootloader). The problem is that
>     we can't do it early enough to preserve the earlyprintk
>     functionality. Maybe you have suggestions on how to achieve that?

I can't forsee how Linux could do this reprogramming - not only do you
have to move the registers, you'd also have to reprogram the DRAM
bases, while running from the DRAM. Not only does that have to be done
early, but the DT would need to describe the DRAM ranks, and the code
would have to parse it.. On top of that it would have to be very
careful not to wack any DRAM that has already been touched by the
kernel.. Tricky stuff :)

Jason

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Sebastian Hesselbarth]

On 03/21/2013 10:41 PM, Jason Gunthorpe wrote:
> On Thu, Mar 21, 2013 at 10:15:23PM +0100, Thomas Petazzoni wrote:
>> Dear Jason Gunthorpe,
>>
>> On Thu, 21 Mar 2013 14:55:45 -0600, Jason Gunthorpe wrote:
>>
>>> Or, better, locate all the internal registers above 8G and use
>>> contiguous DRAM mapping from 0 ->  8GB
>>
>> I see two potential issues with this idea:
>>
>>   *) It only works when LPAE is enabled, so we would have to have
>>      different internal register addresses depending on whether LPAE is
>>      enabled or not. Probably not impossible, but not very
>>      straightforward either.
>
> Ideally the internal register space address would come from the DT -
> we are getting very close to that on Marvell, I think.. Things like
> earlyprintk should ideally early parse the DT, harder I know :(
>
>>   *) It would require Linux to change the internal registers address
>>      (for now the kernel relies on the bootloader). The problem is that
>>      we can't do it early enough to preserve the earlyprintk
>>      functionality. Maybe you have suggestions on how to achieve that?
>
> I can't forsee how Linux could do this reprogramming - not only do you
> have to move the registers, you'd also have to reprogram the DRAM
> bases, while running from the DRAM. Not only does that have to be done
> early, but the DT would need to describe the DRAM ranks, and the code
> would have to parse it.. On top of that it would have to be very
> careful not to wack any DRAM that has already been touched by the
> kernel.. Tricky stuff :)

Jason, Lior,

at least for Dove you are missing one important fact:

The Internal Registers Base Address register is within ... the internal
registers. You can't even look it up without knowing where it is.

That doesn't mean we can parse the DT for the current register base
address and move it to where we expect it. But for DRAM, I suggest not
to mess with it. The boot loader is there for a reason.

Sebastian

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Arnd Bergmann]

On Thursday 21 March 2013, Jason Gunthorpe wrote:
> On Thu, Mar 21, 2013 at 10:15:23PM +0100, Thomas Petazzoni wrote:
> > Dear Jason Gunthorpe,

> > 
> >  *) It only works when LPAE is enabled, so we would have to have
> >     different internal register addresses depending on whether LPAE is
> >     enabled or not. Probably not impossible, but not very
> >     straightforward either.
> 
> Ideally the internal register space address would come from the DT -
> we are getting very close to that on Marvell, I think.. Things like
> earlyprintk should ideally early parse the DT, harder I know :(

Let's not worry about earlyprintk for now. earlyprintk is supposed to be
really really simple, and the solution that we are using now works
well enough: You pick a hardcoded implementation at build time when
you are debugging, but disable debug_ll for a multiplatform release
kernel.

	Arnd

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Andrew Lunn]

> Or, better, locate all the internal registers above 8G and use
> contiguous DRAM mapping from 0 -> 8GB

Wouldn't that get us into a 640KBs is enough for everyone situation?
Why not put the IO at the very top of the 1TB address range?

    Andrew

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Andrew Lunn]

Hi all,

So, lets see if i have all this right.

IO space needs to stay where it is, somewhere in the top 1GB, because
it is limited to the 32bit address space.

We must have some SDRAM in the bottom of the 40bit address range in
order that DMA works. Bounce buffers are used for anything which is
outside of the bottom 32bits.

SDRAM can only be split on ranks. I assume this is also a synonym for
chip select? 

Ideally we want the boot loader to setup the split, since it is not
very easy to move stuff around in a running system. Alternatively, the
boot loader could setup the lowest rank/chip select and leave the
others disabled. That gives enough that Linux can boot and decide
where it wants to put the rest, without the problem of having to remap
the SDRAM its currently running from.

It might be possible, in theory, to copy code into the SRAM and run
from SRAM while moving the SDRAM around. But i get the feeling its not
very easy.

If there is 4GB in the system, it is probably going to be split with
2GB low and 2GB high. This is probably the most important use case,
since throwing away 1/4 of your SDRAM is much worse than 1/8, or 1/16.

If there is 8GB in the system, it is probably going to be split with
2GB low and 6GB high, assuming it has 4 ranks of 2GB. What might be
considered is 4/4 split, with 1GB discarded, if that gives better
overall performance. If it happens to be 2x 4GB you have no choice but
to discard 1GB.

With 16GB, there is no choice other than 4GB/12GB with 1GB discarded.

Is that all correct?

Thanks
	Andrew

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Jason Gunthorpe]

On Fri, Mar 22, 2013 at 07:28:54AM +0100, Andrew Lunn wrote:

> IO space needs to stay where it is, somewhere in the top 1GB, because
> it is limited to the 32bit address space.

Yes
 
> We must have some SDRAM in the bottom of the 40bit address range in
> order that DMA works. Bounce buffers are used for anything which is
> outside of the bottom 32bits.

Yes :(
 
> SDRAM can only be split on ranks. I assume this is also a synonym for
> chip select? 

Yes, rank is the JEDEC term. All CS inputs on every DRAM die in a rank
should be asserted at once. Note that splitting the SDRAM address map
by rank is only one choice, other chipsets can do bank interleaving
across ranks for greater performance..

> Ideally we want the boot loader to setup the split, since it is not
> very easy to move stuff around in a running system. Alternatively,
> the

Yes, there are several possibilities, but they will all require the DT
to describe the ranks on the DRAM bus directly.

> It might be possible, in theory, to copy code into the SRAM and run
> from SRAM while moving the SDRAM around. But i get the feeling its not
> very easy.

This doesn't help, the issue is the kernel has already started and is
already placed in SDRAM, so you can't really touch the mapping that
contains the kernel. The work is more of a complexity to determine
what the bootloader did and re-do it in a way that dosen't move any
memory the kernel has touched up to that point.

> If there is 4GB in the system, it is probably going to be split with
> 2GB low and 2GB high. This is probably the most important use case,
> since throwing away 1/4 of your SDRAM is much worse than 1/8, or 1/16.

Think about things in terms of rank size. A 4GB system could have 1, 2
or 4 GB ranks, and each is a different case. 1GB ranks should have a
3/1 split, 2GB ranks a 2/2 or 3/0 split, and 4GB ranks are forced into
a 2/0 split (due to alignment needs).

> With 16GB, there is no choice other than 4GB/12GB with 1GB discarded.

4x4GB ranks would be a 2/12GB split with 2GB of discard, a rank size
must be a power of two.

At a certain point low mem exhaustion becomes a serious issue for
Linux, a system that can't DMA to 85% of its memory is incredibly
broken, IMHO.

Jason

Re: [PATCH 5/5] arm: dts: Convert mvebu device tree files to 64 bits

[Arnd Bergmann]

On Friday 22 March 2013, Jason Gunthorpe wrote:
> At a certain point low mem exhaustion becomes a serious issue for
> Linux, a system that can't DMA to 85% of its memory is incredibly
> broken, IMHO.

A lot of workloads will also suffer from lowmem exhaustion even without
the DMA zone problem. If course the combination is particularly nasty
because now the kernel has a to kmap() a user page in order to copy the
dma data in and out of lowmem.

I'm sure there are even workloads that perform well with more than
16GB, but I can't think of any right now. I would expect that 4 or 8 GB
is where it gets quite silly for most workloads, but troubles start as
soon as you go beyond 1GB.

	Arnd