[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Christopher Covington]

On an LPAE system, the physical addresses used by VirtIO-MMIO may
be larger than 32 bits, even if the header and configuration space
addresses fit into 32 bits. For example with the Versatile Express
memory map using 4G memory, the following error occured when trying
to use a VirtIO-MMIO block device.

EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: : unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223

To fix this, select ARCH_DMA_ADDR_T_64BIT when both LPAE and
VIRTIO_MMIO are selected.

Signed-off-by: Christopher Covington <cov@codeaurora.org>
---
 arch/arm/mm/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index 1f8fed9..a62bcc9 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -617,6 +617,7 @@ config ARM_LPAE
 	bool "Support for the Large Physical Address Extension"
 	depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
 		!CPU_32v4 && !CPU_32v3
+	select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
 	help
 	  Say Y if you have an ARMv7 processor supporting the LPAE page
 	  table format and you would like to access memory beyond the
-- 
Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Catalin Marinas]

On Wed, Mar 19, 2014 at 05:35:19PM +0000, Christopher Covington wrote:
> On an LPAE system, the physical addresses used by VirtIO-MMIO may
> be larger than 32 bits, even if the header and configuration space
> addresses fit into 32 bits. For example with the Versatile Express
> memory map using 4G memory, the following error occured when trying
> to use a VirtIO-MMIO block device.
> 
> EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: :
> unaligned directory entry - offset=0, inode=3755990991, rec_len=57311,
> name_len=223
> 
> To fix this, select ARCH_DMA_ADDR_T_64BIT when both LPAE and
> VIRTIO_MMIO are selected.
> 
> Signed-off-by: Christopher Covington <cov@codeaurora.org>
> ---
>  arch/arm/mm/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
> index 1f8fed9..a62bcc9 100644
> --- a/arch/arm/mm/Kconfig
> +++ b/arch/arm/mm/Kconfig
> @@ -617,6 +617,7 @@ config ARM_LPAE
>  	bool "Support for the Large Physical Address Extension"
>  	depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
>  		!CPU_32v4 && !CPU_32v3
> +	select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO

That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
platform with >32-bit physical address space? If yes, it should be
selected there.

-- 
Catalin

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Christopher Covington]

Hi Catalin,

On 03/21/2014 12:27 PM, Catalin Marinas wrote:
> On Wed, Mar 19, 2014 at 05:35:19PM +0000, Christopher Covington wrote:
>> On an LPAE system, the physical addresses used by VirtIO-MMIO may
>> be larger than 32 bits, even if the header and configuration space
>> addresses fit into 32 bits. For example with the Versatile Express
>> memory map using 4G memory, the following error occured when trying
>> to use a VirtIO-MMIO block device.
>>
>> EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: :
>> unaligned directory entry - offset=0, inode=3755990991, rec_len=57311,
>> name_len=223
>>
>> To fix this, select ARCH_DMA_ADDR_T_64BIT when both LPAE and
>> VIRTIO_MMIO are selected.
>>
>> Signed-off-by: Christopher Covington <cov@codeaurora.org>
>> ---
>>  arch/arm/mm/Kconfig | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
>> index 1f8fed9..a62bcc9 100644
>> --- a/arch/arm/mm/Kconfig
>> +++ b/arch/arm/mm/Kconfig
>> @@ -617,6 +617,7 @@ config ARM_LPAE
>>  	bool "Support for the Large Physical Address Extension"
>>  	depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
>>  		!CPU_32v4 && !CPU_32v3
>> +	select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
> 
> That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
> platform with >32-bit physical address space? If yes, it should be
> selected there.

The platforms I'm currently using are models like the Versatile Express
RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Catalin Marinas]

On 21 Mar 2014, at 19:44, Christopher Covington <cov@codeaurora.org> wrote:
> On 03/21/2014 12:27 PM, Catalin Marinas wrote:
>> On Wed, Mar 19, 2014 at 05:35:19PM +0000, Christopher Covington wrote:
>>> On an LPAE system, the physical addresses used by VirtIO-MMIO may
>>> be larger than 32 bits, even if the header and configuration space
>>> addresses fit into 32 bits. For example with the Versatile Express
>>> memory map using 4G memory, the following error occured when trying
>>> to use a VirtIO-MMIO block device.
>>> 
>>> EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: :
>>> unaligned directory entry - offset=0, inode=3755990991, rec_len=57311,
>>> name_len=223
>>> 
>>> To fix this, select ARCH_DMA_ADDR_T_64BIT when both LPAE and
>>> VIRTIO_MMIO are selected.
>>> 
>>> Signed-off-by: Christopher Covington <cov@codeaurora.org>
>>> ---
>>> arch/arm/mm/Kconfig | 1 +
>>> 1 file changed, 1 insertion(+)
>>> 
>>> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
>>> index 1f8fed9..a62bcc9 100644
>>> --- a/arch/arm/mm/Kconfig
>>> +++ b/arch/arm/mm/Kconfig
>>> @@ -617,6 +617,7 @@ config ARM_LPAE
>>> 	bool "Support for the Large Physical Address Extension"
>>> 	depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
>>> 		!CPU_32v4 && !CPU_32v3
>>> +	select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
>> 
>> That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
>> platform with >32-bit physical address space? If yes, it should be
>> selected there.
> 
> The platforms I'm currently using are models like the Versatile Express
> RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.

But do you use RAM beyond 32-bit on such models?

Catalin

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Arnd Bergmann]

On Friday 21 March 2014 23:27:24 Catalin Marinas wrote:
> On 21 Mar 2014, at 19:44, Christopher Covington <cov@codeaurora.org> wrote:
> > On 03/21/2014 12:27 PM, Catalin Marinas wrote:
> >>> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
> >>> index 1f8fed9..a62bcc9 100644
> >>> --- a/arch/arm/mm/Kconfig
> >>> +++ b/arch/arm/mm/Kconfig
> >>> @@ -617,6 +617,7 @@ config ARM_LPAE
> >>>     bool "Support for the Large Physical Address Extension"
> >>>     depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
> >>>             !CPU_32v4 && !CPU_32v3
> >>> +   select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
> >> 
> >> That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
> >> platform with >32-bit physical address space? If yes, it should be
> >> selected there.
> > 
> > The platforms I'm currently using are models like the Versatile Express
> > RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.
> 
> But do you use RAM beyond 32-bit on such models?

I think the more important question here is what the normal behavior is
for these platforms. I believe in most cases you don't have RAM above
the boundary, so we should not enable the option by default as it can
have noticeable overhead (we'd turn it on all the time if it didn't).

How about one of these two 

a)

diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index 9ea4b7b..6e3b6db 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -628,7 +628,9 @@ config ARCH_PHYS_ADDR_T_64BIT
 	def_bool ARM_LPAE
 
 config ARCH_DMA_ADDR_T_64BIT
-	bool
+	def_bool "Allow DMA to high (>4GB) addresses"
+	depends on ARCH_PHYS_ADDR_T_64BIT
+	default y
 	help

 config ARM_THUMB
 	bool "Support Thumb user binaries" if !CPU_THUMBONLY


b)

diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index 9ea4b7b..4a21b1e 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -619,16 +619,28 @@ config ARM_LPAE
 	help
 	  Say Y if you have an ARMv7 processor supporting the LPAE page
 	  table format and you would like to access memory beyond the
-	  4GB limit. The resulting kernel image will not run on
-	  processors without the LPA extension.
+	  4GB limit, or run virtual machines usign KVM.
+	  The resulting kernel image will not run on processors without
+	  the LPA extension.
 
 	  If unsure, say N.
 
 config ARCH_PHYS_ADDR_T_64BIT
-	def_bool ARM_LPAE
+	bool "Support more than 4GB of physical address space" if EXPERT
+	depends on ARM_LPAE
+	default y
+	help
+	  Say Y if you use LPAE to access RAM or MMIO registers at
+	  addresses beyond the low 4GB of address space. If this
+	  option is disabled, the kernel will use the LPAE page table
+	  format and it can offer KVM support, but will use a slightly
+	  more efficient representation of physical memory addresses
+	  that restricts access to 32-bit addressable locations.
+
+	  If unsure, say Y.
 
 config ARCH_DMA_ADDR_T_64BIT
-	bool
+	def_bool ARCH_PHYS_ADDR_T_64BIT
 
 config ARM_THUMB
 	bool "Support Thumb user binaries" if !CPU_THUMBONLY



In either case, platforms that need this support can always
'select' it, while a kernel built only for platforms that don't
need it can offer this as a user-selectable option.

	Arnd

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Catalin Marinas]

On Fri, Mar 21, 2014 at 11:52:33PM +0000, Arnd Bergmann wrote:
> On Friday 21 March 2014 23:27:24 Catalin Marinas wrote:
> > On 21 Mar 2014, at 19:44, Christopher Covington <cov@codeaurora.org> wrote:
> > > On 03/21/2014 12:27 PM, Catalin Marinas wrote:
> > >>> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
> > >>> index 1f8fed9..a62bcc9 100644
> > >>> --- a/arch/arm/mm/Kconfig
> > >>> +++ b/arch/arm/mm/Kconfig
> > >>> @@ -617,6 +617,7 @@ config ARM_LPAE
> > >>>     bool "Support for the Large Physical Address Extension"
> > >>>     depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
> > >>>             !CPU_32v4 && !CPU_32v3
> > >>> +   select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
> > >> 
> > >> That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
> > >> platform with >32-bit physical address space? If yes, it should be
> > >> selected there.
> > > 
> > > The platforms I'm currently using are models like the Versatile Express
> > > RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.
> > 
> > But do you use RAM beyond 32-bit on such models?
> 
> I think the more important question here is what the normal behavior is
> for these platforms. I believe in most cases you don't have RAM above
> the boundary, so we should not enable the option by default as it can
> have noticeable overhead (we'd turn it on all the time if it didn't).
> 
> How about one of these two 
> 
> a)
> 
> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
> index 9ea4b7b..6e3b6db 100644
> --- a/arch/arm/mm/Kconfig
> +++ b/arch/arm/mm/Kconfig
> @@ -628,7 +628,9 @@ config ARCH_PHYS_ADDR_T_64BIT
>  	def_bool ARM_LPAE
>  
>  config ARCH_DMA_ADDR_T_64BIT
> -	bool
> +	def_bool "Allow DMA to high (>4GB) addresses"
> +	depends on ARCH_PHYS_ADDR_T_64BIT
> +	default y
>  	help

We should probably call this "Allow DMA to high (>4GB) _bus_ addresses"
to distinguish it from the actual CPU physical address.

> b)
> 
> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
> index 9ea4b7b..4a21b1e 100644
> --- a/arch/arm/mm/Kconfig
> +++ b/arch/arm/mm/Kconfig
[...]
>  config ARCH_PHYS_ADDR_T_64BIT
> -	def_bool ARM_LPAE
> +	bool "Support more than 4GB of physical address space" if EXPERT
> +	depends on ARM_LPAE
> +	default y

I don't think I ever tested LPAE with ARCH_PHYS_ADDR_T_64BIT off. Even
if you enable EXPERT, it's a risk that some bit shifting of phys_addr_t
would break (the pte expects 64-bit entries).

>  config ARCH_DMA_ADDR_T_64BIT
> -	bool
> +	def_bool ARCH_PHYS_ADDR_T_64BIT

I recall Russell objecting to this when we reviewed the LPAE patches
because the CPU address is different from the bus address. I assume
those arguments still stand.

I would go for (a) but with individual SoCs still selecting
ARCH_DMA_ADDR_T_64BIT explicitly if they expect bus addresses over
32-bit.

-- 
Catalin

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Christopher Covington]

On 03/21/2014 07:27 PM, Catalin Marinas wrote:
> On 21 Mar 2014, at 19:44, Christopher Covington <cov@codeaurora.org> wrote:
>> On 03/21/2014 12:27 PM, Catalin Marinas wrote:
>>> On Wed, Mar 19, 2014 at 05:35:19PM +0000, Christopher Covington wrote:
>>>> On an LPAE system, the physical addresses used by VirtIO-MMIO may
>>>> be larger than 32 bits, even if the header and configuration space
>>>> addresses fit into 32 bits. For example with the Versatile Express
>>>> memory map using 4G memory, the following error occured when trying
>>>> to use a VirtIO-MMIO block device.
>>>>
>>>> EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: :
>>>> unaligned directory entry - offset=0, inode=3755990991, rec_len=57311,
>>>> name_len=223
>>>>
>>>> To fix this, select ARCH_DMA_ADDR_T_64BIT when both LPAE and
>>>> VIRTIO_MMIO are selected.
>>>>
>>>> Signed-off-by: Christopher Covington <cov@codeaurora.org>
>>>> ---
>>>> arch/arm/mm/Kconfig | 1 +
>>>> 1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
>>>> index 1f8fed9..a62bcc9 100644
>>>> --- a/arch/arm/mm/Kconfig
>>>> +++ b/arch/arm/mm/Kconfig
>>>> @@ -617,6 +617,7 @@ config ARM_LPAE
>>>> 	bool "Support for the Large Physical Address Extension"
>>>> 	depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
>>>> 		!CPU_32v4 && !CPU_32v3
>>>> +	select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
>>>
>>> That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
>>> platform with >32-bit physical address space? If yes, it should be
>>> selected there.
>>
>> The platforms I'm currently using are models like the Versatile Express
>> RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.
> 
> But do you use RAM beyond 32-bit on such models?

Yes. On the Versatile Express memory map only 2G RAM is 32-bit accessible, and
I'm using 4G as I tried to note in the commit message.

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0575h/Bbajihec.html

(For reference, the device tree I'm passing the A32 kernel is
arch/arm64/boot/dts/rtsm_ve-aemv8a.dts.)

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH] ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO

[Christopher Covington]

Hi Arnd,

On 03/21/2014 07:52 PM, Arnd Bergmann wrote:
> On Friday 21 March 2014 23:27:24 Catalin Marinas wrote:
>> On 21 Mar 2014, at 19:44, Christopher Covington <cov@codeaurora.org> wrote:
>>> On 03/21/2014 12:27 PM, Catalin Marinas wrote:
>>>>> diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
>>>>> index 1f8fed9..a62bcc9 100644
>>>>> --- a/arch/arm/mm/Kconfig
>>>>> +++ b/arch/arm/mm/Kconfig
>>>>> @@ -617,6 +617,7 @@ config ARM_LPAE
>>>>>     bool "Support for the Large Physical Address Extension"
>>>>>     depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \
>>>>>             !CPU_32v4 && !CPU_32v3
>>>>> +   select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
>>>>
>>>> That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a
>>>> platform with >32-bit physical address space? If yes, it should be
>>>> selected there.
>>>
>>> The platforms I'm currently using are models like the Versatile Express
>>> RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.
>>
>> But do you use RAM beyond 32-bit on such models?
> 
> I think the more important question here is what the normal behavior is
> for these platforms. I believe in most cases you don't have RAM above
> the boundary, so we should not enable the option by default as it can
> have noticeable overhead (we'd turn it on all the time if it didn't).

I don't understand why people concerned with the overhead of 64-bit pointers
to memory are turning on LPAE.

> In either case, platforms that need this support can always
> 'select' it, while a kernel built only for platforms that don't
> need it can offer this as a user-selectable option.

I was hoping to fix the issue of things mysteriously breaking at runtime,
regardless of what the user or randconfig selects. I'll have to investigate
the details, but it seems like one could enforce zone DMA at run-time if the
memory size is greater than the DMA pointers are able to address.

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH v2] ARM: virt, vexpress: Use 64-bit DMA for LPAE

[Christopher Covington]

On an LPAE system, the physical addresses used by VirtIO-MMIO may be
higher than 32 bits. For example on a Versatile Express system with
4G RAM, the following error surfaces when trying to use a VirtIO-MMIO
block device.

EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: : unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223

To fix this, select ARCH_DMA_ADDR_T_64BIT when LPAE is selected
and mach-vexpress, mach-virt, or both are being built.

Signed-off-by: Christopher Covington <cov@codeaurora.org>
---
 arch/arm/Kconfig               | 1 +
 arch/arm/mach-vexpress/Kconfig | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 245058b..8407c28 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -901,6 +901,7 @@ endmenu
 
 config ARCH_VIRT
 	bool "Dummy Virtual Machine" if ARCH_MULTI_V7
+	select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE
 	select ARM_AMBA
 	select ARM_GIC
 	select ARM_PSCI
diff --git a/arch/arm/mach-vexpress/Kconfig b/arch/arm/mach-vexpress/Kconfig
index 99c1f15..5f33e16 100644
--- a/arch/arm/mach-vexpress/Kconfig
+++ b/arch/arm/mach-vexpress/Kconfig
@@ -1,5 +1,6 @@
 menuconfig ARCH_VEXPRESS
 	bool "ARM Ltd. Versatile Express family" if ARCH_MULTI_V7
+	select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE
 	select ARCH_REQUIRE_GPIOLIB
 	select ARCH_SUPPORTS_BIG_ENDIAN
 	select ARM_AMBA
-- 
Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by the Linux Foundation.