[PATCH] net: macb: Align the dma and coherent dma masks

[PATCH] net: macb: Align the dma and coherent dma masks

[Harini Katakam]

From: Marc St-Amand <mstamand@ciena.com>

Single page and coherent memory blocks can use different DMA masks
when the macb accesses physical memory directly. The kernel is clever
enough to allocate pages that fit into the requested address width.

When using the ARM SMMU, the DMA mask must be the same for single
pages and big coherent memory blocks. Otherwise the translation
tables turn into one big mess.

  [   74.959909] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
  [   74.959989] arm-smmu fd800000.smmu: Unhandled context fault: fsr=0x402, iova=0x3165687460, fsynr=0x20001, cbfrsynra=0x877, cb=1
  [   75.173939] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
  [   75.173955] arm-smmu fd800000.smmu: Unhandled context fault: fsr=0x402, iova=0x3165687460, fsynr=0x20001, cbfrsynra=0x877, cb=1

Since using the same DMA mask does not hurt direct 1:1 physical
memory mappings, this commit always aligns DMA and coherent masks.

Signed-off-by: Marc St-Amand <mstamand@ciena.com>
Signed-off-by: Harini Katakam <harini.katakam@xilinx.com>
---
 drivers/net/ethernet/cadence/macb_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
index 1ce20bf52f72..4c231159b562 100644
--- a/drivers/net/ethernet/cadence/macb_main.c
+++ b/drivers/net/ethernet/cadence/macb_main.c
@@ -4765,7 +4765,7 @@ static int macb_probe(struct platform_device *pdev)
 
 #ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
 	if (GEM_BFEXT(DAW64, gem_readl(bp, DCFG6))) {
-		dma_set_mask(&pdev->dev, DMA_BIT_MASK(44));
+		dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(44));
 		bp->hw_dma_cap |= HW_DMA_CAP_64B;
 	}
 #endif
-- 
2.17.1

Re: [PATCH] net: macb: Align the dma and coherent dma masks

[Nicolas Ferre]

On 09/02/2022 at 10:43, Harini Katakam wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> 
> From: Marc St-Amand <mstamand@ciena.com>
> 
> Single page and coherent memory blocks can use different DMA masks
> when the macb accesses physical memory directly. The kernel is clever
> enough to allocate pages that fit into the requested address width.
> 
> When using the ARM SMMU, the DMA mask must be the same for single
> pages and big coherent memory blocks. Otherwise the translation
> tables turn into one big mess.
> 
>    [   74.959909] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
>    [   74.959989] arm-smmu fd800000.smmu: Unhandled context fault: fsr=0x402, iova=0x3165687460, fsynr=0x20001, cbfrsynra=0x877, cb=1
>    [   75.173939] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
>    [   75.173955] arm-smmu fd800000.smmu: Unhandled context fault: fsr=0x402, iova=0x3165687460, fsynr=0x20001, cbfrsynra=0x877, cb=1
> 
> Since using the same DMA mask does not hurt direct 1:1 physical
> memory mappings, this commit always aligns DMA and coherent masks.
> 
> Signed-off-by: Marc St-Amand <mstamand@ciena.com>
> Signed-off-by: Harini Katakam <harini.katakam@xilinx.com>

Ok:
Acked-by: Nicolas Ferre <nicolas.ferre@microchip.com>

Best regards,
   Nicolas

> ---
>   drivers/net/ethernet/cadence/macb_main.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
> index 1ce20bf52f72..4c231159b562 100644
> --- a/drivers/net/ethernet/cadence/macb_main.c
> +++ b/drivers/net/ethernet/cadence/macb_main.c
> @@ -4765,7 +4765,7 @@ static int macb_probe(struct platform_device *pdev)
> 
>   #ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
>          if (GEM_BFEXT(DAW64, gem_readl(bp, DCFG6))) {
> -               dma_set_mask(&pdev->dev, DMA_BIT_MASK(44));
> +               dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(44));
>                  bp->hw_dma_cap |= HW_DMA_CAP_64B;
>          }
>   #endif
> --
> 2.17.1
> 


-- 
Nicolas Ferre

Re: [PATCH] net: macb: Align the dma and coherent dma masks

[conor.dooley]

> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> 
> From: Marc St-Amand <mstamand@ciena.com>
> 
> Single page and coherent memory blocks can use different DMA masks
> when the macb accesses physical memory directly. The kernel is clever
> enough to allocate pages that fit into the requested address width.
> 
> When using the ARM SMMU, the DMA mask must be the same for single
> pages and big coherent memory blocks. Otherwise the translation
> tables turn into one big mess.
> 
>   [   74.959909] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
>   [   74.959989] arm-smmu fd800000.smmu: Unhandled context fault: fsr=0x402, iova=0x3165687460, fsynr=0x20001, cbfrsynra=0x877, cb=1
>   [   75.173939] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
>   [   75.173955] arm-smmu fd800000.smmu: Unhandled context fault: fsr=0x402, iova=0x3165687460, fsynr=0x20001, cbfrsynra=0x877, cb=1
> 
> Since using the same DMA mask does not hurt direct 1:1 physical
> memory mappings, this commit always aligns DMA and coherent masks.
> 
> Signed-off-by: Marc St-Amand <mstamand@ciena.com>
> Signed-off-by: Harini Katakam <harini.katakam@xilinx.com>

Tested-by: Conor Dooley <conor.dooley@microchip.com>

Fixes a DMA allocation failure for me on a non arm board, seen if
I only assigned DDR to 64 bit addresses. Without the change I was
getting the following page allocation error:

Starting network:
[    2.911830] ip: page allocation failure: order:2, mode:0xcc4(GFP_KERNEL|GFP_DMA32),nodemask=(null)
[    2.921256] CPU: 3 PID: 171 Comm: ip Not tainted 5.17.0-rc1-00640-g6cc001edd9ad #1
[    2.928915] Hardware name: Microchip PolarFire-SoC Icicle Kit (DT)
[    2.935147] Call Trace:
[    2.937626] [<ffffffff800047e0>] dump_backtrace+0x1c/0x24
[    2.943087] [<ffffffff807fcfce>] dump_stack_lvl+0x40/0x58
[    2.948547] [<ffffffff807fcffa>] dump_stack+0x14/0x1c
[    2.953649] [<ffffffff80104bfc>] warn_alloc+0xd6/0x14c
[    2.958849] [<ffffffff801053e0>] __alloc_pages_slowpath.constprop.0+0x76e/0x882
[    2.966242] [<ffffffff80105618>] __alloc_pages+0x124/0x174
[    2.971783] [<ffffffff80061440>] __dma_direct_alloc_pages+0x12c/0x28c
[    2.978286] [<ffffffff800616f2>] dma_direct_alloc+0x40/0x13e
[    2.983996] [<ffffffff80060bf2>] dma_alloc_attrs+0x78/0x86
[    2.989541] [<ffffffff805cdfb6>] macb_open+0x84/0x42c
[    2.994645] [<ffffffff806e2468>] __dev_open+0xb0/0x142
[    2.999845] [<ffffffff806e2884>] __dev_change_flags+0x180/0x1ec
[    3.005827] [<ffffffff806e290e>] dev_change_flags+0x1e/0x54
[    3.011461] [<ffffffff8076960e>] devinet_ioctl+0x1fc/0x612
[    3.017015] [<ffffffff8076b27e>] inet_ioctl+0x96/0xfa
[    3.022109] [<ffffffff806bf5e2>] sock_ioctl+0x256/0x29e
[    3.027396] [<ffffffff8012bc7c>] sys_ioctl+0x340/0x7f8
[    3.032595] [<ffffffff80003014>] ret_from_syscall+0x0/0x2
<snip>
[    3.176712] macb 20110000.ethernet eth0: Unable to allocate DMA memory (error -12)
ip: SIOCSIFFLAGS: Cannot allocate memory
FAIL

Re: [PATCH] net: macb: Align the dma and coherent dma masks

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (master)
by David S. Miller <davem@davemloft.net>:

On Wed, 9 Feb 2022 15:13:25 +0530 you wrote:
> From: Marc St-Amand <mstamand@ciena.com>
> 
> Single page and coherent memory blocks can use different DMA masks
> when the macb accesses physical memory directly. The kernel is clever
> enough to allocate pages that fit into the requested address width.
> 
> When using the ARM SMMU, the DMA mask must be the same for single
> pages and big coherent memory blocks. Otherwise the translation
> tables turn into one big mess.
> 
> [...]

Here is the summary with links:
  - net: macb: Align the dma and coherent dma masks
    https://git.kernel.org/netdev/net/c/37f7860602b5

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html