[PATCH net-next 0/2] net: axienet: fix DMA Tx error

[PATCH net-next 0/2] net: axienet: fix DMA Tx error

[Andy Chiu]

We ran into multiple DMA TX errors while writing files over a network
block device running on top of a DMA-connected AXI Ethernet device on
64-bit RISC-V machines. The errors indicated that the DMA had fetched a
null descriptor and we found that the reason for this is that AXI DMA had
unexpectedly processed a partially updated tail descriptor pointer. To
fix it, we suggest that the driver should use one 64-bit write instead
of two 32-bit writes to perform such update if possible. For those
archectures where double-word load/stores are unavailable, e.g. 32-bit
archectures, force a driver probe failure if the driver finds 64-bit
capability on DMA.

Andy Chiu (2):
  net: axienet: make the 64b addresable DMA depends on 64b archectures
  net: axienet: Use iowrite64 to write all 64b descriptor pointers

 drivers/net/ethernet/xilinx/xilinx_axienet.h  | 51 +++++++++++++++++++
 .../net/ethernet/xilinx/xilinx_axienet_main.c | 28 ++--------
 2 files changed, 55 insertions(+), 24 deletions(-)

-- 
2.36.0

[PATCH net-next 1/2] net: axienet: make the 64b addresable DMA depends on 64b archectures

[Andy Chiu]

Currently it is not safe to config the IP as 64-bit addressable on 32-bit
archectures, which cannot perform a double-word store on its descriptor
pointers. The pointer is 64-bit wide if the IP is configured as 64-bit,
and the device would process the partially updated pointer on some
states if the pointer was updated via two store-words. To prevent such
condition, we force a probe fail if we discover that the IP has 64-bit
capability but it is not running on a 64-Bit kernel.

This is a series of patch (1/2). The next patch must be applied in order
to make 64b DMA safe on 64b archectures.

Signed-off-by: Andy Chiu <andy.chiu@sifive.com>
Reported-by: Max Hsu <max.hsu@sifive.com>
---
 drivers/net/ethernet/xilinx/xilinx_axienet.h  | 36 +++++++++++++++++++
 .../net/ethernet/xilinx/xilinx_axienet_main.c | 28 +++------------
 2 files changed, 40 insertions(+), 24 deletions(-)

diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet.h b/drivers/net/ethernet/xilinx/xilinx_axienet.h
index 4225efbeda3d..6c95676ba172 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet.h
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet.h
@@ -547,6 +547,42 @@ static inline void axienet_iow(struct axienet_local *lp, off_t offset,
 	iowrite32(value, lp->regs + offset);
 }
 
+/**
+ * axienet_dma_out32 - Memory mapped Axi DMA register write.
+ * @lp:		Pointer to axienet local structure
+ * @reg:	Address offset from the base address of the Axi DMA core
+ * @value:	Value to be written into the Axi DMA register
+ *
+ * This function writes the desired value into the corresponding Axi DMA
+ * register.
+ */
+
+static inline void axienet_dma_out32(struct axienet_local *lp,
+				     off_t reg, u32 value)
+{
+	iowrite32(value, lp->dma_regs + reg);
+}
+
+#ifdef CONFIG_64BIT
+static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
+				 dma_addr_t addr)
+{
+	axienet_dma_out32(lp, reg, lower_32_bits(addr));
+
+	if (lp->features & XAE_FEATURE_DMA_64BIT)
+		axienet_dma_out32(lp, reg + 4, upper_32_bits(addr));
+}
+
+#else /* CONFIG_64BIT */
+
+static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
+				 dma_addr_t addr)
+{
+	axienet_dma_out32(lp, reg, lower_32_bits(addr));
+}
+
+#endif /* CONFIG_64BIT */
+
 /* Function prototypes visible in xilinx_axienet_mdio.c for other files */
 int axienet_mdio_enable(struct axienet_local *lp);
 void axienet_mdio_disable(struct axienet_local *lp);
diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
index 93c9f305bba4..fa7bcd2c1892 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
@@ -133,30 +133,6 @@ static inline u32 axienet_dma_in32(struct axienet_local *lp, off_t reg)
 	return ioread32(lp->dma_regs + reg);
 }
 
-/**
- * axienet_dma_out32 - Memory mapped Axi DMA register write.
- * @lp:		Pointer to axienet local structure
- * @reg:	Address offset from the base address of the Axi DMA core
- * @value:	Value to be written into the Axi DMA register
- *
- * This function writes the desired value into the corresponding Axi DMA
- * register.
- */
-static inline void axienet_dma_out32(struct axienet_local *lp,
-				     off_t reg, u32 value)
-{
-	iowrite32(value, lp->dma_regs + reg);
-}
-
-static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
-				 dma_addr_t addr)
-{
-	axienet_dma_out32(lp, reg, lower_32_bits(addr));
-
-	if (lp->features & XAE_FEATURE_DMA_64BIT)
-		axienet_dma_out32(lp, reg + 4, upper_32_bits(addr));
-}
-
 static void desc_set_phys_addr(struct axienet_local *lp, dma_addr_t addr,
 			       struct axidma_bd *desc)
 {
@@ -2061,6 +2037,10 @@ static int axienet_probe(struct platform_device *pdev)
 			iowrite32(0x0, desc);
 		}
 	}
+	if (!IS_ENABLED(CONFIG_64BIT) && lp->features & XAE_FEATURE_DMA_64BIT) {
+		dev_err(&pdev->dev, "64-bit addressable DMA is not compatible with 32-bit archecture\n");
+		goto cleanup_clk;
+	}
 
 	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(addr_width));
 	if (ret) {
-- 
2.36.0

Re: [PATCH net-next 1/2] net: axienet: make the 64b addresable DMA depends on 64b archectures

[Andy Chiu]

Reviewed-by: Greentime Hu <greentime.hu@sifive.com>

On Mon, Jun 13, 2022 at 11:45 AM Andy Chiu <andy.chiu@sifive.com> wrote:
>
> Currently it is not safe to config the IP as 64-bit addressable on 32-bit
> archectures, which cannot perform a double-word store on its descriptor
> pointers. The pointer is 64-bit wide if the IP is configured as 64-bit,
> and the device would process the partially updated pointer on some
> states if the pointer was updated via two store-words. To prevent such
> condition, we force a probe fail if we discover that the IP has 64-bit
> capability but it is not running on a 64-Bit kernel.
>
> This is a series of patch (1/2). The next patch must be applied in order
> to make 64b DMA safe on 64b archectures.
>
> Signed-off-by: Andy Chiu <andy.chiu@sifive.com>
> Reported-by: Max Hsu <max.hsu@sifive.com>
> ---
>  drivers/net/ethernet/xilinx/xilinx_axienet.h  | 36 +++++++++++++++++++
>  .../net/ethernet/xilinx/xilinx_axienet_main.c | 28 +++------------
>  2 files changed, 40 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet.h b/drivers/net/ethernet/xilinx/xilinx_axienet.h
> index 4225efbeda3d..6c95676ba172 100644
> --- a/drivers/net/ethernet/xilinx/xilinx_axienet.h
> +++ b/drivers/net/ethernet/xilinx/xilinx_axienet.h
> @@ -547,6 +547,42 @@ static inline void axienet_iow(struct axienet_local *lp, off_t offset,
>         iowrite32(value, lp->regs + offset);
>  }
>
> +/**
> + * axienet_dma_out32 - Memory mapped Axi DMA register write.
> + * @lp:                Pointer to axienet local structure
> + * @reg:       Address offset from the base address of the Axi DMA core
> + * @value:     Value to be written into the Axi DMA register
> + *
> + * This function writes the desired value into the corresponding Axi DMA
> + * register.
> + */
> +
> +static inline void axienet_dma_out32(struct axienet_local *lp,
> +                                    off_t reg, u32 value)
> +{
> +       iowrite32(value, lp->dma_regs + reg);
> +}
> +
> +#ifdef CONFIG_64BIT
> +static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
> +                                dma_addr_t addr)
> +{
> +       axienet_dma_out32(lp, reg, lower_32_bits(addr));
> +
> +       if (lp->features & XAE_FEATURE_DMA_64BIT)
> +               axienet_dma_out32(lp, reg + 4, upper_32_bits(addr));
> +}
> +
> +#else /* CONFIG_64BIT */
> +
> +static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
> +                                dma_addr_t addr)
> +{
> +       axienet_dma_out32(lp, reg, lower_32_bits(addr));
> +}
> +
> +#endif /* CONFIG_64BIT */
> +
>  /* Function prototypes visible in xilinx_axienet_mdio.c for other files */
>  int axienet_mdio_enable(struct axienet_local *lp);
>  void axienet_mdio_disable(struct axienet_local *lp);
> diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> index 93c9f305bba4..fa7bcd2c1892 100644
> --- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> +++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
> @@ -133,30 +133,6 @@ static inline u32 axienet_dma_in32(struct axienet_local *lp, off_t reg)
>         return ioread32(lp->dma_regs + reg);
>  }
>
> -/**
> - * axienet_dma_out32 - Memory mapped Axi DMA register write.
> - * @lp:                Pointer to axienet local structure
> - * @reg:       Address offset from the base address of the Axi DMA core
> - * @value:     Value to be written into the Axi DMA register
> - *
> - * This function writes the desired value into the corresponding Axi DMA
> - * register.
> - */
> -static inline void axienet_dma_out32(struct axienet_local *lp,
> -                                    off_t reg, u32 value)
> -{
> -       iowrite32(value, lp->dma_regs + reg);
> -}
> -
> -static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
> -                                dma_addr_t addr)
> -{
> -       axienet_dma_out32(lp, reg, lower_32_bits(addr));
> -
> -       if (lp->features & XAE_FEATURE_DMA_64BIT)
> -               axienet_dma_out32(lp, reg + 4, upper_32_bits(addr));
> -}
> -
>  static void desc_set_phys_addr(struct axienet_local *lp, dma_addr_t addr,
>                                struct axidma_bd *desc)
>  {
> @@ -2061,6 +2037,10 @@ static int axienet_probe(struct platform_device *pdev)
>                         iowrite32(0x0, desc);
>                 }
>         }
> +       if (!IS_ENABLED(CONFIG_64BIT) && lp->features & XAE_FEATURE_DMA_64BIT) {
> +               dev_err(&pdev->dev, "64-bit addressable DMA is not compatible with 32-bit archecture\n");
> +               goto cleanup_clk;
> +       }
>
>         ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(addr_width));
>         if (ret) {
> --
> 2.36.0
>

[PATCH net-next 2/2] net: axienet: Use iowrite64 to write all 64b descriptor pointers

[Andy Chiu]

According to commit f735c40ed93c ("net: axienet: Autodetect 64-bit DMA
capability") and AXI-DMA spec (pg021), on 64-bit capable dma, only
writing MSB part of tail descriptor pointer causes DMA engine to start
fetching descriptors. However, we found that it is true only if dma is in
idle state. In other words, dma would use a tailp even if it only has LSB
updated, when the dma is running.

The non-atomicity of this behavior could be problematic if enough
delay were introduced in between the 2 writes. For example, if an
interrupt comes right after the LSB write and the cpu spends long
enough time in the handler for the dma to get back into idle state by
completing descriptors, then the seconcd write to MSB would treat dma
to start fetching descriptors again. Since the descriptor next to the
one pointed by current tail pointer is not filled by the kernel yet,
fetching a null descriptor here causes a dma internal error and halt
the dma engine down.

We suggest that the dma engine should start process a 64-bit MMIO write
to the descriptor pointer only if ONE 32-bit part of it is written on all
states. Or we should restrict the use of 64-bit addressable dma on 32-bit
platforms, since those devices have no instruction to guarantee the write
to LSB and MSB part of tail pointer occurs atomically to the dma.

initial condition:
curp =  x-3;
tailp = x-2;
LSB = x;
MSB = 0;

cpu:                       |dma:
 iowrite32(LSB, tailp)     |  completes #(x-3) desc, curp = x-3
 ...                       |  tailp updated
 => irq                    |  completes #(x-2) desc, curp = x-2
    ...                    |  completes #(x-1) desc, curp = x-1
    ...                    |  ...
    ...                    |  completes #x desc, curp = tailp = x
 <= irqreturn              |  reaches tailp == curp = x, idle
 iowrite32(MSB, tailp + 4) |  ...
                           |  tailp updated, starts fetching...
                           |  fetches #(x + 1) desc, sees cntrl = 0
                           |  post Tx error, halt

Signed-off-by: Andy Chiu <andy.chiu@sifive.com>
Reported-by: Max Hsu <max.hsu@sifive.com>
---
 drivers/net/ethernet/xilinx/xilinx_axienet.h | 21 +++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet.h b/drivers/net/ethernet/xilinx/xilinx_axienet.h
index 6c95676ba172..97ddc0273b8a 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet.h
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet.h
@@ -564,13 +564,28 @@ static inline void axienet_dma_out32(struct axienet_local *lp,
 }
 
 #ifdef CONFIG_64BIT
+/**
+ * axienet_dma_out64 - Memory mapped Axi DMA register write.
+ * @lp:		Pointer to axienet local structure
+ * @reg:	Address offset from the base address of the Axi DMA core
+ * @value:	Value to be written into the Axi DMA register
+ *
+ * This function writes the desired value into the corresponding Axi DMA
+ * register.
+ */
+static inline void axienet_dma_out64(struct axienet_local *lp,
+				     off_t reg, u64 value)
+{
+	iowrite64(value, lp->dma_regs + reg);
+}
+
 static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
 				 dma_addr_t addr)
 {
-	axienet_dma_out32(lp, reg, lower_32_bits(addr));
-
 	if (lp->features & XAE_FEATURE_DMA_64BIT)
-		axienet_dma_out32(lp, reg + 4, upper_32_bits(addr));
+		axienet_dma_out64(lp, reg, addr);
+	else
+		axienet_dma_out32(lp, reg, lower_32_bits(addr));
 }
 
 #else /* CONFIG_64BIT */
-- 
2.36.0

Re: [PATCH net-next 2/2] net: axienet: Use iowrite64 to write all 64b descriptor pointers

[Andy Chiu]

Reviewed-by: Greentime Hu <greentime.hu@sifive.com>

On Mon, Jun 13, 2022 at 11:45 AM Andy Chiu <andy.chiu@sifive.com> wrote:
>
> According to commit f735c40ed93c ("net: axienet: Autodetect 64-bit DMA
> capability") and AXI-DMA spec (pg021), on 64-bit capable dma, only
> writing MSB part of tail descriptor pointer causes DMA engine to start
> fetching descriptors. However, we found that it is true only if dma is in
> idle state. In other words, dma would use a tailp even if it only has LSB
> updated, when the dma is running.
>
> The non-atomicity of this behavior could be problematic if enough
> delay were introduced in between the 2 writes. For example, if an
> interrupt comes right after the LSB write and the cpu spends long
> enough time in the handler for the dma to get back into idle state by
> completing descriptors, then the seconcd write to MSB would treat dma
> to start fetching descriptors again. Since the descriptor next to the
> one pointed by current tail pointer is not filled by the kernel yet,
> fetching a null descriptor here causes a dma internal error and halt
> the dma engine down.
>
> We suggest that the dma engine should start process a 64-bit MMIO write
> to the descriptor pointer only if ONE 32-bit part of it is written on all
> states. Or we should restrict the use of 64-bit addressable dma on 32-bit
> platforms, since those devices have no instruction to guarantee the write
> to LSB and MSB part of tail pointer occurs atomically to the dma.
>
> initial condition:
> curp =  x-3;
> tailp = x-2;
> LSB = x;
> MSB = 0;
>
> cpu:                       |dma:
>  iowrite32(LSB, tailp)     |  completes #(x-3) desc, curp = x-3
>  ...                       |  tailp updated
>  => irq                    |  completes #(x-2) desc, curp = x-2
>     ...                    |  completes #(x-1) desc, curp = x-1
>     ...                    |  ...
>     ...                    |  completes #x desc, curp = tailp = x
>  <= irqreturn              |  reaches tailp == curp = x, idle
>  iowrite32(MSB, tailp + 4) |  ...
>                            |  tailp updated, starts fetching...
>                            |  fetches #(x + 1) desc, sees cntrl = 0
>                            |  post Tx error, halt
>
> Signed-off-by: Andy Chiu <andy.chiu@sifive.com>
> Reported-by: Max Hsu <max.hsu@sifive.com>
> ---
>  drivers/net/ethernet/xilinx/xilinx_axienet.h | 21 +++++++++++++++++---
>  1 file changed, 18 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet.h b/drivers/net/ethernet/xilinx/xilinx_axienet.h
> index 6c95676ba172..97ddc0273b8a 100644
> --- a/drivers/net/ethernet/xilinx/xilinx_axienet.h
> +++ b/drivers/net/ethernet/xilinx/xilinx_axienet.h
> @@ -564,13 +564,28 @@ static inline void axienet_dma_out32(struct axienet_local *lp,
>  }
>
>  #ifdef CONFIG_64BIT
> +/**
> + * axienet_dma_out64 - Memory mapped Axi DMA register write.
> + * @lp:                Pointer to axienet local structure
> + * @reg:       Address offset from the base address of the Axi DMA core
> + * @value:     Value to be written into the Axi DMA register
> + *
> + * This function writes the desired value into the corresponding Axi DMA
> + * register.
> + */
> +static inline void axienet_dma_out64(struct axienet_local *lp,
> +                                    off_t reg, u64 value)
> +{
> +       iowrite64(value, lp->dma_regs + reg);
> +}
> +
>  static void axienet_dma_out_addr(struct axienet_local *lp, off_t reg,
>                                  dma_addr_t addr)
>  {
> -       axienet_dma_out32(lp, reg, lower_32_bits(addr));
> -
>         if (lp->features & XAE_FEATURE_DMA_64BIT)
> -               axienet_dma_out32(lp, reg + 4, upper_32_bits(addr));
> +               axienet_dma_out64(lp, reg, addr);
> +       else
> +               axienet_dma_out32(lp, reg, lower_32_bits(addr));
>  }
>
>  #else /* CONFIG_64BIT */
> --
> 2.36.0
>

Re: [PATCH net-next 2/2] net: axienet: Use iowrite64 to write all 64b descriptor pointers

[kernel test robot]

Hi Andy,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Andy-Chiu/net-axienet-fix-DMA-Tx-error/20220613-114738
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 27f2533bcc6e909b85d3c1b738fa1f203ed8a835
config: x86_64-randconfig-r002-20220613 (https://download.01.org/0day-ci/archive/20220614/202206140650.4x173WyJ-lkp@intel.com/config)
compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project d378268ead93c85803c270277f0243737b536ae7)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/intel-lab-lkp/linux/commit/6c45bbcce325f6c9c907ed0a11ddc13d8026bbcf
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Andy-Chiu/net-axienet-fix-DMA-Tx-error/20220613-114738
        git checkout 6c45bbcce325f6c9c907ed0a11ddc13d8026bbcf
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

>> ld.lld: error: undefined symbol: iowrite64
   >>> referenced by amd_bus.c
   >>>               vmlinux.o:(axienet_dma_out_addr)

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

Re: [PATCH net-next 0/2] net: axienet: fix DMA Tx error

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net.git (master)
by David S. Miller <davem@davemloft.net>:

On Mon, 13 Jun 2022 11:42:00 +0800 you wrote:
> We ran into multiple DMA TX errors while writing files over a network
> block device running on top of a DMA-connected AXI Ethernet device on
> 64-bit RISC-V machines. The errors indicated that the DMA had fetched a
> null descriptor and we found that the reason for this is that AXI DMA had
> unexpectedly processed a partially updated tail descriptor pointer. To
> fix it, we suggest that the driver should use one 64-bit write instead
> of two 32-bit writes to perform such update if possible. For those
> archectures where double-word load/stores are unavailable, e.g. 32-bit
> archectures, force a driver probe failure if the driver finds 64-bit
> capability on DMA.
> 
> [...]

Here is the summary with links:
  - [net-next,1/2] net: axienet: make the 64b addresable DMA depends on 64b archectures
    https://git.kernel.org/netdev/net/c/00be43a74ca2
  - [net-next,2/2] net: axienet: Use iowrite64 to write all 64b descriptor pointers
    https://git.kernel.org/netdev/net/c/b690f8df6497

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html