Re: [PATCH] net: mlx5: Use writeX() to ring doorbell and remove reduntant wmb()

[Jason Gunthorpe <jgg@ziepe.ca>]

On Thu, Jan 02, 2020 at 07:44:36PM +0200, Liran Alon wrote:
> Currently, mlx5e_notify_hw() executes wmb() to complete writes to cache-coherent
> memory before ringing doorbell. Doorbell is written to by mlx5_write64()
> which use __raw_writeX().
> 
> This is semantically correct but executes reduntant wmb() in some architectures.
> For example, in x86, a write to UC memory guarantees that any previous write to
> WB memory will be globally visible before the write to UC memory. Therefore, there
> is no need to also execute wmb() before write to doorbell which is mapped as UC memory.
> 
> The consideration regarding this between different architectures is handled
> properly by the writeX() macro. Which is defined differently for different
> architectures. E.g. On x86, it is just a memory write. However, on ARM, it
> is defined as __iowmb() folowed by a memory write. __iowmb() is defined
> as wmb().

This reasoning seems correct, though I would recommend directly
refering to locking/memory-barriers.txt which explains this.

> Therefore, change mlx5_write64() to use writeX() and remove wmb() from
> it's callers.

Yes, wmb(); writel(); is always redundant
  
> diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
> index 40748fc1b11b..28744a725e64 100644
> +++ b/include/linux/mlx5/cq.h
> @@ -162,11 +162,6 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd,
>  
>  	*cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci);
>  
> -	/* Make sure that the doorbell record in host memory is
> -	 * written before ringing the doorbell via PCI MMIO.
> -	 */
> -	wmb();
> -

Why did this one change? The doorbell memory here is not a writel():

>  	doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
>  	doorbell[1] = cpu_to_be32(cq->cqn);

>  static inline void mlx5_write64(__be32 val[2], void __iomem *dest)
>  {
>  #if BITS_PER_LONG == 64
> -	__raw_writeq(*(u64 *)val, dest);
> +	writeq(*(u64 *)val, dest);

I want to say this might cause problems with endian swapping as writeq
also does some swaps that __raw does not? Is this true?

ie writeq does not accept a be32

Some time ago I reworked this similar code in userspace to use a u64
and remove the swapping from the caller.

Jason