Re: [PATCH rdma-next v2 1/4] IB/mlx5: Introduce ODP diagnostic counters

[Jason Gunthorpe <jgg@mellanox.com>]

On Thu, Oct 10, 2019 at 12:02:03PM +0300, Leon Romanovsky wrote:

> > >  static inline bool is_odp_mr(struct mlx5_ib_mr *mr)
> > > diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
> > > index 95cf0249b015..966783bfb557 100644
> > > +++ b/drivers/infiniband/hw/mlx5/odp.c
> > > @@ -261,6 +261,10 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start,
> > >  				blk_start_idx = idx;
> > >  				in_block = 1;
> > >  			}
> > > +
> > > +			/* Count page invalidations */
> > > +			mlx5_update_odp_stats(mr, invalidations,
> > > +					      (idx - blk_start_idx + 1));
> >
> > I feel like these should be batched and the atomic done once at the
> > end of the routine..
> 
> We can, but does it worth it?

Probably since it is so simple, atomics are very expensive

> For various reasons we are delaying this series for months already.
> Let's drop "prefetch" counter for now and merge everything without
> it.

OK, I guess the counters are extendible as we go along, however see below:

> > This is also not quite right for prefetch as we are doing a form of
> > prefetching in the mlx5_ib_mr_rdma_pfault_handler() too, although it
> > is less clear how to count those. Maybe this should be split to SQ/RQ
> > faults?
> 
> mlx5_ib_mr_rdma_pfault_handler() calls to pagefault_single_data_segment()
> without MLX5_PF_FLAGS_PREFETCH, so I'm unsure that this counter should
> count mlx5_ib_mr_rdma_pfault_handler() pagefaults.
> 
> However the idea to separate SQ/RQ for everything sounds appealing.

Let's at least have a well defined counter design. SQ/RQ seems like a
good split to me as they have quite different behavior on mlx5
hardware, so splitting the existing counter seems good anyhow

Jason