[PATCH for-next 00/10] Add completion timestamping support

Or Gerlitz
2015-05-17T13:36:16Z

Hi Doug,

This patchset adds completion timestamping supports for verbs consumers.

Timestamping is used by applications in order to know when a WQE was
received/transmitted by the HW. The value is given is HCA hardware cycles,
but could be easily converted as the hardware's core clock frequecny is
available through extension of query device.

Moreover, we add an ability to read the HCA's current clock. This could be
useful on order to synchronize events to the wall clock.

This functionality is achieved by adding/extending the following verbs:

create_cq - create_cq is extended in order to allow passing creation flags
to the CQ creation function. We change IB/core --> vendors API
to be easily extendible by passing a struct which contains
comp_vectors, cqe and the new flags parameter. In order to create
CQ which supports timestamping, IB_CQ_FLAGS_TIMESTAMP should be given.

query_device - We extend query_device uverb further by giving the hardware's
clock frequency and the timestamp mask (the number of timestamp
bits which are supported). If timestamp isn't supported, 0 is returned.

In order to read the timestamp in the WQE, the user needs to query the device
for support, create an appropriate CQ (using the extanded uverb with
IB_CQ_FLAGS_TIMESTAMP) and poll the CQ with an extended poll_cq verb (currently,
only implemented in user-space).

In mlx4, allowing the user to read the core clock efficiently involves mapping
this area of the hardware to user-space (being done by using a mmap command)
and reading the clock from the correct offset of the page.

This offset is returned in the vendor's specific data from mlx4's kernel driver
to the mlx4's user-space driver. query_device is modified in order to support
passing this vendor specific data. A user-space application could use a new
verb in order to read the hardware's clock.

Translating the hardware's clock into ms could be done by dividing this
value by hca_core_clock (which is returned by the extended version of
query_device uverb).

Matan and Or.

Matan Barak (10):
IB/core: Change provider's API of create_cq to be extendible
IB/core: Add flags parameter for ib_create_cq
IB/core: Extend ib_uverbs_create_cq
IB/core: Add timestamp_mask and hca_core_clock to query_device
IB/core: Pass hardware specific data in query_device
IB/mlx4: Add mmap call to map the hardware clock
IB/mlx4: Support extended create_cq and query_device uverbs
IB/mlx4: Add support for timestamp in cq creation
IB/mlx4: Add timestamp_mask and hca_core_clock to query_device
IB/mlx4: Return hca core clock's offset in query_device vendor's data

drivers/infiniband/core/device.c | 6 +-
drivers/infiniband/core/mad.c | 4 +-
drivers/infiniband/core/uverbs.h | 1 +
drivers/infiniband/core/uverbs_cmd.c | 188 ++++++++++++++++----
drivers/infiniband/core/uverbs_main.c | 1 +
drivers/infiniband/core/verbs.c | 6 +-
drivers/infiniband/hw/amso1100/c2_provider.c | 14 ++-
drivers/infiniband/hw/cxgb3/iwch_provider.c | 19 ++-
drivers/infiniband/hw/cxgb4/cq.c | 9 +-
drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 8 +-
drivers/infiniband/hw/cxgb4/provider.c | 8 +-
drivers/infiniband/hw/ehca/ehca_cq.c | 7 +-
drivers/infiniband/hw/ehca/ehca_hca.c | 6 +-
drivers/infiniband/hw/ehca/ehca_iverbs.h | 6 +-
drivers/infiniband/hw/ehca/ehca_main.c | 3 +-
drivers/infiniband/hw/ipath/ipath_cq.c | 9 +-
drivers/infiniband/hw/ipath/ipath_verbs.c | 7 +-
drivers/infiniband/hw/ipath/ipath_verbs.h | 3 +-
drivers/infiniband/hw/mlx4/cq.c | 13 ++-
drivers/infiniband/hw/mlx4/mad.c | 2 +-
drivers/infiniband/hw/mlx4/main.c | 64 +++++++-
drivers/infiniband/hw/mlx4/mlx4_ib.h | 19 ++-
drivers/infiniband/hw/mlx5/cq.c | 10 +-
drivers/infiniband/hw/mlx5/main.c | 14 +-
drivers/infiniband/hw/mlx5/mlx5_ib.h | 5 +-
drivers/infiniband/hw/mthca/mthca_provider.c | 15 ++-
drivers/infiniband/hw/nes/nes_verbs.c | 17 ++-
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 13 ++-
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 9 +-
drivers/infiniband/hw/qib/qib_cq.c | 11 +-
drivers/infiniband/hw/qib/qib_verbs.c | 6 +-
drivers/infiniband/hw/qib/qib_verbs.h | 5 +-
drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 16 ++-
drivers/infiniband/hw/usnic/usnic_ib_verbs.h | 10 +-
drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 5 +-
drivers/infiniband/ulp/iser/iser_verbs.c | 8 +-
drivers/infiniband/ulp/isert/ib_isert.c | 7 +-
drivers/infiniband/ulp/srp/ib_srp.c | 4 +-
drivers/infiniband/ulp/srpt/ib_srpt.c | 2 +-
drivers/net/ethernet/mellanox/mlx4/main.c | 19 ++
.../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 7 +-
include/linux/mlx4/device.h | 9 +
include/rdma/ib_verbs.h | 21 ++-
include/uapi/rdma/ib_user_verbs.h | 19 ++
net/9p/trans_rdma.c | 2 +-
net/rds/ib_cm.c | 4 +-
net/rds/iw_cm.c | 8 +-
net/sunrpc/xprtrdma/svc_rdma_transport.c | 15 +-
net/sunrpc/xprtrdma/verbs.c | 8 +-
49 files changed, 520 insertions(+), 152 deletions(-)