[PATCH V1 for-next 0/7] Change IDR usage and locking in uverbs

[PATCH V1 for-next 0/7] Change IDR usage and locking in uverbs

[Matan Barak]

Hi Doug,

This series is the first part of introducing the new ABI.
It changes the IDR and locking mechanism in ib_uverbs.
This will allow handling all types in the same way, both IB/core
standard types and driver specific types. In that case, type is
referred to either IDR (such as QP, CQ, etc.) or FD such as
completion channel based uobjects.

Previously, we had global IDR tables per type and a per-type list on
the user context, which implied that the type system was fixed.
This patch series changes that into a uverbs-file IDR table (since
in the new proposed ABI each device could have a different type
system and a uverbs-file refers for a specific device) and a unified
per-context objects list. Objects in this list could either be IDR
based objects or FD based objects.

A type is represented by an identifier, an allocation size, a free
function which is used in the context tear down and a release order.
The allocation function is actually the handler (for example,
create_qp). The release order determines the relations between objects.
Some relations are set in the hardware or user-space. For example,
a MW could be created before MR, but if this MW is bounded to the MR,
the MW should be released before the MR.
Later on, we'll add actions in an object-oriented programming manner
to each type. So, a type contains all required information for
allocating its size and freeing it.

The locking mechanism is changed as well. Previously, the uverbs_cmd
handlers created new objects themselves and dealt with the internals
of locking/unlocking them. This is now moved to a separate code which
either creates a new object, destroys an object or locks it for
read/write. This is possible since we have a descriptive type system.
Hence, we could have a shared code to create, lock and destroy types.

In contrast to the previous locking approach, we don't block the
user-space thread if an object is already locked, but we just return
-EBUSY and expect the user to handle this. In order to maintain
backward compatibility, we've added explicit locks to the uverbs_cmd
handlers (in non-racy scenarios of course), thus for sane flows the
behaviour should be the same as previous one.
The incentive here is to provide a robust infrastructure to add new
actions that can't create a dead-lock (e.g. two commands that try to
lock objects in AB-BA manner).
In addition, since objects creation and locking is dealt in a
centralized place, the new proposed ABI won't need to deal with it
explicitly in its handlers.

A typical flow of a handler will be:
1. Serialize the user-space command to kernel-space structures.
2. Get all objects the handle needs (through the new centralized
     mechanism).
3. Do the actual command.
4. Commit or abort objects allocation/fetching/destruction.
5. Write response to the user.

We use the following locks/krefs:
1. ib_uobject: currently_used
	o Protects from concurrent read and write/destroy
2. ib_ucontext: lock
	o Protect from concurrently modifying the context uobject's list
3. ib_uobject: ref [kref]
	o Let's a handler delay the memory release of an uobject even if
	  it was already removed from the ucontext's objects list
	  (e.g. CQ takes a reference count on the completion handler)
	o rdma_core takes one reference count for all objects in its
	  repository.
4. ib_uverbs_file: cleanup_mutex [existing lock]
	o Protects concurrently closing a file from releasing the ucontext.
	  Since when removing a uobject, we need to remove it from the
	  uobjects list and thus we use the ucontext lock (clause 2),
	  we need to make sure the ucontext is alive throughout this process.

This patch-set is applied on top of Doug's k.o/for-4.11 branch.

Regards,
Matan

Changes from V0 - address some of Jason's comments:
1. Change idr table to be per uverbs_file and not per device.
2. Make changes more gradual
    a. Instead of doing a flags day approach, first introduce the idr
       uobjects with a new API and change their usages:
	 o Ditch the live flag
	 o Manage all idr objects through rdma_core
	 o create and cleanup context through rdma_core
    b. Add a lock to preserve old attch_mcast/detach_mcast behaviour.
    c. Add fd based objects infrastructure
    d. Change current completion channel to use this fd infrastructure.
3. Ditch the switch-case enums and favor a more OOP approach through a
   vtable.
4. Instead of having a kref-ed lock, put a new lock on the uverbs_file
   to synchronize fd objects deletion from context removal
    a. We favored this approach from keeping the ucontext alive, as the
       ucontext is allocated by the driver and thus we want to deallocate
       it through the same entity. We don't want to defer the whole
       deallocation process as in the current state we could rmmod the
       drivers code when applications are still running and trigger
       a disassociate flow.
5. Reduce the amount of macros - use only a macro for declaring idr
   types and a macro to declare fd types.
6. Use kref to manage all uobjects.
7. Use proper types for idr.

Leon Romanovsky (1):
  IB/core: Refactor idr to be per uverbs_file

Matan Barak (6):
  IB/core: Add support for idr types
  IB/core: Add idr based standard types
  IB/core: Change idr objects to use the new schema
  IB/core: Add lock to multicast handlers
  IB/core: Add support for fd objects
  IB/core: Change completion channel to use the reworked objects schema

 drivers/infiniband/core/Makefile           |   3 +-
 drivers/infiniband/core/rdma_core.c        | 437 +++++++++++++
 drivers/infiniband/core/rdma_core.h        |  74 +++
 drivers/infiniband/core/uverbs.h           |  50 +-
 drivers/infiniband/core/uverbs_cmd.c       | 980 +++++++++--------------------
 drivers/infiniband/core/uverbs_main.c      | 436 ++++++-------
 drivers/infiniband/core/uverbs_std_types.c | 193 ++++++
 include/rdma/ib_verbs.h                    |  27 +-
 include/rdma/uverbs_std_types.h            |  51 ++
 include/rdma/uverbs_types.h                | 148 +++++
 10 files changed, 1424 insertions(+), 975 deletions(-)
 create mode 100644 drivers/infiniband/core/rdma_core.c
 create mode 100644 drivers/infiniband/core/rdma_core.h
 create mode 100644 drivers/infiniband/core/uverbs_std_types.c
 create mode 100644 include/rdma/uverbs_std_types.h
 create mode 100644 include/rdma/uverbs_types.h

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 1/7] IB/core: Refactor idr to be per uverbs_file

[Matan Barak]

From: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

The current code creates an idr per type. Since types are currently
common for all drivers and known in advance, this was good enough.
However, the proposed ioctl based infrastructure allows each driver
to declare only some of the common types and declare its own specific
types.

Thus, we decided to implement idr to be per uverbs_file.

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Haggai Eran <haggaie-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/uverbs.h      |  19 ++---
 drivers/infiniband/core/uverbs_cmd.c  | 153 +++++++++++++++++-----------------
 drivers/infiniband/core/uverbs_main.c |  45 ++++------
 include/rdma/ib_verbs.h               |   1 +
 4 files changed, 95 insertions(+), 123 deletions(-)

diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 455034a..662a592 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -123,6 +123,10 @@ struct ib_uverbs_file {
 	struct ib_uverbs_event_file	       *async_file;
 	struct list_head			list;
 	int					is_closed;
+
+	struct idr		idr;
+	/* spinlock protects write access to idr */
+	spinlock_t		idr_lock;
 };
 
 struct ib_uverbs_event {
@@ -176,20 +180,7 @@ struct ib_ucq_object {
 	u32			async_events_reported;
 };
 
-extern spinlock_t ib_uverbs_idr_lock;
-extern struct idr ib_uverbs_pd_idr;
-extern struct idr ib_uverbs_mr_idr;
-extern struct idr ib_uverbs_mw_idr;
-extern struct idr ib_uverbs_ah_idr;
-extern struct idr ib_uverbs_cq_idr;
-extern struct idr ib_uverbs_qp_idr;
-extern struct idr ib_uverbs_srq_idr;
-extern struct idr ib_uverbs_xrcd_idr;
-extern struct idr ib_uverbs_rule_idr;
-extern struct idr ib_uverbs_wq_idr;
-extern struct idr ib_uverbs_rwq_ind_tbl_idr;
-
-void idr_remove_uobj(struct idr *idp, struct ib_uobject *uobj);
+void idr_remove_uobj(struct ib_uobject *uobj);
 
 struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file,
 					struct ib_device *ib_dev,
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index 7007822..29c621d 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -120,37 +120,36 @@ static void put_uobj_write(struct ib_uobject *uobj)
 	put_uobj(uobj);
 }
 
-static int idr_add_uobj(struct idr *idr, struct ib_uobject *uobj)
+static int idr_add_uobj(struct ib_uobject *uobj)
 {
 	int ret;
 
 	idr_preload(GFP_KERNEL);
-	spin_lock(&ib_uverbs_idr_lock);
+	spin_lock(&uobj->context->ufile->idr_lock);
 
-	ret = idr_alloc(idr, uobj, 0, 0, GFP_NOWAIT);
+	ret = idr_alloc(&uobj->context->ufile->idr, uobj, 0, 0, GFP_NOWAIT);
 	if (ret >= 0)
 		uobj->id = ret;
 
-	spin_unlock(&ib_uverbs_idr_lock);
+	spin_unlock(&uobj->context->ufile->idr_lock);
 	idr_preload_end();
 
 	return ret < 0 ? ret : 0;
 }
 
-void idr_remove_uobj(struct idr *idr, struct ib_uobject *uobj)
+void idr_remove_uobj(struct ib_uobject *uobj)
 {
-	spin_lock(&ib_uverbs_idr_lock);
-	idr_remove(idr, uobj->id);
-	spin_unlock(&ib_uverbs_idr_lock);
+	spin_lock(&uobj->context->ufile->idr_lock);
+	idr_remove(&uobj->context->ufile->idr, uobj->id);
+	spin_unlock(&uobj->context->ufile->idr_lock);
 }
 
-static struct ib_uobject *__idr_get_uobj(struct idr *idr, int id,
-					 struct ib_ucontext *context)
+static struct ib_uobject *__idr_get_uobj(int id, struct ib_ucontext *context)
 {
 	struct ib_uobject *uobj;
 
 	rcu_read_lock();
-	uobj = idr_find(idr, id);
+	uobj = idr_find(&context->ufile->idr, id);
 	if (uobj) {
 		if (uobj->context == context)
 			kref_get(&uobj->ref);
@@ -162,12 +161,12 @@ static struct ib_uobject *__idr_get_uobj(struct idr *idr, int id,
 	return uobj;
 }
 
-static struct ib_uobject *idr_read_uobj(struct idr *idr, int id,
-					struct ib_ucontext *context, int nested)
+static struct ib_uobject *idr_read_uobj(int id, struct ib_ucontext *context,
+					int nested)
 {
 	struct ib_uobject *uobj;
 
-	uobj = __idr_get_uobj(idr, id, context);
+	uobj = __idr_get_uobj(id, context);
 	if (!uobj)
 		return NULL;
 
@@ -183,12 +182,11 @@ static struct ib_uobject *idr_read_uobj(struct idr *idr, int id,
 	return uobj;
 }
 
-static struct ib_uobject *idr_write_uobj(struct idr *idr, int id,
-					 struct ib_ucontext *context)
+static struct ib_uobject *idr_write_uobj(int id, struct ib_ucontext *context)
 {
 	struct ib_uobject *uobj;
 
-	uobj = __idr_get_uobj(idr, id, context);
+	uobj = __idr_get_uobj(id, context);
 	if (!uobj)
 		return NULL;
 
@@ -201,18 +199,18 @@ static struct ib_uobject *idr_write_uobj(struct idr *idr, int id,
 	return uobj;
 }
 
-static void *idr_read_obj(struct idr *idr, int id, struct ib_ucontext *context,
+static void *idr_read_obj(int id, struct ib_ucontext *context,
 			  int nested)
 {
 	struct ib_uobject *uobj;
 
-	uobj = idr_read_uobj(idr, id, context, nested);
+	uobj = idr_read_uobj(id, context, nested);
 	return uobj ? uobj->object : NULL;
 }
 
 static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(&ib_uverbs_pd_idr, pd_handle, context, 0);
+	return idr_read_obj(pd_handle, context, 0);
 }
 
 static void put_pd_read(struct ib_pd *pd)
@@ -222,7 +220,7 @@ static void put_pd_read(struct ib_pd *pd)
 
 static struct ib_cq *idr_read_cq(int cq_handle, struct ib_ucontext *context, int nested)
 {
-	return idr_read_obj(&ib_uverbs_cq_idr, cq_handle, context, nested);
+	return idr_read_obj(cq_handle, context, nested);
 }
 
 static void put_cq_read(struct ib_cq *cq)
@@ -232,7 +230,7 @@ static void put_cq_read(struct ib_cq *cq)
 
 static struct ib_ah *idr_read_ah(int ah_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(&ib_uverbs_ah_idr, ah_handle, context, 0);
+	return idr_read_obj(ah_handle, context, 0);
 }
 
 static void put_ah_read(struct ib_ah *ah)
@@ -242,12 +240,12 @@ static void put_ah_read(struct ib_ah *ah)
 
 static struct ib_qp *idr_read_qp(int qp_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(&ib_uverbs_qp_idr, qp_handle, context, 0);
+	return idr_read_obj(qp_handle, context, 0);
 }
 
 static struct ib_wq *idr_read_wq(int wq_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(&ib_uverbs_wq_idr, wq_handle, context, 0);
+	return idr_read_obj(wq_handle, context, 0);
 }
 
 static void put_wq_read(struct ib_wq *wq)
@@ -258,7 +256,7 @@ static void put_wq_read(struct ib_wq *wq)
 static struct ib_rwq_ind_table *idr_read_rwq_indirection_table(int ind_table_handle,
 							       struct ib_ucontext *context)
 {
-	return idr_read_obj(&ib_uverbs_rwq_ind_tbl_idr, ind_table_handle, context, 0);
+	return idr_read_obj(ind_table_handle, context, 0);
 }
 
 static void put_rwq_indirection_table_read(struct ib_rwq_ind_table *ind_table)
@@ -270,7 +268,7 @@ static struct ib_qp *idr_write_qp(int qp_handle, struct ib_ucontext *context)
 {
 	struct ib_uobject *uobj;
 
-	uobj = idr_write_uobj(&ib_uverbs_qp_idr, qp_handle, context);
+	uobj = idr_write_uobj(qp_handle, context);
 	return uobj ? uobj->object : NULL;
 }
 
@@ -286,7 +284,7 @@ static void put_qp_write(struct ib_qp *qp)
 
 static struct ib_srq *idr_read_srq(int srq_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(&ib_uverbs_srq_idr, srq_handle, context, 0);
+	return idr_read_obj(srq_handle, context, 0);
 }
 
 static void put_srq_read(struct ib_srq *srq)
@@ -297,7 +295,7 @@ static void put_srq_read(struct ib_srq *srq)
 static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context,
 				     struct ib_uobject **uobj)
 {
-	*uobj = idr_read_uobj(&ib_uverbs_xrcd_idr, xrcd_handle, context, 0);
+	*uobj = idr_read_uobj(xrcd_handle, context, 0);
 	return *uobj ? (*uobj)->object : NULL;
 }
 
@@ -305,7 +303,6 @@ static void put_xrcd_read(struct ib_uobject *uobj)
 {
 	put_uobj_read(uobj);
 }
-
 ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file,
 			      struct ib_device *ib_dev,
 			      const char __user *buf,
@@ -342,6 +339,8 @@ ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file,
 	}
 
 	ucontext->device = ib_dev;
+	/* ufile is required when some objects are released */
+	ucontext->ufile = file;
 	INIT_LIST_HEAD(&ucontext->pd_list);
 	INIT_LIST_HEAD(&ucontext->mr_list);
 	INIT_LIST_HEAD(&ucontext->mw_list);
@@ -575,7 +574,7 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
 	atomic_set(&pd->usecnt, 0);
 
 	uobj->object = pd;
-	ret = idr_add_uobj(&ib_uverbs_pd_idr, uobj);
+	ret = idr_add_uobj(uobj);
 	if (ret)
 		goto err_idr;
 
@@ -599,7 +598,7 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_pd_idr, uobj);
+	idr_remove_uobj(uobj);
 
 err_idr:
 	ib_dealloc_pd(pd);
@@ -622,7 +621,7 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(&ib_uverbs_pd_idr, cmd.pd_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.pd_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 	pd = uobj->object;
@@ -640,7 +639,7 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
 	uobj->live = 0;
 	put_uobj_write(uobj);
 
-	idr_remove_uobj(&ib_uverbs_pd_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -816,7 +815,7 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 
 	atomic_set(&obj->refcnt, 0);
 	obj->uobject.object = xrcd;
-	ret = idr_add_uobj(&ib_uverbs_xrcd_idr, &obj->uobject);
+	ret = idr_add_uobj(&obj->uobject);
 	if (ret)
 		goto err_idr;
 
@@ -860,7 +859,7 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 	}
 
 err_insert_xrcd:
-	idr_remove_uobj(&ib_uverbs_xrcd_idr, &obj->uobject);
+	idr_remove_uobj(&obj->uobject);
 
 err_idr:
 	ib_dealloc_xrcd(xrcd);
@@ -894,7 +893,7 @@ ssize_t ib_uverbs_close_xrcd(struct ib_uverbs_file *file,
 		return -EFAULT;
 
 	mutex_lock(&file->device->xrcd_tree_mutex);
-	uobj = idr_write_uobj(&ib_uverbs_xrcd_idr, cmd.xrcd_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.xrcd_handle, file->ucontext);
 	if (!uobj) {
 		ret = -EINVAL;
 		goto out;
@@ -927,7 +926,7 @@ ssize_t ib_uverbs_close_xrcd(struct ib_uverbs_file *file,
 	if (inode && !live)
 		xrcd_table_delete(file->device, inode);
 
-	idr_remove_uobj(&ib_uverbs_xrcd_idr, uobj);
+	idr_remove_uobj(uobj);
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
 	mutex_unlock(&file->mutex);
@@ -1020,7 +1019,7 @@ ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file,
 	atomic_inc(&pd->usecnt);
 
 	uobj->object = mr;
-	ret = idr_add_uobj(&ib_uverbs_mr_idr, uobj);
+	ret = idr_add_uobj(uobj);
 	if (ret)
 		goto err_unreg;
 
@@ -1048,7 +1047,7 @@ ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file,
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_mr_idr, uobj);
+	idr_remove_uobj(uobj);
 
 err_unreg:
 	ib_dereg_mr(mr);
@@ -1093,8 +1092,7 @@ ssize_t ib_uverbs_rereg_mr(struct ib_uverbs_file *file,
 	     (cmd.start & ~PAGE_MASK) != (cmd.hca_va & ~PAGE_MASK)))
 			return -EINVAL;
 
-	uobj = idr_write_uobj(&ib_uverbs_mr_idr, cmd.mr_handle,
-			      file->ucontext);
+	uobj = idr_write_uobj(cmd.mr_handle, file->ucontext);
 
 	if (!uobj)
 		return -EINVAL;
@@ -1163,7 +1161,7 @@ ssize_t ib_uverbs_dereg_mr(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(&ib_uverbs_mr_idr, cmd.mr_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.mr_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 
@@ -1178,7 +1176,7 @@ ssize_t ib_uverbs_dereg_mr(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	idr_remove_uobj(&ib_uverbs_mr_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -1238,7 +1236,7 @@ ssize_t ib_uverbs_alloc_mw(struct ib_uverbs_file *file,
 	atomic_inc(&pd->usecnt);
 
 	uobj->object = mw;
-	ret = idr_add_uobj(&ib_uverbs_mw_idr, uobj);
+	ret = idr_add_uobj(uobj);
 	if (ret)
 		goto err_unalloc;
 
@@ -1265,7 +1263,7 @@ ssize_t ib_uverbs_alloc_mw(struct ib_uverbs_file *file,
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_mw_idr, uobj);
+	idr_remove_uobj(uobj);
 
 err_unalloc:
 	uverbs_dealloc_mw(mw);
@@ -1291,7 +1289,7 @@ ssize_t ib_uverbs_dealloc_mw(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof(cmd)))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(&ib_uverbs_mw_idr, cmd.mw_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.mw_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 
@@ -1306,7 +1304,7 @@ ssize_t ib_uverbs_dealloc_mw(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	idr_remove_uobj(&ib_uverbs_mw_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -1420,7 +1418,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	atomic_set(&cq->usecnt, 0);
 
 	obj->uobject.object = cq;
-	ret = idr_add_uobj(&ib_uverbs_cq_idr, &obj->uobject);
+	ret = idr_add_uobj(&obj->uobject);
 	if (ret)
 		goto err_free;
 
@@ -1446,7 +1444,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	return obj;
 
 err_cb:
-	idr_remove_uobj(&ib_uverbs_cq_idr, &obj->uobject);
+	idr_remove_uobj(&obj->uobject);
 
 err_free:
 	ib_destroy_cq(cq);
@@ -1716,7 +1714,7 @@ ssize_t ib_uverbs_destroy_cq(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(&ib_uverbs_cq_idr, cmd.cq_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.cq_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 	cq      = uobj->object;
@@ -1732,7 +1730,7 @@ ssize_t ib_uverbs_destroy_cq(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	idr_remove_uobj(&ib_uverbs_cq_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -1939,7 +1937,7 @@ static int create_qp(struct ib_uverbs_file *file,
 	qp->uobject = &obj->uevent.uobject;
 
 	obj->uevent.uobject.object = qp;
-	ret = idr_add_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
+	ret = idr_add_uobj(&obj->uevent.uobject);
 	if (ret)
 		goto err_destroy;
 
@@ -1987,7 +1985,7 @@ static int create_qp(struct ib_uverbs_file *file,
 
 	return 0;
 err_cb:
-	idr_remove_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
+	idr_remove_uobj(&obj->uevent.uobject);
 
 err_destroy:
 	ib_destroy_qp(qp);
@@ -2173,7 +2171,7 @@ ssize_t ib_uverbs_open_qp(struct ib_uverbs_file *file,
 	qp->uobject = &obj->uevent.uobject;
 
 	obj->uevent.uobject.object = qp;
-	ret = idr_add_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
+	ret = idr_add_uobj(&obj->uevent.uobject);
 	if (ret)
 		goto err_destroy;
 
@@ -2202,7 +2200,7 @@ ssize_t ib_uverbs_open_qp(struct ib_uverbs_file *file,
 	return in_len;
 
 err_remove:
-	idr_remove_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject);
+	idr_remove_uobj(&obj->uevent.uobject);
 
 err_destroy:
 	ib_destroy_qp(qp);
@@ -2498,7 +2496,7 @@ ssize_t ib_uverbs_destroy_qp(struct ib_uverbs_file *file,
 
 	memset(&resp, 0, sizeof resp);
 
-	uobj = idr_write_uobj(&ib_uverbs_qp_idr, cmd.qp_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.qp_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 	qp  = uobj->object;
@@ -2521,7 +2519,7 @@ ssize_t ib_uverbs_destroy_qp(struct ib_uverbs_file *file,
 	if (obj->uxrcd)
 		atomic_dec(&obj->uxrcd->refcnt);
 
-	idr_remove_uobj(&ib_uverbs_qp_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -2982,7 +2980,7 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
 	ah->uobject  = uobj;
 	uobj->object = ah;
 
-	ret = idr_add_uobj(&ib_uverbs_ah_idr, uobj);
+	ret = idr_add_uobj(uobj);
 	if (ret)
 		goto err_destroy;
 
@@ -3007,7 +3005,7 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_ah_idr, uobj);
+	idr_remove_uobj(uobj);
 
 err_destroy:
 	ib_destroy_ah(ah);
@@ -3032,7 +3030,7 @@ ssize_t ib_uverbs_destroy_ah(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(&ib_uverbs_ah_idr, cmd.ah_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.ah_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 	ah = uobj->object;
@@ -3046,7 +3044,7 @@ ssize_t ib_uverbs_destroy_ah(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	idr_remove_uobj(&ib_uverbs_ah_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -3345,7 +3343,7 @@ int ib_uverbs_ex_create_wq(struct ib_uverbs_file *file,
 	atomic_inc(&cq->usecnt);
 	wq->uobject = &obj->uevent.uobject;
 	obj->uevent.uobject.object = wq;
-	err = idr_add_uobj(&ib_uverbs_wq_idr, &obj->uevent.uobject);
+	err = idr_add_uobj(&obj->uevent.uobject);
 	if (err)
 		goto destroy_wq;
 
@@ -3372,7 +3370,7 @@ int ib_uverbs_ex_create_wq(struct ib_uverbs_file *file,
 	return 0;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_wq_idr, &obj->uevent.uobject);
+	idr_remove_uobj(&obj->uevent.uobject);
 destroy_wq:
 	ib_destroy_wq(wq);
 err_put_cq:
@@ -3421,7 +3419,7 @@ int ib_uverbs_ex_destroy_wq(struct ib_uverbs_file *file,
 		return -EOPNOTSUPP;
 
 	resp.response_length = required_resp_len;
-	uobj = idr_write_uobj(&ib_uverbs_wq_idr, cmd.wq_handle,
+	uobj = idr_write_uobj(cmd.wq_handle,
 			      file->ucontext);
 	if (!uobj)
 		return -EINVAL;
@@ -3436,7 +3434,7 @@ int ib_uverbs_ex_destroy_wq(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	idr_remove_uobj(&ib_uverbs_wq_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -3604,7 +3602,7 @@ int ib_uverbs_ex_create_rwq_ind_table(struct ib_uverbs_file *file,
 	for (i = 0; i < num_wq_handles; i++)
 		atomic_inc(&wqs[i]->usecnt);
 
-	err = idr_add_uobj(&ib_uverbs_rwq_ind_tbl_idr, uobj);
+	err = idr_add_uobj(uobj);
 	if (err)
 		goto destroy_ind_tbl;
 
@@ -3632,7 +3630,7 @@ int ib_uverbs_ex_create_rwq_ind_table(struct ib_uverbs_file *file,
 	return 0;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_rwq_ind_tbl_idr, uobj);
+	idr_remove_uobj(uobj);
 destroy_ind_tbl:
 	ib_destroy_rwq_ind_table(rwq_ind_tbl);
 err_uobj:
@@ -3675,7 +3673,7 @@ int ib_uverbs_ex_destroy_rwq_ind_table(struct ib_uverbs_file *file,
 	if (cmd.comp_mask)
 		return -EOPNOTSUPP;
 
-	uobj = idr_write_uobj(&ib_uverbs_rwq_ind_tbl_idr, cmd.ind_tbl_handle,
+	uobj = idr_write_uobj(cmd.ind_tbl_handle,
 			      file->ucontext);
 	if (!uobj)
 		return -EINVAL;
@@ -3691,7 +3689,7 @@ int ib_uverbs_ex_destroy_rwq_ind_table(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	idr_remove_uobj(&ib_uverbs_rwq_ind_tbl_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -3830,7 +3828,7 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
 	flow_id->uobject = uobj;
 	uobj->object = flow_id;
 
-	err = idr_add_uobj(&ib_uverbs_rule_idr, uobj);
+	err = idr_add_uobj(uobj);
 	if (err)
 		goto destroy_flow;
 
@@ -3855,7 +3853,7 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
 		kfree(kern_flow_attr);
 	return 0;
 err_copy:
-	idr_remove_uobj(&ib_uverbs_rule_idr, uobj);
+	idr_remove_uobj(uobj);
 destroy_flow:
 	ib_destroy_flow(flow_id);
 err_free:
@@ -3890,8 +3888,7 @@ int ib_uverbs_ex_destroy_flow(struct ib_uverbs_file *file,
 	if (cmd.comp_mask)
 		return -EINVAL;
 
-	uobj = idr_write_uobj(&ib_uverbs_rule_idr, cmd.flow_handle,
-			      file->ucontext);
+	uobj = idr_write_uobj(cmd.flow_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 	flow_id = uobj->object;
@@ -3902,7 +3899,7 @@ int ib_uverbs_ex_destroy_flow(struct ib_uverbs_file *file,
 
 	put_uobj_write(uobj);
 
-	idr_remove_uobj(&ib_uverbs_rule_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
@@ -3990,7 +3987,7 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	atomic_set(&srq->usecnt, 0);
 
 	obj->uevent.uobject.object = srq;
-	ret = idr_add_uobj(&ib_uverbs_srq_idr, &obj->uevent.uobject);
+	ret = idr_add_uobj(&obj->uevent.uobject);
 	if (ret)
 		goto err_destroy;
 
@@ -4024,7 +4021,7 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	return 0;
 
 err_copy:
-	idr_remove_uobj(&ib_uverbs_srq_idr, &obj->uevent.uobject);
+	idr_remove_uobj(&obj->uevent.uobject);
 
 err_destroy:
 	ib_destroy_srq(srq);
@@ -4200,7 +4197,7 @@ ssize_t ib_uverbs_destroy_srq(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(&ib_uverbs_srq_idr, cmd.srq_handle, file->ucontext);
+	uobj = idr_write_uobj(cmd.srq_handle, file->ucontext);
 	if (!uobj)
 		return -EINVAL;
 	srq = uobj->object;
@@ -4221,7 +4218,7 @@ ssize_t ib_uverbs_destroy_srq(struct ib_uverbs_file *file,
 		atomic_dec(&us->uxrcd->refcnt);
 	}
 
-	idr_remove_uobj(&ib_uverbs_srq_idr, uobj);
+	idr_remove_uobj(uobj);
 
 	mutex_lock(&file->mutex);
 	list_del(&uobj->list);
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index b3f95d4..8344afd 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -66,19 +66,6 @@ enum {
 
 static struct class *uverbs_class;
 
-DEFINE_SPINLOCK(ib_uverbs_idr_lock);
-DEFINE_IDR(ib_uverbs_pd_idr);
-DEFINE_IDR(ib_uverbs_mr_idr);
-DEFINE_IDR(ib_uverbs_mw_idr);
-DEFINE_IDR(ib_uverbs_ah_idr);
-DEFINE_IDR(ib_uverbs_cq_idr);
-DEFINE_IDR(ib_uverbs_qp_idr);
-DEFINE_IDR(ib_uverbs_srq_idr);
-DEFINE_IDR(ib_uverbs_xrcd_idr);
-DEFINE_IDR(ib_uverbs_rule_idr);
-DEFINE_IDR(ib_uverbs_wq_idr);
-DEFINE_IDR(ib_uverbs_rwq_ind_tbl_idr);
-
 static DEFINE_SPINLOCK(map_lock);
 static DECLARE_BITMAP(dev_map, IB_UVERBS_MAX_DEVICES);
 
@@ -235,7 +222,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 	list_for_each_entry_safe(uobj, tmp, &context->ah_list, list) {
 		struct ib_ah *ah = uobj->object;
 
-		idr_remove_uobj(&ib_uverbs_ah_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_destroy_ah(ah);
 		kfree(uobj);
 	}
@@ -244,7 +231,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 	list_for_each_entry_safe(uobj, tmp, &context->mw_list, list) {
 		struct ib_mw *mw = uobj->object;
 
-		idr_remove_uobj(&ib_uverbs_mw_idr, uobj);
+		idr_remove_uobj(uobj);
 		uverbs_dealloc_mw(mw);
 		kfree(uobj);
 	}
@@ -252,7 +239,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 	list_for_each_entry_safe(uobj, tmp, &context->rule_list, list) {
 		struct ib_flow *flow_id = uobj->object;
 
-		idr_remove_uobj(&ib_uverbs_rule_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_destroy_flow(flow_id);
 		kfree(uobj);
 	}
@@ -262,7 +249,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 		struct ib_uqp_object *uqp =
 			container_of(uobj, struct ib_uqp_object, uevent.uobject);
 
-		idr_remove_uobj(&ib_uverbs_qp_idr, uobj);
+		idr_remove_uobj(uobj);
 		if (qp == qp->real_qp)
 			ib_uverbs_detach_umcast(qp, uqp);
 		ib_destroy_qp(qp);
@@ -274,7 +261,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 		struct ib_rwq_ind_table *rwq_ind_tbl = uobj->object;
 		struct ib_wq **ind_tbl = rwq_ind_tbl->ind_tbl;
 
-		idr_remove_uobj(&ib_uverbs_rwq_ind_tbl_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_destroy_rwq_ind_table(rwq_ind_tbl);
 		kfree(ind_tbl);
 		kfree(uobj);
@@ -285,7 +272,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 		struct ib_uwq_object *uwq =
 			container_of(uobj, struct ib_uwq_object, uevent.uobject);
 
-		idr_remove_uobj(&ib_uverbs_wq_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_destroy_wq(wq);
 		ib_uverbs_release_uevent(file, &uwq->uevent);
 		kfree(uwq);
@@ -296,7 +283,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 		struct ib_uevent_object *uevent =
 			container_of(uobj, struct ib_uevent_object, uobject);
 
-		idr_remove_uobj(&ib_uverbs_srq_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_destroy_srq(srq);
 		ib_uverbs_release_uevent(file, uevent);
 		kfree(uevent);
@@ -308,7 +295,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 		struct ib_ucq_object *ucq =
 			container_of(uobj, struct ib_ucq_object, uobject);
 
-		idr_remove_uobj(&ib_uverbs_cq_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_destroy_cq(cq);
 		ib_uverbs_release_ucq(file, ev_file, ucq);
 		kfree(ucq);
@@ -317,7 +304,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 	list_for_each_entry_safe(uobj, tmp, &context->mr_list, list) {
 		struct ib_mr *mr = uobj->object;
 
-		idr_remove_uobj(&ib_uverbs_mr_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_dereg_mr(mr);
 		kfree(uobj);
 	}
@@ -328,7 +315,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 		struct ib_uxrcd_object *uxrcd =
 			container_of(uobj, struct ib_uxrcd_object, uobject);
 
-		idr_remove_uobj(&ib_uverbs_xrcd_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_uverbs_dealloc_xrcd(file->device, xrcd);
 		kfree(uxrcd);
 	}
@@ -337,7 +324,7 @@ static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
 	list_for_each_entry_safe(uobj, tmp, &context->pd_list, list) {
 		struct ib_pd *pd = uobj->object;
 
-		idr_remove_uobj(&ib_uverbs_pd_idr, uobj);
+		idr_remove_uobj(uobj);
 		ib_dealloc_pd(pd);
 		kfree(uobj);
 	}
@@ -966,6 +953,8 @@ static int ib_uverbs_open(struct inode *inode, struct file *filp)
 	}
 
 	file->device	 = dev;
+	spin_lock_init(&file->idr_lock);
+	idr_init(&file->idr);
 	file->ucontext	 = NULL;
 	file->async_file = NULL;
 	kref_init(&file->ref);
@@ -1003,6 +992,7 @@ static int ib_uverbs_close(struct inode *inode, struct file *filp)
 		file->ucontext = NULL;
 	}
 	mutex_unlock(&file->cleanup_mutex);
+	idr_destroy(&file->idr);
 
 	mutex_lock(&file->device->lists_mutex);
 	if (!file->is_closed) {
@@ -1376,13 +1366,6 @@ static void __exit ib_uverbs_cleanup(void)
 	unregister_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES);
 	if (overflow_maj)
 		unregister_chrdev_region(overflow_maj, IB_UVERBS_MAX_DEVICES);
-	idr_destroy(&ib_uverbs_pd_idr);
-	idr_destroy(&ib_uverbs_mr_idr);
-	idr_destroy(&ib_uverbs_mw_idr);
-	idr_destroy(&ib_uverbs_ah_idr);
-	idr_destroy(&ib_uverbs_cq_idr);
-	idr_destroy(&ib_uverbs_qp_idr);
-	idr_destroy(&ib_uverbs_srq_idr);
 }
 
 module_init(ib_uverbs_init);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index b1ac973..ab3cae4 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1333,6 +1333,7 @@ struct ib_fmr_attr {
 
 struct ib_ucontext {
 	struct ib_device       *device;
+	struct ib_uverbs_file  *ufile;
 	struct list_head	pd_list;
 	struct list_head	mr_list;
 	struct list_head	mw_list;
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/7] IB/core: Refactor idr to be per uverbs_file

[Jason Gunthorpe]

On Wed, Feb 01, 2017 at 02:38:59PM +0200, Matan Barak wrote:
>  static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
>  {
> -	return idr_read_obj(&ib_uverbs_pd_idr, pd_handle, context, 0);
> +	return idr_read_obj(pd_handle, context, 0);
>  }

As a note: At this point in the series we loose the type checking on
the IDR handles. Eg at this moment idr_read_obj() could return
something that is not a PD and resulting ugliness..

Since that is ultimately fixed as we go through the series I'm not
sure it is worth addressing.

>  
>  	ucontext->device = ib_dev;
> +	/* ufile is required when some objects are released */
> +	ucontext->ufile = file;

Hurm. This reads weirdly, but it is OK..

Overall this is starting to really look weird. ucontext has a pointer
to ufile and ufile has a pointer to ucontext. That tells me they
should be in the same allocation. To make this saner you'd have to
embed ib_ucontext into ib_uverbs_file and use a ->priv for the driver
data instead of sticking at the end of the ib_ucontext allocation.

Probably too big for this patch series but keep it in mind...

Otherwise the patch, in context, seems fine to me..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 1/7] IB/core: Refactor idr to be per uverbs_file

[Matan Barak]

On Fri, Feb 10, 2017 at 7:53 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Feb 01, 2017 at 02:38:59PM +0200, Matan Barak wrote:
>>  static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
>>  {
>> -     return idr_read_obj(&ib_uverbs_pd_idr, pd_handle, context, 0);
>> +     return idr_read_obj(pd_handle, context, 0);
>>  }
>
> As a note: At this point in the series we loose the type checking on
> the IDR handles. Eg at this moment idr_read_obj() could return
> something that is not a PD and resulting ugliness..
>
> Since that is ultimately fixed as we go through the series I'm not
> sure it is worth addressing.
>

Agreed

>>
>>       ucontext->device = ib_dev;
>> +     /* ufile is required when some objects are released */
>> +     ucontext->ufile = file;
>
> Hurm. This reads weirdly, but it is OK..
>
> Overall this is starting to really look weird. ucontext has a pointer
> to ufile and ufile has a pointer to ucontext. That tells me they
> should be in the same allocation. To make this saner you'd have to
> embed ib_ucontext into ib_uverbs_file and use a ->priv for the driver
> data instead of sticking at the end of the ib_ucontext allocation.
>
> Probably too big for this patch series but keep it in mind...
>

Yeah, that could come as a later cleanup.

> Otherwise the patch, in context, seems fine to me..
>

May I add your Reviewed-by to later submissions of this patch?

> Jason

Thanks for reviewing.

Matan

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 2/7] IB/core: Add support for idr types

[Matan Barak]

The new ioctl infrastructure supports driver specific objects.
Each such object type has a hot unplug function, allocation size and
an order of destruction.

When a ucontext is created, a new list is created in this ib_ucontext.
This list contains all objects created under this ib_ucontext.
When a ib_ucontext is destroyed, we traverse this list several time
destroying the various objects by the order mentioned in the object
type description. If few object types have the same destruction order,
they are destroyed in an order opposite to their creation.

Adding an object is done in two parts.
First, an object is allocated and added to idr tree. Then, the
command's handlers (in downstream patches) could work on this object
and fill in its required details.
After a successful command, the commit part is called and the user
objects become ucontext visible. If the handler failed, alloc_abort
should be called.

Removing an uboject is done by calling lookup_get with the write flag
and finalizing it with destroy_commit.

We should make sure idr (per-uverbs-file) and list (per-ucontext) could
be accessed concurrently without corrupting them.

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/Makefile    |   3 +-
 drivers/infiniband/core/rdma_core.c | 277 ++++++++++++++++++++++++++++++++++++
 drivers/infiniband/core/rdma_core.h |  55 +++++++
 include/rdma/ib_verbs.h             |   7 +
 include/rdma/uverbs_types.h         | 104 ++++++++++++++
 5 files changed, 445 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/core/rdma_core.c
 create mode 100644 drivers/infiniband/core/rdma_core.h
 create mode 100644 include/rdma/uverbs_types.h

diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index edaae9f..1819623 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -28,4 +28,5 @@ ib_umad-y :=			user_mad.o
 
 ib_ucm-y :=			ucm.o
 
-ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_marshall.o
+ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
+				rdma_core.o
diff --git a/drivers/infiniband/core/rdma_core.c b/drivers/infiniband/core/rdma_core.c
new file mode 100644
index 0000000..7ce4d67
--- /dev/null
+++ b/drivers/infiniband/core/rdma_core.c
@@ -0,0 +1,277 @@
+/*
+ * Copyright (c) 2016, Mellanox Technologies inc.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/file.h>
+#include <linux/anon_inodes.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/uverbs_types.h>
+#include "uverbs.h"
+#include "rdma_core.h"
+
+static void uverbs_uobject_put_ref(struct kref *ref)
+{
+	struct ib_uobject *uobj =
+		container_of(ref, struct ib_uobject, ref);
+
+	uobj->type->ops->release(uobj);
+}
+
+void uverbs_uobject_put(struct ib_uobject *uobj)
+{
+	kref_put(&uobj->ref, uverbs_uobject_put_ref);
+}
+
+static int uverbs_lock_object(struct ib_uobject *uobj, bool write)
+{
+	if (!write)
+		return down_read_trylock(&uobj->currently_used) == 1 ? 0 :
+			-EBUSY;
+
+	/* lock is either WRITE or DESTROY - should be exclusive */
+	return down_write_trylock(&uobj->currently_used) == 1 ? 0 : -EBUSY;
+}
+
+static void init_uobj(struct ib_uobject *uobj, struct ib_ucontext *context,
+		      const struct uverbs_obj_type *type)
+{
+	/*
+	 * user_handle should be filled by the handler,
+	 * The object is added to the list in the commit stage.
+	 */
+	init_rwsem(&uobj->currently_used);
+	uobj->context     = context;
+	uobj->type	  = type;
+	kref_init(&uobj->ref);
+}
+
+static int idr_add_uobj(struct ib_uobject *uobj)
+{
+	int ret;
+
+	idr_preload(GFP_KERNEL);
+	spin_lock(&uobj->context->ufile->idr_lock);
+
+	/*
+	 * We start with allocating an idr pointing to NULL. This represents an
+	 * object which isn't initialized yet. We'll replace it later on with
+	 * the real object once we commit.
+	 */
+	ret = idr_alloc(&uobj->context->ufile->idr, NULL, 0,
+			min_t(unsigned long, U32_MAX - 1, INT_MAX), GFP_NOWAIT);
+	if (ret >= 0)
+		uobj->id = ret;
+
+	spin_unlock(&uobj->context->ufile->idr_lock);
+	idr_preload_end();
+
+	return ret < 0 ? ret : 0;
+}
+
+static void uverbs_idr_remove_uobj(struct ib_uobject *uobj)
+{
+	spin_lock(&uobj->context->ufile->idr_lock);
+	idr_remove(&uobj->context->ufile->idr, uobj->id);
+	spin_unlock(&uobj->context->ufile->idr_lock);
+}
+
+/* Returns the ib_uobject or an error. The caller should check for IS_ERR. */
+static struct ib_uobject *lookup_get_idr_uobject(const struct uverbs_obj_type *type,
+						 struct ib_ucontext *ucontext,
+						 int id, bool write)
+{
+	struct ib_uobject *uobj;
+	int ret;
+
+	rcu_read_lock();
+	/* object won't be released as we're protected in rcu */
+	uobj = idr_find(&ucontext->ufile->idr, id);
+	if (!uobj) {
+		uobj = ERR_PTR(-ENOENT);
+		goto free;
+	}
+
+	if (uobj->type != type) {
+		uobj = ERR_PTR(-EINVAL);
+		goto free;
+	}
+
+	ret = uverbs_lock_object(uobj, write);
+	if (ret)
+		uobj = ERR_PTR(ret);
+free:
+	rcu_read_unlock();
+	return uobj;
+}
+
+static struct ib_uobject *alloc_begin_idr_uobject(const struct uverbs_obj_type *type,
+						  struct ib_ucontext *ucontext)
+{
+	int ret;
+	const struct uverbs_obj_idr_type *idr_type =
+		container_of(type, struct uverbs_obj_idr_type, type);
+	struct ib_uobject *uobj = kmalloc(idr_type->obj_size, GFP_KERNEL);
+
+	if (!uobj)
+		return ERR_PTR(-ENOMEM);
+
+	init_uobj(uobj, ucontext, type);
+	ret = idr_add_uobj(uobj);
+	if (ret) {
+		kfree(uobj);
+		return ERR_PTR(ret);
+	}
+
+	return uobj;
+}
+
+static void uverbs_uobject_add(struct ib_uobject *uobject)
+{
+	mutex_lock(&uobject->context->lock);
+	list_add(&uobject->list, &uobject->context->uobjects);
+	mutex_unlock(&uobject->context->lock);
+}
+
+static void alloc_commit_idr_uobject(struct ib_uobject *uobj)
+{
+	uverbs_uobject_add(uobj);
+	spin_lock(&uobj->context->ufile->idr_lock);
+	/*
+	 * We already allocated this IDR with a NULL object, so
+	 * this shouldn't fail.
+	 */
+	WARN_ON(idr_replace(&uobj->context->ufile->idr,
+			    uobj, uobj->id));
+	spin_unlock(&uobj->context->ufile->idr_lock);
+}
+
+static void _put_uobj_ref(struct kref *ref)
+{
+	kfree(container_of(ref, struct ib_uobject, ref));
+}
+
+static void alloc_abort_idr_uobject(struct ib_uobject *uobj)
+{
+	uverbs_idr_remove_uobj(uobj);
+	/*
+	 * we don't need kfree_rcu here, as the uobject wasn't exposed to any
+	 * other verb.
+	 */
+	kref_put(&uobj->ref, _put_uobj_ref);
+}
+
+static void lookup_put_idr_uobject(struct ib_uobject *uobj, bool write)
+{
+	if (write)
+		up_write(&uobj->currently_used);
+	else
+		up_read(&uobj->currently_used);
+}
+
+static void destroy_commit_idr_uobject(struct ib_uobject *uobj)
+{
+	uverbs_idr_remove_uobj(uobj);
+	mutex_lock(&uobj->context->lock);
+	list_del(&uobj->list);
+	mutex_unlock(&uobj->context->lock);
+	uverbs_uobject_put(uobj);
+}
+
+static void hot_unplug_idr_uobject(struct ib_uobject *uobj, bool device_removed)
+{
+	const struct uverbs_obj_idr_type *idr_type =
+		container_of(uobj->type, struct uverbs_obj_idr_type, type);
+
+	idr_type->hot_unplug(uobj);
+	uverbs_idr_remove_uobj(uobj);
+	uverbs_uobject_put(uobj);
+}
+
+static void release_idr_uobject(struct ib_uobject *uobj)
+{
+	/*
+	 * When we destroy an object, we first just lock it for WRITE and
+	 * actually DESTROY it in the finalize stage. So, the problematic
+	 * scenario is when we just started the finalize stage of the
+	 * destruction (nothing was executed yet). Now, the other thread
+	 * fetched the object for READ access, but it didn't lock it yet.
+	 * The DESTROY thread continues and starts destroying the object.
+	 * When the other thread continue - without the RCU, it would
+	 * access freed memory. However, the rcu_read_lock delays the free
+	 * until the rcu_read_lock of the READ operation quits. Since the
+	 * write lock of the object is still taken by the DESTROY flow, the
+	 * READ operation will get -EBUSY and it'll just bail out.
+	 */
+	kfree_rcu(uobj, rcu);
+}
+
+const struct uverbs_obj_type_ops uverbs_idr_ops = {
+	.alloc_begin = alloc_begin_idr_uobject,
+	.lookup_get = lookup_get_idr_uobject,
+	.alloc_commit = alloc_commit_idr_uobject,
+	.alloc_abort = alloc_abort_idr_uobject,
+	.lookup_put = lookup_put_idr_uobject,
+	.destroy_commit = destroy_commit_idr_uobject,
+	.hot_unplug = hot_unplug_idr_uobject,
+	.release = release_idr_uobject,
+};
+
+void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
+{
+	unsigned int cur_order = 0;
+
+	while (!list_empty(&ucontext->uobjects)) {
+		struct ib_uobject *obj, *next_obj;
+		unsigned int next_order = UINT_MAX;
+
+		/*
+		 * This shouldn't run while executing other commands on this
+		 * context, thus no lock is required.
+		 */
+		list_for_each_entry_safe(obj, next_obj, &ucontext->uobjects,
+					 list)
+			if (obj->type->destroy_order == cur_order) {
+				list_del(&obj->list);
+				obj->type->ops->hot_unplug(obj, device_removed);
+			} else {
+				next_order = min(next_order,
+						 obj->type->destroy_order);
+			}
+		cur_order = next_order;
+	}
+}
+
+void uverbs_initialize_ucontext(struct ib_ucontext *ucontext)
+{
+	mutex_init(&ucontext->lock);
+	INIT_LIST_HEAD(&ucontext->uobjects);
+}
+
diff --git a/drivers/infiniband/core/rdma_core.h b/drivers/infiniband/core/rdma_core.h
new file mode 100644
index 0000000..ab665a6
--- /dev/null
+++ b/drivers/infiniband/core/rdma_core.h
@@ -0,0 +1,55 @@
+/*
+ * Copyright (c) 2005 Topspin Communications.  All rights reserved.
+ * Copyright (c) 2005, 2006 Cisco Systems.  All rights reserved.
+ * Copyright (c) 2005-2017 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2005 Voltaire, Inc. All rights reserved.
+ * Copyright (c) 2005 PathScale, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef RDMA_CORE_H
+#define RDMA_CORE_H
+
+#include <linux/idr.h>
+#include <rdma/uverbs_types.h>
+#include <rdma/ib_verbs.h>
+#include <linux/mutex.h>
+
+/*
+ * These functions initialize the context and cleanups its uobjects.
+ * The context has a list of objects which is protected by a mutex
+ * on the context. initialize_ucontext should be called when we create
+ * a context.
+ * cleanup_ucontext removes all uobjects from the context and puts them.
+ */
+void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed);
+void uverbs_initialize_ucontext(struct ib_ucontext *ucontext);
+
+#endif /* RDMA_CORE_H */
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index ab3cae4..4a2a0fc 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1347,6 +1347,10 @@ struct ib_ucontext {
 	struct list_head	rwq_ind_tbl_list;
 	int			closing;
 
+	/* locking the uobjects_list */
+	struct mutex		lock;
+	struct list_head	uobjects;
+
 	struct pid             *tgid;
 #ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
 	struct rb_root      umem_tree;
@@ -1374,8 +1378,11 @@ struct ib_uobject {
 	int			id;		/* index into kernel idr */
 	struct kref		ref;
 	struct rw_semaphore	mutex;		/* protects .live */
+	struct rw_semaphore	currently_used;	/* protects exclusive access */
 	struct rcu_head		rcu;		/* kfree_rcu() overhead */
 	int			live;
+
+	const struct uverbs_obj_type *type;
 };
 
 struct ib_udata {
diff --git a/include/rdma/uverbs_types.h b/include/rdma/uverbs_types.h
new file mode 100644
index 0000000..bb263f0
--- /dev/null
+++ b/include/rdma/uverbs_types.h
@@ -0,0 +1,104 @@
+/*
+ * Copyright (c) 2017, Mellanox Technologies inc.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef _UVERBS_TYPES_
+#define _UVERBS_TYPES_
+
+#include <linux/kernel.h>
+#include <rdma/ib_verbs.h>
+
+struct uverbs_obj_type;
+
+struct uverbs_obj_type_ops {
+	/*
+	 * Get an ib_uobject that corresponds to the given id from ucontext,
+	 * These functions could create or destroy objects if required.
+	 * The action will be finalized only when commit, abort or put fops are
+	 * called.
+	 * The flow of the different actions is:
+	 * [alloc]:	 Starts with alloc_begin. The handlers logic is than
+	 *		 executed. If the handler is successful, alloc_commit
+	 *		 is called and the object is inserted to the repository.
+	 *		 Otherwise, alloc_abort is called and the object is
+	 *		 destroyed.
+	 * [lookup]:	 Starts with lookup_get which fetches and locks the
+	 *		 object. After the handler finished using the object, it
+	 *		 needs to call lookup_put to unlock it. The write flag
+	 *		 indicates if the object is locked for exclusive access.
+	 * [destroy]:	 Starts with lookup_get with write flag set. This locks
+	 *		 the object for exclusive access. If the handler code
+	 *		 completed successfully, destroy_commit is called and
+	 *		 the ib_uobject is removed from the context's uobjects
+	 *		 repository and put. Otherwise, lookup_put should be
+	 *		 called.
+	 * [hot_unplug]: Used when the context is destroyed (process
+	 *		 termination, reset flow).
+	 * [release]:    Releases the memory of the ib_uobject. This is called
+	 *		 when the last reference is put. We favored a callback
+	 *		 here as this might require tricks like kfree_rcu.
+	 *		 This shouldn't be called explicitly by the user as it's
+	 *		 used by uverbs_uobject_put.
+	 */
+	struct ib_uobject *(*alloc_begin)(const struct uverbs_obj_type *type,
+					  struct ib_ucontext *ucontext);
+	void (*alloc_commit)(struct ib_uobject *uobj);
+	void (*alloc_abort)(struct ib_uobject *uobj);
+
+	struct ib_uobject *(*lookup_get)(const struct uverbs_obj_type *type,
+					 struct ib_ucontext *ucontext, int id,
+					 bool write);
+	void (*lookup_put)(struct ib_uobject *uobj, bool write);
+	void (*destroy_commit)(struct ib_uobject *uobj);
+
+	void (*hot_unplug)(struct ib_uobject *uobj, bool device_removed);
+
+	void (*release)(struct ib_uobject *uobj);
+};
+
+struct uverbs_obj_type {
+	const struct uverbs_obj_type_ops * const ops;
+	unsigned int destroy_order;
+};
+
+struct uverbs_obj_idr_type {
+	/*
+	 * In idr based objects, uverbs_obj_type_ops points to a generic
+	 * idr operations. In order to specialize the underlying types (e.g. CQ,
+	 * QPs, etc.), we add obj_size and hot_unplug specific callbacks here.
+	 */
+	struct uverbs_obj_type  type;
+	size_t			obj_size;
+	/* The hot_unplug in ops initialized by idr, calls this callback */
+	void (*hot_unplug)(struct ib_uobject *uobj);
+};
+
+#endif
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 2/7] IB/core: Add support for idr types

[Jason Gunthorpe]

On Wed, Feb 01, 2017 at 02:39:00PM +0200, Matan Barak wrote:

> +static int uverbs_lock_object(struct ib_uobject *uobj, bool write)
> +{
> +	if (!write)
> +		return down_read_trylock(&uobj->currently_used) == 1 ? 0 :
> +			-EBUSY;
> +
> +	/* lock is either WRITE or DESTROY - should be exclusive */
> +	return down_write_trylock(&uobj->currently_used) == 1 ? 0 : -EBUSY;

Is currently_used ever used as an actual blocking rwsem? Looks like
no to me right now, is that the long term plan?

It actually seems to be important that this not be blocking, so it
might be better to code this a simple atomic.

And lets call it uverbs_try_lock_object() for clarity.

> +static void uverbs_uobject_put_ref(struct kref *ref)
> +{
> +	struct ib_uobject *uobj =
> +		container_of(ref, struct ib_uobject, ref);
> +
> +	uobj->type->ops->release(uobj);
> +}

This sort of approach starts to become very dangerous when you
contemplate having 'release' be a function in a module. Is that the
case?

But, broadly speaking, I expect to see kfree() in a release function,
and from what I can tell the only issue here is that the IDR sometimes
needs kfree_rcu.

So lets drop the function pointer and just add a 1 bit flag:

if (uobj->needs_rcu)
   kfree_rcu(uobj);
else
   kfree(uobj);

This is more clear, don't overdesign things at this point because it
becomes hard to see if down the road things are done wrong (eg putting
release in a different module)

> +static void init_uobj(struct ib_uobject *uobj, struct ib_ucontext *context,
> +		      const struct uverbs_obj_type *type)
> +{
> +	/*
> +	 * user_handle should be filled by the handler,
> +	 * The object is added to the list in the commit stage.
> +	 */
> +	init_rwsem(&uobj->currently_used);
> +	uobj->context     = context;
> +	uobj->type	  = type;

.. and can you please not add the bogus whitespace in new commits
please? That is really not the typical kernel style and makes
everything hard to maintain and read.

> +	kref_init(&uobj->ref);
> +}

Lets call this alloc_uobj, and just use kzalloc(type->obj_size) to do
it.

Due to the kref and the above comment about release it is not a good
idea to have a type specific allocation, we always want kalloc'd
memory here.

> +static void uverbs_idr_remove_uobj(struct ib_uobject *uobj)
> +{
> +	spin_lock(&uobj->context->ufile->idr_lock);
> +	idr_remove(&uobj->context->ufile->idr, uobj->id);
> +	spin_unlock(&uobj->context->ufile->idr_lock);
> +}

Add a clarifying comment

/* The caller must uverbs_uobject_put() uobj */

> +	rcu_read_lock();
> +	/* object won't be released as we're protected in rcu */
> +	uobj = idr_find(&ucontext->ufile->idr, id);

> +	if (!uobj) {
> +		uobj = ERR_PTR(-ENOENT);
> +		goto free;
> +	}
> +
> +	if (uobj->type != type) {
> +		uobj = ERR_PTR(-EINVAL);
> +		goto free;
> +	}
> +
> +	ret = uverbs_lock_object(uobj, write);

Hum. This RCU is a bit exciting.. So this relies on the write lock
being held whenever uverbs_idr_remove is called. Can you add a
LOCKDEP style of of assertion to uverbs_idr_remove to prove that?

> +static struct ib_uobject *alloc_begin_idr_uobject(const struct uverbs_obj_type *type,
> +						  struct ib_ucontext *ucontext)
> +{
> +	int ret;
> +	const struct uverbs_obj_idr_type *idr_type =
> +		container_of(type, struct uverbs_obj_idr_type, type);
> +	struct ib_uobject *uobj = kmalloc(idr_type->obj_size, GFP_KERNEL);

> +
> +	if (!uobj)
> +		return ERR_PTR(-ENOMEM);

This would be the thing to move to alloc_uobj. Ditto for fd.

> +	init_uobj(uobj, ucontext, type);
> +	ret = idr_add_uobj(uobj);
> +	if (ret) {
> +		kfree(uobj);

This should be a uverbs_uobject_put()

> +static void uverbs_uobject_add(struct ib_uobject *uobject)
> +{
> +	mutex_lock(&uobject->context->lock);
> +	list_add(&uobject->list, &uobject->context->uobjects);
> +	mutex_unlock(&uobject->context->lock);
> +}

I'd open code this into alloc_commit_idr_uobject. The only place the
list should be updated is directly after it has been placed in the
IDR, it is confusing to see it in a function as though it can be
called from someplace else

> +static void _put_uobj_ref(struct kref *ref)
> +{
> +	kfree(container_of(ref, struct ib_uobject, ref));
> +}
> +
> +static void alloc_abort_idr_uobject(struct ib_uobject *uobj)
> +{
> +	uverbs_idr_remove_uobj(uobj);
> +	/*
> +	 * we don't need kfree_rcu here, as the uobject wasn't exposed to any
> +	 * other verb.
> +	 */
> +	kref_put(&uobj->ref, _put_uobj_ref);
> +}

Once needs_rcu is added then this ugly stuff goes away. Set needs_rcu
only when the uboject has been added to the IDR.

> +static void destroy_commit_idr_uobject(struct ib_uobject *uobj)

This is a confusing name.. See below..

> +{
> +	uverbs_idr_remove_uobj(uobj);
> +	mutex_lock(&uobj->context->lock);
> +	list_del(&uobj->list);
> +	mutex_unlock(&uobj->context->lock);
> +	uverbs_uobject_put(uobj);
> +}

And this flow is weird, hot_unplug calls an idr_type op, but this does
not? Why?

> +void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
> +{
> +	unsigned int cur_order = 0;
> +
> +	while (!list_empty(&ucontext->uobjects)) {
> +		struct ib_uobject *obj, *next_obj;
> +		unsigned int next_order = UINT_MAX;
> +
> +		/*
> +		 * This shouldn't run while executing other commands on this
> +		 * context, thus no lock is required.
> +		 */
> +		list_for_each_entry_safe(obj, next_obj, &ucontext->uobjects,
> +					 list)
> +			if (obj->type->destroy_order == cur_order) {
> +				list_del(&obj->list);
> +				obj->type->ops->hot_unplug(obj, device_removed);

Let's add a safety:

   WARN_ON(uverbs_lock_object(obj, true));

It would be really nice to see a simpler 'built-in' scheme to
guarentee that WARN_ON never hits. I expect this is relying on the
external SRCU system to make it work out..

For instance since we now have lookup_get/lookup_put could we
ultimately move the SRCU into those functions?

Also, the call to hot_unplug has to be checked

> +	/* locking the uobjects_list */
> +	struct mutex		lock;

Prefer uobjects_lock for clarity

> @@ -1374,8 +1378,11 @@ struct ib_uobject {
>  	int			id;		/* index into kernel idr */

> +
> +struct uverbs_obj_type_ops {
> +	/*
> +	 * Get an ib_uobject that corresponds to the given id from ucontext,
> +	 * These functions could create or destroy objects if required.
> +	 * The action will be finalized only when commit, abort or put fops are
> +	 * called.
> +	 * The flow of the different actions is:
> +	 * [alloc]:	 Starts with alloc_begin. The handlers logic is than
> +	 *		 executed. If the handler is successful, alloc_commit
> +	 *		 is called and the object is inserted to the repository.
> +	 *		 Otherwise, alloc_abort is called and the object is
> +	 *		 destroyed.

Add:

 Once alloc_commit completes the object is visible to other threads
 and userspace.

> +	 * [lookup]:	 Starts with lookup_get which fetches and locks the
> +	 *		 object. After the handler finished using the object, it
> +	 *		 needs to call lookup_put to unlock it. The write flag
> +	 *		 indicates if the object is locked for exclusive access.
> +	 * [destroy]:	 Starts with lookup_get with write flag set. This locks
> +	 *		 the object for exclusive access. If the handler code
> +	 *		 completed successfully, destroy_commit is called and
> +	 *		 the ib_uobject is removed from the context's uobjects
> +	 *		 repository and put. Otherwise, lookup_put should be
> +	 *		 called.

Lets call this 'remove' please.

And add:

 Once remove succeeds new krefs to the object cannot be acquired by
 other threads or userspace and the hardware driver is removed from
 the object. Other krefs on the object may still exist.

> +	 * [hot_unplug]: Used when the context is destroyed (process
> +	 *		 termination, reset flow).

I don't think we need a dedicated entry point. I think you should add
an enum argument to remove:

enum rdma_remove_reason {
 RDMA_REMOVE_DESTROY, // Userspace requested uobject deletion
 RDMA_REMOVE_CLOSE,   // Context deletion. Call cannot fail.
 RDMA_REMOVE_DRIVER_REMOVE, // Driver is being hot-unplugged. Call cannot fail.
};

This gives the drivers a more information to the right thing to do.

> +	 * [release]:    Releases the memory of the ib_uobject. This is called
> +	 *		 when the last reference is put. We favored a callback
> +	 *		 here as this might require tricks like kfree_rcu.
> +	 *		 This shouldn't be called explicitly by the user as it's
> +	 *		 used by uverbs_uobject_put.

Delete as discussed


> +	void (*destroy_commit)(struct ib_uobject *uobj);

Use:
   /* Must be called with the write lock held. If successful uboj is
      invalid on return. On failure uobject is left completely unchanged */
   int __must_check (*remove)(struct ib_uobject *uobj,enum rdma_remove_reason why);

> +struct uverbs_obj_idr_type {
> +	/*
> +	 * In idr based objects, uverbs_obj_type_ops points to a generic
> +	 * idr operations. In order to specialize the underlying types (e.g. CQ,
> +	 * QPs, etc.), we add obj_size and hot_unplug specific callbacks here.
> +	 */
> +	struct uverbs_obj_type  type;
> +	size_t			obj_size;

Move to type, do not support no kalloc configurations

> +	/* The hot_unplug in ops initialized by idr, calls this callback */
> +	void (*hot_unplug)(struct ib_uobject *uobj);

Use:

/* Free driver resources from the uobject, make the driver uncallable,
   and move the uobject to the detached state. On failure the uboject
   is left completely unchanged. */
int __must_check (*destroy)(struct ib_uobject *uobj,enum rdma_remove_reason why);

And probably add some commentary what objects support a detached
state. Any object that cannot be detached can only exist in the IDR
and cannot have a kref taken. Perhaps we should enforce that directly
for clarity.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 2/7] IB/core: Add support for idr types

[Matan Barak]

On Fri, Feb 10, 2017 at 9:56 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Feb 01, 2017 at 02:39:00PM +0200, Matan Barak wrote:
>
>> +static int uverbs_lock_object(struct ib_uobject *uobj, bool write)
>> +{
>> +     if (!write)
>> +             return down_read_trylock(&uobj->currently_used) == 1 ? 0 :
>> +                     -EBUSY;
>> +
>> +     /* lock is either WRITE or DESTROY - should be exclusive */
>> +     return down_write_trylock(&uobj->currently_used) == 1 ? 0 : -EBUSY;
>
> Is currently_used ever used as an actual blocking rwsem? Looks like
> no to me right now, is that the long term plan?
>

Yeah, it's never used for blocking and that's the long term plan.

> It actually seems to be important that this not be blocking, so it
> might be better to code this a simple atomic.
>

Actually, in the first version it was coded as an atomic and we decided
to change it to rwsem after some comments. Anyway, no problem
changing it back to the original version (see
http://www.spinics.net/lists/linux-rdma/msg38346.html).

> And lets call it uverbs_try_lock_object() for clarity.
>

sure

>> +static void uverbs_uobject_put_ref(struct kref *ref)
>> +{
>> +     struct ib_uobject *uobj =
>> +             container_of(ref, struct ib_uobject, ref);
>> +
>> +     uobj->type->ops->release(uobj);
>> +}
>
> This sort of approach starts to become very dangerous when you
> contemplate having 'release' be a function in a module. Is that the
> case?
>

Well, I expect these core meta-types (e.g. idr, fd) to be declared in ib_uverbs
or ib_core.

> But, broadly speaking, I expect to see kfree() in a release function,
> and from what I can tell the only issue here is that the IDR sometimes
> needs kfree_rcu.
>
> So lets drop the function pointer and just add a 1 bit flag:
>
> if (uobj->needs_rcu)
>    kfree_rcu(uobj);
> else
>    kfree(uobj);
>
> This is more clear, don't overdesign things at this point because it
> becomes hard to see if down the road things are done wrong (eg putting
> release in a different module)
>

Ok, we could use that instead of an explicit release function.

>> +static void init_uobj(struct ib_uobject *uobj, struct ib_ucontext *context,
>> +                   const struct uverbs_obj_type *type)
>> +{
>> +     /*
>> +      * user_handle should be filled by the handler,
>> +      * The object is added to the list in the commit stage.
>> +      */
>> +     init_rwsem(&uobj->currently_used);
>> +     uobj->context     = context;
>> +     uobj->type        = type;
>
> .. and can you please not add the bogus whitespace in new commits
> please? That is really not the typical kernel style and makes
> everything hard to maintain and read.
>

Which bogus whitespace? This is from the kernel coding style doc:
The preferred style for long (multi-line) comments is:

.. code-block:: c

        /*
         * This is the preferred style for multi-line
         * comments in the Linux kernel source code.
         * Please use it consistently.
         *
         * Description:  A column of asterisks on the left side,
         * with beginning and ending almost-blank lines.
         */

Only the net subsystem starts the comment from the first line.

>> +     kref_init(&uobj->ref);
>> +}
>
> Lets call this alloc_uobj, and just use kzalloc(type->obj_size) to do
> it.
>
> Due to the kref and the above comment about release it is not a good
> idea to have a type specific allocation, we always want kalloc'd
> memory here.
>

I can move that. I don't think we really need kzalloc as all fields are
initialized anyway.

>> +static void uverbs_idr_remove_uobj(struct ib_uobject *uobj)
>> +{
>> +     spin_lock(&uobj->context->ufile->idr_lock);
>> +     idr_remove(&uobj->context->ufile->idr, uobj->id);
>> +     spin_unlock(&uobj->context->ufile->idr_lock);
>> +}
>
> Add a clarifying comment
>
> /* The caller must uverbs_uobject_put() uobj */
>

It could actually call kfree directly if the object is guaranteed not
to be used, but
this is a micro optimization that we probably shouldn't really care about.
I prefer to put needs_rcu on the type and not on each uobject.

>> +     rcu_read_lock();
>> +     /* object won't be released as we're protected in rcu */
>> +     uobj = idr_find(&ucontext->ufile->idr, id);
>
>> +     if (!uobj) {
>> +             uobj = ERR_PTR(-ENOENT);
>> +             goto free;
>> +     }
>> +
>> +     if (uobj->type != type) {
>> +             uobj = ERR_PTR(-EINVAL);
>> +             goto free;
>> +     }
>> +
>> +     ret = uverbs_lock_object(uobj, write);
>
> Hum. This RCU is a bit exciting.. So this relies on the write lock
> being held whenever uverbs_idr_remove is called. Can you add a
> LOCKDEP style of of assertion to uverbs_idr_remove to prove that?
>

if we remove the lock and use atomics instead, there's nothing to do
lockdep on, right?

>> +static struct ib_uobject *alloc_begin_idr_uobject(const struct uverbs_obj_type *type,
>> +                                               struct ib_ucontext *ucontext)
>> +{
>> +     int ret;
>> +     const struct uverbs_obj_idr_type *idr_type =
>> +             container_of(type, struct uverbs_obj_idr_type, type);
>> +     struct ib_uobject *uobj = kmalloc(idr_type->obj_size, GFP_KERNEL);
>
>> +
>> +     if (!uobj)
>> +             return ERR_PTR(-ENOMEM);
>
> This would be the thing to move to alloc_uobj. Ditto for fd.
>

Ok

>> +     init_uobj(uobj, ucontext, type);
>> +     ret = idr_add_uobj(uobj);
>> +     if (ret) {
>> +             kfree(uobj);
>
> This should be a uverbs_uobject_put()
>

It'll just postpone this to rcu, but I guess this doesn't really matter.

>> +static void uverbs_uobject_add(struct ib_uobject *uobject)
>> +{
>> +     mutex_lock(&uobject->context->lock);
>> +     list_add(&uobject->list, &uobject->context->uobjects);
>> +     mutex_unlock(&uobject->context->lock);
>> +}
>
> I'd open code this into alloc_commit_idr_uobject. The only place the
> list should be updated is directly after it has been placed in the
> IDR, it is confusing to see it in a function as though it can be
> called from someplace else
>

It'll be later used in the fds path, so I prefer to leave that as is.

>> +static void _put_uobj_ref(struct kref *ref)
>> +{
>> +     kfree(container_of(ref, struct ib_uobject, ref));
>> +}
>> +
>> +static void alloc_abort_idr_uobject(struct ib_uobject *uobj)
>> +{
>> +     uverbs_idr_remove_uobj(uobj);
>> +     /*
>> +      * we don't need kfree_rcu here, as the uobject wasn't exposed to any
>> +      * other verb.
>> +      */
>> +     kref_put(&uobj->ref, _put_uobj_ref);
>> +}
>
> Once needs_rcu is added then this ugly stuff goes away. Set needs_rcu
> only when the uboject has been added to the IDR.
>

We could do that but needs_rcu is actually a type feature.

>> +static void destroy_commit_idr_uobject(struct ib_uobject *uobj)
>
> This is a confusing name.. See below..
>
>> +{
>> +     uverbs_idr_remove_uobj(uobj);
>> +     mutex_lock(&uobj->context->lock);
>> +     list_del(&uobj->list);
>> +     mutex_unlock(&uobj->context->lock);
>> +     uverbs_uobject_put(uobj);
>> +}
>
> And this flow is weird, hot_unplug calls an idr_type op, but this does
> not? Why?
>

In hot unplug or context removal, we need to cleanup the object. In regular
destruction, the handler should take care of destructing the actual object.
It's more symmetrical this way (the handler created the object and
thus it should
destroy it), it lets the handler decide what to do if the destruction
failed or if the destruction
succeeded but copy_to_user failed.
Scenarios like this won't be supported as well:

ret = ib_destroy_xxxx(uobj_xxxx->object, &resp);
if (!ret) {
    copy_to_user(user_resp, &resp, sizeof(resp));
    destroy_commit(uobj_xxxx)
}

>> +void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
>> +{
>> +     unsigned int cur_order = 0;
>> +
>> +     while (!list_empty(&ucontext->uobjects)) {
>> +             struct ib_uobject *obj, *next_obj;
>> +             unsigned int next_order = UINT_MAX;
>> +
>> +             /*
>> +              * This shouldn't run while executing other commands on this
>> +              * context, thus no lock is required.
>> +              */
>> +             list_for_each_entry_safe(obj, next_obj, &ucontext->uobjects,
>> +                                      list)
>> +                     if (obj->type->destroy_order == cur_order) {
>> +                             list_del(&obj->list);
>> +                             obj->type->ops->hot_unplug(obj, device_removed);
>
> Let's add a safety:
>
>    WARN_ON(uverbs_lock_object(obj, true));
>

Ok

> It would be really nice to see a simpler 'built-in' scheme to
> guarentee that WARN_ON never hits. I expect this is relying on the
> external SRCU system to make it work out..
>
> For instance since we now have lookup_get/lookup_put could we
> ultimately move the SRCU into those functions?
>
> Also, the call to hot_unplug has to be checked
>

We assume all handlers finished executing and new handlers won't
execute anymore. It's required here that all handlers aren't executing and
the SRCU guarantees that. We could add another such mechanism or move it,
but it would just complicate the code. SRCU protects the commands handling
path and not just objects fetching and locking.

>> +     /* locking the uobjects_list */
>> +     struct mutex            lock;
>
> Prefer uobjects_lock for clarity
>

Ok

>> @@ -1374,8 +1378,11 @@ struct ib_uobject {
>>       int                     id;             /* index into kernel idr */
>
>> +
>> +struct uverbs_obj_type_ops {
>> +     /*
>> +      * Get an ib_uobject that corresponds to the given id from ucontext,
>> +      * These functions could create or destroy objects if required.
>> +      * The action will be finalized only when commit, abort or put fops are
>> +      * called.
>> +      * The flow of the different actions is:
>> +      * [alloc]:      Starts with alloc_begin. The handlers logic is than
>> +      *               executed. If the handler is successful, alloc_commit
>> +      *               is called and the object is inserted to the repository.
>> +      *               Otherwise, alloc_abort is called and the object is
>> +      *               destroyed.
>> +      * [lookup]:     Starts with lookup_get which fetches and locks the
>> +      *               object. After the handler finished using the object, it
>> +      *               needs to call lookup_put to unlock it. The write flag
>> +      *               indicates if the object is locked for exclusive access.
>> +      * [destroy]:    Starts with lookup_get with write flag set. This locks
>> +      *               the object for exclusive access. If the handler code
>> +      *               completed successfully, destroy_commit is called and
>> +      *               the ib_uobject is removed from the context's uobjects
>> +      *               repository and put. Otherwise, lookup_put should be
>> +      *               called.
>
> Lets call this 'remove' please.
>
> And add:
>
>  Once remove succeeds new krefs to the object cannot be acquired by
>  other threads or userspace and the hardware driver is removed from
>  the object. Other krefs on the object may still exist.
>

Ok

>> +      * [hot_unplug]: Used when the context is destroyed (process
>> +      *               termination, reset flow).
>
> I don't think we need a dedicated entry point. I think you should add
> an enum argument to remove:
>
> enum rdma_remove_reason {
>  RDMA_REMOVE_DESTROY, // Userspace requested uobject deletion
>  RDMA_REMOVE_CLOSE,   // Context deletion. Call cannot fail.
>  RDMA_REMOVE_DRIVER_REMOVE, // Driver is being hot-unplugged. Call cannot fail.
> };
>

The last two differs vastly from the first one. The first one only
deletes the uobject, assumes
that the handler itself deleted the actual object. The last two are
invoked when the context
is destroyed and we need to delete both the objects and the uobject.
Therefore, I can
rename destroy_commit to remove_commit and the hot_unplug to cleanup.
Cleanup could
get the reason.
None of these calls could fail.

> This gives the drivers a more information to the right thing to do.
>
>> +      * [release]:    Releases the memory of the ib_uobject. This is called
>> +      *               when the last reference is put. We favored a callback
>> +      *               here as this might require tricks like kfree_rcu.
>> +      *               This shouldn't be called explicitly by the user as it's
>> +      *               used by uverbs_uobject_put.
>
> Delete as discussed
>
>

Ok

>> +     void (*destroy_commit)(struct ib_uobject *uobj);
>
> Use:
>    /* Must be called with the write lock held. If successful uboj is
>       invalid on return. On failure uobject is left completely unchanged */
>    int __must_check (*remove)(struct ib_uobject *uobj,enum rdma_remove_reason why);
>

I don't think we need to check the result. It just can't fail.

>> +struct uverbs_obj_idr_type {
>> +     /*
>> +      * In idr based objects, uverbs_obj_type_ops points to a generic
>> +      * idr operations. In order to specialize the underlying types (e.g. CQ,
>> +      * QPs, etc.), we add obj_size and hot_unplug specific callbacks here.
>> +      */
>> +     struct uverbs_obj_type  type;
>> +     size_t                  obj_size;
>
> Move to type, do not support no kalloc configurations
>

Ok, I guess this should be enough for the expected types.

>> +     /* The hot_unplug in ops initialized by idr, calls this callback */
>> +     void (*hot_unplug)(struct ib_uobject *uobj);
>
> Use:
>
> /* Free driver resources from the uobject, make the driver uncallable,
>    and move the uobject to the detached state. On failure the uboject
>    is left completely unchanged. */
> int __must_check (*destroy)(struct ib_uobject *uobj,enum rdma_remove_reason why);
>

As above, this can't fail.

> And probably add some commentary what objects support a detached
> state. Any object that cannot be detached can only exist in the IDR
> and cannot have a kref taken. Perhaps we should enforce that directly
> for clarity.
>

Actually, I don't see why we can't support detached IDR objects....
As long as you don't execute commands on this IDR (for example, removing
it from the objects list), you could use the memory as long as you want
after you called uverbs_uobject_get (until you call uverbs_uobject_put).

> Jason

Thanks for the review

Matan

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 2/7] IB/core: Add support for idr types

[Jason Gunthorpe]

On Wed, Feb 15, 2017 at 03:43:31PM +0200, Matan Barak wrote:
> On Fri, Feb 10, 2017 at 9:56 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > On Wed, Feb 01, 2017 at 02:39:00PM +0200, Matan Barak wrote:
> >
> >> +static int uverbs_lock_object(struct ib_uobject *uobj, bool write)
> >> +{
> >> +     if (!write)
> >> +             return down_read_trylock(&uobj->currently_used) == 1 ? 0 :
> >> +                     -EBUSY;
> >> +
> >> +     /* lock is either WRITE or DESTROY - should be exclusive */
> >> +     return down_write_trylock(&uobj->currently_used) == 1 ? 0 : -EBUSY;
> >
> > Is currently_used ever used as an actual blocking rwsem? Looks like
> > no to me right now, is that the long term plan?
> >
> 
> Yeah, it's never used for blocking and that's the long term plan.
> 
> > It actually seems to be important that this not be blocking, so it
> > might be better to code this a simple atomic.
> >
> 
> Actually, in the first version it was coded as an atomic and we decided
> to change it to rwsem after some comments. Anyway, no problem
> changing it back to the original version (see
> http://www.spinics.net/lists/linux-rdma/msg38346.html).

Yes, in general I don't like seeing open coded locks, but in this case
the non-block is actually an important property so it may be clearer,
but the atomic is a messy construct as well.

Maybe a comment on currently_used that the rwsem may never be used
blocking is enough?

> > This sort of approach starts to become very dangerous when you
> > contemplate having 'release' be a function in a module. Is that the
> > case?
> >
> 
> Well, I expect these core meta-types (e.g. idr, fd) to be declared in ib_uverbs
> or ib_core.

So that is two modules and it starts to become tricky..

> >> +     init_rwsem(&uobj->currently_used);
> >> +     uobj->context     = context;
> >> +     uobj->type        = type;
> >
> > .. and can you please not add the bogus whitespace in new commits
> > please? That is really not the typical kernel style and makes
> > everything hard to maintain and read.
> 
> Which bogus whitespace?

The stuff before the = to 'line up' the rhs of the assignment.

> >> +static void uverbs_idr_remove_uobj(struct ib_uobject *uobj)
> >> +{
> >> +     spin_lock(&uobj->context->ufile->idr_lock);
> >> +     idr_remove(&uobj->context->ufile->idr, uobj->id);
> >> +     spin_unlock(&uobj->context->ufile->idr_lock);
> >> +}
> >
> > Add a clarifying comment
> >
> > /* The caller must uverbs_uobject_put() uobj */
> 
> It could actually call kfree directly if the object is guaranteed not
> to be used, but
> this is a micro optimization that we probably shouldn't really care about.
> I prefer to put needs_rcu on the type and not on each uobject.

If skipping the kfree_rcu is important the needs_rcu should be
per-object, otherwise per type is probably fine.


> > Hum. This RCU is a bit exciting.. So this relies on the write lock
> > being held whenever uverbs_idr_remove is called. Can you add a
> > LOCKDEP style of of assertion to uverbs_idr_remove to prove that?
> 
> if we remove the lock and use atomics instead, there's nothing to do
> lockdep on, right?

Well, lockdep-like, if it is an atomic then do an atomic test
protected by CONFIG_LOCKDEP or some other such approach

> >> +     init_uobj(uobj, ucontext, type);
> >> +     ret = idr_add_uobj(uobj);
> >> +     if (ret) {
> >> +             kfree(uobj);
> >
> > This should be a uverbs_uobject_put()
> >
> 
> It'll just postpone this to rcu, but I guess this doesn't really matter.

Idiomatically, once the kref is inited then all frees must go through
kref_put.

> >> +static void _put_uobj_ref(struct kref *ref)
> >> +{
> >> +     kfree(container_of(ref, struct ib_uobject, ref));
> >> +}
> >> +
> >> +static void alloc_abort_idr_uobject(struct ib_uobject *uobj)
> >> +{
> >> +     uverbs_idr_remove_uobj(uobj);
> >> +     /*
> >> +      * we don't need kfree_rcu here, as the uobject wasn't exposed to any
> >> +      * other verb.
> >> +      */
> >> +     kref_put(&uobj->ref, _put_uobj_ref);
> >> +}
> >
> > Once needs_rcu is added then this ugly stuff goes away. Set needs_rcu
> > only when the uboject has been added to the IDR.
> 
> We could do that but needs_rcu is actually a type feature.

Either way this stuff goes away.

> >> +{
> >> +     uverbs_idr_remove_uobj(uobj);
> >> +     mutex_lock(&uobj->context->lock);
> >> +     list_del(&uobj->list);
> >> +     mutex_unlock(&uobj->context->lock);
> >> +     uverbs_uobject_put(uobj);
> >> +}
> >
> > And this flow is weird, hot_unplug calls an idr_type op, but this does
> > not? Why?
> >
> 
> In hot unplug or context removal, we need to cleanup the object. In regular
> destruction, the handler should take care of destructing the actual
> object.

So, I think this is a mistake. There is nothing special the handler
does compared to the hot unplug case and it makes no sense to have two
places to duplicate the call out to the driver unplug code. Having
three copies when the ioctl stuff comes along is even worse..

Having them be different is just going to create bugs down the road...

I don't mind the asymmetry because our entire model assumes complex
creation and consistent/trivial destruction.

It is not unlike the device_allocate/device_destroy/device_put sort of
flow where the allocate step has many variations but the destroy is
always uniform.

> destroy it), it lets the handler decide what to do if the
> destruction failed or if the destruction succeeded but copy_to_user
> failed.  Scenarios like this won't be supported as well:

This is why I added the return code to (*destroy) - exactly so the one
place that cares can return the error code, and all the other places
that can't be allowed to fail can have common recovery handling in the
core code.

Too many of our destroy calls toward the driver can can fail, we need
this to be handled in one place, not sprinkled throughout the code.

> >> +      * [hot_unplug]: Used when the context is destroyed (process
> >> +      *               termination, reset flow).
> >
> > I don't think we need a dedicated entry point. I think you should add
> > an enum argument to remove:
> >
> > enum rdma_remove_reason {
> >  RDMA_REMOVE_DESTROY, // Userspace requested uobject deletion
> >  RDMA_REMOVE_CLOSE,   // Context deletion. Call cannot fail.
> >  RDMA_REMOVE_DRIVER_REMOVE, // Driver is being hot-unplugged. Call cannot fail.
> > };
> >
> 
> The last two differs vastly from the first one. The first one only
> deletes the uobject, assumes

No, that isn't my proposal. The idea is that all three do the same
thing excep that:
 - RDMA_REMOVE_DESTROY the driver can return an error code if it
   wishes, the code is returned to userspace
 - RDMA_REMOVE_CLOSE the driver cannot return an error code and must
   force-destroy the object
 - RDMA_REMOVE_DRIVER_REMOVE the driver cannot return an error code,
   must force-destroy the object, and must leave behind any stuff
   to let userspace keep working (eg a zero page mmap, or whatever)

All flows will call the driver destroy function.

The approach is that the one or two call sites that use
RDMA_REMOVE_CLOSE/RDMA_REMOVE_DRIVER_REMOVE would also include the
WARN_ON to prove the driver is working as expected and then leave
whatever dangling ref around in some kind of controlled way.

> None of these calls could fail.

But that isn't really true :)

> > And probably add some commentary what objects support a detached
> > state. Any object that cannot be detached can only exist in the IDR
> > and cannot have a kref taken. Perhaps we should enforce that directly
> > for clarity.
> >
> 
> Actually, I don't see why we can't support detached IDR objects....
> As long as you don't execute commands on this IDR (for example, removing
> it from the objects list), you could use the memory as long as you want
> after you called uverbs_uobject_get (until you call uverbs_uobject_put).

The question about 'detached' revolves around what the users of the
object do. Eg if the users blindly do

ib_foo_action(uboj->driver_obj)

after the driver is disconnected then they will crash. Those objects
do not support 'detatch'

You are right in general the the IDR framework is OK and doesn't
care. But I think the current state of affairs is that only some call
sites, probably the FD related ones, are actually safe for this usage.

This becomes a bit more important down the road if people want to do
things outside the ucontext - eg enumerate all of the QPs in a
process - then we need clear rules for how that continues to work.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 3/7] IB/core: Add idr based standard types

[Matan Barak]

This patch adds the standard idr based types. These types are
used in downstream patches in order to initialize, destroy and
lookup IB standard objects which are based on idr objects.

An idr object requires filling out several parameters. Its op pointer
should point to uverbs_idr_ops and its size should be at least the
size of ib_uobject. We add a macro to make the type declaration easier.

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/Makefile           |   2 +-
 drivers/infiniband/core/uverbs.h           |   2 +
 drivers/infiniband/core/uverbs_main.c      |   4 +-
 drivers/infiniband/core/uverbs_std_types.c | 165 +++++++++++++++++++++++++++++
 include/rdma/uverbs_std_types.h            |  50 +++++++++
 include/rdma/uverbs_types.h                |  16 +++
 6 files changed, 236 insertions(+), 3 deletions(-)
 create mode 100644 drivers/infiniband/core/uverbs_std_types.c
 create mode 100644 include/rdma/uverbs_std_types.h

diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index 1819623..c336d9c 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -29,4 +29,4 @@ ib_umad-y :=			user_mad.o
 ib_ucm-y :=			ucm.o
 
 ib_uverbs-y :=			uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
-				rdma_core.o
+				rdma_core.o uverbs_std_types.o
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 662a592..dd8048f 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -204,6 +204,8 @@ void ib_uverbs_event_handler(struct ib_event_handler *handler,
 void ib_uverbs_dealloc_xrcd(struct ib_uverbs_device *dev, struct ib_xrcd *xrcd);
 
 int uverbs_dealloc_mw(struct ib_mw *mw);
+void ib_uverbs_detach_umcast(struct ib_qp *qp,
+			     struct ib_uqp_object *uobj);
 
 struct ib_uverbs_flow_spec {
 	union {
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index 8344afd..ba8b1eb 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -200,8 +200,8 @@ void ib_uverbs_release_uevent(struct ib_uverbs_file *file,
 	spin_unlock_irq(&file->async_file->lock);
 }
 
-static void ib_uverbs_detach_umcast(struct ib_qp *qp,
-				    struct ib_uqp_object *uobj)
+void ib_uverbs_detach_umcast(struct ib_qp *qp,
+			     struct ib_uqp_object *uobj)
 {
 	struct ib_uverbs_mcast_entry *mcast, *tmp;
 
diff --git a/drivers/infiniband/core/uverbs_std_types.c b/drivers/infiniband/core/uverbs_std_types.c
new file mode 100644
index 0000000..9bbc4eb
--- /dev/null
+++ b/drivers/infiniband/core/uverbs_std_types.c
@@ -0,0 +1,165 @@
+/*
+ * Copyright (c) 2017, Mellanox Technologies inc.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <rdma/uverbs_std_types.h>
+#include <rdma/ib_user_verbs.h>
+#include <rdma/ib_verbs.h>
+#include <linux/bug.h>
+#include <linux/file.h>
+#include "rdma_core.h"
+#include "uverbs.h"
+
+void uverbs_free_ah(struct ib_uobject *uobject)
+{
+	ib_destroy_ah((struct ib_ah *)uobject->object);
+}
+
+void uverbs_free_flow(struct ib_uobject *uobject)
+{
+	ib_destroy_flow((struct ib_flow *)uobject->object);
+}
+
+void uverbs_free_mw(struct ib_uobject *uobject)
+{
+	uverbs_dealloc_mw((struct ib_mw *)uobject->object);
+}
+
+void uverbs_free_qp(struct ib_uobject *uobject)
+{
+	struct ib_qp *qp = uobject->object;
+	struct ib_uqp_object *uqp =
+		container_of(uobject, struct ib_uqp_object, uevent.uobject);
+
+	if (qp == qp->real_qp)
+		ib_uverbs_detach_umcast(qp, uqp);
+	ib_destroy_qp(qp);
+	ib_uverbs_release_uevent(uobject->context->ufile, &uqp->uevent);
+}
+
+void uverbs_free_rwq_ind_tbl(struct ib_uobject *uobject)
+{
+	struct ib_rwq_ind_table *rwq_ind_tbl = uobject->object;
+	struct ib_wq **ind_tbl = rwq_ind_tbl->ind_tbl;
+
+	ib_destroy_rwq_ind_table(rwq_ind_tbl);
+	kfree(ind_tbl);
+}
+
+void uverbs_free_wq(struct ib_uobject *uobject)
+{
+	struct ib_wq *wq = uobject->object;
+	struct ib_uwq_object *uwq =
+		container_of(uobject, struct ib_uwq_object, uevent.uobject);
+
+	ib_destroy_wq(wq);
+	ib_uverbs_release_uevent(uobject->context->ufile, &uwq->uevent);
+}
+
+void uverbs_free_srq(struct ib_uobject *uobject)
+{
+	struct ib_srq *srq = uobject->object;
+	struct ib_uevent_object *uevent =
+		container_of(uobject, struct ib_uevent_object, uobject);
+
+	ib_destroy_srq(srq);
+	ib_uverbs_release_uevent(uobject->context->ufile, uevent);
+}
+
+void uverbs_free_cq(struct ib_uobject *uobject)
+{
+	struct ib_cq *cq = uobject->object;
+	struct ib_uverbs_event_file *ev_file = cq->cq_context;
+	struct ib_ucq_object *ucq =
+		container_of(uobject, struct ib_ucq_object, uobject);
+
+	ib_destroy_cq(cq);
+	ib_uverbs_release_ucq(uobject->context->ufile, ev_file, ucq);
+}
+
+void uverbs_free_mr(struct ib_uobject *uobject)
+{
+	ib_dereg_mr((struct ib_mr *)uobject->object);
+}
+
+void uverbs_free_xrcd(struct ib_uobject *uobject)
+{
+	struct ib_xrcd *xrcd = uobject->object;
+
+	mutex_lock(&uobject->context->ufile->device->xrcd_tree_mutex);
+	ib_uverbs_dealloc_xrcd(uobject->context->ufile->device, xrcd);
+	mutex_unlock(&uobject->context->ufile->device->xrcd_tree_mutex);
+}
+
+void uverbs_free_pd(struct ib_uobject *uobject)
+{
+	ib_dealloc_pd((struct ib_pd *)uobject->object);
+}
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_cq =
+	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_ucq_object), 0,
+				 uverbs_free_cq);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_qp =
+	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_uqp_object), 0,
+				 uverbs_free_qp);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_mw =
+	UVERBS_TYPE_ALLOC_IDR(0, uverbs_free_mw);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_mr =
+	/* 1 is used in order to free the MR after all the MWs */
+	UVERBS_TYPE_ALLOC_IDR(1, uverbs_free_mr);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_srq =
+	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_usrq_object), 0,
+				 uverbs_free_srq);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_ah =
+	UVERBS_TYPE_ALLOC_IDR(0, uverbs_free_ah);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_flow =
+	UVERBS_TYPE_ALLOC_IDR(0, uverbs_free_flow);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_wq =
+	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_uwq_object), 0,
+				 uverbs_free_wq);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_rwq_ind_table =
+	UVERBS_TYPE_ALLOC_IDR(0, uverbs_free_rwq_ind_tbl);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_xrcd =
+	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_uxrcd_object), 0,
+				 uverbs_free_xrcd);
+
+const struct uverbs_obj_idr_type uverbs_type_attrs_pd =
+	/* 2 is used in order to free the PD after MRs */
+	UVERBS_TYPE_ALLOC_IDR(2, uverbs_free_pd);
diff --git a/include/rdma/uverbs_std_types.h b/include/rdma/uverbs_std_types.h
new file mode 100644
index 0000000..2edb776
--- /dev/null
+++ b/include/rdma/uverbs_std_types.h
@@ -0,0 +1,50 @@
+/*
+ * Copyright (c) 2017, Mellanox Technologies inc.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef _UVERBS_STD_TYPES__
+#define _UVERBS_STD_TYPES__
+
+#include <rdma/uverbs_types.h>
+
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_cq;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_qp;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_rwq_ind_table;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_wq;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_srq;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_ah;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_flow;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_mr;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_mw;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_pd;
+extern const struct uverbs_obj_idr_type uverbs_type_attrs_xrcd;
+#endif
+
diff --git a/include/rdma/uverbs_types.h b/include/rdma/uverbs_types.h
index bb263f0..cdbf352 100644
--- a/include/rdma/uverbs_types.h
+++ b/include/rdma/uverbs_types.h
@@ -101,4 +101,20 @@ struct uverbs_obj_idr_type {
 	void (*hot_unplug)(struct ib_uobject *uobj);
 };
 
+extern const struct uverbs_obj_type_ops uverbs_idr_ops;
+
+#define UVERBS_BUILD_BUG_ON(cond) (sizeof(char[1 - 2 * !!(cond)]) -	\
+				   sizeof(char))
+#define UVERBS_TYPE_ALLOC_IDR_SZ(_size, _order, _hot_unplug)		\
+	 {.type = {							\
+		.destroy_order = _order,				\
+		.ops = &uverbs_idr_ops,					\
+	 },								\
+	 .hot_unplug = _hot_unplug,					\
+	 .obj_size = (_size) +						\
+		UVERBS_BUILD_BUG_ON((_size) < sizeof(struct		\
+						     ib_uobject))}
+#define UVERBS_TYPE_ALLOC_IDR(_order, _hot_unplug)			\
+	 UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_uobject), _order,	\
+				  _hot_unplug)
 #endif
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 3/7] IB/core: Add idr based standard types

[Jason Gunthorpe]

On Wed, Feb 01, 2017 at 02:39:01PM +0200, Matan Barak wrote:

> +void uverbs_free_qp(struct ib_uobject *uobject)

This should be static...

Based on the last email I expect this to be

static int uverbs_destroy_qp(struct ib_uobject *uobject, enum rdma_remove_reason why)

> +{
> +	struct ib_qp *qp = uobject->object;
> +	struct ib_uqp_object *uqp =
> +		container_of(uobject, struct ib_uqp_object, uevent.uobject);
> +
> +	if (qp == qp->real_qp)
> +		ib_uverbs_detach_umcast(qp, uqp);
> +	ib_destroy_qp(qp);

This silently eats the error code from ib_destroy_qp, not OK. See
discussion in prior email. Lots of cases of this.

> +const struct uverbs_obj_idr_type uverbs_type_attrs_cq =
> +	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_ucq_object), 0,
> +				 uverbs_free_cq);
>

It would be more readable if the struct decl was near the functions in
blocks.

Still don't like these macros. The open coded version would look like this:

const struct uverbs_obj_idr_type uverbs_type_attrs_cq = {
      .type = {
      	    .obj_size = sizeof(struct ib_ucq_object),
      	    .ops = &uverbs_idr_ops,
      },
      .destroy = uverbs_destroy_cq,
};

Whichs is more understandable and typical of the kernel style. If we
wish to add a new callback function for instance we don't need a new
macro or big churn.

You could do this I guess:

const struct uverbs_obj_idr_type uverbs_type_attrs_cq = {
      .type = UVERBS_IDR_TYPE(cq, 0),
      .destroy = uverbs_destroy_cq,
};

> +extern const struct uverbs_obj_type_ops uverbs_idr_ops;
> +
> +#define UVERBS_BUILD_BUG_ON(cond) (sizeof(char[1 - 2 * !!(cond)]) -	\
> +				   sizeof(char))

This probably doesn't add value, but you could make this a WARN_ON in
alloc_uobj().

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 3/7] IB/core: Add idr based standard types

[Matan Barak]

On Fri, Feb 10, 2017 at 10:08 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Feb 01, 2017 at 02:39:01PM +0200, Matan Barak wrote:
>
>> +void uverbs_free_qp(struct ib_uobject *uobject)
>
> This should be static...
>

Yeah

> Based on the last email I expect this to be
>
> static int uverbs_destroy_qp(struct ib_uobject *uobject, enum rdma_remove_reason why)
>

Since we don't care about the reason here and it's called from the
destroy callback,
we could leave the deceleration as is.

>> +{
>> +     struct ib_qp *qp = uobject->object;
>> +     struct ib_uqp_object *uqp =
>> +             container_of(uobject, struct ib_uqp_object, uevent.uobject);
>> +
>> +     if (qp == qp->real_qp)
>> +             ib_uverbs_detach_umcast(qp, uqp);
>> +     ib_destroy_qp(qp);
>
> This silently eats the error code from ib_destroy_qp, not OK. See
> discussion in prior email. Lots of cases of this.
>

I can add a WARN_ON, but this is essentially the same as ib_destroy_qp
fails when a context is destroyed (either because of hot unplug or process
termination). You won't get here because of regular destroy handler.
This is the current behavior as well.

>> +const struct uverbs_obj_idr_type uverbs_type_attrs_cq =
>> +     UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_ucq_object), 0,
>> +                              uverbs_free_cq);
>>
>
> It would be more readable if the struct decl was near the functions in
> blocks.
>

But then it can't be used by the hardware drivers to declare new types.

> Still don't like these macros. The open coded version would look like this:
>
> const struct uverbs_obj_idr_type uverbs_type_attrs_cq = {
>       .type = {
>             .obj_size = sizeof(struct ib_ucq_object),
>             .ops = &uverbs_idr_ops,
>       },
>       .destroy = uverbs_destroy_cq,
> };
>
> Whichs is more understandable and typical of the kernel style. If we
> wish to add a new callback function for instance we don't need a new
> macro or big churn.
>
> You could do this I guess:
>
> const struct uverbs_obj_idr_type uverbs_type_attrs_cq = {
>       .type = UVERBS_IDR_TYPE(cq, 0),
>       .destroy = uverbs_destroy_cq,
> };
>

Yeah, seems reasonable - a good trade off.

>> +extern const struct uverbs_obj_type_ops uverbs_idr_ops;
>> +
>> +#define UVERBS_BUILD_BUG_ON(cond) (sizeof(char[1 - 2 * !!(cond)]) -  \
>> +                                sizeof(char))
>
> This probably doesn't add value, but you could make this a WARN_ON in
> alloc_uobj().
>

Why should we calculate this on runtime if we could know this in compile time?

> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 3/7] IB/core: Add idr based standard types

[Jason Gunthorpe]

On Wed, Feb 15, 2017 at 03:44:22PM +0200, Matan Barak wrote:

> > static int uverbs_destroy_qp(struct ib_uobject *uobject, enum rdma_remove_reason why)
> >
> 
> Since we don't care about the reason here and it's called from the
> destroy callback,
> we could leave the deceleration as is.

The point is to ultimately push the reason down to the driver so the
driver can do the proper action. Eg force-destroy a HCA object.

> I can add a WARN_ON, but this is essentially the same as ib_destroy_qp
> fails when a context is destroyed (either because of hot unplug or process
> termination). You won't get here because of regular destroy handler.
> This is the current behavior as well.

I don't think the current behavior is very good. We've had driver
submissions that include random failures inside their destroy routines
- the current situation is a mess.
 
> Why should we calculate this on runtime if we could know this in compile time?

Because then you need the ugly macro :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 4/7] IB/core: Change idr objects to use the new schema

[Matan Barak]

This changes only the handlers which deals with idr based objects to
use the new idr allocation, fetching and destruction schema.
This patch consists of the following changes:
(1) Allocation, fetching and destruction is done via idr ops.
(2) Context initializing and release is done through
    uverbs_initialize_ucontext and uverbs_cleanup_ucontext.
(3) Ditching the live flag. Mostly, this is pretty straight
    forward. The only place that is a bit trickier is in
    ib_uverbs_open_qp. Commit [1] added code to check whether
    the uobject is already live and initialized. This mostly
    happens because of a race between open_qp and events.
    We delayed assigning the uobject's pointer in order to
    eliminate this race without using the live variable.

[1] commit a040f95dc819
	("IB/core: Fix XRC race condition in ib_uverbs_open_qp")

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/uverbs.h      |   2 -
 drivers/infiniband/core/uverbs_cmd.c  | 909 ++++++++++------------------------
 drivers/infiniband/core/uverbs_main.c | 124 +----
 include/rdma/ib_verbs.h               |  13 -
 4 files changed, 260 insertions(+), 788 deletions(-)

diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index dd8048f..1e31ad8 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -180,8 +180,6 @@ struct ib_ucq_object {
 	u32			async_events_reported;
 };
 
-void idr_remove_uobj(struct ib_uobject *uobj);
-
 struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file,
 					struct ib_device *ib_dev,
 					int is_async);
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index 29c621d..17c6dcf 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -40,177 +40,29 @@
 
 #include <linux/uaccess.h>
 
+#include <rdma/uverbs_types.h>
+#include <rdma/uverbs_std_types.h>
+#include "rdma_core.h"
+
 #include "uverbs.h"
 #include "core_priv.h"
 
-struct uverbs_lock_class {
-	struct lock_class_key	key;
-	char			name[16];
-};
-
-static struct uverbs_lock_class pd_lock_class	= { .name = "PD-uobj" };
-static struct uverbs_lock_class mr_lock_class	= { .name = "MR-uobj" };
-static struct uverbs_lock_class mw_lock_class	= { .name = "MW-uobj" };
-static struct uverbs_lock_class cq_lock_class	= { .name = "CQ-uobj" };
-static struct uverbs_lock_class qp_lock_class	= { .name = "QP-uobj" };
-static struct uverbs_lock_class ah_lock_class	= { .name = "AH-uobj" };
-static struct uverbs_lock_class srq_lock_class	= { .name = "SRQ-uobj" };
-static struct uverbs_lock_class xrcd_lock_class = { .name = "XRCD-uobj" };
-static struct uverbs_lock_class rule_lock_class = { .name = "RULE-uobj" };
-static struct uverbs_lock_class wq_lock_class = { .name = "WQ-uobj" };
-static struct uverbs_lock_class rwq_ind_table_lock_class = { .name = "IND_TBL-uobj" };
-
-/*
- * The ib_uobject locking scheme is as follows:
- *
- * - ib_uverbs_idr_lock protects the uverbs idrs themselves, so it
- *   needs to be held during all idr write operations.  When an object is
- *   looked up, a reference must be taken on the object's kref before
- *   dropping this lock.  For read operations, the rcu_read_lock()
- *   and rcu_write_lock() but similarly the kref reference is grabbed
- *   before the rcu_read_unlock().
- *
- * - Each object also has an rwsem.  This rwsem must be held for
- *   reading while an operation that uses the object is performed.
- *   For example, while registering an MR, the associated PD's
- *   uobject.mutex must be held for reading.  The rwsem must be held
- *   for writing while initializing or destroying an object.
- *
- * - In addition, each object has a "live" flag.  If this flag is not
- *   set, then lookups of the object will fail even if it is found in
- *   the idr.  This handles a reader that blocks and does not acquire
- *   the rwsem until after the object is destroyed.  The destroy
- *   operation will set the live flag to 0 and then drop the rwsem;
- *   this will allow the reader to acquire the rwsem, see that the
- *   live flag is 0, and then drop the rwsem and its reference to
- *   object.  The underlying storage will not be freed until the last
- *   reference to the object is dropped.
- */
-
-static void init_uobj(struct ib_uobject *uobj, u64 user_handle,
-		      struct ib_ucontext *context, struct uverbs_lock_class *c)
-{
-	uobj->user_handle = user_handle;
-	uobj->context     = context;
-	kref_init(&uobj->ref);
-	init_rwsem(&uobj->mutex);
-	lockdep_set_class_and_name(&uobj->mutex, &c->key, c->name);
-	uobj->live        = 0;
-}
-
-static void release_uobj(struct kref *kref)
-{
-	kfree_rcu(container_of(kref, struct ib_uobject, ref), rcu);
-}
-
-static void put_uobj(struct ib_uobject *uobj)
-{
-	kref_put(&uobj->ref, release_uobj);
-}
+#define idr_get_xxxx(_type, _write, _handle, _context) ({		\
+	const struct uverbs_obj_idr_type * const type =			\
+			&uverbs_type_attrs_##_type;			\
+	struct ib_uobject *uobj = type->type.ops->lookup_get(&type->type, \
+					_context, _handle, _write);	\
+									\
+	IS_ERR(uobj) ? NULL : uobj->object; })
 
 static void put_uobj_read(struct ib_uobject *uobj)
 {
-	up_read(&uobj->mutex);
-	put_uobj(uobj);
-}
-
-static void put_uobj_write(struct ib_uobject *uobj)
-{
-	up_write(&uobj->mutex);
-	put_uobj(uobj);
-}
-
-static int idr_add_uobj(struct ib_uobject *uobj)
-{
-	int ret;
-
-	idr_preload(GFP_KERNEL);
-	spin_lock(&uobj->context->ufile->idr_lock);
-
-	ret = idr_alloc(&uobj->context->ufile->idr, uobj, 0, 0, GFP_NOWAIT);
-	if (ret >= 0)
-		uobj->id = ret;
-
-	spin_unlock(&uobj->context->ufile->idr_lock);
-	idr_preload_end();
-
-	return ret < 0 ? ret : 0;
-}
-
-void idr_remove_uobj(struct ib_uobject *uobj)
-{
-	spin_lock(&uobj->context->ufile->idr_lock);
-	idr_remove(&uobj->context->ufile->idr, uobj->id);
-	spin_unlock(&uobj->context->ufile->idr_lock);
-}
-
-static struct ib_uobject *__idr_get_uobj(int id, struct ib_ucontext *context)
-{
-	struct ib_uobject *uobj;
-
-	rcu_read_lock();
-	uobj = idr_find(&context->ufile->idr, id);
-	if (uobj) {
-		if (uobj->context == context)
-			kref_get(&uobj->ref);
-		else
-			uobj = NULL;
-	}
-	rcu_read_unlock();
-
-	return uobj;
-}
-
-static struct ib_uobject *idr_read_uobj(int id, struct ib_ucontext *context,
-					int nested)
-{
-	struct ib_uobject *uobj;
-
-	uobj = __idr_get_uobj(id, context);
-	if (!uobj)
-		return NULL;
-
-	if (nested)
-		down_read_nested(&uobj->mutex, SINGLE_DEPTH_NESTING);
-	else
-		down_read(&uobj->mutex);
-	if (!uobj->live) {
-		put_uobj_read(uobj);
-		return NULL;
-	}
-
-	return uobj;
-}
-
-static struct ib_uobject *idr_write_uobj(int id, struct ib_ucontext *context)
-{
-	struct ib_uobject *uobj;
-
-	uobj = __idr_get_uobj(id, context);
-	if (!uobj)
-		return NULL;
-
-	down_write(&uobj->mutex);
-	if (!uobj->live) {
-		put_uobj_write(uobj);
-		return NULL;
-	}
-
-	return uobj;
-}
-
-static void *idr_read_obj(int id, struct ib_ucontext *context,
-			  int nested)
-{
-	struct ib_uobject *uobj;
-
-	uobj = idr_read_uobj(id, context, nested);
-	return uobj ? uobj->object : NULL;
+	uobj->type->ops->lookup_put(uobj, false);
 }
 
 static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(pd_handle, context, 0);
+	return idr_get_xxxx(pd, false, pd_handle, context);
 }
 
 static void put_pd_read(struct ib_pd *pd)
@@ -218,9 +70,9 @@ static void put_pd_read(struct ib_pd *pd)
 	put_uobj_read(pd->uobject);
 }
 
-static struct ib_cq *idr_read_cq(int cq_handle, struct ib_ucontext *context, int nested)
+static struct ib_cq *idr_read_cq(int cq_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(cq_handle, context, nested);
+	return idr_get_xxxx(cq, false, cq_handle, context);
 }
 
 static void put_cq_read(struct ib_cq *cq)
@@ -230,7 +82,7 @@ static void put_cq_read(struct ib_cq *cq)
 
 static struct ib_ah *idr_read_ah(int ah_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(ah_handle, context, 0);
+	return idr_get_xxxx(ah, false, ah_handle, context);
 }
 
 static void put_ah_read(struct ib_ah *ah)
@@ -240,12 +92,12 @@ static void put_ah_read(struct ib_ah *ah)
 
 static struct ib_qp *idr_read_qp(int qp_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(qp_handle, context, 0);
+	return idr_get_xxxx(qp, false, qp_handle, context);
 }
 
 static struct ib_wq *idr_read_wq(int wq_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(wq_handle, context, 0);
+	return idr_get_xxxx(wq, false, wq_handle, context);
 }
 
 static void put_wq_read(struct ib_wq *wq)
@@ -256,7 +108,8 @@ static void put_wq_read(struct ib_wq *wq)
 static struct ib_rwq_ind_table *idr_read_rwq_indirection_table(int ind_table_handle,
 							       struct ib_ucontext *context)
 {
-	return idr_read_obj(ind_table_handle, context, 0);
+	return idr_get_xxxx(rwq_ind_table, false,
+			    ind_table_handle, context);
 }
 
 static void put_rwq_indirection_table_read(struct ib_rwq_ind_table *ind_table)
@@ -264,39 +117,28 @@ static void put_rwq_indirection_table_read(struct ib_rwq_ind_table *ind_table)
 	put_uobj_read(ind_table->uobject);
 }
 
-static struct ib_qp *idr_write_qp(int qp_handle, struct ib_ucontext *context)
-{
-	struct ib_uobject *uobj;
-
-	uobj = idr_write_uobj(qp_handle, context);
-	return uobj ? uobj->object : NULL;
-}
-
 static void put_qp_read(struct ib_qp *qp)
 {
 	put_uobj_read(qp->uobject);
 }
 
-static void put_qp_write(struct ib_qp *qp)
+static void put_srq_read(struct ib_srq *srq)
 {
-	put_uobj_write(qp->uobject);
+	put_uobj_read(srq->uobject);
 }
 
 static struct ib_srq *idr_read_srq(int srq_handle, struct ib_ucontext *context)
 {
-	return idr_read_obj(srq_handle, context, 0);
-}
-
-static void put_srq_read(struct ib_srq *srq)
-{
-	put_uobj_read(srq->uobject);
+	return idr_get_xxxx(srq, false, srq_handle, context);
 }
 
 static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context,
 				     struct ib_uobject **uobj)
 {
-	*uobj = idr_read_uobj(xrcd_handle, context, 0);
-	return *uobj ? (*uobj)->object : NULL;
+	*uobj = uverbs_type_attrs_xrcd.type.ops->lookup_get(&uverbs_type_attrs_xrcd.type,
+							context, xrcd_handle,
+							false);
+	return IS_ERR(*uobj) ? NULL : (*uobj)->object;
 }
 
 static void put_xrcd_read(struct ib_uobject *uobj)
@@ -341,17 +183,8 @@ ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file,
 	ucontext->device = ib_dev;
 	/* ufile is required when some objects are released */
 	ucontext->ufile = file;
-	INIT_LIST_HEAD(&ucontext->pd_list);
-	INIT_LIST_HEAD(&ucontext->mr_list);
-	INIT_LIST_HEAD(&ucontext->mw_list);
-	INIT_LIST_HEAD(&ucontext->cq_list);
-	INIT_LIST_HEAD(&ucontext->qp_list);
-	INIT_LIST_HEAD(&ucontext->srq_list);
-	INIT_LIST_HEAD(&ucontext->ah_list);
-	INIT_LIST_HEAD(&ucontext->wq_list);
-	INIT_LIST_HEAD(&ucontext->rwq_ind_tbl_list);
-	INIT_LIST_HEAD(&ucontext->xrcd_list);
-	INIT_LIST_HEAD(&ucontext->rule_list);
+	uverbs_initialize_ucontext(ucontext);
+
 	rcu_read_lock();
 	ucontext->tgid = get_task_pid(current->group_leader, PIDTYPE_PID);
 	rcu_read_unlock();
@@ -555,12 +388,10 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
 		   (unsigned long) cmd.response + sizeof resp,
 		   in_len - sizeof cmd, out_len - sizeof resp);
 
-	uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
-	if (!uobj)
-		return -ENOMEM;
-
-	init_uobj(uobj, 0, file->ucontext, &pd_lock_class);
-	down_write(&uobj->mutex);
+	uobj = uverbs_type_attrs_pd.type.ops->alloc_begin(&uverbs_type_attrs_pd.type,
+						      file->ucontext);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	pd = ib_dev->alloc_pd(ib_dev, file->ucontext, &udata);
 	if (IS_ERR(pd)) {
@@ -574,10 +405,6 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
 	atomic_set(&pd->usecnt, 0);
 
 	uobj->object = pd;
-	ret = idr_add_uobj(uobj);
-	if (ret)
-		goto err_idr;
-
 	memset(&resp, 0, sizeof resp);
 	resp.pd_handle = uobj->id;
 
@@ -587,24 +414,15 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
 		goto err_copy;
 	}
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&uobj->list, &file->ucontext->pd_list);
-	mutex_unlock(&file->mutex);
-
-	uobj->live = 1;
-
-	up_write(&uobj->mutex);
+	uobj->type->ops->alloc_commit(uobj);
 
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(uobj);
-
-err_idr:
 	ib_dealloc_pd(pd);
 
 err:
-	put_uobj_write(uobj);
+	uobj->type->ops->alloc_abort(uobj);
 	return ret;
 }
 
@@ -621,9 +439,11 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(cmd.pd_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_pd.type.ops->lookup_get(&uverbs_type_attrs_pd.type,
+						     file->ucontext,
+						     cmd.pd_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 	pd = uobj->object;
 
 	if (atomic_read(&pd->usecnt)) {
@@ -636,21 +456,12 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
 	if (ret)
 		goto err_put;
 
-	uobj->live = 0;
-	put_uobj_write(uobj);
-
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	return in_len;
 
 err_put:
-	put_uobj_write(uobj);
+	uobj->type->ops->lookup_put(uobj, true);
 	return ret;
 }
 
@@ -788,16 +599,14 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 		}
 	}
 
-	obj = kmalloc(sizeof *obj, GFP_KERNEL);
-	if (!obj) {
-		ret = -ENOMEM;
+	obj = (struct ib_uxrcd_object *)
+		uverbs_type_attrs_xrcd.type.ops->alloc_begin(&uverbs_type_attrs_xrcd.type,
+							 file->ucontext);
+	if (IS_ERR(obj)) {
+		ret = PTR_ERR(obj);
 		goto err_tree_mutex_unlock;
 	}
 
-	init_uobj(&obj->uobject, 0, file->ucontext, &xrcd_lock_class);
-
-	down_write(&obj->uobject.mutex);
-
 	if (!xrcd) {
 		xrcd = ib_dev->alloc_xrcd(ib_dev, file->ucontext, &udata);
 		if (IS_ERR(xrcd)) {
@@ -815,10 +624,6 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 
 	atomic_set(&obj->refcnt, 0);
 	obj->uobject.object = xrcd;
-	ret = idr_add_uobj(&obj->uobject);
-	if (ret)
-		goto err_idr;
-
 	memset(&resp, 0, sizeof resp);
 	resp.xrcd_handle = obj->uobject.id;
 
@@ -827,7 +632,7 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 			/* create new inode/xrcd table entry */
 			ret = xrcd_table_insert(file->device, inode, xrcd);
 			if (ret)
-				goto err_insert_xrcd;
+				goto err_dealloc_xrcd;
 		}
 		atomic_inc(&xrcd->usecnt);
 	}
@@ -841,12 +646,7 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 	if (f.file)
 		fdput(f);
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&obj->uobject.list, &file->ucontext->xrcd_list);
-	mutex_unlock(&file->mutex);
-
-	obj->uobject.live = 1;
-	up_write(&obj->uobject.mutex);
+	(&obj->uobject)->type->ops->alloc_commit(&obj->uobject);
 
 	mutex_unlock(&file->device->xrcd_tree_mutex);
 	return in_len;
@@ -858,14 +658,11 @@ ssize_t ib_uverbs_open_xrcd(struct ib_uverbs_file *file,
 		atomic_dec(&xrcd->usecnt);
 	}
 
-err_insert_xrcd:
-	idr_remove_uobj(&obj->uobject);
-
-err_idr:
+err_dealloc_xrcd:
 	ib_dealloc_xrcd(xrcd);
 
 err:
-	put_uobj_write(&obj->uobject);
+	(&obj->uobject)->type->ops->alloc_abort(&obj->uobject);
 
 err_tree_mutex_unlock:
 	if (f.file)
@@ -886,57 +683,50 @@ ssize_t ib_uverbs_close_xrcd(struct ib_uverbs_file *file,
 	struct ib_xrcd              *xrcd = NULL;
 	struct inode                *inode = NULL;
 	struct ib_uxrcd_object      *obj;
-	int                         live;
 	int                         ret = 0;
+	bool			    destroyed = false;
 
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
 	mutex_lock(&file->device->xrcd_tree_mutex);
-	uobj = idr_write_uobj(cmd.xrcd_handle, file->ucontext);
-	if (!uobj) {
-		ret = -EINVAL;
-		goto out;
+	uobj = uverbs_type_attrs_xrcd.type.ops->lookup_get(&uverbs_type_attrs_xrcd.type,
+						       file->ucontext,
+						       cmd.xrcd_handle, true);
+	if (IS_ERR(uobj)) {
+		mutex_unlock(&file->device->xrcd_tree_mutex);
+		return PTR_ERR(uobj);
 	}
 
 	xrcd  = uobj->object;
 	inode = xrcd->inode;
 	obj   = container_of(uobj, struct ib_uxrcd_object, uobject);
 	if (atomic_read(&obj->refcnt)) {
-		put_uobj_write(uobj);
 		ret = -EBUSY;
 		goto out;
 	}
 
 	if (!inode || atomic_dec_and_test(&xrcd->usecnt)) {
 		ret = ib_dealloc_xrcd(uobj->object);
-		if (!ret)
-			uobj->live = 0;
+		destroyed = !ret;
 	}
 
-	live = uobj->live;
 	if (inode && ret)
 		atomic_inc(&xrcd->usecnt);
 
-	put_uobj_write(uobj);
-
 	if (ret)
 		goto out;
 
-	if (inode && !live)
+	if (inode && destroyed)
 		xrcd_table_delete(file->device, inode);
 
-	idr_remove_uobj(uobj);
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
-	ret = in_len;
-
 out:
+	if (!ret)
+		uobj->type->ops->destroy_commit(uobj);
+	else
+		uobj->type->ops->lookup_put(uobj, true);
 	mutex_unlock(&file->device->xrcd_tree_mutex);
-	return ret;
+	return ret ?: in_len;
 }
 
 void ib_uverbs_dealloc_xrcd(struct ib_uverbs_device *dev,
@@ -984,12 +774,10 @@ ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file,
 	if (ret)
 		return ret;
 
-	uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
-	if (!uobj)
-		return -ENOMEM;
-
-	init_uobj(uobj, 0, file->ucontext, &mr_lock_class);
-	down_write(&uobj->mutex);
+	uobj = uverbs_type_attrs_mr.type.ops->alloc_begin(&uverbs_type_attrs_mr.type,
+						      file->ucontext);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	pd = idr_read_pd(cmd.pd_handle, file->ucontext);
 	if (!pd) {
@@ -1019,9 +807,6 @@ ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file,
 	atomic_inc(&pd->usecnt);
 
 	uobj->object = mr;
-	ret = idr_add_uobj(uobj);
-	if (ret)
-		goto err_unreg;
 
 	memset(&resp, 0, sizeof resp);
 	resp.lkey      = mr->lkey;
@@ -1036,27 +821,18 @@ ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file,
 
 	put_pd_read(pd);
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&uobj->list, &file->ucontext->mr_list);
-	mutex_unlock(&file->mutex);
-
-	uobj->live = 1;
-
-	up_write(&uobj->mutex);
+	uobj->type->ops->alloc_commit(uobj);
 
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(uobj);
-
-err_unreg:
 	ib_dereg_mr(mr);
 
 err_put:
 	put_pd_read(pd);
 
 err_free:
-	put_uobj_write(uobj);
+	uobj->type->ops->alloc_abort(uobj);
 	return ret;
 }
 
@@ -1092,10 +868,11 @@ ssize_t ib_uverbs_rereg_mr(struct ib_uverbs_file *file,
 	     (cmd.start & ~PAGE_MASK) != (cmd.hca_va & ~PAGE_MASK)))
 			return -EINVAL;
 
-	uobj = idr_write_uobj(cmd.mr_handle, file->ucontext);
-
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_mr.type.ops->lookup_get(&uverbs_type_attrs_mr.type,
+						     file->ucontext,
+						     cmd.mr_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	mr = uobj->object;
 
@@ -1142,8 +919,7 @@ ssize_t ib_uverbs_rereg_mr(struct ib_uverbs_file *file,
 		put_pd_read(pd);
 
 put_uobjs:
-
-	put_uobj_write(mr->uobject);
+	uobj->type->ops->lookup_put(uobj, true);
 
 	return ret;
 }
@@ -1161,28 +937,22 @@ ssize_t ib_uverbs_dereg_mr(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(cmd.mr_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_mr.type.ops->lookup_get(&uverbs_type_attrs_mr.type,
+						     file->ucontext,
+						     cmd.mr_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	mr = uobj->object;
 
 	ret = ib_dereg_mr(mr);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
 
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
+	}
 
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	return in_len;
 }
@@ -1206,12 +976,10 @@ ssize_t ib_uverbs_alloc_mw(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof(cmd)))
 		return -EFAULT;
 
-	uobj = kmalloc(sizeof(*uobj), GFP_KERNEL);
-	if (!uobj)
-		return -ENOMEM;
-
-	init_uobj(uobj, 0, file->ucontext, &mw_lock_class);
-	down_write(&uobj->mutex);
+	uobj = uverbs_type_attrs_mw.type.ops->alloc_begin(&uverbs_type_attrs_mw.type,
+						      file->ucontext);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	pd = idr_read_pd(cmd.pd_handle, file->ucontext);
 	if (!pd) {
@@ -1236,9 +1004,6 @@ ssize_t ib_uverbs_alloc_mw(struct ib_uverbs_file *file,
 	atomic_inc(&pd->usecnt);
 
 	uobj->object = mw;
-	ret = idr_add_uobj(uobj);
-	if (ret)
-		goto err_unalloc;
 
 	memset(&resp, 0, sizeof(resp));
 	resp.rkey      = mw->rkey;
@@ -1251,28 +1016,16 @@ ssize_t ib_uverbs_alloc_mw(struct ib_uverbs_file *file,
 	}
 
 	put_pd_read(pd);
-
-	mutex_lock(&file->mutex);
-	list_add_tail(&uobj->list, &file->ucontext->mw_list);
-	mutex_unlock(&file->mutex);
-
-	uobj->live = 1;
-
-	up_write(&uobj->mutex);
+	uobj->type->ops->alloc_commit(uobj);
 
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(uobj);
-
-err_unalloc:
 	uverbs_dealloc_mw(mw);
-
 err_put:
 	put_pd_read(pd);
-
 err_free:
-	put_uobj_write(uobj);
+	uobj->type->ops->alloc_abort(uobj);
 	return ret;
 }
 
@@ -1289,28 +1042,21 @@ ssize_t ib_uverbs_dealloc_mw(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof(cmd)))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(cmd.mw_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_mw.type.ops->lookup_get(&uverbs_type_attrs_mw.type,
+						     file->ucontext,
+						     cmd.mw_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	mw = uobj->object;
 
 	ret = uverbs_dealloc_mw(mw);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
+	}
 
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	return in_len;
 }
@@ -1376,12 +1122,11 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	if (cmd->comp_vector >= file->device->num_comp_vectors)
 		return ERR_PTR(-EINVAL);
 
-	obj = kmalloc(sizeof *obj, GFP_KERNEL);
-	if (!obj)
-		return ERR_PTR(-ENOMEM);
-
-	init_uobj(&obj->uobject, cmd->user_handle, file->ucontext, &cq_lock_class);
-	down_write(&obj->uobject.mutex);
+	obj = (struct ib_ucq_object *)
+		uverbs_type_attrs_cq.type.ops->alloc_begin(&uverbs_type_attrs_cq.type,
+						       file->ucontext);
+	if (IS_ERR(obj))
+		return obj;
 
 	if (cmd->comp_channel >= 0) {
 		ev_file = ib_uverbs_lookup_comp_file(cmd->comp_channel);
@@ -1391,6 +1136,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 		}
 	}
 
+	obj->uobject.user_handle = cmd->user_handle;
 	obj->uverbs_file	   = file;
 	obj->comp_events_reported  = 0;
 	obj->async_events_reported = 0;
@@ -1403,8 +1149,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	if (cmd_sz > offsetof(typeof(*cmd), flags) + sizeof(cmd->flags))
 		attr.flags = cmd->flags;
 
-	cq = ib_dev->create_cq(ib_dev, &attr,
-					     file->ucontext, uhw);
+	cq = ib_dev->create_cq(ib_dev, &attr, file->ucontext, uhw);
 	if (IS_ERR(cq)) {
 		ret = PTR_ERR(cq);
 		goto err_file;
@@ -1418,10 +1163,6 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	atomic_set(&cq->usecnt, 0);
 
 	obj->uobject.object = cq;
-	ret = idr_add_uobj(&obj->uobject);
-	if (ret)
-		goto err_free;
-
 	memset(&resp, 0, sizeof resp);
 	resp.base.cq_handle = obj->uobject.id;
 	resp.base.cqe       = cq->cqe;
@@ -1433,20 +1174,11 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	if (ret)
 		goto err_cb;
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&obj->uobject.list, &file->ucontext->cq_list);
-	mutex_unlock(&file->mutex);
-
-	obj->uobject.live = 1;
-
-	up_write(&obj->uobject.mutex);
+	(&obj->uobject)->type->ops->alloc_commit(&obj->uobject);
 
 	return obj;
 
 err_cb:
-	idr_remove_uobj(&obj->uobject);
-
-err_free:
 	ib_destroy_cq(cq);
 
 err_file:
@@ -1454,7 +1186,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 		ib_uverbs_release_ucq(file, ev_file, obj);
 
 err:
-	put_uobj_write(&obj->uobject);
+	(&obj->uobject)->type->ops->alloc_abort(&obj->uobject);
 
 	return ERR_PTR(ret);
 }
@@ -1577,7 +1309,7 @@ ssize_t ib_uverbs_resize_cq(struct ib_uverbs_file *file,
 		   (unsigned long) cmd.response + sizeof resp,
 		   in_len - sizeof cmd, out_len - sizeof resp);
 
-	cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
+	cq = idr_read_cq(cmd.cq_handle, file->ucontext);
 	if (!cq)
 		return -EINVAL;
 
@@ -1639,7 +1371,7 @@ ssize_t ib_uverbs_poll_cq(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
+	cq = idr_read_cq(cmd.cq_handle, file->ucontext);
 	if (!cq)
 		return -EINVAL;
 
@@ -1686,7 +1418,7 @@ ssize_t ib_uverbs_req_notify_cq(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
+	cq = idr_read_cq(cmd.cq_handle, file->ucontext);
 	if (!cq)
 		return -EINVAL;
 
@@ -1714,36 +1446,29 @@ ssize_t ib_uverbs_destroy_cq(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(cmd.cq_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_cq.type.ops->lookup_get(&uverbs_type_attrs_cq.type,
+						     file->ucontext,
+						     cmd.cq_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
+
 	cq      = uobj->object;
 	ev_file = cq->cq_context;
 	obj     = container_of(cq->uobject, struct ib_ucq_object, uobject);
 
 	ret = ib_destroy_cq(cq);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
-
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
+	}
 
 	ib_uverbs_release_ucq(file, ev_file, obj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	memset(&resp, 0, sizeof resp);
 	resp.comp_events_reported  = obj->comp_events_reported;
 	resp.async_events_reported = obj->async_events_reported;
 
-	put_uobj(uobj);
-
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
 			 &resp, sizeof resp))
 		return -EFAULT;
@@ -1779,13 +1504,14 @@ static int create_qp(struct ib_uverbs_file *file,
 	if (cmd->qp_type == IB_QPT_RAW_PACKET && !capable(CAP_NET_RAW))
 		return -EPERM;
 
-	obj = kzalloc(sizeof *obj, GFP_KERNEL);
-	if (!obj)
-		return -ENOMEM;
+	obj = (struct ib_uqp_object *)
+		uverbs_type_attrs_qp.type.ops->alloc_begin(&uverbs_type_attrs_qp.type,
+						       file->ucontext);
+	if (IS_ERR(obj))
+		return PTR_ERR(obj);
+	obj->uxrcd = NULL;
+	obj->uevent.uobject.user_handle = cmd->user_handle;
 
-	init_uobj(&obj->uevent.uobject, cmd->user_handle, file->ucontext,
-		  &qp_lock_class);
-	down_write(&obj->uevent.uobject.mutex);
 	if (cmd_sz >= offsetof(typeof(*cmd), rwq_ind_tbl_handle) +
 		      sizeof(cmd->rwq_ind_tbl_handle) &&
 		      (cmd->comp_mask & IB_UVERBS_CREATE_QP_MASK_IND_TABLE)) {
@@ -1838,7 +1564,7 @@ static int create_qp(struct ib_uverbs_file *file,
 			if (!ind_tbl) {
 				if (cmd->recv_cq_handle != cmd->send_cq_handle) {
 					rcq = idr_read_cq(cmd->recv_cq_handle,
-							  file->ucontext, 0);
+							  file->ucontext);
 					if (!rcq) {
 						ret = -EINVAL;
 						goto err_put;
@@ -1848,7 +1574,7 @@ static int create_qp(struct ib_uverbs_file *file,
 		}
 
 		if (has_sq)
-			scq = idr_read_cq(cmd->send_cq_handle, file->ucontext, !!rcq);
+			scq = idr_read_cq(cmd->send_cq_handle, file->ucontext);
 		if (!ind_tbl)
 			rcq = rcq ?: scq;
 		pd  = idr_read_pd(cmd->pd_handle, file->ucontext);
@@ -1937,9 +1663,6 @@ static int create_qp(struct ib_uverbs_file *file,
 	qp->uobject = &obj->uevent.uobject;
 
 	obj->uevent.uobject.object = qp;
-	ret = idr_add_uobj(&obj->uevent.uobject);
-	if (ret)
-		goto err_destroy;
 
 	memset(&resp, 0, sizeof resp);
 	resp.base.qpn             = qp->qp_num;
@@ -1975,19 +1698,10 @@ static int create_qp(struct ib_uverbs_file *file,
 	if (ind_tbl)
 		put_rwq_indirection_table_read(ind_tbl);
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&obj->uevent.uobject.list, &file->ucontext->qp_list);
-	mutex_unlock(&file->mutex);
-
-	obj->uevent.uobject.live = 1;
-
-	up_write(&obj->uevent.uobject.mutex);
+	(&obj->uevent.uobject)->type->ops->alloc_commit(&obj->uevent.uobject);
 
 	return 0;
 err_cb:
-	idr_remove_uobj(&obj->uevent.uobject);
-
-err_destroy:
 	ib_destroy_qp(qp);
 
 err_put:
@@ -2004,7 +1718,7 @@ static int create_qp(struct ib_uverbs_file *file,
 	if (ind_tbl)
 		put_rwq_indirection_table_read(ind_tbl);
 
-	put_uobj_write(&obj->uevent.uobject);
+	(&obj->uevent.uobject)->type->ops->alloc_abort(&obj->uevent.uobject);
 	return ret;
 }
 
@@ -2140,12 +1854,11 @@ ssize_t ib_uverbs_open_qp(struct ib_uverbs_file *file,
 		   (unsigned long) cmd.response + sizeof resp,
 		   in_len - sizeof cmd, out_len - sizeof resp);
 
-	obj = kmalloc(sizeof *obj, GFP_KERNEL);
-	if (!obj)
-		return -ENOMEM;
-
-	init_uobj(&obj->uevent.uobject, cmd.user_handle, file->ucontext, &qp_lock_class);
-	down_write(&obj->uevent.uobject.mutex);
+	obj = (struct ib_uqp_object *)
+		uverbs_type_attrs_qp.type.ops->alloc_begin(&uverbs_type_attrs_qp.type,
+						       file->ucontext);
+	if (IS_ERR(obj))
+		return PTR_ERR(obj);
 
 	xrcd = idr_read_xrcd(cmd.pd_handle, file->ucontext, &xrcd_uobj);
 	if (!xrcd) {
@@ -2165,15 +1878,11 @@ ssize_t ib_uverbs_open_qp(struct ib_uverbs_file *file,
 	qp = ib_open_qp(xrcd, &attr);
 	if (IS_ERR(qp)) {
 		ret = PTR_ERR(qp);
-		goto err_put;
+		goto err_xrcd;
 	}
 
-	qp->uobject = &obj->uevent.uobject;
-
 	obj->uevent.uobject.object = qp;
-	ret = idr_add_uobj(&obj->uevent.uobject);
-	if (ret)
-		goto err_destroy;
+	obj->uevent.uobject.user_handle = cmd.user_handle;
 
 	memset(&resp, 0, sizeof resp);
 	resp.qpn       = qp->qp_num;
@@ -2182,32 +1891,25 @@ ssize_t ib_uverbs_open_qp(struct ib_uverbs_file *file,
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
 			 &resp, sizeof resp)) {
 		ret = -EFAULT;
-		goto err_remove;
+		goto err_destroy;
 	}
 
 	obj->uxrcd = container_of(xrcd_uobj, struct ib_uxrcd_object, uobject);
 	atomic_inc(&obj->uxrcd->refcnt);
+	qp->uobject = &obj->uevent.uobject;
 	put_xrcd_read(xrcd_uobj);
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&obj->uevent.uobject.list, &file->ucontext->qp_list);
-	mutex_unlock(&file->mutex);
-
-	obj->uevent.uobject.live = 1;
 
-	up_write(&obj->uevent.uobject.mutex);
+	(&obj->uevent.uobject)->type->ops->alloc_commit(&obj->uevent.uobject);
 
 	return in_len;
 
-err_remove:
-	idr_remove_uobj(&obj->uevent.uobject);
-
 err_destroy:
 	ib_destroy_qp(qp);
-
-err_put:
+err_xrcd:
 	put_xrcd_read(xrcd_uobj);
-	put_uobj_write(&obj->uevent.uobject);
+err_put:
+	(&obj->uevent.uobject)->type->ops->alloc_abort(&obj->uevent.uobject);
 	return ret;
 }
 
@@ -2496,40 +2198,34 @@ ssize_t ib_uverbs_destroy_qp(struct ib_uverbs_file *file,
 
 	memset(&resp, 0, sizeof resp);
 
-	uobj = idr_write_uobj(cmd.qp_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_qp.type.ops->lookup_get(&uverbs_type_attrs_qp.type,
+						     file->ucontext,
+						     cmd.qp_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
+
 	qp  = uobj->object;
 	obj = container_of(uobj, struct ib_uqp_object, uevent.uobject);
 
 	if (!list_empty(&obj->mcast_list)) {
-		put_uobj_write(uobj);
+		uobj->type->ops->lookup_put(uobj, true);
 		return -EBUSY;
 	}
 
 	ret = ib_destroy_qp(qp);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
+	}
 
 	if (obj->uxrcd)
 		atomic_dec(&obj->uxrcd->refcnt);
 
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
 	ib_uverbs_release_uevent(file, &obj->uevent);
 
 	resp.events_reported = obj->uevent.events_reported;
 
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
 			 &resp, sizeof resp))
@@ -2845,13 +2541,13 @@ ssize_t ib_uverbs_post_recv(struct ib_uverbs_file *file,
 	ret = qp->device->post_recv(qp->real_qp, wr, &bad_wr);
 
 	put_qp_read(qp);
-
-	if (ret)
+	if (ret) {
 		for (next = wr; next; next = next->next) {
 			++resp.bad_wr;
 			if (next == bad_wr)
 				break;
 		}
+	}
 
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
 			 &resp, sizeof resp))
@@ -2941,12 +2637,10 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
 		   (unsigned long)cmd.response + sizeof(resp),
 		   in_len - sizeof(cmd), out_len - sizeof(resp));
 
-	uobj = kmalloc(sizeof *uobj, GFP_KERNEL);
-	if (!uobj)
-		return -ENOMEM;
-
-	init_uobj(uobj, cmd.user_handle, file->ucontext, &ah_lock_class);
-	down_write(&uobj->mutex);
+	uobj = uverbs_type_attrs_ah.type.ops->alloc_begin(&uverbs_type_attrs_ah.type,
+						      file->ucontext);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	pd = idr_read_pd(cmd.pd_handle, file->ucontext);
 	if (!pd) {
@@ -2978,12 +2672,9 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
 	ah->pd      = pd;
 	atomic_inc(&pd->usecnt);
 	ah->uobject  = uobj;
+	uobj->user_handle = cmd.user_handle;
 	uobj->object = ah;
 
-	ret = idr_add_uobj(uobj);
-	if (ret)
-		goto err_destroy;
-
 	resp.ah_handle = uobj->id;
 
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
@@ -2993,28 +2684,18 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
 	}
 
 	put_pd_read(pd);
-
-	mutex_lock(&file->mutex);
-	list_add_tail(&uobj->list, &file->ucontext->ah_list);
-	mutex_unlock(&file->mutex);
-
-	uobj->live = 1;
-
-	up_write(&uobj->mutex);
+	uobj->type->ops->alloc_commit(uobj);
 
 	return in_len;
 
 err_copy:
-	idr_remove_uobj(uobj);
-
-err_destroy:
 	ib_destroy_ah(ah);
 
 err_put:
 	put_pd_read(pd);
 
 err:
-	put_uobj_write(uobj);
+	uobj->type->ops->alloc_abort(uobj);
 	return ret;
 }
 
@@ -3030,28 +2711,21 @@ ssize_t ib_uverbs_destroy_ah(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(cmd.ah_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_ah.type.ops->lookup_get(&uverbs_type_attrs_ah.type,
+						     file->ucontext,
+						     cmd.ah_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
+
 	ah = uobj->object;
 
 	ret = ib_destroy_ah(ah);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
+	}
 
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
-
+	uobj->type->ops->destroy_commit(uobj);
 	return in_len;
 }
 
@@ -3069,7 +2743,7 @@ ssize_t ib_uverbs_attach_mcast(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	qp = idr_write_qp(cmd.qp_handle, file->ucontext);
+	qp = idr_read_qp(cmd.qp_handle, file->ucontext);
 	if (!qp)
 		return -EINVAL;
 
@@ -3098,7 +2772,7 @@ ssize_t ib_uverbs_attach_mcast(struct ib_uverbs_file *file,
 		kfree(mcast);
 
 out_put:
-	put_qp_write(qp);
+	put_qp_read(qp);
 
 	return ret ? ret : in_len;
 }
@@ -3117,16 +2791,16 @@ ssize_t ib_uverbs_detach_mcast(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	qp = idr_write_qp(cmd.qp_handle, file->ucontext);
+	qp = idr_read_qp(cmd.qp_handle, file->ucontext);
 	if (!qp)
 		return -EINVAL;
 
+	obj = container_of(qp->uobject, struct ib_uqp_object, uevent.uobject);
+
 	ret = ib_detach_mcast(qp, (union ib_gid *) cmd.gid, cmd.mlid);
 	if (ret)
 		goto out_put;
 
-	obj = container_of(qp->uobject, struct ib_uqp_object, uevent.uobject);
-
 	list_for_each_entry(mcast, &obj->mcast_list, list)
 		if (cmd.mlid == mcast->lid &&
 		    !memcmp(cmd.gid, mcast->gid.raw, sizeof mcast->gid.raw)) {
@@ -3136,8 +2810,7 @@ ssize_t ib_uverbs_detach_mcast(struct ib_uverbs_file *file,
 		}
 
 out_put:
-	put_qp_write(qp);
-
+	put_qp_read(qp);
 	return ret ? ret : in_len;
 }
 
@@ -3298,20 +2971,19 @@ int ib_uverbs_ex_create_wq(struct ib_uverbs_file *file,
 	if (cmd.comp_mask)
 		return -EOPNOTSUPP;
 
-	obj = kmalloc(sizeof(*obj), GFP_KERNEL);
-	if (!obj)
-		return -ENOMEM;
+	obj = (struct ib_uwq_object *)
+		uverbs_type_attrs_wq.type.ops->alloc_begin(&uverbs_type_attrs_wq.type,
+						       file->ucontext);
+	if (IS_ERR(obj))
+		return PTR_ERR(obj);
 
-	init_uobj(&obj->uevent.uobject, cmd.user_handle, file->ucontext,
-		  &wq_lock_class);
-	down_write(&obj->uevent.uobject.mutex);
 	pd  = idr_read_pd(cmd.pd_handle, file->ucontext);
 	if (!pd) {
 		err = -EINVAL;
 		goto err_uobj;
 	}
 
-	cq = idr_read_cq(cmd.cq_handle, file->ucontext, 0);
+	cq = idr_read_cq(cmd.cq_handle, file->ucontext);
 	if (!cq) {
 		err = -EINVAL;
 		goto err_put_pd;
@@ -3343,9 +3015,6 @@ int ib_uverbs_ex_create_wq(struct ib_uverbs_file *file,
 	atomic_inc(&cq->usecnt);
 	wq->uobject = &obj->uevent.uobject;
 	obj->uevent.uobject.object = wq;
-	err = idr_add_uobj(&obj->uevent.uobject);
-	if (err)
-		goto destroy_wq;
 
 	memset(&resp, 0, sizeof(resp));
 	resp.wq_handle = obj->uevent.uobject.id;
@@ -3360,25 +3029,17 @@ int ib_uverbs_ex_create_wq(struct ib_uverbs_file *file,
 
 	put_pd_read(pd);
 	put_cq_read(cq);
-
-	mutex_lock(&file->mutex);
-	list_add_tail(&obj->uevent.uobject.list, &file->ucontext->wq_list);
-	mutex_unlock(&file->mutex);
-
-	obj->uevent.uobject.live = 1;
-	up_write(&obj->uevent.uobject.mutex);
+	(&obj->uevent.uobject)->type->ops->alloc_commit(&obj->uevent.uobject);
 	return 0;
 
 err_copy:
-	idr_remove_uobj(&obj->uevent.uobject);
-destroy_wq:
 	ib_destroy_wq(wq);
 err_put_cq:
 	put_cq_read(cq);
 err_put_pd:
 	put_pd_read(pd);
 err_uobj:
-	put_uobj_write(&obj->uevent.uobject);
+	(&obj->uevent.uobject)->type->ops->alloc_abort(&obj->uevent.uobject);
 
 	return err;
 }
@@ -3419,30 +3080,23 @@ int ib_uverbs_ex_destroy_wq(struct ib_uverbs_file *file,
 		return -EOPNOTSUPP;
 
 	resp.response_length = required_resp_len;
-	uobj = idr_write_uobj(cmd.wq_handle,
-			      file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_ah.type.ops->lookup_get(&uverbs_type_attrs_ah.type,
+						     file->ucontext,
+						     cmd.wq_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
 	wq = uobj->object;
 	obj = container_of(uobj, struct ib_uwq_object, uevent.uobject);
 	ret = ib_destroy_wq(wq);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
-
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
+	}
 
 	ib_uverbs_release_uevent(file, &obj->uevent);
 	resp.events_reported = obj->uevent.events_reported;
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	ret = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	if (ret)
@@ -3575,14 +3229,13 @@ int ib_uverbs_ex_create_rwq_ind_table(struct ib_uverbs_file *file,
 		wqs[num_read_wqs] = wq;
 	}
 
-	uobj = kmalloc(sizeof(*uobj), GFP_KERNEL);
-	if (!uobj) {
-		err = -ENOMEM;
+	uobj = uverbs_type_attrs_rwq_ind_table.type.ops->alloc_begin(&uverbs_type_attrs_rwq_ind_table.type,
+								 file->ucontext);
+	if (IS_ERR(uobj)) {
+		err = PTR_ERR(uobj);
 		goto put_wqs;
 	}
 
-	init_uobj(uobj, 0, file->ucontext, &rwq_ind_table_lock_class);
-	down_write(&uobj->mutex);
 	init_attr.log_ind_tbl_size = cmd.log_ind_tbl_size;
 	init_attr.ind_tbl = wqs;
 	rwq_ind_tbl = ib_dev->create_rwq_ind_table(ib_dev, &init_attr, uhw);
@@ -3602,10 +3255,6 @@ int ib_uverbs_ex_create_rwq_ind_table(struct ib_uverbs_file *file,
 	for (i = 0; i < num_wq_handles; i++)
 		atomic_inc(&wqs[i]->usecnt);
 
-	err = idr_add_uobj(uobj);
-	if (err)
-		goto destroy_ind_tbl;
-
 	resp.ind_tbl_handle = uobj->id;
 	resp.ind_tbl_num = rwq_ind_tbl->ind_tbl_num;
 	resp.response_length = required_resp_len;
@@ -3620,21 +3269,13 @@ int ib_uverbs_ex_create_rwq_ind_table(struct ib_uverbs_file *file,
 	for (j = 0; j < num_read_wqs; j++)
 		put_wq_read(wqs[j]);
 
-	mutex_lock(&file->mutex);
-	list_add_tail(&uobj->list, &file->ucontext->rwq_ind_tbl_list);
-	mutex_unlock(&file->mutex);
-
-	uobj->live = 1;
-
-	up_write(&uobj->mutex);
+	uobj->type->ops->alloc_commit(uobj);
 	return 0;
 
 err_copy:
-	idr_remove_uobj(uobj);
-destroy_ind_tbl:
 	ib_destroy_rwq_ind_table(rwq_ind_tbl);
 err_uobj:
-	put_uobj_write(uobj);
+	uobj->type->ops->alloc_abort(uobj);
 put_wqs:
 	for (j = 0; j < num_read_wqs; j++)
 		put_wq_read(wqs[j]);
@@ -3673,29 +3314,23 @@ int ib_uverbs_ex_destroy_rwq_ind_table(struct ib_uverbs_file *file,
 	if (cmd.comp_mask)
 		return -EOPNOTSUPP;
 
-	uobj = idr_write_uobj(cmd.ind_tbl_handle,
-			      file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_rwq_ind_table.type.ops->lookup_get(&uverbs_type_attrs_rwq_ind_table.type,
+								file->ucontext,
+								cmd.ind_tbl_handle,
+								true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
+
 	rwq_ind_tbl = uobj->object;
 	ind_tbl = rwq_ind_tbl->ind_tbl;
 
 	ret = ib_destroy_rwq_ind_table(rwq_ind_tbl);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
+	}
 
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 	kfree(ind_tbl);
 	return ret;
 }
@@ -3771,13 +3406,12 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
 		kern_flow_attr = &cmd.flow_attr;
 	}
 
-	uobj = kmalloc(sizeof(*uobj), GFP_KERNEL);
-	if (!uobj) {
-		err = -ENOMEM;
+	uobj = uverbs_type_attrs_flow.type.ops->alloc_begin(&uverbs_type_attrs_flow.type,
+							file->ucontext);
+	if (IS_ERR(uobj)) {
+		err = PTR_ERR(uobj);
 		goto err_free_attr;
 	}
-	init_uobj(uobj, 0, file->ucontext, &rule_lock_class);
-	down_write(&uobj->mutex);
 
 	qp = idr_read_qp(cmd.qp_handle, file->ucontext);
 	if (!qp) {
@@ -3828,10 +3462,6 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
 	flow_id->uobject = uobj;
 	uobj->object = flow_id;
 
-	err = idr_add_uobj(uobj);
-	if (err)
-		goto destroy_flow;
-
 	memset(&resp, 0, sizeof(resp));
 	resp.flow_handle = uobj->id;
 
@@ -3841,27 +3471,19 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
 		goto err_copy;
 
 	put_qp_read(qp);
-	mutex_lock(&file->mutex);
-	list_add_tail(&uobj->list, &file->ucontext->rule_list);
-	mutex_unlock(&file->mutex);
-
-	uobj->live = 1;
-
-	up_write(&uobj->mutex);
+	uobj->type->ops->alloc_commit(uobj);
 	kfree(flow_attr);
 	if (cmd.flow_attr.num_of_specs)
 		kfree(kern_flow_attr);
 	return 0;
 err_copy:
-	idr_remove_uobj(uobj);
-destroy_flow:
 	ib_destroy_flow(flow_id);
 err_free:
 	kfree(flow_attr);
 err_put:
 	put_qp_read(qp);
 err_uobj:
-	put_uobj_write(uobj);
+	uobj->type->ops->alloc_abort(uobj);
 err_free_attr:
 	if (cmd.flow_attr.num_of_specs)
 		kfree(kern_flow_attr);
@@ -3888,25 +3510,19 @@ int ib_uverbs_ex_destroy_flow(struct ib_uverbs_file *file,
 	if (cmd.comp_mask)
 		return -EINVAL;
 
-	uobj = idr_write_uobj(cmd.flow_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_flow.type.ops->lookup_get(&uverbs_type_attrs_flow.type,
+						       file->ucontext,
+						       cmd.flow_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
+
 	flow_id = uobj->object;
 
 	ret = ib_destroy_flow(flow_id);
 	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
-	put_uobj(uobj);
-
+		uobj->type->ops->destroy_commit(uobj);
+	else
+		uobj->type->ops->lookup_put(uobj, true);
 	return ret;
 }
 
@@ -3923,12 +3539,11 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	struct ib_srq_init_attr          attr;
 	int ret;
 
-	obj = kmalloc(sizeof *obj, GFP_KERNEL);
-	if (!obj)
-		return -ENOMEM;
-
-	init_uobj(&obj->uevent.uobject, cmd->user_handle, file->ucontext, &srq_lock_class);
-	down_write(&obj->uevent.uobject.mutex);
+	obj = (struct ib_usrq_object *)
+		uverbs_type_attrs_srq.type.ops->alloc_begin(&uverbs_type_attrs_srq.type,
+							file->ucontext);
+	if (IS_ERR(obj))
+		return PTR_ERR(obj);
 
 	if (cmd->srq_type == IB_SRQT_XRC) {
 		attr.ext.xrc.xrcd  = idr_read_xrcd(cmd->xrcd_handle, file->ucontext, &xrcd_uobj);
@@ -3940,7 +3555,7 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 		obj->uxrcd = container_of(xrcd_uobj, struct ib_uxrcd_object, uobject);
 		atomic_inc(&obj->uxrcd->refcnt);
 
-		attr.ext.xrc.cq  = idr_read_cq(cmd->cq_handle, file->ucontext, 0);
+		attr.ext.xrc.cq  = idr_read_cq(cmd->cq_handle, file->ucontext);
 		if (!attr.ext.xrc.cq) {
 			ret = -EINVAL;
 			goto err_put_xrcd;
@@ -3987,9 +3602,7 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	atomic_set(&srq->usecnt, 0);
 
 	obj->uevent.uobject.object = srq;
-	ret = idr_add_uobj(&obj->uevent.uobject);
-	if (ret)
-		goto err_destroy;
+	obj->uevent.uobject.user_handle = cmd->user_handle;
 
 	memset(&resp, 0, sizeof resp);
 	resp.srq_handle = obj->uevent.uobject.id;
@@ -4005,25 +3618,15 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 	}
 
 	if (cmd->srq_type == IB_SRQT_XRC) {
-		put_uobj_read(xrcd_uobj);
+		put_xrcd_read(xrcd_uobj);
 		put_cq_read(attr.ext.xrc.cq);
 	}
 	put_pd_read(pd);
-
-	mutex_lock(&file->mutex);
-	list_add_tail(&obj->uevent.uobject.list, &file->ucontext->srq_list);
-	mutex_unlock(&file->mutex);
-
-	obj->uevent.uobject.live = 1;
-
-	up_write(&obj->uevent.uobject.mutex);
+	(&obj->uevent.uobject)->type->ops->alloc_commit(&obj->uevent.uobject);
 
 	return 0;
 
 err_copy:
-	idr_remove_uobj(&obj->uevent.uobject);
-
-err_destroy:
 	ib_destroy_srq(srq);
 
 err_put:
@@ -4036,11 +3639,11 @@ static int __uverbs_create_xsrq(struct ib_uverbs_file *file,
 err_put_xrcd:
 	if (cmd->srq_type == IB_SRQT_XRC) {
 		atomic_dec(&obj->uxrcd->refcnt);
-		put_uobj_read(xrcd_uobj);
+		put_xrcd_read(xrcd_uobj);
 	}
 
 err:
-	put_uobj_write(&obj->uevent.uobject);
+	(&obj->uevent.uobject)->type->ops->alloc_abort(&obj->uevent.uobject);
 	return ret;
 }
 
@@ -4197,39 +3800,33 @@ ssize_t ib_uverbs_destroy_srq(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	uobj = idr_write_uobj(cmd.srq_handle, file->ucontext);
-	if (!uobj)
-		return -EINVAL;
+	uobj = uverbs_type_attrs_srq.type.ops->lookup_get(&uverbs_type_attrs_srq.type,
+						      file->ucontext,
+						      cmd.srq_handle, true);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
+
 	srq = uobj->object;
 	obj = container_of(uobj, struct ib_uevent_object, uobject);
 	srq_type = srq->srq_type;
 
 	ret = ib_destroy_srq(srq);
-	if (!ret)
-		uobj->live = 0;
-
-	put_uobj_write(uobj);
-
-	if (ret)
+	if (ret) {
+		uobj->type->ops->lookup_put(uobj, true);
 		return ret;
+	}
 
 	if (srq_type == IB_SRQT_XRC) {
 		us = container_of(obj, struct ib_usrq_object, uevent);
 		atomic_dec(&us->uxrcd->refcnt);
 	}
 
-	idr_remove_uobj(uobj);
-
-	mutex_lock(&file->mutex);
-	list_del(&uobj->list);
-	mutex_unlock(&file->mutex);
-
 	ib_uverbs_release_uevent(file, obj);
 
 	memset(&resp, 0, sizeof resp);
 	resp.events_reported = obj->events_reported;
 
-	put_uobj(uobj);
+	uobj->type->ops->destroy_commit(uobj);
 
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
 			 &resp, sizeof resp))
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index ba8b1eb..0eb4538 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -51,6 +51,7 @@
 #include <rdma/ib.h>
 
 #include "uverbs.h"
+#include "rdma_core.h"
 
 MODULE_AUTHOR("Roland Dreier");
 MODULE_DESCRIPTION("InfiniBand userspace verbs access");
@@ -213,122 +214,11 @@ void ib_uverbs_detach_umcast(struct ib_qp *qp,
 }
 
 static int ib_uverbs_cleanup_ucontext(struct ib_uverbs_file *file,
-				      struct ib_ucontext *context)
+				      struct ib_ucontext *context,
+				      bool device_removed)
 {
-	struct ib_uobject *uobj, *tmp;
-
 	context->closing = 1;
-
-	list_for_each_entry_safe(uobj, tmp, &context->ah_list, list) {
-		struct ib_ah *ah = uobj->object;
-
-		idr_remove_uobj(uobj);
-		ib_destroy_ah(ah);
-		kfree(uobj);
-	}
-
-	/* Remove MWs before QPs, in order to support type 2A MWs. */
-	list_for_each_entry_safe(uobj, tmp, &context->mw_list, list) {
-		struct ib_mw *mw = uobj->object;
-
-		idr_remove_uobj(uobj);
-		uverbs_dealloc_mw(mw);
-		kfree(uobj);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->rule_list, list) {
-		struct ib_flow *flow_id = uobj->object;
-
-		idr_remove_uobj(uobj);
-		ib_destroy_flow(flow_id);
-		kfree(uobj);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->qp_list, list) {
-		struct ib_qp *qp = uobj->object;
-		struct ib_uqp_object *uqp =
-			container_of(uobj, struct ib_uqp_object, uevent.uobject);
-
-		idr_remove_uobj(uobj);
-		if (qp == qp->real_qp)
-			ib_uverbs_detach_umcast(qp, uqp);
-		ib_destroy_qp(qp);
-		ib_uverbs_release_uevent(file, &uqp->uevent);
-		kfree(uqp);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->rwq_ind_tbl_list, list) {
-		struct ib_rwq_ind_table *rwq_ind_tbl = uobj->object;
-		struct ib_wq **ind_tbl = rwq_ind_tbl->ind_tbl;
-
-		idr_remove_uobj(uobj);
-		ib_destroy_rwq_ind_table(rwq_ind_tbl);
-		kfree(ind_tbl);
-		kfree(uobj);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->wq_list, list) {
-		struct ib_wq *wq = uobj->object;
-		struct ib_uwq_object *uwq =
-			container_of(uobj, struct ib_uwq_object, uevent.uobject);
-
-		idr_remove_uobj(uobj);
-		ib_destroy_wq(wq);
-		ib_uverbs_release_uevent(file, &uwq->uevent);
-		kfree(uwq);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->srq_list, list) {
-		struct ib_srq *srq = uobj->object;
-		struct ib_uevent_object *uevent =
-			container_of(uobj, struct ib_uevent_object, uobject);
-
-		idr_remove_uobj(uobj);
-		ib_destroy_srq(srq);
-		ib_uverbs_release_uevent(file, uevent);
-		kfree(uevent);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->cq_list, list) {
-		struct ib_cq *cq = uobj->object;
-		struct ib_uverbs_event_file *ev_file = cq->cq_context;
-		struct ib_ucq_object *ucq =
-			container_of(uobj, struct ib_ucq_object, uobject);
-
-		idr_remove_uobj(uobj);
-		ib_destroy_cq(cq);
-		ib_uverbs_release_ucq(file, ev_file, ucq);
-		kfree(ucq);
-	}
-
-	list_for_each_entry_safe(uobj, tmp, &context->mr_list, list) {
-		struct ib_mr *mr = uobj->object;
-
-		idr_remove_uobj(uobj);
-		ib_dereg_mr(mr);
-		kfree(uobj);
-	}
-
-	mutex_lock(&file->device->xrcd_tree_mutex);
-	list_for_each_entry_safe(uobj, tmp, &context->xrcd_list, list) {
-		struct ib_xrcd *xrcd = uobj->object;
-		struct ib_uxrcd_object *uxrcd =
-			container_of(uobj, struct ib_uxrcd_object, uobject);
-
-		idr_remove_uobj(uobj);
-		ib_uverbs_dealloc_xrcd(file->device, xrcd);
-		kfree(uxrcd);
-	}
-	mutex_unlock(&file->device->xrcd_tree_mutex);
-
-	list_for_each_entry_safe(uobj, tmp, &context->pd_list, list) {
-		struct ib_pd *pd = uobj->object;
-
-		idr_remove_uobj(uobj);
-		ib_dealloc_pd(pd);
-		kfree(uobj);
-	}
-
+	uverbs_cleanup_ucontext(context, device_removed);
 	put_pid(context->tgid);
 
 	return context->device->dealloc_ucontext(context);
@@ -570,7 +460,7 @@ void ib_uverbs_qp_event_handler(struct ib_event *event, void *context_ptr)
 	struct ib_uevent_object *uobj;
 
 	/* for XRC target qp's, check that qp is live */
-	if (!event->element.qp->uobject || !event->element.qp->uobject->live)
+	if (!event->element.qp->uobject)
 		return;
 
 	uobj = container_of(event->element.qp->uobject,
@@ -988,7 +878,7 @@ static int ib_uverbs_close(struct inode *inode, struct file *filp)
 
 	mutex_lock(&file->cleanup_mutex);
 	if (file->ucontext) {
-		ib_uverbs_cleanup_ucontext(file, file->ucontext);
+		ib_uverbs_cleanup_ucontext(file, file->ucontext, false);
 		file->ucontext = NULL;
 	}
 	mutex_unlock(&file->cleanup_mutex);
@@ -1238,7 +1128,7 @@ static void ib_uverbs_free_hw_resources(struct ib_uverbs_device *uverbs_dev,
 			 * (e.g mmput).
 			 */
 			ib_dev->disassociate_ucontext(ucontext);
-			ib_uverbs_cleanup_ucontext(file, ucontext);
+			ib_uverbs_cleanup_ucontext(file, ucontext, true);
 		}
 
 		mutex_lock(&uverbs_dev->lists_mutex);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 4a2a0fc..7ddb08f 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1334,17 +1334,6 @@ struct ib_fmr_attr {
 struct ib_ucontext {
 	struct ib_device       *device;
 	struct ib_uverbs_file  *ufile;
-	struct list_head	pd_list;
-	struct list_head	mr_list;
-	struct list_head	mw_list;
-	struct list_head	cq_list;
-	struct list_head	qp_list;
-	struct list_head	srq_list;
-	struct list_head	ah_list;
-	struct list_head	xrcd_list;
-	struct list_head	rule_list;
-	struct list_head	wq_list;
-	struct list_head	rwq_ind_tbl_list;
 	int			closing;
 
 	/* locking the uobjects_list */
@@ -1377,10 +1366,8 @@ struct ib_uobject {
 	struct list_head	list;		/* link to context's list */
 	int			id;		/* index into kernel idr */
 	struct kref		ref;
-	struct rw_semaphore	mutex;		/* protects .live */
 	struct rw_semaphore	currently_used;	/* protects exclusive access */
 	struct rcu_head		rcu;		/* kfree_rcu() overhead */
-	int			live;
 
 	const struct uverbs_obj_type *type;
 };
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 4/7] IB/core: Change idr objects to use the new schema

[Jason Gunthorpe]

On Wed, Feb 01, 2017 at 02:39:02PM +0200, Matan Barak wrote:

> +#define idr_get_xxxx(_type, _write, _handle, _context) ({		\
> +	const struct uverbs_obj_idr_type * const type =			\
> +			&uverbs_type_attrs_##_type;			\
> +	struct ib_uobject *uobj = type->type.ops->lookup_get(&type->type, \
> +					_context, _handle, _write);	\
> +									\
> +	IS_ERR(uobj) ? NULL : uobj->object; })

I think you should follow the usual convention and have a full
suite of accessor helpers in a header. This header could even be in
patch #2

static struct ib_uobject *__uobj_get(const struct uverbs_obj_type *type, bool write, struct ib_ucontext *ucontext, int id)
{
  [..]
}
#define uobj_get_read(_type,...) ((struct ib_##_type *)_uobj_get(&uverbs_type_attrs_##_type, false, __VA_ARGS__))
#define uobj_get_write(_type,...) ((struct ib_##_type *)_uobj_get(&uverbs_type_attrs_##_type, false, __VA_ARGS__))

static inline void uobj_put_write(struct ib_uobject *uobj)
{
    uobj->type->ops->lookup_put(uobj, true);
}

static inline void uobj_put_read(struct ib_uobject *uobj)
{
    uobj->type->ops->lookup_put(uobj, false);
}

static inline struct ib_uobject *__uobj_alloc(const struct uverbs_obj_type *type,struct ib_ucontext *ucontext)
{
   return type->alloc_begin(type, ucontext);
}

#define uobj_alloc_begin(_type, ...) ((struct ib_##_type *)_uobj_alloc(&uverbs_type_attrs_##_type, __VA_ARGS__))

and so on

>  static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
>  {
> -	return idr_read_obj(pd_handle, context, 0);
> +	return idr_get_xxxx(pd, false, pd_handle, context);
>  }

And you can just totally drop all of these inlines and use

  uobj_get_read(pd, pd_handle, context);

At the call site.

>  static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context,
>  				     struct ib_uobject **uobj)
>  {
> -	*uobj = idr_read_uobj(xrcd_handle, context, 0);
> -	return *uobj ? (*uobj)->object : NULL;
> +	*uobj = uverbs_type_attrs_xrcd.type.ops->lookup_get(&uverbs_type_attrs_xrcd.type,
> +							context, xrcd_handle,
> +							false);
> +	return IS_ERR(*uobj) ? NULL : (*uobj)->object;

Why didn't this use idr_get_xxx ? Why is it so weird?

>  
> @@ -621,9 +439,11 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
>  	if (copy_from_user(&cmd, buf, sizeof cmd))
>  		return -EFAULT;
>  
> -	uobj = idr_write_uobj(cmd.pd_handle, file->ucontext);
> -	if (!uobj)
> -		return -EINVAL;
> +	uobj = uverbs_type_attrs_pd.type.ops->lookup_get(&uverbs_type_attrs_pd.type,
> +						     file->ucontext,
> +						     cmd.pd_handle, true);
> +	if (IS_ERR(uobj))
> +		return PTR_ERR(uobj);
>  	pd = uobj->object;
>  
>  	if (atomic_read(&pd->usecnt)) {
> @@ -636,21 +456,12 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
>  	if (ret)
>  		goto err_put;
>  
> -	uobj->live = 0;
> -	put_uobj_write(uobj);
> -
> -	idr_remove_uobj(uobj);
> -
> -	mutex_lock(&file->mutex);
> -	list_del(&uobj->list);
> -	mutex_unlock(&file->mutex);
> -
> -	put_uobj(uobj);
> +	uobj->type->ops->destroy_commit(uobj);

And here I think you should move the pd->device_dealloc_pd into
destroy_commit by having it call what you called hot_plug, and turn
this simply into:

	return uobj_remove_or_put(uobj, RDMA_REMOVE_DESTROY);

static inline int uobj_remove_or_put(struct ib_uobject *uobj, enum rdma_remove_reason why)
{
    int rc = uobj->type->ops->remove(uobj, why);
    if (!rc)
        uverbs_uobject_put(uobj);

    /* If we are using a forced destruction mode then the driver is
        not permitted to fail. */
    WARN_ON(!rc && why != RDMA_REMOVE_DESTROY);
    return rc;
}

And generally use this pattern for all the destroy verbs.

I didn't study this patch closely since it will change a lot, but I
think this:

 4 files changed, 260 insertions(+), 788 deletions(-)

Pretty much confirms this is the right overall approach.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 4/7] IB/core: Change idr objects to use the new schema

[Matan Barak]

On Fri, Feb 10, 2017 at 10:30 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Feb 01, 2017 at 02:39:02PM +0200, Matan Barak wrote:
>
>> +#define idr_get_xxxx(_type, _write, _handle, _context) ({            \
>> +     const struct uverbs_obj_idr_type * const type =                 \
>> +                     &uverbs_type_attrs_##_type;                     \
>> +     struct ib_uobject *uobj = type->type.ops->lookup_get(&type->type, \
>> +                                     _context, _handle, _write);     \
>> +                                                                     \
>> +     IS_ERR(uobj) ? NULL : uobj->object; })
>
> I think you should follow the usual convention and have a full
> suite of accessor helpers in a header. This header could even be in
> patch #2
>

I prefer to put it in this patch, but I'll introduce them in
uverbs_std_types.h as you suggested.

> static struct ib_uobject *__uobj_get(const struct uverbs_obj_type *type, bool write, struct ib_ucontext *ucontext, int id)
> {
>   [..]
> }
> #define uobj_get_read(_type,...) ((struct ib_##_type *)_uobj_get(&uverbs_type_attrs_##_type, false, __VA_ARGS__))
> #define uobj_get_write(_type,...) ((struct ib_##_type *)_uobj_get(&uverbs_type_attrs_##_type, false, __VA_ARGS__))
>
> static inline void uobj_put_write(struct ib_uobject *uobj)
> {
>     uobj->type->ops->lookup_put(uobj, true);
> }
>
> static inline void uobj_put_read(struct ib_uobject *uobj)
> {
>     uobj->type->ops->lookup_put(uobj, false);
> }
>
> static inline struct ib_uobject *__uobj_alloc(const struct uverbs_obj_type *type,struct ib_ucontext *ucontext)
> {
>    return type->alloc_begin(type, ucontext);
> }
>
> #define uobj_alloc_begin(_type, ...) ((struct ib_##_type *)_uobj_alloc(&uverbs_type_attrs_##_type, __VA_ARGS__))
>
> and so on
>

I'll add something similar.

>>  static struct ib_pd *idr_read_pd(int pd_handle, struct ib_ucontext *context)
>>  {
>> -     return idr_read_obj(pd_handle, context, 0);
>> +     return idr_get_xxxx(pd, false, pd_handle, context);
>>  }
>
> And you can just totally drop all of these inlines and use
>
>   uobj_get_read(pd, pd_handle, context);
>
> At the call site.
>

Sure

>>  static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context,
>>                                    struct ib_uobject **uobj)
>>  {
>> -     *uobj = idr_read_uobj(xrcd_handle, context, 0);
>> -     return *uobj ? (*uobj)->object : NULL;
>> +     *uobj = uverbs_type_attrs_xrcd.type.ops->lookup_get(&uverbs_type_attrs_xrcd.type,
>> +                                                     context, xrcd_handle,
>> +                                                     false);
>> +     return IS_ERR(*uobj) ? NULL : (*uobj)->object;
>
> Why didn't this use idr_get_xxx ? Why is it so weird?
>

This was the original code as well and I wanted to refactor as closely as I can.
However, this might be a good time to cleanup this as well.

>>
>> @@ -621,9 +439,11 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
>>       if (copy_from_user(&cmd, buf, sizeof cmd))
>>               return -EFAULT;
>>
>> -     uobj = idr_write_uobj(cmd.pd_handle, file->ucontext);
>> -     if (!uobj)
>> -             return -EINVAL;
>> +     uobj = uverbs_type_attrs_pd.type.ops->lookup_get(&uverbs_type_attrs_pd.type,
>> +                                                  file->ucontext,
>> +                                                  cmd.pd_handle, true);
>> +     if (IS_ERR(uobj))
>> +             return PTR_ERR(uobj);
>>       pd = uobj->object;
>>
>>       if (atomic_read(&pd->usecnt)) {
>> @@ -636,21 +456,12 @@ ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file,
>>       if (ret)
>>               goto err_put;
>>
>> -     uobj->live = 0;
>> -     put_uobj_write(uobj);
>> -
>> -     idr_remove_uobj(uobj);
>> -
>> -     mutex_lock(&file->mutex);
>> -     list_del(&uobj->list);
>> -     mutex_unlock(&file->mutex);
>> -
>> -     put_uobj(uobj);
>> +     uobj->type->ops->destroy_commit(uobj);
>
> And here I think you should move the pd->device_dealloc_pd into
> destroy_commit by having it call what you called hot_plug, and turn
> this simply into:
>
>         return uobj_remove_or_put(uobj, RDMA_REMOVE_DESTROY);
>
> static inline int uobj_remove_or_put(struct ib_uobject *uobj, enum rdma_remove_reason why)
> {
>     int rc = uobj->type->ops->remove(uobj, why);
>     if (!rc)
>         uverbs_uobject_put(uobj);
>
>     /* If we are using a forced destruction mode then the driver is
>         not permitted to fail. */
>     WARN_ON(!rc && why != RDMA_REMOVE_DESTROY);
>     return rc;
> }
>
> And generally use this pattern for all the destroy verbs.
>

I prefer to keep this separation. In handlers code, the handler is
responsible to destroy
the actual object and to just call remove_commit or lookup_put (via
the accessor functions of course).
There's a clear separation here between the uobjects management and
the objects themselves.
It's also more symmetrical - the user created this pd and [s]he shall
destroy it.

> I didn't study this patch closely since it will change a lot, but I
> think this:
>
>  4 files changed, 260 insertions(+), 788 deletions(-)
>
> Pretty much confirms this is the right overall approach.
>
> Jason

Thanks for the review.

Matan

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 4/7] IB/core: Change idr objects to use the new schema

[Jason Gunthorpe]

On Wed, Feb 15, 2017 at 03:45:00PM +0200, Matan Barak wrote:

> > And here I think you should move the pd->device_dealloc_pd into
> > destroy_commit by having it call what you called hot_plug, and turn
> > this simply into:
> >
> >         return uobj_remove_or_put(uobj, RDMA_REMOVE_DESTROY);
> >
> > static inline int uobj_remove_or_put(struct ib_uobject *uobj, enum rdma_remove_reason why)
> > {
> >     int rc = uobj->type->ops->remove(uobj, why);
> >     if (!rc)
> >         uverbs_uobject_put(uobj);
> >
> >     /* If we are using a forced destruction mode then the driver is
> >         not permitted to fail. */
> >     WARN_ON(!rc && why != RDMA_REMOVE_DESTROY);
> >     return rc;
> > }
> >
> > And generally use this pattern for all the destroy verbs.
> >
> 
> I prefer to keep this separation. In handlers code, the handler is
> responsible to destroy the actual object and to just call
> remove_commit or lookup_put (via the accessor functions of course).
> There's a clear separation here between the uobjects management and
> the objects themselves.  It's also more symmetrical - the user
> created this pd and [s]he shall destroy it.

unless the context is closed, or hot unplug is done, this argument
makes no sense.

The above is very clear and forces the needed sanity on the whole
broken destory system

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 5/7] IB/core: Add lock to multicast handlers

[Matan Barak]

When two handlers used the same object in the old schema, we blocked
the process in the kernel. The new schema just returns -EBUSY. This
could lead to different behaviour in applications between the old
schema and the new schema. In most cases, using such handlers
concurrently could lead to crashing the process. For example, if
thread A destroys a QP and thread B modifies it, we could have the
destruction happens before the modification. In this case, we are
accessing freed memory which could lead to crashing the process.
This is true for most cases. However, attaching and detaching
a multicast address from QP concurrently is safe. Therefore, we
preserve the original behaviour by adding a lock there.

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/uverbs.h     | 2 ++
 drivers/infiniband/core/uverbs_cmd.c | 5 +++++
 2 files changed, 7 insertions(+)

diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 1e31ad8..9fe5e02 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -163,6 +163,8 @@ struct ib_usrq_object {
 
 struct ib_uqp_object {
 	struct ib_uevent_object	uevent;
+	/* lock for mcast list */
+	struct mutex		mcast_lock;
 	struct list_head 	mcast_list;
 	struct ib_uxrcd_object *uxrcd;
 };
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index 17c6dcf..bb8ad1f 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -1511,6 +1511,7 @@ static int create_qp(struct ib_uverbs_file *file,
 		return PTR_ERR(obj);
 	obj->uxrcd = NULL;
 	obj->uevent.uobject.user_handle = cmd->user_handle;
+	mutex_init(&obj->mcast_lock);
 
 	if (cmd_sz >= offsetof(typeof(*cmd), rwq_ind_tbl_handle) +
 		      sizeof(cmd->rwq_ind_tbl_handle) &&
@@ -2749,6 +2750,7 @@ ssize_t ib_uverbs_attach_mcast(struct ib_uverbs_file *file,
 
 	obj = container_of(qp->uobject, struct ib_uqp_object, uevent.uobject);
 
+	mutex_lock(&obj->mcast_lock);
 	list_for_each_entry(mcast, &obj->mcast_list, list)
 		if (cmd.mlid == mcast->lid &&
 		    !memcmp(cmd.gid, mcast->gid.raw, sizeof mcast->gid.raw)) {
@@ -2772,6 +2774,7 @@ ssize_t ib_uverbs_attach_mcast(struct ib_uverbs_file *file,
 		kfree(mcast);
 
 out_put:
+	mutex_unlock(&obj->mcast_lock);
 	put_qp_read(qp);
 
 	return ret ? ret : in_len;
@@ -2796,6 +2799,7 @@ ssize_t ib_uverbs_detach_mcast(struct ib_uverbs_file *file,
 		return -EINVAL;
 
 	obj = container_of(qp->uobject, struct ib_uqp_object, uevent.uobject);
+	mutex_lock(&obj->mcast_lock);
 
 	ret = ib_detach_mcast(qp, (union ib_gid *) cmd.gid, cmd.mlid);
 	if (ret)
@@ -2810,6 +2814,7 @@ ssize_t ib_uverbs_detach_mcast(struct ib_uverbs_file *file,
 		}
 
 out_put:
+	mutex_unlock(&obj->mcast_lock);
 	put_qp_read(qp);
 	return ret ? ret : in_len;
 }
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 6/7] IB/core: Add support for fd objects

[Matan Barak]

The completion channel we use in verbs infrastructure is FD based.
Previously, we had a separate way to manage this object. Since we
strive for a single way to manage any kind of object in this
infrastructure, we conceptually treat all objects as subclasses
of ib_uobject.

This commit adds the necessary mechanism to support FD based objects
like their IDR counterparts. FD objects release need to be synchronized
with context release. We use the cleanup_mutex on the uverbs_file for
that.

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/rdma_core.c   | 157 +++++++++++++++++++++++++++++++++-
 drivers/infiniband/core/rdma_core.h   |   7 ++
 drivers/infiniband/core/uverbs.h      |   1 +
 drivers/infiniband/core/uverbs_main.c |   4 +-
 include/rdma/ib_verbs.h               |   6 ++
 include/rdma/uverbs_types.h           |  16 ++++
 6 files changed, 189 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/rdma_core.c b/drivers/infiniband/core/rdma_core.c
index 7ce4d67..1d24f26 100644
--- a/drivers/infiniband/core/rdma_core.c
+++ b/drivers/infiniband/core/rdma_core.c
@@ -160,6 +160,73 @@ static void uverbs_uobject_add(struct ib_uobject *uobject)
 	mutex_unlock(&uobject->context->lock);
 }
 
+static struct ib_uobject *alloc_begin_fd_uobject(const struct uverbs_obj_type *type,
+						 struct ib_ucontext *ucontext)
+{
+	const struct uverbs_obj_fd_type *fd_type =
+		container_of(type, struct uverbs_obj_fd_type, type);
+	int new_fd;
+	struct ib_uobject_file *uobj_file = NULL;
+	struct file *filp;
+
+	new_fd = get_unused_fd_flags(O_CLOEXEC);
+	if (new_fd < 0)
+		return ERR_PTR(new_fd);
+
+	uobj_file = kmalloc(fd_type->obj_size, GFP_KERNEL);
+	if (!uobj_file) {
+		put_unused_fd(new_fd);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	filp = anon_inode_getfile(fd_type->name,
+				  fd_type->fops,
+				  uobj_file,
+				  fd_type->flags);
+	if (IS_ERR(filp)) {
+		put_unused_fd(new_fd);
+		kfree(uobj_file);
+		return (void *)filp;
+	}
+
+	init_uobj(&uobj_file->uobj, ucontext, type);
+	uobj_file->uobj.id = new_fd;
+	uobj_file->uobj.object = filp;
+	uobj_file->ufile = ucontext->ufile;
+
+	return &uobj_file->uobj;
+}
+
+static struct ib_uobject *lookup_get_fd_uobject(const struct uverbs_obj_type *type,
+						struct ib_ucontext *ucontext,
+						int id, bool write)
+{
+	struct file *f;
+	struct ib_uobject *uobject;
+	const struct uverbs_obj_fd_type *fd_type =
+		container_of(type, struct uverbs_obj_fd_type, type);
+
+	if (write)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	f = fget(id);
+	if (!f)
+		return ERR_PTR(-EBADF);
+
+	uobject = f->private_data;
+	if (f->f_op != fd_type->fops ||
+	    !uobject->context) {
+		fput(f);
+		return ERR_PTR(-EBADF);
+	}
+
+	/*
+	 * No need to protect it with a ref count, as fget increases
+	 * f_count.
+	 */
+	return uobject;
+}
+
 static void alloc_commit_idr_uobject(struct ib_uobject *uobj)
 {
 	uverbs_uobject_add(uobj);
@@ -196,6 +263,38 @@ static void lookup_put_idr_uobject(struct ib_uobject *uobj, bool write)
 		up_read(&uobj->currently_used);
 }
 
+static void lookup_put_fd_uobject(struct ib_uobject *uobj, bool write)
+{
+	struct file *filp = uobj->object;
+
+	WARN_ON(write);
+	fput(filp);
+}
+
+static void alloc_commit_fd_uobject(struct ib_uobject *uobj)
+{
+	struct ib_uobject_file *uobj_file =
+		container_of(uobj, struct ib_uobject_file, uobj);
+
+	kref_get(&uobj_file->ufile->ref);
+	uverbs_uobject_add(&uobj_file->uobj);
+	fd_install(uobj_file->uobj.id, uobj->object);
+	/* This shouldn't be used anymore. Use the file object instead */
+	uobj_file->uobj.id = 0;
+}
+
+static void alloc_abort_fd_uobject(struct ib_uobject *uobj)
+{
+	struct ib_uobject_file *uobj_file =
+		container_of(uobj, struct ib_uobject_file, uobj);
+	struct file *filp = uobj->object;
+
+	/* Unsuccessful NEW */
+	fput(filp);
+	put_unused_fd(uobj_file->uobj.id);
+	uverbs_uobject_put(&uobj_file->uobj);
+}
+
 static void destroy_commit_idr_uobject(struct ib_uobject *uobj)
 {
 	uverbs_idr_remove_uobj(uobj);
@@ -233,6 +332,16 @@ static void release_idr_uobject(struct ib_uobject *uobj)
 	kfree_rcu(uobj, rcu);
 }
 
+static void release_fd_uobject(struct ib_uobject *uobj)
+{
+	kfree(container_of(uobj, struct ib_uobject_file, uobj));
+}
+
+static void destroy_commit_null_uobject(struct ib_uobject *uobj)
+{
+	WARN_ON(true);
+}
+
 const struct uverbs_obj_type_ops uverbs_idr_ops = {
 	.alloc_begin = alloc_begin_idr_uobject,
 	.lookup_get = lookup_get_idr_uobject,
@@ -254,8 +363,15 @@ void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
 
 		/*
 		 * This shouldn't run while executing other commands on this
-		 * context, thus no lock is required.
+		 * context. Thus, the only thing we should take care of is
+		 * releasing a FD while traversing this list. The FD could be
+		 * closed and released from the _release fop of this FD.
+		 * In order to mitigate this, we add a lock.
+		 * We take and release the lock per order traversal in order
+		 * to let other threads (which might still use the FDs) chance
+		 * to run.
 		 */
+		mutex_lock(&ucontext->lock);
 		list_for_each_entry_safe(obj, next_obj, &ucontext->uobjects,
 					 list)
 			if (obj->type->destroy_order == cur_order) {
@@ -265,6 +381,7 @@ void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
 				next_order = min(next_order,
 						 obj->type->destroy_order);
 			}
+		mutex_unlock(&ucontext->lock);
 		cur_order = next_order;
 	}
 }
@@ -275,3 +392,41 @@ void uverbs_initialize_ucontext(struct ib_ucontext *ucontext)
 	INIT_LIST_HEAD(&ucontext->uobjects);
 }
 
+static void hot_unplug_fd_uobject(struct ib_uobject *uobj, bool device_removed)
+{
+	const struct uverbs_obj_fd_type *fd_type =
+		container_of(uobj->type, struct uverbs_obj_fd_type, type);
+	struct ib_uobject_file *uobj_file =
+		container_of(uobj, struct ib_uobject_file, uobj);
+
+	fd_type->hot_unplug(uobj_file, device_removed);
+	uobj_file->uobj.context = NULL;
+}
+
+const struct uverbs_obj_type_ops uverbs_fd_ops = {
+	.alloc_begin = alloc_begin_fd_uobject,
+	.lookup_get = lookup_get_fd_uobject,
+	.alloc_commit = alloc_commit_fd_uobject,
+	.alloc_abort = alloc_abort_fd_uobject,
+	.lookup_put = lookup_put_fd_uobject,
+	.destroy_commit = destroy_commit_null_uobject,
+	.hot_unplug = hot_unplug_fd_uobject,
+	.release = release_fd_uobject,
+};
+
+void uverbs_close_fd(struct file *f)
+{
+	struct ib_uobject_file *uobj_file = f->private_data;
+
+	mutex_lock(&uobj_file->ufile->cleanup_mutex);
+	if (uobj_file->uobj.context) {
+		mutex_lock(&uobj_file->uobj.context->lock);
+		list_del(&uobj_file->uobj.list);
+		mutex_unlock(&uobj_file->uobj.context->lock);
+		uobj_file->uobj.context = NULL;
+	}
+	mutex_unlock(&uobj_file->ufile->cleanup_mutex);
+	kref_put(&uobj_file->ufile->ref, ib_uverbs_release_file);
+	uverbs_uobject_put(&uobj_file->uobj);
+}
+
diff --git a/drivers/infiniband/core/rdma_core.h b/drivers/infiniband/core/rdma_core.h
index ab665a6..da4e808 100644
--- a/drivers/infiniband/core/rdma_core.h
+++ b/drivers/infiniband/core/rdma_core.h
@@ -52,4 +52,11 @@
 void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed);
 void uverbs_initialize_ucontext(struct ib_ucontext *ucontext);
 
+/*
+ * Indicate this fd is no longer used by this consumer, but its memory isn't
+ * released yet. Internally we call uverbs_uobject_put. When the last reference
+ * is put, we release the memory.
+ */
+void uverbs_close_fd(struct file *f);
+
 #endif /* RDMA_CORE_H */
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 9fe5e02..20632ff 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -193,6 +193,7 @@ void ib_uverbs_release_ucq(struct ib_uverbs_file *file,
 			   struct ib_ucq_object *uobj);
 void ib_uverbs_release_uevent(struct ib_uverbs_file *file,
 			      struct ib_uevent_object *uobj);
+void ib_uverbs_release_file(struct kref *ref);
 
 void ib_uverbs_comp_handler(struct ib_cq *cq, void *cq_context);
 void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr);
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index 0eb4538..784eccc 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -229,7 +229,7 @@ static void ib_uverbs_comp_dev(struct ib_uverbs_device *dev)
 	complete(&dev->comp);
 }
 
-static void ib_uverbs_release_file(struct kref *ref)
+void ib_uverbs_release_file(struct kref *ref)
 {
 	struct ib_uverbs_file *file =
 		container_of(ref, struct ib_uverbs_file, ref);
@@ -1128,7 +1128,9 @@ static void ib_uverbs_free_hw_resources(struct ib_uverbs_device *uverbs_dev,
 			 * (e.g mmput).
 			 */
 			ib_dev->disassociate_ucontext(ucontext);
+			mutex_lock(&file->cleanup_mutex);
 			ib_uverbs_cleanup_ucontext(file, ucontext, true);
+			mutex_unlock(&file->cleanup_mutex);
 		}
 
 		mutex_lock(&uverbs_dev->lists_mutex);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 7ddb08f..941a764 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1372,6 +1372,12 @@ struct ib_uobject {
 	const struct uverbs_obj_type *type;
 };
 
+struct ib_uobject_file {
+	struct ib_uobject	uobj;
+	/* ufile contains the lock between context release and file close */
+	struct ib_uverbs_file	*ufile;
+};
+
 struct ib_udata {
 	const void __user *inbuf;
 	void __user *outbuf;
diff --git a/include/rdma/uverbs_types.h b/include/rdma/uverbs_types.h
index cdbf352..48b5e4e 100644
--- a/include/rdma/uverbs_types.h
+++ b/include/rdma/uverbs_types.h
@@ -101,6 +101,22 @@ struct uverbs_obj_idr_type {
 	void (*hot_unplug)(struct ib_uobject *uobj);
 };
 
+struct uverbs_obj_fd_type {
+	/*
+	 * In fd based objects, uverbs_obj_type_ops points to a generic
+	 * fd operations. In order to specialize the underlying types (e.g.
+	 * completion_channel), we use obj_size, fops, name and flags for fd
+	 * creation and hot_unplug for specific release callback.
+	 */
+	struct uverbs_obj_type  type;
+	size_t			obj_size;
+	void (*hot_unplug)(struct ib_uobject_file *uobj_file,
+			   bool device_removed);
+	const struct file_operations	*fops;
+	const char			*name;
+	int				flags;
+};
+
 extern const struct uverbs_obj_type_ops uverbs_idr_ops;
 
 #define UVERBS_BUILD_BUG_ON(cond) (sizeof(char[1 - 2 * !!(cond)]) -	\
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 6/7] IB/core: Add support for fd objects

[Jason Gunthorpe]

On Wed, Feb 01, 2017 at 02:39:04PM +0200, Matan Barak wrote:
> The completion channel we use in verbs infrastructure is FD based.
> Previously, we had a separate way to manage this object. Since we
> strive for a single way to manage any kind of object in this
> infrastructure, we conceptually treat all objects as subclasses
> of ib_uobject.
> 
> This commit adds the necessary mechanism to support FD based objects
> like their IDR counterparts. FD objects release need to be synchronized
> with context release. We use the cleanup_mutex on the uverbs_file for
> that.
> 
> Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>  drivers/infiniband/core/rdma_core.c   | 157 +++++++++++++++++++++++++++++++++-
>  drivers/infiniband/core/rdma_core.h   |   7 ++
>  drivers/infiniband/core/uverbs.h      |   1 +
>  drivers/infiniband/core/uverbs_main.c |   4 +-
>  include/rdma/ib_verbs.h               |   6 ++
>  include/rdma/uverbs_types.h           |  16 ++++
>  6 files changed, 189 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/infiniband/core/rdma_core.c b/drivers/infiniband/core/rdma_core.c
> index 7ce4d67..1d24f26 100644
> +++ b/drivers/infiniband/core/rdma_core.c
> @@ -160,6 +160,73 @@ static void uverbs_uobject_add(struct ib_uobject *uobject)
>  	mutex_unlock(&uobject->context->lock);
>  }
>  
> +static struct ib_uobject *alloc_begin_fd_uobject(const struct uverbs_obj_type *type,
> +						 struct ib_ucontext *ucontext)
> +{
> +	const struct uverbs_obj_fd_type *fd_type =
> +		container_of(type, struct uverbs_obj_fd_type, type);
> +	int new_fd;
> +	struct ib_uobject_file *uobj_file = NULL;
> +	struct file *filp;
> +
> +	new_fd = get_unused_fd_flags(O_CLOEXEC);
> +	if (new_fd < 0)
> +		return ERR_PTR(new_fd);
> +
> +	uobj_file = kmalloc(fd_type->obj_size, GFP_KERNEL);
> +	if (!uobj_file) {
> +		put_unused_fd(new_fd);
> +		return ERR_PTR(-ENOMEM);
> +	}

Move to alloc_uobj, see prior patches

> +static struct ib_uobject *lookup_get_fd_uobject(const struct uverbs_obj_type *type,
> +						struct ib_ucontext *ucontext,
> +						int id, bool write)
> +{
> +	struct file *f;
> +	struct ib_uobject *uobject;
> +	const struct uverbs_obj_fd_type *fd_type =
> +		container_of(type, struct uverbs_obj_fd_type, type);
> +
> +	if (write)
> +		return ERR_PTR(-EOPNOTSUPP);
> +
> +	f = fget(id);
> +	if (!f)
> +		return ERR_PTR(-EBADF);
> +
> +	uobject = f->private_data;
> +	if (f->f_op != fd_type->fops ||
> +	    !uobject->context) {
> +		fput(f);
> +		return ERR_PTR(-EBADF);
> +	}

The read of uobject->context needs locking

> +static void alloc_commit_fd_uobject(struct ib_uobject *uobj)
> +{
> +	struct ib_uobject_file *uobj_file =
> +		container_of(uobj, struct ib_uobject_file, uobj);
> +
> +	kref_get(&uobj_file->ufile->ref);

Something has gone wrong with the krefs here..

kref_get needs to be done for anon_inode_getfile. That kref_get pairs
with the put in uverbs_close_fd.

> +	uverbs_uobject_add(&uobj_file->uobj);
> +	fd_install(uobj_file->uobj.id, uobj->object);
> +	/* This shouldn't be used anymore. Use the file object instead */
> +	uobj_file->uobj.id = 0;

Needs a put to pair with the kref_get done by alloc_uobj.

> +static void alloc_abort_fd_uobject(struct ib_uobject *uobj)
> +{
> +	struct ib_uobject_file *uobj_file =
> +		container_of(uobj, struct ib_uobject_file, uobj);
> +	struct file *filp = uobj->object;
> +
> +	/* Unsuccessful NEW */
> +	fput(filp);
> +	put_unused_fd(uobj_file->uobj.id);

And here is the bug that fixes: since fput(filp) will call
uverbs_close_fd it just put the last kref and free'd the struct.

> +static void destroy_commit_null_uobject(struct ib_uobject *uobj)
> +{
> +	WARN_ON(true);
> +}

So in the model I suggest this would be remove and the purpose is to
disconnect the driver and leave the FD 'detached'.

> +
>  const struct uverbs_obj_type_ops uverbs_idr_ops = {
>  	.alloc_begin = alloc_begin_idr_uobject,
>  	.lookup_get = lookup_get_idr_uobject,
> @@ -254,8 +363,15 @@ void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
>  
>  		/*
>  		 * This shouldn't run while executing other commands on this
> -		 * context, thus no lock is required.
> +		 * context. Thus, the only thing we should take care of is
> +		 * releasing a FD while traversing this list. The FD could be
> +		 * closed and released from the _release fop of this FD.
> +		 * In order to mitigate this, we add a lock.
> +		 * We take and release the lock per order traversal in order
> +		 * to let other threads (which might still use the FDs) chance
> +		 * to run.
>  		 */
> +		mutex_lock(&ucontext->lock);

I think you may as well move this lock to patch 2, reads better that way.

>  		list_for_each_entry_safe(obj, next_obj, &ucontext->uobjects,
>  					 list)
>  			if (obj->type->destroy_order == cur_order) {
> @@ -265,6 +381,7 @@ void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
>  				next_order = min(next_order,
>  						 obj->type->destroy_order);
>  			}
> +		mutex_unlock(&ucontext->lock);

We had lots of problems with this sort of scheme in the hotplug
series, is this OK like this? Feels potentially dangerous..

I feel like we need a rwsem for cleanup. Write lock the rwsem above
which will inhibit changes to the list and have all the remove paths
try_read the rwsem and if that fails then skip removing from the list
and doing the destroy because we know uverbs_cleanup_ucontext is
running and will do both.

That sounds like a much more self-contained and clearer mechanism
compared to where we are today.

> +void uverbs_close_fd(struct file *f)
> +{
> +	struct ib_uobject_file *uobj_file = f->private_data;
> +
> +	mutex_lock(&uobj_file->ufile->cleanup_mutex);

I don't get what cleanup_mutex is doing. Nothing in here seems like it
needs that lock...

> +	if (uobj_file->uobj.context) {
> +		mutex_lock(&uobj_file->uobj.context->lock);
> +		list_del(&uobj_file->uobj.list);
> +		mutex_unlock(&uobj_file->uobj.context->lock);

Again, I don't like this. If we remove something from the list then it
should go through the whole destruction cycle. Why do we skip
is_closed=1 on this path, for instance? Seems wrong.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 6/7] IB/core: Add support for fd objects

[Matan Barak]

On Fri, Feb 10, 2017 at 11:03 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Feb 01, 2017 at 02:39:04PM +0200, Matan Barak wrote:
>> The completion channel we use in verbs infrastructure is FD based.
>> Previously, we had a separate way to manage this object. Since we
>> strive for a single way to manage any kind of object in this
>> infrastructure, we conceptually treat all objects as subclasses
>> of ib_uobject.
>>
>> This commit adds the necessary mechanism to support FD based objects
>> like their IDR counterparts. FD objects release need to be synchronized
>> with context release. We use the cleanup_mutex on the uverbs_file for
>> that.
>>
>> Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>  drivers/infiniband/core/rdma_core.c   | 157 +++++++++++++++++++++++++++++++++-
>>  drivers/infiniband/core/rdma_core.h   |   7 ++
>>  drivers/infiniband/core/uverbs.h      |   1 +
>>  drivers/infiniband/core/uverbs_main.c |   4 +-
>>  include/rdma/ib_verbs.h               |   6 ++
>>  include/rdma/uverbs_types.h           |  16 ++++
>>  6 files changed, 189 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/infiniband/core/rdma_core.c b/drivers/infiniband/core/rdma_core.c
>> index 7ce4d67..1d24f26 100644
>> +++ b/drivers/infiniband/core/rdma_core.c
>> @@ -160,6 +160,73 @@ static void uverbs_uobject_add(struct ib_uobject *uobject)
>>       mutex_unlock(&uobject->context->lock);
>>  }
>>
>> +static struct ib_uobject *alloc_begin_fd_uobject(const struct uverbs_obj_type *type,
>> +                                              struct ib_ucontext *ucontext)
>> +{
>> +     const struct uverbs_obj_fd_type *fd_type =
>> +             container_of(type, struct uverbs_obj_fd_type, type);
>> +     int new_fd;
>> +     struct ib_uobject_file *uobj_file = NULL;
>> +     struct file *filp;
>> +
>> +     new_fd = get_unused_fd_flags(O_CLOEXEC);
>> +     if (new_fd < 0)
>> +             return ERR_PTR(new_fd);
>> +
>> +     uobj_file = kmalloc(fd_type->obj_size, GFP_KERNEL);
>> +     if (!uobj_file) {
>> +             put_unused_fd(new_fd);
>> +             return ERR_PTR(-ENOMEM);
>> +     }
>
> Move to alloc_uobj, see prior patches
>

Ok

>> +static struct ib_uobject *lookup_get_fd_uobject(const struct uverbs_obj_type *type,
>> +                                             struct ib_ucontext *ucontext,
>> +                                             int id, bool write)
>> +{
>> +     struct file *f;
>> +     struct ib_uobject *uobject;
>> +     const struct uverbs_obj_fd_type *fd_type =
>> +             container_of(type, struct uverbs_obj_fd_type, type);
>> +
>> +     if (write)
>> +             return ERR_PTR(-EOPNOTSUPP);
>> +
>> +     f = fget(id);
>> +     if (!f)
>> +             return ERR_PTR(-EBADF);
>> +
>> +     uobject = f->private_data;
>> +     if (f->f_op != fd_type->fops ||
>> +         !uobject->context) {
>> +             fput(f);
>> +             return ERR_PTR(-EBADF);
>> +     }
>
> The read of uobject->context needs locking
>

All these callbacks are assumed to run when the context is alive.
Some upper synchronization mechanism (SRCU) should take care of not calling
uverbs_cleanup_ucontext while executing some other action on a uobject.
The only call we allow the user to execute while this is in process is
uverbs_close_fd and that's
why this is the only call which is protected by the cleanup_mutex.
close_fd is called from the file's release function. This could happen
in two cases - either the object
wasn't exposed to user-space and then commit_abort does the last fput
or the user-space closes the
file. In the first case, nobody should use the fd object as it wasn't
committed yet.
In the second case, fget and __close_fd should protect us.

>> +static void alloc_commit_fd_uobject(struct ib_uobject *uobj)
>> +{
>> +     struct ib_uobject_file *uobj_file =
>> +             container_of(uobj, struct ib_uobject_file, uobj);
>> +
>> +     kref_get(&uobj_file->ufile->ref);
>
> Something has gone wrong with the krefs here..
>
> kref_get needs to be done for anon_inode_getfile. That kref_get pairs
> with the put in uverbs_close_fd.
>

The code assumes close_fd shall always be called from the release
function. So, I guess
alloc_uobj should kref_init and close_fd should be always paired with it.
You could reach close_fd either from alloc_abort, close system call or
some external kernel code
calling fput (with file_get that was called before. In this case
lookup_get isn't valid anymore).

>> +     uverbs_uobject_add(&uobj_file->uobj);
>> +     fd_install(uobj_file->uobj.id, uobj->object);
>> +     /* This shouldn't be used anymore. Use the file object instead */
>> +     uobj_file->uobj.id = 0;
>
> Needs a put to pair with the kref_get done by alloc_uobj.
>
>> +static void alloc_abort_fd_uobject(struct ib_uobject *uobj)
>> +{
>> +     struct ib_uobject_file *uobj_file =
>> +             container_of(uobj, struct ib_uobject_file, uobj);
>> +     struct file *filp = uobj->object;
>> +
>> +     /* Unsuccessful NEW */
>> +     fput(filp);
>> +     put_unused_fd(uobj_file->uobj.id);
>
> And here is the bug that fixes: since fput(filp) will call
> uverbs_close_fd it just put the last kref and free'd the struct.
>

We could either remove uverbs_uobject_put call.

>> +static void destroy_commit_null_uobject(struct ib_uobject *uobj)
>> +{
>> +     WARN_ON(true);
>> +}
>
> So in the model I suggest this would be remove and the purpose is to
> disconnect the driver and leave the FD 'detached'.
>

As stated in the previous patch, remove_commit (the new name) is
entirely different
than cleanup. This just removes the object from the objects list and
decrease its
reference count.
I guess if we want to enable that for fds too, we need some lighter
version of uverbs_close_fd:

Something like:
        mutex_lock(&uobj_file->ufile->cleanup_mutex);
        if (uobj_file->uobj.context) {
                mutex_lock(&uobj_file->uobj.context->uobjects_lock);
                list_del(&uobj_file->uobj.list);
                mutex_unlock(&uobj_file->uobj.context->uobjects_lock);
                uobj_file->uobj.context = NULL;
        }
        mutex_unlock(&uobj_file->ufile->cleanup_mutex);

It removes it from the list and nullify the context that we wouldn't do that
again in close_fd.

The cleanup callback runs from the context termination. It actually destroys
the real object (or marks it as closed in the fd case).

>> +
>>  const struct uverbs_obj_type_ops uverbs_idr_ops = {
>>       .alloc_begin = alloc_begin_idr_uobject,
>>       .lookup_get = lookup_get_idr_uobject,
>> @@ -254,8 +363,15 @@ void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
>>
>>               /*
>>                * This shouldn't run while executing other commands on this
>> -              * context, thus no lock is required.
>> +              * context. Thus, the only thing we should take care of is
>> +              * releasing a FD while traversing this list. The FD could be
>> +              * closed and released from the _release fop of this FD.
>> +              * In order to mitigate this, we add a lock.
>> +              * We take and release the lock per order traversal in order
>> +              * to let other threads (which might still use the FDs) chance
>> +              * to run.
>>                */
>> +             mutex_lock(&ucontext->lock);
>
> I think you may as well move this lock to patch 2, reads better that way.
>

Ok

>>               list_for_each_entry_safe(obj, next_obj, &ucontext->uobjects,
>>                                        list)
>>                       if (obj->type->destroy_order == cur_order) {
>> @@ -265,6 +381,7 @@ void uverbs_cleanup_ucontext(struct ib_ucontext *ucontext, bool device_removed)
>>                               next_order = min(next_order,
>>                                                obj->type->destroy_order);
>>                       }
>> +             mutex_unlock(&ucontext->lock);
>
> We had lots of problems with this sort of scheme in the hotplug
> series, is this OK like this? Feels potentially dangerous..
>
> I feel like we need a rwsem for cleanup. Write lock the rwsem above
> which will inhibit changes to the list and have all the remove paths
> try_read the rwsem and if that fails then skip removing from the list
> and doing the destroy because we know uverbs_cleanup_ucontext is
> running and will do both.
>
> That sounds like a much more self-contained and clearer mechanism
> compared to where we are today.
>

We use SRCU to make sure all handlers are completed and no new handlers
could be executed. This means we're safe (I'll add WARN_ON to be sure).
We could only race with closing fd objects and this is handled with
the cleanup_mutex.
If we run handlers and context cleanup at the same time, we could end
up with double
cleanups of the objects themselves (without additional locks of course).

>> +void uverbs_close_fd(struct file *f)
>> +{
>> +     struct ib_uobject_file *uobj_file = f->private_data;
>> +
>> +     mutex_lock(&uobj_file->ufile->cleanup_mutex);
>
> I don't get what cleanup_mutex is doing. Nothing in here seems like it
> needs that lock...
>

cleanup_mutex protects a race when the context is cleaned up and
uverbs_close_fd is called. They both tries to alter the uobjects list,
so they need the lock on the context. The problem here is that the
context might be released and its pointer could be nullified.
It could be even nullified after the "if (uobj_file->uobj.context) "
statement. In order to solve that, the cleanup_context delays
"file->ucontext = NULL;" and thus makes this safe.


>> +     if (uobj_file->uobj.context) {
>> +             mutex_lock(&uobj_file->uobj.context->lock);
>> +             list_del(&uobj_file->uobj.list);
>> +             mutex_unlock(&uobj_file->uobj.context->lock);
>
> Again, I don't like this. If we remove something from the list then it
> should go through the whole destruction cycle. Why do we skip
> is_closed=1 on this path, for instance? Seems wrong.
>

An object could be destroyed in two ways:
a. handler + remove_commit
b. context cleanup

is_closed = 1 is part of the free function of the completion channel objects.
This series only refactors how these objects are managed and tries hard
not to change the internal implementation there.

> Jason

Thanks for the review

Matan

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 6/7] IB/core: Add support for fd objects

[Jason Gunthorpe]

On Wed, Feb 15, 2017 at 03:48:40PM +0200, Matan Barak wrote:
> >> +static struct ib_uobject *lookup_get_fd_uobject(const struct uverbs_obj_type *type,
> >> +                                             struct ib_ucontext *ucontext,
> >> +                                             int id, bool write)
> >> +{
> >> +     struct file *f;
> >> +     struct ib_uobject *uobject;
> >> +     const struct uverbs_obj_fd_type *fd_type =
> >> +             container_of(type, struct uverbs_obj_fd_type, type);
> >> +
> >> +     if (write)
> >> +             return ERR_PTR(-EOPNOTSUPP);
> >> +
> >> +     f = fget(id);
> >> +     if (!f)
> >> +             return ERR_PTR(-EBADF);
> >> +
> >> +     uobject = f->private_data;
> >> +     if (f->f_op != fd_type->fops ||
> >> +         !uobject->context) {
> >> +             fput(f);
> >> +             return ERR_PTR(-EBADF);
> >> +     }
> >
> > The read of uobject->context needs locking
> >
> 
> All these callbacks are assumed to run when the context is alive.

'uboject->context' is touched outside the SRCU, eg in
uverbs_close_fd() - so it absolutely needs locking.

> The only call we allow the user to execute while this is in process
> is uverbs_close_fd and that's why this is the only call which is
> protected by the cleanup_mutex.

So cleanup_mutex would need to be held during lookup

> >> +static void alloc_commit_fd_uobject(struct ib_uobject *uobj)
> >> +{
> >> +     struct ib_uobject_file *uobj_file =
> >> +             container_of(uobj, struct ib_uobject_file, uobj);
> >> +
> >> +     kref_get(&uobj_file->ufile->ref);
> >
> > Something has gone wrong with the krefs here..
> >
> > kref_get needs to be done for anon_inode_getfile. That kref_get pairs
> > with the put in uverbs_close_fd.
> >
> 
> The code assumes close_fd shall always be called from the release
> function. So, I guess
> alloc_uobj should kref_init and close_fd should be always paired
> with it.

You could make this work, but is is not a great approach. For clarity
a kref should never be shared by two pointers, that provides the
fewest surprises. For instance this is very confusing

uobj = alloc_uobj();   // uobj hold the kref
filp = anon_inode_getfile(); // now another uobj pointer is in filp
[..]
if (err) {
   fput(filp)
   // Oops! Now uobj is toast, but we still have a uobj pointer on our stack!

Make filp totally self-contained and make uobj on the stack totally
self contained with regard to the krefs.

> Something like:
>         mutex_lock(&uobj_file->ufile->cleanup_mutex);
>         if (uobj_file->uobj.context) {
>                 mutex_lock(&uobj_file->uobj.context->uobjects_lock);
>                 list_del(&uobj_file->uobj.list);
>                 mutex_unlock(&uobj_file->uobj.context->uobjects_lock);
>                 uobj_file->uobj.context = NULL;
>         }
>         mutex_unlock(&uobj_file->ufile->cleanup_mutex);
> 
> It removes it from the list and nullify the context that we wouldn't do that
> again in close_fd.
> 
> The cleanup callback runs from the context termination. It actually destroys
> the real object (or marks it as closed in the fd case).

Maybe, I'd have to see this to comment I think.. But this locking is
looking nasty

> > We had lots of problems with this sort of scheme in the hotplug
> > series, is this OK like this? Feels potentially dangerous..
> >
> > I feel like we need a rwsem for cleanup. Write lock the rwsem above
> > which will inhibit changes to the list and have all the remove paths
> > try_read the rwsem and if that fails then skip removing from the list
> > and doing the destroy because we know uverbs_cleanup_ucontext is
> > running and will do both.
> >
> > That sounds like a much more self-contained and clearer mechanism
> > compared to where we are today.

I think you should try something like this:

void add_object_to_cleanup(struct ib_uobject *uobj)
{
    // Cleanup is running. Calling this should have been impossible
    if (!down_read_trylock(&ucontext->cleanup_rwsem)) {
       WARN();
       return;
    }
    
    mutex_lock(&icontext->uobjects_lock);
    // kref is held while the object is on the list
    get_uobj(uobj);
    list_add(&uobj->list, &ucontext->uobjects);
    mutex_unlock(&iucontext->uobjects_lock);

    up_read(&ucontext->cleanup_rwsem);
}

/* For an object on the cleanup list, remove it from the list and
   call the remove function. This is either done immediately
   or defered to an ongoing cleanup. Caller must hold a kref on uobj */
void destory_object_from_cleanup(struct ib_uobject *uobj,
                                 enum rdma_remove_reason why)
{
    // Cleanup is running, the object will be destroyed by the cleanup.
    if (!down_read_trylock(&ucontext->cleanup_rwsem))
       return;

    // Remove it from the list
    mutex_lock(&ucontext->uobjects_lock);
    list_del(&uobj->list);
    uobj->context = NULL;
    put_uobj(uobj);
    mutex_unlock(&ucontext->uobjects_lock);

    [..]->remove(uobj, why);

    up_read(&ucontext->cleanup_rwsem);
}

void cleanup()
{
    /* Claim ownership of the context list. Every add attempt will
       fail and every remove attempt will defer to this loop */
    down_write(&ucontext->cleanup_rwsem);

    /* Note all uobjects_lock's are nested inside cleanup_rwsem, by
       holding the write side we prove we have exclusive list access */
    
    for_each_safe(uobj) {
        [..]->remove(uobj, why);
        put_uobj(uobj);
    }

    // rwsem remains held forever, uncontext is going to be destroyed
}

void uverbs_close_fd(struct file *f)
{
   destroy_object_from_cleanup(uobj);
   put_uobj(uobj);
}

> >> +     if (uobj_file->uobj.context) {
> >> +             mutex_lock(&uobj_file->uobj.context->lock);
> >> +             list_del(&uobj_file->uobj.list);
> >> +             mutex_unlock(&uobj_file->uobj.context->lock);
> >
> > Again, I don't like this. If we remove something from the list then it
> > should go through the whole destruction cycle. Why do we skip
> > is_closed=1 on this path, for instance? Seems wrong.
> >
> 
> An object could be destroyed in two ways:
> a. handler + remove_commit
> b. context cleanup

Again, this doesn't really make sense and is just going to cause bugs

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 6/7] IB/core: Add support for fd objects

[Matan Barak]

On Wed, Feb 15, 2017 at 8:54 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Feb 15, 2017 at 03:48:40PM +0200, Matan Barak wrote:
>> >> +static struct ib_uobject *lookup_get_fd_uobject(const struct uverbs_obj_type *type,
>> >> +                                             struct ib_ucontext *ucontext,
>> >> +                                             int id, bool write)
>> >> +{
>> >> +     struct file *f;
>> >> +     struct ib_uobject *uobject;
>> >> +     const struct uverbs_obj_fd_type *fd_type =
>> >> +             container_of(type, struct uverbs_obj_fd_type, type);
>> >> +
>> >> +     if (write)
>> >> +             return ERR_PTR(-EOPNOTSUPP);
>> >> +
>> >> +     f = fget(id);
>> >> +     if (!f)
>> >> +             return ERR_PTR(-EBADF);
>> >> +
>> >> +     uobject = f->private_data;
>> >> +     if (f->f_op != fd_type->fops ||
>> >> +         !uobject->context) {
>> >> +             fput(f);
>> >> +             return ERR_PTR(-EBADF);
>> >> +     }
>> >
>> > The read of uobject->context needs locking
>> >
>>
>> All these callbacks are assumed to run when the context is alive.
>
> 'uboject->context' is touched outside the SRCU, eg in
> uverbs_close_fd() - so it absolutely needs locking.
>

uobject->context isn't protected by SRCU as intended.
lookup_get_fd_uobject can't be
executed concurrently with uverbs_close_fd, as uverbs_close_fd is
called when the
last reference of the fd is put and in that case fget called by
lookup_get_fd_uobject
would simply fail. Since fput and fget are atomically safe, this should be safe.

>> The only call we allow the user to execute while this is in process
>> is uverbs_close_fd and that's why this is the only call which is
>> protected by the cleanup_mutex.
>
> So cleanup_mutex would need to be held during lookup
>

You can't run lookup_get and uverbs_close_fd concurrently as stated above.

Regarding the locking, assuming uverbs_close_fd is called from the
release file operation and SRCU
protects us (as currently implemented), we don't see locking issues here.

Yishai and Matan.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH V1 for-next 6/7] IB/core: Add support for fd objects

[Jason Gunthorpe]

On Sun, Feb 19, 2017 at 02:47:02PM +0200, Matan Barak wrote:

> >>
> >> All these callbacks are assumed to run when the context is alive.
> >
> > 'uboject->context' is touched outside the SRCU, eg in
> > uverbs_close_fd() - so it absolutely needs locking.
> >
> 
> uobject->context isn't protected by SRCU as intended.
> lookup_get_fd_uobject can't be executed concurrently with
> uverbs_close_fd, as uverbs_close_fd is called when the last
> reference of the fd is put and in that case fget called by
> lookup_get_fd_uobject would simply fail. Since fput and fget are
> atomically safe, this should be safe.

That is so tricky it needs a comment, but yes, that argument makes
sense for uverbs_close_fd().

But, you've just explained away every reason for this test:

+	if (f->f_op != fd_type->fops ||
+	    !uobject->context) {

It should probably be

	if (f->f_op != fd_type->fops) {
		fput(f);
		return ERR_PTR(-EBADF);
	}

	/* fget(id) ensures we are not currently running
	 * uverbs_close_fd, and the caller is expected to ensure
	 * that uverbs_cleanup_ucontext is never done while
	 * a call top lookup is possible. */
	WARN_ON(!uobject->context);
	if (!uobject->context) {
	        // Try to recover
		fput(f);
		return ERR_PTR(-EBADF);
	}

Or you need to explain where context = null comes from..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 for-next 7/7] IB/core: Change completion channel to use the reworked objects schema

[Matan Barak]

This patch adds the standard fd based type - completion_channel.
The completion_channel is now prefixed with ib_uobject, similarly
to the rest of the uobjects.
This requires a few changes:
(1) We define a new completion channel fd based object type.
(2) completion_event and async_event are now two different types.
    This means they use different fops.
(3) We release the completion_channel exactly as we release other
    idr based objects.
(4) Since ib_uobjects are already kref-ed, we only add the kref to the
    async event.

A fd object requires filling out several parameters. Its op pointer
should point to uverbs_fd_ops and its size should be at least the
size if ib_uobject. We use a macro to make the type declaration
easier.

Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/rdma_core.c        |   5 +
 drivers/infiniband/core/rdma_core.h        |  12 ++
 drivers/infiniband/core/uverbs.h           |  26 ++-
 drivers/infiniband/core/uverbs_cmd.c       |  61 +++++--
 drivers/infiniband/core/uverbs_main.c      | 281 +++++++++++++++++------------
 drivers/infiniband/core/uverbs_std_types.c |  30 ++-
 include/rdma/uverbs_std_types.h            |   1 +
 include/rdma/uverbs_types.h                |  12 ++
 8 files changed, 281 insertions(+), 147 deletions(-)

diff --git a/drivers/infiniband/core/rdma_core.c b/drivers/infiniband/core/rdma_core.c
index 1d24f26..f193798 100644
--- a/drivers/infiniband/core/rdma_core.c
+++ b/drivers/infiniband/core/rdma_core.c
@@ -37,6 +37,11 @@
 #include "uverbs.h"
 #include "rdma_core.h"
 
+void uverbs_uobject_get(struct ib_uobject *uobj)
+{
+	kref_get(&uobj->ref);
+}
+
 static void uverbs_uobject_put_ref(struct kref *ref)
 {
 	struct ib_uobject *uobj =
diff --git a/drivers/infiniband/core/rdma_core.h b/drivers/infiniband/core/rdma_core.h
index da4e808..257cf22 100644
--- a/drivers/infiniband/core/rdma_core.h
+++ b/drivers/infiniband/core/rdma_core.h
@@ -59,4 +59,16 @@
  */
 void uverbs_close_fd(struct file *f);
 
+/*
+ * uverbs_uobject_get is called in order to increase the reference count on an
+ * uobject. This is useful when a handler wants to keep the uobject's memory
+ * alive, regardless if this uobject is still alive in the context's objects
+ * repository.
+ * In order to indicate we no longer needs this uobject, uverbs_uobject_put is
+ * called. When the reference count is decreased, the uobject is freed.
+ * For example, this is used when attaching a completion channel to a CQ.
+ */
+void uverbs_uobject_get(struct ib_uobject *uobj);
+void uverbs_uobject_put(struct ib_uobject *uobj);
+
 #endif /* RDMA_CORE_H */
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index 20632ff..98d9924 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -102,17 +102,25 @@ struct ib_uverbs_device {
 };
 
 struct ib_uverbs_event_file {
-	struct kref				ref;
-	int					is_async;
-	struct ib_uverbs_file		       *uverbs_file;
 	spinlock_t				lock;
 	int					is_closed;
 	wait_queue_head_t			poll_wait;
 	struct fasync_struct		       *async_queue;
 	struct list_head			event_list;
+};
+
+struct ib_uverbs_async_event_file {
+	struct ib_uverbs_event_file		ev_file;
+	struct ib_uverbs_file		       *uverbs_file;
+	struct kref				ref;
 	struct list_head			list;
 };
 
+struct ib_uverbs_completion_event_file {
+	struct ib_uobject_file			uobj_file;
+	struct ib_uverbs_event_file		ev_file;
+};
+
 struct ib_uverbs_file {
 	struct kref				ref;
 	struct mutex				mutex;
@@ -120,7 +128,7 @@ struct ib_uverbs_file {
 	struct ib_uverbs_device		       *device;
 	struct ib_ucontext		       *ucontext;
 	struct ib_event_handler			event_handler;
-	struct ib_uverbs_event_file	       *async_file;
+	struct ib_uverbs_async_event_file       *async_file;
 	struct list_head			list;
 	int					is_closed;
 
@@ -182,14 +190,14 @@ struct ib_ucq_object {
 	u32			async_events_reported;
 };
 
-struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file,
-					struct ib_device *ib_dev,
-					int is_async);
+extern const struct file_operations uverbs_event_fops;
+void ib_uverbs_init_event_file(struct ib_uverbs_event_file *ev_file);
+struct file *ib_uverbs_alloc_async_event_file(struct ib_uverbs_file *uverbs_file,
+					      struct ib_device *ib_dev);
 void ib_uverbs_free_async_event_file(struct ib_uverbs_file *uverbs_file);
-struct ib_uverbs_event_file *ib_uverbs_lookup_comp_file(int fd);
 
 void ib_uverbs_release_ucq(struct ib_uverbs_file *file,
-			   struct ib_uverbs_event_file *ev_file,
+			   struct ib_uverbs_completion_event_file *ev_file,
 			   struct ib_ucq_object *uobj);
 void ib_uverbs_release_uevent(struct ib_uverbs_file *file,
 			      struct ib_uevent_object *uobj);
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index bb8ad1f..01ad621 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -145,6 +145,25 @@ static void put_xrcd_read(struct ib_uobject *uobj)
 {
 	put_uobj_read(uobj);
 }
+
+static struct ib_uverbs_completion_event_file *
+ib_uverbs_lookup_comp_file(int fd, struct ib_ucontext *context)
+{
+	struct ib_uobject *uobj;
+
+	uobj = uverbs_type_attrs_comp_channel.type.ops->lookup_get(
+		&uverbs_type_attrs_comp_channel.type, context, fd, false);
+
+	if (IS_ERR(uobj))
+		return NULL;
+
+	uverbs_uobject_get(uobj);
+	put_uobj_read(uobj);
+
+	return container_of(uobj, struct ib_uverbs_completion_event_file,
+			    uobj_file.uobj);
+}
+
 ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file,
 			      struct ib_device *ib_dev,
 			      const char __user *buf,
@@ -208,7 +227,7 @@ ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file,
 		goto err_free;
 	resp.async_fd = ret;
 
-	filp = ib_uverbs_alloc_event_file(file, ib_dev, 1);
+	filp = ib_uverbs_alloc_async_event_file(file, ib_dev);
 	if (IS_ERR(filp)) {
 		ret = PTR_ERR(filp);
 		goto err_fd;
@@ -1068,8 +1087,8 @@ ssize_t ib_uverbs_create_comp_channel(struct ib_uverbs_file *file,
 {
 	struct ib_uverbs_create_comp_channel	   cmd;
 	struct ib_uverbs_create_comp_channel_resp  resp;
-	struct file				  *filp;
-	int ret;
+	struct ib_uobject			  *uobj;
+	struct ib_uverbs_completion_event_file	  *ev_file;
 
 	if (out_len < sizeof resp)
 		return -ENOSPC;
@@ -1077,25 +1096,24 @@ ssize_t ib_uverbs_create_comp_channel(struct ib_uverbs_file *file,
 	if (copy_from_user(&cmd, buf, sizeof cmd))
 		return -EFAULT;
 
-	ret = get_unused_fd_flags(O_CLOEXEC);
-	if (ret < 0)
-		return ret;
-	resp.fd = ret;
+	uobj = uverbs_type_attrs_comp_channel.type.ops->alloc_begin(
+			&uverbs_type_attrs_comp_channel.type, file->ucontext);
+	if (IS_ERR(uobj))
+		return PTR_ERR(uobj);
 
-	filp = ib_uverbs_alloc_event_file(file, ib_dev, 0);
-	if (IS_ERR(filp)) {
-		put_unused_fd(resp.fd);
-		return PTR_ERR(filp);
-	}
+	resp.fd = uobj->id;
+
+	ev_file = container_of(uobj, struct ib_uverbs_completion_event_file,
+			       uobj_file.uobj);
+	ib_uverbs_init_event_file(&ev_file->ev_file);
 
 	if (copy_to_user((void __user *) (unsigned long) cmd.response,
 			 &resp, sizeof resp)) {
-		put_unused_fd(resp.fd);
-		fput(filp);
+		uobj->type->ops->alloc_abort(uobj);
 		return -EFAULT;
 	}
 
-	fd_install(resp.fd, filp);
+	uobj->type->ops->alloc_commit(uobj);
 	return in_len;
 }
 
@@ -1113,7 +1131,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 				       void *context)
 {
 	struct ib_ucq_object           *obj;
-	struct ib_uverbs_event_file    *ev_file = NULL;
+	struct ib_uverbs_completion_event_file    *ev_file = NULL;
 	struct ib_cq                   *cq;
 	int                             ret;
 	struct ib_uverbs_ex_create_cq_resp resp;
@@ -1129,7 +1147,8 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 		return obj;
 
 	if (cmd->comp_channel >= 0) {
-		ev_file = ib_uverbs_lookup_comp_file(cmd->comp_channel);
+		ev_file = ib_uverbs_lookup_comp_file(cmd->comp_channel,
+						     file->ucontext);
 		if (!ev_file) {
 			ret = -EINVAL;
 			goto err;
@@ -1159,7 +1178,7 @@ static struct ib_ucq_object *create_cq(struct ib_uverbs_file *file,
 	cq->uobject       = &obj->uobject;
 	cq->comp_handler  = ib_uverbs_comp_handler;
 	cq->event_handler = ib_uverbs_cq_event_handler;
-	cq->cq_context    = ev_file;
+	cq->cq_context    = &ev_file->ev_file;
 	atomic_set(&cq->usecnt, 0);
 
 	obj->uobject.object = cq;
@@ -1462,7 +1481,11 @@ ssize_t ib_uverbs_destroy_cq(struct ib_uverbs_file *file,
 		return ret;
 	}
 
-	ib_uverbs_release_ucq(file, ev_file, obj);
+	ib_uverbs_release_ucq(file, ev_file ?
+			      container_of(ev_file,
+					   struct ib_uverbs_completion_event_file,
+					   ev_file) : NULL,
+			      obj);
 	uobj->type->ops->destroy_commit(uobj);
 
 	memset(&resp, 0, sizeof resp);
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index 784eccc..cf86db4 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -155,37 +155,37 @@ static void ib_uverbs_release_dev(struct kobject *kobj)
 	.release = ib_uverbs_release_dev,
 };
 
-static void ib_uverbs_release_event_file(struct kref *ref)
+static void ib_uverbs_release_async_event_file(struct kref *ref)
 {
-	struct ib_uverbs_event_file *file =
-		container_of(ref, struct ib_uverbs_event_file, ref);
+	struct ib_uverbs_async_event_file *file =
+		container_of(ref, struct ib_uverbs_async_event_file, ref);
 
 	kfree(file);
 }
 
 void ib_uverbs_release_ucq(struct ib_uverbs_file *file,
-			  struct ib_uverbs_event_file *ev_file,
+			  struct ib_uverbs_completion_event_file *ev_file,
 			  struct ib_ucq_object *uobj)
 {
 	struct ib_uverbs_event *evt, *tmp;
 
 	if (ev_file) {
-		spin_lock_irq(&ev_file->lock);
+		spin_lock_irq(&ev_file->ev_file.lock);
 		list_for_each_entry_safe(evt, tmp, &uobj->comp_list, obj_list) {
 			list_del(&evt->list);
 			kfree(evt);
 		}
-		spin_unlock_irq(&ev_file->lock);
+		spin_unlock_irq(&ev_file->ev_file.lock);
 
-		kref_put(&ev_file->ref, ib_uverbs_release_event_file);
+		uverbs_uobject_put(&ev_file->uobj_file.uobj);
 	}
 
-	spin_lock_irq(&file->async_file->lock);
+	spin_lock_irq(&file->async_file->ev_file.lock);
 	list_for_each_entry_safe(evt, tmp, &uobj->async_list, obj_list) {
 		list_del(&evt->list);
 		kfree(evt);
 	}
-	spin_unlock_irq(&file->async_file->lock);
+	spin_unlock_irq(&file->async_file->ev_file.lock);
 }
 
 void ib_uverbs_release_uevent(struct ib_uverbs_file *file,
@@ -193,12 +193,12 @@ void ib_uverbs_release_uevent(struct ib_uverbs_file *file,
 {
 	struct ib_uverbs_event *evt, *tmp;
 
-	spin_lock_irq(&file->async_file->lock);
+	spin_lock_irq(&file->async_file->ev_file.lock);
 	list_for_each_entry_safe(evt, tmp, &uobj->event_list, obj_list) {
 		list_del(&evt->list);
 		kfree(evt);
 	}
-	spin_unlock_irq(&file->async_file->lock);
+	spin_unlock_irq(&file->async_file->ev_file.lock);
 }
 
 void ib_uverbs_detach_umcast(struct ib_qp *qp,
@@ -249,10 +249,12 @@ void ib_uverbs_release_file(struct kref *ref)
 	kfree(file);
 }
 
-static ssize_t ib_uverbs_event_read(struct file *filp, char __user *buf,
-				    size_t count, loff_t *pos)
+static ssize_t ib_uverbs_event_read(struct ib_uverbs_event_file *file,
+				    struct ib_uverbs_file *uverbs_file,
+				    struct file *filp, char __user *buf,
+				    size_t count, loff_t *pos,
+				    bool is_async)
 {
-	struct ib_uverbs_event_file *file = filp->private_data;
 	struct ib_uverbs_event *event;
 	int eventsz;
 	int ret = 0;
@@ -271,12 +273,12 @@ static ssize_t ib_uverbs_event_read(struct file *filp, char __user *buf,
 			 * and wake_up() guarentee this will see the null set
 			 * without using RCU
 			 */
-					     !file->uverbs_file->device->ib_dev)))
+					     !uverbs_file->device->ib_dev)))
 			return -ERESTARTSYS;
 
 		/* If device was disassociated and no event exists set an error */
 		if (list_empty(&file->event_list) &&
-		    !file->uverbs_file->device->ib_dev)
+		    !uverbs_file->device->ib_dev)
 			return -EIO;
 
 		spin_lock_irq(&file->lock);
@@ -284,7 +286,7 @@ static ssize_t ib_uverbs_event_read(struct file *filp, char __user *buf,
 
 	event = list_entry(file->event_list.next, struct ib_uverbs_event, list);
 
-	if (file->is_async)
+	if (is_async)
 		eventsz = sizeof (struct ib_uverbs_async_event_desc);
 	else
 		eventsz = sizeof (struct ib_uverbs_comp_event_desc);
@@ -314,11 +316,31 @@ static ssize_t ib_uverbs_event_read(struct file *filp, char __user *buf,
 	return ret;
 }
 
-static unsigned int ib_uverbs_event_poll(struct file *filp,
+static ssize_t ib_uverbs_async_event_read(struct file *filp, char __user *buf,
+					  size_t count, loff_t *pos)
+{
+	struct ib_uverbs_async_event_file *file = filp->private_data;
+
+	return ib_uverbs_event_read(&file->ev_file, file->uverbs_file, filp,
+				    buf, count, pos, true);
+}
+
+static ssize_t ib_uverbs_comp_event_read(struct file *filp, char __user *buf,
+					 size_t count, loff_t *pos)
+{
+	struct ib_uverbs_completion_event_file *comp_ev_file =
+		filp->private_data;
+
+	return ib_uverbs_event_read(&comp_ev_file->ev_file,
+				    comp_ev_file->uobj_file.ufile, filp,
+				    buf, count, pos, false);
+}
+
+static unsigned int ib_uverbs_event_poll(struct ib_uverbs_event_file *file,
+					 struct file *filp,
 					 struct poll_table_struct *wait)
 {
 	unsigned int pollflags = 0;
-	struct ib_uverbs_event_file *file = filp->private_data;
 
 	poll_wait(filp, &file->poll_wait, wait);
 
@@ -330,49 +352,100 @@ static unsigned int ib_uverbs_event_poll(struct file *filp,
 	return pollflags;
 }
 
-static int ib_uverbs_event_fasync(int fd, struct file *filp, int on)
+static unsigned int ib_uverbs_async_event_poll(struct file *filp,
+					       struct poll_table_struct *wait)
+{
+	return ib_uverbs_event_poll(filp->private_data, filp, wait);
+}
+
+static unsigned int ib_uverbs_comp_event_poll(struct file *filp,
+					      struct poll_table_struct *wait)
+{
+	struct ib_uverbs_completion_event_file *comp_ev_file =
+		filp->private_data;
+
+	return ib_uverbs_event_poll(&comp_ev_file->ev_file, filp, wait);
+}
+
+static int ib_uverbs_async_event_fasync(int fd, struct file *filp, int on)
 {
 	struct ib_uverbs_event_file *file = filp->private_data;
 
 	return fasync_helper(fd, filp, on, &file->async_queue);
 }
 
-static int ib_uverbs_event_close(struct inode *inode, struct file *filp)
+static int ib_uverbs_comp_event_fasync(int fd, struct file *filp, int on)
 {
-	struct ib_uverbs_event_file *file = filp->private_data;
+	struct ib_uverbs_completion_event_file *comp_ev_file =
+		filp->private_data;
+
+	return fasync_helper(fd, filp, on, &comp_ev_file->ev_file.async_queue);
+}
+
+static int ib_uverbs_async_event_close(struct inode *inode, struct file *filp)
+{
+	struct ib_uverbs_async_event_file *file = filp->private_data;
+	struct ib_uverbs_file *uverbs_file = file->uverbs_file;
 	struct ib_uverbs_event *entry, *tmp;
 	int closed_already = 0;
 
-	mutex_lock(&file->uverbs_file->device->lists_mutex);
-	spin_lock_irq(&file->lock);
-	closed_already = file->is_closed;
-	file->is_closed = 1;
-	list_for_each_entry_safe(entry, tmp, &file->event_list, list) {
+	mutex_lock(&uverbs_file->device->lists_mutex);
+	spin_lock_irq(&file->ev_file.lock);
+	closed_already = file->ev_file.is_closed;
+	file->ev_file.is_closed = 1;
+	list_for_each_entry_safe(entry, tmp, &file->ev_file.event_list, list) {
 		if (entry->counter)
 			list_del(&entry->obj_list);
 		kfree(entry);
 	}
-	spin_unlock_irq(&file->lock);
+	spin_unlock_irq(&file->ev_file.lock);
 	if (!closed_already) {
 		list_del(&file->list);
-		if (file->is_async)
-			ib_unregister_event_handler(&file->uverbs_file->
-				event_handler);
+		ib_unregister_event_handler(&uverbs_file->event_handler);
+	}
+	mutex_unlock(&uverbs_file->device->lists_mutex);
+
+	kref_put(&uverbs_file->ref, ib_uverbs_release_file);
+	kref_put(&file->ref, ib_uverbs_release_async_event_file);
+
+	return 0;
+}
+
+static int ib_uverbs_comp_event_close(struct inode *inode, struct file *filp)
+{
+	struct ib_uverbs_completion_event_file *file = filp->private_data;
+	struct ib_uverbs_event *entry, *tmp;
+	struct ib_uverbs_file *ufile;
+
+	spin_lock_irq(&file->ev_file.lock);
+	list_for_each_entry_safe(entry, tmp, &file->ev_file.event_list, list) {
+		if (entry->counter)
+			list_del(&entry->obj_list);
+		kfree(entry);
 	}
-	mutex_unlock(&file->uverbs_file->device->lists_mutex);
+	spin_unlock_irq(&file->ev_file.lock);
 
-	kref_put(&file->uverbs_file->ref, ib_uverbs_release_file);
-	kref_put(&file->ref, ib_uverbs_release_event_file);
+	ufile = file->uobj_file.ufile;
+	uverbs_close_fd(filp);
 
 	return 0;
 }
 
-static const struct file_operations uverbs_event_fops = {
+const struct file_operations uverbs_event_fops = {
 	.owner	 = THIS_MODULE,
-	.read	 = ib_uverbs_event_read,
-	.poll    = ib_uverbs_event_poll,
-	.release = ib_uverbs_event_close,
-	.fasync  = ib_uverbs_event_fasync,
+	.read	 = ib_uverbs_comp_event_read,
+	.poll    = ib_uverbs_comp_event_poll,
+	.release = ib_uverbs_comp_event_close,
+	.fasync  = ib_uverbs_comp_event_fasync,
+	.llseek	 = no_llseek,
+};
+
+static const struct file_operations uverbs_async_event_fops = {
+	.owner	 = THIS_MODULE,
+	.read	 = ib_uverbs_async_event_read,
+	.poll    = ib_uverbs_async_event_poll,
+	.release = ib_uverbs_async_event_close,
+	.fasync  = ib_uverbs_async_event_fasync,
 	.llseek	 = no_llseek,
 };
 
@@ -419,15 +492,15 @@ static void ib_uverbs_async_handler(struct ib_uverbs_file *file,
 	struct ib_uverbs_event *entry;
 	unsigned long flags;
 
-	spin_lock_irqsave(&file->async_file->lock, flags);
-	if (file->async_file->is_closed) {
-		spin_unlock_irqrestore(&file->async_file->lock, flags);
+	spin_lock_irqsave(&file->async_file->ev_file.lock, flags);
+	if (file->async_file->ev_file.is_closed) {
+		spin_unlock_irqrestore(&file->async_file->ev_file.lock, flags);
 		return;
 	}
 
 	entry = kmalloc(sizeof *entry, GFP_ATOMIC);
 	if (!entry) {
-		spin_unlock_irqrestore(&file->async_file->lock, flags);
+		spin_unlock_irqrestore(&file->async_file->ev_file.lock, flags);
 		return;
 	}
 
@@ -436,13 +509,13 @@ static void ib_uverbs_async_handler(struct ib_uverbs_file *file,
 	entry->desc.async.reserved   = 0;
 	entry->counter               = counter;
 
-	list_add_tail(&entry->list, &file->async_file->event_list);
+	list_add_tail(&entry->list, &file->async_file->ev_file.event_list);
 	if (obj_list)
 		list_add_tail(&entry->obj_list, obj_list);
-	spin_unlock_irqrestore(&file->async_file->lock, flags);
+	spin_unlock_irqrestore(&file->async_file->ev_file.lock, flags);
 
-	wake_up_interruptible(&file->async_file->poll_wait);
-	kill_fasync(&file->async_file->async_queue, SIGIO, POLL_IN);
+	wake_up_interruptible(&file->async_file->ev_file.poll_wait);
+	kill_fasync(&file->async_file->ev_file.async_queue, SIGIO, POLL_IN);
 }
 
 void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr)
@@ -505,15 +578,23 @@ void ib_uverbs_event_handler(struct ib_event_handler *handler,
 
 void ib_uverbs_free_async_event_file(struct ib_uverbs_file *file)
 {
-	kref_put(&file->async_file->ref, ib_uverbs_release_event_file);
+	kref_put(&file->async_file->ref, ib_uverbs_release_async_event_file);
 	file->async_file = NULL;
 }
 
-struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file,
-					struct ib_device	*ib_dev,
-					int is_async)
+void ib_uverbs_init_event_file(struct ib_uverbs_event_file *ev_file)
 {
-	struct ib_uverbs_event_file *ev_file;
+	spin_lock_init(&ev_file->lock);
+	INIT_LIST_HEAD(&ev_file->event_list);
+	init_waitqueue_head(&ev_file->poll_wait);
+	ev_file->is_closed   = 0;
+	ev_file->async_queue = NULL;
+}
+
+struct file *ib_uverbs_alloc_async_event_file(struct ib_uverbs_file *uverbs_file,
+					      struct ib_device	*ib_dev)
+{
+	struct ib_uverbs_async_event_file *ev_file;
 	struct file *filp;
 	int ret;
 
@@ -521,16 +602,11 @@ struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file,
 	if (!ev_file)
 		return ERR_PTR(-ENOMEM);
 
-	kref_init(&ev_file->ref);
-	spin_lock_init(&ev_file->lock);
-	INIT_LIST_HEAD(&ev_file->event_list);
-	init_waitqueue_head(&ev_file->poll_wait);
+	ib_uverbs_init_event_file(&ev_file->ev_file);
 	ev_file->uverbs_file = uverbs_file;
 	kref_get(&ev_file->uverbs_file->ref);
-	ev_file->async_queue = NULL;
-	ev_file->is_closed   = 0;
-
-	filp = anon_inode_getfile("[infinibandevent]", &uverbs_event_fops,
+	kref_init(&ev_file->ref);
+	filp = anon_inode_getfile("[infinibandevent]", &uverbs_async_event_fops,
 				  ev_file, O_RDONLY);
 	if (IS_ERR(filp))
 		goto err_put_refs;
@@ -540,64 +616,33 @@ struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file,
 		      &uverbs_file->device->uverbs_events_file_list);
 	mutex_unlock(&uverbs_file->device->lists_mutex);
 
-	if (is_async) {
-		WARN_ON(uverbs_file->async_file);
-		uverbs_file->async_file = ev_file;
-		kref_get(&uverbs_file->async_file->ref);
-		INIT_IB_EVENT_HANDLER(&uverbs_file->event_handler,
-				      ib_dev,
-				      ib_uverbs_event_handler);
-		ret = ib_register_event_handler(&uverbs_file->event_handler);
-		if (ret)
-			goto err_put_file;
-
-		/* At that point async file stuff was fully set */
-		ev_file->is_async = 1;
-	}
+	WARN_ON(uverbs_file->async_file);
+	uverbs_file->async_file = ev_file;
+	kref_get(&uverbs_file->async_file->ref);
+	INIT_IB_EVENT_HANDLER(&uverbs_file->event_handler,
+			      ib_dev,
+			      ib_uverbs_event_handler);
+	ret = ib_register_event_handler(&uverbs_file->event_handler);
+	if (ret)
+		goto err_put_file;
+
+	/* At that point async file stuff was fully set */
 
 	return filp;
 
 err_put_file:
 	fput(filp);
-	kref_put(&uverbs_file->async_file->ref, ib_uverbs_release_event_file);
+	kref_put(&uverbs_file->async_file->ref,
+		 ib_uverbs_release_async_event_file);
 	uverbs_file->async_file = NULL;
 	return ERR_PTR(ret);
 
 err_put_refs:
 	kref_put(&ev_file->uverbs_file->ref, ib_uverbs_release_file);
-	kref_put(&ev_file->ref, ib_uverbs_release_event_file);
+	kref_put(&ev_file->ref, ib_uverbs_release_async_event_file);
 	return filp;
 }
 
-/*
- * Look up a completion event file by FD.  If lookup is successful,
- * takes a ref to the event file struct that it returns; if
- * unsuccessful, returns NULL.
- */
-struct ib_uverbs_event_file *ib_uverbs_lookup_comp_file(int fd)
-{
-	struct ib_uverbs_event_file *ev_file = NULL;
-	struct fd f = fdget(fd);
-
-	if (!f.file)
-		return NULL;
-
-	if (f.file->f_op != &uverbs_event_fops)
-		goto out;
-
-	ev_file = f.file->private_data;
-	if (ev_file->is_async) {
-		ev_file = NULL;
-		goto out;
-	}
-
-	kref_get(&ev_file->ref);
-
-out:
-	fdput(f);
-	return ev_file;
-}
-
 static int verify_command_mask(struct ib_device *ib_dev, __u32 command)
 {
 	u64 mask;
@@ -892,7 +937,8 @@ static int ib_uverbs_close(struct inode *inode, struct file *filp)
 	mutex_unlock(&file->device->lists_mutex);
 
 	if (file->async_file)
-		kref_put(&file->async_file->ref, ib_uverbs_release_event_file);
+		kref_put(&file->async_file->ref,
+			 ib_uverbs_release_async_event_file);
 
 	kref_put(&file->ref, ib_uverbs_release_file);
 	kobject_put(&dev->kobj);
@@ -1091,7 +1137,7 @@ static void ib_uverbs_free_hw_resources(struct ib_uverbs_device *uverbs_dev,
 					struct ib_device *ib_dev)
 {
 	struct ib_uverbs_file *file;
-	struct ib_uverbs_event_file *event_file;
+	struct ib_uverbs_async_event_file *event_file;
 	struct ib_event event;
 
 	/* Pending running commands to terminate */
@@ -1140,21 +1186,20 @@ static void ib_uverbs_free_hw_resources(struct ib_uverbs_device *uverbs_dev,
 	while (!list_empty(&uverbs_dev->uverbs_events_file_list)) {
 		event_file = list_first_entry(&uverbs_dev->
 					      uverbs_events_file_list,
-					      struct ib_uverbs_event_file,
+					      struct ib_uverbs_async_event_file,
 					      list);
-		spin_lock_irq(&event_file->lock);
-		event_file->is_closed = 1;
-		spin_unlock_irq(&event_file->lock);
+		spin_lock_irq(&event_file->ev_file.lock);
+		event_file->ev_file.is_closed = 1;
+		spin_unlock_irq(&event_file->ev_file.lock);
 
 		list_del(&event_file->list);
-		if (event_file->is_async) {
-			ib_unregister_event_handler(&event_file->uverbs_file->
-						    event_handler);
-			event_file->uverbs_file->event_handler.device = NULL;
-		}
+		ib_unregister_event_handler(
+			&event_file->uverbs_file->event_handler);
+		event_file->uverbs_file->event_handler.device =
+			NULL;
 
-		wake_up_interruptible(&event_file->poll_wait);
-		kill_fasync(&event_file->async_queue, SIGIO, POLL_IN);
+		wake_up_interruptible(&event_file->ev_file.poll_wait);
+		kill_fasync(&event_file->ev_file.async_queue, SIGIO, POLL_IN);
 	}
 	mutex_unlock(&uverbs_dev->lists_mutex);
 }
diff --git a/drivers/infiniband/core/uverbs_std_types.c b/drivers/infiniband/core/uverbs_std_types.c
index 9bbc4eb..cd75ac6 100644
--- a/drivers/infiniband/core/uverbs_std_types.c
+++ b/drivers/infiniband/core/uverbs_std_types.c
@@ -102,7 +102,11 @@ void uverbs_free_cq(struct ib_uobject *uobject)
 		container_of(uobject, struct ib_ucq_object, uobject);
 
 	ib_destroy_cq(cq);
-	ib_uverbs_release_ucq(uobject->context->ufile, ev_file, ucq);
+	ib_uverbs_release_ucq(uobject->context->ufile, ev_file ?
+			      container_of(ev_file,
+					   struct ib_uverbs_completion_event_file,
+					   ev_file) : NULL,
+			      ucq);
 }
 
 void uverbs_free_mr(struct ib_uobject *uobject)
@@ -124,6 +128,30 @@ void uverbs_free_pd(struct ib_uobject *uobject)
 	ib_dealloc_pd((struct ib_pd *)uobject->object);
 }
 
+void uverbs_hot_unplug_completion_event_file(struct ib_uobject_file *uobj_file,
+					     bool device_removed)
+{
+	struct ib_uverbs_completion_event_file *comp_event_file =
+		container_of(uobj_file, struct ib_uverbs_completion_event_file,
+			     uobj_file);
+	struct ib_uverbs_event_file *event_file = &comp_event_file->ev_file;
+
+	spin_lock_irq(&event_file->lock);
+	event_file->is_closed = 1;
+	spin_unlock_irq(&event_file->lock);
+
+	if (device_removed) {
+		wake_up_interruptible(&event_file->poll_wait);
+		kill_fasync(&event_file->async_queue, SIGIO, POLL_IN);
+	}
+};
+
+const struct uverbs_obj_fd_type uverbs_type_attrs_comp_channel =
+	UVERBS_TYPE_ALLOC_FD(0, sizeof(struct ib_uverbs_completion_event_file),
+			     uverbs_hot_unplug_completion_event_file,
+			     &uverbs_event_fops,
+			     "[infinibandevent]", O_RDONLY);
+
 const struct uverbs_obj_idr_type uverbs_type_attrs_cq =
 	UVERBS_TYPE_ALLOC_IDR_SZ(sizeof(struct ib_ucq_object), 0,
 				 uverbs_free_cq);
diff --git a/include/rdma/uverbs_std_types.h b/include/rdma/uverbs_std_types.h
index 2edb776..8037b41 100644
--- a/include/rdma/uverbs_std_types.h
+++ b/include/rdma/uverbs_std_types.h
@@ -35,6 +35,7 @@
 
 #include <rdma/uverbs_types.h>
 
+extern const struct uverbs_obj_fd_type uverbs_type_attrs_comp_channel;
 extern const struct uverbs_obj_idr_type uverbs_type_attrs_cq;
 extern const struct uverbs_obj_idr_type uverbs_type_attrs_qp;
 extern const struct uverbs_obj_idr_type uverbs_type_attrs_rwq_ind_table;
diff --git a/include/rdma/uverbs_types.h b/include/rdma/uverbs_types.h
index 48b5e4e..08df9d5 100644
--- a/include/rdma/uverbs_types.h
+++ b/include/rdma/uverbs_types.h
@@ -118,9 +118,21 @@ struct uverbs_obj_fd_type {
 };
 
 extern const struct uverbs_obj_type_ops uverbs_idr_ops;
+extern const struct uverbs_obj_type_ops uverbs_fd_ops;
 
 #define UVERBS_BUILD_BUG_ON(cond) (sizeof(char[1 - 2 * !!(cond)]) -	\
 				   sizeof(char))
+#define UVERBS_TYPE_ALLOC_FD(_order, _obj_size, _hot_unplug, _fops, _name, _flags)\
+	 {.type = {							\
+		.destroy_order = _order,				\
+		.ops = &uverbs_fd_ops,					\
+	 },								\
+	 .obj_size = (_obj_size) +					\
+		UVERBS_BUILD_BUG_ON((_obj_size) < sizeof(struct ib_uobject_file)), \
+	 .hot_unplug = _hot_unplug,					\
+	 .fops = _fops,							\
+	 .name = _name,							\
+	 .flags = _flags}
 #define UVERBS_TYPE_ALLOC_IDR_SZ(_size, _order, _hot_unplug)		\
 	 {.type = {							\
 		.destroy_order = _order,				\
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html